fix(orchestration-v2): make stuck and resurrected run states unrepresentable

[Iweisc]

Problem

State-management bugs in orchestration all share one root cause: a persisted status that claims something ("this run is working") with nothing forcing the claim to stay true. Concretely at HEAD:

• A fiber dying without finalizing its run left the thread at "Working …" forever; recovery only ran at boot, so every in-process leak persisted until a restart.
• The effect outbox wrote lease_expires_at but never read it — a hung executor wedged its thread's effect queue for the life of the process.
• Nothing (not the event log, not the projections) prevented a buggy write from moving a terminal run back to a live status.
• The thread shell derived its status from the newest run row, so a cancelled queued run (e.g. after promoting a queued message to a steer) could outrank the actually-running turn and anchor the "Working for Xm" timer to the queued message's stale requestedAt.

Fix

Three mechanisms enforce one invariant — every non-terminal run is owned by live work, covered by durable queued work, or on a bounded clock toward terminalization; terminal runs stay terminal:

1. Transition gate — RunLifecycle.ts is the single source of truth for legal status transitions; EventSink validates every status-bearing event inside the commit transaction (folding statuses through the batch) and rejects illegal commits. Migration 045 adds matching SQLite triggers as a storage backstop, with a replay exemption so projection rebuilds of pre-gate history still work. The policy was derived from an exhaustive sweep of every status writer in the codebase: provider-owned subagent rows and subagent/root_turn nodes are deliberately reopenable and stay ungated; checkpoint rollback's completed → rolled_back stays legal.
2. Liveness leases — the provider-event ingestion fiber (and launch preparation) holds a renewed lease row per run; the renewal fiber is scoped to the work, so it dies with its owner on any exit path and the lease expires on its own. No code path has to "remember to clear" anything.
3. Continuous janitor — started right after boot recovery (which stays as the sweep's synchronous first tick): settles expired outbox claims (requeue replay-safe, fail process-bound, per-claim lease tokens so an abandoned claimer can't settle a reclaimed row) and cancels orphaned runs using the boot sweep's cascade, sparing leased, queued-behind, effect-covered, and recently-active work — and never touching live idle provider sessions.

The thread shell now picks the live run (lowest-ordinal non-terminal) over the newest row, fixing the queued-steer "Working 17m" symptom at the derivation layer as well.

Timings are env-tunable (PIKU_ORCH_RUN_LEASE_TTL_MS, PIKU_ORCH_RUN_LEASE_RENEW_MS, PIKU_ORCH_RUN_JANITOR_INTERVAL_MS, PIKU_ORCH_RUN_JANITOR_GRACE_MS, PIKU_ORCH_EFFECT_LEASE_MS). Internals doc: docs/internals/run-lifecycle-guarantees.md.

Tests

• RunLifecycle.test.ts: transition-policy unit coverage (absorbing terminals, rollback/reopen exemptions).
• RunLifecycleGuarantees.test.ts: gate rejection (single event + folded batch) with no partial persistence, rollback retirement, storage-trigger rejection + replay exemption, live-run shell derivation, janitor sweep (orphan terminalized; leased/effect-covered/fresh work spared; shield removal makes work sweepable), outbox expired-lease settlement with stale-token refusal.
• Existing orchestration-v2 suites (foundation persistence, effect worker, recovery, projection store, run execution, thread launch, checkpoint rollback, runtime layer, migrations) pass.

Known follow-up: a replay fixture for a successful queued-message promote-to-steer (the only existing promote test covers the rejected/archived path).

Implemented by Claude Fable 5 via Claude Code.

🤖 Generated with Claude Code

---

Note

High Risk
Release and deploy workflows lose Windows builds, relay auto-config, and stable-release paths—operational behavior changes beyond naming. Broad env var and home-dir renames can break local/CI setups until secrets and docs are aligned.

Overview
This diff rebrands the repo from T3 Code to Piku Code across agent skills, devcontainer/cursor notes, .env.example, .gitignore (.pikucode), and planning docs—replacing T3CODE_* / ~/.t3 with PIKU_* / ~/.pikucode, t3-sqlite-state with piku-sqlite-state, and hosted defaults like app.pikucode.dev.

Agent workflows drop the OpenAI iOS debugger/simulator-browser skills and the test-t3-app / test-t3-mobile skills, and add test-piku-app for isolated web dev, pairing-token recovery, and SQLite fixture seeding.

CI and release shift package filters to @piku/* and piku, add build-essential on the test job so ACP process-tree fixtures compile, and remove mobile native lint, EAS preview/production, showcase screenshots, and the T3 Connect relay deploy workflow. The release workflow becomes nightly-only (scheduled + manual dispatch): no stable tag path, no Windows desktop builds or WSL node-pty prebuild, no relay deploy job for public Clerk/relay config (uses PIKU_* GitHub vars instead), no post-release version-bump finalize job, and CLI publish always uses the nightly npm tag; resource monitor binaries rename to piku-resource-monitor.

Issue templates no longer list apps/mobile. A new plan doc adds thread lineage / lazy context transfer for orchestration v2 (separate from the run-lifecycle work described in the PR title/description if that lives in other commits).

^{Reviewed by Cursor Bugbot for commit 990ecc0. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add orchestration-v2 run lifecycle with unrepresentable stuck and resurrected states

• Introduces a new orchestration-v2 subsystem under apps/server/src/orchestration-v2/ with services for run lifecycle management, provider session management, thread forking, checkpointing, and recovery — designed so stuck and resurrected run states cannot be represented in the type system.
• Adds 10+ new SQLite migrations (036–045) covering orchestration-v2 schema, subagents, effect cancellation, scheduled tasks, and run lifecycle guarantees.
• Adds new contract modules (orchestrationV2.ts, scheduledTask.ts, rpc.ts, applicationEvent.ts, etc.) and a ./legacy-orchestration subpath export on @piku/contracts for the old orchestration schema.
• Renames all internal package namespaces, service identifiers, cookie names, event names, and URNs from t3/t3tools to piku/@piku across the entire monorepo.
• Behavioral Change: session cookie name changes from t3_session to piku_session, the environment descriptor endpoint moves from /.well-known/t3/environment to /.well-known/piku/environment, and the WebAssembly import key changes from t3_write_pty to piku_write_pty — all of which are breaking for existing clients.

📊 Macroscope summarized 990ecc0. 185 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 87402f57-0b84-43f1-8bc1-4f5677717f19

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 990ecc0. Configure here.}

[cursor]

Manual release dispatch skips jobs

Medium Severity

After this change, workflow_dispatch is the only manual release trigger, but preflight still needs check_changes, which runs only on schedule. GitHub Actions skips dependent jobs when a needed job is skipped, so a manual run never reaches build or publish.

 

^{Reviewed by Cursor Bugbot for commit 990ecc0. Configure here.}

[macroscopeapp]

Effect service conventions review of the orchestration run-lifecycle changes. The new RunJanitorService.ts, RunLifecycle.ts and the EventSink gate wiring largely follow the conventions (inline service interface, make + layer, exported Schema.is predicate, multi-value operation discriminator). Four findings, all in RunLeaseService.ts and EventSink.ts.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Effect Service Conventions — findings

Inline anchoring failed, so the findings are listed here with file/line references.

1. apps/server/src/orchestration-v2/RunLeaseService.ts:55-70 — standalone RunLeaseServiceV2Shape

The service interface should be declared inline in the Context.Service call rather than kept as a standalone shape type (the sibling RunJanitorV2 added in this PR already does this). Consider moving the withRunLease member (with its doc comment) into the Context.Service<RunLeaseServiceV2, { ... }>() type argument, dropping the RunLeaseServiceV2Shape export, and referring to the inferred shape as RunLeaseServiceV2["Service"]["withRunLease"] at line 133.

2. apps/server/src/orchestration-v2/RunLeaseService.ts:72 — no exported make

This module owns construction, so it should export a real make and keep the canonical order (tag, make, then layer), as RunJanitorService.ts does in this same PR:

export const make = Effect.gen(function* () { /* current layer body */ });

export const layer: Layer.Layer<RunLeaseServiceV2, never, SqlClient.SqlClient> = Layer.effect(
  RunLeaseServiceV2,
  make,
);

3. apps/server/src/orchestration-v2/RunLeaseService.ts:37-53 — process.env read in a module-scope helper

envMs bypasses the server's effect/Config convention (see apps/server/src/cli/config.ts, telemetry/AnalyticsService.ts) and keeps the timings out of the Effect environment. Consider a Config.all({ ttlMs: Config.integer("PIKU_ORCH_RUN_LEASE_TTL_MS").pipe(Config.withDefault(30_000)), ... }) descriptor yielded where the timings are produced, keeping the Context.Reference as the test/override seam. The same applies to the module-load IIFE at apps/server/src/orchestration-v2/EffectWorker.ts:336-341.

4. apps/server/src/orchestration-v2/EventSink.ts:45-61 — EventSinkIllegalTransitionError

• detail is prose derived from the attributes the error already carries (kind, from, to), and message is built from it. Consider dropping detail and deriving the message from the structural fields directly, e.g. return `Rejected orchestration V2 event: a ${this.kind} ${this.entityId} in terminal status "${this.from}" cannot move to "${this.to}".`;
• The error is not part of the declared error channel (EventSinkV2Error = Schema.Union([EventSinkWriteError, EventSinkStreamError])), so the boundary mappers at lines 578-627 bury this structured domain failure inside EventSinkWriteError.cause as an opaque defect — which is why RunLifecycleGuarantees.test.ts has to inspect .cause with a predicate instead of using Effect.catchTags. Consider adding EventSinkIllegalTransitionError to the union and passing it through the boundary mappers, wrapping only genuinely lower-level failures.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Approvability

Verdict: Unable to determine

Macroscope's correctness review was unable to post its findings for this PR. Approvability cannot proceed without a successful correctness review.

^{You can customize Macroscope's approvability policy. Learn more.}

[Iweisc]

gh pr create silently resolved the fork's upstream remote as the base repo.....this was meant for my own fork. Sorry guys.