Transient "database is locked" during an event write fails the run but leaves the Claude process running unsupervised #15542

Activity

1. juliusmarminge commented on Oct 4, 2026

   

   juliusmarminge

   on Oct 4, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Triage

   Thanks for the careful diagnosis and the timeline, @areidyOTH. I confirmed this on main (40596072), which still has the code from your nightly (73799330). There are two separate bugs here, and it isn't a duplicate of #15447, #5099, or #6097.

   What I found

   The lock fails immediately instead of waiting. packages/shared/src/nodeSqliteClient.ts never sets beginTransaction, so transactions open with a plain deferred BEGIN. An event write reads first (to allocate the turn-item position) and then writes. When a deferred transaction tries to upgrade from its read lock to a write lock while another connection is writing, SQLite returns SQLITE_BUSY right away and skips the busy handler, so busy_timeout never applies.

   • Tested with node:sqlite and PRAGMA busy_timeout = 5000: the deferred upgrade failed in 0 ms with database is locked, while BEGIN IMMEDIATE waited about 5 s as expected.
   • Your 458 ms EventSink.write span ending in a rollback matches the immediate failure. A timeout expiry would take at least 5 s.
   • Upstream @effect/sql-sqlite-node uses BEGIN IMMEDIATE on writable connections. The local client doesn't.
   • [Bug]: SQLite busy_timeout is never set — concurrent CLI/server writes fail with "database is locked" #5099 only added busy_timeout. The other writer here was a separate process (t3 auth session issue / revoke), which is exactly what the timeout should wait out.

   A failed write ends the run but not the Claude process. In RunExecutionService.startRootRun, a failed ingest writes a terminal failure through makeProviderFailure({ class: "unknown" }), which is the "Provider error / Provider turn failed" item you saw. The fiber then just closes its event subscription and never calls interruptTurn.

   • ClaudeAdapterV2.interruptTurn is what calls query.interrupt and query.close, and nothing on this path calls it, so startTurn keeps running.
   • Once the run is failed with no running provider turn, run.interrupt returns "Run … is not interruptible," so Stop can't reach the process either.
   • The process could keep calling tools because the thread was in full-access mode (bypassPermissions).

   #15447 has the same result for Cursor (run marked failed, provider keeps going) with a different trigger.

   Likely fix area

   • Transactions: one option is starting event-write transactions with BEGIN IMMEDIATE so a brief lock waits on busy_timeout. Retrying the write is another.
   • Orphaned provider: when ingest fails, the run could interrupt and close the live provider session. That would probably cover [Bug]: Cursor run continues after T3 marks parent and native subagents failed #15447 too.

   A maintainer will decide on the fix direction.
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Oct 4, 2026
3. whatever1111 commented on Oct 4, 2026

   

   whatever1111

   on Oct 4, 2026

   More actions

   Another occurrence on the same nightly, plus a variant where the run never leaves running

   Same build as this report (0.0.46-nightly.20261004.2644, 7379933), headless t3 serve under a systemd user service on Linux x86_64 (kernel 7.0.0-22), Claude Agent SDK provider (Claude Code 2.1.288) plus Codex. The other writer is the same too: a local MCP bridge runs t3 auth session issue / revoke around every API call it makes.

   Frequency. orchestration V2 provider event ingestion failed was logged 8 times between 08:07 and 09:31 UTC on 2026-10-04, across 8 different runs (Claude and Codex, root threads and delegated tasks). In 7 of the 8, a CLI auth session issue/revoke write had landed 11–1343 ms earlier (6 of them within 600 ms). For the remaining one there was no CLI auth write in the preceding 3 s, so the other writer there is unknown.

   Trace. 5 of the 8 were still in the rotated trace files, and all 5 fail the same way, fast:

   • orchestrationV2.EventSink.write (from providerTurnStart.start) → sql.transaction → sql.execute of INSERT INTO orchestration_v2_turn_item_positions (thread_id, turn_item_id, ordinal) SELECT ?, ?, COALESCE(MAX(ordinal), ?) + 1 FROM … → TurnItemPositionStoreError ← SqlError ← Error: database is locked.
   • The failing statement returned after 0.2–1.4 ms (the whole EventSink.write span took 2.8–4.1 ms). That is the immediate busy return described in the triage, not a 5 s busy_timeout expiry.

   Variant: the failed terminal is lost as well. In 4 of those 5, the compensating orchestrationV2.EventSink.writeIfRunCurrent ran 5–26 ms later and failed with the same database is locked. Those runs never got a terminal at all, so they stayed running instead of showing "Provider turn failed". An interrupt request on one of them did not settle it either. Only 1 of the 5 (08:52:40Z) got its failed terminal written. This is the path #14856 addresses.

   In one of the stuck cases (a root Claude thread, failure at 09:20:51Z):

   • The Claude CLI kept working and finished the turn normally about 6.5 minutes later (final answer and Stop hook at 09:27:33Z in its transcript). None of it reached the projection, and the UI stayed on "Thinking".
   • The user's next message (09:29:05Z) was turned into a steer of the stuck run. The provider-turn.steer effect failed 5 times with … is not the active turn. A delegated-task completion notice at 09:34:57Z also failed to steer.

   In another (a root Claude thread, failure at 09:31:06Z), the provider session was still alive: a later steer was delivered and the agent kept working, but nothing it produced after 09:31:06Z was shown.

   Recovery. Only t3 service restart cleared them. Startup reconciliation cancelled the stuck runs, and with them the in-flight delegated children, whose parents were then woken by the cancellation notices.

   Happy to re-test once a nightly includes #15488. Since the failed terminal can be lost in the same lock window, the retry and interrupt handling in #14856 still seems needed on top of that.