Repository navigation
perf(daemon): resume the transcript rescan from a checkpoint - #1314
Merged
Merged
Conversation
The daemon rebuilds several derived session fields from transcripts at
startup — seq counter, message count, last-message/last-error snippets,
token tally (spec 0103), context gauge (spec 0104), fleet token history
(spec 0167) — so records written before a field existed, or that lagged a
crash, heal themselves.
It did that by reading every transcript in full, serially, before binding
the IPC socket. That is proportional to all history ever recorded. On a
fleet with 340 transcripts totalling 7 GB it put ~30s between launch and
the socket existing, every single start:
loaded config 02:12:34.387
listening 02:13:01.626 (+27.2s)
`construct daemon start` waits 5s, so it reported a daemon that had failed
while it was in fact still reading (#1313).
Checkpoint the fold. Each session stores its derived state in `scan.json`
next to the transcript, with the byte length that state reflects; the next
start `stat`s the file and reads only what was appended past that offset.
Transcripts are append-only, so a recorded length is always a line
boundary and equal lengths mean there is nothing to read.
The checkpoint is a discardable cache, never an authority: missing, stale
past EOF, or written at another version all fall back to a full walk.
Anything carried across the resume point is part of it — including the
model in effect where the scan stopped, without which usage recovered
after a resume attributes differently than a full walk (spec 0167).
Samples are pruned to the retention window as they are written, so the
file cannot accumulate history no consumer would accept back.
Checkpoints are written during startup only, before any adapter resumes
and while nothing can append — sampling a length and a fold state as one
consistent pair needs that quiet, or the race skips events in one
direction and double-counts them in the other.
Also raise `daemon start`'s readiness budget to the 60s the auto-start
path already allows: the first start after this change still pays one
full walk to seed the checkpoints.
Startup now logs what it read, so a slow start is a readable fact:
transcript scan complete sessions=340 up_to_date=338 tail=2
full=0 mib_read=3 elapsed_ms=214
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The daemon rebuilds several derived session fields from transcripts at
startup — seq counter, message count, last-message/last-error snippets, token
tally (spec 0103), context gauge (spec 0104), fleet token history (spec 0167)
— so records written before a field existed, or that lagged a crash, heal
themselves.
It did that by reading every transcript in full, serially, before binding
the IPC socket. That is proportional to all history ever recorded.
On my fleet (340 transcripts, 7.0 GB, largest 294 MB) it put ~30s between
launch and the socket existing — on every start:
construct daemon startwaits 5s, so it reported a daemon that had failedwhile it was in fact still reading:
Note the auto-start path (
ensure_daemon_running) already polls for 60s witha comment about "reported ~30s slow starts" — this exact problem was hit
before, fixed there, and missed in
daemon start.The fix
Checkpoint the fold. Each session stores its derived state in
scan.jsonnextto the transcript, along with the byte length that state reflects. The next
start
stats the file and reads only what was appended past that offset.Transcripts are append-only, so a recorded length is always a line boundary
and equal lengths mean there is nothing to read.
The checkpoint is a discardable cache, never an authority — missing, stale
past EOF, or written at another version all fall back to a full walk. Deleting
every
scan.jsonon disk costs one slow start and nothing else.Two details that are easy to get wrong:
cost sample is attributed to the model in effect at that point in the
transcript (spec 0167). A tail scan starts after the
ModelChangedthatset it, so without persisting it, usage recovered after a resume would be
silently unattributed where a full walk attributes it correctly.
and while nothing can append. Sampling a length and a fold state as one
consistent pair needs that quiet: race it one way and events are skipped
forever, race it the other and they are counted twice.
Samples are pruned to the retention window as written, so the file cannot
accumulate history no consumer would accept back.
Also raises
daemon start's readiness budget to the 60s auto-start alreadyallows — the first start after this change still pays one full walk to seed
the checkpoints.
Measured, against a copy of that real 7 GB fleet
APFS copy-on-write clones of all 341 real session transcripts (distinct
inodes, so nothing could reach the live data), isolated daemon dirs:
transcript scan complete sessions=339 up_to_date=336 tail=2 full=1 mib_read=0 elapsed_ms=24Total checkpoint cost on disk: 0.2 MB across 339 files.
Correctness at scale
Booted checkpointed, snapshotted every summary, deleted all checkpoints,
booted cold again, diffed:
Tests
7 unit tests in
session::scan— the central one asserts field-by-field thata tail scan lands exactly where a full scan would; plus resume-carries-model,
current-checkpoint-reads-nothing, past-EOF rejection, version rejection,
window pruning, missing transcript.
1 end-to-end test through
SessionManager::new. "Did not re-read" is assertedthe only way it can be observed from outside: the checkpoint is doctored to a
value the transcript does not support, and the next boot is required to report
it — a boot that rescanned would overwrite it.
Full workspace suite: 2863 passed, 0 failed.
Specs
specs/0211-transcript-scan-checkpoint.md.specs/0167said "Nothing new is persisted for it" — that free ride existedonly because the daemon walked every transcript anyway. Updated, along with
the matching claim in
cost_history.rs's module doc.Binaries
Daemon-side change; the relevant binary is
construct:.claude/worktrees/transcript-scan-checkpoint/target/debug/constructNo recording — daemon startup logic with no user-visible TUI surface.