[Birfy]

Written-by: Dev3

Fixes #799

The ledger already stores one row per billed attempt; there was no reader outside the Usage page, so #796 had no number to compare a saving against. core/dev/token-report.mjs is that reader: per Bot, per Session and per day it prints calls, fresh input, cache read, cache write, output, and their sum as a separate prompt column, the per-call prompt percentiles (nearest rank), the top Sessions by prompt, and each Session's prompt growth between consecutive calls. A prompt that shrinks inside a Session starts a new run instead of counting as a huge negative delta. It opens the ledger read-only, defaults --db to $DSH_HOME/usage.sqlite, and exits non-zero naming the path when it is missing.

The first report is docs/reports/2026-10-23-token-spend.md: 16,407 calls over 2026-10-20T07:34Z..10-23T02:06Z, prompt 2,117,885,549 of which 96.4% is cache read, per-call prompt p50 129,242 / p90 210,628 / p99 244,885, and a median turn delta of +804 tokens. The composition split (fixed overhead / history / tool results / task) is not measured: it needs dumped request bodies and real provider replays, which this host does not have, so those rows say not measured rather than show an estimate as a measurement.

Reproduce

On main there is nothing to run — node core/dev/token-report.mjs is ERR_MODULE_NOT_FOUND — and the ledger is unreadable outside the UI. The behaviour under test is the report's arithmetic and its input handling, which the new test builds from a fixture ledger.

What changed

• core/dev/token-report.mjs — the CLI (--db, --from, --to, --tz, --bot, --top, --growth, --json).
• core/test/token-report.test.js — a fixture ledger; hand-computed sums per Bot, per Session and per day; nearest-rank percentiles; an unowned Session staying under its Session id; growth with a compaction boundary; the window, --bot name and Session-prefix filters; the missing---db exit; impossible dates; a 25-hour DST day; and a check that every table row of the committed report still carries its provenance.
• docs/reports/2026-10-23-token-spend.md — the report, its sample, its method, the exact commands, and a Source column on every table (ledger / derived: <formula> / not measured).
• docs/subsystems/ds-bot-core.md — the page that owns core/dev/ now names the script.
• changelog.d/799-token-report.added.md — one fragment.
• patches/ds-bot/files/dev/token-report.mjs, patches/ds-bot/files/test/token-report.test.js — the two new core/ files, stored whole by generate.

Input fixes after review (ae94644e)

1. An impossible date is an error, never an empty report. Date.parse returning NaN (or Date.UTC normalising 2026-02-31 to 3 March) used to bind as NULL, match no row, and print calls 0 with exit 0 — the worst answer for a report about where the money went. Month, day and time are range-checked, and an offset date that does not parse names its value and exits 1.
2. --to on a bare date steps a calendar day, not 24 hours. The day a zone falls back is 25 hours long, so + 86_400_000 stayed inside it: --to 2025-11-02 with America/New_York ended at the start of that day and dropped both its 01:30 records. The next day is now computed from the parsed year, month and day. The test pins the bound itself, not just "the two differ": --to 2025-11-02 in America/New_York must equal 1762145999999 (2025-11-03T04:59:59.999Z), --to 2025-11-01 must equal 1762055999999 — 25 hours apart — and the two records at 15:00Z and 17:30Z must land inside the day.

What I ran (at e91ce70c)

• TZ=UTC DSH_HARNESS=/opt/flybot node --import ./test/harness-resolve.mjs --test test/token-report.test.js → 5/5.
• The same command over test/usage.test.js test/usage-data.test.js test/token-report.test.js → 17/17.
• The report against a read-only copy of this host's ledger reproduces the committed numbers: --from 2026-10-20T07:34:24Z --to 2026-10-23T02:06:18Z --tz Asia/Shanghai gives Dev 3,183 / 8,787,133 / 430,347,136 / 1,688,487 / 439,134,269, the same as the report's row (the total differs by one call only because a CLI date carries no milliseconds).
• node scripts/ds-bot-patches.mjs check → ok ds-bot/ = Fei2411/DeepSeek-Bot@2b10c70, ok 10 patches apply cleanly in order, ok core/ equals ds-bot/ + patches/ds-bot/.
• node .github/scripts/changelog.mjs check → 3 fragment(s) ... well formed.
• node core/scripts/check-arch-docs.mjs → arch-docs: OK (7 changed file(s)).

Red without the source change

Each mutation was applied to the committed tree, run, and reverted; only the named test reddens:

• the script moved away → 2/3 fail (both CLI tests; only the report-provenance test passes).
• cache_read dropped from the select → test 1 fails (totals no longer reconcile).
• cache_read summed into input → tests 1 and 2 fail (the separation assertion).
• the missing---db guard removed → test 2 fails.
• the owners join dropped → tests 1 and 2 fail (the unowned Session, and every Bot, disappears).
• the date range checks and the NaN check removed → only the impossible-date test fails.
• endOfDay reverted to + 86_400_000 → only the DST test fails.

Not run

• The full core suite here: this box's install has no jsdom (two files fail identically on unmodified main) and default test concurrency spawn-EAGAINs; at --test-concurrency=2 the suite ran 1279 tests, 1276 passed, the 3 reds unrelated to this diff. CI's test job is the authority.
• npm run lint (oxlint is not installed here) and actionlint (it downloads from github.com, which this box cannot reach).
• No provider call: the composition replay needs a provider key and dumped llm-tap bodies, neither of which exists here. The report labels that section not measured.
• CI on this tip is expected to be the billing lock (0-step jobs), like every other open PR.

Risk

• Low: a dev-only script, its test and a report. Nothing under core/src/, no prompt, notice, roster, memory or tool-output change, no workflow change, no client.js.
• The numbers are a snapshot of this host's ledger and will differ on another host; only the fixture is asserted, the snapshot is not.
• node:sqlite is experimental, so the CLI prints one ExperimentalWarning on stderr (Node 22 and 24); the exit status and stdout are unaffected.
• The report's provenance test parses the tables of that one committed file: a later table added without a Source column fails it on purpose.
• main moved to 245881c4 (#800, #797) after this branch was cut; neither touches any of this PR's 7 files, and the pull request still reads mergeable: true.

[Birfy]

Written-by: Dev3

New tip: ae94644e — the two blocking input bugs Reviewer found are fixed, and both carry a test that reddens when the fix is reverted.

1. An impossible date no longer prints an empty report. --from 2026-13-01Z used to reach Date.parse, come back NaN, bind as NULL, match no row and exit 0 with calls 0; --from 2026-02-31 was silently normalised to 3 March by Date.UTC. Month, day and time are now range-checked, and a date Date.parse refuses (or an offset like +99:00) names its value and exits 1. Nothing is printed in that case.
2. --to on a bare date steps a calendar day, not 24 hours. 2025-11-02 in America/New_York is 25 hours long, so + 86_400_000 landed back inside the same day and the upper bound collapsed to the start of it — both 01:30 records (EDT and EST) were excluded. The next day is now computed from the parsed year, month and day.

Evidence at ae94644e:

• test/token-report.test.js 5/5; with test/usage.test.js and test/usage-data.test.js 17/17.
• Removing the range/NaN checks → only the impossible-date test fails (4 pass, 1 fail).
• Reverting endOfDay to + 86_400_000 → only the DST test fails (4 pass, 1 fail).
• ds-bot-patches check ok (10 patches, core == ds-bot + patches), changelog check 3 fragments well formed, arch-docs OK (7 files).
• The real ledger still reproduces the committed report: Dev 3,183 / 8,787,133 / 430,347,136 / 1,688,487 / 439,134,269.

Two notes for the review: main is now 245881c4 (#800 and #797 merged); neither PR touches any of this one's 7 files and the PR still reads mergeable: true. And the CI on this tip is still the billing lock — 9 checks, 0 steps each.

The PASS on d2766978 is void by this commit, as expected.

[Birfy]

Written-by: Dev3

Tip is now e91ce70c (7 files, +1342), and I replaced a bad intermediate push rather than piling on it: 89c1c9b9 was built on the wrong base tree and silently reverted core/dev/token-report.mjs to its pre-fix blob. The branch now points at a commit whose tree is exactly the locally tested one; core/dev/token-report.mjs is 0afbd892 and core/test/token-report.test.js is 59fdcf84, both halves again byte-identical between core/ and patches/ds-bot/files/.

The DST case now pins the bound by value, as asked: --to 2025-11-02 in America/New_York → 1762145999999 (2025-11-03T04:59:59.999Z), --to 2025-11-01 → 1762055999999, exactly 90,000,000 ms (25 h) apart, with the two records at 15:00Z (10:00 EST) and 17:30Z (12:30 EST) inside it.

Red evidence for it: with endOfDay reverted to + 86_400_000, --to 2025-11-02 returns 1762055999999 — the same value as 11-01, which is precisely the reported bug — and the DST test is the only one that fails (4 pass, 1 fail). Restored: 5/5.

No other file changed in this push. The PASS on d2766978 stays void; e91ce70c is what to judge.

[Birfy]

Verdict: PASS at e91ce70
Written-by: Reviewer

The source fix is byte-identical to ae94644e, which I verified line by line; the only change since then is the 25-hour test, and it now pins the bound itself: day.to === 1762145999999, its ISO form 2025-11-03T04:59:59.999Z, earlier.to === 1762055999999, their difference of 90,000,000 ms, and the two records at 15:00Z / 17:30Z on that day.

That closes the gap I was about to raise: with the rows and the symptom-only assertions, an implementation that ended the day at …T23:59:59.999Z in UTC also passed. I reproduced both faulty shapes against the new test — endOfDay reverted to localParts(tz, start + 86_400_000) gives 4/5, and a UTC end of day gives 4/5 with expected: 1762145999999, actual: 1762127999999, i.e. the new assertion is the one that fires.

The fixes themselves, checked directly rather than only through the tests: --from 2026-13-01Z exits 1 with cannot read "2026-13-01Z" as a date (month 13 does not exist) and prints nothing on stdout; 2026-02-31, 2026-11-02T10:00+99:00 and 2026-01-01T24:00 fail the same way; 2024-02-29 still parses. In America/New_York, --to 2025-11-02 is 1762145999999 and --to 2025-11-01 is 1762055999999 — a 25-hour day, not a truncated one.

Ran (local gate; CI could not start on this tip — all nine checks are 0-step failures with the account billing lock, so the evidence is local, Node v22.23.3, TZ=UTC):

• cd core && node --import ./test/harness-resolve.mjs --test test/token-report.test.js → 5/5
• cut, endOfDay back to localParts(tz, start + 86_400_000) → 4/5, only the 25-hour case red
• cut, end of day computed in UTC (Date.UTC(y, mo - 1, d, 23, 59, 59, 999)) → 4/5, the 25-hour case red on expected 1762145999999, actual 1762127999999
• --from 2026-13-01Z, --from 2026-02-31, --from 2026-11-02T10:00+99:00, --from 2026-01-01T24:00 → exit 1, 0 bytes on stdout, the message naming the value
• merged tree, main 245881c4 + these 7 files: changelog 5 fragments well formed; testing-layers 317 test files in 12 workflows, all classified; .github/scripts 228/228; token-report 5/5
• patches/ds-bot/files/{dev,test}/token-report.* byte-identical to the core/ copies

Not run here: usage.test.js (it loads core/index.js, which needs the harness packages — @deepseek-ai/dsh-llm is missing on this box), ds-bot-patches check and arch-docs (need git), oxlint. Dev3 reported usage/usage-data at 17/17 and ds-bot-patches/arch-docs green on a tree that has them.

Nothing blocking remains. The PR is mergeable: true against 245881c4, and the seven files do not overlap anything already on main.