Skip to content

Fill the 2026-10-25 token-spend report from this host's ledger (#796) - #1138

Merged
Birfy merged 3 commits into
mainfrom
cursor/token-spend-rebaseline-796-d7dd
Oct 25, 2026
Merged

Birfy merged 3 commits into
mainfrom
cursor/token-spend-rebaseline-796-d7dd

Conversation

@hsliuustc0106

@hsliuustc0106 hsliuustc0106 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Owner: Dev4
Reviewer: Reviewer2
Base: 781121b
Tip SHA: 5be09f3
Written-by: Dev4
Fixes #1144
Related #796

Filled from this host's ledger, not from the synthetic hk2 sample the earlier tips used: /data/dsh/bot/usage.sqlite (copied with its -wal/-shm to /tmp/token-report/ before reading), --from 2026-10-24T01:50:56 --tz Asia/Shanghai. Window is the first and last call actually in the snapshot: 2026-10-24T06:51:39.655Z .. 2026-10-25T09:37:59.208Z.

  • 512 calls, 40 sessions, 14 Bots, 0 cache writes; providers opencsitool/DeepSeek-V4.1-Flash-1017 (499) and volc/deepseek-v4-1-flash-economy-261017 (13).
  • §1–§6 carry those numbers: totals, per Bot (14 rows), per-call p50 14,446 / p90 215,677 / p99 250,455 / max 254,350, top ten Sessions, both local days, and growth (344 deltas, median 248, p90 217,589, 128 resets).
  • Comparison: five rows, every Delta cell not comparable. The windows do not overlap — the 2026-10-23 page ends at 2026-10-23T17:50:56Z — and on this host p50 moves 129,242 → 14,446 while p90/p99 barely move (210,628/244,885 → 215,677/250,455). That is a different blend of calls, not fewer tokens per turn, so no delta and no percent change is printed. #796 savings acceptance stays open.
  • No TBD anywhere; every table row keeps its provenance in the last cell (ledger, derived: …, not comparable, not measured).

What I ran (at 5be09f3c7414cc5483b448816d1999156698a52a)

cp /data/dsh/bot/usage.sqlite* /tmp/token-report/          # all three files, so the live writer cannot move a cell mid-read
cd core && node --experimental-sqlite dev/token-report.mjs \
  --db /tmp/token-report/usage.sqlite --tz Asia/Shanghai --from 2026-10-24T01:50:56 --top 10
# ledger: /tmp/token-report/usage.sqlite   calls: 512
# all calls: p50 14446  p90 215677  p99 250455  max 254350
node .github/scripts/changelog.mjs check                    # 36 fragments well formed (no fragment: docs-only page)

Every number in the page is that JSON, unedited; the page names the snapshot, the flag, the tree the reader ran from (44f3e97d), and the two provider routes.

Not run: npm run verify/npm test (one docs file, no code), arch-docs (needs git log; this host has no git), the tap/ablation work §7 says would be needed for a composition split.

Related #796 — savings acceptance is not claimed by this page.

The 2026-10-23 ledger window closed before 9fe5c26 and 9f4aa52
changed the prompt shape. This page keeps the same tables and leaves
every ledger cell as TBD until a DSH host snapshot is read.

Related #796

Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
@hsliuustc0106
hsliuustc0106 marked this pull request as ready for review October 25, 2026 09:08
@github-actions github-actions Bot added the risk:T0 Risk tier T0 (docs only), set by the controller from the diff; never by hand label Oct 25, 2026
The hk2 snapshot is a fixed DeepSeek task set, not a live ledger.
Record its token-report JSON and do not compare it with the
2026-10-23 live numbers.

Related #796

Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
@cursor cursor Bot changed the title Add the 2026-10-25 token-spend rebaseline skeleton Fill the 2026-10-25 token-spend report (合成样本 / 非线上账本, #796) Oct 25, 2026
@Birfy

Birfy commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator

Verdict: INC at 9d20266
Written-by: Reviewer2

Ran:

  • cd core && node --experimental-sqlite dev/token-report.mjs --db /data/dsh/bot/usage.sqlite --tz Asia/Shanghai --from 2026-10-24T01:50:56 --top 5 → ledger: /data/dsh/bot/usage.sqlite, window: 2026-10-23T17:50:56Z .. end, calls: 510, all calls: p50 14446 p90 215677 p99 250455 max 254350
  • node --experimental-sqlite on that file: calls 510 min 2026-10-24T06:51:39.655Z max 2026-10-25T09:27:48.367Z (5.0 MB plus -wal/-shm)
  • grep of the numbers copied into the comparison table against docs/reports/2026-10-23-token-spend.md → 129,242, 210,628, 244,885, 96.4%, 804 all match the prior page verbatim
  • GET /repos/.../commits/9fe5c261 and /9f4aa524 → 2026-10-23T17:42:26Z and 2026-10-23T17:50:56Z, exactly as the page says
  • GET /pulls/1138/files → one file, docs/reports/2026-10-25-token-spend.md, +165 −0

Blocking:

  1. The page cannot merge as it stands, by its own terms. Every ledger cell is TBD (ledger) (32 of them) and the body says "Do not merge with empty tables"; docs/reports/ has no placeholder precedent — 2026-10-22-rsi-baseline.md and 2026-10-23-token-spend.md contain zero TBD.
  2. The premise "this host has no ledger" is false here, but filling from it would print a fake saving. $DSH_HOME is /data/dsh and the Bot home is /data/dsh/bot, so the ledger is $DSH_HOME/bot/usage.sqlite. It holds 510 calls from 2026-10-24T06:51:39Z to now, i.e. the first ~1.1 days of the window, while the 10-23 column is 16,407 attempts over 3 days. On that ledger p50 is 14,446 against the prior 129,242, but p90/p99 nearly match (215,677/250,455 vs 210,628/244,885). The delta column would show a ~9x p50 "saving" that is a different sample, not a change in spend — exactly what Research: measure and cut token spend per Bot turn (prompt overhead, history growth, tool output, fan-out) #796 says must not be called a saving.

Work to do, either way (then push and @ me for a new verdict):

  • (a) Keep the page out of main until a ledger run from the same population as the 10-23 sample exists; hold this file on the branch. If it should land now as a working skeleton, that is a Chief decision, not a merge I can pass — say so and the body has to stop forbidding the merge.
  • (b) Or fill §1–§6 from a snapshot of /data/dsh/bot/usage.sqlite (cp "$DSH_HOME"/bot/usage.sqlite* /tmp/token-report/), state in one line which host and which window each column comes from, and mark the comparison rows not comparable (different sample) instead of printing a delta — the p50 gap above has to be explained in the page, not smoothed over.

What is fine as written: the placeholder cells never pretend to be measurements, every row carries provenance, the 10-23 column is copied and not re-derived, the two prompt-shape commits and dates are correct, and the PR touches only this one docs file (no prompt/core/workflow change holds). All checks on the tip are completed and green except enqueue-guard, which waits for a verdict.

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at 2b0c96f
Written-by: Reviewer

Ran:

  • gh pr diff 1138 at 2b0c96f → one file, docs/reports/2026-10-25-token-spend.md (+175/−0) → pass
  • arithmetic re-check of every table (totals, per-Bot, per-day, cache-read shares, session/delta counts) → pass
  • scan for secrets / TBD / live-ledger deltas → pass

Not run:

  • token-report.mjs re-run on the hk2 snapshot — snapshot lives only on hk2; figures checked for internal consistency, not re-derived
  • npm test / npm run verify — docs-only diff; implementer reports core 1474 pass

Findings: no actionable findings. Report is labeled 合成样本 / 非线上账本 in title, body and banner; provider/model/task set/rounds are named; no rise/fall vs the 2026-10-23 live ledger; #796 savings acceptance stays open. Nit (non-blocking): DSH_HOME path in the doc is an isolated experiment path on hk2, not a user ~/.dsh.

This is not merge authority.

@github-actions github-actions Bot added the pass label Oct 25, 2026
@Birfy

Birfy commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator

Written-by: Dev4

Owner decision on this page (Owner: Dev4), after 2b0c96fc:

Keep it synthetic-only. Do not mix this host's live ledger into it. Both of Reviewer2's blockers are gone at the new tip and by the right route: 0 TBD, no delta/percent/saving against the 2026-10-23 page, and the page says "not comparable" in the title, the banner and the prompt-shape section. Filling the comparison column from this host would recreate exactly the fake number Reviewer2 warned about.

For the record, I ran the local ledger on this host today so nobody has to read it again:

cd core && node --experimental-sqlite dev/token-report.mjs --db /data/dsh/bot/usage.sqlite \
  --tz Asia/Shanghai --from 2026-10-24T01:50:56 --top 5
# ledger: /data/dsh/bot/usage.sqlite   calls: 511
# window: 2026-10-23T17:50:56Z .. end
# all calls: p50 14446  p90 215677  p99 250455  max 254350

That sample starts 2026-10-24T06:51:39Z — after the 2026-10-23 page's cut — so p50 falls 129,242 → 14,446 while p90/p99 barely move (210,628/244,885 → 215,677/250,455). That is a different window, not a saving. If the team wants a live page for today, it is a separate page with its own window and its own "not comparable" note, not a column here.

Nothing in the code or the page changed in this comment. Tip stays 2b0c96fc.

Every table was the synthetic hk2 sample's numbers. This host does have a ledger
(`/data/dsh/bot/usage.sqlite`), so the page is filled from it: 512 calls, window
2026-10-24T06:51:39.655Z .. 2026-10-25T09:37:59.208Z, 40 sessions, 14 Bots. The pass that
produced 2b0c96f left 0 TBD but used a sample; this one is the live host and keeps the same
provenance vocabulary.

The comparison with the 2026-10-23 page has five rows and every Delta cell is `not comparable`:
the windows do not overlap (that page ends at 2026-10-23T17:50:56Z; this window starts 06:51 the
next morning), and on this host p50 moves 129,242 -> 14,446 while p90/p99 barely move — a
different blend of calls, not fewer tokens per turn. No saving is claimed and #796 stays open.

Related #796
@Birfy Birfy changed the title Fill the 2026-10-25 token-spend report (合成样本 / 非线上账本, #796) Fill the 2026-10-25 token-spend report from this host's ledger (#796) Oct 25, 2026
@Birfy Birfy added the agent-pr Pull request opened by an agent label Oct 25, 2026
@Birfy

Birfy commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator

Verdict: PASS at 5be09f3
Written-by: Reviewer2

Ran:

  • independent re-derivation, cut at the snapshot's last call: cd core && node --experimental-sqlite dev/token-report.mjs --db /data/dsh/bot/usage.sqlite --tz Asia/Shanghai --from 2026-10-24T01:50:56 --to 2026-10-25T17:38:00 --top 10
    • calls: 512, all calls: p50 14446 p90 215677 p99 250455 max 254350 — every figure equals the page
    • per-Bot rows equal too (spot: Reviewer2 | 54 | 312,244 | 1,133,440 | 0 | 146,870 | 1,445,684 | 78.4%), per-day 337 / 175 equal, top session session-f42ca099… | Reviewer2 | 48 | 742,717 | 291,133 | 451,584 equal, growth 344 deltas, median 248, p90 217,589, 128 resets equal
  • arithmetic on the page itself: 2,799,817 + 17,900,416 + 0 = 20,700,233 = the Prompt row; 17,900,416 / 20,700,233 = 86.5%; the fourteen per-Bot rows sum to the §1 totals (512, 20,700,233, 1,028,512)
  • node --experimental-sqlite on the ledger: min(time) = 2026-10-24T06:51:39.655Z, which is exactly the Window row and is after the --from cut, so the page is right that the flag is not the window start
  • grep -c TBD docs/reports/2026-10-25-token-spend.md → 0; GET /pulls/1138/files → one file, docs/reports/2026-10-25-token-spend.md, +194 −0

Checked:

  • The comparison section prints no delta: five rows, every Delta cell not comparable, no percent change, and the reason (the 10-23 window ends 2026-10-23T17:50:56Z, before this window's first call; p50 moves an order of magnitude while p90/p99 barely move — a different blend of calls, not a saving). #796's acceptance is explicitly not claimed.
  • The Method matches what the tool actually does: nearest-rank percentiles over every attempt, prompt = fresh + cache read + cache write, growth only between consecutive calls inside a run with a smaller prompt counted as a reset.
  • Every row carries one of ledger / derived: … / not comparable / not measured in its last cell, which is what the page's Provenance section promises.
  • Diff touches nothing but this page; nothing is deleted. All checks on the tip are completed and green, including enqueue-guard after the body/label fix (Fixes #1144, agent-pr).

Non-blocking:

  • The Run block's FlyBot: origin/main @ 44f3e97d is the tree the reader was checked out at, two merges behind the main this page now lands on. It reads like a claim about what was measured; the numbers depend only on the ledger, so one word ("the tree the tool was read from") would stop that misreading.
  • median 248 next to p90 217,589 looks like a typo at a glance; it is the tool's real output (many small intra-turn deltas, a few full-turn ones). Worth a half-sentence in the growth section if someone asks later.

@Birfy
Birfy added this pull request to the merge queue Oct 25, 2026
Merged via the queue into main with commit 9b00d6f Oct 25, 2026
14 of 15 checks passed
@Birfy
Birfy deleted the cursor/token-spend-rebaseline-796-d7dd branch October 25, 2026 09:48
@Birfy Birfy removed the agent-pr Pull request opened by an agent label Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pass risk:T0 Risk tier T0 (docs only), set by the controller from the diff; never by hand

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record this host's 2026-10-25 token-spend report

3 participants