Skip to content

perf(db): use range predicates and covering indexes for the monthly aggregates - #234

Merged
argszero merged 1 commit into
mainfrom
perf/month-aggregate-indexes
Sep 14, 2026
Merged

argszero merged 1 commit into
mainfrom
perf/month-aggregate-indexes

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

Make the monthly aggregate queries index-usable, so they stop taking 4~10s on the dev deployment.

Host report (rant 2026-09-14T16:51:14): monthly aggregates on the dev (NAS-backed) database take
4~10 seconds.

The cause is not NAS throughput. Measured on the same file: a 4KB hot read on the NAS costs
1.9µs, on par with a local disk. The real cost is how many pages one query touches -
strftime('%Y-%m', time) = strftime('%Y-%m', 'now') wraps the indexed column in a function, so
SQLite cannot use the index on time and scans the whole table. Same db, same query:

form time
strftime('%Y-%m', time) = strftime('%Y-%m','now') 3845 ms
range predicate (time >= … AND time < …) 2321 ms
covering index 119 ms

Related Issue

No issue exists for this - it was reported directly by the host. Left empty on purpose rather than
fabricating one.

Changes

  • 12 production predicates rewritten to closed ranges (semantically equivalent to the old form
    and index-usable):

    • src/routes/wallet.rs: month consume, month earn, dashboard month-by-type, and the 7-day series
      LEFT JOIN (date(t.time) = days.day -> t.time >= days.day AND t.time < date(days.day,'+1 day');
      the user filter stays in the JOIN).
    • src/routes/ops.rs: month_calls, month_in, month_out, and today-by-hour.
    • src/routes/admin.rs: per-member LEFT JOIN, per-model, per-dept.
    • src/routes/org.rs: per-dept month_cost subquery.
  • ⚠️ The closed upper bound is load-bearing: the lower bound alone would admit future-month rows.
    This is asserted (see the equivalence test below).

  • src/db.rs: SCHEMA_VERSION 13 -> 14, adding four indexes -
    transactions(user_id, time, type, pts) and transactions(time, type, pts) for the
    with/without-user_id month aggregates, plus usage_records(time) and usage_records(user_id, time)
    (that table previously had zero indexes: a full scan cost 4525ms, 84ms after (time)).

  • New src/perf_gate.rs (test-only module, like i18n_pack.rs / catalog_gate.rs / table_gate.rs):
    fails the suite if a date function wraps a time column in production src/routes/*.rs again. It has
    a positive control on the rewrites (so an empty scan cannot pass), a check that the test-module spec
    oracle is preserved, and a self-test proving the detector itself fires.
    No new dependency, no SQL parser, no database - byte scanning only.

  • The wallet.rs test that still uses the old strftime form is kept unchanged as the independent
    spec oracle: the rewritten production query must keep satisfying the old semantics.

  • Config/data-structure changes are mirrored into the example file (the schema bump is additive
    CREATE INDEX IF NOT EXISTS - no example-file counterpart)

Tests

  • cargo test all green - 245 passed / 0 failed (was 239; +4 perf_gate, +2 db)
  • cargo fmt --check exit 0
  • cargo clippy --all-targets -- -D warnings clean on CI's stable toolchain. Disclosure: the only
    complete toolchain installable in the sandbox is rustc/clippy 1.95.0, which additionally reports
    one collapsible_match at src/protocol.rs:662 - pre-existing, out of this diff (landed in fix(protocol): give every announced responses item a unique output_index and its own order #207),
    green CI ever since; deliberately not touched here.
  • Unit tests added:
    • perf_gate ×4: no date function wraps a time column in production; all closed-range rewrites are
      present (positive control); the spec oracle is kept; the detector flags the pre-fix shape and spares
      the range form.
    • db::tests::v14_usage_indexes_and_schema_version: the two usage_records indexes exist and the
      schema-version gate is idempotent.
    • db::tests::month_range_predicate_is_equivalent_to_strftime_and_excludes_future_rows: a
      semantic spec with a fixture containing a previous-month, a this-month and a next-month
      row; the closed range must equal the old strftime result (2.0) and the lower-bound-only form must
      not (6.0) - that future row is what makes "closed vs open" assertable.

A/B evidence

cargo test re-run against five trees (no --quiet: libtest's --quiet prints one char per test,
which would silently empty every red set).

leg mutation red set
live none {} (245 passed)
M1_wrap one site back to strftime('%Y-%m', time) = … {no_date_function_wraps_a_time_column_in_production, the_closed_range_rewrites_are_all_present, the_test_module_oracle_kept_the_old_form}
M2_upper keep the lower bound only (drop the upper bound) {the_closed_range_rewrites_are_all_present}
M3_index drop one new usage_records index {db::tests::v14_usage_indexes_and_schema_version}
M4_fixture remove the future-month row from the equivalence fixture {db::tests::month_range_predicate_is_equivalent_to_strftime_and_excludes_future_rows}

Every leg reds a non-empty set, the sets are pairwise distinct (M2 ⊆ M1, recorded honestly), and
the mutated files were restored byte-identically (md5 asserted before/after).

Checklist

  • Branch naming follows the convention: perf/month-aggregate-indexes
  • Commit message uses Conventional Commits
  • Single responsibility, minimal change

Deployment note

The new indexes are created by the migration (CREATE INDEX IF NOT EXISTS), so a restart is enough;
no data rewrite and no config change is needed. Index creation on a large table takes a moment on
first start only.

…ggregates

Host report (rant 2026-09-14T16:51:14): monthly aggregate queries take 4~10s on
the dev deployment.

The root cause is not NAS throughput (measured: a 4KB hot read on the NAS is
1.9µs, on par with a local disk) but **how many pages a query touches**:
`strftime('%Y-%m', time) = strftime('%Y-%m', 'now')` wraps the indexed column in
a function, so the index is unusable and SQLite falls back to a full scan. Same
db, same query: strftime 3845ms -> range predicate 2321ms -> covering index
119ms (18-32x).

- 12 production predicates rewritten to **closed** ranges (equivalent to the old
  form and index-usable): wallet.rs (month consume / month earn / dashboard
  month-by-type + the 7-day series JOIN), ops.rs (month_calls / month_in /
  month_out + today-by-hour), admin.rs (per-member, per-model, per-dept),
  org.rs (per-dept month_cost). The closed upper bound matters: the lower bound
  alone would admit future-month rows (asserted).
- src/db.rs: SCHEMA_VERSION 13 -> 14 with four new indexes -
  `transactions(user_id, time, type, pts)` and `transactions(time, type, pts)`
  for the with/without-user_id month aggregates, plus `usage_records(time)` and
  `usage_records(user_id, time)` (that table previously had zero indexes).
- New src/perf_gate.rs (test-only module): fails the suite if a date function
  wraps a time column in production again, with a positive control on the
  rewrites and a self-test proving the detector actually fires.
- The wallet test that still uses the old form is kept as the independent spec
  oracle - the rewrite must keep satisfying the old semantics.

Tests: `cargo test` 239 -> 245 passed / 0 failed (4 perf_gate + 2 db).
`cargo fmt --check` exit 0. Clippy is clean on CI's stable toolchain; the
sandbox's only complete toolchain (1.95.0) additionally reports one pre-existing
`collapsible_match` in `src/protocol.rs:662` (out of this diff, landed in #207),
left untouched.
@argszero
argszero merged commit 93b2ebd into main Sep 14, 2026
1 check passed
@argszero
argszero deleted the perf/month-aggregate-indexes branch September 14, 2026 11:14
argszero added a commit that referenced this pull request Sep 14, 2026
Rant 2026-09-14T16:51:14, acceptance item 2: "对月度聚合 `EXPLAIN QUERY PLAN`
不得出现 `SCAN transactions`".

The `perf_gate` module added in #234 only asserts the source *shape* (no date
function wraps a time column). If the covering indexes were dropped, the shape
would still be right and the suite would stay green - the plan-level half was
missing. This adds it: against a migrated db holding 3000 rows spread over 180
days, each rewritten query shape must show `SEARCH <table>` and must not show
`SCAN <table>`.

Five shapes: per-user month consume (wallet), global month in (ops, no user_id),
monthly usage count (ops), today-by-hour (ops), and the per-member LEFT JOIN
(admin). The admin case asserts on the **alias** (`ur`) because that is what the
planner prints for an aliased table - asserting the table name there reports a
false failure.

Also evaluates the rant's open question on index de-duplication: the new
`transactions(user_id, time, type, pts)` has `transactions(user_id, time)` as a
prefix, so the older `idx_transactions_user_id_time` is prefix-redundant. Left in
place (minimal change) and recorded as a follow-up candidate, not dropped here.

Honest scope note: this test pins the *permitted plan for the range shape*; the
production source is bound to that shape by `perf_gate`. Neither alone would be
enough - together they mean "production has the shape" and "the shape gets a
plan with a SEARCH".

Tests: 245 -> 246 passed / 0 failed. Teeth verified: removing the two new
covering indexes reds this test (and the v12/v14 index-list test), and the file
was restored byte-identically (md5 asserted).
@argszero argszero mentioned this pull request Sep 14, 2026
11 tasks
argszero added a commit that referenced this pull request Sep 14, 2026
Ships the 76 PRs merged since v0.7.22 (#161-#237), the largest release so far.
Database schema moves 12 -> 14:

- v13 (#217): `keys.used` changes unit from tokens to points, and the live
  values are healed from the ledger (`used = SUM(transactions.pts WHERE
  type='consume')`), gated on schema_version < 13.
- v14 (#234): four indexes for the monthly aggregates —
  `transactions(user_id, time, type, pts)`, `transactions(time, type, pts)`,
  `usage_records(time)`, `usage_records(user_id, time)`.

Deployments must apply the DeepSeek flash rename (#233) to their own
config.toml: `seed_models` is a full sync, so a model absent from the config is
deleted at startup; the retired names are gone from config.example.toml.

- Cargo.toml / Cargo.lock: 0.7.22 -> 0.7.23.
- ui/index.html: asset cache-bust 20260912-4 / 20260912-5 / 20260914-1 /
  20260914-4 / 20260914-9 -> 20260914-10 (all five refs).
- CHANGELOG.md: v0.7.23 entry, grouped by area with the PR and hash of each fix.

Gates: `cargo test` 249 passed / 0 failed, `cargo fmt --check` clean,
`node --check` on the four ui/js files OK.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant