perf(db): use range predicates and covering indexes for the monthly aggregates - #234
Merged
Merged
Conversation
…ggregates
Host report (rant 2026-09-14T16:51:14): monthly aggregate queries take 4~10s on
the dev deployment.
The root cause is not NAS throughput (measured: a 4KB hot read on the NAS is
1.9µs, on par with a local disk) but **how many pages a query touches**:
`strftime('%Y-%m', time) = strftime('%Y-%m', 'now')` wraps the indexed column in
a function, so the index is unusable and SQLite falls back to a full scan. Same
db, same query: strftime 3845ms -> range predicate 2321ms -> covering index
119ms (18-32x).
- 12 production predicates rewritten to **closed** ranges (equivalent to the old
form and index-usable): wallet.rs (month consume / month earn / dashboard
month-by-type + the 7-day series JOIN), ops.rs (month_calls / month_in /
month_out + today-by-hour), admin.rs (per-member, per-model, per-dept),
org.rs (per-dept month_cost). The closed upper bound matters: the lower bound
alone would admit future-month rows (asserted).
- src/db.rs: SCHEMA_VERSION 13 -> 14 with four new indexes -
`transactions(user_id, time, type, pts)` and `transactions(time, type, pts)`
for the with/without-user_id month aggregates, plus `usage_records(time)` and
`usage_records(user_id, time)` (that table previously had zero indexes).
- New src/perf_gate.rs (test-only module): fails the suite if a date function
wraps a time column in production again, with a positive control on the
rewrites and a self-test proving the detector actually fires.
- The wallet test that still uses the old form is kept as the independent spec
oracle - the rewrite must keep satisfying the old semantics.
Tests: `cargo test` 239 -> 245 passed / 0 failed (4 perf_gate + 2 db).
`cargo fmt --check` exit 0. Clippy is clean on CI's stable toolchain; the
sandbox's only complete toolchain (1.95.0) additionally reports one pre-existing
`collapsible_match` in `src/protocol.rs:662` (out of this diff, landed in #207),
left untouched.
8 tasks
argszero
added a commit
that referenced
this pull request
Sep 14, 2026
Rant 2026-09-14T16:51:14, acceptance item 2: "对月度聚合 `EXPLAIN QUERY PLAN` 不得出现 `SCAN transactions`". The `perf_gate` module added in #234 only asserts the source *shape* (no date function wraps a time column). If the covering indexes were dropped, the shape would still be right and the suite would stay green - the plan-level half was missing. This adds it: against a migrated db holding 3000 rows spread over 180 days, each rewritten query shape must show `SEARCH <table>` and must not show `SCAN <table>`. Five shapes: per-user month consume (wallet), global month in (ops, no user_id), monthly usage count (ops), today-by-hour (ops), and the per-member LEFT JOIN (admin). The admin case asserts on the **alias** (`ur`) because that is what the planner prints for an aliased table - asserting the table name there reports a false failure. Also evaluates the rant's open question on index de-duplication: the new `transactions(user_id, time, type, pts)` has `transactions(user_id, time)` as a prefix, so the older `idx_transactions_user_id_time` is prefix-redundant. Left in place (minimal change) and recorded as a follow-up candidate, not dropped here. Honest scope note: this test pins the *permitted plan for the range shape*; the production source is bound to that shape by `perf_gate`. Neither alone would be enough - together they mean "production has the shape" and "the shape gets a plan with a SEARCH". Tests: 245 -> 246 passed / 0 failed. Teeth verified: removing the two new covering indexes reds this test (and the v12/v14 index-list test), and the file was restored byte-identically (md5 asserted).
argszero
added a commit
that referenced
this pull request
Sep 14, 2026
Ships the 76 PRs merged since v0.7.22 (#161-#237), the largest release so far. Database schema moves 12 -> 14: - v13 (#217): `keys.used` changes unit from tokens to points, and the live values are healed from the ledger (`used = SUM(transactions.pts WHERE type='consume')`), gated on schema_version < 13. - v14 (#234): four indexes for the monthly aggregates — `transactions(user_id, time, type, pts)`, `transactions(time, type, pts)`, `usage_records(time)`, `usage_records(user_id, time)`. Deployments must apply the DeepSeek flash rename (#233) to their own config.toml: `seed_models` is a full sync, so a model absent from the config is deleted at startup; the retired names are gone from config.example.toml. - Cargo.toml / Cargo.lock: 0.7.22 -> 0.7.23. - ui/index.html: asset cache-bust 20260912-4 / 20260912-5 / 20260914-1 / 20260914-4 / 20260914-9 -> 20260914-10 (all five refs). - CHANGELOG.md: v0.7.23 entry, grouped by area with the PR and hash of each fix. Gates: `cargo test` 249 passed / 0 failed, `cargo fmt --check` clean, `node --check` on the four ui/js files OK.
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the monthly aggregate queries index-usable, so they stop taking 4~10s on the dev deployment.
Host report (rant
2026-09-14T16:51:14): monthly aggregates on the dev (NAS-backed) database take4~10 seconds.
The cause is not NAS throughput. Measured on the same file: a 4KB hot read on the NAS costs
1.9µs, on par with a local disk. The real cost is how many pages one query touches -
strftime('%Y-%m', time) = strftime('%Y-%m', 'now')wraps the indexed column in a function, soSQLite cannot use the index on
timeand scans the whole table. Same db, same query:strftime('%Y-%m', time) = strftime('%Y-%m','now')time >= … AND time < …)Related Issue
No issue exists for this - it was reported directly by the host. Left empty on purpose rather than
fabricating one.
Changes
12 production predicates rewritten to closed ranges (semantically equivalent to the old form
and index-usable):
src/routes/wallet.rs: month consume, month earn, dashboard month-by-type, and the 7-day seriesLEFT JOIN(date(t.time) = days.day->t.time >= days.day AND t.time < date(days.day,'+1 day');the user filter stays in the JOIN).
src/routes/ops.rs:month_calls,month_in,month_out, and today-by-hour.src/routes/admin.rs: per-memberLEFT JOIN, per-model, per-dept.src/routes/org.rs: per-deptmonth_costsubquery.This is asserted (see the equivalence test below).
src/db.rs:SCHEMA_VERSION13 -> 14, adding four indexes -transactions(user_id, time, type, pts)andtransactions(time, type, pts)for thewith/without-
user_idmonth aggregates, plususage_records(time)andusage_records(user_id, time)(that table previously had zero indexes: a full scan cost 4525ms, 84ms after
(time)).New
src/perf_gate.rs(test-only module, likei18n_pack.rs/catalog_gate.rs/table_gate.rs):fails the suite if a date function wraps a time column in production
src/routes/*.rsagain. It hasa positive control on the rewrites (so an empty scan cannot pass), a check that the test-module spec
oracle is preserved, and a self-test proving the detector itself fires.
No new dependency, no SQL parser, no database - byte scanning only.
The
wallet.rstest that still uses the oldstrftimeform is kept unchanged as the independentspec oracle: the rewritten production query must keep satisfying the old semantics.
Config/data-structure changes are mirrored into the example file (the schema bump is additive
CREATE INDEX IF NOT EXISTS- no example-file counterpart)Tests
cargo testall green - 245 passed / 0 failed (was 239; +4perf_gate, +2db)cargo fmt --checkexit 0cargo clippy --all-targets -- -D warningsclean on CI's stable toolchain. Disclosure: the onlycomplete toolchain installable in the sandbox is rustc/clippy 1.95.0, which additionally reports
one
collapsible_matchatsrc/protocol.rs:662- pre-existing, out of this diff (landed in fix(protocol): give every announced responses item a unique output_index and its own order #207),green CI ever since; deliberately not touched here.
perf_gate×4: no date function wraps a time column in production; all closed-range rewrites arepresent (positive control); the spec oracle is kept; the detector flags the pre-fix shape and spares
the range form.
db::tests::v14_usage_indexes_and_schema_version: the twousage_recordsindexes exist and theschema-version gate is idempotent.
db::tests::month_range_predicate_is_equivalent_to_strftime_and_excludes_future_rows: asemantic spec with a fixture containing a previous-month, a this-month and a next-month
row; the closed range must equal the old
strftimeresult (2.0) and the lower-bound-only form mustnot (6.0) - that future row is what makes "closed vs open" assertable.
A/B evidence
cargo testre-run against five trees (no--quiet: libtest's--quietprints one char per test,which would silently empty every red set).
live{}(245 passed)M1_wrapstrftime('%Y-%m', time) = …{no_date_function_wraps_a_time_column_in_production, the_closed_range_rewrites_are_all_present, the_test_module_oracle_kept_the_old_form}M2_upper{the_closed_range_rewrites_are_all_present}M3_indexusage_recordsindex{db::tests::v14_usage_indexes_and_schema_version}M4_fixture{db::tests::month_range_predicate_is_equivalent_to_strftime_and_excludes_future_rows}Every leg reds a non-empty set, the sets are pairwise distinct (
M2 ⊆ M1, recorded honestly), andthe mutated files were restored byte-identically (md5 asserted before/after).
Checklist
perf/month-aggregate-indexesDeployment note
The new indexes are created by the migration (
CREATE INDEX IF NOT EXISTS), so a restart is enough;no data rewrite and no config change is needed. Index creation on a large table takes a moment on
first start only.