Repository navigation
Conversation
Cold or changed Antigravity databases were read with one statement.all() per blob table, so every row was in memory and the event loop was blocked until the whole table returned. Decoded usage maps also held subarray views of each row blob, which kept every blob alive until the read finished. Page gen_metadata, steps and trajectory_metadata_blob by rowid in groups of 256 only for ordinary rowid tables with no index and no column named rowid. Yield between pages inside the same read-only transaction. For these tables, main's ORDER BY idx sorts a rowid-order scan, so a stable sort by idx after paging returns the same order. Every other table runs main's exact query as one read, preserving its results and order. Read the rowid cursor as text and bind it as a bigint so rowids above 2^53 do not fail. Tests cover those rowids, WITHOUT ROWID tables, a column named rowid, covering indexes that retain main's tie order, and trajectory paging. Copy each usage message before decoding it so the row blob can be freed. Refs pingdotgg#13852 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-actions
Bot
added
vouch:unvouched
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This changes a production Antigravity usage reader’s database access, ordering, memory retention, scheduling, and trajectory-decoding behavior, with schema-dependent paging and early termination. Extensive tests cover the intended cases, but the implementation is complex and usage/metering-sensitive rather than a purely mechanical performance tweak. You can add or adjust custom eligibility rules. Learn more. |
juliusmarminge
added
the
macroscope-review
ChatGPT Codex Connector
Dismissing prior approval to re-evaluate 39769cd
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to This change reads Antigravity usage data in bounded pages, which lowers memory use and event-loop stalls on large databases. Results for valid data are unchanged. No actionable merge-blocking risk remains. The real Antigravity table schemas were not checked. Security Architecture Review
Pre-merge checks |
 |
| Check name | Status | Explanation |
|---|---|---|
| Linked Issues check |  |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
| Title check | |
The title clearly and concisely describes the main change: reducing memory use and event-loop stalls during cold Antigravity reads. |
| Description check | |
The description includes all required sections, explains the problem and implementation, links the related issue and maintainer triage, documents focused verification results, and identifies untested … |
✨ Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Qualify the reader comment on equal-idx order: it rests on SQLite's default single-threaded sorter, and read the pageable probe with Number() so it also holds under safe-integer mode. Tests now guard the large-rowid read against an empty spy and cover rowids 0 and the minimum, use an equal-idx fixture where rowid order differs from blob byte order, check that unpageable tables run main's single query, and fold the two keyed-table cases into one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The INTEGER PRIMARY KEY case now checks that both tables are read in rowid pages, and the WITHOUT ROWID case that each keeps main's single query. The large-rowid fixture starts at 2^53 + 2, so the first page's cursor is a rowid that a Number would round. The reader comment notes that the equal-key sort order is observed, not documented by SQLite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
only21mil
changed the title
|
Note 🤖 Claude Opus 5.5 responding on behalf of Theo Taken over in #19446. Your three commits are cherry-picked unchanged and credited there. The no-index check skips the real schema. Antigravity declares #19446 pages both tables by rowid (one generation, or up to 64 steps, per statement), keeps your |
3 participants
Problem
Cold or changed Antigravity databases load whole blob tables synchronously before yielding. #17615 switched each query to
statement.all(); decoded usage maps also retain the original row blobs. #17139 caches unchanged databases, but cannot avoid this first read.On a synthetic 318 MiB history, peak arrayBuffers fall from 308.3 to 76.8 MiB and the longest event-loop stall from 93.8 to 47.2 ms, medians of 6 reads per reader.
Change
Before #17615, the metadata reader iterated rows and yielded every 256 rows. This restores that granularity on the shared SqlClient by paging.
readDatabasereadsgen_metadata,stepsandtrajectory_metadata_blobin 256-row keyset pages only for ordinary rowid tables with no index and no column namedrowid. It usesWHERE rowid > ? ORDER BY rowid LIMIT ?and yields between full pages inside the same read-only transaction. The cursor starts below every rowid, then continues from the last rowid read as text and bound as a bigint, so rowids above 2^53, zero and negative rowids all page correctly. Other schemas keep main's exact single query.A stable
idxsort restores metadata order. Without anidxrowid alias, main sorts a rowid-order scan; its default single-threaded sorter preserves rowid order for equal keys, checked on SQLite 3.53.1 for in-memory and spilled sorts. SQLite does not guarantee tie order in general. On anidx INTEGER PRIMARY KEYalias, main'sORDER BY idxis a rowid scan with no sort step; indexed tables keep main's query and order.metadata()decodesfields(usage.slice())andfields(retryUsage.slice()), so retained fields hold small usage-message copies instead of whole row blobs. Paging alone reduced stalls but left retained blob memory unchanged; both changes address the same cold-read cost.Valid blob results, merge/filter rules, the read-only SQLite layer, single transaction,
busy_timeout = 100, database/WAL cache invalidation and exports stay the same. Main's??=already skipped protobuf decoding after the first trajectory timestamp. A paged read stops after the page holding that timestamp, so a value node:sqlite cannot convert on a later page no longer fails the database; one on that same page still does.Scope and approval
Issue #13852 is labelled
accepted, and the maintainer's triage of 2026-09-26 confirms the cold-read stall on the server thread. #17615 later made each table read a singlestatement.all()on 2026-10-09. This establishes the problem, not approval of every implementation. Paging bounds row count, not blob bytes; worker isolation and a single very large blob remain follow-ups.Disk persistence of decoded results was proposed in #17391, now closed, and stays out of scope here; nothing was copied or carried forward from it. Open #17400 fixes a different problem, double counting, and rewrites the same two
fields(...)calls inmetadata(), so whichever lands second must rebase and keep both.slice()copies.Verification
Linux x64, Node 24.21.0. Focused checks ran from the repository root at this branch's head
e146a97964:ed0f7b908b, with this branch's tests: exit 1, 4 failed and 15 passed.Test details
Main's four failures:
AssertionError: SELECT idx, data FROM gen_metadata ORDER BY idx: expected 600 to be at most 256.AssertionError: SELECT idx, data FROM gen_metadata ORDER BY idx: expected 300 to be at most 256.AssertionError: SELECT data FROM trajectory_metadata_blob: expected 600 to be at most 256.AssertionError: expected gen_metadata to be read in rowid pages, got: SELECT idx, data FROM gen_metadata ORDER BY idx: expected false to be true.The large fixture checks 600 generation and 600 step rows, 1,200 records, all 1,200 blob rows and at most 256 per read. The large-rowid fixture checks the minimum rowid, zero and values starting at
2^53 + 2, all 900 blob rows and ordered results. Its first full-page cursor is2^53 + 255, which Number conversion would round. The trajectory fixture checks a timestamp in page two followed by malformed protobufs, exactly two 256-row reads and the expected timestamp. All assert no errors.Other fixtures cover equal-
idxrowid order with reversed blob byte order, parameterized INTEGER PRIMARY KEY/WITHOUT ROWID cases, a one-row-per-tablerowid-column fallback, and indexed metadata/trajectory order. The INTEGER PRIMARY KEY case asserts paged SQL for both metadata tables; WITHOUT ROWID asserts main's exact query once for each, with 3 generation and 2 step blobs. The rowid-column and indexed tests also assert main's exact SQL runs once, unpaged; indexed order assertions record SQLite's current plan.Mutations that remove the index check or the rowid-column check fail the matching fallback tests. Forcing
pagedoff fails the four tests above, and converting the cursor to Number fails the large-rowid test.The focused test file on main's reader has 4 failures, exit 1; on this branch all 19 tests pass, exit 0.
The synthetic 318 MiB benchmark used Node 24.18.1 and ran on the first pushed head
39769cd575. The later commits change only tests, one code comment and how the probe result is read, and the benchmark was not rerun. All 12 runs returned 5 files, 9,856 records and no errors, with identical full-result SHA-256 after making database paths relative.309d8a23572b309d8a23572bMedian of 6 interleaved cold reads per reader, fresh process each, warm OS page cache, shared host under load. Stall ranges: main 85.7 to 119.3 ms, branch 35.7 to 63.8 ms. Wall-time ranges: main 364.8 to 517.8 ms, branch 360.1 to 529.5 ms. Every branch stall was below the best main stall, but overlapping wall-time ranges establish no wall-time improvement or maximum stall guarantee.
Verbatim summary stdout for the six interleaved cold reads per reader.
Unpublished benchmark harness, commands and measurement method
The harness is not published or part of this PR. It generates one 309.5 MiB database with 9,600 rows per metadata table and four 2.1 MiB databases with 64 rows per table. Blobs contain 16 KiB of protobuf metadata with deterministic unused padding; generation/step response IDs match. Each database has four trajectory rows, uses WAL and closes cleanly before reading.
gen_metadataandstepsdeclareidx INTEGER PRIMARY KEY, so main'sORDER BY idxis a rowid scan without a sort. With an unindexed non-keyidx, main also sorts every blob; this benchmark does not measure that cost. The trajectory table is an ordinary unindexed rowid table.Scratch files live in
$TMPDIR/antigravity-bench, with reader copies inbefore/andafter/and dependencies from the server:Collect six pairs in ABBA order: main/branch on odd pairs, branch/main on even pairs. Each calls
readAntigravityUsage(dir, 0)once in a fresh process without a reader cache. The OS page cache stays warm.Stall uses
monitorEventLoopDelayat 1 ms resolution, checked by a 1 ms interval watchdog. RSS uses the kernel high-water mark; 1 ms memory sampling may miss peaks during stalls. Added RSS takes each run's maximum minus baseline before the median. Full result hash:309d8a23572b67f06fe50fb612a4b880032ac5dd64fb0f42430322c62eb74f51.Not checked:
Model and harness: Claude Opus 5.5 in Claude Code and GPT-6.1 Sol in Codex, run from T3 Code.
🤖 Generated with Claude Code