Skip to content

perf: move the project and thread index from sessions.json to redb - #588

Closed
Tryanks wants to merge 1 commit into
feat/storage-log-compactionfrom
feat/storage-redb-index
Closed

Tryanks wants to merge 1 commit into
feat/storage-log-compactionfrom
feat/storage-redb-index

Conversation

@Tryanks

@Tryanks Tryanks commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Summary

Third slice of the storage work in #535, stacked on #587. Closes #523.

The project and thread index moves from one rewritten-whole sessions.json to a redb database, so an index write costs what changed rather than the whole file.

 <data dir>/
-├── sessions.json            # rewritten whole on every meta change
+├── tcode.redb               # tables projects, sessions, meta; values are the row's JSON
+├── sessions.json.migrated   # the original, byte-identical, kept after the one-time import
 crates/services/src/
-├── store.rs                 # JSON index read/write (−273 lines)
+└── store/index.rs           # SessionIndex: open, import, commit, compact
first start with no database
  parse sessions.json with today's tolerance
  write tcode.redb.tmp in one transaction
  read back, compare row by row
  rename sessions.json → sessions.json.migrated, tmp → tcode.redb
once the database exists it is the only truth; a reappearing sessions.json is ignored with a warning

SessionIndex::commit(batch)                     # one transaction on HostCx::unblock
  store writer drains back-to-back index writes into one commit
host stop joins the store writer                # "stopped" means committed and closed
  • Single writer, concurrent readers via redb's experimental-multiprocess in ConcurrencyMode::SingleWriter (maintainer's decision, Principle 8; already in the locked 4.3.0). A second host on the same data directory fails: headless exits 1, the desktop shows a critical alert and exits 1. A start after a relaunch marker waits up to 15 s for the outgoing instance.
  • Per-row decode failures are skipped with a warning and never rewritten or deleted; before, one bad value failed the whole file.
  • The file is compacted at open once it has doubled since the last compaction.
  • SessionStore::open_at stays directory-only: only a host opens the database; set-password, pair and clients never do.

Departure from the issue. #523 proposed Durability::None for index-only updates. redb refuses it in multi-process modes, so every commit is durable. See the table below.

Evidence

One meta upsert on a 2,957-session index:

Before (sessions.json) After (redb)
Median 3.8 ms on the executor thread 7.0 ms on the blocking pool
Bytes written 2.19 MB, no fsync ≈ 120 KB, two fsyncs

Three batched writes cost the same as one.

  • Import: a copy of the maintainer's sessions.json (23 projects, 2,957 sessions) imports with every row equal field for field to a plain parse; .migrated is byte-identical; a second start changes no row. Import 44 ms, database 4.2 MB.
  • Crash safety: SIGKILL during import, 100 runs, all reopen to the full index. An earlier design that created tcode.redb in place failed 1 of 60, which is why the import builds under a temporary name. SIGKILL during commits, 60 runs: no torn transaction, no acknowledged commit lost.
  • Processes: a second headless host exits 1 in 0.04 s; a read-only open from another process succeeds while the host runs; a relaunching instance waits and starts once the first exits.
  • Tests added in store/index.rs: literal legacy inputs import once; the four interrupted-import states reopen to the full index; an unparseable sessions.json is preserved and the index starts empty; undecodable rows are skipped and kept; a second host is refused until the first closes; replacing or removing a project deletes only its managed icon; compaction after doubling.
  • Removed: session_index_upserts_orders_and_removes_only_the_selected_session (ordering is the host's sort at load; the rest is covered by the index and cascade tests). Two tests that ran a second host while the first was alive, which the app never does, now stop the first or read the live host's rows.
cargo fmt / clippy --workspace -D warnings / machete   ✓
cargo nextest run --workspace --locked                 900 passed, 6 skipped
Web and iOS checks with -D warnings                    ✓
desktop release binary                                 +808 KiB; Cargo.lock gains one edge, no new crate

Merge Danger

Door: one-way

After the first start, tcode.redb is the truth. Rolling back to a JSON build means restoring from sessions.json.migrated and losing every index change made since. The migrated file is byte-identical, so that rollback is possible but manual.

Blast Radius: every host's data directory

  • A data directory on a file system without byte-range locks now fails to start with an explicit error; there is no fallback to an exclusive whole-file lock.
  • Every index commit is durable (two fsyncs), so the median upsert is slower than before while writing far fewer bytes.
  • Windows LockFileEx range locks and Linux OFD locks are verified only by the second-host test in CI; not run locally on Android, Windows or Linux. The startup-failure alert was exercised on macOS.

tcode.redb replaces sessions.json. Each index change writes the rows it
touches in one transaction on the blocking pool, and the store writer
commits back-to-back index writes together, instead of re-reading and
rewriting the whole file for every change.

The host opens the database as redb's single writer under
experimental-multiprocess: another process can read it while the host
runs, and a second host on the same data directory fails with an explicit
error instead of silently overwriting. A relaunch waits for the outgoing
instance to close it.

sessions.json is imported once into a temporary database, read back and
compared, then kept as sessions.json.migrated. A row this build cannot
decode is skipped and left in place rather than failing the whole index.

Closes #523
@Tryanks Tryanks changed the title perf: keep the project and thread index in redb perf: move the project and thread index from sessions.json to redb Oct 6, 2026
@Tryanks

Tryanks commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

Folded into #591, which carries this commit unchanged as one PR for the whole storage phase.

@Tryanks Tryanks closed this Oct 6, 2026
@Tryanks
Tryanks deleted the feat/storage-redb-index branch October 6, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant