feat(sketch_db): implement EpochSource for SketchIndex - #162
Merged
Merged
Conversation
Phase 5 M2.3.6b — first step of porting persistence over to `SketchIndex`. Lets the existing persistence flusher (`store/persistence/flusher.rs`) drive `SketchIndex` without any flusher-side changes; the `agg_id: u64` field on `SealedEpochRef` / `EpochSnapshot` is reused to carry a `sid`. Implementation lives next to the SketchIndex impl block in `index/mod.rs`. Four trait methods: - `list_sealed_epochs` — iterates the `series` DashMap, walks each sid's `sealed_epochs` BTreeMap, emits one `SealedEpochRef` per (sid, epoch_id) with its approximate byte size summed across payloads. - `snapshot_sealed_epoch` — resolves the sid's metadata to label sketch payloads with their `sketch_kind` for read-back dispatch; precompute payloads emit their `AggregateCore::type_name()`. Serializes sketch bytes (already opaque) and precompute bytes (via `SerializableToSink`). - `evict_sealed_epoch` — drops the named epoch from `sealed_epochs`. - `approx_memory_bytes` — on-demand walk over all sealed payloads. O(total sealed entries); fine at the flusher's tick cadence. `AggPayload::approx_bytes()` helper computes per-payload size (`bytes.len()` + struct overhead for sketches, `approx_memory_bytes()` for precomputes). 4 new unit tests: list-only-sealed, snapshot-round-trips-sketch, evict-drops-epoch, approx_memory-grows-with-sealed-state. 808/810 data_plane lib tests pass (2 pre-existing schema_timeline_dispatch failures unrelated). Next: M2.3.6c wires the flusher's startup against SketchIndex in main.rs and implements the on-restart read-back path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
zzylol
added a commit
that referenced
this pull request
May 13, 2026
Phase 5 M2.3.6c — adds `SketchIndex::start_persistence` and `SketchIndexPersistence` (the live persistence harness — manifest + part cache + flusher handle), and constructs one in `main.rs` when `--persistence-enabled` is passed. The new persistence layer writes to `<disk_path>/sketch_index/`, a subdir distinct from the legacy `SketchStore` layout — keeps the two coexisting until the legacy can be deleted in subsequent sub-PRs. The flusher loop is unchanged — it's generic over `EpochSource` (PR #162's `SketchIndex` impl drives the new path; the legacy `SketchStorePerKey` impl drives the deprecated one which now sees no writes after M2.3.6a / PR #160). Read-back from disk on query is NOT in scope here — the in-memory `query_range` / `query_precomputes_by_agg` paths don't yet consult `PartCache`. That's a follow-up: subsequent M2.3.6 sub-PRs add the disk-read glue (the legacy `SketchStorePerKey` similarly stubbed this; see `source.rs:35`'s "Ok(None) until then" comment). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6 tasks
zzylol
added a commit
that referenced
this pull request
Jul 27, 2026
64df20d is ASAPController main's actual tip now that #162 merged -- replaces the temporary pre-merge pin at e4cea66. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Jul 27, 2026
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 5 M2.3.6b — first step of porting persistence over to
SketchIndex. The existing persistence flusher (store/persistence/flusher.rs) can now driveSketchIndexwithout any flusher-side changes; theagg_id: u64field onSealedEpochRef/EpochSnapshotis reused to carry asid.Implementation
Four trait methods in
index/mod.rs:list_sealed_epochs— iteratesseriesDashMap; emits oneSealedEpochRefper (sid, epoch_id) with summed payload size.snapshot_sealed_epoch— resolves sid metadata to label sketch payloads with theirsketch_kind; precompute payloads carry theirAggregateCore::type_name(). Sketches use raw bytes; precomputes serialize viaSerializableToSink.evict_sealed_epoch— drops the named epoch.approx_memory_bytes— on-demand walk; fine at flusher tick cadence.Plus an
AggPayload::approx_bytes()helper.Test plan
cargo test -p data_plane --lib— 808/810 (2 pre-existingschema_timeline_dispatchfailures unrelated).cargo check --workspaceclean.What's next
M2.3.6c: wire the flusher in
main.rsagainstSketchIndexinstead ofSketchStorePerKey. Implement the deferred on-restart read-back path.🤖 Generated with Claude Code