feat(sketch_db): sid-level reconcile_from_streaming_config - #186
Merged
Merged
Conversation
Schema retirement #4. Add the sid-level mirror of `SchemaRegistry::reconcile` under `sketch_db/lifecycle/reconcile.rs`: pub fn reconcile_from_streaming_config( store: &SketchStore, config: &StreamingConfig, retention: Duration, ) -> SidReconcileSummary Iterates every sid in `store`, computes its content signature `(metric_name, agg_kind, group_by_keys)`, and force-retires any sid whose signature is not represented in the new config. Mirrors the "retire orphans" half of the schema-version reconcile; the "add new ids" half is implicit in the sid model (sids are minted lazily by the ingest path on first write). Wired alongside the existing `SchemaRegistry::reconcile` at every reconcile call site: - `route_otlp_to_precompute` in `drivers/ingest/otel.rs` (raw-OTLP ingest path) - `route_modified_otlp_sketches_to_precompute` in `drivers/ingest/otel.rs` (modified-OTLP sketch path) - `streaming-config` swap handler in `drivers/query/servers/http.rs` (event-driven entry) Both registries run in parallel for now: the §6.3 ingest barrier `ingest_state.schemas.is_writable(agg_id)` still depends on `SchemaRegistry` lifecycle state, so we keep the schema reconcile alive until retirement #5 deletes the registry and replaces the barrier with a sid-level check. Six new unit tests cover empty-config, matching-signature, different-signature, idempotency (already-retired sid not re-retired), distinct-metrics isolation, and different-agg-type discrimination. Sketch-typed agg-configs are not yet covered (the `AggregationConfig` shape doesn't carry sketch params today — parallel capability-routing channel handles that lifecycle). 793/793 lib tests pass; 5 ignored. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 task
zzylol
added a commit
that referenced
this pull request
May 13, 2026
…ble from otel (#187) Schema retirement step #3. Move the §6.3 write-side ingest barrier from the `SchemaRegistry`'s agg_id-keyed `is_writable` into the sid catalog directly. `SketchStore::ingest_precompute_for_agg_config` now returns `None` when the sid it resolves to already exists in `Retired` or `Expired` status — the same drop semantics the retired `schemas.is_writable(agg_id)` check provided, just keyed on the sid the sketch_db actually understands. New sids (instance not yet registered) still proceed to register + append; the gate only applies once `lifecycle::reconcile_from_streaming_config` (PR #186) has flipped the sid out of `Active`. Drop the three `schemas.is_writable(agg_id)` checks in otel.rs: - `route_otlp_to_precompute` raw-sample loop - `route_otlp_to_precompute` sketch-envelope loop - `route_modified_otlp_sketches_to_precompute` legacy worker-push loop The `samples_blocked_by_schema_barrier` counter and its companion Prometheus metric stay alive but now read 0 in steady state — the sid-level drops happen inside SketchStore without crossing the schema-keyed counter. Once the schema/ module is fully retired the counter wiring moves to sid-level (or gets renamed). 793/793 lib tests pass; 5 ignored. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Schema retirement #4 — add the sid-level mirror of `SchemaRegistry::reconcile` under `sketch_db/lifecycle/reconcile.rs`:
```rust
pub fn reconcile_from_streaming_config(
store: &SketchStore,
config: &StreamingConfig,
retention: Duration,
) -> SidReconcileSummary
```
Iterates every sid, computes its content signature `(metric_name, agg_kind, group_by_keys)`, and force-retires any sid whose signature is not represented in the new config. The "add new ids" half of schema-version reconcile is implicit in the sid model — sids are minted lazily by the ingest path on first write.
Wired alongside the existing `SchemaRegistry::reconcile` at every reconcile call site:
Both registries run in parallel until retirement #5 deletes `SchemaRegistry` and replaces the §6.3 `is_writable` ingest barrier with a sid-level check.
Test plan
🤖 Generated with Claude Code