feat(sketch_db): sid-level §6.3 ingest barrier; drop schemas.is_writable - #187
Merged
Merged
Conversation
…ble from otel Schema retirement step #3. Move the §6.3 write-side ingest barrier from the `SchemaRegistry`'s agg_id-keyed `is_writable` into the sid catalog directly. `SketchStore::ingest_precompute_for_agg_config` now returns `None` when the sid it resolves to already exists in `Retired` or `Expired` status — the same drop semantics the retired `schemas.is_writable(agg_id)` check provided, just keyed on the sid the sketch_db actually understands. New sids (instance not yet registered) still proceed to register + append; the gate only applies once `lifecycle::reconcile_from_streaming_config` (PR #186) has flipped the sid out of `Active`. Drop the three `schemas.is_writable(agg_id)` checks in otel.rs: - `route_otlp_to_precompute` raw-sample loop - `route_otlp_to_precompute` sketch-envelope loop - `route_modified_otlp_sketches_to_precompute` legacy worker-push loop The `samples_blocked_by_schema_barrier` counter and its companion Prometheus metric stay alive but now read 0 in steady state — the sid-level drops happen inside SketchStore without crossing the schema-keyed counter. Once the schema/ module is fully retired the counter wiring moves to sid-level (or gets renamed). 793/793 lib tests pass; 5 ignored. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 13, 2026
After cherry-picking the four parallel slices (backfill, engine, ingest, http) onto feat/schema-retire-final, three cross-slice mismatches needed fixing: 1. `drivers/query/servers/http.rs` backfill handler — X3 assumed `create_checked` would take `&SketchStore`, but X1 changed it to `&AggregationConfig + created_at_ms: u64`. Rewired the handler to look up the agg-config from the hot-reload streaming-config snapshot (404 on miss) and derive `created_at_ms` as the earliest live `first_seen_unix_ms` from the sid catalog, falling back to wall-clock-now when no sid has ingested data yet. Test fixture `setup_test_server_with_backfill_and_sids` now builds a `StreamingConfig` + `HotReloadStreamingConfig` matching the active agg_ids so the lookup succeeds. 2. `drivers/ingest/otel.rs` `flush_barrier_drops` — X4 deleted `IngestState::record_barrier_drop` (the §6.3 counter was neutered in PR #187), but the helper was still calling it. Stubbed the helper to log-only. 3. `precompute_engine/output_sink.rs` test — `sketch_db::schema` path was hardcoded; switched to the new `sketch_db::lifecycle::AggStatus` location. 758/758 lib tests pass; 5 ignored. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 13, 2026
* wip: schema retirement big-bang — types relocated + eviction migrated, consumers not yet fixed
Mid-refactor checkpoint so a follow-up agent can pick up cleanly. Compile is BROKEN with ~17 unresolved imports — see PR/agent prompt for the remaining consumer list.
Done:
* AggStatus + DEFAULT_RETIREMENT_RETENTION → lifecycle/status.rs
* TimelineSegment + TimelineCoverage → query/timeline.rs
* sketch_db/mod.rs re-exports updated
* sketch_db/schema/ DELETED
* lifecycle/eviction.rs rewritten sid-only, tests rewritten
* Internal imports in index/, lifecycle/reconcile.rs migrated
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* wip(backfill): schema retirement slice X1
Drop `SchemaRegistry` / `AggSchema` references from the backfill
module tree as part of the schema-retirement big-bang. Specifically:
* `storage_engines/mod.rs` no longer re-exports `AggSchema` /
`SchemaRegistry` (only `AggStatus` remains).
* `BackfillService` and `BackfillWindowProcessor` drop their
`schemas: Arc<SchemaRegistry>` field; constructors lose the
`schemas` parameter.
* `BackfillRegistry::create_checked` swaps its `&SchemaRegistry`
+ `agg_id` parameters for `&AggregationConfig` +
`created_at_ms: u64`. The caller (currently the HTTP handler
in drivers/) already has both on hand from its
`StreamingConfig` snapshot. The `CreateError::UnknownAgg`
variant is retained for HTTP error mapping but is no longer
produced by `create_checked` itself — the caller proves the
agg exists by holding its config.
* Tests that previously dereferenced `schemas.get(agg_id)
.created_at_ms` now stamp their own wall-clock millis via a
local `now_ms()` helper, matching the registry's own helper.
* `create_checked_rejects_unknown_agg` is removed (its case is
unreachable in the new signature).
The data_plane crate still has ~10 errors outside this slice's
scope (precompute_engine / query_engines / drivers / controller)
that sibling agents in the parallel retirement big-bang own.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* wip(engine): schema retirement slice X2 — drop schema_registry
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* wip(ingest): schema retirement slice X4 — drop SchemaRegistry from IngestState + main wiring
* wip(http): schema retirement slice X3 — drop schemas state + sid-rewire endpoints
Remove the per-`agg_id` `SchemaRegistry` wiring from the HTTP server's
state + builder + lifecycle endpoints and rewire each one against the
sid catalog (`SketchStore`). The legacy `SchemaRegistry` type was
deleted upstream by the schema retirement big-bang; this slice closes
out the consumers in `drivers/query/servers/http.rs`.
* `HttpServer` / `AppState`: delete the `schemas:
Option<Arc<SchemaRegistry>>` field, the matching `with_schemas`
builder, and every clone site (both `run` and `start_test_server`).
* `POST /api/v1/streaming-config`: drop the legacy
`schemas.reconcile(snap)` path entirely; only the sid-level
`lifecycle::reconcile_from_streaming_config(&SketchStore, &cfg,
DEFAULT_RETIREMENT_RETENTION)` survives. Response shape changes:
`schemas_created` / `schemas_retired` →
`sids_retired: [<u64>...]` (no `added` — sids are minted lazily by
the ingest path on first write).
* `GET /api/v1/db/schemas`: same path, sid-level body. Returns
`count` + `schemas: [{sid, metric_name, status, first_seen_unix_ms,
retired_at_ms, expires_at_ms, group_by_keys, agg_kind}]`. Empty
catalog returns 200 + empty array (no more 503).
* `POST /api/v1/db/schemas/:sid/retire` (was `:agg_id`):
`SketchStore::force_retire(sid, DEFAULT_RETIREMENT_RETENTION)`.
* `POST /api/v1/db/schemas/:sid/expire` (was `:agg_id`):
`SketchStore::force_expire(sid)`.
* Backfill handler: drop the inner `state.schemas` guard and pass
`state.sketch_index.as_ref()` to `BackfillRegistry::create_checked`
— the sibling backfill slice owns the `&SchemaRegistry → &SketchStore`
parameter swap on `create_checked` itself.
* Tests: rewrite `setup_test_server_with_hot_reload_and_schemas` →
`setup_test_server_with_hot_reload_and_sketch_index` and
`setup_test_server_with_backfill_and_schemas` →
`setup_test_server_with_backfill_and_sids`. Add a
`register_precompute_sid` test helper. Rewrite
`test_streaming_config_swap_drives_schema_reconcile` →
`_sid_reconcile` (asserts on `sids_retired`), drop the
`_without_schemas_still_succeeds` test (the `schemas` notion is gone
entirely) and replace with `_response_shape_with_empty_catalog`.
Rewrite `test_get_schemas_returns_active_and_retired_with_status_filter`
for the sid body shape; add
`test_post_schema_retire_and_expire_endpoints_drive_sid_catalog`.
Re-target the two timeline tests at the new sketch_index helper.
The route paths `/api/v1/db/schemas{,/:sid/retire,/:sid/expire}` are
kept under their existing `schemas` prefix to avoid breaking external
callers; only the `:agg_id` path segment was renamed to `:sid`.
`AppState` no longer holds a `schemas` field; `main.rs` will drop the
`.with_schemas(...)` builder call in its sibling slice.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: integration glue for parallel schema-retirement slices
After cherry-picking the four parallel slices (backfill, engine,
ingest, http) onto feat/schema-retire-final, three cross-slice
mismatches needed fixing:
1. `drivers/query/servers/http.rs` backfill handler — X3 assumed
`create_checked` would take `&SketchStore`, but X1 changed it
to `&AggregationConfig + created_at_ms: u64`. Rewired the
handler to look up the agg-config from the hot-reload
streaming-config snapshot (404 on miss) and derive
`created_at_ms` as the earliest live `first_seen_unix_ms` from
the sid catalog, falling back to wall-clock-now when no sid
has ingested data yet. Test fixture
`setup_test_server_with_backfill_and_sids` now builds a
`StreamingConfig` + `HotReloadStreamingConfig` matching the
active agg_ids so the lookup succeeds.
2. `drivers/ingest/otel.rs` `flush_barrier_drops` — X4 deleted
`IngestState::record_barrier_drop` (the §6.3 counter was
neutered in PR #187), but the helper was still calling it.
Stubbed the helper to log-only.
3. `precompute_engine/output_sink.rs` test — `sketch_db::schema`
path was hardcoded; switched to the new
`sketch_db::lifecycle::AggStatus` location.
758/758 lib tests pass; 5 ignored.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Schema retirement step #3 — move the §6.3 write-side ingest barrier from `SchemaRegistry::is_writable(agg_id)` to the sid catalog.
`SketchStore::ingest_precompute_for_agg_config` now returns `None` when the sid it resolves to already exists in `Retired`/`Expired` status — same drop semantics, keyed on the sid the sketch_db understands. New sids still register normally; the gate only kicks in once `lifecycle::reconcile_from_streaming_config` (#186) has flipped a sid out of `Active`.
Drop the three `schemas.is_writable(agg_id)` checks in `otel.rs`:
The `samples_blocked_by_schema_barrier` counter / metric is retained for /metrics compatibility but reads 0 — the sid-level drop happens inside SketchStore without crossing the schema-keyed counter. Counter rewiring follows when retirement #5 deletes `schema/`.
Test plan
🤖 Generated with Claude Code