Skip to content

feat(sketch_db): retire schema/ module — sid-only lifecycle - #189

Merged
zzylol merged 6 commits into
mainfrom
feat/schema-retire-final
May 13, 2026
Merged

feat(sketch_db): retire schema/ module — sid-only lifecycle#189
zzylol merged 6 commits into
mainfrom
feat/schema-retire-final

Conversation

@zzylol

@zzylol zzylol commented May 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Schema retirement big-bang (plan step #5). Deletes the sketch_db::schema module and the Arc<SchemaRegistry> field on every consumer. The §6 lifecycle now lives entirely in the sid catalog (SketchStore); the §7 timeline + §6.3 ingest barrier + reconcile-on-config-swap all run sid-level.

Built as four parallel slices off feat/schema-retire-final@e5eb1d3 (type relocations + sid-only lifecycle/eviction.rs), cherry-picked into a single branch with three small integration fixups.

What landed

Types (moved out of the deleted schema/):

  • AggStatus + DEFAULT_RETIREMENT_RETENTION → lifecycle/status.rs
  • TimelineSegment + TimelineCoverage → query/timeline.rs (the producer)

Consumers migrated:

Slice Files Effect
X1 (backfill) sketch_db/backfill/{mod,processor,service}.rs + storage_engines/mod.rs BackfillRegistry::create_checked now takes &AggregationConfig + created_at_ms: u64. BackfillService and BackfillWindowProcessor drop their schemas field. Re-exports of AggSchema/SchemaRegistry removed.
X2 (query engine) query_engines/asap_query_engine/engine.rs + tests/schema_timeline_dispatch_tests.rs ASAPQueryEngine::schema_registry field + with_schema_registry builder deleted. Dispatcher tests stripped of AggSchema/SchemaRegistry; two #[ignore]'d tests stubbed panic!() pending sid-level cross-reconfigure dispatch.
X3 (http) drivers/query/servers/http.rs AppState.schemas + HttpServer::with_schemas deleted. Swap handler drops the legacy schemas.reconcile; JSON response surfaces sids_retired from SidReconcileSummary. /api/v1/db/schemas endpoints rewired sid-level: GET returns snapshot_instances(); /retire//expire take :sid instead of :agg_id.
X4 (ingest + main) precompute_engine/{engine,ingest_handler}.rs + drivers/ingest/otel.rs + main.rs IngestState.schemas field + record_barrier_drop method deleted. otel.rs drops schemas.reconcile, uses DEFAULT_RETIREMENT_RETENTION for the sid-level reconcile retention. main.rs stops constructing SchemaRegistry and drops every .with_schema_registry/.with_schemas chain call. SchemaEvictionService::new rewired to sid-only.

Integration fixups (single commit f698555):

  1. Backfill HTTP handler: look up AggregationConfig from hot-reload streaming-config snapshot, derive created_at_ms from sid catalog (fall back to wall-clock-now when no live ingest yet).
  2. flush_barrier_drops stubbed to log-only after record_barrier_drop removal.
  3. One leftover schema::AggStatus path in output_sink.rs test routed to lifecycle::AggStatus.

Test plan

  • cargo build --lib -p data_plane → 0 errors
  • cargo test --lib -p data_plane → 758 passed; 5 ignored

Follow-ups (separate PRs)

  • The two #[ignore]'d cross-reconfigure dispatch tests need sid-level rewrites (no longer reproducible from SchemaRegistry-shaped fixtures because two distinct AggregationConfigs with the same content signature collapse into one sid-level segment).
  • SchemaEvictionService no longer cancels in-flight backfills on retirement — backfill jobs remain agg_id-keyed; rekeying on sid + restoring the cancel step is a follow-up.

🤖 Generated with Claude Code

zzylol and others added 6 commits May 13, 2026 09:08
@zzylol @claude
wip: schema retirement big-bang — types relocated + eviction migrated…
e5eb1d3 
…, consumers not yet fixed

Mid-refactor checkpoint so a follow-up agent can pick up cleanly. Compile is BROKEN with ~17 unresolved imports — see PR/agent prompt for the remaining consumer list.

Done:
  * AggStatus + DEFAULT_RETIREMENT_RETENTION → lifecycle/status.rs
  * TimelineSegment + TimelineCoverage → query/timeline.rs
  * sketch_db/mod.rs re-exports updated
  * sketch_db/schema/ DELETED
  * lifecycle/eviction.rs rewritten sid-only, tests rewritten
  * Internal imports in index/, lifecycle/reconcile.rs migrated

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol @claude
wip(backfill): schema retirement slice X1
e28d965 
Drop `SchemaRegistry` / `AggSchema` references from the backfill
module tree as part of the schema-retirement big-bang. Specifically:

* `storage_engines/mod.rs` no longer re-exports `AggSchema` /
  `SchemaRegistry` (only `AggStatus` remains).
* `BackfillService` and `BackfillWindowProcessor` drop their
  `schemas: Arc<SchemaRegistry>` field; constructors lose the
  `schemas` parameter.
* `BackfillRegistry::create_checked` swaps its `&SchemaRegistry`
  + `agg_id` parameters for `&AggregationConfig` +
  `created_at_ms: u64`. The caller (currently the HTTP handler
  in drivers/) already has both on hand from its
  `StreamingConfig` snapshot. The `CreateError::UnknownAgg`
  variant is retained for HTTP error mapping but is no longer
  produced by `create_checked` itself — the caller proves the
  agg exists by holding its config.
* Tests that previously dereferenced `schemas.get(agg_id)
  .created_at_ms` now stamp their own wall-clock millis via a
  local `now_ms()` helper, matching the registry's own helper.
* `create_checked_rejects_unknown_agg` is removed (its case is
  unreachable in the new signature).

The data_plane crate still has ~10 errors outside this slice's
scope (precompute_engine / query_engines / drivers / controller)
that sibling agents in the parallel retirement big-bang own.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol @claude
wip(engine): schema retirement slice X2 — drop schema_registry
018aabf 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
wip(ingest): schema retirement slice X4 — drop SchemaRegistry from In…
13c088d 
…gestState + main wiring
@zzylol @claude
wip(http): schema retirement slice X3 — drop schemas state + sid-rewi…
eb40690 
…re endpoints

Remove the per-`agg_id` `SchemaRegistry` wiring from the HTTP server's
state + builder + lifecycle endpoints and rewire each one against the
sid catalog (`SketchStore`). The legacy `SchemaRegistry` type was
deleted upstream by the schema retirement big-bang; this slice closes
out the consumers in `drivers/query/servers/http.rs`.

* `HttpServer` / `AppState`: delete the `schemas:
  Option<Arc<SchemaRegistry>>` field, the matching `with_schemas`
  builder, and every clone site (both `run` and `start_test_server`).
* `POST /api/v1/streaming-config`: drop the legacy
  `schemas.reconcile(snap)` path entirely; only the sid-level
  `lifecycle::reconcile_from_streaming_config(&SketchStore, &cfg,
  DEFAULT_RETIREMENT_RETENTION)` survives. Response shape changes:
  `schemas_created` / `schemas_retired` →
  `sids_retired: [<u64>...]` (no `added` — sids are minted lazily by
  the ingest path on first write).
* `GET /api/v1/db/schemas`: same path, sid-level body. Returns
  `count` + `schemas: [{sid, metric_name, status, first_seen_unix_ms,
  retired_at_ms, expires_at_ms, group_by_keys, agg_kind}]`. Empty
  catalog returns 200 + empty array (no more 503).
* `POST /api/v1/db/schemas/:sid/retire` (was `:agg_id`):
  `SketchStore::force_retire(sid, DEFAULT_RETIREMENT_RETENTION)`.
* `POST /api/v1/db/schemas/:sid/expire` (was `:agg_id`):
  `SketchStore::force_expire(sid)`.
* Backfill handler: drop the inner `state.schemas` guard and pass
  `state.sketch_index.as_ref()` to `BackfillRegistry::create_checked`
  — the sibling backfill slice owns the `&SchemaRegistry → &SketchStore`
  parameter swap on `create_checked` itself.
* Tests: rewrite `setup_test_server_with_hot_reload_and_schemas` →
  `setup_test_server_with_hot_reload_and_sketch_index` and
  `setup_test_server_with_backfill_and_schemas` →
  `setup_test_server_with_backfill_and_sids`. Add a
  `register_precompute_sid` test helper. Rewrite
  `test_streaming_config_swap_drives_schema_reconcile` →
  `_sid_reconcile` (asserts on `sids_retired`), drop the
  `_without_schemas_still_succeeds` test (the `schemas` notion is gone
  entirely) and replace with `_response_shape_with_empty_catalog`.
  Rewrite `test_get_schemas_returns_active_and_retired_with_status_filter`
  for the sid body shape; add
  `test_post_schema_retire_and_expire_endpoints_drive_sid_catalog`.
  Re-target the two timeline tests at the new sketch_index helper.

The route paths `/api/v1/db/schemas{,/:sid/retire,/:sid/expire}` are
kept under their existing `schemas` prefix to avoid breaking external
callers; only the `:agg_id` path segment was renamed to `:sid`.

`AppState` no longer holds a `schemas` field; `main.rs` will drop the
`.with_schemas(...)` builder call in its sibling slice.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol @claude
fix: integration glue for parallel schema-retirement slices
f698555 
After cherry-picking the four parallel slices (backfill, engine,
ingest, http) onto feat/schema-retire-final, three cross-slice
mismatches needed fixing:

1. `drivers/query/servers/http.rs` backfill handler — X3 assumed
   `create_checked` would take `&SketchStore`, but X1 changed it
   to `&AggregationConfig + created_at_ms: u64`. Rewired the
   handler to look up the agg-config from the hot-reload
   streaming-config snapshot (404 on miss) and derive
   `created_at_ms` as the earliest live `first_seen_unix_ms` from
   the sid catalog, falling back to wall-clock-now when no sid
   has ingested data yet. Test fixture
   `setup_test_server_with_backfill_and_sids` now builds a
   `StreamingConfig` + `HotReloadStreamingConfig` matching the
   active agg_ids so the lookup succeeds.

2. `drivers/ingest/otel.rs` `flush_barrier_drops` — X4 deleted
   `IngestState::record_barrier_drop` (the §6.3 counter was
   neutered in PR #187), but the helper was still calling it.
   Stubbed the helper to log-only.

3. `precompute_engine/output_sink.rs` test — `sketch_db::schema`
   path was hardcoded; switched to the new
   `sketch_db::lifecycle::AggStatus` location.

758/758 lib tests pass; 5 ignored.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 7ad6255 into main May 13, 2026
This was referenced May 17, 2026
Closed
Merged
zzylol added a commit that referenced this pull request May 18, 2026
@zzylol @claude
docs(test): clarify ignore reason for cross-reconfigure timeline test (…
eb87a44 
…#272 step 4) (#288)

Issue #272 step 4 asks to "re-enable the two ignored cross-reconfigure
dispatch tests in `tests/schema_timeline_dispatch_tests.rs`". That file
no longer exists: it was deleted in PR #280 (B7.5 v2) along with the
legacy `handle_query_promql` family it exercised, and its two ignored
tests were `panic!()` stubs with no real implementations.

The only remaining ignored cross-reconfigure test
(`test_get_timeline_returns_segments_after_reconfigure` in
`http.rs:2440`) is wedded to the pre-#189 `SchemaRegistry::reconcile()`
contract that eagerly populated the timeline source on YAML POST.
Post-#189 the sid lifecycle is explicit (see
`lifecycle::reconcile_from_streaming_config` module doc): "There is no
'added' set: sids are minted lazily by the ingest path on first
write". Re-enabling the test as written requires either an
architectural switch to eager sid minting (contradicts the documented
sid lifecycle) or rewriting the test to interleave an ingest step
(changes the contract being tested) — both out of scope for #272.

Updates the `#[ignore]` reason + comment to document this resolution
so the next maintainer doesn't re-investigate. No production change;
no test count delta (baseline stays 727 lib + 2 ignored).

Closes step 4 of #272 as obsoleted by #280.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol deleted the feat/schema-retire-final branch July 17, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zzylol