feat(sketch-db): manual schema retire/expire HTTP endpoints - #42
Merged
Merged
Conversation
Adds the operator/debug surface missing from the eviction chain: - `SchemaRegistry::force_retire(agg_id)` — transitions an Active schema to Retired, starting the configured retention clock. Idempotent; returns the new state or None if unknown. - `SchemaRegistry::force_expire(agg_id)` — sets retired_at_ms and expires_at_ms to now so status() returns Expired immediately; the next SchemaEvictionService tick drops data + removes the schema. - `POST /api/v1/db/schemas/:agg_id/retire` (200 / 404 / 503) - `POST /api/v1/db/schemas/:agg_id/expire` (200 / 404 / 503) Without these, e2e testing the eviction path required either driving a StreamingConfig swap (couples schema lifecycle to controller state) or waiting out the 24h retirement retention. Also factors the inline JSON serialization in handle_get_schemas into a shared `schema_to_json` helper so /retire and /expire return the same shape. 5 new unit tests: force_retire happy-path, idempotent on already- retired, None on unknown; force_expire happy-path, None on unknown. 723 lib tests total pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
zzylol
added a commit
that referenced
this pull request
Apr 20, 2026
The quickstart precompute compose was missing `--enable-schema-eviction`, so Expired schemas (produced by a `StreamingConfig` swap that removed an agg_id, or by `POST /api/v1/db/schemas/:id/expire`) stayed resident in the registry forever and their data never got dropped from the store. The e2e that added retire/expire endpoints in #42 surfaced this: calling `force_expire` flipped the schema's status immediately but the data kept hanging around until a backend restart (which wipes the registry entirely — the opposite of what the §5 service is designed to manage). Also sets `--schema-eviction-poll-secs=60` (down from the 300s default) to make the eviction visible inside a demo run. Production deployments should raise this. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
zzylol
added a commit
that referenced
this pull request
Apr 20, 2026
The write-side schema barrier at `ingest_handler.rs` already drops samples for retired/expired agg_ids (§6.3 of the sketch DB design), but the drop was silent: no counter, no log. The e2e after #42 tried to confirm the barrier fires by looking at the logs and, finding nothing, concluded the barrier wasn't wired — a false alarm that would have sent a bug report down the wrong path. Changes: - `IngestState::samples_blocked_by_schema_barrier: AtomicU64`. Incremented exactly when `is_writable(agg_id)` returns false for a matching agg. One ingest sample can contribute more than once when multiple agg configs match the same metric. - Per-batch `debug!` log after the routing loop: total_dropped + per-agg_id breakdown. Batched (not per-sample) so a hot ingest path doesn't flood the log. - `route_decoded_samples` promoted to `pub(crate)` so the test module can call it directly. - Three unit tests in `precompute_engine::ingest_handler::tests`: - Active schema → counter stays 0. - `force_expire` → counter increments by the matching sample count in the next batch. - Non-matching metrics → counter stays 0 even after expire. 726 lib tests pass (was 723; +3), clippy clean, fmt clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 20, 2026
zzylol
added a commit
that referenced
this pull request
Apr 20, 2026
…art (#44) The quickstart precompute compose was missing `--enable-schema-eviction`, so Expired schemas (produced by a `StreamingConfig` swap that removed an agg_id, or by `POST /api/v1/db/schemas/:id/expire`) stayed resident in the registry forever and their data never got dropped from the store. The e2e that added retire/expire endpoints in #42 surfaced this: calling `force_expire` flipped the schema's status immediately but the data kept hanging around until a backend restart (which wipes the registry entirely — the opposite of what the §5 service is designed to manage). Also sets `--schema-eviction-poll-secs=60` (down from the 300s default) to make the eviction visible inside a demo run. Production deployments should raise this. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 20, 2026
…45) The write-side schema barrier at `ingest_handler.rs` already drops samples for retired/expired agg_ids (§6.3 of the sketch DB design), but the drop was silent: no counter, no log. The e2e after #42 tried to confirm the barrier fires by looking at the logs and, finding nothing, concluded the barrier wasn't wired — a false alarm that would have sent a bug report down the wrong path. Changes: - `IngestState::samples_blocked_by_schema_barrier: AtomicU64`. Incremented exactly when `is_writable(agg_id)` returns false for a matching agg. One ingest sample can contribute more than once when multiple agg configs match the same metric. - Per-batch `debug!` log after the routing loop: total_dropped + per-agg_id breakdown. Batched (not per-sample) so a hot ingest path doesn't flood the log. - `route_decoded_samples` promoted to `pub(crate)` so the test module can call it directly. - Three unit tests in `precompute_engine::ingest_handler::tests`: - Active schema → counter stays 0. - `force_expire` → counter increments by the matching sample count in the next batch. - Non-matching metrics → counter stays 0 even after expire. 726 lib tests pass (was 723; +3), clippy clean, fmt clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
May 5, 2026
* refactor: retire sketch-core mirror * refactor: switch consumer imports to asap_sketchlib::sketches::* Update PR #73 against the reorganized asap_sketchlib (PR #36): the runtime sketches no longer live under a dedicated `asap::` module — they were merged into the existing `src/sketches/` layout (single home per sketch concept, ASAP-runtime types appended to the file that already holds the high-throughput in-process variant). Mechanical path swaps in asap-query-engine: - `asap_sketchlib::asap::dd_sketch::*` → `::sketches::ddsketch::*` - `asap_sketchlib::asap::count_min::*` → `::sketches::countmin::*` - `asap_sketchlib::asap::count_sketch::*` → `::sketches::count::*` - `asap_sketchlib::asap::hll_sketch::*` → `::sketches::hll::*` - `asap_sketchlib::asap::kll::*` → `::sketches::kll::*` - `asap_sketchlib::asap::count_min_with_heap::*` → `::sketches::cms_heap::*` - `asap_sketchlib::asap::hydra_kll::*` → `::sketches::hydra_kll::*` - `asap_sketchlib::asap::set_aggregator::*` → `::sketches::set_aggregator::*` - `asap_sketchlib::asap::delta_set_aggregator::*`→ `::sketches::delta_set_aggregator::*` - `asap_sketchlib::asap::config::*` → `::asap_runtime::*` Naming-conflict renames carried through to the consumers: - `HllDelta` → `HllSketchDelta` (octo_delta::HllDelta still wins the short name) - `HeapItem` → `CmsHeapItem` (common::input::HeapItem still wins the short name) main.rs aliases `asap_sketchlib::asap_runtime as config` so the existing clap derive references (`config::DEFAULT_CMS_IMPL`, `config::configure(...)`) still work without touching the rest of the bin. Tests: - `cargo build --workspace` → clean - `cargo test -p query_engine_rust --lib precompute_operators` → 141 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed `e473ccc`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: align CountSketchDelta consumer with sketchlib-go wire format Track the additive `hh_keys` field on `asap_sketchlib::CountSketchDelta` so the proto delta path constructs the type with all fields filled in. Sends an empty `hh_keys` for now: the vendored Rust proto bindings in `asap_otel_proto::sketchlib::v1` haven't been regenerated against the latest `.proto` (which carries `hh_keys` on the Go side). The TopK rebuild on the proto-delta path will fire once those bindings sync; the sketchlib-go-aligned semantics are already in place underneath. Bumps the asap_sketchlib git dep to `refactor/wire-format-align-go` (see asap_sketchlib PR #37). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(asap_sketchlib): bump pin past PR #39 module renames Three asap_sketchlib modules were renamed in upstream PR #39: - sketches::countmin → sketches::countminsketch - sketches::count → sketches::countsketch - sketches::cms_heap → sketches::countminsketch_topk Backend consumers updated. Cargo.toml pin moved from refactor/wire-format-align-go branch to main (which now also has hh_keys restoration via PR #42 and DDSketch + KLL byte parity via #40 + #41). Unblocks ASAPCollector Phase 3 step 3 (backend consumes asap-precompute-rs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the operator/debug HTTP surface needed to exercise the eviction chain end-to-end without waiting out the 24h retirement retention.
What's in this PR
SchemaRegistry::force_retire(agg_id)— transitions an Active schema to Retired using the configured retention. Idempotent on already-Retired/Expired. Returns new state orNoneif unknown.SchemaRegistry::force_expire(agg_id)— setsretired_at_msandexpires_at_msto now sostatus()returnsExpiredimmediately; the nextSchemaEvictionServicetick drops the data + removes the schema.POST /api/v1/db/schemas/:agg_id/retire→ 200 (with new schema state) / 404 / 503POST /api/v1/db/schemas/:agg_id/expire→ 200 / 404 / 503handle_get_schemasinline JSON into a sharedschema_to_jsonhelper so retire/expire return the same shape.Why
Before this PR, exercising the eviction path in e2e required either driving a
StreamingConfigswap (couples schema lifecycle to controller state) or waiting out the 24h default retirement retention. Neither is practical for manual e2e or a short-lived integration test.Test plan
cargo test -p query_engine_rust --lib— 723 tests pass (up from 718)cargo clippy --all-targets -- -D warningscleancargo fmt --all -- --checkclean🤖 Generated with Claude Code