Skip to content

feat: PrecomputeJob execution endpoint for DataCollector integration - #1

Merged
zzylol merged 3 commits into
mainfrom
feat/precompute-job-endpoint
Apr 13, 2026
Merged

zzylol merged 3 commits into
mainfrom
feat/precompute-job-endpoint

Conversation

@zzylol

@zzylol zzylol commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

Adds controller integration endpoints:

  • POST /api/v1/precompute — execute PrecomputeJob (PromQL on stored sketches)
  • GET /api/v1/health — health check for controller
  • GET /api/v1/store/metrics — list stored aggregation IDs

DataCollector controller → POST precompute job → backend evaluates → returns result.

🤖 Generated with Claude Code

zzylol and others added 3 commits April 3, 2026 23:34
New endpoints for DataCollector controller integration:

POST /api/v1/precompute
  - Receives PrecomputeJob from DataCollector controller
  - Executes PromQL query_expr against stored sketches
  - Returns query result or 404 if not answerable

GET /api/v1/health
  - Health check for controller to verify backend is alive

GET /api/v1/store/metrics
  - Returns list of metrics/aggregation IDs in store
  - Controller can use this to verify data is flowing

Usage with DataCollector controller:
  Controller creates PrecomputeJob with query_expr (e.g.,
  "topk(10, count_over_time(m{env=\"prod\"}[1m]) by (svc))")
  → POSTs to backend /api/v1/precompute
  → Backend evaluates against SimpleMapStore
  → Returns approximate result

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Satisfies clippy dead_code lint on fields previously accepted but unused.
These fields are part of the wire format from the DataCollector controller
and are now surfaced in the tracing span for observability.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 5f714df into main Apr 13, 2026
6 of 8 checks passed
@zzylol
zzylol deleted the feat/precompute-job-endpoint branch April 13, 2026 02:48
zzylol added a commit that referenced this pull request Apr 13, 2026
Rebuilds the OTLP ingest path on top of the merged PR #1 + PR #2
changes so that OTLP-delivered metrics (and pre-built sketches from the
DataCollector OTel collector) flow through the precompute engine's
worker pool. The precompute engine performs window-aligned aggregation
per StreamingConfig before writing to SimpleMapStore, matching the
Prometheus / VictoriaMetrics ingest pattern.

Architecture:
  DataCollector OTel collector
    → OTLP gRPC/HTTP (OtlpReceiver)
    → precompute engine ingest router
    → per-(agg_id, group_key) worker panes
    → StoreOutputSink → SimpleMapStore
    → query engine

Key changes
-----------
- PrecomputeEngine::new() now eagerly builds channels, router, agg_configs
  and a shared Arc<IngestState>. The state is exposed via a new
  `ingest_state()` getter so other ingest sources can push into the same
  worker pool without duplicating setup.
- IngestState is promoted from pub(crate) to pub, alongside an
  `extract_group_key_for(series_key, config)` associated helper that other
  drivers (OTLP here, Kafka potentially later) reuse for label→group
  extraction.
- New WorkerMessage::AccumulatorInput variant carrying a pre-built
  Box<dyn AggregateCore> with agg_id, group_key, and timestamp. Routed
  to workers by the same (agg_id, group_key) hash as GroupSamples.
- GroupState gains a `sketch_panes: BTreeMap<i64, Box<dyn AggregateCore>>`
  alongside the existing `active_panes`, plus a new
  `process_accumulator_input()` worker method that:
    * merges incoming accumulators into the covering pane via merge_with,
    * honors late-data policy (Drop / ForwardToStore),
    * emits both raw-sample and sketch-pane outputs on window close.
  `flush_all` is also extended to drain sketch panes for closed windows.
- New `merge_sketch_panes_for_window` helper mirrors
  `merge_panes_for_window`: oldest pane destructively taken, later panes
  cloned via `clone_boxed_core` for still-open sliding windows.
- OtlpReceiver gains `with_ingest_state()` constructor. When wired to
  the precompute engine, OTLP requests are dispatched as GroupSamples
  (raw metric points) and AccumulatorInput (sketch payloads) through the
  engine's router. Label semantics are preserved — each point is
  formatted into a standard metric{k1="v1",k2="v2"} series key and run
  through the same extract_group_key pipeline used by the Prometheus
  path, so StreamingConfig.grouping_labels drives pane keying uniformly.
- SketchPayload tuple is promoted to a structured `SketchPoint` that
  carries name, attr_name, labels, timestamp, and opaque payload bytes.
  Labels are now preserved through the sketch path (previously lost).
- Incoming sketch payloads are wrapped in
  SketchEnvelopeAccumulator::from_proto_bytes (introduced by PR #2) and
  sent to the worker as AccumulatorInput. This preserves the full
  sketch state end-to-end; per-variant concrete decoding (CountMin →
  CountMinSketchAccumulator, KLL → DatasketchesKLLAccumulator, …) can
  layer on top later without changing the routing contract.
- main.rs constructs the precompute engine BEFORE the OTLP receiver so
  the receiver can obtain an Arc<IngestState>. Without the precompute
  engine OTLP falls back to log-only mode.

Rebase notes
------------
This commit subsumes the earlier PR #3 commit dd22379 (which wrote
directly to SimpleMapStore with SumAccumulator placeholders). The
original commit conflicted with PR #2's own OTLP changes after PR #2
landed on main; rather than carry two overlapping commits forward, the
earlier placeholder work is replaced in-place by this coherent refactor.

Verification
------------
- cargo check --all-targets: clean
- cargo clippy --all-targets -- -D warnings: clean
- cargo fmt --check: clean
- cargo test --lib: 435 passed, 0 failed, 5 ignored

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 20, 2026
…able stats (#48)

Task #34 gap #1 of 3. The §7 schema-timeline primitive
(`SchemaRegistry::timeline_for_metric`) and the cross-schema
combiner (`engines::timeline_dispatch::combine_statistic`)
landed in PRs #20 / #22 / #25, but `SimpleEngine::handle_query_promql`
was still resolving a single `agg_id` via `resolve_agg_info_promql`
and running the full query against it. A query whose time range
spans a reconfigure boundary (old `agg_id` retired, new
`agg_id` created) saw a data cliff for the pre-boundary slice.

This PR wires the dispatcher:

* New `SimpleEngine::try_handle_query_promql_via_timeline`:
  1. Parse + pattern-match the query, extract metric name.
  2. Build a probe `QueryExecutionContext` to read the
     resolved `[t1, t2]` + `Statistic`.
  3. Call `timeline_for_query(metric, t1, t2)`. Bail out
     with `None` (fall-through to default single-agg path)
     if fewer than two segments, or if the statistic is
     non-combinable (quantile / topk / cardinality / rate
     / increase — those follow in PR B2 with a Partial
     HTTP response surface).
  4. Per segment: reuse `build_query_execution_context_promql_for_agg_id`
     from PR #37 (the extracted forced-agg-id entry point),
     clip the store plan's `[start, end]` to the segment's
     bounds, execute, collect results.
  5. Group by label-tuple and fold per-group per-segment
     scalars through `combine_statistic`. Emit the combined
     scalar as an `InstantVectorElement`. Purged segments
     or segments whose `agg_id` is no longer in the config
     go into `unresolved` so the combiner sees them.

* `handle_query_promql` now tries the timeline path first;
  returns immediately on `Some`, falls through to the
  existing single-agg path on `None`. Zero behavior change
  when the timeline has 0–1 segments for the query's metric
  (the common case today).

## Scope

Combinable stats only: Count / Sum / Min / Max. Non-combinable
stats still take the single-agg path — PR B2 will surface
`CombinedResult::Partial` on the HTTP response so users see
`{covered, missing: [segments]}` explicitly instead of a
silent data cliff.

## Validation

- `cargo test -p query_engine_rust --lib` — 728 pass (baseline
  unchanged; the dispatcher stays dormant when tests only
  register one schema per metric).
- `cargo clippy --all-targets -- -D warnings` — clean
- `cargo fmt --all -- --check` — clean

## Follow-ups (explicit non-scope here)

- **Integration test** seeding two agg_ids + cross-boundary
  Sum query. Requires the full `PrecomputeEngine` setup
  harness the existing e2e tests use; deferred as a
  dedicated PR so this one stays a focused dispatcher
  patch.
- **PR B2**: Partial response surface for non-combinable
  stats on the HTTP adapter.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 21, 2026
Addresses TODO.md blocker #1. Unblocks the paper's "sketches for
hot, exact for cold" story: capability-miss queries now serve
exact answers from raw observability samples in the cold tier
instead of hitting Prometheus (or failing) unconditionally.

## What's new

* `drivers/query/fallback/cold_store/` — storage-agnostic
  `ColdStore` trait + `LocalFsColdStore` impl. On-disk layout
  (`raw/<metric>/YYYY/MM/DD/HH/part-NNNNNN.jsonl`) is identical
  to what a future S3 cold store will use, so the adapter stays
  source-compatible when we swap backends.
* `drivers/query/fallback/s3_adapter.rs` — `ColdFallback<S>`
  implements `FallbackClient`. Parses PromQL, extracts
  `(metric, predicates, op)`, scans the cold store, computes
  the answer. Supported shapes for v1:
  - bare instant vector selector (`metric{labels}`)
  - no-grouping scalar aggregation
    (`sum|count|avg|min|max(metric{...})`)
  Label matchers: `=`, `!=` — regex delegated upstream.
  Anything outside this surface falls through to the optional
  inner `FallbackClient` (chain-of-responsibility, typically
  the existing Prometheus proxy).
* `drivers/query/fallback/metrics.rs` — hot/cold telemetry
  counters (`queryengine_hot_queries_total`,
  `queryengine_cold_queries_total`,
  `queryengine_cold_bytes_served_total`), keyed by
  `(metric, shape)`. Mirrors the PR #51
  schema-barrier-counter pattern.
* `AdapterConfig::prometheus_promql_with_cold` convenience
  constructor that composes the cold adapter in front of a
  Prometheus proxy.

## Routing

No engine changes needed. The existing `process_query_request`
already falls through to `FallbackClient` on engine-miss —
configuring the fallback as a `ColdFallback` naturally routes
Purged-segment / capability-miss queries through the cold tier
with the Prometheus proxy as the tail-of-chain for unsupported
shapes.

## Tests

752 → 777 tests (+25):

* 5 unit tests on the JSONL format + hour-prefix helpers
* 4 on `LocalFsColdStore::scan` (range filter, missing prefix,
  hour-boundary span)
* 10 on `ColdFallback` plan extraction + aggregation + float
  formatting
* 6 end-to-end HTTP integration tests in
  `tests/cold_fallback_tests.rs` covering: bare selector, sum
  aggregation, label filtering, unsupported-shape delegation,
  telemetry-counter increments, and the "Purged range served
  from raw" paper story.
zzylol added a commit that referenced this pull request May 12, 2026
…ema helper (#140)

First sub-PR of Step γ (variant-by-variant consumer migration). Adopts
the (c) bridge strategy: keep `legacy_expr::QueryExpr::Aggregate` as
the L2 emit shape; add a one-way canonical-builder helper consumers
call on demand. No construction site rewritten; no bridge enum
variant added to legacy QueryExpr.

This avoids the "child must be canonical" coupling that would otherwise
force γ1 to migrate every other legacy variant (SketchAgg, WindowedAgg,
TopK, etc.) in the same PR.

## What landed

### New: `controller/src/intent_algebra/aggregate_bridge.rs` (328 lines, 8 tests)

```rust
pub fn bridge_aggregate_to_canonical(
    keys: &[ColumnRef],
    aggs: &[AggItem],
    having: &Option<legacy::Predicate>,
    schema: &Schema,
) -> Result<BridgedAggregate, BridgeError>;

pub struct BridgedAggregate {
    pub by: Vec<ColumnId>,
    pub aggs: Vec<AggIntent>,
    pub having: Option<HavingPredicate>,
}

pub enum BridgeError {
    UnresolvedKey(ResolveError),
    HavingDeferred(QueryExprError),
}
```

The `having` translation routes through `from_legacy_scalar` (Batch 2).
E-deferred ScalarExpr variants (`FunctionCall`, `ScalarSubquery`,
`InList`, `Between`) surface as `BridgeError::HavingDeferred(...)`.
Test `bridge_having_deferred_e_variant_surfaces_error` is the contract;
no in-tree construction site builds a HAVING with E-variants today.

### `column_resolution.rs` extension (+211 lines, +6 tests)

```rust
pub fn output_schema_for_aggregate(
    input: &Schema,
    by: &[ColumnId],
    aggs: &[AggIntent],
) -> Schema;

pub fn resolve_named_keys(keys: &[ColumnRef], schema: &Schema)
    -> Result<Vec<ColumnId>, ResolveError>;
```

Mirrors canonical `query_expr::QueryExpr::output_schema_in`'s Aggregate
arm: outputs `by`-columns positionally + one column per
`AggIntent::output_column(probe)`, strips `time_index`, sets
`unique_keys = [by]`. This was Step β TODO #1.

Step γ2-γ4 consumers descending into legacy `Aggregate.input` should
pass `output_schema_for_aggregate(parent_schema, &bridged.by, &bridged.aggs)`
as the inner subtree's `parent_schema`.

### Demo wire in `physical/allocator.rs::alloc_node`

The legacy `QueryExpr::Aggregate { keys, aggs, having, input }` arm
now calls the bridge to derive canonical-shape data and enrich
`NodeAnnotation.rationale` with intent kinds + group-by column count.
Emit shape stays legacy; behavior unchanged (only the rationale string
carries extra info). Proves the bridge is reachable.

## Construction-site migration: 0 (intentional)

Per strategy (c), all ~10 construction sites in
`query_parser/{promql,sql}.rs`, `legacy_lower::lower_aggregate`, and
the optimizer rewrite path still emit `legacy_expr::QueryExpr::Aggregate`.
They migrate in γ7 once no legacy `input` subtree remains (SketchAgg,
WindowedAgg, TopK migrations land first in γ2-γ4).

## Build + test

- `cargo build --release -p controller` — clean
- `cargo build --release -p query_engine_rust` — clean
- `cargo test -p controller --lib` — **688 passed** (was 674; +14 new
  bridge + column_resolution tests)
- `cargo test -p controller --bin controller` — 27 passed

## Known caveats

- Canonical `QueryExpr::Aggregate.having` field is `Option<HavingPredicate>`
  where `HavingPredicate(pub String)`, NOT typed `Option<Predicate>` as
  the spec text suggested. Bridge renders the converted `Predicate` via
  `format!("{pred:?}")`. Round-tripping from the string is a follow-up
  (when canonical `having` upgrades to typed `Predicate`).
- `BridgeError` can't derive `PartialEq` (QueryExprError doesn't); tests
  use `matches!`. Non-blocking.
- `optimizer::engine::HydraConversion` (Step β TODO #7) NOT migrated —
  out of γ1's stated scope, deferred to a later γ sub-PR.

## Diff: 5 files, +618 / -4

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 13, 2026
Post-M2.3 reorg #1 of 8. Moves stateless payload types out of
`store/mod.rs` into a new `sketch_db::data` module. The split
clarifies that:

- `data/` = stateless data taxonomy (payload kinds, sid hashing,
  accuracy derivation, capability re-exports)
- `store/` = stateful sid registry + per-sid columnar substrate

Moved to `sketch_db::data`:

- `AggKind` enum (sketch-vs-precompute discriminator)
- `AggPayload` enum (sketch bytes vs accumulator)
- `SketchConfig` (per-variant tuning params)
- `SketchSampleState`, `SketchEncoding`
- `SketchTimeSeries` (read-side row shape)
- `AccuracyBound` + `from_config` derivation
- `compute_sid` / `compute_sketch_sid` (canonical sid hash)
- `canonical_parameters` helper
- Re-exports: `Capability`, `SketchKindHandle`, `AggregationType`

`store/mod.rs` re-exports each type at the legacy path
(`sketch_db::store::AggKind` etc.) so no external caller needs
to change spelling. The canonical home is now `sketch_db::data::*`.

783/783 lib tests pass.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 13, 2026
Schema retirement #1 of 5. Adds
`sketch_db::query::timeline::timeline_for_metric(&SketchStore, metric,
t1_ms, t2_ms) -> Vec<TimelineSegment>` that computes the per-metric
historical timeline entirely from the sid catalog
(`SketchStore::instances`).

Algorithm:

1. Snapshot all `SketchInstanceMetadata` for `metric` from the sid
   catalog.
2. Group by content signature `(metric, agg_kind, group_by_keys)`.
   Multiple sids sharing the same agg-config fold into one group.
3. Per group, fold the lifecycle fields: `min(first_seen_unix_ms)`,
   `Some(min(retired_at_ms))` iff every sid is retired,
   status = Active > Retired > Expired.
4. Apply the same segmenting + clipping as
   `SchemaRegistry::timeline_for_metric`.

`TimelineSegment.agg_id` now carries a stable xxh64 of the content
signature (the `(metric, agg_kind, group_by_keys)` tuple) — same
content-derived id idiom as `compute_agg_config_id` (PR #151). HTTP
callers see deterministic ids that don't depend on which specific
sid was first seen.

7 new unit tests cover: empty store, inverted range, single-active
signature, two-signatures-in-sequence, fold-of-many-sids-same-sig,
metric isolation, and signature-id determinism. Schema/'s
implementation stays alive in parallel until the consumers migrate.

Adds `SketchStore::snapshot_instances()` to support read-side scans.

783 + 7 = 790 lib tests pass.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 18, 2026
…-Sum deferred) (#282)

Two fixes that all live in control_plane/src/emit/ and adjacent
files, bundled to avoid sequential merge churn. The third
intended fix (B2 — Sum aggregation alongside sketch) was
deferred because the existing workload_store keys by metric name
and stores ONE QueryWorkload per metric — the multi-aggregation
shape needed for `sum by (zone)` alongside `quantile_over_time`
requires either a store-level restructure or a second
PhysicalExpr per metric, both substantially bigger than #1+#2.
Left as a follow-up; the YAML's existing sum-shaped entries
(MVP entries 2/3/4 targeting http_requests_total) still collapse
to the last write on a per-metric basis.

== Fix #1: B3 population gap ==

Symptom (smoke test): every emitted `transform/keep_for_<metric>`
block had `keep_keys(datapoint.attributes, [])` — strips ALL
attrs instead of keeping `grouping_labels`. Sid catalog landed
with one sid per metric instead of one per (metric, zone).

Root cause: the OpAMP-push path's `metric_to_grouping_labels`
WAS being populated from the WorkloadStore (in `main::
emit_bootstrap_typed`, `main::handle_plan`'s typed-stage-split
branch, and `replan::Replanner::try_emit_typed_edge_yaml`). But
the source `QueryWorkload.group_by_labels` was empty for the
canonical MVP query `quantile_over_time(0.99,
http_requests_total_latency_ms[30s])` — the PromQL parser only
surfaces grouping labels from `by (...)` clauses, and a bare
`quantile_over_time` has none. The pre-pop loop in `main.rs`
also passed `group_by_labels: vec![]` in the QuerySpec, leaving
nothing to merge with the empty parsed value.

Fix: add a declarative `grouping_labels: Vec<String>` field to
`WorkloadEntry` so the YAML can state the streaming-config
grouping contract directly. Thread `entry.grouping_labels` into
`QuerySpec.group_by_labels` in the pre-pop loop (main.rs ~276)
so `analyzer.analyze()` merges the YAML-declared labels with any
PromQL `by` keys into `QueryWorkload.group_by_labels`, which
`collect_metric_to_grouping_labels` then drops into
`EdgeStageConfig.metric_to_grouping_labels` → the emitter's
`keep_keys(datapoint.attributes, [...])` list. Same plumbing in
`emit::runtime_tests::populate_store_from_registry` so the
existing 5-sketch round-trip test keeps tracking main.rs.

Regression coverage (2 new tests in emit/mod.rs):
  * workload_entry_grouping_labels_round_trip_through_emit_to_keep_keys
  * workload_entry_grouping_labels_surface_in_emit_keep_keys_list

== Fix #2: B4 — controller picks window_duration from query ==

Symptom: agent's sketch processor's `window_duration` was
hardcoded at 300s (5m) for KLL/CMS/etc., 60s for others. The
backend's streaming-config `windowSize` likewise drifted from
whatever the user wrote in their PromQL `[range]`. Queries with
`[30s]` ranges always landed inside an open sketch window and
returned NoData.

Root cause: the pre-pop QuerySpec hardcoded
`time_window: "5m".into()` (main.rs ~278), which the analyzer
prefers over the PromQL-parsed value at pipeline.rs:203. The
parsed `[30s]` was thrown away. No clamp existed downstream to
catch this either, so a `[5m]` workload landed a 300s sketch
window that fell outside every sensible replay range.

Fix:
  1. Pass `time_window: ""` from main.rs and emit/runtime_tests
     when the entry HAS a `query_string` — lets the analyzer
     extract the matrix-selector range itself. Falls back to
     "5m" only when query_string is None (so the analyzer
     doesn't error at Step 4).
  2. Add `clamp_window_secs(Option<u64>) -> Option<u64>` in
     emit/stage_config.rs with bounds [5, 60]:
       * Lower 5s — below this the sketch processor mints new
         windows before it has enough samples for the family's
         quality bound, and per-flush sid-catalog cardinality
         explodes.
       * Upper 60s (= MAX_WINDOW_SECS, the historical default).
         Above this the user's replay range no longer contains
         a closed sketch window.
  3. Apply the clamp at every emission site so the agent's
     `window_duration` and the backend's `windowSize` agree
     exactly (drift de-syncs warm-tier replay):
       * `build_edge_processor_block` callers in the legacy
         single-pipeline and 5-sketch routing emit paths
         (stage_config.rs ~169 and ~1003).
       * `build_backend_aggregation_json` (stage_config.rs
         ~1636 — the streaming-config JSON path).
       * `generate_streaming_config_yaml` (asapquery_backend.rs
         — the legacy YAML emit path; same clamp so legacy /
         typed paths agree).
       * `build_processor_block` in the legacy agent emitter
         (agent.rs ~138).

Regression coverage:
  * 4 unit tests for `clamp_window_secs` itself (in-range,
    above-max, below-min, None).
  * 4 emit-level tests: legacy edge-yaml clamps 300 → 60,
    legacy edge-yaml preserves 30, 5-sketch routing clamps
    across all 5 family processors, streaming-config JSON
    clamps `windowSize`.
  * 3 legacy `agent.rs` tests covering the same clamp
    contract (clamps_oversize, clamps_undersize, preserves_inrange).
  * 1 pre-existing test (`contains_window_duration`) updated
    to assert the post-clamp 60s value instead of the
    fixture's pre-clamp 5m.

Test plan:
  * `cargo test -p control_plane --lib`: 719 pass (was 706
    pre-change baseline; +13 new tests covering both fixes)
  * `cargo test -p data_plane --lib`: 712 pass (no data_plane
    changes — verification only)
  * `cargo check -p control_plane`: clean

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 25, 2026
…agg from disk, and report real memory (#330)

PR #329's durable tier passed its unit tests but the first live run
(--persistence-seal-window-count=4 --persistence-hot-window-secs=120)
left parts/ empty after 13 min of ingest, lost all data on docker
restart, dropped [5m]/HLL queries under persistence, and reported
~0 KB sealed bytes. Three root causes:

1. Flush never fired (most severe). The flusher only ever flushes
   SEALED epochs, and sealing only fires on the count cadence
   (seal_window_count distinct windows). A slow/stalled series never
   reaches the cadence, so its aged windows sit un-sealed in
   current_epoch forever — never made durable. Fix: a time-driven
   "phase 0" seal — the flusher now rolls every current_epoch window
   older than the hot window into a sealed epoch each tick
   (EpochSource::seal_aged_epochs / SidStoreData::seal_aged_windows /
   MutableEpoch::split_window_ends_before) so it becomes flushable
   regardless of cadence. Parts now commit during runtime and survive
   restart.

2. Exact-agg disk read-back missing. query_exact_agg_range and
   exact_agg_coverage_bounds read only in-memory epochs, so a
   `sum by (...)` / rate query returned "No result" once its windows
   were flushed-then-evicted. Fix: both now union the durable tier,
   reconstructing scalar accumulators (Sum/Increase/MinMax + Multiple*)
   from disk via reconstruct_exact_agg, keyed by the rebuilt label map.

3. approx_memory_bytes ignored current_epoch, so the MEMORY_DIAG
   under-reported and the flusher's memory-pressure trigger was blind
   to the bulk of memory (which under persistence lives un-sealed in
   current_epoch). Fix: count hot current_epoch + sealed; relabel the
   diagnostic.

Persistence-OFF default path is unchanged (seal_aged is a no-op when
persistence_enabled is false; the disk unions are no-ops without a read
handle). Reproducing tests fail on origin/main and pass here:
live_aged_unsealed_panes_flush_and_survive_restart (#1),
live_exact_agg_resolves_from_disk_after_evict (#2),
live_total_memory_accounts_for_current_epoch (#3), plus columnar/seal
and flusher-level unit tests.

Remaining follow-up: MultipleMinMaxAccumulator (needs an out-of-band
min/max sub_type) and the sketch-backed accumulator forms still have no
generic byte factory, so their evicted-to-disk exact-agg portion is
skipped; they remain served from memory.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Jul 6, 2026
Broadcast only cells that moved by more than the §7 F2 per-cell threshold
T = ε‖C‖/(2k√(dw)) instead of every Δ≠0 cell (sparse_delta_cells_thresholded),
and fold ONLY the shipped cells into last_broadcast (apply_cells) so
sub-threshold changes accumulate and eventually ship — bounding the edge's
C_ref approximation error to T per cell. sparse_delta_cells now delegates with
T=0 (exact), so the sparse regime is unchanged: H=4 eval byte-identical
(531,132), alerts fire, references stay consistent.

Honest scope: on a Count-Sketch the payoff is small (H=2048/w=256 geometric
ramp 1,647,966 -> 1,316,158, ~20%; no better on Zipf) because random-sign
hashing + collisions homogenize cell magnitudes, so input skew does not become
cell skew for the threshold to exploit. Right mechanism, capped by CS
structure; the effective high-cardinality lever remains sizing w to the key
count. Test: thresholded_broadcast_drops_subthreshold_and_tracks_last.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant