feat: PrecomputeJob execution endpoint for DataCollector integration - #1
Merged
Merged
Conversation
New endpoints for DataCollector controller integration:
POST /api/v1/precompute
- Receives PrecomputeJob from DataCollector controller
- Executes PromQL query_expr against stored sketches
- Returns query result or 404 if not answerable
GET /api/v1/health
- Health check for controller to verify backend is alive
GET /api/v1/store/metrics
- Returns list of metrics/aggregation IDs in store
- Controller can use this to verify data is flowing
Usage with DataCollector controller:
Controller creates PrecomputeJob with query_expr (e.g.,
"topk(10, count_over_time(m{env=\"prod\"}[1m]) by (svc))")
→ POSTs to backend /api/v1/precompute
→ Backend evaluates against SimpleMapStore
→ Returns approximate result
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Satisfies clippy dead_code lint on fields previously accepted but unused. These fields are part of the wire format from the DataCollector controller and are now surfaced in the tracing span for observability. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 13, 2026
Rebuilds the OTLP ingest path on top of the merged PR #1 + PR #2 changes so that OTLP-delivered metrics (and pre-built sketches from the DataCollector OTel collector) flow through the precompute engine's worker pool. The precompute engine performs window-aligned aggregation per StreamingConfig before writing to SimpleMapStore, matching the Prometheus / VictoriaMetrics ingest pattern. Architecture: DataCollector OTel collector → OTLP gRPC/HTTP (OtlpReceiver) → precompute engine ingest router → per-(agg_id, group_key) worker panes → StoreOutputSink → SimpleMapStore → query engine Key changes ----------- - PrecomputeEngine::new() now eagerly builds channels, router, agg_configs and a shared Arc<IngestState>. The state is exposed via a new `ingest_state()` getter so other ingest sources can push into the same worker pool without duplicating setup. - IngestState is promoted from pub(crate) to pub, alongside an `extract_group_key_for(series_key, config)` associated helper that other drivers (OTLP here, Kafka potentially later) reuse for label→group extraction. - New WorkerMessage::AccumulatorInput variant carrying a pre-built Box<dyn AggregateCore> with agg_id, group_key, and timestamp. Routed to workers by the same (agg_id, group_key) hash as GroupSamples. - GroupState gains a `sketch_panes: BTreeMap<i64, Box<dyn AggregateCore>>` alongside the existing `active_panes`, plus a new `process_accumulator_input()` worker method that: * merges incoming accumulators into the covering pane via merge_with, * honors late-data policy (Drop / ForwardToStore), * emits both raw-sample and sketch-pane outputs on window close. `flush_all` is also extended to drain sketch panes for closed windows. - New `merge_sketch_panes_for_window` helper mirrors `merge_panes_for_window`: oldest pane destructively taken, later panes cloned via `clone_boxed_core` for still-open sliding windows. - OtlpReceiver gains `with_ingest_state()` constructor. When wired to the precompute engine, OTLP requests are dispatched as GroupSamples (raw metric points) and AccumulatorInput (sketch payloads) through the engine's router. Label semantics are preserved — each point is formatted into a standard metric{k1="v1",k2="v2"} series key and run through the same extract_group_key pipeline used by the Prometheus path, so StreamingConfig.grouping_labels drives pane keying uniformly. - SketchPayload tuple is promoted to a structured `SketchPoint` that carries name, attr_name, labels, timestamp, and opaque payload bytes. Labels are now preserved through the sketch path (previously lost). - Incoming sketch payloads are wrapped in SketchEnvelopeAccumulator::from_proto_bytes (introduced by PR #2) and sent to the worker as AccumulatorInput. This preserves the full sketch state end-to-end; per-variant concrete decoding (CountMin → CountMinSketchAccumulator, KLL → DatasketchesKLLAccumulator, …) can layer on top later without changing the routing contract. - main.rs constructs the precompute engine BEFORE the OTLP receiver so the receiver can obtain an Arc<IngestState>. Without the precompute engine OTLP falls back to log-only mode. Rebase notes ------------ This commit subsumes the earlier PR #3 commit dd22379 (which wrote directly to SimpleMapStore with SumAccumulator placeholders). The original commit conflicted with PR #2's own OTLP changes after PR #2 landed on main; rather than carry two overlapping commits forward, the earlier placeholder work is replaced in-place by this coherent refactor. Verification ------------ - cargo check --all-targets: clean - cargo clippy --all-targets -- -D warnings: clean - cargo fmt --check: clean - cargo test --lib: 435 passed, 0 failed, 5 ignored Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
Apr 20, 2026
…able stats (#48) Task #34 gap #1 of 3. The §7 schema-timeline primitive (`SchemaRegistry::timeline_for_metric`) and the cross-schema combiner (`engines::timeline_dispatch::combine_statistic`) landed in PRs #20 / #22 / #25, but `SimpleEngine::handle_query_promql` was still resolving a single `agg_id` via `resolve_agg_info_promql` and running the full query against it. A query whose time range spans a reconfigure boundary (old `agg_id` retired, new `agg_id` created) saw a data cliff for the pre-boundary slice. This PR wires the dispatcher: * New `SimpleEngine::try_handle_query_promql_via_timeline`: 1. Parse + pattern-match the query, extract metric name. 2. Build a probe `QueryExecutionContext` to read the resolved `[t1, t2]` + `Statistic`. 3. Call `timeline_for_query(metric, t1, t2)`. Bail out with `None` (fall-through to default single-agg path) if fewer than two segments, or if the statistic is non-combinable (quantile / topk / cardinality / rate / increase — those follow in PR B2 with a Partial HTTP response surface). 4. Per segment: reuse `build_query_execution_context_promql_for_agg_id` from PR #37 (the extracted forced-agg-id entry point), clip the store plan's `[start, end]` to the segment's bounds, execute, collect results. 5. Group by label-tuple and fold per-group per-segment scalars through `combine_statistic`. Emit the combined scalar as an `InstantVectorElement`. Purged segments or segments whose `agg_id` is no longer in the config go into `unresolved` so the combiner sees them. * `handle_query_promql` now tries the timeline path first; returns immediately on `Some`, falls through to the existing single-agg path on `None`. Zero behavior change when the timeline has 0–1 segments for the query's metric (the common case today). ## Scope Combinable stats only: Count / Sum / Min / Max. Non-combinable stats still take the single-agg path — PR B2 will surface `CombinedResult::Partial` on the HTTP response so users see `{covered, missing: [segments]}` explicitly instead of a silent data cliff. ## Validation - `cargo test -p query_engine_rust --lib` — 728 pass (baseline unchanged; the dispatcher stays dormant when tests only register one schema per metric). - `cargo clippy --all-targets -- -D warnings` — clean - `cargo fmt --all -- --check` — clean ## Follow-ups (explicit non-scope here) - **Integration test** seeding two agg_ids + cross-boundary Sum query. Requires the full `PrecomputeEngine` setup harness the existing e2e tests use; deferred as a dedicated PR so this one stays a focused dispatcher patch. - **PR B2**: Partial response surface for non-combinable stats on the HTTP adapter. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 21, 2026
Addresses TODO.md blocker #1. Unblocks the paper's "sketches for hot, exact for cold" story: capability-miss queries now serve exact answers from raw observability samples in the cold tier instead of hitting Prometheus (or failing) unconditionally. ## What's new * `drivers/query/fallback/cold_store/` — storage-agnostic `ColdStore` trait + `LocalFsColdStore` impl. On-disk layout (`raw/<metric>/YYYY/MM/DD/HH/part-NNNNNN.jsonl`) is identical to what a future S3 cold store will use, so the adapter stays source-compatible when we swap backends. * `drivers/query/fallback/s3_adapter.rs` — `ColdFallback<S>` implements `FallbackClient`. Parses PromQL, extracts `(metric, predicates, op)`, scans the cold store, computes the answer. Supported shapes for v1: - bare instant vector selector (`metric{labels}`) - no-grouping scalar aggregation (`sum|count|avg|min|max(metric{...})`) Label matchers: `=`, `!=` — regex delegated upstream. Anything outside this surface falls through to the optional inner `FallbackClient` (chain-of-responsibility, typically the existing Prometheus proxy). * `drivers/query/fallback/metrics.rs` — hot/cold telemetry counters (`queryengine_hot_queries_total`, `queryengine_cold_queries_total`, `queryengine_cold_bytes_served_total`), keyed by `(metric, shape)`. Mirrors the PR #51 schema-barrier-counter pattern. * `AdapterConfig::prometheus_promql_with_cold` convenience constructor that composes the cold adapter in front of a Prometheus proxy. ## Routing No engine changes needed. The existing `process_query_request` already falls through to `FallbackClient` on engine-miss — configuring the fallback as a `ColdFallback` naturally routes Purged-segment / capability-miss queries through the cold tier with the Prometheus proxy as the tail-of-chain for unsupported shapes. ## Tests 752 → 777 tests (+25): * 5 unit tests on the JSONL format + hour-prefix helpers * 4 on `LocalFsColdStore::scan` (range filter, missing prefix, hour-boundary span) * 10 on `ColdFallback` plan extraction + aggregation + float formatting * 6 end-to-end HTTP integration tests in `tests/cold_fallback_tests.rs` covering: bare selector, sum aggregation, label filtering, unsupported-shape delegation, telemetry-counter increments, and the "Purged range served from raw" paper story.
4 tasks
zzylol
added a commit
that referenced
this pull request
May 12, 2026
…ema helper (#140) First sub-PR of Step γ (variant-by-variant consumer migration). Adopts the (c) bridge strategy: keep `legacy_expr::QueryExpr::Aggregate` as the L2 emit shape; add a one-way canonical-builder helper consumers call on demand. No construction site rewritten; no bridge enum variant added to legacy QueryExpr. This avoids the "child must be canonical" coupling that would otherwise force γ1 to migrate every other legacy variant (SketchAgg, WindowedAgg, TopK, etc.) in the same PR. ## What landed ### New: `controller/src/intent_algebra/aggregate_bridge.rs` (328 lines, 8 tests) ```rust pub fn bridge_aggregate_to_canonical( keys: &[ColumnRef], aggs: &[AggItem], having: &Option<legacy::Predicate>, schema: &Schema, ) -> Result<BridgedAggregate, BridgeError>; pub struct BridgedAggregate { pub by: Vec<ColumnId>, pub aggs: Vec<AggIntent>, pub having: Option<HavingPredicate>, } pub enum BridgeError { UnresolvedKey(ResolveError), HavingDeferred(QueryExprError), } ``` The `having` translation routes through `from_legacy_scalar` (Batch 2). E-deferred ScalarExpr variants (`FunctionCall`, `ScalarSubquery`, `InList`, `Between`) surface as `BridgeError::HavingDeferred(...)`. Test `bridge_having_deferred_e_variant_surfaces_error` is the contract; no in-tree construction site builds a HAVING with E-variants today. ### `column_resolution.rs` extension (+211 lines, +6 tests) ```rust pub fn output_schema_for_aggregate( input: &Schema, by: &[ColumnId], aggs: &[AggIntent], ) -> Schema; pub fn resolve_named_keys(keys: &[ColumnRef], schema: &Schema) -> Result<Vec<ColumnId>, ResolveError>; ``` Mirrors canonical `query_expr::QueryExpr::output_schema_in`'s Aggregate arm: outputs `by`-columns positionally + one column per `AggIntent::output_column(probe)`, strips `time_index`, sets `unique_keys = [by]`. This was Step β TODO #1. Step γ2-γ4 consumers descending into legacy `Aggregate.input` should pass `output_schema_for_aggregate(parent_schema, &bridged.by, &bridged.aggs)` as the inner subtree's `parent_schema`. ### Demo wire in `physical/allocator.rs::alloc_node` The legacy `QueryExpr::Aggregate { keys, aggs, having, input }` arm now calls the bridge to derive canonical-shape data and enrich `NodeAnnotation.rationale` with intent kinds + group-by column count. Emit shape stays legacy; behavior unchanged (only the rationale string carries extra info). Proves the bridge is reachable. ## Construction-site migration: 0 (intentional) Per strategy (c), all ~10 construction sites in `query_parser/{promql,sql}.rs`, `legacy_lower::lower_aggregate`, and the optimizer rewrite path still emit `legacy_expr::QueryExpr::Aggregate`. They migrate in γ7 once no legacy `input` subtree remains (SketchAgg, WindowedAgg, TopK migrations land first in γ2-γ4). ## Build + test - `cargo build --release -p controller` — clean - `cargo build --release -p query_engine_rust` — clean - `cargo test -p controller --lib` — **688 passed** (was 674; +14 new bridge + column_resolution tests) - `cargo test -p controller --bin controller` — 27 passed ## Known caveats - Canonical `QueryExpr::Aggregate.having` field is `Option<HavingPredicate>` where `HavingPredicate(pub String)`, NOT typed `Option<Predicate>` as the spec text suggested. Bridge renders the converted `Predicate` via `format!("{pred:?}")`. Round-tripping from the string is a follow-up (when canonical `having` upgrades to typed `Predicate`). - `BridgeError` can't derive `PartialEq` (QueryExprError doesn't); tests use `matches!`. Non-blocking. - `optimizer::engine::HydraConversion` (Step β TODO #7) NOT migrated — out of γ1's stated scope, deferred to a later γ sub-PR. ## Diff: 5 files, +618 / -4 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
zzylol
added a commit
that referenced
this pull request
May 13, 2026
Post-M2.3 reorg #1 of 8. Moves stateless payload types out of `store/mod.rs` into a new `sketch_db::data` module. The split clarifies that: - `data/` = stateless data taxonomy (payload kinds, sid hashing, accuracy derivation, capability re-exports) - `store/` = stateful sid registry + per-sid columnar substrate Moved to `sketch_db::data`: - `AggKind` enum (sketch-vs-precompute discriminator) - `AggPayload` enum (sketch bytes vs accumulator) - `SketchConfig` (per-variant tuning params) - `SketchSampleState`, `SketchEncoding` - `SketchTimeSeries` (read-side row shape) - `AccuracyBound` + `from_config` derivation - `compute_sid` / `compute_sketch_sid` (canonical sid hash) - `canonical_parameters` helper - Re-exports: `Capability`, `SketchKindHandle`, `AggregationType` `store/mod.rs` re-exports each type at the legacy path (`sketch_db::store::AggKind` etc.) so no external caller needs to change spelling. The canonical home is now `sketch_db::data::*`. 783/783 lib tests pass. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
zzylol
added a commit
that referenced
this pull request
May 13, 2026
Schema retirement #1 of 5. Adds `sketch_db::query::timeline::timeline_for_metric(&SketchStore, metric, t1_ms, t2_ms) -> Vec<TimelineSegment>` that computes the per-metric historical timeline entirely from the sid catalog (`SketchStore::instances`). Algorithm: 1. Snapshot all `SketchInstanceMetadata` for `metric` from the sid catalog. 2. Group by content signature `(metric, agg_kind, group_by_keys)`. Multiple sids sharing the same agg-config fold into one group. 3. Per group, fold the lifecycle fields: `min(first_seen_unix_ms)`, `Some(min(retired_at_ms))` iff every sid is retired, status = Active > Retired > Expired. 4. Apply the same segmenting + clipping as `SchemaRegistry::timeline_for_metric`. `TimelineSegment.agg_id` now carries a stable xxh64 of the content signature (the `(metric, agg_kind, group_by_keys)` tuple) — same content-derived id idiom as `compute_agg_config_id` (PR #151). HTTP callers see deterministic ids that don't depend on which specific sid was first seen. 7 new unit tests cover: empty store, inverted range, single-active signature, two-signatures-in-sequence, fold-of-many-sids-same-sig, metric isolation, and signature-id determinism. Schema/'s implementation stays alive in parallel until the consumers migrate. Adds `SketchStore::snapshot_instances()` to support read-side scans. 783 + 7 = 790 lib tests pass. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 17, 2026
Closed
zzylol
added a commit
that referenced
this pull request
May 18, 2026
…-Sum deferred) (#282) Two fixes that all live in control_plane/src/emit/ and adjacent files, bundled to avoid sequential merge churn. The third intended fix (B2 — Sum aggregation alongside sketch) was deferred because the existing workload_store keys by metric name and stores ONE QueryWorkload per metric — the multi-aggregation shape needed for `sum by (zone)` alongside `quantile_over_time` requires either a store-level restructure or a second PhysicalExpr per metric, both substantially bigger than #1+#2. Left as a follow-up; the YAML's existing sum-shaped entries (MVP entries 2/3/4 targeting http_requests_total) still collapse to the last write on a per-metric basis. == Fix #1: B3 population gap == Symptom (smoke test): every emitted `transform/keep_for_<metric>` block had `keep_keys(datapoint.attributes, [])` — strips ALL attrs instead of keeping `grouping_labels`. Sid catalog landed with one sid per metric instead of one per (metric, zone). Root cause: the OpAMP-push path's `metric_to_grouping_labels` WAS being populated from the WorkloadStore (in `main:: emit_bootstrap_typed`, `main::handle_plan`'s typed-stage-split branch, and `replan::Replanner::try_emit_typed_edge_yaml`). But the source `QueryWorkload.group_by_labels` was empty for the canonical MVP query `quantile_over_time(0.99, http_requests_total_latency_ms[30s])` — the PromQL parser only surfaces grouping labels from `by (...)` clauses, and a bare `quantile_over_time` has none. The pre-pop loop in `main.rs` also passed `group_by_labels: vec![]` in the QuerySpec, leaving nothing to merge with the empty parsed value. Fix: add a declarative `grouping_labels: Vec<String>` field to `WorkloadEntry` so the YAML can state the streaming-config grouping contract directly. Thread `entry.grouping_labels` into `QuerySpec.group_by_labels` in the pre-pop loop (main.rs ~276) so `analyzer.analyze()` merges the YAML-declared labels with any PromQL `by` keys into `QueryWorkload.group_by_labels`, which `collect_metric_to_grouping_labels` then drops into `EdgeStageConfig.metric_to_grouping_labels` → the emitter's `keep_keys(datapoint.attributes, [...])` list. Same plumbing in `emit::runtime_tests::populate_store_from_registry` so the existing 5-sketch round-trip test keeps tracking main.rs. Regression coverage (2 new tests in emit/mod.rs): * workload_entry_grouping_labels_round_trip_through_emit_to_keep_keys * workload_entry_grouping_labels_surface_in_emit_keep_keys_list == Fix #2: B4 — controller picks window_duration from query == Symptom: agent's sketch processor's `window_duration` was hardcoded at 300s (5m) for KLL/CMS/etc., 60s for others. The backend's streaming-config `windowSize` likewise drifted from whatever the user wrote in their PromQL `[range]`. Queries with `[30s]` ranges always landed inside an open sketch window and returned NoData. Root cause: the pre-pop QuerySpec hardcoded `time_window: "5m".into()` (main.rs ~278), which the analyzer prefers over the PromQL-parsed value at pipeline.rs:203. The parsed `[30s]` was thrown away. No clamp existed downstream to catch this either, so a `[5m]` workload landed a 300s sketch window that fell outside every sensible replay range. Fix: 1. Pass `time_window: ""` from main.rs and emit/runtime_tests when the entry HAS a `query_string` — lets the analyzer extract the matrix-selector range itself. Falls back to "5m" only when query_string is None (so the analyzer doesn't error at Step 4). 2. Add `clamp_window_secs(Option<u64>) -> Option<u64>` in emit/stage_config.rs with bounds [5, 60]: * Lower 5s — below this the sketch processor mints new windows before it has enough samples for the family's quality bound, and per-flush sid-catalog cardinality explodes. * Upper 60s (= MAX_WINDOW_SECS, the historical default). Above this the user's replay range no longer contains a closed sketch window. 3. Apply the clamp at every emission site so the agent's `window_duration` and the backend's `windowSize` agree exactly (drift de-syncs warm-tier replay): * `build_edge_processor_block` callers in the legacy single-pipeline and 5-sketch routing emit paths (stage_config.rs ~169 and ~1003). * `build_backend_aggregation_json` (stage_config.rs ~1636 — the streaming-config JSON path). * `generate_streaming_config_yaml` (asapquery_backend.rs — the legacy YAML emit path; same clamp so legacy / typed paths agree). * `build_processor_block` in the legacy agent emitter (agent.rs ~138). Regression coverage: * 4 unit tests for `clamp_window_secs` itself (in-range, above-max, below-min, None). * 4 emit-level tests: legacy edge-yaml clamps 300 → 60, legacy edge-yaml preserves 30, 5-sketch routing clamps across all 5 family processors, streaming-config JSON clamps `windowSize`. * 3 legacy `agent.rs` tests covering the same clamp contract (clamps_oversize, clamps_undersize, preserves_inrange). * 1 pre-existing test (`contains_window_duration`) updated to assert the post-clamp 60s value instead of the fixture's pre-clamp 5m. Test plan: * `cargo test -p control_plane --lib`: 719 pass (was 706 pre-change baseline; +13 new tests covering both fixes) * `cargo test -p data_plane --lib`: 712 pass (no data_plane changes — verification only) * `cargo check -p control_plane`: clean Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
zzylol
added a commit
that referenced
this pull request
May 25, 2026
…agg from disk, and report real memory (#330) PR #329's durable tier passed its unit tests but the first live run (--persistence-seal-window-count=4 --persistence-hot-window-secs=120) left parts/ empty after 13 min of ingest, lost all data on docker restart, dropped [5m]/HLL queries under persistence, and reported ~0 KB sealed bytes. Three root causes: 1. Flush never fired (most severe). The flusher only ever flushes SEALED epochs, and sealing only fires on the count cadence (seal_window_count distinct windows). A slow/stalled series never reaches the cadence, so its aged windows sit un-sealed in current_epoch forever — never made durable. Fix: a time-driven "phase 0" seal — the flusher now rolls every current_epoch window older than the hot window into a sealed epoch each tick (EpochSource::seal_aged_epochs / SidStoreData::seal_aged_windows / MutableEpoch::split_window_ends_before) so it becomes flushable regardless of cadence. Parts now commit during runtime and survive restart. 2. Exact-agg disk read-back missing. query_exact_agg_range and exact_agg_coverage_bounds read only in-memory epochs, so a `sum by (...)` / rate query returned "No result" once its windows were flushed-then-evicted. Fix: both now union the durable tier, reconstructing scalar accumulators (Sum/Increase/MinMax + Multiple*) from disk via reconstruct_exact_agg, keyed by the rebuilt label map. 3. approx_memory_bytes ignored current_epoch, so the MEMORY_DIAG under-reported and the flusher's memory-pressure trigger was blind to the bulk of memory (which under persistence lives un-sealed in current_epoch). Fix: count hot current_epoch + sealed; relabel the diagnostic. Persistence-OFF default path is unchanged (seal_aged is a no-op when persistence_enabled is false; the disk unions are no-ops without a read handle). Reproducing tests fail on origin/main and pass here: live_aged_unsealed_panes_flush_and_survive_restart (#1), live_exact_agg_resolves_from_disk_after_evict (#2), live_total_memory_accounts_for_current_epoch (#3), plus columnar/seal and flusher-level unit tests. Remaining follow-up: MultipleMinMaxAccumulator (needs an out-of-band min/max sub_type) and the sketch-backed accumulator forms still have no generic byte factory, so their evicted-to-disk exact-agg portion is skipped; they remain served from memory. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Jul 6, 2026
Broadcast only cells that moved by more than the §7 F2 per-cell threshold T = ε‖C‖/(2k√(dw)) instead of every Δ≠0 cell (sparse_delta_cells_thresholded), and fold ONLY the shipped cells into last_broadcast (apply_cells) so sub-threshold changes accumulate and eventually ship — bounding the edge's C_ref approximation error to T per cell. sparse_delta_cells now delegates with T=0 (exact), so the sparse regime is unchanged: H=4 eval byte-identical (531,132), alerts fire, references stay consistent. Honest scope: on a Count-Sketch the payoff is small (H=2048/w=256 geometric ramp 1,647,966 -> 1,316,158, ~20%; no better on Zipf) because random-sign hashing + collisions homogenize cell magnitudes, so input skew does not become cell skew for the threshold to exploit. Right mechanism, capped by CS structure; the effective high-cardinality lever remains sizing w to the key count. Test: thresholded_broadcast_drops_subthreshold_and_tracks_last. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds controller integration endpoints:
DataCollector controller → POST precompute job → backend evaluates → returns result.
🤖 Generated with Claude Code