feat(deploy): all-sketches single-agent demo config (paper §architecture) - #271
Merged
Merged
Conversation
…ure)
One sketchcol agent runs all five sketch families
(`[ddsketch, KLL, HLL, countsketch, countmin, batch]`) concurrently
in a single OTLP metrics pipeline, so the backend serves all five
PromQL query families from a single agent. Paper §3 architecture
walkthrough material; pairs with the unified all-sketch
`backend-inference.yaml` warm-tier dispatch table.
Not §5 evaluation data — the per-sketch sweep
(`run_e2e_sweep.sh` + `baseline-{b3-delta,…}.yml` + per-sketch
`e2e-overlay-*.yml`) covers per-family CPU / bandwidth / accuracy
numbers; this config covers the multi-sketch concurrent story.
Files:
- `deploy/configs/sketchcol-agent-allsketches.yaml`
- `deploy/docker-compose/baseline-allsketches.yml`
- `deploy/configs/README-allsketches-demo.md`
- `PROGRESS.md` one-line entry
Output naming verified against `backend-inference.yaml`:
DDSketch + KLL co-emit `<input>_quantile` (different typed
`Metric.data` variants disambiguate), HLL emits `<input>_hll`,
CountSketch + CountMin both preserve raw metric name (different
typed variants). HLL coverage in the unified inference table is a
known gap (raw-name dispatch incompatible with the HLL processor's
mandatory-suffix encode path); documented in the README, tracked
as a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 6, 2026
…, #79, #82, #83, #80, #266, #271) PR #153 has been open ~3 weeks; refresh the doc-only facts that drifted without changing the doc's structure or its 9 query classes. Specifically: - Repo rename DataCollector → ASAPCollector (URLs + prose). - §2.1: SketchEnvelope is now sourced from sketchlib-go (Go) / asap_sketchlib (Rust); post-#270 the SDK aggregator for DDSketch was migrated off DataDog sketches-go onto sketchlib-go, so every modern processor's wire bytes now go through the SketchEnvelope carried as the typed *SketchDataPoint.sketch field of the modified OTLP proto. Drop the "feat/s3-files-mode worktree" mention — the proto file lives in opentelemetry-proto-patch/ on main. - §2.3: PromQL coverage table refreshed with the post-#79 33-pattern set (quantile_over_time × {0.5,0.9,0.95,0.99} × {1m,2m,5m}, sum/count/rate/increase over time, topk × {5,10,50}); add a pointer to PR #266's deploy-side mirror of the same patterns into the 5 backend-inference YAML overlays. - §5.1: cold-fallback parse_jsonl tolerates torn trailing lines per ASAPQuery-backend #80; drop the "modified-OTLP not yet adopted by backend" row — that gap closed. - §5.4: rewrite from "committed adoption plan" to "landed". Backend ingest path-deps the modified proto + asap-precompute-rs (PR #76, 2026-05-05); warm-tier sketch persistence pinned by #82 + wall-clock watermark fallback in worker.rs::flush_all from #83. The 2026-05-01 e2e shipped real PromQL responses for all 5 sketch families. SketchEnvelopeAccumulator + attribute-bytes path is now legacy only. - §6.1: backend ingest is no longer "after the modified proto is vendored" — it's running today via PR #76. - §10 item 1: mark backend modified-proto adoption as landed (kept as a landing record); cross-language byte-format parity #243 noted as the companion fix. - §11: add canonical pointers to sketchlib-go::SketchEnvelope (the cross-language wire format), the backend's design-phase3-asap- precompute-rs.md, and the all-sketches single-agent demo config (PR #271, deploy/configs/sketchcol-agent-allsketches.yaml). - Top: add a "Last refreshed: 2026-05-05" stamp. Doc structure unchanged: §§1–11 intact, the 5-stage block diagram, accumulator table, and 9 Q-C catalog rows are byte-for-byte preserved (only fact text changed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 6, 2026
* docs: pipeline query catalog (OTel + precompute engine + store)
Adds docs/pipeline-query-catalog.md describing which query classes can
be answered end-to-end by the current physical pipeline:
workloads → OTel sketchcol → OTLP → ASAPQuery-backend precompute engine
→ SimpleMapStore → SimpleEngine (PromQL / SQL / Elastic)
This is a *catalog*, not a compilation reference. For compile-time
query → sketch algebra mapping see docs/sketch-algebra-query-mapping.md
and controller/docs/query-to-sketch-translation.md; this doc only
describes what the implementation can physically serve today, row by
row, and where the integration gaps are.
Contents:
- pipeline block diagram of the five stages
- building blocks table for each stage (OTel sketch processors,
precompute-engine accumulators and their mergeability, SimpleEngine
PromQL operators)
- query catalog with 9 supported classes (frequency per group, top-K,
cardinality, quantiles, exact min/max/sum/increase, set change),
each row linking OTel op → precompute merge → stored accumulator →
PromQL surface → accuracy
- explanation of two-stage window aggregation (short OTel batches
folded into long backend windows, losslessly per mergeable sketch)
- multi-label GROUP BY constraint (grouping_labels must be ⊆ labels
preserved by the OTel processor)
- not-yet-answerable patterns and the reason for each (cross-metric
binary ops, exact-required PromQL functions, undecoded envelope
variants)
- integration gap writeup: SketchEnvelopeAccumulator preserves the
opaque bytes but merge_with is a no-op and query_statistic errors;
per-variant SketchEnvelope → concrete accumulator decoders are the
remaining work to make every catalog row hot end-to-end
- worked examples: DEBS Q1 EMA (DDSketch path), ClickBench Q17 TopK
(CountMin+heap path), and the two-stage 10s → 5min aggregation
- configuration knobs users actually tune (window/slide, grouping and
aggregated labels, allowed lateness, late data policy)
- future work list
No code changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): add §4 resource & bandwidth analysis
Adds a new section answering "is multi-stage execution actually a win
on bandwidth and resources?" — separating savings that come from
agent-side sketching vs savings that come from the precompute merge.
Contents of §4:
- three-scenario comparison (raw passthrough / agent sketches direct
to store / multi-stage), with explicit roles per stage
- reference workload (1000 agents, 10k events/s, CMS 2000×5, 50
backend groups, 300s window, 100 QPS) used for all the numbers
- link-by-link bandwidth table (events→agent, agent→backend,
backend→store write, store footprint, store→query read)
- resource-usage table (agent CPU/mem, backend ingest CPU/mem,
storage, query CPU, query latency)
- breakdown of where each multi-stage win comes from:
* agent→backend savings come from agent sketching, NOT multi-stage
* store write rate / footprint savings come from precompute merge
* query-side savings come from merge amortization (factor of Q,
the per-window query rate)
* cross-agent spatial collapse via grouping_labels projection
- summary table mapping each saving to its source and typical
magnitude (10×–10000×)
- §4.7 "when multi-stage does NOT pay off" with the four cases:
low query rate, identity windowing, non-mergeable operators,
bursty workloads with mostly-idle groups
Also renumbered §4–§8 → §5–§9 (and the corresponding subsection
numbers + §4.1 → §5.1 cross-reference) so the new analysis section
slots in between the catalog (§3) and the not-yet-answerable list
(now §5) without disturbing the rest of the structure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): add §7 use cases from real workloads
Brainstorms a curated set of aggregation queries — both simple and
complex — sourced from the project's own open issues, organized by
domain and by which pipeline benefit each one exercises.
Issues referenced for ground truth:
- #47 (benchmark datasets — finance, cluster telemetry, IoT, network,
mobility, healthcare)
- #78 / #85 (DEBS 2022 financial Q1–Q12, with full sketch mappings)
- #46 (MVP reduction targets — transmission, query latency, e2e cost)
- #49–#52 (the three orthogonal aggregation patterns: window per
series, series at each timestamp, matrix aggregation — the
three-pattern framework that organizes §7.1)
§7 contents:
- §7.1 the three aggregation patterns the collector supports, with
the matrix pattern flagged as the killer multi-stage win
- §7.2 financial: DEBS Q1 (EMA), Q3 (top-K active), Q4 (high/low/range),
Q5 (volatility via IQR), Q6 (distinct symbols)
- §7.3 cluster & cloud: heavy-hitter routes, p99 latency per service
per hour, container cardinality, noisy-neighbor pods (Google /
Alibaba / BOOM / MIT Supercloud workloads)
- §7.4 IoT & smart grid: rolling p95 household power, top transformers
by load, vibration percentile per turbofan engine
- §7.5 network & 5G: top source IPs per cell per second, packet size
distribution per BSS
- §7.6 mobility & healthcare: NYC Taxi top zones, p95 trip duration
per zone pair, MIMIC-IV HR per ward
- §7.7 complex multi-stage compositions:
- §7.7.1 cross-tenant fairness — distinct active users per region
per day (the matrix-aggregation killer example, ~500 000× store
reduction)
- §7.7.2 anomaly detection by comparing a host to its cluster's
p99 (two parallel sketches, cross-metric join flagged as §5
future work)
- §7.7.3 Bollinger bands chained on top of DEBS Q1 outputs
(downstream consumer changes window without re-asking the agent)
- §7.7.4 two-stage volume-weighted top-K (CMS+heap × KLL on the
same stream)
- §7.7.5 long-horizon distinct error fingerprints per service per
week (2016 5-min sketches merged into one HLL per week)
- §7.8 mapping the catalog to the three #46 reduction targets, with
cross-references to the §4 magnitudes (10×–100× transmission, 10×–
1000× tail-latency, 100×–10000× store reduction)
Each query in §§7.2–7.6 is listed end-to-end: OTel processor + agent
partition_by → backend AggregationConfig (aggregation_type +
grouping_labels + window_size) → stored accumulator → PromQL surface,
so a reader can verify against the building-blocks tables in §2 and
the resource model in §4 without leaving the doc.
Also bumped §8/§9/§10 numbering (Configuration knobs / Future work /
Pointers) and the stale §8 → §9 cross-reference inside §7.7.2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): §7.8 queries from production observability stacks
Adds a new subsection catalogueing queries lifted from three public
production observability tools, showing that the pipeline covers the
same question surface they do while inheriting §4's bandwidth/resource
savings. Each query is mapped end-to-end: OTel processor + agent
partition → backend AggregationConfig → stored accumulator → PromQL,
and every source URL is recorded in §10.
§7.8 contents:
§7.8.1 ClickHouse / ClickStack — OTel trace analytics
- top-10 endpoints by p99 latency over 24h
- per-service p50/p95/p99 dashboard
- top-K customer IDs causing errors (CMS + heap)
- distinct active users per service per day (HLL, equivalent to
ClickHouse uniq())
Sourced from ClickStack OTel-trace blog posts.
§7.8.2 VictoriaMetrics — MetricsQL dashboard patterns
- histogram_quantile(…_bucket[5m]) → replaced by direct KLL query,
avoids shipping bucket vectors entirely
- rate(http_requests_total[5m]) → Sum/Increase accumulator
- topk(10, sum by (instance) rate(node_cpu[5m])) → CountMin+heap
- count(count_over_time(… status=~5..[1h]) by (path)) → HLL on path
- quantile_over_time(0.99, mysql_query_duration[10m]) by (db_user) → KLL
- bottomk(5, avg_over_time(disk_io_time[1h]) by (device)) → KLL
Cross-reference to ASAPQuery#253 (k8s-mixin audit).
§7.8.3 NCCL Inspector — GPU collective communication observability
Ground truth lifted from NVIDIA's NCCL Inspector blog (per-collective
metrics: algorithmic bandwidth, bus bandwidth, execution time,
message size, collective type; dimensions: host / gpu_id / comm_id /
op_type / comm_size / pattern ∈ {nvlink-only, hca-only, mixed}).
Queries:
- p99 AllReduce execution time per communicator per minute (KLL),
with 800:1 cross-rank collapse
- top-K slowest ranks over 1h (KLL + topk)
- bus-bandwidth distribution per collective type per job (DDSketch)
- distinct active communicators per host per minute (HLL)
- message-size histogram per (comm, op_type) per minute (KLL)
- communication-pattern frequency breakdown per job (keyed CMS
using aggregated_labels=[pattern])
Also bumps existing §7.8 "Targets from #46" to §7.9 and adds an
"External references for §7.8" block at the end of §10 Pointers with
source URLs for ClickStack, ClickHouse OTel trace blogs, MetricsQL
docs, and NCCL Inspector / 2.26 / 2.24 blog posts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): fix PromQL count / distinct-count semantics
Two bugs in the PromQL examples, reported on the PR:
1. **Syntax:** `count_over_time(m[w]) by (d)` is a PromQL parse error
— rollup functions don't accept `by`/`without`; the clause
attaches to the outer aggregator. Rewritten everywhere as
`count by (d) (count_over_time(m[w]))` for distinct counting and
`sum by (d) (count_over_time(m[w]))` for per-group frequency.
2. **Semantics:** `count()` in PromQL counts *time series*, not
distinct values, and `count_over_time` counts *samples per
series*. The distinct-count effect comes from the idiom
`count by (X) (expression with label Y)`, which relies on
Prometheus's label-set uniqueness of series and requires `Y` to
actually be a label on the metric.
Changes:
- §2.3 — adds an inline note box explaining PromQL `count` vs
`count_distinct`, the idiomatic `count by (X) (count_over_time(…))`
form, the two common pitfalls (rollup-with-by and missing
grouping), and the fact that SimpleEngine recognizes the outer-
`count` shape and dispatches it to `SetAggregator` / `HLL`.
- §2.3 table row `count(count_over_time(m[w]) by (l))` →
`count by (l) (count_over_time(m[w]))`.
- §3 catalog:
- Q-C1 / Q-C2 (frequency, top-K by count) — rewritten with
`sum by (d) (count_over_time(m{f}[w]))`.
- Q-C3 (distinct / cardinality) — rewritten with
`count by (d) (count_over_time(m{f}[w]))` and an explicit
pointer to the §2.3 note.
- §7.2 DEBS Q3 top-K active symbols → `topk(10, sum by (symbol)(...))`.
- §7.2 DEBS Q6 distinct active symbols → global
`count(count_over_time(financial.last_trade_price[5m]))` with a
comment clarifying it's the cardinality read.
- §7.3 heavy-hitter routes → `topk(10, sum by (route)(...))`.
- §7.3 active containers → `count by (namespace)(count_over_time(
container_running[1m]))` with a clarifying comment.
- §7.5 top source IPs per cell → `topk(10, sum by (src_ip)(...))`.
- §7.6 NYC taxi top pickup zones → `topk(20, sum by (pickup_zone)(...))`.
- §7.7.1 distinct users per region per day → `count by (region)
(count_over_time(active_users_total[1d]))`.
- §7.7.4 two-stage volume-weighted top-K → `topk(10, sum by (symbol)
(count_over_time(order_events_total[5m])))`.
- §7.7.5 distinct error fingerprints per service per week →
`count by (service) (count_over_time(errors_total[1w]))`.
- §7.8.1 ClickHouse top-K customer IDs → `topk(20, sum by (customer_id)
(count_over_time(error_logs[1h])))`.
- §7.8.2 MetricsQL row for distinct error paths → global
`count(count_over_time(http_requests_total{status=~"5.."}[1h]))`.
- §7.8.3 NCCL distinct active communicators per host → rewritten to
`count by (host) (count_over_time(nccl_comm_active[1m]))`, with a
note that this relies on `comm_id` being a label on
`nccl_comm_active` and is dispatched to the HLL accumulator by
SimpleEngine. Also filled in `aggregated_labels=[comm_id]` in the
backend AggregationConfig snippet.
- §7.8.3 NCCL communication-pattern frequency breakdown per job →
`sum by (job, pattern) (count_over_time(nccl_op_total[1m]))`.
- §3.2 multi-label GROUP BY sentence — same fix for its illustrative
`topk(k, ... by (d1, d2))` snippet.
No structural changes; just correctness fixes on PromQL shapes and a
prominent note so the convention is unambiguous to readers translating
from SQL COUNT(DISTINCT …) or ClickHouse uniq(…).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): honest note about rollup-with-by shorthand
Previous commit (3a58214) rewrote count/count_over_time queries to
strict PromQL and claimed "the catalog below uses the correct
count by (...) form throughout" — but only for count queries. The
catalog still has 20+ quantile_over_time / avg_over_time / sum_over_time
/ min_over_time / max_over_time queries written as
`rollup(m[w]) by (d)`, which is the same kind of strict-PromQL parse
error. These were inherited from docs/sketch-algebra-query-mapping.md
which uses the same shorthand consistently.
Two options: rewrite all 20+ occurrences to strict PromQL (loses the
reader hint about which label the stored sketch is keyed by, and for
quantile/avg/min/max there is no clean `sum by` / `avg by` wrapper
that preserves semantics), or document the shorthand honestly. Taking
the second path.
The §2.3 note is now split into two parts:
① count vs distinct counting — standard PromQL, strict form
`count by (X) (count_over_time(...))`, used for every Q-C3-style
distinct-count query in the catalog.
② project shorthand `rollup(m[w]) by (d)` — inherited from
sketch-algebra-query-mapping.md, not strict PromQL, and explicitly
a reader hint that says "the stored backend sketch is keyed by
`d` via StreamingConfig.grouping_labels = [d]." A strict-PromQL
engine will reject it, but SimpleEngine picks the right stored
sketch from config regardless — dropping the `by (d)` yields an
identical runtime behaviour.
This matches the existing reference doc and doesn't require churning
the 20+ quantile / avg / min / max / sum_over_time queries in the
catalog. The note is unambiguous about which shorthand is strict
PromQL and which is project convention.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): rewrite §5 for fallback + split execution; add §8 design choices
Two related reframings driven by PR discussion.
§5 — was "Not (yet) answerable end-to-end", implying the listed
queries are dead-ends. They aren't. The pipeline is an *accelerator*
sitting on top of an exact backend DB, and every query has one of
three dispositions:
1. Full sketch path (§3 catalog) — ms-scale reads from merged state
2. Full fallback — forwarded to Prometheus / VictoriaMetrics /
ClickHouse / Elasticsearch via the adapters in
drivers/query/adapters/*.rs (prometheus_http, clickhouse_http,
elastic_http; tests in prometheus_forwarding_tests,
clickhouse_forwarding_tests, elastic_forwarding_tests)
3. Split execution — some sub-trees sketch-served, others exact,
combined at the SimpleEngine planner. This maps onto the
"Partial coverage" classification already documented in
docs/sketch-algebra-query-mapping.md §6.
§5 is now structured as:
§5.1 fully unaccelerated shapes served by the exact backend
(exact-required PromQL: last_over_time, deriv, delta,
predict_linear, bare selectors; cross-sketch reinterpretation;
undecoded SketchEnvelope variants; metrics with no
aggregation; cold-start queries awaiting plan push)
§5.2 the fallback architecture (prometheus_http.rs,
clickhouse_http.rs, elastic_http.rs adapters; the
fallback_always / fallback_disabled / fallback_url config
surface; explicit note that Scenario A from §4 is the
fallback's baseline performance floor)
§5.3 split execution with two concrete examples — the ClickBench
Q17-style `SELECT SearchPhrase, MIN(URL), COUNT(*)` (COUNT
sketched, MIN(URL) from exact backend, outer ORDER BY
combined), and the PromQL host-vs-cluster-p99 alert (RHS
quantile_over_time → KLL, LHS bare selector → Prometheus
exact, `>` at the SimpleEngine planner). Plus a list of
four common split patterns.
§5.4 the one real gap — the SketchEnvelopeAccumulator decoder
(previously called §5.1). Explicitly notes that even this
gap is masked operationally by the fallback today: users
get correct answers via forwarding while waiting for the
decoder to land. The gap is about losing §4's resource
savings on sketch-bearing queries, not about losing
correctness.
Bottom line added: "the pipeline never returns 'not supported' to a
user. Everything either hits the sketch path, falls through to the
exact backend, or splits between the two and recombines."
§8 — new section "Design choices: where does sketching start, and
where do raw metrics go?" covering two cross-cutting axes:
§8.1 three plausible sketching boundaries (SDK / agent / backend),
with a pros-cons table keyed to §4's three scenarios. The
current design places the boundary at the agent collector
because backend-side sketching reproduces §4 Scenario A and
loses the 10×–100× agent → backend bandwidth saving, while
SDK-side sketching is strictly better on the (usually cheap)
SDK → agent link but requires per-language SDK implementation
and app redeploy for reconfiguration. Edge cases where SDK
or backend are the right boundary are enumerated.
§8.2 the sketch lane vs the archive lane — why the pipeline has
TWO parallel lanes (lossy sketches + lossless Gorilla/S3),
not one. The Gorilla lane uses the gorillacol in
opentelemetry-collector-contrib-patch/cmd/ and the in-progress
feat/s3-files-mode S3 Files exporter.
§8.3 four reasons the two-lane design is justified (sketches are
lossy, Gorilla is lossless but compact ~10×, fallback reads
the archive lane, flush rates decouple).
§8.4 where to place the tee (in-agent vs in-backend vs hybrid)
and the per-metric policy matrix (sketch-and-archive /
sketch-only / archive-only / raw-passthrough), referencing
ASAPQuery#242 for runtime control.
§8.5 summary table mapping each design question to its chosen
answer and why.
Renumbering: old §8 Configuration knobs → §9, old §9 Future work →
§10, old §10 Pointers → §11, and the stale in-doc cross-references
to §§9/10 are bumped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): §5.4 rewrite for modified OTLP proto + corrections in §§1/2.1/5.1/6.1/6.2/10/11
Earlier passes of the doc described DataCollector's sketch emit path
as "SketchEnvelope bytes stuffed into OTLP DataPoint attributes
(kll.sketch_payload / cms.sketch_payload / ddsketch.sketch_payload)"
with backend-side decoding via SketchEnvelopeAccumulator. That is
NOT what DataCollector actually does on the wire.
DataCollector ships a modified opentelemetry-proto submodule
(see opentelemetry-proto/opentelemetry/proto/metrics/v1/metrics.proto
under the feat/s3-files-mode worktree) that extends Metric.data
with first-class sketch variants:
Metric.data oneof {
...
DDSketch ddsketch = 13;
KLLSketch kllsketch = 14;
CountSketch countsketch = 15;
CountMinSketch countminsketch = 16;
HLLSketch hllsketch = 17;
}
Each variant has a typed data-point message with attributes (labels),
start_time_unix_nano / time_unix_nano, per-window count/sum/min/max,
sketch bytes, and an encoding enum that distinguishes PROTO full
snapshots from *_DELTA delta-transmission payloads (tracking
DataCollector issues #62–#67). Every data point also carries an
optional series_id for stable series descriptors.
The DataCollector sketch processors — verified by grepping
opentelemetry-collector-contrib-patch/processor/*/processor.go —
emit via these native variants:
- kllprocessor: pmetric.MetricTypeKLLSketch, appendKLLSketchDataPoint
- countminsketchprocessor: pmetric.MetricTypeCountMinSketch
- (ddsketch/countsketch/hll processors follow the same pattern)
Meanwhile, ASAPQuery-backend's asap-query-engine/Cargo.toml depends on
the stock `opentelemetry-proto = "0.28"` crate, which does NOT have
these sketch variants. The backend's drivers/ingest/otel.rs match
statement only handles Gauge/Sum/Histogram/ExponentialHistogram/
Summary, so any KLLSketch/DDSketch/... metric from a DataCollector
processor lands in the `None => {}` branch and is silently dropped.
The PR #3 SketchEnvelopeAccumulator + attribute-scraping code path
(looking for kll.sketch_payload etc.) is effectively dead code —
nothing DataCollector emits today uses attribute-stuffing.
The practical consequence: sketch queries return correct answers today
but only because they fall through to the exact backend via the §5.2
forwarding adapters. The sketch fast path is entirely cold for
DataCollector-emitted metrics, not because of a decoder gap but
because the backend and the collector speak slightly different OTLP
dialects.
The fix is two sub-tasks:
1. Vendor/path-depend on DataCollector's modified opentelemetry-proto
so the tonic Rust bindings generate Data::Kllsketch /
Data::Ddsketch / Data::Countsketch / Data::Countminsketch /
Data::Hllsketch oneof variants.
2. Add per-variant handlers in drivers/ingest/otel.rs that read
the typed fields (attributes → labels, time_unix_nano →
timestamp, etc.) and decode the `sketch` bytes using the
`encoding` enum into the matching concrete accumulator via
WorkerMessage::AccumulatorInput.
Once both land, labels / timestamps / count/sum/min/max come for free
as typed proto fields (no attribute scraping, no SketchEnvelopeAccumulator
wrapper), and delta-transmission + series_id optimisations come along
with the vendored proto.
Sections updated:
§1 pipeline diagram — corrected "emits SketchEnvelope bytes in OTLP
attributes" to "emits via MODIFIED OTLP proto as first-class
Metric.data variants: DDSketch / KLLSketch / CountSketch /
CountMinSketch / HLLSketch; typed fields for attributes,
timestamps, count/sum/min/max, sketch bytes, encoding (with
DELTA variants for bandwidth-optimised transmission)".
§2.1 OTel processors table — rewrote the intro paragraph to say
processors emit "via first-class sketch variants of the modified
OTLP proto, not via DataPoint attribute bytes," and updated each
row to name the pmetric.MetricType* variant and its encoding
enum values. Added a paragraph clarifying that the older
asap_sketchlib SketchEnvelope is an in-process serialisation
type, *not* the wire format, and that attribute-bytes encoding is
a legacy path only used by standard-OTLP clients that cannot
speak the modified proto.
§5.1 "Fully unaccelerated" table — replaced the vague "SketchEnvelope
variants whose concrete decoder is not wired yet" row with an
accurate description: backend's stock opentelemetry-proto 0.28
does not know the modified OTLP's sketch variants; metrics are
silently dropped at proto-decode time; queries still answered
via the fallback.
§5.4 substantial rewrite — new title "The one real gap — backend
adoption of DataCollector's modified OTLP". Full walk-through:
- the modified proto's Metric.data oneof additions (tags 13–17)
- typed-field data-point shapes (with KLLSketchDataPoint as the
concrete example)
- verification that sketch processors use the native variants
(pmetric.MetricTypeKLLSketch / MetricTypeCountMinSketch / …)
- why typed fields are better than attribute-stuffing: labels,
timestamps, count/sum/min/max, encoding, series_id all typed
- the encoding enum with PROTO + PROTO_DELTA variants for
native delta transmission
- series_id invariant for stable series descriptors
- concrete state of ASAPQuery-backend: stock opentelemetry-proto
0.28, otel.rs match only handles Gauge/Sum/Histogram/etc.,
SketchEnvelopeAccumulator is a stub (merge_with no-op,
query_statistic Err)
- the two concrete fix tasks (vendor the proto + add per-variant
handlers), with a Rust pseudo-snippet of what the handlers
look like
- explicit note that PR #3's attribute-bytes path is dead code
for modern processors and can become a legacy hatch or be
deleted after the fix
- bottom-line restatement: the pipeline never returns "not
supported" to the user, because the fallback handles everything
today; the sketch fast path is cold because the dialects
differ.
§6.1 DEBS Q1 walk-through — replaced the fake "OTLP DataPoint with
ddsketch.sketch_payload attribute" wire-format block with the
real Metric.data = DDSketch{data_points: [DDSketchDataPoint{…}]}
shape, showing typed attributes / timestamps / count/sum/min/max
/ sketch bytes / encoding fields. Rewrote the backend-ingest
pseudo-code from the SketchEnvelopeAccumulator wrapper to a
Data::Ddsketch(dd) handler that reads the typed fields directly.
Updated the closing sentence to point at §5.4 instead of §5.1.
§6.2 ClickBench Q17 walk-through — same correction, showing
Metric.data = CountMinSketch{…} with typed CountMinSketchDataPoint
fields (sample_count, rows, cols, encoding, attributes) instead
of the "SketchEnvelope::CountMin in DataPoint attributes" fiction.
§10 future work item #1 — rewrote to name the two sub-tasks
(vendor the modified proto + add per-variant handlers) and point
at delta-transmission issues #62–#67.
§11 Pointers — added the modified OTLP wire protobuf as a first-class
reference, clarified that the asap_sketchlib SketchEnvelope is
the legacy in-process serialisation path (not the wire format).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): commit to modified OTLP proto; rename "tee" → "split point"
Two cosmetic but meaningful changes:
1. The modified OTLP proto is the chosen end-to-end wire format.
§5.4 is reframed from "The one real gap" to "Committed adoption
plan — backend support for DataCollector's modified OTLP proto".
All "gap" / "missing piece" language around §5.4 is softened to
"adoption plan" / "remaining piece to land". The technical
content (proto description, typed-field walkthrough, two-task
breakdown, code snippet) is unchanged — only the framing moves
from "unresolved problem" to "committed work item".
Cross-references updated:
- §5 intro (was "the one real gap that the fallback masks but
doesn't fix" → "the committed plan for closing the one
remaining gap")
- §6.1 worked example (was "that is the §5.4 gap" → "both are
tracked as the §5.4 adoption plan")
- §10 future work item #1 (now says "committed adoption plan"
and "remaining piece needed to make every row in the catalog
hot end-to-end")
2. "Tee" renamed throughout to "split" / "split point" — clearer
for readers who don't know the Unix `tee` / plumbing metaphor.
Five occurrences updated across §8.2–§8.4:
- "tee-and-switch" → "split at a single well-defined point"
- "§8.4 Where to place the tee" → "§8.4 Where to place the split
point, and what per-metric policy to use"
- "| Tee point | Trade-off |" → "| Split location | Trade-off |"
- "agent-side tee" → "agent-side split"
- "Where is the tee?" → "Where is the split between the two lanes?"
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): §10 — add asap-planner-rs → DataCollector controller consolidation
Adds a new item #2 in the Future work list: retire asap-planner-rs
from ASAPQuery-backend and consolidate on DataCollector's controller
as the single planner in the architecture.
Today there are two planners doing overlapping work:
- ASAPQuery-backend links asap-planner-rs in-process for query →
config generation (tracked in ASAPQuery #240, #241, #250, all
closed — the in-process library was integrated successfully).
- DataCollector's controller runs its own five-layer planner
(controller/docs/query-to-sketch-translation.md).
Having two planners means duplicated maintenance, ambiguity about
who owns a plan for a given query, and a risk that the
capability-miss path (§10 item for "SimpleEngine capability-miss →
ControllerClient.create_plan") could still go through the local
asap-planner-rs instead of the controller if the wiring is done
naively.
The consolidation item enumerates the six sub-tasks:
(a) audit asap-planner-rs call sites in asap-query-engine
(b) replace each with ControllerClient::create_plan(query_spec)
— the HTTP client from ASAPQuery-backend PR #2 already exists
(c) port any asap-planner-rs logic missing from the controller
into the controller (e.g. label auto-inference from Prometheus,
ASAPQuery#250)
(d) drop asap-planner-rs as a workspace member and dependency
(e) update tests
(f) document in this doc and in
controller/docs/query-to-sketch-translation.md
Sequencing note: should land after or alongside the capability-miss
→ controller call-out so there is never an ambiguous middle state
with both planners live.
Also bumped subsequent future-work item numbers 3 → 3, 4 → 4, 5 → 5
(was originally 2-5, now 2-6 after inserting the new item 2; fixed
a duplicate "3." label in the process).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): §10 — add MessagePack as parallel sketch payload encoding
New §10 future work item #2: add MessagePack as an opt-in
alternative to protobuf for the bytes inside the modified OTLP
*SketchDataPoint.sketch field. Decision committed in PR review;
sketchlib-go gains a parallel msgpack ser/de stack alongside its
existing protobuf path, the modified opentelemetry-proto extends
each per-sketch encoding enum with *_ENCODING_MSGPACK = 2, the
sketch processors gain a per-processor `payload_encoding` config
knob, and the ASAPQuery-backend per-variant handlers + concrete
accumulators dispatch on the encoding enum between from_proto and
from_msgpack constructors.
Ground rules baked into the item:
- protobuf stays the default everywhere
- MessagePack is opt-in per metric at the sketchcol processor config
- delta transmission stays protobuf-only (no *_ENCODING_MSGPACK_DELTA)
- the OTLP envelope itself always stays protobuf
- every sketch type must support both encodings (no msgpack-only types)
- cross-format equality test is the regression anchor
Trade-off context (qualitative + benchmark) is captured in
ProjectASAP/sketchlib-go#26 — Go-side msgpack serialize is ~5×
faster on small sketches, Rust-side protobuf deserialize is
~1.86× faster, payload sizes are within 15% of each other. The
opt-in covers the deployment case where Go-side serialize is the
edge bottleneck.
Sequencing: depends on the modified OTLP adoption work (item 1)
landing first, so there is a working protobuf baseline to compare
the msgpack path against.
Bumped subsequent items in §10 by one (was #2-6, now #3-7).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): §10 — MessagePack delta transmission gets full parity with protobuf
Earlier pass on the §10 item carved out delta transmission as
"protobuf-only to avoid double-implementation drift." Reverting
that decision per design discussion — MessagePack should have full
feature parity with protobuf, including delta transmission. The
extra discipline cost (a 2x2 matrix of encoding × mode instead of
2x1) is bounded by a single regression-anchor test that asserts
all four (format, mode) combinations produce identical query
results on the same source data.
What changed in §10 item #2:
- Encoding enum extension: each per-sketch encoding enum now
gains TWO new variants instead of one:
*_ENCODING_MSGPACK = 2 (full snapshot)
*_ENCODING_MSGPACK_DELTA = 3 (delta)
Applied to KLLSketchEncoding, DDSketchEncoding,
CountSketchEncoding, CountMinSketchEncoding, HLLSketchEncoding.
- sketchlib-go scope grows: msgpack ser/de needs both full and
delta variants, with cross-language round-trip test for both.
- Sketch processor config: existing `delta_transmission` knob
composes orthogonally with the new `payload_encoding` knob.
Cross-product is {proto, msgpack} × {full, delta} = four
output paths per processor.
- Backend per-variant handlers: dispatch on dp.encoding() with
FOUR paths per sketch type (was two): from_proto / proto-delta
apply / from_msgpack / msgpack-delta apply.
- Accumulator API: each accumulator type gains a parallel
merge_msgpack_delta(&self, bytes) -> Self method alongside the
existing merge_proto_delta. The per-series baseline is shared
across encodings — a series can switch between proto-delta and
msgpack-delta mid-stream.
- Test: was "two-way cross-format equality", now "four-way
correctness across all (format, mode) combinations."
Ground-rule line that was removed:
- Old: "delta transmission stays protobuf-only"
- New: "both formats support both full and delta transmission
with no asymmetry"
Other constraints unchanged: protobuf default everywhere,
MessagePack opt-in per metric, OTLP envelope always protobuf,
every sketch type supports both encodings (no msgpack-only types).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(pipeline-query-catalog): refresh stale facts (post-#243, #270, #76, #79, #82, #83, #80, #266, #271)
PR #153 has been open ~3 weeks; refresh the doc-only facts that drifted
without changing the doc's structure or its 9 query classes.
Specifically:
- Repo rename DataCollector → ASAPCollector (URLs + prose).
- §2.1: SketchEnvelope is now sourced from sketchlib-go (Go) /
asap_sketchlib (Rust); post-#270 the SDK aggregator for DDSketch
was migrated off DataDog sketches-go onto sketchlib-go, so every
modern processor's wire bytes now go through the SketchEnvelope
carried as the typed *SketchDataPoint.sketch field of the modified
OTLP proto. Drop the "feat/s3-files-mode worktree" mention — the
proto file lives in opentelemetry-proto-patch/ on main.
- §2.3: PromQL coverage table refreshed with the post-#79 33-pattern
set (quantile_over_time × {0.5,0.9,0.95,0.99} × {1m,2m,5m},
sum/count/rate/increase over time, topk × {5,10,50}); add a pointer
to PR #266's deploy-side mirror of the same patterns into the 5
backend-inference YAML overlays.
- §5.1: cold-fallback parse_jsonl tolerates torn trailing lines per
ASAPQuery-backend #80; drop the "modified-OTLP not yet adopted by
backend" row — that gap closed.
- §5.4: rewrite from "committed adoption plan" to "landed". Backend
ingest path-deps the modified proto + asap-precompute-rs (PR #76,
2026-05-05); warm-tier sketch persistence pinned by #82 + wall-clock
watermark fallback in worker.rs::flush_all from #83. The 2026-05-01
e2e shipped real PromQL responses for all 5 sketch families.
SketchEnvelopeAccumulator + attribute-bytes path is now legacy only.
- §6.1: backend ingest is no longer "after the modified proto is
vendored" — it's running today via PR #76.
- §10 item 1: mark backend modified-proto adoption as landed (kept as
a landing record); cross-language byte-format parity #243 noted as
the companion fix.
- §11: add canonical pointers to sketchlib-go::SketchEnvelope (the
cross-language wire format), the backend's design-phase3-asap-
precompute-rs.md, and the all-sketches single-agent demo config
(PR #271, deploy/configs/sketchcol-agent-allsketches.yaml).
- Top: add a "Last refreshed: 2026-05-05" stamp.
Doc structure unchanged: §§1–11 intact, the 5-stage block diagram,
accumulator table, and 9 Q-C catalog rows are byte-for-byte preserved
(only fact text changed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
(DDSketch + KLL + HLL + CountSketch + CountMinSketch) concurrently
in a single OTLP metrics pipeline (
[ddsketch, KLL, HLL, countsketch, countmin, batch]).docs/paper-outline.md§Architecture: backendserves all five PromQL query families from one agent. Pairs with
the unified all-sketch
backend-inference.yaml.data (per-sketch sweep covers per-family numbers, this config
covers the multi-sketch concurrent story).
Files
deploy/configs/sketchcol-agent-allsketches.yamldeploy/docker-compose/baseline-allsketches.ymlbaseline-b3-delta.yml)deploy/configs/README-allsketches-demo.mdPROGRESS.mdOutput naming
Verified against
backend-inference.yaml(the unified warm-tierdispatch table):
<input>_quantile*_quantilequantile family)<input>_quantile(same name as DDSketch — typedMetric.datavariant disambiguates)<input>(raw — processor has nometric_suffix)http_requests_totaltopk/sum)<input>(raw — typedCountMinSketchDataPointdisambiguates from CountSketch)<input>_hllOpen question
The HLL processor's encode path
(
hllprocessor/encode.go::cardinalityMetricName) always appends anon-empty suffix —
<input>_hll(withmetric_suffix: "_hll") or<input>_hll_cardinality(default); there is no way to emit on thebare raw name. The unified
backend-inference.yamllistscount(http_requests_total)(raw name) under HLL coverage, so HLLqueries against this agent currently fall through to cold tier.
Documented in the README and tracked as a follow-up — orthogonal to
the §3 architecture-figure story which is "five concurrent sketch
pipelines emit to backend", not "every query is warm-tier ready".
Validation (static-only — sweep agent owns docker)
python3 -c "import yaml; yaml.safe_load(...)"succeeds for both new YAML files.opentelemetry-collector-contrib-patch/processor/*/factory.go + metadata.go'sMustNewType:ddsketch,KLL,HLL,countsketch,countmin.backend-inference.yaml/ per-sketchbackend-inference-*.yaml.(
opentelemetry-collector-contrib-patch/cmd/sketchcollector/builder-config-sketches.yaml)compiles all five processors — the binary supports the chain.
Test plan
AGENT_CONFIG=sketchcol-agent-allsketches.yaml docker compose -f base.yml -f agents-N1.yml -f baseline-allsketches.yml -f e2e-overlay.yml up -dand verify each example PromQL from the README returns a non-empty result.🤖 Generated with Claude Code