fix(otlp-ingest): bump tonic max_decoding_message_size to 64 MiB - #70
Merged
Merged
Conversation
The OTLP gRPC receiver was using tonic's default 4 MiB receive
cap. A single agent window's typed-DDSketch full-state batch
(1000 series at the e2e harness's cardinality) runs ~17 MiB on
the wire, so the gateway-to-backend export hit:
rpc error: code = OutOfRange desc = Error,
decoded message length too large:
found 17103632 bytes, the limit is: 4194304 bytes
…and the OTLP exporter on the gateway retried forever with
exponential backoff. Result: the backend's OTLP receiver never
saw a single byte from the agent, even though all three tiers
were operationally healthy.
Bump the receiver's `max_decoding_message_size` to 64 MiB to
match what the agent and gateway already declare on their
own receivers (`max_recv_msg_size_mib: 64` in the YAMLs).
Companion fix on the agent side
(ASAPCollector#210) implements the missing
delta-emit path, which keeps subsequent-window payloads small
(only changed buckets) — but the FIRST window of every series
still ships full state, so the cap matters regardless.
## Verification
End-to-end with b3-delta agent (1000-series cardinality, 60s
window). Pre-fix: backend OTLP receiver shows nothing; gateway
log loops `decoded message length too large`. Post-fix:
OTLP modified-proto Ddsketch received
(metric=http_requests_total_latency_ms_quantile, dps=1000)
OTLP modified-proto sketch ingest:
1000 routed, 0 decode-failed (fallback), 0 unconfigured
…on every window, full or delta.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 1, 2026
Capture this session's merged backend work — #70 (tonic OTLP gRPC max_decoding_message_size bumped to 64 MiB) and #71 (store range_query_into overlap filter + engine closest-pane selection + Prometheus-adapter precompute_window annotation) — at the top of TODO.md so the runtime-warm-tier-actually-works claim is testable from the doc. Cited the matching collector-side PRs (ASAPCollector#210 + #211) in the companion-changes note so future readers can see both halves of the wire fix. Added one new entry under "Known reconciliation gap": `IngestState.sketch_snapshots` is RAM-only, so backend restarts break delta ingest until the agent restarts too. Same item is mirrored in the collector's PROGRESS.md follow-up list — fix on either side closes the gap. `_Last updated_` set to 2026-05-01. Docs only; no code changes. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 2, 2026
* docs(todo): sync after 2026-05-01 query-path-closes-the-loop work (#72) Capture this session's merged backend work — #70 (tonic OTLP gRPC max_decoding_message_size bumped to 64 MiB) and #71 (store range_query_into overlap filter + engine closest-pane selection + Prometheus-adapter precompute_window annotation) — at the top of TODO.md so the runtime-warm-tier-actually-works claim is testable from the doc. Cited the matching collector-side PRs (ASAPCollector#210 + #211) in the companion-changes note so future readers can see both halves of the wire fix. Added one new entry under "Known reconciliation gap": `IngestState.sketch_snapshots` is RAM-only, so backend restarts break delta ingest until the agent restarts too. Same item is mirrored in the collector's PROGRESS.md follow-up list — fix on either side closes the gap. `_Last updated_` set to 2026-05-01. Docs only; no code changes. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: retire sketch-core mirror (#73) * refactor: retire sketch-core mirror * refactor: switch consumer imports to asap_sketchlib::sketches::* Update PR #73 against the reorganized asap_sketchlib (PR #36): the runtime sketches no longer live under a dedicated `asap::` module — they were merged into the existing `src/sketches/` layout (single home per sketch concept, ASAP-runtime types appended to the file that already holds the high-throughput in-process variant). Mechanical path swaps in asap-query-engine: - `asap_sketchlib::asap::dd_sketch::*` → `::sketches::ddsketch::*` - `asap_sketchlib::asap::count_min::*` → `::sketches::countmin::*` - `asap_sketchlib::asap::count_sketch::*` → `::sketches::count::*` - `asap_sketchlib::asap::hll_sketch::*` → `::sketches::hll::*` - `asap_sketchlib::asap::kll::*` → `::sketches::kll::*` - `asap_sketchlib::asap::count_min_with_heap::*` → `::sketches::cms_heap::*` - `asap_sketchlib::asap::hydra_kll::*` → `::sketches::hydra_kll::*` - `asap_sketchlib::asap::set_aggregator::*` → `::sketches::set_aggregator::*` - `asap_sketchlib::asap::delta_set_aggregator::*`→ `::sketches::delta_set_aggregator::*` - `asap_sketchlib::asap::config::*` → `::asap_runtime::*` Naming-conflict renames carried through to the consumers: - `HllDelta` → `HllSketchDelta` (octo_delta::HllDelta still wins the short name) - `HeapItem` → `CmsHeapItem` (common::input::HeapItem still wins the short name) main.rs aliases `asap_sketchlib::asap_runtime as config` so the existing clap derive references (`config::DEFAULT_CMS_IMPL`, `config::configure(...)`) still work without touching the rest of the bin. Tests: - `cargo build --workspace` → clean - `cargo test -p query_engine_rust --lib precompute_operators` → 141 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed `e473ccc`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: align CountSketchDelta consumer with sketchlib-go wire format Track the additive `hh_keys` field on `asap_sketchlib::CountSketchDelta` so the proto delta path constructs the type with all fields filled in. Sends an empty `hh_keys` for now: the vendored Rust proto bindings in `asap_otel_proto::sketchlib::v1` haven't been regenerated against the latest `.proto` (which carries `hh_keys` on the Go side). The TopK rebuild on the proto-delta path will fire once those bindings sync; the sketchlib-go-aligned semantics are already in place underneath. Bumps the asap_sketchlib git dep to `refactor/wire-format-align-go` (see asap_sketchlib PR #37). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The OTLP gRPC receiver was using tonic's default 4 MiB receive cap. A single agent window's typed-DDSketch full-state batch (1000 series at the e2e harness's cardinality) runs ~17 MiB on the wire, so the gateway-to-backend export hit:
```
rpc error: code = OutOfRange desc = Error,
decoded message length too large:
found 17103632 bytes, the limit is: 4194304 bytes
```
…and the OTLP exporter on the gateway retried forever with exponential backoff. Result: the backend never saw a single byte from the agent, even though all three tiers were operationally healthy.
Bump the receiver's `max_decoding_message_size` to 64 MiB to match what the agent and gateway already declare on their own receivers (`max_recv_msg_size_mib: 64` in the YAMLs).
Companion fix
The agent-side delta-emit path was a stub returning full state on every window (fixed in ASAPCollector#210). Once that lands, subsequent windows ship sparse delta diffs — but the FIRST window of every series still ships full state, so this cap matters regardless.
Verification
End-to-end with the b3-delta agent (1000-series cardinality, 60s window).
Pre-fix: backend OTLP receiver shows nothing; gateway loops `decoded message length too large`.
Post-fix (with #210 also applied):
```
OTLP modified-proto Ddsketch received
(metric=http_requests_total_latency_ms_quantile, dps=1000)
OTLP modified-proto sketch ingest:
1000 routed, 0 decode-failed (fallback), 0 unconfigured
```
…on every window, full or delta.
Test plan
🤖 Generated with Claude Code