From 5778e86b71eed44b66d282dde15f45f0da0ea18d Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Fri, 1 May 2026 12:08:41 -0400 Subject: [PATCH] fix(otlp-ingest): bump tonic max_decoding_message_size to 64 MiB MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The OTLP gRPC receiver was using tonic's default 4 MiB receive cap. A single agent window's typed-DDSketch full-state batch (1000 series at the e2e harness's cardinality) runs ~17 MiB on the wire, so the gateway-to-backend export hit: rpc error: code = OutOfRange desc = Error, decoded message length too large: found 17103632 bytes, the limit is: 4194304 bytes …and the OTLP exporter on the gateway retried forever with exponential backoff. Result: the backend's OTLP receiver never saw a single byte from the agent, even though all three tiers were operationally healthy. Bump the receiver's `max_decoding_message_size` to 64 MiB to match what the agent and gateway already declare on their own receivers (`max_recv_msg_size_mib: 64` in the YAMLs). Companion fix on the agent side (ASAPCollector#210) implements the missing delta-emit path, which keeps subsequent-window payloads small (only changed buckets) — but the FIRST window of every series still ships full state, so the cap matters regardless. ## Verification End-to-end with b3-delta agent (1000-series cardinality, 60s window). Pre-fix: backend OTLP receiver shows nothing; gateway log loops `decoded message length too large`. Post-fix: OTLP modified-proto Ddsketch received (metric=http_requests_total_latency_ms_quantile, dps=1000) OTLP modified-proto sketch ingest: 1000 routed, 0 decode-failed (fallback), 0 unconfigured …on every window, full or delta. Co-Authored-By: Claude Opus 4.7 (1M context) --- asap-query-engine/src/drivers/ingest/otel.rs | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/asap-query-engine/src/drivers/ingest/otel.rs b/asap-query-engine/src/drivers/ingest/otel.rs index fca11ea1..72f7ba72 100644 --- a/asap-query-engine/src/drivers/ingest/otel.rs +++ b/asap-query-engine/src/drivers/ingest/otel.rs @@ -97,10 +97,20 @@ impl OtlpReceiver { let grpc_svc = MetricsServiceImpl { shared: shared.clone(), }; + // Bump tonic's default 4 MiB receive cap. A single agent + // window emits ~1000 series, each carrying a typed + // DDSketch / KLLSketch / ... state — the full-state + // payloads run 17+ MiB at the cardinalities the e2e + // harness uses. With the default cap, the gateway's + // OTLP exporter retries forever with + // `decoded message length too large`. Match the + // gateway/agent receiver caps (`max_recv_msg_size_mib: 64` + // in their YAMLs) so all three tiers agree. let grpc_svc = asap_otel_proto::tonic::collector::metrics::v1::metrics_service_server::MetricsServiceServer::new( grpc_svc, - ); + ) + .max_decoding_message_size(64 * 1024 * 1024); let app = Router::new() .route("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/v1/metrics", post(handle_otlp_http))