Connecting DataCollector and ASAPQuery

[zzylol]

TODOs:

• sketchlib-go and sketchlib-rust share the serialization and deserialization methods, and use the same hash functions per sketch @SieDeta
• DataCollector's telegraf and OTel collector should output sketches, and ASAPQuery ingestion drivers should receive the sketches, being able to deserialize the transmission protocol and sketch payloads. @GnaneshGnani
• What protocols should we use for transmission between DataCollector and ASAPQuery? -> This determines which ingest driver(s) are implemented in ASAPQuery, and what configuration we should write for OTel collector/telegraf. @GnaneshGnani
• ASAPQuery precompute engine, and store engine should accept the ingested sketches. @zzylol
• ASAPQuery controller should decide what sketches to compute in DataCollector path. @zzylol

[zzylol]

@milindsrivastava1997 FYI.

[GnaneshGnani]

Datacollector Integration Plan

Integrate ASAPQuery with the OpenTelemetry Collector so OTLP metrics can be ingested directly

---

Design

Our design follows the OTLP Specification:

Choice	OTLP Spec Reference
gRPC port 4317	"The default network port for OTLP/gRPC is 4317."
HTTP port 4318	"The default network port for OTLP/HTTP is 4318."
HTTP path /v1/metrics	"The default URL path for requests that carry metric data is /v1/metrics and the request body is a Protobuf-encoded ExportMetricsServiceRequest message."
Content-Type application/x-protobuf	"The client and the server MUST set 'Content-Type: application/x-protobuf' request and response headers when sending binary Protobuf encoded payload."
Same port multiplexing	"Server implementations MAY accept OTLP/gRPC and OTLP/HTTP requests on the same port and multiplex the connections to the corresponding transport handler based on the 'Content-Type' request header." Normally gRPC uses port 4317 and HTTP uses 4318. This option lets a server listen on one port and inspect the Content-Type header to decide whether to handle the request as gRPC or HTTP. Both transports still send Protobuf-encoded ExportMetricsServiceRequest.

Using standard OTLP ensures compatibility with the OpenTelemetry Collector, SDKs, and ecosystem tooling.

---

End-to-End Flow

Instrumentation / Exporters
    |
    v
OpenTelemetry Collector
    |  receivers: otlp
    |  processors: batch (raw) OR KLL/CMS/HLL (sketch)
    |  exporters: otlphttp (Protobuf)
    |
    |  HTTP POST to ASAPQuery
    |  Content-Type: application/x-protobuf
    |  Path: /v1/metrics (ExportMetricsServiceRequest)
    v
ASAPQuery OTLP Receiver (gRPC :4317, HTTP :4318)
    |
    +-- Pathway 1 (raw metrics)     --> TODO: forward to precompute engine
    |
    +-- Pathway 2 (sketch payloads)  --> TODO: forward to precompute engine

Pathway 1 — Collector sends raw samples (e.g. with a batch processor). ASAPQuery receives them and runs aggregation in the precompute engine.

Pathway 2 — Collector runs KLL/CMS/HLL processors that aggregate at the edge and emit pre-built sketches. ASAPQuery receives sketch payloads and forwards them to the store (no aggregation needed).

---

Protocol Choice

OTLP over gRPC and HTTP (Protobuf) — standard, well-supported. Per the spec, binary Protobuf is the canonical encoding. Sketch payloads can be carried in data point attributes (e.g. kll.sketch_payload).

---

OTLP Protobuf vs Sketch Payload (sketchlib-go)

It's easy to assume they are the same: OTLP uses Protobuf (application/x-protobuf), and sketch payloads can be Protobuf or MessagePack. Both use binary encodings. The confusion is that the outer format (OTLP) and the inner sketch payload are different schemas. The OTLP transport uses opentelemetry-proto: the HTTP body is an ExportMetricsServiceRequest that structures metrics as resource → scope → metric → data point → attributes. For Pathway 2, we carry sketches as opaque bytes in a data point attribute (e.g. kll.sketch_payload). Those bytes can be a SketchEnvelope protobuf (sketchlib) or MessagePack, depending on the chosen encoding. DataCollector produces SketchEnvelope with sketch.SerializePortable() + proto.Marshal(env) (sketchlib-go); ASAPQuery consumes with SketchEnvelope::decode() (sketchlib-rust). OTLP Protobuf is the outer container; the sketch payload is the inner bytes (Protobuf or MessagePack) embedded in an attribute.

---

Implementation Plan

DataCollector

1. KLL, CMS, HLL processors switched to SerializePortable() + protobuf (sketchlib-go)
2. Reference configs for OTLP export to ASAPQuery

ASAPQuery

1. Pathway 1 — Forward raw OTLP metrics to precompute engine
2. Pathway 2 — Forward sketches to precompute engine, forward to store

---

References

• OTLP Specification 1.10.0 — encoding, transport, ports, Content-Type
• OTLP Protobuf definitions — ExportMetricsServiceRequest schema

[GnaneshGnani]

PRs Planned

DataCollector - 1 PR

1. KLL, CMS, HLL processors → SerializePortable() + proto.Marshal(env); add config-otlp-pathway1.yaml, config-otlp-pathway2.yaml (~100 LOC)

ASAPQuery - 1 PR

1. Extend process_otlp_request: for raw metrics, log series keys and counts; for sketch payloads, detect kll.sketch_payload / cms.sketch_payload in attributes → SketchEnvelope::decode() → log sketch type (~150 LOC)

@zzylol, pls check. The otel ingest driver has already been mainlined. So we just need to deserialize the sketches and print them for now. In the data collector, we need to use sketchlib-go for serializing and set the exporters to export the data to ASAPQuery