Skip to content

Add self-monitoring metrics for processors and SDK exports - #70

Merged
zzylol merged 3 commits into
mainfrom
68-datacolletor-path-self-monitoring
Mar 21, 2026
Merged

zzylol merged 3 commits into
mainfrom
68-datacolletor-path-self-monitoring

Conversation

@zzylol

@zzylol zzylol commented Mar 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add self-monitoring for the sketch collector processors and the SDK periodic metric reader
  • add YAML configuration switches for collector processors and the SDK reader via enable_self_monitoring
  • report processor traffic, runtime, and memory metrics, including explicit inbound and outbound throughput

Supported metrics

Collector processors

Enabled per processor with enable_self_monitoring: true.

Supported processors:

  • ddsketchprocessor
  • kllprocessor
  • hllprocessor
  • countsketchprocessor
  • countminsketchprocessor

Reported metrics:

  • otelcol_datacollector_processor_input_bytes
  • otelcol_datacollector_processor_output_bytes
  • otelcol_datacollector_processor_input_metric_points
  • otelcol_datacollector_processor_output_metric_points
  • otelcol_datacollector_processor_input_bandwidth
  • otelcol_datacollector_processor_output_bandwidth
  • otelcol_datacollector_processor_input_throughput
  • otelcol_datacollector_processor_output_throughput
  • otelcol_datacollector_processor_process_cpu_user_time
  • otelcol_datacollector_processor_process_cpu_system_time
  • otelcol_datacollector_processor_process_resident_memory
  • otelcol_datacollector_processor_heap_alloc
  • otelcol_datacollector_processor_heap_sys
  • otelcol_datacollector_processor_goroutines
  • otelcol_datacollector_processor_active_series

SDK periodic metric reader

Enabled in the SDK YAML reader config with enable_self_monitoring: true.

Reported metrics:

  • otel.sdk.metric.reader.export.duration
  • otel.sdk.metric.reader.export.bytes
  • otel.sdk.metric.reader.export.bandwidth
  • otel.sdk.metric.reader.process.cpu.user_time
  • otel.sdk.metric.reader.process.cpu.system_time
  • otel.sdk.metric.reader.process.memory.rss
  • otel.sdk.metric.reader.process.memory.heap_alloc
  • otel.sdk.metric.reader.process.memory.heap_sys
  • otel.sdk.metric.reader.runtime.goroutines

Testing

  • cargo test in controller
  • go test in opentelemetry-go/sdk/metric
  • go test ./internal/observ in opentelemetry-go/sdk/metric
  • targeted go test runs in ddsketchprocessor, kllprocessor, hllprocessor, countsketchprocessor, and countminsketchprocessor

@zzylol zzylol linked an issue Mar 21, 2026 that may be closed by this pull request
@zzylol
zzylol merged commit edacfbd into main Mar 21, 2026
@zzylol
zzylol deleted the 68-datacolletor-path-self-monitoring branch March 21, 2026 05:38
SieDeta pushed a commit that referenced this pull request Apr 17, 2026
…itoring

Add self-monitoring metrics for processors and SDK exports
zzylol added a commit that referenced this pull request May 1, 2026
…able e2e (#212)

Capture this session's merged work at the top, retire stale
follow-ups, add new ones discovered along the way.

New top section: "Single-pipeline multi-sketch + delta +
queryable warm tier (2026-05-01)" — itemises #210 (DDSketch
delta wire fixes — three connected bugs), #211 (windowed
processors pass-through so chained `[ddsketch, HLL, batch]`
works), and the companion backend PRs (#70 OTLP gRPC max msg
size, #71 store overlap filter + closest-pane + window
annotation). Live e2e capture pasted in as evidence.

Retired follow-up #1 ("Warm-tier sketch ingest is dropped at
the gateway") — superseded by the new top section. Replaced
the lingering "agent-side ddsketch→batch wiring quirk" note
that was the chain bug now fixed by #211.

Replaced the abbreviated follow-up list (#2 cold reader,
#3 reducer self-check) with the actual residual after this
session, in priority order:
  1. Run the P7 sweep + accuracy reducer over all five sketches.
  2. Backend in-memory sketch_snapshots cache loss on restart.
  3. Inference config breadth (only [1m] patterns today).
  4. Cold reader torn-line tolerance (carried over).
  5. Reducer self-check vs engine (carried over).
  6. Inline GOPRIVATE / GOTOOLCHAIN env vars in build_sketchcollector.sh.
  7. Decide on OTel submodule bumps still dirty in working tree.

Updated `_Last updated_` to 2026-05-01.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataColletor path self-monitoring

1 participant