Add batch/window modes, support sketch/raw SDK collection and benchmarking

[GnaneshGnani]

DDSketch processor

• Modes and inputs
  • Adds mode: batch (per-batch flush) and mode: window (tumbling window) to ddsketchprocessor.
  • Both modes accept Gauge and DDSketch inputs; batch mode appends summaries, window mode aggregates over time windows.
• Correctness and behavior
  • Window mode now preserves AggregationTemporality for DDSketch inputs, matching batch-mode semantics.
  • Fixes Start/Shutdown so window-mode shutdown does not block when the window goroutine was never started.
• Documentation
  • Rewrites ddsketchprocessor/README.md to treat input type (sketch vs. raw) and output mode (batch vs. window) as orthogonal dimensions.
  • Notes that only a partial DDSketch window-mode benchmark table (10k and 20k MPS) is currently recorded, with batch-mode and higher-rate rows to follow.

KLL processor

• Modes and behavior
  • Keeps support for both mode: batch and mode: window over Gauge inputs, with batch mode appending quantile metrics and window mode emitting per-window quantiles only.
  • Aligns lifecycle with DDSketch by applying the same non-blocking window-mode shutdown pattern.
• Correctness and tests
  • Retains the Prometheus-based correctness check in bench.sh that:
    • Scrapes p50, p90, p99 quantiles.
    • Verifies existence, monotonicity (p50 <= p90 <= p99), and bounds within the expected Zipf range.
  • Keeps unit tests that cover batch/window behavior and config validation.
• Documentation
  • otel_collector_benchmark/README.md already includes full KLL batch and window tables (10k–50k MPS) plus comparative analysis rows; this PR keeps that coverage in sync with the code.

CountSketch processor

• Modes and behavior
  • Uses mode: batch for per-batch CountSketch snapshots and mode: window for ticker-driven, per-window snapshots.
  • Batch mode can keep or drop original metrics via drop_original; window mode is tuned for high storage reduction with summaries only.
• Correctness and tests
  • Adds a Prometheus-based correctness check in cmd/bench.sh that:
    • Scrapes the exporter on port 8889.
    • Verifies both countsketch_row and countsketch_col metadata metrics are present; missing metrics cause the scenario to be flagged as a failure.
  • Keeps unit tests that validate pass-through, window flushing, batch semantics, and DropOriginal behavior.
• Documentation
  • Updates the CountSketch section in otel_collector_benchmark/README.md to:
    • Describe both batch and window modes with full 10k–50k MPS tables.
    • Call out the new correctness check alongside the existing performance and storage-reduction commentary.

Benchmark harness (cmd/bench.sh and otel_collector_benchmark/README.md)

• Processor targets
  • Ensures dedicated targets for:
    • ddsketchcol-batch, ddsketchcol-window
    • kll-batch, kll-window, legacy kll
    • countsketchcol-batch, countsketchcol-window
  • All targets share a single benchmark script and write results under otel_collector_benchmark/benchmark_results/{processor}/.
• Correctness checks
  • DDSketch: quantile existence, monotonicity, and bounds check via Prometheus on port 8889.
  • KLL: _p50, _p90, _p99 existence, monotonicity, and bounds check via the same exporter.
  • CountSketch: presence of countsketch_row and countsketch_col metadata metrics after each run.
• Environment hygiene
  • Frees both telemetry port 8888 and Prometheus port 8889 before each scenario to avoid stale metrics influencing correctness checks.
• Documentation
  • Updates otel_collector_benchmark/README.md to:
    • Describe the configuration and behavior of DDSketch, KLL, and CountSketch in both batch and window modes.
    • Include or explicitly call out the current status of benchmark tables (full for KLL/CountSketch, partial for DDSketch).
    • Summarize comparative performance at 50k MPS across the available sketch processors, with a note that DDSketch rows will be added once those runs are captured.

[Copilot]

Pull request overview

This PR adds two output-timing modes to ddsketchprocessor (per-batch and tumbling-window) so it can aggregate either Gauge samples or DDSketch inputs and then emit either merged DDSketch payloads or quantile gauges. It also wires these new modes into the benchmark harness and updates documentation with configuration guidance and representative benchmark results.

Changes:

• Implement mode: batch (flush on each incoming batch) and mode: window (periodic tumbling-window flush) in ddsketchprocessor.
• Extend tests to cover Gauge input in both modes and DDSketch input merging across multiple batches in window mode.
• Update benchmark script/configs/docs to run ddsketchcol-batch / ddsketchcol-window scenarios and perform a basic Prometheus-scrape quantile correctness check.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
otel_collector_benchmark/README.md	Documents DDSketch benchmark modes, correctness check, and sample window-mode results.
opentelemetry-collector-contrib-patch/processor/ddsketchprocessor/processor.go	Adds window-mode state, periodic flushing, and mode-aware ConsumeMetrics behavior.
opentelemetry-collector-contrib-patch/processor/ddsketchprocessor/factory.go	Switches from processorhelper.NewMetrics to returning the processor implementation directly.
opentelemetry-collector-contrib-patch/processor/ddsketchprocessor/config.go	Introduces mode/window_duration configuration with validation and defaults.
opentelemetry-collector-contrib-patch/processor/ddsketchprocessor/config_test.go	Adds mode-specific validation test cases.
opentelemetry-collector-contrib-patch/processor/ddsketchprocessor/processor_test.go	Adds tests for Gauge input in batch/window modes and DDSketch merging across batches in window mode.
opentelemetry-collector-contrib-patch/processor/ddsketchprocessor/README.md	Updates processor documentation for new modes/config, but currently contains conflicting legacy docs.
opentelemetry-collector-contrib-patch/cmd/otelcontribcol/go.mod	Adds generated builder module with a local replace for the patched ddsketchprocessor.
opentelemetry-collector-contrib-patch/cmd/ddsketchcol/config.yaml	Sets ddsketch processor mode explicitly to batch.
opentelemetry-collector-contrib-patch/cmd/ddsketchcol/config-window.yaml	Adds a new collector config for window mode, exporting Prometheus on 8889.
opentelemetry-collector-contrib-patch/cmd/bench.sh	Adds ddsketch batch/window targets, Prometheus correctness check, and output/input ratio reporting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 34 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.