feat(metrics): expose §6.3 barrier drops as Prometheus counter - #47
Merged
Merged
Conversation
Follow-up from #45: the barrier counter was only visible via the in-process AtomicU64 + debug log, so production deployments running at INFO had no graphable signal for silent drops. Changes: - New `stores::sketch_db::metrics` module with a `CounterVec` `queryengine_ingest_samples_blocked_by_schema_barrier_total`, keyed by `agg_id`, registered through the global `prometheus::default_registry()` so the existing `/metrics` handler (`handle_metrics` in drivers/query/servers/http.rs) scrapes it automatically. - Per-agg increment right alongside the atomic bump in `route_decoded_samples`, fed by the same per-batch `dropped_by_barrier` tally so the counter and the log stay consistent. - New unit test `barrier_prom_counter_increments_per_agg_label` that keys on a unique agg_id (9001) to get a deterministic baseline in the process-global registry. Why label by agg_id (not a scalar counter): lets operators alert on "drops on a specific agg while the registry still reports that agg Active" — the silent-drop regression the counter is meant to catch. 729 lib tests pass (+1), clippy clean, fmt clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
Apr 20, 2026
) Follow-up to #45 / #47. The Prometheus `CounterVec` `queryengine_ingest_samples_blocked_by_schema_barrier_total` (keyed by `agg_id`) was only bumped by the Prometheus / VictoriaMetrics remote-write path. The three OTLP barrier sites in `drivers/ingest/otel.rs` still silently dropped. This PR completes the observability story — no matter which driver the DataCollector ships through, a drop increments the same counter. Changes: - **`IngestState::record_barrier_drop(agg_id, count)`** — single helper that bumps both the in-process atomic AND the Prometheus `CounterVec`. All five ingest drivers funnel through it, so the `/metrics` number is a unified sum. - **`ingest_handler.rs`** — refactored `route_decoded_samples` to call the helper instead of maintaining its own inline atomic-increment + Prometheus-increment loop. Same batched debug-log semantics; fewer moving parts. - **`otel.rs`** — new local `flush_barrier_drops(state, map, driver_tag)` helper; each of the three barrier sites (`otlp-raw`, `otlp-sketch-envelope`, `otlp-modified-proto`) tallies drops in a `HashMap<agg_id, count>` and calls flush at loop exit. One summary debug log per driver per batch. Also: a new unit test `record_barrier_drop_advances_atomic_and_prom_counter` that asserts both sides of the helper's contract (atomic delta == 7 and Prometheus CounterVec delta == 7 when called with count=7 on a fresh agg_id label). 738 lib tests pass (+1), clippy clean, fmt clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #45. The §6.3 write-side schema barrier had a counter (`IngestState::samples_blocked_by_schema_barrier`) and a batched debug log, but both were in-process only. Production deployments running at `INFO` had no graphable signal for silent drops — the exact scenario the counter was meant to make visible.
What's in this PR
Why label by agg_id
Ops can alert on "drops on a specific agg_id while the registry still reports that agg Active" — the silent-drop regression the counter is meant to catch. A scalar counter (no labels) would hide which agg is leaking.
Validation
Follow-ups
🤖 Generated with Claude Code