test: add production process E2E tests and manual runner

[zzylol]

Summary

• replace readiness-only and smoke-named coverage with real process-boundary E2E flows
• provide one manual runner, ./scripts/e2e.sh, with focused component targets; no CI setup is required
• add production control-plane, data-plane, monitor, Gorilla merger, and whole controller-to-query flows
• remove stale tests and every tracked Rust #[ignore]; ./scripts/e2e.sh list now audits all tracked Rust files and fails if an ignore marker returns
• add a strict in-process sketch/query diagnostic matrix with no expected-pass masking
• add production-process raw-oracle coverage for every sketch family:
  • DDSketch and KLL: expected quantiles computed from sorted raw values
  • HLL: expected cardinality computed from an exact raw HashSet
  • CountSketch and Count-Min Sketch: expected top-k identities/counts computed from an exact raw frequency map
• use sketch implementations only to encode collector-compatible modified-OTLP fixtures, never to compute expected results
• prove queries stayed in ASAPQuery (data_source: asap_query) and validate labels/result values
• remove the DDSketch test's fabricated future watermark; both fixture timestamps are real past event times
• describe DDSketch range coverage accurately as endpoint consistency, not unsupported Prometheus step resampling

Commands

# Stable local component and whole-path acceptance suites
./scripts/e2e.sh

# All five sketch families: production binary + independent raw oracle
./scripts/e2e.sh sketch-oracles

# Production oracle matrix and all in-process query diagnostics; both run even if one fails
./scripts/e2e.sh differential-all

# Repository-wide ignored-test audit
./scripts/e2e.sh list

PromQL coverage

PromQL shape	Sketch / path	E2E boundary and status at merge
quantile_over_time(0.5, metric[10s])	DDSketch	production binary, instant + range endpoint, exact raw median oracle — passing
quantile_over_time(0.5, metric[10s])	KLL	production binary, exact raw median oracle — strict No result regression
quantile_over_time(0.99, metric[10s])	DDSketch	production component and controller/backend HTTP matrix — passing representative path
quantile_over_time(0.99, metric[30s])	DDSketch	production controller -> BackendPlan -> OTLP -> PromQL, exact raw p99 oracle — passing
quantile_over_time(0.99, metric[3m])	DDSketch delta/sub-window	strict multi-window reconstruction diagnostic
count(metric)	HLL	production binary + exact raw distinct-set oracle, plus multi-SID diagnostic — strict No result regression
count_over_time(metric[10s])	CountSketch / CMS	instant and CMS range diagnostic coverage — strict planner/query regressions
topk(3, metric)	heap-backed CountSketch / CMS	production binary + exact raw frequency-map oracle and in-process matrix — strict No result/binding regressions
sum(metric)	capability-miss/fallback path	real HTTP feedback/routing E2E

The range endpoint is also tested for invalid start == end and step == 0
parameters. PR #478 does not claim Prometheus range-step resampling semantics;
that product gap is tracked separately in #487.

Data coverage

Data fixture	What it validates
DDSketch raw values [10, 12, 15, 20, 30, 45, 60, 80, 100]	deterministic exact median oracle and DDSketch relative-error bound
Whole-process DDSketch raw values 1..=100	controller-planned alpha/window and exact interpolated p99 oracle (99.01)
KLL raw values 1..=101, retained below K=200	exact median oracle without KLL compaction ambiguity
HLL 2,000 unique user IDs plus 500 duplicate observations	duplicate insensitivity and comparison with an exact HashSet cardinality of 2,000
Frequency keys gamma=200, alpha=100, delta=75, beta=50	CountSketch/CMS top-3 identities, ordering-independent label map, and exact raw counts (gamma, alpha, delta)
DDSketch delta windows 1..=30, 100..=130, 1000..=1030	delta reconstruction, sub-window ingest, cross-window merge, and union-distribution p99
Repeated past-timestamp fixtures followed by later event-time observations	real tumbling-window closure without fabricated future timestamps
service=checkout, service=oracle-e2e, and matrix service=e2e-test labels	policy fingerprint matching, grouping-label preservation, and public response labels
Multiple HLL series with distinct service labels	multi-SID cardinality classification/merge diagnostic
Modified OTLP protobuf and MessagePack payloads	DDSketch/KLL/HLL proto decoding and family-specific heap-bearing CountSketch/CMS MessagePack decoding
Empty/malformed range parameters (start == end, step == 0)	public API validation and explicit Prometheus-shaped errors

The expected values come from raw arrays, HashSet, or HashMap calculations.
Sketch readout APIs are not used as the oracle; sketch implementations are used
only to produce collector-compatible wire payloads.

Summary / materialization coverage

The terms in this table follow ASAPPlanner's canonical post-ASAP IR. Backend
implementation names such as Capability::QuantileApprox are not treated as a
public or second logical vocabulary.

ASAPPlanner summary/readout	Covered behavior
SketchAlgorithm::DDSketch + SketchQuery::Quantile	quantile readout, alpha propagation, instant/range endpoint consistency, grouped SID routing, multi-window and delta reconstruction
SketchAlgorithm::Kll + SketchQuery::Quantile	K parameter propagation, proto decode, selected quantile readout execution
SketchAlgorithm::Hll + SketchQuery::Cardinality	precision propagation, proto decode, duplicate-resistant cardinality readout, and multi-SID merge diagnostic
SketchAlgorithm::CountSketch + SketchQuery::PointCount	signed-frequency summary materialization and count_over_time readout diagnostic
SketchAlgorithm::Cms + SketchQuery::PointCount	count-min materialization, proto matrix decode, instant/range frequency readout diagnostic
SketchAlgorithm::CountSketchWithHeap + SketchQuery::TopK	typed SketchParams, family-specific MessagePack ingest, item labels/counts, and top-k readout
SketchAlgorithm::CmsWithHeap + SketchQuery::TopK	typed SketchParams, MessagePack ingest, item labels/counts, and top-k readout
Backend materialization	typed workload planning -> three-stage split -> BackendStageConfig -> BackendPlan publication/install/readback -> serving-time materialization lookup
Policy and storage routing	aggregation parameters -> policy fingerprint -> SID registration -> grouped SketchStore window -> ASAP query candidate
Summary execution	public PromQL -> Planner-selected summary/readout -> SummaryExecutor decode/merge/readout -> Prometheus vector/matrix adapter
Window lifecycle	tumbling-window closure, event-time advancement, allowed-lateness/flush behavior, delta/sub-window reconstruction, and live/shadow serving
Unsupported readout / fallback	real HTTP miss feedback and controlled exact-backend routing instead of treating a fallback response as ASAP summary success

Coverage means the scenario is present and executable. The strict KLL/HLL/
CountSketch/CMS failures documented below remain product gaps; they are not
reported as passing summary support.

Verification

Passing:

• cargo fmt --all -- --check
• bash -n scripts/e2e.sh
• git diff --check
• ./scripts/e2e.sh list: no tracked Rust #[ignore]
• ./scripts/e2e.sh differential: DDSketch production binary + raw median oracle + range endpoint consistency, 1 passed / 0 failed / 0 ignored
• previously verified component suites and ./scripts/e2e.sh whole, including the production controller -> backend -> OTLP -> SketchStore -> p99 PromQL path

Strict diagnostic result:

• ./scripts/e2e.sh sketch-oracles: DDSketch passes; production KLL, HLL, CountSketch, and CMS each return ASAP-local No result for query; 0 ignored
• these four failures are executable product regressions, not smoke assertions, skips, or expected failures; backend Production KLL, HLL, Count-Min, and CountSketch queries return No result #491 tracks serving triage
• the in-process matrix remains strict at 7 passed / 8 failed / 0 ignored and tracks additional planner/query/range gaps

Scope notes

• system remains explicit opt-in because it delegates to the sibling ASAPCollector multi-node harness and may build images, start containers, and use SSH.
• The VictoriaMetrics comparison remains an environment-gated external integration test when VM_ADDR is provided.
• Existing .worktrees/ content is not included.

[zzylol]

Added production-binary + independent raw-data oracle coverage for every sketch family in commits 2c7a5ed and 2a36b95. Run ./scripts/e2e.sh sketch-oracles: DDSketch passes; KLL/HLL/CountSketch/CMS remain strict ASAP-local No result failures tracked in backend #491. Also fixed the future-watermark fixture (#488), made the ignored-test audit repository-wide and fail-closed (#490), and kept differential-all running both diagnostic suites after failures.

[zzylol]

@milindsrivastava1997 FYI, and this integrate ASAPQuery PR#630