fix: preserve per-series population in ERP execution - #616
Merged
Merged
Conversation
…xecution # Conflicts: # crates/asap_types/src/aggregation_config.rs # crates/asap_types/src/precompute_plan/catalog.rs # crates/asap_types/src/sds.rs # crates/asap_types/src/summary_catalog.rs # data_plane/src/drivers/ingest/prometheus_remote_write.rs # data_plane/src/drivers/query/servers/http.rs # data_plane/src/precompute_engine/output_sink.rs # data_plane/src/storage_engines/sketch_db/lifecycle/eviction.rs # data_plane/src/tests/test_utilities/engine_factories.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A Planner-selected per-series KLL materialization was rejected by the backend compiler because an empty grouping list could not distinguish independent series from a global reduction. The runtime also compared semantic slide frequency with stored pane duration during installation.
What
Represent population partitioning explicitly in catalog descriptors and precompute materializations. Preserve full series identity for supported raw per-entity updates, validate the scope against the installed DAG, and use one shared stored-window duration definition during compilation and installation.
How
PopulationPartitioningdistinguishesPerEntityfromGroupedand contributes to materialization identity. The admission gate remains limited to update forms the producer executes. A real process test measures KLL profiles, passes them through Planner selection, installs the selected plan, ingests two independent series, and verifies warm readout with a health-only fallback server.Before this PR
quantile_over_time(0.9, erp_latency[5s])could select a KLL configuration but lose the materialization to whole-query fallback. A physical pane size different from the semantic slide could also reject installation.After this PR
Two measured profile catalogs cause the real Planner to select K=32 and K=128 respectively. Both configurations produce different materialization identities, install successfully, and execute warm readouts for both labeled series. The evidence is a correctness fixture, not a latency or cost-reduction benchmark.
Verification
/mydata/erp-production-study/process-evidence.json(local artifact, not committed).Limitations
Stacked on #609 and shared executable/precompute contract changes. Finite replay tests use the production drain endpoint to wait for ingestion; this does not establish continuous population completeness. Empirical evidence supplies observed rank error, not a calibrated failure probability. Distributed collector scope and arbitrary summary update forms are not newly admitted.