fix(erp): bound retained runtime feedback across sources - #639
Merged
Merged
Conversation
…-values # Conflicts: # control_plane/src/query_plan/clickhouse_exact.rs # data_plane/src/query_engines/asap_clickhouse_query_engine/relational_adapter.rs
feat(precompute): execute immutable summary subDAGs
feat(erp): publish bounded materialization observations after finite completion
…rocess Enable typed Array and Tuple SQL planning
…ectASAP/ASAPQuery-backend into fix/bounded-runtime-feedback
…orkflow feat(erp): integrate bounded runtime observations into main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runtime ERP feedback must remain bounded across sources and catalog generations. A per-key ring alone allows total retained metadata to grow indefinitely and can leave an older fit eligible after a rejected observation.
This change adds global key and serialized-byte budgets, evicts whole oldest keys, and invalidates previous evidence for a key when its latest payload is malformed or oversized. RPC acknowledgements count only retained records. Size accounting streams into a counter without allocating an encoded payload copy.
Before this PR: each new source could allocate another ring; rejecting its latest observation could preserve an older fit.
After this PR: the default store retains at most 1,024 keys and 16 MiB of serialized records. Invalid latest evidence removes the previous records for that key, allowing ERP consumers to take their existing fallback path.
Verification: six focused runtime feedback tests passed, including global eviction, byte limits, malformed/oversized invalidation, and disabled retention. Independent source review passed.
The byte budget measures serialized records, not process RSS. This is stacked on #638; it does not introduce continuous replanning. Performance measurements and screenshots are not applicable.