Unify window candidate planning and preserve evaluation alignment - #712
Merged
Merged
Conversation
…andidates # Conflicts: # data_plane/src/query_engines/asap_query_engine/post_asap_readout.rs
This was referenced Sep 13, 2026
zzylol
added a commit
that referenced
this pull request
Sep 13, 2026
Rebuilds #700 on top of main (ca944a6) as the current-series feature alone. The window/full-window half of the original PR is dropped: #712 landed an equivalent, and main's version is kept wherever the two overlapped. Multiple current-value quantiles and TopK limits over the same metric now share one bounded maintained population instead of each planning its own state. Lowering follows the Planner-selected IR rather than reconstructing intent from query text, so a catalog rename cannot change the physical plan. - `SeriesPopulation` / `SeriesReadout` contract and `LogicalOperator::CurrentSeries` - Backend `CurrentSeriesStore`: bounded members, lookback expiry, shared ordered state feeding quantile / TopK / sum / count / avg readouts - Planner revision bumped to b8b5d705 for typed `ExactKind::Min`, with the Min readout wired through catalog resolution and the exact-agg merge path - `asap_current_series_populations` / `_cache_builds_total` on /metrics Review fixes carried in (from a review of the original PR): - Budget overflow no longer latches permanently. It records when the budget blew and re-arms once the lookback window has moved past it, instead of stranding the plan on Prometheus until an operator forces a replan. - Quantile readout guards an empty value set instead of underflowing `values.len() - 1` while holding the lock the remote-write path waits on. - `CompileError::Query` reports `query_id`, not the query text. - Full-window sketch reads skip a layout-provably-empty sid instead of discarding the sids already accumulated and dropping to the exact path. Deliberately not included: the original PR also reversed `count()` over HLL from register-merge to declined. That is a behavioral change unrelated to current-series and belongs in its own PR with its own argument; main's semantics are kept here. Verified on this tree: cargo fmt --check, clippy --workspace --all-targets -D warnings, and cargo test --workspace -- --test-threads=1 (2139 passed, 0 failed, 4 ignored). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 13, 2026
zzylol
added a commit
that referenced
this pull request
Sep 14, 2026
Rebased onto main (84109cb). Carries only the current-series feature; the window/full-window half of the original #700 was dropped when #712 landed an equivalent, and main's version is kept wherever the two overlapped. Multiple current-value quantiles and TopK limits over the same metric share one bounded maintained population instead of each planning its own state. Lowering follows the Planner-selected IR rather than reconstructing intent from query text, so a catalog rename cannot change the physical plan. - `SeriesPopulation` / `SeriesReadout` contract and the `CurrentSeries` residual operator - Backend `CurrentSeriesStore`: bounded membership, lookback expiry, shared ordered state feeding quantile / TopK / sum / count / avg readouts - Planner revision bumped to b8b5d705 for typed `ExactKind::Min`, wired through catalog resolution, the exact-agg merge path, and the backend wire sub-type - `asap_current_series_populations` / `_cache_builds_total` on /metrics Adopts main's current naming throughout (`PhysicalCompilationRequest`, `PhysicalDeploymentContext`, `ResidualQueryOperator`, `query_plan::residual`, `selected_plan_root`, `summary_store`, `allow_mixed_summary_and_exact_execution`) rather than the deprecated compatibility aliases. Serde wire names and JSON fixture keys are unchanged from main. Review fixes carried in: - Budget overflow no longer latches permanently. It records when the budget blew and re-arms once the lookback window has moved past it, instead of stranding the plan on Prometheus until an operator forces a replan. - Quantile readout guards an empty value set instead of underflowing `values.len() - 1` while holding the lock the remote-write path waits on. - Full-window sketch reads skip a layout-provably-empty sid instead of discarding the sids already accumulated and dropping to the exact path. Verified on this tree: cargo fmt --check, clippy --workspace --all-targets -D warnings, and cargo test --workspace -- --test-threads=1 (1750 passed, 0 failed, 14 ignored). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Sep 14, 2026
…700) Rebased onto main (84109cb). Carries only the current-series feature; the window/full-window half of the original #700 was dropped when #712 landed an equivalent, and main's version is kept wherever the two overlapped. Multiple current-value quantiles and TopK limits over the same metric share one bounded maintained population instead of each planning its own state. Lowering follows the Planner-selected IR rather than reconstructing intent from query text, so a catalog rename cannot change the physical plan. - `SeriesPopulation` / `SeriesReadout` contract and the `CurrentSeries` residual operator - Backend `CurrentSeriesStore`: bounded membership, lookback expiry, shared ordered state feeding quantile / TopK / sum / count / avg readouts - Planner revision bumped to b8b5d705 for typed `ExactKind::Min`, wired through catalog resolution, the exact-agg merge path, and the backend wire sub-type - `asap_current_series_populations` / `_cache_builds_total` on /metrics Adopts main's current naming throughout (`PhysicalCompilationRequest`, `PhysicalDeploymentContext`, `ResidualQueryOperator`, `query_plan::residual`, `selected_plan_root`, `summary_store`, `allow_mixed_summary_and_exact_execution`) rather than the deprecated compatibility aliases. Serde wire names and JSON fixture keys are unchanged from main. Review fixes carried in: - Budget overflow no longer latches permanently. It records when the budget blew and re-arms once the lookback window has moved past it, instead of stranding the plan on Prometheus until an operator forces a replan. - Quantile readout guards an empty value set instead of underflowing `values.len() - 1` while holding the lock the remote-write path waits on. - Full-window sketch reads skip a layout-provably-empty sid instead of discarding the sids already accumulated and dropping to the exact path. Verified on this tree: cargo fmt --check, clippy --workspace --all-targets -D warnings, and cargo test --workspace -- --test-threads=1 (1750 passed, 0 failed, 14 ignored). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Fixes #705. Window candidates previously came from multiple input paths and were rewritten after generation. In particular, a 60s lookback evaluated every 45s became a 60s tumbling window, leaving most scheduled reads unable to use summaries.
What
window_cost_modelcontract and one generator after logical selection. Remove the snapshot candidate map, HTTP candidate list and request-levelsynthesized_window_queriesflag; no compatibility aliases.How
gcd(W,E); complete windows preserve E, including E>W. Derived maintenance cohorts retain their actual executor restriction.Before this PR
For W=60/E=45, the backend offered a single 60s tumbling pane. At t=90 the requested
(30,90]could not be assembled. Snapshot and HTTP candidates followed different paths, and different E values prevented otherwise useful pane sharing.After this PR
The same demand offers 15s panes and a separately priced complete-window alternative. Both preserve the scheduled query range. Compatible workloads can share smaller panes when total cost falls; an off-grid request still uses exact fallback.
Evidence
Execution examples below are covered by the focused planning/runtime tests. Screenshots and performance measurements are not applicable to this correctness/interface change.
所有例子
下面 W 表示查询最近多少秒,E 表示每隔多少秒求值。区间统一采用 PromQL 的
(start,end]。E<W,且能整除:W=60,E=20。 存 20s 小块。t=60 读取
(0,20]+(20,40]+(40,60],t=80 读取(20,40]+(40,60]+(60,80]。也可以分别维护(0,60]、(20,80]等完整窗口,按维护与读取成本选择。E<W,但不能整除:W=60,E=45。 使用
gcd(60,45)=15s小块。t=90 读取(30,45]+(45,60]+(60,75]+(75,90]。45s 小块拼不出 60s;原先的 60s 大块也拼不出(30,90]。E=W:W=60,E=60。 存 60s 块,每次只读一块:t=60 读取
(0,60],t=120 读取(60,120]。E>W,且是 W 的整数倍:W=60,E=120。 60s 块可以回答 t=120 的
(60,120]和 t=240 的(180,240]。查询仍然每 120s 执行;连续维护的小块中有些不会被该查询使用。完整窗口方案也可按求值网格维护,跳过间隙。E>W,但不是 W 的整数倍:W=60,E=90。 使用 30s 块:t=90 读取
(30,60]+(60,90],t=180 读取(120,150]+(150,180]。后端本地完整窗口也支持这种不重叠且有间隙的求值网格。求值起点偏移:W=60,E=20,在 t=65、85、105 求值。 小块起点相位为 5s,t=65 读取
(5,25]+(25,45]+(45,65]。如果仍从 0s 分块,就无法拼出(5,65]。完整窗口则由求值终点减去 W,得到正确的窗口起点相位。临时查询不对齐的时刻。 已有从 0s 开始的 20s 块,用户在 t=67 查询最近 60s,需要
(7,67]。摘要读取报告 capability miss,由精确路径回答,不取整、不拼接错误边界。多个查询共享。 A 为 W=60/E=20,B 为 W=90/E=30。分别维护时,各读 3 块;共享一套 10s 块后,A 读 6 块、B 读 9 块。系统重新计算创建、维护、保留、回收和读取成本,仅在总成本更低时共享。来源、计算状态、准确率、运行策略和相位也必须兼容;第三个不兼容或代价过高的查询不会阻止其余查询共享。
想用比 E 更大的块:W=60,E=20,提议 30s 块。 t=60 可以用
(0,30]+(30,60]回答,但 t=80 的(20,80]无法拼出,因此拒绝这个普通 pane 布局。细块与粗块结合的 hierarchical rollup 需要额外运行时实现,本 PR 不把它冒充为可执行方案。后端或计算方式不支持某种布局。 生成候选前检查目标能力:Collector 不接受小于完整窗口的 pane,也不接受当前未支持的稀疏完整窗口方案;派生维护及其源状态使用运行时要求的完整 cohort。本地不支持的叶子或不对齐读取走精确回退;严格 Collector 计划无可执行方案时明确拒绝。
提供了实测成本。 如果 20s pane 的有效实测成本为 8,完整窗口成本为 12,选择较便宜的可执行方案。但把 20s pane 改成共享 10s pane 后,不能继续声称成本为 8。当前实现保护已选择的实测方案,自动共享只重算公式生成的方案;成本证据不能绕过布局、时效或身份校验。
Input migration
implementation.window_implementation_idandimplementation.implementation_costwithimplementation.window_cost_model.implementation_idand.cost. Removeimplementation.window_candidates.window_implementationswithwindow_cost_model; declareevaluation_phase_msexplicitly.quotescontain workload-scoped measured layout quotes. Production candidate generation is always automatic.Verification
asap_typestests, 746 control-plane library tests, 31 control-plane HTTP/binary tests, and 1,201 data-plane library tests passed. Process fixtures exposed by the stricter contract were corrected and rerun below; the initial workspace command itself was not a clean pass.cargo +1.98.0 test --workspace --test asapquery_compatibility_process_e2e -- --test-threads=4— 12 passed, 1 existing ignored Collector-schema test.cargo +1.98.0 test --workspace --test e2e_controller_plans_and_backend_serves -- --test-threads=4— 12 passed.--test-threads=4: backend process, ClickHouse differential/Q05, component process, edge runtime, monitor gRPC/process, Remote Write wire, and PromQL differential tests passed. All four sketch process-oracle tests also passed in the workspace run.cargo +1.98.0 clippy --workspace --all-targets -- -D warningsandcargo +1.98.0 fmt --all -- --check— passed after the final fixture changes.python3 -m unittest discover -s tools/o11y-execution -p 'test_*.py'— 50 passed.ASAP_TEST_TIMEOUT_SCALE=4andASAP_E2E_CONTROL_PLANE_BINpointing to the built control-plane binary. No hosted-CI completion is implied by these local results.Fixture migrations preserve each test's purpose: persistence/ERP tests declare complete 5s populations, transport tests declare their actual 1s states, and the warm counter matrix supplies both queried windows. ERP assertions check the full sample count and cardinality. Missing data still uses exact fallback.
New/updated focused test purposes:
derived_window_candidates_cover_dividing_nondividing_and_sparse_cadences: legal panes and unchanged E for all five cadence relations.scheduled_window_layouts_preserve_cadence_phase_and_readout: compiled pane/full-window bindings match scheduled ranges and reject off-grid boundaries.raw_leaf_keeps_its_cadence_beside_a_same_window_derived_cohort: state-specific cohort constraints do not rewrite a neighboring raw leaf.different_cadences_share_common_panes_only_when_cheaper: select sharing under expensive maintenance, retain independent panes under expensive reads.sharing_keeps_profitable_subsets_with_other_phases_or_finer_cadences: a third incompatible/expensive consumer cannot disable useful pair reuse.fractional_cadence_uses_native_fallback_without_truncation: unsupported time precision remains exact rather than changing demand.measured_window_quote_preserves_shape_and_price: preserve the matching measured quote while keeping other feasible alternatives.collector_generation_excludes_partial_panes_and_sparse_full_windows: enforce target capabilities before selection.quotes_cannot_bypass_layout_or_runtime_constraints: reject oversized flat panes and unimplemented hierarchical rollups.conflicting_quote_ids_and_duplicate_shapes_are_rejected: reject ambiguous evidence instead of silently dropping offers.serialized_generated_quote_loses_compiler_provenance: incoming evidence cannot authorize automatic repricing.compiled_window_schedules_execute_exact_ranges: runtime bucket assignment and installed readouts match raw integer samples for cadence/phase/layout combinations; off-grid reads miss.full_window_sketch_read_excludes_neighboring_windows: overlapping KLL states are not merged into the requested complete window.repeated_dashboard_executes_multiple_selected_panes: the production process derives and serves 5s panes for 10s lookbacks without fabricated layout prices.Independent static review covered phase handling, full-window reads, quote identities and pane sharing; its duplicate-ID and incompatible-third-consumer findings were addressed. No performance measurements or screenshots are claimed.
Architectural decisions
WindowCostModelreplaces alternate candidate injection APIs; measured evidence supplements the same feasibility generator.Limitations and follow-up
Runtime layouts retain whole-second precision: fractional cadences are not truncated and use native exact fallback. Hierarchical rollups remain unsupported. Missing/open panes still fall back instead of being assumed zero. Shared-pane selection is a deterministic cost comparison with greedy profitable subsets, not a claim of globally optimal workload partitioning.
Human review — do not complete with an agent