Skip to content

Unify window candidate planning and preserve evaluation alignment - #712

Merged
zzylol merged 4 commits into
mainfrom
feat/unified-window-candidates
Sep 13, 2026
Merged

zzylol merged 4 commits into
mainfrom
feat/unified-window-candidates

Conversation

@zzylol

@zzylol zzylol commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Why

Fixes #705. Window candidates previously came from multiple input paths and were rewritten after generation. In particular, a 60s lookback evaluated every 45s became a 60s tumbling window, leaving most scheduled reads unable to use summaries.

What

  • Snapshot and HTTP planning now use one window_cost_model contract and one generator after logical selection. Remove the snapshot candidate map, HTTP candidate list and request-level synthesized_window_queries flag; no compatibility aliases.

How

  • Generate layouts from each selected state's requirements and the deployment target. Ordinary backend-local panes use gcd(W,E); complete windows preserve E, including E>W. Derived maintenance cohorts retain their actual executor restriction.
  • Preserve evaluation phase, distinguish full-window slide from stored width, and read only the requested complete window instead of merging overlapping neighbors.
  • Compare independently selected additive panes with a freshly priced shared common-divisor producer. Charge each consumer's own read cadence. If a whole group cannot share, retain profitable compatible subsets.
  • Bind measured costs to their exact workload/layout. Protect selected measured quotes from automatic repricing; reject unsupported layouts and conflicting quote identities. Tests specify layouts directly when testing a particular executor, rather than inventing zero prices.

Before this PR

For W=60/E=45, the backend offered a single 60s tumbling pane. At t=90 the requested (30,90] could not be assembled. Snapshot and HTTP candidates followed different paths, and different E values prevented otherwise useful pane sharing.

After this PR

The same demand offers 15s panes and a separately priced complete-window alternative. Both preserve the scheduled query range. Compatible workloads can share smaller panes when total cost falls; an off-grid request still uses exact fallback.

Evidence

Execution examples below are covered by the focused planning/runtime tests. Screenshots and performance measurements are not applicable to this correctness/interface change.

所有例子

下面 W 表示查询最近多少秒,E 表示每隔多少秒求值。区间统一采用 PromQL 的 (start,end]

  1. E<W,且能整除:W=60,E=20。 存 20s 小块。t=60 读取 (0,20]+(20,40]+(40,60],t=80 读取 (20,40]+(40,60]+(60,80]。也可以分别维护 (0,60](20,80] 等完整窗口,按维护与读取成本选择。

  2. E<W,但不能整除:W=60,E=45。 使用 gcd(60,45)=15s 小块。t=90 读取 (30,45]+(45,60]+(60,75]+(75,90]。45s 小块拼不出 60s;原先的 60s 大块也拼不出 (30,90]

  3. E=W:W=60,E=60。 存 60s 块,每次只读一块:t=60 读取 (0,60],t=120 读取 (60,120]

  4. E>W,且是 W 的整数倍:W=60,E=120。 60s 块可以回答 t=120 的 (60,120] 和 t=240 的 (180,240]。查询仍然每 120s 执行;连续维护的小块中有些不会被该查询使用。完整窗口方案也可按求值网格维护,跳过间隙。

  5. E>W,但不是 W 的整数倍:W=60,E=90。 使用 30s 块:t=90 读取 (30,60]+(60,90],t=180 读取 (120,150]+(150,180]。后端本地完整窗口也支持这种不重叠且有间隙的求值网格。

  6. 求值起点偏移:W=60,E=20,在 t=65、85、105 求值。 小块起点相位为 5s,t=65 读取 (5,25]+(25,45]+(45,65]。如果仍从 0s 分块,就无法拼出 (5,65]。完整窗口则由求值终点减去 W,得到正确的窗口起点相位。

  7. 临时查询不对齐的时刻。 已有从 0s 开始的 20s 块,用户在 t=67 查询最近 60s,需要 (7,67]。摘要读取报告 capability miss,由精确路径回答,不取整、不拼接错误边界。

  8. 多个查询共享。 A 为 W=60/E=20,B 为 W=90/E=30。分别维护时,各读 3 块;共享一套 10s 块后,A 读 6 块、B 读 9 块。系统重新计算创建、维护、保留、回收和读取成本,仅在总成本更低时共享。来源、计算状态、准确率、运行策略和相位也必须兼容;第三个不兼容或代价过高的查询不会阻止其余查询共享。

  9. 想用比 E 更大的块:W=60,E=20,提议 30s 块。 t=60 可以用 (0,30]+(30,60] 回答,但 t=80 的 (20,80] 无法拼出,因此拒绝这个普通 pane 布局。细块与粗块结合的 hierarchical rollup 需要额外运行时实现,本 PR 不把它冒充为可执行方案。

  10. 后端或计算方式不支持某种布局。 生成候选前检查目标能力:Collector 不接受小于完整窗口的 pane,也不接受当前未支持的稀疏完整窗口方案;派生维护及其源状态使用运行时要求的完整 cohort。本地不支持的叶子或不对齐读取走精确回退;严格 Collector 计划无可执行方案时明确拒绝。

  11. 提供了实测成本。 如果 20s pane 的有效实测成本为 8,完整窗口成本为 12,选择较便宜的可执行方案。但把 20s pane 改成共享 10s pane 后,不能继续声称成本为 8。当前实现保护已选择的实测方案,自动共享只重算公式生成的方案;成本证据不能绕过布局、时效或身份校验。

Input migration

  • Snapshot: replace implementation.window_implementation_id and implementation.implementation_cost with implementation.window_cost_model.implementation_id and .cost. Remove implementation.window_candidates.
  • HTTP query: replace window_implementations with window_cost_model; declare evaluation_phase_ms explicitly.
  • Optional model quotes contain workload-scoped measured layout quotes. Production candidate generation is always automatic.
  • Update checked-in snapshots, calibration tools and process-test clients together.

Verification

  • The W=60/E=45 regression failed against the original implementation before the fix.
  • Workspace validation completed all unit suites: 107 asap_types tests, 746 control-plane library tests, 31 control-plane HTTP/binary tests, and 1,201 data-plane library tests passed. Process fixtures exposed by the stricter contract were corrected and rerun below; the initial workspace command itself was not a clean pass.
  • cargo +1.98.0 test --workspace --test asapquery_compatibility_process_e2e -- --test-threads=4 — 12 passed, 1 existing ignored Collector-schema test.
  • cargo +1.98.0 test --workspace --test e2e_controller_plans_and_backend_serves -- --test-threads=4 — 12 passed.
  • Remaining compiled data-plane integration targets were run directly with --test-threads=4: backend process, ClickHouse differential/Q05, component process, edge runtime, monitor gRPC/process, Remote Write wire, and PromQL differential tests passed. All four sketch process-oracle tests also passed in the workspace run.
  • cargo +1.98.0 clippy --workspace --all-targets -- -D warnings and cargo +1.98.0 fmt --all -- --check — passed after the final fixture changes.
  • python3 -m unittest discover -s tools/o11y-execution -p 'test_*.py' — 50 passed.
  • Process runs used ASAP_TEST_TIMEOUT_SCALE=4 and ASAP_E2E_CONTROL_PLANE_BIN pointing to the built control-plane binary. No hosted-CI completion is implied by these local results.

Fixture migrations preserve each test's purpose: persistence/ERP tests declare complete 5s populations, transport tests declare their actual 1s states, and the warm counter matrix supplies both queried windows. ERP assertions check the full sample count and cardinality. Missing data still uses exact fallback.

New/updated focused test purposes:

  • derived_window_candidates_cover_dividing_nondividing_and_sparse_cadences: legal panes and unchanged E for all five cadence relations.
  • scheduled_window_layouts_preserve_cadence_phase_and_readout: compiled pane/full-window bindings match scheduled ranges and reject off-grid boundaries.
  • raw_leaf_keeps_its_cadence_beside_a_same_window_derived_cohort: state-specific cohort constraints do not rewrite a neighboring raw leaf.
  • different_cadences_share_common_panes_only_when_cheaper: select sharing under expensive maintenance, retain independent panes under expensive reads.
  • sharing_keeps_profitable_subsets_with_other_phases_or_finer_cadences: a third incompatible/expensive consumer cannot disable useful pair reuse.
  • fractional_cadence_uses_native_fallback_without_truncation: unsupported time precision remains exact rather than changing demand.
  • measured_window_quote_preserves_shape_and_price: preserve the matching measured quote while keeping other feasible alternatives.
  • collector_generation_excludes_partial_panes_and_sparse_full_windows: enforce target capabilities before selection.
  • quotes_cannot_bypass_layout_or_runtime_constraints: reject oversized flat panes and unimplemented hierarchical rollups.
  • conflicting_quote_ids_and_duplicate_shapes_are_rejected: reject ambiguous evidence instead of silently dropping offers.
  • serialized_generated_quote_loses_compiler_provenance: incoming evidence cannot authorize automatic repricing.
  • compiled_window_schedules_execute_exact_ranges: runtime bucket assignment and installed readouts match raw integer samples for cadence/phase/layout combinations; off-grid reads miss.
  • full_window_sketch_read_excludes_neighboring_windows: overlapping KLL states are not merged into the requested complete window.
  • repeated_dashboard_executes_multiple_selected_panes: the production process derives and serves 5s panes for 10s lookbacks without fabricated layout prices.

Independent static review covered phase handling, full-window reads, quote identities and pane sharing; its duplicate-ID and incompatible-third-consumer findings were addressed. No performance measurements or screenshots are claimed.

Architectural decisions

  • A single WindowCostModel replaces alternate candidate injection APIs; measured evidence supplements the same feasibility generator.
  • One optional full-window slide on a binding is necessary to distinguish stored extent from the query grid and prevent overlap double-counting.
  • Candidate feasibility is scoped to the selected state, so a derived cohort cannot impose its cadence on an unrelated raw leaf with the same W.
  • Keep hierarchical rollups unsupported until an executor exists; do not change query cadence to make an unsupported pane legal.

Limitations and follow-up

Runtime layouts retain whole-second precision: fractional cadences are not truncated and use native exact fallback. Hierarchical rollups remain unsupported. Missing/open panes still fall back instead of being assumed zero. Shared-pane selection is a deterministic cost comparison with greedy profitable subsets, not a claim of globally optimal workload partitioning.

Human review — do not complete with an agent

  • The MVP boundary is correct.
  • New conceptual layers or public interfaces are necessary.
  • The before/after description matches the intended product behavior.
  • Human reviewer:
  • Decision and rationale:

@zzylol
zzylol merged commit d07a9b3 into main Sep 13, 2026
1 check passed
zzylol added a commit that referenced this pull request Sep 13, 2026
Rebuilds #700 on top of main (ca944a6) as the current-series feature alone.
The window/full-window half of the original PR is dropped: #712 landed an
equivalent, and main's version is kept wherever the two overlapped.

Multiple current-value quantiles and TopK limits over the same metric now
share one bounded maintained population instead of each planning its own
state. Lowering follows the Planner-selected IR rather than reconstructing
intent from query text, so a catalog rename cannot change the physical plan.

- `SeriesPopulation` / `SeriesReadout` contract and `LogicalOperator::CurrentSeries`
- Backend `CurrentSeriesStore`: bounded members, lookback expiry, shared
  ordered state feeding quantile / TopK / sum / count / avg readouts
- Planner revision bumped to b8b5d705 for typed `ExactKind::Min`, with the
  Min readout wired through catalog resolution and the exact-agg merge path
- `asap_current_series_populations` / `_cache_builds_total` on /metrics

Review fixes carried in (from a review of the original PR):
- Budget overflow no longer latches permanently. It records when the budget
  blew and re-arms once the lookback window has moved past it, instead of
  stranding the plan on Prometheus until an operator forces a replan.
- Quantile readout guards an empty value set instead of underflowing
  `values.len() - 1` while holding the lock the remote-write path waits on.
- `CompileError::Query` reports `query_id`, not the query text.
- Full-window sketch reads skip a layout-provably-empty sid instead of
  discarding the sids already accumulated and dropping to the exact path.

Deliberately not included: the original PR also reversed `count()` over HLL
from register-merge to declined. That is a behavioral change unrelated to
current-series and belongs in its own PR with its own argument; main's
semantics are kept here.

Verified on this tree: cargo fmt --check, clippy --workspace --all-targets
-D warnings, and cargo test --workspace -- --test-threads=1
(2139 passed, 0 failed, 4 ignored).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 14, 2026
Rebased onto main (84109cb). Carries only the current-series feature; the
window/full-window half of the original #700 was dropped when #712 landed an
equivalent, and main's version is kept wherever the two overlapped.

Multiple current-value quantiles and TopK limits over the same metric share one
bounded maintained population instead of each planning its own state. Lowering
follows the Planner-selected IR rather than reconstructing intent from query
text, so a catalog rename cannot change the physical plan.

- `SeriesPopulation` / `SeriesReadout` contract and the `CurrentSeries` residual
  operator
- Backend `CurrentSeriesStore`: bounded membership, lookback expiry, shared
  ordered state feeding quantile / TopK / sum / count / avg readouts
- Planner revision bumped to b8b5d705 for typed `ExactKind::Min`, wired through
  catalog resolution, the exact-agg merge path, and the backend wire sub-type
- `asap_current_series_populations` / `_cache_builds_total` on /metrics

Adopts main's current naming throughout (`PhysicalCompilationRequest`,
`PhysicalDeploymentContext`, `ResidualQueryOperator`, `query_plan::residual`,
`selected_plan_root`, `summary_store`, `allow_mixed_summary_and_exact_execution`)
rather than the deprecated compatibility aliases. Serde wire names and JSON
fixture keys are unchanged from main.

Review fixes carried in:
- Budget overflow no longer latches permanently. It records when the budget blew
  and re-arms once the lookback window has moved past it, instead of stranding
  the plan on Prometheus until an operator forces a replan.
- Quantile readout guards an empty value set instead of underflowing
  `values.len() - 1` while holding the lock the remote-write path waits on.
- Full-window sketch reads skip a layout-provably-empty sid instead of
  discarding the sids already accumulated and dropping to the exact path.

Verified on this tree: cargo fmt --check, clippy --workspace --all-targets
-D warnings, and cargo test --workspace -- --test-threads=1
(1750 passed, 0 failed, 14 ignored).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 14, 2026
…700)

Rebased onto main (84109cb). Carries only the current-series feature; the
window/full-window half of the original #700 was dropped when #712 landed an
equivalent, and main's version is kept wherever the two overlapped.

Multiple current-value quantiles and TopK limits over the same metric share one
bounded maintained population instead of each planning its own state. Lowering
follows the Planner-selected IR rather than reconstructing intent from query
text, so a catalog rename cannot change the physical plan.

- `SeriesPopulation` / `SeriesReadout` contract and the `CurrentSeries` residual
  operator
- Backend `CurrentSeriesStore`: bounded membership, lookback expiry, shared
  ordered state feeding quantile / TopK / sum / count / avg readouts
- Planner revision bumped to b8b5d705 for typed `ExactKind::Min`, wired through
  catalog resolution, the exact-agg merge path, and the backend wire sub-type
- `asap_current_series_populations` / `_cache_builds_total` on /metrics

Adopts main's current naming throughout (`PhysicalCompilationRequest`,
`PhysicalDeploymentContext`, `ResidualQueryOperator`, `query_plan::residual`,
`selected_plan_root`, `summary_store`, `allow_mixed_summary_and_exact_execution`)
rather than the deprecated compatibility aliases. Serde wire names and JSON
fixture keys are unchanged from main.

Review fixes carried in:
- Budget overflow no longer latches permanently. It records when the budget blew
  and re-arms once the lookback window has moved past it, instead of stranding
  the plan on Prometheus until an operator forces a replan.
- Quantile readout guards an empty value set instead of underflowing
  `values.len() - 1` while holding the lock the remote-write path waits on.
- Full-window sketch reads skip a layout-provably-empty sid instead of
  discarding the sids already accumulated and dropping to the exact path.

Verified on this tree: cargo fmt --check, clippy --workspace --all-targets
-D warnings, and cargo test --workspace -- --test-threads=1
(1750 passed, 0 failed, 14 ignored).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Window implementation candidates: sources, rewrites, and simplification questions

1 participant