Conversation
Phase 1 of the BackendPlan end-to-end wiring (control_plane emit -> data_plane serving). Pure, additive: a new from_stage_config() function builds a typed BackendPlan from the same BackendStageConfig input emit_backend_streaming_config_json already turns into the legacy JSON wire format. No wire/push change yet -- that's Phase 2/3. Fingerprint parity with the legacy StreamingConfig path is the critical invariant: Materialization.fingerprint is derived by round-tripping through build_backend_aggregation_json + AggregationConfig::from_yaml_data (the exact JSON shape and parser data_plane's real POST /api/v1/streaming-config handler uses), not a re-derived field mapping, so the two identity spaces can never drift apart. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Combines the wire push and endpoint pieces of the BackendPlan cutover (control_plane/docs/design-backend-plan-wire-format.md): control_plane side: - BackendClient::post_backend_plan_typed posts encoded BackendPlan bytes to POST /api/v1/backend-plan (sibling endpoint derived from the configured streaming-config URL, same convention as storage_routing). - emit::backend_push::push_cumulative_entries now also builds a BackendPlan from the same cumulative_be snapshot used for the legacy streaming-config/storage-routing documents (via backend_plan::from_stage_config from the prior PR) and fires a best-effort, single-attempt push alongside the existing coupled push. This is the real plan-time call site (Phase 3) -- the only place that builds BackendStageConfig for a live push. - The plan push's outcome never affects PushOutcome, which existing callers key real (legacy-path) behavior on -- nothing consumes BackendPlan yet, so its failure must stay invisible to them. No in-function retry loop (unlike the coupled documents): the next replan cycle is the retry backstop, matching push_or_log's existing fire-and-forget contract for the legacy YAML path. data_plane side: - HotReloadBackendPlan (storage_engines/types/hot_reload_config.rs): same ArcSwap-snapshot/swap shape as HotReloadStreamingConfig, applied to control_plane::backend_plan::BackendPlan. - GET/POST /api/v1/backend-plan handlers, registered alongside (not replacing) /api/v1/streaming-config. POST decodes protobuf bytes and atomically swaps; does not touch the sid catalog or SketchStore reconciliation. - main.rs installs an empty handle at startup so the endpoints don't 503 before the control plane's first push lands, mirroring the existing backend-storage-routing bootstrap pattern. Nothing consumes the installed BackendPlan yet -- that's Phase 4 (serving-time cutover), which also retires ObservedFamilyCostModel's SketchStore-reconstruction to a fallback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace forward references to "Phase 4"/"nothing consumes this yet" (stale now that the serving-time consumer exists) with descriptions of the actual current architecture: BackendPlan is read by ASAPQueryEngine's serving-time lookup, sits alongside (not in place of) the legacy streaming-config path, and a dropped push just falls back to SketchStore reconstruction until the next replan cycle. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ore reconstruction (Phase 4) Completes the BackendPlan wire-format cutover (control_plane/docs/design-backend-plan-wire-format.md). Serving time now reads planning's real family/params decision directly off an installed BackendPlan when one covers the query's metric, instead of always reconstructing it from SketchStore metadata. - l4_lowering.rs: new observed_family_for_metric_from_plan() reads Materialization.kind/.params directly (no AggregationConfig reconstruction needed -- they already are the (SummaryKind, SummaryParams) pair this needs). lower_promql_to_l4node() now takes an Option<&BackendPlan> and tries the plan first, falling back to the existing SketchStore-reconstruction path (observed_family_for_metric / ObservedFamilyCostModel) when no plan is installed or it doesn't cover the metric. Three new unit tests prove: (1) SketchStore reconstruction still works with no plan, (2) a plan materialization WINS over a disagreeing SketchStore registration for the same metric -- the load-bearing proof for this cutover, (3) a plan that doesn't cover the metric falls through to SketchStore reconstruction rather than silently failing to observe anything. - l4_readout.rs / live_serve.rs: thread the same Option<&BackendPlan> parameter through unchanged otherwise. - engine.rs: ASAPQueryEngine gains a HotReloadBackendPlan handle (with_hot_reload_backend_plan) and a backend_plan_snapshot() helper; both live-serving call sites (range + instant query) pass the current snapshot through. - main.rs: the HotReloadBackendPlan handle is now shared between the query engine and the HTTP server (previously only the latter), so a POST /api/v1/backend-plan is observable by the next query, same sharing contract as hot_reload_config. - cost_model.rs: ObservedFamilyCostModel's doc updated to reflect its new fallback status -- not dead code, just no longer the first thing consulted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ehavior Drop "this cutover"/"the load-bearing proof for this cutover"-style narration in favor of stating what the code actually does now: prefer BackendPlan's materializations, fall back to SketchStore reconstruction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
zzylol
force-pushed
the
feat/backend-plan-serving-cutover
branch
from
July 30, 2026 04:56
62ba731 to
f84452b
Compare
Merged
4 tasks
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #438 (Phase 2+3) and #437 (Phase 1). Final phase of the BackendPlan cutover (
control_plane/docs/design-backend-plan-wire-format.md): serving time (l4_lowering.rs) now reads planning's real family/params decision directly off an installedBackendPlanwhen it covers the query's metric, instead of always reconstructing it fromSketchStoremetadata viaObservedFamilyCostModel.observed_family_for_metric_from_plan— readsMaterialization.kind/.paramsdirectly off the plan; noAggregationConfigreconstruction needed (they already are the pairObservedFamilyCostModelwants).lower_promql_to_l4nodetries the plan first, falls back to the existingSketchStore-reconstruction path when no plan is installed or it doesn't cover the metric — additive, not a replacement.ASAPQueryEnginegets aHotReloadBackendPlanhandle, shared with the HTTP server inmain.rs(same sharing contract ashot_reload_config), so aPOST /api/v1/backend-planis observable by the next query.ObservedFamilyCostModel's doc updated: fallback status, not dead code.Test plan
l4_lowering.rs::tests::backend_plan_cutover— the load-bearing one (a_plan_materialization_wins_over_sketchstore_reconstruction) registers a Kll sid inSketchStorebut installs a plan declaring DDSketch for the same metric, and asserts the boundL4Nodeuses DDSketch (inspected via theSummaryEstimate{SummaryAgg{summary,..}}tree shape). Two companion tests confirm the no-plan and plan-doesn't-cover-this-metric fallback paths are unchanged.cargo test --release --lib -p control_plane— 732/732.cargo test --release --lib -p data_plane— 915/915 (2 pre-existing ignores).cargo build --release --workspace— clean.e2e_controller_plans_and_backend_serves— same 2 pre-existing*_topkfailures asmain(unrelated, confirmed earlier in this stack).