backend-plan: wire push + data_plane consume endpoint (Phase 2+3) - #440
Merged
Merged
Conversation
Phase 1 of the BackendPlan end-to-end wiring (control_plane emit -> data_plane serving). Pure, additive: a new from_stage_config() function builds a typed BackendPlan from the same BackendStageConfig input emit_backend_streaming_config_json already turns into the legacy JSON wire format. No wire/push change yet -- that's Phase 2/3. Fingerprint parity with the legacy StreamingConfig path is the critical invariant: Materialization.fingerprint is derived by round-tripping through build_backend_aggregation_json + AggregationConfig::from_yaml_data (the exact JSON shape and parser data_plane's real POST /api/v1/streaming-config handler uses), not a re-derived field mapping, so the two identity spaces can never drift apart. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Combines the wire push and endpoint pieces of the BackendPlan cutover (control_plane/docs/design-backend-plan-wire-format.md): control_plane side: - BackendClient::post_backend_plan_typed posts encoded BackendPlan bytes to POST /api/v1/backend-plan (sibling endpoint derived from the configured streaming-config URL, same convention as storage_routing). - emit::backend_push::push_cumulative_entries now also builds a BackendPlan from the same cumulative_be snapshot used for the legacy streaming-config/storage-routing documents (via backend_plan::from_stage_config from the prior PR) and fires a best-effort, single-attempt push alongside the existing coupled push. This is the real plan-time call site (Phase 3) -- the only place that builds BackendStageConfig for a live push. - The plan push's outcome never affects PushOutcome, which existing callers key real (legacy-path) behavior on -- nothing consumes BackendPlan yet, so its failure must stay invisible to them. No in-function retry loop (unlike the coupled documents): the next replan cycle is the retry backstop, matching push_or_log's existing fire-and-forget contract for the legacy YAML path. data_plane side: - HotReloadBackendPlan (storage_engines/types/hot_reload_config.rs): same ArcSwap-snapshot/swap shape as HotReloadStreamingConfig, applied to control_plane::backend_plan::BackendPlan. - GET/POST /api/v1/backend-plan handlers, registered alongside (not replacing) /api/v1/streaming-config. POST decodes protobuf bytes and atomically swaps; does not touch the sid catalog or SketchStore reconciliation. - main.rs installs an empty handle at startup so the endpoints don't 503 before the control plane's first push lands, mirroring the existing backend-storage-routing bootstrap pattern. Nothing consumes the installed BackendPlan yet -- that's Phase 4 (serving-time cutover), which also retires ObservedFamilyCostModel's SketchStore-reconstruction to a fallback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace forward references to "Phase 4"/"nothing consumes this yet" (stale now that the serving-time consumer exists) with descriptions of the actual current architecture: BackendPlan is read by ASAPQueryEngine's serving-time lookup, sits alongside (not in place of) the legacy streaming-config path, and a dropped push just falls back to SketchStore reconstruction until the next replan cycle. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Jul 30, 2026
Merged
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recreated after #438 was auto-closed when its stacked base branch (#437) was deleted on merge — same content, now based directly on
main(which already has #437'sBackendPlan::from_stage_config).Summary
Phases 2+3 of the BackendPlan cutover (
control_plane/docs/design-backend-plan-wire-format.md): the control plane now actually pushes an encodedBackendPlanto a newPOST /api/v1/backend-planendpoint, alongside (not instead of) the legacystreaming-config/storage_routingdocuments.BackendClient::post_backend_plan_typed— new typed POST method, same transient/permanent classification as the existing typed methods, hitting a sibling endpoint derived from the configured streaming-config URL.emit::backend_push::push_cumulative_entries(the one real call site that buildsBackendStageConfigfor a live push) now also builds aBackendPlanviabackend_plan::from_stage_configfrom the samecumulative_besnapshot, and fires a best-effort, single-attempt push. Its outcome never affectsPushOutcome— a dropped push just falls back toSketchStorereconstruction at serving time until the next replan cycle.data_plane:HotReloadBackendPlan(sameArcSwapshape asHotReloadStreamingConfig) +GET/POST /api/v1/backend-planhandlers, wired intomain.rswith an empty bootstrap handle (mirrors the existing storage-routing bootstrap pattern).Test plan
cargo test --release -p control_plane --lib backend_client:: backend_push::— including a dedicated test proving the plan push fires alongside the coupled push, and a test proving a plan-push failure does NOT flipPushOutcome.cargo test --release -p data_plane --lib— 915/915 pass (2 pre-existing ignores), including newHotReloadBackendPlanround-trip + 503 + bad-bytes tests.cargo build --release --workspace— clean.