docs(control_plane): design doc for BackendPlan wire format - #433
Merged
Merged
Conversation
Clean design for the typed control-plane -> data-plane wire contract, written against this deployment's current target architecture (control_plane depends on ASAPController's real crates; SummaryExecutor is the live serving path) rather than as a migration narrative. Key design points: - Materialization's payload is (SummaryKind, SummaryParams) directly -- no separate ExactAggregate variant, since asap_sketch::SummaryKind already unifies exact accumulators and approximate sketches (ASAPController#170). - RoutingIndex has two read modes at different match precisions: family-level Capability match for whole-query/miss-detection routing, exact (SummaryKind, SummaryParams) match for SummaryExecutor's find_candidates. - Once RoutingIndex exists, serving-time L4 lowering reads the plan control_plane already decided directly, instead of reconstructing it from registered sid metadata the way ObservedFamilyCostModel does today -- CostModel becomes planning-time-only, never re-invoked per query. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Jul 29, 2026
This was referenced Jul 29, 2026
zzylol
added a commit
that referenced
this pull request
Jul 30, 2026
Implements the typed control-plane -> data-plane wire contract from control_plane/docs/design-backend-plan-wire-format.md (#433) §3: the BackendPlan/Materialization/RoutingEntry Rust types, a proto schema (proto/backend_plan.proto), and round-trip conversions between them. Reuses this deployment's canonical vocabulary directly rather than re-encoding it: asap_sketch::{SummaryKind, SummaryParams} for the materialization payload (one wire shape for both exact and approximate families, matching ASAPController#170's unification -- no separate ExactAggregate variant), asap_ir/control_plane::intent_algebra's Source/ColumnRef/WindowKind for the L3 IR fragments, control_plane::sketch_algebra::capability::Capability for routing, and asap_types::{PolicyFingerprint, MonitorSpec} (already shared with data_plane for this exact cross-crate reason). StorageBackend/RetentionPolicy are new, deployment-local types -- control_plane cannot depend on data_plane's existing StorageBackend (the dependency runs the other way), so this mirrors it rather than reusing it directly, same constraint asap_types::MonitorSpec's own doc already documents for that type. Scope: types + round-trip tests only, no behavior change. Not wired into emit/, main.rs's planning path, or any data_plane consumer -- RoutingIndex (design doc §4), which is what actually reads this at query time, is separate follow-on work. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Jul 30, 2026
Implements the Tier-2 half of control_plane/docs/design-backend-plan-wire-format.md (#433) §4's RoutingIndex, sourced from PolicyRegistry (StreamingConfig's content-addressed view -- genuinely "what control_plane planned" today, not reconstructed from ingest side effects) rather than the not-yet-built BackendPlan wire format. Tier 1 (exact PolicyFingerprint lookup) is already PolicyRegistry::get, unchanged. find_matching_policies (control_plane::asap_tier_analysis) used to scan every policy in the registry for every candidate, checking each one's metric name first -- paying for every OTHER metric's policies on every lookup. RoutingIndex buckets by metric once at construction, so a lookup only touches policies for the relevant metric. Same match predicate, same observable behavior (verified: all ~18 existing find_matching_policies tests pass unchanged, only their PolicyRegistry construction is now wrapped in RoutingIndex::build). engine.rs's two candidate-resolution loops (range-query, instant-query) now build a RoutingIndex once per query snapshot instead of a raw PolicyRegistry, matching the design doc's stated caller pattern. Building RoutingIndex once per StreamingConfig hot-reload swap instead of once per query (StreamingConfig::policy_registry's own doc comment already flags this as a further, larger optimization "if it shows up in profiles") is not done here -- would require threading a cached derived value through HotReloadStreamingConfig's swap path, a separate, larger change. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supersedes #389. That PR's design doc was written as a migration narrative (motivation citing the now-retired
analyzer-parity-matrix.md, phase sequencing, "what's broken today") from a point in time before this repo adopted ASAPController's real crates directly, beforeSummaryExecutorbecame the live serving path (#427), and beforesketch_reducer.rs/shadow_compare.rswere retired (#430). Rather than patch that narrative to catch up, this is a clean rewrite of the design itself, written against where the codebase actually stands today, with no reference to what's implemented vs. not.Key differences from #389's version, driven by things learned since:
Materialization's payload is just(SummaryKind, SummaryParams)— no separateSketch/ExactAggregatesplit. ASAPController#170 already unified exact accumulators and approximate sketches into oneSummaryKindvocabulary (is_exact()tells them apart); the wire format shouldn't reintroduce a distinction the canonical type has already closed.RoutingIndexis framed around two genuinely different match precisions, tied to two real callers: family-levelCapabilitymatch for whole-query/miss-detection routing, and exact(SummaryKind, SummaryParams)match forSummaryExecutor::find_candidates(which must return every match formerge_statesto fold, not one ranked winner).data_plane'sObservedFamilyCostModel(landed in retire(data_plane): remove shadow_compare.rs and sketch_reducer.rs #430) reconstructs a metric's planned(SummaryKind, SummaryParams)by inspecting registered sid metadata at query time — a working stopgap for the absence of exactly this lookup. OnceRoutingIndexexists, serving-time lowering reads the plan directly instead of reconstructing it, andCostModelbecomes planning-time-only.SummaryKindto name yet — tracked separately at Exact TopK (accuracy: Exact) has no warm-tier materialization -- always falls to archive #432 —RoutingIndexperformance).Test plan
🤖 Generated with Claude Code