Unify catalog generation and typed data identity - #575
Merged
Merged
Conversation
zzylol
force-pushed
the
refactor/typed-summary-definition-refs
branch
from
September 10, 2026 15:57
9c69b67 to
316660a
Compare
zzylol
force-pushed
the
refactor/unify-sds-catalog-contract
branch
from
September 10, 2026 16:00
a2ba424 to
445b60b
Compare
zzylol
force-pushed
the
refactor/unify-sds-catalog-contract
branch
from
September 10, 2026 16:08
445b60b to
0163b27
Compare
zzylol
changed the base branch from
refactor/typed-summary-definition-refs
to
main
September 10, 2026 16:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The SDS contract represented one catalog snapshot with both
CatalogGenerationandSummaryCatalogReference, using different digest field names.DataDescriptoralso identified every population by a metric-name string, so a time-series metric and a relational table could collide and SQL value-column changes did not change descriptor identity.Before this PR
Catalog consumers converted between two equivalent generation structs.
DataDescriptor(metric_name, filter, group_by, observation_semantics)lost whether the source was a metric or table and which value was summarized.After this PR
CatalogGenerationis the single immutable snapshot identity used by the catalog, plans, inventories, and instances. Its canonical digest field issnapshot_sha256; the oldsnapshot_digestspelling is accepted only as a deserialization alias.DataDescriptornow carries language-neutral typed identities:DataSourceIdentity::TimeSeries { metric }orTable { table_ref }ValueProjectionIdentity::SampleValueorColumn { name }Both are included in the content-addressed descriptor ID. The catalog schema advances to version 2 because its canonical descriptor shape and identity changed. SummaryStore indexes only time-series descriptors by metric while retaining relational descriptors without forcing them into that index.
Example
A metric named
events, tableevents.value, and tableevents.costnow produce three distinctDataDescriptorIdvalues. They can share an operator descriptor while remaining distinct summarized populations.Verification
cargo check --workspace --all-targetscargo test -p asap_types sds::tests(14 passed)cargo test -p control_plane --lib physical::compiler::tests(67 passed)cargo test -p control_plane --lib summary_reconcile(3 passed)cargo test -p data_plane --lib storage_engines::sketch_db::sds(5 passed)git diff --checkDepends on #572.
ClickHouse migration surface: use
DataSourceIdentity::Tablewith an explicitValueProjectionIdentity::Column; publish and persist the sameCatalogGenerationreturned bySummaryCatalog::reference().