Skip to content

Unify catalog generation and typed data identity - #575

Merged
zzylol merged 1 commit into
mainfrom
refactor/unify-sds-catalog-contract
Sep 10, 2026
Merged

zzylol merged 1 commit into
mainfrom
refactor/unify-sds-catalog-contract

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why

The SDS contract represented one catalog snapshot with both CatalogGeneration and SummaryCatalogReference, using different digest field names. DataDescriptor also identified every population by a metric-name string, so a time-series metric and a relational table could collide and SQL value-column changes did not change descriptor identity.

Before this PR

Catalog consumers converted between two equivalent generation structs. DataDescriptor(metric_name, filter, group_by, observation_semantics) lost whether the source was a metric or table and which value was summarized.

After this PR

CatalogGeneration is the single immutable snapshot identity used by the catalog, plans, inventories, and instances. Its canonical digest field is snapshot_sha256; the old snapshot_digest spelling is accepted only as a deserialization alias.

DataDescriptor now carries language-neutral typed identities:

  • DataSourceIdentity::TimeSeries { metric } or Table { table_ref }
  • ValueProjectionIdentity::SampleValue or Column { name }

Both are included in the content-addressed descriptor ID. The catalog schema advances to version 2 because its canonical descriptor shape and identity changed. SummaryStore indexes only time-series descriptors by metric while retaining relational descriptors without forcing them into that index.

Example

A metric named events, table events.value, and table events.cost now produce three distinct DataDescriptorId values. They can share an operator descriptor while remaining distinct summarized populations.

Verification

  • cargo check --workspace --all-targets
  • cargo test -p asap_types sds::tests (14 passed)
  • cargo test -p control_plane --lib physical::compiler::tests (67 passed)
  • cargo test -p control_plane --lib summary_reconcile (3 passed)
  • cargo test -p data_plane --lib storage_engines::sketch_db::sds (5 passed)
  • git diff --check

Depends on #572.

ClickHouse migration surface: use DataSourceIdentity::Table with an explicit ValueProjectionIdentity::Column; publish and persist the same CatalogGeneration returned by SummaryCatalog::reference().

@zzylol
zzylol force-pushed the refactor/typed-summary-definition-refs branch from 9c69b67 to 316660a Compare September 10, 2026 15:57
@zzylol
zzylol force-pushed the refactor/unify-sds-catalog-contract branch from a2ba424 to 445b60b Compare September 10, 2026 16:00
@zzylol
zzylol force-pushed the refactor/unify-sds-catalog-contract branch from 445b60b to 0163b27 Compare September 10, 2026 16:08
@zzylol
zzylol changed the base branch from refactor/typed-summary-definition-refs to main September 10, 2026 16:08
@zzylol
zzylol merged commit 98a6522 into main Sep 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant