Skip to content

[feat] Collapse entity-per-table warehouses in schema_index (composite digest) #760

Description

@sahrizvi

Problem

When a database has many tables that share the same column structure — one ticker per table, one tenant per table, time-partitioned tables, per-region tables — the existing `schema_index` emits per-table schemas. N copies of the same schema fill the agent's context with redundancy. With `schema_index`'s truncation cap, the agent sometimes can't see all tables at all.

Concrete example we hit on a benchmark workload:

  • stockmarket dataset has 2754 ticker tables, every one with identical columns (`Date`, `Open`, `High`, `Low`, `Close`, `Adj Close`, `CloseUSD`).
  • `schema_index` truncates to 200 → agent only sees ticker symbols A–Bxxx and has to discover the rest at runtime via SQL probes.
  • Score on this dataset: 1/5 vs. yelp (per-table-is-different schemas) at 6/7. Direct attribution to redundant per-table schemas filling context.

Proposed mechanism

Heuristic in `schema_index`'s scan logic, after enumerating tables for a connection:

  1. Group tables by column-structure fingerprint — sorted tuple of `(column_name, column_type)`.
  2. Detect entity-per-table pattern — if any single fingerprint group covers ≥50% of tables AND has ≥20 tables, mark that group as the "entity" group.
  3. Emit a different digest format for the entity group:

```
schema: .
pattern: entity-per-table
table_count: 2754
composite_columns: [{name: "Date", type: "VARCHAR"}, {name: "Open", type: "DOUBLE"}, …]
sample_table: AAPL
table_names: [AAPL, AAA, AAL, AAN, …] ← full list, searchable
```

  1. Tables NOT in the entity group still get per-table schemas. The two formats coexist.
  2. `schema_search` should still surface table names from entity groups in lookups.

What it enables

  • ~10× context-saving on entity-per-table warehouses.
  • Agent sees ALL tables (full list of names) instead of a truncated subset.
  • Per-table queries still work — composite schema describes the columns; the agent uses table names directly.
  • Backwards compatible: tables with unique structures still get per-table schemas as today.

Beyond benchmark / general use cases

This is a real production pattern, common in:

  • Multi-tenant SaaS (Stripe-style "one table per customer for isolation")
  • Time-partitioned analytics (`events_2024_01_01`, `events_2024_01_02`, …)
  • Per-region tables (`orders_us`, `orders_eu`, `orders_apac`)
  • Per-environment (dev/staging/prod schemas with same tables)
  • Per-source ingestion (`events_segment`, `events_amplitude`, `events_mixpanel`)
  • Sharded analytics (per-customer, per-tenant)

Customers using altimate-code on these warehouses today are losing context capacity to redundant per-table schemas they could collapse to one composite + a name list.

Implementation pointers

  • Heuristic detector: new helper in `packages/opencode/src/altimate/native/schema/cache.ts`
  • Tool wrapper: `packages/opencode/src/altimate/tools/schema-index.ts` (or wherever it lives)
  • Output format reference: `packages/opencode/src/altimate/tools/schema-inspect.ts`
  • Estimated scope: ~150-200 LOC + tests

Tests to add

  • Detector unit tests:
    • 100 same-shape tables → marked entity
    • 5 tables of various shapes → not marked
    • 50 same-shape + 50 different-shape → not marked (under 50% threshold)
  • Integration test against in-memory SQLite fixture with 30 identical-schema tables
  • Backwards-compat test: existing single-table warehouse output unchanged

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpriority:mediumMedium prioritysql-engineSQL analysis, lineage, translationwarehouseWarehouse connectivity and drivers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions