Skip to content

Use collision-free interned group keys in DAG execution - #585

Merged
zzylol merged 1 commit into
mainfrom
perf/canonical-group-key
Sep 10, 2026
Merged

zzylol merged 1 commit into
mainfrom
perf/canonical-group-key

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Prometheus and OTLP ingress previously flattened GROUP BY values with ;. Distinct groups such as (x="a;b", y="c") and (x="a", y="b;c") therefore shared worker state, and downstream code had to reconstruct labels by splitting an ambiguous string.

This change introduces one typed GroupKey for router-to-worker execution. Its versioned, length-prefixed encoding preserves label names, values, and projection order. A bounded interner lets shared DAG consumers reuse the same immutable key and avoids rebuilding name/value allocations for repeated series. Remote-write series population keys now use the same representation, while persisted series IDs and SketchDB keys remain unchanged and require no migration.

The worker emits label values and population labels directly from the typed key. Tests cover delimiter collisions, different label names, projection order, missing values, interner reuse, and ingress routing. A Criterion case measures 100,000 high-cardinality, delimiter-rich two-label projections.

Validation:

  • cargo test -p data_plane --no-run
  • cargo test -p data_plane group_key --lib (4 passed)
  • cargo bench -p data_plane --bench sketch_db -- group_key_projection --sample-size 10 --measurement-time 1 --warm-up-time 1
    • high-cardinality projection: median 108.48 ms / 100,000 keys (921.85k keys/s)
    • repeated shared projection: median 109.51 ms / 100,000 keys (913.13k keys/s)
  • Full library run: 1,067 passed; three pre-existing stack/environment failures remain (pane-origin HTTP assertion, missing-handle HTTP status, and a fixed-port AddrInUse).

@zzylol
zzylol changed the base branch from feat/runtime-window-layouts to main September 10, 2026 17:45
@zzylol
zzylol force-pushed the perf/canonical-group-key branch from 151ea66 to a6eb235 Compare September 10, 2026 17:45
@zzylol
zzylol merged commit 55ac42b into main Sep 10, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant