Skip to content

DeltaSetAggregator must be restricted to tumbling windows #588

Description

@milindsrivastava1997

DeltaSetAggregator only tracks added/removed keys since the last window, so it's only correct for non-overlapping (tumbling) windows. Nothing currently enforces this: the planner (agg_config.rs) copies whatever window_type the surrounding query config has when emitting a DeltaSetAggregator config, and the precompute engine's DeltaSetAggregatorAccumulator/pane-merge path is window-type-agnostic. A sliding-window DeltaSetAggregator config can be planned and executed today, silently producing incorrect merged add/remove sets across overlapping panes.

Consequence for the query engine: within a single query, key-cardinality accumulators may need different merge strategies per window type — DeltaSetAggregator panes must be read/merged as tumbling windows, while other accumulators (e.g. SetAggregator, CountMinSketch, HydraKLL) tied to the same query may legitimately use sliding windows and need the sliding merge path.

Needs:

  • Planner-side validation/gate rejecting or refusing to select DeltaSetAggregator when window_type is Sliding.
  • Query-engine read path to handle mixed window types within one query (tumbling merge for DeltaSetAggregator, sliding merge for co-located accumulators).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions