Skip to content

Process differential E2E advances event time with a future watermark #488

Description

@zzylol

Context

Found while reviewing #478.

The stable process differential creates a data point at wall-clock now and immediately sends an empty watermark at now + 2 seconds:

ddsketch_export(METRIC, sample_ns, &raw_values)
ddsketch_export(METRIC, sample_ns + 2_000_000_000, &[])

This makes the test fast, but it exercises acceptance of future event time rather than the normal collector/backend timing contract. It may hide bugs in real window close, allowed lateness, watermark progression, and query evaluation relative to wall clock. The whole-backend E2E already avoids this by waiting for the controller-selected window to actually end.

Reproduction

./scripts/e2e.sh differential

Location: data_plane/tests/promql_differential_process_e2e.rs, around lines 235-242.

Expected

The stable production-process differential should not require future-dated telemetry.

Preferred options:

  1. wait for the one-second window to end, then emit a wall-clock watermark; or
  2. expose and wait for an explicit ingest/window-commit signal without modifying event timestamps.

Done when

The test passes using realistic non-future timestamps and deterministically observes the committed window without a fixed blind sleep.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions