Skip to content

Prevent missed updates by using stable scheduler checkpoints #13

Open

Description

@666mxvbee

Problem

The checkpoint is updated to DateTimeOffset.UtcNow after external API calls and message creation:

An event created after the API response but before the checkpoint update can be permanently skipped.

External clients also read only the first page:

Proposed solution

  • Capture checkStartedAt before requesting the external API.
  • Persist checkStartedAt only after detected events have been stored.
  • Store stable source event identifiers where available.
  • Handle external API pagination until the previous checkpoint is reached.
  • Deduplicate source events by provider and external event ID.

Acceptance criteria

  • Events created during a check are detected during the next check.
  • A checkpoint advances only after events are persisted successfully.
  • GitHub and StackOverflow pagination is supported.
  • Repeated API pages do not create duplicate events.
  • Tests cover events created between request start and checkpoint commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    criticalpriority: critical

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions