Skip to content

fix(graphql): resume collectors from durable checkpoints after restart - #9148

Closed
ryo-ebata wants to merge 1 commit into
apache:mainfrom
ryo-ebata:codex/graphql-resume-9142
Closed

ryo-ebata wants to merge 1 commit into
apache:mainfrom
ryo-ebata:codex/graphql-resume-9142

Conversation

@ryo-ebata

Copy link
Copy Markdown

Summary

Persist GraphQL collector progress per task, raw scope, nested collector and input so restarting DevLake can resume from the last committed page instead of clearing raw data and starting over.

  • Commit raw rows, transactional parser cleanup and the input checkpoint together.
  • Retain completed run markers across the collector/runner completion window.
  • Keep nested collectors from flushing each other's raw rows.
  • Anchor incremental watermarks to the original task start; do not skip failed/panicking subtasks or duplicate their rows after restart.
  • Propagate iterator, cancellation and partial-response failures; reject incompatible or superseded checkpoints.
  • Stabilize input ordering and remove per-page full raw COUNT scans for pure parsers.
  • Add disposable MySQL/PostgreSQL SIGKILL tests and CI jobs.

Does this close any open issues?

Closes #9142.

Validation

  • Related helper, runner, migration, server, GitHub GraphQL and Linear packages: go test -race -count=1.
  • Reproducible integration suite: bash backend/test/graphql-resume-crash.sh and DEVLAKE_CRASH_DRIVER=postgres bash backend/test/graphql-resume-crash.sh.
  • Each database runs 36 top-level checks and 14 SIGKILLs, including two interruptions of the same real Jobs task, exact extracted Job assertions, initialization/page/completion crash windows, rollback and lost-commit-acknowledgement injection, resume-disabled/rerun API behavior, and real PR/Issue/Linear collector recovery.
  • Additional unit assertions ensure the append-only page path does not run COUNT, and failed commits do not advance the in-memory row count.

Compatibility and limits

The migration adds _devlake_graphql_collector_states; task-less callers retain legacy behavior. Known input/code/page-setting changes are rejected and require a new task. Unknown GraphQL partial errors no longer complete a persisted task even when IgnoreQueryErrors is set. Inputs are staged to disk before fetching to detect iterator failure and manifest drift.

Raw integrity uses row counts, not content checksums. Arbitrary connection/environment changes and upstream cursor/result-set changes are not snapshot-guaranteed. The subtask progress display remains per-attempt; persisted data/cursors determine what is refetched. No live GitHub/Linear service or production database was used for the integration tests.

See backend/test/graphql-resume-coverage.md for the full audit matrix and reproduction instructions. Local results do not imply upstream GitHub Actions approval or success.

Signed-off-by: ryo-ebata <r.ebata@hajimari.inc>
@ryo-ebata
ryo-ebata marked this pull request as draft September 18, 2026 03:55
@ryo-ebata ryo-ebata closed this Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][github_graphql] Collector re-collects from page 1 after pipeline resume (pagination cursor not persisted)

1 participant