Skip to content

fix(dbapi): fail pending queries on SQL connection loss - #74

Draft
rbavery wants to merge 1 commit into
wherobots:mainfrom
rbavery:codex/wbc-1051-disconnect
Draft

fix(dbapi): fail pending queries on SQL connection loss#74
rbavery wants to merge 1 commit into
wherobots:mainfrom
rbavery:codex/wbc-1051-disconnect

Conversation

@rbavery

@rbavery rbavery commented Sep 5, 2026

Copy link
Copy Markdown
Member

What/Why

Implements WBC-1051. A MERGE of 51,101,631 Sentinel-2 STAC rows into wherobots_open_data.sentinel2.l2a_source_items on the org-default Micro runtime lost staging SQL session 6vqukpvch2cytl at 2026-09-05 02:47:42 UTC. Kubernetes evicted its pod for ephemeral-storage pressure (~73.1 GiB container usage); the client disconnected one second later and remained waiting. Replacement and later idle cleanup obscured the original cause. This was metadata publication after a successful eight-worker crawl, not COG ingestion. Grafana eviction evidence. WBC-1002 is related keepalive work, not the proven cause of this incident.

Fail every pending cursor on abrupt or clean WebSocket closure instead of leaving it blocked in Queue.get.

How

Atomically claim terminal query delivery and reject new work after connection loss. Drain buffered results before interpreting closure; already delivered results remain intact. Send failures and explicit close also complete pending cursors. No SQL is automatically retried.

Retain the authenticated session-status URL from connection creation. One best-effort HTTP lookup per failed connection adds firstFailure.message when available, with a one-second socket timeout, no redirects, and a two-second total wait shared by all pending cursors. HTTP errors, malformed/missing details, and stalled lookup fall back to OperationalError with session/execution IDs and an unknown commit outcome warning. Direct connections without a status URL use the same fallback (session ID unavailable).

Release independently for immediate hang prevention. Deploy the companion studio-backend API/compute-worker changes before its agent watcher to enable durable enrichment. A status update may arrive after the lookup deadline; verify catalog/write outcome before manually retrying. No sql-session image change is required.

Verified

  • uv run pytest -q: 143 passed, including multiple cursors, abrupt/clean closure, buffered/delivered result races, send failure, HTTP failure/missing details, and stalled enrichment.
  • uv run pre-commit run --all-files and git diff --check: passed.
  • Controlled fixtures only; no production changes, workload launches, crawl reruns, or full MERGE re-execution. Not deployed; hosted CI tracked separately.

Co-authored with Codex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant