fix(dbapi): fail pending queries on SQL connection loss - #74
Draft
rbavery wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What/Why
Implements WBC-1051. A MERGE of 51,101,631 Sentinel-2 STAC rows into
wherobots_open_data.sentinel2.l2a_source_itemson the org-default Micro runtime lost staging SQL session6vqukpvch2cytlat 2026-09-05 02:47:42 UTC. Kubernetes evicted its pod for ephemeral-storage pressure (~73.1 GiB container usage); the client disconnected one second later and remained waiting. Replacement and later idle cleanup obscured the original cause. This was metadata publication after a successful eight-worker crawl, not COG ingestion. Grafana eviction evidence. WBC-1002 is related keepalive work, not the proven cause of this incident.Fail every pending cursor on abrupt or clean WebSocket closure instead of leaving it blocked in Queue.get.
How
Atomically claim terminal query delivery and reject new work after connection loss. Drain buffered results before interpreting closure; already delivered results remain intact. Send failures and explicit close also complete pending cursors. No SQL is automatically retried.
Retain the authenticated session-status URL from connection creation. One best-effort HTTP lookup per failed connection adds firstFailure.message when available, with a one-second socket timeout, no redirects, and a two-second total wait shared by all pending cursors. HTTP errors, malformed/missing details, and stalled lookup fall back to OperationalError with session/execution IDs and an unknown commit outcome warning. Direct connections without a status URL use the same fallback (session ID unavailable).
Release independently for immediate hang prevention. Deploy the companion studio-backend API/compute-worker changes before its agent watcher to enable durable enrichment. A status update may arrive after the lookup deadline; verify catalog/write outcome before manually retrying. No sql-session image change is required.
Verified
uv run pytest -q: 143 passed, including multiple cursors, abrupt/clean closure, buffered/delivered result races, send failure, HTTP failure/missing details, and stalled enrichment.uv run pre-commit run --all-filesandgit diff --check: passed.Co-authored with Codex.