Repository navigation
Fix two races when a replica starts - #175
Merged
Merged
Conversation
A replica that has just started could answer its first request with 503. The Tests job failed on this once, on Postgres, at test_a_lost_database_degrades_instead_of_crashing. - catch_up() recorded the check as done before the lamp's state had arrived. A request in that gap found no state and a recent check, and reported the lamp unreachable. The check is now recorded last. - Store.pulse() updated a replica's row and inserted it if nothing was updated. Two reports at once (the background loop and a first request) both inserted, and on Postgres the second failed on the primary key. It is now one INSERT ... ON CONFLICT DO UPDATE. Each has a test that failed ten times out of ten before the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Tests job for #173 failed once on
main, on the Postgres run, in a test that change did not touch:Because that run failed, #173 was never published by itself; #174's run, a minute later, passed and published both.
The failure is not a test problem. A replica that has just started can answer its first request with 503, in two ways.
1. The check was recorded before the answer arrived
catch_up()asks the database twice: who is watching, then the lamp's state. It sethub.checked_atbetween the two. A request arriving in that gap saw no state and a check made a moment ago, so it did not ask for itself and reported the lamp unreachable.Fix: record the check after the state is in.
2. Two first reports from one replica collided
Store.pulse()ran an UPDATE and, if no row was updated, an INSERT. At startup the background loop and the first request both report. On Postgres both updates find nothing, both insert, and the second fails on the primary key. SQLite is unaffected because its writers take turns.Fix: one
INSERT ... ON CONFLICT DO UPDATE, which both databases support.Checked
Against a local Postgres 17 and SQLite, ten runs each:
test_a_request_during_the_first_sync_is_not_turned_away(new)assert 503 == 200test_a_replica_reporting_twice_at_once_is_counted_once(new)UniqueViolation; passes on SQLitetest_a_lost_database_degrades_instead_of_crashing(the CI failure)The whole suite passes on both (14 tests).
Which of the two caused the CI failure cannot be told from its log: both end in the same 503.
馃 Generated with Claude Code