fix(coordinator): stop a stale waiter withdrawal cancelling the fetch that replaced it - #2069
Merged
Merged
Conversation
… that replaced it
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by a review pass over the batch merged in #2062, #2063, #2064 and #2065. This is a defect in #2063, which I wrote.
The bug
SchemaColumnStoreshares one fetch between every caller asking for the same key, and counts live waiters so the fetch is only cancelled once the last of them has gone. The cleanup runs fromwithTaskCancellationHandler'sonCancel, which is not main actor isolated, so it hops:That hop is deferred, and
withdrawWaiterlooked uploads[key]positionally with no check that the load it finds is the one the cancelled waiter joined. So:K.loads[K]holds taskT1, one waiter.T1finishes on its own, W1 resumes and its own cleanup removesloads[K].T2,loads[K]now holdsT2with one waiter.T2.W2's fetch dies with nobody having abandoned it. In the app that is a table whose column details silently fail to load, so its default sort and hidden columns do not apply.
The same missing identity check let a cancelled-but-still-running fetch write its result over a newer one, because the write to
entries[key]was unconditional.The fix
Every
Loadcarries anid. The three places that act on a load check it first:withdrawWaiter(from:loadID:)ignores a withdrawal aimed at a load the key no longer holds.load's completion cleanup only removesloads[key]if it still owns it.entries[key]if it still owns the key, so a superseded fetch cannot overwrite a newer result, and a fetch outstanding acrossremoveAll()can no longer repopulate a cleared cache.The separate
generationcounter thatremoveAll()used is gone: the per-load id subsumes it and covers the per-key races it never could.Tests
Two added, both deterministic rather than timing dependent. The stale-withdrawal test drives the exact interleaving through a
#if DEBUGseam (loadID(for:)andwithdrawWaiterForTesting(from:loadID:)), which is the same shape asDatabaseManager.injectSession. Reproducing it by racing real hops would be flaky in both directions. The second test pins that a superseded fetch cannot overwrite a newer result.28 tests pass across
SchemaColumnStoreCancellationTestsandSQLSchemaProviderTests.swiftlint lint --strictreports 0 violations in 1322 files.No CHANGELOG entry: #2063 has not shipped yet, so per the repo's rule this folds into that entry rather than adding a "Fixed" line for something unreleased.