Give every process its own lease name, so two on one machine cannot both hold an item - #435
Merged
davidmckayv merged 2 commits intoSep 8, 2026
Conversation
zopeVaibhav
requested review from
MikeRyanDev,
davidmckayv,
guidovizoso and
tylerslaton
as code owners
September 8, 2026 19:16
davidmckayv
approved these changes
Sep 8, 2026
davidmckayv
left a comment
Contributor
There was a problem hiding this comment.
Deep-reviewed against live code (correctness, governance, no vendor/secret/scale issues). Composed build+tests green. CHANGELOG/format rebase on CI-validated substance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
oursinserver/src/work/queue.tsdecides whether a claimed item is still yours by comparingclaimed_byagainst the caller's own name, and nothing else — no lease comparison, becauseclaimalready refuses a row whose lease has not lapsed. So the name has to be the identity of a process. It was the identity of a machine.Six places built it, all as a role plus the hostname, with randomness only as a fallback for a hostname that is missing. Two processes on one machine got the same string, and
ourscannot tell them apart: after A's lease lapses and B claims the item, A'srenewstill answers true, and both dispatch. Five of the six also used??, which falls back only onundefined, soHOSTNAME=""collapsed the owner to the constanthandoff/— one name for every replica in a deployment.shared/work-owner.tsis now the one place that builds it, and it always appends a random suffix. The hostname is kept in front of it, so a stuck claim still traces back to the machine holding it, which is what the comments at those call sites ask the name for. A blank or whitespace-onlyHOSTNAMEis treated as absent rather than as a name.The
generateIdparameter onloadWorkerEnvgoes with it: it existed to make the fallback testable, and there is no longer a fallback to test — the suffix is unconditional.Fixes #434.
Where it runs
Squarely a replica question, which is why it is worth stating plainly rather than ticking.
for update skip lockedand lease comparison do what they were written to do. Two replicas on different hosts were already fine unlessHOSTNAMEwas empty, which is the??half of this.Boundary and audit
skippedreport.Changelog
CHANGELOG.mdunderUnreleased.Proof
The defect and the fix measured the same way, against the real
createWorkQueueand Postgres: one item, two claimants, the lease allowed to lapse between them.The control that isolates the cause is in the before/after itself: the only thing that changes is how the two processes name themselves, and a run with deliberately distinct names (
routines/pod-a,routines/pod-b) already answered false before this change.shared/work-owner.test.tscovers the contract directly: two calls with one environment never match, a blank or whitespaceHOSTNAMEnever becomes the whole owner, and the hostname survives in the name.worker/tests/env.test.tsmoved from asserting an exactroutines/laptopto asserting the shape, and gained one for two workers on a host not sharing an owner.bun run typecheckclean acrossapp,serverandworker.biome lint --error-on-warningsclean over 582 files;biome formatclean over 578. Worker tests 19 pass / 0 fail. The repository suite runs 2577 tests here against 2572 onmain— the five added — with the same single failure in both,agent-langgraph/tests/history.test.tsfailing to resolve@langchain/core/messages, which is a missing dependency in the local checkout and not touched by this.Not covered: two real worker processes were not run side by side against one deployment. The defect is in what those two processes compute as their names and how the queue compares them, and both halves are exercised above directly.