Skip to content

Give every process its own lease name, so two on one machine cannot both hold an item - #435

Merged
davidmckayv merged 2 commits into
CopilotKit:mainfrom
zopeVaibhav:fix/work-owner-unique-per-process
Sep 8, 2026
Merged

davidmckayv merged 2 commits into
CopilotKit:mainfrom
zopeVaibhav:fix/work-owner-unique-per-process

Conversation

@zopeVaibhav

Copy link
Copy Markdown
Contributor

What this changes

ours in server/src/work/queue.ts decides whether a claimed item is still yours by comparing claimed_by against the caller's own name, and nothing else — no lease comparison, because claim already refuses a row whose lease has not lapsed. So the name has to be the identity of a process. It was the identity of a machine.

Six places built it, all as a role plus the hostname, with randomness only as a fallback for a hostname that is missing. Two processes on one machine got the same string, and ours cannot tell them apart: after A's lease lapses and B claims the item, A's renew still answers true, and both dispatch. Five of the six also used ??, which falls back only on undefined, so HOSTNAME="" collapsed the owner to the constant handoff/ — one name for every replica in a deployment.

shared/work-owner.ts is now the one place that builds it, and it always appends a random suffix. The hostname is kept in front of it, so a stuck claim still traces back to the machine holding it, which is what the comments at those call sites ask the name for. A blank or whitespace-only HOSTNAME is treated as absent rather than as a name.

The generateId parameter on loadWorkerEnv goes with it: it existed to make the fallback testable, and there is no longer a fallback to test — the suffix is unconditional.

Fixes #434.

Where it runs

Squarely a replica question, which is why it is worth stating plainly rather than ticking.

  • New state that outlives a request? None. This changes a string a process computes about itself at start-up.
  • What happens on the second replica? This is the change. Before, a second process on the same machine shared the first one's lease name and both could hold one queue item at once; after, every process is its own claimant and the queue's existing for update skip locked and lease comparison do what they were written to do. Two replicas on different hosts were already fine unless HOSTNAME was empty, which is the ?? half of this.
  • Anything serialised? The queue's own claim/renew/finish, unchanged. This restores the identity those depend on rather than adding a mechanism beside them.
  • Anything fanned out to a browser? No.
  • New listener, port, or schedule? None.

Boundary and audit

  • Every acting call still goes through the gateway: untouched.
  • New refusals and new failures each write a row: no new refusal or failure. A firing that is correctly skipped because the lease went elsewhere was already reported through the sweep's skipped report.
  • Nothing new is trusted from the client: the owner is derived from this process's own environment and a random suffix, never from a request.

Changelog

  • A line in CHANGELOG.md under Unreleased.

Proof

The defect and the fix measured the same way, against the real createWorkQueue and Postgres: one item, two claimants, the lease allowed to lapse between them.

BEFORE  A = routines/laptop            B = routines/laptop
        B took it after the lapse:    true
        A still believes it holds it: true      <- both dispatch

AFTER   A = routines/laptop-07b5e731   B = routines/laptop-b63a54be
        B took it after the lapse:    true
        A still believes it holds it: false     <- A is told it lost the lease

The control that isolates the cause is in the before/after itself: the only thing that changes is how the two processes name themselves, and a run with deliberately distinct names (routines/pod-a, routines/pod-b) already answered false before this change.

shared/work-owner.test.ts covers the contract directly: two calls with one environment never match, a blank or whitespace HOSTNAME never becomes the whole owner, and the hostname survives in the name. worker/tests/env.test.ts moved from asserting an exact routines/laptop to asserting the shape, and gained one for two workers on a host not sharing an owner.

bun run typecheck clean across app, server and worker. biome lint --error-on-warnings clean over 582 files; biome format clean over 578. Worker tests 19 pass / 0 fail. The repository suite runs 2577 tests here against 2572 on main — the five added — with the same single failure in both, agent-langgraph/tests/history.test.ts failing to resolve @langchain/core/messages, which is a missing dependency in the local checkout and not touched by this.

Not covered: two real worker processes were not run side by side against one deployment. The defect is in what those two processes compute as their names and how the queue compares them, and both halves are exercised above directly.

@davidmckayv davidmckayv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep-reviewed against live code (correctness, governance, no vendor/secret/scale issues). Composed build+tests green. CHANGELOG/format rebase on CI-validated substance.

@davidmckayv
davidmckayv merged commit d9bb793 into CopilotKit:main Sep 8, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two processes on one machine share a lease name, so a routine can fire twice

2 participants