Scale-to-zero works once per Bot. A computer that is suspended, then resumed, used, and left alone again is offered for suspension on every sweep after that and never suspended, and stays awake until the next day.
The chain, each step observed against 88078a4 rather than read:
server/src/work/culler.ts:103 offers with key: computer.botId. Nothing in the key varies per idle window.
server/src/work/queue.ts:216-225 finish marks finishedAt rather than deleting, deliberately: the row is what a later offer of the same key collides with, and deleting it made the recovery path a duplicate-run path.
server/src/work/queue.ts:133 offer is onConflictDoNothing on (kind, key), and its comment says a finished row still counts as a conflict, on purpose.
claim requires finishedAt is null, so the surviving row is never handed out.
server/scripts/cull-idle-computers.ts purges rows finished more than 24 hours ago.
So the second offer is a no-op for a day. Nothing reports it: a sweep that offers work and suspends nothing looks exactly like a fleet that is busy.
Reproduced with an integration test driving two idle windows for one Bot against a real queue and database. First window: offered, claimed, suspended, provider.stop called. The Bot is used again an hour later and goes quiet; second window: considered 0, suspended empty, stop never called again. Reading the table directly rather than the report: after the first offer, one row with finishedAt null; after the suspend, the same row with finishedAt set; after the second offer, still one row with the same finishedAt; claimable items for that key, zero. Deleting the finished row first, which is what purge does at 24 hours, makes the second window suspend normally.
The queue already describes this wedge for the other half of the same table. queue.ts:263-272, on items at the attempt cap: "The culler keys on the Bot id, so five failed suspends meant that Bot never scaled to zero again, silently and for ever." That half was given a window to be reaped on. The finished half was left at a day.
Three remedies that look obvious and are not:
- Key on the last acted time. Still repeats for a Bot that is resumed and never used, because
since falls back to the same audit row.
- Key on
startedAt. Dead on arrival: server/src/computer/sandbox.ts:378 takes it from metadata.creationTimestamp, which survives suspend and resume by design, so every later attempt collides.
- Shorten
purge's single olderThanMs. That window also governs the give-up half, where a day is the deliberate backoff before a failing suspension is tried again. Cutting it to the idle threshold turns a permanently failing suspend into a retry every few minutes.
What is left is to keep the two kinds of done on separate clocks: a finished row only has to outlast a sweep, and a row that gave up is kept because that window is also the backoff. A finished suspension kept for the idle window lines the two up exactly, since a Bot cannot qualify as idle again until that long after its last action, by which point its row has gone. A Bot resumed and never used is delayed by at most one idle window rather than a day.
server/tests/computer-culler.integration.test.ts is where this belongs and has no case that runs a second sweep, which is why nothing caught it. PR to follow.
Scale-to-zero works once per Bot. A computer that is suspended, then resumed, used, and left alone again is offered for suspension on every sweep after that and never suspended, and stays awake until the next day.
The chain, each step observed against
88078a4rather than read:server/src/work/culler.ts:103offers withkey: computer.botId. Nothing in the key varies per idle window.server/src/work/queue.ts:216-225finishmarksfinishedAtrather than deleting, deliberately: the row is what a later offer of the same key collides with, and deleting it made the recovery path a duplicate-run path.server/src/work/queue.ts:133offerisonConflictDoNothingon(kind, key), and its comment says a finished row still counts as a conflict, on purpose.claimrequiresfinishedAt is null, so the surviving row is never handed out.server/scripts/cull-idle-computers.tspurges rows finished more than 24 hours ago.So the second offer is a no-op for a day. Nothing reports it: a sweep that offers work and suspends nothing looks exactly like a fleet that is busy.
Reproduced with an integration test driving two idle windows for one Bot against a real queue and database. First window: offered, claimed, suspended,
provider.stopcalled. The Bot is used again an hour later and goes quiet; second window:considered0,suspendedempty,stopnever called again. Reading the table directly rather than the report: after the first offer, one row withfinishedAtnull; after the suspend, the same row withfinishedAtset; after the second offer, still one row with the samefinishedAt; claimable items for that key, zero. Deleting the finished row first, which is whatpurgedoes at 24 hours, makes the second window suspend normally.The queue already describes this wedge for the other half of the same table.
queue.ts:263-272, on items at the attempt cap: "The culler keys on the Bot id, so five failed suspends meant that Bot never scaled to zero again, silently and for ever." That half was given a window to be reaped on. The finished half was left at a day.Three remedies that look obvious and are not:
sincefalls back to the same audit row.startedAt. Dead on arrival:server/src/computer/sandbox.ts:378takes it frommetadata.creationTimestamp, which survives suspend and resume by design, so every later attempt collides.purge's singleolderThanMs. That window also governs the give-up half, where a day is the deliberate backoff before a failing suspension is tried again. Cutting it to the idle threshold turns a permanently failing suspend into a retry every few minutes.What is left is to keep the two kinds of done on separate clocks: a finished row only has to outlast a sweep, and a row that gave up is kept because that window is also the backoff. A finished suspension kept for the idle window lines the two up exactly, since a Bot cannot qualify as idle again until that long after its last action, by which point its row has gone. A Bot resumed and never used is delayed by at most one idle window rather than a day.
server/tests/computer-culler.integration.test.tsis where this belongs and has no case that runs a second sweep, which is why nothing caught it. PR to follow.