Skip to content

A computer that has been suspended once never suspends again for a day #253

Description

@beardthelion

Scale-to-zero works once per Bot. A computer that is suspended, then resumed, used, and left alone again is offered for suspension on every sweep after that and never suspended, and stays awake until the next day.

The chain, each step observed against 88078a4 rather than read:

  1. server/src/work/culler.ts:103 offers with key: computer.botId. Nothing in the key varies per idle window.
  2. server/src/work/queue.ts:216-225 finish marks finishedAt rather than deleting, deliberately: the row is what a later offer of the same key collides with, and deleting it made the recovery path a duplicate-run path.
  3. server/src/work/queue.ts:133 offer is onConflictDoNothing on (kind, key), and its comment says a finished row still counts as a conflict, on purpose.
  4. claim requires finishedAt is null, so the surviving row is never handed out.
  5. server/scripts/cull-idle-computers.ts purges rows finished more than 24 hours ago.

So the second offer is a no-op for a day. Nothing reports it: a sweep that offers work and suspends nothing looks exactly like a fleet that is busy.

Reproduced with an integration test driving two idle windows for one Bot against a real queue and database. First window: offered, claimed, suspended, provider.stop called. The Bot is used again an hour later and goes quiet; second window: considered 0, suspended empty, stop never called again. Reading the table directly rather than the report: after the first offer, one row with finishedAt null; after the suspend, the same row with finishedAt set; after the second offer, still one row with the same finishedAt; claimable items for that key, zero. Deleting the finished row first, which is what purge does at 24 hours, makes the second window suspend normally.

The queue already describes this wedge for the other half of the same table. queue.ts:263-272, on items at the attempt cap: "The culler keys on the Bot id, so five failed suspends meant that Bot never scaled to zero again, silently and for ever." That half was given a window to be reaped on. The finished half was left at a day.

Three remedies that look obvious and are not:

  • Key on the last acted time. Still repeats for a Bot that is resumed and never used, because since falls back to the same audit row.
  • Key on startedAt. Dead on arrival: server/src/computer/sandbox.ts:378 takes it from metadata.creationTimestamp, which survives suspend and resume by design, so every later attempt collides.
  • Shorten purge's single olderThanMs. That window also governs the give-up half, where a day is the deliberate backoff before a failing suspension is tried again. Cutting it to the idle threshold turns a permanently failing suspend into a retry every few minutes.

What is left is to keep the two kinds of done on separate clocks: a finished row only has to outlast a sweep, and a row that gave up is kept because that window is also the backoff. A finished suspension kept for the idle window lines the two up exactly, since a Bot cannot qualify as idle again until that long after its last action, by which point its row has gone. A Bot resumed and never used is delayed by at most one idle window rather than a day.

server/tests/computer-culler.integration.test.ts is where this belongs and has no case that runs a second sweep, which is why nothing caught it. PR to follow.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions