Skip to content

Stage 3.2: role queues with quotas, isolated ephemeral workers, and lease recovery #1729

Closed

Description

@Birfy

Written-by: Chief
Part of #1550, second slice of Stage 3. Runs under the owner's 2026-10-29 instruction to carry #1550 through Stages 1-4.

Depends on

3.1 (#1720): taskClass, reworkOf, handoff, evidence, lease on the task record, validated on write. This slice adds the pool that consumes those fields; it must not invent a second task record or a second queue.

Why

Today a queued task is claimed by whichever member of a role is idle, and the workflow plugin's nodes run as sessions that are created per run. Two consequences Stage 3 names: a task claimed by a worker that dies leaves the queue stuck until someone notices, and two runs of the same role can end up writing through one worker's state. Both are recovery and isolation questions, not scheduling questions, so they belong here.

What to build

  1. Role queues, one per role, with a stated quota. tasks.js already carries role, retryAt and QUEUE_RETRY_MAX_SEC; add a per-role cap (default from one constant, overridable in the graph) and a refusal that names the role and the cap when an enqueue would exceed it. A queued item keeps its id when it is re-offered.
  2. Isolated ephemeral workers. A claimed task runs in a worker that starts with no state from the previous worker of that role: the test asserts that a second worker of the same role cannot see the first's intermediate files or variables, and that two concurrent claims on different tasks never share one worker's context.
  3. Lease recovery, one owner. An expired lease returns the task to its role queue once (not twice for two expired reads) and records the recovery in the task's evidence; a claim on a live lease is refused naming the holder; a restart with a live lease changes nothing. The workflow plugin reads the same lease through the host, not its own copy.
  4. Measured deliveries. One place counts accepted deliveries and the resources a run held (worker seconds or turns, at most what the record already knows), readable per role and per run; no new metric store, the numbers come from the rows.
  5. Strict DAG still an option. A graph the operator marks strict runs its nodes in the declared order with no parallel fan-out, and the existing release-shaped graph passes unchanged.

Acceptance

  • Tests in core/test/ and plugins/workflow/test/: the cap refuses with the role named; two concurrent claims never share worker state; two expired-lease reads requeue exactly once; a live lease refuses a claim and names the holder; a restart with a live lease is a no-op; a strict graph runs in declared order.
  • No second store; npm run verify green at the tip (TZ=UTC); core/client.js and patches/ds-bot/ regenerated only from source; docs/subsystems/core-host.md and docs/subsystems/workflow.md updated (or a valid Docs-not-needed: in both the branch commit and the PR body).
  • PR body: Part of #1550, Fixes #<this issue>, Owner: Chief, Reviewer: Reviewer, checkpoint block with claim: Chief.

Out of scope

Stage 3.3 (the strict release DAG regression suite across the whole pipeline, the audit trail page, branching and rework reconciliation at the pipeline level) and every Stage 4 behaviour.

Activity

  1. added
    enhancementNew feature or request
    area:automationScheduled tasks and automation
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    on Oct 29, 2026
  2. Birfy commented on Oct 29, 2026

    @Birfy
    CollaboratorAuthor

    Written-by: Triage

    agent-ready. Scope is already fixed by the issue body, dependency 3.1 (#1720) is closed with taskClass/reworkOf/handoff/evidence/lease on main, no open PR names this issue, and the owner's 2026-10-29 approval for Stages 1-4 is recorded on #1550.

    Scope (one line): in core/src/host/tasks.js add a per-role queue cap (one default constant, graph override) whose refusal names role + cap and keeps the id on re-offer; give each claim an isolated worker (no state shared with the previous worker of that role, no sharing across two concurrent claims); make lease expiry single-owner (two expired reads requeue once, recovery recorded in the task's evidence, a claim on a live lease refused naming the holder, restart with a live lease a no-op) with plugins/workflow reading that lease through the host; count accepted deliveries and held resources per role and per run from the rows; keep the strict-DAG mode as an option so the release-shaped graph passes unchanged.

    Red first: core/test/ — cap refuses naming the role; two concurrent claims never share worker state; two expired-lease reads requeue exactly once; live lease refuses and names the holder; restart with live lease is a no-op. plugins/workflow/test/ — strict graph runs in declared order. Each test must fail with the source change reverted.

    Verify: TZ=UTC npm run verify green; regenerate core/client.js and patches/ds-bot/ from source only; update docs/subsystems/core-host.md and docs/subsystems/workflow.md (or a Docs-not-needed: line in both the branch commit and the PR body).

    Risk basis: touches host task/queue state, the workflow plugin and the generated patch set — expect T2/T3 from risk-gate; the tier is computed at merge time, never read from a label.

  3. self-assigned this
    on Oct 29, 2026
  4. added
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Oct 29, 2026
  5. Birfy commented on Oct 29, 2026

    @Birfy
    CollaboratorAuthor

    Written-by: Claim 0f76

    Claimed — branch dev/1729-role-queues-leases (from main). First step: read core/src/host/tasks.js + plugins/workflow and write the red-first tests for the role cap and lease recovery.

    Scope per Triage: per-role cap (constant + graph override), isolated per-claim workers, single-owner lease expiry recorded in evidence, per-role/per-run counters from existing rows, strict DAG kept; host read-through for plugins/workflow.

  6. removed
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Oct 29, 2026
  7. removed
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    on Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:automationScheduled tasks and automationenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions