Skip to content

workflow: a step that is still working is killed at the 45-minute deadline and reported as "no reply" #1868

Closed

Description

@Birfy

Written-by: Chief

A workflow step that is still producing output is killed at the workflow's stepTimeoutMinutes deadline and reported as "no reply for N minutes", which reads as silence. On this box the two facts are different: a build step can be working the whole time and still be killed by the deadline.

What happened (2026-10-30, FlyBot 实现线 wf-67229151)

  • i1862-build2 started 18:43:35Z, was still writing its session file at 19:23:07Z (session-08566f9f…/session.v4.jsonl.zstd, 484 KB, mtime 19:23:07Z), and its repo tree had files written at 18:52:10Z. It failed at 19:28:35Z — exactly 45 minutes after start — with {"code":"step-failed","nodeName":"实现","text":"no reply for 45 minutes"}.
  • i1855-build2 did the same: started 18:08:04Z, failed 18:53:04Z (45m00s) with the identical text.
  • The lane had no stepTimeoutMinutes, so it used DEFAULT_STEP_TIMEOUT_MINUTES = 45 (plugins/workflow/src/service.js:36).

So the step was not silent: the run history says "no reply", while the session log shows a turn being written minutes before the kill. The message sends the reader looking for a missing reply that never existed, and the run is marked failed in a way that looks like a model or message-delivery fault (#1866 was exactly that misreading for a different symptom).

Why it matters here

A build step on this host does: tarball download (10 MB) → unpack → npm ci → npm run harness → red-first test → edit → full targeted tests → build → patches generate/check. That routinely exceeds 45 minutes; two of three dispatches of the same issue died on this deadline, and each loss is 45 minutes of a lane slot.

Expected

  1. Say which failure it is. A deadline overrun reads step deadline reached (45 min), not no reply for 45 minutes. If the service can see the step was still producing turns, say that too (last activity 5 min ago).
  2. Let a deadline be extended while the step runs, or at least document and expose the setting where a lane is created — today stepTimeoutMinutes (1..120, workflow/step-timeout) exists but nothing on the lane-creation path sets it, so every lane silently inherits 45.
  3. Optional: raising DEFAULT_STEP_TIMEOUT_MINUTES, or scaling it with stepBudget / node count, so a long lane does not need a manual API call.

Reproduction

curl -s -X POST .../api/bot -d '{"endpoint":"flybot/workflow/assign","payload":{"workflowId":"<lane without stepTimeoutMinutes>","taskId":"slow","title":"slow","brief":"sleep or build for 50 minutes"}}'
# ~45 min later the run history carries: step-failed, "no reply for 45 minutes"

Workaround applied now

workflow/step-timeout set to 120 on wf-67229151 (实现线), wf-3161899a (bot-social issue 实现线), wf-973735fc (issue-build), wf-957f985b (变脏 PR 修复线), wf-7ed167b6 (PR 快线). Verified in workflows.json.

Docs-not-needed: not used — this is an issue, not a pull request.

Activity

  1. added
    bugSomething isn't working
    area:automationScheduled tasks and automation
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    on Oct 30, 2026
  2. Birfy commented on Oct 30, 2026

    @Birfy
    CollaboratorAuthor

    Written-by: Triage

    agent-ready added (labels: bug, area:automation, agent-ready). Scope is frozen below — read it before starting.

    Verified on main 2ffd9fc398696808bde7ca16d4d33296e79958db (2026-10-30T20:05Z):

    • plugins/workflow/src/service.js expireStep() (lines 244–267) reports the deadline as silence: the stage message key is {node} timed out: no reply for {minutes} minutes, the run was marked failed. / …超时:{minutes} 分钟没有回包…, and run.failure.text is no reply for ${minutes} minutes — the text quoted in this issue. fix(workflow): give a step the time its work needs #1581 (merged 59fb637) made the number real but kept the wrong cause.
    • define_workflow is the lane-creation path a Bot uses (plugins/workflow/index.js:135–145): its parameters carry no stepTimeoutMinutes, and service.define() (plugins/workflow/src/service.js:1537–1558) forwards only name/nodes/edges/description, so a lane created that way always inherits the 45-minute default. workflow/create already accepts and validates the field (service.js:1111–1113), so this is a gap, not a new design.

    Files to change

    • plugins/workflow/src/service.js — expireStep(): the stage message (English template key and Chinese text) and run.failure.text say the deadline was reached (step deadline reached (45 min)), not that no reply arrived; keep code: 'step-failed', nodeName and the release/drain path. define(): pass stepTimeoutMinutes through to create() so the existing stepTimeoutValue() validates it — a bad value is refused, never dropped.
    • plugins/workflow/index.js — add optional stepTimeoutMinutes (integer 1..120) to the define_workflow parameters and mention it in that tool's description.
    • plugins/workflow/test/deadline.test.js — the two cases below; use plugins/workflow/test/assign.test.js for the define() pass-through case if that file already exercises define.
    • docs/subsystems/workflow.md — the deadline sentence says the message names the deadline, and define_workflow is listed with workflow/create as accepting stepTimeoutMinutes.
    • changelog.d/1868.fixed.md — user-visible message change.

    Red first (write these before the fix; both must fail on main)

    In plugins/workflow/test/deadline.test.js, beside the existing #1530 deadline test:

    1. Define a lane via svc.define({ name, nodes, edges }), assign, tick to the 45-minute deadline, then assert row.failure.text matches /step deadline reached \(45 min\)/ and does not match /no reply/; assert the stage chat does not say 没有回包.
    2. svc.define({ name, nodes, edges, stepTimeoutMinutes: 90 }) → the stored workflow's stepTimeoutMinutes === 90; stepTimeoutMinutes: 0 → refused with stepTimeoutMinutes is an integer from 1 to 120; omitted → DEFAULT_STEP_TIMEOUT_MINUTES (45).
      On main: (1) fails with no reply for 45 minutes; (2) fails with undefined instead of 90 because define() drops the field.

    Acceptance

    cd plugins/workflow && node --test test/deadline.test.js && npm test
    cd ../.. && TZ=UTC npm run verify
    

    Green must include the new cases. A node that never replies must still fail at the deadline (hard deadline, unchanged) — only the words change.

    Out of scope (do not do in this fix)

    • Making a running step's deadline movable, or letting a retry push it out: service.js:228 states the deadline measures the wait the owner saw, and fix(workflow): give a step the time its work needs #1581 merged that as a hard deadline.
    • Changing DEFAULT_STEP_TIMEOUT_MINUTES (45) or scaling it with node count: that is an owner decision, not this bug. The five live lanes already carry 120 through workflow/step-timeout.
    • Reporting “last activity N min ago”: expireStep() has no partial-activity signal today; the issue makes that conditional, so it is separate work.
  3. removed
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    on Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:automationScheduled tasks and automationbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions