Repository navigation
The aid pty tests flake on a loaded cargo test --workspace run #401
Description
Finding
rust/aid/tests/interactive.rs fails intermittently on a loaded cargo test --workspace, and passes reliably when the suite is run alone.
Measured on this machine (devcontainer, pixi run cargo test --workspace, main at 9392779):
- suite alone,
cargo test -p aid --test interactive: 3/3 green - full workspace run: 2 failures in 4 runs
The failure seen was a_pasted_multi_line_prompt_arrives_whole_rather_than_leaking (aid/tests/interactive.rs:315), with "press Enter" never appeared after the 60s deadline in wait_for (:205).
The tests drive the real binary through a pty and poll wait_for at 25ms against a 60s deadline, so they are sensitive to scheduling pressure from the rest of the workspace run. Each also spawns python3 launch_scenario.py to build a scratch world first.
Why it matters beyond the noise
CI green becomes probabilistic, and a real regression in the interactive flow arrives looking exactly like the flake — which is the state in which flakes stop being investigated.
Shape
Unclear, deliberately. Worth establishing first whether the deadline is simply too tight under load, or whether there is a genuine lost-wakeup between the pty reader thread (:148) and wait_for. Options in rough order of cost: raise the deadline, serialise the pty tests against each other, or hand the reader thread a condvar instead of polling.
Found while adding a pty regression test in the DEVLAUNCH_NO_TTY fix; that test is not the cause — the baseline flakes more often without it.
Activity
blooop commentedon Aug 24, 2026
Diagnosed and fixed in #414 (open, not merged).
Mechanism, confirmed. Not a lost wakeup, not a dead reader thread, not a deadline that is too tight.
PtyAid::send_linewrote the typed line and its Enter as twowrite_allcalls on the pty master. A master write reaches the line discipline in one go, so two writes are two of those with a gap between them: the first completesfix thisand wakesaid's read,read_terminal_submissiondrains what the terminal holds right now (its zero-timeout poll is deliberate), and while the second write is still in flight the paste's tail is not yet a completed line, so the prompt arrives cut at line one. The test's own comment claimed "One write, as a terminal delivers a paste"; the helper it called had never made that true. Onewrite_allof line plus newline, and every completed line of a paste becomes readable at the same instant.Confirmed by knob rather than by argument: inserting a deliberate 20ms sleep between the two writes moved the failure rate from 14 in 100 to 19 in 20.
Numbers. Suite binary run back to back on 8 cores under a load average of 12 to 25:
interactive.rs:291(paste truncated to line one)The reported symptom did not reproduce.
"press Enter" never appearedafter the 60s deadline was not seen once in 280 runs of this suite or in any of the 6 full-workspace runs measured. The deadline inwait_foris left alone: raising it for a mechanism nothing confirmed would buy only a slower failure. Every failure this ticket did reproduce was the paste-tail assertion, in the test the ticket names.Other flakes seen while measuring, not this ticket's and not touched:
flows::launch::tests::a_contended_up_of_a_running_workspace_runs_no_up_at_all,timing::tests::stages_closed_out_of_order_do_not_stay_on_the_clock(twice), anddl/tests/interrupt.rs'sclosing_the_terminal_mid_up_removes_the_token_file_and_kills_the_up. Also:a_ctrl_c_at_the_editor_tears_the_whole_boot_downhangs its "the orphaneddevpod upmust have been killed" assertion on await_forwhose subject is asleep 30, so a kill that never happened still satisfies it 30 seconds later. Separate weakness, separate ticket.blooop commentedon Aug 24, 2026
breadcrumb: mechanism confirmed and fixed in PR #414 (86/100 → 100/100). Four further flaky tests and one vacuous assertion found while measuring, carried to #416 rather than folded in — and the symptom this ticket originally recorded (
"press Enter" never appeared) did not reproduce in 280 runs, which #416 also carries.blooop commentedon Aug 24, 2026
breadcrumb: wf-review on #414 — approve. Mechanism reproduced independently (two writes + 20ms gap fails every time; base helper 2 failures in 60 on an idle box, this PR 0 in 40), and the test still grips production: disabling the drain at
dl/src/lib.rs:287fails it with exactly the truncated paste. Note for whoever closes this ticket: the symptom recorded here ("press Enter" never appeared) is not what was fixed — close against #414 and #416. #414The opt-out's truthy direction was the half nothing held
Gate the worktree metadata prune on what the run did
blooop commentedon Aug 28, 2026
Closing against PR #414 and #416, per the review's instruction. #414 fixed the one flake this ticket names at its mechanism (the paste arrived as two pty writes; it is now one
write_all), measured 86/100 before and 100/100 after, knob-confirmed. The symptom string recorded here ("press Enter" never appearing) never reproduced in 280 runs; it and the four other flakes found while measuring are carried by #416.