Skip to content

Wait on events, not a fixed budget, in two flaky test waits (#2102) - #2104

Merged
Birfy merged 3 commits into
mainfrom
dev/2102-time-flake-tests
Nov 2, 2026
Merged

Birfy merged 3 commits into
mainfrom
dev/2102-time-flake-tests

Conversation

@Birfy

@Birfy Birfy commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Written-by: Dev

Fixes #2102
Owner: Chief
Reviewer: Reviewer
Branch: dev/2102-time-flake-tests
Base: 1778add
Tip SHA: 4e24b45
Docs-not-needed: test-only waits; no subsystem page owns a test's wait helper.

问题

两处测试把「等到某个事件」写成了「烧掉一个固定预算」,慢机上没坏也红:

  • core/test/follow-through.test.js 的 waitFor() mocked 分支用 20000 轮 setImmediate 自旋当预算。一轮约几微秒,20000 轮 ~70–150ms 的纯自旋,而它等的真实工作(inflight.json/state.json 写盘、Session flush、resolve Agent)在 2 核 CI 上被这层自旋抢 CPU,于是 1840b05e 的 push Test 里 settle() 抛 condition not reached。
  • plugins/social/test/social.test.js 的 #90 hide-remote 用例用 Promise.race + 60ms 墙钟判定「没有等 room leave」。60ms 测的是调度延迟:慢机上实现完全正确也会报 hide-remote waited for room leave(c4e425f9 的 push Test)。

改法

  1. core/test/follow-through.test.js:waitFor() 每一步改成 realSetTimeout(REAL_WAIT_STEP_MS=5ms) 让出 CPU,同时仍 mockedTimers.tick(1) 推进被 mock 的时钟;删掉轮数下限(rounds=20000 参数),只剩真实时钟 deadline,并把预算提到 15s。判定条件完全不变:真丢的调用照样在 deadline 内失败。
  2. plugins/social/test/social.test.js:#90 用例改成门闩。在 ctx.flybot.scope 上包一层(plugin 自己 build scoped API,所以必须在 createSocial 之前),让 removeParticipant 先 await held;断言 hide-remote 在门闩未放开时已经返回且 leftRooms 含两个 room、两个 room 里的座位仍在,然后 release() 再用文件里已有的 waitUntil 等座位落定。删掉 60ms 的 Promise.race 与墙钟断言。
  3. core/ 是 ds-bot/ + patches/ds-bot/:用 node scripts/ds-bot-patches.mjs generate 重建系列,只变了 patches/ds-bot/0001-core-fixes.patch;check 与 canonical 都绿。

测试

  • node --test core/test/follow-through.test.js:19/19 pass,连跑 30/30 轮全绿。
  • node --import ./test/resolve.mjs --test test/social.test.js(plugins/social):450/450 pass,连跑 30/30 轮全绿。
  • 红先反证(日志在评论区):follow-through 打断 resume pickup → 15.4s 后 condition not reached 红;social 让 hide-remote 先 await leave → 门闩永不放开、timed out 红。
  • 本 PR 的 Test:tests-ok、package、各 plugins (*) 全绿。

风险

T1(test-only)。无产品代码、无 workflow、无依赖变化;放宽的是等待方式,不是判定条件。

回滚

revert 本 PR 即可。

Checkpoint

{
  "pr": 2104,
  "claim": "Chief",
  "issue": 2102,
  "summary": "两处测试把等事件写成了烧固定预算,慢机上没坏也红。follow-through 的 waitFor 改成 5ms realSetTimeout 步进并只留 15s 真实时钟 deadline,删掉 20000 轮下限。social 的 #90 hide-remote 用例改成门闩,证明返回时 room leave 仍被挡住,再放开等座位落定,删掉 60ms race 与墙钟断言。core 的改动同步重新生成了 patches/ds-bot/0001-core-fixes.patch。",
  "files": [
    "core/test/follow-through.test.js",
    "plugins/social/test/social.test.js",
    "patches/ds-bot/0001-core-fixes.patch"
  ],
  "checks": {
    "ran": "node --test core/test/follow-through.test.js (19/19, 30x); plugins/social node --import ./test/resolve.mjs --test test/social.test.js (450/450, 30x); ds-bot-patches check/canonical; PR Test runs green",
    "result": "green",
    "at": "4e24b4571f31d6e79fd7c2b128cf5c7b1654b978"
  },
  "next": "Reviewer 在 tip 上出 PASS;然后入队合并,main 的 push Test 连绿两次兑现最后一条验收"
}

Birfy added 2 commits November 2, 2026 10:22
Both waits decided a real event by burning a fixed budget, so a loaded runner made them fail
while nothing was wrong (push Test at 1840b05 and c4e425f, green again at 1778add).

core/test/follow-through.test.js: a mocked-clock wait now steps in real time
(realSetTimeout + tick) instead of 20000 setImmediate rounds, and the round floor is gone, so the
real-clock deadline is the only bound (5s -> 15s). A genuine lost call still runs out.

plugins/social/test/social.test.js: hide-remote's #90 case holds the room leave itself instead of
racing a 60ms wall clock, and proves the call returned with the seats still held; waitUntil then
waits for the released leave.

Fixes #2102

Docs-not-needed: test-only; no subsystem behaviour changes.
createSocial() builds its own scoped API, so patching ctx.flybot.rooms never reached the leave
path; wrap ctx.flybot.scope before the plugin exists instead. Same latch, same assertions.
@Birfy Birfy added the agent-pr Pull request opened by an agent label Nov 2, 2026
@Birfy Birfy self-assigned this Nov 2, 2026
core/ is ds-bot/ + patches/ds-bot/ (AGENTS.md), so the edit to core/test/follow-through.test.js
has to land in the series too. Rebuilt with scripts/ds-bot-patches.mjs generate (git 2.56.0, the
dugite build); check and canonical are both green, and only 0001-core-fixes.patch changed.
@Birfy

Birfy commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

Written-by: Dev

验收证据(tip 4e24b4571f31d6e79fd7c2b128cf5c7b1654b978)

连跑 30 遍

  • core/test/follow-through.test.js:30/30 轮 # fail 0(每轮 19/19 pass)。
    $ for i in $(seq 1 30); do node --test core/test/follow-through.test.js; done | grep '^# fail' | sort | uniq -c
         30 # fail 0
    
  • plugins/social:30/30 轮 # fail 0,每轮 450/450 pass。
    $ for i in $(seq 1 30); do node --import ./test/resolve.mjs --test test/social.test.js; done | grep '^# fail' | sort | uniq -c
         30 # fail 0
         30 # pass 450
    

红先反证(把产品代码改坏,新用例必须红)

  1. follow-through —— 把重启后的 pickup 打断(core/src/host/resume.js 的 resumeTurn() 里 room 分支不再 deliver),用例在 deadline 上红:
    not ok 1 - a group turn cut off by a restart is picked up once, and the group is told
      error: 'condition not reached'
      name: 'AssertionError'
    # fail 1
    duration_ms 15456.9
    
    15.4s 说明它等满了真实时钟 deadline 才失败——放宽的是等待方式,不是判定条件(真丢的调用照样红)。
  2. social —— 让 hide-remote 先 await afterHideRemote(key) 再返回(即等 room leave),[Team] Remove a friend's Bot, and approve held Bot replies inside the chat #90 用例红:
    not ok 1 - #90: hide-remote leaves independent rooms in parallel and does not wait for them
      error: 'timed out'
    # fail 1
    
    门闩没放开时调用不返回,所以断言「返回时座位还占着」失败。

core/ 与 patches/ds-bot/

core/ 是 ds-bot/ + patches/ds-bot/(AGENTS.md)。第一次 push 后 CI 的 package job 红在
ds-bot-patches: core/test/follow-through.test.js: differs from ds-bot/ + patches;已用
node scripts/ds-bot-patches.mjs generate 重建系列(只变了 patches/ds-bot/0001-core-fixes.patch):

ok   ds-bot/ = Fei2411/DeepSeek-Bot@2b10c70 (tree 5834c00, v0.1.5)
ok   10 patches apply cleanly in order
ok   core/ equals ds-bot/ + patches/ds-bot/
ok   patches/ds-bot/ is what generate writes (generate is a no-op)

push Test 连绿两次

dev 分支的 push 不触发 Test(本分支三次 push 一条 push run 都没有;push 事件只对 main 开),
所以「连推两次全绿」只能在本 PR 的 pull_request Test 上先看、合入 main 后由 main 的两次 push 兑现。
本 PR 的 Test 在修 patch 系列后重跑,结果见下面的 check 摘要。

@Birfy

Birfy commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

Written-by: Dev

修了 checkpoint 块(原来是项目符号,guard 要的是 <!-- flybot:checkpoint --> 之间一个 JSON 对象,末尾 <!-- /flybot:checkpoint -->),并把 Owner 与 claim 对齐(都是 Chief)。这条评论也会让 Enqueue guard 重新判一次这个 head。

现在 guard 只剩一条正常的红:「no Verifier report names the tip 4e24b45」——等 Reviewer 在 tip 上的 PASS。

@Birfy

Birfy commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at 4e24b45
Written-by: Reviewer
Attestation: 4e24b45 role=Reviewer

Ran at this tip:

$ node --test core/test/follow-through.test.js
# tests 19  # pass 19  # fail 0
$ cd plugins/social && DSH_HARNESS=$PWD/../.. node --import ./test/resolve.mjs --test test/social.test.js
# tests 450  # pass 450  # fail 0
$ node .github/scripts/changelog.mjs check
$ node scripts/ds-bot-patches.mjs check
$ node risk-gate.mjs --pr 2104

审查要点(2026-11-02T02:52Z,tip 4e24b45,risk-tier T2)

  • 干净副本(base 1778add 与 tip 都从 API tarball 取),core/node_modules 装好后跑:follow-through 19/19、social 450/450 全绿。
  • 红先:把两处测试文件换回 base 的 blob 重跑,follow-through 19/19、social 450/450 也全绿 —— 这台 16 核机器上基线的时序 flake 复现不出来(基线的 20000 轮自旋与 60ms race 只在 2 核 CI 上才红),因此红先在本机不成立;能落地的反证是 Dev 在评论区做的实现反向修改(打断 resume pickup → 15.4s 后 condition not reached;hide-remote 先 await leave → 门闩不放行超时红)。改动本身是判定条件不变、只把等待方式从自旋/墙钟改成事件门闩,断言是加强不是放松:social 新增 leaves > 0 且断言离开前座位仍在。
  • 范围:compare base...tip 恰好 3 个文件(core/test/follow-through.test.js、plugins/social/test/social.test.js、patches/ds-bot/0001-core-fixes.patch),ahead_by 3、behind_by 0,无无关 core/ 源码、无其它 patch。
  • ds-bot-patches check:10 个 patch 顺序应用干净,core/ 与 ds-bot/ + patches 一致;changelog check 通过(7 个片段合规)。
  • tip 上 check-runs 全终态:tests-ok、package、plugins (social/timeline/tree/workflow)、ubuntu-latest·Node 24、risk tier、reviewed tip、merge-gate 均 success;cancelled 的是 agent-audience / judge(非失败);唯一的 failure 是 enqueue-guard,缺本 tip 的 PASS 属预期红。

@Birfy

Birfy commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at 4e24b45
Written-by: Reviewer
Attestation: 4e24b45 role=Reviewer
Tip SHA: 4e24b45

Ran at this tip:

$ tar xzf p2104tip.tgz -C p2104tip --strip-components=1; tar xzf p2104base.tgz -C p2104base --strip-components=1
$ cd p2104tip && node --test core/test/follow-through.test.js
# tests 19  # pass 19  # fail 0
$ cd p2104tip/plugins/social && node --import ./test/resolve.mjs --test test/social.test.js
# tests 450  # pass 450  # fail 0
$ cd p2104tip && node scripts/ds-bot-patches.mjs check
# ok 10 patches apply cleanly in order; ok core/ equals ds-bot/ + patches/ds-bot/
$ # red check: core/src/host/resume.js room pickup deliver removed, tests untouched
$ cd p2104tip && node --test --test-name-pattern "group turn cut off by a restart" core/test/follow-through.test.js
# tests 1  # pass 0  # fail 1   (condition not reached)
$ # red check: plugins/social/src/social.js hide-remote awaits afterHideRemote, tests untouched
$ cd p2104tip/plugins/social && node --import ./test/resolve.mjs --test --test-name-pattern "#90: hide-remote leaves independent rooms" test/social.test.js
# tests 1  # pass 0  # fail 1   (timed out)

Review points:

  • 红先成立(两条各自独立)。 只把 core/src/host/resume.js 的房间 pickup deliver 拿掉、测试保持 tip 版,main push CI 两处时间型 flake:follow-through 群回合恢复 settle / social hide-remote 等 room leave #2102 的那条 condition not reached 照样红;只把 plugins/social/src/social.js 的 hide-remote 改成 await afterHideRemote(key)、测试保持 tip 版,[Team] Remove a friend's Bot, and approve held Bot replies inside the chat #90 在门闩上 timed out 红。tip 上同一组命令 19/19、450/450 全绿。被改的是等待方式,判定条件没放宽:真丢的调用仍然在 deadline 上失败。
  • 范围与 compare 一致。 compare 1778add0ac…4e24b4571f 是 3 ahead / 0 behind、3 个 commit、3 个文件 +80/−81,全部是 PR 正文说的两个测试文件与 patches/ds-bot/0001-core-fixes.patch 镜像;无产品代码、无 workflow、无依赖变化,T1。
  • tip check-runs。 15 条里 14 绿(tests-ok、package、merge-gate、四条 plugins、Node 24、risk tier、reviewed tip、agent-audience 的 cancelled 是并发取消不算失败),唯一的红是 enqueue-guard:no Verifier report names the tip 4e24b4571f31 —— 本条即那份 report。
  • 正文元数据。 agent-pr 标签在、Owner: Chief、Reviewer: Reviewer、checkpoint 块字段齐(pr/claim/issue/summary/files/checks/next),与 tip 一致;base 仍是当前 main,不欠 rebase。
  • 限制。 本机只跑了受影响的 core/follow-through 与 plugins/social 两套,没有重跑全量 npm run verify 与 30 轮连跑;全量与连跑证据取自 PR 评论里 Dev 的日志和 tip 的 tests-ok/package 绿。

@github-actions github-actions Bot added the pass label Nov 2, 2026
@Birfy
Birfy added this pull request to the merge queue Nov 2, 2026
Merged via the queue into main with commit 51afc79 Nov 2, 2026
11 of 15 checks passed
@Birfy
Birfy deleted the dev/2102-time-flake-tests branch November 2, 2026 02:50
@github-actions github-actions Bot removed the agent-pr Pull request opened by an agent label Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

main push CI 两处时间型 flake:follow-through 群回合恢复 settle / social hide-remote 等 room leave

1 participant