Skip to content

test(kernel): dump daemon-side state when a dispatch never arrives (#174) - #178

Merged
kjgbot merged 1 commit into
mainfrom
diag176
Sep 5, 2026
Merged

kjgbot merged 1 commit into
mainfrom
diag176

Conversation

@kjgbot

@kjgbot kjgbot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Replaces #176, which GitHub auto-closed when #175 merged and its base branch was deleted. Same change, rebuilt directly on main now that both #175 and #177 have landed.

Why

#174 produced four occurrences and, between them, four test names. Nothing else. The state that would explain it was all daemon-side and none of it survived: wait_with_output() is never reached, so the child's output is dropped in the unwind, and the run's journal was never read.

What it does

On a missing dispatch the test attempts to terminate the resumed child and reaps it, then reports its stdout, stderr, and the run's journal entries with seq, type and step.

The child may be stalled or may have already exited, and those are indistinguishable from the test's side — which is exactly why the dump matters. #174 turned out to be the second: the resume died instantly with run_not_found while the test waited 60s on it.

The journal is the important half. The question a missing dispatch raises is did the daemon resume and stall, or never resume at all? Nothing else answers it. The listing covers the current segment — every entry these non-compacting crash tests produce, though not every entry under compaction.

This is what found #174's root cause, fixed in #177.

Evidence

Forcing the read ceiling to 1ms:

before-first: no step.dispatch after resume: timed out after 1ms waiting for a
protocol frame; the daemon sent nothing (see #174)
--- resume child ---
stdout (0 bytes):

stderr (0 bytes):

--- journal (1 entries) ---
  seq=1 type=RunSpawned step=None

sha256 19f3431a → 361572ef → restored 19f3431a, no 1ms literal left in the tree.

Measured at this head: workspace 152 passed, 0 failed, no warnings; crash_resume 34 passed in 37.98s.

Signoff: local 3-lens preswarm at this head — maintainability / history / structure all REVIEW_PASSED. The history lens caught that an earlier message asserted the child "is still running" while also describing a child that had already exited; that contradiction is corrected in both the message and the code comment.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

#174 produced four occurrences and, between them, four test names.
Nothing else. The state that would explain it was all on the daemon side
and none of it survived: `wait_with_output` is never reached, so the
child's output is dropped in the unwind, and the run's journal was never
read at all.

On a missing dispatch the test now attempts to terminate the resumed
child and reaps it, then reports its stdout, its stderr, and the run's
journal entries with seq, type and step.

The child may be STALLED or may have ALREADY EXITED, and the two are
indistinguishable from the test's side -- which is exactly why the dump
matters. #174 turned out to be the second case: the resume died
instantly with `run_not_found` while the test waited 60s on it. An
earlier revision of this message asserted the child "is still running"
and was "wedged by definition", which contradicted the very failure it
described; the helper terminates and reaps either way, discarding both
results because "already gone" is a normal outcome here rather than an
error.

The journal is the important half. The question a missing dispatch
raises is whether the daemon resumed and stalled partway or never
resumed at all, and nothing else answers it. The listing covers the
CURRENT segment, which is what `journal_entries` scans -- every entry
these non-compacting crash tests produce, though not every entry under
compaction.

This is what found #174's root cause: a run whose journal existed but
whose registry row did not, because `Engine::start` registers last.
Fixed in #177.

Verified by forcing the read ceiling to 1ms:

  before-first: no step.dispatch after resume: timed out after 1ms
  waiting for a protocol frame; the daemon sent nothing (see #174)
  --- resume child ---
  stdout (0 bytes):
  stderr (0 bytes):
  --- journal (1 entries) ---
    seq=1 type=RunSpawned step=None

sha256 19f3431a -> 361572ef -> restored 19f3431a, no 1ms literal left in
the tree.

Measured at THIS head, on main after #175 and #177: workspace 152
passed, 0 failed, no warnings; crash_resume 34 passed in 37.98s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
@coderabbitai

coderabbitai Bot commented Sep 5, 2026 •

Copy link
Copy Markdown

Review Change Stack

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Free

Run ID: d773fd7e-6100-4fd3-b6f6-c46d61f5507b

📥 Commits

Reviewing files that changed from the base of the PR and between 696d3e3 and e6a557d.

📒 Files selected for processing (3)
  • kernel/relayflowd/tests/crash_resume/agent.rs
  • kernel/relayflowd/tests/crash_resume/llm.rs
  • kernel/relayflowd/tests/crash_resume/support.rs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Crash-resume tests now capture resume dispatch errors instead of unwrapping them. A shared helper terminates the resume process, collects stdout and stderr, and reports current journal state in labeled panic diagnostics.

Changes

Crash Resume Diagnostics

Layer / File(s) Summary
Resume diagnostic collection
kernel/relayflowd/tests/crash_resume/support.rs
Added describe_stalled_resume to reap the resume process, capture both output streams, and report current journal entries or an absent-journal status.
Dispatch failure reporting
kernel/relayflowd/tests/crash_resume/agent.rs, kernel/relayflowd/tests/crash_resume/llm.rs
Resume dispatch now handles protocol errors explicitly and includes stalled-resume diagnostics in panic output. Successful dispatch remains unchanged.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

A rabbit checks the resume trail,
And gathers logs when steps derail.
The journal speaks, the streams reply,
While labeled panics point the why.
Crash tests hop onward, ears held high.

Warning

Some tools did not complete. Review the errors below.

🔧 Clippy (1.97.1)

Clippy execution failed


Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

@kjgbot
kjgbot merged commit e0ecae3 into main Sep 5, 2026
2 of 4 checks passed
@kjgbot
kjgbot deleted the diag176 branch September 5, 2026 15:50
kjgbot pushed a commit that referenced this pull request Sep 5, 2026
 merged

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant