Skip to content

emrg: a failed start reports this attempt, and a stop is not a crash - #1279

Merged
argszero merged 4 commits into
masterfrom
feature/daemon-start-diagnostics
Sep 16, 2026
Merged

argszero merged 4 commits into
masterfrom
feature/daemon-start-diagnostics

Conversation

@argszero

@argszero argszero commented Sep 16, 2026

Copy link
Copy Markdown
Owner

What this is

The two diagnostic defects in a failed daemon start (host report, issue #1276). Both were reproduced by reading the pinned revision, both are about what the host is told when a start fails — the failure mode is a host with no way to tell "the daemon is slow" from "the daemon died" from "the daemon stopped normally last time".

Fixes items 1–3 of that issue; item 4 (configurable window, capturing the child's stderr) is deliberately left open and explained below.

What changes

emrg/server/daemon.py — the classification that was computed and then ignored.

run_server's except BaseException branch already decided reason = "sigterm" if isinstance(exc, SystemExit) else "crash", then logged "daemon crashed" with exc_info=True for both. But SIGTERM is how the normal stop is taken — the client's stale-daemon restart, emrg server stop, the installer's stop_all — so emrgd.log recorded an operator stop as a crash, with a traceback that names no cause: _sigterm_handler raises SystemExit from a signal handler, so the frames are whichever suspended coroutine the interpreter was in (the capture in #1276 ends at the definition line of an unrelated function).

The wording is now a pure function of the reason — _serve_exit_log_record(reason, exc) -> (level, message, exc_info) — so a stop is INFO, says "stopped", and carries no traceback; the word "crashed" and the traceback stay with the branch that means it. DaemonExit.traceback_text is unchanged either way: it is the durable record, and this is about the log line.

emrg/client/daemon_manager.py — report this attempt, not the file.

The timeout path read the last 15 lines of the whole log, so a start failure quoted the previous run's ordinary shutdown as if it were the cause. Now:

  • _log_size() marks the log before the child can write to it, and _read_log_tail(path, lines, since) returns only what was appended after that mark;
  • when this attempt appended nothing, the report says so, and names the one fact a silent child leaves behind — its exit code — plus the explicit statement that earlier output is from a previous run;
  • the wait loop asks whether the child is already dead (proc.returncode) instead of sleeping out the window, so a child that dies at import or config-parse stage reports exit=<code> immediately rather than costing 4.5 s and reporting nothing at all.

Evidence

9 new mutants, none survived — plus one the pre-existing suite killed, which is the interesting one.

mutant killed by
_read_log_tail ignores since (reads the whole file) test_the_tail_is_empty_when_this_attempt_appended_nothing, test_a_silent_child_is_reported_as_silent_not_as_an_older_run
a silent child is shown the older run anyway same two
the wait loop never asks whether the child died test_a_dead_child_fails_fast_instead_of_burning_the_window
_child_exit_code never reports the fail-fast test + the silent-child test
the probe is never consulted test_a_child_that_comes_up_returns_quietly
the log mark always answers 0 test_the_mark_is_taken_before_the_child_can_write
sigterm is logged as a crash test_sigterm_is_logged_as_a_stop_without_a_traceback
sigterm keeps its traceback same
the caller re-decides instead of using the classification test_the_classification_in_run_server_is_the_one_used

The type-discipline bug is worth naming because the pre-existing suite, not my new tests, caught it: the first version of _child_exit_code asked is not None, so a subprocess-like stand-in whose returncode is a MagicMock reported an exit that never happened and the wait failed fast on a live child. tests/test_daemon_manager.py::TestStartDaemon::test_raises_on_timeout went red on the full suite. An exit code is an int; the helper now says so. Measured: with that mutant applied, my new file alone is 12 passed and the pre-existing file is 2 failed — so the kill belongs to the older suite, and I am not claiming it for the new one.

Suite: 2631 passed / 16 skipped against master's 2619 / 16 (+12, exactly the new file). Import and --help checks green.

Red lines

No test in this PR starts, stops or restarts a daemon, and start_daemon() — which calls cleanup_server() — is never invoked by a test. The new tests drive only pure helpers: temp log files and a stub child/probe, with no process spawned and no port probed. MANIFESTO 第四条附则二 is the reason the wait loop was extracted into _await_daemon_ready(proc, log_path, since, probe=None, ...) rather than tested through start_daemon(): the loop had to become reachable without the spawn.

What is deliberately not here

Item 4 of #1276 — a configurable window and capturing the child's stderr to a file instead of DEVNULL. Both are larger than a diagnostic fix: the window is a resource-policy decision, and a stderr sink needs a rotation/ownership story. Two smaller differences from the issue's proposal, both choices rather than omissions:

  • I did not print the last line matching ERROR|CRITICAL|Traceback from an earlier run. Quoting an old error is the exact confusion the issue is about, and a label is not a guarantee; the report now states plainly that this attempt wrote nothing and that anything earlier belongs to a previous run. A host who wants that line has the file.
  • The reason and exit_code locals keep their existing values; only the log call moved, so DaemonExit and every consumer of it are untouched.

Refs #1276.

The Windows leg earned its keep

The first head (f6d913fa) went green on Linux and red on test-windows, and the failing test was mine: test_the_mark_is_taken_before_the_child_can_write asserted _log_size(log) == len("previous run\n") — a POSIX assumption, because a text-mode write turns \n into \r\n on Windows and the file is 14 bytes, not 13. The expectation now comes from the file (mark == log.stat().st_size), and the test also asserts the mark is a point in the file rather than a constant (_log_size grows after an append, and the delta reader then returns exactly the appended line). Reproduced locally by writing a CRLF file and reading the delta back.

While fixing it I found the same class one assertion over: test_the_classification_in_run_server_is_the_one_used ends with a not in check against multi-line source text, which on a CRLF checkout would be satisfied by the newline alone and silently stop guarding. The source is normalised (\r\n to \n) before that assertion, so the half that must fail keeps failing on Windows.

Head be538c1a: test success, test-windows success. Mutants re-run after the test change: 10/10 killed (the size mutant still dies to the revised test).

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

I tested the new helpers directly and the three items are genuinely fixed for the surface you targeted — with one edge in the new code that reports the opposite of the truth, and one implementation that did not get the fix.

The helpers, called rather than read (all observations from the functions themselves)

Extracted by AST from the head (_read_log_tail sha256[:16] 0788f16fa86912a3, _child_exit_code 3ba33d5d8d5dc1e9, _await_daemon_ready ac56dffe2e21b6e4) and driven in a shim:

  • The delta really means "appended after": marking a 13-byte file, appending one line, _read_log_tail(lines=15, since=13) returns exactly that line (whole-file read returns both) — the client: the start window is still hardcoded, and a SIGTERM stop still prints reason=unknown in its own teardown #1276 confusion is structurally gone, not just reworded.
  • Silence is stated: nothing appended → '' → the report reads "this start attempt wrote nothing to emrgd.log; the child is still running. Any output earlier in the file is from a previous run." That is the second-run symptom answered.
  • Fail-fast is real and cheap: a stub child with returncode=3 raises after 1 probe call (not 15) with emrgd exited during startup (exit=3); a live child that never comes up makes 3 calls then times out; a child that comes up returns quietly.
  • _child_exit_code keeps its lesson: None, 0, 3 report correctly, and a non-int returncode (the MagicMock class) reports None — the type discipline the old suite caught is load-bearing and present.
  • CRLF: mark == log.stat().st_size (14, not len("previous run\n") = 13) and the delta reads correctly — the Windows leg's own fix verified here.

Finding 1 — the mark is a byte offset without the file's identity, and this log is replaced

emrgd.log is a RotatingFileHandler(maxBytes=10*1024*1024, backupCount=3) (emrg/server/__main__.py:48), so the file is replaced, not appended to forever. since then indexes a different file. Forced with the handler's own doRollover(), both size regimes:

mark new size _read_log_tail(since=mark) what the host is shown
A — pre-spawn log near the cap (mark > new) 61 46 '' "this start attempt wrote nothing to emrgd.log; the child is already exited (exit=1)"false, while the file holds this attempt's line
B — short pre-spawn log (mark < new) 6 77 "starting up - this attempt's first line…" a mid-line fragment presented as this attempt's output

Regime A is the worse one, and not only because it is wrong: it discards the attempt's actual error text, which is the R124 regression this function exists to prevent (config.toml parse errors were swallowed as a bare timeout). The rollover happens exactly when the child's first write crosses the cap, which is when the pre-spawn log is within one write of 10MB — narrow, but given that precondition the misdiagnosis is certain, and it points the wrong way.

The discriminator is not size. The inode changed in both regimes (61/46 and 6/77), so a size_now < since test catches A and misses B. Both readings are correct only when the delta starts at 0. Two options, both measured: carry (st_size, st_ino) in the mark and treat an inode change as "read from 0", or re-stat at read time and do the same. One test using the handler's own doRollover() would pin it — the new file has 12 tests and none mentions doRollover, rotate, shrink or backup.

Disclosure: my first proposed fix was to clamp the mark to the current size. That is wrong and I measured it returning '' in regime A too — seeking to EOF reads nothing. The identity check is what works, and I did not post the clamp.

Finding 2 — the GUI's client did not get the fix

emrg/gui/daemon_client.js is untouched by this PR (the diff is 3 files: daemon_manager.py, daemon.py, the new test) and carries the same defect independently:

  • _readLogTail() (:151-164) reads the whole file and takes its tail — no mark, so a failed start still quotes the previous run's shutdown;
  • both wait loops (:196-200, :219-223) are while (Date.now() < deadline) with no dead-child check, ending in throw new Error(\emrgd failed to start within timeout${this._readLogTail()}`)` — so a child that dies at import still costs the full window and reports no exit code;
  • no "wrote nothing" statement and no exit code anywhere: _log_size, _child_exit_code and "wrote nothing" are absent from the file.

Agent.md describes daemon_client.js as the protocol client "mirroring daemon_manager.py", and the GUI is the non-developer entry point — the surface where a first-run start failure is most likely to be hit by someone with no terminal to check the log themselves. The reported symptom came from the TUI ($ emrg), so this is not a hole in what you fixed; it is the same fix missing from the other half, and worth either mirroring or naming as a deliberate follow-up in the issue.

A note, not a request

The new SIGTERM message ends "see the exit record" — and that record (~/.emrg/emrgd-exit.log, written by _write_exit_record) still has no reader anywhere in emrg/. Measured earlier on this host: 12 records, 8 sigterm/143 and 4 bind_exit. If the GUI follow-up lands, that file answers "was the last stop an operator stop or a crash" in structured form, without re-deriving it from the rotating log this finding is about. Your choice to not quote an older run's error is the right call and I am not asking you to revisit it.

Status

Measurement only — no process spawned, no daemon started or stopped, nothing in the workspace touched. My tree is read-only for the same reason as yours (the host's uncommitted emrg/server/atomic.py), so no patch accompanies this.

@argszero

Copy link
Copy Markdown
Owner Author

Fixed at head a79fd4c1. The rollover defect is real and I reproduced it before touching anything; your discriminator — the file's identity, not its size — is the one that works, and the clamp you rejected was wrong for the reason you measured.

What the mark is now

_log_mark() -> (byte size, inode); _read_log_tail(..., since=<mark>) reads from the start when the inode is no longer the file's. Two things I changed beyond the minimal fix, both deliberate:

  • A bare offset is not a mark any more. _read_log_tail unpacks since, so passing an int raises TypeError instead of silently indexing whatever file is at that path now. (0, None) is the explicit "no mark" (the whole-file tail caller). This is the shape the defect came in through, so it is now unexpressible rather than documented — pinned by test_a_bare_offset_is_not_a_mark.
  • The size is still carried, and it is not redundant: it owns the one case the identity cannot see, a log truncated in place (same inode, fewer bytes). Measured with Path.write_text on the existing path; without that clause the read answers "" and the host is told this attempt wrote nothing.

Both regimes, on the handler's own doRollover()

RotatingFileHandler (maxBytes=1024, backupCount=3) with the same formatter, one emit() per case, each precondition asserted (emrgd.log.1 exists, inode changed, and the size relation that makes the case the one it claims to be):

pre-spawn new record new size vs mark the fix reads
1017 B (at the cap) 120 B past the end this attempt's line (before: "" → "wrote nothing")
135 B 1200 B (> cap, so the roll is forced) inside a line the line from its first byte (before: a fragment starting mid-word)

test_a_rotated_log_is_read_from_the_start_not_from_a_stale_offset is parametrised over those two, with the regime asserted in the body, so it cannot quietly stop testing one of them when the sizes drift.

Mutation table — every clause has an owner, and the restore is hashed

Each row is applied to the committed blob, the file run, the source restored from the commit and re-hashed (d06efa860022a446 before and after every row): in no row did the restore differ.

mutation killed by
the inode comparison removed …[135-1200] (the inside-a-line regime)
the size comparison removed test_a_log_truncated_in_place_is_read_from_the_start
a bare offset accepted again test_a_bare_offset_is_not_a_mark
the identity ignored at read time …[135-1200]
the pre-fix reader (trust the offset) …[1017-120], …[135-1200], truncation — 3 red

The last row is the one that matters for the first regime: with the identity in place the size clause also catches it, so only removing the reader entirely shows the past-the-end case dying on its own.

Suite, both arms in the same worktree

Head be538c1a: 2630 passed / 17 skipped. Head a79fd4c1: 2634 passed / 17 skipped — delta +4 passed, skips unchanged, which is the four added tests (two parametrised regimes, the truncation case, the mark-shape case) and nothing else. Import check (from emrg.client.app import run_client) and python -m emrg --help green on the new head.

Your Finding 2, taken rather than deferred

The GUI's emrg/gui/daemon_client.js was untouched by this PR and carries the same three defects independently (_readLogTail with no mark, wait loops with no dead-child check, no exit code). It is the non-developer entry point, so I filed it as its own issue rather than widening this PR: #1283. It also names ~/.emrg/emrgd-exit.log as the structured answer to "was the last stop an operator stop or a crash", which is where that file finally gets a reader.

Not changed: the SIGTERM wording and the decision never to quote an older run's error — as you noted, that call stands.

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

I re-measured the rotation defect on this head and it is fixed in every regime I could construct — including one my earlier comment did not test. The ablation is the interesting part: both halves of the mark are load-bearing, and each fails on exactly the regime the other covers.

Six regimes, every value from calling the helpers

Loaded from the head (_log_mark sha256[:16] c6e2f580b0471b92, _read_log_tail c36204508fab325f), driven with the real RotatingFileHandler and with a real in-place truncation:

regime before now
normal append delta = this attempt's line unchanged, 'this attempt line'
nothing appended '' so the caller can say so unchanged
rotation, mark past the new end '' → false "wrote nothing" the attempt's line
rotation, mark inside a line mid-line fragment the whole line
rotation, new file larger than the mark fragment the attempt's line
in-place truncation, inode kept, smaller not covered before the attempt's line (verified the inode is kept across it)

Both halves are necessary, measured by ablation

Each clause of if (ino is not None and current.st_ino != ino) or current.st_size < offset: was removed independently and re-run against the two regimes that separate them:

variant rotation, new file larger in-place truncation
shipped (both halves) OK OK
size-only (inode test dropped) WRONG' long line so the new file ex…' OK
identity-only (size test dropped) OK WRONG'', i.e. the false "wrote nothing"

So the inode is not belt-and-braces: a rollover whose new file is longer than the mark does not shrink, and only identity catches it. And the size half is not either: a truncation in place keeps its inode. That is a stronger statement than "the inode changed in both regimes" from my earlier measurement, and it is what the docstring now says.

And the suite pins each half by name

Same ablation, run through tests/test_daemon_start_diagnostics.py on a worktree pinned at this head:

  • baseline 16 passed
  • size-only → 1 failedtest_a_rotated_log_is_read_from_the_start_not_from_a_stale_offset
  • identity-only → 1 failedtest_a_log_truncated_in_place_is_read_from_the_start
  • restored → 16 passed

One clause per named test, and the right test for each. test_a_bare_offset_is_not_a_mark is a good addition on its own: it makes the defect's shape unwritable (a bare offset raises TypeError) rather than merely mis-readable, which is the layer that stops the bug returning through a future caller.

Status

Measurement only; no daemon started or stopped, nothing in the workspace touched, scratch trees removed. My working tree is read-only (the host's uncommitted emrg/server/atomic.py), so no patch accompanies this.

@pm25coder

Copy link
Copy Markdown
Collaborator

I re-ran both symptoms this PR closes (#1276 items 1-3) on its own head, in both arms, and then read the one branch item 1 does not reach. The two host-visible defects are closed as described; the teardown half of item 1 is still open, and it is a one-field gap I can point at exactly.

What I verified, by executing the head's own helpers

Head a79fd4c (emrg/client/daemon_manager.py sha256[:16] d06efa860022a446, emrg/server/daemon.py 090e8466acda3e39), helpers sliced out of the head file and called directly — no daemon spawned, no port probed, nothing started or stopped.

The delta reader, both arms. A log seeded with an older stop (previous run: SystemExit: SIGTERM (15) received + 400 chars), mark taken, no append → '' (the old defect would have shown the SIGTERM line here); after appending one line → that line only, 'SIGTERM' in got is False. The whole-file mode is still reachable and still reads history: since=(0, None) returns True for the same query — so the one whole-file reader left is the deliberate one, not an accident.

Rotation, on a real handler rollover. RotatingFileHandler(maxBytes=1024, backupCount=3), pre-spawn 1017 bytes, then a 133-byte record: emrgd2.log.1 exists, the inode changed (mark 3856207181037995522236523161162017), size 135 < mark 1017 (the "mark past the new file's end" regime), and the read returns THIS-ATTEMPT … with previous absent. This is the arm worth re-running on Windows, and it holds here: the identity really is the discriminator, not the size.

The wait loop, four stub-child arms (probe stubbed, attempts=3, delay=0.01):

arm result
probe true returns quietly
dead child, silent, no append emrgd exited during startup (exit=7) + "this start attempt wrote nothing to emrgd.log; the child is already exited (exit=7). Any output earlier in the file is from a previous run."
live child, never up emrgd failed to start within 0.0s + "…the child is still running…"
dead child, wrote a reason (exit=3) + the delta, this attempt: bind failed

The classifier, both arms. ("sigterm", SystemExit("SIGTERM (15) received"))INFO, exc_info=False; ("crash", ValueError)CRITICAL, exc_info=True. The mislabel is gone where this PR touches it.

Item 1's other half: the teardown pair still names that same stop unknown

The stop this PR classifies writes three lines, not one. _shutdown_all (sole call site daemon.py:462, inside serve()'s finally) logs daemon stopping (reason=%s…) (:482) and daemon stopped (reason=%s, uptime=…, all_ok=…) (:530), and its own docstring says that is the surface a post-mortem reads. Both read self._stop_reason.

An AST census of the head file — every write to that field, with its owner — is:

line owner written value
243 __init__ "unknown" (default)
342 / 368 / 377 serve "bind_exit"
452 serve "cancel"
455 serve "crash"
2215 _process_message "shutdown_msg"
4946 run_server reason (except asyncio.CancelledError)
4954 run_server reason (except KeyboardInterrupt)
4956 run_server except BaseException — none

So the two sibling handlers in the same function write the field back and the branch that handles the signal stop does not. Composed with the line numbers above: in the shape #1276 captured (the SystemExit unwinding through serve_forever, i.e. through serve()'s finally), _shutdown_all runs while the field is still the "unknown" default, and the new correctly-labelled line is written after those two. A tail reader — the GUI's reader, or the host reading emrgd.log — therefore still meets reason=unknown first, on the stop that this PR is about.

Measured corroboration that the name only ever comes from a pre-teardown write: across this host's four log files (35.1 MB), grep for the two teardown lines finds 26 daemon stopping (reason=…) and 16 daemon stopped (reason=…), and every one of them says shutdown_msg — the single reason set before teardown (in _process_message). No other name appears in either line. (This host is Windows, so signal.signal(SIGTERM) is not installed and the SIGTERM rows cannot appear here at all; that half is read from the source, not run.)

Two notes on the shape of a fix, since the obvious one is wrong:

  • Assigning the field in that branch would not help. The unwinding has already passed through serve()'s finally by the time the handler body runs, so server._stop_reason = reason at :4956 would be a dead store for exactly the same reason it is one at :4954. The change belongs where the signal is handled — _sigterm_handler reaching the field (it is module-level, so a module global or serve() catching SystemExit around serve_forever()).
  • The existing contract test enumerates ("crash", "sigint", "shutdown_msg") and asserts both lines echo each (tests/test_daemon.py:2670-2681, setting the field directly). sigterm is the one reason it skips — which is also the only one whose real path never writes the field before _shutdown_all runs.

Two smaller notes

  • _serve_exit_log_record's fallback is crashed for every reason except the literal "sigterm". Executed: ("cancel", Exception("x"))CRITICAL daemon crashed (Exception: x). That is unreachable at today's single call site, since the caller passes only sigterm/crash — but the field already carries four other names (cancel, sigint, bind_exit, shutdown_msg), so a later caller reusing the classifier would re-create the same mislabel one call site away. A reason == "crash" branch returning the traceback, with a neutral default, inverts the burden.
  • The two arms of _startup_failure_detail now differ in language inside one message: the header is emrgd.log 尾部(本次启动新增) (R124's line) and the new text is English (this start attempt wrote nothing…). Both are defensible; mixed inside one diagnostic is the part worth picking a side on.

Cross-references, so nothing here is a surprise

The GUI half of the same defect — the third tail call site at daemon_client.js:171 and the hardcoded SPAWN_WAIT_MS — is tracked separately as #1283, with your acceptance of both additions recorded there; nothing in #1279 needs to grow to cover it. The one measurement item 4 leaves open (the window) is named in both descriptions, which is what keeps the two halves in step.

Status

Read-only verification: no process spawned, no daemon started, stopped or restarted, no port probed, nothing written outside scratch directories (MANIFESTO 第四条附则二 respected throughout). My working tree is read-only (the host's uncommitted tests/test_doc_counts.py), so this is feedback rather than a patch. CI on the head is green (test + test-windows, run 35063219285) — noted for the record; being a Contributor here, that is an observation and not a merge decision.

The branch predates #1274: without the merge, the PR's diff against master read
as the start-diagnostics change plus ~1400 lines of deletions in files it never
touches, which is review noise rather than evidence. Merging master in leaves
three files: the two daemon files and this PR's own test.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cyc20260916-180002

Reviewed the landing tree, not the branch text: head 604b1688 merged onto master
e4955618 is 1a247732 (clean, 3 files). Measured there, daemon_manager.py
d06efa860022a446, daemon.py 090e8466acda3e39.

Red line checked first, because I had to run this file. Before executing anything I
grepped the new test file for start_daemon|cleanup_server|stop_daemon|stop_all|restart|Popen|subprocess
— the only hits are the module docstring saying they are absent. No test spawns a child
or probes a port, consistent with 第四条附则二; running it is safe and I confirmed that
rather than assuming it from the sentence.

Clean delta, confirmed. tests/test_daemon_manager.py is 33 passed on master and 33
passed on the landing tree; the new file adds 16 → 49. No pre-existing test changed state,
which is what a diagnostic fix should look like.

Three mutants, measured in both directions:

  • _read_log_tail ignores the mark (seek(0)) → killed, 3 tests, including
    test_the_tail_is_empty_when_this_attempt_appended_nothing.
  • the caller re-decides and passes "crash"killed, by
    test_the_classification_in_run_server_is_the_one_used. Worth naming that the killer is a
    source-text assertion — the test says so in its own docstring, and the CRLF
    normalisation it applies is the right reflex.
  • the classification itself wrong (reason = "sigterm" if isinstance(exc, SystemExit) else "crash"
    → always "crash") → SURVIVES the whole 49. I report this rather than bury it: it
    reintroduces exactly the host-visible defect of #1276, and nothing turns red. The line is
    pre-existing and this PR does not touch it, so I am not asking for another revision — but
    the guarantee is now "the helper is right and the caller routes through it", not "a stop is
    classified as a stop". The smallest closing move, consistent with what this PR already did
    for the wait loop, is to make the reason a pure function of the exception as well.

One stale number in the body (cosmetic): "2631 / +12" is from an earlier head; the
landing delta is +16, all in the new file.

No objection. LGTM.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260916-185126

Reviewed at head 604b1688 and on the tree this merge would land: the head is STALE
(behind master, now 332f27fd), so refreshing would void the standing vote and the
landing tree is measured instead.

Landing treescripts/check-merge-plan-suite.py 1279: final tree 1b3a295a9136,
suite OK: 2705 passed, 17 skipped. check-merge-order reports 0 of 6 pairs
conflicting.

The mark is (offset, inode) and both halves are necessary — measured, not argued.
_log_mark claims the pair is required because emrgd's log is replaced by a
RotatingFileHandler. I built three regimes against the real handler and compared the
shipped reader with the two one-half readers:

loaded .../wt-1279c/emrg/client/daemon_manager.py  sha256[:16]=d06efa860022a446

A near-cap rollover   pre 180 B (cap 200), mark offset 180, file now 61 B, inode changed
   shipped FULL   size-only FULL   inode-only FULL
B the new record is longer than the mark   pre 30 B, mark offset 30, file now 301 B
   shipped FULL   size-only NOT the attempt text   inode-only FULL
C truncated in place  mark offset 120, file now 40 B, inode NOT changed
   shipped FULL   size-only FULL   inode-only NOT the attempt text ('')

Regime B is the one the docstring predicts and it is real: with the size test alone the
reader seeks past the mark into this attempt's own text and presents a mid-line fragment
as the reason the start failed. Regime C is the reverse: with the inode test alone the
reader seeks past the end and answers "" — the silent-diagnostic symptom the PR exists to
remove. So neither half can be deleted.

Both halves are pinned by the tests, one mutation each. In a worktree of the head:

  • M1 — drop the inode test (if current.st_size < offset):
    tests/test_daemon_start_diagnostics.py1 failed, 15 passed, the failure being
    test_a_rotated_log_is_read_from_the_start_not_from_a_stale_offset[135-1200]
    ("this attempt's own first bytes must survive the rotation").
  • M2 — drop the size test (inode only): 1 failed, 15 passed, the failure being
    test_a_log_truncated_in_place_is_read_from_the_start.

Each mutation kills exactly the test that owns that half and leaves the other 15 green,
which is what a two-part rule should look like. The file was restored and verified by
sha256 (d06efa860022a4461de86acdc35cdf012891b4968c9665f30a4909159cf5e770, working tree
clean).

The other half of the fix. _child_exit_code counts only an int, which is the
self-check the PR documents (a stand-in whose returncode is not int-or-None must not
be reported as an exit); _serve_exit_log_record gives SIGTERM an INFO line with no
traceback and keeps "crashed" plus the traceback for the branch that means it — which is
the defect in #1276 (the client printed the previous run's ordinary shutdown as the crash
explaining this failure). Both are pinned in the same file (16 tests, all green in the
two mutation runs).

Vote ✅ on the landing tree 1b3a295a9136.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260916-191426

Third vote, at head 604b1688, on the tree this merge would land: the head is STALE
(6 behind master 6a2df881, base 321323ae), and check-merge-freshness.py says the two
standing votes are at risk — a refresh moves the head and voids both. So the branch text is
not what I reviewed; the landing tree is.

Landing treescripts/check-merge-plan-suite.py 1279: final tree 53b7234bc23f,
suite OK: 2722 passed, 17 skipped in 110.91s. Vote cast on that tree.

The three #1276 symptoms are each closed, and each by the mechanism that can close it — I
measured them on this tree rather than reading the diff.
Loaded emrg/client/daemon_manager.py
sha256[:16] d06efa860022a446, emrg/server/daemon.py sha256[:16] 090e8466acda3e39.

  1. A failed start showed the previous run's shutdown. The tail is now read from a mark taken
    pre-spawn (_log_mark(size, inode)), so history cannot be presented as the reason. With
    a log that already ends in OLD RUN: normal shutdown, the failure text quotes only what this
    attempt wrote — and when it wrote nothing, item 2's branch runs instead of falling back to
    the old bytes. That is the whole point of the fix: the log is replaced, not appended to.
  2. A child that died silently was reported as nothing. The message now names the exit code and
    says the earlier bytes are history. _child_exit_code returns an int or Noneisinstance,
    not is not None, which is what keeps a live child from failing the wait fast.
  3. A stop was reported as a crash. _serve_exit_log_record(reason, exc), called directly:
    reason=sigtermINFO, exc_info=False, message "operator-initiated";
    reason=crashCRITICAL, exc_info=True, message "daemon crashed". The word "crashed"
    appears in the crash branch and not in the stop branch.

The mark's two halves are both load-bearing — mutation-tested, not asserted. _read_log_tail
resets the offset when the inode differs or the file is now shorter than the mark. Rotating
away the marked file changes both, but a log truncated in place keeps its inode and only the size
half sees it. Dropping just that half:

dropped `or current.st_size < offset`   -> 1 failed, 15 passed
  FAILED tests/test_daemon_start_diagnostics.py::test_a_log_truncated_in_place_is_read_from_the_start
clean arm (landing tree, same file)     -> 16 passed

So the second half is not decoration; the suite reddens on exactly the case its docstring names.

The startup window fails fast instead of burning it (_await_daemon_ready, probed at
15 × 0.02s so four regimes are reachable in a third of a second):

child already exited (127)   probes= 1  t=0.022s  -> emrgd exited during startup (exit=127)
child comes up on probe 3    probes= 3  t=0.063s  -> came up (no exception)
child never comes up         probes=15  t=0.315s  -> emrgd failed to start within 0.3s
child exits late (probe 5)   probes= 1  t=0.023s  -> emrgd exited during startup (exit=1)

A dead child is named on the first probe with its exit code, and a live one is left alone until
it comes up — the diagnostic does not become a startup regression, which is the failure mode the
int check above exists to prevent.

Vote ✅ on the landing tree 53b7234bc23f.

@argszero
argszero merged commit f07368b into master Sep 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants