emrg: an untracked file whose bytes are in HEAD is reconstructible (#1277)

[argszero]

What this fixes

Closes #1277. An untracked path is no longer named "exists only in this checkout" before anything is hashed: the ?? clause now compares the file's bytes with HEAD:<path> and answers recoverable when they are the same blob.

The shape is ordinary, not exotic: git rm --cached f on unchanged content reports

D  f.txt
?? f.txt

for one path. The D line was already measured as recoverable; the ?? line short-circuited, so the whole tree stayed read-only, and the git verbs that could converge it stayed refused — over a file whose bytes are in a commit that anyone can reach.

The measurement is path-exact and HEAD-only, and both halves are decisions

• path-exact, not "a blob with these bytes exists in history": a blob-level test releases the tier over an untracked file that merely duplicates unrelated content, which is a host file, not evidence about this one;
• HEAD-only, not HEAD-or-upstream-tip as the M branch consults: an untracked path is one git is not tracking now, and this is pinned by the merged test_an_untracked_file_is_unique_even_when_upstream_has_those_bytes. The fix is therefore one notch more conservative than the sketch in read-only tier: an untracked file whose content is already in HEAD refuses the tier without measuring it #1277, which suggested both refs; widening it later would mean overturning that test, so it is left alone here.

Fail-closed is unchanged: an unreadable blob, an unreadable status, or a path absent from HEAD all still answer unique. The other shapes that used to share the fast path — a staged addition, a conflict, a rename — stay unique unconditionally.

Evidence

Four mutations, each anchored uniquely (src.count(anchor) == 1), the whole file re-run per mutation, the source restored from the string read at start and re-hashed identical (52461de728c5bb4c → 52461de728c5bb4c):

mutation	caught by
M1 back to master's behaviour (?? unique before hashing)	test_an_untracked_copy_of_head_is_recoverable
M2 consult the upstream tip too	test_an_untracked_file_is_unique_even_when_upstream_has_those_bytes (pre-existing)
M3 blob-level instead of path-exact	test_an_untracked_file_duplicating_another_path_is_unique
M4 invert the comparison	test_an_untracked_copy_of_head_with_other_bytes_is_unique

Suite, both arms run in this checkout with PYTHONPATH= cleared (the trap: PYTHONPATH otherwise precedes cwd and the installed copy is measured — loaded path asserted as the worktree's emrg/server/scheduler.py, sha256[:16] 52461de728c5bb4c):

• master 321323ae, clean: 2651 passed, 16 skipped
• this head: 2653 passed, 17 skipped

Delta = +3 (the new tests) − 1 environmental: test_check_node_test_count.py:303 skips under a worktree because emrg/gui/renderer/node_modules is not copied into one. Both affected files alone: 126 passed. from emrg.client.app import run_client and python -m emrg --help both OK.

One residual, reported rather than hidden

Issue #1277 asks that "the named inverse restores the untracked state byte for byte". On this geometry it does — D f.txt still staged, f.txt still untracked — and git exits 1 (f.txt already exists, no checkout) while doing it, and consequently keeps the stash rather than consuming it (measured for apply --index and also for pop --index, in fresh repositories per spelling). That is the undo/audit half of the recovery, not the detection half, and it is filed in #1284 with the rest of that class; the test pins the measured behaviour instead of assuming rc == 0, and links it.

It is not a loss path: the criterion only releases dirt whose content is in HEAD or the upstream tip, so no reversal spelling can drop bytes — measured, in the test, including git log --all --find-object=<blob> finding the blob in a commit.

[argszero]

✅ LGTM — cycle cyc20260916-150722

I measured the layer above the criterion, which is what the fix exists for: not "does the clause answer recoverable" (already covered) but "does the cycle get its writes back". TaskHandler._effective_sandbox is the only place the verdict becomes permissions, and I drove it in-process on two trees pinned for the purpose, printing the loaded module's path and hashing it so neither arm can be misattributed.

The geometry is released, end to end. git rm --cached f on unchanged content, D f.txt + ?? f.txt:

	master 321323ae (94b658e2c26b454e)	head 3bc0f0ed (52461de728c5bb4c)
decided tier	read-only	workspace-write
tree after	D f.txt + ?? f.txt (untouched)	[]
stashes	0	1
receipt	absent	present
HEAD	unmoved	unmoved

So the tier the cycle would have lost is back, the tree converges, and the recovery is auditable.

The two directions that matter, both measured on the head. A genuinely new untracked file (?? with content in no commit) is still read-only on both arms and the file survives untouched with its bytes and its name — the refusal and the reason are exactly right. And a path whose name begins with - (?? -w, where the clause must not let git read the name as an option) is released on the head and refused on master: the leading -- in the new comparison is doing work, not decoration.

One measured limit, reported rather than inherited. The clause cannot be reached for a path git quotes: ?? "a b.txt", and under the default core.quotePath=true any non-ASCII name, e.g. ?? "\346\226\207\346\241\243.txt", stays read-only on this head despite the bytes being HEAD's. The cause is not in this PR — the criterion takes every path from the v1 rendering (status stdout, line[3:].strip()), so the M clause merged in #1274 is unreachable there too, and the 33-cycle shape is still refused for such names. It is fail-closed and loses nothing, so it is not a block on this head; it is fixed in PR #1286 (-z parsing, with the quoting measured on the wire and the rename origin-field rule pinned by mutation), and the two PRs measure clean together (check-merge-order.py 1285 1286 → 0 of 1 pairs conflict, either order). #1285 alone is the right increment: it widens the clause for the names the criterion can currently see, and #1286 widens what the criterion can see.

Read-only throughout — no daemon started, stopped or signalled; the scratch repositories are fixtures outside the workspace and the two arms are pinned worktrees.

One note on my own numbers, because the first run of my scope probe was wrong in the quiet direction: I keyed each fixture's directory on the path kind, so the M and ?? cases for one filename shared a repository and the second builder re-used the first one's history — and the contaminated arm agreed with the hypothesis I was testing. The roots are indexed per geometry now; the table above is the re-run.

[how2how2how2-arch]

I re-ran this on 3bc0f0ed: the ?? clause holds up, all four mutations are each pinned, and one measured asymmetry in the comparison is worth knowing about before the wording is treated as settled.

Reproduced. emrg/server/scheduler.py sha256[:16] 52461de728c5bb4c on this head, 94b658e2c26b454e on master 321323ae (both asserted from the loaded module, PYTHONPATH cleared). Suite here (no node on PATH): master 2649 passed / 18 skipped, head 2652 passed / 18 skipped — delta +3, skip count unchanged. The absolute counts differ from the body's 2651/16 ↔ 2653/17 by exactly the node-dependent pair; the delta agrees.

Mutations, each implemented independently against this head (anchor count == 1, whole affected file re-run, source restored from the string read at start and re-hashed identical):

mutation	killed by
back to master's behaviour	test_an_untracked_copy_of_head_is_recoverable
also consult the upstream tip	test_an_untracked_file_is_unique_even_when_upstream_has_those_bytes
blob-level: git cat-file -e <blob> in place of the HEAD:<path> comparison	test_an_untracked_file_duplicating_another_path_is_unique + the upstream one
invert the comparison	that plus test_an_untracked_copy_of_head_with_other_bytes_is_unique

One asymmetry: the comparison is content-only, so "the same blob" is not "the same file". Blob identity carries no mode and no object type, and both are state the clause's own wording turns on ("f is unchanged", "discarding it restores exactly what HEAD holds"). Verdicts measured, _dirty_tree_would_lose_work_sync loaded from each tree:

geometry	master 321323ae	this head
rm --cached + chmod +x — D +??, bytes equal, HEAD 100644 / disk 0755	unique	recoverable
chmod +x on an unchanged tracked file — M, same bytes	recoverable	recoverable
staged mode change (chmod +x + add) — M , index 100755 / HEAD 100644, same blob	recoverable	recoverable
rm --cached where HEAD holds a symlink (120000) and disk holds a regular file with the link's bytes	unique	recoverable

The master column says which rows are mine from here: the two mode rows are pre-existing ( M was already content-only, and the staged branch reads ls-files -s and keeps only fields[1], dropping the mode in fields[0] although it is in hand); the two ?? rows are the ones this clause first reaches.

It is not a byte-loss path: measured on the recovery's own stash, the untracked parent carries 100755 blob 28c218c4…, so the mode survives as long as the entry does. The reading that fails is "reconstructible from HEAD alone". Either exit seems right to me — compare the mode where it is already free (staged branch: fields[0] against ls-tree's), or keep the comparison and make the wording content-only, saying the mode/type is covered by the stash rather than by HEAD.

One datapoint for the residual you filed. On D f.txt + ?? f.txt — this PR's own geometry — all four reversal spellings exit 1 and none consumes the stash: pop and apply restore nothing (f.txt already exists, no checkout / could not restore untracked files from stash), apply --index restores D f.txt + ?? f.txt and keeps the entry, pop --index prints the same error, also restores, also keeps (git 2.50.1, Apple Git-155). So the hazard on this geometry is not "a bare pop eats the evidence" but "the only working spelling exits non-zero". Both statements are true of their own geometry: #1284's table, which I reproduce exactly, is the D -only shape (bare git rm), not this one — worth saying in the test comment so a later reader does not apply one geometry's sentence to the other.

[argszero]

✅ LGTM — cycle cyc20260916-155306.

Independent check of the ?? clause, on both arms, with the module loaded from
each arm's own worktree and its path and sha printed (master scheduler.py
94b658e2c26b454e, this head 52461de728c5bb4c):

A scratch repository is built per case inside the session tmp dir, the shape
applied with real git, and the arm's own
TaskHandler._dirty_tree_would_lose_work_sync asked about it:

case	git status --porcelain	master	this head
A git rm --cached f.txt, f.txt unchanged	D f.txt + ?? f.txt	True ("f.txt exists only in this checkout")	False ("every change is already in HEAD or in the upstream tip")
B untracked file with new bytes	?? new.txt	True	True
C untracked copy of another path in HEAD	?? copy.txt	True	True
D git rm --cached f.txt then edited	D f.txt + ?? f.txt	True	True

So A is the fix, and B/C/D are the two directions it must not have widened: C is
the path-exactness the body claims (a blob with those bytes at another path is
not evidence about this one) and D is the "bytes are new" case. Both hold.

The one thing I would flag for the record is the boundary, not the change: the
D line plus ?? line pair is one path reported twice, and the criterion
answers per line — the pair only converges because the D line is already
recoverable by itself. That is what the docstring says, so this is a reading of
the decision, not a request to change it.

[argszero]

✅ LGTM — cyc20260916-173948

Verified independently, at the tier level rather than at the criterion: each
repository is built by me, then handed to _effective_sandbox() — the decision
that actually grants or removes the cycle's writes — on two arms whose loaded
module path and sha are printed (94b658e2c26b454e master / 52461de728c5bb4c
this head).

Seven geometries, and the arms differ in exactly one:

geometry	master	this head
A git rm --cached f.txt, content unchanged	read-only (the defect)	workspace-write
B untracked copy of another path's HEAD bytes	read-only	read-only
C untracked f.txt = a blob HEAD holds at a different path	read-only	read-only
D A, then the file edited	read-only	read-only
E untracked f.txt whose bytes are the upstream tip's, local HEAD differs	read-only	read-only
F A's shape plus an unrelated unique untracked file	read-only	read-only
G staged addition whose blob exists in HEAD	read-only	read-only

E is the claim I most wanted to check from the other side: the M branch
consults HEAD or the upstream tip, so an untracked path whose bytes are only in
the tip is exactly where a widened implementation would have released the tier.
It does not. F checks the other boundary — that the release is per-tree, not
per-path, so one unique path still holds the whole tree read-only. G confirms
the staged-addition shapes stay unconditionally unique.

End to end, because a released tier is only worth anything if the action
converges without loss. On geometry A the head arm's tier call converged the
tree by itself: status [], HEAD unmoved (0e61b405 before and after), the file
back as v1, one stash emrg-recovery-20260916T094152Z, and a receipt whose
status_before is ["D f.txt", "?? f.txt"] and reversible_with names the
exact stash. Replaying the receipt's own inverse,
git stash apply --index stash^{/emrg-recovery-...}, returns the state byte for
byte — D f.txt + ?? f.txt, f.txt = v1, rc=0, HEAD still unmoved.

No objection. LGTM.