Repository navigation
Compare filesystem programs against a fixture CRuby can see (fixes #43) - #124
Open
jiro-by-nori[bot] wants to merge 2 commits into
Open
jiro-by-nori[bot] wants to merge 2 commits into
jiro-by-nori[bot] wants to merge 2 commits into
Conversation
Nothing exercised the model's filesystem against CRuby (issue #43). The model's files are boot-heap data, so a program that reads `/tmp/readme.txt` answers from the fixture on the model side and from whatever the host has on CRuby's side: the two were never the same program, and the whole VFS surface was validated by hand. The trailing-slash bug fixed in #42 is the kind of defect a corpus of these would have caught. A case now names its tree in its sidecar (`{"vfs": "v1"}`); `difftest/vfs.py` describes the tree the model boots with and materializes it before the case and restores it afterwards. Every CRuby run resets the tree first, so a program that writes still starts both runs of the determinism check from the same bytes, and the SUT is handed the tree the control started from rather than the control's last write. Materializing refuses if a fixture path already exists and restores everything it created, so the host's `/tmp` is never overwritten or departed from. The corpus is tier 5 (`difftest/corpus/vfs/`), run by `make conformance` with `difftest run --tier vfs --sut lean`. A gate is allowed there — the model's filesystem is built one step at a time, so a descriptor program gating is a coverage measurement, not a pass — but a disagreement, a control failure or an unusable fixture is not, and this tier owns that verdict rather than leaving it to a baseline. The first five programs pin the fixture's bytes and sizes, path resolution (`.`, `..`, `//`, trailing slash), a write read back, and the descriptor form that is still gated. Validation: `difftest run --tier vfs --sut lean` agrees on four programs and gates the descriptor one; `--sut identity` agrees on all five, which is the harness's own smoke test; the tier exits 1 on a fabricated disagreement and on an occupied fixture path, and leaves `/tmp` as it found it. `replay corpus/tier3` (70 programs) and `run --tier regressions` (222 programs) are unchanged: no disagreement, 221 held and one gated. `difftest/tests` passes in CI mode (204 passed, 0 skipped, including 15 new tests), `ruff check` and `ruff format --check` are clean, `mkdocs build --strict` passes, the difftest coverage baseline moves up (72.77/68.18 -> 73.27/68.67) and `make check-not-weakened` reports CHECKS NOT WEAKENED. The full `make conformance` target is left to CI: this machine has no C toolchain, so `make deps` (bundler and the pinned gems) cannot run here. Authored by Jiro, an autonomous agent (Nori sessions), on behalf of @sam-xif. Co-authored-by: Nori <contact@tilework.tech>
This was referenced Nov 10, 2026
`make conformance` treats "no disagreement" as a pass and has no baseline for this tier, so a corpus that loaded zero cases (a moved directory, a sidecar mistake) would print "the model and CRuby agree" having compared nothing. `load_vfs_corpus` now raises `VfsError` when the directory holds no cases, which `difftest run --tier vfs` reports and exits 2 on, and a unit test pins it. The difftest coverage baseline moves up again (73.27/68.67 -> 73.28/68.72). Authored by Jiro, an autonomous agent (Nori sessions), on behalf of @sam-xif. Co-authored-by: Nori <contact@tilework.tech>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #43.
What was wrong
Every filesystem program was a comparison of two different programs. The model's
files live in the boot heap (
ruby-lean/RubyCore/Heap.lean§Virtual filesystem),so
File.read("/tmp/readme.txt")answered"hello\n"on the model side whileCRuby read whatever the host had at that path — usually nothing. Nothing in
either corpus mentioned
File./Dir./IO., so the whole VFS surface wasvalidated by hand. The trailing-slash bug fixed in #42 is the kind of defect a
corpus of these programs would have caught.
The change
difftest/vfs.py: the fixture the model boots with (v1), and aMaterializedcontext that writes it to the real filesystem, restores it andcan reset it. It refuses to materialize over an existing path (so a user's
/tmp/readme.txtis never overwritten) and removes everything it created.runner.run_case: a case whose sidecar names a fixture ({"vfs": "v1"})materializes it for the case and restores it at the end, including on a
harness error. An unknown fixture name is a
harness_error, never a skip.control.CRubyRunner.run/run_deterministic: reset the fixture beforeevery CRuby run, so a program's writes cannot leak into the second run of
the determinism check. The SUT is handed the tree the control started from,
not the control's last write.
difftest/corpus/vfs/: tier 5 — five programs pinning the fixture's bytes andsizes, path resolution (
.,..,//, trailing slash), a write read back,and the descriptor form that is still gated. Sidecars carry
{"vfs": "v1"}.difftest run --tier vfs --sut leanruns the corpus whole and owns its ownverdict: a gate is allowed (the model's filesystem is built one step at a
time, so a gated step is a coverage measurement, not a pass), but a
disagreement, a control failure or an unusable fixture exits 1. There is no
baseline to absorb a silent one.
VfsError, exit 2), not a silent pass: with nobaseline for this tier, a moved directory would otherwise report "the model
and CRuby agree" having compared nothing.
make conformanceruns it (Makefile— one line in the conformance target,plus the comment above it; this file needs the owner's review and it is the
only owner-owned file touched, no other changes are in it).
docs/testing.md,difftest/README.md,difftest/corpus/vfs/README.mddescribe the fixture and the rules for writing a case. The difftest coverage
baseline moves up with the new module and tests.
Test plan
uv run difftest run --tier vfs --sut lean→ 4 agree, 1sut_unsupported(the descriptor program), 0 disagree; run twice,
/tmpleft as found.uv run difftest run --tier vfs --sut identity→ 5 agree. CRuby againstCRuby through the fixture is the harness's own smoke test: it would fail if
materialization differed from the model's tree.
Dir.exist?("/tmp/readme.txt"), which the modelcurrently answers wrongly — filed separately) → exit 1 and the case is listed
under
unexpected; occupied fixture path → exit 1 with a message naming thepath and the user's file untouched; missing sidecar → exit 2.
replay corpus/tier3→ 57 agree, 13 gated, 0 disagree;run --tier regressions→ 221 held, 1 gated, 0 regressed.DIFFTEST_NO_SKIPS=1 uv run pytest→ 204 passed, 0 skipped (15 new tests);ruff check+ruff format --checkclean;mkdocs build --strictclean;scripts/check-coverage.pyOK at the raised baseline;make check-not-weakened→ CHECKS NOT WEAKENED.make conformance/make check(this machine has noC toolchain, so
make depscannot install the pinned gems). CI runs it.Found while writing the corpus (filed separately)
Dir.exist?answersFile.exist?:Dir.exist?("/tmp/readme.txt")istrueinthe model and
falsein CRuby. The new corpus caught it on the first run; thefix is in
RubyCore/Builtins/FS.leanand is not in this PR (a model changeneeds the proof books, and this PR is the harness). The corpus case will land
with that fix.
Authored by Jiro, an autonomous agent (Nori sessions), on behalf of
@sam-xif. The code and this description were written by the agent.