Skip to content

Compare filesystem programs against a fixture CRuby can see (fixes #43) - #124

Open
jiro-by-nori[bot] wants to merge 2 commits into
mainfrom
jiro/vfs-difftest-corpus
Open

jiro-by-nori[bot] wants to merge 2 commits into
mainfrom
jiro/vfs-difftest-corpus

Conversation

@jiro-by-nori

@jiro-by-nori jiro-by-nori Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Closes #43.

What was wrong

Every filesystem program was a comparison of two different programs. The model's
files live in the boot heap (ruby-lean/RubyCore/Heap.lean §Virtual filesystem),
so File.read("/tmp/readme.txt") answered "hello\n" on the model side while
CRuby read whatever the host had at that path — usually nothing. Nothing in
either corpus mentioned File./Dir./IO., so the whole VFS surface was
validated by hand. The trailing-slash bug fixed in #42 is the kind of defect a
corpus of these programs would have caught.

The change

  • difftest/vfs.py: the fixture the model boots with (v1), and a
    Materialized context that writes it to the real filesystem, restores it and
    can reset it. It refuses to materialize over an existing path (so a user's
    /tmp/readme.txt is never overwritten) and removes everything it created.
  • runner.run_case: a case whose sidecar names a fixture ({"vfs": "v1"})
    materializes it for the case and restores it at the end, including on a
    harness error. An unknown fixture name is a harness_error, never a skip.
  • control.CRubyRunner.run / run_deterministic: reset the fixture before
    every CRuby run, so a program's writes cannot leak into the second run of
    the determinism check. The SUT is handed the tree the control started from,
    not the control's last write.
  • difftest/corpus/vfs/: tier 5 — five programs pinning the fixture's bytes and
    sizes, path resolution (., .., //, trailing slash), a write read back,
    and the descriptor form that is still gated. Sidecars carry {"vfs": "v1"}.
  • difftest run --tier vfs --sut lean runs the corpus whole and owns its own
    verdict: a gate is allowed (the model's filesystem is built one step at a
    time, so a gated step is a coverage measurement, not a pass), but a
    disagreement, a control failure or an unusable fixture exits 1. There is no
    baseline to absorb a silent one.
  • An empty corpus is an error (VfsError, exit 2), not a silent pass: with no
    baseline for this tier, a moved directory would otherwise report "the model
    and CRuby agree" having compared nothing.
  • make conformance runs it (Makefile — one line in the conformance target,
    plus the comment above it; this file needs the owner's review and it is the
    only owner-owned file touched, no other changes are in it).
  • docs/testing.md, difftest/README.md, difftest/corpus/vfs/README.md
    describe the fixture and the rules for writing a case. The difftest coverage
    baseline moves up with the new module and tests.

Test plan

  • uv run difftest run --tier vfs --sut lean → 4 agree, 1 sut_unsupported
    (the descriptor program), 0 disagree; run twice, /tmp left as found.
  • uv run difftest run --tier vfs --sut identity → 5 agree. CRuby against
    CRuby through the fixture is the harness's own smoke test: it would fail if
    materialization differed from the model's tree.
  • Fabricated disagreement (Dir.exist?("/tmp/readme.txt"), which the model
    currently answers wrongly — filed separately) → exit 1 and the case is listed
    under unexpected; occupied fixture path → exit 1 with a message naming the
    path and the user's file untouched; missing sidecar → exit 2.
  • No regression in the existing corpora through the refactored runner:
    replay corpus/tier3 → 57 agree, 13 gated, 0 disagree; run --tier regressions → 221 held, 1 gated, 0 regressed.
  • DIFFTEST_NO_SKIPS=1 uv run pytest → 204 passed, 0 skipped (15 new tests);
    ruff check + ruff format --check clean; mkdocs build --strict clean;
    scripts/check-coverage.py OK at the raised baseline;
    make check-not-weakened → CHECKS NOT WEAKENED.
  • Not run locally: the full make conformance/make check (this machine has no
    C toolchain, so make deps cannot install the pinned gems). CI runs it.

Found while writing the corpus (filed separately)

Dir.exist? answers File.exist?: Dir.exist?("/tmp/readme.txt") is true in
the model and false in CRuby. The new corpus caught it on the first run; the
fix is in RubyCore/Builtins/FS.lean and is not in this PR (a model change
needs the proof books, and this PR is the harness). The corpus case will land
with that fix.


Authored by Jiro, an autonomous agent (Nori sessions), on behalf of
@sam-xif. The code and this description were written by the agent.

Nothing exercised the model's filesystem against CRuby (issue #43). The model's
files are boot-heap data, so a program that reads `/tmp/readme.txt` answers from
the fixture on the model side and from whatever the host has on CRuby's side:
the two were never the same program, and the whole VFS surface was validated by
hand. The trailing-slash bug fixed in #42 is the kind of defect a corpus of
these would have caught.

A case now names its tree in its sidecar (`{"vfs": "v1"}`); `difftest/vfs.py`
describes the tree the model boots with and materializes it before the case and
restores it afterwards. Every CRuby run resets the tree first, so a program that
writes still starts both runs of the determinism check from the same bytes, and
the SUT is handed the tree the control started from rather than the control's
last write. Materializing refuses if a fixture path already exists and restores
everything it created, so the host's `/tmp` is never overwritten or departed
from.

The corpus is tier 5 (`difftest/corpus/vfs/`), run by `make conformance` with
`difftest run --tier vfs --sut lean`. A gate is allowed there — the model's
filesystem is built one step at a time, so a descriptor program gating is a
coverage measurement, not a pass — but a disagreement, a control failure or an
unusable fixture is not, and this tier owns that verdict rather than leaving it
to a baseline. The first five programs pin the fixture's bytes and sizes, path
resolution (`.`, `..`, `//`, trailing slash), a write read back, and the
descriptor form that is still gated.

Validation: `difftest run --tier vfs --sut lean` agrees on four programs and
gates the descriptor one; `--sut identity` agrees on all five, which is the
harness's own smoke test; the tier exits 1 on a fabricated disagreement and on
an occupied fixture path, and leaves `/tmp` as it found it. `replay corpus/tier3`
(70 programs) and `run --tier regressions` (222 programs) are unchanged: no
disagreement, 221 held and one gated. `difftest/tests` passes in CI mode (204
passed, 0 skipped, including 15 new tests), `ruff check` and `ruff format
--check` are clean, `mkdocs build --strict` passes, the difftest coverage
baseline moves up (72.77/68.18 -> 73.27/68.67) and `make check-not-weakened`
reports CHECKS NOT WEAKENED. The full `make conformance` target is left to CI:
this machine has no C toolchain, so `make deps` (bundler and the pinned gems)
cannot run here.

Authored by Jiro, an autonomous agent (Nori sessions), on behalf of @sam-xif.

Co-authored-by: Nori <contact@tilework.tech>
`make conformance` treats "no disagreement" as a pass and has no baseline for
this tier, so a corpus that loaded zero cases (a moved directory, a sidecar
mistake) would print "the model and CRuby agree" having compared nothing.
`load_vfs_corpus` now raises `VfsError` when the directory holds no cases, which
`difftest run --tier vfs` reports and exits 2 on, and a unit test pins it.

The difftest coverage baseline moves up again (73.27/68.67 -> 73.28/68.72).

Authored by Jiro, an autonomous agent (Nori sessions), on behalf of @sam-xif.

Co-authored-by: Nori <contact@tilework.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The VFS builtins have no differential-test coverage: nothing in the difficulty corpora runs File/Dir/IO against CRuby

0 participants