Repository navigation
difftest: cover the VFS step-1 builtins; fix Dir.exist? on a file - #85
Open
jiro-by-nori[bot] wants to merge 1 commit into
Open
jiro-by-nori[bot] wants to merge 1 commit into
jiro-by-nori[bot] wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Reviewed head b7582f2.
Checks observed (all success at this head): CI, checks not weakened, docs, lint, front-end (the desugarer against CRuby), conformance (the model against CRuby), proofs (build, every book, axioms, soundness, comparator). No unresolved review threads.
Files reviewed (all 25): the seven difftest/corpus/vfs/* programs and sidecars; difftest/difftest/{vfs.py,control.py,runner.py,sources.py}; difftest/tests/{marks.py,test_vfs_corpus.py,test_sorbet_check.py,coverage-baseline.json}; ruby-lean/RubyCore/Builtins/FS.lean; difftest/README.md; docs/testing.md.
Validation:
- The session has no CRuby, Lean, or
uv, so I did not run the full differential suite locally; theconformance,front-end, andproofsjobs cover the model build and CRuby comparison. - Ran a standalone Python harness against the PR head's
difftest/vfs.pyanddifftest/sources.py: the corpus loads all 7 cases, every sidecar names a known fixture, all fixture paths are absolute, andload_vfs_corpusrejects an unknown fixture at load time. - Exercised
materialized()directly: an overwritten pre-existing file is restored byte-for-byte, files it created are removed, directories it created are removed, a directory the program wrote into is kept,Noneis a no-op, and two consecutive materialize/write cycles leave no residue (the property case 002 relies on for its determinism double-run). - Compared
VFS_FIXTURES["step1"]with the boot fixture inRubyCore/Heap.lean(lines ~680-690):/tmp/readme.txt=hello\n,/tmp/vfsdata/nums.txt=1\n2\n3\n,/tmp/vfsdata,/tmp, and/all match, so the control's tree is the model's tree. - Traced
VFS.resolvefor the corpus path shapes: trailing slash on a file yieldsnonewhile on a directory it resolves,/../tmp/...stays at the root, and""/relative paths yieldnone, matching every expectation in 000, 001, and 004. - Confirmed the
Dir#exist?split now requires a.dirpayload. Case 003 assertsDir.exist?("/tmp/readme.txt")is false; under the old sharedFile#exist?arm it would be true and the case would DISAGREE with CRuby, so the corpus is a real regression test for the fix. - Confirmed the moved
requires_rubycoremarker uses the same binary path as before (LeanSUT().lean_binandFragmentChecker().lean_binboth resolve toruby-lean/.lake/build/bin/rubycore), so skip behavior is unchanged andmake difftest-test'sDIFFTEST_NO_SKIPS=1still turns a missing binary into a failure. - Scope: one commit on top of current
main, 25 files / 372 changed lines (at, not over, the file cap), no human-gated paths, and the coverage baseline is raised, not lowered.
Residual risk:
- The fixture lives at fixed absolute
/tmppaths, so two concurrent pytest workers or two concurrent runs on one machine would race and could restore each other's bytes. The suite is single-process today (no xdist indifftest/pyproject.toml), so this is latent; worth revisiting if the suite is parallelized. - The 006 sidecar describes
File.sizeon a directory as anErrno::EISDIRgate, but CRuby'sFile.sizeon a directory returns the stat size rather than raising. This does not change the verdict (the case declaressut_unsupportedeither way), but the wording overstates CRuby's behavior. - I did not execute the CRuby/Lean differential replay locally; that rests on the green
conformanceandfront-endjobs.
Verdict: approve.
This was referenced Nov 6, 2026
`Dir.exist?` shared the `File.exist?` arm, so it answered true for a regular file — a wrong answer, not a gate. Split the arm and check that the resolved object is a directory. Add `difftest/corpus/vfs/`, an advisory corpus with versioned fixtures. A sidecar names a fixture; control runs materialize it under /tmp and restore the host's files, including between the determinism double-run. The corpus covers reads, writes, predicates, path components, and the shapes step 1 must gate. Authored by Jiro, an autonomous agent. Co-authored-by: Nori <contact@tilework.tech>
jiro-by-nori
Bot
force-pushed
the
vfs-differential-corpus
branch
from
November 8, 2026 23:15
b7582f2 to
caffc59
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The symbolic filesystem's step-1 builtins had no differential-test coverage (issue #43). Adding a corpus for them immediately found a wrong answer:
Dir.exist?returnedtruefor a regular file. It shared theFile.exist?arm, which only asks whether the path resolves. Split the arm soDir.exist?checks the resolved payload is a directory, as CRuby does.File.exist?is unchanged.The corpus and harness:
difftest/corpus/vfs/, an advisory corpus outside tier 0 so it cannot move the semantics watermark.difftest/vfs.py. A sidecar names one ("vfs": "step1"); the control materializes the same tree under/tmpbefore each run, restores any pre-existing files afterwards, and cleans up directories it created. Resetting per run keeps a writing program deterministic across the double-run.run_casefor both the control and any SUT that executes under CRuby (identity, desugar); the Lean model carries its own copy in the heap.size, all three predicates,Dir.exist?, duplicate separators,./.., trailing slashes, relative paths, and the shapes step 1 must gate (Errnois step 7).requires_rubycoremarker intodifftest/tests/marks.pyso the new and existing tests use one definition. The skip count is unchanged.Scope note: steps 2–5 of the VFS (descriptors,
Dir.entries,gets/pos/rewind) are in open PRs #26, #30, #34 and #36, so their cases land with those changes. This PR covers every filesystem builtin currently onmain.Authored by Jiro, an autonomous agent (code and this description).
Test plan
cd ruby-lean && lake build rubycore— builds.cd difftest && uv run pytestwith the pinned CRuby/Sorbet — 173 passed.cd difftest && uv run difftest replay corpus/vfs --sut lean— 5 agree, 2 sut_unsupported, 0 disagree. The two gate cases assert that missing-file reads and directoryread/size/writedecline rather than guess.cd difftest && uv run difftest replay corpus/tier3 --sut lean— 57 agree, 13 sut_unsupported, 0 disagree (no regressions).python3 scripts/check-not-weakened.py --base origin/main— CHECKS NOT WEAKENED.scripts/check-coverage.py difftest— line 73.63%, branch 69.17% (previous 72.77% / 68.18%); baseline updated,difftest/vfs.pyis 100% covered.ruff format --checkandruff checkon the changed Python — clean.Addresses #43. The descriptor coverage in that issue's proposed work lands with the still-open VFS step PRs (#26, #30, #34, #36); this PR covers every filesystem builtin on
maintoday.