Skip to content

t27b: per-test reference differential -- mismatch 0 today compares the JIT with an interpreter over the same IR #6441

Description

@gHashTag

Refs #6434
Refs #6063

Problem

"mismatch 0" on the t27b lab is not a comparison with the reference path.

  • In cli/t27b/src/main.rs (the --check block), MISMATCH means the JIT disagrees with eval.rs::Interp. Both read the same IR from lower.rs. If lower.rs mistranslates t27 semantics (wrap vs trap, promotion, float rounding, layout), both sides compute the same wrong thing and agree.
  • The only comparison with the reference is per file. blockers.rs::parse_test_report keeps only the tests and FAIL totals of t27c test-report, even though t27c already prints each test's outcome (bootstrap/src/test_report.rs, Report.outcomes).
  • steward.t27 is_alarm(fail, fail) is false, so the 7 files where both fail (lab run 03362c1) count as agreement without checking that the same test failed. Example: specs/port/trinity/src/tri/gen_vector2d.t27: t27b fails Vec2_length with assert_eq left 4627730092099942521 vs right 4627730092099895296 (one f64 ulp). The reference says only "1 of 4 tests fail".
  • Stop::Fuel and Stop::Depth are also counted as agreement whatever the JIT did.

Change

  1. parse_test_report returns the per-test outcome list. The reference cache stores it (new row format; old rows are read as file-level only).
  2. t27b corpus --reference compares test by test. A test whose t27b verdict differs from the reference verdict is ref_mismatch, a new verdict counted in summary.ref_mismatch and treated by the doctor as LAB-MISMATCH.
  3. steward.t27 is_alarm judges per test when per-test data exists.
  4. Interpreter Fuel/Depth stops report unknown, not agreement.

Done when

  • The lab JSON carries ref_mismatch per file and in the summary, and the 7 fail/fail files show which tests disagree, if any.
  • steward.t27 has tests for the per-test is_alarm.
  • New subcommand tri t27b diff <spec> prints both verdict lists side by side.

🤖 Generated with Claude Code

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority/highMust fix before next milestone

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions