ip65_hw_checks: 17 of 46 mutants survive — untested defence-in-depth branches, none of which changes a verdict today #201

What was measured

An adversarial review of #200 built a 46-mutant set against tools/ip65_hw_checks.py — larger than the 14 in the in-tree tools/mutate_ip65_hw_checks.py. 29 were caught; 17 survived.

The decisive part is what came next: applying each of the 17 survivors and re-running the "the C64 did nothing" adversarial capture shows that not one of them changes a single verdict. They are defence-in-depth branches that are subsumed at the call site — an input that would exercise them is already rejected by an earlier or a neighbouring check.

So this is not "the checker is wrong". It is "17 branches exist that nothing distinguishes", which is a maintenance hazard rather than a live defect: a future caller that reaches one of them directly gets behaviour no test pins.

Why it is not fixed in #200

The in-tree harness catches 13/13 of its own detectable set and closed four genuine coverage gaps when it was written (each one a red case that passed whether or not the checker worked). Extending it to 46 mutants and then either pinning or deleting the 17 unreachable branches is a separate, larger piece of work, and #200 is already a first-hardware-result PR carrying its own evidence burden.

Suggested shape

For each survivor, one of:

• pin it — add a red case that reaches the branch directly, if a caller could plausibly reach it;
• delete it — if it is genuinely unreachable given how the rig calls the function, say so and remove it, the way check_mac_on_wire's redundancy is now documented rather than silently carried;
• record it as known-equivalent — tools/mutate_ip65_hw_checks.py already has that category and reports it rather than suppressing it (check_http_response's length test is the existing example).

The third option is the one to reach for last: a growing known-equivalent list is how a mutation score stops meaning anything.

Related

• First real-hardware RR-Net end-to-end for the ip65 product: 15 independent hardware checks on real CS8900a silicon #200 — the PR this came out of.
• Sibling issue on defences dropped from the c64-wireguard reference implementation.