[P2] Behavioral-coverage & diagnostic-accuracy metrics in reporting #15

Activity

1. pro-utkarshM

   added this to the v0.1 completion milestone on May 25, 2026
2. pro-utkarshM

   added

   P2Validation / polishValidation / polish

   on May 25, 2026
3. pro-utkarshM commented on Oct 15, 2026

   

   pro-utkarshM

   on Oct 15, 2026 – with Claude

   MemberAuthor

   More actions

   Status: diagnostic accuracy is measured, but not in the reports this issue asks for.

   • Diagnostic accuracy. code2test benchmark computes it on failures of known cause: mutant-induced failures are expected to be CODE_BUG, tests with a perturbed expected value TEST_WRONG. It is written to .code2test/benchmark.json and shown in the benchmark table, not in the --report html|json reports.
   • Behavioral coverage. Not implemented. The closest measure today is mutation score, from code2test benchmark.
   • Intent Alignment Score (proposal §7.1). Not implemented.

   Keeping this open. Remaining work: compute behavioral coverage, and add it and the latest diagnostic accuracy to the HTML/JSON reports.

   ---

   Generated by Claude Code
4. pro-utkarshM

   added a commit that references this issue on Oct 15, 2026

   feat: behavioral coverage and diagnostic accuracy in reports (#15)

   ...

   35e5a76
5. pro-utkarshM commented on Oct 15, 2026

   

   pro-utkarshM

   on Oct 15, 2026 – with Claude

   MemberAuthor

   More actions

   Done in #22.

   code2test report coverage (previously "not implemented") and the code2test test --report html|json output now include, per component:

   • Behavioral coverage, in two forms:
     • Behaviors tested: the distinct behaviors the component's passing tests check, as stated for each generated test, shown next to its inferred intent.
     • Mutants killed: the share of injected bugs in the component the suite catches. code2test benchmark now records mutants per component and when it ran.
   • Diagnostic accuracy, overall and per expected cause, from the last benchmark. That is where ground truth exists: failures of known cause.
   • Shown in the HTML and JSON reports, and in report coverage --format text|html|json.

   The report flags a benchmark that predates the current tests, and says how to get the numbers when none has run. Code is in code2test/core/quality.py; tests are in tests/test_reporting_and_models.py and tests/test_cli.py.

   Not covered: the proposal's Intent Alignment Score (§7.1), which needs semantic similarity between assertions and intents. It's listed under "Not done" in docs/task.md.

   ---

   Generated by Claude Code
6. pro-utkarshM

   closed this as completedon Oct 15, 2026