Skip to content

Behavioral coverage in reports, lint/format/3.13 in CI, second runs in the paper - #22

Merged
pro-utkarshM merged 5 commits into
masterfrom
claude/amazing-johnson-ehm81c
Oct 15, 2026
Merged

Behavioral coverage in reports, lint/format/3.13 in CI, second runs in the paper#22
pro-utkarshM merged 5 commits into
masterfrom
claude/amazing-johnson-ehm81c

Conversation

@pro-utkarshM

@pro-utkarshM pro-utkarshM commented Oct 15, 2026

Copy link
Copy Markdown
Member

Summary

Closes #13 and #15, and adds the second experiment runs to the paper.

#15: behavioral coverage and diagnostic accuracy in reports

code2test report coverage (previously "not implemented") and the code2test test --report html|json output now report test quality per component:

  • Behaviors tested: the distinct behaviors the component's passing tests check (as stated for each generated test), next to its inferred intent.
  • Mutants killed: the share of injected bugs in that component the suite catches. benchmark.json now records mutants per component and when the benchmark ran.
  • Diagnostic accuracy: overall and per expected cause, from the last code2test benchmark.

The report says when that benchmark predates the current tests, and how to get the numbers when none has run. --format html|json writes tests/reports/coverage.*. The Intent Alignment Score from the proposal is not implemented; it needs semantic similarity, and is listed in docs/task.md.

#13: CI matrix, lint, format check

  • The test job runs on Python 3.12 and 3.13. A new lint job runs ruff check and ruff format --check.
  • ruff uses an explicit error-level rule set (E4, E7, E9, F) and is pinned to 0.17.x, so CI does not change with ruff releases. black is dropped in favor of ruff format.
  • All lint findings are fixed (separate commit, no behavior change). The code is reformatted with ruff format (separate commit). Each reformatted file's AST is unchanged apart from docstrings, and every command's --help output is identical.

Paper: second runs and the campus defect

  • A second run of each configuration; paper Section 5.5 and Table 3 compare the two runs.
  • A real defect found by the tool: inflection 0.5.1 singularizes "campuses" to "campuse". The test that exposed it is now Listing 1.
  • The data is in research/data/second-runs/.

Test plan

  • pytest: 117 passed with pydantic-ai 2.31.1 and with 1.0.6 (Python 3.13 locally; CI adds 3.12).
  • ruff check and ruff format --check clean.
  • code2test report coverage checked on a real generated suite (text, JSON, HTML) and after a fresh benchmark (per-component mutants).
  • Paper builds cleanly; arXiv bundle compiles standalone.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2

Generated by Claude Code

claude added 5 commits October 15, 2026 14:50
@claude
research: second runs of every configuration, and the campus defect
b65d98b 
Each configuration (inflection and TinyDB, MiniMax Flash and M3) was run
a second time. The paper now reports both runs (new Section 5.5 and
Table 3) instead of one run per configuration; all second runs met the
three MVP targets again.

In the second run of inflection with M3, Code2Test kept one test as an
expected failure, and it exposes a real defect: inflection 0.5.1
singularizes "campuses" to "campuse" (and viruses, bonuses, censuses
the same way). The paper describes it, Listing 1 now shows this real
test instead of an illustrative one, and the abstract, RQ1, threats and
conclusion are updated. The paper no longer claims both libraries are
correct.

- research/data/second-runs/: the four runs (tests, intents, metrics,
  benchmark output, verify logs), with a README on how they were made.
- collect_results.py takes an optional second runs directory; macros for
  second runs carry a "Second" tag; results.csv gains a `run` column.
  The first runs' data regenerates byte-identically.
- run_experiments.sh runs every configuration twice.
- README results mention the second runs and the defect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
@claude
feat: behavioral coverage and diagnostic accuracy in reports (#15)
35e5a76 
`code2test report coverage` (until now "not implemented") and the
`code2test test --report html|json` output report test quality per
component:

- behaviors tested: the distinct behaviors that the component's passing
  tests check, as stated for each generated test, next to its intent;
- mutants killed: the share of injected bugs in the component that the
  generated suite catches;
- diagnostic accuracy, overall and per expected cause.

The last two come from the most recent `code2test benchmark` run, which
now records mutants run and caught per component and when it ran. The
report says when that run predates the current tests, and how to get
the numbers when no benchmark has run. `--format html|json` writes to
tests/reports/coverage.*; text prints a table.

is_xfail/passing_count move from the generator to core/models so the
report can use them without importing the agents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
@claude
chore: lint with ruff (errors only) and fix every finding (#13)
1061ccf 
pyproject selects ruff's error rules explicitly (E4, E7, E9, F) so CI
does not change with ruff releases, pins ruff to 0.17.x, and drops
black, which ruff format replaces.

Fixes: unused imports and f-strings without placeholders (ruff --fix),
unused variables (display labels that were never shown, counters never
read, an unused path), bare `except:` -> `except Exception:`, an
ambiguous variable name, a lambda assignment, and CLI command imports
moved to the top of cli/main.py. No behavior changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
@claude
style: format the code with ruff format (#13)
b6449f9 
Mechanical: `ruff format code2test tests research/scripts`. Checked
before committing: every file's AST is unchanged apart from docstrings,
and every command's --help output is identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
@claude
ci: test on Python 3.12 and 3.13; check lint and formatting (#13)
f20d51f 
A lint job runs `ruff check` and `ruff format --check` on code2test,
tests and research/scripts with the pinned ruff 0.17.x; the test job
runs on Python 3.12 and 3.13 (fail-fast off). README and docs/task.md
describe the checks; pyproject lists Python 3.13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
greptile-apps[bot]
greptile-apps Bot reviewed Oct 15, 2026
View reviewed changes

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@pro-utkarshM
pro-utkarshM merged commit 075623c into master Oct 15, 2026
6 checks passed
This was referenced Oct 15, 2026
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[P1] GitHub Actions CI (install + lint + test matrix)

2 participants

@pro-utkarshM @claude