Repository navigation
Measure generated test quality, evaluate on two libraries with two models, add research paper - #20
Merged
Merged
Conversation
…curacy) Every 'code2test test' run now records acceptance rate, first-run pass rate, repairs, xfails and drops (.code2test/metrics.json) and shows them against the MVP targets from docs/proposal.md. 'code2test benchmark' measures generated test quality: it injects single-site mutants into tested functions (always restoring the source) and reports the share the tests catch, and it scores the diagnosis agent on failures with known causes (mutated code -> CODE_BUG, deliberately broken expected values -> TEST_WRONG). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
- Prompts show up to 16k chars of source (was 6k) plus the bodiless API of the component's module and the project modules it imports, and ask the model to trace expected values instead of guessing. Found on TinyDB: the model built TinyDB(storage=MemoryStorage()) without seeing the API. - Agents allow 3 output retries (MiniMax sometimes needs a second try at valid structured output); generation errors name the component. - Logs show the actual error line instead of pytest's 'collection failure'. - Benchmark mutations no longer leave tell-tale '_mutated'/'_wrong' markers, and diagnosis cases are sampled across all killed mutants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
…irs start Real-model runs on TinyDB showed fixtures with literal \" and \n mixed into multi-line code, which failed every test in the file on the first run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
…etrics Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
… TinyDB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
No emojis or marketing language; one clear structure (requirements, installation, model choice, quick start, how it works, command reference, files written, results, limitations, development). Removes claims the code does not support (integration test suggestions, editing tests in the prompt, coverage reports) and a link to a missing document. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
Benchmark: - Restore mutated sources byte for byte (CRLF files were rewritten) and keep an on-disk backup per mutant; files a killed run left mutated are restored on the next run, and SIGTERM/SIGHUP now run the restore. - Don't mutate type annotations (unkillable mutants lowered the score). - TEST_WRONG probes keep the rest of the file (sibling helpers/fixtures) and only count when the changed assertion is what fails. - A failed diagnosis call is unscored instead of counting as TEST_WRONG; an unreachable model keeps the mutation results. - CODE_BUG cases only when the mutated line is within the source the diagnosis agent sees (now 16k chars, was 2k). - All test files of a component run against its mutants; components outside the repo are skipped; --timeout applies to the baseline run; per-cause counts exclude unscored cases; empty-string asserts skipped. Metrics: - Unchanged re-runs count already-verified components instead of reporting 0% acceptance; low-confidence skips are reported separately. - Skipped tests don't count as passing; acceptance needs a real pass. - One syntax error no longer hides the first-run result of the other tests in its file. - Rates are n/a, not 0%, when nothing was generated; xfail and passing counts follow tests dropped by the whole-suite check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
…README corrections The linked docs described Jest/JUnit generation, a template engine, a sandboxed verifier and Python 3.9, none of which match the code. README: Google key model selection, --dry-run still calls the model, test file name for package __init__ modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
A TinyDB run left a 'some/path' file in the project: an unrepaired generated test wrote a relative path. pytest now runs from a throwaway working directory with --rootdir at the repository (imports, config and test ids unchanged), and the prompt asks for tmp_path for any file the code creates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
Standard mutation testing runs the full suite. Running only the mutated component's own tests undercounted kills, e.g. inflection's _irregular() runs at import and its bugs break pluralize/singularize tests: 32 of 34 'survivors' in one run. The per-component number is still reported as a secondary metric, and only own-test failures become diagnosis cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
research/paper: LaTeX source of the paper (tool and experience report), TikZ pipeline figure, verified bibliography, and a Makefile that builds the PDF and the arXiv bundle (with main.bbl, since arXiv does not run BibTeX). Every measured number comes from results.tex, which collect_results.py generates from the run data; abstract.txt is the plain-text abstract for the arXiv form. research/data: for each of the four runs (inflection and TinyDB, MiniMax Flash and M3) the generated test suite, inferred intents (and whether the model re-inferred them), run metrics, benchmark output and verify log. data/development holds the earlier runs cited for run-to-run variance. research/scripts: run_experiments.sh reproduces the runs; collect_results.py builds data/, results.tex and abstract.txt, computing rates from counts and rounding half up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
…aper The results table now shows all four final runs (two libraries, two models) and explains where the suites are weak. The quickstart sample output matches the final inflection run, and docs/real_model_run.md is marked as a record of the first development runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2
pro-utkarshM
changed the title
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds measurement of Code2Test against the MVP targets in
docs/proposal.md, evaluates it on two open-source libraries with two models, fixes what those runs exposed, and adds an arXiv-ready paper with its data. v1 is scoped to Python.Results (final runs, one per library and model)
code2test verifyModels: MiniMax
MiniMax-M3.1-Flash-Preview(Flash) andMiniMax-M3(M3). Every run met all three targets. Half of inflection's sampled mutants fall in_irregular, which the Flash run tested only by counting inserted rules; most TinyDB survivors change string literals.What changed
Measurement
code2test test --autorecords acceptance rate, first-run pass rate, repairs, xfails, drops and reused components in.code2test/metrics.json.code2test benchmark: mutation score (each single-site mutant against the whole generated suite, plus the mutated component's own tests), and diagnosis accuracy on failures of known cause. Sources are restored byte-exactly, including after SIGTERM or a killed run.Fixes found with a real model
tmp_path, so they cannot write into the user's repository.Research (
research/)paper/: LaTeX paper (tool and experience report), figure, bibliography, Makefile for the PDF and the arXiv bundle. Measured numbers come from generated macros.data/: per run, the generated test suite, inferred intents, metrics, benchmark output and verify log.scripts/: reproduce the runs and regenerate the data and macros.Docs
Test plan
pytest: 111 passed🤖 Generated with Claude Code
https://claude.ai/code/session_01VGBhCd3VqGRk2KLZg1Tqu2