Release 0.1.0: project settings, real-model fixes, PyPI publishing

.github/workflows/publish.yml

@@ -0,0 +1,48 @@
+name: Publish to PyPI
+
+# Runs when a GitHub release is published. Uploads with PyPI trusted publishing:
+# no token is stored in the repository. One-time setup on pypi.org: add a trusted
+# publisher for project "code2test", repository Cirkitly/code2test, workflow
+# "publish.yml" and environment "pypi".
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Check that the release tag matches the package version
+        run: |
+          version=$(sed -n 's/^__version__ = "\(.*\)"$/\1/p' code2test/__init__.py)
+          if [ "v$version" != "$GITHUB_REF_NAME" ]; then
+            echo "Release tag $GITHUB_REF_NAME does not match package version $version"
+            exit 1
+          fi
+      - name: Build and check
+        run: |
+          python -m pip install --upgrade build twine
+          python -m build
+          twine check dist/*
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write  # trusted publishing
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

CHANGELOG.md

@@ -0,0 +1,41 @@
+# Changelog
+
+All notable changes to Code2Test are listed here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow [Semantic Versioning](https://semver.org/).
+
+## [0.1.0] - 2026-10-15
+
+First release: an alpha for Python projects.
+
+### Added
+
+- `code2test test` generates a pytest suite for existing Python code. For each top-level function and class it infers an intent from the docstring, type hints, naming and call sites, and asks the model when that inference is uncertain. It then generates tests, runs them, repairs failures from their pytest output (up to two rounds), and diagnoses tests that still fail as a wrong test, a code bug or a wrong intent. Suspected code bugs are kept as `xfail` tests that name the bug; everything else that fails is dropped, and a whole-suite run removes tests that only fail alongside others. The written suite therefore always passes.
+  - Interactive by default: review, edit or skip each intent and set of tests. `--auto` runs without prompts, and `--exit-code` makes it usable in CI.
+- `code2test verify` runs the generated suite; `--diagnose` explains failures and `--fix` repairs generated tests after the code changed, without touching tests you edited.
+- `code2test benchmark` measures the suite: the share of injected bugs it catches (mutation score), and how often the diagnosis names the right cause of a failure whose cause is known.
+- `code2test report coverage` and `code2test test --report html|json|all` report, per component, the behaviors its passing tests check, the injected bugs they catch and the diagnostic accuracy.
+- `code2test intent show|edit|export` to review and correct intents. Edited intents are never overwritten by later runs.
+- `code2test init` saves project defaults for `code2test test`: confidence threshold, output directory and exclude patterns.
+- `code2test config set|show|validate` stores the API key and model; `validate` sends the model one test request.
+- Works with any model pydantic-ai supports, including OpenAI-compatible endpoints through `OPENAI_BASE_URL`.
+
+### Measured
+
+On inflection 0.5.1 and TinyDB 4.8.2, with their own tests removed, using MiniMax `MiniMax-M3.1-Flash-Preview` and `MiniMax-M3`, two runs each:
+
+| Measure | Result | Target |
+| --- | --- | --- |
+| Components with passing tests | 96–100% | 70% |
+| Generated tests passing before any repair | 84–99% | 80% |
+| Injected bugs caught | 59–84% | |
+| Diagnosis correct on failures with a known cause | 97–100% | 75% |
+
+One run kept a test as an expected failure that exposes a real defect in inflection 0.5.1: `singularize("campuses")` returns `"campuse"`. Details are in the README and in [`research/`](research/).
+
+### Known limitations
+
+- Python only; files in other languages are analyzed for the dependency graph but get no tests.
+- Each run sends source code to the configured model provider and costs model calls.
+- Evaluated on two small libraries with one provider's models; large codebases and other models are not yet measured.
+- Not yet available: Markdown reports, and regenerating tests only for code that changed.
+
+[0.1.0]: https://github.com/Cirkitly/code2test/releases/tag/v0.1.0

README.md

@@ -29,12 +29,12 @@ On two open-source libraries whose own tests were removed, Code2Test produced pa
 ## Installation
 
 ```bash
-git clone https://github.com/Cirkitly/code2test.git
-cd code2test
-pip install .
-code2test --help
+pip install code2test
+code2test --version
 ```
 
+To install a release straight from GitHub instead, use `pip install git+https://github.com/Cirkitly/code2test@v0.1.0`. To work on Code2Test itself, see [Development](#development). Changes in each release are listed in [CHANGELOG.md](CHANGELOG.md).
+
 ## Choosing a model
 
 Code2Test picks the model in this order:
@@ -100,7 +100,7 @@ The suite Code2Test writes therefore always passes: every test either passes or
 | `code2test benchmark [PATH]` | Measure mutation score and diagnosis accuracy of the generated tests |
 | `code2test intent show\|edit\|export [NAME]` | Review or correct inferred intents |
 | `code2test report summary\|coverage` | Summarize stored intents, or report behavioral coverage and diagnostic accuracy |
-| `code2test init` | Write a starter `.code2test/config.json` (the other commands take their settings from options, not from this file) |
+| `code2test init` | Save project defaults for `code2test test` (confidence threshold, output directory, exclude patterns) in `.code2test/config.json` |
 | `code2test config set\|show\|validate` | Store the API key and model (`--api-key`, `--main-model`); `validate` sends the model one test request |
 | `code2test version` | Show the installed version |
 
@@ -110,12 +110,15 @@ The suite Code2Test writes therefore always passes: every test either passes or
 | --- | --- | --- |
 | `--auto` | off | Run without prompts |
 | `--confidence FLOAT` | 0.6 | Intent confidence needed to generate tests in `--auto` mode |
-| `--output-dir PATH` | `tests` | Where generated tests are written |
-| `--include`, `--exclude GLOB` | | Limit which files are analyzed |
+| `--output-dir PATH` | `tests` | Where generated tests are written, relative to the project root |
+| `--include GLOB` | | Analyze only matching files |
+| `--exclude GLOB` | | Leave matching files out of the analysis; repeatable, and added to the built-in exclusions |
 | `--dry-run` | off | Generate tests (this calls the model) without writing or running test files |
 | `--exit-code` | off | Exit 1 if any written file failed verification (for CI) |
 | `--report html\|json\|all` | none | Write a run report to `<output-dir>/reports/` |
 
+Defaults for `--confidence`, `--output-dir` and extra exclude patterns can be saved per project with `code2test init`, for example `code2test init --confidence 0.7 --exclude "*migrations*"`. They are stored in `.code2test/config.json`; options given on the command line take precedence. The output directory and the usual test, cache and virtual-environment folders are never analyzed.
+
 `PATH` can be the repository, a subdirectory or a single file. Imports, the `tests/` directory and `.code2test/` always resolve to the project root, which is the nearest directory containing `.git`, `pyproject.toml`, `setup.py`, `setup.cfg` or `.code2test`.
 
 ### `code2test verify`
@@ -172,7 +175,7 @@ The last two come from the most recent `code2test benchmark` run; the report say
 | `.code2test/metrics.json` | Metrics of the last `test --auto` run |
 | `.code2test/benchmark.json` | Results of the last `benchmark` run, including uncaught mutants |
 | `.code2test/dependency_graphs/` | The analyzed dependency graph |
-| `.code2test/config.json` | Starter settings written by `code2test init` (not read by other commands) |
+| `.code2test/config.json` | Project defaults written by `code2test init` and read by `code2test test` |
 | `tests/reports/report.html`, `report.json` | Run reports, when `--report` is used |
 | `tests/reports/coverage.html`, `coverage.json` | Coverage reports from `code2test report coverage --format html\|json` |
 

code2test/__main__.py

@@ -1,5 +1,5 @@
 """
-Entry point for running codewiki as a module: python -m codewiki
+Entry point for running Code2Test as a module: python -m code2test
 """
 
 from code2test.cli.main import cli

code2test/cli/__init__.py

@@ -1,3 +1,3 @@
-"""CLI module for CodeWiki."""
+"""Command-line interface for Code2Test."""
 
 __all__ = []

code2test/cli/commands/init.py

@@ -2,69 +2,75 @@
 Init command for Code2Test.
 """
 
-import json
 from pathlib import Path
+from typing import Optional, Tuple
+
 import click
-from rich.prompt import Confirm, Prompt
+from rich.prompt import Confirm
 
 from code2test.cli.display import DisplayManager
+from code2test.cli.utils.project_config import (
+    PROJECT_DEFAULTS,
+    project_config_path,
+    write_project_settings,
+)
 
 
 @click.command("init")
 @click.argument("path", type=click.Path(exists=True, file_okay=False, dir_okay=True), default=".")
-def init_command(path: str) -> None:
+@click.option(
+    "--confidence",
+    type=click.FloatRange(0.0, 1.0),
+    default=None,
+    help=f"Intent confidence needed to generate tests in --auto mode (default {PROJECT_DEFAULTS['confidence_threshold']})",
+)
+@click.option(
+    "--output-dir",
+    type=str,
+    default=None,
+    help=f"Where generated tests are written (default {PROJECT_DEFAULTS['output_dir']})",
+)
+@click.option(
+    "--exclude",
+    type=str,
+    multiple=True,
+    help="Glob pattern of files to leave out of the analysis (repeatable)",
+)
+def init_command(
+    path: str, confidence: Optional[float], output_dir: Optional[str], exclude: Tuple[str, ...]
+) -> None:
     """
-    Initialize Code2Test in a repository.
+    Save project settings for 'code2test test' in .code2test/config.json.
+
+    'code2test test' uses these settings as its defaults: the confidence
+    threshold, the output directory and extra exclude patterns. Options
+    given to 'code2test test' still take precedence. Edit the file to
+    change them later.
 
-    Creates a .code2test configuration directory and detects project settings.
+    \b
+    Examples:
+        code2test init
+        code2test init --confidence 0.7 --exclude "*migrations*"
     """
     display = DisplayManager()
     project_root = Path(path).resolve()
+    config_file = project_config_path(project_root)
 
-    display.info(f"Initializing Code2Test in: {project_root}")
-
-    config_dir = project_root / ".code2test"
-    config_file = config_dir / "config.json"
-
-    if config_file.exists():
-        if not Confirm.ask("Configuration already exists. Overwrite?"):
-            display.info("Aborted initialization.")
-            return
+    if config_file.exists() and not Confirm.ask(f"{config_file} already exists. Overwrite?"):
+        display.info("Aborted initialization.")
+        return
 
-    config_dir.mkdir(parents=True, exist_ok=True)
+    if not any(project_root.rglob("*.py")):
+        display.warning("No Python files found; Code2Test generates tests for Python code only.")
 
-    # Auto-detect language
-    language = "python"  # Default
-    framework = "pytest"
-
-    if (project_root / "package.json").exists():
-        language = "javascript"
-        framework = "jest"
-        display.info("Detected JavaScript/TypeScript project")
-    elif (project_root / "pom.xml").exists() or (project_root / "build.gradle").exists():
-        language = "java"
-        framework = "junit"
-        display.info("Detected Java project")
-    else:
-        display.info("Detected Python project (default)")
-
-    # User prompts
-    language = Prompt.ask(
-        "Project language", choices=["python", "javascript", "java"], default=language
-    )
-    framework = Prompt.ask("Test framework", default=framework)
-
-    config = {
-        "language": language,
-        "framework": framework,
-        "confidence_threshold": 0.6,
-        "auto_accept": False,
-        "max_concurrency": 5,
-        "exclude_patterns": ["node_modules", "venv", ".git", "__pycache__", "dist", "build"],
+    settings = {
+        "confidence_threshold": PROJECT_DEFAULTS["confidence_threshold"]
+        if confidence is None
+        else confidence,
+        "output_dir": output_dir or PROJECT_DEFAULTS["output_dir"],
+        "exclude_patterns": list(exclude),
     }
+    written = write_project_settings(project_root, settings)
 
-    with open(config_file, "w") as f:
-        json.dump(config, f, indent=2)
-
-    display.success(f"Initialized Code2Test configuration in {config_file}")
+    display.success(f"Saved project settings to {written}")
     display.info("Run 'code2test test .' to start generating tests.")