Repository navigation
release: FixMap v0.8.0 — close all 22 open reports - #155
Merged
Merged
Conversation
…s we document Diagnostics quote the task back at the reader because "no file matched auth, reset, mailer" is actionable where "no match" is not. But the input is arbitrary text, so a pasted blob traveled verbatim into JSON reports, CI logs, and pull request comments — an observed message was 30,155 characters (#136). The same applied to a mistyped --diff, twice over, because git echoes the failing command back inside its own error (#139). Truncating at the point of interpolation was not enough. Profiling the 30k case found the file-mention pattern taking 2.4 seconds on it: its body run contains ".", so it competes with the "\." that follows, and on an unbroken run with no extension the engine matched everything, failed, and retried one character shorter from every start position. That is quadratic, and the Action feeds this pattern issue text from public pull requests. Bounding the run makes it linear — 120k characters now cost 297ms where they would have cost roughly 38 seconds. Alongside that, three documented flags did not do what --help said: - verify accepted --output, then printed to stdout and created no file (#141). - Only --issue rejected duplicates. --repo, --format, --diff, --base, --head, --output, --report and --explain silently kept the last value, which is worst for --repo (scans a different tree than the one named first) and --format (hands a consumer the wrong contract) (#142). - An issue-only plan suggested `--diff <base>...HEAD`. v0.7.4 stopped inventing main...HEAD and replaced it with a placeholder that looks runnable and is not. The command is now runnable as printed, with the missing part on its own line outside it (#137). MCP verify now takes the report inline or as a path, mirroring CLI --report. Requiring the object form rejected agents that had used the CLI first without naming a shape that would work, and forced the model to re-embed an entire plan in the tool call (#138). Verify results carry scan diagnostics, so both commands hand an agent the same three concepts and consuming either no longer needs a branch on output shape (#133). An empty diff explains itself instead of printing two "- None found" sections that read like a display bug (#135). All four evaluation suites are unchanged: internal 62.5/87.5, external 67/100/100, held-out 58.3/75/75, adversarial false-confidence 0.0. Closes #133, #135, #136, #137, #138, #139, #141, #142 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…commands FixMap decided a repository was Python by asking whether any scanned file ended in .py. clap-rs/clap is Rust and keeps one helper script under .github/, so it was told "This Python repository has no supported package-script route; inspect pyproject.toml, tox.ini..." — advice for a toolchain it does not use (#152). A repository declares its language at the root, in the manifest its toolchain requires. That declaration is deliberate where a file extension is incidental, so detectPrimaryLanguage reads Cargo.toml, go.mod, pyproject.toml and package.json first, and only falls back to counting source files when the root is silent or two toolchains both claim it. That unlocks the larger gap: Go and Rust repositories were returning ranked files and an empty command list, which is ranking without any way to check the change. Both toolchains have exactly one test command, so it can be routed rather than guessed at. A cargo command in a workspace is scoped to the crate holding the ranked file, for the same reason package scripts already are — a command that cannot reach a file should not list it (#153). Python is deliberately not routed. pytest, tox, unittest and nox are all plausible for a repository carrying a pyproject.toml, and FixMap cannot read which one it configures, because .toml and .cfg are outside the sampled source extensions. So Python gets a named runner in the no-test-route diagnostic — tox when tox.ini is present, otherwise pytest — with an instruction to confirm it, rather than a routed command claiming more than FixMap knows (#134). Verified against the repositories in the reports, not only fixtures: clap-rs/clap cargo test --manifest-path clap_builder/Cargo.toml, no warning spf13/cobra go test ./..., no warning All four evaluation suites unchanged: internal 62.5/87.5, external 67/100/100, held-out 58.3/75/75, adversarial false-confidence 0.0. Closes #134, #152, #153 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…means Confidence came from an absolute score threshold — 14 and above was high — with nothing about whether the file had actually beaten anything. On a real Zod task the top eight scored 43, 24, 22, 20, 20, 20, 19, 19 and every one was labeled high, while the leader was nineteen points clear. That teaches an agent the eighth guess is as safe to edit as the first, which is the opposite of what the ranking knows (#143). High now requires leading: rank one, within two points of it (the window isClusteredRanking already treats as indistinguishable), or definition-site evidence of its own. Everything else caps at medium, which is what medium already meant — a neighborhood to read. A leader can also be contested. #102 established that a file defining the symbol a task names is a better answer than a file that merely talks about it a lot, so when the leader carries no definition evidence and something below it does, the lead is not decisive and is capped at medium. Score margin is deliberately not part of that test: a definition site six points back is still the definition site, and ranking it lower is exactly the vocabulary-density bias the boost exists to correct. Measured, on the repositories in the reports: zod stringProcessor task 8/8 high -> 1/8, correct file keeps high dayjs Y/YYY format task 8/8 high -> 1/8, src/constant.js keeps high while the parsing plugin drops to medium That last line is the outcome #140 asks for as its acceptance criterion. Its primary claim — customParseFormat ranked above src/constant.js — did not reproduce: at pin acf21cd, stock v0.7.4 already puts src/constant.js first, which I confirmed by measuring v0.7.4 directly rather than trusting this branch. The issue text was elided in the report, so the exact input could not be rebuilt. What the evidence does and does not show. All three accuracy suites are unchanged (internal 62.5/87.5, external 67/100/100, held-out 58.3/75/75) and the adversarial gate stays at false-confidence 0.0. Top-1 calibration is unchanged too — high 9/15, medium 6/8, low 2/4 — because these rules almost never disturb a genuine leader. That also means the suites cannot see this change: they score only the top-ranked file, and the fix is about the other seven rows. The Zod and dayjs numbers above are the evidence for it, not the benchmarks. Also corrects an overclaim found while checking those numbers. The README and the fixmap_plan tool description both told readers a high-confidence top result is right "about three quarters of the time". The measured figure in the table directly below it is 9/15 — 60%. Three quarters is the medium band. Both now say 9 out of 15 and point at the table. Closes #143. Addresses #140. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ode, and MCP explain Seven gaps from the dogfooding sweep, all of them things an agent or a person hit and had no way through. fixmap doctor (#148). The first thing that went wrong in the original session and the least interesting: a stale global install shadowed the version npx was asked for, so `verify` "did not exist" in a release that shipped it, and the time went into doubting the feature rather than the install. doctor names the running version, the resolved path, any conflicting global, and the Node version, and exits non-zero on a shadow so CI fails rather than reads on. Run against this machine it immediately found a real 0.3.1 global shadowing 0.7.4. plan --compare previous.json (#154). The habit worth having is: plan, add the identifier the task was missing, re-plan, and check whether the fix site rose. That meant two JSON files and a manual diff. Comparing this repository's own "ranking confidence" against "confidenceForEntry gives every top-8 file high confidence" now prints what the loop is for: rank.ts rose from 2 to 1, score 9 to 42, medium to high, grounding descriptive to anchored. --exclude and .fixmapignore (#145). FixMap's penalties know about examples/ and generated output because those are conventions. They cannot know a particular repository keeps a marketing site whose copy deliberately contains every symptom word the product documents. Excluded paths are removed before anything reads the candidate set, because the boilerplate threshold is a share of that set and removing files later would score against a population that no longer exists. --explain reports an excluded file as excluded rather than as scoring too low, which would be a false answer to the exact question it exists to answer. --limit (#149). Reports always returned eight files. The useful signal is usually the top one to three, and the rest burns agent context and invites drive-by edits. Also on MCP, where token budget is the whole constraint. --working-tree (#147). Reaching this through --diff HEAD worked but swept in untracked files, which on an agent-driven checkout means .claude/ metadata ranking beside the edit. Staged and unstaged tracked changes are what "what I am working on" means; untracked files are opt-in via --include-untracked. Verified on this branch: 13 tracked paths, no agent metadata. Progress phases (#150). A cold remote clone sat silent for thirty to ninety seconds, which reads as hung in CI logs and agent transcripts. Phases go to stderr, and only to a TTY or under FIXMAP_PROGRESS=1, so stdout stays a clean pipe. fixmap_explain over MCP (#144). --explain was CLI-only, so an agent with no shell had to re-invent the ranking to ask why a file was missing. The renderer moved to core rather than being copied, so the two surfaces cannot drift. One fix found while verifying doctor rather than from a report: reading the global version through execFile with npm.cmd fails on Node 24, which refuses .cmd without a shell, and the failure was swallowed — doctor would have reported "Global install: none" on Windows, the one platform where this footgun bites. Evaluations unchanged: internal 62.5/87.5, external 67/100/100, held-out 58.3/75/75, adversarial false-confidence 0.0. Exclusions and limits are opt-in, so the default path is byte-identical. Closes #144, #145, #147, #148, #149, #150, #154 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pull request URLs are what people actually paste. "Map what this PR is about" is a more common starting point than an issue link, for agents and humans both, and until now it hard-failed with advice to paste the text manually (#146). GitHub serves a pull request's title and body from the same /issues/N endpoint an issue uses, so accepting them costs one path segment and no new request. Compare, tree, discussion and file URLs stay rejected. They carry no task text to rank against, which was the real reason #112 rejected the whole set — pull requests were caught by a rule aimed at something else. An /issues/N URL that GitHub resolves to a pull request still fails, because the caller got something other than what they typed, but the error now names the /pull/N URL that would work instead of telling them to give up. Fetched pull requests are reported as remote-pull-fetched, so a reader can tell which kind of thing supplied the task. Verified against a real pull request, not only fixtures: --issue #132 -> remote-pull-fetched, title and body used as task context --issue .../compare/main...dev -> still refused, naming the URL kinds that work Action verify mode (#151). MCP gained verify in #120 and the Action did not, so workflows that never touch the CLI or MCP had no way to check a pushed fix against the plan it came from. `mode: verify` with `report-path` compares a saved plan — restored from an artifact a prior plan step uploaded — against the pull request diff, exposing finding-count and changed-file-count. It fails the step only for an edit in a generated or retired location, the one finding that is wrong regardless of the task; everything else stays advisory, because a plan can be wrong and a change can still be right. Closes #146, #151 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bumps root, core, cli, action, apps/web, and server.json, and rewrites the README and CHANGELOG around what this release actually changed. A minor bump rather than a patch: v0.8.0 adds a command (doctor), an MCP tool (fixmap_explain), six CLI flags, an Action mode, and pull request URLs as task input. The lockfile was patched by hand — regenerating it on Windows drops the @emnapi entries Linux `npm ci` needs, which has broken CI three times. All 28 are intact and `npm ci --dry-run` passes. Two documentation corrections went in with the release rather than being left for later, because both were claims about how reliable the tool is: - The README and the fixmap_plan tool description said a high-confidence top result is right "about three quarters of the time". The measured figure in the table directly below it is 9/15 — 60%. Three quarters is the medium band. - The README now states plainly that the benchmark suites cannot see the confidence change, because they score only the top-ranked file while the fix is about the other seven rows. The evidence for it is the measured Zod and dayjs behavior, not the suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hero terminal carries a comment saying the listing is the real output of that exact command on the demo repository, and that changing one means re-running the other. Re-running it found three drifts: - The second file was labeled high. It scores 19 against a leader at 47, so under the new confidence rule it is medium. This is the change working, shown where a visitor sees it first. - The risk line said "authentication - high severity". Issue-only plans have reported low severity since #126; the site kept the old wording for a release. - The leading file's score was quoted as 27. It is 47. Also updates the MCP description from two tools to three, now that fixmap_explain exists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 31, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes every open report from the v0.7.4 dogfooding sweep. Seven commits, each with a green
npm run ci. Tests 234 → 309.Two themes
FixMap was accurate about JavaScript and imprecise about everything else. Go and Rust repos returned ranked files and no test command at all — ranking without any way to check the change. Both now route
go test ./...andcargo test, workspace-scoped to the crate being edited. Language is read from the root manifest instead of asking whether any file ends in.py, which had labeled clap-rs/clap — a Rust project with one helper script under.github/— a Python repository.High confidence meant "in the list", not "the answer". It came from an absolute score threshold, so a real Zod task labeled all eight results high while the leader was nineteen points clear. High now requires leading, tying the lead within two points, or carrying definition-site evidence; and a leader that merely out-talks a definition site below it is capped at medium, because that competitor has the stronger kind of evidence.
Commits
f9580ebdiagnostics, CLI contracts, MCP parity332326dlanguage detection, go/cargo test routing9202892confidence calibration77c5e17doctor, compare, exclude, limit, working-tree, MCP explain82ac912PR URLs, Action verify mode5612922release v0.8.0a6ac6a5site example reportThree things reviewers should know
1. #140 does not reproduce. It claims dayjs
customParseFormatranks abovesrc/constant.jsat pinacf21cd. Stock v0.7.4 already putsconstant.jsfirst — I built v0.7.4 and measured it directly rather than trusting this branch. The issue text was elided with…so the exact input could not be rebuilt. The confidence half is real and fixed: the misleading parsing plugin now drops to medium, which is the fallback #140 names as acceptable. Marked "Addresses #140" rather than closing it; I'll comment there with the finding.2. A ReDoS nobody filed. Fixing #136 surfaced the file-mention pattern backtracking quadratically — its body run contains
., so it competes with the\.after it. 30,000 characters took 2.4 seconds, on a code path the Action feeds issue text from public pull requests. Now linear: 120,000 characters cost 297 ms where they would have cost roughly 38 seconds.3. The benchmark suites cannot see the confidence fix. They score only the top-1 file's label, and these rules almost never disturb a genuine leader — so all three suites and the top-1 calibration came back byte-identical. The evidence for the fix is measured behavior on the named repositories, not the suites, and the CHANGELOG says so rather than letting "suites unchanged" imply validation.
Evidence
All accuracy unchanged — internal 62.5/87.5, regression 67/100/100, held-out 58.3/75/75, adversarial false-confidence 0.0. Calibration unchanged at high 9/15, medium 6/8, low 2/4.
Verified against the repositories in the reports, not fixtures alone:
cargo test --manifest-path clap_builder/Cargo.toml, Python mislabel gonego test ./...remote-pull-fetched; compare/tree URLs still refusedfixmap doctor→ found a genuine 0.3.1 global shadowing 0.7.4 on the dev machineCorrections shipped alongside
The README and the
fixmap_plantool description both said a high-confidence top result is right "about three quarters of the time". The measured figure in the table directly below it is 9/15 — 60%. Three quarters is the medium band. The site's hero terminal also showedauthentication · high severity, stale since #126 made issue-only risks low.The lockfile was patched by hand. Regenerating it on Windows drops the
@emnapientries Linuxnpm cineeds, which has broken CI three times; all 28 are intact andnpm ci --dry-runpasses.🤖 Generated with Claude Code