diff --git a/.changeset/churn-complexity-hotspots.md b/.changeset/churn-complexity-hotspots.md new file mode 100644 index 00000000..bc9689ab --- /dev/null +++ b/.changeset/churn-complexity-hotspots.md @@ -0,0 +1,5 @@ +--- +"@stainless-code/codemap": minor +--- + +Add churn × complexity hotspot ranking: `file_churn` refreshed on every index from git history, with `codemap ingest-churn`, MCP/HTTP `ingest_churn`, and config `churn.file` for non-git repos. New `churn-complexity-hotspots` recipe ranks files or symbols (`by_symbol`) by change frequency × complexity with normalized 0–100 scores and `churn_trend`. Outcome alias `hotspots` still maps to fan-in. diff --git a/README.md b/README.md index fdb51ca7..317b8bf7 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,8 @@ codemap validate --json # detect stale / mi codemap context --compact --for "refactor auth" # JSON envelope + intent-matched recipes codemap ingest-coverage coverage/coverage-final.json --json # Istanbul / LCOV (auto-detected) → coverage table; joins with symbols NODE_V8_COVERAGE=.cov bun test && codemap ingest-coverage .cov --runtime --json # V8 protocol (per-process dumps); local-only +codemap ingest-churn metrics/churn.json --json # precomputed file_churn → churn-complexity-hotspots (non-git / CI) +codemap query --json --recipe churn-complexity-hotspots # change-frequency × complexity (not the hotspots alias) codemap agents init # scaffold .agents/ rules + skills codemap agents init --mcp # PM-aware project MCP config (see docs/agents.md) codemap apply rename-preview --params old=foo,new=bar --dry-run # preview recipe-driven edits (substrate executor) @@ -86,7 +88,7 @@ codemap query --json --recipe fan-out-sample codemap dead-code --json # → query --recipe untested-and-dead codemap deprecated --ci # → query --recipe deprecated-symbols --ci codemap boundaries --format sarif > boundary-findings.sarif # → query --recipe boundary-violations --format sarif -codemap hotspots --json --group-by directory # → query --recipe fan-in --json --group-by directory +codemap hotspots --json --group-by directory # → query --recipe fan-in (import hubs — not churn×complexity) codemap coverage-gaps --json --summary # → query --recipe worst-covered-exports --json --summary # Parametrised recipes validate params from .md frontmatter before SQL binding. codemap query --json --recipe find-symbol-by-kind --params kind=function,name_pattern=%Query% @@ -238,12 +240,12 @@ codemap skill # full codemap S codemap rule # full codemap rule markdown to stdout # MCP server (Model Context Protocol) — for agent hosts (Claude Code, Cursor, Codex, generic MCP clients) -codemap mcp # JSON-RPC on stdio (20 tools; watcher default-ON) -# Tools (20): query, query_batch, query_recipe, audit, save_baseline, +codemap mcp # JSON-RPC on stdio (21 tools; watcher default-ON) +# Tools (21): query, query_batch, query_recipe, audit, save_baseline, # list_baselines, drop_baseline, context, validate, show, snippet, impact, # affected, trace, explore, node, apply, apply_rows, apply_diff_input, -# ingest_coverage -# CLI twins: query batch, trace, explore, node, file, schema, symbols, context --include-snippets, ingest-coverage (same JSON as MCP/HTTP). +# ingest_coverage, ingest_churn +# CLI twins: query batch, trace, explore, node, file, schema, symbols, context --include-snippets, ingest-coverage, ingest-churn (same JSON as MCP/HTTP). # query / query_recipe also accept baseline (same diff envelope as codemap query --baseline). # Resources: codemap://schema, codemap://skill, codemap://rule, codemap://mcp-instructions (lazy-cached); # codemap://recipes, codemap://recipes/{id} (live read-per-call — recency fields stay fresh); diff --git a/docs/agents.md b/docs/agents.md index 04d09a75..ae720387 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -133,7 +133,7 @@ See [architecture.md § Session lifecycle wiring](./architecture.md#session-life **`context.index_freshness`** — session bootstrap includes index-level freshness metadata: `commit_drift` (HEAD ≠ `last_indexed_commit`), `pending_sync` (watcher debounce queue or in-flight reindex), optional disk-drift counts when watch is off, and a single `warning` string when agents should pause or re-index. **`context.start_here`** (non-compact) adds inline index summary, intent-ranked `query_recipe` cards, and top hub files with export signatures (adaptive caps by file count; optional MCP/HTTP `include_snippets` for one-line previews). Debug intent biases `sample_markers` toward FIXME/TODO. **MCP:** array-shaped JSON tools (`query`, …) keep row payloads verbatim and append a second `content` block prefixed `@codemap/index_freshness`; object-shaped tools merge `index_freshness` inline. **HTTP:** `POST /tool/*` adds `X-Codemap-Pending-Sync`, `X-Codemap-Commit-Drift`, and `X-Codemap-Warning` headers without changing JSON bodies; **`GET /health`** includes full cheap `index_freshness` when the DB is readable. Complements per-file `validate` / snippet `stale`. See [architecture.md § Context wiring](./architecture.md#context-wiring). -**MCP ToolAnnotations** — `tools/list` (and HTTP `GET /tools`) expose advisory `readOnlyHint` / `destructiveHint` / `idempotentHint` per tool so clients can gate auto-approval. Read paths (`query`, `show`, `audit`, …) → `readOnlyHint: true`; disk-write apply tools → `destructiveHint: true` (writes still require `yes: true`); index mutators (`save_baseline`, `drop_baseline`, `ingest_coverage`) → `readOnlyHint: false` without `destructiveHint`. +**MCP ToolAnnotations** — `tools/list` (and HTTP `GET /tools`) expose advisory `readOnlyHint` / `destructiveHint` / `idempotentHint` per tool so clients can gate auto-approval. Read paths (`query`, `show`, `audit`, …) → `readOnlyHint: true`; disk-write apply tools → `destructiveHint: true` (writes still require `yes: true`); index mutators (`save_baseline`, `drop_baseline`, `ingest_coverage`, `ingest_churn`) → `readOnlyHint: false` without `destructiveHint`. **`CODEMAP_MCP_TOOLS`** — comma-separated snake_case MCP tool names. When set, only listed tools register (stderr lists the active set). Unknown names are ignored with a warning. Unset = all tools (default). **`query_batch`** registers only when listed or when unset (eval ablation). diff --git a/docs/architecture.md b/docs/architecture.md index 611131b3..30bfc20a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -212,7 +212,7 @@ Three **mutually exclusive** CLI entry shapes; all converge on `applyDiffPayload **`src/application/session-lifecycle.ts`** — transport-specific start/stop rules for long-running `mcp` / `serve` processes (one-shot CLI unchanged). **`createStdioDisconnectMonitor`** (MCP only) exits the process when the agent host is actually gone: stdin EOF, stdout `EPIPE`, boot parent PID no longer alive (2s poll), or SIGINT/SIGTERM. The MCP SDK's stdio `transport.onclose` alone is insufficient — it fires only after an explicit `transport.close()`, not when the parent crashes without tearing down the pipe. **`createManagedWatchSession`** refcount-gates chokidar: MCP acquires one client before `connect` and **`forceStop`** drains the watcher on disconnect; HTTP acquires per authenticated request (after auth; **`GET /health`** excluded) and **`releaseClient`** stops the watcher when the count hits zero. **No MCP idle timeout:** `codemap mcp` does **not** exit after N minutes without tool calls while the stdio pipe stays open. IDE hosts spawn MCP once per session and do not reliably respawn it mid-conversation — an idle shutdown would break long pauses (human think time, reading, multi-step plans) with no recovery path. Orphan cleanup is handled by **disconnect detection**, not inactivity timers. **HTTP watch release grace (`HTTP_WATCH_RELEASE_GRACE_MS` = 5000):** distinct from idle timeout — only stops chokidar between stateless requests so the watcher is not started/stopped on every POST; the HTTP listener keeps running. **`GET /health`** liveness probes do not acquire a watch client (probes must not keep chokidar hot). Future **`MCP shared daemon per project`** could revisit opt-in idle policies with explicit client reconnect; not planned for stdio MCP today. -**Performance wiring:** **`--performance`** plumbs through **`RunIndexOptions.performance`** → **`indexFiles({ performance, collectMs })`**. `parse-worker-core.ts` records per-file **`parseMs`** on each `ParsedFile`; main thread times the eight phases (`collect`, `parse`, `insert`, `index_create`, `bindings`, `module_cycles`, `re_export_chains`, `heritage`) and assembles **`IndexPerformanceReport`** under `IndexRunStats.performance`. Note: `total_ms` is `indexFiles` wall-clock (parse + insert + DDL + bindings + cycles + re_exports + heritage), **not** end-to-end run wall — `collect_ms` happens before `indexFiles` and is reported separately. Env var **`CODEMAP_PERFORMANCE_JSON=`** dumps the report as JSON post-run (consumed by [`bun run check:perf-baseline`](./benchmark.md#perf-baseline-regression-guardrail) for local + weekly scheduled drift checks — not a PR merge gate). +**Performance wiring:** **`--performance`** plumbs through **`RunIndexOptions.performance`** → **`indexFiles({ performance, collectMs })`**. `parse-worker-core.ts` records per-file **`parseMs`** on each `ParsedFile`; main thread times the eight phases (`collect`, `parse`, `insert`, `index_create`, `bindings`, `module_cycles`, `re_export_chains`, `heritage`) and assembles **`IndexPerformanceReport`** under `IndexRunStats.performance`. Post-index **`refreshFileChurn`** records **`churn_ms`** separately (patched into the performance JSON when `CODEMAP_PERFORMANCE_JSON` is set). Note: `total_ms` is `indexFiles` wall-clock (parse + insert + DDL + bindings + cycles + re_exports + heritage), **not** end-to-end run wall — `collect_ms` and `churn_ms` happen outside `indexFiles` and are reported separately. Env var **`CODEMAP_PERFORMANCE_JSON=`** dumps the report as JSON post-run (consumed by [`bun run check:perf-baseline`](./benchmark.md#perf-baseline-regression-guardrail) for local + weekly scheduled drift checks — not a PR merge gate). **Agent templates:** `codemap agents init` writes thin pointer files (~18-line SKILL + ~25-line rule) to consumer disk; full content is served live by `codemap skill` / `codemap rule` (CLI) and `codemap://skill` / `codemap://rule` (MCP / HTTP) from `templates/agent-content//*.md`. Section files concatenate in lexical order; `*.gen.md` sections dispatch to renderers in `application/agent-content.ts` so recipe catalog + schema DDL auto-register. Pointer-version stamp (``) + once-per-process stderr nag (`maybeWarnStalePointers`) flag stale consumer templates; cure is `codemap agents init --force`. Full matrix: [agents.md](./agents.md). @@ -496,6 +496,23 @@ One row per leaf parameter binding, ordered by `position`. Pattern params (`func | column_start | INTEGER | 0-based column of the binding token | | column_end | INTEGER | One-past-last column | +### `file_churn` — Git churn metrics per indexed file (`STRICT`) + +One row per indexed file with git history in scope. Populated on **every index pass** by `refreshFileChurn` — git repos via `ingestFileChurnFromGit` (`git log --numstat` scoped to the project root pathspec); when config **`churn.file`** is set, JSON ingest runs instead and **skips** git log. Tunable via `churn.halfLifeDays` (default 90) and optional `churn.since` / CLI `--churn-since `. Non-git repos skip automatic git ingest (table empty until seeded). `churn_trend` is `accelerating` \| `stable` \| `cooling` when enough history exists, else NULL. + +| Column | Type | Description | +| ---------------- | ------- | -------------------------------------------------------------------------- | +| file_path | TEXT PK | FK → `files(path)` CASCADE | +| commit_count | INTEGER | Distinct commits touching the file in scope | +| weighted_commits | REAL | Recency-weighted commit count (default 90-day half-life exponential decay) | +| lines_added | INTEGER | Sum of added lines from numstat | +| lines_removed | INTEGER | Sum of removed lines from numstat | +| last_commit_at | TEXT | ISO timestamp of most recent commit touching the file | +| churn_trend | TEXT | `"accelerating"` \| `"stable"` \| `"cooling"` — nullable in v1 | +| computed_at | TEXT | ISO timestamp when ingest last ran | + +Powers **`churn-complexity-hotspots`** recipe (`hotspot_score`, `hotspot_score_normalized`; file or symbol grain via `by_symbol`). Non-git / fixtures: **`codemap ingest-churn`**, MCP/HTTP **`ingest_churn`**, or config **`churn.file`**. Distinct from outcome alias **`hotspots`** → `fan-in`. + ### `file_metrics` — Per-file aggregate metrics (`STRICT`) One row per indexed TS/JS file. Line classification is regex-light (blank if `/^\s*$/`; comment if line starts with `//`, `/*`, `*`, `*/`). diff --git a/docs/benchmark.md b/docs/benchmark.md index ff6090d3..2a8890a2 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -207,9 +207,9 @@ Independent of the consumer-facing scenarios above, the repo carries a **per-pha ### Mechanism -1. `bun src/index.ts --full --performance` populates [`IndexPerformanceReport`](../src/application/types.ts) with `collect_ms` / `parse_ms` / `insert_ms` / `index_create_ms` / `bindings_ms` / `module_cycles_ms` / `re_export_chains_ms` / `heritage_ms` / `total_ms`. +1. `bun src/index.ts --full --performance` populates [`IndexPerformanceReport`](../src/application/types.ts) with `collect_ms` / `parse_ms` / `insert_ms` / `index_create_ms` / `bindings_ms` / `module_cycles_ms` / `re_export_chains_ms` / `heritage_ms` / `total_ms`, plus post-index **`churn_ms`** (git churn ingest; patched after `indexFiles` completes). 2. Setting `CODEMAP_PERFORMANCE_JSON=` dumps that report as JSON to `` after the run (no CLI flag added; env-var only). -3. [`scripts/check-perf-baseline.ts`](../scripts/check-perf-baseline.ts) (alias `bun run check:perf-baseline`) runs the indexer 3× on this repo, takes per-phase **medians**, and compares **`collect_ms`**, **`parse_ms`**, **`insert_ms`**, **`index_create_ms`**, **`bindings_ms`**, and **`total_ms`** to `fixtures/benchmark/perf-baseline.json`. Other `IndexPerformanceReport` fields (`module_cycles_ms`, `re_export_chains_ms`, `heritage_ms`, …) appear in `--performance` JSON only — not baseline-gated. +3. [`scripts/check-perf-baseline.ts`](../scripts/check-perf-baseline.ts) (alias `bun run check:perf-baseline`) runs the indexer `CODEMAP_PERF_RUNS`× on this repo (`--full --performance`), then the same count idle incremental (`codemap --performance`, no `--full`), takes per-phase **medians**, and compares **`collect_ms`**, **`parse_ms`**, **`insert_ms`**, **`index_create_ms`**, **`bindings_ms`**, **`churn_ms`**, **`churn_idle_ms`** (idle incremental `churn_ms` when HEAD unchanged), and **`total_ms`** to `fixtures/benchmark/perf-baseline.json`. Phases under their noise floor skip gating (`noise_floor_ms` default 10ms; `churn_idle_ms` uses a 5ms floor in the checker). Idle runs also fail when `churn_ms` exceeds `CODEMAP_PERF_IDLE_CHURN_MAX_MS` (default 50ms) — catches accidental full git churn on the idle path. Other `IndexPerformanceReport` fields (`module_cycles_ms`, `re_export_chains_ms`, `heritage_ms`, …) appear in `--performance` JSON only — not baseline-gated. 4. **Local / scheduled only** — run before perf-sensitive PRs; [`.github/workflows/perf-baseline.yml`](../.github/workflows/perf-baseline.yml) fires weekly + `workflow_dispatch` for drift visibility. **Not** on the PR CI path (6 min × 3 runs + bimodal GHA runners → flaky merge gate). ### Why this is separate from `src/benchmark.ts` diff --git a/docs/glossary.md b/docs/glossary.md index c44bee8b..dc92d645 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -260,6 +260,10 @@ Number of edges _into_ a file in the `dependencies` table — `COUNT(*) FROM dep Number of edges _out of_ a file — `COUNT(*) FROM dependencies WHERE from_path = ?`. Surfaces as the `fan-out` recipe. +### `file_churn` (table) + +Per-file git churn metrics refreshed on **every index pass** (incremental scoped recompute; idle HEAD cache). Git via `ingestFileChurnFromGit`; non-git via **`codemap ingest-churn`** or config **`churn.file`** (skips git when `churn.file` is set). MCP/HTTP twin: **`ingest_churn`** (`{path}`). Config: `churn.halfLifeDays`, `churn.since` / `--churn-since`. Column **`churn_trend`**: `accelerating` \| `stable` \| `cooling` when enough history exists. Powers **`churn-complexity-hotspots`** (file or symbol grain, normalized score) — distinct from outcome alias **`hotspots`** → `fan-in`. Empty table → `context` **`churn_hint`** on `start_here`. + ### `files` (table) Header row for every indexed file. `path` is the primary key; all other tables FK to it with `ON DELETE CASCADE`. Flags: `is_barrel` (100% re-exports, no local value symbols) and `has_side_effects` (module-level call/assignment seen at parse time). See `FileRow`. @@ -378,7 +382,7 @@ Rust-based CSS parser (NAPI bindings). Codemap's `src/css-parser.ts` uses its vi ### `codemap mcp` / MCP server -Stdio MCP (Model Context Protocol) server exposing codemap's structural-query surface to agent hosts (Claude Code, Cursor, Codex, generic MCP clients) as JSON-RPC tools — eliminates the bash round-trip on every agent invocation. **20 tools:** `query`, `query_batch`, `query_recipe`, `audit`, `save_baseline`, `list_baselines`, `drop_baseline`, `context`, `validate`, `show`, `snippet`, `impact`, `affected`, `trace`, `explore`, `node`, `apply`, `apply_rows`, `apply_diff_input`, `ingest_coverage`. Each has a CLI twin with the same JSON payload except transport-only MCP resources (`codemap://mcp-instructions`, initialize `instructions`). Subset via **`CODEMAP_MCP_TOOLS`** ([agents.md § MCP tool allowlist](./agents.md#mcp-tool-allowlist)). **Resources:** `codemap://schema`, `codemap://skill`, `codemap://rule`, `codemap://mcp-instructions`, `codemap://recipes`, `codemap://recipes/{id}`, `codemap://files/{path}`, `codemap://symbols/{name}`. Resource freshness is split by contract: schema / skill / rule / mcp-instructions are lazy-cached per server process; recipes, files, and symbols are live read-per-call so inline recency fields and index mutations under `--watch` don't freeze at first read. HTTP's `GET /resources/{encoded-uri}` uses the same resource handler. **Baseline tools** (`save_baseline`, `list_baselines`, `drop_baseline`) mirror `query --save-baseline` / `--baselines` / `--drop-baseline`; **`query` / `query_recipe`** also accept optional `baseline` for one-shot row diff vs saved snapshots (same envelope as CLI `query --baseline`). **CLI twins:** `query batch`, `trace`, `explore`, `node`, `file`, `schema`, `symbols`, `context --include-snippets`, `ingest-coverage`. Tool input/output keys are snake_case on MCP/HTTP — Codemap's convention; CLI stays kebab. Output shape matches each tool's CLI JSON payload; MCP wraps payloads in `{content: [{type: "text", text: …}]}`. Bootstrap once at server boot; tool handlers (in `application/tool-handlers.ts`) and resource handlers (in `application/resource-handlers.ts`) are pure transport-agnostic — the same handlers serve `codemap serve` (HTTP) via `POST /tool/{name}` and `GET /resources/{encoded-uri}`. **Session lifecycle:** exits on client disconnect (stdin EOF, stdout broken pipe, parent process exit, SIGINT/SIGTERM) via `session-lifecycle.ts`; **no idle timeout** — the process stays up while the pipe is open even without tool calls (see [§ Session lifecycle](./architecture.md#cli-usage)). With `--watch`, the watcher starts before connect and drains on exit. Implementation: `src/cli/cmd-mcp.ts` (CLI shell) + `src/application/mcp-server.ts` (engine). See [`architecture.md` § MCP wiring](./architecture.md#cli-usage). +Stdio MCP (Model Context Protocol) server exposing codemap's structural-query surface to agent hosts (Claude Code, Cursor, Codex, generic MCP clients) as JSON-RPC tools — eliminates the bash round-trip on every agent invocation. **21 tools:** `query`, `query_batch`, `query_recipe`, `audit`, `save_baseline`, `list_baselines`, `drop_baseline`, `context`, `validate`, `show`, `snippet`, `impact`, `affected`, `trace`, `explore`, `node`, `apply`, `apply_rows`, `apply_diff_input`, `ingest_coverage`, `ingest_churn`. Each has a CLI twin with the same JSON payload except transport-only MCP resources (`codemap://mcp-instructions`, initialize `instructions`). Subset via **`CODEMAP_MCP_TOOLS`** ([agents.md § MCP tool allowlist](./agents.md#mcp-tool-allowlist)). **Resources:** `codemap://schema`, `codemap://skill`, `codemap://rule`, `codemap://mcp-instructions`, `codemap://recipes`, `codemap://recipes/{id}`, `codemap://files/{path}`, `codemap://symbols/{name}`. Resource freshness is split by contract: schema / skill / rule / mcp-instructions are lazy-cached per server process; recipes, files, and symbols are live read-per-call so inline recency fields and index mutations under `--watch` don't freeze at first read. HTTP's `GET /resources/{encoded-uri}` uses the same resource handler. **Baseline tools** (`save_baseline`, `list_baselines`, `drop_baseline`) mirror `query --save-baseline` / `--baselines` / `--drop-baseline`; **`query` / `query_recipe`** also accept optional `baseline` for one-shot row diff vs saved snapshots (same envelope as CLI `query --baseline`). **CLI twins:** `query batch`, `trace`, `explore`, `node`, `file`, `schema`, `symbols`, `context --include-snippets`, `ingest-coverage`. Tool input/output keys are snake_case on MCP/HTTP — Codemap's convention; CLI stays kebab. Output shape matches each tool's CLI JSON payload; MCP wraps payloads in `{content: [{type: "text", text: …}]}`. Bootstrap once at server boot; tool handlers (in `application/tool-handlers.ts`) and resource handlers (in `application/resource-handlers.ts`) are pure transport-agnostic — the same handlers serve `codemap serve` (HTTP) via `POST /tool/{name}` and `GET /resources/{encoded-uri}`. **Session lifecycle:** exits on client disconnect (stdin EOF, stdout broken pipe, parent process exit, SIGINT/SIGTERM) via `session-lifecycle.ts`; **no idle timeout** — the process stays up while the pipe is open even without tool calls (see [§ Session lifecycle](./architecture.md#cli-usage)). With `--watch`, the watcher starts before connect and drains on exit. Implementation: `src/cli/cmd-mcp.ts` (CLI shell) + `src/application/mcp-server.ts` (engine). See [`architecture.md` § MCP wiring](./architecture.md#cli-usage). ### `query_batch` @@ -410,7 +414,7 @@ Key-value metadata table. Holds `schema_version`, `last_indexed_commit`, `indexe ### outcome aliases (`dead-code` / `deprecated` / `boundaries` / `hotspots` / `coverage-gaps`) -Top-level CLI verbs that thin-wrap `query --recipe `: `dead-code` → `untested-and-dead`, `deprecated` → `deprecated-symbols`, `boundaries` → `boundary-violations`, `hotspots` → `fan-in`, `coverage-gaps` → `worst-covered-exports`. Every `query` flag passes through (`--json`, `--format`, `--ci`, `--summary`, `--changed-since`, `--group-by`, `--params`, `--save-baseline`, `--baseline`). Mapping lives in `src/cli/aliases.ts` (`OUTCOME_ALIASES`). Capped at 5 to avoid alias-sprawl — promote a sixth only when the recipe becomes a headline outcome. Moat-A clean: the alias is a one-line rewrite, not a new primitive; the recipe IS the SQL. **Write alias (distinct):** `codemap rename` thin-wraps `apply rename-preview` (not `query --recipe`) — mapping in `src/cli/rename-alias.ts`; same Moat-A rule (no new write semantics). +Top-level CLI verbs that thin-wrap `query --recipe `: `dead-code` → `untested-and-dead`, `deprecated` → `deprecated-symbols`, `boundaries` → `boundary-violations`, `hotspots` → `fan-in`, `coverage-gaps` → `worst-covered-exports`. For **change-frequency × complexity** refactor targets use recipe **`churn-complexity-hotspots`** (not the `hotspots` alias). Every `query` flag passes through (`--json`, `--format`, `--ci`, `--summary`, `--changed-since`, `--group-by`, `--params`, `--save-baseline`, `--baseline`). Mapping lives in `src/cli/aliases.ts` (`OUTCOME_ALIASES`). Capped at 5 to avoid alias-sprawl — promote a sixth only when the recipe becomes a headline outcome. Moat-A clean: the alias is a one-line rewrite, not a new primitive; the recipe IS the SQL. **Write alias (distinct):** `codemap rename` thin-wraps `apply rename-preview` (not `query --recipe`) — mapping in `src/cli/rename-alias.ts`; same Moat-A rule (no new write semantics). ### oxc-parser @@ -561,7 +565,7 @@ Long-running process that subscribes to filesystem changes via [chokidar v5](htt ### `codemap serve` / HTTP server -Long-running HTTP server exposing the same tool taxonomy as `codemap mcp` over `POST /tool/{name}` for non-MCP consumers (CI scripts, simple `curl`, IDE plugins that don't speak MCP). Default bind **`127.0.0.1:7878`** (loopback only — refuse `0.0.0.0` unless explicitly opted in via `--host 0.0.0.0`); optional `--token ` requires `Authorization: Bearer ` on every request. HTTP returns each tool's native JSON payload directly (NOT MCP's `{content: [...]}` wrapper); `query` / `query_recipe` match `codemap query --json` row arrays unless `summary` / `group_by` reshape the envelope, or `baseline` returns a diff envelope (incompatible with non-`json` `format` / `group_by`; save/list/drop remain separate tools); parity twins (`query batch`, `trace`, `explore`, `node`, `file`, `schema`, `symbols`, `context`, `ingest-coverage`) always emit JSON on CLI without `--json`; other tools match their CLI `--json` payloads when that flag is set; `format: "sarif"` payloads ship as `application/sarif+json`, `format: "annotations"` / `"mermaid"` / `"diff"` / `"badge"` (markdown) as `text/plain; charset=utf-8`, `format: "diff-json"` / `"codeclimate"` / `"badge"` + `badge_style: "json"` as `application/json; charset=utf-8`. Routes: `POST /tool/{name}` (every MCP tool), `GET /resources/{encoded-uri}` (resource handler for `codemap://recipes`, `codemap://recipes/{id}`, `codemap://schema`, `codemap://skill`, `codemap://rule`, `codemap://mcp-instructions`, `codemap://files/{path}`, and `codemap://symbols/{name}`), `GET /health` (auth-exempt liveness probe — does not start the watcher), `GET /tools` / `GET /resources` (catalogs). With `--watch`, chokidar is refcount-gated per request and stops 5s after the last client (`HTTP_WATCH_RELEASE_GRACE_MS`) — distinct from MCP idle shutdown; the HTTP process keeps listening. Pure transport — same `tool-handlers.ts` / `resource-handlers.ts` MCP uses; no engine duplication. Errors → `{"error": "..."}` with HTTP status 400 / 401 / 403 / 404 / 500. SIGINT / SIGTERM → graceful drain. Every response carries `X-Codemap-Version: `. **CSRF + DNS-rebinding guard:** every request (including auth-exempt `/health`) is evaluated against `Sec-Fetch-Site` / `Origin` / `Host` when present — modern browsers send `Sec-Fetch-Site` and `Origin` on cross-origin fetches (header presence varies by request type, browser, and privacy settings), so the guard rejects browser-driven cross-origin requests like a malicious local webpage `fetch`-ing `http://127.0.0.1:7878/tool/save_baseline` to mutate `.codemap/index.db`. `Host` mismatch on a loopback bind blocks DNS rebinding (an attacker resolving `evil.com` to `127.0.0.1` post-load). Non-browser clients (curl, fetch from Node, MCP hosts, CI scripts) typically omit these headers and pass through. Implementation: `src/cli/cmd-serve.ts` (CLI shell) + `src/application/http-server.ts` (transport). See [`architecture.md` § HTTP wiring](./architecture.md#cli-usage). +Long-running HTTP server exposing the same tool taxonomy as `codemap mcp` over `POST /tool/{name}` for non-MCP consumers (CI scripts, simple `curl`, IDE plugins that don't speak MCP). Default bind **`127.0.0.1:7878`** (loopback only — refuse `0.0.0.0` unless explicitly opted in via `--host 0.0.0.0`); optional `--token ` requires `Authorization: Bearer ` on every request. HTTP returns each tool's native JSON payload directly (NOT MCP's `{content: [...]}` wrapper); `query` / `query_recipe` match `codemap query --json` row arrays unless `summary` / `group_by` reshape the envelope, or `baseline` returns a diff envelope (incompatible with non-`json` `format` / `group_by`; save/list/drop remain separate tools); parity twins (`query batch`, `trace`, `explore`, `node`, `file`, `schema`, `symbols`, `context`, `ingest-coverage`, `ingest-churn`) always emit JSON on CLI without `--json`; other tools match their CLI `--json` payloads when that flag is set; `format: "sarif"` payloads ship as `application/sarif+json`, `format: "annotations"` / `"mermaid"` / `"diff"` / `"badge"` (markdown) as `text/plain; charset=utf-8`, `format: "diff-json"` / `"codeclimate"` / `"badge"` + `badge_style: "json"` as `application/json; charset=utf-8`. Routes: `POST /tool/{name}` (every MCP tool), `GET /resources/{encoded-uri}` (resource handler for `codemap://recipes`, `codemap://recipes/{id}`, `codemap://schema`, `codemap://skill`, `codemap://rule`, `codemap://mcp-instructions`, `codemap://files/{path}`, and `codemap://symbols/{name}`), `GET /health` (auth-exempt liveness probe — does not start the watcher), `GET /tools` / `GET /resources` (catalogs). With `--watch`, chokidar is refcount-gated per request and stops 5s after the last client (`HTTP_WATCH_RELEASE_GRACE_MS`) — distinct from MCP idle shutdown; the HTTP process keeps listening. Pure transport — same `tool-handlers.ts` / `resource-handlers.ts` MCP uses; no engine duplication. Errors → `{"error": "..."}` with HTTP status 400 / 401 / 403 / 404 / 500. SIGINT / SIGTERM → graceful drain. Every response carries `X-Codemap-Version: `. **CSRF + DNS-rebinding guard:** every request (including auth-exempt `/health`) is evaluated against `Sec-Fetch-Site` / `Origin` / `Host` when present — modern browsers send `Sec-Fetch-Site` and `Origin` on cross-origin fetches (header presence varies by request type, browser, and privacy settings), so the guard rejects browser-driven cross-origin requests like a malicious local webpage `fetch`-ing `http://127.0.0.1:7878/tool/save_baseline` to mutate `.codemap/index.db`. `Host` mismatch on a loopback bind blocks DNS rebinding (an attacker resolving `evil.com` to `127.0.0.1` post-load). Non-browser clients (curl, fetch from Node, MCP hosts, CI scripts) typically omit these headers and pass through. Implementation: `src/cli/cmd-serve.ts` (CLI shell) + `src/application/http-server.ts` (transport). See [`architecture.md` § HTTP wiring](./architecture.md#cli-usage). ### Code Climate format (`codeclimate`) diff --git a/docs/golden-queries.md b/docs/golden-queries.md index 5ced88d6..c453cf1d 100644 --- a/docs/golden-queries.md +++ b/docs/golden-queries.md @@ -62,7 +62,7 @@ We **do not** commit another product’s source tree, paths, business strings, o ## Scenario shape (implemented) -Scenarios live in **`fixtures/golden/scenarios.json`** (Tier A) or optional **`scenarios.external.json`** / **example** (Tier B). The file may be a **bare array** of scenarios (legacy) or an object `{ "setup": [...], "scenarios": [...] }`. Optional top-level **`setup`** runs once after index, before scenarios — today **`ingest-coverage`** and **`clear-coverage`** (see [run-setup.ts](../scripts/query-golden/run-setup.ts)); missing coverage files are skipped with a warning. Per-scenario **`preSetup`** runs after global setup (global setup restores after the scenario when `preSetup` mutates the index). Each scenario has **`id`**, **`sql` or `recipe`**, optional **`match`** (`exact`, `minRows`, `everyRowContains`, `everyRowFieldEquals`), optional **`budgetMs`**. Goldens: **`fixtures/golden/minimal/*.json`** etc. Refresh: **`bun scripts/query-golden.ts --update`**. +Scenarios live in **`fixtures/golden/scenarios.json`** (Tier A) or optional **`scenarios.external.json`** / **example** (Tier B). The file may be a **bare array** of scenarios (legacy) or an object `{ "setup": [...], "scenarios": [...] }`. Optional top-level **`setup`** runs once after index, before scenarios — today **`ingest-coverage`**, **`clear-coverage`**, and **`seed-file-churn`** (see [run-setup.ts](../scripts/query-golden/run-setup.ts)); missing coverage files are skipped with a warning. Per-scenario **`preSetup`** runs after global setup (global setup restores after the scenario when `preSetup` mutates the index). Each scenario has **`id`**, **`sql` or `recipe`**, optional **`match`** (`exact`, `minRows`, `everyRowContains`, `everyRowFieldEquals`), optional **`budgetMs`**. Goldens: **`fixtures/golden/minimal/*.json`** etc. Refresh: **`bun scripts/query-golden.ts --update`**. **Prompts** in JSON are **intent labels**, not pasted chat logs — pair with queries whose literals come from **fixture-owned** data (see [fixtures/qa/prompts.external.template.md](../fixtures/qa/prompts.external.template.md) for optional chat QA). @@ -78,6 +78,10 @@ Some bundled recipes add optional **`reason`** (TEXT) and **`evidence_json`** (T `coverage-confirmed-dead` adds **`confidence`** (`high` \| `medium`) on each row — **`high`** when static dead and ingested `coverage_pct = 0`; **`medium`** when static dead but the symbol has no ingested coverage row. Also **`reason`**, **`caller_count`**. Goldens: `coverage-confirmed-dead` (post-ingest mix) and `coverage-confirmed-dead-no-ingest` (`preSetup: clear-coverage`, `everyRowFieldEquals` on `confidence: medium`). +### Churn / hotspot columns (`churn-complexity-hotspots` recipe) + +`churn-complexity-hotspots` ranks indexed files or symbols by git churn × cyclomatic complexity. File grain (default): **`file_path`**, null **`symbol_name`** / **`symbol_kind`** / **`line_start`**, **`max_complexity`**, **`avg_complexity`**, churn fields, scores. Symbol grain (`by_symbol=true`): **`symbol_name`**, **`symbol_kind`**, **`line_start`**, **`max_complexity`**. Optional **`path_prefix`** scopes to a subtree. Goldens: `churn-complexity-hotspots`, `churn-complexity-hotspots-by-symbol`, `churn-complexity-hotspots-path-prefix` (fixture churn seeded via **`seed-file-churn`**). + ### Duplication columns (`duplicates` recipe) `duplicates` returns one row per function-shaped symbol in a **`body_hash`** collision group: **`name`**, **`kind`**, **`file_path`**, **`line_start`**, **`line_end`**, **`body_hash`**, **`body_line_count`**, **`duplicate_count`** (in-scope group size after `path_prefix` / `min_body_lines`). Substrate column **`symbols.body_hash`** is populated at index for function-shaped symbols (`function`, `method`, `getter`, `setter`) when `body_line_count >= 2`. Goldens: `duplicates` (includes `src/bench/duplicate-body-{a,b}.ts` pair). False positives possible when unrelated functions share control-flow skeleton or sync vs async/generator bodies match — triage with `snippet`. Recipe caps at **50 rows** (no truncation marker). diff --git a/docs/plans/churn-complexity-hotspots.md b/docs/plans/churn-complexity-hotspots.md deleted file mode 100644 index b3e21d00..00000000 --- a/docs/plans/churn-complexity-hotspots.md +++ /dev/null @@ -1,140 +0,0 @@ -# Churn × complexity hotspots — plan - -> **Status:** open · **Priority:** P2 · **Effort:** L–M (~2–3 weeks) -> -> **Motivator:** Agents and maintainers prioritize refactors by **change frequency × structural complexity** — files that churn often _and_ carry heavy symbols are higher-risk touch points than either signal alone. Today `symbols.complexity` exists but git churn is not indexed; `codemap hotspots` alias maps to import **fan-in**, not change×complexity. -> -> **Roadmap:** [§ Core substrate & platform](../roadmap.md#core-substrate--platform) - ---- - -## Agent start here - -Ship **schema + mocked churn rows + recipe SQL** before wiring real `git log` ingest — proves the JOIN path without git subprocess flakiness in CI. Read `symbols.complexity` and outcome aliases in [`src/cli/aliases.ts`](../../src/cli/aliases.ts); do **not** repoint `hotspots` alias (stays `fan-in`). - -### Key touchpoints - -| File | What to read | -| -------------------------------------------------------------------------- | ------------------------------------------------------------- | -| [`src/db.ts`](../../src/db.ts) | `SCHEMA_VERSION`, table DDL, migrations | -| [`src/application/index-engine.ts`](../../src/application/index-engine.ts) | Full / incremental index hooks (where churn refresh attaches) | -| [`src/extractors/complexity.ts`](../../src/extractors/complexity.ts) | Existing `symbols.complexity` population | -| [`templates/recipes/`](../../templates/recipes/) | Recipe `.sql` + `.md` pair pattern (e.g. `fan-in`) | -| [`src/cli/aliases.ts`](../../src/cli/aliases.ts) | Outcome alias `hotspots` → `fan-in` — leave unchanged | -| [`src/cli/cmd-query.ts`](../../src/cli/cmd-query.ts) | Recipe catalog registration path | - -### Architecture - -```text -index (full or incremental) - → churn-ingest: git log --numstat scoped to indexed files.path - → file_churn rows (weighted_commits, trend, …) -recipe churn-complexity-hotspots - → SQL JOIN file_churn × symbols.complexity → hotspot_score - → query / MCP / HTTP (Moat A — no new verb) -``` - -### Tracer bullet (slice 1) - -1. `file_churn` table + migration. 2. Insert fixture churn rows in test. 3. `churn-complexity-hotspots.sql` returns ranked paths. 4. Wire real git ingest in slice 2. - -### Out of scope (v1) - -Repurposing `codemap hotspots` CLI alias; symbol-level hotspot rows (file-level only unless Q1 resolves otherwise); non-git VCS import (`--churn-file`). - ---- - -## Pre-locked decisions - -| # | Decision | Source | -| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | -| H.1 | **Moat-B substrate** — new `file_churn` table populated at index time (or lazy first-query with cache in `meta`). | [Moat B](../roadmap.md#moats-load-bearing) | -| H.2 | **Moat-A exposure** — bundled recipe `churn-complexity-hotspots` (distinct from outcome alias `hotspots` → `fan-in`). | [Moat A](../roadmap.md#moats-load-bearing); `aliases.ts` | -| H.3 | **Git-native churn v1** — `git log --numstat` (or `git log --format` + diff stat) scoped to indexed `files.path` set; optional `--churn-since ` CLI flag on ingest. | No network; matches incremental git invalidation story | -| H.4 | **Recency weighting** — store `weighted_commits` with exponential half-life (default 90 days) so recent edits rank above ancient history. | Tunable via config `churn.halfLifeDays` | -| H.5 | **Score is a recipe column, not a verdict** — `hotspot_score` computed in SQL JOIN (`file_churn` × aggregated `symbols.complexity` / `file_metrics`); consumer applies `LIMIT`. | Moat A | -| H.6 | **Optional trend column** — `churn_trend: "accelerating" \| "stable" \| "cooling"` from recent-vs-older window ratio; nullable when insufficient history. | v1 nice-to-have; ship score first if schedule tight | - ---- - -## Schema sketch - -```sql -CREATE TABLE file_churn ( - file_path TEXT PRIMARY KEY, - commit_count INTEGER NOT NULL, - weighted_commits REAL NOT NULL, - lines_added INTEGER NOT NULL, - lines_removed INTEGER NOT NULL, - last_commit_at TEXT, - churn_trend TEXT, - computed_at TEXT NOT NULL -) STRICT; -``` - -Recipe joins `file_churn` to per-file max/avg `symbols.complexity` (and optionally `file_metrics.line_count`): - -```sql --- sketch only; final SQL lives in templates/recipes/churn-complexity-hotspots.sql -SELECT f.path, - fc.weighted_commits, - MAX(s.complexity) AS max_complexity, - (fc.weighted_commits * MAX(s.complexity)) AS hotspot_score -FROM files f -JOIN file_churn fc ON fc.file_path = f.path -JOIN symbols s ON s.file_path = f.path -WHERE s.complexity IS NOT NULL -GROUP BY f.path -ORDER BY hotspot_score DESC; -``` - -Normalize score to 0–100 in recipe if cross-repo comparability matters (divide by corpus max). - ---- - -## Implementation steps - -1. **Churn ingest module** — `src/application/churn-ingest.ts`: spawn bounded git subprocess; map paths to indexed files only; respect `.gitignore` / codemap excludes. -2. **Index hook** — run churn refresh on full rebuild; incremental path refreshes churn for changed files + ancestors if needed (v1: full churn recompute acceptable on incremental if <2s on medium repos — measure in plan PR). -3. **`file_churn` table** + `SCHEMA_VERSION` bump + migration in `db.ts`. -4. **Recipe** — `churn-complexity-hotspots` with params `limit`, optional `min_complexity`. -5. **CLI** — no new outcome alias (fan-in keeps `hotspots`); document recipe in catalog + `context` intent keywords ("refactor priority", "hotspot", "churn"). -6. **Golden fixture** — synthetic git history in test repo or mocked churn rows. -7. Docs — `architecture.md` schema row; `glossary.md` disambiguate `hotspots` alias vs `churn-complexity-hotspots` recipe. - ---- - -### Verification - -```bash -bun test src/application/churn-ingest.test.ts # after module lands -bun src/index.ts query --recipe churn-complexity-hotspots --json -bun src/index.ts query --recipe fan-in --json # alias hotspots unchanged -bun run typecheck # db.ts schema + SymbolRow if touched -``` - ---- - -## Acceptance - -- [ ] Recipe returns files ranked by churn×complexity on codemap self-index -- [ ] Outcome alias `codemap hotspots` still resolves to `fan-in` -- [ ] Churn ingest skips non-git repos gracefully (empty `file_churn`, recipe returns empty set + stderr hint) -- [ ] No new pass/fail CLI verb - ---- - -## Open decisions (resolve in plan PR) - -| # | Question | -| --- | -------------------------------------------------------------------------------------------- | -| Q1 | File-level vs symbol-level hotspot rows — v1 file-level only? | -| Q2 | Recompute churn on every incremental index vs explicit `codemap index --refresh-churn` flag? | -| Q3 | Import `--churn-file` JSON for non-git VCS — defer unless consumer asks? | - ---- - -## Dependencies - -- Existing: `symbols.complexity`, `files`, `file_metrics`, git helpers in incremental index -- Independent of [C.9 plugin layer](./c9-plugin-layer.md) diff --git a/docs/plans/substrate-extraction.md b/docs/plans/substrate-extraction.md index 1d37e3c8..c34aaa79 100644 --- a/docs/plans/substrate-extraction.md +++ b/docs/plans/substrate-extraction.md @@ -393,7 +393,7 @@ New recipe candidates: `strict-mode-audit`; `missing-types-fields`; `monorepo-pa ### Tier 11 — Metrics expansion — **PARTIAL (2026-05-15)** -**Shipped:** `file_metrics` + per-symbol metric columns (see [`glossary.md`](../glossary.md)). **Open:** churn/git metrics → [`churn-complexity-hotspots.md`](./churn-complexity-hotspots.md). +**Shipped:** `file_metrics` + per-symbol metric columns (see [`glossary.md`](../glossary.md)). **Shipped:** `file_churn` + **`churn-complexity-hotspots`** recipe (see [architecture § `file_churn`](../architecture.md#file_churn--git-churn-metrics-per-indexed-file-strict)). ### Tier 12 — Module-graph topology — **PARTIAL (2026-05-15)** diff --git a/docs/roadmap.md b/docs/roadmap.md index 37d3733b..7513e951 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -111,7 +111,7 @@ Predicate-as-API only — enrich row shape and audit deltas; no standalone pass/ - [ ] **`history` table** (deferred — revisit-triggered) — temporal queries: "when did symbol X get `@deprecated`?", "coverage trend over last 50 commits", "files that became dead this week". `audit --base ` covers the most-common temporal question (PR-scoped diff) without schema growth, so the table earns its place only when bigger questions emerge. Two shapes (per-commit snapshots ~N × DB size; append-only event log heavier CTE walks); both pay an N-reindexes backfill cost (~30s per reindex). **Revisit triggers:** two consumers ship `jq`-based "audit-runs-over-time" workflows, OR `query_baselines` evolution becomes a recurring agent need. - [ ] **`codemap audit` verdict + thresholds** (v1.x) — `verdict: "pass" | "warn" | "fail"` driven by an `audit.deltas[].{added_max, action}` field on the config object (`.codemap/config.{ts,js,json}`). Triggers: two consumers ship `jq`-based threshold scripts with similar shapes, OR one consumer asks with a concrete config sketch. Until then, raw deltas + consumer-side `jq` is the CI exit-code idiom. **Likely accelerant:** the Marketplace Action (next item) shipping is the most plausible path to firing the trigger — once `- uses: stainless-code/codemap@v1` is the dominant CI path, real `jq` threshold scripts will surface. - [ ] **GitHub Marketplace Action — publish + listing finish** — core Action implementation is in-tree: root `action.yml`, `query --ci`, `audit --format sarif` / `--ci`, package-manager detection, dogfood smoke, and opt-in `pr-comment` summary renderer have shipped. Remaining work is the release/listing slice: `MARKETPLACE.md`, `v1.0.0` / floating `v1` tags, Marketplace setup, sacrificial-repo smoke, and making `action-smoke` blocking once the Action tag exists. Action version stream is independent of CLI version (`package.json` currently drives CLI/npm version; Action publishes at its own `v1.0.0`). Plan: [`plans/github-marketplace-action.md`](./plans/github-marketplace-action.md). Effort: S. -- [ ] **Churn × complexity hotspots** — `file_churn` table (git `log --numstat` over indexed paths, recency-weighted commits, optional trend) + bundled recipe **`churn-complexity-hotspots`** JOINing `symbols.complexity` for ranked refactor targets. Distinct from outcome alias `hotspots` → `fan-in`. Score is a recipe column, not a verdict ([Moat A](./roadmap.md#moats-load-bearing)). Plan: [`plans/churn-complexity-hotspots.md`](./plans/churn-complexity-hotspots.md). Effort: L–M. +- [x] **Churn × complexity hotspots** — `file_churn` (git ingest every index + `ingest-churn` / `churn.file` fallback), recipe **`churn-complexity-hotspots`** (file/symbol grain, normalized score, trend). Alias `hotspots` → `fan-in`. Contract: [architecture § `file_churn`](./architecture.md#file_churn--git-churn-metrics-per-indexed-file-strict), [glossary § file_churn](./glossary.md#file_churn). - [x] **AST-hash duplication** — `symbols.body_hash` (canonical body AST, identifiers → `$id`, literals → kind, absent returns → `Literal:nullish`; function-shaped symbols; skip `body_line_count < 2`) + partial index + bundled `duplicates` recipe (per-symbol rows, CTE `GROUP BY`). **Different shape from token-level suffix-array dupes.** Contract: [architecture § `symbols` table](./architecture.md#symbols--functions-constants-classes-interfaces-types-enums-strict), [glossary § body_hash](./glossary.md#symbolsbody_hash--structural-duplicate-bodies). Effort: M. - [ ] **Falsifiable benchmark CI on named external fixtures** — structural-cost A/B (indexed queries vs `find` + `grep` + `Read`-loop discovery) on zod, fastify, vue-core, next.js. Numbers land in [`docs/benchmark.md`](./benchmark.md); headline figures surface in `MARKETPLACE.md` only after external runs land. Harness: [benchmark § Agent eval harness](./benchmark.md#agent-eval-harness) + external fixture extension; pair with **Agent eval: quality × tokens × wall** for scored completion metrics. **Partial:** manual [`.github/workflows/agent-eval-external.yml`](../.github/workflows/agent-eval-external.yml) for in-repo fixture paths (not zod/fastify/nightly). Effort: M. **Self-index regression guardrail shipped** (#96): `bun run check:perf-baseline` + weekly scheduled workflow (demoted from PR hard gate — GHA runner variance). - [ ] **In-repo test bench scale (optional)** — if `fixtures/minimal` outgrows one corpus: add committed `fixtures/bench/` or rename `minimal`→`bench`. Harness map: [`testing-coverage.md`](./testing-coverage.md), [`fixtures/README.md`](../fixtures/README.md). diff --git a/docs/testing-coverage.md b/docs/testing-coverage.md index a9c9e8c0..423c40b1 100644 --- a/docs/testing-coverage.md +++ b/docs/testing-coverage.md @@ -54,30 +54,31 @@ Every `templates/recipes/.sql` has **≥1** scenario in `fixtures/golden/sce Recipe goldens prove query _behavior_; these scenarios pin **persisted rows** on the minimal corpus. Pin-down ids enforced in CI: `SUBSTRATE_SCENARIO_BY_TABLE` in `scripts/query-golden-coverage-matrix.test.mjs` (plus every bundled recipe id). `index-table-stats` and `call-resolution-stats` are additional aggregate/residual scenarios on the same corpus. -| Table | Scenario id | -| ------------------------------------- | ------------------------------------------------------- | -| Aggregate counts (all indexed tables) | `index-table-stats` | -| `meta` | `meta-fts5-enabled`, `call-resolution-stats` (residual) | -| `source_fts` | `source-fts-row-count` | -| `file_metrics` | `file-metrics-complexity-fixture` | -| `scopes` | `scopes-product-card` | -| `references` | `references-product-card-perms` | -| `bindings` | `bindings-createClient` | -| `import_specifiers` | `import-specifiers-consumer` | -| `async_calls` | `async-calls-prefetch` | -| `decorators` | `decorators-sealed` | -| `dynamic_imports` | `dynamic-imports-prefetch` | -| `module_cycles` | `module-cycles-cache-store` | -| `re_export_chains` | `re-export-chains-product-card` | -| `runtime_markers` | `runtime-markers-env` | -| `function_params` | `function-params-createClient` | -| `boundary_rules` | `boundary-rules-ui-no-api` | -| `unresolved_calls` | `unresolved-call-sites` | -| `calls` (resolution) | `calls-createClient-resolved` | -| `try_catch` | `try-catch-rethrow-heuristics` | -| `jsdoc_tags` | `jsdoc-tags-createClient` | -| `suppressions` | `suppressions-orphan` | -| `coverage` | `coverage-rows-after-ingest` (+ killer recipes) | +| Table | Scenario id | +| ------------------------------------- | -------------------------------------------------------- | +| Aggregate counts (all indexed tables) | `index-table-stats` | +| `meta` | `meta-fts5-enabled`, `call-resolution-stats` (residual) | +| `source_fts` | `source-fts-row-count` | +| `file_metrics` | `file-metrics-complexity-fixture` | +| `file_churn` | `churn-complexity-hotspots` (seed via `seed-file-churn`) | +| `scopes` | `scopes-product-card` | +| `references` | `references-product-card-perms` | +| `bindings` | `bindings-createClient` | +| `import_specifiers` | `import-specifiers-consumer` | +| `async_calls` | `async-calls-prefetch` | +| `decorators` | `decorators-sealed` | +| `dynamic_imports` | `dynamic-imports-prefetch` | +| `module_cycles` | `module-cycles-cache-store` | +| `re_export_chains` | `re-export-chains-product-card` | +| `runtime_markers` | `runtime-markers-env` | +| `function_params` | `function-params-createClient` | +| `boundary_rules` | `boundary-rules-ui-no-api` | +| `unresolved_calls` | `unresolved-call-sites` | +| `calls` (resolution) | `calls-createClient-resolved` | +| `try_catch` | `try-catch-rethrow-heuristics` | +| `jsdoc_tags` | `jsdoc-tags-createClient` | +| `suppressions` | `suppressions-orphan` | +| `coverage` | `coverage-rows-after-ingest` (+ killer recipes) | Core graph tables (`files`, `symbols`, `imports`, …) are covered by many existing SQL/recipe scenarios — see `fixtures/golden/scenarios.json`. diff --git a/fixtures/CAPABILITIES.json b/fixtures/CAPABILITIES.json index cbebddd7..aefaafec 100644 --- a/fixtures/CAPABILITIES.json +++ b/fixtures/CAPABILITIES.json @@ -184,6 +184,21 @@ ], "goldenScenarios": ["duplicates"] }, + { + "id": "churn.file-churn", + "description": "file_churn git metrics and churn-complexity-hotspots recipe", + "fixtureFiles": [ + "src/lib/complexity-fixture.ts", + "src/lib/cache.ts", + "src/api/client.ts", + "file-churn-seed.json" + ], + "goldenScenarios": [ + "churn-complexity-hotspots", + "churn-complexity-hotspots-by-symbol", + "churn-complexity-hotspots-path-prefix" + ] + }, { "id": "boundaries.suppressions", "description": "boundary_rules, suppressions, config-driven violations", diff --git a/fixtures/benchmark/perf-baseline.json b/fixtures/benchmark/perf-baseline.json index f420cdac..fd721b75 100644 --- a/fixtures/benchmark/perf-baseline.json +++ b/fixtures/benchmark/perf-baseline.json @@ -1,13 +1,15 @@ { - "captured_at": "2026-05-25T16:09:06.000Z", - "commit": "3911abf", + "captured_at": "2026-06-10T13:38:17.300Z", + "commit": "36106ff66b691bfbe682da5e5cb29a1779e24a1e", "phases": { - "collect_ms": 18, - "parse_ms": 559, - "insert_ms": 231, - "index_create_ms": 137, - "bindings_ms": 153, - "total_ms": 1117 + "collect_ms": 14, + "parse_ms": 299, + "insert_ms": 282, + "index_create_ms": 149, + "bindings_ms": 174, + "churn_ms": 182, + "churn_idle_ms": 7, + "total_ms": 1194 }, "regression_pct": 25, "noise_floor_ms": 10 diff --git a/fixtures/golden/minimal/churn-complexity-hotspots-by-symbol.json b/fixtures/golden/minimal/churn-complexity-hotspots-by-symbol.json new file mode 100644 index 00000000..069d5dbb --- /dev/null +++ b/fixtures/golden/minimal/churn-complexity-hotspots-by-symbol.json @@ -0,0 +1,132 @@ +[ + { + "file_path": "src/lib/complexity-fixture.ts", + "symbol_name": "labyrinth", + "symbol_kind": "function", + "line_start": 22, + "weighted_commits": 22.5, + "commit_count": 28, + "churn_trend": "accelerating", + "max_complexity": 19, + "avg_complexity": 19, + "hotspot_score": 427.5, + "hotspot_score_normalized": 100 + }, + { + "file_path": "src/lib/complexity-fixture.ts", + "symbol_name": "deeplyNested", + "symbol_kind": "function", + "line_start": 6, + "weighted_commits": 22.5, + "commit_count": 28, + "churn_trend": "accelerating", + "max_complexity": 5, + "avg_complexity": 5, + "hotspot_score": 112.5, + "hotspot_score_normalized": 26.3 + }, + { + "file_path": "src/lib/complexity-fixture.ts", + "symbol_name": "catchDirectRethrow", + "symbol_kind": "function", + "line_start": 98, + "weighted_commits": 22.5, + "commit_count": 28, + "churn_trend": "accelerating", + "max_complexity": 2, + "avg_complexity": 2, + "hotspot_score": 45, + "hotspot_score_normalized": 10.5 + }, + { + "file_path": "src/lib/complexity-fixture.ts", + "symbol_name": "catchInnerArrowRethrow", + "symbol_kind": "function", + "line_start": 86, + "weighted_commits": 22.5, + "commit_count": 28, + "churn_trend": "accelerating", + "max_complexity": 2, + "avg_complexity": 2, + "hotspot_score": 45, + "hotspot_score_normalized": 10.5 + }, + { + "file_path": "src/lib/complexity-fixture.ts", + "symbol_name": "relay", + "symbol_kind": "function", + "line_start": 90, + "weighted_commits": 22.5, + "commit_count": 28, + "churn_trend": "accelerating", + "max_complexity": 1, + "avg_complexity": 1, + "hotspot_score": 22.5, + "hotspot_score_normalized": 5.3 + }, + { + "file_path": "src/api/client.ts", + "symbol_name": "createClient", + "symbol_kind": "function", + "line_start": 20, + "weighted_commits": 8, + "commit_count": 14, + "churn_trend": "cooling", + "max_complexity": 2, + "avg_complexity": 2, + "hotspot_score": 16, + "hotspot_score_normalized": 3.7 + }, + { + "file_path": "src/lib/cache.ts", + "symbol_name": "get", + "symbol_kind": "function", + "line_start": 8, + "weighted_commits": 4.2, + "commit_count": 9, + "churn_trend": "stable", + "max_complexity": 3, + "avg_complexity": 3, + "hotspot_score": 12.6, + "hotspot_score_normalized": 2.9 + }, + { + "file_path": "src/lib/cache.ts", + "symbol_name": "invalidate", + "symbol_kind": "function", + "line_start": 16, + "weighted_commits": 4.2, + "commit_count": 9, + "churn_trend": "stable", + "max_complexity": 3, + "avg_complexity": 3, + "hotspot_score": 12.6, + "hotspot_score_normalized": 2.9 + }, + { + "file_path": "src/api/client.ts", + "symbol_name": "handshake", + "symbol_kind": "function", + "line_start": 37, + "weighted_commits": 8, + "commit_count": 14, + "churn_trend": "cooling", + "max_complexity": 1, + "avg_complexity": 1, + "hotspot_score": 8, + "hotspot_score_normalized": 1.9 + }, + { + "file_path": "src/api/client.ts", + "symbol_name": "legacyClient", + "symbol_kind": "function", + "line_start": 46, + "weighted_commits": 8, + "commit_count": 14, + "churn_trend": "cooling", + "max_complexity": 1, + "avg_complexity": 1, + "hotspot_score": 8, + "hotspot_score_normalized": 1.9 + } +] diff --git a/fixtures/golden/minimal/churn-complexity-hotspots-path-prefix.json b/fixtures/golden/minimal/churn-complexity-hotspots-path-prefix.json new file mode 100644 index 00000000..a3a02611 --- /dev/null +++ b/fixtures/golden/minimal/churn-complexity-hotspots-path-prefix.json @@ -0,0 +1,28 @@ +[ + { + "file_path": "src/lib/complexity-fixture.ts", + "symbol_name": null, + "symbol_kind": null, + "line_start": null, + "weighted_commits": 22.5, + "commit_count": 28, + "churn_trend": "accelerating", + "max_complexity": 19, + "avg_complexity": 5.8, + "hotspot_score": 427.5, + "hotspot_score_normalized": 100 + }, + { + "file_path": "src/lib/cache.ts", + "symbol_name": null, + "symbol_kind": null, + "line_start": null, + "weighted_commits": 4.2, + "commit_count": 9, + "churn_trend": "stable", + "max_complexity": 3, + "avg_complexity": 3, + "hotspot_score": 12.6, + "hotspot_score_normalized": 2.9 + } +] diff --git a/fixtures/golden/minimal/churn-complexity-hotspots.json b/fixtures/golden/minimal/churn-complexity-hotspots.json new file mode 100644 index 00000000..d359d4c2 --- /dev/null +++ b/fixtures/golden/minimal/churn-complexity-hotspots.json @@ -0,0 +1,41 @@ +[ + { + "file_path": "src/lib/complexity-fixture.ts", + "symbol_name": null, + "symbol_kind": null, + "line_start": null, + "weighted_commits": 22.5, + "commit_count": 28, + "churn_trend": "accelerating", + "max_complexity": 19, + "avg_complexity": 5.8, + "hotspot_score": 427.5, + "hotspot_score_normalized": 100 + }, + { + "file_path": "src/api/client.ts", + "symbol_name": null, + "symbol_kind": null, + "line_start": null, + "weighted_commits": 8, + "commit_count": 14, + "churn_trend": "cooling", + "max_complexity": 2, + "avg_complexity": 1.2, + "hotspot_score": 16, + "hotspot_score_normalized": 3.7 + }, + { + "file_path": "src/lib/cache.ts", + "symbol_name": null, + "symbol_kind": null, + "line_start": null, + "weighted_commits": 4.2, + "commit_count": 9, + "churn_trend": "stable", + "max_complexity": 3, + "avg_complexity": 3, + "hotspot_score": 12.6, + "hotspot_score_normalized": 2.9 + } +] diff --git a/fixtures/golden/minimal/files-count.json b/fixtures/golden/minimal/files-count.json index c68f12eb..6ae0eac1 100644 --- a/fixtures/golden/minimal/files-count.json +++ b/fixtures/golden/minimal/files-count.json @@ -1,5 +1,5 @@ [ { - "n": 45 + "n": 46 } ] diff --git a/fixtures/golden/minimal/files-hashes.json b/fixtures/golden/minimal/files-hashes.json index f181d16f..8c5d6bb0 100644 --- a/fixtures/golden/minimal/files-hashes.json +++ b/fixtures/golden/minimal/files-hashes.json @@ -17,6 +17,12 @@ "language": "md", "line_count": 60 }, + { + "path": "file-churn-seed.json", + "content_hash": "9f5241c6e2dc081b0398228178dd10dd6e37f9efc0ef2157e7d417a81b7ffd0b", + "language": "json", + "line_count": 33 + }, { "path": "package.json", "content_hash": "a8e79efc943697cee4d4435360de9150bd7261376abe9e2c34c0c7238177c7de", diff --git a/fixtures/golden/minimal/files-largest.json b/fixtures/golden/minimal/files-largest.json index 2797e814..c44a7998 100644 --- a/fixtures/golden/minimal/files-largest.json +++ b/fixtures/golden/minimal/files-largest.json @@ -23,6 +23,12 @@ "size": 1068, "language": "ts" }, + { + "path": "file-churn-seed.json", + "line_count": 33, + "size": 819, + "language": "json" + }, { "path": "src/utils/date.ts", "line_count": 29, @@ -112,11 +118,5 @@ "line_count": 11, "size": 418, "language": "md" - }, - { - "path": "src/bench/jsx-synthesis/PageShell.tsx", - "line_count": 10, - "size": 153, - "language": "tsx" } ] diff --git a/fixtures/golden/minimal/index-summary.json b/fixtures/golden/minimal/index-summary.json index fee69116..cbb12431 100644 --- a/fixtures/golden/minimal/index-summary.json +++ b/fixtures/golden/minimal/index-summary.json @@ -1,6 +1,6 @@ [ { - "files": 45, + "files": 46, "symbols": 110, "imports": 26, "components": 5, diff --git a/fixtures/golden/minimal/index-table-stats.json b/fixtures/golden/minimal/index-table-stats.json index b618b394..220d5987 100644 --- a/fixtures/golden/minimal/index-table-stats.json +++ b/fixtures/golden/minimal/index-table-stats.json @@ -1,6 +1,6 @@ [ { - "files": 45, + "files": 46, "symbols": 110, "imports": 26, "exports": 64, @@ -32,6 +32,7 @@ "jsdoc_tags": 10, "jsx_attributes": 10, "boundary_rules": 1, - "suppressions": 1 + "suppressions": 1, + "file_churn": 46 } ] diff --git a/fixtures/golden/minimal/source-fts-row-count.json b/fixtures/golden/minimal/source-fts-row-count.json index c68f12eb..6ae0eac1 100644 --- a/fixtures/golden/minimal/source-fts-row-count.json +++ b/fixtures/golden/minimal/source-fts-row-count.json @@ -1,5 +1,5 @@ [ { - "n": 45 + "n": 46 } ] diff --git a/fixtures/golden/scenarios.json b/fixtures/golden/scenarios.json index bbd6d341..2e202b69 100644 --- a/fixtures/golden/scenarios.json +++ b/fixtures/golden/scenarios.json @@ -44,6 +44,11 @@ "prompt": "Row counts for main tables (same SQL as --recipe index-summary)", "recipe": "index-summary" }, + { + "id": "index-table-stats", + "prompt": "Row counts for all indexed substrate tables (mirrors fetchTableStats + post-pass tables)", + "sql": "SELECT (SELECT COUNT(*) FROM files) AS files, (SELECT COUNT(*) FROM symbols) AS symbols, (SELECT COUNT(*) FROM imports) AS imports, (SELECT COUNT(*) FROM exports) AS exports, (SELECT COUNT(*) FROM components) AS components, (SELECT COUNT(*) FROM dependencies) AS dependencies, (SELECT COUNT(*) FROM markers) AS markers, (SELECT COUNT(*) FROM type_members) AS type_members, (SELECT COUNT(*) FROM type_heritage) AS type_heritage, (SELECT COUNT(*) FROM calls) AS calls, (SELECT COUNT(*) FROM css_variables) AS css_vars, (SELECT COUNT(*) FROM css_classes) AS css_classes, (SELECT COUNT(*) FROM css_keyframes) AS css_keyframes, (SELECT COUNT(*) FROM scopes) AS scopes, (SELECT COUNT(*) FROM \"references\") AS ref_count, (SELECT COUNT(*) FROM bindings) AS bindings, (SELECT COUNT(*) FROM import_specifiers) AS import_specifiers, (SELECT COUNT(*) FROM function_params) AS function_params, (SELECT COUNT(*) FROM runtime_markers) AS runtime_markers, (SELECT COUNT(*) FROM test_suites) AS test_suites, (SELECT COUNT(*) FROM re_export_chains) AS re_export_chains, (SELECT COUNT(*) FROM module_cycles) AS module_cycles, (SELECT COUNT(*) FROM dynamic_imports) AS dynamic_imports, (SELECT COUNT(*) FROM file_metrics) AS file_metrics, (SELECT COUNT(*) FROM unresolved_calls) AS unresolved_calls, (SELECT COUNT(*) FROM jsx_elements) AS jsx_elements, (SELECT COUNT(*) FROM async_calls) AS async_calls, (SELECT COUNT(*) FROM decorators) AS decorators, (SELECT COUNT(*) FROM try_catch) AS try_catch, (SELECT COUNT(*) FROM jsdoc_tags) AS jsdoc_tags, (SELECT COUNT(*) FROM jsx_attributes) AS jsx_attributes, (SELECT COUNT(*) FROM boundary_rules) AS boundary_rules, (SELECT COUNT(*) FROM suppressions) AS suppressions, (SELECT COUNT(*) FROM file_churn) AS file_churn" + }, { "id": "imports-consumer-alias", "prompt": "Which alias import does consumer.ts use for the API client?", @@ -462,6 +467,32 @@ "prompt": "Function-shaped symbols with identical structural body_hash across files.", "recipe": "duplicates" }, + { + "id": "churn-complexity-hotspots", + "prompt": "Files ranked by git churn × cyclomatic complexity (hotspot_score).", + "recipe": "churn-complexity-hotspots", + "preSetup": [ + { "kind": "seed-file-churn", "path": "file-churn-seed.json" } + ] + }, + { + "id": "churn-complexity-hotspots-by-symbol", + "prompt": "Symbols ranked by file churn × per-symbol complexity.", + "recipe": "churn-complexity-hotspots", + "params": { "by_symbol": true, "row_limit": 10 }, + "preSetup": [ + { "kind": "seed-file-churn", "path": "file-churn-seed.json" } + ] + }, + { + "id": "churn-complexity-hotspots-path-prefix", + "prompt": "Churn hotspots scoped to src/lib/ via path_prefix.", + "recipe": "churn-complexity-hotspots", + "params": { "path_prefix": "src/lib/" }, + "preSetup": [ + { "kind": "seed-file-churn", "path": "file-churn-seed.json" } + ] + }, { "id": "circular-imports", "prompt": "Files in import cycles (SCCs of size >= 2) via Tarjan.", @@ -650,11 +681,6 @@ "prompt": "JSX attribute substrate on ProductCard", "sql": "SELECT a.name, a.value_kind FROM jsx_attributes a JOIN jsx_elements e ON e.id = a.element_id WHERE e.file_path = 'src/components/shop/ProductCard.tsx' ORDER BY a.name" }, - { - "id": "index-table-stats", - "prompt": "Row counts for all indexed substrate tables (mirrors fetchTableStats + post-pass tables)", - "sql": "SELECT (SELECT COUNT(*) FROM files) AS files, (SELECT COUNT(*) FROM symbols) AS symbols, (SELECT COUNT(*) FROM imports) AS imports, (SELECT COUNT(*) FROM exports) AS exports, (SELECT COUNT(*) FROM components) AS components, (SELECT COUNT(*) FROM dependencies) AS dependencies, (SELECT COUNT(*) FROM markers) AS markers, (SELECT COUNT(*) FROM type_members) AS type_members, (SELECT COUNT(*) FROM type_heritage) AS type_heritage, (SELECT COUNT(*) FROM calls) AS calls, (SELECT COUNT(*) FROM css_variables) AS css_vars, (SELECT COUNT(*) FROM css_classes) AS css_classes, (SELECT COUNT(*) FROM css_keyframes) AS css_keyframes, (SELECT COUNT(*) FROM scopes) AS scopes, (SELECT COUNT(*) FROM \"references\") AS ref_count, (SELECT COUNT(*) FROM bindings) AS bindings, (SELECT COUNT(*) FROM import_specifiers) AS import_specifiers, (SELECT COUNT(*) FROM function_params) AS function_params, (SELECT COUNT(*) FROM runtime_markers) AS runtime_markers, (SELECT COUNT(*) FROM test_suites) AS test_suites, (SELECT COUNT(*) FROM re_export_chains) AS re_export_chains, (SELECT COUNT(*) FROM module_cycles) AS module_cycles, (SELECT COUNT(*) FROM dynamic_imports) AS dynamic_imports, (SELECT COUNT(*) FROM file_metrics) AS file_metrics, (SELECT COUNT(*) FROM unresolved_calls) AS unresolved_calls, (SELECT COUNT(*) FROM jsx_elements) AS jsx_elements, (SELECT COUNT(*) FROM async_calls) AS async_calls, (SELECT COUNT(*) FROM decorators) AS decorators, (SELECT COUNT(*) FROM try_catch) AS try_catch, (SELECT COUNT(*) FROM jsdoc_tags) AS jsdoc_tags, (SELECT COUNT(*) FROM jsx_attributes) AS jsx_attributes, (SELECT COUNT(*) FROM boundary_rules) AS boundary_rules, (SELECT COUNT(*) FROM suppressions) AS suppressions" - }, { "id": "meta-fts5-enabled", "prompt": "meta.fts5_enabled after indexing minimal with fts5: true", diff --git a/fixtures/minimal/file-churn-seed.json b/fixtures/minimal/file-churn-seed.json new file mode 100644 index 00000000..c0cd9eab --- /dev/null +++ b/fixtures/minimal/file-churn-seed.json @@ -0,0 +1,32 @@ +[ + { + "file_path": "src/lib/complexity-fixture.ts", + "commit_count": 28, + "weighted_commits": 22.5, + "lines_added": 520, + "lines_removed": 180, + "last_commit_at": "2026-06-08T12:00:00Z", + "churn_trend": "accelerating", + "computed_at": "2026-06-10T00:00:00Z" + }, + { + "file_path": "src/lib/cache.ts", + "commit_count": 9, + "weighted_commits": 4.2, + "lines_added": 95, + "lines_removed": 40, + "last_commit_at": "2026-05-15T09:00:00Z", + "churn_trend": "stable", + "computed_at": "2026-06-10T00:00:00Z" + }, + { + "file_path": "src/api/client.ts", + "commit_count": 14, + "weighted_commits": 8.0, + "lines_added": 210, + "lines_removed": 90, + "last_commit_at": "2026-06-01T00:00:00Z", + "churn_trend": "cooling", + "computed_at": "2026-06-10T00:00:00Z" + } +] diff --git a/scripts/agent-eval/scenarios.json b/scripts/agent-eval/scenarios.json index c11eff39..8c7e6602 100644 --- a/scripts/agent-eval/scenarios.json +++ b/scripts/agent-eval/scenarios.json @@ -171,6 +171,15 @@ "regex": "export\\s+function\\s+\\w+", "mode": "files" } + }, + { + "id": "churn-complexity-hotspots-recipe", + "goldenId": "churn-complexity-hotspots", + "traditional": { + "globs": ["src/lib/**/*.ts", "src/api/**/*.ts"], + "regex": "export\\s+(function|const)\\s+\\w+", + "mode": "files" + } } ] } diff --git a/scripts/check-perf-baseline.ts b/scripts/check-perf-baseline.ts index 40cf282a..289dfb19 100644 --- a/scripts/check-perf-baseline.ts +++ b/scripts/check-perf-baseline.ts @@ -22,8 +22,13 @@ const BASELINE = join(REPO_ROOT, "fixtures/benchmark/perf-baseline.json"); const TMP_JSON = join(REPO_ROOT, ".perf-run.json"); const UPDATE_MODE = process.argv.includes("--update"); +/** Idle skip should stay well under full git churn; catches fallback to git log. */ +const IDLE_CHURN_MAX_MS = Number( + process.env.CODEMAP_PERF_IDLE_CHURN_MAX_MS ?? 50, +); -type Phase = +type ReportPhase = keyof Pick< + IndexPerformanceReport, | "collect_ms" | "parse_ms" | "insert_ms" @@ -31,17 +36,29 @@ type Phase = | "bindings_ms" | "module_cycles_ms" | "re_export_chains_ms" - | "total_ms"; + | "churn_ms" + | "total_ms" +>; -const GATED_PHASES: Phase[] = [ +/** Baseline-gated phases — `churn_idle_ms` is idle incremental `churn_ms`, not full-rebuild. */ +type GatedPhase = ReportPhase | "churn_idle_ms"; + +const GATED_PHASES: GatedPhase[] = [ "collect_ms", "parse_ms", "insert_ms", "index_create_ms", "bindings_ms", + "churn_ms", + "churn_idle_ms", "total_ms", ]; +/** Per-phase noise floors — `churn_idle_ms` is meta + rev-parse only, often < global floor. */ +const PHASE_NOISE_FLOOR_MS: Partial> = { + churn_idle_ms: 5, +}; + interface PhaseStats { median: number; min: number; @@ -57,7 +74,7 @@ function median(values: number[]): number { : sorted[mid]!; } -async function runOnce(): Promise { +async function runFullOnce(): Promise { if (existsSync(TMP_JSON)) rmSync(TMP_JSON); const proc = Bun.spawn(["bun", INDEXER, "--full", "--performance"], { cwd: REPO_ROOT, @@ -76,20 +93,69 @@ async function runOnce(): Promise { return JSON.parse(readFileSync(TMP_JSON, "utf-8")) as IndexPerformanceReport; } -async function collectStats(): Promise> { +async function runIdleChurnOnce(): Promise { + if (existsSync(TMP_JSON)) rmSync(TMP_JSON); + const proc = Bun.spawn(["bun", INDEXER, "--performance"], { + cwd: REPO_ROOT, + env: { ...process.env, CODEMAP_PERFORMANCE_JSON: TMP_JSON }, + stdout: "pipe", + stderr: "pipe", + }); + const exitCode = await proc.exited; + if (exitCode !== 0) { + const stderr = await new Response(proc.stderr).text(); + throw new Error(`idle indexer failed (exit ${exitCode}): ${stderr}`); + } + if (!existsSync(TMP_JSON)) { + throw new Error(`idle indexer did not write ${TMP_JSON}`); + } + const report = JSON.parse( + readFileSync(TMP_JSON, "utf-8"), + ) as IndexPerformanceReport; + if (report.churn_ms > IDLE_CHURN_MAX_MS) { + throw new Error( + `idle churn_ms=${report.churn_ms} exceeds ${IDLE_CHURN_MAX_MS} — idle skip may have failed (full git churn path?)`, + ); + } + return report.churn_ms; +} + +function noiseFloorFor(phase: GatedPhase, baseline: BaselineFile): number { + return PHASE_NOISE_FLOOR_MS[phase] ?? baseline.noise_floor_ms; +} + +async function collectStats(): Promise> { console.error(`Running indexer ${RUNS}× to collect per-phase medians…`); const reports: IndexPerformanceReport[] = []; for (let i = 0; i < RUNS; i++) { - const r = await runOnce(); + const r = await runFullOnce(); reports.push(r); console.error( - ` run ${i + 1}/${RUNS}: total_ms=${r.total_ms} bindings_ms=${r.bindings_ms}`, + ` run ${i + 1}/${RUNS}: total_ms=${r.total_ms} bindings_ms=${r.bindings_ms} churn_ms=${r.churn_ms}`, ); } + + console.error(`Running idle incremental ${RUNS}× for churn_idle_ms…`); + const idleRuns: number[] = []; + for (let i = 0; i < RUNS; i++) { + const churnMs = await runIdleChurnOnce(); + idleRuns.push(churnMs); + console.error(` idle ${i + 1}/${RUNS}: churn_ms=${churnMs}`); + } + if (existsSync(TMP_JSON)) rmSync(TMP_JSON); - const out = {} as Record; + const out = {} as Record; for (const phase of GATED_PHASES) { + if (phase === "churn_idle_ms") { + out[phase] = { + median: median(idleRuns), + min: Math.min(...idleRuns), + max: Math.max(...idleRuns), + runs: idleRuns, + }; + continue; + } const runs = reports.map((r) => r[phase]); out[phase] = { median: median(runs), @@ -104,7 +170,7 @@ async function collectStats(): Promise> { interface BaselineFile { captured_at: string; commit: string; - phases: Record; + phases: Record; /** Percent above baseline median that triggers a regression. */ regression_pct: number; /** Phases under this median (ms) skip gating — jitter dominates. */ @@ -143,7 +209,7 @@ async function main() { commit: gitHead(), phases: Object.fromEntries( GATED_PHASES.map((p) => [p, stats[p].median]), - ) as Record, + ) as Record, regression_pct: REGRESSION_PCT, noise_floor_ms: NOISE_FLOOR_MS, }; @@ -181,7 +247,7 @@ async function main() { for (const phase of GATED_PHASES) { const base = baseline.phases[phase]; const cur = stats[phase].median; - const gated = base >= baseline.noise_floor_ms; + const gated = base >= noiseFloorFor(phase, baseline); const overBudget = gated && cur > base * (1 + regressionPct / 100); if (overBudget) regressed = true; const flag = gated ? (overBudget ? "REGRESS" : "ok") : "skip(noise)"; diff --git a/scripts/churn-hotspots-recipe-scope.test.mjs b/scripts/churn-hotspots-recipe-scope.test.mjs new file mode 100644 index 00000000..12cb309d --- /dev/null +++ b/scripts/churn-hotspots-recipe-scope.test.mjs @@ -0,0 +1,33 @@ +import { describe, expect, it } from "bun:test"; +import { join } from "node:path"; + +import { $ } from "bun"; + +const REPO_ROOT = join(import.meta.dir, ".."); + +async function indexAndSeedChurn() { + await $`bun src/index.ts --full --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); + await $`bun src/index.ts ingest-churn file-churn-seed.json --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); +} + +describe("churn-complexity-hotspots path_prefix", () => { + it("path_prefix excludes files outside the subtree", async () => { + await indexAndSeedChurn(); + const result = + await $`bun src/index.ts query --recipe churn-complexity-hotspots --json --params path_prefix=src/lib/ --root fixtures/minimal` + .cwd(REPO_ROOT) + .quiet(); + expect(result.exitCode).toBe(0); + const rows = JSON.parse(result.stdout.toString()); + expect(rows.every((r) => r.file_path.startsWith("src/lib/"))).toBe(true); + expect(rows.some((r) => r.file_path === "src/api/client.ts")).toBe(false); + expect(rows.map((r) => r.file_path)).toEqual([ + "src/lib/complexity-fixture.ts", + "src/lib/cache.ts", + ]); + }); +}); diff --git a/scripts/query-golden-coverage-matrix.test.mjs b/scripts/query-golden-coverage-matrix.test.mjs index 5570da6c..cc03bbb0 100644 --- a/scripts/query-golden-coverage-matrix.test.mjs +++ b/scripts/query-golden-coverage-matrix.test.mjs @@ -15,6 +15,7 @@ const GOLDEN_DIR = join(REPO_ROOT, "fixtures/golden/minimal"); /** Tables that must have a dedicated SQL pin-down scenario (not recipe-only). */ const SUBSTRATE_SCENARIO_BY_TABLE = { file_metrics: "file-metrics-complexity-fixture", + file_churn: "churn-complexity-hotspots", scopes: "scopes-product-card", references: "references-product-card-perms", bindings: "bindings-createClient", @@ -78,6 +79,7 @@ describe("golden coverage matrix", () => { expect(scenarioIds.has("index-table-stats")).toBe(true); const scenario = scenarios.find((s) => s.id === "index-table-stats"); expect(scenario?.sql).toContain("FROM file_metrics"); + expect(scenario?.sql).toContain("FROM file_churn"); expect(scenario?.sql).toContain("FROM unresolved_calls"); expect(scenario?.sql).toContain('FROM "references"'); }); diff --git a/scripts/query-golden/run-setup.ts b/scripts/query-golden/run-setup.ts index fa1ee497..b0e27065 100644 --- a/scripts/query-golden/run-setup.ts +++ b/scripts/query-golden/run-setup.ts @@ -5,7 +5,9 @@ import { ingestIstanbul, ingestLcov, } from "../../src/application/coverage-engine"; -import { closeDb, openDb } from "../../src/db"; +import { parseChurnJsonPayload } from "../../src/application/ingest-churn-run"; +import { closeDb, openDb, replaceFileChurn } from "../../src/db"; +import type { FileChurnRow } from "../../src/db"; import type { GoldenSetupStep } from "./schema"; /** @@ -26,6 +28,35 @@ export function runGoldenSetup( db.run("DELETE FROM coverage"); continue; } + if (step.kind === "seed-file-churn") { + const absPath = resolve(fixtureRoot, step.path); + if ( + absPath !== fixtureAbs && + !absPath.startsWith(`${fixtureAbs}${sep}`) + ) { + throw new Error( + `query-golden setup: path must stay under fixture root (${step.path})`, + ); + } + if (!existsSync(absPath)) { + throw new Error( + `query-golden setup: missing file-churn seed ${absPath}`, + ); + } + let rows: FileChurnRow[]; + try { + rows = parseChurnJsonPayload( + JSON.parse(readFileSync(absPath, "utf-8")) as unknown, + ); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + throw new Error( + `query-golden setup: invalid file-churn seed ${absPath}: ${msg}`, + ); + } + replaceFileChurn(db, rows); + continue; + } if (step.kind !== "ingest-coverage") continue; const absPath = resolve(fixtureRoot, step.path); if ( diff --git a/scripts/query-golden/schema.ts b/scripts/query-golden/schema.ts index 49eaeb21..d457cca4 100644 --- a/scripts/query-golden/schema.ts +++ b/scripts/query-golden/schema.ts @@ -41,6 +41,11 @@ export const setupStepSchema = z.discriminatedUnion("kind", [ z.object({ kind: z.literal("clear-coverage"), }), + z.object({ + kind: z.literal("seed-file-churn"), + /** Path relative to fixture root (JSON array of FileChurnRow). */ + path: z.string().min(1), + }), ]); export type GoldenSetupStep = z.infer; diff --git a/src/application/churn-ingest.test.ts b/src/application/churn-ingest.test.ts new file mode 100644 index 00000000..015e908c --- /dev/null +++ b/src/application/churn-ingest.test.ts @@ -0,0 +1,496 @@ +import { afterEach, beforeEach, describe, expect, it } from "bun:test"; +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { resolveCodemapConfig } from "../config"; +import { + closeDb, + createSchema, + getMeta, + insertFile, + META_CHURN_CONFIG_FINGERPRINT, + META_CHURN_INDEXED_COMMIT, + setMeta, +} from "../db"; +import { initCodemap } from "../runtime"; +import { openCodemapDatabase } from "../sqlite-db"; +import { + computeChurnTrend, + ingestFileChurnFromGit, + refreshFileChurn, +} from "./churn-ingest"; + +let projectRoot: string; + +function fixtureEnv(dates: { + author: string; + committer: string; +}): NodeJS.ProcessEnv { + const e: NodeJS.ProcessEnv = {}; + for (const [k, v] of Object.entries(process.env)) { + if (k.startsWith("GIT_") || k.startsWith("HUSKY")) continue; + e[k] = v; + } + e.GIT_AUTHOR_DATE = dates.author; + e.GIT_COMMITTER_DATE = dates.committer; + return e; +} + +function git(args: string[], env?: NodeJS.ProcessEnv): void { + const r = spawnSync("git", args, { + cwd: projectRoot, + env: + env ?? + fixtureEnv({ + author: "2026-06-01T12:00:00Z", + committer: "2026-06-01T12:00:00Z", + }), + }); + if (r.status !== 0) { + throw new Error(`git ${args.join(" ")}: ${r.stderr.toString().trim()}`); + } +} + +function commitAll(message: string, env?: NodeJS.ProcessEnv): void { + git(["add", "."], env); + git(["commit", "-m", message, "--no-gpg-sign"], env); +} + +describe("computeChurnTrend", () => { + it("returns null when commit_count is below threshold", () => { + expect( + computeChurnTrend({ + commit_count: 3, + recent_weighted: 2, + older_weighted: 1, + }), + ).toBeNull(); + }); + + it("classifies accelerating when recent mass dominates", () => { + expect( + computeChurnTrend({ + commit_count: 8, + recent_weighted: 7, + older_weighted: 2, + }), + ).toBe("accelerating"); + }); + + it("classifies cooling when older mass dominates", () => { + expect( + computeChurnTrend({ + commit_count: 10, + recent_weighted: 1, + older_weighted: 9, + }), + ).toBe("cooling"); + }); + + it("classifies stable when recent and older mass are balanced", () => { + expect( + computeChurnTrend({ + commit_count: 8, + recent_weighted: 5, + older_weighted: 5, + }), + ).toBe("stable"); + }); +}); + +describe("ingestFileChurnFromGit", () => { + beforeEach(() => { + projectRoot = mkdtempSync(join(tmpdir(), "codemap-churn-")); + git(["init", "-q", "-b", "main"]); + git(["config", "user.email", "t@example.com"]); + git(["config", "user.name", "T"]); + git(["config", "commit.gpgsign", "false"]); + }); + + afterEach(() => { + rmSync(projectRoot, { recursive: true, force: true }); + }); + + it("populates file_churn for indexed paths from git history", () => { + mkdirSync(join(projectRoot, "src"), { recursive: true }); + writeFileSync(join(projectRoot, "src/hot.ts"), "export const a = 1;\n"); + commitAll("add hot"); + writeFileSync(join(projectRoot, "src/hot.ts"), "export const a = 2;\n"); + commitAll("edit hot"); + + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/hot.ts", + content_hash: "h", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + + const result = ingestFileChurnFromGit(db, { + projectRoot, + quiet: true, + }); + expect(result.ok).toBe(true); + expect(result.rowCount).toBe(1); + expect(result.elapsedMs).toBeGreaterThanOrEqual(0); + + const row = db + .query<{ + file_path: string; + commit_count: number; + weighted_commits: number; + }>( + "SELECT file_path, commit_count, weighted_commits FROM file_churn WHERE file_path = ?", + ) + .get("src/hot.ts"); + expect(row?.commit_count).toBe(2); + expect(row?.weighted_commits).toBeGreaterThan(0); + } finally { + closeDb(db); + } + }); + + it("respects since ref (only commits after anchor)", () => { + mkdirSync(join(projectRoot, "src"), { recursive: true }); + writeFileSync(join(projectRoot, "src/a.ts"), "v1\n"); + commitAll( + "old", + fixtureEnv({ + author: "2024-01-01T12:00:00Z", + committer: "2024-01-01T12:00:00Z", + }), + ); + const anchor = spawnSync("git", ["rev-parse", "HEAD"], { + cwd: projectRoot, + encoding: "utf-8", + }) + .stdout.toString() + .trim(); + writeFileSync(join(projectRoot, "src/a.ts"), "v2\n"); + commitAll( + "new", + fixtureEnv({ + author: "2026-06-01T12:00:00Z", + committer: "2026-06-01T12:00:00Z", + }), + ); + + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/a.ts", + content_hash: "a", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + const result = ingestFileChurnFromGit(db, { + projectRoot, + since: anchor, + quiet: true, + }); + expect(result.ok).toBe(true); + const row = db + .query<{ commit_count: number }>( + "SELECT commit_count FROM file_churn WHERE file_path = 'src/a.ts'", + ) + .get(); + expect(row?.commit_count).toBe(1); + } finally { + closeDb(db); + } + }); + + it("skips gracefully when not a git repository", () => { + const noGit = mkdtempSync(join(tmpdir(), "codemap-no-git-")); + try { + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/x.ts", + content_hash: "x", + size: 1, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + const result = ingestFileChurnFromGit(db, { + projectRoot: noGit, + quiet: true, + }); + expect(result.ok).toBe(false); + expect(result.reason).toContain("not a git"); + const n = db + .query<{ c: number }>("SELECT COUNT(*) AS c FROM file_churn") + .get()?.c; + expect(n).toBe(0); + } finally { + closeDb(db); + } + } finally { + rmSync(noGit, { recursive: true, force: true }); + } + }); +}); + +describe("refreshFileChurn", () => { + beforeEach(() => { + projectRoot = mkdtempSync(join(tmpdir(), "codemap-churn-refresh-")); + initCodemap(resolveCodemapConfig(projectRoot, undefined)); + git(["init", "-q", "-b", "main"]); + git(["config", "user.email", "t@example.com"]); + git(["config", "user.name", "T"]); + git(["config", "commit.gpgsign", "false"]); + mkdirSync(join(projectRoot, "src"), { recursive: true }); + writeFileSync(join(projectRoot, "src/a.ts"), "export const a = 1;\n"); + commitAll("seed"); + }); + + afterEach(() => { + rmSync(projectRoot, { recursive: true, force: true }); + }); + + it("idle skip requires populated file_churn even when HEAD meta matches", () => { + const head = spawnSync("git", ["rev-parse", "HEAD"], { + cwd: projectRoot, + encoding: "utf-8", + }) + .stdout.toString() + .trim(); + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/a.ts", + content_hash: "a", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + setMeta(db, META_CHURN_INDEXED_COMMIT, head); + setMeta(db, META_CHURN_CONFIG_FINGERPRINT, "90|"); + + const result = refreshFileChurn(db, { + projectRoot, + mode: "idle", + halfLifeDays: 90, + since: null, + quiet: true, + }); + expect(result.reason).not.toBe("skipped: HEAD unchanged"); + expect(result.ok).toBe(true); + expect(result.rowCount).toBeGreaterThan(0); + } finally { + closeDb(db); + } + }); + + it("skips git when config churn.file is set", () => { + writeFileSync( + join(projectRoot, "churn.json"), + JSON.stringify([ + { + file_path: "src/a.ts", + commit_count: 99, + weighted_commits: 88, + lines_added: 1, + lines_removed: 0, + last_commit_at: "2026-06-01T00:00:00Z", + churn_trend: "stable", + computed_at: "2026-06-10T00:00:00Z", + }, + ]), + ); + initCodemap( + resolveCodemapConfig(projectRoot, { churn: { file: "churn.json" } }), + ); + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/a.ts", + content_hash: "a", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + const result = refreshFileChurn(db, { + projectRoot, + quiet: true, + }); + expect(result.ok).toBe(true); + expect(result.reason).toBe("config churn.file"); + const row = db + .query<{ commit_count: number }>( + "SELECT commit_count FROM file_churn WHERE file_path = 'src/a.ts'", + ) + .get(); + expect(row?.commit_count).toBe(99); + } finally { + closeDb(db); + } + }); + + it("idle skip when HEAD, rows, and config fingerprint match", () => { + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/a.ts", + content_hash: "a", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + ingestFileChurnFromGit(db, { projectRoot, quiet: true }); + const result = refreshFileChurn(db, { + projectRoot, + mode: "idle", + halfLifeDays: 90, + since: null, + quiet: true, + }); + expect(result.reason).toBe("skipped: HEAD unchanged"); + expect(result.elapsedMs).toBeLessThan(50); + } finally { + closeDb(db); + } + }); + + it("incremental scope merges churn for changed paths only", () => { + mkdirSync(join(projectRoot, "src"), { recursive: true }); + writeFileSync(join(projectRoot, "src/a.ts"), "export const a = 1;\n"); + writeFileSync(join(projectRoot, "src/b.ts"), "export const b = 1;\n"); + commitAll("seed both"); + writeFileSync(join(projectRoot, "src/a.ts"), "export const a = 2;\n"); + commitAll("edit a once"); + + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + for (const path of ["src/a.ts", "src/b.ts"]) { + insertFile(db, { + path, + content_hash: path, + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + } + ingestFileChurnFromGit(db, { projectRoot, quiet: true }); + const bBefore = db + .query<{ commit_count: number }>( + "SELECT commit_count FROM file_churn WHERE file_path = 'src/b.ts'", + ) + .get()?.commit_count; + + writeFileSync(join(projectRoot, "src/a.ts"), "export const a = 3;\n"); + commitAll("edit a again"); + const scoped = ingestFileChurnFromGit(db, { + projectRoot, + scopePaths: ["src/a.ts"], + quiet: true, + }); + expect(scoped.ok).toBe(true); + const aAfter = db + .query<{ commit_count: number }>( + "SELECT commit_count FROM file_churn WHERE file_path = 'src/a.ts'", + ) + .get()?.commit_count; + const bAfter = db + .query<{ commit_count: number }>( + "SELECT commit_count FROM file_churn WHERE file_path = 'src/b.ts'", + ) + .get()?.commit_count; + expect(aAfter).toBe(3); + expect(bAfter).toBe(bBefore); + } finally { + closeDb(db); + } + }); + + it("incremental scope falls back to full refresh when config fingerprint drifts", () => { + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/a.ts", + content_hash: "a", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + setMeta(db, META_CHURN_CONFIG_FINGERPRINT, "30|v1.0.0"); + ingestFileChurnFromGit(db, { projectRoot, quiet: true }); + const fullCount = db + .query<{ c: number }>("SELECT COUNT(*) AS c FROM file_churn") + .get()?.c; + const scoped = refreshFileChurn(db, { + projectRoot, + mode: "incremental", + changedPaths: ["src/a.ts"], + halfLifeDays: 90, + since: null, + quiet: true, + }); + expect(scoped.ok).toBe(true); + const afterCount = db + .query<{ c: number }>("SELECT COUNT(*) AS c FROM file_churn") + .get()?.c; + expect(afterCount).toBe(fullCount); + expect(getMeta(db, META_CHURN_CONFIG_FINGERPRINT)).toBe("90|"); + } finally { + closeDb(db); + } + }); + + it("deletions mode stamps meta without running git log", () => { + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/a.ts", + content_hash: "a", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + ingestFileChurnFromGit(db, { projectRoot, quiet: true }); + const result = refreshFileChurn(db, { + projectRoot, + mode: "deletions", + halfLifeDays: 90, + since: null, + quiet: true, + }); + expect(result.reason).toBe("deletions: churn pruned via CASCADE"); + expect(getMeta(db, META_CHURN_INDEXED_COMMIT)).toBeTruthy(); + } finally { + closeDb(db); + } + }); +}); diff --git a/src/application/churn-ingest.ts b/src/application/churn-ingest.ts new file mode 100644 index 00000000..6e5bedde --- /dev/null +++ b/src/application/churn-ingest.ts @@ -0,0 +1,505 @@ +import { spawnSync } from "node:child_process"; +import { existsSync, realpathSync } from "node:fs"; +import { relative, resolve } from "node:path"; + +import { + getMeta, + mergeFileChurnForPaths, + META_CHURN_CONFIG_FINGERPRINT, + META_CHURN_INDEXED_COMMIT, + pruneFileChurnOrphans, + replaceFileChurn, + setMeta, +} from "../db"; +import type { FileChurnRow } from "../db"; +import { + getChurnFilePath, + getChurnHalfLifeDays, + getChurnSince, + getProjectRoot, +} from "../runtime"; +import type { CodemapDatabase } from "../sqlite-db"; +import { ingestChurnFromConfigPath } from "./ingest-churn-run"; + +export const DEFAULT_CHURN_HALF_LIFE_DAYS = 90; +const MIN_COMMITS_FOR_TREND = 4; +const TREND_ACCELERATING_RATIO = 0.6; +const TREND_COOLING_RATIO = 0.4; + +/** Strip inherited GIT_* so subprocess targets the resolved repo root. */ +function gitSpawnEnv(): NodeJS.ProcessEnv { + const e: NodeJS.ProcessEnv = {}; + for (const [k, v] of Object.entries(process.env)) { + if (k.startsWith("GIT_")) continue; + e[k] = v; + } + return e; +} + +function gitTopLevel(projectRoot: string): string | null { + const r = spawnSync("git", ["rev-parse", "--show-toplevel"], { + cwd: projectRoot, + env: gitSpawnEnv(), + }); + if (r.status !== 0) return null; + return r.stdout.toString().trim(); +} + +function toPosix(p: string): string { + return p.split("\\").join("/"); +} + +function resolveRealPath(path: string): string { + try { + return realpathSync(path); + } catch { + return resolve(path); + } +} + +/** Map a git path (repo-relative) to an indexed `files.path` under `projectRoot`. */ +function indexedPathFromGit( + gitPath: string, + projectPrefix: string, +): string | null { + const p = toPosix(gitPath); + if (projectPrefix === "") return p; + const prefix = `${projectPrefix}/`; + if (!p.startsWith(prefix)) return null; + return p.slice(prefix.length); +} + +function indexedToGitPath(filePath: string, projectPrefix: string): string { + if (projectPrefix === "" || projectPrefix === ".") return filePath; + return `${projectPrefix}/${filePath}`; +} + +export type ChurnTrend = "accelerating" | "stable" | "cooling"; + +export type ChurnRefreshMode = "full" | "incremental" | "idle" | "deletions"; + +export interface ChurnIngestResult { + ok: boolean; + rowCount: number; + elapsedMs: number; + /** User/agent-readable skip reason (non-git, git error, idle cache hit). */ + reason?: string; +} + +interface ChurnAcc { + commit_count: number; + weighted_commits: number; + recent_weighted: number; + older_weighted: number; + lines_added: number; + lines_removed: number; + last_commit_at: string | null; + last_commit_ts: number; +} + +/** + * Classify churn trend from recency-split weighted commit mass. + * Recent window = commits within `halfLifeDays / 2`; older = the rest. + */ +export function computeChurnTrend( + acc: Pick, +): ChurnTrend | null { + if (acc.commit_count < MIN_COMMITS_FOR_TREND) return null; + const total = acc.recent_weighted + acc.older_weighted; + if (total <= 0) return null; + const ratio = acc.recent_weighted / total; + if (ratio >= TREND_ACCELERATING_RATIO) return "accelerating"; + if (ratio <= TREND_COOLING_RATIO) return "cooling"; + return "stable"; +} + +function accsToRows( + byFile: Map, + computedAt: string, +): FileChurnRow[] { + return [...byFile.entries()].map(([file_path, a]) => ({ + file_path, + commit_count: a.commit_count, + weighted_commits: Math.round(a.weighted_commits * 1000) / 1000, + lines_added: a.lines_added, + lines_removed: a.lines_removed, + last_commit_at: a.last_commit_at, + churn_trend: computeChurnTrend(a), + computed_at: computedAt, + })); +} + +function parseGitNumstatLog( + stdout: string, + options: { + indexedPaths: Set; + projectPrefix: string; + halfLife: number; + scopeFilter?: Set; + }, +): Map { + const nowSec = Math.floor(Date.now() / 1000); + const recentWindowDays = options.halfLife / 2; + const byFile = new Map(); + let commitTs = 0; + + for (const line of stdout.split("\n")) { + if (line.startsWith("COMMIT ")) { + const parts = line.split(" "); + commitTs = Number(parts[2] ?? 0); + continue; + } + if (!line.trim() || commitTs <= 0) continue; + const tab = line.indexOf("\t"); + if (tab < 0) continue; + const rest = line.slice(tab + 1); + const tab2 = rest.indexOf("\t"); + if (tab2 < 0) continue; + const addedRaw = line.slice(0, tab); + const removedRaw = rest.slice(0, tab2); + const gitPath = rest.slice(tab2 + 1); + if (addedRaw === "-" || removedRaw === "-") continue; + + const filePath = indexedPathFromGit(gitPath, options.projectPrefix); + if (!filePath || !options.indexedPaths.has(filePath)) continue; + if (options.scopeFilter && !options.scopeFilter.has(filePath)) continue; + + const added = Number(addedRaw) || 0; + const removed = Number(removedRaw) || 0; + const ageDays = Math.max(0, (nowSec - commitTs) / 86_400); + const weight = 0.5 ** (ageDays / options.halfLife); + + let acc = byFile.get(filePath); + if (!acc) { + acc = { + commit_count: 0, + weighted_commits: 0, + recent_weighted: 0, + older_weighted: 0, + lines_added: 0, + lines_removed: 0, + last_commit_at: null, + last_commit_ts: 0, + }; + byFile.set(filePath, acc); + } + acc.commit_count += 1; + acc.weighted_commits += weight; + if (ageDays <= recentWindowDays) { + acc.recent_weighted += weight; + } else { + acc.older_weighted += weight; + } + acc.lines_added += added; + acc.lines_removed += removed; + if (commitTs >= acc.last_commit_ts) { + acc.last_commit_ts = commitTs; + acc.last_commit_at = new Date(commitTs * 1000).toISOString(); + } + } + return byFile; +} + +function countFileChurn(db: CodemapDatabase): number { + return ( + db.query<{ n: number }>("SELECT COUNT(*) AS n FROM file_churn").get()?.n ?? + 0 + ); +} + +function resolveGitHead(projectRoot: string): string | null { + const r = spawnSync("git", ["rev-parse", "HEAD"], { + cwd: projectRoot, + env: gitSpawnEnv(), + }); + if (r.status !== 0) return null; + const head = r.stdout.toString().trim(); + return head.length > 0 ? head : null; +} + +function churnConfigFingerprint( + halfLifeDays: number, + since: string | null, +): string { + return `${halfLifeDays}|${since ?? ""}`; +} + +function stampChurnMeta( + db: CodemapDatabase, + projectRoot: string, + halfLifeDays: number, + since: string | null, +): void { + const head = resolveGitHead(projectRoot); + if (head) setMeta(db, META_CHURN_INDEXED_COMMIT, head); + setMeta( + db, + META_CHURN_CONFIG_FINGERPRINT, + churnConfigFingerprint(halfLifeDays, since), + ); +} + +function canIdleSkipChurn( + db: CodemapDatabase, + head: string | null, + prevHead: string | null, + halfLifeDays: number, + since: string | null, +): boolean { + if (!head || !prevHead || head !== prevHead) return false; + if (countFileChurn(db) === 0) return false; + const fp = churnConfigFingerprint(halfLifeDays, since); + return getMeta(db, META_CHURN_CONFIG_FINGERPRINT) === fp; +} + +function tryConfigChurnFallback( + db: CodemapDatabase, + projectRoot: string, + quiet: boolean, + finish: (partial: Omit) => ChurnIngestResult, +): ChurnIngestResult | null { + let churnFile: string | null = null; + try { + churnFile = getChurnFilePath(); + } catch { + return null; + } + if (!churnFile) return null; + const loaded = ingestChurnFromConfigPath(db, { + projectRoot, + churnFile, + }); + if (!loaded || !loaded.ok) return null; + if (!quiet) { + console.error( + `[churn] file_churn loaded from config churn.file: ${loaded.ingested} files`, + ); + } + return finish({ ok: true, rowCount: loaded.ingested }); +} + +/** + * Populate `file_churn` from `git log --numstat` scoped to indexed paths. + */ +export function ingestFileChurnFromGit( + db: CodemapDatabase, + options: { + projectRoot: string; + halfLifeDays?: number; + since?: string | null; + quiet?: boolean; + /** When set, only these indexed paths are recomputed (merge, not full replace). */ + scopePaths?: string[]; + }, +): ChurnIngestResult { + const t0 = performance.now(); + const projectRoot = resolveRealPath(resolve(options.projectRoot)); + const halfLife = options.halfLifeDays ?? DEFAULT_CHURN_HALF_LIFE_DAYS; + const since = options.since?.trim() || null; + const quiet = options.quiet ?? false; + const scopePaths = options.scopePaths; + const merge = scopePaths !== undefined && scopePaths.length > 0; + + const finish = ( + partial: Omit, + ): ChurnIngestResult => ({ + ...partial, + elapsedMs: Math.round(performance.now() - t0), + }); + + if (!existsSync(resolve(projectRoot, ".git"))) { + const top = gitTopLevel(projectRoot); + if (!top) { + if (!merge) replaceFileChurn(db, []); + const reason = "skipped: not a git repository (file_churn empty)"; + if (!quiet) console.error(`[churn] ${reason}`); + const fallback = tryConfigChurnFallback(db, projectRoot, quiet, finish); + return fallback ?? finish({ ok: false, rowCount: 0, reason }); + } + } + + const gitRootRaw = gitTopLevel(projectRoot); + if (!gitRootRaw) { + if (!merge) replaceFileChurn(db, []); + const reason = "skipped: git unavailable (file_churn empty)"; + if (!quiet) console.error(`[churn] ${reason}`); + const fallback = tryConfigChurnFallback(db, projectRoot, quiet, finish); + return fallback ?? finish({ ok: false, rowCount: 0, reason }); + } + + const indexedPaths = new Set( + db + .query<{ path: string }>("SELECT path FROM files") + .all() + .map((r) => r.path), + ); + if (indexedPaths.size === 0) { + replaceFileChurn(db, []); + return finish({ ok: true, rowCount: 0 }); + } + + const gitRoot = resolveRealPath(gitRootRaw); + const projectPrefix = toPosix(relative(gitRoot, projectRoot)); + + let pathspecArgs: string[]; + let scopeFilter: Set | undefined; + if (merge && scopePaths) { + scopeFilter = new Set(scopePaths); + pathspecArgs = scopePaths.map((p) => indexedToGitPath(p, projectPrefix)); + } else { + pathspecArgs = [ + projectPrefix === "" || projectPrefix === "." ? "." : projectPrefix, + ]; + } + + const logArgs = [ + "log", + "--numstat", + "--format=COMMIT %H %ct", + ...(since ? [`${since}..HEAD`] : []), + "--", + ...pathspecArgs, + ]; + const log = spawnSync("git", logArgs, { + cwd: gitRootRaw, + env: gitSpawnEnv(), + encoding: "utf-8", + maxBuffer: 64 * 1024 * 1024, + }); + if (log.status !== 0) { + const reason = `skipped: git log failed (${log.stderr?.toString().trim() || "unknown"})`; + if (!quiet) console.error(`[churn] ${reason}`); + const fallback = tryConfigChurnFallback(db, projectRoot, quiet, finish); + return fallback ?? finish({ ok: false, rowCount: 0, reason }); + } + + const computedAt = new Date().toISOString(); + const byFile = parseGitNumstatLog(log.stdout, { + indexedPaths, + projectPrefix, + halfLife, + scopeFilter, + }); + const rows = accsToRows(byFile, computedAt); + + if (merge && scopePaths) { + mergeFileChurnForPaths(db, rows, scopePaths); + pruneFileChurnOrphans(db); + } else { + replaceFileChurn(db, rows); + } + + const rowCount = countFileChurn(db); + if (!quiet && rows.length > 0) { + console.error( + `[churn] file_churn ${merge ? "merged" : "populated"}: ${rows.length} files (${rowCount} total)`, + ); + } + stampChurnMeta(db, projectRoot, halfLife, since); + return finish({ ok: true, rowCount }); +} + +/** Index-time churn refresh — git-native with config JSON fallback. */ +export function refreshFileChurn( + db: CodemapDatabase, + options?: { + quiet?: boolean; + projectRoot?: string; + halfLifeDays?: number; + since?: string | null; + mode?: ChurnRefreshMode; + changedPaths?: string[]; + }, +): ChurnIngestResult { + const t0 = performance.now(); + const quiet = options?.quiet ?? false; + const mode = options?.mode ?? "full"; + const projectRoot = options?.projectRoot ?? getProjectRoot(); + const halfLifeDays = options?.halfLifeDays ?? getChurnHalfLifeDays(); + const since = options?.since !== undefined ? options.since : getChurnSince(); + + let configChurnFile: string | null = null; + try { + configChurnFile = getChurnFilePath(); + } catch { + configChurnFile = null; + } + if (configChurnFile) { + const loaded = ingestChurnFromConfigPath(db, { + projectRoot, + churnFile: configChurnFile, + }); + const rowCount = countFileChurn(db); + if (!loaded?.ok) { + const reason = loaded?.error ?? "config churn.file ingest failed"; + if (!quiet) console.error(`[churn] ${reason}`); + return { + ok: false, + rowCount, + elapsedMs: Math.round(performance.now() - t0), + reason, + }; + } + if (!quiet) { + console.error( + `[churn] file_churn loaded from config churn.file: ${loaded.ingested} files`, + ); + } + return { + ok: true, + rowCount: loaded.ingested, + elapsedMs: Math.round(performance.now() - t0), + reason: "config churn.file", + }; + } + + const head = resolveGitHead(projectRoot); + const prevHead = getMeta(db, META_CHURN_INDEXED_COMMIT) ?? null; + + if ( + mode === "idle" && + canIdleSkipChurn(db, head, prevHead, halfLifeDays, since ?? null) + ) { + const rowCount = countFileChurn(db); + return { + ok: true, + rowCount, + elapsedMs: Math.round(performance.now() - t0), + reason: "skipped: HEAD unchanged", + }; + } + + if (mode === "deletions") { + const rowCount = countFileChurn(db); + stampChurnMeta(db, projectRoot, halfLifeDays, since ?? null); + return { + ok: true, + rowCount, + elapsedMs: Math.round(performance.now() - t0), + reason: "deletions: churn pruned via CASCADE", + }; + } + + const base = { + projectRoot, + halfLifeDays, + since: since ?? null, + quiet, + }; + + const fp = churnConfigFingerprint(halfLifeDays, since ?? null); + const storedFp = getMeta(db, META_CHURN_CONFIG_FINGERPRINT) ?? null; + if ( + mode === "incremental" && + options?.changedPaths && + options.changedPaths.length > 0 && + storedFp === fp + ) { + return ingestFileChurnFromGit(db, { + ...base, + scopePaths: options.changedPaths, + }); + } + + return ingestFileChurnFromGit(db, base); +} diff --git a/src/application/context-engine.test.ts b/src/application/context-engine.test.ts index dc6f4a9f..a828bffb 100644 --- a/src/application/context-engine.test.ts +++ b/src/application/context-engine.test.ts @@ -224,6 +224,14 @@ describe("readRecipeSqlLimit", () => { }); }); +describe("classifyIntent", () => { + it("maps hotspot/churn intents to refactor-priority recipe cards", () => { + const c = classifyIntent("which files churn often and are complex"); + expect(c.classified_as).toBe("refactor-priority"); + expect(c.matched_recipes[0]).toBe("churn-complexity-hotspots"); + }); +}); + describe("composeStartHere", () => { it("includes intent-ranked recipe cards and hub leaders with signatures", () => { withSeededDb((db) => { @@ -234,10 +242,10 @@ describe("composeStartHere", () => { ); expect(start.classified_as).toBe("refactor"); expect(start.recipes.map((r) => r.id)).toEqual([ + "churn-complexity-hotspots", "fan-in", "fan-out", "barrel-files", - "deprecated-symbols", ]); expect(start.index_summary.files).toBe(3); expect(start.recipes[0]?.tool).toBe("query_recipe"); @@ -252,6 +260,20 @@ describe("composeStartHere", () => { }); }); + it("sets churn_hint when file_churn is empty", () => { + withSeededDb((db) => { + const start = composeStartHere( + db, + defaultStartHereClassification(), + composeOpts(), + ); + expect(start.index_summary.file_churn).toBe(0); + expect(start.churn_hint).toContain("file_churn is empty"); + expect(start.churn_hint).toContain("ingest_churn"); + expect(start.churn_hint).toContain("churn-complexity-hotspots"); + }); + }); + it("uses explore defaults when no intent is supplied at envelope build time", () => { withSeededDb((db) => { const start = composeStartHere( @@ -491,10 +513,10 @@ describe("buildContextEnvelope", () => { }); expect(envelope.start_here?.classified_as).toBe("refactor"); expect(envelope.start_here?.recipes.map((r) => r.id)).toEqual([ + "churn-complexity-hotspots", "fan-in", "fan-out", "barrel-files", - "deprecated-symbols", ]); expect(envelope.intent?.classified_as).toBe("refactor"); }); diff --git a/src/application/context-engine.ts b/src/application/context-engine.ts index f4169290..f9b4b848 100644 --- a/src/application/context-engine.ts +++ b/src/application/context-engine.ts @@ -121,6 +121,7 @@ export interface ContextIndexSummary { imports: number; components: number; dependencies: number; + file_churn: number; } export interface ContextStartHere { @@ -130,6 +131,8 @@ export interface ContextStartHere { index_summary: ContextIndexSummary; recipes: ContextRecipeStarter[]; hub_leaders: ContextHubLeader[]; + /** Set when `file_churn` is empty — nudges git index or `ingest-churn` / `churn.file`. */ + churn_hint?: string; } export interface BuildContextEnvelopeOpts { @@ -149,16 +152,33 @@ export function classifyIntent(intent: string): { hint: string; } { const t = intent.toLowerCase(); + if ( + /hotspot|churn|refactor priority|risky to refactor|high.?churn|change.?often/.test( + t, + ) + ) { + return { + classified_as: "refactor-priority", + matched_recipes: [ + "churn-complexity-hotspots", + "refactor-risk-ranking", + "high-complexity-untested", + "fan-in", + ], + hint: "churn-complexity-hotspots ranks files by git churn × complexity (distinct from `hotspots` alias → fan-in); pair with refactor-risk-ranking and snippet before large edits.", + }; + } if (/refactor|rename|restructur|extract|move\b/.test(t)) { return { classified_as: "refactor", matched_recipes: [ + "churn-complexity-hotspots", "fan-in", "fan-out", "barrel-files", "deprecated-symbols", ], - hint: "Inspect fan-in / fan-out before moving symbols; barrel-files surfaces public-API hubs; deprecated-symbols flags risky callers.", + hint: "churn-complexity-hotspots surfaces high-churn × high-complexity files; inspect fan-in / fan-out before moving symbols.", }; } if (/bug|fix|debug|error|crash|broken|regress/.test(t)) { @@ -325,10 +345,11 @@ export function composeStartHere( ): ContextStartHere { const budget = resolveContextBudget(opts.fileCount); const fanInRows = opts.fanInRows ?? readFanInHubs(db, budget.hub_limit); - return { + const index_summary = readIndexSummary(db); + const start: ContextStartHere = { classified_as: classification.classified_as, hint: classification.hint, - index_summary: readIndexSummary(db), + index_summary, recipes: composeRecipeStarters(classification.matched_recipes), hub_leaders: composeHubLeadersFromRows(db, fanInRows, { projectRoot: opts.projectRoot, @@ -337,6 +358,11 @@ export function composeStartHere( signaturesPerHub: budget.signatures_per_hub, }), }; + if (index_summary.file_churn === 0) { + start.churn_hint = + "file_churn is empty — run `codemap` (git history) or `ingest_churn` / `codemap ingest-churn ` / config `churn.file` before `churn-complexity-hotspots`."; + } + return start; } export interface FanInHubRow { @@ -391,14 +417,18 @@ function stripTrailingSqlLineComments(sql: string): string { function readIndexSummary(db: CodemapDatabase): ContextIndexSummary { const row = db.query(QUERY_RECIPES["index-summary"]!.sql).get() as - | ContextIndexSummary + | Omit | undefined; + const file_churn = + db.query<{ n: number }>("SELECT COUNT(*) AS n FROM file_churn").get()?.n ?? + 0; return { files: row?.files ?? 0, symbols: row?.symbols ?? 0, imports: row?.imports ?? 0, components: row?.components ?? 0, dependencies: row?.dependencies ?? 0, + file_churn, }; } diff --git a/src/application/http-server.test.ts b/src/application/http-server.test.ts index 15b5ef0c..3886a942 100644 --- a/src/application/http-server.test.ts +++ b/src/application/http-server.test.ts @@ -888,6 +888,44 @@ describe("http-server — POST /tool/{other tools}", () => { expect(r.json.error).toContain("corrupt rows_json"); }); + it("ingest_churn returns 400 when path is missing", async () => { + serverHandle = await startServer(); + const r = await postTool(serverHandle.port, "ingest_churn", { + path: "no-such/churn.json", + }); + expect(r.status).toBe(400); + expect(r.json.error).toContain("churn file not found"); + }); + + it("ingest_churn ingests churn JSON successfully", async () => { + const churnDir = join(benchDir, "fixtures"); + mkdirSync(churnDir); + writeFileSync( + join(churnDir, "churn.json"), + JSON.stringify([ + { + file_path: "src/a.ts", + commit_count: 5, + weighted_commits: 4, + lines_added: 10, + lines_removed: 2, + last_commit_at: "2026-06-01T00:00:00Z", + churn_trend: "stable", + computed_at: "2026-06-10T00:00:00Z", + }, + ]), + ); + serverHandle = await startServer(); + const r = await postTool(serverHandle.port, "ingest_churn", { + path: "fixtures/churn.json", + }); + expect(r.status).toBe(200); + expect(r.json).toMatchObject({ + ingested: 1, + skipped_unindexed: 0, + }); + }); + it("ingest_coverage ingests istanbul artifact successfully", async () => { const db = openDb(); try { diff --git a/src/application/http-server.ts b/src/application/http-server.ts index befa787e..5948902a 100644 --- a/src/application/http-server.ts +++ b/src/application/http-server.ts @@ -43,6 +43,7 @@ import { handleAffected, handleContext, handleDropBaseline, + handleIngestChurn, handleIngestCoverage, handleExplore, handleImpact, @@ -57,6 +58,7 @@ import { handleSnippet, handleValidate, impactArgsSchema, + ingestChurnArgsSchema, ingestCoverageArgsSchema, nodeArgsSchema, traceArgsSchema, @@ -598,6 +600,12 @@ async function dispatchTool( result = await handleIngestCoverage(r.value, opts.root); break; } + case "ingest_churn": { + const r = validate(ingestChurnArgsSchema, args, "ingest_churn"); + if (!r.ok) return writeJson(res, 400, { error: r.error }, opts.version); + result = handleIngestChurn(r.value, opts.root); + break; + } default: { // Reachable only if MCP_TOOL_NAMES gains an entry without a switch arm — // the route guard above catches user-typed unknown names. @@ -648,7 +656,7 @@ function validate( * The browser sends the request (CORS only blocks the *response* from * being read by JS — the request itself reaches us and any side effect * executes). For state-changing tools (`save_baseline`, `drop_baseline`, - * `ingest_coverage`) this lets a malicious page mutate the developer's + * `ingest_coverage`, `ingest_churn`) this lets a malicious page mutate the developer's * `.codemap/index.db`. * * DNS rebinding extends the same attack: `evil.com` resolves to diff --git a/src/application/index-engine.ts b/src/application/index-engine.ts index 7860c6b3..07b23121 100644 --- a/src/application/index-engine.ts +++ b/src/application/index-engine.ts @@ -417,7 +417,8 @@ export function fetchTableStats(db: CodemapDatabase): IndexTableStats { (SELECT COUNT(*) FROM re_export_chains) as re_export_chains, (SELECT COUNT(*) FROM module_cycles) as module_cycles, (SELECT COUNT(*) FROM dynamic_imports) as dynamic_imports, - (SELECT COUNT(*) FROM file_metrics) as file_metrics`, + (SELECT COUNT(*) FROM file_metrics) as file_metrics, + (SELECT COUNT(*) FROM file_churn) as file_churn`, ) .get()!; return row as IndexTableStats; @@ -636,6 +637,7 @@ export async function indexFiles( module_cycles_ms: Math.round(moduleCyclesMs), re_export_chains_ms: Math.round(reExportChainsMs), heritage_ms: Math.round(heritageMs), + churn_ms: 0, total_ms: elapsed, slowest_files: slowest, }; diff --git a/src/application/ingest-churn-run.test.ts b/src/application/ingest-churn-run.test.ts new file mode 100644 index 00000000..42420b34 --- /dev/null +++ b/src/application/ingest-churn-run.test.ts @@ -0,0 +1,124 @@ +import { describe, expect, it } from "bun:test"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { closeDb, createSchema, insertFile } from "../db"; +import { openCodemapDatabase } from "../sqlite-db"; +import { + ingestChurnFromJsonFile, + parseChurnJsonPayload, +} from "./ingest-churn-run"; + +describe("ingestChurnFromJsonFile", () => { + it("loads indexed paths and skips unindexed rows", () => { + const root = mkdtempSync(join(tmpdir(), "codemap-ingest-churn-")); + try { + const jsonPath = join(root, "churn.json"); + writeFileSync( + jsonPath, + JSON.stringify([ + { + file_path: "src/a.ts", + commit_count: 5, + weighted_commits: 4, + lines_added: 10, + lines_removed: 2, + last_commit_at: "2026-06-01T00:00:00Z", + churn_trend: "stable", + computed_at: "2026-06-10T00:00:00Z", + }, + { + file_path: "src/missing.ts", + commit_count: 1, + weighted_commits: 1, + lines_added: 1, + lines_removed: 0, + last_commit_at: null, + churn_trend: null, + computed_at: "2026-06-10T00:00:00Z", + }, + ]), + ); + + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/a.ts", + content_hash: "a", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + const result = ingestChurnFromJsonFile(db, { + projectRoot: root, + path: "churn.json", + }); + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.ingested).toBe(1); + expect(result.skipped_unindexed).toBe(1); + const n = db + .query<{ c: number }>("SELECT COUNT(*) AS c FROM file_churn") + .get()?.c; + expect(n).toBe(1); + } finally { + closeDb(db); + } + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + it("rejects invalid churn_trend values", () => { + expect(() => + parseChurnJsonPayload([ + { + file_path: "src/a.ts", + commit_count: 1, + weighted_commits: 1, + churn_trend: "spiking", + }, + ]), + ).toThrow(/churn_trend must be accelerating/); + }); + + it("rejects empty JSON without wiping file_churn", () => { + const root = mkdtempSync(join(tmpdir(), "codemap-ingest-churn-empty-")); + try { + const jsonPath = join(root, "empty.json"); + writeFileSync(jsonPath, "[]"); + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/a.ts", + content_hash: "a", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + const result = ingestChurnFromJsonFile(db, { + projectRoot: root, + path: "empty.json", + }); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.error).toContain("at least one row"); + const n = db + .query<{ c: number }>("SELECT COUNT(*) AS c FROM file_churn") + .get()?.c; + expect(n).toBe(0); + } finally { + closeDb(db); + } + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/src/application/ingest-churn-run.ts b/src/application/ingest-churn-run.ts new file mode 100644 index 00000000..eec0ba78 --- /dev/null +++ b/src/application/ingest-churn-run.ts @@ -0,0 +1,178 @@ +import { spawnSync } from "node:child_process"; +import { existsSync, readFileSync } from "node:fs"; +import { isAbsolute, resolve } from "node:path"; + +import { + META_CHURN_CONFIG_FINGERPRINT, + META_CHURN_INDEXED_COMMIT, + pruneFileChurnOrphans, + replaceFileChurn, + setMeta, +} from "../db"; +import type { FileChurnRow } from "../db"; +import type { CodemapDatabase } from "../sqlite-db"; + +export interface IngestChurnRunOk { + ok: true; + ingested: number; + skipped_unindexed: number; + sourcePath: string; +} + +export interface IngestChurnRunError { + ok: false; + error: string; +} + +export type IngestChurnRunResult = IngestChurnRunOk | IngestChurnRunError; + +/** Strip inherited GIT_* so subprocess targets the project repo. */ +function gitSpawnEnv(): NodeJS.ProcessEnv { + const e: NodeJS.ProcessEnv = {}; + for (const [k, v] of Object.entries(process.env)) { + if (k.startsWith("GIT_")) continue; + e[k] = v; + } + return e; +} + +const CHURN_TREND_VALUES = new Set(["accelerating", "stable", "cooling"]); + +function parseChurnTrendField(value: unknown): string | null { + if (value === null || value === undefined) return null; + if (typeof value !== "string") { + throw new TypeError( + "churn_trend must be accelerating, stable, cooling, or null", + ); + } + if (!CHURN_TREND_VALUES.has(value)) { + throw new TypeError( + `churn_trend must be accelerating, stable, or cooling (got ${value})`, + ); + } + return value; +} + +export function parseChurnJsonPayload(raw: unknown): FileChurnRow[] { + if (!Array.isArray(raw)) { + throw new TypeError("churn JSON must be an array of file_churn rows"); + } + const rows: FileChurnRow[] = []; + for (const item of raw) { + if (item === null || typeof item !== "object") { + throw new TypeError("each churn row must be an object"); + } + const r = item as Record; + const file_path = r.file_path; + if (typeof file_path !== "string" || file_path.length === 0) { + throw new TypeError("file_path must be a non-empty string"); + } + if (rows.some((existing) => existing.file_path === file_path)) { + throw new TypeError(`duplicate file_path in churn JSON: ${file_path}`); + } + rows.push({ + file_path, + commit_count: Number(r.commit_count) || 0, + weighted_commits: Number(r.weighted_commits) || 0, + lines_added: Number(r.lines_added) || 0, + lines_removed: Number(r.lines_removed) || 0, + last_commit_at: + r.last_commit_at === null || r.last_commit_at === undefined + ? null + : String(r.last_commit_at), + churn_trend: parseChurnTrendField(r.churn_trend), + computed_at: + typeof r.computed_at === "string" + ? r.computed_at + : new Date().toISOString(), + }); + } + return rows; +} + +/** + * Load churn rows from JSON and replace `file_churn` (indexed paths only). + * Used by `codemap ingest-churn` and config `churn.file` fallback. + */ +export function ingestChurnFromJsonFile( + db: CodemapDatabase, + options: { projectRoot: string; path: string }, +): IngestChurnRunResult { + const absPath = isAbsolute(options.path) + ? options.path + : resolve(options.projectRoot, options.path); + if (!existsSync(absPath)) { + return { ok: false, error: `churn file not found: ${absPath}` }; + } + let rows: FileChurnRow[]; + try { + rows = parseChurnJsonPayload( + JSON.parse(readFileSync(absPath, "utf-8")) as unknown, + ); + } catch (err) { + return { + ok: false, + error: err instanceof Error ? err.message : String(err), + }; + } + + const indexed = new Set( + db + .query<{ path: string }>("SELECT path FROM files") + .all() + .map((r) => r.path), + ); + const kept: FileChurnRow[] = []; + let skipped = 0; + for (const row of rows) { + if (!indexed.has(row.file_path)) { + skipped += 1; + continue; + } + kept.push(row); + } + + if (kept.length === 0) { + return { + ok: false, + error: + rows.length === 0 + ? "churn JSON must contain at least one row" + : `churn JSON has no rows for indexed files (${skipped} skipped)`, + }; + } + + replaceFileChurn(db, kept); + pruneFileChurnOrphans(db); + const headResult = spawnSync("git", ["rev-parse", "HEAD"], { + cwd: options.projectRoot, + env: gitSpawnEnv(), + }); + if (headResult.status === 0) { + const head = headResult.stdout.toString().trim(); + if (head) { + setMeta(db, META_CHURN_INDEXED_COMMIT, head); + // JSON ingest has no half-life/since knobs — fingerprint marks manual import. + setMeta(db, META_CHURN_CONFIG_FINGERPRINT, "json|"); + } + } + + return { + ok: true, + ingested: kept.length, + skipped_unindexed: skipped, + sourcePath: absPath, + }; +} + +/** Config `churn.file` ingest when git churn is unavailable. */ +export function ingestChurnFromConfigPath( + db: CodemapDatabase, + options: { projectRoot: string; churnFile: string | null }, +): IngestChurnRunResult | null { + if (!options.churnFile) return null; + return ingestChurnFromJsonFile(db, { + projectRoot: options.projectRoot, + path: options.churnFile, + }); +} diff --git a/src/application/mcp-server.test.ts b/src/application/mcp-server.test.ts index 427f7aca..923917cf 100644 --- a/src/application/mcp-server.test.ts +++ b/src/application/mcp-server.test.ts @@ -1121,6 +1121,70 @@ describe("MCP server — ingest_coverage tool", () => { }); }); +describe("MCP server — ingest_churn tool", () => { + it("lists ingest_churn in tools/list", async () => { + const { client, server } = await makeClient(); + try { + const tools = await client.listTools(); + const names = tools.tools.map((t) => t.name); + expect(names).toContain("ingest_churn"); + } finally { + await server.close(); + } + }); + + it("ingests churn JSON and returns ingest envelope", async () => { + const churnDir = join(benchDir, "fixtures"); + mkdirSync(churnDir); + writeFileSync( + join(churnDir, "churn.json"), + JSON.stringify([ + { + file_path: "src/a.ts", + commit_count: 5, + weighted_commits: 4, + lines_added: 10, + lines_removed: 2, + last_commit_at: "2026-06-01T00:00:00Z", + churn_trend: "stable", + computed_at: "2026-06-10T00:00:00Z", + }, + ]), + ); + + const { client, server } = await makeClient(); + try { + const r = await client.callTool({ + name: "ingest_churn", + arguments: { path: "fixtures/churn.json" }, + }); + expect((r as { isError?: boolean }).isError).toBeUndefined(); + expect(readJson(r)).toMatchObject({ + ingested: 1, + skipped_unindexed: 0, + }); + } finally { + await server.close(); + } + }); + + it("returns isError when path is missing", async () => { + const { client, server } = await makeClient(); + try { + const r = await client.callTool({ + name: "ingest_churn", + arguments: { path: "no-such/churn.json" }, + }); + expect((r as { isError?: boolean }).isError).toBe(true); + expect(readJson(r)).toMatchObject({ + error: expect.stringContaining("churn file not found"), + }); + } finally { + await server.close(); + } + }); +}); + describe("MCP server — query baseline compare", () => { it("query with missing baseline returns isError", async () => { const { client, server } = await makeClient(); diff --git a/src/application/mcp-server.ts b/src/application/mcp-server.ts index a2dafe8b..7a25e093 100644 --- a/src/application/mcp-server.ts +++ b/src/application/mcp-server.ts @@ -50,6 +50,7 @@ import { handleAffected, handleContext, handleDropBaseline, + handleIngestChurn, handleIngestCoverage, exploreArgsSchema, handleExplore, @@ -67,6 +68,7 @@ import { handleSnippet, handleValidate, impactArgsSchema, + ingestChurnArgsSchema, ingestCoverageArgsSchema, listBaselinesArgsSchema, queryArgsSchema, @@ -89,7 +91,7 @@ import { * MCP server engine — owns the tool / resource registry. CLI shell * (`src/cli/cmd-mcp.ts`) handles argv + lifecycle only; this module is * the thin wrapper around `@modelcontextprotocol/sdk` that registers - * 20 JSON-RPC tools (CLI mirrors plus MCP/HTTP resource URIs) and MCP resources + * 21 JSON-RPC tools (CLI mirrors plus MCP/HTTP resource URIs) and MCP resources * (static + templates). Tool bodies are pure handlers in * `application/tool-handlers.ts` — same handlers `codemap serve` (HTTP) * dispatches. See [`docs/architecture.md` § MCP wiring]. @@ -181,6 +183,7 @@ export function createMcpServer(opts: ServerOpts): McpServer { maybeRegister("ingest_coverage", () => registerIngestCoverageTool(server, opts), ); + maybeRegister("ingest_churn", () => registerIngestChurnTool(server, opts)); maybeRegister("show", () => registerShowTool(server, opts)); maybeRegister("snippet", () => registerSnippetTool(server, opts)); maybeRegister("impact", () => registerImpactTool(server)); @@ -307,6 +310,18 @@ function registerIngestCoverageTool(server: McpServer, opts: ServerOpts): void { ); } +function registerIngestChurnTool(server: McpServer, opts: ServerOpts): void { + server.registerTool( + "ingest_churn", + withToolAnnotations("ingest_churn", { + description: + "Import precomputed git churn metrics into `file_churn` for non-git repos or CI fixtures. Same JSON envelope as `codemap ingest-churn` (no `--json` flag on MCP — payload is always JSON). Requires a prior index. Enables `churn-complexity-hotspots`. Args: `path` (required, relative to project root or absolute).", + inputSchema: ingestChurnArgsSchema, + }), + (args) => wrapToolResult(handleIngestChurn(args, opts.root)), + ); +} + function registerDropBaselineTool(server: McpServer): void { server.registerTool( "drop_baseline", diff --git a/src/application/mcp-tool-allowlist.ts b/src/application/mcp-tool-allowlist.ts index 17d6a9cb..89b08f57 100644 --- a/src/application/mcp-tool-allowlist.ts +++ b/src/application/mcp-tool-allowlist.ts @@ -24,6 +24,7 @@ export const MCP_TOOL_NAMES = [ "apply_rows", "apply_diff_input", "ingest_coverage", + "ingest_churn", ] as const; export type McpToolName = (typeof MCP_TOOL_NAMES)[number]; diff --git a/src/application/mcp-tool-annotations.test.ts b/src/application/mcp-tool-annotations.test.ts index e2d65d71..66cdfbe8 100644 --- a/src/application/mcp-tool-annotations.test.ts +++ b/src/application/mcp-tool-annotations.test.ts @@ -68,6 +68,7 @@ describe("mcp-tool-annotations", () => { "save_baseline", "drop_baseline", "ingest_coverage", + "ingest_churn", ] as const) { expect(getMcpToolAnnotations(name)).toMatchObject({ readOnlyHint: false, diff --git a/src/application/mcp-tool-annotations.ts b/src/application/mcp-tool-annotations.ts index 32b79eb2..d5078de0 100644 --- a/src/application/mcp-tool-annotations.ts +++ b/src/application/mcp-tool-annotations.ts @@ -82,6 +82,11 @@ export const MCP_TOOL_ANNOTATIONS = { destructiveHint: false, idempotentHint: false, }, + ingest_churn: { + readOnlyHint: false, + destructiveHint: false, + idempotentHint: false, + }, apply: { readOnlyHint: false, destructiveHint: true, diff --git a/src/application/run-index.ts b/src/application/run-index.ts index b4ec0fc4..8dbf6e43 100644 --- a/src/application/run-index.ts +++ b/src/application/run-index.ts @@ -1,3 +1,5 @@ +import { existsSync, readFileSync, writeFileSync } from "node:fs"; + import { createSchema, getMeta, @@ -8,6 +10,8 @@ import { import type { CodemapDatabase } from "../db"; import { getBoundaryRules, getFts5Enabled } from "../runtime"; import { getStateDir } from "../runtime"; +import { refreshFileChurn } from "./churn-ingest"; +import type { ChurnRefreshMode } from "./churn-ingest"; import { expandHeritageResolveScope } from "./heritage-resolver"; import { collectFiles, @@ -20,7 +24,11 @@ import { targetedReindex, } from "./index-engine"; import { acquireIndexLock } from "./index-lock"; -import type { IndexResult, IndexTableStats } from "./types"; +import type { + IndexPerformanceReport, + IndexResult, + IndexTableStats, +} from "./types"; /** * Returns `true` when the persisted `meta.fts5_enabled` differs from the @@ -73,9 +81,38 @@ function emptyStats(): IndexTableStats { module_cycles: 0, dynamic_imports: 0, file_metrics: 0, + file_churn: 0, }; } +function patchPerformanceJsonWithChurn(churnMs: number): void { + const perfJsonPath = process.env.CODEMAP_PERFORMANCE_JSON; + if (perfJsonPath === undefined || perfJsonPath === "") return; + try { + if (!existsSync(perfJsonPath)) { + writeFileSync( + perfJsonPath, + JSON.stringify({ churn_ms: churnMs } satisfies Pick< + IndexPerformanceReport, + "churn_ms" + >), + ); + return; + } + const perf = JSON.parse( + readFileSync(perfJsonPath, "utf-8"), + ) as IndexPerformanceReport; + writeFileSync( + perfJsonPath, + JSON.stringify({ ...perf, churn_ms: churnMs }, null, 2), + ); + } catch (err) { + console.error( + `[churn] failed to patch performance JSON: ${err instanceof Error ? err.message : err}`, + ); + } +} + /** * - `incremental` — git-based diff vs last indexed commit (default). * - `full` — re-glob and re-index everything. @@ -163,8 +200,12 @@ async function runCodemapIndexBody( // `finally` because full rebuild calls `dropAll` inside `indexFiles` which // wipes `boundary_rules` (config-derived); reconciling AFTER the index // pipeline returns survives that drop on every code path. + let result: IndexResult; + let churnMode: ChurnRefreshMode = "full"; + let churnChangedPaths: string[] | undefined; try { if (mode === "full") { + churnMode = "full"; if (!quiet) console.log(" Full rebuild requested..."); const collectStart = performance.now(); const files = collectFiles(); @@ -175,19 +216,18 @@ async function runCodemapIndexBody( collectMs, commit: options.commit, }); - return { + result = { mode: "full", indexed: run.indexed, skipped: run.skipped, elapsedMs: run.elapsedMs, stats: run.stats, }; - } - - if (mode === "files") { + } else if (mode === "files") { const targetFiles = options.files ?? []; if (targetFiles.length === 0) { - return { + churnMode = "idle"; + result = { mode: "files", indexed: 0, skipped: 0, @@ -195,91 +235,110 @@ async function runCodemapIndexBody( stats: emptyStats(), idle: true, }; - } - const run = await targetedReindex(db, targetFiles, quiet); - return { - mode: "files", - indexed: run.indexed, - skipped: run.skipped, - elapsedMs: run.elapsedMs, - stats: run.stats, - }; - } - - // getChangedFiles reads `meta`; the up-front createSchema above (before the toggle check) covers it. - const diff = getChangedFiles(db); - if (diff) { - if (!quiet) { - console.log( - ` Incremental: ${diff.changed.length} changed, ${diff.deleted.length} deleted`, - ); - } - if (diff.changed.length > 0) { - const indexedPaths = diff.existingPaths; - for (const f of diff.changed) indexedPaths.add(f); - const run = await indexFiles(db, diff.changed, false, indexedPaths, { - quiet, - sourceCache: diff.sourceCache, - existingHashes: diff.existingHashes, - deletedPaths: diff.deleted, - }); - return { - mode: "incremental", + } else { + churnMode = "incremental"; + churnChangedPaths = targetFiles; + const run = await targetedReindex(db, targetFiles, quiet); + result = { + mode: "files", indexed: run.indexed, skipped: run.skipped, elapsedMs: run.elapsedMs, stats: run.stats, }; } - if (diff.deleted.length > 0) { - deleteFilesFromIndex(db, diff.deleted, quiet); - const callScope = expandHeritageResolveScope(db, diff.deleted); - if (callScope.length > 0) { - runCallResolveAndSynthesis(db, callScope); + } else { + // getChangedFiles reads `meta`; the up-front createSchema above covers it. + const diff = getChangedFiles(db); + if (diff) { + if (!quiet) { + console.log( + ` Incremental: ${diff.changed.length} changed, ${diff.deleted.length} deleted`, + ); } - setMeta(db, "last_indexed_commit", getCurrentCommit()); - if (!quiet) console.log(" Index updated (deletions only)"); - return { - mode: "incremental", - indexed: 0, - skipped: 0, - elapsedMs: 0, - stats: fetchTableStats(db), - idle: true, + if (diff.changed.length > 0) { + churnMode = "incremental"; + churnChangedPaths = diff.changed; + const indexedPaths = diff.existingPaths; + for (const f of diff.changed) indexedPaths.add(f); + const run = await indexFiles(db, diff.changed, false, indexedPaths, { + quiet, + sourceCache: diff.sourceCache, + existingHashes: diff.existingHashes, + deletedPaths: diff.deleted, + }); + result = { + mode: "incremental", + indexed: run.indexed, + skipped: run.skipped, + elapsedMs: run.elapsedMs, + stats: run.stats, + }; + } else if (diff.deleted.length > 0) { + churnMode = "deletions"; + deleteFilesFromIndex(db, diff.deleted, quiet); + const callScope = expandHeritageResolveScope(db, diff.deleted); + if (callScope.length > 0) { + runCallResolveAndSynthesis(db, callScope); + } + setMeta(db, "last_indexed_commit", getCurrentCommit()); + if (!quiet) console.log(" Index updated (deletions only)"); + result = { + mode: "incremental", + indexed: 0, + skipped: 0, + elapsedMs: 0, + stats: fetchTableStats(db), + idle: true, + }; + } else { + churnMode = "idle"; + if (!quiet) console.log(" Index is up to date"); + result = { + mode: "incremental", + indexed: 0, + skipped: 0, + elapsedMs: 0, + stats: fetchTableStats(db), + idle: true, + }; + } + } else { + churnMode = "full"; + if (!quiet) { + console.log( + " No previous index or incompatible history, doing full rebuild...", + ); + } + const fallbackCollectStart = performance.now(); + const files = collectFiles(); + const fallbackCollectMs = performance.now() - fallbackCollectStart; + const run = await indexFiles(db, files, true, undefined, { + quiet, + performance: wantPerformance, + collectMs: fallbackCollectMs, + }); + result = { + mode: "full", + indexed: run.indexed, + skipped: run.skipped, + elapsedMs: run.elapsedMs, + stats: run.stats, }; } - if (!quiet) console.log(" Index is up to date"); - return { - mode: "incremental", - indexed: 0, - skipped: 0, - elapsedMs: 0, - stats: fetchTableStats(db), - idle: true, - }; - } - - if (!quiet) { - console.log( - " No previous index or incompatible history, doing full rebuild...", - ); } - const fallbackCollectStart = performance.now(); - const files = collectFiles(); - const fallbackCollectMs = performance.now() - fallbackCollectStart; - const run = await indexFiles(db, files, true, undefined, { - quiet, - performance: wantPerformance, - collectMs: fallbackCollectMs, - }); - return { - mode: "full", - indexed: run.indexed, - skipped: run.skipped, - elapsedMs: run.elapsedMs, - stats: run.stats, - }; } finally { reconcileBoundaryRules(db, getBoundaryRules()); } + + const churn = refreshFileChurn(db, { + quiet, + mode: churnMode, + changedPaths: churnChangedPaths, + }); + patchPerformanceJsonWithChurn(churn.elapsedMs); + if (wantPerformance && !quiet && churn.elapsedMs > 0) { + console.error(`[churn] ingest: ${churn.elapsedMs}ms`); + } + return { ...result, stats: fetchTableStats(db) }; } diff --git a/src/application/tool-handlers.ts b/src/application/tool-handlers.ts index c3e9ef9b..88a55f79 100644 --- a/src/application/tool-handlers.ts +++ b/src/application/tool-handlers.ts @@ -20,6 +20,7 @@ import { z } from "zod"; import { closeDb, + createSchema, deleteQueryBaseline, listQueryBaselines, openDb, @@ -53,6 +54,7 @@ import { buildContextEnvelope } from "./context-engine"; import { findImpact } from "./impact-engine"; import type { ImpactBackend, ImpactDirection } from "./impact-engine"; import { getCurrentCommit } from "./index-engine"; +import { ingestChurnFromJsonFile } from "./ingest-churn-run"; import { runIngestCoverageOnDb } from "./ingest-coverage-run"; import type { BadgeStyle } from "./output-formatters"; import { @@ -1258,6 +1260,50 @@ export async function handleIngestCoverage( } } +// === ingest_churn =========================================================== + +export const ingestChurnArgsSchema = { + path: z.string().min(1, "path must be a non-empty string"), +}; + +export interface IngestChurnArgs { + path: string; +} + +export function handleIngestChurn( + args: IngestChurnArgs, + root: string, +): ToolResult { + try { + const db = openDb(); + try { + createSchema(db); + const indexedCount = + db.query<{ n: number }>("SELECT COUNT(*) AS n FROM files").get()?.n ?? + 0; + if (indexedCount === 0) { + return err( + "codemap ingest-churn: no indexed files — run `codemap` or `codemap --full` first", + ); + } + const outcome = ingestChurnFromJsonFile(db, { + projectRoot: root, + path: args.path, + }); + if (!outcome.ok) return err(outcome.error); + return ok({ + ingested: outcome.ingested, + skipped_unindexed: outcome.skipped_unindexed, + sourcePath: outcome.sourcePath, + }); + } finally { + closeDb(db); + } + } catch (e) { + return err(e instanceof Error ? e.message : String(e), 500); + } +} + // === shared format helpers =================================================== /** diff --git a/src/application/types.ts b/src/application/types.ts index 797539c1..3e9bb944 100644 --- a/src/application/types.ts +++ b/src/application/types.ts @@ -29,6 +29,7 @@ export interface IndexTableStats extends Record { module_cycles: number; dynamic_imports: number; file_metrics: number; + file_churn: number; } /** @@ -53,6 +54,8 @@ export interface IndexPerformanceReport { re_export_chains_ms: number; /** `resolveTypeHeritage` + persist wall. */ heritage_ms: number; + /** `refreshFileChurn` wall (every index pass). */ + churn_ms: number; /** * `indexFiles` wall-clock — `parse + insert + index_create + DDL + bindings * + module_cycles + re_export_chains + heritage_ms`. Does **not** include diff --git a/src/cli/bootstrap-codemap.ts b/src/cli/bootstrap-codemap.ts index cc9f7f16..183dfd35 100644 --- a/src/cli/bootstrap-codemap.ts +++ b/src/cli/bootstrap-codemap.ts @@ -24,6 +24,8 @@ export interface BootstrapCodemapOpts { stateDir?: string | undefined; /** CLI `--with-fts`; `undefined` defers to `.codemap/config.ts` `fts5`. */ fts5Cli?: boolean | undefined; + /** CLI `--churn-since `; overrides config `churn.since`. */ + churnSinceCli?: string | undefined; } export async function bootstrapCodemap( @@ -39,7 +41,11 @@ export async function bootstrapCodemap( const user = await loadUserConfig(opts.root, opts.configFile, { stateDir }); initCodemap( - resolveCodemapConfig(opts.root, user, { stateDir, fts5Cli: opts.fts5Cli }), + resolveCodemapConfig(opts.root, user, { + stateDir, + fts5Cli: opts.fts5Cli, + churnSinceCli: opts.churnSinceCli, + }), ); configureResolver(getProjectRoot(), getTsconfigPath()); // Sanity: getStateDir() must mirror what we passed into resolveCodemapConfig. diff --git a/src/cli/bootstrap.ts b/src/cli/bootstrap.ts index 8e1d58cc..8ece0483 100644 --- a/src/cli/bootstrap.ts +++ b/src/cli/bootstrap.ts @@ -41,7 +41,7 @@ PR comment renderer (audit/SARIF → markdown summary): codemap pr-comment [--shape audit|sarif] [--json] # - for stdin MCP server (Model Context Protocol — for agent hosts): - codemap mcp # stdio JSON-RPC (20 tools; watcher default-ON) + codemap mcp # stdio JSON-RPC (21 tools; watcher default-ON) # CLI parity: query batch, trace, explore, node, file, schema, symbols, context --include-snippets HTTP server (for non-MCP consumers — CI scripts, curl, IDE plugins): @@ -79,6 +79,9 @@ Apply (substrate-shaped fix executor; diff-json row contract): Coverage ingest (Istanbul JSON or LCOV from any test runner): codemap ingest-coverage [--json] # path = file or dir; format auto-detected +Churn ingest (precomputed file_churn JSON for non-git repos or fixtures): + codemap ingest-churn [--json] + Other: codemap unlock [--force] Remove stale cross-process index lock codemap version @@ -91,6 +94,9 @@ Options: --state-dir DIR State directory for codemap-managed files (default .codemap/ under root) --performance Print per-phase timing breakdown + top-10 slowest files (full rebuild only) + --churn-since REF + Only count git commits after REF for file_churn ingest + (overrides config churn.since) --help, -h Show this help `); } @@ -147,6 +153,7 @@ export function validateIndexModeArgs(rest: string[]): void { if (rest[0] === "affected") return; if (rest[0] === "apply") return; if (rest[0] === "ingest-coverage") return; + if (rest[0] === "ingest-churn") return; if (rest[0] === "pr-comment") return; if (rest[0] === "trace") return; if (rest[0] === "explore") return; @@ -171,6 +178,14 @@ export function validateIndexModeArgs(rest: string[]): void { i++; continue; } + if (a === "--churn-since") { + if (i + 1 >= rest.length || rest[i + 1].startsWith("-")) { + console.error("codemap: --churn-since requires a git revision"); + process.exit(1); + } + i += 2; + continue; + } if (a === "--files") { i++; const start = i; @@ -199,6 +214,7 @@ export function parseBootstrapArgs(argv: string[]) { let configFile: string | undefined; let stateDir: string | undefined; let fts5Cli: boolean | undefined; + let churnSinceCli: string | undefined; const rest: string[] = []; for (let i = 0; i < argv.length; i++) { const a = argv[i]; @@ -221,10 +237,15 @@ export function parseBootstrapArgs(argv: string[]) { rest.push(a); continue; } + if (a === "--churn-since" && argv[i + 1]) { + churnSinceCli = argv[++i]; + rest.push("--churn-since", churnSinceCli); + continue; + } rest.push(a); } if (!root) root = process.cwd(); // --state-dir wins over CODEMAP_STATE_DIR (precedence per plan §D7). if (!stateDir) stateDir = process.env.CODEMAP_STATE_DIR; - return { root, configFile, stateDir, fts5Cli, rest }; + return { root, configFile, stateDir, fts5Cli, churnSinceCli, rest }; } diff --git a/src/cli/cmd-index.ts b/src/cli/cmd-index.ts index 0104709d..a3b02739 100644 --- a/src/cli/cmd-index.ts +++ b/src/cli/cmd-index.ts @@ -13,6 +13,7 @@ export async function runIndexCmd(opts: { configFile: string | undefined; stateDir?: string | undefined; fts5Cli?: boolean | undefined; + churnSinceCli?: string | undefined; rest: string[]; }): Promise { await bootstrapCodemap(opts); diff --git a/src/cli/cmd-ingest-churn.ts b/src/cli/cmd-ingest-churn.ts new file mode 100644 index 00000000..d5f27687 --- /dev/null +++ b/src/cli/cmd-ingest-churn.ts @@ -0,0 +1,123 @@ +import { ingestChurnFromJsonFile } from "../application/ingest-churn-run"; +import { closeDb, createSchema, openDb } from "../db"; +import { bootstrapCodemap } from "./bootstrap-codemap"; + +interface IngestChurnOpts { + root: string; + configFile: string | undefined; + stateDir?: string | undefined; + path: string; + json: boolean; +} + +export function printIngestChurnCmdHelp(): void { + console.log(`Usage: codemap ingest-churn [--json] + +Import precomputed git churn metrics into \`file_churn\` for non-git +repositories or CI fixtures (same JSON shape as config \`churn.file\`). +Replaces all \`file_churn\` rows — include every indexed path you want +retained. Enables \`churn-complexity-hotspots\`. JSON must be an array of +objects with \`file_path\`, \`commit_count\`, \`weighted_commits\`, optional +\`lines_added\` / \`lines_removed\` (default 0), \`last_commit_at\` / +\`churn_trend\` (\`accelerating\` \| \`stable\` \| \`cooling\`) / +\`computed_at\` (defaults to current time). Run \`codemap\` (index) first — +only indexed paths are kept; unindexed paths are skipped. + +Args: + Path to JSON file (relative to project root or absolute) + +Flags: + --json Emit result envelope on stdout + --help, -h Show this help + +Output (JSON): + { "ingested": N, "skipped_unindexed": K, "sourcePath": "..." } + +Examples: + codemap ingest-churn metrics/churn.json + codemap ingest-churn metrics/churn.json --json +`); +} + +export function parseIngestChurnRest( + rest: string[], +): + | { kind: "help" } + | { kind: "error"; message: string } + | { kind: "run"; path: string; json: boolean } { + if (rest[0] !== "ingest-churn") { + throw new Error("parseIngestChurnRest: expected ingest-churn"); + } + let path: string | undefined; + let json = false; + for (let i = 1; i < rest.length; i++) { + const a = rest[i]!; + if (a === "--help" || a === "-h") return { kind: "help" }; + if (a === "--json") { + json = true; + continue; + } + if (a.startsWith("-")) { + return { + kind: "error", + message: `codemap ingest-churn: unknown option "${a}"`, + }; + } + if (path !== undefined) { + return { + kind: "error", + message: "codemap ingest-churn: unexpected extra path argument", + }; + } + path = a; + } + if (!path) { + return { + kind: "error", + message: "codemap ingest-churn: missing argument", + }; + } + return { kind: "run", path, json }; +} + +export async function runIngestChurnCmd(opts: IngestChurnOpts): Promise { + await bootstrapCodemap(opts); + const db = openDb(); + try { + createSchema(db); + const indexedCount = + db.query<{ n: number }>("SELECT COUNT(*) AS n FROM files").get()?.n ?? 0; + if (indexedCount === 0) { + console.error( + "codemap ingest-churn: no indexed files — run `codemap` or `codemap --full` first", + ); + process.exit(1); + } + const result = ingestChurnFromJsonFile(db, { + projectRoot: opts.root, + path: opts.path, + }); + if (!result.ok) { + console.error(result.error); + process.exit(1); + } + if (opts.json) { + console.log( + JSON.stringify({ + ingested: result.ingested, + skipped_unindexed: result.skipped_unindexed, + sourcePath: result.sourcePath, + }), + ); + } else { + console.log( + ` Ingested ${result.ingested} file_churn rows from ${result.sourcePath}` + + (result.skipped_unindexed > 0 + ? ` (${result.skipped_unindexed} skipped — not in index)` + : ""), + ); + } + } finally { + closeDb(db); + } +} diff --git a/src/cli/cmd-mcp.ts b/src/cli/cmd-mcp.ts index e424bbe8..47191b35 100644 --- a/src/cli/cmd-mcp.ts +++ b/src/cli/cmd-mcp.ts @@ -85,7 +85,7 @@ Spawns an MCP (Model Context Protocol) server on stdio. Designed to be launched by an agent host (Claude Code, Cursor, Codex, generic MCP clients) — JSON-RPC on stdin/stdout, logs on stderr. -Tools (20; snake_case — mirrors CLI verbs where a shell twin exists): +Tools (21; snake_case — mirrors CLI verbs where a shell twin exists): query One read-only SQL statement (optional \`baseline\` for row diff; incompatible with non-json \`format\` / \`group_by\`). query_batch N statements in one round-trip (CLI: codemap query batch). @@ -97,6 +97,7 @@ Tools (20; snake_case — mirrors CLI verbs where a shell twin exists): list_baselines Catalog of saved baselines. drop_baseline Delete a baseline. ingest_coverage Load Istanbul/LCOV/V8 coverage into the index. + ingest_churn Load precomputed file_churn JSON into the index. context Project bootstrap envelope. validate On-disk hash vs indexed hash. show Symbol metadata: file:line + signature. @@ -129,7 +130,7 @@ Resources: codemap symbols . Output shape matches each tool's CLI JSON payload (always JSON for -query batch, trace, explore, node, file, schema, symbols, context, ingest_coverage; +query batch, trace, explore, node, file, schema, symbols, context, ingest_coverage, ingest_churn; optional \`--json\` on query/show/snippet/impact/affected/validate). MCP wraps payloads in \`{content: [{type: "text", text: …}]}\`; HTTP returns raw JSON. Run \`codemap skill\` or fetch \`codemap://skill\` for query examples. diff --git a/src/cli/main.ts b/src/cli/main.ts index 2b4a8f02..2194e9a3 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -18,7 +18,7 @@ import { emitJsonError } from "./emit-tool-result.js"; */ export async function main(): Promise { const argv = process.argv.slice(2); - const { root, configFile, stateDir, fts5Cli, rest } = + const { root, configFile, stateDir, fts5Cli, churnSinceCli, rest } = parseBootstrapArgs(argv); if (rest[0] === "--help" || rest[0] === "-h") { @@ -430,6 +430,28 @@ Copies bundled agent templates into .agents/ under the project root. return; } + if (rest[0] === "ingest-churn") { + const { parseIngestChurnRest, printIngestChurnCmdHelp, runIngestChurnCmd } = + await import("./cmd-ingest-churn.js"); + const parsed = parseIngestChurnRest(rest); + if (parsed.kind === "help") { + printIngestChurnCmdHelp(); + return; + } + if (parsed.kind === "error") { + console.error(parsed.message); + process.exit(1); + } + await runIngestChurnCmd({ + root, + configFile, + stateDir, + path: parsed.path, + json: parsed.json, + }); + return; + } + if (rest[0] === "ingest-coverage") { const { parseIngestCoverageRest, @@ -710,5 +732,12 @@ Copies bundled agent templates into .agents/ under the project root. } const { runIndexCmd } = await import("./cmd-index.js"); - await runIndexCmd({ root, configFile, stateDir, fts5Cli, rest }); + await runIndexCmd({ + root, + configFile, + stateDir, + fts5Cli, + churnSinceCli, + rest, + }); } diff --git a/src/config.test.ts b/src/config.test.ts index 3f6024f2..bcc0abd1 100644 --- a/src/config.test.ts +++ b/src/config.test.ts @@ -118,6 +118,30 @@ describe("resolveCodemapConfig", () => { expect(r.excludeDirNames).not.toEqual(DEFAULT_EXCLUDE_DIR_NAMES); }); + it("defaults churn halfLifeDays and null since/file", () => { + const r = resolveCodemapConfig(dir, undefined); + expect(r.churn.halfLifeDays).toBe(90); + expect(r.churn.since).toBeNull(); + expect(r.churn.file).toBeNull(); + }); + + it("resolves churn config and CLI since override", () => { + const r = resolveCodemapConfig( + dir, + { churn: { halfLifeDays: 30, since: "v1.0.0" } }, + { churnSinceCli: "abc123" }, + ); + expect(r.churn.halfLifeDays).toBe(30); + expect(r.churn.since).toBe("abc123"); + }); + + it("resolves churn.file to absolute path", () => { + const r = resolveCodemapConfig(dir, { + churn: { file: "churn-data.json" }, + }); + expect(r.churn.file).toBe(join(dir, "churn-data.json")); + }); + it("defaults boundaries to []", () => { const r = resolveCodemapConfig(dir, undefined); expect(r.boundaries).toEqual([]); diff --git a/src/config.ts b/src/config.ts index f0b2f63f..fd7de83e 100644 --- a/src/config.ts +++ b/src/config.ts @@ -100,6 +100,35 @@ export const codemapUserConfigSchema = z .describe( "Track per-recipe `last_run_at` + `run_count` in the `recipe_recency` table; surfaces inline on `--recipes-json` for agent-host ranking. Default `true` (opt-out). Set `false` to short-circuit every write — no rows ever land. Local-only — no upload primitive. See `docs/architecture.md` § `recipe_recency`.", ), + churn: z + .object({ + halfLifeDays: z + .number() + .positive() + .optional() + .describe( + "Exponential half-life (days) for `file_churn.weighted_commits`. Default `90`.", + ), + since: z + .string() + .min(1) + .optional() + .describe( + "Git revision (commit/tag/branch) — only commits after this ref contribute to churn. CLI `--churn-since` overrides.", + ), + file: z + .string() + .min(1) + .optional() + .describe( + "JSON array of `file_churn` rows (relative to project root). Used when git churn is unavailable; also loadable via `codemap ingest-churn`.", + ), + }) + .strict() + .optional() + .describe( + "Git churn ingest for `file_churn` (runs after every index pass).", + ), synthesis: z .object({ heuristicCalls: z @@ -225,6 +254,14 @@ export interface ResolvedCodemapConfig { readonly to_glob: string; readonly action: "deny" | "allow"; }>; + /** Git churn ingest tuning for `file_churn`. */ + readonly churn: { + readonly halfLifeDays: number; + /** When set, only commits after this git ref are counted. */ + readonly since: string | null; + /** Optional JSON churn import when git is unavailable. */ + readonly file: string | null; + }; /** When `heuristicCalls` is true, runs callback-synthesis after `resolveCalls`. */ readonly synthesis: { readonly heuristicCalls: boolean; @@ -273,6 +310,8 @@ export interface ResolveCodemapConfigOpts { * parsing in the bootstrap layer. */ fts5Cli?: boolean | undefined; + /** CLI `--churn-since ` — overrides config `churn.since`. */ + churnSinceCli?: string | undefined; } /** @@ -339,6 +378,20 @@ export function resolveCodemapConfig( const heuristicCalls = parsed?.synthesis?.heuristicCalls === true; + const churnHalfLifeDays = parsed?.churn?.halfLifeDays ?? 90; + const churnFile = parsed?.churn?.file + ? resolve(absRoot, parsed.churn.file) + : null; + let churnSince: string | null = parsed?.churn?.since ?? null; + if (opts.churnSinceCli !== undefined && opts.churnSinceCli !== "") { + churnSince = opts.churnSinceCli; + if (parsed?.churn?.since && parsed.churn.since !== opts.churnSinceCli) { + console.error( + `[churn] CLI override: --churn-since ${opts.churnSinceCli} (config churn.since ignored)`, + ); + } + } + const autoApply = parsed?.apply?.autoApplyRecipes; const applyAutoApplyRecipes = autoApply !== undefined && autoApply.length > 0 @@ -355,6 +408,11 @@ export function resolveCodemapConfig( fts5, boundaries, recipeRecency, + churn: { + halfLifeDays: churnHalfLifeDays, + since: churnSince, + file: churnFile, + }, synthesis: { heuristicCalls }, applyAutoApplyRecipes, }; diff --git a/src/db.ts b/src/db.ts index 0e7ebf3e..c889a609 100644 --- a/src/db.ts +++ b/src/db.ts @@ -3,7 +3,7 @@ import type { CodemapDatabase, BindValues } from "./sqlite-db"; /** Bump only on rebuild-forcing DDL changes (NOT on additive tables/columns). * See `docs/architecture.md` § Schema Versioning. */ -export const SCHEMA_VERSION = 39; +export const SCHEMA_VERSION = 40; /** Moat-A: default call-graph surfaces exclude callback-synthesis edges. */ export const CALLS_AST_ONLY_SQL = "(provenance IS NULL OR provenance = 'ast')"; @@ -93,6 +93,18 @@ export function createTables(db: CodemapDatabase) { export_count INTEGER NOT NULL DEFAULT 0 ) STRICT; + -- Git churn metrics per indexed file (populated each index pass via churn-ingest, golden uses seed-file-churn). + CREATE TABLE IF NOT EXISTS file_churn ( + file_path TEXT PRIMARY KEY REFERENCES files(path) ON DELETE CASCADE, + commit_count INTEGER NOT NULL, + weighted_commits REAL NOT NULL, + lines_added INTEGER NOT NULL, + lines_removed INTEGER NOT NULL, + last_commit_at TEXT, + churn_trend TEXT, + computed_at TEXT NOT NULL + ) STRICT; + CREATE TABLE IF NOT EXISTS imports ( id INTEGER PRIMARY KEY AUTOINCREMENT, file_path TEXT NOT NULL REFERENCES files(path) ON DELETE CASCADE, @@ -763,6 +775,7 @@ export function dropAll(db: CodemapDatabase) { DROP TABLE IF EXISTS runtime_markers; DROP TABLE IF EXISTS test_suites; DROP TABLE IF EXISTS file_metrics; + DROP TABLE IF EXISTS file_churn; DROP TABLE IF EXISTS unresolved_calls; DROP TABLE IF EXISTS bindings; DROP TABLE IF EXISTS "references"; @@ -1697,6 +1710,74 @@ export function insertFileMetrics(db: CodemapDatabase, rows: FileMetricsRow[]) { ); } +/** One row per indexed file with git churn metrics (see `file_churn` table). */ +export interface FileChurnRow { + file_path: string; + commit_count: number; + weighted_commits: number; + lines_added: number; + lines_removed: number; + last_commit_at: string | null; + churn_trend: string | null; + computed_at: string; +} + +export function insertFileChurn(db: CodemapDatabase, rows: FileChurnRow[]) { + batchInsert( + db, + rows, + "INSERT INTO file_churn (file_path, commit_count, weighted_commits, lines_added, lines_removed, last_commit_at, churn_trend, computed_at)", + "(?,?,?,?,?,?,?,?)", + (r, v) => + v.push( + r.file_path, + r.commit_count, + r.weighted_commits, + r.lines_added, + r.lines_removed, + r.last_commit_at, + r.churn_trend, + r.computed_at, + ), + ); +} + +/** Replace all churn rows (full-rebuild git ingest or golden seed-file-churn). */ +export function replaceFileChurn(db: CodemapDatabase, rows: FileChurnRow[]) { + const persist = db.transaction(() => { + db.run("DELETE FROM file_churn"); + insertFileChurn(db, rows); + }); + persist(); +} + +/** `meta` key: last `HEAD` when `file_churn` was refreshed (idle skip). */ +export const META_CHURN_INDEXED_COMMIT = "churn_indexed_commit"; +/** `meta` key: `halfLifeDays|since` fingerprint for idle skip after config changes. */ +export const META_CHURN_CONFIG_FINGERPRINT = "churn_config_fingerprint"; + +/** Replace churn rows for `scopePaths` only; other paths are left unchanged. */ +export function mergeFileChurnForPaths( + db: CodemapDatabase, + rows: FileChurnRow[], + scopePaths: Iterable, +) { + const persist = db.transaction(() => { + for (const p of scopePaths) { + db.run("DELETE FROM file_churn WHERE file_path = ?", [p]); + } + insertFileChurn(db, rows); + }); + persist(); +} + +/** Drop churn rows whose `file_path` is no longer indexed. */ +export function pruneFileChurnOrphans(db: CodemapDatabase) { + db.run( + "DELETE FROM file_churn WHERE file_path NOT IN (SELECT path FROM files)", + ); +} + /** * One row per leaf parameter binding, ordered by position. Owner-kind * lets `(file_path, owner_name, owner_kind)` disambiguate same-name diff --git a/src/file-churn.test.ts b/src/file-churn.test.ts new file mode 100644 index 00000000..f4db5265 --- /dev/null +++ b/src/file-churn.test.ts @@ -0,0 +1,184 @@ +import { describe, expect, it } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { + closeDb, + createSchema, + insertFile, + insertFileChurn, + replaceFileChurn, +} from "./db"; +import type { FileChurnRow } from "./db"; +import { openCodemapDatabase } from "./sqlite-db"; + +const REPO_ROOT = join(import.meta.dir, ".."); + +describe("file_churn + churn-complexity-hotspots recipe", () => { + it("ranks files by weighted_commits × max complexity", () => { + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/hot.ts", + content_hash: "a", + size: 100, + line_count: 10, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + insertFile(db, { + path: "src/cold.ts", + content_hash: "b", + size: 50, + line_count: 5, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + const churn: FileChurnRow[] = [ + { + file_path: "src/hot.ts", + commit_count: 20, + weighted_commits: 15, + lines_added: 200, + lines_removed: 50, + last_commit_at: "2026-06-01T00:00:00Z", + churn_trend: "accelerating", + computed_at: "2026-06-10T00:00:00Z", + }, + { + file_path: "src/cold.ts", + commit_count: 2, + weighted_commits: 1, + lines_added: 10, + lines_removed: 2, + last_commit_at: "2026-01-01T00:00:00Z", + churn_trend: "cooling", + computed_at: "2026-06-10T00:00:00Z", + }, + ]; + replaceFileChurn(db, churn); + db.run( + `INSERT INTO symbols (file_path, name, kind, line_start, line_end, signature, is_exported, is_default_export, members, doc_comment, value, parent_name, visibility, name_column_start, name_column_end, scope_local_id, body_line_count, param_count, complexity) + VALUES ('src/hot.ts', 'hotFn', 'function', 1, 5, 'hotFn()', 1, 0, NULL, NULL, NULL, NULL, NULL, 1, 4, 0, 5, 0, 12), + ('src/cold.ts', 'coldFn', 'function', 1, 3, 'coldFn()', 1, 0, NULL, NULL, NULL, NULL, NULL, 1, 5, 0, 3, 0, 3)`, + ); + + const sql = readFileSync( + join(REPO_ROOT, "templates/recipes/churn-complexity-hotspots.sql"), + "utf-8", + ); + const rows = db.query(sql).all(20, 1, 0, "") as Array<{ + file_path: string; + hotspot_score: number; + hotspot_score_normalized: number; + symbol_name: string | null; + }>; + expect(rows.length).toBe(2); + expect(rows[0]?.file_path).toBe("src/hot.ts"); + expect(rows[0]?.hotspot_score).toBe(180); + expect(rows[0]?.hotspot_score_normalized).toBe(100); + expect(rows[0]?.symbol_name).toBeNull(); + expect(rows[1]?.file_path).toBe("src/cold.ts"); + expect(rows[1]?.hotspot_score).toBe(3); + expect(rows[1]?.hotspot_score_normalized).toBe(1.7); + } finally { + closeDb(db); + } + }); + + it("path_prefix limits results to a subtree", () => { + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + for (const path of ["src/lib/hot.ts", "src/other/cold.ts"]) { + insertFile(db, { + path, + content_hash: path, + size: 50, + line_count: 5, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + } + replaceFileChurn(db, [ + { + file_path: "src/lib/hot.ts", + commit_count: 10, + weighted_commits: 8, + lines_added: 50, + lines_removed: 10, + last_commit_at: "2026-06-01T00:00:00Z", + churn_trend: "stable", + computed_at: "2026-06-10T00:00:00Z", + }, + { + file_path: "src/other/cold.ts", + commit_count: 2, + weighted_commits: 20, + lines_added: 5, + lines_removed: 1, + last_commit_at: "2026-01-01T00:00:00Z", + churn_trend: "cooling", + computed_at: "2026-06-10T00:00:00Z", + }, + ]); + db.run( + `INSERT INTO symbols (file_path, name, kind, line_start, line_end, signature, is_exported, is_default_export, members, doc_comment, value, parent_name, visibility, name_column_start, name_column_end, scope_local_id, body_line_count, param_count, complexity) + VALUES ('src/lib/hot.ts', 'hotFn', 'function', 1, 3, 'hotFn()', 1, 0, NULL, NULL, NULL, NULL, NULL, 1, 4, 0, 3, 0, 5), + ('src/other/cold.ts', 'coldFn', 'function', 1, 3, 'coldFn()', 1, 0, NULL, NULL, NULL, NULL, NULL, 1, 5, 0, 3, 0, 50)`, + ); + + const sql = readFileSync( + join(REPO_ROOT, "templates/recipes/churn-complexity-hotspots.sql"), + "utf-8", + ); + const rows = db.query(sql).all(20, 1, 0, "src/lib/") as Array<{ + file_path: string; + hotspot_score: number; + }>; + expect(rows.map((r) => r.file_path)).toEqual(["src/lib/hot.ts"]); + expect(rows[0]?.hotspot_score).toBe(40); + } finally { + closeDb(db); + } + }); + + it("replaceFileChurn clears prior rows", () => { + const db = openCodemapDatabase(":memory:"); + try { + createSchema(db); + insertFile(db, { + path: "src/x.ts", + content_hash: "x", + size: 10, + line_count: 1, + language: "typescript", + last_modified: 1, + indexed_at: 1, + }); + insertFileChurn(db, [ + { + file_path: "src/x.ts", + commit_count: 1, + weighted_commits: 1, + lines_added: 1, + lines_removed: 0, + last_commit_at: null, + churn_trend: null, + computed_at: "2026-06-10T00:00:00Z", + }, + ]); + replaceFileChurn(db, []); + const count = db + .query<{ n: number }>("SELECT COUNT(*) AS n FROM file_churn") + .get(); + expect(count?.n).toBe(0); + } finally { + closeDb(db); + } + }); +}); diff --git a/src/runtime.ts b/src/runtime.ts index fc82a5b8..9d5c8a62 100644 --- a/src/runtime.ts +++ b/src/runtime.ts @@ -63,6 +63,18 @@ export function getApplyAutoApplyRecipes(): readonly string[] | undefined { return getCodemapConfig().applyAutoApplyRecipes; } +export function getChurnHalfLifeDays(): number { + return getCodemapConfig().churn.halfLifeDays; +} + +export function getChurnSince(): string | null { + return getCodemapConfig().churn.since; +} + +export function getChurnFilePath(): string | null { + return getCodemapConfig().churn.file; +} + /** True if any path segment matches an excluded directory name (e.g. `node_modules`). */ export function isPathExcluded(relPath: string): boolean { const set = getExcludeDirNames(); diff --git a/src/worker-pool.dist.test.ts b/src/worker-pool.dist.test.ts index b1c4dc80..e33a05b9 100644 --- a/src/worker-pool.dist.test.ts +++ b/src/worker-pool.dist.test.ts @@ -4,7 +4,8 @@ */ import { describe, expect, test } from "bun:test"; -import { existsSync } from "node:fs"; +import { existsSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; import { join } from "node:path"; import { globSync } from "./glob-sync"; @@ -42,14 +43,25 @@ describe("node dist --full exit delay", () => { const files = globSync(["**/*.ts", "**/*.tsx", "**/*.css"], minimalRoot); expect(files.length).toBeGreaterThan(WORKER_POOL_INLINE_PARSE_MAX); - await expectSubprocessExits(() => - Bun.spawn(["node", distEntry, "--full"], { - cwd: repoRoot, - env: { ...process.env, CODEMAP_ROOT: minimalRoot }, - stdout: "ignore", - stderr: "pipe", - }), - ); + const stateDir = mkdtempSync(join(tmpdir(), "codemap-dist-full-")); + try { + await expectSubprocessExits(() => + Bun.spawn(["node", distEntry, "--full"], { + cwd: repoRoot, + env: { + ...process.env, + CODEMAP_ROOT: minimalRoot, + CODEMAP_STATE_DIR: stateDir, + // Prevent git from walking above the fixture (parent monorepo history). + GIT_CEILING_DIRECTORIES: minimalRoot, + }, + stdout: "ignore", + stderr: "pipe", + }), + ); + } finally { + rmSync(stateDir, { recursive: true, force: true }); + } }, 8_000, ); diff --git a/templates/agent-content/mcp-instructions.md b/templates/agent-content/mcp-instructions.md index 44dd3cb7..e287ee9b 100644 --- a/templates/agent-content/mcp-instructions.md +++ b/templates/agent-content/mcp-instructions.md @@ -29,40 +29,42 @@ Key fields: `pending_sync` (watcher debounce queue or in-flight reindex), `commi ## Common tasks -| Goal | MCP tool | Recipe twin (`query_recipe`) | -| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Exact symbol lookup | **`show`** (`name`, optional `in`) | `find-symbol-definitions` | -| Field-qualified symbol discovery | **`show`** or **`snippet`** (`query` with `kind:` / `name:` / `path:` / `in:` + free text) | `find-symbol-by-kind` for kind-heavy patterns; CLI `codemap show --query '…' --print-sql` to inspect generated SQL (no MCP `print_sql` arg) | -| Kind / pattern lookup | **`query_recipe`** | `find-symbol-by-kind` | -| Source at symbol | **`snippet`** | same rows as `show` + disk text | -| Blast radius | **`impact`** (`target`, `direction`, `via`, `depth`) | `fan-in` for file hubs; symbol call graph via SQL or `impact` | -| Call path + snippets | **`trace`** (`from`, `to`, `via?`, `max_depth?`, `budget_chars?`) — adaptive snippet caps 15k/10k/6k when omitted | `call-path` | -| Type extends / implements chain | **`query_recipe`** | `type-ancestors`, `type-descendants` (`file_path` when homonyms; on `type-descendants` also scopes output to that file) | -| Multi-symbol survey | **`explore`** (`names`, `depth?`, `kind?`, `budget_chars?`) — row cap always adaptive (500/250/125); snippets 15k/10k/6k when `budget_chars` omitted | `symbol-neighborhood` (once per name) | -| One-hop symbol card | **`node`** (`name`, `kind?`, `in?`, `include_snippets?`, `budget_chars?`) — adaptive snippet caps when snippets enabled | `show` + `symbol-neighborhood` with `depth=1` | -| Affected tests | **`affected`** (`paths?`, `changed_since?`, `test_glob?`, `max_depth?`) | `affected-tests` (RS-delimit multiple paths in `query_recipe` params) | -| CI / SARIF | **`query_recipe`** + `format: "sarif"` | `deprecated-symbols`, `boundary-violations`, … | -| GitLab Code Quality | **`query_recipe`** + `format: "codeclimate"` | `boundary-violations`, … — locatable rows only; flat `minor` severity | -| CI badge / issue count | **`query_recipe`** + `format: "badge"` (+ `badge_style: "json"` for gates) | presentation only — triage via JSON rows / `--summary` | -| Ad-hoc SQL | **`query`** | — | -| N statements / one round-trip | **`query_batch`** | **`codemap query batch`** | -| Index freshness (index-level) | **`context`** (`index_freshness`) + tool metadata above | — | -| Per-file staleness | **`validate`** | — | -| Drift vs baseline | **`audit`** (`baseline_prefix` and/or per-delta `baselines`) or **`query`** / **`query_recipe`** + `baseline` (one-shot row diff vs `query_baselines`; incompatible with non-`json` `format` / `group_by`) | save via **`save_baseline`**; `summary: true` → count-only diff | -| PR merge-base drift | **`audit`** `base: ` (git committish; sha-keyed cache) | `attribution: introduced` (branch-new) \| `inherited` (pre-existing at merge base) on each `added` row; `jq '.deltas.deprecated.added[] \| select(.attribution == "introduced")'`; `summary: true` → `added_introduced` / `added_inherited` | -| Load coverage data | **`ingest_coverage`** (`path`, optional `runtime` for V8 dirs; auto-detects Istanbul `.json` / LCOV `.info`) | enables `worst-covered-exports`, `files-by-coverage`, `untested-and-dead`, **`coverage-confirmed-dead`** (`confidence: high`); **`high-crap-score`** measured override | -| Complex + undertested (no ingest) | **`query_recipe`** `high-crap-score` | graph-estimated 85/40/0% tiers — parse `coverage_source` before CI gates; prefer after **`ingest_coverage`** when possible | -| High-judgment recipe triage | **`query_recipe`** `unimported-exports`, `boundary-violations`, `deprecated-symbols` | rows include `reason` / `evidence_json` — cite before **`apply`** or deletion | -| Apply recipe diff rows | **`apply`** (`recipe`, `params?`, `dry_run?`, `yes?`, `force?`, `until_empty?`, `max_passes?`, `commit_message?`) | recipe must emit `{file_path, line_start, before_pattern, after_pattern}` rows; `yes: true` required for writes; non-`auto_fixable` recipes need `force: true` | -| Apply agent/codemod rows | **`apply_rows`** (`rows`, `dry_run?`, `yes?`) | same row contract; bypasses recipe `auto_fixable` / allowlist gates | -| Apply unified diff text | **`apply_diff_input`** (`diff_text`, `dry_run?`, `yes?`, `commit_message?`) | parses git-style hunks; same executor as `apply_rows` | +| Goal | MCP tool | Recipe twin (`query_recipe`) | +| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Exact symbol lookup | **`show`** (`name`, optional `in`) | `find-symbol-definitions` | +| Field-qualified symbol discovery | **`show`** or **`snippet`** (`query` with `kind:` / `name:` / `path:` / `in:` + free text) | `find-symbol-by-kind` for kind-heavy patterns; CLI `codemap show --query '…' --print-sql` to inspect generated SQL (no MCP `print_sql` arg) | +| Kind / pattern lookup | **`query_recipe`** | `find-symbol-by-kind` | +| Source at symbol | **`snippet`** | same rows as `show` + disk text | +| Blast radius | **`impact`** (`target`, `direction`, `via`, `depth`) | `fan-in` for file hubs; symbol call graph via SQL or `impact` | +| Call path + snippets | **`trace`** (`from`, `to`, `via?`, `max_depth?`, `budget_chars?`) — adaptive snippet caps 15k/10k/6k when omitted | `call-path` | +| Type extends / implements chain | **`query_recipe`** | `type-ancestors`, `type-descendants` (`file_path` when homonyms; on `type-descendants` also scopes output to that file) | +| Multi-symbol survey | **`explore`** (`names`, `depth?`, `kind?`, `budget_chars?`) — row cap always adaptive (500/250/125); snippets 15k/10k/6k when `budget_chars` omitted | `symbol-neighborhood` (once per name) | +| One-hop symbol card | **`node`** (`name`, `kind?`, `in?`, `include_snippets?`, `budget_chars?`) — adaptive snippet caps when snippets enabled | `show` + `symbol-neighborhood` with `depth=1` | +| Affected tests | **`affected`** (`paths?`, `changed_since?`, `test_glob?`, `max_depth?`) | `affected-tests` (RS-delimit multiple paths in `query_recipe` params) | +| CI / SARIF | **`query_recipe`** + `format: "sarif"` | `deprecated-symbols`, `boundary-violations`, … | +| GitLab Code Quality | **`query_recipe`** + `format: "codeclimate"` | `boundary-violations`, … — locatable rows only; flat `minor` severity | +| CI badge / issue count | **`query_recipe`** + `format: "badge"` (+ `badge_style: "json"` for gates) | presentation only — triage via JSON rows / `--summary` | +| Ad-hoc SQL | **`query`** | — | +| N statements / one round-trip | **`query_batch`** | **`codemap query batch`** | +| Index freshness (index-level) | **`context`** (`index_freshness`) + tool metadata above | — | +| Per-file staleness | **`validate`** | — | +| Drift vs baseline | **`audit`** (`baseline_prefix` and/or per-delta `baselines`) or **`query`** / **`query_recipe`** + `baseline` (one-shot row diff vs `query_baselines`; incompatible with non-`json` `format` / `group_by`) | save via **`save_baseline`**; `summary: true` → count-only diff | +| PR merge-base drift | **`audit`** `base: ` (git committish; sha-keyed cache) | `attribution: introduced` (branch-new) \| `inherited` (pre-existing at merge base) on each `added` row; `jq '.deltas.deprecated.added[] \| select(.attribution == "introduced")'`; `summary: true` → `added_introduced` / `added_inherited` | +| Load coverage data | **`ingest_coverage`** (`path`, optional `runtime` for V8 dirs; auto-detects Istanbul `.json` / LCOV `.info`) | enables `worst-covered-exports`, `files-by-coverage`, `untested-and-dead`, **`coverage-confirmed-dead`** (`confidence: high`); **`high-crap-score`** measured override | +| Load churn data (non-git / fixture) | **`ingest_churn`** (`path`) | precomputed `file_churn` JSON — CLI twin `codemap ingest-churn`; enables `churn-complexity-hotspots` when git history is unavailable | +| Complex + undertested (no ingest) | **`query_recipe`** `high-crap-score` | graph-estimated 85/40/0% tiers — parse `coverage_source` before CI gates; prefer after **`ingest_coverage`** when possible | +| Churn × complexity refactor targets | **`query_recipe`** `churn-complexity-hotspots` (`by_symbol?`, `min_complexity?`, `row_limit?`, `path_prefix?`) | git `file_churn` by default each index; config **`churn.file`** replaces git when set; fixtures: **`ingest_churn`** / `ingest-churn` — **not** the `hotspots` alias (`fan-in`); empty `file_churn` → `context` `churn_hint` | +| High-judgment recipe triage | **`query_recipe`** `unimported-exports`, `boundary-violations`, `deprecated-symbols` | rows include `reason` / `evidence_json` — cite before **`apply`** or deletion | +| Apply recipe diff rows | **`apply`** (`recipe`, `params?`, `dry_run?`, `yes?`, `force?`, `until_empty?`, `max_passes?`, `commit_message?`) | recipe must emit `{file_path, line_start, before_pattern, after_pattern}` rows; `yes: true` required for writes; non-`auto_fixable` recipes need `force: true` | +| Apply agent/codemod rows | **`apply_rows`** (`rows`, `dry_run?`, `yes?`) | same row contract; bypasses recipe `auto_fixable` / allowlist gates | +| Apply unified diff text | **`apply_diff_input`** (`diff_text`, `dry_run?`, `yes?`, `commit_message?`) | parses git-style hunks; same executor as `apply_rows` | ## Chains - Rename: `find-symbol-definitions` → `find-symbol-references` (both via **`query_recipe`**). - Call path: **`trace`** (`from`, `to`) or **`query_recipe`** `call-path`; add snippets via **`trace`** / **`node`** / **`explore`** (adaptive snippet caps 15k/10k/6k; explore row cap 500/250/125 always adaptive) or **`snippet`** per row. Dependency hops may return `snippets_skipped_reason` — fall back to **`query_recipe`** + **`snippet`** per hop. - Type hierarchy: **`query_recipe`** `type-ancestors` / `type-descendants`; pass `file_path` when symbol names collide across files. On **`type-descendants`**, `file_path` also limits results to descendants defined in that file. -- Refactor risk: `fan-in` + `refactor-risk-ranking`. +- Refactor risk: `churn-complexity-hotspots` → `fan-in` + `refactor-risk-ranking` → **`snippet`** per row (alias `hotspots` = import fan-in only — not churn×complexity). - Edit path: **`show`** → **`snippet`**; if `stale: true`, line range may have drifted. - Apply path: **`query_recipe`** (or audit baseline `added`) with `format: "diff-json"` → **`apply`** `dry_run: true` → **`apply`** `yes: true` (+ `force: true` when recipe is not `auto_fixable`). Pre-built rows: **`apply_rows`**; unified diff: **`apply_diff_input`**. Fixpoint: **`apply`** `until_empty: true`; git: **`commit_message`** on **`apply`** / **`apply_diff_input`**. **`rename-preview` homonyms:** pass `define_in` in `params` (definition `file_path` anchor — not the same as `in_file`). @@ -75,6 +77,6 @@ Key fields: `pending_sync` (watcher debounce queue or in-flight reindex), `commi ## Recipe ids cited here -`find-symbol-definitions`, `find-symbol-by-kind`, `find-symbol-references`, `fan-in`, `call-path`, `symbol-neighborhood`, `type-ancestors`, `type-descendants`, `affected-tests`, `deprecated-symbols`, `boundary-violations`, `unimported-exports`, `coverage-confirmed-dead`, `high-crap-score`, `refactor-risk-ranking`, `calls-including-heuristic` (opt-in; requires `synthesis.heuristicCalls: true` in config). Default call-graph recipes exclude `provenance = 'heuristic'`. Others: list via **`codemap://recipes`** before **`query_recipe`**. +`find-symbol-definitions`, `find-symbol-by-kind`, `find-symbol-references`, `fan-in`, `call-path`, `symbol-neighborhood`, `type-ancestors`, `type-descendants`, `affected-tests`, `deprecated-symbols`, `boundary-violations`, `unimported-exports`, `coverage-confirmed-dead`, `high-crap-score`, `churn-complexity-hotspots`, `refactor-risk-ranking`, `calls-including-heuristic` (opt-in; requires `synthesis.heuristicCalls: true` in config). Default call-graph recipes exclude `provenance = 'heuristic'`. Others: list via **`codemap://recipes`** before **`query_recipe`**. - + diff --git a/templates/agent-content/rule/00-full.md b/templates/agent-content/rule/00-full.md index 83de3b50..3b903742 100644 --- a/templates/agent-content/rule/00-full.md +++ b/templates/agent-content/rule/00-full.md @@ -6,7 +6,7 @@ alwaysApply: true > **STOP.** Before you call Grep, Glob, SemanticSearch, or Read to answer a **structural** question about **this project** — query the Codemap SQLite index first. -This project is indexed by **Codemap** — a local SQLite database (default **`.codemap/index.db`**) of structure: `files`, `symbols`, `imports`, `exports`, `dependencies`, `calls`, `components`, `markers`, `type_members`, `type_heritage`, `import_specifiers`, `scopes`, `references`, `bindings`, `function_params`, `dynamic_imports`, `jsx_elements`, `jsx_attributes`, `async_calls`, `try_catch`, `decorators`, `jsdoc_tags`, `runtime_markers`, `test_suites`, `re_export_chains`, `module_cycles`, `file_metrics`, `css_variables`, `css_classes`, `css_keyframes`, `suppressions`, `boundary_rules`, and (after `codemap ingest-coverage `) `coverage`. Full DDL: `codemap query --json "SELECT sql FROM sqlite_schema WHERE type='table'"` or MCP resource `codemap://schema`. +This project is indexed by **Codemap** — a local SQLite database (default **`.codemap/index.db`**) of structure: `files`, `symbols`, `imports`, `exports`, `dependencies`, `calls`, `components`, `markers`, `type_members`, `type_heritage`, `import_specifiers`, `scopes`, `references`, `bindings`, `function_params`, `dynamic_imports`, `jsx_elements`, `jsx_attributes`, `async_calls`, `try_catch`, `decorators`, `jsdoc_tags`, `runtime_markers`, `test_suites`, `re_export_chains`, `module_cycles`, `file_metrics`, `file_churn` (every index pass or `codemap ingest-churn`), `css_variables`, `css_classes`, `css_keyframes`, `suppressions`, `boundary_rules`, and (after `codemap ingest-coverage `) `coverage`. Full DDL: `codemap query --json "SELECT sql FROM sqlite_schema WHERE type='table'"` or MCP resource `codemap://schema`. ## How to query @@ -24,6 +24,8 @@ codemap query --recipes-json # canonical list of every bundled + p **Coverage columns:** `high-crap-score` rows add **`coverage_source`** (`measured` \| `estimated`) and **`effective_coverage_pct`** — measured when `ingest-coverage` has a symbol row; else graph tiers 85/40/0% from test reachability (heuristic, not execution). +**Churn / hotspot columns:** `churn-complexity-hotspots` rows add **`hotspot_score`**, **`hotspot_score_normalized`**, **`churn_trend`** — distinct from outcome alias **`hotspots`** → `fan-in`. Non-git: `ingest_churn` / `ingest-churn` / `churn.file`. + **Confidence columns:** `coverage-confirmed-dead` rows add **`confidence`** (`high` \| `medium`) — `high` when static dead and ingested `coverage_pct = 0`; `medium` when dead but unmeasured. Parse before deletion. **Audit attribution:** `codemap audit --base ` (and MCP/HTTP `audit` with `base`) tags each `added` row with **`attribution: introduced | inherited`** — branch-new vs pre-existing at merge base. Filter actionable PR deltas with `jq '.deltas.deprecated.added[] | select(.attribution == "introduced")'`. @@ -32,45 +34,46 @@ codemap query --recipes-json # canonical list of every bundled + p If the question matches any of these, use the index instead of grepping: -| Question shape | Table(s) / Recipe | -| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -| "What/which files import X?" | `imports` (by `source`) or `dependencies` (by `to_path`) | -| "Where is X defined?" | `symbols` | -| "What does file X export?" | `exports` | -| "Who depends on file X?" / "What does file X depend on?" | `dependencies` | -| "Who calls X?" / "What does X call?" | `calls` | -| "Where is X used?" / "Every reference to X" | `--recipe find-references` (name-keyed) | -| "Every reference to X defined in file Y" (precise rename) | `--recipe find-symbol-references` (bindings-precise) | -| Homonym-safe rename (scoped definition anchor) | `--recipe rename-preview` with `define_in=`; CLI `codemap rename [--define-in ] [--in-file ] [--kind ]` | -| "Every write to X" | `--recipe find-write-sites` | -| "Every fn taking a `User` param" | `--recipe find-by-param-type` (params `type_text=...`) | -| "What hooks does component X use?" / "List React components" | `components` | -| "What are the CSS variables/tokens for X?" | `css_variables` | -| "What CSS classes / keyframes are in X?" | `css_classes` / `css_keyframes` | -| "Find all TODOs / FIXMEs / HACKs / NOTEs" | `markers` | -| "What fields does interface/type X have?" | `type_members` | -| "What does X extend / implement?" / type hierarchy | `type_heritage` / `--recipe type-ancestors` / `--recipe type-descendants` | -| "Is X deprecated?" / "What's `@beta` / `@internal`?" | `symbols.doc_comment` / `symbols.visibility` | -| "Leftover `console.log` calls" | `--recipe find-leftover-console` (or `runtime_markers`) | -| "What `process.env.X` vars does this app read?" | `--recipe env-var-audit` | -| "Find `.skip` / `.only` / `.todo` tests" | `--recipe find-skipped-tests` | -| "Tests per file (counts + framework)" | `--recipe tests-by-file` | -| "Are there import cycles?" / "Files in cycles" | `--recipe circular-imports` / `module_cycles` | -| "Where do barrel files re-export from?" | `--recipe barrel-chains` / `re_export_chains` | -| "Functions over 50 lines / deeply nested" | `--recipe large-functions` / `deeply-nested-functions` | -| "What's the cyclomatic / cognitive complexity of X?" | `symbols.complexity` / `symbols.cognitive_complexity` (Sonar-inspired; class methods included) | -| "What's the nesting depth of X?" | `symbols.nesting_depth` | -| "Is symbol X tested?" / "What's the coverage of file Y?" | `coverage` (after `codemap ingest-coverage`) | -| "What's structurally dead AND untested?" | `--recipe untested-and-dead` | -| "Dead exports with ingested zero coverage?" | `--recipe coverage-confirmed-dead` (check `confidence`: `high` vs `medium`) | -| "Worst-covered exported functions" | `--recipe worst-covered-exports` | -| "Which exports has nobody imported?" | `--recipe unimported-exports` | -| "Which components touch deprecated APIs?" | `--recipe components-touching-deprecated` | -| "What's risky to refactor right now?" | `--recipe refactor-risk-ranking` | -| "What's high-complexity AND undertested?" | `--recipe high-complexity-untested` (needs `ingest-coverage`; without ingest prefer `high-crap-score`) | -| "Complex + undertested without coverage ingest?" | `--recipe high-crap-score` (graph-estimated tiers; `coverage_source: estimated`) | -| "What's cognitively complex (nesting-heavy)?" | `--recipe high-cognitive-complexity` (default `min_score=15`; `--params min_score=20` to tighten) | -| "Structurally duplicate function bodies?" | `--recipe duplicates` (rename-insensitive `body_hash`; triage with `snippet` before refactor) | +| Question shape | Table(s) / Recipe | +| ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| "What/which files import X?" | `imports` (by `source`) or `dependencies` (by `to_path`) | +| "Where is X defined?" | `symbols` | +| "What does file X export?" | `exports` | +| "Who depends on file X?" / "What does file X depend on?" | `dependencies` | +| "Who calls X?" / "What does X call?" | `calls` | +| "Where is X used?" / "Every reference to X" | `--recipe find-references` (name-keyed) | +| "Every reference to X defined in file Y" (precise rename) | `--recipe find-symbol-references` (bindings-precise) | +| Homonym-safe rename (scoped definition anchor) | `--recipe rename-preview` with `define_in=`; CLI `codemap rename [--define-in ] [--in-file ] [--kind ]` | +| "Every write to X" | `--recipe find-write-sites` | +| "Every fn taking a `User` param" | `--recipe find-by-param-type` (params `type_text=...`) | +| "What hooks does component X use?" / "List React components" | `components` | +| "What are the CSS variables/tokens for X?" | `css_variables` | +| "What CSS classes / keyframes are in X?" | `css_classes` / `css_keyframes` | +| "Find all TODOs / FIXMEs / HACKs / NOTEs" | `markers` | +| "What fields does interface/type X have?" | `type_members` | +| "What does X extend / implement?" / type hierarchy | `type_heritage` / `--recipe type-ancestors` / `--recipe type-descendants` | +| "Is X deprecated?" / "What's `@beta` / `@internal`?" | `symbols.doc_comment` / `symbols.visibility` | +| "Leftover `console.log` calls" | `--recipe find-leftover-console` (or `runtime_markers`) | +| "What `process.env.X` vars does this app read?" | `--recipe env-var-audit` | +| "Find `.skip` / `.only` / `.todo` tests" | `--recipe find-skipped-tests` | +| "Tests per file (counts + framework)" | `--recipe tests-by-file` | +| "Are there import cycles?" / "Files in cycles" | `--recipe circular-imports` / `module_cycles` | +| "Where do barrel files re-export from?" | `--recipe barrel-chains` / `re_export_chains` | +| "Functions over 50 lines / deeply nested" | `--recipe large-functions` / `deeply-nested-functions` | +| "What's the cyclomatic / cognitive complexity of X?" | `symbols.complexity` / `symbols.cognitive_complexity` (Sonar-inspired; class methods included) | +| "What's the nesting depth of X?" | `symbols.nesting_depth` | +| "Is symbol X tested?" / "What's the coverage of file Y?" | `coverage` (after `codemap ingest-coverage`) | +| "What's structurally dead AND untested?" | `--recipe untested-and-dead` | +| "Dead exports with ingested zero coverage?" | `--recipe coverage-confirmed-dead` (check `confidence`: `high` vs `medium`) | +| "Worst-covered exported functions" | `--recipe worst-covered-exports` | +| "Which exports has nobody imported?" | `--recipe unimported-exports` | +| "Which components touch deprecated APIs?" | `--recipe components-touching-deprecated` | +| "What's risky to refactor right now?" | `--recipe refactor-risk-ranking` | +| "What's high-complexity AND undertested?" | `--recipe high-complexity-untested` (needs `ingest-coverage`; without ingest prefer `high-crap-score`) | +| "Complex + undertested without coverage ingest?" | `--recipe high-crap-score` (graph-estimated tiers; `coverage_source: estimated`) | +| "What's cognitively complex (nesting-heavy)?" | `--recipe high-cognitive-complexity` (default `min_score=15`; `--params min_score=20` to tighten) | +| "Structurally duplicate function bodies?" | `--recipe duplicates` (rename-insensitive `body_hash`; triage with `snippet` before refactor) | +| "What files churn often AND are complex?" | `--recipe churn-complexity-hotspots` (or `--params by_symbol=true`; non-git: `ingest_churn` / `ingest-churn` / `churn.file`; alias `hotspots` → `fan-in`) | ## Quick reference queries diff --git a/templates/agent-content/skill/10-recipes-context.md b/templates/agent-content/skill/10-recipes-context.md index e143daf1..62231299 100644 --- a/templates/agent-content/skill/10-recipes-context.md +++ b/templates/agent-content/skill/10-recipes-context.md @@ -2,7 +2,7 @@ Replace placeholders (`'...'`) with your module path, file glob, or symbol name. -**Outcome aliases:** **`codemap dead-code`** · **`deprecated`** · **`boundaries`** · **`hotspots`** · **`coverage-gaps`** — thin wrappers over `query --recipe `. Every `query` flag passes through (`--json`, `--format sarif|codeclimate|badge`, `--badge-style markdown|json`, `--ci`, `--summary`, `--changed-since`, `--group-by`, `--params`, `--save-baseline`, `--baseline`). Run **`codemap --help`** for the wrapped recipe id. Capped at 5 to avoid sprawl. **Write alias (outside the cap):** **`codemap rename`** → **`apply rename-preview`** (homonym-safe via `--define-in` / `define_in` in params). +**Outcome aliases:** **`codemap dead-code`** · **`deprecated`** · **`boundaries`** · **`hotspots`** · **`coverage-gaps`** — thin wrappers over `query --recipe `. Every `query` flag passes through (`--json`, `--format sarif|codeclimate|badge`, `--badge-style markdown|json`, `--ci`, `--summary`, `--changed-since`, `--group-by`, `--params`, `--save-baseline`, `--baseline`). Run **`codemap --help`** for the wrapped recipe id. Capped at 5 to avoid sprawl. **`hotspots` → `fan-in` (import hubs)** — for **change-frequency × complexity** refactor targets use recipe **`churn-complexity-hotspots`** (not the alias). **Write alias (outside the cap):** **`codemap rename`** → **`apply rename-preview`** (homonym-safe via `--define-in` / `define_in` in params). **Suppressions (opt-in):** `// codemap-ignore-next-line ` and `// codemap-ignore-file ` (also `#`, `--`, ` ## Recipe catalog diff --git a/templates/agent-content/skill/30-schema.gen.md b/templates/agent-content/skill/30-schema.gen.md index 22d1eeed..7878218e 100644 --- a/templates/agent-content/skill/30-schema.gen.md +++ b/templates/agent-content/skill/30-schema.gen.md @@ -1,9 +1,7 @@ ## Schema reference diff --git a/templates/agent-content/skill/40-query-patterns.md b/templates/agent-content/skill/40-query-patterns.md index 560cc41b..31c36e65 100644 --- a/templates/agent-content/skill/40-query-patterns.md +++ b/templates/agent-content/skill/40-query-patterns.md @@ -103,7 +103,7 @@ SELECT DISTINCT callee_name FROM calls WHERE caller_name = 'processUser' AND (provenance IS NULL OR provenance = 'ast'); --- Most-called functions (hotspots) +-- Most-called functions (import/call fan-in — not churn×complexity; use recipe churn-complexity-hotspots for that) SELECT callee_name, COUNT(*) as fan_in FROM calls WHERE (provenance IS NULL OR provenance = 'ast') GROUP BY callee_name ORDER BY fan_in DESC LIMIT 10; @@ -146,7 +146,7 @@ SELECT DISTINCT from_path FROM dependencies WHERE to_path LIKE '%format-date%'; -- Direct dependencies (what does this file import?) SELECT DISTINCT to_path FROM dependencies WHERE from_path LIKE '%OrderRow%'; --- Most-imported files (hotspots) +-- Most-imported files (dependency fan-in — not churn×complexity; use recipe churn-complexity-hotspots for that) SELECT to_path, COUNT(*) as importers FROM dependencies GROUP BY to_path ORDER BY importers DESC LIMIT 15; diff --git a/templates/agent-content/skill/50-maintenance.md b/templates/agent-content/skill/50-maintenance.md index cd59f23f..f92651d4 100644 --- a/templates/agent-content/skill/50-maintenance.md +++ b/templates/agent-content/skill/50-maintenance.md @@ -23,4 +23,4 @@ codemap validate --json Same flags as **`npx @stainless-code/codemap`**, **`pnpm dlx @stainless-code/codemap`**, etc. **`codemap --root /path/to/project`** indexes another working tree. -**Full-text search (opt-in):** pass **`--with-fts`** on index runs, or set **`fts5: true`** in `.codemap/config.ts` — populates the `source_fts` virtual table for `show --query` / `snippet --query` / MCP `show` / `snippet` when `with_fts: true`. Default OFF until measurement closes [FTS default-on evaluation](../../docs/plans/fts-default-on-evaluation.md). +**Full-text search (opt-in):** pass **`--with-fts`** on index runs, or set **`fts5: true`** in `.codemap/config.ts` — populates the `source_fts` virtual table for `show --query` / `snippet --query` / MCP `show` / `snippet` when `with_fts: true`. Default OFF on index runs; enable when you need FTS-backed field search. diff --git a/templates/recipes/churn-complexity-hotspots.md b/templates/recipes/churn-complexity-hotspots.md new file mode 100644 index 00000000..4212d8d6 --- /dev/null +++ b/templates/recipes/churn-complexity-hotspots.md @@ -0,0 +1,49 @@ +--- +actions: + - type: review-churn-hotspot + auto_fixable: false + description: "High git churn × cyclomatic complexity — read source with snippet; check fan-in before refactor. Not the codemap hotspots alias (import fan-in)." +params: + - name: row_limit + type: number + required: false + default: 20 + description: Maximum rows to return (default 20) + - name: min_complexity + type: number + required: false + default: 1 + description: Minimum cyclomatic complexity (default 1) + - name: by_symbol + type: boolean + required: false + default: false + description: When true, one row per symbol; default false ranks files (max complexity in-file) + - name: path_prefix + type: string + required: false + default: "" + description: Limit to files under this path prefix (e.g. src/lib/) +--- + +# churn-complexity-hotspots + +Files or symbols ranked by **git churn × cyclomatic complexity**. Distinct from the outcome alias `codemap hotspots` (import **fan-in** via `fan-in` recipe). + +Default: git refresh every index (`churn.halfLifeDays`, `churn.since` / `--churn-since`). When config `churn.file` is set, that JSON loads on every index and **skips** git log (including in git repos). Manual twin: `codemap ingest-churn ` for non-git repos and CI fixtures. + +```bash +codemap query --recipe churn-complexity-hotspots +codemap query --recipe churn-complexity-hotspots --params min_complexity=10,row_limit=10 +codemap query --recipe churn-complexity-hotspots --params by_symbol=true +codemap query --recipe churn-complexity-hotspots --params path_prefix=src/lib/ +codemap ingest-churn churn-metrics.json +``` + +`hotspot_score` = `weighted_commits × complexity`. `hotspot_score_normalized` is 0–100 vs the corpus max in the result set. + +**Output columns:** shared — `file_path`, `weighted_commits`, `commit_count`, `churn_trend`, `hotspot_score`, `hotspot_score_normalized`. File grain (`by_symbol=false`, default) — `symbol_name`/`symbol_kind`/`line_start` null; `max_complexity`, `avg_complexity`. Symbol grain (`by_symbol=true`) — `symbol_name`, `symbol_kind`, `line_start`; `max_complexity` = symbol complexity. `churn_trend` is `accelerating`, `stable`, or `cooling` when enough history exists. + +**Ingest JSON** (`ingest-churn` / `ingest_churn` / `churn.file`): array of `{file_path, commit_count, weighted_commits, lines_added?, lines_removed?, last_commit_at?, churn_trend?, computed_at?}` — indexed paths only. + +Triage with `snippet` before large refactors. diff --git a/templates/recipes/churn-complexity-hotspots.sql b/templates/recipes/churn-complexity-hotspots.sql new file mode 100644 index 00000000..1a6fc2c6 --- /dev/null +++ b/templates/recipes/churn-complexity-hotspots.sql @@ -0,0 +1,68 @@ +WITH params(row_limit, min_complexity, by_symbol, path_prefix) AS ( + SELECT ?, ?, ?, ? +), +base AS ( + SELECT + fc.file_path, + s.name AS symbol_name, + s.kind AS symbol_kind, + s.line_start, + fc.weighted_commits, + fc.commit_count, + fc.churn_trend, + s.complexity, + ROUND(fc.weighted_commits * s.complexity, 2) AS hotspot_score + FROM file_churn fc + JOIN symbols s ON s.file_path = fc.file_path + CROSS JOIN params p + WHERE s.complexity IS NOT NULL + AND s.complexity >= p.min_complexity + AND (p.path_prefix = '' OR fc.file_path LIKE p.path_prefix || '%') +), +file_rows AS ( + SELECT + file_path, + NULL AS symbol_name, + NULL AS symbol_kind, + NULL AS line_start, + weighted_commits, + commit_count, + churn_trend, + MAX(complexity) AS max_complexity, + ROUND(AVG(complexity), 1) AS avg_complexity, + MAX(hotspot_score) AS hotspot_score + FROM base + GROUP BY file_path, weighted_commits, commit_count, churn_trend +), +symbol_rows AS ( + SELECT + file_path, + symbol_name, + symbol_kind, + line_start, + weighted_commits, + commit_count, + churn_trend, + complexity AS max_complexity, + complexity AS avg_complexity, + hotspot_score + FROM base +), +combined AS ( + SELECT * FROM file_rows WHERE (SELECT by_symbol FROM params) = 0 + UNION ALL + SELECT * FROM symbol_rows WHERE (SELECT by_symbol FROM params) != 0 +), +normalized AS ( + SELECT + c.*, + ROUND( + 100.0 * c.hotspot_score / NULLIF(MAX(c.hotspot_score) OVER (), 0), + 1 + ) AS hotspot_score_normalized + FROM combined c +) +SELECT * +FROM normalized +ORDER BY hotspot_score DESC, file_path, symbol_name +LIMIT (SELECT row_limit FROM params); diff --git a/templates/recipes/high-complexity-untested.md b/templates/recipes/high-complexity-untested.md index 3e007107..c35ff620 100644 --- a/templates/recipes/high-complexity-untested.md +++ b/templates/recipes/high-complexity-untested.md @@ -32,6 +32,8 @@ Each row also includes **SonarSource cognitive complexity** for the same symbol - Low coverage alone surfaces too many false positives — a one-line getter with 0% coverage is barely worth testing. - The intersection is the actionable list: _complex code that nobody's exercising = bug magnet_. +**Compose with [`churn-complexity-hotspots`](./churn-complexity-hotspots.md)** when recent edit frequency matters — complexity + coverage alone miss frequently touched complexity magnets. + ## Tuning axes for project-local overrides `/recipes/high-complexity-untested.sql` (default `.codemap/recipes/`): diff --git a/templates/recipes/refactor-risk-ranking.md b/templates/recipes/refactor-risk-ranking.md index e5229318..546ce6a9 100644 --- a/templates/recipes/refactor-risk-ranking.md +++ b/templates/recipes/refactor-risk-ranking.md @@ -29,3 +29,5 @@ Suggested tuning axes for project-local overrides: - **LOC weight** scale by file `line_count` (already on the `files` table). **Per-symbol vs per-file:** the original design specified per-symbol ranking; empirical testing showed per-symbol output got dominated by long-tail symbols from a single hot file all tied at the same score (file-level `fan_in` inherited). v1 ships file-level aggregation as the more useful default; per-symbol via `calls` is one of the documented tuning axes above. + +**Compose with [`churn-complexity-hotspots`](./churn-complexity-hotspots.md)** when edit frequency matters — fan-in and coverage alone miss files that change every sprint but still carry structural risk.