diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 8496e23..d946c2b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -17,7 +17,7 @@ jobs: version: 10 - uses: actions/setup-node@v4 with: - node-version: '20' + node-version: '22' cache: 'pnpm' - run: pnpm install --frozen-lockfile @@ -42,7 +42,7 @@ jobs: version: 10 - uses: actions/setup-node@v4 with: - node-version: '20' + node-version: '22' cache: 'pnpm' - run: pnpm install --frozen-lockfile diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c7647d..33f0294 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,22 @@ # Changelog +## Product candidate (not published) + +- New projects index and search without embedding or reranker models. Existing index modes are preserved; explicit mode switches rebuild fully. Transformers and LanceDB are optional peers with actionable preflight errors. +- Setup detects supported coding tools and configures the selected tools together. It preserves existing settings, saves version-pinned launches for the selected repository, and leaves indexing to the first search. +- Added `get_code_relationships`: directed import/re-export paths up to three hops, current resolution and source checks, confined endpoints, unresolved/stale diagnostics, and a 4,000-byte response cap. Resolution selects the root's `tsconfig.json` or, when absent, `tsconfig.base.json`, without inheriting an unrelated parent configuration. +- AST chunks preserve contiguous source and static property ownership, including concise descriptor methods, callback bodies and same-line siblings. Generated scope labels remain presentation metadata. Index format 4 rebuilds incompatible indexes even when sources are unchanged, preserves the selected embedding mode, and retains previous artifacts if rebuilding fails. +- Angular TypeScript eligibility uses parsed Angular imports; ordinary `resolve` or `validate` methods, comments, strings and imports from other packages no longer select the Angular analyzer. Other eligible analyzers retain the source and declaration context. The registry order and existing template/style routing are unchanged. +- Compact `signaturePreview` uses a contiguous window of up to three lines around the strongest query-term match in the returned snippet, falling back to its opening lines. Default and explicit exploration searches omit edit preflight while retaining search quality and incomplete-source warnings. Explicit `edit`, `refactor`, and `migrate` intents retain mutation guards. +- Search excludes ordinary stale/unavailable nominees without discarding verified current lower results or triggering an automatic full rebuild. Partial coverage, bounded exhaustion and failed semantic replenishment are explicit; corrupt metadata, build identity, paths and source ranges still fail closed. +- Symbol references scan same-build manifest sources in bounded read batches and skip parsing when a simple ASCII symbol occurs only inside longer identifiers. Unicode or arbitrary query spellings retain raw-text nomination. Coverage keeps raw-text absence separate from parser work and uncertain lexical fallback. Keyword artifact identity uses the shared index validator. Graph traversal verifies edges before consuming output caps, with bounded inspection and honest continuation. +- Watcher shutdown awaits handle closure and cancels pending callbacks. Connected stdio sessions have no default inactivity shutdown; explicit positive idle timeouts remain supported alongside handshake, EOF and parent-death guards. +- Chunk paths are persisted relative to the selected project root, including explicit-root indexing from another working directory. Previously incorrect relative paths can require a full reindex. +- Query advice is limited to grounded returned source, and abstained searches omit global pattern advice. Same-named symbols in different files remain eligible after file-level deduplication; TypeScript type arguments alone no longer select the JSX analyzer. +- The candidate requires Node 22+. CLI dispatch compares canonical entrypoint identity, including npm symlink launchers, while preserving the exported server's initialization on imports. +- Updated the runtime parser and watcher to remove the legacy braces dependency chain. Watcher exclusions use exact directory names; quoted import and export names retain their public symbol names. +- Known limits: model-free conceptual relevance is not guaranteed. Short requests can miss implementations when query words differ from source names; syntactic edges are not runtime calls; workspace/language-specific resolution can remain unresolved. Packed/protocol tests and scoped negative cases do not establish general agent-quality improvement or customer adoption. No npm publication is implied by this entry. + ## [2.3.0](https://github.com/PatrickSys/codebase-context/compare/v2.2.0...v2.3.0) (2026-04-30) diff --git a/README.md b/README.md index 6249c02..f68583b 100644 --- a/README.md +++ b/README.md @@ -8,53 +8,27 @@ Coding agents can read files, but they still have to discover how your repositor Codebase Context gives an agent a local view of that information through code search, team patterns, strong examples, and project memory. It runs as an MCP server - a local tool that your editor or command-line agent can call while it works - and keeps the index on your machine by default. -## Set up your AI client +## Set up your coding tools -Choose your coding tool and run its command once. Use Node.js 22 or newer. These commands use published npm `2.2.0` and do not require a project folder in your configuration: +Automatic multi-tool setup is awaiting publication. The unpinned prompt below will support this flow once a release containing it is available; until then, use the published manual recipes in the [client setup guide](./docs/client-setup.md). -```bash -# Claude Code -claude mcp add --scope user --transport stdio codebase-context -- npx -y codebase-context@2.2.0 - -# Codex CLI -codex mcp add codebase-context -- npx -y codebase-context@2.2.0 - -# OpenCode 1.x (keep the quoted separator on Windows) -opencode mcp add codebase-context '--' npx -y codebase-context@2.2.0 -``` +Use Node.js 22 or newer. Copy this prompt to your coding agent: -Start a new agent session in your project, then ask: +> Set up Codebase Context for this repository and any other repositories I explicitly name. Read https://github.com/PatrickSys/codebase-context#readme and the installed init help. Detect my supported coding tools and configure all of them, using `npx --yes codebase-context init --yes` once per repository. Preserve existing settings and instructions; explain any conflicting entry you cannot safely preserve. Add only a short, relevant usage note where this repository keeps agent instructions. Pass each repository's actual directory with --root. Once available, call Codebase Context to find a real source symbol and open its returned file. Report what works; if a tool needs reopening or trust approval, give me that exact next step and do not claim it connected. -> Use Codebase Context to find [feature] in this repository. Pass this repository's absolute path as project when checking get_indexing_status and searching. Wait for indexing if needed, read codebase://context, then search_codebase and open a returned source file. Show me the relevant files. - -Replace `[feature]` with something you want to find. The agent supplies the repository path in its tool calls, so the registration can serve different projects. Initial indexing may need a local model download. The October 6 isolated checks proved these client registrations and published-package project selection/search; they did not establish a full native agent investigation. - -For Codex Desktop, create or merge `.codex/config.toml` in the project you want to search: - -```toml -[mcp_servers.codebase-context] -command = "npx" -args = ["-y", "codebase-context@2.2.0"] -startup_timeout_sec = 120 -``` +After this flow is published, you can run `npx --yes codebase-context init` in your project. In a terminal, detected Claude Code, Codex, Cursor and OpenCode tools start selected; use Space to change the selection and Enter to apply. Use `--yes` to select all detected tools or repeat `--client` to choose explicitly. Pass `--root` when setting up a different repository. -Trust the project if asked, restart Codex, and start a new task there. Use the prompt above. Config placement alone does not establish successful first use; this no-folder recipe has not been accepted in a fresh native Desktop task. +Each tool starts its own local server when needed. Reopen the tools and approve the server or project trust if asked. The first search builds the repository's index; setup itself does not index or change agent instructions. -Other clients use their own setup commands: +New indexes use keywords, symbols, and structural analysis without embedding or reranker models. Existing indexes retain their successful mode. Semantic mode requires optional runtimes installed alongside CBC with `npm install --prefix /path/to/cbc --no-save --package-lock=false @huggingface/transformers@^4.2.0 @lancedb/lancedb@^0.4.0`. Reindex the target repository through that installation's actual entrypoint, for example `CODEBASE_ROOT=/path/to/repository node /path/to/cbc/node_modules/codebase-context/dist/index.js reindex --embeddings`; do not run `npx --no-install` from a different repository. Installing the runtimes only in a separately selected source folder does not repair another CBC installation. Use `--no-embeddings` for a full model-free rebuild; omitting both preserves the successful mode. Local semantic models may download on opt-in. Missing runtimes produce an actionable error before index replacement. -| Client | Shortest current setup | -| --------------------------- | ---------------------------------------------------------------------------- | -| Gemini CLI | `gemini mcp add --scope user codebase-context npx -y codebase-context@2.2.0` | -| Cursor | Add `.cursor/mcp.json` | -| VS Code with GitHub Copilot | Add `.vscode/mcp.json` | -| GitHub Copilot CLI | `copilot mcp add codebase-context -- npx -y codebase-context@2.2.0` | -| Windsurf | Add `~/.codeium/windsurf/mcp_config.json` | +The candidate also adds `get_code_relationships` for current, directed import/re-export paths up to three hops. It returns source citations, unresolved/stale diagnostics, and explicit truncation. This is syntactic source navigation; conceptual search quality and runtime behavior still require reading and checking the actual code. See the [capabilities reference](./docs/capabilities.md). -Check an existing same-name entry before replacing it. To give the server a default folder, append that folder's absolute path to the `npx` arguments. The [client setup guide](./docs/client-setup.md) covers scopes, optional fixed-folder configuration, verification limits and the unreleased installer. Published `2.2.0`'s interactive `init` has registration bugs; use the commands above. +Relationship resolution uses the selected root's `tsconfig.json`, or its `tsconfig.base.json` when the primary file is absent, and preserves explicit `extends`. This candidate uses index format 4: older indexes rebuild even when source files are unchanged, preserving the last successful model-free or semantic mode. A failed rebuild preserves the previous index files. For other same-format graph changes, request a full `refresh_index` with `incrementalOnly:false` and the repository's absolute `project` path; omit `skipEmbedding` to preserve the existing mode. -The [client setup guide](./docs/client-setup.md) has the exact commands and config for every client, plus what was checked locally and what still relies on official instructions. +AST chunks retain contiguous source, including callable property bodies, with scope labels kept separate from persisted text. Ordinary discovery reports search quality and incomplete source coverage; edit readiness is returned only for explicit `edit`, `refactor`, or `migrate` searches. Read the returned source before changing it. -The default connection is `stdio` (standard input/output): your client starts the server when it needs it. HTTP is an advanced, client-dependent option; verify the setup guide and client support before relying on it. +The [client setup guide](./docs/client-setup.md) retains manual recipes for published packages, including Gemini, GitHub Copilot and Windsurf, optional fixed-folder configuration and advanced HTTP. The default connection is `stdio` (standard input/output): your client starts the server when it needs it. HTTP is an advanced, client-dependent option; verify the setup guide and client support before relying on it. ## What your agent gets diff --git a/docs/capabilities.md b/docs/capabilities.md index 748d5a1..79c3bbd 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -1,6 +1,20 @@ # Capabilities Reference -Technical reference for what `codebase-context` ships today. The public product posture is map first, find second: the bounded conventions map is the first-call surface, and search narrows to the right local example after that. For the user-facing overview, see [README.md](../README.md). +Technical reference for this source tree. The relaunch changes below are an unreleased candidate, distinct from published npm `2.2.0`. The bounded conventions map and source search provide local navigation; useful engineering still requires opening current source and checking behavior. For setup and version boundaries, see [README.md](../README.md). + +## Candidate source navigation + +`get_code_relationships` accepts an indexed repository-relative `file`, `direction` (`outgoing` or `incoming`), `max_hops` (1-3, default 2), and `max_nodes` (1-100, default 50), plus the shared project selector. Directed paths distinguish imports from re-exports and cite importer lines/specifiers. Current module resolution, both source fingerprints, and realpath confinement are checked before a path is returned. Unresolved specifiers are diagnostics; legacy proofless or changed edges are omitted with refresh guidance. The returned JSON text, including project-routing metadata, is capped at 4,000 UTF-8 bytes; this excludes the MCP transport envelope. Truncation is explicit, and dense neighborhoods may require following a returned file in another call to see a full three-hop chain. + +TypeScript resolution reads the selected root's `tsconfig.json`, falling back to that root's `tsconfig.base.json` only when the primary file is absent. Explicit `extends` is supported; an unrelated ancestor's configuration is not selected implicitly. An existing malformed primary file does not silently select the base file. Index format 4 rebuilds older indexes even when source hashes are unchanged. For other same-format graph changes, run one full `refresh_index` (`incrementalOnly:false`, explicit `project`, omit `skipEmbedding` to preserve the existing mode); a no-change incremental refresh otherwise retains its prior graph. + +TypeScript configuration/module resolution supports applicable relative paths, aliases, and config inheritance. Workspace package exports, missing sparse-checkout files, dynamic imports/calls, and language-specific resolution can remain unresolved. Three hops do not establish runtime calls, inheritance, or complete program reachability. Existing search decision-card impact remains at two hops. + +Search snippets and symbol references read confined current files. Search excludes changed, deleted, or unreadable nominees while retaining verified current results from its bounded candidate window. Incomplete coverage is explicit and cannot produce an edit-ready claim; stale-only evidence gives a refresh action. Candidate-budget exhaustion and failed optional semantic replenishment are disclosed separately. Malformed indexes, mismatched build identity, invalid ranges and escaping paths remain integrity errors. Compact search results offer a current source read and the relationship tool as next steps. Scores and readiness signals are heuristics, not proof that a result answers the question. + +Reference lookup checks current text before parsing, including references added after indexing in files recorded by the last successful index. It reads every recorded source, even when the keyword index's chunk cap retained chunks from only some files; files created after indexing require a reindex. Unavailable sources yield explicitly incomplete coverage; lexical fallback can include comments or strings and is labeled uncertain. If the reference inventory is missing or too old, run `codebase-context reindex` once without `--incremental`. Directed graph traversal verifies edges before consuming the returned path/node caps, with a separate inspection budget and explicit exhaustion. Dense stale neighborhoods can still require refreshing the index. + +Newly written chunk paths are relative to the selected project root, independent of the caller's working directory. An existing index created with incorrect working-directory-relative paths may require a full `reindex`; an unchanged incremental run does not migrate those artifacts. ## Transport Modes @@ -40,7 +54,7 @@ For a command gallery with examples, see `docs/cli.md`. | `search --query ` | `--intent explore\|edit\|refactor\|migrate`, `--limit `, `--lang `, `--framework `, `--layer ` | `search_codebase` | | `metadata` | — | `get_codebase_metadata` | | `status` | — | `get_indexing_status` | -| `reindex` | `--incremental`, `--reason ` | equivalent to `refresh_index` | +| `reindex` | `--incremental`, `--reason `, candidate-only `--embeddings` / `--no-embeddings` | equivalent to `refresh_index` | | `style-guide` | `--query `, `--category ` | `get_style_guide` | | `patterns` | `--category all\|di\|state\|testing\|libraries` | `get_team_patterns` | | `refs --symbol ` | `--limit ` | `get_symbol_references` | @@ -62,7 +76,7 @@ npx -y codebase-context@2.2.0 reindex --incremental ## Tool Surface -10 MCP tools + active/project-scoped context resources. +12 MCP tools + active/project-scoped context resources. Shared selector inputs: @@ -77,10 +91,14 @@ Shared selector inputs: | ----------------------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `search_codebase` | `query`, optional `intent`, `limit`, `filters`, `includeSnippets`, shared `project`/`project_directory` | Compact mode returns a bounded result set with `file`, `summary`, `score`, lightweight structural metadata (`symbol`, `symbolKind`, `scope`, `signaturePreview`), and `searchQuality` (`status`, `confidence`, optional `hint`, `tokenEstimate`, `warning`, `rerankerStatus`). Full mode adds richer relationships plus chunk-level `imports`, `exports`, and `complexity`. When `intent="edit"`, a decision card is returned with `ready`, `patterns`, `bestExample`, `impact`, and `whatWouldHelp`. | | `get_team_patterns` | optional `category`, shared `project`/`project_directory` | Pattern frequencies, trends, golden files, conflicts | -| `get_symbol_references` | `symbol`, optional `limit`, shared `project`/`project_directory` | Concrete symbol usage evidence: `usageCount` + top usage snippets + `confidence` + `isComplete`. `confidence: "syntactic"` means static/source-based only (no runtime or dynamic dispatch). When Tree-sitter + file content are available, comments and string literals are excluded from the scan — the count reflects real identifier nodes only. Replaces the removed `get_component_usage`. | +| `get_symbol_references` | `symbol`, optional `limit`, shared `project`/`project_directory` | Current-source matches: observed `usageCount`, bounded snippets, `confidence`, `isComplete`, and coverage counts. Replaces the removed `get_component_usage`. | | `remember` | `type`, `category`, `memory`, `reason`, shared `project`/`project_directory` | Persists to `.codebase-context/memory.json` | | `get_memory` | optional `category`, `type`, `query`, `limit`, shared `project`/`project_directory` | Memories with confidence decay scoring | +`confidence: "syntactic"` means Tree-sitter identifier-node matches; comments and strings are excluded, but references are not limited to call sites. `confidence: "lexical"` means raw-text token-boundary fallback because parsing was unavailable or unsupported; comments and strings may be included, so treat matches as uncertain. + +`coverage` counts deduplicated manifest-listed sources (`candidateFiles`), sources read (`sourcesRead`), read sources without the symbol text (`filesWithoutText`), unavailable sources (`filesUnavailable`), parser attempts (`parserCalls`), and lexical fallback scans (`lexicalFallbackFiles`). Each available candidate source is read before checking whether parser work can be skipped. `usageCount` reports matches across the manifest-listed source set, and `isComplete: false` can mean the result limit was reached or candidate sources were unavailable; files added after the index are outside that coverage. + ### Utility Tools | Tool | Purpose | @@ -90,6 +108,10 @@ Shared selector inputs: | `detect_circular_dependencies` | Import cycles in the file graph | | `refresh_index` | Full or incremental re-index + git memory extraction | | `get_indexing_status` | Index state, progress, last stats | +| `get_codebase_health` | Highest-risk indexed files and health reasons; optionally one file | +| `get_code_relationships` | Verified directed import/re-export paths with source citations and freshness diagnostics | + +The `--embeddings` and `--no-embeddings` reindex flags, including how omitted mode flags preserve the project's successful mode, are candidate-only; see the [CLI reindex guide](./cli.md#reindex). Published `2.2.0` examples above do not include these flags. ## Project Routing @@ -175,14 +197,14 @@ Ordered by execution: 1. **Intent classification** — EXACT_NAME (for symbols), CONCEPTUAL, FLOW, CONFIG, WIRING. Sets keyword/semantic weight ratio. 2. **Query expansion** — bounded domain term expansion for conceptual queries. -3. **Dual retrieval** — keyword (Fuse.js) + semantic (local embeddings or OpenAI). +3. **Retrieval** — keyword (Fuse.js) by default for new indexes; semantic retrieval joins it only for an embedding-enabled index. 4. **RRF fusion** — Reciprocal Rank Fusion (k=60) across all retrieval channels. 5. **Definition-first boost** — for EXACT_NAME intent, results matching the symbol name get +15% score boost (e.g., defining file ranks above using files). 6. **Structure-aware boosting** — import centrality, composition root boost, path overlap, definition demotion for action queries. 7. **Contamination control** — test file filtering for non-test queries. 8. **File deduplication** — best chunk per file. 9. **Symbol-level deduplication** — within each `symbolPath` group, keep only the highest-scoring chunk (prevents duplicate methods from same class clogging results). -10. **Stage-2 reranking** — cross-encoder (`Xenova/ms-marco-MiniLM-L-6-v2`) triggers when the score between the top files are very close. CPU-only, top-10 bounded. +10. **Optional stage-2 reranking** — for embedding-enabled indexes, a cross-encoder (`Xenova/ms-marco-MiniLM-L-6-v2`) can trigger when top scores are close. Model-free indexes skip it. CPU-only, top-10 bounded. 11. **Result enrichment** — compact type (`componentType:layer`), pattern momentum (`trend` Rising/Declining only, Stable omitted), `patternWarning`, condensed relationships (`importedByCount`/`hasTests`), structured hints (capped callers/consumers/tests ranked by frequency), scope header for symbol-aware snippets (`// ClassName.methodName`), related memories (capped to 3), search quality assessment with `hint` when low confidence. 12. **Payload budgeting** — final serialized search responses include `searchQuality.tokenEstimate`; warnings only appear above the 4K-token threshold and differ between compact and full mode. 13. **Full-mode chunk metadata** — when available, full-mode results surface chunk-level `imports` (top 5), `exports` (top 5), and cyclomatic `complexity`. @@ -190,9 +212,12 @@ Ordered by execution: ### Defaults - **Chunk size**: 50 lines, 0 overlap -- **Reranker trigger**: activates when top-3 results are within 0.08 score of each other -- **Embedding model**: `Xenova/bge-small-en-v1.5` (512 token context, fast, local-first) via `@huggingface/transformers`. Override: `EMBEDDING_MODEL=onnx-community/granite-embedding-small-english-r2-ONNX` for Granite (8192 ctx, slower). -- **Vector DB**: LanceDB with cosine distance +- **New projects**: model-free; no embedding/reranker initialization. Existing valid indexes preserve their successful mode. +- **Opt-in**: CLI `reindex --embeddings` or MCP `refresh_index` with `skipEmbedding:false`; `--no-embeddings`/`skipEmbedding:true` select model-free. Mode switches rebuild fully. Explicit startup/config choices are enforced before search. +- **Optional semantic runtimes**: `@huggingface/transformers` and `@lancedb/lancedb` are optional peers, not default install dependencies. Missing runtimes fail before index replacement. +- **Semantic model**: `Xenova/bge-small-en-v1.5` (512 token context); model overrides apply only in semantic mode. Semantic storage uses LanceDB with cosine distance. + +Compact results retain the `signaturePreview` key: it contains one contiguous window of up to three lines around the strongest query-term match in the returned source snippet, or the first three lines when none match. It does not combine distant source lines. Default and explicit exploration report retrieval quality and incomplete source coverage without an edit-readiness card. ## Decision Card (Edit Intent) @@ -259,18 +284,18 @@ Impact is 2-hop transitive: direct importers (hop 1) and their importers (hop 2) ## Indexing -- Initial: full scan → chunking (50 lines, 0 overlap) → embedding → vector DB (LanceDB) + keyword index (Fuse.js) -- Incremental: SHA-256 manifest diffing, selective embed/delete, full intelligence regeneration +- Initial: full scan → chunking → keyword/symbol/structural artifacts; embeddings and LanceDB are optional semantic additions. +- Incremental: source-hash diffing, selective update/delete, full intelligence regeneration; drift during analysis fails closed instead of endorsing older analyzed bytes with newer hashes. - Auto-refresh (MCP server mode only): chokidar file watcher triggers incremental reindex after a debounce on any source file change — `node_modules/`, `.git/`, `dist/`, and `.codebase-context/` are excluded. One-shot CLI runs skip the watcher entirely. -- Version gating: `index-meta.json` tracks format version; mismatches trigger automatic rebuild +- Version gating: index format 4 also changes analyzer selection for TypeScript files without Angular imports, invalidating format 3 and earlier indexes. Earlier changes corrected AST boundaries and persisted chunk/embedding input, including descriptor methods omitted by format 2. Incompatible indexes rebuild even on an unchanged incremental request; the last successful model-free or semantic mode is preserved. Previously generated embeddings are not reused across this migration. - Crash-safe rebuilds: full rebuilds write to `.staging/` and swap atomically only on success - Auto-heal: corrupted index triggers automatic full re-index on next search -- Relationships sidecar: `relationships.json` contains file import graph, symbol export index, and per-edge import details (`importDetails`: line number + imported symbol names where available) +- Relationships sidecar: `relationships.json` contains the file import graph, symbol export index, typed import/re-export source citations and endpoint fingerprints. The query checks current resolution and source before following stored edges. - Storage: `.codebase-context/` directory (memory.json + generated files) ## Analyzers -- **Angular**: signals, standalone components, control flow syntax, lifecycle hooks, DI patterns, component metadata +- **Angular**: signals, standalone components, control flow syntax, lifecycle hooks, DI patterns, component metadata. TypeScript eligibility requires parsed imports from the Angular namespace; method or decorator spellings alone do not establish Angular identity. Files without that evidence use another eligible analyzer, including Generic AST analysis. - **React**: function/class components, custom hooks, context usage, memoization, Suspense, ecosystem signal extraction - **Next.js**: App Router and Pages Router detection, route/API classification, route paths, `"use client"`, metadata exports - **Generic**: 30+ have indexing/retrieval coverage including PHP, Ruby, Swift, Scala, Shell, config/markup., 10 languages have full symbol extraction (Tree-sitter: TypeScript, JavaScript, Python, Java, Kotlin, C, C++, C#, Go, Rust). @@ -278,7 +303,7 @@ Impact is 2-hop transitive: direct importers (hop 1) and their importers (hop 2) Notes: - Language detection covers common extensions including `.pyi`, `.kt`/`.kts`, `.cc`/`.cxx`, and config formats like `.toml`/`.xml`. -- When Tree-sitter grammars are present, the Generic analyzer uses AST-aligned chunking and scope-aware prefixes for symbol-aware snippets (with fallbacks). +- When Tree-sitter grammars are present, the Generic analyzer uses AST-aligned chunking (with fallbacks). Static callable properties retain their owning descriptor context; dynamic property names do not invent a static ownership path. Chunks persist contiguous source spans, with oversized chunks split from that source; scope labels are presentation metadata, not stored source or declaration evidence. ## Evaluation Harness @@ -299,8 +324,9 @@ Reproducible evaluation is shipped as a CLI entrypoint backed by shared scoring ## Limitations -- **Symbol refs are not a call-graph.** `get_symbol_references` counts identifier-node occurrences in the AST (comments/strings excluded via Tree-sitter). It does not distinguish call sites from type annotations, variable assignments, or imports. Full call-site-specific analysis (`call_expression` nodes only) is a roadmap item. -- **Impact is 2-hop max.** `computeImpactCandidates` walks direct importers then their importers. Full BFS reachability is on the roadmap. +- **Symbol refs are not a call-graph.** With syntactic confidence, `get_symbol_references` counts identifier-node occurrences in the AST (comments/strings excluded via Tree-sitter). It does not distinguish call sites from type annotations, variable assignments, or imports. Lexical fallback uses raw text and may include comments or strings. Full call-site-specific analysis (`call_expression` nodes only) is a roadmap item. + Reference scans still read current manifest sources. A conservative ASCII boundary check can skip parsing longer-identifier interiors; Unicode and arbitrary query spellings retain raw-text nomination. `filesWithoutText` counts actual raw-text absence, so it does not count those interior-only skips. +- **Decision-card impact is 2-hop max.** The separate relationship tool follows verified imports/re-exports up to three hops; neither establishes a runtime call graph. - **Published 2.2.0 has dedicated Angular, React, and Next.js analyzers.** NestJS support is in a newer source candidate and is not part of published 2.2.0. Other languages use the Generic analyzer (30+ languages, chunking + import graph, no framework-specific signal extraction). -- **Default embedding model is `bge-small-en-v1.5` (512-token context).** Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` — sends code externally. +- **Model-free conceptual search has limits.** Keyword/symbol matches can miss meaning or return an unrelated high-scoring file. Follow current source and relationship evidence; no general retrieval-quality improvement is claimed. Embeddings are explicit opt-in. OpenAI additionally requires an opt-in provider/key and sends code externally. - **Patterns are file-level frequency counts.** Not semantic clustering. Rising/Declining trend is derived from git commit recency for files using each pattern, not from usage semantics. diff --git a/docs/cli.md b/docs/cli.md index dfda760..5f03f14 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -46,6 +46,19 @@ CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context@2.2.0 patterns - `memory list|add|remove` — manage team memory (stored in `.codebase-context/memory.json`) - MCP client registration — use the published 2.2.0 commands in the [client setup guide](./client-setup.md); published `init` is not a reliable setup route +## `init` + +`init` detects Claude Code, Codex, Cursor and OpenCode, then configures the selected tools together. In a terminal, detected tools start selected; use Space to change the selection and Enter to apply it. Without a terminal, it configures all detected tools. + +```powershell +codebase-context init --help +codebase-context init +codebase-context init --yes +codebase-context init --client codex --client cursor --root "C:\path\to\project" +``` + +Use repeatable `--client` options to choose tools explicitly. `--yes` configures every detected tool when no clients are named. `--root` selects another repository; otherwise `CODEBASE_ROOT` or the current directory is used. Setup writes client registrations but does not build an index or change agent instructions. The first search creates the index. The multi-tool flow is not in published npm `2.2.0` yet; see the [main setup guide](../README.md#set-up-your-coding-tools) for its publication status and the [client setup guide](./client-setup.md) for current manual recipes. + --- ## First use @@ -217,6 +230,18 @@ Example output: ## `reindex` +In the unreleased model-free candidate, use the locally installed package: + +```bash +CODEBASE_ROOT=/path/to/repository node /path/to/cbc/node_modules/codebase-context/dist/index.js reindex +CODEBASE_ROOT=/path/to/repository node /path/to/cbc/node_modules/codebase-context/dist/index.js reindex --incremental +CODEBASE_ROOT=/path/to/repository node /path/to/cbc/node_modules/codebase-context/dist/index.js reindex --no-embeddings +# Requires the optional Transformers and LanceDB runtimes; may download models. +CODEBASE_ROOT=/path/to/repository node /path/to/cbc/node_modules/codebase-context/dist/index.js reindex --embeddings +``` + +New projects default to model-free. Omit mode flags to preserve the successful mode; explicit changes require a full rebuild even with `--incremental`. The two mode flags are mutually exclusive and index failures return nonzero. The examples below target the older published package. + ```bash npx -y codebase-context@2.2.0 reindex npx -y codebase-context@2.2.0 reindex --incremental --reason "changed watcher logic" diff --git a/docs/client-setup.md b/docs/client-setup.md index 7770e7f..efccc81 100644 --- a/docs/client-setup.md +++ b/docs/client-setup.md @@ -1,8 +1,8 @@ # Client Setup -Codebase Context runs as a local MCP server: a tool your AI editor or command-line client can call while it works. Register it without a fixed folder, then let the agent supply its current repository through the `project` tool parameter. These examples use published npm `2.2.0`. +Codebase Context runs as a local MCP server: a tool your AI editor or command-line client can call while it works. Register it without a fixed folder, then let the agent supply its current repository through the `project` tool parameter. These examples use published npm `2.2.0`. The one-prompt multi-tool flow is described in [Set up your coding tools](../README.md#set-up-your-coding-tools) and is awaiting publication. -Use Node.js 22 or newer for these recipes. The package declares Node 18+, but native dependency requirements are narrower than that broad declaration; these checks do not establish Node 18.0 compatibility. +Use Node.js 22 or newer for these recipes. The local product candidate requires Node 22+ and is checked on Windows Node 22.23.3. Published 2.2.0 declares Node 18+, but its current UUID and SDK dependency chain does not establish Node 18 compatibility; use Node 22 with the published recipes too. The copyable registration commands target PowerShell or a POSIX terminal. Windows Command Prompt was not tested; OpenCode's quoted separator is shell-dependent. @@ -73,7 +73,7 @@ Equivalent project config: [mcp_servers.codebase-context] command = "npx" args = ["-y", "codebase-context@2.2.0"] -startup_timeout_sec = 120 +startup_timeout_sec = 300 ``` **What was checked:** The command ran with an isolated `CODEX_HOME`, wrote `config.toml`, and `codex mcp get` returned the same command and arguments. @@ -90,50 +90,26 @@ Create or merge `.codex/config.toml` in the project you want to search. Preserve [mcp_servers.codebase-context] command = "npx" args = ["-y", "codebase-context@2.2.0"] -startup_timeout_sec = 120 +startup_timeout_sec = 300 ``` Trust the project if asked, restart Codex and start a new task there. Use the first-use prompt above so the agent selects the current repository. The CLI command in the preceding section writes user-level config; this recipe uses a trusted project's config instead. -**Verification boundary:** isolated Windows CLI and SDK/MCP checks on October 5 indexed a six-file fixture with published `2.2.0`, selected its project, retrieved context and found a source that the host could open. The exact `npx -y codebase-context@2.2.0 ` entrypoint also passed using a warmed isolated npm cache and prepared index. This does not establish a fully cold download or a fresh Codex Desktop agent session. The September Desktop session below used a packed candidate; keep it separate from published-package acceptance. +**Verification boundary:** isolated Windows CLI and SDK/MCP checks on October 5 indexed a six-file fixture with published `2.2.0`, selected its project, retrieved context and found a source that the host could open. The exact `npx -y codebase-context@2.2.0 ` entrypoint also passed using a warmed isolated npm cache and prepared index. This does not establish a fully cold download or a fresh Codex Desktop agent session. On October 6 the exact published `npx` entrypoint was also tested without a folder, environment root or MCP roots, launched from an unrelated directory. Its first tool call returned `selection_required`; an explicit `project` call selected the prepared fixture, search found the expected symbol, context returned its map, and the runner host opened the returned source. A later call without `project` retained that selection. This is SDK/protocol and runner-host evidence with warmed caches and a pre-indexed fixture, not a native Desktop task. -## Unreleased installer candidate - -The next release is expected to support one project-only command from the target project: - -The `init --client codex --yes` flow is an unreleased source candidate. Published npm `2.2.0` does not include these flags. Use the published configuration above rather than an `@latest` installer command; candidate verification uses an exact built or installed candidate. - -A successful `init --client codex --yes` writes the project-scoped `.codex/config.toml` and prepares the index for that root. If config writing or index preparation fails, setup is incomplete and Codex is not ready to use. - -For candidate verification before publication, build the source checkout and run its absolute `dist/index.js` path from the project you want to configure: - -```powershell -# In the codebase-context source checkout -npm run build - -# In the target project, using the built CLI path -node C:\path\to\codebase-context\dist\index.js init --client codex --yes -``` - -Use `--root C:\path\to\project` when the command is run from elsewhere. Here, `root` means the project folder to configure and index. The candidate writes the project-scoped `/.codex/config.toml`, preserves unrelated Codex settings, prepares the index for that folder, and does not write `AGENTS.md` or global client configuration. After a successful result, trust the project in Codex Desktop if prompted, restart or reopen Codex, start a new task in that project, and ask a small project question. A written config and ready index do not prove that Codex loaded or queried the server. - -Before release, the generated `npx` server command still resolves the published package unless the candidate is installed in the target project's `node_modules`. Our packaged test installs the candidate in a disposable project and checks its file hash. Running the built setup command alone does not prove that Codex will launch that same candidate. - -**Checked on 2026-09-15:** the packed candidate installed and indexed an isolated Windows project. A fresh Codex Desktop task then called the actual `search_codebase` MCP tool and located the requested chart component with its source file and lines. The normal Desktop task flow established project trust; no trust-file edits or replacement CLI/SDK search were used for that answer. This covers the small prepared project with shared caches, not a cold-machine install or the published npm package. - ## Optional fixed-folder setup and CLI preparation To set a default project before the first tool call, append its full path to the server arguments, for example `args = ["-y", "codebase-context@2.2.0", "C:/projects/my-app"]`. Use a quoted literal shell argument in registration commands. This makes the default fixed across repositories when the registration is user-scoped; explicit tool selection can still choose another allowed project. -To prepare the index separately, run from the project folder: +To prepare the index separately with the published CLI, run from the project folder: ```powershell npx -y codebase-context@2.2.0 reindex ``` -Do not use published `2.2.0`'s interactive `init` as the setup route. October 6 isolated checks found that it ignores `--help`, `--client` and `--yes`; its Claude/Codex registration executes `mcp` instead of the client executable, and its HTTP recipes require a separately started server. It can exit zero after registration fails. The unreleased installer above is a separate implementation. +Do not use published `2.2.0`'s interactive `init` as the setup route. October 6 isolated checks found that it ignores `--help`, `--client` and `--yes`; its Claude/Codex registration executes `mcp` instead of the client executable, and its HTTP recipes require a separately started server. It can exit zero after registration fails. Until the updated multi-tool setup is published, use the manual recipes above. ## Gemini CLI @@ -367,3 +343,5 @@ Then replace the `npx` command in your client config with the local entry point: ``` For one explicit project, append its path to the arguments or set `CODEBASE_ROOT`. + +The unreleased source candidate adds an installation diagnostic to `get_indexing_status`: it reports both `rootPath` (the selected repository) and `loadedStatusModulePath` (the status code this server process actually loaded). If the loaded module points to an unexpected cache or checkout, update that client's server command to the intended installation and reconnect; CBC does not change client settings automatically. diff --git a/package.json b/package.json index 2b75047..7e49f4b 100644 --- a/package.json +++ b/package.json @@ -60,7 +60,7 @@ ], "packageManager": "pnpm@10.27.0", "engines": { - "node": ">=18.0.0" + "node": ">=22.0.0" }, "keywords": [ "mcp", @@ -118,7 +118,7 @@ "sync:grammars": "node scripts/sync-grammars.mjs", "prebuild": "pnpm run sync:grammars", "build": "tsc", - "prepack": "pnpm run sync:grammars", + "prepack": "pnpm run build", "prepublishOnly": "pnpm run build", "pretest": "pnpm run sync:grammars", "start": "node dist/index.js", @@ -135,12 +135,10 @@ "prepare": "husky" }, "dependencies": { - "@huggingface/transformers": "^4.2.0", "@inquirer/prompts": "^3.0.0", - "@lancedb/lancedb": "^0.4.0", "@modelcontextprotocol/sdk": "^1.27.1", - "@typescript-eslint/typescript-estree": "^7.0.0", - "chokidar": "^3.6.0", + "@typescript-eslint/typescript-estree": "^8.51.0", + "chokidar": "^4.0.3", "fuse.js": "^7.0.0", "glob": "^10.3.10", "hono": "^4.12.12", @@ -150,7 +148,21 @@ "web-tree-sitter": "^0.25.10", "zod": "^4.3.4" }, + "peerDependencies": { + "@huggingface/transformers": "^4.2.0", + "@lancedb/lancedb": "^0.4.0" + }, + "peerDependenciesMeta": { + "@huggingface/transformers": { + "optional": true + }, + "@lancedb/lancedb": { + "optional": true + } + }, "devDependencies": { + "@huggingface/transformers": "^4.2.0", + "@lancedb/lancedb": "^0.4.0", "@eslint/js": "^9.39.2", "@types/glob": "^8.1.0", "@types/node": "^20.11.24", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index b550f49..925d285 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -23,24 +23,18 @@ importers: .: dependencies: - '@huggingface/transformers': - specifier: ^4.2.0 - version: 4.2.0 '@inquirer/prompts': specifier: ^3.0.0 version: 3.3.2 - '@lancedb/lancedb': - specifier: ^0.4.0 - version: 0.4.20(zod@4.3.4) '@modelcontextprotocol/sdk': specifier: ^1.27.1 version: 1.27.1(zod@4.3.4) '@typescript-eslint/typescript-estree': - specifier: ^7.0.0 - version: 7.18.0(typescript@5.9.3) + specifier: ^8.51.0 + version: 8.51.0(typescript@5.9.3) chokidar: - specifier: ^3.6.0 - version: 3.6.0 + specifier: ^4.0.3 + version: 4.0.3 fuse.js: specifier: ^7.0.0 version: 7.1.0 @@ -69,6 +63,12 @@ importers: '@eslint/js': specifier: ^9.39.2 version: 9.39.2 + '@huggingface/transformers': + specifier: ^4.2.0 + version: 4.2.0 + '@lancedb/lancedb': + specifier: ^0.4.0 + version: 0.4.20(zod@4.3.4) '@types/glob': specifier: ^8.1.0 version: 8.1.0 @@ -380,105 +380,89 @@ packages: resolution: {integrity: sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==} cpu: [arm64] os: [linux] - libc: [glibc] '@img/sharp-libvips-linux-arm@1.2.4': resolution: {integrity: sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==} cpu: [arm] os: [linux] - libc: [glibc] '@img/sharp-libvips-linux-ppc64@1.2.4': resolution: {integrity: sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==} cpu: [ppc64] os: [linux] - libc: [glibc] '@img/sharp-libvips-linux-riscv64@1.2.4': resolution: {integrity: sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==} cpu: [riscv64] os: [linux] - libc: [glibc] '@img/sharp-libvips-linux-s390x@1.2.4': resolution: {integrity: sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==} cpu: [s390x] os: [linux] - libc: [glibc] '@img/sharp-libvips-linux-x64@1.2.4': resolution: {integrity: sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==} cpu: [x64] os: [linux] - libc: [glibc] '@img/sharp-libvips-linuxmusl-arm64@1.2.4': resolution: {integrity: sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==} cpu: [arm64] os: [linux] - libc: [musl] '@img/sharp-libvips-linuxmusl-x64@1.2.4': resolution: {integrity: sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==} cpu: [x64] os: [linux] - libc: [musl] '@img/sharp-linux-arm64@0.34.5': resolution: {integrity: sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [arm64] os: [linux] - libc: [glibc] '@img/sharp-linux-arm@0.34.5': resolution: {integrity: sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [arm] os: [linux] - libc: [glibc] '@img/sharp-linux-ppc64@0.34.5': resolution: {integrity: sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [ppc64] os: [linux] - libc: [glibc] '@img/sharp-linux-riscv64@0.34.5': resolution: {integrity: sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [riscv64] os: [linux] - libc: [glibc] '@img/sharp-linux-s390x@0.34.5': resolution: {integrity: sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [s390x] os: [linux] - libc: [glibc] '@img/sharp-linux-x64@0.34.5': resolution: {integrity: sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [x64] os: [linux] - libc: [glibc] '@img/sharp-linuxmusl-arm64@0.34.5': resolution: {integrity: sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [arm64] os: [linux] - libc: [musl] '@img/sharp-linuxmusl-x64@0.34.5': resolution: {integrity: sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [x64] os: [linux] - libc: [musl] '@img/sharp-wasm32@0.34.5': resolution: {integrity: sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==} @@ -571,14 +555,12 @@ packages: engines: {node: '>= 18'} cpu: [arm64] os: [linux] - libc: [glibc] '@lancedb/lancedb-linux-x64-gnu@0.4.20': resolution: {integrity: sha512-TVicKBhT3Q73gtWmMdR/0+r4gBve5gw3KbcrvzHP/5bNpo2sffETbLEMgLjulunu/Jp8IgC921WW6IFJzNFwrQ==} engines: {node: '>= 18'} cpu: [x64] os: [linux] - libc: [glibc] '@lancedb/lancedb-win32-x64-msvc@0.4.20': resolution: {integrity: sha512-7RsTJAx93NbVD//9m0/TpgQmPsTHlCBfGNFD2zkdZ0j7iacURdG9+egz+Os0veTBqMgwnr6VfMf3T7S0wuu99g==} @@ -602,18 +584,6 @@ packages: '@cfworker/json-schema': optional: true - '@nodelib/fs.scandir@2.1.5': - resolution: {integrity: sha512-vq24Bq3ym5HEQm2NKCr3yXDwjc7vTsEThRDnkp2DK9p1uqLR+DHurm/NOTo0KG7HYHU7eppKZj3MyqYuMBf62g==} - engines: {node: '>= 8'} - - '@nodelib/fs.stat@2.0.5': - resolution: {integrity: sha512-RkhPPp2zrqDAQA/2jNhnztcPAlv64XdhIp7a7454A5ovI7Bukxgt7MX7udwAu3zg1DcpPU0rz3VV1SeaqvY4+A==} - engines: {node: '>= 8'} - - '@nodelib/fs.walk@1.2.8': - resolution: {integrity: sha512-oGB+UxlgWcgQkgwo8GcEGwemoTFt3FIO9ababBmaGwXIoBKZ+GTy0pP185beGg7Llih/NSHSV2XAs1lnznocSg==} - engines: {node: '>= 8'} - '@pkgjs/parseargs@0.11.0': resolution: {integrity: sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==} engines: {node: '>=14'} @@ -682,79 +652,66 @@ packages: resolution: {integrity: sha512-t4ONHboXi/3E0rT6OZl1pKbl2Vgxf9vJfWgmUoCEVQVxhW6Cw/c8I6hbbu7DAvgp82RKiH7TpLwxnJeKv2pbsw==} cpu: [arm] os: [linux] - libc: [glibc] '@rollup/rollup-linux-arm-musleabihf@4.59.0': resolution: {integrity: sha512-CikFT7aYPA2ufMD086cVORBYGHffBo4K8MQ4uPS/ZnY54GKj36i196u8U+aDVT2LX4eSMbyHtyOh7D7Zvk2VvA==} cpu: [arm] os: [linux] - libc: [musl] '@rollup/rollup-linux-arm64-gnu@4.59.0': resolution: {integrity: sha512-jYgUGk5aLd1nUb1CtQ8E+t5JhLc9x5WdBKew9ZgAXg7DBk0ZHErLHdXM24rfX+bKrFe+Xp5YuJo54I5HFjGDAA==} cpu: [arm64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-arm64-musl@4.59.0': resolution: {integrity: sha512-peZRVEdnFWZ5Bh2KeumKG9ty7aCXzzEsHShOZEFiCQlDEepP1dpUl/SrUNXNg13UmZl+gzVDPsiCwnV1uI0RUA==} cpu: [arm64] os: [linux] - libc: [musl] '@rollup/rollup-linux-loong64-gnu@4.59.0': resolution: {integrity: sha512-gbUSW/97f7+r4gHy3Jlup8zDG190AuodsWnNiXErp9mT90iCy9NKKU0Xwx5k8VlRAIV2uU9CsMnEFg/xXaOfXg==} cpu: [loong64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-loong64-musl@4.59.0': resolution: {integrity: sha512-yTRONe79E+o0FWFijasoTjtzG9EBedFXJMl888NBEDCDV9I2wGbFFfJQQe63OijbFCUZqxpHz1GzpbtSFikJ4Q==} cpu: [loong64] os: [linux] - libc: [musl] '@rollup/rollup-linux-ppc64-gnu@4.59.0': resolution: {integrity: sha512-sw1o3tfyk12k3OEpRddF68a1unZ5VCN7zoTNtSn2KndUE+ea3m3ROOKRCZxEpmT9nsGnogpFP9x6mnLTCaoLkA==} cpu: [ppc64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-ppc64-musl@4.59.0': resolution: {integrity: sha512-+2kLtQ4xT3AiIxkzFVFXfsmlZiG5FXYW7ZyIIvGA7Bdeuh9Z0aN4hVyXS/G1E9bTP/vqszNIN/pUKCk/BTHsKA==} cpu: [ppc64] os: [linux] - libc: [musl] '@rollup/rollup-linux-riscv64-gnu@4.59.0': resolution: {integrity: sha512-NDYMpsXYJJaj+I7UdwIuHHNxXZ/b/N2hR15NyH3m2qAtb/hHPA4g4SuuvrdxetTdndfj9b1WOmy73kcPRoERUg==} cpu: [riscv64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-riscv64-musl@4.59.0': resolution: {integrity: sha512-nLckB8WOqHIf1bhymk+oHxvM9D3tyPndZH8i8+35p/1YiVoVswPid2yLzgX7ZJP0KQvnkhM4H6QZ5m0LzbyIAg==} cpu: [riscv64] os: [linux] - libc: [musl] '@rollup/rollup-linux-s390x-gnu@4.59.0': resolution: {integrity: sha512-oF87Ie3uAIvORFBpwnCvUzdeYUqi2wY6jRFWJAy1qus/udHFYIkplYRW+wo+GRUP4sKzYdmE1Y3+rY5Gc4ZO+w==} cpu: [s390x] os: [linux] - libc: [glibc] '@rollup/rollup-linux-x64-gnu@4.59.0': resolution: {integrity: sha512-3AHmtQq/ppNuUspKAlvA8HtLybkDflkMuLK4DPo77DfthRb71V84/c4MlWJXixZz4uruIH4uaa07IqoAkG64fg==} cpu: [x64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-x64-musl@4.59.0': resolution: {integrity: sha512-2UdiwS/9cTAx7qIUZB/fWtToJwvt0Vbo0zmnYt7ED35KPg13Q0ym1g442THLC7VyI6JfYTP4PiSOWyoMdV2/xg==} cpu: [x64] os: [linux] - libc: [musl] '@rollup/rollup-openbsd-x64@4.59.0': resolution: {integrity: sha512-M3bLRAVk6GOwFlPTIxVBSYKUaqfLrn8l0psKinkCFxl4lQvOSz8ZrKDz2gxcBwHFpci0B6rttydI4IpS4IS/jQ==} @@ -878,23 +835,10 @@ packages: eslint: ^8.57.0 || ^9.0.0 typescript: '>=4.8.4 <6.0.0' - '@typescript-eslint/types@7.18.0': - resolution: {integrity: sha512-iZqi+Ds1y4EDYUtlOOC+aUmxnE9xS/yCigkjA7XpTKV6nCBd3Hp/PRGGmdwnfkV2ThMyYldP1wRpm/id99spTQ==} - engines: {node: ^18.18.0 || >=20.0.0} - '@typescript-eslint/types@8.51.0': resolution: {integrity: sha512-TizAvWYFM6sSscmEakjY3sPqGwxZRSywSsPEiuZF6d5GmGD9Gvlsv0f6N8FvAAA0CD06l3rIcWNbsN1e5F/9Ag==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} - '@typescript-eslint/typescript-estree@7.18.0': - resolution: {integrity: sha512-aP1v/BSPnnyhMHts8cf1qQ6Q1IFwwRvAQGRvBFkWlo3/lH29OXA3Pts+c10nxRxIBrDnoMqzhgdwVe5f2D6OzA==} - engines: {node: ^18.18.0 || >=20.0.0} - peerDependencies: - typescript: '*' - peerDependenciesMeta: - typescript: - optional: true - '@typescript-eslint/typescript-estree@8.51.0': resolution: {integrity: sha512-1qNjGqFRmlq0VW5iVlcyHBbCjPB7y6SxpBkrbhNWMy/65ZoncXCEPJxkRZL8McrseNH6lFhaxCIaX+vBuFnRng==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} @@ -908,10 +852,6 @@ packages: eslint: ^8.57.0 || ^9.0.0 typescript: '>=4.8.4 <6.0.0' - '@typescript-eslint/visitor-keys@7.18.0': - resolution: {integrity: sha512-cDF0/Gf81QpY3xYyJKDV14Zwdmid5+uuENhjH2EqFaF0ni+yAyq/LzMaIJdhNJXZI7uLzwIlA+V7oWoyn6Curg==} - engines: {node: ^18.18.0 || >=20.0.0} - '@typescript-eslint/visitor-keys@8.51.0': resolution: {integrity: sha512-mM/JRQOzhVN1ykejrvwnBRV3+7yTKK8tVANVN3o1O0t0v7o+jqdVu9crPy5Y9dov15TJk/FTIgoUGHrTOVL3Zg==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} @@ -1005,10 +945,6 @@ packages: resolution: {integrity: sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg==} engines: {node: '>=12'} - anymatch@3.1.3: - resolution: {integrity: sha512-KMReFUr0B4t+D+OBkjR3KYqvocp2XaSzO55UcB6mgQMd3KbcE+mWTyvVV7D/zsdEbNnV6acZUutkiHQXvTr1Rw==} - engines: {node: '>= 8'} - apache-arrow@15.0.2: resolution: {integrity: sha512-RvwlFxLRpO405PLGffx4N2PYLiF7FD86Q1hHl6J2XCWiq+tTCzpb9ngFw0apFDcXZBMpCzMuwAvA7hjyL1/73A==} hasBin: true @@ -1032,10 +968,6 @@ packages: resolution: {integrity: sha512-FmeCCAenzH0KH381SPT5FZmiA/TmpndpcaShhfgEN9eCVjnFBqq3l1xrI42y8+PPLI6hypzou4GXw00WHmPBLQ==} engines: {node: '>= 0.4'} - array-union@2.1.0: - resolution: {integrity: sha512-HGyxoOTYUyCM6stUe6EJgnd4EoewAI7zMdfqO+kGjnlZmBDz/cR5pf8r/cR4Wq60sL/p0IkcjUEEPwS3GFrIyw==} - engines: {node: '>=8'} - array.prototype.findlastindex@1.2.6: resolution: {integrity: sha512-F/TKATkzseUExPlfvmwQKGITM3DGTK+vkAsCZoDc5daVygbJBnjEUCbgkAvVFsgfXfX4YIqZ/27G3k3tdXrTxQ==} engines: {node: '>= 0.4'} @@ -1071,10 +1003,6 @@ packages: resolution: {integrity: sha512-1pHv8LX9CpKut1Zp4EXey7Z8OfH11ONNH6Dhi2WDUt31VVZFXZzKwXcysBgqSumFCmR+0dqjMK5v5JiFHzi0+g==} engines: {node: 20 || >=22} - binary-extensions@2.3.0: - resolution: {integrity: sha512-Ceh+7ox5qe7LJuLHoY0feh3pHuUDHAcRUeyL2VYghZwfpkNIy/+8Ocg0a3UuSoYzavmylwuLWQOf3hl0jjMMIw==} - engines: {node: '>=8'} - body-parser@2.2.2: resolution: {integrity: sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA==} engines: {node: '>=18'} @@ -1087,10 +1015,6 @@ packages: resolution: {integrity: sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==} engines: {node: 18 || 20 || >=22} - braces@3.0.3: - resolution: {integrity: sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==} - engines: {node: '>=8'} - bytes@3.1.2: resolution: {integrity: sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==} engines: {node: '>= 0.8'} @@ -1126,9 +1050,9 @@ packages: chardet@0.7.0: resolution: {integrity: sha512-mT8iDcrh03qDGRRmoA2hmBJnxpllMR+0/0qlzjqZES6NdiWDcZkCNAk4rPFZ9Q85r27unkiNNg8ZOiwZXBHwcA==} - chokidar@3.6.0: - resolution: {integrity: sha512-7VT13fmjotKpGipCW9JEQAusEPE+Ei8nl6/g4FBAmIm0GOOLMua9NDDo/DWp0ZAxCr3cPq5ZpBqmPAQgDda2Pw==} - engines: {node: '>= 8.10.0'} + chokidar@4.0.3: + resolution: {integrity: sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==} + engines: {node: '>= 14.16.0'} cli-spinners@2.9.2: resolution: {integrity: sha512-ywqV+5MmyL4E7ybXgKys4DugZbX0FC6LnwrhjuykIjnK9k8OQacQ7axGKnjDXWNhns0xot3bZI5h55H8yo9cJg==} @@ -1236,10 +1160,6 @@ packages: detect-node@2.1.0: resolution: {integrity: sha512-T0NIuQpnTvFDATNuHN5roPwSBG83rFsuO+MXXH9/3N1eFbn4wcPjttvjMLEPWJ0RGUYgQE7cGgS3tNxbqCGM7g==} - dir-glob@3.0.1: - resolution: {integrity: sha512-WkrWp9GR4KXfKGYzOLmTuGVi1UWFfws377n9cc55/tb6DuqyF6pcQ5AbiHEshaDpY9v6oaSr2XCDidGmMwdzIA==} - engines: {node: '>=8'} - doctrine@2.1.0: resolution: {integrity: sha512-35mSku4ZXK0vfCuHEDAwt55dg2jNajHZ1odvF+8SSr82EsZY4QmXfuWso8oEd8zRhVObSN18aM0CjSdoBX7zIw==} engines: {node: '>=0.10.0'} @@ -1369,6 +1289,7 @@ packages: eslint@9.39.3: resolution: {integrity: sha512-VmQ+sifHUbI/IcSopBCF/HO3YiHQx/AVd3UVyYL6weuwW+HvON9VYn5l6Zl1WZzPWXPNZrSQpxwkkZ/VuvJZzg==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + deprecated: This version is no longer supported. Please see https://eslint.org/version-support for other options. hasBin: true peerDependencies: jiti: '*' @@ -1436,10 +1357,6 @@ packages: fast-deep-equal@3.1.3: resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==} - fast-glob@3.3.3: - resolution: {integrity: sha512-7MptL8U0cqcFdzIzwOTHoilX9x5BrNqye7Z/LuC7kCMRio1EMSyqRK3BEAUD7sXRq4iT4AzTVuZdhgQ2TCvYLg==} - engines: {node: '>=8.6.0'} - fast-json-stable-stringify@2.1.0: resolution: {integrity: sha512-lhd/wF+Lk98HZoTCtlVraHtfh5XYijIjalXck7saUtuanSDyLMxnHhSXEDJqHxD7msR8D0uCmqlkwjCV8xvwHw==} @@ -1449,9 +1366,6 @@ packages: fast-uri@3.1.0: resolution: {integrity: sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA==} - fastq@1.19.1: - resolution: {integrity: sha512-GwLTyxkCXjXbxqIhTsMI2Nui8huMPtnxg7krajPJAjnEG/iiOS7i+zCtWGZR9G0NBKbXKh6X9m9UIsYX/N6vvQ==} - fdir@6.5.0: resolution: {integrity: sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==} engines: {node: '>=12.0.0'} @@ -1469,10 +1383,6 @@ packages: resolution: {integrity: sha512-XXTUwCvisa5oacNGRP9SfNtYBNAMi+RPwBFmblZEF7N7swHYQS6/Zfk7SRwx4D5j3CH211YNRco1DEMNVfZCnQ==} engines: {node: '>=16.0.0'} - fill-range@7.1.1: - resolution: {integrity: sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==} - engines: {node: '>=8'} - finalhandler@2.1.1: resolution: {integrity: sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==} engines: {node: '>= 18.0.0'} @@ -1563,10 +1473,6 @@ packages: get-tsconfig@4.13.0: resolution: {integrity: sha512-1VKTZJCwBrvbd+Wn3AOgQP/2Av+TfTCOlE4AcRJE72W1ksZXbAx8PPBR9RzgTeSPzlPMHrbANMH3LbltH73wxQ==} - glob-parent@5.1.2: - resolution: {integrity: sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow==} - engines: {node: '>= 6'} - glob-parent@6.0.2: resolution: {integrity: sha512-XxwI8EOhVQgWp6iDL+3b0r86f4d6AX6zSU55HfB4ydCEuXLXc5FcYeOu+nnGftS4TEju/11rt4KJPTMgbfmv4A==} engines: {node: '>=10.13.0'} @@ -1592,10 +1498,6 @@ packages: resolution: {integrity: sha512-DpLKbNU4WylpxJykQujfCcwYWiV/Jhm50Goo0wrVILAv5jOr9d+H+UR3PhSCD2rCCEIg0uc+G+muBTwD54JhDQ==} engines: {node: '>= 0.4'} - globby@11.1.0: - resolution: {integrity: sha512-jhIXaOzy1sb8IyocaruWSn1TjmnBVs8Ayhcy83rmxNJ8q2uWKCAj3CnJY+KpGSXCueAPc0i05kVvVKtP1t9S3g==} - engines: {node: '>=10'} - gopd@1.2.0: resolution: {integrity: sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==} engines: {node: '>= 0.4'} @@ -1697,10 +1599,6 @@ packages: resolution: {integrity: sha512-n4ZT37wG78iz03xPRKJrHTdZbe3IicyucEtdRsV5yglwc3GyUfbAfpSeD0FJ41NbUNSt5wbhqfp1fS+BgnvDFQ==} engines: {node: '>= 0.4'} - is-binary-path@2.1.0: - resolution: {integrity: sha512-ZMERYes6pDydyuGidse7OsHxtbI7WVeUEozgR/g7rd0xUimYNlvZRE/K2MgZTjWy725IfelLeVcEM97mmtRGXw==} - engines: {node: '>=8'} - is-boolean-object@1.2.2: resolution: {integrity: sha512-wa56o2/ElJMYqjCjGkXri7it5FbebW5usLw/nPmCMs5DeZ7eziSYZhSmPRn0txqeW4LnAmQQU7FgqLpsEFKM4A==} engines: {node: '>= 0.4'} @@ -1753,10 +1651,6 @@ packages: resolution: {integrity: sha512-lZhclumE1G6VYD8VHe35wFaIif+CTy5SJIi5+3y4psDgWu4wPDoBhF8NxUOinEc7pHgiTsT6MaBb92rKhhD+Xw==} engines: {node: '>= 0.4'} - is-number@7.0.0: - resolution: {integrity: sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==} - engines: {node: '>=0.12.0'} - is-promise@4.0.0: resolution: {integrity: sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==} @@ -1880,14 +1774,6 @@ packages: resolution: {integrity: sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==} engines: {node: '>=18'} - merge2@1.4.1: - resolution: {integrity: sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==} - engines: {node: '>= 8'} - - micromatch@4.0.8: - resolution: {integrity: sha512-PXwfBhYu0hBCPw8Dn0E+WDYb7af3dSLVWKi3HGv84IdF4TyFoC0ysxFd0Goxw7nSv4T/PzEJQxsYsEiFCKo2BA==} - engines: {node: '>=8.6'} - mime-db@1.52.0: resolution: {integrity: sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==} engines: {node: '>= 0.6'} @@ -1948,10 +1834,6 @@ packages: encoding: optional: true - normalize-path@3.0.0: - resolution: {integrity: sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==} - engines: {node: '>=0.10.0'} - object-assign@4.1.1: resolution: {integrity: sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==} engines: {node: '>=0.10.0'} @@ -2060,20 +1942,12 @@ packages: path-to-regexp@8.4.0: resolution: {integrity: sha512-PuseHIvAnz3bjrM2rGJtSgo1zjgxapTLZ7x2pjhzWwlp4SJQgK3f3iZIQwkpEnBaKz6seKBADpM4B4ySkuYypg==} - path-type@4.0.0: - resolution: {integrity: sha512-gDKb8aZMDeD/tZWs9P6+q0J9Mwkdl6xMV8TjnGP3qJVJ06bdMgkbBlLU8IdfOsIsFz2BW1rNVT3XuNEl8zPAvw==} - engines: {node: '>=8'} - pathe@2.0.3: resolution: {integrity: sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==} picocolors@1.1.1: resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==} - picomatch@2.3.2: - resolution: {integrity: sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==} - engines: {node: '>=8.6'} - picomatch@4.0.4: resolution: {integrity: sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==} engines: {node: '>=12'} @@ -2118,9 +1992,6 @@ packages: resolution: {integrity: sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==} engines: {node: '>=0.6'} - queue-microtask@1.2.3: - resolution: {integrity: sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A==} - range-parser@1.2.1: resolution: {integrity: sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==} engines: {node: '>= 0.6'} @@ -2129,9 +2000,9 @@ packages: resolution: {integrity: sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==} engines: {node: '>= 0.10'} - readdirp@3.6.0: - resolution: {integrity: sha512-hOS089on8RduqdbhvQ5Z37A0ESjsqz6qnRcffsMU3495FuTdqSm+7bhJ29JvIOsBDEEnan5DPu9t3To9VRlMzA==} - engines: {node: '>=8.10.0'} + readdirp@4.1.2: + resolution: {integrity: sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==} + engines: {node: '>= 14.18.0'} reflect.getprototypeof@1.0.10: resolution: {integrity: sha512-00o4I+DVrefhv+nX0ulyi3biSHCPDe+yLv5o/p6d/UVlirijB8E16FtfwSAi4g3tcqrQ4lRAqQSoFEZJehYEcw==} @@ -2157,10 +2028,6 @@ packages: engines: {node: '>= 0.4'} hasBin: true - reusify@1.1.0: - resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==} - engines: {iojs: '>=1.0.0', node: '>=0.10.0'} - roarr@2.15.4: resolution: {integrity: sha512-CHhPh+UNHD2GTXNYhPWLnU8ONHdI+5DI+4EYIAOaiD63rHeYlZvyh8P+in5999TTSFgUYuKUAjzRI4mdh/p+2A==} engines: {node: '>=8.0'} @@ -2178,9 +2045,6 @@ packages: resolution: {integrity: sha512-540WwVDOMxA6dN6We19EcT9sc3hkXPw5mzRNGM3FkdN/vtE9NFvj5lFAPNwUDmJjXidm3v7TC1cTE7t17Ulm1Q==} engines: {node: '>=0.12.0'} - run-parallel@1.2.0: - resolution: {integrity: sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA==} - safe-array-concat@1.1.3: resolution: {integrity: sha512-AURm5f0jYEOydBj7VQlVvDrjeFgthDdEF5H1dP+6mNpoXOMo1quQqJ4wvJDyRZ9+pO3kGWoOdmV08cSv2aJV6Q==} engines: {node: '>=0.4'} @@ -2270,10 +2134,6 @@ packages: resolution: {integrity: sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==} engines: {node: '>=14'} - slash@3.0.0: - resolution: {integrity: sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q==} - engines: {node: '>=8'} - source-map-js@1.2.1: resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==} engines: {node: '>=0.10.0'} @@ -2362,10 +2222,6 @@ packages: resolution: {integrity: sha512-UdiSoX6ypifLmrfQ/XfiawN6hkjSBpCjhKxxZcWlUUmoXLaCKQU0bx4HF/tdDK2uzRuchf1txGvrWBzYREssoQ==} engines: {node: '>=14.14'} - to-regex-range@5.0.1: - resolution: {integrity: sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==} - engines: {node: '>=8.0'} - toidentifier@1.0.1: resolution: {integrity: sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==} engines: {node: '>=0.6'} @@ -2376,12 +2232,6 @@ packages: tree-sitter-wasms@0.1.13: resolution: {integrity: sha512-wT+cR6DwaIz80/vho3AvSF0N4txuNx/5bcRKoXouOfClpxh/qqrF4URNLQXbbt8MaAxeksZcZd1j8gcGjc+QxQ==} - ts-api-utils@1.4.3: - resolution: {integrity: sha512-i3eMG77UTMD0hZhgRS562pv83RC6ukSAC2GMNWc+9dieh/+jDM5u5YG+NHX6VNDRHQcHwmsTHctP9LhbC3WxVw==} - engines: {node: '>=16'} - peerDependencies: - typescript: '>=4.2.0' - ts-api-utils@2.3.0: resolution: {integrity: sha512-6eg3Y9SF7SsAvGzRHQvvc1skDAhwI4YQ32ui1scxD1Ccr0G5qIIbUBT3pFTKX8kmWIQClHobtUdNuaBgwdfdWg==} engines: {node: '>=18.12'} @@ -3036,18 +2886,6 @@ snapshots: transitivePeerDependencies: - supports-color - '@nodelib/fs.scandir@2.1.5': - dependencies: - '@nodelib/fs.stat': 2.0.5 - run-parallel: 1.2.0 - - '@nodelib/fs.stat@2.0.5': {} - - '@nodelib/fs.walk@1.2.8': - dependencies: - '@nodelib/fs.scandir': 2.1.5 - fastq: 1.19.1 - '@pkgjs/parseargs@0.11.0': optional: true @@ -3260,25 +3098,8 @@ snapshots: transitivePeerDependencies: - supports-color - '@typescript-eslint/types@7.18.0': {} - '@typescript-eslint/types@8.51.0': {} - '@typescript-eslint/typescript-estree@7.18.0(typescript@5.9.3)': - dependencies: - '@typescript-eslint/types': 7.18.0 - '@typescript-eslint/visitor-keys': 7.18.0 - debug: 4.4.3 - globby: 11.1.0 - is-glob: 4.0.3 - minimatch: 10.2.3 - semver: 7.7.3 - ts-api-utils: 1.4.3(typescript@5.9.3) - optionalDependencies: - typescript: 5.9.3 - transitivePeerDependencies: - - supports-color - '@typescript-eslint/typescript-estree@8.51.0(typescript@5.9.3)': dependencies: '@typescript-eslint/project-service': 8.51.0(typescript@5.9.3) @@ -3305,11 +3126,6 @@ snapshots: transitivePeerDependencies: - supports-color - '@typescript-eslint/visitor-keys@7.18.0': - dependencies: - '@typescript-eslint/types': 7.18.0 - eslint-visitor-keys: 3.4.3 - '@typescript-eslint/visitor-keys@8.51.0': dependencies: '@typescript-eslint/types': 8.51.0 @@ -3407,11 +3223,6 @@ snapshots: ansi-styles@6.2.3: {} - anymatch@3.1.3: - dependencies: - normalize-path: 3.0.0 - picomatch: 2.3.2 - apache-arrow@15.0.2: dependencies: '@swc/helpers': 0.5.17 @@ -3446,8 +3257,6 @@ snapshots: is-string: 1.1.1 math-intrinsics: 1.1.0 - array-union@2.1.0: {} - array.prototype.findlastindex@1.2.6: dependencies: call-bind: 1.0.8 @@ -3494,8 +3303,6 @@ snapshots: balanced-match@4.0.3: {} - binary-extensions@2.3.0: {} - body-parser@2.2.2: dependencies: bytes: 3.1.2 @@ -3516,10 +3323,6 @@ snapshots: dependencies: balanced-match: 4.0.3 - braces@3.0.3: - dependencies: - fill-range: 7.1.1 - bytes@3.1.2: {} call-bind-apply-helpers@1.0.2: @@ -3554,17 +3357,9 @@ snapshots: chardet@0.7.0: {} - chokidar@3.6.0: + chokidar@4.0.3: dependencies: - anymatch: 3.1.3 - braces: 3.0.3 - glob-parent: 5.1.2 - is-binary-path: 2.1.0 - is-glob: 4.0.3 - normalize-path: 3.0.0 - readdirp: 3.6.0 - optionalDependencies: - fsevents: 2.3.3 + readdirp: 4.1.2 cli-spinners@2.9.2: {} @@ -3661,10 +3456,6 @@ snapshots: detect-node@2.1.0: {} - dir-glob@3.0.1: - dependencies: - path-type: 4.0.0 - doctrine@2.1.0: dependencies: esutils: 2.0.3 @@ -3985,24 +3776,12 @@ snapshots: fast-deep-equal@3.1.3: {} - fast-glob@3.3.3: - dependencies: - '@nodelib/fs.stat': 2.0.5 - '@nodelib/fs.walk': 1.2.8 - glob-parent: 5.1.2 - merge2: 1.4.1 - micromatch: 4.0.8 - fast-json-stable-stringify@2.1.0: {} fast-levenshtein@2.0.6: {} fast-uri@3.1.0: {} - fastq@1.19.1: - dependencies: - reusify: 1.1.0 - fdir@6.5.0(picomatch@4.0.4): optionalDependencies: picomatch: 4.0.4 @@ -4015,10 +3794,6 @@ snapshots: dependencies: flat-cache: 4.0.1 - fill-range@7.1.1: - dependencies: - to-regex-range: 5.0.1 - finalhandler@2.1.1: dependencies: debug: 4.4.3 @@ -4126,10 +3901,6 @@ snapshots: dependencies: resolve-pkg-maps: 1.0.0 - glob-parent@5.1.2: - dependencies: - is-glob: 4.0.3 - glob-parent@6.0.2: dependencies: is-glob: 4.0.3 @@ -4161,15 +3932,6 @@ snapshots: define-properties: 1.2.1 gopd: 1.2.0 - globby@11.1.0: - dependencies: - array-union: 2.1.0 - dir-glob: 3.0.1 - fast-glob: 3.3.3 - ignore: 5.3.2 - merge2: 1.4.1 - slash: 3.0.0 - gopd@1.2.0: {} guid-typescript@1.0.9: {} @@ -4261,10 +4023,6 @@ snapshots: dependencies: has-bigints: 1.1.0 - is-binary-path@2.1.0: - dependencies: - binary-extensions: 2.3.0 - is-boolean-object@1.2.2: dependencies: call-bound: 1.0.4 @@ -4316,8 +4074,6 @@ snapshots: call-bound: 1.0.4 has-tostringtag: 1.0.2 - is-number@7.0.0: {} - is-promise@4.0.0: {} is-regex@1.2.1: @@ -4428,13 +4184,6 @@ snapshots: merge-descriptors@2.0.0: {} - merge2@1.4.1: {} - - micromatch@4.0.8: - dependencies: - braces: 3.0.3 - picomatch: 2.3.2 - mime-db@1.52.0: {} mime-db@1.54.0: {} @@ -4471,8 +4220,6 @@ snapshots: dependencies: whatwg-url: 5.0.0 - normalize-path@3.0.0: {} - object-assign@4.1.1: {} object-inspect@1.13.4: {} @@ -4595,14 +4342,10 @@ snapshots: path-to-regexp@8.4.0: {} - path-type@4.0.0: {} - pathe@2.0.3: {} picocolors@1.1.1: {} - picomatch@2.3.2: {} - picomatch@4.0.4: {} pkce-challenge@5.0.1: {} @@ -4647,8 +4390,6 @@ snapshots: dependencies: side-channel: 1.1.0 - queue-microtask@1.2.3: {} - range-parser@1.2.1: {} raw-body@3.0.2: @@ -4658,9 +4399,7 @@ snapshots: iconv-lite: 0.7.2 unpipe: 1.0.0 - readdirp@3.6.0: - dependencies: - picomatch: 2.3.2 + readdirp@4.1.2: {} reflect.getprototypeof@1.0.10: dependencies: @@ -4694,8 +4433,6 @@ snapshots: path-parse: 1.0.7 supports-preserve-symlinks-flag: 1.0.0 - reusify@1.1.0: {} - roarr@2.15.4: dependencies: boolean: 3.2.0 @@ -4748,10 +4485,6 @@ snapshots: run-async@3.0.0: {} - run-parallel@1.2.0: - dependencies: - queue-microtask: 1.2.3 - safe-array-concat@1.1.3: dependencies: call-bind: 1.0.8 @@ -4901,8 +4634,6 @@ snapshots: signal-exit@4.1.0: {} - slash@3.0.0: {} - source-map-js@1.2.1: {} sprintf-js@1.1.3: {} @@ -4989,20 +4720,12 @@ snapshots: tmp@0.2.4: {} - to-regex-range@5.0.1: - dependencies: - is-number: 7.0.0 - toidentifier@1.0.1: {} tr46@0.0.3: {} tree-sitter-wasms@0.1.13: {} - ts-api-utils@1.4.3(typescript@5.9.3): - dependencies: - typescript: 5.9.3 - ts-api-utils@2.3.0(typescript@5.9.3): dependencies: typescript: 5.9.3 diff --git a/src/analyzers/angular/index.ts b/src/analyzers/angular/index.ts index aa748ad..527d856 100644 --- a/src/analyzers/angular/index.ts +++ b/src/analyzers/angular/index.ts @@ -103,9 +103,20 @@ export class AngularAnalyzer implements FrameworkAnalyzer { return false; } - // For TypeScript files, check if it contains Angular decorators + // Eligibility comes from actual framework imports. Method/decorator spellings + // classify confirmed Angular code later; comments and lookalikes are not proof. if (ext === '.ts' && content) { - return Object.values(this.angularPatterns).some((pattern) => pattern.test(content)); + try { + const ast = parse(content); + return ast.body.some( + (node) => + node.type === 'ImportDeclaration' && + typeof node.source.value === 'string' && + node.source.value.startsWith('@angular/') + ); + } catch { + return false; + } } // Angular component templates and styles @@ -171,7 +182,9 @@ export class AngularAnalyzer implements FrameworkAnalyzer { if (s.type === 'ImportDefaultSpecifier') return 'default'; if (s.type === 'ImportNamespaceSpecifier') return '*'; const specifier = s as TSESTree.ImportSpecifier; - return specifier.imported.name || specifier.local.name; + return specifier.imported.type === 'Identifier' + ? specifier.imported.name + : specifier.imported.value; }), isDefault: node.specifiers.some( (s: TSESTree.ImportClause) => s.type === 'ImportDefaultSpecifier' diff --git a/src/analyzers/generic/index.ts b/src/analyzers/generic/index.ts index 6dcfcaa..78ed320 100644 --- a/src/analyzers/generic/index.ts +++ b/src/analyzers/generic/index.ts @@ -8,6 +8,7 @@ import path from 'path'; import type { TSESTree } from '@typescript-eslint/typescript-estree'; import { FrameworkAnalyzer, + AnalysisOptions, AnalysisResult, CodebaseMetadata, CodeChunk, @@ -99,7 +100,11 @@ export class GenericAnalyzer implements FrameworkAnalyzer { return this.supportedExtensions.includes(ext); } - async analyze(filePath: string, content: string): Promise { + async analyze( + filePath: string, + content: string, + options?: AnalysisOptions + ): Promise { const language = detectLanguage(filePath); const relativePath = path.relative(process.cwd(), filePath); @@ -173,7 +178,8 @@ export class GenericAnalyzer implements FrameworkAnalyzer { filePath, language, framework: 'generic', - componentType: 'module' + componentType: 'module', + includeScopePrefix: options?.includeScopePrefix }); // Enrich AST chunks with the correct relativePath for (const chunk of chunks) { @@ -362,7 +368,9 @@ export class GenericAnalyzer implements FrameworkAnalyzer { if (s.type === 'ImportDefaultSpecifier') return 'default'; if (s.type === 'ImportNamespaceSpecifier') return '*'; const specifier = s as TSESTree.ImportSpecifier; - return specifier.imported.name || specifier.local.name; + return specifier.imported.type === 'Identifier' + ? specifier.imported.name + : specifier.imported.value; }), isDefault: node.specifiers.some( (s: TSESTree.ImportClause) => s.type === 'ImportDefaultSpecifier' @@ -440,7 +448,8 @@ export class GenericAnalyzer implements FrameworkAnalyzer { for (const spec of node.specifiers) { if (spec.type === 'ExportSpecifier') { exports.push({ - name: spec.exported.name, + name: + spec.exported.type === 'Identifier' ? spec.exported.name : spec.exported.value, isDefault: false, type: 'named' }); diff --git a/src/analyzers/react/index.ts b/src/analyzers/react/index.ts index 5838c09..193658f 100644 --- a/src/analyzers/react/index.ts +++ b/src/analyzers/react/index.ts @@ -87,12 +87,21 @@ export class ReactAnalyzer implements FrameworkAnalyzer { return false; } - return ( + const hasReactMarker = /\bfrom\s+['"]react['"]/.test(content) || /\brequire\(\s*['"]react['"]\s*\)/.test(content) || - /\bReact\.createElement\b/.test(content) || - /<[A-Za-z][^>]*>/.test(content) - ); + /\bReact\.createElement\b/.test(content); + if (hasReactMarker) { + return true; + } + + // TypeScript JSX requires .tsx. Plain TypeScript angle brackets also + // start generics and type assertions, so they cannot identify React. + if (extension === '.ts' || extension === '.mts' || extension === '.cts') { + return false; + } + + return /<[A-Za-z][^>]*>/.test(content); } async analyze(filePath: string, content: string): Promise { diff --git a/src/cli-formatters.ts b/src/cli-formatters.ts index 3a8049d..c89295f 100644 --- a/src/cli-formatters.ts +++ b/src/cli-formatters.ts @@ -480,7 +480,25 @@ export function formatRefs(data: RefsResponse, rootPath: string): void { if (data.isComplete === false) { lines.push(''); - lines.push(`${g.arrow} results capped at limit — use --limit to see more`); + const coverage = 'coverage' in data ? data.coverage : undefined; + const filesUnavailable = + coverage !== null && + typeof coverage === 'object' && + 'filesUnavailable' in coverage && + typeof coverage.filesUnavailable === 'number' + ? coverage.filesUnavailable + : 0; + if (filesUnavailable > 0) { + const noun = filesUnavailable === 1 ? 'source' : 'sources'; + lines.push(`${g.arrow} incomplete coverage: ${filesUnavailable} indexed ${noun} unavailable`); + } else { + lines.push(`${g.arrow} results capped at limit — use --limit to see more`); + } + } + + if (confidence === 'lexical') { + lines.push(''); + lines.push('Raw-text fallback may match comments or strings.'); } lines.push(''); diff --git a/src/cli-init.ts b/src/cli-init.ts index e9efc76..3918705 100644 --- a/src/cli-init.ts +++ b/src/cli-init.ts @@ -1,6 +1,6 @@ /** * Interactive setup wizard for codebase-context. - * Handles `codebase-context init` — generates MCP config and instruction block + * Handles `codebase-context init` — detects clients and merges project MCP configs * for the four first-wave AI clients: Claude Code, Cursor, Codex, OpenCode. * * Depends only on Node built-ins and @inquirer/prompts. @@ -10,282 +10,574 @@ import path from 'path'; import * as fs from 'node:fs/promises'; -import { execFileSync } from 'child_process'; -import { select, confirm } from '@inquirer/prompts'; +import { checkbox } from '@inquirer/prompts'; +import { homedir } from 'node:os'; +import { constants } from 'node:fs'; +import { createRequire } from 'node:module'; -export type Client = 'claude-code' | 'cursor' | 'codex' | 'opencode'; +export type Client = (typeof CLIENTS)[number]['id']; -export type McpConfigResult = - | { kind: 'file'; path: string; content: string } - | { kind: 'command'; args: string[] }; +export type McpConfigResult = { kind: 'file'; path: string; content: string }; + +export type McpConfigOptions = { + rootPath?: string; + platform?: NodeJS.Platform; + packageVersion?: string; +}; type JsonObject = Record; -export function generateMcpConfig(client: Client): McpConfigResult { - switch (client) { - case 'claude-code': - return { - kind: 'command', - args: ['mcp', 'add', '--transport', 'http', 'codebase-context', 'http://127.0.0.1:3100/mcp'] - }; - case 'cursor': - return { - kind: 'file', - path: '.cursor/mcp.json', - content: JSON.stringify( - { - mcpServers: { - 'codebase-context': { - type: 'http', - url: 'http://127.0.0.1:3100/mcp' - } - } - }, - null, - 2 +const { version: packageVersion } = createRequire(import.meta.url)('../package.json') as { + version: string; +}; + +export const CLIENTS = [ + { id: 'claude-code', name: 'Claude Code', executable: 'claude', directory: '.claude' }, + { id: 'codex', name: 'Codex', executable: 'codex', directory: '.codex' }, + { id: 'cursor', name: 'Cursor', executable: 'cursor', directory: '.cursor' }, + { id: 'opencode', name: 'OpenCode', executable: 'opencode', directory: '.opencode' } +] as const; + +export async function detectClients( + rootPath: string, + env: NodeJS.ProcessEnv = process.env, + platform: NodeJS.Platform = process.platform +): Promise { + const paths = platform === 'win32' ? path.win32 : path.posix; + const home = env.HOME || env.USERPROFILE || homedir(); + const configHome = env.XDG_CONFIG_HOME || paths.join(home, '.config'); + const searchPath = (env.PATH || env.Path || '').split(platform === 'win32' ? ';' : ':'); + const extensions = platform === 'win32' ? ['', '.exe', '.cmd', '.bat', '.ps1'] : ['']; + const detected: Client[] = []; + for (const client of CLIENTS) { + const userDirectory = + client.id === 'codex' + ? env.CODEX_HOME || paths.join(home, '.codex') + : client.id === 'claude-code' + ? env.CLAUDE_CONFIG_DIR || paths.join(home, '.claude') + : client.id === 'opencode' + ? paths.join(configHome, 'opencode') + : paths.join(home, client.directory); + const directories = [paths.join(rootPath, client.directory), userDirectory]; + const executables = searchPath + .filter(Boolean) + .flatMap((directory) => + extensions.map((extension) => paths.join(directory, client.executable + extension)) + ); + const candidates = [ + ...directories.map((candidate) => ({ path: candidate, directory: true })), + ...executables.map((candidate) => ({ path: candidate, directory: false })) + ]; + for (const candidate of candidates) { + try { + await fs.access( + candidate.path, + candidate.directory || platform === 'win32' ? constants.F_OK : constants.X_OK + ); + const info = await fs.stat(candidate.path); + if (candidate.directory ? !info.isDirectory() : !info.isFile()) continue; + detected.push(client.id); + break; + } catch (error) { + if ( + !isFileNotFound(error) && + !( + typeof error === 'object' && + error !== null && + 'code' in error && + (error.code === 'EACCES' || error.code === 'ENOTDIR') + ) ) - }; - case 'codex': - return { - kind: 'command', - args: ['mcp', 'add', 'codebase-context', 'http://127.0.0.1:3100/mcp'] - }; - case 'opencode': - return { - kind: 'file', - path: 'opencode.json', - content: JSON.stringify( - { + throw error; + } + } + } + return detected; +} + +export function generateMcpConfig(client: Client, options: McpConfigOptions = {}): McpConfigResult { + const rootPath = path.resolve(options.rootPath ?? process.cwd()); + const version = options.packageVersion ?? packageVersion; + const windows = (options.platform ?? process.platform) === 'win32'; + const command = windows ? 'npx.cmd' : 'npx'; + const args = ['--yes', `codebase-context@${version}`, rootPath]; + if (client === 'codex') { + return { + kind: 'file', + path: path.join(rootPath, '.codex', 'config.toml'), + content: + `[mcp_servers.codebase-context]\ncommand = ${JSON.stringify(command)}\n` + + `args = [${args.map((value) => JSON.stringify(value)).join(', ')}]\nstartup_timeout_sec = 300\n` + }; + } + const configPath = + client === 'claude-code' + ? '.mcp.json' + : client === 'cursor' + ? '.cursor/mcp.json' + : 'opencode.json'; + return { + kind: 'file', + path: path.join(rootPath, configPath), + content: JSON.stringify( + client === 'opencode' + ? { $schema: 'https://opencode.ai/config.json', - mcp: { + mcp: { 'codebase-context': { type: 'local', command: [command, ...args] } } + } + : { + mcpServers: { 'codebase-context': { - type: 'remote', - url: 'http://127.0.0.1:3100/mcp' + ...(client === 'cursor' ? { type: 'stdio' } : {}), + command, + args } } }, - null, - 2 - ) - }; - } + null, + 2 + ) + }; } -export function generateInstructionBlock(): string { - return ` -## Codebase Context (MCP) - -**Start of every task:** Call \`get_memory\` to load team conventions before writing any code. +export function resolveInitRoot( + args: string[], + envRoot: string | undefined, + cwd = process.cwd() +): string { + let explicitRoot: string | undefined; + + for (let index = 0; index < args.length; index++) { + const arg = args[index]; + if (arg === '--root') { + const value = args[index + 1]; + if (!value || value.startsWith('--')) { + throw new Error('Expected a directory path after --root.'); + } + explicitRoot = value; + index++; + } else if (arg?.startsWith('--root=')) { + const value = arg.slice('--root='.length); + if (!value) { + throw new Error('Expected a directory path after --root.'); + } + explicitRoot = value; + } + } -**Before editing existing code:** Call \`search_codebase\` with \`intent: "edit"\`. If the preflight card says \`ready: false\`, read the listed files before touching anything. + return path.resolve(explicitRoot ?? (envRoot ? envRoot : cwd)); +} -**Before writing new code:** Call \`get_team_patterns\` to check how the team handles DI, state, testing, and library wrappers — don't introduce a new pattern if one already exists. +function isJsonObject(value: unknown): value is JsonObject { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} -**When asked to "remember" or "record" something:** Call \`remember\` immediately, before doing anything else. +function normalizeJson(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(normalizeJson); + } + if (isJsonObject(value)) { + return Object.fromEntries( + Object.entries(value) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([key, entry]) => [key, normalizeJson(entry)]) + ); + } + return value; +} -**When adding imports that cross module boundaries:** Call \`detect_circular_dependencies\` with the relevant scope after adding the import. - -`; +function jsonValuesEqual(left: unknown, right: unknown): boolean { + return JSON.stringify(normalizeJson(left)) === JSON.stringify(normalizeJson(right)); } -export function resolveInstructionFilePath(client: Client, cwd: string): string | null { - switch (client) { - case 'claude-code': - return path.join(cwd, 'CLAUDE.md'); - case 'cursor': - return path.join(cwd, '.cursorrules'); - case 'codex': - return path.join(cwd, 'AGENTS.md'); - case 'opencode': - return null; - } +function isFileNotFound(error: unknown): boolean { + return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT'; } -/** - * Appends the instruction block to an existing file, or skips if already present. - * Exported with leading underscore to signal internal/test-only use. - */ -export async function _appendInstructionBlock(filePath: string): Promise<'written' | 'skipped'> { - let existing = ''; +async function readExistingConfig(filePath: string): Promise { try { - existing = await fs.readFile(filePath, 'utf8'); - } catch { - // file does not exist — write fresh - await fs.writeFile(filePath, generateInstructionBlock(), 'utf8'); - return 'written'; + return await fs.readFile(filePath, 'utf8'); + } catch (error) { + if (isFileNotFound(error)) return null; + throw error; } +} - if (existing.includes('')) { - return 'skipped'; - } +function isAbsolutePath(value: string): boolean { + return path.posix.isAbsolute(value) || path.win32.isAbsolute(value); +} - await fs.writeFile(filePath, existing + '\n' + generateInstructionBlock(), 'utf8'); - return 'written'; +/** Recognize only CBC-specific launch shapes written by init. */ +function isGeneratedLaunch(command: unknown, args: unknown): boolean { + if ( + typeof command !== 'string' || + !Array.isArray(args) || + !args.every((arg) => typeof arg === 'string') + ) + return false; + const executable = path.basename(command.replace(/\\/g, '/')).toLowerCase(); + if (executable === 'node' || executable === 'node.exe') { + return ( + args.length === 2 && + /[\\/]node_modules[\\/]codebase-context[\\/]dist[\\/]index\.js$/.test(args[0]) && + isAbsolutePath(args[0]) && + isAbsolutePath(args[1]) + ); + } + const launch = command === 'npx' || command === 'npx.cmd' ? args : []; + return ( + launch.length === 3 && + (launch[0] === '--yes' || launch[0] === '-y') && + /^codebase-context@\d+\.\d+\.\d+(?:-[\w.-]+)?$/.test(launch[1]) && + isAbsolutePath(launch[2]) + ); } -function isJsonObject(value: unknown): value is JsonObject { - return typeof value === 'object' && value !== null && !Array.isArray(value); +function mergeGeneratedServer(existing: unknown, generated: unknown, client: Client): unknown { + if (jsonValuesEqual(existing, generated)) return existing; + if (!isJsonObject(existing) || !isJsonObject(generated) || 'url' in existing) return null; + const local = client === 'opencode'; + const command = local && Array.isArray(existing.command) ? existing.command[0] : existing.command; + const args = local && Array.isArray(existing.command) ? existing.command.slice(1) : existing.args; + if ( + (existing.type !== undefined && existing.type !== (local ? 'local' : 'stdio')) || + !isGeneratedLaunch(command, args) + ) + return null; + return { + ...existing, + command: generated.command, + ...(!local ? { args: generated.args } : {}), + ...(generated.type ? { type: generated.type } : {}) + }; } /** * Merge generated MCP config into an existing JSON config file when possible. - * Preserves unrelated keys and only updates the codebase-context server entry. + * Preserves unrelated keys and updates only recognized generated launches. */ export async function _buildMergedMcpContent( filePath: string, generatedContent: string, - client: Extract + client: Extract ): Promise<{ content: string; mergedFromExisting: boolean }> { + const existingText = await readExistingConfig(filePath); + if (existingText === null) { + return { content: generatedContent, mergedFromExisting: false }; + } + let existing: unknown; try { - existing = JSON.parse(await fs.readFile(filePath, 'utf8')); - } catch { - return { content: generatedContent, mergedFromExisting: false }; + existing = JSON.parse(existingText) as unknown; + } catch (error) { + throw new Error(`Cannot merge ${client} config at ${filePath}: existing JSON is invalid.`, { + cause: error + }); } const generated = JSON.parse(generatedContent) as unknown; if (!isJsonObject(existing) || !isJsonObject(generated)) { - return { content: generatedContent, mergedFromExisting: false }; + throw new Error(`Cannot merge ${client} config at ${filePath}: expected a JSON object.`); } - if (client === 'cursor') { - const existingServers = isJsonObject(existing.mcpServers) ? existing.mcpServers : {}; - const generatedServers = isJsonObject(generated.mcpServers) ? generated.mcpServers : {}; - return { - content: JSON.stringify( - { - ...existing, - ...generated, - mcpServers: { - ...existingServers, - ...generatedServers - } - }, - null, - 2 - ), - mergedFromExisting: true - }; + const serverKey = client === 'opencode' ? 'mcp' : 'mcpServers'; + const existingServersValue = existing[serverKey]; + if (existingServersValue !== undefined && !isJsonObject(existingServersValue)) { + throw new Error( + `Cannot merge ${client} config at ${filePath}: ${serverKey} must be a JSON object.` + ); } - - const existingMcp = isJsonObject(existing.mcp) ? existing.mcp : {}; - const generatedMcp = isJsonObject(generated.mcp) ? generated.mcp : {}; + const generatedServersValue = generated[serverKey]; + if (!isJsonObject(generatedServersValue)) { + throw new Error(`Invalid generated ${client} MCP config.`); + } + const existingServers = isJsonObject(existingServersValue) ? existingServersValue : {}; + const hasExistingCodebaseContext = Object.prototype.hasOwnProperty.call( + existingServers, + 'codebase-context' + ); + const generatedCodebaseContext = generatedServersValue['codebase-context']; + const mergedServer = hasExistingCodebaseContext + ? mergeGeneratedServer(existingServers['codebase-context'], generatedCodebaseContext, client) + : generatedCodebaseContext; + if (mergedServer === null) { + throw new Error( + `Cannot merge ${client} config at ${filePath}: it already contains a different codebase-context configuration.` + ); + } + const merged = { + ...generated, + ...existing, + [serverKey]: { ...existingServers, 'codebase-context': mergedServer } + }; return { - content: JSON.stringify( - { - ...existing, - ...generated, - mcp: { - ...existingMcp, - ...generatedMcp - } - }, - null, - 2 - ), + content: jsonValuesEqual(existing, merged) ? existingText : JSON.stringify(merged, null, 2), mergedFromExisting: true }; } -export async function handleInitCli(_argv: string[]): Promise { - console.log('\nSet up codebase-context for your AI client\n'); - - const client = await select({ - message: 'Which client?', - choices: [ - { name: 'Claude Code', value: 'claude-code' }, - { name: 'Cursor', value: 'cursor' }, - { name: 'Codex', value: 'codex' }, - { name: 'OpenCode', value: 'opencode' } - ] - }); - - const mcpResult = generateMcpConfig(client); - - console.log('\n--- MCP Config Preview ---'); - if (mcpResult.kind === 'file') { - try { - await fs.access(mcpResult.path); - console.log(`Warning: ${mcpResult.path} already exists; existing entries will be preserved.`); - } catch { - // file does not exist - } - console.log(`File: ${mcpResult.path}\n${mcpResult.content}`); - } else { - console.log(`Command to run: ${mcpResult.args[0]} ${mcpResult.args.slice(1).join(' ')}`); - } - - const applyMcp = await confirm({ - message: 'Apply MCP config? [y/N]', - default: false - }); - - const instructionPath = resolveInstructionFilePath(client, process.cwd()); - - let applyInstruction = false; - if (instructionPath !== null) { - let fileExists = false; - try { - await fs.access(instructionPath); - fileExists = true; - } catch { - // does not exist - } +function normalizeTomlNewlines(value: string): string { + return value.replace(/\r\n/g, '\n'); +} - if (fileExists) { - console.log(`\n--- Instruction Block Preview (will append to ${instructionPath}) ---`); - } else { - console.log(`\n--- Instruction Block Preview (will create ${instructionPath}) ---`); - } - console.log(generateInstructionBlock()); +function tomlKeyPattern(key: string): string { + return `(?:${key}|"${key}"|'${key}')`; +} - applyInstruction = await confirm({ - message: 'Write instruction block? [y/N]', - default: false - }); +/** + * Merges the generated Codex launch, preserving unrelated settings and custom entries. + */ +export async function _buildMergedCodexContent( + filePath: string, + generatedContent: string +): Promise<{ content: string; mergedFromExisting: boolean }> { + const existing = await readExistingConfig(filePath); + if (existing === null) { + return { content: generatedContent, mergedFromExisting: false }; } - // Execute confirmed actions - - if (applyMcp) { - if (mcpResult.kind === 'file') { - const dir = path.dirname(mcpResult.path); - if (dir && dir !== '.') { - await fs.mkdir(dir, { recursive: true }); - } - const mergedConfig = - client === 'cursor' || client === 'opencode' - ? await _buildMergedMcpContent(mcpResult.path, mcpResult.content, client) - : { content: mcpResult.content, mergedFromExisting: false }; - await fs.writeFile(mcpResult.path, mergedConfig.content, 'utf8'); - if (mergedConfig.mergedFromExisting) { - console.log(`Merged: ${mcpResult.path} (existing entries preserved)`); - } - console.log(`Written: ${mcpResult.path}`); - } else { - const [cmd, ...rest] = mcpResult.args; + const normalizedExisting = normalizeTomlNewlines(existing); + const normalizedGenerated = normalizeTomlNewlines(generatedContent).trim(); + const mcpServersKey = tomlKeyPattern('mcp_servers'); + const codebaseContextKey = tomlKeyPattern('codebase-context'); + const entryHeader = new RegExp( + `^\\s*\\[{1,2}\\s*${mcpServersKey}\\s*\\.\\s*${codebaseContextKey}(?:\\s*\\.\\s*[^\\]]+)?\\s*\\]{1,2}\\s*(?:#.*)?$`, + 'gm' + ); + const existingEntry = entryHeader.exec(normalizedExisting); + + if (existingEntry) { + const nextTable = /^\s*\[{1,2}[^\r\n]+\]{1,2}\s*(?:#.*)?$/gm; + nextTable.lastIndex = entryHeader.lastIndex; + const nextTableMatch = nextTable.exec(normalizedExisting); + const sectionEnd = nextTableMatch?.index ?? normalizedExisting.length; + const section = normalizedExisting.slice(existingEntry.index, sectionEnd).trim(); + const exactEntryHeader = new RegExp( + `^\\[\\s*${mcpServersKey}\\s*\\.\\s*${codebaseContextKey}\\s*\\]\\s*(?:#.*)?$`, + 'm' + ); + const comparableSection = section.replace(exactEntryHeader, '[mcp_servers.codebase-context]'); + if (comparableSection === normalizedGenerated) { + return { content: existing, mergedFromExisting: true }; + } + const commandLine = /^([ \t]*command[ \t]*=[ \t]*)("(?:[^"\\]|\\.)*")([ \t]*(?:#.*)?)$/m; + const argsLine = /^([ \t]*args[ \t]*=[ \t]*)(\[[^\r\n]*\])([ \t]*(?:#.*)?)$/m; + const oldCommand = commandLine.exec(section); + const oldArgs = argsLine.exec(section); + const newCommand = commandLine.exec(normalizedGenerated); + const newArgs = argsLine.exec(normalizedGenerated); + if ( + exactEntryHeader.test(section) && + oldCommand && + oldArgs && + newCommand && + newArgs && + !/^[ \t]*url[ \t]*=/m.test(section) && + section.match(/^[ \t]*command[ \t]*=/gm)?.length === 1 && + section.match(/^[ \t]*args[ \t]*=/gm)?.length === 1 + ) { try { - execFileSync(cmd, rest, { stdio: 'inherit' }); + if (isGeneratedLaunch(JSON.parse(oldCommand[2]), JSON.parse(oldArgs[2]))) { + if ( + jsonValuesEqual(JSON.parse(oldCommand[2]), JSON.parse(newCommand[2])) && + jsonValuesEqual(JSON.parse(oldArgs[2]), JSON.parse(newArgs[2])) + ) { + return { content: existing, mergedFromExisting: true }; + } + const updated = normalizedExisting + .slice(existingEntry.index, sectionEnd) + .replace( + commandLine, + (_line, prefix: string, _value: string, comment: string) => + prefix + newCommand[2] + comment + ) + .replace( + argsLine, + (_line, prefix: string, _value: string, comment: string) => + prefix + newArgs[2] + comment + ); + const content = + normalizedExisting.slice(0, existingEntry.index) + + updated + + normalizedExisting.slice(sectionEnd); + return { + content: existing.includes('\r\n') ? content.replace(/\n/g, '\r\n') : content, + mergedFromExisting: true + }; + } } catch { - console.log( - `Could not run '${cmd}' automatically. Run it yourself:\n ${cmd} ${rest.join(' ')}` - ); + // Non-generated TOML values remain untouched and are reported as conflicts. } } + throw new Error( + `Cannot merge Codex config at ${filePath}: it already contains a different codebase-context configuration.` + ); } - if (applyInstruction && instructionPath !== null) { - const status = await _appendInstructionBlock(instructionPath); - if (status === 'skipped') { - console.log(`Instruction block already present in ${instructionPath}, skipping.`); - } else { - console.log(`Written: ${instructionPath}`); + const dottedEntry = new RegExp( + `^\\s*${mcpServersKey}\\s*\\.\\s*${codebaseContextKey}\\s*(?:=|\\.)`, + 'm' + ); + if (dottedEntry.test(normalizedExisting)) { + throw new Error( + `Cannot merge Codex config at ${filePath}: it already contains a different codebase-context configuration.` + ); + } + + const parentTableHeader = new RegExp(`^\\s*\\[\\s*${mcpServersKey}\\s*\\]\\s*(?:#.*)?$`, 'gm'); + let parentTable = parentTableHeader.exec(normalizedExisting); + while (parentTable) { + const nextTable = /^\s*\[{1,2}[^\r\n]+\]{1,2}\s*(?:#.*)?$/gm; + nextTable.lastIndex = parentTableHeader.lastIndex; + const nextTableMatch = nextTable.exec(normalizedExisting); + const sectionEnd = nextTableMatch?.index ?? normalizedExisting.length; + const parentSettings = normalizedExisting.slice(parentTableHeader.lastIndex, sectionEnd); + const childEntry = new RegExp(`^\\s*${codebaseContextKey}\\s*(?:=|\\.)`, 'm'); + if (childEntry.test(parentSettings)) { + throw new Error( + `Cannot merge Codex config at ${filePath}: it already contains a different codebase-context configuration.` + ); } + parentTable = parentTableHeader.exec(normalizedExisting); + } + + const firstTable = /^\s*\[/m.exec(normalizedExisting); + const rootSettings = firstTable + ? normalizedExisting.slice(0, firstTable.index) + : normalizedExisting; + const inlineServers = new RegExp(`^\\s*${mcpServersKey}\\s*=`, 'm'); + if (inlineServers.test(rootSettings)) { + throw new Error( + `Cannot merge Codex config at ${filePath}: mcp_servers is already configured inline.` + ); } + const prefix = normalizedExisting.trimEnd(); + return { + content: `${prefix ? `${prefix}\n\n` : ''}${normalizedGenerated}\n`, + mergedFromExisting: true + }; +} + +type InitOptions = { + clients: Client[]; + rootPath: string; + yes: boolean; + help: boolean; +}; + +function parseClient(value: string): Client { + const client = CLIENTS.find((client) => client.id === value); + if (client) return client.id; + throw new Error( + `Unknown client "${value}". Choose ${CLIENTS.map((client) => client.id).join(', ')}.` + ); +} + +function parseInitOptions(args: string[]): InitOptions { + if (args.includes('--help') || args.includes('-h')) + return { clients: [], rootPath: process.cwd(), yes: false, help: true }; + const clients: Client[] = []; + let yes = false; + for (let index = 0; index < args.length; index++) { + const arg = args[index]; + if (arg === '--yes' || arg === '-y') yes = true; + else if (arg === '--client') { + const value = args[++index]; + if (!value || value.startsWith('--')) throw new Error('Expected a client after --client.'); + clients.push(parseClient(value)); + } else if (arg?.startsWith('--client=')) + clients.push(parseClient(arg.slice('--client='.length))); + else if (arg === '--root') index++; + else if (!arg?.startsWith('--root=')) throw new Error(`Unknown init option "${arg}".`); + } + return { + clients: [...new Set(clients)], + rootPath: resolveInitRoot(args, process.env.CODEBASE_ROOT), + yes, + help: false + }; +} + +function printInitUsage(): void { + console.log('codebase-context init [options]'); + console.log( + 'Detect and configure coding tools together. With a terminal, Space toggles and Enter applies.' + ); console.log( - '\nNext steps:\n' + - ' Run `npx codebase-context map` to see your codebase conventions\n' + - ' Start the HTTP server: npx codebase-context --http\n' + ' --client Choose a tool (repeatable): ' + + CLIENTS.map((client) => client.id).join(', ') ); + console.log(' --root Project root (default: CODEBASE_ROOT or cwd)'); + console.log(' --yes, -y Configure every detected tool without prompts'); + console.log(' --help, -h Show this help'); +} + +export async function handleInitCli(args: string[]): Promise { + try { + const options = parseInitOptions(args); + if (options.help) { + printInitUsage(); + return; + } + if (!(await fs.stat(options.rootPath)).isDirectory()) + throw new Error(`Project root is not a directory: ${options.rootPath}`); + console.log(`Set up Codebase Context in ${options.rootPath}`); + let clients = options.clients; + if (!clients.length) { + const detected = await detectClients(options.rootPath); + clients = + !options.yes && process.stdin.isTTY && process.stdout.isTTY + ? await checkbox({ + message: 'Choose tools (Space toggles, Enter sets up):', + choices: CLIENTS.map((client) => ({ + name: client.name + (detected.includes(client.id) ? ' (detected)' : ''), + value: client.id, + checked: detected.includes(client.id) + })) + }) + : detected; + } + if (!clients.length) + throw new Error( + 'No coding tools selected or detected. Choose with --client claude-code, --client codex, --client cursor, or --client opencode.' + ); + + const prepared = []; + for (const client of clients) { + const config = generateMcpConfig(client, { rootPath: options.rootPath }); + const merged = + client === 'codex' + ? await _buildMergedCodexContent(config.path, config.content) + : await _buildMergedMcpContent(config.path, config.content, client); + prepared.push({ + ...config, + content: merged.content, + unchanged: (await readExistingConfig(config.path)) === merged.content + }); + } + for (const config of prepared) { + if (!config.unchanged) { + await fs.mkdir(path.dirname(config.path), { recursive: true }); + await fs.writeFile(config.path, config.content, 'utf8'); + } + console.log( + `${config.unchanged ? 'Unchanged' : 'Configured'}: ${path.relative(options.rootPath, config.path)}` + ); + } + console.log('Reopen your tools; approve the server or project trust if asked.'); + console.log("The first search builds this repository's index."); + } catch (error) { + if ( + error instanceof Error && + (error.name === 'ExitPromptError' || + error.name === 'AbortPromptError' || + error.name === 'CancelPromptError') + ) { + console.log('Setup cancelled; no files written.'); + return; + } + console.error(`Init failed: ${error instanceof Error ? error.message : String(error)}`); + process.exitCode = 1; + } } diff --git a/src/cli.ts b/src/cli.ts index c7a0ed4..17fb25a 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -82,12 +82,14 @@ function printUsage(): void { console.log(' [--limit ] [--lang ] [--framework ] [--layer ]'); console.log(' metadata Project structure, frameworks, deps'); console.log(' status Index state and progress'); - console.log(' reindex [--incremental] [--reason ] Re-index the codebase'); + console.log(' reindex [--incremental] [--embeddings|--no-embeddings] Re-index the codebase'); console.log(' style-guide [--query ] [--category ] Style guide rules'); console.log(' patterns [--category all|di|state|testing|libraries] Team patterns'); console.log(' refs --symbol [--limit ] Symbol references'); console.log(' cycles [--scope ] Circular dependency detection'); - console.log(' init Interactive setup wizard for AI clients'); + console.log( + ' init [--client ] [--yes] Set up a native client (init --help for options)' + ); console.log(' map [--json] [--pretty] Conventions map'); console.log(''); console.log('Global flags:'); @@ -129,7 +131,11 @@ async function initToolContext(): Promise { status: indexExists ? 'ready' : 'idle' }; - const performIndexing = async (incrementalOnly?: boolean, reason?: string): Promise => { + const performIndexing = async ( + incrementalOnly?: boolean, + reason?: string, + skipEmbedding?: boolean + ): Promise => { indexState.status = 'indexing'; const mode = incrementalOnly ? 'incremental' : 'full'; console.error(`Indexing (${mode})${reason ? ` — ${reason}` : ''}: ${rootPath}`); @@ -138,7 +144,12 @@ async function initToolContext(): Promise { let lastLoggedProgress = { phase: '', percentage: -1 }; const indexer = new CodebaseIndexer({ rootPath, - ...(projectConfig?.parsing ? { config: { parsing: projectConfig.parsing } } : {}), + config: { + ...(projectConfig?.parsing ? { parsing: projectConfig.parsing } : {}), + ...((skipEmbedding ?? projectConfig?.skipEmbedding) !== undefined + ? { skipEmbedding: skipEmbedding ?? projectConfig?.skipEmbedding } + : {}) + }, incrementalOnly, onProgress: (progress) => { const shouldLog = @@ -167,6 +178,7 @@ async function initToolContext(): Promise { indexState.status = 'error'; indexState.error = error instanceof Error ? error.message : String(error); console.error('Indexing failed:', indexState.error); + throw error; } }; @@ -282,6 +294,11 @@ export async function handleCliCommand(argv: string[]): Promise { return handleMapCli(argv.slice(1)); } + if (argv.slice(1).includes('--help')) { + printUsage(); + return; + } + const useJson = argv.includes('--json'); const flags = parseFlags(argv); @@ -359,10 +376,23 @@ export async function handleCliCommand(argv: string[]): Promise { break; } case 'reindex': { - const usage = 'codebase-context reindex [--incremental] [--reason ]'; + const usage = + 'codebase-context reindex [--incremental] [--reason ] [--embeddings|--no-embeddings]'; const reason = optionalStringFlag(flags, 'reason', usage); const incremental = booleanFlag(flags, 'incremental', usage); - await ctx.performIndexing(incremental, reason); + if ('embeddings' in flags && 'no-embeddings' in flags) { + exitWithError( + `Error: --embeddings and --no-embeddings are mutually exclusive\nUsage: ${usage}` + ); + } + for (const modeFlag of ['embeddings', 'no-embeddings']) { + if (modeFlag in flags && flags[modeFlag] !== true) { + exitWithError(`Error: --${modeFlag} takes no value\nUsage: ${usage}`); + } + } + const skipEmbedding = + 'no-embeddings' in flags ? true : 'embeddings' in flags ? false : undefined; + await ctx.performIndexing(incremental, reason, skipEmbedding); const statusResult = await dispatchTool('get_indexing_status', {}, ctx); formatJson(extractText(statusResult), useJson, 'status', ctx.rootPath); return; diff --git a/src/constants/codebase-context.ts b/src/constants/codebase-context.ts index 473748f..cd87795 100644 --- a/src/constants/codebase-context.ts +++ b/src/constants/codebase-context.ts @@ -11,7 +11,7 @@ export const CODEBASE_CONTEXT_DIRNAME = '.codebase-context' as const; * - Embedding input string changes (e.g. scope-prefix content, prepended metadata) * - Required persisted fields/artifact headers change */ -export const INDEX_FORMAT_VERSION = 1 as const; +export const INDEX_FORMAT_VERSION = 4 as const; /** Schema version for `.codebase-context/index-meta.json` itself. */ export const INDEX_META_VERSION = 1 as const; diff --git a/src/core/analyzer-registry.ts b/src/core/analyzer-registry.ts index 6f4df74..d857c2e 100644 --- a/src/core/analyzer-registry.ts +++ b/src/core/analyzer-registry.ts @@ -4,9 +4,9 @@ */ import path from 'path'; -import { FrameworkAnalyzer, AnalysisResult } from '../types/index.js'; +import { FrameworkAnalyzer, AnalysisResult, AnalysisOptions } from '../types/index.js'; -export interface AnalyzerSelectionOptions { +export interface AnalyzerSelectionOptions extends AnalysisOptions { preferredAnalyzer?: string; extraFileExtensions?: string[]; } @@ -118,7 +118,7 @@ export class AnalyzerRegistry { } try { - return await analyzer.analyze(filePath, content); + return await analyzer.analyze(filePath, content, options); } catch (error) { console.error(`Error analyzing ${filePath} with ${analyzer.name}:`, error); return null; diff --git a/src/core/file-watcher.ts b/src/core/file-watcher.ts index cf5542e..4ab4017 100644 --- a/src/core/file-watcher.ts +++ b/src/core/file-watcher.ts @@ -1,6 +1,6 @@ import chokidar from 'chokidar'; import path from 'path'; -import { EXCLUDED_GLOB_PATTERNS } from '../constants/codebase-context.js'; +import { EXCLUDED_DIRECTORY_NAMES } from '../constants/codebase-context.js'; import { getSupportedExtensions } from '../utils/language-detection.js'; export interface FileWatcherOptions { @@ -27,16 +27,19 @@ function isTrackedSourcePath(filePath: string, trackedExtensions: Set): /** * Watch rootPath for source file changes and call onChanged (debounced). - * Returns a stop() function that cancels the debounce timer and closes the watcher. + * Returns a stop() function that cancels pending work and resolves when the watcher closes. */ -export function startFileWatcher(opts: FileWatcherOptions): () => void { +export function startFileWatcher(opts: FileWatcherOptions): () => Promise { const { rootPath, debounceMs = 2000, extraExtensions, onReady, onChanged } = opts; const trackedExtensions = new Set( getSupportedExtensions(extraExtensions).map((extension) => extension.toLowerCase()) ); let debounceTimer: ReturnType | undefined; + let stopped = false; + let closePromise: Promise | undefined; const trigger = (filePath: string) => { + if (stopped) return; if (!isTrackedSourcePath(filePath, trackedExtensions)) return; if (debounceTimer !== undefined) clearTimeout(debounceTimer); debounceTimer = setTimeout(() => { @@ -46,21 +49,35 @@ export function startFileWatcher(opts: FileWatcherOptions): () => void { }; const watcher = chokidar.watch(rootPath, { - ignored: [...EXCLUDED_GLOB_PATTERNS], + ignored: (filePath) => { + const pathParts = filePath.split(/[\\/]+/); + return EXCLUDED_DIRECTORY_NAMES.some((directoryName) => pathParts.includes(directoryName)); + }, persistent: true, ignoreInitial: true, awaitWriteFinish: { stabilityThreshold: 200, pollInterval: 100 } }); watcher - .on('ready', () => onReady?.()) + .on('ready', () => { + if (!stopped) onReady?.(); + }) .on('add', trigger) .on('change', trigger) .on('unlink', trigger) .on('error', (err: unknown) => console.error('[file-watcher] error:', err)); return () => { + if (closePromise) return closePromise; + stopped = true; if (debounceTimer !== undefined) clearTimeout(debounceTimer); - void watcher.close(); + debounceTimer = undefined; + try { + closePromise = watcher.close(); + } catch (error) { + closePromise = Promise.reject(error); + void closePromise.catch(() => undefined); + } + return closePromise; }; } diff --git a/src/core/index-meta.ts b/src/core/index-meta.ts index 4994235..52f66a3 100644 --- a/src/core/index-meta.ts +++ b/src/core/index-meta.ts @@ -10,16 +10,19 @@ import { INDEX_META_VERSION, INTELLIGENCE_FILENAME, KEYWORD_INDEX_FILENAME, + MEMORY_FILENAME, RELATIONSHIPS_FILENAME, VECTOR_DB_DIRNAME } from '../constants/codebase-context.js'; import { IndexCorruptedError } from '../errors/index.js'; -const ArtifactHeaderSchema = z.object({ +export const ArtifactHeaderSchema = z.object({ buildId: z.string().min(1), formatVersion: z.number().int().nonnegative() }); +export type ArtifactHeader = z.infer; + const KeywordIndexFileSchema = z.object({ header: ArtifactHeaderSchema, chunks: z.array(z.unknown()) @@ -81,7 +84,8 @@ export const IndexMetaSchema = z.object({ path: z.string().min(1), provider: z.string().min(1), embeddingProvider: z.string().optional(), - embeddingModel: z.string().optional() + embeddingModel: z.string().optional(), + embeddingEnabled: z.boolean().optional() }), intelligence: z .object({ @@ -89,6 +93,11 @@ export const IndexMetaSchema = z.object({ }) .optional(), health: z + .object({ + path: z.string().min(1) + }) + .optional(), + manifest: z .object({ path: z.string().min(1) }) @@ -99,12 +108,50 @@ export const IndexMetaSchema = z.object({ export type IndexMeta = z.infer; +type KeywordIndexFile = z.infer; + +export function validateArtifactBuildIdentity( + header: ArtifactHeader, + meta: IndexMeta, + artifactLabel: string, + artifactFile: string +): void { + if (header.formatVersion !== meta.formatVersion) { + throw new IndexCorruptedError( + `${artifactLabel} formatVersion mismatch (rebuild required): meta=${meta.formatVersion}, ${artifactFile}=${header.formatVersion}` + ); + } + if (header.buildId !== meta.buildId) { + throw new IndexCorruptedError( + `${artifactLabel} buildId mismatch (rebuild required): meta=${meta.buildId}, ${artifactFile}=${header.buildId}` + ); + } +} + +/** Validate the keyword artifact's shape and identity against the active index metadata. */ +export function validateKeywordIndexArtifact(json: unknown, meta: IndexMeta): KeywordIndexFile { + const parsed = KeywordIndexFileSchema.safeParse(json); + if (!parsed.success) { + throw new IndexCorruptedError( + `Keyword index schema mismatch (rebuild required): ${parsed.error.message}` + ); + } + + validateArtifactBuildIdentity(parsed.data.header, meta, 'Keyword index', 'index.json'); + + return parsed.data; +} + export function checkEmbeddingMismatch( meta: IndexMeta, currentProvider: string, - currentModel: string + currentModel: string, + currentEmbeddingEnabled = true ): boolean { const stored = meta.artifacts.vectorDb; + const storedEmbeddingEnabled = stored.embeddingEnabled ?? true; + if (storedEmbeddingEnabled !== currentEmbeddingEnabled) return true; + if (!currentEmbeddingEnabled) return false; if (!stored.embeddingProvider || !stored.embeddingModel) return false; // legacy index, no info return stored.embeddingProvider !== currentProvider || stored.embeddingModel !== currentModel; } @@ -176,6 +223,52 @@ export async function readIndexMeta(rootDir: string): Promise { return meta; } +/** Keep an existing index's successful mode; use model-free mode only for a fresh folder. */ +export async function resolveSkipEmbedding( + rootDir: string, + explicitChoice?: boolean +): Promise { + if (explicitChoice !== undefined) { + if (typeof explicitChoice !== 'boolean') throw new TypeError('skipEmbedding must be a boolean'); + return explicitChoice; + } + + try { + const meta = await readIndexMeta(rootDir); + // Metadata written before the mode field describes a semantic index. + return meta.artifacts.vectorDb.embeddingEnabled === false; + } catch (metaError) { + const contextDir = path.join(rootDir, CODEBASE_CONTEXT_DIRNAME); + // A valid older index may retain a trustworthy mode despite requiring a migration. + try { + const raw = await fs.readFile(path.join(contextDir, INDEX_META_FILENAME), 'utf8'); + const legacy = IndexMetaSchema.safeParse(JSON.parse(raw)); + if ( + legacy.success && + legacy.data.metaVersion <= INDEX_META_VERSION && + legacy.data.formatVersion <= INDEX_FORMAT_VERSION + ) { + return legacy.data.artifacts.vectorDb.embeddingEnabled === false; + } + } catch { + // Malformed metadata cannot establish a safe automatic mode. + } + + let entries: string[]; + try { + entries = await fs.readdir(contextDir); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return true; + throw asIndexCorrupted('Cannot recover embedding mode (rebuild required)', error); + } + if (entries.every((entry) => entry === MEMORY_FILENAME)) return true; + throw asIndexCorrupted( + 'Cannot recover embedding mode (rebuild required). Select --no-embeddings or --embeddings explicitly', + metaError + ); + } +} + export async function validateIndexArtifacts(rootDir: string, meta: IndexMeta): Promise { const contextDir = path.join(rootDir, CODEBASE_CONTEXT_DIRNAME); @@ -190,25 +283,7 @@ export async function validateIndexArtifacts(rootDir: string, meta: IndexMeta): // Keyword index header (required) try { const raw = await fs.readFile(keywordPath, 'utf-8'); - const json = JSON.parse(raw); - const parsed = KeywordIndexFileSchema.safeParse(json); - if (!parsed.success) { - throw new IndexCorruptedError( - `Keyword index schema mismatch (rebuild required): ${parsed.error.message}` - ); - } - - const { buildId, formatVersion } = parsed.data.header; - if (formatVersion !== meta.formatVersion) { - throw new IndexCorruptedError( - `Keyword index formatVersion mismatch (rebuild required): meta=${meta.formatVersion}, index.json=${formatVersion}` - ); - } - if (buildId !== meta.buildId) { - throw new IndexCorruptedError( - `Keyword index buildId mismatch (rebuild required): meta=${meta.buildId}, index.json=${buildId}` - ); - } + validateKeywordIndexArtifact(JSON.parse(raw), meta); } catch (error) { if (error instanceof IndexCorruptedError) throw error; throw asIndexCorrupted('Keyword index corrupted (rebuild required)', error); diff --git a/src/core/indexer.ts b/src/core/indexer.ts index 93aebbe..a7aed05 100644 --- a/src/core/indexer.ts +++ b/src/core/indexer.ts @@ -3,7 +3,7 @@ * Scans files, delegates to analyzers, creates embeddings, stores in vector DB */ -import { randomUUID } from 'crypto'; +import { createHash, randomUUID } from 'crypto'; import { promises as fs } from 'fs'; import path from 'path'; import { glob } from 'glob'; @@ -24,18 +24,24 @@ import { analyzerRegistry } from './analyzer-registry.js'; import type { AnalyzerSelectionOptions } from './analyzer-registry.js'; import { getSupportedExtensions, isBinaryFile, isCodeFile } from '../utils/language-detection.js'; import { + assertEmbeddingRuntimeAvailable, getEmbeddingProvider, getConfiguredDimensions, DEFAULT_MODEL, parseEmbeddingProviderName } from '../embeddings/index.js'; -import { getStorageProvider, CodeChunkWithEmbedding } from '../storage/index.js'; +import { + assertStorageRuntimeAvailable, + getStorageProvider, + CodeChunkWithEmbedding +} from '../storage/index.js'; import { LibraryUsageTracker, PatternDetector, ImportGraph, InternalFileGraph, - FileExport + FileExport, + ImportEdgeSource } from '../utils/usage-tracker.js'; import { mergeSmallChunks } from '../utils/chunking.js'; import { getFileCommitDates } from '../utils/git-dates.js'; @@ -53,21 +59,30 @@ import { RELATIONSHIPS_FILENAME, VECTOR_DB_DIRNAME } from '../constants/codebase-context.js'; +import { isPathWithin } from '../utils/project-discovery.js'; import { deriveCodebaseHealth } from '../health/derive.js'; +import { + extractSourceReexports, + InternalSourceResolver, + loadTsconfigCompilerOptions +} from '../utils/internal-source-resolver.js'; const STAGING_DIRNAME = '.staging'; const PREVIOUS_DIRNAME = '.previous'; const DEFAULT_MAX_CHUNKS = 5000; +class SourceSnapshotChangedError extends Error {} + import { computeFileHashes, + hashFileContent, readManifest, writeManifest, diffManifest, type FileManifest, type ManifestDiff } from './manifest.js'; -import { readIndexMeta, checkEmbeddingMismatch } from './index-meta.js'; +import { readIndexMeta, checkEmbeddingMismatch, resolveSkipEmbedding } from './index-meta.js'; let cachedToolVersion: string | null = null; @@ -232,6 +247,78 @@ interface PersistedIndexingStats { generatedAt: string; } +const MAX_INDEX_FILES = 10000; +const MAX_PREVIEW_SKIP_SAMPLES = 20; + +export interface IndexFileSelection { + selected: string[]; + overLimitCount: number; + overLimitSamples: string[]; +} + +export interface IndexPreview { + rootPath: string; + eligibleFiles: string[]; + skippedFiles: Array<{ path: string; reason: string }>; + skippedFileCount: number; + /** Patterns applied before glob enumeration; matching files are not individually observed. */ + excludedPatterns: string[]; +} + +interface FileScanResult { + files: string[]; + skippedFiles: Array<{ path: string; reason: string }>; + skippedFileCount: number; +} + +type SourcePathResolution = + | { status: 'contained'; path: string } + | { status: 'outside' } + | { status: 'alias' } + | { status: 'unavailable' }; + +function normalizeComparablePath(filePath: string): string { + const normalized = path.normalize(path.resolve(filePath)); + return process.platform === 'win32' ? normalized.toLowerCase() : normalized; +} + +async function resolveContainedSourcePath( + filePath: string, + canonicalRootPath: string +): Promise { + let canonicalFilePath: string; + try { + canonicalFilePath = await fs.realpath(filePath); + } catch { + return { status: 'unavailable' }; + } + if (!isPathWithin(canonicalRootPath, canonicalFilePath)) return { status: 'outside' }; + if (normalizeComparablePath(filePath) !== normalizeComparablePath(canonicalFilePath)) { + return { status: 'alias' }; + } + return { status: 'contained', path: canonicalFilePath }; +} + +function addSkippedFile( + scan: FileScanResult, + filePath: string, + reason: string, + rootPath: string +): void { + scan.skippedFileCount++; + if (scan.skippedFiles.length < MAX_PREVIEW_SKIP_SAMPLES) { + scan.skippedFiles.push({ path: path.relative(rootPath, filePath).replace(/\\/g, '/'), reason }); + } +} + +export function selectFilesForIndex(files: string[]): IndexFileSelection { + return { + selected: files.slice(0, MAX_INDEX_FILES), + overLimitCount: Math.max(0, files.length - MAX_INDEX_FILES), + overLimitSamples: files.slice(MAX_INDEX_FILES, MAX_INDEX_FILES + MAX_PREVIEW_SKIP_SAMPLES) + }; +} + export class CodebaseIndexer { private rootPath: string; private config: CodebaseConfig; @@ -240,10 +327,12 @@ export class CodebaseIndexer { private progress: IndexingProgress; private onProgressCallback?: (progress: IndexingProgress) => void; private incrementalOnly: boolean; + private readonly explicitSkipEmbedding?: boolean; constructor(options: IndexerOptions) { this.rootPath = path.resolve(options.rootPath); this.config = this.mergeConfig(options.config); + this.explicitSkipEmbedding = options.config?.skipEmbedding; this.projectOptions = { preferredAnalyzer: options.projectOptions?.preferredAnalyzer, extraFileExtensions: options.projectOptions?.extraFileExtensions @@ -324,7 +413,7 @@ export class CodebaseIndexer { model: defaultModel, batchSize: 32 }, - skipEmbedding: false, + skipEmbedding: true, storage: { provider: 'lancedb', path: './codebase-index' @@ -385,6 +474,38 @@ export class CodebaseIndexer { return Array.from(new Set([...includePatterns, ...extraPatterns])); } + /** Enumerate the exact files the next index pass would select, without indexing or writing. */ + async previewFiles(): Promise { + const scan = await this.scanFiles(false); + const selection = selectFilesForIndex(scan.files); + const skippedFiles = [...scan.skippedFiles]; + for (const file of selection.overLimitSamples.slice( + 0, + MAX_PREVIEW_SKIP_SAMPLES - skippedFiles.length + )) { + skippedFiles.push({ + path: path.relative(this.rootPath, file).replace(/\\/g, '/'), + reason: 'exceeds the indexer file-count limit' + }); + } + const toRelativePath = (file: string): string => + path.relative(this.rootPath, file).replace(/\\/g, '/'); + const comparePaths = (left: string, right: string): number => + left < right ? -1 : left > right ? 1 : 0; + const eligibleFiles = selection.selected.map(toRelativePath).sort(comparePaths); + skippedFiles.sort( + (left, right) => + comparePaths(left.path, right.path) || comparePaths(left.reason, right.reason) + ); + return { + rootPath: this.rootPath, + eligibleFiles, + skippedFiles, + skippedFileCount: scan.skippedFileCount + selection.overLimitCount, + excludedPatterns: [...(this.config.exclude ?? [])] + }; + } + async index(): Promise { const startTime = Date.now(); const stats: IndexingStats = { @@ -414,6 +535,14 @@ export class CodebaseIndexer { let stagingDir: string | null = null; try { + this.config.skipEmbedding = await resolveSkipEmbedding( + this.rootPath, + this.explicitSkipEmbedding + ); + if (!this.config.skipEmbedding) { + assertEmbeddingRuntimeAvailable(this.config.embedding); + assertStorageRuntimeAvailable(); + } // Ensure there is at least a generic fallback analyzer registered when the indexer // is used directly (e.g. in tests or standalone scripts). if (analyzerRegistry.getAll().length === 0) { @@ -429,18 +558,18 @@ export class CodebaseIndexer { // Phase 1: Scanning this.updateProgress('scanning', 0); - let files = await this.scanFiles(); + const scan = await this.scanFiles(); + const selection = selectFilesForIndex(scan.files); + const files = selection.selected; // Memory safety: limit total files to prevent heap exhaustion - const MAX_FILES = 10000; - if (files.length > MAX_FILES) { + if (selection.overLimitCount > 0) { console.warn( - `WARNING: Found ${files.length} files, limiting to ${MAX_FILES} to prevent memory issues.` + `WARNING: Found ${scan.files.length} files, limiting to ${MAX_INDEX_FILES} to prevent memory issues.` ); console.warn( `Consider using more specific include patterns or excluding large directories.` ); - files = files.slice(0, MAX_FILES); } stats.totalFiles = files.length; @@ -482,15 +611,24 @@ export class CodebaseIndexer { const existingMeta = await readIndexMeta(this.rootPath); const currentProvider = this.config.embedding?.provider ?? 'transformers'; const currentModel = this.config.embedding?.model ?? DEFAULT_MODEL; - if (checkEmbeddingMismatch(existingMeta, currentProvider, currentModel)) { + if ( + checkEmbeddingMismatch( + existingMeta, + currentProvider, + currentModel, + !this.config.skipEmbedding + ) + ) { const stored = existingMeta.artifacts.vectorDb; console.error( - `Embedding provider/model changed (stored: ${stored.embeddingProvider}:${stored.embeddingModel}, current: ${currentProvider}:${currentModel}) — forcing full rebuild` + `Embedding configuration changed (stored: ${stored.embeddingEnabled === false ? 'disabled' : `${stored.embeddingProvider ?? 'unknown'}:${stored.embeddingModel ?? 'unknown'}`}, current: ${this.config.skipEmbedding ? 'disabled' : `${currentProvider}:${currentModel}`}); forcing full rebuild` ); diff = null; } } catch { - // No meta yet or legacy index without embedding fields — proceed with incremental + // Chunk semantics and embedding input may have changed even when source hashes did not. + // Incompatible metadata must never reuse chunks from the previous format. + diff = null; } // Short-circuit: nothing changed @@ -585,6 +723,30 @@ export class CodebaseIndexer { // When incremental, track which files need embedding const filesToProcessSet = diff ? new Set(filesToProcess.map((f) => f)) : null; + // Resolve graph edges only to discovered, root-confined source files. + let tsconfigPaths: Record | undefined; + const tsconfigCompilerOptions = loadTsconfigCompilerOptions(this.rootPath); + if (tsconfigCompilerOptions.paths) { + tsconfigPaths = Object.fromEntries( + Object.entries(tsconfigCompilerOptions.paths).map(([alias, targets]) => [ + alias, + [...targets] + ]) + ); + } + const sourceResolver = await InternalSourceResolver.create(this.rootPath, files, { + compilerOptions: tsconfigCompilerOptions + }); + const sourceFingerprints = new Map(); + const analyzedManifestHashes = new Map(); + const pendingResolvedEdges: Array<{ + from: string; + to: string; + line: number; + importedSymbols: string[]; + source: Omit; + }> = []; + for (let i = 0; i < files.length; i++) { const file = files[i]; this.progress.currentFile = file; @@ -593,13 +755,47 @@ export class CodebaseIndexer { try { // Normalize line endings to \n for consistent cross-platform output - const rawContent = await fs.readFile(file, 'utf-8'); + const resolution = await resolveContainedSourcePath(file, this.rootPath); + if (resolution.status !== 'contained') { + throw new Error( + resolution.status === 'outside' + ? `Source resolves outside selected root: ${file}` + : resolution.status === 'alias' + ? `Source alias is not indexed: ${file}` + : `Source could not be resolved before reading: ${file}` + ); + } + const resolvedFile = resolution.path; + const rawBytes = await fs.readFile(resolvedFile); + const sourceFingerprint = createHash('sha256').update(rawBytes).digest('hex'); + sourceFingerprints.set(resolvedFile, sourceFingerprint); + const rawContent = rawBytes.toString('utf-8'); + const relativeFile = path.relative(this.rootPath, resolvedFile).replace(/\\/g, '/'); + const analyzedManifestHash = hashFileContent(rawContent); + analyzedManifestHashes.set(relativeFile, analyzedManifestHash); + if ( + currentHashes && + diff?.unchanged.includes(relativeFile) && + currentHashes[relativeFile] !== analyzedManifestHash + ) { + throw new SourceSnapshotChangedError( + `Source changed during incremental scan; retry indexing: ${relativeFile}` + ); + } const content = rawContent.replace(/\r\n/g, '\n'); - const result = await analyzerRegistry.analyzeFile(file, content, resolvedProjectOptions); + const result = await analyzerRegistry.analyzeFile(file, content, { + ...resolvedProjectOptions, + includeScopePrefix: false + }); if (result) { const isFileChanged = !filesToProcessSet || filesToProcessSet.has(file); + // Analyzer paths are derived from process.cwd(); persisted paths belong to this index root. + for (const chunk of result.chunks) { + chunk.relativePath = relativeFile; + } + const mergedChunks = mergeSmallChunks(result.chunks, 15); allChunks.push(...mergedChunks); @@ -614,24 +810,58 @@ export class CodebaseIndexer { libraryTracker.track(imp.source, file); importGraph.trackImport(imp.source, file, imp.line || 1); - // Track internal file-to-file imports (relative paths) - if (imp.source.startsWith('.')) { - // Resolve the relative import to an absolute path - const fileDir = path.dirname(file); - let resolvedPath = path.resolve(fileDir, imp.source); - - // Try common extensions if not already specified - const ext = path.extname(resolvedPath); - if (!ext) { - for (const tryExt of ['.ts', '.tsx', '.js', '.jsx']) { - const withExt = resolvedPath + tryExt; - // We don't check if file exists for performance - just track what's referenced - resolvedPath = withExt; - break; + const resolution = await sourceResolver.resolve(file, imp.source); + if (resolution.status === 'resolved' && resolution.path) { + pendingResolvedEdges.push({ + from: file, + to: resolution.path, + line: imp.line || 1, + importedSymbols: imp.imports, + source: { + kind: 'import', + specifier: imp.source, + sourceFingerprint } - } + }); + } else { + internalFileGraph.trackUnresolved( + file, + { + line: imp.line || 1, + specifier: imp.source, + kind: 'import', + sourceFingerprint, + targetFingerprint: '' + }, + resolution.reason ?? 'not-found' + ); + } + } - internalFileGraph.trackImport(file, resolvedPath, imp.line || 1, imp.imports); + // Re-export relationships are separate typed edges and require AST source evidence. + for (const reexport of extractSourceReexports(content, file)) { + const resolution = await sourceResolver.resolve(file, reexport.specifier); + const proof = { + line: reexport.line, + specifier: reexport.specifier, + kind: 'reexport' as const, + sourceFingerprint, + targetFingerprint: '' + }; + if (resolution.status === 'resolved' && resolution.path) { + pendingResolvedEdges.push({ + from: file, + to: resolution.path, + line: reexport.line, + importedSymbols: [], + source: { + kind: proof.kind, + specifier: proof.specifier, + sourceFingerprint: proof.sourceFingerprint + } + }); + } else { + internalFileGraph.trackUnresolved(file, proof, resolution.reason ?? 'not-found'); } } @@ -727,6 +957,18 @@ export class CodebaseIndexer { stats.skippedFiles++; } } catch (error) { + if (error instanceof SourceSnapshotChangedError) throw error; + const relativeFile = path.relative(this.rootPath, file).replace(/\\/g, '/'); + if ( + diff?.unchanged.includes(relativeFile) && + error instanceof Error && + 'code' in error && + error.code === 'ENOENT' + ) { + throw new SourceSnapshotChangedError( + `Source disappeared during incremental scan; retry indexing: ${relativeFile}` + ); + } stats.skippedFiles++; stats.errors.push({ filePath: file, @@ -741,6 +983,28 @@ export class CodebaseIndexer { } } + // Finish edges only after each endpoint has been read for this build. This binds both + // fingerprints to the exact source snapshots analyzed above, without retaining file bytes. + for (const edge of pendingResolvedEdges) { + const targetFingerprint = sourceFingerprints.get(edge.to); + if (targetFingerprint) { + internalFileGraph.trackImport(edge.from, edge.to, edge.line, edge.importedSymbols, { + ...edge.source, + targetFingerprint + }); + } else { + internalFileGraph.trackUnresolved( + edge.from, + { + ...edge.source, + line: edge.line, + targetFingerprint: '' + }, + 'not-found' + ); + } + } + stats.totalChunks = allChunks.length; stats.avgChunkSize = allChunks.length > 0 @@ -910,20 +1174,8 @@ export class CodebaseIndexer { const intelligencePath = path.join(activeContextDir, INTELLIGENCE_FILENAME); const libraryStats = libraryTracker.getStats(); - // Extract tsconfig paths for AI to understand import aliases - let tsconfigPaths: Record | undefined; - try { - const tsconfigPath = path.join(this.rootPath, 'tsconfig.json'); - const tsconfigContent = await fs.readFile(tsconfigPath, 'utf-8'); - const tsconfig = JSON.parse(tsconfigContent); - if (tsconfig.compilerOptions?.paths) { - tsconfigPaths = tsconfig.compilerOptions.paths; - console.error( - `Found ${Object.keys(tsconfigPaths!).length} path aliases in tsconfig.json` - ); - } - } catch (_error) { - // No tsconfig.json or no paths defined + if (tsconfigPaths && Object.keys(tsconfigPaths).length > 0) { + console.error(`Found ${Object.keys(tsconfigPaths).length} path aliases in tsconfig.json`); } const intelligence = { @@ -980,6 +1232,7 @@ export class CodebaseIndexer { graph: { imports: graphData.imports || {}, ...(graphData.importDetails ? { importDetails: graphData.importDetails } : {}), + ...(graphData.unresolvedEdges ? { unresolvedEdges: graphData.unresolvedEdges } : {}), importedBy, exports: graphData.exports || {} }, @@ -1003,10 +1256,12 @@ export class CodebaseIndexer { // Write manifest (both full and incremental) // For full rebuild, write to staging; for incremental, write to active const activeManifestPath = path.join(activeContextDir, MANIFEST_FILENAME); + const analyzedHashes = Object.fromEntries(analyzedManifestHashes); const manifest: FileManifest = { version: 1, generatedAt: new Date().toISOString(), - files: currentHashes ?? (await computeFileHashes(files, this.rootPath)) + header: { buildId, formatVersion: INDEX_FORMAT_VERSION }, + files: analyzedHashes }; await writeManifest(activeManifestPath, manifest); @@ -1036,7 +1291,8 @@ export class CodebaseIndexer { path: VECTOR_DB_DIRNAME, provider: 'lancedb', embeddingProvider: this.config.embedding?.provider ?? 'transformers', - embeddingModel: this.config.embedding?.model ?? DEFAULT_MODEL + embeddingModel: this.config.embedding?.model ?? DEFAULT_MODEL, + embeddingEnabled: !this.config.skipEmbedding }, intelligence: { path: INTELLIGENCE_FILENAME }, manifest: { path: MANIFEST_FILENAME }, @@ -1091,9 +1347,10 @@ export class CodebaseIndexer { } } - private async scanFiles(): Promise { - const files: string[] = []; + private async scanFiles(warnOnLargeFiles = true): Promise { + const scan: FileScanResult = { files: [], skippedFiles: [], skippedFileCount: 0 }; const seen = new Set(); + this.rootPath = await fs.realpath(this.rootPath); // Read .gitignore if respecting it let ig: ReturnType | null = null; @@ -1127,33 +1384,55 @@ export class CodebaseIndexer { seen.add(normalizedFile); const relativePath = path.relative(this.rootPath, file); + const sourcePath = await resolveContainedSourcePath(file, this.rootPath); + if (sourcePath.status !== 'contained') { + addSkippedFile( + scan, + file, + sourcePath.status === 'outside' + ? 'source resolves outside selected root' + : sourcePath.status === 'alias' + ? 'source alias is not indexed; use its canonical path' + : 'file could not be inspected', + this.rootPath + ); + continue; + } // Check gitignore if (ig && ig.ignores(relativePath)) { + addSkippedFile(scan, file, 'ignored by .gitignore', this.rootPath); continue; } // Check if it's a code file - if (!isCodeFile(file, this.supportedCodeExtensions) || isBinaryFile(file)) { + if ( + !isCodeFile(sourcePath.path, this.supportedCodeExtensions) || + isBinaryFile(sourcePath.path) + ) { + addSkippedFile(scan, file, 'unsupported or binary file', this.rootPath); continue; } // Check file size try { - const stats = await fs.stat(file); + const stats = await fs.stat(sourcePath.path); if (stats.size > (this.config.parsing?.maxFileSize || 1048576)) { - console.warn(`Skipping large file: ${file} (${stats.size} bytes)`); + if (warnOnLargeFiles) + console.warn(`Skipping large file: ${file} (${stats.size} bytes)`); + addSkippedFile(scan, file, 'exceeds configured maximum file size', this.rootPath); continue; } } catch (_error) { + addSkippedFile(scan, file, 'file could not be inspected', this.rootPath); continue; } - files.push(file); + scan.files.push(sourcePath.path); } } - return files; + return scan; } private updateProgress(phase: IndexingPhase, percentage: number): void { diff --git a/src/core/manifest.ts b/src/core/manifest.ts index 04f7c61..36bbaa8 100644 --- a/src/core/manifest.ts +++ b/src/core/manifest.ts @@ -6,13 +6,34 @@ import { createHash } from 'crypto'; import { promises as fs } from 'fs'; import path from 'path'; +import { z } from 'zod'; + +import { CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME } from '../constants/codebase-context.js'; +import { IndexCorruptedError } from '../errors/index.js'; +import { + ArtifactHeaderSchema, + validateArtifactBuildIdentity, + type ArtifactHeader, + type IndexMeta +} from './index-meta.js'; +import { isPathWithin } from '../utils/project-discovery.js'; export interface FileManifest { version: 1; generatedAt: string; files: Record; // relativePath → SHA-256 hash (first 16 hex chars) + header?: ArtifactHeader; } +const ReferenceManifestSchema = z + .object({ + version: z.literal(1), + generatedAt: z.string().datetime(), + header: ArtifactHeaderSchema, + files: z.record(z.string(), z.string().regex(/^[a-f0-9]{16}$/i)) + }) + .strict(); + export interface ManifestDiff { added: string[]; // new files (not in old manifest) changed: string[]; // hash differs @@ -44,6 +65,107 @@ export async function readManifest(manifestPath: string): Promise !segment || segment === '.' || segment === '..')) { + return false; + } + + const resolved = path.resolve(rootPath, relativePath); + if (!isPathWithin(rootPath, resolved)) return false; + return path.relative(rootPath, resolved).replace(/\\/g, '/') === normalized; +} + +/** Read the build-bound source nominator used by current-source reference lookup. */ +export async function readReferenceManifest( + rootPath: string, + meta: IndexMeta +): Promise { + if (meta.artifacts.manifest?.path !== MANIFEST_FILENAME) { + throw fullReindexError('Reference manifest path is missing or invalid.'); + } + + let realRoot: string; + let realContext: string; + try { + realRoot = await fs.realpath(rootPath); + realContext = await fs.realpath(path.join(realRoot, CODEBASE_CONTEXT_DIRNAME)); + } catch (error) { + throw fullReindexError( + 'Reference manifest directory is missing or unreadable.', + error instanceof Error ? error.message : String(error) + ); + } + if (!isPathWithin(realRoot, realContext)) { + throw fullReindexError('Reference manifest directory resolves outside the selected project.'); + } + + const expectedManifestPath = path.join(realContext, MANIFEST_FILENAME); + let realManifestPath: string; + try { + realManifestPath = await fs.realpath(expectedManifestPath); + } catch (error) { + throw fullReindexError( + 'Reference manifest is missing or unreadable.', + error instanceof Error ? error.message : String(error) + ); + } + if (!isPathWithin(realContext, realManifestPath)) { + throw fullReindexError('Reference manifest resolves outside the index directory.'); + } + + let json: unknown; + try { + json = JSON.parse(await fs.readFile(realManifestPath, 'utf8')); + } catch (error) { + throw fullReindexError( + 'Reference manifest is malformed or unreadable.', + error instanceof Error ? error.message : String(error) + ); + } + + const parsed = ReferenceManifestSchema.safeParse(json); + if (!parsed.success) { + const details = parsed.error.issues.some((issue) => issue.path[0] === 'header') + ? 'The source manifest is legacy or has an invalid build header.' + : `Manifest schema mismatch: ${parsed.error.message}`; + throw fullReindexError('Reference manifest is invalid.', details); + } + + try { + validateArtifactBuildIdentity( + parsed.data.header, + meta, + 'Reference manifest', + MANIFEST_FILENAME + ); + } catch (error) { + throw fullReindexError( + 'Reference manifest build identity does not match index metadata.', + error instanceof Error ? error.message : String(error) + ); + } + + for (const relativePath of Object.keys(parsed.data.files)) { + if (!isSafeManifestSourcePath(realRoot, relativePath)) { + throw new IndexCorruptedError( + `Reference manifest contains an invalid or escaping source path (${relativePath}). Rebuild required.` + ); + } + } + + return parsed.data; +} + /** * Write a manifest to disk. */ diff --git a/src/core/search.ts b/src/core/search.ts index 4a9be1b..4bcc2e9 100644 --- a/src/core/search.ts +++ b/src/core/search.ts @@ -2,22 +2,30 @@ * Hybrid search combining semantic vector search with keyword matching */ -import Fuse from 'fuse.js'; +import Fuse, { type FuseResult, type IFuseOptions } from 'fuse.js'; import path from 'path'; import { promises as fs } from 'fs'; import { CodeChunk, SearchResult, SearchFilters, IntelligenceData } from '../types/index.js'; import { EmbeddingProvider, EmbeddingConfig, getEmbeddingProvider } from '../embeddings/index.js'; import { VectorStorageProvider, getStorageProvider } from '../storage/index.js'; import { analyzerRegistry } from './analyzer-registry.js'; -import { IndexCorruptedError } from '../errors/index.js'; +import { IndexCorruptedError, OptionalDependencyError } from '../errors/index.js'; import { isTestingRelatedQuery } from '../preflight/query-scope.js'; import { assessSearchQuality } from './search-quality.js'; -import { rerank, getRerankerStatus } from './reranker.js'; +import { rerank } from './reranker.js'; import { type IndexMeta, readIndexMeta, validateIndexArtifacts } from './index-meta.js'; +import { hashFileContent, readManifest, type FileManifest } from './manifest.js'; +import { isPathWithin } from '../utils/project-discovery.js'; +import { + InternalSourceResolver, + loadTsconfigCompilerOptions +} from '../utils/internal-source-resolver.js'; +import type { ImportEdgeDetail } from '../utils/usage-tracker.js'; import { CODEBASE_CONTEXT_DIRNAME, INTELLIGENCE_FILENAME, KEYWORD_INDEX_FILENAME, + MANIFEST_FILENAME, VECTOR_DB_DIRNAME } from '../constants/codebase-context.js'; @@ -36,6 +44,39 @@ export interface SearchOptions { export type SearchIntentProfile = 'explore' | 'edit' | 'refactor' | 'migrate'; +/** Coverage of nominated sources, not a claim that the entire repository was searched. */ +export interface SearchSourceCoverage { + status: 'current' | 'partial' | 'no_fresh_results'; + checkedFiles: number; + skippedFiles: number; + candidateBudgetExhausted?: true; + candidateRetrievalFailed?: true; + refreshAction?: string; +} + +export interface SearchOutcome { + results: SearchResult[]; + sourceCoverage: SearchSourceCoverage; +} + +// Null records ordinary drift/unavailability; integrity failures still throw. +type CheckedSources = Map; + +interface SourceValidationContext { + sources: CheckedSources; + realRootPath?: Promise; + candidateBudgetExhausted?: true; + candidateRetrievalFailed?: true; +} + +interface SourceInspectionBudget { + limit: number; + inspected: Set; +} + +// Reuse the lexical rescue work ceiling for bounded stale-candidate replenishment. +const MAX_CANDIDATE_WORK = 750; + type QueryIntent = 'EXACT_NAME' | 'CONCEPTUAL' | 'FLOW' | 'CONFIG' | 'WIRING'; interface QueryVariant { @@ -108,6 +149,44 @@ const QUERY_STOP_WORDS = new Set([ 'before' ]); +// Syntax and question scaffolding are weak evidence after strict keyword +// retrieval fails, so they cannot independently support a rescue match. +const NON_DISCRIMINATIVE_FALLBACK_TERMS = new Set([ + 'class', + 'code', + 'const', + 'does', + 'export', + 'file', + 'function', + 'import', + 'return', + 'something', + 'still', + 'thing', + 'value', + 'what', + 'when', + 'where', + 'why' +]); + +const KEYWORD_FUSE_OPTIONS: IFuseOptions = { + keys: [ + { name: 'content', weight: 0.4 }, + { name: 'metadata.componentName', weight: 0.25 }, + { name: 'filePath', weight: 0.15 }, + { name: 'relativePath', weight: 0.15 }, + { name: 'componentType', weight: 0.15 }, + { name: 'layer', weight: 0.1 }, + { name: 'tags', weight: 0.15 } + ], + includeScore: true, + threshold: 0.4, + useExtendedSearch: true, + ignoreLocation: true +}; + export class CodebaseSearcher { private rootPath: string; private storagePath: string; @@ -121,6 +200,7 @@ export class CodebaseSearcher { private storageProvider: VectorStorageProvider | null = null; private initialized = false; + private sourceManifest: FileManifest | null = null; // Pattern intelligence for trend detection private patternIntelligence: { @@ -130,6 +210,7 @@ export class CodebaseSearcher { } | null = null; private importCentrality: Map | null = null; + private indexedImports = new Map>(); constructor(rootPath: string) { this.rootPath = rootPath; @@ -145,23 +226,29 @@ export class CodebaseSearcher { await validateIndexArtifacts(this.rootPath, this.indexMeta); await this.loadKeywordIndex(); + await this.loadSourceManifest(); await this.loadPatternIntelligence(); // Use the embedding config the index was built with, not the current env-var defaults. // This ensures query vectors are in the same space as stored vectors. // Legacy indexes (no embeddingProvider stored) fall back to env-var defaults. - const storedProvider = this.indexMeta.artifacts.vectorDb.embeddingProvider; - const storedModel = this.indexMeta.artifacts.vectorDb.embeddingModel; - const embeddingConfig: Partial = storedProvider - ? { provider: storedProvider as EmbeddingConfig['provider'], model: storedModel } - : {}; - this.embeddingProvider = await getEmbeddingProvider(embeddingConfig); - this.storageProvider = await getStorageProvider({ - path: this.storagePath - }); + if (this.indexMeta.artifacts.vectorDb.embeddingEnabled !== false) { + const storedProvider = this.indexMeta.artifacts.vectorDb.embeddingProvider; + const storedModel = this.indexMeta.artifacts.vectorDb.embeddingModel; + const embeddingConfig: Partial = storedProvider + ? { provider: storedProvider as EmbeddingConfig['provider'], model: storedModel } + : {}; + this.embeddingProvider = await getEmbeddingProvider(embeddingConfig); + this.storageProvider = await getStorageProvider({ + path: this.storagePath + }); + } this.initialized = true; } catch (error) { + if (error instanceof OptionalDependencyError) { + throw error; + } if (error instanceof IndexCorruptedError) { throw error; // Propagate to handler for auto-heal } @@ -170,6 +257,148 @@ export class CodebaseSearcher { } } + private getIndexedSourcePath(chunk: CodeChunk): string { + if (typeof chunk.filePath !== 'string' || !chunk.filePath.trim()) { + throw new IndexCorruptedError('Indexed source location is invalid. Full rebuild required.'); + } + + const indexedPath = path.resolve(this.rootPath, chunk.filePath); + if (!isPathWithin(this.rootPath, indexedPath)) { + throw new IndexCorruptedError( + 'Indexed source belongs to a different project location. Full rebuild required.' + ); + } + + if (typeof chunk.relativePath === 'string' && chunk.relativePath.trim()) { + const rawRelativePath = chunk.relativePath.trim(); + if (!path.isAbsolute(rawRelativePath)) { + const relativeSourcePath = path.resolve(this.rootPath, rawRelativePath); + if ( + isPathWithin(this.rootPath, relativeSourcePath) && + path.relative(relativeSourcePath, indexedPath) !== '' + ) { + throw new IndexCorruptedError( + 'Indexed source belongs to a different project location. Full rebuild required.' + ); + } + } + } + + // Some existing analyzers recorded relativePath from process.cwd(). The absolute + // filePath remains authoritative when it is confined to the selected root. + return indexedPath; + } + + private async loadSourceManifest(): Promise { + const manifest = await readManifest( + path.join(this.rootPath, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME) + ); + if (!manifest?.files || Array.isArray(manifest.files)) { + throw new IndexCorruptedError( + 'Source manifest is missing or invalid. Full rebuild required.' + ); + } + + // Reject copied indexes with foreign absolute source locations before loading models. + for (const chunk of this.chunks) this.getIndexedSourcePath(chunk); + this.sourceManifest = manifest; + } + + private async validateCandidateSources( + chunks: Iterable, + validation: SourceValidationContext, + budget?: SourceInspectionBudget + ): Promise { + const checkedSources = validation.sources; + if (!this.sourceManifest) { + throw new IndexCorruptedError('Source manifest is unavailable. Full rebuild required.'); + } + validation.realRootPath ??= fs.realpath(this.rootPath).catch(() => { + throw new IndexCorruptedError('Selected project is unavailable. Full rebuild required.'); + }); + const realRootPath = await validation.realRootPath; + for (const chunk of chunks) { + const sourcePath = this.getIndexedSourcePath(chunk); + const sourceKey = path.resolve(this.rootPath, chunk.filePath); + const relativePath = path.relative(this.rootPath, sourcePath).replace(/\\/g, '/'); + const expectedHash = this.sourceManifest.files[relativePath]; + if (typeof expectedHash !== 'string' || !/^[a-f0-9]{16}$/.test(expectedHash)) { + throw new IndexCorruptedError( + 'Indexed source is absent from its manifest or its hash is invalid. Full rebuild required.' + ); + } + this.validateSourceRange(chunk); + if (!checkedSources.has(sourceKey)) { + if (budget && budget.inspected.size >= budget.limit) { + validation.candidateBudgetExhausted = true; + continue; + } + budget?.inspected.add(sourceKey); + try { + const realSourcePath = await fs.realpath(sourcePath); + if (!isPathWithin(realRootPath, realSourcePath)) { + throw new IndexCorruptedError( + 'Source resolves outside the selected project. Full rebuild required.' + ); + } + if (!(await fs.stat(realSourcePath)).isFile()) { + throw new IndexCorruptedError( + 'Indexed source is not a regular file. Full rebuild required.' + ); + } + const content = await fs.readFile(realSourcePath, 'utf-8'); + checkedSources.set( + sourceKey, + hashFileContent(content) === expectedHash + ? content.replace(/\r\n/g, '\n').split('\n') + : null + ); + } catch (error) { + if (error instanceof IndexCorruptedError) throw error; + // Source reads can fail after a successful stat (delete, permissions, I/O). + // Preserve other candidates and report incomplete coverage without rebuilding. + checkedSources.set(sourceKey, null); + } + } + const lines = checkedSources.get(sourceKey); + if (lines) this.validateSourceRange(chunk, lines); + } + } + + private async selectCurrentCandidates( + candidates: Iterable, + limit: number, + getChunk: (candidate: T) => CodeChunk, + validation: SourceValidationContext, + budget: SourceInspectionBudget = { limit, inspected: new Set() } + ): Promise { + const current: T[] = []; + for (const candidate of candidates) { + if (current.length >= limit) break; + const chunk = getChunk(candidate); + await this.validateCandidateSources([chunk], validation, budget); + const source = validation.sources.get(path.resolve(this.rootPath, chunk.filePath)); + if (source === undefined) break; // Inspection budget ended before this source. + if (source !== null) current.push(candidate); + } + return current; + } + + private validateSourceRange( + range: Pick, + lines?: string[] + ): void { + if ( + !Number.isInteger(range.startLine) || + !Number.isInteger(range.endLine) || + range.startLine < 1 || + range.endLine < range.startLine || + (lines !== undefined && range.endLine > lines.length) + ) { + throw new IndexCorruptedError('Indexed source line range is invalid. Full rebuild required.'); + } + } + private async loadKeywordIndex(): Promise { try { const indexPath = path.join(this.rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); @@ -193,21 +422,7 @@ export class CodebaseSearcher { this.chunks = chunks; - this.fuseIndex = new Fuse(this.chunks, { - keys: [ - { name: 'content', weight: 0.4 }, - { name: 'metadata.componentName', weight: 0.25 }, - { name: 'filePath', weight: 0.15 }, - { name: 'relativePath', weight: 0.15 }, - { name: 'componentType', weight: 0.15 }, - { name: 'layer', weight: 0.1 }, - { name: 'tags', weight: 0.15 } - ], - includeScore: true, - threshold: 0.4, - useExtendedSearch: true, - ignoreLocation: true - }); + this.fuseIndex = new Fuse(this.chunks, KEYWORD_FUSE_OPTIONS); } catch (error) { if (error instanceof IndexCorruptedError) { throw error; @@ -222,6 +437,7 @@ export class CodebaseSearcher { * Load pattern intelligence for trend detection and warnings */ private async loadPatternIntelligence(): Promise { + this.indexedImports.clear(); try { const intelligencePath = path.join( this.rootPath, @@ -272,12 +488,37 @@ export class CodebaseSearcher { // Count how many files import each file (in-degree centrality) const importCounts = new Map(); - for (const [_importingFile, importedFiles] of Object.entries( + const details = intelligence.internalFileGraph.importDetails as + | Record> + | undefined; + for (const [importingFile, importedFiles] of Object.entries( intelligence.internalFileGraph.imports )) { const imports = importedFiles as string[]; for (const imported of imports) { importCounts.set(imported, (importCounts.get(imported) || 0) + 1); + const sources = details?.[importingFile]?.[imported]?.sources; + const sourceHash = this.sourceManifest?.files[importingFile]; + const targetHash = this.sourceManifest?.files[imported]; + if (!sourceHash || !targetHash || !Array.isArray(sources)) continue; + const witnesses = sources.filter( + (source) => + source.kind === 'import' && + source.line > 0 && + !!source.specifier && + /^[a-f0-9]{64}$/.test(source.sourceFingerprint) && + /^[a-f0-9]{64}$/.test(source.targetFingerprint) && + source.sourceFingerprint.slice(0, 16) === sourceHash && + source.targetFingerprint.slice(0, 16) === targetHash + ); + if (witnesses.length === 0) continue; + const key = this.indexedFileKey(importingFile); + const targets = this.indexedImports.get(key) ?? new Map(); + targets.set( + this.indexedFileKey(imported), + witnesses.map((source) => source.specifier) + ); + this.indexedImports.set(key, targets); } } @@ -298,6 +539,7 @@ export class CodebaseSearcher { ); this.patternIntelligence = null; this.importCentrality = null; + this.indexedImports.clear(); } } @@ -352,6 +594,361 @@ export class CodebaseSearcher { .filter((term) => term.length > 2 && !QUERY_STOP_WORDS.has(term)); } + private normalizeLexicalTerm(term: string): string { + const lower = term.toLowerCase(); + if (lower.length > 5 && lower.endsWith('ies')) { + return `${lower.slice(0, -3)}y`; + } + if (lower.length > 4 && lower.endsWith('s') && !/(ss|us)$/.test(lower)) { + return lower.slice(0, -1); + } + return lower; + } + + private lexicalWords(text: string): string[] { + return ( + text + .replace(/([A-Z]+)([A-Z][a-z])/g, '$1 $2') + .replace(/([a-z\d])([A-Z])/g, '$1 $2') + .match(/[a-zA-Z0-9]+/g) ?? [] + ).map((word) => this.normalizeLexicalTerm(word)); + } + + private lexicalAliases(word: string): string[] { + // Keep the original word: conservative suffix aliases improve recall but + // must not erase exact evidence. Query and indexed words use identical rules. + const aliases = new Set([word]); + const stem = word.replace(/(?:ed|ing)$/, ''); + if (stem !== word && stem.length >= 4) { + aliases.add(stem); + aliases.add(`${stem}e`); + if (/([b-df-hj-np-tv-z])\1$/.test(stem)) aliases.add(stem.slice(0, -1)); + } + return [...aliases]; + } + + private lexicalQueryTerms(query: string): Map { + const groups = new Map(); + // An explicit code identifier remains one anchor, governed by the indexed + // edge proof below. Its component words cannot corroborate that anchor. + const naturalTerms = query.replace(/[a-zA-Z_][a-zA-Z0-9_]*/g, (word) => + /[a-z][A-Z]|_/.test(word) ? '' : word + ); + const words = [...new Set(this.lexicalWords(naturalTerms))] + .filter( + (term) => + term.length >= 4 && + !QUERY_STOP_WORDS.has(term) && + !NON_DISCRIMINATIVE_FALLBACK_TERMS.has(term) + ) + .slice(0, 12); + for (const word of words) { + const aliases = new Set(this.lexicalAliases(word)); + let key = word; + // Repeated inflections are one clue, never independent corroboration. + for (const [existing, values] of groups) { + if (!values.some((value) => aliases.has(value))) continue; + key = existing; + for (const value of values) aliases.add(value); + groups.delete(existing); + } + groups.set(key, [...aliases]); + } + return groups; + } + + private lexicalEvidence(text: string): Set { + return new Set(this.lexicalWords(text).flatMap((word) => this.lexicalAliases(word))); + } + + private indexedDeclarationNames(chunk: Pick): string[] { + return [ + chunk.metadata?.symbolName, + chunk.metadata?.componentName, + chunk.metadata?.functionName, + chunk.metadata?.className + ] + .filter((name): name is string => typeof name === 'string') + .flatMap((name) => name.split('+')); + } + + private hasExactQueryDeclaration(chunk: Pick, query: string): boolean { + const identifiers = new Set( + (/^[a-zA-Z_][a-zA-Z0-9_]*$/.test(query.trim()) + ? [query.trim()] + : (query.match(/[a-zA-Z_][a-zA-Z0-9_]*/g) ?? []).filter((word) => /[a-z][A-Z]|_/.test(word)) + ).map((word) => word.toLowerCase()) + ); + return this.indexedDeclarationNames(chunk).some((name) => identifiers.has(name.toLowerCase())); + } + + private matchesSearchFilters(chunk: CodeChunk, filters?: SearchFilters): boolean { + if (!filters) return true; + if (filters.componentType && chunk.componentType !== filters.componentType) return false; + if (filters.layer && chunk.layer !== filters.layer) return false; + if (filters.framework && chunk.framework !== filters.framework) return false; + if (filters.language && chunk.language !== filters.language) return false; + if (filters.tags?.length && !filters.tags.some((tag) => chunk.tags?.includes(tag))) { + return false; + } + return true; + } + + private indexedFileKey(file: string): string { + const absolute = path.resolve(this.rootPath, file).replace(/\\/g, '/'); + return process.platform === 'win32' ? absolute.toLowerCase() : absolute; + } + + /** + * Extended Fuse treats spaces as AND within one chunk. If strict retrieval + * yields nothing, admit files with multiple lexical clues or an exact symbol + * whose indexed import edge connects it to a multi-clue companion. + */ + private async rescueKeywordMatches( + query: string, + limit: number, + filters?: SearchFilters, + validation: SourceValidationContext = { sources: new Map() }, + sourceBudget?: SourceInspectionBudget, + retrievedCandidates?: FuseResult[] + ): Promise[]> { + const checkedSources = validation.sources; + if (!this.fuseIndex || /[|^$!=]/.test(query)) return []; + + const queryTerms = this.lexicalQueryTerms(query); + const terms = [...queryTerms.keys()]; + if (terms.length < 2) return []; + + const symbolAnchors = (query.match(/[a-zA-Z_][a-zA-Z0-9_]*/g) ?? []) + .filter((term) => /[a-z][A-Z]/.test(term) && term.length >= 8) + .map((term) => term.toLowerCase()); + + const candidateLimit = Math.min(MAX_CANDIDATE_WORK, Math.max(256, limit * 80)); + const budget = sourceBudget ?? { limit: candidateLimit, inspected: new Set() }; + const aliases = [...new Set([...queryTerms.values()].flat())]; + // Invert the existing suffix aliases conservatively: policy/policies share + // "polic", and archive/archiving share "archiv". These fragments only nominate; + // the unchanged exact lexical evidence below still decides admission. + const fragments = [...new Set(aliases.map((alias) => alias.replace(/[ey]$/, '')))]; + const lookupTerms = fragments.filter( + (fragment) => !fragments.some((other) => other !== fragment && fragment.includes(other)) + ); + // Keep original chunk ordinals, including ties, and nominate across every + // exact evidence zone. Filters apply before the bounded evidence window. + const rawCandidates = + retrievedCandidates ?? + this.chunks.flatMap((item, refIndex): FuseResult[] => { + if (!this.matchesSearchFilters(item, filters)) return []; + const fields = [ + item.content, + path.basename(item.relativePath), + ...this.indexedDeclarationNames(item), + ...(item.exports ?? []) + ]; + const matches = fields.some((field) => { + const text = field.toLowerCase(); + return ( + lookupTerms.some((term) => text.includes(term)) || + symbolAnchors.some((anchor) => text.includes(anchor)) + ); + }); + return matches ? [{ item, refIndex, score: 0 }] : []; + }); + const availableCandidates = rawCandidates.filter( + ({ item }) => checkedSources.get(path.resolve(this.rootPath, item.filePath)) !== null + ); + if (availableCandidates.length > candidateLimit) validation.candidateBudgetExhausted = true; + const candidates = availableCandidates.slice(0, candidateLimit); + const unavailableBefore = [...checkedSources.values()].filter( + (source) => source === null + ).length; + + const queryAliases = new Map( + terms.map((term) => [term, new Set(queryTerms.get(term)!)] as const) + ); + const evidence = candidates.map((result) => { + const { item } = result; + const bodyWords = this.lexicalWords(item.content); + const bodyLexemes = new Set(); + const nonBodyLexemes = this.lexicalEvidence( + [ + item.metadata?.componentName ?? '', + item.metadata?.symbolName ?? '', + path.basename(item.relativePath) + ].join(' ') + ); + const termFrequencies = new Map(terms.map((term) => [term, 0])); + for (const word of bodyWords) { + const wordAliases = this.lexicalAliases(word); + for (const alias of wordAliases) bodyLexemes.add(alias); + for (const term of terms) { + if (wordAliases.some((alias) => queryAliases.get(term)!.has(alias))) { + termFrequencies.set(term, termFrequencies.get(term)! + 1); + } + } + } + const lexemes = new Set([...bodyLexemes, ...nonBodyLexemes]); + const matching = new Set( + terms.filter((term) => queryTerms.get(term)!.some((alias) => lexemes.has(alias))) + ); + const density = + terms.reduce((total, term) => total + Math.sqrt(termFrequencies.get(term)!), 0) / + Math.sqrt(Math.max(1, bodyWords.length)); + // Export names remain useful lexical/graph clues, but can be re-exports; + // exact definition priority below requires own declaration metadata. + const indexedSymbols = [...this.indexedDeclarationNames(item), ...(item.exports ?? [])]; + const exactAnchor = symbolAnchors.some((symbol) => + indexedSymbols.some( + (indexedSymbol) => + typeof indexedSymbol === 'string' && indexedSymbol.toLowerCase() === symbol + ) + ); + const declarationLexemes = this.lexicalEvidence(indexedSymbols.filter(Boolean).join(' ')); + const declarationMatching = new Set( + terms.filter((term) => queryTerms.get(term)!.some((alias) => declarationLexemes.has(alias))) + ); + return { result, matching, declarationMatching, exactAnchor, density }; + }); + + const supportedCompanions = new Map( + evidence + .filter(({ matching }) => matching.size >= 2) + .map(({ result }) => [this.indexedFileKey(result.item.filePath), result.item] as const) + ); + const anchors = new Map( + evidence + .filter(({ exactAnchor }) => exactAnchor) + .map(({ result }) => [this.indexedFileKey(result.item.filePath), result.item] as const) + ); + const corroboratedFiles = new Set(); + const corroboratingTargets = new Map>(); + const corroboratingSources = new Map(); + // Visit each anchor's indexed adjacency once; do not rescan all chunks or + // infer target extensions from source text. + for (const [key, anchor] of anchors) { + for (const target of this.indexedImports.get(key)?.keys() ?? []) { + const companion = supportedCompanions.get(target); + if (!companion || target === key) continue; + const targets = corroboratingTargets.get(key) ?? new Set(); + targets.add(target); + corroboratingTargets.set(key, targets); + corroboratingSources.set(key, anchor); + corroboratingSources.set(target, companion); + } + } + // A stale endpoint cannot lend confidence to a current anchor, even when + // the companion would otherwise fall outside the returned result limit. + if (corroboratingSources.size > 0) { + await this.validateCandidateSources(corroboratingSources.values(), validation, budget); + // Only candidate endpoints enter the resolver. Re-read compiler options for + // each query: unchanged endpoint bytes do not prove unchanged alias routing. + const resolver = await InternalSourceResolver.create( + this.rootPath, + [...corroboratingSources.values()] + .filter( + (chunk) => checkedSources.get(path.resolve(this.rootPath, chunk.filePath)) != null + ) + .map((chunk) => path.resolve(this.rootPath, chunk.filePath)), + { compilerOptions: loadTsconfigCompilerOptions(this.rootPath) } + ); + for (const [key, targets] of corroboratingTargets) { + const anchor = corroboratingSources.get(key)!; + if (checkedSources.get(path.resolve(this.rootPath, anchor.filePath)) == null) continue; + for (const target of targets) { + const companion = corroboratingSources.get(target)!; + if (checkedSources.get(path.resolve(this.rootPath, companion.filePath)) == null) continue; + for (const specifier of this.indexedImports.get(key)?.get(target) ?? []) { + const current = await resolver.resolve( + path.resolve(this.rootPath, anchor.filePath), + specifier + ); + if (current.reason === 'outside-root') { + throw new IndexCorruptedError( + 'Import resolves outside the selected project. Full rebuild required.' + ); + } + if ( + current.status === 'resolved' && + current.path && + this.indexedFileKey(current.path) === target + ) { + corroboratedFiles.add(key); + break; + } + } + if (corroboratedFiles.has(key)) break; + } + } + } + + const fileEvidence = new Map>(); + for (const { result, matching } of evidence) { + const key = this.indexedFileKey(result.item.filePath); + const termsForFile = fileEvidence.get(key) ?? new Set(); + for (const term of matching) termsForFile.add(term); + fileEvidence.set(key, termsForFile); + } + + const seenFiles = new Set(); + const nominated = evidence + .filter(({ result, exactAnchor }) => { + const key = this.indexedFileKey(result.item.filePath); + return ( + (exactAnchor && corroboratedFiles.has(key)) || (fileEvidence.get(key)?.size ?? 0) >= 2 + ); + }) + .sort((a, b) => { + const keyA = this.indexedFileKey(a.result.item.filePath); + const keyB = this.indexedFileKey(b.result.item.filePath); + const score = (entry: typeof a, key: string) => + ((fileEvidence.get(key)?.size ?? 0) + entry.declarationMatching.size) * 8 + + entry.matching.size * 3 + + (entry.exactAnchor ? 8 : 0) - + (this.isTestFile(entry.result.item.filePath) ? 2 : 0) - + (entry.result.score ?? 0); + const primaryDifference = score(b, keyB) - score(a, keyA); + if (primaryDifference !== 0) return primaryDifference; + return b.density - a.density || a.result.refIndex - b.result.refIndex; + }) + .filter(({ result }) => { + const key = this.indexedFileKey(result.item.filePath); + if (seenFiles.has(key)) return false; + seenFiles.add(key); + return true; + }) + .map(({ result, declarationMatching }) => ({ + ...result, + // Equal document zones: distinct source clues nominate the file; + // indexed declaration clues distinguish its own work from mentions in + // imports or configuration. This remains bounded to [0, 1]. + score: + 1 - + ((fileEvidence.get(this.indexedFileKey(result.item.filePath))?.size ?? 0) + + declarationMatching.size) / + (terms.length * 2) + })); + const current = await this.selectCurrentCandidates( + nominated, + limit, + (result) => result.item, + validation, + budget + ); + const unavailableAfter = [...checkedSources.values()].filter( + (source) => source === null + ).length; + if ( + current.length < limit && + unavailableAfter > unavailableBefore && + candidates.length === candidateLimit + ) { + // Refill the same bounded evidence window after excluding every chunk of + // a rejected source. Fresh-prefix queries retain the original single pass. + return this.rescueKeywordMatches(query, limit, filters, validation, budget, rawCandidates); + } + return current; + } + /** * Classify query intent based on heuristic patterns */ @@ -708,18 +1305,23 @@ export class CodebaseSearcher { }) .sort((a, b) => b.score - a.score); - // SEARCH-01: Definition-first boost for EXACT_NAME intent - // Boost results where symbolName matches query (case-insensitive) - if (intent === 'EXACT_NAME') { - const queryNormalized = query.toLowerCase(); + // Definition first for explicit identifiers, regardless of which + // analyzer populated the existing declaration metadata. Coarse component or + // layer boosts must not move a fuzzy same-suffix mention above that match. + if ( + intent === 'EXACT_NAME' || + scoredResults.some((result) => this.hasExactQueryDeclaration(result, query)) + ) { for (const result of scoredResults) { - const symbolName = result.metadata?.symbolName; - if (symbolName && symbolName.toLowerCase() === queryNormalized) { + if (this.hasExactQueryDeclaration(result, query)) { result.score *= 1.15; // +15% boost for definition } } - // Re-sort after boost - scoredResults.sort((a, b) => b.score - a.score); + scoredResults.sort( + (a, b) => + Number(this.hasExactQueryDeclaration(b, query)) - + Number(this.hasExactQueryDeclaration(a, query)) || b.score - a.score + ); } // File-level deduplication @@ -733,35 +1335,9 @@ export class CodebaseSearcher { if (deduped.length >= limit) break; } - // SEARCH-01: Symbol-level deduplication - // Within each symbol group (symbolPath), keep only the highest-scoring chunk - const seenSymbols = new Map(); - const symbolDeduped: SearchResult[] = []; - for (const result of deduped) { - const symbolPath = result.metadata?.symbolPath; - if (!symbolPath) { - // No symbol info, keep as-is - symbolDeduped.push(result); - continue; - } - - const symbolPathKey = Array.isArray(symbolPath) ? symbolPath.join('.') : String(symbolPath); - const existing = seenSymbols.get(symbolPathKey); - if (!existing || result.score > existing.score) { - if (existing) { - // Replace lower-scoring version - const idx = symbolDeduped.indexOf(existing); - if (idx >= 0) { - symbolDeduped[idx] = result; - } - } else { - symbolDeduped.push(result); - } - seenSymbols.set(symbolPathKey, result); - } - } - - const finalResults = symbolDeduped; + // File-level deduplication above already keeps only one result per file. + // Keep same-named symbols in different files as distinct search results. + const finalResults = deduped; if ( isNonTestQuery && @@ -838,7 +1414,8 @@ export class CodebaseSearcher { useSemanticSearch: boolean, useKeywordSearch: boolean, semanticWeight: number, - keywordWeight: number + keywordWeight: number, + validation: SourceValidationContext ): Promise<{ semantic: Map }>; keyword: Map }>; @@ -856,7 +1433,12 @@ export class CodebaseSearcher { if (useSemanticSearch && this.embeddingProvider && this.storageProvider) { try { for (const variant of queryVariants) { - const vectorResults = await this.semanticSearch(variant.query, candidateLimit, filters); + const vectorResults = await this.semanticSearch( + variant.query, + candidateLimit, + filters, + validation + ); // Assign ranks based on retrieval order (0-indexed) vectorResults.forEach((result, index) => { @@ -886,13 +1468,23 @@ export class CodebaseSearcher { if (useKeywordSearch && this.fuseIndex) { try { for (const variant of queryVariants) { - const keywordResults = await this.keywordSearch(variant.query, candidateLimit, filters); + const keywordResults = await this.keywordSearch( + variant.query, + candidateLimit, + filters, + validation + ); + // Grounded multi-clue rescue needs its evidence distinction retained + // through fusion. Single/exact identifier searches keep their existing + // rank semantics and confidence calibration. + const weightLexicalEvidence = this.lexicalQueryTerms(variant.query).size >= 2; // Assign ranks based on retrieval order (0-indexed) keywordResults.forEach((result, index) => { const id = result.chunk.id; const rank = index; // 0-indexed rank - const weight = keywordWeight * variant.weight; + const weight = + keywordWeight * variant.weight * (weightLexicalEvidence ? result.score : 1); const existing = keywordRanks.get(id); if (existing) { @@ -906,6 +1498,9 @@ export class CodebaseSearcher { }); } } catch (error) { + if (error instanceof IndexCorruptedError) { + throw error; // Preserve the caller's existing auto-heal path. + } console.warn('Keyword search failed:', error); } } @@ -919,6 +1514,15 @@ export class CodebaseSearcher { filters?: SearchFilters, options: SearchOptions = DEFAULT_SEARCH_OPTIONS ): Promise { + return (await this.searchWithCoverage(query, limit, filters, options)).results; + } + + async searchWithCoverage( + query: string, + limit: number = 5, + filters?: SearchFilters, + options: SearchOptions = DEFAULT_SEARCH_OPTIONS + ): Promise { if (!this.initialized) { await this.initialize(); } @@ -927,6 +1531,10 @@ export class CodebaseSearcher { ...DEFAULT_SEARCH_OPTIONS, ...options }; + if (this.indexMeta?.artifacts.vectorDb.embeddingEnabled === false) { + merged.useSemanticSearch = false; + merged.enableReranker = false; + } const { useSemanticSearch, useKeywordSearch, @@ -945,6 +1553,8 @@ export class CodebaseSearcher { const candidateLimit = Math.max(limit * 2, candidateFloor || 30); const primaryVariants = this.buildQueryVariants(query, enableQueryExpansion ? 1 : 0); + const validation: SourceValidationContext = { sources: new Map() }; + const checkedSources = validation.sources; const primaryMatches = await this.collectHybridMatches( primaryVariants, candidateLimit, @@ -952,8 +1562,28 @@ export class CodebaseSearcher { Boolean(useSemanticSearch), Boolean(useKeywordSearch), finalSemanticWeight, - finalKeywordWeight + finalKeywordWeight, + validation + ); + + const keepCurrentMatches = (matches: typeof primaryMatches): void => { + for (const channel of [matches.semantic, matches.keyword]) { + for (const [id, { chunk }] of channel) { + if (checkedSources.get(path.resolve(this.rootPath, chunk.filePath)) == null) + channel.delete(id); + } + } + }; + + // Validate every nominated source before scoring and limiting so stale top + // matches cannot hide a valid lower-ranked current file. + await this.validateCandidateSources( + [...primaryMatches.semantic.values(), ...primaryMatches.keyword.values()].map( + (match) => match.chunk + ), + validation ); + keepCurrentMatches(primaryMatches); const primaryTotalWeight = primaryVariants.reduce((sum, v) => sum + v.weight, 0) * @@ -984,9 +1614,18 @@ export class CodebaseSearcher { Boolean(useSemanticSearch), Boolean(useKeywordSearch), finalSemanticWeight, - finalKeywordWeight + finalKeywordWeight, + validation ); + await this.validateCandidateSources( + [...rescueMatches.semantic.values(), ...rescueMatches.keyword.values()].map( + (match) => match.chunk + ), + validation + ); + keepCurrentMatches(rescueMatches); + const rescueVariantWeights = rescueVariants.map((_, i) => (i === 0 ? 1 : 0.8)); const rescueTotalWeight = rescueVariantWeights.reduce((sum, w) => sum + w, 0) * @@ -1015,7 +1654,41 @@ export class CodebaseSearcher { } } - return bestResults; + const results = bestResults.map((result) => { + const lines = checkedSources.get(path.resolve(this.rootPath, result.filePath)); + if (lines == null) { + throw new IndexCorruptedError('Result source was not validated. Full rebuild required.'); + } + this.validateSourceRange(result, lines); + return { + ...result, + snippet: this.generateSnippet(lines.slice(result.startLine - 1, result.endLine).join('\n')) + }; + }); + const skippedFiles = [...checkedSources.values()].filter((source) => source === null).length; + return { + results, + sourceCoverage: { + status: + skippedFiles === 0 && + !validation.candidateBudgetExhausted && + !validation.candidateRetrievalFailed + ? 'current' + : results.length > 0 + ? 'partial' + : 'no_fresh_results', + checkedFiles: checkedSources.size, + skippedFiles, + ...(validation.candidateBudgetExhausted && { candidateBudgetExhausted: true }), + ...(validation.candidateRetrievalFailed && { candidateRetrievalFailed: true }), + ...((skippedFiles > 0 || + validation.candidateBudgetExhausted || + validation.candidateRetrievalFailed) && { + refreshAction: + 'Run refresh_index with incrementalOnly=true to refresh changed or unavailable sources, then retry.' + }) + } + }; } private generateSummary(chunk: CodeChunk): string { @@ -1081,7 +1754,8 @@ export class CodebaseSearcher { private async semanticSearch( query: string, limit: number, - filters?: SearchFilters + filters?: SearchFilters, + validation: SourceValidationContext = { sources: new Map() } ): Promise<{ chunk: CodeChunk; score: number }[]> { if (!this.embeddingProvider || !this.storageProvider) { return []; @@ -1089,18 +1763,71 @@ export class CodebaseSearcher { const queryVector = await this.embeddingProvider.embed(query); - const results = await this.storageProvider.search(queryVector, limit, filters); - - return results.map((r) => ({ - chunk: r.chunk, - score: r.score - })); + const budget: SourceInspectionBudget = { + limit: Math.min(limit, MAX_CANDIDATE_WORK), + inspected: new Set() + }; + const retainedCount = this.chunks.length; + let requested = Math.min(limit, MAX_CANDIDATE_WORK); + let retrievedWork = 0; + let retained: Array<{ chunk: CodeChunk; score: number }> | undefined; + for (;;) { + const results = await this.storageProvider + .search(queryVector, requested, filters) + .catch((error: unknown) => { + if (error instanceof IndexCorruptedError || retained === undefined) throw error; + validation.candidateRetrievalFailed = true; + console.warn('Semantic refill failed; retaining validated current candidates:', error); + return null; + }); + if (results === null) return retained ?? []; + retrievedWork += results.length; + const current = await this.selectCurrentCandidates( + results, + limit, + (result) => result.chunk, + validation, + budget + ); + retained = current.map(({ chunk, score }) => ({ chunk, score })); + if ( + current.length >= limit || + results.length < requested || + (retainedCount > 0 && requested >= retainedCount) + ) { + return retained; + } + const staleSources = new Set( + [...validation.sources].filter(([, source]) => source === null).map(([file]) => file) + ); + if (staleSources.size === 0) { + if (requested < limit && results.length === requested) + validation.candidateBudgetExhausted = true; + return retained; + } + const rejectedChunks = this.chunks.filter((chunk) => + staleSources.has(path.resolve(this.rootPath, chunk.filePath)) + ).length; + // Widen by known rejected rows, never by an arbitrary multiplier. Sum of + // retrieved rows is bounded, and the query vector is reused for all retries. + const next = Math.min( + retainedCount || requested, + limit + rejectedChunks, + MAX_CANDIDATE_WORK - retrievedWork + ); + if (next <= requested || budget.inspected.size >= budget.limit) { + validation.candidateBudgetExhausted = true; + return retained; + } + requested = next; + } } private async keywordSearch( query: string, limit: number, - filters?: SearchFilters + filters?: SearchFilters, + validation: SourceValidationContext = { sources: new Map() } ): Promise<{ chunk: CodeChunk; score: number }[]> { if (!this.fuseIndex || this.chunks.length === 0) { return []; @@ -1109,68 +1836,63 @@ export class CodebaseSearcher { let fuseResults = this.fuseIndex.search(query); if (filters) { - fuseResults = fuseResults.filter((r) => { - const chunk = r.item; - - if (filters.componentType && chunk.componentType !== filters.componentType) { - return false; - } - if (filters.layer && chunk.layer !== filters.layer) { - return false; - } - if (filters.framework && chunk.framework !== filters.framework) { - return false; - } - if (filters.language && chunk.language !== filters.language) { - return false; - } - if (filters.tags && filters.tags.length > 0) { - const chunkTags = chunk.tags || []; - if (!filters.tags.some((tag) => chunkTags.includes(tag))) { - return false; - } - } - - return true; - }); + fuseResults = fuseResults.filter(({ item }) => this.matchesSearchFilters(item, filters)); } - return fuseResults.slice(0, limit).map((r) => { - const chunk = r.item; - let score = 1 - (r.score || 0); + if (fuseResults.length === 0 || (fuseResults[0].score ?? 0) > KEYWORD_FUSE_OPTIONS.threshold!) { + const rescue = await this.rescueKeywordMatches(query, limit, filters, validation); + const seen = new Set(rescue.map(({ item }) => item.id)); + fuseResults = [...rescue, ...fuseResults.filter(({ item }) => !seen.has(item.id))].sort( + (a, b) => (a.score ?? 0) - (b.score ?? 0) + ); + } + const current = await this.selectCurrentCandidates( + fuseResults, + limit, + (result) => result.item, + validation + ); + return current + .map((r) => { + const chunk = r.item; + let score = 1 - (r.score || 0); - // Boost exact matches on class name or file path - const queryLower = query.toLowerCase(); - const fileName = path.basename(chunk.filePath).toLowerCase(); - const relativePathLower = chunk.relativePath.toLowerCase(); - const componentName = chunk.metadata?.componentName?.toLowerCase() || ''; + // Boost exact matches on class name or file path + const queryLower = query.toLowerCase(); + const fileName = path.basename(chunk.filePath).toLowerCase(); + const relativePathLower = chunk.relativePath.toLowerCase(); - // Exact class name match - if (componentName && queryLower === componentName) { - score = Math.min(1.0, score + 0.3); - } + // Use the same analyzer-neutral declaration evidence as final ranking. + if (this.hasExactQueryDeclaration(chunk, query)) { + score = Math.min(1.0, score + 0.3); + } - // Exact file name match - if ( - fileName === queryLower || - fileName.replace(/\.ts$/, '') === queryLower.replace(/\.ts$/, '') - ) { - score = Math.min(1.0, score + 0.2); - } + // Exact file name match + if ( + fileName === queryLower || + fileName.replace(/\.ts$/, '') === queryLower.replace(/\.ts$/, '') + ) { + score = Math.min(1.0, score + 0.2); + } - // File path contains query - if ( - chunk.filePath.toLowerCase().includes(queryLower) || - relativePathLower.includes(queryLower) - ) { - score = Math.min(1.0, score + 0.1); - } + // File path contains query + if ( + chunk.filePath.toLowerCase().includes(queryLower) || + relativePathLower.includes(queryLower) + ) { + score = Math.min(1.0, score + 0.1); + } - return { - chunk, - score - }; - }); + return { + chunk, + score + }; + }) + .sort( + (a, b) => + Number(this.hasExactQueryDeclaration(b.chunk, query)) - + Number(this.hasExactQueryDeclaration(a.chunk, query)) || b.score - a.score + ); } private generateRelevanceReason(chunk: CodeChunk, query: string): string { diff --git a/src/core/symbol-references.ts b/src/core/symbol-references.ts index f4c1986..150c49e 100644 --- a/src/core/symbol-references.ts +++ b/src/core/symbol-references.ts @@ -2,13 +2,18 @@ import { promises as fs } from 'fs'; import path from 'path'; import { CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME } from '../constants/codebase-context.js'; import { IndexCorruptedError } from '../errors/index.js'; +import { readIndexMeta, validateKeywordIndexArtifact } from './index-meta.js'; +import { readReferenceManifest } from './manifest.js'; import type { UsageLocation } from '../types/index.js'; import { detectLanguage } from '../utils/language-detection.js'; -import { findIdentifierOccurrences } from '../utils/tree-sitter.js'; +import { isPathWithin } from '../utils/project-discovery.js'; +import { + hasIdentifierCandidate, + findIdentifierOccurrences, + findLexicalIdentifierOccurrences +} from '../utils/tree-sitter.js'; interface IndexedChunk { - content?: unknown; - startLine?: unknown; relativePath?: unknown; filePath?: unknown; } @@ -22,8 +27,16 @@ interface SymbolReferencesSuccess { symbol: string; usageCount: number; usages: SymbolUsage[]; - confidence: 'syntactic'; + confidence: 'syntactic' | 'lexical'; isComplete: boolean; + coverage: { + candidateFiles: number; + sourcesRead: number; + filesWithoutText: number; + filesUnavailable: number; + parserCalls: number; + lexicalFallbackFiles: number; + }; } interface SymbolReferencesError { @@ -33,33 +46,16 @@ interface SymbolReferencesError { export type SymbolReferencesResult = SymbolReferencesSuccess | SymbolReferencesError; -function escapeRegex(value: string): string { - return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); -} +type SourceReadResult = + | { status: 'source'; source: string; realSourcePath: string } + | { status: 'without-text' } + | { status: 'without-candidate' } + | { status: 'unavailable' } + | SymbolReferencesError; -function getUsageFile(rootPath: string, chunk: IndexedChunk): string { - if (typeof chunk.relativePath === 'string' && chunk.relativePath.trim()) { - return chunk.relativePath.replace(/\\/g, '/'); - } - - if (typeof chunk.filePath === 'string' && chunk.filePath.trim()) { - const relativePath = path.relative(rootPath, chunk.filePath); - if (!relativePath || relativePath.startsWith('..')) { - return path.basename(chunk.filePath); - } - return relativePath.replace(/\\/g, '/'); - } - - return 'unknown'; -} - -function buildPreview(content: string, lineOffset: number): string { - const lines = content.split('\n'); - const start = Math.max(0, lineOffset - 1); - const end = Math.min(lines.length, lineOffset + 2); - const previewLines = lines.slice(start, end); - return previewLines.join('\n').trim(); -} +// Bound both outstanding source I/O and buffered source contents. Parsing remains +// sequential because Tree-sitter caches a mutable parser for each language. +const SOURCE_READ_BATCH_SIZE = 4; function buildPreviewFromFileLines(lines: string[], line: number): string { const start = Math.max(0, line - 2); @@ -67,12 +63,47 @@ function buildPreviewFromFileLines(lines: string[], line: number): string { return lines.slice(start, end).join('\n').trim(); } +function isOrdinarySourceAvailabilityError(error: unknown): boolean { + if (error === null || typeof error !== 'object' || !('code' in error)) return false; + const code = error.code; + return code === 'ENOENT' || code === 'ENOTDIR' || code === 'EACCES' || code === 'EPERM'; +} + +async function readReferenceSource( + relativePath: string, + absolutePath: string, + realRootPath: string, + symbol: string +): Promise { + try { + const realSourcePath = await fs.realpath(absolutePath); + if (!isPathWithin(realRootPath, realSourcePath)) { + return { + status: 'error', + message: `Indexed source resolves outside the selected project (${relativePath}). Refresh the index before trusting reference results.` + }; + } + if (!(await fs.stat(realSourcePath)).isFile()) throw new Error('not a file'); + const source = (await fs.readFile(realSourcePath, 'utf-8')).replace(/\r\n/g, '\n'); + if (!source.includes(symbol)) return { status: 'without-text' }; + // The boundary screen only suppresses simple ASCII identifier interiors; + // other spellings and non-ASCII adjacency stay on the existing parser path. + return hasIdentifierCandidate(source, symbol) + ? { status: 'source', source, realSourcePath } + : { status: 'without-candidate' }; + } catch (error) { + if (isOrdinarySourceAvailabilityError(error)) return { status: 'unavailable' }; + return { + status: 'error', + message: `Indexed source is missing or unreadable (${relativePath}). Refresh the index before trusting reference results.` + }; + } +} + function resolveAbsoluteChunkPath(rootPath: string, chunk: IndexedChunk): string | null { const resolvedRoot = path.resolve(rootPath); const isWithinRoot = (candidate: string): boolean => { - const resolvedCandidate = path.resolve(candidate); - const relative = path.relative(resolvedRoot, resolvedCandidate); - return Boolean(relative) && !relative.startsWith('..') && !path.isAbsolute(relative); + return isPathWithin(resolvedRoot, candidate); }; if (typeof chunk.filePath === 'string' && chunk.filePath.trim()) { @@ -92,15 +123,6 @@ function resolveAbsoluteChunkPath(rootPath: string, chunk: IndexedChunk): string return null; } -async function fileExists(targetPath: string): Promise { - try { - const stat = await fs.stat(targetPath); - return stat.isFile(); - } catch { - return false; - } -} - export async function findSymbolReferences( rootPath: string, symbol: string, @@ -117,11 +139,13 @@ export async function findSymbolReferences( } const indexPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const meta = await readIndexMeta(rootPath); + const manifest = await readReferenceManifest(rootPath, meta); - let chunksRaw: unknown; + let keywordIndexRaw: unknown; try { const content = await fs.readFile(indexPath, 'utf-8'); - chunksRaw = JSON.parse(content); + keywordIndexRaw = JSON.parse(content); } catch (error) { throw new IndexCorruptedError( `Keyword index missing or unreadable (rebuild required): ${ @@ -130,126 +154,141 @@ export async function findSymbolReferences( ); } - if (Array.isArray(chunksRaw)) { + if (Array.isArray(keywordIndexRaw)) { throw new IndexCorruptedError( 'Legacy keyword index format detected (missing header). Rebuild required.' ); } - const chunks = - chunksRaw !== null && - typeof chunksRaw === 'object' && - 'chunks' in chunksRaw && - Array.isArray(chunksRaw.chunks) - ? (chunksRaw.chunks as unknown[]) - : null; - - if (!chunks) { - throw new IndexCorruptedError('Keyword index corrupted: expected { header, chunks }'); - } + const { chunks } = validateKeywordIndexArtifact(keywordIndexRaw, meta); const usages: SymbolUsage[] = []; let usageCount = 0; - - const escapedSymbol = escapeRegex(normalizedSymbol); - const prefilter = new RegExp(`\\b${escapedSymbol}\\b`); - const matcher = new RegExp(`\\b${escapedSymbol}\\b`, 'g'); - - // Prefilter candidate files from the keyword index. We do not trust chunk contents for - // exact reference counting when Tree-sitter is available; chunks only guide which files to scan. - const chunksByFile = new Map< - string, - { relPath: string; absPath: string | null; chunks: IndexedChunk[] } - >(); + let sourcesRead = 0; + let filesWithoutText = 0; + let filesUnavailable = 0; + let parserCalls = 0; + let lexicalFallbackFiles = 0; + + // The manifest nominates every analyzed source; capped keyword chunks cannot define + // the reference scope. Current source bytes determine matches, locations, and previews. + const candidateFiles = new Map(); + + for (const relativePath of Object.keys(manifest.files)) { + const absPath = resolveAbsoluteChunkPath(rootPath, { relativePath }); + if (!absPath) { + throw new IndexCorruptedError( + `Reference manifest contains an invalid source path (${relativePath}). Rebuild required.` + ); + } + const relPath = path.relative(rootPath, absPath).replace(/\\/g, '/'); + candidateFiles.set(relPath, absPath); + } for (const chunkRaw of chunks) { const chunk = chunkRaw as IndexedChunk; - if (typeof chunk.content !== 'string') continue; - if (!prefilter.test(chunk.content)) continue; - - const relPath = getUsageFile(rootPath, chunk); const absPath = resolveAbsoluteChunkPath(rootPath, chunk); - - const entry = chunksByFile.get(relPath); - if (entry) { - entry.chunks.push(chunk); - // Prefer a real absolute path when available - if (!entry.absPath && absPath) { - entry.absPath = absPath; - } - } else { - chunksByFile.set(relPath, { relPath, absPath, chunks: [chunk] }); + if (!absPath) { + return { + status: 'error', + message: + 'Indexed source location is invalid. Refresh the index before trusting reference results.' + }; + } + const relPath = path.relative(rootPath, absPath).replace(/\\/g, '/'); + if (!Object.hasOwn(manifest.files, relPath)) { + throw new IndexCorruptedError( + `Keyword index source is absent from the build-bound manifest (${relPath}). Rebuild required.` + ); } } - for (const entry of chunksByFile.values()) { - const relPath = entry.relPath; - const absPath = entry.absPath; - - // Preferred: Tree-sitter identifier walk on the real file content. - if (absPath && (await fileExists(absPath))) { - try { - const raw = await fs.readFile(absPath, 'utf-8'); - const content = raw.replace(/\r\n/g, '\n'); - const language = detectLanguage(absPath); - const occurrences = await findIdentifierOccurrences(content, language, normalizedSymbol); - - if (occurrences) { - usageCount += occurrences.length; - - if (usages.length < normalizedLimit && occurrences.length > 0) { - const lines = content.split('\n'); - for (const occ of occurrences) { - if (usages.length >= normalizedLimit) break; - usages.push({ - file: relPath, - line: occ.line, - preview: buildPreviewFromFileLines(lines, occ.line) - }); - } - } - - continue; - } - } catch { - // Fall through to chunk-regex fallback (missing grammar, parse failure, etc.) + let realRootPath: string; + try { + realRootPath = await fs.realpath(rootPath); + } catch { + return { status: 'error', message: 'Selected project path is missing or unreadable.' }; + } + const sourceFiles = [...candidateFiles]; + for (let offset = 0; offset < sourceFiles.length; offset += SOURCE_READ_BATCH_SIZE) { + const batch = sourceFiles.slice(offset, offset + SOURCE_READ_BATCH_SIZE); + // Reads return values even on failure. Settle every started read before a fatal + // return, then consume in manifest order so snippets and totals stay deterministic. + const sources = await Promise.all( + batch.map(([relativePath, absolutePath]) => + readReferenceSource(relativePath, absolutePath, realRootPath, normalizedSymbol) + ) + ); + const fatalSource = sources.find((source) => source.status === 'error'); + if (fatalSource?.status === 'error') return fatalSource; + + for (let index = 0; index < batch.length; index += 1) { + const [relativePath] = batch[index]; + const result = sources[index]; + if (result.status === 'error') return result; + if (result.status === 'unavailable') { + filesUnavailable += 1; + continue; } - } - - // Fallback: regex scan inside the matched chunks (legacy behavior). - for (const chunk of entry.chunks) { - if (typeof chunk.content !== 'string') continue; - - const chunkContent = chunk.content; - const startLine = typeof chunk.startLine === 'number' ? chunk.startLine : 1; - matcher.lastIndex = 0; - - let match: RegExpExecArray | null; - while ((match = matcher.exec(chunkContent)) !== null) { - usageCount += 1; - - if (usages.length >= normalizedLimit) { - continue; + sourcesRead += 1; + if (result.status === 'without-text') { + filesWithoutText += 1; + continue; + } + // Raw query text was present but only inside longer ASCII identifiers; + // keep this distinct from filesWithoutText coverage. + if (result.status === 'without-candidate') continue; + const { source, realSourcePath } = result; + const lines = source.split('\n'); + parserCalls += 1; + const occurrences = await findIdentifierOccurrences( + source, + detectLanguage(realSourcePath), + normalizedSymbol + ).catch(() => null); + + if (occurrences) { + usageCount += occurrences.length; + for (const occurrence of occurrences) { + if (usages.length >= normalizedLimit) break; + usages.push({ + file: relativePath, + line: occurrence.line, + preview: buildPreviewFromFileLines(lines, occurrence.line) + }); } + continue; + } - const prefix = chunkContent.slice(0, match.index); - const lineOffset = prefix.split('\n').length - 1; - + lexicalFallbackFiles += 1; + const lexicalOccurrences = findLexicalIdentifierOccurrences(source, normalizedSymbol); + usageCount += lexicalOccurrences.length; + for (const occurrence of lexicalOccurrences) { + if (usages.length >= normalizedLimit) break; usages.push({ - file: relPath, - line: startLine + lineOffset, - preview: buildPreview(chunkContent, lineOffset) + file: relativePath, + line: occurrence.line, + preview: buildPreviewFromFileLines(lines, occurrence.line) }); } } } + const coverageComplete = filesUnavailable === 0; return { status: 'success', symbol: normalizedSymbol, usageCount, usages, - confidence: 'syntactic', - isComplete: usageCount < normalizedLimit + confidence: lexicalFallbackFiles === 0 ? 'syntactic' : 'lexical', + isComplete: coverageComplete && usageCount < normalizedLimit, + coverage: { + candidateFiles: candidateFiles.size, + sourcesRead, + filesWithoutText, + filesUnavailable, + parserCalls, + lexicalFallbackFiles + } }; } diff --git a/src/embeddings/index.ts b/src/embeddings/index.ts index aabb3fa..f8c1718 100644 --- a/src/embeddings/index.ts +++ b/src/embeddings/index.ts @@ -9,6 +9,14 @@ import { parseEmbeddingProviderName } from './types.js'; import { TransformersEmbeddingProvider, MODEL_CONFIGS } from './transformers.js'; +import { assertOptionalDependency } from '../utils/optional-dependency.js'; + +export function assertEmbeddingRuntimeAvailable(config: Partial = {}): void { + const provider = config.provider ?? DEFAULT_EMBEDDING_CONFIG.provider; + if (provider !== 'openai' && provider !== 'custom') { + assertOptionalDependency('@huggingface/transformers', '^4.2.0'); + } +} /** * Returns expected embedding dimensions for a given config without initializing any provider. @@ -32,6 +40,7 @@ let cachedProviderType: string | null = null; export async function getEmbeddingProvider( config: Partial = {} ): Promise { + assertEmbeddingRuntimeAvailable(config); const mergedConfig = { ...DEFAULT_EMBEDDING_CONFIG, ...config }; const providerKey = `${mergedConfig.provider}:${mergedConfig.model}`; diff --git a/src/embeddings/transformers.ts b/src/embeddings/transformers.ts index 7fc62e8..bb358d1 100644 --- a/src/embeddings/transformers.ts +++ b/src/embeddings/transformers.ts @@ -1,6 +1,7 @@ import { EmbeddingProvider, DEFAULT_MODEL } from './types.js'; import type { FeatureExtractionPipeline } from '@huggingface/transformers'; import os from 'os'; +import { assertOptionalDependency } from '../utils/optional-dependency.js'; /** * Returns the number of ONNX intra-op threads to use. @@ -56,6 +57,8 @@ export class TransformersEmbeddingProvider implements EmbeddingProvider { if (this.ready) return; if (this.initPromise) return this.initPromise; + assertOptionalDependency('@huggingface/transformers', '^4.2.0'); + this.initPromise = this._initialize(); return this.initPromise; } diff --git a/src/errors/index.ts b/src/errors/index.ts index 96a9a30..7f8afe1 100644 --- a/src/errors/index.ts +++ b/src/errors/index.ts @@ -8,3 +8,13 @@ export class IndexCorruptedError extends Error { this.name = 'IndexCorruptedError'; } } + +/** Missing opt-in runtime, distinct from damaged index data. */ +export class OptionalDependencyError extends Error { + constructor(packageName: string, versionRange: string) { + super( + `Semantic mode needs ${packageName}. In the environment where CBC is installed, run npm install "${packageName}@${versionRange}", then retry.` + ); + this.name = 'OptionalDependencyError'; + } +} diff --git a/src/index.ts b/src/index.ts index ab539b8..0aeff3d 100644 --- a/src/index.ts +++ b/src/index.ts @@ -5,7 +5,7 @@ * Provides codebase indexing and semantic search capabilities */ -import { promises as fs } from 'fs'; +import { promises as fs, realpathSync } from 'fs'; import path from 'path'; import { fileURLToPath } from 'url'; @@ -47,6 +47,7 @@ import { } from './utils/project-discovery.js'; import { readIndexMeta, validateIndexArtifacts } from './core/index-meta.js'; import { TOOLS, dispatchTool, type ToolContext, type ToolResponse } from './tools/index.js'; +import { finalizeRelationshipPayload } from './tools/get-code-relationships.js'; import { finalizeSearchPayloadText } from './tools/search-payload-budget.js'; import type { ProjectDescriptor, ToolPaths } from './tools/types.js'; import { @@ -57,6 +58,8 @@ import { makeLegacyPaths, normalizeRootKey, removeProject, + stopProjectWatcher, + stopAllProjectWatchers, type ProjectRuntimeOverrides, type ProjectState } from './project-state.js'; @@ -81,7 +84,7 @@ let mcpRuntimeReady = false; let mcpRuntimePromise: Promise | undefined; // Flags that are NOT project paths — skip them when resolving the bootstrap root. -const KNOWN_FLAGS = new Set(['--http', '--port', '--help']); +const KNOWN_FLAGS = new Set(['--http', '--port', '--help', '--embeddings', '--no-embeddings']); const CLI_SUBCOMMANDS = [ 'memory', 'search', @@ -96,16 +99,41 @@ const CLI_SUBCOMMANDS = [ 'map' ]; -// Check if this module is the entry point. +function canonicalEntrypointPath(filePath: string | undefined): string | undefined { + if (!filePath) return undefined; + try { + const canonicalPath = path.normalize(realpathSync(filePath)); + return process.platform === 'win32' ? canonicalPath.toLowerCase() : canonicalPath; + } catch { + return undefined; + } +} + +// Resolve symlink launchers while keeping imports and unrelated index files inert. +const moduleEntrypointPath = canonicalEntrypointPath(fileURLToPath(import.meta.url)); +const processEntrypointPath = canonicalEntrypointPath(process.argv[1]); const isDirectRun = - process.argv[1]?.replace(/\\/g, '/').endsWith('index.js') || - process.argv[1]?.replace(/\\/g, '/').endsWith('index.ts'); + moduleEntrypointPath !== undefined && moduleEntrypointPath === processEntrypointPath; const directSubcommand = process.argv[2]; const isDirectCliSubcommand = isDirectRun && typeof directSubcommand === 'string' && (CLI_SUBCOMMANDS.includes(directSubcommand) || directSubcommand === '--help'); +if ( + !isDirectCliSubcommand && + process.argv.includes('--embeddings') && + process.argv.includes('--no-embeddings') +) { + console.error('Choose either --embeddings or --no-embeddings, not both.'); + process.exit(1); +} +const bootstrapSkipEmbedding = process.argv.includes('--no-embeddings') + ? true + : process.argv.includes('--embeddings') + ? false + : undefined; + // Resolve optional bootstrap root with validation handled later in main(). function resolveRootPath(): string | undefined { const envPath = process.env.CODEBASE_ROOT; @@ -154,9 +182,7 @@ const stdioIdleTimeoutEnv = Number.parseInt( 10 ); const STDIO_IDLE_TIMEOUT_MS = - Number.isFinite(stdioIdleTimeoutEnv) && stdioIdleTimeoutEnv >= 0 - ? stdioIdleTimeoutEnv - : 10 * 60 * 1000; + Number.isFinite(stdioIdleTimeoutEnv) && stdioIdleTimeoutEnv >= 0 ? stdioIdleTimeoutEnv : 0; let noteSessionActivity: () => void = () => undefined; let beginTrackedSessionWork: () => void = () => undefined; let endTrackedSessionWork: () => void = () => undefined; @@ -207,7 +233,14 @@ type ProjectResolution = | { ok: true; project: ProjectState } | { ok: false; response: ToolResponse }; -function finalizeJsonTextPayload(payload: Record): string { +export function finalizeJsonTextPayload( + payload: Record, + relationshipResponse = false +): string { + if (relationshipResponse) { + return finalizeRelationshipPayload(payload); + } + const mode = typeof payload.budget === 'object' && payload.budget !== null && @@ -430,7 +463,10 @@ function setActiveProject(rootPath: string): void { rememberProjectPath(resolvedRootPath); } -function syncKnownRoots(rootEntries: Array<{ rootPath: string; label?: string }>): void { +async function syncKnownRoots( + rootEntries: Array<{ rootPath: string; label?: string }> +): Promise { + const watcherClosures: Promise[] = []; const nextRoots = new Map(); const normalizedRoots = rootEntries.length > 0 ? rootEntries : primaryRootPath ? [{ rootPath: primaryRootPath }] : []; @@ -452,7 +488,7 @@ function syncKnownRoots(rootEntries: Array<{ rootPath: string; label?: string }> for (const [rootKey, existingRoot] of knownRoots.entries()) { if (!nextRoots.has(rootKey)) { - removeProject(existingRoot.rootPath); + watcherClosures.push(removeProject(existingRoot.rootPath)); forgetProjectPath(existingRoot.rootPath); } } @@ -462,7 +498,7 @@ function syncKnownRoots(rootEntries: Array<{ rootPath: string; label?: string }> isPathWithin(knownRoot.rootPath, project.rootPath) ); if (!stillAllowed) { - removeProject(project.rootPath); + watcherClosures.push(removeProject(project.rootPath)); forgetProjectPath(project.rootPath); } } @@ -479,6 +515,7 @@ function syncKnownRoots(rootEntries: Array<{ rootPath: string; label?: string }> activeProjectKey = undefined; } } + await Promise.all(watcherClosures); } function parseProjectSelector(value: unknown): string | undefined { @@ -618,7 +655,8 @@ function createToolContext(project: ProjectState): ToolContext { paths: project.paths, rootPath: project.rootPath, project: buildProjectDescriptor(project.rootPath), - performIndexing: (incrementalOnly?: boolean) => performIndexing(project, incrementalOnly), + performIndexing: (incrementalOnly, _reason, skipEmbedding) => + performIndexing(project, incrementalOnly, skipEmbedding), listProjects: () => listProjectDescriptors(), getActiveProject: () => getActiveProjectDescriptor() }; @@ -643,6 +681,7 @@ if (primaryRootPath) { export const INDEX_CONSUMING_TOOL_NAMES = [ 'search_codebase', 'get_symbol_references', + 'get_code_relationships', 'detect_circular_dependencies', 'get_team_patterns', 'get_codebase_metadata', @@ -662,12 +701,28 @@ export type IndexSignal = { reason?: string; }; -async function requireValidIndex(rootPath: string, paths: ToolPaths): Promise { - const meta = await readIndexMeta(rootPath); - await validateIndexArtifacts(rootPath, meta); +async function requireValidIndex(project: ProjectState): Promise { + const meta = await readIndexMeta(project.rootPath); + await validateIndexArtifacts(project.rootPath, meta); + + const requestedSkipEmbedding = project.runtimeOverrides.skipEmbedding ?? bootstrapSkipEmbedding; + if ( + requestedSkipEmbedding !== undefined && + (meta.artifacts.vectorDb.embeddingEnabled ?? true) !== !requestedSkipEmbedding + ) { + const reason = `Index embedding mode differs from explicit configuration; rebuilding ${project.rootPath}`; + console.error(`[Index] ${reason}`); + void performIndexing(project); + return { + status: 'indexing', + confidence: 'low', + action: 'rebuild-started', + reason + }; + } // Optional artifact presence informs confidence. - const hasIntelligence = await fileExists(paths.intelligence); + const hasIntelligence = await fileExists(project.paths.intelligence); return { status: 'ready', @@ -688,7 +743,7 @@ async function ensureValidIndexOrAutoHeal(project: ProjectState): Promise { +async function performIndexing( + project: ProjectState, + incrementalOnly?: boolean, + skipEmbedding?: boolean +): Promise { + const previousSkipEmbedding = project.runtimeOverrides.skipEmbedding; + if (skipEmbedding !== undefined) project.runtimeOverrides.skipEmbedding = skipEmbedding; let nextMode = incrementalOnly; for (;;) { await performIndexingOnce(project, nextMode); + if (project.indexState.status === 'error' && skipEmbedding !== undefined) { + if (previousSkipEmbedding === undefined) { + delete project.runtimeOverrides.skipEmbedding; + } else { + project.runtimeOverrides.skipEmbedding = previousSkipEmbedding; + } + } + const shouldRunQueuedRefresh = project.autoRefresh.consumeQueuedRefresh( project.indexState.status ); @@ -1397,10 +1470,10 @@ async function refreshKnownRootsFromClient(): Promise { ); clientRootsEnabled = fileRoots.length > 0; - syncKnownRoots(fileRoots); + await syncKnownRoots(fileRoots); } catch { clientRootsEnabled = false; - syncKnownRoots(primaryRootPath ? [{ rootPath: primaryRootPath }] : []); + await syncKnownRoots(primaryRootPath ? [{ rootPath: primaryRootPath }] : []); } await refreshDiscoveredProjectsForKnownRoots(); @@ -1555,12 +1628,20 @@ async function ensureProjectInitialized(project: ProjectState): Promise { await project.initPromise; } -function ensureProjectWatcher(project: ProjectState, debounceMs: number): void { +async function ensureProjectWatcher(project: ProjectState, debounceMs: number): Promise { if (project.stopWatcher) { touchProject(project.rootPath); return; } + if (project.watcherClosePromise) { + await project.watcherClosePromise; + if (project.stopWatcher) { + touchProject(project.rootPath); + return; + } + } + project.stopWatcher = startFileWatcher({ rootPath: project.rootPath, debounceMs, @@ -1602,8 +1683,7 @@ function ensureProjectWatcher(project: ProjectState, debounceMs: number): void { const projectToEvict = evictionCandidates[0]; if (projectToEvict?.stopWatcher) { - projectToEvict.stopWatcher(); - delete projectToEvict.stopWatcher; + await stopProjectWatcher(projectToEvict); } } @@ -1618,7 +1698,7 @@ async function initProject( touchProject(project.rootPath); if (options.enableWatcher) { - ensureProjectWatcher(project, debounceMs); + await ensureProjectWatcher(project, debounceMs); } } @@ -1642,6 +1722,10 @@ function normalizeRuntimeExtensions(extensions?: string[]): string[] | undefined function buildProjectRuntimeOverrides(projectConfig: ProjectConfig): ProjectRuntimeOverrides { const runtimeOverrides: ProjectRuntimeOverrides = {}; + if ((bootstrapSkipEmbedding ?? projectConfig.skipEmbedding) !== undefined) { + runtimeOverrides.skipEmbedding = bootstrapSkipEmbedding ?? projectConfig.skipEmbedding; + } + if (projectConfig.excludePatterns?.length) { runtimeOverrides.extraExcludePatterns = [...projectConfig.excludePatterns]; } @@ -1754,12 +1838,6 @@ async function main() { // ── Cleanup guards (normal MCP lifecycle) ────────────────────────────────── let shuttingDown = false; - const stopAllWatchers = () => { - for (const project of getAllProjects()) { - project.stopWatcher?.(); - } - }; - let idleTimer: NodeJS.Timeout | undefined; let activeSessionWork = 0; let mcpClientInitialized = false; @@ -1777,8 +1855,13 @@ async function main() { } shuttingDown = true; clearIdleTimer(); - stopAllWatchers(); - process.exit(code); + void stopAllProjectWatchers().then( + () => process.exit(code), + (error: unknown) => { + console.error('[file-watcher] Shutdown cleanup failed:', error); + process.exit(code); + } + ); }; const scheduleIdleShutdown = () => { @@ -1812,7 +1895,11 @@ async function main() { scheduleIdleShutdown(); }; - process.once('exit', stopAllWatchers); + process.once('exit', () => { + void stopAllProjectWatchers().catch((error: unknown) => { + console.error('[file-watcher] Exit cleanup failed:', error); + }); + }); process.once('SIGINT', () => { shutdown(0); }); @@ -1945,22 +2032,32 @@ async function startHttp(explicitPort?: number): Promise { }); // Register cleanup — no parent death guard or stdin listeners in HTTP mode - const stopAllWatchers = () => { - for (const project of getAllProjects()) { - project.stopWatcher?.(); - } - }; - const shutdown = async () => { console.error('[HTTP] Shutting down...'); - stopAllWatchers(); + await stopAllProjectWatchers(); await handle.close(); process.exit(0); }; - process.once('SIGINT', () => void shutdown()); - process.once('SIGTERM', () => void shutdown()); - process.once('exit', stopAllWatchers); + process.once( + 'SIGINT', + () => + void shutdown().catch((error: unknown) => { + console.error('[HTTP] Shutdown failed:', error); + }) + ); + process.once( + 'SIGTERM', + () => + void shutdown().catch((error: unknown) => { + console.error('[HTTP] Shutdown failed:', error); + }) + ); + process.once('exit', () => { + void stopAllProjectWatchers().catch((error: unknown) => { + console.error('[file-watcher] Exit cleanup failed:', error); + }); + }); // If a bootstrap root was provided, auto-init it if (primaryRootPath) { diff --git a/src/project-state.ts b/src/project-state.ts index deaa5a6..b30b728 100644 --- a/src/project-state.ts +++ b/src/project-state.ts @@ -12,6 +12,8 @@ import type { AutoRefreshController } from './core/auto-refresh.js'; import type { ToolPaths, IndexState } from './tools/types.js'; export interface ProjectRuntimeOverrides { + /** Explicit embedding mode; absent preserves the recorded project mode. */ + skipEmbedding?: boolean; /** Extra glob exclusion patterns merged with the default index-time exclusions. */ extraExcludePatterns?: string[]; /** Maximum chunks retained in semantic and keyword indexes for this project. */ @@ -28,7 +30,8 @@ export interface ProjectState { indexState: IndexState; autoRefresh: AutoRefreshController; initPromise?: Promise; - stopWatcher?: () => void; + stopWatcher?: () => Promise; + watcherClosePromise?: Promise; runtimeOverrides: ProjectRuntimeOverrides; } @@ -94,16 +97,45 @@ export function getAllProjects(): ProjectState[] { return Array.from(projects.values()); } -export function removeProject(rootPath: string): void { +export function stopAllProjectWatchers(): Promise { + const closing = Array.from(projects.values(), stopProjectWatcher); + return Promise.all(closing).then(() => undefined); +} + +export function stopProjectWatcher(project: ProjectState): Promise { + if (project.watcherClosePromise) return project.watcherClosePromise; + const stop = project.stopWatcher; + if (!stop) return Promise.resolve(); + + let closing: Promise; + try { + closing = Promise.resolve(stop()); + } catch (error) { + console.error(`[file-watcher] Failed to close watcher for ${project.rootPath}:`, error); + delete project.stopWatcher; + return Promise.resolve(); + } + + delete project.stopWatcher; + project.watcherClosePromise = closing + .catch((error: unknown) => { + console.error(`[file-watcher] Failed to close watcher for ${project.rootPath}:`, error); + }) + .finally(() => { + delete project.watcherClosePromise; + }); + return project.watcherClosePromise; +} + +export function removeProject(rootPath: string): Promise { const key = normalizeRootKey(rootPath); const project = projects.get(key); - project?.stopWatcher?.(); projects.delete(key); + return project ? stopProjectWatcher(project) : Promise.resolve(); } -export function clearProjects(): void { - for (const project of projects.values()) { - project.stopWatcher?.(); - } +export function clearProjects(): Promise { + const closing = stopAllProjectWatchers(); projects.clear(); + return closing; } diff --git a/src/review-context.ts b/src/review-context.ts index f3b76a7..7d97490 100644 --- a/src/review-context.ts +++ b/src/review-context.ts @@ -116,6 +116,7 @@ export interface ReviewSearchResult { sourceFiles: string[]; searchQuality?: SearchQuality; preflight?: SearchResponse['preflight']; + sourceCoverage?: SearchResponse['sourceCoverage']; results: ReviewRelatedResult[]; error?: string; } @@ -398,8 +399,12 @@ export async function buildReviewContextPacket( sourceFiles: reviewQuery.sourceFiles, searchQuality: response.searchQuality, ...(response.preflight ? { preflight: response.preflight } : {}), + ...(response.sourceCoverage ? { sourceCoverage: response.sourceCoverage } : {}), results: response.results.slice(0, maxResultsPerQuery).map(compactSearchResult) }); + if (response.sourceCoverage && response.sourceCoverage.status !== 'current') { + warnings.push(`Search source coverage is incomplete for query: ${reviewQuery.query}`); + } } catch (error) { searches.push({ query: reviewQuery.query, diff --git a/src/server/config.ts b/src/server/config.ts index ff6c1d6..8a8533f 100644 --- a/src/server/config.ts +++ b/src/server/config.ts @@ -4,6 +4,7 @@ import path from 'node:path'; export interface ProjectConfig { root: string; + skipEmbedding?: boolean; excludePatterns?: string[]; parsing?: { maxChunks?: number; @@ -85,6 +86,13 @@ export async function loadServerConfig(): Promise { const resolvedRoot = path.resolve(expandTilde(rawRoot)); const parsedProject: ProjectConfig = { root: resolvedRoot }; + if (typeof project.skipEmbedding === 'boolean') { + parsedProject.skipEmbedding = project.skipEmbedding; + } else if (project.skipEmbedding !== undefined) { + console.error( + `[config] Ignoring invalid project skipEmbedding: ${project.skipEmbedding}` + ); + } const excludePatterns = parseStringArray(project.excludePatterns); if (excludePatterns) { parsedProject.excludePatterns = excludePatterns; diff --git a/src/storage/index.ts b/src/storage/index.ts index fb9d9e2..d776902 100644 --- a/src/storage/index.ts +++ b/src/storage/index.ts @@ -8,6 +8,11 @@ export * from './lancedb.js'; import { VectorStorageProvider, StorageConfig, DEFAULT_STORAGE_CONFIG } from './types.js'; import { LanceDBStorageProvider } from './lancedb.js'; +import { assertOptionalDependency } from '../utils/optional-dependency.js'; + +export function assertStorageRuntimeAvailable(): void { + assertOptionalDependency('@lancedb/lancedb', '^0.4.0'); +} /** * Get a storage provider based on configuration @@ -16,6 +21,7 @@ export async function getStorageProvider( config: Partial = {}, options?: { expectExisting?: boolean; expectedDimensions?: number } ): Promise { + assertStorageRuntimeAvailable(); const mergedConfig = { ...DEFAULT_STORAGE_CONFIG, ...config }; const provider = new LanceDBStorageProvider(); diff --git a/src/storage/lancedb.ts b/src/storage/lancedb.ts index 9351d94..bd71063 100644 --- a/src/storage/lancedb.ts +++ b/src/storage/lancedb.ts @@ -8,6 +8,7 @@ import type { Connection, Table } from '@lancedb/lancedb'; import { VectorStorageProvider, CodeChunkWithEmbedding, VectorSearchResult } from './types.js'; import { CodeChunk, SearchFilters } from '../types/index.js'; import { IndexCorruptedError } from '../errors/index.js'; +import { assertOptionalDependency } from '../utils/optional-dependency.js'; interface LanceDBRecord { id: string; @@ -48,6 +49,7 @@ export class LanceDBStorageProvider implements VectorStorageProvider { options?: { expectExisting?: boolean; expectedDimensions?: number } ): Promise { if (this.initialized) return; + assertOptionalDependency('@lancedb/lancedb', '^0.4.0'); try { this.storagePath = storagePath; diff --git a/src/tools/get-code-relationships.ts b/src/tools/get-code-relationships.ts new file mode 100644 index 0000000..9c1ef17 --- /dev/null +++ b/src/tools/get-code-relationships.ts @@ -0,0 +1,340 @@ +import type { Tool } from '@modelcontextprotocol/sdk/types.js'; +import { createHash } from 'crypto'; +import { promises as fs } from 'fs'; +import path from 'path'; +import { RELATIONSHIPS_FILENAME } from '../constants/codebase-context.js'; +import { readIndexMeta } from '../core/index-meta.js'; +import { + traverseFileGraphWithVerification, + type FileGraphEdge, + type ImportEdgeDetail, + type UnresolvedImportEdge +} from '../utils/usage-tracker.js'; +import { + InternalSourceResolver, + loadTsconfigCompilerOptions +} from '../utils/internal-source-resolver.js'; +import type { ToolContext, ToolResponse } from './types.js'; + +const RESPONSE_BYTE_BUDGET = 4000; +const MAX_INSPECTIONS = 1000; +const MAX_DIAGNOSTICS = 20; + +function serializeBoundedResponse(value: Record): { + text: string; + truncated: boolean; +} { + const bounded: Record = { ...value }; + const collectionKeys = ['paths', 'unresolved', 'staleEdges', 'staleUnresolvedEdges']; + for (const key of collectionKeys) { + if (Array.isArray(bounded[key])) bounded[key] = [...bounded[key]]; + } + let truncated = value.truncated === true; + const encode = (): string => JSON.stringify(bounded); + let payload = encode(); + const trimOrder = ['staleUnresolvedEdges', 'staleEdges', 'unresolved', 'paths']; + while (Buffer.byteLength(payload, 'utf8') > RESPONSE_BYTE_BUDGET) { + const key = trimOrder.find( + (candidate) => + Array.isArray(bounded[candidate]) && (bounded[candidate] as unknown[]).length > 0 + ); + if (!key) { + return { + text: JSON.stringify({ + status: 'output_limited', + truncated: true, + message: + 'The relationship response exceeded its byte budget; narrow the starting file or hop limit.' + }), + truncated: true + }; + } + (bounded[key] as unknown[]).pop(); + truncated = true; + bounded.truncated = true; + bounded.outputNote = [ + value.outputNote, + 'The serialized response reached its 4000-byte budget; some paths or diagnostics were omitted. Narrow the starting file or hop limit for more.' + ] + .filter(Boolean) + .join(' '); + payload = encode(); + } + return { text: payload, truncated }; +} + +export function finalizeRelationshipPayload(value: Record): string { + return serializeBoundedResponse(value).text; +} + +export const definition: Tool = { + name: 'get_code_relationships', + description: + 'Follow verified import or re-export relationships up to three hops from one indexed file. Returns source lines and freshness diagnostics; imports never imply runtime calls.', + inputSchema: { + type: 'object', + properties: { + file: { type: 'string', description: 'Indexed repository-relative file path.' }, + direction: { type: 'string', enum: ['outgoing', 'incoming'] }, + max_hops: { type: 'number', description: 'Hop limit from 1 to 3; defaults to 2.' }, + max_nodes: { type: 'number', description: 'Output node cap from 1 to 100; defaults to 50.' } + }, + required: ['file'] + } +}; + +interface Sidecar { + header?: { buildId?: string; formatVersion?: number }; + graph?: { + imports?: Record; + importDetails?: Record>; + exports?: Record; + unresolvedEdges?: UnresolvedImportEdge[]; + }; +} + +function asObject(value: unknown): Record { + return value !== null && typeof value === 'object' && !Array.isArray(value) + ? (value as Record) + : {}; +} + +function safeRelativePath(value: string): string | null { + const normalized = value.replace(/\\/g, '/').replace(/^\.\//, ''); + if ( + !normalized || + path.posix.isAbsolute(normalized) || + /^[A-Za-z]:\//.test(normalized) || + normalized.startsWith('//') || + normalized.split('/').some((part) => part === '..') + ) { + return null; + } + return normalized; +} + +function matchesRelativePath(root: string, absolute: string, expected: string): boolean { + const relative = path.relative(root, absolute).replace(/\\/g, '/'); + return process.platform === 'win32' + ? relative.toLowerCase() === expected.replace(/\\/g, '/').toLowerCase() + : relative === expected.replace(/\\/g, '/'); +} + +async function resolveConfined(rootRealPath: string, relativePath: string): Promise { + const safe = safeRelativePath(relativePath); + if (!safe) return null; + try { + const realPath = await fs.realpath(path.resolve(rootRealPath, ...safe.split('/'))); + const relative = path.relative(rootRealPath, realPath); + return relative === '' || + (!relative.startsWith(`..${path.sep}`) && relative !== '..' && !path.isAbsolute(relative)) + ? realPath + : null; + } catch { + return null; + } +} + +async function currentFingerprint(filePath: string): Promise { + try { + const content = await fs.readFile(filePath); + return createHash('sha256').update(content).digest('hex'); + } catch { + return null; + } +} + +async function verifyEdge( + rootRealPath: string, + edge: FileGraphEdge, + resolver: InternalSourceResolver +): Promise { + if ( + !edge.verified || + !edge.source.specifier || + !edge.source.sourceFingerprint || + !edge.source.targetFingerprint + ) { + return false; + } + const [from, to] = await Promise.all([ + resolveConfined(rootRealPath, edge.from), + resolveConfined(rootRealPath, edge.to) + ]); + if (!from || !to) return false; + const currentResolution = await resolver.resolve(from, edge.source.specifier, true); + if (currentResolution.status !== 'resolved' || !currentResolution.path) return false; + const currentTarget = await fs.realpath(currentResolution.path).catch(() => null); + if (!currentTarget || !matchesRelativePath(rootRealPath, currentTarget, edge.to)) { + return false; + } + const [fromFingerprint, toFingerprint] = await Promise.all([ + currentFingerprint(from), + currentFingerprint(to) + ]); + return ( + fromFingerprint === edge.source.sourceFingerprint && + toFingerprint === edge.source.targetFingerprint + ); +} + +export async function handle( + args: Record, + ctx: ToolContext +): Promise { + const fileArg = typeof args.file === 'string' ? args.file : ''; + const direction = args.direction === 'incoming' ? 'incoming' : 'outgoing'; + const maxHops = + args.max_hops === 1 || args.max_hops === 2 || args.max_hops === 3 ? args.max_hops : 2; + const maxNodes = + typeof args.max_nodes === 'number' && Number.isFinite(args.max_nodes) + ? Math.max(1, Math.min(100, Math.floor(args.max_nodes))) + : 50; + const reply = (value: Record, isError = false): ToolResponse => ({ + content: [{ type: 'text', text: finalizeRelationshipPayload(value) }], + ...(isError ? { isError: true } : {}) + }); + + try { + const rootRealPath = await fs.realpath(ctx.rootPath); + const start = safeRelativePath(fileArg); + if (!start) + return reply( + { status: 'invalid_path', message: 'Provide a repository-relative file path.' }, + true + ); + const startRealPath = await resolveConfined(rootRealPath, start); + if (!startRealPath) + return reply({ + status: 'not_indexed_or_outside_root', + file: start, + message: 'The requested file is missing or resolves outside this project root.' + }); + + const sidecarPath = path.join(path.dirname(ctx.paths.intelligence), RELATIONSHIPS_FILENAME); + const parsed: unknown = JSON.parse(await fs.readFile(sidecarPath, 'utf8')); + const sidecar = asObject(parsed) as Sidecar; + const meta = await readIndexMeta(ctx.rootPath); + if (!sidecar.header?.buildId || sidecar.header.buildId !== meta.buildId) { + return reply({ + status: 'stale_index', + message: + 'Relationship sidecar and index metadata have different build identities. Refresh the index before relying on graph results.' + }); + } + if (!sidecar.graph?.imports) { + return reply({ + status: 'graph_unavailable', + message: 'The current index has no file relationship graph. Refresh indexing to build it.' + }); + } + + // Every edge is resolved against current confined files with allowUnindexed=true. + // Avoid resolving every stored graph file during setup for a nearby relationship query. + const resolver = await InternalSourceResolver.create(rootRealPath, [], { + compilerOptions: loadTsconfigCompilerOptions(rootRealPath) + }); + const verificationCache = new Map(); + const result = await traverseFileGraphWithVerification( + sidecar.graph, + start, + { direction, maxHops, maxNodes, maxPaths: 40, maxInspections: MAX_INSPECTIONS }, + async (edge) => { + const proofKey = JSON.stringify([ + edge.from, + edge.to, + edge.kind, + edge.source.specifier, + edge.source.line, + edge.source.sourceFingerprint, + edge.source.targetFingerprint + ]); + let verified = verificationCache.get(proofKey); + if (verified === undefined) { + verified = await verifyEdge(rootRealPath, edge, resolver); + verificationCache.set(proofKey, verified); + } + return verified; + } + ); + const staleEdges = result.rejectedEdges.slice(0, MAX_DIAGNOSTICS).map((edge) => ({ + from: edge.from, + to: edge.to, + kind: edge.kind, + line: edge.source.line, + specifier: edge.source.specifier + })); + + const unresolved: UnresolvedImportEdge[] = []; + const staleUnresolvedEdges: UnresolvedImportEdge[] = []; + for (const edge of result.unresolved) { + const importer = await resolveConfined(rootRealPath, edge.from); + const importerFingerprint = importer ? await currentFingerprint(importer) : null; + if (importerFingerprint === edge.source.sourceFingerprint) unresolved.push(edge); + else staleUnresolvedEdges.push(edge); + } + const stale = result.rejectedEdges.length > 0 || staleUnresolvedEdges.length > 0; + const visiblePaths = result.paths; + const visibleUnresolved = unresolved.slice(0, MAX_DIAGNOSTICS); + const visibleStaleEdges = staleEdges; + const visibleStaleUnresolved = staleUnresolvedEdges.slice(0, MAX_DIAGNOSTICS); + const diagnosticsLimited = + staleEdges.length < result.rejectedEdges.length || + visibleUnresolved.length < unresolved.length || + visibleStaleUnresolved.length < staleUnresolvedEdges.length; + const outputLimited = result.truncated || diagnosticsLimited; + return reply({ + status: stale ? 'stale_edges' : 'success', + file: start, + direction, + maxHops, + paths: visiblePaths.map((entry) => ({ + nodes: entry.nodes, + edges: entry.edges.map((edge) => ({ + from: edge.from, + to: edge.to, + kind: edge.kind, + source: { line: edge.source.line, specifier: edge.source.specifier } + })) + })), + unresolved: visibleUnresolved.map((edge) => ({ + from: edge.from, + kind: edge.kind, + source: { line: edge.source.line, specifier: edge.source.specifier }, + reason: edge.reason + })), + ...(visibleStaleEdges.length > 0 ? { staleEdges: visibleStaleEdges } : {}), + ...(visibleStaleUnresolved.length > 0 + ? { + staleUnresolvedEdges: visibleStaleUnresolved.map((edge) => ({ + from: edge.from, + kind: edge.kind, + source: { line: edge.source.line, specifier: edge.source.specifier }, + reason: edge.reason + })) + } + : {}), + truncated: outputLimited, + message: stale + ? 'Some stored relationships could not be proven against current confined source files. Omitted those edges; refresh indexing to update this graph.' + : unresolved.length > 0 + ? 'Resolved relationships are current. Unresolved source specifiers are diagnostics and were not followed.' + : 'Relationships are limited to verified imports and re-exports; dynamic calls are not inferred.', + ...(outputLimited + ? { + outputNote: result.inspectionLimited + ? 'The relationship inspection budget was reached; additional relationships were not checked. Refresh indexing to remove stale edges, then retry from this file or a returned endpoint.' + : result.truncated + ? 'The accepted node or path cap was reached; narrow the hop limit or continue from a returned endpoint for more.' + : 'Some freshness or unresolved diagnostics were omitted. Refresh indexing to update this graph.' + } + : {}) + }); + } catch (error) { + return reply({ + status: 'graph_unavailable', + message: 'Could not read a current relationship sidecar. Refresh indexing and retry.', + detail: error instanceof Error ? error.message : String(error) + }); + } +} diff --git a/src/tools/get-indexing-status.ts b/src/tools/get-indexing-status.ts index 9da86ca..1dc5e59 100644 --- a/src/tools/get-indexing-status.ts +++ b/src/tools/get-indexing-status.ts @@ -1,6 +1,9 @@ +import { fileURLToPath } from 'node:url'; import type { Tool } from '@modelcontextprotocol/sdk/types.js'; import type { ToolContext, ToolResponse } from './types.js'; +const loadedStatusModulePath = fileURLToPath(import.meta.url); + export const definition: Tool = { name: 'get_indexing_status', description: @@ -26,6 +29,7 @@ export async function handle( { status: ctx.indexState.status, rootPath: ctx.rootPath, + loadedStatusModulePath, lastIndexed: ctx.indexState.lastIndexed?.toISOString(), stats: ctx.indexState.stats ? { diff --git a/src/tools/get-symbol-references.ts b/src/tools/get-symbol-references.ts index 0536f3c..88d6b33 100644 --- a/src/tools/get-symbol-references.ts +++ b/src/tools/get-symbol-references.ts @@ -1,11 +1,12 @@ import type { Tool } from '@modelcontextprotocol/sdk/types.js'; import type { ToolContext, ToolResponse } from './types.js'; -import { findSymbolReferences } from '../core/symbol-references.js'; +import { findSymbolReferences, type SymbolReferencesResult } from '../core/symbol-references.js'; +import { IndexCorruptedError } from '../errors/index.js'; export const definition: Tool = { name: 'get_symbol_references', description: - 'Find concrete references to a symbol in indexed chunks. Returns total usageCount and top usage snippets.', + 'Find current-source references across the files recorded by the last successful index. Returns observed usageCount, bounded snippets, parser confidence, and coverage.', inputSchema: { type: 'object', properties: { @@ -51,7 +52,29 @@ export async function handle( }; } - const result = await findSymbolReferences(ctx.rootPath, normalizedSymbol, normalizedLimit); + let result: SymbolReferencesResult; + try { + result = await findSymbolReferences(ctx.rootPath, normalizedSymbol, normalizedLimit); + } catch (error) { + if (!(error instanceof IndexCorruptedError)) throw error; + return { + content: [ + { + type: 'text', + text: JSON.stringify( + { + status: 'error', + symbol: normalizedSymbol, + message: error.message + }, + null, + 2 + ) + } + ], + isError: true + }; + } if (result.status === 'error') { return { @@ -68,7 +91,8 @@ export async function handle( 2 ) } - ] + ], + isError: true }; } @@ -83,7 +107,8 @@ export async function handle( usageCount: result.usageCount, usages: result.usages, confidence: result.confidence, - isComplete: result.isComplete + isComplete: result.isComplete, + coverage: result.coverage }, null, 2 diff --git a/src/tools/index.ts b/src/tools/index.ts index def32ac..7f5ea4b 100644 --- a/src/tools/index.ts +++ b/src/tools/index.ts @@ -13,6 +13,7 @@ import { definition as d8, handle as h8 } from './detect-circular-dependencies.j import { definition as d9, handle as h9 } from './remember.js'; import { definition as d10, handle as h10 } from './get-memory.js'; import { definition as d11, handle as h11 } from './get-codebase-health.js'; +import { definition as d12, handle as h12 } from './get-code-relationships.js'; import type { ToolContext, ToolResponse } from './types.js'; @@ -52,7 +53,7 @@ function withProjectSelector(definition: Tool): Tool { }; } -export const TOOLS: Tool[] = [d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11].map( +export const TOOLS: Tool[] = [d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12].map( withProjectSelector ); @@ -84,6 +85,8 @@ export async function dispatchTool( return h10(args, ctx); case 'get_codebase_health': return h11(args, ctx); + case 'get_code_relationships': + return h12(args, ctx); default: return { content: [{ type: 'text', text: JSON.stringify({ error: `Unknown tool: ${name}` }) }], diff --git a/src/tools/refresh-index.ts b/src/tools/refresh-index.ts index 3c1b647..b2d1726 100644 --- a/src/tools/refresh-index.ts +++ b/src/tools/refresh-index.ts @@ -1,11 +1,14 @@ import type { Tool } from '@modelcontextprotocol/sdk/types.js'; +import { assertEmbeddingRuntimeAvailable } from '../embeddings/index.js'; +import { OptionalDependencyError } from '../errors/index.js'; +import { assertStorageRuntimeAvailable } from '../storage/index.js'; import type { ToolContext, ToolResponse } from './types.js'; export const definition: Tool = { name: 'refresh_index', description: 'Re-index the codebase. Supports full re-index or incremental mode. ' + - 'Use incrementalOnly=true to only process files changed since last index.', + 'Use incrementalOnly=true to process changed files. Omit skipEmbedding to preserve the last successful mode; true disables embedding and reranker models, false opts into semantic embeddings. Mode changes require a full rebuild.', inputSchema: { type: 'object', properties: { @@ -17,6 +20,11 @@ export const definition: Tool = { type: 'boolean', description: 'If true, only re-index files changed since last full index (faster). Default: false (full re-index)' + }, + skipEmbedding: { + type: 'boolean', + description: + 'True selects model-free indexing; false explicitly opts into semantic embeddings. Omit to preserve the last successful mode.' } } } @@ -26,12 +34,45 @@ export async function handle( args: Record, ctx: ToolContext ): Promise { - const { reason, incrementalOnly } = args as { reason?: string; incrementalOnly?: boolean }; + if (args.skipEmbedding !== undefined && typeof args.skipEmbedding !== 'boolean') { + return { + isError: true, + content: [{ type: 'text', text: 'skipEmbedding must be a boolean' }] + }; + } + if (ctx.indexState.status === 'indexing') { + return { + isError: true, + content: [ + { type: 'text', text: 'An index build is already running; retry after it completes.' } + ] + }; + } + const { reason, incrementalOnly, skipEmbedding } = args as { + reason?: string; + incrementalOnly?: boolean; + skipEmbedding?: boolean; + }; + + if (skipEmbedding === false) { + try { + assertEmbeddingRuntimeAvailable(); + assertStorageRuntimeAvailable(); + } catch (error) { + if (error instanceof OptionalDependencyError) { + return { + isError: true, + content: [{ type: 'text', text: error.message }] + }; + } + throw error; + } + } const mode = incrementalOnly ? 'incremental' : 'full'; console.error(`Refresh requested (${mode}): ${reason || 'Manual trigger'}`); - ctx.performIndexing(incrementalOnly); + void ctx.performIndexing(incrementalOnly, reason, skipEmbedding); return { content: [ @@ -42,7 +83,7 @@ export async function handle( status: 'started', mode, message: incrementalOnly - ? 'Incremental re-indexing started. Only changed files will be re-embedded.' + ? 'Incremental re-indexing started. A changed embedding mode requires a full rebuild.' : 'Full re-indexing started. Check status with get_indexing_status.', reason }, diff --git a/src/tools/search-codebase.ts b/src/tools/search-codebase.ts index 7a4f822..de90f94 100644 --- a/src/tools/search-codebase.ts +++ b/src/tools/search-codebase.ts @@ -3,16 +3,14 @@ import { promises as fs } from 'fs'; import path from 'path'; import type { ToolContext, ToolResponse, DecisionCard } from './types.js'; import { CodebaseSearcher } from '../core/search.js'; -import type { SearchIntentProfile } from '../core/search.js'; +import type { SearchIntentProfile, SearchSourceCoverage } from '../core/search.js'; import type { SearchResult, IntelligenceData, PatternsData, - IntelligenceGoldenFile, ChunkMetadata } from '../types/index.js'; import { buildEvidenceLock } from '../preflight/evidence-lock.js'; -import type { EvidenceLock } from '../preflight/evidence-lock.js'; import { shouldIncludePatternConflictCategory } from '../preflight/query-scope.js'; import { isComplementaryPatternConflict, @@ -21,7 +19,7 @@ import { import { assessSearchQuality } from '../core/search-quality.js'; import { getRerankerStatus } from '../core/reranker.js'; import { IndexCorruptedError } from '../errors/index.js'; -import { formatMemoryScopeText, readMemoriesFile, withConfidence } from '../memory/store.js'; +import { readMemoriesFile, withConfidence } from '../memory/store.js'; import type { MemoryWithConfidence } from '../memory/store.js'; import { indexHealthByFile, normalizeHealthLookupKey, readHealthFile } from '../health/store.js'; import { InternalFileGraph } from '../utils/usage-tracker.js'; @@ -104,7 +102,7 @@ export const definition: Tool = { includeSnippets: { type: 'boolean', description: - 'Include code snippets in results (default: false). If you need code, prefer read_file instead.', + 'Include code snippets in results (default: false). Open returned paths with your host file reader; CBC does not expose a file-reading tool.', default: false }, filters: { @@ -214,14 +212,17 @@ export async function handle( const searcher = new CodebaseSearcher(ctx.rootPath); let results: SearchResult[]; + let sourceCoverage: SearchSourceCoverage; const searchProfile = ( intent && ['explore', 'edit', 'refactor', 'migrate'].includes(intent) ? intent : 'explore' ) as SearchIntentProfile; try { - results = await searcher.search(queryStr, limit || 5, filters, { + const outcome = await searcher.searchWithCoverage(queryStr, limit || 5, filters, { profile: searchProfile }); + results = outcome.results; + sourceCoverage = outcome.sourceCoverage; } catch (error) { if (error instanceof IndexCorruptedError) { console.error('[Auto-Heal] Index corrupted. Triggering background re-index...'); @@ -245,6 +246,34 @@ export async function handle( throw error; } + if (sourceCoverage.status === 'no_fresh_results') { + return { + content: [ + { + type: 'text', + text: finalizeSearchPayloadText( + { + status: 'no_fresh_results', + message: sourceCoverage.candidateRetrievalFailed + ? 'No fresh results in the inspected candidates. Semantic refill failed; retry or refresh the index.' + : sourceCoverage.candidateBudgetExhausted + ? 'No fresh results in the bounded candidate window. Some candidates remain uninspected; refresh the index or narrow the query and retry.' + : 'No fresh results: nominated sources changed or could not be read. Refresh the index and retry.', + sourceCoverage, + searchQuality: { status: 'low_confidence', confidence: 0 }, + budget: { mode: mode === 'full' ? 'full' : 'compact', resultCount: 0 }, + ...(searchProfile !== 'explore' && { + preflight: { ready: false, reason: sourceCoverage.refreshAction } + }), + results: [] + }, + { mode: mode === 'full' ? 'full' : 'compact', pretty: true, transportAware: true } + ) + } + ] + }; + } + // Load memories for keyword matching, enriched with confidence const allMemories = await readMemoriesFile(ctx.paths.memory); const allMemoriesWithConf = withConfidence(allMemories); @@ -336,6 +365,38 @@ export async function handle( .filter((value): value is string => value !== null) ); + function canonicalRootRelativePath(filePath: string): string | undefined { + if (!filePath || path.isAbsolute(filePath)) return undefined; + const normalized = filePath.replace(/\\/g, '/'); + const parts = normalized.split('/'); + if ( + /^[a-z]:/i.test(parts[0]) || + parts.some((part) => part === '' || part === '.' || part === '..') + ) { + return undefined; + } + return normalized; + } + + function hasGroundedCanonicalExample(candidate: { + canonicalExample?: { file: string; snippet?: string }; + }): boolean { + const example = candidate.canonicalExample; + const examplePath = example && canonicalRootRelativePath(example.file); + const snippet = example?.snippet; + if (!examplePath || !snippet?.trim()) return false; + + const matches = results.filter((result) => { + const resultPath = canonicalRootRelativePath(normalizeGraphPath(result.filePath)); + return resultPath === examplePath; + }); + return matches.length === 1 && matches[0].snippet.includes(snippet); + } + + function isPatternInQueryScope(category: string): boolean { + return shouldIncludePatternConflictCategory(category, queryStr); + } + function getMemoryScopeBoost(memory: MemoryWithConfidence): number { if (!memory.scope || memory.scope.kind === 'global') return 0; @@ -354,10 +415,13 @@ export async function handle( return 0; } + function getMemoryText(memory: MemoryWithConfidence): string { + return `${memory.memory} ${memory.reason}`.toLowerCase(); + } + function getMemoryTextMatchCount(memory: MemoryWithConfidence): number { - const haystack = - `${memory.memory} ${memory.reason} ${formatMemoryScopeText(memory.scope)}`.toLowerCase(); - return queryTerms.filter((term) => haystack.includes(term)).length; + const text = getMemoryText(memory); + return queryTerms.filter((term) => text.includes(term)).length; } function formatMemoryForOutput(memory: MemoryWithConfidence): string { @@ -628,7 +692,7 @@ export async function handle( if (terms.length === 0) return []; return memories .filter((m) => { - const text = `${m.memory} ${m.reason} ${formatMemoryScopeText(m.scope)}`.toLowerCase(); + const text = getMemoryText(m); const matchCount = terms.filter((t) => text.includes(t)).length; return (matchCount >= 2 || getMemoryScopeBoost(m) >= 2) && m.effectiveConfidence >= 0.5; }) @@ -682,55 +746,35 @@ export async function handle( function buildPatternSummary(): string | undefined { const patterns = intelligence?.patterns; if (!patterns) return undefined; - const entries = Object.values(patterns) - .filter((data) => Boolean(data.primary)) + const entries = Object.entries(patterns) + .filter(([category, data]) => + Boolean( + data.primary && + isPatternInQueryScope(category) && + hasGroundedCanonicalExample(data.primary) + ) + ) .slice(0, 3) - .map((data) => { + .map(([, data]) => { const p = data.primary!; return `${p.name} (${p.frequency}, ${p.trend ?? 'Stable'})`; }); return entries.length > 0 ? entries.join(', ') : undefined; } - // Return the golden file path most relevant to the given search results, using directory - // overlap as a query-aware proximity signal. Falls back to goldenFiles[0] when no overlap. - // Track B: use cross-encoder reranker to score golden files against the query for stronger - // relevance — deferred to avoid reranker latency on compact mode in v2. + // A repository-wide example is not evidence for this query. Recommend it only + // when search independently returned the same file, preserving result order. function getBestExample(topResults: Array<{ filePath: string }>): string | undefined { const goldenFiles = intelligence?.goldenFiles; - if (!goldenFiles?.length) return undefined; - if (!topResults?.length) return goldenFiles[0].file; - - // Extract directory paths from top-5 results - const resultDirs = topResults.slice(0, 5).map((r) => path.dirname(r.filePath)); - - let bestFile = goldenFiles[0]; - let bestScore = -1; - - for (const gf of goldenFiles) { - const gfParts = path.dirname(gf.file).split(path.sep).filter(Boolean); - let maxShared = 0; - - for (const resDir of resultDirs) { - const resParts = resDir.split(path.sep).filter(Boolean); - let shared = 0; - const minLen = Math.min(gfParts.length, resParts.length); - for (let i = 0; i < minLen; i++) { - if (gfParts[i] === resParts[i]) shared++; - else break; - } - if (shared > maxShared) maxShared = shared; - } - - // Use golden file score as tie-breaker - const score = maxShared * 1000 + (gf.score ?? 0); - if (score > bestScore) { - bestScore = score; - bestFile = gf; - } + if (!goldenFiles?.length || !topResults.length) return undefined; + const examples = new Map( + goldenFiles.map((file) => [path.resolve(ctx.rootPath, file.file), file.file]) + ); + for (const result of topResults.slice(0, 5)) { + const example = examples.get(path.resolve(ctx.rootPath, result.filePath)); + if (example) return example; } - - return bestFile.file; + return undefined; } // Build up to 2 dynamic next-hop suggestions (compact mode) @@ -739,74 +783,30 @@ export async function handle( sq: { status: string } ): Array<{ tool: string; args?: Record; why: string }> { const hops: Array<{ tool: string; args?: Record; why: string }> = []; + // Only advertise tools this MCP server actually supplies. Source reads belong + // to the host; returned result paths already identify what to open there. if (compactResults.length > 0) { - hops.push({ - tool: 'read_file', - args: { path: compactResults[0].filePath }, - why: 'Read the top match for implementation details' - }); - } - if (sq.status === 'low_confidence') { - hops.push({ tool: 'get_team_patterns', why: 'Review pattern guidance — confidence is low' }); - } else if (hops.length < 2) { - hops.push({ tool: 'get_team_patterns', why: 'Review team patterns and best examples' }); + const relativeFile = path.relative(ctx.rootPath, compactResults[0].filePath); + if ( + !path.isAbsolute(relativeFile) && + relativeFile !== '..' && + !relativeFile.startsWith(`..${path.sep}`) + ) { + hops.push({ + tool: 'get_code_relationships', + args: { file: relativeFile.replace(/\\/g, '/'), direction: 'outgoing' }, + why: + sq.status === 'low_confidence' + ? 'Follow current import evidence while continuing source discovery' + : 'Follow verified imports to related implementation files' + }); + } } return hops.slice(0, 2); } const searchQuality = assessSearchQuality(queryStr, results); - // Always-on edit preflight (lite): do not require intent and keep payload small. - let editPreflight: - | { mode: string; riskLevel: string; confidence: string; evidenceLock: EvidenceLock } - | undefined = undefined; - if (intelligence && (!intent || intent === 'explore')) { - try { - const resultPaths = results.map((r) => r.filePath); - const impactCandidates = computeImpactCandidates(resultPaths); - - // Use existing pattern intelligence for evidenceLock scoring, but keep the output payload lite. - const preferredPatternsForEvidence: Array<{ pattern: string; example?: string }> = []; - const patterns: PatternsData = intelligence.patterns || {}; - for (const [_, data] of Object.entries(patterns)) { - if (data.primary) { - const p = data.primary; - if (p.trend === 'Rising' || p.trend === 'Stable') { - preferredPatternsForEvidence.push({ - pattern: p.name, - ...(p.canonicalExample && { example: p.canonicalExample.file }) - }); - } - } - } - - let riskLevel: 'low' | 'medium' | 'high' = 'low'; - if (impactCandidates.length > 10) { - riskLevel = 'high'; - } else if (impactCandidates.length > 3) { - riskLevel = 'medium'; - } - - editPreflight = { - mode: 'lite', - riskLevel, - confidence: computeIndexConfidence(), - evidenceLock: buildEvidenceLock({ - results, - preferredPatterns: preferredPatternsForEvidence.slice(0, 5), - relatedMemories, - failureWarnings: [], - patternConflicts: [], - searchQualityStatus: searchQuality.status - // Note: indexFreshness intentionally omitted — explore intent is for navigation, - // not mutation; freshness gating does not apply. - }) - }; - } catch { - // editPreflight is best-effort - never fail search over it - } - } - // Compose preflight card for edit/refactor/migrate intents let preflight: DecisionCard | undefined = undefined; const preflightIntents = ['edit', 'refactor', 'migrate']; @@ -839,7 +839,11 @@ export async function handle( // Primary pattern = preferred if Rising or Stable if (data.primary) { const p = data.primary; - if (p.trend === 'Rising' || p.trend === 'Stable') { + if ( + (p.trend === 'Rising' || p.trend === 'Stable') && + isPatternInQueryScope(category) && + hasGroundedCanonicalExample(p) + ) { preferredPatternsList.push({ pattern: p.name, category, @@ -853,7 +857,11 @@ export async function handle( // Also-detected patterns that are Declining = avoid if (data.alsoDetected) { for (const alt of data.alsoDetected) { - if (alt.trend === 'Declining') { + if ( + alt.trend === 'Declining' && + isPatternInQueryScope(category) && + hasGroundedCanonicalExample(alt) + ) { avoidPatternsList.push({ pattern: alt.name, category, @@ -870,7 +878,7 @@ export async function handle( const resultPaths = results.map((r) => r.filePath); const impactCandidates = computeImpactCandidates(resultPaths); - // PREF-02: Compute impact coverage (callers of result files that appear in results) + // Compute impact coverage (callers of result files that appear in results) const callerFiles = resultPaths.flatMap((p) => { const importers: string[] = []; for (const [dep, importerList] of reverseImports) { @@ -928,14 +936,6 @@ export async function handle( riskLevel = 'medium'; } - // --- Golden files (exemplar code) --- - const goldenFiles = (intelligence.goldenFiles ?? []) - .slice(0, 3) - .map((g: IntelligenceGoldenFile) => ({ - file: g.file, - score: g.score - })); - // --- Confidence (index freshness) --- // TODO: Review this confidence calculation //const confidence = computeIndexConfidence(); @@ -964,9 +964,11 @@ export async function handle( if (shouldSkipLegacyTestingFrameworkCategory(cat, patterns)) continue; if (!shouldIncludePatternConflictCategory(cat, queryStr)) continue; if (!data.primary || !data.alsoDetected?.length) continue; + if (!hasGroundedCanonicalExample(data.primary)) continue; const primaryFreq = parseFloat(data.primary.frequency) || 100; if (primaryFreq >= 80) continue; for (const alt of data.alsoDetected) { + if (!hasGroundedCanonicalExample(alt)) continue; const altFreq = parseFloat(alt.frequency) || 0; if (altFreq >= 20) { if (isComplementaryPatternConflict(cat, data.primary.name, alt.name)) continue; @@ -991,7 +993,7 @@ export async function handle( indexFreshness: computeIndexConfidence() }); - // Build clean decision card (PREF-01 to PREF-04) + // Build clean decision card const decisionCard: DecisionCard = { ready: evidenceLock.readyToEdit }; @@ -1014,13 +1016,15 @@ export async function handle( decisionCard.warnings = [...(decisionCard.warnings ?? []), ...freshnessGaps]; } - // Add patterns (do/avoid, capped at 3 each, with adoption %) - const doPatterns = preferredPatternsForOutput - .slice(0, 3) - .map((p) => `${p.pattern} — ${p.adoption ? `${p.adoption} adoption` : ''}`); - const avoidPatterns = avoidPatternsForOutput - .slice(0, 3) - .map((p) => `${p.pattern} — ${p.adoption ? `${p.adoption} adoption` : ''} (declining)`); + // Pattern advice is actionable only when search itself did not abstain. + const suppressPatternAdvice = + evidenceLock.status === 'block' || searchQuality.status === 'low_confidence'; + const doPatterns = suppressPatternAdvice + ? [] + : preferredPatternsForOutput.slice(0, 3).map((p) => p.pattern); + const avoidPatterns = suppressPatternAdvice + ? [] + : avoidPatternsForOutput.slice(0, 3).map((p) => `${p.pattern} (declining)`); if (doPatterns.length > 0 || avoidPatterns.length > 0) { decisionCard.patterns = { ...(doPatterns.length > 0 && { do: doPatterns }), @@ -1091,22 +1095,23 @@ export async function handle( } } - // For edit/refactor/migrate: return clean decision card. - // For explore or lite-only: return lightweight { ready, reason }. + // Discovery returns retrieval quality and source coverage; mutation intents get edit readiness. let preflightPayload: { ready: boolean; reason?: string } | Record | undefined; if (preflight) { // preflight is already a clean decision card (DecisionCard type) preflightPayload = preflight; - } else if (editPreflight) { - // Lite preflight for explore intent - const el = editPreflight.evidenceLock; + } + if (sourceCoverage.status === 'partial' && preflightIntents.includes(searchProfile)) { preflightPayload = { - ready: el?.readyToEdit ?? false, - ...(el && !el.readyToEdit && el.nextAction && { reason: el.nextAction }) + ...preflightPayload, + ready: false, + abstain: true, + reason: sourceCoverage.refreshAction, + nextAction: sourceCoverage.refreshAction }; } - // Helper: Build scope header for symbol-aware chunks (SEARCH-02) + // Helper: Build scope header for symbol-aware chunks function buildScopeHeader(metadata: ChunkMetadata): string | null { // Try symbolPath first (most reliable for AST-based symbols) if (metadata?.symbolPath && Array.isArray(metadata.symbolPath)) { @@ -1168,25 +1173,35 @@ export async function handle( ...(rerankerStatus === 'unavailable' && { rerankerStatus: 'unavailable' }) }; - type SearchResponsePayload = { - status: 'success'; - searchQuality: typeof searchQualityBlock & { - tokenEstimate?: number; - warning?: string; - }; - budget: { mode: 'compact' | 'full'; resultCount: number }; - preflight?: typeof preflightPayload; - patternSummary?: string; - bestExample?: string; - nextHops?: Array<{ tool: string; why: string; args?: Record }>; - results: Array>; - totalResults?: number; - relatedMemories?: string[]; - }; - // Compact mode (default): bounded response with light graph context const isCompact = mode !== 'full'; + function buildSignaturePreview(snippet: string | undefined): string | undefined { + if (!snippet) return undefined; + + const lines = snippet.replace(/^\r?\n+/, '').split('\n'); + const previewTerms = [...new Set(queryTerms.filter((term) => term.length > 3))]; + const strongestMatch = lines + .map((line, index) => ({ + index, + matches: previewTerms.filter((term) => line.toLowerCase().includes(term)).length + })) + .filter((entry) => entry.matches > 0) + .sort((a, b) => b.matches - a.matches || a.index - b.index)[0]; + + let previewLines = lines.slice(0, 3); + if (strongestMatch) { + const windowLength = Math.min(3, lines.length); + const windowStart = Math.max( + 0, + Math.min(strongestMatch.index - 1, lines.length - windowLength) + ); + previewLines = lines.slice(windowStart, windowStart + windowLength); + } + + return previewLines.join('\n').trim() || undefined; + } + if (isCompact) { const compactResults = results.slice(0, 6); const strongMemories = filterStrongMemories(relatedMemories, queryStr); @@ -1196,6 +1211,7 @@ export async function handle( const payloadText = finalizeSearchPayloadText( { status: 'success', + ...(sourceCoverage.status === 'partial' && { sourceCoverage }), searchQuality: searchQualityBlock, budget: { mode: 'compact', resultCount: compactResults.length }, ...(preflightPayload && { preflight: preflightPayload }), @@ -1207,15 +1223,7 @@ export async function handle( const topExports = getTopExports(r.filePath); const scope = buildScopeHeader(r.metadata); const health = getResultHealth(r.filePath); - // First 3 lines of chunk content as a lightweight signature preview - const signaturePreview = r.snippet - ? r.snippet - .replace(/^\r?\n+/, '') - .split('\n') - .slice(0, 3) - .join('\n') - .trim() || undefined - : undefined; + const signaturePreview = buildSignaturePreview(r.snippet); return { file: `${r.filePath}:${r.startLine}-${r.endLine}`, summary: r.summary, @@ -1258,6 +1266,7 @@ export async function handle( const payloadText = finalizeSearchPayloadText( { status: 'success', + ...(sourceCoverage.status === 'partial' && { sourceCoverage }), searchQuality: searchQualityBlock, budget: { mode: 'full', resultCount: results.length }, ...(preflightPayload && { preflight: preflightPayload }), diff --git a/src/tools/types.ts b/src/tools/types.ts index 5de5ee8..43cabb8 100644 --- a/src/tools/types.ts +++ b/src/tools/types.ts @@ -1,5 +1,6 @@ import type { CodebaseIndexer } from '../core/indexer.js'; import type { IndexingStats } from '../types/index.js'; +import type { SearchSourceCoverage } from '../core/search.js'; export interface DecisionCard { ready: boolean; @@ -56,7 +57,11 @@ export interface ToolContext { paths: ToolPaths; rootPath: string; project?: ProjectDescriptor; - performIndexing: (incrementalOnly?: boolean, reason?: string) => void; + performIndexing: ( + incrementalOnly?: boolean, + reason?: string, + skipEmbedding?: boolean + ) => void | Promise; listProjects?: () => ProjectDescriptor[]; getActiveProject?: () => ProjectDescriptor | undefined; } @@ -132,6 +137,7 @@ export interface SearchResultItem { export interface SearchResponse { status: string; + sourceCoverage?: SearchSourceCoverage; searchQuality: SearchQuality; budget?: SearchBudget; preflight?: DecisionCard | SearchLitePreflight; diff --git a/src/types/index.ts b/src/types/index.ts index 9010843..619b5e9 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -21,7 +21,7 @@ export interface FrameworkAnalyzer { canAnalyze(filePath: string, content?: string): boolean; /** Parse a file and extract structured information */ - analyze(filePath: string, content: string): Promise; + analyze(filePath: string, content: string, options?: AnalysisOptions): Promise; /** Detect framework-specific patterns and metadata from the entire codebase */ detectCodebaseMetadata(rootPath: string): Promise; @@ -39,6 +39,10 @@ export interface FrameworkAnalyzer { getSnippetPattern?(category: string, name: string): RegExp | null; } +export interface AnalysisOptions { + includeScopePrefix?: boolean; +} + // ============================================================================ // ANALYSIS RESULTS // ============================================================================ diff --git a/src/utils/ast-chunker.ts b/src/utils/ast-chunker.ts index 9e7e8be..0d245f1 100644 --- a/src/utils/ast-chunker.ts +++ b/src/utils/ast-chunker.ts @@ -56,8 +56,10 @@ export const MAX_AST_CHUNK_FILE_LINES = 10_000; // 10K lines */ export function generateScopePrefix(node: SymbolNode, ancestors: SymbolNode[]): string { // Build scope path: ancestor names joined with " > ", then current node - const pathParts = ancestors.map((a) => a.symbol.name); - pathParts.push(node.symbol.name); + const pathParts = node.symbol.symbolPath ?? [ + ...ancestors.map((a) => a.symbol.name), + node.symbol.name + ]; const scopePath = pathParts.join(' > '); // Build signature hint from the symbol content @@ -69,7 +71,7 @@ export function generateScopePrefix(node: SymbolNode, ancestors: SymbolNode[]): /** * Extract a short signature hint from a symbol's content. */ -function extractSignatureHint(sym: TreeSitterSymbol): string { +function extractSignatureHint(sym: Pick): string { const kind = sym.kind; if (kind === 'class') return 'class'; @@ -132,13 +134,18 @@ export function buildSymbolTree(symbols: TreeSitterSymbol[]): SymbolNode[] { const cs = candidate.symbol; const ns = node.symbol; - // candidate must fully contain node (and not be identical range unless different symbol) - if ( - cs.startLine <= ns.startLine && - cs.endLine >= ns.endLine && - !(cs.startLine === ns.startLine && cs.endLine === ns.endLine) - ) { - const span = cs.endLine - cs.startLine; + // Real parser offsets disambiguate nested and sibling symbols on the same line. + // Synthetic callers without offsets retain the public line-range contract. + const hasOffsets = cs.endIndex > cs.startIndex && ns.endIndex > ns.startIndex; + const contains = hasOffsets + ? cs.startIndex <= ns.startIndex && + cs.endIndex >= ns.endIndex && + (cs.startIndex !== ns.startIndex || cs.endIndex !== ns.endIndex) + : cs.startLine <= ns.startLine && + cs.endLine >= ns.endLine && + !(cs.startLine === ns.startLine && cs.endLine === ns.endLine); + if (contains) { + const span = hasOffsets ? cs.endIndex - cs.startIndex : cs.endLine - cs.startLine; if (span < bestSpan) { bestSpan = span; bestParent = candidate; @@ -172,146 +179,111 @@ export function generateASTChunks( options: ASTChunkOptions ): CodeChunk[] { const lines = content.split('\n'); - const totalLines = lines.length; + if (symbols.length === 0) return [makeFillerChunk(lines, 1, lines.length, options)]; - if (symbols.length === 0) { - // No symbols: entire file is one filler chunk - return [makeFillerChunk(lines, 1, totalLines, options)]; + const lineStarts = [0]; + for (let i = 0; i < content.length; i++) { + if (content[i] === '\n') lineStarts.push(i + 1); } - - const roots = buildSymbolTree(symbols); - - // Sort roots by startLine for ordered processing - roots.sort((a, b) => a.symbol.startLine - b.symbol.startLine); - - const chunks: CodeChunk[] = []; - let cursor = 1; // 1-based line cursor - - const shouldPrefix = options.includeScopePrefix !== false; - - for (const root of roots) { - // Gap before this root - if (root.symbol.startLine > cursor) { - chunks.push(makeFillerChunk(lines, cursor, root.symbol.startLine - 1, options)); - } - // Process the root symbol (recurse for containers) - chunks.push(...processNode(root, lines, options, null, [], shouldPrefix)); - cursor = root.symbol.endLine + 1; - } - - // Suffix after last root - if (cursor <= totalLines) { - chunks.push(makeFillerChunk(lines, cursor, totalLines, options)); - } - - return chunks; -} - -/** Recursively process a SymbolNode into chunks. */ -function processNode( - node: SymbolNode, - lines: string[], - options: ASTChunkOptions, - parentName: string | null, - ancestors: SymbolNode[], - shouldPrefix: boolean -): CodeChunk[] { - const sym = node.symbol; - const symbolPath = parentName ? [parentName, sym.name] : [sym.name]; - - if (node.children.length === 0) { - // Leaf symbol → single chunk - const chunk = makeSymbolChunk(sym, lines, options, symbolPath, parentName); - if (shouldPrefix) { - const prefix = generateScopePrefix(node, ancestors); - chunk.content = prefix + '\n' + chunk.content; - } - if (chunk.metadata) { - (chunk.metadata as Record).symbolPath = symbolPath; + const lineAt = (offset: number): number => { + let low = 0; + let high = lineStarts.length; + while (low + 1 < high) { + const middle = Math.floor((low + high) / 2); + if (lineStarts[middle] <= offset) low = middle; + else high = middle; } - return [chunk]; - } - - // Container symbol — split into header, children, footer - const chunks: CodeChunk[] = []; - const sortedChildren = [...node.children].sort((a, b) => a.symbol.startLine - b.symbol.startLine); - - // Header: from container start to first child start - 1 - const headerEnd = sortedChildren[0].symbol.startLine - 1; - if (headerEnd >= sym.startLine) { - const headerLines = extractLines(lines, sym.startLine, headerEnd); - const nonBlank = headerLines.filter((l) => l.trim().length > 0).length; - if (nonBlank > 2) { - const headerChunk = makeSymbolChunk( - { - ...sym, - name: `${sym.name}:header`, - startLine: sym.startLine, - endLine: headerEnd, - content: headerLines.join('\n') - }, - lines, - options, - symbolPath, - parentName, - true // use provided content - ); - if (shouldPrefix) { - const prefix = generateScopePrefix(node, ancestors); - headerChunk.content = `${prefix}\n${headerChunk.content}`; + return low + 1; + }; + const ranges = new Map(); + for (const symbol of symbols) { + let start = lineStarts[symbol.startLine - 1] ?? content.length; + let end = symbol.endLine < lineStarts.length ? lineStarts[symbol.endLine] - 1 : content.length; + if (symbol.endIndex > symbol.startIndex) { + start = symbol.startIndex; + end = symbol.endIndex; + // web-tree-sitter releases may expose byte offsets or UTF-16 offsets. + // Match the already extracted source rather than guessing from non-ASCII text. + if (content.slice(start, end) !== symbol.content) { + const bytes = Buffer.from(content, 'utf8'); + start = bytes.subarray(0, symbol.startIndex).toString('utf8').length; + end = bytes.subarray(0, symbol.endIndex).toString('utf8').length; } - chunks.push(headerChunk); } + ranges.set(symbol, { start, end }); } - // Children - let childCursor = sortedChildren[0].symbol.startLine; - for (const child of sortedChildren) { - // Gap between children within container - if (child.symbol.startLine > childCursor) { - const gapStart = childCursor; - const gapEnd = child.symbol.startLine - 1; - const gapLines = extractLines(lines, gapStart, gapEnd); - const nonBlank = gapLines.filter((l) => l.trim().length > 0).length; - if (nonBlank > 0) { - chunks.push(makeFillerChunk(lines, gapStart, gapEnd, options)); - } + const chunks: CodeChunk[] = []; + const appendRange = ( + start: number, + end: number, + owner?: SymbolNode, + ancestors: SymbolNode[] = [], + suffix = '' + ): void => { + const originalStart = start; + const fragment = content.slice(start, end); + const first = fragment.search(/\S/); + if (first < 0) return; + const last = fragment.search(/\s*$/); + // Drop only whitespace crossing a line boundary. Same-line separators remain + // exact source; indentation on an adjacent symbol's line is not a body span. + if (fragment.slice(0, first).includes('\n')) start = lineStarts[lineAt(start + first) - 1]; + if (fragment.slice(last).includes('\n')) end = originalStart + last; + const startLine = lineAt(start); + const endLine = lineAt(Math.max(start, end - 1)); + const source = content.slice(start, end); + if (!owner) { + chunks.push({ ...makeFillerChunk(lines, startLine, endLine, options), content: source }); + return; } - chunks.push( - ...processNode(child, lines, options, sym.name, [...ancestors, node], shouldPrefix) + const symbol = owner.symbol; + const symbolPath = symbol.symbolPath ?? [ + ...ancestors.map((ancestor) => ancestor.symbol.name), + symbol.name + ]; + const chunk = makeSymbolChunk( + { ...symbol, name: symbol.name + suffix, startLine, endLine, content: source }, + lines, + options, + symbolPath, + symbolPath.at(-2) ?? null, + true ); - childCursor = child.symbol.endLine + 1; - } - - // Footer: from last child end + 1 to container end - const lastChildEnd = sortedChildren[sortedChildren.length - 1].symbol.endLine; - if (lastChildEnd < sym.endLine) { - const footerStart = lastChildEnd + 1; - const footerLines = extractLines(lines, footerStart, sym.endLine); - const nonBlank = footerLines.filter((l) => l.trim().length > 0).length; - if (nonBlank > 2) { - const footerChunk = makeSymbolChunk( - { - ...sym, - name: `${sym.name}:footer`, - startLine: footerStart, - endLine: sym.endLine, - content: footerLines.join('\n') - }, - lines, - options, - symbolPath, - parentName, - true - ); - if (shouldPrefix) { - const prefix = generateScopePrefix(node, ancestors); - footerChunk.content = `${prefix}\n${footerChunk.content}`; - } - chunks.push(footerChunk); + if (options.includeScopePrefix !== false) + chunk.content = generateScopePrefix(owner, ancestors) + '\n' + chunk.content; + chunks.push(chunk); + }; + const processNode = (node: SymbolNode, ancestors: SymbolNode[]): void => { + const range = ranges.get(node.symbol)!; + const children = [...node.children].sort( + (a, b) => ranges.get(a.symbol)!.start - ranges.get(b.symbol)!.start + ); + if (children.length === 0) { + appendRange(range.start, range.end, node, ancestors); + return; + } + let cursor = range.start; + for (let i = 0; i < children.length; i++) { + const child = children[i]; + const childRange = ranges.get(child.symbol)!; + appendRange(cursor, childRange.start, node, ancestors, i === 0 ? ':header' : ':body'); + processNode(child, [...ancestors, node]); + cursor = childRange.end; } + appendRange(cursor, range.end, node, ancestors, ':footer'); + }; + const roots = buildSymbolTree(symbols).sort( + (a, b) => ranges.get(a.symbol)!.start - ranges.get(b.symbol)!.start + ); + let cursor = 0; + for (const root of roots) { + const range = ranges.get(root.symbol)!; + appendRange(cursor, range.start); + processNode(root, []); + cursor = range.end; } - + appendRange(cursor, content.length); return chunks; } @@ -490,16 +462,43 @@ function findSafeSplitPoint(lines: string[], targetIdx: number): number { /** * Main public API: generate AST-aligned chunks from content + symbols, - * then merge tiny chunks and split oversized ones. + * then split oversized source spans and optionally add presentation context. */ export function createASTAlignedChunks( content: string, symbols: TreeSitterSymbol[], options: ASTChunkOptions ): CodeChunk[] { - const raw = generateASTChunks(content, symbols, options); - const merged = mergeSmallSymbolChunks(raw, options.minChunkLines); - const final = splitOversizedChunks(merged, options.maxChunkLines); + // Partition source before adding presentation context. Persisted callers opt out + // of the final prefix; they receive exactly the same source ranges and bytes. + const raw = generateASTChunks(content, symbols, { ...options, includeScopePrefix: false }); + // Even identical owner labels can describe distinct declarations. The public + // merge helper is retained for callers, but persisted source spans stay separate. + const final = splitOversizedChunks(raw, options.maxChunkLines); + if (options.includeScopePrefix !== false) { + const prefixes = new Map(); + for (const chunk of raw) { + if (!chunk.metadata.symbolAware) continue; + const path = chunk.metadata.symbolPath ?? []; + const key = JSON.stringify(path); + if (!prefixes.has(key)) { + const symbol = symbols.find( + (candidate) => + candidate.name === path.at(-1) && + candidate.startLine <= chunk.startLine && + candidate.endLine >= chunk.endLine + ); + prefixes.set( + key, + `// ${path.join(' > ')} :: ${extractSignatureHint(symbol ?? { kind: chunk.metadata.symbolKind ?? '', content: chunk.content })}` + ); + } + } + for (const chunk of final) { + const prefix = prefixes.get(JSON.stringify(chunk.metadata.symbolPath ?? [])); + if (chunk.metadata.symbolAware && prefix) chunk.content = `${prefix}\n${chunk.content}`; + } + } return final; } @@ -547,9 +546,10 @@ function makeSymbolChunk( parentName: string | null, useProvidedContent = false ): CodeChunk { - const chunkContent = useProvidedContent - ? sym.content - : extractLines(lines, sym.startLine, sym.endLine).join('\n'); + const chunkContent = + useProvidedContent || sym.endIndex > sym.startIndex + ? sym.content + : extractLines(lines, sym.startLine, sym.endLine).join('\n'); return { id: uuidv4(), diff --git a/src/utils/internal-source-resolver.ts b/src/utils/internal-source-resolver.ts new file mode 100644 index 0000000..a77dd92 --- /dev/null +++ b/src/utils/internal-source-resolver.ts @@ -0,0 +1,196 @@ +import { promises as fs } from 'fs'; +import path from 'path'; +import ts from 'typescript'; +import { parse } from '@typescript-eslint/typescript-estree'; + +export interface TsconfigResolverOptions { + baseUrl?: string; + paths?: Record; + compilerOptions?: ts.CompilerOptions; +} + +export interface SourceResolution { + status: 'resolved' | 'unresolved'; + path?: string; + reason?: 'external' | 'not-found' | 'outside-root' | 'unsupported-alias'; +} + +function isWithin(root: string, candidate: string): boolean { + const relative = path.relative(root, candidate); + return ( + relative === '' || + (relative !== '..' && !relative.startsWith(`..${path.sep}`) && !path.isAbsolute(relative)) + ); +} + +export function loadTsconfigCompilerOptions(rootPath: string): ts.CompilerOptions { + // Keep discovery within this root and config paths normalized for TypeScript diagnostics. + const configPath = ['tsconfig.json', 'tsconfig.base.json'] + .map((name) => path.join(rootPath, name).split(path.sep).join('/')) + .find((candidate) => ts.sys.fileExists(candidate)); + if (!configPath) { + return { + module: ts.ModuleKind.Preserve, + moduleResolution: ts.ModuleResolutionKind.Bundler + }; + } + const readResult = ts.readConfigFile(configPath, ts.sys.readFile); + if (readResult.error) { + return { + module: ts.ModuleKind.Preserve, + moduleResolution: ts.ModuleResolutionKind.Bundler + }; + } + const parsed = ts.parseJsonConfigFileContent(readResult.config, ts.sys, path.dirname(configPath)); + return { + module: ts.ModuleKind.Preserve, + moduleResolution: ts.ModuleResolutionKind.Bundler, + ...parsed.options + }; +} + +function matchesPathAlias( + specifier: string, + paths: Record +): Array<{ target: string; suffix: string }> { + const matches: Array<{ target: string; suffix: string; specificity: number }> = []; + for (const [pattern, targets] of Object.entries(paths)) { + const star = pattern.indexOf('*'); + if (star < 0) { + if (specifier === pattern) + for (const target of targets) + matches.push({ target, suffix: '', specificity: pattern.length }); + continue; + } + const prefix = pattern.slice(0, star); + const suffix = pattern.slice(star + 1); + if ( + specifier.startsWith(prefix) && + specifier.endsWith(suffix) && + specifier.length >= prefix.length + suffix.length + ) { + const capture = specifier.slice(prefix.length, specifier.length - suffix.length); + for (const target of targets) + matches.push({ + target: target.replace('*', capture), + suffix: '', + specificity: prefix.length + suffix.length + }); + } + } + return matches + .sort((a, b) => b.specificity - a.specificity) + .map(({ target, suffix }) => ({ target, suffix })); +} + +export class InternalSourceResolver { + private readonly rootPath: string; + private readonly rootRealPath: string; + private readonly indexedFiles = new Map(); + private readonly options: TsconfigResolverOptions; + private readonly compilerOptions: ts.CompilerOptions; + + private constructor( + rootPath: string, + rootRealPath: string, + indexedFiles: Map, + options: TsconfigResolverOptions + ) { + this.rootPath = rootPath; + this.rootRealPath = rootRealPath; + this.indexedFiles = indexedFiles; + this.options = options; + this.compilerOptions = options.compilerOptions ?? { + module: ts.ModuleKind.Preserve, + moduleResolution: ts.ModuleResolutionKind.Bundler, + ...(options.baseUrl ? { baseUrl: path.resolve(rootPath, options.baseUrl) } : {}), + ...(options.paths ? { paths: options.paths } : {}) + }; + } + + static async create( + rootPath: string, + files: string[], + options: TsconfigResolverOptions = {} + ): Promise { + const rootRealPath = await fs.realpath(rootPath); + const indexedFiles = new Map(); + for (const file of files) { + const absolute = path.resolve(file); + const relative = path.relative(rootPath, absolute); + if (!isWithin(rootPath, absolute)) continue; + const rel = relative.split(path.sep).join('/'); + indexedFiles.set(process.platform === 'win32' ? rel.toLowerCase() : rel, absolute); + const real = await fs.realpath(absolute).catch(() => null); + if (real && isWithin(rootRealPath, real)) { + const realRelative = path.relative(rootRealPath, real).split(path.sep).join('/'); + indexedFiles.set( + process.platform === 'win32' ? realRelative.toLowerCase() : realRelative, + absolute + ); + } + } + return new InternalSourceResolver(path.resolve(rootPath), rootRealPath, indexedFiles, options); + } + + async resolve( + importer: string, + specifier: string, + allowUnindexed = false + ): Promise { + const importerPath = path.resolve(importer); + const importerRealPath = await fs.realpath(importerPath).catch(() => null); + if (!importerRealPath || !isWithin(this.rootRealPath, importerRealPath)) + return { status: 'unresolved', reason: 'outside-root' }; + const matchingAlias = matchesPathAlias(specifier, this.compilerOptions.paths ?? {}); + const bareSpecifier = !specifier.startsWith('.') && !path.isAbsolute(specifier); + if (bareSpecifier && matchingAlias.length === 0 && !this.compilerOptions.baseUrl) { + return { status: 'unresolved', reason: 'external' }; + } + const resolved = ts.resolveModuleName(specifier, importerPath, this.compilerOptions, ts.sys) + .resolvedModule?.resolvedFileName; + if (!resolved) + return { + status: 'unresolved', + reason: matchingAlias.length > 0 ? 'not-found' : bareSpecifier ? 'external' : 'not-found' + }; + const real = await fs.realpath(resolved).catch(() => null); + if (!real) return { status: 'unresolved', reason: 'not-found' }; + if (!isWithin(this.rootRealPath, real)) { + return { + status: 'unresolved', + reason: bareSpecifier && matchingAlias.length === 0 ? 'external' : 'outside-root' + }; + } + const relative = path.relative(this.rootRealPath, real).split(path.sep).join('/'); + const indexedPath = this.indexedFiles.get( + process.platform === 'win32' ? relative.toLowerCase() : relative + ); + if (indexedPath) return { status: 'resolved', path: indexedPath }; + if (allowUnindexed) return { status: 'resolved', path: real }; + return { status: 'unresolved', reason: 'not-found' }; + } +} + +export function extractSourceReexports( + source: string, + filePath: string +): Array<{ specifier: string; line: number }> { + if (!/\.[cm]?[jt]sx?$/i.test(filePath)) return []; + if (!/\bexport\s+(?:type\s+)?(?:\*|\{)/.test(source)) return []; + try { + const ast = parse(source, { loc: true, jsx: /\.[jt]sx$/i.test(filePath) }); + const reexports: Array<{ specifier: string; line: number }> = []; + for (const node of ast.body) { + if ( + (node.type === 'ExportNamedDeclaration' || node.type === 'ExportAllDeclaration') && + node.source + ) { + reexports.push({ specifier: node.source.value, line: node.loc.start.line }); + } + } + return reexports; + } catch { + return []; + } +} diff --git a/src/utils/optional-dependency.ts b/src/utils/optional-dependency.ts new file mode 100644 index 0000000..707c503 --- /dev/null +++ b/src/utils/optional-dependency.ts @@ -0,0 +1,17 @@ +import { createRequire } from 'node:module'; +import { OptionalDependencyError } from '../errors/index.js'; + +const resolve = createRequire(import.meta.url).resolve; + +/** Check an opt-in dependency without loading its native runtime or model. */ +export function assertOptionalDependency(packageName: string, versionRange: string): void { + try { + resolve(packageName); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code === 'MODULE_NOT_FOUND' || code === 'ERR_MODULE_NOT_FOUND') { + throw new OptionalDependencyError(packageName, versionRange); + } + throw error; + } +} diff --git a/src/utils/tree-sitter.ts b/src/utils/tree-sitter.ts index f24d69c..a268c66 100644 --- a/src/utils/tree-sitter.ts +++ b/src/utils/tree-sitter.ts @@ -15,6 +15,7 @@ export interface TreeSitterSymbol { endIndex: number; content: string; nodeType: string; + symbolPath?: string[]; } export interface TreeSitterSymbolExtraction { @@ -42,6 +43,9 @@ const SYMBOL_CANDIDATE_NODE_TYPES = [ 'method', 'method_declaration', 'method_definition', + 'pair', + 'field_definition', + 'public_field_definition', 'struct_item', 'struct_specifier', 'trait_item', @@ -218,6 +222,10 @@ function maybeGetNameNode(node: Node): Node | null { } function extractNodeName(node: Node): string { + if (node.type === 'pair') return staticPropertyName(node.childForFieldName('key')) ?? 'anonymous'; + if (['method_definition', 'field_definition', 'public_field_definition'].includes(node.type)) { + return staticPropertyName(node.childForFieldName('name')) ?? 'anonymous'; + } const nameNode = maybeGetNameNode(node); if (nameNode?.text) { const normalized = normalizeSymbolName(nameNode.text); @@ -237,6 +245,114 @@ function extractNodeName(node: Node): string { return 'anonymous'; } +/** Read only literal property keys; a computed identifier is not its runtime value. */ +function staticPropertyName(node: Node | null): string | null { + if (!node) return null; + if (node.type === 'computed_property_name') { + const expression = node.namedChildren[0]; + return expression?.type === 'string' || expression?.type === 'number' + ? staticPropertyName(expression) + : null; + } + if (node.type === 'string') { + const text = node.text.slice(1, -1); + return text.replace( + /\\(?:u\{([\da-f]+)\}|u([\da-f]{4})|x([\da-f]{2})|([\s\S]))/gi, + ( + _match: string, + codePoint: string | undefined, + unicode: string | undefined, + hex: string | undefined, + escaped: string | undefined + ) => { + const digits = codePoint ?? unicode ?? hex; + if (digits) return String.fromCodePoint(Number.parseInt(digits, 16)); + const escapes: Record = { + n: '\n', + r: '\r', + t: '\t', + b: '\b', + f: '\f', + v: '\v', + '0': '\0', + '\n': '', + '\r': '' + }; + return escaped === undefined ? '' : (escapes[escaped] ?? escaped); + } + ); + } + return ['identifier', 'property_identifier', 'private_property_identifier', 'number'].includes( + node.type + ) + ? normalizeSymbolName(node.text) + : null; +} + +function isCallableValue(node: Node | null): boolean { + if (node?.type === 'parenthesized_expression') + return isCallableValue(node.namedChildren[0] ?? null); + return ( + node?.type === 'arrow_function' || + node?.type === 'function_expression' || + node?.type === 'generator_function' + ); +} + +/** Property descriptors keep their immediate named owner and implementation together. */ +function isCallableDescriptor(node: Node | null): boolean { + if (node?.type !== 'object') return false; + const fields = node.namedChildren.filter( + (child): child is Node => child !== null && child.type !== 'comment' + ); + const descriptorKeys = new Set(['value', 'get', 'set', 'writable', 'enumerable', 'configurable']); + const fieldKey = (field: Node): string | null => + staticPropertyName(field.childForFieldName(field.type === 'pair' ? 'key' : 'name')); + return ( + fields.length > 0 && + fields.every( + (field) => + (field.type === 'pair' || field.type === 'method_definition') && + descriptorKeys.has(fieldKey(field) ?? '') + ) && + fields.some((field) => { + return ( + ['value', 'get', 'set'].includes(fieldKey(field) ?? '') && + (field.type === 'method_definition' + ? field.childForFieldName('body') !== null + : isCallableValue(field.childForFieldName('value'))) + ); + }) + ); +} + +function getSymbolPath(node: Node): string[] { + const names: string[] = []; + let current: Node | null = node; + while (current) { + if ( + current.type === 'pair' || + current.type === 'variable_declarator' || + ((SYMBOL_CANDIDATE_NODE_TYPES as readonly string[]).includes(current.type) && + !['lexical_declaration', 'type_declaration'].includes(current.type)) + ) { + const name = extractNodeName(current); + // A dynamic property interrupts the static route; omitting it would falsely + // connect the inner declaration directly to an outer owner. + if ( + ['pair', 'method_definition', 'field_definition', 'public_field_definition'].includes( + current.type + ) && + name === 'anonymous' + ) + break; + if (name !== 'anonymous') names.unshift(name); + } + current = current.parent; + } + return names; +} + function isFunctionVariableDeclarator(node: Node): boolean { if (node.type !== 'variable_declarator') { return false; @@ -251,6 +367,20 @@ function isFunctionVariableDeclarator(node: Node): boolean { } function shouldSkipNode(language: string, node: Node): boolean { + if ( + (node.type === 'pair' || node.type === 'method_definition') && + node.parent?.parent?.type === 'pair' && + isCallableDescriptor(node.parent) + ) { + return true; + } + if (node.type === 'pair') { + const value = node.childForFieldName('value'); + return !isCallableValue(value) && !isCallableDescriptor(value); + } + if (node.type === 'field_definition' || node.type === 'public_field_definition') { + return !isCallableValue(node.childForFieldName('value')); + } if (node.type === 'variable_declarator') { return ( !['javascript', 'javascriptreact', 'typescript', 'typescriptreact'].includes(language) || @@ -288,7 +418,8 @@ function buildSymbol(node: Node, content: string): TreeSitterSymbol { startIndex: rangeNode.startIndex, endIndex: rangeNode.endIndex, content: extractNodeContent(rangeNode, content), - nodeType: node.type + nodeType: node.type, + symbolPath: getSymbolPath(node) }; } @@ -346,7 +477,7 @@ export async function extractTreeSitterSymbols( continue; } - const key = `${symbol.kind}:${symbol.name}:${symbol.startLine}:${symbol.endLine}`; + const key = `${symbol.kind}:${symbol.name}:${symbol.startIndex}:${symbol.endIndex}`; if (seen.has(key)) { continue; } @@ -389,6 +520,87 @@ export interface IdentifierOccurrence { nodeType: string; } +const LEXICAL_IDENTIFIER_CONTINUE = /(?:\p{ID_Continue}|[$]|\u200c|\u200d)/u; +const ASCII_IDENTIFIER_CONTINUE = /[A-Za-z0-9_$]/; +const ASCII_IDENTIFIER = /^[A-Za-z_$][A-Za-z0-9_$]*$/; + +/** + * Conservatively nominate source for identifier matching without parsing longer + * ASCII identifiers. Non-ASCII and non-identifier query spellings retain raw + * substring behavior, so grammar-specific and uncertain cases remain included. + */ +export function hasIdentifierCandidate(content: string, symbol: string): boolean { + const normalizedSymbol = symbol.trim(); + if (!normalizedSymbol) return false; + if (!ASCII_IDENTIFIER.test(normalizedSymbol)) return content.includes(normalizedSymbol); + + let searchFrom = 0; + while (searchFrom <= content.length - normalizedSymbol.length) { + const startIndex = content.indexOf(normalizedSymbol, searchFrom); + if (startIndex === -1) return false; + + const endIndex = startIndex + normalizedSymbol.length; + const previous = startIndex > 0 ? content[startIndex - 1] : undefined; + const next = endIndex < content.length ? content[endIndex] : undefined; + const hasLeftBoundary = previous === undefined || !ASCII_IDENTIFIER_CONTINUE.test(previous); + const hasRightBoundary = next === undefined || !ASCII_IDENTIFIER_CONTINUE.test(next); + if (hasLeftBoundary && hasRightBoundary) return true; + + searchFrom = startIndex + Math.max(1, normalizedSymbol.length); + } + return false; +} + +/** + * Conservative language-neutral token matching for files without a usable parser. + * This preserves whole identifier boundaries for Unicode, `_`, and `$` spellings. + */ +export function findLexicalIdentifierOccurrences( + content: string, + symbol: string +): IdentifierOccurrence[] { + const normalizedSymbol = symbol.trim(); + if (!normalizedSymbol) return []; + + const occurrences: IdentifierOccurrence[] = []; + let searchFrom = 0; + let lineCursor = 0; + let line = 1; + while (searchFrom <= content.length - normalizedSymbol.length) { + const startIndex = content.indexOf(normalizedSymbol, searchFrom); + if (startIndex === -1) break; + const endIndex = startIndex + normalizedSymbol.length; + const previousIndex = startIndex > 0 ? startIndex - 1 : -1; + const previousCodePointIndex = + previousIndex >= 0 && + content.charCodeAt(previousIndex) >= 0xdc00 && + content.charCodeAt(previousIndex) <= 0xdfff + ? previousIndex - 1 + : previousIndex; + const previousCodePoint = + previousCodePointIndex >= 0 + ? String.fromCodePoint(content.codePointAt(previousCodePointIndex) ?? 0) + : undefined; + const nextCodePoint = + endIndex < content.length + ? String.fromCodePoint(content.codePointAt(endIndex) ?? 0) + : undefined; + const hasIdentifierBoundary = + (!previousCodePoint || !LEXICAL_IDENTIFIER_CONTINUE.test(previousCodePoint)) && + (!nextCodePoint || !LEXICAL_IDENTIFIER_CONTINUE.test(nextCodePoint)); + + while (lineCursor < startIndex) { + if (content.charCodeAt(lineCursor) === 10) line += 1; + lineCursor += 1; + } + if (hasIdentifierBoundary) { + occurrences.push({ line, startIndex, endIndex, nodeType: 'lexical_identifier' }); + } + searchFrom = startIndex + Math.max(1, normalizedSymbol.length); + } + return occurrences; +} + const IDENTIFIER_NODE_TYPES = [ 'identifier', 'type_identifier', diff --git a/src/utils/usage-tracker.ts b/src/utils/usage-tracker.ts index 23bedcb..8889ba0 100644 --- a/src/utils/usage-tracker.ts +++ b/src/utils/usage-tracker.ts @@ -625,6 +625,184 @@ export interface UnusedExport { export interface ImportEdgeDetail { line?: number; importedSymbols?: string[]; + /** Source declarations that produced this adjacency. Legacy details have no proof and stay unverified. */ + sources?: ImportEdgeSource[]; +} + +export type GraphRelationKind = 'import' | 'reexport'; + +export interface ImportEdgeSource { + line: number; + specifier: string; + kind: GraphRelationKind; + sourceFingerprint: string; + targetFingerprint: string; +} + +export interface UnresolvedImportEdge { + from: string; + kind: GraphRelationKind; + source: ImportEdgeSource; + reason: 'external' | 'not-found' | 'outside-root' | 'unsupported-alias'; +} + +export interface FileGraphEdge { + from: string; + to: string; + kind: GraphRelationKind; + source: ImportEdgeSource; + verified: boolean; +} + +export interface FileGraphTraversal { + paths: Array<{ nodes: string[]; edges: FileGraphEdge[] }>; + unresolved: UnresolvedImportEdge[]; + truncated: boolean; +} + +interface TraversableFileGraph { + imports?: Record; + importDetails?: Record>; + unresolvedEdges?: UnresolvedImportEdge[]; +} + +interface FileGraphTraversalOptions { + direction: 'outgoing' | 'incoming'; + maxHops: 1 | 2 | 3; + maxNodes?: number; + maxPaths?: number; + /** Candidate edge attempts and unresolved diagnostics share this work budget. */ + maxInspections?: number; +} + +interface CheckedFileGraphTraversal extends FileGraphTraversal { + rejectedEdges: FileGraphEdge[]; + inspectionLimited: boolean; +} + +/** A shared BFS lets asynchronous freshness checks run before a path consumes output limits. */ +function* fileGraphTraversal( + graph: TraversableFileGraph, + start: string, + options: FileGraphTraversalOptions +): Generator { + const imports = graph.imports ?? {}; + const incoming = new Map(); + if (options.direction === 'incoming') { + // The sidecar stores forward adjacency only. Build its reverse lookup once, without source IO. + for (const [from, targets] of Object.entries(imports)) { + for (const to of targets) { + const importers = incoming.get(to); + if (importers) importers.push(from); + else incoming.set(to, [from]); + } + } + } + const neighbors = function* (node: string): Generator { + const targets = + options.direction === 'outgoing' ? (imports[node] ?? []) : (incoming.get(node) ?? []); + for (const neighbor of targets) { + const from = options.direction === 'outgoing' ? node : neighbor; + const to = options.direction === 'outgoing' ? neighbor : node; + const detail = graph.importDetails?.[from]?.[to]; + const sources = detail?.sources ?? []; + if (sources.length > 0) { + for (const source of sources) yield { from, to, kind: source.kind, source, verified: true }; + } else { + yield { + from, + to, + kind: 'import', + verified: false, + source: { + line: detail?.line ?? 1, + specifier: '', + kind: 'import', + sourceFingerprint: '', + targetFingerprint: '' + } + }; + } + } + }; + + const maxNodes = Math.max(1, Math.min(500, options.maxNodes ?? 100)); + const maxPaths = Math.max(1, Math.min(500, options.maxPaths ?? 200)); + const maxInspections = Math.max(1, Math.min(10000, options.maxInspections ?? 1000)); + const visitedNodes = new Set([start]); + const queue: Array<{ nodes: string[]; edges: FileGraphEdge[] }> = [{ nodes: [start], edges: [] }]; + const paths: FileGraphTraversal['paths'] = []; + const rejectedEdges: FileGraphEdge[] = []; + let truncated = false; + let inspections = 0; + let inspectionLimited = false; + traversal: for (let cursor = 0; cursor < queue.length; cursor++) { + const current = queue[cursor]!; + if (current.edges.length >= options.maxHops) continue; + for (const edge of neighbors(current.nodes[current.nodes.length - 1]!)) { + if (inspections >= maxInspections) { + truncated = inspectionLimited = true; + break traversal; + } + inspections++; + const next = options.direction === 'outgoing' ? edge.to : edge.from; + if (current.nodes.includes(next)) continue; + if (paths.length >= maxPaths) { + truncated = true; + break traversal; + } + if (!(yield edge)) { + rejectedEdges.push(edge); + continue; + } + if (!visitedNodes.has(next) && visitedNodes.size >= maxNodes) { + truncated = true; + continue; + } + visitedNodes.add(next); + const nextPath = { + nodes: [...current.nodes, next], + edges: [...current.edges, edge] + }; + paths.push(nextPath); + queue.push(nextPath); + } + } + const unresolved: UnresolvedImportEdge[] = []; + for (const edge of graph.unresolvedEdges ?? []) { + if (!visitedNodes.has(edge.from)) continue; + if (inspections >= maxInspections) { + truncated = inspectionLimited = true; + break; + } + inspections++; + unresolved.push(edge); + } + return { paths, unresolved, truncated, rejectedEdges, inspectionLimited }; +} + +export function traverseFileGraph( + graph: TraversableFileGraph, + start: string, + options: FileGraphTraversalOptions +): FileGraphTraversal { + const traversal = fileGraphTraversal(graph, start, options); + let step = traversal.next(); + while (!step.done) step = traversal.next(true); + const { paths, unresolved, truncated } = step.value; + return { paths, unresolved, truncated }; +} + +export async function traverseFileGraphWithVerification( + graph: TraversableFileGraph, + start: string, + options: FileGraphTraversalOptions, + acceptEdge: (edge: FileGraphEdge) => Promise +): Promise { + const traversal = fileGraphTraversal(graph, start, options); + let step = traversal.next(); + while (!step.done) step = traversal.next(await acceptEdge(step.value)); + return step.value; } export class InternalFileGraph { @@ -636,6 +814,7 @@ export class InternalFileGraph { private importedSymbols: Map> = new Map(); // Map: fromFile -> toFile -> edge details (line/symbols) private importDetails: Map> = new Map(); + private unresolvedEdges: UnresolvedImportEdge[] = []; // Root path for relative path conversion private rootPath: string; @@ -674,7 +853,8 @@ export class InternalFileGraph { importingFile: string, importedFile: string, line?: number, - importedSymbols?: string[] + importedSymbols?: string[], + sourceProof?: Omit ): void { const fromFile = this.normalizePath(importingFile); const toFile = this.normalizePath(importedFile); @@ -694,7 +874,7 @@ export class InternalFileGraph { ? importedSymbols.filter((sym) => sym !== '*' && sym !== 'default') : []; - if (normalizedLine || normalizedSymbols.length > 0) { + if (normalizedLine || normalizedSymbols.length > 0 || sourceProof) { if (!this.importDetails.has(fromFile)) { this.importDetails.set(fromFile, new Map()); } @@ -710,9 +890,31 @@ export class InternalFileGraph { for (const sym of normalizedSymbols) mergedSymbolsSet.add(sym); const mergedSymbols = Array.from(mergedSymbolsSet); + const sources = existing.sources ? [...existing.sources] : []; + if (sourceProof && normalizedLine) { + const nextSource: ImportEdgeSource = { ...sourceProof, line: normalizedLine }; + if ( + !sources.some( + (source) => + source.line === nextSource.line && + source.specifier === nextSource.specifier && + source.kind === nextSource.kind + ) + ) { + sources.push(nextSource); + } + } + detailsForFrom.set(toFile, { ...(mergedLine ? { line: mergedLine } : {}), - ...(mergedSymbols.length > 0 ? { importedSymbols: mergedSymbols.sort() } : {}) + ...(mergedSymbols.length > 0 ? { importedSymbols: mergedSymbols.sort() } : {}), + ...(sources.length > 0 + ? { + sources: sources.sort( + (a, b) => a.line - b.line || a.specifier.localeCompare(b.specifier) + ) + } + : {}) }); } @@ -727,6 +929,30 @@ export class InternalFileGraph { } } + trackUnresolved( + importingFile: string, + source: ImportEdgeSource, + reason: UnresolvedImportEdge['reason'], + kind: GraphRelationKind = source.kind + ): void { + const edge: UnresolvedImportEdge = { + from: this.normalizePath(importingFile), + kind, + source, + reason + }; + if ( + !this.unresolvedEdges.some( + (existing) => + existing.from === edge.from && + existing.source.line === edge.source.line && + existing.source.specifier === edge.source.specifier + ) + ) { + this.unresolvedEdges.push(edge); + } + } + /** * Track exports from a file */ @@ -865,6 +1091,7 @@ export class InternalFileGraph { imports: Record; exports: Record; importDetails?: Record>; + unresolvedEdges?: UnresolvedImportEdge[]; stats: { files: number; edges: number; avgDependencies: number }; } { const imports: Record = {}; @@ -894,6 +1121,7 @@ export class InternalFileGraph { imports, exports, ...(Object.keys(importDetails).length > 0 ? { importDetails } : {}), + ...(this.unresolvedEdges.length > 0 ? { unresolvedEdges: this.unresolvedEdges } : {}), stats: this.getStats() }; } @@ -906,6 +1134,7 @@ export class InternalFileGraph { imports?: Record; exports?: Record; importDetails?: Record>; + unresolvedEdges?: UnresolvedImportEdge[]; }, rootPath: string ): InternalFileGraph { @@ -944,6 +1173,11 @@ export class InternalFileGraph { } } + graph.unresolvedEdges = (data.unresolvedEdges ?? []).map((edge) => ({ + ...edge, + from: graph.normalizePath(edge.from) + })); + return graph; } } diff --git a/tests/analyzer-registry.test.ts b/tests/analyzer-registry.test.ts index 703ceb7..202007a 100644 --- a/tests/analyzer-registry.test.ts +++ b/tests/analyzer-registry.test.ts @@ -5,6 +5,8 @@ import { NextJsAnalyzer } from '../src/analyzers/nextjs/index'; import { NestJsAnalyzer } from '../src/analyzers/nestjs/index'; import { ReactAnalyzer } from '../src/analyzers/react/index'; import { GenericAnalyzer } from '../src/analyzers/generic/index'; +import angularRoutingFixture from './fixtures/angular-routing-regressions.json'; +import { createHash } from 'node:crypto'; // Register default analyzers analyzerRegistry.register(new AngularAnalyzer()); @@ -13,6 +15,45 @@ analyzerRegistry.register(new NestJsAnalyzer()); analyzerRegistry.register(new ReactAnalyzer()); analyzerRegistry.register(new GenericAnalyzer()); +describe('frozen Angular-local routing source facts', () => { + const registry = new AnalyzerRegistry(); + const angular = new AngularAnalyzer(); + registry.register(angular); + registry.register(new NextJsAnalyzer()); + registry.register(new NestJsAnalyzer()); + registry.register(new ReactAnalyzer()); + registry.register(new GenericAnalyzer()); + + for (const fixture of angularRoutingFixture.cases) { + it(`routes according to actual source evidence: ${fixture.id}`, async () => { + const filePath = `/virtual/${fixture.filePath}`; + expect(angular.canAnalyze(filePath, fixture.source)).toBe(fixture.expectedAngularEligible); + expect(registry.findAnalyzer(filePath, fixture.source)?.name).toBe(fixture.expectedAnalyzer); + if ('origin' in fixture && fixture.origin) { + expect(createHash('sha256').update(fixture.source, 'utf8').digest('hex')).toBe( + fixture.origin.sourceSha256 + ); + } + if ('expectedSourceSymbols' in fixture && fixture.expectedSourceSymbols) { + const result = await registry.analyzeFile(filePath, fixture.source, { + includeScopePrefix: false + }); + expect(result?.metadata.analyzer).toBe('generic'); + expect(result?.metadata.chunkStrategy).toBe('ast-aligned'); + for (const name of fixture.expectedSourceSymbols) { + const symbol = result?.chunks.find((chunk) => chunk.metadata.symbolName === name); + expect(symbol, name).toBeDefined(); + expect(symbol!.metadata.symbolAware).toBe(true); + expect(symbol!.metadata.symbolPath).toContain(name); + expect(fixture.source).toContain(symbol!.content); + expect(symbol!.startLine).toBeGreaterThanOrEqual(1); + expect(symbol!.endLine).toBeLessThanOrEqual(fixture.source.split('\n').length); + } + } + }); + } +}); + describe('AnalyzerRegistry', () => { describe('missing analyzer logging', () => { it('should stay quiet by default when no analyzer matches', async () => { @@ -110,10 +151,56 @@ export class PaymentsService {} expect( analyzerRegistry.findAllAnalyzers('/tmp/payments.service.ts', content).map((a) => a.name) - ).toEqual(['nestjs', 'angular', 'generic']); + ).toEqual(['nestjs', 'generic']); expect(analyzerRegistry.findAnalyzer('/tmp/payments.service.ts', content)?.name).toBe( 'nestjs' ); }); + + it('selects Generic for plain TypeScript generics and retains symbol-aware definitions', async () => { + const source = ` +interface ProjectConfiguration { targets?: Record } +const map = new Map(); + +export function readTargetDefaultsForTarget( + project: ProjectConfiguration, + targetName: string +) { + return map.get(targetName) ?? project.targets?.[targetName]; +} +`; + + expect(analyzerRegistry.findAnalyzer('/tmp/target-defaults.ts', source)?.name).toBe( + 'generic' + ); + const result = await analyzerRegistry.analyzeFile('/tmp/target-defaults.ts', source); + + expect(result?.metadata.analyzer).toBe('generic'); + expect( + result?.chunks.some( + (chunk) => + chunk.metadata.symbolName === 'readTargetDefaultsForTarget' && + chunk.metadata.symbolAware === true + ) + ).toBe(true); + }); + + it('preserves quoted names in TypeScript imports and exports', async () => { + const source = `import { "remote-name" as localName } from 'some-package'; +export { localName as "public-name" };`; + const result = await new GenericAnalyzer().analyze('/tmp/quoted-names.ts', source); + + expect(result.imports).toMatchObject([{ source: 'some-package', imports: ['remote-name'] }]); + expect(result.exports).toContainEqual({ + name: 'public-name', + isDefault: false, + type: 'named' + }); + + const angularResult = await new AngularAnalyzer().analyze('/tmp/quoted-names.ts', source); + expect(angularResult.imports).toMatchObject([ + { source: 'some-package', imports: ['remote-name'] } + ]); + }); }); }); diff --git a/tests/ast-chunker-integration.test.ts b/tests/ast-chunker-integration.test.ts index 16ee73e..6ea2360 100644 --- a/tests/ast-chunker-integration.test.ts +++ b/tests/ast-chunker-integration.test.ts @@ -1,6 +1,9 @@ import { describe, expect, it } from 'vitest'; import { GenericAnalyzer } from '../src/analyzers/generic/index'; import { MAX_AST_CHUNK_FILE_LINES } from '../src/utils/ast-chunker'; +import { createASTAlignedChunks } from '../src/utils/ast-chunker'; +import { extractTreeSitterSymbols } from '../src/utils/tree-sitter'; +import structuralFixture from './fixtures/structural-context-regressions.json'; // --------------------------------------------------------------------------- // Fixtures @@ -69,6 +72,197 @@ export function createUserService(db: Database): UserService { const analyzer = new GenericAnalyzer(); +describe('frozen structural raw chunks', () => { + for (const descriptor of [ + { + name: 'concise-value', + fields: 'value() { closeConnections(); }', + markers: ['closeConnections()'] + }, + { name: 'concise-get', fields: 'get() { return readState(); }', markers: ['readState()'] }, + { + name: 'concise-set', + fields: 'set(next) { writeState(next); }', + markers: ['writeState(next)'] + }, + { + name: 'mixed-accessors', + fields: 'get() { return readState(); }, set: next => writeState(next), enumerable: true', + markers: ['readState()', 'writeState(next)'] + } + ]) { + it(`keeps the nearest concise descriptor owner with its body: ${descriptor.name}`, async () => { + const source = `function createHandlers() { return { shutdown: { ${descriptor.fields} }, sibling: { value() { reloadConnections(); } }, services: { billing: { reconcile() { persistInvoice(); } } } }; }`; + const parsed = await extractTreeSitterSymbols(source, 'javascript'); + expect(parsed).not.toBeNull(); + const owner = parsed!.symbols.find((symbol) => symbol.name === 'shutdown'); + expect(owner).toBeDefined(); + expect(owner!.symbolPath).toEqual(['createHandlers', 'shutdown']); + expect(owner!.content).toBe(`shutdown: { ${descriptor.fields} }`); + const chunks = createASTAlignedChunks(source, parsed!.symbols, { + filePath: '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/virtual/concise-descriptor.js', + language: 'javascript', + minChunkLines: 10, + maxChunkLines: 150, + includeScopePrefix: false + }); + const implementation = chunks.find((chunk) => chunk.metadata.symbolName === 'shutdown'); + expect(implementation).toBeDefined(); + expect(implementation!.metadata.symbolPath).toEqual(['createHandlers', 'shutdown']); + expect(implementation!.content).toBe(owner!.content); + for (const marker of descriptor.markers) expect(implementation!.content).toContain(marker); + expect(implementation!.content).not.toContain('reloadConnections()'); + expect(implementation!.content).not.toContain('persistInvoice()'); + expect( + chunks.find((chunk) => chunk.metadata.symbolName === 'reconcile')?.metadata.symbolPath + ).toEqual(['createHandlers', 'services', 'billing', 'reconcile']); + expect( + parsed!.symbols.some( + (symbol) => symbol.symbolPath?.join('.') === 'createHandlers.shutdown.value' + ) + ).toBe(false); + }); + } + + for (const fields of [ + 'value() { closeConnections(); }, unknown: true', + '[operation]() { closeConnections(); }' + ]) { + it(`does not coalesce an unresolved descriptor shape: ${fields}`, async () => { + const source = `function createHandlers() { return { shutdown: { ${fields} } }; }`; + const parsed = await extractTreeSitterSymbols(source, 'javascript'); + expect(parsed).not.toBeNull(); + expect(parsed!.symbols.some((symbol) => symbol.name === 'shutdown')).toBe(false); + }); + } + + it('preserves same-line container statements before and after a nested callback', async () => { + const source = 'function start(){ before(); const hooks={onReady:()=>openQueue()}; after(); }'; + const parsed = await extractTreeSitterSymbols(source, 'javascript'); + const chunks = createASTAlignedChunks(source, parsed!.symbols, { + filePath: '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/virtual/container.js', + language: 'javascript', + minChunkLines: 10, + maxChunkLines: 150, + includeScopePrefix: false + }); + expect(chunks.map((chunk) => chunk.content).join('')).toBe(source); + const callback = chunks.find((chunk) => chunk.metadata.symbolName === 'onReady'); + expect(callback).toBeDefined(); + expect(callback!.content).toContain('openQueue()'); + expect(callback!.content).not.toContain('before()'); + expect(callback!.content).not.toContain('after()'); + for (const chunk of chunks) { + expect(chunk.startLine).toBe(1); + expect(chunk.endLine).toBe(1); + expect(source).toContain(chunk.content); + } + }); + + for (const fixture of structuralFixture.cases) { + it(`keeps raw owner bodies isolated: ${fixture.id}`, async () => { + const parsed = await extractTreeSitterSymbols(fixture.source, fixture.language); + expect(parsed).not.toBeNull(); + const chunks = createASTAlignedChunks(fixture.source, parsed!.symbols, { + filePath: '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/virtual/structural.ts', + language: fixture.language, + minChunkLines: 10, + maxChunkLines: 150, + includeScopePrefix: false + }); + for (const fact of fixture.sourceSymbols) { + if ('scopeOnly' in fact && fact.scopeOnly) { + expect( + chunks.some((chunk) => + fact.bodyMarkers.every((marker) => chunk.content.includes(marker)) + ) + ).toBe(true); + continue; + } + const matching = chunks.filter( + (chunk) => JSON.stringify(chunk.metadata.symbolPath) === JSON.stringify(fact.ownerChain) + ); + expect(matching.length, `${fixture.id}: ${fact.ownerChain.join(' > ')}`).toBeGreaterThan(0); + for (const marker of fact.bodyMarkers) + expect(matching.some((chunk) => chunk.content.includes(marker))).toBe(true); + for (const chunk of matching) { + expect(fixture.source).toContain(chunk.content); + if ('forbiddenSiblingBodyMarkers' in fact) { + for (const marker of fact.forbiddenSiblingBodyMarkers ?? []) + expect(chunk.content).not.toContain(marker); + } + } + } + }); + } + + it('coalesces only an immediate callable descriptor, without swallowing singleton wrappers', async () => { + const source = [ + 'function buildHandlers() {', + ' return {', + ' shutdown: {', + ' configurable: true,', + ' value: () => { closeConnections(); }', + ' },', + ' services: { billing: { reconcile: () => persistInvoice() } }', + ' };', + '}' + ].join('\n'); + const parsed = await extractTreeSitterSymbols(source, 'javascript'); + const chunks = createASTAlignedChunks(source, parsed!.symbols, { + filePath: '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/virtual/descriptor.js', + language: 'javascript', + minChunkLines: 10, + maxChunkLines: 150, + includeScopePrefix: false + }); + const shutdown = chunks.find((chunk) => chunk.metadata.symbolName === 'shutdown'); + expect(shutdown).toBeDefined(); + expect(shutdown!.content).toContain('shutdown: {'); + expect(shutdown!.content).toContain('closeConnections()'); + expect(shutdown!.content).not.toContain('persistInvoice()'); + const reconcile = chunks.find((chunk) => chunk.metadata.symbolName === 'reconcile'); + expect(reconcile).toBeDefined(); + expect(reconcile!.content).not.toContain('services:'); + expect(reconcile!.content).not.toContain('billing:'); + }); + + it('splits raw source before display prefixes without shifting source coordinates', async () => { + const source = [ + 'function consume() {', + '// consume :: ()', + ...Array.from({ length: 175 }, (_, i) => ` step${i}();`), + '}' + ].join('\n'); + const parsed = await extractTreeSitterSymbols(source, 'javascript'); + const options = { + filePath: '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/virtual/large.js', + language: 'javascript', + minChunkLines: 10, + maxChunkLines: 35 + }; + const raw = createASTAlignedChunks(source, parsed!.symbols, { + ...options, + includeScopePrefix: false + }); + const displayed = createASTAlignedChunks(source, parsed!.symbols, options); + expect(displayed.map(({ startLine, endLine }) => [startLine, endLine])).toEqual( + raw.map(({ startLine, endLine }) => [startLine, endLine]) + ); + for (const chunk of raw) { + expect(chunk.content).toBe( + source + .split('\n') + .slice(chunk.startLine - 1, chunk.endLine) + .join('\n') + ); + expect(chunk.endLine - chunk.startLine + 1).toBeLessThanOrEqual(35); + } + expect(raw.map((chunk) => chunk.content).join('\n')).toBe(source); + expect(raw[0].content).toContain('// consume :: ()'); + }); +}); + describe('AST Chunker Integration', () => { // Test 1: Supported language, normal file — AST chunks with scope prefixes it('produces AST-aligned chunks with scope prefixes for a normal TypeScript file', async () => { diff --git a/tests/auto-refresh-e2e.test.ts b/tests/auto-refresh-e2e.test.ts index 6e6a7bc..b848ea6 100644 --- a/tests/auto-refresh-e2e.test.ts +++ b/tests/auto-refresh-e2e.test.ts @@ -146,7 +146,7 @@ describe('Auto-refresh E2E', () => { expect(updatedContent).toContain('UPDATED_TOKEN'); expect(incrementalRuns).toBeGreaterThan(0); } finally { - stopWatcher(); + await stopWatcher(); } }, 20000); }); diff --git a/tests/cli-entrypoint-runtime.test.ts b/tests/cli-entrypoint-runtime.test.ts index 7bdb291..1c353ac 100644 --- a/tests/cli-entrypoint-runtime.test.ts +++ b/tests/cli-entrypoint-runtime.test.ts @@ -1,5 +1,8 @@ import { spawnSync } from 'node:child_process'; -import { resolve } from 'node:path'; +import { mkdtempSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; +import { pathToFileURL } from 'node:url'; import { describe, expect, it } from 'vitest'; const root = resolve(import.meta.dirname, '..'); @@ -54,4 +57,69 @@ describe('CLI entrypoint runtime', () => { expect(parsed.rootPath).toBe(root); expect(parsed.hint).toContain('refresh_index'); }, 120_000); + + it('dispatches CLI help when invoked through a symlink entrypoint', (context) => { + const tempDirectory = mkdtempSync(join(tmpdir(), 'cbc-cli-entrypoint-')); + const linkedEntrypoint = join(tempDirectory, 'codebase-context.ts'); + + try { + symlinkSync(entrypoint, linkedEntrypoint, 'file'); + } catch (error) { + const code = + typeof error === 'object' && + error !== null && + 'code' in error && + typeof error.code === 'string' + ? error.code + : undefined; + rmSync(tempDirectory, { recursive: true, force: true }); + if (process.platform === 'win32' && ['EPERM', 'EACCES', 'ENOSYS'].includes(code ?? '')) { + context.skip(); + return; + } + throw error; + } + + try { + const result = spawnSync(process.execPath, ['--import', 'tsx', linkedEntrypoint, '--help'], { + cwd: root, + env: { ...process.env, CODEBASE_ROOT: root }, + encoding: 'utf8', + timeout: 120_000 + }); + + expect(result.status).toBe(0); + expect(result.stdout).toContain('codebase-context [options]'); + expect(result.stdout).toContain('init [--client ]'); + expect(result.stderr).not.toContain('ERR_MODULE_NOT_FOUND'); + } finally { + rmSync(tempDirectory, { recursive: true, force: true }); + } + }, 120_000); + + it('does not run the server or CLI when imported by an unrelated index.js wrapper', () => { + const tempDirectory = mkdtempSync(join(tmpdir(), 'cbc-cli-import-')); + const wrapper = join(tempDirectory, 'index.js'); + const importUrl = pathToFileURL(entrypoint).href; + writeFileSync( + wrapper, + `process.argv[2] = '--help'; import(${JSON.stringify(importUrl)}).then(() => console.log('IMPORT_COMPLETE'), (error) => { console.error(error); process.exitCode = 1; });`, + 'utf8' + ); + + try { + const result = spawnSync(process.execPath, ['--import', 'tsx', wrapper], { + cwd: root, + env: { ...process.env, CODEBASE_ROOT: root }, + encoding: 'utf8', + timeout: 120_000 + }); + + expect(result.status).toBe(0); + expect(result.stdout.trim()).toBe('IMPORT_COMPLETE'); + expect(result.stderr).toBe(''); + } finally { + rmSync(tempDirectory, { recursive: true, force: true }); + } + }, 120_000); }); diff --git a/tests/cli-init-bootstrap.test.ts b/tests/cli-init-bootstrap.test.ts new file mode 100644 index 0000000..bb25481 --- /dev/null +++ b/tests/cli-init-bootstrap.test.ts @@ -0,0 +1,158 @@ +import path from 'node:path'; +import * as fs from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { describe, expect, it } from 'vitest'; +import { + detectClients, + generateMcpConfig, + handleInitCli, + _buildMergedMcpContent, + _buildMergedCodexContent +} from '../src/cli-init.js'; +import { resolveRootPath } from '../src/index.js'; + +describe('init bootstrap root', () => { + it('passes the configured project root through the MCP server argv path', () => { + const projectRoot = path.resolve('C:/projects/An App With Spaces'); + const previousArgv = process.argv; + const previousEnvRoot = process.env.CODEBASE_ROOT; + const generated = generateMcpConfig('claude-code', { + rootPath: projectRoot, + platform: 'linux', + packageVersion: '9.8.7' + }); + if (generated.kind !== 'file') throw new Error('expected project MCP config'); + + const parsed = JSON.parse(generated.content) as { + mcpServers: { 'codebase-context': { command: string; args: string[] } }; + }; + const server = parsed.mcpServers['codebase-context']; + expect(server.command).toBe('npx'); + expect(server.args).toEqual(['--yes', 'codebase-context@9.8.7', projectRoot]); + process.argv = ['node', 'codebase-context', server.args.at(-1) ?? '']; + process.env.CODEBASE_ROOT = path.resolve('C:/projects/Other App'); + + try { + expect(resolveRootPath()).toBe(projectRoot); + } finally { + process.argv = previousArgv; + if (previousEnvRoot === undefined) delete process.env.CODEBASE_ROOT; + else process.env.CODEBASE_ROOT = previousEnvRoot; + } + }); +}); + +describe('init preserves custom launchers', () => { + it('distinguishes executable files from directories while detecting tools', async () => { + const root = await fs.mkdtemp(path.join(tmpdir(), 'cbc-detect-files-')); + const bin = path.join(root, 'bin'); + const home = path.join(root, 'home'); + await fs.mkdir(path.join(bin, 'claude'), { recursive: true }); + await fs.writeFile(path.join(root, '.codex'), 'This is a file, not a tool directory.'); + const env = { + PATH: bin, + HOME: home, + USERPROFILE: home, + XDG_CONFIG_HOME: path.join(home, '.config') + }; + try { + expect(await detectClients(root, env)).toEqual([]); + const executable = path.join(bin, process.platform === 'win32' ? 'codex.cmd' : 'codex'); + await fs.writeFile(executable, 'Detection must never execute this file.'); + await fs.chmod(executable, 0o755); + expect(await detectClients(root, env)).toEqual(['codex']); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } + }); + + it('leaves a matching Codex launch and extra settings byte-for-byte unchanged', async () => { + const root = await fs.mkdtemp(path.join(tmpdir(), 'cbc-codex-format-')); + const generated = generateMcpConfig('codex', { rootPath: root }); + const original = ( + 'model = "example"\n\n' + + generated.content + + 'custom_label = "keep"\n' + ).replace(/\n/g, '\r\n'); + await fs.mkdir(path.dirname(generated.path), { recursive: true }); + await fs.writeFile(generated.path, original); + try { + await handleInitCli(['--client', 'codex', '--yes', '--root', root]); + expect(await fs.readFile(generated.path, 'utf8')).toBe(original); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } + }); + + it.each([ + ['claude-code', 'other package'], + ['codex', 'other package'], + ['claude-code', 'custom npx'], + ['codex', 'custom npx'] + ] as const)('preserves %s config with a %s launcher', async (client, kind) => { + const root = await fs.mkdtemp(path.join(tmpdir(), 'cbc-custom-launch-')); + const generated = generateMcpConfig(client, { rootPath: root }); + const launcher = path.join(root, 'node_modules', 'other-package', 'dist', 'index.js'); + const command = kind === 'custom npx' ? path.join(root, 'custom', 'npx.cmd') : 'node'; + const args = + kind === 'custom npx' ? ['--yes', 'codebase-context@2.3.0', root] : [launcher, root]; + const original = + client === 'codex' + ? `[mcp_servers.codebase-context]\ncommand = ${JSON.stringify(command)}\nargs = ${JSON.stringify(args)}\nstartup_timeout_sec = 75\n` + : JSON.stringify({ + mcpServers: { + 'codebase-context': { + command, + args, + env: { KEEP: 'yes' } + } + }, + customSetting: true + }); + await fs.mkdir(path.dirname(generated.path), { recursive: true }); + await fs.writeFile(generated.path, original); + try { + const merge = + client === 'codex' + ? _buildMergedCodexContent(generated.path, generated.content) + : _buildMergedMcpContent(generated.path, generated.content, client); + await expect(merge).rejects.toThrow('different codebase-context'); + expect(await fs.readFile(generated.path, 'utf8')).toBe(original); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } + }); + + it('writes no selected configs when a later client uses a custom launcher', async () => { + const root = await fs.mkdtemp(path.join(tmpdir(), 'cbc-batch-launch-')); + const claude = path.join(root, '.mcp.json'); + const codex = path.join(root, '.codex', 'config.toml'); + const originalClaude = JSON.stringify({ + mcpServers: { other: { command: 'custom' } }, + customSetting: true + }); + const originalCodex = `[mcp_servers.codebase-context]\ncommand = "node"\nargs = [${JSON.stringify(path.join(root, 'other-package', 'dist', 'index.js'))}, ${JSON.stringify(root)}]\n`; + await fs.mkdir(path.dirname(codex), { recursive: true }); + await fs.writeFile(claude, originalClaude); + await fs.writeFile(codex, originalCodex); + const previousExitCode = process.exitCode; + try { + process.exitCode = undefined; + await handleInitCli([ + '--client', + 'claude-code', + '--client', + 'codex', + '--yes', + '--root', + root + ]); + expect(process.exitCode).toBe(1); + expect(await fs.readFile(claude, 'utf8')).toBe(originalClaude); + expect(await fs.readFile(codex, 'utf8')).toBe(originalCodex); + } finally { + process.exitCode = previousExitCode; + await fs.rm(root, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/cli-init-windows.test.ts b/tests/cli-init-windows.test.ts new file mode 100644 index 0000000..50c765f --- /dev/null +++ b/tests/cli-init-windows.test.ts @@ -0,0 +1,57 @@ +import { mkdtemp, writeFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { describe, expect, it } from 'vitest'; +import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'; +import { generateMcpConfig } from '../src/cli-init.js'; + +describe.skipIf(process.platform !== 'win32')('Windows init launch', () => { + it.each(['An App', 'App&MissingCbcLaunchCommand', 'App%cbc_literal%End'])( + 'passes the repository path %s as one literal argument', + async (name) => { + const directory = await mkdtemp(path.join(tmpdir(), 'cbc-launch-')); + const root = path.join(directory, name); + await writeFile( + path.join(directory, 'npx.cmd'), + '@echo off\r\nnode "%~dp0capture.cjs" %*\r\n' + ); + await writeFile( + path.join(directory, 'capture.cjs'), + 'console.log(JSON.stringify({jsonrpc:"2.0",method:"captured",params:{args:process.argv.slice(2)}}));' + ); + const generated = generateMcpConfig('claude-code', { rootPath: root }); + const server = JSON.parse(generated.content).mcpServers['codebase-context'] as { + command: string; + args: string[]; + }; + const transport = new StdioClientTransport({ + ...server, + env: { PATH: directory + path.delimiter + process.env.PATH, cbc_literal: 'expanded' }, + stderr: 'pipe' + }); + const captured = new Promise((resolve, reject) => { + transport.onmessage = (message) => resolve(message); + transport.onerror = reject; + }); + try { + await transport.start(); + const result = await Promise.race([ + captured, + new Promise((_, reject) => { + const timer = setTimeout( + () => reject(new Error('Launch did not return arguments')), + 3000 + ); + timer.unref(); + }) + ]); + expect(result).toMatchObject({ + params: { args: ['--yes', expect.stringMatching(/^codebase-context@/), root] } + }); + } finally { + await transport.close(); + await rm(directory, { recursive: true, force: true }); + } + } + ); +}); diff --git a/tests/cli-init.test.ts b/tests/cli-init.test.ts index 1dc718e..1e3b11c 100644 --- a/tests/cli-init.test.ts +++ b/tests/cli-init.test.ts @@ -1,5 +1,10 @@ -import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { describe, it, expect, vi, beforeEach, afterAll } from 'vitest'; import path from 'path'; +import { readFileSync } from 'node:fs'; + +const { version: packageVersion } = JSON.parse( + readFileSync(new URL('../package.json', import.meta.url), 'utf8') +) as { version: string }; // Mock node:fs/promises at the top level so Vitest can hoist it correctly. vi.mock('node:fs/promises', async (importOriginal) => { @@ -7,149 +12,138 @@ vi.mock('node:fs/promises', async (importOriginal) => { return { ...actual, access: vi.fn(), + stat: vi.fn(), readFile: vi.fn(), writeFile: vi.fn().mockResolvedValue(undefined), mkdir: vi.fn().mockResolvedValue(undefined) }; }); +vi.mock('@inquirer/prompts', () => ({ + checkbox: vi.fn() +})); + import * as fsMod from 'node:fs/promises'; +import { checkbox } from '@inquirer/prompts'; import { + CLIENTS, generateMcpConfig, - generateInstructionBlock, - resolveInstructionFilePath, - _appendInstructionBlock, - _buildMergedMcpContent + detectClients, + resolveInitRoot, + _buildMergedMcpContent, + _buildMergedCodexContent, + handleInitCli } from '../src/cli-init.js'; // --- generateMcpConfig --- describe('generateMcpConfig', () => { - it('claude-code returns a command with mcp add --transport http', () => { - const result = generateMcpConfig('claude-code'); - expect(result.kind).toBe('command'); - if (result.kind !== 'command') return; - expect(result.args[0]).toBe('mcp'); - expect(result.args).toContain('--transport'); - expect(result.args).toContain('http'); - expect(result.args).toContain('codebase-context'); - expect(result.args).toContain('http://127.0.0.1:3100/mcp'); - }); - - it('cursor returns a file at .cursor/mcp.json with http type', () => { - const result = generateMcpConfig('cursor'); + const projectRoot = path.resolve('C:/projects/Example Project'); + + it('generates a version-pinned npx.cmd launch for Claude Code on Windows', () => { + const result = generateMcpConfig('claude-code', { + rootPath: projectRoot, + platform: 'win32', + packageVersion: '9.8.7' + }); + expect(result.kind).toBe('file'); if (result.kind !== 'file') return; - expect(result.path).toBe('.cursor/mcp.json'); - const parsed = JSON.parse(result.content) as { - mcpServers: { 'codebase-context': { type: string } }; - }; - expect(parsed.mcpServers['codebase-context'].type).toBe('http'); - }); - - it('codex returns a command with mcp add', () => { - const result = generateMcpConfig('codex'); - expect(result.kind).toBe('command'); - if (result.kind !== 'command') return; - expect(result.args).toContain('mcp'); - expect(result.args).toContain('add'); - expect(result.args).toContain('codebase-context'); - expect(result.args).toContain('http://127.0.0.1:3100/mcp'); + expect(result.path).toBe(path.join(projectRoot, '.mcp.json')); + expect(JSON.parse(result.content)).toEqual({ + mcpServers: { + 'codebase-context': { + command: 'npx.cmd', + args: ['--yes', 'codebase-context@9.8.7', projectRoot] + } + } + }); }); - it('opencode returns a file at opencode.json with remote type', () => { - const result = generateMcpConfig('opencode'); + it('Cursor gets a project stdio config', () => { + const result = generateMcpConfig('cursor', { rootPath: projectRoot, platform: 'linux' }); expect(result.kind).toBe('file'); if (result.kind !== 'file') return; - expect(result.path).toBe('opencode.json'); + expect(result.path).toBe(path.join(projectRoot, '.cursor', 'mcp.json')); const parsed = JSON.parse(result.content) as { - mcp: { 'codebase-context': { type: string } }; + mcpServers: { 'codebase-context': { type: string; command: string; args: string[] } }; }; - expect(parsed.mcp['codebase-context'].type).toBe('remote'); - }); -}); - -// --- generateInstructionBlock --- - -describe('generateInstructionBlock', () => { - it('contains opening and closing delimiters', () => { - const block = generateInstructionBlock(); - expect(block).toContain(''); - expect(block).toContain(''); - }); - - it('contains all five tool call rules', () => { - const block = generateInstructionBlock(); - expect(block).toContain('get_memory'); - expect(block).toContain('search_codebase'); - expect(block).toContain('get_team_patterns'); - expect(block).toContain('remember'); - expect(block).toContain('detect_circular_dependencies'); + expect(parsed.mcpServers['codebase-context']).toEqual({ + type: 'stdio', + command: 'npx', + args: ['--yes', `codebase-context@${packageVersion}`, projectRoot] + }); }); -}); - -// --- resolveInstructionFilePath --- - -describe('resolveInstructionFilePath', () => { - const cwd = '/test-cwd'; - it('claude-code resolves to CLAUDE.md', () => { - const result = resolveInstructionFilePath('claude-code', cwd); - expect(result).toBe(path.join(cwd, 'CLAUDE.md')); + it('defaults to a pinned npx launch using the current platform', () => { + const result = generateMcpConfig('claude-code', { rootPath: projectRoot }); + if (result.kind !== 'file') throw new Error('expected file config'); + const parsed = JSON.parse(result.content) as { + mcpServers: { 'codebase-context': { command: string; args: string[] } }; + }; + const server = parsed.mcpServers['codebase-context']; + const windows = process.platform === 'win32'; + expect(server.command).toBe(windows ? 'npx.cmd' : 'npx'); + expect(server.args).toEqual([ + '--yes', + expect.stringMatching(/^codebase-context@\d+\.\d+\.\d+/), + projectRoot + ]); }); - it('cursor resolves to .cursorrules', () => { - const result = resolveInstructionFilePath('cursor', cwd); - expect(result).toBe(path.join(cwd, '.cursorrules')); - }); + it('Codex allows a cold package install before connecting', () => { + const result = generateMcpConfig('codex', { + rootPath: projectRoot, + platform: 'linux', + packageVersion: '9.8.7' + }); - it('codex resolves to AGENTS.md', () => { - const result = resolveInstructionFilePath('codex', cwd); - expect(result).toBe(path.join(cwd, 'AGENTS.md')); + expect(result.kind).toBe('file'); + if (result.kind !== 'file') return; + expect(result.path).toBe(path.join(projectRoot, '.codex', 'config.toml')); + expect(result.content).toBe( + `[mcp_servers.codebase-context]\ncommand = "npx"\nargs = ["--yes", "codebase-context@9.8.7", ${JSON.stringify(projectRoot)}]\nstartup_timeout_sec = 300\n` + ); }); - it('opencode returns null', () => { - const result = resolveInstructionFilePath('opencode', cwd); - expect(result).toBeNull(); + it('OpenCode uses a local stdio command array', () => { + const result = generateMcpConfig('opencode', { rootPath: projectRoot, platform: 'linux' }); + expect(result.kind).toBe('file'); + if (result.kind !== 'file') return; + expect(result.path).toBe(path.join(projectRoot, 'opencode.json')); + const parsed = JSON.parse(result.content) as { + mcp: { 'codebase-context': { type: string; command: string[] } }; + }; + expect(parsed.mcp['codebase-context']).toEqual({ + type: 'local', + command: ['npx', '--yes', `codebase-context@${packageVersion}`, projectRoot] + }); }); }); -// --- _appendInstructionBlock file-write behavior --- - -describe('_appendInstructionBlock', () => { - const accessMock = vi.mocked(fsMod.access); - const readFileMock = vi.mocked(fsMod.readFile); - const writeFileMock = vi.mocked(fsMod.writeFile); - - beforeEach(() => { - vi.resetAllMocks(); - writeFileMock.mockResolvedValue(undefined); +describe('resolveInitRoot', () => { + it('uses explicit root before CODEBASE_ROOT and cwd', () => { + expect( + resolveInitRoot(['--root', 'C:/projects/Explicit Project'], 'C:/projects/Env', 'C:/other') + ).toBe(path.resolve('C:/projects/Explicit Project')); }); - it('appends block when file exists and block not yet present', async () => { - // Simulate file exists (access resolves) and has content without the delimiter - accessMock.mockResolvedValue(undefined); - readFileMock.mockResolvedValue('# existing content' as unknown as Buffer); - - await _appendInstructionBlock('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/test/CLAUDE.md'); - - expect(writeFileMock).toHaveBeenCalledOnce(); - const callArgs = writeFileMock.mock.calls[0] as [string, string, ...unknown[]]; - const content = callArgs[1]; - expect(content).toContain('# existing content'); - expect(content).toContain(''); + it('uses CODEBASE_ROOT when no explicit root is supplied', () => { + expect(resolveInitRoot([], 'C:/projects/Env Project', 'C:/other')).toBe( + path.resolve('C:/projects/Env Project') + ); }); - it('skips when block already present', async () => { - accessMock.mockResolvedValue(undefined); - readFileMock.mockResolvedValue( - '# existing content\n\nsome block\n' as unknown as Buffer + it('uses cwd when neither root override is supplied', () => { + expect(resolveInitRoot([], undefined, 'C:/projects/Current Project')).toBe( + path.resolve('C:/projects/Current Project') ); + }); - const result = await _appendInstructionBlock('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/test/CLAUDE.md'); - - expect(result).toBe('skipped'); - expect(writeFileMock).not.toHaveBeenCalled(); + it('treats an empty CODEBASE_ROOT as unset', () => { + expect(resolveInitRoot([], '', 'C:/projects/Current Project')).toBe( + path.resolve('C:/projects/Current Project') + ); }); }); @@ -170,12 +164,16 @@ describe('_buildMergedMcpContent', () => { }) as unknown as Buffer ); - const generated = generateMcpConfig('cursor'); + const generated = generateMcpConfig('cursor', { platform: 'linux' }); if (generated.kind !== 'file') throw new Error('expected file config'); - const merged = await _buildMergedMcpContent('/test/.cursor/mcp.json', generated.content, 'cursor'); + const merged = await _buildMergedMcpContent( + '/test/.cursor/mcp.json', + generated.content, + 'cursor' + ); const parsed = JSON.parse(merged.content) as { - mcpServers: Record; + mcpServers: Record; someOtherKey: boolean; }; @@ -185,13 +183,73 @@ describe('_buildMergedMcpContent', () => { type: 'http', url: 'http://127.0.0.1:4000/mcp' }); + expect(parsed.mcpServers['codebase-context']).toMatchObject({ type: 'stdio', command: 'npx' }); + }); + + it('preserves existing Claude Code mcpServers entries and adds codebase-context', async () => { + readFileMock.mockResolvedValue( + JSON.stringify({ + mcpServers: { + 'existing-server': { type: 'stdio', command: 'node', args: ['server.js'] } + }, + someOtherKey: true + }) as unknown as Buffer + ); + + const generated = generateMcpConfig('claude-code', { + rootPath: path.resolve('C:/projects/app'), + platform: 'win32' + }); + if (generated.kind !== 'file') throw new Error('expected file config'); + + const merged = await _buildMergedMcpContent( + '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/test/.mcp.json', + generated.content, + 'claude-code' + ); + const parsed = JSON.parse(merged.content) as { + mcpServers: Record; + someOtherKey: boolean; + }; + + expect(merged.mergedFromExisting).toBe(true); + expect(parsed.someOtherKey).toBe(true); + expect(parsed.mcpServers['existing-server']).toEqual({ + type: 'stdio', + command: 'node', + args: ['server.js'] + }); expect(parsed.mcpServers['codebase-context']).toEqual({ - type: 'http', - url: 'http://127.0.0.1:3100/mcp' + command: 'npx.cmd', + args: ['--yes', `codebase-context@${packageVersion}`, path.resolve('C:/projects/app')] + }); + }); + + it('refuses to overwrite a disabled Claude Code codebase-context entry', async () => { + readFileMock.mockResolvedValue( + JSON.stringify({ + mcpServers: { + 'codebase-context': { + type: 'stdio', + command: 'node', + args: ['server.js'], + disabled: true + } + } + }) as unknown as Buffer + ); + const generated = generateMcpConfig('claude-code', { + rootPath: path.resolve('C:/projects/app'), + platform: 'win32' }); + if (generated.kind !== 'file') throw new Error('expected file config'); + + await expect( + _buildMergedMcpContent('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/test/.mcp.json', generated.content, 'claude-code') + ).rejects.toThrow(/already contains a different codebase-context configuration/i); }); - it('preserves existing opencode mcp entries and updates codebase-context deterministically', async () => { + it('refuses to overwrite an existing conflicting OpenCode codebase-context entry', async () => { readFileMock.mockResolvedValue( JSON.stringify({ $schema: 'https://opencode.ai/config.json', @@ -206,21 +264,574 @@ describe('_buildMergedMcpContent', () => { const generated = generateMcpConfig('opencode'); if (generated.kind !== 'file') throw new Error('expected file config'); - const merged = await _buildMergedMcpContent('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/test/opencode.json', generated.content, 'opencode'); + await expect( + _buildMergedMcpContent('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/test/opencode.json', generated.content, 'opencode') + ).rejects.toThrow(/already contains a different codebase-context configuration/i); + }); + + it('treats a custom Cursor HTTP entry as a conflict', async () => { + readFileMock.mockResolvedValue( + JSON.stringify({ + mcpServers: { 'codebase-context': { type: 'http', url: 'https://custom/mcp' } } + }) as unknown as Buffer + ); + const generated = generateMcpConfig('cursor', { rootPath: '/test', platform: 'linux' }); + if (generated.kind !== 'file') throw new Error('expected file config'); + + await expect( + _buildMergedMcpContent('/test/.cursor/mcp.json', generated.content, 'cursor') + ).rejects.toThrow(/already contains a different codebase-context configuration/i); + }); + + it('updates a generated JSON launch when the project root moves and preserves entry extras', async () => { + const oldRoot = path.resolve('C:/projects/Old App'); + const newRoot = path.resolve('C:/projects/New App'); + readFileMock.mockResolvedValue( + JSON.stringify({ + customSetting: true, + mcpServers: { + 'codebase-context': { + command: 'C:/node.exe', + args: ['C:/cache/node_modules/codebase-context/dist/index.js', oldRoot], + customTimeout: 8 + } + } + }) as unknown as Buffer + ); + const generated = generateMcpConfig('claude-code', { + rootPath: newRoot, + platform: 'win32', + packageVersion: '9.8.7' + }); + if (generated.kind !== 'file') throw new Error('expected file config'); + + const merged = await _buildMergedMcpContent( + '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/test/.mcp.json', + generated.content, + 'claude-code' + ); + const parsed = JSON.parse(merged.content) as { + customSetting: boolean; + mcpServers: { 'codebase-context': Record }; + }; + expect(parsed.customSetting).toBe(true); + expect(parsed.mcpServers['codebase-context']).toEqual({ + command: 'npx.cmd', + args: ['--yes', 'codebase-context@9.8.7', newRoot], + customTimeout: 8 + }); + }); + + it('updates a previously generated pinned npx command when the project root moves', async () => { + const oldRoot = path.resolve('C:/projects/Old Pinned App'); + const newRoot = path.resolve('C:/projects/New Pinned App'); + readFileMock.mockResolvedValue( + JSON.stringify({ + mcpServers: { + 'codebase-context': { + command: 'npx', + args: ['--yes', 'codebase-context@2.3.0', oldRoot], + customLabel: 'preserve' + } + } + }) as unknown as Buffer + ); + const generated = generateMcpConfig('claude-code', { + rootPath: newRoot, + platform: 'linux', + packageVersion: '9.8.7' + }); + if (generated.kind !== 'file') throw new Error('expected file config'); + + const merged = await _buildMergedMcpContent( + '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/test/.mcp.json', + generated.content, + 'claude-code' + ); const parsed = JSON.parse(merged.content) as { - mcp: Record; - extra: string; + mcpServers: { 'codebase-context': { command: string; args: string[]; customLabel: string } }; }; + expect(parsed.mcpServers['codebase-context']).toEqual({ + command: 'npx', + args: ['--yes', 'codebase-context@9.8.7', newRoot], + customLabel: 'preserve' + }); + }); +}); + +describe('detectClients', () => { + const accessMock = vi.mocked(fsMod.access); + const missing = () => Object.assign(new Error('missing'), { code: 'ENOENT' }); + + beforeEach(() => { + vi.resetAllMocks(); + accessMock.mockRejectedValue(missing()); + vi.mocked(fsMod.stat).mockImplementation(async (candidate) => { + const executable = String(candidate) + .replace(/\\/g, '/') + .toLowerCase() + .startsWith('c:/tools/'); + return { isFile: () => executable, isDirectory: () => !executable } as Awaited< + ReturnType + >; + }); + }); + + it('uses one four-client table and detects installed tools through PATH candidates', async () => { + accessMock.mockImplementation(async (candidate) => { + const pathValue = String(candidate).toLowerCase(); + if (pathValue.startsWith('c:\\tools\\') && pathValue.endsWith('codex.exe')) return undefined; + throw missing(); + }); + + const clients = await detectClients('/project', { PATH: 'C:/tools' }, 'win32'); + + expect(CLIENTS.map(({ id }) => id)).toEqual(['claude-code', 'codex', 'cursor', 'opencode']); + expect(clients).toContain('codex'); + expect(clients).not.toContain('claude-code'); + expect( + accessMock.mock.calls.some(([candidate]) => String(candidate).includes('C:\\tools')) + ).toBe(true); + }); + + it.each([ + { + name: 'CODEX_HOME', + env: { PATH: '', CODEX_HOME: 'C:/custom-codex-home' }, + marker: 'custom-codex-home', + client: 'codex' + }, + { + name: 'CLAUDE_CONFIG_DIR', + env: { PATH: '', CLAUDE_CONFIG_DIR: 'C:/custom-claude-config' }, + marker: 'custom-claude-config', + client: 'claude-code' + }, + { + name: 'XDG_CONFIG_HOME', + env: { PATH: '', XDG_CONFIG_HOME: 'C:/custom-xdg-config' }, + marker: 'custom-xdg-config', + client: 'opencode' + } + ])('checks $name when detecting installed clients', async ({ env, marker, client }) => { + accessMock.mockImplementation(async (candidate) => { + if (String(candidate).toLowerCase().includes(marker)) return undefined; + throw missing(); + }); + + const clients = await detectClients('/project', env, 'win32'); + + expect(clients).toContain(client); + expect(accessMock.mock.calls.some(([candidate]) => String(candidate).includes(marker))).toBe( + true + ); + }); +}); + +describe('handleInitCli', () => { + const readFileMock = vi.mocked(fsMod.readFile); + const writeFileMock = vi.mocked(fsMod.writeFile); + const mkdirMock = vi.mocked(fsMod.mkdir); + const accessMock = vi.mocked(fsMod.access); + const statMock = vi.mocked(fsMod.stat); + const previousExitCode = process.exitCode; + const previousIsTTY = process.stdin.isTTY; + const previousStdoutIsTTY = process.stdout.isTTY; + + const missing = () => Object.assign(new Error('missing'), { code: 'ENOENT' }); + const detectDirectories = (...names: string[]) => { + accessMock.mockImplementation(async (candidate) => { + const basename = String(candidate).toLowerCase().split(/[\\/]/).at(-1); + if (names.some((name) => basename === (name === 'opencode' ? 'opencode' : `.${name}`))) + return undefined; + throw missing(); + }); + }; + + beforeEach(() => { + vi.resetAllMocks(); + writeFileMock.mockResolvedValue(undefined); + mkdirMock.mockResolvedValue(undefined); + readFileMock.mockRejectedValue(missing()); + accessMock.mockRejectedValue(missing()); + statMock.mockResolvedValue({ isDirectory: () => true } as Awaited< + ReturnType + >); + process.exitCode = undefined; + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: false }); + Object.defineProperty(process.stdout, 'isTTY', { configurable: true, value: false }); + }); + + afterAll(() => { + process.exitCode = previousExitCode; + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: previousIsTTY }); + Object.defineProperty(process.stdout, 'isTTY', { + configurable: true, + value: previousStdoutIsTTY + }); + }); + + it('configures all detected clients with --yes', async () => { + const projectRoot = path.resolve('C:/projects/Configured App'); + detectDirectories('claude', 'codex'); + + await handleInitCli(['--yes', '--root', projectRoot]); + + expect(checkbox).not.toHaveBeenCalled(); + expect(writeFileMock.mock.calls.map(([filePath]) => filePath)).toEqual( + expect.arrayContaining([ + path.join(projectRoot, '.mcp.json'), + path.join(projectRoot, '.codex', 'config.toml') + ]) + ); + expect(writeFileMock).toHaveBeenCalledTimes(2); + }); + + it('configures all detected clients without a terminal even without --yes', async () => { + const projectRoot = path.resolve('C:/projects/No Terminal'); + detectDirectories('cursor', 'opencode'); + + await handleInitCli(['--root', projectRoot]); + + expect(checkbox).not.toHaveBeenCalled(); + expect(writeFileMock.mock.calls.map(([filePath]) => filePath)).toEqual( + expect.arrayContaining([ + path.join(projectRoot, '.cursor', 'mcp.json'), + path.join(projectRoot, 'opencode.json') + ]) + ); + expect(writeFileMock).toHaveBeenCalledTimes(2); + }); + + it('lets repeated --client options override detection', async () => { + const projectRoot = path.resolve('C:/projects/Explicit Clients'); + + await handleInitCli([ + '--client', + 'claude-code', + '--client', + 'codex', + '--yes', + '--root', + projectRoot + ]); + + expect(accessMock).not.toHaveBeenCalled(); + expect(writeFileMock.mock.calls.map(([filePath]) => filePath)).toEqual( + expect.arrayContaining([ + path.join(projectRoot, '.mcp.json'), + path.join(projectRoot, '.codex', 'config.toml') + ]) + ); + expect(writeFileMock).toHaveBeenCalledTimes(2); + }); + + it('reports no detected clients without writing when there is no terminal', async () => { + await handleInitCli(['--yes', '--root', path.resolve('C:/projects/Empty')]); + + expect(writeFileMock).not.toHaveBeenCalled(); + expect(mkdirMock).not.toHaveBeenCalled(); + expect(process.exitCode).toBe(1); + }); + + it('shows detected clients checked in the terminal multi-select', async () => { + detectDirectories('claude', 'codex'); + vi.mocked(checkbox).mockResolvedValue(['claude-code']); + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: true }); + Object.defineProperty(process.stdout, 'isTTY', { configurable: true, value: true }); + + await handleInitCli(['--root', path.resolve('C:/projects/Interactive')]); + + const choices = vi.mocked(checkbox).mock.calls[0]?.[0].choices; + expect(choices).toEqual( + expect.arrayContaining([ + expect.objectContaining({ value: 'claude-code', checked: true }), + expect.objectContaining({ value: 'codex', checked: true }), + expect.objectContaining({ value: 'cursor', checked: false }), + expect.objectContaining({ value: 'opencode', checked: false }) + ]) + ); + expect(writeFileMock).toHaveBeenCalledOnce(); + }); + + it('leaves all no-detection terminal choices unchecked and writes nothing on empty selection', async () => { + vi.mocked(checkbox).mockResolvedValue([]); + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: true }); + Object.defineProperty(process.stdout, 'isTTY', { configurable: true, value: true }); + + await handleInitCli(['--root', path.resolve('C:/projects/Cancelled')]); + + const choices = vi.mocked(checkbox).mock.calls[0]?.[0].choices; + expect(choices).toEqual( + expect.arrayContaining([ + expect.objectContaining({ value: 'claude-code', checked: false }), + expect.objectContaining({ value: 'codex', checked: false }), + expect.objectContaining({ value: 'cursor', checked: false }), + expect.objectContaining({ value: 'opencode', checked: false }) + ]) + ); + expect(writeFileMock).not.toHaveBeenCalled(); + expect(mkdirMock).not.toHaveBeenCalled(); + expect(process.exitCode).toBe(1); + }); + + it('does not write when the terminal prompt throws its cancellation error', async () => { + detectDirectories('claude'); + vi.mocked(checkbox).mockRejectedValue( + Object.assign(new Error('User force closed the prompt'), { name: 'ExitPromptError' }) + ); + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: true }); + Object.defineProperty(process.stdout, 'isTTY', { configurable: true, value: true }); + + await handleInitCli(['--root', path.resolve('C:/projects/Prompt Cancelled')]); + + expect(writeFileMock).not.toHaveBeenCalled(); + expect(mkdirMock).not.toHaveBeenCalled(); + }); + + it('requires an existing directory for --root', async () => { + statMock.mockResolvedValue({ isDirectory: () => false } as Awaited< + ReturnType + >); + + await handleInitCli(['--client', 'codex', '--yes', '--root', path.resolve('C:/projects/File')]); + + expect(mkdirMock).not.toHaveBeenCalled(); + expect(writeFileMock).not.toHaveBeenCalled(); + expect(process.exitCode).toBe(1); + }); + + it('does not create configs when --root does not exist', async () => { + statMock.mockRejectedValue(missing()); + + await handleInitCli([ + '--client', + 'codex', + '--yes', + '--root', + path.resolve('C:/projects/Missing Directory') + ]); + + expect(mkdirMock).not.toHaveBeenCalled(); + expect(writeFileMock).not.toHaveBeenCalled(); + expect(process.exitCode).toBe(1); + }); + + it('leaves an identical generated config untouched', async () => { + const projectRoot = path.resolve('C:/projects/Already Configured'); + const generated = generateMcpConfig('claude-code', { rootPath: projectRoot }); + readFileMock.mockResolvedValue(generated.content as unknown as Buffer); + + await handleInitCli(['--client', 'claude-code', '--yes', '--root', projectRoot]); + + expect(writeFileMock).not.toHaveBeenCalled(); + }); + + it('prepares every selected config before writing when a later client conflicts', async () => { + const projectRoot = path.resolve('C:/projects/Batch Conflict'); + const opencodePath = path.join(projectRoot, 'opencode.json'); + readFileMock.mockImplementation(async (filePath) => { + if (String(filePath) === opencodePath) { + return JSON.stringify({ + mcp: { 'codebase-context': { type: 'remote', url: 'https://custom' } } + }) as unknown as Buffer; + } + throw missing(); + }); + + await handleInitCli([ + '--client', + 'claude-code', + '--client', + 'opencode', + '--yes', + '--root', + projectRoot + ]); + + expect(process.exitCode).toBe(1); + expect(mkdirMock).not.toHaveBeenCalled(); + expect(writeFileMock).not.toHaveBeenCalled(); + }); + + it('shows help without prompting or writing files', async () => { + await handleInitCli(['--help']); + + expect(checkbox).not.toHaveBeenCalled(); + expect(writeFileMock).not.toHaveBeenCalled(); + }); +}); + +describe('_buildMergedCodexContent', () => { + const readFileMock = vi.mocked(fsMod.readFile); + const writeFileMock = vi.mocked(fsMod.writeFile); + const mkdirMock = vi.mocked(fsMod.mkdir); + const statMock = vi.mocked(fsMod.stat); + + beforeEach(() => { + vi.resetAllMocks(); + writeFileMock.mockResolvedValue(undefined); + mkdirMock.mockResolvedValue(undefined); + statMock.mockResolvedValue({ isDirectory: () => true } as Awaited< + ReturnType + >); + readFileMock.mockRejectedValue(Object.assign(new Error('missing'), { code: 'ENOENT' })); + }); + + const runCodexSetupWithExistingConfig = async (existingToml: string) => { + const projectRoot = path.resolve('C:/projects/Codex Existing'); + const initialBytes = Buffer.from(existingToml, 'utf8'); + let persistedBytes = Buffer.from(initialBytes); + readFileMock.mockResolvedValue(existingToml as unknown as Buffer); + writeFileMock.mockImplementation(async (_filePath, content) => { + persistedBytes = Buffer.from(content as string, 'utf8'); + }); + + const previousExitCode = process.exitCode; + process.exitCode = undefined; + try { + await handleInitCli(['--client', 'codex', '--yes', '--root', projectRoot]); + return { initialBytes, persistedBytes, exitCode: process.exitCode }; + } finally { + process.exitCode = previousExitCode; + } + }; + + it('preserves unrelated Codex TOML settings when adding the project server', async () => { + readFileMock.mockResolvedValue( + '[model]\nmodel = "gpt-example"\n\n[mcp_servers.other]\ncommand = "other"\n' as unknown as Buffer + ); + const generated = generateMcpConfig('codex', { + rootPath: 'C:/projects/app', + platform: 'win32' + }); + if (generated.kind !== 'file') throw new Error('expected file config'); + + const merged = await _buildMergedCodexContent(generated.path, generated.content); expect(merged.mergedFromExisting).toBe(true); - expect(parsed.extra).toBe('value'); - expect(parsed.mcp['existing-server']).toEqual({ - type: 'remote', - url: 'http://127.0.0.1:5000/mcp' + expect(merged.content).toContain('[model]\nmodel = "gpt-example"'); + expect(merged.content).toContain('[mcp_servers.other]\ncommand = "other"'); + expect(merged.content).toContain('[mcp_servers.codebase-context]'); + }); + + it('leaves the same Codex codebase-context entry unchanged on a repeated setup', async () => { + const generated = generateMcpConfig('codex', { + rootPath: path.resolve('C:/projects/app'), + platform: 'win32' }); - expect(parsed.mcp['codebase-context']).toEqual({ - type: 'remote', - url: 'http://127.0.0.1:3100/mcp' + if (generated.kind !== 'file') throw new Error('expected file config'); + readFileMock.mockResolvedValue(generated.content as unknown as Buffer); + + const merged = await _buildMergedCodexContent(generated.path, generated.content); + + expect(merged).toEqual({ content: generated.content, mergedFromExisting: true }); + }); + + it('treats quoted table keys as the same idempotent Codex entry', async () => { + const generated = generateMcpConfig('codex', { + rootPath: path.resolve('C:/projects/app'), + platform: 'win32' + }); + if (generated.kind !== 'file') throw new Error('expected file config'); + const existing = generated.content.replace( + '[mcp_servers.codebase-context]', + '["mcp_servers"."codebase-context"]' + ); + readFileMock.mockResolvedValue(existing as unknown as Buffer); + + const merged = await _buildMergedCodexContent(generated.path, generated.content); + + expect(merged).toEqual({ content: existing, mergedFromExisting: true }); + }); + + it('refuses a conflicting or disabled Codex codebase-context entry', async () => { + readFileMock.mockResolvedValue( + '[mcp_servers.codebase-context]\ndisabled = true\ncommand = "old"\n' as unknown as Buffer + ); + const generated = generateMcpConfig('codex', { + rootPath: 'C:/projects/app', + platform: 'win32' + }); + if (generated.kind !== 'file') throw new Error('expected file config'); + + await expect(_buildMergedCodexContent(generated.path, generated.content)).rejects.toThrow( + /already contains a different codebase-context configuration/i + ); + }); + + it('updates a generated Codex launch for a moved root while preserving custom settings', async () => { + const oldRoot = path.resolve('C:/projects/Old Codex App'); + const newRoot = path.resolve('C:/projects/New Codex App'); + const existing = + '[mcp_servers.codebase-context]\n' + + 'command = "node"\n' + + `args = ["C:/cache/node_modules/codebase-context/dist/index.js", ${JSON.stringify(oldRoot)}]\n` + + 'startup_timeout_sec = 75\n' + + 'custom_label = "keep me"\n'; + readFileMock.mockResolvedValue(existing as unknown as Buffer); + const generated = generateMcpConfig('codex', { + rootPath: newRoot, + platform: 'win32', + packageVersion: '9.8.7' }); + if (generated.kind !== 'file') throw new Error('expected file config'); + + const merged = await _buildMergedCodexContent(generated.path, generated.content); + + expect(merged.content).toContain('command = "npx.cmd"'); + expect(merged.content).toContain( + `args = ["--yes", "codebase-context@9.8.7", ${JSON.stringify(newRoot)}]` + ); + expect(merged.content).toContain('startup_timeout_sec = 75'); + expect(merged.content).toContain('custom_label = "keep me"'); }); + + it.each([ + { + name: 'double-quoted parent and child keys', + toml: '["mcp_servers"."codebase-context"]\ncommand = "old"\n' + }, + { + name: 'double-quoted parent and bare child key', + toml: '["mcp_servers".codebase-context]\ncommand = "old"\n' + } + ])( + 'refuses to replace a Codex entry under $name without changing its bytes', + async ({ toml }) => { + const result = await runCodexSetupWithExistingConfig(toml); + + expect(result.exitCode).toBe(1); + expect(writeFileMock).not.toHaveBeenCalled(); + expect(result.persistedBytes).toEqual(result.initialBytes); + } + ); + + it.each([ + { + name: 'inline child in the bare parent table', + toml: '[mcp_servers]\ncodebase-context = { command = "old", args = [] }\n' + }, + { + name: 'dotted child in the double-quoted parent table', + toml: '["mcp_servers"]\ncodebase-context.command = "old"\n' + }, + { + name: 'dotted child in the single-quoted parent table', + toml: "['mcp_servers']\n'codebase-context'.command = \"old\"\n" + }, + { + name: 'inline root table under a double-quoted key', + toml: '"mcp_servers" = { "codebase-context" = { command = "old", args = [] } }\n' + } + ])( + 'refuses to replace a Codex entry under $name without changing its bytes', + async ({ toml }) => { + const result = await runCodexSetupWithExistingConfig(toml); + + expect(result.exitCode).toBe(1); + expect(writeFileMock).not.toHaveBeenCalled(); + expect(result.persistedBytes).toEqual(result.initialBytes); + } + ); }); diff --git a/tests/cli.test.ts b/tests/cli.test.ts index 6ee46d7..2debe67 100644 --- a/tests/cli.test.ts +++ b/tests/cli.test.ts @@ -2,15 +2,32 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import path from 'path'; import os from 'os'; import { promises as fs } from 'fs'; +import { formatRefs } from '../src/cli-formatters.js'; const toolMocks = vi.hoisted(() => ({ dispatchTool: vi.fn() })); +const cliContextMocks = vi.hoisted(() => ({ + loadProjectConfig: vi.fn(), + indexer: vi.fn() +})); + vi.mock('../src/tools/index.js', () => ({ dispatchTool: toolMocks.dispatchTool })); +vi.mock('../src/server/config.js', () => ({ + loadProjectConfig: cliContextMocks.loadProjectConfig +})); + +vi.mock('../src/core/indexer.js', () => ({ + CodebaseIndexer: vi.fn().mockImplementation(() => ({ + index: cliContextMocks.indexer, + getProgress: vi.fn() + })) +})); + import { handleCliCommand, handleMemoryCli } from '../src/cli.js'; describe('CLI', () => { @@ -21,6 +38,12 @@ describe('CLI', () => { beforeEach(() => { toolMocks.dispatchTool.mockReset(); + cliContextMocks.loadProjectConfig.mockReset().mockResolvedValue(undefined); + cliContextMocks.indexer.mockReset().mockResolvedValue({ + indexedFiles: 0, + totalChunks: 0, + duration: 0 + }); originalEnvRoot = process.env.CODEBASE_ROOT; delete process.env.CODEBASE_ROOT; @@ -41,6 +64,23 @@ describe('CLI', () => { else process.env.CODEBASE_ROOT = originalEnvRoot; }); + it('init configures the selected repository without preparing its index', async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-init-only-')); + const previousExitCode = process.exitCode; + try { + process.exitCode = undefined; + await handleCliCommand(['init', '--client', 'codex', '--yes', '--root', root]); + expect(cliContextMocks.indexer).not.toHaveBeenCalled(); + expect(cliContextMocks.loadProjectConfig).not.toHaveBeenCalled(); + expect(await fs.readFile(path.join(root, '.codex', 'config.toml'), 'utf8')).toContain( + '[mcp_servers.codebase-context]' + ); + } finally { + process.exitCode = previousExitCode; + await fs.rm(root, { recursive: true, force: true }); + } + }); + it('search errors when --query has no value', async () => { await expect(handleCliCommand(['search', '--query', '--json'])).rejects.toThrow( /process\.exit:1/ @@ -49,6 +89,25 @@ describe('CLI', () => { expect(errorSpy).toHaveBeenCalled(); }); + it('reindex help returns before context creation, indexing, or dispatch', async () => { + await handleCliCommand(['reindex', '--help']); + + const output = logSpy.mock.calls.map((call) => String(call[0] ?? '')).join('\n'); + expect(output).toContain('reindex [--incremental]'); + expect(cliContextMocks.loadProjectConfig).not.toHaveBeenCalled(); + expect(cliContextMocks.indexer).not.toHaveBeenCalled(); + expect(toolMocks.dispatchTool).not.toHaveBeenCalled(); + }); + + it('read-command help returns before context creation or dispatch', async () => { + await handleCliCommand(['status', '--help']); + + const output = logSpy.mock.calls.map((call) => String(call[0] ?? '')).join('\n'); + expect(output).toContain('status'); + expect(cliContextMocks.loadProjectConfig).not.toHaveBeenCalled(); + expect(toolMocks.dispatchTool).not.toHaveBeenCalled(); + }); + it('search dispatches with typed args', async () => { toolMocks.dispatchTool.mockResolvedValue({ content: [{ type: 'text', text: JSON.stringify({ ok: true }) }] @@ -114,7 +173,7 @@ describe('CLI', () => { const out = logSpy.mock.calls.map((c) => String(c[0] ?? '')).join('\n'); expect(out).toMatch(/Index Status/); - expect(out).toMatch(/\+\- Index Status/); + expect(out).toMatch(/\+- Index Status/); expect(out).toMatch(/Progress:/); } finally { if (originalAscii === undefined) delete process.env.CODEBASE_CONTEXT_ASCII; @@ -132,6 +191,45 @@ describe('CLI', () => { expect(logSpy).toHaveBeenCalled(); }); + it('reports unavailable reference sources and labels lexical fallback uncertainty', () => { + const result = { + symbol: 'activeToken', + usageCount: 1, + confidence: 'lexical', + usages: [{ file: 'src/active.ts', line: 1, preview: 'activeToken();' }], + isComplete: false, + coverage: { filesUnavailable: 1 } + }; + + formatRefs(result, 'C:/repo'); + + const output = logSpy.mock.calls + .map((call) => String(call[0] ?? '')) + .join('\n') + .replace(/\s+/g, ' '); + expect(output).toContain('incomplete coverage: 1 indexed source unavailable'); + expect(output).not.toContain('results capped at limit'); + expect(output).toContain('lexical'); + expect(output).toContain('Raw-text fallback may match comments or strings.'); + }); + + it('retains the limit message when complete reference coverage reaches the limit', () => { + const result = { + symbol: 'activeToken', + usageCount: 10, + confidence: 'syntactic', + usages: [{ file: 'src/active.ts', line: 1, preview: 'activeToken();' }], + isComplete: false, + coverage: { filesUnavailable: 0 } + }; + + formatRefs(result, 'C:/repo'); + + const output = logSpy.mock.calls.map((call) => String(call[0] ?? '')).join('\n'); + expect(output).toContain('results capped at limit'); + expect(output).not.toContain('incomplete coverage'); + }); + it('memory list errors on invalid --type', async () => { await expect(handleMemoryCli(['list', '--type', 'nope'])).rejects.toThrow(/process\.exit:1/); expect(errorSpy).toHaveBeenCalled(); diff --git a/tests/contextbench-baseline-runner.test.ts b/tests/contextbench-baseline-runner.test.ts index 61064a4..881f3ae 100644 --- a/tests/contextbench-baseline-runner.test.ts +++ b/tests/contextbench-baseline-runner.test.ts @@ -1,5 +1,12 @@ import { execFileSync, spawnSync } from 'node:child_process'; -import { appendFileSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { + appendFileSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync +} from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; import { describe, expect, it, vi } from 'vitest'; @@ -51,22 +58,17 @@ function childEnv(overrides: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv { return { ...env, ...overrides }; } -function ignoreWindowsTempCleanupRace(error: unknown): void { - const code = (error as NodeJS.ErrnoException).code; - if (!['EBUSY', 'ENOTEMPTY', 'EPERM'].includes(code ?? '')) throw error; +function cleanupTempRoot(tempRoot: string): void { + rmSync(tempRoot, { + recursive: true, + force: true, + maxRetries: 10, + retryDelay: 200 + }); } function cleanupSessionRoot(sessionRoot: string): void { - try { - rmSync(path.dirname(path.dirname(path.dirname(path.dirname(sessionRoot)))), { - recursive: true, - force: true, - maxRetries: 10, - retryDelay: 200 - }); - } catch (error) { - ignoreWindowsTempCleanupRace(error); - } + cleanupTempRoot(path.dirname(path.dirname(path.dirname(path.dirname(sessionRoot))))); } function tempSessionRoot(phase: 'phase40' | 'phase41' = 'phase40'): string { @@ -94,7 +96,15 @@ function createCleanGitRepo(root: string): string { execFileSync('git', ['add', 'README.md'], { cwd: repoPath, encoding: 'utf8', env: childEnv() }); execFileSync( 'git', - ['-c', 'user.name=ContextBench Test', '-c', 'user.email=contextbench@example.invalid', 'commit', '-m', 'fixture'], + [ + '-c', + 'user.name=ContextBench Test', + '-c', + 'user.email=contextbench@example.invalid', + 'commit', + '-m', + 'fixture' + ], { cwd: repoPath, encoding: 'utf8', env: childEnv() } ); return repoPath; @@ -131,7 +141,11 @@ function writeStubClaude(root: string): string { return stubPath; } -function writeStubEvaluator(root: string, exitCode: 0 | 1, output = JSON.stringify({ score: 1 })): string { +function writeStubEvaluator( + root: string, + exitCode: 0 | 1, + output = JSON.stringify({ score: 1 }) +): string { const stubPath = path.join(root, `stub-evaluator-${exitCode}.cjs`); const serializedOutput = JSON.stringify(output); writeFileSync( @@ -146,14 +160,22 @@ describe('ContextBench Phase 40 baseline runner', () => { it('reserves every required slot and writes terminal missing-evidence rows for blocked lanes', () => { const sessionRoot = tempSessionRoot(); try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); const reservations = JSON.parse( readFileSync(path.join(sessionRoot, 'slot-reservations.json'), 'utf8') - ) as { reservations: Array<{ laneId: string; status: string; terminalStatus: string | null }> }; + ) as { + reservations: Array<{ laneId: string; status: string; terminalStatus: string | null }>; + }; expect(reservations.reservations).toHaveLength(20 * 6 * 3); - const blocked = reservations.reservations.filter((slot) => slot.status === 'terminal_missing_evidence'); + const blocked = reservations.reservations.filter( + (slot) => slot.status === 'terminal_missing_evidence' + ); expect(blocked).toHaveLength(20 * 2 * 3); expect([...new Set(blocked.map((slot) => slot.laneId))].sort()).toEqual([ 'codebase-memory-mcp', @@ -176,10 +198,16 @@ describe('ContextBench Phase 40 baseline runner', () => { it('rejects duplicate primary baseline rows during validation', () => { const sessionRoot = tempSessionRoot('phase41'); try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); - const firstRow = readFileSync(path.join(sessionRoot, 'run-manifest.jsonl'), 'utf8').trim().split('\n')[0]; + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); + const firstRow = readFileSync(path.join(sessionRoot, 'run-manifest.jsonl'), 'utf8') + .trim() + .split('\n')[0]; appendFileSync(path.join(sessionRoot, 'run-manifest.jsonl'), `${firstRow}\n`, 'utf8'); const result = spawnSync( @@ -191,10 +219,7 @@ describe('ContextBench Phase 40 baseline runner', () => { expect(result.status).not.toBe(0); expect(result.stderr).toContain('duplicate primary baseline row for reservation'); } finally { - rmSync(path.dirname(path.dirname(path.dirname(path.dirname(sessionRoot)))), { - recursive: true, - force: true - }); + cleanupSessionRoot(sessionRoot); } }); @@ -202,9 +227,13 @@ describe('ContextBench Phase 40 baseline runner', () => { const sessionRoot = tempSessionRoot(); const taskId = manifest.tasks[0].instance_id; try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); execFileSync( 'node', [ @@ -265,7 +294,13 @@ describe('ContextBench Phase 40 baseline runner', () => { it('records official evaluator invocation metadata for overridden live executor attempts', () => { const tempRoot = mkdtempSync(path.join(tmpdir(), 'contextbench-official-evaluator-')); - const sessionRoot = path.join(tempRoot, 'benchmark-runs', 'contextbench', 'phase41', 'runner-smoke'); + const sessionRoot = path.join( + tempRoot, + 'benchmark-runs', + 'contextbench', + 'phase41', + 'runner-smoke' + ); const taskId = manifest.tasks[0].instance_id; const repoPath = createCleanGitRepo(tempRoot); const payloadPath = writePayloadFile(tempRoot, taskId, repoPath); @@ -275,14 +310,22 @@ describe('ContextBench Phase 40 baseline runner', () => { CONTEXTBENCH_CLAUDE_COMMAND: JSON.stringify([process.execPath, stubClaude]), CONTEXTBENCH_OFFICIAL_EVALUATOR_COMMAND: JSON.stringify([process.execPath, stubEvaluator]), CONTEXTBENCH_LANE_TELEMETRY_JSON: JSON.stringify({ - 'raw-native': { sourceKind: 'proxy', proofSource: 'stubbed_test_proxy', observedTools: ['native-read'] } + 'raw-native': { + sourceKind: 'proxy', + proofSource: 'stubbed_test_proxy', + observedTools: ['native-read'] + } }) }); try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8', - env - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8', + env + } + ); execFileSync( 'node', [ @@ -329,7 +372,12 @@ describe('ContextBench Phase 40 baseline runner', () => { expect(score.stdoutPath).toBeTruthy(); expect(score.stderrPath).toBeTruthy(); const rawTrace = JSON.parse(readFileSync(attempt?.raw_trace_path ?? '', 'utf8')) as { - laneIsolation?: { proven: boolean; sourceKind: string; proofSource: string; observedTools: string[] }; + laneIsolation?: { + proven: boolean; + sourceKind: string; + proofSource: string; + observedTools: string[]; + }; }; expect(rawTrace.laneIsolation).toMatchObject({ proven: true, @@ -338,13 +386,19 @@ describe('ContextBench Phase 40 baseline runner', () => { observedTools: ['native-read'] }); } finally { - rmSync(tempRoot, { recursive: true, force: true }); + cleanupTempRoot(tempRoot); } }); it('rejects malformed official evaluator output as judge_failed diagnostic evidence', () => { const tempRoot = mkdtempSync(path.join(tmpdir(), 'contextbench-official-evaluator-malformed-')); - const sessionRoot = path.join(tempRoot, 'benchmark-runs', 'contextbench', 'phase41', 'runner-smoke'); + const sessionRoot = path.join( + tempRoot, + 'benchmark-runs', + 'contextbench', + 'phase41', + 'runner-smoke' + ); const taskId = manifest.tasks[0].instance_id; const repoPath = createCleanGitRepo(tempRoot); const payloadPath = writePayloadFile(tempRoot, taskId, repoPath); @@ -354,14 +408,22 @@ describe('ContextBench Phase 40 baseline runner', () => { CONTEXTBENCH_CLAUDE_COMMAND: JSON.stringify([process.execPath, stubClaude]), CONTEXTBENCH_OFFICIAL_EVALUATOR_COMMAND: JSON.stringify([process.execPath, stubEvaluator]), CONTEXTBENCH_LANE_TELEMETRY_JSON: JSON.stringify({ - 'raw-native': { sourceKind: 'proxy', proofSource: 'stubbed_test_proxy', observedTools: ['native-read'] } + 'raw-native': { + sourceKind: 'proxy', + proofSource: 'stubbed_test_proxy', + observedTools: ['native-read'] + } }) }); try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8', - env - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8', + env + } + ); execFileSync( 'node', [ @@ -401,7 +463,7 @@ describe('ContextBench Phase 40 baseline runner', () => { expect(score.fallbackReason).toBe('official_evaluator_malformed_jsonl'); expect(score.outputHash).toMatch(/^sha256:[a-f0-9]{64}$/); } finally { - rmSync(tempRoot, { recursive: true, force: true }); + cleanupTempRoot(tempRoot); } }); @@ -415,8 +477,16 @@ describe('ContextBench Phase 40 baseline runner', () => { ]; for (const testCase of cases) { - const tempRoot = mkdtempSync(path.join(tmpdir(), 'contextbench-official-evaluator-envelope-')); - const sessionRoot = path.join(tempRoot, 'benchmark-runs', 'contextbench', 'phase41', 'runner-smoke'); + const tempRoot = mkdtempSync( + path.join(tmpdir(), 'contextbench-official-evaluator-envelope-') + ); + const sessionRoot = path.join( + tempRoot, + 'benchmark-runs', + 'contextbench', + 'phase41', + 'runner-smoke' + ); const taskId = manifest.tasks[0].instance_id; const repoPath = createCleanGitRepo(tempRoot); const payloadPath = writePayloadFile(tempRoot, taskId, repoPath); @@ -426,14 +496,22 @@ describe('ContextBench Phase 40 baseline runner', () => { CONTEXTBENCH_CLAUDE_COMMAND: JSON.stringify([process.execPath, stubClaude]), CONTEXTBENCH_OFFICIAL_EVALUATOR_COMMAND: JSON.stringify([process.execPath, stubEvaluator]), CONTEXTBENCH_LANE_TELEMETRY_JSON: JSON.stringify({ - 'raw-native': { sourceKind: 'proxy', proofSource: 'stubbed_test_proxy', observedTools: ['native-read'] } + 'raw-native': { + sourceKind: 'proxy', + proofSource: 'stubbed_test_proxy', + observedTools: ['native-read'] + } }) }); try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8', - env - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8', + env + } + ); execFileSync( 'node', [ @@ -460,14 +538,20 @@ describe('ContextBench Phase 40 baseline runner', () => { expect(attempt?.status).toBe('judge_failed'); expect(attempt?.scoring.fallbackReason).toBe(testCase.reason); } finally { - rmSync(tempRoot, { recursive: true, force: true }); + cleanupTempRoot(tempRoot); } } }); it('records official evaluator failure as judge_failed without making claims', () => { const tempRoot = mkdtempSync(path.join(tmpdir(), 'contextbench-official-evaluator-fail-')); - const sessionRoot = path.join(tempRoot, 'benchmark-runs', 'contextbench', 'phase41', 'runner-smoke'); + const sessionRoot = path.join( + tempRoot, + 'benchmark-runs', + 'contextbench', + 'phase41', + 'runner-smoke' + ); const taskId = manifest.tasks[0].instance_id; const repoPath = createCleanGitRepo(tempRoot); const payloadPath = writePayloadFile(tempRoot, taskId, repoPath); @@ -478,10 +562,14 @@ describe('ContextBench Phase 40 baseline runner', () => { CONTEXTBENCH_OFFICIAL_EVALUATOR_COMMAND: JSON.stringify([process.execPath, stubEvaluator]) }); try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8', - env - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8', + env + } + ); execFileSync( 'node', [ @@ -525,16 +613,20 @@ describe('ContextBench Phase 40 baseline runner', () => { expect(score.exitCode).toBe(1); expect(score.exitStatus).toBe(1); } finally { - rmSync(tempRoot, { recursive: true, force: true }); + cleanupTempRoot(tempRoot); } }); it('chunks all-ready-lane execution with max-attempts so live runs are resumable', () => { const sessionRoot = tempSessionRoot('phase41'); try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); execFileSync( 'node', [ @@ -591,9 +683,13 @@ describe('ContextBench Phase 40 baseline runner', () => { const sessionRoot = tempSessionRoot('phase41'); const taskId = manifest.tasks[0].instance_id; try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); execFileSync( 'node', [ @@ -607,8 +703,16 @@ describe('ContextBench Phase 40 baseline runner', () => { { encoding: 'utf8' } ); const measurement = JSON.parse( - readFileSync(path.join(sessionRoot, 'setup-index', 'raw-native', 'setup-index.json'), 'utf8') - ) as { claimBearing: boolean; setupStatus: string; indexStatus: string; setupLogPath: string }; + readFileSync( + path.join(sessionRoot, 'setup-index', 'raw-native', 'setup-index.json'), + 'utf8' + ) + ) as { + claimBearing: boolean; + setupStatus: string; + indexStatus: string; + setupLogPath: string; + }; expect(measurement).toMatchObject({ claimBearing: false, setupStatus: 'not_required', @@ -652,9 +756,13 @@ describe('ContextBench Phase 40 baseline runner', () => { const sessionRoot = tempSessionRoot('phase41'); const taskId = manifest.tasks[0].instance_id; try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); execFileSync( 'node', [ @@ -674,7 +782,8 @@ describe('ContextBench Phase 40 baseline runner', () => { { encoding: 'utf8' } ); const attempt = readRows(sessionRoot).find( - (row) => row.lane_id === 'codebase-context' && row.task_id === taskId && row.repeat_index === 1 + (row) => + row.lane_id === 'codebase-context' && row.task_id === taskId && row.repeat_index === 1 ); expect(attempt?.status).toBe('setup_failed'); expect(attempt?.scoring.fallbackReason).toContain('missing_setup_index_measurement'); @@ -688,16 +797,23 @@ describe('ContextBench Phase 40 baseline runner', () => { const sessionRoot = tempSessionRoot('phase41'); const taskId = manifest.tasks[0].instance_id; try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); const logsDir = path.join(sessionRoot, 'manual-setup-index-logs', 'codebase-context'); mkdirSync(logsDir, { recursive: true }); const setupLogPath = path.join(logsDir, 'setup.stdout.log'); const indexLogPath = path.join(logsDir, 'index.stdout.log'); writeFileSync(setupLogPath, 'setup completed\n', 'utf8'); writeFileSync(indexLogPath, 'index completed\n', 'utf8'); - const importPath = path.join(path.dirname(sessionRoot), 'codebase-context-setup-index-import.json'); + const importPath = path.join( + path.dirname(sessionRoot), + 'codebase-context-setup-index-import.json' + ); writeFileSync( importPath, JSON.stringify( @@ -749,7 +865,8 @@ describe('ContextBench Phase 40 baseline runner', () => { { encoding: 'utf8' } ); const attempt = readRows(sessionRoot).find( - (row) => row.lane_id === 'codebase-context' && row.task_id === taskId && row.repeat_index === 1 + (row) => + row.lane_id === 'codebase-context' && row.task_id === taskId && row.repeat_index === 1 ); expect(attempt?.status).toBe('completed'); expect(attempt?.setupIndex).toMatchObject({ @@ -769,9 +886,13 @@ describe('ContextBench Phase 40 baseline runner', () => { const sessionRoot = tempSessionRoot('phase41'); const taskId = manifest.tasks[0].instance_id; try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); const logsDir = path.join(sessionRoot, 'manual-setup-index-logs', 'codebase-context'); mkdirSync(logsDir, { recursive: true }); const setupLogPath = path.join(logsDir, 'setup.stdout.log'); @@ -896,7 +1017,8 @@ describe('ContextBench Phase 40 baseline runner', () => { { encoding: 'utf8' } ); const attempt = readRows(sessionRoot).find( - (row) => row.lane_id === 'codebase-context' && row.task_id === taskId && row.repeat_index === 1 + (row) => + row.lane_id === 'codebase-context' && row.task_id === taskId && row.repeat_index === 1 ); expect(attempt?.status).toBe('setup_failed'); expect(attempt?.scoring.fallbackReason).toContain('missing_setup_index_measurement'); @@ -922,9 +1044,13 @@ describe('ContextBench Phase 40 baseline runner', () => { const sessionRoot = tempSessionRoot(); const taskId = manifest.tasks[0].instance_id; try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); execFileSync( 'node', [ @@ -942,7 +1068,9 @@ describe('ContextBench Phase 40 baseline runner', () => { { encoding: 'utf8' } ); const rows = readRows(sessionRoot); - const diagnosticRows = rows.filter((row) => row.run_id.startsWith('codebase-context-current')); + const diagnosticRows = rows.filter((row) => + row.run_id.startsWith('codebase-context-current') + ); expect(diagnosticRows.length).toBeGreaterThanOrEqual(3); expect(diagnosticRows.every((row) => row.lane_id === 'codebase-context')).toBe(true); expect(diagnosticRows.every((row) => row.scoring.claimBearing === false)).toBe(true); @@ -957,13 +1085,23 @@ describe('ContextBench Phase 40 baseline runner', () => { it('runs Phase 42 verification as read-only artifact-derived evidence and fails diagnostic sessions closed', () => { const tempRoot = mkdtempSync(path.join(tmpdir(), 'contextbench-phase42-verify-')); - const sessionRoot = path.join(tempRoot, 'benchmark-runs', 'contextbench', 'phase41', 'runner-smoke'); + const sessionRoot = path.join( + tempRoot, + 'benchmark-runs', + 'contextbench', + 'phase41', + 'runner-smoke' + ); const reportPath = path.join(tempRoot, 'phase42-report.json'); const taskId = manifest.tasks[0].instance_id; try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); execFileSync( 'node', [ @@ -1022,21 +1160,33 @@ describe('ContextBench Phase 40 baseline runner', () => { expect(report.failureCounts.official_evaluator_missing).toBeGreaterThan(0); expect(report.failureCounts.missing_required_run).toBeGreaterThan(0); expect(report.blockedClaims).toContain('Phase 42 passed'); - expect(readFileSync(path.join(sessionRoot, 'BASELINE-SESSION.json'), 'utf8')).toBe(sessionBefore); + expect(readFileSync(path.join(sessionRoot, 'BASELINE-SESSION.json'), 'utf8')).toBe( + sessionBefore + ); } finally { - rmSync(tempRoot, { recursive: true, force: true }); + cleanupTempRoot(tempRoot); } }); it('classifies diagnostic baseline arms as supplemental during Phase 42 verification', () => { const tempRoot = mkdtempSync(path.join(tmpdir(), 'contextbench-phase42-arms-')); - const sessionRoot = path.join(tempRoot, 'benchmark-runs', 'contextbench', 'phase41', 'runner-smoke'); + const sessionRoot = path.join( + tempRoot, + 'benchmark-runs', + 'contextbench', + 'phase41', + 'runner-smoke' + ); const reportPath = path.join(tempRoot, 'phase42-report.json'); const taskId = manifest.tasks[0].instance_id; try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); execFileSync( 'node', [ @@ -1076,16 +1226,20 @@ describe('ContextBench Phase 40 baseline runner', () => { expect(report.failureCounts.unexpected_run_row ?? 0).toBe(0); expect(report.failureCounts.denominator_count_mismatch).toBe(1); } finally { - rmSync(tempRoot, { recursive: true, force: true }); + cleanupTempRoot(tempRoot); } }); it('blocks baseline seal when terminal row completeness lacks Phase 42 claim evidence', () => { const sessionRoot = tempSessionRoot('phase41'); try { - execFileSync('node', ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], { - encoding: 'utf8' - }); + execFileSync( + 'node', + ['scripts/contextbench-runner.mjs', '--baseline-snapshot', '--out', sessionRoot], + { + encoding: 'utf8' + } + ); execFileSync( 'node', [ diff --git a/tests/contextbench-baseline-schema-gate.test.ts b/tests/contextbench-baseline-schema-gate.test.ts index 69b7a93..6ba2a42 100644 --- a/tests/contextbench-baseline-schema-gate.test.ts +++ b/tests/contextbench-baseline-schema-gate.test.ts @@ -44,22 +44,13 @@ function childEnv(overrides: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv { return { ...env, ...overrides }; } -function ignoreWindowsTempCleanupRace(error: unknown): void { - const code = (error as NodeJS.ErrnoException).code; - if (!['EBUSY', 'ENOTEMPTY', 'EPERM'].includes(code ?? '')) throw error; -} - function cleanupSessionRoot(sessionRoot: string): void { - try { - rmSync(path.dirname(path.dirname(path.dirname(path.dirname(sessionRoot)))), { - recursive: true, - force: true, - maxRetries: 10, - retryDelay: 200 - }); - } catch (error) { - ignoreWindowsTempCleanupRace(error); - } + rmSync(path.dirname(path.dirname(path.dirname(path.dirname(sessionRoot)))), { + recursive: true, + force: true, + maxRetries: 10, + retryDelay: 200 + }); } function normalizeFilesystemPath(filePath: string): string { @@ -815,14 +806,8 @@ describe('ContextBench Phase 40 schema gate', () => { expect.arrayContaining(['base_commit_mismatch', 'problem_statement_hash_mismatch']) ); } finally { - rmSync(path.dirname(path.dirname(path.dirname(path.dirname(missingSessionRoot)))), { - recursive: true, - force: true - }); - rmSync(path.dirname(path.dirname(path.dirname(path.dirname(wrongCommitSessionRoot)))), { - recursive: true, - force: true - }); + cleanupSessionRoot(missingSessionRoot); + cleanupSessionRoot(wrongCommitSessionRoot); rmSync(payloadDir, { recursive: true, force: true }); rmSync(wrongCommitRepo, { recursive: true, force: true }); } diff --git a/tests/eval-harness.test.ts b/tests/eval-harness.test.ts index 9d483bb..c1367ae 100644 --- a/tests/eval-harness.test.ts +++ b/tests/eval-harness.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it, vi } from 'vitest'; import { CodebaseSearcher } from '../src/core/search.js'; import type { CodeChunk, SearchResult } from '../src/types/index.js'; +import { asSearcherTestHarness, materializeSourceSnapshot } from './helpers/source-snapshot.js'; import type { EvalFixture, EvalQuery } from '../src/eval/types.js'; import { countUtf8Bytes, @@ -49,10 +50,11 @@ function setupSearcherWithResultsByQuery( byQuery: Record, fallback: SearchResult[] = [] ): CodebaseSearcher { - const searcher = new CodebaseSearcher('C:/repo') as any; - searcher.initialized = true; - searcher.search = vi.fn(async (query: string) => byQuery[query] ?? fallback); - return searcher as CodebaseSearcher; + const searcher = new CodebaseSearcher('C:/repo'); + const harness = asSearcherTestHarness(searcher); + harness.initialized = true; + harness.search = vi.fn(async (query: string) => byQuery[query] ?? fallback); + return searcher; } function fixtureForSingleQuery(query: EvalQuery): EvalFixture { @@ -268,23 +270,30 @@ describe('Eval Harness - integration with CodebaseSearcher (mocked)', () => { { componentType: 'unknown' } ); - const searcher = new CodebaseSearcher('C:/repo') as any; - searcher.initialized = true; - searcher.embeddingProvider = { - embed: vi.fn(async () => new Array(384).fill(0.01)) - }; - searcher.storageProvider = { - search: vi.fn(async () => [ - { chunk: authServiceChunk, score: 0.65 }, - { chunk: e2eChunk, score: 0.55 } - ]), - count: vi.fn(async () => 2) - }; - searcher.fuseIndex = null; - searcher.patternIntelligence = null; + const snapshot = await materializeSourceSnapshot([authServiceChunk, e2eChunk]); + try { + const searcher = new CodebaseSearcher(snapshot.rootPath); + const harness = asSearcherTestHarness(searcher); + harness.initialized = true; + harness.sourceManifest = snapshot.manifest; + harness.embeddingProvider = { + embed: vi.fn(async () => new Array(384).fill(0.01)) + }; + harness.storageProvider = { + search: vi.fn(async () => [ + { chunk: authServiceChunk, score: 0.65 }, + { chunk: e2eChunk, score: 0.55 } + ]), + count: vi.fn(async () => 2) + }; + harness.fuseIndex = null; + harness.patternIntelligence = null; - const results = await (searcher as CodebaseSearcher).search('AuthService', 5); - expect(results.length).toBeGreaterThan(0); - expect(results[0].filePath).toContain('auth.service.ts'); + const results = await searcher.search('AuthService', 5); + expect(results.length).toBeGreaterThan(0); + expect(results[0].filePath).toContain('auth.service.ts'); + } finally { + await snapshot.cleanup(); + } }); }); diff --git a/tests/file-graph-freshness.test.ts b/tests/file-graph-freshness.test.ts new file mode 100644 index 0000000..75ae93e --- /dev/null +++ b/tests/file-graph-freshness.test.ts @@ -0,0 +1,164 @@ +import { createHash } from 'crypto'; +import { promises as fs } from 'fs'; +import os from 'os'; +import path from 'path'; +import { afterEach, expect, it, vi } from 'vitest'; +import { + CODEBASE_CONTEXT_DIRNAME, + INDEX_FORMAT_VERSION, + INDEX_META_FILENAME, + INDEX_META_VERSION, + RELATIONSHIPS_FILENAME +} from '../src/constants/codebase-context.js'; +import { handle as getCodeRelationships } from '../src/tools/get-code-relationships.js'; +import type { ToolContext } from '../src/tools/types.js'; +import { InternalFileGraph } from '../src/utils/usage-tracker.js'; + +const roots: string[] = []; +afterEach(async () => { + vi.restoreAllMocks(); + await Promise.all(roots.splice(0).map((root) => fs.rm(root, { recursive: true, force: true }))); +}); + +async function fixture() { + const root = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-graph-freshness-')); + roots.push(root); + const graph = new InternalFileGraph(root); + const sources = new Map(); + const baseDir = path.join(root, CODEBASE_CONTEXT_DIRNAME); + await fs.mkdir(baseDir); + const write = async (file: string, source: string) => { + sources.set(file, source); + await fs.writeFile(path.join(root, file), source); + }; + const fingerprint = (file: string) => + createHash('sha256').update(sources.get(file) ?? '').digest('hex'); + const edge = (from: string, to: string, line = 1, kind: 'import' | 'reexport' = 'import') => { + graph.trackImport(path.join(root, from), path.join(root, to), line, [], { + kind, + specifier: `./${to.slice(0, -3)}`, + sourceFingerprint: fingerprint(from), + targetFingerprint: fingerprint(to) + }); + }; + const save = async () => { + await fs.writeFile(path.join(baseDir, INDEX_META_FILENAME), JSON.stringify({ + metaVersion: INDEX_META_VERSION, + formatVersion: INDEX_FORMAT_VERSION, + buildId: 'freshness-fixture', + generatedAt: '2026-10-15T00:00:00.000Z', + toolVersion: 'fixture', + artifacts: { + keywordIndex: { path: 'keyword-index.json' }, + vectorDb: { path: 'vector-db', provider: 'none', embeddingEnabled: false } + } + })); + await fs.writeFile(path.join(baseDir, RELATIONSHIPS_FILENAME), JSON.stringify({ + header: { buildId: 'freshness-fixture', formatVersion: INDEX_FORMAT_VERSION }, + graph: graph.toJSON() + })); + }; + const ctx: ToolContext = { + rootPath: root, + paths: { + baseDir, + intelligence: path.join(baseDir, 'intelligence.json'), + memory: path.join(baseDir, 'memory.json'), + health: path.join(baseDir, 'health.json'), + keywordIndex: path.join(baseDir, 'keyword-index.json'), + vectorDb: path.join(baseDir, 'vector-db') + }, + indexState: { status: 'ready' }, + performIndexing: () => undefined + }; + const query = async (file: string, direction: 'incoming' | 'outgoing', maxHops = 1, maxNodes = 100) => { + const response = await getCodeRelationships({ file, direction, max_hops: maxHops, max_nodes: maxNodes }, ctx); + const text = response.content?.[0]?.text ?? '{}'; + expect(Buffer.byteLength(text, 'utf8')).toBeLessThanOrEqual(4000); + return JSON.parse(text) as { + status: string; + paths: Array<{ nodes: string[]; edges: Array<{ kind: string }> }>; + truncated: boolean; + outputNote?: string; + message: string; + }; + }; + return { root, write, edge, save, query }; +} + +it.each([2, 100])('keeps a current outgoing edge after forty stale targets with accepted node cap %i', async (maxNodes) => { + const f = await fixture(); + const stale = Array.from({ length: 40 }, (_, n) => `stale-${n}.ts`); + await f.write('entry.ts', [...stale, 'support.ts'].map((file) => `import './${file.slice(0, -3)}';`).join('\n')); + for (const [index, file] of [...stale, 'support.ts'].entries()) { + await f.write(file, 'export const value = true;\n'); + f.edge('entry.ts', file, index + 1); + } + await f.save(); + for (const file of stale) await f.write(file, 'export const changed = true;\n'); + + const result = await f.query('entry.ts', 'outgoing', 1, maxNodes); + expect(result.status).toBe('stale_edges'); + expect(result.paths.map(({ nodes }) => nodes)).toEqual([['entry.ts', 'support.ts']]); + expect(result.paths.some(({ nodes }) => nodes.some((node) => node.startsWith('stale-')))).toBe(false); +}); + +it.each([2, 100])('keeps a current incoming importer after forty stale importers with accepted node cap %i', async (maxNodes) => { + const f = await fixture(); + await f.write('target.ts', 'export const value = true;\n'); + const stale = Array.from({ length: 40 }, (_, n) => `stale-${n}.ts`); + for (const file of [...stale, 'current.ts']) { + await f.write(file, "import './target';\n"); + f.edge(file, 'target.ts'); + } + await f.save(); + for (const file of stale) await f.write(file, 'export const changed = true;\n'); + + const result = await f.query('target.ts', 'incoming', 1, maxNodes); + expect(result.paths.map(({ nodes }) => nodes)).toEqual([['target.ts', 'current.ts']]); +}); + +it('follows a current typed three-hop chain past stale branches and terminates its cycle', async () => { + const f = await fixture(); + const stale = Array.from({ length: 40 }, (_, n) => `stale-${n}.ts`); + await f.write('entry.ts', [...stale, 'barrel.ts'].map((file) => `import './${file.slice(0, -3)}';`).join('\n')); + await f.write('barrel.ts', "export * from './middle';\n"); + await f.write('middle.ts', "import './leaf';\n"); + await f.write('leaf.ts', "import './barrel';\n"); + for (const file of stale) { + await f.write(file, 'export const value = true;\n'); + f.edge('entry.ts', file); + } + f.edge('entry.ts', 'barrel.ts'); + f.edge('barrel.ts', 'middle.ts', 1, 'reexport'); + f.edge('middle.ts', 'leaf.ts'); + f.edge('leaf.ts', 'barrel.ts'); + await f.save(); + for (const file of stale) await f.write(file, 'export const changed = true;\n'); + + const result = await f.query('entry.ts', 'outgoing', 3, 4); + const chain = result.paths.find(({ nodes }) => nodes.length === 4); + expect(chain?.nodes).toEqual(['entry.ts', 'barrel.ts', 'middle.ts', 'leaf.ts']); + expect(chain?.edges.map(({ kind }) => kind)).toEqual(['import', 'reexport', 'import']); + expect(result.paths.every(({ nodes }) => new Set(nodes).size === nodes.length)).toBe(true); +}); + +it('bounds stale edge inspections and explains when a current edge remains beyond that budget', async () => { + const f = await fixture(); + await f.write('entry.ts', `${"import './stale';\n".repeat(1100)}import './current';\n`); + await f.write('stale.ts', 'export const value = true;\n'); + await f.write('current.ts', 'export const current = true;\n'); + for (let line = 1; line <= 1100; line++) f.edge('entry.ts', 'stale.ts', line); + f.edge('entry.ts', 'current.ts', 1101); + await f.save(); + await f.write('stale.ts', 'export const changed = true;\n'); + const reads = vi.spyOn(fs, 'readFile'); + + const result = await f.query('entry.ts', 'outgoing'); + const staleReads = reads.mock.calls.filter(([file]) => String(file) === path.join(f.root, 'stale.ts')); + expect(staleReads.length).toBeLessThanOrEqual(1000); + expect(result.paths).toEqual([]); + expect(result.truncated).toBe(true); + expect(result.outputNote).toMatch(/inspection/i); + expect(result.message).toMatch(/refresh indexing/i); +}); diff --git a/tests/file-graph-indexing-negatives.test.ts b/tests/file-graph-indexing-negatives.test.ts new file mode 100644 index 0000000..a5123da --- /dev/null +++ b/tests/file-graph-indexing-negatives.test.ts @@ -0,0 +1,31 @@ +import { afterAll, expect, it } from 'vitest'; +import { promises as fs } from 'fs'; +import os from 'os'; +import path from 'path'; +import { CodebaseIndexer } from '../src/core/indexer.js'; +import { CODEBASE_CONTEXT_DIRNAME, RELATIONSHIPS_FILENAME } from '../src/constants/codebase-context.js'; + +const root = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-graph-negative-')); +afterAll(async () => fs.rm(root, { recursive: true, force: true })); + +it('never serializes a guessed .ts target when only a differently named target or no target exists', async () => { + const source = path.join(root, 'src'); + await fs.mkdir(source, { recursive: true }); + await fs.writeFile(path.join(source, 'entry.ts'), "import './missing';\nimport './present';\n"); + await fs.writeFile(path.join(source, 'present.tsx'), 'export const present = true;\n'); + const indexer = new CodebaseIndexer({ rootPath: root, config: { skipEmbedding: true } }); + await indexer.index(); + + const sidecarPath = path.join(root, CODEBASE_CONTEXT_DIRNAME, RELATIONSHIPS_FILENAME); + const sidecar = JSON.parse(await fs.readFile(sidecarPath, 'utf8')) as { + graph: { imports: Record; importDetails?: Record }>>; unresolvedEdges?: Array<{ source: { specifier: string }; reason: string }> }; + }; + const importer = Object.keys(sidecar.graph.imports).find((file) => file.endsWith('src/entry.ts')); + expect(importer).toBeDefined(); + const targets = sidecar.graph.imports[importer ?? ''] ?? []; + expect(targets).toContain('src/present.tsx'); + expect(targets).not.toContain('src/missing.ts'); + expect(sidecar.graph.unresolvedEdges).toContainEqual(expect.objectContaining({ + source: expect.objectContaining({ specifier: './missing' }), reason: 'not-found' + })); +}); diff --git a/tests/file-graph-integration.test.ts b/tests/file-graph-integration.test.ts new file mode 100644 index 0000000..ddef78a --- /dev/null +++ b/tests/file-graph-integration.test.ts @@ -0,0 +1,232 @@ +import { afterAll, expect, it } from 'vitest'; +import { promises as fs } from 'fs'; +import os from 'os'; +import path from 'path'; +import { CodebaseIndexer } from '../src/core/indexer.js'; +import { CODEBASE_CONTEXT_DIRNAME, RELATIONSHIPS_FILENAME } from '../src/constants/codebase-context.js'; +import { handle as getCodeRelationships } from '../src/tools/get-code-relationships.js'; +import type { ToolContext } from '../src/tools/types.js'; + +const root = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-graph-integration-')); +afterAll(async () => fs.rm(root, { recursive: true, force: true })); + +async function write(relative: string, content: string, projectRoot = root): Promise { + const fullPath = path.join(projectRoot, relative); + await fs.mkdir(path.dirname(fullPath), { recursive: true }); + await fs.writeFile(fullPath, content); +} + +async function readGraph(projectRoot = root): Promise<{ header: { buildId: string }; graph: { imports: Record; importDetails: Record }>>; unresolvedEdges?: Array<{ source: { specifier: string }; reason: string }> } }> { + return JSON.parse(await fs.readFile(path.join(projectRoot, CODEBASE_CONTEXT_DIRNAME, RELATIONSHIPS_FILENAME), 'utf8')) as { + header: { buildId: string }; + graph: { imports: Record; importDetails: Record }>>; unresolvedEdges?: Array<{ source: { specifier: string }; reason: string }> }; + }; +} + +function context(projectRoot = root): ToolContext { + const contextDir = path.join(projectRoot, CODEBASE_CONTEXT_DIRNAME); + return { + rootPath: projectRoot, + paths: { + baseDir: contextDir, + intelligence: path.join(contextDir, 'intelligence.json'), + memory: path.join(contextDir, 'memory.json'), + health: path.join(contextDir, 'health.json'), + keywordIndex: path.join(contextDir, 'keyword-index.json'), + vectorDb: path.join(contextDir, 'vector-db') + }, + indexState: { status: 'ready' }, + performIndexing: () => undefined + }; +} + +it('verifies the incoming three-hop alias chain in a nested root with only tsconfig.base.json', async () => { + const host = path.join(root, 'base-config-host'); + const projectRoot = path.join(host, 'checkout with spaces'); + const utility = 'libs/web/shared/utils/src/lib/time-util.ts'; + const barrel = 'libs/web/shared/utils/src/index.ts'; + const pipe = 'libs/web/shared/pipes/duration-pipe/src/lib/duration.pipe.ts'; + const module = 'libs/web/shared/pipes/duration-pipe/src/lib/duration-pipe.module.ts'; + const alias = '@angular-spotify/web/shared/utils'; + await write('tsconfig.json', JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { [alias]: ['wrong-parent-target.ts'] } } + }), host); + await write('tsconfig.base.json', JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { [alias]: [barrel] } } + }), projectRoot); + await write(utility, 'export class TimeUtil { static formatDuration(value: number) { return String(value); } }\n', projectRoot); + await write(barrel, "export * from './lib/time-util';\n", projectRoot); + await write(pipe, `import { TimeUtil } from '${alias}';\nimport '@unmapped/package';\nexport class DurationPipe { transform(value: number) { return TimeUtil.formatDuration(value); } }\n`, projectRoot); + await write(module, "import { DurationPipe } from './duration.pipe';\nexport const pipes = [DurationPipe];\n", projectRoot); + + await new CodebaseIndexer({ rootPath: projectRoot, config: { skipEmbedding: true } }).index(); + const graph = await readGraph(projectRoot); + expect(graph.graph.imports[barrel]).toContain(utility); + expect(graph.graph.imports[pipe]).toContain(barrel); + expect(graph.graph.imports[module]).toContain(pipe); + expect(graph.graph.importDetails[pipe]?.[barrel]?.sources?.[0]).toMatchObject({ + kind: 'import', specifier: alias, sourceFingerprint: expect.any(String), targetFingerprint: expect.any(String) + }); + expect(graph.graph.unresolvedEdges).toEqual(expect.arrayContaining([ + expect.objectContaining({ source: expect.objectContaining({ specifier: '@unmapped/package' }), reason: 'external' }) + ])); + expect(graph.graph.unresolvedEdges?.some(({ source }) => source.specifier === alias)).toBe(false); + + const response = await getCodeRelationships({ file: utility, direction: 'incoming', max_hops: 3, max_nodes: 10 }, context(projectRoot)); + const result = JSON.parse(response.content?.[0]?.text ?? '{}') as { + status: string; paths: Array<{ nodes: string[] }>; staleEdges?: unknown[]; truncated: boolean; + }; + expect(result.status).toBe('success'); + expect(result.paths.map(({ nodes }) => nodes)).toContainEqual([utility, barrel, pipe, module]); + expect(result.staleEdges ?? []).toHaveLength(0); + expect(result.truncated).toBe(false); +}); + +it('resolves source aliases, directory indexes, typed barrels and three hops, then refreshes after delete/rename', async () => { + await write('tsconfig.json', JSON.stringify({ compilerOptions: { baseUrl: '.', paths: { '@core/*': ['packages/core/src/*'] } } })); + await write('src/main.ts', "import './barrel';\nimport './present';\nimport './missing';\nimport './dir';\nimport '@core/actual';\nimport 'external-pkg';\n"); + await write('src/barrel.ts', "export { actual } from '@core/actual';\n"); + await write('src/present.tsx', 'export const present = true;\n'); + await write('src/dir/index.ts', 'export const indexed = true;\n'); + await write('packages/core/src/actual.ts', "import './impl';\nexport const actual = true;\n"); + await write('packages/core/src/impl.ts', "import { actual } from '@core/actual';\nexport const impl = actual;\n"); + + await new CodebaseIndexer({ rootPath: root, config: { skipEmbedding: true } }).index(); + const graph = await readGraph(); + const main = Object.keys(graph.graph.imports).find((file) => file.endsWith('src/main.ts')); + expect(main).toBeDefined(); + const targets = graph.graph.imports[main ?? ''] ?? []; + expect(targets).toContain('src/present.tsx'); + expect(targets).toContain('src/dir/index.ts'); + expect(targets).not.toContain('src/missing.ts'); + expect(graph.graph.unresolvedEdges).toEqual(expect.arrayContaining([ + expect.objectContaining({ source: expect.objectContaining({ specifier: './missing' }), reason: 'not-found' }), + expect.objectContaining({ source: expect.objectContaining({ specifier: 'external-pkg' }), reason: 'external' }) + ])); + expect(graph.graph.importDetails['src/barrel.ts']?.['packages/core/src/actual.ts']?.sources?.[0]).toMatchObject({ + kind: 'reexport', specifier: '@core/actual', sourceFingerprint: expect.any(String), targetFingerprint: expect.any(String) + }); + + const response = await getCodeRelationships({ file: 'src/main.ts', direction: 'outgoing', max_hops: 3, max_nodes: 30 }, context()); + const result = JSON.parse(response.content?.[0]?.text ?? '{}') as { status: string; paths: Array<{ nodes: string[] }>; unresolved: unknown[] }; + expect(result.status).toBe('success'); + expect(result.paths.some(({ nodes }) => nodes.join('|') === 'src/main.ts|src/barrel.ts|packages/core/src/actual.ts|packages/core/src/impl.ts')).toBe(true); + expect(result.unresolved).toHaveLength(2); + + await fs.rename(path.join(root, 'packages/core/src/actual.ts'), path.join(root, 'packages/core/src/actual-renamed.ts')); + await fs.rm(path.join(root, 'packages/core/src/impl.ts')); + await new CodebaseIndexer({ rootPath: root, config: { skipEmbedding: true }, incrementalOnly: true }).index(); + const refreshed = await readGraph(); + expect(refreshed.header.buildId).not.toBe(graph.header.buildId); + expect(refreshed.graph.imports['src/barrel.ts'] ?? []).not.toContain('packages/core/src/actual.ts'); + expect(refreshed.graph.unresolvedEdges).toEqual(expect.arrayContaining([ + expect.objectContaining({ from: 'src/barrel.ts', source: expect.objectContaining({ specifier: '@core/actual' }) }) + ])); +}); + +it('rejects graph edges after either current endpoint changes without reindexing', async () => { + await Promise.all([ + fs.rm(path.join(root, 'src'), { recursive: true, force: true }), + fs.rm(path.join(root, 'packages'), { recursive: true, force: true }), + fs.rm(path.join(root, CODEBASE_CONTEXT_DIRNAME), { recursive: true, force: true }), + fs.rm(path.join(root, 'tsconfig.json'), { force: true }) + ]); + await write('tsconfig.json', JSON.stringify({ compilerOptions: { baseUrl: '.', paths: { '@core/*': ['packages/core/src/*'] } } })); + await write('src/main.ts', "import '@core/actual';\n"); + await write('packages/core/src/actual.ts', 'export const actual = true;\n'); + await new CodebaseIndexer({ rootPath: root, config: { skipEmbedding: true } }).index(); + await fs.appendFile(path.join(root, 'packages/core/src/actual.ts'), 'export const changed = true;\n'); + + const response = await getCodeRelationships({ file: 'src/main.ts', max_hops: 1 }, context()); + const result = JSON.parse(response.content?.[0]?.text ?? '{}') as { status: string; paths: unknown[]; staleEdges?: unknown[] }; + expect(result.status).toBe('stale_edges'); + expect(result.paths).toHaveLength(0); + expect(result.staleEdges).toHaveLength(1); + + await new CodebaseIndexer({ rootPath: root, config: { skipEmbedding: true } }).index(); + await fs.appendFile(path.join(root, 'src/main.ts'), '// changed importer\n'); + const changedSourceResponse = await getCodeRelationships({ file: 'src/main.ts', max_hops: 1 }, context()); + const changedSource = JSON.parse(changedSourceResponse.content?.[0]?.text ?? '{}') as { + status: string; paths: unknown[]; staleEdges?: unknown[]; + }; + expect(changedSource.status).toBe('stale_edges'); + expect(changedSource.paths).toHaveLength(0); + expect(changedSource.staleEdges).toHaveLength(1); +}); + +it('marks edges stale when current tsconfig resolution changes or a higher priority target appears', async () => { + await Promise.all([ + fs.rm(path.join(root, 'src'), { recursive: true, force: true }), + fs.rm(path.join(root, 'packages'), { recursive: true, force: true }), + fs.rm(path.join(root, CODEBASE_CONTEXT_DIRNAME), { recursive: true, force: true }), + fs.rm(path.join(root, 'tsconfig.json'), { force: true }) + ]); + await write('tsconfig.json', JSON.stringify({ compilerOptions: { baseUrl: '.', paths: { '@core/*': ['packages/core/src/*'] } } })); + await write('src/main.ts', "import './feature';\nimport '@core/actual';\n"); + await write('src/feature.tsx', 'export const feature = true;\n'); + await write('packages/core/src/actual.ts', 'export const actual = true;\n'); + await write('packages/alternate/src/actual.ts', 'export const actual = false;\n'); + await new CodebaseIndexer({ rootPath: root, config: { skipEmbedding: true } }).index(); + const original = await getCodeRelationships({ file: 'src/main.ts', max_hops: 1 }, context()); + expect(JSON.parse(original.content?.[0]?.text ?? '{}')).toMatchObject({ status: 'success' }); + + await write('src/feature.ts', 'export const feature = false;\n'); + await write('tsconfig.json', JSON.stringify({ compilerOptions: { baseUrl: '.', paths: { '@core/*': ['packages/alternate/src/*'] } } })); + const changedResolution = await getCodeRelationships({ file: 'src/main.ts', max_hops: 1 }, context()); + const result = JSON.parse(changedResolution.content?.[0]?.text ?? '{}') as { + status: string; paths: unknown[]; staleEdges: Array<{ from: string; to: string; specifier: string }>; + }; + expect(result.status).toBe('stale_edges'); + expect(result.paths).toHaveLength(0); + expect(result.staleEdges).toEqual(expect.arrayContaining([ + expect.objectContaining({ specifier: './feature', to: 'src/feature.tsx' }), + expect.objectContaining({ specifier: '@core/actual', to: 'packages/core/src/actual.ts' }) + ])); +}); + +it('bounds the complete serialized branching response and reports truncation', async () => { + await Promise.all([ + fs.rm(path.join(root, 'src'), { recursive: true, force: true }), + fs.rm(path.join(root, CODEBASE_CONTEXT_DIRNAME), { recursive: true, force: true }), + fs.rm(path.join(root, 'tsconfig.json'), { force: true }) + ]); + const imports: string[] = []; + for (let index = 0; index < 36; index += 1) { + const target = `branch-${String(index).padStart(2, '0')}-long-diagnostic-target-name.ts`; + imports.push(`import './${target.slice(0, -3)}';`); + await write(`src/${target}`, `export const branch${index} = ${index};\n`); + } + await write('src/main.ts', `${imports.join('\n')}\n`); + await new CodebaseIndexer({ rootPath: root, config: { skipEmbedding: true } }).index(); + + const response = await getCodeRelationships({ file: 'src/main.ts', max_hops: 1, max_nodes: 100 }, context()); + const payload = response.content?.[0]?.text ?? ''; + const result = JSON.parse(payload) as { paths: unknown[]; truncated: boolean }; + expect(Buffer.byteLength(payload, 'utf8')).toBeLessThanOrEqual(4000); + expect(result.paths.length).toBeGreaterThan(0); + expect(result.truncated).toBe(true); +}); + +it('rejects legacy proofless edges and reports a missing sidecar', async () => { + await Promise.all([ + fs.rm(path.join(root, 'src'), { recursive: true, force: true }), + fs.rm(path.join(root, CODEBASE_CONTEXT_DIRNAME), { recursive: true, force: true }), + fs.rm(path.join(root, 'tsconfig.json'), { force: true }) + ]); + await write('src/from.ts', "import './to';\n"); + await write('src/to.ts', 'export const value = true;\n'); + await new CodebaseIndexer({ rootPath: root, config: { skipEmbedding: true } }).index(); + + const sidecarPath = path.join(root, CODEBASE_CONTEXT_DIRNAME, RELATIONSHIPS_FILENAME); + const sidecar = JSON.parse(await fs.readFile(sidecarPath, 'utf8')) as { + graph: { imports: Record; importDetails: Record> }; + }; + sidecar.graph.importDetails = { 'src/from.ts': { 'src/to.ts': {} } }; + await fs.writeFile(sidecarPath, JSON.stringify(sidecar)); + const legacy = await getCodeRelationships({ file: 'src/from.ts', max_hops: 1 }, context()); + expect(JSON.parse(legacy.content?.[0]?.text ?? '{}')).toMatchObject({ status: 'stale_edges', paths: [], truncated: false }); + + await fs.rm(sidecarPath); + const missing = await getCodeRelationships({ file: 'src/from.ts', max_hops: 1 }, context()); + expect(JSON.parse(missing.content?.[0]?.text ?? '{}')).toMatchObject({ status: 'graph_unavailable' }); +}); diff --git a/tests/file-graph-traversal.test.ts b/tests/file-graph-traversal.test.ts new file mode 100644 index 0000000..260dc22 --- /dev/null +++ b/tests/file-graph-traversal.test.ts @@ -0,0 +1,68 @@ +import { describe, expect, it } from 'vitest'; +import { InternalFileGraph, traverseFileGraph } from '../src/utils/usage-tracker.js'; + +describe('verified file graph traversal', () => { + it('round-trips typed source proofs and returns a three-hop path with edge evidence', () => { + const graph = new InternalFileGraph('C:/repo'); + graph.trackImport('C:/repo/src/a.ts', 'C:/repo/src/b.ts', 2, ['B'], { + kind: 'import', specifier: './b', sourceFingerprint: 'sha256:a', targetFingerprint: 'sha256:b' + }); + graph.trackImport('C:/repo/src/b.ts', 'C:/repo/src/c/index.ts', 4, [], { + kind: 'reexport', specifier: '../c', sourceFingerprint: 'sha256:b', targetFingerprint: 'sha256:c' + }); + graph.trackImport('C:/repo/src/c/index.ts', 'C:/repo/src/d.tsx', 1, [], { + kind: 'import', specifier: '../d', sourceFingerprint: 'sha256:c', targetFingerprint: 'sha256:d' + }); + graph.trackUnresolved('C:/repo/src/a.ts', { + kind: 'import', line: 5, specifier: './missing', sourceFingerprint: 'sha256:a', targetFingerprint: '' + }, 'not-found'); + + const json = graph.toJSON(); + const restored = InternalFileGraph.fromJSON(json, 'C:/repo').toJSON(); + expect(Object.keys(restored.importDetails ?? {})).toHaveLength(3); + expect(restored.importDetails?.['src/b.ts']?.['src/c/index.ts']?.sources?.[0]).toMatchObject({ kind: 'reexport', line: 4, specifier: '../c' }); + expect(restored.unresolvedEdges?.[0]?.reason).toBe('not-found'); + + const result = traverseFileGraph(restored, 'src/a.ts', { direction: 'outgoing', maxHops: 3 }); + expect(result.paths.map((entry) => entry.nodes)).toContainEqual([ + 'src/a.ts', 'src/b.ts', 'src/c/index.ts', 'src/d.tsx' + ]); + expect(result.paths.find((entry) => entry.nodes.length === 4)?.edges[1]).toMatchObject({ + kind: 'reexport', source: { line: 4, specifier: '../c', sourceFingerprint: 'sha256:b', targetFingerprint: 'sha256:c' } + }); + expect(result.unresolved).toHaveLength(1); + }); + + it('supports incoming direction, caps hops and terminates cycles', () => { + const graph = new InternalFileGraph('C:/repo'); + graph.trackImport('C:/repo/a.ts', 'C:/repo/b.ts', 1, [], { kind: 'import', specifier: './b', sourceFingerprint: 'a', targetFingerprint: 'b' }); + graph.trackImport('C:/repo/b.ts', 'C:/repo/c.ts', 2, [], { kind: 'import', specifier: './c', sourceFingerprint: 'b', targetFingerprint: 'c' }); + graph.trackImport('C:/repo/c.ts', 'C:/repo/d.ts', 3, [], { kind: 'import', specifier: './d', sourceFingerprint: 'c', targetFingerprint: 'd' }); + graph.trackImport('C:/repo/d.ts', 'C:/repo/a.ts', 4, [], { kind: 'import', specifier: './a', sourceFingerprint: 'd', targetFingerprint: 'a' }); + const serialized = graph.toJSON(); + + const outgoing = traverseFileGraph(serialized, 'a.ts', { direction: 'outgoing', maxHops: 2 }); + expect(outgoing.paths.every((entry) => entry.edges.length <= 2)).toBe(true); + expect(outgoing.truncated).toBe(false); + const incoming = traverseFileGraph(serialized, 'd.ts', { direction: 'incoming', maxHops: 1 }); + expect(incoming.paths).toContainEqual({ nodes: ['d.ts', 'c.ts'], edges: [expect.objectContaining({ from: 'c.ts', to: 'd.ts' })] }); + const capped = traverseFileGraph(serialized, 'a.ts', { direction: 'outgoing', maxHops: 3, maxNodes: 2 }); + expect(capped.truncated).toBe(true); + const outputCapped = traverseFileGraph(serialized, 'a.ts', { direction: 'outgoing', maxHops: 3, maxPaths: 1 }); + expect(outputCapped.paths).toHaveLength(1); + expect(outputCapped.truncated).toBe(true); + }); + + it.each(['incoming', 'outgoing'] as const)('terminates a cycle reachable inside three hops in %s direction', (direction) => { + const graph = new InternalFileGraph('C:/repo'); + const pairs = [['a.ts', 'b.ts'], ['b.ts', 'a.ts'], ['b.ts', 'c.ts'], ['c.ts', 'b.ts']] as const; + for (const [from, to] of pairs) { + graph.trackImport(`C:/repo/${from}`, `C:/repo/${to}`, 1, [], { + kind: 'import', specifier: `./${to.slice(0, -3)}`, sourceFingerprint: from, targetFingerprint: to + }); + } + const result = traverseFileGraph(graph.toJSON(), 'a.ts', { direction, maxHops: 3 }); + expect(result.paths.map(({ nodes }) => nodes)).toEqual([['a.ts', 'b.ts'], ['a.ts', 'b.ts', 'c.ts']]); + expect(result.truncated).toBe(false); + }); +}); diff --git a/tests/file-watcher.test.ts b/tests/file-watcher.test.ts index cde1f45..5c11e17 100644 --- a/tests/file-watcher.test.ts +++ b/tests/file-watcher.test.ts @@ -1,7 +1,9 @@ -import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import { promises as fs } from 'fs'; import path from 'path'; import os from 'os'; +import { EventEmitter } from 'node:events'; +import chokidar, { type FSWatcher } from 'chokidar'; import { startFileWatcher } from '../src/core/file-watcher.js'; import { rmWithRetries } from './test-helpers.js'; @@ -16,6 +18,55 @@ describe('FileWatcher', () => { await rmWithRetries(tempDir); }); + it('waits for close completion once and ignores events after stop', async () => { + let finishClose!: () => void; + const closeCompletion = new Promise((resolve) => { + finishClose = resolve; + }); + const watcher = new EventEmitter() as unknown as FSWatcher; + watcher.close = vi.fn(() => closeCompletion); + const watchSpy = vi.spyOn(chokidar, 'watch').mockReturnValue(watcher); + const onReady = vi.fn(); + const onChanged = vi.fn(); + let stop: (() => Promise) | undefined; + try { + stop = startFileWatcher({ + rootPath: tempDir, + debounceMs: 20, + onReady, + onChanged + }); + watcher.emit('add', path.join(tempDir, 'before-stop.ts')); + + const firstStop: unknown = stop(); + const secondStop: unknown = stop(); + expect(firstStop).toBeInstanceOf(Promise); + expect(secondStop).toBe(firstStop); + expect(watcher.close).toHaveBeenCalledTimes(1); + + watcher.emit('add', path.join(tempDir, 'after-stop.ts')); + watcher.emit('ready'); + await new Promise((resolve) => setTimeout(resolve, 40)); + expect(onChanged).not.toHaveBeenCalled(); + expect(onReady).not.toHaveBeenCalled(); + + let stopResolved = false; + const completion = Promise.resolve(firstStop).then(() => { + stopResolved = true; + }); + await Promise.resolve(); + expect(stopResolved).toBe(false); + + finishClose(); + await completion; + expect(stopResolved).toBe(true); + } finally { + finishClose(); + watchSpy.mockRestore(); + if (stop) await stop(); + } + }); + it('triggers onChanged after debounce window', async () => { const debounceMs = 400; let callCount = 0; @@ -29,7 +80,9 @@ describe('FileWatcher', () => { rootPath: tempDir, debounceMs, onReady: () => resolveReady(), - onChanged: () => { callCount++; }, + onChanged: () => { + callCount++; + } }); try { @@ -40,7 +93,7 @@ describe('FileWatcher', () => { await new Promise((resolve) => setTimeout(resolve, debounceMs + 1000)); expect(callCount).toBe(1); } finally { - stop(); + await stop(); } }, 8000); @@ -57,7 +110,9 @@ describe('FileWatcher', () => { rootPath: tempDir, debounceMs, onReady: () => resolveReady(), - onChanged: () => { callCount++; }, + onChanged: () => { + callCount++; + } }); try { @@ -72,7 +127,7 @@ describe('FileWatcher', () => { await new Promise((resolve) => setTimeout(resolve, debounceMs + 1200)); expect(callCount).toBe(1); } finally { - stop(); + await stop(); } }, 8000); @@ -89,19 +144,25 @@ describe('FileWatcher', () => { rootPath: tempDir, debounceMs, onReady: () => resolveReady(), - onChanged: () => { callCount++; }, - }); - - // Give chokidar a moment to finish initializing before the first write - await ready; - await fs.writeFile(path.join(tempDir, 'cancel.ts'), 'export const y = 99;'); - // Let chokidar detect the event (including awaitWriteFinish stabilityThreshold) - // but stop before the debounce window expires. - await new Promise((resolve) => setTimeout(resolve, 350)); - stop(); - // Wait past where debounce would have fired - await new Promise((resolve) => setTimeout(resolve, debounceMs + 200)); - expect(callCount).toBe(0); + onChanged: () => { + callCount++; + } + }); + + try { + // Give chokidar a moment to finish initializing before the first write + await ready; + await fs.writeFile(path.join(tempDir, 'cancel.ts'), 'export const y = 99;'); + // Let chokidar detect the event (including awaitWriteFinish stabilityThreshold) + // but stop before the debounce window expires. + await new Promise((resolve) => setTimeout(resolve, 350)); + await stop(); + // Wait past where debounce would have fired + await new Promise((resolve) => setTimeout(resolve, debounceMs + 200)); + expect(callCount).toBe(0); + } finally { + await stop(); + } }, 5000); it('ignores changes to non-tracked file extensions', async () => { @@ -128,7 +189,7 @@ describe('FileWatcher', () => { await new Promise((resolve) => setTimeout(resolve, debounceMs + 700)); expect(callCount).toBe(0); } finally { - stop(); + await stop(); } }, 5000); @@ -156,7 +217,7 @@ describe('FileWatcher', () => { await new Promise((resolve) => setTimeout(resolve, debounceMs + 700)); expect(callCount).toBe(1); } finally { - stop(); + await stop(); } }, 5000); @@ -185,7 +246,84 @@ describe('FileWatcher', () => { await new Promise((resolve) => setTimeout(resolve, debounceMs + 700)); expect(callCount).toBe(1); } finally { - stop(); + await stop(); + } + }, 5000); + + it('prunes excluded directories at any depth without hiding similar names', async () => { + const debounceMs = 150; + let callCount = 0; + + let resolveReady!: () => void; + const ready = new Promise((resolve) => { + resolveReady = resolve; + }); + + const stop = startFileWatcher({ + rootPath: tempDir, + debounceMs, + onReady: () => resolveReady(), + onChanged: () => { + callCount++; + } + }); + + try { + await ready; + for (const excluded of ['nested/node_modules', 'dist', '.codebase-context']) { + const directory = path.join(tempDir, 'parent', excluded); + await fs.mkdir(directory, { recursive: true }); + await fs.writeFile(path.join(directory, 'ignored.ts'), 'export const ignored = true;'); + } + await new Promise((resolve) => setTimeout(resolve, debounceMs + 700)); + expect(callCount).toBe(0); + + const allowed = path.join(tempDir, 'nested', 'node_modules-copy'); + await fs.mkdir(allowed, { recursive: true }); + await fs.writeFile(path.join(allowed, 'included.ts'), 'export const included = true;'); + await new Promise((resolve) => setTimeout(resolve, debounceMs + 700)); + expect(callCount).toBe(1); + } finally { + await stop(); + } + }, 7000); + + it('watches a literal brace-named root and reports add, change, and unlink', async () => { + const braceRoot = path.join(tempDir, '{literal-root}'); + await fs.mkdir(braceRoot); + const debounceMs = 100; + let callCount = 0; + + let resolveReady!: () => void; + const ready = new Promise((resolve) => { + resolveReady = resolve; + }); + + const stop = startFileWatcher({ + rootPath: braceRoot, + debounceMs, + onReady: () => resolveReady(), + onChanged: () => { + callCount++; + } + }); + + const filePath = path.join(braceRoot, 'tracked.ts'); + try { + await ready; + await fs.writeFile(filePath, 'export const value = 1;'); + await new Promise((resolve) => setTimeout(resolve, debounceMs + 450)); + expect(callCount).toBe(1); + + await fs.writeFile(filePath, 'export const value = 2;'); + await new Promise((resolve) => setTimeout(resolve, debounceMs + 450)); + expect(callCount).toBe(2); + + await fs.unlink(filePath); + await new Promise((resolve) => setTimeout(resolve, debounceMs + 450)); + expect(callCount).toBe(3); + } finally { + await stop(); } }, 5000); }); diff --git a/tests/fixtures/angular-routing-regressions.json b/tests/fixtures/angular-routing-regressions.json new file mode 100644 index 0000000..2d71e4b --- /dev/null +++ b/tests/fixtures/angular-routing-regressions.json @@ -0,0 +1,330 @@ +{ + "version": 1, + "purpose": "Frozen source facts for Angular-local analyzer eligibility and registry routing. These controls do not change or score retrieval gold.", + "eligibilityContract": "Ordinary names such as resolve, validate, intercept and canActivate are not framework evidence. Actual Angular imports and Angular-bound syntax establish eligibility; comments, string literals, filenames and other packages do not. Existing extension support is retained. Classification inside confirmed Angular files is a separate responsibility.", + "cases": [ + { + "id": "ordinary-promise-resolve", + "filePath": "src/example.ts", + "source": "export const saveRecord = (data: object) => Promise.resolve(data);", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic", + "expectedSourceSymbols": ["saveRecord"] + }, + { + "id": "node-path-resolve", + "filePath": "src/example.ts", + "source": "import path from 'node:path'; export const resolveFile = (file: string) => path.resolve(file);", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic", + "expectedSourceSymbols": ["resolveFile"] + }, + { + "id": "user-defined-resolve", + "filePath": "src/example.ts", + "source": "export class Resolver { resolve(value: string) { return value; } }", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "ordinary-validator", + "filePath": "src/example.ts", + "source": "export class Schema { validate(value: unknown) { return value !== null; } }", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "ordinary-interceptor", + "filePath": "src/example.ts", + "source": "export class Middleware { intercept(request: object) { return request; } }", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "ordinary-guard-method", + "filePath": "src/example.ts", + "source": "export class Permissions { canActivate() { return true; } canMatch() { return false; } }", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "locally-defined-resolve-type", + "filePath": "src/example.ts", + "source": "type ResolveFn = () => string; export const lookup: ResolveFn = () => \"found\";", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "locally-defined-validator-interface", + "filePath": "src/example.ts", + "source": "interface Validator { validate(value: unknown): boolean; } export class Schema implements Validator { validate(value: unknown) { return !!value; } }", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "decorator-lookalikes-in-comments", + "filePath": "src/example.ts", + "source": "// @Component({selector: \"demo\"})\n/* @Injectable() */\nexport function normalize(value: string) { return value.trim(); }", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic", + "expectedSourceSymbols": ["normalize"] + }, + { + "id": "decorator-lookalikes-in-strings", + "filePath": "src/example.ts", + "source": "export const hints = \"@Component({}) @Injectable() CanActivateFn ResolveFn\"; export const ready = () => true;", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "angular-import-lookalike-in-comment", + "filePath": "src/example.ts", + "source": "// import { Component } from '@angular/core';\nexport const value = 1;", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "angular-import-lookalike-in-string", + "filePath": "src/example.ts", + "source": "export const example = \"import { Component } from '@angular/core';\";", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "similarly-named-nonangular-package", + "filePath": "src/example.ts", + "source": "import { Component } from 'ui-component'; @Component({}) export class Card {}", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "nestjs-injectable-is-not-angular", + "filePath": "src/example.ts", + "source": "import { Injectable } from '@nestjs/common'; @Injectable() export class PaymentsService {}", + "expectedAngularEligible": false, + "expectedAnalyzer": "nestjs" + }, + { + "id": "component-filename-is-not-proof", + "filePath": "src/plain.component.ts", + "source": "export function resolve(value: string) { return value; }", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic", + "expectedSourceSymbols": ["resolve"] + }, + { + "id": "angular-component-import", + "filePath": "src/example.ts", + "source": "import { Component } from '@angular/core'; @Component({selector: 'demo', template: ''}) export class DemoComponent {}", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-service-import", + "filePath": "src/example.ts", + "source": "import { Injectable } from '@angular/core'; @Injectable({providedIn: 'root'}) export class CacheService {}", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-directive-import", + "filePath": "src/example.ts", + "source": "import { Directive } from '@angular/core'; @Directive({selector: '[focus]'}) export class FocusDirective {}", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-pipe-import", + "filePath": "src/example.ts", + "source": "import { Pipe } from '@angular/core'; @Pipe({name: 'title'}) export class TitlePipe {}", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-module-import", + "filePath": "src/example.ts", + "source": "import { NgModule } from '@angular/core'; @NgModule({}) export class FeatureModule {}", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-aliased-component-binding", + "filePath": "src/example.ts", + "source": "import { Component as View } from '@angular/core'; @View({selector: 'demo', template: ''}) export class DemoView {}", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-namespace-decorator-binding", + "filePath": "src/example.ts", + "source": "import * as ng from '@angular/core'; @ng.Component({selector: 'demo', template: ''}) export class DemoView {}", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-class-resolver-binding", + "filePath": "src/example.ts", + "source": "import { Resolve } from '@angular/router'; export class RecordResolver implements Resolve { resolve() { return 'record'; } }", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-functional-resolvefn-binding", + "filePath": "src/example.ts", + "source": "import { ResolveFn } from '@angular/router'; export const recordResolver: ResolveFn = () => 'record';", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-aliased-functional-resolver", + "filePath": "src/example.ts", + "source": "import { ResolveFn as Lookup } from '@angular/router'; export const recordResolver: Lookup = () => 'record';", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-class-guard-binding", + "filePath": "src/example.ts", + "source": "import { CanActivate } from '@angular/router'; export class AccessGuard implements CanActivate { canActivate() { return true; } }", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-functional-guard-binding", + "filePath": "src/example.ts", + "source": "import { CanActivateChildFn } from '@angular/router'; export const childGuard: CanActivateChildFn = () => true;", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-functional-interceptor-binding", + "filePath": "src/example.ts", + "source": "import { HttpInterceptorFn as Interceptor } from '@angular/common/http'; export const attachHeaders: Interceptor = (request, next) => next(request);", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-functional-validator-binding", + "filePath": "src/example.ts", + "source": "import { AsyncValidatorFn } from '@angular/forms'; export const checkInput: AsyncValidatorFn = control => Promise.resolve(null);", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-template-extension-retained", + "filePath": "src/card.html", + "source": "
@if (ready) {

Ready

}
", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-style-extension-retained-css", + "filePath": "src/card.css", + "source": ".card { display: block; }", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-style-extension-retained-scss", + "filePath": "src/card.scss", + "source": ".card { display: block; }", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-style-extension-retained-sass", + "filePath": "src/card.sass", + "source": ".card { display: block; }", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "angular-style-extension-retained-less", + "filePath": "src/card.less", + "source": ".card { display: block; }", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular" + }, + { + "id": "javascript-eligibility-unchanged", + "filePath": "src/util.js", + "source": "export const resolveValue = value => Promise.resolve(value);", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "other-language-eligibility-unchanged", + "filePath": "src/util.py", + "source": "def resolve(value):\n return value\n", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic" + }, + { + "id": "public-excalidraw-json", + "filePath": "packages/excalidraw/data/json.ts", + "source": "import {\n EXPORT_DATA_TYPES,\n getExportSource,\n MIME_TYPES,\n VERSIONS,\n} from \"@excalidraw/common\";\n\nimport type { ExcalidrawElement, NonDeleted } from \"@excalidraw/element/types\";\n\nimport type { MaybePromise } from \"@excalidraw/common/utility-types\";\n\nimport { cleanAppStateForExport, clearAppStateForDatabase } from \"../appState\";\n\nimport { isImageFileHandle, loadFromBlob } from \"./blob\";\nimport { fileOpen, fileSave } from \"./filesystem\";\n\nimport type { AppState, BinaryFiles, LibraryItems } from \"../types\";\nimport type {\n ExportedDataState,\n ImportedDataState,\n ExportedLibraryData,\n ImportedLibraryData,\n} from \"./types\";\n\nexport type JSONExportData = {\n elements: readonly NonDeleted[];\n appState: AppState;\n files: BinaryFiles;\n};\n\n/**\n * Strips out files which are only referenced by deleted elements\n */\nconst filterOutDeletedFiles = (\n elements: readonly ExcalidrawElement[],\n files: BinaryFiles,\n) => {\n const nextFiles: BinaryFiles = {};\n for (const element of elements) {\n if (\n !element.isDeleted &&\n \"fileId\" in element &&\n element.fileId &&\n files[element.fileId]\n ) {\n nextFiles[element.fileId] = files[element.fileId];\n }\n }\n return nextFiles;\n};\n\nexport const serializeAsJSON = (\n elements: readonly ExcalidrawElement[],\n appState: Partial,\n files: BinaryFiles,\n type: \"local\" | \"database\",\n): string => {\n const data: ExportedDataState = {\n type: EXPORT_DATA_TYPES.excalidraw,\n version: VERSIONS.excalidraw,\n source: getExportSource(),\n elements,\n appState:\n type === \"local\"\n ? cleanAppStateForExport(appState)\n : clearAppStateForDatabase(appState),\n files:\n type === \"local\"\n ? filterOutDeletedFiles(elements, files)\n : // will be stripped from JSON\n undefined,\n };\n\n return JSON.stringify(data, null, 2);\n};\n\nexport const saveAsJSON = async ({\n data,\n filename,\n fileHandle,\n}: {\n data: MaybePromise;\n filename: string;\n fileHandle: AppState[\"fileHandle\"];\n}) => {\n const blob = Promise.resolve(data).then(({ elements, appState, files }) => {\n const serialized = serializeAsJSON(elements, appState, files, \"local\");\n return new Blob([serialized], {\n type: MIME_TYPES.excalidraw,\n });\n });\n\n const savedFileHandle = await fileSave(blob, {\n name: filename,\n extension: \"excalidraw\",\n description: \"Excalidraw file\",\n fileHandle: isImageFileHandle(fileHandle) ? null : fileHandle,\n });\n return { fileHandle: savedFileHandle };\n};\n\nexport const loadFromJSON = async (\n localAppState: AppState,\n localElements: readonly ExcalidrawElement[] | null,\n) => {\n const file = await fileOpen({\n description: \"Excalidraw files\",\n // ToDo: Be over-permissive until https://bugs.webkit.org/show_bug.cgi?id=34442\n // gets resolved. Else, iOS users cannot open `.excalidraw` files.\n // extensions: [\"json\", \"excalidraw\", \"png\", \"svg\"],\n });\n return loadFromBlob(file, localAppState, localElements, file.handle);\n};\n\nexport const isValidExcalidrawData = (data?: {\n type?: any;\n elements?: any;\n appState?: any;\n}): data is ImportedDataState => {\n return (\n data?.type === EXPORT_DATA_TYPES.excalidraw &&\n (!data.elements ||\n (Array.isArray(data.elements) &&\n (!data.appState || typeof data.appState === \"object\")))\n );\n};\n\nexport const isValidLibrary = (json: any): json is ImportedLibraryData => {\n return (\n typeof json === \"object\" &&\n json &&\n json.type === EXPORT_DATA_TYPES.excalidrawLibrary &&\n (json.version === 1 || json.version === 2)\n );\n};\n\nexport const serializeLibraryAsJSON = (libraryItems: LibraryItems) => {\n const data: ExportedLibraryData = {\n type: EXPORT_DATA_TYPES.excalidrawLibrary,\n version: VERSIONS.excalidrawLibrary,\n source: getExportSource(),\n libraryItems,\n };\n return JSON.stringify(data, null, 2);\n};\n\nexport const saveLibraryAsJSON = async (libraryItems: LibraryItems) => {\n const serialized = serializeLibraryAsJSON(libraryItems);\n await fileSave(\n new Blob([serialized], {\n type: MIME_TYPES.excalidrawlib,\n }),\n {\n name: \"library\",\n extension: \"excalidrawlib\",\n description: \"Excalidraw library file\",\n },\n );\n};\n", + "expectedAngularEligible": false, + "expectedAnalyzer": "generic", + "expectedSourceSymbols": ["serializeAsJSON", "saveAsJSON"], + "origin": { + "repositoryUrl": "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/excalidraw/excalidraw", + "path": "packages/excalidraw/data/json.ts", + "sourceSha256": "8ae8ad064f9a2f2060fe6ca62b8a5cf3cebeaa7dbc7c4ee8b227ed449cf9555b", + "lineStart": 1, + "lineEnd": 159, + "referenceFixture": "tests/fixtures/discovery-excalidraw.json", + "fixtureRepositoryRef": "e18c1dd213000dde0ae94ef7eb00aab537b39708", + "snapshotRefVerification": "not independently established; source bytes and hash are authoritative", + "licenseNotice": "excalidraw-mit" + } + }, + { + "id": "public-angular-spotify-component", + "filePath": "apps/angular-spotify/src/app/app.component.ts", + "source": "import { Component } from '@angular/core';\r\n\r\n@Component({\r\n selector: 'angular-spotify-root',\r\n template: ` `,\r\n styleUrls: ['./app.component.scss']\r\n})\r\nexport class AppComponent {}\r\n", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular", + "origin": { + "repositoryUrl": "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/trungk18/angular-spotify", + "path": "apps/angular-spotify/src/app/app.component.ts", + "sourceSha256": "187e85f943f37ed8cd87eb7ecfee0b00850128e4e82fadbcf8510170ae962b18", + "lineStart": 1, + "lineEnd": 8, + "repositoryRef": "ff9efa765c53cfde78c9a172c62d515ae8ef9fe0", + "licenseNotice": "angular-spotify-mit" + } + }, + { + "id": "public-angular-spotify-auth-interceptor", + "filePath": "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "source": "import { Injectable, Provider } from '@angular/core';\r\nimport {\r\n HttpEvent,\r\n HttpInterceptor,\r\n HttpHandler,\r\n HttpRequest,\r\n HTTP_INTERCEPTORS\r\n} from '@angular/common/http';\r\nimport { Observable } from 'rxjs';\r\nimport { AuthStore } from '@angular-spotify/web/auth/data-access';\r\nimport { switchMap, take } from 'rxjs/operators';\r\nimport { shouldSkipInterceptor } from './skip-urls';\r\n\r\n@Injectable()\r\nexport class AuthInterceptor implements HttpInterceptor {\r\n constructor(private authStore: AuthStore) {}\r\n\r\n intercept(req: HttpRequest, next: HttpHandler): Observable> {\r\n if (shouldSkipInterceptor(req.url)) {\r\n return next.handle(req);\r\n }\r\n\r\n return this.authStore.token$.pipe(\r\n take(1),\r\n switchMap((token) => {\r\n if (!token) {\r\n return next.handle(req);\r\n }\r\n const headers = req.headers.set('Authorization', `Bearer ${token}`);\r\n const authReq = req.clone({\r\n headers\r\n });\r\n return next.handle(authReq);\r\n })\r\n );\r\n }\r\n}\r\n\r\nexport const authInterceptorProvider: Provider = {\r\n provide: HTTP_INTERCEPTORS,\r\n useClass: AuthInterceptor,\r\n multi: true\r\n};\r\n", + "expectedAngularEligible": true, + "expectedAnalyzer": "angular", + "origin": { + "repositoryUrl": "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/trungk18/angular-spotify", + "path": "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "sourceSha256": "186562e4372376eb2ca1c86d51ac0255d35ef9132f7ee9281cd8eefb1d052e3a", + "lineStart": 1, + "lineEnd": 43, + "repositoryRef": "ff9efa765c53cfde78c9a172c62d515ae8ef9fe0", + "licenseNotice": "angular-spotify-mit" + } + } + ], + "thirdPartyNotices": [ + { + "id": "excalidraw-mit", + "repositoryUrl": "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/excalidraw/excalidraw", + "licensePath": "LICENSE", + "licenseSourceSha256": "1352d204fdb90d5e482c13139d848bc2c9300a9a7de358a0a558a23e72d0d8be", + "licenseText": "MIT License\n\nCopyright (c) 2020 Excalidraw\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" + }, + { + "id": "angular-spotify-mit", + "repositoryUrl": "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/trungk18/angular-spotify", + "licensePath": "LICENSE", + "licenseSourceSha256": "002ff3de0a6eebc81bf78cd524b87c5388e334e796c7f8fe29ad829d1e9b64da", + "licenseText": "MIT License\r\n\r\nCopyright (c) 2021 Trung Vo\r\n\r\nPermission is hereby granted, free of charge, to any person obtaining a copy\r\nof this software and associated documentation files (the \"Software\"), to deal\r\nin the Software without restriction, including without limitation the rights\r\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\ncopies of the Software, and to permit persons to whom the Software is\r\nfurnished to do so, subject to the following conditions:\r\n\r\nThe above copyright notice and this permission notice shall be included in all\r\ncopies or substantial portions of the Software.\r\n\r\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\nSOFTWARE.\r\n" + } + ] +} diff --git a/tests/fixtures/eval-angular-spotify-supplemental.json b/tests/fixtures/eval-angular-spotify-supplemental.json new file mode 100644 index 0000000..45ec2b7 --- /dev/null +++ b/tests/fixtures/eval-angular-spotify-supplemental.json @@ -0,0 +1,133 @@ +{ + "description": "Supplemental source-grounded retrieval evaluation for angular-spotify.", + "codebase": "angular-spotify", + "repository": "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/trungk18/angular-spotify", + "repositoryUrl": "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/trungk18/angular-spotify", + "repositoryRef": "ff9efa765c53cfde78c9a172c62d515ae8ef9fe0", + "repositoryRefStatus": "Task-supplied pinned revision; snapshot has no .git metadata for independent verification.", + "license": "MIT", + "frozenDate": "2026-10-15", + "notes": "Supplemental set authored after candidate-04 source/archive freeze; it is not a pre-implementation evaluation and does not replace the existing evaluation set. Do not infer adoption, release quality, or broad generalization from this set alone.", + "queries": [ + { + "id": 1, + "query": "When a Spotify API call comes back unauthorized, where is that converted into the expired-session UI signal, and what does the interceptor actually do?", + "category": "conceptual_behavior", + "expectedPatterns": ["libs/web/auth/util/src/lib/interceptors/unauthorized.interceptor.ts"], + "expectedNotPatterns": ["libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts"], + "notes": "Distinguishes 401 UI behavior from adding credentials to requests." + }, + { + "id": 2, + "query": "Which URLs bypass the shared HTTP interceptors, including the token endpoint and the external lyrics service, and what is the URL matching rule?", + "category": "exact_identifier_and_configuration", + "expectedPatterns": ["libs/web/auth/util/src/lib/interceptors/skip-urls.ts"], + "expectedNotPatterns": [ + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/auth/util/src/lib/interceptors/unauthorized.interceptor.ts" + ], + "notes": "The helper holds the endpoint list and substring predicate." + }, + { + "id": 3, + "query": "Trace the Spotify PKCE callback: where is the verifier recovered, the authorization code exchanged, tokens saved, and the pre-login route restored?", + "category": "multi_concept_wiring", + "expectedPatterns": [ + "libs/web/auth/data-access/src/lib/store/auth.store.ts", + "libs/web/auth/data-access/src/lib/models/spotify-authorize.ts" + ], + "expectedNotPatterns": [ + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/auth/util/src/lib/interceptors/skip-urls.ts" + ], + "notes": "Requires the auth flow store plus its authorize URL helper." + }, + { + "id": 4, + "query": "How does search replace results for a new term but append tracks, artists, albums, and playlists when another page is requested?", + "category": "conceptual_pagination", + "expectedPatterns": ["libs/web/search/data-access/src/lib/store/search.store.ts"], + "expectedNotPatterns": ["libs/web/search/feature/src/lib/search.component.ts"], + "notes": "Targets state/effect behavior, not the search page presentation." + }, + { + "id": 5, + "query": "Where are recent-played pages fetched with Spotify after-cursors and appended into the existing track list?", + "category": "multi_file_data_flow", + "expectedPatterns": [ + "libs/web/home/data-access/src/lib/store/recent-played-tracks/recent-played-tracks.effect.ts", + "libs/web/home/data-access/src/lib/store/recent-played-tracks/recent-played-tracks.reducer.ts" + ], + "expectedNotPatterns": [ + "libs/web/home/ui/recent-played/src/lib/recent-played.component.html" + ], + "notes": "Fetch/cursor and accumulation cross the effect and reducer." + }, + { + "id": 6, + "query": "What shared sentinel directive emits when the bottom enters view, and how does the recent-played component turn that into a load-more action?", + "category": "cross_layer_wiring", + "expectedPatterns": [ + "libs/web/shared/ui/infinite-scroll/src/lib/infinite-scroll.directive.ts", + "libs/web/home/ui/recent-played/src/lib/recent-played.component.ts" + ], + "expectedNotPatterns": [ + "libs/web/home/data-access/src/lib/store/recent-played-tracks/recent-played-tracks.reducer.ts" + ], + "notes": "Separates viewport detection from feature-level dispatch." + }, + { + "id": 7, + "query": "Where are Spotify audio-analysis segments converted into energy, pitch, timbre and beat-phase values before the selected visualizer renderer is updated?", + "category": "conceptual_data_transformation", + "expectedPatterns": ["libs/web/visualizer/ui/src/lib/web-visualizer-ui.component.ts"], + "expectedNotPatterns": ["libs/web/visualizer/data-access/src/lib/store/visualizer.store.ts"], + "notes": "Targets the live analysis-to-audio-data conversion, not selection state." + }, + { + "id": 8, + "query": "How does the visualizer switch into picture-in-picture after leaving its route, and what happens when the route is visited again?", + "category": "navigation_state_behavior", + "expectedPatterns": ["libs/web/visualizer/data-access/src/lib/store/visualizer.store.ts"], + "expectedNotPatterns": ["libs/web/lyrics/ui/lyrics-pip/src/lib/lyrics-pip.component.ts"], + "notes": "Tests route-event state transitions, not the separate lyrics PiP widget." + }, + { + "id": 9, + "query": "Which effect persists the current settings after the volume-persist action, and which service applies the app-specific storage key prefix?", + "category": "exact_identifier_and_wiring", + "expectedPatterns": [ + "libs/web/settings/data-access/src/lib/store/settings.effects.ts", + "libs/web/settings/data-access/src/lib/services/local-storage.service.ts" + ], + "expectedNotPatterns": ["libs/web/auth/data-access/src/lib/store/auth.store.ts"], + "notes": "Connects the settings effect to its storage utility." + }, + { + "id": 10, + "query": "Where does the Browse shell choose between the category index and a category-id route, while registering the categories and category-playlists state/effects?", + "category": "routing_and_registration", + "expectedPatterns": ["libs/web/browse/feature/shell/src/lib/browse-shell.module.ts"], + "expectedNotPatterns": [ + "libs/web/browse/feature/categories/src/lib/categories.component.ts", + "libs/web/browse/feature/category/src/lib/category.component.ts" + ], + "notes": "Looks for the route/state composition module, not either routed view." + } + ], + "idProvenance": { + "mapping": { + "ang-01": 1, + "ang-02": 2, + "ang-03": 3, + "ang-04": 4, + "ang-05": 5, + "ang-06": 6, + "ang-07": 7, + "ang-08": 8, + "ang-09": 9, + "ang-10": 10 + }, + "reason": "declared EvalQuery.id:number; factual schema alignment before first score, with question, expected paths, categories, and order unchanged." + } +} diff --git a/tests/fixtures/eval-excalidraw-supplemental.json b/tests/fixtures/eval-excalidraw-supplemental.json new file mode 100644 index 0000000..15f39b2 --- /dev/null +++ b/tests/fixtures/eval-excalidraw-supplemental.json @@ -0,0 +1,124 @@ +{ + "description": "Supplemental source-grounded retrieval evaluation for excalidraw/excalidraw.", + "codebase": "excalidraw/excalidraw", + "repository": "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/excalidraw/excalidraw", + "repositoryUrl": "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/excalidraw/excalidraw", + "repositoryRef": "e18c1dd213000dde0ae94ef7eb00aab537b39708", + "repositoryRefStatus": "Task-supplied pinned revision; snapshot has no .git metadata for independent verification.", + "license": "MIT", + "frozenDate": "2026-10-15", + "notes": "Supplemental set authored after candidate-04 source/archive freeze; it is not a pre-implementation evaluation and does not replace the existing evaluation set. Do not infer adoption, release quality, or broad generalization from this set alone.", + "queries": [ + { + "id": 1, + "query": "Where does the app debounce saving the non-deleted scene and app state to browser storage, then save image binaries separately?", + "category": "multi_concept_storage_flow", + "expectedPatterns": ["excalidraw-app/data/LocalData.ts"], + "expectedNotPatterns": ["excalidraw-app/data/tabSync.ts"], + "notes": "The local persistence entrypoint coordinates both scene storage and file storage." + }, + { + "id": 2, + "query": "How does a local .excalidraw JSON export filter out image files used only by deleted elements, and where is the resulting file saved?", + "category": "edge_case_export", + "expectedPatterns": [ + "packages/excalidraw/data/json.ts", + "packages/excalidraw/data/filesystem.ts" + ], + "expectedNotPatterns": [ + "packages/excalidraw/data/image.ts", + "packages/excalidraw/clipboard.ts" + ], + "notes": "Distinguishes standard JSON-file export from PNG embedding and clipboard serialization." + }, + { + "id": 3, + "query": "When old diagrams are restored, where are legacy text font and line-height values repaired, and how are incomplete arrow point lists normalized?", + "category": "compatibility_repair", + "expectedPatterns": ["packages/excalidraw/data/restore.ts"], + "expectedNotPatterns": ["packages/excalidraw/tests/data/restore.test.ts"], + "notes": "Request asks for implementation; the nearby restore test is an explicit distractor." + }, + { + "id": 4, + "query": "How is scene JSON compressed into a PNG text chunk and decoded back out, including the legacy unencoded scene case?", + "category": "exact_identifier_and_compatibility", + "expectedPatterns": ["packages/excalidraw/data/image.ts"], + "expectedNotPatterns": [ + "packages/excalidraw/data/json.ts", + "packages/excalidraw/tests/data/restore.test.ts" + ], + "notes": "Targets PNG metadata helpers rather than the ordinary .excalidraw JSON serializer." + }, + { + "id": 5, + "query": "How does the app map the browser language to a supported locale, fall back to the default, and cache a user selection?", + "category": "configuration_and_wiring", + "expectedPatterns": [ + "excalidraw-app/app-language/language-detector.ts", + "excalidraw-app/app-language/language-state.ts", + "excalidraw-app/app-language/LanguageList.tsx" + ], + "expectedNotPatterns": ["excalidraw-app/tests/LanguageList.test.tsx"], + "notes": "The detector, state hook and selector are distinct steps; test file is the negative match." + }, + { + "id": 6, + "query": "Where are collaboration scene updates encrypted before socket emission, and where are incoming packets decrypted and reconciled into the scene?", + "category": "cross_module_protocol", + "expectedPatterns": ["excalidraw-app/collab/Portal.tsx", "excalidraw-app/collab/Collab.tsx"], + "expectedNotPatterns": ["excalidraw-app/data/FileManager.ts"], + "notes": "Separates outbound transport framing from inbound room-state handling." + }, + { + "id": 7, + "query": "Which file manager tracks image file versions as fetching, saved or errored so repeated upload and fetch work can be deduplicated?", + "category": "implementation_convention", + "expectedPatterns": ["excalidraw-app/data/FileManager.ts"], + "expectedNotPatterns": ["excalidraw-app/data/fileStatusStore.ts"], + "notes": "Distinguishes per-file manager bookkeeping from the shared UI status snapshot store." + }, + { + "id": 8, + "query": "Where does the legacy localStorage library adapter load and clear saved library items as they migrate to an IndexedDB adapter, and where is that migration wired at startup?", + "category": "migration_and_startup_wiring", + "expectedPatterns": ["excalidraw-app/data/LocalData.ts", "excalidraw-app/App.tsx"], + "expectedNotPatterns": ["packages/excalidraw/data/library.ts"], + "notes": "App startup wires the adapters; LocalData defines both persistence adapters." + }, + { + "id": 9, + "query": "How does this tab decide browser-stored scene or file state is newer than its in-memory version, and update the timestamp marker?", + "category": "state_synchronization", + "expectedPatterns": ["excalidraw-app/data/tabSync.ts"], + "expectedNotPatterns": ["excalidraw-app/data/LocalData.ts"], + "notes": "Targets cross-tab timestamp comparison rather than the main local save path." + }, + { + "id": 10, + "query": "How are clipboard and drag/drop transfer items parsed when the payload mixes text and files, and how are Excalidraw element payloads recognized?", + "category": "mixed_input_edge_case", + "expectedPatterns": ["packages/excalidraw/clipboard.ts"], + "expectedNotPatterns": [ + "packages/excalidraw/tests/clipboard.test.tsx", + "packages/excalidraw/clipboard.test.ts" + ], + "notes": "Implementation source is distinct from both clipboard test suites." + } + ], + "idProvenance": { + "mapping": { + "exc-01": 1, + "exc-02": 2, + "exc-03": 3, + "exc-04": 4, + "exc-05": 5, + "exc-06": 6, + "exc-07": 7, + "exc-08": 8, + "exc-09": 9, + "exc-10": 10 + }, + "reason": "declared EvalQuery.id:number; factual schema alignment before first score, with question, expected paths, categories, and order unchanged." + } +} diff --git a/tests/fixtures/reference-nomination-parity.json b/tests/fixtures/reference-nomination-parity.json new file mode 100644 index 0000000..3778afd --- /dev/null +++ b/tests/fixtures/reference-nomination-parity.json @@ -0,0 +1,344 @@ +{ + "contract": "Freeze current exact-source reference behavior before any production nomination change. Each expected result is the current parser path (AST when parse succeeds, otherwise lexical fallback).", + "cases": [ + { + "id": "ts-exact-reference", + "language": "typescript", + "symbol": "Scene", + "source": "const Scene = 1; Scene;", + "expected": { "mode": "ast", "count": 2 } + }, + { + "id": "ts-long-identifier", + "language": "typescript", + "symbol": "Scene", + "source": "const SceneData = 1;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-dollar-identifier", + "language": "typescript", + "symbol": "Scene", + "source": "const $Scene = 1;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-underscore-identifier", + "language": "typescript", + "symbol": "Scene", + "source": "const _Scene = 1;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-underscore-suffix", + "language": "typescript", + "symbol": "Scene", + "source": "const Scene_Thing = 1;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-unicode-start-continuation", + "language": "typescript", + "symbol": "Scene", + "source": "const πScene = 1;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-unicode-end-continuation", + "language": "typescript", + "symbol": "Scene", + "source": "const Sceneπ = 1;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-combining-mark-continuation", + "language": "typescript", + "symbol": "Scene", + "source": "const Scené = 1;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-joiner-continuation", + "language": "typescript", + "symbol": "Scene", + "source": "const Scene‌part = 1;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-string-only", + "language": "typescript", + "symbol": "Scene", + "source": "const message = \"Scene\";", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-comment-only", + "language": "typescript", + "symbol": "Scene", + "source": "// Scene", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-quoted-property", + "language": "typescript", + "symbol": "Scene", + "source": "const value = { \"Scene\": 1 };", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-unicode-escape-semantic-name", + "language": "typescript", + "symbol": "Scene", + "source": "const \\u0053cene = 1;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "ts-unicode-escape-raw-query", + "language": "typescript", + "symbol": "\\u0053cene", + "source": "const \\u0053cene = 1;", + "expected": { "mode": "ast", "count": 1 } + }, + { + "id": "ts-incomplete-syntax-fallback", + "language": "typescript", + "symbol": "Scene", + "source": "const Scene =", + "expected": { "mode": "lexical", "count": 1 } + }, + { + "id": "ts-number-error-split", + "language": "typescript", + "symbol": "Scene", + "source": "const value = 0xScene;", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "javascript-number-suffix", + "language": "javascript", + "symbol": "Scene", + "source": "const value = 1Scene;", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "python-number-suffix", + "language": "python", + "symbol": "Scene", + "source": "value = 1Scene", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "go-number-suffix", + "language": "go", + "symbol": "Scene", + "source": "package p\nvar value = 1Scene", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "rust-number-suffix", + "language": "rust", + "symbol": "Scene", + "source": "fn f() { let value = 1Scene; }", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "java-number-suffix", + "language": "java", + "symbol": "Scene", + "source": "class C { int value = 1Scene; }", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "c-number-suffix", + "language": "c", + "symbol": "Scene", + "source": "int value = 1Scene;", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "cpp-number-suffix", + "language": "cpp", + "symbol": "Scene", + "source": "int value = 1Scene;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "c-hex-prefix-split", + "language": "c", + "symbol": "Scene", + "source": "int value = 0xScene;", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "cpp-hex-prefix-split", + "language": "cpp", + "symbol": "Scene", + "source": "int value = 0xScene;", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "csharp-number-suffix", + "language": "csharp", + "symbol": "Scene", + "source": "class C { int value = 1Scene; }", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "kotlin-number-suffix", + "language": "kotlin", + "symbol": "Scene", + "source": "val value = 1Scene", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "js-exact-reference", + "language": "javascript", + "symbol": "Scene", + "source": "const Scene = 1; Scene;", + "expected": { "mode": "ast", "count": 2 } + }, + { + "id": "tsx-jsx-identifier", + "language": "typescriptreact", + "symbol": "Scene", + "source": "const view = ;", + "expected": { "mode": "ast", "count": 1 } + }, + { + "id": "python-exact-reference", + "language": "python", + "symbol": "Scene", + "source": "Scene = 1\nprint(Scene)", + "expected": { "mode": "ast", "count": 2 } + }, + { + "id": "python-long-identifier", + "language": "python", + "symbol": "Scene", + "source": "SceneData = 1", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "go-exact-reference", + "language": "go", + "symbol": "Scene", + "source": "package p\nvar Scene int\nfunc f() { _ = Scene }", + "expected": { "mode": "ast", "count": 2 } + }, + { + "id": "rust-exact-reference", + "language": "rust", + "symbol": "Scene", + "source": "fn f() { let Scene = 1; println!(\"{}\", Scene); }", + "expected": { "mode": "ast", "count": 2 } + }, + { + "id": "java-type-reference", + "language": "java", + "symbol": "Scene", + "source": "class Scene { Scene value; }", + "expected": { "mode": "ast", "count": 2 } + }, + { + "id": "cpp-exact-reference", + "language": "cpp", + "symbol": "Scene", + "source": "int Scene; int f() { return Scene; }", + "expected": { "mode": "ast", "count": 2 } + }, + { + "id": "c-exact-reference", + "language": "c", + "symbol": "Scene", + "source": "int Scene; int f() { return Scene; }", + "expected": { "mode": "ast", "count": 2 } + }, + { + "id": "csharp-exact-reference", + "language": "csharp", + "symbol": "Scene", + "source": "class Scene { int value; }", + "expected": { "mode": "ast", "count": 1 } + }, + { + "id": "kotlin-current-node-name-contract", + "language": "kotlin", + "symbol": "Scene", + "source": "fun main() { val Scene = 1; println(Scene) }", + "expected": { "mode": "ast", "count": 0 } + }, + { + "id": "unknown-quoted-and-comment-fallback", + "language": "unknown", + "symbol": "Scene", + "source": "val x = \"Scene\" // Scene", + "expected": { "mode": "lexical", "count": 2 } + }, + { + "id": "unknown-long-identifier", + "language": "unknown", + "symbol": "Scene", + "source": "SceneData", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "unknown-dollar-boundary", + "language": "unknown", + "symbol": "Scene", + "source": "$Scene Scene$", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "unknown-non-ascii-boundaries", + "language": "unknown", + "symbol": "Scene", + "source": "πScene Sceneπ", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "unknown-number-error-split", + "language": "unknown", + "symbol": "Scene", + "source": "0xScene", + "expected": { "mode": "lexical", "count": 0 } + }, + { + "id": "unknown-composite-symbol", + "language": "unknown", + "symbol": "foo.bar", + "source": "foo.bar + foobar", + "expected": { "mode": "lexical", "count": 1 } + }, + { + "id": "unknown-raw-escape-spelling", + "language": "unknown", + "symbol": "\\u0053cene", + "source": "\\u0053cene", + "expected": { "mode": "lexical", "count": 1 } + }, + { + "id": "angular-spotify-source-token", + "language": "typescript", + "symbol": "Component", + "source": "Component", + "provenance": { + "repository": "angular-spotify", + "file": "apps/angular-spotify/src/app/app.component.ts", + "sha256": "187e85f943f37ed8cd87eb7ecfee0b00850128e4e82fadbcf8510170ae962b18" + }, + "expected": { "mode": "ast", "count": 1 } + }, + { + "id": "excalidraw-source-token-inside-long-identifier", + "language": "typescript", + "symbol": "Scene", + "source": "initializeScene", + "provenance": { + "repository": "excalidraw-monorepo", + "file": "excalidraw-app/App.tsx", + "sha256": "59789614d257b77bd89a3e304de3247fed4c821401e0e30173ce52d58b71816b" + }, + "expected": { "mode": "ast", "count": 0 } + } + ] +} diff --git a/tests/fixtures/search-rescue-density.json b/tests/fixtures/search-rescue-density.json new file mode 100644 index 0000000..8195bdd --- /dev/null +++ b/tests/fixtures/search-rescue-density.json @@ -0,0 +1,478 @@ +{ + "schemaVersion": 1, + "purpose": "Frozen public controls for a secondary source-local normalized term-frequency density tie-break.", + "formula": "For each existing normalized query group g, tf_g counts normalized body-word occurrences whose existing lexicalAliases(normalizedBodyWord) intersect the aliases in g. Count that body-word occurrence at most once per group, regardless of alias-expansion multiplicity. density = sum_g sqrt(tf_g) / sqrt(max(1, normalizedBodyWordCount)).", + "ordering": "Lexicographic comparison: preserve the existing primary rank, then compare descending density only for equal primary ranks, then preserve original chunk ordinal. Never add density to or otherwise alter the primary score.", + "scope": "Synthetic source snippets and relative file paths. Expected order is a consequence of the frozen lexical formula and existing primary evidence classes; it does not assert semantic correctness.", + "cases": [ + { + "id": "density-breaks-primary-tie", + "query": "invoice approval", + "chunks": [ + { + "id": "long-noise", + "filePath": "src/long-noise.ts", + "content": "invoice approval billing invoice mail report network" + }, + { + "id": "compact", + "filePath": "src/compact.ts", + "content": "invoice approval" + } + ], + "expectedIds": ["compact", "long-noise"], + "expectedPathPatterns": ["src/compact.ts", "src/long-noise.ts"], + "expectedProperties": [ + "primary evidence ties", + "unmatched body vocabulary increases the denominator" + ] + }, + { + "id": "declaration-primary-dominates-density", + "query": "invoice approval", + "chunks": [ + { + "id": "declared-implementation", + "filePath": "src/invoice-approval.ts", + "content": "export function apply(invoice) { const approval = true; return applyPolicy(invoice, approval); } billing mail report network logging validation storage adapter response", + "metadata": { + "functionName": "invoiceApproval" + } + }, + { + "id": "dense-body", + "filePath": "src/repeated.ts", + "content": "invoice approval invoice approval invoice approval" + } + ], + "expectedIds": ["declared-implementation", "dense-body"], + "expectedPathPatterns": ["src/invoice-approval.ts", "src/repeated.ts"], + "expectedProperties": [ + "declaration evidence remains primary", + "density cannot promote a weaker primary candidate" + ] + }, + { + "id": "exact-anchor-primary-dominates-density", + "query": "InvoicePolicy invoice policy", + "chunks": [ + { + "id": "exact-anchor", + "filePath": "src/invoice-policy.ts", + "content": "invoice policy resolves a rule through a versioned registry and validates the result against the selected tenant before returning it", + "metadata": { + "symbolName": "InvoicePolicy" + } + }, + { + "id": "dense-body", + "filePath": "src/repeated-policy.ts", + "content": "invoice policy invoice policy invoice policy" + } + ], + "expectedIds": ["exact-anchor", "dense-body"], + "expectedPathPatterns": ["src/invoice-policy.ts", "src/repeated-policy.ts"], + "expectedProperties": [ + "exact indexed anchor remains primary", + "body density is only a secondary tie-break" + ] + }, + { + "id": "implementation-primary-dominates-test-density", + "query": "invoice approval", + "chunks": [ + { + "id": "implementation", + "filePath": "src/invoice-approval.ts", + "content": "invoice approval validates the request and records the accepted result after checking the account state and policy" + }, + { + "id": "test", + "filePath": "src/__tests__/invoice-approval.test.ts", + "content": "invoice approval invoice approval invoice approval invoice approval" + } + ], + "expectedIds": ["implementation", "test"], + "expectedPathPatterns": ["src/invoice-approval.ts", "src/__tests__/invoice-approval.test.ts"], + "expectedProperties": [ + "test-file primary penalty remains effective", + "repeated test-body terms do not override primary order" + ] + }, + { + "id": "own-implementation-outranks-repeated-type-mentions", + "query": "cache refresh", + "chunks": [ + { + "id": "own-implementation", + "filePath": "src/cache-refresh.ts", + "content": "refreshCache updates the cache after validating the new value and then returns the stored entry", + "metadata": { + "functionName": "refreshCache" + } + }, + { + "id": "repeated-type-mentions", + "filePath": "src/cache-types.ts", + "content": "type CacheRefresh = Cache & Refresh & CacheRefresh & CacheRefresh & CacheRefresh" + } + ], + "expectedIds": ["own-implementation", "repeated-type-mentions"], + "expectedPathPatterns": ["src/cache-refresh.ts", "src/cache-types.ts"], + "expectedProperties": [ + "own declaration evidence remains primary", + "repetition in a type body is lexical evidence only" + ] + }, + { + "id": "tiny-wrapper-does-not-beat-stronger-declaration", + "query": "invoice approval", + "chunks": [ + { + "id": "implementation", + "filePath": "src/invoice-approval.ts", + "content": "invoice approval validates the account, checks the pending state, records the decision, and returns the persisted result", + "metadata": { + "functionName": "invoiceApproval" + } + }, + { + "id": "tiny-wrapper", + "filePath": "src/wrapper.ts", + "content": "invoice approval" + } + ], + "expectedIds": ["implementation", "tiny-wrapper"], + "expectedPathPatterns": ["src/invoice-approval.ts", "src/wrapper.ts"], + "expectedProperties": [ + "declaration primary rank dominates the wrapper's higher density", + "shortness alone does not imply a better result" + ] + }, + { + "id": "metadata-repetition-is-not-body-frequency", + "query": "invoice approval", + "chunks": [ + { + "id": "repeated-metadata-first", + "filePath": "src/a.ts", + "content": "invoice approval validates a request", + "metadata": { + "componentName": "InvoiceApproval InvoiceApproval InvoiceApproval" + } + }, + { + "id": "single-metadata-second", + "filePath": "src/b.ts", + "content": "invoice approval validates a request", + "metadata": { + "componentName": "InvoiceApproval" + } + } + ], + "expectedIds": ["repeated-metadata-first", "single-metadata-second"], + "expectedPathPatterns": ["src/a.ts", "src/b.ts"], + "expectedProperties": [ + "body density is identical", + "metadata repetition does not increase term frequency", + "original ordinal breaks the tie" + ] + }, + { + "id": "body-inflection-alias-matching-preserves-ordinal", + "query": "archive policy", + "chunks": [ + { + "id": "inflected-first", + "filePath": "src/a.ts", + "content": "archiving policy" + }, + { + "id": "canonical-second", + "filePath": "src/b.ts", + "content": "archive policy" + } + ], + "expectedIds": ["inflected-first", "canonical-second"], + "expectedPathPatterns": ["src/a.ts", "src/b.ts"], + "expectedProperties": [ + "existing body lexical aliases are intersected with the query-group aliases", + "archiving and archive each contribute one occurrence to the archive group", + "both bodies have equal density and original ordinal is preserved" + ] + }, + { + "id": "overlapping-aliases-count-one-occurrence-per-group", + "query": "archive archiving policy", + "chunks": [ + { + "id": "archiving-first", + "filePath": "src/a.ts", + "content": "archiving policy" + }, + { + "id": "archive-second", + "filePath": "src/b.ts", + "content": "archive policy" + } + ], + "expectedIds": ["archiving-first", "archive-second"], + "expectedPathPatterns": ["src/a.ts", "src/b.ts"], + "expectedProperties": [ + "archive and archiving belong to one existing alias group", + "one body occurrence contributes once to that group", + "equal density preserves ordinal" + ] + }, + { + "id": "repeat-one-group-does-not-win-on-raw-count", + "query": "archive policy", + "chunks": [ + { + "id": "balanced", + "filePath": "src/balanced.ts", + "content": "archive policy" + }, + { + "id": "repeated-archive", + "filePath": "src/repeated.ts", + "content": "archive archive policy" + } + ], + "expectedIds": ["balanced", "repeated-archive"], + "expectedPathPatterns": ["src/balanced.ts", "src/repeated.ts"], + "expectedProperties": [ + "balanced density is 2/sqrt(2)", + "repeated density is (sqrt(2)+1)/sqrt(3)", + "square-root TF and body length are both applied" + ] + }, + { + "id": "unrelated-billing-mail-report-network-words-only-add-length", + "query": "archive policy", + "chunks": [ + { + "id": "compact", + "filePath": "src/compact.ts", + "content": "archive policy" + }, + { + "id": "unrelated-vocabulary", + "filePath": "src/unrelated.ts", + "content": "archive policy billing invoice mail report network" + } + ], + "expectedIds": ["compact", "unrelated-vocabulary"], + "expectedPathPatterns": ["src/compact.ts", "src/unrelated.ts"], + "expectedProperties": [ + "billing, mail, report, and network words are not query clues", + "unmatched body words increase the denominator only" + ] + }, + { + "id": "overlapping-chunks-are-scored-separately", + "query": "invoice approval", + "chunks": [ + { + "id": "long-overlap", + "filePath": "src/ledger.ts", + "startLine": 1, + "content": "invoice approval\ninvoice approval\nreturn approved\nledger entries retain stable boundaries" + }, + { + "id": "short-overlap", + "filePath": "src/ledger.ts", + "startLine": 2, + "content": "invoice approval\nreturn approved" + } + ], + "expectedIds": ["short-overlap"], + "expectedPathPatterns": ["src/ledger.ts"], + "expectedProperties": [ + "the second chunk exactly matches overlapping source lines", + "rescue returns one representative per source file", + "density ranks body evidence per chunk before file deduplication", + "query term counts are not aggregated across chunks" + ] + }, + { + "id": "identical-full-bodies-preserve-original-ordinal", + "query": "invoice approval", + "chunks": [ + { + "id": "z-first", + "filePath": "src/z.ts", + "content": "invoice approval validates the request" + }, + { + "id": "a-second", + "filePath": "src/a.ts", + "content": "invoice approval validates the request" + } + ], + "expectedIds": ["z-first", "a-second"], + "expectedPathPatterns": ["src/z.ts", "src/a.ts"], + "expectedProperties": [ + "identical normalized bodies have equal density", + "lexicographic path or id does not replace ordinal tie-breaking" + ] + }, + { + "id": "body-evidence-beats-metadata-only-density-zero", + "query": "invoice approval", + "chunks": [ + { + "id": "metadata-only", + "filePath": "src/metadata.ts", + "content": " \n ", + "metadata": { + "componentName": "InvoiceApproval" + } + }, + { + "id": "body-present", + "filePath": "src/body.ts", + "content": "invoice approval", + "metadata": { + "componentName": "InvoiceApproval" + } + } + ], + "expectedIds": ["body-present", "metadata-only"], + "expectedPathPatterns": ["src/body.ts", "src/metadata.ts"], + "expectedDensityById": { + "metadata-only": 0 + }, + "expectedProperties": [ + "primary evidence ties", + "empty body uses max(1, body length)", + "metadata does not enter the density numerator or denominator" + ] + }, + { + "id": "empty-metadata-only-bodies-have-finite-zero-density", + "query": "invoice approval", + "chunks": [ + { + "id": "empty-first", + "filePath": "src/empty-a.ts", + "content": "", + "metadata": { + "componentName": "InvoiceApproval" + } + }, + { + "id": "empty-second", + "filePath": "src/empty-b.ts", + "content": " \n ", + "metadata": { + "componentName": "InvoiceApproval" + } + } + ], + "expectedIds": ["empty-first", "empty-second"], + "expectedPathPatterns": ["src/empty-a.ts", "src/empty-b.ts"], + "expectedDensityById": { + "empty-first": 0, + "empty-second": 0 + }, + "expectedProperties": [ + "density is finite and exactly zero", + "equal primary evidence and density preserve ordinal" + ] + }, + { + "id": "equal-primary-tiny-wrapper-wins-density-with-explicit-limitation", + "query": "archive policy", + "chunks": [ + { + "id": "tiny-wrapper", + "filePath": "src/wrapper.ts", + "content": "archive policy" + }, + { + "id": "repeated-long-body", + "filePath": "src/implementation.ts", + "content": "archive archive policy policy filler filler filler filler" + } + ], + "expectedIds": ["tiny-wrapper", "repeated-long-body"], + "expectedPathPatterns": ["src/wrapper.ts", "src/implementation.ts"], + "limitation": "Primary lexical evidence ties, so the fixed formula ranks the short wrapper first (sqrt(1)+sqrt(1) over sqrt(2), versus sqrt(2)+sqrt(2) over sqrt(8)). This order is not evidence that the wrapper is semantically more useful; preserving this counterexample prevents an unsupported semantic claim.", + "expectedProperties": [ + "equal primary ranks invoke density as a separate secondary comparison", + "the higher-density result may be a wrapper", + "density does not establish semantic usefulness" + ] + }, + { + "id": "ambiguous-primary-density-order-is-not-semantic-proof", + "query": "invoice approval", + "chunks": [ + { + "id": "comment-repetition", + "filePath": "src/comment.ts", + "content": "// invoice approval invoice approval invoice approval" + }, + { + "id": "implementation", + "filePath": "src/implementation.ts", + "content": "export function approve(invoice) { return policyAllowsApproval(invoice); }" + } + ], + "expectedIds": ["comment-repetition", "implementation"], + "expectedPathPatterns": ["src/comment.ts", "src/implementation.ts"], + "limitation": "Primary lexical evidence ties. The fixed formula places the repeated comment first; the available evidence cannot establish whether the comment or implementation is semantically more useful. This expected order is algorithmic, not a semantic winner claim.", + "expectedProperties": [ + "ambiguous exact-primary tie is retained explicitly", + "density does not prove action meaning or semantic relevance" + ] + } + ], + "sourceValidationCases": [ + { + "id": "stale-sources-within-nomination-pool-are-skipped", + "query": "invoice approval", + "staleChunkGenerator": { + "count": 30, + "idPattern": "stale-{index}", + "filePathPattern": "src/stale-{index}.ts", + "content": "invoice approval" + }, + "currentChunks": [ + { + "id": "current", + "filePath": "src/current.ts", + "content": "invoice approval validates current source" + } + ], + "mutateFilePathPattern": "src/stale-{index}.ts", + "replacementSource": "export const unrelated = true;", + "expectedIds": ["current"], + "expectedProperties": [ + "31 candidates fit inside the existing nomination pool", + "stale sources are excluded before results are returned", + "density cannot bypass source validation", + "no beyond-window refill is required" + ] + }, + { + "id": "changed-only-source-cannot-supply-density-evidence", + "query": "invoice approval", + "chunks": [ + { + "id": "changed", + "filePath": "src/changed.ts", + "content": "invoice approval validates the indexed source" + } + ], + "mutateFilePath": "src/changed.ts", + "replacementSource": "export const unrelated = true;", + "expectedIds": [], + "expectedProperties": [ + "changed source is invalidated by the existing harness", + "density cannot bypass source integrity checks" + ] + } + ] +} diff --git a/tests/fixtures/search-rescue-nomination.json b/tests/fixtures/search-rescue-nomination.json new file mode 100644 index 0000000..efd922e --- /dev/null +++ b/tests/fixtures/search-rescue-nomination.json @@ -0,0 +1,382 @@ +{ + "version": 1, + "purpose": "Frozen general rescue-only nomination witnesses and negative controls; synthetic unit contract, not a retrieval accuracy benchmark.", + "contract": { + "scope": "Replace rescue fuzzy OR nomination deliberately; keep strict fuzzy retrieval and exact rescue admission unchanged.", + "nomination": "Conservative literal fragments across content, basename, componentName, symbolName, functionName, className and exports; filters before the dynamic cap; stable original chunk ordinal.", + "window": "min(750,max(256,limit*80)); overflow is disclosed through candidateBudgetExhausted. Witness completeness applies inside the nominated window, not beyond it.", + "ranking": "Existing exact evidence rank and score remain; equal exact evidence uses original ordinal instead of fuzzy nomination score. This is an intentional nomination and tie-order change, not parity.", + "normalization": "Existing ASCII lexical splitting and case folding only; no new Unicode transliteration, synonyms or language-specific rules.", + "fallback": "No fuzzy OR rescan in rescue, including overflow/refill; original strict fuzzy search remains the typo and extended-syntax route." + }, + "lexicalCases": [ + { + "id": "plural-policy-to-singular", + "query": "policy archive", + "content": "/* policies archiving */", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "singular-to-plural-query", + "query": "policies archives", + "content": "/* policy archive */", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "plural-library", + "query": "library archive", + "content": "/* libraries archived */", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "silent-e-ing", + "query": "archive policy", + "content": "/* archiving policies */", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "ed-to-ing", + "query": "archived policy", + "content": "/* archiving policy */", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "double-consonant-format", + "query": "formatting invoice", + "content": "/* formatted invoices */", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "double-consonant-stop", + "query": "stopping invoice", + "content": "/* stopped invoices */", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "camel-case", + "query": "invoice archive", + "content": "export const archiveInvoices = true;", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "snake-case", + "query": "credit ledger", + "content": "credit_ledger = True", + "language": "python", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "acronym-boundary", + "query": "http archive", + "content": "type HTTPArchive = {};", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "uppercase", + "query": "POLICY ARCHIVE", + "content": "/* POLICIES ARCHIVING */", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "unicode-surroundings", + "query": "policy archive", + "content": "/* λ archiving 政策 policies café */", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "unicode-lookalikes-not-folded", + "query": "policy archive", + "content": "/* рolicy аrchive */", + "expected": false + }, + { + "id": "diacritic-not-transliterated", + "query": "policy archive", + "content": "/* policies arçhive */", + "expected": false + }, + { + "id": "repeated-inflections-one-clue", + "query": "policy policies", + "content": "/* policy policies */", + "expected": false + }, + { + "id": "source-inflections-one-clue", + "query": "archive policy", + "content": "/* archived archiving archive */", + "expected": false + }, + { + "id": "substring-collision", + "query": "archive policy", + "content": "/* monarchivel policies */", + "expected": false + }, + { + "id": "single-natural-clue", + "query": "archive", + "content": "/* archive policy */", + "expected": false + }, + { + "id": "stop-word-not-corroboration", + "query": "where archive", + "content": "/* where archive */", + "expected": false + }, + { + "id": "comment-clues-are-lexical", + "query": "archive policy", + "content": "// archive policy", + "expected": true, + "expectedScore": 0.5 + }, + { + "id": "tags-do-not-admit", + "query": "archive policy", + "content": "export const value = 0;", + "tags": ["archive", "policy"], + "expected": false + }, + { + "id": "component-name-only", + "query": "archive policy", + "content": "export const value = 0;", + "metadata": { + "componentName": "ArchivePolicy" + }, + "expected": true, + "expectedScore": 0 + }, + { + "id": "symbol-name-only", + "query": "archive policy", + "content": "export const value = 0;", + "metadata": { + "symbolName": "ArchivePolicy" + }, + "expected": true, + "expectedScore": 0 + }, + { + "id": "basename-only", + "query": "archive policy", + "content": "export const value = 0;", + "filePath": "src/archive-policy.ts", + "expected": true, + "expectedScore": 0.5 + } + ], + "representativeCases": [ + { + "id": "function-metadata-representative", + "query": "archive policy", + "chunks": [ + { + "id": "declaration", + "filePath": "src/a.ts", + "content": "export const value = 0;", + "metadata": { + "functionName": "ArchivePolicy" + } + }, + { + "id": "body", + "filePath": "src/a.ts", + "startLine": 3, + "content": "// archive policy" + } + ], + "expectedIds": ["declaration"] + }, + { + "id": "class-metadata-representative", + "query": "archive policy", + "chunks": [ + { + "id": "declaration", + "filePath": "src/a.ts", + "content": "export const value = 0;", + "metadata": { + "className": "ArchivePolicy" + } + }, + { + "id": "body", + "filePath": "src/a.ts", + "startLine": 3, + "content": "// archive policy" + } + ], + "expectedIds": ["declaration"] + }, + { + "id": "export-metadata-representative", + "query": "archive policy", + "chunks": [ + { + "id": "export", + "filePath": "src/a.ts", + "content": "export const value = 0;", + "exports": ["ArchivePolicy"] + }, + { + "id": "body", + "filePath": "src/a.ts", + "startLine": 3, + "content": "// archive policy" + } + ], + "expectedIds": ["export"] + }, + { + "id": "merged-symbol-representative", + "query": "archive policy", + "chunks": [ + { + "id": "declaration", + "filePath": "src/a.ts", + "content": "export const value = 0;", + "metadata": { + "symbolName": "ArchivePolicy+Other" + } + }, + { + "id": "body", + "filePath": "src/a.ts", + "startLine": 3, + "content": "// archive policy" + } + ], + "expectedIds": ["declaration"] + }, + { + "id": "same-file-disjoint-clues", + "query": "archive policy", + "chunks": [ + { + "id": "first", + "filePath": "src/a.ts", + "content": "// archive" + }, + { + "id": "second", + "filePath": "src/a.ts", + "startLine": 3, + "content": "// policy" + } + ], + "expectedIds": ["first"] + }, + { + "id": "equal-evidence-original-ordinal", + "query": "archive policy", + "chunks": [ + { + "id": "z-first", + "filePath": "src/z.ts", + "content": "// archive policy" + }, + { + "id": "a-second", + "filePath": "src/a.ts", + "content": "// archive policy" + } + ], + "expectedIds": ["z-first", "a-second"] + }, + { + "id": "equal-evidence-reversed-ordinal", + "query": "archive policy", + "chunks": [ + { + "id": "a-first", + "filePath": "src/a.ts", + "content": "// archive policy" + }, + { + "id": "z-second", + "filePath": "src/z.ts", + "content": "// archive policy" + } + ], + "expectedIds": ["a-first", "z-second"] + } + ], + "populationCases": [ + { + "id": "below-minimum-window", + "limit": 1, + "count": 255, + "cap": 256, + "exhausted": false + }, + { + "id": "at-minimum-window", + "limit": 1, + "count": 256, + "cap": 256, + "exhausted": false + }, + { + "id": "over-minimum-window", + "limit": 1, + "count": 257, + "cap": 256, + "exhausted": true + }, + { + "id": "intermediate-window", + "limit": 9, + "count": 721, + "cap": 720, + "exhausted": true + }, + { + "id": "at-maximum-window", + "limit": 10, + "count": 750, + "cap": 750, + "exhausted": false + }, + { + "id": "over-maximum-window", + "limit": 10, + "count": 751, + "cap": 750, + "exhausted": true + } + ], + "lifecycleCases": ["changed", "missing", "directory", "invalid-manifest", "outside-root"], + "graphCases": ["current", "no-edge", "changed-companion", "directory-companion"], + "strictCases": [ + { + "id": "typo-fuzzy-retained", + "query": "archve polcy", + "expectsResult": true + }, + { + "id": "extended-or-retained", + "query": "archive | policy", + "expectsResult": true + }, + { + "id": "exact-identifier-retained", + "query": "ArchivePolicy", + "expectsResult": true + } + ] +} diff --git a/tests/fixtures/structural-context-regressions.json b/tests/fixtures/structural-context-regressions.json new file mode 100644 index 0000000..2d90c33 --- /dev/null +++ b/tests/fixtures/structural-context-regressions.json @@ -0,0 +1,176 @@ +{ + "version": 1, + "purpose": "General JavaScript and TypeScript source facts for structural ownership, bounded source witnesses, and source-versus-synthetic-context regressions. This is a human-checkable regression fixture, not a retrieval-ranking benchmark.", + "ownerChainDefinition": "Each sourceSymbols entry names the exact static lexical route in source from its outer named declaration/property through the executable leaf. Static string keys use their decoded key. For an unresolved dynamic computed key, the route may stop at the nearest established owner; scopeOnly marks that route as context and does not claim that owner declares the callback. Dynamic computed keys are omitted from the route and listed in unresolvedNames. bodyMarkers belong to that entry's implementation body; forbiddenSiblingBodyMarkers are source markers owned by another listed sibling.", + "cases": [ + { + "id": "nested-object-static-owner-chain", + "language": "typescript", + "source": "const services = { billing: { reconcile: async (invoice: Invoice) => { return persistInvoice(invoice); } } };", + "sourceSymbols": [{ "ownerChain": ["services", "billing", "reconcile"], "bodyMarkers": ["persistInvoice(invoice)"] }] + }, + { + "id": "same-leaf-name-under-distinct-owners", + "language": "javascript", + "source": "const billing = {\n refresh: () => billingCache.reload()\n};\nconst catalog = {\n refresh: () => catalogCache.reload()\n};", + "sourceSymbols": [ + { "ownerChain": ["billing", "refresh"], "bodyMarkers": ["billingCache.reload()"], "forbiddenSiblingBodyMarkers": ["catalogCache.reload()"] }, + { "ownerChain": ["catalog", "refresh"], "bodyMarkers": ["catalogCache.reload()"], "forbiddenSiblingBodyMarkers": ["billingCache.reload()"] } + ] + }, + { + "id": "sibling-body-isolation", + "language": "typescript", + "source": "const jobs = {\n export: { run: () => writeArchive() },\n import: { run: () => readArchive() }\n};", + "sourceSymbols": [ + { "ownerChain": ["jobs", "export", "run"], "bodyMarkers": ["writeArchive()"], "forbiddenSiblingBodyMarkers": ["readArchive()"] }, + { "ownerChain": ["jobs", "import", "run"], "bodyMarkers": ["readArchive()"], "forbiddenSiblingBodyMarkers": ["writeArchive()"] } + ] + }, + { + "id": "same-line-property-callback-bodies-stay-distinct", + "language": "javascript", + "source": "const operations = { archive: { run: () => writeArchiveFile() }, restore: { run: () => readArchiveFile() } };", + "sourceSymbols": [ + { "ownerChain": ["operations", "archive", "run"], "bodyMarkers": ["writeArchiveFile()"], "forbiddenSiblingBodyMarkers": ["readArchiveFile()"] }, + { "ownerChain": ["operations", "restore", "run"], "bodyMarkers": ["readArchiveFile()"], "forbiddenSiblingBodyMarkers": ["writeArchiveFile()"] } + ] + }, + { + "id": "static-string-computed-property", + "language": "javascript", + "source": "const actions = { [\"sync-now\"]: () => syncRemote() };", + "sourceSymbols": [{ "ownerChain": ["actions", "sync-now"], "bodyMarkers": ["syncRemote()"] }], + "unresolvedNames": [] + }, + { + "id": "dynamic-computed-property-name-remains-unresolved", + "language": "typescript", + "source": "const actions = { [actionName]: () => runAction() };", + "sourceSymbols": [{ "ownerChain": ["actions"], "scopeOnly": true, "bodyMarkers": ["runAction()"] }], + "unresolvedNames": ["actions[actionName]"] + }, + { + "id": "property-arrow-callback-body", + "language": "typescript", + "source": "const listeners = { onMessage: (message: Message) => { recordMessage(message); acknowledge(message.id); } };", + "sourceSymbols": [{ "ownerChain": ["listeners", "onMessage"], "bodyMarkers": ["recordMessage(message)", "acknowledge(message.id)"] }] + }, + { + "id": "class-method-with-property-callback", + "language": "typescript", + "source": "class QueueWorker { start() { const hooks = { onReady: () => openQueue() }; return hooks; } }", + "sourceSymbols": [{ "ownerChain": ["QueueWorker", "start", "hooks", "onReady"], "bodyMarkers": ["openQueue()"] }] + }, + { + "id": "class-method-sibling-isolation", + "language": "javascript", + "source": "class Cache {\n load() { return readDisk(); }\n save() { return writeDisk(); }\n}", + "sourceSymbols": [ + { "ownerChain": ["Cache", "load"], "bodyMarkers": ["readDisk()"], "forbiddenSiblingBodyMarkers": ["writeDisk()"] }, + { "ownerChain": ["Cache", "save"], "bodyMarkers": ["writeDisk()"], "forbiddenSiblingBodyMarkers": ["readDisk()"] } + ] + }, + { + "id": "standalone-function-declaration", + "language": "typescript", + "source": "export function normalizeRecord(record: RecordInput): RecordOutput { return mapRecord(record); }", + "sourceSymbols": [{ "ownerChain": ["normalizeRecord"], "bodyMarkers": ["mapRecord(record)"] }] + }, + { + "id": "standalone-function-expression", + "language": "javascript", + "source": "const calculateTotal = function (items) { return sumPrices(items); };", + "sourceSymbols": [{ "ownerChain": ["calculateTotal"], "bodyMarkers": ["sumPrices(items)"] }] + }, + { + "id": "class-and-function-compatible-source-facts", + "language": "typescript", + "source": "class Ledger {\n close() { flushEntries(); }\n}\nfunction openLedger() { return createLedger(); }", + "sourceSymbols": [ + { "ownerChain": ["Ledger", "close"], "bodyMarkers": ["flushEntries()"] }, + { "ownerChain": ["openLedger"], "bodyMarkers": ["createLedger()"] } + ] + }, + { + "id": "nested-static-property-chain-three-levels", + "language": "javascript", + "source": "const app = { api: { reports: { build: () => assembleReport() } } };", + "sourceSymbols": [{ "ownerChain": ["app", "api", "reports", "build"], "bodyMarkers": ["assembleReport()"] }] + }, + { + "id": "static-computed-key-inside-nested-owner", + "language": "typescript", + "source": "const routes = { admin: { [\"user-list\"]: () => fetchUsers() } };", + "sourceSymbols": [{ "ownerChain": ["routes", "admin", "user-list"], "bodyMarkers": ["fetchUsers()"] }], + "unresolvedNames": [] + }, + { + "id": "nested-container-retains-distinct-bounded-witness-markers", + "language": "javascript", + "source": "const pipeline = {\n ingest: { run: () => { validateInput(); decodePayload(); persistPayload(); } },\n report: { run: () => renderSummary() }\n};", + "sourceSymbols": [ + { "ownerChain": ["pipeline", "ingest", "run"], "bodyMarkers": ["validateInput()", "decodePayload()", "persistPayload()"], "boundedWitnessMarkers": ["validateInput()", "persistPayload()"], "forbiddenSiblingBodyMarkers": ["renderSummary()"] }, + { "ownerChain": ["pipeline", "report", "run"], "bodyMarkers": ["renderSummary()"], "boundedWitnessMarkers": ["renderSummary()"], "forbiddenSiblingBodyMarkers": ["validateInput()", "decodePayload()", "persistPayload()"] } + ] + }, + { + "id": "method-body-has-interior-witness-and-sibling-control", + "language": "typescript", + "source": "class Importer {\n importRows(rows: Row[]) {\n beginImport();\n const normalized = normalizeRows(rows);\n const checked = checkRows(normalized);\n const saved = saveRows(checked);\n finishImport(saved);\n }\n exportRows() { return writeRows(); }\n}", + "sourceSymbols": [ + { "ownerChain": ["Importer", "importRows"], "bodyMarkers": ["beginImport()", "normalizeRows(rows)", "checkRows(normalized)", "saveRows(checked)", "finishImport(saved)"], "boundedWitnessMarkers": ["beginImport()", "checkRows(normalized)", "finishImport(saved)"], "forbiddenSiblingBodyMarkers": ["writeRows()"] }, + { "ownerChain": ["Importer", "exportRows"], "bodyMarkers": ["writeRows()"], "boundedWitnessMarkers": ["writeRows()"], "forbiddenSiblingBodyMarkers": ["beginImport()", "normalizeRows(rows)", "checkRows(normalized)", "saveRows(checked)", "finishImport(saved)"] } + ] + }, + { + "id": "same-method-name-in-separate-classes", + "language": "typescript", + "source": "class LocalStore {\n clear() { clearLocalFiles(); }\n}\nclass RemoteStore {\n clear() { clearRemoteFiles(); }\n}", + "sourceSymbols": [ + { "ownerChain": ["LocalStore", "clear"], "bodyMarkers": ["clearLocalFiles()"], "forbiddenSiblingBodyMarkers": ["clearRemoteFiles()"] }, + { "ownerChain": ["RemoteStore", "clear"], "bodyMarkers": ["clearRemoteFiles()"], "forbiddenSiblingBodyMarkers": ["clearLocalFiles()"] } + ] + }, + { + "id": "type-declaration-is-not-function-body", + "language": "typescript", + "source": "type RowMapper = (row: Row) => MappedRow;\nfunction mapRow(row: Row): MappedRow { return convertRow(row); }", + "sourceSymbols": [{ "ownerChain": ["mapRow"], "bodyMarkers": ["convertRow(row)"] }], + "supportingOnlyMarkers": ["type RowMapper = (row: Row) => MappedRow"] + }, + { + "id": "test-companion-midchunk-line-is-supporting-context", + "language": "typescript", + "source": "describe(\"invoice parser\", () => {\n it(\"keeps the account id\", () => {\n const parsed = parseInvoice(fixture);\n expect(parsed.accountId).toBe(\"acct-7\");\n });\n});", + "sourceSymbols": [], + "supportingOnlyMarkers": ["expect(parsed.accountId).toBe(\"acct-7\")"] + }, + { + "id": "declaration-companion-midchunk-line-is-supporting-context", + "language": "typescript", + "source": "export interface InvoiceParser {\n parse(source: string): Invoice;\n}\nexport function parseInvoice(source: string): Invoice { return decodeInvoice(source); }", + "sourceSymbols": [{ "ownerChain": ["parseInvoice"], "bodyMarkers": ["decodeInvoice(source)"] }], + "supportingOnlyMarkers": ["parse(source: string): Invoice"] + }, + { + "id": "scope-label-text-is-not-source-body-evidence", + "language": "javascript", + "source": "const workers = {\n email: { send: () => deliverEmail() },\n sms: { send: () => deliverText() }\n};", + "sourceSymbols": [ + { "ownerChain": ["workers", "email", "send"], "bodyMarkers": ["deliverEmail()"], "forbiddenSiblingBodyMarkers": ["deliverText()"] }, + { "ownerChain": ["workers", "sms", "send"], "bodyMarkers": ["deliverText()"], "forbiddenSiblingBodyMarkers": ["deliverEmail()"] } + ], + "syntheticLabelsAbsentFromSource": ["workers > email > send", "workers > sms > send"] + }, + { + "id": "overlapping-name-words-do-not-cross-attribute-sibling-body", + "language": "typescript", + "source": "const tasks = {\n syncUsers: { run: () => uploadUsers() },\n syncTeams: { run: () => uploadTeams() }\n};", + "sourceSymbols": [ + { "ownerChain": ["tasks", "syncUsers", "run"], "bodyMarkers": ["uploadUsers()"], "forbiddenSiblingBodyMarkers": ["uploadTeams()"] }, + { "ownerChain": ["tasks", "syncTeams", "run"], "bodyMarkers": ["uploadTeams()"], "forbiddenSiblingBodyMarkers": ["uploadUsers()"] } + ] + } + ] +} diff --git a/tests/get-symbol-references.test.ts b/tests/get-symbol-references.test.ts index 6481b46..0cbd99b 100644 --- a/tests/get-symbol-references.test.ts +++ b/tests/get-symbol-references.test.ts @@ -7,7 +7,8 @@ import { INDEX_FORMAT_VERSION, INDEX_META_FILENAME, INDEX_META_VERSION, - KEYWORD_INDEX_FILENAME + KEYWORD_INDEX_FILENAME, + MANIFEST_FILENAME } from '../src/constants/codebase-context.js'; type ToolCallRequest = { @@ -22,7 +23,9 @@ type ToolCallResponse = { isError?: boolean; }; -function getToolCallHandler(server: unknown): (request: ToolCallRequest) => Promise { +function getToolCallHandler( + server: unknown +): (request: ToolCallRequest) => Promise { const handlers = (server as { _requestHandlers?: unknown })._requestHandlers; if (!(handlers instanceof Map)) { throw new Error('Expected server._requestHandlers to be a Map'); @@ -34,6 +37,41 @@ function getToolCallHandler(server: unknown): (request: ToolCallRequest) => Prom return handler as (request: ToolCallRequest) => Promise; } +async function writeProjectSource( + rootPath: string, + relativePath: string, + content: string +): Promise { + const absolutePath = path.join(rootPath, relativePath); + await fs.mkdir(path.dirname(absolutePath), { recursive: true }); + await fs.writeFile(absolutePath, content, 'utf-8'); + return absolutePath; +} + +async function writeReferenceManifest( + rootPath: string, + buildId: string, + sourcePaths: string[] +): Promise { + const contextDir = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + const metadataPath = path.join(contextDir, INDEX_META_FILENAME); + const meta = JSON.parse(await fs.readFile(metadataPath, 'utf8')) as { + artifacts: Record; + }; + meta.artifacts.manifest = { path: MANIFEST_FILENAME }; + await fs.writeFile(metadataPath, JSON.stringify(meta, null, 2), 'utf8'); + await fs.writeFile( + path.join(contextDir, MANIFEST_FILENAME), + JSON.stringify({ + version: 1, + generatedAt: new Date().toISOString(), + header: { buildId, formatVersion: INDEX_FORMAT_VERSION }, + files: Object.fromEntries(sourcePaths.map((sourcePath) => [sourcePath, 'a'.repeat(16)])) + }), + 'utf8' + ); +} + describe('get_symbol_references MCP tool', () => { let tempRoot: string | null = null; let originalArgv: string[] | null = null; @@ -72,6 +110,36 @@ describe('get_symbol_references MCP tool', () => { expect(TOOLS.some((tool) => tool.name === 'get_symbol_references')).toBe(true); }); + it('marks a manifest-listed source integrity failure as an MCP error', async () => { + if (!tempRoot) throw new Error('tempRoot not initialized'); + + const { server } = await import('../src/index.js'); + const { CodebaseIndexer } = await import('../src/core/indexer.js'); + const sourcePath = await writeProjectSource( + tempRoot, + 'src/a.ts', + 'export const alpha = 1;\nconsole.log(alpha);\n' + ); + await new CodebaseIndexer({ rootPath: tempRoot }).index(); + await fs.unlink(sourcePath); + await fs.mkdir(sourcePath); + + const response = await getToolCallHandler(server)({ + jsonrpc: '2.0', + id: 1, + method: 'tools/call', + params: { name: 'get_symbol_references', arguments: { symbol: 'alpha' } } + }); + + expect(response.isError).toBe(true); + const payload = JSON.parse(response.content[0].text) as { + status: string; + message: string; + }; + expect(payload.status).toBe('error'); + expect(payload.message).toContain('src/a.ts'); + }); + it('returns usageCount and top usages from keyword index', async () => { if (!tempRoot) { throw new Error('tempRoot not initialized'); @@ -90,16 +158,22 @@ describe('get_symbol_references MCP tool', () => { 'utf-8' ); + const sourceA = 'export function alpha() {\n return beta(alpha);\n}\n'; + const sourceB = 'const beta = alpha + 1;\n'; + const filePathA = await writeProjectSource(tempRoot, 'src/a.ts', sourceA); + const filePathB = await writeProjectSource(tempRoot, 'src/b.ts', sourceB); const chunks = [ { - content: 'export function alpha() {\n return beta(alpha);\n}', + content: sourceA, startLine: 10, - relativePath: 'src/a.ts' + relativePath: 'src/a.ts', + filePath: filePathA }, { - content: 'const beta = alpha + 1;\n', + content: sourceB, startLine: 2, - relativePath: 'src/b.ts' + relativePath: 'src/b.ts', + filePath: filePathB } ]; @@ -128,6 +202,7 @@ describe('get_symbol_references MCP tool', () => { ), 'utf-8' ); + await writeReferenceManifest(tempRoot, buildId, ['src/a.ts', 'src/b.ts']); const { server } = await import('../src/index.js'); const handler = getToolCallHandler(server); @@ -151,6 +226,11 @@ describe('get_symbol_references MCP tool', () => { expect(payload.usages.length).toBeLessThanOrEqual(2); expect(payload.confidence).toBe('syntactic'); expect(typeof payload.isComplete).toBe('boolean'); + expect(payload.coverage).toMatchObject({ + candidateFiles: 2, + sourcesRead: 2, + filesUnavailable: 0 + }); for (const usage of payload.usages) { expect(usage.file).toBeTypeOf('string'); @@ -160,6 +240,60 @@ describe('get_symbol_references MCP tool', () => { } }); + it('returns a structured full-reindex error when the reference manifest is missing', async () => { + if (!tempRoot) { + throw new Error('tempRoot not initialized'); + } + + const contextDir = path.join(tempRoot, CODEBASE_CONTEXT_DIRNAME); + await fs.mkdir(contextDir, { recursive: true }); + const buildId = 'test-build-symbol-refs-missing-manifest'; + await fs.mkdir(path.join(contextDir, 'index'), { recursive: true }); + await fs.writeFile( + path.join(contextDir, 'index', 'index-build.json'), + JSON.stringify({ buildId, formatVersion: INDEX_FORMAT_VERSION }), + 'utf8' + ); + await fs.writeFile( + path.join(contextDir, KEYWORD_INDEX_FILENAME), + JSON.stringify({ + header: { buildId, formatVersion: INDEX_FORMAT_VERSION }, + chunks: [] + }), + 'utf8' + ); + await fs.writeFile( + path.join(contextDir, INDEX_META_FILENAME), + JSON.stringify({ + metaVersion: INDEX_META_VERSION, + formatVersion: INDEX_FORMAT_VERSION, + buildId, + generatedAt: new Date().toISOString(), + toolVersion: 'test', + artifacts: { + keywordIndex: { path: KEYWORD_INDEX_FILENAME }, + vectorDb: { path: 'index', provider: 'test' }, + manifest: { path: MANIFEST_FILENAME } + } + }), + 'utf8' + ); + + const { server } = await import('../src/index.js'); + const handler = getToolCallHandler(server); + const response = await handler({ + jsonrpc: '2.0', + id: 2, + method: 'tools/call', + params: { name: 'get_symbol_references', arguments: { symbol: 'alpha' } } + }); + + expect(response.isError).toBe(true); + const payload = JSON.parse(response.content[0].text) as { status: string; message: string }; + expect(payload.status).toBe('error'); + expect(payload.message).toMatch(/full reindex.*codebase-context reindex without --incremental/i); + }); + it('excludes comment and string matches when file exists and Tree-sitter is supported', async () => { if (!tempRoot) { throw new Error('tempRoot not initialized'); @@ -186,7 +320,7 @@ describe('get_symbol_references MCP tool', () => { 'const alpha = 1;', '// alpha only in comment', 'console.log(alpha);', - 'const s = \"alpha\";' + 'const s = "alpha";' ].join('\n'); await fs.writeFile(filePath, fileContent, 'utf-8'); @@ -224,6 +358,7 @@ describe('get_symbol_references MCP tool', () => { ), 'utf-8' ); + await writeReferenceManifest(tempRoot, buildId, ['src/example.ts']); const { server } = await import('../src/index.js'); const handler = getToolCallHandler(server); @@ -272,11 +407,14 @@ describe('get_symbol_references MCP tool', () => { 'utf-8' ); + const source = 'export function alpha() {\n return alpha;\n}\n'; + const filePath = await writeProjectSource(tempRoot, 'src/test.ts', source); const chunks = [ { - content: 'export function alpha() {\n return alpha;\n}', + content: source, startLine: 1, - relativePath: 'src/test.ts' + relativePath: 'src/test.ts', + filePath } ]; @@ -305,6 +443,7 @@ describe('get_symbol_references MCP tool', () => { ), 'utf-8' ); + await writeReferenceManifest(tempRoot, buildId, ['src/test.ts']); const { server } = await import('../src/index.js'); const handler = getToolCallHandler(server); @@ -351,11 +490,13 @@ describe('get_symbol_references MCP tool', () => { // Create a chunk with many matches const content = 'foo foo foo foo foo foo foo foo foo foo foo'; + const filePath = await writeProjectSource(tempRoot, 'src/test.ts', content); const chunks = [ { content, startLine: 1, - relativePath: 'src/test.ts' + relativePath: 'src/test.ts', + filePath } ]; @@ -384,6 +525,7 @@ describe('get_symbol_references MCP tool', () => { ), 'utf-8' ); + await writeReferenceManifest(tempRoot, buildId, ['src/test.ts']); const { server } = await import('../src/index.js'); const handler = getToolCallHandler(server); diff --git a/tests/helpers/source-snapshot.ts b/tests/helpers/source-snapshot.ts new file mode 100644 index 0000000..a67293b --- /dev/null +++ b/tests/helpers/source-snapshot.ts @@ -0,0 +1,104 @@ +import { promises as fs } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import type { CodebaseSearcher } from '../../src/core/search.js'; +import { + hashFileContent, + readManifest, + writeManifest, + type FileManifest +} from '../../src/core/manifest.js'; +import { + CODEBASE_CONTEXT_DIRNAME, + MANIFEST_FILENAME +} from '../../src/constants/codebase-context.js'; +import type { CodeChunk } from '../../src/types/index.js'; + +export interface SourceSnapshot { + rootPath: string; + manifest: FileManifest; + cleanup: () => Promise; +} + +export interface SearcherTestHarness { + initialized: boolean; + sourceManifest: FileManifest | null; + embeddingProvider: unknown; + storageProvider: unknown; + fuseIndex: unknown; + patternIntelligence: unknown; + semanticSearch: unknown; + keywordSearch: unknown; + collectHybridMatches: unknown; + search: CodebaseSearcher['search']; +} + +export function asSearcherTestHarness(searcher: CodebaseSearcher): SearcherTestHarness { + return searcher as unknown as SearcherTestHarness; +} + +function toRelativeSourcePath(filePath: string): string { + const normalized = filePath.replace(/\\/g, '/'); + const relativePath = normalized.replace(/^[A-Za-z]:\/repo\//i, '').replace(/^\/repo\//, ''); + if (path.isAbsolute(relativePath) || relativePath.split('/').includes('..')) { + throw new Error(`Fixture source path must be relative to the synthetic repo: ${filePath}`); + } + return relativePath; +} + +/** Materialize indexed chunks as a confined source tree plus the standard file manifest. */ +export async function materializeSourceSnapshot( + chunks: CodeChunk[], + prefix = 'search-source-fixture-' +): Promise { + const rootPath = await fs.mkdtemp(path.join(os.tmpdir(), prefix)); + try { + const byRelativePath = new Map(); + for (const chunk of chunks) { + const relativePath = toRelativeSourcePath(chunk.filePath); + const sameFile = byRelativePath.get(relativePath) ?? []; + sameFile.push(chunk); + byRelativePath.set(relativePath, sameFile); + } + + const files: Record = {}; + for (const [relativePath, fileChunks] of byRelativePath) { + const finalLine = Math.max(...fileChunks.map((chunk) => chunk.endLine)); + const lines = Array(Math.max(finalLine, 1)).fill('') as string[]; + for (const chunk of fileChunks) { + const contentLines = chunk.content.split('\n'); + contentLines.forEach((line, offset) => { + const lineIndex = chunk.startLine - 1 + offset; + if (lineIndex < lines.length) lines[lineIndex] = line; + }); + } + + const content = `${lines.join('\n')}\n`; + const sourcePath = path.join(rootPath, ...relativePath.split('/')); + await fs.mkdir(path.dirname(sourcePath), { recursive: true }); + await fs.writeFile(sourcePath, content); + files[relativePath] = hashFileContent(content); + for (const chunk of fileChunks) { + chunk.filePath = sourcePath; + chunk.relativePath = relativePath; + } + } + + const manifestPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME); + await writeManifest(manifestPath, { + version: 1, + generatedAt: new Date().toISOString(), + files + }); + const manifest = await readManifest(manifestPath); + if (!manifest) throw new Error('Could not read generated source fixture manifest'); + return { + rootPath, + manifest, + cleanup: () => fs.rm(rootPath, { recursive: true, force: true }) + }; + } catch (error) { + await fs.rm(rootPath, { recursive: true, force: true }); + throw error; + } +} diff --git a/tests/index-versioning-migration.test.ts b/tests/index-versioning-migration.test.ts index a541f17..57e03d5 100644 --- a/tests/index-versioning-migration.test.ts +++ b/tests/index-versioning-migration.test.ts @@ -27,6 +27,12 @@ vi.mock('../src/core/search.js', async () => { async search(query: string, limit: number, filters?: unknown, options?: unknown) { return searchMocks.search(query, limit, filters, options); } + async searchWithCoverage(query: string, limit: number, filters?: unknown, options?: unknown) { + return { + results: await this.search(query, limit, filters, options), + sourceCoverage: { status: 'current', checkedFiles: 0, skippedFiles: 0 } + }; + } } return { CodebaseSearcher }; @@ -207,7 +213,7 @@ describe('index versioning migration (MIGR-01)', () => { afterEach(async () => { const { clearProjects } = await import('../src/project-state.js'); - clearProjects(); + await clearProjects(); if (originalArgv) process.argv = originalArgv; if (originalEnvRoot === undefined) delete process.env.CODEBASE_ROOT; @@ -425,7 +431,7 @@ describe('index-consuming allowlist enforcement', () => { afterEach(async () => { const { clearProjects } = await import('../src/project-state.js'); - clearProjects(); + await clearProjects(); if (originalArgv) process.argv = originalArgv; if (originalEnvRoot === undefined) delete process.env.CODEBASE_ROOT; @@ -481,7 +487,8 @@ describe('index-consuming allowlist enforcement', () => { } })); - vi.doMock('../src/core/index-meta.js', () => ({ + vi.doMock('../src/core/index-meta.js', async (importOriginal) => ({ + ...(await importOriginal()), readIndexMeta, validateIndexArtifacts })); diff --git a/tests/indexer-model-mode.test.ts b/tests/indexer-model-mode.test.ts new file mode 100644 index 0000000..f6c0f54 --- /dev/null +++ b/tests/indexer-model-mode.test.ts @@ -0,0 +1,469 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { promises as fs } from 'node:fs'; +import { writeFileSync } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import structuralFixture from './fixtures/structural-context-regressions.json'; +import type { CodeChunk } from '../src/types/index.js'; +import { + resolveSkipEmbedding, + checkEmbeddingMismatch, + type IndexMeta +} from '../src/core/index-meta.js'; +import { + CODEBASE_CONTEXT_DIRNAME, + INDEX_FORMAT_VERSION, + INDEX_META_FILENAME, + INDEX_META_VERSION, + HEALTH_FILENAME, + INTELLIGENCE_FILENAME, + KEYWORD_INDEX_FILENAME, + MANIFEST_FILENAME, + RELATIONSHIPS_FILENAME, + VECTOR_DB_DIRNAME +} from '../src/constants/codebase-context.js'; + +const { embeddingInit, storageInit, embeddingRuntimeAssert, storageRuntimeAssert, rerankInit } = + vi.hoisted(() => ({ + embeddingInit: vi.fn(), + storageInit: vi.fn(), + embeddingRuntimeAssert: vi.fn(), + storageRuntimeAssert: vi.fn(), + rerankInit: vi.fn(async (_query: string, results: unknown) => results) + })); +vi.mock('../src/embeddings/index.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + assertEmbeddingRuntimeAvailable: embeddingRuntimeAssert, + getEmbeddingProvider: embeddingInit + }; +}); +vi.mock('../src/storage/index.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + assertStorageRuntimeAvailable: storageRuntimeAssert, + getStorageProvider: storageInit + }; +}); +vi.mock('../src/core/reranker.js', () => ({ + rerank: rerankInit, + getRerankerStatus: vi.fn(() => 'fallback') +})); + +import { CodebaseSearcher } from '../src/core/search.js'; +import { CodebaseIndexer } from '../src/core/indexer.js'; + +describe('persisted embedding mode', () => { + let rootPath: string | undefined; + + async function retainLegacyFormat(legacyRoot: string, formatVersion: number): Promise { + const contextDir = path.join(legacyRoot, CODEBASE_CONTEXT_DIRNAME); + const metaPath = path.join(contextDir, INDEX_META_FILENAME); + const meta = JSON.parse(await fs.readFile(metaPath, 'utf8')) as IndexMeta; + meta.formatVersion = formatVersion; + await fs.writeFile(metaPath, JSON.stringify(meta)); + for (const filename of [ + KEYWORD_INDEX_FILENAME, + INTELLIGENCE_FILENAME, + RELATIONSHIPS_FILENAME, + HEALTH_FILENAME, + MANIFEST_FILENAME + ]) { + const artifactPath = path.join(contextDir, filename); + const artifact = JSON.parse(await fs.readFile(artifactPath, 'utf8')) as { + header: { formatVersion: number }; + }; + artifact.header.formatVersion = formatVersion; + await fs.writeFile(artifactPath, JSON.stringify(artifact)); + } + await fs.writeFile( + path.join(contextDir, VECTOR_DB_DIRNAME, 'index-build.json'), + JSON.stringify({ buildId: meta.buildId, formatVersion }) + ); + return meta; + } + + afterEach(async () => { + if (rootPath) await fs.rm(rootPath, { recursive: true, force: true }); + rootPath = undefined; + }); + + it('defaults a genuinely fresh folder to model-free and honors explicit semantic opt-in', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-mode-fresh-')); + await expect(resolveSkipEmbedding(rootPath)).resolves.toBe(true); + await expect(resolveSkipEmbedding(rootPath, false)).resolves.toBe(false); + }); + + it.each([1, 2, 3])( + 'preserves a format-%i recorded semantic or model-free mode when no choice is supplied', + async (formatVersion) => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-mode-existing-')); + await fs.mkdir(path.join(rootPath, '.codebase-context'), { recursive: true }); + const writeMeta = async (embeddingEnabled: boolean): Promise => { + const meta: IndexMeta = { + metaVersion: 1, + formatVersion, + buildId: 'build-1', + generatedAt: new Date().toISOString(), + toolVersion: '1.0.0', + artifacts: { + keywordIndex: { path: 'index.json' }, + vectorDb: { + path: 'vector-db', + provider: 'lancedb', + embeddingProvider: 'transformers', + embeddingModel: 'Xenova/bge-small-en-v1.5', + embeddingEnabled + } + } + }; + await fs.writeFile( + path.join(rootPath!, '.codebase-context', 'index-meta.json'), + JSON.stringify(meta) + ); + }; + + await writeMeta(false); + await expect(resolveSkipEmbedding(rootPath)).resolves.toBe(true); + await expect(resolveSkipEmbedding(rootPath, false)).resolves.toBe(false); + await writeMeta(true); + await expect(resolveSkipEmbedding(rootPath)).resolves.toBe(false); + } + ); + + it('fails closed when index artifacts exist but mode metadata cannot be read', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-mode-corrupt-')); + const contextDir = path.join(rootPath, '.codebase-context'); + await fs.mkdir(contextDir); + await fs.writeFile(path.join(contextDir, 'index.json'), '{}'); + await expect(resolveSkipEmbedding(rootPath)).rejects.toThrow(/Cannot recover embedding mode/); + await expect(resolveSkipEmbedding(rootPath, true)).resolves.toBe(true); + }); + + it('fails closed on unknown state under the index directory but accepts memory-only state', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-mode-unknown-')); + const contextDir = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + await fs.mkdir(contextDir); + await fs.writeFile(path.join(contextDir, 'unrecognized-state.json'), '{}'); + await expect(resolveSkipEmbedding(rootPath)).rejects.toThrow(/Cannot recover embedding mode/); + + await fs.rm(contextDir, { recursive: true, force: true }); + await fs.mkdir(contextDir); + await fs.writeFile(path.join(contextDir, 'memory.json'), '{}'); + await expect(resolveSkipEmbedding(rootPath)).resolves.toBe(true); + }); + + it('forces rebuilds when enabled state changes, while mode-free provider changes do not matter', () => { + const meta = { + metaVersion: 1, + formatVersion: 1, + buildId: 'build-1', + generatedAt: new Date().toISOString(), + toolVersion: '1.0.0', + artifacts: { + keywordIndex: { path: 'index.json' }, + vectorDb: { + path: 'vector-db', + provider: 'lancedb', + embeddingEnabled: false + } + } + } satisfies IndexMeta; + expect(checkEmbeddingMismatch(meta, 'openai', 'some-model', false)).toBe(false); + expect(checkEmbeddingMismatch(meta, 'transformers', 'some-model', true)).toBe(true); + }); + + it('initializes keyword search without embedding or vector storage in model-free mode', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-search-model-free-')); + const contextDir = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + const vectorDir = path.join(contextDir, VECTOR_DB_DIRNAME); + const buildId = 'model-free-build'; + await fs.mkdir(vectorDir, { recursive: true }); + await fs.writeFile( + path.join(contextDir, KEYWORD_INDEX_FILENAME), + JSON.stringify({ header: { buildId, formatVersion: INDEX_FORMAT_VERSION }, chunks: [] }) + ); + await fs.writeFile( + path.join(contextDir, INTELLIGENCE_FILENAME), + JSON.stringify({ header: { buildId, formatVersion: INDEX_FORMAT_VERSION }, patterns: {} }) + ); + await fs.writeFile( + path.join(contextDir, MANIFEST_FILENAME), + JSON.stringify({ version: 1, generatedAt: new Date().toISOString(), files: {} }) + ); + await fs.writeFile( + path.join(vectorDir, 'index-build.json'), + JSON.stringify({ buildId, formatVersion: INDEX_FORMAT_VERSION }) + ); + await fs.writeFile( + path.join(contextDir, INDEX_META_FILENAME), + JSON.stringify({ + metaVersion: INDEX_META_VERSION, + formatVersion: INDEX_FORMAT_VERSION, + buildId, + generatedAt: new Date().toISOString(), + toolVersion: '1.0.0', + artifacts: { + keywordIndex: { path: KEYWORD_INDEX_FILENAME }, + vectorDb: { + path: VECTOR_DB_DIRNAME, + provider: 'lancedb', + embeddingEnabled: false + } + } + }) + ); + + const searcher = new CodebaseSearcher(rootPath); + await searcher.initialize(); + await searcher.search('source', 5, undefined, { + useSemanticSearch: true, + enableReranker: true + }); + expect(embeddingInit).not.toHaveBeenCalled(); + expect(storageInit).not.toHaveBeenCalled(); + expect(rerankInit).not.toHaveBeenCalled(); + }); + + it('indexes real source by default and preserves the model-free mode on rebuild', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-index-model-free-')); + const sourcePath = path.join(rootPath, 'source.ts'); + await fs.writeFile(sourcePath, 'export const sourcePreserved = "original source";\n'); + + const first = new CodebaseIndexer({ rootPath }); + const firstStats = await first.index(); + expect(firstStats.errors).toEqual([]); + + const contextDir = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + const firstMeta = JSON.parse( + await fs.readFile(path.join(contextDir, INDEX_META_FILENAME), 'utf8') + ) as { artifacts: { vectorDb: { embeddingEnabled: boolean } } }; + expect(firstMeta.artifacts.vectorDb.embeddingEnabled).toBe(false); + expect(embeddingRuntimeAssert).not.toHaveBeenCalled(); + expect(storageRuntimeAssert).not.toHaveBeenCalled(); + expect(embeddingInit).not.toHaveBeenCalled(); + const indexPath = path.join(contextDir, KEYWORD_INDEX_FILENAME); + const initialIndex = await fs.readFile(indexPath, 'utf8'); + expect(initialIndex).toContain('original source'); + + await fs.writeFile(sourcePath, 'export const sourcePreserved = "updated source";\n'); + const second = new CodebaseIndexer({ rootPath }); + const secondStats = await second.index(); + expect(secondStats.errors).toEqual([]); + const secondMeta = JSON.parse( + await fs.readFile(path.join(contextDir, INDEX_META_FILENAME), 'utf8') + ) as { artifacts: { vectorDb: { embeddingEnabled: boolean } } }; + expect(secondMeta.artifacts.vectorDb.embeddingEnabled).toBe(false); + const rebuiltIndex = await fs.readFile(indexPath, 'utf8'); + expect(rebuiltIndex).toContain('updated source'); + expect(rebuiltIndex).not.toContain('original source'); + expect(embeddingInit).not.toHaveBeenCalled(); + }, 60000); + + it('persists exact raw AST bodies, keeps same-line siblings separate, and preserves prefix-like source comments', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-raw-ast-')); + const fixture = structuralFixture.cases.find( + (item) => item.id === 'same-line-property-callback-bodies-stay-distinct' + ); + expect(fixture).toBeDefined(); + await fs.writeFile(path.join(rootPath, 'handlers.js'), fixture!.source); + const commented = 'function consume() {\n// consume :: ()\n consumeMessage();\n}\n'; + await fs.writeFile(path.join(rootPath, 'consume.js'), commented); + await new CodebaseIndexer({ rootPath }).index(); + const persisted = JSON.parse( + await fs.readFile( + path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME), + 'utf8' + ) + ) as { chunks: CodeChunk[] }; + for (const fact of fixture!.sourceSymbols) { + const own = persisted.chunks.filter( + (chunk) => JSON.stringify(chunk.metadata.symbolPath) === JSON.stringify(fact.ownerChain) + ); + expect(own.length).toBeGreaterThan(0); + for (const marker of fact.bodyMarkers) + expect(own.some((chunk) => chunk.content.includes(marker))).toBe(true); + for (const chunk of own) { + expect(fixture!.source).toContain(chunk.content); + if ('forbiddenSiblingBodyMarkers' in fact) { + for (const marker of fact.forbiddenSiblingBodyMarkers ?? []) + expect(chunk.content).not.toContain(marker); + } + } + } + const consume = persisted.chunks.find((chunk) => chunk.metadata.symbolName === 'consume'); + expect(consume).toBeDefined(); + expect(consume!.content).toBe(commented.trimEnd()); + expect(consume!.content.match(/\/\/ consume :: \(\)/g)).toHaveLength(1); + }, 60000); + + it('persists concise descriptor implementations under their nearest static owner', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-concise-descriptor-')); + const source = + 'function createHandlers() { return { shutdown: { value() { closeConnections(); } }, state: { get() { return readState(); }, set: next => writeState(next) }, sibling: { value() { reloadConnections(); } } }; }'; + await fs.writeFile(path.join(rootPath, 'handlers.js'), source); + await new CodebaseIndexer({ rootPath }).index(); + const persisted = JSON.parse( + await fs.readFile( + path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME), + 'utf8' + ) + ) as { chunks: CodeChunk[] }; + for (const fact of [ + { + name: 'shutdown', + body: 'shutdown: { value() { closeConnections(); } }', + forbidden: ['readState()', 'reloadConnections()'] + }, + { + name: 'state', + body: 'state: { get() { return readState(); }, set: next => writeState(next) }', + forbidden: ['closeConnections()', 'reloadConnections()'] + } + ]) { + const own = persisted.chunks.filter((chunk) => chunk.metadata.symbolName === fact.name); + expect(own).toHaveLength(1); + expect(own[0].content).toBe(fact.body); + expect(own[0].metadata.symbolPath).toEqual(['createHandlers', fact.name]); + for (const marker of fact.forbidden) expect(own[0].content).not.toContain(marker); + } + }, 60000); + + it.each([1, 2, 3])( + 'rebuilds an otherwise consistent format-%i index on an unchanged incremental pass', + async (formatVersion) => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-format-recovery-')); + await fs.writeFile( + path.join(rootPath, 'source.ts'), + 'export function handle() { return dispatch(); }\n' + ); + await new CodebaseIndexer({ rootPath }).index(); + const contextDir = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + const metaPath = path.join(contextDir, INDEX_META_FILENAME); + const oldMeta = await retainLegacyFormat(rootPath, formatVersion); + await expect(resolveSkipEmbedding(rootPath)).resolves.toBe(true); + const stats = await new CodebaseIndexer({ rootPath, incrementalOnly: true }).index(); + expect(stats.errors).toEqual([]); + const rebuilt = JSON.parse(await fs.readFile(metaPath, 'utf8')) as IndexMeta; + expect(rebuilt.formatVersion).toBe(4); + expect(rebuilt.buildId).not.toBe(oldMeta.buildId); + expect(rebuilt.artifacts.vectorDb.embeddingEnabled).toBe(false); + }, + 60000 + ); + + it.each([1, 2, 3])( + 'preserves a format-%i successful index when semantic migration opt-in cannot initialize', + async (formatVersion) => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-format-atomic-')); + await fs.writeFile( + path.join(rootPath, 'source.ts'), + 'export function handle() { return dispatch(); }\n' + ); + await new CodebaseIndexer({ rootPath }).index(); + const contextDir = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + const metaPath = path.join(contextDir, INDEX_META_FILENAME); + await retainLegacyFormat(rootPath, formatVersion); + const originalMeta = await fs.readFile(metaPath, 'utf8'); + const originalIndex = await fs.readFile( + path.join(contextDir, KEYWORD_INDEX_FILENAME), + 'utf8' + ); + await expect(resolveSkipEmbedding(rootPath)).resolves.toBe(true); + const unavailable = new Error('migration semantic runtime unavailable'); + embeddingRuntimeAssert.mockImplementationOnce(() => { + throw unavailable; + }); + await expect( + new CodebaseIndexer({ + rootPath, + incrementalOnly: true, + config: { skipEmbedding: false } + }).index() + ).rejects.toBe(unavailable); + expect(await fs.readFile(metaPath, 'utf8')).toBe(originalMeta); + expect(await fs.readFile(path.join(contextDir, KEYWORD_INDEX_FILENAME), 'utf8')).toBe( + originalIndex + ); + }, + 60000 + ); + + it('checks semantic dependencies before creating or replacing index artifacts', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-semantic-runtime-')); + await fs.writeFile(path.join(rootPath, 'source.ts'), 'export const source = 1;\n'); + const missingRuntime = new Error('semantic dependency unavailable'); + embeddingRuntimeAssert.mockImplementationOnce(() => { + throw missingRuntime; + }); + + const indexer = new CodebaseIndexer({ rootPath, config: { skipEmbedding: false } }); + await expect(indexer.index()).rejects.toBe(missingRuntime); + await expect(fs.access(path.join(rootPath, CODEBASE_CONTEXT_DIRNAME))).rejects.toMatchObject({ + code: 'ENOENT' + }); + expect(storageRuntimeAssert).not.toHaveBeenCalled(); + }); + + it('binds the manifest digest to the exact bytes analyzed, even if the file changes during analysis', async () => { + // Frozen S2 regression: analyzer receives snapshot A, source changes to B before publish. + // The manifest must not certify stale chunks as if they came from B. + const { analyzerRegistry } = await import('../src/core/analyzer-registry.js'); + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-manifest-snapshot-')); + const sourcePath = path.join(rootPath, 'snapshot.ts'); + const analyzed = 'export const snapshotTarget = "A";\n'; + const changed = 'export const replacementTarget = "B";\n'; + await fs.writeFile(sourcePath, analyzed); + const original = analyzerRegistry.analyzeFile.bind(analyzerRegistry); + const spy = vi.spyOn(analyzerRegistry, 'analyzeFile').mockImplementation(async (...args) => { + const result = await original(...args); + if (args[0] === sourcePath) await fs.writeFile(sourcePath, changed); + return result; + }); + try { + await new CodebaseIndexer({ rootPath }).index(); + } finally { + spy.mockRestore(); + } + const manifest = JSON.parse( + await fs.readFile(path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME), 'utf8') + ) as { files: Record }; + const expected = (await import('node:crypto')) + .createHash('sha256') + .update(analyzed) + .digest('hex') + .slice(0, 16); + expect(manifest.files['snapshot.ts']).toBe(expected); + }); + + it('fails an incremental build when a pre-hashed unchanged file drifts before analysis', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-incremental-snapshot-')); + const triggerPath = path.join(rootPath, 'a-trigger.ts'); + const unchangedPath = path.join(rootPath, 'z-unchanged.ts'); + await fs.writeFile(triggerPath, 'export const trigger = 1;\n'); + await fs.writeFile(unchangedPath, 'export const stableTarget = "old";\n'); + await new CodebaseIndexer({ rootPath }).index(); + const metaPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, INDEX_META_FILENAME); + const priorMeta = JSON.parse(await fs.readFile(metaPath, 'utf8')) as { buildId: string }; + + await fs.writeFile(triggerPath, 'export const trigger = 2;\n'); + let mutated = false; + await expect( + new CodebaseIndexer({ + rootPath, + incrementalOnly: true, + onProgress: (progress) => { + if (progress.phase === 'analyzing' && !mutated) { + mutated = true; + writeFileSync(unchangedPath, 'export const replacementTarget = "new";\n'); + } + } + }).index() + ).rejects.toThrow(/Source changed during incremental scan/); + expect(mutated).toBe(true); + const currentMeta = JSON.parse(await fs.readFile(metaPath, 'utf8')) as { buildId: string }; + expect(currentMeta.buildId).toBe(priorMeta.buildId); + }, 60000); +}); diff --git a/tests/indexer-preview.test.ts b/tests/indexer-preview.test.ts new file mode 100644 index 0000000..944ad94 --- /dev/null +++ b/tests/indexer-preview.test.ts @@ -0,0 +1,278 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { promises as fs } from 'node:fs'; +import { renameSync, unlinkSync } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { CodebaseIndexer, selectFilesForIndex } from '../src/core/indexer.js'; +import { handleCliCommand } from '../src/cli.js'; + +describe('index preview enumeration', () => { + let root: string; + + afterEach(async () => { + if (root) await fs.rm(root, { recursive: true, force: true }); + }); + + it('reports only files the indexer will select and explains observed skips', async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-preview-')); + await fs.mkdir(path.join(root, 'packages', 'api'), { recursive: true }); + await fs.mkdir(path.join(root, 'packages', 'web', 'dist'), { recursive: true }); + await fs.writeFile(path.join(root, '.gitignore'), 'ignored.ts\n'); + await fs.writeFile(path.join(root, 'packages', 'api', 'keep.ts'), 'export const keep = 1;\n'); + await fs.writeFile(path.join(root, 'ignored.ts'), 'export const ignored = 1;\n'); + await fs.writeFile(path.join(root, 'large.ts'), `export const large = '${'x'.repeat(80)}';\n`); + await fs.writeFile( + path.join(root, 'packages', 'web', 'dist', 'bundle.ts'), + 'export const bundle = 1;\n' + ); + await fs.writeFile(path.join(root, 'notes.txt'), 'not a supported source file'); + + const preview = await new CodebaseIndexer({ + rootPath: root, + config: { skipEmbedding: true, parsing: { maxFileSize: 48 } } + }).previewFiles(); + + expect(preview.rootPath).toBe(root); + expect(preview.eligibleFiles).toEqual(['packages/api/keep.ts']); + expect(preview.skippedFiles).toEqual([ + { path: 'ignored.ts', reason: 'ignored by .gitignore' }, + { path: 'large.ts', reason: 'exceeds configured maximum file size' } + ]); + expect(preview.excludedPatterns).toContain('**/dist/**'); + expect(preview.excludedPatterns.some((pattern) => pattern.includes('dist/bundle.ts'))).toBe( + false + ); + await expect(fs.access(path.join(root, '.codebase-context'))).rejects.toMatchObject({ + code: 'ENOENT' + }); + }); + + it('honors project config exclusions in the Codex init preview', async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-preview-config-')); + await fs.mkdir(path.join(root, 'src'), { recursive: true }); + await fs.mkdir(path.join(root, 'private'), { recursive: true }); + await fs.writeFile(path.join(root, 'src', 'keep.ts'), 'export const keep = 1;\n'); + await fs.writeFile(path.join(root, 'private', 'secret.ts'), 'export const secret = 1;\n'); + const configPath = path.join(root, 'server-config.json'); + await fs.writeFile( + configPath, + JSON.stringify({ projects: [{ root, excludePatterns: ['**/private/**'] }] }) + ); + const previousConfigPath = process.env.CODEBASE_CONTEXT_CONFIG_PATH; + process.env.CODEBASE_CONTEXT_CONFIG_PATH = configPath; + const output = vi.spyOn(console, 'log').mockImplementation(() => undefined); + try { + await handleCliCommand(['init', '--client', 'codex', '--root', root, '--preview']); + const rendered = output.mock.calls.flat().join('\n'); + expect(rendered).toContain('src/keep.ts'); + expect(rendered).not.toContain('private/secret.ts'); + expect(rendered).toContain('**/private/**'); + } finally { + output.mockRestore(); + if (previousConfigPath === undefined) delete process.env.CODEBASE_CONTEXT_CONFIG_PATH; + else process.env.CODEBASE_CONTEXT_CONFIG_PATH = previousConfigPath; + } + }); + + it('reports overflow with a count and bounded examples, without materializing an overflow list', () => { + const files = Array.from({ length: 10025 }, (_, index) => `file-${index}.ts`); + + const selection = selectFilesForIndex(files); + + expect(selection.selected).toHaveLength(10000); + expect(selection.overLimitCount).toBe(25); + expect(selection.overLimitSamples).toHaveLength(20); + expect(selection.overLimitSamples[0]).toBe('file-10000.ts'); + }); + + it('rechecks gitignore after preview instead of indexing a stale selection', async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-preview-stale-')); + await fs.writeFile(path.join(root, 'keep.ts'), 'export const keep = 1;\n'); + await fs.writeFile(path.join(root, 'stale.ts'), 'export const stale = 1;\n'); + const indexer = new CodebaseIndexer({ + rootPath: root, + config: { skipEmbedding: true } + }); + + const preview = await indexer.previewFiles(); + expect(preview.eligibleFiles).toContain('stale.ts'); + await fs.writeFile(path.join(root, '.gitignore'), 'stale.ts\n'); + + const stats = await indexer.index(); + + expect(stats.totalFiles).toBe(1); + }); + + it('does not enumerate source symlinks whose canonical target is outside the root', async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-preview-symlink-')); + const outsideRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-preview-outside-')); + try { + const outsideFile = path.join(outsideRoot, 'outside.ts'); + const linkPath = path.join(root, 'linked.ts'); + await fs.writeFile(outsideFile, 'export const outside = true;\n'); + try { + await fs.symlink(outsideFile, linkPath, 'file'); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (process.platform === 'win32' && (code === 'EPERM' || code === 'EACCES')) { + throw new Error( + `Symlink containment oracle could not run: Windows denied symlink creation (${code}).` + ); + } + throw error; + } + + const preview = await new CodebaseIndexer({ rootPath: root }).previewFiles(); + + expect(preview.eligibleFiles).not.toContain('linked.ts'); + expect(preview.skippedFiles).toContainEqual({ + path: 'linked.ts', + reason: 'source resolves outside selected root' + }); + } finally { + await fs.rm(outsideRoot, { recursive: true, force: true }); + } + }); + + it('does not let an in-root alias bypass a configured exclusion', async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-preview-inroot-alias-')); + const privateDir = path.join(root, 'private'); + await fs.mkdir(privateDir, { recursive: true }); + const canonicalFile = path.join(privateDir, 'secret.ts'); + const aliasPath = path.join(root, 'alias.ts'); + await fs.writeFile(canonicalFile, 'export const secret = true;\n'); + try { + await fs.symlink(canonicalFile, aliasPath, 'file'); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (process.platform === 'win32' && (code === 'EPERM' || code === 'EACCES')) { + throw new Error( + `In-root alias oracle could not run: Windows denied symlink creation (${code}).` + ); + } + throw error; + } + + const preview = await new CodebaseIndexer({ + rootPath: root, + config: { exclude: ['**/private/**'] } + }).previewFiles(); + + expect(preview.eligibleFiles).not.toContain('alias.ts'); + expect(preview.eligibleFiles).not.toContain('private/secret.ts'); + expect(preview.skippedFiles).toContainEqual({ + path: 'alias.ts', + reason: 'source alias is not indexed; use its canonical path' + }); + }); + + it('counts one canonical source when multiple in-root aliases point to it', async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-preview-aliases-')); + const canonicalFile = path.join(root, 'real.ts'); + await fs.writeFile(canonicalFile, 'export const real = true;\n'); + for (const name of ['first.ts', 'second.ts']) { + try { + await fs.symlink(canonicalFile, path.join(root, name), 'file'); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (process.platform === 'win32' && (code === 'EPERM' || code === 'EACCES')) { + throw new Error( + `Alias deduplication oracle could not run: Windows denied symlink creation (${code}).` + ); + } + throw error; + } + } + + const preview = await new CodebaseIndexer({ rootPath: root }).previewFiles(); + + expect(preview.eligibleFiles).toEqual(['real.ts']); + expect(preview.skippedFileCount).toBe(2); + expect(preview.skippedFiles.map((file) => file.path).sort()).toEqual(['first.ts', 'second.ts']); + }); + + it('rechecks containment immediately before reading a source replaced by an outside symlink', async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-read-symlink-')); + const outsideRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-read-outside-')); + const sourcePath = path.join(root, 'source.ts'); + const replacementPath = path.join(root, 'replacement-link'); + try { + const outsideFile = path.join(outsideRoot, 'outside.ts'); + await fs.writeFile(sourcePath, 'export const inside = true;\n'); + await fs.writeFile(outsideFile, 'export const outside = true;\n'); + try { + await fs.symlink(outsideFile, replacementPath, 'file'); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (process.platform === 'win32' && (code === 'EPERM' || code === 'EACCES')) { + throw new Error( + `Pre-read containment oracle could not run: Windows denied symlink creation (${code}).` + ); + } + throw error; + } + + let replaced = false; + const indexer = new CodebaseIndexer({ + rootPath: root, + config: { skipEmbedding: true }, + onProgress: ({ phase }) => { + if (phase !== 'analyzing' || replaced) return; + unlinkSync(sourcePath); + renameSync(replacementPath, sourcePath); + replaced = true; + } + }); + + const stats = await indexer.index(); + + expect(replaced).toBe(true); + expect(stats.indexedFiles).toBe(0); + expect( + stats.errors.some((error) => error.error.includes('Source resolves outside selected root')) + ).toBe(true); + } finally { + await fs.rm(outsideRoot, { recursive: true, force: true }); + } + }); + + it('rejects a source replaced before read by an alias to another in-root file', async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), 'indexer-read-inroot-alias-')); + const sourcePath = path.join(root, 'source.ts'); + const canonicalPath = path.join(root, 'canonical.ts'); + const replacementPath = path.join(root, 'replacement-link'); + await fs.writeFile(sourcePath, 'export const original = true;\n'); + await fs.writeFile(canonicalPath, 'export const canonical = true;\n'); + try { + await fs.symlink(canonicalPath, replacementPath, 'file'); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (process.platform === 'win32' && (code === 'EPERM' || code === 'EACCES')) { + throw new Error( + `In-root read-alias oracle could not run: Windows denied symlink creation (${code}).` + ); + } + throw error; + } + + let replaced = false; + const indexer = new CodebaseIndexer({ + rootPath: root, + config: { skipEmbedding: true }, + onProgress: ({ phase }) => { + if (phase !== 'analyzing' || replaced) return; + unlinkSync(sourcePath); + renameSync(replacementPath, sourcePath); + replaced = true; + } + }); + + const stats = await indexer.index(); + + expect(replaced).toBe(true); + expect(stats.indexedFiles).toBe(1); + expect(stats.errors.some((error) => error.error.includes('Source alias is not indexed'))).toBe( + true + ); + }); +}); diff --git a/tests/internal-file-graph-serialization.test.ts b/tests/internal-file-graph-serialization.test.ts index 1f773a6..42829f3 100644 --- a/tests/internal-file-graph-serialization.test.ts +++ b/tests/internal-file-graph-serialization.test.ts @@ -12,7 +12,19 @@ describe('InternalFileGraph serialization', () => { const importingFile = path.join(rootPath, 'src', 'importer.ts'); graph.trackExports(exportedFile, [{ name: 'Foo', type: 'function' }]); - graph.trackImport(importingFile, exportedFile, 12, ['Foo']); + graph.trackImport(importingFile, exportedFile, 12, ['Foo'], { + kind: 'reexport', + specifier: './exported', + sourceFingerprint: 'source-sha256', + targetFingerprint: 'target-sha256' + }); + graph.trackUnresolved(importingFile, { + line: 18, + specifier: './missing', + kind: 'import', + sourceFingerprint: 'source-sha256', + targetFingerprint: '' + }, 'not-found'); const json = graph.toJSON(); expect(json.importDetails).toBeDefined(); @@ -20,6 +32,10 @@ describe('InternalFileGraph serialization', () => { const restored = InternalFileGraph.fromJSON(json, rootPath); const restoredJson = restored.toJSON(); expect(restoredJson.importDetails).toEqual(json.importDetails); + expect(restoredJson.unresolvedEdges).toEqual(json.unresolvedEdges); + expect(restoredJson.importDetails?.['src/importer.ts']?.['src/exported.ts']?.sources?.[0]).toMatchObject({ + specifier: './exported', kind: 'reexport', sourceFingerprint: 'source-sha256', targetFingerprint: 'target-sha256' + }); const unused = restored.findUnusedExports(); expect(unused.length).toBe(0); diff --git a/tests/internal-source-resolver.test.ts b/tests/internal-source-resolver.test.ts new file mode 100644 index 0000000..9680cd8 --- /dev/null +++ b/tests/internal-source-resolver.test.ts @@ -0,0 +1,133 @@ +import { afterAll, expect, it } from 'vitest'; +import { promises as fs } from 'fs'; +import os from 'os'; +import path from 'path'; +import ts from 'typescript'; +import { extractSourceReexports, InternalSourceResolver, loadTsconfigCompilerOptions } from '../src/utils/internal-source-resolver.js'; + +const root = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-source-resolver-')); +afterAll(async () => fs.rm(root, { recursive: true, force: true })); + +async function configProject(name: string): Promise { + const project = path.join(root, name, 'project'); + await fs.mkdir(project, { recursive: true }); + await fs.writeFile(path.join(root, name, 'tsconfig.json'), JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { '@ancestor/*': ['ancestor/*'] } } + })); + return project; +} + +it('loads aliases from the selected root tsconfig.base.json instead of an ancestor config', async () => { + const project = await configProject('base-only'); + await fs.writeFile(path.join(project, 'tsconfig.base.json'), JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { '@shared/*': ['src/*'] } } + })); + await fs.mkdir(path.join(project, 'src')); + const importer = path.join(project, 'src/entry.ts'); + const target = path.join(project, 'src/util.ts'); + await Promise.all([fs.writeFile(importer, ''), fs.writeFile(target, 'export const value = true;')]); + + const compilerOptions = loadTsconfigCompilerOptions(project); + expect(path.resolve(compilerOptions.baseUrl ?? '')).toBe(project); + expect(compilerOptions.paths).toEqual({ '@shared/*': ['src/*'] }); + const resolver = await InternalSourceResolver.create(project, [importer, target], { compilerOptions }); + expect(await resolver.resolve(importer, '@shared/util')).toEqual({ status: 'resolved', path: target }); + expect(await resolver.resolve(importer, '@unmapped/util')).toEqual({ status: 'unresolved', reason: 'external' }); +}); + +it('prefers the selected root tsconfig.json without merging its base-config sibling', async () => { + const project = await configProject('primary'); + await fs.writeFile(path.join(project, 'tsconfig.json'), JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { '@primary/*': ['primary/*'] } } + })); + await fs.writeFile(path.join(project, 'tsconfig.base.json'), JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { '@base/*': ['base/*'] } } + })); + + const compilerOptions = loadTsconfigCompilerOptions(project); + expect(path.resolve(compilerOptions.baseUrl ?? '')).toBe(project); + expect(compilerOptions.paths).toEqual({ '@primary/*': ['primary/*'] }); +}); + +it('uses resolver defaults when the selected root has no config, without borrowing an ancestor', async () => { + const project = await configProject('no-config'); + expect(loadTsconfigCompilerOptions(project)).toEqual({ + module: ts.ModuleKind.Preserve, + moduleResolution: ts.ModuleResolutionKind.Bundler + }); +}); + +it.each(['tsconfig.json', 'tsconfig.base.json'])('preserves explicit extends from the selected %s', async (configName) => { + const project = await configProject(`extends-${configName}`); + await fs.writeFile(path.join(path.dirname(project), 'shared.json'), JSON.stringify({ + compilerOptions: { baseUrl: './project', paths: { '@inherited/*': ['src/*'] }, strict: true } + })); + await fs.writeFile(path.join(project, configName), JSON.stringify({ extends: '../shared.json' })); + + const compilerOptions = loadTsconfigCompilerOptions(project); + expect(path.resolve(compilerOptions.baseUrl ?? '')).toBe(project); + expect(compilerOptions.paths).toEqual({ '@inherited/*': ['src/*'] }); + expect(compilerOptions.strict).toBe(true); +}); + +it('keeps defaults for a malformed primary config instead of silently loading the base sibling', async () => { + const project = await configProject('malformed-primary'); + await fs.writeFile(path.join(project, 'tsconfig.json'), '{'); + await fs.writeFile(path.join(project, 'tsconfig.base.json'), JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { '@base/*': ['src/*'] } } + })); + expect(loadTsconfigCompilerOptions(project)).toEqual({ + module: ts.ModuleKind.Preserve, + moduleResolution: ts.ModuleResolutionKind.Bundler + }); +}); + +it('resolves only indexed TS/TSX files, directory indexes, aliases, and JS specifiers to source', async () => { + const importer = path.join(root, 'apps/web/src/entry.ts'); + const jsAliasTarget = path.join(root, 'apps/web/src/actual.ts'); + const targets = [ + path.join(root, 'apps/web/src/view.tsx'), + path.join(root, 'packages/core/src/index.ts'), + path.join(root, 'packages/core/src/actual.ts') + ]; + await fs.mkdir(path.dirname(importer), { recursive: true }); + await fs.mkdir(path.dirname(targets[0]!), { recursive: true }); + await fs.mkdir(path.dirname(targets[1]!), { recursive: true }); + await Promise.all([fs.writeFile(importer, ''), fs.writeFile(jsAliasTarget, ''), ...targets.map((file) => fs.writeFile(file, ''))]); + const resolver = await InternalSourceResolver.create(root, [importer, jsAliasTarget, ...targets], { + baseUrl: '.', paths: { '@core/*': ['packages/core/src/*'] } + }); + + expect(await resolver.resolve(importer, './view')).toMatchObject({ status: 'resolved', path: targets[0] }); + expect(await resolver.resolve(importer, '../../../packages/core/src')).toMatchObject({ status: 'resolved', path: targets[1] }); + expect(await resolver.resolve(importer, '@core/actual')).toMatchObject({ status: 'resolved', path: targets[2] }); + expect(await resolver.resolve(importer, '@core/missing')).toMatchObject({ status: 'unresolved', reason: 'not-found' }); + expect(await resolver.resolve(importer, 'external-package')).toMatchObject({ status: 'unresolved', reason: 'external' }); + expect(await resolver.resolve(importer, './ghost')).toMatchObject({ status: 'unresolved', reason: 'not-found' }); + expect(await resolver.resolve(importer, './actual.js')).toMatchObject({ status: 'resolved', path: jsAliasTarget }); +}); + +it('extracts only explicit AST re-exports and rejects paths resolving outside the root', async () => { + expect(extractSourceReexports("export { thing } from './impl';\nexport * from './other';", 'index.ts')) + .toEqual([{ specifier: './impl', line: 1 }, { specifier: './other', line: 2 }]); + expect(extractSourceReexports("import './impl';", 'index.ts')).toEqual([]); + + const outside = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-outside-')); + const importer = path.join(root, 'src/in-root.ts'); + const escape = path.join(root, 'src/escape'); + await fs.mkdir(path.dirname(importer), { recursive: true }); + await fs.writeFile(importer, ''); + let symlinkCreated = false; + try { + await fs.writeFile(path.join(outside, 'file.ts'), 'export const outside = true;'); + await fs.symlink(outside, escape, 'junction'); + symlinkCreated = true; + } catch { + // Junction creation can be disabled by the host; direct traversal tests still cover outside-root paths. + } + if (symlinkCreated) { + const resolver = await InternalSourceResolver.create(root, [importer, path.join(escape, 'file.ts')]); + expect(await resolver.resolve(importer, './escape/file')).toMatchObject({ status: 'unresolved', reason: 'outside-root' }); + } + await fs.rm(outside, { recursive: true, force: true }); +}); diff --git a/tests/multi-project-routing.test.ts b/tests/multi-project-routing.test.ts index fa8f181..2fb47df 100644 --- a/tests/multi-project-routing.test.ts +++ b/tests/multi-project-routing.test.ts @@ -64,6 +64,12 @@ vi.mock('../src/core/search.js', async () => { async search(query: string, limit: number, filters?: unknown, options?: unknown) { return searchMocks.search(this.rootPath, query, limit, filters, options); } + async searchWithCoverage(query: string, limit: number, filters?: unknown, options?: unknown) { + return { + results: await this.search(query, limit, filters, options), + sourceCoverage: { status: 'current', checkedFiles: 0, skippedFiles: 0 } + }; + } } return { CodebaseSearcher }; @@ -221,7 +227,7 @@ describe('multi-project routing', () => { afterEach(async () => { const { clearProjects } = await import('../src/project-state.js'); - clearProjects(); + await clearProjects(); if (originalArgv) process.argv = originalArgv; if (originalEnvRoot === undefined) { @@ -480,6 +486,76 @@ describe('multi-project routing', () => { ); }); + it('reserves watcher slots while concurrent opens evict and reopen a closing project', async () => { + const roots = [primaryRoot, secondaryRoot, nestedProjectRoot]; + for (let index = 0; index < 4; index += 1) { + const root = await fs.mkdtemp(path.join(os.tmpdir(), `cc-watcher-limit-${index}-`)); + roots.push(root); + await seedValidIndex(root); + } + + const oldestRoot = roots[0]!; + const [openingRootA, openingRootB] = roots.slice(5); + if (!openingRootA || !openingRootB) throw new Error('expected two new watcher roots'); + + let finishOldestClose!: () => void; + const oldestClose = new Promise((resolve) => { + finishOldestClose = resolve; + }); + const stopCalls = new Map(); + watcherMocks.start.mockImplementation(({ rootPath }: { rootPath: string }) => () => { + stopCalls.set(rootPath, (stopCalls.get(rootPath) ?? 0) + 1); + return rootPath === oldestRoot ? oldestClose : Promise.resolve(); + }); + + const { server } = await import('../src/index.js'); + const handler = (server as unknown as TestServer)._requestHandlers.get('tools/call'); + if (!handler) throw new Error('tools/call handler not registered'); + + try { + for (const [id, root] of roots.slice(0, 5).entries()) { + await callTool(handler, 200 + id, 'search_codebase', { query: 'feature', project: root }); + } + + const opening = [ + callTool(handler, 210, 'search_codebase', { query: 'feature', project: openingRootA }), + callTool(handler, 211, 'search_codebase', { query: 'feature', project: openingRootB }) + ]; + await vi.waitFor(() => { + expect(stopCalls.get(oldestRoot)).toBeGreaterThan(0); + }); + + const reopening = [ + callTool(handler, 212, 'search_codebase', { query: 'feature', project: oldestRoot }), + callTool(handler, 213, 'search_codebase', { query: 'feature', project: oldestRoot }) + ]; + await vi.waitFor(() => { + expect(watcherMocks.start).toHaveBeenCalledWith( + expect.objectContaining({ rootPath: oldestRoot }) + ); + }); + expect( + watcherMocks.start.mock.calls.filter(([options]) => options.rootPath === oldestRoot) + ).toHaveLength(1); + + finishOldestClose(); + await Promise.all([...opening, ...reopening]); + + const { getAllProjects } = await import('../src/project-state.js'); + expect(getAllProjects().filter((project) => project.stopWatcher)).toHaveLength(5); + expect( + watcherMocks.start.mock.calls.filter(([options]) => options.rootPath === oldestRoot) + ).toHaveLength(2); + expect(stopCalls.get(oldestRoot)).toBe(1); + } finally { + finishOldestClose(); + await fs.rm(roots[3]!, { recursive: true, force: true }); + await fs.rm(roots[4]!, { recursive: true, force: true }); + await fs.rm(roots[5]!, { recursive: true, force: true }); + await fs.rm(roots[6]!, { recursive: true, force: true }); + } + }); + it('uses the active project for later tool calls and returns project metadata', async () => { const { server } = await import('../src/index.js'); const handler = (server as unknown as TestServer)._requestHandlers.get('tools/call'); @@ -591,7 +667,9 @@ describe('multi-project routing', () => { it('lists bounded and full context resources for active and project-scoped flows', async () => { const { server } = await import('../src/index.js'); const toolHandler = (server as unknown as TestServer)._requestHandlers.get('tools/call'); - const resourcesHandler = (server as unknown as TestServer)._requestHandlers.get('resources/list'); + const resourcesHandler = (server as unknown as TestServer)._requestHandlers.get( + 'resources/list' + ); if (!toolHandler || !resourcesHandler) { throw new Error('required handlers not registered'); diff --git a/tests/project-state.test.ts b/tests/project-state.test.ts index 92820e5..49314a7 100644 --- a/tests/project-state.test.ts +++ b/tests/project-state.test.ts @@ -11,8 +11,8 @@ import { import { CODEBASE_CONTEXT_DIRNAME } from '../src/constants/codebase-context.js'; describe('project-state', () => { - afterEach(() => { - clearProjects(); + afterEach(async () => { + await clearProjects(); vi.restoreAllMocks(); }); @@ -85,33 +85,54 @@ describe('project-state', () => { }); describe('clearProjects', () => { - it('empties the map', () => { + it('empties the map', async () => { getOrCreateProject('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/projects/app-a'); getOrCreateProject('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/projects/app-b'); - clearProjects(); + await clearProjects(); // After clearing, getOrCreateProject should return a fresh instance const p = getOrCreateProject('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/projects/app-a'); expect(p.indexState.status).toBe('idle'); }); - it('calls stopWatcher on projects with watchers', () => { + it('calls stopWatcher on projects with watchers', async () => { const p = getOrCreateProject('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/projects/my-app'); - const stopSpy = vi.fn(); + const stopSpy = vi.fn(async () => undefined); p.stopWatcher = stopSpy; - clearProjects(); + await clearProjects(); expect(stopSpy).toHaveBeenCalledTimes(1); }); + + it('waits for watcher close after synchronously clearing project state', async () => { + let finishClose!: () => void; + const closeCompletion = new Promise((resolve) => { + finishClose = resolve; + }); + const project = getOrCreateProject('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/projects/closing-app'); + project.stopWatcher = vi.fn(() => closeCompletion); + + let clearResolved = false; + const clearing = clearProjects().then(() => { + clearResolved = true; + }); + expect(getProject('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/projects/closing-app')).toBeUndefined(); + await Promise.resolve(); + expect(clearResolved).toBe(false); + + finishClose(); + await clearing; + expect(clearResolved).toBe(true); + }); }); describe('removeProject', () => { - it('removes a project and stops its watcher', () => { + it('removes a project and stops its watcher', async () => { const project = getOrCreateProject('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/projects/my-app'); - const stopSpy = vi.fn(); + const stopSpy = vi.fn(async () => undefined); project.stopWatcher = stopSpy; - removeProject('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/projects/my-app'); + await removeProject('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/projects/my-app'); expect(stopSpy).toHaveBeenCalledTimes(1); expect(getProject('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/projects/my-app')).toBeUndefined(); diff --git a/tests/react-analyzer.test.ts b/tests/react-analyzer.test.ts index 2cb7939..eaed855 100644 --- a/tests/react-analyzer.test.ts +++ b/tests/react-analyzer.test.ts @@ -14,6 +14,48 @@ describe('ReactAnalyzer', () => { expect(analyzer.canAnalyze('/tmp/file.js', 'React.createElement("div");')).toBe(true); }); + it.each(['.ts', '.mts', '.cts'])( + 'does not claim %s files for TypeScript generic or angle-assertion syntax alone', + (extension) => { + const analyzer = new ReactAnalyzer(); + + expect( + analyzer.canAnalyze( + `/tmp/file${extension}`, + 'const map = new Map();' + ) + ).toBe(false); + expect( + analyzer.canAnalyze(`/tmp/file${extension}`, 'const value = input;') + ).toBe(false); + } + ); + + it.each(['.ts', '.mts', '.cts'])( + 'continues to recognize explicit React markers in %s files', + (extension) => { + const analyzer = new ReactAnalyzer(); + + expect( + analyzer.canAnalyze(`/tmp/file${extension}`, 'import { useState } from "react";') + ).toBe(true); + expect(analyzer.canAnalyze(`/tmp/file${extension}`, 'const React = require("react");')).toBe( + true + ); + expect(analyzer.canAnalyze(`/tmp/file${extension}`, 'React.createElement("div");')).toBe( + true + ); + } + ); + + it('continues to claim JSX extensions and JavaScript JSX syntax', () => { + const analyzer = new ReactAnalyzer(); + + expect(analyzer.canAnalyze('/tmp/file.tsx')).toBe(true); + expect(analyzer.canAnalyze('/tmp/file.jsx')).toBe(true); + expect(analyzer.canAnalyze('/tmp/file.js', 'const element =
;')).toBe(true); + }); + it('detects components, hooks, context, and ecosystem signals', async () => { const analyzer = new ReactAnalyzer(); const filePath = path.join(process.cwd(), 'src', 'components', 'MyWidget.tsx'); @@ -54,7 +96,9 @@ export class LegacyWidget extends Component { expect(result.framework).toBe('react'); expect( - result.components.some((component) => component.componentType === 'hook' && component.name === 'useTheme') + result.components.some( + (component) => component.componentType === 'hook' && component.name === 'useTheme' + ) ).toBe(true); expect( result.components.some( @@ -67,7 +111,10 @@ export class LegacyWidget extends Component { ) ).toBe(true); - const patterns = (result.metadata.detectedPatterns || []) as Array<{ category: string; name: string }>; + const patterns = (result.metadata.detectedPatterns || []) as Array<{ + category: string; + name: string; + }>; expect(patterns).toContainEqual({ category: 'stateManagement', name: 'React Context' }); expect(patterns).toContainEqual({ category: 'reactivity', name: 'Suspense' }); expect(patterns).toContainEqual({ category: 'reactivity', name: 'Memoization' }); @@ -99,7 +146,10 @@ export function CheckoutForm({ cartPromise }: { cartPromise: Promise } `; const result = await analyzer.analyze(filePath, code); - const patterns = (result.metadata.detectedPatterns || []) as Array<{ category: string; name: string }>; + const patterns = (result.metadata.detectedPatterns || []) as Array<{ + category: string; + name: string; + }>; expect(patterns).toContainEqual({ category: 'reactHooks', name: 'Built-in hooks' }); expect(patterns).toContainEqual({ category: 'reactivity', name: 'Actions' }); diff --git a/tests/reference-nomination-parity.test.ts b/tests/reference-nomination-parity.test.ts new file mode 100644 index 0000000..0ee36dc --- /dev/null +++ b/tests/reference-nomination-parity.test.ts @@ -0,0 +1,88 @@ +import { readFileSync } from 'node:fs'; +import path from 'node:path'; +import { describe, expect, it } from 'vitest'; +import { + hasIdentifierCandidate, + findIdentifierOccurrences, + findLexicalIdentifierOccurrences +} from '../src/utils/tree-sitter'; + +interface ParityCase { + id: string; + language: string; + symbol: string; + source: string; + expected: { mode: 'ast' | 'lexical'; count: number }; +} + +interface ParityFixture { + contract: string; + cases: ParityCase[]; +} + +const fixturePath = path.join( + process.cwd(), + 'tests', + 'fixtures', + 'reference-nomination-parity.json' +); +const fixture = JSON.parse(readFileSync(fixturePath, 'utf8')) as ParityFixture; + +describe('reference nomination parity fixture', () => { + it('freezes current parser and lexical behavior across grammar and uncertainty cases', async () => { + expect(fixture.cases.length).toBeGreaterThanOrEqual(20); + + for (const example of fixture.cases) { + const ast = await findIdentifierOccurrences(example.source, example.language, example.symbol); + const mode = ast === null ? 'lexical' : 'ast'; + const occurrences = + ast === null ? findLexicalIdentifierOccurrences(example.source, example.symbol) : ast; + + expect({ mode, count: occurrences.length }, example.id).toEqual(example.expected); + } + }); + + it('preserves the frozen result with the production ASCII-conservative nominator', async () => { + const omittedCandidates: string[] = []; + for (const example of fixture.cases) { + const nominated = hasIdentifierCandidate(example.source, example.symbol); + if (example.source.includes(example.symbol) && !nominated) { + omittedCandidates.push(example.id); + } + const ast = nominated + ? await findIdentifierOccurrences(example.source, example.language, example.symbol) + : []; + const occurrences = !nominated + ? [] + : ast === null + ? findLexicalIdentifierOccurrences(example.source, example.symbol) + : ast; + + expect(occurrences.length, example.id).toBe(example.expected.count); + } + + expect(omittedCandidates).toEqual([ + 'ts-long-identifier', + 'ts-dollar-identifier', + 'ts-underscore-identifier', + 'ts-underscore-suffix', + 'ts-number-error-split', + 'javascript-number-suffix', + 'python-number-suffix', + 'go-number-suffix', + 'rust-number-suffix', + 'java-number-suffix', + 'c-number-suffix', + 'cpp-number-suffix', + 'c-hex-prefix-split', + 'cpp-hex-prefix-split', + 'csharp-number-suffix', + 'kotlin-number-suffix', + 'python-long-identifier', + 'unknown-long-identifier', + 'unknown-dollar-boundary', + 'unknown-number-error-split', + 'excalidraw-source-token-inside-long-identifier' + ]); + }); +}); diff --git a/tests/refresh-index-mode.test.ts b/tests/refresh-index-mode.test.ts new file mode 100644 index 0000000..c6412fc --- /dev/null +++ b/tests/refresh-index-mode.test.ts @@ -0,0 +1,110 @@ +import { describe, expect, it, vi } from 'vitest'; +import * as embeddings from '../src/embeddings/index.js'; +import { OptionalDependencyError } from '../src/errors/index.js'; +import * as storage from '../src/storage/index.js'; +import { handle } from '../src/tools/refresh-index.js'; +import type { ToolContext } from '../src/tools/types.js'; + +function makeContext(status: ToolContext['indexState']['status'] = 'idle') { + const indexer = {} as NonNullable; + const stats = {} as NonNullable; + return { + indexState: { status, lastIndexed: new Date(1), indexer, stats }, + paths: {} as ToolContext['paths'], + rootPath: '/project', + performIndexing: vi.fn() + } satisfies ToolContext; +} + +describe('refresh_index embedding mode', () => { + it('forwards explicit semantic opt-in and preserves omission for recorded mode resolution', async () => { + const context = makeContext(); + const embeddingAssert = vi + .spyOn(embeddings, 'assertEmbeddingRuntimeAvailable') + .mockImplementation(() => {}); + const storageAssert = vi + .spyOn(storage, 'assertStorageRuntimeAvailable') + .mockImplementation(() => {}); + try { + await handle( + { skipEmbedding: false, incrementalOnly: true, reason: 'owner choice' }, + context + ); + await handle({ incrementalOnly: true }, context); + + expect(embeddingAssert).toHaveBeenCalledTimes(1); + expect(storageAssert).toHaveBeenCalledTimes(1); + expect(context.performIndexing).toHaveBeenNthCalledWith(1, true, 'owner choice', false); + expect(context.performIndexing).toHaveBeenNthCalledWith(2, true, undefined, undefined); + } finally { + embeddingAssert.mockRestore(); + storageAssert.mockRestore(); + } + }); + + it('rejects invalid or busy mode changes before starting indexing', async () => { + const invalidContext = makeContext(); + const invalid = await handle({ skipEmbedding: 'false' }, invalidContext); + expect(invalid.isError).toBe(true); + expect(invalidContext.performIndexing).not.toHaveBeenCalled(); + + const busyContext = makeContext('indexing'); + const busy = await handle({ skipEmbedding: false }, busyContext); + expect(busy.isError).toBe(true); + expect(busyContext.performIndexing).not.toHaveBeenCalled(); + }); + + it('rejects missing embedding runtime before dispatch and preserves prior ready state', async () => { + const context = makeContext('ready'); + const originalState = { ...context.indexState }; + const missingRuntime = new OptionalDependencyError('@huggingface/transformers', '^4.2.0'); + const embeddingAssert = vi + .spyOn(embeddings, 'assertEmbeddingRuntimeAvailable') + .mockImplementationOnce(() => { + throw missingRuntime; + }); + const storageAssert = vi.spyOn(storage, 'assertStorageRuntimeAvailable'); + + try { + const response = await handle({ skipEmbedding: false }, context); + expect(response.isError).toBe(true); + expect(response.content?.[0].text).toContain('@huggingface/transformers'); + expect(response.content?.[0].text).toContain('npm install'); + expect(context.performIndexing).not.toHaveBeenCalled(); + expect(context.indexState).toEqual(originalState); + expect(embeddingAssert).toHaveBeenCalledTimes(1); + expect(storageAssert).not.toHaveBeenCalled(); + } finally { + embeddingAssert.mockRestore(); + storageAssert.mockRestore(); + } + }); + + it('rejects missing vector storage after embedding preflight without changing project state', async () => { + const context = makeContext('ready'); + const originalState = { ...context.indexState }; + const embeddingAssert = vi + .spyOn(embeddings, 'assertEmbeddingRuntimeAvailable') + .mockImplementation(() => {}); + const missingRuntime = new OptionalDependencyError('@lancedb/lancedb', '^0.4.0'); + const storageAssert = vi + .spyOn(storage, 'assertStorageRuntimeAvailable') + .mockImplementationOnce(() => { + throw missingRuntime; + }); + + try { + const response = await handle({ skipEmbedding: false }, context); + expect(response.isError).toBe(true); + expect(response.content?.[0].text).toContain('@lancedb/lancedb'); + expect(response.content?.[0].text).toContain('npm install'); + expect(context.performIndexing).not.toHaveBeenCalled(); + expect(context.indexState).toEqual(originalState); + expect(embeddingAssert).toHaveBeenCalledTimes(1); + expect(storageAssert).toHaveBeenCalledTimes(1); + } finally { + embeddingAssert.mockRestore(); + storageAssert.mockRestore(); + } + }); +}); diff --git a/tests/relationship-response-budget.test.ts b/tests/relationship-response-budget.test.ts new file mode 100644 index 0000000..f7eb50b --- /dev/null +++ b/tests/relationship-response-budget.test.ts @@ -0,0 +1,52 @@ +import { expect, it } from 'vitest'; +import { finalizeJsonTextPayload } from '../src/index.js'; + +it('keeps the final relationship text within budget after routing metadata is added', () => { + const graphPayload = { + status: 'success', + file: 'packages/example/src/main.ts', + direction: 'outgoing', + maxHops: 3, + paths: Array.from({ length: 16 }, (_, index) => ({ + nodes: [ + 'packages/example/src/main.ts', + `packages/example/src/branch-${index}-with-a-long-source-path.ts` + ], + edges: [ + { + from: 'packages/example/src/main.ts', + to: `packages/example/src/branch-${index}-with-a-long-source-path.ts`, + kind: 'import', + source: { line: index + 1, specifier: `./branch-${index}-with-a-long-source-path` } + } + ] + })), + unresolved: [], + truncated: false, + message: + 'Relationships are limited to verified imports and re-exports; dynamic calls are not inferred.' + }; + const routedPayload = { + ...graphPayload, + index: { status: 'ready', lastIndexed: '2026-10-11T04:30:00.000Z', buildId: 'b'.repeat(64) }, + project: { + id: 'p'.repeat(64), + rootPath: 'C:/projects/consumer', + source: 'root', + label: 'consumer project' + } + }; + + const finalText = finalizeJsonTextPayload(routedPayload, true); + expect(Buffer.byteLength(finalText, 'utf8')).toBeLessThanOrEqual(4000); + const finalized = JSON.parse(finalText) as { + paths: unknown[]; + truncated: boolean; + index: unknown; + project: unknown; + }; + expect(finalized.truncated).toBe(true); + expect(finalized.paths.length).toBeGreaterThan(0); + expect(finalized.index).toEqual(routedPayload.index); + expect(finalized.project).toEqual(routedPayload.project); +}); diff --git a/tests/relationship-sidecar.test.ts b/tests/relationship-sidecar.test.ts index d4ac6da..d3ce7d2 100644 --- a/tests/relationship-sidecar.test.ts +++ b/tests/relationship-sidecar.test.ts @@ -6,7 +6,8 @@ import { readIndexMeta, validateIndexArtifacts } from '../src/core/index-meta.js import { CODEBASE_CONTEXT_DIRNAME, RELATIONSHIPS_FILENAME, - INDEX_META_FILENAME + INDEX_META_FILENAME, + INDEX_FORMAT_VERSION } from '../src/constants/codebase-context.js'; async function createTempDir(): Promise { @@ -74,7 +75,7 @@ describe('Relationship Sidecar', () => { expect(relationships.header).toBeDefined(); expect(relationships.header.buildId).toBeDefined(); expect(typeof relationships.header.buildId).toBe('string'); - expect(relationships.header.formatVersion).toBe(1); + expect(relationships.header.formatVersion).toBe(INDEX_FORMAT_VERSION); expect(relationships.generatedAt).toBeDefined(); expect(relationships.graph).toBeDefined(); expect(typeof relationships.graph.imports).toBe('object'); @@ -189,7 +190,7 @@ describe('Relationship Sidecar', () => { const relationships = JSON.parse(raw); expect(relationships.header).toBeDefined(); - expect(relationships.header.formatVersion).toBe(1); + expect(relationships.header.formatVersion).toBe(INDEX_FORMAT_VERSION); expect(relationships.graph).toBeDefined(); expect(relationships.symbols).toBeDefined(); expect(relationships.stats).toBeDefined(); diff --git a/tests/reranker.test.ts b/tests/reranker.test.ts index 7bee16c..c73754c 100644 --- a/tests/reranker.test.ts +++ b/tests/reranker.test.ts @@ -1,6 +1,7 @@ -import { describe, it, expect } from 'vitest'; +import { describe, it, expect, vi } from 'vitest'; import { isAmbiguous } from '../src/core/reranker.js'; -import type { SearchResult } from '../src/types/index.js'; +import type { CodeChunk, SearchResult } from '../src/types/index.js'; +import { asSearcherTestHarness, materializeSourceSnapshot } from './helpers/source-snapshot.js'; function makeResult(score: number, filePath: string): SearchResult { return { @@ -29,23 +30,20 @@ describe('Reranker ambiguity detection', () => { const results = [ makeResult(0.95, '/a.ts'), makeResult(0.75, '/b.ts'), - makeResult(0.60, '/c.ts') + makeResult(0.6, '/c.ts') ]; expect(isAmbiguous(results)).toBe(false); }); it('returns false for fewer than 3 results', () => { - const results = [ - makeResult(0.85, '/a.ts'), - makeResult(0.84, '/b.ts') - ]; + const results = [makeResult(0.85, '/a.ts'), makeResult(0.84, '/b.ts')]; expect(isAmbiguous(results)).toBe(false); }); it('correctly handles edge case at threshold boundary', () => { // Gap of exactly 0.08 is NOT below the < 0.08 threshold const results = [ - makeResult(0.90, '/a.ts'), + makeResult(0.9, '/a.ts'), makeResult(0.85, '/b.ts'), makeResult(0.82, '/c.ts') ]; @@ -53,7 +51,7 @@ describe('Reranker ambiguity detection', () => { // Gap of 0.07 IS below the threshold const ambiguous = [ - makeResult(0.90, '/a.ts'), + makeResult(0.9, '/a.ts'), makeResult(0.86, '/b.ts'), makeResult(0.83, '/c.ts') ]; @@ -67,93 +65,94 @@ describe('File-level dedupe in search results', () => { // The dedupe logic is in scoreAndSortResults — tested indirectly by // ensuring results contain unique file paths const { CodebaseSearcher } = await import('../src/core/search.js'); - const { vi } = await import('vitest'); - - const searcher = new CodebaseSearcher('C:/repo') as any; - searcher.initialized = true; - searcher.embeddingProvider = { - embed: vi.fn(async () => [0.1, 0.2]) - }; - searcher.storageProvider = { - search: vi.fn(async () => [ - { - chunk: { - id: 'chunk1', - content: 'class Foo {}', - filePath: 'C:/repo/src/foo.ts', - relativePath: 'src/foo.ts', - startLine: 1, - endLine: 10, - language: 'typescript', - framework: 'generic', - componentType: 'service', - layer: 'core', - dependencies: [], - imports: [], - exports: [], - tags: [], - metadata: {} - }, - score: 0.9 - }, - { - chunk: { - id: 'chunk2', - content: 'class Bar extends Foo {}', - filePath: 'C:/repo/src/foo.ts', // same file, different chunk - relativePath: 'src/foo.ts', - startLine: 11, - endLine: 20, - language: 'typescript', - framework: 'generic', - componentType: 'service', - layer: 'core', - dependencies: [], - imports: [], - exports: [], - tags: [], - metadata: {} - }, - score: 0.85 - }, - { - chunk: { - id: 'chunk3', - content: 'class Baz {}', - filePath: 'C:/repo/src/baz.ts', // different file - relativePath: 'src/baz.ts', - startLine: 1, - endLine: 10, - language: 'typescript', - framework: 'generic', - componentType: 'service', - layer: 'core', - dependencies: [], - imports: [], - exports: [], - tags: [], - metadata: {} - }, - score: 0.80 - } - ]), - count: vi.fn(async () => 3) - }; - searcher.fuseIndex = null; - searcher.patternIntelligence = null; + const chunks: CodeChunk[] = [ + { + id: 'chunk1', + content: 'class Foo {}', + filePath: 'C:/repo/src/foo.ts', + relativePath: 'src/foo.ts', + startLine: 1, + endLine: 10, + language: 'typescript', + framework: 'generic', + componentType: 'service', + layer: 'core', + dependencies: [], + imports: [], + exports: [], + tags: [], + metadata: {} + }, + { + id: 'chunk2', + content: 'class Bar extends Foo {}', + filePath: 'C:/repo/src/foo.ts', + relativePath: 'src/foo.ts', + startLine: 11, + endLine: 20, + language: 'typescript', + framework: 'generic', + componentType: 'service', + layer: 'core', + dependencies: [], + imports: [], + exports: [], + tags: [], + metadata: {} + }, + { + id: 'chunk3', + content: 'class Baz {}', + filePath: 'C:/repo/src/baz.ts', + relativePath: 'src/baz.ts', + startLine: 1, + endLine: 10, + language: 'typescript', + framework: 'generic', + componentType: 'service', + layer: 'core', + dependencies: [], + imports: [], + exports: [], + tags: [], + metadata: {} + } + ]; + const snapshot = await materializeSourceSnapshot(chunks, 'reranker-source-'); + try { + const searcher = new CodebaseSearcher(snapshot.rootPath); + const harness = asSearcherTestHarness(searcher); + harness.initialized = true; + harness.sourceManifest = snapshot.manifest; + harness.embeddingProvider = { + embed: vi.fn(async () => [0.1, 0.2]) + }; + harness.storageProvider = { + search: vi.fn(async () => [ + { chunk: chunks[0], score: 0.9 }, + { chunk: chunks[1], score: 0.85 }, + { chunk: chunks[2], score: 0.8 } + ]), + count: vi.fn(async () => 3) + }; + harness.fuseIndex = null; + harness.patternIntelligence = null; - const results = await (searcher as any).search('Foo class', 5, undefined, { - useSemanticSearch: true, - useKeywordSearch: false, - enableReranker: false, - enableLowConfidenceRescue: false, - enableQueryExpansion: false - }); + const results = await searcher.search('Foo class', 5, undefined, { + useSemanticSearch: true, + useKeywordSearch: false, + enableReranker: false, + enableLowConfidenceRescue: false, + enableQueryExpansion: false + }); - // Should have 2 unique files, not 3 results (two from foo.ts) - const filePaths = results.map((r: any) => r.filePath); - expect(filePaths.length).toBe(2); - expect(filePaths[0]).toContain('foo.ts'); - expect(filePaths[1]).toContain('baz.ts'); + // Should have 2 unique files, not 3 results (two from foo.ts) + const filePaths = results.map((result) => result.filePath); + expect(filePaths.length).toBe(2); + expect(filePaths[0]).toContain('foo.ts'); + expect(filePaths[1]).toContain('baz.ts'); + } finally { + await snapshot.cleanup(); + } }); }); diff --git a/tests/review-context.test.ts b/tests/review-context.test.ts index 52dca41..c097455 100644 --- a/tests/review-context.test.ts +++ b/tests/review-context.test.ts @@ -13,7 +13,9 @@ import type { SearchResponse } from '../src/tools/types.js'; describe('review context', () => { it('parses modified, added, deleted and renamed files', () => { expect( - parseNameStatus(['M\tsrc/a.ts', 'A\tsrc/b.ts', 'D\tsrc/c.ts', 'R100\tsrc/old.ts\tsrc/new.ts'].join('\n')) + parseNameStatus( + ['M\tsrc/a.ts', 'A\tsrc/b.ts', 'D\tsrc/c.ts', 'R100\tsrc/old.ts\tsrc/new.ts'].join('\n') + ) ).toEqual([ { path: 'src/a.ts', status: 'M', rawStatus: 'M' }, { path: 'src/b.ts', status: 'A', rawStatus: 'A' }, @@ -54,7 +56,9 @@ describe('review context', () => { }); it('detects binary patches', () => { - expect(inspectPatch('diff --git a/a.png b/a.png\nBinary files a/a.png and b/a.png differ')).toEqual({ + expect( + inspectPatch('diff --git a/a.png b/a.png\nBinary files a/a.png and b/a.png differ') + ).toEqual({ additions: 0, deletions: 0, binary: true @@ -119,6 +123,45 @@ describe('review context', () => { expect(fingerprintDiff('diff body\n')).not.toBe(fingerprintDiff('diff body')); }); + it.each(['partial', 'no_fresh_results'] as const)( + 'keeps %s source coverage visible to review-context consumers', + async (status) => { + const rawDiff = '@@ -1 +1 @@\n-oldSession();\n+rotateSessionToken();'; + const sourceCoverage = { + status, + checkedFiles: status === 'partial' ? 2 : 1, + skippedFiles: 1, + refreshAction: 'refresh_index' + }; + const packet = await buildReviewContextPacket({ + base: 'main', + head: 'HEAD', + baseCommit: 'a'.repeat(40), + headCommit: 'b'.repeat(40), + rawDiff, + changedFiles: [{ path: 'src/auth.ts', status: 'M', rawStatus: 'M' }], + patchesByPath: new Map([['src/auth.ts', rawDiff]]), + search: async () => ({ + status: 'ok', + searchQuality: { status: 'ok', confidence: 0.9 }, + sourceCoverage, + preflight: { ready: false, reason: 'Indexed candidates are no longer current.' }, + results: + status === 'partial' + ? [{ file: 'src/current.ts:1-2', summary: 'Current source', score: 0.9 }] + : [] + }) + }); + + expect(packet.searches[0]?.sourceCoverage).toEqual(sourceCoverage); + expect(packet.searches[0]?.preflight?.ready).toBe(false); + expect(packet.warnings).toContain( + 'Search source coverage is incomplete for query: rotateSessionToken oldSession' + ); + expect(packet.searches[0]?.results).toHaveLength(status === 'partial' ? 1 : 0); + } + ); + it('builds a bounded packet and keeps failed searches explicit', async () => { const rawDiff = [ 'diff --git a/src/auth.ts b/src/auth.ts', diff --git a/tests/search-codebase-auto-heal.test.ts b/tests/search-codebase-auto-heal.test.ts index 13cd1a0..2bd3d16 100644 --- a/tests/search-codebase-auto-heal.test.ts +++ b/tests/search-codebase-auto-heal.test.ts @@ -19,6 +19,21 @@ const indexerMocks = vi.hoisted(() => ({ index: vi.fn() })); +function getToolsCallHandler(server: unknown) { + const handlers = ( + server as { + _requestHandlers: { + get( + name: string + ): ((request: unknown) => Promise<{ content: Array<{ text: string }> }>) | undefined; + }; + } + )._requestHandlers; + const handler = handlers.get('tools/call'); + if (!handler) throw new Error('tools/call handler not registered'); + return handler; +} + vi.mock('../src/core/search.js', async () => { class CodebaseSearcher { constructor(_rootPath: string) {} @@ -26,6 +41,13 @@ vi.mock('../src/core/search.js', async () => { async search(query: string, limit: number, filters?: unknown) { return searchMocks.search(query, limit, filters); } + + async searchWithCoverage(query: string, limit: number, filters?: unknown) { + return { + results: await this.search(query, limit, filters), + sourceCoverage: { status: 'current', checkedFiles: 0, skippedFiles: 0 } + }; + } } return { CodebaseSearcher }; @@ -151,7 +173,7 @@ describe('search_codebase auto-heal', () => { ); const { server } = await import('../src/index.js'); - const handler = (server as any)._requestHandlers.get('tools/call'); + const handler = getToolsCallHandler(server); const response = await handler({ jsonrpc: '2.0', @@ -171,9 +193,26 @@ describe('search_codebase auto-heal', () => { expect(searchMocks.search).toHaveBeenCalledTimes(1); }, 15000); + it('rebuilds an existing semantic index before search when startup explicitly requests model-free mode', async () => { + process.argv.push('--no-embeddings'); + const { server } = await import('../src/index.js'); + const handler = getToolsCallHandler(server); + const response = await handler({ + jsonrpc: '2.0', + id: 2, + method: 'tools/call', + params: { name: 'search_codebase', arguments: { query: 'test' } } + }); + const payload = JSON.parse(response.content[0].text); + expect(payload.status).toBe('indexing'); + expect(payload.message).toContain('retry'); + expect(indexerMocks.index).toHaveBeenCalledTimes(1); + expect(searchMocks.search).not.toHaveBeenCalled(); + }, 15000); + it('returns invalid_params when search_codebase query is missing', async () => { const { server } = await import('../src/index.js'); - const handler = (server as any)._requestHandlers.get('tools/call'); + const handler = getToolsCallHandler(server); const response = await handler({ jsonrpc: '2.0', @@ -213,7 +252,7 @@ describe('search_codebase auto-heal', () => { ); const { server } = await import('../src/index.js'); - const handler = (server as any)._requestHandlers.get('tools/call'); + const handler = getToolsCallHandler(server); const response = await handler({ jsonrpc: '2.0', diff --git a/tests/search-compact-mode.test.ts b/tests/search-compact-mode.test.ts index c4d573c..9c0ca75 100644 --- a/tests/search-compact-mode.test.ts +++ b/tests/search-compact-mode.test.ts @@ -2,7 +2,9 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import { promises as fs } from 'fs'; import os from 'os'; import path from 'path'; +import { hashFileContent, writeManifest } from '../src/core/manifest.js'; import type { SearchResponse } from '../src/tools/types.js'; +import structuralContextFixture from './fixtures/structural-context-regressions.json'; import { CODEBASE_CONTEXT_DIRNAME, INDEX_FORMAT_VERSION, @@ -10,6 +12,7 @@ import { INDEX_META_VERSION, INTELLIGENCE_FILENAME, KEYWORD_INDEX_FILENAME, + MANIFEST_FILENAME, RELATIONSHIPS_FILENAME, VECTOR_DB_DIRNAME } from '../src/constants/codebase-context.js'; @@ -25,6 +28,13 @@ vi.mock('../src/core/search.js', async () => { async search(query: string, limit: number, filters?: unknown) { return searchMocks.search(query, limit, filters); } + + async searchWithCoverage(query: string, limit: number, filters?: unknown) { + return { + results: await this.search(query, limit, filters), + sourceCoverage: { status: 'current', checkedFiles: 0, skippedFiles: 0 } + }; + } } return { CodebaseSearcher }; @@ -125,7 +135,11 @@ describe('search_codebase compact/full mode', () => { primary: { name: 'Signals', frequency: '78%', - trend: 'Rising' + trend: 'Rising', + canonicalExample: { + file: 'src/auth/auth.service.ts', + snippet: 'export class AuthService { getToken() { return token; } }' + } } }, componentArchitecture: { @@ -287,7 +301,9 @@ describe('search_codebase compact/full mode', () => { // Test 4: Compact response includes budget, patternSummary, bestExample, nextHops it('compact response includes budget, patternSummary, bestExample, and nextHops', async () => { - searchMocks.search.mockResolvedValueOnce([makeResult()]); + searchMocks.search.mockResolvedValueOnce([ + makeResult({ filePath: path.join(tempRoot!, 'src/auth/auth.service.ts') }) + ]); const { server } = await import('../src/index.js'); const handler = ( @@ -310,14 +326,168 @@ describe('search_codebase compact/full mode', () => { const payload = parseSearchResponse(response.content[0].text); expect(payload.budget).toBeDefined(); expect(payload.budget?.mode).toBe('compact'); - // patternSummary should include pattern info from intelligence.json + // Only patterns backed by source in a returned result belong in this summary. expect(typeof payload.patternSummary).toBe('string'); expect(payload.patternSummary).toContain('Signals'); + expect(payload.patternSummary).not.toContain('Standalone Components'); // bestExample should be the top golden file expect(payload.bestExample).toBe('src/auth/auth.service.ts'); // nextHops should have at least 1 entry expect(Array.isArray(payload.nextHops)).toBe(true); expect(payload.nextHops?.length ?? 0).toBeGreaterThan(0); + expect(payload.nextHops?.some((hop) => hop.tool === 'read_file')).toBe(false); + expect(payload.nextHops?.[0].tool).toBe('get_code_relationships'); + }); + + it('does not recommend an unrelated golden example for a different source result', async () => { + searchMocks.search.mockResolvedValueOnce([ + makeResult({ filePath: path.join(tempRoot!, 'src/cli-init.ts') }) + ]); + const { server } = await import('../src/index.js'); + const handler = ( + server as unknown as { + _requestHandlers?: Map< + string, + (r: unknown) => Promise<{ content: Array<{ type: string; text: string }> }> + >; + } + )._requestHandlers?.get('tools/call'); + if (!handler) throw new Error('Expected tools/call handler'); + const response = await handler({ + jsonrpc: '2.0', + id: 1, + method: 'tools/call', + params: { name: 'search_codebase', arguments: { query: 'setup wizard' } } + }); + const payload = parseSearchResponse(response.content[0].text); + expect(payload.bestExample).toBeUndefined(); + expect(payload.nextHops?.some((hop) => hop.tool === 'read_file')).toBe(false); + }); + + it('omits stale pattern examples from summaries, advice, and evidence', async () => { + const intelligencePath = path.join(tempRoot!, CODEBASE_CONTEXT_DIRNAME, INTELLIGENCE_FILENAME); + const intelligence = JSON.parse(await fs.readFile(intelligencePath, 'utf-8')) as { + patterns: Record< + string, + { primary?: Record; alsoDetected?: Array> } + >; + }; + intelligence.patterns.stateManagement.primary = { + name: 'Signals', + frequency: '78%', + trend: 'Rising', + canonicalExample: { + file: 'src/auth/auth.service.ts', + snippet: 'export function obsoleteSignalStore() {}' + } + }; + intelligence.patterns.stateManagement.alsoDetected = [ + { + name: 'Legacy Store', + frequency: '22%', + trend: 'Declining', + canonicalExample: { + file: 'src/auth/auth.service.ts', + snippet: 'export function obsoleteLegacyStore() {}' + } + } + ]; + await fs.writeFile(intelligencePath, JSON.stringify(intelligence), 'utf-8'); + searchMocks.search.mockResolvedValueOnce([ + makeResult(), + makeResult({ filePath: 'src/auth/token.service.ts' }), + makeResult({ filePath: 'src/auth/session.service.ts' }) + ]); + + const { server } = await import('../src/index.js'); + const handler = ( + server as { + _requestHandlers?: Map< + string, + (r: unknown) => Promise<{ content: Array<{ type: string; text: string }> }> + >; + } + )._requestHandlers?.get('tools/call'); + if (!handler) throw new Error('Expected tools/call handler'); + + const response = await handler({ + jsonrpc: '2.0', + id: 1, + method: 'tools/call', + params: { + name: 'search_codebase', + arguments: { query: 'auth service', intent: 'edit' } + } + }); + const payload = parseSearchResponse(response.content[0].text); + expect(payload.patternSummary).toBeUndefined(); + expect(payload.preflight?.ready).toBe(false); + expect(payload.preflight?.patterns).toBeUndefined(); + }); + + it('keeps advice for a current canonical example and excludes unrelated patterns', async () => { + searchMocks.search.mockResolvedValueOnce([makeResult()]); + + const { server } = await import('../src/index.js'); + const handler = ( + server as { + _requestHandlers?: Map< + string, + (r: unknown) => Promise<{ content: Array<{ type: string; text: string }> }> + >; + } + )._requestHandlers?.get('tools/call'); + if (!handler) throw new Error('Expected tools/call handler'); + + const response = await handler({ + jsonrpc: '2.0', + id: 1, + method: 'tools/call', + params: { + name: 'search_codebase', + arguments: { query: 'auth service', intent: 'edit' } + } + }); + const payload = parseSearchResponse(response.content[0].text); + expect(payload.searchQuality?.status).toBe('ok'); + expect(payload.preflight?.patterns?.do).toContain('Signals'); + expect(payload.preflight?.patterns?.do).not.toContain('Standalone Components'); + expect(payload.preflight?.patterns?.do?.some((pattern) => pattern.includes('adoption'))).toBe( + false + ); + }); + + it('suppresses grounded pattern advice when search quality requires abstention', async () => { + searchMocks.search.mockResolvedValueOnce([ + makeResult({ score: 0.2 }), + makeResult({ filePath: 'src/auth/auth.service2.ts', score: 0.19 }), + makeResult({ filePath: 'src/auth/auth.service3.ts', score: 0.18 }) + ]); + + const { server } = await import('../src/index.js'); + const handler = ( + server as { + _requestHandlers?: Map< + string, + (r: unknown) => Promise<{ content: Array<{ type: string; text: string }> }> + >; + } + )._requestHandlers?.get('tools/call'); + if (!handler) throw new Error('Expected tools/call handler'); + + const response = await handler({ + jsonrpc: '2.0', + id: 1, + method: 'tools/call', + params: { + name: 'search_codebase', + arguments: { query: 'auth service', intent: 'edit' } + } + }); + const payload = parseSearchResponse(response.content[0].text); + expect(payload.searchQuality?.status).toBe('low_confidence'); + expect(payload.preflight?.abstain).toBe(true); + expect(payload.preflight?.patterns).toBeUndefined(); }); it('compact mode serializes symbol intelligence fields from chunk metadata', async () => { @@ -361,6 +531,109 @@ describe('search_codebase compact/full mode', () => { ); }); + it.each([ + { + id: 'test-companion-midchunk-line-is-supporting-context', + query: 'accountId', + context: [ + "import { parseInvoice } from './invoice-parser';", + "import { fixture } from './invoice-fixture';", + 'const options = {};' + ] + }, + { + id: 'declaration-companion-midchunk-line-is-supporting-context', + query: 'InvoiceParser parse', + context: [ + "import type { Invoice } from './invoice';", + "import type { ParserOptions } from './parser-options';", + 'const source = readSource();' + ] + } + ])('compact previews retain query-relevant source lines for $id', async (testCase) => { + const fixtureCase = structuralContextFixture.cases.find( + (entry) => entry.id === testCase.id + ); + if (!fixtureCase) throw new Error(`Missing structural fixture case: ${testCase.id}`); + const marker = fixtureCase.supportingOnlyMarkers?.[0]; + if (!marker) throw new Error(`Missing supporting source marker: ${testCase.id}`); + + searchMocks.search.mockResolvedValueOnce([ + makeResult({ snippet: [...testCase.context, fixtureCase.source].join('\n') }) + ]); + + const { server } = await import('../src/index.js'); + const handler = ( + server as { + _requestHandlers?: Map< + string, + (request: unknown) => Promise<{ content: Array<{ type: string; text: string }> }> + >; + } + )._requestHandlers?.get('tools/call'); + if (!handler) throw new Error('Expected tools/call handler'); + + const response = await handler({ + jsonrpc: '2.0', + id: 1, + method: 'tools/call', + params: { + name: 'search_codebase', + arguments: { query: testCase.query } + } + }); + const payload = parseSearchResponse(response.content[0].text); + const preview = payload.results[0]?.signaturePreview; + + expect(preview).toContain(marker); + expect((preview ?? '').split('\n').length).toBeLessThanOrEqual(3); + }); + + it('keeps compact preview lines contiguous around the strongest source match', async () => { + const fixtureCase = structuralContextFixture.cases.find( + (entry) => entry.id === 'method-body-has-interior-witness-and-sibling-control' + ); + if (!fixtureCase) throw new Error('Missing bounded method-body fixture case'); + + searchMocks.search.mockResolvedValueOnce([ + makeResult({ snippet: fixtureCase.source, startLine: 1, endLine: 12 }) + ]); + + const { server } = await import('../src/index.js'); + const handler = ( + server as { + _requestHandlers?: Map< + string, + (request: unknown) => Promise<{ content: Array<{ type: string; text: string }> }> + >; + } + )._requestHandlers?.get('tools/call'); + if (!handler) throw new Error('Expected tools/call handler'); + + const response = await handler({ + jsonrpc: '2.0', + id: 1, + method: 'tools/call', + params: { + name: 'search_codebase', + arguments: { query: 'beginImport finishImport saved' } + } + }); + const payload = parseSearchResponse(response.content[0].text); + const previewLines = (payload.results[0]?.signaturePreview ?? '') + .split('\n') + .map((line) => line.trim()); + const sourceLines = fixtureCase.source.split('\n').map((line) => line.trim()); + const previewStart = sourceLines.indexOf(previewLines[0] ?? ''); + + expect(previewLines).toContain('finishImport(saved);'); + expect(previewLines.length).toBeGreaterThan(0); + expect(previewLines.length).toBeLessThanOrEqual(3); + expect(sourceLines.slice(previewStart, previewStart + previewLines.length)).toEqual( + previewLines + ); + }); + it('adds an exact tokenEstimate advisory to compact responses', async () => { searchMocks.search.mockResolvedValueOnce([makeResult()]); @@ -517,64 +790,78 @@ describe('search_codebase compact/full mode', () => { expect(payload.results[0].complexity).toBe(7); }); - it('real CodebaseSearcher preserves chunk imports and exports', async () => { - if (!tempRoot) throw new Error('tempRoot not initialized'); - - const ctxDir = path.join(tempRoot, CODEBASE_CONTEXT_DIRNAME); - const actualChunk = { - id: 'auth-chunk', - content: - 'import { tokenStore } from "./token-store";\nexport class AuthService {\n getToken() { return tokenStore.read(); }\n}\nexport const AUTH_TOKEN = "auth";', - filePath: path.join(tempRoot, 'src', 'auth', 'auth.service.ts'), - relativePath: 'src/auth/auth.service.ts', - startLine: 1, - endLine: 5, - language: 'ts', - dependencies: [], - imports: [ - 'src/auth/token-store.ts', - 'src/auth/session.ts', - 'src/shared/logger.ts', - 'src/config/env.ts', - 'src/http/client.ts' - ], - exports: ['AuthService', 'AUTH_TOKEN'], - tags: ['service'], - metadata: { - className: 'AuthService', - symbolAware: true, - symbolName: 'AuthService', - symbolKind: 'class' - } - }; - - await fs.writeFile( - path.join(ctxDir, KEYWORD_INDEX_FILENAME), - JSON.stringify( - { - header: { buildId: 'test-build-compact', formatVersion: INDEX_FORMAT_VERSION }, - chunks: [actualChunk] - }, - null, - 2 - ), - 'utf-8' - ); - - const actualSearchModule = - await vi.importActual('../src/core/search.js'); - const searcher = new actualSearchModule.CodebaseSearcher(tempRoot); - const results = await searcher.search('AuthService token', 5, undefined, { - useSemanticSearch: false, - useKeywordSearch: true, - enableReranker: false - }); + it( + 'real CodebaseSearcher preserves chunk imports and exports', + async () => { + if (!tempRoot) throw new Error('tempRoot not initialized'); + + const ctxDir = path.join(tempRoot, CODEBASE_CONTEXT_DIRNAME); + const actualChunk = { + id: 'auth-chunk', + content: + 'import { tokenStore } from "./token-store";\nexport class AuthService {\n getToken() { return tokenStore.read(); }\n}\nexport const AUTH_TOKEN = "auth";', + filePath: path.join(tempRoot, 'src', 'auth', 'auth.service.ts'), + relativePath: 'src/auth/auth.service.ts', + startLine: 1, + endLine: 5, + language: 'ts', + dependencies: [], + imports: [ + 'src/auth/token-store.ts', + 'src/auth/session.ts', + 'src/shared/logger.ts', + 'src/config/env.ts', + 'src/http/client.ts' + ], + exports: ['AuthService', 'AUTH_TOKEN'], + tags: ['service'], + metadata: { + className: 'AuthService', + symbolAware: true, + symbolName: 'AuthService', + symbolKind: 'class' + } + }; + + await fs.writeFile( + path.join(ctxDir, KEYWORD_INDEX_FILENAME), + JSON.stringify( + { + header: { buildId: 'test-build-compact', formatVersion: INDEX_FORMAT_VERSION }, + chunks: [actualChunk] + }, + null, + 2 + ), + 'utf-8' + ); - expect(results).toHaveLength(1); - expect(results[0].filePath).toBe(actualChunk.filePath); - expect(results[0].imports).toEqual(actualChunk.imports); - expect(results[0].exports).toEqual(actualChunk.exports); - }, SLOW_WINDOWS_TEST_TIMEOUT_MS); + // This real-search fixture must represent actual indexed source. Older setup + // only wrote cached chunks, which cannot establish current-source validity. + await fs.mkdir(path.dirname(actualChunk.filePath), { recursive: true }); + await fs.writeFile(actualChunk.filePath, actualChunk.content, 'utf-8'); + await writeManifest(path.join(ctxDir, MANIFEST_FILENAME), { + version: 1, + generatedAt: new Date().toISOString(), + files: { [actualChunk.relativePath]: hashFileContent(actualChunk.content) } + }); + + const actualSearchModule = + await vi.importActual('../src/core/search.js'); + const searcher = new actualSearchModule.CodebaseSearcher(tempRoot); + const results = await searcher.search('AuthService token', 5, undefined, { + useSemanticSearch: false, + useKeywordSearch: true, + enableReranker: false + }); + + expect(results).toHaveLength(1); + expect(results[0].filePath).toBe(actualChunk.filePath); + expect(results[0].imports).toEqual(actualChunk.imports); + expect(results[0].exports).toEqual(actualChunk.exports); + }, + SLOW_WINDOWS_TEST_TIMEOUT_MS + ); it('adds a warning only when the final full payload exceeds the compact budget threshold', async () => { const oversizedSummary = 'Token-heavy summary '.repeat(1200); diff --git a/tests/search-decision-card.test.ts b/tests/search-decision-card.test.ts index c6d77ae..66bc05a 100644 --- a/tests/search-decision-card.test.ts +++ b/tests/search-decision-card.test.ts @@ -263,7 +263,7 @@ export class ProfileService { } }, SLOW_WINDOWS_TEST_TIMEOUT_MS); - it('intent="explore" returns lightweight preflight', async () => { + it('intent="explore" omits edit readiness', async () => { if (!tempRoot) throw new Error('tempRoot not initialized'); const { server } = await import('../src/index.js'); @@ -284,14 +284,7 @@ export class ProfileService { const content = response.content[0]; const parsed = JSON.parse(content.text) as SearchToolPayload; - const preflight = parsed.preflight; - - // For explore intent, preflight should be lite: { ready, reason? } - if (preflight) { - expect(preflight.ready).toBeDefined(); - expect(typeof preflight.ready).toBe('boolean'); - // Should NOT have full decision card fields for explore - } + expect(parsed.preflight).toBeUndefined(); }, SLOW_WINDOWS_TEST_TIMEOUT_MS); it('includes snippet field when includeSnippets=true', async () => { diff --git a/tests/search-edit-preflight-lite.test.ts b/tests/search-edit-preflight-lite.test.ts index 5ee335e..8435e2f 100644 --- a/tests/search-edit-preflight-lite.test.ts +++ b/tests/search-edit-preflight-lite.test.ts @@ -12,10 +12,48 @@ import { VECTOR_DB_DIRNAME } from '../src/constants/codebase-context.js'; +type SourceCoverageMock = { + status: 'current' | 'partial' | 'no_fresh_results'; + checkedFiles: number; + skippedFiles: number; + refreshAction?: string; +}; + +type ToolCallRequest = { + jsonrpc: '2.0'; + id: number; + method: 'tools/call'; + params: { name: string; arguments: Record }; +}; + +type ToolCallResponse = { content: Array<{ type: string; text: string }> }; + +type SearchPayload = { + status: string; + searchQuality?: { status: string; confidence: number }; + sourceCoverage?: SourceCoverageMock; + preflight?: { ready: boolean; abstain?: boolean; reason?: string }; +}; + const searchMocks = vi.hoisted(() => ({ - search: vi.fn() + search: vi.fn(), + sourceCoverage: vi.fn((): SourceCoverageMock => ({ + status: 'current', + checkedFiles: 0, + skippedFiles: 0 + })) })); +function getToolCallHandler( + server: unknown +): (request: ToolCallRequest) => Promise { + const handlers = (server as { _requestHandlers?: unknown })._requestHandlers; + if (!(handlers instanceof Map)) throw new Error('Expected server._requestHandlers to be a Map'); + const handler = handlers.get('tools/call'); + if (typeof handler !== 'function') throw new Error('Expected tools/call handler'); + return handler as (request: ToolCallRequest) => Promise; +} + vi.mock('../src/core/search.js', async () => { class CodebaseSearcher { constructor(_rootPath: string) {} @@ -23,6 +61,13 @@ vi.mock('../src/core/search.js', async () => { async search(query: string, limit: number, filters?: unknown) { return searchMocks.search(query, limit, filters); } + + async searchWithCoverage(query: string, limit: number, filters?: unknown) { + return { + results: await this.search(query, limit, filters), + sourceCoverage: searchMocks.sourceCoverage() + }; + } } return { CodebaseSearcher }; @@ -35,6 +80,11 @@ describe('search_codebase preflight', () => { beforeEach(async () => { searchMocks.search.mockReset(); + searchMocks.sourceCoverage.mockReset().mockReturnValue({ + status: 'current', + checkedFiles: 0, + skippedFiles: 0 + }); vi.resetModules(); originalArgv = [...process.argv]; @@ -122,7 +172,7 @@ describe('search_codebase preflight', () => { } }); - it('returns preflight with ready flag when intent is omitted (intelligence present)', async () => { + it('omits edit readiness for discovery when intent is omitted', async () => { searchMocks.search.mockResolvedValueOnce([ { summary: 'Auth interceptor implementation', @@ -137,7 +187,7 @@ describe('search_codebase preflight', () => { ]); const { server } = await import('../src/index.js'); - const handler = (server as any)._requestHandlers.get('tools/call'); + const handler = getToolCallHandler(server); const response = await handler({ jsonrpc: '2.0', @@ -151,11 +201,51 @@ describe('search_codebase preflight', () => { } }); - const payload = JSON.parse(response.content[0].text); + const payload = JSON.parse(response.content[0].text) as SearchPayload; expect(payload.status).toBe('success'); - // Preflight is flattened to { ready, reason } - expect(payload.preflight).toBeTruthy(); - expect(typeof payload.preflight.ready).toBe('boolean'); + expect(payload.searchQuality).toBeDefined(); + expect(payload.preflight).toBeUndefined(); + }); + + it('keeps currentness and retrieval uncertainty visible for explicit exploration', async () => { + searchMocks.search.mockResolvedValueOnce([ + { + summary: 'Local cache recovery path', + snippet: 'reloadCache() { return readSnapshot(); }', + filePath: 'src/cache/recovery.ts', + startLine: 14, + endLine: 19, + score: 0.05, + language: 'ts', + metadata: {} + } + ]); + searchMocks.sourceCoverage.mockReturnValueOnce({ + status: 'partial', + checkedFiles: 2, + skippedFiles: 1, + refreshAction: 'Refresh the index to inspect remaining candidates.' + }); + + const { server } = await import('../src/index.js'); + const handler = getToolCallHandler(server); + const response = await handler({ + jsonrpc: '2.0', + id: 2, + method: 'tools/call', + params: { + name: 'search_codebase', + arguments: { query: 'cache recovery', intent: 'explore' } + } + }); + + const payload = JSON.parse(response.content[0].text) as SearchPayload; + expect(payload.sourceCoverage?.status).toBe('partial'); + expect(payload.sourceCoverage?.refreshAction).toBe( + 'Refresh the index to inspect remaining candidates.' + ); + expect(payload.searchQuality?.status).toBeDefined(); + expect(payload.preflight).toBeUndefined(); }); it('returns preflight with ready flag when intent="edit"', async () => { @@ -173,7 +263,7 @@ describe('search_codebase preflight', () => { ]); const { server } = await import('../src/index.js'); - const handler = (server as any)._requestHandlers.get('tools/call'); + const handler = getToolCallHandler(server); const response = await handler({ jsonrpc: '2.0', @@ -188,10 +278,49 @@ describe('search_codebase preflight', () => { } }); - const payload = JSON.parse(response.content[0].text); + const payload = JSON.parse(response.content[0].text) as SearchPayload; expect(payload.status).toBe('success'); - // Preflight is flattened to { ready, reason } expect(payload.preflight).toBeTruthy(); - expect(typeof payload.preflight.ready).toBe('boolean'); + expect(typeof payload.preflight?.ready).toBe('boolean'); + }); + + it('keeps the explicit edit gate closed when source coverage is partial', async () => { + searchMocks.search.mockResolvedValueOnce([ + { + summary: 'Local cache recovery path', + snippet: 'reloadCache() { return readSnapshot(); }', + filePath: 'src/cache/recovery.ts', + startLine: 14, + endLine: 19, + score: 0.05, + language: 'ts', + metadata: {} + } + ]); + searchMocks.sourceCoverage.mockReturnValueOnce({ + status: 'partial', + checkedFiles: 2, + skippedFiles: 1, + refreshAction: 'Refresh the index to inspect remaining candidates.' + }); + + const { server } = await import('../src/index.js'); + const handler = getToolCallHandler(server); + const response = await handler({ + jsonrpc: '2.0', + id: 3, + method: 'tools/call', + params: { + name: 'search_codebase', + arguments: { query: 'cache recovery', intent: 'edit' } + } + }); + + const payload = JSON.parse(response.content[0].text) as SearchPayload; + expect(payload.preflight?.ready).toBe(false); + expect(payload.preflight?.abstain).toBe(true); + expect(payload.preflight?.reason).toBe( + 'Refresh the index to inspect remaining candidates.' + ); }); }); diff --git a/tests/search-health-scope.test.ts b/tests/search-health-scope.test.ts index 0e175bf..4fd1acb 100644 --- a/tests/search-health-scope.test.ts +++ b/tests/search-health-scope.test.ts @@ -16,6 +16,13 @@ vi.mock('../src/core/search.js', async () => { async search(query: string, limit: number, filters?: unknown) { return searchMocks.search(query, limit, filters); } + + async searchWithCoverage(query: string, limit: number, filters?: unknown) { + return { + results: await this.search(query, limit, filters), + sourceCoverage: { status: 'current', checkedFiles: 0, skippedFiles: 0 } + }; + } } return { CodebaseSearcher }; diff --git a/tests/search-memory-scope.test.ts b/tests/search-memory-scope.test.ts new file mode 100644 index 0000000..9b544c5 --- /dev/null +++ b/tests/search-memory-scope.test.ts @@ -0,0 +1,124 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { promises as fs } from 'fs'; +import os from 'os'; +import path from 'path'; +import { handle } from '../src/tools/search-codebase.js'; +import type { ToolContext } from '../src/tools/types.js'; + +const searchMocks = vi.hoisted(() => ({ search: vi.fn() })); + +vi.mock('../src/core/search.js', async () => { + class CodebaseSearcher { + constructor(_rootPath: string) {} + + async search(query: string, limit: number, filters?: unknown) { + return searchMocks.search(query, limit, filters); + } + + async searchWithCoverage(query: string, limit: number, filters?: unknown) { + return { + results: await this.search(query, limit, filters), + sourceCoverage: { status: 'current', checkedFiles: 1, skippedFiles: 0 } + }; + } + } + + return { CodebaseSearcher }; +}); + +describe('search_codebase memory relevance at the tool seam', () => { + let tempRoot: string; + let ctx: ToolContext; + + beforeEach(async () => { + searchMocks.search.mockReset(); + tempRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'search-memory-scope-')); + const contextDir = path.join(tempRoot, '.codebase-context'); + await fs.mkdir(contextDir, { recursive: true }); + + ctx = { + indexState: { status: 'ready' }, + paths: { + baseDir: contextDir, + memory: path.join(contextDir, 'memory.json'), + intelligence: path.join(contextDir, 'intelligence.json'), + keywordIndex: path.join(contextDir, 'index.json'), + vectorDb: path.join(contextDir, 'index') + }, + rootPath: tempRoot, + performIndexing: () => undefined + }; + + const date = new Date().toISOString(); + await fs.writeFile( + ctx.paths.memory, + JSON.stringify([ + { + id: 'auth-path-only', + type: 'decision', + category: 'architecture', + memory: 'Prefer React Signals for local view state', + reason: 'Keep the widget state independent of network retry behavior.', + date, + scope: { kind: 'file', file: 'src/auth/auth.service.ts' } + }, + { + id: 'cross-file-auth-decision', + type: 'decision', + category: 'architecture', + memory: 'Use AuthService for CLI token bootstrap', + reason: 'Keep CLI setup on the shared token refresh path.', + date, + scope: { + kind: 'symbol', + file: 'src/auth/auth.service.ts', + symbol: 'AuthService' + } + } + ]), + 'utf-8' + ); + await fs.writeFile( + ctx.paths.intelligence, + JSON.stringify({ + header: { buildId: 'scope-test-build', formatVersion: 1 }, + generatedAt: date, + patterns: {}, + goldenFiles: [] + }), + 'utf-8' + ); + }); + + afterEach(async () => { + await fs.rm(tempRoot, { recursive: true, force: true }); + }); + + it('excludes path-only auth memories while retaining a relevant cross-file decision', async () => { + searchMocks.search.mockResolvedValueOnce([ + { + summary: 'Auth CLI bootstrap', + snippet: 'export function bootstrapAuthCli() { return authService.refreshToken(); }', + filePath: 'src/cli/bootstrap.ts', + startLine: 1, + endLine: 4, + score: 0.92, + language: 'ts', + metadata: { symbolName: 'bootstrapAuthCli', symbolKind: 'function' }, + relevanceReason: 'Matches auth CLI setup' + } + ]); + + const response = await handle({ query: 'auth CLI', intent: 'edit', mode: 'full' }, ctx); + const payload = JSON.parse(response.content?.[0]?.text ?? '{}') as { + relatedMemories?: string[]; + }; + + expect(payload.relatedMemories).toContainEqual( + expect.stringContaining('Use AuthService for CLI token bootstrap') + ); + expect(payload.relatedMemories).not.toContainEqual( + expect.stringContaining('Prefer React Signals for local view state') + ); + }); +}); diff --git a/tests/search-model-free-fallback.test.ts b/tests/search-model-free-fallback.test.ts new file mode 100644 index 0000000..b1aaa6e --- /dev/null +++ b/tests/search-model-free-fallback.test.ts @@ -0,0 +1,816 @@ +import { describe, expect, it } from 'vitest'; +import { createHash } from 'node:crypto'; +import { promises as fs } from 'node:fs'; +import path from 'node:path'; +import Fuse from 'fuse.js'; +import type { CodeChunk, SearchFilters } from '../src/types/index.js'; +import { CodebaseSearcher } from '../src/core/search.js'; +import { assessSearchQuality } from '../src/core/search-quality.js'; +import { GenericAnalyzer } from '../src/analyzers/generic/index.js'; +import { CodebaseIndexer } from '../src/core/indexer.js'; +import { analyzerRegistry } from '../src/core/analyzer-registry.js'; +import { InternalFileGraph } from '../src/utils/usage-tracker.js'; +import { + InternalSourceResolver, + loadTsconfigCompilerOptions +} from '../src/utils/internal-source-resolver.js'; +import { + CODEBASE_CONTEXT_DIRNAME, + INTELLIGENCE_FILENAME, + MANIFEST_FILENAME +} from '../src/constants/codebase-context.js'; +import { asSearcherTestHarness, materializeSourceSnapshot } from './helpers/source-snapshot.js'; + +function chunk(filePath: string, content: string, symbolName?: string): CodeChunk { + return { + id: filePath, + content, + filePath, + relativePath: filePath, + startLine: 1, + endLine: content.split('\n').length, + language: 'typescript', + framework: 'generic', + componentType: 'module', + dependencies: [], + imports: [], + exports: [], + tags: [], + metadata: symbolName ? { symbolName, componentName: symbolName } : {} + }; +} + +async function searchFixture(chunks: CodeChunk[], withImportGraph = false) { + const snapshot = await materializeSourceSnapshot(chunks, 'cbc-model-free-query-'); + const searcher = new CodebaseSearcher(snapshot.rootPath); + const harness = asSearcherTestHarness(searcher) as ReturnType & { + chunks: CodeChunk[]; + loadPatternIntelligence: () => Promise; + semanticSearch: ( + query: string, + limit: number, + filters?: SearchFilters + ) => Promise>; + }; + harness.initialized = true; + harness.sourceManifest = snapshot.manifest; + harness.chunks = chunks; + harness.embeddingProvider = null; + harness.storageProvider = null; + harness.fuseIndex = new Fuse(chunks, { + keys: [ + { name: 'content', weight: 0.4 }, + { name: 'metadata.componentName', weight: 0.25 }, + { name: 'filePath', weight: 0.15 }, + { name: 'relativePath', weight: 0.15 }, + { name: 'componentType', weight: 0.15 }, + { name: 'layer', weight: 0.1 }, + { name: 'tags', weight: 0.15 } + ], + includeScore: true, + threshold: 0.4, + useExtendedSearch: true, + ignoreLocation: true + }); + // Use the same analyzer, resolver and graph serializer as indexing. Text in + // comments/strings and unproven chunk.imports must not become graph edges. + const graph = new InternalFileGraph(snapshot.rootPath); + const reloadGraph = async (data: ReturnType) => { + await fs.writeFile( + path.join(snapshot.rootPath, CODEBASE_CONTEXT_DIRNAME, INTELLIGENCE_FILENAME), + JSON.stringify({ internalFileGraph: data }) + ); + await harness.loadPatternIntelligence(); + }; + if (withImportGraph) { + const files = [...new Set(chunks.map((item) => item.filePath))]; + const resolver = await InternalSourceResolver.create(snapshot.rootPath, files, { + compilerOptions: { + ...loadTsconfigCompilerOptions(snapshot.rootPath), + resolveJsonModule: true + } + }); + const analyzer = new GenericAnalyzer(); + for (const file of files.filter((file) => file.endsWith('.ts'))) { + const content = await fs.readFile(file, 'utf8'); + const analysis = await analyzer.analyze(file, content); + for (const imported of analysis.imports) { + const target = await resolver.resolve(file, imported.source); + if (target.status !== 'resolved' || !target.path) continue; + graph.trackImport(file, target.path, imported.line, imported.imports, { + kind: 'import', + specifier: imported.source, + sourceFingerprint: createHash('sha256').update(content).digest('hex'), + targetFingerprint: createHash('sha256') + .update(await fs.readFile(target.path)) + .digest('hex') + }); + } + } + await reloadGraph(graph.toJSON()); + } + const search = (query: string, filters?: SearchFilters) => + searcher.search(query, 5, filters, { + useSemanticSearch: false, + enableQueryExpansion: false, + enableLowConfidenceRescue: false, + enableReranker: false + }); + const searchWithSemantic = (query: string) => + searcher.search(query, 5, undefined, { + useSemanticSearch: true, + useKeywordSearch: true, + enableQueryExpansion: false, + enableLowConfidenceRescue: false, + enableReranker: false + }); + return { + rootPath: snapshot.rootPath, + search, + searchWithSemantic, + searcher: harness, + graph: graph.toJSON(), + reloadGraph, + cleanup: snapshot.cleanup + }; +} + +describe('model-free query fallback', () => { + it.each(['ParcelApiClient', 'parcelapiclient', 'Where is ParcelApiClient retry policy?'])( + 'keeps an exact declared class ahead of a same-suffix consumer for %s', + async (query) => { + const declaration = chunk( + 'src/parcels.ts', + 'export class ParcelApiClient { retryPolicy = "bounded"; }' + ); + declaration.componentType = 'unknown'; + declaration.metadata = { componentName: 'ParcelApiClient' }; + const consumer = chunk( + 'src/ParcelApiClient-consumer.ts', + 'export class CachedParcelApiClient { client: ParcelApiClient; retryPolicy = "cached"; }' + ); + consumer.metadata = { componentName: 'CachedParcelApiClient' }; + consumer.componentType = 'service'; + consumer.layer = 'business'; + const fixture = await searchFixture([consumer, declaration]); + try { + const results = await fixture.search(query); + expect(results[0]?.filePath).toContain('parcels.ts'); + const exact = await fixture.search('ParcelApiClient'); + expect(assessSearchQuality('ParcelApiClient', exact).status).toBe('ok'); + expect(await fixture.search('QuantumPineappleService')).toEqual([]); + } finally { + await fixture.cleanup(); + } + } + ); + + it('keeps a re-export mention below its indexed declaration for an exact class query', async () => { + const declaration = chunk( + 'src/parcels.ts', + 'export class ParcelApiClient { retryPolicy = "bounded"; }' + ); + declaration.metadata = { componentName: 'ParcelApiClient' }; + declaration.componentType = 'unknown'; + const barrel = chunk( + 'src/ParcelApiClient.ts', + 'export { ParcelApiClient } from "./parcels.js";' + ); + barrel.exports = ['ParcelApiClient']; + barrel.componentType = 'service'; + barrel.layer = 'business'; + const fixture = await searchFixture([barrel, declaration]); + try { + expect((await fixture.search('ParcelApiClient'))[0]?.filePath).toContain('parcels.ts'); + } finally { + await fixture.cleanup(); + } + }); + + it('retrieves an inflected action in a source declaration ahead of incidental configuration words', async () => { + const fixture = await searchFixture([ + chunk( + 'src/order-batch.ts', + 'export function batchOrders(orders: string[]) { return orders.map(order => [order]); }', + 'batchOrders' + ), + chunk( + 'jest.config.ts', + 'export default { displayName: "orders", coverageDirectory: "coverage/order-processing" };' + ) + ]); + try { + const results = await fixture.search('How are orders batched for processing?'); + expect(results[0]?.filePath).toContain('order-batch.ts'); + } finally { + await fixture.cleanup(); + } + }); + + it('uses the same identifier boundaries for natural questions and Python snake_case source', async () => { + const implementation = chunk( + 'billing/archive.py', + 'def archive_invoices(records):\n return [record for record in records if record.paid]', + 'archive_invoices' + ); + implementation.language = 'python'; + const fixture = await searchFixture([ + implementation, + chunk('src/invoice.ts', 'export const invoices = [];') + ]); + try { + const results = await fixture.search('Where are invoices archived before deletion?', { + language: 'python' + }); + expect(results[0]?.filePath).toContain('archive.py'); + expect(await fixture.search('Where are photons archived before teleportation?')).toEqual([]); + } finally { + await fixture.cleanup(); + } + }); + + it('does not let fuzzy full-question configuration hits hide grounded declaration evidence', async () => { + const fixture = await searchFixture([ + chunk( + 'src/channel.ts', + 'export function setAudioVolume(volume: number) { return volume; }', + 'setAudioVolume' + ), + chunk( + 'tsconfig.json', + '{ "compilerOptions": { "allowJs": true }, "adjust": "auditor", "volume": "logs" }' + ) + ]); + try { + const results = await fixture.search('adjust audio volume'); + expect(results[0]?.filePath).toContain('channel.ts'); + const exact = await fixture.search('setAudioVolume'); + expect(exact[0]?.filePath).toContain('channel.ts'); + } finally { + await fixture.cleanup(); + } + }); + + it('does not count inflections of one clue as independent corroboration', async () => { + const fixture = await searchFixture([ + chunk('src/archive.ts', 'export function archive() { return true; }', 'archive'), + chunk('src/organization.ts', 'export const organization = "archive";') + ]); + try { + expect(await fixture.search('Where are archives archived holographically?')).toEqual([]); + expect(await fixture.search('Where are organic organs archived?')).toEqual([]); + } finally { + await fixture.cleanup(); + } + }); + + it('keeps concrete declaration relevance above unrelated component and layer labels', async () => { + const declaration = chunk( + 'src/budget.ts', + 'export function calculateRetryBudget() { return 3; }', + 'calculateRetryBudget' + ); + declaration.componentType = 'unknown'; + const mention = chunk( + 'src/controller.ts', + 'export class Controller { label = "retry budget"; }', + 'Controller' + ); + mention.componentType = 'service'; + mention.layer = 'business'; + const fixture = await searchFixture([mention, declaration]); + try { + expect((await fixture.search('How is the retry budget calculated?'))[0]?.filePath).toContain( + 'budget.ts' + ); + } finally { + await fixture.cleanup(); + } + }); + + it.each([ + { + file: 'jobs/batch.ts', + source: + 'export function batchOrders(orders: string[]) { return orders.map(order => [order]); }', + query: 'How are orders batched for processing?', + exact: 'batchOrders' + }, + { + file: 'billing/archive.py', + source: + 'def archive_invoices(records):\n return [record for record in records if record.paid]', + query: 'Where are invoices archived before deletion?', + exact: 'archive_invoices' + } + ])('retrieves real GenericAnalyzer chunks for $file', async ({ file, source, query, exact }) => { + const analyzer = new GenericAnalyzer(); + const analysis = await analyzer.analyze(path.resolve(file), `${source}\n`); + for (const item of analysis.chunks) { + item.relativePath = file; + item.filePath = file; + } + expect(analysis.chunks.length).toBeGreaterThan(0); + const fixture = await searchFixture(analysis.chunks); + try { + expect((await fixture.search(query))[0]?.filePath).toContain(path.basename(file)); + expect((await fixture.search(exact))[0]?.filePath).toContain(path.basename(file)); + } finally { + await fixture.cleanup(); + } + }); + + it('uses default model-free indexing and search for inflected TypeScript and Python questions', async () => { + const sources = [ + chunk( + 'jobs/batch.ts', + 'export function batchOrders(orders: string[]) { return orders.map(order => [order]); }' + ), + chunk( + 'billing/archive.py', + 'def archive_invoices(records):\n return [record for record in records if record.paid]' + ) + ]; + const snapshot = await materializeSourceSnapshot(sources, 'cbc-model-free-indexed-'); + analyzerRegistry.register(new GenericAnalyzer()); + try { + // The helper's manifest supports in-memory tests; this case starts from + // ordinary unindexed source and lets the real indexer own all artifacts. + await fs.unlink(path.join(snapshot.rootPath, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME)); + await fs.rmdir(path.join(snapshot.rootPath, CODEBASE_CONTEXT_DIRNAME)); + const indexer = new CodebaseIndexer({ rootPath: snapshot.rootPath }); + await indexer.index(); + const searcher = new CodebaseSearcher(snapshot.rootPath); + const typescript = await searcher.search('How are orders batched for processing?'); + const python = await searcher.search('Where are invoices archived before deletion?'); + expect(typescript[0]?.filePath).toContain('batch.ts'); + expect(python[0]?.filePath).toContain('archive.py'); + expect((await searcher.search('batchOrders'))[0]?.filePath).toContain('batch.ts'); + expect((await searcher.search('archive_invoices'))[0]?.filePath).toContain('archive.py'); + } finally { + await snapshot.cleanup(); + } + }); + + it('replenishes a saturated rescue evidence window after one source with many chunks changes', async () => { + const stale = Array.from({ length: 800 }, (_, index) => ({ + ...chunk('src/billing.ts', 'export const billing = { invoice: true, refund: true };'), + id: `billing-${index}` + })); + const support = chunk( + 'src/support.ts', + 'export const support = { invoice: true, refund: true };' + ); + const fixture = await searchFixture([...stale, support]); + try { + await fs.writeFile(stale[0].filePath, 'export const changed = true;\n'); + const matches = await fixture.search('billing invoice refund scheduling'); + expect(matches.map((result) => path.basename(result.filePath))).toEqual(['support.ts']); + expect(matches[0].snippet).toContain('support'); + } finally { + await fixture.cleanup(); + } + }); + + it('recovers separate source and policy files without requiring a manually written OR query', async () => { + const fixture = await searchFixture( + [ + chunk( + 'src/target.ts', + 'import { REQUIRED_POLICY } from "./policy.js";\nexport function resolvePilotPolicy(kind: string) { return kind + REQUIRED_POLICY; }', + 'resolvePilotPolicy' + ), + chunk('src/policy.ts', 'export const REQUIRED_POLICY = "review-before-deploy";') + ], + true + ); + try { + const results = await fixture.search('resolvePilotPolicy review-before-deploy'); + expect(results.map((result) => result.filePath.split(/[\\/]/).pop())).toEqual( + expect.arrayContaining(['target.ts', 'policy.ts']) + ); + const explicitOr = await fixture.search('resolvePilotPolicy | review-before-deploy'); + expect(explicitOr.map((result) => result.filePath.split(/[\\/]/).pop())).toEqual( + expect.arrayContaining(['target.ts', 'policy.ts']) + ); + } finally { + await fixture.cleanup(); + } + }); + + it('finds an implementation from an ordinary negative-limit suggestion question', async () => { + const fixture = await searchFixture([ + chunk( + 'packages/nx/src/utils/string-similarity.ts', + 'Returns up to limit candidates. Candidates not similar enough to input are excluded from suggestions.\nexport function findClosestMatches(input: string, candidates: string[], limit = 3) { return candidates.slice(0, limit); }', + 'findClosestMatches' + ), + chunk('packages/nx/src/utils/worker-limit.ts', 'export const workerLimit = 100;'), + chunk( + 'packages/nx/src/utils/catalog-helper.ts', + 'export function formatSuggestions() { return "suggestions"; }' + ) + ]); + try { + const results = await fixture.search( + 'Why do negative limits still yield multiple Nx typo suggestions?' + ); + expect(results[0]?.filePath).toContain('string-similarity.ts'); + expect( + assessSearchQuality( + 'Why do negative limits still yield multiple Nx typo suggestions?', + results + ).status + ).toBe('low_confidence'); + } finally { + await fixture.cleanup(); + } + }); + + it('does not manufacture matches from irrelevant text or weaken exact symbol search', async () => { + const fixture = await searchFixture([ + chunk( + 'src/session.ts', + 'export function authenticateUser() { return "token"; }', + 'authenticateUser' + ), + chunk('src/policy.ts', 'export const SESSION_POLICY = "login-required";') + ]); + try { + expect(await fixture.search('holographic pineapple zebra quantum')).toEqual([]); + const exact = await fixture.search('authenticateUser'); + expect(exact[0]?.filePath).toContain('session.ts'); + } finally { + await fixture.cleanup(); + } + }); + + it('does not promote unrelated single-term hits into a cross-file answer', async () => { + const fixture = await searchFixture([ + chunk('src/green.ts', 'export const hydroponics = 1;'), + chunk('src/red.ts', 'export const transcendence = 1;') + ]); + try { + expect(await fixture.search('hydroponics transcendence')).toEqual([]); + expect(await fixture.search('hydroponics transcendence', { language: 'python' })).toEqual([]); + } finally { + await fixture.cleanup(); + } + }); + + it('respects filters when the fallback has enough supporting words', async () => { + const fixture = await searchFixture([ + chunk('src/policy.ts', 'export const deploymentPolicy = "review before deploy";') + ]); + try { + expect(await fixture.search('review deploy', { language: 'python' })).toEqual([]); + expect((await fixture.search('review deploy')).at(0)?.filePath).toContain('policy.ts'); + } finally { + await fixture.cleanup(); + } + }); + + it('combines evidence from separate chunks in one source file without repeating that file', async () => { + const first = chunk('src/combined.ts', 'export const hydroponics = true;'); + first.id = 'combined-first'; + const second = chunk('src/combined.ts', 'export const transcendence = true;'); + second.id = 'combined-second'; + second.startLine = 2; + second.endLine = 2; + const fixture = await searchFixture([first, second]); + try { + const results = await fixture.search('hydroponics transcendence'); + expect(results).toHaveLength(1); + expect(results[0].filePath).toContain('combined.ts'); + } finally { + await fixture.cleanup(); + } + }); + + it('rejects an exact symbol when all the other substantive query terms are unsupported', async () => { + const fixture = await searchFixture([ + chunk( + 'src/auth.ts', + 'export function authenticateUser() { return true; }', + 'authenticateUser' + ) + ]); + try { + expect(await fixture.search('authenticateUser holographic pineapple zebra quantum')).toEqual( + [] + ); + expect((await fixture.search('authenticateUser'))[0]?.filePath).toContain('auth.ts'); + } finally { + await fixture.cleanup(); + } + }); + + it('does not corroborate an exact symbol using evidence from an unrelated file', async () => { + const fixture = await searchFixture([ + chunk( + 'src/auth.ts', + 'export function authenticateUser() { return true; }', + 'authenticateUser' + ), + chunk('src/billing.ts', 'export const billing = { invoice: "paid", refund: "pending" };') + ]); + try { + const matches = await fixture.search('authenticateUser billing invoice refund'); + expect(matches.map((result) => result.filePath.split(/[\\/]/).pop())).toEqual(['billing.ts']); + } finally { + await fixture.cleanup(); + } + }); + + it('does not treat a longer indexed symbol as an exact query symbol anchor', async () => { + const fixture = await searchFixture( + [ + chunk( + 'src/auth.ts', + 'import "./billing.js";\nexport function authenticateUserAdmin() { return true; }', + 'authenticateUserAdmin' + ), + chunk('src/billing.ts', 'export const billing = { invoice: "paid", refund: "pending" };') + ], + true + ); + try { + const matches = await fixture.search('authenticateUser billing invoice refund'); + expect(matches.map((result) => path.basename(result.filePath))).toEqual(['billing.ts']); + } finally { + await fixture.cleanup(); + } + }); + + it('does not use a comment-only symbol mention as an exact import anchor', async () => { + const fixture = await searchFixture( + [ + chunk('src/auth.ts', 'import "./billing.js";\n// authenticateUser was renamed last year.'), + chunk('src/billing.ts', 'export const billing = { invoice: "paid", refund: "pending" };') + ], + true + ); + try { + const matches = await fixture.search('authenticateUser billing invoice refund'); + expect(matches.map((result) => path.basename(result.filePath))).toEqual(['billing.ts']); + } finally { + await fixture.cleanup(); + } + }); + + it.each([ + '// import "./billing.js";', + '/* import "./billing.js"; */', + 'const example = \'import "./billing.js";\';', + 'const example = `import "./billing.js";`;', + 'const example = \'import("./billing.js")\';', + 'const example = \'require("./billing.js")\';', + 'const example = \'from "./billing.js"\';' + ])('does not use fake import text as cross-file evidence: %s', async (fakeImport) => { + const auth = chunk( + 'src/auth.ts', + `${fakeImport}\nexport function authenticateUser() { return true; }`, + 'authenticateUser' + ); + auth.imports = ['./billing.js']; // Raw chunk metadata has no source proof either. + const fixture = await searchFixture( + [ + auth, + chunk('src/billing.ts', 'export const billing = { invoice: "paid", refund: "pending" };') + ], + true + ); + try { + expect(fixture.graph.imports['src/auth.ts'] ?? []).toEqual([]); + const matches = await fixture.search('authenticateUser billing invoice refund'); + expect(matches.map((result) => path.basename(result.filePath))).toEqual(['billing.ts']); + } finally { + await fixture.cleanup(); + } + }); + + it('fails closed when a genuine import has no indexed graph proof', async () => { + const fixture = await searchFixture([ + chunk( + 'src/auth.ts', + 'import "./billing.js";\nexport function authenticateUser() { return true; }', + 'authenticateUser' + ), + chunk('src/billing.ts', 'export const billing = { invoice: "paid", refund: "pending" };') + ]); + try { + const matches = await fixture.search('authenticateUser billing invoice refund'); + expect(matches.map((result) => path.basename(result.filePath))).toEqual(['billing.ts']); + } finally { + await fixture.cleanup(); + } + }); + + it.each(['legacy', 'sourceFingerprint', 'targetFingerprint'] as const)( + 'fails closed for %s graph proof', + async (invalidProof) => { + const fixture = await searchFixture( + [ + chunk( + 'src/auth.ts', + 'import "./billing.js";\nexport function authenticateUser() { return true; }', + 'authenticateUser' + ), + chunk('src/billing.ts', 'export const billing = { invoice: "paid", refund: "pending" };') + ], + true + ); + try { + const detail = fixture.graph.importDetails!['src/auth.ts']['src/billing.ts']; + expect(detail.sources).toHaveLength(1); + if (invalidProof === 'legacy') delete detail.sources; + else detail.sources![0][invalidProof] = '0'.repeat(64); + await fixture.reloadGraph(fixture.graph); + const matches = await fixture.search('authenticateUser billing invoice refund'); + expect(matches.map((result) => path.basename(result.filePath))).toEqual(['billing.ts']); + } finally { + await fixture.cleanup(); + } + } + ); + + it('does not conflate a resolved JSON import with a same-stem TypeScript file', async () => { + const json = chunk('src/billing.json', '{}'); + json.language = 'json'; + const fixture = await searchFixture( + [ + chunk( + 'src/auth.ts', + 'import "./billing.json";\nexport function authenticateUser() { return true; }', + 'authenticateUser' + ), + json, + chunk('src/billing.ts', 'export const billing = { invoice: "paid", refund: "pending" };') + ], + true + ); + try { + expect(fixture.graph.imports['src/auth.ts']).toEqual(['src/billing.json']); + const matches = await fixture.search('authenticateUser billing invoice refund', { + language: 'typescript' + }); + expect(matches.map((result) => path.basename(result.filePath))).toEqual(['billing.ts']); + } finally { + await fixture.cleanup(); + } + }); + + it('excludes an anchor when its only graph companion no longer matches the snapshot', async () => { + const auth = chunk( + 'src/auth.ts', + 'import "./billing.js";\nexport function authenticateUser() { return true; }', + 'authenticateUser' + ); + const billing = chunk( + 'src/billing.ts', + 'export const billing = { invoice: "paid", refund: "pending" };' + ); + const fixture = await searchFixture([auth, billing], true); + try { + await fs.writeFile(billing.filePath, 'export const unrelated = true;'); + // Contract migration: drift must not become corruption or lend stale graph proof. + expect(await fixture.search('authenticateUser billing invoice refund')).toEqual([]); + } finally { + await fixture.cleanup(); + } + }); + + it('preserves a fresh semantic nominee while excluding its stale graph companion', async () => { + const auth = chunk( + 'src/auth.ts', + 'import "./billing.js";\nexport function authenticateUser() { return true; }', + 'authenticateUser' + ); + const billing = chunk( + 'src/billing.ts', + 'export const billing = { invoice: "paid", refund: "pending" };' + ); + const fixture = await searchFixture([auth, billing], true); + try { + await fs.writeFile(billing.filePath, 'export const unrelated = true;'); + fixture.searcher.embeddingProvider = {}; + fixture.searcher.storageProvider = {}; + fixture.searcher.semanticSearch = async () => [{ chunk: auth, score: 0.9 }]; + + const results = await fixture.searchWithSemantic('authenticateUser billing invoice refund'); + expect(results.map((result) => path.basename(result.filePath))).toEqual(['auth.ts']); + expect(results[0].snippet).toContain('authenticateUser'); + } finally { + await fixture.cleanup(); + } + }); + + it('does not corroborate an anchor after alias configuration redirects its unchanged import', async () => { + const fixture = await searchFixture([ + chunk( + 'src/auth.ts', + 'import "@billing";\nexport function authenticateUser() { return true; }', + 'authenticateUser' + ), + chunk('src/billing.ts', 'export const billing = { invoice: "paid", refund: "pending" };'), + chunk('src/other.ts', 'export const unrelated = true;') + ]); + const configPath = path.join(fixture.rootPath, 'tsconfig.json'); + try { + await fs.writeFile( + configPath, + JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { '@billing': ['src/billing.ts'] } } + }) + ); + const authPath = path.join(fixture.rootPath, 'src/auth.ts'); + const billingPath = path.join(fixture.rootPath, 'src/billing.ts'); + const graph = new InternalFileGraph(fixture.rootPath); + graph.trackImport(authPath, billingPath, 1, [], { + kind: 'import', + specifier: '@billing', + sourceFingerprint: createHash('sha256') + .update(await fs.readFile(authPath)) + .digest('hex'), + targetFingerprint: createHash('sha256') + .update(await fs.readFile(billingPath)) + .digest('hex') + }); + await fixture.reloadGraph(graph.toJSON()); + expect( + (await fixture.search('authenticateUser billing invoice refund')) + .map((result) => path.basename(result.filePath)) + .sort() + ).toEqual(['auth.ts', 'billing.ts']); + await fs.writeFile( + configPath, + JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { '@billing': ['src/other.ts'] } } + }) + ); + // Same endpoint bytes and same initialized searcher: resolution is a per-query witness. + expect( + (await fixture.search('authenticateUser billing invoice refund')).map((result) => + path.basename(result.filePath) + ) + ).toEqual(['billing.ts']); + } finally { + await fixture.cleanup(); + } + }); + + it.skipIf(process.platform !== 'linux')( + 'keeps case-distinct indexed import targets separate', + async () => { + const fixture = await searchFixture( + [ + chunk( + 'src/auth.ts', + 'import "./Billing.js";\nexport function authenticateUser() { return true; }', + 'authenticateUser' + ), + chunk('src/Billing.ts', 'export const unrelated = true;'), + chunk('src/billing.ts', 'export const billing = { invoice: "paid", refund: "pending" };') + ], + true + ); + try { + expect(fixture.graph.imports['src/auth.ts']).toEqual(['src/Billing.ts']); + const matches = await fixture.search('authenticateUser billing invoice refund'); + expect(matches.map((result) => path.basename(result.filePath))).toEqual(['billing.ts']); + } finally { + await fixture.cleanup(); + } + } + ); + + it('does not turn generic TypeScript syntax words into a confident match', async () => { + const fixture = await searchFixture([ + chunk('src/unconnected.ts', 'export function unconnected() { return 1; }') + ]); + try { + expect(await fixture.search('configure function return on thing')).toEqual([]); + } finally { + await fixture.cleanup(); + } + }); + + it('applies a language filter before the bounded rescue-candidate cutoff', async () => { + const decoys = Array.from({ length: 1051 }, (_, index) => { + const item = chunk(`src/noise-${index}.py`, 'system error algorithm'); + item.language = 'python'; + return item; + }); + const target = chunk('src/algorithm.ts', 'export const algorithm = "cohesion";'); + const fixture = await searchFixture([...decoys, target]); + try { + const matches = await fixture.search('system error algorithm cohesion', { + language: 'typescript' + }); + expect(matches[0]?.filePath).toContain('algorithm.ts'); + expect(matches).toHaveLength(1); + } finally { + await fixture.cleanup(); + } + }); +}); diff --git a/tests/search-ranking.test.ts b/tests/search-ranking.test.ts index 24449e4..4330978 100644 --- a/tests/search-ranking.test.ts +++ b/tests/search-ranking.test.ts @@ -1,8 +1,18 @@ import { describe, it, expect, vi } from 'vitest'; import type { CodeChunk } from '../src/types/index.js'; import { CodebaseSearcher } from '../src/core/search.js'; +import { + asSearcherTestHarness, + materializeSourceSnapshot, + type SourceSnapshot +} from './helpers/source-snapshot.js'; -function createChunk(id: string, filePath: string, content: string): CodeChunk { +function createChunk( + id: string, + filePath: string, + content: string, + metadata: CodeChunk['metadata'] = {} +): CodeChunk { return { id, content, @@ -18,28 +28,84 @@ function createChunk(id: string, filePath: string, content: string): CodeChunk { imports: [], exports: [], tags: [], - metadata: {} + metadata }; } function setupSemanticOnlySearcher( - results: { chunk: CodeChunk; score: number }[] + results: { chunk: CodeChunk; score: number }[], + snapshot: SourceSnapshot ): CodebaseSearcher { - const searcher = new CodebaseSearcher('C:/repo') as any; - searcher.initialized = true; - searcher.embeddingProvider = { + const searcher = new CodebaseSearcher(snapshot.rootPath); + const harness = asSearcherTestHarness(searcher); + harness.initialized = true; + harness.sourceManifest = snapshot.manifest; + harness.embeddingProvider = { embed: vi.fn(async () => [0.1, 0.2]) }; - searcher.storageProvider = { + harness.storageProvider = { search: vi.fn(async () => results), count: vi.fn(async () => results.length) }; - searcher.fuseIndex = null; - searcher.patternIntelligence = null; - return searcher as CodebaseSearcher; + harness.fuseIndex = null; + harness.patternIntelligence = null; + return searcher; } describe('CodebaseSearcher query-aware ranking', () => { + it('keeps same-named symbols from different files and deduplicates chunks within each file', async () => { + const symbolPath = ['readTargetDefaultsForTarget']; + const metadata = { + symbolName: 'readTargetDefaultsForTarget', + symbolPath, + symbolAware: true + }; + const firstFileBest = createChunk( + 'target-best', + 'C:/repo/packages/nx/target-defaults.ts', + 'export function readTargetDefaultsForTarget() { return "nx"; }', + metadata + ); + const firstFileDuplicate = createChunk( + 'target-duplicate', + 'C:/repo/packages/nx/target-defaults.ts', + 'export function readTargetDefaultsForTarget() { return "nx"; }', + metadata + ); + const secondFile = createChunk( + 'wrapper', + 'C:/repo/packages/devkit/target-defaults.ts', + 'export function readTargetDefaultsForTarget() { return "devkit"; }', + metadata + ); + + const snapshot = await materializeSourceSnapshot( + [firstFileBest, firstFileDuplicate, secondFile], + 'search-symbol-file-identity-' + ); + try { + const searcher = setupSemanticOnlySearcher( + [ + { chunk: firstFileBest, score: 0.9 }, + { chunk: firstFileDuplicate, score: 0.85 }, + { chunk: secondFile, score: 0.8 } + ], + snapshot + ); + + const results = await searcher.search('readTargetDefaultsForTarget', 2); + + expect(results).toHaveLength(2); + expect(results.map((result) => result.filePath)).toEqual([ + firstFileBest.filePath, + secondFile.filePath + ]); + expect(results[0].score).toBeGreaterThan(results[1].score); + } finally { + await snapshot.cleanup(); + } + }); + it('de-prioritizes spec files for non-testing queries', async () => { const specChunk = createChunk( 'spec', @@ -52,14 +118,24 @@ describe('CodebaseSearcher query-aware ranking', () => { 'export class SessionManager {}' ); - const searcher = setupSemanticOnlySearcher([ - { chunk: specChunk, score: 0.75 }, - { chunk: implChunk, score: 0.68 } - ]); - - const results = await searcher.search('Refactor session management flow', 2); - expect(results[0].filePath).toContain('session-manager.ts'); - expect(results[0].filePath).not.toContain('.spec.ts'); + const snapshot = await materializeSourceSnapshot( + [specChunk, implChunk], + 'search-ranking-source-' + ); + try { + const searcher = setupSemanticOnlySearcher( + [ + { chunk: specChunk, score: 0.75 }, + { chunk: implChunk, score: 0.68 } + ], + snapshot + ); + const results = await searcher.search('Refactor session management flow', 2); + expect(results[0].filePath).toContain('session-manager.ts'); + expect(results[0].filePath).not.toContain('.spec.ts'); + } finally { + await snapshot.cleanup(); + } }); it('keeps spec files prioritized for testing queries', async () => { @@ -74,13 +150,23 @@ describe('CodebaseSearcher query-aware ranking', () => { 'export class SessionManager {}' ); - const searcher = setupSemanticOnlySearcher([ - { chunk: specChunk, score: 0.75 }, - { chunk: implChunk, score: 0.68 } - ]); - - const results = await searcher.search('Update unit tests for session manager with mocks', 2); - expect(results[0].filePath).toContain('.spec.ts'); + const snapshot = await materializeSourceSnapshot( + [specChunk, implChunk], + 'search-ranking-source-' + ); + try { + const searcher = setupSemanticOnlySearcher( + [ + { chunk: specChunk, score: 0.75 }, + { chunk: implChunk, score: 0.68 } + ], + snapshot + ); + const results = await searcher.search('Update unit tests for session manager with mocks', 2); + expect(results[0].filePath).toContain('.spec.ts'); + } finally { + await snapshot.cleanup(); + } }); it('de-prioritizes Windows e2e paths for non-testing queries', async () => { @@ -95,12 +181,22 @@ describe('CodebaseSearcher query-aware ranking', () => { 'export class SessionManager {}' ); - const searcher = setupSemanticOnlySearcher([ - { chunk: e2eChunk, score: 0.75 }, - { chunk: implChunk, score: 0.72 } - ]); - - const results = await searcher.search('session login flow', 2); - expect(results[0].filePath.toLowerCase()).toContain('session-manager.ts'); + const snapshot = await materializeSourceSnapshot( + [e2eChunk, implChunk], + 'search-ranking-source-' + ); + try { + const searcher = setupSemanticOnlySearcher( + [ + { chunk: e2eChunk, score: 0.75 }, + { chunk: implChunk, score: 0.72 } + ], + snapshot + ); + const results = await searcher.search('session login flow', 2); + expect(results[0].filePath.toLowerCase()).toContain('session-manager.ts'); + } finally { + await snapshot.cleanup(); + } }); }); diff --git a/tests/search-reranker-health.test.ts b/tests/search-reranker-health.test.ts index aea0cfd..6ff8ae8 100644 --- a/tests/search-reranker-health.test.ts +++ b/tests/search-reranker-health.test.ts @@ -29,6 +29,13 @@ vi.mock('../src/core/search.js', async () => { async search(query: string, limit: number, filters?: unknown) { return searchMocks.search(query, limit, filters); } + + async searchWithCoverage(query: string, limit: number, filters?: unknown) { + return { + results: await this.search(query, limit, filters), + sourceCoverage: { status: 'current', checkedFiles: 0, skippedFiles: 0 } + }; + } } return { CodebaseSearcher }; diff --git a/tests/search-rescue-density.test.ts b/tests/search-rescue-density.test.ts new file mode 100644 index 0000000..b88cb8a --- /dev/null +++ b/tests/search-rescue-density.test.ts @@ -0,0 +1,151 @@ +import { describe, expect, it } from 'vitest'; +import { promises as fs } from 'node:fs'; +import Fuse, { type IFuseOptions } from 'fuse.js'; +import { CodebaseSearcher } from '../src/core/search.js'; +import type { CodeChunk } from '../src/types/index.js'; +import { asSearcherTestHarness, materializeSourceSnapshot } from './helpers/source-snapshot.js'; + +interface DensityChunkSpec { + id: string; + filePath: string; + content: string; + startLine?: number; + metadata?: CodeChunk['metadata']; +} + +interface DensityCase { + id: string; + query: string; + chunks: DensityChunkSpec[]; + expectedIds: string[]; + expectedPathPatterns: string[]; + limitation?: string; +} + +interface GeneratedChunkCase { + count: number; + idPattern: string; + filePathPattern: string; + content: string; +} + +interface SourceValidationCase { + id: string; + query: string; + chunks?: DensityChunkSpec[]; + staleChunkGenerator?: GeneratedChunkCase; + currentChunks?: DensityChunkSpec[]; + mutateFilePath?: string; + mutateFilePathPattern?: string; + replacementSource: string; + expectedIds: string[]; +} + +interface DensityFixture { + cases: DensityCase[]; + sourceValidationCases: SourceValidationCase[]; +} + +interface DensityHarness { + chunks: CodeChunk[]; + fuseIndex: Fuse; + rescueKeywordMatches: ( + query: string, + limit: number + ) => Promise>; +} + +const fixture = JSON.parse( + await fs.readFile(new URL('./fixtures/search-rescue-density.json', import.meta.url), 'utf8') +) as DensityFixture; + +const fuseOptions: IFuseOptions = { + keys: [ + { name: 'content', weight: 0.4 }, + { name: 'metadata.componentName', weight: 0.25 }, + { name: 'filePath', weight: 0.15 }, + { name: 'relativePath', weight: 0.15 }, + { name: 'componentType', weight: 0.15 }, + { name: 'layer', weight: 0.1 }, + { name: 'tags', weight: 0.15 } + ], + includeScore: true, + threshold: 0.4, + useExtendedSearch: true, + ignoreLocation: true +}; + +function makeChunk(spec: DensityChunkSpec, index = 0): CodeChunk { + const startLine = spec.startLine ?? 1; + return { + id: spec.id || `density-${index}`, + filePath: spec.filePath, + relativePath: spec.filePath, + content: spec.content, + startLine, + endLine: startLine + spec.content.split('\n').length - 1, + language: 'typescript', + framework: 'generic', + componentType: 'module', + dependencies: [], + imports: [], + exports: [], + tags: [], + metadata: spec.metadata ?? {} + }; +} + +async function setup(chunks: CodeChunk[]) { + const snapshot = await materializeSourceSnapshot(chunks, 'cbc-rescue-density-'); + const searcher = new CodebaseSearcher(snapshot.rootPath); + const base = asSearcherTestHarness(searcher); + base.initialized = true; + base.sourceManifest = snapshot.manifest; + base.embeddingProvider = null; + base.storageProvider = null; + const fuse = new Fuse(chunks, fuseOptions); + base.fuseIndex = fuse; + const harness = searcher as unknown as DensityHarness; + harness.chunks = chunks; + return { ...snapshot, harness, fuse }; +} + +describe('frozen rescue density behavior controls', () => { + it.each(fixture.cases)('$id', async (control) => { + const subject = await setup(control.chunks.map(makeChunk)); + try { + const results = await subject.harness.rescueKeywordMatches(control.query, 10); + expect(results.map(({ item }) => item.id)).toEqual(control.expectedIds); + expect(results.map(({ item }) => item.relativePath)).toEqual(control.expectedPathPatterns); + } finally { + await subject.cleanup(); + } + }); + + it.each(fixture.sourceValidationCases)('$id', async (control) => { + const chunks = control.staleChunkGenerator + ? Array.from({ length: control.staleChunkGenerator.count }, (_, index) => { + const generated = control.staleChunkGenerator!; + return makeChunk({ + id: generated.idPattern.replace('{index}', String(index)), + filePath: generated.filePathPattern.replace('{index}', String(index)), + content: generated.content + }); + }).concat((control.currentChunks ?? []).map(makeChunk)) + : (control.chunks ?? []).map(makeChunk); + const subject = await setup(chunks); + try { + const changedChunks = control.mutateFilePathPattern + ? chunks.filter(({ id }) => id.startsWith('stale-')) + : chunks.filter(({ relativePath }) => relativePath === control.mutateFilePath); + for (const stale of changedChunks) { + await fs.writeFile(stale.filePath, control.replacementSource); + } + + const results = await subject.harness.rescueKeywordMatches(control.query, 5); + expect(results.map(({ item }) => item.id)).toEqual(control.expectedIds); + } finally { + await subject.cleanup(); + } + }); +}); diff --git a/tests/search-rescue-nomination.test.ts b/tests/search-rescue-nomination.test.ts new file mode 100644 index 0000000..1585be9 --- /dev/null +++ b/tests/search-rescue-nomination.test.ts @@ -0,0 +1,378 @@ +import { describe, expect, it, vi } from 'vitest'; +import { promises as fs } from 'node:fs'; +import path from 'node:path'; +import Fuse, { type FuseResult, type IFuseOptions } from 'fuse.js'; +import { CodebaseSearcher } from '../src/core/search.js'; +import { IndexCorruptedError } from '../src/errors/index.js'; +import type { CodeChunk, SearchFilters } from '../src/types/index.js'; +import { asSearcherTestHarness, materializeSourceSnapshot } from './helpers/source-snapshot.js'; + +interface ChunkSpec { + id?: string; + filePath?: string; + content: string; + startLine?: number; + language?: string; + metadata?: CodeChunk['metadata']; + exports?: string[]; + tags?: string[]; +} + +interface NominationFixture { + lexicalCases: Array< + ChunkSpec & { id: string; query: string; expected: boolean; expectedScore?: number } + >; + representativeCases: Array<{ + id: string; + query: string; + chunks: Array; + expectedIds: string[]; + }>; + populationCases: Array<{ + id: string; + limit: number; + count: number; + cap: number; + exhausted: boolean; + }>; + lifecycleCases: Array<'changed' | 'missing' | 'directory' | 'invalid-manifest' | 'outside-root'>; + graphCases: Array<'current' | 'no-edge' | 'changed-companion' | 'directory-companion'>; + strictCases: Array<{ id: string; query: string; expectsResult: boolean }>; +} + +interface Validation { + sources: Map; + candidateBudgetExhausted?: true; +} + +interface RescueHarness { + chunks: CodeChunk[]; + indexedImports: Map>; + indexedFileKey: (file: string) => string; + lexicalEvidence: (text: string) => Set; + rescueKeywordMatches: ( + query: string, + limit: number, + filters?: SearchFilters, + validation?: Validation + ) => Promise[]>; + keywordSearch: ( + query: string, + limit: number, + filters?: SearchFilters + ) => Promise>; +} + +const fixture = JSON.parse( + await fs.readFile(new URL('./fixtures/search-rescue-nomination.json', import.meta.url), 'utf8') +) as NominationFixture; + +const options: IFuseOptions = { + keys: [ + { name: 'content', weight: 0.4 }, + { name: 'metadata.componentName', weight: 0.25 }, + { name: 'filePath', weight: 0.15 }, + { name: 'relativePath', weight: 0.15 }, + { name: 'componentType', weight: 0.15 }, + { name: 'layer', weight: 0.1 }, + { name: 'tags', weight: 0.15 } + ], + includeScore: true, + threshold: 0.4, + useExtendedSearch: true, + ignoreLocation: true +}; + +function chunk(spec: ChunkSpec, index = 0): CodeChunk { + const startLine = spec.startLine ?? 1; + return { + id: spec.id ?? `chunk-${index}`, + filePath: spec.filePath ?? 'src/a.ts', + relativePath: spec.filePath ?? 'src/a.ts', + content: spec.content, + startLine, + endLine: startLine + spec.content.split('\n').length - 1, + language: spec.language ?? 'typescript', + framework: 'generic', + componentType: 'module', + dependencies: [], + imports: [], + exports: spec.exports ?? [], + tags: spec.tags ?? [], + metadata: spec.metadata ?? {} + }; +} + +async function setup(chunks: CodeChunk[]) { + const snapshot = await materializeSourceSnapshot(chunks, 'cbc-nomination-contract-'); + const searcher = new CodebaseSearcher(snapshot.rootPath); + const base = asSearcherTestHarness(searcher); + base.initialized = true; + base.sourceManifest = snapshot.manifest; + base.embeddingProvider = null; + base.storageProvider = null; + const fuse = new Fuse(chunks, options); + base.fuseIndex = fuse; + const harness = searcher as unknown as RescueHarness; + harness.chunks = chunks; + return { ...snapshot, searcher, harness, fuse }; +} + +describe('frozen rescue-only literal nomination contract', () => { + // These are lexical unit controls, not an accuracy score. In particular, a + // comment can provide a lexical clue without proving a program relationship. + it.each(fixture.lexicalCases)('$id', async (control) => { + const source = chunk(control); + const subject = await setup([source]); + try { + const results = await subject.harness.rescueKeywordMatches(control.query, 5); + expect(results.map(({ item }) => item.id)).toEqual(control.expected ? [source.id] : []); + if (control.expected) expect(results[0]?.score).toBe(control.expectedScore); + } finally { + await subject.cleanup(); + } + }); + + it.each(fixture.representativeCases)('$id', async (control) => { + const subject = await setup(control.chunks.map(chunk)); + try { + const results = await subject.harness.rescueKeywordMatches(control.query, 5); + expect(results.map(({ item }) => item.id)).toEqual(control.expectedIds); + expect(results.every(({ score }) => score !== undefined && score >= 0 && score <= 1)).toBe( + true + ); + } finally { + await subject.cleanup(); + } + }); + + it('does not invoke corpus fuzzy matching to nominate or rank a rescue witness', async () => { + const subject = await setup([chunk({ content: '// archiving policies' })]); + const fuzzy = vi.spyOn(Fuse.prototype, 'search'); + try { + expect(await subject.harness.rescueKeywordMatches('archive policy', 5)).toHaveLength(1); + expect(fuzzy).not.toHaveBeenCalled(); + } finally { + fuzzy.mockRestore(); + await subject.cleanup(); + } + }); + + it.each(fixture.populationCases)('$id', async (control) => { + const chunks = Array.from({ length: control.count }, (_, index) => + chunk({ + id: `nominee-${index}`, + filePath: index === control.cap ? 'src/b.ts' : 'src/a.ts', + content: '// archive policy' + }) + ); + const subject = await setup(chunks); + const evidence = vi.spyOn(subject.harness, 'lexicalEvidence'); + const validation: Validation = { sources: new Map() }; + try { + const results = await subject.harness.rescueKeywordMatches( + 'archive policy', + control.limit, + undefined, + validation + ); + // Overflow is a declared bounded miss, not an invented complete search. + expect(results.map(({ item }) => item.id)).toEqual(['nominee-0']); + expect(Boolean(validation.candidateBudgetExhausted)).toBe(control.exhausted); + expect(evidence.mock.calls.length).toBeLessThanOrEqual(control.cap * 2); + } finally { + await subject.cleanup(); + } + }); + + it.each(['language', 'framework', 'componentType', 'layer', 'tags'] as const)( + 'applies %s before the candidate window', + async (field) => { + const rejected = Array.from({ length: 760 }, (_, index) => + chunk({ + id: `excluded-${index}`, + filePath: 'src/a.ts', + content: '// archive policy' + }) + ); + const target = chunk({ id: 'included', filePath: 'src/b.ts', content: '// archive policy' }); + const filters: SearchFilters = {}; + if (field === 'language') { + filters.language = 'python'; + target.language = 'python'; + } + if (field === 'framework') { + filters.framework = 'test-framework'; + target.framework = 'test-framework'; + } + if (field === 'componentType') { + filters.componentType = 'function'; + target.componentType = 'function'; + } + if (field === 'layer') { + filters.layer = 'business'; + target.layer = 'business'; + } + if (field === 'tags') { + filters.tags = ['included']; + target.tags = ['included']; + } + const subject = await setup([...rejected, target]); + const validation: Validation = { sources: new Map() }; + try { + const results = await subject.harness.rescueKeywordMatches( + 'archive policy', + 1, + filters, + validation + ); + expect(results.map(({ item }) => item.id)).toEqual(['included']); + expect(validation.candidateBudgetExhausted).toBeUndefined(); + } finally { + await subject.cleanup(); + } + } + ); + + it.each(fixture.lifecycleCases)('preserves %s source handling', async (state) => { + const source = chunk({ content: '// archive policy' }); + const subject = await setup([source]); + const validation: Validation = { sources: new Map() }; + try { + if (state === 'changed') await fs.writeFile(source.filePath, '// replacement\n'); + if (state === 'missing' || state === 'directory') await fs.unlink(source.filePath); + if (state === 'directory') await fs.mkdir(source.filePath); + if (state === 'invalid-manifest') subject.manifest.files['src/a.ts'] = 'invalid'; + if (state === 'outside-root') + source.filePath = path.resolve(subject.rootPath, '../escape.ts'); + const operation = subject.harness.rescueKeywordMatches( + 'archive policy', + 5, + undefined, + validation + ); + if (state === 'changed' || state === 'missing') { + expect(await operation).toEqual([]); + expect(validation.sources.get(source.filePath)).toBeNull(); + } else { + await expect(operation).rejects.toBeInstanceOf(IndexCorruptedError); + } + } finally { + await subject.cleanup(); + } + }); + + it('refills past all chunks of a stale source without rerunning Fuse', async () => { + const stale = Array.from({ length: 270 }, (_, index) => + chunk({ + id: `stale-${index}`, + content: '// archive policy' + }) + ); + const current = chunk({ id: 'current', filePath: 'src/b.ts', content: '// archive policy' }); + const subject = await setup([...stale, current]); + const fuzzy = vi.spyOn(Fuse.prototype, 'search'); + try { + await fs.writeFile(stale[0].filePath, '// replacement\n'); + const results = await subject.harness.rescueKeywordMatches('archive policy', 1); + expect(results.map(({ item }) => item.id)).toEqual(['current']); + expect(fuzzy).not.toHaveBeenCalled(); + } finally { + fuzzy.mockRestore(); + await subject.cleanup(); + } + }); + + it('exposes nomination overflow through the public source coverage result', async () => { + const chunks = Array.from({ length: 752 }, (_, index) => + chunk({ + id: `split-${index}`, + content: index % 2 === 0 ? '// archive' : '// policy', + startLine: index % 2 === 0 ? 1 : 3 + }) + ); + const subject = await setup(chunks); + try { + const outcome = await subject.searcher.searchWithCoverage('archive policy', 1, undefined, { + useSemanticSearch: false, + enableQueryExpansion: false, + enableLowConfidenceRescue: false, + enableReranker: false + }); + expect(outcome.results).toHaveLength(1); + expect(outcome.sourceCoverage.status).toBe('partial'); + expect(outcome.sourceCoverage.candidateBudgetExhausted).toBe(true); + } finally { + await subject.cleanup(); + } + }); + + it.each(fixture.graphCases)('requires current resolved graph proof: %s', async (state) => { + const anchor = chunk({ + id: 'anchor', + content: 'import { rules } from "./b.js";\nexport class LedgerGateway {}', + metadata: { className: 'LedgerGateway' } + }); + const companion = chunk({ + id: 'companion', + filePath: 'src/b.ts', + content: 'export const rules = { archive: true, policy: true };' + }); + const subject = await setup([anchor, companion]); + if (state !== 'no-edge') { + subject.harness.indexedImports = new Map([ + [ + subject.harness.indexedFileKey(anchor.filePath), + new Map([[subject.harness.indexedFileKey(companion.filePath), ['./b.js']]]) + ] + ]); + } + try { + if (state === 'changed-companion') + await fs.writeFile(companion.filePath, 'export const rules = {};\n'); + if (state === 'directory-companion') { + await fs.unlink(companion.filePath); + await fs.mkdir(companion.filePath); + } + const operation = subject.harness.rescueKeywordMatches('LedgerGateway archive policy', 5); + if (state === 'directory-companion') { + await expect(operation).rejects.toBeInstanceOf(IndexCorruptedError); + } else { + const ids = (await operation).map(({ item }) => item.id); + expect(ids.includes('anchor')).toBe(state === 'current'); + } + } finally { + await subject.cleanup(); + } + }); + + it.each(fixture.strictCases)('retains original strict Fuse route: $id', async (control) => { + const source = chunk({ content: 'export class ArchivePolicy {}\n// archive policy' }); + const subject = await setup([source]); + const fuzzy = vi.spyOn(subject.fuse, 'search'); + try { + const results = await subject.harness.keywordSearch(control.query, 5); + expect(fuzzy).toHaveBeenCalledWith(control.query); + expect(results.length > 0).toBe(control.expectsResult); + } finally { + await subject.cleanup(); + } + }); + + it.each([ + 'archive | policy', + '^archive policy', + 'archive policy$', + '!archive policy', + '=archive policy' + ])('keeps explicit syntax out of lexical rescue: %s', async (query) => { + const subject = await setup([chunk({ content: '// archive policy' })]); + const fuzzy = vi.spyOn(subject.fuse, 'search'); + try { + expect(await subject.harness.rescueKeywordMatches(query, 5)).toEqual([]); + expect(fuzzy).not.toHaveBeenCalled(); + } finally { + await subject.cleanup(); + } + }); +}); diff --git a/tests/search-retrieval-strategy.test.ts b/tests/search-retrieval-strategy.test.ts index 0cb92a2..7010342 100644 --- a/tests/search-retrieval-strategy.test.ts +++ b/tests/search-retrieval-strategy.test.ts @@ -1,6 +1,10 @@ import { describe, it, expect, vi } from 'vitest'; import type { CodeChunk } from '../src/types/index.js'; import { CodebaseSearcher } from '../src/core/search.js'; +import { asSearcherTestHarness, materializeSourceSnapshot } from './helpers/source-snapshot.js'; +import { promises as fs } from 'node:fs'; +import path from 'node:path'; +import { IndexCorruptedError } from '../src/errors/index.js'; function createChunk(id: string, filePath: string, content: string): CodeChunk { return { @@ -22,31 +26,226 @@ function createChunk(id: string, filePath: string, content: string): CodeChunk { }; } +async function createSourceBackedSearcher(chunks: CodeChunk[]) { + const snapshot = await materializeSourceSnapshot(chunks, 'retrieval-source-'); + const searcher = new CodebaseSearcher(snapshot.rootPath); + const harness = asSearcherTestHarness(searcher); + harness.initialized = true; + harness.sourceManifest = snapshot.manifest; + harness.embeddingProvider = {}; + harness.storageProvider = {}; + harness.fuseIndex = null; + harness.patternIntelligence = null; + harness.semanticSearch = vi.fn(async () => []); + harness.keywordSearch = vi.fn(async () => []); + return { + searcher, + harness, + cleanup: snapshot.cleanup + }; +} + describe('CodebaseSearcher retrieval strategy', () => { + it.each(['ordinary', 'integrity'] as const)( + 'preserves validated candidates on ordinary refill failure and propagates integrity failure (%s)', + async (failure) => { + const stale = Array.from({ length: 100 }, (_, n) => + createChunk( + `stale-${n}`, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return true; }' + ) + ); + const support = createChunk( + 'support', + 'src/support.ts', + 'export function support() { return quartzBeacon(); }' + ); + const chunks = [...stale, support]; + const fixture = await createSourceBackedSearcher(chunks); + const harness = fixture.harness as typeof fixture.harness & { chunks: CodeChunk[] }; + harness.chunks = chunks; + Reflect.deleteProperty(harness, 'semanticSearch'); + const embed = vi.fn(async () => [0.1, 0.2]); + const vectorSearch = vi + .fn() + .mockResolvedValueOnce( + [...stale.slice(0, 29), support].map((chunk) => ({ chunk, score: 0.9, distance: 0.1 })) + ) + .mockRejectedValueOnce( + failure === 'integrity' + ? new IndexCorruptedError('Invalid vector index build identity') + : new Error('Semantic refill temporarily unavailable') + ); + harness.embeddingProvider = { embed }; + harness.storageProvider = { search: vectorSearch }; + try { + await fs.writeFile(stale[0].filePath, 'export const changed = true;\n'); + const outcome = fixture.searcher.searchWithCoverage( + 'ordinary semantic request', + 1, + undefined, + { + useKeywordSearch: false, + useSemanticSearch: true, + enableQueryExpansion: false, + enableLowConfidenceRescue: false, + enableReranker: false + } + ); + if (failure === 'integrity') { + await expect(outcome).rejects.toBeInstanceOf(IndexCorruptedError); + } else { + const result = await outcome; + expect(result.results.map((match) => path.basename(match.filePath))).toEqual([ + 'support.ts' + ]); + expect(result.results[0].snippet).toContain('support'); + expect(result.sourceCoverage).toMatchObject({ + status: 'partial', + checkedFiles: 2, + skippedFiles: 1, + candidateRetrievalFailed: true + }); + expect(result.sourceCoverage.candidateBudgetExhausted).toBeUndefined(); + } + expect(embed).toHaveBeenCalledTimes(1); + expect(vectorSearch).toHaveBeenCalledTimes(2); + } finally { + await fixture.cleanup(); + } + } + ); + + it('discloses semantic exhaustion when widening reaches the bounded cumulative retrieval work', async () => { + const stale = Array.from({ length: 1000 }, (_, n) => + createChunk( + `stale-${n}`, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return true; }' + ) + ); + const support = createChunk( + 'support', + 'src/support.ts', + 'export function support() { return quartzBeacon(); }' + ); + const chunks = [...stale, support]; + const fixture = await createSourceBackedSearcher(chunks); + const harness = fixture.harness as typeof fixture.harness & { chunks: CodeChunk[] }; + harness.chunks = chunks; + Reflect.deleteProperty(harness, 'semanticSearch'); + const embed = vi.fn(async () => [0.1, 0.2]); + const vectorSearch = vi.fn(async (_vector: number[], limit: number) => + chunks.slice(0, limit).map((chunk) => ({ chunk, score: 0.9, distance: 0.1 })) + ); + harness.embeddingProvider = { embed }; + harness.storageProvider = { search: vectorSearch }; + try { + await fs.writeFile(stale[0].filePath, 'export const changed = true;\n'); + const outcome = await fixture.searcher.searchWithCoverage('quartzBeacon', 1, undefined, { + useKeywordSearch: false, + useSemanticSearch: true, + enableQueryExpansion: false, + enableLowConfidenceRescue: false, + enableReranker: false + }); + expect(outcome.results).toEqual([]); + expect(outcome.sourceCoverage).toMatchObject({ + status: 'no_fresh_results', + checkedFiles: 1, + skippedFiles: 1, + candidateBudgetExhausted: true + }); + expect(embed).toHaveBeenCalledTimes(1); + expect(vectorSearch).toHaveBeenCalledTimes(2); + expect( + vectorSearch.mock.calls.reduce((sum, [, limit]) => sum + limit, 0) + ).toBeLessThanOrEqual(750); + expect(Math.max(...vectorSearch.mock.calls.map(([, limit]) => limit))).toBeLessThanOrEqual( + chunks.length + ); + } finally { + await fixture.cleanup(); + } + }); + + it.each([false, true])( + 'replenishes semantic stale prefixes only when needed (changed=%s) with one embedding', + async (changed) => { + const staleChunks = Array.from({ length: 100 }, (_, n) => + createChunk( + `stale-${n}`, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return true; }' + ) + ); + const support = createChunk( + 'support', + 'src/support.ts', + 'export function support() { return quartzBeacon(); }' + ); + const chunks = [...staleChunks, support]; + const fixture = await createSourceBackedSearcher(chunks); + const harness = fixture.harness as typeof fixture.harness & { chunks: CodeChunk[] }; + harness.chunks = chunks; + Reflect.deleteProperty(harness, 'semanticSearch'); + const embed = vi.fn(async () => [0.1, 0.2]); + const vectorSearch = vi.fn(async (_vector: number[], limit: number) => + chunks.slice(0, limit).map((chunk) => ({ chunk, score: 0.9, distance: 0.1 })) + ); + harness.embeddingProvider = { embed }; + harness.storageProvider = { search: vectorSearch }; + try { + if (changed) await fs.writeFile(staleChunks[0].filePath, 'export const changed = true;\n'); + const outcome = await fixture.searcher.searchWithCoverage('quartzBeacon', 1, undefined, { + useKeywordSearch: false, + useSemanticSearch: true, + enableQueryExpansion: false, + enableLowConfidenceRescue: false, + enableReranker: false + }); + expect(outcome.results).toHaveLength(1); + expect(path.basename(outcome.results[0].filePath)).toBe( + changed ? 'support.ts' : 'quartzBeacon.ts' + ); + expect(embed).toHaveBeenCalledTimes(1); + expect(vectorSearch).toHaveBeenCalledTimes(changed ? 2 : 1); + expect(vectorSearch.mock.calls[0][1]).toBe(30); + if (changed) { + expect(vectorSearch.mock.calls[1][1]).toBe(101); + expect(outcome.sourceCoverage).toMatchObject({ + status: 'partial', + checkedFiles: 2, + skippedFiles: 1 + }); + } + } finally { + await fixture.cleanup(); + } + } + ); + it('enforces candidate floor independently from user limit', async () => { const implChunk = createChunk( 'impl', 'src/core/auth-service.ts', 'export class AuthService {}' ); - const searcher = new CodebaseSearcher('C:/repo') as any; - - searcher.initialized = true; - searcher.embeddingProvider = {}; - searcher.storageProvider = {}; - searcher.fuseIndex = null; - searcher.patternIntelligence = null; - searcher.semanticSearch = vi.fn(async () => [{ chunk: implChunk, score: 0.7 }]); - searcher.keywordSearch = vi.fn(async () => []); - - await searcher.search('authentication login', 1, undefined, { - enableQueryExpansion: false, - enableLowConfidenceRescue: false, - candidateFloor: 30 - }); - - expect(searcher.semanticSearch).toHaveBeenCalledTimes(1); - expect(searcher.semanticSearch.mock.calls[0][1]).toBe(30); + const { searcher, harness, cleanup } = await createSourceBackedSearcher([implChunk]); + const semanticSearch = vi.fn(async () => [{ chunk: implChunk, score: 0.7 }]); + harness.semanticSearch = semanticSearch; + try { + await searcher.search('authentication login', 1, undefined, { + enableQueryExpansion: false, + enableLowConfidenceRescue: false, + candidateFloor: 30 + }); + expect(semanticSearch).toHaveBeenCalledTimes(1); + expect(semanticSearch.mock.calls[0][1]).toBe(30); + } finally { + await cleanup(); + } }); it('uses bounded query expansion for intent-heavy queries', async () => { @@ -55,24 +254,20 @@ describe('CodebaseSearcher retrieval strategy', () => { 'src/core/router-service.ts', 'export class RouterService {}' ); - const searcher = new CodebaseSearcher('C:/repo') as any; - - searcher.initialized = true; - searcher.embeddingProvider = {}; - searcher.storageProvider = {}; - searcher.fuseIndex = null; - searcher.patternIntelligence = null; - searcher.semanticSearch = vi.fn(async () => [{ chunk: implChunk, score: 0.65 }]); - searcher.keywordSearch = vi.fn(async () => []); - - await searcher.search('authentication login', 3, undefined, { - enableQueryExpansion: true, - enableLowConfidenceRescue: false - }); - - const semanticQueries = searcher.semanticSearch.mock.calls.map((call: any[]) => call[0]); - expect(semanticQueries[0]).toBe('authentication login'); - expect(semanticQueries.length).toBeLessThanOrEqual(2); + const { searcher, harness, cleanup } = await createSourceBackedSearcher([implChunk]); + const semanticSearch = vi.fn(async (_query: string) => [{ chunk: implChunk, score: 0.65 }]); + harness.semanticSearch = semanticSearch; + try { + await searcher.search('authentication login', 3, undefined, { + enableQueryExpansion: true, + enableLowConfidenceRescue: false + }); + const semanticQueries = semanticSearch.mock.calls.map(([query]) => query); + expect(semanticQueries[0]).toBe('authentication login'); + expect(semanticQueries.length).toBeLessThanOrEqual(2); + } finally { + await cleanup(); + } }); it('runs low-confidence rescue and can replace poor primary ranking', async () => { @@ -87,14 +282,9 @@ describe('CodebaseSearcher retrieval strategy', () => { 'export class AuthCallbackComponent {}' ); - const searcher = new CodebaseSearcher('C:/repo') as any; - searcher.initialized = true; - searcher.embeddingProvider = {}; - searcher.storageProvider = {}; - searcher.fuseIndex = null; - searcher.patternIntelligence = null; + const { searcher, harness, cleanup } = await createSourceBackedSearcher([specChunk, implChunk]); - searcher.semanticSearch = vi.fn(async (query: string) => { + const semanticSearch = vi.fn(async (query: string) => { if (query.includes('router') || query.includes('navigation')) { return [ { chunk: implChunk, score: 0.8 }, @@ -107,16 +297,18 @@ describe('CodebaseSearcher retrieval strategy', () => { { chunk: implChunk, score: 0.35 } ]; }); - searcher.keywordSearch = vi.fn(async () => []); - - const results = await searcher.search('navigate to page after login redirect', 2, undefined, { - enableQueryExpansion: false, - enableLowConfidenceRescue: true, - profile: 'edit' - }); - - expect(results[0].filePath).toContain('auth-callback.component.ts'); - expect(results[0].filePath).not.toContain('.spec.ts'); + harness.semanticSearch = semanticSearch; + try { + const results = await searcher.search('navigate to page after login redirect', 2, undefined, { + enableQueryExpansion: false, + enableLowConfidenceRescue: true, + profile: 'edit' + }); + expect(results[0].filePath).toContain('auth-callback.component.ts'); + expect(results[0].filePath).not.toContain('.spec.ts'); + } finally { + await cleanup(); + } }); it('applies intent-based keyword-heavy weights for exact-name queries', async () => { @@ -126,24 +318,19 @@ describe('CodebaseSearcher retrieval strategy', () => { 'export class AuthService { login() {} }' ); - const searcher = new CodebaseSearcher('C:/repo') as any; - searcher.initialized = true; - searcher.embeddingProvider = {}; - searcher.storageProvider = {}; - searcher.fuseIndex = null; - searcher.patternIntelligence = null; + const { searcher, harness, cleanup } = await createSourceBackedSearcher([implChunk]); let capturedSemanticWeight: number | undefined; let capturedKeywordWeight: number | undefined; // Intercept collectHybridMatches to capture the weights used - searcher.collectHybridMatches = vi.fn( + harness.collectHybridMatches = vi.fn( async ( - _variants: any, - _limit: any, - _filters: any, - _useSemantic: any, - _useKeyword: any, + _variants: unknown, + _limit: unknown, + _filters: unknown, + _useSemantic: unknown, + _useKeyword: unknown, semWeight: number, kwWeight: number ) => { @@ -153,22 +340,23 @@ describe('CodebaseSearcher retrieval strategy', () => { semantic: new Map([ ['impl', { chunk: implChunk, ranks: [{ rank: 0, weight: semWeight }] }] ]), - keyword: new Map([ - ['impl', { chunk: implChunk, ranks: [{ rank: 0, weight: kwWeight }] }] - ]) + keyword: new Map([['impl', { chunk: implChunk, ranks: [{ rank: 0, weight: kwWeight }] }]]) }; } ); // AuthService is PascalCase → EXACT_NAME intent → keyword: 0.6, semantic: 0.4 - await searcher.search('AuthService', 3, undefined, { - enableQueryExpansion: false, - enableLowConfidenceRescue: false - }); - - expect(capturedKeywordWeight).toBeGreaterThan(capturedSemanticWeight!); - expect(capturedKeywordWeight).toBeCloseTo(0.6, 1); - expect(capturedSemanticWeight).toBeCloseTo(0.4, 1); + try { + await searcher.search('AuthService', 3, undefined, { + enableQueryExpansion: false, + enableLowConfidenceRescue: false + }); + expect(capturedKeywordWeight).toBeGreaterThan(capturedSemanticWeight!); + expect(capturedKeywordWeight).toBeCloseTo(0.6, 1); + expect(capturedSemanticWeight).toBeCloseTo(0.4, 1); + } finally { + await cleanup(); + } }); it('produces scores below 1.0 for weak single-list retrievals', async () => { @@ -179,22 +367,20 @@ describe('CodebaseSearcher retrieval strategy', () => { ); const weakChunk = createChunk('weak', 'src/utils/helpers.ts', 'export function helper() {}'); - const searcher = new CodebaseSearcher('C:/repo') as any; - searcher.initialized = true; - searcher.embeddingProvider = {}; - searcher.storageProvider = {}; - searcher.fuseIndex = null; - searcher.patternIntelligence = null; + const { searcher, harness, cleanup } = await createSourceBackedSearcher([ + strongChunk, + weakChunk + ]); // Mock collectHybridMatches: strong chunk rank 0 in both channels, // weak chunk rank 5 in keyword only - searcher.collectHybridMatches = vi.fn( + harness.collectHybridMatches = vi.fn( async ( - _variants: any, - _limit: any, - _filters: any, - _useSemantic: any, - _useKeyword: any, + _variants: unknown, + _limit: unknown, + _filters: unknown, + _useSemantic: unknown, + _useKeyword: unknown, semWeight: number, kwWeight: number ) => ({ @@ -208,17 +394,20 @@ describe('CodebaseSearcher retrieval strategy', () => { }) ); - const results = await searcher.search('authentication login', 10, undefined, { - enableQueryExpansion: false, - enableLowConfidenceRescue: false - }); - - expect(results.length).toBeGreaterThanOrEqual(2); - expect(results[0].score).toBeGreaterThan(0); - // Key invariant: strong result (rank 0, both channels) must score - // meaningfully higher than weak result (rank 5, keyword only). - // With normalization-by-actual-max, the ratio collapses. - const ratio = results[0].score / results[1].score; - expect(ratio).toBeGreaterThan(1.5); + try { + const results = await searcher.search('authentication login', 10, undefined, { + enableQueryExpansion: false, + enableLowConfidenceRescue: false + }); + expect(results.length).toBeGreaterThanOrEqual(2); + expect(results[0].score).toBeGreaterThan(0); + // Key invariant: strong result (rank 0, both channels) must score + // meaningfully higher than weak result (rank 5, keyword only). + // With normalization-by-actual-max, the ratio collapses. + const ratio = results[0].score / results[1].score; + expect(ratio).toBeGreaterThan(1.5); + } finally { + await cleanup(); + } }); }); diff --git a/tests/search-safe-01.test.ts b/tests/search-safe-01.test.ts index bee7c75..e2b7cc0 100644 --- a/tests/search-safe-01.test.ts +++ b/tests/search-safe-01.test.ts @@ -24,6 +24,13 @@ vi.mock('../src/core/search.js', async () => { async search(query: string, limit: number, filters?: unknown) { return searchMocks.search(query, limit, filters); } + + async searchWithCoverage(query: string, limit: number, filters?: unknown) { + return { + results: await this.search(query, limit, filters), + sourceCoverage: { status: 'current', checkedFiles: 0, skippedFiles: 0 } + }; + } } return { CodebaseSearcher }; diff --git a/tests/search-scoped-memory.test.ts b/tests/search-scoped-memory.test.ts index 0f6df64..ae21450 100644 --- a/tests/search-scoped-memory.test.ts +++ b/tests/search-scoped-memory.test.ts @@ -16,6 +16,13 @@ vi.mock('../src/core/search.js', async () => { async search(query: string, limit: number, filters?: unknown) { return searchMocks.search(query, limit, filters); } + + async searchWithCoverage(query: string, limit: number, filters?: unknown) { + return { + results: await this.search(query, limit, filters), + sourceCoverage: { status: 'current', checkedFiles: 0, skippedFiles: 0 } + }; + } } return { CodebaseSearcher }; diff --git a/tests/search-source-validity.test.ts b/tests/search-source-validity.test.ts new file mode 100644 index 0000000..98428a2 --- /dev/null +++ b/tests/search-source-validity.test.ts @@ -0,0 +1,962 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { promises as fs } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { CodebaseIndexer } from '../src/core/indexer.js'; +import { CodebaseSearcher, type SearchOptions } from '../src/core/search.js'; +import { analyzerRegistry } from '../src/core/analyzer-registry.js'; +import { GenericAnalyzer } from '../src/analyzers/generic/index.js'; +import { IndexCorruptedError } from '../src/errors/index.js'; +import { + CODEBASE_CONTEXT_DIRNAME, + HEALTH_FILENAME, + INTELLIGENCE_FILENAME, + KEYWORD_INDEX_FILENAME, + MANIFEST_FILENAME, + MEMORY_FILENAME, + VECTOR_DB_DIRNAME +} from '../src/constants/codebase-context.js'; +import { findSymbolReferences } from '../src/core/symbol-references.js'; +import type { CodeChunk, SearchResult } from '../src/types/index.js'; +import { handle as searchCodebase } from '../src/tools/search-codebase.js'; +import type { SearchResponse, ToolContext } from '../src/tools/types.js'; + +const KEYWORD_ONLY_OPTIONS: SearchOptions = { + useSemanticSearch: false, + useKeywordSearch: true, + enableQueryExpansion: false, + enableLowConfidenceRescue: false, + enableReranker: false, + profile: 'explore' +}; + +function isContained(rootPath: string, candidatePath: string): boolean { + const relativePath = path.relative(rootPath, candidatePath); + return ( + relativePath === '' || + (relativePath !== '..' && + !relativePath.startsWith(`..${path.sep}`) && + !path.isAbsolute(relativePath)) + ); +} + +async function indexProject(rootPath: string, extraFileExtensions: string[] = []): Promise { + const indexer = new CodebaseIndexer({ + rootPath, + config: { + skipEmbedding: true, + embedding: { provider: 'openai', model: 'text-embedding-3-small' }, + parsing: { maxFileSize: 1024 * 1024 } + }, + projectOptions: { + preferredAnalyzer: 'generic', + extraFileExtensions + } + }); + await indexer.index(); +} + +async function createSourceFile( + rootPath: string, + relativePath: string, + content: string +): Promise { + const filePath = path.join(rootPath, relativePath); + await fs.mkdir(path.dirname(filePath), { recursive: true }); + await fs.writeFile(filePath, content, 'utf8'); +} + +function searchContext(rootPath: string): ToolContext { + const baseDir = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + return { + rootPath, + indexState: { status: 'ready' }, + paths: { + baseDir, + keywordIndex: path.join(baseDir, KEYWORD_INDEX_FILENAME), + intelligence: path.join(baseDir, INTELLIGENCE_FILENAME), + memory: path.join(baseDir, MEMORY_FILENAME), + health: path.join(baseDir, HEALTH_FILENAME), + vectorDb: path.join(baseDir, VECTOR_DB_DIRNAME) + }, + performIndexing: vi.fn() + }; +} + +async function expectSearchResultsOpenInsideRoot( + rootPath: string, + results: SearchResult[], + expectedRelativePath: string +): Promise { + expect(results.length).toBeGreaterThan(0); + for (const result of results) { + expect(path.isAbsolute(result.filePath)).toBe(true); + expect(isContained(rootPath, result.filePath)).toBe(true); + expect(path.relative(rootPath, result.filePath).replace(/\\/g, '/')).toBe(expectedRelativePath); + const source = await fs.readFile(result.filePath, 'utf8'); + const sourceLines = source.replace(/\r\n/g, '\n').split('\n'); + const indexedRange = sourceLines.slice(result.startLine - 1, result.endLine).join('\n'); + expect(indexedRange).toContain('quartzBeacon'); + expect(indexedRange).toContain(result.snippet.replace(/\n\n\.\.\. \[\d+ more lines\]$/, '')); + expect(result.startLine).toBeGreaterThan(0); + expect(result.endLine).toBeGreaterThanOrEqual(result.startLine); + } +} + +describe('search source-path validity', () => { + let tempRoot: string; + let originalEmbeddingProvider: string | undefined; + let originalOpenAiKey: string | undefined; + let consoleWarnSpy: ReturnType; + + beforeEach(async () => { + tempRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-search-source-validity-')); + originalEmbeddingProvider = process.env.EMBEDDING_PROVIDER; + originalOpenAiKey = process.env.OPENAI_API_KEY; + // CodebaseSearcher uses its real provider-selection path, which fails closed before any + // network request when OpenAI is selected without a key; keyword search remains available. + process.env.EMBEDDING_PROVIDER = 'openai'; + delete process.env.OPENAI_API_KEY; + consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + analyzerRegistry.register(new GenericAnalyzer()); + }); + + afterEach(async () => { + consoleWarnSpy.mockRestore(); + if (originalEmbeddingProvider === undefined) delete process.env.EMBEDDING_PROVIDER; + else process.env.EMBEDDING_PROVIDER = originalEmbeddingProvider; + if (originalOpenAiKey === undefined) delete process.env.OPENAI_API_KEY; + else process.env.OPENAI_API_KEY = originalOpenAiKey; + await fs.rm(tempRoot, { recursive: true, force: true }); + }); + + it.each(['changed', 'deleted', 'unreadable'] as const)( + 'keeps a fresh lower nominee when the top source is %s without rebuilding', + async (drift) => { + const rootPath = path.join(tempRoot, drift); + const topPath = path.join(rootPath, 'src/quartzBeacon.ts'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return "old top"; }\n' + ); + await createSourceFile( + rootPath, + 'src/support.ts', + 'export function support() { return quartzBeacon(); }\n' + ); + await indexProject(rootPath); + const indexPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const indexBefore = await fs.readFile(indexPath, 'utf8'); + if (drift === 'changed') await fs.writeFile(topPath, 'export const changed = true;\n'); + if (drift === 'deleted') await fs.rm(topPath); + const readFile = fs.readFile.bind(fs); + const readSpy = + drift === 'unreadable' + ? vi + .spyOn(fs, 'readFile') + .mockImplementation((...args: Parameters) => { + if (path.resolve(String(args[0])) === path.resolve(topPath)) { + return Promise.reject( + Object.assign(new Error('Permission denied'), { code: 'EACCES' }) + ); + } + return readFile(...args); + }) + : undefined; + try { + const ctx = searchContext(rootPath); + const response = await searchCodebase( + { query: 'quartzBeacon', limit: 1, intent: 'edit' }, + ctx + ); + const payload = JSON.parse(response.content![0].text) as SearchResponse; + expect(payload.status).toBe('success'); + expect(payload.results).toHaveLength(1); + expect(payload.results[0].file).toContain('support.ts:'); + expect(payload.sourceCoverage).toMatchObject({ + status: 'partial', + checkedFiles: 2, + skippedFiles: 1 + }); + expect(payload.preflight?.ready).toBe(false); + expect(payload.preflight?.nextAction).toMatch(/refresh/i); + expect(JSON.stringify(payload)).not.toContain('old top'); + expect(ctx.performIndexing).not.toHaveBeenCalled(); + expect(await fs.readFile(indexPath, 'utf8')).toBe(indexBefore); + } finally { + readSpy?.mockRestore(); + } + } + ); + + it.each(['compact', 'full'])( + 'distinguishes stale-only evidence from no nominations in %s mode', + async (mode) => { + const rootPath = path.join(tempRoot, mode); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return "old top"; }\n' + ); + await indexProject(rootPath); + await fs.writeFile( + path.join(rootPath, 'src/quartzBeacon.ts'), + 'export const different = true;\n' + ); + const ctx = searchContext(rootPath); + const stale = await searchCodebase({ query: 'quartzBeacon', mode }, ctx); + const payload = JSON.parse(stale.content![0].text) as SearchResponse & { message: string }; + expect(payload.status).toBe('no_fresh_results'); + expect(payload.message).toMatch(/no fresh/i); + expect(payload.sourceCoverage).toMatchObject({ status: 'no_fresh_results', skippedFiles: 1 }); + expect(payload.sourceCoverage?.refreshAction).toMatch(/refresh_index/); + expect(payload.results).toEqual([]); + expect(payload.preflight).toBeUndefined(); + expect(ctx.performIndexing).not.toHaveBeenCalled(); + expect(JSON.stringify(payload)).not.toContain('old top'); + for (const intent of ['edit', 'refactor', 'migrate'] as const) { + const guarded = await searchCodebase({ query: 'quartzBeacon', mode, intent }, ctx); + const guardedPayload = JSON.parse(guarded.content![0].text) as SearchResponse; + expect(guardedPayload.status).toBe('no_fresh_results'); + expect(guardedPayload.sourceCoverage?.status).toBe('no_fresh_results'); + expect(guardedPayload.preflight?.ready).toBe(false); + } + const empty = await searchCodebase({ query: 'nonexistentUnrelatedNomination', mode }, ctx); + const emptyPayload = JSON.parse(empty.content![0].text) as SearchResponse; + expect(emptyPayload.status).toBe('success'); + expect(emptyPayload.results).toEqual([]); + expect(emptyPayload.sourceCoverage).toBeUndefined(); + } + ); + + it('rejects an injected nominee absent from the manifest even with a fresh lower result', async () => { + const rootPath = path.join(tempRoot, 'injected source'); + await createSourceFile(rootPath, 'src/quartzBeacon.ts', 'export const quartzBeacon = true;\n'); + await indexProject(rootPath); + const indexPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const index = JSON.parse(await fs.readFile(indexPath, 'utf8')) as { chunks: CodeChunk[] }; + await createSourceFile( + rootPath, + 'src/injected.ts', + 'export const quartzBeacon = "injected";\n' + ); + index.chunks.push({ + ...index.chunks[0], + id: 'injected', + filePath: path.join(rootPath, 'src/injected.ts'), + relativePath: 'src/injected.ts' + }); + await fs.writeFile(indexPath, JSON.stringify(index)); + await expect( + new CodebaseSearcher(rootPath).search('quartzBeacon', 1, undefined, KEYWORD_ONLY_OPTIONS) + ).rejects.toThrow(/absent from its manifest/); + }); + + it('keeps concurrent query coverage independent on one initialized searcher', async () => { + const rootPath = path.join(tempRoot, 'concurrent coverage'); + await createSourceFile(rootPath, 'src/quartzBeacon.ts', 'export const quartzBeacon = true;\n'); + await createSourceFile(rootPath, 'src/stableBeacon.ts', 'export const stableBeacon = true;\n'); + await indexProject(rootPath); + await fs.writeFile( + path.join(rootPath, 'src/quartzBeacon.ts'), + 'export const changed = true;\n' + ); + const searcher = new CodebaseSearcher(rootPath); + await searcher.initialize(); + const [stale, current] = await Promise.all([ + searcher.searchWithCoverage('quartzBeacon', 1, undefined, KEYWORD_ONLY_OPTIONS), + searcher.searchWithCoverage('stableBeacon', 1, undefined, KEYWORD_ONLY_OPTIONS) + ]); + expect(stale.results).toEqual([]); + expect(stale.sourceCoverage).toMatchObject({ + status: 'no_fresh_results', + checkedFiles: 1, + skippedFiles: 1 + }); + expect(current.results).toHaveLength(1); + expect(current.sourceCoverage).toEqual({ status: 'current', checkedFiles: 1, skippedFiles: 0 }); + }); + + it.each(['compact', 'full'] as const)( + 'replenishes current candidates while retaining mutation gates in %s mode', + async (mode) => { + const rootPath = path.join(tempRoot, 'one stale source many chunks'); + const sourcePath = path.join(rootPath, 'src/quartzBeacon.ts'); + const supportPath = path.join(rootPath, 'src/support.ts'); + const source = + Array.from( + { length: 100 }, + (_, n) => + `export function quartzBeacon${n}() {\n${Array.from( + { length: 30 }, + (_, m) => ` const helper${m} = "quartzBeacon";` + ).join('\n')}\n return "quartzBeacon";\n}` + ).join('\n') + '\n'; + await createSourceFile(rootPath, 'src/quartzBeacon.ts', source); + await createSourceFile( + rootPath, + 'src/support.ts', + 'export function support() { return quartzBeacon0(); }\n' + ); + await indexProject(rootPath); + const indexPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const indexBefore = await fs.readFile(indexPath, 'utf8'); + const index = JSON.parse(indexBefore) as { chunks: CodeChunk[] }; + expect( + index.chunks.filter((chunk) => chunk.relativePath === 'src/quartzBeacon.ts').length + ).toBeGreaterThan(30); + const readSpy = vi.spyOn(fs, 'readFile'); + const realPathSpy = vi.spyOn(fs, 'realpath'); + try { + const fresh = await new CodebaseSearcher(rootPath).searchWithCoverage( + 'quartzBeacon', + 1, + undefined, + KEYWORD_ONLY_OPTIONS + ); + expect(fresh.results).toHaveLength(1); + expect(fresh.sourceCoverage).toEqual({ status: 'current', checkedFiles: 1, skippedFiles: 0 }); + await fs.writeFile(sourcePath, 'export const changed = true;\n'); + for (const limit of [1, 5]) { + readSpy.mockClear(); + realPathSpy.mockClear(); + const result = await new CodebaseSearcher(rootPath).searchWithCoverage( + 'quartzBeacon', + limit, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(rootPath, result.results, 'src/support.ts'); + expect(result.results.length).toBeLessThanOrEqual(limit); + expect(result.sourceCoverage).toMatchObject({ + status: 'partial', + checkedFiles: 2, + skippedFiles: 1 + }); + expect(result.sourceCoverage.candidateBudgetExhausted).toBeUndefined(); + const reads = readSpy.mock.calls.map(([file]) => path.resolve(String(file))); + expect(reads.filter((file) => file === sourcePath)).toHaveLength(1); + // The assertion helper opens the returned file once as a host would. + expect(reads.filter((file) => file === supportPath)).toHaveLength(2); + expect( + realPathSpy.mock.calls.filter(([file]) => path.resolve(String(file)) === rootPath) + ).toHaveLength(1); + } + const ctx = searchContext(rootPath); + const response = await searchCodebase({ query: 'quartzBeacon', limit: 1, mode }, ctx); + const payload = JSON.parse(response.content![0].text) as SearchResponse; + expect(payload.status).toBe('success'); + expect(payload.results).toHaveLength(1); + expect(payload.results[0].file).toContain('support.ts:'); + expect(payload.sourceCoverage?.status).toBe('partial'); + expect(payload.preflight).toBeUndefined(); + for (const intent of ['edit', 'refactor', 'migrate'] as const) { + const guarded = await searchCodebase( + { query: 'quartzBeacon', limit: 1, mode, intent }, + ctx + ); + const guardedPayload = JSON.parse(guarded.content![0].text) as SearchResponse; + expect(guardedPayload.sourceCoverage?.status).toBe('partial'); + expect(guardedPayload.preflight?.ready).toBe(false); + expect(guardedPayload.preflight?.nextAction).toMatch(/refresh/i); + } + expect(ctx.performIndexing).not.toHaveBeenCalled(); + expect(await fs.readFile(indexPath, 'utf8')).toBe(indexBefore); + } finally { + readSpy.mockRestore(); + realPathSpy.mockRestore(); + } + } + ); + + it('discloses exhausted source inspection budget instead of claiming all nominated evidence is stale', async () => { + const rootPath = path.join(tempRoot, 'bounded stale sources'); + for (let n = 0; n < 35; n++) { + await createSourceFile( + rootPath, + `src/quartzBeacon${n}.ts`, + `export function quartzBeacon${n}() { return "quartzBeacon"; }\n` + ); + } + await createSourceFile( + rootPath, + 'src/support.ts', + 'export function support() { return quartzBeacon0(); }\n' + ); + await indexProject(rootPath); + for (let n = 0; n < 35; n++) { + await fs.writeFile( + path.join(rootPath, `src/quartzBeacon${n}.ts`), + 'export const changed = true;\n' + ); + } + const outcome = await new CodebaseSearcher(rootPath).searchWithCoverage( + 'quartzBeacon', + 1, + undefined, + KEYWORD_ONLY_OPTIONS + ); + expect(outcome.results).toEqual([]); + expect(outcome.sourceCoverage).toMatchObject({ + status: 'no_fresh_results', + candidateBudgetExhausted: true + }); + expect(outcome.sourceCoverage.checkedFiles).toBeLessThanOrEqual(30); + const ctx = searchContext(rootPath); + const response = await searchCodebase({ query: 'quartzBeacon', limit: 1 }, ctx); + const payload = JSON.parse(response.content![0].text) as SearchResponse & { message: string }; + expect(payload.status).toBe('no_fresh_results'); + expect(payload.message).toMatch(/bounded candidate window.*uninspected/i); + expect(payload.sourceCoverage?.candidateBudgetExhausted).toBe(true); + expect(ctx.performIndexing).not.toHaveBeenCalled(); + }); + + it('rejects malformed ranges in lower nominated evidence before limiting to a fresh top file', async () => { + const rootPath = path.join(tempRoot, 'lower invalid range'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return true; }\n' + ); + await createSourceFile(rootPath, 'src/support.ts', 'export const support = quartzBeacon;\n'); + await indexProject(rootPath); + const before = await new CodebaseSearcher(rootPath).search( + 'quartzBeacon', + 1, + undefined, + KEYWORD_ONLY_OPTIONS + ); + expect(path.basename(before[0].filePath)).toBe('quartzBeacon.ts'); + const indexPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const index = JSON.parse(await fs.readFile(indexPath, 'utf8')) as { chunks: CodeChunk[] }; + for (const chunk of index.chunks.filter((chunk) => chunk.relativePath === 'src/support.ts')) { + chunk.endLine = 99999; + } + await fs.writeFile(indexPath, JSON.stringify(index)); + await expect( + new CodebaseSearcher(rootPath).search('quartzBeacon', 1, undefined, KEYWORD_ONLY_OPTIONS) + ).rejects.toThrow(/source line range is invalid/i); + }); + + it('stores root-relative chunk paths when indexing nested roots from an ancestor cwd', async () => { + const ancestorPath = path.join(tempRoot, 'consumer'); + const rootPath = path.join(ancestorPath, 'project-a'); + const independentRootPath = path.join(ancestorPath, 'project-b'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return "first"; }\nquartzBeacon();\n' + ); + await createSourceFile( + rootPath, + 'src/stableBeacon.ts', + 'export function stableBeacon() { return "stable"; }\nstableBeacon();\n' + ); + await createSourceFile( + independentRootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return "independent"; }\nquartzBeacon();\n' + ); + + const cwdSpy = vi.spyOn(process, 'cwd').mockReturnValue(ancestorPath); + try { + await indexProject(rootPath); + + const indexPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const initialIndex = JSON.parse(await fs.readFile(indexPath, 'utf8')) as { + chunks: Array<{ + content: string; + filePath: string; + id: string; + relativePath: string; + startLine: number; + endLine: number; + }>; + }; + expect(initialIndex.chunks.length).toBeGreaterThan(0); + expect( + initialIndex.chunks.every((chunk) => !chunk.relativePath.startsWith('project-a/')) + ).toBe(true); + expect(initialIndex.chunks.every((chunk) => chunk.relativePath.startsWith('src/'))).toBe( + true + ); + + const firstResults = await new CodebaseSearcher(rootPath).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(rootPath, firstResults, 'src/quartzBeacon.ts'); + + const stableBefore = initialIndex.chunks.filter((chunk) => + chunk.relativePath.endsWith('stableBeacon.ts') + ); + expect(stableBefore.length).toBeGreaterThan(0); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return "edited"; }\nquartzBeacon();\n' + ); + cwdSpy.mockReturnValue(tempRoot); + await new CodebaseIndexer({ + rootPath, + config: { + skipEmbedding: true, + embedding: { provider: 'openai', model: 'text-embedding-3-small' }, + parsing: { maxFileSize: 1024 * 1024 } + }, + projectOptions: { preferredAnalyzer: 'generic' }, + incrementalOnly: true + }).index(); + + const incrementallyIndexed = JSON.parse(await fs.readFile(indexPath, 'utf8')) as { + chunks: Array<{ + content: string; + filePath: string; + id: string; + relativePath: string; + startLine: number; + endLine: number; + }>; + }; + expect(incrementallyIndexed.chunks.length).toBeGreaterThan(0); + expect( + incrementallyIndexed.chunks.every( + (chunk) => + chunk.relativePath === 'src/quartzBeacon.ts' || + chunk.relativePath === 'src/stableBeacon.ts' + ) + ).toBe(true); + const stableAfter = incrementallyIndexed.chunks.filter((chunk) => + chunk.relativePath.endsWith('stableBeacon.ts') + ); + expect(stableAfter).toHaveLength(stableBefore.length); + expect( + stableAfter.map(({ relativePath, filePath, content, startLine, endLine }) => ({ + relativePath, + filePath, + content, + startLine, + endLine + })) + ).toEqual( + stableBefore.map(({ relativePath, filePath, content, startLine, endLine }) => ({ + relativePath, + filePath, + content, + startLine, + endLine + })) + ); + + const editedResults = await new CodebaseSearcher(rootPath).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(rootPath, editedResults, 'src/quartzBeacon.ts'); + expect(editedResults.some((result) => result.snippet.includes('edited'))).toBe(true); + + cwdSpy.mockReturnValue(ancestorPath); + await indexProject(independentRootPath); + const independentIndex = JSON.parse( + await fs.readFile( + path.join(independentRootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME), + 'utf8' + ) + ) as { chunks: Array<{ relativePath: string }> }; + expect(independentIndex.chunks.length).toBeGreaterThan(0); + expect(independentIndex.chunks.every((chunk) => chunk.relativePath.startsWith('src/'))).toBe( + true + ); + const independentResults = await new CodebaseSearcher(independentRootPath).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot( + independentRootPath, + independentResults, + 'src/quartzBeacon.ts' + ); + } finally { + cwdSpy.mockRestore(); + } + }); + + it('rejects a copied index that points outside the selected root, then searches rebuilt current ranges', async () => { + const originalRoot = path.join(tempRoot, 'source repo with spaces'); + const copiedRoot = path.join(tempRoot, 'copied repo with spaces'); + const initialSource = [ + 'export function quartzBeacon() {', + ' return "original";', + '}', + 'quartzBeacon();', + '' + ].join('\n'); + await createSourceFile(originalRoot, 'src/quartzBeacon.ts', initialSource); + await indexProject(originalRoot); + await fs.cp(originalRoot, copiedRoot, { recursive: true }); + await createSourceFile( + copiedRoot, + 'src/quartzBeacon.ts', + `// copied header one\n// copied header two\n${initialSource}` + ); + + const staleSearcher = new CodebaseSearcher(copiedRoot); + await expect( + staleSearcher.search('quartzBeacon', 5, undefined, KEYWORD_ONLY_OPTIONS) + ).rejects.toBeInstanceOf(IndexCorruptedError); + + // The existing recovery action is a complete index rebuild, which must replace absolute + // chunk paths and stale line ranges with evidence from the selected copied repository. + await indexProject(copiedRoot); + const rebuiltResults = await new CodebaseSearcher(copiedRoot).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(copiedRoot, rebuiltResults, 'src/quartzBeacon.ts'); + expect(rebuiltResults.some((result) => result.startLine >= 3)).toBe(true); + }); + + it('renders only current cited source lines, not indexing-only context headers', async () => { + const rootPath = path.join(tempRoot, 'annotated chunks'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon(): string {\r\n return "current source";\r\n}\r\nquartzBeacon();\r\n' + ); + await indexProject(rootPath); + const indexPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const index = JSON.parse(await fs.readFile(indexPath, 'utf8')) as { + chunks: Array<{ content: string }>; + }; + for (const chunk of index.chunks) + chunk.content = '// indexing-only signature header\n' + chunk.content; + const indexedBytes = JSON.stringify(index); + await fs.writeFile(indexPath, indexedBytes, 'utf8'); + + const results = await new CodebaseSearcher(rootPath).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + + await expectSearchResultsOpenInsideRoot(rootPath, results, 'src/quartzBeacon.ts'); + expect( + results.every((result) => !result.snippet.includes('indexing-only signature header')) + ).toBe(true); + expect(await fs.readFile(indexPath, 'utf8')).toBe(indexedBytes); + }); + + it('preserves source projection when Windows opens the same root with different casing', async () => { + const rootPath = path.join(tempRoot, 'mixed-case-project'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return 1; }\n' + ); + await indexProject(rootPath); + const differentlyCasedRoot = path.join(tempRoot, 'MIXED-CASE-PROJECT'); + const caseVariantExists = await fs.access(differentlyCasedRoot).then( + () => true, + () => false + ); + const search = new CodebaseSearcher(differentlyCasedRoot).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + if (caseVariantExists) { + await expectSearchResultsOpenInsideRoot( + differentlyCasedRoot, + await search, + 'src/quartzBeacon.ts' + ); + } else { + // Filesystem behavior, not the OS name, decides whether this is the same project. + await expect(search).rejects.toBeInstanceOf(IndexCorruptedError); + } + }); + + it('rejects an indexed directory redirected outside the root even when source bytes match', async () => { + const rootPath = path.join(tempRoot, 'junction-project'); + const outsidePath = path.join(tempRoot, 'external-source'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return 1; }\n' + ); + await indexProject(rootPath); + // Only this test's owned directory is moved; the index retains the original path/hash. + await fs.rename(path.join(rootPath, 'src'), outsidePath); + await fs.symlink( + outsidePath, + path.join(rootPath, 'src'), + process.platform === 'win32' ? 'junction' : 'dir' + ); + await expect( + new CodebaseSearcher(rootPath).search('quartzBeacon', 5, undefined, KEYWORD_ONLY_OPTIONS) + ).rejects.toBeInstanceOf(IndexCorruptedError); + expect(await fs.readFile(path.join(outsidePath, 'quartzBeacon.ts'), 'utf8')).toContain( + 'quartzBeacon' + ); + }); + + it('permits an indexed directory linked to a source location still inside its own root', async () => { + const rootPath = path.join(tempRoot, 'real-project'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return 1; }\n' + ); + await indexProject(rootPath); + const internalSourcePath = path.join(rootPath, 'internal-source'); + await fs.rename(path.join(rootPath, 'src'), internalSourcePath); + await fs.symlink( + internalSourcePath, + path.join(rootPath, 'src'), + process.platform === 'win32' ? 'junction' : 'dir' + ); + const results = await new CodebaseSearcher(rootPath).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(rootPath, results, 'src/quartzBeacon.ts'); + }); + + it.each([-1, 1.5, 99999])( + 'rejects invalid returned source bounds (%s) without fabricating a snippet', + async (endLine) => { + const rootPath = path.join(tempRoot, 'invalid range'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return 1; }\n' + ); + await indexProject(rootPath); + const indexPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const index = JSON.parse(await fs.readFile(indexPath, 'utf8')) as { + chunks: Array<{ startLine: number; endLine: number }>; + }; + for (const chunk of index.chunks) { + chunk.startLine = 1; + chunk.endLine = endLine; + } + await fs.writeFile(indexPath, JSON.stringify(index), 'utf8'); + await expect( + new CodebaseSearcher(rootPath).search('quartzBeacon', 5, undefined, KEYWORD_ONLY_OPTIONS) + ).rejects.toThrow(/source line range is invalid.*rebuild required/i); + } + ); + + it('excludes stale ranges after a same-root source shift until an explicit refresh', async () => { + const rootPath = path.join(tempRoot, 'shifted source repo with spaces'); + const originalSource = [ + 'export function quartzBeacon() {', + ' return "original";', + '}', + 'quartzBeacon();', + '' + ].join('\n'); + await createSourceFile(rootPath, 'src/quartzBeacon.ts', originalSource); + await indexProject(rootPath); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + `// shifted header one\n// shifted header two\n${originalSource}` + ); + + // Owner-selected contract migration: ordinary drift is excluded with explicit + // coverage, while missing membership, malformed ranges and confinement remain errors. + const stale = await new CodebaseSearcher(rootPath).searchWithCoverage( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + expect(stale.results).toEqual([]); + expect(stale.sourceCoverage).toMatchObject({ status: 'no_fresh_results', skippedFiles: 1 }); + + await indexProject(rootPath); + const rebuiltResults = await new CodebaseSearcher(rootPath).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(rootPath, rebuiltResults, 'src/quartzBeacon.ts'); + expect(rebuiltResults.some((result) => result.startLine >= 3)).toBe(true); + }); + it('excludes a deleted top result while returning the valid lower file before and after refresh', async () => { + const rootPath = path.join(tempRoot, 'deleted source repo with spaces'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() {\n return "top";\n}\nquartzBeacon();\n' + ); + await createSourceFile( + rootPath, + 'src/support.ts', + 'export function useBeacon() {\n return quartzBeacon();\n}\n' + ); + await indexProject(rootPath); + await fs.rm(path.join(rootPath, 'src', 'quartzBeacon.ts')); + + const staleSearcher = new CodebaseSearcher(rootPath); + const current = await staleSearcher.searchWithCoverage( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(rootPath, current.results, 'src/support.ts'); + expect(current.sourceCoverage).toMatchObject({ status: 'partial', skippedFiles: 1 }); + + await indexProject(rootPath); + const rebuiltResults = await new CodebaseSearcher(rootPath).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(rootPath, rebuiltResults, 'src/support.ts'); + }); + + it('fails closed when the source manifest is missing, then recovers after a full rebuild', async () => { + const rootPath = path.join(tempRoot, 'missing manifest repo with spaces'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() {\n return "source";\n}\nquartzBeacon();\n' + ); + await indexProject(rootPath); + await fs.rm(path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME)); + + await expect( + new CodebaseSearcher(rootPath).search('quartzBeacon', 5, undefined, KEYWORD_ONLY_OPTIONS) + ).rejects.toBeInstanceOf(IndexCorruptedError); + + await indexProject(rootPath); + const rebuiltResults = await new CodebaseSearcher(rootPath).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(rootPath, rebuiltResults, 'src/quartzBeacon.ts'); + }); + + it('rejects traversal paths even when the outside target exists', async () => { + const rootPath = path.join(tempRoot, 'path traversal repo with spaces'); + const outsidePath = path.join(tempRoot, 'outside.ts'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() {\n return "inside";\n}\nquartzBeacon();\n' + ); + await createSourceFile( + tempRoot, + 'outside.ts', + 'export function quartzBeacon() {\n return "outside";\n}\nquartzBeacon();\n' + ); + await indexProject(rootPath); + + const indexPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const parsedIndex = JSON.parse(await fs.readFile(indexPath, 'utf8')) as { + header: { buildId: string; formatVersion: number }; + chunks: Array<{ content: string; filePath: string; relativePath: string }>; + }; + const candidateChunk = parsedIndex.chunks.find( + (chunk) => + path.resolve(rootPath, chunk.filePath) === path.join(rootPath, 'src', 'quartzBeacon.ts') + ); + if (!candidateChunk) throw new Error('Indexed source chunk missing from test fixture'); + candidateChunk.filePath = outsidePath; + candidateChunk.relativePath = '../outside.ts'; + await fs.writeFile(indexPath, JSON.stringify(parsedIndex), 'utf8'); + + await expect( + new CodebaseSearcher(rootPath).search('quartzBeacon', 5, undefined, KEYWORD_ONLY_OPTIONS) + ).rejects.toBeInstanceOf(IndexCorruptedError); + + await indexProject(rootPath); + const rebuiltResults = await new CodebaseSearcher(rootPath).search( + 'quartzBeacon', + 5, + undefined, + KEYWORD_ONLY_OPTIONS + ); + await expectSearchResultsOpenInsideRoot(rootPath, rebuiltResults, 'src/quartzBeacon.ts'); + expect( + rebuiltResults.every( + (result) => !path.resolve(result.filePath).startsWith(path.resolve(outsidePath)) + ) + ).toBe(true); + }); + it('finds new references in current parser-unavailable source even when stale chunks lack the symbol', async () => { + const rootPath = path.join(tempRoot, 'changed source repo with spaces'); + const originalSource = [ + 'export function staleToken() {', + ' return "old";', + '}', + 'staleToken();', + '' + ].join('\n'); + await createSourceFile(rootPath, 'src/widget.sfc', originalSource); + await indexProject(rootPath, ['.sfc']); + + const currentSource = [ + 'export function currentOnly() {', + ' return "new";', + '}', + 'currentOnly();', + '' + ].join('\n'); + await createSourceFile(rootPath, 'src/widget.sfc', currentSource); + + const staleSymbol = await findSymbolReferences(rootPath, 'staleToken'); + const newSymbol = await findSymbolReferences(rootPath, 'currentOnly'); + expect(staleSymbol.status).toBe('success'); + if (staleSymbol.status !== 'success') throw new Error('Expected a successful reference query'); + expect(staleSymbol.usageCount).toBe(0); + expect(staleSymbol.usages).toEqual([]); + expect(newSymbol.status).toBe('success'); + if (newSymbol.status !== 'success') throw new Error('Expected a successful reference query'); + expect(newSymbol.usageCount).toBe(2); + expect(newSymbol.usages.map((usage) => usage.line)).toEqual([1, 4]); + expect(newSymbol.usages.every((usage) => usage.file === 'src/widget.sfc')).toBe(true); + }); + + it('does not return symbol references through a directory junction outside the project', async () => { + const rootPath = path.join(tempRoot, 'symbol-junction-project'); + const outsidePath = path.join(tempRoot, 'symbol-external-source'); + await createSourceFile( + rootPath, + 'src/quartzBeacon.ts', + 'export function quartzBeacon() { return 1; }\nquartzBeacon();\n' + ); + await indexProject(rootPath); + await fs.rename(path.join(rootPath, 'src'), outsidePath); + await fs.symlink( + outsidePath, + path.join(rootPath, 'src'), + process.platform === 'win32' ? 'junction' : 'dir' + ); + + const result = await findSymbolReferences(rootPath, 'quartzBeacon'); + expect(result.status).toBe('error'); + if (result.status !== 'error') throw new Error('Expected source-confinement error'); + expect(result.message).toMatch(/outside the selected project/i); + }); +}); diff --git a/tests/server-config.test.ts b/tests/server-config.test.ts index eb7f6c6..e2ba8a2 100644 --- a/tests/server-config.test.ts +++ b/tests/server-config.test.ts @@ -113,10 +113,7 @@ describe('loadServerConfig', () => { const projA = path.join(os.tmpdir(), 'ccc-test-proj-a'); const projB = path.join(os.tmpdir(), 'ccc-test-proj-b'); const config = JSON.stringify({ - projects: [ - { root: projA, excludePatterns: ['**/dist/**'] }, - { root: projB } - ], + projects: [{ root: projA, excludePatterns: ['**/dist/**'] }, { root: projB }], server: { port: 5199, host: '0.0.0.0' } }); await withTempConfig(config, async (filePath) => { @@ -171,6 +168,26 @@ describe('loadServerConfig', () => { }); }); + it('preserves explicit per-project embedding mode, including false opt-in', async () => { + const config = JSON.stringify({ + projects: [ + { root: '~/model-free', skipEmbedding: true }, + { root: '~/semantic', skipEmbedding: false }, + { root: '~/invalid', skipEmbedding: 'false' } + ] + }); + + await withTempConfig(config, async (filePath) => { + process.env.CODEBASE_CONTEXT_CONFIG_PATH = filePath; + const result = await loadServerConfig(); + expect(result?.projects?.map((project) => project.skipEmbedding)).toEqual([ + true, + false, + undefined + ]); + }); + }); + it('loads parsing configuration for the requested CLI project root', async () => { const projectRoot = path.join(os.tmpdir(), 'ccc-large-cli-project'); const config = JSON.stringify({ diff --git a/tests/symbol-references.test.ts b/tests/symbol-references.test.ts new file mode 100644 index 0000000..bcf4b22 --- /dev/null +++ b/tests/symbol-references.test.ts @@ -0,0 +1,800 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { promises as fs } from 'fs'; +import os from 'os'; +import path from 'path'; +import { + CODEBASE_CONTEXT_DIRNAME, + INDEX_FORMAT_VERSION, + INDEX_META_FILENAME, + INDEX_META_VERSION, + KEYWORD_INDEX_FILENAME, + MANIFEST_FILENAME, + MEMORY_FILENAME +} from '../src/constants/codebase-context.js'; +import { CodebaseIndexer } from '../src/core/indexer.js'; +import { analyzerRegistry } from '../src/core/analyzer-registry.js'; +import { GenericAnalyzer } from '../src/analyzers/generic/index.js'; + +const { findIdentifierOccurrences } = vi.hoisted(() => ({ + findIdentifierOccurrences: vi.fn() +})); + +vi.mock('../src/utils/tree-sitter.js', async (importOriginal) => ({ + ...(await importOriginal()), + findIdentifierOccurrences +})); + +import { findSymbolReferences } from '../src/core/symbol-references.js'; + +if (!analyzerRegistry.get('generic')) analyzerRegistry.register(new GenericAnalyzer()); + +function createGate() { + let release!: () => void; + const promise = new Promise((resolve) => { + release = resolve; + }); + return { promise, release }; +} + +describe('findSymbolReferences', () => { + let rootPath: string | null = null; + + afterEach(async () => { + vi.restoreAllMocks(); + vi.clearAllMocks(); + if (rootPath) { + await fs.rm(rootPath, { recursive: true, force: true }); + rootPath = null; + } + }); + + async function createIndex(files: Array<{ path: string; exists?: boolean; source?: string }>) { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-symbol-references-')); + const chunks = []; + for (const entry of files) { + const absolutePath = path.join(rootPath, entry.path); + if (entry.exists !== false) { + await fs.mkdir(path.dirname(absolutePath), { recursive: true }); + await fs.writeFile(absolutePath, entry.source ?? '', 'utf8'); + } + chunks.push({ relativePath: entry.path, filePath: absolutePath }); + } + const contextPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + await fs.mkdir(contextPath, { recursive: true }); + await fs.writeFile( + path.join(contextPath, KEYWORD_INDEX_FILENAME), + JSON.stringify({ + header: { buildId: 'test', formatVersion: INDEX_FORMAT_VERSION }, + chunks + }), + 'utf8' + ); + await fs.writeFile( + path.join(contextPath, INDEX_META_FILENAME), + JSON.stringify({ + metaVersion: INDEX_META_VERSION, + formatVersion: INDEX_FORMAT_VERSION, + buildId: 'test', + generatedAt: new Date().toISOString(), + toolVersion: 'test', + artifacts: { + keywordIndex: { path: KEYWORD_INDEX_FILENAME }, + vectorDb: { path: 'index', provider: 'test' }, + manifest: { path: MANIFEST_FILENAME } + } + }), + 'utf8' + ); + await fs.writeFile( + path.join(contextPath, MANIFEST_FILENAME), + JSON.stringify({ + version: 1, + generatedAt: new Date().toISOString(), + header: { buildId: 'test', formatVersion: INDEX_FORMAT_VERSION }, + // Deliberately stale hashes prove source text, not stored hashes, decides matches. + files: Object.fromEntries(files.map((file) => [file.path, '0'.repeat(16)])) + }), + 'utf8' + ); + return rootPath; + } + + async function rewriteManifest(root: string, value: unknown): Promise { + const contextPath = path.join(root, CODEBASE_CONTEXT_DIRNAME); + await fs.writeFile(path.join(contextPath, MANIFEST_FILENAME), JSON.stringify(value), 'utf8'); + } + + async function rewriteManifestArtifactPath(root: string, manifestPath: string): Promise { + const metadataPath = path.join(root, CODEBASE_CONTEXT_DIRNAME, INDEX_META_FILENAME); + const meta = JSON.parse(await fs.readFile(metadataPath, 'utf8')) as { + artifacts: Record; + }; + meta.artifacts.manifest = { path: manifestPath }; + await fs.writeFile(metadataPath, JSON.stringify(meta), 'utf8'); + } + + it.each([ + ['missing keyword index header', (index: Record) => delete index.header], + [ + 'mismatched keyword index build identity', + (index: Record) => { + const header = index.header as Record; + index.header = { ...header, buildId: 'different-build' }; + } + ] + ])('rejects a real-built index with %s', async (_description, mutate) => { + findIdentifierOccurrences.mockResolvedValue([]); + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-symbol-index-identity-')); + const sourcePath = path.join(rootPath, 'src', 'identity.ts'); + await fs.mkdir(path.dirname(sourcePath), { recursive: true }); + await fs.writeFile(sourcePath, 'export function identity() { return identity; }\n', 'utf8'); + await new CodebaseIndexer({ + rootPath, + config: { skipEmbedding: true }, + projectOptions: { preferredAnalyzer: 'generic' } + }).index(); + + const keywordPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + const parsed = JSON.parse(await fs.readFile(keywordPath, 'utf8')) as Record; + mutate(parsed); + await fs.writeFile(keywordPath, JSON.stringify(parsed), 'utf8'); + + await expect(findSymbolReferences(rootPath, 'identity')).rejects.toThrow(/keyword index/i); + }); + + it('finds current references in files omitted by the keyword chunk cap', async () => { + findIdentifierOccurrences.mockResolvedValue([ + { line: 1, startIndex: 0, endIndex: 1, nodeType: 'identifier' } + ]); + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-symbol-index-cap-')); + const sources = new Map([ + ['src/a-first.ts', 'export function firstOnlyToken() { return 1; }\n'], + ['src/z-second.ts', 'export function secondOnlyToken() { return 2; }\n'] + ]); + for (const [relativePath, source] of sources) { + const absolutePath = path.join(rootPath, relativePath); + await fs.mkdir(path.dirname(absolutePath), { recursive: true }); + await fs.writeFile(absolutePath, source, 'utf8'); + } + + const stats = await new CodebaseIndexer({ + rootPath, + config: { skipEmbedding: true, parsing: { maxChunks: 1, maxFileSize: 1024 * 1024 } }, + projectOptions: { preferredAnalyzer: 'generic' } + }).index(); + + const contextPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + const keywordIndex = JSON.parse( + await fs.readFile(path.join(contextPath, KEYWORD_INDEX_FILENAME), 'utf8') + ) as { + header: { buildId: string; formatVersion: number }; + chunks: Array<{ relativePath?: string }>; + }; + const manifest = JSON.parse( + await fs.readFile(path.join(contextPath, MANIFEST_FILENAME), 'utf8') + ) as { + header?: { buildId: string; formatVersion: number }; + files: Record; + }; + const indexedPaths = new Set( + keywordIndex.chunks.flatMap((chunk) => (chunk.relativePath ? [chunk.relativePath] : [])) + ); + const omittedPath = Object.keys(manifest.files).find((file) => !indexedPaths.has(file)); + expect(stats.indexedFiles).toBe(2); + expect(stats.totalChunks).toBeGreaterThan(keywordIndex.chunks.length); + expect(keywordIndex.chunks).toHaveLength(1); + expect(manifest.header).toEqual(keywordIndex.header); + expect(omittedPath).toBeDefined(); + if (!omittedPath) throw new Error('Expected one manifest-listed file omitted by the chunk cap'); + + const omittedSymbol = new Map([ + ['src/a-first.ts', 'firstOnlyToken'], + ['src/z-second.ts', 'secondOnlyToken'] + ]).get(omittedPath); + expect(omittedSymbol).toBeDefined(); + if (!omittedSymbol) throw new Error(`Unexpected omitted source path: ${omittedPath}`); + + const result = await findSymbolReferences(rootPath, omittedSymbol); + + expect(result.status).toBe('success'); + if (result.status !== 'success') throw new Error('Expected successful current-source scan'); + expect(result.usageCount).toBe(1); + expect(result.usages).toMatchObject([{ file: omittedPath, line: 1 }]); + expect(result.isComplete).toBe(true); + expect(result.coverage).toMatchObject({ + candidateFiles: 2, + sourcesRead: 2, + filesUnavailable: 0 + }); + }); + + it('writes a build-bound manifest after an incremental add/change/delete pass', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-symbol-manifest-incremental-')); + const sourceFiles = new Map([ + ['src/keep.ts', 'export const beforeChange = 1;\n'], + ['src/remove.ts', 'export const removedSource = 1;\n'] + ]); + for (const [relativePath, source] of sourceFiles) { + const absolutePath = path.join(rootPath, relativePath); + await fs.mkdir(path.dirname(absolutePath), { recursive: true }); + await fs.writeFile(absolutePath, source, 'utf8'); + } + const config = { skipEmbedding: true, parsing: { maxChunks: 1, maxFileSize: 1024 * 1024 } }; + await new CodebaseIndexer({ + rootPath, + config, + projectOptions: { preferredAnalyzer: 'generic' } + }).index(); + + const contextPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + const firstMeta = JSON.parse( + await fs.readFile(path.join(contextPath, INDEX_META_FILENAME), 'utf8') + ) as { buildId: string; formatVersion: number }; + const firstManifest = JSON.parse( + await fs.readFile(path.join(contextPath, MANIFEST_FILENAME), 'utf8') + ) as { header?: { buildId: string; formatVersion: number } }; + expect(firstManifest.header).toEqual({ + buildId: firstMeta.buildId, + formatVersion: firstMeta.formatVersion + }); + + await fs.rm(path.join(rootPath, 'src', 'remove.ts')); + await fs.writeFile(path.join(rootPath, 'src', 'keep.ts'), 'export const changedToken = 2;\n'); + await fs.writeFile(path.join(rootPath, 'src', 'added.ts'), 'export const addedToken = 3;\n'); + const incremental = await new CodebaseIndexer({ + rootPath, + config, + projectOptions: { preferredAnalyzer: 'generic' }, + incrementalOnly: true + }).index(); + + const nextMeta = JSON.parse( + await fs.readFile(path.join(contextPath, INDEX_META_FILENAME), 'utf8') + ) as { buildId: string; formatVersion: number }; + const nextManifest = JSON.parse( + await fs.readFile(path.join(contextPath, MANIFEST_FILENAME), 'utf8') + ) as { header?: { buildId: string; formatVersion: number }; files: Record }; + expect(incremental.incremental).toMatchObject({ added: 1, changed: 1, deleted: 1 }); + expect(nextManifest.header).toEqual({ + buildId: nextMeta.buildId, + formatVersion: nextMeta.formatVersion + }); + expect(Object.keys(nextManifest.files).sort()).toEqual(['src/added.ts', 'src/keep.ts']); + }); + + it('does not stamp a headerless legacy manifest on a no-op incremental run', async () => { + rootPath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-symbol-manifest-legacy-')); + const sourcePath = path.join(rootPath, 'src', 'legacy.ts'); + await fs.mkdir(path.dirname(sourcePath), { recursive: true }); + await fs.writeFile(sourcePath, 'export const legacyToken = 1;\n', 'utf8'); + await new CodebaseIndexer({ + rootPath, + config: { skipEmbedding: true }, + projectOptions: { preferredAnalyzer: 'generic' } + }).index(); + + const contextPath = path.join(rootPath, CODEBASE_CONTEXT_DIRNAME); + const manifestPath = path.join(contextPath, MANIFEST_FILENAME); + const initialManifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as Record; + expect(initialManifest.header).toBeDefined(); + delete initialManifest.header; + await fs.writeFile(manifestPath, JSON.stringify(initialManifest), 'utf8'); + + await new CodebaseIndexer({ + rootPath, + config: { skipEmbedding: true }, + projectOptions: { preferredAnalyzer: 'generic' }, + incrementalOnly: true + }).index(); + + const afterIncremental = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as Record; + const meta = JSON.parse( + await fs.readFile(path.join(contextPath, INDEX_META_FILENAME), 'utf8') + ) as { artifacts: { vectorDb: { embeddingEnabled?: boolean } } }; + expect(afterIncremental.header).toBeUndefined(); + expect(meta.artifacts.vectorDb.embeddingEnabled).toBe(false); + await expect(findSymbolReferences(rootPath, 'legacyToken')).rejects.toThrow(/full reindex/i); + + const memoryPath = path.join(contextPath, MEMORY_FILENAME); + const memory = JSON.stringify([{ type: 'note', memory: 'preserve this note' }]); + await fs.writeFile(memoryPath, memory, 'utf8'); + await new CodebaseIndexer({ + rootPath, + config: { skipEmbedding: true }, + projectOptions: { preferredAnalyzer: 'generic' } + }).index(); + + const repairedManifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as { + header?: { buildId: string; formatVersion: number }; + }; + const repairedMeta = JSON.parse( + await fs.readFile(path.join(contextPath, INDEX_META_FILENAME), 'utf8') + ) as { buildId: string; formatVersion: number; artifacts: { vectorDb: { embeddingEnabled?: boolean } } }; + expect(repairedManifest.header).toEqual({ + buildId: repairedMeta.buildId, + formatVersion: repairedMeta.formatVersion + }); + expect(repairedMeta.artifacts.vectorDb.embeddingEnabled).toBe(false); + expect(await fs.readFile(memoryPath, 'utf8')).toBe(memory); + findIdentifierOccurrences.mockResolvedValue([ + { line: 1, startIndex: 0, endIndex: 1, nodeType: 'identifier' } + ]); + const repairedResult = await findSymbolReferences(rootPath, 'legacyToken'); + expect(repairedResult.status).toBe('success'); + if (repairedResult.status !== 'success') throw new Error('Expected reference lookup after full reindex'); + expect(repairedResult.usageCount).toBe(1); + }); + + it('requires a readable versioned manifest for reference coverage', async () => { + const root = await createIndex([{ path: 'src/a.ts', source: 'alpha();' }]); + await fs.rm(path.join(root, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME)); + + await expect(findSymbolReferences(root, 'alpha')).rejects.toThrow(/manifest.*full reindex/i); + }); + + it('requires a build-bound manifest instead of accepting the legacy headerless format', async () => { + const root = await createIndex([{ path: 'src/a.ts', source: 'alpha();' }]); + const manifestPath = path.join(root, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME); + const manifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as Record; + delete manifest.header; + await rewriteManifest(root, manifest); + + await expect(findSymbolReferences(root, 'alpha')).rejects.toThrow(/manifest.*full reindex/i); + }); + + it.each([ + ['buildId', { buildId: 'different-build', formatVersion: INDEX_FORMAT_VERSION }], + ['formatVersion', { buildId: 'test', formatVersion: INDEX_FORMAT_VERSION + 1 }] + ])('rejects manifest header with mismatched %s', async (_field, header) => { + const root = await createIndex([{ path: 'src/a.ts', source: 'alpha();' }]); + const manifestPath = path.join(root, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME); + const manifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as { + header: unknown; + [key: string]: unknown; + }; + manifest.header = header; + await rewriteManifest(root, manifest); + + await expect(findSymbolReferences(root, 'alpha')).rejects.toThrow(/manifest.*(buildId|formatVersion)/i); + }); + + it.each([ + ['null', null], + ['an array', []], + ['a malformed hash', { 'src/a.ts': 'not-a-hash' }] + ])('rejects manifest file entries supplied as %s', async (_description, files) => { + const root = await createIndex([{ path: 'src/a.ts', source: 'alpha();' }]); + const manifestPath = path.join(root, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME); + const manifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as { + [key: string]: unknown; + }; + manifest.files = files; + await rewriteManifest(root, manifest); + + await expect(findSymbolReferences(root, 'alpha')).rejects.toThrow(/manifest/i); + }); + + it('rejects manifest artifact metadata that traverses outside the context directory', async () => { + const root = await createIndex([{ path: 'src/a.ts', source: 'alpha();' }]); + await fs.writeFile( + path.join(root, MANIFEST_FILENAME), + await fs.readFile(path.join(root, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME), 'utf8'), + 'utf8' + ); + await rewriteManifestArtifactPath(root, `../${MANIFEST_FILENAME}`); + + await expect(findSymbolReferences(root, 'alpha')).rejects.toThrow(/manifest.*path/i); + }); + + it('rejects traversal in manifest source paths before scanning sources', async () => { + const root = await createIndex([{ path: 'src/a.ts', source: 'alpha();' }]); + const manifestPath = path.join(root, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME); + const manifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as { + [key: string]: unknown; + }; + manifest.files = { '../outside.ts': '0'.repeat(16) }; + await rewriteManifest(root, manifest); + + await expect(findSymbolReferences(root, 'alpha')).rejects.toThrow(/manifest.*path/i); + }); + + it('skips parser work when current source text cannot contain the symbol', async () => { + const root = await createIndex([{ path: 'src/a.ts', source: 'const beta = 1;' }]); + findIdentifierOccurrences.mockResolvedValue([]); + + const result = await findSymbolReferences(root, 'alpha'); + + expect(result.status).toBe('success'); + expect(findIdentifierOccurrences).not.toHaveBeenCalled(); + if (result.status !== 'success') throw new Error('Expected complete current-source scan'); + expect(result.coverage).toMatchObject({ + candidateFiles: 1, + sourcesRead: 1, + filesWithoutText: 1, + parserCalls: 0 + }); + }); + + it('skips longer-identifier candidates without changing exact references or text coverage', async () => { + const root = await createIndex([ + { path: 'src/long.ts', source: 'const SceneData = 1;' }, + { path: 'src/exact.ts', source: 'const Scene = 1; Scene();' } + ]); + findIdentifierOccurrences.mockResolvedValue([ + { line: 1, startIndex: 6, endIndex: 11, nodeType: 'identifier' } + ]); + + const result = await findSymbolReferences(root, 'Scene'); + + expect(result.status).toBe('success'); + if (result.status !== 'success') throw new Error('Expected successful current-source scan'); + expect(findIdentifierOccurrences).toHaveBeenCalledTimes(1); + expect(findIdentifierOccurrences).toHaveBeenCalledWith( + 'const Scene = 1; Scene();', + 'typescript', + 'Scene' + ); + expect(findIdentifierOccurrences).not.toHaveBeenCalledWith( + 'const SceneData = 1;', + 'typescript', + 'Scene' + ); + expect(result.usageCount).toBe(1); + expect(result.usages).toMatchObject([{ file: 'src/exact.ts', line: 1 }]); + expect(result.coverage).toMatchObject({ + candidateFiles: 2, + sourcesRead: 2, + filesWithoutText: 0, + parserCalls: 1 + }); + }); + + it('finds references added after indexing by checking current source', async () => { + const root = await createIndex([{ path: 'src/a.ts', source: 'const previousOnly = 1;' }]); + await fs.writeFile( + path.join(root, 'src/a.ts'), + 'const currentOnly = 1; currentOnly();', + 'utf8' + ); + findIdentifierOccurrences.mockResolvedValue(null); + + const result = await findSymbolReferences(root, 'currentOnly'); + + expect(result.status).toBe('success'); + if (result.status !== 'success') throw new Error('Expected successful current-source scan'); + expect(result.usageCount).toBe(2); + expect(result.coverage).toMatchObject({ sourcesRead: 1, parserCalls: 1 }); + }); + + it.each(['$alpha', 'προβολή'])( + 'matches the full identifier spelling %s in lexical fallback', + async (symbol) => { + const source = `x${symbol} suffix_${symbol} $${symbol} ${symbol} `; + const root = await createIndex([{ path: 'src/a.unknown', source }]); + findIdentifierOccurrences.mockResolvedValue(null); + + const result = await findSymbolReferences(root, symbol); + + expect(result.status).toBe('success'); + if (result.status !== 'success') throw new Error('Expected successful lexical scan'); + expect(result.usageCount).toBe(1); + expect(result.confidence).toBe('lexical'); + expect(result.coverage.lexicalFallbackFiles).toBe(1); + } + ); + + it('counts observed references beyond the usage limit and reports truncation', async () => { + const root = await createIndex([ + { path: 'src/a.unknown', source: 'activeToken(); activeToken(); activeToken();' } + ]); + findIdentifierOccurrences.mockResolvedValue(null); + + const result = await findSymbolReferences(root, 'activeToken', 2); + + expect(result.status).toBe('success'); + if (result.status !== 'success') throw new Error('Expected successful bounded scan'); + expect(result.usages).toHaveLength(2); + expect(result.usageCount).toBe(3); + expect(result.isComplete).toBe(false); + }); + + it('keeps observed references when an indexed source is unavailable', async () => { + const root = await createIndex([ + { path: 'src/missing.ts', exists: false }, + { path: 'src/readable.unknown', source: 'activeToken();' } + ]); + findIdentifierOccurrences.mockResolvedValue(null); + + const result = await findSymbolReferences(root, 'activeToken'); + + expect(result.status).toBe('success'); + if (result.status !== 'success') throw new Error('Expected partial successful scan'); + expect(result.usageCount).toBe(1); + expect(result.isComplete).toBe(false); + expect(result.coverage.filesUnavailable).toBe(1); + }); + + it('bounds source reads to four and preserves manifest order with sequential parsing', async () => { + const files = Array.from({ length: 9 }, (_, index) => ({ + path: `src/${index}.unknown`, + source: index < 2 ? 'previousOnly();' : `// file ${index}\nπροβολή();\n` + })); + const root = await createIndex(files); + await fs.writeFile(path.join(root, files[0].path), '// current source\r\nπροβολή();\r\n'); + // Every other source is manifest-only; capped keyword chunks are not the inventory. + await fs.writeFile( + path.join(root, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME), + JSON.stringify({ + header: { buildId: 'test', formatVersion: INDEX_FORMAT_VERSION }, + chunks: [{ relativePath: files[0].path }] + }) + ); + const sourcePaths = new Map(files.map((file, index) => [path.join(root, file.path), index])); + const gates = files.map(() => createGate()); + const parserGate = createGate(); + const parserStarted = createGate(); + const readsStarted: number[] = []; + const readsCompleted: number[] = []; + let activeReads = 0; + let maxActiveReads = 0; + let activeParsers = 0; + let maxActiveParsers = 0; + const originalReadFile = fs.readFile; + vi.spyOn(fs, 'readFile').mockImplementation(async (...args) => { + const index = typeof args[0] === 'string' ? sourcePaths.get(args[0]) : undefined; + if (index === undefined) return originalReadFile(...args); + readsStarted.push(index); + activeReads += 1; + maxActiveReads = Math.max(maxActiveReads, activeReads); + try { + await gates[index].promise; + const content = await originalReadFile(...args); + readsCompleted.push(index); + return content; + } finally { + activeReads -= 1; + } + }); + findIdentifierOccurrences.mockImplementation(async () => { + activeParsers += 1; + maxActiveParsers = Math.max(maxActiveParsers, activeParsers); + parserStarted.release(); + await parserGate.promise; + activeParsers -= 1; + return null; + }); + const scan = findSymbolReferences(root, 'προβολή', 3); + try { + await vi.waitFor(() => expect(readsStarted).toHaveLength(4)); + expect([...readsStarted].sort()).toEqual([0, 1, 2, 3]); + for (const index of [3, 2, 1, 0]) { + gates[index].release(); + await vi.waitFor(() => expect(readsCompleted).toContain(index)); + } + await parserStarted.promise; + // Completed buffers remain bounded while the shared parser is busy. + expect(readsStarted).toHaveLength(4); + expect(findIdentifierOccurrences).toHaveBeenCalledTimes(1); + parserGate.release(); + for (const gate of gates) gate.release(); + const result = await scan; + + expect(maxActiveReads).toBe(4); + expect(maxActiveParsers).toBe(1); + expect(readsCompleted.slice(0, 4)).toEqual([3, 2, 1, 0]); + expect(new Set(readsStarted).size).toBe(9); + expect(result.status).toBe('success'); + if (result.status !== 'success') throw new Error('Expected bounded current-source scan'); + expect(result.usageCount).toBe(8); + expect(result.usages.map((usage) => usage.file)).toEqual([ + 'src/0.unknown', 'src/2.unknown', 'src/3.unknown' + ]); + expect(result.usages[0]).toEqual({ + file: 'src/0.unknown', line: 2, preview: '// current source\nπροβολή();' + }); + expect(result.coverage).toEqual({ + candidateFiles: 9, sourcesRead: 9, filesWithoutText: 1, + filesUnavailable: 0, parserCalls: 8, lexicalFallbackFiles: 8 + }); + expect(result.confidence).toBe('lexical'); + expect(result.isComplete).toBe(false); + } finally { + for (const gate of gates) gate.release(); + parserGate.release(); + await scan; + findIdentifierOccurrences.mockReset(); + } + }); + + it.each(['ENOENT', 'ENOTDIR', 'EACCES', 'EPERM'])( + 'keeps full observed counts across batches when a source read reports %s', + async (code) => { + const files = Array.from({ length: 6 }, (_, index) => ({ + path: `src/${index}.unknown`, source: 'alpha();' + })); + const root = await createIndex(files); + const unavailable = path.join(root, files[1].path); + const originalReadFile = fs.readFile; + vi.spyOn(fs, 'readFile').mockImplementation(async (...args) => { + if (args[0] === unavailable) throw Object.assign(new Error('unavailable'), { code }); + return originalReadFile(...args); + }); + findIdentifierOccurrences.mockResolvedValue(null); + + const result = await findSymbolReferences(root, 'alpha', 1); + + expect(result.status).toBe('success'); + if (result.status !== 'success') throw new Error('Expected partial current-source scan'); + expect(result.usageCount).toBe(5); + expect(result.usages.map((usage) => usage.file)).toEqual(['src/0.unknown']); + expect(result.coverage).toMatchObject({ + candidateFiles: 6, sourcesRead: 5, filesUnavailable: 1, parserCalls: 5 + }); + expect(result.isComplete).toBe(false); + } + ); + + it.each(['realpath', 'stat', 'readFile'] as const)( + 'fails the entire batch closed on an unexpected %s error without parsing it', + async (operation) => { + const files = Array.from({ length: 5 }, (_, index) => ({ + path: `src/${index}.unknown`, source: 'alpha();' + })); + const root = await createIndex(files); + const fatal = path.join(root, files[1].path); + const later = path.join(root, files[4].path); + const originalReadFile = fs.readFile; + const laterReads: unknown[] = []; + // The operation-specific spies retain the real fs signatures. + if (operation === 'realpath') { + const realpath = fs.realpath; + vi.spyOn(fs, 'realpath').mockImplementation(async (...args) => { + if (args[0] === fatal) throw Object.assign(new Error('fatal'), { code: 'EIO' }); + return realpath(...args); + }); + } else if (operation === 'stat') { + const stat = fs.stat; + vi.spyOn(fs, 'stat').mockImplementation(async (...args) => { + if (args[0] === fatal) throw Object.assign(new Error('fatal'), { code: 'EIO' }); + return stat(...args); + }); + } + vi.spyOn(fs, 'readFile').mockImplementation(async (...args) => { + if (args[0] === later) laterReads.push(args[0]); + if (operation === 'readFile' && args[0] === fatal) { + throw Object.assign(new Error('fatal'), { code: 'EIO' }); + } + return originalReadFile(...args); + }); + findIdentifierOccurrences.mockResolvedValue(null); + + const result = await findSymbolReferences(root, 'alpha'); + + expect(result.status).toBe('error'); + if (result.status !== 'error') throw new Error('Expected fail-closed scan'); + expect(result.message).toContain(files[1].path); + expect(findIdentifierOccurrences).not.toHaveBeenCalled(); + expect(laterReads).toEqual([]); + } + ); + + it('settles pending source read failures before returning a fatal batch', async () => { + const files = Array.from({ length: 5 }, (_, index) => ({ + path: `src/${index}.unknown`, source: 'alpha();' + })); + const root = await createIndex(files); + const first = path.join(root, files[0].path); + const pending = path.join(root, files[1].path); + const pendingRelease = createGate(); + const originalReadFile = fs.readFile; + let pendingStarted = false; + let pendingSettled = false; + let scanSettled = false; + vi.spyOn(fs, 'readFile').mockImplementation(async (...args) => { + if (args[0] === first) throw Object.assign(new Error('first failure'), { code: 'EIO' }); + if (args[0] === pending) { + pendingStarted = true; + await pendingRelease.promise; + pendingSettled = true; + throw Object.assign(new Error('later failure'), { code: 'EMFILE' }); + } + return originalReadFile(...args); + }); + findIdentifierOccurrences.mockResolvedValue(null); + const scan = findSymbolReferences(root, 'alpha').finally(() => { scanSettled = true; }); + try { + await vi.waitFor(() => expect(pendingStarted).toBe(true)); + expect(scanSettled).toBe(false); + expect(pendingSettled).toBe(false); + pendingRelease.release(); + const result = await scan; + expect(result.status).toBe('error'); + if (result.status !== 'error') throw new Error('Expected fail-closed scan'); + expect(result.message).toContain(files[0].path); + expect(pendingSettled).toBe(true); + expect(findIdentifierOccurrences).not.toHaveBeenCalled(); + } finally { + pendingRelease.release(); + await scan; + } + }); + + it('fails a batch closed when a manifest-listed source becomes a directory', async () => { + const files = Array.from({ length: 5 }, (_, index) => ({ + path: `src/${index}.unknown`, source: 'alpha();' + })); + const root = await createIndex(files); + const invalidSource = path.join(root, files[1].path); + await fs.unlink(invalidSource); + await fs.mkdir(invalidSource); + findIdentifierOccurrences.mockResolvedValue(null); + + const result = await findSymbolReferences(root, 'alpha'); + + expect(result.status).toBe('error'); + if (result.status !== 'error') throw new Error('Expected file-integrity error'); + expect(result.message).toContain(files[1].path); + expect(findIdentifierOccurrences).not.toHaveBeenCalled(); + }); + + it('fails closed when an indexed source resolves outside the project', async () => { + const root = await createIndex([{ path: 'src/a.ts', source: 'alpha();' }]); + const outsidePath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-symbol-outside-')); + try { + const externalFile = path.join(outsidePath, 'external.ts'); + await fs.writeFile(externalFile, 'alpha();', 'utf8'); + const indexPath = path.join(root, CODEBASE_CONTEXT_DIRNAME, KEYWORD_INDEX_FILENAME); + await fs.writeFile( + indexPath, + JSON.stringify({ + header: { buildId: 'test', formatVersion: INDEX_FORMAT_VERSION }, + chunks: [{ filePath: externalFile }] + }), + 'utf8' + ); + + const result = await findSymbolReferences(root, 'alpha'); + + expect(result.status).toBe('error'); + if (result.status !== 'error') throw new Error('Expected path-confinement error'); + expect(result.message).toMatch(/invalid|outside/i); + } finally { + await fs.rm(outsidePath, { recursive: true, force: true }); + } + }); + + it('fails closed when a manifest-listed source symlink resolves outside the project', async ({ skip }) => { + const root = await createIndex([{ path: 'src/anchor.ts', source: 'safeOnly();' }]); + const outsidePath = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-symbol-manifest-link-')); + try { + const externalFile = path.join(outsidePath, 'external.ts'); + await fs.writeFile(externalFile, 'export const outsideOnly = 1;\n', 'utf8'); + const linkPath = path.join(root, 'src', 'link.ts'); + await fs.mkdir(path.dirname(linkPath), { recursive: true }); + try { + await fs.symlink(externalFile, linkPath, 'file'); + } catch (error) { + const code = error !== null && typeof error === 'object' && 'code' in error ? error.code : undefined; + if (code === 'EPERM' || code === 'EACCES' || code === 'ENOSYS') { + skip(`Symlink creation is unavailable in this environment (${String(code)})`); + } + throw error; + } + const manifestPath = path.join(root, CODEBASE_CONTEXT_DIRNAME, MANIFEST_FILENAME); + const manifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as { + files: Record; + [key: string]: unknown; + }; + manifest.files['src/link.ts'] = '0'.repeat(16); + await rewriteManifest(root, manifest); + + const result = await findSymbolReferences(root, 'outsideOnly'); + + expect(result.status).toBe('error'); + if (result.status !== 'error') throw new Error('Expected symlink confinement error'); + expect(result.message).toMatch(/outside/i); + } finally { + await fs.rm(outsidePath, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/tools/dispatch.test.ts b/tests/tools/dispatch.test.ts index d51f313..366d642 100644 --- a/tests/tools/dispatch.test.ts +++ b/tests/tools/dispatch.test.ts @@ -1,10 +1,11 @@ import { describe, it, expect } from 'vitest'; +import { fileURLToPath } from 'node:url'; import { TOOLS, dispatchTool } from '../../src/tools/index.js'; import type { ToolContext } from '../../src/tools/types.js'; describe('Tool Dispatch', () => { - it('exports all 11 tools', () => { - expect(TOOLS.length).toBe(11); + it('exports all 12 tools including source relationship navigation', () => { + expect(TOOLS.length).toBe(12); expect(TOOLS.map((t) => t.name)).toEqual([ 'search_codebase', 'get_codebase_metadata', @@ -16,7 +17,8 @@ describe('Tool Dispatch', () => { 'detect_circular_dependencies', 'remember', 'get_memory', - 'get_codebase_health' + 'get_codebase_health', + 'get_code_relationships' ]); }); @@ -96,5 +98,11 @@ describe('Tool Dispatch', () => { expect(result.content![0].type).toBe('text'); const text = result.content![0].text; expect(text).toContain('idle'); + const status = JSON.parse(text) as { rootPath: string; loadedStatusModulePath: string }; + expect(status.rootPath).toBe('/tmp'); + expect(status.loadedStatusModulePath).toBe( + fileURLToPath(new URL('../../src/tools/get-indexing-status.ts', import.meta.url)) + ); + expect(status.loadedStatusModulePath).not.toBe(status.rootPath); }); }); diff --git a/tests/tree-sitter-symbols.test.ts b/tests/tree-sitter-symbols.test.ts index b8ce9dd..9511df2 100644 --- a/tests/tree-sitter-symbols.test.ts +++ b/tests/tree-sitter-symbols.test.ts @@ -1,6 +1,11 @@ import { describe, expect, it } from 'vitest'; import { GenericAnalyzer } from '../src/analyzers/generic/index'; -import { extractTreeSitterSymbols, supportsTreeSitter } from '../src/utils/tree-sitter'; +import structuralFixture from './fixtures/structural-context-regressions.json'; +import { + extractTreeSitterSymbols, + findIdentifierOccurrences, + supportsTreeSitter +} from '../src/utils/tree-sitter'; describe('Tree-sitter symbol extraction', () => { it('extracts TypeScript symbols including function variables', async () => { @@ -80,4 +85,70 @@ describe('Tree-sitter symbol extraction', () => { it('reports unsupported language grammars', () => { expect(supportsTreeSitter('markdown')).toBe(false); }); + + it('matches complete dollar, underscore, and Unicode identifiers in parsed code', async () => { + const source = [ + 'const alpha = $alpha;', + 'const alphaSuffix = 1;', + 'const _alpha = 2;', + 'const προβολή = alpha;', + '// alpha and $alpha are comments', + 'const message = "alpha $alpha";' + ].join('\n'); + + const alpha = await findIdentifierOccurrences(source, 'typescript', 'alpha'); + const dollar = await findIdentifierOccurrences(source, 'typescript', '$alpha'); + const unicode = await findIdentifierOccurrences(source, 'typescript', 'προβολή'); + + expect(alpha?.map((occurrence) => occurrence.line)).toEqual([1, 4]); + expect(dollar?.map((occurrence) => occurrence.line)).toEqual([1]); + expect(unicode?.map((occurrence) => occurrence.line)).toEqual([4]); + }); +}); + +describe('frozen structural source facts', () => { + it('does not connect static ancestors across a dynamic method or field boundary', async () => { + const cases = [ + 'class C { [name](){ const inner=()=>run(); } }', + 'class C { [name]=()=>{ const inner=()=>run(); }; }' + ]; + for (const source of cases) { + const parsed = await extractTreeSitterSymbols(source, 'javascript'); + expect(parsed).not.toBeNull(); + const inner = parsed!.symbols.find((symbol) => symbol.name === 'inner') as + | (NonNullable['symbols'][number] & { symbolPath?: string[] }) + | undefined; + expect(inner).toBeDefined(); + expect(inner!.symbolPath).toEqual(['inner']); + } + }); + + for (const fixture of structuralFixture.cases) { + it(`preserves exact static ownership and implementation: ${fixture.id}`, async () => { + const extracted = await extractTreeSitterSymbols(fixture.source, fixture.language); + expect(extracted).not.toBeNull(); + for (const fact of fixture.sourceSymbols) { + if ('scopeOnly' in fact && fact.scopeOnly) { + expect(extracted!.symbols.some((symbol) => symbol.name.includes('[actionName]'))).toBe( + false + ); + continue; + } + const symbol = extracted!.symbols.find((candidate) => { + const sourceSymbol = candidate as typeof candidate & { symbolPath?: string[] }; + return JSON.stringify(sourceSymbol.symbolPath) === JSON.stringify(fact.ownerChain); + }); + expect(symbol, `${fixture.id}: ${fact.ownerChain.join(' > ')}`).toBeDefined(); + for (const marker of fact.bodyMarkers) expect(symbol!.content).toContain(marker); + if ('forbiddenSiblingBodyMarkers' in fact) { + for (const marker of fact.forbiddenSiblingBodyMarkers ?? []) { + expect(symbol!.content).not.toContain(marker); + } + } + expect(fixture.source).toContain(symbol!.content); + expect(symbol!.startLine).toBeGreaterThanOrEqual(1); + expect(symbol!.endLine).toBeLessThanOrEqual(fixture.source.split('\n').length); + } + }); + } });