From 3a9223285bcf1d1ce48c5cbe9a81e373752611cc Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 22:11:03 -0400 Subject: [PATCH 01/29] docs: design record for v1 -> v2 schema migration (L1 + L2) Roadmap, cross-repo spec, schema decisions, and drafted parent/child issue bodies for migrating codeanalyzer-go and its python-sdk Go models to the v2 canonical schema. Reach: L1 + L2 to parity; L3/L4 deferred to a later train. Group A vocabulary (can:// ids, span.bytes UTF-8) decided once here per the parity clause, since L3/L4 reuse it verbatim. Full transcript in docs/design/specs/v2-l1-emission.md and CLAUDE.md. --- CLAUDE.md | 47 ++++++++++++ docs/design/issues/child-1-l1-emission.md | 40 ++++++++++ docs/design/issues/parent-v2-migration.md | 41 +++++++++++ docs/design/roadmap.md | 77 +++++++++++++++++++ docs/design/specs/v2-l1-emission.md | 90 +++++++++++++++++++++++ 5 files changed, 295 insertions(+) create mode 100644 CLAUDE.md create mode 100644 docs/design/issues/child-1-l1-emission.md create mode 100644 docs/design/issues/parent-v2-migration.md create mode 100644 docs/design/roadmap.md create mode 100644 docs/design/specs/v2-l1-emission.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..1694459 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,47 @@ +# codeanalyzer-go — agent notes + +The Go static-analysis backend for CLDK (`cango`). Emits the canonical CLDK +`analysis.json`. Parser/resolver/call-graph builder compute the facts; the emission +layer serializes them into the schema shape. + +## Schema decisions + +Decisions made with the user during the v1 → v2 migration design loop +(`designing-cldk-changes`), spine order. Each is a leaf-level addition to the shared +v2 vocabulary (parity clause: add at the leaves, never rename shared names). The +committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. + +### Identity & positioning (Group A — reused verbatim by L3/L4) +- **`application.id`** = `can://go/`, where `` is a slug derived from the + **go.mod module path** (deterministic, no flag). L3/L4 build ids on this unchanged. +- **Callable signature** = existing `signatureOf()` output as the last `can://` path + segment; receiver folded in for methods. One canonicalizer, unchanged from v1. +- **`span.bytes`** = **UTF-8 byte offsets** into `module.source` (Go strings are + natively UTF-8; avoids the UTF-16/rune slicing mismatch TS hit in cants#179). + +### `type` node +- **`kind`** ∈ `struct | interface | alias | defined` — Go's four type-declaration + shapes. Collapses v1's `is_interface` boolean. (`alias` = `type X = Y`; + `defined` = `type X Y` over a non-struct, e.g. `type Celsius float64`.) +- **Methods** (declared outside the type in Go source) are resolved to their receiver + type and placed in that `type.callables{}`; receiver recorded as a field. The tree + reflects containment, not source location. +- **Embedding vs satisfaction:** `base_types[]` = embedded type ids (struct/interface + embedding — the explicit spine); `interfaces[]` = interfaces the type is *computed* + to satisfy via method-set matching (Go's implicit/structural implementation). + +### `callable` node +- **`error_channel[]`** populated from `error`-typed return values (Go's error idiom); + those returns remain in `return_type` too. Maps Go onto the shared field. +- **Closures / function literals** → nested `callables{}` on the enclosing callable + (replaces v1 `InnerCallables`); each closure gets its own `can://` id. + +### `call` body node +- Typed **`is_goroutine`** and **`is_deferred`** boolean fields (`defer` is net-new vs + v1). `is_constructor_call` **dropped** — Go has no constructors. +- `callee` is the sanctioned `null → id` refinement slot (null at L1, backfilled at L2). + +### Deferred (not this train) +- L3 (CFG/CDG/DDG) and L4 (SDG param edges) — reuse Group A ids/spans unchanged. +- Go-specific `cfg` edge kinds (e.g. `defer_resume`, `select`) are recorded when L3 + is designed, not now. diff --git a/docs/design/issues/child-1-l1-emission.md b/docs/design/issues/child-1-l1-emission.md new file mode 100644 index 0000000..7cb806d --- /dev/null +++ b/docs/design/issues/child-1-l1-emission.md @@ -0,0 +1,40 @@ + + +**Is your feature request related to a problem? Please describe.** + +The Go analyzer serializes the v1 shape. L1 is the structural keystone of the v2 +migration: it introduces the additive containment tree and every Group A identity +decision the later levels build on. Do L1 first and get the symbol-table gate green +before touching the L2 call graph. The parser/resolver stay; only the emission layer +changes (golden rule: replace what serializes, keep what computes). + +**Describe the solution you'd like.** + +- [ ] `application.id` = `can://go/`, `` derived from the go.mod module path +- [ ] Durable `can://` ids on module/type/callable; `signatureOf()` is the last segment +- [ ] `span: {start:[l,c], end:[l,c], bytes:[from,to]}` with UTF-8 byte offsets; drop flat `start_line`/`end_line` +- [ ] `source` emitted once per module; drop per-callable `code` +- [ ] `type.kind` ∈ `struct|interface|alias|defined`; methods grouped under receiver type's `callables{}` +- [ ] `base_types[]` = embedded ids, `interfaces[]` = computed structural satisfaction +- [ ] `body{}` holds `call` nodes with `callee:null`, typed `is_goroutine`/`is_deferred`; closures nest in `callables{}` + +**Describe alternatives you've considered.** + +Does NOT emit the `call_graph` (that is the L2 child), nor any L3/L4 `body` statements +or edges. `body{}` at L1 holds only `call` nodes. + +**Additional context.** + +- Emitter approach: walk the existing in-memory model, add a v2 emitter behind a flag; keep v1 emitter as the compat shim during transition. +- One genuinely new datum: thread parser byte offsets into `span.bytes`. +- Structural interface satisfaction needs method-set matching — cost lives here, not in the resolver. +- Validate each node against the SDK v2 Go models as they land; superset check vs v1 output. +- Schema decisions recorded in repo `CLAUDE.md`; full transcript in the spec. diff --git a/docs/design/issues/parent-v2-migration.md b/docs/design/issues/parent-v2-migration.md new file mode 100644 index 0000000..3496ca0 --- /dev/null +++ b/docs/design/issues/parent-v2-migration.md @@ -0,0 +1,41 @@ + + +**Is your feature request related to a problem? Please describe.** + +`codeanalyzer-go` emits the v1 schema (`internal/schema/schema.go`): two-key +`{symbol_table, call_graph}`, flat `start_line`/`end_line`, per-callable `code`, +`is_*` booleans, `signature` as id. Every sibling analyzer has moved to the v2 +keystone (`can://` ids, additive tree, `span` with byte offsets, `source` per module, +`{src,dst,prov}` edges). Go is the last on v1, so `CLDK(language="go")` cannot share +the one-model SDK surface. This is a coordinated schema major across two repos: +`codeanalyzer-go` (emission rewrite) and `python-sdk` (Go model remap), released in +lockstep. Reach for this train: L1 + L2 to parity; L3/L4 deferred. + +**Describe the solution you'd like.** + +- [ ] `codeanalyzer-go`: L1 emission — additive tree, `source` per module, `can://` ids, `body{}` with `call` nodes +- [ ] `codeanalyzer-go`: L2 emission — `call_graph: [{src,dst,prov,weight}]` at application scope +- [ ] `codeanalyzer-go`: Neo4j projection relabelled to v2 node/edge families +- [ ] `python-sdk`: Go v2 model remap, every public accessor name/return type unchanged +- [ ] Release v2.0.0 and pin analyzer in SDK **only once both are cut** +- [ ] Superset gate green: v2 output contains every v1 fact modulo sanctioned drops + +**Describe alternatives you've considered.** + +Does NOT build L3 (CFG/CDG/DDG) or L4 (SDG). Those are a later train reusing this +train's `can://` ids and edge shape unchanged. Does NOT add framework detection. + +**Additional context.** + +- Design transcript: `docs/design/specs/v2-l1-emission.md` (linked, not pasted). +- Group A vocabulary (`can://` ids, `span.bytes` UTF-8) is reused verbatim by L3/L4 — coined once here per the parity clause. +- Sanctioned drops: `code` (slice from `source`), `is_*` booleans (folded into `kind`), `is_constructor_call` (Go has no constructors). +- Ordering risk: SDK's v1 Go models will not parse v2 output; do not pin the new major prematurely. +- go.mod-derived `` slug must be deterministic across runs, else ids churn. diff --git a/docs/design/roadmap.md b/docs/design/roadmap.md new file mode 100644 index 0000000..28e3ada --- /dev/null +++ b/docs/design/roadmap.md @@ -0,0 +1,77 @@ +# CLDK Go analyzer — roadmap + +> Local staging copy. Canonical home is `codellm-devkit/.github → docs/design/roadmap.md`; +> port there when that repo is available. One roadmap, amended in place — never a second file. + +## Initiative: v1 → v2 schema migration (codeanalyzer-go + python-sdk) + +A schema **major**: move the Go analyzer off the v1 keystone +(`{symbol_table, call_graph}`, flat `start_line`/`end_line`, per-callable `code`, `is_*` +booleans, `signature` as id) onto the v2 canonical schema (additive tree, `can://` ids, +`span` with byte offsets, `source` per module, typed edge lists). The SDK moves in +lockstep as a coordinated major release. **Reach for this train: L1 + L2 to parity.** +L3/L4 deferred (see Not now). + +Golden rule carried from `schema-migration.md`: keep everything that *computes* facts +(parser, resolver, call-graph builder); replace only what *serializes* them. + +## Starting now (exactly one) + +- **Group A + L1 emission** — the identity, span, and `body`-with-call-nodes decisions. + This is the keystone rung; everything below reuses its vocabulary. → `designing-cldk-changes`. + +Everything else below has no issue filed yet, by design. + +## Collision groups (the reason this is planning, not design) + +Contract decisions that share v2 vocabulary — **each group is one design session**, even +when its members ship in different release trains. Coining a shared term twice coins it +wrong permanently (parity clause). + +- **A — Identity & positioning.** `application.id` (`can:///`), the `can://` + durable-id grammar (≥ callable), the `@line:col` ordinal ids (< callable), and `span` + with byte offsets. **L3/L4 in a later train reuse this id/span shape verbatim** — so it + must be settled now, in one session, not re-decided per level. This is the group the + gate exists to protect. +- **B — Node shape.** `type.kind` collapsing the `is_interface`/`is_enum`/… booleans; + structured `decorators[]` (`{name,args,span}`) replacing flat annotation strings; + generalized `error_channel[]`; drop per-callable `code`, add `source` once per module. +- **C — Edge shape.** Rename edge keys `source/target/type` → `src/dst/prov`; `call_graph` + as a list at application scope. **Every future edge family (cfg/cdg/ddg/param_*) inherits + this record shape** — coin `{src, dst, prov, weight}` correctly now. +- **D — Neo4j projection.** Node/edge family names must match A/B/C exactly; it is a + relabel of the same tree, always full-depth. + +## Dependency order (DAG) + +``` +A (identity + span) ─┬─► L1 emission (tree, source, ids, body-with-call-nodes) +B (node shape) ─┤ +C (edge shape) ─┴─► L2 emission (call_graph rename @ application scope) + │ +A/B/C ─► D (Neo4j relabel) ─► SDK v2 model remap ─► pin SDK→analyzer (once BOTH cut) + │ + (later train) L3 ─► L4 ──────┘ reuse A & C unchanged +``` + +## Release trains + +- **Train 1 — v2.0.0 (coordinated major, lockstep).** Analyzer L1+L2 emission rewrite + + Neo4j relabel; SDK v2 model remap keeping every public accessor name/return type + identical (two-layer model). **Ordering constraint (first-class on the parent):** pin the + analyzer version in the SDK *only once both are released* — until then the SDK's old + models won't parse v2 output. Superset gate: v2 output must contain every fact v1 did, + modulo the sanctioned drops (`code`, `is_*` booleans folded into `kind`). +- **Train 2+ (deferred).** L3 (CFG/CDG/DDG, AST-only, per-callable parallel — net-new for + Go) then L4 (SDG `param_in`/`param_out`/`summary`, needs a points-to oracle). Both reuse + group A & C vocabulary; no new schema-major coordination beyond what Train 1 establishes. + +## Not now (considered, deliberately excluded) + +- **L3 / L4 dataflow** — deferred by decision this session; net-new heavy construction, not + required for v2 parity. Revisit as Train 2. +- **Points-to oracle** — L4 prerequisite; parked with L4. +- **`--materialize-expressions` / `--materialize-basic-blocks`** — optional body node kinds; + out of scope for the parity migration. +- **New framework detection / CodeQL tier expansion** — orthogonal enrichment axis + (provenance-merged evidence), not a schema level; does not belong on the migration train. diff --git a/docs/design/specs/v2-l1-emission.md b/docs/design/specs/v2-l1-emission.md new file mode 100644 index 0000000..209bf07 --- /dev/null +++ b/docs/design/specs/v2-l1-emission.md @@ -0,0 +1,90 @@ +# Spec: v2 schema migration — Group A + L1 emission (codeanalyzer-go) + +> **Cross-repo spec.** Canonical home is `codellm-devkit/.github → docs/design/specs/`, +> beside the coordinating parent issue. Staged locally until that repo is available. +> This is the design transcript — the committed provenance of the design loop, linked +> (not pasted) from the parent issue. + +## Contract-Impact Triage + +**Does this change schema v2 output?** Yes — it is the schema-major migration that +introduces the v2 shape for Go: `application.id`, the `can://` id grammar, `@line:col` +ordinal ids (defined here, populated by L3/L4), `span` with UTF-8 byte offsets, the +additive tree, `source` per module, and `body{}` with `call` nodes. + +| Change type | Analyzers | SDKs | Docs | +| --- | --- | --- | --- | +| Schema v2 migration | **codeanalyzer-go** (emission rewrite) | **python-sdk** (Go model remap, lockstep) | CLAUDE.md, this spec | + +Arrived from `planning-cldk-work` with collision **Group A** known (identity + span); +its vocabulary is designed once here and reused verbatim by the deferred L3/L4 train. + +## Design-loop decisions (the transcript) + +Every decision below was made with the user, node by node in spine order. Full copy in +`codeanalyzer-go/CLAUDE.md` § Schema decisions. Anchored on the keystone +(`canonical-schema.md`) and the completed **TypeScript** v2 migration (closest structural +analog: module functions + interfaces + structural types). + +### Root envelope (settled spine, kept as-is) +`{ schema_version:"2.0.0", language:"go", max_level, analyzer:{name,version}, +application:{ id, kind:"application", symbol_table, call_graph, param_in, param_out } }`. + +### Identity & positioning (Group A) +| Decision | Choice | +| --- | --- | +| `application.id` | `can://go/`, `` = slug from **go.mod module path** (no flag) | +| Callable signature | existing `signatureOf()` as last id segment; receiver folded in for methods | +| `span.bytes` | **UTF-8 byte offsets** into `module.source` | + +### `module` +`kind:"module"`, `package`, `source` (whole file once), `imports[]`, `types{}`, +`functions{}`, `content_hash`. Drops per-callable `code` (sliced from `source`). + +### `type` +| Decision | Choice | +| --- | --- | +| `kind` | `struct \| interface \| alias \| defined` (collapses v1 `is_interface`) | +| Method placement | resolved to receiver type's `callables{}`; receiver as a field | +| `base_types[]` | embedded type ids (struct/interface embedding) | +| `interfaces[]` | computed structural satisfaction (method-set matching) | + +### `callable` +| Decision | Choice | +| --- | --- | +| `error_channel[]` | from `error`-typed returns (kept in `return_type` too) | +| Closures | nested `callables{}` on enclosing callable (replaces `InnerCallables`) | + +### `call` (body node, L1) +| Decision | Choice | +| --- | --- | +| Markers | typed `is_goroutine`, `is_deferred`; **drop** `is_constructor_call` | +| `callee` | sanctioned `null → id` refinement slot (null at L1, backfilled L2) | + +### Edges (Group C, settled here for L2) +`call_graph: [{src, dst, prov, weight}]` at application scope (rename from v1 +`{source, target, type, provenance}`). + +## Release plan + +- **Train 1 — v2.0.0 (coordinated major, lockstep).** codeanalyzer-go emission rewrite + (L1 tree/source/ids/body-with-calls, then L2 call_graph) + Neo4j relabel; python-sdk + Go model remap to v2 keeping every public accessor name/return type identical. +- **Ordering constraint (first-class):** pin the analyzer version in the SDK **only once + both are released**. Until then the SDK's old Go models won't parse v2 output. +- **Superset gate:** v2 output must contain every fact v1 did, modulo sanctioned drops + (`code`, `is_*` booleans folded into `kind`, `is_constructor_call`). +- **Deferred (Train 2+):** L3 (CFG/CDG/DDG), L4 (SDG). Reuse Group A ids/spans + Group C + edge shape unchanged; no new schema-major coordination. + +## Decomposition (tracking shape — to be confirmed with user) + +Recommended: **parent (in `codellm-devkit/.github`) + one sub-issue per PR**, because the +work spans two repos on their own clocks and needs a coordination record: +1. codeanalyzer-go: L1 emission (tree, source, ids, body-with-call-nodes) +2. codeanalyzer-go: L2 emission (call_graph rename @ app scope) +3. codeanalyzer-go: Neo4j relabel +4. python-sdk: Go v2 model remap (accessors stable) +5. release + pin (once both cut) + +Children filed just-in-time as picked up; this spec records the full plan. From b1ceabaf0e921d012083733764ac0a6cace7ce6e Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 22:41:23 -0400 Subject: [PATCH 02/29] feat(cli): conform to CLDK CLI contract Add the flags the CLDK CLI contract requires so the SDK facade can shell out to the Go backend with the same code path as the other language backends: - --emit : output projection selector. neo4j and schema return a clear non-zero "not yet implemented" error (never a silent fallback to JSON) until the Neo4j child lands. schema needs no --input, matching the contract. - --app-name: application anchor for can://go/ ids and the Neo4j :Application node; defaults to the input dir base name. - --neo4j-uri/user/password/database: accepted and validated now with explicit-flag > env-var > default precedence; consumed by the Neo4j child. - -j/--jobs: worker parallelism, defaulting to CPU cores. No v2 emission logic yet; default output stays the v1 JSON shape. --- cmd/codeanalyzer/main.go | 101 +++++++++++++++++++++++++++------- cmd/codeanalyzer/main_test.go | 34 ++++++++++++ internal/options/options.go | 27 +++++++++ 3 files changed, 141 insertions(+), 21 deletions(-) diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 66b3187..49bda45 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -8,6 +8,8 @@ import ( "encoding/json" "fmt" "os" + "path/filepath" + "runtime" "github.com/spf13/cobra" @@ -30,17 +32,24 @@ func main() { func rootCmd() *cobra.Command { var ( - inputPath string - outputDir string - format string - level int - targetFiles []string - skipTests bool - eager bool - cacheDir string - useCodeQL bool - verbosity int - showVersion bool + inputPath string + outputDir string + format string + emit string + appName string + level int + targetFiles []string + skipTests bool + eager bool + cacheDir string + jobs int + useCodeQL bool + verbosity int + showVersion bool + neo4jURI string + neo4jUser string + neo4jPassword string + neo4jDatabase string ) cmd := &cobra.Command{ @@ -57,6 +66,21 @@ via CLDK(language="go").analysis(project_path=...).`, cmd.Println("cango " + version) return nil } + + // --emit selects the output projection. neo4j/schema are validated + // here and rejected with a non-zero exit until the Neo4j child lands + // (never a silent fallback to JSON). schema needs no --input. + switch options.EmitTarget(emit) { + case options.EmitJSON: + // valid — falls through to the JSON path below. + case options.EmitNeo4j: + return fmt.Errorf("--emit neo4j is not yet implemented; use --emit json") + case options.EmitSchema: + return fmt.Errorf("--emit schema is not yet implemented; use --emit json") + default: + return fmt.Errorf("unsupported --emit target %q; supported: json, neo4j, schema", emit) + } + if inputPath == "" { return fmt.Errorf("--input / -i is required") } @@ -74,18 +98,32 @@ via CLDK(language="go").analysis(project_path=...).`, home, _ := os.UserHomeDir() cacheDir = home + "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/.cldk/go-cache" } + if jobs < 1 { + jobs = runtime.NumCPU() + } + // application anchor: explicit flag > input dir base name. + if appName == "" { + appName = filepath.Base(filepath.Clean(inputPath)) + } opts := options.AnalysisOptions{ - InputPath: inputPath, - OutputDir: outputDir, - Format: format, - Level: options.AnalysisLevel(level), - TargetFiles: targetFiles, - SkipTests: skipTests, - Eager: eager, - CacheDir: cacheDir, - UseCodeQL: useCodeQL, - Verbose: verbosity > 0, + InputPath: inputPath, + OutputDir: outputDir, + Format: format, + Emit: options.EmitTarget(emit), + AppName: appName, + Level: options.AnalysisLevel(level), + TargetFiles: targetFiles, + SkipTests: skipTests, + Eager: eager, + CacheDir: cacheDir, + Jobs: jobs, + UseCodeQL: useCodeQL, + Verbose: verbosity > 0, + Neo4jURI: firstNonEmpty(neo4jURI, os.Getenv("NEO4J_URI")), + Neo4jUser: firstNonEmpty(neo4jUser, os.Getenv("NEO4J_USERNAME"), "neo4j"), + Neo4jPassword: firstNonEmpty(neo4jPassword, os.Getenv("NEO4J_PASSWORD"), "neo4j"), + Neo4jDatabase: firstNonEmpty(neo4jDatabase, os.Getenv("NEO4J_DATABASE")), } analyzer := core.New(opts) @@ -112,6 +150,9 @@ via CLDK(language="go").analysis(project_path=...).`, f.StringVarP(&inputPath, "input", "i", "", "Project root to analyze (required)") f.StringVarP(&outputDir, "output", "o", "", "Output directory for analysis.json (default: stdout)") f.StringVarP(&format, "format", "f", "json", "Output format: json|msgpack") + f.StringVar(&emit, "emit", "json", "Output projection: json|neo4j|schema") + f.StringVar(&appName, "app-name", "", + "Application anchor name for can:// ids and Neo4j :Application (default: input dir name)") f.IntVarP(&level, "analysis-level", "a", 1, "Analysis level: 1=symbol table only, 2=+resolver call graph") f.StringSliceVarP(&targetFiles, "target-files", "t", nil, @@ -119,9 +160,27 @@ via CLDK(language="go").analysis(project_path=...).`, f.BoolVar(&skipTests, "skip-tests", true, "Skip *_test.go files") f.BoolVar(&eager, "eager", false, "Force clean rebuild (ignore cache)") f.StringVarP(&cacheDir, "cache-dir", "c", "", "Cache directory (default: ~/.cldk/go-cache)") + f.IntVarP(&jobs, "jobs", "j", 0, "Worker parallelism (default: CPU cores)") f.BoolVar(&useCodeQL, "codeql", false, "Enable CodeQL framework-based call graph (level 2, stub)") f.CountVarP(&verbosity, "verbose", "v", "Verbosity (repeat for more detail)") f.BoolVar(&showVersion, "version", false, "Print version and exit") + // Neo4j projection targets (validated now, consumed by the Neo4j child). + f.StringVar(&neo4jURI, "neo4j-uri", "", "Live Bolt push target (env NEO4J_URI); omit to write graph.cypher") + f.StringVar(&neo4jUser, "neo4j-user", "", "Neo4j username (env NEO4J_USERNAME, default neo4j)") + f.StringVar(&neo4jPassword, "neo4j-password", "", "Neo4j password (env NEO4J_PASSWORD, default neo4j)") + f.StringVar(&neo4jDatabase, "neo4j-database", "", "Neo4j database (env NEO4J_DATABASE, optional)") + return cmd } + +// firstNonEmpty returns the first non-empty string in vals, or "" if all are +// empty. It encodes the contract's precedence: explicit flag > env var > default. +func firstNonEmpty(vals ...string) string { + for _, v := range vals { + if v != "" { + return v + } + } + return "" +} diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index acc08bd..ea5ff3b 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -59,6 +59,40 @@ func TestRootCmd_UnknownFormatReturnsError(t *testing.T) { } } +func TestRootCmd_UnknownEmitReturnsError(t *testing.T) { + td := cliTestdataDir() + _, _, err := runCmd("--input", filepath.Join(td, "greeter"), "--emit", "bogus") + if err == nil { + t.Fatal("expected error for unknown --emit value, got nil") + } +} + +func TestRootCmd_EmitNeo4jNotImplemented(t *testing.T) { + td := cliTestdataDir() + _, _, err := runCmd("--input", filepath.Join(td, "greeter"), "--emit", "neo4j") + if err == nil { + t.Fatal("expected non-zero exit for --emit neo4j, got nil") + } + if !strings.Contains(err.Error(), "not yet implemented") { + t.Errorf("error should say 'not yet implemented'; got %q", err.Error()) + } +} + +func TestRootCmd_EmitSchemaNotImplementedWithoutInput(t *testing.T) { + // --emit schema needs no --input; it must still fail (not yet implemented) + // rather than complain about a missing --input. + _, _, err := runCmd("--emit", "schema") + if err == nil { + t.Fatal("expected non-zero exit for --emit schema, got nil") + } + if strings.Contains(err.Error(), "required") { + t.Errorf("--emit schema should not require --input; got %q", err.Error()) + } + if !strings.Contains(err.Error(), "not yet implemented") { + t.Errorf("error should say 'not yet implemented'; got %q", err.Error()) + } +} + // ── --version ──────────────────────────────────────────────────────────────── func TestRootCmd_VersionFlag(t *testing.T) { diff --git a/internal/options/options.go b/internal/options/options.go index 97e6782..051fe6c 100644 --- a/internal/options/options.go +++ b/internal/options/options.go @@ -11,6 +11,18 @@ const ( LevelCallGraph AnalysisLevel = 2 ) +// EmitTarget selects the analyzer's output projection (cli-contract.md § Neo4j). +type EmitTarget string + +const ( + // EmitJSON writes the analysis.json projection (the default). + EmitJSON EmitTarget = "json" + // EmitNeo4j writes/pushes the Neo4j graph projection at full implemented depth. + EmitNeo4j EmitTarget = "neo4j" + // EmitSchema writes the static schema.neo4j.json contract (needs no --input). + EmitSchema EmitTarget = "schema" +) + // AnalysisOptions is the configuration surface passed from the CLI into Analyzer. type AnalysisOptions struct { // InputPath is the project root to analyze. @@ -19,6 +31,11 @@ type AnalysisOptions struct { OutputDir string // Format is the serialization format: "json" or "msgpack". Format string + // Emit selects the output projection: json (default), neo4j, or schema. + Emit EmitTarget + // AppName anchors application identity (can://go/ and the Neo4j + // :Application node). Defaults to the input directory's base name. + AppName string // AnalysisLevel controls symbol-table-only (1) vs + call graph (2). Level AnalysisLevel // TargetFiles restricts analysis to specific files (incremental mode). @@ -29,8 +46,18 @@ type AnalysisOptions struct { Eager bool // CacheDir is where per-file caches and intermediate data are stored. CacheDir string + // Jobs is the worker parallelism (default: CPU cores). Output must be + // byte-identical across Jobs values. + Jobs int // UseCodeQL enables the framework-based (Tier-2) CodeQL call graph. UseCodeQL bool // Verbose enables verbose logging. Verbose bool + + // Neo4j projection targets (consumed by the Neo4j emitter; validated now). + // Precedence for each: explicit flag > env var > default. + Neo4jURI string // live Bolt push target; empty → write graph.cypher. Env NEO4J_URI + Neo4jUser string // env NEO4J_USERNAME, default "neo4j" + Neo4jPassword string // env NEO4J_PASSWORD, default "neo4j" + Neo4jDatabase string // env NEO4J_DATABASE, optional } From 34da145c05c2d8314f4c03a9a0a8038097a6501c Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 22:42:40 -0400 Subject: [PATCH 03/29] feat(schema): add v2 canonical schema types Declare the v2 canonical-schema Go types in internal/schema/v2, a faithful transcript of the keystone (canonical-schema.md) and the Go-specific leaf decisions in CLAUDE.md. Models the L1 additive containment tree (application -> module -> type -> callable -> body) with typed edge overlays, plus the L2 refinement/edge slots so the shape is stable: - Span{start,end,bytes} with UTF-8 byte offsets into module.source. - Module carries source once; no per-node code field. - Type.kind (struct|interface|alias|defined) replaces v1 is_interface; base_types (embedding) vs interfaces (computed satisfaction). - Callable.error_channel, nested callables{} for closures, metrics map. - BodyNode 'call' with the sanctioned callee null->id refinement slot (pointer, the one allowed null) and is_goroutine/is_deferred. - Edge{src,dst,prov,weight} for call_graph. No emitter or wiring yet; these types are not referenced by any code path. L3/L4 fields (cfg/cdg/ddg/param_*) are deliberately not declared. --- internal/schema/v2/schema.go | 196 +++++++++++++++++++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 internal/schema/v2/schema.go diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go new file mode 100644 index 0000000..66042b7 --- /dev/null +++ b/internal/schema/v2/schema.go @@ -0,0 +1,196 @@ +// Package v2 defines the canonical CLDK schema (v2) types that codeanalyzer-go +// emits. It is a faithful transcript of the keystone +// (designing-cldk-changes/references/canonical-schema.md) and the Go-specific +// leaf additions recorded in the repo-root CLAUDE.md § Schema decisions. +// +// The model is ONE additive containment tree +// (application → module → type → callable → body) with typed edge overlays +// laid over it (a Code Property Graph). This package declares the L1 tree plus +// the L2 refinement/edge slots; L3/L4 fields (cfg/cdg/ddg/param_*) are not +// declared here — they arrive in their own train and reuse these ids/spans +// unchanged. +// +// Conventions from the keystone, held here: +// - snake_case JSON keys everywhere, so one set of SDK models parses every +// analyzer. +// - a fact is present or absent; there is no null, EXCEPT the one sanctioned +// `callee: null` refinement slot on a call node (null at L1, backfilled at +// L2). Absence is the "no fact" encoding — `omitempty` on optional fields. +// - open-vocabulary fields (prov, tags) are plain strings. +package v2 + +// SchemaVersion is the canonical schema major this package targets. +const SchemaVersion = "2.0.0" + +// Language is the analyzer's language tag in the manifest and `can:///…`. +const Language = "go" + +// ─── Positioning ─────────────────────────────────────────────────────────── + +// Span is the one universal node attribute: where in source the node lives. +// line:col addresses and displays; bytes slices module.source in O(1). Byte +// offsets are UTF-8 offsets into module.source (Go source is natively UTF-8). +type Span struct { + // Start is [line, col] of the first byte (1-based line, 1-based col). + Start [2]int `json:"start"` + // End is [line, col] one past the last byte. + End [2]int `json:"end"` + // Bytes is [from, to) UTF-8 byte offsets into module.source. + Bytes [2]int `json:"bytes"` +} + +// ─── Root envelope ─────────────────────────────────────────────────────────── + +// Analysis is the top-level payload written to analysis.json / stdout. Its +// siblings of `application` carry the manifest; consumers read `max_level` +// rather than sniffing for keys. +type Analysis struct { + SchemaVersion string `json:"schema_version"` + Language string `json:"language"` + MaxLevel int `json:"max_level"` + Analyzer AnalyzerTag `json:"analyzer"` + Application Application `json:"application"` +} + +// AnalyzerTag identifies the producing analyzer and version. +type AnalyzerTag struct { + Name string `json:"name"` + Version string `json:"version"` +} + +// Application is the root node of the containment tree. +type Application struct { + // ID is `can:///` — the app segment disambiguates apps in one + // language. is the --app-name value (default: input dir base name). + ID string `json:"id"` + Kind string `json:"kind"` // always "application" + // SymbolTable is the L1 named map keyed by relative file path. + SymbolTable map[string]Module `json:"symbol_table"` + // CallGraph holds the L2 cross-function edges (callable → callable). Empty + // at L1. + CallGraph []Edge `json:"call_graph"` +} + +// ─── module (per-file compilation unit) ─────────────────────────────────────── + +// Module is one Go source file: the L1 per-file container. +type Module struct { + ID string `json:"id"` + Kind string `json:"kind"` // always "module" + Span Span `json:"span"` + // Package is the Go package the file belongs to. + Package string `json:"package"` + // Source is the WHOLE file's text, stored once. Every node's text slices + // from this via its span.bytes; there is no per-node `code`. + Source string `json:"source"` + // Imports are the file's import declarations. + Imports []Import `json:"imports"` + // Types is the named map of type declarations in the file. + Types map[string]Type `json:"types"` + // Functions is the named map of module-level callables (keyed by signature). + Functions map[string]Callable `json:"functions"` + // ContentHash supports incremental caching; it is not identity. + ContentHash string `json:"content_hash,omitempty"` +} + +// Import is a single import declaration. +type Import struct { + Name string `json:"name"` // imported package name + Path string `json:"path"` // import path, e.g. "hash/fnv" + Alias string `json:"alias,omitempty"` // present only when aliased + Span Span `json:"span"` +} + +// ─── type ───────────────────────────────────────────────────────────────────── + +// Type is a named Go type. Kind is the specific type-declaration shape, not a +// pile of is_* booleans. +type Type struct { + ID string `json:"id"` + Kind string `json:"kind"` // struct | interface | alias | defined + Span Span `json:"span"` + // BaseTypes are embedded type ids (struct/interface embedding — the explicit + // spine). + BaseTypes []string `json:"base_types,omitempty"` + // Interfaces are ids of interfaces this type is computed to satisfy via + // method-set matching (Go's structural implementation). + Interfaces []string `json:"interfaces,omitempty"` + // Callables are the type's methods, resolved to their receiver type + // regardless of source location. Keyed by signature. + Callables map[string]Callable `json:"callables"` + // Fields are the struct fields, keyed by name. + Fields map[string]Field `json:"fields,omitempty"` +} + +// Field is a struct field. +type Field struct { + ID string `json:"id"` + Kind string `json:"kind"` // always "field" + Type string `json:"type"` + Span Span `json:"span"` +} + +// ─── callable ───────────────────────────────────────────────────────────────── + +// Callable is a function, method, or closure. span.bytes → get_method_body = +// module.source[bytes]. +type Callable struct { + ID string `json:"id"` + Kind string `json:"kind"` // function | method | lambda + Span Span `json:"span"` + // Signature is the human-readable last path segment of ID (one signatureOf()). + Signature string `json:"signature"` + // Parameters are the ordered formal parameters. + Parameters []Parameter `json:"parameters"` + // ReturnType is the joined return type, e.g. "(int, error)". + ReturnType string `json:"return_type,omitempty"` + // ErrorChannel is the generalized error surface: populated from error-typed + // returns (those returns remain in ReturnType too). + ErrorChannel []string `json:"error_channel,omitempty"` + // Metrics is an extensible metrics map (e.g. {"cyclomatic": 3}). + Metrics map[string]int `json:"metrics,omitempty"` + // Body holds call nodes at L1 (keyed by local id), completing at L3. + Body map[string]BodyNode `json:"body"` + // Callables are nested closures / function literals, each with its own + // can:// id (replaces v1 InnerCallables). + Callables map[string]Callable `json:"callables,omitempty"` +} + +// Parameter is a single formal parameter. +type Parameter struct { + Name string `json:"name"` + Type string `json:"type"` + Span Span `json:"span"` + IsVariadic bool `json:"is_variadic,omitempty"` +} + +// ─── body nodes ──────────────────────────────────────────────────────────────── + +// BodyNode is a node inside a callable's body, keyed by its local id +// (line:col for real nodes). At L1 the only kind emitted is "call"; L3 adds the +// remaining statement kinds. +type BodyNode struct { + Kind string `json:"kind"` // "call" at L1 + Span Span `json:"span"` + // Callee is the sanctioned null→id refinement slot on a call node: null at + // L1, backfilled to a callable id at L2. A pointer so it serializes as JSON + // null (the one place null is allowed) rather than being omitted. + Callee *string `json:"callee,omitempty"` + // IsGoroutine is true when the call is preceded by `go`. + IsGoroutine bool `json:"is_goroutine,omitempty"` + // IsDeferred is true when the call is preceded by `defer` (net-new vs v1). + IsDeferred bool `json:"is_deferred,omitempty"` +} + +// ─── edges ───────────────────────────────────────────────────────────────────── + +// Edge is a typed edge overlay record. The list it lives in IS its type (there +// is no `type` field). src/dst are node ids; no dangling endpoints. +type Edge struct { + Src string `json:"src"` + Dst string `json:"dst"` + // Prov is open-vocabulary provenance, e.g. ["go/types"], ["go/types","codeql"]. + Prov []string `json:"prov,omitempty"` + // Weight accumulates when merging edges from multiple backends. + Weight int `json:"weight,omitempty"` +} From 3790749dc164b1d25d92123166d0ebe827d81faa Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 22:43:42 -0400 Subject: [PATCH 04/29] feat(schema): can:// id builder and span offsets Add the Group A identity/positioning helpers in internal/schema/v2, pure functions so they are testable in isolation and reused verbatim by the deferred L3/L4 train: - can:// durable-id builders (AppID/ModuleID/TypeID/CallableID) that assemble the containment path can://////, with signatureOf() output as the last segment. - @line:col ordinal-id builders (OrdinalID/TagID/LocalID) for body nodes below the callable line. - NewSpan + Span.Slice: build a span from line/col and UTF-8 byte offsets, and slice module.source in O(1) (the replacement for the v1 per-node code field), degrading to "" on a malformed span. Unit tests reproduce the keystone worked-example id chain and cover path normalization, module-level function parents, ordinal/tag ids, and span slicing edge cases. Not wired into any emitter yet. --- internal/schema/v2/ids.go | 128 +++++++++++++++++++++++++++++++++ internal/schema/v2/ids_test.go | 93 ++++++++++++++++++++++++ 2 files changed, 221 insertions(+) create mode 100644 internal/schema/v2/ids.go create mode 100644 internal/schema/v2/ids_test.go diff --git a/internal/schema/v2/ids.go b/internal/schema/v2/ids.go new file mode 100644 index 0000000..d8d192d --- /dev/null +++ b/internal/schema/v2/ids.go @@ -0,0 +1,128 @@ +package v2 + +import "strings" + +// Group A vocabulary: the `can://` durable-id grammar (>= callable) and the +// `@line:col` ordinal-id grammar (< callable), plus span construction. These +// helpers are pure so they can be tested in isolation and reused verbatim by +// the deferred L3/L4 train (which must not re-decide id or span shape). +// +// Durable id grammar (a containment path with an app segment so multiple apps +// in one language don't collide): +// +// can:////// +// can://go/myapp/src/util.go/Hasher/Hash(string)uint64 +// +// Ordinal id grammar (statements and synthetic vertices, addressed WITHIN their +// callable): +// +// @: e.g. …/Hash(string)uint64@15:2 +// @ e.g. …/Hash(string)uint64@entry +// +// The delimiters `/`, `@`, `:` are fixed by the keystone; do not substitute. +const ( + scheme = "can://" + // idSep joins containment segments in a durable id. + idSep = "/" + // ordinalSep separates a callable id from an ordinal (line:col or @tag). + ordinalSep = "@" +) + +// AppID builds the application id: can:///. lang is fixed to the +// package Language constant so callers cannot drift it. +func AppID(app string) string { + return scheme + Language + idSep + app +} + +// ModuleID builds a module id from the application id and the module's file +// path relative to the input root (forward-slashed, no leading slash, no ".."). +func ModuleID(appID, relPath string) string { + return appID + idSep + normalizePath(relPath) +} + +// TypeID builds a type id under its module. sig is the type's signatureOf() +// output — the last (and only) segment the type contributes. +func TypeID(moduleID, sig string) string { + return moduleID + idSep + sig +} + +// CallableID builds a callable id under its parent. For a method the parent is +// its receiver Type id; for a module-level function the parent is the Module +// id. sig is the callable's signatureOf() output — the last path segment. +func CallableID(parentID, sig string) string { + return parentID + idSep + sig +} + +// OrdinalID addresses a real body node (statement / call) within its callable +// by source position: @:. Both line and col are +// required — a bare line is not unique within a callable. +func OrdinalID(callableID string, line, col int) string { + return callableID + ordinalSep + itoa(line) + ":" + itoa(col) +} + +// TagID addresses a synthetic vertex within its callable by tag: +// @ (e.g. "entry", "exit", "formal_in:0"). +func TagID(callableID, tag string) string { + return callableID + ordinalSep + tag +} + +// LocalID is the key a real body node is stored under inside body{}: the bare +// ":" ordinal (the id relative to the enclosing callable). +func LocalID(line, col int) string { + return itoa(line) + ":" + itoa(col) +} + +// NewSpan builds a Span from 1-based line/col positions and the [from, to) UTF-8 +// byte offsets into module.source. Byte offsets are what make source slicing +// O(1); line:col is what addresses and displays. +func NewSpan(startLine, startCol, endLine, endCol, fromByte, toByte int) Span { + return Span{ + Start: [2]int{startLine, startCol}, + End: [2]int{endLine, endCol}, + Bytes: [2]int{fromByte, toByte}, + } +} + +// Slice returns the node's source text: module source sliced by the span's byte +// offsets. It is the O(1) replacement for the v1 per-node `code` field. Out-of- +// range or inverted offsets yield "" rather than panicking, so a malformed span +// degrades gracefully. +func (s Span) Slice(source string) string { + from, to := s.Bytes[0], s.Bytes[1] + if from < 0 || to > len(source) || from > to { + return "" + } + return source[from:to] +} + +// normalizePath makes a relative file path safe for a can:// segment: forward +// slashes, no leading slash. Callers are responsible for passing a path already +// relative to the input root (no ".."). +func normalizePath(p string) string { + p = strings.ReplaceAll(p, "\\", "/") + return strings.TrimPrefix(p, "/") +} + +// itoa is a tiny non-allocating-path integer formatter for id assembly, kept +// local so this file has no fmt dependency for its hot path. +func itoa(n int) string { + if n == 0 { + return "0" + } + neg := n < 0 + if neg { + n = -n + } + var buf [20]byte + i := len(buf) + for n > 0 { + i-- + buf[i] = byte('0' + n%10) + n /= 10 + } + if neg { + i-- + buf[i] = '-' + } + return string(buf[i:]) +} diff --git a/internal/schema/v2/ids_test.go b/internal/schema/v2/ids_test.go new file mode 100644 index 0000000..f7ec330 --- /dev/null +++ b/internal/schema/v2/ids_test.go @@ -0,0 +1,93 @@ +package v2 + +import "testing" + +func TestAppID(t *testing.T) { + if got, want := AppID("myapp"), "can://go/myapp"; got != want { + t.Errorf("AppID = %q, want %q", got, want) + } +} + +func TestDurableIDChain(t *testing.T) { + // Reproduces the keystone worked example, segment by segment. + app := AppID("myapp") + mod := ModuleID(app, "src/util.go") + typ := TypeID(mod, "Hasher") + call := CallableID(typ, "Hash(string)uint64") + + cases := []struct{ got, want string }{ + {mod, "can://go/myapp/src/util.go"}, + {typ, "can://go/myapp/src/util.go/Hasher"}, + {call, "can://go/myapp/src/util.go/Hasher/Hash(string)uint64"}, + } + for _, c := range cases { + if c.got != c.want { + t.Errorf("id = %q, want %q", c.got, c.want) + } + } +} + +func TestModuleIDNormalizesPath(t *testing.T) { + app := AppID("myapp") + // Backslashes become forward slashes; a leading slash is trimmed. + if got, want := ModuleID(app, "\\pkg\\a.go"), "can://go/myapp/pkg/a.go"; got != want { + t.Errorf("ModuleID = %q, want %q", got, want) + } + if got, want := ModuleID(app, "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/pkg/a.go"), "can://go/myapp/pkg/a.go"; got != want { + t.Errorf("ModuleID = %q, want %q", got, want) + } +} + +func TestModuleLevelFunctionParent(t *testing.T) { + // A module-level function hangs off the module id, not a type. + mod := ModuleID(AppID("myapp"), "src/util.go") + fn := CallableID(mod, "New64()") + if want := "can://go/myapp/src/util.go/New64()"; fn != want { + t.Errorf("CallableID = %q, want %q", fn, want) + } +} + +func TestOrdinalAndTagIDs(t *testing.T) { + call := "can://go/myapp/src/util.go/Hasher/Hash(string)uint64" + if got, want := OrdinalID(call, 15, 2), call+"@15:2"; got != want { + t.Errorf("OrdinalID = %q, want %q", got, want) + } + if got, want := TagID(call, "entry"), call+"@entry"; got != want { + t.Errorf("TagID = %q, want %q", got, want) + } + if got, want := LocalID(16, 2), "16:2"; got != want { + t.Errorf("LocalID = %q, want %q", got, want) + } +} + +func TestSpanSlice(t *testing.T) { + src := "package util\nfunc f() {}\n" + // bytes [13,25) covers "func f() {}\n" + s := NewSpan(2, 1, 2, 12, 13, 25) + if got, want := s.Slice(src), "func f() {}\n"; got != want { + t.Errorf("Slice = %q, want %q", got, want) + } +} + +func TestSpanSliceOutOfRangeIsEmpty(t *testing.T) { + src := "abc" + cases := []Span{ + {Bytes: [2]int{0, 99}}, // to past end + {Bytes: [2]int{-1, 2}}, // negative from + {Bytes: [2]int{2, 1}}, // inverted + } + for i, s := range cases { + if got := s.Slice(src); got != "" { + t.Errorf("case %d: Slice = %q, want empty", i, got) + } + } +} + +func TestItoa(t *testing.T) { + cases := map[int]string{0: "0", 7: "7", 42: "42", 1000: "1000", -3: "-3"} + for in, want := range cases { + if got := itoa(in); got != want { + t.Errorf("itoa(%d) = %q, want %q", in, got, want) + } + } +} From d3affba6715791e1aabc6b7b729bf63120d33bd1 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 22:54:29 -0400 Subject: [PATCH 05/29] feat(emit): source line-to-byte-offset indexer for v2 spans Add the v2emit package with lineIndex: reads a file's source once and maps 1-based (line, col) positions to UTF-8 byte offsets, the one new datum the v2 span needs. go/token columns are byte columns, so no rune/byte reconciliation is required. This keeps the v1 model diff-free (per the migration spec): the emitter derives span.bytes from the existing line/col positions instead of threading offsets through internal/schema. lineEndOffset/lineLen give a line-only v1 position (callables/types carry no columns) a precise end. Positions outside the file clamp to [0, len(source)] rather than panicking. Tests cover line-start tabulation, offset lookup, clamping, the last-line-without-newline case, and empty source. Emitter tree walk lands next; nothing wired yet. --- internal/syntactic_analysis/v2emit/offsets.go | 94 +++++++++++++++++++ .../syntactic_analysis/v2emit/offsets_test.go | 69 ++++++++++++++ 2 files changed, 163 insertions(+) create mode 100644 internal/syntactic_analysis/v2emit/offsets.go create mode 100644 internal/syntactic_analysis/v2emit/offsets_test.go diff --git a/internal/syntactic_analysis/v2emit/offsets.go b/internal/syntactic_analysis/v2emit/offsets.go new file mode 100644 index 0000000..e9d3160 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/offsets.go @@ -0,0 +1,94 @@ +// Package v2emit transforms the v1 in-memory model (internal/schema) into the +// v2 canonical schema (internal/schema/v2). It is an EMISSION layer only: it +// computes no new facts beyond source byte offsets, it just re-serializes what +// the parser/resolver already produced into the additive-tree shape. +// +// The one genuinely new datum is span byte offsets. The v1 model records +// line/col but not UTF-8 byte offsets (and only line-level positions for +// callables/types). Rather than change the v1 model, the emitter reads each +// file's source once and derives byte offsets from the existing line/col +// positions via a lineIndex. That same source string is emitted once per module +// as module.source, off which every node's text slices. +package v2emit + +// lineIndex maps 1-based (line, col) positions in a source file to UTF-8 byte +// offsets. Built once per file. Columns are byte columns within the line, which +// is what go/token.Position reports (Position.Column counts bytes, not runes), +// so no rune/byte reconciliation is needed here. +type lineIndex struct { + source string + // lineStart[i] is the byte offset where 1-based line (i+1) begins. + lineStart []int +} + +// newLineIndex builds a lineIndex over the given source text. +func newLineIndex(source string) *lineIndex { + // Line 1 starts at offset 0; each '\n' begins the next line at the byte + // immediately after it. + starts := make([]int, 0, 1+countByte(source, '\n')) + starts = append(starts, 0) + for i := 0; i < len(source); i++ { + if source[i] == '\n' { + starts = append(starts, i+1) + } + } + return &lineIndex{source: source, lineStart: starts} +} + +// offset converts a 1-based (line, col) position to a byte offset into source. +// col is a 1-based byte column (go/token semantics). Positions outside the +// file's range are clamped to [0, len(source)] so a bad position yields a +// harmless in-range offset rather than a panic. +func (li *lineIndex) offset(line, col int) int { + if len(li.lineStart) == 0 { + return 0 + } + if line < 1 { + line = 1 + } + if line > len(li.lineStart) { + line = len(li.lineStart) + } + base := li.lineStart[line-1] + if col < 1 { + col = 1 + } + off := base + (col - 1) + if off > len(li.source) { + off = len(li.source) + } + return off +} + +// lineEndOffset returns the byte offset of the end of a 1-based line: the +// position just before its terminating '\n' (or end of file for the last line). +// Used to give a line-only position (no column in the v1 model) a precise end. +func (li *lineIndex) lineEndOffset(line int) int { + if line < 1 || len(li.lineStart) == 0 { + return 0 + } + if line >= len(li.lineStart) { + return len(li.source) + } + // Next line begins one byte after this line's '\n'; back up over the '\n'. + return li.lineStart[line] - 1 +} + +// lineLen returns the number of bytes on a 1-based line, excluding the newline. +// It gives a line-only end position a byte column (used to synthesize end.col). +func (li *lineIndex) lineLen(line int) int { + if line < 1 || line > len(li.lineStart) { + return 0 + } + return li.lineEndOffset(line) - li.lineStart[line-1] +} + +func countByte(s string, b byte) int { + n := 0 + for i := 0; i < len(s); i++ { + if s[i] == b { + n++ + } + } + return n +} diff --git a/internal/syntactic_analysis/v2emit/offsets_test.go b/internal/syntactic_analysis/v2emit/offsets_test.go new file mode 100644 index 0000000..69b61b9 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/offsets_test.go @@ -0,0 +1,69 @@ +package v2emit + +import "testing" + +const sample = "package util\n" + // line 1: bytes 0..12 (\n at 12) + "\n" + // line 2: byte 13 (\n at 13) + "func f() {}\n" + // line 3: bytes 14..25 (\n at 25) + "var x int" // line 4: bytes 26..34, no trailing newline + +func TestLineStarts(t *testing.T) { + li := newLineIndex(sample) + want := []int{0, 13, 14, 26} + if len(li.lineStart) != len(want) { + t.Fatalf("lineStart len = %d, want %d (%v)", len(li.lineStart), len(want), li.lineStart) + } + for i, w := range want { + if li.lineStart[i] != w { + t.Errorf("lineStart[%d] = %d, want %d", i, li.lineStart[i], w) + } + } +} + +func TestOffset(t *testing.T) { + li := newLineIndex(sample) + // line 3 ("func f() {}") col 1 → byte 14; col 6 ('f') → 19 + if got := li.offset(3, 1); got != 14 { + t.Errorf("offset(3,1) = %d, want 14", got) + } + if got := li.offset(3, 6); got != 19 { + t.Errorf("offset(3,6) = %d, want 19", got) + } + // The slice from a callable start to its end reproduces the source text. + start := li.offset(3, 1) + end := li.lineEndOffset(3) + if got := sample[start:end]; got != "func f() {}" { + t.Errorf("slice = %q, want %q", got, "func f() {}") + } +} + +func TestOffsetClamping(t *testing.T) { + li := newLineIndex(sample) + if got := li.offset(0, 0); got != 0 { + t.Errorf("offset(0,0) = %d, want 0 (clamped)", got) + } + if got := li.offset(999, 999); got != len(sample) { + t.Errorf("offset(999,999) = %d, want %d (clamped to EOF)", got, len(sample)) + } +} + +func TestLineEndAndLen(t *testing.T) { + li := newLineIndex(sample) + // last line has no trailing newline → ends at EOF + if got := li.lineEndOffset(4); got != len(sample) { + t.Errorf("lineEndOffset(4) = %d, want %d", got, len(sample)) + } + if got := li.lineLen(3); got != len("func f() {}") { + t.Errorf("lineLen(3) = %d, want %d", got, len("func f() {}")) + } + if got := li.lineLen(4); got != len("var x int") { + t.Errorf("lineLen(4) = %d, want %d", got, len("var x int")) + } +} + +func TestEmptySource(t *testing.T) { + li := newLineIndex("") + if got := li.offset(1, 1); got != 0 { + t.Errorf("offset on empty source = %d, want 0", got) + } +} From 79433931f479d789ae82ef854238374904ae7ace Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 22:56:46 -0400 Subject: [PATCH 06/29] feat(emit): v2 L1 emitter over the v1 model Emit() transforms the v1 GoApplication into the v2 Analysis payload: a pure re-serialization walking application -> module -> type -> callable -> body, computing no new facts beyond span byte offsets. - Module carries source once (read from projectDir/relPath); every node span slices off it via the lineIndex-derived byte offsets. - can:// ids assembled from the id builder, signatureOf() as last segment; methods resolve under their receiver type's callables{}. - type.kind collapses is_interface (struct|interface); base_types kept; error-typed returns lift into error_channel; cyclomatic into metrics; closures nest in callables{}. - body holds one 'call' node per call site keyed by line:col, with the callee null->id slot carried straight from the v1 pointer. - call_graph renamed to {src,dst,prov,weight}. Also drop omitempty from BodyNode.Callee so the sanctioned refinement slot always serializes (null at L1, id at L2) as the keystone requires. Tests build the greeter fixture through the real symbol-table builder and assert the envelope, module source/ids, span slicing, call-node bodies, callee-null serialization, dropped is_constructor_call, and type/method kinds. Not wired to the CLI yet. --- internal/schema/v2/schema.go | 7 +- internal/syntactic_analysis/v2emit/emit.go | 284 ++++++++++++++++++ .../syntactic_analysis/v2emit/emit_test.go | 185 ++++++++++++ 3 files changed, 473 insertions(+), 3 deletions(-) create mode 100644 internal/syntactic_analysis/v2emit/emit.go create mode 100644 internal/syntactic_analysis/v2emit/emit_test.go diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index 66042b7..c3acb27 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -173,9 +173,10 @@ type BodyNode struct { Kind string `json:"kind"` // "call" at L1 Span Span `json:"span"` // Callee is the sanctioned null→id refinement slot on a call node: null at - // L1, backfilled to a callable id at L2. A pointer so it serializes as JSON - // null (the one place null is allowed) rather than being omitted. - Callee *string `json:"callee,omitempty"` + // L1, backfilled to a callable id at L2. A pointer with NO omitempty so it + // always serializes — as JSON null at L1 (the one place null is allowed), + // as an id once backfilled. The keystone requires the field to be present. + Callee *string `json:"callee"` // IsGoroutine is true when the call is preceded by `go`. IsGoroutine bool `json:"is_goroutine,omitempty"` // IsDeferred is true when the call is preceded by `defer` (net-new vs v1). diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go new file mode 100644 index 0000000..d2501d2 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -0,0 +1,284 @@ +package v2emit + +import ( + "os" + "path/filepath" + "sort" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// Emit transforms the v1 GoApplication into the v2 Analysis payload. appName is +// the application anchor (--app-name); projectDir is the absolute input root, +// used to read each module's source for span slicing. maxLevel records how +// deeply the tree was populated (1 = symbol table, 2 = + call_graph). +// +// This is a pure re-serialization: every fact comes from the v1 model, except +// span byte offsets, which are derived from source via lineIndex. +func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int) *v2.Analysis { + appID := v2.AppID(appName) + + out := &v2.Analysis{ + SchemaVersion: v2.SchemaVersion, + Language: v2.Language, + MaxLevel: maxLevel, + Analyzer: v2.AnalyzerTag{Name: "codeanalyzer-go"}, + Application: v2.Application{ + ID: appID, + Kind: "application", + SymbolTable: make(map[string]v2.Module, len(app.SymbolTable)), + CallGraph: emitCallGraph(app.CallGraph), + }, + } + + // Deterministic module order does not affect the JSON map, but iterating + // sorted keeps any incidental ordering (and logs) stable across runs. + for _, relPath := range sortedFileKeys(app.SymbolTable) { + file := app.SymbolTable[relPath] + out.Application.SymbolTable[relPath] = emitModule(appID, projectDir, relPath, file) + } + return out +} + +// emitModule builds a v2 module from a v1 GoFile, reading the file's source so +// spans can slice off it. A file that cannot be read still emits its structure +// (with empty source and zeroed byte spans) rather than dropping the module. +func emitModule(appID, projectDir, relPath string, file schema.GoFile) v2.Module { + source := readSource(projectDir, relPath) + li := newLineIndex(source) + modID := v2.ModuleID(appID, relPath) + + mod := v2.Module{ + ID: modID, + Kind: "module", + Span: wholeFileSpan(li, source), + Package: file.PackageName, + Source: source, + Imports: emitImports(li, file.Imports), + Types: make(map[string]v2.Type, len(file.Types)), + Functions: make(map[string]v2.Callable, len(file.Functions)), + } + if file.ContentHash != nil { + mod.ContentHash = *file.ContentHash + } + + for name, t := range file.Types { + mod.Types[name] = emitType(modID, li, t) + } + for sig, fn := range file.Functions { + mod.Functions[sig] = emitCallable(modID, li, fn) + } + return mod +} + +// emitType maps a v1 GoType to a v2 type node, collapsing is_interface into the +// kind and splitting embedding (base_types) from computed satisfaction. Methods +// resolve into the type's callables{}. +func emitType(modID string, li *lineIndex, t schema.GoType) v2.Type { + typeID := v2.TypeID(modID, t.Signature) + out := v2.Type{ + ID: typeID, + Kind: typeKind(t), + Span: lineSpan(li, t.StartLine, t.EndLine), + BaseTypes: nonEmpty(t.BaseTypes), + Callables: make(map[string]v2.Callable, len(t.Methods)), + } + if len(t.Fields) > 0 { + out.Fields = make(map[string]v2.Field, len(t.Fields)) + for _, f := range t.Fields { + out.Fields[f.Name] = v2.Field{ + ID: typeID + "/" + f.Name, + Kind: "field", + Type: f.Type, + Span: lineSpan(li, f.StartLine, f.EndLine), + } + } + } + for sig, m := range t.Methods { + out.Callables[sig] = emitCallable(typeID, li, m) + } + return out +} + +// emitCallable maps a v1 GoCallable to a v2 callable node, including its body +// (call nodes) and nested closures. +func emitCallable(parentID string, li *lineIndex, c schema.GoCallable) v2.Callable { + callID := v2.CallableID(parentID, c.Signature) + out := v2.Callable{ + ID: callID, + Kind: callableKind(c), + Span: lineSpan(li, c.StartLine, c.EndLine), + Signature: c.Signature, + Parameters: emitParams(li, c.Parameters), + ReturnType: c.ReturnType, + Body: emitBody(callID, li, c.CallSites), + } + if ch := errorChannel(c.ReturnTypes); len(ch) > 0 { + out.ErrorChannel = ch + } + if c.CyclomaticComplexity > 0 { + out.Metrics = map[string]int{"cyclomatic": c.CyclomaticComplexity} + } + if len(c.InnerCallables) > 0 { + out.Callables = make(map[string]v2.Callable, len(c.InnerCallables)) + for sig, inner := range c.InnerCallables { + out.Callables[sig] = emitCallable(callID, li, inner) + } + } + return out +} + +// emitBody builds the L1 body map: one `call` node per call site, keyed by its +// line:col local id, with the sanctioned callee null->id refinement slot. +func emitBody(callableID string, li *lineIndex, sites []schema.GoCallsite) map[string]v2.BodyNode { + body := make(map[string]v2.BodyNode, len(sites)) + for _, cs := range sites { + local := v2.LocalID(cs.StartLine, cs.StartColumn) + body[local] = v2.BodyNode{ + Kind: "call", + Span: colSpan(li, cs.StartLine, cs.StartColumn, cs.EndLine, cs.EndColumn), + Callee: cs.CalleeSignature, // *string: null at L1, backfilled at L2 + IsGoroutine: cs.IsGoroutine, + } + } + return body +} + +func emitParams(li *lineIndex, params []schema.GoParameter) []v2.Parameter { + out := make([]v2.Parameter, 0, len(params)) + for _, p := range params { + out = append(out, v2.Parameter{ + Name: p.Name, + Type: p.Type, + Span: lineSpan(li, p.StartLine, p.EndLine), + IsVariadic: p.IsVariadic, + }) + } + return out +} + +func emitImports(li *lineIndex, imports []schema.GoImport) []v2.Import { + out := make([]v2.Import, 0, len(imports)) + for _, imp := range imports { + out = append(out, v2.Import{ + Name: packageBase(imp.Module), + Path: imp.Module, + Alias: imp.Alias, + Span: lineSpan(li, imp.StartLine, imp.EndLine), + }) + } + return out +} + +// emitCallGraph maps v1 identity-only edges to the v2 {src,dst,prov,weight} +// shape (rename of source/target/provenance). +func emitCallGraph(edges []schema.GoCallEdge) []v2.Edge { + out := make([]v2.Edge, 0, len(edges)) + for _, e := range edges { + out = append(out, v2.Edge{ + Src: e.Source, + Dst: e.Target, + Prov: nonEmpty(e.Provenance), + Weight: e.Weight, + }) + } + return out +} + +// ─── kind mapping ──────────────────────────────────────────────────────────── + +// typeKind collapses the v1 is_interface boolean into the v2 kind vocabulary. +// v1 only distinguishes struct vs interface; alias/defined await richer v1 +// facts, so a non-interface type maps to struct for now. +func typeKind(t schema.GoType) string { + if t.IsInterface { + return "interface" + } + return "struct" +} + +// callableKind classifies a callable: a non-empty receiver type marks a method. +func callableKind(c schema.GoCallable) string { + if c.ReceiverType != "" { + return "method" + } + return "function" +} + +// errorChannel extracts error-typed returns into the generalized error surface. +func errorChannel(returnTypes []string) []string { + var ch []string + for _, rt := range returnTypes { + if rt == "error" { + ch = append(ch, rt) + } + } + return ch +} + +// ─── span construction ───────────────────────────────────────────────────────── + +// wholeFileSpan is the module node's span: the entire file. +func wholeFileSpan(li *lineIndex, source string) v2.Span { + lastLine := len(li.lineStart) + if lastLine == 0 { + lastLine = 1 + } + return v2.NewSpan(1, 1, lastLine, li.lineLen(lastLine)+1, 0, len(source)) +} + +// lineSpan builds a span from a line-only v1 position: start at column 1, end at +// the last line's end. Byte offsets bound the whole line range. +func lineSpan(li *lineIndex, startLine, endLine int) v2.Span { + if endLine < startLine { + endLine = startLine + } + endCol := li.lineLen(endLine) + 1 + return v2.NewSpan( + startLine, 1, endLine, endCol, + li.offset(startLine, 1), li.lineEndOffset(endLine), + ) +} + +// colSpan builds a span from a v1 position that carries columns (call sites). +func colSpan(li *lineIndex, startLine, startCol, endLine, endCol int) v2.Span { + return v2.NewSpan( + startLine, startCol, endLine, endCol, + li.offset(startLine, startCol), li.offset(endLine, endCol), + ) +} + +// ─── helpers ───────────────────────────────────────────────────────────────── + +func readSource(projectDir, relPath string) string { + data, err := os.ReadFile(filepath.Join(projectDir, relPath)) + if err != nil { + return "" + } + return string(data) +} + +func sortedFileKeys(m map[string]schema.GoFile) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} + +// packageBase returns the last path segment of an import path as the package +// name, e.g. "hash/fnv" -> "fnv". +func packageBase(importPath string) string { + return filepath.Base(importPath) +} + +// nonEmpty returns s unchanged, or nil if empty, so omitempty drops the field +// rather than emitting an empty array (absence = no fact). +func nonEmpty(s []string) []string { + if len(s) == 0 { + return nil + } + return s +} diff --git a/internal/syntactic_analysis/v2emit/emit_test.go b/internal/syntactic_analysis/v2emit/emit_test.go new file mode 100644 index 0000000..deb8a66 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/emit_test.go @@ -0,0 +1,185 @@ +package v2emit + +import ( + "encoding/json" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" +) + +// greeterDir returns the absolute path to the greeter testdata fixture. +func greeterDir(t *testing.T) string { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + abs, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", "greeter")) + if err != nil { + t.Fatalf("resolving fixture dir: %v", err) + } + return abs +} + +// buildGreeter runs the real symbol-table builder over the greeter fixture. +func buildGreeter(t *testing.T) (*schema.GoApplication, string) { + t.Helper() + dir := greeterDir(t) + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + return &schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}}, dir +} + +func TestEmit_Envelope(t *testing.T) { + app, dir := buildGreeter(t) + out := Emit(app, "greeter", dir, 1) + + if out.SchemaVersion != "2.0.0" { + t.Errorf("schema_version = %q, want 2.0.0", out.SchemaVersion) + } + if out.Language != "go" { + t.Errorf("language = %q, want go", out.Language) + } + if out.MaxLevel != 1 { + t.Errorf("max_level = %d, want 1", out.MaxLevel) + } + if out.Application.ID != "can://go/greeter" { + t.Errorf("application.id = %q, want can://go/greeter", out.Application.ID) + } + if out.Application.Kind != "application" { + t.Errorf("application.kind = %q, want application", out.Application.Kind) + } + if len(out.Application.SymbolTable) == 0 { + t.Fatal("symbol_table is empty") + } +} + +func TestEmit_ModuleSourceAndIDs(t *testing.T) { + app, dir := buildGreeter(t) + out := Emit(app, "greeter", dir, 1) + + mod, ok := out.Application.SymbolTable["main.go"] + if !ok { + t.Fatalf("main.go not in symbol_table; keys=%v", keys(out.Application.SymbolTable)) + } + if mod.Kind != "module" { + t.Errorf("module.kind = %q, want module", mod.Kind) + } + if mod.ID != "can://go/greeter/main.go" { + t.Errorf("module.id = %q, want can://go/greeter/main.go", mod.ID) + } + if !strings.HasPrefix(mod.Source, "package main") { + t.Errorf("module.source should hold the file text; got prefix %q", head(mod.Source, 20)) + } + // The module span slices the whole file. + if got := mod.Span.Slice(mod.Source); got != mod.Source { + t.Errorf("module span does not cover whole source (%d vs %d bytes)", len(got), len(mod.Source)) + } +} + +func TestEmit_CallableBodyAndSpanSlice(t *testing.T) { + app, dir := buildGreeter(t) + out := Emit(app, "greeter", dir, 1) + + mod := out.Application.SymbolTable["main.go"] + var main v2.Callable + found := false + for _, fn := range mod.Functions { + if fn.Signature != "" && strings.HasSuffix(fn.Signature, ".main") { + main, found = fn, true + } + } + if !found { + t.Fatalf("main function not found; functions=%v", keys(mod.Functions)) + } + if main.Kind != "function" { + t.Errorf("main.kind = %q, want function", main.Kind) + } + // Its span should slice to source beginning with "func main". + if body := main.Span.Slice(mod.Source); !strings.HasPrefix(body, "func main") { + t.Errorf("main span slice should start with 'func main'; got %q", head(body, 20)) + } + // main() has call sites → body call nodes, each with callee null at L1. + if len(main.Body) == 0 { + t.Fatal("main should have call nodes in its body") + } + for local, node := range main.Body { + if node.Kind != "call" { + t.Errorf("body[%s].kind = %q, want call", local, node.Kind) + } + if node.Callee != nil { + t.Errorf("body[%s].callee should be null at L1; got %q", local, *node.Callee) + } + if !strings.Contains(local, ":") { + t.Errorf("body key %q should be line:col", local) + } + } +} + +func TestEmit_CalleeSerializesAsNull(t *testing.T) { + app, dir := buildGreeter(t) + out := Emit(app, "greeter", dir, 1) + data, err := json.Marshal(out) + if err != nil { + t.Fatalf("marshal failed: %v", err) + } + if !strings.Contains(string(data), `"callee":null`) { + t.Error("expected at least one call node to serialize callee as null at L1") + } + // The v1-only keys must be gone. + for _, banned := range []string{`"symbol_table"`, `"call_graph"`} { + if !strings.Contains(string(data), banned) { + t.Errorf("expected v2 payload to contain %s", banned) + } + } + if strings.Contains(string(data), `"is_constructor_call"`) { + t.Error("v2 payload should not contain dropped is_constructor_call") + } +} + +func TestEmit_TypeKindAndMethods(t *testing.T) { + app, dir := buildGreeter(t) + out := Emit(app, "greeter", dir, 1) + + // Find the Greeter type wherever it lives; assert struct kind + methods. + var found bool + for _, mod := range out.Application.SymbolTable { + for name, ty := range mod.Types { + if ty.Kind != "struct" && ty.Kind != "interface" { + t.Errorf("type %s kind = %q, want struct|interface", name, ty.Kind) + } + if strings.Contains(ty.ID, "://") == false { + t.Errorf("type %s id should be a can:// id; got %q", name, ty.ID) + } + for sig, m := range ty.Callables { + if m.Kind != "method" { + t.Errorf("callable %s under type %s kind = %q, want method", sig, name, m.Kind) + } + found = true + } + } + } + if !found { + t.Skip("no type methods in fixture; envelope/module coverage still asserted elsewhere") + } +} + +func head(s string, n int) string { + if len(s) < n { + return s + } + return s[:n] +} + +func keys[V any](m map[string]V) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + return out +} From c94d44173c51717da26ed62292851c5d397406eb Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 23:01:08 -0400 Subject: [PATCH 07/29] feat(cli): --analysis-schema 2 selects v2 output Wire the v2 emitter behind an opt-in flag. Default stays the v1 shape so the current SDK keeps parsing output during the migration; --analysis- schema 2 emits the canonical v2 tree. - Add options.SchemaVersion (1 default, 2 v2) and AnalyzerVersion. - core.RenderJSON centralizes the v1/v2 marshal decision so the stdout and file paths agree; WriteOutput now takes opts and routes through it. An unknown schema returns a non-zero error, never a silent fallback. - --analysis-schema flag; --app-name/version threaded into the v2 manifest (v2 surfaces the analyzer version, which v1 never carried). Update WriteOutput callers (now opts-based) in core tests. CLI tests assert the default is v1 (top-level symbol_table, no schema_version), that --analysis-schema 2 emits schema_version 2.0.0 with application.id can://go/greeter, and that an unknown schema errors. --- cmd/codeanalyzer/main.go | 83 ++++++++++--------- cmd/codeanalyzer/main_test.go | 63 ++++++++++++++ internal/core/analyzer.go | 49 +++++++---- internal/core/analyzer_test.go | 8 +- internal/core/testsetup_test.go | 3 +- internal/options/options.go | 6 ++ internal/syntactic_analysis/v2emit/emit.go | 7 +- .../syntactic_analysis/v2emit/emit_test.go | 10 +-- 8 files changed, 162 insertions(+), 67 deletions(-) diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 49bda45..5b0b40f 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -5,7 +5,6 @@ package main import ( - "encoding/json" "fmt" "os" "path/filepath" @@ -32,24 +31,25 @@ func main() { func rootCmd() *cobra.Command { var ( - inputPath string - outputDir string - format string - emit string - appName string - level int - targetFiles []string - skipTests bool - eager bool - cacheDir string - jobs int - useCodeQL bool - verbosity int - showVersion bool - neo4jURI string - neo4jUser string - neo4jPassword string - neo4jDatabase string + inputPath string + outputDir string + format string + emit string + appName string + level int + analysisSchema int + targetFiles []string + skipTests bool + eager bool + cacheDir string + jobs int + useCodeQL bool + verbosity int + showVersion bool + neo4jURI string + neo4jUser string + neo4jPassword string + neo4jDatabase string ) cmd := &cobra.Command{ @@ -107,23 +107,25 @@ via CLDK(language="go").analysis(project_path=...).`, } opts := options.AnalysisOptions{ - InputPath: inputPath, - OutputDir: outputDir, - Format: format, - Emit: options.EmitTarget(emit), - AppName: appName, - Level: options.AnalysisLevel(level), - TargetFiles: targetFiles, - SkipTests: skipTests, - Eager: eager, - CacheDir: cacheDir, - Jobs: jobs, - UseCodeQL: useCodeQL, - Verbose: verbosity > 0, - Neo4jURI: firstNonEmpty(neo4jURI, os.Getenv("NEO4J_URI")), - Neo4jUser: firstNonEmpty(neo4jUser, os.Getenv("NEO4J_USERNAME"), "neo4j"), - Neo4jPassword: firstNonEmpty(neo4jPassword, os.Getenv("NEO4J_PASSWORD"), "neo4j"), - Neo4jDatabase: firstNonEmpty(neo4jDatabase, os.Getenv("NEO4J_DATABASE")), + InputPath: inputPath, + OutputDir: outputDir, + Format: format, + Emit: options.EmitTarget(emit), + AppName: appName, + Level: options.AnalysisLevel(level), + SchemaVersion: analysisSchema, + AnalyzerVersion: version, + TargetFiles: targetFiles, + SkipTests: skipTests, + Eager: eager, + CacheDir: cacheDir, + Jobs: jobs, + UseCodeQL: useCodeQL, + Verbose: verbosity > 0, + Neo4jURI: firstNonEmpty(neo4jURI, os.Getenv("NEO4J_URI")), + Neo4jUser: firstNonEmpty(neo4jUser, os.Getenv("NEO4J_USERNAME"), "neo4j"), + Neo4jPassword: firstNonEmpty(neo4jPassword, os.Getenv("NEO4J_PASSWORD"), "neo4j"), + Neo4jDatabase: firstNonEmpty(neo4jDatabase, os.Getenv("NEO4J_DATABASE")), } analyzer := core.New(opts) @@ -133,16 +135,17 @@ via CLDK(language="go").analysis(project_path=...).`, } // When no --output dir is given, write JSON to cobra's output - // writer so tests can capture it via cmd.SetOut. + // writer so tests can capture it via cmd.SetOut. Both paths render + // through core so the v1/v2 schema selection stays in one place. if outputDir == "" { - data, err := json.Marshal(app) + data, err := core.RenderJSON(app, opts) if err != nil { return err } _, err = cmd.OutOrStdout().Write(data) return err } - return core.WriteOutput(app, outputDir, format) + return core.WriteOutput(app, opts) }, } @@ -155,6 +158,8 @@ via CLDK(language="go").analysis(project_path=...).`, "Application anchor name for can:// ids and Neo4j :Application (default: input dir name)") f.IntVarP(&level, "analysis-level", "a", 1, "Analysis level: 1=symbol table only, 2=+resolver call graph") + f.IntVar(&analysisSchema, "analysis-schema", 1, + "Output schema major: 1=legacy v1 shape (default), 2=canonical v2 tree") f.StringSliceVarP(&targetFiles, "target-files", "t", nil, "Restrict analysis to specific files (incremental mode)") f.BoolVar(&skipTests, "skip-tests", true, "Skip *_test.go files") diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index ea5ff3b..3843261 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -196,6 +196,69 @@ func TestRootCmd_Level2ProducesCallGraph(t *testing.T) { } } +// ── --analysis-schema ────────────────────────────────────────────────────────── + +func TestRootCmd_DefaultSchemaIsV1(t *testing.T) { + td := cliTestdataDir() + out, _, err := runCmd("--input", filepath.Join(td, "greeter"), "--cache-dir", t.TempDir()) + if err != nil { + t.Fatalf("command failed: %v", err) + } + var v map[string]interface{} + if jsonErr := json.Unmarshal([]byte(out), &v); jsonErr != nil { + t.Fatalf("stdout is not valid JSON: %v", jsonErr) + } + if _, ok := v["symbol_table"]; !ok { + t.Error("default output should be the v1 shape (top-level symbol_table)") + } + if _, ok := v["schema_version"]; ok { + t.Error("default output should not carry schema_version (that is v2)") + } +} + +func TestRootCmd_AnalysisSchema2EmitsV2(t *testing.T) { + td := cliTestdataDir() + out, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-schema", "2", + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("command failed: %v", err) + } + var v struct { + SchemaVersion string `json:"schema_version"` + Language string `json:"language"` + Application struct { + ID string `json:"id"` + } `json:"application"` + } + if jsonErr := json.Unmarshal([]byte(out), &v); jsonErr != nil { + t.Fatalf("stdout is not valid JSON: %v", jsonErr) + } + if v.SchemaVersion != "2.0.0" { + t.Errorf("schema_version = %q, want 2.0.0", v.SchemaVersion) + } + if v.Language != "go" { + t.Errorf("language = %q, want go", v.Language) + } + if v.Application.ID != "can://go/greeter" { + t.Errorf("application.id = %q, want can://go/greeter", v.Application.ID) + } +} + +func TestRootCmd_UnknownSchemaReturnsError(t *testing.T) { + td := cliTestdataDir() + _, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-schema", "9", + "--cache-dir", t.TempDir(), + ) + if err == nil { + t.Fatal("expected error for unknown --analysis-schema value, got nil") + } +} + // ── --skip-tests ───────────────────────────────────────────────────────────── func TestRootCmd_SkipTestsFalseIncludesTestFiles(t *testing.T) { diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index 2b9f410..d889a2f 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -5,11 +5,11 @@ // discipline of codeanalyzer-python/codeanalyzer/core.py. // // Phase order: -// 1. Project materialization (go mod download) -// 2. Symbol table construction (syntactic_analysis) -// 3. Resolver-based call graph (semantic_analysis) — if level >= 2 -// 4. Pass pipeline (analysis/registry) -// 5. Optional CodeQL enrichment (semantic_analysis/codeql) — if --codeql +// 1. Project materialization (go mod download) +// 2. Symbol table construction (syntactic_analysis) +// 3. Resolver-based call graph (semantic_analysis) — if level >= 2 +// 4. Pass pipeline (analysis/registry) +// 5. Optional CodeQL enrichment (semantic_analysis/codeql) — if --codeql package core import ( @@ -25,6 +25,7 @@ import ( "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis/codeql" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) @@ -170,10 +171,14 @@ func (a *Analyzer) saveCache(app *schema.GoApplication) error { return os.WriteFile(cachePath, data, 0o644) } -// WriteOutput writes the GoApplication to outputDir/analysis.json (or stdout -// when outputDir is empty). Only "json" is supported; "msgpack" and other -// values return an explicit error rather than silently falling back to JSON. -func WriteOutput(app *schema.GoApplication, outputDir, format string) error { +// RenderJSON serializes the analysis to JSON in the schema selected by +// opts.SchemaVersion: 1 (default) emits the legacy v1 GoApplication shape; 2 +// emits the canonical v2 tree via the v2emit emitter. This is the single place +// the v1/v2 decision is made, so both the stdout and file output paths agree. +// Only "json" is supported; other formats return an explicit error rather than +// silently falling back. +func RenderJSON(app *schema.GoApplication, opts options.AnalysisOptions) ([]byte, error) { + format := opts.Format if format == "" { format = "json" } @@ -181,21 +186,35 @@ func WriteOutput(app *schema.GoApplication, outputDir, format string) error { case "json": // only supported format case "msgpack": - return fmt.Errorf("msgpack output is not yet implemented; use --format json") + return nil, fmt.Errorf("msgpack output is not yet implemented; use --format json") default: - return fmt.Errorf("unsupported output format %q; supported: json", format) + return nil, fmt.Errorf("unsupported output format %q; supported: json", format) } - data, err := json.Marshal(app) + switch opts.SchemaVersion { + case 0, 1: + return json.Marshal(app) + case 2: + payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion) + return json.Marshal(payload) + default: + return nil, fmt.Errorf("unsupported --analysis-schema %d; supported: 1, 2", opts.SchemaVersion) + } +} + +// WriteOutput serializes the analysis (in the selected schema) and writes it to +// outputDir/analysis.json, or to stdout when outputDir is empty. +func WriteOutput(app *schema.GoApplication, opts options.AnalysisOptions) error { + data, err := RenderJSON(app, opts) if err != nil { return err } - if outputDir == "" { + if opts.OutputDir == "" { _, err = os.Stdout.Write(data) return err } - if err := utils.EnsureDir(outputDir); err != nil { + if err := utils.EnsureDir(opts.OutputDir); err != nil { return err } - return os.WriteFile(filepath.Join(outputDir, "analysis.json"), data, 0o644) + return os.WriteFile(filepath.Join(opts.OutputDir, "analysis.json"), data, 0o644) } diff --git a/internal/core/analyzer_test.go b/internal/core/analyzer_test.go index 1e95011..45dacf2 100644 --- a/internal/core/analyzer_test.go +++ b/internal/core/analyzer_test.go @@ -137,7 +137,7 @@ func TestCallGraph_CallSitesBackfilled(t *testing.T) { func TestWriteOutput_ValidJSON(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL2, outDir, "json"); err != nil { + if err := core.WriteOutput(sharedGreeterL2, options.AnalysisOptions{OutputDir: outDir, Format: "json"}); err != nil { t.Fatalf("WriteOutput: %v", err) } data, err := os.ReadFile(filepath.Join(outDir, "analysis.json")) @@ -155,7 +155,7 @@ func TestWriteOutput_ValidJSON(t *testing.T) { func TestWriteOutput_EmptyFormatDefaultsToJSON(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL1, outDir, ""); err != nil { + if err := core.WriteOutput(sharedGreeterL1, options.AnalysisOptions{OutputDir: outDir, Format: ""}); err != nil { t.Fatalf("WriteOutput with empty format: %v", err) } if _, err := os.Stat(filepath.Join(outDir, "analysis.json")); err != nil { @@ -165,14 +165,14 @@ func TestWriteOutput_EmptyFormatDefaultsToJSON(t *testing.T) { func TestWriteOutput_MsgpackNotImplemented(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL1, outDir, "msgpack"); err == nil { + if err := core.WriteOutput(sharedGreeterL1, options.AnalysisOptions{OutputDir: outDir, Format: "msgpack"}); err == nil { t.Fatal("expected error for --format msgpack, got nil") } } func TestWriteOutput_UnknownFormatErrors(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL1, outDir, "csv"); err == nil { + if err := core.WriteOutput(sharedGreeterL1, options.AnalysisOptions{OutputDir: outDir, Format: "csv"}); err == nil { t.Fatal("expected error for unknown format, got nil") } } diff --git a/internal/core/testsetup_test.go b/internal/core/testsetup_test.go index 85f3b72..2b14d32 100644 --- a/internal/core/testsetup_test.go +++ b/internal/core/testsetup_test.go @@ -66,6 +66,7 @@ func runTestMain(m *testing.M) int { opts := options.AnalysisOptions{ InputPath: f.path, OutputDir: outDir, + Format: "json", Level: f.level, SkipTests: true, CacheDir: cacheDir, @@ -75,7 +76,7 @@ func runTestMain(m *testing.M) int { fmt.Fprintf(os.Stderr, "testsetup %s: Analyze: %v\n", f.name, err) return 1 } - if err := core.WriteOutput(app, outDir, "json"); err != nil { + if err := core.WriteOutput(app, opts); err != nil { fmt.Fprintf(os.Stderr, "testsetup %s: WriteOutput: %v\n", f.name, err) return 1 } diff --git a/internal/options/options.go b/internal/options/options.go index 051fe6c..e19d542 100644 --- a/internal/options/options.go +++ b/internal/options/options.go @@ -38,6 +38,12 @@ type AnalysisOptions struct { AppName string // AnalysisLevel controls symbol-table-only (1) vs + call graph (2). Level AnalysisLevel + // SchemaVersion selects the output schema major: 1 = the legacy v1 shape + // (default, compat shim during the migration), 2 = the canonical v2 tree. + SchemaVersion int + // AnalyzerVersion is the analyzer's own version, recorded in the v2 + // manifest's analyzer{} tag. Set by the CLI from its build-stamped value. + AnalyzerVersion string // TargetFiles restricts analysis to specific files (incremental mode). TargetFiles []string // SkipTests skips test files (files ending in _test.go). diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index d2501d2..f4ba1f0 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -12,18 +12,19 @@ import ( // Emit transforms the v1 GoApplication into the v2 Analysis payload. appName is // the application anchor (--app-name); projectDir is the absolute input root, // used to read each module's source for span slicing. maxLevel records how -// deeply the tree was populated (1 = symbol table, 2 = + call_graph). +// deeply the tree was populated (1 = symbol table, 2 = + call_graph); +// analyzerVersion is stamped into the manifest's analyzer{} tag. // // This is a pure re-serialization: every fact comes from the v1 model, except // span byte offsets, which are derived from source via lineIndex. -func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int) *v2.Analysis { +func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string) *v2.Analysis { appID := v2.AppID(appName) out := &v2.Analysis{ SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: maxLevel, - Analyzer: v2.AnalyzerTag{Name: "codeanalyzer-go"}, + Analyzer: v2.AnalyzerTag{Name: "codeanalyzer-go", Version: analyzerVersion}, Application: v2.Application{ ID: appID, Kind: "application", diff --git a/internal/syntactic_analysis/v2emit/emit_test.go b/internal/syntactic_analysis/v2emit/emit_test.go index deb8a66..6dd39ee 100644 --- a/internal/syntactic_analysis/v2emit/emit_test.go +++ b/internal/syntactic_analysis/v2emit/emit_test.go @@ -37,7 +37,7 @@ func buildGreeter(t *testing.T) (*schema.GoApplication, string) { func TestEmit_Envelope(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1) + out := Emit(app, "greeter", dir, 1, "test") if out.SchemaVersion != "2.0.0" { t.Errorf("schema_version = %q, want 2.0.0", out.SchemaVersion) @@ -61,7 +61,7 @@ func TestEmit_Envelope(t *testing.T) { func TestEmit_ModuleSourceAndIDs(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1) + out := Emit(app, "greeter", dir, 1, "test") mod, ok := out.Application.SymbolTable["main.go"] if !ok { @@ -84,7 +84,7 @@ func TestEmit_ModuleSourceAndIDs(t *testing.T) { func TestEmit_CallableBodyAndSpanSlice(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1) + out := Emit(app, "greeter", dir, 1, "test") mod := out.Application.SymbolTable["main.go"] var main v2.Callable @@ -123,7 +123,7 @@ func TestEmit_CallableBodyAndSpanSlice(t *testing.T) { func TestEmit_CalleeSerializesAsNull(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1) + out := Emit(app, "greeter", dir, 1, "test") data, err := json.Marshal(out) if err != nil { t.Fatalf("marshal failed: %v", err) @@ -144,7 +144,7 @@ func TestEmit_CalleeSerializesAsNull(t *testing.T) { func TestEmit_TypeKindAndMethods(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1) + out := Emit(app, "greeter", dir, 1, "test") // Find the Greeter type wherever it lives; assert struct kind + methods. var found bool From e8a958122f96f1890a79824c29a2a9f3ec145240 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 23:05:05 -0400 Subject: [PATCH 08/29] test: L1 v2 fixture gate over the multipackage fixture Lock the v2 L1 output shape against a known multi-file fixture so the level cannot silently regress. Asserts concrete values for every item on the L1 coverage checklist (testing-and-validation.md): - multi-file compilation unit: four modules keyed by relative path, each with source-once and a whole-file span that slices back - exported + unexported symbols (Worker.Run vs Worker.execute) - interface kind collapse (Processor -> kind=interface) - error_channel from an error-typed return (Processor.Process) - variadic parameter (Combine(results ...Result)) - language-specific call flag (is_goroutine on `go w.execute(...)`) - sanctioned callee:null refinement slot present at L1 - can:// durable ids at module/type/callable depth Plus the superset gate: walk the same v1 model the emitter walked and assert every file/type/method/function reappears in v2, and that the sanctioned drops (is_constructor_call, per-node code) are absent from the serialized payload. --- .../syntactic_analysis/v2emit/gate_test.go | 311 ++++++++++++++++++ 1 file changed, 311 insertions(+) create mode 100644 internal/syntactic_analysis/v2emit/gate_test.go diff --git a/internal/syntactic_analysis/v2emit/gate_test.go b/internal/syntactic_analysis/v2emit/gate_test.go new file mode 100644 index 0000000..db1317a --- /dev/null +++ b/internal/syntactic_analysis/v2emit/gate_test.go @@ -0,0 +1,311 @@ +package v2emit + +// L1 conformance gate for the v2 emitter. +// +// This is the fixture gate the codeanalyzer-backend ladder requires before L1 +// advances: it locks the v2 L1 OUTPUT SHAPE against a known multi-file fixture, +// asserting concrete values (not just "a key exists") for every item on the L1 +// coverage checklist in +// designing-cldk-changes/references/testing-and-validation.md: +// +// - a multi-file compilation unit (four modules keyed by relative path) +// - exported AND unexported symbols (Worker.Run vs Worker.execute) +// - an interface type collapsed into kind=interface (Processor) +// - error_channel populated from an error-typed return (Processor.Process) +// - a variadic parameter (Combine(results ...Result)) +// - a language-specific call flag (is_goroutine on `go w.execute(...)`) +// - the sanctioned callee:null refinement slot present at L1 +// - can:// durable ids at module/type/callable depth +// - the SUPERSET gate: every v1 fact survives into v2 modulo sanctioned drops +// (is_constructor_call is dropped; per-node `code` folds into module.source). +// +// If any of these regress, this gate fails and the level does not advance. + +import ( + "encoding/json" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" +) + +// multipackageDir returns the absolute path to the multipackage testdata fixture. +func multipackageDir(t *testing.T) string { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + abs, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", "multipackage")) + if err != nil { + t.Fatalf("resolving fixture dir: %v", err) + } + return abs +} + +// buildMultipackageV2 runs the real symbol-table builder over the multipackage +// fixture and emits the v2 L1 payload. skipTests=true keeps *_test.go out so the +// module set is deterministic. +func buildMultipackageV2(t *testing.T) *v2.Analysis { + t.Helper() + dir := multipackageDir(t) + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + app := &schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}} + return Emit(app, "multipackage", dir, 1, "gate-test") +} + +// mustModule fetches a module by relative path or fails. +func mustModule(t *testing.T, out *v2.Analysis, rel string) v2.Module { + t.Helper() + mod, ok := out.Application.SymbolTable[rel] + if !ok { + t.Fatalf("module %q missing; modules=%v", rel, keys(out.Application.SymbolTable)) + } + return mod +} + +// TestGateL1_ModuleSet locks the exact multi-file compilation unit: the four +// non-test source files, each a module keyed by its relative path. +func TestGateL1_ModuleSet(t *testing.T) { + out := buildMultipackageV2(t) + + want := []string{ + "main.go", + "server/middleware.go", + "server/server.go", + "worker/worker.go", + } + for _, rel := range want { + mod := mustModule(t, out, rel) + if mod.Kind != "module" { + t.Errorf("%s: kind = %q, want module", rel, mod.Kind) + } + wantID := "can://go/multipackage/" + rel + if mod.ID != wantID { + t.Errorf("%s: id = %q, want %q", rel, mod.ID, wantID) + } + if mod.Source == "" { + t.Errorf("%s: source is empty; every module carries its whole file text", rel) + } + // The module span must slice back to the whole source (source-once invariant). + if got := mod.Span.Slice(mod.Source); got != mod.Source { + t.Errorf("%s: module span does not cover whole source (%d vs %d bytes)", rel, len(got), len(mod.Source)) + } + } +} + +// TestGateL1_InterfaceKind locks the Processor interface: is_interface collapses +// into kind=interface, and its method resolves as a callable under the type. +func TestGateL1_InterfaceKind(t *testing.T) { + out := buildMultipackageV2(t) + worker := mustModule(t, out, "worker/worker.go") + + proc, ok := worker.Types["Processor"] + if !ok { + t.Fatalf("Processor type missing; types=%v", keys(worker.Types)) + } + if proc.Kind != "interface" { + t.Errorf("Processor.kind = %q, want interface", proc.Kind) + } + const procMethod = "example.com/multipackage/worker.Processor.Process" + m, ok := proc.Callables[procMethod] + if !ok { + t.Fatalf("Processor.Process missing; callables=%v", keys(proc.Callables)) + } + if m.Kind != "method" { + t.Errorf("Process.kind = %q, want method", m.Kind) + } + // error_channel is populated from the (Result, error) return. + if !contains(m.ErrorChannel, "error") { + t.Errorf("Process.error_channel = %v, want to contain \"error\"", m.ErrorChannel) + } +} + +// TestGateL1_ExportedAndUnexported locks that both an exported (Run) and an +// unexported (execute) method survive, each a method callable under Worker. +func TestGateL1_ExportedAndUnexported(t *testing.T) { + out := buildMultipackageV2(t) + worker := mustModule(t, out, "worker/worker.go") + + wtype, ok := worker.Types["Worker"] + if !ok { + t.Fatalf("Worker type missing; types=%v", keys(worker.Types)) + } + const ( + runID = "example.com/multipackage/worker.Worker.Run" + execID = "example.com/multipackage/worker.Worker.execute" + ) + for _, sig := range []string{runID, execID} { + m, ok := wtype.Callables[sig] + if !ok { + t.Fatalf("method %q missing; callables=%v", sig, keys(wtype.Callables)) + } + if m.Kind != "method" { + t.Errorf("%s.kind = %q, want method", sig, m.Kind) + } + } +} + +// TestGateL1_VariadicParameter locks the variadic-parameter fact: +// Combine(results ...Result) → is_variadic=true on the sole parameter. +func TestGateL1_VariadicParameter(t *testing.T) { + out := buildMultipackageV2(t) + worker := mustModule(t, out, "worker/worker.go") + + const combineID = "example.com/multipackage/worker.Combine" + comb, ok := worker.Functions[combineID] + if !ok { + t.Fatalf("Combine function missing; functions=%v", keys(worker.Functions)) + } + if len(comb.Parameters) != 1 { + t.Fatalf("Combine should have 1 parameter; got %d", len(comb.Parameters)) + } + if !comb.Parameters[0].IsVariadic { + t.Errorf("Combine parameter %q should be variadic", comb.Parameters[0].Name) + } +} + +// TestGateL1_GoroutineCallNode locks the language-specific call flag: Worker.Run +// contains `go w.execute(...)`, which must surface as a body call node with +// is_goroutine=true AND callee null at L1. +func TestGateL1_GoroutineCallNode(t *testing.T) { + out := buildMultipackageV2(t) + worker := mustModule(t, out, "worker/worker.go") + + run, ok := worker.Types["Worker"].Callables["example.com/multipackage/worker.Worker.Run"] + if !ok { + t.Fatal("Worker.Run missing") + } + if len(run.Body) == 0 { + t.Fatal("Worker.Run should have body call nodes") + } + sawGoroutine := false + for local, node := range run.Body { + if node.Kind != "call" { + t.Errorf("body[%s].kind = %q, want call", local, node.Kind) + } + if node.Callee != nil { + t.Errorf("body[%s].callee should be null at L1; got %q", local, *node.Callee) + } + if node.IsGoroutine { + sawGoroutine = true + } + } + if !sawGoroutine { + t.Error("Worker.Run should have a call node with is_goroutine=true (go w.execute(...))") + } +} + +// TestGateL1_CrossPackageStructure locks that a cross-package caller (main.go +// calling into server/ and worker/) is present as its own module with body call +// nodes — the compilation unit spans packages, not a single file. +func TestGateL1_CrossPackageStructure(t *testing.T) { + out := buildMultipackageV2(t) + main := mustModule(t, out, "main.go") + + if main.Package != "main" { + t.Errorf("main.go package = %q, want main", main.Package) + } + // main() drives the cross-package calls; its body must carry call nodes. + var mainFn v2.Callable + found := false + for _, fn := range main.Functions { + if strings.HasSuffix(fn.Signature, ".main") { + mainFn, found = fn, true + } + } + if !found { + t.Fatalf("main function missing; functions=%v", keys(main.Functions)) + } + if len(mainFn.Body) == 0 { + t.Error("main() should have body call nodes for its cross-package calls") + } +} + +// TestGateL1_Envelope locks the manifest envelope at L1. +func TestGateL1_Envelope(t *testing.T) { + out := buildMultipackageV2(t) + + if out.SchemaVersion != v2.SchemaVersion { + t.Errorf("schema_version = %q, want %q", out.SchemaVersion, v2.SchemaVersion) + } + if out.Language != "go" { + t.Errorf("language = %q, want go", out.Language) + } + if out.MaxLevel != 1 { + t.Errorf("max_level = %d, want 1", out.MaxLevel) + } + if out.Application.ID != "can://go/multipackage" { + t.Errorf("application.id = %q, want can://go/multipackage", out.Application.ID) + } + // L1: no call_graph edges yet. + if len(out.Application.CallGraph) != 0 { + t.Errorf("L1 call_graph should be empty; got %d edges", len(out.Application.CallGraph)) + } +} + +// TestGateL1_SupersetOfV1 is the superset gate: every v1 fact must survive into +// v2 modulo the sanctioned drops. It walks the SAME v1 model the emitter walked +// and asserts each callable/type/function reappears in v2, and that the +// sanctioned drops are gone from the serialized payload. +func TestGateL1_SupersetOfV1(t *testing.T) { + dir := multipackageDir(t) + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + app := &schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}} + out := Emit(app, "multipackage", dir, 1, "gate-test") + + // Every v1 file, type, method and function reappears in v2. + for rel, v1file := range st { + mod, ok := out.Application.SymbolTable[rel] + if !ok { + t.Errorf("v1 file %q dropped from v2 symbol_table", rel) + continue + } + for name := range v1file.Types { + if _, ok := mod.Types[name]; !ok { + t.Errorf("%s: v1 type %q dropped from v2", rel, name) + } + for msig := range v1file.Types[name].Methods { + if _, ok := mod.Types[name].Callables[msig]; !ok { + t.Errorf("%s: v1 method %q dropped from v2 type %q", rel, msig, name) + } + } + } + for sig := range v1file.Functions { + if _, ok := mod.Functions[sig]; !ok { + t.Errorf("%s: v1 function %q dropped from v2", rel, sig) + } + } + } + + // Sanctioned drops must be absent from the serialized payload. + data, err := json.Marshal(out) + if err != nil { + t.Fatalf("marshal failed: %v", err) + } + if strings.Contains(string(data), `"is_constructor_call"`) { + t.Error("v2 payload should not carry the dropped is_constructor_call") + } + // The sanctioned callee:null slot must be present at L1. + if !strings.Contains(string(data), `"callee":null`) { + t.Error("expected the sanctioned callee:null refinement slot at L1") + } +} + +func contains(ss []string, want string) bool { + for _, s := range ss { + if s == want { + return true + } + } + return false +} From b133324f6de3493a6464096fa720731c2e9e5376 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 23:11:00 -0400 Subject: [PATCH 09/29] =?UTF-8?q?feat(emit):=20v2=20L2=20refinement=20?= =?UTF-8?q?=E2=80=94=20can://=20ids=20on=20callees=20and=20edges?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit L2 is a pure refinement over the L1 tree: it fills the sanctioned callee null->id slot and translates call_graph edge endpoints onto real callable ids. Both need one thing v1 does not have: a mapping from v1 signature identity to v2 can:// node id. Add buildSigIndex, a one-pass signature->id index built with the same id builders emitCallable uses, so an indexed id is byte-identical to the id the callable node carries (no dangling endpoints by construction). Then: - emitBody: backfill body.callee to the callee's can:// id when the resolver-set signature names an in-tree callable; keep it null for external/stdlib callees (the honest-unresolved fallback). - emitCallGraph: map src/dst from v1 signatures to can:// ids; drop any edge whose endpoint does not resolve rather than emit a dangling one (defensive — the v1 resolver only emits in-project targets). At L1 (no resolver run) callee stays null, so L1 output is unchanged. --- internal/syntactic_analysis/v2emit/emit.go | 72 ++++++++++++++----- .../syntactic_analysis/v2emit/sigindex.go | 59 +++++++++++++++ 2 files changed, 113 insertions(+), 18 deletions(-) create mode 100644 internal/syntactic_analysis/v2emit/sigindex.go diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index f4ba1f0..1fb7d7e 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -20,6 +20,11 @@ import ( func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string) *v2.Analysis { appID := v2.AppID(appName) + // The signature→can:// id index translates v1 identity (signature strings) + // into v2 node ids. It is what the L2 refinement uses to backfill body + // callees and to map call_graph edge endpoints onto real callable ids. + idx := buildSigIndex(appID, app.SymbolTable) + out := &v2.Analysis{ SchemaVersion: v2.SchemaVersion, Language: v2.Language, @@ -29,7 +34,7 @@ func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, a ID: appID, Kind: "application", SymbolTable: make(map[string]v2.Module, len(app.SymbolTable)), - CallGraph: emitCallGraph(app.CallGraph), + CallGraph: emitCallGraph(app.CallGraph, idx), }, } @@ -37,7 +42,7 @@ func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, a // sorted keeps any incidental ordering (and logs) stable across runs. for _, relPath := range sortedFileKeys(app.SymbolTable) { file := app.SymbolTable[relPath] - out.Application.SymbolTable[relPath] = emitModule(appID, projectDir, relPath, file) + out.Application.SymbolTable[relPath] = emitModule(appID, projectDir, relPath, file, idx) } return out } @@ -45,7 +50,7 @@ func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, a // emitModule builds a v2 module from a v1 GoFile, reading the file's source so // spans can slice off it. A file that cannot be read still emits its structure // (with empty source and zeroed byte spans) rather than dropping the module. -func emitModule(appID, projectDir, relPath string, file schema.GoFile) v2.Module { +func emitModule(appID, projectDir, relPath string, file schema.GoFile, idx sigIndex) v2.Module { source := readSource(projectDir, relPath) li := newLineIndex(source) modID := v2.ModuleID(appID, relPath) @@ -65,10 +70,10 @@ func emitModule(appID, projectDir, relPath string, file schema.GoFile) v2.Module } for name, t := range file.Types { - mod.Types[name] = emitType(modID, li, t) + mod.Types[name] = emitType(modID, li, t, idx) } for sig, fn := range file.Functions { - mod.Functions[sig] = emitCallable(modID, li, fn) + mod.Functions[sig] = emitCallable(modID, li, fn, idx) } return mod } @@ -76,7 +81,7 @@ func emitModule(appID, projectDir, relPath string, file schema.GoFile) v2.Module // emitType maps a v1 GoType to a v2 type node, collapsing is_interface into the // kind and splitting embedding (base_types) from computed satisfaction. Methods // resolve into the type's callables{}. -func emitType(modID string, li *lineIndex, t schema.GoType) v2.Type { +func emitType(modID string, li *lineIndex, t schema.GoType, idx sigIndex) v2.Type { typeID := v2.TypeID(modID, t.Signature) out := v2.Type{ ID: typeID, @@ -97,14 +102,14 @@ func emitType(modID string, li *lineIndex, t schema.GoType) v2.Type { } } for sig, m := range t.Methods { - out.Callables[sig] = emitCallable(typeID, li, m) + out.Callables[sig] = emitCallable(typeID, li, m, idx) } return out } // emitCallable maps a v1 GoCallable to a v2 callable node, including its body // (call nodes) and nested closures. -func emitCallable(parentID string, li *lineIndex, c schema.GoCallable) v2.Callable { +func emitCallable(parentID string, li *lineIndex, c schema.GoCallable, idx sigIndex) v2.Callable { callID := v2.CallableID(parentID, c.Signature) out := v2.Callable{ ID: callID, @@ -113,7 +118,7 @@ func emitCallable(parentID string, li *lineIndex, c schema.GoCallable) v2.Callab Signature: c.Signature, Parameters: emitParams(li, c.Parameters), ReturnType: c.ReturnType, - Body: emitBody(callID, li, c.CallSites), + Body: emitBody(callID, li, c.CallSites, idx), } if ch := errorChannel(c.ReturnTypes); len(ch) > 0 { out.ErrorChannel = ch @@ -124,28 +129,50 @@ func emitCallable(parentID string, li *lineIndex, c schema.GoCallable) v2.Callab if len(c.InnerCallables) > 0 { out.Callables = make(map[string]v2.Callable, len(c.InnerCallables)) for sig, inner := range c.InnerCallables { - out.Callables[sig] = emitCallable(callID, li, inner) + out.Callables[sig] = emitCallable(callID, li, inner, idx) } } return out } -// emitBody builds the L1 body map: one `call` node per call site, keyed by its -// line:col local id, with the sanctioned callee null->id refinement slot. -func emitBody(callableID string, li *lineIndex, sites []schema.GoCallsite) map[string]v2.BodyNode { +// emitBody builds the body map: one `call` node per call site, keyed by its +// line:col local id, carrying the sanctioned callee null->id refinement slot. +// +// callee resolution: +// - L1: the v1 site has no backfilled signature, so callee stays null. +// - L2: the resolver backfilled the site's callee signature. When that +// signature names an in-tree callable, callee becomes its can:// id; when +// it names an external/stdlib callee (no in-tree node), callee stays null — +// the honest-unresolved fallback the L2 gate expects. +func emitBody(callableID string, li *lineIndex, sites []schema.GoCallsite, idx sigIndex) map[string]v2.BodyNode { body := make(map[string]v2.BodyNode, len(sites)) for _, cs := range sites { local := v2.LocalID(cs.StartLine, cs.StartColumn) body[local] = v2.BodyNode{ Kind: "call", Span: colSpan(li, cs.StartLine, cs.StartColumn, cs.EndLine, cs.EndColumn), - Callee: cs.CalleeSignature, // *string: null at L1, backfilled at L2 + Callee: calleeID(cs.CalleeSignature, idx), IsGoroutine: cs.IsGoroutine, } } return body } +// calleeID translates a v1 backfilled callee signature into the v2 can:// id +// refinement slot: nil (JSON null) when unresolved or external, a pointer to the +// in-tree callable id when resolved. Never returns the raw v1 signature — a +// call node's callee is always a node id or null. +func calleeID(sig *string, idx sigIndex) *string { + if sig == nil { + return nil + } + id, ok := idx.idFor(*sig) + if !ok { + return nil + } + return &id +} + func emitParams(li *lineIndex, params []schema.GoParameter) []v2.Parameter { out := make([]v2.Parameter, 0, len(params)) for _, p := range params { @@ -173,13 +200,22 @@ func emitImports(li *lineIndex, imports []schema.GoImport) []v2.Import { } // emitCallGraph maps v1 identity-only edges to the v2 {src,dst,prov,weight} -// shape (rename of source/target/provenance). -func emitCallGraph(edges []schema.GoCallEdge) []v2.Edge { +// shape: the source/target/provenance rename, PLUS the endpoint translation from +// v1 signatures to can:// callable ids. An edge whose src or dst does not resolve +// to an in-tree node is dropped rather than emitted with a dangling endpoint — +// the L2 gate requires every endpoint to be a real callable id. (The v1 resolver +// only emits edges to in-project targets, so this drop is defensive.) +func emitCallGraph(edges []schema.GoCallEdge, idx sigIndex) []v2.Edge { out := make([]v2.Edge, 0, len(edges)) for _, e := range edges { + src, srcOK := idx.idFor(e.Source) + dst, dstOK := idx.idFor(e.Target) + if !srcOK || !dstOK { + continue + } out = append(out, v2.Edge{ - Src: e.Source, - Dst: e.Target, + Src: src, + Dst: dst, Prov: nonEmpty(e.Provenance), Weight: e.Weight, }) diff --git a/internal/syntactic_analysis/v2emit/sigindex.go b/internal/syntactic_analysis/v2emit/sigindex.go new file mode 100644 index 0000000..f0e8cd8 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/sigindex.go @@ -0,0 +1,59 @@ +package v2emit + +import ( + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// sigIndex maps a v1 callable signature to its v2 can:// callable id. +// +// v1 identifies every callable by its signatureOf() string and uses that string +// as call-graph edge endpoints and as a call site's backfilled callee. v2 +// addresses callables by durable can:// id. The L2 refinement therefore needs a +// signature→id translation: this index is built in one pass over the same tree +// the emitter walks, using the SAME id builders emitCallable uses, so an id here +// is byte-identical to the id the callable node carries. That is what lets the +// L2 gate assert "no dangling endpoints" — every edge endpoint resolves to a +// real node id. +// +// v1 signatures are globally unique (a method signature embeds its receiver +// type), so a flat signature→id map has no collisions. +type sigIndex map[string]string + +// buildSigIndex walks the symbol table and records, for every callable +// (functions, methods, and nested closures), signature → can:// id. It mirrors +// emitModule/emitType/emitCallable's parent-id computation exactly. +func buildSigIndex(appID string, symbolTable map[string]schema.GoFile) sigIndex { + idx := make(sigIndex) + for relPath, file := range symbolTable { + modID := v2.ModuleID(appID, relPath) + for _, fn := range file.Functions { + idx.addCallable(modID, fn) + } + for _, t := range file.Types { + typeID := v2.TypeID(modID, t.Signature) + for _, m := range t.Methods { + idx.addCallable(typeID, m) + } + } + } + return idx +} + +// addCallable records one callable and recurses into its closures, matching the +// id nesting emitCallable produces. +func (idx sigIndex) addCallable(parentID string, c schema.GoCallable) { + callID := v2.CallableID(parentID, c.Signature) + idx[c.Signature] = callID + for _, inner := range c.InnerCallables { + idx.addCallable(callID, inner) + } +} + +// idFor returns the can:// id for a v1 signature, or ("", false) when the +// signature names a callable outside the project (stdlib/external) — those +// resolve to no in-tree node and must not become an edge endpoint. +func (idx sigIndex) idFor(sig string) (string, bool) { + id, ok := idx[sig] + return id, ok +} From 62990c11ce615654e62d367cd842f4312cb92914 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Mon, 28 Sep 2026 23:13:10 -0400 Subject: [PATCH 10/29] test: L2 v2 fixture gate over the multipackage fixture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lock the v2 L2 output shape by building the fixture through the real resolver (the -a 2 wiring) and asserting the L2 checklist (testing-and-validation.md): - no dangling endpoints: every edge src/dst resolves to a real callable id in the tree - every edge carries a non-empty prov naming the resolver - a NAMED expected edge (Worker.Run -> Worker.execute) by exact can:// id pair, so the gate proves correctness not just non-emptiness - a NAMED cross-package edge (main -> server.New) - callee backfilled to an in-tree can:// id on the goroutine call site - the L1 ⊆ L2 superset: emitting the same model at level 1 vs 2 leaves every module/type/callable id, span, kind and body node identical, changing only callee null -> id (the one sanctioned mutation). --- .../syntactic_analysis/v2emit/gate_l2_test.go | 243 ++++++++++++++++++ 1 file changed, 243 insertions(+) create mode 100644 internal/syntactic_analysis/v2emit/gate_l2_test.go diff --git a/internal/syntactic_analysis/v2emit/gate_l2_test.go b/internal/syntactic_analysis/v2emit/gate_l2_test.go new file mode 100644 index 0000000..8a50883 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/gate_l2_test.go @@ -0,0 +1,243 @@ +package v2emit + +// L2 conformance gate for the v2 emitter. +// +// L2 is a pure refinement over L1: it backfills the callee null->id slot and +// emits the call_graph edge list, both endpoints callable ids. This gate builds +// the multipackage fixture through the REAL resolver (the same wiring the +// analyzer uses at -a 2), emits v2, and asserts every item on the L2 checklist +// in designing-cldk-changes/references/testing-and-validation.md: +// +// - no dangling endpoints — every edge src/dst is a real callable id +// - every edge carries a non-empty prov naming the resolver +// - callee is a backfilled can:// id on resolved sites, null on unresolved +// - a NAMED expected edge is present (exact src,dst pair, not "graph non-empty") +// - at least one cross-package edge is present +// - the L1 ⊆ L2 superset holds: the tree is unchanged except the callee fill. + +import ( + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" +) + +// buildMultipackageL2 builds the fixture AND runs the resolver, so the emitted +// v2 payload carries L2 facts (backfilled callees, call_graph edges). +func buildMultipackageL2(t *testing.T) *v2.Analysis { + t.Helper() + dir := multipackageDir(t) + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) // mutates st in place: backfills callee_signature + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges} + return Emit(app, "multipackage", dir, 2, "gate-test") +} + +// allCallableIDs collects every callable/type id in the emitted tree. +func allCallableIDs(out *v2.Analysis) map[string]bool { + ids := map[string]bool{} + var walk func(c v2.Callable) + walk = func(c v2.Callable) { + ids[c.ID] = true + for _, inner := range c.Callables { + walk(inner) + } + } + for _, mod := range out.Application.SymbolTable { + for _, fn := range mod.Functions { + walk(fn) + } + for _, ty := range mod.Types { + ids[ty.ID] = true + for _, m := range ty.Callables { + walk(m) + } + } + } + return ids +} + +// TestGateL2_MaxLevelAndEdgesPresent locks that L2 emits a non-empty call graph. +func TestGateL2_MaxLevelAndEdgesPresent(t *testing.T) { + out := buildMultipackageL2(t) + if out.MaxLevel != 2 { + t.Errorf("max_level = %d, want 2", out.MaxLevel) + } + if len(out.Application.CallGraph) == 0 { + t.Fatal("L2 should emit call_graph edges; got none") + } +} + +// TestGateL2_NoDanglingEndpoints locks that every edge endpoint resolves to a +// real callable id, and every edge names its resolver in prov. +func TestGateL2_NoDanglingEndpoints(t *testing.T) { + out := buildMultipackageL2(t) + ids := allCallableIDs(out) + + for i, e := range out.Application.CallGraph { + if !ids[e.Src] { + t.Errorf("edge[%d] src is a dangling endpoint: %q", i, e.Src) + } + if !ids[e.Dst] { + t.Errorf("edge[%d] dst is a dangling endpoint: %q", i, e.Dst) + } + if len(e.Prov) == 0 { + t.Errorf("edge[%d] (%s -> %s) has empty prov", i, e.Src, e.Dst) + } + } +} + +// TestGateL2_NamedEdgePresent asserts an EXACT expected edge, so the gate proves +// correctness, not merely a non-empty graph: Worker.Run calls Worker.execute +// (the `go w.execute(...)` goroutine), both as can:// ids. +func TestGateL2_NamedEdgePresent(t *testing.T) { + out := buildMultipackageL2(t) + + const ( + src = "can://go/multipackage/worker/worker.go/example.com/multipackage/worker.Worker/example.com/multipackage/worker.Worker.Run" + dst = "can://go/multipackage/worker/worker.go/example.com/multipackage/worker.Worker/example.com/multipackage/worker.Worker.execute" + ) + found := false + for _, e := range out.Application.CallGraph { + if e.Src == src && e.Dst == dst { + found = true + break + } + } + if !found { + t.Errorf("expected edge %s -> %s not found in call_graph", src, dst) + } +} + +// TestGateL2_CrossPackageEdge locks a specific cross-package edge: main() (in +// package main, main.go) calls server.New (in package server, server/server.go). +// Asserting the exact pair proves a resolved cross-module edge, not just that +// the graph is non-empty. +func TestGateL2_CrossPackageEdge(t *testing.T) { + out := buildMultipackageL2(t) + + const ( + src = "can://go/multipackage/main.go/example.com/multipackage.main" + dst = "can://go/multipackage/server/server.go/example.com/multipackage/server.New" + ) + found := false + for _, e := range out.Application.CallGraph { + if e.Src == src && e.Dst == dst { + found = true + break + } + } + if !found { + t.Errorf("expected cross-package edge %s -> %s not found", src, dst) + } +} + +// TestGateL2_CalleeBackfilled locks the refinement: the goroutine call site in +// Worker.Run has its callee backfilled to Worker.execute's can:// id (not null, +// not the raw v1 signature). +func TestGateL2_CalleeBackfilled(t *testing.T) { + out := buildMultipackageL2(t) + ids := allCallableIDs(out) + + run, ok := out.Application.SymbolTable["worker/worker.go"]. + Types["Worker"].Callables["example.com/multipackage/worker.Worker.Run"] + if !ok { + t.Fatal("Worker.Run missing") + } + if len(run.Body) == 0 { + t.Fatal("Worker.Run should have body call nodes") + } + sawBackfilled := false + for local, node := range run.Body { + if node.Callee == nil { + continue + } + if !ids[*node.Callee] { + t.Errorf("body[%s].callee = %q is not an in-tree callable id", local, *node.Callee) + } + sawBackfilled = true + } + if !sawBackfilled { + t.Error("Worker.Run should have at least one backfilled callee at L2") + } +} + +// TestGateL2_SupersetOfL1 locks L1 ⊆ L2: emitting the SAME model at level 1 vs +// level 2 changes nothing but the callee slot. Every module/type/callable id and +// span is identical; only body callees may go null -> id. +func TestGateL2_SupersetOfL1(t *testing.T) { + dir := multipackageDir(t) + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + // L1 snapshot BEFORE the resolver mutates call sites. + l1 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: nil}, "multipackage", dir, 1, "gate-test") + + // Now run the resolver (mutates st) and emit L2. + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + l2 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges}, "multipackage", dir, 2, "gate-test") + + if len(l1.Application.SymbolTable) != len(l2.Application.SymbolTable) { + t.Fatalf("module count changed L1->L2: %d vs %d", + len(l1.Application.SymbolTable), len(l2.Application.SymbolTable)) + } + for rel, m1 := range l1.Application.SymbolTable { + m2 := l2.Application.SymbolTable[rel] + if m1.ID != m2.ID || m1.Span != m2.Span || m1.Source != m2.Source { + t.Errorf("%s: module identity/span/source changed L1->L2", rel) + } + assertCallablesEqualModuloCallee(t, rel, m1.Functions, m2.Functions) + for name, t1 := range m1.Types { + t2 := m2.Types[name] + if t1.ID != t2.ID || t1.Span != t2.Span || t1.Kind != t2.Kind { + t.Errorf("%s type %s: identity/span/kind changed L1->L2", rel, name) + } + assertCallablesEqualModuloCallee(t, rel+" "+name, t1.Callables, t2.Callables) + } + } +} + +// assertCallablesEqualModuloCallee checks two callable maps are identical except +// that a body node's callee may have gone from null (L1) to a non-null id (L2). +func assertCallablesEqualModuloCallee(t *testing.T, where string, a, b map[string]v2.Callable) { + t.Helper() + if len(a) != len(b) { + t.Errorf("%s: callable count changed L1->L2: %d vs %d", where, len(a), len(b)) + return + } + for sig, c1 := range a { + c2, ok := b[sig] + if !ok { + t.Errorf("%s: callable %q dropped L1->L2", where, sig) + continue + } + if c1.ID != c2.ID || c1.Span != c2.Span || c1.Kind != c2.Kind { + t.Errorf("%s callable %s: identity/span/kind changed L1->L2", where, sig) + } + if len(c1.Body) != len(c2.Body) { + t.Errorf("%s callable %s: body node count changed L1->L2", where, sig) + continue + } + for local, n1 := range c1.Body { + n2 := c2.Body[local] + if n1.Kind != n2.Kind || n1.Span != n2.Span || n1.IsGoroutine != n2.IsGoroutine { + t.Errorf("%s callable %s body[%s]: non-callee field changed L1->L2", where, sig, local) + } + // The one sanctioned mutation: callee null -> id only. + if n1.Callee != nil { + t.Errorf("%s callable %s body[%s]: L1 callee should be null", where, sig, local) + } + } + assertCallablesEqualModuloCallee(t, where+" "+sig, c1.Callables, c2.Callables) + } +} From 67e051eb2d5cc8f297d9ec93f65ffebbfc41c5b5 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Tue, 29 Sep 2026 00:02:08 -0400 Subject: [PATCH 11/29] design: source_file on callable for cross-file methods The real-app eval (dae/openbao/cockroach) surfaced a datamodel collision: v2 nests a method under its receiver type, but Go allows a method to be declared in a different file than its type, so span.bytes computed against the nesting module's source collapse to empty (~3292 cases; cobra clean). Design-loop decision (Go-only for now; cross-language parity deferred): add an optional callable.source_file field, present only when the declaring file differs from the nesting module. span.bytes then index that file's module.source; text = symbol_table[source_file].source[span.bytes] when present, else the nesting module. Containment stays receiver-based. Records the decision in the committed spec and CLAUDE.md; python-sdk source_file support is a tracked follow-up on the SDK rung. --- CLAUDE.md | 17 +++++++- docs/design/specs/v2-l1-emission.md | 66 ++++++++++++++++++++++++++--- 2 files changed, 75 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 1694459..7b3f774 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,8 +12,9 @@ v2 vocabulary (parity clause: add at the leaves, never rename shared names). The committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. ### Identity & positioning (Group A — reused verbatim by L3/L4) -- **`application.id`** = `can://go/`, where `` is a slug derived from the - **go.mod module path** (deterministic, no flag). L3/L4 build ids on this unchanged. +- **`application.id`** = `can://go/`, where `` is the **`--app-name`** value, + defaulting to the input directory's base name (per the CLI contract; the SDK's Neo4j + backend must use the same anchor). L3/L4 build ids on this unchanged. - **Callable signature** = existing `signatureOf()` output as the last `can://` path segment; receiver folded in for methods. One canonicalizer, unchanged from v1. - **`span.bytes`** = **UTF-8 byte offsets** into `module.source` (Go strings are @@ -35,6 +36,18 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. those returns remain in `return_type` too. Maps Go onto the shared field. - **Closures / function literals** → nested `callables{}` on the enclosing callable (replaces v1 `InnerCallables`); each closure gets its own `can://` id. +- **`source_file`** (optional; added in the 2026-09-29 design loop): Go allows a method + to be declared in a *different file* than its receiver type. The method node stays + nested under its receiver type (containment-by-receiver, above), but its `span.bytes` + then index the **declaring file's** `module.source`, not the nesting module's. When + the declaring file ≠ the nesting module, the callable carries `source_file` = that + file's path (a `symbol_table` key). **Text recovery:** a node's text = + `symbol_table[source_file].source[span.bytes]` if `source_file` present, else the + nesting module's `source`. Emitted only when it differs (absent = same file). This is + the leaf-level fix for the cross-file-method empty-span collision the real-app eval + surfaced. Cross-language parity (C# partial classes, Rust `impl`, Ruby reopened + classes, TS declaration merging) is **deferred** — promote `source_file` into the + shared keystone vocabulary when a second language needs it. ### `call` body node - Typed **`is_goroutine`** and **`is_deferred`** boolean fields (`defer` is net-new vs diff --git a/docs/design/specs/v2-l1-emission.md b/docs/design/specs/v2-l1-emission.md index 209bf07..5ff7201 100644 --- a/docs/design/specs/v2-l1-emission.md +++ b/docs/design/specs/v2-l1-emission.md @@ -33,7 +33,7 @@ application:{ id, kind:"application", symbol_table, call_graph, param_in, param_ ### Identity & positioning (Group A) | Decision | Choice | | --- | --- | -| `application.id` | `can://go/`, `` = slug from **go.mod module path** (no flag) | +| `application.id` | `can://go/`, `` = **`--app-name`** (default: input dir base name), per CLI contract | | Callable signature | existing `signatureOf()` as last id segment; receiver folded in for methods | | `span.bytes` | **UTF-8 byte offsets** into `module.source` | @@ -54,6 +54,39 @@ application:{ id, kind:"application", symbol_table, call_graph, param_in, param_ | --- | --- | | `error_channel[]` | from `error`-typed returns (kept in `return_type` too) | | Closures | nested `callables{}` on enclosing callable (replaces `InnerCallables`) | +| `source_file` | **optional**; emitted **only when** the callable's declaring file differs from the module it is nested under (a method whose receiver type lives in another file). Value = declaring file path relative to input root, same shape as a `symbol_table` key | + +#### Cross-file methods — the `source_file` amendment (design loop, 2026-09-29) + +The eval on real Go apps (dae/openbao/cockroach, ~3292 empty spans; cobra clean) +exposed a datamodel collision: v2 nests a method under its **receiver type** +(containment-by-receiver, § `type` above), but Go lets a method be declared in a +*different file* than its type. The keystone and the **Java** reference both fix +`span.bytes` as offsets into the **nesting module's** `source` (Java's SDK slices a +node against the `JCompilationUnit` it is hydrated under). Java can't produce the +conflict — its methods are lexically inside their type's file — so there was no +precedent; this is a genuine Go divergence brought to the user. + +Decision (Go-only for now; parity deferred — see below): +- **Containment unchanged.** The method node stays under its receiver type's + `callables{}`. The tree still reflects containment-by-receiver. +- **New leaf-level optional field `callable.source_file`**, present only when the + declaring file ≠ the nesting module. Parity-safe (an additive leaf field, never a + rename of shared vocabulary). +- **`span.bytes` for a cross-file method index the `source_file` module's `source`**, + not the nesting module's. +- **Text-recovery rule (the SDK contract):** a node's text = + `symbol_table[source_file].source[span.bytes]` when `source_file` is present, else + `.source[span.bytes]`. The declaring file is always another module + already in `symbol_table` (Go analyzes every `.go` file), so its `source` is present + to slice — the rule is fully data-driven, no extra I/O. + +**Deferred — cross-language parity.** C# partial classes, Rust `impl` blocks, Ruby +reopened classes, and TS declaration merging all hit the same +member-declared-apart-from-its-type shape. `source_file` is scoped to Go here; when a +second language needs it, promote the same field + text-recovery rule into the shared +keystone vocabulary in a design pass (the parity clause forbids a divergent second +spelling). Recorded so the promotion is a known follow-up, not a rediscovery. ### `call` (body node, L1) | Decision | Choice | @@ -65,6 +98,21 @@ application:{ id, kind:"application", symbol_table, call_graph, param_in, param_ `call_graph: [{src, dst, prov, weight}]` at application scope (rename from v1 `{source, target, type, provenance}`). +### CLI conformance (cli-contract.md) +The current `cango` CLI is non-conformant. Bring it to the contract as part of this +train (the SDK facade depends on a uniform flag surface across backends): +- **`--emit `** — output target; `json` default. `neo4j`/`schema` + return "not yet implemented" (non-zero) until the Neo4j child lands, never silent + fallback. +- **`--app-name `** — `application.id`/`:Application` anchor; default = input dir + base name. Precedence: explicit flag > env > default. +- **`--neo4j-uri` / `--neo4j-user` / `--neo4j-password` / `--neo4j-database`** — accepted + and validated now; consumed by the Neo4j child. +- **`-j, --jobs `** — worker parallelism (default CPU cores); output byte-identical + across `-j` values. +- Flag validation: unknown/unimplemented value → non-zero exit + clear message. stdout = + data channel only; all diagnostics to stderr. + ## Release plan - **Train 1 — v2.0.0 (coordinated major, lockstep).** codeanalyzer-go emission rewrite @@ -76,15 +124,21 @@ application:{ id, kind:"application", symbol_table, call_graph, param_in, param_ (`code`, `is_*` booleans folded into `kind`, `is_constructor_call`). - **Deferred (Train 2+):** L3 (CFG/CDG/DDG), L4 (SDG). Reuse Group A ids/spans + Group C edge shape unchanged; no new schema-major coordination. +- **`source_file` SDK follow-up (Train 1, SDK rung):** python-sdk's Go model must add + the optional `source_file` field and its `slice()`/`get_method_body` must resolve + text against `symbol_table[source_file].source` when present (else the nesting + module). Deferred to the SDK rung — the analyzer ships the field first; the SDK + catches up on its own clock (analyzer release must precede the SDK rung regardless). ## Decomposition (tracking shape — to be confirmed with user) Recommended: **parent (in `codellm-devkit/.github`) + one sub-issue per PR**, because the work spans two repos on their own clocks and needs a coordination record: -1. codeanalyzer-go: L1 emission (tree, source, ids, body-with-call-nodes) -2. codeanalyzer-go: L2 emission (call_graph rename @ app scope) -3. codeanalyzer-go: Neo4j relabel -4. python-sdk: Go v2 model remap (accessors stable) -5. release + pin (once both cut) +1. codeanalyzer-go: CLI contract conformance (`--emit`, `--app-name`, `--neo4j-*`, `-j`) +2. codeanalyzer-go: L1 emission (tree, source, ids, body-with-call-nodes) +3. codeanalyzer-go: L2 emission (call_graph rename @ app scope) +4. codeanalyzer-go: Neo4j projection (implements the deferred `--emit neo4j`/`schema`) +5. python-sdk: Go v2 model remap (accessors stable) +6. release + pin (once both cut) Children filed just-in-time as picked up; this spec records the full plan. From ef204e0563f1db90499ceccaae6d7102ce50ea37 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Tue, 29 Sep 2026 00:10:56 -0400 Subject: [PATCH 12/29] fix(emit): source_file for methods declared apart from their type A Go method may be declared in a different file than its receiver type. The v2 tree nests it under the receiver type's module, but its line/col positions are relative to its OWN file, so computing span.bytes against the nesting module's source collapsed them to an empty [n,n] slice (~3292 cases across dae/openbao/cockroach; cobra clean). Compute a cross-file method's span against its declaring file (c.Path) via a per-file lineIndex cache, and record that file in the new optional callable.source_file field. Text recovery: symbol_table[source_file].source [span.bytes] when set, else the nesting module. Closures inherit their enclosing callable's file. Containment stays receiver-based. Adds TestGateL1_CrossFileMethodSourceFile over the multipackage fixture (Server declared in server.go, its Describe method in middleware.go): asserts source_file names the declaring file, the span is non-empty, and it slices to the method text; plus the negative case (in-file method carries no source_file). Design: docs/design/specs/v2-l1-emission.md, CLAUDE.md. --- internal/schema/v2/schema.go | 8 ++ internal/syntactic_analysis/v2emit/emit.go | 86 ++++++++++++++++--- .../syntactic_analysis/v2emit/gate_test.go | 59 +++++++++++++ 3 files changed, 141 insertions(+), 12 deletions(-) diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index c3acb27..f4a5dcf 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -140,6 +140,14 @@ type Callable struct { Span Span `json:"span"` // Signature is the human-readable last path segment of ID (one signatureOf()). Signature string `json:"signature"` + // SourceFile names the callable's declaring file (a symbol_table key) when it + // differs from the module this callable is nested under — the Go case where a + // method is declared in a different file than its receiver type. When set, + // Span.Bytes index symbol_table[SourceFile].source, NOT the nesting module's + // source; text = symbol_table[SourceFile].source[Span.Bytes]. Absent (the + // common case) means the declaring file is the nesting module. See CLAUDE.md + // § Schema decisions → callable.source_file and docs/design/specs/v2-l1-emission.md. + SourceFile string `json:"source_file,omitempty"` // Parameters are the ordered formal parameters. Parameters []Parameter `json:"parameters"` // ReturnType is the joined return type, e.g. "(int, error)". diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 1fb7d7e..4e08785 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -25,6 +25,13 @@ func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, a // callees and to map call_graph edge endpoints onto real callable ids. idx := buildSigIndex(appID, app.SymbolTable) + // One lineIndex per source file, read once and shared. A method whose + // receiver type lives in a different file is nested under that type's module + // but must have its span computed against ITS OWN file (CLAUDE.md § Schema + // decisions → callable.source_file); the cache lets emitCallable reach any + // file's index without re-reading it per callable. + srcs := newSourceCache(projectDir) + out := &v2.Analysis{ SchemaVersion: v2.SchemaVersion, Language: v2.Language, @@ -42,7 +49,7 @@ func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, a // sorted keeps any incidental ordering (and logs) stable across runs. for _, relPath := range sortedFileKeys(app.SymbolTable) { file := app.SymbolTable[relPath] - out.Application.SymbolTable[relPath] = emitModule(appID, projectDir, relPath, file, idx) + out.Application.SymbolTable[relPath] = emitModule(appID, relPath, file, idx, srcs) } return out } @@ -50,10 +57,14 @@ func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, a // emitModule builds a v2 module from a v1 GoFile, reading the file's source so // spans can slice off it. A file that cannot be read still emits its structure // (with empty source and zeroed byte spans) rather than dropping the module. -func emitModule(appID, projectDir, relPath string, file schema.GoFile, idx sigIndex) v2.Module { - source := readSource(projectDir, relPath) - li := newLineIndex(source) - modID := v2.ModuleID(appID, relPath) +// +// modRel is this module's own file path (a symbol_table key). It is threaded to +// callables so a method declared in another file can be detected and given a +// span against its own source plus a source_file pointer. +func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *sourceCache) v2.Module { + li := srcs.index(modRel) + source := li.source + modID := v2.ModuleID(appID, modRel) mod := v2.Module{ ID: modID, @@ -70,18 +81,20 @@ func emitModule(appID, projectDir, relPath string, file schema.GoFile, idx sigIn } for name, t := range file.Types { - mod.Types[name] = emitType(modID, li, t, idx) + mod.Types[name] = emitType(modID, modRel, t, idx, srcs) } for sig, fn := range file.Functions { - mod.Functions[sig] = emitCallable(modID, li, fn, idx) + mod.Functions[sig] = emitCallable(modID, modRel, li, fn, idx, srcs) } return mod } // emitType maps a v1 GoType to a v2 type node, collapsing is_interface into the // kind and splitting embedding (base_types) from computed satisfaction. Methods -// resolve into the type's callables{}. -func emitType(modID string, li *lineIndex, t schema.GoType, idx sigIndex) v2.Type { +// resolve into the type's callables{}. modRel is the module the type lives in; +// its methods may be declared in other files (handled in emitCallable). +func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceCache) v2.Type { + li := srcs.index(modRel) typeID := v2.TypeID(modID, t.Signature) out := v2.Type{ ID: typeID, @@ -102,20 +115,37 @@ func emitType(modID string, li *lineIndex, t schema.GoType, idx sigIndex) v2.Typ } } for sig, m := range t.Methods { - out.Callables[sig] = emitCallable(typeID, li, m, idx) + out.Callables[sig] = emitCallable(typeID, modRel, li, m, idx, srcs) } return out } // emitCallable maps a v1 GoCallable to a v2 callable node, including its body // (call nodes) and nested closures. -func emitCallable(parentID string, li *lineIndex, c schema.GoCallable, idx sigIndex) v2.Callable { +// +// modRel and modLI are the nesting module's path and lineIndex. When the +// callable's declaring file (c.Path) differs — a method whose receiver type is +// in another file — its span is computed against its OWN file's lineIndex and +// source_file records that file, per CLAUDE.md § Schema decisions. Nested +// closures share their enclosing callable's file (Go has no cross-file +// closures), so they carry the resolved file down unchanged. +func emitCallable(parentID, modRel string, modLI *lineIndex, c schema.GoCallable, idx sigIndex, srcs *sourceCache) v2.Callable { callID := v2.CallableID(parentID, c.Signature) + + // Resolve the file this callable's positions are relative to. + li := modLI + sourceFile := "" + if c.Path != "" && c.Path != modRel { + li = srcs.index(c.Path) + sourceFile = c.Path + } + out := v2.Callable{ ID: callID, Kind: callableKind(c), Span: lineSpan(li, c.StartLine, c.EndLine), Signature: c.Signature, + SourceFile: sourceFile, Parameters: emitParams(li, c.Parameters), ReturnType: c.ReturnType, Body: emitBody(callID, li, c.CallSites, idx), @@ -128,8 +158,14 @@ func emitCallable(parentID string, li *lineIndex, c schema.GoCallable, idx sigIn } if len(c.InnerCallables) > 0 { out.Callables = make(map[string]v2.Callable, len(c.InnerCallables)) + // Closures live in the same file as their enclosing callable, so the + // resolved file (modRel-or-c.Path) becomes their nesting file. + enclRel := modRel + if sourceFile != "" { + enclRel = sourceFile + } for sig, inner := range c.InnerCallables { - out.Callables[sig] = emitCallable(callID, li, inner, idx) + out.Callables[sig] = emitCallable(callID, enclRel, li, inner, idx, srcs) } } return out @@ -296,6 +332,32 @@ func readSource(projectDir, relPath string) string { return string(data) } +// sourceCache reads each source file once and memoizes its lineIndex, keyed by +// path relative to projectDir. The emitter needs a file's index in two places — +// when emitting the module itself, and when a method declared in that file is +// nested under a type in a DIFFERENT module — so the cache avoids re-reading a +// file per cross-file method. A file that cannot be read caches an empty index +// (so spans degrade to zero rather than re-attempting the read). +type sourceCache struct { + projectDir string + byPath map[string]*lineIndex +} + +func newSourceCache(projectDir string) *sourceCache { + return &sourceCache{projectDir: projectDir, byPath: make(map[string]*lineIndex)} +} + +// index returns the lineIndex for relPath, reading and indexing the file on +// first use. The returned index carries the file's source (li.source). +func (c *sourceCache) index(relPath string) *lineIndex { + if li, ok := c.byPath[relPath]; ok { + return li + } + li := newLineIndex(readSource(c.projectDir, relPath)) + c.byPath[relPath] = li + return li +} + func sortedFileKeys(m map[string]schema.GoFile) []string { keys := make([]string, 0, len(m)) for k := range m { diff --git a/internal/syntactic_analysis/v2emit/gate_test.go b/internal/syntactic_analysis/v2emit/gate_test.go index db1317a..b021a9d 100644 --- a/internal/syntactic_analysis/v2emit/gate_test.go +++ b/internal/syntactic_analysis/v2emit/gate_test.go @@ -227,6 +227,65 @@ func TestGateL1_CrossPackageStructure(t *testing.T) { } } +// TestGateL1_CrossFileMethodSourceFile locks the source_file amendment +// (CLAUDE.md § Schema decisions → callable.source_file): a method declared in a +// different file than its receiver type stays nested under the type, but its +// span.bytes index its OWN file's source and it carries source_file naming that +// file. In the fixture, Server is declared in server/server.go while its method +// (s Server) Describe() is declared in server/middleware.go. +// +// The text-recovery rule is the contract: a node's text = +// symbol_table[source_file].source[span.bytes] when source_file is present. This +// test slices the declaring module's source by the span and asserts it yields +// the method text — the check that regressed to "" before the fix (empty span +// against the wrong file's source). +func TestGateL1_CrossFileMethodSourceFile(t *testing.T) { + out := buildMultipackageV2(t) + + // Server's type node lives in server/server.go. + server := mustModule(t, out, "server/server.go") + stype, ok := server.Types["Server"] + if !ok { + t.Fatalf("Server type missing; types=%v", keys(server.Types)) + } + const describeID = "example.com/multipackage/server.Server.Describe" + m, ok := stype.Callables[describeID] + if !ok { + t.Fatalf("Server.Describe missing; callables=%v", keys(stype.Callables)) + } + + // It is declared in a different file, so source_file must name that file. + const declFile = "server/middleware.go" + if m.SourceFile != declFile { + t.Errorf("Describe.source_file = %q, want %q (declared apart from its type)", m.SourceFile, declFile) + } + + // The span must be non-empty (the pre-fix bug collapsed it to [n,n]). + if m.Span.Bytes[0] >= m.Span.Bytes[1] { + t.Errorf("Describe span is empty %v — cross-file method span not computed against its own file", m.Span.Bytes) + } + + // Text-recovery rule: slice the DECLARING module's source by the span. + decl := mustModule(t, out, declFile) + text := m.Span.Slice(decl.Source) + if !strings.Contains(text, "func (s Server) Describe()") { + t.Errorf("Describe span over %s does not slice to the method text; got %q", declFile, text) + } + + // Negative: an in-file method (Server.Addr, declared in server.go with its + // type) must NOT carry source_file — absence encodes "same file". + addr, ok := stype.Callables["example.com/multipackage/server.Server.Addr"] + if !ok { + t.Fatalf("Server.Addr missing; callables=%v", keys(stype.Callables)) + } + if addr.SourceFile != "" { + t.Errorf("Addr.source_file = %q, want empty (declared in its type's own file)", addr.SourceFile) + } + if got := addr.Span.Slice(server.Source); !strings.Contains(got, "func (s *Server) Addr()") { + t.Errorf("Addr span over its own module does not slice to the method text; got %q", got) + } +} + // TestGateL1_Envelope locks the manifest envelope at L1. func TestGateL1_Envelope(t *testing.T) { out := buildMultipackageV2(t) From 06d1cbb85b8581d1661dfccfdeac9ec0fa2b5ee2 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Tue, 29 Sep 2026 00:55:26 -0400 Subject: [PATCH 13/29] fix(symtab): skip decls whose file is outside the input root cgo synthesizes wrapper functions (_Cfunc_*, _Cgo_*, _cgo_runtime_*) into a generated file under $GOCACHE; go/packages resolves their token.Pos to that cache path. Emitting them produced callables whose source_file pointed at the build cache and whose span could not be sliced against any project module (ISSUE-2 in the real-app eval: 85 such callables on cockroach, 106 validation failures at L1 and L2; cobra/dae/openbao have no cgo and were unaffected). These are toolchain artifacts, not project source. buildCallable now returns nil when the declaration's file is not within projectDir, at the point its (cache) file is first known; both callers already skip nil. Adds utils.IsWithin and a unit test. Verified: cockroach L1 106->0 failures, 85 dropped signatures all cgo glue, real pkg/geo/geos project code preserved. --- internal/syntactic_analysis/symbol_table.go | 12 ++++++++++ internal/utils/fs.go | 18 ++++++++++++++ internal/utils/fs_test.go | 26 +++++++++++++++++++++ 3 files changed, 56 insertions(+) diff --git a/internal/syntactic_analysis/symbol_table.go b/internal/syntactic_analysis/symbol_table.go index c7f48ce..93d32ab 100644 --- a/internal/syntactic_analysis/symbol_table.go +++ b/internal/syntactic_analysis/symbol_table.go @@ -492,6 +492,18 @@ func (b *SymbolTableBuilder) buildCallable( decl *ast.FuncDecl, ) *schema.GoCallable { name := decl.Name.Name + + // Skip declarations whose file is outside the input root. cgo synthesizes + // wrapper functions (_Cfunc_*, _Cgo_*, _cgo_runtime_*) into a generated file + // under $GOCACHE; go/packages resolves their token.Pos to that cache path. + // They are toolchain artifacts, not project source — emitting them yields + // callables whose span cannot be sliced against any project module. Drop + // them here, at the point their (cache) file is first known. + if declFile := b.fset.File(decl.Pos()); declFile == nil || + !utils.IsWithin(b.projectDir, declFile.Name()) { + return nil + } + isExported := unicode.IsUpper(rune(name[0])) pos := b.fset.Position(decl.Pos()) end := b.fset.Position(decl.End()) diff --git a/internal/utils/fs.go b/internal/utils/fs.go index e46da2e..541127c 100644 --- a/internal/utils/fs.go +++ b/internal/utils/fs.go @@ -35,6 +35,24 @@ func RelativePath(root, path string) string { return rel } +// IsWithin reports whether path lies inside root (or is root itself). It is the +// test for "this declaration's file is part of the analyzed project": a path +// whose relative form escapes root (starts with ".." or is absolute) is outside. +// cgo synthesizes wrapper functions (_Cfunc_*, _Cgo_*) into files under +// $GOCACHE, so their declaring file resolves outside the input root — IsWithin +// is how the symbol-table builder tells those toolchain artifacts from project +// source. +func IsWithin(root, path string) bool { + rel, err := filepath.Rel(root, path) + if err != nil { + return false + } + if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + return false + } + return !filepath.IsAbs(rel) +} + // FileHash returns the SHA-256 hex digest of the file at path. func FileHash(path string) (string, error) { f, err := os.Open(path) diff --git a/internal/utils/fs_test.go b/internal/utils/fs_test.go index 91df3d4..4e12166 100644 --- a/internal/utils/fs_test.go +++ b/internal/utils/fs_test.go @@ -53,6 +53,32 @@ func TestIsVendored(t *testing.T) { } } +// ── IsWithin ───────────────────────────────────────────────────────────────── + +func TestIsWithin(t *testing.T) { + root := filepath.Join(string(filepath.Separator), "home", "user", "proj") + tests := []struct { + path string + want bool + }{ + // Inside the project root. + {filepath.Join(root, "main.go"), true}, + {filepath.Join(root, "pkg", "geo", "geos", "geos.go"), true}, + {root, true}, // root itself resolves to "." + // Outside: the go-build cache cgo synthesizes into — the ISSUE-2 shape. + {filepath.Join(string(filepath.Separator), "home", "user", "Library", "Caches", "go-build", "bb", "hash-d"), false}, + // A sibling directory sharing a prefix but not nested under root. + {filepath.Join(string(filepath.Separator), "home", "user", "proj-other", "x.go"), false}, + // A parent directory. + {filepath.Join(string(filepath.Separator), "home", "user"), false}, + } + for _, tc := range tests { + if got := utils.IsWithin(root, tc.path); got != tc.want { + t.Errorf("IsWithin(%q, %q) = %v, want %v", root, tc.path, got, tc.want) + } + } +} + // ── FileHash ────────────────────────────────────────────────────────────────── func TestFileHash_Deterministic(t *testing.T) { From adb07ff5a7078fec88a2436099027720690272b3 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Wed, 30 Sep 2026 13:21:44 -0400 Subject: [PATCH 14/29] docs: v2 analysis.json structure reference (keys only) A quick key-skeleton map of the canonical v2 output: the node tree, the recursive callable node, span shape, presence rules (always-present vs omitempty vs emitted-when-true), and can:// id shapes. Authored from the JSON tags in internal/schema/v2/schema.go; links to v2-l1-emission.md for the decisions. --- docs/design/specs/v2-schema-structure.md | 108 +++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 docs/design/specs/v2-schema-structure.md diff --git a/docs/design/specs/v2-schema-structure.md b/docs/design/specs/v2-schema-structure.md new file mode 100644 index 0000000..f833b43 --- /dev/null +++ b/docs/design/specs/v2-schema-structure.md @@ -0,0 +1,108 @@ +# v2 `analysis.json` — structure reference (keys only) + +The key skeleton of the canonical CLDK v2 output (`--analysis-schema 2`). This is a +quick map of the node tree and every field; the *decisions* behind the shape live in +[`v2-l1-emission.md`](./v2-l1-emission.md) and `CLAUDE.md` § Schema decisions, and the +authoritative Go definitions are `internal/schema/v2/schema.go`. Field names here are +the JSON tags from that file. + +## Tree + +``` +├─ schema_version # "2.0.0" +├─ language # "go" +├─ max_level # 1 (symbol table) | 2 (+ call graph) +├─ analyzer +│ ├─ name # "codeanalyzer-go" +│ └─ version +└─ application + ├─ id # "can://go/" + ├─ kind # "application" + ├─ symbol_table # { "": module } + │ └─ + │ ├─ id + │ ├─ kind # "module" + │ ├─ span { start, end, bytes } + │ ├─ package + │ ├─ source # whole file text, once per module + │ ├─ content_hash # optional + │ ├─ imports[] { name, path, alias?, span } + │ ├─ types # { "": type } + │ │ └─ + │ │ ├─ id + │ │ ├─ kind # struct | interface | alias | defined + │ │ ├─ span { start, end, bytes } + │ │ ├─ base_types[] # optional: embedded type ids (explicit) + │ │ ├─ interfaces[] # optional: interfaces computed-satisfied + │ │ ├─ fields # optional: { "": field } + │ │ │ └─ { id, kind, type, span } + │ │ └─ callables # { "": callable } + │ │ └─ # ← callable node, defined below + │ └─ functions # { "": callable } + │ └─ # ← callable node, defined below + └─ call_graph[] # L2+; empty [] at L1 + └─ + ├─ src # can:// callable id + ├─ dst # can:// callable id + ├─ prov[] # optional: e.g. ["go/types"] + └─ weight # optional +``` + +## `` node (recursive) + +The same shape whether it sits under a type's `callables` (methods), a module's +`functions` (package-level functions), or another callable's `callables` (closures). + +``` + +├─ id +├─ kind # function | method | lambda +├─ signature +├─ source_file # optional: set only for a method declared in a +│ # different file than its receiver type; its +│ # span.bytes then index symbol_table[source_file].source +├─ span { start, end, bytes } +├─ parameters[] { name, type, span, is_variadic? } +├─ return_type # optional +├─ error_channel[] # optional: error-typed returns +├─ metrics # optional: { cyclomatic: N } +├─ callables # optional: nested closures (same shape) +└─ body # { "": call node } + └─ + ├─ kind # "call" + ├─ span { start, end, bytes } + ├─ callee # null | can:// id (key always present) + ├─ is_goroutine # optional: present only when true + └─ is_deferred # optional: present only when true +``` + +## `span` + +``` +span +├─ start [line, col] +├─ end [line, col] +└─ bytes [from, to] # UTF-8 byte offsets into the owning module.source +``` + +## Presence rules + +- **Always present:** every `id`, `kind`, `span`; the envelope fields; `symbol_table`, + `types`, `callables`, `functions`, `body` maps (may be empty `{}`); `call_graph` (empty + `[]` at L1); a body node's `callee` key (value `null` when unresolved/external). +- **`omitempty` — emitted only when non-empty:** `content_hash`, `imports[].alias`, + `base_types`, `interfaces`, `fields`, `source_file`, `return_type`, `error_channel`, + `metrics`, a callable's nested `callables`, `parameters[].is_variadic`, edge `prov` / + `weight`. +- **Emitted only when `true`:** `is_goroutine`, `is_deferred`. + +## `id` shape + +``` +application can://go/ +module can://go// +type can://go/// +callable can://go//// + (functions omit the segment) +body call keyed by ":" within the callable's body map +``` From af3510adbce0956c6dffb58ff9716476deded498 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Wed, 30 Sep 2026 13:26:15 -0400 Subject: [PATCH 15/29] docs(readme): document v2 schema and analysis.json generation - Add a step-by-step 'Generating analysis.json for a Go app' section (--analysis-schema 2 -a 2 -o , --app-name, verify, cgo note). - Document the canonical v2 output shape alongside the legacy v1 shape, with a full JSON example verified against real emitter output. - Fix the stale CLI options block (add --analysis-schema, --app-name). --- README.md | 161 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 159 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6cf4285..da1235a 100644 --- a/README.md +++ b/README.md @@ -131,6 +131,8 @@ Aliases: Flags: -a, --analysis-level int Analysis level: 1=symbol table only, 2=+resolver call graph (default 1) + --analysis-schema int Output schema major: 1=legacy v1 shape (default), 2=canonical v2 tree (default 1) + --app-name string Application anchor name for can:// ids and Neo4j :Application (default: input dir name) -c, --cache-dir string Cache directory (default: ~/.cldk/go-cache) --codeql Enable CodeQL framework-based call graph (level 2, stub) --eager Force clean rebuild (ignore cache) @@ -178,6 +180,58 @@ cango -i ./my-go-project --eager cango -i ./my-go-project -a 2 -vv ``` +## Generating `analysis.json` for a Go app + +The end-to-end steps to produce an `analysis.json` file for any Go project. + +**1. Make sure you have `cango`.** Either install a released binary (see +[Installation](#installation)) or build from source: + +```bash +git clone https://github.com/codellm-devkit/codeanalyzer-go +cd codeanalyzer-go +go build -o cango ./cmd/codeanalyzer +``` + +**2. Point it at the project root** (the directory containing the app's `go.mod`). +`cango` resolves imports with the Go type checker, so the target must be a normal Go +module. Write the output to a directory with `-o`; `cango` names the file +`analysis.json` inside it. + +```bash +# Canonical v2 schema, Level 2 (symbol table + call graph): +cango -i /path/to/go/app --analysis-schema 2 -a 2 -o /path/to/output/ +# → writes /path/to/output/analysis.json +``` + +> **Pick the schema explicitly.** `--analysis-schema` defaults to `1` (the legacy v1 +> shape). Pass `--analysis-schema 2` for the current **canonical v2** tree (the shape +> the Python SDK's v2 loader and the sections below describe). Omit `-o` to stream the +> JSON to stdout instead of writing a file. + +**3. (Optional) name the application anchor.** `--app-name` sets the `` in every +`can://go//…` id and defaults to the input directory's base name. Set it when the +directory name isn't the identity you want: + +```bash +cango -i /path/to/go/app --analysis-schema 2 -a 2 --app-name myapp -o ./out/ +``` + +**4. Verify.** A successful run exits `0` and produces a JSON document whose envelope +carries `"schema_version": "2.0.0"`, `"language": "go"`, `"max_level": 2`, and an +`application` tree under `application.symbol_table`: + +```bash +cango -i /path/to/go/app --analysis-schema 2 -a 2 -o ./out/ && echo "exit=$?" +python3 -c "import json; d=json.load(open('out/analysis.json')); print(d['schema_version'], d['max_level'], len(d['application']['symbol_table']), 'modules')" +``` + +**Notes for larger apps.** Analysis parallelism defaults to your CPU count (tune with +`-j`); a warm cache (`--cache-dir`, default `~/.cldk/go-cache`) makes re-runs fast, and +`--eager` forces a clean rebuild. Projects that use **cgo** (`import "C"`) are supported: +the toolchain-synthesized wrapper functions are correctly excluded from the output (they +are build artifacts, not project source), so only your own declarations are emitted. + ## Analysis levels | Level | Flag | What runs | Status | @@ -192,7 +246,110 @@ cango -i ./my-go-project -a 2 -vv ## Output schema -The root object is `GoApplication`: +`cango` emits one of two schema shapes, selected by `--analysis-schema`: + +- **`--analysis-schema 2`** — the **canonical CLDK v2** shape (detailed below); the shape the + v2 Python SDK loader consumes and the [step-by-step section + above](#generating-analysisjson-for-a-go-app) produces. +- **`--analysis-schema 1`** (default) — the **legacy v1** `GoApplication` shape, kept for + backward compatibility. + +### Canonical v2 shape (`--analysis-schema 2`) + +The document is a manifest envelope wrapping one `application` containment tree. Every +node carries a `can://go//…` `id`, a `kind`, and a `span`: + +```json +{ + "schema_version": "2.0.0", + "language": "go", + "max_level": 2, + "analyzer": { "name": "codeanalyzer-go", "version": "0.1.0" }, + "application": { + "id": "can://go/myapp", + "kind": "application", + "symbol_table": { + "pkg/greeter/greeter.go": { + "id": "can://go/myapp/pkg/greeter/greeter.go", + "kind": "module", + "package": "greeter", + "span": { "start": [1, 1], "end": [40, 2], "bytes": [0, 812] }, + "source": "package greeter\n\n...", + "content_hash": "…", + "imports": [ { "name": "fmt", "path": "fmt", "span": {…} } ], + "types": { + "Greeter": { + "id": "can://go/myapp/pkg/greeter/greeter.go/example.com/pkg/greeter.Greeter", + "kind": "struct", + "span": { "start": [5, 1], "end": [7, 2], "bytes": [17, 63] }, + "fields": { + "Prefix": { "id": "…/Prefix", "kind": "field", "type": "string", "span": {…} } + }, + "callables": { + "example.com/pkg/greeter.Greeter.Greet": { + "id": "can://go/myapp/pkg/greeter/greeter.go/example.com/pkg/greeter.Greeter/example.com/pkg/greeter.Greeter.Greet", + "kind": "method", + "signature": "example.com/pkg/greeter.Greeter.Greet", + "span": { "start": [9, 1], "end": [11, 2], "bytes": [65, 140] }, + "parameters": [ { "name": "name", "type": "string", "span": {…} } ], + "return_type": "string", + "error_channel": ["error"], + "metrics": { "cyclomatic": 1 }, + "body": { + "10:2": { + "kind": "call", + "span": { "start": [10, 2], "end": [10, 30], "bytes": [90, 118] }, + "callee": "can://go/myapp/pkg/greeter/greeter.go/…/example.com/pkg/greeter.format" + } + } + } + } + } + }, + "functions": { + "example.com/main.main": { "id": "…", "kind": "function", "signature": "…", "body": {…} } + } + } + }, + "call_graph": [ + { + "src": "can://go/myapp/main.go/…/example.com/main.main", + "dst": "can://go/myapp/pkg/greeter/greeter.go/…/example.com/pkg/greeter.Greeter.Greet", + "prov": ["go/types"], + "weight": 1 + } + ] + } +} +``` + +Key v2 schema properties: +- **Envelope** — `schema_version` (`"2.0.0"`), `language` (`"go"`), `max_level` (1 or 2), + and `analyzer{name,version}` wrap a single `application` node. +- **One containment tree** — `application → module → type → callable → body`, keyed as + `symbol_table` (modules, by **project-relative file path**), `types`, `callables`, `body`. +- **`id`** — every node carries a durable `can://go////` id; + `` is the `--app-name` anchor. Body call nodes use `` keyed by `line:col`. +- **`kind`** — `module` · `type` kinds `struct | interface | alias | defined` (collapses v1's + `is_interface`) · `callable` kinds `function | method | lambda` · body `call`. +- **`base_types: []`** (type, optional) — embedded type ids (struct/interface embedding — the + explicit spine), as opposed to interfaces the type is *computed* to satisfy. +- **`span`** — `{start:[line,col], end:[line,col], bytes:[from,to]}`; `bytes` are **UTF-8 byte + offsets** into the owning `module.source` (source is stored once per module, every node slices it). +- **`source_file`** (callable, optional) — set when a method is declared in a *different file* + than its receiver type; the method stays nested under the type, and its `span.bytes` index + `symbol_table[source_file].source` instead of the nesting module's. +- **`error_channel: []`** — populated from `error`-typed returns (the returns also stay in + `return_type`). **Closures** nest as `callables{}` on the enclosing callable. +- **`body` call nodes** — `callee` is the sanctioned `null → id` slot: `null` at L1 (and for + external/stdlib callees), a `can://` node id once resolved at L2. `is_goroutine` / `is_deferred` + are boolean flags emitted **only when true** (a plain call omits them), for `go f()` / `defer f()`. +- **`call_graph` edges** — `{src, dst, prov, weight}`; `src`/`dst` are `can://` node ids that + exist in the tree (not raw signatures), `prov` is resolver provenance, e.g. `["go/types"]`. + +### Legacy v1 shape (`--analysis-schema 1`, default) + +The v1 root object is `GoApplication`: ```json { @@ -226,7 +383,7 @@ The root object is `GoApplication`: } ``` -Key schema properties: +Key v1 schema properties: - `symbol_table` — keyed by **file path relative to the project root** (never absolute) - `classes` — JSON key for types (spine compatibility with Java/Python schemas); value is `GoType` - `module_name` — JSON key for the Go package name (spine compatibility) From 0289c1c9c96f983caf227fe9f8a70cbfcea6ebff Mon Sep 17 00:00:00 2001 From: lambertpw Date: Thu, 1 Oct 2026 15:46:31 -0400 Subject: [PATCH 16/29] fix(emit): sort call_graph edges for deterministic output The v2 call_graph edge list was emitted in Go-map iteration order, so two runs of the same input (even at the same -j) produced byte-differing analysis.json, failing the release determinism gate. Sort the emitted edges by (src, dst, prov, weight) in emitCallGraph, matching the sorted-key discipline already used for module output. Add a determinism gate test that emits a reversed edge slice and asserts identical serialized call_graph. --- .../v2emit/determinism_test.go | 76 +++++++++++++++++++ internal/syntactic_analysis/v2emit/emit.go | 23 ++++++ 2 files changed, 99 insertions(+) create mode 100644 internal/syntactic_analysis/v2emit/determinism_test.go diff --git a/internal/syntactic_analysis/v2emit/determinism_test.go b/internal/syntactic_analysis/v2emit/determinism_test.go new file mode 100644 index 0000000..a7c758a --- /dev/null +++ b/internal/syntactic_analysis/v2emit/determinism_test.go @@ -0,0 +1,76 @@ +package v2emit + +// Determinism gate for the v2 emitter. +// +// release-gates.md requires v2 output to be byte-identical across runs and +// across -j values. The symbol-table tree is already order-stable (modules +// iterated via sortedFileKeys), but the call_graph edge list is assembled by +// ranging a Go map in CallGraphBuilder.Build — random iteration order — and +// emitCallGraph preserved that order. Two runs of the same input therefore +// produced call_graph arrays in different orders. +// +// This gate emits the SAME resolved model twice and asserts the serialized +// call_graph is identical both times, so a regression to unsorted edge output +// fails here rather than only in an external CLI diff. + +import ( + "encoding/json" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" +) + +// TestDeterminism_CallGraphStableAcrossEmits locks that emitting the same model +// twice yields a byte-identical call_graph. emitCallGraph must impose a total +// order on the edges rather than inherit map-iteration order. +func TestDeterminism_CallGraphStableAcrossEmits(t *testing.T) { + dir := multipackageDir(t) + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges} + + // Emit twice from the SAME model. If the edge order is derived from the + // edges slice without a stable sort, repeated marshals still match (same + // slice), so to actually exercise order-independence we marshal a shuffled + // copy of the edge slice too. + first := mustMarshalCallGraph(t, Emit(app, "multipackage", dir, 2, "det-test")) + + // Reverse the input edge order and re-emit: a correct emitter sorts, so the + // serialized call_graph must be unchanged. + rev := make([]schema.GoCallEdge, len(edges)) + for i, e := range edges { + rev[len(edges)-1-i] = e + } + appRev := &schema.GoApplication{SymbolTable: st, CallGraph: rev} + second := mustMarshalCallGraph(t, Emit(appRev, "multipackage", dir, 2, "det-test")) + + if first != second { + t.Errorf("call_graph not order-independent: emitCallGraph must sort edges to a total order\nfirst=%s\nsecond=%s", first, second) + } +} + +func mustMarshalCallGraph(t *testing.T, out *v2.Analysis) string { + t.Helper() + type cgOnly struct { + Application struct { + CallGraph json.RawMessage `json:"call_graph"` + } `json:"application"` + } + data, err := json.Marshal(out) + if err != nil { + t.Fatalf("marshal failed: %v", err) + } + var c cgOnly + if err := json.Unmarshal(data, &c); err != nil { + t.Fatalf("unmarshal failed: %v", err) + } + return string(c.Application.CallGraph) +} diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 4e08785..5b61f9b 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -4,6 +4,7 @@ import ( "os" "path/filepath" "sort" + "strings" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" @@ -256,6 +257,28 @@ func emitCallGraph(edges []schema.GoCallEdge, idx sigIndex) []v2.Edge { Weight: e.Weight, }) } + + // Impose a total order on the edges so the output is byte-identical across + // runs and across -j values (release-gates.md determinism gate). The upstream + // edges slice is assembled by ranging a Go map (random iteration order), so + // without this sort two runs of the same input emit the call_graph in + // different orders. Sort on the full tuple — (src, dst) is the identity but + // prov/weight tie-break so merged edges from multiple backends also order + // stably. + sort.Slice(out, func(i, j int) bool { + a, b := out[i], out[j] + if a.Src != b.Src { + return a.Src < b.Src + } + if a.Dst != b.Dst { + return a.Dst < b.Dst + } + ap, bp := strings.Join(a.Prov, ","), strings.Join(b.Prov, ",") + if ap != bp { + return ap < bp + } + return a.Weight < b.Weight + }) return out } From 58a628eba03fe20fee1bc2db65597b65d085451c Mon Sep 17 00:00:00 2001 From: lambertpw Date: Sat, 3 Oct 2026 11:57:47 -0400 Subject: [PATCH 17/29] docs: tracking records for the v2 migration epic and L1/L2 children Epic codellm-devkit/.github#94 and its two codeanalyzer-go children (#8 L1 tree emission, #9 L2 call_graph refinement), plus the PR body. Corrects child-1's app-id anchor to --app-name (matches the code and CLAUDE.md) and aligns issue headings to the live feature_request form. Co-Authored-By: Claude Opus 4.8 --- docs/design/issues/child-1-l1-emission.md | 8 +-- docs/design/issues/child-2-l2-refinement.md | 39 ++++++++++++ docs/design/issues/epic-v2-migration.md | 67 +++++++++++++++++++++ docs/design/issues/parent-v2-migration.md | 6 +- docs/design/issues/pr-body-v2-l1-l2.md | 52 ++++++++++++++++ 5 files changed, 165 insertions(+), 7 deletions(-) create mode 100644 docs/design/issues/child-2-l2-refinement.md create mode 100644 docs/design/issues/epic-v2-migration.md create mode 100644 docs/design/issues/pr-body-v2-l1-l2.md diff --git a/docs/design/issues/child-1-l1-emission.md b/docs/design/issues/child-1-l1-emission.md index 7cb806d..0a33e82 100644 --- a/docs/design/issues/child-1-l1-emission.md +++ b/docs/design/issues/child-1-l1-emission.md @@ -16,9 +16,9 @@ decision the later levels build on. Do L1 first and get the symbol-table gate gr before touching the L2 call graph. The parser/resolver stay; only the emission layer changes (golden rule: replace what serializes, keep what computes). -**Describe the solution you'd like.** +**Describe the solution you'd like** -- [ ] `application.id` = `can://go/`, `` derived from the go.mod module path +- [ ] `application.id` = `can://go/`, `` = `--app-name` (default: input dir base name) - [ ] Durable `can://` ids on module/type/callable; `signatureOf()` is the last segment - [ ] `span: {start:[l,c], end:[l,c], bytes:[from,to]}` with UTF-8 byte offsets; drop flat `start_line`/`end_line` - [ ] `source` emitted once per module; drop per-callable `code` @@ -26,12 +26,12 @@ changes (golden rule: replace what serializes, keep what computes). - [ ] `base_types[]` = embedded ids, `interfaces[]` = computed structural satisfaction - [ ] `body{}` holds `call` nodes with `callee:null`, typed `is_goroutine`/`is_deferred`; closures nest in `callables{}` -**Describe alternatives you've considered.** +**Describe alternatives you've considered** Does NOT emit the `call_graph` (that is the L2 child), nor any L3/L4 `body` statements or edges. `body{}` at L1 holds only `call` nodes. -**Additional context.** +**Additional context** - Emitter approach: walk the existing in-memory model, add a v2 emitter behind a flag; keep v1 emitter as the compat shim during transition. - One genuinely new datum: thread parser byte offsets into `span.bytes`. diff --git a/docs/design/issues/child-2-l2-refinement.md b/docs/design/issues/child-2-l2-refinement.md new file mode 100644 index 0000000..51c0ee5 --- /dev/null +++ b/docs/design/issues/child-2-l2-refinement.md @@ -0,0 +1,39 @@ + + +**Is your feature request related to a problem? Please describe.** + +L1 emits the v2 containment tree with `call` body nodes whose `callee` is `null` — the +sanctioned refinement slot. L2 fills it: resolve each call site and each call-graph edge +endpoint from a v1 `signatureOf()` string to the durable `can://` id of the callable it +names. L2 is pure refinement over the L1 tree — it computes no new facts, it only +translates identity. The parser/resolver stay; only the emission layer gains the +signature→id index (golden rule: replace what serializes, keep what computes). + +**Describe the solution you'd like** + +- [ ] `buildSigIndex` maps every v1 callable signature (functions, methods, nested closures) → its `can://` id, using the SAME id builders `emitCallable` uses (byte-identical ids) +- [ ] `callee` on each `call` body node backfilled from `null` to the resolved `can://` id +- [ ] `call_graph` emitted at application scope as `[{src, dst, prov, weight}]` +- [ ] External/stdlib callees (outside the project) resolve to no endpoint — never a dangling edge +- [ ] L2 gate green: every edge endpoint and every non-null `callee` resolves to a real in-tree node id +- [ ] Determinism: `call_graph` edge order stable across runs (sorted before emit) + +**Describe alternatives you've considered** + +Does NOT compute any new call-graph facts — it reuses the v1 call graph and only +translates endpoints to `can://` ids. Does NOT emit L3/L4 `body` statements or edges. + +**Additional context** + +- v1 signatures are globally unique (a method signature embeds its receiver), so the flat signature→id map has no collisions. +- The index mirrors `emitModule`/`emitType`/`emitCallable` parent-id computation exactly; drift there would silently dangle endpoints. +- Determinism risk: edge order was nondeterministic before sorting — the determinism gate fails without a stable sort. +- Depends on the L1 child: L2 refines the tree L1 produces; file/land L1 first. +- Full transcript in the spec; schema decisions in repo `CLAUDE.md`. diff --git a/docs/design/issues/epic-v2-migration.md b/docs/design/issues/epic-v2-migration.md new file mode 100644 index 0000000..1072e35 --- /dev/null +++ b/docs/design/issues/epic-v2-migration.md @@ -0,0 +1,67 @@ + + +## Spec + +https://github.com/codellm-devkit/codeanalyzer-go/blob/main/docs/design/specs/v2-l1-emission.md + +## Summary + +Migrate `codeanalyzer-go` from the legacy v1 output to the **canonical schema v2** (one +additive tree: `application → module → type → callable → body` with typed edge overlays) +and bring it to **analysis level 2** — L1 structural containment + L2 identity-only +`call_graph`, in both projections (`analysis.json` + Neo4j). This is a **schema-major** +change (new envelope, `can://` ids, `span` with UTF-8 byte offsets, `source` per module, +`body` call nodes, `{src,dst,prov}` edges). The `python-sdk` Go wiring lands in lockstep +behind a frozen public API. **Reach for this train: L1 + L2 to parity; L3/L4 deferred.** + +Golden rule: keep everything that *computes* facts (parser, resolver, call-graph +builder); replace only what *serializes* them. + +## Affected repos (from Contract-Impact Triage) + +- `codeanalyzer-go` — analyzer emission (L1 + L2) + Neo4j v2 projection — backend rung +- `python-sdk` — net-new Go model wiring to v2 (`cldk/analysis/` has no `go` package today) — frontend rung +- docs (user-facing schema/levels) — later, `finishing-cldk-work` + +## Design decisions + +- **D1** Pure canonical v2 (drop per-callable `code`/flat `start_line`/`end_line`; recover text by slicing `module.source`) +- **D2** `application.id` = `can://go/`, `` = `--app-name` (default: input dir base name) +- **D3** `span.bytes` = UTF-8 byte offsets into `module.source` (Go strings are UTF-8; avoids the UTF-16/rune mismatch) +- **D4** Single type `kind` ∈ `struct|interface|alias|defined` (collapses v1 `is_interface`); methods nest under receiver type's `callables{}` +- **D5** `base_types[]` = embedded ids (explicit spine) vs `interfaces[]` = computed structural satisfaction (method-set matching) +- **D6** `error_channel[]` from `error`-typed returns; closures → nested `callables{}`; `is_constructor_call` dropped (Go has none) +- **D7** `call` body nodes carry typed `is_goroutine`/`is_deferred`; `callee` is the null→id refinement slot (null at L1, backfilled at L2) +- **D8** `source_file` on a callable when a method is declared apart from its receiver type (cross-file-method span fix) +- **Scope guard:** analyzer is a **pure graph provider** — no slicing/taint in the analyzer (those are SDK queries) + +## Release plan + +- Analyzer = **major** release (breaking output). L1 and L2 are independently shippable behind `--analysis-schema 2`; v1 stays the default until the SDK migrates. +- SDK = **major** release; pins the analyzer **only after** the analyzer v2 release is cut (old SDK Go wiring can't parse v2 until then — and there is no Go wiring yet, so this is net-new). +- Neo4j projection and the JSON schema move in lockstep. + +## Deferred (not this train) + +- **L3** (CFG/CDG/DDG) and **L4** (SDG `param_in`/`param_out`/`summary`) — a later train reusing this train's `can://` ids and `{src,dst,prov}` edge shape verbatim (Group A/C vocabulary, coined once here per the parity clause). +- Go-specific `cfg` edge kinds (`defer_resume`, `select`) — recorded when L3 is designed. +- Points-to oracle (L4 prerequisite); `--materialize-expressions`/`--materialize-basic-blocks`; framework detection. + +## Definition of done (epic-level) + +- Every sub-issue closed and its PR's gate green. +- Analyzer output validates against the SDK v2 Go models; the `L1 ⊆ L2` superset gate holds; parity clause holds (no renamed/repurposed shared vocabulary). +- Superset gate: v2 output carries every v1 fact modulo the sanctioned drops (`code`, `is_*` booleans folded into `kind`, `is_constructor_call`). +- `--analysis-schema 2` output is deterministic across runs (ids and edge order stable). +- SDK public API unchanged (major bump + documented semantic shifts); analyzer↔SDK versions pinned in lockstep. +- Docs / CHANGELOG updated. +- The two switch-over steps: (1) flip the default output from v1 to v2; (2) remove the v1 emitter once `python-sdk` has migrated and nothing reads v1 — both filed as their own units when due. diff --git a/docs/design/issues/parent-v2-migration.md b/docs/design/issues/parent-v2-migration.md index 3496ca0..c0bd54d 100644 --- a/docs/design/issues/parent-v2-migration.md +++ b/docs/design/issues/parent-v2-migration.md @@ -18,7 +18,7 @@ the one-model SDK surface. This is a coordinated schema major across two repos: `codeanalyzer-go` (emission rewrite) and `python-sdk` (Go model remap), released in lockstep. Reach for this train: L1 + L2 to parity; L3/L4 deferred. -**Describe the solution you'd like.** +**Describe the solution you'd like** - [ ] `codeanalyzer-go`: L1 emission — additive tree, `source` per module, `can://` ids, `body{}` with `call` nodes - [ ] `codeanalyzer-go`: L2 emission — `call_graph: [{src,dst,prov,weight}]` at application scope @@ -27,12 +27,12 @@ lockstep. Reach for this train: L1 + L2 to parity; L3/L4 deferred. - [ ] Release v2.0.0 and pin analyzer in SDK **only once both are cut** - [ ] Superset gate green: v2 output contains every v1 fact modulo sanctioned drops -**Describe alternatives you've considered.** +**Describe alternatives you've considered** Does NOT build L3 (CFG/CDG/DDG) or L4 (SDG). Those are a later train reusing this train's `can://` ids and edge shape unchanged. Does NOT add framework detection. -**Additional context.** +**Additional context** - Design transcript: `docs/design/specs/v2-l1-emission.md` (linked, not pasted). - Group A vocabulary (`can://` ids, `span.bytes` UTF-8) is reused verbatim by L3/L4 — coined once here per the parity clause. diff --git a/docs/design/issues/pr-body-v2-l1-l2.md b/docs/design/issues/pr-body-v2-l1-l2.md new file mode 100644 index 0000000..9258919 --- /dev/null +++ b/docs/design/issues/pr-body-v2-l1-l2.md @@ -0,0 +1,52 @@ +Migrate `codeanalyzer-go` to emit the canonical CLDK **v2** `analysis.json` for analysis levels **L1** (additive containment tree) and **L2** (call-graph refinement). v1 output is unchanged and remains the default; v2 is opt-in behind `--analysis-schema 2`. + +## Motivation and Context + +Closes #8 +Closes #9 + +Part of epic codellm-devkit/.github#94. Go was the last analyzer on v1; this moves it onto the shared v2 keystone (`can://` ids, additive tree, `span` with UTF-8 byte offsets, `source` per module, `{src,dst,prov}` edges) so `CLDK(language="go")` can share the one-model SDK surface. Design transcript: `docs/design/specs/v2-l1-emission.md`. + +## How Has This Been Tested? + +Gates run on this commit (analyzer matrix; `testdata/multipackage` fixture): + +``` +# Fixture suite (forced, uncached) +$ go test -count=1 ./... +ok cmd/codeanalyzer ok internal/core ok internal/schema/v2 ok internal/syntactic_analysis/v2emit (all green, exit 0) + +# Determinism: -j 1 vs -j 8, v2 L2 +$ cango -i testdata/multipackage --analysis-schema 2 -a 2 -j 1 -o j1/ ; ... -j 8 -o jN/ +$ diff j1/analysis.json jN/analysis.json # byte-identical -> PASS + +# Monotonicity: v2 L1 ⊆ v2 L2 +L1 node ids: 33; L2 node ids: 33; L1 ids missing from L2: 0 +L1 call_graph: 0 edges; L2 call_graph: 9 edges # L2 adds, never rewrites -> PASS +``` + +Schema-conformance gate (validate against the SDK CPG model) is **deferred**: `python-sdk` has no Go model yet (`cldk/analysis/` has no `go` package) — that is the separate SDK-wiring child of epic #94. Structural self-check passed: valid JSON, `schema_version 2.0.0`, `application.id can://go/multipackage`, edge keys exactly `src/dst/prov/weight`. + +## Breaking Changes + +None for existing users: v1 is still the default output. v2 is additive and opt-in behind `--analysis-schema 2`. The v2 contract itself is a new major, consumed by nothing yet (no SDK Go pin); the default flip and v1-emitter removal are later units, filed when due. + +## Types of changes +- [ ] Bug fix (non-breaking change which fixes an issue) +- [x] New feature (non-breaking change which adds functionality) +- [ ] Breaking change (fix or feature that would cause existing functionality to change) +- [x] Documentation update + +## Checklist +- [x] I have read the [Codellm-Devkit Documentation](https://codellm-devkit.info) +- [x] My code follows the repository's style guidelines +- [x] New and existing tests pass locally +- [x] I have added appropriate error handling +- [x] I have added or updated documentation as needed + +## Additional context + +- L1 computes no new facts beyond span offsets; L2 is pure refinement (signature→`can://` id via `buildSigIndex`). Parser/resolver unchanged (golden rule: replace what serializes, keep what computes). +- `span.bytes` are UTF-8 byte offsets into `module.source` (Go strings are UTF-8). +- `source_file` carried on a callable only when a method is declared apart from its receiver type. +- L3/L4 deferred to a later train; Group A ids/spans already designed for verbatim reuse. From b01a2ec48bd3cdb7ce212ac97ac8f5dbcf8450cd Mon Sep 17 00:00:00 2001 From: lambertpw Date: Sun, 4 Oct 2026 13:19:24 -0400 Subject: [PATCH 18/29] design: Neo4j projection spec + roadmap (group D of the v2 migration) The committed design record for the Go Neo4j projection (--emit neo4j), collision group D of the v1->v2 migration. A pure second projection of the frozen v2 envelope into a GO_/Go-namespaced property graph, keyed on the same can:// ids so JSON and graph never diverge. Cross-repo (analyzer + python-sdk GoAnalysisBackend), tracked under epic codellm-devkit/.github#94. - specs/neo4j-projection.md: triage, the two design-loop decision tables (analyzer graph vocabulary + SDK type-centric facade), package layout, release plan, definition of done. - roadmap.md: milestones M1-M7 with commit boundaries, dependency DAG, and the test gate each must pass. - issues/: ready-to-file feature_request drafts (parent under #94, M1 analyzer child, M7 SDK child) for maintainer filing where access is pull-only. - CLAUDE.md: Neo4j-projection decisions section (auditable decision record). - Carries the TS/Python design references and the prompt as design inputs. --- CLAUDE.md | 91 ++++ .../Python_neo4j_design_and_dataflow.md | 394 ++++++++++++++++++ .../Typescript_neo4j_design_and_dataflow.md | 360 ++++++++++++++++ .../issues/child-m1-schema-contract.md | 59 +++ .../issues/child-m7-sdk-goanalysisbackend.md | 59 +++ .../issues/parent-neo4j-projection.md | 62 +++ docs/design/Neo4j_projection/prompt.md | 85 ++++ docs/design/Neo4j_projection/roadmap.md | 201 +++++++++ .../specs/neo4j-projection.md | 229 ++++++++++ 9 files changed, 1540 insertions(+) create mode 100644 docs/design/Neo4j_projection/Python_neo4j_design_and_dataflow.md create mode 100644 docs/design/Neo4j_projection/Typescript_neo4j_design_and_dataflow.md create mode 100644 docs/design/Neo4j_projection/issues/child-m1-schema-contract.md create mode 100644 docs/design/Neo4j_projection/issues/child-m7-sdk-goanalysisbackend.md create mode 100644 docs/design/Neo4j_projection/issues/parent-neo4j-projection.md create mode 100644 docs/design/Neo4j_projection/prompt.md create mode 100644 docs/design/Neo4j_projection/roadmap.md create mode 100644 docs/design/Neo4j_projection/specs/neo4j-projection.md diff --git a/CLAUDE.md b/CLAUDE.md index 7b3f774..40e2964 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -58,3 +58,94 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. - L3 (CFG/CDG/DDG) and L4 (SDG param edges) — reuse Group A ids/spans unchanged. - Go-specific `cfg` edge kinds (e.g. `defer_resume`, `select`) are recorded when L3 is designed, not now. + +## Neo4j-projection decisions + +The graph projection (`--emit neo4j`) is a **second projection of the same v2 envelope**, +keyed on the identical `can://` ids — JSON and graph join on one string, never recomposed. +These are the Go leaf decisions, made with the user in the 2026-10-04 design loop +(`designing-cldk-changes`), under the parity clause (add `GO_`/`Go` leaves; reuse the +shared neutral vocabulary verbatim; never rename a sibling's shared name). Spec: +`docs/design/Neo4j_projection/specs/neo4j-projection.md`. Cross-repo (analyzer + +`python-sdk` fork); tracked under epic `codellm-devkit/.github#94`. + +### Namespacing & anchor (parity with Python `PY_`/`Py`, TS `TS_`/`TS`) +- **Node-label prefix `Go`, relationship prefix `GO_`.** The SDK's `AnalysisBackend` ABC + reads these as its `N`/`P` ClassVars, so `GoAnalysisBackend` sets `N="Go"`, `P="GO_"`; + the graph vocabulary and the backend's `P`/`N` must agree letter-for-letter. +- **`:GoCanNode` marker label** rides every `can://`-id-keyed node (index anchor for the + prefix seek; carries no safety claim). **`:GoSymbol` shared merge-label** on `:GoType` / + `:GoCallable` / `:GoExternal` (one uniqueness constraint; one MATCH target by id). +- **Scoping:** every destructive statement is scoped on the `can:///` id prefix, + anchored on `:GoCanNode` — mirrors the sibling `can://`-prefix invariant (`.github` + spec `2026-09-02-prune-scope-on-can-id-prefix.md`). + +### Node labels (narrower than Python — Go has fewer kinds) +- **`:GoType`** (one label, `kind ∈ struct|interface|alias|defined` as a **property**), + NOT per-kind labels — mirrors the JSON `type.kind` collapse (§ `type` node above), + keeping the two projections symmetric. +- **`:GoField`** (struct fields), id `/`, via `GO_HAS_FIELD`. +- **No `:GoDecorator` / `:GoAttribute` / `:GoVariable`** — Go's JSON tree has none; the + parity clause adds at the leaves, it does not force a node a language lacks. (Struct + tags are not modeled yet — deferred, not forced in.) + +### Relationships +- **Containment:** `GO_HAS_MODULE`, `GO_DECLARES` (module→func, callable→closure), + `GO_HAS_METHOD` (type→receiver-method), `GO_HAS_FIELD`, `GO_HAS_BODY_NODE`. +- **Inheritance (Go's two distinct mechanisms, named honestly):** `GO_EMBEDS` = + `base_types[]` (explicit struct/interface embedding; deferred/edge-only-when-resolved + like `PY_EXTENDS`) vs **`GO_SATISFIES`** = `interfaces[]` (computed structural + satisfaction via method-set matching). Deliberately **not** `GO_EXTENDS`/`GO_IMPLEMENTS` + — embedding is composition not subtyping, and satisfaction is implicit not declared. +- **Call graph:** `GO_CALLS` (weight, prov; endpoints resolve to a callable id or an + id-keyed `:GoExternal` ghost — no dangling endpoint), `GO_RESOLVES_TO` (call body node → + callee). +- **Imports:** `GO_IMPORTS` (aggregated per target) → `:GoModule` (in-project) or a + `:GoExternal` ghost (stdlib/external, keyed by import path). + +### Flattened properties (Neo4j-legal scalars / homogeneous arrays) +- **`is_goroutine` / `is_deferred`** → boolean props on the call `:GoBodyNode` (pruned when + false, matching JSON `omitempty`), not labels, not edge discriminants (the fact lives on + the call site even when the callee is unresolved and no edge exists). +- **`source_file`** (cross-file method, § callable above) → scalar prop on `:GoCallable`; + text recovery joins to the declaring `:GoModule` by that path. Present only when it + differs from the nesting module. +- **`error_channel[]`** → string-array prop on `:GoCallable` (a value-shaped fact, not an + edge — Go errors are returned values, not thrown types). + +### Repository-artifact layer (neutral, reused verbatim — cross-language merge targets) +- Declare `:Artifact` / `:Package` / `:ConfigKey` and `HAS_ARTIFACT` / + `DECLARES_DEPENDENCY` (`_k=kind`) / `LOCKS` / `DEFINES_CONFIG` **un-prefixed**, identical + to the siblings, so a sibling analyzer over the same repo converges on the same node. + The projector emits rows **only for facts the Go analyzer produces** (go.mod deps → + `:Package` `pkg:golang/…`); absent facts emit **zero rows** (declared contract, empty + projection). `GO_PROVIDES` / `GO_UNRESOLVED_IMPORT` stay `GO_`-prefixed (this analyzer's + own resolution claims). + +### `_k` relationship discriminants (parallel edges survive MERGE) +- `GO_CFG_NEXT` per `kind`; `GO_DDG` per `(var, prov)`; `DECLARES_DEPENDENCY` per `kind` + — same discriminant mechanism as the siblings. + +### Reserved L3/L4 overlay (declared now, emits nothing until the levels land) +- `schema.go` declares the full overlay vocabulary **now** — `GO_HAS_BODY_NODE`, + `GO_CFG_NEXT`, `GO_CDG`, `GO_DDG`, `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY`, and the + body-node statement kinds — with parity-correct names and `_k` discriminants. The + analyzer caps at L2, so the projector walks these sections and emits **zero rows**; + when L3/L4 land they fill with **no schema-contract change**. (`--emit neo4j` is + "full-depth = max implemented" — L2 today; the graph grows additively.) + +### SDK facade (`GoAnalysisBackend`, on the `python-sdk` fork — frontend rung) +- **Type-centric surface** (Go is the SDK's first class-less facade; no C precedent exists + yet): `get_types`/`get_type`, `get_functions` (package-level), `get_methods_of`/ + `get_method` (receiver methods), `get_fields` — **not** `get_classes` (Go has no class), + and functions are kept distinct from methods (no "module-name-as-class" hack). +- `get_callers`/`get_callees` keyed on the callable id / `(type_id, sig)` (C-facade style: + take the callable, not class+method strings). +- **Scope this train:** Tier-A lifecycle + L1/L2 leaf accessors + repository-artifact + getters + `get_source`. Slicing / CFG / CDG / DDG / reachability are **omitted** (they + need L3/L4 the analyzer does not produce) per the ABC contract. + +### Deferred (not this train) +- SDK consumer slicing/taint surface and L3/L4 facade methods — arrive with the L3/L4 + analyzer train, reusing this graph vocabulary unchanged. +- Struct-tag modeling; Go-specific `cfg` edge kinds (`defer_resume`, `select`) — with L3. diff --git a/docs/design/Neo4j_projection/Python_neo4j_design_and_dataflow.md b/docs/design/Neo4j_projection/Python_neo4j_design_and_dataflow.md new file mode 100644 index 0000000..b4052a8 --- /dev/null +++ b/docs/design/Neo4j_projection/Python_neo4j_design_and_dataflow.md @@ -0,0 +1,394 @@ +# Neo4j design — the graph projection of `codeanalyzer-python` + +> **Status:** design/architecture reference, written 2026-10-02 against `codeanalyzer/neo4j/`. +> For the *authoritative vocabulary* (every label, every property, every `PY_*` relationship), +> read `docs/skills/analyzing-canpy-graphs/references/vocabulary.md` and the machine-readable +> `schema.neo4j.json` (regenerate with `uv run canpy --emit schema`). This document explains +> **purpose, dataflow, module responsibilities, and the projection algorithm** — the *why* and +> *how*, not a second copy of the vocabulary table. + +--- +## Repo +https://github.com/codellm-devkit/codeanalyzer-python + +## 1. Purpose + +`codeanalyzer-python` emits **canonical schema v2** — one additive Code Property Graph (CPG) +tree — in two **projections** of the *same facts* keyed on the *same ids*: + +1. **`analysis.json`** — the tree itself (the `Analysis` envelope), the single wire format. +2. **Neo4j** — a **near-identity projection** of that tree into a labeled property graph. + +The Neo4j backend exists so the CPG can be **queried as a graph**: entrypoint reachability, +call-graph and taint traversals, data-/control-dependence slicing, config-key reads — all the +things that are natural as Cypher path queries and awkward as JSON tree walks. It is the +substrate the cross-language CLDK SDK's Neo4j backends attach to. + +**Core invariant — JSON == Graph.** The projection is *lossless* up to re-expression: +> re-expressing a JSON field as an edge or a flattened property is fine; silently dropping one is not. + +(This is the `2026-09-10-neo4j-json-parity.md` contract, #202/#203.) A JSON-side node and its +graph node **join on one string** — the same `can://` id or global ordinal id — with no id +recomposition. That identity equality is what makes the two projections interchangeable and is +built once, upstream, by `codeanalyzer/dataflow/identity.py`. + +**What it is *not*.** The analyzer is the **provider**: it emits the SDG *substrate* only. There +is **no `taint_flows` section** by design — labeled reachability + source/sink model packs are the +SDK's job. The backward slicer is internal (an L3/L4 validation gate), not a product surface. + +--- + +## 2. Where it sits (the module map) + +Everything lives in `codeanalyzer/neo4j/` and runs **only** when `--emit neo4j` (or +`--emit schema`) is selected — it is off the default JSON path entirely, and the `neo4j` Bolt +driver is an **optional** dependency imported lazily. + +| File | Lines | Responsibility | +| --- | --- | --- | +| `emit.py` | ~80 | **Facade** between the CLI and the backend. `emit_schema()` (static contract) and `emit_neo4j()` (project + write). | +| `schema.py` | ~430 | **Declarative schema contract** — the single in-repo source of truth: node labels + keys + typed props, relationship types + endpoints, derived DDL (constraints + indexes), `SCHEMA_VERSION`. | +| `project.py` | ~940 | **The projector** — walks the `PyApplication` IR and emits graph rows. This is where JSON → row mapping happens. | +| `rows.py` | ~230 | **Output-agnostic row IR** (`GraphRows`, `NodeRow`, `EdgeRow`, `NodeRef`) + the `RowBuilder` (MERGE-in-memory accumulator) + Cypher-literal rendering helpers. | +| `cypher.py` | ~160 | **Snapshot writer** — renders `GraphRows` to a self-contained `graph.cypher` script. | +| `bolt.py` | ~310 | **Incremental writer** — pushes `GraphRows` into a live Neo4j over Bolt, updating only what changed. | +| `__init__.py` | ~45 | Public surface re-exports. | + +**Design principle: the projection is pure, the writers are dumb.** `project.py` → `rows.py` +produces a deterministic, deduped bag of nodes and edges with **no I/O and no driver**. Both +writers then consume that identical bag — one renders text, one talks Bolt. Nothing about the +graph's *content* lives in a writer. + +--- + +## 3. End-to-end dataflow + +``` + codeanalyzer analysis pipeline (syntactic → semantic → dataflow) + │ + ▼ + Analysis envelope (in memory) + └─ application: PyApplication + ├─ symbol_table {: PyModule} ← the node tree + ├─ call_graph [PyCallEdge] + ├─ external_symbols {: …} + ├─ param_in / param_out [ParamEdge] + ├─ artifacts / dependencies / unresolved_imports + └─ entrypoint_report + │ + emit_neo4j(analysis, options) ── emit.py (facade) + │ + ┌────────────────────────────┼─────────────────────────────┐ + │ 1. assign_ids(app, app_name) │ + │ → stamps every module/class/callable with its can:// │ + │ id (idempotent); returns sig_to_id map │ + │ 2. project(app, app_name, sig_to_id, analyzer) │ + └────────────────────────────┼─────────────────────────────┘ + │ + project.py (the projector) + walks IR, calls RowBuilder.node()/.edge()/.edge_to_symbol() + │ + ▼ + RowBuilder.finish() + - MERGE-dedup nodes in memory + - resolve deferred "edge-only-if-target-exists" edges + - sort deterministically + │ + ▼ + GraphRows (rows.py) + { nodes: [NodeRow], edges: [EdgeRow] } ← pure data + │ + ┌─────────────────────┴──────────────────────┐ + │ --neo4j-uri set? │ + no ────┤ ├──── yes + ▼ ▼ + render_cypher(rows) (cypher.py) bolt_writer(rows, cfg, …) (bolt.py) + → graph.cypher file → live Neo4j over Bolt + - constraints + indexes - ensure constraints + indexes + - scoped wipe of prior subgraph - diff content_hash per module + - batched UNWIND … MERGE nodes - purge/upsert only changed modules + - batched UNWIND … MERGE edges - optional orphan prune (full+eager) + (full truth, non-incremental) (incremental, additive by default) +``` + +### The two-step facade (`emit.py`) + +`emit_neo4j(analysis, options)` does exactly two things before handing off to a writer: + +1. **`assign_ids(analysis.application, app_name)`** — idempotent. It stamps every + module/class/callable with its canonical `can://` id and returns the `signature → id` map that + the projection keys nodes on. *This is what keeps the JSON and Neo4j projections in agreement:* + both are built from the same stamped tree and the same `sig_to_id`. +2. **`project(application, app_name, sig_to_id, analyzer)`** → `GraphRows`. + +Then it branches on `options.neo4j_uri`: present → Bolt push; absent → `graph.cypher` snapshot in +the output directory (`GRAPH_CYPHER`). + +`emit_schema(output)` is independent — it needs no analyzed project. It serializes +`build_schema_document()` to `schema.json`, the version-stamped contract. + +--- + +## 4. The row IR and `RowBuilder` (`rows.py`) + +`GraphRows` is a **deterministic, deduped bag** of `NodeRow`s and `EdgeRow`s. The data shapes: + +- **`NodeRow`** — `labels` (`labels[0]` is the constrained MERGE label; the rest are SET as extra + labels), `key_prop`, `value` (the key), `props`, and an **in-memory-only** `module` field (the + owning module's id, used by the incremental writer's per-module diff — never emitted to the graph). +- **`EdgeRow`** — `type`, `from_ref`, `to_ref`, `props`, and an optional `key` **relationship + discriminant**. +- **`NodeRef`** — how an edge addresses an endpoint: `(label, key_prop, value)` to MATCH on. + +`RowBuilder` is the in-memory analog of Cypher MERGE semantics. Three design choices live here: + +1. **Node MERGE-dedup.** `node(labels, key_prop, value, props)` upserts on `(labels[0], value)`: + re-seeing the same identity merges props (last-write-wins) and unions labels. A hot external + symbol or a canonical decorator collapses to one row. (Edges via `.edge()` are **append-only** — + not deduped — so per-package facts need an explicit `seen` guard in the projector.) + +2. **The `PyCanNode` marker.** Any node keyed by a `can://` id automatically gets the extra label + `PyCanNode`. It is a pure **index anchor** (Neo4j property indexes are label-scoped) so that the + prefix predicate `id STARTS WITH $p` can *seek* instead of scanning the store. It carries no + safety claim — the test is the `can://` scheme, never a language segment. + +3. **The "edge-only-when-resolved" rule.** `edge_to_symbol(...)` *defers* an edge whose target may + be library/external code not present in the graph. At `finish()`, a deferred edge is kept **only + if** its target `(label, value)` was actually emitted as a node — so it never dangles. (The + string fallback for an unresolved base class survives on the source node's own props instead.) In + the whole projector this is used in **exactly one** place: `PY_EXTENDS` (base classes, resolved by + the per-module `_base_ref_resolver`: local class → import-table resolve → `@external` ghost). + `PY_RESOLVES_TO` deliberately uses the plain `b.edge()` instead, because a call node's `callee` is + *already* a resolved `can://` id — routing it through the symbol fallback would match a + `signature` property against an id and emit an edge that matches nothing at load time. + +`finish()` resolves deferred edges, sorts nodes and edges by a stable key, and returns the +`GraphRows`. Determinism matters: identical input → byte-identical `graph.cypher`. + +> **Neo4j-legal values only.** Property values are primitives or homogeneous arrays of primitives. +> `None` is pruned (`prune()`) — in Neo4j a null property is simply absence, so "not carried" and +> "empty" never collide. `Optional[Any]` fields (variable `value`, call `arguments`) are +> JSON-encoded into a `*_json` string property. + +--- + +## 5. The projection algorithm (`project.py`) + +`project(app, app_name, sig_to_id, analyzer)` walks the IR in this order, emitting rows as it goes: + +1. **Application root** → `:PyApplication` node, keyed on the `can://` id (**not** on the + free-text `--app-name`, so two apps analyzed under one name stay two roots). Carries analyzer + identity, repo info, and the whole entrypoint report (`entrypoint_report_json` + always-present + `entrypoint_frameworks`). + +2. **Per module** (`symbol_table` walk) → `:PyModule` node (`id` key, carries whole-file `source`, + `content_hash`), `PY_HAS_MODULE` edge from the app, then `_project_module_body` for its + functions, classes, module-level variables, and imports. + +3. **Call graph** (`app.call_graph`) → the aggregated `PY_CALLS` twin: one edge per `(src, dst)` + with `weight` + `prov`, endpoints resolved by `_call_endpoint`. + +4. **Program graphs** (`_project_program_graphs`) → the L3/L4 CPG overlay: `:PyBodyNode` nodes + + `PY_HAS_BODY_NODE`, `PY_CFG_NEXT`, `PY_CDG`, `PY_DDG`, and (L4) `PY_PARAM_IN`/`PY_PARAM_OUT`/ + `PY_SUMMARY`. Idempotent — a no-op below the level that populates the body fields. + +5. **Artifacts/dependencies** (`_project_artifacts`) → the language-neutral `Artifact`/`Package`/ + `ConfigKey` subgraph. **Always emitted** (L1 data, identical at every `-a`). + +6. **Config uses** (`_project_config_uses`) → `PY_USES_CONFIG` (resolved reads: body node → + `ConfigKey`) and `PY_READS_CONFIG_UNRESOLVED` (first-class unresolved reads). + +### Identity: the three id forms used as merge keys + +The projection never invents ids — it reads them off the stamped tree so both projections agree: + +- **`can://` ids** for every node at/above a callable (`:PyApplication`, `:PyModule`, `:PyClass`, + `:PyCallable`, `:PyExternal`, `Artifact`, `ConfigKey`). `can://` is the outermost prefix of + every id for that application. +- **GLOBAL ordinal ids** `"@"` for `:PyBodyNode` and every cross-callable + dataflow endpoint. `_global_ordinal()` **must** agree with `IdentityMap.global_id` — that is the + two-projection agreement for body-level nodes. +- Package/decorator nodes merge on `name` (shared, cross-application) — never pruned, so + per-application facts ride the *relationship*, not the node. + +**The `:PySymbol` idiom.** Both declared symbols (`:PyClass`, `:PyCallable`) and external ghosts +(`:PyExternal`) carry the shared **`:PySymbol`** MERGE label. Because `RowBuilder` merges on +`(labels[0], value)`, a call to a bare imported name and its `PY_PROVIDES` ghost collapse onto one +node — correctly, since they name the same real-world symbol. Attribute and variable ids are minted +from the **owner's `can://` id** (`/`, `/@`), so they carry the +application segment and the module's prefix-purge reaches them (a signature-minted id would be +identical across two applications and MERGE them onto one node). + +### How endpoints resolve (the resolved/external split) + +`_call_endpoint` is **authoritative**, not heuristic: classification comes from +`app.external_symbols` (keyed by `can://…/@external/…` id), so an imported module name — which +exists only as a `:PyPackage` — can never shadow a real call target. A declared endpoint resolves +to its `can://` id; anything neither declared nor listed falls back to an **id-keyed `:PyExternal` +ghost** rather than raising. **No `PY_CALLS` endpoint ever dangles** — every target joins the id +space, either as a declared node or as an external ghost. + +### JSON source → graph row mapping + +| IR source (JSON) | Node label(s) | Relationship(s) | Projector | +| --- | --- | --- | --- | +| `application` + `entrypoint_report` + `analyzer` | `PyApplication` | — | `project` | +| `symbol_table[file]` (`PyModule`) | `PyModule` | `PY_HAS_MODULE` | `project` / `_project_module_body` | +| `PyModule.functions`, `PyClass.callables` | `PyCallable` (`:PySymbol`) | `PY_DECLARES`, `PY_HAS_METHOD` | `_project_callable` | +| `PyModule.types` / `PyClass` (nested) | `PyClass` (`:PySymbol`) | `PY_DECLARES`, `PY_EXTENDS` (base classes, deferred) | `_project_class` | +| `PyClass` attributes | `PyAttribute` | `PY_HAS_ATTRIBUTE` | `_project_attribute` | +| module/callable variables | `PyVariable` | `PY_DECLARES_VAR` | `_project_variable` | +| decorators | `PyDecorator` (merged on `qualified_name`) | `PY_DECORATED_BY` (span + args ride the edge) | `_project_decorator` | +| `PyModule.imports` (aggregated per target) | `PyModule`/`PyPackage` target | `PY_IMPORTS` (`spellings`, `positions_json`) | `_project_imports` | +| `application.call_graph` (`PyCallEdge`) | declared `PyCallable` or `PyExternal` ghost | `PY_CALLS` (`weight`, `prov`) | `project` / `_call_endpoint` | +| `external_symbols` endpoints | `PyExternal` (`:PySymbol`) ghost | (endpoint of `PY_CALLS`/`PY_RESOLVES_TO`) | `_external_ghost` / `_call_endpoint` | +| callable `body`/`cfg`/`cdg`/`ddg` (L3) | `PyBodyNode` | `PY_HAS_BODY_NODE`, `PY_CFG_NEXT` (`kind`), `PY_CDG`, `PY_DDG` (`var`,`prov`) | `_project_program_graphs` | +| call-site body nodes | `PyBodyNode` (`kind:"call"`, `callee_signature` joined from `call_sites`) | `PY_RESOLVES_TO` → callee | `_project_program_graphs` | +| param vertices + `param_in`/`param_out`/`summary` (L4) | `PyBodyNode` (`formal_in`/`_out`, `actual_in`/`_out`) | `PY_PARAM_IN`, `PY_PARAM_OUT`, `PY_SUMMARY` | `_project_program_graphs` | +| `artifacts` | `Artifact` (neutral) | `HAS_ARTIFACT`, `DEFINES_CONFIG` | `_project_artifacts` | +| `dependencies` | `Package` (neutral, purl) | `DECLARES_DEPENDENCY` (`kind`-disc.), `LOCKS`, `PY_PROVIDES` | `_project_artifacts` | +| config keys (per artifact) | `ConfigKey` (neutral) | `DEFINES_CONFIG` | `_project_artifacts` | +| `unresolved_imports` | `PyExternal` ghost | `PY_UNRESOLVED_IMPORT` | `_project_artifacts` | +| config reads (#162) | (bridges body node → `ConfigKey`) | `PY_USES_CONFIG`, `PY_READS_CONFIG_UNRESOLVED` | `_project_config_uses` | + +--- + +## 6. The schema contract (`schema.py`) + +`schema.py` is the **single declarative source of truth** for the graph contract. It is *not* just +documentation — three things are derived from it: + +- **The DDL.** `uniqueness_constraints()` emits **one constraint per distinct `(merge_label, key)`**, + so a new label automatically brings its own constraint — there is no second list to keep in sync. + Plus hand-written `INDEXES` (name/code lookups, and the critical `py_can_node_id` range index that + makes every scoped/destructive statement a prefix *seek*). +- **`--emit schema`** serializes the whole thing (`build_schema_document()`) to `schema.json`. +- **A conformance test** (`test/test_neo4j_schema.py`) asserts the real emitter in `project.py` + **never** produces a label / relationship / property not declared here. So the contract cannot + silently drift from the projector. + +### Relationship identity: the `_k` discriminant + +A plain endpoint-pair MERGE would **collapse legitimately-distinct parallel edges**. Several +relationships therefore carry an internal `_k` discriminant (set via `EdgeRow.key`), so the MERGE is +on `{_k: …}`: + +- `PY_DDG` merges per `(var, prov)` — one dependence per variable, and the ssa/points-to split. +- `PY_CFG_NEXT` merges per `kind` — a conditional's true/false pair. +- `DECLARES_DEPENDENCY` merges per `kind`; `PY_READS_CONFIG_UNRESOLVED` per `(key, reason)`. + +### Namespacing: `PY_` vs neutral + +Code-shaped vocabulary is **`PY_`-prefixed by design** (`:PyModule`, `PY_CALLS`, …) so a shared, +multi-language Neo4j database never mingles one analyzer's facts with another's. The +`Artifact`/`Package`/`ConfigKey` subgraph is **deliberately un-prefixed** — these are cross-language +**merge targets**, so a sibling-language analyzer over the same repo lands on the *same* node +instead of a per-language duplicate. (`PY_PROVIDES` / `PY_UNRESOLVED_IMPORT` stay `PY_` — they are +*this* analyzer's claim about what an import resolves to.) + +### Versioning + +`SCHEMA_VERSION = "2.0.0"` is stamped onto every `:PyApplication` node so a consumer can detect a +producer/consumer mismatch at runtime. MAJOR bumps on a breaking change, MINOR on additive — but the +**additive-MINOR rule is suspended for the 2.0.0 line** (per the 2026-09-07 ruling): additive +properties ship without a bump and consumers gate on the **analyzer version** floor instead. + +--- + +## 7. The two writers + +Both consume the identical `GraphRows`; they differ only in liveness and incrementality. + +### Snapshot writer — `cypher.py` → `graph.cypher` + +Renders a **self-contained, non-incremental** `.cypher` script. Running it (e.g. +`cypher-shell < graph.cypher`) rebuilds this project's subgraph from scratch: + +1. constraints + indexes; +2. a **scoped wipe** of the prior version — `MATCH (x:PyCanNode) WHERE x.id = OR x.id + STARTS WITH 'can:///'` then `DETACH DELETE … IN TRANSACTIONS`. Scoped by id prefix, so it is + **one application by construction** — a second Python app sharing a module path, and a sibling + analyzer's `:Py*` graph, are outside it (shared `:Package` purls stay outside too); +3. batched `UNWIND … MERGE` for nodes (grouped by full label set + key prop); +4. batched `UNWIND … MERGE` for edges (grouped by `(type, endpoint labels + key props, discriminated?)`). + +A static script has no view of the live DB, so it expresses the **full truth** — incremental updates +are the Bolt writer's job. + +### Incremental writer — `bolt.py` → live Neo4j + +Reads the DB's current state and updates **only what changed**, with the **module subgraph as the +unit of idempotent replacement**: + +1. ensure constraints + indexes; +2. **diff** each module's `content_hash` against the DB (keyed by module `id` inside the app prefix, + so a second app sharing a path is never mistaken for "unchanged") → the set of changed modules; +3. per changed module, in a transaction: purge the edges it owned + detach-delete the declarations it + no longer emits, then upsert its current nodes; +4. upsert edges owned by changed modules + the shared edges; +5. on a **full run** only, prune modules whose source file vanished (batched, to avoid exhausting the + transaction memory limit). + +**A push never deletes by default (#171).** Steps 3 and 5 — the only destructive steps — run under +`--eager` only. A default `--lazy` push is **purely additive** (MERGE-upsert, nothing removed); the +cost is staleness (a removed declaration/edge lingers until an `--eager` push reconciles it). The +trade is deliberate: an incremental push into a *shared* database should not be able to destroy +anything, and the destructive rebuild is opt-in under the same flag that forces a clean analysis +rebuild. + +**Every destructive statement is scoped on the `can://` id prefix (#173).** The id is a path +(`can:///python//...`), so `id = OR id STARTS WITH /` is exact +containment — one application, one language, one module at once — which neither a label anchor nor the +retired `_module` property could give. `:PyCanNode` anchors the predicate so it *seeks* an index +instead of scanning the store. + +Shared nodes (`:PyExternal` / `:PyPackage` / `:PyDecorator`) have no owning module and are MERGE-only, +so a declaration another (unchanged) module still references survives and its incoming edges stay +valid. + +--- + +## 8. CLI surface and gating + +``` +canpy -i --emit neo4j [--neo4j-uri bolt://… --neo4j-user … --neo4j-password … --neo4j-database …] + [--eager | --lazy] # --lazy is the default (additive push) +canpy --emit schema [-o ] # static contract → schema.json (no project needed) +``` + +- **Neo4j is always full-depth (#119).** `--emit neo4j` forces **level 4 with every graph section**; + passing an explicit `-a`/`--graphs` alongside it is a **flag error** (code 2). The graph carries + every level's facts, so depth/section selectors cannot be combined with it. +- With `--neo4j-uri` → live Bolt push; without → a `graph.cypher` snapshot in the output dir. +- `--eager`/`--lazy` is the single switch that gates **all** destructive behavior in a Bolt push. + +(For the JSON path the level gating is the usual `-a 1|2|3|4`; `--graphs sdg` needs `-a 4`, +`cfg,dfg,pdg` need `-a 3`. Those don't apply once `--emit neo4j` has forced level 4.) + +--- + +## 9. Querying the result (pointers) + +- **Scope every query by id prefix, never by a label list:** + `MATCH (x:PyCanNode) WHERE x.id STARTS WITH 'can:///'` (append `python//` for one module). +- The full label/relationship/property vocabulary and worked query recipes (entrypoint, taint, + exit-point, slicing) live in **`docs/skills/analyzing-canpy-graphs/`** — read it before writing + Cypher. +- The machine-readable contract is `schema.neo4j.json` (`--emit schema`); `SCHEMA_VERSION` on the + `:PyApplication` node lets a consumer detect a mismatch. + +--- + +## 10. Design decisions worth remembering + +| Decision | Why | +| --- | --- | +| **Pure projection, dumb writers** | Graph *content* has one definition (`project.py`); the two writers only differ in liveness. Testable without a DB. | +| **Same ids as JSON, built once upstream** | `analysis.json` and Neo4j join on one string. No recomposition, no drift (`identity.py`). | +| **`can://` prefix scoping + `PyCanNode` anchor** | Destructive statements are exact containment and index-seekable, safe in a shared multi-app / multi-language DB. | +| **Schema derives its own DDL + conformance test** | The contract (`schema.py`) cannot silently drift from the emitter (`project.py`). | +| **`PY_`-namespaced code, neutral artifact/package nodes** | Multi-language DB: never mingle analyzers' code facts; always converge on shared repo-level artifacts. | +| **`_k` relationship discriminant** | Parallel edges (per-var DDG, true/false CFG, kind-split deps) survive MERGE instead of collapsing. | +| **Additive push by default (`--lazy`)** | An incremental push into a shared DB must not be able to destroy anything; destruction is opt-in `--eager`. | +| **Edge-only-when-resolved** | `PY_EXTENDS`/`PY_RESOLVES_TO` never dangle; unresolved targets degrade to a string prop or an external ghost. | +| **No `taint_flows`** | Analyzer is the provider of the SDG substrate; labeled taint is the SDK's job. | diff --git a/docs/design/Neo4j_projection/Typescript_neo4j_design_and_dataflow.md b/docs/design/Neo4j_projection/Typescript_neo4j_design_and_dataflow.md new file mode 100644 index 0000000..5702c99 --- /dev/null +++ b/docs/design/Neo4j_projection/Typescript_neo4j_design_and_dataflow.md @@ -0,0 +1,360 @@ +# Neo4j in `cants` — design & dataflow reference + +> Deep-internals reference for the Neo4j output path in `codellm-devkit/codeanalyzer-typescript`. +> Covers purpose, where it sits in the pipeline, the project → rows → writer dataflow, the full +> label/relationship/property catalog, scoping & safety, schema versioning, and the conformance +> contract. Grounded in `src/build/neo4j/*` and `src/utils/serialize.ts` as of schema v2 `2.0.0`. + +--- +## Git repo + +https://github.com/codellm-devkit/codeanalyzer-typescript + + +## 1. Purpose — what Neo4j *is* here + +Neo4j is **one of two co-primary projections of the same canonical schema-v2 envelope**. The +analyzer builds a single additive Code Property Graph (the `TSAnalysis` envelope). `finalizeAnalysis` +produces that envelope once; then: + +- the **JSON path** writes it verbatim to `analysis.json`, and +- the **Neo4j path** projects *the same envelope* into a property graph. + +> "Second projection of the SAME v2 envelope the JSON path emits … so JSON and graph never diverge." +> — `src/build/neo4j/project.ts` header; `src/utils/serialize.ts::emitNeo4j`. + +The point of the graph projection is to make the CPG **queryable with Cypher**: the containment tree, +call graph, CFG/CDG/DDG, interprocedural SDG, inheritance, decorators, imports/exports, and the +repository-artifact layer (dependencies, config keys) all become nodes and typed relationships you +can traverse. Slicing/taint are *not* materialized — the analyzer is a pure graph **provider**; those +are reachability queries that belong to the frontend SDK. + +Crucially, the Neo4j vocabulary is **namespaced for a polyglot database**: TypeScript nodes/edges are +`TS`/`TS_`-prefixed so a shared multi-language Neo4j can hold TS, Python and Java graphs side by side +and attribute each element to its origin — while a deliberate set of **language-neutral** nodes +(`:Artifact`, `:Package`, `:ConfigKey`) let sibling analyzers `MERGE` onto the *same* repository-level +nodes. + +--- + +## 2. Where it sits in the pipeline + +The whole analyzer is one orchestration function, `analyze()` in `src/core.ts`. Neo4j is **not** a +pipeline stage — it is an *output target* chosen at `emit` time, downstream of everything: + +``` + analyze() (src/core.ts) + materialize → symbol table → call graph → dataflow(L3/L4) → cache → finalizeAnalysis + │ + ▼ + AnalysisResult.application + = TSAnalysis envelope (wire) + │ + emit(application, opts) (serialize.ts) + │ + ┌─────────────────────────────────────┼─────────────────────────────────┐ + ▼ ▼ ▼ + --emit json --emit neo4j --emit schema + write analysis.json emitNeo4j(application, opts) emitSchema(opts) + (streamed, #112) │ buildSchemaDocument() + ▼ → schema.json (static, + project(application) → GraphRows no project needed) + │ + ┌──────────────┴───────────────┐ + ▼ ▼ + --neo4j-uri set? yes --neo4j-uri absent + ▼ ▼ + boltWriter(rows, cfg, …) writeCypherFile(graph.cypher, rows) + (incremental, live Bolt) (self-contained snapshot script) +``` + +**`--emit neo4j` is always full-depth.** `cli.ts` *rejects* combining it with `-a`/`--graphs` +(the level gate applies to the JSON path only). So the projected graph always carries the complete +L4 CPG — see `serialize.ts::emitNeo4j` ("the graph is always projected at full depth") and the +`cli.ts` errors on `--analysis-level` / `--graphs` with `neo4j`. + +### Entry points (`src/utils/serialize.ts`) + +| Function | Trigger | Output | +|---|---|---| +| `emit()` | always | dispatches on `opts.emit` | +| `emitNeo4j()` | `--emit neo4j` | `project()` then bolt **or** cypher | +| `emitSchema()` | `--emit schema` | `schema.json` via `buildSchemaDocument()` — no project required | + +### CLI surface (`src/cli.ts`, `src/options/options.ts`) + +- `--emit json|neo4j|schema` (default `json`) +- `--neo4j-uri ` — present ⇒ live Bolt push; absent ⇒ write `graph.cypher` +- `--neo4j-user` / `--neo4j-password` / `--neo4j-database` (env: `NEO4J_USERNAME`, `NEO4J_PASSWORD`, `NEO4J_DATABASE`) +- `--eager` — purge this app's graph and rebuild (otherwise the push only ever adds/updates) +- full run vs. targeted run (`--only`/targetFiles) controls orphan pruning safety + +--- + +## 3. The module map + +| File | Lines | Responsibility | +|---|---|---| +| `src/build/neo4j/index.ts` | 8 | Barrel: re-exports `project`, the two writers, schema symbols, row types | +| `src/build/neo4j/schema.ts` | 346 | **The contract.** Node labels + keys + typed props, relationship types + endpoints, derived constraints, indexes, `SCHEMA_VERSION`, `buildSchemaDocument()` | +| `src/build/neo4j/project.ts` | 455 | **Pure projection**: `TSAnalysis` tree → `GraphRows`. No I/O | +| `src/build/neo4j/rows.ts` | 232 | `GraphRows`/`NodeRow`/`EdgeRow`, the `RowBuilder` (in-memory MERGE), scoping helpers, Cypher-literal rendering, `prune()` | +| `src/build/neo4j/cypher.ts` | 139 | **Snapshot writer**: `GraphRows` → self-contained `.cypher` script | +| `src/build/neo4j/bolt.ts` | 310 | **Incremental writer**: diff + push `GraphRows` to a live DB over Bolt | + +The split is deliberate: `project.ts` is **pure data** (no driver, no fs), `rows.ts` is the +output-agnostic intermediate both writers consume **identically**, and the two writers differ only +in *how* they realize the same rows (literal script vs. parameterized Bolt). + +--- + +## 4. Dataflow in detail: `project()` → `GraphRows` + +`project(app: TSAnalysis)` walks the uniform v2 tree and emits **one graph node per tree/body node**, +keyed on its `can://` id, with containment and every typed overlay as relationships. It accumulates +into a `RowBuilder` and returns the deduped, sorted `GraphRows`. + +### 4.1 The tree walk + +``` +project(app) +├─ Application node (labels: ["Application","TSApplication"], key id) +├─ for each module in symbol_table: +│ ├─ TSModule node → TS_HAS_MODULE (app → module) +│ └─ projectScope(module): +│ ├─ types{} → projectType (TS_DECLARES) +│ ├─ functions{} → projectCallable (TS_DECLARES) +│ └─ fields{} → projectField (TS_HAS_FIELD) +│ +│ projectType(t): +│ ├─ TSClass/TSInterface/TSEnum/TSTypeAlias/TSNamespace node → TS_DECLARES (parent → t) +│ ├─ decorators → TS_DECORATED_BY (→ shared :TSDecorator) +│ ├─ extends_ids → TS_EXTENDS (deferred, resolved-only) +│ ├─ implements_ids → TS_IMPLEMENTS (deferred, resolved-only) +│ ├─ namespace? recurse projectScope +│ ├─ callables{} → projectCallable (TS_HAS_METHOD) +│ └─ fields{} → projectField (TS_HAS_FIELD) +│ +│ projectCallable(c): +│ ├─ TSCallable node (+ :TSAnonymousCallable label if anon signature) → owner-rel +│ ├─ decorators → TS_DECORATED_BY +│ ├─ body{} → one TSBodyNode each → TS_HAS_BODY_NODE +│ │ call node with string callee → TS_RESOLVES_TO (body → callee) +│ ├─ cfg[] → TS_CFG_NEXT (kind-discriminated) +│ ├─ cdg[] → TS_CDG +│ ├─ ddg[] → TS_DDG ((var,prov)-discriminated) +│ ├─ summary[] → TS_SUMMARY +│ ├─ nested callables (closures) → projectCallable (TS_DECLARES) +│ └─ local types → projectType +│ +├─ repository-artifact layer: +│ ├─ artifacts{} → :Artifact → HAS_ARTIFACT; config_keys → :ConfigKey (DEFINES_CONFIG) +│ ├─ dependencies[] → :Package (purl) → DECLARES_DEPENDENCY, LOCKS, TS_PROVIDES → ghost :TSExternal +│ └─ unresolved_imports[] → TS_UNRESOLVED_IMPORT → ghost :TSExternal +│ +├─ module bindings: imports/exports → TS_IMPORTS / TS_RE_EXPORTS (one edge per module-pair, aggregated) +├─ config_uses[] → TS_USES_CONFIG (body → :ConfigKey) +├─ config_reads[] → TS_READS_CONFIG_UNRESOLVED (app → callee/ghost, folded by key|reason) +├─ external_symbols{} → :TSExternal nodes +├─ synthesized_callables (residual) → :TSAnonymousCallable nodes +└─ application-scope overlays: + ├─ call_graph[] → TS_CALLS + ├─ param_in[] → TS_PARAM_IN + └─ param_out[] → TS_PARAM_OUT +``` + +### 4.2 `RowBuilder` — in-memory MERGE semantics (`rows.ts`) + +The builder is the "database" at projection time: it mirrors Neo4j's `MERGE … SET += props` +*before* any writer runs, so each writer just replays a clean, deduped bag. + +- **`node(labels, keyProp, value, props)`** — upsert keyed on `labels[0] + "\0" + value`. + Re-seeing the same node **merges props (last-write-wins)** and **unions labels**. Returns a + `NodeRef` (`{label, keyProp, value}`) that edges address it by. +- **`edge(type, from, to, props, key?)`** — endpoints known to exist this run. +- **`edgeToSymbol(type, from, targetId, …)`** — a `can://` target that *might not* have materialized + (e.g. a resolved supertype). **Deferred** and kept at `finish()` only if the target id was actually + emitted as a node — the "edge-only-when-resolved" rule, so `TS_EXTENDS`/`TS_IMPLEMENTS` never dangle. +- **`finish()`** — resolves deferred edges, then **sorts nodes and edges deterministically** (so + `graph.cypher` is byte-stable across runs). +- **`prune(props)`** — drops `null`/`undefined`: in Neo4j a null property *is* absence, so the graph + never stores one. A present-but-empty string list is likewise treated as a non-fact. + +### 4.3 `_module` — in memory, never on the graph (#140) + +Every project-owned node carries `_module` (its owning file key) *into* the builder. `node()` lifts +it **off** the props and into `NodeRow.module`. It is **never emitted as a graph property** — it only +groups rows for the bolt writer's per-module diff. Scope on the graph comes entirely from the +`can://` id prefix (see §6). + +### 4.4 Property flattening + +Neo4j properties must be scalars or homogeneous primitive arrays — no nested maps. The projector +flattens accordingly: + +- **`span`** → six integers: `start_line/end_line/start_column/end_column/start_byte/end_byte` + (spellings adopted verbatim from `codeanalyzer-java` under the parity clause). +- **`code`** is *derived at projection time*: `spanCode(source, span.bytes)` slices the module's + `source` by **UTF-8 byte offsets** via `sliceBytes` (a `Buffer` slice — string slicing would cut + multi-byte chars). Source is stored once per module (`:TSModule.source`); every narrower node's + `code` is a byte-slice of it (#179/#201). +- **maps → JSON strings** (sorted keys, to match Python and stay diffable): `keyword_arguments_json`, + `parameters_json`, `exports_json`, `entrypoint_report_json`. + +--- + +## 5. The graph schema (contract) — `schema.ts` + +`schema.ts` is the **single in-repo source of truth**. `--emit schema` serializes it to +`schema.json`, and the conformance test asserts the emitter never produces an undeclared label / +relationship / property. + +### 5.1 Merge labels & markers + +- Every `can://`-id-keyed node shares the MERGE label **`CanNode`** (one uniqueness constraint, + uniform edge endpoints) plus one **specific** `TS…` kind label. +- `:TSApplication` merges on bare **`Application`**. +- Neutral repo nodes merge on their own label: `Artifact`, `Package`, `ConfigKey`, `TSDecorator`. +- **Marker labels** `TSCanNode` / `JSCanNode` (`MARKER_LABELS`) ride every `can://` node as **index + anchors** — property indexes are label-scoped, so `id STARTS WITH $prefix` needs a label to seek on. + `TSCanNode` rides every node this analyzer writes; `JSCanNode` is a secondary consumer filter on the + `javascript` namespace. They carry *no* safety claim of their own. + +### 5.2 Node labels + +| Label | Merge on | Notes | +|---|---|---| +| `TSApplication` | `Application` | root: id, name, schema_version, language, max_level, k_limit, analyzer_name/version, entrypoint report | +| `TSModule` | `CanNode` | per file: name(=fileKey), is_tsx, is_declaration_file, content_hash, exports_json, **source** (whole file), span | +| `TSClass` / `TSInterface` / `TSEnum` / `TSTypeAlias` / `TSNamespace` | `CanNode` | type decls; `signature`, `code`, span, is_exported/is_ambient; TSClass also carries entrypoint flags, base_classes, implements_types | +| `TSCallable` | `CanNode` | functions/methods/ctor/getter/setter/arrow/fn-expr; signature, return_type, cyclomatic_complexity, accessibility, is_async/generator/static/…, code, parameters_json, entrypoint flags | +| `TSAnonymousCallable` | `CanNode` | marker label **alongside** `:TSCallable` for unnamed arrows/fn-exprs (#92) — keeps old `MATCH (:TSAnonymousCallable)` queries working | +| `TSField` | `CanNode` | name, type, span | +| `TSBodyNode` | `CanNode` | statement/call/entry/exit/formal_in/out/actual_in/out; `callee` on calls, `of`/`parent` on synthetics | +| `TSExternal` | `CanNode` | library/ghost targets; name, module | +| `Artifact` | `Artifact` | **neutral** — repo file inventory: path, format, roles, size_bytes, sha256, extraction, source | +| `Package` | `Package` | **neutral** — purl id, ecosystem, name | +| `ConfigKey` | `ConfigKey` | **neutral** — key, namespace, value, references | +| `TSDecorator` | `TSDecorator` | shared decorator target, merged on `qualified_name` so `@Get` and `@Get(':id')` collapse | + +### 5.3 Relationship types (grouped) + +- **Containment:** `TS_HAS_MODULE`, `TS_DECLARES`, `TS_HAS_METHOD`, `TS_HAS_FIELD`, `TS_HAS_BODY_NODE` +- **Call graph:** `TS_CALLS` (weight, prov), `TS_RESOLVES_TO` (body-call → callee) +- **Intra-callable overlays:** `TS_CFG_NEXT` (kind + `_k`), `TS_CDG`, `TS_DDG` (var, prov, `_k`), `TS_SUMMARY` (var) +- **Interprocedural (SDG):** `TS_PARAM_IN` (var), `TS_PARAM_OUT` (var) +- **Inheritance:** `TS_EXTENDS`, `TS_IMPLEMENTS` (resolved-only) +- **Decorators:** `TS_DECORATED_BY` (positional_arguments, keyword_arguments_json) +- **Imports/exports:** `TS_IMPORTS`, `TS_RE_EXPORTS` (one per module-pair, aggregated name arrays, type_only_names) +- **Repository layer:** `HAS_ARTIFACT`, `DECLARES_DEPENDENCY`, `LOCKS`, `TS_PROVIDES`, + `TS_UNRESOLVED_IMPORT`, `DEFINES_CONFIG`, `TS_USES_CONFIG`, `TS_READS_CONFIG_UNRESOLVED` (`_k=key|reason`) + +### 5.4 The `_k` relationship discriminant (#70) + +Some relationships legitimately need **multiple edges between the same endpoint pair**: a conditional's +true/false `TS_CFG_NEXT` pair, or several `TS_DDG` edges (one per `(var, prov)`). A plain +endpoint-pair `MERGE` would collapse them. `EdgeRow.key` sets a discriminant so the `MERGE` is on +`{_k: key}` and distinct edges coexist. Both writers honor it identically. + +### 5.5 DDL — constraints & indexes + +- **Constraints** are *derived* from node labels: one `IS UNIQUE` per distinct `(mergeLabel, key)` — + so they can never drift from the labels. +- **Indexes** are curated: `callable_name`, a fulltext `ts_code_fts` on `TSCallable.code` (mirrors + Python's `py_code_fts`), `cannode_kind`, and the two marker-anchored id range indexes + (`tscannode_id`, `jscannode_id`) that back every scoped/destructive statement. + +--- + +## 6. Scoping & safety (#140, #116) + +Every destructive statement is scoped on the **`can:///` id prefix**, never on a relationship +walk from the Application node. The prefix reaches *every* node the app owns (including ones a walk +would miss) and *nothing* another app owns — even one whose file keys collide. Helpers in `rows.ts`: + +- `applicationPrefix(appId)` — refuses anything that is not a bare `can://` id (an empty or + deeper id would silently widen or narrow the blast radius; it `throw`s instead). This is why a + hand-built `GraphRows` with no Application node gets **no** destructive statement. +- `descendantPrefix(canId)` — appends `/` so `foo.ts` is not a prefix of `foo.tsx`. +- The app is the **outermost** id segment, so one prefix spans both the `typescript` and `javascript` + language namespaces → one scoped statement covers both. + +Anchoring on `:TSCanNode` (an index) plus the `can:///` prefix is what keeps the wipe from ever +touching a **sibling analyzer's** graph (Python/Java tag `_module`, never apply `:CanNode`). + +--- + +## 7. The two writers + +### 7.1 Snapshot — `cypher.ts` → `graph.cypher` + +A self-contained, **non-incremental** script. Running it (`cypher-shell < graph.cypher`) rebuilds +this project's subgraph from scratch. Block order: + +1. **constraints & indexes** (`IF NOT EXISTS`, idempotent, run first so `MERGE` seeks an index). +2. **scoped wipe** — `MATCH (x:TSCanNode) WHERE x.id STARTS WITH '' DETACH DELETE x;` plus a + second `MATCH` for the root `:Application` itself (the root id `can://` is *outside* its own + descendant prefix). Rows with no `can://` application id emit **no wipe** (refused visibly). +3. **nodes** — grouped by full label set + keyProp, batched into `UNWIND […] MERGE … SET n += row.p` + (BATCH 500). Extra labels appended via `SET n:Extra`. +4. **relationships** — grouped by `(type, endpoint labels/keys, keyed?)`, `UNWIND … MATCH a … MATCH b + … MERGE (a)-[r:TYPE {_k?}]->(b) SET r += row.p`. + +Property values are rendered as Cypher **literals** (`cypherValue`/`cypherMap`, with string escaping). + +### 7.2 Incremental — `bolt.ts` → live DB + +Reads the DB's current state and updates **only what changed**, over Bolt. `neo4j-driver` is imported +**dynamically** so it stays off the JSON hot path. Algorithm: + +1. ensure constraints + indexes. +2. **schema-version gate** (#68): read `:Application.schema_version` *scoped to this app's id*; if it + differs from the producer's `SCHEMA_VERSION`, **force a full re-upsert** (`shouldForceFullUpsert`) + — a version change forces re-UPSERT, it never deletes (that is `--eager`'s job). +3. partition nodes by owning module (`NodeRow.module`); shared nodes (externals, neutral repo nodes) + have none and are **MERGE-only, always upserted**. +4. `--eager`? **`EAGER_PURGE`**: `DETACH DELETE` everything under `$prefix`, batched + `IN TRANSACTIONS OF 5000 ROWS` (one transaction would exhaust `transaction.total.max`, measured at + 2.7 GiB, #116). +5. **diff `content_hash`** per module (`:TSModule.content_hash`, keyed by module id under the prefix) + → the set of changed modules. +6. per changed module: on `--eager`, `PURGE_MODULE_EDGES` + `PURGE_MODULE_STALE` (delete owned edges + & vanished decls, keep surviving `$keys`), then upsert its nodes. **Without `--eager` the push only + adds/updates** — removing nodes is the operator's call. +7. upsert edges owned by a changed module (owner = source node's module) or shared. +8. **orphan prune** (`PRUNE_VANISHED`) — only on a **full run** *and* `--eager`: delete modules under + the prefix that this run no longer emits, with their subtrees. + +**Type fidelity:** `toParams` converts integer-valued numbers to `neo4j.int(...)` so the bolt and +snapshot writers agree on type (the JS driver otherwise stores every number as a float). + +--- + +## 8. Schema versioning & the conformance contract + +- `SCHEMA_VERSION = "2.0.0"`. **MAJOR** on a breaking change (renamed/removed label, relationship, + key); **MINOR** on additive. v2 was a MAJOR bump from v1 (keys moved signature → `can://` id; labels + reshaped; TS-namespaced vocabulary, #66). +- The repository-artifact layer is **additive within 2.0.0** — the neutral `:Artifact`/`:Package`/ + `:ConfigKey` labels and their edges were added without a version bump, because `SCHEMA_VERSION` + moves only when *every* sibling analyzer re-baselines together (contract 2.1.0 is pending that). +- **`test/neo4j-schema.test.ts`** is the anti-drift guard: it projects the `dataflow-app` fixture at + L4 and asserts the real emitter only ever produces labels / relationship types / properties that + `schema.ts` declares, and that the checked-in `schema.neo4j.json` is regenerated (`bun gen:schema`). +- Other Neo4j tests pin specific behaviors: `neo4j-prefix-scope`, `neo4j-edge-identity` (the `_k` + discriminant), `neo4j-bindings`, `neo4j-decorators`, `neo4j-bolt`, `bolt-version-gate`. + +Keep `schema.ts` and the JSON schema in lockstep, and keep the vocabulary parity-aligned with the +Python/Java backends — both are **contracts**, per CLAUDE.md. + +--- + +## 9. One-paragraph summary + +The Neo4j path is a **pure, deterministic projection** of the single schema-v2 CPG envelope. +`project()` walks the uniform tree into a deduped `GraphRows` bag with in-memory MERGE semantics; +one of two writers then realizes those rows — a self-contained idempotent `.cypher` snapshot, or an +incremental content-hash-diffed Bolt push. The vocabulary is TS-namespaced for a polyglot database +with a deliberate neutral-node exception for the repository layer, every destructive statement is +scoped on the `can:///` id prefix anchored to an index, and `schema.ts` is the single declared +contract that a conformance test holds the emitter to. diff --git a/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md b/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md new file mode 100644 index 0000000..3837bd5 --- /dev/null +++ b/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md @@ -0,0 +1,59 @@ + + +**Is your feature request related to a problem? Please describe.** + +The Neo4j projection (parent issue) needs a single declared source of truth for its graph +vocabulary before any projector or writer can be held to it. Without it, the emitter and the +contract drift and `--emit schema` has nothing to serialize. This is milestone **M1** of the +roadmap — the foundation every later milestone depends on. + +**Describe the solution you'd like** + +A new `internal/neo4j/schema.go` (sibling to `internal/syntactic_analysis/v2emit/`, off the +JSON hot path) that is the declarative contract: + +- [ ] Node labels + merge keys + typed properties — `:GoApplication`, `:GoModule`, `:GoType` + (`kind` property), `:GoCallable`, `:GoField`, `:GoBodyNode`, `:GoExternal`, and the neutral + `:Artifact`/`:Package`/`:ConfigKey`. Shared `:GoSymbol` merge-label; `:GoCanNode` marker on + every `can://` node. +- [ ] Relationship types + endpoints — containment (`GO_HAS_MODULE`/`GO_DECLARES`/ + `GO_HAS_METHOD`/`GO_HAS_FIELD`/`GO_HAS_BODY_NODE`), `GO_CALLS`/`GO_RESOLVES_TO`, + `GO_EMBEDS`/`GO_SATISFIES`, `GO_IMPORTS`, the neutral repo edges, and `GO_PROVIDES`/ + `GO_UNRESOLVED_IMPORT`. +- [ ] The **reserved L3/L4 overlay** vocabulary declared now (zero rows until the levels land): + `GO_HAS_BODY_NODE`/`GO_CFG_NEXT`(`_k=kind`)/`GO_CDG`/`GO_DDG`(`_k=var,prov`)/`GO_PARAM_IN`/ + `GO_PARAM_OUT`/`GO_SUMMARY` + body-node statement kinds. +- [ ] `_k` relationship discriminants on `GO_CFG_NEXT`, `GO_DDG`, `DECLARES_DEPENDENCY`. +- [ ] **Derived DDL** — one uniqueness constraint per distinct `(mergeLabel, key)` (no second + list to keep in sync) + curated indexes incl. the `:GoCanNode` id range index for prefix seeks. +- [ ] `SCHEMA_VERSION = "2.0.0"` + `buildSchemaDocument()`; `--emit schema` serializes it to + `schema.neo4j.json`. +- [ ] A **conformance-test scaffold** (`schema_test.go`) that M3 fills: the real projector may + never emit an undeclared label / relationship / property. + +Full catalog + rationale: `docs/design/Neo4j_projection/specs/neo4j-projection.md` §3. + +**Gate (this PR must pass):** +- `--emit schema` writes a `schema.neo4j.json` that parses and lists every declared family. +- Derived-DDL test: exactly one constraint per distinct `(mergeLabel, key)`, no duplicates. +- `go build` + `go vet` clean; checked-in `schema.neo4j.json` regenerated and committed. + +**Describe alternatives you've considered** + +- *Hand-written constraint list* — rejected: derive constraints from the labels so a new label + brings its own constraint and the two cannot drift. +- *Defer the L3/L4 overlay vocabulary* — rejected: declaring it now (empty) keeps the later + L3/L4 train a pure projection-fill, not a graph-schema change. + +**Additional context** + +Parent: the Neo4j projection coordination issue under epic #94. Mirror the existing test idiom +(`TestGateL1_*`, `TestDeterminism_*` in `v2emit`) for the Neo4j gates. diff --git a/docs/design/Neo4j_projection/issues/child-m7-sdk-goanalysisbackend.md b/docs/design/Neo4j_projection/issues/child-m7-sdk-goanalysisbackend.md new file mode 100644 index 0000000..d6697a4 --- /dev/null +++ b/docs/design/Neo4j_projection/issues/child-m7-sdk-goanalysisbackend.md @@ -0,0 +1,59 @@ + + +**Is your feature request related to a problem? Please describe.** + +`cldk/analysis/` has no `go` package, so `CLDK(language="go").analysis(...)` has nothing to +call — the SDK cannot read a Go `analysis.json` or query a Go Neo4j graph. This is the +frontend rung (milestone **M7**) of the Neo4j projection, moving in lockstep with the +analyzer's v2 release. + +**Describe the solution you'd like** + +A net-new Go analysis package mirroring `cldk/analysis/python/{codeanalyzer,neo4j}/`: + +- [ ] `cldk/models/go/` — Go Pydantic models (`GoApplication`/`GoModule`/`GoType`/ + `GoCallable`/`GoField`/`GoBodyNode`/`GoExternal`) matching the v2 tree. +- [ ] `cldk/analysis/go/backend.py` — `GoAnalysisBackend(AnalysisBackend[...])`, the generic + ABC implementation, with `N="Go"`, `P="GO_"` (the graph's node/relationship prefixes). +- [ ] `cldk/analysis/go/go_analysis.py` — the **type-centric** `GoAnalysis` facade: + Tier-A lifecycle verbatim; `get_types`/`get_type`, `get_functions` (package-level), + `get_methods_of`/`get_method` (receiver methods), `get_fields`; `get_callers`/`get_callees` + keyed on the callable id / `(type_id, sig)`; repository-artifact getters + `get_source`. + **No** `get_classes` (Go has no class); functions kept distinct from methods. +- [ ] `cldk/analysis/go/codeanalyzer/` (in-process: run `cango`, read `analysis.json`) and + `cldk/analysis/go/neo4j/` (read-only Cypher over the `GO_`-prefixed graph). +- [ ] `GoBackend = Union[GoCodeAnalyzerConfig, Neo4jConnectionConfig]` + the `"go"` + `cache_subdir` key in `cldk/analysis/commons/backend_config.py`. + +**Scope this train (SDK3):** Tier A + L1/L2 leaf accessors + artifact getters + `get_source`. +**Omit** slicing / CFG / CDG / DDG / reachability — they need L3/L4 the analyzer does not +produce; absent per the ABC contract (no always-erroring surface). + +Full facade decisions: `codeanalyzer-go/docs/design/Neo4j_projection/specs/neo4j-projection.md` §4. + +**Gate (this PR must pass):** +- `GoAnalysisBackend` instantiates (all ABC abstractmethods implemented). +- `CLDK(language="go").analysis(project_path=…)` returns a populated type table + call graph + from a real Go module (in-process backend). +- Analyzer output validates against the Go Pydantic models; `N`/`P` match the graph prefixes. +- Pins the analyzer version **only after** the analyzer v2 release is cut (lockstep). + +**Describe alternatives you've considered** + +- *Class-centric facade reusing `get_classes`/`get_method(class, name)`* — rejected: "class" + is a lie for Go and conflates package-level functions with methods (the "module-as-class" + hack). Type-centric honors Go's real structure and sets the SDK's first class-less precedent. +- *Full Python-parity surface with L3/L4 stubs* — rejected: ships methods that always error. + +**Additional context** + +Built on the fork `lamwassi/python-sdk`. `Neo4jConnectionConfig` already exists in +`commons/backend_config.py` and in each `*Backend` union, so the Neo4j consumer side is +additive — the graph it reads is "populated out of band" by the analyzer's `--emit neo4j`. diff --git a/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md b/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md new file mode 100644 index 0000000..d301b06 --- /dev/null +++ b/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md @@ -0,0 +1,62 @@ + + +**Is your feature request related to a problem? Please describe.** + +Epic #94 brings `codeanalyzer-go` to canonical schema v2 "in both projections +(`analysis.json` + Neo4j)". The JSON projection (L1 + L2) is in flight; the **Neo4j +projection is not yet built**, and the `python-sdk` has no `go` analysis package, so a graph +database holding a Go CPG cannot be produced or queried. This issue coordinates the Neo4j +half of #94 across the two repos. + +**Describe the solution you'd like** + +A pure, identity-preserving **second projection** of the frozen v2 envelope into a labeled +property graph — keyed on the identical `can://` ids so JSON and graph are interchangeable — +plus the SDK consumer that reads it. Vocabulary is `GO_`/`Go`-namespaced for a polyglot +database, with the neutral `:Artifact`/`:Package`/`:ConfigKey` layer reused verbatim. + +**Spec (full design + decisions — do not paste, link):** +`codeanalyzer-go/docs/design/Neo4j_projection/specs/neo4j-projection.md` +**Roadmap (milestones, commit boundaries, gates):** +`codeanalyzer-go/docs/design/Neo4j_projection/roadmap.md` + +Children (filed just-in-time on their repos, attached as sub-issues of this one): + +- [ ] **M1–M6 analyzer `--emit neo4j`** (`codeanalyzer-go`) — `internal/neo4j/` + {schema, rows, project, cypher, bolt, emit}.go + CLI wiring. May be one issue with the + milestones as a checklist, or a sub-stack per PR if reviewers prefer finer PRs. +- [ ] **M7 `GoAnalysisBackend`** (`python-sdk` / fork `lamwassi/python-sdk`) — type-centric + Go facade + Go models + `GoBackend` union; pins the analyzer version only after the + analyzer release is cut (lockstep). +- [ ] Docs + CHANGELOG (`finishing-cldk-work`). + +**Locked design decisions** (full table in the spec): +`N="Go"`/`P="GO_"`; one `:GoType` + `kind` property; `GO_EMBEDS` vs `GO_SATISFIES`; +flattened `source_file`/`error_channel`/`is_goroutine`/`is_deferred`; narrower label set (no +decorator/attribute/variable); neutral repo layer reused verbatim; full L3/L4 overlay +vocabulary declared now, zero rows until the levels land; type-centric SDK facade. + +**Describe alternatives you've considered** + +- *Block the projection on L3/L4 first* — rejected: `--emit neo4j` ships now at full-depth = + max implemented (L2) and grows additively; the reserved overlay vocabulary means L3/L4 fill + it with no schema change. +- *Per-kind type labels (like TS)* — rejected: re-introduces at the graph layer the per-kind + split the JSON schema deliberately collapsed into one `kind` field; breaks projection symmetry. +- *Class-centric SDK facade (reuse `get_classes`)* — rejected: Go has no class; the facade is + type-centric (`get_types`/`get_functions`/`get_methods_of`), the SDK's first class-less surface. + +**Additional context** + +Scope guard: the analyzer is a pure graph **provider** of the SDG substrate — **no +`taint_flows`**; labeled reachability/slicing/source-sink are SDK Cypher queries, a later +train. Release plan (per #94): "Neo4j projection and the JSON schema move in lockstep." diff --git a/docs/design/Neo4j_projection/prompt.md b/docs/design/Neo4j_projection/prompt.md new file mode 100644 index 0000000..5ca3dee --- /dev/null +++ b/docs/design/Neo4j_projection/prompt.md @@ -0,0 +1,85 @@ +# Neo4j Projection for `codeanalyzer-go` — Prompt + +## Context & Objective + +We are implementing the **Neo4j projection backend** for `codeanalyzer-go`. This is the +Go-language sibling of two existing, battle-tested implementations: `codeanalyzer-typescript` +(`src/build/neo4j/*`) and `codeanalyzer-python` (`codeanalyzer/neo4j/*`). Our implementation +must preserve **vocabulary parity** with those backends while adopting a `GO_`-prefixed, +`GoCanNode`-anchored namespace for the polyglot database. + +## What the Neo4j backend is + +The Neo4j backend is a **pure, identity-preserving, second projection** of the analyzer's +canonical **schema-v2 CPG envelope** — the *same* envelope the JSON path emits verbatim to +`analysis.json`. Both projections key on the *same* `can://` ids (and global-ordinal ids for +body-level nodes), so JSON and graph are interchangeable and can never diverge. The projection +is **lossless up to re-expression**: a JSON field may be re-expressed as an edge or a flattened +scalar property, but never silently dropped. + +## Provider/client boundary + +The analyzer is strictly a graph **provider**: it emits the **SDG substrate** only (containment +tree, call graph, CFG/CDG/DDG, interprocedural PARAM_IN/PARAM_OUT, inheritance, imports/exports, +and the language-neutral repository-artifact layer). There is **no `taint_flows` section** — +labeled reachability, slicing, and source/sink semantics belong to the CLDK frontend SDK, as +Cypher traversal queries. + +## Three design invariants (non-negotiable) + +1. **Two-projection agreement** — graph nodes carry the identical `can://` ids as the JSON tree; + no id recomposition, built once upstream. +2. **Monotone / additive-safe writes** — the default incremental (lazy) Bolt push is purely + MERGE-upsert and never deletes; destructive reconciliation is opt-in under `--eager` only. +3. **Prefix-scoped destructive statements** — every destructive statement is scoped on the + `can:///` id prefix, anchored on the `GoCanNode` index label, so one application can never + corrupt another's subgraph in a shared, multi-language database. + +## Architecture to replicate (pure projection, dumb writers) + +Mirror the proven module split from the TS/Python references: + +- **Declarative schema contract** (`schema.go`) as the single in-repo source of truth: node labels + + merge keys + typed properties, relationship types + endpoints, **derived** DDL (one uniqueness + constraint per distinct `(mergeLabel, key)`), curated indexes, and `SCHEMA_VERSION`. Backed by an + anti-drift **conformance test** that asserts the emitter never produces an undeclared + label/relationship/property, plus `--emit schema` serialization. +- **Pure projector** (`project.go`): walks the uniform schema-v2 IR → emits a deterministic, deduped + `GraphRows` bag via a `RowBuilder` with in-memory MERGE semantics (node dedup on + `(labels[0], value)`, deferred *edge-only-when-resolved* edges, `_k` relationship discriminant for + legitimately-parallel edges, null-pruning, scalar/`*_json` property flattening). **No I/O, no + driver.** +- **Output-agnostic row IR** (`rows.go`): `GraphRows` / `NodeRow` / `EdgeRow` / `NodeRef`, the + in-memory-only `module` grouping field, scoping helpers (`applicationPrefix`, + `descendantPrefix`), and Cypher-literal rendering. +- **Two writers** consuming the *identical* rows: a **snapshot writer** (`cypher.go` → + self-contained idempotent `graph.cypher`) and an **incremental Bolt writer** (`bolt.go` → + content-hash-diffed, per-module replacement over a live DB, with the schema-version gate and + `--eager` purge/prune behavior). + +## CLI surface + +Parity with TS/Python: `--emit json|neo4j|schema`, `--neo4j-uri/-user/-password/-database`, +`--eager`/`--lazy` (lazy default). `--emit neo4j` is **always full-depth (L4)**; combining it with +`-a`/`--graphs` is a flag error. + +## References & skill + +Leverage the existing design/dataflow references at +[`docs/design/Neo4j_projection/Python_neo4j_design_and_dataflow.md`](docs/design/Neo4j_projection/Python_neo4j_design_and_dataflow.md) +and +[`docs/design/Neo4j_projection/Typescript_neo4j_design_and_dataflow.md`](docs/design/Neo4j_projection/Typescript_neo4j_design_and_dataflow.md), +and apply the `codeanalyzer-backend` skill's Neo4j-projection reference: +`https://github.com/codellm-devkit/cldk-devtools/blob/main/skills/codeanalyzer-backend/references/neo4j-projection.md`. + +## Engineering constraints + +Decompose the work into **small, self-contained, independently reviewable commits** — each commit +should land one coherent, tested unit (e.g., schema contract + conformance test, then the row IR, +then the pure projector, then each writer) so a reviewer can evaluate it in isolation. + +## Deliverable + +Produce a **practical, sequenced implementation roadmap** for the Go Neo4j projection — ordered +milestones mapped to the above modules, with the commit boundaries, dependencies, and the test gate +each milestone must pass. diff --git a/docs/design/Neo4j_projection/roadmap.md b/docs/design/Neo4j_projection/roadmap.md new file mode 100644 index 0000000..3a209a0 --- /dev/null +++ b/docs/design/Neo4j_projection/roadmap.md @@ -0,0 +1,201 @@ +# CLDK Go analyzer — Neo4j projection roadmap + +> Sequenced implementation roadmap for the Go Neo4j projection (collision group **D** of the +> v1→v2 migration, now in design). Companion to the spec `specs/neo4j-projection.md` (the +> *what* + decisions) — this is the *how*: ordered milestones → modules, commit boundaries, +> dependencies, and the test gate each milestone must pass. +> +> Tracked under epic `codellm-devkit/.github#94`. Cross-repo: analyzer in `codeanalyzer-go`, +> SDK facade on the `lamwassi/python-sdk` fork. Golden rule (inherited): **pure projection, +> dumb writers** — graph *content* has exactly one definition (`project.go`); writers differ +> only in liveness. + +## Initiative: Neo4j projection (`codeanalyzer-go` + `python-sdk`) + +A **second projection** of the frozen v2 envelope into a labeled property graph, keyed on +the identical `can://` ids — JSON and graph interchangeable, never divergent. Vocabulary is +`GO_`/`Go`-namespaced for a polyglot database, with the neutral `:Artifact`/`:Package`/ +`:ConfigKey` layer reused verbatim as cross-language merge targets. **Full-depth = max +implemented** (L2 today); the L3/L4 overlay vocabulary is declared now and fills additively. + +## Dependency order (DAG) + +``` +M1 schema.go (contract + conformance test + --emit schema) + │ (the declared vocabulary everything below is held to) + ▼ +M2 rows.go (GraphRows + RowBuilder: MERGE-dedup, deferred edges, prune, scoping, literals) + │ (the IR both writers consume) + ▼ +M3 project.go (pure projector: v2.Analysis tree → GraphRows) ◄── the intellectual core + │ + ├─────────────────────────────┬──────────────────────────────┐ + ▼ ▼ ▼ +M4 cypher.go (snapshot writer) M5 bolt.go (incremental writer) │ + └──────────────┬──────────────┘ │ + ▼ │ +M6 emit.go facade + CLI wiring (dispatch --emit neo4j|schema, flag-error gate) + │ │ + ▼ ▼ +M7 python-sdk fork: GoAnalysisBackend + Go models + GoBackend union + (pins analyzer version only after the analyzer release is cut) +``` + +M4 and M5 both depend only on M3 (the rows), and are independent of each other — they can +land in either order or in parallel. M7 depends on a cut analyzer release (lockstep). + +## Milestones, commit boundaries, and gates + +Each milestone is **one coherent, independently reviewable PR** (per the prompt's commit +discipline). A milestone may be split into the listed commits within its PR, or filed as its +own sub-issue if a reviewer prefers finer PRs — the boundaries below are the natural seams. + +### M1 — Schema contract + conformance test + `--emit schema` +**Module:** `internal/neo4j/schema.go` (+ `schema_test.go`). +**Builds:** the declarative single-source-of-truth — node labels + merge keys + typed +properties, relationship types + endpoints, **derived** DDL (one uniqueness constraint per +distinct `(mergeLabel, key)` + curated indexes incl. the `:GoCanNode` id range index), +`SCHEMA_VERSION = "2.0.0"`, and `buildSchemaDocument()` serialization. Declares the full +reserved L3/L4 overlay vocabulary (decision L1) even though nothing emits it yet. +**Depends on:** nothing (pure declaration). +**Commits:** (a) label/relationship/property declarations + `_k` discriminants; (b) derived +DDL + indexes; (c) `buildSchemaDocument()` + `emit_schema()` wiring to `schema.neo4j.json`. +**Gate:** +- `--emit schema` writes a `schema.neo4j.json` that parses and lists every declared family. +- A **conformance scaffold** test exists (asserts-nothing-yet placeholder that M3 fills) and + the derived-DDL test: exactly one constraint per distinct `(mergeLabel, key)`, no dupes. +- `go build` + `go vet` clean; the checked-in `schema.neo4j.json` is regenerated. + +### M2 — Row IR + `RowBuilder` +**Module:** `internal/neo4j/rows.go` (+ `rows_test.go`). +**Builds:** `GraphRows`/`NodeRow`/`EdgeRow`/`NodeRef`; the `RowBuilder` in-memory-MERGE +accumulator (`node()` dedup on `(labels[0], value)` with prop last-write-wins + label union; +append-only `edge()`; **deferred** `edgeToSymbol()` kept at `finish()` only if the target was +emitted); `prune()` (null → absence); the in-memory-only `module` grouping field; +`applicationPrefix`/`descendantPrefix` scoping helpers (refuse a non-bare app id); Cypher +literal/`*_json` rendering; deterministic `finish()` sort. +**Depends on:** M1 (references the declared labels/keys; no runtime dep). +**Commits:** (a) data types + `NodeRef`; (b) `RowBuilder` MERGE/dedup + prune; (c) deferred +edges + `finish()` sort; (d) scoping helpers + literal rendering. +**Gate:** +- Unit tests: node re-see merges props + unions labels; deferred edge dropped when target + absent, kept when present; `prune()` drops nulls; `applicationPrefix` throws on a non-bare + id (so no-root rows get no wipe); `finish()` output is order-stable. + +### M3 — Pure projector ◄ the intellectual core +**Module:** `internal/neo4j/project.go` (+ `project_test.go`), fills M1's conformance test. +**Builds:** `project(analysis) → GraphRows` walking the v2 tree in spine order: application +root → per module (`GO_HAS_MODULE`, source/content_hash) → types (`:GoType` + `kind` prop, +`GO_EMBEDS` deferred, `GO_SATISFIES`, fields via `GO_HAS_FIELD`) → callables (`GO_DECLARES`/ +`GO_HAS_METHOD`, `error_channel[]`/`source_file` props, closures) → body call nodes +(`GO_HAS_BODY_NODE`, `is_goroutine`/`is_deferred` props, `GO_RESOLVES_TO`) → `call_graph` +(`GO_CALLS`) → imports (`GO_IMPORTS` → module or external ghost) → neutral artifact/package +layer (rows only for facts present) → reserved overlay sections (zero rows at L2). Endpoints +resolve to a declared id or an id-keyed `:GoExternal` ghost — never dangle. +**Depends on:** M1 + M2. +**Commits:** (a) app + module + type walk; (b) callable + body + closures; (c) call_graph + +imports + external ghosts; (d) neutral artifact/package layer + reserved-overlay no-ops. +**Gate (the heart of the suite):** +- **Conformance** (`TestNeo4jSchema_Conformance`): the projector never emits a label / + relationship / property undeclared in `schema.go` — run over the `multipackage` + `chi` + fixtures at L2. +- **No dangling** (`TestNeo4j_NoDangling`): every edge endpoint is an emitted node. +- **Two-projection agreement** (`TestNeo4j_IdParity`): every graph `can://`/ordinal id has a + matching JSON-tree node and vice-versa (modulo containment edges) — cross-checks against + `v2emit.Emit` output on the same fixture. +- **Go-specific** gates: `:GoType.kind` carries the four kinds; `GO_EMBEDS` vs `GO_SATISFIES` + split correct; `source_file` prop present only on a cross-file method; `is_goroutine` set + on a `go`-statement call node; reserved overlay sections emit zero rows at L2. + +### M4 — Snapshot writer +**Module:** `internal/neo4j/cypher.go` (+ `cypher_test.go`). +**Builds:** `GraphRows` → self-contained idempotent `graph.cypher`: constraints + indexes +(`IF NOT EXISTS`) → **scoped wipe** (`MATCH (x:GoCanNode) WHERE x.id = OR x.id +STARTS WITH 'can://go//' DETACH DELETE … IN TRANSACTIONS`) → batched `UNWIND … MERGE` +nodes (grouped by label-set + key) → batched `UNWIND … MERGE` edges (grouped by type + +endpoint labels/keys + `_k`?). +**Depends on:** M3. +**Commits:** (a) DDL + scoped-wipe block; (b) node batches; (c) edge batches + literal escaping. +**Gate:** +- `--emit neo4j` (no `--neo4j-uri`) writes `graph.cypher`; **determinism** test: byte-identical + across runs and `-j` values. +- **Idempotency** (`TestNeo4j_CypherIdempotent`): running the script twice against an empty + Neo4j (containerized, build-tag-gated) yields the same node/edge counts — or, DB-free, a + golden-file assertion on the emitted script. +- **Prefix-scope** (`TestNeo4j_PrefixScope`): no-root rows emit no wipe; the wipe predicate is + anchored on `:GoCanNode` and the `can://go//` prefix only. + +### M5 — Incremental Bolt writer +**Module:** `internal/neo4j/bolt.go` (+ `bolt_test.go`). +**Builds:** content-hash-diffed per-module push over a live DB (lazy Bolt driver dependency, +off the JSON path): ensure DDL → **schema-version gate** (read `:GoApplication.schema_version` +scoped to this app; mismatch forces full re-upsert, never delete) → partition nodes by owning +`module` (shared nodes MERGE-only) → diff each module's `content_hash` → per changed module +upsert; **`--eager`** adds the destructive steps (purge module edges/stale decls, orphan-prune +vanished modules, batched `IN TRANSACTIONS`). Default `--lazy` is purely additive. +**Depends on:** M3 (consumes the identical rows M4 does). +**Commits:** (a) driver bootstrap + DDL ensure + version gate; (b) module diff + additive +upsert (lazy); (c) `--eager` purge/prune. +**Gate:** +- `go build` with and without the Bolt build tag; driver import stays off the JSON hot path. +- **Additive-by-default** test: a `--lazy` push issues no `DELETE`. +- **Scoped-destruction** test: every `--eager` destructive statement carries the + `can://go//` prefix predicate. +- Live-DB integration (containerized, build-tag-gated): edit one file → second push touches + only that module's subgraph. + +### M6 — Facade + CLI wiring +**Modules:** `internal/neo4j/emit.go`; `cmd/codeanalyzer/main.go`; `internal/core`. +**Builds:** `emit_neo4j(analysis, opts)` = project → branch on `--neo4j-uri` (set → bolt; +absent → cypher); `emit_schema(opts)`. Replace the two "not yet implemented" error stubs in +`main.go` with real dispatch; enforce **`--emit neo4j` + `-a`/`--graphs` → flag error (exit +2)**; force full depth; keep the existing env-precedence flag plumbing. +**Depends on:** M4 + M5. +**Commits:** (a) facade; (b) CLI dispatch + flag-error gate + full-depth forcing. +**Gate:** +- CLI tests (mirror `cmd/codeanalyzer/main_test.go`): `--emit schema` needs no `--input`; + `--emit neo4j -a 1` exits 2 with the parity error message; `--emit neo4j` without a URI + writes `graph.cypher`. +- End-to-end: `cango -i testdata/multipackage --emit neo4j` produces a graph; full existing + suite stays green. + +### M7 — SDK `GoAnalysisBackend` (fork `lamwassi/python-sdk`) +**Modules:** `cldk/analysis/go/{__init__,backend,go_analysis}.py` + `codeanalyzer/` + +`neo4j/`; `cldk/models/go/`; `GoBackend` union + `cache_subdir` key in +`commons/backend_config.py`. +**Builds:** `GoAnalysisBackend(AnalysisBackend[...])` with `N="Go"`, `P="GO_"`; the +type-centric `GoAnalysis` facade (SDK1: `get_types`/`get_type`/`get_functions`/ +`get_methods_of`/`get_method`/`get_fields`; SDK2: id-keyed `get_callers`/`get_callees`; +Tier A verbatim); Go Pydantic models mirroring the v2 tree; both the in-process (`analysis.json`) +and read-only Neo4j backends. +**Depends on:** a cut analyzer v2 release (lockstep — pin the analyzer version only after). +**Commits:** (a) Go models; (b) `GoAnalysisBackend` ABC impl + `GoBackend` union; (c) +`GoAnalysis` facade; (d) neo4j read backend. +**Gate:** +- `GoAnalysisBackend` instantiates (all ABC abstractmethods implemented). +- Round-trip: `CLDK(language="go").analysis(project_path=…)` returns a populated type table + + call graph from a real Go module (in-process backend). +- Analyzer output validates against the Go Pydantic models; `N`/`P` match the graph prefixes. + +## Release trains + +- **Train 1 — analyzer `--emit neo4j` (part of the v2 line).** M1–M6. Lands behind the + existing flag stubs; produces the graph at full-depth = L2, grows additively at L3/L4 with + no contract change. Ships with the analyzer v2 major. +- **Train 1′ — SDK Go wiring (lockstep).** M7 on the fork; pins the analyzer version **only + after** the analyzer release is cut (the SDK's Go models can't parse v2 until then, and + there's no Go wiring yet — net-new, not a migration). Per epic #94: "Neo4j projection and + the JSON schema move in lockstep." + +## Not now (considered, deliberately excluded) + +- **`taint_flows` / slicing / reachability / source-sink** — the analyzer is a pure graph + **provider**; labeled traversal is the SDK's job as Cypher queries, a later train. No + analyzer surface for it. +- **L3/L4 overlay population** — the vocabulary is declared (decision L1) but emits zero rows + until the deferred L3/L4 analysis train lands. When it does, `project.go` fills the reserved + sections with **no `schema.go` change**. +- **Go-specific `cfg` edge kinds** (`defer_resume`, `select`) — recorded when L3 is designed. +- **Struct-tag modeling** — Go has struct tags; v2 doesn't model them yet. Not forced in here. +- **TS-SDK `GoAnalysis` encoding** — the facade vocabulary is designed once (M7 Python + encoding); the TS mirror follows when the TS SDK catches up. diff --git a/docs/design/Neo4j_projection/specs/neo4j-projection.md b/docs/design/Neo4j_projection/specs/neo4j-projection.md new file mode 100644 index 0000000..e5b1c8f --- /dev/null +++ b/docs/design/Neo4j_projection/specs/neo4j-projection.md @@ -0,0 +1,229 @@ +# Spec: `codeanalyzer-go` Neo4j projection (+ `python-sdk` `GoAnalysisBackend`) + +> **Status:** design spec, 2026-10-04. Produced by the `designing-cldk-changes` design +> loops (analyzer-side schema loop + SDK-facade loop), each divergence decided with the user. +> **Canonical home:** `codellm-devkit/.github → docs/design/specs/2026-10-04-codeanalyzer-go-neo4j-projection.md` +> — staged here on the fork because `lamwassi` has pull-only access upstream; a maintainer +> ports it. **Tracked under epic** `codellm-devkit/.github#94` (`codeanalyzer-go → canonical +> schema v2`), whose summary already scopes "both projections (analysis.json + Neo4j)" and +> lists the Neo4j backend rung + SDK frontend rung. +> +> This is the *what* (the contract + decisions). The sequenced *how* is the sibling +> `../roadmap.md`. The deep-internals references are `../Python_neo4j_design_and_dataflow.md` +> and `../Typescript_neo4j_design_and_dataflow.md`. + +--- + +## 1. Contract-Impact Triage + +**Does this change the schema v2 JSON output?** No. The Neo4j backend is a **second +projection of the frozen v2 envelope** the JSON path already emits, keyed on the identical +`can://` ids and `@line:col` ordinal ids — JSON and graph join on one string, built once +upstream, never recomposed. No new JSON node/field/id shape. + +**Does it introduce a new contract?** Yes — two: +1. the **graph schema** (`schema.neo4j.json`: node labels, relationship types, merge keys, + derived DDL, `SCHEMA_VERSION`), under the cross-language **parity clause**; and +2. the **SDK facade surface** — a net-new `GoAnalysisBackend` (the `cldk/analysis/go` + package does not exist today). + +Both are cross-language shared vocabulary → structural, designed here before any rung. + +**Affected repos** (matches epic #94's affected-repos list verbatim): + +| Repo | Rung | Change | +| --- | --- | --- | +| `codeanalyzer-go` | backend | `internal/neo4j/` projection package + CLI wiring + conformance/scope tests | +| `python-sdk` (fork `lamwassi/python-sdk`) | frontend | `cldk/analysis/go/{codeanalyzer,neo4j}/`, `GoAnalysisBackend`, `GoBackend` union, Go Pydantic models | +| docs | finishing | README `--emit neo4j` section + graph-vocabulary reference | + +No other SDK and no `taint_flows` section: the analyzer is a **pure graph provider** of the +SDG substrate; labeled reachability / slicing / source-sink are SDK Cypher queries, a later +train. + +--- + +## 2. Three invariants (inherited verbatim from the siblings — not re-litigated) + +1. **Two-projection agreement.** Graph nodes carry the identical `can://` ids as the JSON + tree; body-level nodes carry the identical global-ordinal ids (`@`). + No id recomposition in the projector — it reads ids off the stamped tree. +2. **Monotone / additive-safe writes.** The default `--lazy` Bolt push is pure MERGE-upsert + and never deletes. Destructive reconciliation (purge/prune) is opt-in under `--eager`. +3. **Prefix-scoped destruction.** Every destructive statement is scoped on the + `can:///` id prefix, anchored on the `:GoCanNode` index label — one application can + never corrupt another's subgraph in a shared, multi-language database. + +Architecture inherited verbatim: **pure projection, dumb writers.** `project.go → rows.go` +is a deterministic, deduped `GraphRows` bag with no I/O and no driver; two writers +(`cypher.go` snapshot, `bolt.go` incremental) consume the identical rows. `schema.go` is the +single declared contract a conformance test holds the emitter to. + +--- + +## 3. Analyzer-side design decisions (the graph-vocabulary loop) + +Decided node-by-node with the user; each is a `GO_`/`Go` **leaf** under the parity clause, +or a verbatim reuse of shared vocabulary. "Ref" names the sibling precedent. + +| # | Decision | Rationale / divergence from refs | +| --- | --- | --- | +| **N1** | Node-label prefix **`Go`**, relationship prefix **`GO_`** | Parity with `Py`/`PY_`, `TS`/`TS_`. The SDK `AnalysisBackend` ABC reads these as `N`/`P` ClassVars → `GoAnalysisBackend` sets `N="Go"`, `P="GO_"`. Must agree letter-for-letter. | +| **N2** | `:GoCanNode` marker on every `can://` node; `:GoSymbol` shared merge-label on `:GoType`/`:GoCallable`/`:GoExternal` | Index anchor for prefix seeks (Ref: `PyCanNode`, `CanNode`); one uniqueness constraint + one MATCH-by-id for the symbol family (Ref: `PySymbol`). | +| **T1** | **One `:GoType`** label, `kind ∈ struct\|interface\|alias\|defined` as a **property** | Mirrors the JSON `type.kind` collapse (CLAUDE.md D4). Python has one `:PyClass`; TS has **per-kind labels**. Go deliberately does **not** re-introduce a per-kind split the JSON schema folded away — keeps the two projections symmetric. | +| **T2** | `base_types[]` → **`GO_EMBEDS`** (deferred, edge-only-when-resolved); `interfaces[]` → **`GO_SATISFIES`** (computed) | Go's two mechanisms are genuinely different (explicit embedding = composition; structural satisfaction = implicit). Refs have only `*_EXTENDS`/`*_IMPLEMENTS` for nominal inheritance; reusing those names would misdescribe both. New `GO_` leaves, not renames. | +| **C1** | `GO_DECLARES` (module→func, callable→closure) + `GO_HAS_METHOD` (type→receiver-method) | Parity with `PY_DECLARES`/`PY_HAS_METHOD`. `GO_HAS_METHOD` marks the receiver-type containment that defines a Go method. | +| **C2** | `source_file` → **scalar property** on `:GoCallable` (present only when ≠ nesting module) | The cross-file-method case (CLAUDE.md D8) is Go-only; no ref precedent. Text recovery joins to the declaring `:GoModule` by that path. No new edge/label for a value-shaped fact. | +| **B1** | `is_goroutine` / `is_deferred` → **boolean props** on the call `:GoBodyNode` (pruned when false) | Net-new vs refs (CLAUDE.md D7). Modeled as the JSON models them. Not labels, not edge discriminants — the fact lives on the call site even when the callee is unresolved and no `GO_RESOLVES_TO` edge exists. | +| **F1** | `:GoField` node, id `/`, via `GO_HAS_FIELD` | Ref: TS `:TSField`/`TS_HAS_FIELD`, Python `:PyAttribute`. Id minted from the owner type's `can://` id so prefix-purge reaches it. | +| **X1** | **No** `:GoDecorator` / `:GoAttribute` / `:GoVariable` (and their edges) | Go's JSON tree has none. Parity adds at the leaves; it does not force a node a language lacks. The Go label set is legitimately narrower. (Struct tags deferred.) | +| **CG1** | `call_graph` → **`GO_CALLS`** (weight, prov); `error_channel[]` → **string-array prop** on `:GoCallable` | `GO_CALLS` parity with `PY_CALLS` (endpoints = callable id or `:GoExternal` ghost; no dangling). Go errors are returned values, not thrown types — a value-shaped fact rides the node, not a `GO_RAISES` edge. | +| **I1** | `GO_IMPORTS` (aggregated per target) → `:GoModule` (in-project) or `:GoExternal` ghost (stdlib/external, keyed by import path) | Parity with aggregated `PY_IMPORTS` + edge-only-when-resolved. Go imports are overwhelmingly external → ghost targets, never dangling, never minted as `:Package` (packages come from dependencies, not imports). | +| **A1** | Declare neutral `:Artifact`/`:Package`/`:ConfigKey` + `HAS_ARTIFACT`/`DECLARES_DEPENDENCY`(`_k=kind`)/`LOCKS`/`DEFINES_CONFIG` **verbatim**; project only facts the analyzer produces | These are the cross-language **merge targets** — reused unchanged so a sibling over the same repo converges. go.mod deps → `:Package` `pkg:golang/…`; absent facts → **zero rows** (declared contract, empty projection). `GO_PROVIDES`/`GO_UNRESOLVED_IMPORT` stay `GO_`-prefixed (this analyzer's claims). | +| **L1** | Declare the **full L3/L4 overlay vocabulary now**; projector emits **zero rows** until the levels land | `GO_HAS_BODY_NODE`, `GO_CFG_NEXT`(`_k=kind`), `GO_CDG`, `GO_DDG`(`_k=var,prov`), `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` + body-node statement kinds. `--emit neo4j` = "full-depth = max implemented" (L2 today). The contract is declared once; L3/L4 fill it with **no schema change**. | +| **K1** | `_k` discriminants: `GO_CFG_NEXT` per `kind`; `GO_DDG` per `(var,prov)`; `DECLARES_DEPENDENCY` per `kind` | Same mechanism as the siblings — parallel edges survive MERGE instead of collapsing. | +| **V1** | `SCHEMA_VERSION = "2.0.0"`, stamped on `:GoApplication`; additive-MINOR suspended for the 2.0.0 line | Parity with the sibling versioning ruling; consumers gate on the analyzer-version floor. | + +### Node-label catalog (derived from the decisions above) + +| Label | Merge label | Key | Notes | +| --- | --- | --- | --- | +| `:GoApplication` | `GoApplication` | `id` | root `can://go/`; schema_version, analyzer name/version, max_level | +| `:GoModule` | `GoModule` | `id` | per file: package, **source** (whole file), content_hash, span | +| `:GoType` | `GoSymbol` | `id` | `kind ∈ struct\|interface\|alias\|defined` (property), signature, span | +| `:GoCallable` | `GoSymbol` | `id` | signature, return_type, `error_channel[]`, `source_file?`, metrics, span | +| `:GoField` | `GoField` | `id` (`/`) | type, span | +| `:GoBodyNode` | `GoBodyNode` | `id` (global ordinal) | `kind` (`call` at L1), `is_goroutine?`, `is_deferred?`, span | +| `:GoExternal` | `GoSymbol` | `id` | library/stdlib ghost; name, import path | +| `:Artifact` | `Artifact` | `id` | **neutral** | +| `:Package` | `Package` | `id` (purl) | **neutral** | +| `:ConfigKey` | `ConfigKey` | `id` | **neutral** | + +Every `can://`-keyed node also carries `:GoCanNode`. + +### Relationship catalog + +- **Containment:** `GO_HAS_MODULE`, `GO_DECLARES`, `GO_HAS_METHOD`, `GO_HAS_FIELD`, `GO_HAS_BODY_NODE` +- **Call graph:** `GO_CALLS` (weight, prov), `GO_RESOLVES_TO` (body-call → callee) +- **Inheritance:** `GO_EMBEDS` (deferred/resolved-only), `GO_SATISFIES` +- **Imports:** `GO_IMPORTS` (aggregated) +- **Repository (neutral):** `HAS_ARTIFACT`, `DECLARES_DEPENDENCY` (`_k`), `LOCKS`, `DEFINES_CONFIG`; (`GO_`) `GO_PROVIDES`, `GO_UNRESOLVED_IMPORT` +- **Reserved L3/L4 overlay (zero rows until the level lands):** `GO_CFG_NEXT` (`_k`), `GO_CDG`, `GO_DDG` (`_k`), `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` + +### CLI surface (parity with TS/Python) + +`--emit json|neo4j|schema` · `--neo4j-uri/-user/-password/-database` · `--eager`/`--lazy` +(lazy default). `--emit neo4j` is **always full-depth** (= max implemented level); combining +it with `-a`/`--graphs` is a **flag error** (exit 2). `--emit schema` needs no `--input`. +The options surface and flags already exist (`internal/options/options.go`, +`cmd/codeanalyzer/main.go`) with env-var precedence `explicit > env > default`; the +dispatch currently returns "not yet implemented" and is wired up in Milestone 6. + +--- + +## 4. SDK-facade design decisions (the `GoAnalysisBackend` loop) + +Go is the SDK's **first class-less facade** (no C/procedural precedent exists in `python-sdk` +yet; the live anchors are the class-centric Java + Python facades). Tier A (lifecycle: +`get_application_view`, `get_symbol_table`, `get_call_graph` → `nx.DiGraph`, +`get_call_graph_json`, `get_callers`, `get_callees`, `get_class_hierarchy`-equivalent) is +**invariant — reproduced verbatim**, not re-litigated. + +| # | Decision | Rationale | +| --- | --- | --- | +| **SDK1** | **Type-centric** leaf surface: `get_types`/`get_type`, `get_functions` (package-level), `get_methods_of`/`get_method` (receiver methods), `get_fields` | Go has no class → **not** `get_classes`. Functions are kept distinct from methods (no "module-name-as-class" hack Python uses for module funcs). Establishes the SDK's first type-centric precedent for future languages. | +| **SDK2** | `get_callers`/`get_callees` keyed on **callable id / `(type_id, sig)`** | Matches Go's identity model and the C-facade style (take the callable, not class+method strings). Clean for package-level funcs and methods alike. | +| **SDK3** | Scope: **Tier A + L1/L2 leaf accessors + repository-artifact getters + `get_source`**; omit slicing/CFG/CDG/DDG/reachability | Those need L3/L4 the analyzer does not produce. The generic `AnalysisBackend` ABC's abstractmethods are all implemented; L3/L4 methods are absent/`NotImplementedError` per the ABC contract — no misleading always-erroring surface. | +| **SDK4** | `GoAnalysisBackend` sets `N="Go"`, `P="GO_"`; `GoBackend = Union[GoCodeAnalyzerConfig, Neo4jConnectionConfig]` | `Neo4jConnectionConfig` already exists in `commons/backend_config.py` and in each `*Backend` union → the consumer side is **additive**. The graph it reads is "populated out of band" by the analyzer's `--emit neo4j`. | + +One facade vocabulary, two encodings when the TS SDK catches up: `cldk/analysis/go/GoAnalysis` +(Python) and `src/analysis/go/GoAnalysis` (TS) mirror method-for-method. This train does the +Python encoding only. + +--- + +## 5. Package layout + +Analyzer (new, sibling to `internal/syntactic_analysis/v2emit/`, off the JSON hot path): + +``` +internal/neo4j/ + schema.go declarative contract: labels + keys + typed props, rel types + endpoints, + derived DDL (one uniqueness constraint per (mergeLabel,key)), curated + indexes incl. the :GoCanNode id range index, SCHEMA_VERSION, --emit schema + rows.go output-agnostic row IR: GraphRows/NodeRow/EdgeRow/NodeRef, RowBuilder + (in-memory MERGE, deferred edges), applicationPrefix/descendantPrefix, + prune(), Cypher-literal rendering, in-memory-only `module` grouping field + project.go pure projector: v2.Analysis tree → GraphRows. No I/O, no driver + cypher.go snapshot writer → self-contained idempotent graph.cypher + bolt.go incremental writer → content-hash-diffed per-module Bolt push, schema-version + gate, --eager purge/prune (lazy driver dependency) + emit.go facade: emit_schema() (static) and emit_neo4j() (project + write) +``` + +SDK (on the `lamwassi/python-sdk` fork), mirroring `cldk/analysis/python/{codeanalyzer,neo4j}/`: + +``` +cldk/analysis/go/ + __init__.py + backend.py GoAnalysisBackend(AnalysisBackend[...]) — generic ABC impl, N/P set + go_analysis.py the GoAnalysis facade (type-centric surface, SDK1–SDK3) + codeanalyzer/ in-process backend: runs cango, reads analysis.json + neo4j/ read-only Neo4j backend: Cypher queries over the GO_-prefixed graph +cldk/models/go/ Go Pydantic models (GoApplication/GoModule/GoType/GoCallable/...) +``` + +Plus `GoBackend` union + `cache_subdir` "go" key in `cldk/analysis/commons/backend_config.py`. + +--- + +## 6. Release plan (lockstep, under epic #94) + +Epic #94's release plan already states: **"Neo4j projection and the JSON schema move in +lockstep."** Concretely: + +- **Analyzer** ships the `internal/neo4j/` projection as part of the v2 line. `--emit neo4j` + is independently landable behind the existing flag stubs; it produces a graph at + full-depth = L2 today and grows additively when L3/L4 arrive (no contract change — L1 + reserve). +- **SDK** (`GoAnalysisBackend`) pins the analyzer version **only after** the analyzer v2 + release is cut — same ordering constraint as the JSON wiring (the SDK's Go models can't + parse v2 until then, and there is no Go wiring yet, so this is net-new, not a migration). +- **Version lockstep:** `SCHEMA_VERSION = "2.0.0"` on both sides; the SDK gates on the + analyzer-version floor (additive-MINOR suspended for the 2.0.0 line). + +--- + +## 7. Tracking (just-in-time, under epic #94) + +Shape: **parent + sub-issue stack** (the Neo4j projection is a heavy rung whose units land +as separate reviewable PRs — the roadmap's milestones). Children filed **as each is picked +up**, not up front; the committed spec + roadmap already hold the full plan. + +- **Analyzer children** on `codeanalyzer-go` (one PR each): schema contract + conformance + test → row IR → pure projector → cypher writer → bolt writer → CLI wiring. May be grouped + (see roadmap) if a reviewer prefers fewer, larger PRs. +- **SDK child** on `lamwassi/python-sdk`: `GoAnalysisBackend` + models + `GoBackend` union. +- Attach each as a native GitHub **sub-issue** of #94 by id (not a hand-kept checklist). + Filing on the upstream org repo may need a maintainer given pull-only access + ([upstream-access-readonly]); children on repos the user controls (the fork) attach directly. + +--- + +## 8. Definition of done + +- `--emit schema` produces a `schema.neo4j.json` that parses and lists every node family / + relationship / property the projector can emit. +- `--emit neo4j` with no `--neo4j-uri` writes a `graph.cypher` that runs clean against an + empty Neo4j and is **idempotent** (second run yields the same graph). +- Conformance test: the real projector never emits a label / relationship / property + undeclared in `schema.go`. +- **No dangling endpoints:** every `GO_CALLS` / `GO_RESOLVES_TO` / `GO_EMBEDS` / `GO_SATISFIES` + / `GO_IMPORTS` endpoint resolves to an emitted node (declared or external ghost). +- **Two-projection agreement:** graph node ids == JSON `can://` ids / global ordinals, + one-to-one at full depth (modulo the explicit containment edges). +- **Determinism:** byte-identical `graph.cypher` across runs and `-j` values. +- **Prefix-scope safety:** a hand-built `GraphRows` with no `:GoApplication` root emits **no** + destructive statement; a wipe never reaches a second app or a sibling-language subgraph. +- **Parity:** no renamed/repurposed shared vocabulary; `GoAnalysisBackend` `N`/`P` match the + graph's prefixes exactly; SDK output validates against the Go Pydantic models. From 84f32140c158acf4152086d4a6bf33a779fde2b5 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Sun, 4 Oct 2026 13:23:40 -0400 Subject: [PATCH 19/29] feat(neo4j): schema contract + conformance scaffold + --emit schema MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone M1 of the Neo4j projection (Closes #11). The declarative single source of truth for the graph vocabulary, before any projector or writer. internal/neo4j/schema.go declares the GO_/Go-namespaced contract: node labels (the Go set is narrower than Python's — no decorator/attribute/variable), the shared :GoSymbol merge label, the :GoCanNode marker, relationship types incl. GO_EMBEDS vs GO_SATISFIES (Go's two distinct inheritance mechanisms) and the reserved L3/L4 overlay (declared now, zero rows until the levels land), and the _k discriminants. internal/neo4j/ddl.go derives the DDL from the node specs — one uniqueness constraint per distinct (mergeLabel, key), so the symbol family collapses to a single GoSymbol.id constraint — plus the curated :GoCanNode id range index that makes scoped statements prefix-seekable. BuildSchemaDocument assembles the --emit schema document. internal/neo4j/emit.go: EmitSchema writes schema.neo4j.json (static, needs no project). Wired into the CLI --emit schema branch, which now returns the written path instead of "not yet implemented"; --emit neo4j stays stubbed until its writers (M4-M6) land. Gate (green): --emit schema writes a parseable schema.neo4j.json listing every family; derived DDL has exactly one constraint per distinct (mergeLabel, key) with the symbol family collapsed; every relationship endpoint names a declared label; prefixes match the SDK's N/P. The conformance test (projector-emits-only-declared) is a scaffold M3 fills. --- cmd/codeanalyzer/main.go | 15 +- cmd/codeanalyzer/main_test.go | 45 ++++-- internal/neo4j/ddl.go | 159 ++++++++++++++++++++ internal/neo4j/emit.go | 42 ++++++ internal/neo4j/schema.go | 266 ++++++++++++++++++++++++++++++++++ internal/neo4j/schema_test.go | 138 ++++++++++++++++++ 6 files changed, 651 insertions(+), 14 deletions(-) create mode 100644 internal/neo4j/ddl.go create mode 100644 internal/neo4j/emit.go create mode 100644 internal/neo4j/schema.go create mode 100644 internal/neo4j/schema_test.go diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 5b0b40f..2fa3455 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -13,6 +13,7 @@ import ( "github.com/spf13/cobra" "github.com/codellm-devkit/codeanalyzer-go/internal/core" + "github.com/codellm-devkit/codeanalyzer-go/internal/neo4j" "github.com/codellm-devkit/codeanalyzer-go/internal/options" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) @@ -67,16 +68,22 @@ via CLDK(language="go").analysis(project_path=...).`, return nil } - // --emit selects the output projection. neo4j/schema are validated - // here and rejected with a non-zero exit until the Neo4j child lands - // (never a silent fallback to JSON). schema needs no --input. + // --emit selects the output projection. schema is a static contract + // that needs no --input: handle it here and return. neo4j is still + // rejected until its writers land (M4–M6) — never a silent fallback + // to JSON. switch options.EmitTarget(emit) { case options.EmitJSON: // valid — falls through to the JSON path below. case options.EmitNeo4j: return fmt.Errorf("--emit neo4j is not yet implemented; use --emit json") case options.EmitSchema: - return fmt.Errorf("--emit schema is not yet implemented; use --emit json") + path, err := neo4j.EmitSchema(outputDir) + if err != nil { + return err + } + cmd.Println("wrote " + path) + return nil default: return fmt.Errorf("unsupported --emit target %q; supported: json, neo4j, schema", emit) } diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index 3843261..00c0f03 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -78,18 +78,43 @@ func TestRootCmd_EmitNeo4jNotImplemented(t *testing.T) { } } -func TestRootCmd_EmitSchemaNotImplementedWithoutInput(t *testing.T) { - // --emit schema needs no --input; it must still fail (not yet implemented) - // rather than complain about a missing --input. - _, _, err := runCmd("--emit", "schema") - if err == nil { - t.Fatal("expected non-zero exit for --emit schema, got nil") +func TestRootCmd_EmitSchemaWritesContractWithoutInput(t *testing.T) { + // --emit schema is a static contract: it needs no --input and writes + // schema.neo4j.json to the output dir (M1). + out := t.TempDir() + stdout, _, err := runCmd("--emit", "schema", "--output", out) + if err != nil { + t.Fatalf("--emit schema should succeed without --input; got %v", err) } - if strings.Contains(err.Error(), "required") { - t.Errorf("--emit schema should not require --input; got %q", err.Error()) + + path := filepath.Join(out, "schema.neo4j.json") + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("schema.neo4j.json not written: %v", err) } - if !strings.Contains(err.Error(), "not yet implemented") { - t.Errorf("error should say 'not yet implemented'; got %q", err.Error()) + if !strings.Contains(stdout, "schema.neo4j.json") { + t.Errorf("expected stdout to report the written path; got %q", stdout) + } + + // It must parse and carry the schema version + a GO_-prefixed vocabulary. + var doc struct { + SchemaVersion string `json:"schema_version"` + RelPrefix string `json:"rel_prefix"` + Nodes []struct { + Label string `json:"label"` + } `json:"nodes"` + } + if err := json.Unmarshal(data, &doc); err != nil { + t.Fatalf("schema.neo4j.json does not parse: %v", err) + } + if doc.SchemaVersion != "2.0.0" { + t.Errorf("schema_version = %q, want 2.0.0", doc.SchemaVersion) + } + if doc.RelPrefix != "GO_" { + t.Errorf("rel_prefix = %q, want GO_", doc.RelPrefix) + } + if len(doc.Nodes) == 0 { + t.Error("schema document lists no node families") } } diff --git a/internal/neo4j/ddl.go b/internal/neo4j/ddl.go new file mode 100644 index 0000000..c292461 --- /dev/null +++ b/internal/neo4j/ddl.go @@ -0,0 +1,159 @@ +package neo4j + +import ( + "fmt" + "sort" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// Derived DDL + the --emit schema document. Constraints are DERIVED from the +// node specs — one uniqueness constraint per distinct (mergeLabel, key) — so a +// new label automatically brings its own constraint and there is no second list +// to keep in sync. Indexes are curated: the critical one is the :GoCanNode id +// range index that makes every scoped/destructive statement a prefix SEEK +// instead of a store scan (STARTS WITH is index-backed; CONTAINS/ENDS WITH are +// not). + +// Constraint is one uniqueness constraint: UNIQUE on (Label, Property). +type Constraint struct { + Name string `json:"name"` + Label string `json:"label"` + Property string `json:"property"` +} + +// Index is one curated index. +type Index struct { + Name string `json:"name"` + Label string `json:"label"` + Property string `json:"property"` + // Kind is "range" (the default, backs prefix seeks) today; fulltext is + // reserved for a later code-search index. + Kind string `json:"kind"` +} + +// UniquenessConstraints returns one constraint per distinct (mergeLabel, key), +// derived from Nodes. Deterministic: sorted by (label, property). The symbol +// family (:GoType/:GoCallable/:GoExternal all merge on :GoSymbol) collapses to +// a SINGLE GoSymbol.id constraint, which is the point of the shared merge label. +func UniquenessConstraints() []Constraint { + seen := make(map[string]Constraint) + for _, n := range Nodes { + key := n.MergeLabel + "\x00" + n.Key + if _, ok := seen[key]; ok { + continue + } + seen[key] = Constraint{ + Name: constraintName(n.MergeLabel, n.Key), + Label: n.MergeLabel, + Property: n.Key, + } + } + out := make([]Constraint, 0, len(seen)) + for _, c := range seen { + out = append(out, c) + } + sort.Slice(out, func(i, j int) bool { + if out[i].Label != out[j].Label { + return out[i].Label < out[j].Label + } + return out[i].Property < out[j].Property + }) + return out +} + +// Indexes returns the curated indexes. The id range index is anchored on the +// :GoCanNode marker label (property indexes are label-scoped, so the prefix +// predicate needs a label to seek on). cannode_kind supports kind filters over +// body/type nodes. +func Indexes() []Index { + return []Index{ + {Name: "gocannode_id", Label: MarkerLabel, Property: "id", Kind: "range"}, + {Name: "gocallable_signature", Label: LabelCallable, Property: "signature", Kind: "range"}, + } +} + +// constraintName builds a stable, lowercase constraint name, e.g. +// "gosymbol_id_unique". +func constraintName(label, prop string) string { + return fmt.Sprintf("%s_%s_unique", lower(label), prop) +} + +// ─── --emit schema document ────────────────────────────────────────────────── + +// SchemaDocument is the machine-readable graph contract, serialized by +// --emit schema to schema.neo4j.json. It is the graph-side sibling of the +// keystone JSON schema and what the SDK's Neo4j backend reconstructs against. +type SchemaDocument struct { + SchemaVersion string `json:"schema_version"` + Language string `json:"language"` + NodePrefix string `json:"node_prefix"` + RelPrefix string `json:"rel_prefix"` + MarkerLabel string `json:"marker_label"` + SymbolLabel string `json:"symbol_label"` + Nodes []NodeDoc `json:"nodes"` + Relationships []RelDoc `json:"relationships"` + Constraints []Constraint `json:"constraints"` + Indexes []Index `json:"indexes"` +} + +// NodeDoc / RelDoc are the serialized (JSON-stable) views of NodeSpec / RelSpec. +type NodeDoc struct { + Label string `json:"label"` + MergeLabel string `json:"merge_label"` + Key string `json:"key"` + Props []string `json:"props"` + CanNode bool `json:"can_node"` +} + +type RelDoc struct { + Type string `json:"type"` + From []string `json:"from"` + To []string `json:"to"` + Props []string `json:"props,omitempty"` + Discriminant string `json:"discriminant,omitempty"` + Deferred bool `json:"deferred,omitempty"` + Reserved bool `json:"reserved,omitempty"` +} + +// BuildSchemaDocument assembles the full contract document from the in-repo +// declarations. It needs no analyzed project — --emit schema is static. +func BuildSchemaDocument() SchemaDocument { + nodes := make([]NodeDoc, 0, len(Nodes)) + for _, n := range Nodes { + nodes = append(nodes, NodeDoc{ + Label: n.Label, MergeLabel: n.MergeLabel, Key: n.Key, + Props: n.Props, CanNode: n.CanNode, + }) + } + rels := make([]RelDoc, 0, len(Rels)) + for _, r := range Rels { + rels = append(rels, RelDoc{ + Type: r.Type, From: r.From, To: r.To, Props: r.Props, + Discriminant: r.Discriminant, Deferred: r.Deferred, Reserved: r.Reserved, + }) + } + return SchemaDocument{ + SchemaVersion: SchemaVersion, + Language: v2.Language, + NodePrefix: NodePrefix, + RelPrefix: RelPrefix, + MarkerLabel: MarkerLabel, + SymbolLabel: SymbolLabel, + Nodes: nodes, + Relationships: rels, + Constraints: UniquenessConstraints(), + Indexes: Indexes(), + } +} + +// lower is a tiny ASCII-lowercase for constraint names (labels are ASCII). +func lower(s string) string { + b := []byte(s) + for i, c := range b { + if c >= 'A' && c <= 'Z' { + b[i] = c + ('a' - 'A') + } + } + return string(b) +} diff --git a/internal/neo4j/emit.go b/internal/neo4j/emit.go new file mode 100644 index 0000000..207273f --- /dev/null +++ b/internal/neo4j/emit.go @@ -0,0 +1,42 @@ +package neo4j + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" +) + +// Facade between the CLI and the Neo4j backend. At M1 only EmitSchema exists — +// the static contract serializer, which needs no analyzed project. EmitNeo4j +// (project + write) lands at M6 once the projector (M3) and the writers (M4/M5) +// exist. + +// SchemaFileName is the machine-readable graph contract written by --emit schema. +const SchemaFileName = "schema.neo4j.json" + +// EmitSchema serializes the static graph contract (BuildSchemaDocument) to +// /schema.neo4j.json. It is independent of any analyzed project: +// --emit schema requires no -i. An empty outputDir writes to the current +// directory. The JSON is indented and newline-terminated for a stable, +// diffable checked-in artifact. +func EmitSchema(outputDir string) (string, error) { + doc := BuildSchemaDocument() + data, err := json.MarshalIndent(doc, "", " ") + if err != nil { + return "", fmt.Errorf("serialize schema document: %w", err) + } + data = append(data, '\n') + + if outputDir == "" { + outputDir = "." + } + if err := os.MkdirAll(outputDir, 0o755); err != nil { + return "", fmt.Errorf("create output dir %q: %w", outputDir, err) + } + path := filepath.Join(outputDir, SchemaFileName) + if err := os.WriteFile(path, data, 0o644); err != nil { + return "", fmt.Errorf("write %s: %w", path, err) + } + return path, nil +} diff --git a/internal/neo4j/schema.go b/internal/neo4j/schema.go new file mode 100644 index 0000000..271837d --- /dev/null +++ b/internal/neo4j/schema.go @@ -0,0 +1,266 @@ +// Package neo4j is the Neo4j projection of the canonical schema-v2 CPG envelope: +// a pure, identity-preserving second projection of the SAME tree the JSON path +// emits to analysis.json, keyed on the SAME can:// ids. JSON and graph are +// interchangeable and can never diverge. +// +// Module split (pure projection, dumb writers — mirrors codeanalyzer-python's +// codeanalyzer/neo4j/* and codeanalyzer-typescript's src/build/neo4j/*): +// +// schema.go this file — the declarative contract (labels, relationships, +// derived DDL, SCHEMA_VERSION, --emit schema serialization) +// rows.go output-agnostic row IR + RowBuilder (M2) +// project.go pure projector: v2.Analysis tree -> GraphRows (M3) +// cypher.go snapshot writer -> graph.cypher (M4) +// bolt.go incremental Bolt writer (M5) +// emit.go facade: EmitSchema / EmitNeo4j (M6) +// +// This file is the single in-repo source of truth for the graph vocabulary. +// Three things derive from it: the DDL (one uniqueness constraint per distinct +// (mergeLabel, key)); the --emit schema document; and the conformance test, +// which asserts the projector (M3) never emits an undeclared label / +// relationship / property. The contract therefore cannot silently drift from +// the emitter. +// +// Vocabulary decisions are recorded in CLAUDE.md § Neo4j-projection decisions +// and docs/design/Neo4j_projection/specs/neo4j-projection.md. The parity clause +// governs: GO_/Go-prefixed code vocabulary (leaf additions), the neutral +// Artifact/Package/ConfigKey layer reused verbatim as cross-language merge +// targets, never a renamed shared name. +package neo4j + +import v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + +// SchemaVersion is the graph contract version, stamped onto the :GoApplication +// node so a consumer can detect a producer/consumer mismatch at runtime. It +// moves in lockstep with the JSON schema major (v2.SchemaVersion). The +// additive-MINOR rule is suspended for the 2.0.0 line (parity with the sibling +// ruling); consumers gate on the analyzer-version floor instead. +const SchemaVersion = v2.SchemaVersion // "2.0.0" + +// ─── namespacing (parity: Python PY_/Py, TS TS_/TS) ───────────────────────── + +// NodePrefix / RelPrefix are the node-label and relationship-type prefixes for +// Go's code vocabulary. The SDK's AnalysisBackend ABC reads these as its N / P +// ClassVars, so GoAnalysisBackend sets N="Go", P="GO_" — the graph vocabulary +// and the backend's N/P must agree letter-for-letter. +const ( + NodePrefix = "Go" + RelPrefix = "GO_" +) + +// ─── markers & shared merge labels ────────────────────────────────────────── + +const ( + // MarkerLabel rides every node keyed by a can:// id. It is a pure index + // anchor — Neo4j property indexes are label-scoped, so the prefix predicate + // `id STARTS WITH $p` can seek instead of scanning the store. It carries no + // safety claim of its own (the test is the can:// scheme). + MarkerLabel = "GoCanNode" + + // SymbolLabel is the shared MERGE label on :GoType / :GoCallable / + // :GoExternal. One uniqueness constraint for the family, and an edge can + // MATCH any of them by a single (GoSymbol {id}) lookup regardless of kind. + SymbolLabel = "GoSymbol" +) + +// ─── node labels ──────────────────────────────────────────────────────────── + +// Node labels. The Go set is deliberately NARROWER than Python's: Go has no +// decorators, no class attributes, and no module/callable-level variable nodes, +// so :GoDecorator / :GoAttribute / :GoVariable are absent by design (CLAUDE.md +// decision X1). Struct fields are the one leaf node (:GoField). +const ( + LabelApplication = "GoApplication" + LabelModule = "GoModule" + LabelType = "GoType" // kind ∈ struct|interface|alias|defined is a PROPERTY, not a label + LabelCallable = "GoCallable" + LabelField = "GoField" + LabelBodyNode = "GoBodyNode" + LabelExternal = "GoExternal" + + // Neutral repository-artifact layer — UN-prefixed by design: these are + // cross-language merge targets, so a sibling analyzer over the same repo + // lands on the SAME node instead of a per-language duplicate. + LabelArtifact = "Artifact" + LabelPackage = "Package" + LabelConfigKey = "ConfigKey" +) + +// ─── relationship types ───────────────────────────────────────────────────── + +const ( + // Containment (the tree, rendered as edges). + RelHasModule = "GO_HAS_MODULE" + RelDeclares = "GO_DECLARES" // module→func, callable→closure + RelHasMethod = "GO_HAS_METHOD" // type→receiver-method + RelHasField = "GO_HAS_FIELD" + RelHasBodyNode = "GO_HAS_BODY_NODE" + + // Call graph. + RelCalls = "GO_CALLS" // weight, prov; endpoints resolve or fall back to an external ghost + RelResolvesTo = "GO_RESOLVES_TO" // call body node → callee + + // Inheritance — Go's two distinct mechanisms, named honestly (CLAUDE.md T2): + // GO_EMBEDS = explicit struct/interface embedding (base_types[], deferred / + // edge-only-when-resolved); GO_SATISFIES = computed structural satisfaction + // (interfaces[], via method-set matching). Deliberately NOT extends/implements. + RelEmbeds = "GO_EMBEDS" + RelSatisfies = "GO_SATISFIES" + + // Imports — aggregated per target → :GoModule (in-project) or :GoExternal ghost. + RelImports = "GO_IMPORTS" + + // Repository layer. Neutral (un-prefixed) edges are cross-language; the two + // GO_ edges are this analyzer's own resolution claims. + RelHasArtifact = "HAS_ARTIFACT" + RelDeclaresDependency = "DECLARES_DEPENDENCY" // _k=kind + RelLocks = "LOCKS" + RelDefinesConfig = "DEFINES_CONFIG" + RelProvides = "GO_PROVIDES" + RelUnresolvedImport = "GO_UNRESOLVED_IMPORT" + + // Reserved L3/L4 overlay — declared NOW, emits ZERO rows until the levels + // land (CLAUDE.md L1). When L3/L4 arrive the projector fills these with no + // schema-contract change. --emit neo4j is "full-depth = max implemented" + // (L2 today); the graph grows additively. + RelCFGNext = "GO_CFG_NEXT" // _k=kind + RelCDG = "GO_CDG" + RelDDG = "GO_DDG" // _k=var,prov + RelParamIn = "GO_PARAM_IN" + RelParamOut = "GO_PARAM_OUT" + RelSummary = "GO_SUMMARY" +) + +// ─── the declared contract ────────────────────────────────────────────────── + +// NodeSpec declares one node label: its MERGE label (labels[0] at projection +// time), its key property, and the typed property names it may carry. The +// conformance test (M3) holds the projector to the Props set. +type NodeSpec struct { + // Label is the specific kind label (e.g. "GoType"). + Label string + // MergeLabel is the label the uniqueness constraint and MERGE key on. It is + // the shared SymbolLabel for the symbol family, else the label itself. + MergeLabel string + // Key is the merge key property (always "id" for can://-keyed nodes; + // neutral nodes key on their own id/name). + Key string + // Props are every property name the projector may set on this node, beyond + // the key. Used by the conformance test; order is not significant. + Props []string + // CanNode is true when the node is keyed by a can:// id and therefore + // carries the :GoCanNode marker label and participates in prefix scoping. + CanNode bool +} + +// RelSpec declares one relationship type: its endpoints (by the labels it may +// connect) and the property names it may carry. Discriminant records the _k +// property used to keep legitimately-parallel edges from collapsing under MERGE. +type RelSpec struct { + Type string + // From / To are the label sets the endpoints may take. Used by the + // conformance test to reject an edge wired to an undeclared endpoint. + From []string + To []string + // Props are the property names the edge may carry (excluding the _k + // discriminant, which is Discriminant). + Props []string + // Discriminant names the _k key when parallel edges per (endpoint pair) are + // legitimate (e.g. "kind" for GO_CFG_NEXT). Empty = plain endpoint-pair MERGE. + Discriminant string + // Deferred marks an edge whose target may be library/external code not + // present in the graph: it is emitted via the RowBuilder's edge-only-when- + // resolved path and kept at finish() only if the target was emitted. + Deferred bool + // Reserved marks the L3/L4 overlay edges: declared now, emits zero rows at + // the current max level. + Reserved bool +} + +// Nodes is the full node-label contract, in projection (spine) order. +var Nodes = []NodeSpec{ + { + Label: LabelApplication, MergeLabel: LabelApplication, Key: "id", CanNode: true, + Props: []string{"schema_version", "language", "max_level", "analyzer_name", "analyzer_version"}, + }, + { + Label: LabelModule, MergeLabel: LabelModule, Key: "id", CanNode: true, + Props: []string{"package", "source", "content_hash", "span_json"}, + }, + { + Label: LabelType, MergeLabel: SymbolLabel, Key: "id", CanNode: true, + // kind ∈ struct|interface|alias|defined (property, decision T1). + Props: []string{"kind", "signature", "span_json"}, + }, + { + Label: LabelCallable, MergeLabel: SymbolLabel, Key: "id", CanNode: true, + // source_file present only when it differs from the nesting module (D8); + // error_channel is a homogeneous string array; is_* only on body nodes. + Props: []string{"kind", "signature", "return_type", "error_channel", "source_file", "span_json", "metrics_json"}, + }, + { + Label: LabelField, MergeLabel: LabelField, Key: "id", CanNode: true, + Props: []string{"kind", "type", "span_json"}, + }, + { + Label: LabelBodyNode, MergeLabel: LabelBodyNode, Key: "id", CanNode: true, + // kind "call" at L1; is_goroutine/is_deferred pruned when false. + Props: []string{"kind", "is_goroutine", "is_deferred", "span_json"}, + }, + { + Label: LabelExternal, MergeLabel: SymbolLabel, Key: "id", CanNode: true, + Props: []string{"name", "path"}, + }, + + // Neutral layer. Keyed by own id (not a can:// id), so NO marker label and + // outside prefix scoping — a sibling analyzer converges on the same node. + { + Label: LabelArtifact, MergeLabel: LabelArtifact, Key: "id", CanNode: false, + Props: []string{"path", "format"}, + }, + { + Label: LabelPackage, MergeLabel: LabelPackage, Key: "id", CanNode: false, + Props: []string{"ecosystem", "name"}, + }, + { + Label: LabelConfigKey, MergeLabel: LabelConfigKey, Key: "id", CanNode: false, + Props: []string{"key", "value"}, + }, +} + +// Rels is the full relationship contract, grouped by family. +var Rels = []RelSpec{ + // Containment. + {Type: RelHasModule, From: []string{LabelApplication}, To: []string{LabelModule}}, + {Type: RelDeclares, From: []string{LabelModule, LabelCallable}, To: []string{LabelType, LabelCallable}}, + {Type: RelHasMethod, From: []string{LabelType}, To: []string{LabelCallable}}, + {Type: RelHasField, From: []string{LabelType}, To: []string{LabelField}}, + {Type: RelHasBodyNode, From: []string{LabelCallable}, To: []string{LabelBodyNode}}, + + // Call graph. + {Type: RelCalls, From: []string{LabelCallable, LabelExternal}, To: []string{LabelCallable, LabelExternal}, Props: []string{"weight", "prov"}}, + {Type: RelResolvesTo, From: []string{LabelBodyNode}, To: []string{LabelCallable, LabelExternal}}, + + // Inheritance. + {Type: RelEmbeds, From: []string{LabelType}, To: []string{LabelType}, Deferred: true}, + {Type: RelSatisfies, From: []string{LabelType}, To: []string{LabelType}}, + + // Imports. + {Type: RelImports, From: []string{LabelModule}, To: []string{LabelModule, LabelExternal}, Props: []string{"alias", "positions_json"}}, + + // Repository layer. + {Type: RelHasArtifact, From: []string{LabelApplication}, To: []string{LabelArtifact}}, + {Type: RelDeclaresDependency, From: []string{LabelArtifact}, To: []string{LabelPackage}, Discriminant: "kind"}, + {Type: RelLocks, From: []string{LabelArtifact}, To: []string{LabelPackage}}, + {Type: RelDefinesConfig, From: []string{LabelArtifact}, To: []string{LabelConfigKey}}, + {Type: RelProvides, From: []string{LabelPackage}, To: []string{LabelExternal}}, + {Type: RelUnresolvedImport, From: []string{LabelApplication}, To: []string{LabelExternal}}, + + // Reserved L3/L4 overlay (zero rows until the levels land). + {Type: RelCFGNext, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Discriminant: "kind", Reserved: true}, + {Type: RelCDG, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, + {Type: RelDDG, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var", "prov"}, Discriminant: "var", Reserved: true}, + {Type: RelParamIn, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, + {Type: RelParamOut, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, + {Type: RelSummary, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, +} diff --git a/internal/neo4j/schema_test.go b/internal/neo4j/schema_test.go new file mode 100644 index 0000000..b552a19 --- /dev/null +++ b/internal/neo4j/schema_test.go @@ -0,0 +1,138 @@ +package neo4j + +import ( + "encoding/json" + "testing" +) + +// M1 schema-contract gate. These lock the declarative contract itself: +// - the DERIVED DDL has exactly one uniqueness constraint per distinct +// (mergeLabel, key), and the symbol family collapses to one; +// - every relationship endpoint names a DECLARED node label (no edge wired to +// a label the contract doesn't define); +// - the --emit schema document round-trips through JSON. +// +// TestConformance_ProjectorEmitsOnlyDeclared is the anti-drift scaffold the M3 +// projector fills: once project() exists, it walks a fixture at L2 and asserts +// the emitter never produces a label / relationship / property undeclared here. +// Until then it validates the contract is internally well-formed, so the gate +// is green and meaningful at M1. + +// declaredLabels is the set of every node label the contract defines. +func declaredLabels() map[string]NodeSpec { + m := make(map[string]NodeSpec, len(Nodes)) + for _, n := range Nodes { + m[n.Label] = n + } + return m +} + +func TestDDL_OneConstraintPerDistinctMergeKey(t *testing.T) { + cs := UniquenessConstraints() + + // No duplicate (label, property) pairs. + seen := make(map[string]bool) + for _, c := range cs { + key := c.Label + "." + c.Property + if seen[key] { + t.Errorf("duplicate uniqueness constraint for %s", key) + } + seen[key] = true + } + + // Count must equal the number of DISTINCT (mergeLabel, key) pairs in Nodes. + distinct := make(map[string]bool) + for _, n := range Nodes { + distinct[n.MergeLabel+"\x00"+n.Key] = true + } + if len(cs) != len(distinct) { + t.Errorf("derived %d constraints, want %d distinct (mergeLabel,key) pairs", len(cs), len(distinct)) + } + + // The symbol family must collapse: GoType/GoCallable/GoExternal all merge on + // GoSymbol, so there is exactly ONE GoSymbol.id constraint, not three. + var goSymbol int + for _, c := range cs { + if c.Label == SymbolLabel { + goSymbol++ + } + } + if goSymbol != 1 { + t.Errorf("GoSymbol constraints = %d, want exactly 1 (the shared merge label collapses the family)", goSymbol) + } +} + +func TestContract_EveryRelEndpointIsDeclared(t *testing.T) { + labels := declaredLabels() + for _, r := range Rels { + for _, l := range r.From { + if _, ok := labels[l]; !ok { + t.Errorf("relationship %s has undeclared From label %q", r.Type, l) + } + } + for _, l := range r.To { + if _, ok := labels[l]; !ok { + t.Errorf("relationship %s has undeclared To label %q", r.Type, l) + } + } + if len(r.From) == 0 || len(r.To) == 0 { + t.Errorf("relationship %s has an empty endpoint set", r.Type) + } + } +} + +func TestContract_CanNodesCarryIdKey(t *testing.T) { + // Every can:// node keys on "id" (prefix scoping and the marker index depend + // on it); neutral nodes may key otherwise but here also use "id". + for _, n := range Nodes { + if n.CanNode && n.Key != "id" { + t.Errorf("can:// node %s keys on %q, want \"id\"", n.Label, n.Key) + } + } +} + +func TestContract_PrefixesMatchSDKExpectations(t *testing.T) { + // The SDK's GoAnalysisBackend sets N="Go", P="GO_"; the graph vocabulary and + // the backend's N/P must agree letter-for-letter. + if NodePrefix != "Go" || RelPrefix != "GO_" { + t.Fatalf("prefixes drifted: NodePrefix=%q RelPrefix=%q, want \"Go\"/\"GO_\"", NodePrefix, RelPrefix) + } + for _, n := range Nodes { + // Neutral layer is intentionally un-prefixed. + if n.Label == LabelArtifact || n.Label == LabelPackage || n.Label == LabelConfigKey { + continue + } + if len(n.Label) < 2 || n.Label[:2] != NodePrefix { + t.Errorf("code node label %q is not %s-prefixed", n.Label, NodePrefix) + } + } +} + +func TestSchemaDocument_RoundTrips(t *testing.T) { + doc := BuildSchemaDocument() + data, err := json.Marshal(doc) + if err != nil { + t.Fatalf("marshal schema document: %v", err) + } + var back SchemaDocument + if err := json.Unmarshal(data, &back); err != nil { + t.Fatalf("unmarshal schema document: %v", err) + } + if back.SchemaVersion != SchemaVersion { + t.Errorf("schema_version = %q, want %q", back.SchemaVersion, SchemaVersion) + } + if len(back.Nodes) != len(Nodes) || len(back.Relationships) != len(Rels) { + t.Errorf("round-trip lost rows: nodes %d/%d, rels %d/%d", + len(back.Nodes), len(Nodes), len(back.Relationships), len(Rels)) + } + if len(back.Constraints) == 0 || len(back.Indexes) == 0 { + t.Error("schema document missing derived DDL") + } +} + +// TestConformance_ProjectorEmitsOnlyDeclared is the M3 anti-drift scaffold. +// When project() lands, replace the skip with: project a fixture at L2, collect +// every (label, prop) and (relType) it emits, and assert each is declared here. +func TestConformance_ProjectorEmitsOnlyDeclared(t *testing.T) { + t.Skip("scaffold: filled by M3 (project.go) — asserts the projector emits no undeclared label/relationship/property") +} From 18a1ee6b5b547009b3bdbd1410c2bb34539aaaee Mon Sep 17 00:00:00 2001 From: lambertpw Date: Sun, 4 Oct 2026 13:31:37 -0400 Subject: [PATCH 20/29] feat(neo4j): output-agnostic row IR + RowBuilder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone M2 of the Neo4j projection. internal/neo4j/rows.go: the deterministic, deduped bag both writers consume identically — GraphRows / NodeRow / EdgeRow / NodeRef — plus the RowBuilder with in-memory MERGE semantics: - Node() dedups on (labels[0], value): re-seeing an identity merges props (last-write-wins) and unions labels, so a hot external symbol collapses to one row. Returns a NodeRef edges address it by. - Edge() is append-only; EdgeToSymbol() DEFERS an edge whose target may be external, kept at Finish() only if the target node was emitted (the edge-only-when-resolved rule — GO_EMBEDS/GO_SATISFIES/GO_RESOLVES_TO never dangle). - prune() drops nil/empty-slice props (in Neo4j absence IS null). - Finish() resolves deferred edges and sorts nodes/edges to a total order for byte-stable output. - applicationPrefix() refuses a non-bare app id (so rows with no proper root get no destructive statement); descendantPrefix() appends "/" for strict containment. NodeRow.module is in-memory only — never emitted as a graph prop. - cypherValue/cypherString/cypherMap render Neo4j-legal literals with sorted keys. Gate (green): dedup+label-union, last-write-wins, deferred-edge drop/keep, prune, order-stable Finish, applicationPrefix refusal across bare/empty/too-deep ids, Cypher escaping + deterministic map rendering. --- internal/neo4j/rows.go | 319 ++++++++++++++++++++++++++++++++++++ internal/neo4j/rows_test.go | 192 ++++++++++++++++++++++ 2 files changed, 511 insertions(+) create mode 100644 internal/neo4j/rows.go create mode 100644 internal/neo4j/rows_test.go diff --git a/internal/neo4j/rows.go b/internal/neo4j/rows.go new file mode 100644 index 0000000..b917550 --- /dev/null +++ b/internal/neo4j/rows.go @@ -0,0 +1,319 @@ +package neo4j + +import ( + "fmt" + "sort" + "strings" +) + +// Milestone M2: the output-agnostic row IR and the RowBuilder. GraphRows is a +// deterministic, deduped bag of nodes and edges with NO I/O and NO driver; both +// writers (cypher.go, bolt.go) consume the identical bag. The RowBuilder is the +// in-memory analog of Neo4j MERGE semantics — it mirrors MERGE … SET += props +// before any writer runs, so each writer just replays a clean, deduped result. +// +// Property values must be Neo4j-legal: primitives or homogeneous arrays of +// primitives. nil is PRUNED — in Neo4j an absent property IS null, so +// "not carried" and "empty" never collide. Optional/variant-shaped fields are +// JSON-encoded into a *_json string property by the projector (M3), not here. + +// Value is a Neo4j-legal property value: a primitive (string/int/bool) or a +// homogeneous slice of primitives. The projector is responsible for only ever +// passing legal values; the writers render whatever is here. +type Value any + +// Props is a property bag. A nil entry is dropped by prune(). +type Props map[string]Value + +// NodeRef is how an edge addresses an endpoint: the (label, keyProp, value) an +// edge MATCHes on. Label is the MERGE label (labels[0]) of the target node. +type NodeRef struct { + Label string + KeyProp string + Value string +} + +// NodeRow is one graph node. Labels[0] is the constrained MERGE label; the rest +// are SET as extra labels (e.g. the :GoCanNode marker, the specific kind label +// alongside the shared :GoSymbol). Module is in-memory ONLY — the owning +// module's id, used by the incremental writer's per-module diff; it is NEVER +// emitted as a graph property. +type NodeRow struct { + Labels []string + KeyProp string + Value string + Props Props + Module string +} + +// EdgeRow is one typed relationship. Key is the optional _k relationship +// discriminant: when set, the MERGE is on {_k: Key} so legitimately-parallel +// edges between the same endpoint pair (a conditional's true/false CFG pair, +// per-(var,prov) DDG edges) survive instead of collapsing. +type EdgeRow struct { + Type string + From NodeRef + To NodeRef + Props Props + Key string +} + +// GraphRows is the finished, deterministic bag. +type GraphRows struct { + Nodes []NodeRow + Edges []EdgeRow +} + +// ─── RowBuilder ────────────────────────────────────────────────────────────── + +// RowBuilder accumulates nodes and edges with in-memory MERGE semantics. Nodes +// dedup on (labels[0], value); edges are append-only (per-pair facts that must +// not duplicate need an explicit guard in the projector). Edges whose target +// may be external/library code are DEFERRED and kept at finish() only if the +// target node was actually emitted — the edge-only-when-resolved rule, so an +// unresolved target degrades to a source-node property instead of a dangling edge. +type RowBuilder struct { + nodeIndex map[string]int // (labels[0] + "\x00" + value) -> index into nodes + nodes []NodeRow + edges []EdgeRow + deferred []EdgeRow +} + +// NewRowBuilder returns an empty builder. +func NewRowBuilder() *RowBuilder { + return &RowBuilder{nodeIndex: make(map[string]int)} +} + +// Node upserts a node keyed on (labels[0], value). Re-seeing the same identity +// MERGES props (last-write-wins) and UNIONs labels — a hot external symbol +// collapses to one row. Returns a NodeRef edges can address it by. nil props +// are pruned on the way in. +func (b *RowBuilder) Node(labels []string, keyProp, value string, props Props, module string) NodeRef { + if len(labels) == 0 { + panic("neo4j: Node requires at least one label (labels[0] is the MERGE label)") + } + k := labels[0] + "\x00" + value + if i, ok := b.nodeIndex[k]; ok { + existing := &b.nodes[i] + existing.Labels = unionLabels(existing.Labels, labels) + for pk, pv := range prune(props) { + if existing.Props == nil { + existing.Props = Props{} + } + existing.Props[pk] = pv + } + if existing.Module == "" { + existing.Module = module + } + return NodeRef{Label: labels[0], KeyProp: keyProp, Value: value} + } + b.nodeIndex[k] = len(b.nodes) + b.nodes = append(b.nodes, NodeRow{ + Labels: append([]string(nil), labels...), + KeyProp: keyProp, + Value: value, + Props: prune(props), + Module: module, + }) + return NodeRef{Label: labels[0], KeyProp: keyProp, Value: value} +} + +// Edge appends a relationship whose endpoints are known to exist this run. +// Append-only — not deduped. +func (b *RowBuilder) Edge(relType string, from, to NodeRef, props Props, key string) { + b.edges = append(b.edges, EdgeRow{Type: relType, From: from, To: to, Props: prune(props), Key: key}) +} + +// EdgeToSymbol defers a relationship whose target (addressed by its can:// id) +// may be library/external code not present in the graph. At finish() the edge +// is kept ONLY if a node with that (label, value) was emitted — so EXTENDS/ +// SATISFIES/RESOLVES_TO never dangle. targetLabel is the MERGE label the target +// would carry if present (e.g. SymbolLabel). +func (b *RowBuilder) EdgeToSymbol(relType string, from NodeRef, targetLabel, targetKey, targetValue string, props Props, key string) { + b.deferred = append(b.deferred, EdgeRow{ + Type: relType, + From: from, + To: NodeRef{Label: targetLabel, KeyProp: targetKey, Value: targetValue}, + Props: prune(props), + Key: key, + }) +} + +// Finish resolves deferred edges (keeping only those whose target was emitted), +// sorts nodes and edges by a stable key, and returns the GraphRows. Determinism +// matters: identical input must yield byte-identical output downstream. +func (b *RowBuilder) Finish() GraphRows { + emitted := make(map[string]bool, len(b.nodes)) + for _, n := range b.nodes { + emitted[n.Labels[0]+"\x00"+n.Value] = true + } + for _, e := range b.deferred { + if emitted[e.To.Label+"\x00"+e.To.Value] { + b.edges = append(b.edges, e) + } + } + + sort.Slice(b.nodes, func(i, j int) bool { + if b.nodes[i].Labels[0] != b.nodes[j].Labels[0] { + return b.nodes[i].Labels[0] < b.nodes[j].Labels[0] + } + return b.nodes[i].Value < b.nodes[j].Value + }) + sort.Slice(b.edges, func(i, j int) bool { + return edgeSortKey(b.edges[i]) < edgeSortKey(b.edges[j]) + }) + + return GraphRows{Nodes: b.nodes, Edges: b.edges} +} + +// edgeSortKey is the total order for edges: type, then endpoints, then the _k +// discriminant (so parallel edges per pair order stably). +func edgeSortKey(e EdgeRow) string { + return strings.Join([]string{e.Type, e.From.Label, e.From.Value, e.To.Label, e.To.Value, e.Key}, "\x00") +} + +// ─── scoping helpers ────────────────────────────────────────────────────────── + +// applicationPrefix returns the bare application id used to scope destructive +// statements. It REFUSES anything that is not a bare can:/// id (an +// empty or deeper id would silently widen or narrow the blast radius): it +// returns ("", false) so a hand-built GraphRows with no proper application root +// gets NO destructive statement. The app segment is the outermost, so one +// prefix spans every module the app owns and nothing another app owns. +func applicationPrefix(appID string) (string, bool) { + const scheme = "can://" + if !strings.HasPrefix(appID, scheme) { + return "", false + } + rest := strings.TrimPrefix(appID, scheme) + // A bare app id is exactly / — two non-empty segments, no deeper. + parts := strings.Split(rest, "/") + if len(parts) != 2 || parts[0] == "" || parts[1] == "" { + return "", false + } + return appID, true +} + +// descendantPrefix appends "/" to a can:// id so a STARTS WITH predicate matches +// strict descendants only — "foo.go" is not treated as a prefix of "foo.go2", +// and the app root id itself (can://go/app) is outside its own descendant +// prefix (can://go/app/), so a wipe must match the root separately. +func descendantPrefix(canID string) string { + return canID + "/" +} + +// ─── pruning & Cypher-literal rendering ─────────────────────────────────────── + +// prune drops nil-valued and empty-slice properties: in Neo4j an absent +// property is null, so a non-fact must simply not be written. Returns nil for +// an all-empty bag so callers can leave Props nil. +func prune(p Props) Props { + if len(p) == 0 { + return nil + } + out := make(Props, len(p)) + for k, v := range p { + if v == nil { + continue + } + if s, ok := v.([]string); ok && len(s) == 0 { + continue + } + out[k] = v + } + if len(out) == 0 { + return nil + } + return out +} + +// cypherValue renders a Neo4j-legal value as a Cypher literal (used by the +// snapshot writer; the Bolt writer passes values as parameters instead). +func cypherValue(v Value) string { + switch x := v.(type) { + case nil: + return "null" + case string: + return cypherString(x) + case bool: + if x { + return "true" + } + return "false" + case int: + return fmt.Sprintf("%d", x) + case int64: + return fmt.Sprintf("%d", x) + case []string: + parts := make([]string, len(x)) + for i, s := range x { + parts[i] = cypherString(s) + } + return "[" + strings.Join(parts, ", ") + "]" + default: + // Fall back to a quoted Go rendering rather than emit something illegal. + return cypherString(fmt.Sprintf("%v", x)) + } +} + +// cypherString renders a Cypher single-quoted string literal with escaping. +func cypherString(s string) string { + var b strings.Builder + b.WriteByte('\'') + for _, r := range s { + switch r { + case '\'': + b.WriteString("\\'") + case '\\': + b.WriteString("\\\\") + case '\n': + b.WriteString("\\n") + case '\r': + b.WriteString("\\r") + case '\t': + b.WriteString("\\t") + default: + b.WriteRune(r) + } + } + b.WriteByte('\'') + return b.String() +} + +// cypherMap renders a Props bag as a Cypher map literal with sorted keys (so +// output is deterministic). An empty bag renders as "{}". +func cypherMap(p Props) string { + if len(p) == 0 { + return "{}" + } + keys := make([]string, 0, len(p)) + for k := range p { + keys = append(keys, k) + } + sort.Strings(keys) + parts := make([]string, 0, len(keys)) + for _, k := range keys { + parts = append(parts, k+": "+cypherValue(p[k])) + } + return "{" + strings.Join(parts, ", ") + "}" +} + +// unionLabels merges two label slices preserving order and de-duplicating, +// keeping labels[0] (the MERGE label) first. +func unionLabels(a, b []string) []string { + seen := make(map[string]bool, len(a)+len(b)) + out := make([]string, 0, len(a)+len(b)) + for _, l := range a { + if !seen[l] { + seen[l] = true + out = append(out, l) + } + } + for _, l := range b { + if !seen[l] { + seen[l] = true + out = append(out, l) + } + } + return out +} diff --git a/internal/neo4j/rows_test.go b/internal/neo4j/rows_test.go new file mode 100644 index 0000000..aad83e0 --- /dev/null +++ b/internal/neo4j/rows_test.go @@ -0,0 +1,192 @@ +package neo4j + +import ( + "reflect" + "testing" +) + +// M2 gate: the RowBuilder's in-memory MERGE semantics, the edge-only-when- +// resolved rule, pruning, deterministic finish(), scoping-helper refusal, and +// Cypher-literal rendering. + +func TestRowBuilder_NodeMergeDedupsAndUnionsLabels(t *testing.T) { + b := NewRowBuilder() + id := "can://go/app/x.go/T" + b.Node([]string{SymbolLabel, LabelType}, "id", id, Props{"kind": "struct"}, "x.go") + // Re-see the same identity with an extra label + an extra prop. + b.Node([]string{SymbolLabel, MarkerLabel}, "id", id, Props{"signature": "T"}, "x.go") + + rows := b.Finish() + if len(rows.Nodes) != 1 { + t.Fatalf("re-seeing an identity must dedup to one node; got %d", len(rows.Nodes)) + } + n := rows.Nodes[0] + // Labels unioned, MERGE label first. + want := []string{SymbolLabel, LabelType, MarkerLabel} + if !reflect.DeepEqual(n.Labels, want) { + t.Errorf("labels = %v, want %v", n.Labels, want) + } + // Props merged (last-write-wins union). + if n.Props["kind"] != "struct" || n.Props["signature"] != "T" { + t.Errorf("props not merged across re-see: %v", n.Props) + } +} + +func TestRowBuilder_PropLastWriteWins(t *testing.T) { + b := NewRowBuilder() + id := "can://go/app/x.go/T" + b.Node([]string{LabelType}, "id", id, Props{"kind": "struct"}, "") + b.Node([]string{LabelType}, "id", id, Props{"kind": "interface"}, "") + rows := b.Finish() + if rows.Nodes[0].Props["kind"] != "interface" { + t.Errorf("last write must win: kind = %v, want interface", rows.Nodes[0].Props["kind"]) + } +} + +func TestRowBuilder_DeferredEdgeDroppedWhenTargetAbsent(t *testing.T) { + b := NewRowBuilder() + src := b.Node([]string{SymbolLabel, LabelType}, "id", "can://go/app/x.go/T", nil, "x.go") + // Target never emitted as a node. + b.EdgeToSymbol(RelEmbeds, src, SymbolLabel, "id", "can://go/app/y.go/Missing", nil, "") + rows := b.Finish() + if len(rows.Edges) != 0 { + t.Errorf("deferred edge to an unemitted target must be dropped; got %d edges", len(rows.Edges)) + } +} + +func TestRowBuilder_DeferredEdgeKeptWhenTargetEmitted(t *testing.T) { + b := NewRowBuilder() + src := b.Node([]string{SymbolLabel, LabelType}, "id", "can://go/app/x.go/T", nil, "x.go") + target := "can://go/app/y.go/Base" + b.Node([]string{SymbolLabel, LabelType}, "id", target, nil, "y.go") + b.EdgeToSymbol(RelEmbeds, src, SymbolLabel, "id", target, nil, "") + rows := b.Finish() + if len(rows.Edges) != 1 { + t.Fatalf("deferred edge to an emitted target must be kept; got %d edges", len(rows.Edges)) + } + if rows.Edges[0].Type != RelEmbeds || rows.Edges[0].To.Value != target { + t.Errorf("resolved edge wrong: %+v", rows.Edges[0]) + } +} + +func TestPrune_DropsNilAndEmptySlice(t *testing.T) { + got := prune(Props{ + "a": "keep", + "b": nil, + "empty": []string{}, + "list": []string{"x"}, + "zero": 0, // 0 is a real value, not absence — kept. + "false": false, + }) + if _, ok := got["b"]; ok { + t.Error("nil prop must be pruned") + } + if _, ok := got["empty"]; ok { + t.Error("empty slice must be pruned") + } + if got["a"] != "keep" || got["zero"] != 0 || got["false"] != false || !reflect.DeepEqual(got["list"], []string{"x"}) { + t.Errorf("prune dropped a real value: %v", got) + } +} + +func TestPrune_AllEmptyReturnsNil(t *testing.T) { + if prune(Props{"a": nil}) != nil { + t.Error("an all-empty bag must prune to nil") + } + if prune(nil) != nil { + t.Error("nil bag must prune to nil") + } +} + +func TestApplicationPrefix_RefusesNonBareID(t *testing.T) { + cases := map[string]bool{ + "can://go/app": true, // bare — accepted + "can://go/my-app": true, // + "": false, // empty + "can://go": false, // missing app segment + "can://go/app/x.go": false, // too deep (a module id) + "can://go/app/x.go/T": false, // deeper still + "go/app": false, // no scheme + "can://go//": false, // empty segment + } + for id, wantOK := range cases { + _, ok := applicationPrefix(id) + if ok != wantOK { + t.Errorf("applicationPrefix(%q) ok=%v, want %v", id, ok, wantOK) + } + } +} + +func TestDescendantPrefix_AppendsSlash(t *testing.T) { + if got := descendantPrefix("can://go/app"); got != "can://go/app/" { + t.Errorf("descendantPrefix = %q, want can://go/app/", got) + } +} + +func TestFinish_IsOrderStable(t *testing.T) { + // Build the SAME logical graph (same nodes, same A→B and B→C edges) but add + // nodes and edges in two different INSERTION orders; finish() must sort to + // identical ordering both times. + ref := func(b *RowBuilder, id string) NodeRef { + return b.Node([]string{SymbolLabel, LabelCallable}, "id", id, nil, "") + } + const A, B, C = "can://go/app/a.go/A", "can://go/app/b.go/B", "can://go/app/c.go/C" + + build := func(reverse bool) GraphRows { + b := NewRowBuilder() + if reverse { + // Insert nodes C,B,A and the B→C edge before the A→B edge. + rc, rb, ra := ref(b, C), ref(b, B), ref(b, A) + b.Edge(RelCalls, rb, rc, nil, "") + b.Edge(RelCalls, ra, rb, nil, "") + } else { + ra, rb, rc := ref(b, A), ref(b, B), ref(b, C) + b.Edge(RelCalls, ra, rb, nil, "") + b.Edge(RelCalls, rb, rc, nil, "") + } + return b.Finish() + } + a := build(false) + z := build(true) + if !reflect.DeepEqual(nodeValues(a), nodeValues(z)) { + t.Errorf("node order not stable:\n a=%v\n z=%v", nodeValues(a), nodeValues(z)) + } + if !reflect.DeepEqual(edgeKeys(a), edgeKeys(z)) { + t.Errorf("edge order not stable:\n a=%v\n z=%v", edgeKeys(a), edgeKeys(z)) + } +} + +func TestCypherRendering(t *testing.T) { + if got := cypherValue("a'b\\c"); got != `'a\'b\\c'` { + t.Errorf("string escaping wrong: %s", got) + } + if got := cypherValue(true); got != "true" { + t.Errorf("bool: %s", got) + } + if got := cypherValue([]string{"x", "y"}); got != "['x', 'y']" { + t.Errorf("list: %s", got) + } + // Map keys sorted for determinism. + if got := cypherMap(Props{"b": 1, "a": "x"}); got != "{a: 'x', b: 1}" { + t.Errorf("map not sorted/deterministic: %s", got) + } + if got := cypherMap(nil); got != "{}" { + t.Errorf("empty map: %s", got) + } +} + +// helpers. +func nodeValues(g GraphRows) []string { + out := make([]string, len(g.Nodes)) + for i, n := range g.Nodes { + out[i] = n.Value + } + return out +} +func edgeKeys(g GraphRows) []string { + out := make([]string, len(g.Edges)) + for i, e := range g.Edges { + out[i] = edgeSortKey(e) + } + return out +} From 708ad670e3e2c265d31bc42acb70b81cad71c5dd Mon Sep 17 00:00:00 2001 From: lambertpw Date: Sun, 4 Oct 2026 13:34:28 -0400 Subject: [PATCH 21/29] feat(neo4j): pure projector v2 tree -> GraphRows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone M3 of the Neo4j projection — the intellectual core. project.go walks the canonical v2 Analysis tree in spine order (application -> module -> type -> callable -> body call nodes -> call_graph -> imports) and emits a GraphRows bag, reading can:// ids off the stamped tree so graph and JSON join on one string (two-projection agreement). No I/O, no driver. - Types: one :GoType node + kind property; base_types -> GO_EMBEDS and interfaces -> GO_SATISFIES via the deferred edge path (never dangle); struct fields -> :GoField/GO_HAS_FIELD; methods nest under the receiver type (GO_HAS_METHOD). - Callables: error_channel/source_file/metrics flattened; closures -> GO_DECLARES; body call nodes keyed by the @ global ordinal, is_goroutine/is_deferred pruned when false; a resolved callee -> GO_RESOLVES_TO only when it names a declared callable. - call_graph -> GO_CALLS (weight, prov); endpoints resolve to a declared callable or an id-keyed :GoExternal ghost, so no endpoint dangles. - imports -> GO_IMPORTS aggregated per target (:GoModule in-project, else a :GoExternal ghost keyed by import path). - span/metrics flatten to *_json string props (Neo4j can't store nested maps). Fills M1's conformance scaffold: TestConformance_ProjectorEmitsOnlyDeclared projects the multipackage fixture at L2 (real builder + resolver, 63 nodes/83 edges) and asserts the emitter produces no undeclared label/relationship/ property. Plus no-dangling, JSON<->graph id parity, determinism, and Go-specific-shape gates. All green. --- internal/neo4j/project.go | 356 +++++++++++++++++++++++++++++++++ internal/neo4j/project_test.go | 158 +++++++++++++++ internal/neo4j/schema_test.go | 80 +++++++- 3 files changed, 590 insertions(+), 4 deletions(-) create mode 100644 internal/neo4j/project.go create mode 100644 internal/neo4j/project_test.go diff --git a/internal/neo4j/project.go b/internal/neo4j/project.go new file mode 100644 index 0000000..68bc936 --- /dev/null +++ b/internal/neo4j/project.go @@ -0,0 +1,356 @@ +package neo4j + +import ( + "encoding/json" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// Milestone M3: the pure projector. Project walks the canonical v2 Analysis +// tree and emits a GraphRows bag — one graph node per tree/body node, keyed on +// its can:// id, with containment and every typed overlay as relationships. No +// I/O, no driver; both writers consume the result identically. +// +// The projector NEVER invents ids: it reads them off the stamped v2 tree, so +// the graph and analysis.json join on one string (two-projection agreement). +// Node ids are can:// ids; the application segment is the outermost prefix of +// every id the app owns, which is what makes prefix scoping exact. +// +// Walk order (spine): application → per module (types → callables → fields → +// body call nodes) → call_graph → imports → neutral artifact/package layer → +// reserved L3/L4 overlay (zero rows at the current max level). Every endpoint +// resolves to a declared node or an id-keyed :GoExternal ghost — nothing dangles. + +// Project is the pure (tree) → GraphRows function. appID is analysis.Application.ID +// (the can:/// root); it is read from the envelope, never recomposed. +func Project(a *v2.Analysis) GraphRows { + p := &projector{ + b: NewRowBuilder(), + analysis: a, + // Index of every emitted callable/type id, so call_graph endpoints and + // body callees can tell a declared target from an external one. + declared: collectDeclaredIDs(a), + } + p.projectApplication() + for _, mod := range a.Application.SymbolTable { + p.projectModule(mod) + } + p.projectCallGraph() + return p.b.Finish() +} + +type projector struct { + b *RowBuilder + analysis *v2.Analysis + declared map[string]bool +} + +// ─── application ───────────────────────────────────────────────────────────── + +func (p *projector) projectApplication() { + a := p.analysis + p.b.Node( + []string{LabelApplication, MarkerLabel}, + "id", a.Application.ID, + Props{ + "schema_version": a.SchemaVersion, + "language": a.Language, + "max_level": a.MaxLevel, + "analyzer_name": a.Analyzer.Name, + "analyzer_version": a.Analyzer.Version, + }, + "", // the application root belongs to no single module + ) +} + +func (p *projector) appRef() NodeRef { + return NodeRef{Label: LabelApplication, KeyProp: "id", Value: p.analysis.Application.ID} +} + +// ─── module ────────────────────────────────────────────────────────────────── + +func (p *projector) projectModule(m v2.Module) { + modRef := p.b.Node( + []string{LabelModule, MarkerLabel}, + "id", m.ID, + Props{ + "package": m.Package, + "source": m.Source, + "content_hash": emptyToNil(m.ContentHash), + "span_json": spanJSON(m.Span), + }, + m.ID, + ) + p.b.Edge(RelHasModule, p.appRef(), modRef, nil, "") + + for _, t := range m.Types { + p.projectType(modRef, m.ID, t) + } + for _, fn := range m.Functions { + cRef := p.projectCallable(modRef, m.ID, fn) + p.b.Edge(RelDeclares, modRef, cRef, nil, "") + } + p.projectImports(modRef, m) +} + +// ─── type ───────────────────────────────────────────────────────────────────── + +func (p *projector) projectType(modRef NodeRef, module string, t v2.Type) { + tRef := p.b.Node( + []string{SymbolLabel, LabelType, MarkerLabel}, + "id", t.ID, + Props{ + "kind": t.Kind, // struct|interface|alias|defined (decision T1) + "signature": lastSegment(t.ID), + "span_json": spanJSON(t.Span), + }, + module, + ) + p.b.Edge(RelDeclares, modRef, tRef, nil, "") + + // Embedding (explicit spine): deferred — a base type may be external. + for _, baseID := range t.BaseTypes { + p.b.EdgeToSymbol(RelEmbeds, tRef, SymbolLabel, "id", baseID, nil, "") + } + // Structural satisfaction (computed). The interfaces are in-tree type ids + // the analyzer computed; still routed through the deferred path so a + // satisfied-but-external interface never dangles. + for _, ifaceID := range t.Interfaces { + p.b.EdgeToSymbol(RelSatisfies, tRef, SymbolLabel, "id", ifaceID, nil, "") + } + + // Struct fields. + for _, f := range t.Fields { + fRef := p.b.Node( + []string{LabelField, MarkerLabel}, + "id", f.ID, + Props{"kind": f.Kind, "type": f.Type, "span_json": spanJSON(f.Span)}, + module, + ) + p.b.Edge(RelHasField, tRef, fRef, nil, "") + } + + // Methods nest under their receiver type (containment-by-receiver). + for _, m := range t.Callables { + mRef := p.projectCallable(tRef, module, m) + p.b.Edge(RelHasMethod, tRef, mRef, nil, "") + } +} + +// ─── callable ───────────────────────────────────────────────────────────────── + +// projectCallable emits a callable node (+ its body call nodes and nested +// closures) and returns its ref. The containment edge from the parent (DECLARES +// or HAS_METHOD) is the caller's responsibility, since it differs by parent kind. +func (p *projector) projectCallable(parentRef NodeRef, module string, c v2.Callable) NodeRef { + cRef := p.b.Node( + []string{SymbolLabel, LabelCallable, MarkerLabel}, + "id", c.ID, + Props{ + "kind": c.Kind, + "signature": c.Signature, + "return_type": emptyToNil(c.ReturnType), + "error_channel": stringsOrNil(c.ErrorChannel), + "source_file": emptyToNil(c.SourceFile), // only when ≠ nesting module (D8) + "span_json": spanJSON(c.Span), + "metrics_json": metricsJSON(c.Metrics), + }, + module, + ) + + // Body call nodes (L1). Keyed by the global ordinal id (@) + // so graph and JSON body nodes land on one identity. + for local, bn := range c.Body { + p.projectBodyNode(cRef, module, c.ID, local, bn) + } + + // Nested closures → GO_DECLARES (callable → closure). + for _, inner := range c.Callables { + innerRef := p.projectCallable(cRef, module, inner) + p.b.Edge(RelDeclares, cRef, innerRef, nil, "") + } + return cRef +} + +func (p *projector) projectBodyNode(callableRef NodeRef, module, callableID, local string, bn v2.BodyNode) { + id := callableID + "@" + local + bRef := p.b.Node( + []string{LabelBodyNode, MarkerLabel}, + "id", id, + Props{ + "kind": bn.Kind, + "is_goroutine": boolOrNil(bn.IsGoroutine), + "is_deferred": boolOrNil(bn.IsDeferred), + "span_json": spanJSON(bn.Span), + }, + module, + ) + p.b.Edge(RelHasBodyNode, callableRef, bRef, nil, "") + + // A resolved callee (backfilled at L2) → GO_RESOLVES_TO. The callee is + // already a resolved can:// id; route through the plain edge only when it + // names a declared callable (an external callee stays null in JSON and + // emits no edge here). + if bn.Callee != nil && p.declared[*bn.Callee] { + p.b.Edge(RelResolvesTo, bRef, + NodeRef{Label: SymbolLabel, KeyProp: "id", Value: *bn.Callee}, nil, "") + } +} + +// ─── imports ────────────────────────────────────────────────────────────────── + +// projectImports aggregates a module's imports into one GO_IMPORTS edge per +// resolved target. An import whose path names an in-tree module resolves to that +// :GoModule; otherwise it resolves to an id-keyed :GoExternal ghost (stdlib / +// third-party), which never dangles. +func (p *projector) projectImports(modRef NodeRef, m v2.Module) { + appID := p.analysis.Application.ID + for _, imp := range m.Imports { + // Is the import path an in-project module? (symbol_table is keyed by + // relative file path; Go imports name a package path, not a file — so an + // in-project hit is rare and handled conservatively via the id space.) + targetModuleID := v2.ModuleID(appID, imp.Path) + if _, ok := p.analysis.Application.SymbolTable[imp.Path]; ok { + p.b.Edge(RelImports, modRef, + NodeRef{Label: LabelModule, KeyProp: "id", Value: targetModuleID}, + Props{"alias": emptyToNil(imp.Alias)}, imp.Path) + continue + } + // External/stdlib → :GoExternal ghost keyed by import path. + ghostID := appID + "/@external/" + imp.Path + ghostRef := p.b.Node( + []string{SymbolLabel, LabelExternal, MarkerLabel}, + "id", ghostID, + Props{"name": imp.Name, "path": imp.Path}, + "", // shared — belongs to no owning module + ) + p.b.Edge(RelImports, modRef, ghostRef, Props{"alias": emptyToNil(imp.Alias)}, imp.Path) + } +} + +// ─── call graph ───────────────────────────────────────────────────────────── + +// projectCallGraph emits the aggregated GO_CALLS edges. Endpoints are already +// can:// callable ids in the v2 edge; a declared id resolves to its callable +// node, anything else to an id-keyed external ghost — so no GO_CALLS endpoint +// ever dangles. +func (p *projector) projectCallGraph() { + for _, e := range p.analysis.Application.CallGraph { + from := p.callEndpoint(e.Src) + to := p.callEndpoint(e.Dst) + p.b.Edge(RelCalls, from, to, Props{ + "weight": intOrNil(e.Weight), + "prov": stringsOrNil(e.Prov), + }, "") + } +} + +// callEndpoint resolves a call endpoint id to a NodeRef. A declared callable +// addresses its node by the shared symbol label; an undeclared id is minted as +// an external ghost node (so the MATCH finds something at load time). +func (p *projector) callEndpoint(id string) NodeRef { + ref := NodeRef{Label: SymbolLabel, KeyProp: "id", Value: id} + if p.declared[id] { + return ref + } + // Emit the ghost so the endpoint is a real node, not a dangling MATCH. + p.b.Node([]string{SymbolLabel, LabelExternal, MarkerLabel}, "id", id, + Props{"name": lastSegment(id)}, "") + return ref +} + +// ─── id collection ───────────────────────────────────────────────────────── + +// collectDeclaredIDs walks the tree once to gather every declared type and +// callable id, so endpoint resolution can distinguish a declared target from an +// external one without emitting it twice. +func collectDeclaredIDs(a *v2.Analysis) map[string]bool { + ids := make(map[string]bool) + var walkCallable func(c v2.Callable) + walkCallable = func(c v2.Callable) { + ids[c.ID] = true + for _, inner := range c.Callables { + walkCallable(inner) + } + } + for _, m := range a.Application.SymbolTable { + for _, t := range m.Types { + ids[t.ID] = true + for _, meth := range t.Callables { + walkCallable(meth) + } + } + for _, fn := range m.Functions { + walkCallable(fn) + } + } + return ids +} + +// ─── flattening helpers ────────────────────────────────────────────────────── + +// spanJSON flattens a span to a compact JSON string property. (A span is a +// nested map, which Neo4j cannot store as a property, so it rides as *_json — +// the same approach the siblings use for variant-shaped fields.) +func spanJSON(s v2.Span) Value { + data, err := json.Marshal(s) + if err != nil { + return nil + } + return string(data) +} + +func metricsJSON(m map[string]int) Value { + if len(m) == 0 { + return nil + } + data, err := json.Marshal(m) + if err != nil { + return nil + } + return string(data) +} + +// emptyToNil returns nil for "" so prune() drops the property (absence = no fact). +func emptyToNil(s string) Value { + if s == "" { + return nil + } + return s +} + +// boolOrNil returns nil for false so a false flag is simply absent (matching the +// JSON omitempty on is_goroutine/is_deferred). +func boolOrNil(b bool) Value { + if !b { + return nil + } + return true +} + +func intOrNil(n int) Value { + if n == 0 { + return nil + } + return n +} + +func stringsOrNil(s []string) Value { + if len(s) == 0 { + return nil + } + return s +} + +// lastSegment returns the final "/" or "@"-delimited segment of a can:// id, +// used as a human-readable signature/name fallback for ghosts. +func lastSegment(id string) string { + last := id + for i := len(id) - 1; i >= 0; i-- { + if id[i] == '/' || id[i] == '@' { + last = id[i+1:] + break + } + } + return last +} diff --git a/internal/neo4j/project_test.go b/internal/neo4j/project_test.go new file mode 100644 index 0000000..971e219 --- /dev/null +++ b/internal/neo4j/project_test.go @@ -0,0 +1,158 @@ +package neo4j + +import ( + "path/filepath" + "runtime" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" +) + +// M3 gate: the pure projector. These project the multipackage fixture at L2 +// (the real builder + resolver, the same wiring -a 2 uses) and assert the +// conformance contract, no-dangling, two-projection id agreement, determinism, +// and the Go-specific shapes. + +// buildMultipackageL2 builds the fixture through the real symbol-table builder +// and resolver and emits the v2 L2 analysis — the input to Project. +func buildMultipackageL2(t *testing.T) *v2.Analysis { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "multipackage")) + if err != nil { + t.Fatalf("resolving fixture dir: %v", err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges} + return v2emit.Emit(app, "multipackage", dir, 2, "neo4j-gate") +} + +func TestProject_NoDanglingEndpoints(t *testing.T) { + rows := Project(buildMultipackageL2(t)) + + // Every (label, value) that was emitted as a node. + emitted := make(map[string]bool, len(rows.Nodes)) + for _, n := range rows.Nodes { + for _, l := range n.Labels { + emitted[l+"\x00"+n.Value] = true + } + } + for _, e := range rows.Edges { + if !emitted[e.From.Label+"\x00"+e.From.Value] { + t.Errorf("%s edge FROM a non-emitted node: %s %s", e.Type, e.From.Label, e.From.Value) + } + if !emitted[e.To.Label+"\x00"+e.To.Value] { + t.Errorf("%s edge TO a non-emitted node: %s %s", e.Type, e.To.Label, e.To.Value) + } + } +} + +func TestProject_IdParityWithJSON(t *testing.T) { + a := buildMultipackageL2(t) + rows := Project(a) + + // Every can:// id in the JSON tree (application/module/type/callable/field) + // must appear as a graph node, and vice versa for can://-keyed nodes. + jsonIDs := collectDeclaredIDs(a) + jsonIDs[a.Application.ID] = true + for _, m := range a.Application.SymbolTable { + jsonIDs[m.ID] = true + for _, t := range m.Types { + for _, f := range t.Fields { + jsonIDs[f.ID] = true + } + } + } + + graphIDs := make(map[string]bool) + for _, n := range rows.Nodes { + // Only can:// nodes participate (externals/ghosts are can:// too, but are + // not in the JSON tree, so compare the declared-tree subset). + graphIDs[n.Value] = true + } + + for id := range jsonIDs { + if !graphIDs[id] { + t.Errorf("JSON tree id has no graph node: %s", id) + } + } +} + +func TestProject_Deterministic(t *testing.T) { + a := buildMultipackageL2(t) + r1 := Project(a) + r2 := Project(a) + if len(r1.Nodes) != len(r2.Nodes) || len(r1.Edges) != len(r2.Edges) { + t.Fatalf("row counts differ across runs") + } + for i := range r1.Nodes { + if r1.Nodes[i].Value != r2.Nodes[i].Value || r1.Nodes[i].Labels[0] != r2.Nodes[i].Labels[0] { + t.Fatalf("node order not deterministic at %d", i) + } + } + for i := range r1.Edges { + if edgeSortKey(r1.Edges[i]) != edgeSortKey(r2.Edges[i]) { + t.Fatalf("edge order not deterministic at %d", i) + } + } +} + +func TestProject_GoSpecificShapes(t *testing.T) { + rows := Project(buildMultipackageL2(t)) + + var sawType, sawCallable, sawModule, sawCalls bool + var typeKindsSeen = map[string]bool{} + for _, n := range rows.Nodes { + switch n.Labels[0] { + case LabelModule: + sawModule = true + } + if hasLabel(n, LabelType) { + sawType = true + if k, ok := n.Props["kind"].(string); ok { + typeKindsSeen[k] = true + } + } + if hasLabel(n, LabelCallable) { + sawCallable = true + } + } + for _, e := range rows.Edges { + if e.Type == RelCalls { + sawCalls = true + } + } + if !sawModule || !sawType || !sawCallable { + t.Errorf("expected module/type/callable nodes; module=%v type=%v callable=%v", sawModule, sawType, sawCallable) + } + if !sawCalls { + t.Error("expected at least one GO_CALLS edge at L2") + } + // Type kinds must be from the declared vocabulary. + for k := range typeKindsSeen { + switch k { + case "struct", "interface", "alias", "defined": + default: + t.Errorf("undeclared type kind %q", k) + } + } +} + +func hasLabel(n NodeRow, label string) bool { + for _, l := range n.Labels { + if l == label { + return true + } + } + return false +} diff --git a/internal/neo4j/schema_test.go b/internal/neo4j/schema_test.go index b552a19..8634009 100644 --- a/internal/neo4j/schema_test.go +++ b/internal/neo4j/schema_test.go @@ -130,9 +130,81 @@ func TestSchemaDocument_RoundTrips(t *testing.T) { } } -// TestConformance_ProjectorEmitsOnlyDeclared is the M3 anti-drift scaffold. -// When project() lands, replace the skip with: project a fixture at L2, collect -// every (label, prop) and (relType) it emits, and assert each is declared here. +// TestConformance_ProjectorEmitsOnlyDeclared is the anti-drift gate (filled by +// M3). It projects the multipackage fixture at L2 and asserts the real emitter +// never produces a label, relationship type, or property that schema.go does +// not declare — so the contract cannot silently drift from the projector. func TestConformance_ProjectorEmitsOnlyDeclared(t *testing.T) { - t.Skip("scaffold: filled by M3 (project.go) — asserts the projector emits no undeclared label/relationship/property") + rows := Project(buildMultipackageL2(t)) + + // Declared label set + per-label allowed property set. A node's labels[0] is + // the MERGE label, which for the symbol family is the SHARED SymbolLabel; the + // specific kind (GoType/GoCallable/GoExternal) is then an extra label. So the + // NodeSpec is resolved by the specific kind label, and properties validate + // against it. + labelDecl := declaredLabels() + validMerge := map[string]bool{MarkerLabel: true, SymbolLabel: true} + for _, n := range Nodes { + validMerge[n.MergeLabel] = true + } + + for _, n := range rows.Nodes { + // labels[0] must be a legitimate MERGE label (a node's own label, the + // shared SymbolLabel, or — never — the marker). + if !validMerge[n.Labels[0]] { + t.Errorf("node MERGE label %q is undeclared", n.Labels[0]) + continue + } + // Resolve the spec by the specific kind label (labels[0] when it is the + // node's own merge label, else the first extra label that is a declared kind). + spec, ok := labelDecl[n.Labels[0]] + if !ok { + for _, l := range n.Labels[1:] { + if s, found := labelDecl[l]; found { + spec, ok = s, true + break + } + } + } + if !ok { + t.Errorf("node with labels %v matches no declared kind", n.Labels) + continue + } + // Every extra label must be a declared kind or the marker. + for _, l := range n.Labels[1:] { + if _, isKind := labelDecl[l]; !isKind && l != MarkerLabel { + t.Errorf("node carries undeclared extra label %q", l) + } + } + allowed := map[string]bool{spec.Key: true} + for _, pr := range spec.Props { + allowed[pr] = true + } + for pk := range n.Props { + if !allowed[pk] { + t.Errorf("node %s carries undeclared property %q", spec.Label, pk) + } + } + } + + relDecl := make(map[string]RelSpec, len(Rels)) + for _, r := range Rels { + relDecl[r.Type] = r + } + for _, e := range rows.Edges { + spec, ok := relDecl[e.Type] + if !ok { + t.Errorf("relationship type %q is undeclared", e.Type) + continue + } + allowed := map[string]bool{} + for _, pr := range spec.Props { + allowed[pr] = true + } + for pk := range e.Props { + if !allowed[pk] { + t.Errorf("relationship %s carries undeclared property %q", e.Type, pk) + } + } + } } From 7f5028aa5c02796ed9801174ab9f2b0bb2d2aad8 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Sun, 4 Oct 2026 13:43:55 -0400 Subject: [PATCH 22/29] feat(neo4j): snapshot writer -> graph.cypher MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone M4 of the Neo4j projection. cypher.go renders a GraphRows bag into a self-contained, idempotent graph.cypher script that rebuilds this application's subgraph from scratch. Block order: constraints + indexes (IF NOT EXISTS) -> scoped wipe -> batched UNWIND...MERGE nodes -> batched UNWIND...MERGE edges (500/batch). The scoped wipe is anchored on :GoCanNode and the can:/// id prefix, with the root id matched separately (the root is outside its own descendant prefix). It uses applicationPrefix, so a bag with no bare application root emits NO destructive statement (visible as a SKIPPED comment) — one application by construction, never the store, never a sibling analyzer's graph. Symbol-family nodes MERGE on the shared :GoSymbol label and SET the specific kind as an extra label; keyed edges MERGE on {_k: ...}. Gate (green, DB-free): the script carries DDL + wipe + node/edge batches; is byte-identical across runs; every DETACH DELETE is scoped on :GoCanNode + the prefix; a no-root bag emits no wipe. --- internal/neo4j/cypher.go | 217 ++++++++++++++++++++++++++++++++++ internal/neo4j/cypher_test.go | 99 ++++++++++++++++ 2 files changed, 316 insertions(+) create mode 100644 internal/neo4j/cypher.go create mode 100644 internal/neo4j/cypher_test.go diff --git a/internal/neo4j/cypher.go b/internal/neo4j/cypher.go new file mode 100644 index 0000000..548886b --- /dev/null +++ b/internal/neo4j/cypher.go @@ -0,0 +1,217 @@ +package neo4j + +import ( + "fmt" + "sort" + "strings" +) + +// Milestone M4: the snapshot writer. RenderCypher turns a GraphRows bag into a +// self-contained, idempotent graph.cypher script. Running it (e.g. +// `cypher-shell < graph.cypher`) rebuilds this application's subgraph from +// scratch. Block order: +// +// 1. constraints + indexes (IF NOT EXISTS) — first, so MERGE seeks an index; +// 2. scoped wipe of this app's prior subgraph — anchored on :GoCanNode and the +// can:/// id prefix, with the root id matched separately (the root is +// outside its own descendant prefix). Rows with no proper application root +// emit NO wipe (refused visibly), so a hand-built bag can't delete anything; +// 3. batched UNWIND … MERGE for nodes, grouped by full label set + key prop; +// 4. batched UNWIND … MERGE for edges, grouped by (type, endpoint labels/keys, +// keyed?). +// +// A static script has no view of the live DB, so it expresses the FULL truth; +// incremental updates are the Bolt writer's job (M5). Determinism: identical +// GraphRows → byte-identical script (RowBuilder.Finish already sorted the bag). + +// batchSize is the UNWIND batch size (parity with the siblings' 500). +const batchSize = 500 + +// RenderCypher renders the full snapshot script. appID is the application root +// id used to scope the wipe; it is normally rows' application node value. +func RenderCypher(rows GraphRows, appID string) string { + var b strings.Builder + + writeDDL(&b) + writeScopedWipe(&b, appID) + writeNodeBatches(&b, rows.Nodes) + writeEdgeBatches(&b, rows.Edges) + + return b.String() +} + +func writeDDL(b *strings.Builder) { + b.WriteString("// ─── constraints & indexes ───\n") + for _, c := range UniquenessConstraints() { + fmt.Fprintf(b, "CREATE CONSTRAINT %s IF NOT EXISTS FOR (n:%s) REQUIRE n.%s IS UNIQUE;\n", + c.Name, c.Label, c.Property) + } + for _, idx := range Indexes() { + fmt.Fprintf(b, "CREATE INDEX %s IF NOT EXISTS FOR (n:%s) ON (n.%s);\n", + idx.Name, idx.Label, idx.Property) + } + b.WriteByte('\n') +} + +// writeScopedWipe emits the DETACH DELETE scoped to this app. It refuses to emit +// anything unless appID is a bare application id (applicationPrefix), so the +// blast radius is exactly one application — never the whole store, never a +// sibling analyzer's :Go* graph (they lack :GoCanNode), never a second Go app +// sharing a module path. +func writeScopedWipe(b *strings.Builder, appID string) { + root, ok := applicationPrefix(appID) + if !ok { + b.WriteString("// ─── scoped wipe SKIPPED: no bare application root id ───\n\n") + return + } + desc := descendantPrefix(root) + b.WriteString("// ─── scoped wipe (this application only) ───\n") + // Descendants: everything under can:///. + fmt.Fprintf(b, "MATCH (n:%s) WHERE n.id STARTS WITH %s DETACH DELETE n;\n", + MarkerLabel, cypherString(desc)) + // The root itself is outside its own descendant prefix. + fmt.Fprintf(b, "MATCH (n:%s) WHERE n.id = %s DETACH DELETE n;\n", + MarkerLabel, cypherString(root)) + b.WriteByte('\n') +} + +// ─── node batches ───────────────────────────────────────────────────────────── + +func writeNodeBatches(b *strings.Builder, nodes []NodeRow) { + if len(nodes) == 0 { + return + } + b.WriteString("// ─── nodes ───\n") + // Group by (full label set + key prop) so one UNWIND handles a homogeneous + // batch. Keys sorted for deterministic output. + groups := map[string][]NodeRow{} + for _, n := range nodes { + groups[nodeGroupKey(n)] = append(groups[nodeGroupKey(n)], n) + } + for _, gk := range sortedKeys(groups) { + g := groups[gk] + mergeLabel := g[0].Labels[0] + keyProp := g[0].KeyProp + extraLabels := g[0].Labels[1:] + + for _, chunk := range chunkNodes(g, batchSize) { + b.WriteString("UNWIND [\n") + for i, n := range chunk { + sep := "," + if i == len(chunk)-1 { + sep = "" + } + fmt.Fprintf(b, " {k: %s, p: %s}%s\n", cypherString(n.Value), cypherMap(n.Props), sep) + } + b.WriteString("] AS row\n") + fmt.Fprintf(b, "MERGE (n:%s {%s: row.k})\n", mergeLabel, keyProp) + b.WriteString("SET n += row.p") + for _, l := range extraLabels { + fmt.Fprintf(b, "\nSET n:%s", l) + } + b.WriteString(";\n") + } + } + b.WriteByte('\n') +} + +// ─── edge batches ───────────────────────────────────────────────────────────── + +func writeEdgeBatches(b *strings.Builder, edges []EdgeRow) { + if len(edges) == 0 { + return + } + b.WriteString("// ─── relationships ───\n") + groups := map[string][]EdgeRow{} + for _, e := range edges { + groups[edgeGroupKey(e)] = append(groups[edgeGroupKey(e)], e) + } + for _, gk := range sortedEdgeKeys(groups) { + g := groups[gk] + e0 := g[0] + keyed := e0.Key != "" + + for _, chunk := range chunkEdges(g, batchSize) { + b.WriteString("UNWIND [\n") + for i, e := range chunk { + sep := "," + if i == len(chunk)-1 { + sep = "" + } + if keyed { + fmt.Fprintf(b, " {a: %s, b: %s, k: %s, p: %s}%s\n", + cypherString(e.From.Value), cypherString(e.To.Value), cypherString(e.Key), cypherMap(e.Props), sep) + } else { + fmt.Fprintf(b, " {a: %s, b: %s, p: %s}%s\n", + cypherString(e.From.Value), cypherString(e.To.Value), cypherMap(e.Props), sep) + } + } + b.WriteString("] AS row\n") + fmt.Fprintf(b, "MATCH (a:%s {%s: row.a})\n", e0.From.Label, e0.From.KeyProp) + fmt.Fprintf(b, "MATCH (b:%s {%s: row.b})\n", e0.To.Label, e0.To.KeyProp) + if keyed { + fmt.Fprintf(b, "MERGE (a)-[r:%s {_k: row.k}]->(b)\n", e0.Type) + } else { + fmt.Fprintf(b, "MERGE (a)-[r:%s]->(b)\n", e0.Type) + } + b.WriteString("SET r += row.p;\n") + } + } + b.WriteByte('\n') +} + +// ─── grouping keys & chunking ───────────────────────────────────────────────── + +func nodeGroupKey(n NodeRow) string { + return strings.Join(n.Labels, ":") + "\x00" + n.KeyProp +} + +func edgeGroupKey(e EdgeRow) string { + keyed := "0" + if e.Key != "" { + keyed = "1" + } + return strings.Join([]string{e.Type, e.From.Label, e.From.KeyProp, e.To.Label, e.To.KeyProp, keyed}, "\x00") +} + +func chunkNodes(s []NodeRow, n int) [][]NodeRow { + var out [][]NodeRow + for i := 0; i < len(s); i += n { + end := i + n + if end > len(s) { + end = len(s) + } + out = append(out, s[i:end]) + } + return out +} + +func chunkEdges(s []EdgeRow, n int) [][]EdgeRow { + var out [][]EdgeRow + for i := 0; i < len(s); i += n { + end := i + n + if end > len(s) { + end = len(s) + } + out = append(out, s[i:end]) + } + return out +} + +func sortedKeys(m map[string][]NodeRow) []string { + ks := make([]string, 0, len(m)) + for k := range m { + ks = append(ks, k) + } + sort.Strings(ks) + return ks +} + +func sortedEdgeKeys(m map[string][]EdgeRow) []string { + ks := make([]string, 0, len(m)) + for k := range m { + ks = append(ks, k) + } + sort.Strings(ks) + return ks +} diff --git a/internal/neo4j/cypher_test.go b/internal/neo4j/cypher_test.go new file mode 100644 index 0000000..7d3f8fb --- /dev/null +++ b/internal/neo4j/cypher_test.go @@ -0,0 +1,99 @@ +package neo4j + +import ( + "strings" + "testing" +) + +// M4 gate: the snapshot writer. The DDL + scoped wipe + batched node/edge MERGE +// blocks are present, deterministic, and prefix-scoped; a bag with no proper +// application root emits no destructive statement. + +func renderFixture(t *testing.T) string { + t.Helper() + a := buildMultipackageL2(t) + rows := Project(a) + return RenderCypher(rows, a.Application.ID) +} + +func TestCypher_HasDDLWipeAndBatches(t *testing.T) { + script := renderFixture(t) + for _, want := range []string{ + "CREATE CONSTRAINT", + "CREATE INDEX gocannode_id", + "DETACH DELETE", + "UNWIND [", + "MERGE (n:", + "MERGE (a)-[r:GO_", + } { + if !strings.Contains(script, want) { + t.Errorf("script missing %q", want) + } + } +} + +func TestCypher_Deterministic(t *testing.T) { + a := buildMultipackageL2(t) + s1 := RenderCypher(Project(a), a.Application.ID) + s2 := RenderCypher(Project(a), a.Application.ID) + if s1 != s2 { + t.Error("graph.cypher is not byte-identical across runs") + } +} + +func TestCypher_WipeIsPrefixScopedAndAnchored(t *testing.T) { + script := renderFixture(t) + // The wipe must match on the :GoCanNode marker and the can:/// prefix, + // never a bare label like (:GoModule) or the whole store. + if !strings.Contains(script, "MATCH (n:GoCanNode) WHERE n.id STARTS WITH 'can://go/multipackage/'") { + t.Errorf("wipe not anchored on :GoCanNode + descendant prefix:\n%s", wipeBlock(script)) + } + if !strings.Contains(script, "MATCH (n:GoCanNode) WHERE n.id = 'can://go/multipackage'") { + t.Errorf("wipe does not match the root id separately:\n%s", wipeBlock(script)) + } + // A destructive statement must never be unscoped. + for _, line := range strings.Split(script, "\n") { + if strings.Contains(line, "DETACH DELETE") && !strings.Contains(line, "WHERE") { + t.Errorf("unscoped DETACH DELETE: %q", line) + } + } +} + +func TestCypher_NoRootEmitsNoWipe(t *testing.T) { + // A hand-built bag with no bare application id must emit NO destructive + // statement — the applicationPrefix refusal in action. + b := NewRowBuilder() + b.Node([]string{LabelCallable, SymbolLabel}, "id", "can://go/app/x.go/f", nil, "x.go") + script := RenderCypher(b.Finish(), "can://go/app/x.go/f") // not a bare app id + if strings.Contains(script, "DETACH DELETE") { + t.Errorf("a bag with no bare application root must emit no wipe:\n%s", script) + } + if !strings.Contains(script, "scoped wipe SKIPPED") { + t.Error("the skipped wipe should be visible in a comment") + } +} + +func TestCypher_SymbolFamilyMergesOnSharedLabel(t *testing.T) { + // Nodes in the symbol family (labels[0]=GoSymbol) must MERGE on :GoSymbol and + // SET the specific kind as an extra label. + script := renderFixture(t) + if !strings.Contains(script, "MERGE (n:GoSymbol {id: row.k})") { + t.Error("symbol-family nodes must MERGE on the shared :GoSymbol label") + } + if !strings.Contains(script, "SET n:GoCallable") && !strings.Contains(script, "SET n:GoType") { + t.Error("symbol-family nodes must SET their specific kind as an extra label") + } +} + +// wipeBlock extracts the wipe section for readable failure output. +func wipeBlock(script string) string { + start := strings.Index(script, "scoped wipe") + if start < 0 { + return "(no wipe block)" + } + end := strings.Index(script[start:], "// ─── nodes") + if end < 0 { + end = len(script) - start + } + return script[start : start+end] +} From f2a562a0c8a1d78c8d1b8249afe24821b05c4590 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Sun, 4 Oct 2026 13:47:26 -0400 Subject: [PATCH 23/29] feat(neo4j): incremental Bolt writer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone M5 of the Neo4j projection. bolt.go pushes a GraphRows bag into a live Neo4j over Bolt, updating only what changed with the module subgraph as the unit of idempotent replacement. Adds github.com/neo4j/neo4j-go-driver/v5 (pure Go — CGO_ENABLED=0 static builds still link; only exercised when EmitNeo4j runs with a --neo4j-uri, off the JSON path). - ensure constraints + indexes; schema-version gate (a stored-vs-producer mismatch forces a full re-upsert, never a delete); - partition nodes by owning module (NodeRow.module); shared nodes (externals, neutral repo nodes) are MERGE-only, always upserted; - diff each module's content_hash against the DB -> the changed set; - per changed module, upsert; on --eager, purge its owned subtree first; - upsert edges; on a full + eager run, orphan-prune vanished modules. A push NEVER deletes by default (#171): the purge/prune steps run under --eager only; lazy is purely additive. Every destructive statement is scoped on the can:/// id prefix anchored on :GoCanNode (#173), extracted into pure query-builders so the scope invariant is unit-testable. Gate: DB-free unit tests (partitioning, module-hash, param marshalling, and the destructive-queries-are-scoped-and-anchored invariant) run on every `go test`; a live round-trip (push, idempotent re-push, lazy-never-deletes) is in bolt_integration_test.go behind the `neo4j_it` build tag. Static CGO-free build verified. --- go.mod | 1 + go.sum | 2 + internal/neo4j/bolt.go | 434 ++++++++++++++++++++++++ internal/neo4j/bolt_integration_test.go | 130 +++++++ internal/neo4j/bolt_test.go | 107 ++++++ 5 files changed, 674 insertions(+) create mode 100644 internal/neo4j/bolt.go create mode 100644 internal/neo4j/bolt_integration_test.go create mode 100644 internal/neo4j/bolt_test.go diff --git a/go.mod b/go.mod index c913971..e514b7e 100644 --- a/go.mod +++ b/go.mod @@ -3,6 +3,7 @@ module github.com/codellm-devkit/codeanalyzer-go go 1.25.0 require ( + github.com/neo4j/neo4j-go-driver/v5 v5.28.5 github.com/spf13/cobra v1.8.0 golang.org/x/tools v0.46.0 ) diff --git a/go.sum b/go.sum index 88530cf..c30bbc2 100644 --- a/go.sum +++ b/go.sum @@ -3,6 +3,8 @@ github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= +github.com/neo4j/neo4j-go-driver/v5 v5.28.5 h1:YfqEKXt8AxsXRMGu73eNipYWCSXodVI4dl2I8iwcavA= +github.com/neo4j/neo4j-go-driver/v5 v5.28.5/go.mod h1:Vff8OwT7QpLm7L2yYr85XNWe9Rbqlbeb9asNXJTHO4k= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= diff --git a/internal/neo4j/bolt.go b/internal/neo4j/bolt.go new file mode 100644 index 0000000..2c9a848 --- /dev/null +++ b/internal/neo4j/bolt.go @@ -0,0 +1,434 @@ +package neo4j + +import ( + "context" + "fmt" + "sort" + + "github.com/neo4j/neo4j-go-driver/v5/neo4j" +) + +// Milestone M5: the incremental Bolt writer. It pushes a GraphRows bag into a +// live Neo4j over Bolt, updating ONLY what changed, with the module subgraph as +// the unit of idempotent replacement. The neo4j driver is pure Go, so a +// CGO_ENABLED=0 static build still links; it is only exercised when EmitNeo4j +// runs with a --neo4j-uri, never on the JSON path. +// +// Algorithm: +// 1. ensure constraints + indexes; +// 2. schema-version gate: read :GoApplication.schema_version scoped to this +// app's id; a mismatch forces a full re-UPSERT (never a delete — that is +// --eager's job); +// 3. partition nodes by owning module (NodeRow.module); shared nodes (externals, +// neutral repo nodes) have none and are MERGE-only, always upserted; +// 4. --eager? scoped purge of everything under the app prefix, then rebuild; +// 5. diff each module's content_hash against the DB → the changed set; +// 6. per changed module: on --eager, purge its owned edges + stale decls, then +// upsert its nodes; WITHOUT --eager the push only adds/updates; +// 7. upsert edges owned by a changed module (or shared); +// 8. on a full --eager run, orphan-prune modules whose source vanished. +// +// A push NEVER deletes by default (#171): steps 4/6/8 run under --eager only. +// Every destructive statement is scoped on the can:/// id prefix (#173), +// anchored on :GoCanNode so it seeks an index. + +// BoltConfig carries the connection + behavior knobs (populated by the facade +// from AnalysisOptions). +type BoltConfig struct { + URI string + User string + Password string + Database string + // Eager enables the destructive reconciliation steps. Default (lazy) is + // purely additive: MERGE-upsert, nothing removed. + Eager bool + // FullRun is true when the analysis saw every file (not a --target-files + // subset); orphan pruning only runs on a full, eager run. + FullRun bool +} + +// WriteBolt pushes rows into the live DB per BoltConfig. appID scopes every +// destructive statement; it must be the application root id. +func WriteBolt(ctx context.Context, rows GraphRows, appID string, cfg BoltConfig) error { + driver, err := neo4j.NewDriverWithContext(cfg.URI, neo4j.BasicAuth(cfg.User, cfg.Password, "")) + if err != nil { + return fmt.Errorf("neo4j driver: %w", err) + } + defer driver.Close(ctx) + if err := driver.VerifyConnectivity(ctx); err != nil { + return fmt.Errorf("neo4j connectivity: %w", err) + } + + session := driver.NewSession(ctx, neo4j.SessionConfig{DatabaseName: cfg.Database}) + defer session.Close(ctx) + + w := &boltWriter{ctx: ctx, session: session, rows: rows, appID: appID, cfg: cfg} + return w.run() +} + +type boltWriter struct { + ctx context.Context + session neo4j.SessionWithContext + rows GraphRows + appID string + cfg BoltConfig +} + +func (w *boltWriter) run() error { + if err := w.ensureDDL(); err != nil { + return err + } + + // Schema-version gate: a producer/consumer mismatch forces a full re-upsert + // (it never deletes). We detect it but the upsert path below is already a + // full MERGE of every current node, so a mismatch simply means we don't + // trust the per-module content_hash diff and upsert everything. + forceFull, err := w.schemaVersionMismatch() + if err != nil { + return err + } + + root, scoped := applicationPrefix(w.appID) + if w.cfg.Eager && scoped { + if err := w.eagerPurge(root); err != nil { + return err + } + } + + // Partition nodes by owning module; shared nodes (module == "") are always + // upserted. + byModule, shared := partitionNodes(w.rows.Nodes) + + // Diff content_hash per module unless a mismatch/eager forces everything. + changed := map[string]bool{} + if forceFull || w.cfg.Eager { + for m := range byModule { + changed[m] = true + } + } else { + changed, err = w.changedModules(byModule) + if err != nil { + return err + } + } + + // Upsert shared nodes (MERGE-only, every run) then changed-module nodes. + if err := w.upsertNodes(shared); err != nil { + return err + } + for _, m := range sortedModuleKeys(byModule) { + if !changed[m] { + continue + } + if w.cfg.Eager && scoped { + if err := w.purgeModule(m); err != nil { + return err + } + } + if err := w.upsertNodes(byModule[m]); err != nil { + return err + } + } + + // Upsert edges owned by a changed module (owner = source node's module) or + // shared. Determining ownership precisely needs the source node's module; we + // conservatively upsert every edge whose source id is under a changed module + // prefix, plus all shared/app-level edges. A MERGE-upsert of an unchanged + // edge is a harmless no-op. + if err := w.upsertEdges(w.rows.Edges); err != nil { + return err + } + + if w.cfg.Eager && w.cfg.FullRun && scoped { + if err := w.pruneVanishedModules(byModule); err != nil { + return err + } + } + return nil +} + +// ─── DDL + version gate ─────────────────────────────────────────────────────── + +func (w *boltWriter) ensureDDL() error { + return w.write(func(tx neo4j.ManagedTransaction) error { + for _, c := range UniquenessConstraints() { + q := fmt.Sprintf("CREATE CONSTRAINT %s IF NOT EXISTS FOR (n:%s) REQUIRE n.%s IS UNIQUE", + c.Name, c.Label, c.Property) + if _, err := tx.Run(w.ctx, q, nil); err != nil { + return err + } + } + for _, idx := range Indexes() { + q := fmt.Sprintf("CREATE INDEX %s IF NOT EXISTS FOR (n:%s) ON (n.%s)", + idx.Name, idx.Label, idx.Property) + if _, err := tx.Run(w.ctx, q, nil); err != nil { + return err + } + } + return nil + }) +} + +// schemaVersionMismatch reads the stored :GoApplication.schema_version for THIS +// app and compares to the producer's SchemaVersion. A mismatch (or absence) +// returns true → the caller upserts everything rather than trusting the diff. +func (w *boltWriter) schemaVersionMismatch() (bool, error) { + var stored string + err := w.read(func(tx neo4j.ManagedTransaction) error { + res, err := tx.Run(w.ctx, + "MATCH (a:GoApplication {id: $id}) RETURN a.schema_version AS v", + map[string]any{"id": w.appID}) + if err != nil { + return err + } + if res.Next(w.ctx) { + if v, ok := res.Record().Get("v"); ok && v != nil { + stored, _ = v.(string) + } + } + return res.Err() + }) + if err != nil { + return false, err + } + // No stored version (first push) is not a "mismatch" that forces anything + // beyond the normal upsert; a DIFFERENT stored version does. + return stored != "" && stored != SchemaVersion, nil +} + +// ─── destructive steps (--eager only) ───────────────────────────────────────── + +func (w *boltWriter) eagerPurge(root string) error { + q, params := eagerPurgeQuery(root) + return w.write(func(tx neo4j.ManagedTransaction) error { + _, err := tx.Run(w.ctx, q, params) + return err + }) +} + +// purgeModule deletes the edges and declarations a module owned, scoped to its +// id prefix, so a re-analyzed module's stale facts are removed before re-upsert. +func (w *boltWriter) purgeModule(moduleID string) error { + q, params := purgeModuleQuery(moduleID) + return w.write(func(tx neo4j.ManagedTransaction) error { + _, err := tx.Run(w.ctx, q, params) + return err + }) +} + +// eagerPurgeQuery / purgeModuleQuery are pure so the destructive-scope gate can +// assert they are anchored on :GoCanNode and the can:// prefix WITHOUT a live +// DB. Every destructive statement is scoped — never an unqualified DELETE. +func eagerPurgeQuery(root string) (string, map[string]any) { + // Batched to avoid exhausting the transaction memory limit on large apps. + return "MATCH (n:GoCanNode) WHERE n.id STARTS WITH $p OR n.id = $root " + + "CALL (n) { DETACH DELETE n } IN TRANSACTIONS OF 5000 ROWS", + map[string]any{"p": descendantPrefix(root), "root": root} +} + +func purgeModuleQuery(moduleID string) (string, map[string]any) { + return "MATCH (n:GoCanNode) WHERE n.id = $m OR n.id STARTS WITH $p DETACH DELETE n", + map[string]any{"m": moduleID, "p": descendantPrefix(moduleID)} +} + +// pruneVanishedModules removes modules under the app prefix that this run no +// longer emits (their source file vanished). Full + eager only. +func (w *boltWriter) pruneVanishedModules(byModule map[string][]NodeRow) error { + present := make([]string, 0, len(byModule)) + for m := range byModule { + present = append(present, m) + } + root, _ := applicationPrefix(w.appID) + desc := descendantPrefix(root) + return w.write(func(tx neo4j.ManagedTransaction) error { + _, err := tx.Run(w.ctx, + "MATCH (m:GoModule) WHERE m.id STARTS WITH $p AND NOT m.id IN $present "+ + "MATCH (n:GoCanNode) WHERE n.id = m.id OR n.id STARTS WITH m.id + '/' DETACH DELETE n", + map[string]any{"p": desc, "present": present}) + return err + }) +} + +// ─── diff ───────────────────────────────────────────────────────────────────── + +// changedModules compares each module's content_hash in rows against the DB. +func (w *boltWriter) changedModules(byModule map[string][]NodeRow) (map[string]bool, error) { + changed := map[string]bool{} + for _, m := range sortedModuleKeys(byModule) { + localHash := moduleHash(byModule[m]) + var dbHash string + err := w.read(func(tx neo4j.ManagedTransaction) error { + res, err := tx.Run(w.ctx, + "MATCH (m:GoModule {id: $id}) RETURN m.content_hash AS h", + map[string]any{"id": m}) + if err != nil { + return err + } + if res.Next(w.ctx) { + if h, ok := res.Record().Get("h"); ok && h != nil { + dbHash, _ = h.(string) + } + } + return res.Err() + }) + if err != nil { + return nil, err + } + // Changed when the hash differs or the module is new, or when either side + // has no hash (can't prove unchanged → upsert). + if localHash == "" || dbHash == "" || localHash != dbHash { + changed[m] = true + } + } + return changed, nil +} + +// ─── upserts ────────────────────────────────────────────────────────────────── + +func (w *boltWriter) upsertNodes(nodes []NodeRow) error { + if len(nodes) == 0 { + return nil + } + groups := map[string][]NodeRow{} + for _, n := range nodes { + groups[nodeGroupKey(n)] = append(groups[nodeGroupKey(n)], n) + } + for _, gk := range sortedKeys(groups) { + g := groups[gk] + mergeLabel := g[0].Labels[0] + keyProp := g[0].KeyProp + extra := g[0].Labels[1:] + setLabels := "" + for _, l := range extra { + setLabels += " SET n:" + l + } + q := fmt.Sprintf("UNWIND $rows AS row MERGE (n:%s {%s: row.k}) SET n += row.p%s", + mergeLabel, keyProp, setLabels) + params := map[string]any{"rows": nodeParams(g)} + if err := w.write(func(tx neo4j.ManagedTransaction) error { + _, err := tx.Run(w.ctx, q, params) + return err + }); err != nil { + return err + } + } + return nil +} + +func (w *boltWriter) upsertEdges(edges []EdgeRow) error { + if len(edges) == 0 { + return nil + } + groups := map[string][]EdgeRow{} + for _, e := range edges { + groups[edgeGroupKey(e)] = append(groups[edgeGroupKey(e)], e) + } + for _, gk := range sortedEdgeKeys(groups) { + g := groups[gk] + e0 := g[0] + keyed := e0.Key != "" + var mergeClause string + if keyed { + mergeClause = fmt.Sprintf("MERGE (a)-[r:%s {_k: row.k}]->(b)", e0.Type) + } else { + mergeClause = fmt.Sprintf("MERGE (a)-[r:%s]->(b)", e0.Type) + } + q := fmt.Sprintf("UNWIND $rows AS row MATCH (a:%s {%s: row.a}) MATCH (b:%s {%s: row.b}) %s SET r += row.p", + e0.From.Label, e0.From.KeyProp, e0.To.Label, e0.To.KeyProp, mergeClause) + params := map[string]any{"rows": edgeParams(g, keyed)} + if err := w.write(func(tx neo4j.ManagedTransaction) error { + _, err := tx.Run(w.ctx, q, params) + return err + }); err != nil { + return err + } + } + return nil +} + +// ─── tx helpers ─────────────────────────────────────────────────────────────── + +func (w *boltWriter) write(fn func(neo4j.ManagedTransaction) error) error { + _, err := w.session.ExecuteWrite(w.ctx, func(tx neo4j.ManagedTransaction) (any, error) { + return nil, fn(tx) + }) + return err +} + +func (w *boltWriter) read(fn func(neo4j.ManagedTransaction) error) error { + _, err := w.session.ExecuteRead(w.ctx, func(tx neo4j.ManagedTransaction) (any, error) { + return nil, fn(tx) + }) + return err +} + +// ─── param marshalling ──────────────────────────────────────────────────────── + +// nodeParams converts node rows to driver params. Props map directly (the +// driver stores Go ints as integers, strings as strings, []string as a list). +func nodeParams(g []NodeRow) []map[string]any { + out := make([]map[string]any, len(g)) + for i, n := range g { + out[i] = map[string]any{"k": n.Value, "p": propsToAny(n.Props)} + } + return out +} + +func edgeParams(g []EdgeRow, keyed bool) []map[string]any { + out := make([]map[string]any, len(g)) + for i, e := range g { + m := map[string]any{"a": e.From.Value, "b": e.To.Value, "p": propsToAny(e.Props)} + if keyed { + m["k"] = e.Key + } + out[i] = m + } + return out +} + +func propsToAny(p Props) map[string]any { + if len(p) == 0 { + return map[string]any{} + } + out := make(map[string]any, len(p)) + for k, v := range p { + out[k] = v + } + return out +} + +// ─── partitioning & hashing ─────────────────────────────────────────────────── + +func partitionNodes(nodes []NodeRow) (byModule map[string][]NodeRow, shared []NodeRow) { + byModule = map[string][]NodeRow{} + for _, n := range nodes { + if n.Module == "" { + shared = append(shared, n) + continue + } + byModule[n.Module] = append(byModule[n.Module], n) + } + return byModule, shared +} + +// moduleHash returns the content_hash the projector put on the :GoModule node in +// this group (the module's own node carries it), else "". +func moduleHash(nodes []NodeRow) string { + for _, n := range nodes { + if n.Labels[0] == LabelModule { + if h, ok := n.Props["content_hash"].(string); ok { + return h + } + } + } + return "" +} + +func sortedModuleKeys(m map[string][]NodeRow) []string { + ks := make([]string, 0, len(m)) + for k := range m { + ks = append(ks, k) + } + sort.Strings(ks) + return ks +} diff --git a/internal/neo4j/bolt_integration_test.go b/internal/neo4j/bolt_integration_test.go new file mode 100644 index 0000000..72ac8b7 --- /dev/null +++ b/internal/neo4j/bolt_integration_test.go @@ -0,0 +1,130 @@ +//go:build neo4j_it + +// Live-DB integration test for the Bolt writer. Gated behind the `neo4j_it` +// build tag so the default `go test ./...` (and CI without a database) skips it. +// +// Run against a disposable Neo4j: +// +// docker run --rm -p7687:7687 -e NEO4J_AUTH=neo4j/testpass neo4j:5 +// NEO4J_TEST_URI=bolt://localhost:7687 NEO4J_TEST_PASS=testpass \ +// go test -tags neo4j_it ./internal/neo4j/ -run Integration -v +// +// It asserts the behaviors the DB-free unit tests cannot: a push materializes +// the subgraph, a re-push is idempotent (same counts), a lazy push never +// deletes, and an eager push reconciles. +package neo4j + +import ( + "context" + "os" + "testing" + + "github.com/neo4j/neo4j-go-driver/v5/neo4j" +) + +func itConfig(t *testing.T) BoltConfig { + t.Helper() + uri := os.Getenv("NEO4J_TEST_URI") + if uri == "" { + t.Skip("set NEO4J_TEST_URI to run the live Bolt integration test") + } + pass := os.Getenv("NEO4J_TEST_PASS") + if pass == "" { + pass = "neo4j" + } + return BoltConfig{URI: uri, User: "neo4j", Password: pass, Database: os.Getenv("NEO4J_TEST_DB")} +} + +func TestIntegration_PushThenIdempotentRepush(t *testing.T) { + ctx := context.Background() + cfg := itConfig(t) + cfg.Eager = true // start from a clean, reconciled state + cfg.FullRun = true + + a := buildMultipackageL2(t) + rows := Project(a) + + if err := WriteBolt(ctx, rows, a.Application.ID, cfg); err != nil { + t.Fatalf("first push: %v", err) + } + n1, e1 := countSubgraph(t, ctx, cfg, a.Application.ID) + + // A second eager push of identical rows must yield identical counts. + if err := WriteBolt(ctx, rows, a.Application.ID, cfg); err != nil { + t.Fatalf("second push: %v", err) + } + n2, e2 := countSubgraph(t, ctx, cfg, a.Application.ID) + + if n1 != n2 || e1 != e2 { + t.Errorf("re-push not idempotent: nodes %d->%d, edges %d->%d", n1, n2, e1, e2) + } + if n1 == 0 { + t.Error("push materialized no nodes") + } +} + +func TestIntegration_LazyPushNeverDeletes(t *testing.T) { + ctx := context.Background() + cfg := itConfig(t) + a := buildMultipackageL2(t) + rows := Project(a) + + // Seed eagerly. + seed := cfg + seed.Eager, seed.FullRun = true, true + if err := WriteBolt(ctx, rows, a.Application.ID, seed); err != nil { + t.Fatalf("seed: %v", err) + } + before, _ := countSubgraph(t, ctx, cfg, a.Application.ID) + + // A lazy push of an EMPTY row set must delete nothing. + empty := GraphRows{} + if err := WriteBolt(ctx, empty, a.Application.ID, cfg); err != nil { + t.Fatalf("lazy empty push: %v", err) + } + after, _ := countSubgraph(t, ctx, cfg, a.Application.ID) + if after < before { + t.Errorf("lazy push deleted nodes: %d -> %d (lazy must be additive-only)", before, after) + } +} + +// countSubgraph opens a short-lived driver and counts the nodes and +// relationships under this app's can:// prefix (anchored on :GoCanNode). +func countSubgraph(t *testing.T, ctx context.Context, cfg BoltConfig, appID string) (nodes, edges int) { + t.Helper() + drv, err := neo4j.NewDriverWithContext(cfg.URI, neo4j.BasicAuth(cfg.User, cfg.Password, "")) + if err != nil { + t.Fatalf("driver: %v", err) + } + defer drv.Close(ctx) + sess := drv.NewSession(ctx, neo4j.SessionConfig{DatabaseName: cfg.Database}) + defer sess.Close(ctx) + + root, _ := applicationPrefix(appID) + desc := descendantPrefix(root) + _, err = sess.ExecuteRead(ctx, func(tx neo4j.ManagedTransaction) (any, error) { + nr, err := tx.Run(ctx, + "MATCH (n:GoCanNode) WHERE n.id = $root OR n.id STARTS WITH $p RETURN count(n) AS c", + map[string]any{"root": root, "p": desc}) + if err != nil { + return nil, err + } + if nr.Next(ctx) { + nodes = int(nr.Record().Values[0].(int64)) + } + er, err := tx.Run(ctx, + "MATCH (n:GoCanNode)-[r]->() WHERE n.id = $root OR n.id STARTS WITH $p RETURN count(r) AS c", + map[string]any{"root": root, "p": desc}) + if err != nil { + return nil, err + } + if er.Next(ctx) { + edges = int(er.Record().Values[0].(int64)) + } + return nil, nil + }) + if err != nil { + t.Fatalf("count query: %v", err) + } + return nodes, edges +} diff --git a/internal/neo4j/bolt_test.go b/internal/neo4j/bolt_test.go new file mode 100644 index 0000000..49a8e41 --- /dev/null +++ b/internal/neo4j/bolt_test.go @@ -0,0 +1,107 @@ +package neo4j + +import ( + "strings" + "testing" +) + +// M5 unit gate (DB-free): the pure pieces of the Bolt writer — node +// partitioning, module-hash extraction, param marshalling, and the critical +// invariant that every destructive query is scoped on :GoCanNode + the can:// +// prefix (never an unqualified DELETE). The live-DB behavior (diff, additive +// vs eager) is exercised by bolt_integration_test.go behind the `neo4j_it` +// build tag. + +func TestPartitionNodes_SharedHaveNoModule(t *testing.T) { + b := NewRowBuilder() + b.Node([]string{LabelModule, MarkerLabel}, "id", "can://go/app/x.go", Props{"content_hash": "h1"}, "x.go") + b.Node([]string{LabelCallable, SymbolLabel, MarkerLabel}, "id", "can://go/app/x.go/f", nil, "x.go") + b.Node([]string{SymbolLabel, LabelExternal, MarkerLabel}, "id", "can://go/app/@external/fmt", nil, "") // shared + rows := b.Finish() + + byModule, shared := partitionNodes(rows.Nodes) + if len(byModule["x.go"]) != 2 { + t.Errorf("module x.go should own 2 nodes, got %d", len(byModule["x.go"])) + } + if len(shared) != 1 { + t.Errorf("expected 1 shared (no-module) node, got %d", len(shared)) + } +} + +func TestModuleHash_ReadsModuleNode(t *testing.T) { + nodes := []NodeRow{ + {Labels: []string{LabelCallable}, Value: "can://go/app/x.go/f"}, + {Labels: []string{LabelModule}, Value: "can://go/app/x.go", Props: Props{"content_hash": "abc123"}}, + } + if got := moduleHash(nodes); got != "abc123" { + t.Errorf("moduleHash = %q, want abc123", got) + } + if got := moduleHash(nodes[:1]); got != "" { + t.Errorf("moduleHash with no module node = %q, want empty", got) + } +} + +func TestDestructiveQueries_AreScopedAndAnchored(t *testing.T) { + // Both destructive queries MUST filter on :GoCanNode and bind the prefix as a + // parameter — never an unqualified DETACH DELETE. + eq, ep := eagerPurgeQuery("can://go/app") + assertScoped(t, "eagerPurge", eq, ep) + if ep["p"] != "can://go/app/" || ep["root"] != "can://go/app" { + t.Errorf("eagerPurge params not the app prefix+root: %v", ep) + } + + pq, pp := purgeModuleQuery("can://go/app/x.go") + assertScoped(t, "purgeModule", pq, pp) + if pp["p"] != "can://go/app/x.go/" || pp["m"] != "can://go/app/x.go" { + t.Errorf("purgeModule params not the module prefix+id: %v", pp) + } +} + +func assertScoped(t *testing.T, name, q string, params map[string]any) { + t.Helper() + if !strings.Contains(q, "DETACH DELETE") { + t.Fatalf("%s: not a destructive query?", name) + } + if !strings.Contains(q, ":GoCanNode") { + t.Errorf("%s: destructive query not anchored on :GoCanNode: %q", name, q) + } + if !strings.Contains(q, "n.id STARTS WITH $p") { + t.Errorf("%s: destructive query not prefix-scoped on a bound param: %q", name, q) + } + // The prefix param must end in "/" (strict descendant containment). + if p, _ := params["p"].(string); !strings.HasSuffix(p, "/") { + t.Errorf("%s: prefix param %q must end in / for strict containment", name, p) + } +} + +func TestParamMarshalling(t *testing.T) { + g := []NodeRow{{Value: "id1", Props: Props{"kind": "struct", "n": 3}}} + ps := nodeParams(g) + if ps[0]["k"] != "id1" { + t.Errorf("node key not marshalled: %v", ps[0]) + } + p := ps[0]["p"].(map[string]any) + if p["kind"] != "struct" || p["n"] != 3 { + t.Errorf("node props not marshalled: %v", p) + } + + e := []EdgeRow{{From: NodeRef{Value: "a"}, To: NodeRef{Value: "b"}, Key: "kind", Props: Props{"var": "x"}}} + ep := edgeParams(e, true) + if ep[0]["a"] != "a" || ep[0]["b"] != "b" || ep[0]["k"] != "kind" { + t.Errorf("edge endpoints/key not marshalled: %v", ep[0]) + } + // Non-keyed edges omit k. + ep2 := edgeParams(e, false) + if _, ok := ep2[0]["k"]; ok { + t.Error("non-keyed edge must not carry a _k param") + } +} + +func TestBoltConfig_LazyIsDefault(t *testing.T) { + // Documents the invariant: a zero-value BoltConfig is lazy (Eager false), so + // the default push is purely additive. + var cfg BoltConfig + if cfg.Eager { + t.Error("zero-value BoltConfig must be lazy (Eager=false) — additive by default") + } +} From 3dc7fb2d916177bd326c2ef60a294aab575a948c Mon Sep 17 00:00:00 2001 From: lambertpw Date: Sun, 4 Oct 2026 13:49:23 -0400 Subject: [PATCH 24/29] =?UTF-8?q?feat(neo4j):=20facade=20+=20CLI=20wiring?= =?UTF-8?q?=20=E2=80=94=20--emit=20neo4j=20is=20live?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone M6 of the Neo4j projection, completing the analyzer arc. EmitNeo4j projects the v2 Analysis (the SAME tree the JSON path emits) and writes it: --neo4j-uri set → live Bolt push (incremental, additive unless --eager); absent → a self-contained graph.cypher snapshot in the output dir. CLI: --emit neo4j is unstubbed and shares the analysis path, but is ALWAYS full-depth — it forces level 2 (max implemented) + canonical v2, and an explicit -a/--analysis-level alongside it is a flag error ("--analysis-level does not apply to --emit neo4j; the graph is always projected at full depth"). The Neo4j connection config threads through from the existing env-precedence flags; --eager and a full (non-target-files) run gate the destructive push steps. Gate (green): --emit neo4j with no URI writes graph.cypher end-to-end (verified byte-identical across runs — idempotent); the flag-error gate rejects -a with a clear non-zero exit; --emit schema still writes the static contract. Static CGO-free binary builds and runs. Full suite green. --- cmd/codeanalyzer/main.go | 40 ++++++++++++++++++++++++++++---- cmd/codeanalyzer/main_test.go | 35 ++++++++++++++++++++++++---- internal/neo4j/emit.go | 43 +++++++++++++++++++++++++++++++---- 3 files changed, 104 insertions(+), 14 deletions(-) diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 2fa3455..fdaead4 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -15,6 +15,7 @@ import ( "github.com/codellm-devkit/codeanalyzer-go/internal/core" "github.com/codellm-devkit/codeanalyzer-go/internal/neo4j" "github.com/codellm-devkit/codeanalyzer-go/internal/options" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) @@ -69,14 +70,22 @@ via CLDK(language="go").analysis(project_path=...).`, } // --emit selects the output projection. schema is a static contract - // that needs no --input: handle it here and return. neo4j is still - // rejected until its writers land (M4–M6) — never a silent fallback - // to JSON. - switch options.EmitTarget(emit) { + // that needs no --input: handle it here and return. neo4j shares the + // analysis path below but is ALWAYS full-depth, so an explicit + // -a/--analysis-level alongside it is a flag error (the level gate + // applies to the JSON path only). + emitTarget := options.EmitTarget(emit) + switch emitTarget { case options.EmitJSON: // valid — falls through to the JSON path below. case options.EmitNeo4j: - return fmt.Errorf("--emit neo4j is not yet implemented; use --emit json") + if cmd.Flags().Changed("analysis-level") { + return fmt.Errorf("--analysis-level does not apply to --emit neo4j; the graph is always projected at full depth") + } + // Force full depth + canonical v2: the graph carries every + // implemented level's facts in both projections. + level = int(options.LevelCallGraph) + analysisSchema = 2 case options.EmitSchema: path, err := neo4j.EmitSchema(outputDir) if err != nil { @@ -141,6 +150,27 @@ via CLDK(language="go").analysis(project_path=...).`, return err } + // Neo4j projection: build the canonical v2 payload (the SAME tree the + // JSON path emits) and project it. --neo4j-uri set → live Bolt push; + // absent → a graph.cypher snapshot in the output dir. + if emitTarget == options.EmitNeo4j { + payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion) + boltCfg := neo4j.BoltConfig{ + URI: opts.Neo4jURI, + User: opts.Neo4jUser, + Password: opts.Neo4jPassword, + Database: opts.Neo4jDatabase, + Eager: opts.Eager, + FullRun: len(opts.TargetFiles) == 0, + } + msg, err := neo4j.EmitNeo4j(cmd.Context(), payload, outputDir, boltCfg) + if err != nil { + return err + } + cmd.Println(msg) + return nil + } + // When no --output dir is given, write JSON to cobra's output // writer so tests can capture it via cmd.SetOut. Both paths render // through core so the v1/v2 schema selection stays in one place. diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index 00c0f03..26c3c90 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -67,14 +67,39 @@ func TestRootCmd_UnknownEmitReturnsError(t *testing.T) { } } -func TestRootCmd_EmitNeo4jNotImplemented(t *testing.T) { +func TestRootCmd_EmitNeo4jWritesGraphCypher(t *testing.T) { + // --emit neo4j with no --neo4j-uri writes a self-contained graph.cypher + // snapshot into the output dir (M6). td := cliTestdataDir() - _, _, err := runCmd("--input", filepath.Join(td, "greeter"), "--emit", "neo4j") + out := t.TempDir() + stdout, _, err := runCmd("--input", filepath.Join(td, "multipackage"), "--emit", "neo4j", "--output", out) + if err != nil { + t.Fatalf("--emit neo4j should succeed; got %v", err) + } + data, err := os.ReadFile(filepath.Join(out, "graph.cypher")) + if err != nil { + t.Fatalf("graph.cypher not written: %v", err) + } + script := string(data) + for _, want := range []string{"CREATE CONSTRAINT", "DETACH DELETE", "MERGE (n:", "GO_CALLS"} { + if !strings.Contains(script, want) { + t.Errorf("graph.cypher missing %q", want) + } + } + if !strings.Contains(stdout, "graph.cypher") { + t.Errorf("expected stdout to report the written path; got %q", stdout) + } +} + +func TestRootCmd_EmitNeo4jRejectsExplicitAnalysisLevel(t *testing.T) { + // --emit neo4j is always full-depth; combining it with -a is a flag error. + td := cliTestdataDir() + _, _, err := runCmd("--input", filepath.Join(td, "greeter"), "--emit", "neo4j", "-a", "1") if err == nil { - t.Fatal("expected non-zero exit for --emit neo4j, got nil") + t.Fatal("expected a flag error for --emit neo4j -a 1, got nil") } - if !strings.Contains(err.Error(), "not yet implemented") { - t.Errorf("error should say 'not yet implemented'; got %q", err.Error()) + if !strings.Contains(err.Error(), "does not apply to --emit neo4j") { + t.Errorf("error should explain the level gate; got %q", err.Error()) } } diff --git a/internal/neo4j/emit.go b/internal/neo4j/emit.go index 207273f..e5c682a 100644 --- a/internal/neo4j/emit.go +++ b/internal/neo4j/emit.go @@ -1,16 +1,51 @@ package neo4j import ( + "context" "encoding/json" "fmt" "os" "path/filepath" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" ) -// Facade between the CLI and the Neo4j backend. At M1 only EmitSchema exists — -// the static contract serializer, which needs no analyzed project. EmitNeo4j -// (project + write) lands at M6 once the projector (M3) and the writers (M4/M5) -// exist. +// Facade between the CLI and the Neo4j backend: EmitSchema (static contract, +// needs no project) and EmitNeo4j (project the v2 tree, then write — graph.cypher +// snapshot when no URI, live Bolt push when a URI is set). + +// GraphFileName is the snapshot script written when no --neo4j-uri is given. +const GraphFileName = "graph.cypher" + +// EmitNeo4j projects the v2 Analysis into GraphRows and writes it. The projection +// is pure and shared; only the write target differs. When cfg.URI is set it +// pushes over Bolt (incremental, additive unless cfg.Eager); otherwise it writes +// a self-contained graph.cypher snapshot into outputDir (default: current dir). +// It returns a short human-readable description of what was written. +func EmitNeo4j(ctx context.Context, a *v2.Analysis, outputDir string, cfg BoltConfig) (string, error) { + rows := Project(a) + appID := a.Application.ID + + if cfg.URI != "" { + if err := WriteBolt(ctx, rows, appID, cfg); err != nil { + return "", err + } + return fmt.Sprintf("pushed %d nodes, %d relationships to %s", len(rows.Nodes), len(rows.Edges), cfg.URI), nil + } + + if outputDir == "" { + outputDir = "." + } + if err := os.MkdirAll(outputDir, 0o755); err != nil { + return "", fmt.Errorf("create output dir %q: %w", outputDir, err) + } + path := filepath.Join(outputDir, GraphFileName) + script := RenderCypher(rows, appID) + if err := os.WriteFile(path, []byte(script), 0o644); err != nil { + return "", fmt.Errorf("write %s: %w", path, err) + } + return "wrote " + path, nil +} // SchemaFileName is the machine-readable graph contract written by --emit schema. const SchemaFileName = "schema.neo4j.json" From 0c8a3f8ba128d7d11c8606281bf85aa2ebac41d5 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Sun, 4 Oct 2026 20:04:10 -0400 Subject: [PATCH 25/29] feat(cli): default --analysis-schema to 2 (canonical v2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit cango now emits the canonical v2 tree by default; --analysis-schema 1 is the opt-in escape hatch for the legacy v1 shape. Done before the first release, so there are no v1-default consumers to break — the cleanest moment to flip it. - main.go: flag default 1 -> 2, help text reordered to name v2 as default. - main_test.go: DefaultSchemaIsV1 -> DefaultSchemaIsV2 (default output now carries schema_version + application); add AnalysisSchema1EmitsLegacyV1 to pin the escape hatch; the Level1/Level2 call-graph tests now read application.call_graph (v2) instead of the v1 top-level key. - README: the four "v1 is the default" references corrected to v2. The python-sdk GoCodeanalyzer already passes --analysis-schema 2 explicitly and unconditionally, so this does not change SDK behaviour — it fixes the shape a hand-run `cango` produces. --- README.md | 18 +++++++------- cmd/codeanalyzer/main.go | 4 ++-- cmd/codeanalyzer/main_test.go | 44 ++++++++++++++++++++++++++++------- 3 files changed, 47 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index da1235a..cecd8c2 100644 --- a/README.md +++ b/README.md @@ -131,7 +131,7 @@ Aliases: Flags: -a, --analysis-level int Analysis level: 1=symbol table only, 2=+resolver call graph (default 1) - --analysis-schema int Output schema major: 1=legacy v1 shape (default), 2=canonical v2 tree (default 1) + --analysis-schema int Output schema major: 2=canonical v2 tree (default), 1=legacy v1 shape (default 2) --app-name string Application anchor name for can:// ids and Neo4j :Application (default: input dir name) -c, --cache-dir string Cache directory (default: ~/.cldk/go-cache) --codeql Enable CodeQL framework-based call graph (level 2, stub) @@ -204,10 +204,10 @@ cango -i /path/to/go/app --analysis-schema 2 -a 2 -o /path/to/output/ # → writes /path/to/output/analysis.json ``` -> **Pick the schema explicitly.** `--analysis-schema` defaults to `1` (the legacy v1 -> shape). Pass `--analysis-schema 2` for the current **canonical v2** tree (the shape -> the Python SDK's v2 loader and the sections below describe). Omit `-o` to stream the -> JSON to stdout instead of writing a file. +> **Schema.** `--analysis-schema` defaults to `2` — the current **canonical v2** tree (the +> shape the Python SDK's v2 loader and the sections below describe). Pass +> `--analysis-schema 1` only if you need the **legacy v1** shape for backward compatibility. +> Omit `-o` to stream the JSON to stdout instead of writing a file. **3. (Optional) name the application anchor.** `--app-name` sets the `` in every `can://go//…` id and defaults to the input directory's base name. Set it when the @@ -251,10 +251,10 @@ are build artifacts, not project source), so only your own declarations are emit - **`--analysis-schema 2`** — the **canonical CLDK v2** shape (detailed below); the shape the v2 Python SDK loader consumes and the [step-by-step section above](#generating-analysisjson-for-a-go-app) produces. -- **`--analysis-schema 1`** (default) — the **legacy v1** `GoApplication` shape, kept for - backward compatibility. +- **`--analysis-schema 1`** — the **legacy v1** `GoApplication` shape, kept for backward + compatibility (v2 is the default). -### Canonical v2 shape (`--analysis-schema 2`) +### Canonical v2 shape (`--analysis-schema 2`, default) The document is a manifest envelope wrapping one `application` containment tree. Every node carries a `can://go//…` `id`, a `kind`, and a `span`: @@ -347,7 +347,7 @@ Key v2 schema properties: - **`call_graph` edges** — `{src, dst, prov, weight}`; `src`/`dst` are `can://` node ids that exist in the tree (not raw signatures), `prov` is resolver provenance, e.g. `["go/types"]`. -### Legacy v1 shape (`--analysis-schema 1`, default) +### Legacy v1 shape (`--analysis-schema 1`) The v1 root object is `GoApplication`: diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index fdaead4..71f591f 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -195,8 +195,8 @@ via CLDK(language="go").analysis(project_path=...).`, "Application anchor name for can:// ids and Neo4j :Application (default: input dir name)") f.IntVarP(&level, "analysis-level", "a", 1, "Analysis level: 1=symbol table only, 2=+resolver call graph") - f.IntVar(&analysisSchema, "analysis-schema", 1, - "Output schema major: 1=legacy v1 shape (default), 2=canonical v2 tree") + f.IntVar(&analysisSchema, "analysis-schema", 2, + "Output schema major: 2=canonical v2 tree (default), 1=legacy v1 shape") f.StringSliceVarP(&targetFiles, "target-files", "t", nil, "Restrict analysis to specific files (incremental mode)") f.BoolVar(&skipTests, "skip-tests", true, "Skip *_test.go files") diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index 26c3c90..0e827f9 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -212,14 +212,17 @@ func TestRootCmd_Level1ProducesNoCallGraph(t *testing.T) { t.Fatalf("command failed: %v", err) } + // The default is now the v2 shape, where call_graph lives under application (not top-level). var result struct { - CallGraph []interface{} `json:"call_graph"` + Application struct { + CallGraph []interface{} `json:"call_graph"` + } `json:"application"` } if jsonErr := json.Unmarshal([]byte(out), &result); jsonErr != nil { t.Fatalf("stdout is not valid JSON: %v", jsonErr) } - if len(result.CallGraph) != 0 { - t.Errorf("level 1 should produce no call graph edges; got %d", len(result.CallGraph)) + if len(result.Application.CallGraph) != 0 { + t.Errorf("level 1 should produce no call graph edges; got %d", len(result.Application.CallGraph)) } } @@ -235,20 +238,23 @@ func TestRootCmd_Level2ProducesCallGraph(t *testing.T) { t.Fatalf("command failed: %v", err) } + // The default is now the v2 shape, where call_graph lives under application (not top-level). var result struct { - CallGraph []interface{} `json:"call_graph"` + Application struct { + CallGraph []interface{} `json:"call_graph"` + } `json:"application"` } if jsonErr := json.Unmarshal([]byte(out), &result); jsonErr != nil { t.Fatalf("stdout is not valid JSON: %v", jsonErr) } - if len(result.CallGraph) == 0 { + if len(result.Application.CallGraph) == 0 { t.Error("level 2 should produce call graph edges; got none") } } // ── --analysis-schema ────────────────────────────────────────────────────────── -func TestRootCmd_DefaultSchemaIsV1(t *testing.T) { +func TestRootCmd_DefaultSchemaIsV2(t *testing.T) { td := cliTestdataDir() out, _, err := runCmd("--input", filepath.Join(td, "greeter"), "--cache-dir", t.TempDir()) if err != nil { @@ -258,11 +264,33 @@ func TestRootCmd_DefaultSchemaIsV1(t *testing.T) { if jsonErr := json.Unmarshal([]byte(out), &v); jsonErr != nil { t.Fatalf("stdout is not valid JSON: %v", jsonErr) } + if _, ok := v["schema_version"]; !ok { + t.Error("default output should be the v2 shape (top-level schema_version)") + } + if _, ok := v["application"]; !ok { + t.Error("default output should carry an application root (that is v2)") + } +} + +func TestRootCmd_AnalysisSchema1EmitsLegacyV1(t *testing.T) { + td := cliTestdataDir() + out, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-schema", "1", + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("command failed: %v", err) + } + var v map[string]interface{} + if jsonErr := json.Unmarshal([]byte(out), &v); jsonErr != nil { + t.Fatalf("stdout is not valid JSON: %v", jsonErr) + } if _, ok := v["symbol_table"]; !ok { - t.Error("default output should be the v1 shape (top-level symbol_table)") + t.Error("--analysis-schema 1 should still emit the legacy v1 shape (top-level symbol_table)") } if _, ok := v["schema_version"]; ok { - t.Error("default output should not carry schema_version (that is v2)") + t.Error("the legacy v1 shape should not carry schema_version") } } From 3136a7a3094ea7be9cfb869971b712ca426317ef Mon Sep 17 00:00:00 2001 From: lambertpw Date: Tue, 6 Oct 2026 15:33:31 -0400 Subject: [PATCH 26/29] style: resolve golangci-lint findings in neo4j and utils Fix gosimple S1016 in internal/neo4j/ddl.go: convert NodeSpec/RelSpec to NodeDoc/RelDoc via type conversion instead of struct literals. Fix errcheck on best-effort deferred cleanup: - internal/neo4j/bolt.go: driver.Close / session.Close - internal/utils/fs.go: f.Close - internal/core/testsetup_test.go: os.RemoveAll --- internal/core/testsetup_test.go | 2 +- internal/neo4j/bolt.go | 4 ++-- internal/neo4j/ddl.go | 10 ++-------- internal/utils/fs.go | 2 +- 4 files changed, 6 insertions(+), 12 deletions(-) diff --git a/internal/core/testsetup_test.go b/internal/core/testsetup_test.go index 2b14d32..7446447 100644 --- a/internal/core/testsetup_test.go +++ b/internal/core/testsetup_test.go @@ -36,7 +36,7 @@ func runTestMain(m *testing.M) int { fmt.Fprintf(os.Stderr, "testsetup: MkdirTemp: %v\n", err) return 1 } - defer os.RemoveAll(tmpRoot) + defer func() { _ = os.RemoveAll(tmpRoot) }() type fixture struct { name string diff --git a/internal/neo4j/bolt.go b/internal/neo4j/bolt.go index 2c9a848..1a26181 100644 --- a/internal/neo4j/bolt.go +++ b/internal/neo4j/bolt.go @@ -54,13 +54,13 @@ func WriteBolt(ctx context.Context, rows GraphRows, appID string, cfg BoltConfig if err != nil { return fmt.Errorf("neo4j driver: %w", err) } - defer driver.Close(ctx) + defer func() { _ = driver.Close(ctx) }() if err := driver.VerifyConnectivity(ctx); err != nil { return fmt.Errorf("neo4j connectivity: %w", err) } session := driver.NewSession(ctx, neo4j.SessionConfig{DatabaseName: cfg.Database}) - defer session.Close(ctx) + defer func() { _ = session.Close(ctx) }() w := &boltWriter{ctx: ctx, session: session, rows: rows, appID: appID, cfg: cfg} return w.run() diff --git a/internal/neo4j/ddl.go b/internal/neo4j/ddl.go index c292461..eb1e30c 100644 --- a/internal/neo4j/ddl.go +++ b/internal/neo4j/ddl.go @@ -121,17 +121,11 @@ type RelDoc struct { func BuildSchemaDocument() SchemaDocument { nodes := make([]NodeDoc, 0, len(Nodes)) for _, n := range Nodes { - nodes = append(nodes, NodeDoc{ - Label: n.Label, MergeLabel: n.MergeLabel, Key: n.Key, - Props: n.Props, CanNode: n.CanNode, - }) + nodes = append(nodes, NodeDoc(n)) } rels := make([]RelDoc, 0, len(Rels)) for _, r := range Rels { - rels = append(rels, RelDoc{ - Type: r.Type, From: r.From, To: r.To, Props: r.Props, - Discriminant: r.Discriminant, Deferred: r.Deferred, Reserved: r.Reserved, - }) + rels = append(rels, RelDoc(r)) } return SchemaDocument{ SchemaVersion: SchemaVersion, diff --git a/internal/utils/fs.go b/internal/utils/fs.go index 541127c..8d6b43d 100644 --- a/internal/utils/fs.go +++ b/internal/utils/fs.go @@ -59,7 +59,7 @@ func FileHash(path string) (string, error) { if err != nil { return "", err } - defer f.Close() + defer func() { _ = f.Close() }() h := sha256.New() if _, err := io.Copy(h, f); err != nil { return "", err From 2b87d3cbc5c8bab7c1427b83d4f254db6d371cbe Mon Sep 17 00:00:00 2001 From: lambertpw Date: Tue, 6 Oct 2026 22:24:33 -0400 Subject: [PATCH 27/29] Added Neo4j instructions --- README.md | 165 ++++++++++++++++++++++++++++++++++---- scripts/load_and_query.sh | 115 ++++++++++++++++++++++++++ 2 files changed, 266 insertions(+), 14 deletions(-) create mode 100755 scripts/load_and_query.sh diff --git a/README.md b/README.md index cecd8c2..b1ad66c 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,11 @@ The binary is fully self-contained: a single static executable with no runtime d - [Examples](#examples) - [Analysis levels](#analysis-levels) - [Output schema](#output-schema) +- [Neo4j projection (`--emit neo4j`)](#neo4j-projection---emit-neo4j) + - [Emit a snapshot (`graph.cypher`)](#emit-a-snapshot-graphcypher) + - [Push live to a running Neo4j](#push-live-to-a-running-neo4j) + - [Load and query](#load-and-query) + - [Query reference](#query-reference) - [Python SDK (CLDK) integration](#python-sdk-cldk-integration) - [Architecture & Tooling](#architecture--tooling) - [Development](#development) @@ -57,6 +62,9 @@ The binary is fully self-contained: a single static executable with no runtime d dependencies for SDK users. - **CLDK canonical schema** — output is spine-compatible with the Java/Python/TypeScript analyzers, loadable directly by the Python SDK. +- **Neo4j projection** — the same v2 envelope projected into a property graph (`--emit neo4j`), + either as a replayable `graph.cypher` snapshot or pushed live over Bolt, keyed on the identical + `can://` ids so JSON and graph join on one string. ## Installation @@ -130,20 +138,26 @@ Aliases: cango, codeanalyzer-go Flags: - -a, --analysis-level int Analysis level: 1=symbol table only, 2=+resolver call graph (default 1) - --analysis-schema int Output schema major: 2=canonical v2 tree (default), 1=legacy v1 shape (default 2) - --app-name string Application anchor name for can:// ids and Neo4j :Application (default: input dir name) - -c, --cache-dir string Cache directory (default: ~/.cldk/go-cache) - --codeql Enable CodeQL framework-based call graph (level 2, stub) - --eager Force clean rebuild (ignore cache) - -f, --format string Output format: json|msgpack (default "json") - -h, --help help for cango - -i, --input string Project root to analyze (required) - -o, --output string Output directory for analysis.json (default: stdout) - --skip-tests Skip *_test.go files (default true) - -t, --target-files strings Restrict analysis to specific files (incremental mode) - -v, --verbose count Verbosity (repeat for more detail) - --version Print version and exit + -a, --analysis-level int Analysis level: 1=symbol table only, 2=+resolver call graph (default 1) + --analysis-schema int Output schema major: 2=canonical v2 tree (default), 1=legacy v1 shape (default 2) + --app-name string Application anchor name for can:// ids and Neo4j :Application (default: input dir name) + -c, --cache-dir string Cache directory (default: ~/.cldk/go-cache) + --codeql Enable CodeQL framework-based call graph (level 2, stub) + --eager Force clean rebuild (ignore cache) + --emit string Output projection: json|neo4j|schema (default "json") + -f, --format string Output format: json|msgpack (default "json") + -h, --help help for cango + -i, --input string Project root to analyze (required) + -j, --jobs int Worker parallelism (default: CPU cores) + --neo4j-database string Neo4j database (env NEO4J_DATABASE, optional) + --neo4j-password string Neo4j password (env NEO4J_PASSWORD, default neo4j) + --neo4j-uri string Live Bolt push target (env NEO4J_URI); omit to write graph.cypher + --neo4j-user string Neo4j username (env NEO4J_USERNAME, default neo4j) + -o, --output string Output directory for analysis.json (default: stdout) + --skip-tests Skip *_test.go files (default true) + -t, --target-files strings Restrict analysis to specific files (incremental mode) + -v, --verbose count Verbosity (repeat for more detail) + --version Print version and exit ``` ### Examples @@ -394,6 +408,129 @@ Key v1 schema properties: - `GoCallEdge.provenance: List[str]` — resolver identifiers, e.g. `["go/types"]` or `["go/types","codeql"]` - Call edges are **identity-only**: source and target are `GoCallable.signature` strings that exist in the symbol table +## Neo4j projection (`--emit neo4j`) + +`--emit neo4j` projects the **same v2 envelope** into a Neo4j property graph. It is a *second +projection of the identical analysis* — JSON nodes and graph nodes carry the same `can://go//…` +ids, so the two join on one string and are never recomposed. The graph vocabulary is `Go`-prefixed +labels (`:GoModule`, `:GoType`, `:GoCallable`, `:GoField`, `:GoBodyNode`, `:GoExternal`, plus the +shared merge label `:GoSymbol` and marker `:GoCanNode`) and `GO_`-prefixed relationships +(`GO_HAS_MODULE`, `GO_DECLARES`, `GO_HAS_METHOD`, `GO_HAS_FIELD`, `GO_CALLS`, `GO_RESOLVES_TO`, +`GO_IMPORTS`, `GO_EMBEDS`, `GO_SATISFIES`, …). + +There are two output paths, chosen by whether `--neo4j-uri` is set: + +| | `--neo4j-uri` unset | `--neo4j-uri` set | +|---|---|---| +| **What happens** | Writes a replayable `graph.cypher` snapshot | Pushes the graph live over Bolt into a running Neo4j | +| **Where** | `<-o dir>/graph.cypher` (or stdout) | The target database | +| **Use for** | Review, diffing, loading on your own schedule, CI | A DB you already have running | + +Both paths are **scoped to one application**: every destructive statement is anchored on +`:GoCanNode` and the `can://go//` id prefix, so re-emitting replaces exactly that app's subgraph +and never touches another app (or a sibling-language graph) in the same database. The projection is +**deterministic** — a second emit is byte-identical. + +> **`--emit neo4j` is always full depth.** The graph is projected at the max implemented level (L2 +> today — symbol table + call graph) and canonical schema v2. Do **not** pass `-a`/`--analysis-level` +> with it — an explicit level is a flag error (`--analysis-level does not apply to --emit neo4j`), +> because there is no shallower graph to ask for. + +### Emit a snapshot (`graph.cypher`) + +```bash +cango -i /path/to/go/app --emit neo4j -o ./out/ +# → writes ./out/graph.cypher +``` + +The snapshot is a self-contained script: constraints + indexes, a scoped wipe of this app's subgraph, +then `UNWIND … MERGE` batches for nodes and relationships. Replay it into any Neo4j with +`cypher-shell` (see below). Omit `-o` to stream the Cypher to stdout. + +### Push live to a running Neo4j + +Set `--neo4j-uri` (or the `NEO4J_URI` env var) to push directly over Bolt instead of writing a file: + +```bash +./cango -i /path/to/go/app --emit neo4j \ + --neo4j-uri bolt://127.0.0.1:7687 \ + --neo4j-user neo4j --neo4j-password +``` + +Credentials and target also read from the environment (`NEO4J_URI`, `NEO4J_USERNAME`, +`NEO4J_PASSWORD`, `NEO4J_DATABASE`), so you can keep them out of your shell history: + +```bash +export NEO4J_URI=bolt://127.0.0.1:7687 +export NEO4J_PASSWORD= +cango -i /path/to/go/app --emit neo4j +``` + +> Use `127.0.0.1` rather than `localhost` if anything else (an SSH tunnel, another DB) might also be +> listening on `7687` — `localhost` can resolve to the wrong listener. + +### Load and query + +If you emitted a `graph.cypher` snapshot, load it into a running Neo4j with `cypher-shell`: + +```bash +cypher-shell -a bolt://127.0.0.1:7687 -u neo4j -p < ./out/graph.cypher +``` + +A convenience harness, [`scripts/load_and_query.sh`](scripts/load_and_query.sh), loads the snapshot +and runs a set of sample queries — it auto-detects `cypher-shell` from a Neo4j Desktop or Homebrew +install. It reads `graph.cypher` from its own directory, so emit the snapshot into `scripts/` first +(`-o scripts`): + +```bash +# brew install neo4j && neo4j start # (or Neo4j Desktop) — then set a password +cango -i /path/to/go/app --emit neo4j -o scripts # writes scripts/graph.cypher + +NEO4J_PASSWORD= ./scripts/load_and_query.sh # load + query +NEO4J_PASSWORD= ./scripts/load_and_query.sh --query # queries only +NEO4J_PASSWORD= ./scripts/load_and_query.sh --check # just test the connection +``` + +Or explore visually in Neo4j Browser at `http://127.0.0.1:7474` (connect to `bolt://127.0.0.1:7687`), +paste a query, and swap `RETURN p` for `RETURN …` to draw the graph. + +### Query reference + +Always **scope by the `can://` id prefix** (an index-backed seek via the `gocannode_id` range index), +never by a bare label scan — replace `` with your `--app-name`: + +```cypher +// Node counts by label. +MATCH (n:GoCanNode) RETURN labels(n)[0] AS label, count(*) AS n ORDER BY n DESC; + +// Relationship counts by type. +MATCH ()-[r]->() RETURN type(r) AS rel, count(*) AS n ORDER BY n DESC; + +// Everything owned by one application (prefix seek). +MATCH (n:GoCanNode) WHERE n.id STARTS WITH 'can://go//' RETURN count(n) AS app_nodes; + +// Types ranked by method count (the "god struct" finder). +MATCH (t:GoType) OPTIONAL MATCH (t)-[:GO_HAS_METHOD]->(m) +RETURN t.kind AS kind, t.signature AS type, count(m) AS methods ORDER BY methods DESC; + +// Most-called callables (in-degree) — the hot utilities. +MATCH (:GoCallable)-[:GO_CALLS]->(b:GoCallable) +RETURN b.signature AS callee, count(*) AS called_by ORDER BY called_by DESC LIMIT 15; + +// Everything Command.Execute reaches, transitively (any depth). +MATCH (:GoCallable {signature:'github.com/spf13/cobra.Command.Execute'})-[:GO_CALLS*]->(b) +RETURN DISTINCT b.signature AS reachable ORDER BY reachable; + +// External / stdlib packages reached via imports. +MATCH (:GoCanNode)-[:GO_IMPORTS]->(x:GoExternal) RETURN DISTINCT x.path AS import ORDER BY import; +``` + +The full, commented catalog — overview/sanity, containment tree, call graph (callers, callees, +reachability, shortest path, entry points, leaves), call-site resolution, imports, inheritance, +metrics, and scoped maintenance — lives in +[`Eval/Neo4j_eval/queries.cypher`](Eval/Neo4j_eval/queries.cypher), validated against the +`spf13/cobra` graph. + ## Python SDK (CLDK) integration ```python diff --git a/scripts/load_and_query.sh b/scripts/load_and_query.sh new file mode 100755 index 0000000..80a2767 --- /dev/null +++ b/scripts/load_and_query.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# +# Load graph.cypher into a NATIVE Neo4j (Neo4j Desktop or a +# Homebrew/tarball install — no Docker) and run sample queries against it. +# +# ./load_and_query.sh # load graph.cypher, then run the sample queries +# ./load_and_query.sh --query # skip the load, just re-run the queries +# ./load_and_query.sh --check # only verify the connection + find cypher-shell +# +# Connection (override via env): +# NEO4J_URI default bolt://localhost:7687 +# NEO4J_USER default neo4j +# NEO4J_PASSWORD default neo4j (set to your DB's password) +# CYPHER_SHELL explicit path to the cypher-shell binary (else auto-detected) +# +# Example: +# NEO4J_PASSWORD=mypass ./load_and_query.sh +# +# Prerequisite: a running Neo4j and its `cypher-shell`. Get them via EITHER: +# • Neo4j Desktop (neo4j.com/download): create + start a local DBMS; this +# script auto-finds the cypher-shell it bundles. +# • Homebrew: `brew install neo4j && neo4j start` (cypher-shell lands on PATH). +set -euo pipefail + +URI="${NEO4J_URI:-bolt://localhost:7687}" +USER="${NEO4J_USER:-neo4j}" +PASS="${NEO4J_PASSWORD:-neo4j}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CYPHER_FILE="$SCRIPT_DIR/graph.cypher" + +# find_cypher_shell locates the binary: an explicit $CYPHER_SHELL, then PATH, +# then common Neo4j Desktop / Homebrew locations. +find_cypher_shell() { + if [ -n "${CYPHER_SHELL:-}" ] && [ -x "$CYPHER_SHELL" ]; then echo "$CYPHER_SHELL"; return 0; fi + if command -v cypher-shell >/dev/null 2>&1; then command -v cypher-shell; return 0; fi + # Neo4j Desktop bundles it under its application support tree; Homebrew under its cellar. + local c + for c in \ + "$HOME/Library/Application Support/Neo4j Desktop/Application/"*/cypher-shell/bin/cypher-shell \ + "$HOME/Library/Application Support/com.Neo4j.Relate/Data/dbmss/"*/bin/cypher-shell \ + /opt/homebrew/bin/cypher-shell \ + /usr/local/bin/cypher-shell ; do + if [ -x "$c" ]; then echo "$c"; return 0; fi + done + return 1 +} + +CS="$(find_cypher_shell || true)" +if [ -z "$CS" ]; then + cat >&2 </dev/null 2>&1; then + echo "connection OK." + else + cat >&2 < if your DB uses another) +EOF + exit 1 + fi +} + +load() { + [ -f "$CYPHER_FILE" ] || { echo "missing $CYPHER_FILE — run 'cango -i --emit neo4j -o Eval/Neo4j_eval' first" >&2; exit 1; } + echo "loading $(basename "$CYPHER_FILE") ($(wc -c < "$CYPHER_FILE" | tr -d ' ') bytes) ..." + shell < "$CYPHER_FILE" >/dev/null + echo "loaded." +} + +q() { + echo + echo "── $1" + shell "$2" || true +} + +queries() { + q "node counts by label" \ + "MATCH (n:GoCanNode) RETURN labels(n)[0] AS label, count(*) AS n ORDER BY n DESC" + q "relationship counts by type" \ + "MATCH ()-[r]->() RETURN type(r) AS rel, count(*) AS n ORDER BY n DESC" + q "everything scoped to the cobra app (prefix seek)" \ + "MATCH (n:GoCanNode) WHERE n.id STARTS WITH 'can://go/cobra' RETURN count(n) AS cobra_nodes" + q "sample call graph edges" \ + "MATCH (a:GoCallable)-[:GO_CALLS]->(b) RETURN a.signature AS caller, b.signature AS callee LIMIT 15" + q "types and how many methods each has" \ + "MATCH (t:GoType) OPTIONAL MATCH (t)-[:GO_HAS_METHOD]->(m) RETURN t.kind AS kind, t.signature AS type, count(m) AS methods ORDER BY methods DESC LIMIT 15" + q "interfaces structurally satisfied (GO_SATISFIES)" \ + "MATCH (t:GoType)-[:GO_SATISFIES]->(i:GoType) RETURN t.signature AS type, i.signature AS satisfies LIMIT 15" + q "external / stdlib packages reached via imports" \ + "MATCH (:GoCanNode)-[:GO_IMPORTS]->(x:GoExternal) RETURN DISTINCT x.path AS import ORDER BY import" + q "most-called callables (in-degree)" \ + "MATCH (:GoCallable)-[:GO_CALLS]->(b:GoCallable) RETURN b.signature AS callee, count(*) AS called_by ORDER BY called_by DESC LIMIT 10" +} + +case "${1:-}" in + --check) check ;; + --query) check; queries ;; + *) check; load; queries + echo; echo "Re-run queries only: $0 --query" ;; +esac From e92de21d1299b13b5fd1e91f86784b551e0fee12 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Wed, 7 Oct 2026 12:22:54 -0400 Subject: [PATCH 28/29] fix(emit): deterministic call_graph for duplicate init signatures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Go permits many package-level `func init()` per package (one per file), all sharing the signature ".init". These are distinct callables with distinct can:// ids (the id embeds the declaring file via its module), but the emitter's flat signature->id index collapsed them with last-writer-wins while ranging a Go map. The surviving id — and therefore which init an edge's source resolved to — varied run to run, so the call_graph edge set was nondeterministic and failed the release determinism gate (byte-identical across runs and -j values). The downstream edge sort was correct but ordered an unstable set. Resolve an edge's source by (signature, declaring file) instead of bare signature: GoCallEdge now carries SourcePath (the source callable's file, already known where the edge is built), and emitCallGraph routes src through a (signature, path)-keyed index so each init edge points to the init that actually owns its call site. The flat index is kept for the unique common case (and for targets, which are never init) and is now deterministic on collision (smallest id wins) as a backstop. No change to the shared signatureOf canonicalizer; the Neo4j projection reuses the emitter's ids and inherits the fix with no change of its own. Add a fresh-build determinism gate over the chi fixture (the only one with multiple init per package): it rebuilds the symbol table + call graph from disk 16 times and asserts a byte-identical call_graph, catching the class the existing single-build determinism test could not. A focused test asserts the three middleware init nodes are distinct and each init-source edge resolves to its own file. --- internal/schema/schema.go | 11 ++ internal/semantic_analysis/call_graph.go | 5 + .../v2emit/determinism_freshbuild_test.go | 153 ++++++++++++++++++ internal/syntactic_analysis/v2emit/emit.go | 7 +- .../syntactic_analysis/v2emit/sigindex.go | 78 +++++++-- 5 files changed, 241 insertions(+), 13 deletions(-) create mode 100644 internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go diff --git a/internal/schema/schema.go b/internal/schema/schema.go index 6eedd79..c02690e 100644 --- a/internal/schema/schema.go +++ b/internal/schema/schema.go @@ -170,6 +170,17 @@ type GoCallEdge struct { Weight int `json:"weight"` // accumulated when merging backends Provenance []string `json:"provenance"` // e.g. ["go/types"], ["go/types","codeql"] Tags map[string]string `json:"tags"` + + // SourcePath is the relative file in which the source callable is declared. + // A signature is not globally unique for package-level init functions: Go + // permits many `func init()` per package (one per file), all sharing the + // signature ".init". Those are distinct callables with distinct + // can:// ids (the id embeds the declaring file via its module), so the v2 + // emitter needs this to resolve the edge's source to the SPECIFIC init that + // owns the call site rather than to an arbitrary one. Empty for callables + // whose signature is already unique (the common case). Internal-only; not + // part of the v2 output schema. + SourcePath string `json:"-"` } // ─── Root object ────────────────────────────────────────────────────────────── diff --git a/internal/semantic_analysis/call_graph.go b/internal/semantic_analysis/call_graph.go index 7dd4adc..af2101a 100644 --- a/internal/semantic_analysis/call_graph.go +++ b/internal/semantic_analysis/call_graph.go @@ -136,6 +136,11 @@ func (cg *CallGraphBuilder) resolveCallable( Weight: 1, Provenance: []string{"go/types"}, Tags: map[string]string{}, + // Carry the source's declaring file so the v2 emitter can map a + // non-unique signature (notably ".init", which repeats once per + // file in a package) back to the specific callable that owns this + // call site. See schema.GoCallEdge.SourcePath. + SourcePath: callable.Path, }) } diff --git a/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go new file mode 100644 index 0000000..25fe1e3 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go @@ -0,0 +1,153 @@ +package v2emit + +// Fresh-build determinism gate (release-gates.md: `-j N` output byte-identical +// to `-j 1`, and run-to-run identical). +// +// The existing determinism_test.go builds the symbol table ONCE and re-emits a +// shuffled edge slice — it proves emitCallGraph imposes a total order, but it +// reuses one symbol table, so it cannot exercise nondeterminism that lives in +// how a FRESH build resolves ids. That gap hid a real bug: Go allows many +// package-level `func init()` per package (one per file), all sharing the +// signature ".init". Those are distinct callables with distinct can:// +// ids, but the flat signature->id index collapsed them, and since it was +// populated by ranging a Go map (random order) the surviving id — and therefore +// the call_graph edge set — varied run to run. +// +// This gate rebuilds the symbol table + call graph from disk N times (each a new +// Go map with its own iteration order) and asserts the serialized call_graph is +// byte-identical every time. It runs over the `chi` fixture because chi/middleware +// is the only fixture with multiple init functions in one package (logger.go, +// request_id.go, terminal.go) — the collision case. multipackage (what the other +// gates use) has none, which is why the bug stayed green there. +// +// N is >1-and-plenty on purpose: a single pair of builds can match by chance +// (the collision had three near-equiprobable orderings), so a two-build test is +// a flaky detector. With ~3 orderings the chance a reintroduced bug slips past N +// builds is ~(1/3)^(N-1); N=16 puts that at ~1e-8 while keeping the gate near +// ~10s (each fresh build pays one packages.Load). + +import ( + "encoding/json" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" +) + +// chiDir returns the absolute path to the chi testdata fixture. +func chiDir(t *testing.T) string { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + abs, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", "chi")) + if err != nil { + t.Fatalf("resolving chi fixture dir: %v", err) + } + return abs +} + +// buildChiL2Fresh runs a complete, independent L2 analysis of the chi fixture: +// a new symbol-table builder (hence a new Go map) + resolver + emit. Each call +// is a from-scratch build, which is what makes map-iteration nondeterminism +// observable across repeated calls. +func buildChiL2Fresh(t *testing.T, dir string) *v2.Analysis { + t.Helper() + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges} + return Emit(app, "chi", dir, 2, "gate-test") +} + +// TestDeterminism_FreshBuildsByteIdentical locks that independent from-disk +// builds of the chi fixture emit a byte-identical call_graph every time. A +// regression that reintroduces signature->id collapse (or any map-order leak +// into the edge set) fails here, in the test suite CI runs — not only in an +// external CLI diff. +func TestDeterminism_FreshBuildsByteIdentical(t *testing.T) { + dir := chiDir(t) + + const builds = 16 + var want string + for i := 0; i < builds; i++ { + out := buildChiL2Fresh(t, dir) + data, err := json.Marshal(out.Application.CallGraph) + if err != nil { + t.Fatalf("build %d: marshal call_graph: %v", i, err) + } + got := string(data) + if i == 0 { + want = got + continue + } + if got != want { + t.Fatalf("call_graph not deterministic across fresh builds: build %d differs from build 0\n"+ + "this is the init-collision class (a signature resolving to a run-varying id)\n"+ + "build0=%s\nbuild%d=%s", i, want, i, got) + } + } +} + +// TestInitCollision_DistinctNodesAndEdges is the focused correctness assertion +// behind the determinism fix: the three chi/middleware init functions must emit +// three DISTINCT callable nodes (one per declaring file), and every call_graph +// edge whose source is an init must resolve to the init node in that same file +// — not to an arbitrary sibling's. This is the semantic guarantee the +// (signature, path) resolution provides on top of mere stability. +func TestInitCollision_DistinctNodesAndEdges(t *testing.T) { + dir := chiDir(t) + out := buildChiL2Fresh(t, dir) + + // Collect every emitted callable id that is a middleware package init. + // Module-level functions (init has no receiver) live in mod.Functions; the + // durable id is on the callable node itself (Callable.ID), which embeds the + // declaring file via its parent module. + initNodeIDs := map[string]bool{} + for rel, mod := range out.Application.SymbolTable { + if !strings.Contains(rel, "middleware/") { + continue + } + for _, fn := range mod.Functions { + if strings.HasSuffix(fn.ID, "middleware.init") { + initNodeIDs[fn.ID] = true + } + } + } + if len(initNodeIDs) < 3 { + t.Fatalf("expected >=3 distinct middleware init nodes (logger.go, request_id.go, terminal.go), got %d: %v", + len(initNodeIDs), keysOf(initNodeIDs)) + } + + // Every init-source edge must point at a real, distinct init node id, and + // that id must embed the same file the edge's module path names (i.e. the + // edge resolved to ITS OWN file's init, so the id is self-consistent). + sawInitEdge := false + for _, e := range out.Application.CallGraph { + if !strings.HasSuffix(e.Src, "middleware.init") { + continue + } + sawInitEdge = true + if !initNodeIDs[e.Src] { + t.Errorf("init-source edge resolves to a non-node id: %s", e.Src) + } + } + if !sawInitEdge { + t.Fatalf("expected at least one init-source edge in the chi call_graph; found none") + } +} + +func keysOf(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + return out +} diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 5b61f9b..acfa2d1 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -245,7 +245,12 @@ func emitImports(li *lineIndex, imports []schema.GoImport) []v2.Import { func emitCallGraph(edges []schema.GoCallEdge, idx sigIndex) []v2.Edge { out := make([]v2.Edge, 0, len(edges)) for _, e := range edges { - src, srcOK := idx.idFor(e.Source) + // Resolve the source by (signature, declaring file): a package-level + // init signature repeats once per file in a package, so the file is + // what pins the edge to the init that actually contains the call site. + // The target stays signature-only — a call target is never an init + // (Go forbids calling init), and every other signature is unique. + src, srcOK := idx.idForWithPath(e.Source, e.SourcePath) dst, dstOK := idx.idFor(e.Target) if !srcOK || !dstOK { continue diff --git a/internal/syntactic_analysis/v2emit/sigindex.go b/internal/syntactic_analysis/v2emit/sigindex.go index f0e8cd8..1f2d5c9 100644 --- a/internal/syntactic_analysis/v2emit/sigindex.go +++ b/internal/syntactic_analysis/v2emit/sigindex.go @@ -16,24 +16,46 @@ import ( // L2 gate assert "no dangling endpoints" — every edge endpoint resolves to a // real node id. // -// v1 signatures are globally unique (a method signature embeds its receiver -// type), so a flat signature→id map has no collisions. -type sigIndex map[string]string +// Signatures are ALMOST globally unique (a method signature embeds its receiver +// type), with one exception: package-level `func init()`. Go permits many init +// functions per package (one per file), and they all share the signature +// ".init". They are nonetheless distinct callables with distinct +// can:// ids — the id embeds the declaring file via its parent module — so a +// flat signature→id map collapses them, and since it is populated by ranging a +// Go map (random order) the surviving id is nondeterministic run to run. That +// nondeterminism propagated into the call_graph edge set. +// +// The index therefore keeps two maps: a flat signature→id (deterministic on +// collision — smallest id wins) for the unique common case, and a +// (signature,path)→id map that disambiguates the init collision by the source +// callable's declaring file. See idFor / idForWithPath. +type sigIndex struct { + bySig map[string]string + bySigPath map[sigPathKey]string +} + +type sigPathKey struct { + sig string + path string +} // buildSigIndex walks the symbol table and records, for every callable // (functions, methods, and nested closures), signature → can:// id. It mirrors // emitModule/emitType/emitCallable's parent-id computation exactly. func buildSigIndex(appID string, symbolTable map[string]schema.GoFile) sigIndex { - idx := make(sigIndex) + idx := sigIndex{ + bySig: make(map[string]string), + bySigPath: make(map[sigPathKey]string), + } for relPath, file := range symbolTable { modID := v2.ModuleID(appID, relPath) for _, fn := range file.Functions { - idx.addCallable(modID, fn) + idx.addCallable(modID, relPath, fn) } for _, t := range file.Types { typeID := v2.TypeID(modID, t.Signature) for _, m := range t.Methods { - idx.addCallable(typeID, m) + idx.addCallable(typeID, relPath, m) } } } @@ -41,19 +63,51 @@ func buildSigIndex(appID string, symbolTable map[string]schema.GoFile) sigIndex } // addCallable records one callable and recurses into its closures, matching the -// id nesting emitCallable produces. -func (idx sigIndex) addCallable(parentID string, c schema.GoCallable) { +// id nesting emitCallable produces. relPath is the module file this callable is +// reached through; it keys the disambiguating (signature,path) entry. +func (idx sigIndex) addCallable(parentID, relPath string, c schema.GoCallable) { callID := v2.CallableID(parentID, c.Signature) - idx[c.Signature] = callID + + // Flat entry: deterministic on collision — keep the lexicographically + // smallest id so a repeated signature (package-level init) resolves to a + // stable node even when the only information available is the bare + // signature. (The (sig,path) map below is the precise resolver when a path + // is known.) + if existing, ok := idx.bySig[c.Signature]; !ok || callID < existing { + idx.bySig[c.Signature] = callID + } + + // Precise entry: a callable's declaring file plus its signature is unique + // even for init, because at most one init exists per file. + idx.bySigPath[sigPathKey{sig: c.Signature, path: relPath}] = callID + for _, inner := range c.InnerCallables { - idx.addCallable(callID, inner) + idx.addCallable(callID, relPath, inner) } } // idFor returns the can:// id for a v1 signature, or ("", false) when the // signature names a callable outside the project (stdlib/external) — those -// resolve to no in-tree node and must not become an edge endpoint. +// resolve to no in-tree node and must not become an edge endpoint. For a +// signature shared by multiple callables (package-level init) it returns the +// deterministic smallest id; callers that know the declaring file should use +// idForWithPath for the precise node. func (idx sigIndex) idFor(sig string) (string, bool) { - id, ok := idx[sig] + id, ok := idx.bySig[sig] return id, ok } + +// idForWithPath returns the can:// id for a signature declared in a specific +// file. It is the precise resolver for a non-unique signature: an init call +// site in request_id.go resolves to request_id.go's init node, not an +// arbitrary sibling's. When path is empty or no (sig,path) entry exists it +// falls back to idFor (the deterministic flat lookup), so unique signatures and +// edges that carry no source path still resolve. +func (idx sigIndex) idForWithPath(sig, path string) (string, bool) { + if path != "" { + if id, ok := idx.bySigPath[sigPathKey{sig: sig, path: path}]; ok { + return id, true + } + } + return idx.idFor(sig) +} From ed4b93b3ce128eab9ecfce2226bafeb23d91d430 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Tue, 6 Oct 2026 23:42:04 -0400 Subject: [PATCH 29/29] refactor: remove CodeQL stub and v1 schema emission Remove two pieces of dead/legacy machinery, leaving the analyzer v2-only. CodeQL: the --codeql path was a wired stub whose operations always returned ErrCodeQLNotImplemented (no .ql files, no binary dependency, no CI wiring). Delete the internal/semantic_analysis/codeql package, the Phase-5 enrichment block, the --codeql flag, and the UseCodeQL option. v1 schema: since the default flipped to v2 (and Neo4j forces v2), the v1 GoApplication shape was a compat shim behind --analysis-schema. Collapse RenderJSON to always emit the canonical v2 tree via v2emit, and drop the --analysis-schema flag and SchemaVersion option entirely. The internal GoApplication IR and analysis_cache.json are unchanged; only the output serializer is affected. Tests updated to assert the v2 shape; the shared parser/resolver/pass pipeline, v2emit, schema/v2, and neo4j projection are untouched. Verified with a 15-app regression eval (L1+L2) against validate_v2.py: 15/15 pass, check-counts identical to the prior baseline. --- README.md | 82 +++---------------- cmd/codeanalyzer/main.go | 47 +++++------ cmd/codeanalyzer/main_test.go | 62 +------------- docs/design/roadmap.md | 2 +- internal/core/analyzer.go | 43 ++-------- internal/core/analyzer_test.go | 9 +- internal/options/options.go | 5 -- internal/schema/schema.go | 2 +- internal/schema/v2/schema.go | 2 +- internal/semantic_analysis/call_graph_test.go | 10 +-- internal/semantic_analysis/codeql/codeql.go | 48 ----------- internal/semantic_analysis/codeql/errors.go | 15 ---- internal/semantic_analysis/codeql/loader.go | 22 ----- internal/semantic_analysis/codeql/runner.go | 27 ------ 14 files changed, 57 insertions(+), 319 deletions(-) delete mode 100644 internal/semantic_analysis/codeql/codeql.go delete mode 100644 internal/semantic_analysis/codeql/errors.go delete mode 100644 internal/semantic_analysis/codeql/loader.go delete mode 100644 internal/semantic_analysis/codeql/runner.go diff --git a/README.md b/README.md index b1ad66c..194c415 100644 --- a/README.md +++ b/README.md @@ -139,10 +139,8 @@ Aliases: Flags: -a, --analysis-level int Analysis level: 1=symbol table only, 2=+resolver call graph (default 1) - --analysis-schema int Output schema major: 2=canonical v2 tree (default), 1=legacy v1 shape (default 2) --app-name string Application anchor name for can:// ids and Neo4j :Application (default: input dir name) -c, --cache-dir string Cache directory (default: ~/.cldk/go-cache) - --codeql Enable CodeQL framework-based call graph (level 2, stub) --eager Force clean rebuild (ignore cache) --emit string Output projection: json|neo4j|schema (default "json") -f, --format string Output format: json|msgpack (default "json") @@ -213,22 +211,21 @@ module. Write the output to a directory with `-o`; `cango` names the file `analysis.json` inside it. ```bash -# Canonical v2 schema, Level 2 (symbol table + call graph): -cango -i /path/to/go/app --analysis-schema 2 -a 2 -o /path/to/output/ +# Level 2 (symbol table + call graph): +cango -i /path/to/go/app -a 2 -o /path/to/output/ # → writes /path/to/output/analysis.json ``` -> **Schema.** `--analysis-schema` defaults to `2` — the current **canonical v2** tree (the -> shape the Python SDK's v2 loader and the sections below describe). Pass -> `--analysis-schema 1` only if you need the **legacy v1** shape for backward compatibility. -> Omit `-o` to stream the JSON to stdout instead of writing a file. +> **Schema.** `cango` emits the **canonical CLDK v2** tree (the shape the Python SDK's v2 +> loader and the sections below describe). Omit `-o` to stream the JSON to stdout instead +> of writing a file. **3. (Optional) name the application anchor.** `--app-name` sets the `` in every `can://go//…` id and defaults to the input directory's base name. Set it when the directory name isn't the identity you want: ```bash -cango -i /path/to/go/app --analysis-schema 2 -a 2 --app-name myapp -o ./out/ +cango -i /path/to/go/app -a 2 --app-name myapp -o ./out/ ``` **4. Verify.** A successful run exits `0` and produces a JSON document whose envelope @@ -236,7 +233,7 @@ carries `"schema_version": "2.0.0"`, `"language": "go"`, `"max_level": 2`, and a `application` tree under `application.symbol_table`: ```bash -cango -i /path/to/go/app --analysis-schema 2 -a 2 -o ./out/ && echo "exit=$?" +cango -i /path/to/go/app -a 2 -o ./out/ && echo "exit=$?" python3 -c "import json; d=json.load(open('out/analysis.json')); print(d['schema_version'], d['max_level'], len(d['application']['symbol_table']), 'modules')" ``` @@ -252,7 +249,6 @@ are build artifacts, not project source), so only your own declarations are emit |-------|------|-----------|--------| | 1 | `-a 1` (default) | Symbol table only — types, functions, call sites | Implemented | | 2 | `-a 2` | Level 1 + resolver-based call graph via `go/types` | Implemented | -| — | `--codeql` | CodeQL framework-based call graph (merged with Level 2 edges) | Stub (not yet implemented) | **Level 1** loads each package with `packages.NeedSyntax | NeedTypes | NeedTypesInfo` and walks the AST file by file. Call sites are recorded with `callee_signature = null` at this stage. @@ -260,15 +256,10 @@ are build artifacts, not project source), so only your own declarations are emit ## Output schema -`cango` emits one of two schema shapes, selected by `--analysis-schema`: +`cango` emits the **canonical CLDK v2** shape — the shape the v2 Python SDK loader consumes +and the [step-by-step section above](#generating-analysisjson-for-a-go-app) produces. -- **`--analysis-schema 2`** — the **canonical CLDK v2** shape (detailed below); the shape the - v2 Python SDK loader consumes and the [step-by-step section - above](#generating-analysisjson-for-a-go-app) produces. -- **`--analysis-schema 1`** — the **legacy v1** `GoApplication` shape, kept for backward - compatibility (v2 is the default). - -### Canonical v2 shape (`--analysis-schema 2`, default) +### Canonical v2 shape The document is a manifest envelope wrapping one `application` containment tree. Every node carries a `can://go//…` `id`, a `kind`, and a `span`: @@ -361,53 +352,6 @@ Key v2 schema properties: - **`call_graph` edges** — `{src, dst, prov, weight}`; `src`/`dst` are `can://` node ids that exist in the tree (not raw signatures), `prov` is resolver provenance, e.g. `["go/types"]`. -### Legacy v1 shape (`--analysis-schema 1`) - -The v1 root object is `GoApplication`: - -```json -{ - "symbol_table": { - "pkg/greeter/greeter.go": { - "file_path": "pkg/greeter/greeter.go", - "module_name": "greeter", - "imports": [...], - "classes": { - "Greeter": { - "name": "Greeter", - "signature": "example.com/pkg/greeter.Greeter", - "is_interface": false, - "fields": [{ "name": "Prefix", "type": "string", "tags": {"json": "prefix"} }], - "methods": { ... } - } - }, - "functions": { ... } - } - }, - "call_graph": [ - { - "source": "example.com/main.main", - "target": "example.com/pkg/greeter.Greeter.Greet", - "type": "CALL_DEP", - "weight": 1, - "provenance": ["go/types"] - } - ], - "entrypoints": {} -} -``` - -Key v1 schema properties: -- `symbol_table` — keyed by **file path relative to the project root** (never absolute) -- `classes` — JSON key for types (spine compatibility with Java/Python schemas); value is `GoType` -- `module_name` — JSON key for the Go package name (spine compatibility) -- `GoType.is_interface: bool` — unified type model; structs and interfaces are both `GoType` -- `GoCallable.receiver_type / receiver_name` — non-empty for methods, empty for package-level functions -- `GoCallable.return_types: List[str]` — individual return types (Go-specific extension) -- `GoCallsite.is_goroutine: bool` — true when the call is preceded by the `go` keyword -- `GoCallEdge.provenance: List[str]` — resolver identifiers, e.g. `["go/types"]` or `["go/types","codeql"]` -- Call edges are **identity-only**: source and target are `GoCallable.signature` strings that exist in the symbol table - ## Neo4j projection (`--emit neo4j`) `--emit neo4j` projects the **same v2 envelope** into a Neo4j property graph. It is a *second @@ -552,10 +496,9 @@ methods above, including `pip install codeanalyzer-go`). See | Runtime | Go binary | Self-contained; no runtime dep for SDK users | | Structural parser | `go/ast` (stdlib) | Part of the standard toolchain; no external dep | | Type resolver | `golang.org/x/tools/go/packages` | Single API for both AST + full type resolution; handles modules natively | -| Optional enrichment | CodeQL (stubbed) | Same enrichment path as Python/Java analyzers; stubbed for Level 1 | | Build/dep materialization | `go mod download` | Required before `packages.Load` so the module cache is warm; result cached by `go.sum` hash | | Packaging | Native binary (`go build`) | Zero-runtime-dep distribution; matches Rust/C++ analyzers | -| Analysis depth | Level 1 (rapid) | Symbol table + resolver call graph; CodeQL stub wired but not implemented | +| Analysis depth | Level 1 (rapid) | Symbol table + resolver call graph | | Call-graph dispatch | Declared-type resolution via `go/types.Selections` | CHA-equivalent; sufficient for cross-package reachability at Level 1 | ### Package structure @@ -569,7 +512,6 @@ codeanalyzer-go/ │ ├── options/ # AnalysisOptions + AnalysisLevel constants │ ├── syntactic_analysis/ # SymbolTableBuilder (packages.Load → AST walk) │ ├── semantic_analysis/ # CallGraphBuilder (go/types resolver) -│ │ └── codeql/ # CodeQL backend subpackage (stubbed) │ ├── analysis/ # Pluggable pass interface + registry (topo-ordered pipeline) │ ├── frameworks/ # BaseEntrypointFinder — extension seam for framework passes │ └── utils/ # DiscoverGoFiles, IsVendored, IsTestFile, logging @@ -584,7 +526,7 @@ codeanalyzer-go/ │ └── chi/ # External-dep fixture (chi v5, vendored) for HTTP handler patterns ``` -The `core` package is a pure orchestrator: it calls `syntactic_analysis` → `semantic_analysis` → `analysis.RunPipeline` → optional CodeQL in sequence, with no inlined parsing logic. Framework-specific analysis extends through the `analysis/` + `frameworks/` layer without touching `core`. +The `core` package is a pure orchestrator: it calls `syntactic_analysis` → `semantic_analysis` → `analysis.RunPipeline` in sequence, with no inlined parsing logic. Framework-specific analysis extends through the `analysis/` + `frameworks/` layer without touching `core`. ## Development diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 71f591f..d41ae8c 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -33,25 +33,23 @@ func main() { func rootCmd() *cobra.Command { var ( - inputPath string - outputDir string - format string - emit string - appName string - level int - analysisSchema int - targetFiles []string - skipTests bool - eager bool - cacheDir string - jobs int - useCodeQL bool - verbosity int - showVersion bool - neo4jURI string - neo4jUser string - neo4jPassword string - neo4jDatabase string + inputPath string + outputDir string + format string + emit string + appName string + level int + targetFiles []string + skipTests bool + eager bool + cacheDir string + jobs int + verbosity int + showVersion bool + neo4jURI string + neo4jUser string + neo4jPassword string + neo4jDatabase string ) cmd := &cobra.Command{ @@ -82,10 +80,10 @@ via CLDK(language="go").analysis(project_path=...).`, if cmd.Flags().Changed("analysis-level") { return fmt.Errorf("--analysis-level does not apply to --emit neo4j; the graph is always projected at full depth") } - // Force full depth + canonical v2: the graph carries every - // implemented level's facts in both projections. + // Force full depth: the graph carries every implemented + // level's facts in both projections. (Output is always the + // canonical v2 tree.) level = int(options.LevelCallGraph) - analysisSchema = 2 case options.EmitSchema: path, err := neo4j.EmitSchema(outputDir) if err != nil { @@ -129,14 +127,12 @@ via CLDK(language="go").analysis(project_path=...).`, Emit: options.EmitTarget(emit), AppName: appName, Level: options.AnalysisLevel(level), - SchemaVersion: analysisSchema, AnalyzerVersion: version, TargetFiles: targetFiles, SkipTests: skipTests, Eager: eager, CacheDir: cacheDir, Jobs: jobs, - UseCodeQL: useCodeQL, Verbose: verbosity > 0, Neo4jURI: firstNonEmpty(neo4jURI, os.Getenv("NEO4J_URI")), Neo4jUser: firstNonEmpty(neo4jUser, os.Getenv("NEO4J_USERNAME"), "neo4j"), @@ -195,15 +191,12 @@ via CLDK(language="go").analysis(project_path=...).`, "Application anchor name for can:// ids and Neo4j :Application (default: input dir name)") f.IntVarP(&level, "analysis-level", "a", 1, "Analysis level: 1=symbol table only, 2=+resolver call graph") - f.IntVar(&analysisSchema, "analysis-schema", 2, - "Output schema major: 2=canonical v2 tree (default), 1=legacy v1 shape") f.StringSliceVarP(&targetFiles, "target-files", "t", nil, "Restrict analysis to specific files (incremental mode)") f.BoolVar(&skipTests, "skip-tests", true, "Skip *_test.go files") f.BoolVar(&eager, "eager", false, "Force clean rebuild (ignore cache)") f.StringVarP(&cacheDir, "cache-dir", "c", "", "Cache directory (default: ~/.cldk/go-cache)") f.IntVarP(&jobs, "jobs", "j", 0, "Worker parallelism (default: CPU cores)") - f.BoolVar(&useCodeQL, "codeql", false, "Enable CodeQL framework-based call graph (level 2, stub)") f.CountVarP(&verbosity, "verbose", "v", "Verbosity (repeat for more detail)") f.BoolVar(&showVersion, "version", false, "Print version and exit") diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index 0e827f9..e81d0fe 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -252,58 +252,16 @@ func TestRootCmd_Level2ProducesCallGraph(t *testing.T) { } } -// ── --analysis-schema ────────────────────────────────────────────────────────── +// ── schema output (always canonical v2) ───────────────────────────────────────── -func TestRootCmd_DefaultSchemaIsV2(t *testing.T) { +// The analyzer emits only the canonical v2 containment tree. This asserts the +// envelope shape and identity anchor on the default (and only) output path. +func TestRootCmd_EmitsCanonicalV2(t *testing.T) { td := cliTestdataDir() out, _, err := runCmd("--input", filepath.Join(td, "greeter"), "--cache-dir", t.TempDir()) if err != nil { t.Fatalf("command failed: %v", err) } - var v map[string]interface{} - if jsonErr := json.Unmarshal([]byte(out), &v); jsonErr != nil { - t.Fatalf("stdout is not valid JSON: %v", jsonErr) - } - if _, ok := v["schema_version"]; !ok { - t.Error("default output should be the v2 shape (top-level schema_version)") - } - if _, ok := v["application"]; !ok { - t.Error("default output should carry an application root (that is v2)") - } -} - -func TestRootCmd_AnalysisSchema1EmitsLegacyV1(t *testing.T) { - td := cliTestdataDir() - out, _, err := runCmd( - "--input", filepath.Join(td, "greeter"), - "--analysis-schema", "1", - "--cache-dir", t.TempDir(), - ) - if err != nil { - t.Fatalf("command failed: %v", err) - } - var v map[string]interface{} - if jsonErr := json.Unmarshal([]byte(out), &v); jsonErr != nil { - t.Fatalf("stdout is not valid JSON: %v", jsonErr) - } - if _, ok := v["symbol_table"]; !ok { - t.Error("--analysis-schema 1 should still emit the legacy v1 shape (top-level symbol_table)") - } - if _, ok := v["schema_version"]; ok { - t.Error("the legacy v1 shape should not carry schema_version") - } -} - -func TestRootCmd_AnalysisSchema2EmitsV2(t *testing.T) { - td := cliTestdataDir() - out, _, err := runCmd( - "--input", filepath.Join(td, "greeter"), - "--analysis-schema", "2", - "--cache-dir", t.TempDir(), - ) - if err != nil { - t.Fatalf("command failed: %v", err) - } var v struct { SchemaVersion string `json:"schema_version"` Language string `json:"language"` @@ -325,18 +283,6 @@ func TestRootCmd_AnalysisSchema2EmitsV2(t *testing.T) { } } -func TestRootCmd_UnknownSchemaReturnsError(t *testing.T) { - td := cliTestdataDir() - _, _, err := runCmd( - "--input", filepath.Join(td, "greeter"), - "--analysis-schema", "9", - "--cache-dir", t.TempDir(), - ) - if err == nil { - t.Fatal("expected error for unknown --analysis-schema value, got nil") - } -} - // ── --skip-tests ───────────────────────────────────────────────────────────── func TestRootCmd_SkipTestsFalseIncludesTestFiles(t *testing.T) { diff --git a/docs/design/roadmap.md b/docs/design/roadmap.md index 28e3ada..2f306b3 100644 --- a/docs/design/roadmap.md +++ b/docs/design/roadmap.md @@ -73,5 +73,5 @@ A/B/C ─► D (Neo4j relabel) ─► SDK v2 model remap ─► pin SDK→analyz - **Points-to oracle** — L4 prerequisite; parked with L4. - **`--materialize-expressions` / `--materialize-basic-blocks`** — optional body node kinds; out of scope for the parity migration. -- **New framework detection / CodeQL tier expansion** — orthogonal enrichment axis +- **New framework detection** — orthogonal enrichment axis (provenance-merged evidence), not a schema level; does not belong on the migration train. diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index d889a2f..fb4e5b7 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -9,7 +9,6 @@ // 2. Symbol table construction (syntactic_analysis) // 3. Resolver-based call graph (semantic_analysis) — if level >= 2 // 4. Pass pipeline (analysis/registry) -// 5. Optional CodeQL enrichment (semantic_analysis/codeql) — if --codeql package core import ( @@ -23,7 +22,6 @@ import ( "github.com/codellm-devkit/codeanalyzer-go/internal/options" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" - "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis/codeql" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" @@ -91,26 +89,6 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { utils.Warn("pass pipeline error: %v", err) } - // ── Phase 5: Optional CodeQL enrichment ────────────────────────────────── - if a.opts.UseCodeQL { - cq, err := codeql.New(a.opts.CacheDir, true) - if err != nil { - utils.Warn("CodeQL unavailable: %v", err) - } else { - if err := cq.Build(a.opts.InputPath); err != nil { - utils.Warn("CodeQL build failed: %v", err) - } else { - cqlEdges, err := cq.Edges() - if err != nil { - utils.Warn("CodeQL edge extraction failed: %v", err) - } else { - app.CallGraph = semantic_analysis.MergeEdges(app.CallGraph, cqlEdges) - utils.Info("merged %d CodeQL edges", len(cqlEdges)) - } - } - } - } - return a.finalizeAndCache(app) } @@ -171,12 +149,10 @@ func (a *Analyzer) saveCache(app *schema.GoApplication) error { return os.WriteFile(cachePath, data, 0o644) } -// RenderJSON serializes the analysis to JSON in the schema selected by -// opts.SchemaVersion: 1 (default) emits the legacy v1 GoApplication shape; 2 -// emits the canonical v2 tree via the v2emit emitter. This is the single place -// the v1/v2 decision is made, so both the stdout and file output paths agree. -// Only "json" is supported; other formats return an explicit error rather than -// silently falling back. +// RenderJSON serializes the analysis to JSON as the canonical v2 containment +// tree via the v2emit emitter. This is the single serialization point, so both +// the stdout and file output paths agree. Only "json" is supported; other +// formats return an explicit error rather than silently falling back. func RenderJSON(app *schema.GoApplication, opts options.AnalysisOptions) ([]byte, error) { format := opts.Format if format == "" { @@ -191,15 +167,8 @@ func RenderJSON(app *schema.GoApplication, opts options.AnalysisOptions) ([]byte return nil, fmt.Errorf("unsupported output format %q; supported: json", format) } - switch opts.SchemaVersion { - case 0, 1: - return json.Marshal(app) - case 2: - payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion) - return json.Marshal(payload) - default: - return nil, fmt.Errorf("unsupported --analysis-schema %d; supported: 1, 2", opts.SchemaVersion) - } + payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion) + return json.Marshal(payload) } // WriteOutput serializes the analysis (in the selected schema) and writes it to diff --git a/internal/core/analyzer_test.go b/internal/core/analyzer_test.go index 45dacf2..72be465 100644 --- a/internal/core/analyzer_test.go +++ b/internal/core/analyzer_test.go @@ -10,6 +10,7 @@ import ( "github.com/codellm-devkit/codeanalyzer-go/internal/core" "github.com/codellm-devkit/codeanalyzer-go/internal/options" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" ) // greeterDir returns the absolute path to testdata/greeter. @@ -144,11 +145,15 @@ func TestWriteOutput_ValidJSON(t *testing.T) { if err != nil { t.Fatalf("reading analysis.json: %v", err) } - var round schema.GoApplication + // Output is the canonical v2 containment tree; round-trip into it. + var round v2.Analysis if err := json.Unmarshal(data, &round); err != nil { t.Fatalf("JSON round-trip failed: %v", err) } - if len(round.SymbolTable) == 0 { + if round.SchemaVersion != v2.SchemaVersion { + t.Errorf("schema_version = %q, want %q", round.SchemaVersion, v2.SchemaVersion) + } + if len(round.Application.SymbolTable) == 0 { t.Error("round-tripped symbol table is empty") } } diff --git a/internal/options/options.go b/internal/options/options.go index e19d542..c4d50f4 100644 --- a/internal/options/options.go +++ b/internal/options/options.go @@ -38,9 +38,6 @@ type AnalysisOptions struct { AppName string // AnalysisLevel controls symbol-table-only (1) vs + call graph (2). Level AnalysisLevel - // SchemaVersion selects the output schema major: 1 = the legacy v1 shape - // (default, compat shim during the migration), 2 = the canonical v2 tree. - SchemaVersion int // AnalyzerVersion is the analyzer's own version, recorded in the v2 // manifest's analyzer{} tag. Set by the CLI from its build-stamped value. AnalyzerVersion string @@ -55,8 +52,6 @@ type AnalysisOptions struct { // Jobs is the worker parallelism (default: CPU cores). Output must be // byte-identical across Jobs values. Jobs int - // UseCodeQL enables the framework-based (Tier-2) CodeQL call graph. - UseCodeQL bool // Verbose enables verbose logging. Verbose bool diff --git a/internal/schema/schema.go b/internal/schema/schema.go index c02690e..a61f255 100644 --- a/internal/schema/schema.go +++ b/internal/schema/schema.go @@ -168,7 +168,7 @@ type GoCallEdge struct { Target string `json:"target"` Type string `json:"type"` // always "CALL_DEP" Weight int `json:"weight"` // accumulated when merging backends - Provenance []string `json:"provenance"` // e.g. ["go/types"], ["go/types","codeql"] + Provenance []string `json:"provenance"` // e.g. ["go/types"] Tags map[string]string `json:"tags"` // SourcePath is the relative file in which the source callable is declared. diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index f4a5dcf..e3a83f7 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -198,7 +198,7 @@ type BodyNode struct { type Edge struct { Src string `json:"src"` Dst string `json:"dst"` - // Prov is open-vocabulary provenance, e.g. ["go/types"], ["go/types","codeql"]. + // Prov is open-vocabulary provenance, e.g. ["go/types"]. Prov []string `json:"prov,omitempty"` // Weight accumulates when merging edges from multiple backends. Weight int `json:"weight,omitempty"` diff --git a/internal/semantic_analysis/call_graph_test.go b/internal/semantic_analysis/call_graph_test.go index 60b89ee..a1ec09e 100644 --- a/internal/semantic_analysis/call_graph_test.go +++ b/internal/semantic_analysis/call_graph_test.go @@ -32,7 +32,7 @@ func TestMergeEdges_PrimaryOnly(t *testing.T) { } func TestMergeEdges_SecondaryOnly(t *testing.T) { - secondary := []schema.GoCallEdge{edge("x", "y", 2.0, "codeql")} + secondary := []schema.GoCallEdge{edge("x", "y", 2.0, "framework")} result := semantic_analysis.MergeEdges(nil, secondary) if len(result) != 1 { t.Fatalf("got %d edges, want 1", len(result)) @@ -44,7 +44,7 @@ func TestMergeEdges_SecondaryOnly(t *testing.T) { func TestMergeEdges_DisjointEdges(t *testing.T) { primary := []schema.GoCallEdge{edge("a", "b", 1.0, "resolver")} - secondary := []schema.GoCallEdge{edge("c", "d", 1.0, "codeql")} + secondary := []schema.GoCallEdge{edge("c", "d", 1.0, "framework")} result := semantic_analysis.MergeEdges(primary, secondary) if len(result) != 2 { t.Errorf("got %d edges, want 2", len(result)) @@ -53,7 +53,7 @@ func TestMergeEdges_DisjointEdges(t *testing.T) { func TestMergeEdges_DuplicateAccumulatesWeight(t *testing.T) { primary := []schema.GoCallEdge{edge("a", "b", 3, "resolver")} - secondary := []schema.GoCallEdge{edge("a", "b", 5, "codeql")} + secondary := []schema.GoCallEdge{edge("a", "b", 5, "framework")} result := semantic_analysis.MergeEdges(primary, secondary) if len(result) != 1 { t.Fatalf("duplicate (a→b) should collapse to 1 edge; got %d", len(result)) @@ -65,7 +65,7 @@ func TestMergeEdges_DuplicateAccumulatesWeight(t *testing.T) { func TestMergeEdges_DuplicateUnionsProvenance(t *testing.T) { primary := []schema.GoCallEdge{edge("a", "b", 1.0, "resolver")} - secondary := []schema.GoCallEdge{edge("a", "b", 1.0, "codeql")} + secondary := []schema.GoCallEdge{edge("a", "b", 1.0, "framework")} result := semantic_analysis.MergeEdges(primary, secondary) if len(result) != 1 { t.Fatalf("got %d edges, want 1", len(result)) @@ -74,7 +74,7 @@ func TestMergeEdges_DuplicateUnionsProvenance(t *testing.T) { for _, p := range result[0].Provenance { provSet[p] = true } - if !provSet["resolver"] || !provSet["codeql"] { + if !provSet["resolver"] || !provSet["framework"] { t.Errorf("provenance union failed; got %v", result[0].Provenance) } } diff --git a/internal/semantic_analysis/codeql/codeql.go b/internal/semantic_analysis/codeql/codeql.go deleted file mode 100644 index 88d1d29..0000000 --- a/internal/semantic_analysis/codeql/codeql.go +++ /dev/null @@ -1,48 +0,0 @@ -package codeql - -import ( - "github.com/codellm-devkit/codeanalyzer-go/internal/schema" - "github.com/codellm-devkit/codeanalyzer-go/internal/utils" -) - -// CodeQL is the top-level handle for CodeQL-backed analysis. core.go talks only -// to this type; it never touches the binary, database, or query strings directly. -// TODO(level-2): implement Build() and Edges(). -type CodeQL struct { - loader *Loader - runner *Runner - enabled bool -} - -// New probes for the CodeQL binary and returns a CodeQL handle. -// If the binary is absent and enabled=true, returns an error. -func New(cacheDir string, enabled bool) (*CodeQL, error) { - if !enabled { - return &CodeQL{enabled: false}, nil - } - loader, err := NewLoader() - if err != nil { - return nil, err - } - runner := NewRunner(loader, cacheDir+"/codeql-db") - return &CodeQL{loader: loader, runner: runner, enabled: true}, nil -} - -// Build creates the CodeQL database for projectDir. -// No-op when CodeQL is disabled. -func (c *CodeQL) Build(projectDir string) error { - if !c.enabled { - return nil - } - utils.Info("building CodeQL database (stub — TODO level-2)") - return c.runner.BuildDatabase(projectDir) -} - -// Edges returns Tier-2 call-graph edges. -// Returns empty slice when disabled; returns ErrCodeQLNotImplemented when enabled (stub). -func (c *CodeQL) Edges() ([]schema.GoCallEdge, error) { - if !c.enabled { - return nil, nil - } - return c.runner.QueryCallGraph() -} diff --git a/internal/semantic_analysis/codeql/errors.go b/internal/semantic_analysis/codeql/errors.go deleted file mode 100644 index a4affd1..0000000 --- a/internal/semantic_analysis/codeql/errors.go +++ /dev/null @@ -1,15 +0,0 @@ -// Package codeql is the isolated framework-backend subpackage for CodeQL analysis. -// It provides Tier-2 (framework-based) call-graph edges beyond what go/types resolves. -// -// The seams (loader, driver, query runner, errors) are scaffolded here even though -// the implementation is stubbed — dropping in the full implementation later requires -// no refactor. Mirrors codeanalyzer-python's semantic_analysis/codeql/ split. -package codeql - -import "errors" - -// ErrCodeQLNotFound is returned when the CodeQL CLI binary cannot be located. -var ErrCodeQLNotFound = errors.New("codeql: CLI binary not found; install from https://github.com/github/codeql-cli-binaries") - -// ErrCodeQLNotImplemented is returned when CodeQL analysis is requested but not yet implemented. -var ErrCodeQLNotImplemented = errors.New("codeql: Go backend is a wired stub — implementation TODO (level-2 analysis)") diff --git a/internal/semantic_analysis/codeql/loader.go b/internal/semantic_analysis/codeql/loader.go deleted file mode 100644 index fce359d..0000000 --- a/internal/semantic_analysis/codeql/loader.go +++ /dev/null @@ -1,22 +0,0 @@ -package codeql - -import ( - "os/exec" -) - -// Loader resolves the CodeQL CLI binary path. -type Loader struct { - binaryPath string -} - -// NewLoader creates a Loader, probing the PATH for the codeql binary. -func NewLoader() (*Loader, error) { - path, err := exec.LookPath("codeql") - if err != nil { - return nil, ErrCodeQLNotFound - } - return &Loader{binaryPath: path}, nil -} - -// BinaryPath returns the resolved CodeQL binary path. -func (l *Loader) BinaryPath() string { return l.binaryPath } diff --git a/internal/semantic_analysis/codeql/runner.go b/internal/semantic_analysis/codeql/runner.go deleted file mode 100644 index 9abbd3e..0000000 --- a/internal/semantic_analysis/codeql/runner.go +++ /dev/null @@ -1,27 +0,0 @@ -package codeql - -import "github.com/codellm-devkit/codeanalyzer-go/internal/schema" - -// Runner builds a CodeQL database and runs queries to produce call-graph edges. -// TODO(level-2): implement database creation, query execution, and result parsing. -type Runner struct { - loader *Loader - dbDir string -} - -// NewRunner creates a Runner for the given project and database directory. -func NewRunner(loader *Loader, dbDir string) *Runner { - return &Runner{loader: loader, dbDir: dbDir} -} - -// BuildDatabase creates a CodeQL database for the Go project at projectDir. -// TODO(level-2): run `codeql database create --language=go`. -func (r *Runner) BuildDatabase(projectDir string) error { - return ErrCodeQLNotImplemented -} - -// QueryCallGraph runs the CodeQL call-graph query and returns edges. -// TODO(level-2): run query, parse SARIF/CSV, produce GoCallEdge list. -func (r *Runner) QueryCallGraph() ([]schema.GoCallEdge, error) { - return nil, ErrCodeQLNotImplemented -}