diff --git a/.gitignore b/.gitignore index 9214263b..4a6d9d3c 100644 --- a/.gitignore +++ b/.gitignore @@ -53,11 +53,11 @@ scratch* *.json !devcontainer.json -# Blessed TS unit-test fixture: hand-built from a sample app whose source (incl. src/external.ts) -# was never committed, so it cannot be regenerated by running codeanalyzer-typescript again. The -# bulk-accessor tests assert exact-set constants (signature counts, ownerless sets) against this -# exact file -- losing it breaks the suite for every fresh clone (#298). -!tests/resources/typescript/analysis_json/slim/analysis.json +# Blessed TS unit-test fixtures: analysis.json at each level, generated by the pinned +# codeanalyzer-typescript from tests/resources/typescript/application (see the README there), and +# the sample app's own manifests, which the analyzer's artifact layer reads. +!tests/resources/typescript/analysis_json/v2/*/analysis.json +!tests/resources/typescript/application/*.json # Python compiled files and env diff --git a/CHANGELOG.md b/CHANGELOG.md index 000be40c..c72a0910 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,122 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +Leg 2.5a of the CLDK 2.0 facade: the TypeScript layer on schema v2 (see +`docs/design/specs/2026-09-06-leg-2.5-typescript.md`; plan +`docs/design/plans/2026-09-06-leg-2.5a-typescript-schema-v2.md`). Targets `2.0.0-rc.3`. The +leg-1.5/1.6 query surface for TypeScript (addressing, scoping keywords, per-callable graphs, slices, +paths, entrypoints) is 2.5b, on codeanalyzer-typescript 1.3.0 when it is cut. + +### Changed +- **Pinned `codeanalyzer-typescript` 0.4.3 → 1.2.0** (`pyproject.toml` `dependencies` and + `[tool.backend-versions]`). The analyzer emits canonical schema v2, and `cldk.models.typescript` + is rewritten as an `extra="forbid"` mirror of it: `analysis.json` is the `TSAnalysis` envelope + (`analyzer.version`, `max_level`, `application`); every node carries a `can://` `id`, a `kind` and a + `span`; a module's `classes`/`interfaces`/`enums`/`type_aliases`/`namespaces` are read-only views over + one `types` map; `start_line`/`end_line`/`code` are properties over `span` and the module `source`. + A 0.4.x `analysis.json` no longer validates — re-run the analyzer. + **BREAKING by value:** `TSCallEdge` is `TSCallGraphEdge{src, dst, prov, weight}` (was + `source/target/provenance/tags`), with `can://` ids as endpoints; `TSExternalSymbol` / + `TSSynthesizedCallable` are the id-keyed `TSExternalNode` / `TSSynthesizedNode`; `TSCallable` has no + `path`, `call_sites`, `accessed_symbols`, `local_variables` or `code_start_line`. The 1.x class names + stay importable as aliases. **BREAKING (public classmethod):** + `TSCallableOverview.from_callable(c, owner_signature, owner_kind)` now takes a required + keyword-only `path` — a v2 `TSCallable` no longer carries one, so the caller walking + `symbol_table` must pass the module's key. `TSCallableOverview` itself is unchanged. + `is_entrypoint`/`entrypoints` (1.3.0 additive, `Optional`) are declared on **every** type kind, + not only `TSClass`, so a 1.3.0 payload that stamps them on an interface, enum, type alias or + namespace validates under `extra="forbid"` before the pin moves. +- **JavaScript modules are in scope** (`.js/.jsx/.mjs/.cjs`; spec TS-3). The analyzer ids them + `can://javascript//…` beside `can://typescript//…`, and every accessor on both backends + reads both: the symbol table lists them, and the Neo4j backend scopes every statement by the two + prefixes. A single-prefix reading would have dropped every `.js` file silently. +- **`TSAnalysisBackend` inherits the generic `AnalysisBackend`** (`P = "TS"`, `N = "TS"`), so both + TypeScript backends now also answer `get_artifacts` / `get_dependencies` / `get_config_keys` / + `get_config_uses` / `get_unresolved_config_reads` with the shared `PyArtifact` / `PyDependency` / + `PyConfigKey` / `PyConfigUseEdge` / `PyConfigRead` models (a TypeScript dependency's `ecosystem` is + `"npm"`; a non-string config value is rendered as its JSON text). +- **`get_call_graph()` on TypeScript** keys nodes as every other accessor does (module file key, + signature, `"."` for an external) with `id` and `kind` node attributes — `kind` is + one of `module | class | interface | enum | type_alias | namespace | callable | external` — and + edges carry `type="CALL_DEP"`, `weight` and `provenance` (a tuple) as Python's do; the 1.x `tags` + dict is gone with the wire. Module callers and class callees are kept, not dropped as on Python — + filter on `kind == "callable"` for that shape. `get_external_symbols()` is keyed `"."` + to match. `get_call_sites` / `get_calling_lines` / `get_call_targets` / `get_callsites_for` read a + callable's `body` nodes of `kind == "call"`; `get_method_bodies` omits a callable with no source + text (an implicit constructor's `code` is now `""`, not `None`). +- The in-process backend drives the 1.2.0 CLI: `-i --app-name -a <1..4> + -o --cache-dir --skip-tests [--eager] [-t ]...`. Every `AnalysisLevel` is + sent as its integer (the old cap at level 2 is gone). **`TSCodeAnalyzerConfig.tsc_only` is a + deprecated no-op** (the flag was removed upstream in 1.0.0); `True` emits a `DeprecationWarning`. +- **BREAKING: TypeScript Neo4j graph vocabulary migration.** `TSNeo4jBackend` now queries the 1.2.0 + projection — `:Application {id: can://typescript/}`, `TSModule` / `TSClass` / `TSInterface` / + `TSEnum` / `TSTypeAlias` / `TSNamespace` / `TSCallable` / `TSField` / `TSBodyNode {kind:'call'}` / + `TSExternal` under `TS_HAS_MODULE` / `TS_DECLARES` / `TS_HAS_METHOD` / `TS_HAS_FIELD` / + `TS_HAS_BODY_NODE` / `TS_RESOLVES_TO` / `TS_CALLS {weight, prov}` / `TS_DECORATED_BY` — instead of + 0.4.3's `:Symbol` / `:CallSite` / `CALLS` / `HAS_CALLSITE`, with which it shares nothing. There is no + `_module` property to scope on; scope is the two id prefixes above. **Signature-level backwards + compatibility does not extend to graph generation:** a graph emitted by codeanalyzer-typescript + **below 1.2.0**, or one that holds no `:Application` with the requested id (an absent application, + a Python graph), used to answer every query with zero rows and now **raises `GraphSchemaMismatch` + at attach**, naming the relationship types found and missing, or the `analyzer_version` found and + the `1.2.0` floor. **Migration:** re-ingest with `codeanalyzer-typescript>=1.2.0 --emit neo4j`; + there is no in-place upgrade. A graph emitted by an unreleased `main` build stamps `1.2.0` too and + cannot be told apart from a release graph (filed upstream). +- **What the 1.2.0 Neo4j projection does not carry**, stated rather than papered over. On + `TSNeo4jBackend`, four accessors **raise `CodeanalyzerExecutionException` naming the gap**, because + their empty value would read as a fact: `get_imports` and `get_exports` (no import/export + vocabulary in the graph), `get_unresolved_config_reads` (no `config_reads`; `[]` would read as + "every read resolved") and `get_method_parameters` for a **found** method (`:TSCallable` projects + no parameters; a missing method still answers `[]`). Two more join them where a *wrong* value was + the alternative: **`get_extended_classes` and `get_implemented_interfaces` read the split off + `TS_EXTENDS`/`TS_IMPLEMENTS`**, not off `base_classes`/`implements_types` — `base_classes` is the + extends-implements union and `implements_types` is written by no node in the projection (0 of 207 + classes and 0 of 687 interfaces on the reference graph), so subtracting it would have returned a + class's interfaces as its extended classes. They cover **resolved in-repo bases only** (a library + base is in `base_classes` with no node to point at) and each **raises** when its relationship type + is absent from the database — as `TS_IMPLEMENTS` is on superset-frontend. Everything else returns + the model's documented empty where the graph is silent: a callable's `parameters`, `comments`, + `type_parameters`, `overload_signatures`, `body`, `cfg`/`cdg`/`ddg`/`summary`; an enum member's + `value`; a module's `source`, `imports`, `exports`, `comments`; a decorator's position; a call + site's `method_name`/receiver/argument facets and columns. Each rebuilt node's `code` is the text + the graph projected for it, on a line-only span. `get_call_targets` differs by one value: an + unresolved call site contributes `""` from the graph where the in-memory backend contributes the + call's `method_name`. `get_synthesized_callables` is keyed by the anonymous node's own id on Neo4j + (the analyzer's older compatibility key exists only in `analysis.json`); the ABC's docstring now + states that keying as backend-dependent rather than describing the in-memory backend only. + `get_config_uses` returns `[]` both for a corpus the analyzer found no config read in and for a + database that declares no `TS_USES_CONFIG` at all — indistinguishable on this graph, and + deliberately **not** refused at attach, since a project that reads no configuration is a valid + project. `get_nested_classes` is permanently `[]` on **both** backends and is not a projection + gap at all: a schema-v2 class holds only `callables` and `fields`, so no class nests a class + (`TSCallable.inner_classes` is the surviving case). +- **`get_application_view()` on `TSNeo4jBackend` now carries the repository-artifact layer** — + `artifacts` (with their config keys), `dependencies` and `config_uses`, from the same rows the + dedicated accessors return, retyped onto the wire's `TSArtifact`/`TSDependency`/`TSConfigUse` (a + dependency's `ecosystem` has no field there and is dropped; `get_dependencies()` keeps it). It + used to hand back an empty artifact layer while the dedicated accessors answered 832 artifacts, + 3,601 dependencies and 526 config keys on the same attach, so `app.dependencies` read as "this + project declares none". Three overlays stay empty, each stated in the docstring: `param_in`/ + `param_out` (leg 2.5a reads **no** dataflow overlay; 2.5b does), `config_reads` (not projected) + and `unresolved_imports` (no accessor on this surface reads `TS_UNRESOLVED_IMPORT`). +- **Known emitter limitation (declaration merging).** codeanalyzer-typescript mints one id for a + value and a type of the same name (`const X = …` + `interface X`, `const X = …` + `type X`, + `type X` + a field `X`), so its own `MERGE` collapses them onto one node carrying both labels and + the last writer's `kind`. The Neo4j backend rebuilds such a node as the one facet its containment + edge and labels name, keeps it out of the accessors for the other facet, and raises when the + labels name none or several. Three such nodes on the superset-frontend reference graph (1,841 + modules: 1,557 TypeScript, 284 JavaScript). +- Internal: the language-neutral helpers leg 1.5/1.6 built inside `cldk/analysis/python/` moved to + `cldk/analysis/commons/` — `bounds.py` (bounds, keyset paging, `EdgeOrder`), `graphs.py` + (bounded subgraphs, path assembly, the `SDG_RELS`/`VIA` tables as functions of the `P` prefix), + `keys.py` (module keys, scoping selectors, `module_key_of`, `module_dotted` with the extension set + as a parameter), `levels.py` (the `-a` integers), `artifacts.py` (the shared artifact-layer + reconstructors) and `backend.semver` (the version-floor parser). Python re-imports every name; + nothing changed in behaviour or count. + +### Removed +- `TypeScriptAnalysis.get_entry_point_methods` and `get_service_entry_point_methods`, which only ever + raised `NotImplementedError`. Working entrypoint accessors arrive with the query surface (2.5b). ## [v2.0.0-rc.2] - 2026-09-06 Python legs 1, 1.5 and 1.6 of the CLDK 2.0 agent-facing query facade (see diff --git a/CLAUDE.md b/CLAUDE.md index e36c713c..861ebfac 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ an optional read-only Neo4j backend — selected by the *type* of the `backend=` |----------|-------------|---------------|---------------|--------| | Java | `CLDK.java(...)` | `JCodeanalyzer` (bundled JAR, subprocess) | `JNeo4jBackend` | `cldk/models/java/` | | Python | `CLDK.python(...)` | `PyCodeanalyzer` (in-process `codeanalyzer-python`) | `PyNeo4jBackend` | re-exported from `codeanalyzer-python` | -| TypeScript | `CLDK.typescript(...)` | `TSCodeanalyzer` (`codeanalyzer-typescript` binary, subprocess) | `TSNeo4jBackend` | `cldk/models/typescript/` | +| TypeScript (+ JavaScript modules) | `CLDK.typescript(...)` | `TSCodeanalyzer` (`codeanalyzer-typescript` 1.2.0 binary from the wheel, subprocess; `-a 1..4`) | `TSNeo4jBackend` (graphs emitted by ≥ 1.2.0; older refused at attach) | `cldk/models/typescript/` (schema v2 mirror) | The legacy `CLDK(language="").analysis(...)` entry still works as a compat shim. Adding a language means a new factory method + facade + backend ABC/impl(s) + models + tests — **update this diff --git a/cldk/analysis/commons/artifacts.py b/cldk/analysis/commons/artifacts.py new file mode 100644 index 00000000..23b0918e --- /dev/null +++ b/cldk/analysis/commons/artifacts.py @@ -0,0 +1,101 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +"""The shared repository-artifact layer, rebuilt from the Neo4j projection: property maps → +``PyArtifact`` / ``PyConfigKey`` / ``PyDependency``. + +Every codeanalyzer projects this layer identically and unprefixed (``:Artifact``, ``:ConfigKey``, +``:Package``; ``HAS_ARTIFACT``, ``DEFINES_CONFIG``, ``DECLARES_DEPENDENCY``, ``LOCKS``), and the +generic ABC (:mod:`cldk.analysis.commons.backend`) promises the same four ``Py*`` models from every +language, so the reconstructors live here once. Lifted verbatim from +``cldk/analysis/python/neo4j/reconstruct.py`` (leg 2.5a), which re-exports them. +""" + +from __future__ import annotations + +from typing import Any, List, Mapping + +from cldk.models.python import PyArtifact, PyConfigKey, PyDependency, Span + +Props = Mapping[str, Any] + + +def config_key(props: Props) -> PyConfigKey: + """Rebuild a :class:`PyConfigKey` from a ``:ConfigKey`` node's properties. + + Line-only ``span`` (see :func:`body_node`): the projection writes ``start_line``/``end_line`` + and nothing finer, so the columns and byte offsets rehydrate as ``0``. ``span`` stays ``None`` + when the node carries no lines at all (best-effort extraction never located the key in the + artifact's source). + """ + lines = (props.get("start_line"), props.get("end_line")) + return PyConfigKey( + id=props.get("id", ""), + key=props.get("key", ""), + namespace=props.get("namespace", ""), + value=props.get("value"), + span=Span(start=(lines[0], 0), end=(lines[1], 0), bytes=(0, 0)) if None not in lines else None, + references=list(props.get("references", []) or []), + ) + + +def artifact(props: Props, *, config_keys: List[PyConfigKey] | None = None) -> PyArtifact: + """Rebuild a :class:`PyArtifact` from an ``:Artifact`` node's properties plus its fetched + :class:`PyConfigKey` children (``[:DEFINES_CONFIG]``). + + ``kind`` is not a projected property — every ``PyArtifact`` the analyzer emits carries the + model's own default (``"artifact"``; see ``codeanalyzer/artifacts/discovery.py``), so it is + supplied here rather than queried for. + """ + return PyArtifact( + id=props.get("id", ""), + kind="artifact", + path=props.get("path", ""), + format=props.get("format", ""), + roles=list(props.get("roles", []) or []), + size_bytes=props.get("size_bytes", 0), + sha256=props.get("sha256", ""), + source=props.get("source", ""), + extraction=props.get("extraction", "none"), + config_keys=config_keys or [], + ) + + +def dependency(props: Props, *, name: str, ecosystem: str, declared_in: str) -> PyDependency: + """Rebuild a :class:`PyDependency` from a ``[:DECLARES_DEPENDENCY]`` edge's properties plus its + endpoints (``name``/``ecosystem`` off the ``:Package`` node, ``declared_in`` off the + ``:Artifact`` node). ``ecosystem`` is a real ``Package`` property (``neo4j/schema.py``'s + ``Package`` node type carries it); ``"pypi"`` is only ever what the analyzer happens to write + there today (its only ecosystem, per ``PyDependency.ecosystem``'s own docstring) — read off the + node rather than hardcoded, so this doesn't silently go stale the day a second ecosystem ships. + + ``locked_version``/``provides_imports`` are projection-lossy here: the graph carries them on + the separate ``[:LOCKS]``/``[:PY_PROVIDES]`` edges (per-package facts, not per-declaration), and + no caller of this reconstruction chases those yet, so they come back at the model's own empty + defaults — the same class of gap :func:`callsite` documents for ``argument_types``. + """ + return PyDependency( + name=name, + ecosystem=ecosystem, + spec=props.get("spec", ""), + kind=props.get("kind", "runtime"), + extras=list(props.get("extras", []) or []), + declared_in=declared_in, + direct=props.get("direct", True), + provides_imports=[], + prov=list(props.get("prov", []) or []), + ) diff --git a/cldk/analysis/commons/backend.py b/cldk/analysis/commons/backend.py index b2f6d8c1..aea09a19 100644 --- a/cldk/analysis/commons/backend.py +++ b/cldk/analysis/commons/backend.py @@ -52,13 +52,25 @@ from __future__ import annotations +import re from abc import ABC, abstractmethod -from typing import ClassVar, Dict, Generic, List, TypeVar +from typing import Any, ClassVar, Dict, Generic, List, Tuple, TypeVar import networkx as nx from cldk.models.python import PyArtifact, PyConfigKey, PyConfigRead, PyConfigUseEdge, PyDependency + +def semver(raw: Any) -> Tuple[int, int, int] | None: + """``"1.4.1"`` (or ``"1.4.1.post0"``) as ``(1, 4, 1)``; ``None`` for anything that does not + start with three dotted integers, so an unparsable version is *unknown*, never silently zero. + + The parser behind every Neo4j backend's analyzer-version floor: the schema probe compares its + result against the backend's ``_ANALYZER_FLOOR`` and refuses the graph below it.""" + m = re.match(r"(\d+)\.(\d+)\.(\d+)", raw) if isinstance(raw, str) else None + return (int(m[1]), int(m[2]), int(m[3])) if m else None + + AppT = TypeVar("AppT") ModuleT = TypeVar("ModuleT") TypeT = TypeVar("TypeT") diff --git a/cldk/analysis/commons/backend_config.py b/cldk/analysis/commons/backend_config.py index 66c1787c..911b3582 100644 --- a/cldk/analysis/commons/backend_config.py +++ b/cldk/analysis/commons/backend_config.py @@ -77,13 +77,14 @@ class PyCodeAnalyzerConfig(CodeAnalyzerConfig): class TSCodeAnalyzerConfig(CodeAnalyzerConfig): """Select the in-process codeanalyzer backend for TypeScript. - Adds the TypeScript-only call-graph knob on top of :class:`CodeAnalyzerConfig`. + Kept distinct from :class:`CodeAnalyzerConfig` for the one knob it used to add; that knob is + now a no-op, and the class stays so existing call sites keep constructing it. Attributes: - tsc_only: If ``True``, restrict the analyzer to the tsc resolver call graph by passing - ``--tsc-only`` (codeanalyzer-typescript >= 0.4.2). Defaults to ``False`` (let the - binary choose its default). This is the supported replacement for the obsolete - ``--call-graph-provider both``. + tsc_only: **Deprecated, no-op.** codeanalyzer-typescript removed ``--tsc-only`` in 1.0.0: + the call graph always carries both resolvers, each edge tagged with its provenance + (``tsc`` / ``defuse`` / ``import``). Passing ``True`` emits a :class:`DeprecationWarning` + and changes nothing; filter edges by ``provenance`` instead. """ tsc_only: bool = False diff --git a/cldk/analysis/commons/bounds.py b/cldk/analysis/commons/bounds.py new file mode 100644 index 00000000..f32ded56 --- /dev/null +++ b/cldk/analysis/commons/bounds.py @@ -0,0 +1,357 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Bounds, selection and keyset paging: the language-neutral rulings every backend shares. + +Lifted out of the Python backend (leg 2.5a, G4) unchanged. Nothing here knows a language: a +``depth`` is a hop budget, a ``page_size`` is a count, a selector that names nothing is refused +the same way whether the names are Python paths or TypeScript ones. The per-language backends +import these rather than re-deriving them, which is what keeps two backends of one language -- +and the backends of two languages -- from drifting on what a keyword means. +""" + +from __future__ import annotations + +import base64 +import json +from bisect import bisect_right +from typing import Callable, Dict, List, NamedTuple, Sequence, Tuple + +from cldk.analysis.commons.results import EdgePage, SliceNode +from cldk.utils.exceptions import SelectorNotInGraph + + +def reject_bare_string(kind: str, values: object) -> None: + """Refuse a single string where a sequence of names is required. + + ``paths='pkg/mod.py'`` is not a type error to Python — a string *is* a sequence, of ten + characters — so it used to reach :func:`check_selector` as ten requested paths and come back as + ``10 of 10 paths not in graph: 'p', 'k', 'g', '/', …``. The mistake is the likely one because + the sibling keyword ``module=`` genuinely is single-valued, so both spellings look plausible. + + Raises: + TypeError: ``values`` is a ``str``. + """ + if isinstance(values, str): + raise TypeError(f"{kind}= takes a sequence of names, not a string; pass [{values!r}] to select just that one") + + +def check_selector(kind: str, requested: Sequence[str], missing: Sequence[str]) -> None: + """The one place a scoping keyword's *selection* is judged, for both backends. + + Every scoped accessor — ``get_symbol_table(paths=)``, ``get_classes(module=)``, + ``get_call_graph(roots=)`` — narrows a whole-application enumeration to what the caller named. + Two ways of naming nothing must not both come back as an empty result: + + * **an empty sequence** (``paths=[]``, ``roots=[]``) selected nothing while missing nothing. It + is a caller bug — the argument to omit is the argument that means "everything" — and it + raises the same :class:`ValueError` ``depth=`` without ``roots=`` already does. + * **values that match nothing** are the ambiguous empty the parent spec's D7 calls a defect: + a mistyped path and a module that genuinely declares no classes were the same ``{}``. They + raise :class:`~cldk.utils.exceptions.SelectorNotInGraph`, which names them and stops. It + offers no near-miss candidates on purpose — leg 1.5's E8 puts typo-tolerant matching out of + scope "not in the resolver, not in the error path". + + A **partial** miss raises too. Returning the values that did match would make a result whose + size the caller cannot check against what it asked for, which is the same silence one step + quieter. + + Args: + kind: The keyword's name, as it appears in the caller's own call — ``"paths"``, + ``"module"`` or ``"roots"``. + requested: Everything the keyword named, in the caller's spelling. + missing: The subset of ``requested`` that matched nothing. Callers with no membership + information to bring (``call_graph_scope``, which has not seen the graph yet) pass an + empty sequence and get only the empty-selection check. + + Raises: + ValueError: ``requested`` is empty. + SelectorNotInGraph: ``missing`` is non-empty. + """ + if not requested: + raise ValueError(f"{kind}= selected nothing; omit it to enumerate the whole application") + if missing: + # ``roots=`` is an exact filter, unlike every name-taking accessor on this surface, so a + # correct short name and a typo miss the same way -- the message has to say which + # vocabulary it wanted (see the assessment on PythonAnalysisBackend.get_call_graph). + detail = ( + "roots= takes full signatures (as get_callables_overview() reports them) or @external ids, not bare names; " + "to address a callable by name use resolve_callable(name).callable, or backward_cone / callers_of / call_paths_between" + if kind == "roots" + else None + ) + raise SelectorNotInGraph(kind, list(missing), len(requested), detail=detail) + + +def check_depth(depth: int | None) -> int | None: + """``depth`` is a hop budget: ``None`` for unbounded, otherwise an ``int`` of at least 1. + + Type-checked and not merely range-checked, because the two ways of getting it wrong are silent + otherwise: ``depth="2"`` raised ``TypeError`` from somewhere further in, and ``depth=2.5`` was + accepted and truncated to 2 by the Cypher/ego-graph radius. ``bool`` is rejected for the same + reason — ``depth=True`` is ``1`` by accident. + + One function, so ``get_call_graph``, the slices and the reachability accessors cannot come to + disagree about what a hop budget is. + """ + if depth is not None and (not isinstance(depth, int) or isinstance(depth, bool) or depth < 1): + raise ValueError(f"depth must be an int >= 1, got {depth!r}") + return depth + + +# ---------------------------------------------------------------------------------------------- +# Paging the per-callable graphs (E5). +# +# THE CANONICAL ORDER, defined once here because it is the only thing that makes a page mean the +# same thing on both backends. Neo4j returns rows in no order unless told to, and the local +# backend returns the analyzer's emission order; without one stated sort, page two on Neo4j is a +# different set of edges from page two locally. Each backend uses these functions -- the local one +# sorts and slices with them directly, the Neo4j one writes the same components into its ORDER BY +# and rebuilds the cursor from them -- so a change here moves both at once. +# +# The order is over the edge's OWN fields, in the order a reader would name them: source, then +# target, then whatever else the edge carries. Nothing positional and nothing backend-specific +# (no relationship element id, no row number), because a key one backend cannot compute is not a +# shared order. +# +# TOTALITY. A keyset cursor resumes strictly *after* a key, so a repeated key would drop its twin. +# The full field tuple is unique on real data: measured across odoo-slim-19's 5,134,655 PY_DDG, +# 247,906 PY_CFG_NEXT and 139,065 PY_CDG edges, zero (src, dst, ...) tuples repeat -- and for CFG +# the endpoints alone are *not* enough (13,310 node pairs carry two edges of different ``kind``), +# which is why ``kind`` is in the key. On the graph side the emitter MERGEs these relationships on +# exactly these properties, so uniqueness is structural there rather than incidental. Two edges +# equal in every field would be equal as values -- the models carry nothing else -- so their +# relative order is unobservable, and the page boundary is the same either way. +# +# ``or ""`` / ``or []`` is not cosmetic: ``DdgEdge.var`` is ``Optional[str]``, and a ``None`` in a +# sort key raises in Python and silently drops the row in Cypher (``null > x`` is null). The +# Cypher spells the same normalisation with ``coalesce``. + + +#: Edges per page when the caller does not say. 10,000 is where the measured distribution +#: splits: on odoo-slim-19, 15,520 of the 15,549 callables have fewer than 10,000 DDG edges, so +#: this default answers 99.8% of callables completely in one page and no caller of a normal +#: callable ever writes a loop -- while the 29 that are larger, up to 1,386,918 edges, are held to +#: a response a caller can actually hold. CFG and CDG max out at 402 and 314 edges on the same +#: application, so for them it is never reached. +DEFAULT_PAGE_SIZE = 10_000 + + +class EdgeOrder(NamedTuple): + """One edge kind's canonical order, in both spellings that have to agree. + + The Python sort key and the Cypher expressions are the same components said twice, in two + languages, and the whole point of the order is that the two never disagree — so they are + written down once, together, and each backend takes the half it can run. ``len(exprs)`` is + also the order's arity, which is how a cursor from one accessor is refused by another + (:func:`decode_cursor`): the three arities are 3, 2 and 4. + + ``coalesce`` in the expressions is ``or ""`` / ``or []`` in the key: ``DdgEdge.var`` is + optional, and a ``None`` in a sort key raises in Python and silently drops the row in Cypher. + """ + + key: Callable[[object], Tuple] + exprs: Tuple[str, ...] + + +def encode_cursor(scope: str, key: Tuple) -> str: + """An opaque, round-trippable spelling of a sort key, stamped with the callable it came from. + + Opaque on purpose: the caller passes it back and never reads it, so the components of the + order stay an implementation detail rather than joining the caller's vocabulary. Base64 of + JSON, because the key holds strings and a list of strings, and both survive that unchanged. + + ``scope`` is the resolved callable signature, carried so that :func:`decode_cursor` can refuse + a cursor minted for a different callable. Without it, an agent looping over callables and + reusing the wrong ``next_cursor`` would get a plausible page of the *right* callable's edges + resumed from a position in the *wrong* one — silently, since body-node ids sort by callable id + and the filter would simply skip everything or nothing. + """ + return base64.urlsafe_b64encode(json.dumps([scope, list(key)]).encode("utf-8")).decode("ascii") + + +def decode_cursor(cursor: str, scope: str, arity: int) -> Tuple: + """Inverse of :func:`encode_cursor`, checked against the caller it is being used for. + + Three ways a cursor can be wrong, all of them raising rather than being read as "start from + the beginning" — which would silently hand back page one when page nine was asked for: + it does not decode; it was minted for another callable; or it has the wrong number of + components, which is what a cursor from a *different accessor* looks like (the three orders + have arities 3, 2 and 4, so no cursor is silently valid for the wrong graph). + """ + try: + got_scope, key = json.loads(base64.urlsafe_b64decode(cursor.encode("ascii")).decode("utf-8")) + except Exception as exc: # noqa: BLE001 -- any decode failure is the same caller error + raise ValueError(f"not a cursor from a previous page: {cursor!r}") from exc + if got_scope != scope: + raise ValueError(f"this cursor is from a page of {got_scope!r}, not {scope!r}") + if len(key) != arity: + raise ValueError(f"cursor has {len(key)} components, this accessor's order has {arity}: {cursor!r}") + return tuple(key) + + +def check_page_size(page_size: int) -> int: + """``page_size`` must ask for at least one edge. + + Zero is refused rather than treated as "no limit": a page of nothing whose ``next_cursor`` can + never advance is an infinite loop dressed as an empty answer. + """ + if page_size < 1: + raise ValueError(f"page_size must be at least 1, got {page_size}") + return page_size + + +def keyset_where(exprs: Sequence[str]) -> str: + """The Cypher for "strictly after the cursor", written out because Cypher has no tuple + comparison: ``(a, b) > ($c0, $c1)`` has to become + ``a > $c0 OR (a = $c0 AND (b > $c1))``. + + Keyset rather than ``SKIP``: measured on ``Website.configurator_apply`` (1,386,918 DDG edges, + 10,000 per page, query alone), ``SKIP`` costs 2.6s for the first page, 9.0s for the middle one + and 4.3s for the last -- it re-sorts a prefix that grows with the offset -- while this filter + is flat at 3.1s / 2.9s / 2.4s. The offset form is not wrong, it just gets worse the further in + the caller reads, which is the one direction pagination exists to make cheap. + """ + clause = "" + for i in reversed(range(len(exprs))): + expr, param = exprs[i], f"$c{i}" + clause = f"{expr} > {param}" + (f" OR ({expr} = {param} AND ({clause}))" if clause else "") + return clause + + +def cursor_params(cursor: str, scope: str, arity: int) -> Dict[str, object]: + """The ``$c0…$cN`` bindings :func:`keyset_where` reads, from an opaque cursor.""" + return {f"c{i}": v for i, v in enumerate(decode_cursor(cursor, scope, arity))} + + +def edge_page(model, scope: str, edges: List, order: EdgeOrder, page_size: int, cursor: str | None) -> EdgePage: + """One page of an edge set already held in memory. + + The local backend has every edge in hand, so it sorts by ``key`` and slices. The cursor is + resolved by binary search over the sorted keys -- ``bisect_right``, i.e. the first edge + strictly after it -- so it means exactly what :func:`keyset_where` makes it mean on the graph, + rather than an independently-invented position that happens to line up. + """ + check_page_size(page_size) + key = order.key + rows = sorted(edges, key=key) + start = bisect_right([key(e) for e in rows], decode_cursor(cursor, scope, len(order.exprs))) if cursor is not None else 0 + window = rows[start : start + page_size] + more = start + len(window) < len(rows) + return EdgePage[model](edges=window, total=len(rows), next_cursor=encode_cursor(scope, key(window[-1])) if more and window else None) + + +#: Paths per query when the caller does not say. A path list is a set of *witnesses* for a flow, +#: not the flow's extent, and ten worked examples is already more than a reader will follow; the +#: extent question is ``slice_forward``, which reports a ``total``. +DEFAULT_MAX_PATHS = 10 + + +def check_max_paths(max_paths: int) -> int: + """``max_paths`` must admit at least one path. Zero is refused for :func:`check_max_nodes`'s + reason: an empty list whose ``truncated`` says "there were more" answers nothing, and it is + indistinguishable at a glance from "there is no flow".""" + if max_paths < 1: + raise ValueError(f"max_paths must be at least 1, got {max_paths}") + return max_paths + + +def check_distinct_endpoints(src: SliceNode, dst: SliceNode) -> None: + """A path query must have two different endpoints. + + Neo4j's shortest-path search *refuses* a self-question outright ("the shortest path algorithm + does not work when the start and end nodes are the same"), which would otherwise surface as a + raw driver error from one backend and an empty list from the other. Both raise here instead, + and neither answers ``[]``: for a node that genuinely sits on a cycle, ``[]`` would be + indistinguishable from a proved absence of one, which is the ambiguous empty in another + costume. ``reaches(x, x)`` is the accessor that answers the existence question, and it does + terminate (measured: 0.03s, where the obvious ``EXISTS`` spelling never finished). + + Takes the *resolved* endpoints rather than their refs so the message speaks the caller's + vocabulary (E6/E7): a value is named ``'kwargs' within '….configurator_apply'``, a callable + by its signature, and the advice is a call that actually runs -- ``reaches`` takes callable + names, so for a value the cycle question is asked of its enclosing callable. + """ + if src.ref != dst.ref: + return + if src.kind == "callable": + raise ValueError(f"paths from {src.callable!r} to itself are not answered; ask reaches({src.callable!r}, {src.callable!r}) whether a cycle exists") + raise ValueError( + f"paths from {src.name!r} to itself (within {src.callable!r}) are not answered; a value reaches itself only through " + f"recursion, so ask reaches({src.callable!r}, {src.callable!r}) whether the callable is on a call cycle" + ) + + +#: Nodes per slice when the caller does not say. The same 10,000 as :data:`DEFAULT_PAGE_SIZE`, and +#: for a different reason: there, it is where 99.8% of callables fit in one page; here, nothing +#: fits, because the measured distribution has no middle (see +#: :class:`~cldk.analysis.commons.results.Slice`). 10,000 is the largest result that stays +#: readable, and every slice above it is one a caller should be re-asking with ``depth=``. +DEFAULT_MAX_NODES = 10_000 + +#: Hops from the seed when the caller does not say. **Finite, and that is the whole point.** +#: +#: The measured distribution has no middle (see :class:`~cldk.analysis.commons.results.Slice`), so +#: an unbounded default hands a connected seed 10,000 arbitrary nodes of a 195,819-node closure -- +#: an unprincipled 5%, honestly flagged ``truncated`` and useless either way. A finite default +#: answers a *narrower* question *completely* instead, and ``depth=None`` is how a caller asks for +#: the whole cone. +#: +#: 5 is the largest bound at which no measured slice needs ``max_nodes`` at all. Over 120 random +#: ``formal_in`` seeds with callers on odoo-slim-19, node counts by depth: +#: +#: ========= ===== ======= ======= ======= ======== +#: direction depth median p75 max > 10,000 +#: ========= ===== ======= ======= ======= ======== +#: backward 3 14 70 846 0 +#: backward 5 33 188 1,539 0 +#: backward 6 56 324 2,818 0 +#: backward 8 464 2,044 16,028 1 +#: backward None 195,786 195,787 198,306 79 +#: forward 3 12 34 440 0 +#: forward 5 24 63 1,053 0 +#: forward 6 35 166 14,260 1 +#: forward 8 48 402 37,326 2 +#: forward None 71 440,269 440,645 52 +#: ========= ===== ======= ======= ======= ======== +#: +#: 3 is informative but thin; 6 is where a forward slice first exceeds the cap and the default +#: would start truncating again. 5 is the last depth that never does, in either direction. +#: +#: **Which accessors take it, and which deliberately do not.** The three *slices* +#: (``slice_backward``, ``slice_forward``, ``backward_cone``) default to it: a bounded slice is a +#: *complete* answer to a narrower question, and ``total`` says so. The two *predicates* +#: (``reaches``, ``flows_to_call``, ``flows_to_argument``) and the two *path* queries +#: (``paths_between``, ``call_paths_between``) default to ``None`` -- unbounded -- because a hop +#: budget on a boolean or a path list is not a smaller answer but a **wrong** one: "no flow" and +#: "no flow within five hops" collapse into the same ``False`` / ``[]`` with nothing in the result +#: to tell them apart. Measured on odoo-slim-19: ``flows_to_call("kwargs", "Website.create", +#: within="Website.configurator_apply")`` is ``False`` at five hops and ``True`` unbounded, and +#: the matching ``paths_between`` is ``[]`` at five hops and ten paths at eight. ``depth=`` stays +#: on all five as an explicit narrowing a caller can name; it is only the *default* that differs. +DEFAULT_DEPTH = 5 + + +def check_max_nodes(max_nodes: int) -> int: + """``max_nodes`` must admit at least one node — the seed, if nothing else. + + Zero is refused rather than read as "no limit": a slice of nothing whose ``total`` says + 195,784 is a result no caller can act on, and "unbounded" is what ``max_nodes=None`` would + have to mean if it ever meant anything. + """ + if max_nodes < 1: + raise ValueError(f"max_nodes must be at least 1, got {max_nodes}") + return max_nodes diff --git a/cldk/analysis/commons/graphs.py b/cldk/analysis/commons/graphs.py new file mode 100644 index 00000000..5a130044 --- /dev/null +++ b/cldk/analysis/commons/graphs.py @@ -0,0 +1,253 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Graph walks and path assembly: the language-neutral half of slicing and reachability. + +Lifted out of the Python backend (leg 2.5a, G4) unchanged except for three parameters: the +relationship-type prefix (``sdg_rels(P)`` / ``via_table(P)`` where the Python backend had +``PY_``-spelled tables) and the ``via`` map :func:`flow_path` translates through. The per-language +backend binds each once and hands the bound object down. +""" + +from __future__ import annotations + +from typing import Callable, Iterable, List, Literal, Mapping, Sequence, Tuple + +import networkx as nx + +from cldk.analysis.commons.bounds import check_selector, reject_bare_string +from cldk.analysis.commons.results import FlowPath, PathHop, SliceNode + + +def bounded_subgraph(graph: nx.DiGraph, roots: List[str], depth: int | None, declared: Iterable[str]) -> nx.DiGraph: + """The sub-call-graph reachable from ``roots``, within ``depth`` hops when given. + + **Induced**, not path-only: every edge between two reached nodes is kept, including one + pointing back towards a root. A path-only answer would let ``graph.predecessors(n)`` lie about + a node the caller can see, which is a worse defect than the extra edges are a cost. The Neo4j + backend's Cypher is written to produce the same induced shape rather than the cheaper + edges-along-the-path shape, for exactly this reason. + + **The domain a root is judged against — stated here because both backends must judge against + the same one — is the callable inventory, not this graph.** ``graph`` is built from call + *edges* alone, so a callable that neither calls nor is called by anything is not a node in it: + 444 of the live odoo application's 15,549 in-scope callables, 2.9%. Checking membership of + ``graph`` therefore raised for a callable that plainly exists, while the Neo4j backend — whose + Cypher matches a root by node *label*, not by edge participation — returned the one-node graph + it is. ``declared`` closes that gap: it carries every callable the application declares, and a + root is valid when it is **in the inventory or is a node of the graph**. The second disjunct is + not redundant — an ``@external`` ghost is a legitimate root, is a graph node, and is not a + declared callable — and the union is exactly what the Neo4j root match accepts (a + ``:PyCallable`` of this application, or a ``:PyExternal``). + + A root outside that domain raises (:func:`check_selector`) rather than contributing nothing: + "no such callable" and "a callable that calls nothing" are different answers, and before this + they were the same empty graph. + + The returned graph stays **edge-induced**. An isolated root is added back as a lone node — + which is the answer, and the one Neo4j gives — but nothing else the inventory knows about is + seeded into it. Seeding all declared callables would make the unbounded local graph disagree + with Neo4j's node-for-node, trading one parity defect for a larger one. + """ + inventory = set(declared) + check_selector("roots", roots, [r for r in roots if r not in graph and r not in inventory]) + nodes: set = set() + isolated: set = set() + for root in roots: + if root not in graph: + isolated.add(root) # declared, but in no call edge: its own one-node graph + elif depth is None: + nodes |= nx.descendants(graph, root) | {root} + else: + nodes |= set(nx.ego_graph(graph, root, radius=depth).nodes) + sub = graph.subgraph(nodes).copy() + sub.add_nodes_from(isolated) + return sub + + +# The structural half of the per-callable graph orders. The components and their sequence are +# what make a page mean the same thing on both backends (see the paging block in ``bounds``); the +# per-language backend binds each to its own edge model -- ``cfg_sort_key(edge: CfgEdge)`` in the +# Python backend -- so the typed name a reader greps for stays where the type lives. +_EDGE_KEYS: dict[str, Callable[[object], Tuple]] = { + "cfg": lambda e: (e.src, e.dst, e.kind or ""), + "cdg": lambda e: (e.src, e.dst), + "ddg": lambda e: (e.src, e.dst, e.var or "", list(e.prov or [])), +} + + +def edge_sort_key(kind: Literal["cfg", "cdg", "ddg"]) -> Callable[[object], Tuple]: + """The canonical sort key for one per-callable graph kind, over the edge's own fields. + + ``cfg``: source, target, kind. ``cdg``: source, target. ``ddg``: source, target, variable, + provenance. ``or ""`` / ``or []`` because an optional field's ``None`` in a sort key raises in + Python and silently drops the row in Cypher; the Cypher spells it ``coalesce``. + """ + return _EDGE_KEYS[kind] + + +# ---------------------------------------------------------------------------------------------- +# Slicing and reachability (E2, E3, E5). +# +# THE FIVE RELATIONSHIP TYPES A SLICE FOLLOWS, verified against codeanalyzer's own +# ``neo4j/schema.py`` REL_TYPES and against ``CALL db.relationshipTypes()`` on odoo-slim-19 rather +# than copied from a plan -- the names in this leg's plan have been wrong before (PY_CFG_NEXT is +# not PY_CFG). All five exist, with these edge counts on that application: +# +# PY_DDG 5,134,655 data dependence, within a callable (var, prov) +# PY_CDG 139,065 control dependence, within a callable +# PY_PARAM_IN 229,035 actual_in -> formal_in : an argument entering a callee +# PY_PARAM_OUT 133,267 formal_out -> actual_out : a value coming back to the caller +# PY_SUMMARY 453,398 actual_in -> actual_out : a callee's pass-through, at the call site +# +# All five point WITH the flow -- verified on the live graph, where every PY_PARAM_IN runs +# actual_in -> formal_in and every PY_PARAM_OUT runs formal_out -> actual_out, with no exceptions +# in 362,302 edges. So a forward slice follows them and a backward slice follows them reversed; +# there is no per-type direction table to keep straight, which is why they can share one match. +# +# PY_CFG_NEXT is deliberately NOT here. Control *flow* says what runs next; a slice is about what +# a value or a decision depends on, and following successor edges would pull in every later +# statement whether or not it depends on anything -- the "returns the whole callable" bug that a +# non-emptiness assertion cannot catch. +# +# The table is a function of the backend's relationship prefix (``AnalysisBackend.P``): the five +# kinds and their meaning are the analyzer family's, the ``PY_`` / ``TS_`` spelling is one language's. + + +def sdg_rels(P: str) -> tuple[str, ...]: + """The five relationship types a slice follows, spelled with the language's prefix ``P``.""" + return (f"{P}_DDG", f"{P}_CDG", f"{P}_PARAM_IN", f"{P}_PARAM_OUT", f"{P}_SUMMARY") + + +def sdg_rel_pattern(P: str) -> str: + """The Cypher spelling of :func:`sdg_rels` for a relationship-type disjunction.""" + return "|".join(sdg_rels(P)) + + +#: The caller's word for each relationship a path hop can be justified by (E6). The graph's own +#: ``PY_DDG``/``PY_PARAM_IN`` spelling never leaves the backend; both backends translate through +#: this one table so a hop cannot be labelled ``data`` over Neo4j and ``ddg`` locally. +#: +#: ``argument`` and ``return`` are the two interprocedural edges, and they are deliberately not +#: both called "parameter": ``PY_PARAM_IN`` binds a caller's argument to a callee's formal, and +#: ``PY_PARAM_OUT`` binds a callee's result back into the caller. A reader following a path needs +#: to know which way it just crossed a call boundary. +def via_table(P: str) -> dict[str, str]: + """The relationship-type -> caller's-word table above, for the language whose prefix is ``P``.""" + return { + f"{P}_DDG": "data", + f"{P}_CDG": "control", + f"{P}_PARAM_IN": "argument", + f"{P}_PARAM_OUT": "return", + f"{P}_SUMMARY": "summary", + f"{P}_CALLS": "call", + } + + +def hop_sort_key(hops: Sequence[PathHop]) -> Tuple: + """The order two paths are compared in, in the caller's *own* vocabulary. + + E2 makes a path a sequence, which only means something if the *list* of paths is stable too: + ``max_paths`` truncates, and a truncation of a non-deterministic order is not reproducible. + So paths are ordered shortest first, then hop by hop on ``(via, var, to.ref)`` — every term of + which the caller can see in the result it gets back. + + Two hops that are indistinguishable in that vocabulary (parallel edges of the same kind, on + the same variable, between the same two nodes) are left to a backend-local tie-break: the + Neo4j backend appends the relationship's ``elementId``, the local backend keeps the order the + analyzer emitted them in. Either is stable for repeated calls against one graph; neither is + meaningful to a caller, which is why it is last and why nothing above depends on it. + """ + return (len(hops), tuple((h.via, h.var or "", h.to.ref) for h in hops)) + + +def flow_path(nodes: Sequence[SliceNode], edges: Sequence[Tuple[str, "str | None", "Sequence[str] | None"]], *, via: Mapping[str, str]) -> FlowPath: + """Join a walk's ``n`` nodes and its ``n - 1`` edges into a :class:`FlowPath`. + + Both backends build paths through here, which is what makes the joining invariant + (``hops[i].to is hops[i + 1].frm``) a property of the construction rather than something each + backend has to be trusted to preserve. ``edges`` are the graph's own relationship types; they + are translated to the caller's word through ``via`` (the backend's :func:`via_table`) exactly once, here. + + Raises: + KeyError: A relationship type with no word in ``via`` — a new edge kind from a future + analyzer generation, which must be named before it can be reported rather than passed + through in the graph's spelling. + """ + return FlowPath(hops=[PathHop(frm=nodes[i], to=nodes[i + 1], via=via[rel], var=var, prov=list(prov or [])) for i, (rel, var, prov) in enumerate(edges)]) + + +def as_slice_node(node: object) -> SliceNode: + """The :class:`~cldk.analysis.commons.results.SliceNode` for anything carrying an address. + + :meth:`PythonAnalysisBackend.describe` takes "anything with a ``ref``" — slice nodes, the + endpoints of a :class:`~cldk.analysis.commons.results.PathHop`, a + :class:`~cldk.analysis.commons.results.LocateResult` — because the addressing layer hands a + caller three shapes and asking them to convert between shapes to hydrate one is the kind of + friction that gets worked around with string surgery. + + A ``SliceNode`` passes through untouched. A ``LocateResult`` is re-expressed as one, keeping + the vocabulary it already speaks: ``module.path`` is the file, ``callable.signature`` the + enclosing callable, ``node.kind`` the position's kind. + + Raises: + TypeError: ``node`` carries neither a ``ref`` nor a ``node_id``, so there is nothing to + look up. Guessing an address from a file and a line is what ``locate`` is for. + """ + if isinstance(node, SliceNode): + return node + ref = getattr(node, "node_id", None) + if ref is None: + raise TypeError(f"describe() needs something carrying a ref (a SliceNode, a path hop endpoint, a locate() result); got {type(node).__name__}") + module, callable_ref, body = node.module, node.callable, getattr(node, "node", None) + return SliceNode( + file=module.path, + line=node.span.start[0], + callable=callable_ref.signature if callable_ref else "", + kind=body.kind if body else "callable", + name=callable_ref.name if callable_ref else None, + source=node.source or None, + ref=ref, + ) + + +def cone_sinks(resolve: Callable[[str], SliceNode], sinks: Sequence[str]) -> List[SliceNode]: + """Resolve ``backward_cone``'s sinks, refusing the two ways of naming nothing. + + The same discipline :func:`check_selector` applies to ``roots=`` and ``paths=``: a bare string + is ten one-character sinks and is refused as a type error, and an empty sequence is refused + because "everything" is the argument omitted, not the argument emptied — and there is no + "everything" here to fall back to. Each surviving name goes through ``resolve``, so an + ambiguous sink raises listing candidates instead of one of them being picked. + + Duplicates are collapsed by resolved signature, not by the string the caller wrote: naming the + same callable twice, once bare and once qualified, is one sink. + """ + reject_bare_string("sinks", sinks) + if not sinks: + raise ValueError("sinks= names nothing to walk back from; pass at least one callable") + resolved = {node.callable: node for node in (resolve(s) for s in sinks)} + return list(resolved.values()) + + +def slice_resolved(roots: List[SliceNode]) -> str: + """The audit line on a :class:`~cldk.analysis.commons.results.Slice`: what the caller's names + matched, in the caller's vocabulary. + + Both backends build it here rather than each formatting its own, so a caller comparing two + results is comparing answers and not two spellings of one. + """ + return ", ".join(f"{r.callable} {r.kind} {r.name!r}" if r.kind != "callable" else r.callable for r in roots) diff --git a/cldk/analysis/commons/keys.py b/cldk/analysis/commons/keys.py new file mode 100644 index 00000000..496e2ce0 --- /dev/null +++ b/cldk/analysis/commons/keys.py @@ -0,0 +1,158 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Module keys and scoping: how a caller's path names a module, in any language. + +Lifted out of the Python backend and its Neo4j reconstruction (leg 2.5a, G4) unchanged except +that :func:`module_dotted` takes the language's source extensions as a parameter. Keys are the +repo-relative paths the analyzer saw; every ruling here is about matching a caller's spelling of +one to the key that exists, or refusing. +""" + +from __future__ import annotations + +import os +import posixpath +from typing import Collection, Iterable, List, Sequence + +from cldk.analysis.commons.bounds import check_depth, check_selector, reject_bare_string + + +def resolve_module_key(path: str, keys: Iterable[str]) -> str: + """The symbol-table / graph ``file_key`` naming ``path``, or ``path`` unchanged if none does. + + A caller of :meth:`PythonAnalysisBackend.locate` hands over whatever its scanner printed — + ``./src/app.py``, ``src/../src/app.py``, or an absolute path from the machine the scan ran on — + while both backends are keyed by the project-relative path the analyzer saw. Exact key first, + then the normalised form, then the longest known key the normalised path *ends on a segment + boundary* of (which is what an absolute path is). Returning ``path`` unchanged when nothing + matches is deliberate: the caller then gets ``file_not_in_graph`` naming the path it asked + about, not a silently substituted neighbour. + """ + keys = list(keys) + if path in keys: + return path + norm = posixpath.normpath(str(path).replace(os.sep, "/")) + if norm in keys: + return norm + suffix_matches = [k for k in keys if norm.endswith("/" + k)] + return max(suffix_matches, key=len) if suffix_matches else path + + +def scope_paths(paths: Sequence[str] | None, keys: Iterable[str], kind: str = "paths") -> List[str] | None: + """Resolve requested module paths to symbol-table keys, or ``None`` for "the whole application". + + Both backends route their ``paths=`` / ``module=`` keywords through here, so the lenient + resolution (:func:`resolve_module_key` — an absolute path or one with native separators finds + its module) and the strictness (:func:`check_selector` — a path naming no module raises) cannot + drift apart between them. + + Args: + paths: What the caller named, or ``None`` for the unscoped call. + keys: The symbol-table keys that exist — ``symbol_table.keys()`` locally, the + application's module ``file_key``s over Neo4j. + kind: The keyword's name for the error message; ``"module"`` for ``get_classes``, whose + single-valued keyword routes through here as a one-element sequence. + + **Resolution is many-to-one, and the result is de-duplicated.** Leniency is the whole point of + :func:`resolve_module_key` — ``"pkg/a.py"`` and ``"/abs/pkg/a.py"`` are two spellings a scanner + may plausibly hand over for the *same* module — so two requested paths legitimately collapse to + one key and the caller gets one entry back. Raising on the collapse would punish the very + caller the leniency exists for; de-duplicating explicitly is what keeps the returned list from + naming the same module twice and asking both backends to fetch it twice. + + Raises: + TypeError: ``paths`` is a bare string (see :func:`reject_bare_string`). + ValueError: ``paths`` is an empty sequence. + SelectorNotInGraph: a path names no module in this application. + """ + reject_bare_string(kind, paths) + if paths is None: + return None + known = list(keys) + resolved = [resolve_module_key(p, known) for p in paths] + check_selector(kind, list(paths), [p for p, r in zip(paths, resolved) if r not in known]) + return list(dict.fromkeys(resolved)) + + +def call_graph_scope(roots: Sequence[str] | None, depth: int | None) -> List[str] | None: + """Normalise :meth:`PythonAnalysisBackend.get_call_graph`'s scoping keywords. + + Returns the roots as a list, or ``None`` for "the whole application" — the unscoped call, + which must keep behaving exactly as it did before the keywords existed. + + Both backends route through this so the two cannot drift apart on what a keyword combination + means (the failure mode Fix 1 of leg 1.5 had to go back and repair on the child-fetch paths). + Whether each root *exists* is checked later, by whichever backend has the graph in hand, but + through the same :func:`check_selector` — see :func:`bounded_subgraph`. + + Raises: + TypeError: ``roots`` is a bare string (see :func:`reject_bare_string`). + ValueError: ``depth`` that is not a positive ``int``, ``depth`` without ``roots``, or an + empty ``roots``. A hop budget with no origin to count from has no meaning, and quietly + returning all 364,752 edges would be the worst of the available answers — the caller + asked for a bounded graph and would be handed an unbounded one with no signal. + ``depth`` is type-checked rather than merely range-checked because the two ways of + getting it wrong are silent otherwise: ``depth="2"`` raised ``TypeError`` from the + comparison, and ``depth=2.5`` was accepted and truncated to 2 by the Cypher/ego-graph + radius. ``bool`` is rejected for the same reason — ``depth=True`` is ``1`` by accident. + """ + check_depth(depth) + reject_bare_string("roots", roots) + if roots is None: + if depth is not None: + raise ValueError("depth= requires roots=; a hop budget needs an origin to count from") + return None + check_selector("roots", list(roots), ()) + return list(roots) + + +def module_key_of(node_id: str, prefix: str, known: Collection[str]) -> str: + """The repo-relative module key embedded in a ``can://`` id (F4). + + Ids are ``/`` (or exactly ```` for a module), and a + file key can itself contain ``.py/`` as a directory name, so the key is never recovered by + splitting: every ``/``-boundary prefix of the id is tried longest first and the first that is + a member of ``known`` -- the application's verified module keys -- wins. A miss raises: a key + we cannot verify is a defect, not a guess. ``known`` should be a set; this runs once per row. + """ + if not node_id.startswith(prefix): + raise KeyError(node_id) + parts = node_id[len(prefix) :].split("/") + for n in range(len(parts), 0, -1): + candidate = "/".join(parts[:n]) + if candidate in known: + return candidate + raise KeyError(node_id) + + +def module_dotted(path: str, *, extensions: Sequence[str] = (".py",)) -> str: + """The dotted module name a repo-relative path spells: ``"odoo/tools/mail.py"`` → + ``"odoo.tools.mail"``, ``"pkg/__init__.py"`` → ``"pkg"``. The same derivation the analyzer's + signatures embody, so ``in_module=`` can be written the way a signature reads. ``extensions`` is + the language's source suffixes; the Python default keeps every existing call site as it was. + + One step is **Python-specific and unconditional**: a trailing ``/__init__`` is stripped, because + a Python package's ``__init__.py`` is addressed by the package name. No other language's + conventional index file is stripped -- a TypeScript ``src/foo/index.ts`` dots to + ``src.foo.index``, not ``src.foo`` -- and callers that want that must not rely on this helper + for it. Harmless where the convention does not exist (no ``__init__`` segment, nothing to + strip); if a second language ever needs its own index name, that is a parameter, not a + branch here.""" + stem = next((path[: -len(ext)] for ext in extensions if path.endswith(ext)), path) + if stem.endswith("/__init__"): + stem = stem[: -len("/__init__")] + return stem.replace("/", ".") diff --git a/cldk/analysis/commons/levels.py b/cldk/analysis/commons/levels.py new file mode 100644 index 00000000..0eb4de40 --- /dev/null +++ b/cldk/analysis/commons/levels.py @@ -0,0 +1,56 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +"""The SDK's :class:`~cldk.analysis.AnalysisLevel` names mapped to the analyzers' integer levels. + +Every codeanalyzer takes ``-a 1..4`` (symbol table, call graph, intraprocedural dataflow, +interprocedural SDG); the SDK names those levels once, here, so no backend sends the enum's +display string or caps the level on the caller's behalf. +""" + +from __future__ import annotations + +from cldk.analysis import AnalysisLevel + +#: The analyzer's ``-a`` integer for each SDK level. +ANALYZER_LEVELS = { + AnalysisLevel.symbol_table: 1, + AnalysisLevel.call_graph: 2, + AnalysisLevel.program_dependency_graph: 3, + AnalysisLevel.system_dependency_graph: 4, +} + +#: The inverse, by the member name a caller writes (``"call_graph"``, not ``"call graph"``) — so +#: an error about the level in use names it the way it was asked for. +LEVEL_NAMES = {n: lvl.name for lvl, n in ANALYZER_LEVELS.items()} + + +def analyzer_level(level: "AnalysisLevel | str") -> int: + """The analyzer's integer level for one of the SDK's :class:`~cldk.analysis.AnalysisLevel` + names. + + Accepts the enum, its value (``"call graph"``) and its member name (``"call_graph"``): the + facade's parameter is typed ``str``, and the underscore spelling is what a caller writing + ``analysis_level="system_dependency_graph"`` produces. An unrecognised name raises rather than + falling back to a default — a level that silently becomes 1 is the defect this function exists + to close. + """ + key = str(getattr(level, "value", level)).replace("_", " ") + try: + return ANALYZER_LEVELS[AnalysisLevel(key)] + except ValueError: + raise ValueError(f"unknown analysis_level {level!r}; expected one of {[lvl.name for lvl in AnalysisLevel]}") from None diff --git a/cldk/analysis/commons/resolve.py b/cldk/analysis/commons/resolve.py index 34e68cd9..608ed891 100644 --- a/cldk/analysis/commons/resolve.py +++ b/cldk/analysis/commons/resolve.py @@ -49,6 +49,7 @@ from typing import Callable, List, NamedTuple, Optional, Sequence, TypeVar +from cldk.analysis.commons.keys import module_dotted from cldk.utils.exceptions import AmbiguousName, SelectorNotInGraph T = TypeVar("T") @@ -246,15 +247,6 @@ class is excluded outright, not silently kept. return resolve_name(name, [c.signature for c in candidates], kind="callable", narrow_with=narrow_with) -def module_dotted(path: str) -> str: - """The dotted module name a repo-relative path spells: ``"odoo/tools/mail.py"`` → - ``"odoo.tools.mail"``, ``"pkg/__init__.py"`` → ``"pkg"``. The same derivation the analyzer's - signatures embody, so ``in_module=`` can be written the way a signature reads.""" - stem = path[:-3] if path.endswith(".py") else path - if stem.endswith("/__init__"): - stem = stem[: -len("/__init__")] - return stem.replace("/", ".") - def resolve_within(resolve_callable: Callable[[str], "T"], within: str) -> "T": """Resolve a ``within=`` argument, re-raising an ambiguity in terms the caller can act on. diff --git a/cldk/analysis/python/backend.py b/cldk/analysis/python/backend.py index 230b9cef..fec23d73 100644 --- a/cldk/analysis/python/backend.py +++ b/cldk/analysis/python/backend.py @@ -29,19 +29,48 @@ from __future__ import annotations -import base64 -import json -import os -import posixpath from abc import abstractmethod -from bisect import bisect_right -from typing import Callable, Dict, Iterable, List, NamedTuple, Sequence, Tuple +from typing import Dict, List, Sequence, Tuple import networkx as nx from cldk.analysis.commons.backend import AnalysisBackend -from cldk.analysis.commons.results import EdgePage, EntrypointCoverage, FlowPath, FlowPaths, LocateResult, PathHop, Slice, SliceNode -from cldk.utils.exceptions import SelectorNotInGraph + +# The language-neutral rulings live in ``commons`` (leg 2.5a, G4) and are re-exported here under +# the names this backend's callers and tests have always imported them by. +from cldk.analysis.commons.bounds import ( + DEFAULT_DEPTH, + DEFAULT_MAX_NODES, + DEFAULT_MAX_PATHS, + DEFAULT_PAGE_SIZE, + EdgeOrder, + check_depth, + check_distinct_endpoints, + check_max_nodes, + check_max_paths, + check_page_size, + check_selector, + cursor_params, + decode_cursor, + edge_page, + encode_cursor, + keyset_where, + reject_bare_string, +) +from cldk.analysis.commons.graphs import ( + as_slice_node, + bounded_subgraph, + cone_sinks, + edge_sort_key, + flow_path, + hop_sort_key, + sdg_rel_pattern, + sdg_rels, + slice_resolved, + via_table, +) +from cldk.analysis.commons.keys import call_graph_scope, resolve_module_key, scope_paths +from cldk.analysis.commons.results import EdgePage, EntrypointCoverage, FlowPaths, LocateResult, Slice, SliceNode from cldk.models.python import ( CdgEdge, CfgEdge, @@ -58,27 +87,6 @@ ) -def resolve_module_key(path: str, keys: Iterable[str]) -> str: - """The symbol-table / graph ``file_key`` naming ``path``, or ``path`` unchanged if none does. - - A caller of :meth:`PythonAnalysisBackend.locate` hands over whatever its scanner printed — - ``./src/app.py``, ``src/../src/app.py``, or an absolute path from the machine the scan ran on — - while both backends are keyed by the project-relative path the analyzer saw. Exact key first, - then the normalised form, then the longest known key the normalised path *ends on a segment - boundary* of (which is what an absolute path is). Returning ``path`` unchanged when nothing - matches is deliberate: the caller then gets ``file_not_in_graph`` naming the path it asked - about, not a silently substituted neighbour. - """ - keys = list(keys) - if path in keys: - return path - norm = posixpath.normpath(str(path).replace(os.sep, "/")) - if norm in keys: - return norm - suffix_matches = [k for k in keys if norm.endswith("/" + k)] - return max(suffix_matches, key=len) if suffix_matches else path - - def body_key_column(key: str) -> int: """The start column encoded in a body node's local key (``"21:12"`` -> ``12``), or ``-1``. @@ -97,246 +105,19 @@ def body_key_column(key: str) -> int: return int(col) if col.isdigit() else -1 -def reject_bare_string(kind: str, values: object) -> None: - """Refuse a single string where a sequence of names is required. - - ``paths='pkg/mod.py'`` is not a type error to Python — a string *is* a sequence, of ten - characters — so it used to reach :func:`check_selector` as ten requested paths and come back as - ``10 of 10 paths not in graph: 'p', 'k', 'g', '/', …``. The mistake is the likely one because - the sibling keyword ``module=`` genuinely is single-valued, so both spellings look plausible. - - Raises: - TypeError: ``values`` is a ``str``. - """ - if isinstance(values, str): - raise TypeError(f"{kind}= takes a sequence of names, not a string; pass [{values!r}] to select just that one") - - -def check_selector(kind: str, requested: Sequence[str], missing: Sequence[str]) -> None: - """The one place a scoping keyword's *selection* is judged, for both backends. - - Every scoped accessor — ``get_symbol_table(paths=)``, ``get_classes(module=)``, - ``get_call_graph(roots=)`` — narrows a whole-application enumeration to what the caller named. - Two ways of naming nothing must not both come back as an empty result: - - * **an empty sequence** (``paths=[]``, ``roots=[]``) selected nothing while missing nothing. It - is a caller bug — the argument to omit is the argument that means "everything" — and it - raises the same :class:`ValueError` ``depth=`` without ``roots=`` already does. - * **values that match nothing** are the ambiguous empty the parent spec's D7 calls a defect: - a mistyped path and a module that genuinely declares no classes were the same ``{}``. They - raise :class:`~cldk.utils.exceptions.SelectorNotInGraph`, which names them and stops. It - offers no near-miss candidates on purpose — leg 1.5's E8 puts typo-tolerant matching out of - scope "not in the resolver, not in the error path". - - A **partial** miss raises too. Returning the values that did match would make a result whose - size the caller cannot check against what it asked for, which is the same silence one step - quieter. - - Args: - kind: The keyword's name, as it appears in the caller's own call — ``"paths"``, - ``"module"`` or ``"roots"``. - requested: Everything the keyword named, in the caller's spelling. - missing: The subset of ``requested`` that matched nothing. Callers with no membership - information to bring (``call_graph_scope``, which has not seen the graph yet) pass an - empty sequence and get only the empty-selection check. - - Raises: - ValueError: ``requested`` is empty. - SelectorNotInGraph: ``missing`` is non-empty. - """ - if not requested: - raise ValueError(f"{kind}= selected nothing; omit it to enumerate the whole application") - if missing: - # ``roots=`` is an exact filter, unlike every name-taking accessor on this surface, so a - # correct short name and a typo miss the same way -- the message has to say which - # vocabulary it wanted (see the assessment on PythonAnalysisBackend.get_call_graph). - detail = ( - "roots= takes full signatures (as get_callables_overview() reports them) or @external ids, not bare names; " - "to address a callable by name use resolve_callable(name).callable, or backward_cone / callers_of / call_paths_between" - if kind == "roots" - else None - ) - raise SelectorNotInGraph(kind, list(missing), len(requested), detail=detail) - - -def scope_paths(paths: Sequence[str] | None, keys: Iterable[str], kind: str = "paths") -> List[str] | None: - """Resolve requested module paths to symbol-table keys, or ``None`` for "the whole application". - - Both backends route their ``paths=`` / ``module=`` keywords through here, so the lenient - resolution (:func:`resolve_module_key` — an absolute path or one with native separators finds - its module) and the strictness (:func:`check_selector` — a path naming no module raises) cannot - drift apart between them. - - Args: - paths: What the caller named, or ``None`` for the unscoped call. - keys: The symbol-table keys that exist — ``symbol_table.keys()`` locally, the - application's module ``file_key``s over Neo4j. - kind: The keyword's name for the error message; ``"module"`` for ``get_classes``, whose - single-valued keyword routes through here as a one-element sequence. - - **Resolution is many-to-one, and the result is de-duplicated.** Leniency is the whole point of - :func:`resolve_module_key` — ``"pkg/a.py"`` and ``"/abs/pkg/a.py"`` are two spellings a scanner - may plausibly hand over for the *same* module — so two requested paths legitimately collapse to - one key and the caller gets one entry back. Raising on the collapse would punish the very - caller the leniency exists for; de-duplicating explicitly is what keeps the returned list from - naming the same module twice and asking both backends to fetch it twice. - - Raises: - TypeError: ``paths`` is a bare string (see :func:`reject_bare_string`). - ValueError: ``paths`` is an empty sequence. - SelectorNotInGraph: a path names no module in this application. - """ - reject_bare_string(kind, paths) - if paths is None: - return None - known = list(keys) - resolved = [resolve_module_key(p, known) for p in paths] - check_selector(kind, list(paths), [p for p, r in zip(paths, resolved) if r not in known]) - return list(dict.fromkeys(resolved)) - - -def call_graph_scope(roots: Sequence[str] | None, depth: int | None) -> List[str] | None: - """Normalise :meth:`PythonAnalysisBackend.get_call_graph`'s scoping keywords. - - Returns the roots as a list, or ``None`` for "the whole application" — the unscoped call, - which must keep behaving exactly as it did before the keywords existed. - - Both backends route through this so the two cannot drift apart on what a keyword combination - means (the failure mode Fix 1 of leg 1.5 had to go back and repair on the child-fetch paths). - Whether each root *exists* is checked later, by whichever backend has the graph in hand, but - through the same :func:`check_selector` — see :func:`bounded_subgraph`. - - Raises: - TypeError: ``roots`` is a bare string (see :func:`reject_bare_string`). - ValueError: ``depth`` that is not a positive ``int``, ``depth`` without ``roots``, or an - empty ``roots``. A hop budget with no origin to count from has no meaning, and quietly - returning all 364,752 edges would be the worst of the available answers — the caller - asked for a bounded graph and would be handed an unbounded one with no signal. - ``depth`` is type-checked rather than merely range-checked because the two ways of - getting it wrong are silent otherwise: ``depth="2"`` raised ``TypeError`` from the - comparison, and ``depth=2.5`` was accepted and truncated to 2 by the Cypher/ego-graph - radius. ``bool`` is rejected for the same reason — ``depth=True`` is ``1`` by accident. - """ - check_depth(depth) - reject_bare_string("roots", roots) - if roots is None: - if depth is not None: - raise ValueError("depth= requires roots=; a hop budget needs an origin to count from") - return None - check_selector("roots", list(roots), ()) - return list(roots) - - -def check_depth(depth: int | None) -> int | None: - """``depth`` is a hop budget: ``None`` for unbounded, otherwise an ``int`` of at least 1. - - Type-checked and not merely range-checked, because the two ways of getting it wrong are silent - otherwise: ``depth="2"`` raised ``TypeError`` from somewhere further in, and ``depth=2.5`` was - accepted and truncated to 2 by the Cypher/ego-graph radius. ``bool`` is rejected for the same - reason — ``depth=True`` is ``1`` by accident. - - One function, so ``get_call_graph``, the slices and the reachability accessors cannot come to - disagree about what a hop budget is. - """ - if depth is not None and (not isinstance(depth, int) or isinstance(depth, bool) or depth < 1): - raise ValueError(f"depth must be an int >= 1, got {depth!r}") - return depth - - -def bounded_subgraph(graph: nx.DiGraph, roots: List[str], depth: int | None, declared: Iterable[str]) -> nx.DiGraph: - """The sub-call-graph reachable from ``roots``, within ``depth`` hops when given. - - **Induced**, not path-only: every edge between two reached nodes is kept, including one - pointing back towards a root. A path-only answer would let ``graph.predecessors(n)`` lie about - a node the caller can see, which is a worse defect than the extra edges are a cost. The Neo4j - backend's Cypher is written to produce the same induced shape rather than the cheaper - edges-along-the-path shape, for exactly this reason. - - **The domain a root is judged against — stated here because both backends must judge against - the same one — is the callable inventory, not this graph.** ``graph`` is built from call - *edges* alone, so a callable that neither calls nor is called by anything is not a node in it: - 444 of the live odoo application's 15,549 in-scope callables, 2.9%. Checking membership of - ``graph`` therefore raised for a callable that plainly exists, while the Neo4j backend — whose - Cypher matches a root by node *label*, not by edge participation — returned the one-node graph - it is. ``declared`` closes that gap: it carries every callable the application declares, and a - root is valid when it is **in the inventory or is a node of the graph**. The second disjunct is - not redundant — an ``@external`` ghost is a legitimate root, is a graph node, and is not a - declared callable — and the union is exactly what the Neo4j root match accepts (a - ``:PyCallable`` of this application, or a ``:PyExternal``). - - A root outside that domain raises (:func:`check_selector`) rather than contributing nothing: - "no such callable" and "a callable that calls nothing" are different answers, and before this - they were the same empty graph. - - The returned graph stays **edge-induced**. An isolated root is added back as a lone node — - which is the answer, and the one Neo4j gives — but nothing else the inventory knows about is - seeded into it. Seeding all declared callables would make the unbounded local graph disagree - with Neo4j's node-for-node, trading one parity defect for a larger one. - """ - inventory = set(declared) - check_selector("roots", roots, [r for r in roots if r not in graph and r not in inventory]) - nodes: set = set() - isolated: set = set() - for root in roots: - if root not in graph: - isolated.add(root) # declared, but in no call edge: its own one-node graph - elif depth is None: - nodes |= nx.descendants(graph, root) | {root} - else: - nodes |= set(nx.ego_graph(graph, root, radius=depth).nodes) - sub = graph.subgraph(nodes).copy() - sub.add_nodes_from(isolated) - return sub - - -# ---------------------------------------------------------------------------------------------- -# Paging the per-callable graphs (E5). -# -# THE CANONICAL ORDER, defined once here because it is the only thing that makes a page mean the -# same thing on both backends. Neo4j returns rows in no order unless told to, and the local -# backend returns the analyzer's emission order; without one stated sort, page two on Neo4j is a -# different set of edges from page two locally. Each backend uses these functions -- the local one -# sorts and slices with them directly, the Neo4j one writes the same components into its ORDER BY -# and rebuilds the cursor from them -- so a change here moves both at once. -# -# The order is over the edge's OWN fields, in the order a reader would name them: source, then -# target, then whatever else the edge carries. Nothing positional and nothing backend-specific -# (no relationship element id, no row number), because a key one backend cannot compute is not a -# shared order. -# -# TOTALITY. A keyset cursor resumes strictly *after* a key, so a repeated key would drop its twin. -# The full field tuple is unique on real data: measured across odoo-slim-19's 5,134,655 PY_DDG, -# 247,906 PY_CFG_NEXT and 139,065 PY_CDG edges, zero (src, dst, ...) tuples repeat -- and for CFG -# the endpoints alone are *not* enough (13,310 node pairs carry two edges of different ``kind``), -# which is why ``kind`` is in the key. On the graph side the emitter MERGEs these relationships on -# exactly these properties, so uniqueness is structural there rather than incidental. Two edges -# equal in every field would be equal as values -- the models carry nothing else -- so their -# relative order is unobservable, and the page boundary is the same either way. -# -# ``or ""`` / ``or []`` is not cosmetic: ``DdgEdge.var`` is ``Optional[str]``, and a ``None`` in a -# sort key raises in Python and silently drops the row in Cypher (``null > x`` is null). The -# Cypher spells the same normalisation with ``coalesce``. - - -#: Edges per page when the caller does not say. 10,000 is where the measured distribution -#: splits: on odoo-slim-19, 15,520 of the 15,549 callables have fewer than 10,000 DDG edges, so -#: this default answers 99.8% of callables completely in one page and no caller of a normal -#: callable ever writes a loop -- while the 29 that are larger, up to 1,386,918 edges, are held to -#: a response a caller can actually hold. CFG and CDG max out at 402 and 314 edges on the same -#: application, so for them it is never reached. -DEFAULT_PAGE_SIZE = 10_000 +_CFG_KEY, _CDG_KEY, _DDG_KEY = edge_sort_key("cfg"), edge_sort_key("cdg"), edge_sort_key("ddg") def cfg_sort_key(edge: CfgEdge) -> Tuple: """The canonical order for :meth:`PythonAnalysisBackend.get_cfg`: source, target, kind.""" - return (edge.src, edge.dst, edge.kind or "") + return _CFG_KEY(edge) def cdg_sort_key(edge: CdgEdge) -> Tuple: """The canonical order for :meth:`PythonAnalysisBackend.get_cdg`: source, target. A control dependence carries nothing else to break a tie on, and needs nothing else: the pair is unique. """ - return (edge.src, edge.dst) + return _CDG_KEY(edge) def ddg_sort_key(edge: DdgEdge) -> Tuple: @@ -349,24 +130,7 @@ def ddg_sort_key(edge: DdgEdge) -> Tuple: Python and Cypher order lists the same way (element-wise, shorter first on a prefix), verified on the live graph rather than assumed: see ``test_neo4j_orders_a_page_exactly_as_python_would``. """ - return (edge.src, edge.dst, edge.var or "", list(edge.prov or [])) - - -class EdgeOrder(NamedTuple): - """One edge kind's canonical order, in both spellings that have to agree. - - The Python sort key and the Cypher expressions are the same components said twice, in two - languages, and the whole point of the order is that the two never disagree — so they are - written down once, together, and each backend takes the half it can run. ``len(exprs)`` is - also the order's arity, which is how a cursor from one accessor is refused by another - (:func:`decode_cursor`): the three arities are 3, 2 and 4. - - ``coalesce`` in the expressions is ``or ""`` / ``or []`` in the key: ``DdgEdge.var`` is - optional, and a ``None`` in a sort key raises in Python and silently drops the row in Cypher. - """ - - key: Callable[[object], Tuple] - exprs: Tuple[str, ...] + return _DDG_KEY(edge) #: The three orders. ``src``/``dst``/``kind``/``var``/``prov`` are the aliases the backends' @@ -375,335 +139,11 @@ class EdgeOrder(NamedTuple): CDG_ORDER = EdgeOrder(cdg_sort_key, ("src", "dst")) DDG_ORDER = EdgeOrder(ddg_sort_key, ("src", "dst", "coalesce(var,'')", "coalesce(prov,[])")) - -def encode_cursor(scope: str, key: Tuple) -> str: - """An opaque, round-trippable spelling of a sort key, stamped with the callable it came from. - - Opaque on purpose: the caller passes it back and never reads it, so the components of the - order stay an implementation detail rather than joining the caller's vocabulary. Base64 of - JSON, because the key holds strings and a list of strings, and both survive that unchanged. - - ``scope`` is the resolved callable signature, carried so that :func:`decode_cursor` can refuse - a cursor minted for a different callable. Without it, an agent looping over callables and - reusing the wrong ``next_cursor`` would get a plausible page of the *right* callable's edges - resumed from a position in the *wrong* one — silently, since body-node ids sort by callable id - and the filter would simply skip everything or nothing. - """ - return base64.urlsafe_b64encode(json.dumps([scope, list(key)]).encode("utf-8")).decode("ascii") - - -def decode_cursor(cursor: str, scope: str, arity: int) -> Tuple: - """Inverse of :func:`encode_cursor`, checked against the caller it is being used for. - - Three ways a cursor can be wrong, all of them raising rather than being read as "start from - the beginning" — which would silently hand back page one when page nine was asked for: - it does not decode; it was minted for another callable; or it has the wrong number of - components, which is what a cursor from a *different accessor* looks like (the three orders - have arities 3, 2 and 4, so no cursor is silently valid for the wrong graph). - """ - try: - got_scope, key = json.loads(base64.urlsafe_b64decode(cursor.encode("ascii")).decode("utf-8")) - except Exception as exc: # noqa: BLE001 -- any decode failure is the same caller error - raise ValueError(f"not a cursor from a previous page: {cursor!r}") from exc - if got_scope != scope: - raise ValueError(f"this cursor is from a page of {got_scope!r}, not {scope!r}") - if len(key) != arity: - raise ValueError(f"cursor has {len(key)} components, this accessor's order has {arity}: {cursor!r}") - return tuple(key) - - -def check_page_size(page_size: int) -> int: - """``page_size`` must ask for at least one edge. - - Zero is refused rather than treated as "no limit": a page of nothing whose ``next_cursor`` can - never advance is an infinite loop dressed as an empty answer. - """ - if page_size < 1: - raise ValueError(f"page_size must be at least 1, got {page_size}") - return page_size - - -def keyset_where(exprs: Sequence[str]) -> str: - """The Cypher for "strictly after the cursor", written out because Cypher has no tuple - comparison: ``(a, b) > ($c0, $c1)`` has to become - ``a > $c0 OR (a = $c0 AND (b > $c1))``. - - Keyset rather than ``SKIP``: measured on ``Website.configurator_apply`` (1,386,918 DDG edges, - 10,000 per page, query alone), ``SKIP`` costs 2.6s for the first page, 9.0s for the middle one - and 4.3s for the last -- it re-sorts a prefix that grows with the offset -- while this filter - is flat at 3.1s / 2.9s / 2.4s. The offset form is not wrong, it just gets worse the further in - the caller reads, which is the one direction pagination exists to make cheap. - """ - clause = "" - for i in reversed(range(len(exprs))): - expr, param = exprs[i], f"$c{i}" - clause = f"{expr} > {param}" + (f" OR ({expr} = {param} AND ({clause}))" if clause else "") - return clause - - -def cursor_params(cursor: str, scope: str, arity: int) -> Dict[str, object]: - """The ``$c0…$cN`` bindings :func:`keyset_where` reads, from an opaque cursor.""" - return {f"c{i}": v for i, v in enumerate(decode_cursor(cursor, scope, arity))} - - -def edge_page(model, scope: str, edges: List, order: EdgeOrder, page_size: int, cursor: str | None) -> EdgePage: - """One page of an edge set already held in memory. - - The local backend has every edge in hand, so it sorts by ``key`` and slices. The cursor is - resolved by binary search over the sorted keys -- ``bisect_right``, i.e. the first edge - strictly after it -- so it means exactly what :func:`keyset_where` makes it mean on the graph, - rather than an independently-invented position that happens to line up. - """ - check_page_size(page_size) - key = order.key - rows = sorted(edges, key=key) - start = bisect_right([key(e) for e in rows], decode_cursor(cursor, scope, len(order.exprs))) if cursor is not None else 0 - window = rows[start : start + page_size] - more = start + len(window) < len(rows) - return EdgePage[model](edges=window, total=len(rows), next_cursor=encode_cursor(scope, key(window[-1])) if more and window else None) - - -# ---------------------------------------------------------------------------------------------- -# Slicing and reachability (E2, E3, E5). -# -# THE FIVE RELATIONSHIP TYPES A SLICE FOLLOWS, verified against codeanalyzer's own -# ``neo4j/schema.py`` REL_TYPES and against ``CALL db.relationshipTypes()`` on odoo-slim-19 rather -# than copied from a plan -- the names in this leg's plan have been wrong before (PY_CFG_NEXT is -# not PY_CFG). All five exist, with these edge counts on that application: -# -# PY_DDG 5,134,655 data dependence, within a callable (var, prov) -# PY_CDG 139,065 control dependence, within a callable -# PY_PARAM_IN 229,035 actual_in -> formal_in : an argument entering a callee -# PY_PARAM_OUT 133,267 formal_out -> actual_out : a value coming back to the caller -# PY_SUMMARY 453,398 actual_in -> actual_out : a callee's pass-through, at the call site -# -# All five point WITH the flow -- verified on the live graph, where every PY_PARAM_IN runs -# actual_in -> formal_in and every PY_PARAM_OUT runs formal_out -> actual_out, with no exceptions -# in 362,302 edges. So a forward slice follows them and a backward slice follows them reversed; -# there is no per-type direction table to keep straight, which is why they can share one match. -# -# PY_CFG_NEXT is deliberately NOT here. Control *flow* says what runs next; a slice is about what -# a value or a decision depends on, and following successor edges would pull in every later -# statement whether or not it depends on anything -- the "returns the whole callable" bug that a -# non-emptiness assertion cannot catch. -SDG_RELS = ("PY_DDG", "PY_CDG", "PY_PARAM_IN", "PY_PARAM_OUT", "PY_SUMMARY") - -#: The Cypher spelling of :data:`SDG_RELS` for a relationship-type disjunction. -SDG_REL_PATTERN = "|".join(SDG_RELS) - -#: The caller's word for each relationship a path hop can be justified by (E6). The graph's own -#: ``PY_DDG``/``PY_PARAM_IN`` spelling never leaves the backend; both backends translate through -#: this one table so a hop cannot be labelled ``data`` over Neo4j and ``ddg`` locally. -#: -#: ``argument`` and ``return`` are the two interprocedural edges, and they are deliberately not -#: both called "parameter": ``PY_PARAM_IN`` binds a caller's argument to a callee's formal, and -#: ``PY_PARAM_OUT`` binds a callee's result back into the caller. A reader following a path needs -#: to know which way it just crossed a call boundary. -VIA = { - "PY_DDG": "data", - "PY_CDG": "control", - "PY_PARAM_IN": "argument", - "PY_PARAM_OUT": "return", - "PY_SUMMARY": "summary", - "PY_CALLS": "call", -} - -#: Paths per query when the caller does not say. A path list is a set of *witnesses* for a flow, -#: not the flow's extent, and ten worked examples is already more than a reader will follow; the -#: extent question is ``slice_forward``, which reports a ``total``. -DEFAULT_MAX_PATHS = 10 - - -def check_max_paths(max_paths: int) -> int: - """``max_paths`` must admit at least one path. Zero is refused for :func:`check_max_nodes`'s - reason: an empty list whose ``truncated`` says "there were more" answers nothing, and it is - indistinguishable at a glance from "there is no flow".""" - if max_paths < 1: - raise ValueError(f"max_paths must be at least 1, got {max_paths}") - return max_paths - - -def check_distinct_endpoints(src: SliceNode, dst: SliceNode) -> None: - """A path query must have two different endpoints. - - Neo4j's shortest-path search *refuses* a self-question outright ("the shortest path algorithm - does not work when the start and end nodes are the same"), which would otherwise surface as a - raw driver error from one backend and an empty list from the other. Both raise here instead, - and neither answers ``[]``: for a node that genuinely sits on a cycle, ``[]`` would be - indistinguishable from a proved absence of one, which is the ambiguous empty in another - costume. ``reaches(x, x)`` is the accessor that answers the existence question, and it does - terminate (measured: 0.03s, where the obvious ``EXISTS`` spelling never finished). - - Takes the *resolved* endpoints rather than their refs so the message speaks the caller's - vocabulary (E6/E7): a value is named ``'kwargs' within '….configurator_apply'``, a callable - by its signature, and the advice is a call that actually runs -- ``reaches`` takes callable - names, so for a value the cycle question is asked of its enclosing callable. - """ - if src.ref != dst.ref: - return - if src.kind == "callable": - raise ValueError(f"paths from {src.callable!r} to itself are not answered; ask reaches({src.callable!r}, {src.callable!r}) whether a cycle exists") - raise ValueError( - f"paths from {src.name!r} to itself (within {src.callable!r}) are not answered; a value reaches itself only through " - f"recursion, so ask reaches({src.callable!r}, {src.callable!r}) whether the callable is on a call cycle" - ) - - -def hop_sort_key(hops: Sequence[PathHop]) -> Tuple: - """The order two paths are compared in, in the caller's *own* vocabulary. - - E2 makes a path a sequence, which only means something if the *list* of paths is stable too: - ``max_paths`` truncates, and a truncation of a non-deterministic order is not reproducible. - So paths are ordered shortest first, then hop by hop on ``(via, var, to.ref)`` — every term of - which the caller can see in the result it gets back. - - Two hops that are indistinguishable in that vocabulary (parallel edges of the same kind, on - the same variable, between the same two nodes) are left to a backend-local tie-break: the - Neo4j backend appends the relationship's ``elementId``, the local backend keeps the order the - analyzer emitted them in. Either is stable for repeated calls against one graph; neither is - meaningful to a caller, which is why it is last and why nothing above depends on it. - """ - return (len(hops), tuple((h.via, h.var or "", h.to.ref) for h in hops)) - - -def flow_path(nodes: Sequence[SliceNode], edges: Sequence[Tuple[str, "str | None", "Sequence[str] | None"]]) -> FlowPath: - """Join a walk's ``n`` nodes and its ``n - 1`` edges into a :class:`FlowPath`. - - Both backends build paths through here, which is what makes the joining invariant - (``hops[i].to is hops[i + 1].frm``) a property of the construction rather than something each - backend has to be trusted to preserve. ``edges`` are the graph's own relationship types; they - are translated to the caller's word through :data:`VIA` exactly once, here. - - Raises: - KeyError: A relationship type with no word in :data:`VIA` — a new edge kind from a future - analyzer generation, which must be named before it can be reported rather than passed - through in the graph's spelling. - """ - return FlowPath(hops=[PathHop(frm=nodes[i], to=nodes[i + 1], via=VIA[rel], var=var, prov=list(prov or [])) for i, (rel, var, prov) in enumerate(edges)]) - - -def as_slice_node(node: object) -> SliceNode: - """The :class:`~cldk.analysis.commons.results.SliceNode` for anything carrying an address. - - :meth:`PythonAnalysisBackend.describe` takes "anything with a ``ref``" — slice nodes, the - endpoints of a :class:`~cldk.analysis.commons.results.PathHop`, a - :class:`~cldk.analysis.commons.results.LocateResult` — because the addressing layer hands a - caller three shapes and asking them to convert between shapes to hydrate one is the kind of - friction that gets worked around with string surgery. - - A ``SliceNode`` passes through untouched. A ``LocateResult`` is re-expressed as one, keeping - the vocabulary it already speaks: ``module.path`` is the file, ``callable.signature`` the - enclosing callable, ``node.kind`` the position's kind. - - Raises: - TypeError: ``node`` carries neither a ``ref`` nor a ``node_id``, so there is nothing to - look up. Guessing an address from a file and a line is what ``locate`` is for. - """ - if isinstance(node, SliceNode): - return node - ref = getattr(node, "node_id", None) - if ref is None: - raise TypeError(f"describe() needs something carrying a ref (a SliceNode, a path hop endpoint, a locate() result); got {type(node).__name__}") - module, callable_ref, body = node.module, node.callable, getattr(node, "node", None) - return SliceNode( - file=module.path, - line=node.span.start[0], - callable=callable_ref.signature if callable_ref else "", - kind=body.kind if body else "callable", - name=callable_ref.name if callable_ref else None, - source=node.source or None, - ref=ref, - ) - -#: Nodes per slice when the caller does not say. The same 10,000 as :data:`DEFAULT_PAGE_SIZE`, and -#: for a different reason: there, it is where 99.8% of callables fit in one page; here, nothing -#: fits, because the measured distribution has no middle (see -#: :class:`~cldk.analysis.commons.results.Slice`). 10,000 is the largest result that stays -#: readable, and every slice above it is one a caller should be re-asking with ``depth=``. -DEFAULT_MAX_NODES = 10_000 - -#: Hops from the seed when the caller does not say. **Finite, and that is the whole point.** -#: -#: The measured distribution has no middle (see :class:`~cldk.analysis.commons.results.Slice`), so -#: an unbounded default hands a connected seed 10,000 arbitrary nodes of a 195,819-node closure -- -#: an unprincipled 5%, honestly flagged ``truncated`` and useless either way. A finite default -#: answers a *narrower* question *completely* instead, and ``depth=None`` is how a caller asks for -#: the whole cone. -#: -#: 5 is the largest bound at which no measured slice needs ``max_nodes`` at all. Over 120 random -#: ``formal_in`` seeds with callers on odoo-slim-19, node counts by depth: -#: -#: ========= ===== ======= ======= ======= ======== -#: direction depth median p75 max > 10,000 -#: ========= ===== ======= ======= ======= ======== -#: backward 3 14 70 846 0 -#: backward 5 33 188 1,539 0 -#: backward 6 56 324 2,818 0 -#: backward 8 464 2,044 16,028 1 -#: backward None 195,786 195,787 198,306 79 -#: forward 3 12 34 440 0 -#: forward 5 24 63 1,053 0 -#: forward 6 35 166 14,260 1 -#: forward 8 48 402 37,326 2 -#: forward None 71 440,269 440,645 52 -#: ========= ===== ======= ======= ======= ======== -#: -#: 3 is informative but thin; 6 is where a forward slice first exceeds the cap and the default -#: would start truncating again. 5 is the last depth that never does, in either direction. -#: -#: **Which accessors take it, and which deliberately do not.** The three *slices* -#: (``slice_backward``, ``slice_forward``, ``backward_cone``) default to it: a bounded slice is a -#: *complete* answer to a narrower question, and ``total`` says so. The two *predicates* -#: (``reaches``, ``flows_to_call``, ``flows_to_argument``) and the two *path* queries -#: (``paths_between``, ``call_paths_between``) default to ``None`` -- unbounded -- because a hop -#: budget on a boolean or a path list is not a smaller answer but a **wrong** one: "no flow" and -#: "no flow within five hops" collapse into the same ``False`` / ``[]`` with nothing in the result -#: to tell them apart. Measured on odoo-slim-19: ``flows_to_call("kwargs", "Website.create", -#: within="Website.configurator_apply")`` is ``False`` at five hops and ``True`` unbounded, and -#: the matching ``paths_between`` is ``[]`` at five hops and ten paths at eight. ``depth=`` stays -#: on all five as an explicit narrowing a caller can name; it is only the *default* that differs. -DEFAULT_DEPTH = 5 - - -def check_max_nodes(max_nodes: int) -> int: - """``max_nodes`` must admit at least one node — the seed, if nothing else. - - Zero is refused rather than read as "no limit": a slice of nothing whose ``total`` says - 195,784 is a result no caller can act on, and "unbounded" is what ``max_nodes=None`` would - have to mean if it ever meant anything. - """ - if max_nodes < 1: - raise ValueError(f"max_nodes must be at least 1, got {max_nodes}") - return max_nodes - - -def cone_sinks(resolve: Callable[[str], SliceNode], sinks: Sequence[str]) -> List[SliceNode]: - """Resolve ``backward_cone``'s sinks, refusing the two ways of naming nothing. - - The same discipline :func:`check_selector` applies to ``roots=`` and ``paths=``: a bare string - is ten one-character sinks and is refused as a type error, and an empty sequence is refused - because "everything" is the argument omitted, not the argument emptied — and there is no - "everything" here to fall back to. Each surviving name goes through ``resolve``, so an - ambiguous sink raises listing candidates instead of one of them being picked. - - Duplicates are collapsed by resolved signature, not by the string the caller wrote: naming the - same callable twice, once bare and once qualified, is one sink. - """ - reject_bare_string("sinks", sinks) - if not sinks: - raise ValueError("sinks= names nothing to walk back from; pass at least one callable") - resolved = {node.callable: node for node in (resolve(s) for s in sinks)} - return list(resolved.values()) - - -def slice_resolved(roots: List[SliceNode]) -> str: - """The audit line on a :class:`~cldk.analysis.commons.results.Slice`: what the caller's names - matched, in the caller's vocabulary. - - Both backends build it here rather than each formatting its own, so a caller comparing two - results is comparing answers and not two spellings of one. - """ - return ", ".join(f"{r.callable} {r.kind} {r.name!r}" if r.kind != "callable" else r.callable for r in roots) +#: The five relationship types a slice follows and the caller's word for each, in this language's +#: ``PY_`` spelling -- see :func:`~cldk.analysis.commons.graphs.sdg_rels` for what they are and why. +SDG_RELS = sdg_rels("PY") +SDG_REL_PATTERN = sdg_rel_pattern("PY") +VIA = via_table("PY") class PythonAnalysisBackend(AnalysisBackend[PyApplication, PyModule, PyClass, PyCallable, PyClassAttribute, str]): diff --git a/cldk/analysis/python/codeanalyzer/codeanalyzer.py b/cldk/analysis/python/codeanalyzer/codeanalyzer.py index c27f62cd..f64494f5 100644 --- a/cldk/analysis/python/codeanalyzer/codeanalyzer.py +++ b/cldk/analysis/python/codeanalyzer/codeanalyzer.py @@ -60,6 +60,7 @@ from codeanalyzer.schema import Analysis, model_dump_json from cldk.analysis import AnalysisLevel +from cldk.analysis.commons.levels import ANALYZER_LEVELS, LEVEL_NAMES, analyzer_level from cldk.analysis.commons.resolve import CallableCandidate, body_node_kind, resolve_callable_signature, resolve_value_name, resolve_within, value_candidate from cldk.analysis.commons.results import CallableRef, Diagnostic, EdgePage, EntrypointCoverage, FlowPaths, LocateResult, ModuleRef, Slice, SliceNode, TypeRef from cldk.utils.exceptions import CodeanalyzerUsageException @@ -119,39 +120,15 @@ #: table + Jedi call graph, 2 = + defuse-linker call graph, 3 = + intraprocedural dataflow #: (CFG/CDG/DDG), 4 = + interprocedural SDG (``formal_in``/``formal_out`` vertices, alias-aware #: DDG). The four SDK names line up with those four integers in order. -_ANALYZER_LEVELS = { - AnalysisLevel.symbol_table: 1, - AnalysisLevel.call_graph: 2, - AnalysisLevel.program_dependency_graph: 3, - AnalysisLevel.system_dependency_graph: 4, -} - -#: The inverse, by the member name a caller writes (``"call_graph"``, not ``"call graph"``) — so -#: an error about the level in use names it the way it was asked for. -_LEVEL_NAMES = {n: lvl.name for lvl, n in _ANALYZER_LEVELS.items()} - - -def analyzer_level(level: "AnalysisLevel | str") -> int: - """The analyzer's integer level for one of the SDK's :class:`~cldk.analysis.AnalysisLevel` - names. - - Accepts the enum, its value (``"call graph"``) and its member name (``"call_graph"``): the - facade's parameter is typed ``str``, and the underscore spelling is what a caller writing - ``analysis_level="system_dependency_graph"`` produces. An unrecognised name raises rather than - falling back to a default — a level that silently becomes 1 is the defect this function exists - to close. - - ``AnalysisOptions``'s other two dataflow knobs are left at their defaults on purpose: - ``graphs="cfg,dfg,pdg,sdg"`` already selects every section the SDK can surface (``sdg`` is - inert below level 4, where it only widens a ``want_pdg`` that ``pdg`` already sets), and - ``graph_field_depth=3`` is the analyzer's own access-path k-limit, which the SDK exposes no - parameter for. - """ - key = str(getattr(level, "value", level)).replace("_", " ") - try: - return _ANALYZER_LEVELS[AnalysisLevel(key)] - except ValueError: - raise ValueError(f"unknown analysis_level {level!r}; expected one of {[lvl.name for lvl in AnalysisLevel]}") from None +#: The analyzer's ``-a`` integer for each SDK level and its inverse — lifted to +#: :mod:`cldk.analysis.commons.levels` (TypeScript sends the same integers); re-bound here under +#: the names this module and its tests always used. ``AnalysisOptions``'s other two dataflow knobs +#: are left at their defaults on purpose: ``graphs="cfg,dfg,pdg,sdg"`` already selects every +#: section the SDK can surface (``sdg`` is inert below level 4, where it only widens a +#: ``want_pdg`` that ``pdg`` already sets), and ``graph_field_depth=3`` is the analyzer's own +#: access-path k-limit, which the SDK exposes no parameter for. +_ANALYZER_LEVELS = ANALYZER_LEVELS +_LEVEL_NAMES = LEVEL_NAMES def body_node_id(callable_id: str, body_key: str) -> str: @@ -1499,7 +1476,7 @@ def _value_paths(self, a: SliceNode, b: SliceNode, depth: int | None, max_paths: walks = self._shortest_walks(adjacency["forward"], a.ref, b.ref, depth, max_paths + 1) described = {ref: _local_slice_node(nodes[ref], ref) for walk in walks for ref, _ in walk if ref in nodes} described[a.ref] = a - paths = [flow_path([described[a.ref]] + [described[ref] for ref, _ in walk], [label for _, label in walk]) for walk in walks[:max_paths]] + paths = [flow_path([described[a.ref]] + [described[ref] for ref, _ in walk], [label for _, label in walk], via=VIA) for walk in walks[:max_paths]] return FlowPaths(paths=paths, complete=len(walks) <= max_paths) def paths_between(self, src: str, dst: str, *, src_within: str, dst_within: str, depth: int | None = None, max_paths: int = DEFAULT_MAX_PATHS) -> FlowPaths: @@ -1537,7 +1514,7 @@ def call_paths_between(self, src: str, dst: str, *, depth: int | None = None, ma externals = self.get_external_symbols() described = {sig: self._call_graph_node(sig, externals) for walk in walks for sig, _ in walk} described[a] = self._call_graph_node(a, externals) - paths = [flow_path([described[a]] + [described[sig] for sig, _ in walk], [label for _, label in walk]) for walk in walks[:max_paths]] + paths = [flow_path([described[a]] + [described[sig] for sig, _ in walk], [label for _, label in walk], via=VIA) for walk in walks[:max_paths]] return FlowPaths(paths=paths, complete=len(walks) <= max_paths) def _call_graph_node(self, signature: str, externals: Dict[str, PyExternalSymbol]) -> SliceNode: diff --git a/cldk/analysis/python/neo4j/neo4j_backend.py b/cldk/analysis/python/neo4j/neo4j_backend.py index 30097ea5..e868312b 100644 --- a/cldk/analysis/python/neo4j/neo4j_backend.py +++ b/cldk/analysis/python/neo4j/neo4j_backend.py @@ -81,7 +81,6 @@ from __future__ import annotations import logging -import re from collections import defaultdict from contextlib import contextmanager from functools import cached_property @@ -92,6 +91,8 @@ from codeanalyzer.schema.ids import application_id, module_id from codeanalyzer.schema.py_schema import PyEntrypointReport +from cldk.analysis.commons.backend import semver as _semver +from cldk.analysis.commons.keys import module_key_of from cldk.analysis.commons.resolve import CallableCandidate, body_node_kind, resolve_callable_signature, resolve_value_name, resolve_within, value_candidate from cldk.analysis.commons.results import CallableRef, Diagnostic, EdgePage, EntrypointCoverage, FlowPath, FlowPaths, LocateResult, ModuleRef, PathHop, Slice, SliceNode, TypeRef from cldk.analysis.python.backend import ( @@ -151,11 +152,6 @@ logger = logging.getLogger(__name__) -def _semver(raw: Any) -> Tuple[int, int, int] | None: - """``"1.4.1"`` (or ``"1.4.1.post0"``) as ``(1, 4, 1)``; ``None`` for anything that does not - start with three dotted integers, so an unparsable version is *unknown*, never silently zero.""" - m = re.match(r"(\d+)\.(\d+)\.(\d+)", raw) if isinstance(raw, str) else None - return (int(m[1]), int(m[2]), int(m[3])) if m else None # One statement per parent->child collection, each fetching that whole collection for the *entire* # application in a single round trip and returning the parent's key as ``pk``. These are the bulk @@ -485,7 +481,7 @@ def _scope_prefix(self) -> str: @cached_property def _module_set(self) -> FrozenSet[str]: - """:attr:`_modules` as a set -- the membership side of :func:`~cldk.analysis.python.neo4j.reconstruct.module_key_of`. + """:attr:`_modules` as a set -- the membership side of :func:`~cldk.analysis.commons.keys.module_key_of`. The list stays the Cypher parameter (the driver does not pack a set); this is the view every projected row's key is verified against, built once.""" return frozenset(self._modules) @@ -502,13 +498,13 @@ def _module_key(self, node_id: str) -> str: miss is a genuine defect and is raised as such, without the id (E6). """ try: - return R.module_key_of(node_id, self._scope_prefix, self._module_set) + return module_key_of(node_id, self._scope_prefix, self._module_set) except KeyError: pass self._modules = self._load_module_keys() self.__dict__.pop("_module_set", None) # drop the cached frozenset; rebuilt on next read try: - return R.module_key_of(node_id, self._scope_prefix, self._module_set) + return module_key_of(node_id, self._scope_prefix, self._module_set) except KeyError: raise CodeanalyzerExecutionException( f"A node of application {self.application_name!r} belongs to none of the {len(self._module_set)} module keys the graph " @@ -1612,7 +1608,7 @@ def _paths(self, query: str, node_of, a: SliceNode, b: SliceNode, *, src: str, d are the keys the query matches them by.""" check_distinct_endpoints(a, b) rows = self._run(query.format(rels=SDG_REL_PATTERN, depth="" if depth is None else depth), src=src, dst=dst, cap=max_paths + 1, prefix=self._scope_prefix) - paths = [flow_path([node_of(n, self._module_key) for n in r["ns"]], [(e["via"], e["var"], e["prov"]) for e in r["rs"]]) for r in rows[:max_paths]] + paths = [flow_path([node_of(n, self._module_key) for n in r["ns"]], [(e["via"], e["var"], e["prov"]) for e in r["rs"]], via=VIA) for r in rows[:max_paths]] return FlowPaths(paths=paths, complete=len(rows) <= max_paths) # Argument validation precedes name resolution on every accessor below, as it does on the diff --git a/cldk/analysis/python/neo4j/reconstruct.py b/cldk/analysis/python/neo4j/reconstruct.py index 20b895a9..165377b9 100644 --- a/cldk/analysis/python/neo4j/reconstruct.py +++ b/cldk/analysis/python/neo4j/reconstruct.py @@ -38,20 +38,24 @@ from __future__ import annotations import json -from typing import Any, Collection, Dict, List, Mapping +from typing import Any, Dict, List, Mapping +# ``module_key_of`` lives in ``commons.keys`` (leg 2.5a, G4); re-exported for the callers that have +# always addressed it through this module. +from cldk.analysis.commons.keys import module_key_of # noqa: F401 + +# The artifact-layer reconstructors (``artifact`` / ``config_key`` / ``dependency``) live in +# ``commons.artifacts`` (leg 2.5a): the layer is projected identically by every analyzer. +from cldk.analysis.commons.artifacts import artifact, config_key, dependency # noqa: F401 from cldk.models.python import ( BodyNode, - PyArtifact, PyCallable, PyCallableOverview, PyClass, PyClassAttribute, PyClassOverview, PyComment, - PyConfigKey, PyConfigRead, - PyDependency, PyExternalSymbol, PyImport, PyModule, @@ -68,26 +72,6 @@ Props = Mapping[str, Any] -# -----[ ids ]----- -def module_key_of(node_id: str, prefix: str, known: Collection[str]) -> str: - """The repo-relative module key embedded in a ``can://`` id (F4). - - Ids are ``/`` (or exactly ```` for a module), and a - file key can itself contain ``.py/`` as a directory name, so the key is never recovered by - splitting: every ``/``-boundary prefix of the id is tried longest first and the first that is - a member of ``known`` -- the application's verified module keys -- wins. A miss raises: a key - we cannot verify is a defect, not a guess. ``known`` should be a set; this runs once per row. - """ - if not node_id.startswith(prefix): - raise KeyError(node_id) - parts = node_id[len(prefix) :].split("/") - for n in range(len(parts), 0, -1): - candidate = "/".join(parts[:n]) - if candidate in known: - return candidate - raise KeyError(node_id) - - # -----[ helpers ]----- def comments(props: Props) -> List[PyComment]: """Rebuild the (lossy) comment list from the single ``docstring`` property.""" @@ -209,73 +193,6 @@ def body_node(props: Props) -> BodyNode: ) -def config_key(props: Props) -> PyConfigKey: - """Rebuild a :class:`PyConfigKey` from a ``:ConfigKey`` node's properties. - - Line-only ``span`` (see :func:`body_node`): the projection writes ``start_line``/``end_line`` - and nothing finer, so the columns and byte offsets rehydrate as ``0``. ``span`` stays ``None`` - when the node carries no lines at all (best-effort extraction never located the key in the - artifact's source). - """ - lines = (props.get("start_line"), props.get("end_line")) - return PyConfigKey( - id=props.get("id", ""), - key=props.get("key", ""), - namespace=props.get("namespace", ""), - value=props.get("value"), - span=Span(start=(lines[0], 0), end=(lines[1], 0), bytes=(0, 0)) if None not in lines else None, - references=list(props.get("references", []) or []), - ) - - -def artifact(props: Props, *, config_keys: List[PyConfigKey] | None = None) -> PyArtifact: - """Rebuild a :class:`PyArtifact` from an ``:Artifact`` node's properties plus its fetched - :class:`PyConfigKey` children (``[:DEFINES_CONFIG]``). - - ``kind`` is not a projected property — every ``PyArtifact`` the analyzer emits carries the - model's own default (``"artifact"``; see ``codeanalyzer/artifacts/discovery.py``), so it is - supplied here rather than queried for. - """ - return PyArtifact( - id=props.get("id", ""), - kind="artifact", - path=props.get("path", ""), - format=props.get("format", ""), - roles=list(props.get("roles", []) or []), - size_bytes=props.get("size_bytes", 0), - sha256=props.get("sha256", ""), - source=props.get("source", ""), - extraction=props.get("extraction", "none"), - config_keys=config_keys or [], - ) - - -def dependency(props: Props, *, name: str, ecosystem: str, declared_in: str) -> PyDependency: - """Rebuild a :class:`PyDependency` from a ``[:DECLARES_DEPENDENCY]`` edge's properties plus its - endpoints (``name``/``ecosystem`` off the ``:Package`` node, ``declared_in`` off the - ``:Artifact`` node). ``ecosystem`` is a real ``Package`` property (``neo4j/schema.py``'s - ``Package`` node type carries it); ``"pypi"`` is only ever what the analyzer happens to write - there today (its only ecosystem, per ``PyDependency.ecosystem``'s own docstring) — read off the - node rather than hardcoded, so this doesn't silently go stale the day a second ecosystem ships. - - ``locked_version``/``provides_imports`` are projection-lossy here: the graph carries them on - the separate ``[:LOCKS]``/``[:PY_PROVIDES]`` edges (per-package facts, not per-declaration), and - no caller of this reconstruction chases those yet, so they come back at the model's own empty - defaults — the same class of gap :func:`callsite` documents for ``argument_types``. - """ - return PyDependency( - name=name, - ecosystem=ecosystem, - spec=props.get("spec", ""), - kind=props.get("kind", "runtime"), - extras=list(props.get("extras", []) or []), - declared_in=declared_in, - direct=props.get("direct", True), - provides_imports=[], - prov=list(props.get("prov", []) or []), - ) - - def unresolved_config_read(props: Props, *, callee: str) -> PyConfigRead: """Rebuild a :class:`PyConfigRead` from a ``[:PY_READS_CONFIG_UNRESOLVED]`` edge's properties plus its ``:PyExternal`` ghost endpoint (``callee``). diff --git a/cldk/analysis/typescript/backend.py b/cldk/analysis/typescript/backend.py index 2ba00044..0df5d031 100644 --- a/cldk/analysis/typescript/backend.py +++ b/cldk/analysis/typescript/backend.py @@ -24,23 +24,29 @@ * :class:`~cldk.analysis.typescript.neo4j.TSNeo4jBackend` — answers the *same* queries with Cypher over the graph ``codeanalyzer-typescript`` emits with ``--emit neo4j``. -This ABC formalizes the surface those two share so the façade↔backend relationship is enforced by -the type system (and at instantiation time) instead of matching only by convention. Both backends -subclass it; the façade is typed against it. Backend-specific lifecycle (e.g. the Neo4j driver's -``close()`` / context-manager support) is intentionally *not* part of the contract. - -The vocabulary mirrors :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer` / -:class:`~cldk.analysis.python.codeanalyzer.PyCodeanalyzer`, but the node kinds are TypeScript-native -(interfaces, type aliases, enums, namespaces, decorators, ...). +The shape shared with every other language — application view, symbol table, call graph, the +class/method/field lookups and the repository-artifact layer — is inherited from the generic +:class:`~cldk.analysis.commons.backend.AnalysisBackend`; what is declared here is the +TypeScript-native remainder (interfaces, type aliases, enums, namespaces, decorators, the +1.x call-site accessors and the bulk projections). Both backends subclass it; the façade is typed +against it. Backend-specific lifecycle (e.g. the Neo4j driver's ``close()`` / context-manager +support) is intentionally *not* part of the contract. + +The call graph both backends return keeps TypeScript's own endpoints (decision TS-11): cants emits +a module as the caller of its top-level code and a class as the callee of ``new X()``, and both +are kept, tagged with a ``kind`` node attribute (:data:`CALL_GRAPH_NODE_KINDS`) so a +caller wanting Python's callable-only shape filters in one line rather than the SDK erasing every +top-level call. """ from __future__ import annotations -from abc import ABC, abstractmethod -from typing import Dict, List, Set, Tuple +from abc import abstractmethod +from typing import ClassVar, Dict, List, Set, Tuple import networkx as nx +from cldk.analysis.commons.backend import AnalysisBackend from cldk.models.typescript import ( TSApplication, TSCallable, @@ -53,45 +59,63 @@ TSEnumMember, TSExport, TSExternalSymbol, + TSField, TSImport, TSInterface, TSModule, TSSynthesizedCallable, + TSType, TSTypeAlias, TSVariableDeclaration, ) -class TSAnalysisBackend(ABC): +#: The ``kind`` vocabulary of a call-graph node: what the id index holds — a module, any of the +#: five type kinds (a class is the callee of ``new X()``; the others are indexed and would be kept +#: if the analyzer ever emitted an edge to one), a callable, or an external. +CALL_GRAPH_NODE_KINDS = frozenset({"module", "class", "interface", "enum", "type_alias", "namespace", "callable", "external"}) + + +class TSAnalysisBackend(AnalysisBackend[TSApplication, TSModule, TSType, TSCallable, TSField, str]): """Abstract base every TypeScript analysis backend implements. A backend owns *all* indexing and query logic for a TypeScript application; the :class:`TypeScriptAnalysis` façade is a one-line-delegation shim over it. Implementations must return the canonical ``cldk.models.typescript`` pydantic objects (or the documented NetworkX / dict / list shapes) so the two backends are behaviorally interchangeable. - """ - # -----[ application / whole-program ]----- - @abstractmethod - def get_application(self) -> TSApplication: - """The whole application view (symbol table + call graph + external symbols).""" + Inherited abstract (see :class:`~cldk.analysis.commons.backend.AnalysisBackend`): + ``get_application_view``, ``get_symbol_table``, ``get_call_graph``, ``get_all_classes``, + ``get_class``, ``get_all_methods_in_class``, ``get_method``, ``get_all_fields``, + ``get_method_parameters``, ``get_artifacts``, ``get_dependencies``, ``get_config_keys``, + ``get_config_uses``, ``get_unresolved_config_reads``. + """ - @abstractmethod - def get_symbol_table(self) -> Dict[str, TSModule]: - """The per-file symbol table, keyed by module file path.""" + P: ClassVar[str] = "TS" + N: ClassVar[str] = "TS" + # -----[ application / whole-program ]----- @abstractmethod def get_modules(self) -> List[TSModule]: """All modules (compilation units).""" @abstractmethod def get_external_symbols(self) -> Dict[str, TSExternalSymbol]: - """Phantom (external) call targets — imported/required library members.""" + """Phantom (external) call targets — imported/required library members and builtins — + keyed ``"."``, the key the call graph uses for them; the wire's ``can://`` + id is on the value.""" @abstractmethod def get_synthesized_callables(self) -> Dict[str, TSSynthesizedCallable]: - """Anonymous-callback endpoints the symbol table never names (Jelly-resolved). Keyed by the - synthesized signature that ``call_graph`` edges reference. Empty for the ``tsc`` resolver.""" + """The application's anonymous callables, each value carrying the ``can://`` tree id of + the callable it stands for. Empty below level 2. + + **The key is backend-dependent**, and each backend's own docstring says which it uses: a + backend reading ``analysis.json`` passes the analyzer's compatibility index through as + emitted, so the key is the *older* anonymous id and the value's ``id`` is the tree id that + replaced it (key != ``id``); a backend reading the Neo4j projection has the tree nodes and + not the index, so it keys by the node's own id (key == ``id``). Do not key a cross-backend + lookup on this map -- ask for the value's ``id``.""" @abstractmethod def get_typescript_file(self, qualified_name: str) -> str | None: @@ -102,10 +126,6 @@ def get_typescript_module(self, file_path: str) -> TSModule | None: """The module for a file path.""" # -----[ call graph ]----- - @abstractmethod - def get_call_graph(self) -> nx.DiGraph: - """NetworkX DiGraph of callable signatures (and phantom external symbols) + call edges.""" - @abstractmethod def get_call_graph_json(self) -> str: """The application serialized as JSON.""" @@ -129,7 +149,8 @@ def get_class_hierarchy(self) -> nx.DiGraph: # -----[ call sites ]----- @abstractmethod def get_call_sites(self, qualified_callable_name: str) -> List[TSCallsite]: - """The rich, syntactic call sites inside a callable.""" + """The syntactic call sites inside a callable — its ``body`` nodes of ``kind == "call"``, + with the resolved callee mapped to its signature.""" @abstractmethod def get_calling_lines(self, target_signature: str) -> List[int]: @@ -139,15 +160,7 @@ def get_calling_lines(self, target_signature: str) -> List[int]: def get_call_targets(self, source_signature: str) -> Set[str]: """The call targets invoked from a callable, derived from its call sites.""" - # -----[ classes / interfaces / enums / type-aliases ]----- - @abstractmethod - def get_all_classes(self) -> Dict[str, TSClass]: - """Every class, keyed by signature.""" - - @abstractmethod - def get_class(self, qualified_class_name: str) -> TSClass | None: - """A single class by signature.""" - + # -----[ interfaces / enums / type-aliases ]----- @abstractmethod def get_all_interfaces(self) -> Dict[str, TSInterface]: """Every interface, keyed by signature.""" @@ -166,7 +179,9 @@ def get_all_type_aliases(self) -> Dict[str, TSTypeAlias]: @abstractmethod def get_all_nested_classes(self, qualified_class_name: str) -> List[TSClass]: - """The classes declared inside a class.""" + """The classes declared inside a class -- on schema v2 always ``[]``, on every backend: a + class holds only ``callables`` and ``fields``, so no class nests a type. A class declared + inside a *callable* survives as ``TSCallable.inner_classes``. Kept for the 1.x surface.""" @abstractmethod def get_all_sub_classes(self, qualified_class_name: str) -> Dict[str, TSClass]: @@ -185,21 +200,6 @@ def get_implemented_interfaces(self, qualified_class_name: str) -> List[str]: def get_all_methods_in_application(self) -> Dict[str, Dict[str, TSCallable]]: """All methods grouped by their owning class/interface signature.""" - @abstractmethod - def get_all_methods_in_class(self, qualified_class_name: str) -> Dict[str, TSCallable]: - """The methods of a class/interface, keyed by short name.""" - - @abstractmethod - def get_method(self, qualified_class_name: str, qualified_method_name: str) -> TSCallable | None: - """A single method of a class/interface, or a module/namespace-level function. - ``qualified_class_name`` accepts either a class/interface signature (resolving to that - type's methods) or a module/namespace scope, in which case module-level functions are - resolved as a fallback; returns ``None`` if nothing resolves.""" - - @abstractmethod - def get_method_parameters(self, qualified_class_name: str, qualified_method_name: str) -> List[str]: - """The parameter names of a method.""" - @abstractmethod def get_all_constructors(self, qualified_class_name: str) -> Dict[str, TSCallable]: """The constructors of a class.""" @@ -208,10 +208,6 @@ def get_all_constructors(self, qualified_class_name: str) -> Dict[str, TSCallabl def get_all_functions(self) -> Dict[str, TSCallable]: """Top-level (module/namespace) functions, keyed by signature.""" - @abstractmethod - def get_all_fields(self, qualified_class_name: str) -> List[TSClassAttribute]: - """The attributes/fields of a class.""" - @abstractmethod def get_interface_properties(self, qualified_interface_name: str) -> List[TSClassAttribute]: """The properties of an interface.""" @@ -263,9 +259,9 @@ def get_callables_overview(self) -> List[TSCallableOverview]: @abstractmethod def get_method_bodies(self, signatures: List[str]) -> Dict[str, str]: """Source bodies for the given callable signatures, keyed by signature. Signatures with no - matching callable are omitted, as are callables whose ``code`` is ``None`` (e.g. implicit - constructors the analyzer synthesizes with no source text) — every returned value is a - real ``str``.""" + matching callable are omitted, as are callables with no source text (an implicit + constructor the analyzer synthesizes has an empty span, so its ``code`` is ``""``; 1.x + carried ``None``) — every returned value is a real, non-empty ``str``.""" @abstractmethod def get_decorated_callables(self, markers: List[str]) -> List[TSCallableOverview]: diff --git a/cldk/analysis/typescript/codeanalyzer/codeanalyzer.py b/cldk/analysis/typescript/codeanalyzer/codeanalyzer.py index d173ec91..f31bc8dd 100644 --- a/cldk/analysis/typescript/codeanalyzer/codeanalyzer.py +++ b/cldk/analysis/typescript/codeanalyzer/codeanalyzer.py @@ -16,10 +16,10 @@ """TypeScript Codeanalyzer backend wrapper. -Subprocess wrapper around the ``codeanalyzer-typescript`` binary (built from ``codeanalyzer-ts`` -with ``bun build --compile``). Mirrors the Java ``JCodeanalyzer`` / Python ``PyCodeanalyzer`` -pattern: shell out to the analyzer, read ``analysis.json`` from stdout (or an output dir), -validate it into a ``TSApplication`` pydantic model, **and own all query/indexing logic**. The +Subprocess wrapper around the ``codeanalyzer-typescript`` binary (``cants``). Mirrors the Java +``JCodeanalyzer`` / Python ``PyCodeanalyzer`` pattern: shell out to the analyzer, read the +``analysis.json`` envelope (:class:`TSAnalysis`) from stdout or an output dir, keep its +``application`` as the queried :class:`TSApplication`, **and own all query/indexing logic**. The ``TypeScriptAnalysis`` facade is a thin delegating shell over this backend. """ @@ -30,22 +30,26 @@ import os import shlex import subprocess -from collections import deque +import warnings from pathlib import Path from subprocess import CompletedProcess from typing import Dict, Iterator, List, Set, Tuple, Union import networkx as nx -from cldk.analysis import AnalysisLevel +from cldk.analysis.commons.levels import analyzer_level from cldk.analysis.typescript.backend import TSAnalysisBackend +from cldk.models.python import PyArtifact, PyConfigKey, PyConfigRead, PyConfigUseEdge, PyDependency from cldk.models.typescript import ( + TSAnalysis, TSApplication, + TSBodyNode, TSCallable, TSCallableOverview, TSCallsite, TSClass, TSClassAttribute, + TSConfigKey, TSDecorator, TSEnum, TSEnumMember, @@ -63,29 +67,36 @@ logger = logging.getLogger(__name__) +#: The codeanalyzer-typescript release that removed ``--tsc-only`` (the resolver is no longer a +#: choice; 1.x's ``tsc`` and ``defuse`` provenances are both emitted and tagged per edge). +_TSC_ONLY_REMOVED_IN = "1.0.0" + class TSCodeanalyzer(TSAnalysisBackend): """Build and query the application view of a TypeScript project by invoking the codeanalyzer-typescript binary as a subprocess. This backend owns all indexing and query logic (symbol lookups, the NetworkX call graph, - class hierarchy, call sites, entrypoints, decorators, ...). The :class:`TypeScriptAnalysis` - facade simply delegates to it, mirroring how :class:`PythonAnalysis` delegates to - :class:`PyCodeanalyzer`. + class hierarchy, call sites, decorators, the artifact layer, ...). The + :class:`TypeScriptAnalysis` facade simply delegates to it, mirroring how + :class:`PythonAnalysis` delegates to :class:`PyCodeanalyzer`. Args: project_dir: Path to the root of the TypeScript project. - analysis_backend_path: Directory containing the ``codeanalyzer-typescript`` binary. If - None, falls back to ``$CODEANALYZER_TS_BIN`` then the ``codeanalyzer-typescript`` - PyPI package (``pip install codeanalyzer-typescript``). analysis_json_path: Directory to persist ``analysis.json``. If None, output is read from the subprocess stdout pipe. - analysis_level: ``AnalysisLevel.symbol_table`` (1) or ``AnalysisLevel.call_graph`` (2). - eager_analysis: If True, re-run the analyzer even if a cached ``analysis.json`` exists. + analysis_level: Any :class:`~cldk.analysis.AnalysisLevel` (or its name); sent to the + analyzer as ``-a 1..4`` — the backend requests what the caller asked for. + eager_analysis: If True, re-run the analyzer even if a cached ``analysis.json`` exists, and + tell the analyzer to rebuild its own cache (``--eager``). target_files: Restrict analysis to these files (incremental). - tsc_only: If True, restrict the analyzer to the tsc resolver call graph by passing - ``--tsc-only`` (codeanalyzer-typescript >= 0.4.2). Defaults to False (let the binary - choose its default). Replaces reliance on the obsolete ``--call-graph-provider both``. + tsc_only: Deprecated no-op. The flag was removed from codeanalyzer-typescript at 1.0.0; + passing ``True`` emits a :class:`DeprecationWarning` and changes nothing. + + Attributes: + analysis: The whole ``analysis.json`` envelope — ``max_level``, ``k_limit``, + ``analyzer.version`` — for callers that need to know what generation produced the view. + application: ``analysis.application``, the queried view. """ def __init__( @@ -102,10 +113,14 @@ def __init__( self.analysis_level = analysis_level self.eager_analysis = eager_analysis self.target_files = target_files - self.tsc_only = tsc_only - self.application: TSApplication = self._init_codeanalyzer( - analysis_level=1 if analysis_level == AnalysisLevel.symbol_table else 2 - ) + if tsc_only: + warnings.warn( + f"tsc_only is a no-op: codeanalyzer-typescript removed --tsc-only in {_TSC_ONLY_REMOVED_IN}; " "every call edge now carries its resolver in `prov` instead.", + DeprecationWarning, + stacklevel=4, # warn at the CLDK.typescript(...) call, through the facade + ) + self.analysis: TSAnalysis = self._init_codeanalyzer(analysis_level=analyzer_level(analysis_level)) + self.application: TSApplication = self.analysis.application self._call_graph: nx.DiGraph | None = None self._index() @@ -126,53 +141,46 @@ def _get_codeanalyzer_exec(self) -> List[str]: import codeanalyzer_typescript return [str(codeanalyzer_typescript.bin_path())] - except (ModuleNotFoundError, FileNotFoundError): - pass - - raise CodeanalyzerExecutionException( - "codeanalyzer-typescript binary not found. Install it with `pip install codeanalyzer-typescript`, " - "or set $CODEANALYZER_TS_BIN." - ) - - @staticmethod - def _init_tsapplication(data: str) -> TSApplication: - """Build a TSApplication from a stringified analysis.json.""" - return TSApplication(**json.loads(data)) - - def _init_codeanalyzer(self, analysis_level: int = 1) -> TSApplication: - """Run the analyzer and return the validated TSApplication.""" - codeanalyzer_exec = self._get_codeanalyzer_exec() - target_args: List[str] = [] - if self.target_files: - for tf in self.target_files: - target_args += ["-t", str(tf).strip()] - # Restrict the call graph to the tsc resolver path when requested, replacing the obsolete - # `--call-graph-provider both`. The `--tsc-only` flag lands in codeanalyzer-typescript - # 0.4.2; older binaries reject it, so only opt in when running >= 0.4.2. - if self.tsc_only: - target_args += ["--tsc-only"] - + except (ModuleNotFoundError, FileNotFoundError) as e: + raise CodeanalyzerExecutionException( + "codeanalyzer-typescript binary not found: $CODEANALYZER_TS_BIN is unset and the " + f"`codeanalyzer-typescript` wheel is not importable or carries no binary for this platform ({e}). " + "Install it with `pip install codeanalyzer-typescript`, or set $CODEANALYZER_TS_BIN." + ) from e + + def _argv(self, analysis_level: int, output_dir: Path | None) -> List[str]: + """The 1.2.0 command line: ``-i --app-name -a <1..4> [-o + --cache-dir ] --skip-tests [--eager] [-t ]...``. The application name is what + the analyzer stamps into every ``can://typescript//...`` id.""" + project = Path(self.project_dir) + args = self._get_codeanalyzer_exec() + ["-i", str(project), "--app-name", project.name, "-a", str(analysis_level)] + if output_dir is not None: + args += ["-o", str(output_dir), "--cache-dir", str(output_dir)] + args += ["--skip-tests"] + if self.eager_analysis: + args += ["--eager"] + for tf in self.target_files or []: + args += ["-t", str(tf).strip()] + return args + + def _init_codeanalyzer(self, analysis_level: int) -> TSAnalysis: + """Run the analyzer and return the validated envelope.""" if self.analysis_json_path is None: # Read compact JSON from the stdout pipe. - args = codeanalyzer_exec + ["-i", str(Path(self.project_dir)), "-a", str(analysis_level)] + target_args + args = self._argv(analysis_level, None) try: logger.info(f"Running codeanalyzer-typescript: {' '.join(args)}") - console_out: CompletedProcess[str] = subprocess.run( - args, capture_output=True, text=True, check=True - ) - return self._init_tsapplication(console_out.stdout) + console_out: CompletedProcess[str] = subprocess.run(args, capture_output=True, text=True, check=True) + return TSAnalysis.model_validate_json(console_out.stdout) except Exception as e: # noqa: BLE001 raise CodeanalyzerExecutionException(str(e)) from e # Persist to an output directory and read analysis.json back. - analysis_json_file = Path(self.analysis_json_path).joinpath("analysis.json") + output_dir = Path(self.analysis_json_path) + analysis_json_file = output_dir / "analysis.json" needs_run = self.eager_analysis or not analysis_json_file.exists() or bool(self.target_files) if needs_run: - args = ( - codeanalyzer_exec - + ["-i", str(Path(self.project_dir)), "-a", str(analysis_level), "-o", str(self.analysis_json_path)] - + target_args - ) + args = self._argv(analysis_level, output_dir) try: logger.info(f"Running codeanalyzer-typescript: {' '.join(args)}") subprocess.run(args, capture_output=True, text=True, check=True) @@ -180,12 +188,12 @@ def _init_codeanalyzer(self, analysis_level: int = 1) -> TSApplication: raise CodeanalyzerExecutionException("codeanalyzer-typescript did not generate analysis.json.") except Exception as e: # noqa: BLE001 raise CodeanalyzerExecutionException(str(e)) from e - with open(analysis_json_file, encoding="utf-8") as f: - return self._init_tsapplication(json.dumps(json.load(f))) + return TSAnalysis.model_validate_json(analysis_json_file.read_text(encoding="utf-8")) # -----[ indexing ]----- def _index(self) -> None: - """Flatten the (recursive) symbol table into signature-keyed lookups, built once.""" + """Flatten the (recursive) symbol table into signature-keyed lookups, built once, and the + id → (graph key, kind) index that joins ``can://`` edge endpoints to those keys (TS-10).""" self._classes: Dict[str, TSClass] = {} self._interfaces: Dict[str, TSInterface] = {} self._enums: Dict[str, TSEnum] = {} @@ -194,8 +202,12 @@ def _index(self) -> None: self._functions: Dict[str, TSCallable] = {} self._methods_by_class: Dict[str, Dict[str, TSCallable]] = {} self._file_of: Dict[str, str] = {} + #: ``can://`` id → (call-graph node key, kind). Modules key on the file key, classes and + #: callables on ``signature``, externals on ``"."``. + self._id_index: Dict[str, Tuple[str, str]] = {} for fp, mod in self.application.symbol_table.items(): + self._id_index[mod.id] = (fp, "module") for f in mod.functions.values(): self._add_callable(f, fp) self._functions[f.signature] = f @@ -205,16 +217,41 @@ def _index(self) -> None: self._add_interface(it, fp) for en in mod.enums.values(): self._enums[en.signature] = en - self._file_of[en.signature] = fp + self._add_type(en, fp) for ta in mod.type_aliases.values(): self._type_aliases[ta.signature] = ta - self._file_of[ta.signature] = fp + self._add_type(ta, fp) for ns in mod.namespaces.values(): self._add_namespace(ns, fp) + for key, ext in (self.application.external_symbols or {}).items(): + node = (f"{ext.module}.{ext.name}", "external") + self._id_index[key] = node + self._id_index[ext.id] = node + # The compatibility index: keyed by the older anonymous id, the value's id is the tree id + # that replaced it (already indexed above); a residual fallback node (key == id, no tree + # home) is keyed by its own name. Anything else is an analyzer defect, never a node keyed + # by a raw id. + for key, syn in (self.application.synthesized_callables or {}).items(): + if syn.id in self._id_index: + node = self._id_index[syn.id] + elif syn.id == key and syn.name: + node = (syn.name, "callable") + else: + raise CodeanalyzerExecutionException( + f"synthesized callable {key!r} resolves to {syn.id!r}, which is neither a callable of application " + f"{self.application.id!r} nor a named residual node: codeanalyzer-typescript " + f"{self.analysis.analyzer.version} emitted an unhomed endpoint" + ) + self._id_index.setdefault(key, node) + + def _add_type(self, t, fp: str) -> None: + self._file_of[t.signature] = fp + self._id_index[t.id] = (t.signature, t.kind) def _add_callable(self, c: TSCallable, fp: str) -> None: self._callables[c.signature] = c self._file_of[c.signature] = fp + self._id_index[c.id] = (c.signature, "callable") for ic in c.inner_callables.values(): self._add_callable(ic, fp) for cl in c.inner_classes.values(): @@ -222,18 +259,16 @@ def _add_callable(self, c: TSCallable, fp: str) -> None: def _add_class(self, cl: TSClass, fp: str) -> None: self._classes[cl.signature] = cl - self._file_of[cl.signature] = fp + self._add_type(cl, fp) methods: Dict[str, TSCallable] = {} for m in cl.methods.values(): self._add_callable(m, fp) methods[m.name] = m self._methods_by_class[cl.signature] = methods - for ic in cl.inner_classes.values(): - self._add_class(ic, fp) def _add_interface(self, it: TSInterface, fp: str) -> None: self._interfaces[it.signature] = it - self._file_of[it.signature] = fp + self._add_type(it, fp) methods: Dict[str, TSCallable] = {} for m in it.methods.values(): self._add_callable(m, fp) @@ -241,6 +276,7 @@ def _add_interface(self, it: TSInterface, fp: str) -> None: self._methods_by_class[it.signature] = methods def _add_namespace(self, ns: TSNamespace, fp: str) -> None: + self._add_type(ns, fp) for f in ns.functions.values(): self._add_callable(f, fp) self._functions[f.signature] = f @@ -250,13 +286,56 @@ def _add_namespace(self, ns: TSNamespace, fp: str) -> None: self._add_interface(it, fp) for en in ns.enums.values(): self._enums[en.signature] = en - self._file_of[en.signature] = fp + self._add_type(en, fp) for ta in ns.type_aliases.values(): self._type_aliases[ta.signature] = ta - self._file_of[ta.signature] = fp + self._add_type(ta, fp) for n in ns.namespaces.values(): self._add_namespace(n, fp) + def _node_of(self, node_id: str) -> Tuple[str, str]: + """The (graph key, kind) an endpoint id resolves to. Every endpoint the analyzer emits is + homed on the tree, the externals or the synthesized index; one that is not is the + analyzer's defect, surfaced rather than skipped or keyed by a raw id.""" + try: + return self._id_index[node_id] + except KeyError: + raise CodeanalyzerExecutionException( + f"call-graph endpoint {node_id!r} is not a module, type, callable, external or synthesized callable " + f"of application {self.application.id!r}: codeanalyzer-typescript {self.analysis.analyzer.version} " + "emitted an unhomed endpoint" + ) from None + + def _callee_signature(self, node: TSBodyNode) -> str | None: + """The graph key a call node's resolved ``callee`` id maps to; ``None`` when the analyzer + left it unresolved (``null``). A ``callee`` that is neither is the same unhomed-endpoint + defect :meth:`_node_of` raises for — a raw id never reaches a return field.""" + if node.callee is None: + return None + return self._node_of(node.callee)[0] + + def _callsite(self, key: str, node: TSBodyNode) -> TSCallsite: + """The 1.x per-call record, read off a ``kind == "call"`` body node.""" + span = node.span + return TSCallsite( + method_name=node.method_name or "", + receiver_expr=node.receiver_expr, + receiver_type=node.receiver_type, + argument_types=list(node.argument_types), + type_arguments=list(node.type_arguments), + return_type=node.return_type, + callee_signature=self._callee_signature(node), + is_constructor_call=node.is_constructor_call, + is_optional_chain=node.is_optional_chain, + start_line=span.start[0] if span else -1, + start_column=span.start[1] if span else -1, + end_line=span.end[0] if span else -1, + end_column=span.end[1] if span else -1, + ) + + def _call_nodes(self, c: TSCallable) -> Iterator[Tuple[str, TSBodyNode]]: + return ((k, n) for k, n in c.body.items() if n.kind == "call") + def _resolve_callable(self, class_or_module: str, method: str | None = None) -> TSCallable | None: """Resolve a callable from either a full signature (``method is None``) or a ``(class/module, member)`` pair. Mirrors :meth:`PyCodeanalyzer.get_method` resolution.""" @@ -285,7 +364,7 @@ def _resolve_signature(self, class_or_sig: str, member: str | None = None) -> st return callable_.signature if callable_ else f"{class_or_sig}.{member}" # -----[ application / whole-program ]----- - def get_application(self) -> TSApplication: + def get_application_view(self) -> TSApplication: return self.application def get_symbol_table(self) -> Dict[str, TSModule]: @@ -295,12 +374,10 @@ def get_modules(self) -> List[TSModule]: return list(self.application.symbol_table.values()) def get_external_symbols(self) -> Dict[str, TSExternalSymbol]: - return self.application.external_symbols + return {f"{ext.module}.{ext.name}": ext for ext in (self.application.external_symbols or {}).values()} def get_synthesized_callables(self) -> Dict[str, TSSynthesizedCallable]: - """Anonymous-callback endpoints Jelly resolves that the symbol table never names. Keyed by - the synthesized signature an edge's ``source``/``target`` references.""" - return self.application.synthesized_callables + return dict(self.application.synthesized_callables or {}) def get_typescript_file(self, qualified_name: str) -> str | None: return self._file_of.get(qualified_name) @@ -310,29 +387,19 @@ def get_typescript_module(self, file_path: str) -> TSModule | None: # -----[ call graph ]----- def get_call_graph(self) -> nx.DiGraph: - """Build (and cache) a NetworkX DiGraph whose nodes are callable signatures (plus phantom - external symbols and synthesized anonymous callbacks) and whose edges are the identity-only - call edges.""" + """Build (and cache) the call graph: nodes keyed as every other accessor keys them (module + file key, type/callable signature, ``"."`` for an external) with ``id`` and + ``kind`` attributes; edges carry ``type="CALL_DEP"``, ``weight`` and ``provenance`` as the + Python backend's do. Module callers and class callees are kept (TS-11).""" if self._call_graph is not None: return self._call_graph graph = nx.DiGraph() - for sig, callable_ in self._callables.items(): - graph.add_node(sig, callable=callable_, external=False) - # Phantom (external) nodes so that import-attributed edges don't dangle. - for sig, ext in self.application.external_symbols.items(): - graph.add_node(sig, external=True, module=ext.module, name=ext.name) - # Synthesized anonymous-callback nodes so Jelly's anonymous edges don't dangle. - for sig, syn in self.application.synthesized_callables.items(): - graph.add_node(sig, external=False, synthesized=True, name=syn.name, path=syn.path) for edge in self.application.call_graph: - graph.add_edge( - edge.source, - edge.target, - type=edge.type, - weight=edge.weight, - provenance=edge.provenance, - tags=edge.tags, - ) + src, src_kind = self._node_of(edge.src) + dst, dst_kind = self._node_of(edge.dst) + graph.add_node(src, id=edge.src, kind=src_kind) + graph.add_node(dst, id=edge.dst, kind=dst_kind) + graph.add_edge(src, dst, type="CALL_DEP", weight=edge.weight, provenance=tuple(edge.prov)) self._call_graph = graph return graph @@ -348,10 +415,7 @@ def get_all_callers(self, target_class_name: str, target_method_declaration: str target = self._resolve_signature(target_class_name, target_method_declaration) if target not in graph: return {"target_method": target, "caller_details": []} - callers = [ - {"caller_signature": src, "edge": graph.get_edge_data(src, target)} - for src in graph.predecessors(target) - ] + callers = [{"caller_signature": src, "edge": graph.get_edge_data(src, target)} for src in graph.predecessors(target)] return {"target_method": target, "caller_details": callers} def get_all_callees(self, source_class_name: str, source_method_declaration: str | None = None) -> Dict: @@ -361,34 +425,18 @@ def get_all_callees(self, source_class_name: str, source_method_declaration: str source = self._resolve_signature(source_class_name, source_method_declaration) if source not in graph: return {"source_method": source, "callee_details": []} - callees = [ - {"callee_signature": tgt, "edge": graph.get_edge_data(source, tgt)} - for tgt in graph.successors(source) - ] + callees = [{"callee_signature": tgt, "edge": graph.get_edge_data(source, tgt)} for tgt in graph.successors(source)] return {"source_method": source, "callee_details": callees} - def get_class_call_graph( - self, qualified_class_name: str, method_signature: str | None = None - ) -> List[Tuple[str, str]]: - """Call-graph edges reachable from a class (or one of its methods).""" - adjacency: Dict[str, List[str]] = {} - for e in self.application.call_graph: - adjacency.setdefault(e.source, []).append(e.target) + def get_class_call_graph(self, qualified_class_name: str, method_signature: str | None = None) -> List[Tuple[str, str]]: + """Call-graph edges reachable from a class (or one of its methods), in BFS order.""" + graph = self.get_call_graph() if method_signature is not None: seeds = [method_signature] else: seeds = [m.signature for m in self._methods_by_class.get(qualified_class_name, {}).values()] - edges: List[Tuple[str, str]] = [] - seen = set(seeds) - queue = deque(seeds) - while queue: - src = queue.popleft() - for dst in adjacency.get(src, []): - edges.append((src, dst)) - if dst not in seen: - seen.add(dst) - queue.append(dst) - return edges + seeds = [s for s in seeds if s in graph] + return list(nx.edge_bfs(graph, seeds)) if seeds else [] def get_class_hierarchy(self) -> nx.DiGraph: """Inheritance/implementation graph: an edge child → base for every base_class.""" @@ -405,28 +453,29 @@ def get_class_hierarchy(self) -> nx.DiGraph: # -----[ call sites ]----- def get_call_sites(self, qualified_callable_name: str) -> List[TSCallsite]: - """The rich, syntactic call sites *inside* a callable (receiver/argument types, resolved - ``callee_signature``, position). Distinct from the resolved call-graph edges.""" + """The syntactic call sites *inside* a callable (receiver/argument types, resolved + ``callee_signature``, position) — its ``body`` nodes of ``kind == "call"``. Distinct from + the resolved call-graph edges.""" callable_ = self._callables.get(qualified_callable_name) - return list(callable_.call_sites) if callable_ else [] + return [self._callsite(k, n) for k, n in self._call_nodes(callable_)] if callable_ else [] def get_calling_lines(self, target_signature: str) -> List[int]: """Sorted, de-duplicated source lines anywhere in the project where ``target_signature`` - is invoked (matched against each call site's resolved ``callee_signature``).""" + is invoked (matched against each call node's resolved callee).""" lines: Set[int] = set() for callable_ in self._callables.values(): - for cs in callable_.call_sites: - if cs.callee_signature == target_signature and cs.start_line >= 0: - lines.add(cs.start_line) + for _, n in self._call_nodes(callable_): + if n.span is not None and self._callee_signature(n) == target_signature: + lines.add(n.span.start[0]) return sorted(lines) def get_call_targets(self, source_signature: str) -> Set[str]: - """The set of call targets invoked from a callable, taken from its call sites. Resolved - ``callee_signature`` when available, otherwise the bare ``method_name``.""" + """The set of call targets invoked from a callable, taken from its call nodes. Resolved + callee signature when available, otherwise the bare ``method_name``.""" callable_ = self._callables.get(source_signature) if callable_ is None: return set() - return {cs.callee_signature or cs.method_name for cs in callable_.call_sites} + return {self._callee_signature(n) or n.method_name or "" for _, n in self._call_nodes(callable_)} # -----[ classes / interfaces / enums / type-aliases ]----- def get_all_classes(self) -> Dict[str, TSClass]: @@ -449,8 +498,9 @@ def get_all_type_aliases(self) -> Dict[str, TSTypeAlias]: return self._type_aliases def get_all_nested_classes(self, qualified_class_name: str) -> List[TSClass]: - cls = self._classes.get(qualified_class_name) - return list(cls.inner_classes.values()) if cls else [] + # The v2 class facet nests no types (only namespaces and callables do), so a class never + # has nested classes on this wire; kept for the 1.x surface. + return [] def get_all_sub_classes(self, qualified_class_name: str) -> Dict[str, TSClass]: return {sig: cls for sig, cls in self._classes.items() if qualified_class_name in cls.base_classes} @@ -499,11 +549,7 @@ def get_method_parameters(self, qualified_class_name: str, qualified_method_name return [p.name for p in method.parameters] if method else [] def get_all_constructors(self, qualified_class_name: str) -> Dict[str, TSCallable]: - return { - name: m - for name, m in self._methods_by_class.get(qualified_class_name, {}).items() - if m.kind == "constructor" - } + return {name: m for name, m in self._methods_by_class.get(qualified_class_name, {}).items() if m.kind == "constructor"} def get_all_functions(self) -> Dict[str, TSCallable]: return self._functions @@ -527,6 +573,56 @@ def get_all_variables(self) -> Dict[str, List[TSVariableDeclaration]]: """Module-level variable declarations per file.""" return {fp: list(m.variables) for fp, m in self.application.symbol_table.items()} + # -----[ repository artifacts — the shared Py* models, as the generic ABC promises ]----- + @staticmethod + def _py_config_key(ck: TSConfigKey) -> PyConfigKey: + """``TSConfigKey.value`` may be a JSON number or boolean (``"strict": true``); + ``PyConfigKey.value`` is a string, so a non-string value is rendered as its JSON text + (``true``, ``1``, ``1.5``), which is what the artifact itself says.""" + value = ck.value if isinstance(ck.value, str) or ck.value is None else json.dumps(ck.value) + return PyConfigKey(id=ck.id, key=ck.key, namespace=ck.namespace, value=value, span=ck.span.model_dump() if ck.span else None, references=list(ck.references)) + + def get_artifacts(self) -> Dict[str, PyArtifact]: + """Every non-code artifact (see :meth:`AnalysisBackend.get_artifacts`), keyed by + repo-relative path as the wire keys them; every ``TSArtifact`` field has a home on + :class:`PyArtifact`.""" + return { + path: PyArtifact(**a.model_dump(exclude={"config_keys"}), config_keys=[self._py_config_key(ck) for ck in a.config_keys]) + for path, a in self.application.artifacts.items() + } + + def get_dependencies(self, *, direct_only: bool = False, ecosystem: str | None = None, declared_in: str | None = None) -> List[PyDependency]: + """Every declared dependency, optionally filtered (see + :meth:`AnalysisBackend.get_dependencies`). The TypeScript wire carries no ``ecosystem`` + field — every dependency is an npm package (``pkg:npm/``), so that is what the + shared model's field says and what the ``ecosystem`` filter matches.""" + deps = [PyDependency(ecosystem="npm", **d.model_dump()) for d in self.application.dependencies] + if direct_only: + deps = [d for d in deps if d.direct] + if ecosystem is not None: + deps = [d for d in deps if d.ecosystem == ecosystem] + if declared_in is not None: + deps = [d for d in deps if d.declared_in == declared_in] + return deps + + def get_config_keys(self) -> Dict[str, PyConfigKey]: + """Every configuration key, flattened out of the artifact that defines it and keyed by id + (see :meth:`AnalysisBackend.get_config_keys`).""" + return {ck.id: self._py_config_key(ck) for a in self.application.artifacts.values() for ck in a.config_keys} + + def get_config_uses(self, key: str | None = None) -> List[PyConfigUseEdge]: + """Resolved code-to-config edges (see :meth:`AnalysisBackend.get_config_uses`).""" + edges = [PyConfigUseEdge(**u.model_dump()) for u in self.application.config_uses] + if key is None: + return edges + matching_ids = {ck.id for ck in self.get_config_keys().values() if ck.key == key} + return [e for e in edges if e.dst in matching_ids] + + def get_unresolved_config_reads(self) -> List[PyConfigRead]: + """Detector-matched config reads that resolved to no declared key (see + :meth:`AnalysisBackend.get_unresolved_config_reads`) — ``TSApplication.config_reads``.""" + return [PyConfigRead(**r.model_dump()) for r in self.application.config_reads] + # -----[ decorators ]----- def get_decorators(self, qualified_callable_name: str) -> List[TSDecorator]: callable_ = self._callables.get(qualified_callable_name) @@ -580,15 +676,15 @@ def _iter_callables(self) -> Iterator[Tuple[TSCallable, str | None, str | None]] def get_callables_overview(self) -> List[TSCallableOverview]: """Return a lightweight overview of every callable in the application (see :meth:`TSAnalysisBackend.get_callables_overview`).""" - return [TSCallableOverview.from_callable(c, owner_sig, owner_kind) for c, owner_sig, owner_kind in self._iter_callables()] + return [TSCallableOverview.from_callable(c, owner_sig, owner_kind, path=self._file_of[c.signature]) for c, owner_sig, owner_kind in self._iter_callables()] def get_method_bodies(self, signatures: List[str]) -> Dict[str, str]: - """Return ``{signature: code}`` for the requested signatures that exist and have a body - (omits callables whose ``code`` is ``None``, e.g. implicit constructors).""" + """Return ``{signature: code}`` for the requested signatures that exist and have source + text (omits an implicit constructor, whose empty span slices to ``""``).""" result: Dict[str, str] = {} for sig in signatures: c = self._callables.get(sig) - if c is not None and c.code is not None: + if c is not None and c.code: result[sig] = c.code return result @@ -596,7 +692,7 @@ def get_decorated_callables(self, markers: List[str]) -> List[TSCallableOverview """Return overviews of callables decorated with any of ``markers``.""" marker_set = set(markers) return [ - TSCallableOverview.from_callable(c, owner_sig, owner_kind) + TSCallableOverview.from_callable(c, owner_sig, owner_kind, path=self._file_of[c.signature]) for c, owner_sig, owner_kind in self._iter_callables() if marker_set.intersection(d.name for d in c.decorators) ] @@ -607,5 +703,5 @@ def get_callsites_for(self, signatures: List[str]) -> Dict[str, List[TSCallsite] for sig in signatures: c = self._callables.get(sig) if c is not None: - result[sig] = list(c.call_sites) + result[sig] = [self._callsite(k, n) for k, n in self._call_nodes(c)] return result diff --git a/cldk/analysis/typescript/neo4j/neo4j_backend.py b/cldk/analysis/typescript/neo4j/neo4j_backend.py index 02891c7d..c1c87125 100644 --- a/cldk/analysis/typescript/neo4j/neo4j_backend.py +++ b/cldk/analysis/typescript/neo4j/neo4j_backend.py @@ -14,61 +14,88 @@ # limitations under the License. ################################################################################ -"""Neo4j-backed TypeScript analysis backend (read-only Cypher client). - -A drop-in alternative to :class:`TSCodeanalyzer`: it exposes the **same query -method surface** (``get_all_classes``, ``get_call_graph``, ``get_all_callers``, -...) so the :class:`TypeScriptAnalysis` facade can delegate to either one, but -every method answers by running **Cypher over a live Neo4j graph** instead of -walking the in-memory pydantic/NetworkX structures. - -This class is purely a **query client**: it never builds the graph and has no -dependency on the ``codeanalyzer-typescript`` binary or the project sources. It -assumes the database is already populated and just polls it — the shape a cloud -deployment wants, where a third-party job (e.g. inside Kubernetes) loads the -graph out of band and the SDK only reads it. - -The graph is the one ``codeanalyzer-typescript`` emits with ``--emit neo4j`` -(schema: ``codeanalyzer-ts/schema.neo4j.json``). Populating it always happens out -of band — never from this backend. - -Identity model (must match the in-memory backend): - -* a callable/class/interface/enum/type-alias is a ``:Symbol`` keyed by ``signature``; -* call-graph edges are ``(:Symbol)-[:CALLS]->(:Symbol|:External)``; -* every project-owned node carries a ``_module`` provenance property, so a single - database may hold several applications — all queries here are scoped to this - backend's application by the set of its module ``file_key``s. - -Parity caveats (inherent to what the projection stores, not bugs): - -* ``CALLS`` edge ``tags`` only round-trip the three keys the projection keeps - (``ts.dispatch`` / ``ts.external`` / ``ts.module``); -* ``get_imports`` / ``get_all_exports`` are reconstructed from the *aggregated* - ``IMPORTS`` / ``RE_EXPORTS`` edges (individual bindings, aliases and positions - are not stored); -* comments collapse to a single docstring, type-parameters keep only their names. +"""Neo4j-backed TypeScript analysis backend (read-only Cypher client) on the codeanalyzer-typescript +1.2.0 graph vocabulary. + +A drop-in alternative to :class:`TSCodeanalyzer`: the same query surface, every method answered by +Cypher over a live graph that ``codeanalyzer-typescript --emit neo4j`` populated out of band. This +class never writes and needs neither the analyzer binary nor the sources. + +**The graph it reads** (``schema.neo4j.json`` at the 1.2.0 tag; ``main`` renames nothing): +``:Application {id: can://typescript/}`` anchors the application and stamps +``analyzer_version``; every project node carries a ``can://`` ``id`` under the merge label +``CanNode`` -- ``TSModule`` (``name`` holds the file key), ``TSClass``/``TSInterface``/``TSEnum``/ +``TSTypeAlias``/``TSNamespace``, ``TSCallable`` (all seven kinds; anonymous ones also +``TSAnonymousCallable``), ``TSField``, ``TSBodyNode``, ``TSExternal``; containment is +``TS_HAS_MODULE`` / ``TS_DECLARES`` / ``TS_HAS_METHOD`` / ``TS_HAS_FIELD``; calls are ``TS_CALLS +{weight, prov}`` and a call site is a ``TSBodyNode {kind:'call'}`` under ``TS_HAS_BODY_NODE`` +resolving over ``TS_RESOLVES_TO``. + +**Scope (TS-3).** A signature is not application-stamped, so every statement that could match +another application's node carries the two-prefix predicate :func:`_scoped` spells -- +``can://typescript//`` and ``can://javascript//`` -- or is keyed by an id that embeds the +application, or walks out from the ``:Application`` anchor. There is no ``_module`` property to +fall back on (retired on ``main``, #166). + +**Seek labels (measured on the superset graph).** Signature and prefix statements anchor on the +specific label alone (``:TSCallable``): 11,085 callables scan in ~8 ms, and ``:CanNode`` turns the +two-prefix predicate into a slower range-seek union. Id-equality point lookups anchor on +``:CanNode: