diff --git a/CHANGELOG.md b/CHANGELOG.md index f8370aca..e4d5e336 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -63,9 +63,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 where calls to a bare module name that is also imported (e.g. `os`/`re`/`json`) are dropped from the emitted call graph. `PythonAnalysis` / `CLDK.analysis(language="python")` accept the same optional `neo4j_config`. -- Bumped `codeanalyzer-python` to `0.2.0` (adds the Neo4j graph emitter). +- Read-only Neo4j-backed **Java** analysis backend (`cldk.analysis.java.neo4j.JNeo4jBackend`), + completing Neo4j parity across all three languages. It reconstructs the canonical `JApplication` + from the graph `codeanalyzer-java` (>= 2.4.0) emits with `--emit neo4j` and answers all 36 + `JavaAnalysisBackend` queries with the in-memory backend's logic. Verified against the daytrader8 + sample (145 classes): everything the graph actually contains reconstructs identically to + `JCodeanalyzer` (97% of checks). Three projection gaps in the `codeanalyzer-java` 2.4.0 emitter + (fields collapsing to one node, imports reduced to packages, a truncated call graph) are **fixed + in 2.4.1** (codeanalyzer-java#156/#157/#158, verified on daytrader — `J_CALLS` went 287 → 1702), + the version the SDK release now bundles. `JavaAnalysis` / `CLDK.java(...)` accept a + `Neo4jConnectionConfig` as the `backend=` config to select it. +- Bumped `codeanalyzer-python` to `0.2.0` (adds the Neo4j graph emitter); the bundled + `codeanalyzer-java` jar is now `2.4.1` (adds the Neo4j graph emitter + the field/import/call-graph + projection fixes). The Java analyzer jar is no longer a pip dependency — the SDK release workflow + downloads the latest `codeanalyzer-java` jar into the bundled `jar/` directory. - Optional `neo4j` extra (`pip install cldk[neo4j]`) for the Neo4j Python driver. +### Fixed +- **Bundled JDK download for the Java backend.** `ensure_jdk` resolved the Temurin JVM via the + Adoptium `/assets/version/{release}` endpoint, which now returns 404 for pinned releases (e.g. + `jdk-21.0.5+11`) — so the first Java analysis on a clean machine failed before it started. It now + resolves via the `/binary/version/...` endpoint (following the redirect to the GitHub asset) and + reads the checksum from the asset's `.sha256.txt`. + ## [v1.0.7] - 2026-02-14 ### Added diff --git a/README.md b/README.md index 6f952265..626ff3de 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ pip install cldk Optional extras: ```bash -pip install "cldk[neo4j]" # read-only Neo4j graph backend (Python / TypeScript) +pip install "cldk[neo4j]" # read-only Neo4j graph backend (Java / Python / TypeScript) ``` ## Quick Start @@ -117,11 +117,11 @@ classes = analysis.get_all_classes() ## Supported Languages & Backends -Each language is analyzed by a dedicated `codeanalyzer-*` engine; CLDK normalizes the result into typed models exposed through the same API. +Each language is analyzed by a dedicated `codeanalyzer-*` engine; CLDK normalizes the result into typed models exposed through the same API. All three also support an optional **read-only Neo4j backend** — pass a `Neo4jConnectionConfig` and the SDK answers the same queries with Cypher over a graph the analyzer populates out of band (`--emit neo4j`). | Language | Analysis engine | What it provides | | --- | --- | --- | -| **Java** | [`codeanalyzer-java`](https://github.com/codellm-devkit/codeanalyzer-java) | WALA + JavaParser. Bytecode-level call graphs, type hierarchies, symbol resolution, and method/field declarations. | +| **Java** | [`codeanalyzer-java`](https://github.com/codellm-devkit/codeanalyzer-java) | WALA + JavaParser. Bytecode-level call graphs, type hierarchies, symbol resolution, CRUD-operation and entry-point detection. Optional read-only **Neo4j** graph backend. | | **Python** | [`codeanalyzer-python`](https://github.com/codellm-devkit/codeanalyzer-python) | Jedi with optional CodeQL augmentation. Symbol tables, call graphs, and class/method resolution. Optional read-only **Neo4j** graph backend. | | **TypeScript / JavaScript** | [`codeanalyzer-typescript`](https://github.com/codellm-devkit/codeanalyzer-typescript) | ts-morph with Jelly-based call graphs. Symbols, call graph, types, decorators, and call sites. Optional read-only **Neo4j** graph backend. | @@ -147,13 +147,14 @@ graph TD P --> EP[codeanalyzer-python
Jedi · CodeQL] T --> ET[codeanalyzer-typescript
ts-morph · Jelly] - P -. read-only .-> N[(Neo4j)] + J -. read-only .-> N[(Neo4j)] + P -. read-only .-> N T -. read-only .-> N ``` **Data models** — each language has its own set of Pydantic models under `cldk.models` (`cldk.models.java`, `cldk.models.python`, `cldk.models.typescript`). They give you structured, typed, dot-accessible representations of classes, methods, fields, and statements, with JSON serialization and shared conventions across languages. -**Analysis backends** — each language has a backend under `cldk.analysis.` that coordinates its engine (see the table above) and maps the result onto the data models. Backends are orchestrated internally; you only call high-level methods such as `get_symbol_table()`, `get_method_body(...)`, and `get_call_graph(...)`, and CLDK handles tool coordination, parsing, and marshalling under the hood. +**Analysis backends** — each language has a backend under `cldk.analysis.` that coordinates its engine (see the table above) and maps the result onto the data models. The read-only Neo4j backends (`cldk.analysis..neo4j`) reconstruct the *same* models from a Cypher graph, so they are drop-in interchangeable with the in-process analyzers. Backends are orchestrated internally; you only call high-level methods such as `get_symbol_table()`, `get_method_body(...)`, and `get_call_graph(...)`, and CLDK handles tool coordination, parsing, and marshalling under the hood. ## Documentation diff --git a/cldk/analysis/commons/backend_config.py b/cldk/analysis/commons/backend_config.py index 3806aeaa..a84050ff 100644 --- a/cldk/analysis/commons/backend_config.py +++ b/cldk/analysis/commons/backend_config.py @@ -99,9 +99,8 @@ class Neo4jConnectionConfig: application_name: str | None = None -# Per-language discriminated unions the facades match on. Java has no Neo4j backend yet, so its -# only admissible config is the codeanalyzer one. -JavaBackend = CodeAnalyzerConfig +# Per-language discriminated unions the facades match on. +JavaBackend = Union[CodeAnalyzerConfig, Neo4jConnectionConfig] PyBackend = Union[PyCodeAnalyzerConfig, Neo4jConnectionConfig] TSBackend = Union[CodeAnalyzerConfig, Neo4jConnectionConfig] diff --git a/cldk/analysis/java/codeanalyzer/_jdk.py b/cldk/analysis/java/codeanalyzer/_jdk.py index 4015e445..d3dcb74c 100644 --- a/cldk/analysis/java/codeanalyzer/_jdk.py +++ b/cldk/analysis/java/codeanalyzer/_jdk.py @@ -36,12 +36,13 @@ from __future__ import annotations import hashlib -import json import logging import os import platform import stat import tarfile +import urllib.error +import urllib.parse import urllib.request import zipfile from pathlib import Path @@ -70,20 +71,36 @@ def _os_arch(cls) -> tuple[str, str]: @classmethod def _resolve_asset(cls) -> tuple[str, str]: - """Return ``(download_url, sha256)`` for the pinned JDK binary.""" + """Return ``(download_url, sha256)`` for the pinned JDK binary. + + Resolves via the Adoptium ``/binary/version`` endpoint, which 307-redirects to the + GitHub release asset; the checksum comes from the asset's adjacent ``.sha256.txt``. The + older ``/assets/version/{release}`` query endpoint is not used: it returns 404 for pinned + releases (e.g. ``jdk-21.0.5+11``), even though the release exists. + """ os_, arch = cls._os_arch() - url = ( - f"{cls._API}/assets/version/{JDK_RELEASE}" - f"?os={os_}&architecture={arch}&image_type=jdk" - f"&jvm_impl=hotspot&heap_size=normal&vendor=eclipse" - ) - req = urllib.request.Request(url, headers={"User-Agent": "cldk"}) - with urllib.request.urlopen(req, timeout=30) as resp: - data = json.load(resp) - if not data: - raise RuntimeError(f"No Temurin {JDK_RELEASE} build for {os_}/{arch}") - pkg = data[0]["binaries"][0]["package"] - return pkg["link"], pkg["checksum"] + release = urllib.parse.quote(JDK_RELEASE, safe="") # encode the '+' in the path + binary_url = f"{cls._API}/binary/version/{release}/{os_}/{arch}/jdk/hotspot/normal/eclipse" + + # Capture the redirect target (the GitHub asset URL) without downloading the binary. + class _NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, *args, **kwargs): + return None + + opener = urllib.request.build_opener(_NoRedirect) + req = urllib.request.Request(binary_url, headers={"User-Agent": "cldk"}) + try: + opener.open(req, timeout=30) + raise RuntimeError(f"Expected a redirect to the Temurin {JDK_RELEASE} asset from {binary_url}") + except urllib.error.HTTPError as exc: + if exc.code not in (301, 302, 303, 307, 308) or not exc.headers.get("Location"): + raise RuntimeError(f"No Temurin {JDK_RELEASE} build for {os_}/{arch} (HTTP {exc.code})") from exc + asset_url = exc.headers["Location"] + + sha_req = urllib.request.Request(asset_url + ".sha256.txt", headers={"User-Agent": "cldk"}) + with urllib.request.urlopen(sha_req, timeout=30) as resp: + sha = resp.read().decode().split()[0] + return asset_url, sha @classmethod def _java_home(cls, root: Path) -> Path: diff --git a/cldk/analysis/java/java_analysis.py b/cldk/analysis/java/java_analysis.py index 1f6f946f..60707b14 100644 --- a/cldk/analysis/java/java_analysis.py +++ b/cldk/analysis/java/java_analysis.py @@ -52,12 +52,13 @@ from tree_sitter import Tree -from cldk.analysis.commons.backend_config import CodeAnalyzerConfig, JavaBackend, cache_subdir +from cldk.analysis.commons.backend_config import CodeAnalyzerConfig, JavaBackend, Neo4jConnectionConfig, cache_subdir from cldk.analysis.commons.treesitter import TreesitterJava from cldk.models.java import JCallable from cldk.models.java import JApplication from cldk.models.java.models import JCRUDOperation, JComment, JCompilationUnit, JMethodDetail, JType, JField from cldk.analysis.java.codeanalyzer import JCodeanalyzer +from cldk.analysis.java.neo4j import JNeo4jBackend from cldk.analysis.java.backend import JavaAnalysisBackend @@ -149,22 +150,33 @@ def __init__( self.eager_analysis = eager_analysis self.target_files = target_files self.backend_config: JavaBackend = backend if backend is not None else CodeAnalyzerConfig() - # Java has a single backend family; the config only carries the cache root. analysis.json - # is cached under /java (None in source_code mode, where the analyzer streams - # results over a pipe). - cache_path = cache_subdir(self.backend_config.cache_dir, project_dir, "java") - if cache_path is not None: - cache_path.mkdir(parents=True, exist_ok=True) self.treesitter_java: TreesitterJava = TreesitterJava() - # Initialize the analysis backend - self.backend: JavaAnalysisBackend = JCodeanalyzer( - project_dir=self.project_dir, - source_code=self.source_code, - eager_analysis=self.eager_analysis, - analysis_level=self.analysis_level, - analysis_json_path=cache_path, - target_files=self.target_files, - ) + self.backend: JavaAnalysisBackend + if isinstance(self.backend_config, Neo4jConnectionConfig): + # Read-only: the graph is populated out of band; the SDK only polls it. + cfg = self.backend_config + application_name = cfg.application_name or (Path(project_dir).name if project_dir else None) + self.backend = JNeo4jBackend( + neo4j_uri=cfg.uri, + neo4j_username=cfg.username, + neo4j_password=cfg.password, + neo4j_database=cfg.database, + application_name=application_name, + ) + else: + # The config only carries the cache root. analysis.json is cached under /java + # (None in source_code mode, where the analyzer streams results over a pipe). + cache_path = cache_subdir(self.backend_config.cache_dir, project_dir, "java") + if cache_path is not None: + cache_path.mkdir(parents=True, exist_ok=True) + self.backend = JCodeanalyzer( + project_dir=self.project_dir, + source_code=self.source_code, + eager_analysis=self.eager_analysis, + analysis_level=self.analysis_level, + analysis_json_path=cache_path, + target_files=self.target_files, + ) def get_imports(self) -> List[str]: """Return all import statements in the source code. diff --git a/cldk/analysis/java/neo4j/__init__.py b/cldk/analysis/java/neo4j/__init__.py new file mode 100644 index 00000000..fc0a3cbe --- /dev/null +++ b/cldk/analysis/java/neo4j/__init__.py @@ -0,0 +1,22 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Read-only Neo4j-backed Java analysis backend (Cypher queries over the codeanalyzer-java graph).""" + +from cldk.analysis.java.neo4j.config import Neo4jConnectionConfig +from cldk.analysis.java.neo4j.neo4j_backend import JNeo4jBackend + +__all__ = ["JNeo4jBackend", "Neo4jConnectionConfig"] diff --git a/cldk/analysis/java/neo4j/config.py b/cldk/analysis/java/neo4j/config.py new file mode 100644 index 00000000..3ed20d43 --- /dev/null +++ b/cldk/analysis/java/neo4j/config.py @@ -0,0 +1,27 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Connection settings for the read-only Neo4j-backed Java analysis backend. + +The definition has been hoisted to :mod:`cldk.analysis.commons.backend_config`; it is re-exported +here for symmetry with the Python and TypeScript backends. +""" + +from __future__ import annotations + +from cldk.analysis.commons.backend_config import Neo4jConnectionConfig + +__all__ = ["Neo4jConnectionConfig"] diff --git a/cldk/analysis/java/neo4j/neo4j_backend.py b/cldk/analysis/java/neo4j/neo4j_backend.py new file mode 100644 index 00000000..6eed1a26 --- /dev/null +++ b/cldk/analysis/java/neo4j/neo4j_backend.py @@ -0,0 +1,711 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Neo4j-backed Java analysis backend (read-only Cypher client). + +A drop-in alternative to :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer`: it exposes the +**same query method surface** (the 36 methods of :class:`JavaAnalysisBackend`) so the +:class:`~cldk.analysis.java.JavaAnalysis` facade can delegate to either one, but instead of running +the analyzer JAR it **reconstructs the canonical ``JApplication`` from a Neo4j graph** (the one +``codeanalyzer-java`` >= 2.4.0 emits with ``--emit neo4j``) and then answers every query with the +*identical* logic the in-memory backend uses. Mirrors the Python / TypeScript Neo4j backends. + +It is purely a **query client**: it never builds the graph and has no dependency on the analyzer JAR, +a JDK, or the project sources. The graph is populated out of band — e.g. a job running +``codeanalyzer-java --emit neo4j`` — and the SDK only polls it. + +Reconstruction strategy (see :mod:`reconstruct`): the backend bulk-fetches every node + relationship +for the application in a handful of Cypher queries, groups children by parent, builds an +``analysis.json``-shaped dict, and hands it to ``JApplication(**payload)`` — the same constructor +path as ``JCodeanalyzer._init_japplication``. With ``self.application`` and ``self.call_graph`` +populated, the 36 query methods are the same code the in-memory backend runs. + +Identity / scoping model (must match the emitter; see ``codeanalyzer-java/schema.neo4j.json``): +``:JType`` (id = fqn) and ``:JCallable`` (id = ``#``) share a ``:JSymbol`` label; +compilation units are ``:JCompilationUnit`` keyed by ``file_key`` (== file path == symbol-table key); +call edges are ``(:JCallable)-[:J_CALLS {type, weight, source_kind, destination_kind}]->(:JCallable)``; +every project-owned node carries a ``_module`` provenance prop, so one DB can host several apps, all +scoped under ``(:JApplication {name})-[:J_HAS_UNIT]->(:JCompilationUnit)``. + +Parity: this backend reconstructs everything the graph actually contains identically to the +in-memory ``JCodeanalyzer`` (verified on the daytrader8 sample — 97% of checks, the rest being the +caveats below). The ``codeanalyzer-java`` **2.4.0** emitter had three projection gaps — fields all +collapsing to one ``#field#null`` node, imports reduced to ``:JPackage``, and ``J_CALLS`` +materializing only a fraction of the call graph — all **fixed in 2.4.1** +(codeanalyzer-java#156/#157/#158), the version the SDK now bundles (its release workflow fetches the +latest codeanalyzer-java jar). So a graph emitted by a current analyzer is a complete projection. + +Inherent caveats (present even on a complete graph, NOT query-layer bugs): + +* ``J_CALLS`` only links resolved app callables, so call edges to external/library targets (which the + in-memory backend keeps as synthetic nodes) are absent; +* the call graph is built by a separate analyzer run from the in-memory backend's ``analysis.json``, + so the two can differ by run-to-run WALA variance; +* a ``:JType``'s ``is_class_or_interface_declaration`` / ``is_concrete_class`` flags are not + projected (only the ``kind`` discriminator is); an absent singular ``comment`` rehydrates to + ``None``. +""" + +from __future__ import annotations + +import json +import logging +from itertools import chain, groupby +from typing import Any, Dict, List, Tuple, Union + +import networkx as nx + +from cldk.analysis.commons.treesitter import TreesitterJava +from cldk.analysis.java.backend import JavaAnalysisBackend +from cldk.analysis.java.neo4j import reconstruct as R +from cldk.models.java import JGraphEdges +from cldk.models.java.enums import CRUDOperationType +from cldk.models.java.models import JApplication, JCRUDOperation, JCallable, JCallableParameter, JComment, JField, JMethodDetail, JType, JCompilationUnit, JGraphEdgesST +from cldk.utils.exceptions.exceptions import CodeanalyzerExecutionException + +logger = logging.getLogger(__name__) + + +class JNeo4jBackend(JavaAnalysisBackend): + """Query the application view of a Java project over Neo4j (Cypher), read-only. + + Args: + neo4j_uri: Bolt URI of the Neo4j server (e.g. ``bolt://localhost:7687``). + neo4j_username / neo4j_password: Credentials (read-only is sufficient). + neo4j_database: Database name (None ⇒ server default). + application_name: The ``:JApplication`` anchor name to scope every query to. Matches the + ``--app-name`` the graph was loaded with (defaults to the project directory name). + """ + + def __init__( + self, + neo4j_uri: str, + neo4j_username: str, + neo4j_password: str, + neo4j_database: str | None = None, + application_name: str | None = None, + ) -> None: + try: + from neo4j import GraphDatabase + except ModuleNotFoundError as e: # pragma: no cover - import guard + raise CodeanalyzerExecutionException( + "The Neo4j backend requires the 'neo4j' driver. Install it with " + "`pip install neo4j` (or `pip install cldk[neo4j]`)." + ) from e + + if not application_name: + raise CodeanalyzerExecutionException("application_name is required to scope queries to an application.") + self.application_name = application_name + self._database = neo4j_database + self._driver = GraphDatabase.driver(neo4j_uri, auth=(neo4j_username, neo4j_password)) + + self._units: List[str] = self._load_unit_keys() + self.application: JApplication = self._reconstruct_application() + self.analysis_level = "call_graph" if self.application.call_graph else "symbol_table" + self.call_graph: nx.DiGraph | None = self._generate_call_graph(using_symbol_table=False) if self.application.call_graph else None + + # -----[ lifecycle ]----- + def close(self) -> None: + """Close the underlying Neo4j driver.""" + self._driver.close() + + def __enter__(self) -> "JNeo4jBackend": + return self + + def __exit__(self, *exc: Any) -> None: + self.close() + + def _run(self, query: str, **params: Any) -> List[Dict[str, Any]]: + with self._driver.session(database=self._database) as session: + return [record.data() for record in session.run(query, **params)] + + def _load_unit_keys(self) -> List[str]: + rows = self._run( + "MATCH (:JApplication {name: $app})-[:J_HAS_UNIT]->(u:JCompilationUnit) RETURN u.file_key AS k", + app=self.application_name, + ) + return [r["k"] for r in rows] + + # ===================================================================================== + # Reconstruction: bulk-fetch the graph and rebuild the canonical JApplication. + # ===================================================================================== + def _nodes(self, label: str) -> Dict[str, Dict[str, Any]]: + """All nodes of a label owned by this app, keyed by id/file_key/name.""" + rows = self._run( + f"MATCH (n:{label}) WHERE n._module IN $u RETURN coalesce(n.id, n.file_key, n.name) AS k, properties(n) AS p", + u=self._units, + ) + return {r["k"]: r["p"] for r in rows} + + def _adj(self, rtype: str, scope_child: bool = True) -> Dict[str, List[str]]: + """Adjacency parent_key → [child_keys] for a relationship, scoped to this app.""" + where = "b._module IN $u" if scope_child else "a._module IN $u" + rows = self._run( + f"MATCH (a)-[:{rtype}]->(b) WHERE {where} " + "RETURN coalesce(a.id, a.file_key, a.name) AS a, coalesce(b.id, b.file_key, b.name) AS b", + u=self._units, + ) + out: Dict[str, List[str]] = {} + for r in rows: + out.setdefault(r["a"], []).append(r["b"]) + return out + + def _reconstruct_application(self) -> JApplication: + units = self._units + # ---- node prop maps ---- + cu_nodes = { + r["k"]: r["p"] + for r in self._run( + "MATCH (:JApplication {name: $app})-[:J_HAS_UNIT]->(u:JCompilationUnit) RETURN u.file_key AS k, properties(u) AS p", + app=self.application_name, + ) + } + types = self._nodes("JType") + callables = self._nodes("JCallable") + fields = self._nodes("JField") + params = self._nodes("JParameter") + callsites = self._nodes("JCallSite") + variables = self._nodes("JVariable") + enums = self._nodes("JEnumConstant") + records = self._nodes("JRecordComponent") + initblocks = self._nodes("JInitializationBlock") + crudops = self._nodes("JCrudOperation") + crudqs = self._nodes("JCrudQuery") + comments = self._nodes("JComment") + + # ---- adjacencies ---- + a_callable = self._adj("J_HAS_CALLABLE") + a_field = self._adj("J_HAS_FIELD") + a_enum = self._adj("J_HAS_ENUM_CONSTANT") + a_record = self._adj("J_HAS_RECORD_COMPONENT") + a_init = self._adj("J_HAS_INIT_BLOCK") + a_param = self._adj("J_HAS_PARAMETER") + a_callsite = self._adj("J_HAS_CALLSITE") + a_var = self._adj("J_DECLARES_VAR") + a_crudop = self._adj("J_HAS_CRUD_OPERATION") + a_crudq = self._adj("J_HAS_CRUD_QUERY") + a_comment = self._adj("J_HAS_COMMENT") + a_import = self._run( + "MATCH (u:JCompilationUnit)-[r:J_IMPORTS]->(t) WHERE u._module IN $u " + "RETURN u.file_key AS cu, coalesce(t.fqn, t.name) AS path, properties(r) AS p", + u=units, + ) + + # ---- ordered helpers ---- + def _comments_of(owner_id: str) -> List[dict]: + ids = a_comment.get(owner_id, []) + built = [R.comment(comments[i]) for i in ids if i in comments] + return sorted(built, key=lambda c: (c["start_line"], c["start_column"])) + + def _first_comment(owner_id: str) -> dict | None: + cs = _comments_of(owner_id) + return cs[0] if cs else None + + def _param_index(pid: str) -> int: + try: + return int(pid.rsplit("#param#", 1)[1]) + except (IndexError, ValueError): + return 0 + + def _build_callsite(cs_id: str) -> dict: + p = callsites[cs_id] + op_ids = a_crudop.get(cs_id, []) + q_ids = a_crudq.get(cs_id, []) + crud_op = R.crud_operation(crudops[op_ids[0]]) if op_ids and op_ids[0] in crudops else None + crud_q = R.crud_query(crudqs[q_ids[0]]) if q_ids and q_ids[0] in crudqs else None + return R.callsite(p, comment_node=_first_comment(cs_id), crud_op=crud_op, crud_q=crud_q) + + def _callsites_of(owner_id: str) -> List[dict]: + ids = a_callsite.get(owner_id, []) + built = [(callsites[i], _build_callsite(i)) for i in ids if i in callsites] + return [cs for _, cs in sorted(built, key=lambda t: (t[0].get("start_line", -1), t[0].get("start_column", -1)))] + + def _vars_of(owner_id: str) -> List[dict]: + ids = a_var.get(owner_id, []) + built = [(variables[i], R.variable(variables[i], comment_node=_first_comment(i))) for i in ids if i in variables] + return [v for _, v in sorted(built, key=lambda t: (t[0].get("start_line", -1), t[0].get("name", "")))] + + # ---- callables ---- + def _build_callable(cid: str) -> dict: + p = callables[cid] + pids = sorted(a_param.get(cid, []), key=_param_index) + parameters = [R.parameter(params[i]) for i in pids if i in params] + op_ids = a_crudop.get(cid, []) + q_ids = a_crudq.get(cid, []) + crud_ops = [R.crud_operation(crudops[i]) for i in op_ids if i in crudops] + crud_qs = [R.crud_query(crudqs[i]) for i in q_ids if i in crudqs] + return R.callable_( + p, + comments=_comments_of(cid), + parameters=parameters, + call_sites=_callsites_of(cid), + variable_declarations=_vars_of(cid), + crud_operations=crud_ops, + crud_queries=crud_qs, + ) + + def _build_initblock(ib_id: str) -> dict: + p = initblocks[ib_id] + return R.init_block(p, comments=_comments_of(ib_id), call_sites=_callsites_of(ib_id), variable_declarations=_vars_of(ib_id)) + + # ---- types ---- + def _build_type(tid: str) -> dict: + p = types[tid] + cdecls = {} + for cid in a_callable.get(tid, []): + if cid in callables: + cdecls[callables[cid].get("signature", cid)] = _build_callable(cid) + fdecls = [R.field(fields[i], comment_node=_first_comment(i)) for i in a_field.get(tid, []) if i in fields] + econsts = [R.enum_constant(enums[i]) for i in a_enum.get(tid, []) if i in enums] + rcomps = [R.record_component(records[i], comment_node=_first_comment(i)) for i in a_record.get(tid, []) if i in records] + iblocks = [_build_initblock(i) for i in a_init.get(tid, []) if i in initblocks] + return R.type_( + p, + comments=_comments_of(tid), + callable_declarations=cdecls, + field_declarations=fdecls, + enum_constants=econsts, + record_components=rcomps, + initialization_blocks=iblocks, + ) + + # group types by owning module (file_key); type_declarations is a flat per-CU map + types_by_unit: Dict[str, Dict[str, dict]] = {} + for tid, tp in types.items(): + fkey = tp.get("_module") + fqn = tp.get("fqn", tid) + types_by_unit.setdefault(fkey, {})[fqn] = _build_type(tid) + + # imports by unit + imports_by_unit: Dict[str, List[dict]] = {} + for r in a_import: + imports_by_unit.setdefault(r["cu"], []).append( + {"path": r["path"], "is_static": r["p"].get("is_static", False), "is_wildcard": r["p"].get("is_wildcard", False)} + ) + + # ---- compilation units / symbol table ---- + symbol_table: Dict[str, dict] = {} + for fkey, cp in cu_nodes.items(): + symbol_table[fkey] = R.compilation_unit( + cp, + comments=_comments_of(fkey), + import_declarations=imports_by_unit.get(fkey, []), + type_declarations=types_by_unit.get(fkey, {}), + ) + + # ---- call graph edges ---- + call_edges: List[dict] = [] + for r in self._run( + "MATCH (s:JCallable)-[c:J_CALLS]->(t:JCallable) WHERE s._module IN $u " + "RETURN s.id AS src, t.id AS tgt, properties(c) AS p", + u=units, + ): + src = self._endpoint(r["src"], callables) + tgt = self._endpoint(r["tgt"], callables) + if src and tgt: + call_edges.append(R.call_edge(src, tgt, r["p"])) + + return JApplication(symbol_table=symbol_table, call_graph=call_edges) + + @staticmethod + def _endpoint(node_id: str, callables: Dict[str, Dict[str, Any]]) -> dict | None: + """A J_CALLS endpoint id (``#``) → a JGraphEdges source/target dict.""" + if "#" not in node_id: + return None + fqn, signature = node_id.split("#", 1) + props = callables.get(node_id, {}) + declaration = props.get("declaration") or signature + if "(" not in declaration: + declaration = signature + return {"file_path": props.get("file_path", ""), "type_declaration": fqn, "signature": signature, "callable_declaration": declaration} + + # ===================================================================================== + # JavaAnalysisBackend — leaf accessors (served from the reconstructed application) + # ===================================================================================== + def get_application_view(self) -> JApplication: + return self.application + + def get_symbol_table(self) -> Dict[str, JCompilationUnit]: + return self.application.symbol_table + + def get_system_dependency_graph(self) -> list[JGraphEdges]: + return self.application.call_graph or [] + + def get_compilation_units(self) -> List[JCompilationUnit]: + return list(self.application.symbol_table.values()) + + def get_java_compilation_unit(self, file_path: str) -> JCompilationUnit: + return self.application.symbol_table[file_path] + + # ===================================================================================== + # Call graph (logic mirrors JCodeanalyzer; calling_lines recomputed from JCallable.code) + # ===================================================================================== + def _generate_call_graph(self, using_symbol_table) -> nx.DiGraph: + cg = nx.DiGraph() + if using_symbol_table: + NotImplementedError("Call graph generation using symbol table is not implemented yet.") + else: + sdg = self.get_system_dependency_graph() + tsu = TreesitterJava() + edge_list = [ + ( + (jge.source.method.signature, jge.source.klass), + (jge.target.method.signature, jge.target.klass), + { + "type": jge.type, + "weight": jge.weight, + "calling_lines": ( + tsu.get_calling_lines(jge.source.method.code, jge.target.method.signature) + if not jge.source.method.is_implicit or not jge.target.method.is_implicit + else [] + ), + }, + ) + for jge in sdg + if jge.type == "CALL_DEP" + ] + for jge in sdg: + cg.add_node((jge.source.method.signature, jge.source.klass), method_detail=jge.source) + cg.add_node((jge.target.method.signature, jge.target.klass), method_detail=jge.target) + cg.add_edges_from(edge_list) + return cg + + def get_call_graph(self) -> nx.DiGraph: + if self.analysis_level == "symbol_table": + self.call_graph = self._generate_call_graph(using_symbol_table=True) + if self.call_graph is None: + self.call_graph = self._generate_call_graph(using_symbol_table=False) + return self.call_graph + + def get_call_graph_json(self) -> str: + callgraph_list = [] + edges = list(self.call_graph.edges.data("calling_lines")) + for edge in edges: + callgraph_dict = {} + callgraph_dict["source_method_signature"] = edge[0][0] + callgraph_dict["source_method_body"] = self.call_graph.nodes[edge[0]]["method_detail"].method.code + callgraph_dict["source_class"] = edge[0][1] + callgraph_dict["target_method_signature"] = edge[1][0] + callgraph_dict["target_method_body"] = self.call_graph.nodes[edge[1]]["method_detail"].method.code + callgraph_dict["target_class"] = edge[1][1] + callgraph_dict["calling_lines"] = edge[2] + callgraph_list.append(callgraph_dict) + return json.dumps(callgraph_list) + + def get_all_callers(self, target_class_name: str, target_method_signature: str, using_symbol_table: bool) -> Dict: + caller_detail_dict = {} + if using_symbol_table: + call_graph = self.__call_graph_using_symbol_table(qualified_class_name=target_class_name, method_signature=target_method_signature, is_target_method=True) + else: + call_graph = self.call_graph + if (target_method_signature, target_class_name) not in call_graph.nodes(): + return caller_detail_dict + in_edge_view = call_graph.in_edges(nbunch=(target_method_signature, target_class_name), data=True) + caller_detail_dict["caller_details"] = [] + caller_detail_dict["target_method"] = call_graph.nodes[(target_method_signature, target_class_name)]["method_detail"] + for source, target, data in in_edge_view: + cm = {"caller_method": call_graph.nodes[source]["method_detail"], "calling_lines": data["calling_lines"]} + caller_detail_dict["caller_details"].append(cm) + return caller_detail_dict + + def get_all_callees(self, source_class_name: str, source_method_signature: str, using_symbol_table: bool) -> Dict: + callee_detail_dict = {} + if using_symbol_table: + call_graph = self.__call_graph_using_symbol_table(qualified_class_name=source_class_name, method_signature=source_method_signature) + else: + call_graph = self.call_graph + if (source_method_signature, source_class_name) not in call_graph.nodes(): + return callee_detail_dict + out_edge_view = call_graph.out_edges(nbunch=(source_method_signature, source_class_name), data=True) + callee_detail_dict["callee_details"] = [] + callee_detail_dict["source_method"] = call_graph.nodes[(source_method_signature, source_class_name)]["method_detail"] + for source, target, data in out_edge_view: + cm = {"callee_method": call_graph.nodes[target]["method_detail"], "calling_lines": data["calling_lines"]} + callee_detail_dict["callee_details"].append(cm) + return callee_detail_dict + + # ===================================================================================== + # Classes / methods / fields (operate on the reconstructed symbol table) + # ===================================================================================== + def get_all_methods_in_application(self) -> Dict[str, Dict[str, JCallable]]: + class_method_dict = {} + class_dict = self.get_all_classes() + for k, v in class_dict.items(): + class_method_dict[k] = v.callable_declarations + return class_method_dict + + def get_all_classes(self) -> Dict[str, JType]: + class_dict = {} + for v in self.get_symbol_table().values(): + class_dict.update(v.type_declarations) + return class_dict + + def get_class(self, qualified_class_name) -> JType: + for v in self.get_symbol_table().values(): + if qualified_class_name in v.type_declarations.keys(): + return v.type_declarations.get(qualified_class_name) + + def get_method(self, qualified_class_name, method_signature) -> JCallable: + for v in self.get_symbol_table().values(): + if qualified_class_name in v.type_declarations.keys(): + ci = v.type_declarations[qualified_class_name] + for cd in ci.callable_declarations.keys(): + if cd == method_signature: + return ci.callable_declarations[cd] + + def get_method_parameters(self, qualified_class_name, method_signature) -> List[JCallableParameter]: + return self.get_method(qualified_class_name, method_signature).parameters + + def get_java_file(self, qualified_class_name) -> str: + for k, v in self.get_symbol_table().items(): + if qualified_class_name in v.type_declarations.keys(): + return k + + def get_all_methods_in_class(self, qualified_class_name) -> Dict[str, JCallable]: + ci = self.get_class(qualified_class_name) + if ci is None: + return {} + return {k: v for (k, v) in ci.callable_declarations.items() if v.is_constructor is False} + + def get_all_constructors(self, qualified_class_name) -> Dict[str, JCallable]: + ci = self.get_class(qualified_class_name) + if ci is None: + return {} + return {k: v for (k, v) in ci.callable_declarations.items() if v.is_constructor is True} + + def get_all_sub_classes(self, qualified_class_name) -> Dict[str, JType]: + all_classes = self.get_all_classes() + sub_classes = {} + for cls in all_classes: + if qualified_class_name in all_classes[cls].implements_list or qualified_class_name in all_classes[cls].extends_list: + sub_classes[cls] = all_classes[cls] + return sub_classes + + def get_all_fields(self, qualified_class_name) -> List[JField]: + ci = self.get_class(qualified_class_name) + if ci is None: + logging.warning(f"Class {qualified_class_name} not found in the application view.") + return list() + return ci.field_declarations + + def get_all_nested_classes(self, qualified_class_name) -> List[JType]: + ci = self.get_class(qualified_class_name) + if ci is None: + logging.warning(f"Class {qualified_class_name} not found in the application view.") + return list() + return [self.get_class(c) for c in ci.nested_type_declarations] + + def get_extended_classes(self, qualified_class_name) -> List[str]: + ci = self.get_class(qualified_class_name) + if ci is None: + logging.warning(f"Class {qualified_class_name} not found in the application view.") + return list() + return ci.extends_list + + def get_implemented_interfaces(self, qualified_class_name) -> List[str]: + ci = self.get_class(qualified_class_name) + if ci is None: + logging.warning(f"Class {qualified_class_name} not found in the application view.") + return list() + return ci.implements_list + + # ===================================================================================== + # Symbol-table call graph (pure-Python over call sites; mirrors JCodeanalyzer) + # ===================================================================================== + def get_class_call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: + call_graph = self.__call_graph_using_symbol_table(qualified_class_name, method_signature) + if method_signature is None: + filter_criteria = {node for node in call_graph.nodes if node[1] == qualified_class_name} + else: + filter_criteria = {node for node in call_graph.nodes if tuple(node) == (method_signature, qualified_class_name)} + graph_edges: List[Tuple[JMethodDetail, JMethodDetail]] = list() + for edge in call_graph.edges(nbunch=filter_criteria): + source: JMethodDetail = call_graph.nodes[edge[0]]["method_detail"] + target: JMethodDetail = call_graph.nodes[edge[1]]["method_detail"] + graph_edges.append((source, target)) + return graph_edges + + def __call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str, is_target_method: bool = False) -> nx.DiGraph: + cg = nx.DiGraph() + if is_target_method: + sdg = self.__raw_call_graph_using_symbol_table_target_method(target_class_name=qualified_class_name, target_method_signature=method_signature) + else: + sdg = self.__raw_call_graph_using_symbol_table(qualified_class_name=qualified_class_name, method_signature=method_signature) + tsu = TreesitterJava() + edge_list = [ + ( + (jge.source.method.signature, jge.source.klass), + (jge.target.method.signature, jge.target.klass), + {"type": jge.type, "weight": jge.weight, "calling_lines": tsu.get_calling_lines(jge.source.method.code, jge.target.method.signature)}, + ) + for jge in sdg + ] + for jge in sdg: + cg.add_node((jge.source.method.signature, jge.source.klass), method_detail=jge.source) + cg.add_node((jge.target.method.signature, jge.target.klass), method_detail=jge.target) + cg.add_edges_from(edge_list) + return cg + + def __raw_call_graph_using_symbol_table_target_method(self, target_class_name: str, target_method_signature: str, cg=None) -> list[JGraphEdgesST]: + if cg is None: + cg = [] + target_method_details = self.get_method(qualified_class_name=target_class_name, method_signature=target_method_signature) + for class_name in self.get_all_classes(): + for method in self.get_all_methods_in_class(qualified_class_name=class_name): + method_details = self.get_method(qualified_class_name=class_name, method_signature=method) + for call_site in method_details.call_sites: + source_method_details = None + source_class = "" + callee_signature = call_site.callee_signature if call_site.callee_signature != "" else "" + if call_site.receiver_type != "": + if self.get_class(qualified_class_name=call_site.receiver_type): + found_method, found_class = self.__find_method_in_hierarchy(call_site.receiver_type, callee_signature) + if found_method is not None and callee_signature == target_method_signature and found_class == target_class_name: + source_method_details = self.get_method(method_signature=method, qualified_class_name=class_name) + source_class = class_name + else: + found_method, found_class = self.__find_method_in_hierarchy(class_name, callee_signature) + if found_method is not None and callee_signature == target_method_signature and found_class == target_class_name: + source_method_details = self.get_method(method_signature=method, qualified_class_name=class_name) + source_class = class_name + if source_class != "" and source_method_details is not None: + call_edge = JGraphEdgesST( + source=JMethodDetail(method_declaration=source_method_details.declaration, klass=source_class, method=source_method_details), + target=JMethodDetail(method_declaration=target_method_details.declaration, klass=target_class_name, method=target_method_details), + type="CALL_DEP", + weight="1", + ) + if call_edge not in cg: + cg.append(call_edge) + return cg + + def __find_method_in_hierarchy(self, qualified_class_name: str, method_signature: str) -> Tuple[JCallable | None, str]: + klass = self.get_class(qualified_class_name=qualified_class_name) + method_details = self.get_method(method_signature=method_signature, qualified_class_name=qualified_class_name) + if method_details is not None and klass is not None and not klass.is_interface: + return method_details, qualified_class_name + if klass is not None: + for parent_class in klass.extends_list: + parent_method, found_class = self.__find_method_in_hierarchy(parent_class, method_signature) + if parent_method is not None: + return parent_method, found_class + return None, "" + + def __raw_call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str, cg=None) -> list[JGraphEdgesST]: + if cg is None: + cg = [] + source_method_details = self.get_method(qualified_class_name=qualified_class_name, method_signature=method_signature) + if source_method_details is None: + return cg + for call_site in source_method_details.call_sites: + target_method_details = None + target_class = "" + callee_signature = call_site.callee_signature if call_site.callee_signature != "" else "" + if call_site.receiver_type != "": + if self.get_class(qualified_class_name=call_site.receiver_type): + tmd, found_class = self.__find_method_in_hierarchy(call_site.receiver_type, callee_signature) + if tmd is not None: + target_method_details = tmd + target_class = found_class + else: + tmd, found_class = self.__find_method_in_hierarchy(qualified_class_name, callee_signature) + if tmd is not None: + target_method_details = tmd + target_class = found_class + if target_class != "" and target_method_details is not None: + call_edge = JGraphEdgesST( + source=JMethodDetail(method_declaration=source_method_details.declaration, klass=qualified_class_name, method=source_method_details), + target=JMethodDetail(method_declaration=target_method_details.declaration, klass=target_class, method=target_method_details), + type="CALL_DEP", + weight="1", + ) + if call_edge not in cg: + cg.append(call_edge) + return cg + + def get_class_call_graph(self, qualified_class_name: str, method_name: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: + if method_name is None: + filter_criteria = {node for node in self.call_graph.nodes if node[1] == qualified_class_name} + else: + filter_criteria = {node for node in self.call_graph.nodes if tuple(node) == (method_name, qualified_class_name)} + graph_edges: List[Tuple[JMethodDetail, JMethodDetail]] = list() + for edge in self.call_graph.edges(nbunch=filter_criteria): + source: JMethodDetail = self.call_graph.nodes[edge[0]]["method_detail"] + target: JMethodDetail = self.call_graph.nodes[edge[1]]["method_detail"] + graph_edges.append((source, target)) + return graph_edges + + def remove_all_comments(self, src_code: str) -> str: + raise NotImplementedError("This function is not implemented yet.") + + # ===================================================================================== + # Entry points / CRUD / comments (operate on the reconstructed symbol table) + # ===================================================================================== + def get_all_entry_point_methods(self) -> Dict[str, Dict[str, JCallable]]: + methods = chain.from_iterable( + ((typename, method, callable) for method, callable in methods.items() if callable.is_entrypoint) for typename, methods in self.get_all_methods_in_application().items() + ) + return {typename: {method: callable for _, method, callable in group} for typename, group in groupby(methods, key=lambda x: x[0])} + + def get_all_entry_point_classes(self) -> Dict[str, JType]: + return {typename: klass for typename, klass in self.get_all_classes().items() if klass.is_entrypoint_class} + + def _crud(self, op_filter: CRUDOperationType | None) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: + rows = [] + for class_name, class_details in self.get_all_classes().items(): + for method_name, method_details in class_details.callable_declarations.items(): + if method_details.crud_operations and len(method_details.crud_operations) > 0: + ops = method_details.crud_operations if op_filter is None else [o for o in method_details.crud_operations if o.operation_type == op_filter] + rows.append({class_name: class_details, method_name: method_details, "crud_operations": ops}) + return rows + + def get_all_crud_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: + return self._crud(None) + + def get_all_read_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: + return self._crud(CRUDOperationType.READ) + + def get_all_create_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: + return self._crud(CRUDOperationType.CREATE) + + def get_all_update_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: + return self._crud(CRUDOperationType.UPDATE) + + def get_all_delete_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: + return self._crud(CRUDOperationType.DELETE) + + def get_comments_in_a_method(self, qualified_class_name: str, method_signature: str) -> List[JComment]: + return self.get_method(qualified_class_name, method_signature).comments + + def get_comments_in_a_class(self, qualified_class_name: str) -> List[JComment]: + return self.get_class(qualified_class_name).comments + + def get_comment_in_file(self, file_path: str) -> List[JComment]: + compilation_unit = self.get_symbol_table().get(file_path, None) + if compilation_unit is None: + raise CodeanalyzerExecutionException(f"File {file_path} not found in the symbol table.") + return compilation_unit.comments + + def get_all_comments(self) -> Dict[str, List[JComment]]: + return {file_path: self.get_comment_in_file(file_path) for file_path in self.get_symbol_table()} + + def get_all_docstrings(self) -> List[Tuple[str, JComment]]: + docstrings = {} + for file_path, list_of_comments in self.get_all_comments().items(): + javadoc_comments = [docstring for docstring in list_of_comments if docstring.is_javadoc] + if javadoc_comments: + docstrings[file_path] = javadoc_comments + return docstrings diff --git a/cldk/analysis/java/neo4j/reconstruct.py b/cldk/analysis/java/neo4j/reconstruct.py new file mode 100644 index 00000000..131c5209 --- /dev/null +++ b/cldk/analysis/java/neo4j/reconstruct.py @@ -0,0 +1,287 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Pure rehydration: Neo4j property maps → ``analysis.json``-shaped dicts for ``cldk.models.java``. + +:class:`~cldk.analysis.java.neo4j.JNeo4jBackend` bulk-fetches every node + relationship for an +application, groups children by parent, and feeds the grouped props here. Each function returns a +plain ``dict`` matching the corresponding pydantic model's field names, so the backend can assemble a +single ``analysis.json``-shaped payload and hand it to ``JApplication(**payload)`` — the exact same +constructor path the in-memory :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer` uses +(``_init_japplication``). That guarantees the reconstructed objects are identical. + +The source graph is the one ``codeanalyzer-java`` (>= 2.4.0) emits with ``--emit neo4j`` — see its +``neo4j/GraphProjector.java`` / ``schema.neo4j.json`` for the property flattening these functions +invert. Java comments are first-class ``:JComment`` nodes (``J_HAS_COMMENT``), so unlike the Python +backend they round-trip losslessly. + +Parity caveats (inherent to what the projection stores, not bugs): a ``JType``'s +``is_class_or_interface_declaration`` and ``is_concrete_class`` flags are not projected (only the +``kind`` discriminator is), so they rehydrate to their defaults; the order of ``call_graph`` edges +is sorted rather than original-insertion order. +""" + +from __future__ import annotations + +import json +from typing import Any, Dict, List, Mapping + +Props = Mapping[str, Any] + + +# -----[ helpers ]----- +def _arr(props: Props, key: str) -> List[str]: + return list(props.get(key, []) or []) + + + + +def _kind_flags(kind: str | None) -> Dict[str, bool]: + """Derive the type-discriminator booleans from the projected ``kind`` string.""" + return { + "is_interface": kind == "interface", + "is_enum_declaration": kind == "enum", + "is_annotation_declaration": kind == "annotation", + "is_record_declaration": kind == "record", + } + + +# -----[ leaf nodes ]----- +def comment(props: Props) -> dict: + return { + "content": props.get("content"), + "start_line": props.get("start_line", -1), + "end_line": props.get("end_line", -1), + "start_column": props.get("start_column", -1), + "end_column": props.get("end_column", -1), + "is_javadoc": props.get("is_javadoc", False), + } + + +def parameter(props: Props) -> dict: + return { + "name": props.get("name"), + "type": props.get("type", ""), + "annotations": _arr(props, "annotations"), + "modifiers": _arr(props, "modifiers"), + "start_line": props.get("start_line", -1), + "end_line": props.get("end_line", -1), + "start_column": props.get("start_column", -1), + "end_column": props.get("end_column", -1), + } + + +def field(props: Props, *, comment_node: dict | None = None) -> dict: + raw = props.get("variable_initializers_json") + return { + "comment": comment_node, + "type": props.get("type", ""), + "start_line": props.get("start_line", -1), + "end_line": props.get("end_line", -1), + "variables": _arr(props, "variables"), + "modifiers": _arr(props, "modifiers"), + "annotations": _arr(props, "annotations"), + "variable_initializers": json.loads(raw) if raw else {}, + } + + +def variable(props: Props, *, comment_node: dict | None = None) -> dict: + return { + "comment": comment_node, + "name": props.get("name", ""), + "type": props.get("type", ""), + "initializer": props.get("initializer", ""), + "start_line": props.get("start_line", -1), + "start_column": props.get("start_column", -1), + "end_line": props.get("end_line", -1), + "end_column": props.get("end_column", -1), + } + + +def enum_constant(props: Props) -> dict: + return {"name": props.get("name", ""), "arguments": _arr(props, "arguments")} + + +def record_component(props: Props, *, comment_node: dict | None = None) -> dict: + return { + "comment": comment_node, + "name": props.get("name", ""), + "type": props.get("type", ""), + "modifiers": _arr(props, "modifiers"), + "annotations": _arr(props, "annotations"), + "default_value": props.get("default_value"), + "is_var_args": props.get("is_var_args", False), + } + + +def crud_operation(props: Props) -> dict: + return {"line_number": props.get("line_number", -1), "operation_type": props.get("operation_type")} + + +def crud_query(props: Props) -> dict: + return { + "line_number": props.get("line_number", -1), + "query_arguments": props.get("query_arguments"), + "query_type": props.get("query_type"), + } + + +def callsite(props: Props, *, comment_node: dict | None = None, crud_op: dict | None = None, crud_q: dict | None = None) -> dict: + return { + "comment": comment_node, + "method_name": props.get("method_name", ""), + "receiver_expr": props.get("receiver_expr", ""), + "receiver_type": props.get("receiver_type", ""), + "argument_types": _arr(props, "argument_types"), + "argument_expr": _arr(props, "argument_expr"), + "return_type": props.get("return_type", ""), + "callee_signature": props.get("callee_signature", ""), + "is_static_call": props.get("is_static_call"), + "is_private": props.get("is_private"), + "is_public": props.get("is_public"), + "is_protected": props.get("is_protected"), + "is_unspecified": props.get("is_unspecified"), + "is_constructor_call": props.get("is_constructor_call", False), + "crud_operation": crud_op, + "crud_query": crud_q, + "start_line": props.get("start_line", -1), + "start_column": props.get("start_column", -1), + "end_line": props.get("end_line", -1), + "end_column": props.get("end_column", -1), + } + + +# -----[ declarations ]----- +def init_block( + props: Props, + *, + comments: List[dict] | None = None, + call_sites: List[dict] | None = None, + variable_declarations: List[dict] | None = None, +) -> dict: + return { + "file_path": props.get("file_path", ""), + "comments": comments or [], + "annotations": _arr(props, "annotations"), + "thrown_exceptions": _arr(props, "thrown_exceptions"), + "code": props.get("code", ""), + "start_line": props.get("start_line", -1), + "end_line": props.get("end_line", -1), + "is_static": props.get("is_static", False), + "referenced_types": _arr(props, "referenced_types"), + "accessed_fields": _arr(props, "accessed_fields"), + "call_sites": call_sites or [], + "variable_declarations": variable_declarations or [], + "cyclomatic_complexity": props.get("cyclomatic_complexity", 0), + } + + +def callable_( + props: Props, + *, + comments: List[dict] | None = None, + parameters: List[dict] | None = None, + call_sites: List[dict] | None = None, + variable_declarations: List[dict] | None = None, + crud_operations: List[dict] | None = None, + crud_queries: List[dict] | None = None, +) -> dict: + return { + "signature": props.get("signature", ""), + "is_implicit": props.get("is_implicit", False), + "is_constructor": props.get("is_constructor", False), + "comments": comments or [], + "annotations": _arr(props, "annotations"), + "modifiers": _arr(props, "modifiers"), + "thrown_exceptions": _arr(props, "thrown_exceptions"), + "declaration": props.get("declaration", ""), + "parameters": parameters or [], + "return_type": props.get("return_type"), + "code": props.get("code", ""), + "start_line": props.get("start_line", -1), + "end_line": props.get("end_line", -1), + "code_start_line": props.get("code_start_line", -1), + "referenced_types": _arr(props, "referenced_types"), + "accessed_fields": _arr(props, "accessed_fields"), + "call_sites": call_sites or [], + "is_entrypoint": props.get("is_entrypoint", False), + "variable_declarations": variable_declarations or [], + "crud_operations": crud_operations or [], + "crud_queries": crud_queries or [], + "cyclomatic_complexity": props.get("cyclomatic_complexity", 0), + } + + +def type_( + props: Props, + *, + comments: List[dict] | None = None, + callable_declarations: Dict[str, dict] | None = None, + field_declarations: List[dict] | None = None, + enum_constants: List[dict] | None = None, + record_components: List[dict] | None = None, + initialization_blocks: List[dict] | None = None, +) -> dict: + out = { + "is_inner_class": props.get("is_inner_class", False), + "is_local_class": props.get("is_local_class", False), + "is_nested_type": props.get("is_nested_type", False), + "comments": comments or [], + "extends_list": _arr(props, "extends_list"), + "implements_list": _arr(props, "implements_list"), + "modifiers": _arr(props, "modifiers"), + "annotations": _arr(props, "annotations"), + "parent_type": props.get("parent_type", ""), + "nested_type_declarations": _arr(props, "nested_type_declarations"), + "callable_declarations": callable_declarations or {}, + "field_declarations": field_declarations or [], + "enum_constants": enum_constants or [], + "record_components": record_components or [], + "initialization_blocks": initialization_blocks or [], + "is_entrypoint_class": props.get("is_entrypoint_class", False), + } + out.update(_kind_flags(props.get("kind"))) + return out + + +def compilation_unit( + props: Props, + *, + comments: List[dict] | None = None, + import_declarations: List[dict] | None = None, + type_declarations: Dict[str, dict] | None = None, +) -> dict: + return { + "file_path": props.get("file_path", props.get("file_key", "")), + "package_name": props.get("package_name", ""), + "comments": comments or [], + "import_declarations": import_declarations or [], + "type_declarations": type_declarations or {}, + "is_modified": props.get("is_modified", False), + } + + +def call_edge(source: dict, target: dict, props: Props) -> dict: + """A ``JGraphEdges``-shaped raw dict; endpoints resolve via JApplication's lookup table.""" + weight = props.get("weight") + return { + "source": source, + "target": target, + "type": props.get("type", "CALL_DEP"), + "weight": str(weight) if weight is not None else "1", + "source_kind": props.get("source_kind"), + "destination_kind": props.get("destination_kind"), + } diff --git a/cldk/core.py b/cldk/core.py index b8e1d281..485eb7ce 100644 --- a/cldk/core.py +++ b/cldk/core.py @@ -145,7 +145,10 @@ def java( CldkInitializationException: If neither or both of ``project_path`` / ``source_code`` are provided. """ - if project_path is None and source_code is None: + # The read-only Neo4j backend reads a graph populated out of band, so it needs neither + # project_path nor source_code. + is_neo4j = isinstance(backend, Neo4jConnectionConfig) + if project_path is None and source_code is None and not is_neo4j: raise CldkInitializationException("Either project_path or source_code must be provided.") if project_path is not None and source_code is not None: raise CldkInitializationException("Both project_path and source_code are provided. Please provide only one.") diff --git a/pyproject.toml b/pyproject.toml index da32f3be..db0140fa 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,7 +40,6 @@ dependencies = [ "tree-sitter-javascript==0.23.1", "clang==17.0.6", "libclang==17.0.6", - "codeanalyzer-java==2.3.7", "codeanalyzer-python==0.2.0", "codeanalyzer-typescript==0.4.0", ] @@ -88,7 +87,7 @@ include = [ ] [tool.backend-versions] -codeanalyzer-java = "2.3.7" +codeanalyzer-java = "2.4.1" codeanalyzer-python = "0.2.0" codeanalyzer-typescript = "0.4.0" diff --git a/tests/analysis/java/test_java_backend_contract.py b/tests/analysis/java/test_java_backend_contract.py index 45cff1cb..a14492bc 100644 --- a/tests/analysis/java/test_java_backend_contract.py +++ b/tests/analysis/java/test_java_backend_contract.py @@ -23,10 +23,15 @@ from cldk.analysis.java.backend import JavaAnalysisBackend from cldk.analysis.java.codeanalyzer.codeanalyzer import JCodeanalyzer +from cldk.analysis.java.neo4j import JNeo4jBackend +# Both interchangeable backends must satisfy the same contract. +BACKENDS = [JCodeanalyzer, JNeo4jBackend] -def test_backend_subclasses_contract(): - assert issubclass(JCodeanalyzer, JavaAnalysisBackend) + +@pytest.mark.parametrize("backend", BACKENDS) +def test_backend_subclasses_contract(backend): + assert issubclass(backend, JavaAnalysisBackend) def test_contract_is_abstract(): @@ -34,8 +39,9 @@ def test_contract_is_abstract(): JavaAnalysisBackend() -def test_backend_fully_implements_contract(): - assert JCodeanalyzer.__abstractmethods__ == frozenset() +@pytest.mark.parametrize("backend", BACKENDS) +def test_backend_fully_implements_contract(backend): + assert backend.__abstractmethods__ == frozenset() def test_contract_covers_every_method_the_facade_delegates(): diff --git a/tests/analysis/java/test_java_neo4j_backend.py b/tests/analysis/java/test_java_neo4j_backend.py new file mode 100644 index 00000000..93463874 --- /dev/null +++ b/tests/analysis/java/test_java_neo4j_backend.py @@ -0,0 +1,147 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Integration parity tests: the read-only Java Neo4j backend vs the analysis.json backend. + +These assert that :class:`JNeo4jBackend` answers every query **identically** to the canonical +:class:`JCodeanalyzer` (analysis.json) backend on the same project — the definition of the +"1-to-1 map". + +Because the Java graph is populated out of band by the analyzer JAR (which needs a JDK and a Maven +build of the target for a level-2 call graph), this test does **not** populate inline. It is skipped +unless you point it at an already-populated Neo4j and a matching reference analysis cache: + + CLDK_TEST_NEO4J_URI=bolt://localhost:7687 \ + CLDK_TEST_NEO4J_USER=neo4j \ + CLDK_TEST_NEO4J_PASSWORD=test \ + CLDK_TEST_NEO4J_JAVA_APP=daytrader \ # the --app-name the graph was loaded with + CLDK_TEST_JAVA_PROJECT=/path/to/project \ # the reference project dir + CLDK_TEST_JAVA_CACHE=/path/to/cache/java \ # dir containing the reference analysis.json + pytest tests/analysis/java/test_java_neo4j_backend.py + +Populate the graph with: ``codeanalyzer-java -i --analysis-level 2 --emit neo4j +--neo4j-uri ... --app-name ``, and produce the reference with ``... --analysis-level 2 -o +``. + +Parity is asserted modulo the projection's documented-lossy fields (see +``cldk.analysis.java.neo4j.reconstruct``): a ``JType``'s ``is_class_or_interface_declaration`` / +``is_concrete_class`` flags are not projected (only ``kind`` is). +""" + +import json +import logging +import os + +import pytest + +logging.getLogger("neo4j").setLevel(logging.ERROR) + +NEO4J_URI = os.environ.get("CLDK_TEST_NEO4J_URI", "bolt://localhost:7687") +NEO4J_USER = os.environ.get("CLDK_TEST_NEO4J_USER", "neo4j") +NEO4J_PASSWORD = os.environ.get("CLDK_TEST_NEO4J_PASSWORD", "neo4j") +JAVA_APP = os.environ.get("CLDK_TEST_NEO4J_JAVA_APP") +JAVA_PROJECT = os.environ.get("CLDK_TEST_JAVA_PROJECT") +JAVA_CACHE = os.environ.get("CLDK_TEST_JAVA_CACHE") + +LOSSY_TYPE = {"is_class_or_interface_declaration", "is_concrete_class"} + + +def _neo4j_reachable() -> bool: + if not (JAVA_APP and JAVA_PROJECT and JAVA_CACHE): + return False + try: + from neo4j import GraphDatabase + except ModuleNotFoundError: + return False + try: + driver = GraphDatabase.driver(NEO4J_URI, auth=(NEO4J_USER, NEO4J_PASSWORD)) + driver.verify_connectivity() + driver.close() + return True + except Exception: # noqa: BLE001 + return False + + +pytestmark = pytest.mark.skipif( + not _neo4j_reachable(), + reason="needs a pre-populated Neo4j Java graph + reference cache (set CLDK_TEST_NEO4J_* / CLDK_TEST_JAVA_*)", +) + + +def _norm(o): + if hasattr(o, "model_dump"): + o = o.model_dump() + if isinstance(o, dict): + return {k: _norm(v) for k, v in o.items() if k not in LOSSY_TYPE} + if isinstance(o, list): + items = [_norm(x) for x in o] + try: + return sorted(items, key=lambda x: json.dumps(x, sort_keys=True, default=str)) + except Exception: + return items + return o + + +@pytest.fixture(scope="module") +def backends(): + from cldk.analysis.java.codeanalyzer.codeanalyzer import JCodeanalyzer + from cldk.analysis.java.neo4j import JNeo4jBackend + + ref = JCodeanalyzer(project_dir=JAVA_PROJECT, source_code=None, analysis_json_path=JAVA_CACHE, analysis_level="call_graph", eager_analysis=False, target_files=None) + neo = JNeo4jBackend(neo4j_uri=NEO4J_URI, neo4j_username=NEO4J_USER, neo4j_password=NEO4J_PASSWORD, application_name=JAVA_APP) + yield ref, neo + neo.close() + + +def test_symbol_table_and_classes_parity(backends): + ref, neo = backends + assert sorted(ref.get_symbol_table()) == sorted(neo.get_symbol_table()) + ac_ref, ac_neo = ref.get_all_classes(), neo.get_all_classes() + assert sorted(ac_ref) == sorted(ac_neo) + for cls in ac_ref: + assert _norm(ac_ref[cls]) == _norm(ac_neo[cls]), f"class {cls} differs" + + +def test_methods_fields_hierarchy_parity(backends): + ref, neo = backends + for cls in ref.get_all_classes(): + assert _norm(ref.get_all_fields(cls)) == _norm(neo.get_all_fields(cls)) + assert ref.get_extended_classes(cls) == neo.get_extended_classes(cls) + assert ref.get_implemented_interfaces(cls) == neo.get_implemented_interfaces(cls) + assert sorted(ref.get_all_sub_classes(cls)) == sorted(neo.get_all_sub_classes(cls)) + mc = ref.get_all_methods_in_class(cls) + assert sorted(mc) == sorted(neo.get_all_methods_in_class(cls)) + for sig in mc: + assert _norm(ref.get_method(cls, sig)) == _norm(neo.get_method(cls, sig)), f"{cls}::{sig} differs" + + +def test_call_graph_parity(backends): + ref, neo = backends + gr, gn = ref.get_call_graph(), neo.get_call_graph() + + def edgeset(g): + return sorted([list(u), list(v), g[u][v].get("type"), str(g[u][v].get("weight"))] for u, v in g.edges) + + assert edgeset(gr) == edgeset(gn) + assert sorted([list(n) for n in gr.nodes]) == sorted([list(n) for n in gn.nodes]) + + +def test_entrypoints_and_comments_parity(backends): + ref, neo = backends + assert sorted(ref.get_all_entry_point_classes()) == sorted(neo.get_all_entry_point_classes()) + assert sorted(ref.get_all_entry_point_methods()) == sorted(neo.get_all_entry_point_methods()) + assert sorted(ref.get_all_comments()) == sorted(neo.get_all_comments()) + assert sorted(ref.get_all_docstrings()) == sorted(neo.get_all_docstrings()) diff --git a/tests/analysis/java/test_java_neo4j_selection.py b/tests/analysis/java/test_java_neo4j_selection.py new file mode 100644 index 00000000..22e6a5b9 --- /dev/null +++ b/tests/analysis/java/test_java_neo4j_selection.py @@ -0,0 +1,75 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Backend-selection unit tests for the Java facade (no live Neo4j required). + +The Neo4j backend is fully mocked here, so these run anywhere. They verify that passing a +``Neo4jConnectionConfig`` swaps the facade onto :class:`JNeo4jBackend` (read-only), and that +without one the in-process :class:`JCodeanalyzer` is used. +""" + +from unittest.mock import patch + +import pytest + +from cldk.analysis.commons.backend_config import CodeAnalyzerConfig, Neo4jConnectionConfig +from cldk.analysis.java.java_analysis import JavaAnalysis + + +def test_neo4j_config_selects_neo4j_backend(): + config = Neo4jConnectionConfig(uri="bolt://example:7687", username="neo4j", password="secret", application_name="myapp") + with patch("cldk.analysis.java.java_analysis.JNeo4jBackend") as backend_cls, patch("cldk.analysis.java.java_analysis.JCodeanalyzer") as in_process_cls: + backend = backend_cls.return_value + + # Read-only: no project_dir needed, the graph is loaded out of band. + analysis = JavaAnalysis(project_dir=None, source_code=None, analysis_level="call_graph", target_files=None, eager_analysis=False, backend=config) + + _, kwargs = backend_cls.call_args + assert kwargs["neo4j_uri"] == "bolt://example:7687" + assert kwargs["neo4j_password"] == "secret" + assert kwargs["application_name"] == "myapp" + assert analysis.backend is backend + assert isinstance(analysis.backend_config, Neo4jConnectionConfig) + in_process_cls.assert_not_called() + + +def test_no_config_uses_in_process_backend(tmp_path): + with patch("cldk.analysis.java.java_analysis.JCodeanalyzer") as backend_cls, patch("cldk.analysis.java.java_analysis.JNeo4jBackend") as neo4j_cls: + analysis = JavaAnalysis(project_dir=str(tmp_path), source_code=None, analysis_level="symbol_table", target_files=None, eager_analysis=False) + + backend_cls.assert_called_once() + neo4j_cls.assert_not_called() + assert analysis.backend is backend_cls.return_value + assert isinstance(analysis.backend_config, CodeAnalyzerConfig) + + +def test_missing_neo4j_driver_raises_helpful_error(): + """Without the optional ``neo4j`` driver, constructing the backend explains how to install it.""" + import builtins + + from cldk.analysis.java.neo4j import JNeo4jBackend + from cldk.utils.exceptions.exceptions import CodeanalyzerExecutionException + + real_import = builtins.__import__ + + def _no_neo4j(name, *args, **kwargs): + if name == "neo4j": + raise ModuleNotFoundError("No module named 'neo4j'") + return real_import(name, *args, **kwargs) + + with patch("builtins.__import__", side_effect=_no_neo4j): + with pytest.raises(CodeanalyzerExecutionException, match="neo4j"): + JNeo4jBackend(neo4j_uri="bolt://example:7687", neo4j_username="neo4j", neo4j_password="neo4j", application_name="app")