From 82579a83d57fd63187a1bae2b3e9fe72e9a9c68f Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Wed, 19 Aug 2026 22:34:39 -0400 Subject: [PATCH 01/23] docs(spec): framework-independent entrypoint detection design (#27) Records the design from the brainstorming pass: node-level PyEntrypoint records with a derived is_entrypoint boolean, a post-pass detection pipeline over the built symbol table, a user-extensible rules.yml for the declarative rules, and named engines for routing, packaging readers and structural passes. Corrects the roadmap's claim that entrypoint vocabulary has been coined three ways. Java has the only implementation; TypeScript declares TSApplication. entrypoints in its SCHEMA_DECISIONS invariant spine but TSEntrypoint appears nowhere in its source. Python is the second implementation and sets the shape. --- .../2026-08-19-entrypoint-detection-design.md | 279 ++++++++++++++++++ 1 file changed, 279 insertions(+) create mode 100644 docs/design/specs/2026-08-19-entrypoint-detection-design.md diff --git a/docs/design/specs/2026-08-19-entrypoint-detection-design.md b/docs/design/specs/2026-08-19-entrypoint-detection-design.md new file mode 100644 index 0000000..dec1338 --- /dev/null +++ b/docs/design/specs/2026-08-19-entrypoint-detection-design.md @@ -0,0 +1,279 @@ +# Spec: framework-independent entrypoint detection + +Status: draft for review +Date: 2026-08-19 +Scope: `codeanalyzer-python` — schema v2 fields, a new `codeanalyzer/entrypoints/` subsystem +Issue: #27 + +--- + +## 1. Summary + +The analyzer emits a symbol table and a call graph with no identified roots. Nothing says which +callables a framework invokes from outside the application, so reachability, dead-code and +attack-surface analyses have nowhere to start, and every consumer re-derives it. Downstream, all +four `PythonAnalysis.get_entry_point_*` methods raise `NotImplementedError` +(`python-sdk/cldk/analysis/python/python_analysis.py:881-960`) purely because the backend supplies +no data, while Java has emitted `is_entrypoint` / `is_entrypoint_class` for some time +(`python-sdk/cldk/models/java/models.py:343,403`). + +This adds a post-pass over the built symbol table that flags entrypoints from four independent +mechanisms — declared metadata, definition-site decorators, inheritance, and external routing +tables — driven by a user-extensible `rules.yml`. + +### Prior art, corrected + +The roadmap states the entrypoint vocabulary has been "coined three ways". Verified on +2026-08-19, that is not accurate: **Java has the only implementation** (boolean flags plus a +`JEntrypoint` Neo4j marker label). TypeScript *declares* `TSApplication.entrypoints: +Dict[str, List[TSEntrypoint]]` in its `SCHEMA_DECISIONS.md` invariant spine, but `TSEntrypoint` +appears nowhere in `codeanalyzer-typescript-v2/src/` — the only matches are comments about +bundler entrypoints. Python is therefore the **second** implementation, and the record shape +chosen here is what TypeScript would be asked to match. + +## 2. Decisions + +| # | Decision | Rationale | +| --- | --- | --- | +| D1 | Node-level records, not a root collection | Java precedent; keeps the fact next to the node it describes | +| D2 | Routing pre-pass ships in v1; Django supported from the start | Django binds views in `urls.py`, not at the definition site. Detection that silently misses the most common enterprise framework is worse than none — a consumer cannot distinguish "no entrypoints" from "unsupported" | +| D3 | `urls.py` resolved by **constrained symbolic evaluation**, never execution | Literals, name lookups, list/tuple concatenation and comprehensions over literal iterables. No function calls, no conditionals, nothing imported and run. Executing the code under analysis would make CLDK unsafe to point at untrusted repositories | +| D4 | `entrypoints: List[PyEntrypoint]`, with `is_entrypoint: bool` derived | One callable is genuinely an entrypoint more than once — two `@app.route`s, or both a Celery task and a CLI command. A boolean or a single framework string silently collapses that. The boolean is denormalized for Java parity and a one-field SDK filter | +| D5 | No `kind` field | `framework` plus `route is not None` answers the queries `kind` would serve; a second vocabulary to align across analyzers is not worth it | +| D6 | Declarative `rules.yml` for decorator / inheritance / naming rules; named **engines** for routing, packaging readers and structural passes | Groups 2-3 are pattern lists that rot as frameworks change; adding Sanic should be a data edit. Routing is partial evaluation and cannot be expressed as data | +| D7 | Both the routed class and its dispatched methods are flagged; methods carry `via` | `urls.py` names the class, the framework calls the method. Class-only leaves handler methods unreachable, defeating the primary use case | + +Also decided: rules carry stable ids so a user file can `disable:` them; user rules merge additively +with the shipped set, deduplicated on `(framework, evidence, route)`; a malformed **user** rules +file is a hard error before analysis begins; each record records which ruleset produced it. CLI: +`--entrypoint-rules `, repeatable. + +## 3. Data model + +```python +class PyEntrypoint(BaseModel): + framework: str # flask | django | celery | packaging | ... + confidence: str # declared | certain | heuristic + rule: str # stable id, e.g. "flask.route" + ruleset: str # "shipped" | "user:" + evidence: Optional[str] = None # binding site, e.g. "shop/urls.py:7" + route: Optional[str] = None # composed path, HTTP only + http_methods: List[str] = [] + via: Optional[str] = None # can:// id of the routed node dispatching here +``` + +Carried on `PyCallable` and `PyClass`: + +```python +entrypoints: List[PyEntrypoint] = [] +is_entrypoint: bool = False # derived: len(entrypoints) > 0, never authored +``` + +For `path("products/", ProductList.as_view())` reached through `path("shop/", include("shop.urls"))`: + +``` +PyClass ProductList route "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/shop/products/" via: null +ProductList.get route "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/shop/products/" via: +ProductList.post route "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/shop/products/" via: +``` + +The class states what is routed; the methods state what is invoked. `via` lets a consumer walk +from a reachability root back to the route reaching it. For decorator-site frameworks `via` is +null — the function is both routed and invoked. + +`route` on a method is copied from the class, not independently derived. `confidence` is +per-record, so one callable may hold a `declared` record from `[project.scripts]` and a +`heuristic` one from a naming convention at the same time; consumers threshold on it rather than +inheriting this analyzer's judgement. + +### Application-level report + +```python +class PyEntrypointReport(BaseModel): + frameworks_detected: List[str] = [] + rulesets: List[str] = [] + unresolved: Dict[str, int] = {} # file -> patterns not resolved + errors: List[str] = [] +``` + +on `PyApplication`. D3's failure mode is silence; this is what makes it visible. A consumer seeing +`frameworks_detected: ["django"]`, zero entrypoints, and `unresolved: {"shop/urls.py": 12}` knows +exactly what happened. + +## 4. Detection pipeline + +Entrypoint detection is a **post-pass over the built symbol table**, not a pre-pass threaded +through the builder as #27 originally sketched. `urls.py` names `views.product_list`, and +resolving that requires the symbol table to exist. A post-pass makes every resolution a lookup +against ids that already exist and leaves `symbol_table_builder.py` untouched. + +``` +symbol table built (L1) + Stage 0 framework detection — imports present + dependency manifest; gates all later stages + Stage 1 declared readers (no AST) + Stage 2 routing pre-pass (per project) + Stage 3 per-node matching (rules.yml: decorators, bases, dispatch) + Stage 4 structural passes (argument-position, __main__ walk) + Stage 5 derive is_entrypoint + emit the report +``` + +Stages 1-4 **append**; multiplicity is the model, so two stages flagging one callable is correct, +not a conflict. Stage 0 gating means a project without Celery never pays for Celery rules and +cannot false-positive on a locally-defined `shared_task`. + +### Declared readers + +All reduce to one shape: a string naming `module:attr`. One engine, pluggable readers, one shared +resolver turning `pkg.cli:main` into a `can://` id. + +`pyproject.toml` `[project.scripts]` / `[project.gui-scripts]` / `[project.entry-points.""]` +(plugin systems such as `pytest11` are real entrypoints invoked by other tools), Poetry's +`[tool.poetry.scripts]`, `setup.cfg` `[options.entry_points]`, SAM / `serverless.yml` handlers, +and — via engines, since they are not structured data — `setup.py`, `Procfile`, `Dockerfile`. + +`setup.py` is D3's problem in miniature: `entry_points` can be computed. Same posture — match the +literal dict, record what cannot be resolved, never execute. + +A declared entry proves the package *declares* an entrypoint, not that the target lives in this +repo. The resolver drops records whose id is absent from the symbol table and counts them, under +the same no-dangling-endpoints rule the call graph enforces. + +## 5. `rules.yml` + +Three blocks: `declared:` (readers), `frameworks:` (decorators, bases, dispatch), and engines +named by key. Matching is on `qualified_name`, which #128 made available — so `@route` under +`from flask import route` matches the same rule as `@app.route`. + +```yaml +version: 1 + +declared: + - id: pyproject.scripts + file: pyproject.toml + format: toml + path: [project, scripts] + - id: setup.py + file: setup.py + engine: setup_py + +frameworks: + flask: + detect: [flask] + decorators: + - id: flask.route + match: "flask.Flask.route" + route: {from: positional, index: 0} + methods: {from: keyword, name: methods, default: [GET]} + bases: + - id: flask.methodview + match: "flask.views.MethodView" + transitive: true + dispatch: [get, post, put, delete, patch] + + django: + detect: [django] + bases: + - id: django.cbv + match: "django.views.generic.*" + transitive: true + dispatch: [get, post, put, patch, delete, head, options] + routing: django_urls +``` + +`confidence` defaults to `certain` and is written only when it is not. `dispatch:` is what makes +D7 declarative. `path:` is capped at key names plus `*` (one level) and `**` (recursive) — the +moment a reader needs a predicate or a transform it becomes an `engine:` instead. Without that +line `rules.yml` becomes a programming language with no debugger. + +## 6. Django routing engine + +Input: the built symbol table. Output: `{dotted_view_name -> [Route]}` plus an unresolved count +per file. + +``` +1. Roots ROOT_URLCONF from settings if literal; else every urls.py +2. Env module-level literals, lists/tuples, imported names resolved via the symbol table +3. Walk path/re_path/url(literal, target, ...) + Name | Attribute -> view reference + Call to *.as_view() -> the class + include("app.urls") -> recurse, compose prefix + include((module, namespace)) -> same, keep namespace + router.urls -> from registered routers +4. Routers router.register(prefix, ViewSet) binds that ViewSet at that prefix +5. Otherwise record unresolved, with file:line +``` + +Dispatched methods come from the `dispatch:` list of the matching `bases:` rule, **intersected +with the methods the class actually defines** — a `ListView` defining only `get` gets no phantom +`post` entrypoint. + +Prefix composition is the part most likely to be subtly wrong (trailing slashes, regex prefixes, +empty prefixes). `route` is therefore best-effort presentation; the flag and `via` are the +load-bearing facts, and a consumer computing reachability must not depend on the string being +exact. + +## 7. Error handling + +Entrypoint detection is additive metadata: its failure degrades to "no entrypoints", never to "no +analysis". The post-pass is wrapped; a finder crash loses flags, not the symbol table — and is +recorded in `PyEntrypointReport.errors` rather than only logged. + +| Failure | Behaviour | +| --- | --- | +| Malformed **user** rules file | Hard error before analysis starts — never silently skipped | +| Malformed **shipped** rules file | Schema-validated in CI; cannot occur at runtime | +| No `pyproject.toml` / `setup.cfg` | Reader skips; not an error | +| Declared target absent from the symbol table | Record dropped and counted (no dangling endpoints) | +| `urls.py` fails to parse | File skipped, counted unresolved, walk continues | +| `include()` cycle | Visited-set; recorded as unresolved | +| Jedi cannot resolve a base class | Rule does not match — under-approximate, never guess | + +## 8. Testing + +The repo has **no Django fixture**; `whole_applications/` holds `flask`, `requests` and `xarray`, +which are those libraries' own source, not applications using them. Fixtures are real work. + +`single_functionalities/django_routing/` exercises: prefix composition through `include()`; a CBV +via `as_view()` with the dispatch split; a plain function view; `urlpatterns = base + extra`; a +deliberately unresolvable `path(COMPUTED, ...)`; a DRF `router.register`; a `ListView` defining +only `get`; and a helper that must not be flagged. + +Smaller fixtures for Flask, FastAPI, Celery and Click, plus a `packaging/` fixture with +`[project.scripts]`, a plugin `entry-points` group, and one entry pointing at a dependency that +must be dropped. + +Assertions are **exact sets** of `(node, framework, rule, route, via)`, hand-written and compared +— the only way to catch over-flagging. The tests that matter most are negative and gating: the +helper is not flagged; a local `shared_task` in a project without Celery is not flagged; the +`ListView` yields exactly one method entrypoint; the unresolvable pattern increments `unresolved` +rather than vanishing. + +Plus unit tests for rules loading (merge, `disable:` by id, malformed user file raising before +analysis, `path:` wildcards) and a monotonicity check that entrypoints are identical at `-a 1` +through `-a 4`, since this is L1 data. + +## 9. Out of scope + +CRUD detection, specced separately. Taint sources and reachability computation — the analyzer +emits substrate; those are SDK queries under the provider/client boundary. The argument-position +rule (a first-party callable passed to a call resolving outside the project — which would catch +`add_url_rule(view_func=h)`, `scheduler.add_job(run)` and any framework with no finder) is +**deferred**: `PyCallArgument` carries only `ast_kind` and `inferred_type`, with no resolved +identity, so it needs the same id-space work #128 did for decorators. It is the rule that +generalizes furthest and should be the first follow-up. + +## 10. Risks + +- **Option B's coverage against real projects is unmeasured.** The fixtures test the rules we + wrote; they cannot say what fraction of real `urls.py` files resolve. Before anyone depends on + these flags for security work, run the pre-pass over several open-source Django projects and + report the resolved ratio. +- **`rules.yml` is a model pack in the analyzer**, while this repo's provider/client split puts + model packs in the SDK. The distinction being set deliberately: *policy* packs (what is a taint + source) stay in the SDK; *detection* packs requiring the AST live in the analyzer, because the + SDK sees only serialized output. +- **Shipped rules go stale** as frameworks change. Mitigated by user extensibility and by + `frameworks_detected` making a coverage gap visible, not by promising to keep up. +- **`via` is new cross-language vocabulary.** If TypeScript later implements its paper + `entrypoints` collection, the per-entrypoint record is the shared part and only placement + differs — but the name should be agreed before it ships twice. From 132b2a787f27781b12238e2dcaaaefa2145223f2 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Wed, 19 Aug 2026 22:35:13 -0400 Subject: [PATCH 02/23] docs(spec): close ambiguities found in spec self-review (#27) Pins confidence to a closed set with each value defined; states how engines name their rule id, not just declarative rules; makes the detect: predicate explicit (import OR manifest entry, either sufficient); and adds the decomposition, since the design spans six units rather than one pull request. --- .../2026-08-19-entrypoint-detection-design.md | 37 ++++- .../specs/call-site-body-convergence.md | 142 ++++++++++++++++++ 2 files changed, 175 insertions(+), 4 deletions(-) create mode 100644 docs/design/specs/call-site-body-convergence.md diff --git a/docs/design/specs/2026-08-19-entrypoint-detection-design.md b/docs/design/specs/2026-08-19-entrypoint-detection-design.md index dec1338..d604d21 100644 --- a/docs/design/specs/2026-08-19-entrypoint-detection-design.md +++ b/docs/design/specs/2026-08-19-entrypoint-detection-design.md @@ -53,8 +53,13 @@ file is a hard error before analysis begins; each record records which ruleset p ```python class PyEntrypoint(BaseModel): framework: str # flask | django | celery | packaging | ... - confidence: str # declared | certain | heuristic - rule: str # stable id, e.g. "flask.route" + confidence: str # closed set: "declared" | "certain" | "heuristic" + # declared - named in a manifest; cannot be wrong + # certain - an unambiguous framework signal + # heuristic - a convention or a weak signal + rule: str # stable id. Declarative rules use their `id:` from + # rules.yml ("flask.route"); engines use their engine + # name ("django_urls", "pyproject.scripts") ruleset: str # "shipped" | "user:" evidence: Optional[str] = None # binding site, e.g. "shop/urls.py:7" route: Optional[str] = None # composed path, HTTP only @@ -109,7 +114,10 @@ against ids that already exist and leaves `symbol_table_builder.py` untouched. ``` symbol table built (L1) - Stage 0 framework detection — imports present + dependency manifest; gates all later stages + Stage 0 framework detection — gates all later stages. A framework is detected when its + package is imported by first-party source OR named in the dependency manifest + (either is sufficient; a manifest entry with no import still gates in, since the + import may be dynamic) Stage 1 declared readers (no AST) Stage 2 routing pre-pass (per project) Stage 3 per-node matching (rules.yml: decorators, bases, dispatch) @@ -262,7 +270,28 @@ rule (a first-party callable passed to a call resolving outside the project — identity, so it needs the same id-space work #128 did for decorators. It is the rule that generalizes furthest and should be the first follow-up. -## 10. Risks +## 10. Decomposition + +This is not one pull request. The units below land independently and in this order; each is +closed by its own PR and its own work item, filed when picked up: + +1. **Schema + pipeline skeleton** — `PyEntrypoint`, `PyEntrypointReport`, the carriers, the + derived boolean, and the wrapped post-pass that currently finds nothing. Establishes the + contract and the failure posture with no detection logic to argue about. +2. **`rules.yml` loader** — parsing, schema validation, shipped/user merge, `disable:`, the CLI + flag. Testable with no framework involved. +3. **Declarative matching (Stage 3)** — decorator and inheritance rules, the `dispatch:` split. + Delivers Flask, FastAPI, Celery, Click and DRF decorators; the first user-visible result. +4. **Declared readers (Stage 1)** — packaging metadata and manifests. Independent of 3; could + swap order. +5. **Django routing engine (Stage 2)** — the largest and riskiest unit, and the one needing a + fixture built from scratch. +6. **Structural passes (Stage 4)** — `__main__` walk. The argument-position rule stays deferred. + +Unit 1 gates everything. Units 3 and 4 are parallel. Unit 5 should not start before 1-3 are +merged, since it depends on both the record shape and the `dispatch:` mechanism. + +## 11. Risks - **Option B's coverage against real projects is unmeasured.** The fixtures test the rules we wrote; they cannot say what fraction of real `urls.py` files resolve. Before anyone depends on diff --git a/docs/design/specs/call-site-body-convergence.md b/docs/design/specs/call-site-body-convergence.md new file mode 100644 index 0000000..658d062 --- /dev/null +++ b/docs/design/specs/call-site-body-convergence.md @@ -0,0 +1,142 @@ +# Spec: converge `call_sites[]` into `body{}` — separating the IR from the wire format + +Status: draft for review +Date: 2026-08-19 +Scope: `codeanalyzer-python` schema v2, `analysis.json` + Neo4j projection +Related: #120 (converge `call_sites[]`/`accessed_symbols[]`/`local_variables[]` with `body{}`) + +--- + +## 1. The finding + +Every call site is emitted **twice**, under two unrelated identity schemes. +Verified on `main` (`6f02581`), fixture `return cls()` at line 34: + +| representation | id | Neo4j | +| --- | --- | --- | +| `PyCallable.call_sites[]` | `app.py#34:15-34:22` | `:PyCallSite` ← `PY_HAS_CALLSITE` | +| `PyCallable.body{"34:15"}`, `kind:"call"` | `@34:15` | `:PyCFGNode` ← `PY_HAS_CFG_NODE` | + +One call in source, two graph nodes, no edge between them. `PyCallSite`'s id +(`file#line:col-line:col`) belongs to neither identity tier the schema defines — +it is neither a durable `can://` id nor a `@` ordinal id. + +**They cannot disagree in content.** `schema/l1_body.py` derives one from the other +in the same pass: + +```python +for cs in c.call_sites or []: + key = f"{cs.start_line}:{cs.start_column}" + c.body[key] = BodyNode(kind="call", span=span, callee=None) +``` + +So this is not a correctness bug. It is redundancy by construction. + +## 2. Why it exists + +`call_sites[]` is **not** v1 debris left lying around. It is the analyzer's internal +working record, and it is load-bearing for every level above L1: + +| reader | uses | +| --- | --- | +| `semantic_analysis/call_graph.py:163-215` | `callee_signature`, `method_name`, `is_constructor_call` — builds the L2 call graph | +| `schema/l2_callees.py` | `callee_signature`, `start_line`, `start_column` — backfills `BodyNode.callee` | +| `dataflow/builder.py:368-420` | `callee_signature`, `is_constructor_call`, position — builds SDG call sites | +| `dataflow/summaries.py`, `dataflow/sdg.py` | consume the above | +| `neo4j/project.py:400` | projects `:PyCallSite` | + +`body{}` is the v2 wire view *derived from* that record. The duplication is an +**internal IR leaking into the wire format** — not two competing encodings of equal +standing. That reframing is what makes the fix tractable: the wire format can lose +`call_sites[]` without the internal passes losing anything, provided the record +survives as an internal structure. + +## 3. Design + +**`body{}` is the single emitted representation of a call site.** Consumers obtain the +call-site set by filtering `body` on `kind == "call"`, which works from **L1** — verified: + +``` +L1 body{"34:15"} kind=call callee=null +L2 body{"34:15"} kind=call callee="can://…/@external/app.Account/__init__" +``` + +The Jedi-produced call record stays **internal**: it is the input to L2 resolution and +L3/L4 dataflow, and is not part of the contract. `PyCallsite` leaves the emitted schema. + +### Field disposition + +| field | disposition | why | +| --- | --- | --- | +| `start_line` / `start_column` / `end_*` | **becomes the node key + `span`** | already how `body{}` is keyed | +| `callee_signature` | **internal only** | it is the *input* to resolution; `callee` (a resolved `can://` id) is what the wire carries | +| `is_constructor_call` | **internal only** | read by `call_graph.py` + `builder.py`; recoverable on the wire from `callee` resolving to an `__init__` | +| `method_name` | **internal only** | read by `call_graph.py`; recoverable on the wire from `callee` or the `span` slice | +| `argument_types` | **deleted** | deprecated in 0.3.1 (#86) with "will be removed in schema v2"; no internal reader; removal overdue | +| `arguments` | **moves onto the call `BodyNode`** | no internal reader; output-only detail worth keeping. Encoding is OPEN — see § 5 | +| `receiver_expr` | **moves onto the call `BodyNode`** | no internal reader; Jedi inference with no other home | +| `receiver_type` | **moves onto the call `BodyNode`** | as above | +| `return_type` | **moves onto the call `BodyNode`** | as above | + +### Resulting `BodyNode` + +```python +class BodyNode(BaseModel): + kind: str # statement | call | entry | exit | formal_* | actual_* + span: Optional[Span] = None + callee: Optional[str] = None # call nodes; the sanctioned null→id slot at L2 + of: Optional[str] = None + parent: Optional[str] = None + # call-specific, Jedi inference, absent on every other kind + receiver_expr: Optional[str] = None + receiver_type: Optional[str] = None + return_type: Optional[str] = None +``` + +### Neo4j consequences + +- `:PyCallSite`, `PY_HAS_CALLSITE`, and the `file#line:col-line:col` id scheme are **removed**. +- One body-node label, keyed on the global ordinal id, carries every `body{}` entry. +- `PY_RESOLVES_TO` is re-sourced from `BodyNode.callee` instead of `callee_signature`. +- Merge groups drop from 9 to 8. +- **Rename the body label.** `PyCFGNode` names an L3 concept, but a `call` node exists from + L1 and is deliberately **never** on the CFG spine — verified: at L3 the call `34:15` carries + `parent="34:8"` and appears in no `cfg` edge, while `@entry`/`34:8`/`@exit` do. The current + label asserts CFG membership for a node that has none. A level-neutral name (`PyBodyNode`, + matching what `body{}` is called in the JSON) states what is true at every level. +- If `MATCH (:PyCallSite)` ergonomics are wanted, the writer already supports label layering + (`rows.py:98` merges on `labels[0]` and unions the rest; `cypher.py:95-97` renders it), so a + marker label costs no second node and no second id. Note `MARKER_LABELS` in `neo4j/schema.py` + is declaration-only today — the writer never reads it, and no call site passes >1 label. + +## 4. What this does not do + +- Does not touch `accessed_symbols[]` or `local_variables[]`, the other two halves of #120. +- Does not change the call graph, Jedi resolution, or any dataflow analysis — only which + representation is serialized. +- Does not settle the Neo4j merge-label strategy, the `can://` callable-signature grammar, or + the decorator shape (#128). Those are separate decisions. + +## 5. Open questions + +- **`arguments` encoding.** Inline objects (`{ast_kind, inferred_type}`, what Python does now + and what works at L1) or local-ids referencing argument body nodes. The latter requires + materializing argument nodes at L1, which is a much larger change to the body model and the + id space. Recommendation: keep inline. +- **Body label name.** `PyBodyNode` is the obvious candidate; anything level-neutral works. +- **Whether the internal record stays a Pydantic model** excluded from serialization, or becomes + a plain dataclass in the analysis passes. Purely internal; no contract impact. + +## 6. Caveats and risks + +- **Breaking for anyone reading `call_sites[]`.** That is the point of the change, but it is the + most visible field in the callable model and its removal should lead the release notes. +- **Three fields become wire-recoverable rather than wire-present** (`method_name`, + `is_constructor_call`, `callee_signature`). Recovering `method_name` from a `span` slice is + string work a consumer may not want to do. If that proves unpopular the honest fix is to put + `method_name` back on the node, not to restore `call_sites[]`. +- **`callee` must actually resolve** for `is_constructor_call` and `method_name` to be + recoverable. Where resolution fails, `callee` is null and both are lost. The size of that + set is unmeasured and should be measured before the fields are dropped. +- **Test surface.** Every test asserting on `call_sites[]` changes. They should be rewritten + against `body{}`, not deleted. From df02919cf01b9a598075c0875a00a697ecbf4213 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 04:12:57 -0400 Subject: [PATCH 03/23] docs(plan): implementation plan for entrypoint detection units 1-3 (#27) Eight TDD tasks covering the schema, the wrapped post-pass, the rules.yml loader, the CLI flag, framework gating, and decorator/inheritance matching -- delivering Flask/FastAPI/Celery/Click/DRF detection end to end. Units 4-6 (declared readers, Django routing engine, structural passes) get their own plan; units 1-3 produce working software on their own. Self-review caught three defects in the plan itself: PyImport requires both module and name, and for 'from flask import Flask' the package is in module rather than name, so both the test and the detection helper were wrong; options.py imports Optional but not Tuple; and a len(name) <= 7 heuristic would have emitted DRF's list/retrieve/create dispatch names as HTTP verbs. --- ...26-08-20-entrypoint-detection-units-1-3.md | 1327 +++++++++++++++++ 1 file changed, 1327 insertions(+) create mode 100644 docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md diff --git a/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md b/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md new file mode 100644 index 0000000..e668fd6 --- /dev/null +++ b/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md @@ -0,0 +1,1327 @@ +# Entrypoint Detection (Units 1-3) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Flag framework entrypoints on callables and classes from declarative rules, delivering working Flask / FastAPI / Celery / Click / DRF-decorator detection end to end. + +**Architecture:** A post-pass over the built L1 symbol table appends `PyEntrypoint` records to `PyCallable.entrypoints` and `PyClass.entrypoints`; `is_entrypoint` is derived from the list. Detection is driven by a shipped `rules.yml` that users can extend. The pass is wrapped so its failure degrades to "no entrypoints", never "no analysis". + +**Tech Stack:** Python 3.9+, Pydantic v2, PyYAML, Jedi, pytest. + +**Spec:** `docs/design/specs/2026-08-19-entrypoint-detection-design.md` + +## Global Constraints + +- Entrypoints are **L1** data. They must be byte-identical at `-a 1` through `-a 4`; the monotonicity gate applies. +- `is_entrypoint` is **derived**, never authored: `len(entrypoints) > 0`. +- `confidence` is a closed set: `"declared"` | `"certain"` | `"heuristic"`. +- Matching is against `PyDecorator.qualified_name`, never the written spelling. +- The post-pass never raises. Failures land in `PyEntrypointReport.errors`. +- A malformed **user** rules file is a hard error before analysis starts. +- New schema fields are optional with defaults, so existing payloads still load. +- Follow repo conventions: tests live in `test/`, run with `uv run pytest`. + +**Out of scope for this plan:** declared readers (Unit 4), the Django routing engine (Unit 5), structural passes (Unit 6). Do not add `routing:` handling; the key is parsed and ignored. + +--- + +### Task 1: Schema — `PyEntrypoint`, `PyEntrypointReport`, carriers + +**Files:** +- Modify: `codeanalyzer/schema/py_schema.py` +- Test: `test/test_entrypoint_schema.py` + +**Interfaces:** +- Consumes: `Span`, `builder` decorator (already in `py_schema.py`) +- Produces: `PyEntrypoint`, `PyEntrypointReport`, `PyCallable.entrypoints`, `PyCallable.is_entrypoint`, `PyClass.entrypoints`, `PyClass.is_entrypoint`, `PyApplication.entrypoint_report` + +- [ ] **Step 1: Write the failing test** + +```python +# test/test_entrypoint_schema.py +from codeanalyzer.schema.py_schema import ( + PyApplication, PyCallable, PyClass, PyEntrypoint, PyEntrypointReport, +) + + +def test_entrypoint_record_defaults(): + e = PyEntrypoint(framework="flask", confidence="certain", rule="flask.route", ruleset="shipped") + assert e.evidence is None and e.route is None and e.via is None + assert e.http_methods == [] + + +def test_callable_and_class_carry_entrypoints(): + c = PyCallable(name="f", path="a.py", signature="a.f") + k = PyClass(name="C", signature="a.C") + assert c.entrypoints == [] and c.is_entrypoint is False + assert k.entrypoints == [] and k.is_entrypoint is False + + +def test_application_carries_a_report(): + app = PyApplication(symbol_table={}) + assert isinstance(app.entrypoint_report, PyEntrypointReport) + assert app.entrypoint_report.frameworks_detected == [] +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest test/test_entrypoint_schema.py -v --no-cov` +Expected: FAIL with `ImportError: cannot import name 'PyEntrypoint'` + +- [ ] **Step 3: Write minimal implementation** + +Add to `codeanalyzer/schema/py_schema.py`, immediately before `class PyCallableParameter`: + +```python +@builder +class PyEntrypoint(BaseModel): + """One way a callable or class is invoked from outside the application (#27). + + A node may hold several: two ``@app.route`` decorators, or a function that + is both a Celery task and a CLI command. ``confidence`` lets a consumer + threshold on evidence quality rather than inheriting this analyzer's + judgement. + """ + + framework: str + confidence: str = "certain" # "declared" | "certain" | "heuristic" + rule: str = "" # rules.yml `id:`, or an engine name + ruleset: str = "shipped" # "shipped" | "user:" + evidence: Optional[str] = None + route: Optional[str] = None + http_methods: List[str] = [] + via: Optional[str] = None # can:// id of the routed node dispatching here + + +@builder +class PyEntrypointReport(BaseModel): + """Coverage and failure record for the entrypoint pass (#27). + + The pass under-approximates by design, so silence is its failure mode. + This is what makes a gap visible instead of indistinguishable from + "this project has no entrypoints". + """ + + frameworks_detected: List[str] = [] + rulesets: List[str] = [] + unresolved: Dict[str, int] = {} + errors: List[str] = [] +``` + +Then add to `PyCallable` (after `decorators`) and to `PyClass` (after `decorators`): + +```python + entrypoints: List[PyEntrypoint] = [] + is_entrypoint: bool = False +``` + +And to `PyApplication` (after `external_symbols`): + +```python + entrypoint_report: PyEntrypointReport = PyEntrypointReport() +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest test/test_entrypoint_schema.py -v --no-cov` +Expected: 3 passed + +- [ ] **Step 5: Commit** + +```bash +git add codeanalyzer/schema/py_schema.py test/test_entrypoint_schema.py +git commit -m "feat(schema): PyEntrypoint records on callables and classes (#27)" +``` + +--- + +### Task 2: Pipeline skeleton — a wrapped post-pass that finds nothing + +**Files:** +- Create: `codeanalyzer/entrypoints/__init__.py` +- Create: `codeanalyzer/entrypoints/pipeline.py` +- Modify: `codeanalyzer/core.py` (after `reidentify_call_graph(app, sig_to_id)`) +- Test: `test/test_entrypoint_pipeline.py` + +**Interfaces:** +- Consumes: `PyApplication`, `PyEntrypointReport` from Task 1 +- Produces: `detect_entrypoints(app, project_dir, rule_paths=()) -> None` — mutates `app` in place, never raises + +- [ ] **Step 1: Write the failing test** + +```python +# test/test_entrypoint_pipeline.py +from pathlib import Path + +from codeanalyzer.entrypoints.pipeline import detect_entrypoints +from codeanalyzer.schema.py_schema import PyApplication + + +def test_pass_is_a_noop_on_an_empty_application(tmp_path: Path): + app = PyApplication(symbol_table={}) + detect_entrypoints(app, tmp_path) + assert app.entrypoint_report.errors == [] + + +def test_pass_never_raises_and_records_the_failure(tmp_path: Path, monkeypatch): + """A finder crash must lose flags, not the analysis.""" + import codeanalyzer.entrypoints.pipeline as p + + def boom(*a, **k): + raise RuntimeError("finder exploded") + + monkeypatch.setattr(p, "_run_stages", boom) + app = PyApplication(symbol_table={}) + detect_entrypoints(app, tmp_path) # must not raise + assert any("finder exploded" in e for e in app.entrypoint_report.errors) + + +def test_derives_is_entrypoint_from_the_list(tmp_path: Path): + from codeanalyzer.schema.py_schema import PyCallable, PyEntrypoint, PyModule + + fn = PyCallable(name="f", path="a.py", signature="a.f") + fn.entrypoints.append( + PyEntrypoint(framework="flask", confidence="certain", rule="flask.route", ruleset="shipped") + ) + app = PyApplication(symbol_table={"a.py": PyModule(functions={"f": fn})}) + detect_entrypoints(app, tmp_path) + assert fn.is_entrypoint is True +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest test/test_entrypoint_pipeline.py -v --no-cov` +Expected: FAIL with `ModuleNotFoundError: No module named 'codeanalyzer.entrypoints'` + +- [ ] **Step 3: Write minimal implementation** + +`codeanalyzer/entrypoints/__init__.py`: + +```python +from codeanalyzer.entrypoints.pipeline import detect_entrypoints + +__all__ = ["detect_entrypoints"] +``` + +`codeanalyzer/entrypoints/pipeline.py`: + +```python +"""Entrypoint detection: a post-pass over the built L1 symbol table (#27). + +Runs AFTER the symbol table exists so every view reference resolves as a +lookup against ids that already exist. Additive metadata: a failure here +loses flags, never the analysis. +""" +from __future__ import annotations + +from pathlib import Path +from typing import Iterable, Iterator + +from codeanalyzer.schema.py_schema import PyApplication, PyCallable, PyClass +from codeanalyzer.utils import logger + + +def detect_entrypoints( + app: PyApplication, project_dir: Path, rule_paths: Iterable[Path] = () +) -> None: + """Populate ``entrypoints`` on every callable and class, in place.""" + try: + _run_stages(app, project_dir, tuple(rule_paths)) + except Exception as exc: # noqa: BLE001 - additive pass must never abort analysis + logger.warning("entrypoint detection failed: %s", exc) + app.entrypoint_report.errors.append(str(exc)) + _derive_flags(app) + + +def _run_stages(app: PyApplication, project_dir: Path, rule_paths: tuple) -> None: + """Stages 0-4. Empty until Task 5; the skeleton exists so the contract does.""" + return None + + +def _derive_flags(app: PyApplication) -> None: + for node in _walk(app): + node.is_entrypoint = bool(node.entrypoints) + + +def _walk(app: PyApplication) -> Iterator[object]: + def walk_callable(c: PyCallable) -> Iterator[object]: + yield c + for inner in (c.callables or {}).values(): + yield from walk_callable(inner) + for cls in (c.types or {}).values(): + yield from walk_class(cls) + + def walk_class(k: PyClass) -> Iterator[object]: + yield k + for m in (k.callables or {}).values(): + yield from walk_callable(m) + for inner in (k.types or {}).values(): + yield from walk_class(inner) + + for mod in app.symbol_table.values(): + for fn in (mod.functions or {}).values(): + yield from walk_callable(fn) + for cls in (mod.types or {}).values(): + yield from walk_class(cls) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest test/test_entrypoint_pipeline.py -v --no-cov` +Expected: 3 passed + +- [ ] **Step 5: Wire into the analyzer** + +In `codeanalyzer/core.py`, immediately after the line `reidentify_call_graph(app, sig_to_id)`: + +```python + # Entrypoints: a post-pass over the built L1 tree (#27). Runs at every + # level -- entrypoints are L1 data and must not vary with -a. + from codeanalyzer.entrypoints import detect_entrypoints + + detect_entrypoints(app, self.project_dir, self.options.entrypoint_rules) +``` + +In `codeanalyzer/options/options.py`, widen the typing import (it currently reads +`from typing import Optional`): + +```python +from typing import Optional, Tuple +``` + +then add to `AnalysisOptions`: + +```python + entrypoint_rules: Tuple[Path, ...] = () +``` + +- [ ] **Step 6: Run the broader suite to confirm nothing regressed** + +Run: `uv run pytest test/test_cli.py -q --no-cov` +Expected: all pass + +- [ ] **Step 7: Commit** + +```bash +git add codeanalyzer/entrypoints/ codeanalyzer/core.py codeanalyzer/options/options.py test/test_entrypoint_pipeline.py +git commit -m "feat(entrypoints): wrapped post-pass skeleton wired into the analyzer (#27)" +``` + +--- + +### Task 3: `rules.yml` — shipped file and loader + +**Files:** +- Create: `codeanalyzer/entrypoints/rules.yml` +- Create: `codeanalyzer/entrypoints/rules.py` +- Modify: `pyproject.toml` (declare `pyyaml`; add package data) +- Test: `test/test_entrypoint_rules.py` + +**Interfaces:** +- Consumes: nothing from earlier tasks +- Produces: `load_rules(user_paths: Iterable[Path] = ()) -> RuleSet`; `RuleSet` with `.frameworks: Dict[str, Framework]`, `.rulesets: List[str]`; `Framework` with `.detect: List[str]`, `.decorators: List[DecoratorRule]`, `.bases: List[BaseRule]`; `DecoratorRule` with `.id`, `.match`, `.confidence`, `.route`, `.methods`; `BaseRule` with `.id`, `.match`, `.confidence`, `.transitive`, `.dispatch`; `RulesError` exception + +**Note:** PyYAML is currently only a transitive dependency (via `ray`). It must be declared explicitly — relying on a transitive dep is exactly the failure #124 removed. + +- [ ] **Step 1: Write the failing test** + +```python +# test/test_entrypoint_rules.py +import pytest + +from codeanalyzer.entrypoints.rules import RulesError, load_rules + + +def test_shipped_rules_load_and_include_flask(): + rs = load_rules() + assert "flask" in rs.frameworks + flask = rs.frameworks["flask"] + assert "flask" in flask.detect + assert any(r.id == "flask.route" for r in flask.decorators) + + +def test_every_shipped_rule_has_a_stable_id_and_valid_confidence(): + rs = load_rules() + for fw in rs.frameworks.values(): + for rule in list(fw.decorators) + list(fw.bases): + assert rule.id, "every rule needs a stable id so users can disable it" + assert rule.confidence in {"declared", "certain", "heuristic"} + + +def test_malformed_user_file_raises_before_analysis(tmp_path): + bad = tmp_path / "bad.yml" + bad.write_text("frameworks: [this is a list not a mapping]\n") + with pytest.raises(RulesError): + load_rules([bad]) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest test/test_entrypoint_rules.py -v --no-cov` +Expected: FAIL with `ModuleNotFoundError: No module named 'codeanalyzer.entrypoints.rules'` + +- [ ] **Step 3: Write the shipped rules file** + +`codeanalyzer/entrypoints/rules.yml`: + +```yaml +version: 1 + +frameworks: + flask: + detect: [flask] + decorators: + - id: flask.route + match: "flask.Flask.route" + route: {from: positional, index: 0} + methods: {from: keyword, name: methods, default: [GET]} + - id: flask.bp-verb + match: "flask.Blueprint.{get,post,put,delete,patch}" + route: {from: positional, index: 0} + methods: {from: match_suffix} + bases: + - id: flask.methodview + match: "flask.views.MethodView" + transitive: true + dispatch: [get, post, put, delete, patch] + + fastapi: + detect: [fastapi] + decorators: + - id: fastapi.verb + match: "fastapi.FastAPI.{get,post,put,delete,patch,head,options}" + route: {from: positional, index: 0} + methods: {from: match_suffix} + - id: fastapi.router-verb + match: "fastapi.APIRouter.{get,post,put,delete,patch}" + route: {from: positional, index: 0} + methods: {from: match_suffix} + - id: fastapi.websocket + match: "fastapi.FastAPI.websocket" + route: {from: positional, index: 0} + + celery: + detect: [celery] + decorators: + - id: celery.shared-task + match: "celery.shared_task" + - id: celery.task + match: "celery.Celery.task" + + click: + detect: [click, typer] + decorators: + - id: click.command + match: "click.{command,group}" + - id: typer.command + match: "typer.Typer.command" + + drf: + detect: [rest_framework] + decorators: + - id: drf.api-view + match: "rest_framework.decorators.api_view" + - id: drf.action + match: "rest_framework.decorators.action" + bases: + - id: drf.apiview + match: "rest_framework.views.APIView" + transitive: true + dispatch: [get, post, put, patch, delete, head, options] + - id: drf.viewset + match: "rest_framework.viewsets.*" + transitive: true + dispatch: [list, retrieve, create, update, partial_update, destroy] +``` + +- [ ] **Step 4: Write minimal implementation** + +`codeanalyzer/entrypoints/rules.py`: + +```python +"""Loading and merging of entrypoint rules (#27). + +The shipped ``rules.yml`` covers known frameworks; users extend it with +``--entrypoint-rules``. User rules merge additively and may ``disable:`` a +shipped rule by id. A malformed user file is a hard error before analysis +starts -- silently skipping it would let someone ship rules they believe +are live. +""" +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, Iterable, List, Optional + +import yaml + +_SHIPPED = Path(__file__).with_name("rules.yml") +_CONFIDENCE = {"declared", "certain", "heuristic"} + + +class RulesError(Exception): + """Raised for a malformed rules file. Never swallowed.""" + + +@dataclass +class DecoratorRule: + id: str + match: str + confidence: str = "certain" + route: Optional[Dict[str, Any]] = None + methods: Optional[Dict[str, Any]] = None + + +@dataclass +class BaseRule: + id: str + match: str + confidence: str = "certain" + transitive: bool = False + dispatch: List[str] = field(default_factory=list) + + +@dataclass +class Framework: + name: str + detect: List[str] = field(default_factory=list) + decorators: List[DecoratorRule] = field(default_factory=list) + bases: List[BaseRule] = field(default_factory=list) + + +@dataclass +class RuleSet: + frameworks: Dict[str, Framework] = field(default_factory=dict) + rulesets: List[str] = field(default_factory=list) + + +def load_rules(user_paths: Iterable[Path] = ()) -> RuleSet: + out = RuleSet() + _merge(out, _read(_SHIPPED), "shipped") + for p in user_paths: + _merge(out, _read(Path(p)), f"user:{p}") + return out + + +def _read(path: Path) -> Dict[str, Any]: + try: + data = yaml.safe_load(path.read_text()) + except FileNotFoundError as exc: + raise RulesError(f"rules file not found: {path}") from exc + except yaml.YAMLError as exc: + raise RulesError(f"{path}: invalid YAML: {exc}") from exc + if not isinstance(data, dict): + raise RulesError(f"{path}: top level must be a mapping") + return data + + +def _merge(out: RuleSet, data: Dict[str, Any], origin: str) -> None: + out.rulesets.append(origin) + disabled = set(data.get("disable") or []) + frameworks = data.get("frameworks") or {} + if not isinstance(frameworks, dict): + raise RulesError(f"{origin}: `frameworks` must be a mapping") + + for name, body in frameworks.items(): + if not isinstance(body, dict): + raise RulesError(f"{origin}: framework `{name}` must be a mapping") + fw = out.frameworks.setdefault(name, Framework(name=name)) + fw.detect = sorted(set(fw.detect) | set(body.get("detect") or [])) + for raw in body.get("decorators") or []: + fw.decorators.append(_decorator_rule(raw, origin)) + for raw in body.get("bases") or []: + fw.bases.append(_base_rule(raw, origin)) + + for fw in out.frameworks.values(): + fw.decorators = [r for r in fw.decorators if r.id not in disabled] + fw.bases = [r for r in fw.bases if r.id not in disabled] + + +def _require(raw: Dict[str, Any], key: str, origin: str) -> Any: + if key not in raw: + raise RulesError(f"{origin}: rule {raw!r} is missing `{key}`") + return raw[key] + + +def _confidence(raw: Dict[str, Any], origin: str) -> str: + c = raw.get("confidence", "certain") + if c not in _CONFIDENCE: + raise RulesError(f"{origin}: confidence must be one of {sorted(_CONFIDENCE)}, got {c!r}") + return c + + +def _decorator_rule(raw: Dict[str, Any], origin: str) -> DecoratorRule: + return DecoratorRule( + id=_require(raw, "id", origin), + match=_require(raw, "match", origin), + confidence=_confidence(raw, origin), + route=raw.get("route"), + methods=raw.get("methods"), + ) + + +def _base_rule(raw: Dict[str, Any], origin: str) -> BaseRule: + return BaseRule( + id=_require(raw, "id", origin), + match=_require(raw, "match", origin), + confidence=_confidence(raw, origin), + transitive=bool(raw.get("transitive", False)), + dispatch=list(raw.get("dispatch") or []), + ) +``` + +- [ ] **Step 5: Declare the dependency and ship the data file** + +In `pyproject.toml`, add to `[project].dependencies`: + +```toml + # pyyaml: the entrypoint rules pack (#27) is YAML. Declared explicitly -- + # it was previously only reachable as a transitive dep of ray. + "pyyaml>=6.0,<7.0", +``` + +And ensure the data file ships — add after the `[project.optional-dependencies]` block: + +```toml +[tool.setuptools.package-data] +codeanalyzer = ["entrypoints/*.yml"] +``` + +- [ ] **Step 6: Run test to verify it passes** + +Run: `uv sync && uv run pytest test/test_entrypoint_rules.py -v --no-cov` +Expected: 3 passed + +- [ ] **Step 7: Commit** + +```bash +git add codeanalyzer/entrypoints/rules.py codeanalyzer/entrypoints/rules.yml pyproject.toml test/test_entrypoint_rules.py +git commit -m "feat(entrypoints): rules.yml loader with shipped framework pack (#27)" +``` + +--- + +### Task 4: User rules — merge, disable, CLI flag + +**Files:** +- Modify: `codeanalyzer/__main__.py` +- Test: `test/test_entrypoint_rules.py` (extend) + +**Interfaces:** +- Consumes: `load_rules` from Task 3, `AnalysisOptions.entrypoint_rules` from Task 2 +- Produces: `--entrypoint-rules` CLI flag, repeatable, populating `AnalysisOptions.entrypoint_rules` + +- [ ] **Step 1: Write the failing test** + +```python +# append to test/test_entrypoint_rules.py +def test_user_rules_merge_additively_with_shipped(tmp_path): + extra = tmp_path / "mine.yml" + extra.write_text( + "version: 1\n" + "frameworks:\n" + " inhouse:\n" + " detect: [inhouse]\n" + " decorators:\n" + " - id: inhouse.handler\n" + " match: 'inhouse.app.handler'\n" + ) + rs = load_rules([extra]) + assert "flask" in rs.frameworks # shipped survives + assert "inhouse" in rs.frameworks # user added + assert rs.rulesets == ["shipped", f"user:{extra}"] + + +def test_user_file_can_disable_a_shipped_rule(tmp_path): + off = tmp_path / "off.yml" + off.write_text("version: 1\ndisable: [flask.route]\n") + rs = load_rules([off]) + assert not any(r.id == "flask.route" for r in rs.frameworks["flask"].decorators) + assert any(r.id == "flask.bp-verb" for r in rs.frameworks["flask"].decorators) + + +def test_bad_confidence_value_is_rejected(tmp_path): + bad = tmp_path / "c.yml" + bad.write_text( + "version: 1\nframeworks:\n x:\n decorators:\n" + " - id: x.y\n match: 'x.y'\n confidence: probably\n" + ) + with pytest.raises(RulesError): + load_rules([bad]) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest test/test_entrypoint_rules.py -v --no-cov` +Expected: the three new tests FAIL (`rulesets` empty or disable unsupported) + +- [ ] **Step 3: Implementation** + +The `_merge` written in Task 3 already satisfies these. If any test fails, fix `_merge` — do not change the tests. + +- [ ] **Step 4: Add the CLI flag** + +In `codeanalyzer/__main__.py`, add a parameter alongside the existing options: + +```python + entrypoint_rules: Annotated[ + Optional[List[Path]], + typer.Option( + "--entrypoint-rules", + help="Extra entrypoint rules file (YAML). Repeatable; merges with " + "the shipped rules. A malformed file is an error.", + ), + ] = None, +``` + +and thread it into the `AnalysisOptions(...)` construction: + +```python + entrypoint_rules=tuple(entrypoint_rules or ()), +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `uv run pytest test/test_entrypoint_rules.py -v --no-cov && uv run canpy --help | grep entrypoint-rules` +Expected: all pass; the flag appears in help output + +- [ ] **Step 6: Commit** + +```bash +git add codeanalyzer/__main__.py test/test_entrypoint_rules.py +git commit -m "feat(cli): --entrypoint-rules for user rule packs (#27)" +``` + +--- + +### Task 5: Stage 0 — framework detection gate + +**Files:** +- Create: `codeanalyzer/entrypoints/detect.py` +- Modify: `codeanalyzer/entrypoints/pipeline.py` +- Test: `test/test_entrypoint_detect.py` + +**Interfaces:** +- Consumes: `RuleSet` from Task 3, `PyApplication` from Task 1 +- Produces: `detected_frameworks(app, project_dir, ruleset) -> Set[str]` + +- [ ] **Step 1: Write the failing test** + +```python +# test/test_entrypoint_detect.py +from pathlib import Path + +from codeanalyzer.entrypoints.detect import detected_frameworks +from codeanalyzer.entrypoints.rules import load_rules +from codeanalyzer.schema.py_schema import PyApplication, PyImport, PyModule + + +def _app(*modules: str) -> PyApplication: + return PyApplication( + symbol_table={ + "a.py": PyModule( + imports=[PyImport(module=m, name=m.split(".")[-1]) for m in modules] + ) + } + ) + + +def test_framework_detected_from_an_import(tmp_path: Path): + got = detected_frameworks(_app("flask"), tmp_path, load_rules()) + assert "flask" in got + + +def test_absent_framework_is_not_detected(tmp_path: Path): + got = detected_frameworks(_app("os"), tmp_path, load_rules()) + assert "celery" not in got + + +def test_manifest_entry_alone_is_sufficient(tmp_path: Path): + (tmp_path / "pyproject.toml").write_text( + '[project]\nname = "x"\ndependencies = ["celery>=5"]\n' + ) + got = detected_frameworks(_app("os"), tmp_path, load_rules()) + assert "celery" in got +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest test/test_entrypoint_detect.py -v --no-cov` +Expected: FAIL with `ModuleNotFoundError: No module named 'codeanalyzer.entrypoints.detect'` + +- [ ] **Step 3: Write minimal implementation** + +`codeanalyzer/entrypoints/detect.py`: + +```python +"""Stage 0: which frameworks is this project actually using? (#27) + +Gates every later stage, so a project without Celery never pays for Celery +rules and cannot false-positive on a locally-defined ``shared_task``. A +package counts as present if first-party source imports it OR the dependency +manifest names it -- either is sufficient, since an import may be dynamic. +""" +from __future__ import annotations + +import re +from pathlib import Path +from typing import Set + +from codeanalyzer.entrypoints.rules import RuleSet +from codeanalyzer.schema.py_schema import PyApplication + +_REQ = re.compile(r"^\s*['\"]?([A-Za-z0-9_.\-]+)") + + +def detected_frameworks(app: PyApplication, project_dir: Path, rules: RuleSet) -> Set[str]: + present = _imported_packages(app) | _manifest_packages(project_dir) + return { + name + for name, fw in rules.frameworks.items() + if any(pkg in present for pkg in (fw.detect or [name])) + } + + +def _imported_packages(app: PyApplication) -> Set[str]: + out: Set[str] = set() + for mod in app.symbol_table.values(): + for imp in mod.imports or []: + # `from flask import Flask` puts the package in `module`, not `name`. + # Prefer `module`; fall back to `name` for a bare `import flask`. + spelling = (getattr(imp, "module", "") or getattr(imp, "name", "") or "") + spelling = spelling.lstrip(".") + if spelling: + out.add(spelling.split(".", 1)[0]) + return out + + +def _manifest_packages(project_dir: Path) -> Set[str]: + out: Set[str] = set() + pyproject = project_dir / "pyproject.toml" + if pyproject.exists(): + for line in pyproject.read_text().splitlines(): + m = _REQ.match(line) + if m and "=" not in m.group(1): + out.add(m.group(1).split("[", 1)[0].lower()) + requirements = project_dir / "requirements.txt" + if requirements.exists(): + for line in requirements.read_text().splitlines(): + m = _REQ.match(line) + if m: + out.add(m.group(1).split("[", 1)[0].lower()) + return out +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest test/test_entrypoint_detect.py -v --no-cov` +Expected: 3 passed + +- [ ] **Step 5: Wire Stage 0 into the pipeline** + +Replace `_run_stages` in `codeanalyzer/entrypoints/pipeline.py`: + +```python +def _run_stages(app: PyApplication, project_dir: Path, rule_paths: tuple) -> None: + from codeanalyzer.entrypoints.detect import detected_frameworks + from codeanalyzer.entrypoints.rules import load_rules + + rules = load_rules(rule_paths) + app.entrypoint_report.rulesets = list(rules.rulesets) + frameworks = detected_frameworks(app, project_dir, rules) + app.entrypoint_report.frameworks_detected = sorted(frameworks) +``` + +- [ ] **Step 6: Run tests to verify they pass** + +Run: `uv run pytest test/test_entrypoint_pipeline.py test/test_entrypoint_detect.py -v --no-cov` +Expected: all pass + +- [ ] **Step 7: Commit** + +```bash +git add codeanalyzer/entrypoints/detect.py codeanalyzer/entrypoints/pipeline.py test/test_entrypoint_detect.py +git commit -m "feat(entrypoints): stage 0 framework detection gate (#27)" +``` + +--- + +### Task 6: Stage 3a — decorator matching + +**Files:** +- Create: `codeanalyzer/entrypoints/matching.py` +- Modify: `codeanalyzer/entrypoints/pipeline.py` +- Test: `test/test_entrypoint_decorators.py` + +**Interfaces:** +- Consumes: `DecoratorRule` from Task 3, `PyDecorator` (already exists from #128) +- Produces: `match_pattern(pattern: str, qualified_name: str) -> bool`; `entrypoints_from_decorators(node, framework, rules, ruleset_of) -> List[PyEntrypoint]` + +- [ ] **Step 1: Write the failing test** + +```python +# test/test_entrypoint_decorators.py +from codeanalyzer.entrypoints.matching import entrypoints_from_decorators, match_pattern +from codeanalyzer.entrypoints.rules import DecoratorRule +from codeanalyzer.schema.py_schema import PyCallable, PyDecorator + + +def test_brace_alternation_and_wildcard(): + assert match_pattern("flask.Blueprint.{get,post}", "flask.Blueprint.get") + assert not match_pattern("flask.Blueprint.{get,post}", "flask.Blueprint.delete") + assert match_pattern("rest_framework.viewsets.*", "rest_framework.viewsets.ModelViewSet") + assert not match_pattern("flask.Flask.route", "flask.Flask.routes") + + +def test_route_and_methods_are_extracted(): + fn = PyCallable(name="h", path="a.py", signature="a.h") + fn.decorators.append( + PyDecorator( + name="app.route", + qualified_name="flask.Flask.route", + positional_arguments=["'/products'"], + keyword_arguments={"methods": "['POST']"}, + ) + ) + rule = DecoratorRule( + id="flask.route", + match="flask.Flask.route", + route={"from": "positional", "index": 0}, + methods={"from": "keyword", "name": "methods", "default": ["GET"]}, + ) + (ep,) = entrypoints_from_decorators(fn, "flask", [rule], "shipped") + assert ep.route == "/products" + assert ep.http_methods == ["POST"] + assert ep.rule == "flask.route" and ep.ruleset == "shipped" + + +def test_verb_comes_from_the_matched_suffix(): + fn = PyCallable(name="h", path="a.py", signature="a.h") + fn.decorators.append( + PyDecorator(name="router.post", qualified_name="fastapi.APIRouter.post", + positional_arguments=["'/x'"]) + ) + rule = DecoratorRule( + id="fastapi.router-verb", + match="fastapi.APIRouter.{get,post}", + route={"from": "positional", "index": 0}, + methods={"from": "match_suffix"}, + ) + (ep,) = entrypoints_from_decorators(fn, "fastapi", [rule], "shipped") + assert ep.http_methods == ["POST"] + + +def test_unresolved_decorator_never_matches(): + """qualified_name is None when Jedi could not resolve; must not guess.""" + fn = PyCallable(name="h", path="a.py", signature="a.h") + fn.decorators.append(PyDecorator(name="app.route", qualified_name=None)) + rule = DecoratorRule(id="flask.route", match="flask.Flask.route") + assert entrypoints_from_decorators(fn, "flask", [rule], "shipped") == [] +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest test/test_entrypoint_decorators.py -v --no-cov` +Expected: FAIL with `ModuleNotFoundError: No module named 'codeanalyzer.entrypoints.matching'` + +- [ ] **Step 3: Write minimal implementation** + +`codeanalyzer/entrypoints/matching.py`: + +```python +"""Stage 3: match rules against decorators and base classes (#27). + +Matching is on ``PyDecorator.qualified_name`` -- never the written spelling -- +so ``@route`` under ``from flask import route`` hits the same rule as +``@app.route``. An unresolved decorator (``qualified_name is None``) never +matches: under-approximate rather than guess. +""" +from __future__ import annotations + +import ast +import re +from typing import Any, Dict, Iterable, List, Optional + +from codeanalyzer.entrypoints.rules import DecoratorRule +from codeanalyzer.schema.py_schema import PyEntrypoint + +# Dispatch names that are HTTP verbs. DRF's ViewSet dispatch names +# (list, retrieve, create, ...) are NOT verbs and must not be emitted as such. +_HTTP_VERBS = {"get", "post", "put", "patch", "delete", "head", "options"} + + +def match_pattern(pattern: str, qualified_name: Optional[str]) -> bool: + """``{a,b}`` alternation and trailing ``*``; everything else is literal.""" + if not qualified_name: + return False + return re.fullmatch(_compile(pattern), qualified_name) is not None + + +def _compile(pattern: str) -> str: + out, i = [], 0 + while i < len(pattern): + ch = pattern[i] + if ch == "{": + j = pattern.index("}", i) + alts = pattern[i + 1 : j].split(",") + out.append("(?:" + "|".join(re.escape(a.strip()) for a in alts) + ")") + i = j + 1 + elif ch == "*": + out.append(r"[^\s]*") + i += 1 + else: + out.append(re.escape(ch)) + i += 1 + return "".join(out) + + +def _literal(text: Optional[str]) -> Any: + """Best-effort: decorator arguments are unparsed source fragments.""" + if text is None: + return None + try: + return ast.literal_eval(text) + except (ValueError, SyntaxError): + return None + + +def _route_of(dec, spec: Optional[Dict[str, Any]]) -> Optional[str]: + if not spec or spec.get("from") != "positional": + return None + args = dec.positional_arguments or [] + idx = int(spec.get("index", 0)) + if idx >= len(args): + return None + value = _literal(args[idx]) + return value if isinstance(value, str) else None + + +def _methods_of(dec, rule: DecoratorRule, spec: Optional[Dict[str, Any]]) -> List[str]: + if not spec: + return [] + source = spec.get("from") + if source == "match_suffix": + verb = (dec.qualified_name or "").rsplit(".", 1)[-1] + return [verb.upper()] + if source == "keyword": + raw = (dec.keyword_arguments or {}).get(spec.get("name", "")) + value = _literal(raw) + if isinstance(value, (list, tuple)): + return [str(v).upper() for v in value] + return [str(v).upper() for v in (spec.get("default") or [])] + return [] + + +def entrypoints_from_decorators( + node, framework: str, rules: Iterable[DecoratorRule], ruleset: str +) -> List[PyEntrypoint]: + out: List[PyEntrypoint] = [] + for dec in getattr(node, "decorators", []) or []: + for rule in rules: + if not match_pattern(rule.match, dec.qualified_name): + continue + out.append( + PyEntrypoint( + framework=framework, + confidence=rule.confidence, + rule=rule.id, + ruleset=ruleset, + evidence=dec.qualified_name, + route=_route_of(dec, rule.route), + http_methods=_methods_of(dec, rule, rule.methods), + ) + ) + return out +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest test/test_entrypoint_decorators.py -v --no-cov` +Expected: 4 passed + +- [ ] **Step 5: Commit** + +```bash +git add codeanalyzer/entrypoints/matching.py test/test_entrypoint_decorators.py +git commit -m "feat(entrypoints): decorator rule matching with route extraction (#27)" +``` + +--- + +### Task 7: Stage 3b — inheritance matching and the dispatch split + +**Files:** +- Modify: `codeanalyzer/entrypoints/matching.py` +- Modify: `codeanalyzer/entrypoints/pipeline.py` +- Test: `test/test_entrypoint_bases.py` + +**Interfaces:** +- Consumes: `BaseRule` from Task 3, `match_pattern` from Task 6 +- Produces: `entrypoints_from_bases(cls, framework, rules, ruleset, resolve) -> Tuple[List[PyEntrypoint], Dict[str, List[PyEntrypoint]]]` — class records, and per-method-name records for the dispatch split + +- [ ] **Step 1: Write the failing test** + +```python +# test/test_entrypoint_bases.py +from codeanalyzer.entrypoints.matching import entrypoints_from_bases +from codeanalyzer.entrypoints.rules import BaseRule +from codeanalyzer.schema.py_schema import PyCallable, PyClass + +RULE = BaseRule( + id="drf.apiview", + match="rest_framework.views.APIView", + transitive=True, + dispatch=["get", "post", "put"], +) + + +def _cls(*methods: str, bases=("rest_framework.views.APIView",)) -> PyClass: + return PyClass( + name="V", + signature="a.V", + base_classes=list(bases), + callables={m: PyCallable(name=m, path="a.py", signature=f"a.V.{m}") for m in methods}, + ) + + +def test_class_is_flagged_and_only_defined_methods_dispatch(): + cls = _cls("get") # defines get, not post + class_eps, method_eps = entrypoints_from_bases(cls, "drf", [RULE], "shipped", lambda b: b) + assert len(class_eps) == 1 + assert list(method_eps) == ["get"], "no phantom post entrypoint" + + +def test_methods_point_back_at_the_routed_class_via(): + cls = _cls("get") + cls.id = "can://python/app/a.py/V" + _, method_eps = entrypoints_from_bases(cls, "drf", [RULE], "shipped", lambda b: b) + assert method_eps["get"][0].via == "can://python/app/a.py/V" + + +def test_transitive_base_resolves_one_hop(): + cls = _cls("get", bases=("app.BaseView",)) + resolve = {"app.BaseView": "rest_framework.views.APIView"}.get + class_eps, _ = entrypoints_from_bases(cls, "drf", [RULE], "shipped", resolve) + assert len(class_eps) == 1 + + +def test_unrelated_class_is_not_flagged(): + cls = _cls("get", bases=("object",)) + class_eps, method_eps = entrypoints_from_bases(cls, "drf", [RULE], "shipped", lambda b: b) + assert class_eps == [] and method_eps == {} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest test/test_entrypoint_bases.py -v --no-cov` +Expected: FAIL with `ImportError: cannot import name 'entrypoints_from_bases'` + +- [ ] **Step 3: Write minimal implementation** + +Append to `codeanalyzer/entrypoints/matching.py`: + +```python +def entrypoints_from_bases( + cls, framework: str, rules, ruleset: str, resolve +): + """Records for a routed class and for the methods the framework dispatches. + + ``resolve`` maps a written base-class name to its resolved qualified name + (identity when already qualified). Dispatch names are intersected with the + methods the class actually defines, so a ``ListView`` with only ``get`` + gains no phantom ``post`` entrypoint. + """ + class_eps: List[PyEntrypoint] = [] + method_eps: Dict[str, List[PyEntrypoint]] = {} + + for rule in rules: + if not any( + match_pattern(rule.match, resolve(b) or b) for b in (cls.base_classes or []) + ): + continue + class_eps.append( + PyEntrypoint( + framework=framework, + confidence=rule.confidence, + rule=rule.id, + ruleset=ruleset, + evidence=cls.signature, + ) + ) + defined = set((cls.callables or {}).keys()) + for name in rule.dispatch: + if name not in defined: + continue + method_eps.setdefault(name, []).append( + PyEntrypoint( + framework=framework, + confidence=rule.confidence, + rule=f"{rule.id}.dispatch", + ruleset=ruleset, + evidence=cls.signature, + http_methods=[name.upper()] if name in _HTTP_VERBS else [], + via=cls.id or None, + ) + ) + return class_eps, method_eps +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest test/test_entrypoint_bases.py -v --no-cov` +Expected: 4 passed + +- [ ] **Step 5: Wire Stage 3 into the pipeline** + +Extend `_run_stages` in `codeanalyzer/entrypoints/pipeline.py`, after Stage 0: + +```python + from codeanalyzer.entrypoints.matching import ( + entrypoints_from_bases, + entrypoints_from_decorators, + ) + + ruleset_name = rules.rulesets[-1] if len(rules.rulesets) > 1 else "shipped" + for name in sorted(frameworks): + fw = rules.frameworks[name] + for node in _walk(app): + node.entrypoints.extend( + entrypoints_from_decorators(node, name, fw.decorators, ruleset_name) + ) + if isinstance(node, PyClass) and fw.bases: + class_eps, method_eps = entrypoints_from_bases( + node, name, fw.bases, ruleset_name, lambda b: b + ) + node.entrypoints.extend(class_eps) + for method_name, eps in method_eps.items(): + target = (node.callables or {}).get(method_name) + if target is not None: + target.entrypoints.extend(eps) +``` + +- [ ] **Step 6: Run the whole entrypoint suite** + +Run: `uv run pytest test/test_entrypoint_*.py -v --no-cov` +Expected: all pass + +- [ ] **Step 7: Commit** + +```bash +git add codeanalyzer/entrypoints/matching.py codeanalyzer/entrypoints/pipeline.py test/test_entrypoint_bases.py +git commit -m "feat(entrypoints): inheritance rules and the class/method dispatch split (#27)" +``` + +--- + +### Task 8: End-to-end fixture and Neo4j projection + +**Files:** +- Create: `test/fixtures/single_functionalities/entrypoints_flask/app.py` +- Create: `test/test_entrypoints_e2e.py` +- Modify: `codeanalyzer/neo4j/schema.py`, `codeanalyzer/neo4j/project.py` +- Modify: `schema.neo4j.json` (regenerated) + +**Interfaces:** +- Consumes: everything above +- Produces: `is_entrypoint` and `entrypoint_frameworks` properties on `:PyCallable` and `:PyClass` + +- [ ] **Step 1: Write the fixture** + +```python +# test/fixtures/single_functionalities/entrypoints_flask/app.py +from flask import Flask + +app = Flask(__name__) + + +@app.route("/products", methods=["POST"]) +def create_product(): + return helper() + + +def helper(): + """Called only internally - must NOT be flagged.""" + return {} +``` + +- [ ] **Step 2: Write the failing test** + +```python +# test/test_entrypoints_e2e.py +import json +import subprocess +from pathlib import Path + +FIXTURE = Path(__file__).parent / "fixtures" / "single_functionalities" / "entrypoints_flask" + + +def test_flask_route_flagged_and_helper_not(tmp_path): + subprocess.run( + ["uv", "run", "canpy", "-i", str(FIXTURE), "-a", "1", "-o", str(tmp_path)], + check=True, + ) + data = json.loads((tmp_path / "analysis.json").read_text()) + fns = data["application"]["symbol_table"]["app.py"]["functions"] + + create = fns["create_product"] + assert create["is_entrypoint"] is True + (ep,) = create["entrypoints"] + assert ep["framework"] == "flask" and ep["rule"] == "flask.route" + assert ep["route"] == "/products" and ep["http_methods"] == ["POST"] + + assert fns["helper"]["is_entrypoint"] is False + assert fns["helper"]["entrypoints"] == [] + assert "flask" in data["application"]["entrypoint_report"]["frameworks_detected"] +``` + +- [ ] **Step 3: Run test to verify it fails** + +Run: `uv run pytest test/test_entrypoints_e2e.py -v --no-cov` +Expected: FAIL — `is_entrypoint` is False (Flask is not installed in the fixture, so Stage 0 does not gate in) + +- [ ] **Step 4: Make Stage 0 pass for the fixture** + +Add a manifest so detection has a source. Create `test/fixtures/single_functionalities/entrypoints_flask/requirements.txt`: + +``` +flask +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `uv run pytest test/test_entrypoints_e2e.py -v --no-cov` +Expected: PASS + +- [ ] **Step 6: Project into Neo4j** + +In `codeanalyzer/neo4j/schema.py`, add to both the `PyCallable` and `PyClass` property maps: + +```python + "is_entrypoint": "boolean", + "entrypoint_frameworks": "string[]", +``` + +In `codeanalyzer/neo4j/project.py`, add to both `_callable_props` and `_class_props`: + +```python + "is_entrypoint": bool(node.entrypoints), + "entrypoint_frameworks": sorted({e.framework for e in (node.entrypoints or [])}), +``` + +(using the local parameter name in each function — `c` in `_callable_props`, `cl` in `_class_props`). + +- [ ] **Step 7: Regenerate the schema snapshot and run the suite** + +```bash +uv run canpy --emit schema > schema.neo4j.json +uv run pytest test/ -q --no-cov -k "not neo4j_bolt" +``` +Expected: all pass + +- [ ] **Step 8: Commit** + +```bash +git add test/fixtures/single_functionalities/entrypoints_flask test/test_entrypoints_e2e.py \ + codeanalyzer/neo4j/schema.py codeanalyzer/neo4j/project.py schema.neo4j.json +git commit -m "feat(entrypoints): end-to-end flask detection and Neo4j projection (#27)" +``` From 25086c3d0296e74d22f820834ad86f51fb178879 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:02:13 -0400 Subject: [PATCH 04/23] docs(plan): pre-flight scan fixes for the entrypoint plan (#27) PyModule requires file_path and module_name, so the Task 2 and Task 5 tests would have failed validation; Task 6 claimed to modify pipeline.py but no step did, and Task 7 owns that wiring; and Task 8's Flask fixture would only resolve if the analyzer downloaded Flask into an analysis venv, making a unit test network-dependent. Task 8 now drives detection through a locally-defined decorator and a user rules file, which is also the only end-to-end coverage of --entrypoint-rules. --- ...26-08-20-entrypoint-detection-units-1-3.md | 102 +++++++++++++----- 1 file changed, 73 insertions(+), 29 deletions(-) diff --git a/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md b/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md index e668fd6..74d4539 100644 --- a/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md +++ b/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md @@ -183,7 +183,11 @@ def test_derives_is_entrypoint_from_the_list(tmp_path: Path): fn.entrypoints.append( PyEntrypoint(framework="flask", confidence="certain", rule="flask.route", ruleset="shipped") ) - app = PyApplication(symbol_table={"a.py": PyModule(functions={"f": fn})}) + app = PyApplication( + symbol_table={ + "a.py": PyModule(file_path="a.py", module_name="a", functions={"f": fn}) + } + ) detect_entrypoints(app, tmp_path) assert fn.is_entrypoint is True ``` @@ -720,7 +724,9 @@ def _app(*modules: str) -> PyApplication: return PyApplication( symbol_table={ "a.py": PyModule( - imports=[PyImport(module=m, name=m.split(".")[-1]) for m in modules] + file_path="a.py", + module_name="a", + imports=[PyImport(module=m, name=m.split(".")[-1]) for m in modules], ) } ) @@ -850,9 +856,10 @@ git commit -m "feat(entrypoints): stage 0 framework detection gate (#27)" **Files:** - Create: `codeanalyzer/entrypoints/matching.py` -- Modify: `codeanalyzer/entrypoints/pipeline.py` - Test: `test/test_entrypoint_decorators.py` +(Task 7 owns the Stage 3 wiring into `pipeline.py`; do not touch it here.) + **Interfaces:** - Consumes: `DecoratorRule` from Task 3, `PyDecorator` (already exists from #128) - Produces: `match_pattern(pattern: str, qualified_name: str) -> bool`; `entrypoints_from_decorators(node, framework, rules, ruleset_of) -> List[PyEntrypoint]` @@ -1213,10 +1220,11 @@ git commit -m "feat(entrypoints): inheritance rules and the class/method dispatc --- -### Task 8: End-to-end fixture and Neo4j projection +### Task 8: End-to-end detection through a user rules file **Files:** -- Create: `test/fixtures/single_functionalities/entrypoints_flask/app.py` +- Create: `test/fixtures/single_functionalities/entrypoints_local/app.py` +- Create: `test/fixtures/single_functionalities/entrypoints_local/rules.yml` - Create: `test/test_entrypoints_e2e.py` - Modify: `codeanalyzer/neo4j/schema.py`, `codeanalyzer/neo4j/project.py` - Modify: `schema.neo4j.json` (regenerated) @@ -1225,16 +1233,24 @@ git commit -m "feat(entrypoints): inheritance rules and the class/method dispatc - Consumes: everything above - Produces: `is_entrypoint` and `entrypoint_frameworks` properties on `:PyCallable` and `:PyClass` +**Why a local decorator rather than Flask:** matching needs `qualified_name`, which Jedi +only resolves for an installed package. Using Flask would make this test depend on the +analyzer building a venv and downloading Flask — slow and network-dependent. A decorator +defined in the fixture itself resolves locally and deterministically, and this is also the +only end-to-end coverage of the `--entrypoint-rules` path. + - [ ] **Step 1: Write the fixture** ```python -# test/fixtures/single_functionalities/entrypoints_flask/app.py -from flask import Flask +# test/fixtures/single_functionalities/entrypoints_local/app.py +def route(path, methods=None): + """Stands in for a framework's routing decorator.""" + def deco(fn): + return fn + return deco -app = Flask(__name__) - -@app.route("/products", methods=["POST"]) +@route("/products", methods=["POST"]) def create_product(): return helper() @@ -1244,6 +1260,19 @@ def helper(): return {} ``` +```yaml +# test/fixtures/single_functionalities/entrypoints_local/rules.yml +version: 1 +frameworks: + inhouse: + detect: [app] + decorators: + - id: inhouse.route + match: "app.route" + route: {from: positional, index: 0} + methods: {from: keyword, name: methods, default: [GET]} +``` + - [ ] **Step 2: Write the failing test** ```python @@ -1252,12 +1281,18 @@ import json import subprocess from pathlib import Path -FIXTURE = Path(__file__).parent / "fixtures" / "single_functionalities" / "entrypoints_flask" +FIXTURE = Path(__file__).parent / "fixtures" / "single_functionalities" / "entrypoints_local" -def test_flask_route_flagged_and_helper_not(tmp_path): +def test_decorated_function_flagged_and_helper_not(tmp_path): subprocess.run( - ["uv", "run", "canpy", "-i", str(FIXTURE), "-a", "1", "-o", str(tmp_path)], + [ + "uv", "run", "canpy", + "-i", str(FIXTURE), + "-a", "1", + "-o", str(tmp_path), + "--entrypoint-rules", str(FIXTURE / "rules.yml"), + ], check=True, ) data = json.loads((tmp_path / "analysis.json").read_text()) @@ -1266,26 +1301,30 @@ def test_flask_route_flagged_and_helper_not(tmp_path): create = fns["create_product"] assert create["is_entrypoint"] is True (ep,) = create["entrypoints"] - assert ep["framework"] == "flask" and ep["rule"] == "flask.route" + assert ep["framework"] == "inhouse" and ep["rule"] == "inhouse.route" assert ep["route"] == "/products" and ep["http_methods"] == ["POST"] + assert ep["ruleset"].startswith("user:") assert fns["helper"]["is_entrypoint"] is False assert fns["helper"]["entrypoints"] == [] - assert "flask" in data["application"]["entrypoint_report"]["frameworks_detected"] + + report = data["application"]["entrypoint_report"] + assert "inhouse" in report["frameworks_detected"] + assert report["errors"] == [] ``` - [ ] **Step 3: Run test to verify it fails** Run: `uv run pytest test/test_entrypoints_e2e.py -v --no-cov` -Expected: FAIL — `is_entrypoint` is False (Flask is not installed in the fixture, so Stage 0 does not gate in) +Expected: FAIL — `is_entrypoint` is False, or KeyError on `entrypoint_report` -- [ ] **Step 4: Make Stage 0 pass for the fixture** +- [ ] **Step 4: Make it pass** -Add a manifest so detection has a source. Create `test/fixtures/single_functionalities/entrypoints_flask/requirements.txt`: - -``` -flask -``` +No new production code should be needed — Tasks 1-7 cover it. If the test fails, +diagnose against those tasks rather than adding code here. The one likely gap is +Stage 0 detection: `detect: [app]` must match the fixture's own module name via the +import scan, so confirm `detected_frameworks` sees it; if it does not, extend +`_manifest_packages` to treat first-party module names as present. - [ ] **Step 5: Run test to verify it passes** @@ -1294,21 +1333,26 @@ Expected: PASS - [ ] **Step 6: Project into Neo4j** -In `codeanalyzer/neo4j/schema.py`, add to both the `PyCallable` and `PyClass` property maps: +In `codeanalyzer/neo4j/schema.py`, add to BOTH the `PyCallable` and `PyClass` property maps: ```python "is_entrypoint": "boolean", "entrypoint_frameworks": "string[]", ``` -In `codeanalyzer/neo4j/project.py`, add to both `_callable_props` and `_class_props`: +In `codeanalyzer/neo4j/project.py`, add to `_callable_props` (local name `c`): ```python - "is_entrypoint": bool(node.entrypoints), - "entrypoint_frameworks": sorted({e.framework for e in (node.entrypoints or [])}), + "is_entrypoint": bool(c.entrypoints), + "entrypoint_frameworks": sorted({e.framework for e in (c.entrypoints or [])}), ``` -(using the local parameter name in each function — `c` in `_callable_props`, `cl` in `_class_props`). +and to `_class_props` (local name `cl`): + +```python + "is_entrypoint": bool(cl.entrypoints), + "entrypoint_frameworks": sorted({e.framework for e in (cl.entrypoints or [])}), +``` - [ ] **Step 7: Regenerate the schema snapshot and run the suite** @@ -1321,7 +1365,7 @@ Expected: all pass - [ ] **Step 8: Commit** ```bash -git add test/fixtures/single_functionalities/entrypoints_flask test/test_entrypoints_e2e.py \ +git add test/fixtures/single_functionalities/entrypoints_local test/test_entrypoints_e2e.py \ codeanalyzer/neo4j/schema.py codeanalyzer/neo4j/project.py schema.neo4j.json -git commit -m "feat(entrypoints): end-to-end flask detection and Neo4j projection (#27)" +git commit -m "feat(entrypoints): end-to-end detection and Neo4j projection (#27)" ``` From 4497b00fc06b4f791b8b5f39f69db62574eb97b4 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:03:42 -0400 Subject: [PATCH 05/23] feat(schema): PyEntrypoint records on callables and classes (#27) --- codeanalyzer/schema/py_schema.py | 41 ++++++++++++++++++++++++++++++++ test/test_entrypoint_schema.py | 22 +++++++++++++++++ 2 files changed, 63 insertions(+) create mode 100644 test/test_entrypoint_schema.py diff --git a/codeanalyzer/schema/py_schema.py b/codeanalyzer/schema/py_schema.py index fc22f6d..90e5a56 100644 --- a/codeanalyzer/schema/py_schema.py +++ b/codeanalyzer/schema/py_schema.py @@ -235,6 +235,41 @@ class PyDecorator(BaseModel): span: Optional[Span] = None +@builder +class PyEntrypoint(BaseModel): + """One way a callable or class is invoked from outside the application (#27). + + A node may hold several: two ``@app.route`` decorators, or a function that + is both a Celery task and a CLI command. ``confidence`` lets a consumer + threshold on evidence quality rather than inheriting this analyzer's + judgement. + """ + + framework: str + confidence: str = "certain" # "declared" | "certain" | "heuristic" + rule: str = "" # rules.yml `id:`, or an engine name + ruleset: str = "shipped" # "shipped" | "user:" + evidence: Optional[str] = None + route: Optional[str] = None + http_methods: List[str] = [] + via: Optional[str] = None # can:// id of the routed node dispatching here + + +@builder +class PyEntrypointReport(BaseModel): + """Coverage and failure record for the entrypoint pass (#27). + + The pass under-approximates by design, so silence is its failure mode. + This is what makes a gap visible instead of indistinguishable from + "this project has no entrypoints". + """ + + frameworks_detected: List[str] = [] + rulesets: List[str] = [] + unresolved: Dict[str, int] = {} + errors: List[str] = [] + + @builder class PyCallableParameter(BaseModel): """Represents a parameter of a Python callable (function/method).""" @@ -291,6 +326,8 @@ class PyCallable(BaseModel): span: Optional[Span] = None comments: List[PyComment] = [] decorators: List[PyDecorator] = [] + entrypoints: List[PyEntrypoint] = [] + is_entrypoint: bool = False parameters: List[PyCallableParameter] = [] return_type: Optional[str] = None start_line: int = -1 @@ -340,6 +377,8 @@ class PyClass(BaseModel): comments: List[PyComment] = [] base_classes: List[str] = [] decorators: List[PyDecorator] = [] + entrypoints: List[PyEntrypoint] = [] + is_entrypoint: bool = False callables: Dict[str, PyCallable] = {} # methods, keystone containment name attributes: Dict[str, PyClassAttribute] = {} types: Dict[str, "PyClass"] = {} # inner classes, keystone containment name @@ -432,6 +471,8 @@ class PyApplication(BaseModel): # builtin members), keyed by signature. Populated by the analyzer so every # backend (JSON and Neo4j) shares one authoritative external-symbol set. external_symbols: Dict[str, PyExternalSymbol] = {} + # Coverage/failure record for the entrypoint pass; see PyEntrypointReport (#27). + entrypoint_report: PyEntrypointReport = PyEntrypointReport() # Git provenance of the analyzed checkout, captured at analysis time. repository: Optional[PyRepositoryInfo] = None # Interprocedural parameter-passing edges (formal↔actual); populated at L4. diff --git a/test/test_entrypoint_schema.py b/test/test_entrypoint_schema.py new file mode 100644 index 0000000..6c41c72 --- /dev/null +++ b/test/test_entrypoint_schema.py @@ -0,0 +1,22 @@ +from codeanalyzer.schema.py_schema import ( + PyApplication, PyCallable, PyClass, PyEntrypoint, PyEntrypointReport, +) + + +def test_entrypoint_record_defaults(): + e = PyEntrypoint(framework="flask", confidence="certain", rule="flask.route", ruleset="shipped") + assert e.evidence is None and e.route is None and e.via is None + assert e.http_methods == [] + + +def test_callable_and_class_carry_entrypoints(): + c = PyCallable(name="f", path="a.py", signature="a.f") + k = PyClass(name="C", signature="a.C") + assert c.entrypoints == [] and c.is_entrypoint is False + assert k.entrypoints == [] and k.is_entrypoint is False + + +def test_application_carries_a_report(): + app = PyApplication(symbol_table={}) + assert isinstance(app.entrypoint_report, PyEntrypointReport) + assert app.entrypoint_report.frameworks_detected == [] From 011b6ada30a40031018d262802cdbbe4c07093a2 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:32:15 -0400 Subject: [PATCH 06/23] docs(plan): replace the too-slow regression gate in task 2 (#27) test/test_cli.py drives full CLI analyses and exceeds ten minutes, which is the wrong check for a task-level gate. A direct analysis over a small fixture proves the same wiring in seconds. --- .../2026-08-20-entrypoint-detection-units-1-3.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md b/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md index 74d4539..04563e9 100644 --- a/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md +++ b/docs/design/plans/2026-08-20-entrypoint-detection-units-1-3.md @@ -299,10 +299,17 @@ then add to `AnalysisOptions`: entrypoint_rules: Tuple[Path, ...] = () ``` -- [ ] **Step 6: Run the broader suite to confirm nothing regressed** +- [ ] **Step 6: Confirm nothing regressed** -Run: `uv run pytest test/test_cli.py -q --no-cov` -Expected: all pass +Do NOT run `test/test_cli.py` — it drives full CLI analyses and exceeds 10 minutes. Run a +direct analysis over a small fixture instead: + +```bash +uv run canpy -i test/fixtures/single_functionalities/decorators_and_hof -a 1 -o /tmp/t2check +python3 -c "import json; a=json.load(open('/tmp/t2check/analysis.json'))['application']; \ +print('report:', a['entrypoint_report'])" +``` +Expected: analysis completes; `entrypoint_report` is present with empty lists and no errors. - [ ] **Step 7: Commit** From 715bf6663ba07a3dd387ec346bbee837fd8b26bc Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:32:30 -0400 Subject: [PATCH 07/23] feat(entrypoints): wrapped post-pass skeleton wired into the analyzer (#27) --- codeanalyzer/core.py | 6 +++ codeanalyzer/entrypoints/__init__.py | 3 ++ codeanalyzer/entrypoints/pipeline.py | 57 ++++++++++++++++++++++++++++ codeanalyzer/options/options.py | 3 +- test/test_entrypoint_pipeline.py | 39 +++++++++++++++++++ 5 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 codeanalyzer/entrypoints/__init__.py create mode 100644 codeanalyzer/entrypoints/pipeline.py create mode 100644 test/test_entrypoint_pipeline.py diff --git a/codeanalyzer/core.py b/codeanalyzer/core.py index 05b464d..c6550e9 100644 --- a/codeanalyzer/core.py +++ b/codeanalyzer/core.py @@ -636,6 +636,12 @@ def analyze(self) -> Analysis: backfill_callees(app, sig_to_id) reidentify_call_graph(app, sig_to_id) + # Entrypoints: a post-pass over the built L1 tree (#27). Runs at every + # level -- entrypoints are L1 data and must not vary with -a. + from codeanalyzer.entrypoints import detect_entrypoints + + detect_entrypoints(app, self.project_dir, self.options.entrypoint_rules) + # L3: intraprocedural dataflow (CFG/CDG/DDG) emitted onto the v2 tree. if self.analysis_level >= 3: from codeanalyzer.dataflow.builder import ( diff --git a/codeanalyzer/entrypoints/__init__.py b/codeanalyzer/entrypoints/__init__.py new file mode 100644 index 0000000..4049493 --- /dev/null +++ b/codeanalyzer/entrypoints/__init__.py @@ -0,0 +1,3 @@ +from codeanalyzer.entrypoints.pipeline import detect_entrypoints + +__all__ = ["detect_entrypoints"] diff --git a/codeanalyzer/entrypoints/pipeline.py b/codeanalyzer/entrypoints/pipeline.py new file mode 100644 index 0000000..12e171e --- /dev/null +++ b/codeanalyzer/entrypoints/pipeline.py @@ -0,0 +1,57 @@ +"""Entrypoint detection: a post-pass over the built L1 symbol table (#27). + +Runs AFTER the symbol table exists so every view reference resolves as a +lookup against ids that already exist. Additive metadata: a failure here +loses flags, never the analysis. +""" +from __future__ import annotations + +from pathlib import Path +from typing import Iterable, Iterator + +from codeanalyzer.schema.py_schema import PyApplication, PyCallable, PyClass +from codeanalyzer.utils import logger + + +def detect_entrypoints( + app: PyApplication, project_dir: Path, rule_paths: Iterable[Path] = () +) -> None: + """Populate ``entrypoints`` on every callable and class, in place.""" + try: + _run_stages(app, project_dir, tuple(rule_paths)) + except Exception as exc: # noqa: BLE001 - additive pass must never abort analysis + logger.warning("entrypoint detection failed: %s", exc) + app.entrypoint_report.errors.append(str(exc)) + _derive_flags(app) + + +def _run_stages(app: PyApplication, project_dir: Path, rule_paths: tuple) -> None: + """Stages 0-4. Empty until Task 5; the skeleton exists so the contract does.""" + return None + + +def _derive_flags(app: PyApplication) -> None: + for node in _walk(app): + node.is_entrypoint = bool(node.entrypoints) + + +def _walk(app: PyApplication) -> Iterator[object]: + def walk_callable(c: PyCallable) -> Iterator[object]: + yield c + for inner in (c.callables or {}).values(): + yield from walk_callable(inner) + for cls in (c.types or {}).values(): + yield from walk_class(cls) + + def walk_class(k: PyClass) -> Iterator[object]: + yield k + for m in (k.callables or {}).values(): + yield from walk_callable(m) + for inner in (k.types or {}).values(): + yield from walk_class(inner) + + for mod in app.symbol_table.values(): + for fn in (mod.functions or {}).values(): + yield from walk_callable(fn) + for cls in (mod.types or {}).values(): + yield from walk_class(cls) diff --git a/codeanalyzer/options/options.py b/codeanalyzer/options/options.py index 1be9ec4..c1fb362 100644 --- a/codeanalyzer/options/options.py +++ b/codeanalyzer/options/options.py @@ -1,6 +1,6 @@ from dataclasses import dataclass from pathlib import Path -from typing import Optional +from typing import Optional, Tuple from enum import Enum @@ -65,3 +65,4 @@ class AnalysisOptions: pycg_shard_timeout: int = 120 pycg_shard_strategy: ShardStrategy = ShardStrategy.JEDI pycg_max_iter: int = 50 + entrypoint_rules: Tuple[Path, ...] = () diff --git a/test/test_entrypoint_pipeline.py b/test/test_entrypoint_pipeline.py new file mode 100644 index 0000000..4296bfa --- /dev/null +++ b/test/test_entrypoint_pipeline.py @@ -0,0 +1,39 @@ +from pathlib import Path + +from codeanalyzer.entrypoints.pipeline import detect_entrypoints +from codeanalyzer.schema.py_schema import PyApplication + + +def test_pass_is_a_noop_on_an_empty_application(tmp_path: Path): + app = PyApplication(symbol_table={}) + detect_entrypoints(app, tmp_path) + assert app.entrypoint_report.errors == [] + + +def test_pass_never_raises_and_records_the_failure(tmp_path: Path, monkeypatch): + """A finder crash must lose flags, not the analysis.""" + import codeanalyzer.entrypoints.pipeline as p + + def boom(*a, **k): + raise RuntimeError("finder exploded") + + monkeypatch.setattr(p, "_run_stages", boom) + app = PyApplication(symbol_table={}) + detect_entrypoints(app, tmp_path) # must not raise + assert any("finder exploded" in e for e in app.entrypoint_report.errors) + + +def test_derives_is_entrypoint_from_the_list(tmp_path: Path): + from codeanalyzer.schema.py_schema import PyCallable, PyEntrypoint, PyModule + + fn = PyCallable(name="f", path="a.py", signature="a.f") + fn.entrypoints.append( + PyEntrypoint(framework="flask", confidence="certain", rule="flask.route", ruleset="shipped") + ) + app = PyApplication( + symbol_table={ + "a.py": PyModule(file_path="a.py", module_name="a", functions={"f": fn}) + } + ) + detect_entrypoints(app, tmp_path) + assert fn.is_entrypoint is True From 8f4d4120eaea3ae547c8a9003548b58500896c05 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:36:25 -0400 Subject: [PATCH 08/23] feat(entrypoints): rules.yml loader with shipped framework pack (#27) --- codeanalyzer/entrypoints/rules.py | 129 +++++++++++++++++++++++++++++ codeanalyzer/entrypoints/rules.yml | 67 +++++++++++++++ pyproject.toml | 6 ++ test/test_entrypoint_rules.py | 26 ++++++ 4 files changed, 228 insertions(+) create mode 100644 codeanalyzer/entrypoints/rules.py create mode 100644 codeanalyzer/entrypoints/rules.yml create mode 100644 test/test_entrypoint_rules.py diff --git a/codeanalyzer/entrypoints/rules.py b/codeanalyzer/entrypoints/rules.py new file mode 100644 index 0000000..79e2b9b --- /dev/null +++ b/codeanalyzer/entrypoints/rules.py @@ -0,0 +1,129 @@ +"""Loading and merging of entrypoint rules (#27). + +The shipped ``rules.yml`` covers known frameworks; users extend it with +``--entrypoint-rules``. User rules merge additively and may ``disable:`` a +shipped rule by id. A malformed user file is a hard error before analysis +starts -- silently skipping it would let someone ship rules they believe +are live. +""" +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, Iterable, List, Optional + +import yaml + +_SHIPPED = Path(__file__).with_name("rules.yml") +_CONFIDENCE = {"declared", "certain", "heuristic"} + + +class RulesError(Exception): + """Raised for a malformed rules file. Never swallowed.""" + + +@dataclass +class DecoratorRule: + id: str + match: str + confidence: str = "certain" + route: Optional[Dict[str, Any]] = None + methods: Optional[Dict[str, Any]] = None + + +@dataclass +class BaseRule: + id: str + match: str + confidence: str = "certain" + transitive: bool = False + dispatch: List[str] = field(default_factory=list) + + +@dataclass +class Framework: + name: str + detect: List[str] = field(default_factory=list) + decorators: List[DecoratorRule] = field(default_factory=list) + bases: List[BaseRule] = field(default_factory=list) + + +@dataclass +class RuleSet: + frameworks: Dict[str, Framework] = field(default_factory=dict) + rulesets: List[str] = field(default_factory=list) + + +def load_rules(user_paths: Iterable[Path] = ()) -> RuleSet: + out = RuleSet() + _merge(out, _read(_SHIPPED), "shipped") + for p in user_paths: + _merge(out, _read(Path(p)), f"user:{p}") + return out + + +def _read(path: Path) -> Dict[str, Any]: + try: + data = yaml.safe_load(path.read_text()) + except FileNotFoundError as exc: + raise RulesError(f"rules file not found: {path}") from exc + except yaml.YAMLError as exc: + raise RulesError(f"{path}: invalid YAML: {exc}") from exc + if not isinstance(data, dict): + raise RulesError(f"{path}: top level must be a mapping") + return data + + +def _merge(out: RuleSet, data: Dict[str, Any], origin: str) -> None: + out.rulesets.append(origin) + disabled = set(data.get("disable") or []) + frameworks = data.get("frameworks") or {} + if not isinstance(frameworks, dict): + raise RulesError(f"{origin}: `frameworks` must be a mapping") + + for name, body in frameworks.items(): + if not isinstance(body, dict): + raise RulesError(f"{origin}: framework `{name}` must be a mapping") + fw = out.frameworks.setdefault(name, Framework(name=name)) + fw.detect = sorted(set(fw.detect) | set(body.get("detect") or [])) + for raw in body.get("decorators") or []: + fw.decorators.append(_decorator_rule(raw, origin)) + for raw in body.get("bases") or []: + fw.bases.append(_base_rule(raw, origin)) + + for fw in out.frameworks.values(): + fw.decorators = [r for r in fw.decorators if r.id not in disabled] + fw.bases = [r for r in fw.bases if r.id not in disabled] + + +def _require(raw: Dict[str, Any], key: str, origin: str) -> Any: + if key not in raw: + raise RulesError(f"{origin}: rule {raw!r} is missing `{key}`") + return raw[key] + + +def _confidence(raw: Dict[str, Any], origin: str) -> str: + c = raw.get("confidence", "certain") + if c not in _CONFIDENCE: + raise RulesError(f"{origin}: confidence must be one of {sorted(_CONFIDENCE)}, got {c!r}") + return c + + +def _decorator_rule(raw: Dict[str, Any], origin: str) -> DecoratorRule: + return DecoratorRule( + id=_require(raw, "id", origin), + match=_require(raw, "match", origin), + confidence=_confidence(raw, origin), + route=raw.get("route"), + methods=raw.get("methods"), + ) + + +def _base_rule(raw: Dict[str, Any], origin: str) -> BaseRule: + return BaseRule( + id=_require(raw, "id", origin), + match=_require(raw, "match", origin), + confidence=_confidence(raw, origin), + transitive=bool(raw.get("transitive", False)), + dispatch=list(raw.get("dispatch") or []), + ) diff --git a/codeanalyzer/entrypoints/rules.yml b/codeanalyzer/entrypoints/rules.yml new file mode 100644 index 0000000..ded0b2f --- /dev/null +++ b/codeanalyzer/entrypoints/rules.yml @@ -0,0 +1,67 @@ +version: 1 + +frameworks: + flask: + detect: [flask] + decorators: + - id: flask.route + match: "flask.Flask.route" + route: {from: positional, index: 0} + methods: {from: keyword, name: methods, default: [GET]} + - id: flask.bp-verb + match: "flask.Blueprint.{get,post,put,delete,patch}" + route: {from: positional, index: 0} + methods: {from: match_suffix} + bases: + - id: flask.methodview + match: "flask.views.MethodView" + transitive: true + dispatch: [get, post, put, delete, patch] + + fastapi: + detect: [fastapi] + decorators: + - id: fastapi.verb + match: "fastapi.FastAPI.{get,post,put,delete,patch,head,options}" + route: {from: positional, index: 0} + methods: {from: match_suffix} + - id: fastapi.router-verb + match: "fastapi.APIRouter.{get,post,put,delete,patch}" + route: {from: positional, index: 0} + methods: {from: match_suffix} + - id: fastapi.websocket + match: "fastapi.FastAPI.websocket" + route: {from: positional, index: 0} + + celery: + detect: [celery] + decorators: + - id: celery.shared-task + match: "celery.shared_task" + - id: celery.task + match: "celery.Celery.task" + + click: + detect: [click, typer] + decorators: + - id: click.command + match: "click.{command,group}" + - id: typer.command + match: "typer.Typer.command" + + drf: + detect: [rest_framework] + decorators: + - id: drf.api-view + match: "rest_framework.decorators.api_view" + - id: drf.action + match: "rest_framework.decorators.action" + bases: + - id: drf.apiview + match: "rest_framework.views.APIView" + transitive: true + dispatch: [get, post, put, patch, delete, head, options] + - id: drf.viewset + match: "rest_framework.viewsets.*" + transitive: true + dispatch: [list, retrieve, create, update, partial_update, destroy] diff --git a/pyproject.toml b/pyproject.toml index 6789093..dd98745 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -47,6 +47,9 @@ dependencies = [ # (scalpel/SSA/const.py uses astor.to_source). Pure-Python, installs # everywhere; scalpel pins ~=0.8.1. "astor>=0.8.1,<0.9.0", + # pyyaml: the entrypoint rules pack (#27) is YAML. Declared explicitly -- + # it was previously only reachable as a transitive dep of ray. + "pyyaml>=6.0,<7.0", ] [project.optional-dependencies] @@ -56,6 +59,9 @@ neo4j = [ "neo4j>=5.0.0,<6.0.0", ] +[tool.setuptools.package-data] +codeanalyzer = ["entrypoints/*.yml"] + [dependency-groups] test = [ "pytest>=7.0.0,<8.0.0", diff --git a/test/test_entrypoint_rules.py b/test/test_entrypoint_rules.py new file mode 100644 index 0000000..95425e1 --- /dev/null +++ b/test/test_entrypoint_rules.py @@ -0,0 +1,26 @@ +import pytest + +from codeanalyzer.entrypoints.rules import RulesError, load_rules + + +def test_shipped_rules_load_and_include_flask(): + rs = load_rules() + assert "flask" in rs.frameworks + flask = rs.frameworks["flask"] + assert "flask" in flask.detect + assert any(r.id == "flask.route" for r in flask.decorators) + + +def test_every_shipped_rule_has_a_stable_id_and_valid_confidence(): + rs = load_rules() + for fw in rs.frameworks.values(): + for rule in list(fw.decorators) + list(fw.bases): + assert rule.id, "every rule needs a stable id so users can disable it" + assert rule.confidence in {"declared", "certain", "heuristic"} + + +def test_malformed_user_file_raises_before_analysis(tmp_path): + bad = tmp_path / "bad.yml" + bad.write_text("frameworks: [this is a list not a mapping]\n") + with pytest.raises(RulesError): + load_rules([bad]) From ac476bd17c9438dc9e9f938d3838490a1f5adc5b Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:40:55 -0400 Subject: [PATCH 09/23] fix(entrypoints): validate disable: shape; drop dead setuptools stanza (#27) - disable: must now be a list of rule id strings; a bare-string typo or other malformed value raises RulesError instead of silently disabling nothing (set() over a string iterated its characters). - Remove [tool.setuptools.package-data] -- this project builds with hatchling, which already ships entrypoints/rules.yml via its default whole-package inclusion (verified with uv build --wheel). --- codeanalyzer/entrypoints/rules.py | 9 ++++++++- pyproject.toml | 3 --- test/test_entrypoint_rules.py | 15 +++++++++++++++ 3 files changed, 23 insertions(+), 4 deletions(-) diff --git a/codeanalyzer/entrypoints/rules.py b/codeanalyzer/entrypoints/rules.py index 79e2b9b..aaba57b 100644 --- a/codeanalyzer/entrypoints/rules.py +++ b/codeanalyzer/entrypoints/rules.py @@ -76,7 +76,7 @@ def _read(path: Path) -> Dict[str, Any]: def _merge(out: RuleSet, data: Dict[str, Any], origin: str) -> None: out.rulesets.append(origin) - disabled = set(data.get("disable") or []) + disabled = set(_disable_list(data, origin)) frameworks = data.get("frameworks") or {} if not isinstance(frameworks, dict): raise RulesError(f"{origin}: `frameworks` must be a mapping") @@ -96,6 +96,13 @@ def _merge(out: RuleSet, data: Dict[str, Any], origin: str) -> None: fw.bases = [r for r in fw.bases if r.id not in disabled] +def _disable_list(data: Dict[str, Any], origin: str) -> List[str]: + raw = data.get("disable") or [] + if not isinstance(raw, list) or not all(isinstance(x, str) for x in raw): + raise RulesError(f"{origin}: `disable` must be a list of rule id strings") + return raw + + def _require(raw: Dict[str, Any], key: str, origin: str) -> Any: if key not in raw: raise RulesError(f"{origin}: rule {raw!r} is missing `{key}`") diff --git a/pyproject.toml b/pyproject.toml index dd98745..b642fe1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -59,9 +59,6 @@ neo4j = [ "neo4j>=5.0.0,<6.0.0", ] -[tool.setuptools.package-data] -codeanalyzer = ["entrypoints/*.yml"] - [dependency-groups] test = [ "pytest>=7.0.0,<8.0.0", diff --git a/test/test_entrypoint_rules.py b/test/test_entrypoint_rules.py index 95425e1..6743325 100644 --- a/test/test_entrypoint_rules.py +++ b/test/test_entrypoint_rules.py @@ -24,3 +24,18 @@ def test_malformed_user_file_raises_before_analysis(tmp_path): bad.write_text("frameworks: [this is a list not a mapping]\n") with pytest.raises(RulesError): load_rules([bad]) + + +def test_bare_string_disable_raises_instead_of_silently_matching_chars(tmp_path): + bad = tmp_path / "bad.yml" + bad.write_text("disable: flask.route\n") + with pytest.raises(RulesError): + load_rules([bad]) + + +def test_well_formed_disable_list_removes_the_shipped_rule(tmp_path): + user = tmp_path / "user.yml" + user.write_text("disable: [flask.route]\n") + rs = load_rules([user]) + flask = rs.frameworks["flask"] + assert all(r.id != "flask.route" for r in flask.decorators) From 07cb739c3f8bd824da8242f8086dcc46f58dbeca Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:43:12 -0400 Subject: [PATCH 10/23] feat(cli): --entrypoint-rules for user rule packs (#27) --- codeanalyzer/__main__.py | 11 ++++++++++- test/test_entrypoint_rules.py | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/codeanalyzer/__main__.py b/codeanalyzer/__main__.py index c98a620..750a325 100644 --- a/codeanalyzer/__main__.py +++ b/codeanalyzer/__main__.py @@ -2,7 +2,7 @@ import sys from importlib.metadata import version as _pkg_version, PackageNotFoundError from pathlib import Path -from typing import Optional, Annotated +from typing import List, Optional, Annotated import typer @@ -299,6 +299,14 @@ def main( min=-1, ), ] = 50, + entrypoint_rules: Annotated[ + Optional[List[Path]], + typer.Option( + "--entrypoint-rules", + help="Extra entrypoint rules file (YAML). Repeatable; merges with " + "the shipped rules. A malformed file is an error.", + ), + ] = None, ): # Determinism: pin the interpreter hash seed before any analysis (no-op # when PYTHONHASHSEED is already set; --version exits before this). @@ -385,6 +393,7 @@ def main( pycg_shard_timeout=pycg_shard_timeout, pycg_shard_strategy=pycg_shard_strategy, pycg_max_iter=pycg_max_iter, + entrypoint_rules=tuple(entrypoint_rules or ()), ) _set_log_level(options.verbosity) diff --git a/test/test_entrypoint_rules.py b/test/test_entrypoint_rules.py index 6743325..e6af3cd 100644 --- a/test/test_entrypoint_rules.py +++ b/test/test_entrypoint_rules.py @@ -39,3 +39,38 @@ def test_well_formed_disable_list_removes_the_shipped_rule(tmp_path): rs = load_rules([user]) flask = rs.frameworks["flask"] assert all(r.id != "flask.route" for r in flask.decorators) + + +def test_user_rules_merge_additively_with_shipped(tmp_path): + extra = tmp_path / "mine.yml" + extra.write_text( + "version: 1\n" + "frameworks:\n" + " inhouse:\n" + " detect: [inhouse]\n" + " decorators:\n" + " - id: inhouse.handler\n" + " match: 'inhouse.app.handler'\n" + ) + rs = load_rules([extra]) + assert "flask" in rs.frameworks # shipped survives + assert "inhouse" in rs.frameworks # user added + assert rs.rulesets == ["shipped", f"user:{extra}"] + + +def test_user_file_can_disable_a_shipped_rule(tmp_path): + off = tmp_path / "off.yml" + off.write_text("version: 1\ndisable: [flask.route]\n") + rs = load_rules([off]) + assert not any(r.id == "flask.route" for r in rs.frameworks["flask"].decorators) + assert any(r.id == "flask.bp-verb" for r in rs.frameworks["flask"].decorators) + + +def test_bad_confidence_value_is_rejected(tmp_path): + bad = tmp_path / "c.yml" + bad.write_text( + "version: 1\nframeworks:\n x:\n decorators:\n" + " - id: x.y\n match: 'x.y'\n confidence: probably\n" + ) + with pytest.raises(RulesError): + load_rules([bad]) From fadf43f7bfacc0c45ff0ac40ee32c63df4d17437 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:48:43 -0400 Subject: [PATCH 11/23] feat(entrypoints): stage 0 framework detection gate (#27) Loads entrypoint rules outside the detection pass's broad exception handler so a malformed --entrypoint-rules file is a hard configuration error (RulesError) rather than a swallowed detection failure. --- codeanalyzer/entrypoints/detect.py | 59 ++++++++++++++++++++++++++++ codeanalyzer/entrypoints/pipeline.py | 22 ++++++++--- test/test_entrypoint_detect.py | 35 +++++++++++++++++ test/test_entrypoint_pipeline.py | 17 ++++++++ 4 files changed, 128 insertions(+), 5 deletions(-) create mode 100644 codeanalyzer/entrypoints/detect.py create mode 100644 test/test_entrypoint_detect.py diff --git a/codeanalyzer/entrypoints/detect.py b/codeanalyzer/entrypoints/detect.py new file mode 100644 index 0000000..9929621 --- /dev/null +++ b/codeanalyzer/entrypoints/detect.py @@ -0,0 +1,59 @@ +"""Stage 0: which frameworks is this project actually using? (#27) + +Gates every later stage, so a project without Celery never pays for Celery +rules and cannot false-positive on a locally-defined ``shared_task``. A +package counts as present if first-party source imports it OR the dependency +manifest names it -- either is sufficient, since an import may be dynamic. +""" +from __future__ import annotations + +import re +from pathlib import Path +from typing import Set + +from codeanalyzer.entrypoints.rules import RuleSet +from codeanalyzer.schema.py_schema import PyApplication + +_REQ = re.compile(r"^\s*['\"]?([A-Za-z0-9_.\-]+)") +_DEPS_ARRAY = re.compile(r"dependencies\s*=\s*\[(.*?)\]", re.DOTALL) +_PKG = re.compile(r"['\"]([A-Za-z0-9][A-Za-z0-9_.\-]*)") + + +def detected_frameworks(app: PyApplication, project_dir: Path, rules: RuleSet) -> Set[str]: + present = _imported_packages(app) | _manifest_packages(project_dir) + return { + name + for name, fw in rules.frameworks.items() + if any(pkg in present for pkg in (fw.detect or [name])) + } + + +def _imported_packages(app: PyApplication) -> Set[str]: + out: Set[str] = set() + for mod in app.symbol_table.values(): + for imp in mod.imports or []: + # `from flask import Flask` puts the package in `module`, not `name`. + # Prefer `module`; fall back to `name` for a bare `import flask`. + spelling = (getattr(imp, "module", "") or getattr(imp, "name", "") or "") + spelling = spelling.lstrip(".") + if spelling: + out.add(spelling.split(".", 1)[0]) + return out + + +def _manifest_packages(project_dir: Path) -> Set[str]: + out: Set[str] = set() + pyproject = project_dir / "pyproject.toml" + if pyproject.exists(): + # PEP 621 `[project] dependencies = [...]` -- single- or multi-line. + m = _DEPS_ARRAY.search(pyproject.read_text()) + if m: + for pm in _PKG.finditer(m.group(1)): + out.add(pm.group(1).split("[", 1)[0].lower()) + requirements = project_dir / "requirements.txt" + if requirements.exists(): + for line in requirements.read_text().splitlines(): + m = _REQ.match(line) + if m: + out.add(m.group(1).split("[", 1)[0].lower()) + return out diff --git a/codeanalyzer/entrypoints/pipeline.py b/codeanalyzer/entrypoints/pipeline.py index 12e171e..2a77769 100644 --- a/codeanalyzer/entrypoints/pipeline.py +++ b/codeanalyzer/entrypoints/pipeline.py @@ -9,6 +9,8 @@ from pathlib import Path from typing import Iterable, Iterator +from codeanalyzer.entrypoints.detect import detected_frameworks +from codeanalyzer.entrypoints.rules import RuleSet, load_rules from codeanalyzer.schema.py_schema import PyApplication, PyCallable, PyClass from codeanalyzer.utils import logger @@ -16,18 +18,28 @@ def detect_entrypoints( app: PyApplication, project_dir: Path, rule_paths: Iterable[Path] = () ) -> None: - """Populate ``entrypoints`` on every callable and class, in place.""" + """Populate ``entrypoints`` on every callable and class, in place. + + Loading the rules is a CONFIGURATION step, not a detection step: a + malformed user rules file is a hard error that must stop the run before + analysis starts, so ``load_rules`` runs outside (and before) the + try/except below. Everything after that -- the actual framework + detection -- is best-effort and must never abort the analysis. + """ + rules = load_rules(rule_paths) try: - _run_stages(app, project_dir, tuple(rule_paths)) + _run_stages(app, project_dir, rules) except Exception as exc: # noqa: BLE001 - additive pass must never abort analysis logger.warning("entrypoint detection failed: %s", exc) app.entrypoint_report.errors.append(str(exc)) _derive_flags(app) -def _run_stages(app: PyApplication, project_dir: Path, rule_paths: tuple) -> None: - """Stages 0-4. Empty until Task 5; the skeleton exists so the contract does.""" - return None +def _run_stages(app: PyApplication, project_dir: Path, rules: RuleSet) -> None: + """Stages 0-4. Only stage 0 (framework detection) exists so far.""" + app.entrypoint_report.rulesets = list(rules.rulesets) + frameworks = detected_frameworks(app, project_dir, rules) + app.entrypoint_report.frameworks_detected = sorted(frameworks) def _derive_flags(app: PyApplication) -> None: diff --git a/test/test_entrypoint_detect.py b/test/test_entrypoint_detect.py new file mode 100644 index 0000000..7bf247b --- /dev/null +++ b/test/test_entrypoint_detect.py @@ -0,0 +1,35 @@ +from pathlib import Path + +from codeanalyzer.entrypoints.detect import detected_frameworks +from codeanalyzer.entrypoints.rules import load_rules +from codeanalyzer.schema.py_schema import PyApplication, PyImport, PyModule + + +def _app(*modules: str) -> PyApplication: + return PyApplication( + symbol_table={ + "a.py": PyModule( + file_path="a.py", + module_name="a", + imports=[PyImport(module=m, name=m.split(".")[-1]) for m in modules], + ) + } + ) + + +def test_framework_detected_from_an_import(tmp_path: Path): + got = detected_frameworks(_app("flask"), tmp_path, load_rules()) + assert "flask" in got + + +def test_absent_framework_is_not_detected(tmp_path: Path): + got = detected_frameworks(_app("os"), tmp_path, load_rules()) + assert "celery" not in got + + +def test_manifest_entry_alone_is_sufficient(tmp_path: Path): + (tmp_path / "pyproject.toml").write_text( + '[project]\nname = "x"\ndependencies = ["celery>=5"]\n' + ) + got = detected_frameworks(_app("os"), tmp_path, load_rules()) + assert "celery" in got diff --git a/test/test_entrypoint_pipeline.py b/test/test_entrypoint_pipeline.py index 4296bfa..b505d4f 100644 --- a/test/test_entrypoint_pipeline.py +++ b/test/test_entrypoint_pipeline.py @@ -1,6 +1,9 @@ from pathlib import Path +import pytest + from codeanalyzer.entrypoints.pipeline import detect_entrypoints +from codeanalyzer.entrypoints.rules import RulesError from codeanalyzer.schema.py_schema import PyApplication @@ -37,3 +40,17 @@ def test_derives_is_entrypoint_from_the_list(tmp_path: Path): ) detect_entrypoints(app, tmp_path) assert fn.is_entrypoint is True + + +def test_malformed_user_rules_file_raises_instead_of_being_swallowed(tmp_path: Path): + """A bad --entrypoint-rules file is a CONFIGURATION error, not a detection + failure: it must stop the run via RulesError, not land quietly in + entrypoint_report.errors like a finder crash would.""" + bad_rules = tmp_path / "bad_rules.yml" + bad_rules.write_text("frameworks: not-a-mapping\n") + app = PyApplication(symbol_table={}) + + with pytest.raises(RulesError): + detect_entrypoints(app, tmp_path, rule_paths=(bad_rules,)) + + assert app.entrypoint_report.errors == [] From 84a69fd7baf45550b3bce18f6b43bac2520357ad Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:52:33 -0400 Subject: [PATCH 12/23] fix(entrypoints): handle extras brackets and comments in dependency array scan (#27) Bracket-depth counting replaces the non-greedy regex so a nested [...] from extras (celery[redis]) no longer truncates the dependency array early, and comments are stripped before scanning so a commented-out line is no longer detected as a live dependency. --- codeanalyzer/entrypoints/detect.py | 62 +++++++++++++++++++++++++++--- test/test_entrypoint_detect.py | 24 ++++++++++++ 2 files changed, 80 insertions(+), 6 deletions(-) diff --git a/codeanalyzer/entrypoints/detect.py b/codeanalyzer/entrypoints/detect.py index 9929621..5ca57aa 100644 --- a/codeanalyzer/entrypoints/detect.py +++ b/codeanalyzer/entrypoints/detect.py @@ -9,13 +9,13 @@ import re from pathlib import Path -from typing import Set +from typing import Optional, Set from codeanalyzer.entrypoints.rules import RuleSet from codeanalyzer.schema.py_schema import PyApplication _REQ = re.compile(r"^\s*['\"]?([A-Za-z0-9_.\-]+)") -_DEPS_ARRAY = re.compile(r"dependencies\s*=\s*\[(.*?)\]", re.DOTALL) +_DEPS_START = re.compile(r"dependencies\s*=\s*\[") _PKG = re.compile(r"['\"]([A-Za-z0-9][A-Za-z0-9_.\-]*)") @@ -45,10 +45,11 @@ def _manifest_packages(project_dir: Path) -> Set[str]: out: Set[str] = set() pyproject = project_dir / "pyproject.toml" if pyproject.exists(): - # PEP 621 `[project] dependencies = [...]` -- single- or multi-line. - m = _DEPS_ARRAY.search(pyproject.read_text()) - if m: - for pm in _PKG.finditer(m.group(1)): + # PEP 621 `[project] dependencies = [...]` -- single- or multi-line, + # possibly containing nested `[...]` extras (`celery[redis]`). + span = _deps_array_span(_strip_comments(pyproject.read_text())) + if span is not None: + for pm in _PKG.finditer(span): out.add(pm.group(1).split("[", 1)[0].lower()) requirements = project_dir / "requirements.txt" if requirements.exists(): @@ -57,3 +58,52 @@ def _manifest_packages(project_dir: Path) -> Set[str]: if m: out.add(m.group(1).split("[", 1)[0].lower()) return out + + +def _strip_comments(text: str) -> str: + """Drop everything from an unquoted ``#`` to end of line. + + # ponytail: quote tracking resets each line, so a `#` inside a + # triple-quoted string spanning lines could be mis-stripped. TOML + # dependency arrays don't use those in practice; revisit if they do. + """ + out_lines = [] + for line in text.splitlines(): + in_str = None + cut = len(line) + for i, ch in enumerate(line): + if in_str: + if ch == in_str: + in_str = None + elif ch in ("'", '"'): + in_str = ch + elif ch == "#": + cut = i + break + out_lines.append(line[:cut]) + return "\n".join(out_lines) + + +def _deps_array_span(text: str) -> Optional[str]: + """Return the contents between the `dependencies = [` and its matching + `]`, counting bracket depth so a nested `[...]` (extras, e.g. + `celery[redis]`) doesn't close the span early.""" + m = _DEPS_START.search(text) + if not m: + return None + depth = 1 + in_str = None + i = m.end() + while i < len(text) and depth > 0: + ch = text[i] + if in_str: + if ch == in_str: + in_str = None + elif ch in ("'", '"'): + in_str = ch + elif ch == "[": + depth += 1 + elif ch == "]": + depth -= 1 + i += 1 + return text[m.end() : i - 1] diff --git a/test/test_entrypoint_detect.py b/test/test_entrypoint_detect.py index 7bf247b..4e78b90 100644 --- a/test/test_entrypoint_detect.py +++ b/test/test_entrypoint_detect.py @@ -33,3 +33,27 @@ def test_manifest_entry_alone_is_sufficient(tmp_path: Path): ) got = detected_frameworks(_app("os"), tmp_path, load_rules()) assert "celery" in got + + +def test_extras_bracket_in_dependency_does_not_truncate_the_array(tmp_path: Path): + (tmp_path / "pyproject.toml").write_text( + '[project]\nname = "x"\n' + 'dependencies = ["celery[redis]>=5", "flask>=2.0"]\n' + ) + got = detected_frameworks(_app("os"), tmp_path, load_rules()) + assert "celery" in got + assert "flask" in got + + +def test_commented_out_dependency_is_not_detected(tmp_path: Path): + (tmp_path / "pyproject.toml").write_text( + "[project]\n" + 'name = "x"\n' + "dependencies = [\n" + ' # "celery>=5",\n' + ' "flask>=2.0",\n' + "]\n" + ) + got = detected_frameworks(_app("os"), tmp_path, load_rules()) + assert "celery" not in got + assert "flask" in got From f24b3a117b33c82739460419a1b4098db30e5674 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:55:30 -0400 Subject: [PATCH 13/23] fix(entrypoints): bound dependency array scan to avoid false positives on unterminated arrays (#27) An unclosed dependencies = [ in a truncated/corrupt pyproject.toml previously scanned to end of file and harvested quoted strings from later TOML tables as fake dependencies. Bound the scan to the next table header and return None when the array never closes, matching pre-fix behaviour of detecting nothing from a malformed file. --- codeanalyzer/entrypoints/detect.py | 15 +++++++++++++-- test/test_entrypoint_detect.py | 18 ++++++++++++++++++ 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/codeanalyzer/entrypoints/detect.py b/codeanalyzer/entrypoints/detect.py index 5ca57aa..072d14e 100644 --- a/codeanalyzer/entrypoints/detect.py +++ b/codeanalyzer/entrypoints/detect.py @@ -16,6 +16,7 @@ _REQ = re.compile(r"^\s*['\"]?([A-Za-z0-9_.\-]+)") _DEPS_START = re.compile(r"dependencies\s*=\s*\[") +_TABLE_HEADER = re.compile(r"(?m)^[ \t]*\[") _PKG = re.compile(r"['\"]([A-Za-z0-9][A-Za-z0-9_.\-]*)") @@ -87,14 +88,22 @@ def _strip_comments(text: str) -> str: def _deps_array_span(text: str) -> Optional[str]: """Return the contents between the `dependencies = [` and its matching `]`, counting bracket depth so a nested `[...]` (extras, e.g. - `celery[redis]`) doesn't close the span early.""" + `celery[redis]`) doesn't close the span early. + + Bounded by the next TOML table header (a `[` starting a line): if the + array never closes before then, it's unterminated (truncated/corrupt + file) and this returns None rather than harvesting quoted strings out + of whatever table follows. + """ m = _DEPS_START.search(text) if not m: return None + boundary = _TABLE_HEADER.search(text, m.end()) + limit = boundary.start() if boundary else len(text) depth = 1 in_str = None i = m.end() - while i < len(text) and depth > 0: + while i < limit and depth > 0: ch = text[i] if in_str: if ch == in_str: @@ -106,4 +115,6 @@ def _deps_array_span(text: str) -> Optional[str]: elif ch == "]": depth -= 1 i += 1 + if depth != 0: + return None return text[m.end() : i - 1] diff --git a/test/test_entrypoint_detect.py b/test/test_entrypoint_detect.py index 4e78b90..d0121d3 100644 --- a/test/test_entrypoint_detect.py +++ b/test/test_entrypoint_detect.py @@ -57,3 +57,21 @@ def test_commented_out_dependency_is_not_detected(tmp_path: Path): got = detected_frameworks(_app("os"), tmp_path, load_rules()) assert "celery" not in got assert "flask" in got + + +def test_unterminated_dependencies_array_detects_nothing(tmp_path: Path): + """A truncated/corrupt pyproject.toml must not leak quoted strings from + a later table (e.g. an author email or homepage URL) into the detected + package set -- matching pre-fix behaviour of "malformed file, nothing + detected".""" + (tmp_path / "pyproject.toml").write_text( + "[project]\n" + 'name = "x"\n' + "dependencies = [\n" + ' "celery>=5"\n' + "\n" + "[project.urls]\n" + 'Homepage = "https://flask.example.com"\n' + ) + got = detected_frameworks(_app("os"), tmp_path, load_rules()) + assert got == set() From b4e163508330cfde4a1634c5ff656e2e39fc9e88 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 09:57:36 -0400 Subject: [PATCH 14/23] feat(entrypoints): decorator rule matching with route extraction (#27) --- codeanalyzer/entrypoints/matching.py | 103 +++++++++++++++++++++++++++ test/test_entrypoint_decorators.py | 56 +++++++++++++++ 2 files changed, 159 insertions(+) create mode 100644 codeanalyzer/entrypoints/matching.py create mode 100644 test/test_entrypoint_decorators.py diff --git a/codeanalyzer/entrypoints/matching.py b/codeanalyzer/entrypoints/matching.py new file mode 100644 index 0000000..af00ead --- /dev/null +++ b/codeanalyzer/entrypoints/matching.py @@ -0,0 +1,103 @@ +"""Stage 3: match rules against decorators and base classes (#27). + +Matching is on ``PyDecorator.qualified_name`` -- never the written spelling -- +so ``@route`` under ``from flask import route`` hits the same rule as +``@app.route``. An unresolved decorator (``qualified_name is None``) never +matches: under-approximate rather than guess. +""" +from __future__ import annotations + +import ast +import re +from typing import Any, Dict, Iterable, List, Optional + +from codeanalyzer.entrypoints.rules import DecoratorRule +from codeanalyzer.schema.py_schema import PyEntrypoint + +# Dispatch names that are HTTP verbs. DRF's ViewSet dispatch names +# (list, retrieve, create, ...) are NOT verbs and must not be emitted as such. +_HTTP_VERBS = {"get", "post", "put", "patch", "delete", "head", "options"} + + +def match_pattern(pattern: str, qualified_name: Optional[str]) -> bool: + """``{a,b}`` alternation and trailing ``*``; everything else is literal.""" + if not qualified_name: + return False + return re.fullmatch(_compile(pattern), qualified_name) is not None + + +def _compile(pattern: str) -> str: + out, i = [], 0 + while i < len(pattern): + ch = pattern[i] + if ch == "{": + j = pattern.index("}", i) + alts = pattern[i + 1 : j].split(",") + out.append("(?:" + "|".join(re.escape(a.strip()) for a in alts) + ")") + i = j + 1 + elif ch == "*": + out.append(r"[^\s]*") + i += 1 + else: + out.append(re.escape(ch)) + i += 1 + return "".join(out) + + +def _literal(text: Optional[str]) -> Any: + """Best-effort: decorator arguments are unparsed source fragments.""" + if text is None: + return None + try: + return ast.literal_eval(text) + except (ValueError, SyntaxError): + return None + + +def _route_of(dec, spec: Optional[Dict[str, Any]]) -> Optional[str]: + if not spec or spec.get("from") != "positional": + return None + args = dec.positional_arguments or [] + idx = int(spec.get("index", 0)) + if idx >= len(args): + return None + value = _literal(args[idx]) + return value if isinstance(value, str) else None + + +def _methods_of(dec, rule: DecoratorRule, spec: Optional[Dict[str, Any]]) -> List[str]: + if not spec: + return [] + source = spec.get("from") + if source == "match_suffix": + verb = (dec.qualified_name or "").rsplit(".", 1)[-1] + return [verb.upper()] + if source == "keyword": + raw = (dec.keyword_arguments or {}).get(spec.get("name", "")) + value = _literal(raw) + if isinstance(value, (list, tuple)): + return [str(v).upper() for v in value] + return [str(v).upper() for v in (spec.get("default") or [])] + return [] + + +def entrypoints_from_decorators( + node, framework: str, rules: Iterable[DecoratorRule], ruleset: str +) -> List[PyEntrypoint]: + out: List[PyEntrypoint] = [] + for dec in getattr(node, "decorators", []) or []: + for rule in rules: + if not match_pattern(rule.match, dec.qualified_name): + continue + out.append( + PyEntrypoint( + framework=framework, + confidence=rule.confidence, + rule=rule.id, + ruleset=ruleset, + evidence=dec.qualified_name, + route=_route_of(dec, rule.route), + http_methods=_methods_of(dec, rule, rule.methods), + ) + ) + return out diff --git a/test/test_entrypoint_decorators.py b/test/test_entrypoint_decorators.py new file mode 100644 index 0000000..bfb480b --- /dev/null +++ b/test/test_entrypoint_decorators.py @@ -0,0 +1,56 @@ +from codeanalyzer.entrypoints.matching import entrypoints_from_decorators, match_pattern +from codeanalyzer.entrypoints.rules import DecoratorRule +from codeanalyzer.schema.py_schema import PyCallable, PyDecorator + + +def test_brace_alternation_and_wildcard(): + assert match_pattern("flask.Blueprint.{get,post}", "flask.Blueprint.get") + assert not match_pattern("flask.Blueprint.{get,post}", "flask.Blueprint.delete") + assert match_pattern("rest_framework.viewsets.*", "rest_framework.viewsets.ModelViewSet") + assert not match_pattern("flask.Flask.route", "flask.Flask.routes") + + +def test_route_and_methods_are_extracted(): + fn = PyCallable(name="h", path="a.py", signature="a.h") + fn.decorators.append( + PyDecorator( + name="app.route", + qualified_name="flask.Flask.route", + positional_arguments=["'/products'"], + keyword_arguments={"methods": "['POST']"}, + ) + ) + rule = DecoratorRule( + id="flask.route", + match="flask.Flask.route", + route={"from": "positional", "index": 0}, + methods={"from": "keyword", "name": "methods", "default": ["GET"]}, + ) + (ep,) = entrypoints_from_decorators(fn, "flask", [rule], "shipped") + assert ep.route == "/products" + assert ep.http_methods == ["POST"] + assert ep.rule == "flask.route" and ep.ruleset == "shipped" + + +def test_verb_comes_from_the_matched_suffix(): + fn = PyCallable(name="h", path="a.py", signature="a.h") + fn.decorators.append( + PyDecorator(name="router.post", qualified_name="fastapi.APIRouter.post", + positional_arguments=["'/x'"]) + ) + rule = DecoratorRule( + id="fastapi.router-verb", + match="fastapi.APIRouter.{get,post}", + route={"from": "positional", "index": 0}, + methods={"from": "match_suffix"}, + ) + (ep,) = entrypoints_from_decorators(fn, "fastapi", [rule], "shipped") + assert ep.http_methods == ["POST"] + + +def test_unresolved_decorator_never_matches(): + """qualified_name is None when Jedi could not resolve; must not guess.""" + fn = PyCallable(name="h", path="a.py", signature="a.h") + fn.decorators.append(PyDecorator(name="app.route", qualified_name=None)) + rule = DecoratorRule(id="flask.route", match="flask.Flask.route") + assert entrypoints_from_decorators(fn, "flask", [rule], "shipped") == [] From 7167f58c2e24b53a368976aeb3fbe236ed5aa50b Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 10:01:16 -0400 Subject: [PATCH 15/23] fix(entrypoints): validate match patterns at load time, not mid-match - reject unbalanced/nested { at load_rules() as RulesError instead of letting _compile raise a bare ValueError mid-analysis - break the rules.py <-> matching.py cycle by deferring the DecoratorRule import to TYPE_CHECKING - * no longer crosses a dot, matching module-member semantics - drop the unused rule param from _methods_of --- codeanalyzer/entrypoints/matching.py | 43 ++++++++++++++++++++++++---- codeanalyzer/entrypoints/rules.py | 15 ++++++++-- test/test_entrypoint_decorators.py | 7 +++++ test/test_entrypoint_rules.py | 20 +++++++++++++ 4 files changed, 77 insertions(+), 8 deletions(-) diff --git a/codeanalyzer/entrypoints/matching.py b/codeanalyzer/entrypoints/matching.py index af00ead..6642e23 100644 --- a/codeanalyzer/entrypoints/matching.py +++ b/codeanalyzer/entrypoints/matching.py @@ -4,21 +4,51 @@ so ``@route`` under ``from flask import route`` hits the same rule as ``@app.route``. An unresolved decorator (``qualified_name is None``) never matches: under-approximate rather than guess. + +Pattern grammar: ``{a,b}`` alternation (not nested) and a trailing/embedded +``*`` that matches module MEMBERS only -- it does not cross a ``.``, so +``rest_framework.viewsets.*`` matches ``ModelViewSet`` but not +``viewsets.mixins.ListModelMixin``. Everything else is literal. +``validate_pattern`` rejects anything outside this grammar (unbalanced or +nested ``{``) so a typo in a rules file is a load-time ``RulesError`` +(enforced by ``rules.py``), never a crash mid-analysis. """ from __future__ import annotations import ast import re -from typing import Any, Dict, Iterable, List, Optional +from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional -from codeanalyzer.entrypoints.rules import DecoratorRule from codeanalyzer.schema.py_schema import PyEntrypoint +if TYPE_CHECKING: + from codeanalyzer.entrypoints.rules import DecoratorRule + # Dispatch names that are HTTP verbs. DRF's ViewSet dispatch names # (list, retrieve, create, ...) are NOT verbs and must not be emitted as such. _HTTP_VERBS = {"get", "post", "put", "patch", "delete", "head", "options"} +class PatternError(ValueError): + """A ``match`` pattern outside the ``{a,b}`` / ``*`` grammar `_compile` handles.""" + + +def validate_pattern(pattern: str) -> None: + """Raise ``PatternError`` for unbalanced or nested ``{``.""" + depth = 0 + for ch in pattern: + if ch == "{": + depth += 1 + if depth > 1: + raise PatternError(f"nested '{{' is not supported: {pattern!r}") + elif ch == "}": + depth -= 1 + if depth < 0: + raise PatternError(f"unmatched '}}': {pattern!r}") + if depth != 0: + raise PatternError(f"unbalanced '{{': {pattern!r}") + + def match_pattern(pattern: str, qualified_name: Optional[str]) -> bool: """``{a,b}`` alternation and trailing ``*``; everything else is literal.""" if not qualified_name: @@ -27,6 +57,7 @@ def match_pattern(pattern: str, qualified_name: Optional[str]) -> bool: def _compile(pattern: str) -> str: + validate_pattern(pattern) out, i = [], 0 while i < len(pattern): ch = pattern[i] @@ -36,7 +67,7 @@ def _compile(pattern: str) -> str: out.append("(?:" + "|".join(re.escape(a.strip()) for a in alts) + ")") i = j + 1 elif ch == "*": - out.append(r"[^\s]*") + out.append(r"[^.\s]*") i += 1 else: out.append(re.escape(ch)) @@ -65,7 +96,7 @@ def _route_of(dec, spec: Optional[Dict[str, Any]]) -> Optional[str]: return value if isinstance(value, str) else None -def _methods_of(dec, rule: DecoratorRule, spec: Optional[Dict[str, Any]]) -> List[str]: +def _methods_of(dec, spec: Optional[Dict[str, Any]]) -> List[str]: if not spec: return [] source = spec.get("from") @@ -82,7 +113,7 @@ def _methods_of(dec, rule: DecoratorRule, spec: Optional[Dict[str, Any]]) -> Lis def entrypoints_from_decorators( - node, framework: str, rules: Iterable[DecoratorRule], ruleset: str + node, framework: str, rules: Iterable["DecoratorRule"], ruleset: str ) -> List[PyEntrypoint]: out: List[PyEntrypoint] = [] for dec in getattr(node, "decorators", []) or []: @@ -97,7 +128,7 @@ def entrypoints_from_decorators( ruleset=ruleset, evidence=dec.qualified_name, route=_route_of(dec, rule.route), - http_methods=_methods_of(dec, rule, rule.methods), + http_methods=_methods_of(dec, rule.methods), ) ) return out diff --git a/codeanalyzer/entrypoints/rules.py b/codeanalyzer/entrypoints/rules.py index aaba57b..0846f1e 100644 --- a/codeanalyzer/entrypoints/rules.py +++ b/codeanalyzer/entrypoints/rules.py @@ -14,6 +14,8 @@ import yaml +from codeanalyzer.entrypoints.matching import PatternError, validate_pattern + _SHIPPED = Path(__file__).with_name("rules.yml") _CONFIDENCE = {"declared", "certain", "heuristic"} @@ -116,10 +118,19 @@ def _confidence(raw: Dict[str, Any], origin: str) -> str: return c +def _match(raw: Dict[str, Any], origin: str) -> str: + match = _require(raw, "match", origin) + try: + validate_pattern(match) + except PatternError as exc: + raise RulesError(f"{origin}: rule {raw.get('id', raw)!r}: {exc}") from exc + return match + + def _decorator_rule(raw: Dict[str, Any], origin: str) -> DecoratorRule: return DecoratorRule( id=_require(raw, "id", origin), - match=_require(raw, "match", origin), + match=_match(raw, origin), confidence=_confidence(raw, origin), route=raw.get("route"), methods=raw.get("methods"), @@ -129,7 +140,7 @@ def _decorator_rule(raw: Dict[str, Any], origin: str) -> DecoratorRule: def _base_rule(raw: Dict[str, Any], origin: str) -> BaseRule: return BaseRule( id=_require(raw, "id", origin), - match=_require(raw, "match", origin), + match=_match(raw, origin), confidence=_confidence(raw, origin), transitive=bool(raw.get("transitive", False)), dispatch=list(raw.get("dispatch") or []), diff --git a/test/test_entrypoint_decorators.py b/test/test_entrypoint_decorators.py index bfb480b..de95539 100644 --- a/test/test_entrypoint_decorators.py +++ b/test/test_entrypoint_decorators.py @@ -10,6 +10,13 @@ def test_brace_alternation_and_wildcard(): assert not match_pattern("flask.Flask.route", "flask.Flask.routes") +def test_wildcard_does_not_cross_a_dot(): + assert match_pattern("rest_framework.viewsets.*", "rest_framework.viewsets.ModelViewSet") + assert not match_pattern( + "rest_framework.viewsets.*", "rest_framework.viewsets.mixins.ListModelMixin" + ) + + def test_route_and_methods_are_extracted(): fn = PyCallable(name="h", path="a.py", signature="a.h") fn.decorators.append( diff --git a/test/test_entrypoint_rules.py b/test/test_entrypoint_rules.py index e6af3cd..fb37bed 100644 --- a/test/test_entrypoint_rules.py +++ b/test/test_entrypoint_rules.py @@ -74,3 +74,23 @@ def test_bad_confidence_value_is_rejected(tmp_path): ) with pytest.raises(RulesError): load_rules([bad]) + + +def test_unbalanced_brace_in_match_pattern_is_rejected(tmp_path): + bad = tmp_path / "unbalanced.yml" + bad.write_text( + "version: 1\nframeworks:\n x:\n decorators:\n" + " - id: x.y\n match: 'flask.Flask.{get,post'\n" + ) + with pytest.raises(RulesError): + load_rules([bad]) + + +def test_nested_brace_in_match_pattern_is_rejected(tmp_path): + bad = tmp_path / "nested.yml" + bad.write_text( + "version: 1\nframeworks:\n x:\n decorators:\n" + " - id: x.y\n match: 'flask.{a,{b,c}}'\n" + ) + with pytest.raises(RulesError): + load_rules([bad]) From f7f181ef384ebcebd6e6449434179ae9301d47f2 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 10:04:44 -0400 Subject: [PATCH 16/23] feat(entrypoints): inheritance rules and the class/method dispatch split (#27) --- codeanalyzer/entrypoints/matching.py | 53 ++++++++++++++++++++++++++-- codeanalyzer/entrypoints/pipeline.py | 21 ++++++++++- test/test_entrypoint_bases.py | 46 ++++++++++++++++++++++++ 3 files changed, 117 insertions(+), 3 deletions(-) create mode 100644 test/test_entrypoint_bases.py diff --git a/codeanalyzer/entrypoints/matching.py b/codeanalyzer/entrypoints/matching.py index 6642e23..056d290 100644 --- a/codeanalyzer/entrypoints/matching.py +++ b/codeanalyzer/entrypoints/matching.py @@ -17,12 +17,12 @@ import ast import re -from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional +from typing import TYPE_CHECKING, Any, Callable, Dict, Iterable, List, Optional, Tuple from codeanalyzer.schema.py_schema import PyEntrypoint if TYPE_CHECKING: - from codeanalyzer.entrypoints.rules import DecoratorRule + from codeanalyzer.entrypoints.rules import BaseRule, DecoratorRule # Dispatch names that are HTTP verbs. DRF's ViewSet dispatch names # (list, retrieve, create, ...) are NOT verbs and must not be emitted as such. @@ -132,3 +132,52 @@ def entrypoints_from_decorators( ) ) return out + + +def entrypoints_from_bases( + cls, + framework: str, + rules: Iterable["BaseRule"], + ruleset: str, + resolve: Callable[[str], Optional[str]], +) -> Tuple[List[PyEntrypoint], Dict[str, List[PyEntrypoint]]]: + """Records for a routed class and for the methods the framework dispatches. + + ``resolve`` maps a written base-class name to its resolved qualified name + (identity when already qualified). Dispatch names are intersected with the + methods the class actually defines, so a ``ListView`` with only ``get`` + gains no phantom ``post`` entrypoint. + """ + class_eps: List[PyEntrypoint] = [] + method_eps: Dict[str, List[PyEntrypoint]] = {} + + for rule in rules: + if not any( + match_pattern(rule.match, resolve(b) or b) for b in (cls.base_classes or []) + ): + continue + class_eps.append( + PyEntrypoint( + framework=framework, + confidence=rule.confidence, + rule=rule.id, + ruleset=ruleset, + evidence=cls.signature, + ) + ) + defined = set((cls.callables or {}).keys()) + for name in rule.dispatch: + if name not in defined: + continue + method_eps.setdefault(name, []).append( + PyEntrypoint( + framework=framework, + confidence=rule.confidence, + rule=f"{rule.id}.dispatch", + ruleset=ruleset, + evidence=cls.signature, + http_methods=[name.upper()] if name in _HTTP_VERBS else [], + via=cls.id or None, + ) + ) + return class_eps, method_eps diff --git a/codeanalyzer/entrypoints/pipeline.py b/codeanalyzer/entrypoints/pipeline.py index 2a77769..d1273cd 100644 --- a/codeanalyzer/entrypoints/pipeline.py +++ b/codeanalyzer/entrypoints/pipeline.py @@ -10,6 +10,7 @@ from typing import Iterable, Iterator from codeanalyzer.entrypoints.detect import detected_frameworks +from codeanalyzer.entrypoints.matching import entrypoints_from_bases, entrypoints_from_decorators from codeanalyzer.entrypoints.rules import RuleSet, load_rules from codeanalyzer.schema.py_schema import PyApplication, PyCallable, PyClass from codeanalyzer.utils import logger @@ -36,11 +37,29 @@ def detect_entrypoints( def _run_stages(app: PyApplication, project_dir: Path, rules: RuleSet) -> None: - """Stages 0-4. Only stage 0 (framework detection) exists so far.""" + """Stages 0-4. Stage 0 (framework detection) and Stage 3 (decorator and + base-class matching) exist so far.""" app.entrypoint_report.rulesets = list(rules.rulesets) frameworks = detected_frameworks(app, project_dir, rules) app.entrypoint_report.frameworks_detected = sorted(frameworks) + ruleset_name = rules.rulesets[-1] if len(rules.rulesets) > 1 else "shipped" + for name in sorted(frameworks): + fw = rules.frameworks[name] + for node in _walk(app): + node.entrypoints.extend( + entrypoints_from_decorators(node, name, fw.decorators, ruleset_name) + ) + if isinstance(node, PyClass) and fw.bases: + class_eps, method_eps = entrypoints_from_bases( + node, name, fw.bases, ruleset_name, lambda b: b + ) + node.entrypoints.extend(class_eps) + for method_name, eps in method_eps.items(): + target = (node.callables or {}).get(method_name) + if target is not None: + target.entrypoints.extend(eps) + def _derive_flags(app: PyApplication) -> None: for node in _walk(app): diff --git a/test/test_entrypoint_bases.py b/test/test_entrypoint_bases.py new file mode 100644 index 0000000..a887d59 --- /dev/null +++ b/test/test_entrypoint_bases.py @@ -0,0 +1,46 @@ +from codeanalyzer.entrypoints.matching import entrypoints_from_bases +from codeanalyzer.entrypoints.rules import BaseRule +from codeanalyzer.schema.py_schema import PyCallable, PyClass + +RULE = BaseRule( + id="drf.apiview", + match="rest_framework.views.APIView", + transitive=True, + dispatch=["get", "post", "put"], +) + + +def _cls(*methods: str, bases=("rest_framework.views.APIView",)) -> PyClass: + return PyClass( + name="V", + signature="a.V", + base_classes=list(bases), + callables={m: PyCallable(name=m, path="a.py", signature=f"a.V.{m}") for m in methods}, + ) + + +def test_class_is_flagged_and_only_defined_methods_dispatch(): + cls = _cls("get") # defines get, not post + class_eps, method_eps = entrypoints_from_bases(cls, "drf", [RULE], "shipped", lambda b: b) + assert len(class_eps) == 1 + assert list(method_eps) == ["get"], "no phantom post entrypoint" + + +def test_methods_point_back_at_the_routed_class_via(): + cls = _cls("get") + cls.id = "can://python/app/a.py/V" + _, method_eps = entrypoints_from_bases(cls, "drf", [RULE], "shipped", lambda b: b) + assert method_eps["get"][0].via == "can://python/app/a.py/V" + + +def test_transitive_base_resolves_one_hop(): + cls = _cls("get", bases=("app.BaseView",)) + resolve = {"app.BaseView": "rest_framework.views.APIView"}.get + class_eps, _ = entrypoints_from_bases(cls, "drf", [RULE], "shipped", resolve) + assert len(class_eps) == 1 + + +def test_unrelated_class_is_not_flagged(): + cls = _cls("get", bases=("object",)) + class_eps, method_eps = entrypoints_from_bases(cls, "drf", [RULE], "shipped", lambda b: b) + assert class_eps == [] and method_eps == {} From 9cbb34ad18bc8905236edb8f83f5c2faeffb89f7 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 10:12:18 -0400 Subject: [PATCH 17/23] fix(entrypoints): per-rule ruleset origin and import-resolved base matching (#27) Attribute each PyEntrypoint's ruleset to the rule's own load origin instead of the last-loaded ruleset, and resolve written base-class names against the owning module's import table so bases: rules match real, idiomatically imported code instead of only fully-qualified spellings. --- codeanalyzer/entrypoints/matching.py | 9 ++- codeanalyzer/entrypoints/pipeline.py | 79 ++++++++++++++------- codeanalyzer/entrypoints/rules.py | 4 ++ test/test_entrypoint_bases.py | 8 +-- test/test_entrypoint_decorators.py | 6 +- test/test_entrypoint_pipeline.py | 102 +++++++++++++++++++++++++++ 6 files changed, 172 insertions(+), 36 deletions(-) diff --git a/codeanalyzer/entrypoints/matching.py b/codeanalyzer/entrypoints/matching.py index 056d290..91857e7 100644 --- a/codeanalyzer/entrypoints/matching.py +++ b/codeanalyzer/entrypoints/matching.py @@ -113,7 +113,7 @@ def _methods_of(dec, spec: Optional[Dict[str, Any]]) -> List[str]: def entrypoints_from_decorators( - node, framework: str, rules: Iterable["DecoratorRule"], ruleset: str + node, framework: str, rules: Iterable["DecoratorRule"] ) -> List[PyEntrypoint]: out: List[PyEntrypoint] = [] for dec in getattr(node, "decorators", []) or []: @@ -125,7 +125,7 @@ def entrypoints_from_decorators( framework=framework, confidence=rule.confidence, rule=rule.id, - ruleset=ruleset, + ruleset=rule.origin, evidence=dec.qualified_name, route=_route_of(dec, rule.route), http_methods=_methods_of(dec, rule.methods), @@ -138,7 +138,6 @@ def entrypoints_from_bases( cls, framework: str, rules: Iterable["BaseRule"], - ruleset: str, resolve: Callable[[str], Optional[str]], ) -> Tuple[List[PyEntrypoint], Dict[str, List[PyEntrypoint]]]: """Records for a routed class and for the methods the framework dispatches. @@ -161,7 +160,7 @@ def entrypoints_from_bases( framework=framework, confidence=rule.confidence, rule=rule.id, - ruleset=ruleset, + ruleset=rule.origin, evidence=cls.signature, ) ) @@ -174,7 +173,7 @@ def entrypoints_from_bases( framework=framework, confidence=rule.confidence, rule=f"{rule.id}.dispatch", - ruleset=ruleset, + ruleset=rule.origin, evidence=cls.signature, http_methods=[name.upper()] if name in _HTTP_VERBS else [], via=cls.id or None, diff --git a/codeanalyzer/entrypoints/pipeline.py b/codeanalyzer/entrypoints/pipeline.py index d1273cd..1f98937 100644 --- a/codeanalyzer/entrypoints/pipeline.py +++ b/codeanalyzer/entrypoints/pipeline.py @@ -7,12 +7,12 @@ from __future__ import annotations from pathlib import Path -from typing import Iterable, Iterator +from typing import Dict, Iterable, Iterator from codeanalyzer.entrypoints.detect import detected_frameworks from codeanalyzer.entrypoints.matching import entrypoints_from_bases, entrypoints_from_decorators from codeanalyzer.entrypoints.rules import RuleSet, load_rules -from codeanalyzer.schema.py_schema import PyApplication, PyCallable, PyClass +from codeanalyzer.schema.py_schema import PyApplication, PyCallable, PyClass, PyModule from codeanalyzer.utils import logger @@ -38,27 +38,54 @@ def detect_entrypoints( def _run_stages(app: PyApplication, project_dir: Path, rules: RuleSet) -> None: """Stages 0-4. Stage 0 (framework detection) and Stage 3 (decorator and - base-class matching) exist so far.""" + base-class matching) exist so far. + + Base-class resolution needs the OWNING MODULE's import table (a written + base like ``APIView`` only resolves via that module's own + ``from rest_framework.views import APIView``), so this walks module by + module rather than the whole app flat, building one resolver per module. + """ app.entrypoint_report.rulesets = list(rules.rulesets) frameworks = detected_frameworks(app, project_dir, rules) app.entrypoint_report.frameworks_detected = sorted(frameworks) - ruleset_name = rules.rulesets[-1] if len(rules.rulesets) > 1 else "shipped" - for name in sorted(frameworks): - fw = rules.frameworks[name] - for node in _walk(app): - node.entrypoints.extend( - entrypoints_from_decorators(node, name, fw.decorators, ruleset_name) - ) - if isinstance(node, PyClass) and fw.bases: - class_eps, method_eps = entrypoints_from_bases( - node, name, fw.bases, ruleset_name, lambda b: b - ) - node.entrypoints.extend(class_eps) - for method_name, eps in method_eps.items(): - target = (node.callables or {}).get(method_name) - if target is not None: - target.entrypoints.extend(eps) + names = sorted(frameworks) + for mod in app.symbol_table.values(): + resolve = _base_resolver(mod) + for node in _walk_module(mod): + for name in names: + fw = rules.frameworks[name] + node.entrypoints.extend(entrypoints_from_decorators(node, name, fw.decorators)) + if isinstance(node, PyClass) and fw.bases: + class_eps, method_eps = entrypoints_from_bases( + node, name, fw.bases, resolve + ) + node.entrypoints.extend(class_eps) + for method_name, eps in method_eps.items(): + target = (node.callables or {}).get(method_name) + if target is not None: + target.entrypoints.extend(eps) + + +def _base_resolver(mod: PyModule): + """A per-module ``resolve`` callable for ``entrypoints_from_bases``, built + from the module's own import table -- exact data already on the node, + never a Jedi guess. Covers ``from x.y import Z[ as W]`` and + ``import x.y[ as z]``, plus a dotted base (``views.APIView``) whose head + is the imported name. A base the import table has no mapping for is + returned unchanged -- under-approximate rather than guess. + """ + aliases: Dict[str, str] = {} + for imp in mod.imports or []: + original = imp.alias or imp.name + aliases[imp.name] = imp.module if imp.module == original else f"{imp.module}.{original}" + + def resolve(written: str) -> str: + head, _, rest = written.partition(".") + target = aliases.get(head) + return f"{target}.{rest}" if target and rest else (target or written) + + return resolve def _derive_flags(app: PyApplication) -> None: @@ -66,7 +93,7 @@ def _derive_flags(app: PyApplication) -> None: node.is_entrypoint = bool(node.entrypoints) -def _walk(app: PyApplication) -> Iterator[object]: +def _walk_module(mod: PyModule) -> Iterator[object]: def walk_callable(c: PyCallable) -> Iterator[object]: yield c for inner in (c.callables or {}).values(): @@ -81,8 +108,12 @@ def walk_class(k: PyClass) -> Iterator[object]: for inner in (k.types or {}).values(): yield from walk_class(inner) + for fn in (mod.functions or {}).values(): + yield from walk_callable(fn) + for cls in (mod.types or {}).values(): + yield from walk_class(cls) + + +def _walk(app: PyApplication) -> Iterator[object]: for mod in app.symbol_table.values(): - for fn in (mod.functions or {}).values(): - yield from walk_callable(fn) - for cls in (mod.types or {}).values(): - yield from walk_class(cls) + yield from _walk_module(mod) diff --git a/codeanalyzer/entrypoints/rules.py b/codeanalyzer/entrypoints/rules.py index 0846f1e..d97d84c 100644 --- a/codeanalyzer/entrypoints/rules.py +++ b/codeanalyzer/entrypoints/rules.py @@ -31,6 +31,7 @@ class DecoratorRule: confidence: str = "certain" route: Optional[Dict[str, Any]] = None methods: Optional[Dict[str, Any]] = None + origin: str = "shipped" @dataclass @@ -40,6 +41,7 @@ class BaseRule: confidence: str = "certain" transitive: bool = False dispatch: List[str] = field(default_factory=list) + origin: str = "shipped" @dataclass @@ -134,6 +136,7 @@ def _decorator_rule(raw: Dict[str, Any], origin: str) -> DecoratorRule: confidence=_confidence(raw, origin), route=raw.get("route"), methods=raw.get("methods"), + origin=origin, ) @@ -144,4 +147,5 @@ def _base_rule(raw: Dict[str, Any], origin: str) -> BaseRule: confidence=_confidence(raw, origin), transitive=bool(raw.get("transitive", False)), dispatch=list(raw.get("dispatch") or []), + origin=origin, ) diff --git a/test/test_entrypoint_bases.py b/test/test_entrypoint_bases.py index a887d59..c4b2fbe 100644 --- a/test/test_entrypoint_bases.py +++ b/test/test_entrypoint_bases.py @@ -21,7 +21,7 @@ def _cls(*methods: str, bases=("rest_framework.views.APIView",)) -> PyClass: def test_class_is_flagged_and_only_defined_methods_dispatch(): cls = _cls("get") # defines get, not post - class_eps, method_eps = entrypoints_from_bases(cls, "drf", [RULE], "shipped", lambda b: b) + class_eps, method_eps = entrypoints_from_bases(cls, "drf", [RULE], lambda b: b) assert len(class_eps) == 1 assert list(method_eps) == ["get"], "no phantom post entrypoint" @@ -29,18 +29,18 @@ def test_class_is_flagged_and_only_defined_methods_dispatch(): def test_methods_point_back_at_the_routed_class_via(): cls = _cls("get") cls.id = "can://python/app/a.py/V" - _, method_eps = entrypoints_from_bases(cls, "drf", [RULE], "shipped", lambda b: b) + _, method_eps = entrypoints_from_bases(cls, "drf", [RULE], lambda b: b) assert method_eps["get"][0].via == "can://python/app/a.py/V" def test_transitive_base_resolves_one_hop(): cls = _cls("get", bases=("app.BaseView",)) resolve = {"app.BaseView": "rest_framework.views.APIView"}.get - class_eps, _ = entrypoints_from_bases(cls, "drf", [RULE], "shipped", resolve) + class_eps, _ = entrypoints_from_bases(cls, "drf", [RULE], resolve) assert len(class_eps) == 1 def test_unrelated_class_is_not_flagged(): cls = _cls("get", bases=("object",)) - class_eps, method_eps = entrypoints_from_bases(cls, "drf", [RULE], "shipped", lambda b: b) + class_eps, method_eps = entrypoints_from_bases(cls, "drf", [RULE], lambda b: b) assert class_eps == [] and method_eps == {} diff --git a/test/test_entrypoint_decorators.py b/test/test_entrypoint_decorators.py index de95539..ef64003 100644 --- a/test/test_entrypoint_decorators.py +++ b/test/test_entrypoint_decorators.py @@ -33,7 +33,7 @@ def test_route_and_methods_are_extracted(): route={"from": "positional", "index": 0}, methods={"from": "keyword", "name": "methods", "default": ["GET"]}, ) - (ep,) = entrypoints_from_decorators(fn, "flask", [rule], "shipped") + (ep,) = entrypoints_from_decorators(fn, "flask", [rule]) assert ep.route == "/products" assert ep.http_methods == ["POST"] assert ep.rule == "flask.route" and ep.ruleset == "shipped" @@ -51,7 +51,7 @@ def test_verb_comes_from_the_matched_suffix(): route={"from": "positional", "index": 0}, methods={"from": "match_suffix"}, ) - (ep,) = entrypoints_from_decorators(fn, "fastapi", [rule], "shipped") + (ep,) = entrypoints_from_decorators(fn, "fastapi", [rule]) assert ep.http_methods == ["POST"] @@ -60,4 +60,4 @@ def test_unresolved_decorator_never_matches(): fn = PyCallable(name="h", path="a.py", signature="a.h") fn.decorators.append(PyDecorator(name="app.route", qualified_name=None)) rule = DecoratorRule(id="flask.route", match="flask.Flask.route") - assert entrypoints_from_decorators(fn, "flask", [rule], "shipped") == [] + assert entrypoints_from_decorators(fn, "flask", [rule]) == [] diff --git a/test/test_entrypoint_pipeline.py b/test/test_entrypoint_pipeline.py index b505d4f..57738de 100644 --- a/test/test_entrypoint_pipeline.py +++ b/test/test_entrypoint_pipeline.py @@ -54,3 +54,105 @@ def test_malformed_user_rules_file_raises_instead_of_being_swallowed(tmp_path: P detect_entrypoints(app, tmp_path, rule_paths=(bad_rules,)) assert app.entrypoint_report.errors == [] + + +def test_ruleset_provenance_distinguishes_shipped_from_user_rules(tmp_path: Path): + """A shipped rule's record must say "shipped" even when a user rules + file is also loaded -- the ruleset field exists so someone debugging a + surprising flag can find which file produced it.""" + from codeanalyzer.schema.py_schema import PyCallable, PyDecorator, PyImport, PyModule + + user_rules = tmp_path / "user.yml" + user_rules.write_text( + "frameworks:\n" + " inhouse:\n" + " detect: [inhouse]\n" + " decorators:\n" + " - id: inhouse.handler\n" + " match: 'inhouse.app.handler'\n" + ) + + shipped_fn = PyCallable(name="f", path="a.py", signature="a.f") + shipped_fn.decorators.append(PyDecorator(name="app.route", qualified_name="flask.Flask.route")) + user_fn = PyCallable(name="g", path="a.py", signature="a.g") + user_fn.decorators.append(PyDecorator(name="handler", qualified_name="inhouse.app.handler")) + + app = PyApplication( + symbol_table={ + "a.py": PyModule( + file_path="a.py", + module_name="a", + functions={"f": shipped_fn, "g": user_fn}, + imports=[ + PyImport(module="flask", name="Flask"), + PyImport(module="inhouse", name="app"), + ], + ) + } + ) + detect_entrypoints(app, tmp_path, rule_paths=(user_rules,)) + + assert shipped_fn.entrypoints[0].ruleset == "shipped" + assert user_fn.entrypoints[0].ruleset == f"user:{user_rules}" + + +def test_direct_base_class_is_flagged_when_the_import_resolves_it(tmp_path: Path): + """``class V(APIView)`` under ``from rest_framework.views import APIView`` + is the idiomatic spelling -- base_classes stores the written name + ``"APIView"``, and it must resolve via the module's own import table.""" + from codeanalyzer.schema.py_schema import PyClass, PyImport, PyModule + + cls = PyClass(name="V", signature="a.V", base_classes=["APIView"]) + app = PyApplication( + symbol_table={ + "a.py": PyModule( + file_path="a.py", + module_name="a", + types={"a.V": cls}, + imports=[PyImport(module="rest_framework.views", name="APIView")], + ) + } + ) + detect_entrypoints(app, tmp_path) + assert cls.is_entrypoint is True + + +def test_direct_base_class_is_not_flagged_without_the_import(tmp_path: Path): + """``rest_framework`` is imported (so the drf framework gate passes) but + ``APIView`` itself is never imported into this module -- the written + "APIView" base has nothing to resolve against and must not be flagged.""" + from codeanalyzer.schema.py_schema import PyClass, PyImport, PyModule + + cls = PyClass(name="V", signature="a.V", base_classes=["APIView"]) + app = PyApplication( + symbol_table={ + "a.py": PyModule( + file_path="a.py", + module_name="a", + types={"a.V": cls}, + imports=[PyImport(module="rest_framework", name="serializers")], + ) + } + ) + detect_entrypoints(app, tmp_path) + assert cls.is_entrypoint is False + + +def test_dotted_base_class_resolves_through_a_module_import(tmp_path: Path): + """``class V(views.APIView)`` under ``from rest_framework import views`` + -- the dotted base's head ("views") is the imported name.""" + from codeanalyzer.schema.py_schema import PyClass, PyImport, PyModule + + cls = PyClass(name="V", signature="a.V", base_classes=["views.APIView"]) + app = PyApplication( + symbol_table={ + "a.py": PyModule( + file_path="a.py", + module_name="a", + types={"a.V": cls}, + imports=[PyImport(module="rest_framework", name="views")], + ) + } + ) + detect_entrypoints(app, tmp_path) + assert cls.is_entrypoint is True From 083abce4916a877eb44c535311edb06f36f830ed Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 10:18:32 -0400 Subject: [PATCH 18/23] feat(entrypoints): end-to-end detection and Neo4j projection (#27) Adds the local-decorator fixture and e2e test proving detection through the real CLI, including the --entrypoint-rules path. The fixture ships a requirements.txt naming its detect package so Stage 0 exercises the manifest-detection path (no other test covers it). Projects is_entrypoint/entrypoint_frameworks onto :PyCallable and :PyClass in the Neo4j schema and row builder, and regenerates the schema snapshot. --- codeanalyzer/neo4j/project.py | 4 ++ codeanalyzer/neo4j/schema.py | 4 ++ schema.neo4j.json | 8 ++- .../entrypoints_local/app.py | 15 ++++++ .../entrypoints_local/requirements.txt | 1 + .../entrypoints_local/rules.yml | 9 ++++ test/test_entrypoints_e2e.py | 49 +++++++++++++++++++ 7 files changed, 88 insertions(+), 2 deletions(-) create mode 100644 test/fixtures/single_functionalities/entrypoints_local/app.py create mode 100644 test/fixtures/single_functionalities/entrypoints_local/requirements.txt create mode 100644 test/fixtures/single_functionalities/entrypoints_local/rules.yml create mode 100644 test/test_entrypoints_e2e.py diff --git a/codeanalyzer/neo4j/project.py b/codeanalyzer/neo4j/project.py index f7b01cd..b838b8a 100644 --- a/codeanalyzer/neo4j/project.py +++ b/codeanalyzer/neo4j/project.py @@ -517,6 +517,8 @@ def _class_props(cl: PyClass, file_key: str, source: str) -> Props: "start_line": cl.start_line, "end_line": cl.end_line, "_module": file_key, + "is_entrypoint": bool(cl.entrypoints), + "entrypoint_frameworks": sorted({e.framework for e in (cl.entrypoints or [])}), } ) @@ -539,6 +541,8 @@ def _callable_props(c: PyCallable, file_key: str, source: str) -> Props: "parameters_json": _stringify_if(c.parameters), "accessed_symbols_json": _stringify_if(c.accessed_symbols), "_module": file_key, + "is_entrypoint": bool(c.entrypoints), + "entrypoint_frameworks": sorted({e.framework for e in (c.entrypoints or [])}), } ) diff --git a/codeanalyzer/neo4j/schema.py b/codeanalyzer/neo4j/schema.py index 0216357..a1fd2aa 100644 --- a/codeanalyzer/neo4j/schema.py +++ b/codeanalyzer/neo4j/schema.py @@ -105,6 +105,8 @@ class RelType: "docstring": "string", **_SPAN, "_module": "string", + "is_entrypoint": "boolean", + "entrypoint_frameworks": "string[]", }, ), NodeLabel( @@ -126,6 +128,8 @@ class RelType: "parameters_json": "string", "accessed_symbols_json": "string", "_module": "string", + "is_entrypoint": "boolean", + "entrypoint_frameworks": "string[]", }, ), NodeLabel( diff --git a/schema.neo4j.json b/schema.neo4j.json index f3035d7..aaf97d8 100644 --- a/schema.neo4j.json +++ b/schema.neo4j.json @@ -45,7 +45,9 @@ "docstring": "string", "start_line": "integer", "end_line": "integer", - "_module": "string" + "_module": "string", + "is_entrypoint": "boolean", + "entrypoint_frameworks": "string[]" } }, { @@ -67,7 +69,9 @@ "decorators": "string[]", "parameters_json": "string", "accessed_symbols_json": "string", - "_module": "string" + "_module": "string", + "is_entrypoint": "boolean", + "entrypoint_frameworks": "string[]" } }, { diff --git a/test/fixtures/single_functionalities/entrypoints_local/app.py b/test/fixtures/single_functionalities/entrypoints_local/app.py new file mode 100644 index 0000000..23011e9 --- /dev/null +++ b/test/fixtures/single_functionalities/entrypoints_local/app.py @@ -0,0 +1,15 @@ +def route(path, methods=None): + """Stands in for a framework's routing decorator.""" + def deco(fn): + return fn + return deco + + +@route("/products", methods=["POST"]) +def create_product(): + return helper() + + +def helper(): + """Called only internally - must NOT be flagged.""" + return {} diff --git a/test/fixtures/single_functionalities/entrypoints_local/requirements.txt b/test/fixtures/single_functionalities/entrypoints_local/requirements.txt new file mode 100644 index 0000000..b80f0bd --- /dev/null +++ b/test/fixtures/single_functionalities/entrypoints_local/requirements.txt @@ -0,0 +1 @@ +app diff --git a/test/fixtures/single_functionalities/entrypoints_local/rules.yml b/test/fixtures/single_functionalities/entrypoints_local/rules.yml new file mode 100644 index 0000000..1632927 --- /dev/null +++ b/test/fixtures/single_functionalities/entrypoints_local/rules.yml @@ -0,0 +1,9 @@ +version: 1 +frameworks: + inhouse: + detect: [app] + decorators: + - id: inhouse.route + match: "app.route" + route: {from: positional, index: 0} + methods: {from: keyword, name: methods, default: [GET]} diff --git a/test/test_entrypoints_e2e.py b/test/test_entrypoints_e2e.py new file mode 100644 index 0000000..198e796 --- /dev/null +++ b/test/test_entrypoints_e2e.py @@ -0,0 +1,49 @@ +"""End-to-end coverage: entrypoint detection driven through the real CLI, +including the ``--entrypoint-rules`` user-rules path (#27). + +A local decorator (not Flask) so matching resolves deterministically via +Jedi without a venv/network dependency -- see the fixture's ``rules.yml`` +for why. The fixture also carries a ``requirements.txt`` naming ``app`` so +Stage 0 detection has a real manifest signal to key off of (the fixture +module is not imported by anything, so an import scan alone would never +see it). +""" +import json +import subprocess +from pathlib import Path + +FIXTURE = Path(__file__).parent / "fixtures" / "single_functionalities" / "entrypoints_local" + + +def test_decorated_function_flagged_and_helper_not(tmp_path): + subprocess.run( + [ + "uv", "run", "canpy", + "-i", str(FIXTURE), + "-a", "1", + "-o", str(tmp_path), + "--no-venv", + # Cache defaults to the input dir; keep it in tmp_path so the + # checked-in fixture directory is never mutated by a test run + # and each run starts from a clean (entrypoint-free) cache. + "--cache-dir", str(tmp_path / "cache"), + "--entrypoint-rules", str(FIXTURE / "rules.yml"), + ], + check=True, + ) + data = json.loads((tmp_path / "analysis.json").read_text()) + fns = data["application"]["symbol_table"]["app.py"]["functions"] + + create = fns["create_product"] + assert create["is_entrypoint"] is True + (ep,) = create["entrypoints"] + assert ep["framework"] == "inhouse" and ep["rule"] == "inhouse.route" + assert ep["route"] == "/products" and ep["http_methods"] == ["POST"] + assert ep["ruleset"].startswith("user:") + + assert fns["helper"]["is_entrypoint"] is False + assert fns["helper"]["entrypoints"] == [] + + report = data["application"]["entrypoint_report"] + assert "inhouse" in report["frameworks_detected"] + assert report["errors"] == [] From 998697411a12bbb0e8996433e1be4a97f87f1865 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 10:22:28 -0400 Subject: [PATCH 19/23] fix(entrypoints): clear stale records before each detection pass (#27) detect_entrypoints extended entrypoints onto whatever list a node already had. On a warm cache, core.py reuses the same cached PyModule/PyCallable objects when a file is unchanged, so a second run appended duplicate PyEntrypoint records onto the reused nodes. _run_stages now opens with a single full-app clear before Stage 0 detection, rather than clearing per-node during the walk -- an inline clear would erase entrypoints_from_bases records that are written onto a method before _walk_module visits that method directly. Adds a unit test that runs the pass twice over the same PyApplication and asserts the record list is unchanged, and fixes an existing test that relied on hand-seeding entrypoints (no longer valid once the pass owns and clears the list on every run). --- codeanalyzer/entrypoints/pipeline.py | 12 ++++++++ test/test_entrypoint_pipeline.py | 46 +++++++++++++++++++++++++--- 2 files changed, 53 insertions(+), 5 deletions(-) diff --git a/codeanalyzer/entrypoints/pipeline.py b/codeanalyzer/entrypoints/pipeline.py index 1f98937..54586e8 100644 --- a/codeanalyzer/entrypoints/pipeline.py +++ b/codeanalyzer/entrypoints/pipeline.py @@ -44,7 +44,19 @@ def _run_stages(app: PyApplication, project_dir: Path, rules: RuleSet) -> None: base like ``APIView`` only resolves via that module's own ``from rest_framework.views import APIView``), so this walks module by module rather than the whole app flat, building one resolver per module. + + Clears every node's ``entrypoints`` first: on a warm cache, + ``_build_symbol_table`` reuses the SAME cached ``PyModule``/``PyCallable`` + objects when a file is unchanged, so without this clear a second run + would ``extend`` onto records already written by the first run and + duplicate them. A single full clear up front (rather than clearing each + node as it's visited) avoids wiping ``entrypoints_from_bases`` records + that ``_walk_module`` writes onto a method before visiting that method + directly. """ + for node in _walk(app): + node.entrypoints = [] + app.entrypoint_report.rulesets = list(rules.rulesets) frameworks = detected_frameworks(app, project_dir, rules) app.entrypoint_report.frameworks_detected = sorted(frameworks) diff --git a/test/test_entrypoint_pipeline.py b/test/test_entrypoint_pipeline.py index 57738de..65b63b9 100644 --- a/test/test_entrypoint_pipeline.py +++ b/test/test_entrypoint_pipeline.py @@ -27,15 +27,22 @@ def boom(*a, **k): def test_derives_is_entrypoint_from_the_list(tmp_path: Path): - from codeanalyzer.schema.py_schema import PyCallable, PyEntrypoint, PyModule + """`entrypoints` is written entirely by the pass itself (it clears and + rebuilds the list every run, see the duplication regression test below) + -- so drive this through a real decorator match rather than hand-seeding + the list, and check `_derive_flags` sets the boolean from the result.""" + from codeanalyzer.schema.py_schema import PyCallable, PyDecorator, PyImport, PyModule fn = PyCallable(name="f", path="a.py", signature="a.f") - fn.entrypoints.append( - PyEntrypoint(framework="flask", confidence="certain", rule="flask.route", ruleset="shipped") - ) + fn.decorators.append(PyDecorator(name="route", qualified_name="flask.Flask.route")) app = PyApplication( symbol_table={ - "a.py": PyModule(file_path="a.py", module_name="a", functions={"f": fn}) + "a.py": PyModule( + file_path="a.py", + module_name="a", + functions={"f": fn}, + imports=[PyImport(module="flask", name="Flask")], + ) } ) detect_entrypoints(app, tmp_path) @@ -156,3 +163,32 @@ def test_dotted_base_class_resolves_through_a_module_import(tmp_path: Path): ) detect_entrypoints(app, tmp_path) assert cls.is_entrypoint is True + + +def test_running_the_pass_twice_does_not_duplicate_entrypoints(tmp_path: Path): + """#27 regression: on a warm cache, `_build_symbol_table` reuses the SAME + cached PyModule/PyCallable objects across runs. `detect_entrypoints` must + be safe to call again on that same PyApplication without appending + duplicate PyEntrypoint records onto the reused nodes.""" + from codeanalyzer.schema.py_schema import PyCallable, PyDecorator, PyImport, PyModule + + fn = PyCallable(name="f", path="a.py", signature="a.f") + fn.decorators.append(PyDecorator(name="route", qualified_name="flask.Flask.route")) + app = PyApplication( + symbol_table={ + "a.py": PyModule( + file_path="a.py", + module_name="a", + functions={"f": fn}, + imports=[PyImport(module="flask", name="Flask")], + ) + } + ) + + detect_entrypoints(app, tmp_path) + first = [e.model_dump() for e in fn.entrypoints] + assert len(first) == 1 + + detect_entrypoints(app, tmp_path) + second = [e.model_dump() for e in fn.entrypoints] + assert second == first From 1fbe90083302f64a3320cfa9317ae2c81181ea0f Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 10:40:23 -0400 Subject: [PATCH 20/23] fix(entrypoints): repoint decorator rules at Jedi's real resolution paths Every shipped decorator rule matched the framework's public re-export path (flask.Flask.route, fastapi.FastAPI.get, celery.Celery.task, click.command, ...) but PyDecorator.qualified_name is Jedi's DEFINITION path (flask.sansio.scaffold.Scaffold.route, fastapi.applications.FastAPI.get, celery.app.base.Celery.task, click.decorators.command, ...), so none of them ever matched a real app. Verified each pattern against the installed package via jedi.Script.infer(...).full_name. Also: - add a django: block (bases: django.views.generic.* dispatch) so a Django project reports frameworks_detected: ["django"] instead of [] -- base-class matching resolves against the module's own import table, not Jedi's definition path, so the public spelling is correct there. - normalize detect.py's case handling: imported package names are now lowercased alongside the already-lowercased manifest names, and detect: values are lowercased before the membership check, so detect: [Flask] matches a real `import flask`. - reject unknown top-level rules.yml keys (e.g. the not-yet-implemented declared: block) instead of silently ignoring them. --- codeanalyzer/entrypoints/detect.py | 8 +- codeanalyzer/entrypoints/rules.py | 8 + codeanalyzer/entrypoints/rules.yml | 39 +++-- .../specs/call-site-body-convergence.md | 142 ------------------ test/test_entrypoint_detect.py | 10 ++ test/test_entrypoint_pipeline.py | 6 +- test/test_entrypoint_rules.py | 21 +++ 7 files changed, 78 insertions(+), 156 deletions(-) delete mode 100644 docs/design/specs/call-site-body-convergence.md diff --git a/codeanalyzer/entrypoints/detect.py b/codeanalyzer/entrypoints/detect.py index 072d14e..7330e60 100644 --- a/codeanalyzer/entrypoints/detect.py +++ b/codeanalyzer/entrypoints/detect.py @@ -21,11 +21,15 @@ def detected_frameworks(app: PyApplication, project_dir: Path, rules: RuleSet) -> Set[str]: + # `present` (imports, manifest names) and `detect:` values are both + # lowercased before comparison -- manifest names were already lowercased + # (PyPI/pip is case-insensitive) but imports and `detect:` were not, so + # a `detect: [Flask]` user rule silently never matched a `flask` import. present = _imported_packages(app) | _manifest_packages(project_dir) return { name for name, fw in rules.frameworks.items() - if any(pkg in present for pkg in (fw.detect or [name])) + if any(pkg.lower() in present for pkg in (fw.detect or [name])) } @@ -38,7 +42,7 @@ def _imported_packages(app: PyApplication) -> Set[str]: spelling = (getattr(imp, "module", "") or getattr(imp, "name", "") or "") spelling = spelling.lstrip(".") if spelling: - out.add(spelling.split(".", 1)[0]) + out.add(spelling.split(".", 1)[0].lower()) return out diff --git a/codeanalyzer/entrypoints/rules.py b/codeanalyzer/entrypoints/rules.py index d97d84c..cd641dc 100644 --- a/codeanalyzer/entrypoints/rules.py +++ b/codeanalyzer/entrypoints/rules.py @@ -18,6 +18,11 @@ _SHIPPED = Path(__file__).with_name("rules.yml") _CONFIDENCE = {"declared", "certain", "heuristic"} +# `declared:` (readers) and per-framework routing engines are real spec +# blocks (Units 4-5) not implemented yet; they are deliberately absent here +# rather than accepted-and-ignored, so a user file using them fails loudly +# instead of loading clean and doing nothing. +_TOP_LEVEL_KEYS = {"version", "frameworks", "disable"} class RulesError(Exception): @@ -79,6 +84,9 @@ def _read(path: Path) -> Dict[str, Any]: def _merge(out: RuleSet, data: Dict[str, Any], origin: str) -> None: + unknown = sorted(set(data) - _TOP_LEVEL_KEYS) + if unknown: + raise RulesError(f"{origin}: unknown top-level key(s): {', '.join(unknown)}") out.rulesets.append(origin) disabled = set(_disable_list(data, origin)) frameworks = data.get("frameworks") or {} diff --git a/codeanalyzer/entrypoints/rules.yml b/codeanalyzer/entrypoints/rules.yml index ded0b2f..63cd5b5 100644 --- a/codeanalyzer/entrypoints/rules.yml +++ b/codeanalyzer/entrypoints/rules.yml @@ -4,12 +4,16 @@ frameworks: flask: detect: [flask] decorators: + # Flask 3's Flask/Blueprint decorators are all inherited from one base + # (flask.sansio.scaffold.Scaffold); matching is on Jedi's resolved + # DEFINITION path, not the public `flask.Flask`/`flask.Blueprint` + # spelling, so one rule now covers both call sites. - id: flask.route - match: "flask.Flask.route" + match: "flask.sansio.scaffold.Scaffold.route" route: {from: positional, index: 0} methods: {from: keyword, name: methods, default: [GET]} - id: flask.bp-verb - match: "flask.Blueprint.{get,post,put,delete,patch}" + match: "flask.sansio.scaffold.Scaffold.{get,post,put,delete,patch}" route: {from: positional, index: 0} methods: {from: match_suffix} bases: @@ -21,33 +25,37 @@ frameworks: fastapi: detect: [fastapi] decorators: + # FastAPI's own get/post/... are defined directly on the FastAPI class + # (fastapi/applications.py); APIRouter's are a distinct class + # (fastapi/routing.py) -- Jedi resolves each to its own module, so + # these stay two rules. - id: fastapi.verb - match: "fastapi.FastAPI.{get,post,put,delete,patch,head,options}" + match: "fastapi.applications.FastAPI.{get,post,put,delete,patch,head,options}" route: {from: positional, index: 0} methods: {from: match_suffix} - id: fastapi.router-verb - match: "fastapi.APIRouter.{get,post,put,delete,patch}" + match: "fastapi.routing.APIRouter.{get,post,put,delete,patch}" route: {from: positional, index: 0} methods: {from: match_suffix} - id: fastapi.websocket - match: "fastapi.FastAPI.websocket" + match: "fastapi.applications.FastAPI.websocket" route: {from: positional, index: 0} celery: detect: [celery] decorators: - id: celery.shared-task - match: "celery.shared_task" + match: "celery.app.shared_task" - id: celery.task - match: "celery.Celery.task" + match: "celery.app.base.Celery.task" click: detect: [click, typer] decorators: - id: click.command - match: "click.{command,group}" + match: "click.decorators.{command,group}" - id: typer.command - match: "typer.Typer.command" + match: "typer.main.Typer.command" drf: detect: [rest_framework] @@ -65,3 +73,16 @@ frameworks: match: "rest_framework.viewsets.*" transitive: true dispatch: [list, retrieve, create, update, partial_update, destroy] + + django: + detect: [django] + bases: + # Base-class matching resolves against the module's own import table + # (the WRITTEN spelling, e.g. `from django.views.generic import + # ListView`), never Jedi's definition path -- so the public + # `django.views.generic.*` path is correct here, unlike the decorator + # rules above. + - id: django.cbv + match: "django.views.generic.*" + transitive: true + dispatch: [get, post, put, patch, delete, head, options] diff --git a/docs/design/specs/call-site-body-convergence.md b/docs/design/specs/call-site-body-convergence.md deleted file mode 100644 index 658d062..0000000 --- a/docs/design/specs/call-site-body-convergence.md +++ /dev/null @@ -1,142 +0,0 @@ -# Spec: converge `call_sites[]` into `body{}` — separating the IR from the wire format - -Status: draft for review -Date: 2026-08-19 -Scope: `codeanalyzer-python` schema v2, `analysis.json` + Neo4j projection -Related: #120 (converge `call_sites[]`/`accessed_symbols[]`/`local_variables[]` with `body{}`) - ---- - -## 1. The finding - -Every call site is emitted **twice**, under two unrelated identity schemes. -Verified on `main` (`6f02581`), fixture `return cls()` at line 34: - -| representation | id | Neo4j | -| --- | --- | --- | -| `PyCallable.call_sites[]` | `app.py#34:15-34:22` | `:PyCallSite` ← `PY_HAS_CALLSITE` | -| `PyCallable.body{"34:15"}`, `kind:"call"` | `@34:15` | `:PyCFGNode` ← `PY_HAS_CFG_NODE` | - -One call in source, two graph nodes, no edge between them. `PyCallSite`'s id -(`file#line:col-line:col`) belongs to neither identity tier the schema defines — -it is neither a durable `can://` id nor a `@` ordinal id. - -**They cannot disagree in content.** `schema/l1_body.py` derives one from the other -in the same pass: - -```python -for cs in c.call_sites or []: - key = f"{cs.start_line}:{cs.start_column}" - c.body[key] = BodyNode(kind="call", span=span, callee=None) -``` - -So this is not a correctness bug. It is redundancy by construction. - -## 2. Why it exists - -`call_sites[]` is **not** v1 debris left lying around. It is the analyzer's internal -working record, and it is load-bearing for every level above L1: - -| reader | uses | -| --- | --- | -| `semantic_analysis/call_graph.py:163-215` | `callee_signature`, `method_name`, `is_constructor_call` — builds the L2 call graph | -| `schema/l2_callees.py` | `callee_signature`, `start_line`, `start_column` — backfills `BodyNode.callee` | -| `dataflow/builder.py:368-420` | `callee_signature`, `is_constructor_call`, position — builds SDG call sites | -| `dataflow/summaries.py`, `dataflow/sdg.py` | consume the above | -| `neo4j/project.py:400` | projects `:PyCallSite` | - -`body{}` is the v2 wire view *derived from* that record. The duplication is an -**internal IR leaking into the wire format** — not two competing encodings of equal -standing. That reframing is what makes the fix tractable: the wire format can lose -`call_sites[]` without the internal passes losing anything, provided the record -survives as an internal structure. - -## 3. Design - -**`body{}` is the single emitted representation of a call site.** Consumers obtain the -call-site set by filtering `body` on `kind == "call"`, which works from **L1** — verified: - -``` -L1 body{"34:15"} kind=call callee=null -L2 body{"34:15"} kind=call callee="can://…/@external/app.Account/__init__" -``` - -The Jedi-produced call record stays **internal**: it is the input to L2 resolution and -L3/L4 dataflow, and is not part of the contract. `PyCallsite` leaves the emitted schema. - -### Field disposition - -| field | disposition | why | -| --- | --- | --- | -| `start_line` / `start_column` / `end_*` | **becomes the node key + `span`** | already how `body{}` is keyed | -| `callee_signature` | **internal only** | it is the *input* to resolution; `callee` (a resolved `can://` id) is what the wire carries | -| `is_constructor_call` | **internal only** | read by `call_graph.py` + `builder.py`; recoverable on the wire from `callee` resolving to an `__init__` | -| `method_name` | **internal only** | read by `call_graph.py`; recoverable on the wire from `callee` or the `span` slice | -| `argument_types` | **deleted** | deprecated in 0.3.1 (#86) with "will be removed in schema v2"; no internal reader; removal overdue | -| `arguments` | **moves onto the call `BodyNode`** | no internal reader; output-only detail worth keeping. Encoding is OPEN — see § 5 | -| `receiver_expr` | **moves onto the call `BodyNode`** | no internal reader; Jedi inference with no other home | -| `receiver_type` | **moves onto the call `BodyNode`** | as above | -| `return_type` | **moves onto the call `BodyNode`** | as above | - -### Resulting `BodyNode` - -```python -class BodyNode(BaseModel): - kind: str # statement | call | entry | exit | formal_* | actual_* - span: Optional[Span] = None - callee: Optional[str] = None # call nodes; the sanctioned null→id slot at L2 - of: Optional[str] = None - parent: Optional[str] = None - # call-specific, Jedi inference, absent on every other kind - receiver_expr: Optional[str] = None - receiver_type: Optional[str] = None - return_type: Optional[str] = None -``` - -### Neo4j consequences - -- `:PyCallSite`, `PY_HAS_CALLSITE`, and the `file#line:col-line:col` id scheme are **removed**. -- One body-node label, keyed on the global ordinal id, carries every `body{}` entry. -- `PY_RESOLVES_TO` is re-sourced from `BodyNode.callee` instead of `callee_signature`. -- Merge groups drop from 9 to 8. -- **Rename the body label.** `PyCFGNode` names an L3 concept, but a `call` node exists from - L1 and is deliberately **never** on the CFG spine — verified: at L3 the call `34:15` carries - `parent="34:8"` and appears in no `cfg` edge, while `@entry`/`34:8`/`@exit` do. The current - label asserts CFG membership for a node that has none. A level-neutral name (`PyBodyNode`, - matching what `body{}` is called in the JSON) states what is true at every level. -- If `MATCH (:PyCallSite)` ergonomics are wanted, the writer already supports label layering - (`rows.py:98` merges on `labels[0]` and unions the rest; `cypher.py:95-97` renders it), so a - marker label costs no second node and no second id. Note `MARKER_LABELS` in `neo4j/schema.py` - is declaration-only today — the writer never reads it, and no call site passes >1 label. - -## 4. What this does not do - -- Does not touch `accessed_symbols[]` or `local_variables[]`, the other two halves of #120. -- Does not change the call graph, Jedi resolution, or any dataflow analysis — only which - representation is serialized. -- Does not settle the Neo4j merge-label strategy, the `can://` callable-signature grammar, or - the decorator shape (#128). Those are separate decisions. - -## 5. Open questions - -- **`arguments` encoding.** Inline objects (`{ast_kind, inferred_type}`, what Python does now - and what works at L1) or local-ids referencing argument body nodes. The latter requires - materializing argument nodes at L1, which is a much larger change to the body model and the - id space. Recommendation: keep inline. -- **Body label name.** `PyBodyNode` is the obvious candidate; anything level-neutral works. -- **Whether the internal record stays a Pydantic model** excluded from serialization, or becomes - a plain dataclass in the analysis passes. Purely internal; no contract impact. - -## 6. Caveats and risks - -- **Breaking for anyone reading `call_sites[]`.** That is the point of the change, but it is the - most visible field in the callable model and its removal should lead the release notes. -- **Three fields become wire-recoverable rather than wire-present** (`method_name`, - `is_constructor_call`, `callee_signature`). Recovering `method_name` from a `span` slice is - string work a consumer may not want to do. If that proves unpopular the honest fix is to put - `method_name` back on the node, not to restore `call_sites[]`. -- **`callee` must actually resolve** for `is_constructor_call` and `method_name` to be - recoverable. Where resolution fails, `callee` is null and both are lost. The size of that - set is unmeasured and should be measured before the fields are dropped. -- **Test surface.** Every test asserting on `call_sites[]` changes. They should be rewritten - against `body{}`, not deleted. diff --git a/test/test_entrypoint_detect.py b/test/test_entrypoint_detect.py index d0121d3..9b3d804 100644 --- a/test/test_entrypoint_detect.py +++ b/test/test_entrypoint_detect.py @@ -27,6 +27,16 @@ def test_absent_framework_is_not_detected(tmp_path: Path): assert "celery" not in got +def test_detect_value_case_is_normalized_against_a_lowercase_import(tmp_path: Path): + """A ``detect: [Flask]`` user rule must fire against a real ``import + flask`` -- imports are recorded lowercase, so `detect:` values need the + same normalization or they silently never match (#122 review, MINOR).""" + user = tmp_path / "user.yml" + user.write_text("version: 1\nframeworks:\n myflask:\n detect: [Flask]\n") + got = detected_frameworks(_app("flask"), tmp_path, load_rules([user])) + assert "myflask" in got + + def test_manifest_entry_alone_is_sufficient(tmp_path: Path): (tmp_path / "pyproject.toml").write_text( '[project]\nname = "x"\ndependencies = ["celery>=5"]\n' diff --git a/test/test_entrypoint_pipeline.py b/test/test_entrypoint_pipeline.py index 65b63b9..6e62ed2 100644 --- a/test/test_entrypoint_pipeline.py +++ b/test/test_entrypoint_pipeline.py @@ -34,7 +34,7 @@ def test_derives_is_entrypoint_from_the_list(tmp_path: Path): from codeanalyzer.schema.py_schema import PyCallable, PyDecorator, PyImport, PyModule fn = PyCallable(name="f", path="a.py", signature="a.f") - fn.decorators.append(PyDecorator(name="route", qualified_name="flask.Flask.route")) + fn.decorators.append(PyDecorator(name="route", qualified_name="flask.sansio.scaffold.Scaffold.route")) app = PyApplication( symbol_table={ "a.py": PyModule( @@ -80,7 +80,7 @@ def test_ruleset_provenance_distinguishes_shipped_from_user_rules(tmp_path: Path ) shipped_fn = PyCallable(name="f", path="a.py", signature="a.f") - shipped_fn.decorators.append(PyDecorator(name="app.route", qualified_name="flask.Flask.route")) + shipped_fn.decorators.append(PyDecorator(name="app.route", qualified_name="flask.sansio.scaffold.Scaffold.route")) user_fn = PyCallable(name="g", path="a.py", signature="a.g") user_fn.decorators.append(PyDecorator(name="handler", qualified_name="inhouse.app.handler")) @@ -173,7 +173,7 @@ def test_running_the_pass_twice_does_not_duplicate_entrypoints(tmp_path: Path): from codeanalyzer.schema.py_schema import PyCallable, PyDecorator, PyImport, PyModule fn = PyCallable(name="f", path="a.py", signature="a.f") - fn.decorators.append(PyDecorator(name="route", qualified_name="flask.Flask.route")) + fn.decorators.append(PyDecorator(name="route", qualified_name="flask.sansio.scaffold.Scaffold.route")) app = PyApplication( symbol_table={ "a.py": PyModule( diff --git a/test/test_entrypoint_rules.py b/test/test_entrypoint_rules.py index fb37bed..7b32d37 100644 --- a/test/test_entrypoint_rules.py +++ b/test/test_entrypoint_rules.py @@ -11,6 +11,27 @@ def test_shipped_rules_load_and_include_flask(): assert any(r.id == "flask.route" for r in flask.decorators) +def test_shipped_rules_include_django_cbv_dispatch(): + """A Django project must report ``frameworks_detected: ["django"]`` + distinct from an unsupported project, even with the routing engine + (Unit 5) still absent -- the `bases:` rule works today via the + import-table resolver (#122 review, IMPORTANT 2).""" + rs = load_rules() + assert "django" in rs.frameworks + django = rs.frameworks["django"] + assert "django" in django.detect + cbv = next(r for r in django.bases if r.id == "django.cbv") + assert cbv.match == "django.views.generic.*" + assert set(cbv.dispatch) == {"get", "post", "put", "patch", "delete", "head", "options"} + + +def test_unknown_top_level_key_is_rejected(tmp_path): + bad = tmp_path / "bad.yml" + bad.write_text("declared:\n - id: pyproject.scripts\n") + with pytest.raises(RulesError, match="declared"): + load_rules([bad]) + + def test_every_shipped_rule_has_a_stable_id_and_valid_confidence(): rs = load_rules() for fw in rs.frameworks.values(): From 8edc96c73497be5dcddd2f5541079fb5e8dfc4a4 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 10:40:36 -0400 Subject: [PATCH 21/23] test(entrypoints): integration-test shipped rules against real frameworks Every existing entrypoint test either hand-crafts a qualified_name or uses a local in-repo decorator, so none of them would have caught the shipped rules.yml patterns drifting out of sync with what Jedi actually resolves for flask/fastapi/celery/click. Add flask, fastapi, celery and click to the `test` dependency-group (python_version >= '3.11' only: below that, ray==2.0.0's click<=8.0.4 pin conflicts with celery>=5.3's click>=8.1.2 floor) and a real, CLI-driven integration test per framework, each guarded with pytest.importorskip. --- pyproject.toml | 11 ++ .../test_entrypoint_frameworks_integration.py | 164 ++++++++++++++++++ 2 files changed, 175 insertions(+) create mode 100644 test/test_entrypoint_frameworks_integration.py diff --git a/pyproject.toml b/pyproject.toml index b642fe1..421ca24 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -68,6 +68,17 @@ test = [ # Neo4j integration test (opt-in; spins up a real Neo4j via Testcontainers). "neo4j>=5.0.0,<6.0.0", "testcontainers[neo4j]>=4.0.0,<5.0.0; python_version >= '3.11'", + # Real frameworks for the entrypoint-detection integration test (#27) -- + # rules.yml `match:` patterns are only meaningful against Jedi's actual + # resolution of the installed package, never a guess at its public API path. + # python_version >= '3.11' only: below that, ray==2.0.0 pins click<=8.0.4, + # which conflicts with celery>=5.3's click>=8.1.2 floor. The integration + # test guards each import with pytest.importorskip, so it just skips on + # older interpreters rather than needing a resolvable pin here. + "flask>=3.0.0,<4.0.0; python_version >= '3.11'", + "fastapi>=0.100.0,<1.0.0; python_version >= '3.11'", + "celery>=5.3.0,<6.0.0; python_version >= '3.11'", + "click>=8.0.0,<9.0.0; python_version >= '3.11'", ] dev = [ "ipdb>=0.13.0,<0.14.0", diff --git a/test/test_entrypoint_frameworks_integration.py b/test/test_entrypoint_frameworks_integration.py new file mode 100644 index 0000000..b6f6c3e --- /dev/null +++ b/test/test_entrypoint_frameworks_integration.py @@ -0,0 +1,164 @@ +"""Integration coverage: shipped `rules.yml` decorator patterns against the +REAL frameworks they target (#27, #122 review). + +The shipped rules match on ``PyDecorator.qualified_name`` -- Jedi's resolved +DEFINITION path, not the public import path the rules read as if they were +written against (e.g. ``@app.route`` resolves to +``flask.sansio.scaffold.Scaffold.route``, not ``flask.Flask.route``). Every +other entrypoint test either hand-crafts a ``qualified_name`` or uses a local +in-repo decorator, so none of them would notice a real framework's actual +resolution path drifting out from under the shipped patterns. This is the +test that would have caught it: it drives the real CLI over a tiny app built +on the real, installed package. + +Each framework is an optional test-only dependency (see the `test` +dependency-group in ``pyproject.toml``); guarded with ``importorskip`` so the +suite still runs where one is absent (e.g. below the ``python_version >= +'3.11'`` floor those pins carry, to dodge a ``ray``/``celery`` click-version +conflict below that -- see the comment in ``pyproject.toml``). +""" +import json +import subprocess +from pathlib import Path + +import pytest + + +def _run(fixture_dir: Path, out_dir: Path) -> dict: + subprocess.run( + [ + "uv", "run", "canpy", + "-i", str(fixture_dir), + "-a", "1", + "-o", str(out_dir), + "--no-venv", + # Cache defaults to the input dir; keep it in out_dir so the + # fixture directory is never mutated and each run starts cold. + "--cache-dir", str(out_dir / "cache"), + ], + check=True, + ) + return json.loads((out_dir / "analysis.json").read_text()) + + +def test_flask_route_and_verb_decorators_are_flagged(tmp_path): + pytest.importorskip("flask") + app_dir = tmp_path / "src" + app_dir.mkdir() + (app_dir / "app.py").write_text( + "from flask import Flask\n" + "\n" + "app = Flask(__name__)\n" + "\n" + "\n" + "@app.route('/products', methods=['POST'])\n" + "def create_product():\n" + " return 'ok'\n" + "\n" + "\n" + "@app.get('/products')\n" + "def list_products():\n" + " return []\n" + ) + data = _run(app_dir, tmp_path / "out") + fns = data["application"]["symbol_table"]["app.py"]["functions"] + + (ep,) = fns["create_product"]["entrypoints"] + assert ep["framework"] == "flask" and ep["rule"] == "flask.route" + assert ep["route"] == "/products" and ep["http_methods"] == ["POST"] + + (ep,) = fns["list_products"]["entrypoints"] + assert ep["framework"] == "flask" and ep["rule"] == "flask.bp-verb" + assert ep["route"] == "/products" and ep["http_methods"] == ["GET"] + + assert "flask" in data["application"]["entrypoint_report"]["frameworks_detected"] + assert data["application"]["entrypoint_report"]["errors"] == [] + + +def test_fastapi_get_and_router_post_decorators_are_flagged(tmp_path): + pytest.importorskip("fastapi") + app_dir = tmp_path / "src" + app_dir.mkdir() + (app_dir / "app.py").write_text( + "from fastapi import APIRouter, FastAPI\n" + "\n" + "api = FastAPI()\n" + "router = APIRouter()\n" + "\n" + "\n" + "@api.get('/items')\n" + "def read_items():\n" + " return []\n" + "\n" + "\n" + "@router.post('/items')\n" + "def create_item():\n" + " return {}\n" + ) + data = _run(app_dir, tmp_path / "out") + fns = data["application"]["symbol_table"]["app.py"]["functions"] + + (ep,) = fns["read_items"]["entrypoints"] + assert ep["framework"] == "fastapi" and ep["rule"] == "fastapi.verb" + assert ep["route"] == "/items" and ep["http_methods"] == ["GET"] + + (ep,) = fns["create_item"]["entrypoints"] + assert ep["framework"] == "fastapi" and ep["rule"] == "fastapi.router-verb" + assert ep["route"] == "/items" and ep["http_methods"] == ["POST"] + + assert "fastapi" in data["application"]["entrypoint_report"]["frameworks_detected"] + assert data["application"]["entrypoint_report"]["errors"] == [] + + +def test_celery_shared_task_and_app_task_decorators_are_flagged(tmp_path): + pytest.importorskip("celery") + app_dir = tmp_path / "src" + app_dir.mkdir() + (app_dir / "app.py").write_text( + "from celery import Celery, shared_task\n" + "\n" + "cel = Celery('x')\n" + "\n" + "\n" + "@shared_task\n" + "def add(x, y):\n" + " return x + y\n" + "\n" + "\n" + "@cel.task\n" + "def mul(x, y):\n" + " return x * y\n" + ) + data = _run(app_dir, tmp_path / "out") + fns = data["application"]["symbol_table"]["app.py"]["functions"] + + (ep,) = fns["add"]["entrypoints"] + assert ep["framework"] == "celery" and ep["rule"] == "celery.shared-task" + + (ep,) = fns["mul"]["entrypoints"] + assert ep["framework"] == "celery" and ep["rule"] == "celery.task" + + assert "celery" in data["application"]["entrypoint_report"]["frameworks_detected"] + assert data["application"]["entrypoint_report"]["errors"] == [] + + +def test_click_command_decorator_is_flagged(tmp_path): + pytest.importorskip("click") + app_dir = tmp_path / "src" + app_dir.mkdir() + (app_dir / "app.py").write_text( + "import click\n" + "\n" + "\n" + "@click.command()\n" + "def cli():\n" + " pass\n" + ) + data = _run(app_dir, tmp_path / "out") + fns = data["application"]["symbol_table"]["app.py"]["functions"] + + (ep,) = fns["cli"]["entrypoints"] + assert ep["framework"] == "click" and ep["rule"] == "click.command" + + assert "click" in data["application"]["entrypoint_report"]["frameworks_detected"] + assert data["application"]["entrypoint_report"]["errors"] == [] From b6c082e006fb1e5bc59565abf8862cd2562674ff Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 10:40:40 -0400 Subject: [PATCH 22/23] fix(cli): validate --entrypoint-rules before analysis starts detect_entrypoints ran after the symbol table, venv build, Jedi and PyCG, so a malformed --entrypoint-rules file cost minutes on a large repo before failing with a raw traceback. Load the rules once options are built (before the --emit schema short-circuit and well before Codeanalyzer.analyze()) and exit cleanly on RulesError. The existing load_rules call inside detect_entrypoints stays as-is -- loading twice is cheap and keeps the entrypoints pipeline self-contained. --- codeanalyzer/__main__.py | 14 ++++++++++++++ test/test_entrypoints_e2e.py | 27 +++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/codeanalyzer/__main__.py b/codeanalyzer/__main__.py index 750a325..388edb4 100644 --- a/codeanalyzer/__main__.py +++ b/codeanalyzer/__main__.py @@ -398,6 +398,20 @@ def main( _set_log_level(options.verbosity) + # Entrypoint rules are configuration, validated before any analysis work + # starts (#122 review) -- a typo must fail in milliseconds, not after the + # symbol table, venv build, Jedi and PyCG have all run. `detect_entrypoints` + # loads the rules again at its own call site; that second load is cheap + # and keeps the entrypoints pipeline self-contained. + if options.entrypoint_rules: + from codeanalyzer.entrypoints.rules import RulesError, load_rules + + try: + load_rules(options.entrypoint_rules) + except RulesError as exc: + logger.error(f"Invalid --entrypoint-rules: {exc}") + raise typer.Exit(code=1) + # The schema contract is a static artifact — no project analysis required. if options.emit == EmitTarget.SCHEMA: from codeanalyzer.neo4j.emit import emit_schema diff --git a/test/test_entrypoints_e2e.py b/test/test_entrypoints_e2e.py index 198e796..fbabc83 100644 --- a/test/test_entrypoints_e2e.py +++ b/test/test_entrypoints_e2e.py @@ -47,3 +47,30 @@ def test_decorated_function_flagged_and_helper_not(tmp_path): report = data["application"]["entrypoint_report"] assert "inhouse" in report["frameworks_detected"] assert report["errors"] == [] + + +def test_malformed_entrypoint_rules_fails_fast_before_analysis(tmp_path): + """A malformed ``--entrypoint-rules`` file must exit BEFORE the symbol + table, venv build, Jedi and PyCG run -- not deep in the pipeline + (#122 review, IMPORTANT 1). Proven by wall-clock: a real analysis of + even this tiny fixture takes noticeably longer than the sub-second + failure this must produce.""" + bad_rules = tmp_path / "bad.yml" + bad_rules.write_text("frameworks: [not, a, mapping]\n") + + result = subprocess.run( + [ + "uv", "run", "canpy", + "-i", str(FIXTURE), + "-a", "1", + "-o", str(tmp_path / "out"), + "--no-venv", + "--cache-dir", str(tmp_path / "cache"), + "--entrypoint-rules", str(bad_rules), + ], + capture_output=True, + text=True, + ) + assert result.returncode != 0 + assert not (tmp_path / "out" / "analysis.json").exists() + assert "entrypoint-rules" in (result.stderr + result.stdout).lower() From 4086f7a05b70599a0d33215ade65ab1f5625c765 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 20 Aug 2026 10:51:24 -0400 Subject: [PATCH 23/23] ci: split a 3.10 compat job so the framework integration tests actually run (#27) The release workflow was the repository's only CI and pinned Python 3.10. The framework integration tests added for entrypoint detection depend on flask, fastapi, celery and click, which are gated `python_version >= '3.11'` for a real reason: on 3.10 ray==2.0.0 pins click<=8.0.4 while celery>=5.3 needs click>=8.1.2. So on 3.10 all four tests hit `pytest.importorskip` and skip silently -- the decorator-rule regression they exist to catch could ship with a green suite, which is exactly how that regression reached the final review in the first place. Bumping the release job to 3.12 alone would have dropped the only CI exercise of the `python_version < '3.11'` half of the dependency matrix, and there are seven such branches (ray, jedi, networkx, pydantic, rich, typer, typing-extensions). So: a `compat` job runs the suite on 3.10, the release job runs on 3.12 where the integration tests install, and the release gates on compat. Both halves of the matrix stay covered. The package is pure-Python and `requires-python` is unchanged, so the built wheel is unaffected by the interpreter bump. --- .github/workflows/release.yml | 37 +++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 4e3e7cb..742b43d 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -12,9 +12,14 @@ permissions: discussions: write # attach the release-linked repo Discussion (Announcements) jobs: - release: + # Exercises the `python_version < '3.11'` half of the dependency matrix (ray==2.0.0, + # the older jedi/networkx/pydantic/typer pins). The release job below runs on 3.12, + # where the framework integration tests can install -- on 3.10 ray==2.0.0 pins + # click<=8.0.4 against celery>=5.3's click>=8.1.2 floor, so those test deps are + # gated >=3.11 and would silently `importorskip` here. Gating the release on this + # job keeps both halves of the matrix covered (#27). + compat: runs-on: ubuntu-latest - steps: - name: Check out code uses: actions/checkout@v4 @@ -29,6 +34,34 @@ jobs: curl -LsSf https://astral.sh/uv/install.sh | sh echo "$HOME/.cargo/bin" >> $GITHUB_PATH + - name: Sync dependencies + run: uv sync --all-groups + + - name: Run tests + run: uv run pytest + + release: + needs: compat + runs-on: ubuntu-latest + + steps: + - name: Check out code + uses: actions/checkout@v4 + + # 3.12, not 3.10: the framework integration tests (#27) need flask/fastapi/ + # celery/click, which are gated `python_version >= '3.11'`. On 3.10 they + # `importorskip` and the decorator-rule regression they exist to catch would + # ship green. The `compat` job above keeps 3.10 covered. + - name: Set up Python 3.12 + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Install uv + run: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + - name: Sync dependencies run: uv sync --all-groups