From 53be2ad3d083c50361e9a8a32f8ff4e649532d84 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 11:11:28 -0700 Subject: [PATCH 01/10] docs(design): spec the ConfigKey family (closes #152 core; config_use deferred) --- .../2026-08-28-config-key-family-design.md | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 docs/design/specs/2026-08-28-config-key-family-design.md diff --git a/docs/design/specs/2026-08-28-config-key-family-design.md b/docs/design/specs/2026-08-28-config-key-family-design.md new file mode 100644 index 0000000..67c26dc --- /dev/null +++ b/docs/design/specs/2026-08-28-config-key-family-design.md @@ -0,0 +1,98 @@ +# ConfigKey Family: Configuration Keys as First-Class Nodes + +**Date:** 2026-08-28 +**Status:** Approved (design dialogue in-session) +**Scope:** codeanalyzer-python, schema v2 additive; closes #152 (core); config_use deferred +**Builds on:** `2026-08-27-artifacts-and-dependencies-design.md` (the artifact substrate) + +## Problem + +Artifacts capture configuration *files* verbatim, but their *meaning* — the +keys a deployment defines, the values services wire together, the references +that stitch `.env` → compose → CI — is opaque text. #152's config_key family +extracts it. The definitions side lands here; the `config_use` edge (code +reading a key) is the recorded follow-up. + +## Locked decisions + +1. **Neutral vocabulary.** A yaml key is not a Python concept: graph label + `ConfigKey`, edge `DEFINES_CONFIG` (Artifact→ConfigKey) — same + nouns-neutral rule as `Artifact`/`Package`; the Python-specific claim + arrives later as `PY_USES_CONFIG`. (Supersedes #152's `PyConfigKey` / + `PY_DEFINES_CONFIG` naming per the reconciliation on #152.) +2. **Value gated on text capture.** `value: Optional[str]` populated only + when `--artifact-text` is on (default). Keys, namespaces, spans, and + references are always extracted — `--no-artifact-text` drops values and + source together, so the secret off-switch actually switches everything off. +3. **V1 formats**: `env` (`.env`, `.env.*`, `.flaskenv`), `yaml`, `json`, + `toml`, `ini`, `properties`. Extraction is format-driven over existing + artifacts (pyproject.toml gets keys too; overlap with dependency records is + harmless). `references[]` v1 recognizes three syntaxes, recorded as raw + tokens: `${VAR}`/`$VAR`, `%(name)s`, `${{ ... }}`. +4. **Placement: nested.** `PyArtifact.config_keys: List[PyConfigKey]` — + containment mirrors `DEFINES_CONFIG`; L1 data, identical at every level. +5. **Overlay posture.** Parse failure never drops the artifact node; it sets + the artifact's `extraction: "partial"`. Extraction reads full on-disk text + (cap-immune, same decoupling as dependency manifests). + +## Model + +`PyConfigKey` (model name follows `PyArtifact` precedent; label stays neutral): + +| field | type | notes | +| --- | --- | --- | +| `id` | str | `@key/` | +| `key` | str | dotted path; numeric segments for arrays (`services.web.ports.0`) | +| `namespace` | str | `env` \| `yaml` \| `json` \| `toml` \| `ini` \| `properties` | +| `value` | Optional[str] | only when text capture on | +| `span` | Span | into the artifact's source | +| `references` | List[str] | raw recognized tokens | + +## Extraction + +New `codeanalyzer/artifacts/config_keys.py`, invoked in `core.analyze()` +beside `build_dependency_view`: flatteners over tomllib/yaml/json/configparser +plus env/properties line parsers; deterministic (sorted keys); spans from the +source text; value extraction obeys `options.artifact_text`. + +## Neo4j + +`ConfigKey` label (merge key `id`; props key, namespace, value?, references, +start_line, end_line) + `DEFINES_CONFIG` rel; uniqueness constraint (DDL +asset inherits it); conformance fixture grows a config-bearing artifact. +`SCHEMA_VERSION` stays `2.0.0` (no consumers yet — reconciliation decision). + +## Riders (bounded, same PR) + +- `*.tf` discovery rule → new role `iac`, capture-only (no HCL extraction). +- `PyDependency.ecosystem: str = "pypi"` (SDK symmetry with purl). +- `skip_tests` deliberately NOT wired into the inventory — artifacts under + test dirs are signal; decision recorded here rather than silently + diverging from #152. + +## Deferred: config_use + +`PY_USES_CONFIG` (body node → ConfigKey) needs `PyCallArgument.value` +(#152's own boundary). ConfigKey ids minted here are its resolution target. + +## Caveats + +- Multi-line/huge values (yaml block scalars) are captured as parsed; no + per-value cap in v1 — revisit with payload evidence. +- `references[]` is recognition, not resolution: tokens are recorded raw; + cross-artifact joins are the consumer's query (see analyses.md examples). +- env-family files are `format: "text"` with role `env`; namespace `env` is + keyed off the basename rule, not the format. + +## Definition of done + +- Per-format flattener tests (nesting, arrays, quoting, interpolation, + escapes); reference extraction for all three syntaxes; value-gating test + (`--no-artifact-text` → keys/spans/references identical, values absent); + corrupted config → artifact kept, `extraction: "partial"`; span slices + reproduce values from source. +- e2e fixture grows `.env`, `settings.yml`, `app.properties`; level-invariance + and determinism gates extended; full suite green. +- Neo4j: label+rel in catalog, `--emit schema` reflects them, conformance + fixture exercises them; skill vocabulary/analyses updated on the #161 line. +- One PR closing #152, stacked on `feat/issue-157-artifacts-dependencies`. From 5dd9fe88c744229c7a57b6924b7b5a3a7edfca7b Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 11:12:06 -0700 Subject: [PATCH 02/10] docs(plan): ConfigKey family implementation plan --- .../plans/2026-08-28-config-key-family.md | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 docs/design/plans/2026-08-28-config-key-family.md diff --git a/docs/design/plans/2026-08-28-config-key-family.md b/docs/design/plans/2026-08-28-config-key-family.md new file mode 100644 index 0000000..9c77938 --- /dev/null +++ b/docs/design/plans/2026-08-28-config-key-family.md @@ -0,0 +1,54 @@ +# ConfigKey Family Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Extract configuration keys as first-class `ConfigKey` nodes from the six v1 formats, plus three bounded riders (closes #152). + +**Architecture:** New `codeanalyzer/artifacts/config_keys.py` flattens config formats into `PyConfigKey` records nested on `PyArtifact.config_keys`, wired into `core.analyze()` beside `build_dependency_view`; Neo4j projects neutral `ConfigKey` nodes via `DEFINES_CONFIG`. + +**Tech Stack:** tomllib/tomli, yaml, json, configparser (all in-tree), hand-rolled env/properties line parsers, existing `Span`/`byte_offsets` helpers. + +**Spec:** `docs/design/specs/2026-08-28-config-key-family-design.md` + +## Global Constraints + +- No AI attribution anywhere. Conventional Commits. +- Neutral graph vocabulary: label `ConfigKey`, edge `DEFINES_CONFIG` — never `PyConfigKey`/`PY_DEFINES_CONFIG` in the graph. +- `value` populated ONLY when `options.artifact_text` is true; keys/namespace/span/references always extracted, from full on-disk text (never the possibly-truncated stored `source`). +- Parse failure never drops an artifact; it sets `extraction: "partial"`. +- L1 data: identical at every `-a`; deterministic (sorted key order within each artifact). +- ConfigKey id = `@key/`; dotted paths use numeric segments for arrays. +- Namespaces exactly: env, yaml, json, toml, ini, properties. +- references[] recognizes exactly: `${VAR}`/`$VAR`, `%(name)s`, `${{ ... }}` — raw tokens, order of appearance, deduplicated. + +--- + +### Task 1: Model + id helper + +**Files:** Modify `codeanalyzer/schema/py_schema.py` (PyConfigKey near PyArtifact; `config_keys: List[PyConfigKey] = []` on PyArtifact), `codeanalyzer/schema/ids.py` (`config_key_id(artifact_id: str, dotted_key: str) -> str` returning `f"{artifact_id}@key/{dotted_key}"`). Test `test/test_config_key_models.py`. + +Model fields per spec §Model: id, key, namespace, value: Optional[str] = None, span: Optional[Span] = None, references: List[str] = []. Round-trip test via compat helpers; default-empty on old payloads; id shape asserted. +Commit: `feat(schema): PyConfigKey model and config_key_id helper` + +### Task 2: Flatteners + reference recognition + +**Files:** Create `codeanalyzer/artifacts/config_keys.py`. Test `test/test_config_key_extraction.py`. + +Public API: `extract_config_keys(artifact: PyArtifact, full_text: str, capture_value: bool) -> List[PyConfigKey]` dispatching on namespace (env by basename family `.env`/`.env.*`/`.flaskenv`; else by format for yaml/json/toml/ini/properties; other formats → []). Internals: `_flatten(obj, prefix)` producing dotted paths with numeric array segments; env parser (KEY=value, `#` comments, `export ` prefix, single/double quote stripping); properties parser (`key=value`/`key: value`, `\` continuations, `!`/`#` comments); ini via configparser with raw=True (preserve `%(x)s`); `_find_references(text) -> List[str]` with the three regexes (`\$\{\{[^}]*\}\}` FIRST, then `\$\{[A-Za-z_][A-Za-z0-9_]*\}`, then `\$[A-Za-z_][A-Za-z0-9_]*`; dedupe preserving order). Spans: line/col of the key's line (env/properties/ini exact line; yaml/json/toml best-effort via first occurrence search of the final key segment on its own line — record the chosen rule in a comment). Never raises: any exception → return partial list gathered so far and signal failure via return, or raise a single ConfigParseError the caller catches — pick one, test it. +Tests: nested yaml→dotted, arrays→numeric segments, env quoting/comments/export, properties continuations, ini interpolation preserved raw + reference recognized, toml tables, json nesting, all three reference syntaxes incl. dedupe/order, capture_value=False → value None everywhere else identical, malformed input per format → failure signaled without exception escaping. +Commit: `feat(artifacts): config-key flatteners with reference recognition` + +### Task 3: Wiring + riders + +**Files:** Modify `codeanalyzer/core.py` (call after build_dependency_view: iterate sorted artifacts, namespace-eligible ones get `art.config_keys = extract_config_keys(...)` reading full text via the same on-disk read used by dependencies `_full_text`; on failure set `extraction = "partial"`), `codeanalyzer/artifacts/discovery.py` (rule `("*.tf", "text", ["iac"])`), `codeanalyzer/schema/py_schema.py` (`PyDependency.ecosystem: str = "pypi"`), `codeanalyzer/artifacts/dependencies.py` (set ecosystem explicitly where records are built — both _emit and the lock-only transitive branch). Tests: extend `test/test_artifact_pipeline.py` (keys present at `-a 1` and `-a 2`, identical), new asserts in `test_artifact_discovery.py` (`main.tf` → role iac) and `test_dependency_view.py` (ecosystem == "pypi" on every record). +Commit: `feat(artifacts): wire config-key extraction into analyze; tf rule; ecosystem field` + +### Task 4: Neo4j projection + +**Files:** Modify `codeanalyzer/neo4j/schema.py` (NodeLabel ConfigKey merge key id, props: id, key, namespace, value string, references string[], start_line, end_line; RelType DEFINES_CONFIG Artifact→ConfigKey), `codeanalyzer/neo4j/project.py` (in `_project_artifacts`: per artifact, per config_key sorted by key — node + edge; value omitted from props when None), regenerate `schema.neo4j.json`, extend `test/sample_graph_app.py` (one artifact with a config key so the every-label test passes). Test `test/test_neo4j_config_keys.py`: node + edge rows projected, value-absent when model value None, catalog/constraint present. +Commit: `feat(neo4j): neutral ConfigKey nodes via DEFINES_CONFIG` + +### Task 5: e2e + docs + +**Files:** Fixture `test/fixtures/whole_applications/manifests_app/` gains `.env` (with a `${VAR}` reference + a SECRET-looking key), `config/settings.yml` (nested + array), `app.properties`; extend `test/test_artifacts_end_to_end.py` (exact key sets per file, namespace env on .env, references extracted, value-gating via a `--no-artifact-text`-equivalent options run, spans slice source to the value, artifact-set `==` assert updated); CHANGELOG Unreleased line; README Output-shape sentence. Full suite `uv run pytest test/ -q --no-cov` — record counts. +Commit: `test(artifacts): config-key e2e coverage; docs` From 2732c52b153607a1d454b9c15c6237cf80a96405 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 11:22:40 -0700 Subject: [PATCH 03/10] feat(schema): PyConfigKey model and config_key_id helper --- codeanalyzer/schema/ids.py | 7 ++++ codeanalyzer/schema/py_schema.py | 18 ++++++++++ test/test_config_key_models.py | 59 ++++++++++++++++++++++++++++++++ 3 files changed, 84 insertions(+) create mode 100644 test/test_config_key_models.py diff --git a/codeanalyzer/schema/ids.py b/codeanalyzer/schema/ids.py index 09626ff..337fdc8 100644 --- a/codeanalyzer/schema/ids.py +++ b/codeanalyzer/schema/ids.py @@ -32,6 +32,13 @@ def artifact_id(app_name: str, rel_path: str) -> str: return f"can://artifact/{app_name}/{rel_path}" +def config_key_id(artifact_id: str, dotted_key: str) -> str: + """A ``PyConfigKey`` extracted from an artifact: ``@key/``. + ``dotted_key`` uses numeric segments for array indices (e.g. + ``services.web.ports.0``); ids are opaque, do not re-split them.""" + return f"{artifact_id}@key/{dotted_key}" + + def purl_pypi(name: str) -> str: """Package URL for a (PEP 503 normalized) PyPI distribution name.""" return f"pkg:pypi/{name}" diff --git a/codeanalyzer/schema/py_schema.py b/codeanalyzer/schema/py_schema.py index 26db9d7..03ad8d2 100644 --- a/codeanalyzer/schema/py_schema.py +++ b/codeanalyzer/schema/py_schema.py @@ -470,6 +470,23 @@ class PyExternalSymbol(BaseModel): module: Optional[str] = None # best-effort owning module, e.g. "requests" +@builder +class PyConfigKey(BaseModel): + """A configuration key flattened out of a config-bearing ``PyArtifact`` + (#152). Graph vocabulary stays neutral (label ``ConfigKey``, edge + ``DEFINES_CONFIG``) -- the ``Py`` prefix here is only the ``PyArtifact`` + naming precedent, not a Python-specific claim. L1 data, identical at + every analysis level; nested under the owning artifact, containment + mirrors ``DEFINES_CONFIG``.""" + + id: str = "" # @key/ + key: str # dotted path; numeric segments for arrays, e.g. "services.web.ports.0" + namespace: str # env|yaml|json|toml|ini|properties + value: Optional[str] = None # populated only when options.artifact_text is on + span: Optional[Span] = None # into the artifact's source; best-effort for yaml/json/toml + references: List[str] = [] # raw recognized tokens, order of appearance, deduplicated + + @builder class PyArtifact(BaseModel): """Any non-`.py` project file (config, manifest, CI, container spec, or @@ -488,6 +505,7 @@ class PyArtifact(BaseModel): source: str = "" # verbatim by default; "" for binary or when capture is disabled text_truncated: bool = False # True when `source` is a prefix, not the full file extraction: str = "none" # none|partial|full + config_keys: List[PyConfigKey] = [] # flattened config keys (#152); [] when not namespace-eligible @builder diff --git a/test/test_config_key_models.py b/test/test_config_key_models.py new file mode 100644 index 0000000..9ec93c9 --- /dev/null +++ b/test/test_config_key_models.py @@ -0,0 +1,59 @@ +"""Task 1: PyConfigKey model and config_key_id helper.""" +from codeanalyzer.schema import model_validate_json, model_dump_json +from codeanalyzer.schema.ids import artifact_id, config_key_id +from codeanalyzer.schema.py_schema import PyApplication, PyArtifact, PyConfigKey, Span + + +def test_config_key_id_shape(): + assert config_key_id("can://artifact/a/pyproject.toml", "project.name") == \ + "can://artifact/a/pyproject.toml@key/project.name" + + +def test_config_key_id_numeric_array_segment(): + assert config_key_id("can://artifact/a/compose.yml", "services.web.ports.0") == \ + "can://artifact/a/compose.yml@key/services.web.ports.0" + + +def test_models_round_trip(): + art = PyArtifact( + id=artifact_id("a", "config.yaml"), path="config.yaml", + format="yaml", roles=["unknown"], size_bytes=10, + sha256="ab" * 32, source="db:\n host: localhost\n", + ) + key = PyConfigKey( + id=config_key_id(art.id, "db.host"), key="db.host", namespace="yaml", + value="localhost", + span=Span(start=(2, 7), end=(2, 16), bytes=(10, 19)), + references=["${VAR}"], + ) + art.config_keys = [key] + app = PyApplication.builder().symbol_table({}).call_graph([]).build() + app.artifacts = {art.path: art} + back = model_validate_json(PyApplication, model_dump_json(app)) + ck = back.artifacts["config.yaml"].config_keys[0] + assert ck.id == f"{art.id}@key/db.host" + assert ck.key == "db.host" + assert ck.namespace == "yaml" + assert ck.value == "localhost" + assert ck.span.start == (2, 7) and ck.span.end == (2, 16) + assert ck.references == ["${VAR}"] + + +def test_defaults_empty_on_old_payload(): + """A PyArtifact payload written before #152 has no `config_keys` key at + all -- must still validate, defaulting to an empty list.""" + art = PyArtifact( + id=artifact_id("a", "x.txt"), path="x.txt", format="text", + size_bytes=0, sha256="0" * 64, + ) + app = PyApplication.builder().symbol_table({}).call_graph([]).build() + app.artifacts = {art.path: art} + back = model_validate_json(PyApplication, model_dump_json(app)) + assert back.artifacts["x.txt"].config_keys == [] + + +def test_config_key_optional_fields_default(): + key = PyConfigKey(id="x@key/a", key="a", namespace="env") + assert key.value is None + assert key.span is None + assert key.references == [] From 3b0b0ab75108fc516f99855f5e347cb4547ac7f9 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 11:32:25 -0700 Subject: [PATCH 04/10] feat(artifacts): config-key flatteners with reference recognition --- codeanalyzer/artifacts/config_keys.py | 289 ++++++++++++++++++++++++++ test/test_config_key_extraction.py | 230 ++++++++++++++++++++ 2 files changed, 519 insertions(+) create mode 100644 codeanalyzer/artifacts/config_keys.py create mode 100644 test/test_config_key_extraction.py diff --git a/codeanalyzer/artifacts/config_keys.py b/codeanalyzer/artifacts/config_keys.py new file mode 100644 index 0000000..af73c4f --- /dev/null +++ b/codeanalyzer/artifacts/config_keys.py @@ -0,0 +1,289 @@ +"""Config-key flatteners (#152). Pure text-in/records-out, mirroring +`artifacts/parsers.py`'s idiom: one dispatcher over per-format internals, +never raising. + +Namespace dispatch: an env-family basename (`.env`, `.env.*`, `.flaskenv`) +always wins, regardless of the artifact's declared `format`; otherwise the +`format` field selects yaml/json/toml/ini/properties. Any other format +extracts nothing (not a failure -- there's just nothing to flatten). + +Span precision differs by shape: env/properties/ini are line-oriented, so +the parse itself knows the exact defining line. yaml/json/toml are +tree-shaped -- span recovery falls back to a best-effort search for the +final dotted-key segment on its own line (see `_find_key_span`), which can +bind the wrong line when the same leaf name recurs at another nesting level, +or find nothing at all (`span=None`) for a minified/single-line file. +""" +from __future__ import annotations + +import configparser +import json +import re +import sys +from typing import Callable, Dict, List, Optional, Tuple + +if sys.version_info >= (3, 11): + import tomllib +else: # pragma: no cover - exercised on the 3.10 CI leg + import tomli as tomllib + +import yaml + +from codeanalyzer.schema.ids import config_key_id +from codeanalyzer.schema.py_schema import PyArtifact, PyConfigKey, Span, byte_offsets + +# A parsed leaf before it becomes a PyConfigKey: (dotted_key, raw_value, span). +_Entry = Tuple[str, object, Optional[Span]] + + +# --- dotted-path flattening (yaml/json/toml share this over their parsed +# dict/list trees) -------------------------------------------------------- + +def _flatten(obj, prefix: str = ""): + """Yield (dotted_key, leaf_value) pairs; numeric segments for arrays + (e.g. "services.web.ports.0").""" + if isinstance(obj, dict): + for k, v in obj.items(): + yield from _flatten(v, f"{prefix}.{k}" if prefix else str(k)) + elif isinstance(obj, list): + for i, v in enumerate(obj): + yield from _flatten(v, f"{prefix}.{i}" if prefix else str(i)) + else: + yield prefix, obj + + +def _line_span(text: str, lines: List[str], lineno: int) -> Span: + """Exact span covering `lineno`'s full text (1-based).""" + line = lines[lineno - 1] + lo, hi = byte_offsets(text, lineno, 0, lineno, len(line)) + return Span(start=(lineno, 0), end=(lineno, len(line)), bytes=(lo, hi)) + + +_BEST_EFFORT_KEY_TAIL = r'["\']?\s*[:=]' + + +def _find_key_span(text: str, lines: List[str], last_segment: str) -> Optional[Span]: + """Best-effort: the FIRST line (in file order) whose stripped-of-leading + indentation/list-dash content starts with `last_segment` (optionally + quoted) followed by `:` or `=` -- covers yaml (`key:`), json (`"key":`), + and toml (`key =`) without per-format branching. Anchored at column 0 + (not a substring search) so it can't latch onto a leaf name that merely + appears inside a longer token; the cost is that it also can't see keys + packed onto a single minified line, which is an accepted v1 gap given + `span` is `Optional`.""" + pattern = re.compile(r'^[\s\-]*["\']?' + re.escape(last_segment) + _BEST_EFFORT_KEY_TAIL) + for i, line in enumerate(lines, start=1): + if pattern.match(line): + return _line_span(text, lines, i) + return None + + +def _flatten_structured(data, text: str, lines: List[str]) -> List[_Entry]: + return [(k, v, _find_key_span(text, lines, k.rsplit(".", 1)[-1])) for k, v in _flatten(data)] + + +def _parse_yaml(text: str, lines: List[str]) -> List[_Entry]: + return _flatten_structured(yaml.safe_load(text) or {}, text, lines) + + +def _parse_json(text: str, lines: List[str]) -> List[_Entry]: + return _flatten_structured(json.loads(text), text, lines) + + +def _parse_toml(text: str, lines: List[str]) -> List[_Entry]: + return _flatten_structured(tomllib.loads(text), text, lines) + + +# --- env: KEY=value, `#` comments, `export ` prefix, quote stripping ------- + +_ENV_LINE = re.compile(r'^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$') + + +def _strip_quotes(value: str) -> str: + if len(value) >= 2 and value[0] == value[-1] and value[0] in "'\"": + return value[1:-1] + return value + + +def _is_env_family(basename: str) -> bool: + return basename == ".env" or basename.startswith(".env.") or basename == ".flaskenv" + + +def _parse_env(text: str, lines: List[str]) -> List[_Entry]: + out: List[_Entry] = [] + for lineno, raw in enumerate(lines, start=1): + stripped = raw.strip() + if not stripped or stripped.startswith("#"): + continue + m = _ENV_LINE.match(stripped) + if not m: + continue + key, value = m.group(1), _strip_quotes(m.group(2).strip()) + out.append((key, value, _line_span(text, lines, lineno))) + return out + + +# --- properties: key=value / key: value, `\` continuations, `!`/`#` comments + +_PROPS_KV = re.compile(r'^(?P[^=:\s]+)\s*[:=]\s*(?P.*)$') + + +def _parse_properties(text: str, lines: List[str]) -> List[_Entry]: + out: List[_Entry] = [] + i, n = 0, len(lines) + while i < n: + stripped = lines[i].strip() + if not stripped or stripped.startswith(("#", "!")): + i += 1 + continue + start_lineno = i + 1 + parts = [stripped] + while parts[-1].endswith("\\") and i + 1 < n: + parts[-1] = parts[-1][:-1] # drop just the continuation backslash + i += 1 + parts.append(lines[i].strip()) # continuation: leading whitespace stripped + m = _PROPS_KV.match("".join(parts)) + if m: + out.append((m.group("key").strip(), m.group("value").strip(), + _line_span(text, lines, start_lineno))) + i += 1 + return out + + +# --- ini: configparser with raw=True (preserve `%(x)s`); exact line via a +# lightweight parallel section/key scan (configparser gives no line numbers, +# and `strict=True` already guarantees no duplicate (section, key) pairs) -- + +_INI_SECTION = re.compile(r'^\[(?P[^]]+)\]\s*$') +_INI_KEY = re.compile(r'^(?P[^\s#;=:][^=:]*?)\s*[:=]') + + +def _ini_line_map(lines: List[str]) -> Dict[Tuple[str, str], int]: + out: Dict[Tuple[str, str], int] = {} + section: Optional[str] = None + for lineno, line in enumerate(lines, start=1): + if not line.strip() or line.lstrip().startswith((";", "#")): + continue + m = _INI_SECTION.match(line) + if m: + section = m.group("name") + continue + m = _INI_KEY.match(line) + if m and section is not None: + out.setdefault((section, m.group("key")), lineno) + return out + + +def _parse_ini(text: str, lines: List[str]) -> List[_Entry]: + cp = configparser.ConfigParser() + cp.optionxform = str # preserve on-disk case (also needed for the line-map lookup) + cp.read_string(text) + line_map = _ini_line_map(lines) + out: List[_Entry] = [] + for section in cp.sections(): + for key, value in cp.items(section, raw=True): + lineno = line_map.get((section, key)) or line_map.get(("DEFAULT", key)) + span = _line_span(text, lines, lineno) if lineno else None + out.append((f"{section}.{key}", value, span)) + return out + + +_NAMESPACE_PARSERS: Dict[str, Callable[[str, List[str]], List[_Entry]]] = { + "yaml": _parse_yaml, "json": _parse_json, "toml": _parse_toml, + "ini": _parse_ini, "properties": _parse_properties, +} + + +# --- reference recognition --------------------------------------------- + +_REF_TEMPLATE = re.compile(r'\$\{\{[^}]*\}\}') +_REF_BRACED = re.compile(r'\$\{[A-Za-z_][A-Za-z0-9_]*\}') +_REF_BARE = re.compile(r'\$[A-Za-z_][A-Za-z0-9_]*') +_REF_PERCENT = re.compile(r'%\([A-Za-z_][A-Za-z0-9_]*\)s') + + +def _find_references(text: str) -> List[str]: + """Raw reference tokens in `text`, order of appearance, deduplicated. + `${{ ... }}` is matched (and masked out of the working copy) FIRST so a + `${VAR}`/`$VAR` nested inside a template expression is not also counted + as a standalone reference; `%(name)s` never overlaps a `$`-sigil form so + it needs no masking.""" + found: List[Tuple[int, str]] = [] + working = text + for pattern in (_REF_TEMPLATE, _REF_BRACED, _REF_BARE): + for m in pattern.finditer(working): + found.append((m.start(), m.group(0))) + working = pattern.sub(lambda m: " " * len(m.group(0)), working) + for m in _REF_PERCENT.finditer(text): + found.append((m.start(), m.group(0))) + found.sort(key=lambda pair: pair[0]) + seen = set() + out: List[str] = [] + for _, token in found: + if token not in seen: + seen.add(token) + out.append(token) + return out + + +def _stringify(value: object) -> str: + if value is None: + return "" + if isinstance(value, bool): # yaml/json/toml spell it lowercase on disk + return "true" if value else "false" + return str(value) + + +# --- public API ----------------------------------------------------------- + +def extract_config_keys( + artifact: PyArtifact, full_text: str, capture_value: bool, +) -> Tuple[List[PyConfigKey], bool]: + """Flatten `artifact`'s config format into `PyConfigKey` records, reading + `full_text` (the real on-disk text -- never the possibly-truncated + `artifact.source`). + + Returns `(keys, ok)`: the same two-tuple shape as + `artifacts.parsers.parse_manifest`'s `(records, partial)`, but the + OPPOSITE polarity -- here `ok` is `True` on success (including the + not-applicable case: a format with no flattener yields `([], True)`) and + `False` only when parsing raised. Never raises: every code path below is + covered by one `try`/`except`, so a malformed file degrades to `([], + False)` instead of an exception escaping to the caller. `keys` is always + sorted by `key` (L1 determinism). + + `value` is populated only when `capture_value` is True; `key`, + `namespace`, `span`, and `references` are extracted unconditionally + either way (references are recognized in the raw leaf value regardless + of whether that value is exposed).""" + basename = artifact.path.rsplit("/", 1)[-1] + if _is_env_family(basename): + namespace, parser = "env", _parse_env + else: + parser = _NAMESPACE_PARSERS.get(artifact.format) + if parser is None: + return [], True + namespace = artifact.format + + try: + lines = full_text.splitlines() + entries = parser(full_text, lines) + # Last occurrence wins on a duplicate dotted key (env/properties can + # legally redefine a key later in the file; a repeat would otherwise + # collide on `id`, which is derived from `key` alone). + coalesced: Dict[str, Tuple[object, Optional[Span]]] = {} + for dotted_key, value, span in entries: + coalesced[dotted_key] = (value, span) + keys = [ + PyConfigKey( + id=config_key_id(artifact.id, dotted_key), key=dotted_key, + namespace=namespace, + value=_stringify(value) if capture_value else None, + span=span, references=_find_references(_stringify(value)), + ) + for dotted_key, (value, span) in coalesced.items() + ] + keys.sort(key=lambda k: k.key) + return keys, True + except Exception: + return [], False diff --git a/test/test_config_key_extraction.py b/test/test_config_key_extraction.py new file mode 100644 index 0000000..cb438cf --- /dev/null +++ b/test/test_config_key_extraction.py @@ -0,0 +1,230 @@ +"""Task 2: config-key flatteners with reference recognition.""" +import textwrap + +from codeanalyzer.artifacts.config_keys import extract_config_keys +from codeanalyzer.schema.ids import artifact_id, config_key_id +from codeanalyzer.schema.py_schema import PyArtifact + + +def _artifact(path: str, fmt: str) -> PyArtifact: + return PyArtifact(id=artifact_id("app", path), path=path, format=fmt) + + +def _by_key(keys): + return {k.key: k for k in keys} + + +# --- yaml: nesting, arrays, determinism ------------------------------------- + +def test_yaml_nested_dotted_and_numeric_array_segments(): + text = textwrap.dedent("""\ + db: + host: localhost + port: 5432 + services: + - name: web + ports: + - 8080 + - 8081 + """) + art = _artifact("config.yaml", "yaml") + keys, ok = extract_config_keys(art, text, True) + assert ok is True + by = _by_key(keys) + assert by["db.host"].value == "localhost" + assert by["db.port"].value == "5432" + assert by["services.0.name"].value == "web" + assert by["services.0.ports.0"].value == "8080" + assert by["services.0.ports.1"].value == "8081" + assert all(k.namespace == "yaml" for k in keys) + assert all(k.id == config_key_id(art.id, k.key) for k in keys) + # L1 determinism: sorted key order within the artifact. + assert [k.key for k in keys] == sorted(k.key for k in keys) + + +def test_yaml_best_effort_span_slices_the_key_line(): + text = "db:\n host: localhost\n" + art = _artifact("config.yaml", "yaml") + keys, ok = extract_config_keys(art, text, True) + host = _by_key(keys)["db.host"] + assert host.span is not None + assert host.span.start[0] == 2 # " host: localhost" is line 2 + assert "localhost" in text[host.span.bytes[0]:host.span.bytes[1]] + + +# --- json: nesting ------------------------------------------------------ + +def test_json_nesting(): + text = textwrap.dedent("""\ + { + "database": { + "host": "localhost", + "replicas": ["r1", "r2"] + } + } + """) + art = _artifact("config.json", "json") + keys, ok = extract_config_keys(art, text, True) + assert ok is True + by = _by_key(keys) + assert by["database.host"].value == "localhost" + assert by["database.replicas.0"].value == "r1" + assert by["database.replicas.1"].value == "r2" + assert all(k.namespace == "json" for k in keys) + + +# --- toml: tables --------------------------------------------------------- + +def test_toml_tables(): + text = textwrap.dedent("""\ + [server] + host = "0.0.0.0" + port = 8080 + + [server.tls] + enabled = true + """) + art = _artifact("config.toml", "toml") + keys, ok = extract_config_keys(art, text, True) + assert ok is True + by = _by_key(keys) + assert by["server.host"].value == "0.0.0.0" + assert by["server.port"].value == "8080" + assert by["server.tls.enabled"].value == "true" + assert all(k.namespace == "toml" for k in keys) + + +# --- env: quoting, comments, export ---------------------------------------- + +def test_env_quoting_comments_export(): + text = textwrap.dedent("""\ + # a comment + export APP_NAME="myapp" + DEBUG=true + GREETING='hello world' + """) + art = _artifact(".env", "text") + keys, ok = extract_config_keys(art, text, True) + assert ok is True + by = _by_key(keys) + assert by["APP_NAME"].value == "myapp" + assert by["DEBUG"].value == "true" + assert by["GREETING"].value == "hello world" + assert all(k.namespace == "env" for k in keys) + # exact line: APP_NAME is on line 2, and the span slices back to it. + app_name = by["APP_NAME"] + assert app_name.span.start == (2, 0) + assert "APP_NAME" in text[app_name.span.bytes[0]:app_name.span.bytes[1]] + + +def test_env_family_basename_dispatch_regardless_of_format(): + for path in (".env", ".env.local", ".flaskenv"): + art = _artifact(path, "text") + keys, ok = extract_config_keys(art, "FOO=bar\n", True) + assert ok is True + assert keys[0].namespace == "env", path + # a near-miss basename must NOT be treated as env-family. + art = _artifact(".environment", "text") + keys, ok = extract_config_keys(art, "FOO=bar\n", True) + assert ok is True and keys == [] + + +# --- properties: continuations, comments ----------------------------------- + +def test_properties_continuations_and_comments(): + text = textwrap.dedent("""\ + ! a bang comment + # a hash comment + app.name=MyApp + message=Welcome to \\ + Wonderland + """) + art = _artifact("app.properties", "properties") + keys, ok = extract_config_keys(art, text, True) + assert ok is True + by = _by_key(keys) + assert by["app.name"].value == "MyApp" + assert by["message"].value == "Welcome to Wonderland" + assert all(k.namespace == "properties" for k in keys) + + +def test_properties_key_colon_form(): + art = _artifact("app.properties", "properties") + keys, ok = extract_config_keys(art, "greeting: hi\n", True) + assert ok is True + assert _by_key(keys)["greeting"].value == "hi" + + +# --- ini: raw interpolation preserved + reference recognized --------------- + +def test_ini_interpolation_preserved_raw_and_reference_recognized(): + text = textwrap.dedent("""\ + [paths] + home = /usr/local + here = %(home)s/app + """) + art = _artifact("tox.ini", "ini") + keys, ok = extract_config_keys(art, text, True) + assert ok is True + by = _by_key(keys) + assert by["paths.home"].value == "/usr/local" + # raw=True: NOT interpolated to "/usr/local/app". + assert by["paths.here"].value == "%(home)s/app" + assert by["paths.here"].references == ["%(home)s"] + assert all(k.namespace == "ini" for k in keys) + # exact line (not best-effort): "here" is on line 3. + assert by["paths.here"].span.start[0] == 3 + + +# --- references: all three syntaxes, order of appearance, dedupe ----------- + +def test_references_all_three_syntaxes_order_and_dedupe(): + text = ( + 'greeting: "start $FOO middle ${BAR} then ${{ tmpl.expr }} ' + 'and %(baz)s end $FOO"\n' + ) + art = _artifact("config.yaml", "yaml") + keys, ok = extract_config_keys(art, text, True) + assert ok is True + refs = _by_key(keys)["greeting"].references + assert refs == ["$FOO", "${BAR}", "${{ tmpl.expr }}", "%(baz)s"] + + +# --- value gating: capture_value=False ------------------------------------ + +def test_capture_value_false_hides_value_only(): + text = "db:\n host: localhost\n" + art = _artifact("config.yaml", "yaml") + with_value, ok1 = extract_config_keys(art, text, True) + without_value, ok2 = extract_config_keys(art, text, False) + assert ok1 is True and ok2 is True + assert len(with_value) == len(without_value) == 1 + a, b = with_value[0], without_value[0] + assert a.value == "localhost" + assert b.value is None + # everything else identical. + assert a.id == b.id and a.key == b.key and a.namespace == b.namespace + assert a.span == b.span and a.references == b.references + + +# --- dispatch: unsupported format -> ([], True) ----------------------------- + +def test_unsupported_format_returns_empty_ok(): + art = _artifact("Dockerfile", "dockerfile") + keys, ok = extract_config_keys(art, "FROM python:3.12\n", True) + assert keys == [] and ok is True + + +# --- malformed input per format: failure signaled, never raises ------------ + +def test_malformed_input_never_raises_signals_failure(): + cases = [ + ("bad.json", "json", '{"a": '), + ("bad.yaml", "yaml", "key:\n\tbad: 1\n"), + ("bad.toml", "toml", "key = \n"), + ("bad.ini", "ini", "[a]\nx = 1\nx = 2\n"), # duplicate option + ] + for path, fmt, text in cases: + art = _artifact(path, fmt) + keys, ok = extract_config_keys(art, text, True) + assert keys == [] and ok is False, f"{fmt} should signal failure, not raise" From 0c646b9dfa9e0b7c48fb810a1e6015438cfdebf0 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 12:01:26 -0700 Subject: [PATCH 05/10] fix(artifacts): env quote/comment interaction, DEFAULT-section ini keys --- codeanalyzer/artifacts/config_keys.py | 35 ++++++++++-- .../2026-08-28-config-key-family-design.md | 4 ++ test/test_config_key_extraction.py | 57 +++++++++++++++++++ 3 files changed, 91 insertions(+), 5 deletions(-) diff --git a/codeanalyzer/artifacts/config_keys.py b/codeanalyzer/artifacts/config_keys.py index af73c4f..31a4558 100644 --- a/codeanalyzer/artifacts/config_keys.py +++ b/codeanalyzer/artifacts/config_keys.py @@ -99,10 +99,20 @@ def _parse_toml(text: str, lines: List[str]) -> List[_Entry]: _ENV_LINE = re.compile(r'^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$') -def _strip_quotes(value: str) -> str: - if len(value) >= 2 and value[0] == value[-1] and value[0] in "'\"": - return value[1:-1] - return value +def _env_value(raw: str) -> str: + """The text after `KEY=` on one line -> the value. A quoted value ends + at its MATCHING closing quote -- anything after that (including a `#`) + is trailing comment and is discarded, so a `#` INSIDE the quotes (e.g. a + URL fragment) is never reached by comment-stripping. An unquoted value + ends at the first unescaped `" #"` (whitespace then `#`); a bare `#` + stuck directly to a token (no preceding whitespace) is not a comment + marker and stays in the value.""" + raw = raw.strip() + if raw and raw[0] in "'\"": + quote = raw[0] + end = raw.find(quote, 1) + return raw[1:end] if end != -1 else raw[1:] + return re.split(r"\s#", raw, maxsplit=1)[0].strip() def _is_env_family(basename: str) -> bool: @@ -118,7 +128,7 @@ def _parse_env(text: str, lines: List[str]) -> List[_Entry]: m = _ENV_LINE.match(stripped) if not m: continue - key, value = m.group(1), _strip_quotes(m.group(2).strip()) + key, value = m.group(1), _env_value(m.group(2)) out.append((key, value, _line_span(text, lines, lineno))) return out @@ -180,6 +190,21 @@ def _parse_ini(text: str, lines: List[str]) -> List[_Entry]: cp.read_string(text) line_map = _ini_line_map(lines) out: List[_Entry] = [] + # DEFAULT's own keys, unconditionally: `cp.sections()` never includes + # "DEFAULT" (configparser convention), so a file with only a [DEFAULT] + # section would otherwise yield zero keys. `cp.items(section, ...)` + # below ALSO re-inherits every DEFAULT key into each real section + # (configparser's fallback-lookup semantics) -- so a key defined only in + # DEFAULT deliberately appears twice: once as `DEFAULT.` and again + # as `
.` per inheriting section. Both are real, distinct + # facts (the key is DEFINED in DEFAULT; the section's own resolved + # value equals it), so both stay -- this is intended duplication, not a + # bug (see docs/design/specs/2026-08-28-config-key-family-design.md + # Caveats). + for key, value in cp.defaults().items(): + lineno = line_map.get(("DEFAULT", key)) + span = _line_span(text, lines, lineno) if lineno else None + out.append((f"DEFAULT.{key}", value, span)) for section in cp.sections(): for key, value in cp.items(section, raw=True): lineno = line_map.get((section, key)) or line_map.get(("DEFAULT", key)) diff --git a/docs/design/specs/2026-08-28-config-key-family-design.md b/docs/design/specs/2026-08-28-config-key-family-design.md index 67c26dc..9ba1c59 100644 --- a/docs/design/specs/2026-08-28-config-key-family-design.md +++ b/docs/design/specs/2026-08-28-config-key-family-design.md @@ -83,6 +83,10 @@ asset inherits it); conformance fixture grows a config-bearing artifact. cross-artifact joins are the consumer's query (see analyses.md examples). - env-family files are `format: "text"` with role `env`; namespace `env` is keyed off the basename rule, not the format. +- Literal dotted keys are indistinguishable from nesting once flattened into + a dotted-path id (a top-level key literally named `"a.b"` and a nested + `a: {b: ...}` both flatten to `a.b`); colliding forms coalesce last-wins + by design, same as a plain duplicate key within one format. ## Definition of done diff --git a/test/test_config_key_extraction.py b/test/test_config_key_extraction.py index cb438cf..4e8abb6 100644 --- a/test/test_config_key_extraction.py +++ b/test/test_config_key_extraction.py @@ -117,6 +117,27 @@ def test_env_quoting_comments_export(): assert "APP_NAME" in text[app_name.span.bytes[0]:app_name.span.bytes[1]] +def test_env_quoted_value_with_trailing_comment(): + art = _artifact(".env", "text") + keys, ok = extract_config_keys(art, 'SECRET="abc123" # rotate quarterly\n', True) + assert ok is True + assert _by_key(keys)["SECRET"].value == "abc123" + + +def test_env_unquoted_value_with_trailing_comment(): + art = _artifact(".env", "text") + keys, ok = extract_config_keys(art, "FOO=bar # c\n", True) + assert ok is True + assert _by_key(keys)["FOO"].value == "bar" + + +def test_env_hash_inside_quotes_is_preserved(): + art = _artifact(".env", "text") + keys, ok = extract_config_keys(art, 'URL="http://x#frag"\n', True) + assert ok is True + assert _by_key(keys)["URL"].value == "http://x#frag" + + def test_env_family_basename_dispatch_regardless_of_format(): for path in (".env", ".env.local", ".flaskenv"): art = _artifact(path, "text") @@ -176,6 +197,27 @@ def test_ini_interpolation_preserved_raw_and_reference_recognized(): assert by["paths.here"].span.start[0] == 3 +def test_ini_default_only_emits_default_prefixed_keys(): + art = _artifact("tox.ini", "ini") + keys, ok = extract_config_keys(art, "[DEFAULT]\ntimeout = 30\n", True) + assert ok is True + assert _by_key(keys)["DEFAULT.timeout"].value == "30" + + +def test_ini_default_and_section_both_emit_duplicated_key(): + # configparser inherits every DEFAULT key into each real section, so a + # key defined only in DEFAULT deliberately shows up twice: once as + # DEFAULT., once per inheriting section as
.. + text = "[DEFAULT]\ntimeout = 30\n\n[server]\nhost = 0.0.0.0\n" + art = _artifact("tox.ini", "ini") + keys, ok = extract_config_keys(art, text, True) + assert ok is True + by = _by_key(keys) + assert by["DEFAULT.timeout"].value == "30" + assert by["server.timeout"].value == "30" + assert by["server.host"].value == "0.0.0.0" + + # --- references: all three syntaxes, order of appearance, dedupe ----------- def test_references_all_three_syntaxes_order_and_dedupe(): @@ -207,6 +249,21 @@ def test_capture_value_false_hides_value_only(): assert a.span == b.span and a.references == b.references +# --- duplicate dotted key: last-wins coalescing ----------------------------- + +def test_literal_dotted_key_collides_with_nesting_last_wins(): + # A literal top-level key "a.b" and a nested a -> {b: ...} both flatten + # to the same dotted path "a.b" -- indistinguishable once flattened, so + # they coalesce into one record (last occurrence in the file wins). + text = "a.b: 1\na:\n b: 2\n" + art = _artifact("config.yaml", "yaml") + keys, ok = extract_config_keys(art, text, True) + assert ok is True + matches = [k for k in keys if k.key == "a.b"] + assert len(matches) == 1 + assert matches[0].value == "2" + + # --- dispatch: unsupported format -> ([], True) ----------------------------- def test_unsupported_format_returns_empty_ok(): From 4210a32217587bbe866fbd9618e714851087ef74 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 12:14:30 -0700 Subject: [PATCH 06/10] feat(artifacts): wire config-key extraction into analyze; tf rule; ecosystem field Layers extract_config_keys onto every namespace-eligible artifact right after build_dependency_view in core.analyze() (L1 data, every level): env family by basename, else by format in {yaml,json,toml,ini,properties}; full text read fresh from disk (mirrors dependencies._full_text), binary artifacts skipped, a parse failure downgrades extraction to "partial" without dropping the artifact. Riders: *.tf -> new role iac; .flaskenv joins the env basename family; *.properties is a new format (tool-config); a generic *.ini rule after the existing tox.ini rule makes non-tox ini files namespace-eligible too; PyDependency.ecosystem: str = "pypi" for SDK symmetry with purl, set explicitly at both declared and lock-only-transitive construction sites. --- codeanalyzer/artifacts/__init__.py | 6 +++- codeanalyzer/artifacts/config_keys.py | 19 ++++++++++++ codeanalyzer/artifacts/dependencies.py | 4 +-- codeanalyzer/artifacts/discovery.py | 7 +++++ codeanalyzer/core.py | 34 ++++++++++++++++++++- codeanalyzer/schema/py_schema.py | 3 +- test/test_artifact_discovery.py | 20 ++++++++++++ test/test_artifact_pipeline.py | 42 ++++++++++++++++++++++++++ test/test_config_key_extraction.py | 29 +++++++++++++++++- test/test_dependency_view.py | 17 +++++++++++ 10 files changed, 175 insertions(+), 6 deletions(-) diff --git a/codeanalyzer/artifacts/__init__.py b/codeanalyzer/artifacts/__init__.py index a7af4c9..07eddf6 100644 --- a/codeanalyzer/artifacts/__init__.py +++ b/codeanalyzer/artifacts/__init__.py @@ -5,7 +5,11 @@ text or binary -- issue #157 follow-up); extraction is narrow (only dependency manifests are parsed for meaning in this unit).""" +from codeanalyzer.artifacts.config_keys import extract_config_keys, is_config_eligible from codeanalyzer.artifacts.dependencies import build_dependency_view from codeanalyzer.artifacts.discovery import discover_artifacts -__all__ = ["discover_artifacts", "build_dependency_view"] +__all__ = [ + "discover_artifacts", "build_dependency_view", + "extract_config_keys", "is_config_eligible", +] diff --git a/codeanalyzer/artifacts/config_keys.py b/codeanalyzer/artifacts/config_keys.py index 31a4558..1139d2c 100644 --- a/codeanalyzer/artifacts/config_keys.py +++ b/codeanalyzer/artifacts/config_keys.py @@ -261,6 +261,25 @@ def _stringify(value: object) -> str: # --- public API ----------------------------------------------------------- +def is_config_eligible(artifact: PyArtifact) -> bool: + """Whether `artifact` is worth extracting config keys from: an env-family + basename (`.env`/`.env.*`/`.flaskenv`, regardless of declared format), or + a namespace-bearing format (yaml/json/toml/ini/properties). A binary + artifact is never eligible -- there is no decodable text to flatten, and + a rule-matched-but-undecodable file downgrades to `format="binary"` + regardless of its basename (see discovery.py), so the binary check wins + even over an env-family name. + + Callers (core.py's wiring) use this to skip the on-disk read + parse + attempt entirely on artifacts that can never yield config keys, rather + than relying on `extract_config_keys`'s own not-applicable `([], True)` + return after already having paid for the read.""" + if artifact.format == "binary": + return False + basename = artifact.path.rsplit("/", 1)[-1] + return _is_env_family(basename) or artifact.format in _NAMESPACE_PARSERS + + def extract_config_keys( artifact: PyArtifact, full_text: str, capture_value: bool, ) -> Tuple[List[PyConfigKey], bool]: diff --git a/codeanalyzer/artifacts/dependencies.py b/codeanalyzer/artifacts/dependencies.py index 3207b56..c1a8558 100644 --- a/codeanalyzer/artifacts/dependencies.py +++ b/codeanalyzer/artifacts/dependencies.py @@ -104,7 +104,7 @@ def build_dependency_view( def _emit(raw: List[RawDep], declared_in: str, kind_override: Optional[str] = None) -> None: for r in raw: deps.append(PyDependency( - name=r.name, spec=r.spec, + name=r.name, ecosystem="pypi", spec=r.spec, kind=kind_override if kind_override is not None else r.kind, extras=sorted(r.extras), declared_in=declared_in, prov=["declared"], )) @@ -177,7 +177,7 @@ def _emit(raw: List[RawDep], declared_in: str, kind_override: Optional[str] = No declared_names = {d.name for d in deps} for name in sorted(set(pins) - declared_names): deps.append(PyDependency( - name=name, kind="runtime", declared_in=pin_lock_artifact[name], + name=name, ecosystem="pypi", kind="runtime", declared_in=pin_lock_artifact[name], direct=False, locked_version=pins[name], prov=["lockfile"], )) diff --git a/codeanalyzer/artifacts/discovery.py b/codeanalyzer/artifacts/discovery.py index 5bf20c7..6f564f6 100644 --- a/codeanalyzer/artifacts/discovery.py +++ b/codeanalyzer/artifacts/discovery.py @@ -33,11 +33,13 @@ ("kind/*.yaml", "yaml", ["service-topology"]), ("Chart.yaml", "yaml", ["service-topology"]), ("values.yaml", "yaml", ["service-topology"]), + ("*.tf", "text", ["iac"]), (".github/workflows/*.yml", "yaml", ["ci"]), (".github/workflows/*.yaml", "yaml", ["ci"]), (".gitlab-ci.yml", "yaml", ["ci"]), (".env", "text", ["env"]), (".env.*", "text", ["env"]), + (".flaskenv", "text", ["env"]), ("tox.ini", "ini", ["tool-config"]), ("noxfile.py", "text", ["tool-config"]), ("Makefile", "text", ["tool-config"]), @@ -49,6 +51,11 @@ ("*.rst", "text", ["docs"]), ("*.cfg", "ini", ["unknown"]), ("*.toml", "toml", ["unknown"]), + ("*.properties", "properties", ["tool-config"]), + # Generic fallback AFTER the specific tox.ini rule above, so a non-tox + # *.ini file (mypy.ini, pytest.ini, ...) still reaches format="ini" -- + # config-key extraction (#152) is namespace-eligible by format. + ("*.ini", "ini", ["tool-config"]), ] _IGNORED_DIRS = { diff --git a/codeanalyzer/core.py b/codeanalyzer/core.py index f29ba77..07a749e 100644 --- a/codeanalyzer/core.py +++ b/codeanalyzer/core.py @@ -37,6 +37,19 @@ from codeanalyzer.options import AnalysisOptions from codeanalyzer.provenance import analyzer_info, repository_info +def _artifact_full_text(project_dir: Path, path: str, art) -> str: + """Mirrors ``artifacts.dependencies._full_text``: config-key extraction + (#152) must never depend on the stored ``source`` -- capped by + ``text_max_bytes`` and emptied by ``capture_text=False`` (payload-size + controls, not extraction controls). Read the real file fresh instead; + fall back to ``art.source`` only if it's gone (e.g. a synthetic artifact + in a unit test, or the file vanished mid-run).""" + try: + return (project_dir / path).read_bytes().decode("utf-8") + except (OSError, UnicodeDecodeError): + return art.source + + def _ensure_ray() -> None: """Initialize Ray with the driver's pinned hash seed in the workers. @@ -650,7 +663,10 @@ def analyze(self) -> Analysis: # Artifacts + dependencies: L1 data, every level, never varies with -a # (spec 2026-08-27). Deterministic by default; venv probing is opt-in. - from codeanalyzer.artifacts import build_dependency_view, discover_artifacts + from codeanalyzer.artifacts import ( + build_dependency_view, discover_artifacts, extract_config_keys, + is_config_eligible, + ) app.artifacts = discover_artifacts( self.project_dir, app_name, @@ -665,6 +681,22 @@ def analyze(self) -> Analysis: self.options.resolve_installed, ) + # Config keys (#152): L1 data, layered onto the same artifacts, every + # level. Namespace-eligible artifacts only (env-family by basename, + # else by format); a parse failure never drops the artifact -- it + # downgrades `extraction` to "partial" instead (never overwritten on + # success, so it doesn't clobber a "partial" already set above by + # dependency-manifest parsing on the same artifact). + for path in sorted(app.artifacts): + art = app.artifacts[path] + if not is_config_eligible(art): + continue + full_text = _artifact_full_text(self.project_dir, path, art) + keys, ok = extract_config_keys(art, full_text, self.options.artifact_text) + art.config_keys = keys + if not ok: + art.extraction = "partial" + # L3: intraprocedural dataflow (CFG/CDG/DDG) emitted onto the v2 tree. if self.analysis_level >= 3: from codeanalyzer.dataflow.builder import ( diff --git a/codeanalyzer/schema/py_schema.py b/codeanalyzer/schema/py_schema.py index 03ad8d2..6fee577 100644 --- a/codeanalyzer/schema/py_schema.py +++ b/codeanalyzer/schema/py_schema.py @@ -498,7 +498,7 @@ class PyArtifact(BaseModel): id: str = "" kind: str = "artifact" path: str # repo-relative POSIX path (also the map key) - format: str # toml|yaml|json|ini|requirements|dockerfile|text|binary + format: str # toml|yaml|json|ini|properties|requirements|dockerfile|text|binary roles: List[str] = [] size_bytes: int = 0 sha256: str = "" # always the full file's hash, even when source is truncated/empty @@ -513,6 +513,7 @@ class PyDependency(BaseModel): """One declared third-party dependency, evidence-tagged via ``prov``.""" name: str # PEP 503 normalized + ecosystem: str = "pypi" # SDK symmetry with purl (#152 rider); the only ecosystem this analyzer emits spec: str = "" kind: str = "runtime" # runtime|dev|optional|build extras: List[str] = [] diff --git a/test/test_artifact_discovery.py b/test/test_artifact_discovery.py index 50938f4..be3df82 100644 --- a/test/test_artifact_discovery.py +++ b/test/test_artifact_discovery.py @@ -243,3 +243,23 @@ def test_dependency_manifest_still_empty_source_with_capture_text_false(tmp_path arts = discover_artifacts(tmp_path, "a", capture_text=False, text_max_bytes=16) art = arts["pyproject.toml"] assert art.source == "" and art.text_truncated is False + + +def test_discovers_terraform_flaskenv_properties_and_generic_ini(tmp_path): + """Task 3 riders (#152): *.tf -> new role iac; .flaskenv joins the env + basename family; *.properties is a new format; a generic *.ini rule + (placed after the specific tox.ini rule) makes non-tox ini files + namespace-eligible for config-key extraction too.""" + _mk(tmp_path, "main.tf", 'resource "x" "y" {}\n') + _mk(tmp_path, ".flaskenv", "FLASK_ENV=production\n") + _mk(tmp_path, "app.properties", "key=value\n") + _mk(tmp_path, "mypy.ini", "[mypy]\nstrict = true\n") + _mk(tmp_path, "tox.ini", "[tox]\nenvlist = py312\n") + arts = discover_artifacts(tmp_path, "a") + assert arts["main.tf"].format == "text" and arts["main.tf"].roles == ["iac"] + assert arts[".flaskenv"].format == "text" and arts[".flaskenv"].roles == ["env"] + assert arts["app.properties"].format == "properties" + assert arts["app.properties"].roles == ["tool-config"] + assert arts["mypy.ini"].format == "ini" and arts["mypy.ini"].roles == ["tool-config"] + # tox.ini keeps matching its own specific (pre-existing) rule, unshadowed. + assert arts["tox.ini"].format == "ini" and arts["tox.ini"].roles == ["tool-config"] diff --git a/test/test_artifact_pipeline.py b/test/test_artifact_pipeline.py index a6716b8..154641a 100644 --- a/test/test_artifact_pipeline.py +++ b/test/test_artifact_pipeline.py @@ -69,3 +69,45 @@ def test_artifact_text_options_thread_through_core(tmp_path): )).analyze().application assert no_text.artifacts["notes.md"].source == "" assert no_text.artifacts["notes.md"].text_truncated is False + + +def test_config_keys_present_and_identical_across_levels(tmp_path): + """Task 3: config-key extraction is wired into core.analyze() beside + build_dependency_view -- L1 data, identical at every analysis level.""" + proj = tmp_path / "proj" + proj.mkdir() + (proj / "pyproject.toml").write_text( + '[project]\ndependencies = ["requests"]\n\n[tool.demo]\nkey = "val"\n' + ) + (proj / "app.py").write_text("import requests\n") + a1 = _run(tmp_path, proj, 1) + a2 = _run(tmp_path, proj, 2) + d1 = json.loads(model_dump_json(a1)) + d2 = json.loads(model_dump_json(a2)) + keys1 = d1["artifacts"]["pyproject.toml"]["config_keys"] + keys2 = d2["artifacts"]["pyproject.toml"]["config_keys"] + assert keys1 == keys2 + assert keys1, "expected non-empty config_keys on pyproject.toml" + assert {k["key"] for k in keys1} >= {"tool.demo.key"} + + +def test_config_key_eligibility_and_partial_on_parse_failure(tmp_path): + """Namespace-eligibility: env-family by basename regardless of format + (.env is format="text"), a non-eligible format never even attempts + extraction (config_keys stays []), and a parse failure never drops the + artifact -- it downgrades extraction to "partial" instead.""" + proj = tmp_path / "proj" + proj.mkdir() + (proj / ".env").write_text("SECRET=${TOKEN}\n") + (proj / "notes.md").write_text("just docs, not config\n") + (proj / "broken.toml").write_text("key = [unterminated\n") + app = _run(tmp_path, proj, 1) + + env_keys = {k.key: k for k in app.artifacts[".env"].config_keys} + assert env_keys["SECRET"].namespace == "env" + assert env_keys["SECRET"].references == ["${TOKEN}"] + + assert app.artifacts["notes.md"].config_keys == [] + + assert app.artifacts["broken.toml"].config_keys == [] + assert app.artifacts["broken.toml"].extraction == "partial" diff --git a/test/test_config_key_extraction.py b/test/test_config_key_extraction.py index 4e8abb6..1a0c98f 100644 --- a/test/test_config_key_extraction.py +++ b/test/test_config_key_extraction.py @@ -1,7 +1,7 @@ """Task 2: config-key flatteners with reference recognition.""" import textwrap -from codeanalyzer.artifacts.config_keys import extract_config_keys +from codeanalyzer.artifacts.config_keys import extract_config_keys, is_config_eligible from codeanalyzer.schema.ids import artifact_id, config_key_id from codeanalyzer.schema.py_schema import PyArtifact @@ -285,3 +285,30 @@ def test_malformed_input_never_raises_signals_failure(): art = _artifact(path, fmt) keys, ok = extract_config_keys(art, text, True) assert keys == [] and ok is False, f"{fmt} should signal failure, not raise" + + +# --- is_config_eligible: Task 3's core.py wiring uses this to skip a disk +# read + extraction attempt on artifacts that can never yield config keys --- + +def test_is_config_eligible_env_family_regardless_of_format(): + # .env/.env.*/.flaskenv are format="text" on disk (discovery.py) -- still + # eligible via the basename rule, independent of the declared format. + for name in (".env", ".env.production", ".flaskenv"): + assert is_config_eligible(_artifact(name, "text")) is True + + +def test_is_config_eligible_by_namespace_bearing_format(): + for fmt in ("yaml", "json", "toml", "ini", "properties"): + assert is_config_eligible(_artifact(f"config.{fmt}", fmt)) is True + + +def test_is_config_eligible_false_for_other_formats(): + for fmt in ("text", "dockerfile", "requirements"): + assert is_config_eligible(_artifact("misc", fmt)) is False + + +def test_is_config_eligible_false_for_binary_even_if_env_basename(): + # A rule-matched-but-undecodable file downgrades to format="binary" + # regardless of basename (discovery.py) -- never eligible, there is no + # decodable text to flatten. + assert is_config_eligible(_artifact(".env", "binary")) is False diff --git a/test/test_dependency_view.py b/test/test_dependency_view.py index e5ed419..68142b9 100644 --- a/test/test_dependency_view.py +++ b/test/test_dependency_view.py @@ -226,6 +226,23 @@ def test_large_lock_parses_all_pins_even_with_capture_text_false(tmp_path): assert arts["uv.lock"].extraction == "full" +def test_ecosystem_pypi_on_every_record(tmp_path): + """#152 rider: PyDependency.ecosystem is set explicitly (SDK symmetry with + purl) on every record -- both the declared-manifest branch (_emit) and + the lock-only transitive branch.""" + (tmp_path / "pyproject.toml").write_text('[project]\ndependencies = ["requests>=2.31"]\n') + (tmp_path / "uv.lock").write_text( + '[[package]]\nname = "requests"\nversion = "2.32.3"\n' + '[[package]]\nname = "urllib3"\nversion = "2.2.1"\n' + ) + arts = discover_artifacts(tmp_path, "app") + deps, _ = build_dependency_view(arts, {}, tmp_path, None, False) + by = {d.name: d for d in deps} + assert set(by) == {"requests", "urllib3"} # sanity: both branches present + assert by["urllib3"].direct is False # the lock-only transitive branch + assert all(d.ecosystem == "pypi" for d in deps) + + def test_corrupted_lock_extraction_is_partial_not_full(tmp_path): """A lock with real (non-empty) content that parse_lock_pins can't make sense of must not claim extraction="full" for zero pins extracted.""" From 8ca091a149d4b23d1e31c4485296af3085e0cc43 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 12:23:29 -0700 Subject: [PATCH 07/10] feat(neo4j): neutral ConfigKey nodes via DEFINES_CONFIG ConfigKey (merge key id; props key, namespace, value?, references, start_line, end_line) plus the DEFINES_CONFIG containment edge (Artifact -> ConfigKey), projected per artifact in _project_artifacts sorted by key. value is omitted (not null) when the model's value is None, matching prune()'s null-is-absent convention. schema.neo4j.json regenerated; sample_graph_app.py now runs the real extractor over its manifest's own source so the every-label conformance test covers the new label/relationship. --- codeanalyzer/neo4j/project.py | 20 ++++++++ codeanalyzer/neo4j/schema.py | 14 ++++++ schema.neo4j.json | 27 +++++++++- test/sample_graph_app.py | 11 ++++- test/test_neo4j_config_keys.py | 90 ++++++++++++++++++++++++++++++++++ 5 files changed, 160 insertions(+), 2 deletions(-) create mode 100644 test/test_neo4j_config_keys.py diff --git a/codeanalyzer/neo4j/project.py b/codeanalyzer/neo4j/project.py index 39ce62b..38f933c 100644 --- a/codeanalyzer/neo4j/project.py +++ b/codeanalyzer/neo4j/project.py @@ -310,6 +310,26 @@ def _project_artifacts(b: RowBuilder, app: PyApplication, app_name: str, app_ref ) b.edge("HAS_ARTIFACT", app_ref, art_ref) + # Config keys flattened out of this artifact (#152) -- sorted by key + # for deterministic row order, matching the JSON side's L1 determinism. + for ck in sorted(art.config_keys or [], key=lambda k: k.key): + ck_ref = b.node( + ["ConfigKey"], + "id", + ck.id, + prune( + { + "key": ck.key, + "namespace": ck.namespace, + "value": ck.value, + "references": list(ck.references or []), + "start_line": ck.span.start[0] if ck.span else None, + "end_line": ck.span.end[0] if ck.span else None, + } + ), + ) + b.edge("DEFINES_CONFIG", art_ref, ck_ref) + # Every lock artifact present LOCKS every dependency it pinned. The pins # from all lock files are already merged into one `locked_version` per # dependency upstream (Task 5 `build_dependency_view`) -- there is no diff --git a/codeanalyzer/neo4j/schema.py b/codeanalyzer/neo4j/schema.py index c45f134..93e1e80 100644 --- a/codeanalyzer/neo4j/schema.py +++ b/codeanalyzer/neo4j/schema.py @@ -215,6 +215,16 @@ class RelType: NodeLabel("Package", "Package", "id", { "id": "string", "ecosystem": "string", "name": "string", }), + # A configuration key flattened out of a config-bearing Artifact (#152). + # Neutral vocabulary like Artifact/Package -- a yaml/env/ini key is not a + # Python concept. `value` is omitted (not null) when the source model's + # value is None (--no-artifact-text, or a namespace with no value at that + # path); `references` is always present, possibly empty. + NodeLabel("ConfigKey", "ConfigKey", "id", { + "id": "string", "key": "string", "namespace": "string", + "value": "string", "references": "string[]", + **_SPAN, + }), ] _DECL_TARGETS = ["PyClass", "PyCallable"] @@ -266,6 +276,10 @@ class RelType: RelType("PY_SUMMARY", ["PyBodyNode"], ["PyBodyNode"]), # Neutral artifact/dependency subgraph (Task 6). RelType("HAS_ARTIFACT", ["PyApplication"], ["Artifact"]), + # A config key nests under exactly one owning artifact (its id is + # `@key/`) -- a plain containment edge, no + # per-edge properties or discriminant needed (#152). + RelType("DEFINES_CONFIG", ["Artifact"], ["ConfigKey"]), # ``_k`` (merges per ``kind``): the same manifest may declare one package # twice under different kinds (e.g. a runtime dep re-listed under an # optional extra) -- same endpoint pair, so without the discriminant the diff --git a/schema.neo4j.json b/schema.neo4j.json index ffda3f1..53d2452 100644 --- a/schema.neo4j.json +++ b/schema.neo4j.json @@ -177,6 +177,20 @@ "ecosystem": "string", "name": "string" } + }, + { + "label": "ConfigKey", + "merge_label": "ConfigKey", + "key": "id", + "properties": { + "id": "string", + "key": "string", + "namespace": "string", + "value": "string", + "references": "string[]", + "start_line": "integer", + "end_line": "integer" + } } ], "relationship_types": [ @@ -391,6 +405,16 @@ ], "properties": {} }, + { + "type": "DEFINES_CONFIG", + "from": [ + "Artifact" + ], + "to": [ + "ConfigKey" + ], + "properties": {} + }, { "type": "DECLARES_DEPENDENCY", "from": [ @@ -453,7 +477,8 @@ "CREATE CONSTRAINT pyvariable_id IF NOT EXISTS FOR (x:PyVariable) REQUIRE x.id IS UNIQUE", "CREATE CONSTRAINT pybodynode_id IF NOT EXISTS FOR (x:PyBodyNode) REQUIRE x.id IS UNIQUE", "CREATE CONSTRAINT artifact_id IF NOT EXISTS FOR (x:Artifact) REQUIRE x.id IS UNIQUE", - "CREATE CONSTRAINT package_id IF NOT EXISTS FOR (x:Package) REQUIRE x.id IS UNIQUE" + "CREATE CONSTRAINT package_id IF NOT EXISTS FOR (x:Package) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT configkey_id IF NOT EXISTS FOR (x:ConfigKey) REQUIRE x.id IS UNIQUE" ], "indexes": [ "CREATE INDEX py_callable_name IF NOT EXISTS FOR (c:PyCallable) ON (c.name)", diff --git a/test/sample_graph_app.py b/test/sample_graph_app.py index 40eb511..04f7f12 100644 --- a/test/sample_graph_app.py +++ b/test/sample_graph_app.py @@ -6,7 +6,8 @@ ``body``/``cfg``/``cdg``/``ddg`` (level 3), and — new at level 4 — the interprocedural ``param_in``/``param_out``/``summary`` param-passing overlay plus the points-to ``ddg`` delta. Also carries the artifact/dependency subgraph -(Task 6): a manifest + lock artifact and a locked, import-providing dependency. +(Task 6): a manifest + lock artifact and a locked, import-providing dependency, +plus the manifest's own flattened config keys (#152). The symbol table is built from a real (temporary) source file so ``build_function_pdgs`` can recover each callable's AST; ``assign_ids`` + @@ -34,6 +35,7 @@ emit_l3_body, emit_l4, ) +from codeanalyzer.artifacts.config_keys import extract_config_keys from codeanalyzer.dataflow.scalpel_oracle import make_alias_oracle from codeanalyzer.dataflow.syntactic import SyntacticOracle from codeanalyzer.schema import PyApplication, PyExternalSymbol @@ -168,6 +170,13 @@ def make_sample_app() -> Tuple[PyApplication, Dict[str, str]]: source="", extraction="full", ), } + # Config keys flattened out of the manifest (#152) -- the real extractor, + # run over the artifact's own source above, so the id/span shape is + # authentic rather than hand-computed. Exercises the ConfigKey label and + # DEFINES_CONFIG relationship (guarded by + # test_all_catalog_node_kinds_and_relationships_are_exercised). + pyproject = app.artifacts["pyproject.toml"] + pyproject.config_keys, _ = extract_config_keys(pyproject, pyproject.source, True) app.dependencies = [ PyDependency( name="acme", spec=">=1.0", kind="runtime", extras=[], diff --git a/test/test_neo4j_config_keys.py b/test/test_neo4j_config_keys.py new file mode 100644 index 0000000..c36db30 --- /dev/null +++ b/test/test_neo4j_config_keys.py @@ -0,0 +1,90 @@ +"""Task 4: neutral ConfigKey nodes via DEFINES_CONFIG in the Neo4j projection.""" +from codeanalyzer.core import Codeanalyzer +from codeanalyzer.neo4j.project import project +from codeanalyzer.neo4j.schema import CONSTRAINTS, NODE_LABELS, REL_TYPES +from codeanalyzer.options import AnalysisOptions +from codeanalyzer.schema.assign_ids import assign_ids + + +def test_catalog_has_config_key_label_and_rel(): + labels = {n.label: n for n in NODE_LABELS} + config_key = labels["ConfigKey"] + assert config_key.key == "id" and config_key.merge_label == "ConfigKey" + assert config_key.properties == { + "id": "string", "key": "string", "namespace": "string", + "value": "string", "references": "string[]", + "start_line": "integer", "end_line": "integer", + } + rels = {r.type: r for r in REL_TYPES} + defines = rels["DEFINES_CONFIG"] + assert defines.from_labels == ["Artifact"] and defines.to_labels == ["ConfigKey"] + + +def test_constraint_present_for_config_key(): + assert ( + "CREATE CONSTRAINT configkey_id IF NOT EXISTS FOR (x:ConfigKey) " + "REQUIRE x.id IS UNIQUE" + ) in CONSTRAINTS + + +def _analyze(proj, tmp_path, **opts): + return Codeanalyzer(AnalysisOptions( + input=proj, analysis_level=1, no_venv=True, cache_dir=tmp_path / "c", **opts, + )).analyze().application + + +def test_config_key_node_and_edge_projected(tmp_path): + proj = tmp_path / "p" + proj.mkdir() + (proj / "pyproject.toml").write_text( + '[project]\ndependencies = ["requests"]\n\n[tool.demo]\nkey = "val"\n' + ) + app = _analyze(proj, tmp_path) + rows = project(app, "p", assign_ids(app, "p")) + + manifest_id = app.artifacts["pyproject.toml"].id + ck = next( + k for k in app.artifacts["pyproject.toml"].config_keys if k.key == "tool.demo.key" + ) + + node = next(n for n in rows.nodes if n.labels[0] == "ConfigKey" and n.value == ck.id) + assert node.props["key"] == "tool.demo.key" + assert node.props["namespace"] == "toml" + assert node.props["value"] == "val" + assert node.props["references"] == [] # kept (present-but-empty), not pruned + + edge = next( + e for e in rows.edges if e.type == "DEFINES_CONFIG" and e.to_ref.value == ck.id + ) + assert edge.from_ref.label == "Artifact" and edge.from_ref.value == manifest_id + assert edge.to_ref.label == "ConfigKey" + + +def test_config_key_value_omitted_when_model_value_none(tmp_path): + """spec 2026-08-28 constraint 2: --no-artifact-text drops values (and + source) together -- keys/namespace/span/references are still extracted + from the real on-disk text either way.""" + proj = tmp_path / "p" + proj.mkdir() + (proj / "pyproject.toml").write_text('[tool.demo]\nkey = "val"\n') + app = _analyze(proj, tmp_path, artifact_text=False) + ck = app.artifacts["pyproject.toml"].config_keys[0] + assert ck.key == "tool.demo.key" and ck.value is None # sanity: still extracted + + rows = project(app, "p", assign_ids(app, "p")) + node = next(n for n in rows.nodes if n.labels[0] == "ConfigKey" and n.value == ck.id) + assert "value" not in node.props + + +def test_config_key_references_and_span_projected(tmp_path): + proj = tmp_path / "p" + proj.mkdir() + (proj / ".env").write_text("DATABASE_URL=${DB_HOST}\n") + app = _analyze(proj, tmp_path) + ck = app.artifacts[".env"].config_keys[0] + assert ck.references == ["${DB_HOST}"] + + rows = project(app, "p", assign_ids(app, "p")) + node = next(n for n in rows.nodes if n.labels[0] == "ConfigKey" and n.value == ck.id) + assert node.props["references"] == ["${DB_HOST}"] + assert node.props["start_line"] == 1 and node.props["end_line"] == 1 From 56b8575d3fef82ac3e110229e96c3526b48319dc Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 12:39:27 -0700 Subject: [PATCH 08/10] fix(artifacts): upgrade extraction to full on clean config-key parse A clean config-key parse never upgraded a non-manifest artifact's extraction past "none" -- every config-bearing file with no dependency role (values.yaml, bare .env, app.properties, ...) stayed "none" even after successfully yielding keys. Now a success upgrades an untouched "none" to "full", while still never overwriting an existing "partial" left by a failed dependency-manifest parse on the same artifact. Also cross-references core._artifact_full_text and dependencies._full_text in both docstrings so the deliberate duplication can't silently drift. --- codeanalyzer/artifacts/dependencies.py | 6 +++++- codeanalyzer/core.py | 23 +++++++++++++------- test/test_artifact_pipeline.py | 29 ++++++++++++++++++++++++++ 3 files changed, 49 insertions(+), 9 deletions(-) diff --git a/codeanalyzer/artifacts/dependencies.py b/codeanalyzer/artifacts/dependencies.py index c1a8558..8eea79b 100644 --- a/codeanalyzer/artifacts/dependencies.py +++ b/codeanalyzer/artifacts/dependencies.py @@ -85,7 +85,11 @@ def _full_text(project_dir: Path, path: str, art: PyArtifact) -> str: (both payload-size controls on the JSON/Neo4j payload, not extraction controls). Read the real file fresh instead; fall back to ``art.source`` only if it is gone (e.g. a synthetic artifact in a unit test, or the file - vanished mid-run).""" + vanished mid-run). + + Mirrored (not imported -- this name is module-private) by + ``core._artifact_full_text`` for the same reason on config-key + extraction (#152); keep the two in sync if this logic changes.""" try: return (project_dir / path).read_bytes().decode("utf-8") except (OSError, UnicodeDecodeError): diff --git a/codeanalyzer/core.py b/codeanalyzer/core.py index 07a749e..fbc01c0 100644 --- a/codeanalyzer/core.py +++ b/codeanalyzer/core.py @@ -38,12 +38,14 @@ from codeanalyzer.provenance import analyzer_info, repository_info def _artifact_full_text(project_dir: Path, path: str, art) -> str: - """Mirrors ``artifacts.dependencies._full_text``: config-key extraction - (#152) must never depend on the stored ``source`` -- capped by - ``text_max_bytes`` and emptied by ``capture_text=False`` (payload-size + """Mirrors ``artifacts.dependencies._full_text`` verbatim (not imported + -- that name is module-private to ``dependencies.py``): config-key + extraction (#152) must never depend on the stored ``source`` -- capped + by ``text_max_bytes`` and emptied by ``capture_text=False`` (payload-size controls, not extraction controls). Read the real file fresh instead; fall back to ``art.source`` only if it's gone (e.g. a synthetic artifact - in a unit test, or the file vanished mid-run).""" + in a unit test, or the file vanished mid-run). Keep the two in sync if + this logic changes.""" try: return (project_dir / path).read_bytes().decode("utf-8") except (OSError, UnicodeDecodeError): @@ -683,10 +685,13 @@ def analyze(self) -> Analysis: # Config keys (#152): L1 data, layered onto the same artifacts, every # level. Namespace-eligible artifacts only (env-family by basename, - # else by format); a parse failure never drops the artifact -- it - # downgrades `extraction` to "partial" instead (never overwritten on - # success, so it doesn't clobber a "partial" already set above by - # dependency-manifest parsing on the same artifact). + # else by format). `extraction` combines with any prior + # dependency-manifest pass on the same artifact: a parse failure here + # always downgrades to "partial" (never drops the artifact); a clean + # parse upgrades an untouched "none" to "full", but never overwrites + # an existing "partial" (e.g. from a failed dependency-manifest parse + # on the same artifact) -- a successful pass here must not silently + # erase an unrelated failure already recorded on the artifact. for path in sorted(app.artifacts): art = app.artifacts[path] if not is_config_eligible(art): @@ -696,6 +701,8 @@ def analyze(self) -> Analysis: art.config_keys = keys if not ok: art.extraction = "partial" + elif art.extraction == "none": + art.extraction = "full" # L3: intraprocedural dataflow (CFG/CDG/DDG) emitted onto the v2 tree. if self.analysis_level >= 3: diff --git a/test/test_artifact_pipeline.py b/test/test_artifact_pipeline.py index 154641a..4501f36 100644 --- a/test/test_artifact_pipeline.py +++ b/test/test_artifact_pipeline.py @@ -111,3 +111,32 @@ def test_config_key_eligibility_and_partial_on_parse_failure(tmp_path): assert app.artifacts["broken.toml"].config_keys == [] assert app.artifacts["broken.toml"].extraction == "partial" + + +def test_config_key_success_upgrades_none_to_full_on_non_manifest(tmp_path): + """Review fix (HIGH): a clean config-key parse on a non-manifest artifact + (extraction still "none" -- build_dependency_view never touched it) + upgrades extraction to "full", not just left at "none".""" + proj = tmp_path / "proj" + proj.mkdir() + (proj / "values.yaml").write_text("replicas: 3\n") + app = _run(tmp_path, proj, 1) + art = app.artifacts["values.yaml"] + assert art.extraction == "full" + assert art.config_keys and art.config_keys[0].key == "replicas" + + +def test_config_key_success_does_not_clear_existing_partial(tmp_path): + """Review fix (HIGH), other half: a Pipfile with `packages` written as a + TOML array (not a table) breaks the Pipfile-specific dependency parser + (AttributeError on `.items()` -> partial=True) but is still perfectly + valid, flattenable TOML -- config-key extraction on the same text + succeeds. That unrelated success must not clear the dependency parse's + "partial" already recorded on the artifact.""" + proj = tmp_path / "proj" + proj.mkdir() + (proj / "Pipfile").write_text('packages = ["requests"]\n') + app = _run(tmp_path, proj, 1) + art = app.artifacts["Pipfile"] + assert art.extraction == "partial" # from the broken dependency parse + assert art.config_keys and art.config_keys[0].key == "packages.0" # still extracted From b613c4350650081bfda6b5fa89d42d0676ffe2ce Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 12:47:39 -0700 Subject: [PATCH 09/10] test(artifacts): config-key e2e coverage; docs --- CHANGELOG.md | 4 + README.md | 3 +- codeanalyzer/artifacts/config_keys.py | 2 +- codeanalyzer/artifacts/discovery.py | 2 + .../2026-08-28-config-key-family-design.md | 4 +- .../manifests_app/app.properties | 12 ++ .../manifests_app/config/settings.yml | 18 +++ test/test_artifacts_end_to_end.py | 148 ++++++++++++++++++ 8 files changed, 189 insertions(+), 4 deletions(-) create mode 100644 test/fixtures/whole_applications/manifests_app/app.properties create mode 100644 test/fixtures/whole_applications/manifests_app/config/settings.yml diff --git a/CHANGELOG.md b/CHANGELOG.md index d9cb398..fdff66a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The release workflow now also stages `schema.cypher` (the same Neo4j schema contract as runnable, `;`-terminated Cypher DDL -- uniqueness constraints plus indexes) as a GitHub Release asset alongside `schema.json`. +- Configuration keys extracted as first-class `ConfigKey` nodes from six v1 + formats (`.env`, `yaml`, `json`, `toml`, `ini`, `properties`), with + reference recognition and `DEFINES_CONFIG` Neo4j projection; namespace + discriminators per format (#152). ## [1.2.0] - 2026-08-26 diff --git a/README.md b/README.md index c8fd83b..a18e70d 100644 --- a/README.md +++ b/README.md @@ -479,7 +479,8 @@ The application envelope also contains three substrate sections: - **`artifacts`** — discovered non-code files (manifests, configs, Docker files, CI workflows, packaging files, scripts, docs, and legal files) with extraction status (`none`, `partial`, or `full`; default `none`), keyed by relative path; each artifact carries the - `can://artifact//` id namespace. + `can://artifact//` id namespace. Config files carry extracted `config_keys` + (keys, values, namespaces, and references) and `DEFINES_CONFIG` Neo4j edges. - **`dependencies`** — declared packages with kind (`runtime`/`dev`/`optional`/`build`), spec, locked version, and provenance (`prov`): where each binding came from (manifest file, lock file, installed metadata). diff --git a/codeanalyzer/artifacts/config_keys.py b/codeanalyzer/artifacts/config_keys.py index 1139d2c..f6e59c4 100644 --- a/codeanalyzer/artifacts/config_keys.py +++ b/codeanalyzer/artifacts/config_keys.py @@ -224,7 +224,7 @@ def _parse_ini(text: str, lines: List[str]) -> List[_Entry]: _REF_TEMPLATE = re.compile(r'\$\{\{[^}]*\}\}') _REF_BRACED = re.compile(r'\$\{[A-Za-z_][A-Za-z0-9_]*\}') _REF_BARE = re.compile(r'\$[A-Za-z_][A-Za-z0-9_]*') -_REF_PERCENT = re.compile(r'%\([A-Za-z_][A-Za-z0-9_]*\)s') +_REF_PERCENT = re.compile(r'%\([A-Za-z_][A-Za-z0-9_.]*\)s') def _find_references(text: str) -> List[str]: diff --git a/codeanalyzer/artifacts/discovery.py b/codeanalyzer/artifacts/discovery.py index 6f564f6..e6bb965 100644 --- a/codeanalyzer/artifacts/discovery.py +++ b/codeanalyzer/artifacts/discovery.py @@ -33,6 +33,8 @@ ("kind/*.yaml", "yaml", ["service-topology"]), ("Chart.yaml", "yaml", ["service-topology"]), ("values.yaml", "yaml", ["service-topology"]), + ("config/*.yml", "yaml", ["tool-config"]), + ("config/*.yaml", "yaml", ["tool-config"]), ("*.tf", "text", ["iac"]), (".github/workflows/*.yml", "yaml", ["ci"]), (".github/workflows/*.yaml", "yaml", ["ci"]), diff --git a/docs/design/specs/2026-08-28-config-key-family-design.md b/docs/design/specs/2026-08-28-config-key-family-design.md index 9ba1c59..12f2a91 100644 --- a/docs/design/specs/2026-08-28-config-key-family-design.md +++ b/docs/design/specs/2026-08-28-config-key-family-design.md @@ -70,9 +70,9 @@ asset inherits it); conformance fixture grows a config-bearing artifact. test dirs are signal; decision recorded here rather than silently diverging from #152. -## Deferred: config_use +## Next unit in this train: config_use (#162) -`PY_USES_CONFIG` (body node → ConfigKey) needs `PyCallArgument.value` +`config_use` is a standard feature of the artifact layer landing next: `PY_USES_CONFIG` (body node → ConfigKey) needs `PyCallArgument.value` (#152's own boundary). ConfigKey ids minted here are its resolution target. ## Caveats diff --git a/test/fixtures/whole_applications/manifests_app/app.properties b/test/fixtures/whole_applications/manifests_app/app.properties new file mode 100644 index 0000000..b770287 --- /dev/null +++ b/test/fixtures/whole_applications/manifests_app/app.properties @@ -0,0 +1,12 @@ +# Application properties +app.name=MyApplication +app.version=1.0.0 +# Multi-line property with continuation +app.long.description=This is a very long property value that \ + continues on the next line with additional configuration details +server.port=9000 +server.ssl=true +# Property with interpolation reference +db.url=%(db.host)s:%(db.port)s +db.host=localhost +db.port=5432 diff --git a/test/fixtures/whole_applications/manifests_app/config/settings.yml b/test/fixtures/whole_applications/manifests_app/config/settings.yml new file mode 100644 index 0000000..2d2ed9a --- /dev/null +++ b/test/fixtures/whole_applications/manifests_app/config/settings.yml @@ -0,0 +1,18 @@ +server: + host: localhost + port: 8080 + timeouts: + connect: 30 + read: 60 +database: + primary: + url: postgresql://db.example.com/prod + pool_size: 10 + replicas: + - url: postgresql://replica1.example.com/prod + - url: postgresql://replica2.example.com/prod +logging: + level: DEBUG + handlers: + - console + - file diff --git a/test/test_artifacts_end_to_end.py b/test/test_artifacts_end_to_end.py index de37909..73b0ed3 100644 --- a/test/test_artifacts_end_to_end.py +++ b/test/test_artifacts_end_to_end.py @@ -23,6 +23,7 @@ def test_full_surface(tmp_path): "pyproject.toml", "requirements-dev.txt", "setup.py", "uv.lock", "environment.yml", "Dockerfile", "docker-compose.yml", ".github/workflows/ci.yml", "data.csv", "logo.png", + ".env", "config/settings.yml", "app.properties", } assert arts["setup.py"].extraction == "partial" # computed install_requires assert arts["pyproject.toml"].extraction == "full" @@ -71,3 +72,150 @@ def test_level_invariance_artifacts_and_deps(tmp_path): unres_l1 = sorted([u.module for u in app_l1.unresolved_imports]) unres_l4 = sorted([u.module for u in app_l4.unresolved_imports]) assert unres_l1 == unres_l4 + + +def test_config_keys_extraction(tmp_path): + """Verify config key extraction from .env, settings.yml, and app.properties.""" + app = _app(tmp_path, "config_test") + arts = app.artifacts + + # .env: env namespace, exact keys + env_keys = {k.key for k in arts[".env"].config_keys} + assert env_keys == {"DEBUG", "DATABASE_URL", "API_KEY", "SECRET_API_TOKEN", "FLASK_ENV"} + + # All .env keys should be in env namespace + for key in arts[".env"].config_keys: + assert key.namespace == "env" + + # settings.yml: yaml namespace, nested + array keys with numeric indices + yaml_keys = {k.key for k in arts["config/settings.yml"].config_keys} + assert yaml_keys == { + "server.host", "server.port", "server.timeouts.connect", "server.timeouts.read", + "database.primary.url", "database.primary.pool_size", + "database.replicas.0.url", "database.replicas.1.url", + "logging.level", "logging.handlers.0", "logging.handlers.1", + } + + # All yaml keys should be in yaml namespace + for key in arts["config/settings.yml"].config_keys: + assert key.namespace == "yaml" + + # app.properties: properties namespace + props_keys = {k.key for k in arts["app.properties"].config_keys} + assert props_keys == { + "app.name", "app.version", "app.long.description", + "server.port", "server.ssl", "db.url", "db.host", "db.port", + } + + # All properties keys should be in properties namespace + for key in arts["app.properties"].config_keys: + assert key.namespace == "properties" + + +def test_config_keys_references(tmp_path): + """Verify reference extraction from config values.""" + app = _app(tmp_path, "refs_test") + + # .env: should find ${DATABASE_PASSWORD} reference + env_refs = [] + for key in app.artifacts[".env"].config_keys: + env_refs.extend(key.references) + assert "${DATABASE_PASSWORD}" in env_refs + + # app.properties: should find %(db.host)s and %(db.port)s references + props_refs = set() + for key in app.artifacts["app.properties"].config_keys: + props_refs.update(key.references) + assert "%(db.host)s" in props_refs + assert "%(db.port)s" in props_refs + + # settings.yml: no references expected + yaml_refs = [] + for key in app.artifacts["config/settings.yml"].config_keys: + yaml_refs.extend(key.references) + assert len(yaml_refs) == 0 + + +def test_config_keys_values(tmp_path): + """Verify value extraction when artifact_text=True.""" + app = _app(tmp_path, "values_test") + + # With artifact_text=True (default), values should be populated + env_api_key = next((k for k in app.artifacts[".env"].config_keys if k.key == "API_KEY"), None) + assert env_api_key is not None + assert env_api_key.value == "${DATABASE_PASSWORD}" + + secret_token = next((k for k in app.artifacts[".env"].config_keys if k.key == "SECRET_API_TOKEN"), None) + assert secret_token is not None + assert secret_token.value == "sk-12345abcdef-super-secret" + + +def test_config_keys_value_gating(tmp_path): + """Verify value gating when artifact_text=False.""" + # Run with artifact_text=False + app = Codeanalyzer(AnalysisOptions( + input=FIXTURE, analysis_level=1, no_venv=True, cache_dir=tmp_path / "no_text", + artifact_text=False, + )).analyze().application + + # Keys, namespaces, references should still be present + env_keys = {k.key for k in app.artifacts[".env"].config_keys} + assert env_keys == {"DEBUG", "DATABASE_URL", "API_KEY", "SECRET_API_TOKEN", "FLASK_ENV"} + + # But values should all be None + for key in app.artifacts[".env"].config_keys: + assert key.value is None + + # References should still be extracted + env_refs = [] + for key in app.artifacts[".env"].config_keys: + env_refs.extend(key.references) + assert "${DATABASE_PASSWORD}" in env_refs + + +def test_config_keys_spans(tmp_path): + """Verify that spans slice source text correctly to the values.""" + app = _app(tmp_path, "spans_test") + env_source = app.artifacts[".env"].source + + # DATABASE_URL has value "postgresql://localhost/mydb # connection string" + db_url_key = next((k for k in app.artifacts[".env"].config_keys if k.key == "DATABASE_URL"), None) + assert db_url_key is not None + assert db_url_key.span is not None + + # The line containing DATABASE_URL should include the value + line_num = db_url_key.span.start[0] + end_line_num = db_url_key.span.end[0] + assert line_num == end_line_num # span on same line + + # Extract the line and verify it contains both key and value + lines = env_source.split('\n') + line_text = lines[line_num - 1] # line numbers are 1-indexed + assert "DATABASE_URL" in line_text + assert "postgresql://localhost/mydb" in line_text + + +def test_config_keys_extraction_level(tmp_path): + """Verify config_keys extracted at L1 and identical at L4.""" + app_l1 = Codeanalyzer(AnalysisOptions( + input=FIXTURE, analysis_level=1, no_venv=True, cache_dir=tmp_path / "config_l1", + )).analyze().application + app_l4 = Codeanalyzer(AnalysisOptions( + input=FIXTURE, analysis_level=4, no_venv=True, cache_dir=tmp_path / "config_l4", + )).analyze().application + + # Config keys should be identical at both levels + for art_name in [".env", "config/settings.yml", "app.properties"]: + l1_keys = sorted([k.key for k in app_l1.artifacts[art_name].config_keys]) + l4_keys = sorted([k.key for k in app_l4.artifacts[art_name].config_keys]) + assert l1_keys == l4_keys + + +def test_config_keys_extraction_full(tmp_path): + """Verify extraction='full' on the three config files.""" + app = _app(tmp_path, "extraction_test") + + # All three new config files should have extraction='full' + assert app.artifacts[".env"].extraction == "full" + assert app.artifacts["config/settings.yml"].extraction == "full" + assert app.artifacts["app.properties"].extraction == "full" From ee1e3faff185fc54b9c7738c6b01f9f0b7c34770 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Fri, 28 Aug 2026 12:56:14 -0700 Subject: [PATCH 10/10] fix(test): track fixture .env past ignore; spec caveats --- .gitignore | 1 + docs/design/specs/2026-08-28-config-key-family-design.md | 7 +++++++ test/fixtures/whole_applications/manifests_app/.env | 6 ++++++ 3 files changed, 14 insertions(+) create mode 100644 test/fixtures/whole_applications/manifests_app/.env diff --git a/.gitignore b/.gitignore index 3a62449..4148ce5 100644 --- a/.gitignore +++ b/.gitignore @@ -198,3 +198,4 @@ node_modules/ # Track fixture lock files past the repo-root uv.lock ignore !test/fixtures/**/uv.lock +!test/fixtures/**/.env diff --git a/docs/design/specs/2026-08-28-config-key-family-design.md b/docs/design/specs/2026-08-28-config-key-family-design.md index 12f2a91..c0612a6 100644 --- a/docs/design/specs/2026-08-28-config-key-family-design.md +++ b/docs/design/specs/2026-08-28-config-key-family-design.md @@ -87,6 +87,13 @@ asset inherits it); conformance fixture grows a config-bearing artifact. a dotted-path id (a top-level key literally named `"a.b"` and a nested `a: {b: ...}` both flatten to `a.b`); colliding forms coalesce last-wins by design, same as a plain duplicate key within one format. +- Strict configparser: a duplicate option within a single ini section causes + configparser to raise ConfigParserError, downgrading the entire file to + zero keys and extraction `"partial"` (unlike env/properties which use + last-wins semantics). +- Empty collections (yaml/json/toml arrays or objects with no elements) yield + no ConfigKey entries — a `logging.handlers: []` contributes no keys to the + flattened result. ## Definition of done diff --git a/test/fixtures/whole_applications/manifests_app/.env b/test/fixtures/whole_applications/manifests_app/.env new file mode 100644 index 0000000..2f4f465 --- /dev/null +++ b/test/fixtures/whole_applications/manifests_app/.env @@ -0,0 +1,6 @@ +# Application environment configuration +DEBUG=true +DATABASE_URL="postgresql://localhost/mydb # connection string" +API_KEY="${DATABASE_PASSWORD}" +SECRET_API_TOKEN="sk-12345abcdef-super-secret" +export FLASK_ENV=production