From b3a3029895aac823c29633eda2ce49dfc4cd005e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 24 Jun 2026 15:26:33 +0000 Subject: [PATCH 1/3] =?UTF-8?q?Add=20audit/=20=E2=80=94=20static-layer=20o?= =?UTF-8?q?rchestrator=20+=20aggregation=20pipeline=20(Plan.md=20Phase=200?= =?UTF-8?q?=E2=80=931)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First deliverable of the Own.NET Audit plan: the build-free static tier plus the full normalize -> score -> report aggregation, all testable on Linux CI with no target build and no external tools. Aggregation core (audit/aggregate/), each with embedded-fixture --selftest in the oracle_compare style: - normalize.py: reads every tool's SARIF through the SAME parse_sarif as oracle_compare (reused, not duplicated), maps (tool, ruleId) -> Plan.md §2 category via static/taxonomy/categories.yml. Splits the umbrella OWN001 code by its [resource: ...] tag so subscription/timer leaks land in cat. 2/3 instead of collapsing into IDisposable (cat. 1); labels OWN014 as region-escape. DevExpress findings are baseline-suppressed — counted in coverage, never hidden. Unmapped rules surface as pending taxonomy, not dropped. - score.py: generalizes oracle_compare.compare() into cross-tool agreement (>=2 tools at basename+line-window -> high confidence), category-driven P0-P3 severity, and a per-module pain heatmap answering "where does it hurt most". - report.py: markdown health report + machine JSON, with a load-bearing coverage section (NO-TOOL categories, suppressed count, unmapped rules). Static layer (audit/static/): - run_static.py orchestrator (build-free runners -> aggregate -> report), with a full end-to-end --selftest on embedded SARIF fixtures. - tools/owncheck.py (own-check.sh --format sarif, build-free; graceful when dotnet is absent), codeql.sh (build-mode=none, security-and-quality suite), and the build-required Windows runners roslyn_pack.ps1 / infersharp.sh (real skeletons, NO-TOOL exit when the tool is absent). - inject/ analyzer props+targets under MSBuild's recognized names, gated on /p:OwnAudit=true; taxonomy/categories.yml; config/profiles/desktop-wpf.yml. Decoupling: imports nothing from the ownlang core; the only in-repo seam is scripts/oracle_compare.parse_sarif (vendored on lift-out), own-check via CLI only. PyYAML is scoped to audit/ (requirements.txt) so the core suite stays zero-dep; a new CI job runs the four selftests (48 checks). The target itself is audited on a local Windows machine, never in CI. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE --- .github/workflows/ci.yml | 21 ++ audit/README.md | 118 ++++++ audit/aggregate/normalize.py | 347 ++++++++++++++++++ audit/aggregate/report.py | 233 ++++++++++++ audit/aggregate/score.py | 261 +++++++++++++ audit/config/profiles/desktop-wpf.yml | 51 +++ audit/requirements.txt | 6 + .../inject/OwnAudit.Directory.Build.props | 39 ++ .../inject/OwnAudit.Directory.Build.targets | 18 + audit/static/run_static.py | 240 ++++++++++++ audit/static/taxonomy/categories.yml | 77 ++++ audit/static/tools/codeql.sh | 53 +++ audit/static/tools/infersharp.sh | 46 +++ audit/static/tools/owncheck.py | 87 +++++ audit/static/tools/roslyn_pack.ps1 | 56 +++ 15 files changed, 1653 insertions(+) create mode 100644 audit/README.md create mode 100755 audit/aggregate/normalize.py create mode 100755 audit/aggregate/report.py create mode 100755 audit/aggregate/score.py create mode 100644 audit/config/profiles/desktop-wpf.yml create mode 100644 audit/requirements.txt create mode 100644 audit/static/inject/OwnAudit.Directory.Build.props create mode 100644 audit/static/inject/OwnAudit.Directory.Build.targets create mode 100755 audit/static/run_static.py create mode 100644 audit/static/taxonomy/categories.yml create mode 100755 audit/static/tools/codeql.sh create mode 100755 audit/static/tools/infersharp.sh create mode 100755 audit/static/tools/owncheck.py create mode 100644 audit/static/tools/roslyn_pack.ps1 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cadee87b..2e92dc09 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -42,6 +42,27 @@ jobs: python scripts/metamorphic_facts.py --selftest python scripts/benchmark.py --selftest + # Own.NET Audit (audit/) — the aggregation layer's selftests, the only thing the + # Linux CI gates for the audit (the target itself is analyzed on a local Windows + # machine, never in CI; see audit/README.md and Plan.md §3.2). PyYAML is scoped + # to audit/ here so the core test suite stays zero-dependency. + audit-selftests: + name: audit aggregation selftests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install audit deps (PyYAML, audit-scoped) + run: pip install -r audit/requirements.txt + - name: Own.NET Audit selftests (normalize + score + report + orchestrator) + run: | + python audit/aggregate/normalize.py --selftest + python audit/aggregate/score.py --selftest + python audit/aggregate/report.py --selftest + python audit/static/run_static.py --selftest + tests: name: tests (py${{ matrix.python-version }}) runs-on: ubuntu-latest diff --git a/audit/README.md b/audit/README.md new file mode 100644 index 00000000..cd8d0640 --- /dev/null +++ b/audit/README.md @@ -0,0 +1,118 @@ +# Own.NET Audit + +An audit **orchestrator** for a legacy .NET Framework 4.7.2 / WPF / DevExpress +desktop application. It runs a fleet of mature, ready-made analyzers over the +target, normalizes every tool's output to SARIF, scores findings by cross-tool +agreement, and produces a categorized **health report ranked by where it hurts +most** — the "anamnesis" of the codebase. + +Full design: [`../Plan.md`](../Plan.md). This subtree is the first deliverable — +the **static layer** (build-free tier) plus the **aggregation pipeline**. The +runtime layer and the AI-reviewer layer are later phases. + +## Principles (why this is an orchestrator, not a new analyzer) + +- **No new heuristics.** We run existing tools; we do not invent regex detectors. + A category with no reliable tool is marked `NO-TOOL` and deferred to the runtime + layer — never faked. (Mirrors `own-check`'s honest-skip discipline.) +- **SARIF is the one normalized format.** Every tool's output is read through the + *same* `parse_sarif` the oracle uses, then mapped to a category. +- **Honest coverage.** Suppressed third-party (DevExpress) findings are counted, + not hidden. Unmapped rules are surfaced as pending taxonomy, not dropped. + Tiers that did not run are labelled, not silently treated as "clean". +- **Determinism.** A run over a fixed commit is a stable, diffable artifact. + +## Decoupling + +This subtree lifts out as a standalone project (Plan.md §7). It imports **nothing** +from the `ownlang` core. Its only in-repo seams are: + +- `scripts/oracle_compare.parse_sarif` / `norm_path` — a pure SARIF reader, reused + (not duplicated) per Plan.md §3.4. Vendored on lift-out (Phase 4). +- `own-check` is consumed **only** via its CLI (`scripts/own-check.sh`). + +The single third-party Python dependency is PyYAML (see `requirements.txt`), +scoped to this subtree so the zero-dependency core test suite stays untouched. + +## Layout + +```text +audit/ + aggregate/ + normalize.py # SARIF -> categorized findings; OWN001 [resource:] split; DevExpress suppress + score.py # cross-tool agreement + severity + "where it hurts most" heatmap + report.py # markdown + json renderers (health report) + static/ + run_static.py # orchestrator: run build-free runners -> aggregate -> report + tools/ + owncheck.py # build-free runner: own-check.sh --format sarif (needs dotnet) + codeql.sh # build-free runner: CodeQL build-mode=none, security-and-quality + roslyn_pack.ps1 # build-required runner (local Windows): NetAnalyzers/Roslynator/... + infersharp.sh # build-required runner: Infer# over built binaries + inject/ # OwnAudit.Directory.Build.props/.targets (analyzer injection, gated) + taxonomy/ + categories.yml # rule-id -> category knowledge base (Plan.md §2/§3.4) + config/profiles/ + desktop-wpf.yml # which packs / severity floor for the net472 WPF target + requirements.txt # PyYAML (audit-scoped) +``` + +## Tiers (Plan.md §3.2) + +| Tier | Tools | Needs a successful build of the target? | +|---|---|---| +| **build-free** | own-check, CodeQL (`build-mode: none`) | no — works on a solution that does not compile | +| **build-required** | Roslyn analyzer packs, Infer# | yes | + +The entire audit of the target runs on a **local Windows machine** (VS Build Tools ++ DevExpress 12.2). There is no CI run of the target — Own.NET's Linux CI only +gates the Python aggregation selftests (this subtree), exactly as it gates +`oracle_compare --selftest` today. + +## Running it + +```bash +# Build-free tier + report (own-check needs a .NET SDK on PATH; codeql if installed): +python audit/static/run_static.py \ + --target /path/to/legacy/src \ + --profile desktop-wpf \ + --target-name acme/LegacyApp --commit "$(git -C /path/to/legacy rev-parse HEAD)" \ + --out artifacts/own-audit +# -> artifacts/own-audit/report.md and report.json + +# Build-required tier runs on the Windows machine; drop its SARIF into the same +# --out directory and re-run run_static.py to fold it into the report: +pwsh audit/static/tools/roslyn_pack.ps1 -Solution ..\target-audit\Target.sln \ + -AnalyzerCache .\cache -Out artifacts\own-audit +``` + +The aggregation modules also run standalone: + +```bash +python audit/aggregate/normalize.py --sarif own-check=own.sarif --sarif codeql=cq.sarif \ + --json findings.json +python audit/aggregate/report.py --findings findings.json --format markdown +``` + +## Selftests + +Every aggregation module carries embedded-fixture selftests (the +`oracle_compare --selftest` discipline). They need no external tools and gate on +Linux CI: + +```bash +python audit/aggregate/normalize.py --selftest +python audit/aggregate/score.py --selftest +python audit/aggregate/report.py --selftest +python audit/static/run_static.py --selftest # full pipeline end-to-end on fixtures +``` + +## Status + +- **Done (this slice):** static build-free runners, normalization + taxonomy + (incl. the OWN001 `[resource:]` split and OWN014 region-escape labelling), + DevExpress baseline-suppress, cross-tool agreement scoring, the pain heatmap, + markdown + json reports, the analyzer-injection props/targets, and selftests. +- **Deferred:** HTML + merged-SARIF renderers (more views over the same model); + the runtime layer (FlaUI + ClrMD leak-harness, duplicate-immutable detector); + the AI-reviewer layer; feeding confirmed findings back into the OwnLang corpus. diff --git a/audit/aggregate/normalize.py b/audit/aggregate/normalize.py new file mode 100755 index 00000000..004f48e8 --- /dev/null +++ b/audit/aggregate/normalize.py @@ -0,0 +1,347 @@ +#!/usr/bin/env python3 +""" +Own.NET Audit — normalization: SARIF -> categorized findings (Plan.md §3.4). + +Reads every tool's SARIF through the *same* ``parse_sarif`` as ``oracle_compare.py`` +(reused, not duplicated — that parser is the one that stopped the silent 38-line +drop on the ScreenToGif oracle run), then maps each ``(tool, ruleId)`` to a +problem category from Plan.md §2 using the knowledge base in +``static/taxonomy/categories.yml``. + +Three honesty rules, mirroring the rest of the repo: + + * Unmapped rules are **not dropped** — they land in ``uncategorized`` and are + surfaced in coverage, so the taxonomy grows deliberately. + * ``OWN001`` is an umbrella leak code; it is split by its ``[resource: ...]`` + tag so subscription/timer leaks (cat. 2/3) are not collapsed into the + IDisposable bucket (cat. 1). + * DevExpress third-party findings are baseline-suppressed — dropped from the + main report but **counted** in coverage, never hidden silently. + +This module's only in-repo seam is ``scripts/oracle_compare.parse_sarif``; own-check +is consumed solely via its CLI. Neither couples ``audit/`` to the ``ownlang`` core. + +Usage: + normalize.py --sarif own-check=own.sarif --sarif codeql=codeql.sarif \\ + --taxonomy static/taxonomy/categories.yml [--strip PREFIX] [--json out.json] + normalize.py --selftest +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from collections import Counter +from dataclasses import dataclass, field +from fnmatch import fnmatchcase +from pathlib import Path +from typing import Any + +RESOURCE_RE = re.compile(r"\[resource:\s*([^\]]+)\]", re.IGNORECASE) + + +def _import_oracle() -> tuple[Any, Any]: + """Reuse the oracle's SARIF reader (Plan.md §3.4: reuse, don't duplicate). + + Only this one symbol is borrowed from the in-repo scripts/; it is a pure + SARIF->Finding reader. On lift-out (Plan.md Phase 4) it gets vendored.""" + repo = Path(__file__).resolve().parents[2] + sys.path.insert(0, str(repo / "scripts")) + try: + from oracle_compare import norm_path, parse_sarif + except ImportError as exc: # pragma: no cover - environment guard + raise SystemExit( + "normalize: cannot import parse_sarif from scripts/oracle_compare.py " + f"(expected at {repo / 'scripts'}): {exc}" + ) from exc + return parse_sarif, norm_path + + +parse_sarif, norm_path = _import_oracle() + + +@dataclass +class AuditFinding: + """One analyzer result, categorized. ``fkey`` is the basename (lower-cased) for + cross-tool matching — robust to path-prefix differences between tools, exactly + like oracle_compare.Finding.""" + + tool: str + path: str + line: int + rule: str + message: str + category: int = 0 + category_name: str = "uncategorized" + resource: str | None = None + suppressed: bool = False + suppress_reason: str = "" + fkey: str = field(init=False, default="") + + def __post_init__(self) -> None: + self.fkey = self.path.lower().rsplit("/", 1)[-1] + + @property + def module(self) -> str: + """Directory of the finding (the heatmap roll-up unit), or ``(root)``.""" + return self.path.rsplit("/", 1)[0] if "/" in self.path else "(root)" + + +@dataclass +class Taxonomy: + rules: dict[str, Any] + category_severity: dict[int, str] + suppress_tokens: list[str] + + def severity_for(self, category: int) -> str: + return self.category_severity.get(category, "P3") + + +def load_taxonomy(path: str | Path) -> Taxonomy: + import yaml # scoped dep — see audit/requirements.txt + + data = yaml.safe_load(Path(path).read_text(encoding="utf-8")) or {} + sev = {int(k): str(v) for k, v in (data.get("category_severity") or {}).items()} + suppress = list((data.get("suppress") or {}).get("path_or_message_contains") or []) + return Taxonomy(rules=dict(data.get("rules") or {}), + category_severity=sev, suppress_tokens=suppress) + + +def _resource_tag(message: str) -> str | None: + m = RESOURCE_RE.search(message or "") + return m.group(1).strip().lower() if m else None + + +def _match_rule(rule: str, rules: dict[str, Any]) -> dict[str, Any] | None: + """Resolve a ruleId to its taxonomy spec. Exact match wins; otherwise the most + specific glob (longest non-wildcard run) wins.""" + if rule in rules: + return rules[rule] + best: tuple[int, dict[str, Any]] | None = None + for pat, spec in rules.items(): + if any(c in pat for c in "*?[") and fnmatchcase(rule, pat): + specificity = len(pat.replace("*", "").replace("?", "")) + if best is None or specificity > best[0]: + best = (specificity, spec) + return best[1] if best else None + + +def categorize(rule: str, resource: str | None, + rules: dict[str, Any]) -> tuple[int, str]: + """(category, name) for a ruleId, splitting umbrella codes by resource tag.""" + spec = _match_rule(rule, rules) + if spec is None: + return 0, "uncategorized" + if "by_resource" in spec: + by = spec["by_resource"] + chosen = by.get(resource) if resource is not None else None + if chosen is None: + chosen = by.get("*", {}) + return int(chosen.get("category", 0)), str(chosen.get("name", "uncategorized")) + return int(spec.get("category", 0)), str(spec.get("name", "uncategorized")) + + +def _suppressed(path: str, message: str, tokens: list[str]) -> str: + """Reason string if this finding is third-party baseline-suppressed, else ''.""" + hay = f"{path}\n{message}".lower() + for tok in tokens: + if tok.lower() in hay: + return f"third-party: {tok}" + return "" + + +def normalize_results(raw: list[Any], tax: Taxonomy) -> list[AuditFinding]: + """Turn oracle_compare Findings (tool/path/line/rule/message) into categorized + AuditFindings. We ignore the oracle's leak/other ``cls`` and apply our own + richer taxonomy instead.""" + out: list[AuditFinding] = [] + for f in raw: + resource = _resource_tag(f.message) + category, name = categorize(f.rule, resource, tax.rules) + reason = _suppressed(f.path, f.message, tax.suppress_tokens) + out.append(AuditFinding( + tool=f.tool, path=f.path, line=f.line, rule=f.rule, message=f.message, + category=category, category_name=name, resource=resource, + suppressed=bool(reason), suppress_reason=reason)) + return out + + +def coverage(findings: list[AuditFinding]) -> dict[str, Any]: + """The honesty ledger: what was categorized, what was suppressed, and which + rules have no taxonomy entry yet (so they are visibly pending, not lost).""" + kept = [f for f in findings if not f.suppressed] + suppressed = [f for f in findings if f.suppressed] + uncategorized = Counter(f.rule for f in kept if f.category == 0) + return { + "tools": sorted({f.tool for f in findings}), + "total": len(findings), + "kept": len(kept), + "suppressed": len(suppressed), + "suppressed_by": dict(Counter(f.suppress_reason for f in suppressed)), + "by_category": dict(Counter(f.category for f in kept)), + "uncategorized_rules": dict(uncategorized), + } + + +def normalize(sarif_inputs: list[tuple[str, str]], tax: Taxonomy, + strips: list[str]) -> tuple[list[AuditFinding], dict[str, Any]]: + raw: list[Any] = [] + for tool, path in sarif_inputs: + raw += parse_sarif(Path(path).read_text(encoding="utf-8"), tool, strips) + findings = normalize_results(raw, tax) + return findings, coverage(findings) + + +def finding_to_dict(f: AuditFinding) -> dict[str, Any]: + return { + "tool": f.tool, "path": f.path, "line": f.line, "rule": f.rule, + "category": f.category, "category_name": f.category_name, + "resource": f.resource, "suppressed": f.suppressed, + "suppress_reason": f.suppress_reason, "message": f.message, + } + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Normalize SARIF into categorized findings.") + ap.add_argument("--sarif", action="append", default=[], metavar="TOOL=PATH", + help="a tool's SARIF as tool=path (repeatable)") + ap.add_argument("--taxonomy", default=str( + Path(__file__).resolve().parents[1] / "static" / "taxonomy" / "categories.yml"), + help="categories.yml (default: the shipped taxonomy)") + ap.add_argument("--strip", action="append", default=[], metavar="PREFIX", + help="path prefix to strip from finding paths (repeatable)") + ap.add_argument("--json", dest="json_out", default="", + help="write normalized findings + coverage as JSON") + ap.add_argument("--selftest", action="store_true", + help="run built-in checks and exit") + args = ap.parse_args(argv) + + if args.selftest: + return _selftest() + + inputs: list[tuple[str, str]] = [] + for spec in args.sarif: + tool, _, path = spec.partition("=") + if not path: + ap.error(f"--sarif expects tool=path, got {spec!r}") + inputs.append((tool, path)) + + tax = load_taxonomy(args.taxonomy) + findings, cov = normalize(inputs, tax, args.strip) + payload = {"coverage": cov, + "findings": [finding_to_dict(f) for f in findings if not f.suppressed]} + if args.json_out: + Path(args.json_out).write_text(json.dumps(payload, indent=2), encoding="utf-8") + print(json.dumps(cov, indent=2)) + return 0 + + +# --------------------------------------------------------------------------- # +# Selftest — embedded fixtures, in the style of oracle_compare._selftest. # +# --------------------------------------------------------------------------- # + +def _own_sarif() -> str: + """own-check-style SARIF exercising the OWN001 [resource:] split + OWN014.""" + def res(rule: str, msg: str, uri: str, line: int) -> dict[str, Any]: + return {"ruleId": rule, "level": "warning", "message": {"text": msg}, + "locations": [{"physicalLocation": { + "artifactLocation": {"uri": uri}, "region": {"startLine": line}}}]} + return json.dumps({"version": "2.1.0", "runs": [{"tool": {"driver": {"name": "Own.NET"}}, + "results": [ + res("OWN001", "event subscribed but never unsubscribed [resource: subscription token]", + "src/Vm/CustomerViewModel.cs", 12), + res("OWN001", "timer never stopped [resource: timer]", "src/Vm/TimerViewModel.cs", 30), + res("OWN001", "field never disposed [resource: disposable field]", + "src/Vm/ReportViewModel.cs", 7), + res("OWN001", "local IDisposable never disposed", "src/Util/Io.cs", 9), + res("OWN014", "region escape: view-model promoted to App lifetime", + "src/Vm/StaticEventEscapeViewModel.cs", 50), + ]}]}) + + +def _codeql_sarif() -> str: + def res(rule: str, uri: str, line: int) -> dict[str, Any]: + return {"ruleId": rule, "message": {"text": "not disposed"}, + "locations": [{"physicalLocation": { + "artifactLocation": {"uri": uri}, "region": {"startLine": line}}}]} + return json.dumps({"runs": [{"results": [ + res("cs/local-not-disposed", "src/Util/Io.cs", 9), # agrees with own local leak + res("cs/empty-block", "DevExpress.Xpf/Grid/Helper.cs", 4), # third-party -> suppressed + res("FOO999", "src/Util/Misc.cs", 3), # unmapped -> uncategorized + ]}]}) + + +def _selftest() -> int: + tax_path = Path(__file__).resolve().parents[1] / "static" / "taxonomy" / "categories.yml" + fails: list[str] = [] + + # The shipped taxonomy must parse and carry the split that the OWN001 fix needs. + tax = load_taxonomy(tax_path) + if "OWN001" not in tax.rules or "by_resource" not in tax.rules["OWN001"]: + fails.append("shipped categories.yml lost the OWN001 by_resource split") + if tax.rules.get("OWN014", {}).get("name") != "region-escape": + fails.append("shipped categories.yml: OWN014 must be region-escape, not subscription-leak") + + own = normalize_results(parse_sarif(_own_sarif(), "own-check", []), tax) + by_file = {f.fkey: f for f in own} + + # OWN001 umbrella split by [resource: ...] tag. + cases = { + "customerviewmodel.cs": (2, "subscription-leak"), + "timerviewmodel.cs": (3, "timer-leak"), + "reportviewmodel.cs": (1, "idisposable-leak"), + "io.cs": (1, "idisposable-leak"), # no resource tag -> "*" fallback + "staticeventescapeviewmodel.cs": (2, "region-escape"), + } + for fkey, (cat, name) in cases.items(): + got = by_file.get(fkey) + if got is None: + fails.append(f"missing finding for {fkey}") + elif (got.category, got.category_name) != (cat, name): + fails.append( + f"{fkey}: expected ({cat},{name}), got ({got.category},{got.category_name})") + + # Glob mapping + uncategorized + DevExpress suppression on the CodeQL run. + cq = normalize_results(parse_sarif(_codeql_sarif(), "codeql", []), tax) + cq_by = {f.fkey: f for f in cq} + if cq_by["io.cs"].category != 1: + fails.append("cs/local-not-disposed should map to category 1") + if not cq_by["helper.cs"].suppressed: + fails.append("DevExpress finding must be baseline-suppressed") + if cq_by["misc.cs"].category != 0: + fails.append("unmapped FOO999 must be uncategorized (category 0)") + + cov = coverage(own + cq) + if cov["suppressed"] != 1: + fails.append(f"coverage suppressed count wrong: {cov['suppressed']}") + if "FOO999" not in cov["uncategorized_rules"]: + fails.append("uncategorized rule FOO999 must be surfaced in coverage") + if cov["uncategorized_rules"].get("FOO999") != 1: + fails.append("uncategorized count wrong") + # a suppressed finding must not leak into the kept category tally + if cov["by_category"].get(0, 0) != 1: # only FOO999 (the DevExpress one is suppressed) + fails.append(f"suppressed finding leaked into kept categories: {cov['by_category']}") + + # severity baseline comes from the category, not the tool level + if tax.severity_for(1) != "P1" or tax.severity_for(0) != "P3": + fails.append("category severity baseline wrong") + + # glob specificity: CA2000 (exact, cat 1) must beat CA2* (glob, cat 14) + cat, _ = categorize("CA2000", None, tax.rules) + if cat != 1: + fails.append(f"exact CA2000 should win over CA2* glob: got {cat}") + cat, _ = categorize("CA1822", None, tax.rules) + if cat != 14: + fails.append(f"CA1* glob should map to general-quality (14): got {cat}") + + total = 18 + for f in fails: + print(f"NORMALIZE SELFTEST FAIL: {f}") + print(f"normalize selftest: {total - len(fails)}/{total} checks passed") + return 1 if fails else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/audit/aggregate/report.py b/audit/aggregate/report.py new file mode 100755 index 00000000..ac0bca4c --- /dev/null +++ b/audit/aggregate/report.py @@ -0,0 +1,233 @@ +#!/usr/bin/env python3 +""" +Own.NET Audit — renderers (Plan.md §3.5). Turns the scored findings + coverage +ledger into the "health anamnesis": a categorized report ranked by where it hurts +most, with an honest coverage map. + + * **Markdown** — for humans / a GitHub run summary (as oracle/mine do today). + * **JSON** — machine-readable, for the downstream AI layer and regression diffs. + +HTML and merged-SARIF renderers are deferred (Plan.md §3.5) — they are additional +views over the same scored model, not new analysis. + +The coverage section is load-bearing: it states which tiers ran, which categories +are NO-TOOL / deferred-to-runtime, how many DevExpress findings were suppressed, +and which rules have no taxonomy entry yet. A clean report that hid its own gaps +would be worse than useless. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from normalize import AuditFinding, coverage, load_taxonomy, normalize_results +from score import score +from score import to_json as score_to_json + +# Human labels for the Plan.md §2 categories, used in the coverage section. +CATEGORY_LABELS = { + 1: "IDisposable leak", 2: "event/subscription leak & region escape", + 3: "timer leak", 4: "DependencyPropertyDescriptor.AddValueChanged leak", + 5: "INPC correctness", 6: "PropertyChanged storms", 7: "WPF binding errors", + 8: "broken virtualization", 9: "Freezable / per-instance brushes", + 10: "allocations in converters/getters", 11: "duplicated immutable data", + 12: "heavy reference data / LOH / Gen2", 13: "cross-thread / ObjectDisposedException", + 14: "general bugs / perf / best-practice", 15: "architecture metrics", +} + + +def render_markdown(meta: dict[str, Any], cov: dict[str, Any], + scored: dict[str, Any], max_list: int = 40) -> str: + clusters = scored["clusters"] + high = [c for c in clusters if c.confidence == "high"] + candidates = [c for c in clusters if c.confidence == "candidate"] + out: list[str] = [ + f"# Own.NET Audit — health report — `{meta.get('target', '?')}`", + "", + f"- commit: `{meta.get('commit', '?')}`", + f"- generated: {meta.get('generated', '?')}", + f"- profile: `{meta.get('profile', '?')}`", + f"- tools run: {', '.join(cov.get('tools') or []) or '(none)'}", + f"- tiers: {meta.get('tiers', '?')}", + f"- match: basename + line within ±{meta.get('line_tol', 3)}", + "", + f"**{scored['totals']['clusters']} findings** " + f"({scored['totals']['high_confidence']} high-confidence, " + f"{scored['totals']['candidates']} candidate). " + "High-confidence = flagged by ≥2 independent tools at the same spot.", + "", + "## Where it hurts most", + "", + "Modules ranked by pain index (severity weighted by cross-tool agreement, " + "summed). " + "This is the triage order — top is worst, bottom is almost fine.", + "", + "| module | pain | findings | high-conf | top category |", + "|---|---:|---:|---:|---|", + ] + for row in scored["heatmap"][:max_list]: + out.append(f"| `{row['module']}` | {row['pain']} | {row['findings']} | " + f"{row['high_confidence']} | {row['top_category']} |") + if not scored["heatmap"]: + out.append("| _(no findings)_ | | | | |") + + out += ["", f"## High-confidence findings — {len(high)} (≥2 tools agree)", ""] + out += ["_(none)_"] if not high else [ + f"- `{c.path}:{c.line}` **[{c.severity} · {c.category_name}]** " + f"— {', '.join(c.tools)}" for c in high[:max_list] + ] + + out += ["", f"## Candidates — {len(candidates)} (single tool: unique catch or possible FP)", ""] + out += ["_(none)_"] if not candidates else [ + f"- `{c.path}:{c.line}` **[{c.severity} · {c.category_name}]** " + f"({c.tools[0]})" for c in candidates[:max_list] + ] + if len(candidates) > max_list: + out.append(f"- … (+{len(candidates) - max_list} more)") + + out += _coverage_section(meta, cov, scored) + out += [ + "", "## How to read this", "", + "- **Where it hurts most** is the triage order: fix top modules first.", + "- **High-confidence** = two independent tools flag the same spot — start here.", + "- **Candidates** are single-tool: either a unique own-check catch (the leak " + "classes the oracles can't express) or a possible false positive to harden.", + "- **Coverage** is the honesty map: NO-TOOL categories are deferred to the " + "runtime layer, not silently \"clean\"; suppressed DevExpress findings are " + "counted, not hidden; unmapped rules are pending taxonomy, not lost.", + "", + ] + return "\n".join(out) + + +def _coverage_section(meta: dict[str, Any], cov: dict[str, Any], + scored: dict[str, Any]) -> list[str]: + out: list[str] = ["", "## Coverage / honesty", ""] + out.append(f"- findings ingested: {cov.get('total', 0)} " + f"(kept {cov.get('kept', 0)}, suppressed {cov.get('suppressed', 0)})") + if cov.get("suppressed_by"): + for reason, n in sorted(cov["suppressed_by"].items()): + out.append(f" - suppressed — {reason}: {n}") + no_tool = meta.get("no_tool_static") or [] + if no_tool: + labels = ", ".join(f"{c} ({CATEGORY_LABELS.get(c, '?')})" for c in no_tool) + out.append(f"- **NO-TOOL (static)** → deferred to runtime layer: {labels}") + unmapped = cov.get("uncategorized_rules") or {} + if unmapped: + shown = ", ".join(f"`{r}` x{n}" for r, n in sorted(unmapped.items())) + out.append(f"- unmapped rules (pending taxonomy, not dropped): {shown}") + else: + out.append("- unmapped rules: none — every flagged rule is categorized") + by_sev = scored.get("by_severity") or {} + if by_sev: + out.append("- by severity: " + + ", ".join(f"{k}={by_sev[k]}" for k in sorted(by_sev))) + return out + + +def render_json(meta: dict[str, Any], cov: dict[str, Any], + scored: dict[str, Any]) -> dict[str, Any]: + return {"meta": meta, "coverage": cov, **score_to_json(scored)} + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Render the audit health report (markdown/json).") + ap.add_argument("--findings", help="normalize.py --json output") + ap.add_argument("--taxonomy", default=str( + Path(__file__).resolve().parents[1] / "static" / "taxonomy" / "categories.yml")) + ap.add_argument("--format", choices=["markdown", "json"], default="markdown") + ap.add_argument("--target", default="") + ap.add_argument("--commit", default="") + ap.add_argument("--line-tol", type=int, default=3) + ap.add_argument("--selftest", action="store_true") + args = ap.parse_args(argv) + + if args.selftest: + return _selftest() + if not args.findings: + ap.error("--findings is required (or use --selftest)") + + tax = load_taxonomy(args.taxonomy) + payload = json.loads(Path(args.findings).read_text(encoding="utf-8")) + cov = payload.get("coverage") or {} + findings = [AuditFinding( + tool=d["tool"], path=d["path"], line=d["line"], rule=d["rule"], + message=d.get("message", ""), category=d.get("category", 0), + category_name=d.get("category_name", "uncategorized"), + resource=d.get("resource")) for d in payload.get("findings", [])] + scored = score(findings, tax, args.line_tol) + meta = {"target": args.target, "commit": args.commit, "line_tol": args.line_tol} + if args.format == "json": + print(json.dumps(render_json(meta, cov, scored), indent=2)) + else: + print(render_markdown(meta, cov, scored)) + return 0 + + +# --------------------------------------------------------------------------- # +# Selftest # +# --------------------------------------------------------------------------- # + +def _selftest() -> int: + tax_path = Path(__file__).resolve().parents[1] / "static" / "taxonomy" / "categories.yml" + tax = load_taxonomy(tax_path) + fails: list[str] = [] + + findings = normalize_results([], tax) # start empty, then add concrete ones + findings += [ + AuditFinding("own-check", "src/Util/Io.cs", 9, "OWN001", "leak", 1, "idisposable-leak"), + AuditFinding("codeql", "src/Util/Io.cs", 10, "cs/local-not-disposed", "nd", 1, + "idisposable-leak"), + AuditFinding("own-check", "src/Vm/Customer.cs", 12, "OWN001", "sub", 2, + "subscription-leak", resource="subscription token"), + AuditFinding("codeql", "DevExpress.Xpf/G.cs", 4, "cs/empty-block", "x", 0, + "uncategorized", suppressed=True, suppress_reason="third-party: DevExpress."), + AuditFinding("codeql", "src/Util/Misc.cs", 3, "FOO999", "y", 0, "uncategorized"), + ] + cov = coverage(findings) + scored = score(findings, tax) + meta = {"target": "acme/legacy", "commit": "abc123", "generated": "2026-06-24", + "profile": "desktop-wpf", "tiers": "build-free", "line_tol": 3, + "no_tool_static": [6, 11]} + + md = render_markdown(meta, cov, scored) + for needle in ("# Own.NET Audit — health report", "## Where it hurts most", + "## High-confidence findings", "## Candidates", + "## Coverage / honesty", "NO-TOOL", "How to read"): + if needle not in md: + fails.append(f"markdown missing section/marker: {needle!r}") + if "third-party: DevExpress." not in md: + fails.append("coverage must report the suppressed DevExpress count") + if "`FOO999`" not in md: + fails.append("coverage must surface the unmapped FOO999 rule") + if "src/Util" not in md: + fails.append("heatmap must list the worst module (src/Util)") + # the agreed leak (high-confidence) must be the worst module, ahead of src/Vm + util_pos, vm_pos = md.find("`src/Util`"), md.find("`src/Vm`") + if util_pos == -1 or (vm_pos != -1 and util_pos > vm_pos): + fails.append("heatmap ordering: src/Util (agreed leak) must precede src/Vm") + + js = render_json(meta, cov, scored) + if js["meta"]["target"] != "acme/legacy": + fails.append("json lost meta") + if js["totals"]["high_confidence"] != 1: + fails.append(f"json high_confidence wrong: {js['totals']}") + if js["coverage"]["suppressed"] != 1: + fails.append("json coverage lost suppressed count") + if not js["clusters"] or "evidence" not in js["clusters"][0]: + fails.append("json clusters missing evidence") + + total = 13 + for f in fails: + print(f"REPORT SELFTEST FAIL: {f}") + print(f"report selftest: {total - len(fails)}/{total} checks passed") + return 1 if fails else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/audit/aggregate/score.py b/audit/aggregate/score.py new file mode 100755 index 00000000..12dffc5a --- /dev/null +++ b/audit/aggregate/score.py @@ -0,0 +1,261 @@ +#!/usr/bin/env python3 +""" +Own.NET Audit — scoring (Plan.md §3.5). Generalizes ``oracle_compare.compare()`` +from a 3-way leak diff into a cross-tool confidence + severity + heatmap roll-up. + +Three axes, kept independent on purpose: + + 1. **Agreement** — findings at the same ``(basename, line ± window)`` across tools + cluster together (robust to path-prefix differences, exactly as oracle_compare + matches). A cluster confirmed by >= 2 distinct tools is ``high`` confidence; + a lone finding is a ``candidate`` (a unique own-check catch, or a possible FP). + 2. **Severity** — each cluster inherits a baseline ``P0..P3`` from its category + (``category_severity`` in the taxonomy). Severity answers "how bad", agreement + answers "how sure"; conflating them hides one behind the other. + 3. **Heatmap** — clusters roll up per module (directory) into a pain index + ``severity_weight x confidence_weight``, sorted descending. This is the direct + answer to "where does it hurt most / where is it almost fine", not a dump of + 3000 "possible issue" lines. + +Input is the list of ``AuditFinding`` from normalize.py (suppressed ones excluded). +""" + +from __future__ import annotations + +import argparse +import json +import sys +from collections import Counter, defaultdict +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from normalize import ( + AuditFinding, + Taxonomy, + load_taxonomy, +) + +SEV_WEIGHT = {"P0": 8.0, "P1": 4.0, "P2": 2.0, "P3": 1.0} +CONF_WEIGHT = {"high": 2.0, "candidate": 1.0} + + +@dataclass +class Cluster: + findings: list[AuditFinding] + category: int + category_name: str + severity: str + confidence: str + module: str = field(init=False, default="") + path: str = field(init=False, default="") + line: int = field(init=False, default=0) + + def __post_init__(self) -> None: + rep = self.findings[0] + self.module, self.path, self.line = rep.module, rep.path, rep.line + + @property + def tools(self) -> list[str]: + return sorted({f.tool for f in self.findings}) + + @property + def pain(self) -> float: + return SEV_WEIGHT.get(self.severity, 1.0) * CONF_WEIGHT.get(self.confidence, 1.0) + + +def _cluster_one_file(group: list[AuditFinding], tol: int) -> list[list[AuditFinding]]: + """Greedily merge findings in one file whose lines fall within ``tol`` of an + existing cluster's span. Order-independent enough for a line window.""" + clusters: list[list[AuditFinding]] = [] + for f in sorted(group, key=lambda x: x.line): + placed = False + for c in clusters: + if any(abs(f.line - g.line) <= tol for g in c): + c.append(f) + placed = True + break + if not placed: + clusters.append([f]) + return clusters + + +def _pick_category(members: list[AuditFinding], tax: Taxonomy) -> tuple[int, str, str]: + """The cluster's category is its most-severe member (ties -> lowest category id), + so a leak co-located with a generic-quality hit is ranked as a leak.""" + def key(f: AuditFinding) -> tuple[float, int]: + return (SEV_WEIGHT.get(tax.severity_for(f.category), 1.0), -f.category) + best = max(members, key=key) + return best.category, best.category_name, tax.severity_for(best.category) + + +def score(findings: list[AuditFinding], tax: Taxonomy, line_tol: int = 3) -> dict[str, Any]: + kept = [f for f in findings if not f.suppressed] + by_file: dict[str, list[AuditFinding]] = defaultdict(list) + for f in kept: + by_file[f.fkey].append(f) + + clusters: list[Cluster] = [] + for group in by_file.values(): + for members in _cluster_one_file(group, line_tol): + category, name, severity = _pick_category(members, tax) + confidence = "high" if len({m.tool for m in members}) >= 2 else "candidate" + clusters.append(Cluster(findings=members, category=category, + category_name=name, severity=severity, + confidence=confidence)) + + clusters.sort(key=lambda c: (-c.pain, c.module, c.path, c.line)) + + heat: dict[str, dict[str, Any]] = defaultdict( + lambda: {"pain": 0.0, "findings": 0, "high": 0, "categories": Counter()}) + for c in clusters: + h = heat[c.module] + h["pain"] += c.pain + h["findings"] += 1 + h["high"] += 1 if c.confidence == "high" else 0 + h["categories"][c.category_name] += 1 + heatmap = sorted( + ({"module": m, "pain": round(v["pain"], 2), "findings": v["findings"], + "high_confidence": v["high"], + "top_category": (v["categories"].most_common(1)[0][0] if v["categories"] else "")} + for m, v in heat.items()), + key=lambda r: (-r["pain"], r["module"])) + + return { + "clusters": clusters, + "heatmap": heatmap, + "totals": { + "clusters": len(clusters), + "high_confidence": sum(1 for c in clusters if c.confidence == "high"), + "candidates": sum(1 for c in clusters if c.confidence == "candidate"), + }, + "by_category": dict(Counter(c.category_name for c in clusters)), + "by_severity": dict(Counter(c.severity for c in clusters)), + } + + +def cluster_to_dict(c: Cluster) -> dict[str, Any]: + return { + "path": c.path, "line": c.line, "module": c.module, + "category": c.category, "category_name": c.category_name, + "severity": c.severity, "confidence": c.confidence, + "pain": round(c.pain, 2), "tools": c.tools, + "evidence": [f"{f.tool} {f.rule}: {f.message}" for f in c.findings], + } + + +def to_json(scored: dict[str, Any]) -> dict[str, Any]: + return { + "totals": scored["totals"], + "by_category": scored["by_category"], + "by_severity": scored["by_severity"], + "heatmap": scored["heatmap"], + "clusters": [cluster_to_dict(c) for c in scored["clusters"]], + } + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser( + description="Score normalized findings (agreement/severity/heatmap).") + ap.add_argument("--findings", help="normalize.py --json output (findings + coverage)") + ap.add_argument("--taxonomy", default=str( + Path(__file__).resolve().parents[1] / "static" / "taxonomy" / "categories.yml")) + ap.add_argument("--line-tol", type=int, default=3) + ap.add_argument("--json", dest="json_out", default="") + ap.add_argument("--selftest", action="store_true") + args = ap.parse_args(argv) + + if args.selftest: + return _selftest() + if not args.findings: + ap.error("--findings is required (or use --selftest)") + + tax = load_taxonomy(args.taxonomy) + payload = json.loads(Path(args.findings).read_text(encoding="utf-8")) + findings = [AuditFinding( + tool=d["tool"], path=d["path"], line=d["line"], rule=d["rule"], + message=d.get("message", ""), category=d.get("category", 0), + category_name=d.get("category_name", "uncategorized"), + resource=d.get("resource"), suppressed=d.get("suppressed", False), + suppress_reason=d.get("suppress_reason", "")) for d in payload.get("findings", [])] + scored = score(findings, tax, args.line_tol) + out = to_json(scored) + if args.json_out: + Path(args.json_out).write_text(json.dumps(out, indent=2), encoding="utf-8") + print(json.dumps({"totals": out["totals"], "heatmap": out["heatmap"][:10]}, indent=2)) + return 0 + + +# --------------------------------------------------------------------------- # +# Selftest # +# --------------------------------------------------------------------------- # + +def _f(tool: str, path: str, line: int, rule: str, cat: int, name: str, + msg: str = "") -> AuditFinding: + return AuditFinding(tool=tool, path=path, line=line, rule=rule, message=msg, + category=cat, category_name=name) + + +def _selftest() -> int: + tax_path = Path(__file__).resolve().parents[1] / "static" / "taxonomy" / "categories.yml" + tax = load_taxonomy(tax_path) + fails: list[str] = [] + + findings = [ + # two tools at the same spot -> one high-confidence leak cluster (cat 1, P1) + _f("own-check", "src/Util/Io.cs", 9, "OWN001", 1, "idisposable-leak"), + _f("codeql", "src/Util/Io.cs", 10, "cs/local-not-disposed", 1, "idisposable-leak"), + # a lone subscription leak -> candidate, also P1 + _f("own-check", "src/Vm/CustomerViewModel.cs", 12, "OWN001", 2, "subscription-leak"), + # a lone general-quality hit -> candidate, P2 (lower pain) + _f("codeql", "src/Util/Style.cs", 4, "S1118", 14, "general-quality"), + ] + scored = score(findings, tax, line_tol=3) + + if scored["totals"]["clusters"] != 3: + fails.append(f"expected 3 clusters, got {scored['totals']['clusters']}") + if scored["totals"]["high_confidence"] != 1: + fails.append( + f"expected 1 high-confidence cluster, got {scored['totals']['high_confidence']}") + if scored["totals"]["candidates"] != 2: + fails.append(f"expected 2 candidates, got {scored['totals']['candidates']}") + + top = scored["clusters"][0] + if top.module != "src/Util": + fails.append(f"top cluster should be the agreed leak in src/Util, got {top.module}") + if top.confidence != "high": + fails.append("top cluster (highest pain) must be the cross-tool agreement") + if top.tools != ["codeql", "own-check"]: + fails.append(f"agreed cluster tools wrong: {top.tools}") + + # heatmap orders by pain: the agreed P1 leak module outranks the P2 candidate module + pain = {row["module"]: row["pain"] for row in scored["heatmap"]} + # src/Util has the high-conf P1 (4*2=8) + a candidate P2 (2*1=2) = 10; + # src/Vm has a candidate P1 (4*1=4). So src/Util must outrank src/Vm. + if pain.get("src/Util", 0) <= pain.get("src/Vm", 0): + fails.append(f"heatmap pain ordering wrong: {pain}") + if abs(pain.get("src/Util", 0) - 10.0) > 0.01: + fails.append(f"src/Util pain should be 10.0, got {pain.get('src/Util')}") + + # severity stays category-driven: the subscription leak is P1 even though alone + sub = next(c for c in scored["clusters"] if c.module == "src/Vm") + if sub.severity != "P1": + fails.append(f"subscription-leak cluster should be P1, got {sub.severity}") + + js = to_json(scored) + if js["totals"]["clusters"] != 3 or not js["heatmap"]: + fails.append("to_json lost totals/heatmap") + if "evidence" not in js["clusters"][0]: + fails.append("cluster json missing evidence") + + total = 9 + real_fails = [f for f in fails if f] + for f in real_fails: + print(f"SCORE SELFTEST FAIL: {f}") + print(f"score selftest: {total - len(real_fails)}/{total} checks passed") + return 1 if real_fails else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/audit/config/profiles/desktop-wpf.yml b/audit/config/profiles/desktop-wpf.yml new file mode 100644 index 00000000..ac1451b8 --- /dev/null +++ b/audit/config/profiles/desktop-wpf.yml @@ -0,0 +1,51 @@ +# Own.NET Audit — profile: desktop-wpf (Plan.md §3.1). +# +# A profile declares WHICH analyzer packs to inject and the severity floor for an +# audit run. It is the default profile for the legacy net472 / WPF / DevExpress +# target. Profiles are data, not code; run_static.py reads the active one. +# +# build-free tools run even on a solution that does not compile; build-required +# tools need a successful MSBuild build (Plan.md §3.2). The whole audit runs on a +# local Windows machine — there is no CI run of the target. + +name: desktop-wpf +description: > + Legacy .NET Framework 4.7.2 / WPF / DevExpress desktop application. Leak- and + lifetime-focused, with DevExpress third-party noise baseline-suppressed. + +severity_floor: warning # own-check --severity; advisory tier kept in the report + +tiers: + build_free: + - own-check # error-tolerant SemanticModel; works on a broken solution + - codeql # build-mode: none, security-and-quality suite + build_required: + - roslyn-pack # NetAnalyzers, Meziantou, Roslynator, AsyncFixer, + # SonarAnalyzer, IDisposableAnalyzers, WpfAnalyzers, + # PropertyChangedAnalyzers (one build, many analyzers) + - infersharp # needs .dll + .pdb; most fragile step on net472 + +# Analyzer packs restored into the audit cache for the build-required tier. Pin +# versions whose Roslyn runtime is compatible with the target's MSBuild toolchain; +# an incompatible pack is marked NO-TOOL, never forced (Plan.md §3.3). +roslyn_packs: + - Microsoft.CodeAnalysis.NetAnalyzers + - Meziantou.Analyzer + - Roslynator.Analyzers + - AsyncFixer + - SonarAnalyzer.CSharp + - IDisposableAnalyzers + - WpfAnalyzers + - PropertyChangedAnalyzers + +# Categories with no reliable static tool in this profile — reported honestly as +# NO-TOOL / deferred-to-runtime in the coverage section, never faked (Plan.md §2). +no_tool_static: + - 4 # DependencyPropertyDescriptor.AddValueChanged leak -> runtime leak-harness + - 6 # PropertyChanged storms / expensive getters -> runtime + - 7 # WPF binding errors -> runtime + - 8 # broken/disabled virtualization -> runtime + - 10 # allocations in converters/getters -> runtime + - 11 # duplicated immutable data (the project's "gold") -> runtime + - 12 # heavy reference data / LOH / Gen2 bloat -> runtime + - 13 # cross-thread ObjectDisposedException / INPC -> runtime diff --git a/audit/requirements.txt b/audit/requirements.txt new file mode 100644 index 00000000..9d6b7071 --- /dev/null +++ b/audit/requirements.txt @@ -0,0 +1,6 @@ +# Own.NET Audit — the audit subtree's only third-party Python dependency. +# +# The core ownlang package and its test suite are deliberately zero-dependency; +# this requirement is scoped to audit/ alone (its own CI job installs it) so the +# core suite stays dependency-free. PyYAML reads the taxonomy/profile data files. +PyYAML>=6.0 diff --git a/audit/static/inject/OwnAudit.Directory.Build.props b/audit/static/inject/OwnAudit.Directory.Build.props new file mode 100644 index 00000000..59c13579 --- /dev/null +++ b/audit/static/inject/OwnAudit.Directory.Build.props @@ -0,0 +1,39 @@ + + + + + + true + All + true + + $(MSBuildProjectDirectory)\..\artifacts\own-audit\$(MSBuildProjectName).sarif,version=2.1 + + false + false + + + + + + + diff --git a/audit/static/inject/OwnAudit.Directory.Build.targets b/audit/static/inject/OwnAudit.Directory.Build.targets new file mode 100644 index 00000000..ca72ee1d --- /dev/null +++ b/audit/static/inject/OwnAudit.Directory.Build.targets @@ -0,0 +1,18 @@ + + + + + + + + diff --git a/audit/static/run_static.py b/audit/static/run_static.py new file mode 100755 index 00000000..15a06d8f --- /dev/null +++ b/audit/static/run_static.py @@ -0,0 +1,240 @@ +#!/usr/bin/env python3 +""" +Own.NET Audit — static layer orchestrator (Plan.md §3.3/§3.5). + +Runs the build-free tier of analyzers over a target, collects each tool's SARIF, +then normalizes → scores → renders the health report. Build-required runners +(Roslyn packs, Infer#) run on the local Windows machine and drop their SARIF into +the same artifacts directory; this orchestrator picks up whatever is present, so a +partial run still produces a (partial, honestly-labelled) report. + +Every runner is best-effort: an unavailable tool (no dotnet, no codeql, a build +that did not compile) is recorded as a tier gap in the coverage section, never a +crash — the continue-on-error discipline of Plan.md §3.2. + +Usage: + run_static.py --target /path/to/legacy --profile desktop-wpf --out artifacts/own-audit + run_static.py --selftest +""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +_HERE = Path(__file__).resolve().parent +_AUDIT = _HERE.parent +sys.path.insert(0, str(_AUDIT / "aggregate")) +sys.path.insert(0, str(_HERE / "tools")) + +from normalize import coverage, load_taxonomy, normalize_results # noqa: E402 +from owncheck import run_own_check # noqa: E402 +from report import render_json, render_markdown # noqa: E402 +from score import score # noqa: E402 + +try: + from oracle_compare import parse_sarif +except ImportError: # pragma: no cover + sys.path.insert(0, str(_AUDIT.parent / "scripts")) + from oracle_compare import parse_sarif + +DEFAULT_TAXONOMY = _AUDIT / "static" / "taxonomy" / "categories.yml" +DEFAULT_PROFILE_DIR = _AUDIT / "config" / "profiles" + + +def load_profile(name_or_path: str) -> dict[str, Any]: + import yaml + + p = Path(name_or_path) + if not p.exists(): + p = DEFAULT_PROFILE_DIR / f"{name_or_path}.yml" + return yaml.safe_load(p.read_text(encoding="utf-8")) or {} + + +def aggregate(sarif_inputs: list[tuple[str, str]], out_dir: Path, meta: dict[str, Any], + taxonomy: Path = DEFAULT_TAXONOMY, line_tol: int = 3) -> dict[str, Any]: + """Normalize → score → render the SARIFs in ``sarif_inputs`` and write the + markdown + json reports to ``out_dir``. Returns the scored totals.""" + out_dir.mkdir(parents=True, exist_ok=True) + tax = load_taxonomy(taxonomy) + raw: list[Any] = [] + for tool, path in sarif_inputs: + raw += parse_sarif(Path(path).read_text(encoding="utf-8"), tool, meta.get("strip", [])) + findings = normalize_results(raw, tax) + cov = coverage(findings) + scored = score(findings, tax, line_tol) + meta = {**meta, "line_tol": line_tol} + + (out_dir / "report.md").write_text(render_markdown(meta, cov, scored), encoding="utf-8") + (out_dir / "report.json").write_text( + json.dumps(render_json(meta, cov, scored), indent=2), encoding="utf-8") + return {"totals": scored["totals"], "coverage": cov, + "report_md": str(out_dir / "report.md"), + "report_json": str(out_dir / "report.json")} + + +def _run_codeql(target: str, out_dir: Path) -> dict[str, Any]: + """Best-effort build-free CodeQL via the runner shell. Exit 3 = NO-TOOL.""" + runner = _HERE / "tools" / "codeql.sh" + status: dict[str, Any] = {"tool": "codeql", "tier": "build-free", + "available": False, "sarif": None, "reason": ""} + if not runner.exists(): + status["reason"] = "codeql.sh runner missing" + return status + proc = subprocess.run([str(runner), "--target", target, "--out", str(out_dir)], + capture_output=True, text=True, check=False) + sarif = out_dir / "codeql.sarif" + if proc.returncode == 0 and sarif.exists(): + status.update(available=True, sarif=str(sarif)) + else: + status["reason"] = proc.stderr.strip().splitlines()[-1] if proc.stderr.strip() else \ + f"codeql runner exit {proc.returncode}" + return status + + +def run(target: str, profile: dict[str, Any], out_dir: Path, target_name: str = "", + commit: str = "", line_tol: int = 3) -> dict[str, Any]: + out_dir.mkdir(parents=True, exist_ok=True) + severity = profile.get("severity_floor", "warning") + build_free = (profile.get("tiers") or {}).get("build_free") or [] + + tiers: list[dict[str, Any]] = [] + sarif_inputs: list[tuple[str, str]] = [] + if "own-check" in build_free: + st = run_own_check(target, out_dir, severity) + tiers.append(st) + if st["available"] and st["sarif"]: + sarif_inputs.append(("own-check", st["sarif"])) + if "codeql" in build_free: + st = _run_codeql(target, out_dir) + tiers.append(st) + if st["available"] and st["sarif"]: + sarif_inputs.append(("codeql", st["sarif"])) + + # Pick up any build-required SARIFs already dropped here by the Windows runners. + for extra, tool in (("roslyn_pack.sarif", "roslyn-pack"), ("infersharp.sarif", "infersharp")): + p = out_dir / extra + if p.exists(): + sarif_inputs.append((tool, str(p))) + tiers.append({"tool": tool, "tier": "build-required", "available": True, + "sarif": str(p), "reason": ""}) + + meta = { + "target": target_name or target, "commit": commit, + "generated": f"{datetime.now(UTC):%Y-%m-%d %H:%M UTC}", + "profile": profile.get("name", "?"), + "tiers": ", ".join(f"{t['tool']}={'ok' if t['available'] else 'NO-TOOL'}" for t in tiers) + or "(no runners)", + "no_tool_static": profile.get("no_tool_static") or [], + } + result = aggregate(sarif_inputs, out_dir, meta, line_tol=line_tol) + result["tiers"] = tiers + return result + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Run the static audit layer over a target.") + ap.add_argument("--target", help="path to the target source tree") + ap.add_argument("--profile", default="desktop-wpf", help="profile name or path") + ap.add_argument("--out", default="artifacts/own-audit", help="artifacts/report directory") + ap.add_argument("--target-name", default="", help="owner/repo label for the report header") + ap.add_argument("--commit", default="", help="commit SHA for the report header") + ap.add_argument("--line-tol", type=int, default=3) + ap.add_argument("--selftest", action="store_true") + args = ap.parse_args(argv) + + if args.selftest: + return _selftest() + if not args.target: + ap.error("--target is required (or use --selftest)") + + profile = load_profile(args.profile) + result = run(args.target, profile, Path(args.out), args.target_name, + args.commit, args.line_tol) + print(json.dumps({"totals": result["totals"], + "tiers": [{"tool": t["tool"], "available": t["available"], + "reason": t["reason"]} for t in result["tiers"]], + "report_md": result["report_md"]}, indent=2)) + return 0 + + +# --------------------------------------------------------------------------- # +# Selftest — full normalize→score→render pipeline on embedded SARIF fixtures, # +# no external tools needed (so it gates on Linux CI like oracle_compare). # +# --------------------------------------------------------------------------- # + +def _fixture_sarifs(tmp: Path) -> list[tuple[str, str]]: + own = {"version": "2.1.0", "runs": [{"tool": {"driver": {"name": "Own.NET"}}, "results": [ + {"ruleId": "OWN001", "level": "warning", + "message": {"text": "event subscribed, no -= [resource: subscription token]"}, + "locations": [{"physicalLocation": {"artifactLocation": {"uri": "src/Vm/Customer.cs"}, + "region": {"startLine": 12}}}]}, + {"ruleId": "OWN001", "level": "warning", + "message": {"text": "local IDisposable never disposed"}, + "locations": [{"physicalLocation": {"artifactLocation": {"uri": "src/Util/Io.cs"}, + "region": {"startLine": 9}}}]}, + ]}]} + codeql = {"runs": [{"results": [ + {"ruleId": "cs/local-not-disposed", "message": {"text": "not disposed"}, + "locations": [{"physicalLocation": {"artifactLocation": {"uri": "src/Util/Io.cs"}, + "region": {"startLine": 10}}}]}, + {"ruleId": "cs/empty-block", "message": {"text": "empty"}, + "locations": [{"physicalLocation": {"artifactLocation": {"uri": "DevExpress.Xpf/G.cs"}, + "region": {"startLine": 4}}}]}, + ]}]} + (tmp / "own-check.sarif").write_text(json.dumps(own), encoding="utf-8") + (tmp / "codeql.sarif").write_text(json.dumps(codeql), encoding="utf-8") + return [("own-check", str(tmp / "own-check.sarif")), ("codeql", str(tmp / "codeql.sarif"))] + + +def _selftest() -> int: + import tempfile + + fails: list[str] = [] + with tempfile.TemporaryDirectory() as td: + tmp = Path(td) + inputs = _fixture_sarifs(tmp) + meta = {"target": "acme/legacy", "commit": "abc123", "profile": "desktop-wpf", + "tiers": "own-check=ok, codeql=ok", "no_tool_static": [6, 11]} + result = aggregate(inputs, tmp / "report", meta) + + # the agreed Io.cs leak is high-confidence; Customer.cs subscription is a candidate + if result["totals"]["high_confidence"] != 1: + fails.append(f"expected 1 high-confidence cluster, got {result['totals']}") + if result["totals"]["candidates"] != 1: + fails.append(f"expected 1 candidate, got {result['totals']}") + + md = Path(result["report_md"]).read_text(encoding="utf-8") + if "# Own.NET Audit — health report" not in md: + fails.append("report.md missing title") + if "## Where it hurts most" not in md or "## Coverage / honesty" not in md: + fails.append("report.md missing a required section") + if "third-party: DevExpress." not in md: + fails.append("report.md must report the suppressed DevExpress finding") + # src/Util (agreed leak) must outrank src/Vm (lone subscription) in the heatmap + if md.find("`src/Util`") == -1 or (md.find("`src/Vm`") != -1 + and md.find("`src/Util`") > md.find("`src/Vm`")): + fails.append("heatmap ordering: src/Util must precede src/Vm") + + js = json.loads(Path(result["report_json"]).read_text(encoding="utf-8")) + if js["coverage"]["suppressed"] != 1: + fails.append("report.json coverage lost the suppressed count") + if js["meta"]["target"] != "acme/legacy": + fails.append("report.json lost meta") + if not (tmp / "report" / "report.md").exists(): + fails.append("aggregate did not write report.md to disk") + + total = 8 + for f in fails: + print(f"RUN_STATIC SELFTEST FAIL: {f}") + print(f"run_static selftest: {total - len(fails)}/{total} checks passed") + return 1 if fails else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/audit/static/taxonomy/categories.yml b/audit/static/taxonomy/categories.yml new file mode 100644 index 00000000..bf021928 --- /dev/null +++ b/audit/static/taxonomy/categories.yml @@ -0,0 +1,77 @@ +# Own.NET Audit — category taxonomy (knowledge base). +# +# Maps each analyzer's ruleId (exact, or a glob like `IDISP0*` / `*RESOURCE_LEAK`) +# to one of the problem categories defined in Plan.md §2. normalize.py reads this +# and labels every SARIF result. Rules with NO mapping are NOT dropped — they land +# in `uncategorized` and are surfaced in the report's coverage section, so the +# taxonomy grows deliberately instead of swallowing findings (same discipline as +# oracle_compare's unparsed-line bucket). +# +# Precedence: an exact ruleId match wins over a glob; among globs, the most +# specific (longest non-wildcard prefix) wins. + +rules: + # ── Category 1: IDisposable leak (local/field, dispose-on-throw) ────────────── + CA2000: {category: 1, name: idisposable-leak} + CA2213: {category: 1, name: idisposable-leak} + "IDISP0*": {category: 1, name: idisposable-leak} # IDisposableAnalyzers + "cs/local-not-disposed": {category: 1, name: idisposable-leak} + "cs/missing-dispose": {category: 1, name: idisposable-leak} + "cs/dispose-not-called-on-throw": {category: 1, name: idisposable-leak} + "*RESOURCE_LEAK": {category: 1, name: idisposable-leak} # Infer#: (PULSE_)RESOURCE_LEAK + "*MEMORY_LEAK": {category: 1, name: idisposable-leak} + + # ── own-check OWN001 is an UMBRELLA leak code ──────────────────────────────── + # One ruleId covers IDisposable fields/locals, subscription tokens and timers; + # the `[resource: ...]` tag in the message text distinguishes them. A flat + # OWN001 -> cat.1 would collapse subscription/timer leaks (cat. 2/3) into the + # IDisposable bucket and undercount them, so normalize.py splits on the tag. + OWN001: + by_resource: # key matched against [resource: KEY] + "subscription token": {category: 2, name: subscription-leak} + "timer": {category: 3, name: timer-leak} + "*": {category: 1, name: idisposable-leak} # disposable / disposable field / pooled buffer + + # ── Category 2: event/subscription leak & region escape ────────────────────── + OWN014: {category: 2, name: region-escape} # escape to a longer-lived region + # (vm->App, SystemEvents) — NOT a + # generic subscription-leak + + # ── Category 5: INPC correctness ───────────────────────────────────────────── + "INPC0*": {category: 5, name: inpc-correctness} # PropertyChangedAnalyzers + + # ── Category 9: WPF Freezable / per-instance brush-geometry (partial) ──────── + "WPF0*": {category: 9, name: wpf-freezable} # WpfAnalyzers (subset) + + # ── Category 14: general bugs / perf / best-practice / async ────────────────── + "CA1*": {category: 14, name: general-quality} # NetAnalyzers design/perf + "CA2*": {category: 14, name: general-quality} # (CA2000/CA2213 above win by exactness) + "S*": {category: 14, name: general-quality} # SonarAnalyzer.CSharp + "RCS*": {category: 14, name: general-quality} # Roslynator + "MA0*": {category: 14, name: general-quality} # Meziantou.Analyzer + "AsyncFixer*": {category: 14, name: general-quality} + "cs/*": {category: 14, name: general-quality} # CodeQL security-and-quality (leak ids above win) + + # ── Category 15: architecture metrics / hotspots ───────────────────────────── + "RCS1*": {category: 15, name: architecture} # Roslynator maintainability subset + +# Baseline P-level per category (Plan.md §3.5.2). Confidence (cross-tool +# agreement) is a SEPARATE axis handled by score.py; this is severity only. +category_severity: + 1: P1 # leaks — the high-value classes + 2: P1 + 3: P1 + 5: P2 + 9: P2 + 14: P2 + 15: P3 + 0: P3 # uncategorized + +# DevExpress baseline-suppress (Plan.md §3.4, confirmed decision). Findings whose +# path OR message contains one of these tokens are third-party noise: dropped from +# the main report but COUNTED and reported in the coverage section — nothing is +# hidden silently. This is an honest third-party-namespace filter, not a regex +# heuristic over our own code. +suppress: + path_or_message_contains: + - "DevExpress." diff --git a/audit/static/tools/codeql.sh b/audit/static/tools/codeql.sh new file mode 100755 index 00000000..7be78d3c --- /dev/null +++ b/audit/static/tools/codeql.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# +# Own.NET Audit — CodeQL runner (build-free tier). +# +# CodeQL can analyze C# straight from source with `--build-mode=none` (no MSBuild +# build of the target needed) — that is what makes it build-free (Plan.md §3.2). +# +# CRITICAL: the dispose / not-disposed queries live in the `security-and-quality` +# suite, NOT the default `security` suite. Using the default suite makes CodeQL +# silently return zero leak findings — a rake already documented in oracle.yml. +# +# Usage: +# codeql.sh --target --out [--db ] +# +# Emits: /codeql.sarif. Exits 3 (NO-TOOL) if the codeql CLI is not installed, +# so the orchestrator records the tier as unavailable rather than failing the run. + +set -euo pipefail + +target="" +out="artifacts/own-audit" +db="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --target) target="$2"; shift 2 ;; + --out) out="$2"; shift 2 ;; + --db) db="$2"; shift 2 ;; + -h|--help) sed -n '2,18p' "$0"; exit 0 ;; + *) echo "codeql.sh: unknown arg $1" >&2; exit 2 ;; + esac +done + +[[ -n "$target" ]] || { echo "codeql.sh: --target is required" >&2; exit 2; } + +if ! command -v codeql >/dev/null 2>&1; then + echo "NO-TOOL: codeql CLI not installed — skipping the build-free CodeQL tier" >&2 + exit 3 +fi + +mkdir -p "$out" +db="${db:-$(mktemp -d)/codeql-db}" + +# build-mode: none — analyze C# from source, no target build required. +codeql database create "$db" --language=csharp --build-mode=none --source-root="$target" --overwrite + +# security-and-quality carries the dispose/leak quality queries (see header). +codeql database analyze "$db" \ + --format=sarifv2.1.0 \ + --output="$out/codeql.sarif" \ + codeql/csharp-queries:codeql-suites/csharp-security-and-quality.qls + +echo "codeql.sh: wrote $out/codeql.sarif" diff --git a/audit/static/tools/infersharp.sh b/audit/static/tools/infersharp.sh new file mode 100755 index 00000000..d1a035f1 --- /dev/null +++ b/audit/static/tools/infersharp.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# +# Own.NET Audit — Infer# runner (build-required tier). +# +# Infer# analyzes compiled .NET binaries, so it needs a successful build of the +# target with PDBs (`.dll` + `.pdb`) — build-required (Plan.md §3.2/§3.3). For +# net472 on a 12-year-old solution this is the most fragile step, which is why the +# orchestrator treats it as continue-on-error: a failed build yields a partial +# report, not an empty one. +# +# This runs on the LOCAL WINDOWS MACHINE (VS Build Tools + DevExpress). There is no +# CI run of the target. +# +# Usage: +# infersharp.sh --bin --out +# +# Emits: /infersharp.sarif. Exits 3 (NO-TOOL) if the infersharp CLI/container +# is not available. + +set -euo pipefail + +bin="" +out="artifacts/own-audit" + +while [[ $# -gt 0 ]]; do + case "$1" in + --bin) bin="$2"; shift 2 ;; + --out) out="$2"; shift 2 ;; + -h|--help) sed -n '2,18p' "$0"; exit 0 ;; + *) echo "infersharp.sh: unknown arg $1" >&2; exit 2 ;; + esac +done + +[[ -n "$bin" ]] || { echo "infersharp.sh: --bin (built output with .dll+.pdb) is required" >&2; exit 2; } + +if ! command -v infersharp >/dev/null 2>&1 && ! command -v infersharpaction >/dev/null 2>&1; then + echo "NO-TOOL: Infer# CLI not available — skipping the build-required Infer# tier" >&2 + exit 3 +fi + +mkdir -p "$out" +# Infer# writes report.sarif into its output dir; copy it to the audit artifacts. +infersharp "$bin" --sarif --results-dir "$out/infer-out" +cp "$out/infer-out/report.sarif" "$out/infersharp.sarif" + +echo "infersharp.sh: wrote $out/infersharp.sarif" diff --git a/audit/static/tools/owncheck.py b/audit/static/tools/owncheck.py new file mode 100755 index 00000000..ee0b7340 --- /dev/null +++ b/audit/static/tools/owncheck.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 +""" +Own.NET Audit — own-check runner (build-free tier). + +A thin wrapper around the existing ``scripts/own-check.sh --format sarif`` (Plan.md +§3.3: reused as-is, consumed only via its CLI). own-check uses an error-tolerant +Roslyn ``SemanticModel``, so it runs even on a solution that does not compile — +hence build-free. It is the one runner that expresses the subscription / timer / +region-escape leak classes the oracle tools cannot (Plan.md §2, cat. 2-4). + +Requires a .NET SDK on PATH for the C# fact extractor. If ``dotnet`` is missing the +runner reports the tier as unavailable (a partial, honest result) instead of +crashing — the ``continue-on-error`` discipline from Plan.md §3.2. + +Usage: + owncheck.py --target /path/to/legacy/src --out artifacts/own-audit [--severity warning] +""" + +from __future__ import annotations + +import argparse +import json +import shutil +import subprocess +import sys +from pathlib import Path +from typing import Any + +_REPO = Path(__file__).resolve().parents[3] # audit/static/tools/ -> repo root +OWN_CHECK_SH = _REPO / "scripts" / "own-check.sh" + + +def run_own_check(target: str, out_dir: Path, severity: str = "warning", + root: Path | None = None) -> dict[str, Any]: + """Run own-check over ``target`` and write its SARIF to ``out_dir``. + + Returns a status dict: tool, tier, available, and (on success) the SARIF path + plus a finding count, or (on failure) a reason — never raises for an + unavailable toolchain.""" + out_dir.mkdir(parents=True, exist_ok=True) + sarif_path = out_dir / "own-check.sarif" + status: dict[str, Any] = {"tool": "own-check", "tier": "build-free", + "available": False, "sarif": None, "reason": ""} + + if not OWN_CHECK_SH.exists(): + status["reason"] = f"own-check.sh not found at {OWN_CHECK_SH}" + return status + if shutil.which("dotnet") is None: + status["reason"] = "dotnet SDK not on PATH (needed by the C# fact extractor)" + return status + + cmd = [str(OWN_CHECK_SH), "--format", "sarif", "--severity", severity, "--", target] + if root is not None: + cmd[1:1] = ["--root", str(root)] + proc = subprocess.run(cmd, capture_output=True, text=True, check=False) + sarif_text = proc.stdout.strip() + if not sarif_text.startswith("{"): + status["reason"] = (f"own-check did not emit SARIF (exit {proc.returncode}): " + f"{proc.stderr.strip()[:200]}") + return status + + sarif_path.write_text(sarif_text, encoding="utf-8") + try: + doc = json.loads(sarif_text) + n = sum(len(r.get("results", [])) for r in doc.get("runs", [])) + except json.JSONDecodeError: + n = 0 + status.update(available=True, sarif=str(sarif_path), findings=n) + return status + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Run own-check (build-free) -> SARIF.") + ap.add_argument("--target", required=True, help="path to the target source tree") + ap.add_argument("--out", default="artifacts/own-audit", help="SARIF output directory") + ap.add_argument("--severity", default="warning", choices=["error", "warning"]) + ap.add_argument("--root", default=None, help="Own.NET checkout (own-check --root)") + args = ap.parse_args(argv) + + status = run_own_check(args.target, Path(args.out), args.severity, + Path(args.root) if args.root else None) + print(json.dumps(status, indent=2)) + return 0 if status["available"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/audit/static/tools/roslyn_pack.ps1 b/audit/static/tools/roslyn_pack.ps1 new file mode 100644 index 00000000..a5711f92 --- /dev/null +++ b/audit/static/tools/roslyn_pack.ps1 @@ -0,0 +1,56 @@ +<# +.SYNOPSIS + Own.NET Audit - Roslyn analyzer-pack runner (build-required tier). + +.DESCRIPTION + Builds the target ONCE with the audit analyzer cache injected (one build, many + analyzers) and collects a per-project SARIF via . This is the + build-required tier (Plan.md 3.2/3.3): it needs a successful MSBuild build, so + it runs on the LOCAL WINDOWS MACHINE (VS Build Tools + DevExpress 12.2). There is + no CI run of the target. + + Mechanism (Plan.md 3.1): a throwaway `git worktree` of the target with + OwnAudit.Directory.Build.props/.targets copied in under MSBuild's recognized + names, all gated on /p:OwnAudit=true so developer builds are untouched. The + analyzer DLLs come from a pre-restored audit cache pointed to by + $OwnAuditAnalyzers, NOT a PackageReference in the 12-year-old project tree. + + Pin analyzer-pack versions whose Roslyn runtime matches the target's MSBuild + toolchain; an incompatible pack is recorded NO-TOOL, never forced. + +.PARAMETER Solution + Path to the target .sln (inside the audit worktree). + +.PARAMETER AnalyzerCache + Directory of pre-restored analyzer DLLs (sets $OwnAuditAnalyzers). + +.PARAMETER Out + Artifacts directory for the per-project SARIF (default artifacts\own-audit). + +.EXAMPLE + .\roslyn_pack.ps1 -Solution ..\target-audit\Target.sln -AnalyzerCache .\cache -Out artifacts\own-audit +#> +param( + [Parameter(Mandatory = $true)][string]$Solution, + [Parameter(Mandatory = $true)][string]$AnalyzerCache, + [string]$Out = "artifacts\own-audit" +) + +$ErrorActionPreference = "Stop" + +if (-not (Get-Command msbuild -ErrorAction SilentlyContinue)) { + Write-Error "NO-TOOL: msbuild not on PATH - install VS Build Tools (build-required tier runs on Windows)." + exit 3 +} + +New-Item -ItemType Directory -Force -Path $Out | Out-Null + +# continue-on-error: a failed build still yields whatever per-project SARIFs were +# produced before the failure - a partial, honest report, not an empty one. +msbuild $Solution ` + /p:OwnAudit=true ` + /p:OwnAuditAnalyzers=$AnalyzerCache ` + /p:Configuration=Release ` + /bl:"$Out\build.binlog" + +Write-Host "roslyn_pack.ps1: per-project SARIF under $Out (merged by audit/aggregate/)." From 14ce6357d568f3674a6c1f5bd2f8703c83b43741 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 24 Jun 2026 15:44:41 +0000 Subject: [PATCH 2/3] Address Codex + CodeRabbit review on PR #100 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex (4): - OWN050 notes kept out of scored findings: own-check emits OWN050 ("cannot verify — unresolved") as a SARIF note, a non-verdict. The taxonomy now lists it under coverage_notes; normalize routes such rules to the coverage ledger (analysis-skipped count) instead of scoring them, so they no longer surface as phantom uncategorized P3 candidates. - Roslyn ErrorLog now honors the runner's -Out: the injected props anchor the per-project SARIF at $(OwnAuditOutDir) (default unchanged), roslyn_pack.ps1 passes /p:OwnAuditOutDir=, and the targets MakeDir mirrors it. - run_static picks up the Roslyn per-project SARIFs: it globs /roslyn/*.sarif (the props write $(MSBuildProjectName).sarif under roslyn/) instead of a single fixed roslyn_pack.sarif, so analyzer-pack findings reach totals/coverage. - infersharp.sh requires the infersharp CLI and drops the invalid infersharpaction fallback (that is a GitHub Action, not a CLI with this interface); aligns the invocation to Infer#'s documented infer-out/report.sarif output. CodeRabbit (4 + 1 nitpick): - report.py: high-confidence list now appends a "+N more" note past max_list, matching the candidates overflow — no silent truncation against the header count. - owncheck.py: subprocess.run gains a 900s timeout; TimeoutExpired returns a graceful unavailable status (continue-on-error). - selftest totals are now derived from a check() helper in all four modules (normalize/score/report/run_static), so the printed pass ratio can no longer drift from the real check count. - infersharp.sh fallback removed (same as Codex item above). Adds selftest coverage for the OWN050 routing and the roslyn/ pickup. All four selftests green (19/11/15/11), ruff clean. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE --- audit/aggregate/normalize.py | 102 +++++++++++------- audit/aggregate/report.py | 42 ++++---- audit/aggregate/score.py | 57 +++++----- .../inject/OwnAudit.Directory.Build.props | 8 +- .../inject/OwnAudit.Directory.Build.targets | 7 +- audit/static/run_static.py | 81 +++++++++----- audit/static/taxonomy/categories.yml | 9 ++ audit/static/tools/infersharp.sh | 17 ++- audit/static/tools/owncheck.py | 6 +- audit/static/tools/roslyn_pack.ps1 | 10 +- 10 files changed, 205 insertions(+), 134 deletions(-) diff --git a/audit/aggregate/normalize.py b/audit/aggregate/normalize.py index 004f48e8..229cc030 100755 --- a/audit/aggregate/normalize.py +++ b/audit/aggregate/normalize.py @@ -78,6 +78,7 @@ class AuditFinding: resource: str | None = None suppressed: bool = False suppress_reason: str = "" + note: bool = False # analysis-skipped coverage note (e.g. OWN050), not a verdict fkey: str = field(init=False, default="") def __post_init__(self) -> None: @@ -88,12 +89,19 @@ def module(self) -> str: """Directory of the finding (the heatmap roll-up unit), or ``(root)``.""" return self.path.rsplit("/", 1)[0] if "/" in self.path else "(root)" + @property + def scored(self) -> bool: + """A finding counts toward the report only if it is neither third-party + suppressed nor an analysis-skipped coverage note.""" + return not self.suppressed and not self.note + @dataclass class Taxonomy: rules: dict[str, Any] category_severity: dict[int, str] suppress_tokens: list[str] + coverage_note_rules: set[str] def severity_for(self, category: int) -> str: return self.category_severity.get(category, "P3") @@ -105,8 +113,10 @@ def load_taxonomy(path: str | Path) -> Taxonomy: data = yaml.safe_load(Path(path).read_text(encoding="utf-8")) or {} sev = {int(k): str(v) for k, v in (data.get("category_severity") or {}).items()} suppress = list((data.get("suppress") or {}).get("path_or_message_contains") or []) + notes = {str(r) for r in (data.get("coverage_notes") or [])} return Taxonomy(rules=dict(data.get("rules") or {}), - category_severity=sev, suppress_tokens=suppress) + category_severity=sev, suppress_tokens=suppress, + coverage_note_rules=notes) def _resource_tag(message: str) -> str | None: @@ -161,18 +171,21 @@ def normalize_results(raw: list[Any], tax: Taxonomy) -> list[AuditFinding]: resource = _resource_tag(f.message) category, name = categorize(f.rule, resource, tax.rules) reason = _suppressed(f.path, f.message, tax.suppress_tokens) + is_note = f.rule in tax.coverage_note_rules out.append(AuditFinding( tool=f.tool, path=f.path, line=f.line, rule=f.rule, message=f.message, category=category, category_name=name, resource=resource, - suppressed=bool(reason), suppress_reason=reason)) + suppressed=bool(reason), suppress_reason=reason, note=is_note)) return out def coverage(findings: list[AuditFinding]) -> dict[str, Any]: - """The honesty ledger: what was categorized, what was suppressed, and which - rules have no taxonomy entry yet (so they are visibly pending, not lost).""" - kept = [f for f in findings if not f.suppressed] + """The honesty ledger: what was categorized, what was suppressed, which rules + are analysis-skipped coverage notes, and which rules have no taxonomy entry yet + (so they are visibly pending, not lost).""" + kept = [f for f in findings if f.scored] suppressed = [f for f in findings if f.suppressed] + notes = [f for f in findings if f.note and not f.suppressed] uncategorized = Counter(f.rule for f in kept if f.category == 0) return { "tools": sorted({f.tool for f in findings}), @@ -180,6 +193,8 @@ def coverage(findings: list[AuditFinding]) -> dict[str, Any]: "kept": len(kept), "suppressed": len(suppressed), "suppressed_by": dict(Counter(f.suppress_reason for f in suppressed)), + "analysis_skipped": len(notes), + "analysis_skipped_by": dict(Counter(f.rule for f in notes)), "by_category": dict(Counter(f.category for f in kept)), "uncategorized_rules": dict(uncategorized), } @@ -231,7 +246,7 @@ def main(argv: list[str] | None = None) -> int: tax = load_taxonomy(args.taxonomy) findings, cov = normalize(inputs, tax, args.strip) payload = {"coverage": cov, - "findings": [finding_to_dict(f) for f in findings if not f.suppressed]} + "findings": [finding_to_dict(f) for f in findings if f.scored]} if args.json_out: Path(args.json_out).write_text(json.dumps(payload, indent=2), encoding="utf-8") print(json.dumps(cov, indent=2)) @@ -258,6 +273,8 @@ def res(rule: str, msg: str, uri: str, line: int) -> dict[str, Any]: res("OWN001", "local IDisposable never disposed", "src/Util/Io.cs", 9), res("OWN014", "region escape: view-model promoted to App lifetime", "src/Vm/StaticEventEscapeViewModel.cs", 50), + res("OWN050", "cannot verify 'X.Y' — unresolved [resource: unresolved reference]", + "src/Util/Unknown.cs", 3), ]}]}) @@ -275,14 +292,17 @@ def res(rule: str, uri: str, line: int) -> dict[str, Any]: def _selftest() -> int: tax_path = Path(__file__).resolve().parents[1] / "static" / "taxonomy" / "categories.yml" - fails: list[str] = [] + checks: list[str] = [] + + def check(ok: bool, msg: str) -> None: # total derives from the call count + checks.append("" if ok else msg) # The shipped taxonomy must parse and carry the split that the OWN001 fix needs. tax = load_taxonomy(tax_path) - if "OWN001" not in tax.rules or "by_resource" not in tax.rules["OWN001"]: - fails.append("shipped categories.yml lost the OWN001 by_resource split") - if tax.rules.get("OWN014", {}).get("name") != "region-escape": - fails.append("shipped categories.yml: OWN014 must be region-escape, not subscription-leak") + check("by_resource" in tax.rules.get("OWN001", {}), + "shipped categories.yml lost the OWN001 by_resource split") + check(tax.rules.get("OWN014", {}).get("name") == "region-escape", + "shipped categories.yml: OWN014 must be region-escape, not subscription-leak") own = normalize_results(parse_sarif(_own_sarif(), "own-check", []), tax) by_file = {f.fkey: f for f in own} @@ -297,49 +317,49 @@ def _selftest() -> int: } for fkey, (cat, name) in cases.items(): got = by_file.get(fkey) - if got is None: - fails.append(f"missing finding for {fkey}") - elif (got.category, got.category_name) != (cat, name): - fails.append( - f"{fkey}: expected ({cat},{name}), got ({got.category},{got.category_name})") + check(got is not None and (got.category, got.category_name) == (cat, name), + f"{fkey}: expected ({cat},{name}), got " + f"{None if got is None else (got.category, got.category_name)}") + + # OWN050 is an analysis-skipped coverage note, not a verdict: routed out of + # scoring (Codex review on #100), never a phantom uncategorized P3 candidate. + own050 = by_file.get("unknown.cs") + check(own050 is not None and own050.note and not own050.scored, + "OWN050 must be a coverage note (note=True, scored=False)") # Glob mapping + uncategorized + DevExpress suppression on the CodeQL run. cq = normalize_results(parse_sarif(_codeql_sarif(), "codeql", []), tax) cq_by = {f.fkey: f for f in cq} - if cq_by["io.cs"].category != 1: - fails.append("cs/local-not-disposed should map to category 1") - if not cq_by["helper.cs"].suppressed: - fails.append("DevExpress finding must be baseline-suppressed") - if cq_by["misc.cs"].category != 0: - fails.append("unmapped FOO999 must be uncategorized (category 0)") + check(cq_by["io.cs"].category == 1, "cs/local-not-disposed should map to category 1") + check(cq_by["helper.cs"].suppressed, "DevExpress finding must be baseline-suppressed") + check(cq_by["misc.cs"].category == 0, "unmapped FOO999 must be uncategorized (category 0)") cov = coverage(own + cq) - if cov["suppressed"] != 1: - fails.append(f"coverage suppressed count wrong: {cov['suppressed']}") - if "FOO999" not in cov["uncategorized_rules"]: - fails.append("uncategorized rule FOO999 must be surfaced in coverage") - if cov["uncategorized_rules"].get("FOO999") != 1: - fails.append("uncategorized count wrong") + check(cov["suppressed"] == 1, f"coverage suppressed count wrong: {cov['suppressed']}") + check(cov["analysis_skipped"] == 1 and cov["analysis_skipped_by"].get("OWN050") == 1, + "OWN050 must be counted as analysis-skipped in coverage") + check("OWN050" not in cov["uncategorized_rules"], + "OWN050 (a coverage note) must not pollute uncategorized rules") + check(cov["uncategorized_rules"].get("FOO999") == 1, + "uncategorized rule FOO999 must be surfaced in coverage exactly once") # a suppressed finding must not leak into the kept category tally - if cov["by_category"].get(0, 0) != 1: # only FOO999 (the DevExpress one is suppressed) - fails.append(f"suppressed finding leaked into kept categories: {cov['by_category']}") + check(cov["by_category"].get(0, 0) == 1, + f"suppressed finding leaked into kept categories: {cov['by_category']}") # severity baseline comes from the category, not the tool level - if tax.severity_for(1) != "P1" or tax.severity_for(0) != "P3": - fails.append("category severity baseline wrong") + check(tax.severity_for(1) == "P1" and tax.severity_for(0) == "P3", + "category severity baseline wrong") # glob specificity: CA2000 (exact, cat 1) must beat CA2* (glob, cat 14) - cat, _ = categorize("CA2000", None, tax.rules) - if cat != 1: - fails.append(f"exact CA2000 should win over CA2* glob: got {cat}") - cat, _ = categorize("CA1822", None, tax.rules) - if cat != 14: - fails.append(f"CA1* glob should map to general-quality (14): got {cat}") - - total = 18 + check(categorize("CA2000", None, tax.rules)[0] == 1, + "exact CA2000 should win over CA2* glob") + check(categorize("CA1822", None, tax.rules)[0] == 14, + "CA1* glob should map to general-quality (14)") + + fails = [c for c in checks if c] for f in fails: print(f"NORMALIZE SELFTEST FAIL: {f}") - print(f"normalize selftest: {total - len(fails)}/{total} checks passed") + print(f"normalize selftest: {len(checks) - len(fails)}/{len(checks)} checks passed") return 1 if fails else 0 diff --git a/audit/aggregate/report.py b/audit/aggregate/report.py index ac0bca4c..4ec72c06 100755 --- a/audit/aggregate/report.py +++ b/audit/aggregate/report.py @@ -81,6 +81,8 @@ def render_markdown(meta: dict[str, Any], cov: dict[str, Any], f"- `{c.path}:{c.line}` **[{c.severity} · {c.category_name}]** " f"— {', '.join(c.tools)}" for c in high[:max_list] ] + if len(high) > max_list: + out.append(f"- … (+{len(high) - max_list} more)") out += ["", f"## Candidates — {len(candidates)} (single tool: unique catch or possible FP)", ""] out += ["_(none)_"] if not candidates else [ @@ -176,7 +178,10 @@ def main(argv: list[str] | None = None) -> int: def _selftest() -> int: tax_path = Path(__file__).resolve().parents[1] / "static" / "taxonomy" / "categories.yml" tax = load_taxonomy(tax_path) - fails: list[str] = [] + checks: list[str] = [] + + def check(ok: bool, msg: str) -> None: # total derives from the call count + checks.append("" if ok else msg) findings = normalize_results([], tax) # start empty, then add concrete ones findings += [ @@ -199,33 +204,26 @@ def _selftest() -> int: for needle in ("# Own.NET Audit — health report", "## Where it hurts most", "## High-confidence findings", "## Candidates", "## Coverage / honesty", "NO-TOOL", "How to read"): - if needle not in md: - fails.append(f"markdown missing section/marker: {needle!r}") - if "third-party: DevExpress." not in md: - fails.append("coverage must report the suppressed DevExpress count") - if "`FOO999`" not in md: - fails.append("coverage must surface the unmapped FOO999 rule") - if "src/Util" not in md: - fails.append("heatmap must list the worst module (src/Util)") + check(needle in md, f"markdown missing section/marker: {needle!r}") + check("third-party: DevExpress." in md, "coverage must report the suppressed DevExpress count") + check("`FOO999`" in md, "coverage must surface the unmapped FOO999 rule") + check("src/Util" in md, "heatmap must list the worst module (src/Util)") # the agreed leak (high-confidence) must be the worst module, ahead of src/Vm util_pos, vm_pos = md.find("`src/Util`"), md.find("`src/Vm`") - if util_pos == -1 or (vm_pos != -1 and util_pos > vm_pos): - fails.append("heatmap ordering: src/Util (agreed leak) must precede src/Vm") + check(util_pos != -1 and not (vm_pos != -1 and util_pos > vm_pos), + "heatmap ordering: src/Util (agreed leak) must precede src/Vm") js = render_json(meta, cov, scored) - if js["meta"]["target"] != "acme/legacy": - fails.append("json lost meta") - if js["totals"]["high_confidence"] != 1: - fails.append(f"json high_confidence wrong: {js['totals']}") - if js["coverage"]["suppressed"] != 1: - fails.append("json coverage lost suppressed count") - if not js["clusters"] or "evidence" not in js["clusters"][0]: - fails.append("json clusters missing evidence") - - total = 13 + check(js["meta"]["target"] == "acme/legacy", "json lost meta") + check(js["totals"]["high_confidence"] == 1, f"json high_confidence wrong: {js['totals']}") + check(js["coverage"]["suppressed"] == 1, "json coverage lost suppressed count") + check(bool(js["clusters"]) and "evidence" in js["clusters"][0], + "json clusters missing evidence") + + fails = [c for c in checks if c] for f in fails: print(f"REPORT SELFTEST FAIL: {f}") - print(f"report selftest: {total - len(fails)}/{total} checks passed") + print(f"report selftest: {len(checks) - len(fails)}/{len(checks)} checks passed") return 1 if fails else 0 diff --git a/audit/aggregate/score.py b/audit/aggregate/score.py index 12dffc5a..7196c113 100755 --- a/audit/aggregate/score.py +++ b/audit/aggregate/score.py @@ -91,7 +91,7 @@ def key(f: AuditFinding) -> tuple[float, int]: def score(findings: list[AuditFinding], tax: Taxonomy, line_tol: int = 3) -> dict[str, Any]: - kept = [f for f in findings if not f.suppressed] + kept = [f for f in findings if f.scored] by_file: dict[str, list[AuditFinding]] = defaultdict(list) for f in kept: by_file[f.fkey].append(f) @@ -200,7 +200,10 @@ def _f(tool: str, path: str, line: int, rule: str, cat: int, name: str, def _selftest() -> int: tax_path = Path(__file__).resolve().parents[1] / "static" / "taxonomy" / "categories.yml" tax = load_taxonomy(tax_path) - fails: list[str] = [] + checks: list[str] = [] + + def check(ok: bool, msg: str) -> None: # total derives from the call count + checks.append("" if ok else msg) findings = [ # two tools at the same spot -> one high-confidence leak cluster (cat 1, P1) @@ -213,48 +216,40 @@ def _selftest() -> int: ] scored = score(findings, tax, line_tol=3) - if scored["totals"]["clusters"] != 3: - fails.append(f"expected 3 clusters, got {scored['totals']['clusters']}") - if scored["totals"]["high_confidence"] != 1: - fails.append( - f"expected 1 high-confidence cluster, got {scored['totals']['high_confidence']}") - if scored["totals"]["candidates"] != 2: - fails.append(f"expected 2 candidates, got {scored['totals']['candidates']}") + check(scored["totals"]["clusters"] == 3, + f"expected 3 clusters, got {scored['totals']['clusters']}") + check(scored["totals"]["high_confidence"] == 1, + f"expected 1 high-confidence cluster, got {scored['totals']['high_confidence']}") + check(scored["totals"]["candidates"] == 2, + f"expected 2 candidates, got {scored['totals']['candidates']}") top = scored["clusters"][0] - if top.module != "src/Util": - fails.append(f"top cluster should be the agreed leak in src/Util, got {top.module}") - if top.confidence != "high": - fails.append("top cluster (highest pain) must be the cross-tool agreement") - if top.tools != ["codeql", "own-check"]: - fails.append(f"agreed cluster tools wrong: {top.tools}") + check(top.module == "src/Util", + f"top cluster should be the agreed leak in src/Util, got {top.module}") + check(top.confidence == "high", "top cluster (highest pain) must be the cross-tool agreement") + check(top.tools == ["codeql", "own-check"], f"agreed cluster tools wrong: {top.tools}") # heatmap orders by pain: the agreed P1 leak module outranks the P2 candidate module pain = {row["module"]: row["pain"] for row in scored["heatmap"]} # src/Util has the high-conf P1 (4*2=8) + a candidate P2 (2*1=2) = 10; # src/Vm has a candidate P1 (4*1=4). So src/Util must outrank src/Vm. - if pain.get("src/Util", 0) <= pain.get("src/Vm", 0): - fails.append(f"heatmap pain ordering wrong: {pain}") - if abs(pain.get("src/Util", 0) - 10.0) > 0.01: - fails.append(f"src/Util pain should be 10.0, got {pain.get('src/Util')}") + check(pain.get("src/Util", 0) > pain.get("src/Vm", 0), f"heatmap pain ordering wrong: {pain}") + check(abs(pain.get("src/Util", 0) - 10.0) <= 0.01, + f"src/Util pain should be 10.0, got {pain.get('src/Util')}") # severity stays category-driven: the subscription leak is P1 even though alone sub = next(c for c in scored["clusters"] if c.module == "src/Vm") - if sub.severity != "P1": - fails.append(f"subscription-leak cluster should be P1, got {sub.severity}") + check(sub.severity == "P1", f"subscription-leak cluster should be P1, got {sub.severity}") js = to_json(scored) - if js["totals"]["clusters"] != 3 or not js["heatmap"]: - fails.append("to_json lost totals/heatmap") - if "evidence" not in js["clusters"][0]: - fails.append("cluster json missing evidence") - - total = 9 - real_fails = [f for f in fails if f] - for f in real_fails: + check(js["totals"]["clusters"] == 3 and bool(js["heatmap"]), "to_json lost totals/heatmap") + check("evidence" in js["clusters"][0], "cluster json missing evidence") + + fails = [c for c in checks if c] + for f in fails: print(f"SCORE SELFTEST FAIL: {f}") - print(f"score selftest: {total - len(real_fails)}/{total} checks passed") - return 1 if real_fails else 0 + print(f"score selftest: {len(checks) - len(fails)}/{len(checks)} checks passed") + return 1 if fails else 0 if __name__ == "__main__": diff --git a/audit/static/inject/OwnAudit.Directory.Build.props b/audit/static/inject/OwnAudit.Directory.Build.props index 59c13579..183d9d7e 100644 --- a/audit/static/inject/OwnAudit.Directory.Build.props +++ b/audit/static/inject/OwnAudit.Directory.Build.props @@ -25,8 +25,12 @@ true All true - - $(MSBuildProjectDirectory)\..\artifacts\own-audit\$(MSBuildProjectName).sarif,version=2.1 + + $(MSBuildProjectDirectory)\..\artifacts\own-audit + $(OwnAuditOutDir)\roslyn\$(MSBuildProjectName).sarif,version=2.1 false false diff --git a/audit/static/inject/OwnAudit.Directory.Build.targets b/audit/static/inject/OwnAudit.Directory.Build.targets index ca72ee1d..9b2addca 100644 --- a/audit/static/inject/OwnAudit.Directory.Build.targets +++ b/audit/static/inject/OwnAudit.Directory.Build.targets @@ -13,6 +13,11 @@ - + + + $(MSBuildProjectDirectory)\..\artifacts\own-audit + + diff --git a/audit/static/run_static.py b/audit/static/run_static.py index 15a06d8f..1cfadc9a 100755 --- a/audit/static/run_static.py +++ b/audit/static/run_static.py @@ -117,12 +117,21 @@ def run(target: str, profile: dict[str, Any], out_dir: Path, target_name: str = sarif_inputs.append(("codeql", st["sarif"])) # Pick up any build-required SARIFs already dropped here by the Windows runners. - for extra, tool in (("roslyn_pack.sarif", "roslyn-pack"), ("infersharp.sarif", "infersharp")): - p = out_dir / extra - if p.exists(): - sarif_inputs.append((tool, str(p))) - tiers.append({"tool": tool, "tier": "build-required", "available": True, - "sarif": str(p), "reason": ""}) + # Roslyn writes ONE SARIF PER PROJECT under roslyn/ (see the injected props's + # $(MSBuildProjectName).sarif), so glob the directory; Infer# writes a single file. + roslyn_dir = out_dir / "roslyn" + roslyn_sarifs = sorted(roslyn_dir.glob("*.sarif")) if roslyn_dir.is_dir() else [] + for p in roslyn_sarifs: + sarif_inputs.append(("roslyn-pack", str(p))) + if roslyn_sarifs: + tiers.append({"tool": "roslyn-pack", "tier": "build-required", "available": True, + "sarif": f"{len(roslyn_sarifs)} project SARIF(s) under roslyn/", + "reason": ""}) + infer = out_dir / "infersharp.sarif" + if infer.exists(): + sarif_inputs.append(("infersharp", str(infer))) + tiers.append({"tool": "infersharp", "tier": "build-required", "available": True, + "sarif": str(infer), "reason": ""}) meta = { "target": target_name or target, "commit": commit, @@ -195,7 +204,11 @@ def _fixture_sarifs(tmp: Path) -> list[tuple[str, str]]: def _selftest() -> int: import tempfile - fails: list[str] = [] + checks: list[str] = [] + + def check(ok: bool, msg: str) -> None: # total derives from the call count + checks.append("" if ok else msg) + with tempfile.TemporaryDirectory() as td: tmp = Path(td) inputs = _fixture_sarifs(tmp) @@ -204,35 +217,45 @@ def _selftest() -> int: result = aggregate(inputs, tmp / "report", meta) # the agreed Io.cs leak is high-confidence; Customer.cs subscription is a candidate - if result["totals"]["high_confidence"] != 1: - fails.append(f"expected 1 high-confidence cluster, got {result['totals']}") - if result["totals"]["candidates"] != 1: - fails.append(f"expected 1 candidate, got {result['totals']}") + check(result["totals"]["high_confidence"] == 1, + f"expected 1 high-confidence cluster, got {result['totals']}") + check(result["totals"]["candidates"] == 1, f"expected 1 candidate, got {result['totals']}") md = Path(result["report_md"]).read_text(encoding="utf-8") - if "# Own.NET Audit — health report" not in md: - fails.append("report.md missing title") - if "## Where it hurts most" not in md or "## Coverage / honesty" not in md: - fails.append("report.md missing a required section") - if "third-party: DevExpress." not in md: - fails.append("report.md must report the suppressed DevExpress finding") + check("# Own.NET Audit — health report" in md, "report.md missing title") + check("## Where it hurts most" in md and "## Coverage / honesty" in md, + "report.md missing a required section") + check("third-party: DevExpress." in md, + "report.md must report the suppressed DevExpress finding") # src/Util (agreed leak) must outrank src/Vm (lone subscription) in the heatmap - if md.find("`src/Util`") == -1 or (md.find("`src/Vm`") != -1 - and md.find("`src/Util`") > md.find("`src/Vm`")): - fails.append("heatmap ordering: src/Util must precede src/Vm") + check(md.find("`src/Util`") != -1 and not (md.find("`src/Vm`") != -1 + and md.find("`src/Util`") > md.find("`src/Vm`")), + "heatmap ordering: src/Util must precede src/Vm") js = json.loads(Path(result["report_json"]).read_text(encoding="utf-8")) - if js["coverage"]["suppressed"] != 1: - fails.append("report.json coverage lost the suppressed count") - if js["meta"]["target"] != "acme/legacy": - fails.append("report.json lost meta") - if not (tmp / "report" / "report.md").exists(): - fails.append("aggregate did not write report.md to disk") - - total = 8 + check(js["coverage"]["suppressed"] == 1, "report.json coverage lost the suppressed count") + check(js["meta"]["target"] == "acme/legacy", "report.json lost meta") + check((tmp / "report" / "report.md").exists(), "aggregate did not write report.md to disk") + + # Roslyn build-required tier writes one SARIF PER PROJECT under roslyn/; run() + # must glob the directory, not a single fixed filename (Codex review on #100). + with tempfile.TemporaryDirectory() as td2: + out2 = Path(td2) + (out2 / "roslyn").mkdir() + rosl = {"runs": [{"results": [{"ruleId": "CA2000", "message": {"text": "undisposed"}, + "locations": [{"physicalLocation": {"artifactLocation": {"uri": "src/App/Svc.cs"}, + "region": {"startLine": 5}}}]}]}]} + (out2 / "roslyn" / "ProjA.sarif").write_text(json.dumps(rosl), encoding="utf-8") + profile = {"name": "t", "severity_floor": "warning", "tiers": {"build_free": []}} + res2 = run("/nonexistent-target", profile, out2, target_name="t/p") + check(any(t["tool"] == "roslyn-pack" for t in res2["tiers"]), + "roslyn per-project SARIF under roslyn/ must be picked up") + check(res2["totals"]["clusters"] >= 1, "roslyn finding must reach the aggregated totals") + + fails = [c for c in checks if c] for f in fails: print(f"RUN_STATIC SELFTEST FAIL: {f}") - print(f"run_static selftest: {total - len(fails)}/{total} checks passed") + print(f"run_static selftest: {len(checks) - len(fails)}/{len(checks)} checks passed") return 1 if fails else 0 diff --git a/audit/static/taxonomy/categories.yml b/audit/static/taxonomy/categories.yml index bf021928..d51c03ae 100644 --- a/audit/static/taxonomy/categories.yml +++ b/audit/static/taxonomy/categories.yml @@ -75,3 +75,12 @@ category_severity: suppress: path_or_message_contains: - "DevExpress." + +# Analysis-skipped / coverage NOTES — diagnostics that are NOT verdicts. own-check +# emits OWN050 ("cannot verify X — unresolved") at SARIF level: note when it cannot +# analyze something. These must NOT be scored as findings: left in, they would show +# up as uncategorized P3 candidates and add phantom pain. Instead they are routed to +# the coverage ledger (analysis-skipped count) — the honest "we didn't see here", +# the same discipline as own-check's own OWN050. +coverage_notes: + - "OWN050" diff --git a/audit/static/tools/infersharp.sh b/audit/static/tools/infersharp.sh index d1a035f1..edb70341 100755 --- a/audit/static/tools/infersharp.sh +++ b/audit/static/tools/infersharp.sh @@ -33,14 +33,23 @@ done [[ -n "$bin" ]] || { echo "infersharp.sh: --bin (built output with .dll+.pdb) is required" >&2; exit 2; } -if ! command -v infersharp >/dev/null 2>&1 && ! command -v infersharpaction >/dev/null 2>&1; then - echo "NO-TOOL: Infer# CLI not available — skipping the build-required Infer# tier" >&2 +# Require the infersharp CLI specifically. `microsoft/infersharpaction` is a GitHub +# Action (inputs: binary-path / github-sarif), NOT a shell CLI with this interface, +# so it is not a drop-in fallback — in a workflow, use that Action directly. +if ! command -v infersharp >/dev/null 2>&1; then + echo "NO-TOOL: Infer# CLI 'infersharp' not on PATH — skipping the build-required Infer# tier." >&2 + echo " (In a GitHub Action, use microsoft/infersharpaction with binary-path instead.)" >&2 exit 3 fi +bin_abs="$(cd "$bin" 2>/dev/null && pwd)" \ + || { echo "infersharp.sh: --bin directory '$bin' not found" >&2; exit 2; } + mkdir -p "$out" -# Infer# writes report.sarif into its output dir; copy it to the audit artifacts. -infersharp "$bin" --sarif --results-dir "$out/infer-out" +# Infer# always writes its SARIF to infer-out/report.sarif relative to the working +# directory (microsoft/infersharp), so run it from $out, then copy the canonical +# report into the audit artifacts directory. +( cd "$out" && infersharp "$bin_abs" ) cp "$out/infer-out/report.sarif" "$out/infersharp.sarif" echo "infersharp.sh: wrote $out/infersharp.sarif" diff --git a/audit/static/tools/owncheck.py b/audit/static/tools/owncheck.py index ee0b7340..8836b547 100755 --- a/audit/static/tools/owncheck.py +++ b/audit/static/tools/owncheck.py @@ -52,7 +52,11 @@ def run_own_check(target: str, out_dir: Path, severity: str = "warning", cmd = [str(OWN_CHECK_SH), "--format", "sarif", "--severity", severity, "--", target] if root is not None: cmd[1:1] = ["--root", str(root)] - proc = subprocess.run(cmd, capture_output=True, text=True, check=False) + try: + proc = subprocess.run(cmd, capture_output=True, text=True, check=False, timeout=900) + except subprocess.TimeoutExpired: + status["reason"] = "own-check timed out (>900s) — partial/unavailable result" + return status sarif_text = proc.stdout.strip() if not sarif_text.startswith("{"): status["reason"] = (f"own-check did not emit SARIF (exit {proc.returncode}): " diff --git a/audit/static/tools/roslyn_pack.ps1 b/audit/static/tools/roslyn_pack.ps1 index a5711f92..516c7427 100644 --- a/audit/static/tools/roslyn_pack.ps1 +++ b/audit/static/tools/roslyn_pack.ps1 @@ -43,14 +43,18 @@ if (-not (Get-Command msbuild -ErrorAction SilentlyContinue)) { exit 3 } -New-Item -ItemType Directory -Force -Path $Out | Out-Null +# Resolve -Out to an absolute path so the per-project ErrorLog (which the injected +# props anchors at $(OwnAuditOutDir)) lands here, not next to each project dir. +$OutFull = (New-Item -ItemType Directory -Force -Path $Out).FullName # continue-on-error: a failed build still yields whatever per-project SARIFs were # produced before the failure - a partial, honest report, not an empty one. +# /p:OwnAuditOutDir makes the injected props write SARIF under $OutFull\roslyn\. msbuild $Solution ` /p:OwnAudit=true ` /p:OwnAuditAnalyzers=$AnalyzerCache ` + /p:OwnAuditOutDir=$OutFull ` /p:Configuration=Release ` - /bl:"$Out\build.binlog" + /bl:"$OutFull\build.binlog" -Write-Host "roslyn_pack.ps1: per-project SARIF under $Out (merged by audit/aggregate/)." +Write-Host "roslyn_pack.ps1: per-project SARIF under $OutFull\roslyn (merged by audit/aggregate/)." From df08418cd48069841335e0938c36871e7c336ba2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 24 Jun 2026 15:50:18 +0000 Subject: [PATCH 3/3] Guard --bin/--out/--target/--db against missing values in shell runners CodeRabbit (PR #100): infersharp.sh read $2 for --bin/--out without checking a value was supplied, so under `set -u` a flag with no argument aborted with a raw shell error instead of the intended usage exit. Added the `[[ $# -ge 2 ]]` guard that own-check.sh already uses, and applied the same to codeql.sh (--target/ --out/--db) which had the identical pattern. Each missing value now exits 2 with a clear message. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01QDPpNT9Uh8RoTrKPvgcoRE --- audit/static/tools/codeql.sh | 9 ++++++--- audit/static/tools/infersharp.sh | 6 ++++-- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/audit/static/tools/codeql.sh b/audit/static/tools/codeql.sh index 7be78d3c..8e476af1 100755 --- a/audit/static/tools/codeql.sh +++ b/audit/static/tools/codeql.sh @@ -23,9 +23,12 @@ db="" while [[ $# -gt 0 ]]; do case "$1" in - --target) target="$2"; shift 2 ;; - --out) out="$2"; shift 2 ;; - --db) db="$2"; shift 2 ;; + --target) [[ $# -ge 2 ]] || { echo "codeql.sh: --target requires a value" >&2; exit 2; } + target="$2"; shift 2 ;; + --out) [[ $# -ge 2 ]] || { echo "codeql.sh: --out requires a value" >&2; exit 2; } + out="$2"; shift 2 ;; + --db) [[ $# -ge 2 ]] || { echo "codeql.sh: --db requires a value" >&2; exit 2; } + db="$2"; shift 2 ;; -h|--help) sed -n '2,18p' "$0"; exit 0 ;; *) echo "codeql.sh: unknown arg $1" >&2; exit 2 ;; esac diff --git a/audit/static/tools/infersharp.sh b/audit/static/tools/infersharp.sh index edb70341..d3541d1c 100755 --- a/audit/static/tools/infersharp.sh +++ b/audit/static/tools/infersharp.sh @@ -24,8 +24,10 @@ out="artifacts/own-audit" while [[ $# -gt 0 ]]; do case "$1" in - --bin) bin="$2"; shift 2 ;; - --out) out="$2"; shift 2 ;; + --bin) [[ $# -ge 2 ]] || { echo "infersharp.sh: --bin requires a value" >&2; exit 2; } + bin="$2"; shift 2 ;; + --out) [[ $# -ge 2 ]] || { echo "infersharp.sh: --out requires a value" >&2; exit 2; } + out="$2"; shift 2 ;; -h|--help) sed -n '2,18p' "$0"; exit 0 ;; *) echo "infersharp.sh: unknown arg $1" >&2; exit 2 ;; esac