From 220502241fdd1df86f83963433a0f17f60af6377 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 02:10:58 +0000 Subject: [PATCH 1/9] =?UTF-8?q?docs(proposal):=20P-022=20Rust=20core=20mig?= =?UTF-8?q?ration=20=E2=80=94=20bird's-eye=20architecture?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design-only proposal for porting the Python core to a Rust workspace, keeping the Python repo as the differential oracle / golden test. Covers: crate topology as an enforced acyclic DAG (decoupled across the OwnIR seam), Rust idioms that remove the current accidental complexity (enum+match vs assert_never, newtype IDs vs id(sym), persistent state vs deep copy, Lattice/Analysis traits vs the interleaved _Analyzer), library candidates, prior art to study (rust-analyzer, ruff, rustc_mir_dataflow, Polonius, ...), Rust architecture-fitness tooling (crate-graph fitness test, cargo-modules/deny/machete/public-api), and the differential-oracle design that rides the already-frozen evidence/SARIF/JSON contracts. No Rust code yet; Python stays authoritative until parity holds. --- docs/proposals/P-022-rust-core-migration.md | 238 ++++++++++++++++++++ docs/proposals/README.md | 1 + 2 files changed, 239 insertions(+) create mode 100644 docs/proposals/P-022-rust-core-migration.md diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md new file mode 100644 index 00000000..9dda21f3 --- /dev/null +++ b/docs/proposals/P-022-rust-core-migration.md @@ -0,0 +1,238 @@ +# P-022 — Rust core migration: bird's-eye architecture + +Status: **draft / exploratory** (design only — no Rust code committed yet; the +Python core stays the reference implementation and the oracle until parity holds) + +## Why + +The Python core (`ownlang/`) is a working PoC, but its density is a maintenance +tax: a mutable-dict dataflow state keyed by `id(sym)`, an `assert_never` dispatch +that must be hand-updated across three files, and analyses (ownership / lifetime / +effect / DI) interleaved in one `_Analyzer`. The intended end state is a **Rust +core**. This document is the bird's-eye plan: crate topology, patterns, libraries, +prior art, architecture-fitness tooling, and — the load-bearing piece — a +**differential oracle** that pins the Rust core to the Python one output-for-output. + +The recent evidence + SARIF work (P-015, execution-surfaces ADR) is not incidental +here: it turned the core's verdicts into **normalized, diffable contracts** +(sorted diagnostics, SARIF `codeFlows`/`relatedLocations`, `.ownreport.json`). Those +contracts are exactly what a differential oracle compares. The migration rides on +them. + +## Goals / non-goals + +**Goals** +- A Rust workspace whose **crate graph is the architecture** — an enforced acyclic + dependency DAG, high cohesion per crate, decoupled across the OwnIR seam. +- **Behavioural parity** with the Python core on the existing corpus, proven by a + differential oracle in CI (the Python repo is the golden test). +- Idiomatic Rust that removes the Python accidental complexity (enums + exhaustive + `match` instead of `assert_never`; newtype IDs instead of `id(sym)`; persistent + state instead of deep `copy()`). + +**Non-goals (for this phase)** +- No Datalog/Ascent rewrite of the rule layer yet. That is a *later* strategy the + Rust move unlocks (execution-surfaces ADR §8 trigger), not part of the port. +- No reimplementation of the frontends. The Roslyn (C#) extractor and OwnTS (TS) + stay put — they emit **OwnIR** facts; the Rust core consumes OwnIR exactly as the + Python core does. OwnIR is the seam, not a thing to rewrite. +- No behaviour changes. If the Rust core disagrees with Python, Python wins until a + divergence is a deliberate, separately-justified change. + +## The seam: what stays a contract + +Three contracts must not drift during the port — they are the oracle's comparison +surface and the frontend boundary: + +1. **OwnIR** (`ownlang/ownir.py`): the versioned JSON fact schema frontends emit and + the core consumes. Rust re-types it with `serde`; the JSON on the wire is + identical. Additive optional fields tolerated, vocabulary changes fail loudly — + same rule as today. +2. **Diagnostic + Evidence** (`ownlang/diagnostics.py`): code, line, subject, + resource_kind, and the ordered evidence slice. This is the verdict contract. +3. **Output surfaces**: text (`render`/`render_pretty`), SARIF 2.1.0 + (`relatedLocations`/`codeFlows`), `.ownreport.json`, and emitted C# (`emit`). + Normalized, these are byte-comparable across implementations. + +## Crate topology (decoupled = the DAG; Cargo enforces acyclicity at compile time) + +``` + own-ir (OwnIR types + serde; the fact/verdict contract — leaf-ish) + ▲ ▲ + │ └────────────────────────┐ + own-syntax ───┤ │ + (lexer/AST) ▼ │ + own-cfg ──▶ own-analysis ──▶ own-diagnostics ──▶ own-codegen + (lower) (lattice + (Diagnostic/ (C# emit) + worklist + Evidence model, + ownership/ text + SARIF + lifetime/ projection) + effect/DI) │ + ▼ + own-cli (check/emit/cfg/report/ownir/explain) + ▲ + own-oracle ──────┘ (dev/test: differential harness vs Python) +``` + +- **`own-syntax`** — lexer + parser + AST. Zero analysis knowledge. (Prior art: + ruff / rust-analyzer parsers.) +- **`own-ir`** — OwnIR fact types, `serde` (de)serialization, schema version. Shared + by the (future) frontends-in-Rust and the core; kept dependency-light so both + sides can depend on it without pulling the analysis. +- **`own-cfg`** — AST → CFG lowering; CFG/Instr types. The `assert_never` sites + become exhaustive `match` here. +- **`own-analysis`** — the heart: a generic worklist solver over `Lattice` + + `Analysis` traits, and the ownership/loan/lifetime/effect/DI impls. Each analysis + is an independent trait impl, not interleaved (this is the direct de-noodling of + today's `_Analyzer`). +- **`own-diagnostics`** — Diagnostic/Evidence model + presentation (human render, + SARIF projection). A pure consumer of verdict data; knows nothing about the + solver internals. Mirrors today's already-clean split (`evidence.py` is a pure + projection). +- **`own-codegen`** — C# emission from CFG + analysis results. +- **`own-cli`** — the binary; wires the pipeline and owns the CLI surface. +- **`own-oracle`** — dev-only differential harness (see below). + +The dependency arrows only point rightward/inward; Cargo makes a cycle a compile +error, so the layering is not a convention you can accidentally violate — it is the +build graph. + +## Patterns / idioms (what removes the Python accidental complexity) + +| Python pain today | Rust idiom | Payoff | +| --- | --- | --- | +| `assert_never` dispatch updated by hand in cfg/analysis/codegen | `enum` + exhaustive `match` | a missed variant is a **compile error**, not a runtime assert | +| RID = `id(sym)`; handles keyed by object identity | newtype indices `Rid(u32)`, `BlockId(u32)`, `LoanId(u32)` + arena | deterministic, serializable, no identity hacks | +| AST/CFG as object graphs | **arena + indices** (`la-arena`/`id-arena`) | borrow-checker-friendly graphs, cheap clone, cache-friendly | +| `State.copy()` deep-copies dicts every merge | **persistent maps** (`imbl`/`rpds`) — structural sharing | join/clone at merges is O(log n) sharing, not a deep copy | +| ownership/lifetime/effect/DI interleaved in `_Analyzer` | `Lattice` + `DataflowAnalysis` traits; one generic solver, N impls | analyses decoupled + independently testable | +| strings compared everywhere | interning (`lasso`/`string-interner`) | cheap symbol equality; smaller state | +| Diagnostics-as-data (good, keep it) | keep: `Diagnostic` is data, **not** an `Err` | verdicts stay first-class; `thiserror`/`anyhow` only for real I/O errors | + +Direction of the analysis (forward worklist to fixpoint, union at merge, monotone +transfer over a finite lattice) ports directly — it is textbook `rustc_mir_dataflow` +shape. + +## Libraries (candidates — pin/validate maintenance at build time) + +| Concern | Candidate(s) | Notes | +| --- | --- | --- | +| Lexer | `logos` | derive-based, fast; or hand-roll to match Python tokens exactly | +| Parser | hand-written recursive descent (recommended) | matches Python error recovery / golden parity; `chumsky`/`winnow` if we want combinators | +| Arena / IDs | `la-arena` (rust-analyzer), `id-arena`, `slotmap` | index-based AST/CFG | +| Interning | `lasso`, `string-interner` | symbol/string interning | +| Graph (CFG) | `petgraph` or hand-rolled arena | petgraph gives traversals/dominators for free | +| Persistent state | `imbl` (im fork) or `rpds` | copy-on-write dataflow state | +| Serialization | `serde` + `serde_json` | OwnIR, SARIF, `.ownreport.json` | +| SARIF types | `serde-sarif` (evaluate) or hand-rolled structs | we already emit the shape; typed structs prevent drift | +| CLI | `clap` (derive) | mirror the current subcommand surface | +| Errors | `thiserror` (libs) + `anyhow` (bin) | diagnostics are NOT errors | +| Snapshot tests | `insta` | the Rust equivalent of the golden tests | +| Property tests | `proptest` | replaces the Python codegen fuzzer | +| Datalog (later) | `ascent`, `datafrog`, `crepe` | only if/when the rule layer goes declarative | + +## Prior art to study (architecture references) + +| Project | What to steal | +| --- | --- | +| **rust-analyzer** | arena + interning, hand-written error-recovering parser, `salsa` incremental, crate split | +| **ruff** (Astral) | Python-linter-in-Rust: AST visitor, **rule registry**, diagnostics + fixes, `insta` snapshots, workspace layout — the closest analogue to what we're building | +| **rustc `rustc_mir_dataflow`** | the `Analysis`/lattice/worklist framework — our ownership dataflow is the same shape | +| **Polonius** | borrow-check-as-Datalog (uses `datafrog`) — the reference if we later take facts/relations → Datalog | +| **Prusti / Flux / Creusot / MIRAI** | pass architecture for Rust static verification | +| **oxc / biome** | JS/TS toolchains in Rust: crate splitting, diagnostics, codegen, perf | +| **salsa** | incremental recomputation, if we want editor/on-save later | + +The **ruff rule registry** is worth calling out: it is the execution-surfaces ADR §2 +"typed primitive/detector registry" done idiomatically in Rust — so that ADR idea is +better realized *after* the port, natively, than bolted onto Python now. + +## Architecture-fitness tooling for Rust (the "architectural analyzers") + +There is no single "ArchUnit for Rust", but the crate DAG plus a few tools give +strong, CI-enforceable decoupling: + +- **The crate graph itself** — cyclic crate deps are a Cargo compile error, so the + layering is enforced for free. Add a tiny test that parses `cargo metadata` and + asserts the *allowed* edge set (a poor-man's ArchUnit — fails if, say, + `own-diagnostics` ever grows a dep on `own-analysis`). +- **`cargo-modules`** — visualize/asserts module + crate structure and dependencies; + catches orphans and unexpected edges. +- **`cargo-deny`** — dependency policy (licenses, bans, duplicate versions, + advisories) — supply-chain + hygiene gate. +- **`cargo-machete`** — unused dependency detector. +- **`cargo-public-api`** — track each crate's public surface so coupling can't leak + in through accidentally-`pub` internals; diff it in CI. +- **Clippy** (pedantic/nursery selectively) + `#![warn(missing_docs)]` + strict + module privacy (`pub(crate)` by default). + +Fitness function to encode early: **`own-diagnostics` and `own-ir` must not depend on +`own-analysis`** (the verdict/contract layer stays independent of the solver). One +`cargo metadata` test locks that. + +## The differential oracle (Python repo = golden test) + +This is the spine of the migration — it lets us port incrementally with confidence. + +**Principle:** compare on the **stable output contracts**, never on internal state. +Both implementations already sort diagnostics by `(line, code)`; both emit the same +SARIF/JSON shapes. So: + +- **Inputs:** the existing `corpus/`, `examples/`, `tests/fixtures/`, golden `.own` + files, and OwnIR fact files. Plus generated inputs (see below). +- **Run both:** `python -m ownlang check --format sarif ` vs + `own-rs check --format sarif ` (identical CLI surface + SARIF ⇒ clean diff). + Likewise `cfg` (dump CFG as JSON), `report` (`.ownreport.json`), `emit` (C#). +- **Normalize then diff:** canonicalize JSON (sorted keys), normalize file paths, + drop volatile fields; compare the diagnostic set (code, line, subject, evidence + steps) and the SARIF log. A per-file, per-diagnostic divergence report — the same + shape `scripts/oracle_compare.py` already produces for the cross-tool (Infer#/ + CodeQL) diff. **The Rust core is just another SARIF producer that the existing + oracle diffs against Python.** +- **CI ratchet:** a job runs the diff over the corpus and fails on any divergence in + the covered subset. Coverage starts small and grows; the ratchet only tightens. +- **Generative differential + metamorphic:** reuse the codegen fuzzer's spirit — + generate random `.own`/OwnIR, run both, diff. Metamorphic relations (e.g. renaming + a symbol must not change the diagnostic set) catch classes of bugs a fixed corpus + misses. + +Because the oracle compares *contracts we already froze and tested*, the recent +evidence/SARIF hardening is what makes it cheap. Each layer also gets its own diff +seam (`cfg` JSON at the CFG layer, SARIF at the verdict layer), so we can bisect a +divergence to the crate that introduced it. + +## Migration strategy (strangler-fig, bottom-up, oracle-gated) + +1. **Stand up the workspace + `own-ir`** (serde round-trips the existing OwnIR + fixtures — first parity check, at the seam). +2. **`own-syntax`**: port the parser; diff the AST/`cfg` dump against Python. +3. **`own-cfg`**: port lowering; diff CFG JSON. +4. **`own-analysis`**: port the worklist + ownership first, then lifetime/effect/DI; + diff diagnostics (no evidence) → then evidence → then SARIF, layer by layer. +5. **`own-diagnostics` + `own-codegen`**: SARIF/report/text and C# `emit`; diff each. +6. **`own-cli`**: cut over once corpus parity is ~100%. Keep Python frozen as the + oracle/spec. +7. **Only then** revisit the rule layer as Datalog/Ascent (ADR §8: "core moves to + Rust" trigger now satisfied) — natively, not as a Python detour. + +Throughout, Python stays authoritative; the Rust crates light up behind the ratchet. + +## Open questions (to resolve as we go) + +- **Parser**: hand-roll vs `chumsky`/`winnow`. Leaning hand-roll for golden-parity of + error messages. +- **Persistent vs arena-CoW state**: `imbl` maps vs an arena with copy-on-write. Bench + on the corpus before committing. +- **`serde-sarif` vs hand-rolled SARIF structs**: evaluate the crate's maintenance and + whether it matches our exact shape. +- **Incrementality (`salsa`)**: out of scope for parity, but the crate split should not + preclude it (keep queries pure). +- **Repo layout**: a `rust/` subtree in this repo (monorepo, easiest for the oracle to + run both) vs a sibling repo. Monorepo recommended so the oracle and corpus are one + `git` away. + +## Placement + +This document lives in `PhysShell/Own.NET/docs/proposals/`. It is design-only; the +first code deliverable is the workspace skeleton + `own-ir` round-trip + the oracle +harness, on its own branch, gated by the differential ratchet from commit one. diff --git a/docs/proposals/README.md b/docs/proposals/README.md index 703c2d15..0919b313 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -41,6 +41,7 @@ proposal is marked `done` with a pointer. | [P-017](P-017-multi-stack-frontends.md) | Multi-stack frontends (OwnTS / OwnJVM: OwnJava + OwnKotlin) | draft | | [P-020](P-020-ownts-react-effects.md) | OwnTS React effects profile (`Own.React`) — the effect-storm angle | draft | | [P-021](P-021-async-audit-pack.md) | Async audit pack (`Own.Async`) | draft | +| [P-022](P-022-rust-core-migration.md) | Rust core migration: crate DAG, patterns, prior art, differential oracle (Python = golden) | draft / exploratory | > For priorities, milestones, the framing, and the design philosophy across all > of these, see the strategy hub: [`docs/ROADMAP.md`](../ROADMAP.md). P-004 … P-016 From e2e07211a514ec32a7caba41f0d6e7b88d4f0a6e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 02:20:51 +0000 Subject: [PATCH 2/9] docs(P-022): fold CodeRabbit + Codex architecture review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both reviewers converged; all corrections are factual/design refinements (no forks): - Crate DAG: own-codegen is a SIBLING of own-diagnostics (both consume own-cfg/own-analysis), not chained through it. Codegen is verdict-independent (AST/CFG-driven, matching Python codegen.generate), so the old own-diagnostics -> own-codegen edge would have forced diagnostics to re-export solver internals. Added a fitness function locking codegen !-> diagnostics (and !-> analysis for now). (Codex P2, CodeRabbit #1) - Oracle: do NOT reuse scripts/oracle_compare.py as the parity oracle — it is cross-tool fuzzy (leak-only, +-N line tolerance, coarse severity) and would mask off-by-one/label/subject/exit-status divergences. Spec a new exact harness over status+stdout+stderr+SARIF/JSON, with an exit/crash gate first, exact set-equality incl. evidence label text, and intra-tie ordering. (Codex P1, CodeRabbit #4) - CFG seam does not exist yet: python cfg prints human text, not JSON. Add+freeze a canonical cfg --format json before the ratchet uses that seam; added as migration step 0. (Codex P2) - State: arena+CoW likely wins this procedural workload; bench largest real function, wall-clock + RSS. Prior art: clippy lint-pass registry, prusti-viper encoding boundary. Repo-layout revisit trigger. (CodeRabbit #2/#3/#5/#6) --- docs/proposals/P-022-rust-core-migration.md | 131 ++++++++++++++------ 1 file changed, 92 insertions(+), 39 deletions(-) diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md index 9dda21f3..9cbebd55 100644 --- a/docs/proposals/P-022-rust-core-migration.md +++ b/docs/proposals/P-022-rust-core-migration.md @@ -57,23 +57,30 @@ surface and the frontend boundary: ## Crate topology (decoupled = the DAG; Cargo enforces acyclicity at compile time) ``` - own-ir (OwnIR types + serde; the fact/verdict contract — leaf-ish) - ▲ ▲ - │ └────────────────────────┐ - own-syntax ───┤ │ - (lexer/AST) ▼ │ - own-cfg ──▶ own-analysis ──▶ own-diagnostics ──▶ own-codegen - (lower) (lattice + (Diagnostic/ (C# emit) - worklist + Evidence model, - ownership/ text + SARIF - lifetime/ projection) - effect/DI) │ - ▼ - own-cli (check/emit/cfg/report/ownir/explain) - ▲ - own-oracle ──────┘ (dev/test: differential harness vs Python) + own-ir (OwnIR fact/verdict contract; serde — leaf-ish, everyone may depend on it) + own-syntax (lexer / parser / AST) + └─▶ own-cfg (AST → CFG lowering) + └─▶ own-analysis (lattice + worklist; ownership / lifetime / effect / DI) + + own-cfg + own-analysis ─┬─▶ own-diagnostics (Diagnostic/Evidence model, text + SARIF) + └─▶ own-codegen (C# emit — AST/CFG-driven, verdict-independent) + + {all of the above} ─▶ own-cli (check / emit / cfg / report / ownir / explain) + own-cli ◀─ own-oracle (dev/test: differential harness vs Python) ``` +`own-diagnostics` and `own-codegen` are **sibling consumers** of `own-cfg` / +`own-analysis` — neither depends on the other. This matters: codegen must not chain +*through* the verdict renderer (see the fitness function below and Codex's review), +or it would need diagnostics to re-export solver internals — breaking the "diagnostics +knows nothing about the solver" invariant. Today's Python `codegen.generate(mod)` +takes only the AST and never imports `diagnostics`, so codegen is **verdict-independent** +(its policy comes from AST/CFG shape — `_laminar_scopes`, `_buffer_modes`, …), not from +analysis conclusions. Keep it that way; if Rust codegen ever *does* want a solver +verdict (e.g. inserting a `Dispose()` from the ownership conclusion rather than +re-deriving it from shape), that is a **deliberate new `own-codegen → own-analysis` +edge** to flag on its own, not something that sneaks in via the diagnostics arrow. + - **`own-syntax`** — lexer + parser + AST. Zero analysis knowledge. (Prior art: ruff / rust-analyzer parsers.) - **`own-ir`** — OwnIR fact types, `serde` (de)serialization, schema version. Shared @@ -89,7 +96,9 @@ surface and the frontend boundary: SARIF projection). A pure consumer of verdict data; knows nothing about the solver internals. Mirrors today's already-clean split (`evidence.py` is a pure projection). -- **`own-codegen`** — C# emission from CFG + analysis results. +- **`own-codegen`** — C# emission, **driven by AST/CFG shape, not analysis verdicts** + (verdict-independent, matching Python `codegen.generate(mod)`). A sibling of + `own-diagnostics`, not downstream of it. - **`own-cli`** — the binary; wires the pipeline and owns the CLI surface. - **`own-oracle`** — dev-only differential harness (see below). @@ -138,14 +147,16 @@ shape. | **rust-analyzer** | arena + interning, hand-written error-recovering parser, `salsa` incremental, crate split | | **ruff** (Astral) | Python-linter-in-Rust: AST visitor, **rule registry**, diagnostics + fixes, `insta` snapshots, workspace layout — the closest analogue to what we're building | | **rustc `rustc_mir_dataflow`** | the `Analysis`/lattice/worklist framework — our ownership dataflow is the same shape | +| **clippy lint-pass registry** | a *second-party* lint layer bolted onto a compiler's MIR — arguably a closer analogue to our OWN-code registry (analyses atop external-frontend OwnIR facts) than ruff's own | | **Polonius** | borrow-check-as-Datalog (uses `datafrog`) — the reference if we later take facts/relations → Datalog | -| **Prusti / Flux / Creusot / MIRAI** | pass architecture for Rust static verification | +| **Prusti / Flux / Creusot / MIRAI** | pass architecture for Rust static verification; **prusti-viper**'s MIR→Viper *encoding boundary* is a model for seaming a future verification backend (P-002) onto `own-ir` without touching `own-analysis` | | **oxc / biome** | JS/TS toolchains in Rust: crate splitting, diagnostics, codegen, perf | | **salsa** | incremental recomputation, if we want editor/on-save later | -The **ruff rule registry** is worth calling out: it is the execution-surfaces ADR §2 -"typed primitive/detector registry" done idiomatically in Rust — so that ADR idea is -better realized *after* the port, natively, than bolted onto Python now. +The **ruff / clippy rule registries** are worth calling out: they are the +execution-surfaces ADR §2 "typed primitive/detector registry" done idiomatically in +Rust — so that ADR idea is better realized *after* the port, natively, than bolted +onto Python now. ## Architecture-fitness tooling for Rust (the "architectural analyzers") @@ -166,9 +177,14 @@ strong, CI-enforceable decoupling: - **Clippy** (pedantic/nursery selectively) + `#![warn(missing_docs)]` + strict module privacy (`pub(crate)` by default). -Fitness function to encode early: **`own-diagnostics` and `own-ir` must not depend on -`own-analysis`** (the verdict/contract layer stays independent of the solver). One -`cargo metadata` test locks that. +Fitness functions to encode early, each locked by one `cargo metadata` test over the +allowed edge set: +- **`own-diagnostics` and `own-ir` must not depend on `own-analysis`** — the + verdict/contract layer stays independent of the solver. +- **`own-codegen` must not depend on `own-diagnostics`** (they are siblings), and — + for now — not on `own-analysis` either (codegen is verdict-independent). A future + `own-codegen → own-analysis` edge is allowed only as a deliberate, reviewed change + to the allowed set, never implicitly. ## The differential oracle (Python repo = golden test) @@ -182,13 +198,30 @@ SARIF/JSON shapes. So: files, and OwnIR fact files. Plus generated inputs (see below). - **Run both:** `python -m ownlang check --format sarif ` vs `own-rs check --format sarif ` (identical CLI surface + SARIF ⇒ clean diff). - Likewise `cfg` (dump CFG as JSON), `report` (`.ownreport.json`), `emit` (C#). + Likewise `report` (`.ownreport.json`) and `emit` (C#). +- **Exit/crash gate first.** Before diffing *any* output, assert both binaries + produced the same exit status and neither panicked — a Rust panic has no SARIF + representation, so an output-only diff would score a crash-on-valid-input as + "no findings = parity". Diff `stderr` too (e.g. the P-014 OWN050 advisory notes + live there, not in SARIF). +- **Exact, not fuzzy.** This is a **golden** same-input comparison, so it demands + strict set-equality on `(path, line, code, subject)` plus the full evidence slice + **including each step's label text** (a step on the right line with the wrong + `role`/label is a real semantic bug SARIF-location-diffing alone misses). Also pin + intra-tie ordering when two findings share `(line, code)`. + > **Do not reuse `scripts/oracle_compare.py` as the parity oracle.** It was built + > for a *different* job — cross-tool fuzzy matching against external tools (Infer#/ + > CodeQL): it keeps only leak-class findings, matches by basename within a ±N-line + > tolerance (`near()`), and buckets severities coarsely. Reused as-is it would let + > an off-by-one in Rust CFG lowering, a changed evidence label, a wrong + > subject/resource-kind, a non-leak diagnostic, or a differing exit status all pass + > as "parity". Build a **new exact diff harness** (or add a strict `--exact` mode to + > the script) over canonicalized `status + stdout + stderr + SARIF/JSON`. Reuse + > `oracle_compare.py` only as a reference for the SARIF-reading plumbing. - **Normalize then diff:** canonicalize JSON (sorted keys), normalize file paths, - drop volatile fields; compare the diagnostic set (code, line, subject, evidence - steps) and the SARIF log. A per-file, per-diagnostic divergence report — the same - shape `scripts/oracle_compare.py` already produces for the cross-tool (Infer#/ - CodeQL) diff. **The Rust core is just another SARIF producer that the existing - oracle diffs against Python.** + drop only genuinely volatile fields (timestamps, absolute temp dirs) — nothing + semantic. "SARIF-clean" must not be allowed to imply "full parity": `.ownreport.json` + fields not modeled in SARIF are compared on their own seam. - **CI ratchet:** a job runs the diff over the corpus and fails on any divergence in the covered subset. Coverage starts small and grows; the ratchet only tightens. - **Generative differential + metamorphic:** reuse the codegen fuzzer's spirit — @@ -196,17 +229,26 @@ SARIF/JSON shapes. So: a symbol must not change the diagnostic set) catch classes of bugs a fixed corpus misses. -Because the oracle compares *contracts we already froze and tested*, the recent -evidence/SARIF hardening is what makes it cheap. Each layer also gets its own diff -seam (`cfg` JSON at the CFG layer, SARIF at the verdict layer), so we can bisect a -divergence to the crate that introduced it. +**Per-layer seams.** SARIF is the verdict-layer seam and already exists. A **CFG-layer +seam does not** — today `python -m ownlang cfg` prints a *human* dump (`_print_cfg`), +not a contract. So a prerequisite of diffing CFGs is to first **add and freeze a +canonical `cfg --format json` export on the Python side**; mirroring the debug text +dump would bake a non-contract format into the ratchet. Treat "CFG JSON seam" as work +to build, not an existing contract. With SARIF (verdict) present and CFG-JSON added, +a divergence can be bisected to the crate that introduced it. + +Because the oracle compares *contracts we froze and tested*, the recent evidence/SARIF +hardening is what makes the verdict seam cheap — the CFG seam still needs building. ## Migration strategy (strangler-fig, bottom-up, oracle-gated) +0. **Add the missing Python seams first**: a canonical `cfg --format json` export + (and the exact diff harness). Without these the ratchet has nothing to compare the + CFG layer against. 1. **Stand up the workspace + `own-ir`** (serde round-trips the existing OwnIR fixtures — first parity check, at the seam). 2. **`own-syntax`**: port the parser; diff the AST/`cfg` dump against Python. -3. **`own-cfg`**: port lowering; diff CFG JSON. +3. **`own-cfg`**: port lowering; diff the frozen CFG JSON (from step 0). 4. **`own-analysis`**: port the worklist + ownership first, then lifetime/effect/DI; diff diagnostics (no evidence) → then evidence → then SARIF, layer by layer. 5. **`own-diagnostics` + `own-codegen`**: SARIF/report/text and C# `emit`; diff each. @@ -220,16 +262,27 @@ Throughout, Python stays authoritative; the Rust crates light up behind the ratc ## Open questions (to resolve as we go) - **Parser**: hand-roll vs `chumsky`/`winnow`. Leaning hand-roll for golden-parity of - error messages. -- **Persistent vs arena-CoW state**: `imbl` maps vs an arena with copy-on-write. Bench - on the corpus before committing. + error messages — matching Python's exact wording/positions through a combinator's + error model is friction in an oracle context. Re-evaluate a combinator rewrite only + *after* error-parity is frozen, gated by the same oracle. +- **Persistent vs arena-CoW state**: `imbl`/`rpds` maps vs a dense arena + dirty-bitset + copy-on-write. Persistent maps win with frequent merges / high sharing (wide CFGs); + arena+CoW likely wins on *this* workload (procedural functions, modest branching, + small per-RID state) where flat-`Vec` access beats persistent-map lookup constants — + and `imbl` only pulls ahead if we graduate to interprocedural/whole-program dataflow. + Bench on the corpus's **largest real function** (not synthetic), measuring both + wall-clock **and** peak RSS (persistent maps often lose on RSS via node overhead even + when they win on clone time). - **`serde-sarif` vs hand-rolled SARIF structs**: evaluate the crate's maintenance and whether it matches our exact shape. - **Incrementality (`salsa`)**: out of scope for parity, but the crate split should not preclude it (keep queries pure). - **Repo layout**: a `rust/` subtree in this repo (monorepo, easiest for the oracle to - run both) vs a sibling repo. Monorepo recommended so the oracle and corpus are one - `git` away. + run both) vs a sibling repo. Monorepo recommended *for this phase* so the oracle and + corpus are one `git` away and there is no submodule/pinned-SHA ceremony. Revisit the + split once Rust is authoritative and Python is legacy/reference-only — at that point + the coupling inverts and the monorepo's blast radius (Rust CI on every Python-only + doc change) becomes the annoyance rather than the benefit. ## Placement From 8bbc04a3edf0c237bddb22ae2c6147f962828021 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 02:29:06 +0000 Subject: [PATCH 3/9] docs(P-022): add performance principles + compiler-strictness/ratchet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Performance: frontend-specific levers in payoff order — intern everything (u32 symbols), bitset lattice (u8 bitflags for VarState + dense Vec<_> by RID, not HashSet) which also tips the persistent-vs-arena call toward arena+CoW, arenas + u32 indices, FxHashMap, rayon across independent per-function analyses, mimalloc/jemalloc. Micro-opt last, behind a flamegraph; perf locked by an iai-callgrind instruction-count ratchet in CI (deterministic). Added the matching library rows. Strictness: declarative [workspace.lints] block (Rust >= 1.74) — unsafe_code=forbid, unreachable_pub/rust_2018_idioms deny, clippy pedantic/nursery as warn with justified allows, unwrap/indexing/ arithmetic_side_effects/panic/dbg/print_stdout denied surgically. Added cargo-mutants (does the suite/oracle actually catch a broken transfer?), cargo-semver-checks, cargo-hack --feature-powerset, cargo-udeps, and the honest 'no NDepend-for-Rust' limit. Plus the ratchet doctrine: baseline current violations, hold new code to the full bar, tighten half a turn per iteration — same ratchet as the oracle (correctness) and iai-callgrind (perf), honouring 'a false positive is worse than a miss' at the tooling layer. --- docs/proposals/P-022-rust-core-migration.md | 95 ++++++++++++++++++++- 1 file changed, 91 insertions(+), 4 deletions(-) diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md index 9cbebd55..c8b8733b 100644 --- a/docs/proposals/P-022-rust-core-migration.md +++ b/docs/proposals/P-022-rust-core-migration.md @@ -139,7 +139,42 @@ shape. | Snapshot tests | `insta` | the Rust equivalent of the golden tests | | Property tests | `proptest` | replaces the Python codegen fuzzer | | Datalog (later) | `ascent`, `datafrog`, `crepe` | only if/when the rule layer goes declarative | - +| Fast hashing | `rustc-hash` (`FxHashMap`) / `ahash` | keys are small ints (RID/BlockId); SipHash default is a tax | +| Parallelism | `rayon` | per-function analysis is embarrassingly parallel | +| Allocator | `mimalloc` / `jemalloc` (`#[global_allocator]`) | allocation-heavy frontend; often a free 10–20 % | +| Perf regression gate | `iai-callgrind` (CI) + `criterion` (local) + `dhat` (heap) | instruction-count benches are deterministic → CI-safe | + +## Performance principles ("blazingly fast" is earned, not free) + +"Written in Rust" makes speed *possible*, not automatic — needless `.clone()`, the +SipHash default, `Box` in loops, and per-node allocation write slow Rust just +fine. This is a compiler frontend: the cost is **allocation, hashing, and tree/graph +traversal**, not FLOPs, so the levers are specific. In rough order of payoff for our +workload: + +1. **Intern everything** (symbols/strings → `u32`). `lasso`/`string-interner`; + equality becomes an int compare, memory drops sharply. rustc/rust-analyzer intern + universally — the #1 lever for this kind of code. +2. **Bitset lattice, not `HashSet`.** Today's `set[VarState]` over + `{OWNED,MOVED,RELEASED,ESCAPED}` becomes a **`u8` via `bitflags`**, and `State.var` + a *dense* `Vec` indexed by RID — tiny, cache-friendly, trivially cloned. + (rustc's dataflow uses `BitSet`/`ChunkedBitSet`.) This is also the thumb on the + scale for the persistent-map-vs-arena question: with a bitset + dense `Vec`, the + arena+CoW representation almost certainly wins. +3. **Arenas + `u32` indices** for AST/CFG — bump allocation, no per-node `Box`, no + pointer chasing, half the size of 64-bit pointers. +4. **`FxHashMap` everywhere** (`rustc-hash`) — our keys are small ints; SipHash is pure + overhead here. +5. **`rayon` across functions** — each function's worklist is independent, so per- + function (and per-file over OwnIR dumps) analysis parallelizes trivially. +6. **`mimalloc`/`jemalloc`** as the global allocator — a common free win for an + allocation-heavy frontend. + +Micro-optimization (SIMD, `unsafe get_unchecked`, manual bounds-check elision) is the +*last* resort and only behind a flamegraph. **Correctness first:** the differential +oracle proves parity with Python before any of this; perf is then locked by an +**`iai-callgrind` instruction-count ratchet** in CI (deterministic, unlike wall-clock), +so a regression fails the build the same way a divergence does. ## Prior art to study (architecture references) | Project | What to steal | @@ -171,11 +206,63 @@ strong, CI-enforceable decoupling: catches orphans and unexpected edges. - **`cargo-deny`** — dependency policy (licenses, bans, duplicate versions, advisories) — supply-chain + hygiene gate. -- **`cargo-machete`** — unused dependency detector. +- **`cargo-machete`** (stable) / **`cargo-udeps`** (nightly) — unused dependency + detector; machete is CI-cheap, udeps catches a few more. +- **`cargo-semver-checks`** — public-API break detection, for the library-shaped + crates (`own-ir`, and anything consumed externally). +- **`cargo-hack --feature-powerset`** — build/test every feature combination, so a + combo nobody compiled doesn't rot. - **`cargo-public-api`** — track each crate's public surface so coupling can't leak in through accidentally-`pub` internals; diff it in CI. -- **Clippy** (pedantic/nursery selectively) + `#![warn(missing_docs)]` + strict - module privacy (`pub(crate)` by default). +- **`cargo-mutants`** — mutation testing: does the test suite (and the oracle) *catch* + a deliberately broken transfer/lattice, or is it green-but-blind? The sharpest tool + here, and a natural fit with a correctness-first, oracle-gated port. + +Honest limit: there is **no full NDepend/ArchUnit-for-Rust** (no LCOM / +instability-metric / "zone of pain" tooling). The language compensates partly (orphan +rules, default privacy, `unreachable_pub`), and the crate DAG + `cargo-modules` + +`cargo metadata` edge tests cover the structural invariants — but true cohesion/SRP +judgement stays a review concern (human or the CodeRabbit/Codex layer we already run). + +### Compiler strictness — declarative `[workspace.lints]` (Rust ≥ 1.74) + +Lint config lives once in the workspace `Cargo.toml` and every crate inherits it: + +```toml +[workspace.lints.rust] +unsafe_code = "forbid" # forbid where unsafe isn't needed — cannot be overridden +unreachable_pub = "deny" # a pub nobody sees is a lie in the API +missing_debug_implementations = "warn" +rust_2018_idioms = { level = "deny", priority = -1 } + +[workspace.lints.clippy] +pedantic = { level = "warn", priority = -1 } +nursery = { level = "warn", priority = -1 } +unwrap_used = "deny" # restriction group, applied surgically: +expect_used = "warn" +indexing_slicing = "deny" # a panicking `[i]` is a prod incident +arithmetic_side_effects = "deny" # silent release-mode overflow +panic = "deny" +dbg_macro = "deny" +print_stdout = "deny" # libraries speak via `tracing`, not stdout +``` + +Do **not** take `pedantic`/`nursery` wholesale to `deny` — they carry noisy lints; +keep them `warn` with surgical `#[allow(...)]`, and **every `#[allow]` carries a +justification comment** (an unexplained allow is debt). `unsafe_code = "forbid"` per +crate is stronger than `deny` (it cannot be locally overridden) — set it everywhere a +crate has no legitimate `unsafe`. + +### The ratchet (do not boil the ocean) + +Max strictness switched on all at once, even on a *new* codebase, trains the +suppress-without-looking reflex — and a linter you reflexively silence is dead. This +is the same **ratchet** we already run for correctness (the oracle) and perf +(`iai-callgrind`): **baseline the current violations, hold new code to the full bar, +and let old code only get better, never worse.** Tighten half a turn per iteration, do +not strip the thread in one evening. This also honours the project's prime directive — +*a false positive is worse than a miss* — at the tooling layer, not just in the +analyzer's own verdicts. Fitness functions to encode early, each locked by one `cargo metadata` test over the allowed edge set: From e8885a20ea8a4926179e2a13100891ac4040e7cb Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 02:39:58 +0000 Subject: [PATCH 4/9] docs(P-022): fold in troubles.md perf specifics (arena/SmallVec/discipline) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-checked the performance section against the source article (troubles.md, 'Writing words and reading dwords'). Added the concrete frontend-relevant levers it teaches that were missing: - bump/typed arena for the build-once/read-many AST/CFG (bumpalo, typed-arena, rustc TypedArena) alongside u32 indices; - flat backing arrays, never Vec> (a cache miss per row); - SmallVec for the many-small collections (evidence steps, loans, per-block diagnostics) — inline by the law of small numbers; - enums / impl Trait / &dyn over Box; - the article's discipline framing we already run as doctrine: optimize by cumulative cost under perf (not by looks), skip one-time costs, keep the transfer function pure (registers + benchmarkability), algorithms first, #[inline] only on tiny cross-crate hot fns (never inline(always) by reflex), LTO + panic=abort, micro-opt last behind a flamegraph; - further-reading credits (the article, ripgrep post-mortem, Fastware). --- docs/proposals/P-022-rust-core-migration.md | 50 ++++++++++++++++----- 1 file changed, 40 insertions(+), 10 deletions(-) diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md index c8b8733b..55d877f5 100644 --- a/docs/proposals/P-022-rust-core-migration.md +++ b/docs/proposals/P-022-rust-core-migration.md @@ -161,20 +161,50 @@ workload: (rustc's dataflow uses `BitSet`/`ChunkedBitSet`.) This is also the thumb on the scale for the persistent-map-vs-arena question: with a bitset + dense `Vec`, the arena+CoW representation almost certainly wins. -3. **Arenas + `u32` indices** for AST/CFG — bump allocation, no per-node `Box`, no - pointer chasing, half the size of 64-bit pointers. -4. **`FxHashMap` everywhere** (`rustc-hash`) — our keys are small ints; SipHash is pure +3. **Arena-allocate the immutable trees.** An AST/CFG is built once then read many + times — the textbook case for a **bump/typed arena** (`bumpalo`, `typed-arena`, or + rustc's own `TypedArena`): nodes land contiguously, there is no per-node `Box`, and + the whole tree frees at once. Combine with `u32` indices for cross-references (half + the size of a 64-bit pointer, no pointer chasing, and serializable) — `la-arena` / + `id-arena` give exactly that. +4. **Flat backing arrays, never `Vec>`.** Nested vecs scatter each inner vec + across the heap (a cache miss per row). Back per-block instruction lists, the loan + table, etc. with one flat `Vec` + offsets/slices. This is the same cache argument as + the dense-`Vec`-by-RID state above. +5. **`SmallVec` for the many-small collections.** Evidence steps (usually 1–2), loans + per owner (usually 0–1), diagnostics per block — all tiny by the law of small + numbers. `SmallVec<[Evidence; 2]>` keeps them inline (no allocation, same cache + locality as a plain `T`) and only spills to the heap in the rare large case. +6. **`FxHashMap` everywhere** (`rustc-hash`) — our keys are small ints; SipHash is pure overhead here. -5. **`rayon` across functions** — each function's worklist is independent, so per- +7. **Prefer enums / `impl Trait` / `&dyn` over `Box`.** The `Lattice`/`Analysis` + layer is better as enums or generics (monomorphized, inlinable) than boxed trait + objects; where dynamic dispatch is genuinely needed, `&dyn` beats an owning `Box`. +8. **`rayon` across functions** — each function's worklist is independent, so per- function (and per-file over OwnIR dumps) analysis parallelizes trivially. -6. **`mimalloc`/`jemalloc`** as the global allocator — a common free win for an +9. **`mimalloc`/`jemalloc`** as the global allocator — a common free win for an allocation-heavy frontend. -Micro-optimization (SIMD, `unsafe get_unchecked`, manual bounds-check elision) is the -*last* resort and only behind a flamegraph. **Correctness first:** the differential -oracle proves parity with Python before any of this; perf is then locked by an -**`iai-callgrind` instruction-count ratchet** in CI (deterministic, unlike wall-clock), -so a regression fails the build the same way a divergence does. +**Discipline (the article's framing, which we already run as doctrine):** don't +optimize blindly or first — readability first, then optimize by **cumulative cost** +under `perf`/a flamegraph, not by what "looks slow"; one-time costs (CLI/OwnIR parsing) +don't matter, the hot path (the transfer function in the worklist) does — so **keep +the transfer pure** (local writes, no writes through shared pointers) both to help the +optimizer keep values in registers and to make it trivially benchmarkable. Algorithms +before micro-opt. `#[inline]` only on tiny cross-crate hot functions (`next`/`deref` +shape), **never `#[inline(always)]` by reflex**; LTO for cross-crate inlining; +`panic = "abort"` in release. SIMD / `unsafe get_unchecked` / manual bounds-check +elision (consolidate checks into one early `assert!` and let LLVM elide the rest) are +the *last* resort, behind a flamegraph. + +**Correctness first:** the differential oracle proves parity with Python before any of +this; perf is then locked by an **`iai-callgrind` instruction-count ratchet** in CI +(deterministic, unlike wall-clock), so a regression fails the build the same way a +divergence does. + +*Further reading:* troubles.md "Writing words and reading dwords: Achieving warp speed +with Rust" (the source of the arena/`SmallVec`/cache points above), BurntSushi's ripgrep +post-mortem, and Alexandrescu's "Fastware" talk. ## Prior art to study (architecture references) | Project | What to steal | From d2fc4df260fe27a8436f55d6797f07999fdbcddb Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 02:43:15 +0000 Subject: [PATCH 5/9] docs(notes): cross-stack strictness & architecture-fitness policy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture the tooling posture across all three stacks as a companion to P-022, centered on the ratchet doctrine (baseline current violations, hold new code to the full bar, tighten half a turn per iteration) — the same monotone gate we run for correctness (oracle) and perf (iai-callgrind), grounded in 'a false positive is worse than a miss'. Per stack: Rust (cargo-deny/udeps/semver-checks/hack/modules/mutants; [workspace.lints] lives in P-022; honest 'no NDepend-for-Rust' limit); C#/.NET WPF (csproj strictness + editorconfig; IDisposableAnalyzers, VSTHRD, Meziantou, ErrorProne, Sonar, BannedApiAnalyzers; NetArchTest/ ArchUnitNET + NDepend + Stryker.NET); Python ownlang (ruff ALL, xenon, vulture, import-linter, pyright, Hypothesis). Cross-stack: CodeQL + Semgrep. Notes the SOLID/SRP semantic limit (proxy metrics only; real detection = review, human or the CodeRabbit/Codex layer we already run) and a start order for ~80% of the value in a day. P-022 links to it. --- docs/notes/strictness-and-fitness.md | 133 ++++++++++++++++++++ docs/proposals/P-022-rust-core-migration.md | 4 +- 2 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 docs/notes/strictness-and-fitness.md diff --git a/docs/notes/strictness-and-fitness.md b/docs/notes/strictness-and-fitness.md new file mode 100644 index 00000000..768e6b50 --- /dev/null +++ b/docs/notes/strictness-and-fitness.md @@ -0,0 +1,133 @@ +# Strictness & architecture fitness — a cross-stack policy + +Status: **notes / doctrine** (the tooling posture across the three stacks; companion to +`docs/proposals/P-022-rust-core-migration.md`, which carries the Rust specifics) + +## The doctrine: ratchet, not a big bang + +The prime directive is **a false positive is worse than a miss**. That has a direct +consequence for linting/analysis tooling: switching maximum strictness on *all at once* +over a live codebase produces thousands of findings, a week of firefighting, and a +learned habit of slapping `suppress` on without looking — and a linter you reflexively +silence is dead for real. + +So the pattern is a **ratchet**, everywhere: + +- **Baseline** the current violations (built in to ruff, SonarQube, NDepend; in .NET via + editorconfig severities; in Rust via an allow-then-deny sweep). +- **New code** is held to the full bar. +- **Old code** may only get better, never worse. +- Tighten **half a turn per iteration**, not strip the thread in one evening. + +This is the same ratchet we already run for **correctness** (the differential oracle, +P-022) and **performance** (the `iai-callgrind` instruction-count gate). Strictness is +the third axis of the same idea: a monotone gate that only ever tightens. + +## Rust (007, sandboy, snipx, griff) + +Lint config is declarative and workspace-inherited (Rust ≥ 1.74) — the concrete +`[workspace.lints]` block lives in **P-022 § "Compiler strictness"** (`unsafe_code = +forbid`, `unreachable_pub`/`rust_2018_idioms` deny, clippy `pedantic`/`nursery` as +`warn` with justified `#[allow]`, surgical deny of `unwrap_used`/`indexing_slicing`/ +`arithmetic_side_effects`/`panic`/`dbg_macro`/`print_stdout`). + +Supply-chain and structure: + +- **`cargo-deny`** — licenses, advisories, duplicate versions, banned crates. +- **`cargo-udeps`** (nightly) / **`cargo-machete`** (stable) — dead dependencies. +- **`cargo-semver-checks`** — public-API break detection; critical for library-shaped + crates (e.g. `snipx`). +- **`cargo-hack --feature-powerset`** — build/test the feature combos nobody compiled. +- **`cargo-modules`** — module-graph structure + acyclicity. +- **`cargo-mutants`** — mutation testing: do the tests *catch* a deliberately broken + branch, or are they green-but-blind? The sharpest instrument here. + +Honest limit: **no full NDepend-for-Rust** (no LCOM / instability / "zone of pain" +metrics). The language compensates partly (orphan rules, default privacy, +`unreachable_pub`); the rest is `cargo-modules` + hand-written `cargo metadata` +dependency tests. + +## C# / .NET (enterprise WPF; the Roslyn extractor) + +Base in the csproj: + +```xml +true +latest-all +All +true +``` + +plus an `.editorconfig` that raises the IDE rules to `severity = error`. On Framework +4.7.2, `enable` (via `LangVersion`) annotates your own code only — +the BCL stays unannotated — still better than nothing. + +Analyzers, ranked by value here: + +- **IDisposableAnalyzers** — ownership/dispose discipline. This is literally the Own.NET + philosophy applied to live C#: who owns, who releases, who leaked. Install first. +- **Microsoft.VisualStudio.Threading.Analyzers (VSTHRD)** — async rules written in the + VS team's blood: `.Result`/`.Wait()` deadlocks in WPF, `async void`, `ConfigureAwait`. + Mandatory for a WPF app. +- **Meziantou.Analyzer** — culture-dependent strings, forgotten `CancellationToken`s, + allocations. +- **ErrorProne.NET** — struct-performance and exception hygiene. +- **SonarAnalyzer.CSharp** — broad, free bug detection. +- **BannedApiAnalyzers** — your own deny-list (`DateTime.Now`, `Task.Result`, + `File.ReadAllText` without an encoding, …) in one `BannedSymbols.txt`. +- **Roslynator + StyleCop** — style, to taste. + +Architecture as executable invariants (not a wiki page nobody reads): + +- **NetArchTest.eNhanced** / **ArchUnitNET** — "ViewModels don't reference the DAL", + "nothing depends on the UI assembly", "all repositories are internal" — as unit tests. +- **NDepend** (paid, only one in its class) — LCOM, afferent/efferent coupling, + instability `I = Ce/(Ca+Ce)`, abstractness, distance-from-main-sequence (the + zone-of-pain / zone-of-uselessness diagram), and CQLinq (LINQ queries over your own + codebase as CI rules). +- **Stryker.NET** — mutation testing for .NET. + +## Python (ownlang — the reference core) + +`mypy --strict` is already in the gate. Tighten around it: + +- **ruff** with `select = ["ALL"]` and a deliberate, commented ignore list — it includes + the pylint design metrics (`too-many-branches`/`-arguments`/`-statements`), crude but + workable proxies for low cohesion. +- **xenon** — fails the build when cyclomatic complexity exceeds a threshold + (`xenon --max-absolute B`). +- **vulture** — dead code. +- **import-linter** — architectural contracts for Python: "lexer does not import + codegen", "layers go strictly downward" — exactly the pipeline discipline ownlang + needs (and the analogue of the Rust crate-DAG fitness test). +- **pyright** in strict as a second opinion to mypy — they catch different things. +- **Hypothesis** — property tests on lexer/parser; generative inputs find what the + golden tests didn't imagine. + +## Cross-stack + +- **CodeQL** — free for public repos (ours are public). It is the Datalog-over-code we + keep circling back to (P-022 § open questions), ready-made; turn it on in Actions. +- **Semgrep** — custom rules in YAML in ~10 minutes; ideal for project bans ("nobody + calls `subprocess` outside our wrapper in 007"). +- **SonarQube Community** — a dashboard, if wanted. + +## The honest limit: SOLID is semantic + +SRP and DIP are *semantic* properties; statically they are only approximated by proxy +metrics (size, coupling, LCOM). NDepend and ArchUnit give an approximation; the real +"this class does two things" judgement comes from **review — human or LLM**. The +CodeRabbit + Codex layer we already run on every PR is precisely that missing tier, so +that slice is already being built, not bought. + +## Starting order (≈ a day's work, ~80 % of the value) + +1. **IDisposableAnalyzers + VSTHRD** in the WPF app — a direct hit on the ownership + theme. +2. **clippy `[workspace.lints]` + `cargo-deny`** in the Rust workspace. +3. **import-linter** in ownlang. +4. **CodeQL** in Actions across all public repos. + +Then take **NDepend** on trial and run it once over the enterprise solution — the +main-sequence diagram on a legacy codebase is an experience after which people either +refactor or drink (occasionally both). Everything else is applied on the ratchet. diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md index 55d877f5..6b1ec7e0 100644 --- a/docs/proposals/P-022-rust-core-migration.md +++ b/docs/proposals/P-022-rust-core-migration.md @@ -292,7 +292,9 @@ is the same **ratchet** we already run for correctness (the oracle) and perf and let old code only get better, never worse.** Tighten half a turn per iteration, do not strip the thread in one evening. This also honours the project's prime directive — *a false positive is worse than a miss* — at the tooling layer, not just in the -analyzer's own verdicts. +analyzer's own verdicts. The same doctrine across all three stacks (Rust / .NET / +Python) and the cross-stack tools (CodeQL, Semgrep) lives in +[`docs/notes/strictness-and-fitness.md`](../notes/strictness-and-fitness.md). Fitness functions to encode early, each locked by one `cargo metadata` test over the allowed edge set: From b23f5f8e91ba6c1b3c16899d846b3d52b816c21a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 02:51:35 +0000 Subject: [PATCH 6/9] docs(P-022): fold Codex perf review (RPO, I/O budget, state default, OWN050) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four P3 suggestions, all confirmed: - OWN050 is a real SARIF result (level: note, pinned in test_ownir.py), not stderr-only — corrected the oracle exit/crash-gate bullet so the exact harness compares it on the SARIF seam and doesn't diff the wrong stream / double-count the human summary. (Codex @325) - Solver scheduling: added RPO/reverse-postorder worklist ordering + an in_queue bitset (dense bitset-of-blocks for small CFGs) + scratch-state reuse as a first-class lever — block order dominates fixpoint iteration count even with the compact lattice. (Codex @123) - I/O in the perf budget: over large OwnIR dumps serde_json parse + SARIF render are O(input+findings) per invocation and can dominate once the transfer is tight; bench large-OwnIR from_slice / buffered to_writer with interned strings, not only transfer microbenches. Softened the 'one-time costs don't matter' claim. (Codex @191) - Dataflow-state table row: present persistent maps (imbl/rpds) as a benchmark *candidate*, with dense-bitset arena/scratch CoW as the leaning default — don't lock the port into tree-node alloc + O(log n) before the corpus bench. (Codex @116) --- docs/proposals/P-022-rust-core-migration.md | 35 ++++++++++++++++----- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md index 6b1ec7e0..8879c50c 100644 --- a/docs/proposals/P-022-rust-core-migration.md +++ b/docs/proposals/P-022-rust-core-migration.md @@ -131,7 +131,7 @@ shape. | Arena / IDs | `la-arena` (rust-analyzer), `id-arena`, `slotmap` | index-based AST/CFG | | Interning | `lasso`, `string-interner` | symbol/string interning | | Graph (CFG) | `petgraph` or hand-rolled arena | petgraph gives traversals/dominators for free | -| Persistent state | `imbl` (im fork) or `rpds` | copy-on-write dataflow state | +| Dataflow state | dense bitset `Vec` + arena/scratch CoW *(leaning default)*; `imbl`/`rpds` *(benchmark candidate)* | persistent maps are one option to bench, **not** the default idiom — see the persistent-vs-arena open question | | Serialization | `serde` + `serde_json` | OwnIR, SARIF, `.ownreport.json` | | SARIF types | `serde-sarif` (evaluate) or hand-rolled structs | we already emit the shape; typed structs prevent drift | | CLI | `clap` (derive) | mirror the current subcommand surface | @@ -185,13 +185,28 @@ workload: 9. **`mimalloc`/`jemalloc`** as the global allocator — a common free win for an allocation-heavy frontend. +**Solver scheduling (a big lever the compact lattice alone won't buy).** For a monotone +forward analysis with loops, *block visitation order* decides how many times joins and +transfers re-run. Schedule the worklist in **reverse-postorder (RPO)** so a block is +(re)visited after its predecessors are stable — this minimizes fixpoint iterations. Back +the queue with a cheap **`in_queue` bitset** (or, for small CFGs, a dense bitset-of- +blocks) so a block is never enqueued twice, and reuse **scratch state buffers** across +per-block transfers instead of re-allocating a fresh `State` each visit. Without this, +an implementation can burn most of its time revisiting blocks and churning allocations +*even with* the bitset lattice above. + **Discipline (the article's framing, which we already run as doctrine):** don't optimize blindly or first — readability first, then optimize by **cumulative cost** -under `perf`/a flamegraph, not by what "looks slow"; one-time costs (CLI/OwnIR parsing) -don't matter, the hot path (the transfer function in the worklist) does — so **keep -the transfer pure** (local writes, no writes through shared pointers) both to help the -optimizer keep values in registers and to make it trivially benchmarkable. Algorithms -before micro-opt. `#[inline]` only on tiny cross-crate hot functions (`next`/`deref` +under `perf`/a flamegraph, not by what "looks slow"; truly one-time costs don't matter, +the hot path (the transfer function in the worklist) does — so **keep the transfer +pure** (local writes, no writes through shared pointers) both to help the optimizer keep +values in registers and to make it trivially benchmarkable. **Caveat for the batch +scanner:** over large OwnIR dumps, `serde_json` parse and SARIF/report rendering are +`O(input + findings)` and run on *every* invocation — once the transfer loop is tight, +that I/O can dominate wall-time and allocations. So bench it explicitly too: +representative large-OwnIR `from_slice` / buffered `to_writer`, with borrowed/interned +diagnostic strings — not only transfer-function microbenches. Algorithms before +micro-opt. `#[inline]` only on tiny cross-crate hot functions (`next`/`deref` shape), **never `#[inline(always)]` by reflex**; LTO for cross-crate inlining; `panic = "abort"` in release. SIMD / `unsafe get_unchecked` / manual bounds-check elision (consolidate checks into one early `assert!` and let LLVM elide the rest) are @@ -321,8 +336,12 @@ SARIF/JSON shapes. So: - **Exit/crash gate first.** Before diffing *any* output, assert both binaries produced the same exit status and neither panicked — a Rust panic has no SARIF representation, so an output-only diff would score a crash-on-valid-input as - "no findings = parity". Diff `stderr` too (e.g. the P-014 OWN050 advisory notes - live there, not in SARIF). + "no findings = parity". Diff `stderr` too — but note it carries only the + machine-format *summary/chatter* (verbosity lines). The advisory **OWN050** is a + genuine **SARIF result** (`level: "note"`; pinned in `tests/test_ownir.py`), so it is + compared on the SARIF seam like any other finding — do **not** treat it as + stderr-only, or the oracle will diff the wrong stream / double-count the human + summary text. - **Exact, not fuzzy.** This is a **golden** same-input comparison, so it demands strict set-equality on `(path, line, code, subject)` plus the full evidence slice **including each step's label text** (a step on the right line with the wrong From 3322d54cffd5bd7d6dce92046006654e3b04c580 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 02:54:42 +0000 Subject: [PATCH 7/9] docs(P-022): align patterns-table state idiom with perf guidance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex flagged (and fixed in its sandbox) that the Patterns/idioms table still presented persistent maps as THE State.copy() replacement, out of sync with the Libraries-row/perf-section fix. Apply the same wording here so the doc is internally consistent: dense bitset Vec + arena/scratch CoW is the leaning default; imbl/rpds a benchmark candidate. (Codex's own commit 3a38d70 never landed on the branch/GitHub — folded manually.) --- docs/proposals/P-022-rust-core-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md index 8879c50c..764bf704 100644 --- a/docs/proposals/P-022-rust-core-migration.md +++ b/docs/proposals/P-022-rust-core-migration.md @@ -113,7 +113,7 @@ build graph. | `assert_never` dispatch updated by hand in cfg/analysis/codegen | `enum` + exhaustive `match` | a missed variant is a **compile error**, not a runtime assert | | RID = `id(sym)`; handles keyed by object identity | newtype indices `Rid(u32)`, `BlockId(u32)`, `LoanId(u32)` + arena | deterministic, serializable, no identity hacks | | AST/CFG as object graphs | **arena + indices** (`la-arena`/`id-arena`) | borrow-checker-friendly graphs, cheap clone, cache-friendly | -| `State.copy()` deep-copies dicts every merge | **persistent maps** (`imbl`/`rpds`) — structural sharing | join/clone at merges is O(log n) sharing, not a deep copy | +| `State.copy()` deep-copies dicts every merge | dense bitset `Vec` + arena/scratch copy-on-write *(leaning default)*; persistent maps (`imbl`/`rpds`) a *benchmark candidate* | avoid the deep copy at merges — but don't pre-commit to tree-node alloc + O(log n); see the persistent-vs-arena open question | | ownership/lifetime/effect/DI interleaved in `_Analyzer` | `Lattice` + `DataflowAnalysis` traits; one generic solver, N impls | analyses decoupled + independently testable | | strings compared everywhere | interning (`lasso`/`string-interner`) | cheap symbol equality; smaller state | | Diagnostics-as-data (good, keep it) | keep: `Diagnostic` is data, **not** an `Err` | verdicts stay first-class; `thiserror`/`anyhow` only for real I/O errors | From 60b45f9ec6771a279be7d18a4ef82275375d9c26 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 04:36:57 +0000 Subject: [PATCH 8/9] docs(P-022): fix DAG arrow direction + unsafe/forbid vs get_unchecked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two internal inconsistencies CodeRabbit flagged this round: - Crate-DAG arrows were drawn backward for the verdict layer. The solver constructs Diagnostic/Evidence (types owned by own-diagnostics), so the real Cargo edge is own-analysis -> own-diagnostics, NOT the reverse — which is what the fitness function ('own-diagnostics must not depend on own-analysis') and the migration order already imply. Redrew the diagram (own-diagnostics upstream of own-analysis; own-codegen depends on own-cfg only) and rewrote the prose + crate bullets to match. - unsafe_code = 'forbid' (workspace-wide) contradicted the perf section still listing 'unsafe get_unchecked' as a lever. Dropped it: bounds-check elision is done the safe way (one early assert! + LLVM), and a profiled hot path that genuinely needs raw indexing scopes that ONE crate from forbid -> deny with a documented reviewed #[allow(unsafe_code)], never the workspace. --- docs/proposals/P-022-rust-core-migration.md | 58 +++++++++++++-------- 1 file changed, 35 insertions(+), 23 deletions(-) diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md index 764bf704..04db3020 100644 --- a/docs/proposals/P-022-rust-core-migration.md +++ b/docs/proposals/P-022-rust-core-migration.md @@ -56,30 +56,36 @@ surface and the frontend boundary: ## Crate topology (decoupled = the DAG; Cargo enforces acyclicity at compile time) +Arrow = "is depended on by" (dependency → dependent, i.e. build order): + ``` - own-ir (OwnIR fact/verdict contract; serde — leaf-ish, everyone may depend on it) + own-ir (OwnIR fact/verdict contract; serde — leaf, everyone may depend on it) own-syntax (lexer / parser / AST) └─▶ own-cfg (AST → CFG lowering) - └─▶ own-analysis (lattice + worklist; ownership / lifetime / effect / DI) + ├─▶ own-analysis (lattice + worklist; ownership / lifetime / effect / DI) + └─▶ own-codegen (C# emit — AST/CFG only; NOT own-analysis, NOT own-diagnostics) - own-cfg + own-analysis ─┬─▶ own-diagnostics (Diagnostic/Evidence model, text + SARIF) - └─▶ own-codegen (C# emit — AST/CFG-driven, verdict-independent) + own-diagnostics (Diagnostic/Evidence model, text + SARIF) ─▶ own-analysis + # own-analysis *constructs* Diagnostic/Evidence, so it depends on own-diagnostics; + # own-diagnostics depends only on own-ir/own-syntax spans — never on the solver. {all of the above} ─▶ own-cli (check / emit / cfg / report / ownir / explain) own-cli ◀─ own-oracle (dev/test: differential harness vs Python) ``` -`own-diagnostics` and `own-codegen` are **sibling consumers** of `own-cfg` / -`own-analysis` — neither depends on the other. This matters: codegen must not chain -*through* the verdict renderer (see the fitness function below and Codex's review), -or it would need diagnostics to re-export solver internals — breaking the "diagnostics -knows nothing about the solver" invariant. Today's Python `codegen.generate(mod)` -takes only the AST and never imports `diagnostics`, so codegen is **verdict-independent** -(its policy comes from AST/CFG shape — `_laminar_scopes`, `_buffer_modes`, …), not from -analysis conclusions. Keep it that way; if Rust codegen ever *does* want a solver -verdict (e.g. inserting a `Dispose()` from the ownership conclusion rather than -re-deriving it from shape), that is a **deliberate new `own-codegen → own-analysis` -edge** to flag on its own, not something that sneaks in via the diagnostics arrow. +The non-obvious edge is **`own-analysis → own-diagnostics`**: the solver *constructs* +`Diagnostic`/`Evidence` values (in Python, `analysis.py` imports and builds them from +`diagnostics.py` using solver-internal state), and those types are *owned by* +`own-diagnostics` — so analysis depends on diagnostics, **not the reverse**. +`own-diagnostics` therefore stays upstream, depending only on span/location primitives +(`own-ir`/`own-syntax`), never on the solver — which is exactly the fitness function +below. `own-codegen` is the true **sibling**: it hangs off `own-cfg`/AST *only* and is +**verdict-independent** (Python `codegen.generate(mod)` takes just the AST — its policy +comes from `_laminar_scopes`/`_buffer_modes`, not analysis conclusions). Codegen must +not chain through diagnostics or reach into the solver; if Rust codegen ever *does* want +a verdict (e.g. a `Dispose()` inserted from the ownership conclusion rather than +re-derived from shape), that is a **deliberate new `own-codegen → own-analysis` edge** +to flag on its own. - **`own-syntax`** — lexer + parser + AST. Zero analysis knowledge. (Prior art: ruff / rust-analyzer parsers.) @@ -92,13 +98,14 @@ edge** to flag on its own, not something that sneaks in via the diagnostics arro `Analysis` traits, and the ownership/loan/lifetime/effect/DI impls. Each analysis is an independent trait impl, not interleaved (this is the direct de-noodling of today's `_Analyzer`). -- **`own-diagnostics`** — Diagnostic/Evidence model + presentation (human render, - SARIF projection). A pure consumer of verdict data; knows nothing about the - solver internals. Mirrors today's already-clean split (`evidence.py` is a pure +- **`own-diagnostics`** — *owns* the Diagnostic/Evidence types + their presentation + (human render, SARIF projection). It knows nothing about the solver; the solver + depends on **it** to construct verdicts. Depends only on span/location primitives + (`own-ir`/`own-syntax`). Mirrors today's clean split (`evidence.py` is a pure projection). - **`own-codegen`** — C# emission, **driven by AST/CFG shape, not analysis verdicts** - (verdict-independent, matching Python `codegen.generate(mod)`). A sibling of - `own-diagnostics`, not downstream of it. + (verdict-independent, matching Python `codegen.generate(mod)`). Depends on `own-cfg` + only — a sibling of `own-analysis`/`own-diagnostics`, downstream of neither. - **`own-cli`** — the binary; wires the pipeline and owns the CLI surface. - **`own-oracle`** — dev-only differential harness (see below). @@ -208,9 +215,14 @@ representative large-OwnIR `from_slice` / buffered `to_writer`, with borrowed/in diagnostic strings — not only transfer-function microbenches. Algorithms before micro-opt. `#[inline]` only on tiny cross-crate hot functions (`next`/`deref` shape), **never `#[inline(always)]` by reflex**; LTO for cross-crate inlining; -`panic = "abort"` in release. SIMD / `unsafe get_unchecked` / manual bounds-check -elision (consolidate checks into one early `assert!` and let LLVM elide the rest) are -the *last* resort, behind a flamegraph. +`panic = "abort"` in release. Bounds-check elision the **safe** way — consolidate the +checks into one early `assert!` and let LLVM prove the rest unreachable (Rust's safe +iterators already match the C start/end-pointer idiom) — plus SIMD are the *last* +resort, behind a flamegraph. Note this stays inside `unsafe_code = "forbid"`: we do +**not** reach for `unsafe get_unchecked`. If a *profiled* hot path ever proves it needs +raw unchecked indexing beyond what assert-elision buys, that single crate — not the +workspace — drops `forbid` → `deny` with a documented, reviewed `#[allow(unsafe_code)]`; +never a blanket relaxation. **Correctness first:** the differential oracle proves parity with Python before any of this; perf is then locked by an **`iai-callgrind` instruction-count ratchet** in CI From 4d3a37c19f5be0bfb900367bee1d6ff427249936 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 04:41:11 +0000 Subject: [PATCH 9/9] docs(P-022): label crate-topology fence as text (markdownlint MD040) --- docs/proposals/P-022-rust-core-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/P-022-rust-core-migration.md b/docs/proposals/P-022-rust-core-migration.md index 04db3020..e1cbe676 100644 --- a/docs/proposals/P-022-rust-core-migration.md +++ b/docs/proposals/P-022-rust-core-migration.md @@ -58,7 +58,7 @@ surface and the frontend boundary: Arrow = "is depended on by" (dependency → dependent, i.e. build order): -``` +```text own-ir (OwnIR fact/verdict contract; serde — leaf, everyone may depend on it) own-syntax (lexer / parser / AST) └─▶ own-cfg (AST → CFG lowering)