feat: add shared Rust level-one normalizer and pyo3 batch surface

[SkyeAv]

Adds the shared Rust level-one normalizer — clean → Unicode lowercase → Snowball English (Porter2) stemming of all-ASCII-alphabetic tokens → dedupe → byte-wise sort → single-space join — plus a rayon-parallel, GIL-releasing pyo3 batch surface, as the foundation for stronger fullmap matching.

Normalizer (rust/src/nlp.rs)

• normalize_l1: returns Cow<str> with a zero-copy borrowed fast path for already-clean single tokens; digit-bearing (51), punctuation-bearing (tnf-alpha), and non-ASCII tokens pass through unstemmed after lowercasing.
• Shared cleaning: clean_and_lower in rust/src/fullmap.rs becomes pub(crate) and is reused verbatim by nlp, so cleaning semantics literally cannot drift from the fullmap emit path.
• Unit tests: 12 cases with WHY comments — reorder, dedupe, whitespace collapse, stemming folds (genes→gene, inhibiting→inhibit), digit/punctuation pass-through, Unicode lowercase, empty string, Cow borrow fast path.

pyo3 surface

• normalize_terms: index-aligned batch function, rayon-parallel, releases the GIL via py.detach (pyo3 0.29); registered in rust/src/lib.rs, stubbed in src/tablassert/rs.pyi.
• normalize_l1 re-export: plain Rust export for the fullmap build side (landed in a later PR of this stack) and integration tests; intent documented at the export site.

Design

• rust-stemmers = "1" (docs.rs): pure-Rust Snowball implementations (English = Porter2), MIT/BSD-3 with only serde deps, Send+Sync fn-pointer stemmer shared via OnceLock, Cow-returning stem. Rejected alternatives: PyStemmer/snowballstemmer (Python-side cost), lemmatizers (dictionary weight, no throughput), pure-polars token chains (measured ~25× slower: 0.8M rows/s vs 20.5M).
• Deferred: lemmatization (dictionary weight vs. throughput) and stop-word removal (precision); no per-column opt-out exists today.

Testing

• cargo test --manifest-path rust/Cargo.toml nlp → 12 passed
• cargo test --manifest-path rust/Cargo.toml --lib → 124 passed
• cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings → clean
• cargo fmt --check → clean

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: fb6d7e01-2249-4f9f-bbd7-c8ace34bba79

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}