feat!: strengthen level-one normalization and move fullmap to schema v6

[SkyeAv]

BREAKING: strengthens level-one normalization on both sides of the fullmap contract at once — the Python query side routes through the shared Rust normalizer, and fullmap builds derive their keys from the same implementation, moving databases to schema tablassert.fullmap.v6.

Why one PR

• Two-sided exact-match contract: fullmap lookups are exact string matches between Python query keys and Rust-emitted DB keys. Strengthening one side alone leaves the other's keys unreachable (empirically: the intermediate state failed the agent/e2e suites in CI), so query side and build side land as one atomic layer.

Python query side (src/tablassert/nlp.py)

• level_one(lf, col): signature and in-place semantics unchanged; the column is cast to string and mapped through rs.normalize_terms via a null-safe map_batches (map_elements stays forbidden per repo convention). Nulls never reach the pyo3 surface; "" stays "".
• 7 new tests (tests/test_nlp.py) with WHY docstrings: token-order insensitivity, stemming fold, dedupe + whitespace collapse, null/empty preservation, digit/punctuation pass-through, Unicode lowercase, other columns untouched. All 8 pre-existing tests pass unmodified.
• Hand-keyed fake-fullmap fixtures re-keyed (tests/test_lib.py) to normalized forms, each mapping verified against the real normalizer — expectation-preserving, no weakened assertions.

Rust build side (rust/src/fullmap.rs)

• emit_term: level-one keys now derive via nlp::normalize_l1, byte-for-byte matching the query side.
• Quality gates stay pre-stemming: token_qc's banned-token check compares literal spellings (hypothetical, uncharacterized) that Porter2 would mutate (hypothetical→hypothet); a second post-normalization is_dead_term guard blocks terms that normalize onto query-side sentinels. Level-two derivation composes after level one unchanged.

Schema migration

• SCHEMA_VERSION v5 → v6: the key SPACE changed, so old databases can never be hit by normalized queries; validate_schema rejects v1–v5 loudly with the existing rebuild instruction, covered by v4/v5 rejection tests.
• Golden pins regenerated deliberately (rust/tests/build_golden.rs, rust/tests/common/mod.rs, rust/tests/extract_prebuilt.rs) with the documented regeneration command; Python real-DB expectations re-keyed; schema strings updated in docs/fullmap.md and docs/cli.md.

Testing

• cargo test --manifest-path rust/Cargo.toml → --lib 126 passed, fullmap 57 passed, --test build_golden 11 passed, 1 ignored (the documented regeneration helper)
• Targeted Python (test_fullmap, test_cover_fullmap, test_docs_source_of_truth, test_nlp, test_lib) → 471 passed
• Independent verifier exercise: real-DB lock-step probe over 1,232,064 terms → HITS=1,221,965 EXPECTED_DROPS=10,099 UNEXPECTED_MISSES=0; forged v5-tagged DB rejected loudly through both direct lookup and the production resolve_many path
• Throughput: golden build ~0.99×; repeated 2M-row builds ≤1.25× of the pre-change baseline
• CI: python-lint, python-test, python-type, rust all pass on this head

Caveats

• Existing fullmap databases must be rebuilt (tablassert build-fullmap); prebuilt archives need a re-release.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 61e0df30-1120-42c2-8720-7df5968b93b2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}