docs: document the normalized level-one contract

[SkyeAv]

Locks the normalized level-one contract in place: a shared golden fixture consumed by both languages, API docs describing the v6 semantics, and the BREAKING changelog entry.

Golden equivalence

• rust/tests/fixtures/nlp_golden.tsv: representative vectors (reorder, inflections, dedupe, whitespace, digits, punctuation, Unicode lowercase, empty via an explicit @EMPTY encoding that keeps git diff --check clean) consumed by BOTH rust/tests/nlp_golden.rs and a tests/test_nlp.py golden test through the compiled level_one path — the two sides can never silently acquire different normalization contracts.

Docs

• docs/api/fullmap.md, docs/api/lib.md: document the level-one semantics precisely (clean/Unicode lowercase, whitespace tokenization, Porter2 stemming of all-ASCII-alphabetic tokens only, digit/punctuation pass-through, dedupe + byte-wise sort + single-space join, null preservation), the schema-v6 rebuild requirement, and the raw-exact-then-normalized preferred-name ranking; stale "lowercase + strip" passages replaced.
• docs/fullmap.md, docs/cli.md: schema references updated in the migration PR are kept consistent.

CHANGELOG

• ## Unreleased → Breaking Changes: level-one keys now defined by normalized terms; schema-v5 databases rejected, rebuild required; preferred-name ranking compares normalized forms.

Testing

• cargo test --manifest-path rust/Cargo.toml --test nlp_golden → 1 passed; uv run pytest tests/test_nlp.py -k golden -n 0 -q → 1 passed, 16 deselected; full NLP suite → 17 passed
• Full gate at stack top: make check → exit 0 (ruff lint/format, full-repo pyright 0 errors, pytest, cargo test, clippy -D warnings); uv run pytest → 1352 passed, 43 skipped, 0 failed
• Independent final audit: APPROVE — no blockers, no should-fix, no nits

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 1d1d4e57-5417-4c6b-947a-a2741d84b85a

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}