test: add level-one throughput acceptance benchmarks

[SkyeAv]

Adds absolute throughput acceptance benchmarks for the normalized level-one path on both sides of the FFI, gated on the median of three warmed-up runs with bounds calibrated for slow CI hardware.

Python (tests/test_nlp.py)

• test_level_one_performance_one_million_terms: 1M seeded multi-token terms through level_one(...).collect(), generation excluded from timing, one warmup, then three timed runs with the median gated at ≤8.0s; output shape and representative normalized values asserted so the artifact is exercised, not just timed.

Rust (rust/src/nlp.rs)

• normalize_terms_performance_one_million_terms: 1M terms through the pyo3 batch entry with warmup and three timed runs, median gated at ≤12.0s, output length/content asserted; deterministic LCG-seeded workload (seed 5005), locale/network/optional-extra independent, runs as a normal #[test] (never ignored).

Design

• Calibrated bounds, no relative comparisons: measured ~0.6s (Python) and ~0.8s (Rust) on a 12-core dev box vs ~3.2s and 3.6–4.9s per sample on 2-core CI runners; the ceilings keep >2× headroom on the slowest observed environment while still failing immediately for the guarded regression classes: map_elements per-row UDFs, serial fallback, and accidental quadratic token work (minutes at this N, not seconds).
• Median-of-three absorbs scheduler noise without relaxing the ceilings.

Testing

• uv run pytest tests/test_nlp.py::test_level_one_performance_one_million_terms -n 0 -q → 1 passed
• cargo test --manifest-path rust/Cargo.toml nlp::tests::normalize_terms_performance_one_million_terms -- --nocapture --test-threads=1 → normalize_terms median took 0.802s for 1,000,000 terms, 1 passed
• Full NLP suite → 17 passed; clippy -D warnings, cargo fmt --check, pyright, ruff → clean

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 00c2d031-e3cd-402e-83cc-ca0b2bcbdafa

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}