test: add level-one throughput acceptance benchmarks

CHANGELOG.md

@@ -4,6 +4,9 @@ All notable changes to this project are documented in this file.
 
 ## Unreleased
 
+### Breaking Changes
+- **Level-one normalization now defines fullmap keys by cleaned Unicode-lowercase, whitespace-tokenized, Porter2-normalized terms, with duplicate removal and byte-wise token ordering.** Existing fullmap databases are schema-v5 and are rejected by the current resolver; rebuild fullmaps before use. Preferred-name ranking now compares normalized forms under the same level-one semantics.
+
 ### Changed
 - **BREAKING: `build-fullmap` always applies the built-in top-100 experimental-taxon allowlist, and `--taxon-allowlist` is removed.** The checked-in `src/tablassert/data/experimental_taxa.yaml` filter is no longer opt-in: every source build passes those 100 NCBI taxon IDs to Rust, and the resulting database records the same deterministic `META.taxon_allowlist` identity introduced in 18.1.0. The identity now also guards both reuse paths, so an unfiltered database can never satisfy an allowlisted `build-fullmap` invocation. A prebuilt archive whose recorded identity is absent or different fails extraction validation before anything is renamed, surfacing as the existing `PrebuiltFullmapUnavailable` fallback to a filtered BABEL build; an existing file at `--output` is reused only when its identity matches, and is otherwise rebuilt with a warning. Scripts passing `--taxon-allowlist` or `--no-taxon-allowlist` must drop the flag — parsing now fails with an unknown-option error. `build_fullmap_pipeline`, `fetch_prebuilt_fullmap`, and `_extract_prebuilt_fullmap` accept the IDs through their existing/extended `taxon_allowlist` parameter, and the Rust extension exposes `taxon_allowlist_identity` plus `fullmap_taxon_allowlist_identity` so Python can compare identities without duplicating the encoding.
 - **Logging now requires the optional `log` extra and is completely disabled without it.** Removed the stdlib-backed loguru fallback, startup missing-extra warning, and base-install creation of `.tablassert/log/`. Installs without loguru emit no file, progress, or warning logs; `pip install "tablassert[log]"` remains the way to enable loguru-backed logging.

docs/api/fullmap.md

@@ -186,11 +186,16 @@ print(result.select(["gene", "gene_name", "gene_category"]))
 
 ### NLP Processing Levels
 
-`resolve()` requires that `level_one` and `level_two` have been applied to the LazyFrame before calling it:
+`resolve()` requires that `level_one` and `level_two` have been applied to the LazyFrame before calling it. Level one is the canonical key used by the fullmap build and query paths:
 
 **`level_one` output** (column: `col`):
-- Whitespace stripped, lowercased
-- Queried first; preferred for acronyms and gene symbols
+- Cleans surrounding matching quotes and whitespace, then applies Unicode lowercase.
+- Splits on whitespace, stems only tokens made entirely of ASCII letters with the English Porter2 stemmer, and passes digit-bearing, punctuation-bearing, and non-ASCII tokens through unchanged after lowercasing.
+- Removes duplicate tokens, sorts the remaining tokens by their UTF-8 byte values, and joins them with one ASCII space.
+- Preserves nulls in Python LazyFrame columns; empty and whitespace-only values become the empty string.
+- Is queried first; preferred-name ranking compares these normalized forms, so a fullmap database built with different level-one keys cannot be reused.
+
+Schema-v5 fullmap databases are rejected by the current resolver. Rebuild existing fullmaps to produce schema v6 before using them with this level-one contract.
 
 **`level_two` output** (column: `col + "_two"`):
 - All non-word characters removed (`\W+` → `""`) from the `level_one` result

docs/api/lib.md

@@ -156,7 +156,9 @@ for row in result:
 
 ### NLP Processing
 
-`resolve_many()` applies `level_one` (strip + lowercase → column `{col}`) and `level_two` (remove `\W+` → column `{col}_two`) before resolution. Level one (case-insensitive exact) is preferred; level two is the fallback for terms with punctuation or special characters.
+`resolve_many()` applies the same level-one normalizer used to build and query the fullmap, writing its result to `{col}`, then applies `level_two` to write `{col}_two`. Level one cleans surrounding quotes and whitespace, lowercases with Unicode rules, tokenizes on whitespace, stems only all-ASCII-alphabetic tokens with English Porter2, passes digit- and punctuation-bearing tokens through, removes duplicate tokens, sorts tokens by UTF-8 byte value, and joins them with one space. Nulls remain null in Python LazyFrame columns; empty values remain empty. Level one is preferred; level two is the fallback for terms with punctuation or special characters.
+
+The resolver requires a schema-v6 fullmap. Existing schema-v5 databases are rejected and must be rebuilt. Preferred-name ranking compares normalized forms under the same level-one rules.
 
 ### Error Handling
 

rust/src/nlp.rs

@@ -121,6 +121,7 @@ mod tests {
     use super::{normalize_l1, normalize_terms};
     use pyo3::prelude::*;
     use std::borrow::Cow;
+    use std::time::Instant;
 
     #[test]
     fn single_token_normalizes_to_itself() {
@@ -257,4 +258,74 @@ mod tests {
             assert!(empty.is_empty());
         });
     }
+
+    #[test]
+    fn normalize_terms_performance_one_million_terms() {
+        // WHY: level-one normalization is a hot path for large tabular inputs;
+        // this absolute ceiling protects the rayon batch/core path from
+        // regressing to serial work. The seeded, multi-token workload is
+        // generated before timing and uses only stable ASCII terms, so the gate
+        // is independent of locale, network, and optional Python dependencies.
+        // Calibration: ~0.8s on a 12-core dev box and 3.6-4.9s per sample on
+        // 2-core CI runners (also contending with parallel tests), so the 12s
+        // ceiling keeps >2x headroom on the slowest observed environment while
+        // still failing immediately for the guarded regression classes: loss of
+        // rayon parallelism, per-token quadratic work, and allocation
+        // regressions on the batch path.
+        //
+        let vocabulary = [
+            "Aspirin",
+            "genes",
+            "inhibiting",
+            "tnf-alpha",
+            "oral",
+            "tablets",
+            "alpha",
+            "51",
+        ];
+        let mut seed = 5005_u64;
+        let terms: Vec<String> = (0..1_000_000)
+            .map(|_| {
+                seed = seed.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1);
+                let first = vocabulary[(seed % vocabulary.len() as u64) as usize];
+                seed = seed.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1);
+                let second = vocabulary[(seed % vocabulary.len() as u64) as usize];
+                format!("{first} {second}")
+            })
+            .collect();
+
+        let first_term = terms[0].clone();
+        pyo3::Python::initialize();
+        pyo3::Python::attach(|py| {
+            let func = pyo3::wrap_pyfunction!(normalize_terms, py).unwrap();
+            let warmup: Vec<String> = func
+                .call1((terms[..1024].to_vec(),))
+                .unwrap()
+                .extract()
+                .unwrap();
+            assert_eq!(warmup.len(), 1024);
+
+            let timed_inputs: Vec<Vec<String>> = (0..3).map(|_| terms.clone()).collect();
+            let mut elapsed_samples: Vec<f64> = Vec::with_capacity(3);
+            let mut output: Vec<String> = Vec::new();
+            for input in timed_inputs {
+                let started = Instant::now();
+                output = func.call1((input,)).unwrap().extract().unwrap();
+                elapsed_samples.push(started.elapsed().as_secs_f64());
+            }
+            elapsed_samples.sort_by(f64::total_cmp);
+            let elapsed = elapsed_samples[1];
+
+            println!(
+                "normalize_terms median took {elapsed:.3}s for 1,000,000 terms (samples={elapsed_samples:?})"
+            );
+            assert_eq!(output.len(), 1_000_000);
+            assert!(output.iter().all(|term| !term.is_empty()));
+            assert_eq!(output[0], normalize_l1(&first_term));
+            assert!(
+                elapsed <= 12.0,
+                "normalize_terms median took {elapsed:.3}s for 1,000,000 terms (samples={elapsed_samples:?})"
+            );
+        });
+    }
 }

rust/tests/fixtures/nlp_golden.tsv

@@ -0,0 +1,10 @@
+raw	expected
+Oral Aspirin	aspirin oral
+Genes inhibiting	gene inhibit
+aspirin aspirin oral	aspirin oral
+  Oral   Aspirin  	aspirin oral
+51	51
+tnf-alpha	tnf-alpha
+ÄTHÉROGENIC	äthérogenic
+"  GENE  "	gene
+@EMPTY	@EMPTY

rust/tests/nlp_golden.rs

@@ -0,0 +1,32 @@
+//! Shared level-one normalization golden vectors.
+
+const GOLDEN: &str = include_str!("fixtures/nlp_golden.tsv");
+
+#[test]
+fn normalize_l1_matches_shared_golden_fixture() {
+    // WHY: fullmap ingestion uses normalize_l1 directly while Python queries use
+    // rs.normalize_terms; one checked-in fixture keeps those public paths tied
+    // to the same expected outputs without duplicating vectors in each test.
+    let mut rows = GOLDEN.lines();
+    assert_eq!(rows.next(), Some("raw\texpected"));
+
+    let mut count = 0usize;
+    for (line_number, line) in rows.enumerate() {
+        let (raw, expected) = line
+            .split_once('\t')
+            .unwrap_or_else(|| panic!("fixture line {} is not TSV", line_number + 2));
+        let raw = raw.strip_prefix("@EMPTY").map_or(raw, |_| "");
+        let expected = expected.strip_prefix("@EMPTY").map_or(expected, |_| "");
+        assert_eq!(
+            tablassert_rs::normalize_l1(raw),
+            expected,
+            "fixture line {}",
+            line_number + 2
+        );
+        count += 1;
+    }
+    assert!(
+        count >= 8,
+        "fixture must cover representative normalization cases"
+    );
+}

tests/test_nlp.py

@@ -1,9 +1,71 @@
 from __future__ import annotations
 
+import csv
+import random
+import time
+from pathlib import Path
+
 import polars as pl
 
 from tablassert.nlp import level_one, level_two
 
+GOLDEN_FIXTURE: Path = Path(__file__).resolve().parents[1] / "rust" / "tests" / "fixtures" / "nlp_golden.tsv"
+
+
+def test_level_one_golden_matches_rust_fixture() -> None:
+    """The Python Rust-normalizer path agrees with the shared golden fixture.
+
+    WHY: Rust fullmap keys and Python query terms must remain byte-for-byte
+    equivalent; reading one fixture from both integration suites prevents the
+    two sides from silently acquiring different normalization contracts.
+    """
+    with GOLDEN_FIXTURE.open(newline="", encoding="utf-8") as handle:
+        rows: list[tuple[str, str]] = [(raw, expected) for raw, expected in csv.reader(handle, delimiter="\t") if raw != "raw"]
+
+    rows = [("" if raw == "@EMPTY" else raw, "" if expected == "@EMPTY" else expected) for raw, expected in rows]
+    raw_terms: list[str] = [raw for raw, _expected in rows]
+    expected_terms: list[str] = [expected for _raw, expected in rows]
+    assert raw_terms
+    frame: pl.LazyFrame = pl.DataFrame({"name": raw_terms}).lazy()
+    result: pl.DataFrame = level_one(frame, "name").collect()
+    assert result["name"].to_list() == expected_terms
+
+
+def test_level_one_performance_one_million_terms() -> None:
+    """Normalize one million seeded multi-token terms within the Python bound.
+
+    WHY: level-one normalization is a hot path for large tabular inputs; this
+    absolute ceiling protects the Rust-backed batch implementation from
+    regressing to per-row Python work while keeping the workload stable and
+    independent of data generation, locale, network, or optional extras.
+    Calibration: ~0.6s on a 12-core dev box and a stable ~3.2s on 2-core CI
+    runners, so the 8.0s ceiling keeps >2x headroom on the slowest observed
+    environment while still failing immediately (minutes at this N) for the
+    regression classes it guards: `map_elements` per-row UDFs, serial loops,
+    and accidental quadratic token work.
+    """
+    term_parts: tuple[str, ...] = ("Aspirin", "genes", "inhibiting", "tnf-alpha", "oral", "tablets", "alpha", "51")
+    generator = random.Random(5005)
+    terms: list[str] = [f"{generator.choice(term_parts)} {generator.choice(term_parts)}" for _ in range(1_000_000)]
+    frame: pl.DataFrame = pl.DataFrame({"name": terms})
+    lazy_frame: pl.LazyFrame = frame.lazy()
+
+    level_one(lazy_frame, "name").collect()
+    elapsed_samples: list[float] = []
+    result: pl.DataFrame | None = None
+    for _ in range(3):
+        started: float = time.perf_counter()
+        result = level_one(lazy_frame, "name").collect()
+        elapsed_samples.append(time.perf_counter() - started)
+
+    assert result is not None
+    elapsed: float = sorted(elapsed_samples)[1]
+    assert result.height == 1_000_000
+    assert result.columns == ["name"]
+    assert result.schema["name"] == pl.String
+    assert result["name"].head(4).to_list() == ["tnf-alpha", "aspirin tnf-alpha", "51", "oral tablet"]
+    assert elapsed <= 8.0, f"level_one median took {elapsed:.3f}s for 1,000,000 terms (samples={elapsed_samples!r})"
+
 
 def test_level_one_strips_and_lowercases() -> None:
     """level one strips whitespace and lowercases."""