diff --git a/CHANGELOG.md b/CHANGELOG.md index b315bc0e..04df31e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,9 @@ All notable changes to this project are documented in this file. ## Unreleased +### Breaking Changes +- **Level-one normalization now defines fullmap keys by cleaned Unicode-lowercase, whitespace-tokenized, Porter2-normalized terms, with duplicate removal and byte-wise token ordering.** Existing fullmap databases are schema-v5 and are rejected by the current resolver; rebuild fullmaps before use. Preferred-name ranking now compares normalized forms under the same level-one semantics. + ### Changed - **BREAKING: `build-fullmap` always applies the built-in top-100 experimental-taxon allowlist, and `--taxon-allowlist` is removed.** The checked-in `src/tablassert/data/experimental_taxa.yaml` filter is no longer opt-in: every source build passes those 100 NCBI taxon IDs to Rust, and the resulting database records the same deterministic `META.taxon_allowlist` identity introduced in 18.1.0. The identity now also guards both reuse paths, so an unfiltered database can never satisfy an allowlisted `build-fullmap` invocation. A prebuilt archive whose recorded identity is absent or different fails extraction validation before anything is renamed, surfacing as the existing `PrebuiltFullmapUnavailable` fallback to a filtered BABEL build; an existing file at `--output` is reused only when its identity matches, and is otherwise rebuilt with a warning. Scripts passing `--taxon-allowlist` or `--no-taxon-allowlist` must drop the flag — parsing now fails with an unknown-option error. `build_fullmap_pipeline`, `fetch_prebuilt_fullmap`, and `_extract_prebuilt_fullmap` accept the IDs through their existing/extended `taxon_allowlist` parameter, and the Rust extension exposes `taxon_allowlist_identity` plus `fullmap_taxon_allowlist_identity` so Python can compare identities without duplicating the encoding. - **Logging now requires the optional `log` extra and is completely disabled without it.** Removed the stdlib-backed loguru fallback, startup missing-extra warning, and base-install creation of `.tablassert/log/`. Installs without loguru emit no file, progress, or warning logs; `pip install "tablassert[log]"` remains the way to enable loguru-backed logging. diff --git a/docs/api/fullmap.md b/docs/api/fullmap.md index 7f02e2ef..f7f8b772 100644 --- a/docs/api/fullmap.md +++ b/docs/api/fullmap.md @@ -186,11 +186,16 @@ print(result.select(["gene", "gene_name", "gene_category"])) ### NLP Processing Levels -`resolve()` requires that `level_one` and `level_two` have been applied to the LazyFrame before calling it: +`resolve()` requires that `level_one` and `level_two` have been applied to the LazyFrame before calling it. Level one is the canonical key used by the fullmap build and query paths: **`level_one` output** (column: `col`): -- Whitespace stripped, lowercased -- Queried first; preferred for acronyms and gene symbols +- Cleans surrounding matching quotes and whitespace, then applies Unicode lowercase. +- Splits on whitespace, stems only tokens made entirely of ASCII letters with the English Porter2 stemmer, and passes digit-bearing, punctuation-bearing, and non-ASCII tokens through unchanged after lowercasing. +- Removes duplicate tokens, sorts the remaining tokens by their UTF-8 byte values, and joins them with one ASCII space. +- Preserves nulls in Python LazyFrame columns; empty and whitespace-only values become the empty string. +- Is queried first; preferred-name ranking compares these normalized forms, so a fullmap database built with different level-one keys cannot be reused. + +Schema-v5 fullmap databases are rejected by the current resolver. Rebuild existing fullmaps to produce schema v6 before using them with this level-one contract. **`level_two` output** (column: `col + "_two"`): - All non-word characters removed (`\W+` → `""`) from the `level_one` result diff --git a/docs/api/lib.md b/docs/api/lib.md index 150bc7db..8deef39c 100644 --- a/docs/api/lib.md +++ b/docs/api/lib.md @@ -156,7 +156,9 @@ for row in result: ### NLP Processing -`resolve_many()` applies `level_one` (strip + lowercase → column `{col}`) and `level_two` (remove `\W+` → column `{col}_two`) before resolution. Level one (case-insensitive exact) is preferred; level two is the fallback for terms with punctuation or special characters. +`resolve_many()` applies the same level-one normalizer used to build and query the fullmap, writing its result to `{col}`, then applies `level_two` to write `{col}_two`. Level one cleans surrounding quotes and whitespace, lowercases with Unicode rules, tokenizes on whitespace, stems only all-ASCII-alphabetic tokens with English Porter2, passes digit- and punctuation-bearing tokens through, removes duplicate tokens, sorts tokens by UTF-8 byte value, and joins them with one space. Nulls remain null in Python LazyFrame columns; empty values remain empty. Level one is preferred; level two is the fallback for terms with punctuation or special characters. + +The resolver requires a schema-v6 fullmap. Existing schema-v5 databases are rejected and must be rebuilt. Preferred-name ranking compares normalized forms under the same level-one rules. ### Error Handling diff --git a/rust/src/nlp.rs b/rust/src/nlp.rs index c62a54d7..ec2ea6d7 100644 --- a/rust/src/nlp.rs +++ b/rust/src/nlp.rs @@ -121,6 +121,7 @@ mod tests { use super::{normalize_l1, normalize_terms}; use pyo3::prelude::*; use std::borrow::Cow; + use std::time::Instant; #[test] fn single_token_normalizes_to_itself() { @@ -257,4 +258,74 @@ mod tests { assert!(empty.is_empty()); }); } + + #[test] + fn normalize_terms_performance_one_million_terms() { + // WHY: level-one normalization is a hot path for large tabular inputs; + // this absolute ceiling protects the rayon batch/core path from + // regressing to serial work. The seeded, multi-token workload is + // generated before timing and uses only stable ASCII terms, so the gate + // is independent of locale, network, and optional Python dependencies. + // Calibration: ~0.8s on a 12-core dev box and 3.6-4.9s per sample on + // 2-core CI runners (also contending with parallel tests), so the 12s + // ceiling keeps >2x headroom on the slowest observed environment while + // still failing immediately for the guarded regression classes: loss of + // rayon parallelism, per-token quadratic work, and allocation + // regressions on the batch path. + // + let vocabulary = [ + "Aspirin", + "genes", + "inhibiting", + "tnf-alpha", + "oral", + "tablets", + "alpha", + "51", + ]; + let mut seed = 5005_u64; + let terms: Vec = (0..1_000_000) + .map(|_| { + seed = seed.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1); + let first = vocabulary[(seed % vocabulary.len() as u64) as usize]; + seed = seed.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1); + let second = vocabulary[(seed % vocabulary.len() as u64) as usize]; + format!("{first} {second}") + }) + .collect(); + + let first_term = terms[0].clone(); + pyo3::Python::initialize(); + pyo3::Python::attach(|py| { + let func = pyo3::wrap_pyfunction!(normalize_terms, py).unwrap(); + let warmup: Vec = func + .call1((terms[..1024].to_vec(),)) + .unwrap() + .extract() + .unwrap(); + assert_eq!(warmup.len(), 1024); + + let timed_inputs: Vec> = (0..3).map(|_| terms.clone()).collect(); + let mut elapsed_samples: Vec = Vec::with_capacity(3); + let mut output: Vec = Vec::new(); + for input in timed_inputs { + let started = Instant::now(); + output = func.call1((input,)).unwrap().extract().unwrap(); + elapsed_samples.push(started.elapsed().as_secs_f64()); + } + elapsed_samples.sort_by(f64::total_cmp); + let elapsed = elapsed_samples[1]; + + println!( + "normalize_terms median took {elapsed:.3}s for 1,000,000 terms (samples={elapsed_samples:?})" + ); + assert_eq!(output.len(), 1_000_000); + assert!(output.iter().all(|term| !term.is_empty())); + assert_eq!(output[0], normalize_l1(&first_term)); + assert!( + elapsed <= 12.0, + "normalize_terms median took {elapsed:.3}s for 1,000,000 terms (samples={elapsed_samples:?})" + ); + }); + } } diff --git a/rust/tests/fixtures/nlp_golden.tsv b/rust/tests/fixtures/nlp_golden.tsv new file mode 100644 index 00000000..adc48c30 --- /dev/null +++ b/rust/tests/fixtures/nlp_golden.tsv @@ -0,0 +1,10 @@ +raw expected +Oral Aspirin aspirin oral +Genes inhibiting gene inhibit +aspirin aspirin oral aspirin oral + Oral Aspirin aspirin oral +51 51 +tnf-alpha tnf-alpha +ÄTHÉROGENIC äthérogenic +" GENE " gene +@EMPTY @EMPTY diff --git a/rust/tests/nlp_golden.rs b/rust/tests/nlp_golden.rs new file mode 100644 index 00000000..719ef567 --- /dev/null +++ b/rust/tests/nlp_golden.rs @@ -0,0 +1,32 @@ +//! Shared level-one normalization golden vectors. + +const GOLDEN: &str = include_str!("fixtures/nlp_golden.tsv"); + +#[test] +fn normalize_l1_matches_shared_golden_fixture() { + // WHY: fullmap ingestion uses normalize_l1 directly while Python queries use + // rs.normalize_terms; one checked-in fixture keeps those public paths tied + // to the same expected outputs without duplicating vectors in each test. + let mut rows = GOLDEN.lines(); + assert_eq!(rows.next(), Some("raw\texpected")); + + let mut count = 0usize; + for (line_number, line) in rows.enumerate() { + let (raw, expected) = line + .split_once('\t') + .unwrap_or_else(|| panic!("fixture line {} is not TSV", line_number + 2)); + let raw = raw.strip_prefix("@EMPTY").map_or(raw, |_| ""); + let expected = expected.strip_prefix("@EMPTY").map_or(expected, |_| ""); + assert_eq!( + tablassert_rs::normalize_l1(raw), + expected, + "fixture line {}", + line_number + 2 + ); + count += 1; + } + assert!( + count >= 8, + "fixture must cover representative normalization cases" + ); +} diff --git a/tests/test_nlp.py b/tests/test_nlp.py index 00440b73..a39389b2 100644 --- a/tests/test_nlp.py +++ b/tests/test_nlp.py @@ -1,9 +1,71 @@ from __future__ import annotations +import csv +import random +import time +from pathlib import Path + import polars as pl from tablassert.nlp import level_one, level_two +GOLDEN_FIXTURE: Path = Path(__file__).resolve().parents[1] / "rust" / "tests" / "fixtures" / "nlp_golden.tsv" + + +def test_level_one_golden_matches_rust_fixture() -> None: + """The Python Rust-normalizer path agrees with the shared golden fixture. + + WHY: Rust fullmap keys and Python query terms must remain byte-for-byte + equivalent; reading one fixture from both integration suites prevents the + two sides from silently acquiring different normalization contracts. + """ + with GOLDEN_FIXTURE.open(newline="", encoding="utf-8") as handle: + rows: list[tuple[str, str]] = [(raw, expected) for raw, expected in csv.reader(handle, delimiter="\t") if raw != "raw"] + + rows = [("" if raw == "@EMPTY" else raw, "" if expected == "@EMPTY" else expected) for raw, expected in rows] + raw_terms: list[str] = [raw for raw, _expected in rows] + expected_terms: list[str] = [expected for _raw, expected in rows] + assert raw_terms + frame: pl.LazyFrame = pl.DataFrame({"name": raw_terms}).lazy() + result: pl.DataFrame = level_one(frame, "name").collect() + assert result["name"].to_list() == expected_terms + + +def test_level_one_performance_one_million_terms() -> None: + """Normalize one million seeded multi-token terms within the Python bound. + + WHY: level-one normalization is a hot path for large tabular inputs; this + absolute ceiling protects the Rust-backed batch implementation from + regressing to per-row Python work while keeping the workload stable and + independent of data generation, locale, network, or optional extras. + Calibration: ~0.6s on a 12-core dev box and a stable ~3.2s on 2-core CI + runners, so the 8.0s ceiling keeps >2x headroom on the slowest observed + environment while still failing immediately (minutes at this N) for the + regression classes it guards: `map_elements` per-row UDFs, serial loops, + and accidental quadratic token work. + """ + term_parts: tuple[str, ...] = ("Aspirin", "genes", "inhibiting", "tnf-alpha", "oral", "tablets", "alpha", "51") + generator = random.Random(5005) + terms: list[str] = [f"{generator.choice(term_parts)} {generator.choice(term_parts)}" for _ in range(1_000_000)] + frame: pl.DataFrame = pl.DataFrame({"name": terms}) + lazy_frame: pl.LazyFrame = frame.lazy() + + level_one(lazy_frame, "name").collect() + elapsed_samples: list[float] = [] + result: pl.DataFrame | None = None + for _ in range(3): + started: float = time.perf_counter() + result = level_one(lazy_frame, "name").collect() + elapsed_samples.append(time.perf_counter() - started) + + assert result is not None + elapsed: float = sorted(elapsed_samples)[1] + assert result.height == 1_000_000 + assert result.columns == ["name"] + assert result.schema["name"] == pl.String + assert result["name"].head(4).to_list() == ["tnf-alpha", "aspirin tnf-alpha", "51", "oral tablet"] + assert elapsed <= 8.0, f"level_one median took {elapsed:.3f}s for 1,000,000 terms (samples={elapsed_samples!r})" + def test_level_one_strips_and_lowercases() -> None: """level one strips whitespace and lowercases."""