feat(distill): make distill-export schema-robust across a drifting corpus - #184
Merged
Merged
Conversation
…rift
An append-only corpus is the whole point of `--distill`, but
`datasets.load_dataset("json")` infers its column schema from the FIRST BLOCK
OF THE FIRST FILE ONLY and then casts every later table to it — so the moment
v1 and v2 records (or any two differently-shaped files) share a directory,
export either raises CastError or, worse, silently JSON-encodes a
varying-type column into a string. Outcome files sitting in the same corpus
directory made it worse: a record-shaped corpus could not be exported at all
without manually moving files.
`distill-export` now:
- partitions the directory by CONTENT (`is_outcome_file` inspects the first
non-blank line's record_type, so a renamed file still partitions correctly)
and loads only record files; an outcomes-only directory exits 2 with a
message distinguishing it from an empty one;
- reads every record file with the pure `read_ndjson`, unions the corpus
schema with `normalize_rows`, writes ONE uniform temporary NDJSON (in a
system temp dir, never inside --distill-dir, always removed via
try/finally), and passes only that single file to `load_dataset` — the
schema now comes from the whole corpus, not the first 10 lines of one
file. A v1-only corpus still exports: normalization collapses it to its
own union;
- rejects type conflicts loudly: `detect_type_conflicts` runs BEFORE the
[distill] preflight and datasets import, so a column that varies type
(e.g. pmc_id int vs str) exits 2 naming the key and the types with the
why, instead of letting datasets silently stringify the column;
- fails loud on record files that exist but contain zero rows (previously a
bare StopIteration from inside datasets' JSON reader).
Flags, defaults, exit codes, preflight ordering, and the success-message
shape are unchanged; the message's file count now correctly names record
files only. `distill-weigh` output exports with no new flag.
docs/cli.md `## distill-export` prose updated in the same commit (outcome
skip, normalization, type-conflict rejection); option table unchanged.
Testing: `uv run pytest tests/test_cover_cli.py -q -n0 --no-cov` -> 54 passed
(4 new base-env tests: outcome skip via stubbed datasets, outcomes-only exit
2, pure normalize step across v1+v2, type-conflict exit 2);
`tests/test_distill.py tests/test_distill_weigh.py` -> 64 passed (2 new
[distill]-gated round-trips: v1-file-first corpus that naive load_dataset
CastErrors on; weigh->export composition — both genuinely run locally);
docs SSOT gate -> 167 passed; `uv run pytest -q` -> 1497 passed, 15 skipped,
0 failed; pyright 0 errors; ruff check + format clean; mkdocs strict clean.
Independent Tier-2 review verified all seven acceptance points against
source (cli.py:1384-1426) and reproduced a real-CLI artifact: mixed corpus
exit 0 with union schema (v1 rows explicit-null on run_id), type-conflict
exit 2 naming key+types, outcomes-only exit 2, no temp residue, corpus dir
untouched.
SkyeAv
added this pull request to stack #186
September 15, 2026 17:55
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
SkyeAv
changed the title
SkyeAv
marked this pull request as ready for review
September 15, 2026 17:55
SkyeAv
added a commit
that referenced
this pull request
Sep 15, 2026
Cut 19.0.0 and bump the package version in pyproject.toml, CITATION.cff, and uv.lock. Major: five breaking changes ship. The `tablassert` console command now requires the optional `[cli]` extra, since `cyclopts` and `rich` left the base install (#187). Level-one normalization redefines fullmap keys as cleaned Unicode-lowercase, Porter2-normalized, byte-ordered token sets and moves the database to schema v6, so every schema-v5 fullmap is rejected and must be rebuilt (#171, #172, #174, #175, #176). `build-fullmap --aria2c` is gone -- aria2c is used automatically whenever the `[aria2]` extra is installed (#178) -- and `--taxon-allowlist` is gone because the built-in top-100 experimental-taxon allowlist now applies to every build, guarding both reuse paths by `META.taxon_allowlist` identity (#153). Logging now requires the `[log]` extra and is fully disabled without it (#152). Features: `tablassert.net` is a new stdlib-only transient/permanent classification and retry seam shared by the agent and the BABEL downloader (#163, #164), with one bounded jittered retry layer across the inner agent, reflexion, and judge (#167) and machine-readable `error_code` values on skipped checkpoint records (#166). `build-kg` gained an automatic ephemeral shared-prefix TCode cache keyed by content-addressed XXH64 op digests (#155-#159, #161). The `--distill` corpus became schema-uniform v2 with a sibling `outcomes.ndjson`, a deterministic `RewardConfig`-tunable reward, and the new zero-dependency `tablassert distill-weigh` command (#180-#183). `CLASS_FIELD_OVERRIDES` grants the DAKP sparse qualifier stack on the pinned association classes (#188). Fixes: `distill-export` partitions its corpus by content and unions the schema across record files, so a mixed v1/v2 directory no longer CastErrors or silently stringifies a column (#184); BABEL retry warnings survive the loguru sink; PMC downloads are idempotent, atomic, and bounded-parallel (#165); reflexion and judge prompts are hard-bounded (#168). Changelog: - versioned the Unreleased section as 19.0.0 - 2026-09-15 - merged three duplicate `### Added` blocks into one - moved the two `BREAKING:` entries from `### Changed` into `### Breaking Changes` and added a `**Migration:**` note to all five - added PR links to all 17 entries, which carried none - added the missing entries: the distill v2 corpus/reward/distill-weigh stack (#180-#183), the distill-export schema-drift fix (#184), the TCode run-cache detail (#155-#161), and the README badge removal (#154) Docs: none needed here. Every shipped doc change landed with its own PR, and the docs source-of-truth gate passes against the bumped tree. Testing: - make check -> exit 0 - uv run ruff check . -> All checks passed! - uv run pyright -> 0 errors, 0 warnings, 0 informations - uv run pytest -> 1622 passed, 52 skipped - cargo test --manifest-path rust/Cargo.toml -> 156 passed, 0 failed - cargo clippy --all-targets -- -D warnings -> clean - uv lock --check -> resolved 169 packages, lock current - uv run mkdocs build --strict -> exit 0 - docs SSOT + CLI coverage after the changelog edit -> 204 passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An append-only corpus is the point of
--distill, butdatasets.load_dataset("json")infers its column schema from the first block of the first file only and then casts every later table to it — so the moment v1 and v2 records (or any two differently-shaped files) share a directory, export either raisesCastErroror, worse, silently JSON-encodes a varying-type column into a string. Outcome files sitting in the same corpus directory made a mixed corpus unexportable outright. This hardensdistill-exportto treat schema drift as a first-class case.Content-based partition
iter_record_files+is_outcome_file(inspects the first non-blank line'srecord_type) partition the directory, so a renamed outcome file still partitions correctly; only record files are loaded.Corpus-wide schema union
read_ndjson, unioned vianormalize_rows(first-appearance key order), and written as one uniform temporary NDJSON in a system temp dir — never inside--distill-dir, removed viatry/finallyon success and failure — and that single file is the only thing passed toload_dataset. A v1-only corpus still exports: normalization collapses it to its own union.Type-conflict rejection before the extra
detect_type_conflictsruns beforeextras.require("distill")and thedatasetsimport: a key with more than one distinct non-null type exits 2 naming the key and the types with the why (datasets would silently JSON-encode a varying-type column into a string). Base-environment users get the actionable data error, not an extras error.StopIterationfrom inside datasets' JSON reader.Compatibility
[distill]preflight ordering, and the success-message shape are unchanged; the message's file count now correctly names record files only.distill-weighoutput (which legitimately carries extra columns) exports with no new flag — the weigh → export composition works as-is.Docs
docs/cli.md## distill-exportprose updated (outcome skip, normalization, type-conflict rejection); option table unchanged — no flags added.Testing
uv run pytest tests/test_cover_cli.py -q -n0 --no-cov→ 54 passed (4 new base-env tests: outcome skip via stubbeddatasets, outcomes-only exit 2, pure normalize across v1+v2, type-conflict exit 2).uv run pytest tests/test_distill.py tests/test_distill_weigh.py -q -n0 --no-cov→ 64 passed incl. 2 new[distill]-gated round-trips: a v1-file-first corpus that naiveload_datasetCastErrors on exports with the union schema, and the weigh→export composition.uv run pytest -q→ 1512 passed, 15 skipped, 0 failed;uv run pyright→ 0 errors;mkdocs build --strict→ exit 0.