feat(distill): schema-uniform v2 records and an append-only outcome sink - #180
Merged
Merged
Conversation
…ry call
`--distill` records were not joinable to the run that produced them and were not
schema-stable, so the captured corpus could not carry an outcome or be loaded
reliably for fine-tuning.
Make every appended line self-describing and uniform:
- `RECORD_KEYS` is now the canonical ordered key set and every record emits every
key, with an explicit `null` where a value is unknown. This is load-bearing, not
cosmetic: `datasets.load_dataset("json")` infers its features from the first
block of the first file and raises `CastError` when a later block of an
append-only corpus introduces a column the inferred schema lacks.
- Add `record_type` (a content discriminator, so a training row is separable from
an outcome row without inspecting the filename), `schema_version`, and `run_id`
(`"<invocation_id>:<pmc_id>"`, the join key to a run's outcome, opened by the new
no-I/O `begin_run` scope).
- Derive `n_messages`, `input_tokens` and `output_tokens` onto the record itself so
per-example size is queryable without re-parsing `messages` — TRL's
`SFTConfig.max_length` defaults to 1024 and silently drops fully-masked examples.
This also closes a latent v1 drift: `cli.py` wraps the judge and reflexion models
with no `meta` while `agent.py` passes `pmc_id`, and `model_id` was added only when
recoverable, so one append-only file already held three different key sets. All
records are now uniform, pinned by
`test_records_are_schema_uniform_across_purposes`.
v2 is strictly additive: the seven pre-existing keys keep their names, types and
meanings, `call_index` semantics are unchanged, `distill.py` gains no third-party
import, and every write path stays fail-soft so recording can never break a batch.
Testing: `uv run pytest tests/test_distill.py -q -n0 --no-cov` -> 19 passed (12
pre-existing tests unmodified, +7 new); `uv run pytest -q` -> 1406 passed, 15
skipped, 0 failed (baseline 1399); `uv run pyright` -> 0 errors; `uv run ruff check .`
and `ruff format --check .` clean.
…es.ndjson The captured corpus still knew nothing about how each run ended, so an example could not be weighted by the quality of the KG it produced. Add `DistillRecorder.record_outcome()`: one schema-uniform line per supervisor run in a sibling `outcomes.ndjson` (never interleaved into `records.ndjson`, which stays a pure ChatML training file for Unsloth Studio and `distill-export`), joined back onto records at weigh time on `run_id`. - `OUTCOME_KEYS` (35 figures: coverage, Biolink validity incl. strict, demoted-edge rate, node/edge counts, unresolved/advice counts, attempts, config chars + sha256, provenance, QC, tool-call tallies, tokens, steps, judge score/dimensions, gate thresholds, versions) — every key on every line with an explicit `null` when unmeasured, because `datasets` infers features from the first block of the first file and raises `CastError` on a later column; only figures already computed at the run's status-decision point are captured (no new build/audit/KGX read). - Nested structs (`tool_calls`, `gate`, `versions`) are pinned to fixed sub-key tuples and never emitted as bare `null`, so a struct column cannot be silently JSON-demotioned to a string by `on_mixed_types="use_json"`. - `record_type`/`schema_version` are forced, `run_id` falls back to the open `begin_run` scope, `timestamp` to write time; outcome lines never advance `call_index`, which now counts only records that actually reached disk. - Capture stays zero-dependency and fail-soft: one guarded append path shared by `record` and `record_outcome`, failures logged, never raised into a run. Testing: `uv run pytest tests/test_distill.py -q -n0 --no-cov` -> 24 passed (19 unmodified + 5 new); `uv run pytest -q` -> 1411 passed, 15 skipped, 0 failed; `uv run pyright` -> 0 errors; ruff check + format --check clean.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
SkyeAv
added this pull request to stack #186
September 15, 2026 17:55
SkyeAv
changed the title
SkyeAv
marked this pull request as ready for review
September 15, 2026 17:55
SkyeAv
added a commit
that referenced
this pull request
Sep 15, 2026
Cut 19.0.0 and bump the package version in pyproject.toml, CITATION.cff, and uv.lock. Major: five breaking changes ship. The `tablassert` console command now requires the optional `[cli]` extra, since `cyclopts` and `rich` left the base install (#187). Level-one normalization redefines fullmap keys as cleaned Unicode-lowercase, Porter2-normalized, byte-ordered token sets and moves the database to schema v6, so every schema-v5 fullmap is rejected and must be rebuilt (#171, #172, #174, #175, #176). `build-fullmap --aria2c` is gone -- aria2c is used automatically whenever the `[aria2]` extra is installed (#178) -- and `--taxon-allowlist` is gone because the built-in top-100 experimental-taxon allowlist now applies to every build, guarding both reuse paths by `META.taxon_allowlist` identity (#153). Logging now requires the `[log]` extra and is fully disabled without it (#152). Features: `tablassert.net` is a new stdlib-only transient/permanent classification and retry seam shared by the agent and the BABEL downloader (#163, #164), with one bounded jittered retry layer across the inner agent, reflexion, and judge (#167) and machine-readable `error_code` values on skipped checkpoint records (#166). `build-kg` gained an automatic ephemeral shared-prefix TCode cache keyed by content-addressed XXH64 op digests (#155-#159, #161). The `--distill` corpus became schema-uniform v2 with a sibling `outcomes.ndjson`, a deterministic `RewardConfig`-tunable reward, and the new zero-dependency `tablassert distill-weigh` command (#180-#183). `CLASS_FIELD_OVERRIDES` grants the DAKP sparse qualifier stack on the pinned association classes (#188). Fixes: `distill-export` partitions its corpus by content and unions the schema across record files, so a mixed v1/v2 directory no longer CastErrors or silently stringifies a column (#184); BABEL retry warnings survive the loguru sink; PMC downloads are idempotent, atomic, and bounded-parallel (#165); reflexion and judge prompts are hard-bounded (#168). Changelog: - versioned the Unreleased section as 19.0.0 - 2026-09-15 - merged three duplicate `### Added` blocks into one - moved the two `BREAKING:` entries from `### Changed` into `### Breaking Changes` and added a `**Migration:**` note to all five - added PR links to all 17 entries, which carried none - added the missing entries: the distill v2 corpus/reward/distill-weigh stack (#180-#183), the distill-export schema-drift fix (#184), the TCode run-cache detail (#155-#161), and the README badge removal (#154) Docs: none needed here. Every shipped doc change landed with its own PR, and the docs source-of-truth gate passes against the bumped tree. Testing: - make check -> exit 0 - uv run ruff check . -> All checks passed! - uv run pyright -> 0 errors, 0 warnings, 0 informations - uv run pytest -> 1622 passed, 52 skipped - cargo test --manifest-path rust/Cargo.toml -> 156 passed, 0 failed - cargo clippy --all-targets -- -D warnings -> clean - uv lock --check -> resolved 169 packages, lock current - uv run mkdocs build --strict -> exit 0 - docs SSOT + CLI coverage after the changelog edit -> 204 passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gives
--distilla stable, self-describing corpus contract so downstream tooling (and a future training-data pipeline) can trust the files without sniffing them: every record line carries the same canonical key set, and each run's outcome lands in a sibling file instead of being inferred fromstate.json.Schema-uniform v2 records
records.ndjsonnow carries exactly the 13RECORD_KEYS(record_type,schema_version,run_id,timestamp,purpose,call_index,messages,token_usage,input_tokens,output_tokens,n_messages,pmc_id,model_id),SCHEMA_VERSION = 2, with an explicitnullwhere a value is unknown — a key never first appears partway down an append-only file, which is what made v1 corpora unexportable.begin_run()opens a per-article scope and stamps every record withrun_id("<invocation_id>:<pmc_id>") pluspmc_id/n_messages/token scalars so a corpus can be grouped and filtered without re-parsingmessages.metakeys are logged as a warning instead of silently widening the schema; the previous ChatML-only v1 shape still parses (readers tolerate supersets, writers never emit them).Append-only outcome sink
outcomes.ndjson: a sibling file in the same directory gainsrecord_outcome()+canonical_struct()with a fixed 35-keyOUTCOME_KEYSset (build verdict scalars, tool-call tallies, gate thresholds, package versions) — schema-uniform the same way, so outcome lines are joinable, exportable, and diffable._append_json_line, which returnsbooland only advancescall_indexwhen a line actually lands on disk — a failed write is logged, never raised, and never desynchronizes the counter.Design
records.ndjsonstays append-only forever; outcomes go to the sibling file, never a rewrite.Testing
uv run pytest tests/test_distill.py -q -n0 --no-cov→ 26 passed (v2 shape, explicit-null invariants, unknown-key warning, guarded-append counter semantics, v1 back-compat).uv run pytest -q→ 1512 passed, 15 skipped, 0 failed;uv run pyright→ 0 errors;ruff check+ruff format --check→ clean.