feat(distill): add tablassert distill-weigh to emit weighted training rows

[SkyeAv]

Adds a seventh subcommand, tablassert distill-weigh, that composes the corpus into a trainable dataset: join records.ndjson + outcomes.ndjson on run_id, score each matched row, apply a selection policy, and write flat training rows plus a reproducibility manifest. The missing step between --distill and LoRA/QLoRA SFT — until now nothing turned a recorded corpus into weighted rows.

Weigh workflow

• tablassert distill-weigh --distill-dir <corpus> --out train.ndjson with --policy threshold|best-of-n|replication (-p, default threshold), --threshold 0.75, --top-n 2, --replication-k 2, --reward-config YAML/JSON override, --edge-ref breadth-reference override, --manifest (default <out>.manifest.json).
• Filters: --purpose agent|all|<live> (judge and reflexion turns are error-recovery trajectories, not teaching signal); --final-call-only keeps each run's highest call_index and retains rows it cannot rank (warns + counts unrankable_final_call instead of silently dropping them).
• Edge reference: --edge-ref > config edge_ref > corpus median over the same deduplicated outcome pool (a superseded duplicate can no longer shift weights); the resolved value and its source (derived/overridden/null + warning) land in the manifest.

Output contract

• One row per filtered record, keys exactly TRAIN_ROW_KEYS in order (canonical record keys + outcome_matched + outcome_* columns + selection keys); unknown record keys append after the canonical set rather than being dropped.
• Deterministic manifest: resolved reward config, edge_ref + source, policy knobs, join stats, selected_count, selected_zero_weight (best-of-n has no weight floor — an operator must see gated-failed trajectories it selects), unmatched_count, nested distinct diversity counters before/after selection (fixed to read the flattened outcome_config_yaml_sha256 column), absolute source_files. Two identical runs produce byte-identical output and manifest — no timestamps.
• Atomic writes via sibling temp + os.replace.

Fail-loud guards (exit 2, actionable stderr)

• Corpus integrity: derived training output already inside --distill-dir (marker keys) and mixed record/outcome files are rejected by content, not filename — a prior train.ndjson re-ingested would double every example.
• Path validation: --out/--manifest naming a directory, colliding, or landing inside --distill-dir (export ingests every *.ndjson there).
• Input validation: missing/empty corpus, malformed NDJSON naming file + line, matched == 0 with the pre-v2 hint, unknown policy/purpose/knob values, unreadable reward config.

Docs

• docs/cli.md: seven commands, table row, full ## distill-weigh section with every flag/default and the manifest contents — same commit, per the repo's docs-SSOT gates. llms.txt updated to match.

Design

• Zero optional dependencies — stdlib + pyyaml only; no [distill] extra required to weigh a corpus.
• Not RLHF: this is data selection for supervised fine-tuning; no trainer, reward model, or RL import exists anywhere in the command.

Testing

• uv run pytest tests/test_distill_weigh.py -q -n0 --no-cov → 36 passed, 0 skipped (all three policies, both edge_ref override sources, retention counts, exit-2 paths, byte-reproducibility).
• Docs gates: tests/test_docs_cli_coverage.py tests/test_docs_source_of_truth.py tests/test_docs_examples.py tests/test_agent_docs.py → 167 passed.
• Full gate at the stack tip: uv run pytest -q → 1512 passed, 15 skipped, 0 failed; uv run pyright → 0 errors; mkdocs build --strict → exit 0.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 57d933e0-0980-418e-92fc-968bc93c449c

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}