pretraining-data

(ACL 2026 Main) LLMSurgeon recovers the pretraining data mixture of any LLM from only its generated text — no weights, no training data. A calibrated domain classifier plus label-shift correction de-blurs biased predictions. Ships with LLMScan, a benchmark on 8 open-source LLMs.

[EMNLP 2026] HALvest-Contrastive: Retrieval-like Authorship Attribution with Patch-level Late Interaction

Curated papers on auditing the pretraining data of time series foundation models (TSFMs): contamination auditing, membership inference, information leakage, and LLM methods that transfer

LLM pretraining data mixture research insights

Streaming, sharded, resumable pipelines for preparing language-model training data

Notation as an unrecorded pre-training variable: the paper, the instruments, the corpus census, the reader/writer studies, and the pure-frame format. E. M. Freeburg, 2026.

Do quality filters pull benchmark questions into pretraining corpora? Injects MMLU, GSM8K, GPQA, ARC, HellaSwag, PIQA and TruthfulQA items into a corpus and ranks them with six quality classifiers, including DCLM, FineWeb-Edu and Gaperon.

Unofficial reproduction of Nemotron-CLIMB (CLustering-based Iterative Data Mixture Bootstrapping, NeurIPS 2025, arXiv:2504.13161) on Ascend NPUs — 5 rounds, ≈3,900 NPU-hours; learned mixtures beat every fixed-ratio baseline (+0.014–0.043 STEM); the selection mechanism is the weak link — fixed and validated live.