lora.method: pissa with DPO or KTO corrupts both policy and reference at trainer init; the saved artefact differs from the model the in-run gates evaluated (CR-A03-007) #405

Wave 1 · theme D1-5: Training, merging and export correctness · highest severity High · 1 finding(s) · effort small

Summary

An operator selects PiSSA for DPO. Training starts from a degraded model with a degraded KL anchor, the gates pass or fail a model that is never shipped, and the shipped adapter behaves differently again.

Primary finding

CR-A03-007 — lora.method: pissa with DPO or KTO corrupts both policy and reference at trainer init; the saved artefact differs from the model the in-run gates evaluated

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. ForgeLM hands TRL an already-wrapped PeftModel (model.py:609), and TRL 1.7.1 then calls model.add_adapter('ref', default_config) (trl dpo_trainer.py:616-635; kto_trainer.py:569). With init_lora_weights='pissa' (model.py:539-540), adding the second adapter runs PiSSA initialisation again, which subtracts more principal components from the base weights in place. TRL then copies the default adapter into 'ref'. The policy and the reference both stop matching the pretrained model. After training, the in-memory model (which the safety, judge and benchmark gates evaluate) differs from the adapter reloaded by chat, export or safety-eval, because the reload runs PiSSA init only once.

Where.

• forgelm/model.py:539-540
• forgelm/model.py:609
• forgelm/trainer.py:1015
• forgelm/trainer.py:1015-1022 (trainer_kwargs with no peft_config)
• forgelm/trainer.py:1711-1731
• forgelm/inference.py:108-114
• forgelm/trainer.py:1053-1056
• forgelm/cli/_training.py:148
• forgelm/cli/_pipeline.py:1564-1566
• forgelm/config.py:370

Evidence.

• t9/dpo_pissa.py and kto_pissa.py (trl 1.7.1): the initial PiSSA policy matches the base (6.3e-7). After DPOTrainer or KTOTrainer init, the policy minus base is 0.486 and the ref minus base is 0.486 (ref == policy). The plain-LoRA control is 0.0. In t9/pissa_dpo_cmp.py, after 1 DPO step, the in-memory model minus base is 0.549 and the model reloaded by forgelm.inference.load_model differs from the in-memory model by 0.547. The SFT control (pissa_cmp.py) reloads identically (0.0).

Recommended fix. Pass the peft_config to the TRL trainer instead of a pre-wrapped model for preference trainers, or reject lora.method=pissa for dpo/kto/orpo/simpo until conversion is handled (for example PEFT's path_initial_model_for_weight_conversion). Add a logit-equality test before and after trainer init.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}