fix(training): exclude unavailable metrics from checkpoint ranking

[LauraGPT]

Summary

Fix the verified checkpoint-selection defect reported in #3388. This PR does not close that issue or claim to solve catastrophic forgetting.

• Require complete, finite validation metrics across every contributing batch/rank. Missing/None/nonfinite acc is not zero, and acc_rich is not ASR accuracy. Unavailable selected metrics are excluded from best, keep-N ranking and metric-based averaging, with a rank-zero warning.
• Preserve genuine zero-accuracy scores and existing acc/loss tie semantics. Keep explicit criterion selection; do not silently switch to loss or reinterpret historical zero scores.
• Use consistent validation collectives, including asymmetric missing metrics, nonfinite loss, empty ranks and same-name best revalidation. Keep per-batch logs current.
• Persist the actual best and ranked candidates before writing checkpoint files. All new epoch/latest/best writes precede deletion of a pruned candidate; DeepSpeed save decisions/metadata are synchronized while engine writes remain collective. This is not an atomic multi-file rollback guarantee.
• Reject criterion changes during resume. Reject unavailable same-name revalidation of the current best before changing its histories; invalidate stale nonbest entries before serialization. Filter nonfinite scores and missing checkpoint files before selecting N for averaging.
• Set the maintained SenseVoice recipe explicitly to validation loss. Update both continual-finetuning guides with independently audited references, loss versus CER/retention, fresh output directories for changed criteria, and language-token/downstream allowlist checks. Remove the old CER table after the reporter withdrew the new-domain evaluation.
• Add dedicated CPU checkpoint-selection CI.

Verification

On ind-gpu8, Python 3.12 and PyTorch 2.10.0+cu128 with CPU execution:

python -m pytest -q \
  tests/test_checkpoint_metric_presence.py \
  tests/test_checkpoint_metric_distributed.py \
  tests/test_trainer_ds_unvalidated_ckpt.py \
  tests/test_train_ds_distributed_config.py \
  tests/test_sensevoice_continual_finetuning_docs.py

85 passed, no skips. Adjacent training-document/API-signature contracts: 12 passed. Shell syntax and git diff --check pass. Independent static review found no remaining actionable issue.

The tests call actual validation and checkpoint save/load paths with real CPU tensors and files. With validation losses [3, 1, 2], explicit loss selection retains epoch 2 as best; averaging retained epochs 2/3 produces the expected parameter mean. They also cover persisted ranking, resume, self-pruning and simulated latest/best write failures retaining prior candidates.

Real two-rank CPU Gloo tests exercise asymmetric inputs and post-save rank differences. The original implementation reproduced a collective-size mismatch/SIGABRT; the fixed implementation completes and agrees on valid means and unavailable metrics. The initial metric regression was 41 failing/6 passing against the original source, followed by targeted RED/GREEN tests for review findings.

Boundaries

• Acoustic model outputs and the DeepSpeed engine writer boundary are simulated. This is not a full DeepSpeed/FSDP/GPU acoustic-training or CER validation.
• Validation loss is the total objective, not a guarantee of minimum CER or retained-language accuracy. The averaged model still needs independent evaluation.
• Unranked checkpoints remain on disk and can exceed keep-N. Old fabricated-zero histories and lagged metadata are not automatically repaired; previously pruned weights cannot be recovered. Use a new output directory when changing the criterion, and independently reevaluate retained checkpoints from affected runs.
• No universal replay ratio, unused-language-token claim, private CER result, or undocumented pretraining-data proportion is introduced.