feat: add Intern-Decision text training cookbook on Ascend

README.md

@@ -406,6 +406,7 @@ foundation for building customizable, enterprise-grade training services.
 | Component Type | Component Link                                                                                           | Component Function                                                                      | Author              |
 | -------------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ------------------- |
 | Patch          | [qwen3_moe_transformers4_patch](https://www.modelscope.cn/models/twinkle-kit/qwen3_moe_transformers4_patch) | Fixes Qwen3 MoE model hang issue during FSDP2 training, effective for transformers==4.x | ModelScope Official |
+| Training recipe | [Intern-Decision on Ascend](cookbook/transformers/intern_decision_npu/README.md) | Text-only masked decision SFT, joint-field supervision and FSDP2 checkpoint recovery | — |
 
 ## Contributions
 

cookbook/transformers/intern_decision_npu/.gitignore

@@ -0,0 +1,9 @@
+/raw/
+/prepared/
+/decision_data/
+/sources/
+/results/
+/outputs/
+/data-audit.json
+__pycache__/
+*.log

cookbook/transformers/intern_decision_npu/JOINT_REPRODUCTION.md

@@ -0,0 +1,61 @@
+# Joint decision training with Twinkle
+
+## Objective and scope
+
+Start from Qwen3.5-4B instruction weights, train all language parameters and freeze vision/projector parameters. Input messages retain the entire joint JSON skeleton. Gold answer symbols are supplied separately as `decision_targets`, one single-token symbol per marker, in field order. The logit immediately before a marker predicts its answer. The existing `TransformersModel` and `InputProcessor` extension points are reused. `twinkle_adapter` shifts labels once for the external cross-entropy and selects supervised positions without restricting the training vocabulary. No duplicate Qwen model implementation is introduced.
+
+Images, 35B models, sequence parallelism and cross-example packing are outside the validated recipe. Preserve question and option order; do not truncate evidence.
+
+## Environment
+
+Historical four-NPU validation used Ascend 910B, Python 3.12, CANN 9.1.0, Torch 2.10.0+cpu, torch_npu 2.10.0.post2, Transformers 5.15.1 and Accelerate 1.14.0. Twinkle additionally used PEFT 0.19.0. Use the repository installation instructions for the selected platform.
+
+The historical base image is `quay.nju.edu.cn/ascend/ms-swift@sha256:d1b56f2d77882edb92615c45641556c8d5adaf8ed360be73f4f0978f70fa01c1`. The optional `create_container.sh` reproduces its original fixed four-device layout (devices 0–3); it is not a portable scheduler or a complete dependency lock. Inspect device assignment before using that helper. No image export is required.
+
+The historical NPU image used NumPy 1.26.4 and an explicit `--no-deps` package installation. That is a reproduction-only dependency exception, outside the current upstream NumPy >=2.0 requirement. This PR leaves global dependency metadata unchanged. A dependency-compliant NPU environment still needs device validation; do not interpret the historical image as an upstream-supported dependency combination.
+
+## Prepare inputs
+
+From this example directory, supply already prepared case-level splits and the reference Intern-Decision schema:
+
+```bash
+python prepare_joint.py --prepared /path/to/case-splits \
+  --output /path/to/new-joint-data \
+  --official-schema /path/to/Intern-Decision/src/inputs/schema.py
+```
+
+The preparation tool checks compiled messages and targets against the supplied reference, records its SHA256, rejects duplicate/cross-split inputs and refuses an existing output directory. The historical schema hash was `d2fce8ffb8af19dca08a04eb01f4daae70ed43311d73c052ddcab797346d79bd`.
+
+The historical split contained 960 train, 120 validation, 120 calibration and 400 test cases, with five fields per case. Test and calibration records never enter training or checkpoint selection. The unpublished original training corpus is not reconstructed.
+
+## Launch
+
+The following variables override machine-specific paths and device assignment:
+
+```bash
+export PYTHON_BIN=/path/to/training-venv/bin/python
+export MODEL_PATH=/path/to/Qwen3.5-4B
+export DATA_ROOT=/path/to/new-joint-data
+export OUTPUT_ROOT=/path/to/new-training-run
+export ASCEND_RT_VISIBLE_DEVICES='<FOUR_ASSIGNED_NPU_IDS>'
+# Run sequentially, from this directory:
+bash run_joint.sh smoke
+bash run_joint.sh resume
+bash run_joint.sh train
+```
+
+The fixed four-rank recipe uses global batch 16, AdamW, learning rate 2e-6, cosine minimum 2e-7, warmup 0.03, weight decay 0.01 and seed 42. With the historical 960-case data, two epochs give 120 optimizer updates. Do not reuse that step count as an epoch claim for a different dataset.
+
+Smoke runs two steps and saves full state; resume verifies a third step in a fresh directory. Formal training saves a final model-only checkpoint. Model-only export does not support full optimizer/scheduler/RNG recovery. CPU checkpoint fences and Twinkle DCP hooks are scoped to this opt-in recipe; they are not general fixes for every backend.
+
+## Independent evaluation
+
+Use a separate MS-SWIFT HF/NPU inference environment to load either framework's exported weights:
+
+```bash
+ASCEND_RT_VISIBLE_DEVICES='<ONE_ASSIGNED_NPU_ID>' /path/to/eval-venv/bin/python evaluate_joint.py \
+  --checkpoint /path/to/final-hf-checkpoint \
+  --data /path/to/case-splits/test.jsonl --output /path/to/new-evaluation
+```
+
+Only validation loss may influence checkpoint selection. Official seven-suite evaluation is a separate protocol, not this held-out business subset. Historical full training and evaluation completed on the reproduction branch, but aggregate quality remained below the official target. Current PR-head device validation and upstream CI are pending. Keep measured metrics and raw predictions local.

cookbook/transformers/intern_decision_npu/README.md

@@ -0,0 +1,31 @@
+# Intern-Decision text training on Ascend
+
+This example trains the language backbone of Qwen3.5-4B with joint decision supervision in Twinkle, while freezing the vision tower and projector. Each assistant input contains a complete JSON skeleton with `<decision>` markers. Answer symbols appear only in labels. All supervised fields use full-vocabulary causal cross-entropy in the same forward pass.
+
+The existing `TransformersModel` and `InputProcessor` extension points are reused. `twinkle_adapter` shifts labels once for the external cross-entropy and selects supervised positions without restricting the training vocabulary. No duplicate Qwen model implementation is introduced.
+
+## Run the joint-field recipe
+
+Use [JOINT_REPRODUCTION.md](JOINT_REPRODUCTION.md) for environment requirements, input preparation, smoke training, checkpoint recovery and the full run. The example currently requires four Ascend NPUs, text-only inputs and no sequence parallelism or cross-example packing.
+
+```bash
+# From the repository root, after installing its dependencies:
+python -m pytest tests/processor/test_intern_decision.py -q
+```
+
+These synthetic CPU regressions require no checkpoint or benchmark download. They cover causal loss and gradient equivalence; real-tokenizer and distributed checkpoint audits remain separate checks.
+
+## Validation scope
+
+The pinned historical reproduction completed four-rank short training and full-state recovery, 120 optimizer steps, final model export/reload and independent evaluation. The final PR revision still needs device validation and upstream CI. Historical training evidence must not be attributed to a different source revision.
+
+The original Intern-Decision training corpus is not public. This recipe uses public Typed Decisions data and reproduces the supervision method, not the official training mixture or published model quality. It is not a StartLux or 35B training recipe. Keep weights, predictions and measured results outside the source tree.
+
+## Supporting tools
+
+- `prepare_joint.py`: compile all fields of a case together and compare them with a supplied official schema; reject split overlap and answer leakage.
+- `check_checkpoint.py`: check exported model keys, frozen vision weights, sampled language updates and checkpoint artifacts.
+- `evaluate_joint.py`: evaluate a saved model on held-out joint cases using the separately installed MS-SWIFT HF/NPU backend.
+- `run.sh`, `prepare_inputs.py` (where present), `evaluate_decision.py` and `evaluate_laya_suite.py`: earlier single-question experiment utilities. They are not the default joint recipe and their budgets and metrics must not be mixed with it.
+
+See [THIRD_PARTY.md](THIRD_PARTY.md) for schema attribution. Dataset access and licenses remain the caller's responsibility.

cookbook/transformers/intern_decision_npu/THIRD_PARTY.md

@@ -0,0 +1,6 @@
+# Source attribution
+
+`decision_schema.py` is derived from the Apache-2.0 licensed
+[InternLM/Intern-Decision](https://github.com/InternLM/Intern-Decision) input schema.
+Public training inputs come from `LocalLLaMA/typed-decisions`; data is downloaded separately.
+No dataset contents or experimental results are included in this example.

cookbook/transformers/intern_decision_npu/check_checkpoint.py

@@ -0,0 +1,88 @@
+"""Audit saved model changes and full training-state files after a short run."""
+import argparse
+import json
+from pathlib import Path
+
+import torch
+from safetensors import safe_open
+from transformers import AutoTokenizer
+
+
+def weight_map(root):
+    manifest = root / 'model.safetensors.index.json'
+    if manifest.exists():
+        return json.loads(manifest.read_text())['weight_map']
+    with safe_open(root / 'model.safetensors', framework='pt', device='cpu') as handle:
+        return {key: 'model.safetensors' for key in handle.keys()}
+
+
+def tensor(root, mapping, key):
+    with safe_open(root / mapping[key], framework='pt', device='cpu') as handle:
+        return handle.get_tensor(key)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('checkpoint', type=Path)
+    parser.add_argument('--base', type=Path, default=Path('/models/Qwen3.5-4B'))
+    parser.add_argument('--framework', choices=['swift', 'twinkle'], required=True)
+    parser.add_argument('--expected-step', type=int, required=True)
+    parser.add_argument('--output', type=Path, required=True)
+    args = parser.parse_args()
+    before, after = weight_map(args.base), weight_map(args.checkpoint)
+    # HF Qwen3_5ForConditionalGeneration does not load the base MTP predictor.
+    # The training checkpoint may explicitly serialize the tied output head.
+    ignored_mtp = {key for key in before if key.startswith('mtp.')}
+    expected = set(before) - ignored_mtp
+    extra = set(after) - expected
+    assert expected <= set(after), 'A runtime parameter is missing'
+    assert extra <= {'lm_head.weight'}, 'Unexpected exported parameter'
+    config = json.loads((args.checkpoint / 'config.json').read_text())
+    assert config['architectures'] == ['Qwen3_5ForConditionalGeneration']
+    if 'lm_head.weight' in extra:
+        assert config.get('tie_word_embeddings') is True
+        head = tensor(args.checkpoint, after, 'lm_head.weight')
+        embedding = tensor(args.checkpoint, after, 'model.language_model.embed_tokens.weight')
+        torch.testing.assert_close(head, embedding, rtol=0, atol=0)
+        del head, embedding
+    assert all((args.checkpoint / name).is_file() for name in set(after.values()))
+    vision = [key for key in before if '.visual.' in key]
+    language = [key for key in before if '.language_model.layers.0.' in key and key.endswith('weight')][:5]
+    assert vision and language
+    report = {
+        'checkpoint': str(args.checkpoint), 'frozen_tensors_verified': len(vision),
+        'unloaded_base_mtp_tensors': sorted(ignored_mtp), 'language_sample': {}}
+    for key in vision:
+        old = tensor(args.base, before, key).float()
+        new = tensor(args.checkpoint, after, key).float()
+        torch.testing.assert_close(old, new, rtol=0, atol=0)
+    for key in language:
+        old = tensor(args.base, before, key).float()
+        new = tensor(args.checkpoint, after, key).float()
+        report['language_sample'][key] = {
+            'max_abs_delta': float((old - new).abs().max()),
+            'changed_elements': int((old != new).sum())}
+    assert any(row['changed_elements'] for row in report['language_sample'].values())
+    marker = AutoTokenizer.from_pretrained(args.checkpoint).encode('<decision>', add_special_tokens=False)
+    assert len(marker) == 1
+    report['marker_id'] = marker[0]
+    state = json.loads((args.checkpoint / 'trainer_state.json').read_text())
+    report['global_step'] = state['global_step' if args.framework == 'swift' else 'cur_step']
+    assert report['global_step'] == args.expected_step
+    assert any((args.checkpoint / name).exists() for name in ['optimizer.pt', 'optimizer.bin'])
+    optimizer_dir = args.checkpoint / 'optimizer.pt'
+    if optimizer_dir.is_dir():
+        assert (optimizer_dir / '.metadata').is_file()
+        assert len(list(optimizer_dir.glob('*.distcp'))) == 4
+    assert (args.checkpoint / 'scheduler.pt').exists()
+    pattern = 'rng_state*.pth' if args.framework == 'swift' else 'rng_state_rank*.pt'
+    assert len(list(args.checkpoint.glob(pattern))) == 4
+    report['status'] = 'passed'
+    report['scope'] = 'all frozen vision tensors, sampled language updates, saved state; resume validated separately'
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(report, indent=2))
+    print(json.dumps(report, indent=2))
+
+
+if __name__ == '__main__':
+    main()

cookbook/transformers/intern_decision_npu/create_container.sh

@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+# Create a fresh, isolated four-card workspace on an Ascend host.
+set -euo pipefail
+framework=${1:?swift or twinkle}
+container=${2:?new container name}
+runtime=$(realpath "${3:?new runtime directory}")
+source_dir=$(realpath "${4:?framework checkout directory}")
+model=$(realpath "${5:?base checkpoint directory}")
+data=$(realpath "${6:?data directory containing joint and prepared}")
+case "$framework" in swift|twinkle) ;; *) exit 2 ;; esac
+image=quay.nju.edu.cn/ascend/ms-swift@sha256:d1b56f2d77882edb92615c45641556c8d5adaf8ed360be73f4f0978f70fa01c1
+test -z "$(ls -A "$runtime")"
+docker run -d --name "$container" --privileged --network host --ipc host --shm-size 32g \
+  -v "$runtime:/workspace" -v "$source_dir:/workspace/framework:ro" \
+  -v "$model:/models/Qwen3.5-4B:ro" -v "$data:/data:ro" \
+  -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
+  -v /usr/local/dcmi:/usr/local/dcmi:ro \
+  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi:ro \
+  -v /etc/ascend_install.info:/etc/ascend_install.info:ro \
+  "$image" bash -c 'rm -f /dev/davinci4 /dev/davinci5 /dev/davinci6 /dev/davinci7; exec sleep infinity'
+docker exec "$container" python -m venv --system-site-packages /workspace/.venv
+if [[ "$framework" == twinkle ]]; then
+  docker exec "$container" /workspace/.venv/bin/python -m pip install --no-deps peft==0.19.0
+  # PYTHONPATH in run_joint.sh selects this exact checkout; distribution metadata
+  # is installed without copying or modifying the read-only checkout.
+  docker exec "$container" bash -c 'cp -a /workspace/framework /workspace/package-build; /workspace/.venv/bin/python -m pip install --no-deps /workspace/package-build'
+fi