Repository navigation
Phase 2: Temperature STE + EMA shadow weights + bfloat16 (+15-25% speedup) #294
Description
Task Description
Goal
Phase 2 training optimizations after Phase 1 (#293). Cumulative +15-25% speedup.
Tasks
1. Temperature-Annealed STE Quantization
- Start with tau=5.0, anneal to tau=0.1 over training
- Soft ternary during forward, hard ternary during inference
- Expected: 10-15% convergence improvement
2. EMA Shadow Weights
- Maintain exponential moving average of weights (decay=0.999)
- Use EMA weights for evaluation/inference
- Expected: 5-8% PPL improvement
3. bfloat16 Shadow Weights
- Keep shadow copies in bfloat16 for gradient accumulation
- Quantize to ternary after update step
- Reduces memory, enables larger models
4. RMSNorm Before Linear Layers
- From BitNet b1.58: critical for ternary network stability
- Add RMSNorm before every linear projection
Depends on
Priority
P1 - days 3-5
Files to modify
- src/tri-api/ (STE, EMA, optimizer)
- src/vsa.zig (quantization, RMSNorm)
Verification steps
zig build testpasses- Temperature annealing shows smooth loss curve
- EMA weights give lower eval PPL than raw weights
- RMSNorm stabilizes training at larger learning rates
Activity
github-actions commentedon Mar 12, 2026
🚀 Agent container spawned! Deploying on Railway. Tracking in this issue.
github-actions commentedon Mar 12, 2026
🚀 Agent container spawned! Deploying on Railway. Tracking in this issue.
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent | 2026-03-12T05:56:45Z
📋 Step: 1/8 — Authenticating with GitHub
🔄 Status: AWAKENING
⏱️ Elapsed: 0s
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #294
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent | 2026-03-12T05:56:46Z
📋 Step: 1/8 — Authenticating with GitHub
🔄 Status: AWAKENING
⏱️ Elapsed: 0s
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent | 2026-03-12T05:56:47Z
📋 Step: 1/8 — Authenticating with GitHub
🔄 Status: AWAKENING
⏱️ Elapsed: 0s
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #294
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #294
gHashTag commentedon Mar 12, 2026
☠️ Trinity Agent | 2026-03-12T05:56:48Z
📋 Step: 2/8 — Container terminated by signal
🔄 Status: KILLED
⏱️ Elapsed: 1s
gHashTag commentedon Mar 12, 2026
☠️ Trinity Agent | 2026-03-12T05:56:48Z
📋 Step: 2/8 — Container terminated by signal
🔄 Status: KILLED
⏱️ Elapsed: 1s
gHashTag commentedon Mar 12, 2026
☠️ Trinity Agent Dashboard — Issue #294
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #294
gHashTag commentedon Mar 12, 2026
☠️ Trinity Agent Dashboard — Issue #294
gHashTag commentedon Mar 12, 2026
☠️ Trinity Agent | 2026-03-12T05:56:52Z
📋 Step: 3/8 — Container terminated by signal
🔄 Status: KILLED
⏱️ Elapsed: 6s
86 remaining items
gHashTag commentedon Mar 12, 2026
🧪 Trinity Agent | 2026-03-12T22:12:50Z
📋 Step: 5/8 — and finally commit the message referencing the issue. number.I need to verify everything works corre
🔄 Status: TESTING
⏱️ Elapsed: 738s
gHashTag commentedon Mar 12, 2026
💥 Trinity Agent | 2026-03-12T22:12:52Z
📋 Step: 6/8 — if tests fail, I can diagnose and root cause by reading test output or looking at error messages.
🔄 Status: ERROR
⏱️ Elapsed: 740s
gHashTag commentedon Mar 12, 2026
🧪 Trinity Agent | 2026-03-12T22:12:55Z
📋 Step: 7/8 — verification passes. and
zig build testpasses, andzig fmt src/ && zig build, to see the change🔄 Status: TESTING
⏱️ Elapsed: 743s
gHashTag commentedon Mar 12, 2026
💥 Trinity Agent | 2026-03-12T22:13:08Z
📋 Step: 15/8 — and that I didn't see immediate errors or issues during this coding review that."
🔄 Status: ERROR
⏱️ Elapsed: 756s
gHashTag commentedon Mar 12, 2026
🔍 Trinity Agent | 2026-03-12T22:13:15Z
📋 Step: 5/8 — Self-review (advisory)
🔄 Status: REVIEWING
⏱️ Elapsed: 763s
gHashTag commentedon Mar 12, 2026
🔧 Trinity Agent | 2026-03-12T22:13:19Z
📋 Step: 6/8 — Compilation failed, auto-repair attempt 1/3
🔄 Status: REPAIRING
⏱️ Elapsed: 767s
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent | 2026-03-12T22:13:29Z
📋 Step: 1/8 — Authenticating with GitHub
🔄 Status: AWAKENING
⏱️ Elapsed: 0s
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #294
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #294
gHashTag commentedon Mar 12, 2026
📖 Trinity Agent | 2026-03-12T22:13:38Z
📋 Step: 3/8 — Reading issue #294
🔄 Status: READING
⏱️ Elapsed: 9s
gHashTag commentedon Mar 12, 2026
📖 Trinity Agent Dashboard — Issue #294
gHashTag commentedon Mar 12, 2026
⚡ Trinity Agent | 2026-03-12T22:13:43Z
📋 Step: 4/8 — Claude Code running (timeout: 3600s)
🔄 Status: CODING
⏱️ Elapsed: 14s
gHashTag commentedon Mar 12, 2026
🤖 Agent: trinity-agent | 2026-03-12
📋 Step: 1/5 — Exploring codebase and starting Phase 2 implementation
🔄 Status: ACTING
Thought: Need to implement 4 Phase 2 optimizations: Temperature-Annealed STE, EMA Shadow Weights, bfloat16 Shadow Weights, RMSNorm before linear layers.
Action: Analyzed ste.zig, trainer.zig, model.zig, trinity_block.zig, constants.zig
Result: Found existing STE implementation with vanilla/TWN/progressive modes. Training loop uses AdamW/LAMB. RMSNorm already exists in model.zig.
Next: Implement temperature-annealed STE quantization with soft ternary forward, hard ternary inference.