Repository navigation
Phase 3: Modal Labs GPU + PBT hyperparameter search (+20-30% speedup) #295
Description
Goal
Миграция тренинга на GPU для 3-5x ускорения + PBT для быстрого поиска гиперпараметров.
Tasks
1. Modal Labs Integration
- Создать
modal_train.pyобёртку для HSLM тренинга - Docker-native, serverless GPU (A10G/A100)
- Pay-per-use: платишь только во время тренинга
- Ожидание: 3-5x raw speedup
2. Гибридная архитектура
- Railway остаётся для оркестрации (Cloud Dev агенты)
- Modal Labs для GPU compute
tri train --gpuфлаг для диспатча на Modal
3. Population-Based Training (PBT)
- 4-5 параллельных runs с разными гиперпараметрами
- Каждые N шагов: evaluate, мутация bottom-2 с параметрами top-2
- Поиск: LR, batch_size, tau_schedule, label_smoothing_eps
- Ожидание: 30-50% быстрее к оптимуму
4. Альтернативы для оценки
- Vast.ai: самый дешёвый для длинных runs (спот-рынок)
- Runpod: Docker-native, низкий overhead
Depends on
- Phase 1: LAMB optimizer + batch128 + label smoothing + AVX2 SIMD (20-30% speedup) #293 (Phase 1), Phase 2: Temperature STE + EMA shadow weights + bfloat16 (+15-25% speedup) #294 (Phase 2)
Priority
P1 — дни 5-7
Cost
- Modal A10G: ~$0.60/hr, A100: ~$2.50/hr
- Railway CPU 32vCPU: ~$0.89/hr
- Break-even при 3x speedup
Files to modify
src/tri-api/, bin/tri-train, .modal/
Verification
- Modal job запускается и выдаёт логи
- GPU training 3x+ быстрее CPU baseline
- PBT находит лучшие гиперпараметры за 4-5 runs
Activity
github-actions commentedon Mar 12, 2026
🚀 Agent container spawned! Deploying on Railway. Tracking in this issue.
github-actions commentedon Mar 12, 2026
🚀 Agent container spawned! Deploying on Railway. Tracking in this issue.
github-actions commentedon Mar 12, 2026
🚀 Agent container spawned! Deploying on Railway. Tracking in this issue.
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent | 2026-03-12T06:01:09Z
📋 Step: 1/8 — Authenticating with GitHub
🔄 Status: AWAKENING
⏱️ Elapsed: 0s
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #295
gHashTag commentedon Mar 12, 2026
☠️ Trinity Agent | 2026-03-12T06:01:12Z
📋 Step: 2/8 — Container terminated by signal
🔄 Status: KILLED
⏱️ Elapsed: 3s
gHashTag commentedon Mar 12, 2026
☠️ Trinity Agent Dashboard — Issue #295
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent | 2026-03-12T06:01:18Z
📋 Step: 1/8 — Authenticating with GitHub
🔄 Status: AWAKENING
⏱️ Elapsed: 0s
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #295
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #295
gHashTag commentedon Mar 12, 2026
📖 Trinity Agent | 2026-03-12T06:01:26Z
📋 Step: 3/8 — Reading issue #295
🔄 Status: READING
⏱️ Elapsed: 8s
gHashTag commentedon Mar 12, 2026
⚡ Trinity Agent Dashboard — Issue #295
gHashTag commentedon Mar 12, 2026
⚡ Trinity Agent | 2026-03-12T06:01:31Z
📋 Step: 4/8 — Claude Code running (timeout: 3600s)
🔄 Status: CODING
⏱️ Elapsed: 13s
17 remaining items
gHashTag commentedon Mar 12, 2026
❌ Trinity Agent | 2026-03-12T19:29:07Z
📋 Step: 8/8 — Compilation gate failed after 3 repair attempts — skipping PR
🔄 Status: FAILED
⏱️ Elapsed: 698s
gHashTag commentedon Mar 12, 2026
❌ Trinity Agent: Compilation gate FAILED after 3 repair attempts. Code does not build.
PR creation skipped. Issue needs manual attention.
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent | 2026-03-12T19:29:22Z
📋 Step: 1/8 — Authenticating with GitHub
🔄 Status: AWAKENING
⏱️ Elapsed: 0s
gHashTag commentedon Mar 12, 2026
🌅 Trinity Agent Dashboard — Issue #295
gHashTag commentedon Mar 12, 2026
⚡ Trinity Agent Dashboard — Issue #295
gHashTag commentedon Mar 12, 2026
📖 Trinity Agent | 2026-03-12T19:29:29Z
📋 Step: 3/8 — Reading issue #295
🔄 Status: READING
⏱️ Elapsed: 7s
gHashTag commentedon Mar 12, 2026
📖 Trinity Agent Dashboard — Issue #295
gHashTag commentedon Mar 12, 2026
❌ Trinity Agent | 2026-03-12T19:30:03Z
📋 Step: 4/8 — z.ai attempt 1/3, backoff 30s
🔄 Status: FAILED
⏱️ Elapsed: 41s
gHashTag commentedon Mar 12, 2026
⚡ Trinity Agent | 2026-03-12T19:30:38Z
📋 Step: 5/8 — Claude Code running (timeout: 3600s)
🔄 Status: CODING
⏱️ Elapsed: 76s
gHashTag commentedon Mar 12, 2026
📋 Step: 1/7 — Modal Labs GPU wrapper creation
🔄 Status: THINKING
Thought: Modal Labs provides serverless GPU (A10G/A100) with Python SDK. Need to create wrapper that runs Zig HSLM training on GPU-enabled containers.
Action: Creating .modal/hslm_modal.py for GPU training dispatch
Next: Implement PBT in Zig for hyperparameter search
gHashTag commentedon Mar 12, 2026
📋 Step: 2/7 — Modal Labs Integration complete
Thought: Modal wrapper complete, will now dispatch to via tri CLI
Action: Creating . .sm/hslm_modal.py, with updated HSLM training doc
Next: Test PBT implementation in zig
gHashTag commentedon Apr 30, 2026
Requires GPU training infrastructure. Not actionable without hardware access.
feat(hslm): Phase 3 — PBT hyperparameter search (#582)