Skip to content

Wave 9 Multi-Mac Deployment — 48 Workers across 2-3 Macs #409

Closed

Description

@gHashTag

Wave 9 Multi-Mac Deployment

Overview

Deploy Wave 9 training with S3 MultiObj configuration across multiple Mac machines. Each Mac runs a subset of 48 total workers.

Configuration File: .trinity/wave9_multi_mac.yaml

devices:
  - id: 1
    hostname: mac-1.local
    workers_start: 1
    workers_count: 16
  - id: 2
    hostname: mac-2.local
    workers_start: 17
    workers_count: 16
  - id: 3
    hostname: mac-3.local
    workers_start: 33
    workers_count: 16

CLI Commands

# Initialize all devices (generates compose files)
tri farm local-wave9 multi-mac init

# Start all workers
tri farm local-wave9 multi-mac start

# Check status
tri farm local-wave9 multi-mac status

# Verify seeds and S3 MultiObj config
tri farm local-wave9 multi-mac verify

# Stop all workers
tri farm local-wave9 multi-mac stop

Per-Mac Device Init (Alternative)

# Mac 1 (workers 1-16)
tri farm local-wave9 device-init --device-id 1 --workers-start 1 --workers-count 16
docker-compose -f deploy/docker/docker-compose.wave9-mac-1.yml up -d

# Mac 2 (workers 17-32)
tri farm local-wave9 device-init --device-id 2 --workers-start 17 --workers-count 16
docker-compose -f deploy/docker/docker-compose.wave9-mac-2.yml up -d

# Mac 3 (workers 33-48)
tri farm local-wave9 device-init --device-id 3 --workers-start 33 --workers-count 16
docker-compose -f deploy/docker/docker-compose.wave9-mac-3.yml up -d

Verification

# Verify unique seeds (CRITICAL!)
docker ps --format "{{.Names}}" | grep wave9 | while read c; do
  docker exec $c printenv HSLM_SEED
done | sort | uniq -d  # Should be empty

# Verify S3 MultiObj profile
docker exec wave9-worker-1 printenv HSLM_PROFILE  # s3multiobj
docker exec wave9-worker-1 printenv HSLM_NTP_WEIGHT  # 0.5
docker exec wave9-worker-1 printenv HSLM_JEPA_ENABLED  # true

Documentation

Full guide: docs/wave9/multi_mac_deployment.md

Closes #418

Activity

  1. gHashTag commented on Mar 25, 2026

    @gHashTag
    OwnerAuthor

    ✅ Wave 9 Multi-Mac Deployment — COMPLETE

    All CLI commands implemented and tested:

    • — Initialize all devices from config
    • — Start workers across all Macs
    • — Stop all workers
    • — Show aggregate status
    • — Check seeds and S3 MultiObj profile

    Infrastructure files:

    • src/tri/mac_cluster.zig — Mac cluster discovery
    • src/tri/mac_installer.zig — Mac installer
    • src/tri/wave9_device.zig — Device-specific compose
    • src/tri/wave9_generator.zig — Compose generation
    • docs/wave9/multi_mac_deployment.md — Full documentation

    Build: L0 ✅ L1 ✅ tri ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions