allreduce

This repository outlines a comprehensive guide for training a distributed deep learning model.

Topology-aware analytical model for Ring and hierarchical AllReduce traffic.

Pool Apple Silicon Macs into a distributed ML training cluster. Zero-config discovery, ring AllReduce gradient sync, HMAC-authenticated fleet isolation with TLS encryption. PyTorch + MLX engines.

AllReduce/AllGather scaling in ASTRA-sim across torus vs switch topologies on the analytical + ns-3 backends — latency-bound vs bandwidth-bound over message size and node count. Reproducible Docker/Chakra harness + write-up.

Analytical simulator for collective communication on ring, fat tree, and dragonfly topologies: graph based hop costs, bandwidth efficiency, and closed form algorithm crossover up to 1024 nodes. Reproducible sweeps in Python (networkx), with a compiled LaTeX report.

RoCEv2 control plane simulator — DCQCN congestion control on fat-tree GPU clusters with NCCL Ring AllReduce workloads

Adaptive Hybrid Distributed SGD implements and evaluates Parameter Server, Ring AllReduce, and a runtime-adaptive hybrid that switches communication topology during model training based on live metrics.

Event-driven benchmark harness for ring and tree allreduce under topology, message size, and congestion constraints.

Benchmark suite for MPI collectives (Bcast, Alltoall, Allreduce) with from-scratch ring and recursive-doubling allreduce implementations. Fitted Hockney cost models, measured algorithm crossovers, and LaTeX reports generated entirely from the data. C++17, Open MPI, Python analysis pipeline.

Distributed training of a VGG-11 network on the CIFAR-10 dataset

CPU, dependency-free lab for collective-communication algorithms (ring/tree/naive all-reduce): simulate the data movement, verify the volume formulas, model latency vs bandwidth, and parse real nccl-tests output.