miniLLM

A ~101M parameter dense decoder-only transformer, trained from scratch on FineWeb-Edu on a single RTX 4060 Laptop (8GB) — plus chessLLM, the same architecture retrained on chess notation, which builds an internal board representation from move text alone.

Two write-ups:

• This file — architecture, optimizer choices, training pipeline, measured throughput.
• CHESS.md — the chess model: representation learning, the linear board probe, and where the world model breaks. That is the more interesting one.

Weights for both are on the Hub:

Model	Weights	Val loss	What it is
miniLLM	amanm10000/minillm	3.2840 (ppl 26.7)	100.7M non-embedding, FineWeb-Edu, base model
chessLLM	amanm10000/chessllm	0.4150	94.4M non-embedding, Lichess SAN, 99.22% legal

Try the chess model (no training required)

git clone --recurse-submodules https://github.com/amanmprojects/llm.git
cd llm

python -m venv .venv && source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cu130   # see requirements.txt
pip install -r requirements.txt

python scripts/download_weights.py chess   # ~378 MB -> data/chess_out/ckpt.pt

Then either evaluate it:

python chess/eval_chess.py --ckpt data/chess_out/ckpt.pt --mode legal --n-games 20

or play it in the browser (two processes, UI on :8000, model sidecar on :5555):

node chess-ui/server.mjs
python chess-ui/llm_server.py

Pick 🧠 chessLLM (neural) from the Level dropdown. The sidecar binds to 127.0.0.1 and has no authentication — do not expose it to a network.

If you cloned without --recurse-submodules, run git submodule update --init to fetch the UI.

Try the text model

python scripts/download_weights.py text    # ~252 MB + tokenizer
python sample.py --ckpt out/ckpt.pt --prompt 'The capital of France is'

This also fetches data/tokenizer_v2.json, which sample.py needs to decode and which is not in git. It is a base model with no instruction tuning — it continues text, it does not answer questions, and factual details in its output are frequently wrong. Sampling settings matter a lot here: see Decoding matters more than you'd think.

Quick start (training)

source .venv/bin/activate

# 1. data (~10 min, already done)
python prepare_data.py --shards 3 --target-tokens 2_000_000_000

# 2. train overnight (set --hours to your budget)
python train.py --preset base --hours 10 --run-name base_muon

# 3. generate
python sample.py --prompt "The causes of the French Revolution were"

Resume an interrupted run with --resume auto (checkpoints every 20 min, atomic writes). The LR schedule is stored in the checkpoint, so a resume continues the curve the run was actually on — you do not have to re-pass --lr-peak/--warmup-steps, and passing different ones is ignored rather than silently applied mid-run.

Suspend kills a run. A laptop that sleeps takes the CUDA context with it and the process dies with no traceback — this ended run 2 at step 1750 of 7535. Wrap long runs:

systemd-inhibit --what=sleep --why="training" python train.py --preset base --hours 10 ...

No sudo needed. Confirm it took with systemd-inhibit --list.

Checkpoints go straight to out/ckpt.pt — written as ckpt.tmp then os.replaced, so a reader never sees a half-written file and you can sample from a run in progress. Do it on the CPU:

python sample.py --device cpu --prompt "..."     # while training; the GPU has no room to spare

out/ckpt.pt is overwritten in place, so copy it aside before starting a second phase on top of a finished run — the bf16 snapshots in out/snapshots/ carry no optimizer state and cannot resume training.

Phase 2: continued pretraining + chat

# more prose (shards 4-6 -> a second bin; never touches the first)
python prepare_shards.py --shard-start 4 --shard-end 7 --out data/train_b.bin

# reserve Llama-3 control tokens inside the existing 32768 vocab, then build chat bins
python reserve_control_tokens.py
python prepare_qa.py

python train.py --preset base --hours 10 --resume auto --fresh-schedule \
  --train-bins train.bin,train_b.bin \
  --qa-bin train_qa.bin --qa-val-bin val_qa.bin --qa-weight 0.02 \
  --lr-peak 0.5 --run-name base_phase2

python sample.py --chat        # multi-turn, Llama-3 template

--fresh-schedule is required to continue a finished run: phase 1 decayed to LR ~0, so a plain --resume auto would train at zero LR for the whole night. It keeps the weights and optimizer state, restarts step accounting, and re-warms over --warmup-steps to --lr-peak x base LR. Training refuses to start without it rather than silently wasting the budget.

Pass it only when starting a new phase. To resume a phase-2 run that crashed mid-phase, use a plain --resume auto: the checkpoint records that it is on the phase-2 curve, and --fresh-schedule would restart the step counter at 0 and re-warm a second time.

Architecture

Llama-3-shaped dense transformer. 12 layers, d=768, 12 query heads / 4 KV heads, SwiGLU FFN at 2048 hidden, 1024 context, 32768 vocab, tied embeddings.

Component	Choice	Why
Norm	RMSNorm, pre-norm	Cheaper and more stable than LayerNorm
Positions	RoPE (theta=10k)	Relative, zero learned params
FFN	SwiGLU	Same FLOPs as GELU at 8/3·d, reliably lower loss
Attention	GQA 12q/4kv + SDPA	3× fewer KV projections, quality-neutral here
QK-norm	RMSNorm on q,k	Bounds attention logits → tolerates a hot LR
Biases	none	Free params, no quality cost
Embeddings	tied	Saves 25M params = 25% of the model
Optimizer	Muon + AdamW	Measured 2.7× lower perplexity than AdamW (see below)

Optimizer A/B (measured, not assumed)

Identical 200-step / 13.1M-token budget, same WSD schedule, base model:

Config	val loss	val ppl	vs AdamW
AdamW lr=1.5e-3	6.0712	433.2	—
Muon lr=0.02 (chosen)	5.0917	162.7	−0.98 nats, 2.7× lower ppl
Muon lr=0.035	5.1077	165.3	−0.96 nats

Muon costs ~2% throughput (25.9k vs 26.5k tok/s) for that gain. Reproduce with ./ab_test.sh.

The two models, side by side

Same code, same optimizer, same schedule — two presets. The chess model spends its budget on depth rather than vocabulary:

	miniLLM (base)	chessLLM (chess)
Params	101M (76M non-embedding)	94.4M (94.4M non-embedding)
Layers	12	15
d_model	768	768
Heads	12 query / 4 KV (GQA)	12 query / 4 KV (GQA)
Head dim	64	64
FFN hidden	2048 (SwiGLU)	2048 (SwiGLU)
Context	1024	1024
Vocab	32768 BPE	40 characters
Embeddings	tied	tied
Data	FineWeb-Edu	Lichess SAN
Final val	see run log	0.4152

The vocab row drives the rest. A 32768-token embedding table costs ~25M parameters; a 40-token one costs 30k. That frees the budget for three extra layers, and it means essentially every parameter in chessLLM is doing chess rather than storing token identities.

Run 1 (2026-08-05)

12.44h on one RTX 4060 Laptop 8GB, base preset, Muon lr=0.02:

Steps × tokens/step	9,144 × 131,072 = 1.20B tokens
Tokens per param	11.9 (below Chinchilla's 20 — a wall-clock budget, not a compute-optimal one)
Throughput	26.8k tok/s sustained, 5.6 GB of 7.6 GB VRAM
Final val loss	3.2840 (ppl 26.7)

Val loss by step: 3.83 @1.7k → 3.69 @3.4k → 3.63 @5.4k → 3.60 @7.4k → 3.28 @9.1k. The last leg is the WSD decay phase, which is where most of the final drop comes from — stopping before the decay completes would have left ~0.3 nats on the table.

Nine bf16 inference-only snapshots (240 MB each) are in out/snapshots/; walk them with compare_snapshots.py --gpu to hear the model learn.

Decoding matters more than you'd think

The model ranks true facts above plausible false ones 95% of the time (eval_facts.py, 40 common-knowledge cloze pairs, chance = 50%). But at the original temperature 0.8, free generation only produced the correct fact 30% of the time — a 65-point gap between what the model knows and what it says. Sampling noise, not missing knowledge.

Measured over 10 factual + 12 open-ended prompts, 3 seeds, 120-token generations:

temperature	rep. penalty	fact%	worst repeated-4gram%
0.8	1.0	30–50%	~74%	old defaults
0.3	1.25	67%	5%	current defaults
0.0	1.35	80%	2%	max factuality
0.0	1.50	80%	0%	over-penalised: starts padding with enumerations

So: lower the temperature when you want facts, raise it when you want variety. The repetition penalty (CTRL-style, windowed to the last 256 tokens) is what kills the "the National Park of Pakistan became the National Park of Pakistan" failure mode; below ~1.25 this model loops badly, above ~1.4 it starts avoiding words it legitimately needs.

Factual accuracy also improved monotonically through the whole run — 77.5% at step 1,713 to 95.0% at step 9,144, still climbing at the end, with no plateau. Val loss alone hides this: the constant-LR phase looks flat (−0.020 nats/1k steps) while knowledge accumulates.

Run 2 — continued pretraining + chat (2026-08-06)

Phase 1 used 1.20B of the 1.995B tokens it had, and had not plateaued on factual accuracy. Run 2 continues from its checkpoint with three changes:

Prose pool	1.995B + 2.269B (shards 4-6, train_b.bin) = 4.26B tokens
Budget	7,535 steps x 131,072 = 0.99B tokens in 10.0h @ 27.4k tok/s
Cumulative	2.19B tokens = 21.8 tokens/param (past Chinchilla's 20)
LR	re-warmup 200 steps to 0.5x base, then hold, then decay
Chat	2% of rows from a 10.13M-token Llama-3-formatted QA bin (~1.9 epochs)

The second-phase LR peaks at half the original. Full LR on an already-converged model briefly undoes what the first decay settled; half is the usual compromise between that and re-warming enough to actually move.

The QA bin is 75% UltraChat / 25% Dolly by tokens (not by example count — UltraChat conversations are ~4x longer, so an example split would have handed it ~4x the share asked for). They fail in opposite directions: Dolly alone never shows a follow-up turn, UltraChat alone teaches a 100M model to produce long confident prose with nothing grounding it.

Fitting a chat template into a frozen vocabulary

The v1 tokenizer had one special token, so spelling a chat template in ordinary text cost 10 tokens for <|start_header_id|> and 8 for <|eot_id|>. Growing the vocab would mean resizing a tied, already-trained 32768x768 embedding.

Instead, reserve_control_tokens.py reclaims dead slots. A byte-level BPE seeds its vocab with all 256 byte characters, and 49 never occurred in 1.995B tokens — but "never occurred" is not "cannot occur". Only bytes that are structurally impossible in valid UTF-8 are safe:

0xC0, 0xC1   overlong 2-byte encodings, forbidden by the spec
0xF5 - 0xFF  would encode a codepoint beyond U+10FFFF

That yields 13 provably-unreachable slots; 6 are used and 7 reserved. Same vocab size, same parameter count, same checkpoint — and every existing .bin stays valid, because ids are preserved and ordinary text verifiably tokenizes bit-identically to v1.

The trap: the ids are unreachable from real text, but the names are not. Registering them in added_tokens makes the tokenizer match the literal string <unk> or <|eot_id|> anywhere it appears — and shard 6 of FineWeb-Edu contains a machine-translation tutorial with <unk> in an example sentence. Left alone, that document would forge a turn boundary mid-prose. chat_format.scrub_control_tokens breaks these spellings before encoding (1 document in 2.27B tokens), and both data scripts hard-fail if a control id survives into a bin. The security-flavoured framing is deliberate: a corpus that can forge turn boundaries is a corpus that can teach the model to ignore them.

Why not DeepSeek-style (MLA / MoE / sparse attention)

Every V3/V4 headline feature solves a problem that only exists at scale:

• MLA compresses the KV cache. At 100M params / 1024 ctx the KV cache is ~6 MB. It would add complexity and cost quality to solve a non-problem.
• MoE buys more params per FLOP. This setup is VRAM-bound, not FLOP-bound — MoE spends the scarce resource to save the abundant one. Backwards here.
• Sparse attention (DSA) fixes O(L²) at long context. At 1024 ctx attention is ~8% of FLOPs; optimizing it is noise.
• MTP is plausible but unproven sub-1B, and adds real bug surface.

At this scale the levers that actually move loss, in order: token count → data quality → LR schedule → optimizer → architecture. Architecture is last, and dense-Llama is already near the practical ceiling. Muon is the one genuinely high-leverage non-boring choice, and it's applied where it's proven.

DeepSeek's quality is also overwhelmingly data + post-training/RL, not the attention variant — so copying its architecture wouldn't transfer the thing that makes it good anyway.

Training setup

• bf16 autocast (auto fp16+GradScaler fallback on Turing/T4, which has no bf16)
• Auto micro-batch sizing — probes the largest batch that fits; no hand-tuning, no OOM at step 3000
• Wall-clock budgeting — --hours N measures real throughput, then sizes the LR schedule to land exactly at the end of the budget
• WSD LR schedule (2% warmup → flat → linear decay over last 20%) — degrades gracefully if stopped early, unlike cosine
• Gradient accumulation to a 131k-token optimizer step
• torch.compile + fused AdamW + expandable_segments allocator

Measured on an RTX 4060 Laptop (8GB)

Metric	Value
bf16 matmul peak (measured)	25.9 TFLOPS
Training throughput	26.5–27.7k tok/s (~15.9 TFLOPS effective, ~61% of matmul peak)
Max micro-batch @ 1024 ctx	8 (12 OOMs); 5.6 GB of 7.6 GB usable
Tokenizer compression	5.00 chars/token (GPT-2 gets ~4.3 here)
12h run	9,144 steps = 1.20B tokens = 11.9 tokens/param

Note the auto-batcher probes in eager mode (7.04 GiB peak) while the compiled model only uses 5.6 GB, so it deliberately picks a conservative size — good for an unattended run.

Token budget honesty

At this throughput a 12h run gives ~11.9 tokens/param, below Chinchilla's 20. The compute-optimal model for this exact budget would be ~70M params at 1.4B tokens. The loss difference is small (the curve is flat near the optimum) and 100M buys more knowledge capacity for generation, so 100M is the right call here — but longer is better: HOURS=16 ./run_overnight.sh.

Files

File	Purpose
model.py	Architecture + presets (small 22M / base 101M / large 191M)
muon.py	Muon optimizer + param-group splitting
prepare_data.py	Download FineWeb-Edu → train BPE → flat uint16 bins
prepare_shards.py	Tokenize an explicit shard range into a new bin; never overwrites
reserve_control_tokens.py	Reclaim dead vocab slots for Llama-3 control tokens
chat_format.py	Chat template, control-token ids, tokenizer resolution, scrubbing
prepare_qa.py	Blend + block-pack UltraChat/Dolly into chat bins
train.py	Training loop (DDP-ready, resumable, multi-bin mixing)
sample.py	Generation CLI (--chat for multi-turn)
eval_facts.py	Cloze-ranking factuality probe
chess/chess_format.py	Chess SAN encoder / decoder (40-id vocab, frozen id map)
chess/prepare_chess.py	Streaming Lichess PGN → filter → char-encode → uint8 bin
chess/split_chess_bin.py	Carve a val bin off a train bin, on game boundaries
chess/make_dev_corpus.py	Synthesize small Lichess-style PGN zst for pipeline checks
chess/eval_chess.py	Legal-move, Stockfish-Elo, Puzzles, and board-probe evals
chess/train_chess.sh	One-shot: prep + smoke + flags for the overnight run

Chess — miniLLM training a chessboard

What it is. A sub-100M chess model (94.4M) trained from scratch on the same laptop, using a character-level SAN encoding of rated Lichess games. It isolates a clean, checkable question: does a transformer actually build an internal board representation, or just memorize surface statistics? The answer here is the former — a linear probe reads the board out of the residual stream at 85.6% per-square — while play strength lands at ~1131 Elo, well short of the 1300–1600 target.

CHESS.md is the full write-up, covering the representation-learning evidence, the loss curve, and the failure modes. This section covers the pipeline.

Why the tiny vocab. General text spends ~100M+ parameters on the embedding+head matrix. Chess needs almost no vocabulary — hashes and piece letters only — so the same GPU budget goes into a 15-layer stack on a 40-token vocab. Representation cost per move is tiny and the model is parameter-dense where it matters: the layers.

Representation (chess/chess_format.py)

Games are the move sequence preceded by a rating bucket:

;<15> e4 e5 Nf3 Nc6 Bb4 a6 ...\n

A frozen 40-id map: ;=0 game-start, \n=1 game-end, =2, then a-h, 1-8, KQRBN, Ox#=-, and ten <15>..<24> rating buckets (ids 30-39). Characters, not BPE — so any legal move is in-vocabulary by construction, and legality failures are the model's fault, not the tokenizer's.

import sys; sys.path.insert(0, "chess")
import chess_format as cf
ids = cf.encode_game(["e4", "e5", "Nf3"], rating=1800)  # [0, 30, 2, ...]

Namespace-package gotcha: chess/ has no __init__.py, so from chess import chess_format resolves to the installed python-chess package. Always sys.path.insert(0, "chess") then import chess_format. (prepare_chess.py / train.py / eval_chess.py already do this.)

Data (chess/prepare_chess.py)

Streams a Lichess standard-rated monthly PGN zst, filters hard, and writes a flat uint8 bin — no BPE, token ids are the character ids themselves.

Get the dump from the HuggingFace mirror, not database.lichess.org. The official host served this machine at 13–40 KB/s (a ~200-hour ETA for 31 GiB) while a control host on the same connection managed 10.5 MB/s. The mirror below carries the byte-identical original .pgn.zst at ~6–8 MB/s, about an hour:

wget -c -O data/lichess_db_standard_rated_2023-01.pgn.zst \
  https://huggingface.co/datasets/ezipe/lichess_2023_janoct/resolve/main/lichess_db_standard_rated_2023-01.pgn.zst

Avoid carbon225/lichess-elite — its moves are English prose ("White pawn to d4"), not SAN.

python chess/prepare_chess.py \
  --input data/lichess_db_standard_rated_2023-01.pgn.zst \
  --min-elo 1800 --min-time-control 180 \
  --target-tokens 2_100_000_000 --out data/chess_train.bin

Filters: both players ≥1800, Termination == "Normal", move count in [10, 250], time-control ≥180s. On 2023-01 that keeps 13.9% of games at ~293 ids each.

The subsample rate is measured, not guessed: a calibration probe reads the first 400k games and extrapolates by compressed-byte ratio, because a hardcoded estimate undershot this month's token target by 35%. --calibrate-games 0 disables it and keeps every surviving game.

Validate before the multi-hour pass — this replays real games through python-chess and demands 100%:

python chess/prepare_chess.py --input <dump>.pgn.zst --sample 3000 --dry-run
# [validate] 3,000/3,000 sampled games replay legal with python-chess (100.00%).

A 5.9MB synthetic corpus (data/dev.pgn.zst, from make_dev_corpus.py) exercises the pipeline without any download, but its ratings/terminations are random — most games fail the production filters, so it is for plumbing checks only, never for judging model quality.

The val split (chess/split_chess_bin.py)

prepare_chess.py writes one bin; training needs a held-out set. The split must fall on a game boundary — a mid-game cut leaves the val set prefixed by a fragment whose opening the train set already holds, which quietly flatters val loss.

python chess/split_chess_bin.py --bin data/chess_train.bin \
  --val-out data/chess_val.bin --val-tokens 10_000_000     # --dry-run to preview the cut

It takes a suffix, so the train bin is a plain in-place truncation (no multi-GB rewrite) and no game can land in both halves. Lichess dumps are chronological, so this is also a mild time-holdout: val games were played after train games.

Train (train.py --preset chess)

python train.py --preset chess --hours 10 \
  --train-bins chess_train.bin --val-bin chess_val.bin \
  --bin-dtype uint8 --vocab-map chess \
  --tokens-per-step 131072 --run-name chess_v1

The chess preset (15 layers / 768 width / GQA 12q·4kv / 1024 ctx) reuses the same Muon + WSD stack as the language model; --bin-dtype uint8 and --vocab-map chess tell the loader the bins are raw character ids. Sample after a smoke run:

;<20> b3 h6 Na3 d3 e5 Bd2 Nf6 Bf4 ...

The v1 run, from logs/chess_train.log:

[data]  1332.2M train tokens  val=10.0M  vocab=40  bin-dtype=uint8
[model] 94.4M params (94.4M non-embedding)
[autobatch] micro_bs=12  (16 fits at 7.40 GiB but leaves 0.02 GiB < 0.60 headroom)
[batch] micro_bs=12 accum=11 -> 135,168 tokens/step
[train] 7,505 steps = 1.01B tokens (10.7 tokens/param), 10.34h at 27.5k tok/s
[final] val 0.4152  train 0.4188

The val curve is worth looking at because it nearly fooled me: it sits flat at ~0.488 from step 5250 to 6000 — rising at 5750 — and then drops 0.072 over the last 1,500 steps once the WSD decay phase begins. Judging the run at step 6000 would have called it converged with 19% of its total improvement still to come. Full curve in CHESS.md.

Evaluation (chess/eval_chess.py)

The plan's three headline metrics, plus the board probe — the interesting one.

mode	what it measures	why it matters
legal	% of generated moves that parse as legal SAN	a chess model that proposes illegal moves is broken, full stop
stockfish	Elo from win/loss vs Stockfish at fixed skill/depth	actual play strength, not perplexity
puzzle	solve rate on real Lichess puzzle positions	tactical lookahead, independent of any game the model saw
probe	legal-move rate offline; board probe	see below

python chess/eval_chess.py --ckpt data/chess_out/ckpt.pt --mode legal --n-games 200
python chess/eval_chess.py --ckpt data/chess_out/ckpt.pt --mode stockfish --opp-elo 1450
python chess/eval_chess.py --ckpt data/chess_out/ckpt.pt --mode probe --probe-layer 10

The board probe (the headline result). Freeze the model, run real games, read the residual-stream activations at a chosen layer, and fit a linear (logistic) readout that predicts, per square, which piece (if any) sits there — 13-way over 64 squares. If moves follow instantly from a linear decode of the model's hidden state, the board is genuinely internal, not something reconstructed only in the final layer's attention.

Probe train/test are split on whole-game boundaries (a random position split inflates accuracy — positions one ply apart are near-identical), so the number is honest. The baseline to beat is the majority class (an empty square, ~65%): anything meaningfully above that shows real board knowledge.

Measured results (chessLLM v1, step 7505, val 0.4152 — full analysis in CHESS.md, raw logs in logs/evals/):

legal moves      99.22%  (11,216 moves over 200 continuations, python-chess verified)
survival         55.6 plies to first illegal move (27.8 full moves)
board probe      85.6%   (layer 10, vs 64.5% majority-class baseline, 0/64 squares at chance)
Elo              ~1131 +/- 103  (8W-5D-47L over 60 games vs Stockfish skill 3)

Against the §9 targets (99.5% legal / 1300–1600 Elo / >90% probe): legality clears, Elo and probe do not. Two results worth stating plainly rather than burying:

• 40% of Stockfish games ended in an illegal move, from a model that is 99.22% legal in self-play. Not a contradiction — self-play stays inside the training distribution, and an engine does not. The Elo figure therefore conflates weak play with board-tracking collapse under distribution shift.
• The rating-bucket prompt does not control strength. 40 games per bucket: <15> and <20> produced identical records, <24> was slightly worse, all inside ±126. A clean negative result.

stockfish mode needs the engine on PATH. On Arch it is not in the official repos (pacman -S stockfish → "target not found"); it is AUR-only. The official static binary needs no sudo:

curl -sL -o /tmp/sf.tar https://github.com/official-stockfish/Stockfish/releases/download/sf_18/stockfish-ubuntu-x86-64-bmi2.tar
tar xf /tmp/sf.tar -C /tmp && install -m755 /tmp/stockfish/stockfish-ubuntu-x86-64-bmi2 ~/.local/bin/stockfish

Pick the build by CPU (grep -oE "avx512f|avx2|bmi2" /proc/cpuinfo | sort -u); this laptop's i7-13620H has AVX2+BMI2 but no AVX-512, so -bmi2 is the right one.

The code is already DDP-ready and the T4 fp16 path is handled:

torchrun --nproc_per_node=2 train.py --preset large --hours 8 --compile 0

Note T4 is Turing: no bf16, no Flash-Attention, PCIe-only interconnect. It's ~2× this laptop in aggregate, against a 9h session cap and 30 GPU-hr/week quota. Worth it for a large v2 run, not for the first one.