staging → master - #208
Open
george-connito wants to merge 42 commits into
Open
Conversation
…ssions
Measurement-only pass against near-duplicate submissions (copy + noise /
brief finetune), which evade the existing exact-score-tie penalty in
finalize_round_scores. During idle ticks of the bg-eval worker, average
pairs of the round's top-5 positive-scoring submissions, evaluate the
merged model on the round's cached eval batches, and LOG loss(avg) vs
each side plus delta-cosine similarity — raw merge_penalty and
would-flag verdicts at multiple thresholds, under both candidate
predicates. No enforcement: the pass never touches scores, weights,
journal, or round lifecycle.
- connito/validator/dedup.py: pure helpers (average_state_dicts,
delta_cosine, select_pairs, shadow_report, recover_val_loss,
find_submission_path)
- bg-eval worker: incremental one-pair-per-idle-tick pass (real miner
evals always win the GPU), per-round state keyed by round_id (survives
the stuck-lock recycler), deepcopy-per-pair (never mutates
_eval_base_model), gpu_eval_lock acquired in-thread with in-thread
deadline (the _run_eval precedent)
- retention_top_k(): both prune sites keep top-dedup_top_k files while
the filter is active so pair files survive until measured
- EvalCfg: dedup_filter_mode ("off"|"shadow", default off, deliberately
UNLOCKED so shadow can be enabled per-host), dedup_top_k=5,
dedup_max_pairs=10
- MinerEvalJob.val_loss + Round.scores_snapshot() accessors
- telemetry: pairs-evaluated / would-flag counters (no uid labels)
- tests: connito/test/test_dedup_filter.py (20 cases)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t branch `get_base_model` carried a 4-bit BitsAndBytes/Unsloth branch gated on `config.model.use_quantization` / `model.use_unsloth`. Neither field was ever declared on `ModelCfg`, and `BaseConfig` sets `extra="ignore"`, so the gate was permanently False and the branch unreachable. It was also wrong in three further ways had it ever fired: it returned a stock `AutoModelForCausalLM` instead of the custom MoE class, it sat above the `partial` split so it would have handed back a full 64-expert model where a partial one was expected, and `bitsandbytes` is not pinned so `from_pretrained` would have raised ImportError. Delete it, along with the now-unused `unsloth` pin and its doc references. Also collapse the four copy-pasted `precision -> model_dtype` blocks into `helper.resolve_precision` / `helper.resolve_model_dtype`. These have to agree across sites — a model built at one dtype and autocast at another produces silently different losses — and they were drifting. Finally, point pytest at `connito/test` and add a CI test job. `testpaths` had pointed at a `tests/` directory that has never existed, so a bare `pytest` collected nothing and reported success, and there was no test job in CI at all. Four tests already fail on master and are stale rather than broken code; they are deselected explicitly in the workflow, with the reason for each, so the list stays visible and shrinks as they are triaged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…default off)
Adds `model.quantization: off | int8`, a runtime weight-only int8
representation. Weights are still loaded and saved at `model.precision`;
int8 only changes how they are held in memory in between.
Hand-rolled rather than bitsandbytes/torchao, for two reasons that are
properties of this codebase rather than preferences:
* Production never loads weights through `from_pretrained` — both roles take
the partial path in `get_base_model`, which constructs a bare module and
streams safetensors into it, so a `quantization_config` has no effect.
* Both libraries only rewrite `nn.Linear`. The routed experts are stacked raw
3D `nn.Parameter`s consumed by `F.linear` on a slice, so either library
would silently skip the tensors that dominate memory.
The load-bearing property is that `state_dict()` is dequantization
transparent: a quantized module serialises exactly the keys, shapes and dtypes
it would unquantized (int8 values and scales are persistent=False buffers) and
re-quantizes plain fp16/bf16 on the way back in. That is what lets the
checkpoint hash contract, the validator's per-miner graft and the save path
keep working untouched, and it is covered by tests.
Scope differs by role, and not for want of trying. On a validator eval model
the whole module is frozen, so the routed experts can be quantized wholesale.
On a miner they cannot: each layer's stacked tensor interleaves the trainable
group with the frozen helper group, and `freeze_parameters` necessarily marks
the whole thing trainable — freezing helper slices would change what the
trainable experts co-adapt against, and therefore change what miners submit.
Miner scope is the frozen non-expert Linears only.
Quantizing the validator's `global_model` is forbidden outright, and asserted:
int8 weights are buffers, so merge and the outer optimizer would silently skip
every converted tensor — no exception, no warning, just decaying vtrust.
`require_not_quantized` guards merge, the round snapshot, save, and the three
streaming loaders that mutate `state_dict()` tensors in place.
Consensus: this changes `val_loss`. A validator on int8 is not comparable with
an fp16 one for the same `combined_seed`. Default off, deliberately unlocked so
per-host staging validation is possible, and reported as a `connito_validator`
telemetry label so divergence is diagnosable. Lock it if it ever becomes the
fleet default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Foreground eval passed `base_model=global_model` directly while the background worker got its own `deepcopy(global_model)`. That asymmetry is harmless today but blocks int8: `global_model` cannot be quantized (merge and the outer optimizer walk `named_parameters()`, and int8 weights are buffers), so quantizing only the background copy would score some of a round's miners in int8 and others in fp16, against mismatched baselines, and then rank them against each other into the `_RANK_TO_SCORE` cliff. Give foreground its own persistent eval model, re-seeded from the round snapshot each round — the load routes through the re-quantizing `_load_from_state_dict`, so the snapshot stays fp16 and nothing here has to know about int8. Gated on `model.quantization != "off"`. With the toggle off, `resolve_foreground_eval_model` returns `global_model` itself and no third model is ever built, so this is a no-op by construction rather than by measurement. That matters: an unconditional persistent copy would cost every fp16 validator ~8-10 GB of resident VRAM for a dormant feature, and off is the fleet-wide state for the whole shadow period — and permanently if the rank-preservation gate fails. The per-miner `copy.deepcopy` in `load_model_from_path` is deliberately left alone. It copies whatever the base model holds, so an int8 base already makes every per-miner copy ~2x smaller. Replacing it with in-place grafting would trade deepcopy's pristine-base guarantee for bookkeeping, in a system where a 1e-6 perturbation is documented to move a miner 54 places; it saves memory in fp16 too, so it stands alone as its own change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…(dtype=) `Int8Linear.compute_dtype` and `CustomDeepseekV2Experts._compute_dtype` are what `_save_to_state_dict` dequantizes into, so a stale value makes `state_dict()` report a dtype the module no longer uses — which shows up as a dtype mismatch in the validator's graft path rather than as an obvious error. `Module.to(dtype=...)` never updated them. Nothing in production casts a quantized model today: quantization is applied last, and `require_not_quantized` guards the one `.to(dtype=...)` that could reach one. But that is an ordering property, and ordering properties rot. Probe what `_apply`'s fn did to a throwaway float and adopt it, rather than relying on the guard staying in place. Same treatment the fp32 scales already had. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rank-preservation gate ran, and validator int8 failed it. Measured on an
L40S against 7 real miner shards on the production eval mix (C4 +
Nemotron-CC-Math), 21 batches @ seq 1024:
- The top-3 ordering changes. `_RANK_TO_SCORE` pays by position
(2.25/1.5/1.0), so this silently redistributes rewards: brother-winter/bro03
goes 1st -> 3rd, maximso/con1 goes 2nd -> 1st.
- int8 manufactures exact val_loss ties out of a tie-free population — 0 pairs
under fp16, 2 pairs under int8 (con1==con5, con9==con10). The exact-tie
branch in `finalize_round_scores` then zeroes all four, including the two
miners int8 itself ranked first and second. int8 creates the collision and
the scoring rule punishes miners for it.
- Per-miner perturbation is 0.36x the best-to-worst spread, against the <0.1x
bar this was gated on.
(An earlier C4-only run put the noise ratio at 1.65x. Nemotron-Math
discriminates miners much better — spread 0.005062 vs 0.001243 — so the
production number is far kinder. It still fails, on ordering and ties rather
than on raw noise.)
Given that, a startup warning is not enough: both failure modes are silent in
production and one of them costs real miners a round's rewards. So refuse to
start, with the numbers in the error, unless CONNITO_ALLOW_VALIDATOR_INT8=1 is
set — an escape hatch for re-running the gate on a staging hotkey, not a tuning
knob. The miner-side toggle is unaffected and needs no override.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`quantize_eval_model_` raises the first time a round needs an eval model, which is after `Round.freeze` has already locked the roster. An operator who set `model.quantization: int8` should learn about it while reading the startup log, not from a traceback part-way through their first cycle. Check it in `run()` before the telemetry server comes up, routing through the same function so the explanation and the override stay in one place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matches the reference implementation in ~/experiment (partial_moe.py:FP8Linear,
17c878d) so loss measurements transfer between the two codebases. Storage
format and forward math only — scope, attachment points and the
dequantization-transparent state_dict() contract are unchanged.
float8_e4m3fn / QMAX 448, cast without an explicit round or clamp.
Forward dequantizes in the activation dtype, matching the reference exactly
(including the stacked-experts path); serialization keeps fp32 arithmetic.
model.quantization: "int8" -> "fp8"; CONNITO_ALLOW_VALIDATOR_INT8 ->
CONNITO_ALLOW_VALIDATOR_FP8.
Both formats are one byte per weight, so this buys no memory (measured
identical at 8.428 GB) and costs 3.19x fidelity: 2.645% relative weight error
against int8's 0.829%. Comparability is the whole return.
Three traps fp8 has that int8 did not, each found by probing on an L40S and
each now pinned by a test:
* float8_e4m3fn IS a floating dtype, so Module.to(dtype=) upcasts the weight
buffer and the memory saving evaporates with nothing raised. Both _apply
overrides re-assert the storage dtype. Casting back is exactly lossless.
* torch.isfinite raises NotImplementedError for fp8 on CPU and CUDA alike.
The miner's post-setup guard filters buffers on is_floating_point(), which
int8 failed and fp8 passes, so this would have crashed every fp8 miner at
startup. quantization.all_finite uses torch.isnan, which is exhaustive
here: e4m3fn has no infinity encoding.
* e4m3fn overflow produces NaN, not saturation. Safe only because the per-row
scale caps |w/scale| at 448.000031 and everything up to 464 rounds back to
448 — verified over a 200k-row adversarial sweep.
Two dequantization functions, deliberately: in fp16 a row whose scale rounds to
zero (amax < 1.3351e-5) would be silently deleted from a checkpoint by the
activation-dtype multiply, so state_dict() keeps fp32 arithmetic.
The validator refusal now separates int8-measured evidence from fp8 inference:
the rank gate was only ever run under int8 and int8 failed it; fp8 is coarser
and has not been gated in its own right. A test pins that wording.
Also: FP8Linear._save_to_state_dict moves buffers to CPU before dequantizing
rather than after, which is what its comment already claimed (save-path VRAM
overhead 0.168 GB -> 0.000 GB); and _load_from_state_dict no longer allocates a
second full stacked clone per graft just to populate a debug log field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running a second validator on a hotkey that a production validator already
uses lets a test instance be evaluated by the real fleet without registering
(and paying for) a separate hotkey. It is only safe if the second process
acts under that shared identity nowhere. `CONNITO_VALIDATOR_OBSERVER=1`
closes the three channels where it otherwise would.
**Chain extrinsics.** All six writes — the startup commit, MinerCommit1,
both ValidatorCommits, and both `set_weights` paths — funnel through
`ChainSubmitter`, so gating its three public methods covers them with no
change at any call site. Suppressing `set_weights` alone would not be
enough: `ValidatorChainCommit` carries a per-cycle `miner_seed` that every
validator hashes into the seed driving fleet-wide miner assignment, and
`build_chain_checkpoints_from_previous_phase` pairs the commit-1 signature
with the commit-2 hash by hotkey — so interleaved commits make the live
validator's own checkpoint fail signature verification.
**Hivemind all-reduce.** `HotkeyAuthorizer` admits any whitelisted hotkey,
so an observer on a shared hotkey is accepted into the same
`expert_averaging-group{id}` prefix and its gradients average into the
fleet's global model. It has to skip the DHT outright; handing hivemind a
throwaway wallet instead just crashes the process, since peers reject the
unwhitelisted signer and the bootstrap raises outside the run loop's `try`.
`sync_grad_across_validators` iterates `group_averagers`, so an empty dict
leaves the merge phase a clean no-op.
**HF upload** needs no code — pointing `hf.token_env_var` at an unset name
makes `get_hf_upload_readiness` decline. Keep the real `HF_TOKEN` in the
environment: `load_dataset` relies on ambient auth and the eval set is gated.
Env var rather than a config field, because `check_and_prompt_locked`
rewrites the YAML on startup and a deployment that does not set it cannot
inherit it. Off by default, and only the exact value `1` enables it, so a
typo cannot silently stop a live validator from submitting.
Also stamps `observer` onto `connito_validator_info` — hotkey and uid are
identical between the two processes by design, so that label is the only
thing distinguishing the two scrapes.
`sign_hash` was the last thing in the validator that needed the hotkey's private key: no chain extrinsic is submitted in observer mode, and the DHT — whose `HotkeyAuthorizer` signs every request — is skipped. Its only consumer is the `SignedModelHashChainCommit` on the next line, which is suppressed too, so producing the signature was pure cost. Gating it changes what has to be placed on the test host. An observer now needs the hotkey's *address*, which is public and already visible on the metagraph, but never the key that signs with it — so a live validator's private key does not have to be copied onto a shared box to run one. That is a stronger guarantee than the env var alone gives: a public-only keyfile cannot act under the shared identity even if the flag is wrong, and would fail loudly rather than quietly submit. The hash itself is still computed — `model_hash` is read on the next line and drives eval, and `sign_hash` was what lazily triggered it.
feat(validator): observer mode for a validator sharing a live hotkey
…ion-toggle # Conflicts: # connito/shared/telemetry.py # connito/validator/run.py
Review question on #205: after `delattr(self, name)` drops the stacked parameter from `_parameters`, is `self._compute_dtype` still readable? It is, and the code was already correct — but nothing said so. The assignment must stay above the `delattr` because it is the last read of `param`, and it survives because a `torch.dtype` is not a tensor, so `nn.Module.__setattr__` routes it to `__dict__` instead of `_parameters`. Both facts are invisible at the call site and a reordering would break `_stacked_dtype`, `_expert_slice` and `state_dict` at once.
Feat/fp8 quantization toggle
Adds `dedup_filter_mode: "enforce"`, which zeroes BOTH sides of every pair the merge-loss filter confirms redundant. Generalizes the existing exact-score-tie rule from bit-identical to near-identical submissions — a 1-ULP perturbation on 2 scalars out of 1.77B is otherwise enough to slip past it (docs/diagnosis-2026-08-10-duplicate-submissions.md). Threshold is pinned at 0, making the decision a pure SIGN test. Measured on 7 live submissions from round 8814586: pairs containing a miner with a genuine per-cycle training history came out negative (-4.8e-4, -4.9e-4) and are not flagged; near-copy and noise-injected pairs came out >= 0 (+1.2e-5 .. +5.8e-4) and are. Every threshold >= 0.01 in SHADOW_THRESHOLDS flags 100% of live pairs, honest miners included, and is unusable here. - dedup.py: `compute_merge_penalty` + `is_redundant(penalty, threshold)`. Enforcement acts on the UNROUNDED penalty — `shadow_report` rounds to 6 dp, where a tiny negative becomes `-0.0` and `-0.0 >= 0` is True, which would flag an honest pair. - round.py: `dedup_flagged_uids` + lock-guarded `mark_dedup_flagged`. - evaluator.py: flagged UIDs are excluded from the rank ladder and get an explicit 0, alongside `tied_uids`. Read via `getattr` because `finalize_round_scores` also runs against the journal-recovery path's duck-typed `_RecoveryRound`, which has no dedup state. - config.py: `dedup_threshold: float = 0.0`, still unlocked so a single host can enforce ahead of the fleet. - tests: 6 new dedup cases (sign test, threshold-widening, the -0.0 rounding trap, enforce flags/skips, shadow never enforces) + 2 finalize_round_scores cases. KNOWN GAPS, both open before this should be enabled in production: 1. The run-to-run noise floor of `merge_penalty` is unmeasured. Signal is ~5e-4; if repeat evals of one pair vary by that much, the sign is not stable and this zeroes honest miners at random. 2. Zeroing both sides is safe only if in-round copying is impossible. The commit-reveal should guarantee it (MinerCommit1 signs the hash before MinerCommit2 reveals), but it is UNVERIFIED that the signature must land in the MinerCommit1 window. If it need not, copying a top miner becomes a way to zero them. Separately: the pass is still triggered only on bg-eval idle ticks, and on production those never occur — `no pending targets` logged 0 times in 100 minutes against a 7m50s eval window. The trigger needs to become "one pair per K evals" or enforcement will never fire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shadow pass hung entirely off bg-eval idle ticks, and on a full roster those never arrive. Measured on production: `no pending targets — going idle` was logged ZERO times in 100 minutes, against a 7m50s eval window in which the worker completed 44 back-to-back evals before Merge cut it off with work still queued. The pass never executed once — not even to skip a pair — so `validator_dedup_pairs_evaluated_total` sat at 0.0 for the whole deploy. There is no spare GPU time to donate. Adds `dedup_eval_interval` (default 8): run one merged pair after every N completed evals, so measurement is proportional to work actually done. At the observed ~44 evals/window that spends ~5 of the 10-pair budget and adds ~11% to the window. The idle-tick trigger is kept — it is still the cheapest opportunity when one exists. Set the interval to 0 to restore idle-only behaviour. The cadence lives in `_tick_interleaved_dedup()` rather than inline in the loop so it is testable without driving the whole worker. - background_eval_worker.py: `_dedup_evals_since_pair` counter, reset per round alongside the rest of the dedup state; `_tick_interleaved_dedup`. - config.py: `dedup_eval_interval: int = 8`. - tests: cadence fires every N, 0 disables it, counter resets on a new round. Unchanged: every gate inside `_maybe_run_dedup_shadow` still applies, so an interleaved call during Merge or outside the eval window is a no-op, and the whole thing is inert while `dedup_filter_mode` is "off". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…placing evals Reverses the interleave default and replaces it with the window it was always meant to use. WHY INTERLEAVING WAS WRONG The bg-eval window is bounded by phase transitions, not by work remaining (`eval_window_active.clear()` at MinerCommit1 - 5). It cannot absorb extra work. A merged pair run mid-scoring therefore DISPLACES a miner eval, and an unevaluated miner scores 0 for the round — the filter's cost lands on honest miners. `dedup_eval_interval` now defaults to 0. WHY IT IS NOT NEEDED The window spans ValidatorCommit1 → Train, and measured on production the roster was graded in ~27 min of a ~68 min Train (40 miners, ~41 s each), leaving ~41 min idle with the eval base model still resident (10,467 MiB, 0% util). Ten pairs need ~7 min of that. The idle-tick trigger already covers it; `_wait_clear` never tears the model down, so nothing has to be re-materialised. Two earlier diagnoses in this branch were wrong and are corrected here: "the worker never idles" (the `no pending targets` line is suppressed unless idle_ticks == 0, and idle_ticks is also bumped in the gated branch) and "the round is empty during Train" (validator_miner_last_scored_round_id only updates at finalize, so it cannot show the in-flight round). WHAT THIS ADDS - Deadline guard. `eval_window_close_block()` computes MinerCommit1 - 5 from the cycle periods; run.py publishes it as a monotonic deadline when it opens the window. The pass refuses to START a pair with less than `dedup_pair_budget_sec` (120 s) left, because finalize applies verdicts and submits weights in the same step — a later result misses the round entirely. Inactive when unpublished, so behaviour is unchanged without it. - Skip-reason logging. Every early return was silent, which is why a pass that never produced a pair was indistinguishable from one never called. Each gate now names itself, and `no_pairs` carries scored/positive counts — the counts that would have settled this immediately. Logged on CHANGE only; the pass is polled every ~2 s across a 68 min phase. - `dedup_pair_budget_sec` config; `dedup_eval_interval` kept as an escape hatch for a roster whose grading already overruns Train. Guarded: `eval_worker` is None when background_worker_enabled is false, so the deadline publish is skipped rather than warning every cycle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_emit_dedup_summary` returned early when `budget_used == 0 and
pairs_skipped == 0`, which silenced exactly the case worth reporting: a
round where the pass WAS reached but every attempt was gated. At round
granularity "produced nothing" and "never called" were indistinguishable
— the ambiguity that made this take three attempts to diagnose.
Now gates on the filter being enabled rather than on whether work
happened, and carries `last_skip_reason` + `mode`. A disabled filter is
still silent. A blocked round now emits, e.g.:
dedup-shadow: round summary round_id=… mode=shadow budget_used=0
pairs_evaluated=0 last_skip_reason=no_pairs reason=round_transition
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flagged miners were being removed from the ranking list, so the
survivors re-ranked from position 0 and the 2.25 / 1.5 / 1.0 slots
refilled from below. Two things go wrong with that cascade:
* It promotes miners nobody examined. The pairwise filter only ever
compares the top `dedup_top_k` (5), so zeroing the top three hands a
paid slot to #6 — a submission that was never compared against
anything. In round 8814586, 70 of 96 miners sat inside duplicate
clusters, so the promoted miner is more likely than not to come from
the same cluster we just penalised.
* It pays an attacker for taking out a rival. Copy the leader from a
throwaway UID, the pair is flagged, both are zeroed — and every UID
behind the leader gains exactly one rank, including the attacker's
other UIDs. Symmetric zeroing plus a cascade turns the filter into a
weapon aimed at whoever is winning.
Flagged miners now keep their rank position and are paid 0, so the slot
is burned rather than reassigned. No unflagged miner's score changes.
The exact-tie rule keeps its existing cascade on purpose: it applies
even with `dedup_filter_mode: off`, so changing it would alter
production scoring well beyond this feature.
This does NOT withhold emission, and it is not meant to. Weights are
normalized before submission and scores are a rolling ~8-round window,
so the round still pays out in full — burning redirects the share from
"this round's #4-#6" to the recent historical field, which is a much
harder target to game than being ranked sixth once.
Telemetry: `top3` now logs the awarded score rather than the rank's
nominal value (a burned slot must read 0 or the log contradicts the
aggregator), plus a `dedup_flagged_count`.
Tests: the old expectation is replaced by the burn semantics, and an
anti-griefing invariant is added — flagging the leader must leave every
unflagged miner's score byte-for-byte unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`select_pairs` re-reads `scores_snapshot()` on every trigger, so "top K" meant "top K of whoever happens to be graded right now". Miners are graded stalest-first — an order unrelated to quality — so an early sample is a RANDOM subset of the field, not the leaders. That interacts badly with the budgets, which are asymmetric: `dedup_top_k` bounds a single call, `dedup_max_pairs` bounds the whole ROUND. With `dedup_eval_interval: 2` the first pair fires after two evals and all ten are gone inside the first twenty — drawn from a half-filled scoreboard. By the time the real top-5 exists there is no budget left to compare it, so the miners actually about to be paid are the ones never examined. In `enforce` mode this is not merely wasted measurement. `dedup_flagged_uids` is only ever added to and finalize zeroes anything in it, so a miner flagged against a neighbour who later fell down the board is still zeroed — on a comparison the final ranking would never have chosen to make. Reserve the tail of the eval window instead. Once it opens the worker stops claiming miner evals, so the ranking cannot move between one pair and the next, and `select_pairs` reads a settled board. The span is `dedup_max_pairs * dedup_pair_budget_sec`, which is the pass's own worst case for its own budget — the last pair therefore starts exactly where `_dedup_window_allows_pair` still permits one. New `dedup_freeze_field` (default on) turns it off and restores the idle/interval triggers. The freeze only arms when `dedup_filter_mode` is shadow/enforce. Without that check the default `off` would stop claiming evals for the whole reserved span and spend it doing nothing, since the pass's mode gate returns immediately — a 20-minute stall on every validator. Cost, stated plainly: miners still ungraded when the tail opens score 0 for the round. On a validator with a long idle tail (measured: roster graded in ~27 min of a ~68 min Train) that is free. On one whose grading already overruns the window it is real, so the freeze logs `scored` / `positive` / round stats at the transition rather than going quiet. Tests: tail derivation, freeze disabled, filter-off never freezing, and that the pass refuses to spend a single pair before the board settles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shadow-mode merge-loss dedup filter for top submissions
Adds `include_linears` to `quantize_model_` and passes False from `quantize_eval_model_`, so the validator's fp8 scope is expert projections only — the MLA projections, first_k_dense_replace dense MLPs and shared_experts stay at full precision. Why: the reference implementation this format was matched to (`~/experiment/partial_moe.py:quantize_expert_fp8`, 17c878d) quantizes expert projections and nothing else. We already matched its representation (e4m3, per-output-row, activation-dtype dequant in the forward) so that loss measurements transfer between the two codebases — but a delta only transfers if both sides are quantizing the same set of tensors, and ours was also converting 162 backbone Linears. Same scope now, so the numbers are comparable. Little is given up: on a partial model the experts are ~1.8 B params against a ~0.4 B backbone, so experts-only keeps most of the memory saving. The miner path is untouched — `include_linears` defaults to True, and the miner's stacked expert tensors can't be quantized at all (trainable and helper groups share one tensor per layer), so backbone Linears are the only thing its toggle has to work with. No production behaviour change: validator fp8 still refuses to start without CONNITO_ALLOW_VALIDATOR_FP8=1, and `model.quantization` still defaults to off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…real hardware
Records the evidence behind the experts-only scope this branch introduces, and
corrects a refusal message that has since become false.
`tools/quantization/` holds three harnesses that need an L40S, the 30 GB
DeepSeek-V2-Lite checkpoint and the production eval datasets, so they cannot
live in `connito/test/`. The CPU-only invariants stay there; these answer the
questions that only real hardware can.
The parity result, measured on the stock HuggingFace model — the experiment
repo's own base (`model.py:72`) — with our `FP8Linear` applied exactly where its
`quantize_expert_fp8` applies its own:
- relative weight error 2.654% mean across all 4992 expert projections
(min 2.645%, max 2.660%), reproducing the reference's ~2.7%
- val_loss 1.449180 -> 1.448177 on identical seeded batches
- 29.36 -> 16.98 GiB resident
That error figure is what makes loss deltas transfer between the two codebases,
which was the point of matching the scope in the first place.
Two findings that fell out of running the full-experts configuration, both
recorded in the README rather than fixed here since neither is this branch's
subject:
- `partial=False` leaves every routed expert at `_init_weights` random. The
checkpoint names them `...experts.{N}.gate_proj.weight`; we serialise the
fused `...experts.{N}.gate_up_proj`. The partial path translates between the
two, the full path is a bare `load_state_dict(strict=False)` that swallows
4992 unexpected and 52 missing keys. Same weights score 1.449 through the
stock loader and 9.956 through ours.
- `resolve_foreground_eval_model` copies before it quantizes, so a
full-experts fp8 validator needs two full fp16 models at once and OOMs at
43.69 of 44.39 GiB.
The refusal in `quantize_eval_model_` claimed fp8 "has NOT been gated in its own
right". It has been since, and it failed the same way int8 did — 0.21x
perturbation against a <0.1x bar, top-3 reordering, one manufactured exact tie.
Saying so is a stronger argument for the block than the old appeal to fp8 being
coarser, and it stops anyone reading the message from concluding the gate is
still outstanding. Also adds the trap that message was missing: a ~0.001
absolute loss shift is negligible for training and comparable to the entire
inter-miner spread the ranking has to resolve, which is exactly how a harmless
loss delta becomes a reordered podium.
The param counts in the scope comment were wrong (~1.8 B experts against a
~0.4 B backbone); measured, it is 14.39 B routed experts against 1.32 B.
Verified on the L40S: all three harnesses load and run from the repo path, the
refusal raises with the new text, the override path still starts, and
test_validator_observer_mode passes (14 tests).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`get_base_model(partial=False)` returned a model whose every routed expert was
still at its `_init_weights` random value — 14.39 B of 15.71 B parameters — with
no exception and no warning.
The full branch of `load_pretrained_model_low_mem` was a bare
`load_state_dict(strict=False)` against the HuggingFace checkpoint. The
checkpoint names routed experts `...experts.{N}.{gate,up,down}_proj.weight`, one
tensor per expert per projection; `CustomDeepseekV2Experts` stores each layer's
experts stacked, gate and up fused, under `...experts.{N}.gate_up_proj`. Nothing
matched in either direction, and `strict=False` swallowed both halves: 4992
unexpected keys in, 52 missing keys unfilled.
The partial path has translated between the two all along, in
`_apply_pretrained_tensor_to_partial` — it buffers `gate_proj` until `up_proj`
arrives for the same expert, concatenates them, and writes the pair into the
right slice. The full path simply never called it. This routes the full path
through the same translator and the same streaming loaders, so both paths now
share one implementation rather than one path having a silent gap.
`assignments_from_expert_modules` supplies the local->global mapping those
loaders need. A partial build derives it from its group assignment; a full build
has no group, so it reads `expert_indices` off the modules themselves. It does
not assume the identity — a full build happens to be 0..N-1, but the helper
reports whatever layout is there, and a test pins that.
Streaming is a second, free win: the old path materialised a whole CPU model and
a whole pretrained state_dict before copying between them, peaking at 59.6 GB of
host RAM on a 62 GB box. Streaming holds the model plus one mmap'd shard,
~37 GB. For reference, HuggingFace's own loader peaks at 18.9 GB for the same
checkpoint, so there is more to take here later.
Measured on an L40S before this change: the full model scored 9.956 on seeded
eval batches where the stock HuggingFace model scored 1.449 on the same batches.
Expert tensors mismatched the checkpoint while the backbone matched it exactly,
and the loaded expert values had mean magnitude 0.01595 against the initialiser's
0.02*sqrt(2/pi) = 0.01596 — untouched, not partially loaded.
Verified: the 6 new tests pass. The end-to-end check (a corrected full model
should score ~1.45, and `tools/quantization/gpu_full_load_check.py` should report
`experts_loaded: true`) has NOT been run — it needs ~37 GB of host RAM and the
box is currently running the tester validator. Four unrelated tests fail on this
branch; all four fail identically on pristine staging with these changes
reverted, so they are pre-existing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#118 (2a51426, 2026-05-07) made `.safetensors` the expert-shard save format. Two fetch sites were not migrated with it and have asked for `.pt` alone ever since, so both 404 against every checkpoint published in the last three months: - `_build_download_targets` — the validator's global-model fetch from chain. A validator that loses local state can never recover the subnet's model, and no validator ever picks up a peer's newer checkpoint. Observed live: 404 on `g-connito/co/resolve/a26aba5/model_expgroup_4.pt`, followed by a silent cold start from pretrained. - `hydrate_miner_submissions_from_hf` — recovering miner submissions from HF. Dead by the same mechanism. Confirmed against that checkpoint: it carries `model_expgroup_3.safetensors` and `model_expgroup_4.safetensors`, and zero `.pt`. Everywhere else already handles both suffixes — `background_download_worker`, `dedup`, `evaluator` and `checkpoint_helper` all go through `MINER_CHECKPOINT_SUFFIXES`, and the bg worker already names its output after whatever it actually downloaded. These two were the outliers. `.pt` is kept as a fallback rather than deleted. It is tried only when the preferred suffix is missing, and only on `HFFileMissingError` — a dead repo, an auth failure or a timeout means the candidate is unusable, so retrying it would double the latency of every real failure for nothing. The two suffixes go out as separate requests because `download_checkpoint_from_hf` raises on the first absent entry: one call listing both would fail against every repo, since none carries both. `hydrate_miner_submissions_from_hf` had a third defect in the same function: it scanned for already-present submissions with `glob("*.pt")`, which matches none of the `.safetensors` files the background worker writes, so every poll re-downloaded every miner it had already fetched. Now uses the same `MINER_CHECKPOINT_SUFFIXES` sweep as its siblings. The destination filename now carries the suffix that was actually downloaded. `load_state_dict_from_path` dispatches on it, so writing safetensors bytes into a hardcoded `.pt` name would have routed them to `torch.load` and failed — a trap that only appears once the fetch is fixed. `test_hydrate_miner_submissions_from_hf_writes_assigned_miners_only` asserted `model_expgroup_0.pt`, i.e. it pinned the bug rather than catching it; updated, and a companion added for the legacy-fallback path. New `test_expert_shard_suffix.py` covers the target builder and the fallback ladder, including that a non-file error is not retried. Four tests fail on this branch; all four fail identically on pristine staging with these changes reverted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Feat/fp8 experts only validator
🩹 fix(model): load pretrained routed experts in the full-model path
Makes the full expert topology reachable and routes it the way the base checkpoint does. Three findings, one of which would have silently invalidated the comparison it exists to enable. **1. The full branch was unreachable.** `moe.partial_moe` has been declared in `MoECfg` since the initial commit and read by nothing; `validator/run.py` hard-coded `partial=True`. So the full branch of `mycelia.get_base_model` — including the loader fixed in #210 — could not be entered from any config. Wired the existing field to the call site rather than adding a new one. **2. `full_topk` was wrong: 2, not 6.** It has sat at its initial-commit value for four months because nothing read it. DeepSeek-V2-Lite ships `num_experts_per_tok: 6`, and the reference experiment inherits that — it loads through `from_pretrained` and never overrides the gate (`partial_moe.py` reads `top_k` off the stock gate). Turning full experts on at 2 would have scored top-2-of-64 against the experiment's top-6-of-64: a different forward graph, presented as the same one. `partial_topk` got a deliberate 1 -> 6 bump in 917bb16; this was missed in the same pass. **3. Two differences I expected were not real** — verified rather than assumed, and left alone: `n_group=2` never applies (the group-limited branch needs `topk_method == "group_limited_greedy"`, and V2-Lite ships `greedy`), and `norm_topk_prob` is set on the config but read nowhere in our routing. The remaining genuine gap to the experiment is dtype: it runs bf16, we run fp16. `bf16-mixed` is already supported, so this is a per-host config choice; documented at the field. Not changed as a default — that would move scoring on the partial topology the fleet runs today. **Behaviour is unchanged by default.** `partial_moe` defaults True, so the validator builds exactly the model it does now, and `full_topk` is still read by nothing until an operator opts in. **`partial_moe: false` does not fit on a 46 GB card yet**, and the new startup warning says so with the numbers: ~29 GB of fp16 expert weights against partial's ~2.6 GB, and `load_model_from_path` deepcopies the model *before* `quantize_eval_model_` can shrink it, so the peak is two unquantized models. That OOM was measured, not predicted. fp8 shrinks the copy, not the thing that produces it. Deliberately a warning and not a refusal — the option is for reproducing the experiment on a box that can hold it. Left `partial_moe` out of `_LOCKED_FIELDS` on purpose: locking it would have the reset undo a staging host's edit every restart. It should be locked before it ever becomes a fleet default, since two validators on different topologies score on different graphs. Tests: 7 new in `test_full_expert_topology.py`, including the switch asserted against the real `setup_training` call rather than a recomputation of it — the bug was a literal at that call site. Full suite run in the `:staging` image; the 5 failures are identical with and without this change (4 pre-existing, 1 telemetry port artefact of the container). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pology 🧭 feat(validator): selectable expert topology, and fix full_topk
The full-model build peaked at **59.6 GB resident on a 62 GB host** for a
model whose final size is ~31 GB. Now 38.0 GB. Loaded weights and eval
loss are bit-identical either way — only the peak moves.
`nn.Parameter(torch.empty(...))` and `nn.Linear` take torch's *default*
dtype, which is fp32. Constructing a model and casting afterwards keeps
the fp32 storage for every parameter live until that parameter is
individually replaced, so the peak is ~2x the final size. The comment
above the call site already said not to do this:
# Build at `model_dtype` from the start: a fp32 default + later
# cast would peak at ~2x the final size for a 15B-param model.
model = model_class(moe_config).to(dtype=model_dtype)
The comment was right. The line under it constructed fp32 and cast after.
`mycelia.build_at_dtype` wraps construction in `set_default_dtype`.
Applied at all three construction sites — the full path, its
meta-tensor fallback, and the partial path. Partial was the same bug an
order of magnitude smaller (~8 GB peak for a ~4 GB model); fixed because
it is the same one-line remedy, not because it was hurting anything.
Call sites keep their trailing `.to(dtype=...)`. It is free once
construction is already correct (`Tensor.to` returns self, verified by
`data_ptr` in the tests) and it keeps the end-state invariant pinned if a
submodule ever hard-codes fp32.
`set_default_dtype` is process-global, so the restore is load-bearing:
a leaked bf16 default would silently downgrade every tensor built later
in the process. Restored in a `finally`, tested including the exception
and nesting paths.
**Measured, not argued.** Same script, same box, before and after:
| | before | after |
|---|---|---|
| host peak | 59.6 GB | **38.0 GB** |
| expert gate/up/down vs checkpoint | MATCH, max delta 0 | MATCH, max delta 0 |
| backbone (control) | MATCH | MATCH |
| forward loss | 0.30255791544914246 | 0.30255791544914246 |
38.0 and not ~31 because streaming adds a shard mmap plus the tensor
being materialised; the 2x construction overhead is what went away.
**Repo cleanup carried with it.** `gpu_full_load_check.py` is removed: it
reproduced the pre-#210 bug by calling `load_state_dict(strict=False)`
directly, which is a code path the loader no longer takes, so running it
today reports a mismatch that no longer exists. Replaced by
`gpu_full_fix_verify.py`, which exercises the real entry point
(`get_base_model(partial=False)`) and reads ground truth with `safe_open`
one tensor at a time rather than materialising a second 30 GB state_dict
to compare four tensors. The README keeps the old diagnosis as history —
the failure was silent, so the shape of it is worth not losing.
Tests: 5 new. Full suite run in the `:staging` image; failures identical
with and without this change (4 pre-existing, 1 telemetry port artefact
of the container).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`build_grad_buff_from_model` built a gradient buffer for every expert
group; `validator.run` then deleted all but its own on the next line:
group_grad_buff_meta = build_grad_buff_from_model(...) # builds ALL
excluded = [gid for gid in group_grad_buff_meta if gid != active_group_id ...]
for gid in excluded:
del group_grad_buff_meta[gid] # keeps ONE
Each buffer is a real `torch.zeros(total, device="cpu")`, so this
allocated and binned a full second copy on every validator start:
- **7.25 GB** on the partial topology the fleet runs today
- **27.4 GB** on the full topology, where it was a material part of
running out of host RAM
`group_ids` now scopes the allocation. The delete loop stays as the
invariant check — `group_ids` guards the memory, the loop guards what
reaches the averager.
Name bucketing still runs over every group, deliberately. It costs
nothing (it builds lists of strings) and `include_shared` needs the
complete ownership picture: scoping the *assignment* instead of the
*allocation* would make other groups' experts look unowned and drop them
into the shared buffer, to be averaged with every peer rather than
within their group. There is a test pinning that specifically, because
it is the one way this change could have been quietly wrong.
Observed on the tester at full topology, both buffers reporting the same
size:
Built expert group grad buffer - 4 27456 MB total_numel=14394851328
Built expert group grad buffer - 2 27456 MB total_numel=14394851328
That they are *equal* is a second, separate bug: the stacked-mode
bucketing matches by layer rather than by expert, so on a full model
every group claims all 64 experts of every layer it touches. Not fixed
here — it needs the buffer to hold slices of the stacked tensor, which
changes how gradients are packed and unpacked, and that deserves its own
PR and its own tests.
Tests: 5 new. Full suite run in the `:staging` image; failures identical
with and without this change (4 pre-existing, 1 telemetry port artefact
of the container).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eval base
Scores miners on the full expert topology (all 64 routed experts,
`full_topk=6`) without making `global_model` full — the direct route was
measured off a cliff twice over. Opt-in via
`evaluation.full_topology_eval`; default-off changes nothing.
**Why not `partial_moe: false`.** A full global model cannot merge: the
outer step needs params + `.grad` + SGD momentum resident, measured at
87.8 GiB for 15.71 B params (`gpu_option_b_budget.py`, included) against
44.4 GiB VRAM or 62 GB host. Freezing unmerged experts can't rescue it —
`requires_grad` is per-tensor and a layer's 64 experts share one stacked
tensor. And the merge only ever moves own-group rows anyway: a miner
model is a deepcopy of global + shard overlay, so `g.data - p.data` is
exactly zero everywhere the shard didn't reach. Carrying optimizer state
for 1664 experts to merge ~215 was the whole wall.
**The design** keeps `global_model` partial — merge, outer optimizer,
chain hash, peer resync byte-identical to production — and adds a frozen
full-topology base used only for scoring:
- `requires_grad=False` throughout: no grads, no momentum, no
optimizer state, ever;
- one `load_state_dict(global.state_dict())` refreshes it per round —
the expert serializer emits per-expert keys under *global* ids, so
backbone + merged rows carry across with no new translation code;
- miner shards graft in place through the same overlay loader, with
the touched rows (~3.4 GB, not 29) backed up first and restored
bit-identically after;
- parked on CPU for the merge window, so the merge never shares the
GPU with it.
This is the reference experiment's own architecture
(`~/experiment/partial_moe.py`): trainable local experts apart from a
frozen full set that lives on CPU and swaps in for merged-topology eval.
Its paradigm D — full model, assigned experts trainable at original
positions — is exactly partial-global + frozen-full-base.
**Grafting in place is safe here because the base is disposable** — it
is never hashed, submitted, merged or synced. The failure that rules out
grafting into `global_model` (a died restore corrupting what the
validator commits) costs at worst one round of scores, and the next
round's refresh rebuilds the base from global regardless.
**What in-place grafting newly has to defend, and does:**
- *Hostile shards.* Deepcopy isolation made them free — they could
only poison their own copy. In place, an unvalidated write (another
group's experts, lm_head, the router) would survive the restore and
contaminate every later miner of the round. Shards are validated
all-or-nothing before any tensor is read; `ShardRejected` subclasses
ValueError so the existing statedict_parse_failed handler attributes
the failure to the miner.
- *Orphaned evals.* A foreground per-miner timeout detaches the
awaiter but the thread runs on (long-documented in
`evaluate_one_miner_sync`). With a deepcopy that burned VRAM; with a
shared base an orphan's late restore would interleave with the next
miner's graft and silently corrupt that miner's loss. The base
carries a lock acquired inside the eval thread (bg-eval's
gpu_eval_lock pattern), and prepare/park bump a generation so a
*queued* orphan fails its generation check without touching the
base. A graft that dies mid-overlay self-heals before re-raising.
Baseline and miners share the base, so both sides of every delta are the
same topology. v1 constraints, enforced at startup: bg worker off (its
own template would be a second 29.3 GiB copy), quantization off, and
`partial_moe` true.
Budget on the L40S: eval window = partial global 9.2 (idle) + base 29.3
+ backup ~3.4 + activations ≈ 42-43 of 44.4 — tight, and the number the
live run must confirm; merge window = partial peak 27.6 with the base
parked (29.3 GB of 62 host). `full_topology_eval` is deliberately NOT
locked while opt-in (same reasoning as `model.quantization`): full and
partial rank miners differently (Spearman rho -0.45 across 22 real
submissions, per the reference experiment), so it must be lockable
before ever becoming a fleet default, but a staging host has to hold it
across restarts today.
Tests: 12 new pinning bit-identical restore, all-or-nothing rejection,
stale-generation no-touch, refresh-carries-merged-rows, stale-backup
discard, and the frozen contract. Full suite in the `:staging` image:
failures identical with and without this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three fixes from the first live full-topology round. The mechanism
worked — first full-topology baseline 1.9768 against partial's 3.7839,
51 batches, zero NaN — and the eval-window budget came in at 39.5 GiB of
44.4, better than the ~42 projected. Both problems were around it.
**1. The bg-worker constraint starved the round.** v1 required
`background_worker_enabled: false`, reasoning that bg-eval would need a
second full template (true, 29.3 GiB, does not fit). But that flag also
gates bg-*download*, and foreground reads exclusively from the directory
bg-download fills — the compose file says so in as many words. Observed:
a full window with `discovered_total=0` and `scored=0` against four
claimed foreground UIDs. bg-eval is now suppressed on its own and
downloads keep flowing; the startup check no longer demands the flag.
**2. `park()` allocated 29.3 GB at the worst possible moment.**
`.to("cpu")` reuses nothing — the host tensors were released when the
model moved to the GPU — so parking allocated a second full model just
as the merge phase began wanting memory of its own. The validator died
there: last line `Aggregating miner gradient change locally`, next line a
fresh `Validator starting`, no traceback.
The base now holds references to its original CPU tensors for the
process lifetime and `park()` copies back into them. Host cost is a flat
29.3 GB from build to shutdown, peak host never exceeds steady-state
host, and there is no allocation left to mistime. Pinned by `data_ptr`
in the tests — wrapper identity would prove nothing, the storage address
is the claim.
**3. Neither transition logged anything.** The post-mortem above had a
seventeen-second window and no numbers in it. `prepare_for_round`,
`park` and `build` now emit VRAM used/free and host RSS.
Tests: 3 new (park writes back into pre-allocated storage, park is
idempotent, park preserves values), 15 in the file. Full suite in the
`:staging` image: failures identical with and without this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Watchtower replaces the image and never touches an operator's compose file, .env or config.yaml. The only channel that reaches a running validator's settings is check_and_prompt_locked, which resets locked fields to their class defaults on every start (--auto_update_config defaults to True). "Enforce this fleet-wide" therefore means exactly: ship the default, and lock the field. Four fields have to move together, because check_full_topology_eval_supported rejects the mixtures at startup: evaluation.full_topology_eval false -> true the change being enforced model.precision fp16-mixed -> bf16 the base checkpoint's dtype moe.partial_moe true, locked a full global model cannot merge model.quantization "off", locked fp8 + graft is rejected Leaving any one unlocked hands the operator who had edited it a crash-loop on the release that turns full-topology eval on, rather than a config they can argue with. full_topology_eval is the only new field of the four, so it is the only one with a free ride: pydantic fills the default and config.write() persists it. The other three are existing fields whose operator values survive every pull forever — the lock is the whole mechanism there, not a belt on a brace. precision is safe to lock because resolve_precision downgrades bf16 -> fp16 on a device without BF16 compute, so a card that cannot run the default lands on fp16 through the resolver rather than on a broken config. Staging hosts that need to diverge use --no-auto_update_config, which suppresses the reset. That is also why the run.py guard survives, with its fallback pinned false: a config predating the field must not build a 29 GB base before the locked-field reset has had its say. Tests: 6 new pinning the enforced set and the reset mechanism; the partial_moe test inverted to match, keeping its old rationale as history. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first live round with a full eval window scored zero miners. Baseline fitted — 51 batches, baseline_loss=1.7303 on the full topology — and then all three foreground miners OOM'd: foreground eval: starting baseline_loss=1.7303 foreground_uids=[165, 36, 23] evaluate_one_miner: OOM uid=165 evaluate_one_miner: OOM uid=36 evaluate_one_miner: OOM uid=23 No foreground miners evaluated round_id=8872226 [VRAM before GPU cleanup] allocated=42939.7MB free=213.4MB alloc_pct=94.5% Baseline ran at 39.5 GiB of 44.4; the miners hit 42.9. The ~3.4 GiB delta is the graft backup, because `rows[local].clone()` clones onto the *source* device and the base is on the GPU for the eval window. Nothing reads those rows on the GPU — they are written once at graft and read once at restore — so the eval window was carrying 3.4 GiB it never used and the activations were what did not fit. Each OOM landed ~2.5 minutes into its eval, i.e. in the forward pass, not the overlay. The graft mechanism itself worked; it had nowhere to compute. Staged on the host now, and pooled rather than allocated per miner: every miner of a round touches the same expert set (this validator's group), so storage is allocated once and written over. Same reasoning as `_shadow_*` in park() — flat host cost, no per-miner churn on a box with no swap. Pinned memory deliberately not used: 3.4 GB of page-locked storage on a host already running at 47 GB of 62 trades one pressure for another. Also logs VRAM and host RSS at graft and restore. The same round grew host RSS 31.5 -> 47.3 GB across four eval loader builds and then died entering the merge with no traceback; ~15.8 GB of that is unaccounted for, and these two lines are what will localise it to the eval window or rule it out. Tests: 3 new — backups land on CPU, the pool is reused across miners (data_ptr, not values), and restore stays bit-identical through host staging. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In-round evidence costs one attempt per ~105-minute cycle and does not distinguish causes. Two rounds failed to score a miner: the first OOM'd all three (42.9 GiB allocated, 213 MB free), the second never reached a graft because the submission window had closed before the roster froze. Neither told us whether the memory fixes work. This reproduces the eval window on its own, with the two things that make the numbers real: a ballast allocation standing in for the resident partial global model (~9.3 GiB, without which a 45.5 GiB card looks roomy and any budget claim is meaningless), and a real miner shard off disk rather than a synthesised one, since the overlay's cost depends on how many rows the shard actually touches. Forward passes use synthetic (1, 1024) batches — the shape `get_dataloader` builds for eval, per `per_device_train_batch_size`. Activation memory is a function of shape, not content, and pulling real eval batches would add a dataset download to a test about allocation. Overlay correctness is checked by comparing tensors directly, which is stricter than inferring it from a loss. Measured on the L40S, all seven checks passing: build host RSS 30.13 GB (build_at_dtype) base on card 29.33 GiB graft VRAM delta +0.00 GiB (was ~3.4) backup 410 rows, 3.30 GB staged, on host grafted forward peak 39.67 of 44.39 GiB restore 52/52 sampled rows bit-identical pool reuse (2nd miner) 410/410 rows, host flat 34.47 -> 34.47 park host +0.00 GB, card released `vram_free_gb=4.19` here against the live round's 4.17 — the ballast reproduces the real budget rather than approximating it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(validator): two host-RAM fixes — build dtype, and duplicate grad buffers
Feat/full topology eval base (REDO)
…ounds
Full-topology validators crash-looped on a ~2-cycle rhythm overnight —
23:16, 02:47, 06:18, 09:45 — with a CUDA OOM inside
FullTopologyEvalBase.prepare_for_round:
File "connito/validator/full_topology_eval.py", line 213, in prepare_for_round
self.model.to(device)
torch.OutOfMemoryError: Tried to allocate 704.00 MiB. GPU 0 has a total
capacity of 44.39 GiB of which 639.38 MiB is free.
43.75 GiB was already in use before the base moved. The VRAM log says where
it went:
round start, pre-merge allocated = 9,513.3 MB params
steady, post-merge allocated = 19,027.8 MB params + momentum
19,027.8 is almost exactly 2 x 9,513.3. The outer optimizer is DiLoCo-style
SGD with momentum, so step() allocates a momentum_buffer the size of the
parameter set on its first call and holds it for the process lifetime.
That was harmless while the eval model was a per-miner deepcopy of the
partial topology. It stops being harmless with evaluation.full_topology_eval,
which wants the 29.3 GiB frozen base resident in the same window:
19.0 + 29.3 = 48.3 GiB against a 44.39 GiB card. The first round of every
process fitted because momentum did not exist yet; the second always OOM'd.
The eval-window budget in full_topology_eval's module docstring counted the
partial global model at 9.2 GiB idle and omitted the momentum entirely —
even though tools/quantization/gpu_option_b_budget.py documents that the
buffer "is allocated on the first step() and persists for the process
lifetime". gpu_graft_verify.py's ballast reproduced the pre-merge budget
faithfully and the post-merge one not at all, which is why every measurement
taken so far came from a first round and looked fine.
Nothing reads the buffer between steps, so it has no reason to hold the card
while miners are scored. It is now onloaded immediately before step() and
parked back on the host immediately after, with empty_cache() so the freed
blocks go back to the driver rather than sitting in the caching allocator —
the next caller wants large contiguous blocks for the base's .to(device).
Cost is ~9.5 GiB over PCIe each way once per round, well under a second
against a multi-minute round.
Applied unconditionally rather than gated on full_topology_eval: partial
validators get the same 9.5 GiB back, and a fix that only runs in one
configuration is a fix that only gets tested in one configuration.
Tests: 5, the load-bearing one being that momentum is *relocated* and not
reset. Two optimizers stepped over identical gradients, one parked between
every step, required to end bit-identical in both weights and buffers — a
buggy offload that dropped or zeroed the buffer would pass every
device-placement assertion and silently change how every validator merges.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…offload fix(validator): park outer-optimizer momentum on the host between rounds
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rolls up PRs #204–#215.
The headline: validators now score on the full expert set
Today a validator evaluates a miner using only its partial slice of the model. After this release it evaluates using
all 64 experts per layer, the same topology the reference experiment trains and measures with.
evaluation.full_topology_evalships locked attrue, not as an opt-in. Locked fields are reset to their class default on every start (--auto_update_configdefaults toTrue), which is the only channel thatreaches a running validator's settings. Watchtower replaces the image and never touches an operator's
config.yaml.Two fields are locked with it, because the startup guard
(
check_full_topology_eval_supported) rejects the other combinations:evaluation.full_topology_evaltruemodel.precisionbf16-mixedmoe.partial_moetrueHow it fits in memory
The obvious approach (just set
partial_moe: false) was measured off a cliff twice over. The merge step needs params + gradients + SGD momentum all resident: 87.8 GiB for 15.71 B params, against 44.4 GiB of VRAM. Andper-miner eval deepcopies the base, a second 29.3 GiB.
So instead:
global_modelstays partial (merge, outer optimizer, chain hash, peer sync all unchanged), and a separate frozen full-topology model is used only for scoring. It has no gradients, no optimizer state, and isparked on CPU outside the eval window so the merge phase never shares the GPU with it. Miner shards are grafted into it in place, with the touched rows backed up and restored afterward.
This is the reference experiment's own design translated over — it keeps trainable local experts apart from a frozen full expert set on CPU that swaps in for merged-topology eval.
What operators need
Host RAM + swap ≥ 80 GB. The full base is ~29 GB parked on host, plus a
~38 GB transient while it builds. Validators short on host memory will
crash-loop, not degrade quietly. VRAM ≥ 48 GB (46 GB is not enough).
Everything else
Making the full expert path actually work (#210, #211, #212)
The full-model code path existed but had never been exercised. Three real bugs fell out:
strict=Falseswallowed 4992 unmatched keys.pt, not.safetensorsmoe.partial_moewas declared but read by nothing;full_topkwas2Quantization (#205, #209)
Storage format switched int8 → fp8 (e4m3) to match
experimentrepo exactly, so loss measurements transfer between the two codebases. Scope narrowed to expert projections only (the reference quantizes nothing else; we were also converting 162 backbone Linears, which broke comparability).fp8 is weight storage only — weights are dequantized to the activation dtype for compute. Both formats are one byte per weight, so this buys no memory (measured identical at 8.428 GB) and costs 3.19× fidelity.
Validators refuse fp8 at startup unless
CONNITO_ALLOW_VALIDATOR_FP8is set, and it is now additionally incompatible with full-topology scoring — so on this release, validator quantization is off, period.Dedup filter (#204)
Flags near-duplicate submissions (copy + noise, or a brief finetune) by averaging two miners' models and checking whether the merged loss got worse. Catches what the existing exact-score-tie rule misses: a 1-ULP perturbation on
2 scalars out of 1.77 B is enough to slip past it
(
docs/diagnosis-2026-08-10-duplicate-submissions.md).Three modes:
off(fully inert),shadow(measure and log only),enforce(zeroes both sides of a flagged pair).Should run
shadowfor a cycle or two and set the threshold from that data first.Enabling
shadowcosts ~7 GB more disk (shard retention widens fromtop_k_miners_to_rewardtodedup_top_k) and reserves the eval window's tail (20 min by default).Observer mode (#207)
Lets a second validator run on a live production hotkey for testing while submitting no chain extrinsic, skipping the DHT, and declining HF upload. Requires
CONNITO_VALIDATOR_OBSERVER=1exactly.Notably, an observer now only needs the hotkey's public address, never the private key — so a live validator's private key doesn't have to be copied onto a shared test box. A public-only keyfile can't act under the shared identity
even if the flag is wrong; it fails loudly instead.
Stability fixes found by running this live (#213, #214, #215)
Each of these was a crash observed on the tester validator, not a theoretical concern:
park()was itself allocating.Validated live
Observer-mode validator on an L40S sharing production's hotkey:
baseline_loss=1.8578, miners merged, outer optimizer step completed)val_losswithin 0.05% of production