Testnet dress-rehearsal tooling: leoma preflight (launch gate) + leoma smoke (outcome verifier) + runbook

docs/TESTNET_RUNBOOK.md

@@ -0,0 +1,145 @@
+# Testnet dress-rehearsal runbook
+
+A repeatable rehearsal that proves the subnet does the right thing **before** mainnet.
+Every step ends in an assertion, not an eyeball — `leoma preflight` gates the launch,
+and `leoma smoke` confirms each scenario was actually exercised and handled correctly.
+
+Run this on testnet with at least one validator + one eval box (a GPU) and a handful of
+miner hotkeys you control.
+
+---
+
+## 0. Prerequisites — pin the consensus surface
+
+The subnet deliberately burns 100% to UID 0 until these are pinned. This is not optional.
+
+1. **Pick the base-model revision** and pin `chain.toml [seed].seed_digest` (the genesis
+   king) to the exact Wan2.1-I2V-14B-480P revision you will run — either a Hippius OCI
+   digest (`sha256:<64hex>`) or a HuggingFace commit SHA (`hf:<40hex>`). `preflight`
+   rejects anything else as unresolvable.
+2. **Build and publish the corpus:**
+   ```bash
+   leoma corpus build-manifest --corpus-id leoma-testnet-v1   # decides windows, hashes truth
+   leoma corpus publish-manifest manifest.json                # prints the digest
+   ```
+   Paste the printed digest into `chain.toml [corpus].manifest_digest`.
+3. **Verify each eval box** decodes the corpus byte-identically to the manifest:
+   ```bash
+   leoma corpus verify --sample 4
+   ```
+   A box that fails this must not duel — its distances would not be reproducible.
+
+## 1. Calibrate `delta_threshold` (the load-bearing measurement)
+
+This is the single largest open consensus risk. On **each GPU type** in the fleet:
+
+```bash
+leoma calibrate generate --gpu <label> -o box-<label>.json
+```
+
+Then, once, compare them all:
+
+```bash
+leoma calibrate analyze box-*.json
+```
+
+- **PASS** → the current `delta_threshold` clears the measured cross-GPU noise floor.
+  Proceed.
+- **FAIL** → `delta_threshold` is *below* the noise floor. Two honest validators can
+  fork. Raise it to the recommended value **or**, if the recommendation is implausibly
+  large, treat it as a signal that LPIPS-on-generated-frames is too noisy for
+  cross-hardware consensus and make a structural decision (a more reproducible metric,
+  or pin the fleet to one GPU class). **Do not launch through a FAIL.**
+
+## 2. Preflight — the launch gate
+
+On the validator box, with `EVAL_SERVER_URL`, `R2_OWN_BUCKET`, `WALLET_NAME`,
+`HOTKEY_NAME` set:
+
+```bash
+leoma preflight
+```
+
+It exits non-zero (and says exactly why) if the seed or corpus is unpinned, the
+consensus surface is invalid, or the eval box is on a different `chain.toml` / scoring
+code than the validator. **Gate your launch script on it:**
+
+```bash
+leoma preflight && leoma serve
+```
+
+Running several eval-server processes (one per GPU pair on an 8×H100 box)? Set
+`EVAL_SERVER_URLS` (comma-separated) instead of the single `EVAL_SERVER_URL` —
+`preflight` checks every configured URL independently and labels each finding by its
+box, so one stale server can't hide behind a healthy sibling.
+
+## 3. Start the services
+
+```bash
+# GPU box(es) — one process per pair of GPUs to duel on, each pinned via
+# LEOMA_KING_DEVICE/LEOMA_CHALLENGER_DEVICE if running more than one on the same host
+leoma servers eval-server           # binds 127.0.0.1; validator reaches it over an SSH tunnel
+
+# validator box — set EVAL_SERVER_URLS to the comma-separated list if running more than one
+leoma serve
+```
+
+## 4. Drive the scenarios
+
+Submit each of these as a miner (`leoma miner push` + `leoma miner commit`) and let the
+validator pick it up. The goal is to exercise every handling path once.
+
+| # | Scenario | How to produce it | Expected outcome |
+|---|----------|-------------------|------------------|
+| A | **Genuine crown** | A model that actually beats the genesis king | Crowned; king chain grows; weights shift |
+| B | **Fair rejection** | A model weaker than the king | Scored, `lcb < delta`, king holds |
+| C | **Broken repo (quarantine)** | Commit a reveal pointing at a non-existent repo | `error` row, `model_not_found`; quarantined after 2 sightings; **later challengers still run** |
+| D | **Wrong architecture** | A model whose `transformer/config.json` shape differs from the base | Rejected pre-dispatch (`arch_mismatch`) in ~seconds, no GPU spent |
+| E | **Copy of the king** | Re-upload the king's weights under a new hotkey (change only the README) | Rejected pre-duel (`copy_of_king`); no multi-hour duel |
+| F | **Freeze cheat** | A model that emits the conditioning frame repeated | Rejected by the freeze gate (`FROZE OUT`) even if it beat a weak king |
+
+## 5. Smoke — assert the outcomes
+
+Point `smoke` at the validator's published `dashboard.json`:
+
+```bash
+leoma smoke https://<your-state-bucket>/dashboard.json
+```
+
+It reports which rehearsal scenarios have been observed and exits non-zero until every
+one has. Re-run it as you drive more scenarios; a clean run means:
+
+- ✓ a challenger beat the king and was crowned
+- ✓ a challenger was scored and lost fairly
+- ✓ a broken model was recorded as an error (not silently dropped)
+- ✓ a copy of the king was rejected pre-duel
+- ✓ a freeze cheat was rejected by the gate
+- ✓ the validator is dueling, not degraded
+
+Check the **dashboard** during a real duel too — the "in the arena" panel should show
+the live challenger-vs-king, and the quality-over-reigns chart should show models
+sitting well below the dashed freeze cheat floor.
+
+## 6. Resilience spot-checks (optional but recommended)
+
+- **Restart the validator mid-duel.** It should re-attach to the in-flight slot and
+  settle the same duel, not orphan it.
+- **Kill the eval box mid-duel.** The validator should classify it transient and retry;
+  the box should come back with a fresh CUDA context.
+- **Point one eval box at a stale `chain.toml`.** `preflight` and the validator's
+  dispatch preflight should both refuse it (`consensus_mismatch`). With a single
+  configured eval server the dashboard should show the `degraded` reason; with several
+  configured (`EVAL_SERVER_URLS`), the stale one should simply be skipped in favor of a
+  healthy sibling — the validator should **not** show `degraded` as long as at least one
+  configured box is healthy.
+
+---
+
+## Definition of done
+
+- `leoma calibrate analyze` returns **PASS** for the fleet's `delta_threshold`.
+- `leoma preflight` exits 0 on every validator.
+- `leoma smoke` reports **all** scenarios observed.
+- The dashboard shows a live duel and the cheat-floor chart.
+
+Only then is the subnet ready for mainnet.

leoma/app/preflight.py

@@ -0,0 +1,242 @@
+"""Is this validator actually ready to launch? A hard gate, not a hope.
+
+The subnet has several pins that must be set before it does anything useful, and each
+one fails *safe but silent*: an unpinned seed digest, an unpinned corpus, or an eval
+box on a stale config all make the validator burn to UID 0 rather than crown. That is
+the correct behavior, but an operator who flips the switch and walks away would see a
+dead subnet with no single place that says *why*.
+
+``run_preflight`` is that single place. It runs every readiness check, classifies each
+as pass / warn / fail, and returns an overall verdict. ``fail`` means "the validator
+will not crown anyone until you fix this"; ``warn`` means "this will work but you
+probably didn't mean it" (e.g. no eval server configured to check against).
+
+The checks are **pure functions of already-fetched inputs** — the CLI does the I/O
+(HTTP to the eval box, a HEAD on the corpus) and hands the results here — so the
+decision logic is unit-testable with no network, GPU, or chain.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import NamedTuple, Optional
+
+from leoma.infra.model_store import DIGEST_RE
+
+PASS = "pass"
+WARN = "warn"
+FAIL = "fail"
+
+
+@dataclass(frozen=True)
+class CheckResult:
+    name: str
+    status: str
+    detail: str
+
+    @property
+    def ok(self) -> bool:
+        return self.status != FAIL
+
+
+@dataclass(frozen=True)
+class PreflightReport:
+    checks: tuple[CheckResult, ...]
+
+    @property
+    def ready(self) -> bool:
+        """True when nothing is FAIL. Warnings do not block launch."""
+        return all(c.ok for c in self.checks)
+
+    @property
+    def failures(self) -> tuple[CheckResult, ...]:
+        return tuple(c for c in self.checks if c.status == FAIL)
+
+    @property
+    def warnings(self) -> tuple[CheckResult, ...]:
+        return tuple(c for c in self.checks if c.status == WARN)
+
+
+def check_seed(seed_digest: str) -> CheckResult:
+    digest = (seed_digest or "").strip()
+    if not digest:
+        return CheckResult(
+            "seed_digest", FAIL,
+            "chain.toml [seed].seed_digest is empty — with no genesis king the subnet burns "
+            "100% to UID 0. Pin the exact base-model revision before launch.",
+        )
+    if not DIGEST_RE.match(digest):
+        return CheckResult(
+            "seed_digest", FAIL,
+            f"chain.toml [seed].seed_digest ({digest[:23]}…) is not a recognized digest — "
+            "expected 'sha256:<64hex>' (Hippius) or 'hf:<40hex>' (HuggingFace commit SHA). "
+            "A malformed pin means the genesis king can never resolve.",
+        )
+    return CheckResult("seed_digest", PASS, f"genesis king pinned ({digest[:23]}…)")
+
+
+def check_corpus_pin(corpus_pinned: bool, manifest_digest: str) -> CheckResult:
+    if corpus_pinned:
+        return CheckResult("corpus_pin", PASS, f"corpus manifest pinned ({manifest_digest[:23]}…)")
+    return CheckResult(
+        "corpus_pin", FAIL,
+        "chain.toml [corpus].manifest_digest is empty — the duel exam is not reproducible "
+        "and the validator refuses to duel. Publish a manifest (`leoma corpus publish-manifest`) "
+        "and pin its digest.",
+    )
+
+
+def check_consensus_digest(consensus_digest: str) -> CheckResult:
+    # If chain_config imported at all, SPEC validated and this exists — so this is really
+    # a "surface the digest so the operator can compare it across boxes" check.
+    if consensus_digest and consensus_digest.startswith("sha256:"):
+        return CheckResult("consensus_digest", PASS, consensus_digest)
+    return CheckResult("consensus_digest", FAIL, "consensus surface did not produce a digest")
+
+
+def check_corpus_reachable(fetched_digest: Optional[str], pinned_digest: str, error: Optional[str]) -> CheckResult:
+    """Given the digest of the manifest actually fetched from the bucket, does it match?"""
+    if error:
+        return CheckResult("corpus_fetch", WARN, f"could not fetch the corpus manifest to verify it: {error}")
+    if not fetched_digest:
+        return CheckResult("corpus_fetch", WARN, "corpus manifest not checked (no bucket credentials)")
+    if fetched_digest == pinned_digest:
+        return CheckResult("corpus_fetch", PASS, "published manifest matches the pinned digest")
+    return CheckResult(
+        "corpus_fetch", FAIL,
+        f"the bucket's manifest ({fetched_digest[:19]}…) does NOT match the pinned digest "
+        f"({pinned_digest[:19]}…). Republish, or fix the pin.",
+    )
+
+
+def check_eval_server(
+    health: Optional[dict],
+    our_consensus_digest: str,
+    our_eval_code_digest: str,
+    error: Optional[str] = None,
+    *,
+    name: str = "eval_server",
+) -> CheckResult:
+    """Given one eval box's /health, do its consensus + code digests match ours?
+
+    ``name`` lets the caller disambiguate several servers (e.g. ``eval_server[url]``)
+    when checking a whole ``EVAL_SERVER_URLS`` fleet instead of a single box.
+    """
+    if error:
+        return CheckResult(name, WARN, f"eval server not reachable ({error}); skipped")
+    if health is None:
+        return CheckResult(name, WARN, "no eval server configured to check against (set EVAL_SERVER_URL(S))")
+
+    theirs_consensus = health.get("consensus_digest")
+    theirs_code = health.get("eval_code_digest")
+    if theirs_consensus != our_consensus_digest:
+        return CheckResult(
+            name, FAIL,
+            f"eval box pins a DIFFERENT consensus surface (box {str(theirs_consensus)[:19]}…, "
+            f"validator {our_consensus_digest[:19]}…) — one of you is on a stale chain.toml.",
+        )
+    if theirs_code is None:
+        # A current box always reports this field (see eval_server.py's /health). A box
+        # missing it entirely is running a build old enough to predate the field — we
+        # have literally no evidence its scoring code matches, so this must not read as
+        # a silent PASS.
+        return CheckResult(
+            name, WARN,
+            "eval box's /health did not report eval_code_digest (stale build?) — its "
+            "scoring code could not be verified against this validator's.",
+        )
+    if theirs_code != our_eval_code_digest:
+        return CheckResult(
+            name, FAIL,
+            f"eval box runs DIFFERENT scoring code (box {str(theirs_code)[:19]}…, "
+            f"validator {our_eval_code_digest[:19]}…) — its distances would not be reproducible.",
+        )
+    return CheckResult(name, PASS, "eval box matches this validator's consensus surface + code")
+
+
+class EvalServerProbe(NamedTuple):
+    """One configured server's raw /health result, ready for ``check_eval_servers``."""
+    url: str
+    health: Optional[dict]
+    error: Optional[str] = None
+
+
+def check_eval_servers(
+    probes: tuple[EvalServerProbe, ...],
+    our_consensus_digest: str,
+    our_eval_code_digest: str,
+) -> tuple[CheckResult, ...]:
+    """One :func:`check_eval_server` result per configured ``EVAL_SERVER_URLS`` entry.
+
+    A single-server validator gets exactly one ``eval_server`` check, unchanged. A
+    multi-server validator gets one check per URL — silently checking only the first
+    configured server (or none at all) would leave the rest of the fleet unverified.
+    """
+    if not probes:
+        return (check_eval_server(None, our_consensus_digest, our_eval_code_digest),)
+    if len(probes) == 1:
+        p = probes[0]
+        return (check_eval_server(p.health, our_consensus_digest, our_eval_code_digest, p.error),)
+    return tuple(
+        check_eval_server(
+            p.health, our_consensus_digest, our_eval_code_digest, p.error,
+            name=f"eval_server[{p.url}]",
+        )
+        for p in probes
+    )
+
+
+def check_state_bucket(own_bucket: Optional[str]) -> CheckResult:
+    if own_bucket and own_bucket.strip():
+        return CheckResult("state_bucket", PASS, f"king state persists to {own_bucket}")
+    return CheckResult(
+        "state_bucket", FAIL,
+        "R2_OWN_BUCKET is not set — the validator cannot persist king state and will refuse to run.",
+    )
+
+
+def check_wallet(wallet_name: Optional[str], hotkey_name: Optional[str]) -> CheckResult:
+    if wallet_name and hotkey_name:
+        return CheckResult("wallet", PASS, f"{wallet_name}/{hotkey_name}")
+    return CheckResult("wallet", WARN, "wallet/hotkey not both set (using defaults)")
+
+
+def run_preflight(
+    *,
+    seed_digest: str,
+    corpus_pinned: bool,
+    manifest_digest: str,
+    consensus_digest: str,
+    eval_code_digest: str,
+    own_bucket: Optional[str],
+    wallet_name: Optional[str],
+    hotkey_name: Optional[str],
+    corpus_fetched_digest: Optional[str] = None,
+    corpus_error: Optional[str] = None,
+    eval_servers: tuple[EvalServerProbe, ...] = (),
+) -> PreflightReport:
+    """Assemble every readiness check into one verdict. Pure — the caller does the I/O.
+
+    ``eval_servers`` is one probe per configured ``EVAL_SERVER_URLS`` entry (empty when
+    none are configured) — a single-server validator still gets exactly one
+    ``eval_server`` check; a multi-server one gets one per URL, so a stale box can't
+    hide behind a healthy sibling.
+    """
+    checks = [
+        check_seed(seed_digest),
+        check_corpus_pin(corpus_pinned, manifest_digest),
+        check_consensus_digest(consensus_digest),
+        check_corpus_reachable(corpus_fetched_digest, manifest_digest, corpus_error),
+        *check_eval_servers(eval_servers, consensus_digest, eval_code_digest),
+        check_state_bucket(own_bucket),
+        check_wallet(wallet_name, hotkey_name),
+    ]
+    return PreflightReport(tuple(checks))
+
+
+__all__ = [
+    "PASS", "WARN", "FAIL",
+    "CheckResult", "PreflightReport", "EvalServerProbe", "run_preflight",
+    "check_seed", "check_corpus_pin", "check_consensus_digest",
+    "check_corpus_reachable", "check_eval_server", "check_eval_servers",
+    "check_state_bucket", "check_wallet",
+]