chore(proof): staging VM wire checklist + e2e probe harness

AGENTS.md

@@ -74,7 +74,7 @@ When verifying a challenge (local-e2e, staging, or focused tests), **simulate a
 4. **Bounty — fail-closed scorer:** the CortexLM/backend public feed is the only scorer. With no readable `BOUNTY_BACKEND_PUBLIC_URL`, `POST /v1/reports` must answer **503** and the emitter must pay **nobody** — it still covers `E` with `NoScore(ChallengeInternal)`, because a paid challenge with no leaves 409s the seal for every challenge. `BOUNTY_FORCE_SIM` is retired — do not reintroduce an offline bounty scorer. See [`docs/BOUNTY.md`](docs/BOUNTY.md).
 5. **Proof — submit:** `POST /v1/submissions` with a `topic_id`. Missing/unknown/not-open → **400** (no row); a custom topic without `artifact_uri` → **400** (no row). Empty `eval_image_digest`, missing/closed/misconfigured RLM judge `InferenceOffer`, missing judge API key, spoofed topic origin, missing/closed/non-`1x` `EvalExecutorOffer` (Lium path), zero open topics, or an unsealed baseline → **503**. Miners submit claim + code + FLOPs + artifact; they do not bind the judge offer or the executor offer. Contamination / empty manifest persist **rejected** without rent; on custom topics the runner's measured `flops_used` over the budget or over the miner's `declared_flops` persists **rejected** after the run, and a report without a measurement is **503** (no row). `GET /v1/proof/topics` must never leak holdout records.
 6. **Proof — executor:** `GET /v1/proof/executor` is always 200 (`ready` + `reason`); `POST /v1/admin/proof/executor` (operator bearer) rotates or closes the live `1x` offer and 400s anything the pin refuses. Harvest rents the offer's `lium_template_id` at exactly `1x` (any other `rent_gpu_count` aborts before the rent) under `max_proof_deadline_s`; a run cut at the deadline is **503 + `stdout_tail`**. `PROOF_HARVEST_*` env only hot-swaps under the pin ceilings. Never a live Lium rent in CI.
-7. **Proof — topic VMs (custom family):** the RLM runs in one Firecracker microVM per `topic_id` on a **dedicated KVM host** (`proof-vm-orchestrator`, HTTPS + bearer **file**), never on the droplet, never on Lium, never nested; every paid run is a **sister** Firecracker guest with **no network**, and the host stamps `sandboxed` / guest-measured `flops_used` on the report. `PROOF_VM_ORCHESTRATOR_URL` unset → `UnwiredVmOrchestrator` (503); URL set but token file missing/empty, `PROOF_RLM_VM_IMAGE_DIGEST` unpinned, agent down, or a `firecracker_required` run without the sister attestation → **503, no row, no host fallback**. `PROOF_VM_RUNNER_CUSTOM_IDS` is the only thing that registers a runner. The custom family is wired from that env alone — live orchestrator selected + ≥1 id → `FamilyMux::custom_only` when no Lium harvest is wired (custom topics score; `nll` / `throughput` → **503**, no row); never stage a placeholder Lium key to open custom topics, and the unwired stub never carries a mux. `/v1/status` keeps the families apart: `live_harvest_wired` is the **Lium harvest only** (never true because a custom mux exists); the custom family is `custom_family_wired` / `registered_custom` / `custom_ready`. Hard `topic_id ↔ VM` bind on both sides (agent 409 `topic_mismatch`). Do not invent an RLM / sister image digest. **Zero live Firecracker in CI** — every test uses the fake hypervisor. Runbook: [`docs/runbooks/proof-vm-orchestrator.md`](docs/runbooks/proof-vm-orchestrator.md).
+7. **Proof — topic VMs (custom family):** the RLM runs in one Firecracker microVM per `topic_id` on a host with a working `/dev/kvm` (`proof-vm-orchestrator`, HTTPS + bearer **file**) — production: dedicated DO metal preferred, never colocated on the CP; staging: colocating the agent on the CP droplet with nested `/dev/kvm` is an allowed exception, proven on `cortex-staging` (nested stays fragile — if the boot fails, provision metal); never on Lium, never emulated; every paid run is a **sister** Firecracker guest with **no network**, and the host stamps `sandboxed` / guest-measured `flops_used` on the report. `PROOF_VM_ORCHESTRATOR_URL` unset → `UnwiredVmOrchestrator` (503); URL set but token file missing/empty, `PROOF_RLM_VM_IMAGE_DIGEST` unpinned, agent down, or a `firecracker_required` run without the sister attestation → **503, no row, no host fallback**. `PROOF_VM_RUNNER_CUSTOM_IDS` is the only thing that registers a runner. The custom family is wired from that env alone — live orchestrator selected + ≥1 id → `FamilyMux::custom_only` when no Lium harvest is wired (custom topics score; `nll` / `throughput` → **503**, no row); never stage a placeholder Lium key to open custom topics, and the unwired stub never carries a mux. `/v1/status` keeps the families apart: `live_harvest_wired` is the **Lium harvest only** (never true because a custom mux exists); the custom family is `custom_family_wired` / `registered_custom` / `custom_ready`. Hard `topic_id ↔ VM` bind on both sides (agent 409 `topic_mismatch`). Do not invent an RLM / sister image digest. **Zero live Firecracker in CI** — every test uses the fake hypervisor. Probe the wire with `GET /v1/admin/proof/vm-orchestrator` (operator bearer, loopback) or `deploy/scripts/proof-vm-wire-check.sh` (`all`, `boot-probe`, `matrix` + `submit-probe --expect 503 --reason …`). Runbook: [`docs/runbooks/proof-vm-orchestrator.md`](docs/runbooks/proof-vm-orchestrator.md) § DigitalOcean staging.
 8. Leaf emission → `POST /v1/weights/raw` → seal → `GET /v1/weights/latest` with **`sealed: true`** (burn fallback alone is not a real seal).
 
 **Never host Sim in staging/prod** for live scoring. `PROOF_FORCE_SIM=1` is CI/local opt-in only (`deploy/scripts/assert-compose-matrix.sh` fails if a droplet overlay sets one). Live Proof rent requires a digest pin in `config/proof-pin.toml` plus miner BYOK (`LIUM_API_KEY` / `X-Lium-Api-Key`). Never log or commit that key. Do not invent `eval_image_digest`.

bins/proof-challenge/Cargo.toml

@@ -13,6 +13,7 @@ name = "proof-challenge"
 path = "src/main.rs"
 
 [dependencies]
+async-trait = "0.1"
 axum = { version = "0.8", default-features = false, features = ["http1", "tokio"] }
 challenge-keys = { path = "../../crates/challenge-keys" }
 clap = { version = "4", features = ["derive", "env"] }
@@ -36,6 +37,7 @@ tracing = "0.1"
 crypto = { path = "../../crates/crypto" }
 hex = "0.4"
 proof-rlm = { path = "../../crates/proof-rlm", features = ["test-fixtures"] }
+proof-vm-agent = { path = "../../crates/proof-vm-agent", features = ["test-fixtures"] }
 reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] }
 sha2 = "0.10"
 tokio = { version = "1", features = ["macros", "rt-multi-thread", "net", "process", "time"] }