feat(proof): one Firecracker experiment VM per paid job with a generic in-guest runner

[echobt]

Summary

Architecte RELOCK C, generalist form: topics whose signed constraints.params select an in-guest runner get one dedicated Firecracker microVM per Baseline / Evaluate job — created for the job, sized by the topic's ask under a hard operator lock, fed the topic-pinned experiment pack over vsock, run by an operator adaptor baked into the guest image, attested by the host as experiment_vm, destroyed afterwards — and scored only once the orchestrator confirms that VM destroyed. Parallel experiments are parallel VMs, never containers sharing one VM. Path B (vsock → host harness) is not implemented. Rootless podman inside the guest is the accepted C (not Docker-in-VM).

Nothing benchmark-, dataset-, harness-, adaptor-, or pack-specific is compiled in or committed. The words that select the path are generic knobs whose values are topic data:

constraints.params	role
baseline_runner (synonym in_guest_benchmark_runner)	operator adaptor id under /opt/proof/runners/<id>/ (e.g. operator_adaptor_v0) — the adaptor itself lives outside git
experiment_pack_digest (+ optional experiment_pack_path)	sha256: of the pack tar the KVM host stages; required with a runner, never defaulted
experiment_vcpus / experiment_mem_mib / experiment_disk_mib	the topic's ask, held under the caps (over = 503, never clamped)
model_pin, every other param	reach the adaptor as PROOF_MODEL_PIN / PROOF_PARAM_<KEY> (two signed names that collide after upper-casing / -→_ are refused before anything runs)

Resource caps (Mathis LOCK — hard):

• per-experiment VM 16 vCPU / 32 GiB RAM — DEFAULT_EXPERIMENT_VCPUS=16, DEFAULT_EXPERIMENT_MEM_MIB=32768 (what a silent topic gets), DEFAULT_MAX_EXPERIMENT_VCPUS=16, DEFAULT_MAX_EXPERIMENT_MEM_MIB=32768 (the most a topic may ask for), and LOCK_MAX_EXPERIMENT_VCPUS=16 / LOCK_MAX_EXPERIMENT_MEM_MIB=32768 as compiled-in maxima: ExperimentCeilings::validate refuses any CP (PROOF_EXPERIMENT_VM_MAX_*) or host (PROOF_VM_AGENT_EXPERIMENT_MAX_*) ceiling above them so the process does not boot, and shape() / admit() refuse any VM shape above them whatever the ceilings say. A smaller host may only lower a ceiling — and lowering only the MAX knob is enough: an unset default follows it (a default set above its ceiling is refused, never clamped);
• writable disk 32 GiB default (PROOF_EXPERIMENT_VM_DISK_MIB), 16 GiB floor, max raised by the operator when the metal has more (the disk ceiling is not locked);
• PROOF_VM_AGENT_MAX_EXPERIMENT_VMS (default 2) caps how many run at once; the topic VM keeps its locked 4 / 8192;
• rootfs bake target 1.5–2.5 GiB (budget check in the bake).

What changed

• crates/proof-experiment (new): the binding parser (ExperimentBinding::from_params, fail-closed on a half selection), ExperimentCeilings (defaults + maxima + hard lock + disk floor) / VmShape::locked / ExperimentPolicy::from_env (an unset default follows a lowered ceiling; a set default above its ceiling is refused), wire ExperimentSpec.
• proof-vm-proto: HostToRlm::StagePack / RlmToHost::PackStaged (≤ 160 MiB), GuestMode { sister | experiment_vm } on the attestation (serde default = sister, wire-compatible), VmRecord.experiment, health experiment_vms / max_experiment_vms, ErrorCode::Capacity (503).
• proof-rlm: TopicVmSpec.experiment, run_paid_job (create → run → destroy → return the outcome only on a confirmed destroy; Ok(false) or a teardown error is VmError::TeardownUnconfirmed — 503, no row, no baseline — even after a successful run, naming the VM), VmBackedRunner::with_experiments; TopicSetup::baseline routes through it (the baseline is the first experiment VM). Fake orchestrator keeps experiment VMs out of attach, records teardowns, and can inject unconfirmed / failed teardowns.
• proof-vm-agent: experiment VMs bypass the one-per-topic rule, never answer attach, count against the host capacity, ids <topic>-x<n>; the fake hypervisor attests experiment_vm.
• crates/proof-fc-experiment (new, host layer over FirecrackerHypervisor): ceilings (under the lock) → pack resolved under PROOF_VM_AGENT_EXPERIMENT_PACK_DIR and re-hashed with verify_artifact before any jail → boot with disk_mib scratch → StagePack over vsock (destroy on failure) → job bound to the VM's runner + pack → experiment_vm attestation for that VM and job. proof-fc-host changes are 3 lines (scratch size from the spec, attestation mode).
• proof-vm-fc / bins/proof-challenge: policy from env, an experiment attestation must name the VM the job was dispatched to; orchestrator binary gains the --experiment-* / --max-experiment-vms flags (a vCPU / memory ceiling above the lock refuses to boot) and wraps the backend.
• crates/proof-vm-guest + bins/proof-vm-guest-agent (new): the generic guest agent (vsock :5000 via tokio-vsock, or --stdio). Hello, tmpfs secrets (chowned to the run-as user), pack verify + unpack, artefact fetch streamed under a hard 64 MiB cap (cut mid-body, never buffered first), adaptor exec contract (run / inspect / propose_rules, documented PROOF_* env, report.json / checklist.json / rules.json), param-name collision refusal before exec, stdout / stderr drained concurrently into rolling bounded tails while the process runs, deadline kill of the process group, secret redaction in every tail and evidence string. No default value ever: no runner / adaptor / pack / verifying artefact / report / finite value → RlmToHost::Failed → 502 → 503, no row. ProposeRules without an adaptor entrypoint echoes the signed topic's own checklist.
• deploy/guest/: bake-rootfs.sh (Debian minbase + rootless podman/crun/fuse-overlayfs/pasta + catatonit; the podman store on run-as-owned paths init.sh creates — graphroot = /var/lib/proof/containers/storage on the scratch drive, runroot = /run/user/<uid>/containers — never the root-owned /var/lib/containers / /run/containers; generic operator hooks --extra-pkgs / --overlay / --chroot-hook for harness tooling; --check-kernel-config; tree budget 2560 MiB / image 3072 MiB; image named after its own sha256sum; RE-LOCK steps printed; --dry-run without root), init.sh / agent-loop.sh (PID 1, scratch on /dev/vdb, cgroup v2, run-as user, both podman paths pre-created and chowned), runners/README.md (adaptor contract + a fail-closed skeleton + honest rootless-podman limits). No adaptor ships in git — the harness CLI, agent, task format, and trial → primary_value rule are operator artefacts baked outside the repo and selected by signed params; a test greps deploy/guest/ for harness names.
• Docs: docs/runbooks/proof-experiment-vms.md (params, caps incl. the hard lock, flow incl. confirmed destroy, bake + size budget, RE-LOCK on metal, fail-closed rows incl. the new ones, limitations); pointers in the orchestrator runbook, PROOF.md, root / deploy / docs AGENTS.md, ARCHITECTURE.md, COMPLETENESS.md, miner guide; env examples; wire check fails a ceiling knob above the lock, validates integers, the 16 GiB disk floor, the image pin.

Greptile findings addressed (head 85e26c69)

Finding	Fix
P1 Require confirmed VM destruction / P1 Propagate teardown failures (proof-rlm/src/vm.rs)	run_paid_job returns the outcome only on teardown → Ok(true); Ok(false) / Err → VmError::TeardownUnconfirmed (503, no row, no baseline), naming the VM and any job failure too. Tests in proof-rlm (evaluate + baseline, injected Ok(false) and Err) and proof-rlm-scorer (setup with an unconfirmed destroy seals no baseline).
P1 Writable Podman paths (bake-rootfs.sh)	runroot = /run/user/<uid>/containers, graphroot = rootless_storage_path = /var/lib/proof/containers/storage; init.sh creates and chowns both before the agent starts. Verified in a private mount namespace: uid 1000 can write both, cannot create the old paths. Test asserts the bake's paths are what init makes writable.
P1 Stream artifact size checks (fetch.rs)	download_capped reads resp.chunk() incrementally and aborts the moment the cap would be exceeded; honest Content-Length over the cap refused before the body. Test: endless chunked body without Content-Length is cut promptly; bodies at / under the cap arrive whole.
P1 Enforce locked VM maxima (proof-experiment)	LOCK_MAX_EXPERIMENT_* compiled in; validate rejects max_vcpus > 16 / max_mem_mib > 32768 (AboveLock), shape / admit reject any such shape; orchestrator refuses --experiment-max-vcpus 32 etc.; wire check fails such knobs. Tests on both sides.
P1 Remove the Harbor harness / P1 Do not substitute rewards (runners/harbor-podman/*)	Both files deleted; --with-harbor / --harbor-version replaced by generic --extra-pkgs / --overlay / --chroot-hook; README ships the contract + a fail-closed skeleton only and states a trial without a measurement is never scored from another value. Test greps deploy/guest/ for harness names.
P1 Reject parameter-name collisions (runner.rs)	param_env refuses two signed names mapping to one PROOF_PARAM_* (names both + the variable) before run / inspect / propose_rules exec. Test: control (foo-bar alone maps) + collision (adaptor never runs) on all three paths.
P1 Clamp unset experiment defaults (proof-experiment, second review)	ExperimentPolicy::from_env reads the MAX knobs first; an unset default follows a lowered ceiling (host parity), a default set above its ceiling is still refused, the lock still holds. Pure from_lookup + test over the vCPU-only / memory-only / disk-only configurations.
P2 Bound output while draining (runner.rs)	stdout / stderr drained concurrently into rolling Tails of STREAM_TAIL_BYTES each while the process runs (≤ MAX_TAIL_BYTES total, marker for what was cut). Test: 8 MiB floods on both streams complete under the deadline with a bounded, marked tail.

Honest limits (also in the runbook)

• Rootless podman, not Docker-in-VM: Firecracker guests have no nested KVM; containers are namespaces run by an unprivileged user. Harness behaviour under podman's API socket / pasta must be smoked on the baked image before signing a topic on it.
• The guest kernel needs CONFIG_USER_NS, overlayfs, fuse, veth, tun, cgroups, seccomp… — a stock microVM config lacks several (--check-kernel-config names them).
• flops_used on this path is guest-agent-authored (from report.json, relayed by the host); an agentic harness has no FLOP counter, so the topic sets flops_budget: 0 or supplies an accounting param — never invented.
• Inspection needs an adaptor inspect entrypoint (topic-RLM work); without one such a topic cannot reach Evaluate.
• An unconfirmed destroy leaves the VM for the operator to reconcile on the KVM host; the CP withholds the result rather than retrying blindly.
• Host sizing: one experiment VM at the lock (16 / 32 / 32 GiB disk) already exceeds a g-8vcpu-32gb beside the 4/8 topic VM — the dedicated production host must carry it; the colocated staging droplet lowers the caps on both sides (below the lock is allowed) or keeps the count at 1 with a smaller topic ask.
• Nothing has been baked or booted from this change. Everything is tested against the fake hypervisor, a recording stager, shell-script adaptors, loopback HTTP, a mount-namespace probe of init.sh, and the bake's dry run. No digest is invented anywhere; the pack digest and runner id are topic data staged outside git.

Greptile

• Greptile has reviewed this PR; findings are fixed or answered
• If the bot was silent, I commented @greptileai review

Test plan

• cargo test --workspace (all green; new: proof-rlm withheld-unless-destroyed, proof-rlm-scorer baseline withheld on unconfirmed destroy, proof-experiment hard lock + lowered-ceiling-only policy, proof-vm-orchestrator above-lock flags refused, proof-fc-experiment above-lock shape refused, proof-vm-guest param collision / output flood / streamed fetch cap / bake hooks / podman paths / no-harness grep)
• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings, cargo deny check, xtask loc-cap / consensus-lint / spec-check / design-check / external-docs-check, dcap clippy
• Metal: bake, stage image + pack on the KVM host, RE-LOCK per the runbook, run § Verify (not possible from this change; recorded as unknown)

Risk

Additive wire changes (serde(default) everywhere; an old agent given an experiment spec boots a topic VM, stages no pack, attests nothing → the CP refuses the run, 503). No emission, signature-domain, or miner CVM measurement impact. New host env knobs default to the 16 / 32 lock and 2 concurrent experiment VMs and refuse to boot above the lock; size the KVM host for max_experiment_vms × 16 vCPU / 32 GiB / 32 GiB disk on top of the 4/8 topic VM. A host that fails to destroy an experiment VM now surfaces as 503s on that topic (with the VM named) instead of silently leaking capacity.

Naming

I did not rename BASE_* environment variables, deployed host paths (/opt/base, /run/base, …), GHCR baseintelligence/base package names, or base-*-v1 cryptographic domain tags.

  

[echobt]

@greptileai please re-review after Mathis 16vCPU/32GiB caps bump (40bd4bb).

[greptile-apps]

Greptile Summary

Summary

• Unset experiment VM CPU, memory, and disk defaults now follow configured ceilings, allowing smaller hosts to start without redundant default settings.
• Explicit defaults that exceed their configured ceilings remain rejected, while the hard CPU and memory limits remain enforced.

Merge Safety

Safe to merge. No blocking issues remain.

Confidence Score: 5/5

Safe to merge; no outstanding blocking issues remain.

No outstanding findings remain.

_{Reviews (4): Last reviewed commit: "fix(proof-experiment): an unset default ..." | Re-trigger Greptile}

[echobt]

@greptileai please re-review after Greptile P1 fixes (ec1defa).

[echobt]

@greptileai