chore(proof): staging VM wire checklist + e2e probe harness - #247
Conversation
…kvm works Architecte nit on #247: `cortex-staging` (nested DigitalOcean virtualisation) did boot Firecracker and the § 4 fail-closed matrix came back green, so the runbook must not claim a Droplet can never be the KVM host. Corrected wording everywhere the PR (and the #244 lines next to it) said so: - staging may colocate the agent on the control-plane droplet when /dev/kvm works — validated on cortex-staging; - production prefers a dedicated DO bare-metal / KVM host; - nested KVM remains fragile — if the boot fails or /dev/kvm goes away, provision metal, do not patch around it (fragility signs listed); - never Lium, never an emulator, never without /dev/kvm. Runbook § DigitalOcean staging gains the colocated specifics: the CP container reaches the agent on the droplet's VPC address (loopback in the container is the container; FcConfig accepts plain http on loopback only), agent bound on that address with TLS + SAN, :8200 firewalled to the compose network, sizing on top of the compose stack, a local `install` of the bearer copy; sign-off records the placement. The fail-closed probes, the matrix, and the harness are unchanged. Same correction in AGENTS.md item 7, deploy/AGENTS.md, both staging overlays, the generic env examples, the systemd unit comment, docs/PROOF.md, docs/ARCHITECTURE.md, docs/COMPLETENESS.md (staging boot + § 4 green recorded; § 5 / § 6 still to be recorded), and the agent / binary module docs. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Greptile SummaryAdds an operator-authenticated VM-orchestrator readiness probe, staging wire-check and end-to-end probe harness, deployment overlays, and runbook guidance. The follow-up change rejects zero, negative, and non-numeric Confidence Score: 5/5Safe to merge. No outstanding blocking findings remain. The production-host guard now normalizes parsed hostnames before protected-host checks; the boot probe records its topic before creation and reconciles an ambiguous create response; live probes default to the topic FLOPs budget; and explicit FLOPs declarations reject zero, negative, and non-numeric values before any submission request. Reviews (3): Last reviewed commit: "fix(deploy): wire-check refuses a zero d..." | Re-trigger Greptile |
…ated do metal Architecte GO conditionnel on #247: one wording across every placement statement (runbook, AGENTS.md item 7, deploy/AGENTS.md, both staging overlays, the generic env examples, the systemd unit comment, docs/PROOF.md, docs/ARCHITECTURE.md, docs/COMPLETENESS.md, agent and binary module docs): - staging: colocating the agent on the CP droplet with nested /dev/kvm is an allowed exception, proven on cortex-staging (§ 4 green); nested stays fragile — if the boot fails, provision metal; - production: dedicated DO metal preferred — never colocated on the CP. No "a Droplet is not one / no nested" claim remains. Fail-closed probes, matrix, and harness unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com>
…oot-probe, budget flops Greptile on #247: P1 production guard bypass: refuse_prod parsed nothing and matched the URL case-sensitively, so https://NETWORK.CORTEX.FOUNDATION reached authenticated create / submit requests. The guard now parses the host (scheme, userinfo, port, path, query, trailing dot stripped; IPv6 kept bracketed), lower-cases it, and refuses a protected host or any subdomain of one, plus a case-insensitive whole-URL match as belt and braces. It runs at env load, before any other check, on every probe. P2 lost response strands VM: PROBE_TOPIC_LIVE is set before the create goes out; a create whose answer is 000 / 5xx / unparseable is reconciled through GET /v1/vms/by-topic and any VM the agent reports is destroyed in line (probe_reconcile_destroy), and the EXIT trap does the same for anything still in flight (Ctrl-C, unconfirmed teardown). Topic state is cleared only after a confirmed destroy followed by a 404, so a retry on the probe topic is never blocked. PROOF_VM_WIRE_CHECK_FAULT= lose-create-answer is a test-only hook that drops the create's answer. P1 live probe under-declares FLOPs: submit-probe declared 1, so a live run measuring more was a flops_under_declared reject. Fail-closed probes still send 1 (nothing runs); --expect 2xx declares the topic's flops_budget read from GET /v1/proof/topics/<id> (--declared-flops N overrides; unreadable budget = FAIL naming the flag). Tests (fake agent): uppercase / userinfo / trailing-dot / subdomain production origins exit 2 before any request, for submit-probe and for the agent URL at env load; the lost-answer create boots one VM that is found by topic and destroyed, the topic is free, a retry passes. Runbook § 3 / § 5 updated. Co-authored-by: Mathis <echobt@users.noreply.github.com>
GET /v1/admin/proof/vm-orchestrator (operator bearer, read-only, no VM, no spend) reports what proof-challenge resolved for the topic-VM orchestrator and whether the KVM-host agent answers, through the very client the runner drives: `ready()` (bearer file present, RLM image pinned — re-read now), the locked template, one agent health call, and the host's own gates (live_harvest_wired, registered_custom). A broken wire is data, not an error: bearer refused, agent unreachable, digest unpinned, and URL unset each show up by name so a staging operator can prove the wire without cargo on the droplet. Never the bearer value. proof-challenge resolves the orchestrator once per process and shares the Arc between the runner registry and the probe; the registry is still built only over a wired harvest, so log lines are unchanged. Tests: route is 401 / 503 auth_unconfigured / 200; unwired hosts name the env vars; against the in-process fake agent the report shows ready + fake hypervisor, then a bearer rotated on one side, a stopped agent, and an emptied bearer file, each as data. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Operator harness for the Proof topic-VM wire on the staging master
droplet: bash + curl + python3, no cargo, bearer only ever in a 0600
curl config, production hosts refused.
env CP env file: https URL (loopback http = WARN), bearer
file non-empty via --path-map (compose bind mount), RLM
image pin is sha256:<64 hex> (empty = FAIL, never
invented), optional CA is PEM, custom ids well-formed,
locked 4/8192 shape, PROOF_FORCE_SIM off
agent GET /v1/health with the bearer (ready / reason /
hypervisor / vms); no bearer and wrong bearer → 401
cp /v1/status gates (lium, live_harvest_wired,
registered_custom ⊇ ids), no URL / token / path leak,
/v1/proof/topics holdout leak, executor readiness, and
the admin vm-orchestrator probe (the CP's own rustls
client: ready, agent health, agent_error)
boot-probe create → attach → 409 → 409 topic_mismatch → destroy →
404 for ONE RLM VM (no job, no spend); Ctrl-C tears
the VM down
submit-probe POST /v1/submissions on a custom topic asserting the
fail-closed code + reason; 2xx refused without
--allow-live-run; a scored row must carry the
sister-measured flops_used
matrix the fail-closed flips with the expected 503 reasons
Integration test (proof-vm-fc, fake agent on loopback, skipped without
bash/curl/python3): env / agent / boot-probe speak the router's JSON,
the bearer never appears in the output, one boot + one Destroy, a dead
agent fails the check; unpinned digest and an emptied bearer file fail
closed.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
`orchestrator unreachable (...)` travels into the miner-facing 503 body when the KVM-host agent is down; reqwest's Display would print the agent's URL with it. Keep the method + route, strip the URL (`Error::without_url`). Test asserts the route stays and the address does not. Co-authored-by: Mathis <echobt@users.noreply.github.com>
deploy/env/proof-challenge.staging-vm.example: the client-side keys to append to proof-challenge.env on the staging master (https agent URL on the VPC, bearer file, private CA, RLM image pin, locked 4/8192 shape, custom ids, admin tokens file). deploy/env/proof-vm-orchestrator.staging.example: /etc/proof-vm/ orchestrator.env for the dedicated KVM host serving staging (private bind + TLS, bearer file, kernel + sister pins, sizes, egress allowlist for the judge origin / artefact hosts / resolver). Placeholders only, and every REPLACE_WITH_* value fails closed as written: the wire check flags each one, proof-challenge stays unwired or answers 503, the agent refuses to boot. No digest is real. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
…e + probes New § DigitalOcean staging: the CP stays on the existing staging master droplet; the agent needs a KVM host with /dev/kvm — a DO Droplet is not one (no nested virtualisation; the unit's ConditionPathExists refuses by design; no nested-FC workaround), so use DO bare-metal / dedicated hardware or any bare-metal KVM host on the staging VPC / private network, private bind, TLS + bearer. Host pre-checks, per-topic budget, CP preconditions (lium, live_harvest_wired, judge, executor, a signed custom topic with a sealed baseline), KVM-host and CP steps with the staging env overlays, the wire check (env / agent / cp / boot-probe), the fail-closed matrix (flip → restart? → expected 503 reason → what the admin probe shows), the happy path with the exact log lines and row / artefact evidence (sister sandboxed + flops_used, evidence bind, destroy teardown), a sign-off checklist that records unknown / not run rather than a green box, and rollback. § Wire the control plane documents GET /v1/admin/proof/vm-orchestrator (field → root cause table; loopback only — staging's public API is cleartext). Operate table gains "Is the wire up?". Co-authored-by: Mathis <echobt@users.noreply.github.com>
AGENTS.md verification item 7, deploy/AGENTS.md (staging wire section), docs/AGENTS.md runbook index, docs/PROOF.md HTTP surface (the admin vm-orchestrator probe), deploy/secrets/README.md (admin_tokens now also gates the probe), docs/COMPLETENESS.md (probe + harness + overlays; still not on any host, sign-off unfilled). Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
…kvm works Architecte nit on #247: `cortex-staging` (nested DigitalOcean virtualisation) did boot Firecracker and the § 4 fail-closed matrix came back green, so the runbook must not claim a Droplet can never be the KVM host. Corrected wording everywhere the PR (and the #244 lines next to it) said so: - staging may colocate the agent on the control-plane droplet when /dev/kvm works — validated on cortex-staging; - production prefers a dedicated DO bare-metal / KVM host; - nested KVM remains fragile — if the boot fails or /dev/kvm goes away, provision metal, do not patch around it (fragility signs listed); - never Lium, never an emulator, never without /dev/kvm. Runbook § DigitalOcean staging gains the colocated specifics: the CP container reaches the agent on the droplet's VPC address (loopback in the container is the container; FcConfig accepts plain http on loopback only), agent bound on that address with TLS + SAN, :8200 firewalled to the compose network, sizing on top of the compose stack, a local `install` of the bearer copy; sign-off records the placement. The fail-closed probes, the matrix, and the harness are unchanged. Same correction in AGENTS.md item 7, deploy/AGENTS.md, both staging overlays, the generic env examples, the systemd unit comment, docs/PROOF.md, docs/ARCHITECTURE.md, docs/COMPLETENESS.md (staging boot + § 4 green recorded; § 5 / § 6 still to be recorded), and the agent / binary module docs. Co-authored-by: Mathis <echobt@users.noreply.github.com>
…ated do metal Architecte GO conditionnel on #247: one wording across every placement statement (runbook, AGENTS.md item 7, deploy/AGENTS.md, both staging overlays, the generic env examples, the systemd unit comment, docs/PROOF.md, docs/ARCHITECTURE.md, docs/COMPLETENESS.md, agent and binary module docs): - staging: colocating the agent on the CP droplet with nested /dev/kvm is an allowed exception, proven on cortex-staging (§ 4 green); nested stays fragile — if the boot fails, provision metal; - production: dedicated DO metal preferred — never colocated on the CP. No "a Droplet is not one / no nested" claim remains. Fail-closed probes, matrix, and harness unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com>
…oot-probe, budget flops Greptile on #247: P1 production guard bypass: refuse_prod parsed nothing and matched the URL case-sensitively, so https://NETWORK.CORTEX.FOUNDATION reached authenticated create / submit requests. The guard now parses the host (scheme, userinfo, port, path, query, trailing dot stripped; IPv6 kept bracketed), lower-cases it, and refuses a protected host or any subdomain of one, plus a case-insensitive whole-URL match as belt and braces. It runs at env load, before any other check, on every probe. P2 lost response strands VM: PROBE_TOPIC_LIVE is set before the create goes out; a create whose answer is 000 / 5xx / unparseable is reconciled through GET /v1/vms/by-topic and any VM the agent reports is destroyed in line (probe_reconcile_destroy), and the EXIT trap does the same for anything still in flight (Ctrl-C, unconfirmed teardown). Topic state is cleared only after a confirmed destroy followed by a 404, so a retry on the probe topic is never blocked. PROOF_VM_WIRE_CHECK_FAULT= lose-create-answer is a test-only hook that drops the create's answer. P1 live probe under-declares FLOPs: submit-probe declared 1, so a live run measuring more was a flops_under_declared reject. Fail-closed probes still send 1 (nothing runs); --expect 2xx declares the topic's flops_budget read from GET /v1/proof/topics/<id> (--declared-flops N overrides; unreadable budget = FAIL naming the flag). Tests (fake agent): uppercase / userinfo / trailing-dot / subdomain production origins exit 2 before any request, for submit-probe and for the agent URL at env load; the lost-answer create boots one VM that is found by topic and destroyed, the topic is free, a retry passes. Runbook § 3 / § 5 updated. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
… test Co-authored-by: Mathis <echobt@users.noreply.github.com>
…ready, not lium After #246 the custom family is wired from the topic-VM env alone and `live_harvest_wired` is the Lium harvest only (nll / throughput), so a custom-only host reads false by design. The harness `cp` step no longer FAILs on it: the flag is logged as informational, and the custom family is gated on `custom_family_wired` (FAIL when ids are set but the family is not routed), `registered_custom` ⊇ ids, and `custom_ready` ⊇ ids (registered but not ready = bearer file / image pin on this host). The admin probe log line shows the same three fields. `VmOrchestratorReport` carries `custom_family_wired` next to `registered_custom`, and its `live_harvest_wired` is the host's Lium-only answer. Runbook (§ Wire the control plane, admin-probe field table, § 0 preconditions, § 3 cp row), the CP staging overlay, docs/PROOF.md, and docs/COMPLETENESS.md say the same. Verified against a custom-only loopback proof-challenge wired to a loopback agent: custom gates PASS with live_harvest_wired=false logged, not failed. Co-authored-by: Mathis <echobt@users.noreply.github.com>
8e0bf30 to
2acff57
Compare
|
@greptileai please re-review after rebase onto #246 (f013abb) — live_harvest informational, custom_family gates. |
Greptile P2 on #247: `--declared-flops 0` passed the override check and reached the control plane for a live probe, where any measured usage is a flops_under_declared reject instead of the documented awaiting_admin path. Explicit overrides must now be positive integers; zero, negative, and non-numeric values exit 1 with an actionable message before any request. Test covers 0 / abc / -1 on a live-run invocation. Co-authored-by: Mathis <echobt@users.noreply.github.com>
|
@greptileai please re-review after P2 fix (b01d606) — refuse zero declared_flops override. |
* deploy: staging pins for 4252130 images.yml built GHCR digests on main then could not push the pin commit (GH013: PRs required + Greptile). Land the same promote via PR. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(deploy): keep full prior staging release in previous Sequential promote.sh calls were snapshotting previous per service, so rollback of 4252130 would restore a mixed pin set. Capture the 1dd07f7 release once and write it back after the promote loop. Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy: staging pins for f013abb Retarget #245: main moved to #246. Digests from images run 34281980177. previous stays the full 1dd07f7 staging release. Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy: staging pins for 890ce5b Retarget #245: main moved to #247. Digests from images run 34286333214. previous stays the full 1dd07f7 staging release. Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy: staging pins for c0ce094 Retarget #245: main moved through #248/#252. Digests from images run 34297422117. previous stays the full 1dd07f7 staging release. Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy: staging pins for 85ab2b7 Retarget #245: main moved to #251. Digests from images run 34299348722. previous stays the full 1dd07f7 staging release. Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy: staging pins for e30d51d Retarget #245: main moved to #250. Digests from images run 34302039784. previous stays the full 1dd07f7 staging release. Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy: staging pins for a8b9018 Retarget #245: main moved to #253. Digests from images run 34373475123. previous stays the full 1dd07f7 staging release. Co-authored-by: Mathis <echobt@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mathis <echobt@users.noreply.github.com>
Summary
Everything an operator needs to flip staging from
UnwiredVmOrchestrator→FirecrackerOrchestratoron DigitalOcean and prove fail-closed + one happy path. Rebased ontomainatf013abbc(#246): #246'sTopicVm/live_scorer/FamilyMux::custom_onlyare kept as-is; #247's admin probe rides on the sameTopicVm(one orchestrator per process, shared by the runner registry and the probe). No digest is invented anywhere, nothing weakens fail-closed, nothing in theproof-challengecontainer can exec Firecracker.Glue —
GET /v1/admin/proof/vm-orchestrator(operator bearer, read-only, no VM, no spend). Reports through the very client the runner drives:ready()(bearer file + RLM image pin, re-read now), the locked 4 / 8192 template, one agent health call (agent/agent_error), plus the host gatescustom_family_wired,registered_custom, andlive_harvest_wired(Lium only, informational for the custom family). A broken wire is data, not an error — URL unset, bearer refused, agent unreachable, digest unpinned each show up by name — and the bearer value never appears. This is what lets a droplet with nocargoprove the rustls/CA/bearer path the CP actually uses.Harness —
deploy/scripts/proof-vm-wire-check.sh(bash + curl + python3, runs on the staging master, bearer only ever in a 0600 curl config, production hosts refused case-insensitively before any request):envPROOF_VM_ORCHESTRATOR_URLhttps; bearer file non-empty via--path-map(compose bind mount), 0400 / uid 65532;PROOF_RLM_VM_IMAGE_DIGESTissha256:<64 hex>(empty / placeholder = FAIL); CA is PEM; custom ids well-formed; locked shape;PROOF_FORCE_SIMoffagentGET /v1/healthwith the bearer → ready / hypervisor / vms; no bearer → 401; wrong bearer → 401cp/v1/status:lium,custom_family_wired,registered_custom⊇ ids,custom_ready⊇ ids (live_harvest_wiredlogged, Lium-only, never a FAIL), no URL / token / path leak,/v1/proof/topicsholdout leak, executor readiness, then the admin probeboot-probe201, or an unconfirmed teardown all end in a by-topic attach + destroysubmit-probePOST /v1/submissionson a custom topic asserting the fail-closed code and reason; 2xx refused without--allow-live-run; a live run declares the topic'sflops_budget(--declared-flops Noverrides) and must carry the sister-measuredflops_usedmatrixPROOF_VM_ORCHESTRATOR_URL, emptied bearer →PROOF_VM_ORCHESTRATOR_TOKEN_FILE, wrong bearer →refused the bearer, unpinned →PROOF_RLM_VM_IMAGE_DIGEST, agent down →orchestrator unreachable, unknown topic / no locator → 400Runbook —
docs/runbooks/proof-vm-orchestrator.md§ DigitalOcean staging. Placement: staging colo on the CP droplet with nested/dev/kvmis an allowed exception, proven oncortex-staging(§ 4 green); production: dedicated DO metal preferred, never colocated on the CP; nested stays fragile — if the boot fails, provision metal. Colocated specifics (CP container reaches the agent on the droplet's VPC address, TLS + SAN,:8200firewalled to the compose network, sizing on top of the compose stack), host pre-checks, CP preconditions (custom family gated oncustom_family_wired/registered_custom/custom_ready; Lium optional), KVM-host + CP steps, wire check, fail-closed matrix, happy-path evidence table, sign-off checklist, rollback.Env overlays —
deploy/env/proof-challenge.staging-vm.example(CP) anddeploy/env/proof-vm-orchestrator.staging.example(agent host). Placeholders only; everyREPLACE_WITH_*fails closed as written.Small hardening —
proof-vm-fc: theorchestrator unreachable (…)text reaches miners as a 503 body; it now keeps the route and drops the agent URL.Cross-links: root
AGENTS.mditem 7,deploy/AGENTS.md,docs/AGENTS.md,docs/PROOF.md,docs/ARCHITECTURE.md,deploy/secrets/README.md,docs/COMPLETENESS.md, the systemd unit comment and the agent / binary module docs — same placement wording everywhere.Out of scope, as ordered: no SSH to any host, no digests, no nested-FC redesign.
Greptile
83b85f7c, threads resolved)@greptileai reviewTest plan
cargo test --workspace— green on the rebased tree (proof-http admin route 401 / 503 / 200 + unwired + stub report incl.custom_family_wired; proof-challenge probe against the in-process fake agent: ready, bearer rotated on one side →refused the bearer, stopped agent →unreachable, emptied bearer →ready: false; proof-vm-fctests/wire_check_script.rsruns the bash harness against the fake agent — router JSON, one boot + one Destroy, bearer never printed, dead agent fails, unpinned / empty bearer fail closed, uppercase / userinfo / trailing-dot / subdomain production origins exit 2 before any request, a lost create answer is reconciled by topic and destroyed and a retry passes)cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningsxtask loc-cap/consensus-lint/spec-check/design-check/external-docs-checkcargo deny: no new external crate inCargo.lock— same graph asmainproof-challenge(no Lium) wired to a loopback agent —cpPASSes the custom gates withlive_harvest_wired=falselogged, not failed; admin probe returnsorchestrator: firecracker,ready: true,custom_family_wired: true, agent health; both placeholder overlays fail closedassert-compose-matrix.sh— compose files untouched; no docker on this box (CI runs it)Risk
Deploy only: one new operator-bearer admin route on
proof-challenge(read-only; reports env-var names and container paths to the operator, never the bearer), a shell harness, docs, placeholder env examples. No change to scoring, emission, signatures, or miner-facing API;/v1/statusand/v1/proof/topicsunchanged.proof-challengeresolves the orchestrator once and passes it into #246'slive_scorer; boot log lines unchanged.Naming
I did not rename
BASE_*environment variables, deployed host paths(
/opt/base,/run/base, …), GHCRbaseintelligence/basepackage names, orbase-*-v1cryptographic domain tags, unless this PR’s purpose is a coordinatedcutover documented in
docs/NAMING.md.