Conversation
- Add mesh_peer_count() and topic_peer_count() to monitor mesh status - Add refresh_subscription() to force SUBSCRIBE exchange with peers - Add repair_mesh_if_needed() that runs every 10s to fix empty mesh - Trigger subscription refresh when new platform validator is identified - IMPORTANT: Do NOT use add_explicit_peer - it makes peers bypass mesh! - Fixes issue where validators joining existing network don't form mesh
…aim/Lease - Add crates/platform-server with REST API, WebSocket, PostgreSQL - Implement Control Plane API (validators, challenges, config) - Implement Data API with Claim/Lease anti-duplication mechanism - Add Rule Engine for server-side enforcement - Add observability with Sentry integration and audit trail - Add mechanism_id and emission_weight to ChallengeConfig - Add docker-compose.server.yml for subnet owner deployment - Add Dockerfile.server with optimized multi-stage build Architecture per README spec: - Platform Server is single source of truth - Challenges access DB only via Data API - Deterministic weights via /get_weights endpoint - Full audit trail for all data access
- Add api_key and api_provider fields to submission model - Add total_cost_usd tracking for LLM inference costs - Update schema with new columns for centralized API key storage - Update all submission queries to include new fields - Add update_submission_cost() for accumulating inference costs
- Add ws_transport.rs with WebSocket server for broker - Support JWT authentication for WebSocket connections - container-broker supports both Unix socket and WebSocket - Options: --ws-port (default 8090), --ws-only to disable Unix socket - Environment: BROKER_WS_PORT, BROKER_JWT_SECRET Challenge containers connect via WebSocket without socket mounting.
- Add secure-container-runtime as validator-node dependency
- Start WebSocket broker alongside validator (port 8090 default)
- Add --broker-port and --broker-jwt-secret CLI options
- Pass CONTAINER_BROKER_WS_URL and CONTAINER_BROKER_JWT to challenges
- Challenges spawn sandbox containers via WebSocket broker
- No Docker socket needed for challenge containers
Architecture:
validator-node → container broker (ws://0.0.0.0:8090)
→ challenges connect via WebSocket
→ sandbox containers spawned by broker
- challenge-orchestrator/evaluator.rs: remove fixed EvaluateRequest/Response, use generic JSON passthrough (challenge-agnostic) - platform-server/challenge_proxy.rs: increase timeout 30s -> 600s for long evaluations
- Remove hardcoded CHALLENGE_ID/CHALLENGE_URL requirements
- Add challenges table with full metadata (docker_image, resources, etc.)
- Implement ChallengeManager for dynamic container lifecycle
- Add REST API for challenge CRUD: /api/v1/challenges
- Dynamic routing: /api/v1/challenges/{id}/* proxies to containers
- Load challenges from DB on startup, manage via API
- Default owner hotkey: 5GziQCcRpN8NCJktX343brnfuVe3w6gUYieeStXPD1Dag2At
- Server can start with zero challenges (fully dynamic)
Challenge servers may default to different ports (e.g., 8081). Force PORT=8080 since orchestrator expects all challenges on 8080.
- Add --platform-server CLI option for centralized orchestration - Add PlatformServerClient for fetching weights from platform-server - In CommitWindowOpen, prefer platform-server weights over local calculation - Fallback to local weights if platform-server is unavailable
- Create bins/platform with subcommands: server, validator, version
- Server mode: runs platform-server with full orchestration
- Validator mode: delegates to validator-node binary (same args)
- Single Docker image can run both modes:
docker run platform server [OPTIONS]
docker run platform validator --secret-key <KEY> [OPTIONS]
This enables using a single Docker image for both subnet owner (server)
and validators, simplifying deployment and CI/CD pipelines.
- Add /api/v1/submissions endpoints for agent submission - Add /api/v1/evaluations endpoints for evaluation recording - Use simplified registration request (no signature required for dev) - All core APIs now available in unified binary
Validators use existing docker-compose without changes: - ENTRYPOINT: validator-node - Default: --data-dir /data --platform-server https://chain.platform.network - Reads VALIDATOR_SECRET_KEY from environment automatically
debian:bookworm-slim has older glibc which causes: GLIBC_2.38 not found GLIBC_2.39 not found Ubuntu 24.04 has the required glibc version.
No hardcoded challenges - fully dynamic:
- Register via POST /api/v1/challenges
- Start via POST /api/v1/challenges/{id}/start
- Stop via POST /api/v1/challenges/{id}/stop
Supports both formats: - postgresql://host:5432 (base URL) - postgresql://host:5432/postgres (full URL with db name)
|
Caution Review failedThe pull request is closed. 📝 WalkthroughWalkthroughIntroduces a centralized Platform Server as a new Rust crate with comprehensive HTTP APIs, PostgreSQL database integration, WebSocket support, role-based authentication, dynamic challenge orchestration, and task management. Simultaneously introduces a unified platform binary that dispatches to either server or validator modes via CLI. Extends the challenge SDK with new platform client and server modules, modernizing from P2P to centralized architecture. Updates validator-node integration to communicate with the platform server. Adds WebSocket transport to secure container runtime. Refreshes Docker infrastructure, CI/CD workflows, and dependency configurations. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Challenge Client
participant Server as Platform Server
participant DB as Database
participant Broker as Container Broker
participant Validator as Validator Node
rect rgb(200, 220, 255)
Note over Client,Validator: Authentication & Registration
Client->>Server: POST /api/v1/auth<br/>(hotkey, timestamp, signature)
Server->>DB: verify_signature() & check owner
DB-->>Server: valid
Server->>DB: create session (UUID token)
Server-->>Client: AuthResponse (token, session_id)
end
rect rgb(220, 200, 255)
Note over Server,DB: Submission & Task Management
Client->>Server: POST /api/v1/submissions<br/>(miner_hotkey, source_code)
Server->>DB: compute_agent_hash()
Server->>DB: create_submission()
DB-->>Server: submission_id
Server->>Server: broadcast SubmissionEvent
Server-->>Client: SubmitAgentResponse (id, agent_hash)
end
rect rgb(220, 255, 220)
Note over Validator,Server: Task Claim & Evaluation Workflow
Validator->>Server: POST /api/v1/data/tasks/claim<br/>(validator_hotkey)
Server->>DB: claim_task() atomic check
DB-->>Server: TaskLease granted
Server->>Server: broadcast TaskClaimedEvent
Server-->>Validator: task_id, submission_id, data
Validator->>Broker: establish WebSocket connection
Broker->>Validator: challenge execution environment
Validator->>Broker: submit evaluation result
Broker-->>Validator: ack
Validator->>Server: POST /api/v1/data/results<br/>(agent_hash, score, results)
Server->>DB: create_evaluation()
Server->>DB: update_leaderboard()
Server->>Server: broadcast EvaluationEvent
Server-->>Validator: evaluation_id
end
rect rgb(255, 240, 200)
Note over Client,Server: Leaderboard & Network State Query
Client->>Server: GET /api/v1/leaderboard?limit=100
Server->>DB: get_leaderboard(100)
DB-->>Server: Vec\<LeaderboardEntry\>
Server-->>Client: JSON array (rank, agent_hash, score)
Client->>Server: GET /api/v1/challenges/network-state
Server->>DB: multi-query (epoch, stake, validators)
DB-->>Server: aggregated state
Server-->>Client: NetworkStateEvent JSON
end
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: defaults Review profile: CHILL Plan: Pro ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (59)
Comment |
* feat(p2p-consensus): add bootstrap nodes config and peer count methods - Add DEFAULT_BOOTSTRAP_NODES constant for production bootstrap peers - Update production() config to use DEFAULT_BOOTSTRAP_NODES - Export DEFAULT_BOOTSTRAP_NODES from lib.rs - Add len() and is_empty() methods to PeerMapping - Add connected_peer_count() and has_min_peers() to P2PNetwork * feat(p2p): add Bittensor block linking to ChainState - Add bittensor_block and bittensor_block_hash fields to ChainState - Add link_to_bittensor_block() and linked_block() methods - Update Default impl with zero-initialized fields - Link state to Bittensor block on NewBlock events * feat(challenge-sdk): add P2P submission storage and evaluation status support - Add StoreSubmission, RequestEvaluationStatus, EvaluationStatusResponse message types - Add ValidatorEvaluationResult struct for tracking validator evaluations - Add store_submission() method to P2PChallengeClient for distributed storage - Add get_evaluation_status() method to query submission evaluation progress - Update decentralized runner to handle new message types - Add comprehensive tests for new message types and methods - Export ValidatorEvaluationResult from lib.rs and prelude * refactor: make centralized server optional, add deprecation notices for P2P migration * docs: add P2P decentralized mode documentation * feat: add Docker configs for decentralized P2P mode * style: format code with cargo fmt
…tence (#2) * feat: add challenges directory structure and workspace configuration * feat: add challenge-registry crate for challenge lifecycle management Create new platform-challenge-registry crate with: - Challenge discovery and registration - Version management (semver-based) - Lifecycle state machine (registered/starting/running/stopping/stopped) - Health monitoring with configurable checks - State persistence and hot-reload support - Migration planning for version upgrades Modules: - registry: Main registry with CRUD operations - lifecycle: State machine for challenge states - health: Health monitoring and status tracking - state: State snapshots for hot-reload - discovery: Challenge discovery from various sources - migration: Version migration planning - version: Semantic versioning support - error: Registry-specific error types * feat(core): add checkpoint system for state persistence * feat: add restoration system for checkpoint recovery * feat(rpc-server): add health check endpoints for rolling updates * docs: add challenge integration guide * test: add integration tests for checkpoint and restoration system * feat: add graceful shutdown with checkpoint persistence - Add ShutdownHandler struct for checkpoint management - Create periodic checkpoints every 5 minutes - Save final checkpoint on graceful shutdown (Ctrl+C) - Persist evaluation state for hot-reload recovery This enables validators to update without losing evaluation progress.
* feat(p2p-consensus): add bootstrap nodes config and peer count methods - Add DEFAULT_BOOTSTRAP_NODES constant for production bootstrap peers - Update production() config to use DEFAULT_BOOTSTRAP_NODES - Export DEFAULT_BOOTSTRAP_NODES from lib.rs - Add len() and is_empty() methods to PeerMapping - Add connected_peer_count() and has_min_peers() to P2PNetwork * feat(p2p): add Bittensor block linking to ChainState - Add bittensor_block and bittensor_block_hash fields to ChainState - Add link_to_bittensor_block() and linked_block() methods - Update Default impl with zero-initialized fields - Link state to Bittensor block on NewBlock events * feat(challenge-sdk): add P2P submission storage and evaluation status support - Add StoreSubmission, RequestEvaluationStatus, EvaluationStatusResponse message types - Add ValidatorEvaluationResult struct for tracking validator evaluations - Add store_submission() method to P2PChallengeClient for distributed storage - Add get_evaluation_status() method to query submission evaluation progress - Update decentralized runner to handle new message types - Add comprehensive tests for new message types and methods - Export ValidatorEvaluationResult from lib.rs and prelude * refactor: make centralized server optional, add deprecation notices for P2P migration * docs: add P2P decentralized mode documentation * feat: add Docker configs for decentralized P2P mode * style: format code with cargo fmt
…tence (#2) * feat: add challenges directory structure and workspace configuration * feat: add challenge-registry crate for challenge lifecycle management Create new platform-challenge-registry crate with: - Challenge discovery and registration - Version management (semver-based) - Lifecycle state machine (registered/starting/running/stopping/stopped) - Health monitoring with configurable checks - State persistence and hot-reload support - Migration planning for version upgrades Modules: - registry: Main registry with CRUD operations - lifecycle: State machine for challenge states - health: Health monitoring and status tracking - state: State snapshots for hot-reload - discovery: Challenge discovery from various sources - migration: Version migration planning - version: Semantic versioning support - error: Registry-specific error types * feat(core): add checkpoint system for state persistence * feat: add restoration system for checkpoint recovery * feat(rpc-server): add health check endpoints for rolling updates * docs: add challenge integration guide * test: add integration tests for checkpoint and restoration system * feat: add graceful shutdown with checkpoint persistence - Add ShutdownHandler struct for checkpoint management - Create periodic checkpoints every 5 minutes - Save final checkpoint on graceful shutdown (Ctrl+C) - Persist evaluation state for hot-reload recovery This enables validators to update without losing evaluation progress.
FIX #1: encode_dir_archive now archives each top-level child individually instead of adding the source as a root '.' member. A non-root eval uid extracting into a root-owned tmpfs-mode=1777 mount could not chmod/utime the '.' root, so tar exited 2 under the bootstrap's 'set -e' and aborted before the wrapped command ran (workspace READ + artifact WRITE both failed). FIX #2: swarm_backend.run() applied _cap_log(64KB) to the full cross-node stdout, silently truncating drain sections larger than the cap so the executor could not restore a drained archive (e.g. a checkpoint). Now cap only the human-readable remainder and re-append the drain sections uncapped. The frozen BrokerRunResponse schema is unchanged (no new field). Adds non-root-into-1777-tmpfs and >64KB-through-_cap_log regression tests.
CortexLM/relearn PR #2 published the image that implements the harvest contract, so the eval half of live scoring is now pinned: eval_image_digest = sha256:303c6357… (was empty) relearn_git_sha = 82e21442… (was 6e952d1e) Only those two values change. base_model, teacher_model, teacher_nvfp4, teacher_backend, eval_image, relearn_git, holdout_commitment, holdout_size, and public_ids are byte-for-byte unchanged. `committed_pin_cannot_rent_or_score_live_yet` existed to make this moment deliberate; it is replaced by `committed_pin_allows_live_rent` plus `eval_image_is_digest_only`, which pins the exact digest and asserts the tag never migrates into `eval_image`. Pinning the image is not the whole job, and the docs now say so: a live host still needs the harvest wired and a champion baseline recorded, each with its own 503 and boot-log line. `remote-deploy.sh` warns when a master host is missing the relearn holdout or the baseline, since with the image pinned those two files are the difference between scoring and 503 on every submission. Note recorded in COMPLETENESS.md: PR #2 is open, not merged, so the pinned SHA is not yet reachable from that repo's default branch. Re-pin to the merge commit when it lands. Co-authored-by: Mathis <echobt@users.noreply.github.com>
…ge (#202) * feat(relearn): one-challenge subnet v0 eval loop Retire Design and Prism as live products. Wire Relearn HTTP submit, displacement scoring, digest-freeze holdout, and operator promote. Keep Lium/receipt/paired-test rails. Pin CortexLM/relearn (seed in-tree until org write can create the public repo). Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(site): satisfy clippy assigning-clones and match-same-arms Workspace clippy -D warnings failed on Relearn arena mapping. Co-authored-by: Mathis <echobt@users.noreply.github.com> * chore(relearn): pin CortexLM/relearn git sha * docs(relearn): drop anti-TEE slogans from user-facing copy Keep miners-pay-Lium as the trust line. Leave holdout-after-freeze in the eval loop internals, not README/miner Trust rows. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): split miner and validator one-pagers README is what Cortex is plus two links. Validators get a one-screen role; Relearn miners get submit, Lium, promote-wait, and a pointer. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): keep miner and validator pages together Both audience pages live under docs/external-miner/. README still has two links only. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): env-only teacher http api Teacher URL, model, and key come from RELEARN_TEACHER_* only. Missing URL or key skips to sim. No baked host. Miners pay Lium. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(bounty): add bounty challenge beside relearn Wire a second live challenge (id bounty) on the Relearn PR branch: pair a Bittensor hotkey to a dedicated Cortex Chat account, file bug reports, and score by precision vs the previous champion. Default emission is relearn 7000 / bounty 3000. Chat inject stays env-only; no teacher hosts or secrets in git. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(bounty): consume backend public api Cortex reads CortexLM/backend GET /v1/bounty/public/* for scoring. It does not serve a public leaderboard. Unset URL skips (CI). Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-t2i): add Relearn T2I challenge on Cosmos3 + Q-Judger Pin nvidia/Cosmos3-Super-Text2Image (OpenMDW 1.1, verified card) as the generator seed miners fine-tune, and Q-Judger (Qwen/Qwen-Image-Bench, Apache-2.0) as the only judge. Flux-family bases are refused outright. Eval prompts are frozen in config/relearn-t2i-pin.toml so no miner brings its own upsampler to the scored split, and every miner generates the same prompt ids at the same derived seeds. The holdout split is present in git only as a commitment; records come from an operator file and are verified at boot, so a wrong file refuses submissions instead of scoring the public split. Gates: paired displacement on the holdout, per-L1-pillar regression epsilon, seed replay, agentic faithfulness agreement, contamination, and a judge N/A ceiling. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-mm): add Relearn Multimodal challenge beside Relearn LLM Miners attach a permissively licensed vision encoder plus a projector to the champion Relearn LLM. Pin google/siglip2-so400m-patch14-384 (verified Apache-2.0) and restrict miner-supplied encoders to OSI-permissive terms. Two gates, both mandatory. Gate 1 reruns the existing Relearn text holdout on the submitted LM with vision ignored; a drop past epsilon zeroes the submission no matter how good the vision numbers are, and an encoder-only submission must additionally hash-match the champion LM. Gate 2 is a frozen image holdout (captioning, VQA, OCR, spatial relations - not ImageNet or COCO test) plus agentic image-tool traces, each replayed with the pixels shuffled so a model that ignores the image cannot pass. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(challenges): split emission four ways and wire the new services Retune challenges.toml to relearn 4000 / relearn-t2i 1500 / relearn-mm 1500 / bounty 3000 with a dedicated public key per challenge, re-signed under a fresh throwaway owner key (the previous owner secret is off-git by design). Wire relearn-t2i-challenge on :8097 and relearn-mm-challenge on :8098 through the Dockerfile, compose matrix, env examples, GHCR image lanes, remote-deploy, and local-e2e. Local smoke materializes the T2I holdout from the documented dev salt; without it the service still answers /health and 503s submissions, which is the intended fail-closed state. Docs: README lists all four challenges, docs/RELEARN-T2I.md and docs/RELEARN-MM.md carry the control-plane contracts, and the miner guides state that Flux is rejected, Q-Judger is the only T2I judge, and encoder licenses must be OSI-permissive. external-docs-check now pins those claims. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(site): list Relearn T2I and Relearn Multimodal as arenas The site matched emission shares by slug and only knew relearn, so after the four-way split the marketing surface would have shown one challenge's 4000 bps as the whole subnet. Add both arena slugs (wire form equals the trust-root challenge id), frames, and status fetches, and drop the retired design/prism arena list that both landing and metrics were still feeding. A down challenge backend leaves the static frame in place rather than dropping the arena, so the emission column still accounts for every challenge. Added a regression test that every live arena slug resolves to a non-zero trust-root share and that the shares sum to 10000. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): fail-closed holdout and anti-overfit gates Pin a holdout commitment on the live relearn path, load records only from RELEARN_HOLDOUT_FILE, and reject submissions when the file is missing or does not match. Add contamination, public-gap, vision pixel-shuffle, and off-lattice canary-regression gates. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): pin 27b and glm flash nvfp4 on #199 Overlay locked Hugging Face ids on Testeur's holdout gates. Keep public_ids, holdout_commitment, and holdout_size. vLLM serves from RELEARN_TEACHER_LOCAL_DIR, never the repo id. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(repo): complete greptile review gate on the overlay Template, .greptile rules, and a short AGENTS / miner-docs note. Covers fail-closed, holdout-off-git, no Modal/secrets, no Flux, OSI encoders, and BASE_* / domain-tag freeze. Additive; scoring gates unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): pin teacher to incoai glm-5.3 nvfp4 Relock teacher_nvfp4 to incoai/GLM-5.3-NVFP4 (full GLM-5.3). Wire id glm-5.3, not flash. Serve from RELEARN_TEACHER_LOCAL_DIR. Keep #199 holdout_commitment, public_ids, and holdout_size. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): fail closed instead of scoring submissions in sim Live submit could not produce a real verdict. `eval_after_freeze` always called `sim_slice_scores`, so a host with an empty `eval_image_digest` answered `201 CREATED` with simulated numbers, and an empty miner manifest skipped the contamination gate entirely. Eval backend is now explicit, mirroring relearn-mm: - `EvalBackend::{Lium, Sim}` + `force_sim()` / `resolve_eval_backend()`. Sim is selected only by `RELEARN_FORCE_SIM`, never as a fallback. - `eval_after_freeze` takes the pin and the backend. On `Lium` it returns `EvalImageUnpinned` without a `sha256:` pin and `LiveHarvestUnavailable` with one; both map to `503`, so no sim number can reach the lattice. - `/v1/status` publishes `eval_backend`, `force_sim`, and `can_score`; the submit row publishes `eval_backend`. - The sim base champion is only seeded on a host that resolved `Sim`. Contamination now carries its evidence: - `ContaminationEvidence` records the declared id / image-hash / dataset counts next to the hits, and `contamination_evidence` replaces `contaminated_fingerprints`. - An undeclared manifest is `GateFail::ContaminationEvidenceMissing`, the same fail-closed shape as `PublicEvidenceMissing`. The offline harness also could not express any promotion: perturbed and holdout were drawn from independent salts, so the drop was ~0.17 against a 0.05 ceiling and every sim run failed `Perturbation`. Sim slices are now derived from one skill level per artifact (`sim_artifact_skill`), with the retention slices offset from the holdout draw, so `RELEARN_FORCE_SIM=1` can reach `awaiting_admin` and the gate tests compare like with like. Pin ids and `judge_challenger` thresholds are untouched. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): say the committed holdout commitment is not the live seal The pin and COMPLETENESS named the local dev salt next to the committed `holdout_commitment`, which reads as the production seal. Both now describe it as the CI / local commitment, point at `config/CEREMONY.md`, and keep the requirement that production rotate salt *and* catalog before re-signing. The salt literal stays only in `deploy/scripts/local-e2e.sh`, where it is the local default; `committed_pin.rs` asserts it is absent from the pin. Also documents the two fail-closed behaviours this branch adds: the 503 a host returns without a `sha256:` eval-image pin (with `eval_backend` / `can_score` on `/v1/status`), and that an undeclared miner `manifest` fails the contamination gate. Miner example manifest now declares real fields instead of three empty arrays, which would now be rejected. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): let RELEARN_FORCE_SIM=1 boot instead of failing to parse Found on the live HTTP run: `--force-sim` bound `env = "RELEARN_FORCE_SIM"` to a clap `bool`, so the documented `RELEARN_FORCE_SIM=1` was rejected with `invalid value '1' for '--force-sim'` and the service exited at boot. Only compose's `"true"` / `"false"` strings worked, which is why it went unnoticed. Drops the env binding, matching `relearn-mm-challenge` and `relearn-t2i-challenge`: the value is read by `resolve_eval_backend`, which accepts `1` / `true` / `yes`. A test pins every documented spelling. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn-mm): follow the locked LLM base id instead of Flash-Next `cargo test --workspace` is red on the parent branch: #200 moved `relearn_challenge_task::BASE_MODEL_ID` to `Qwen/Qwen3.8-27B`, but `relearn-mm-task` re-exports that constant as `LM_BASE_MODEL_ID` while `config/relearn-mm-pin.toml` and four tests still spelled out `Qwen/Qwen3.8-Flash-Next`. Eight tests failed, including `RelearnMmPin::validate` on the committed pin. The encoder attaches to the Relearn champion's LM, so the MM pin has to carry the live LLM base — this aligns it rather than choosing a new model. `lm_side_tracks_the_relearn_champion_base` now asserts against `relearn_challenge_task::BASE_MODEL_ID` so the two cannot drift again. Also updates the miner-facing seed harness default (`docs/external-miner/relearn-seed/eval/harness/eval.py`), which still pointed `RELEARN_BASE_MODEL` at Flash-Next. Drop this commit if #200 fixes the fallout upstream. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): tell seed miners to declare their training manifest Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): record a live champion baseline and stop banking refusals Two holes the live replay of d7c5920 found behind the digest 503. **A live host could never record a champion baseline.** Making the sim baseline sim-only was right — a live challenger judged against simulated champion scores is not a comparison — but it left the live path with no baseline at all. The moment a `sha256:` eval image is pinned, submit answers `no champion baseline recorded` before contamination, public–holdout gap, or pixel-shuffle can run, so every gate this branch exists to protect is dead code on the only host that matters. Boot now records the baseline with the scorer the host will actually use: - `LiveScorer` is the eval-image harvest seam. `eval_after_freeze` takes it and refuses (`LiveHarvestUnavailable`) rather than substituting sim. - `BaselineMeasurement` (`RELEARN_BASE_CHAMPION_FILE`) is the operator's recorded measurement of the base model. `verify` binds it to the pin's `eval_image_digest` and `holdout_commitment` and refuses a measurement missing a series the gates read, so a junk baseline fails at boot instead of silently rejecting every challenger. - `boot_base_champion` picks the recorded measurement, else the wired harvest, else refuses. Sim numbers are not a candidate on a live host. **Refusals banked rows.** The row was inserted before the eval, so every 503 left an `evaluating` row with no scores that appears on no operator surface — spammable. Nothing is persisted until scoring produced a verdict, and the row is inserted once in its final state instead of insert-then-patch. Check order now puts the root cause first and spends nothing it cannot use: holdout unseal, then `scoring_readiness` (digest pin + harvest), then the baseline, then the eval. An unpinned digest still reports the pin, not the baseline, and a live eval never runs for a submission the host could not judge — that eval is the miner's Lium spend. `/v1/status` gains `live_harvest_wired` and `champion_baseline_recorded` next to `can_score`, since "cannot score" now has three distinct causes. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): document the champion baseline the live gates need Adds the operator surface for the live baseline: what `RELEARN_BASE_CHAMPION_FILE` is, the JSON the eval image has to emit, and why `public` / `general_canary` cannot be empty (a champion the gates cannot read rejects every challenger for a reason the miner cannot act on). Also records the two new `/v1/status` fields (`live_harvest_wired`, `champion_baseline_recorded`) and the three distinct causes of `can_score: false` in the miner troubleshoot table, notes that a refusal persists no row, and adds the operator checklist item plus COMPLETENESS gaps for the unwired live harvest. Co-authored-by: Mathis <echobt@users.noreply.github.com> * test(prism-verda): serialize the two PRISM_VERDA_COMPUTE tests `pick_honors_compute_override` sets a process-wide env var that `pick_compute` reads, so it raced `pick_b200_then_fallback`: whichever ran while the override was set picked `L40S` and failed. It flaked in two of three `cargo test --workspace` runs and passed in isolation, which made it hard to tell whether an unrelated PR was green. Test-only mutex around both. Pre-existing and unrelated to this branch — drop this commit if it lands elsewhere. Co-authored-by: Mathis <echobt@users.noreply.github.com> * test(prism-verda): use the method reference clippy asks for Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): wire the live harvest on the Lium path `live_scorer` was hardcoded `None`, so a digest-pinned host still refused with `live_harvest_wired: false` and the champion could only come from `RELEARN_BASE_CHAMPION_FILE`. This adds the control-plane client. The scoring code is not here: it ships inside `eval_image` from CortexLM/relearn. `relearn-lium-harvest` is the client for it. - `LiveScorer` is now async (`#[async_trait]`, as `EvalJobBackend` already is), so `eval_after_freeze` and `boot_base_champion` are too. The bin builds its runtime before recording the baseline. - `RelearnEvalMetrics` is the image contract: the `BaselineMeasurement` envelope plus run identity, so the operator's baseline file is literally the image's output for the base model — one format, not two. `verify` binds schema, `submission_digest`, and `artifact_digest`, so a pod cannot answer with another artifact's numbers or replay an earlier run. - `LiumHarvest` owns the lifecycle: boot the digest-pinned image, deliver the request, harvest, then terminate and require verified teardown before any score is accepted. An orphan pod outranks the run result because it keeps spending the miner's money. - `LiumEvalPod` is the Lium/SSH transport. Run inputs travel in `request.json` over stdin, never interpolated into the remote command, so a crafted digest cannot be shell injection on the pod. The workdir is scrubbed after the run; termination is the real guarantee. - `build_live_scorer` wires it on `EvalBackend::Lium` only, from `LIUM_API_KEY`. Sim scores in-process and never gets a pod. No key means no harvest and the host refuses. Nothing in this crate can compute a score. A pod that returns no `RELEARN_EVAL_OK`, no metrics document, or a document bound to another run is an error, not a fallback — the tests feed metrics-document fixtures through the real parse/verify path rather than substituting sim numbers. `prism-lium` additionally re-exports `ssh_exec{,_allow_fail,_stdin}` and `SshExecOutput` so the client can reuse its retry/timeout helpers. Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn): require the master SSH key and document the image contract Live replay of the wiring found the spec left `ssh_public_keys` empty, so `provision` refused with "Lium rent requires at least one SSH public key" — the harvest could never boot a pod. A pod with no master key is also unreachable, so the request could not be delivered and no metrics could be read back. - `LiumHarvest` takes the master public key(s) and registers them under `relearn-eval-worker`; it refuses before renting when there is none. - The bin loads it from `LIUM_SSH_PUBLIC_KEY_FILE` (same convention as `prism-challenge`) and does not wire the harvest without it. - `LIUM_API_BASE_URL` can point a staging host at a stand-in provider instead of spending real money. - Provider and eval-image failures are now 503, not 500: the host cannot score right now, which is not the miner's mistake, and retrying is right. Documents the image contract in `docs/RELEARN.md` — entrypoint, marker lines, document shape, and the identity/pin checks — so `CortexLM/relearn` has something to implement against. Also records, in RELEARN.md, COMPLETENESS.md, and the operator checklist, that the harvest request carries the holdout to a miner-paid pod: mitigated by the digest-pinned image, tmp delivery, post-run scrub, and verified termination, but not eliminated. That exposure is an owner decision, not an implementation detail. Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy(relearn): pin the published eval image digest and relearn SHA CortexLM/relearn PR #2 published the image that implements the harvest contract, so the eval half of live scoring is now pinned: eval_image_digest = sha256:303c6357… (was empty) relearn_git_sha = 82e21442… (was 6e952d1e) Only those two values change. base_model, teacher_model, teacher_nvfp4, teacher_backend, eval_image, relearn_git, holdout_commitment, holdout_size, and public_ids are byte-for-byte unchanged. `committed_pin_cannot_rent_or_score_live_yet` existed to make this moment deliberate; it is replaced by `committed_pin_allows_live_rent` plus `eval_image_is_digest_only`, which pins the exact digest and asserts the tag never migrates into `eval_image`. Pinning the image is not the whole job, and the docs now say so: a live host still needs the harvest wired and a champion baseline recorded, each with its own 503 and boot-log line. `remote-deploy.sh` warns when a master host is missing the relearn holdout or the baseline, since with the image pinned those two files are the difference between scoring and 503 on every submission. Note recorded in COMPLETENESS.md: PR #2 is open, not merged, so the pinned SHA is not yet reachable from that repo's default branch. Re-pin to the merge commit when it lands. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): stop implying the baseline file can replace the harvest The boot warning read "submissions will 503 unless RELEARN_BASE_CHAMPION_FILE covers the baseline and a harvest is configured", which is wrong now that the digest is pinned: the replay showed a host with the baseline recorded and no harvest still 503ing every submission. The baseline covers the champion only; each submission needs its own measurement. Points at the env example instead. Co-authored-by: Mathis <echobt@users.noreply.github.com> * refactor(harvest): share one digest-pinned lium pod transport Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-image): fail-closed live eval on the relearn-image id Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-agent): add the replayed-tool-trace challenge Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(relearn-agent): pin the episode set and add the ceremony helper Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(bounty): price bugs by severity and fail closed on ingest Co-authored-by: Mathis <echobt@users.noreply.github.com> * feat(challenges): relock four live ids and re-sign the trust root Co-authored-by: Mathis <echobt@users.noreply.github.com> * deploy(challenges): bring up the four live services and gate mm off Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(challenges): name the four live challenges and their gates Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(challenges): pin image/agent evals to relearn#3 Digest-only pins from CortexLM/relearn PR #3, harvest contracts that speak those images (shared RELEARN_METRICS markers, recorded traces, optional image canary), and miner/operator docs for the four live ids. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn-image): split canary gating out of judge_challenger Keeps the function under clippy's 100-line cap after the both-empty skip, without weakening the one-sided fail-closed rule. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn-image): name the published image-eval pin Primary eval_image is ghcr.io/cortexlm/relearn-image-eval at the relearn#3 digest; t2i-eval stays the same-digest alias. Teacher ids on relearn are untouched. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): pin the eval image that prints RELEARN_EVAL_OK Match #201 HEAD: sha256:00839671… / 822d2729. Refuse sha256:303c6357… which never printed the marker. Image, agent, and bounty pins unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn-agent): can_score stays false until holdout loads Status was advertising can_score on sim/live hosts with no verified episodes while submit already 503d. Holdout is now the first readiness check. LLM digest and image/bounty pins are untouched. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): forward teacher env and fail before rent Port the #201 LLM harvest gate onto the harvest-pod transport: without RELEARN_TEACHER_API_URL, can_score is false and submit 503s naming the var, with no Lium rent. Teacher env is delivered over stdin as teacher.env (umask 077, set -a) rather than interpolated into the remote command. Missing-marker 503s include a redacted run.log tail. Image/agent harvests stay env-free. LLM digest, image/agent pins, and bounty fail-closed are unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): hoist Unready scorer stub for clippy Move the scoring_readiness teacher-gate fixture out of the test body so items-after-statements stays clean. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(challenges): fail-before-rent and screen junk before eval Image and Agent harvest now refuse without a judge/teacher URL, forward stdin teacher.env the same way LLM does, and reject contaminated or empty-evidence manifests before renting. Bounty treats any prior fingerprint as duplicate and refuses title==body / token-thin reports. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(challenges): clippy persist helpers and image happy-path Pre-eval reject helpers satisfy too-many-arguments/field-reassign lints. Image happy-path submits a declared manifest so it still evals instead of being screened as empty evidence. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): can_score waits for champion; resolve eval PATH Live status advertised can_score while submit 503'd without a champion baseline. Harvest run_cmd now prefers /usr/bin/<eval> (else command -v) so a login-less SSH PATH cannot miss the binary. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): pin eval image that ships /usr/bin/relearn-eval Replace sha256:00839671 (exit 127 on a login-less SSH PATH) with sha256:86240d76 from CortexLM/relearn 9998154f. Image/agent pins unchanged. Do not pin sha256:303c6357. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): pin the CUDA scoring image CI pulled sha256:201cc5d2 ships /usr/bin/relearn-eval (relearn d107a7c1). Do not pin 86240d76, 00839671, or 303c6357. Image/agent pins unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(lium): rent digest-pinned harvest templates not recipes Harvest was setting image_digest and then renting prism-recipe-v10, so SSH ran /usr/bin/relearn-eval on the wrong image (exit 127). InstanceSpec now carries docker_image + startup; provision uses that pin and refuses a recipes fallback. Probe the score binary before staging the holdout. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(lium): harvest template is repo@digest with no startup The live relearn-eval image already CMD ["serve"] (tini + entrypoint). Create the Lium template with docker_image=pin@digest, no tag, no startup, and a 12-hex name so recipes cannot be reused. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): pin GLM-5.3 judge image; can_score needs holdout Lock the LLM eval image to sha256:cbc4bbb8 (git f3cfa69). Reject the four earlier judge-broken prefixes. LLM and Image can_score now require the holdout file to be loaded, matching Agent. Bounty HTTP rejects a second miner refiling a closed fingerprint. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): drop pin _token wording; HTTP mismatch 503 Reword the GLM-5.3 pin comment so the committed-pin secret scan does not treat max_tokens as a leaked credential. Add HTTP tests that a commitment-mismatched holdout cannot load, so status stays can_score false and submit is 503 on LLM, Image, and Agent. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): forward base weights; fail closed before rent The eval image does not bake Qwen. TeacherEnv now forwards RELEARN_BASE_MODEL_DIR, HF_HOME, HF_HUB_CACHE, and an explicit RELEARN_ALLOW_MODEL_DOWNLOAD=1 into teacher.env over stdin. ready() refuses to rent when neither DIR nor ALLOW is set. Status publishes base_weights.primed + via (var name only). ALLOW_DOWNLOAD is never defaulted. Pin stays cbc4bbb8. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): bin ready() tests require ALLOW_DOWNLOAD A teacher/judge URL alone is no longer enough to score. Binary readiness tests now fail closed without DIR/ALLOW, then pass with the first-champion ALLOW_DOWNLOAD=1 path. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(lium): rent whole-host ncu b200 for 1-gpu pin Lium NCU hosts reject split even when min_gpu_count_for_rental is 1. Parse ncu_profiling_enabled and rent the whole host instead of posting gpu_count=1 on the live 2× B200. A 1-GPU harvest pin still matches. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): unpin cbc4bbb8; no eval image has scored cbc4bbb8 (f3cfa69) exited 1 on a rented pod without RELEARN_EVAL_OK: no vLLM on the CUDA scoring image, transformers fallback, then Qwen3VLVideoProcessor crashed for want of torchvision. An empty digest is the fail-closed state - submissions 503 rather than renting a B200 that cannot score. The pin now names every digest that reached a pod and failed, and the tests assert the fail-closed state instead of a digest. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): fail closed when gate evidence is missing Perturbation and the known-answer canaries were 'if let Some(..)': a run whose eval document omitted the series took neither gate. Pixel shuffle iterated the challenger's own map, so dropping a family skipped the control. All three now fail closed, with the champion as the reference for which vision families the holdout actually has images in. BaselineMeasurement::verify additionally requires perturbed + canaries, so an image that never emits them is refused at boot rather than rejecting every challenger for a reason the miner cannot act on. EvalError::Integrity is a 503, not a 500: an unverified teardown is this host refusing to believe a run, not the miner's mistake. Co-authored-by: Mathis <echobt@users.noreply.github.com> * docs(relearn): unpinned eval image and fail-closed gate evidence RELEARN.md, COMPLETENESS.md, and the miner-facing external-miner docs now say the digest is empty on purpose and what the next image needs (vLLM + torchvision), and list the new *_evidence_missing rejections so a miner can tell a skipped gate from a failed one. Co-authored-by: Mathis <echobt@users.noreply.github.com> * test(relearn): cover the live path end to end through HTTP One run walks champion recorded -> can_score true -> a clean submission scored on the Lium backend with lattice -> every gate zeroing its own run: contamination, undeclared manifest, public-holdout gap, general canary, and the three that used to be skippable by omitting the series. Each rejection is asserted against its own gate name, so failing closed everywhere would not pass. Second test: promoting moves the bar. A later challenger at the same skill ties the sitting champion instead of beating the base model. relearn-score re-exports ExampleSeries so a consumer can build the slices whose types are already in its public API. Co-authored-by: Mathis <echobt@users.noreply.github.com> * fix(relearn): split judge gates for clippy; name every eval status judge_challenger crossed the 100-line cap once the missing-evidence arms landed, so the retention and pixel-shuffle gates are their own functions. eval_err now matches EvalError exhaustively: a new failure mode has to be given a status deliberately instead of inheriting a 500. Co-authored-by: Mathis <echobt@users.noreply.github.com> * test(relearn): committed pin refuses live scoring at runtime Asserting the digest string is empty says the file is right; calling scoring_readiness with the shipped pin says the host is. The reported cause has to be the pin, not a downstream symptom. Co-authored-by: Mathis <echobt@users.noreply.github.com> * chore(ci): retrigger checks after retargeting to main Co-authored-by: Mathis <echobt@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mathis <echobt@users.noreply.github.com>
This PR introduces the centralized Platform Server architecture, enabling dynamic challenge management and unified validator deployment.
Summary by CodeRabbit
New Features
Infrastructure
✏️ Tip: You can customize this high-level summary in your review settings.