Skip to content

v1.0.0 Platform - #2

Merged
echobt merged 19 commits into
mainfrom
dev
Dec 27, 2025
Merged

v1.0.0 Platform#2
echobt merged 19 commits into
mainfrom
dev

Conversation

@echobt

@echobt echobt commented Dec 27, 2025

Copy link
Copy Markdown
Contributor

This PR introduces the centralized Platform Server architecture, enabling dynamic challenge management and unified validator deployment.

Summary by CodeRabbit

  • New Features

    • New Platform Server application for managing and orchestrating challenges with APIs for validators, submissions, evaluations, and leaderboard management.
    • Unified platform binary supporting both server and validator modes via single CLI.
    • Real-time WebSocket event streaming for challenge updates and server communications.
    • Challenge SDK with HTTP server framework and platform client for improved challenge integration.
    • Container broker for secure WebSocket-based challenge communication.
  • Infrastructure

    • Docker compose configurations for streamlined server deployment.
    • Environment configuration examples for server setup.

✏️ Tip: You can customize this high-level summary in your review settings.

- Add mesh_peer_count() and topic_peer_count() to monitor mesh status
- Add refresh_subscription() to force SUBSCRIBE exchange with peers
- Add repair_mesh_if_needed() that runs every 10s to fix empty mesh
- Trigger subscription refresh when new platform validator is identified
- IMPORTANT: Do NOT use add_explicit_peer - it makes peers bypass mesh!
- Fixes issue where validators joining existing network don't form mesh
…aim/Lease

- Add crates/platform-server with REST API, WebSocket, PostgreSQL
- Implement Control Plane API (validators, challenges, config)
- Implement Data API with Claim/Lease anti-duplication mechanism
- Add Rule Engine for server-side enforcement
- Add observability with Sentry integration and audit trail
- Add mechanism_id and emission_weight to ChallengeConfig
- Add docker-compose.server.yml for subnet owner deployment
- Add Dockerfile.server with optimized multi-stage build

Architecture per README spec:
- Platform Server is single source of truth
- Challenges access DB only via Data API
- Deterministic weights via /get_weights endpoint
- Full audit trail for all data access
- Add api_key and api_provider fields to submission model
- Add total_cost_usd tracking for LLM inference costs
- Update schema with new columns for centralized API key storage
- Update all submission queries to include new fields
- Add update_submission_cost() for accumulating inference costs
- Add ws_transport.rs with WebSocket server for broker
- Support JWT authentication for WebSocket connections
- container-broker supports both Unix socket and WebSocket
- Options: --ws-port (default 8090), --ws-only to disable Unix socket
- Environment: BROKER_WS_PORT, BROKER_JWT_SECRET

Challenge containers connect via WebSocket without socket mounting.
- Add secure-container-runtime as validator-node dependency
- Start WebSocket broker alongside validator (port 8090 default)
- Add --broker-port and --broker-jwt-secret CLI options
- Pass CONTAINER_BROKER_WS_URL and CONTAINER_BROKER_JWT to challenges
- Challenges spawn sandbox containers via WebSocket broker
- No Docker socket needed for challenge containers

Architecture:
  validator-node → container broker (ws://0.0.0.0:8090)
                 → challenges connect via WebSocket
                 → sandbox containers spawned by broker
- challenge-orchestrator/evaluator.rs: remove fixed EvaluateRequest/Response,
  use generic JSON passthrough (challenge-agnostic)
- platform-server/challenge_proxy.rs: increase timeout 30s -> 600s for long evaluations
- Remove hardcoded CHALLENGE_ID/CHALLENGE_URL requirements
- Add challenges table with full metadata (docker_image, resources, etc.)
- Implement ChallengeManager for dynamic container lifecycle
- Add REST API for challenge CRUD: /api/v1/challenges
- Dynamic routing: /api/v1/challenges/{id}/* proxies to containers
- Load challenges from DB on startup, manage via API
- Default owner hotkey: 5GziQCcRpN8NCJktX343brnfuVe3w6gUYieeStXPD1Dag2At
- Server can start with zero challenges (fully dynamic)
Challenge servers may default to different ports (e.g., 8081).
Force PORT=8080 since orchestrator expects all challenges on 8080.
- Add --platform-server CLI option for centralized orchestration
- Add PlatformServerClient for fetching weights from platform-server
- In CommitWindowOpen, prefer platform-server weights over local calculation
- Fallback to local weights if platform-server is unavailable
- Create bins/platform with subcommands: server, validator, version
- Server mode: runs platform-server with full orchestration
- Validator mode: delegates to validator-node binary (same args)
- Single Docker image can run both modes:
    docker run platform server [OPTIONS]
    docker run platform validator --secret-key <KEY> [OPTIONS]

This enables using a single Docker image for both subnet owner (server)
and validators, simplifying deployment and CI/CD pipelines.
- Add /api/v1/submissions endpoints for agent submission
- Add /api/v1/evaluations endpoints for evaluation recording
- Use simplified registration request (no signature required for dev)
- All core APIs now available in unified binary
Validators use existing docker-compose without changes:
- ENTRYPOINT: validator-node
- Default: --data-dir /data --platform-server https://chain.platform.network
- Reads VALIDATOR_SECRET_KEY from environment automatically
debian:bookworm-slim has older glibc which causes:
  GLIBC_2.38 not found
  GLIBC_2.39 not found

Ubuntu 24.04 has the required glibc version.
No hardcoded challenges - fully dynamic:
- Register via POST /api/v1/challenges
- Start via POST /api/v1/challenges/{id}/start
- Stop via POST /api/v1/challenges/{id}/stop
Supports both formats:
- postgresql://host:5432 (base URL)
- postgresql://host:5432/postgres (full URL with db name)
@echobt
echobt merged commit dee38a8 into main Dec 27, 2025
3 of 4 checks passed
@coderabbitai

coderabbitai Bot commented Dec 27, 2025

Copy link
Copy Markdown

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

Introduces a centralized Platform Server as a new Rust crate with comprehensive HTTP APIs, PostgreSQL database integration, WebSocket support, role-based authentication, dynamic challenge orchestration, and task management. Simultaneously introduces a unified platform binary that dispatches to either server or validator modes via CLI. Extends the challenge SDK with new platform client and server modules, modernizing from P2P to centralized architecture. Updates validator-node integration to communicate with the platform server. Adds WebSocket transport to secure container runtime. Refreshes Docker infrastructure, CI/CD workflows, and dependency configurations.

Changes

Cohort / File(s) Summary
Platform Server Core
crates/platform-server/Cargo.toml, crates/platform-server/src/main.rs, crates/platform-server/src/lib.rs
New server crate with complete HTTP/WebSocket infrastructure, database initialization, and router configuration with health, control plane, data API, and dynamic challenge proxy endpoints.
Server API Handlers
crates/platform-server/src/api/{auth,challenges,evaluations,leaderboard,submissions,validators}.rs, crates/platform-server/src/api/mod.rs
Six API modules: authentication with role-based access control, challenge configuration management, evaluation submission/retrieval, leaderboard ranking, agent submissions, and validator registration with heartbeat and event broadcasting.
Database Layer
crates/platform-server/src/db/{mod,queries,schema}.rs
PostgreSQL integration with connection pooling, comprehensive query functions for validators, submissions, evaluations, leaderboard, challenges, tasks, and network state; includes schema definition with migrations and multi-database support for per-challenge isolation.
Server State & Infrastructure
crates/platform-server/src/{state,challenge_proxy,orchestration,data_api}.rs
Application state management with session/event broadcasting, HTTP proxy to challenge containers with health checks and weight fetching, challenge orchestrator integration for Docker-based container management, and data API for task claiming/leasing workflows.
Observability & Security
crates/platform-server/src/{observability,rule_engine}.rs
Sentry integration with structured audit logging, rate limiting, and validation rules; audit event tracking for auth, submissions, evaluations, and tasks; rule engine for submission/evaluation/claim validation with per-hotkey rate limiting.
WebSocket Real-Time
crates/platform-server/src/websocket/{mod,events,handler}.rs
Broadcast-based WebSocket event system with connection tracking, authentication via bearer tokens, and message handling for real-time platform events (submissions, evaluations, leaderboard updates, validator joins, task claims).
Models
crates/platform-server/src/models/mod.rs
Complete data model layer: validators, submissions, evaluations, leaderboards, tasks, challenges, auth sessions, WebSocket events with multiple payload types, and configuration structures.
Dockerfile.server
crates/platform-server/Dockerfile
Multi-stage Docker build (chef, planner, builder, runtime) for server with dependency caching, stripped binary optimization, minimal runtime image with healthcheck and OCI labels.
Unified Platform Binary
bins/platform/Cargo.toml, bins/platform/src/{main,server,validator}.rs
New binary with CLI dispatching to server or validator subcommands via clap; server mode integrates with new platform-server crate; validator mode delegates to validator-node binary with constructed CLI arguments.
Challenge SDK Modernization
crates/challenge-sdk/Cargo.toml, crates/challenge-sdk/src/{lib,error,platform_client,server}.rs
SDK refactored with new centralized client/server architecture: PlatformClient for WebSocket-based challenge-to-server communication, ServerChallenge trait and ChallengeServer for server-mode operation, feature-gated HTTP server, new error variants (Connection, Auth, Config, Io), and updated public exports with deprecation markers.
Validator Integration
bins/validator-node/{Cargo.toml,src/{lib,main}.rs}
Validator-node now integrates with platform server: added PlatformServerClient to fetch challenge weights/health, container broker initialization with JWT security, WebSocket communication to broker, and weight submission to centralized server. Enhanced CLI with platform-server, broker-port, and docker-challenges options.
Secure Container Runtime
crates/secure-container-runtime/{Cargo.toml,src/{lib,types,ws_transport.rs}, crates/secure-container-runtime/src/bin/container-broker.rs
New WebSocket transport module (run_ws_server, WsConfig, WsClaims, generate_token) supporting JWT authentication and per-challenge connection limits. Container broker binary enhanced with optional WebSocket server (--ws-port, --ws-only) alongside Unix socket, environment-based configuration (BROKER_WS_PORT, BROKER_JWT_SECRET), and dev/strict security modes.
Challenge Orchestrator Updates
crates/challenge-orchestrator/src/{docker,evaluator,lib}.rs
Docker containers now receive broker integration (CONTAINER_BROKER_WS_URL, CONTAINER_BROKER_JWT). Evaluator refactored to generic JSON proxy: removed typed EvaluateRequest/Response, added evaluate_generic and proxy_request for flexible per-challenge routing. ChallengeInfo extended with mechanism_id, emission_weight, tasks_count. Health check now returns boolean.
CLI Tools
bins/csudo/src/main.rs
Minor refactoring of Monitor command from nested match to flatter structure; preserves functional behavior for challenges, validators, logs, and health subcommands.
Docker & Compose
Dockerfile, Dockerfile.full, docker-compose.server.yml, docker-compose.yml, docker/Dockerfile.{challenge,validator}
Dockerfile unified for server/validator with simplified multi-stage (cacher, final-builder) and Ubuntu 24.04 base; Dockerfile.full updated to rust:1.92-bookworm. New docker-compose.server.yml for platform server service with healthcheck and networking. Removed version key from docker-compose.yml.
Configuration & CI/CD
.env.server.example, .github/workflows/ci.yml, .githooks/pre-push, Cargo.toml
New environment template for server (database, owner, URLs, logging). CI workflow now requires build/clippy/test for Docker job. Pre-push hook expanded with comprehensive clippy allowances. Workspace members added: crates/platform-server, bins/platform. Dependencies: clap enhanced with "env" feature, reqwest added.

Sequence Diagram(s)

sequenceDiagram
    participant Client as Challenge Client
    participant Server as Platform Server
    participant DB as Database
    participant Broker as Container Broker
    participant Validator as Validator Node

    rect rgb(200, 220, 255)
    Note over Client,Validator: Authentication & Registration
    Client->>Server: POST /api/v1/auth<br/>(hotkey, timestamp, signature)
    Server->>DB: verify_signature() & check owner
    DB-->>Server: valid
    Server->>DB: create session (UUID token)
    Server-->>Client: AuthResponse (token, session_id)
    end

    rect rgb(220, 200, 255)
    Note over Server,DB: Submission & Task Management
    Client->>Server: POST /api/v1/submissions<br/>(miner_hotkey, source_code)
    Server->>DB: compute_agent_hash()
    Server->>DB: create_submission()
    DB-->>Server: submission_id
    Server->>Server: broadcast SubmissionEvent
    Server-->>Client: SubmitAgentResponse (id, agent_hash)
    end

    rect rgb(220, 255, 220)
    Note over Validator,Server: Task Claim & Evaluation Workflow
    Validator->>Server: POST /api/v1/data/tasks/claim<br/>(validator_hotkey)
    Server->>DB: claim_task() atomic check
    DB-->>Server: TaskLease granted
    Server->>Server: broadcast TaskClaimedEvent
    Server-->>Validator: task_id, submission_id, data
    
    Validator->>Broker: establish WebSocket connection
    Broker->>Validator: challenge execution environment
    Validator->>Broker: submit evaluation result
    Broker-->>Validator: ack
    
    Validator->>Server: POST /api/v1/data/results<br/>(agent_hash, score, results)
    Server->>DB: create_evaluation()
    Server->>DB: update_leaderboard()
    Server->>Server: broadcast EvaluationEvent
    Server-->>Validator: evaluation_id
    end

    rect rgb(255, 240, 200)
    Note over Client,Server: Leaderboard & Network State Query
    Client->>Server: GET /api/v1/leaderboard?limit=100
    Server->>DB: get_leaderboard(100)
    DB-->>Server: Vec\<LeaderboardEntry\>
    Server-->>Client: JSON array (rank, agent_hash, score)
    
    Client->>Server: GET /api/v1/challenges/network-state
    Server->>DB: multi-query (epoch, stake, validators)
    DB-->>Server: aggregated state
    Server-->>Client: NetworkStateEvent JSON
    end
Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes


🐰 A platform grand, with server's hand,
Orchestrates each challenge's land,
WebSockets whisper, tasks align,
Where validators and agents entwine—
Hop along this feature design! 🚀

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch dev

📜 Recent review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 616dfac and 19d4298.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (59)
  • .env.server.example
  • .githooks/pre-push
  • .github/workflows/ci.yml
  • Cargo.toml
  • Dockerfile
  • Dockerfile.full
  • Dockerfile.server
  • bins/csudo/src/main.rs
  • bins/platform/Cargo.toml
  • bins/platform/src/main.rs
  • bins/platform/src/server.rs
  • bins/platform/src/validator.rs
  • bins/validator-node/Cargo.toml
  • bins/validator-node/src/lib.rs
  • bins/validator-node/src/main.rs
  • crates/challenge-orchestrator/src/docker.rs
  • crates/challenge-orchestrator/src/evaluator.rs
  • crates/challenge-orchestrator/src/lib.rs
  • crates/challenge-sdk/Cargo.toml
  • crates/challenge-sdk/src/error.rs
  • crates/challenge-sdk/src/lib.rs
  • crates/challenge-sdk/src/platform_client.rs
  • crates/challenge-sdk/src/server.rs
  • crates/core/src/schema_guard.rs
  • crates/network/src/node.rs
  • crates/platform-server/Cargo.toml
  • crates/platform-server/Dockerfile
  • crates/platform-server/src/api/auth.rs
  • crates/platform-server/src/api/challenges.rs
  • crates/platform-server/src/api/evaluations.rs
  • crates/platform-server/src/api/leaderboard.rs
  • crates/platform-server/src/api/mod.rs
  • crates/platform-server/src/api/submissions.rs
  • crates/platform-server/src/api/validators.rs
  • crates/platform-server/src/challenge_proxy.rs
  • crates/platform-server/src/data_api.rs
  • crates/platform-server/src/db/mod.rs
  • crates/platform-server/src/db/queries.rs
  • crates/platform-server/src/db/schema.rs
  • crates/platform-server/src/lib.rs
  • crates/platform-server/src/main.rs
  • crates/platform-server/src/models/mod.rs
  • crates/platform-server/src/observability.rs
  • crates/platform-server/src/orchestration.rs
  • crates/platform-server/src/rule_engine.rs
  • crates/platform-server/src/state.rs
  • crates/platform-server/src/websocket/events.rs
  • crates/platform-server/src/websocket/handler.rs
  • crates/platform-server/src/websocket/mod.rs
  • crates/rpc-server/src/jsonrpc.rs
  • crates/secure-container-runtime/Cargo.toml
  • crates/secure-container-runtime/src/bin/container-broker.rs
  • crates/secure-container-runtime/src/lib.rs
  • crates/secure-container-runtime/src/types.rs
  • crates/secure-container-runtime/src/ws_transport.rs
  • docker-compose.server.yml
  • docker-compose.yml
  • docker/Dockerfile.challenge
  • docker/Dockerfile.validator

Comment @coderabbitai help to get the list of available commands and usage tips.

echobt added a commit that referenced this pull request Feb 21, 2026
* feat(p2p-consensus): add bootstrap nodes config and peer count methods

- Add DEFAULT_BOOTSTRAP_NODES constant for production bootstrap peers
- Update production() config to use DEFAULT_BOOTSTRAP_NODES
- Export DEFAULT_BOOTSTRAP_NODES from lib.rs
- Add len() and is_empty() methods to PeerMapping
- Add connected_peer_count() and has_min_peers() to P2PNetwork

* feat(p2p): add Bittensor block linking to ChainState

- Add bittensor_block and bittensor_block_hash fields to ChainState
- Add link_to_bittensor_block() and linked_block() methods
- Update Default impl with zero-initialized fields
- Link state to Bittensor block on NewBlock events

* feat(challenge-sdk): add P2P submission storage and evaluation status support

- Add StoreSubmission, RequestEvaluationStatus, EvaluationStatusResponse message types
- Add ValidatorEvaluationResult struct for tracking validator evaluations
- Add store_submission() method to P2PChallengeClient for distributed storage
- Add get_evaluation_status() method to query submission evaluation progress
- Update decentralized runner to handle new message types
- Add comprehensive tests for new message types and methods
- Export ValidatorEvaluationResult from lib.rs and prelude

* refactor: make centralized server optional, add deprecation notices for P2P migration

* docs: add P2P decentralized mode documentation

* feat: add Docker configs for decentralized P2P mode

* style: format code with cargo fmt
echobt added a commit that referenced this pull request Feb 21, 2026
…tence (#2)

* feat: add challenges directory structure and workspace configuration

* feat: add challenge-registry crate for challenge lifecycle management

Create new platform-challenge-registry crate with:
- Challenge discovery and registration
- Version management (semver-based)
- Lifecycle state machine (registered/starting/running/stopping/stopped)
- Health monitoring with configurable checks
- State persistence and hot-reload support
- Migration planning for version upgrades

Modules:
- registry: Main registry with CRUD operations
- lifecycle: State machine for challenge states
- health: Health monitoring and status tracking
- state: State snapshots for hot-reload
- discovery: Challenge discovery from various sources
- migration: Version migration planning
- version: Semantic versioning support
- error: Registry-specific error types

* feat(core): add checkpoint system for state persistence

* feat: add restoration system for checkpoint recovery

* feat(rpc-server): add health check endpoints for rolling updates

* docs: add challenge integration guide

* test: add integration tests for checkpoint and restoration system

* feat: add graceful shutdown with checkpoint persistence

- Add ShutdownHandler struct for checkpoint management
- Create periodic checkpoints every 5 minutes
- Save final checkpoint on graceful shutdown (Ctrl+C)
- Persist evaluation state for hot-reload recovery

This enables validators to update without losing evaluation progress.
echobt added a commit that referenced this pull request Apr 21, 2026
echobt added a commit that referenced this pull request Apr 21, 2026
* feat(p2p-consensus): add bootstrap nodes config and peer count methods

- Add DEFAULT_BOOTSTRAP_NODES constant for production bootstrap peers
- Update production() config to use DEFAULT_BOOTSTRAP_NODES
- Export DEFAULT_BOOTSTRAP_NODES from lib.rs
- Add len() and is_empty() methods to PeerMapping
- Add connected_peer_count() and has_min_peers() to P2PNetwork

* feat(p2p): add Bittensor block linking to ChainState

- Add bittensor_block and bittensor_block_hash fields to ChainState
- Add link_to_bittensor_block() and linked_block() methods
- Update Default impl with zero-initialized fields
- Link state to Bittensor block on NewBlock events

* feat(challenge-sdk): add P2P submission storage and evaluation status support

- Add StoreSubmission, RequestEvaluationStatus, EvaluationStatusResponse message types
- Add ValidatorEvaluationResult struct for tracking validator evaluations
- Add store_submission() method to P2PChallengeClient for distributed storage
- Add get_evaluation_status() method to query submission evaluation progress
- Update decentralized runner to handle new message types
- Add comprehensive tests for new message types and methods
- Export ValidatorEvaluationResult from lib.rs and prelude

* refactor: make centralized server optional, add deprecation notices for P2P migration

* docs: add P2P decentralized mode documentation

* feat: add Docker configs for decentralized P2P mode

* style: format code with cargo fmt
echobt added a commit that referenced this pull request Apr 21, 2026
…tence (#2)

* feat: add challenges directory structure and workspace configuration

* feat: add challenge-registry crate for challenge lifecycle management

Create new platform-challenge-registry crate with:
- Challenge discovery and registration
- Version management (semver-based)
- Lifecycle state machine (registered/starting/running/stopping/stopped)
- Health monitoring with configurable checks
- State persistence and hot-reload support
- Migration planning for version upgrades

Modules:
- registry: Main registry with CRUD operations
- lifecycle: State machine for challenge states
- health: Health monitoring and status tracking
- state: State snapshots for hot-reload
- discovery: Challenge discovery from various sources
- migration: Version migration planning
- version: Semantic versioning support
- error: Registry-specific error types

* feat(core): add checkpoint system for state persistence

* feat: add restoration system for checkpoint recovery

* feat(rpc-server): add health check endpoints for rolling updates

* docs: add challenge integration guide

* test: add integration tests for checkpoint and restoration system

* feat: add graceful shutdown with checkpoint persistence

- Add ShutdownHandler struct for checkpoint management
- Create periodic checkpoints every 5 minutes
- Save final checkpoint on graceful shutdown (Ctrl+C)
- Persist evaluation state for hot-reload recovery

This enables validators to update without losing evaluation progress.
echobt added a commit that referenced this pull request Jun 19, 2026
FIX #1: encode_dir_archive now archives each top-level child individually
instead of adding the source as a root '.' member. A non-root eval uid
extracting into a root-owned tmpfs-mode=1777 mount could not chmod/utime the
'.' root, so tar exited 2 under the bootstrap's 'set -e' and aborted before
the wrapped command ran (workspace READ + artifact WRITE both failed).

FIX #2: swarm_backend.run() applied _cap_log(64KB) to the full cross-node
stdout, silently truncating drain sections larger than the cap so the executor
could not restore a drained archive (e.g. a checkpoint). Now cap only the
human-readable remainder and re-append the drain sections uncapped. The frozen
BrokerRunResponse schema is unchanged (no new field).

Adds non-root-into-1777-tmpfs and >64KB-through-_cap_log regression tests.
cursor Bot pushed a commit that referenced this pull request Aug 31, 2026
CortexLM/relearn PR #2 published the image that implements the harvest
contract, so the eval half of live scoring is now pinned:

  eval_image_digest = sha256:303c6357…  (was empty)
  relearn_git_sha   = 82e21442…         (was 6e952d1e)

Only those two values change. base_model, teacher_model, teacher_nvfp4,
teacher_backend, eval_image, relearn_git, holdout_commitment, holdout_size,
and public_ids are byte-for-byte unchanged.

`committed_pin_cannot_rent_or_score_live_yet` existed to make this moment
deliberate; it is replaced by `committed_pin_allows_live_rent` plus
`eval_image_is_digest_only`, which pins the exact digest and asserts the tag
never migrates into `eval_image`.

Pinning the image is not the whole job, and the docs now say so: a live host
still needs the harvest wired and a champion baseline recorded, each with its
own 503 and boot-log line. `remote-deploy.sh` warns when a master host is
missing the relearn holdout or the baseline, since with the image pinned
those two files are the difference between scoring and 503 on every
submission.

Note recorded in COMPLETENESS.md: PR #2 is open, not merged, so the pinned
SHA is not yet reachable from that repo's default branch. Re-pin to the merge
commit when it lands.

Co-authored-by: Mathis <echobt@users.noreply.github.com>
echobt added a commit that referenced this pull request Sep 3, 2026
…ge (#202)

* feat(relearn): one-challenge subnet v0 eval loop

Retire Design and Prism as live products. Wire Relearn HTTP submit,
displacement scoring, digest-freeze holdout, and operator promote.
Keep Lium/receipt/paired-test rails. Pin CortexLM/relearn (seed in-tree
until org write can create the public repo).

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(site): satisfy clippy assigning-clones and match-same-arms

Workspace clippy -D warnings failed on Relearn arena mapping.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* chore(relearn): pin CortexLM/relearn git sha

* docs(relearn): drop anti-TEE slogans from user-facing copy

Keep miners-pay-Lium as the trust line. Leave holdout-after-freeze
in the eval loop internals, not README/miner Trust rows.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* docs(relearn): split miner and validator one-pagers

README is what Cortex is plus two links. Validators get a one-screen
role; Relearn miners get submit, Lium, promote-wait, and a pointer.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* docs(relearn): keep miner and validator pages together

Both audience pages live under docs/external-miner/. README still
has two links only.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn): env-only teacher http api

Teacher URL, model, and key come from RELEARN_TEACHER_* only.
Missing URL or key skips to sim. No baked host. Miners pay Lium.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(bounty): add bounty challenge beside relearn

Wire a second live challenge (id bounty) on the Relearn PR branch:
pair a Bittensor hotkey to a dedicated Cortex Chat account, file
bug reports, and score by precision vs the previous champion.
Default emission is relearn 7000 / bounty 3000. Chat inject stays
env-only; no teacher hosts or secrets in git.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(bounty): consume backend public api

Cortex reads CortexLM/backend GET /v1/bounty/public/* for scoring.
It does not serve a public leaderboard. Unset URL skips (CI).

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn-t2i): add Relearn T2I challenge on Cosmos3 + Q-Judger

Pin nvidia/Cosmos3-Super-Text2Image (OpenMDW 1.1, verified card) as the
generator seed miners fine-tune, and Q-Judger (Qwen/Qwen-Image-Bench,
Apache-2.0) as the only judge. Flux-family bases are refused outright.

Eval prompts are frozen in config/relearn-t2i-pin.toml so no miner brings
its own upsampler to the scored split, and every miner generates the same
prompt ids at the same derived seeds. The holdout split is present in git
only as a commitment; records come from an operator file and are verified
at boot, so a wrong file refuses submissions instead of scoring the public
split. Gates: paired displacement on the holdout, per-L1-pillar regression
epsilon, seed replay, agentic faithfulness agreement, contamination, and a
judge N/A ceiling.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn-mm): add Relearn Multimodal challenge beside Relearn LLM

Miners attach a permissively licensed vision encoder plus a projector to
the champion Relearn LLM. Pin google/siglip2-so400m-patch14-384 (verified
Apache-2.0) and restrict miner-supplied encoders to OSI-permissive terms.

Two gates, both mandatory. Gate 1 reruns the existing Relearn text holdout
on the submitted LM with vision ignored; a drop past epsilon zeroes the
submission no matter how good the vision numbers are, and an encoder-only
submission must additionally hash-match the champion LM. Gate 2 is a frozen
image holdout (captioning, VQA, OCR, spatial relations - not ImageNet or
COCO test) plus agentic image-tool traces, each replayed with the pixels
shuffled so a model that ignores the image cannot pass.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(challenges): split emission four ways and wire the new services

Retune challenges.toml to relearn 4000 / relearn-t2i 1500 / relearn-mm 1500
/ bounty 3000 with a dedicated public key per challenge, re-signed under a
fresh throwaway owner key (the previous owner secret is off-git by design).

Wire relearn-t2i-challenge on :8097 and relearn-mm-challenge on :8098
through the Dockerfile, compose matrix, env examples, GHCR image lanes,
remote-deploy, and local-e2e. Local smoke materializes the T2I holdout from
the documented dev salt; without it the service still answers /health and
503s submissions, which is the intended fail-closed state.

Docs: README lists all four challenges, docs/RELEARN-T2I.md and
docs/RELEARN-MM.md carry the control-plane contracts, and the miner guides
state that Flux is rejected, Q-Judger is the only T2I judge, and encoder
licenses must be OSI-permissive. external-docs-check now pins those claims.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(site): list Relearn T2I and Relearn Multimodal as arenas

The site matched emission shares by slug and only knew relearn, so after
the four-way split the marketing surface would have shown one challenge's
4000 bps as the whole subnet. Add both arena slugs (wire form equals the
trust-root challenge id), frames, and status fetches, and drop the retired
design/prism arena list that both landing and metrics were still feeding.

A down challenge backend leaves the static frame in place rather than
dropping the arena, so the emission column still accounts for every
challenge. Added a regression test that every live arena slug resolves to a
non-zero trust-root share and that the shares sum to 10000.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn): fail-closed holdout and anti-overfit gates

Pin a holdout commitment on the live relearn path, load records only
from RELEARN_HOLDOUT_FILE, and reject submissions when the file is
missing or does not match. Add contamination, public-gap, vision
pixel-shuffle, and off-lattice canary-regression gates.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn): pin 27b and glm flash nvfp4 on #199

Overlay locked Hugging Face ids on Testeur's holdout gates. Keep
public_ids, holdout_commitment, and holdout_size. vLLM serves from
RELEARN_TEACHER_LOCAL_DIR, never the repo id.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* docs(repo): complete greptile review gate on the overlay

Template, .greptile rules, and a short AGENTS / miner-docs note. Covers
fail-closed, holdout-off-git, no Modal/secrets, no Flux, OSI encoders,
and BASE_* / domain-tag freeze. Additive; scoring gates unchanged.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn): pin teacher to incoai glm-5.3 nvfp4

Relock teacher_nvfp4 to incoai/GLM-5.3-NVFP4 (full GLM-5.3). Wire id
glm-5.3, not flash. Serve from RELEARN_TEACHER_LOCAL_DIR. Keep #199
holdout_commitment, public_ids, and holdout_size.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): fail closed instead of scoring submissions in sim

Live submit could not produce a real verdict. `eval_after_freeze` always
called `sim_slice_scores`, so a host with an empty `eval_image_digest`
answered `201 CREATED` with simulated numbers, and an empty miner manifest
skipped the contamination gate entirely.

Eval backend is now explicit, mirroring relearn-mm:

- `EvalBackend::{Lium, Sim}` + `force_sim()` / `resolve_eval_backend()`.
  Sim is selected only by `RELEARN_FORCE_SIM`, never as a fallback.
- `eval_after_freeze` takes the pin and the backend. On `Lium` it returns
  `EvalImageUnpinned` without a `sha256:` pin and `LiveHarvestUnavailable`
  with one; both map to `503`, so no sim number can reach the lattice.
- `/v1/status` publishes `eval_backend`, `force_sim`, and `can_score`; the
  submit row publishes `eval_backend`.
- The sim base champion is only seeded on a host that resolved `Sim`.

Contamination now carries its evidence:

- `ContaminationEvidence` records the declared id / image-hash / dataset
  counts next to the hits, and `contamination_evidence` replaces
  `contaminated_fingerprints`.
- An undeclared manifest is `GateFail::ContaminationEvidenceMissing`, the
  same fail-closed shape as `PublicEvidenceMissing`.

The offline harness also could not express any promotion: perturbed and
holdout were drawn from independent salts, so the drop was ~0.17 against a
0.05 ceiling and every sim run failed `Perturbation`. Sim slices are now
derived from one skill level per artifact (`sim_artifact_skill`), with the
retention slices offset from the holdout draw, so `RELEARN_FORCE_SIM=1`
can reach `awaiting_admin` and the gate tests compare like with like.
Pin ids and `judge_challenger` thresholds are untouched.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* docs(relearn): say the committed holdout commitment is not the live seal

The pin and COMPLETENESS named the local dev salt next to the committed
`holdout_commitment`, which reads as the production seal. Both now describe
it as the CI / local commitment, point at `config/CEREMONY.md`, and keep the
requirement that production rotate salt *and* catalog before re-signing. The
salt literal stays only in `deploy/scripts/local-e2e.sh`, where it is the
local default; `committed_pin.rs` asserts it is absent from the pin.

Also documents the two fail-closed behaviours this branch adds: the 503 a
host returns without a `sha256:` eval-image pin (with `eval_backend` /
`can_score` on `/v1/status`), and that an undeclared miner `manifest` fails
the contamination gate. Miner example manifest now declares real fields
instead of three empty arrays, which would now be rejected.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): let RELEARN_FORCE_SIM=1 boot instead of failing to parse

Found on the live HTTP run: `--force-sim` bound `env = "RELEARN_FORCE_SIM"`
to a clap `bool`, so the documented `RELEARN_FORCE_SIM=1` was rejected with
`invalid value '1' for '--force-sim'` and the service exited at boot. Only
compose's `"true"` / `"false"` strings worked, which is why it went unnoticed.

Drops the env binding, matching `relearn-mm-challenge` and
`relearn-t2i-challenge`: the value is read by `resolve_eval_backend`, which
accepts `1` / `true` / `yes`. A test pins every documented spelling.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn-mm): follow the locked LLM base id instead of Flash-Next

`cargo test --workspace` is red on the parent branch: #200 moved
`relearn_challenge_task::BASE_MODEL_ID` to `Qwen/Qwen3.8-27B`, but
`relearn-mm-task` re-exports that constant as `LM_BASE_MODEL_ID` while
`config/relearn-mm-pin.toml` and four tests still spelled out
`Qwen/Qwen3.8-Flash-Next`. Eight tests failed, including
`RelearnMmPin::validate` on the committed pin.

The encoder attaches to the Relearn champion's LM, so the MM pin has to
carry the live LLM base — this aligns it rather than choosing a new model.
`lm_side_tracks_the_relearn_champion_base` now asserts against
`relearn_challenge_task::BASE_MODEL_ID` so the two cannot drift again.

Also updates the miner-facing seed harness default
(`docs/external-miner/relearn-seed/eval/harness/eval.py`), which still
pointed `RELEARN_BASE_MODEL` at Flash-Next.

Drop this commit if #200 fixes the fallout upstream.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* docs(relearn): tell seed miners to declare their training manifest

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): record a live champion baseline and stop banking refusals

Two holes the live replay of d7c5920 found behind the digest 503.

**A live host could never record a champion baseline.** Making the sim
baseline sim-only was right — a live challenger judged against simulated
champion scores is not a comparison — but it left the live path with no
baseline at all. The moment a `sha256:` eval image is pinned, submit answers
`no champion baseline recorded` before contamination, public–holdout gap, or
pixel-shuffle can run, so every gate this branch exists to protect is dead
code on the only host that matters.

Boot now records the baseline with the scorer the host will actually use:

- `LiveScorer` is the eval-image harvest seam. `eval_after_freeze` takes it
  and refuses (`LiveHarvestUnavailable`) rather than substituting sim.
- `BaselineMeasurement` (`RELEARN_BASE_CHAMPION_FILE`) is the operator's
  recorded measurement of the base model. `verify` binds it to the pin's
  `eval_image_digest` and `holdout_commitment` and refuses a measurement
  missing a series the gates read, so a junk baseline fails at boot instead
  of silently rejecting every challenger.
- `boot_base_champion` picks the recorded measurement, else the wired
  harvest, else refuses. Sim numbers are not a candidate on a live host.

**Refusals banked rows.** The row was inserted before the eval, so every
503 left an `evaluating` row with no scores that appears on no operator
surface — spammable. Nothing is persisted until scoring produced a verdict,
and the row is inserted once in its final state instead of insert-then-patch.

Check order now puts the root cause first and spends nothing it cannot use:
holdout unseal, then `scoring_readiness` (digest pin + harvest), then the
baseline, then the eval. An unpinned digest still reports the pin, not the
baseline, and a live eval never runs for a submission the host could not
judge — that eval is the miner's Lium spend.

`/v1/status` gains `live_harvest_wired` and `champion_baseline_recorded`
next to `can_score`, since "cannot score" now has three distinct causes.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* docs(relearn): document the champion baseline the live gates need

Adds the operator surface for the live baseline: what
`RELEARN_BASE_CHAMPION_FILE` is, the JSON the eval image has to emit, and why
`public` / `general_canary` cannot be empty (a champion the gates cannot read
rejects every challenger for a reason the miner cannot act on).

Also records the two new `/v1/status` fields (`live_harvest_wired`,
`champion_baseline_recorded`) and the three distinct causes of
`can_score: false` in the miner troubleshoot table, notes that a refusal
persists no row, and adds the operator checklist item plus COMPLETENESS gaps
for the unwired live harvest.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* test(prism-verda): serialize the two PRISM_VERDA_COMPUTE tests

`pick_honors_compute_override` sets a process-wide env var that
`pick_compute` reads, so it raced `pick_b200_then_fallback`: whichever ran
while the override was set picked `L40S` and failed. It flaked in two of
three `cargo test --workspace` runs and passed in isolation, which made it
hard to tell whether an unrelated PR was green.

Test-only mutex around both. Pre-existing and unrelated to this branch —
drop this commit if it lands elsewhere.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* test(prism-verda): use the method reference clippy asks for

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn): wire the live harvest on the Lium path

`live_scorer` was hardcoded `None`, so a digest-pinned host still refused
with `live_harvest_wired: false` and the champion could only come from
`RELEARN_BASE_CHAMPION_FILE`. This adds the control-plane client.

The scoring code is not here: it ships inside `eval_image` from
CortexLM/relearn. `relearn-lium-harvest` is the client for it.

- `LiveScorer` is now async (`#[async_trait]`, as `EvalJobBackend` already
  is), so `eval_after_freeze` and `boot_base_champion` are too. The bin
  builds its runtime before recording the baseline.
- `RelearnEvalMetrics` is the image contract: the `BaselineMeasurement`
  envelope plus run identity, so the operator's baseline file is literally
  the image's output for the base model — one format, not two. `verify`
  binds schema, `submission_digest`, and `artifact_digest`, so a pod cannot
  answer with another artifact's numbers or replay an earlier run.
- `LiumHarvest` owns the lifecycle: boot the digest-pinned image, deliver
  the request, harvest, then terminate and require verified teardown before
  any score is accepted. An orphan pod outranks the run result because it
  keeps spending the miner's money.
- `LiumEvalPod` is the Lium/SSH transport. Run inputs travel in
  `request.json` over stdin, never interpolated into the remote command, so
  a crafted digest cannot be shell injection on the pod. The workdir is
  scrubbed after the run; termination is the real guarantee.
- `build_live_scorer` wires it on `EvalBackend::Lium` only, from
  `LIUM_API_KEY`. Sim scores in-process and never gets a pod. No key means
  no harvest and the host refuses.

Nothing in this crate can compute a score. A pod that returns no
`RELEARN_EVAL_OK`, no metrics document, or a document bound to another run
is an error, not a fallback — the tests feed metrics-document fixtures
through the real parse/verify path rather than substituting sim numbers.

`prism-lium` additionally re-exports `ssh_exec{,_allow_fail,_stdin}` and
`SshExecOutput` so the client can reuse its retry/timeout helpers.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn): require the master SSH key and document the image contract

Live replay of the wiring found the spec left `ssh_public_keys` empty, so
`provision` refused with "Lium rent requires at least one SSH public key" —
the harvest could never boot a pod. A pod with no master key is also
unreachable, so the request could not be delivered and no metrics could be
read back.

- `LiumHarvest` takes the master public key(s) and registers them under
  `relearn-eval-worker`; it refuses before renting when there is none.
- The bin loads it from `LIUM_SSH_PUBLIC_KEY_FILE` (same convention as
  `prism-challenge`) and does not wire the harvest without it.
- `LIUM_API_BASE_URL` can point a staging host at a stand-in provider
  instead of spending real money.
- Provider and eval-image failures are now 503, not 500: the host cannot
  score right now, which is not the miner's mistake, and retrying is right.

Documents the image contract in `docs/RELEARN.md` — entrypoint, marker
lines, document shape, and the identity/pin checks — so `CortexLM/relearn`
has something to implement against. Also records, in RELEARN.md,
COMPLETENESS.md, and the operator checklist, that the harvest request
carries the holdout to a miner-paid pod: mitigated by the digest-pinned
image, tmp delivery, post-run scrub, and verified termination, but not
eliminated. That exposure is an owner decision, not an implementation
detail.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* deploy(relearn): pin the published eval image digest and relearn SHA

CortexLM/relearn PR #2 published the image that implements the harvest
contract, so the eval half of live scoring is now pinned:

  eval_image_digest = sha256:303c6357…  (was empty)
  relearn_git_sha   = 82e21442…         (was 6e952d1e)

Only those two values change. base_model, teacher_model, teacher_nvfp4,
teacher_backend, eval_image, relearn_git, holdout_commitment, holdout_size,
and public_ids are byte-for-byte unchanged.

`committed_pin_cannot_rent_or_score_live_yet` existed to make this moment
deliberate; it is replaced by `committed_pin_allows_live_rent` plus
`eval_image_is_digest_only`, which pins the exact digest and asserts the tag
never migrates into `eval_image`.

Pinning the image is not the whole job, and the docs now say so: a live host
still needs the harvest wired and a champion baseline recorded, each with its
own 503 and boot-log line. `remote-deploy.sh` warns when a master host is
missing the relearn holdout or the baseline, since with the image pinned
those two files are the difference between scoring and 503 on every
submission.

Note recorded in COMPLETENESS.md: PR #2 is open, not merged, so the pinned
SHA is not yet reachable from that repo's default branch. Re-pin to the merge
commit when it lands.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): stop implying the baseline file can replace the harvest

The boot warning read "submissions will 503 unless RELEARN_BASE_CHAMPION_FILE
covers the baseline and a harvest is configured", which is wrong now that the
digest is pinned: the replay showed a host with the baseline recorded and no
harvest still 503ing every submission. The baseline covers the champion only;
each submission needs its own measurement. Points at the env example instead.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* refactor(harvest): share one digest-pinned lium pod transport

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn-image): fail-closed live eval on the relearn-image id

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn-agent): add the replayed-tool-trace challenge

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(relearn-agent): pin the episode set and add the ceremony helper

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(bounty): price bugs by severity and fail closed on ingest

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* feat(challenges): relock four live ids and re-sign the trust root

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* deploy(challenges): bring up the four live services and gate mm off

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* docs(challenges): name the four live challenges and their gates

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(challenges): pin image/agent evals to relearn#3

Digest-only pins from CortexLM/relearn PR #3, harvest contracts that
speak those images (shared RELEARN_METRICS markers, recorded traces,
optional image canary), and miner/operator docs for the four live ids.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn-image): split canary gating out of judge_challenger

Keeps the function under clippy's 100-line cap after the both-empty
skip, without weakening the one-sided fail-closed rule.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn-image): name the published image-eval pin

Primary eval_image is ghcr.io/cortexlm/relearn-image-eval at the
relearn#3 digest; t2i-eval stays the same-digest alias. Teacher
ids on relearn are untouched.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): pin the eval image that prints RELEARN_EVAL_OK

Match #201 HEAD: sha256:00839671… / 822d2729. Refuse sha256:303c6357…
which never printed the marker. Image, agent, and bounty pins unchanged.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn-agent): can_score stays false until holdout loads

Status was advertising can_score on sim/live hosts with no verified
episodes while submit already 503d. Holdout is now the first readiness
check. LLM digest and image/bounty pins are untouched.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): forward teacher env and fail before rent

Port the #201 LLM harvest gate onto the harvest-pod transport: without
RELEARN_TEACHER_API_URL, can_score is false and submit 503s naming the
var, with no Lium rent. Teacher env is delivered over stdin as
teacher.env (umask 077, set -a) rather than interpolated into the
remote command. Missing-marker 503s include a redacted run.log tail.

Image/agent harvests stay env-free. LLM digest, image/agent pins, and
bounty fail-closed are unchanged.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): hoist Unready scorer stub for clippy

Move the scoring_readiness teacher-gate fixture out of the test body
so items-after-statements stays clean.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(challenges): fail-before-rent and screen junk before eval

Image and Agent harvest now refuse without a judge/teacher URL, forward
stdin teacher.env the same way LLM does, and reject contaminated or
empty-evidence manifests before renting. Bounty treats any prior
fingerprint as duplicate and refuses title==body / token-thin reports.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(challenges): clippy persist helpers and image happy-path

Pre-eval reject helpers satisfy too-many-arguments/field-reassign
lints. Image happy-path submits a declared manifest so it still
evals instead of being screened as empty evidence.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): can_score waits for champion; resolve eval PATH

Live status advertised can_score while submit 503'd without a
champion baseline. Harvest run_cmd now prefers /usr/bin/<eval>
(else command -v) so a login-less SSH PATH cannot miss the binary.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): pin eval image that ships /usr/bin/relearn-eval

Replace sha256:00839671 (exit 127 on a login-less SSH PATH) with
sha256:86240d76 from CortexLM/relearn 9998154f. Image/agent pins
unchanged. Do not pin sha256:303c6357.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): pin the CUDA scoring image CI pulled

sha256:201cc5d2 ships /usr/bin/relearn-eval (relearn d107a7c1).
Do not pin 86240d76, 00839671, or 303c6357. Image/agent pins
unchanged.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(lium): rent digest-pinned harvest templates not recipes

Harvest was setting image_digest and then renting prism-recipe-v10,
so SSH ran /usr/bin/relearn-eval on the wrong image (exit 127).
InstanceSpec now carries docker_image + startup; provision uses that
pin and refuses a recipes fallback. Probe the score binary before
staging the holdout.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(lium): harvest template is repo@digest with no startup

The live relearn-eval image already CMD ["serve"] (tini + entrypoint).
Create the Lium template with docker_image=pin@digest, no tag, no
startup, and a 12-hex name so recipes cannot be reused.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): pin GLM-5.3 judge image; can_score needs holdout

Lock the LLM eval image to sha256:cbc4bbb8 (git f3cfa69). Reject the
four earlier judge-broken prefixes. LLM and Image can_score now require
the holdout file to be loaded, matching Agent. Bounty HTTP rejects a
second miner refiling a closed fingerprint.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): drop pin _token wording; HTTP mismatch 503

Reword the GLM-5.3 pin comment so the committed-pin secret scan
does not treat max_tokens as a leaked credential. Add HTTP tests
that a commitment-mismatched holdout cannot load, so status stays
can_score false and submit is 503 on LLM, Image, and Agent.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): forward base weights; fail closed before rent

The eval image does not bake Qwen. TeacherEnv now forwards
RELEARN_BASE_MODEL_DIR, HF_HOME, HF_HUB_CACHE, and an explicit
RELEARN_ALLOW_MODEL_DOWNLOAD=1 into teacher.env over stdin.
ready() refuses to rent when neither DIR nor ALLOW is set.
Status publishes base_weights.primed + via (var name only).
ALLOW_DOWNLOAD is never defaulted. Pin stays cbc4bbb8.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): bin ready() tests require ALLOW_DOWNLOAD

A teacher/judge URL alone is no longer enough to score. Binary
readiness tests now fail closed without DIR/ALLOW, then pass with
the first-champion ALLOW_DOWNLOAD=1 path.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(lium): rent whole-host ncu b200 for 1-gpu pin

Lium NCU hosts reject split even when min_gpu_count_for_rental is 1.
Parse ncu_profiling_enabled and rent the whole host instead of posting
gpu_count=1 on the live 2× B200. A 1-GPU harvest pin still matches.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): unpin cbc4bbb8; no eval image has scored

cbc4bbb8 (f3cfa69) exited 1 on a rented pod without RELEARN_EVAL_OK: no
vLLM on the CUDA scoring image, transformers fallback, then
Qwen3VLVideoProcessor crashed for want of torchvision. An empty digest is
the fail-closed state - submissions 503 rather than renting a B200 that
cannot score. The pin now names every digest that reached a pod and
failed, and the tests assert the fail-closed state instead of a digest.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): fail closed when gate evidence is missing

Perturbation and the known-answer canaries were 'if let Some(..)': a run
whose eval document omitted the series took neither gate. Pixel shuffle
iterated the challenger's own map, so dropping a family skipped the
control. All three now fail closed, with the champion as the reference
for which vision families the holdout actually has images in.

BaselineMeasurement::verify additionally requires perturbed + canaries,
so an image that never emits them is refused at boot rather than
rejecting every challenger for a reason the miner cannot act on.

EvalError::Integrity is a 503, not a 500: an unverified teardown is this
host refusing to believe a run, not the miner's mistake.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* docs(relearn): unpinned eval image and fail-closed gate evidence

RELEARN.md, COMPLETENESS.md, and the miner-facing external-miner docs now
say the digest is empty on purpose and what the next image needs (vLLM +
torchvision), and list the new *_evidence_missing rejections so a miner
can tell a skipped gate from a failed one.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* test(relearn): cover the live path end to end through HTTP

One run walks champion recorded -> can_score true -> a clean submission
scored on the Lium backend with lattice -> every gate zeroing its own
run: contamination, undeclared manifest, public-holdout gap, general
canary, and the three that used to be skippable by omitting the series.
Each rejection is asserted against its own gate name, so failing closed
everywhere would not pass.

Second test: promoting moves the bar. A later challenger at the same
skill ties the sitting champion instead of beating the base model.

relearn-score re-exports ExampleSeries so a consumer can build the
slices whose types are already in its public API.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* fix(relearn): split judge gates for clippy; name every eval status

judge_challenger crossed the 100-line cap once the missing-evidence arms
landed, so the retention and pixel-shuffle gates are their own functions.
eval_err now matches EvalError exhaustively: a new failure mode has to be
given a status deliberately instead of inheriting a 500.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* test(relearn): committed pin refuses live scoring at runtime

Asserting the digest string is empty says the file is right; calling
scoring_readiness with the shipped pin says the host is. The reported
cause has to be the pin, not a downstream symptom.

Co-authored-by: Mathis <echobt@users.noreply.github.com>

* chore(ci): retrigger checks after retargeting to main

Co-authored-by: Mathis <echobt@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant