DEC-CA-0036: Miner-funded challenger legs with direct submissions - #235
Open
tensorlink-dev wants to merge 55 commits into
Open
DEC-CA-0036: Miner-funded challenger legs with direct submissions#235tensorlink-dev wants to merge 55 commits into
tensorlink-dev wants to merge 55 commits into
Conversation
…029 groundwork) Miner-funded compute plumbing ported from PRISM's control plane (base repo: prism-lium-payer / lium-rent-pool): keys held memory-first with 0600 TTL-file persistence for restart teardown, never logged or stored anywhere shared; rent failures classified auth / rate_limited / no_capacity / infra with the matching no-burn requeue semantics; funded submissions queue by reveal block (earliest commit is the seniority claim) with a one-live-entry-per-hotkey rule and an elastic-cadence sizing helper (rounds_needed, k-cap preserved). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
…r HTTP surface POST /v1/fund carries X-Lium-Api-Key + X-Miner-Hotkey + X-Commit-Ref with an sr25519 signature over a versioned canonical message; fail-closed intake with PRISM-compatible error codes (missing_lium_api_key, bad_signature, not_revealed, …). Keys go to the vault only after the queue accepts, are refreshed on idempotent re-funds, and are forgotten on withdraw; the public /v1/queue feed carries no key material. Reveal eligibility resolves against the chain's revealed commitments (latest matching reveal per hotkey), with a --trust-refs dev bypass that never grants seniority. No balance pre-check by design: a key is validated by use and an underfunded one surfaces as a classified rent failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
… funded rent/teardown/reconcile LiumProvider grows an api_key field (repr-hidden): when set, every CLI call runs with that key layered into the CHILD env only — argv and this process's environ never carry it, and stderr is scrubbed before it can reach a log. PodInstance records payer_hotkey (drop-when-default, so operator-account ledgers serialise byte-identically); the orphan-reaper name gate accepts the new funded stage. provision/funded.py rents one pod per challenger on the miner's own key (PRISM's per-submission backend shape), classifies failures through the funding taxonomy (only infra burns an attempt; a rent is never auto-retried here — each attempt spends the miner's budget), and routes teardown/reconcile through the payer vault, since a pod on a miner's account is invisible to the operator's key. Keyless leftovers are returned loudly, never swallowed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
… elastic-cadence floor funded_mode off/shadow/required ([round], digest-inert; shipped off). Required mode drains the funded queue into the field at most finalist_cap at a time, earliest reveal block first, ref-matched to the eligible reveal — which trips the heat's fits-the-cap fast path, so no GPU screen runs and every funded entrant duels; unfunded reveals wait unburned. Entries mark in_round at selection, recover to queued on a torn round (the burn-after-heat rule applied to funding), and settle done at the same moment the submission burn fires. skip_unfunded_rounds skips an unfunded boundary entirely — no king leg, no pods, no manifest; validators poll manifests and never schedule, so a skipped boundary is consensus-invisible (the elastic-cadence floor; the ceiling is the epoch grid itself via the existing scheduled epoch_blocks switch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
Funds the miner's revealed submission with their own Lium API key: the key is read from the environment only (never argv), refused over plain http to a non-local intake, and sent as the X-Lium-Api-Key header of one POST signed by the hotkey over the versioned canonical message (action-bound, so a captured fund request can never replay as a withdraw). Wallet is used for signing only — no chain connection. End-to-end test pins the CLI and the intake to the same canonical bytes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
…adence rounds (proposed) The design node for the PRISM (sn100) port, the rollout runbook, and the CLAUDE.md pointer. Records the owner's cost allocation (king + confirmation legs stay operator-funded; a provisional dethrone re-trains on the operator's account before it crowns), the consequences owned (no shared credentials on payer-account pods; DEC-CA-0016 tenure re-denomination gates arming; dedup stays pre-duel; no balance pre-check by design), and the arming order: intake → shadow → required-on-testnet + the scheduled epoch_blocks grid change → mainnet release-then-activate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
…ber miner-funded node to DEC-CA-0036 Main claimed DEC-CA-0029..0035 (fork-anneal, bench-anneal, GP budget, even mix, EMA bundle, init floor, toto2 constants) while this branch was in flight — the same numbering collision the 2026-08-20 renumbering fixed once before. The miner-funded elastic-rounds node moves to DEC-CA-0036; every reference in code, config comments, docs, and the CLAUDE.md pointer moves with it. Also folds in main's validate_corpus_target_points beside validate_funded_mode (both sides added validators at the same site). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
…scoped reap, hardened intake/CLI Ten findings from the 2026-08-29 high-effort review of the DEC-CA-0036 diff: - FundedQueue is now flock-serialised reload-before-mutate (sibling .lock file): the long-lived intake instance can no longer resurrect entries the trainer settled, and the ThreadingHTTPServer handlers stop racing on the shared tmp file. Reads always hit the current file, never a snapshot. - teardown_funded verifies by RE-LISTING the payer's pods after terminate — LiumProvider.terminate's idempotent rm-failure swallow turned a revoked key's 401 into a silent leak on the miner's account; still-live and crashed teardowns are returned loudly, and one failure no longer aborts the sweep for other payers. - reconcile_funded is scoped to payer_pod_pattern (cascade-*-funded-<that payer's slug>): the generic provisioner-name match could kill a miner's own unrelated cascade pods on their account (the 2026-07-13 over-reap class, aimed at hardware the operator never rented). - Dead entries can no longer hold the elastic-cadence floor open: ref_mismatch and burned go terminal at selection, funding_expired kills queued entries outliving the key TTL — no perpetual king-leg drain from one unenterable fund. Shadow mode is strictly read-only (no recover/expire). - cascade fund parses the intake hostname instead of substring-matching (http://localhost.evil.example no longer passes), and decodes non-JSON error bodies without a traceback. cascade-intake requires --queue-path (a CWD default silently split-brained the queue against the trainer) and returns clean 500s on handler errors. - select_field is the trainer filter's ordering and rounds_needed feeds the skip-path telemetry, so neither helper is dead code drifting from an inline copy. Full suite: 1692 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
… publication DEC-CA-0036 second half — nothing miner-hosted, losers private forever: - SubmissionStore: content-addressed private ZIPs (0600, earliest uploader owns a digest), hostile-zip-safe extraction, per-dispatch staging that can only ever ship ONE entry's ZIP (a funded pod's payer can read the box). - Vault refs ride the EXISTING hub repo@digest grammar under the reserved vault/direct namespace, so the chain payload is byte-ordinary: deployed validators parse it unchanged (they never fetch generator code), participant sets cannot fork, and only fetch_from_hub grows a branch (store → staged pod ZIP → published champion object). - Intake POST /v1/submit: ZIP body with the signature bound to its sha256, optional X-Lium-Api-Key in the SAME request — the entry parks pending_reveal and auto-queues with the real chain reveal block when it resolves (lazy sweep on every request). Fail-closed: digest_mismatch / digest_owned / zip_too_large (rejected on declared length) / submissions_disabled. - ChampionPublisher (the sn100 top-model pattern): champions/<digest>.zip + index public-read on the manifest bucket per [round] champion_publish (crown / delay / dethrone); a deposed vault king always reveals at hand-off; losers never publish. Trainer hook per round, best-effort. - Trainer ownership guard drops a copied digest at field entry (byte-copies arriving as uploads still die at the DEC-CA-0008 dedup screen). - CLI: cascade submit (verify → deterministic ZIP → one signed POST → timed-reveal commit; $LIUM_API_KEY funds in the same request) and vault-aware cascade fetch king (published champions resolve anonymously, clear message while the policy keeps the live king private). All inert: submission_vault_dir = "" and champion_publish = "off" shipped. Accepted trade recorded in the node: non-champion entries are no longer third-party re-derivable — PRISM's posture, and the point. Full suite: 1710 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
…ake, provisioner, CLI Second high-effort review of the full DEC-CA-0036 diff (correctness + adversarial security). Ten correctness findings fixed; the security pass then found no exploitable vulnerabilities. - champion.py: a deposed vault king whose hand-off publish fails now lands on a persistent 'unpublished' backlog retried every round — one bucket 500 can no longer erase a reign's audit trail (the 'history always re-derivable' contract the module promised but broke by overwriting state regardless of outcome). - trainer/loop.py: _submission_store exports $CASCADE_VAULT_DIR from [round] submission_vault_dir, so the fetch path (dedup screen, king fetch, local train) can actually resolve vault refs — without it a fully configured store was invisible to every fetch, silently skipping direct submissions past the DEC-CA-0008 content screen. - queue.py: /v1/queue redacts pending_reveal entries to a bare count — listing (hotkey, ref) before the timed chain reveal leaked the sealed next-round field. Entry TTL is now configurable (funded_entry_ttl_hours, tracks --ttl-hours) and expiry keys off last_active not funded_at, so a sold-out entry cycling through no_capacity requeues waits the whole drought while a truly idle one still dies. - store.py: SubmissionStore.put does its ownership check-then-write under a flock — concurrent identical uploads from two hotkeys can no longer let the last meta-write steal 'earliest upload owns the content'. - miner/cli.py: zip_repo_bytes filters to ALLOW_PATTERNS (drops .git, __pycache__, non-code) — a raw rglob shipped commit history + committer emails + any committed secret into the operator store and, on a throne, PUBLIC champions/; matching the Hub filter also keeps both channels' trees dedup-comparable. - funded.py: rent_funded_pod's failure cleanup uses terminate_verified (re-list after rm) and returns the leaked pod id — a revoked key mid-launch no longer leaks a pod silently past the taxonomy. - intake.py: submit_gate runs identity+signature on HEADERS before the HTTP layer buffers the upload body; submit reports funding='blocked-by-existing- entry' truthfully instead of falsely promising pending_reveal when a live entry for another ref already holds the slot. - trainer/loop.py: mark_in_round re-checks the ref inside the lock — an intake ref-replace landing in the select→mark window no longer gets its new entry consumed by a round that trained the old ref (settled 'done' unpaid-for). Full suite: 1719 passed, 21 skipped. Security review: no findings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
… HEAD A focused review of the direct-submissions commit surfaced seven issues; four were already fixed by round 2, one (pending_reveal expiry) is correct-by-design given the entry-TTL/key-TTL coupling (documented), and TWO survived into HEAD: - store.py: extract_zip_safely wraps mkdir/write_bytes in try/except OSError to StorageError, so a hostile ZIP naming 'a' (file) then 'a/b' (path under it) returns a clean 400 bad_zip at the intake instead of an uncaught OSError 500. - champion.py: _publish returns False when the champions/index.json write fails after the ZIP upload — the index anonymous readers consume would otherwise stay stale forever, because returning True marked the reign published and never retried the index (the ZIP re-upload on retry is idempotent). docs: note that a submit-with-key reveal must land within the key TTL (the key lives in the vault under the same clock), so an over-delayed reveal correctly fails rather than outliving its own key. Full suite: 1721 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
…mpotency The round-3 convergence review flagged two adversarial-robustness gaps still live in HEAD: - store.py: extract_zip_safely now enforces MAX_EXTRACTED_BYTES (512MB) — a submission is generator source (a few MB), so a ~128MB deflate stream expanding to 100+GB is a bomb. Checked against the declared sizes (cheap) and the running total of bytes actually inflated (catches a lying header), so put()'s validation probe can't fill the operator disk (the box holds the eval pool + trainer wallet). - champion.py: note_king advances the reign counter only on the FIRST call per round_id — a mid-round retry calling _maybe_publish_champion twice would otherwise inflate reign_rounds and fire the 'delay' reveal a round early, exposing a live king's private code. A king change is still always processed (it may carry a hand-off); publication is idempotent. Full suite: 1723 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
… own zip-bomb guard Reviewing the round-4 commit surfaced two bugs in that very fix: - store.py: zf.read() raises zipfile.BadZipFile (a bad CRC, a truncated member, or a member whose real deflate exceeds its declared file_size) — which is NOT an OSError, so it escaped the except handler into a 500 on a hostile upload. Now caught alongside OSError → clean 400 bad_zip. infolist() is guarded the same way. - store.py: the running-total check was unreachable (CPython caps read() at file_size, so the declared-sum guard already dominates) and its comment falsely claimed it caught lying headers — a lying header actually surfaces as the BadZipFile above. Reworded to what it honestly is: a backstop on the bytes actually written, not reliant on that internal cap. The champion.py reign-idempotency change from round 4 was reviewed and confirmed correct. Full suite: 1724 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
The round-5 review showed chasing exception types stayed incomplete — zf.read() on a hostile member can also raise zlib.error (garbage deflate) and NotImplementedError (unsupported compression method), neither an OSError nor BadZipFile, so both still escaped as a 500. extract_zip_safely now treats the read as the hostile-input boundary it is: ANY failure to inflate an untrusted member becomes StorageError -> a clean 400 bad_zip. The regression test injects zlib.error / NotImplementedError / EOFError directly, so the contract is proven type-independently rather than by one hand-crafted CRC case. Two cleanups the same review flagged: dropped the dead try/except around infolist() (ZipFile.__init__ already reads the central directory and raises there) and simplified the traversal check to the single is_relative_to() conjunct (a path equal to dest is already relative-to it). Full suite: 1724 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
The round-6 review showed symptom-patching the extractor kept missing corners (three of four findings real). Rewrote extract_zip_safely as a proper hostile-input boundary rather than a growing type-list: - Streaming extraction (zf.open + 1MiB chunks) bounds RAM to one chunk regardless of a member's declared size, and enforces the cap on bytes ACTUALLY inflated — the backstop no longer leans on a CPython read()-caps- at-file_size internal. - One try per member wraps the resolve, mkdir, and streamed inflate; ANY input failure (BadZipFile / zlib.error / NotImplementedError / EOFError / OSError / odd-name ValueError) becomes a clean 400 bad_zip. A genuine resource fault (MemoryError) is re-raised so an operator OOM surfaces as a 500, not a mislabeled client error. - The client-facing message no longer echoes the caught exception — an OSError carries operator-internal vault paths that must not reach the miner. - NUL-in-name guard kept as defence-in-depth (CPython truncates at NUL before we see it, so the reviewer's 500 vector is in fact unreachable — the regression test asserts the real benign behavior, not a raise). Dead infolist() guard and a redundant traversal conjunct removed. Regression tests inject every inflate-failure class plus MemoryError and a path-leak OSError, proving the contract type-independently. Full suite: 1726 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
… faults The round-7 review found the extractor still conflated the untrusted-inflate surface with the operator-side write surface (only MemoryError carved out), and the intake picked HTTP status by substring-matching error text an attacker's member name could hijack. Both fixed structurally: - store.py raises typed SubmissionTooLarge / DigestOwned (StorageError subclasses); intake.submit dispatches HTTP status on the exception TYPE, so a bad-zip whose member is NAMED 'zip_too_large' is still 400 bad_zip, never 413. - extraction classifies OSError by errno: a full/unhealthy operator disk (ENOSPC/EDQUOT/EIO/EROFS/EMFILE/ENFILE/ENOMEM) propagates → 500, while a structural fault the hostile ZIP caused (a member path clashing with a file) converts → 400. Operator infra incidents are no longer masked as client errors; MemoryError still propagates. New tests: ENOSPC on the streamed write propagates as an operator fault, and a member named 'zip_too_large' does not steer the intake response code. Full suite: 1728 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
… as StorageError The round-8 review found two remaining gaps in the same operator-vs-input classification: - _OPERATOR_ERRNOS omitted permission faults (EACCES/EPERM) plus EFBIG/EBUSY, so a mis-permissioned vault dir was mislabeled as a client 400 bad_zip instead of surfacing as a 500. Added them — EROFS was already operator, so EACCES/EPERM belong in the same 'operator box is unhealthy' class. - Round 8's raw-OSError propagation (correct for the INTAKE path → 500) regressed the FETCH path: fetch_vault_snapshot feeds fetch_from_hub, whose contract is StorageError-on-failure and whose callers (dedup screen, promotion, heat-status) catch only StorageError. A trainer-side disk hiccup would have escaped as an uncaught OSError and crashed the round. fetch_vault_snapshot now wraps any OSError from extraction into StorageError, so the fetch path degrades gracefully while the intake path keeps its 500. Tests: EACCES propagates from extraction; an ENOSPC during a vault fetch comes back as StorageError. Full suite: 1729 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
…sage hygiene Three real bugs the round-9 review found latent in fetch_vault_snapshot: - The temp dir was keyed by PID only, while its sibling fetch_from_hub uses PID+uuid: two same-process lanes fetching one digest into one dest collided on a single tmp tree, and each lane's finally-rmtree deleted the other's mid-extract. Now PID+uuid, matching fetch_from_hub — the 'concurrent lanes never read a torn tree' guarantee actually holds. - The fetch-path OSError wrapper missed MemoryError, so a large-decompression OOM on a memory-pressured trainer escaped raw past callers that catch only StorageError and crashed the round. Now (OSError, MemoryError) both wrap. - The wrapped message echoed the raw error, re-introducing the operator-path disclosure extract_zip_safely's own boundary was written to avoid. Dropped the interpolation (the cause chain is kept via 'from e' for server logs). Two design-opinion findings (a typed OperatorFault subclass; a shared atomic-extract helper across store.py/hippius.py) are noted but not taken — they are refactors, not defects, and the code is correct as fixed. Tests: both ENOSPC and MemoryError during a vault fetch come back as StorageError with no operator path in the message. Full suite: 1729 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
… types The round-10 review noted the path-leak assertion was vacuous for the MemoryError case (str(MemoryError()) is ''), so the 'no operator path in the reason' guarantee was only really exercised for OSError. Assert the EXACT wrapped message instead — meaningful for both fault types. Declined (with reasons, not defects): the OOM-during-error-handling allocation edge (if a ~40-byte string alloc fails the process is already doomed) and the unguarded fallback-rename race (a pre-existing pattern shared with fetch_from_hub — fixing only this copy would desync the two). Full suite: 1729 passed, 21 skipped. Security review: clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…ed with DEC-CA-0036 gates Main's retry-after-settle path (reuse settled finalists, bypass entry filters) composes with the funded/vault gates deliberately: on the reused path ALL entry filters are bypassed — the settled heat already applied them, and the funded entries were settled done at the original settle, so re-filtering would empty the field and walk the king over. Champion publication runs on every attempt (idempotent per round_id via the reign counter guard), and _mark_funded_done stays unconditional at the settle moment (idempotent on retry; a crash between burn and mark self-corrects next round via the burned-hotkey terminal fail). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
A full-diff review with fresh eyes found seven real bugs the commit-scoped rounds missed by never examining the queue read/write interaction as a whole: 1. Queue read/write corruption (SERIOUS): unlocked reads shared a mutable self._entries with locked writers, so a read's reload could rebind the dict mid-write and serialise a torn/empty map — dropping a just-202'd fund or wiping the file. Removed the cached dict entirely: every op loads a FRESH local dict under a flock (writers LOCK_EX, readers LOCK_SH). 2. Content-Length not validated (SERIOUS): a negative value slipped past the size cap into rfile.read(-1) (read-to-EOF), and non-numeric raised an uncaught ValueError. Now a clean 400 bad_content_length. 3. Champion index erased on a TRANSIENT read failure: a bucket hiccup was treated as 'first publish' and rewrote the index from empty, dropping prior champions. Now distinguishes ObjectNotFound (real first publish) from a transient error (report not-done, retry — ZIP re-upload is idempotent). 4. reconcile_funded used raw terminate: a revoked-key orphan was reported killed while still billing. Now terminate_verified, confirmed kills only. 5. promote_pending held the flock across the chain resolver: a hung substrate poll would stall every queue op incl. the trainer. Now resolves UNLOCKED (snapshot under shared lock → resolve → apply under exclusive lock, with a ref re-check). 6. queue.fail select-race: a fail decided off a snapshot could terminally fail a miner's freshly re-funded entry. Added expect_ref guard (like mark_in_round). 7. Misleading 'blocked-by-existing-entry' on an idempotent same-ref re-submit; now reports 'already-funded' truthfully. Also folds in main's r47/r48 settled-retry path, composed with the DEC-CA-0036 gates (all entry filters bypassed on the reused path; champion publish runs every attempt, idempotent per round_id). Full suite: 1778 passed, 21 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
The final-pass convergence review confirmed no correctness bug survived; two doc-level items: - queue._locked is NOT re-entrant (each call opens its own fd; flock conflicts across fds even in one thread), so calling a public read/mutator inside a lock self-deadlocks. Documented the invariant so a future edit does not nest one — every method already works the yielded dict directly. - intake submit-response docstring listed only pending_reveal|none; added the already-funded / blocked-by-existing-entry values and the bad_content_length 400 this PR introduced. Doc-only; no behavior change. (The latent 'reads need write perm on .lock' narrowing is moot in practice — the transparency feed is served by the intake writer via GET /v1/queue, never read off disk by a separate uid.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
… deadline The intake runs on the orchestrator (trainer wallet + eval pool), so it gets in-app floors under the front proxy the runbook now mandates: - make_server: per-socket request_timeout_s (slowloris eviction; the stdlib default is no timeout) and a BoundedServer capping concurrent handler threads — over the cap, connections get an immediate 503 + Retry-After without spawning a thread. - ChainRevealResolver: the reveal poll joins a worker thread under a deadline; a hung substrate websocket serves the stale reveal table instead of pinning request threads, never stacks a second poll on a hung one, and harvests a late finish. - cascade-intake flags: --max-connections, --request-timeout, --chain-timeout. - docs/MINER_FUNDED_ROUNDS.md: 'Fronting the intake (DoS posture)' — proxy does volumetric defence, backstop knobs + residual risk stated. - tests: slowloris eviction, flood 503 + cap recovery, resolver deadline/stale/harvest, poll-error propagation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc
…funded-compute-twm808
…owercase .wallet) Both signing sites called bittensor.wallet(...), which the branch's own pin (bittensor==10.5.0) does not expose — cascade fund/submit crashed with 'module bittensor has no attribute wallet' before signing anything. shared/chain.py already carries the <9 vs 9+/10 shim; use the same one here. Found live on testnet 259 while rehearsing the funded path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
…s (owner 2026-09-02)
[round] funded_activation_block — the release-then-activate switch for the
whole DEC-CA-0036 machinery: before the block every funded_* knob reads
"off" and (because the flip is announced) boundaries HOLD, so no normal
rounds burn provisioner money between deploy and the flip; from the block
the armed config applies. Stamped from the live height each tick/round;
offline callers treat an unreached gate as CLOSED.
chain.toml armed (mainnet):
- funded_activation_block = 8991900 — projected ≈ Fri 2026-09-04 05:30 UTC,
the 3h grid's closest boundary to the owner's 06:00 target
- epoch seam 3600 → 900 (≈3h rounds) at 8989200 (Thu ~20:30 UTC; boundary
of both grids, load-checked)
- funded_mode=required, skip_unfunded_rounds=true, funded_pods=rent,
field cap 8 + capacity probe, five-type SKU set, funded_king_rent=true,
payer_vault_dir, the pinned worker image; direct submissions stay OFF
(image gate)
- [training] expected_gpu unpinned ("") for per-round GPU choice —
CONTRACT CHANGE: trainer + validators deploy together before the seam
Docs: MINER.md §6b (the miner funding walkthrough + failure-table rows),
llms.txt (funded-era round mechanics, workflow step 8, config values),
MINER_FUNDED_ROUNDS.md go-live checklist + block table, DEC-CA-0036
amendment (incl. the owner-accepted open gates: confirmation leg, tenure
re-denomination), CLAUDE.md pointer.
Tests: conftest neutralizes the armed pins (cadence pair, funded gate,
funded modes) exactly like expected_gpu/commit_floor; seam-literal tests
updated; gate + hold covered in test_funded_field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
…d go-live block (owner 2026-09-02) [eval] scored_horizons / scored_from_block = 8991900 — PR #241's knobs, set to the SAME block as [round] funded_activation_block so the two flips are one coordinated validator update. Ignored by loaders without #241; on the merged tree the ladder is () at 8991899 and (64, 256, 720) at 8991900 (verified). Runbook: #241 must be in the deployed release; check the current pool's >= 784-step eligibility (the 720 rung sizes every rung) before the block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
Same values; the rationale lives in docs/MINER_FUNDED_ROUNDS.md and the DEC-CA-0036 node. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
…uel settle (owner 2026-09-02) - Drop the pre-gate hold: before funded_activation_block the funded knobs read "off" and legacy rounds keep running (last one Thu 20:30 UTC, 6-8h); the first 12h boundary after it, 8992800 (≈ Fri 08:30 UTC), is the go-live block for BOTH the funded field and PR #241's ladder. - Keep the 12h epoch grid (owner): epoch_blocks 3600, no scheduled seam, max_rounds_per_day 2. The 3h grid is a later switch. - one_submission_per_hotkey bug: funded entries burned at the heat settle, BEFORE their legs ran, so a leg requeued unburned (sold-out/rate-limit/ infra) was terminally failed as "burned" by the next round's filter. Funded entries now burn at the DUEL settle per outcome — trained or generator-failed → spent; auth/market/infra → still re-fundable. Testnet runs the rule off, which is why the E2E never showed it. - Docs: spent-hotkey rollover rule (fresh hotkey, reveal after the Thu 20:30 cutoff), 12h cadence, block table; seam tests build their pairs explicitly instead of matching shipped literals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
…eardown (PRISM-level credential model) A funded pod runs on the miner's Lium account and its payer has console access, so everything in the worker's environment is theirs to read — and the funded host was mirroring the operator final profile's forward_env (Hippius S3 keys, Hub login, HF token) onto it. - RemoteHost.isolated: the dispatcher forwards NOTHING from the orchestrator's environment to an isolated host (no forward_env, no WANDB_API_KEY extra) — only the host's own static_env. - cascade/funding/robots.py: Harbor robot accounts, project-scoped, repository:push only, day-expiry backstop; minted per funded pod at rent (robot_id on the ledger), revoked at teardown and by every sweep. - Credential resolution for a payer pod, in order: per-pod robot via a project-admin USER login (CASCADE_HUB_ADMIN_USERNAME/PASSWORD — Harbor forbids robots minting robots, and the operator's own Hub identity IS a project robot, found live) → a static push-only robot from CASCADE_FUNDED_HUB_USERNAME/PASSWORD (user logins in that slot refused) → fail CLOSED (leg skipped unburned, pod torn down). The operator's own login is never an option. - Vault-ZIP staging appends to static_env instead of replacing it. - Docs: credential model + operator env pairs in the runbook, pieces table, DEC-CA-0036 amendment; remaining PRISM delta (credential-free pod via SSH harvest, sealed vault) recorded for the worker-image release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
… is tamper The trainer tracked a funded pod by NAME, which the account owner chooses and can re-use — a relaunched pod under the same name was indistinguishable, and a mid-leg swap only surfaced as an ssh drop classified infra (requeued, unburned). Now, at rent: record Lium's pod id (pod_uid, on the ledger) and ssh-keyscan the container's host key (generated at first boot); dispatch runs StrictHostKeyChecking=yes against a per-pin known_hosts file placed FIRST in the argv (ssh takes the first value); the platform id + RUNNING status are re-verified on the payer's key before dispatch and when the leg returns. Any mismatch — or a host-key refusal mid-leg — settles as error_class 'tamper': miner fault, terminal, hotkey spent. Rent fails closed (no pin → no leg). Live probes established the owner's actual reach on a custom-image pod (rent-caller keys only, backups confined to /workspace, no console), so pod replacement is the move this catches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
…RCE) Every scorer — the validator's verdict, the king-pod bench, the audit replay, the heat screen — imported and EXECUTED forecast_wrapper.py and model.py shipped inside the checkpoint directory and built the model from its config.json (evaluator.py: 'owner-produced and trusted; no static guard or sandbox'). True while only operator pods produced checkpoints; under miner-funded compute that is arbitrary code execution on all validators and on the operator's king pod from any funded miner. cascade/eval/checkpoint_guard.py: the two code files must be byte-identical to this release's copies (model.py is toto2_model.py, already folded into base_arch_digest; forecast_wrapper.py is the _FORECAST_WRAPPER_PY template) — any other .py is refused; config.json['toto2'] must equal Toto2Config.from_contract(contract); weights.safetensors is checked by HEADER (tensor names/dtypes/shapes/offsets vs the pinned model's state dict) and by a byte budget from the pinned param count, before any tensor is allocated. Honest checkpoints pass byte-for-byte. Wired: load_forecaster/evaluate_checkpoint take (validator loop passes the entry's size contract; audit replay the round's; heat screen the screen contract; the bench sidecar refuses non-identical code before invoking); trust_checkpoint_code=True is the explicit opt-out for archived pre-guard checkpoints. The trainer fetches and guards every funded leg's checkpoint before it can enter the manifest — a deviation settles as 'tamper' (hotkey spent), so validators and the bench never see it even before they upgrade. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
…ealed-payer shape
With CASCADE_VAULT_KEY_FILE set (32 raw bytes / 64 hex, shared by the
intake and the trainer) every vault entry is stored as
{sealed: b64(nonce||ct||tag), stored_at, v} under AES-256-GCM with the
hotkey as associated data (pycryptodome, already in the chain extra): a
copied vault dir is useless without the key file, and a renamed entry
fails to unseal. Unset keeps the legacy plaintext form with a loud
startup warning; legacy entries hydrate and are re-sealed on their next
insert/refresh; a configured-but-bad key file fails loud rather than
falling back to plaintext.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
…oxy config PerIpLimiter: sliding-window requests-per-minute and concurrent-connection caps per client IP (429 + Retry-After, /health exempt; rejected at the socket before a handler thread spawns). Behind the mandated TLS proxy the client IP is taken from X-Forwarded-For only when the peer IS --trusted-proxy — the header is attacker-supplied from anyone else. The runbook now carries a reference nginx front (limit_req/limit_conn, body size = the ZIP cap, header/body timeouts, request buffering). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
…licitly Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y65PT2wahVQnhE1SjWXyGW
Funded challengers were never benched: the leg tore its pod down and the post-publish sweep only knows pods, so no funded-round challenger could reach the bench stream or the promotion pool. - [telemetry] funded_bench (default on): the leg keeps its pod through publish and registers it as the bench host; each payer pod is torn down when its own sweep ends, with sweeps at bench-thread exit, round failure, the no-bench path and the next run_round for leftovers. - Payer numbers are a filter, never a published fact: the top funded_bench_verify_top payer-reported challengers are pushed (guard-verified copy, over the pinned ssh) to a <role>-verify work dir on the king pod and re-benched there; within funded_bench_verify_tolerance the operator's numbers stand in, beyond it the entry is dropped (TAMPER). Unverified payer numbers are logged only. Wire format untouched. - bench_hook: no HF_TOKEN to isolated hosts; push_checkpoint helper. - remote: ssh_transport_options/build_scp_argv so the vault-ZIP scp honours the pinned host key (it used accept-new). - _bench_host_for never guesses a pod for a funded challenger. - Docs (MINER_FUNDED_ROUNDS, MINER §6b, llms.txt), DEC-CA-0036 amendment 4, chain.toml/chain.testnet.toml [telemetry] keys, tests. Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9Qtvsw3SjMTWjkpYqRNaT
…or harvest [round] funded_pod_checkpoint = "harvest" (new default): the funded worker trains --local-only and uploads NOTHING (the pod env holds no credential at all — no robot, no token); its receipt carries local_checkpoint_dir instead of a trained_pointer (TrainedEntry still refuses non-hub pointers, keeping un-harvested checkpoints out of manifests by construction). The orchestrator pulls the checkpoint over the pinned ssh (harvest_remote_dir: tar streamed through tarfile filter="data" — traversal/absolute/symlink members off the miner-controlled stream fail the harvest), runs the ingest guard on the local copy (deviation ⇒ tamper), and uploads under its own identity. Harvest transport failures settle infra (requeue), never tamper. - worker: --local-only flag; train_one split into _train_for_entry + train_one/train_one_local. - remote: worker_argv local_only, LocalTrainReceipt/receipt_to_local, dispatch(local_checkpoint=), harvest_remote_dir. - loop: _funded_harvest gate, robot minting skipped in harvest mode, _harvest_funded_checkpoint (pull → verify → upload → real entry). - "robot" mode kept as the fallback for a pre---local-only image. - chain.toml/chain.testnet.toml key, docs, DEC-CA-0036 amendment 5, tests (harvest leg, tamper/infra split, no-robot rent, traversal refusal, worker receipt). Needs the worker image rebuilt from this release (--local-only is new CLI). Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9Qtvsw3SjMTWjkpYqRNaT
…rom the image build context _payer_vault/ (payer Lium keys!) and local orchestrator state were not in .dockerignore — a worker-image build from a live orchestrator tree would have baked them into a public image via COPY . Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9Qtvsw3SjMTWjkpYqRNaT
…ge context too gitignored files still enter docker build contexts; hosts.toml carries live pod addresses and the orchestrator's forward_env layout. Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9Qtvsw3SjMTWjkpYqRNaT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements miner-funded challenger training legs (DEC-CA-0036): miners submit generator code directly to the operator's intake service with their own Lium API key, code stays private unless it wins the throne, and training is billed to the miner's account instead of the operator's.
Summary
This PR adds the complete infrastructure for direct submissions and miner-funded compute:
cascade/funding/store.py): Content-addressed ZIP storage with ownership tracking; vault refs ride the existingrepo@digestgrammar under a reservedvault/directrepo idcascade/funding/vault.py): TTL-bounded, mode-0600 custody of Lium API keys; never logged or persisted beyond the submission lifecyclecascade/funding/queue.py): Reveal-block-ordered entry queue with 1-live-per-hotkey rule; atomic file operations under exclusive flockcascade/funding/intake.py,main.py): HTTP endpoint for/v1/fund,/v1/submit,/v1/withdraw; signed hotkey auth; chain reveal oraclecascade/funding/faults.py): Classifies Lium failures (auth/rate/capacity/infra) with no-burn rules ported from PRISMcascade/provision/funded.py): One pod per miner's key; payer-attributed ledger; teardown hydrates the right key from vaultcascade/funding/champion.py): Publishes winning vault-ref code tochampions/per policy (crown/delay/dethrone)cascade/trainer/loop.py):[round] funded_modefield selection (off/shadow/required); funded entries drain in reveal-block order; terminal entries for stale/burned hotkeyscascade/miner/cli.py):cascade fund(sign and POST key),cascade submit(ZIP and fund in one request),cascade fetch king(anonymous champion fetch)Key implementation details
vault/direct@sha256:<64hex>parses as a standardmetro-v1:gen:hippius:commit on every deployed validator — participant sets cannot fork on the schemeStorageError(→ 400), never 500Configuration
New
[round]settings:funded_mode: "off" (default), "shadow" (read-only reporting), "required" (funded entries only)funded_queue_path: Path to the queue filefunded_entry_ttl_hours: How long entries linger before expirychampion_publish: "off", "crown", "delay", "dethrone"champion_publish_delay_rounds: Reign delay before publicationNew environment variables:
CASCADE_VAULT_DIR: Operator's private submission storeCASCADE_CHAMPION_BASE: Anonymous S3 endpoint for published championsLIUM_API_KEY: Miner's API key (read by CLI, never on command line)Testing
https://claude.ai/code/session_01BYjSCaxgmwWQQXPjD1G6hc