docs: active harnesses auto-enqueue each round - #2
Conversation
Reflect control-plane behavior so miners know they stay in the field without re-POSTing, and clarify manual vs scheduled quota.
|
Warning Review limit reached
Next review available in: 54 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThe documentation now describes automatic enqueueing of accepted harnesses, rolling 10-round scoring, separate manual and scheduled quotas, submission lifecycle rules, elimination cooldowns, and screenshot-only viewer access. ChangesRound Scheduling and Scoring
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/rounds-and-scoring.md`:
- Line 12: Use one automatic-enqueue lifecycle contract across the
documentation: in docs/rounds-and-scoring.md lines 12-12, replace the
accepted-submit trigger with an accepted active harness trigger; in
docs/README.md lines 5-6, state that scheduling continues for every open round
after the harness becomes active; and in docs/README.md lines 32-34, add the
accepted-active state before the next open round in the process diagram.
- Around line 37-38: Update the viewer route documentation in
rounds-and-scoring.md to include the gateway prefix, documenting GET
{GATEWAY}/challenge/design/v1/view/{run_id}/index.png while preserving the
existing statement that .html requests return 410 Gone.
- Line 14: Update the “Split daily quota” entry to define the scheduled cap as
20 per hotkey per UTC day for the scheduled bucket only, derived from 10 rounds
× one prompt per round × a headroom factor of 2. Clarify that it neither
combines with the manual limit of 10 nor limits in-flight runs, that both
buckets reset at the UTC day boundary, and that quota responses expose separate
limit and remaining values.
In `@docs/troubleshoot.md`:
- Around line 6-7: Update the troubleshooting entries for “Quota exhausted” and
“Active harness, no run this round” to distinguish manual versus scheduled
bucket exhaustion and delayed enqueue versus elimination. In the quota guidance,
direct users to inspect both bucket states and wait for the next UTC day only
when the relevant bucket is exhausted; in the harness guidance, direct users to
check scheduled.remaining, round events, and elimination status before waiting
for auto-enqueue.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 8f6ecbdd-ae34-45ee-992a-e6f750d9de32
📒 Files selected for processing (5)
docs/README.mddocs/rounds-and-scoring.mddocs/submit.mddocs/troubleshoot.mdexamples/what-not-to-do/README.md
| | **30 minute agent timeout** | Your `run` has at most 30 minutes in the sandbox. | | ||
| | **10 sandboxed runs / hotkey / UTC day** | Check remaining quota with `GET /v1/quota/{hotkey}`. | | ||
| | **3 prompts per round** | Same deterministic prompt set for every harness in that round. | | ||
| | **Auto-enqueue each round** | After your first accepted submit (next round), your **latest active harness** is queued automatically every open round — you do **not** need to re-POST to keep competing. | |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Use one lifecycle contract for automatic enqueueing. The documentation must require an accepted active harness and an open round before automatic scheduling.
docs/rounds-and-scoring.md#L12-L12: replace “accepted submit” with an accepted active harness trigger.docs/README.md#L5-L6: state that scheduling applies to every open round after the harness becomes active.docs/README.md#L32-L34: add the accepted-active state before the next open round in the process diagram.
📍 Affects 2 files
docs/rounds-and-scoring.md#L12-L12(this comment)docs/README.md#L5-L6docs/README.md#L32-L34
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/rounds-and-scoring.md` at line 12, Use one automatic-enqueue lifecycle
contract across the documentation: in docs/rounds-and-scoring.md lines 12-12,
replace the accepted-submit trigger with an accepted active harness trigger; in
docs/README.md lines 5-6, state that scheduling continues for every open round
after the harness becomes active; and in docs/README.md lines 32-34, add the
accepted-active state before the next open round in the process diagram.
| | **3 prompts per round** | Same deterministic prompt set for every harness in that round. | | ||
| | **Auto-enqueue each round** | After your first accepted submit (next round), your **latest active harness** is queued automatically every open round — you do **not** need to re-POST to keep competing. | | ||
| | **1 shared prompt / round** | Same deterministic prompt for every harness in that round. | | ||
| | **Split daily quota** | **Manual** (your `POST /v1/harness`, 10/day) vs **Scheduled** (organizer auto-enqueue, ~10/day, cap 20). Auto-queue does not spend manual quota. | |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
rg -n -C 5 'manual|scheduled|cap.?20|quota|auto-enqueue' docs examplesRepository: BaseIntelligence/design-challenge
Length of output: 8425
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- candidate files ---'
git ls-files | rg '(^|/)(docs/rounds-and-scoring\.md|.*quota.*|.*round.*|.*contract.*|.*api.*)$' | head -200
printf '%s\n' '--- quota and scheduled references ---'
rg -n -C 4 'runs_used|scheduled|manual|cap.?20|quota|auto.enqueue|auto-enqueue' --glob '!docs/rounds-and-scoring.md' . | head -400Repository: BaseIntelligence/design-challenge
Length of output: 5155
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- repository files ---'
git ls-files | head -300
printf '%s\n' '--- API and quota documentation ---'
cat -n docs/api.md | sed -n '1,100p'
cat -n docs/rounds-and-scoring.md | sed -n '1,70p'
cat -n docs/submit.md | sed -n '60,85p'
printf '%s\n' '--- exact quota-contract terms ---'
rg -n -i 'runs_used|remaining|limit|quota/\{hotkey\}|scheduled|manual|UTC day|cap' . --hidden --glob '!.git/**'Repository: BaseIntelligence/design-challenge
Length of output: 8720
🌐 Web query:
BaseIntelligence BASE monorepo DESIGN_CHALLENGE.md scheduled quota cap 20 runs_used limit remaining
💡 Result:
The BASE monorepo (maintained by BaseIntelligence) handles task scheduling, worker operations, and validator/miner interactions for its decentralized AI agent challenges [1][2]. While the specific DESIGN_CHALLENGE.md file you mentioned is part of the repository structure, authoritative technical documentation regarding scheduled quota caps—specifically the variables runs_used and remaining—is defined by the platform's execution and worker schemas [3][1][4]. In the context of the BASE framework, quota management for tasks and execution runs generally adheres to the following principles: 1. Quota Definitions: Quota tracking, where implemented, typically uses runs_used (current consumption), runs_limit (total allowed), and a calculated remaining value (the difference between limit and used) [5][6][7]. 2. Enforcement: BASE utilizes an execution substrate (broker) to manage job deployments [8]. If a run exceeds a defined quota, the system architecture often supports postponing execution until a slot becomes available, rather than dropping the request entirely [6]. 3. Implementation: In the BASE repository, core constraints (such as execution proofs or VM configurations) are aligned with smart contract parameters [9]. Developers should consult the docs/monorepo.md or docs/miner/reference.md files within the BaseIntelligence/base repository for specific per-challenge quota configurations, as these may vary based on the specific subnet subnet requirements and miner activity [3][4]. If you are encountering a quota cap, ensure your environment is correctly passing authentication and identity headers required for the BaseIntelligence proxy to accurately attribute the runs_used against your specific miner or validator key [1][8].
Citations:
- 1: https://github.com/BaseIntelligence/agent-challenge
- 2: https://github.com/BaseIntelligence
- 3: CortexLM/cortex@BaseIntelligence:85d20e0...BaseIntelligence:5b2c88e
- 4: CortexLM/cortex@main@{1day}...main
- 5: AttiR/OpsCanvas@f17ca40
- 6: https://retoor.molodetz.nl/retoor/devplacepy/commit/a3963611f07d4d77b4ec5f6368e633f652fbafbc.diff
- 7: https://github.com/yannickloth/claude-router-system/blob/main/plugins/infolead-claude-subscription-router/implementation/quota_tracker.py
- 8: https://github.com/PlatformNetwork/platform/blob/main/docs/challenges.md
- 9: CortexLM/cortex@01b1e93
🏁 Script executed:
#!/bin/bash
set -eu
for repo in BaseIntelligence/base BaseIntelligence/agent-challenge; do
printf '%s\n' "--- $repo tree matches ---"
curl -fsSL "https://api.github.com/repos/$repo/git/trees/main?recursive=1" |
jq -r '.tree[]?.path' |
rg -i 'design.?challenge|quota|round' |
head -100 || true
printf '%s\n' "--- $repo DESIGN_CHALLENGE.md ---"
curl -fsSL "https://raw.githubusercontent.com/$repo/main/docs/DESIGN_CHALLENGE.md" |
rg -n -i -C 5 'quota|scheduled|manual|cap|runs_used|remaining|UTC day' |
head -300 || true
doneRepository: BaseIntelligence/design-challenge
Length of output: 14320
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
import json
import re
import urllib.request
base = "https://raw.githubusercontent.com/BaseIntelligence/base/main/"
tree_url = "https://api.github.com/repos/BaseIntelligence/base/git/trees/main?recursive=1"
tree = json.load(urllib.request.urlopen(tree_url))["tree"]
roots = (
"crates/design-challenge-task/",
"crates/design-challenge/",
"bins/design-challenge/",
"crates/db/migrations/",
)
terms = re.compile(r"runs_used|remaining|scheduled|manual|quota|SCHEDULED_DAILY_RUN_CAP|DAILY_RUN", re.I)
for entry in tree:
path = entry.get("path", "")
if not path.startswith(roots) or not path.endswith((".rs", ".sql")):
continue
try:
text = urllib.request.urlopen(base + path).read().decode()
except Exception:
continue
lines = text.splitlines()
hits = [i for i, line in enumerate(lines) if terms.search(line)]
if hits:
print(f"\n--- {path} ---")
shown = set()
for i in hits:
for j in range(max(0, i - 3), min(len(lines), i + 4)):
if j not in shown:
print(f"{j + 1}:{lines[j]}")
shown.add(j)
PYRepository: BaseIntelligence/design-challenge
Length of output: 2457
🏁 Script executed:
#!/bin/bash
set -eu
tree=$(curl -fsSL 'https://api.github.com/repos/BaseIntelligence/base/git/trees/main?recursive=1')
printf '%s\n' '--- quota-related source paths ---'
printf '%s' "$tree" |
jq -r '.tree[]?.path' |
rg '^(crates/design-challenge-task|crates/design-challenge|bins/design-challenge|crates/db/migrations)/.*\.(rs|sql)$' |
while read -r path; do
content=$(curl -fsSL "https://raw.githubusercontent.com/BaseIntelligence/base/main/$path")
if printf '%s\n' "$content" | rg -qi 'runs_used|remaining|scheduled|manual|quota|SCHEDULED_DAILY_RUN_CAP|DAILY_RUN'; then
printf '%s\n' "--- $path ---"
printf '%s\n' "$content" |
rg -n -C 4 'runs_used|remaining|scheduled|manual|quota|SCHEDULED_DAILY_RUN_CAP|DAILY_RUN'
fi
doneRepository: BaseIntelligence/design-challenge
Length of output: 10396
🏁 Script executed:
#!/bin/bash
set -eu
curl -fsSL 'https://api.github.com/repos/BaseIntelligence/base/git/trees/main?recursive=1' |
jq -r '.tree[]?.path' |
rg -i '(^|/)(design|.*quota.*|.*http.*|.*route.*|.*api.*|.*handler.*).*' |
head -300Repository: BaseIntelligence/design-challenge
Length of output: 4828
🏁 Script executed:
#!/bin/bash
set -eu
for path in \
crates/design-http/src/api.rs \
crates/design-http/src/lib.rs \
crates/design-db/src/lib.rs \
crates/design-store/src/store.rs \
crates/design-store/src/lib.rs \
crates/design-store-pg/src/lib.rs; do
printf '%s\n' "--- $path ---"
curl -fsSL "https://raw.githubusercontent.com/BaseIntelligence/base/main/$path" |
rg -n -C 6 'quota|runs_used|manual_runs_used|scheduled|remaining|limit|UTC|day'
doneRepository: BaseIntelligence/design-challenge
Length of output: 15938
Define the scheduled quota cap.
State that cap 20 applies per hotkey per UTC day to the scheduled bucket only. It is the 10-round schedule multiplied by one prompt per round and a headroom factor of 2. It does not combine with the manual limit of 10 or limit in-flight runs. State that both buckets reset with the UTC day and that the quota response reports separate limit and remaining values.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/rounds-and-scoring.md` at line 14, Update the “Split daily quota” entry
to define the scheduled cap as 20 per hotkey per UTC day for the scheduled
bucket only, derived from 10 rounds × one prompt per round × a headroom factor
of 2. Clarify that it neither combines with the manual limit of 10 nor limits
in-flight runs, that both buckets reset at the UTC day boundary, and that quota
responses expose separate limit and remaining values.
| Screenshots only: `GET /v1/view/{run_id}/index.png`. Produced HTML is never | ||
| served (`.html` → `410 Gone`). |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target section ---'
sed -n '1,70p' docs/rounds-and-scoring.md
printf '%s\n' '--- gateway and viewer references ---'
rg -n -C 3 '/challenge/design|/v1/view|GATEWAY|index\.png|410 Gone' README.md docs
printf '%s\n' '--- changed-file summary ---'
git diff --stat -- docs/rounds-and-scoring.mdRepository: BaseIntelligence/design-challenge
Length of output: 11123
Add the gateway prefix to the viewer route.
For gateway access, document GET {GATEWAY}/challenge/design/v1/view/{run_id}/index.png. Keep the .html → 410 Gone behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/rounds-and-scoring.md` around lines 37 - 38, Update the viewer route
documentation in rounds-and-scoring.md to include the gateway prefix,
documenting GET {GATEWAY}/challenge/design/v1/view/{run_id}/index.png while
preserving the existing statement that .html requests return 410 Gone.
| | Quota exhausted | `GET .../quota/{hotkey}` — check `manual` vs `scheduled`; wait for next UTC day | | ||
| | Active harness, no run this round | Wait for round auto-enqueue; or you may be eliminated | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Distinguish quota exhaustion from delayed enqueue and elimination.
The manual and scheduled buckets have separate effects. If only one bucket is exhausted, “wait for next UTC day” is incomplete. For an active harness without a run, users should check scheduled.remaining, round events, and elimination status before waiting.
Proposed wording
- | Quota exhausted | `GET .../quota/{hotkey}` — check `manual` vs `scheduled`; wait for next UTC day |
- | Active harness, no run this round | Wait for round auto-enqueue; or you may be eliminated |
+ | Quota exhausted | Check which bucket is exhausted. Manual quota blocks `POST /v1/harness`; scheduled quota blocks auto-enqueue until reset |
+ | Active harness, no run this round | Check `scheduled.remaining`, round events, and elimination status before waiting for auto-enqueue |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | Quota exhausted | `GET .../quota/{hotkey}` — check `manual` vs `scheduled`; wait for next UTC day | | |
| | Active harness, no run this round | Wait for round auto-enqueue; or you may be eliminated | | |
| | Quota exhausted | Check which bucket is exhausted. Manual quota blocks `POST /v1/harness`; scheduled quota blocks auto-enqueue until reset | | |
| | Active harness, no run this round | Check `scheduled.remaining`, round events, and elimination status before waiting for auto-enqueue | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/troubleshoot.md` around lines 6 - 7, Update the troubleshooting entries
for “Quota exhausted” and “Active harness, no run this round” to distinguish
manual versus scheduled bucket exhaustion and delayed enqueue versus
elimination. In the quota guidance, direct users to inspect both bucket states
and wait for the next UTC day only when the relevant bucket is exhausted; in the
harness guidance, direct users to check scheduled.remaining, round events, and
elimination status before waiting for auto-enqueue.
1 participant
Summary
Pairs with BaseIntelligence/base PR for the control-plane auto-enqueue change.
Test plan
Summary by CodeRabbit
410 Gone.