Skip to content

Studio: design system, reports, live views, Genesis as an autonomous scientist, identity file (features 007-020) - #2

Open
LucasWaki wants to merge 39 commits into
mainfrom
007-benchmark-foundations
Open

Studio: design system, reports, live views, Genesis as an autonomous scientist, identity file (features 007-020)#2
LucasWaki wants to merge 39 commits into
mainfrom
007-benchmark-foundations

Conversation

@LucasWaki

Copy link
Copy Markdown
Collaborator

What changed

This branch carries the Studio work of 8 and 9 Sep on top of the benchmark foundations (features 007 to 011), and closes with feature 019 and the first item of feature 020.

Studio (features 012 to 018)

  • One design system: a token sheet over Radix Colors, IBM Plex and Lucide vendored, no inline styles, no external resources. A CSS test fails on any hex colour, !important, inline style or external URL.
  • Measures computed once on the server from stored results and events: pass rate with Wilson intervals, pass^k, violations, false completion, paired deltas, complete cost. Charts drawn as SVG styled by class.
  • Reports as the front door: run and round reports that read verdict first, caveats written from data only, a historical Bare baseline reused when a finished run matches on model, effort and task hashes, a Terms section, public audience by default.
  • Live views without duplication, a sticky inspector instead of the scrolling right frame, the weekly budget chip.
  • The product graph panel shows any bench version as products with their stored business actions, and the live Product Graph Monarch Enterprise uses, read from Feature Discovery over GET only. Nothing in the bench can write to Monarch.

Genesis (feature 019)

  • Cards as inputs: a link, a run id or a sentence becomes a card with a question. A watcher works queued cards one at a time under US$ 2.00 per card and US$ 6.00 per day, counted from settled receipts, and never launches a run. It ignores runs and sources older than its first start.
  • A daily index of the Monarch checkout at the declared build, a change record filed in the library, and five read-only code tools. Graphify is used when installed.
  • Bounded memory: LAB.md and MONARCH.md at 2,500 characters each, notes per card, an FTS5 record, seven-day probation, thirty-day decay, pinned entries that never decay, a nightly consolidation turn under US$ 0.50.
  • A Trello-like board, a fixed library topic list with Other last, and a daily scheduler.

Feature 020, first item: the identity file

  • SOUL.md holds Genesis's voice, priorities and what it never does. A starter text is written on first start. Only a person edits it, from the Memory tab. It is the first block of every prompt and no Genesis tool reaches it.

Also

  • Windows long-path handling in evidence writes and a short pytest temp root.
  • wb studio takes its port from the environment.
  • Research log, plan, improvement ledger and handoff docs of 8 and 9 Sep.

Why

The plan of 9 Sep (docs/AI-LABS-IMPLEMENTATION-PLAN-2026-09-09.md) mapped the Studio into seven phases. The Genesis design follows the research note docs/AI-LABS-GENESIS-CODE-AWARE-MEMORY-2026-09-09.md; Lucas's choices for what Genesis grows next are in its Part 3.

For the reviewer

  • Python suite: 1832 passed, 3 skipped, 1 failed. The failure is tests/test_run_config.py::test_shipped_create_run_pilot_hash_is_unchanged, which already failed at main and waits on Lucas's sign-off to re-pin the hash (pre-registration rule).
  • Browser suite: node tests/browser/suite.cjs, 27 checks pass. Screenshots under tests/browser/snapshots/ are for review, not pixel diffing.
  • The watcher spent nothing in the wild, but at its first start it backfilled cards for every past run and source and started four turns that failed the ledger check. Both causes are fixed in this branch.
  • Carlos's two doc commits on main (handoff of 8 Sep, deferred item on the public front door) are merged in; the one conflict, in deferred.md, kept both sides.
  • Kept out by .gitignore: the Monarch Enterprise checkout snapshot under artifacts/enterprise-local, Genesis test sessions under artifacts/genesis-check, and local scratch folders.
  • Nothing here changes the methodology in PLAN.md §1. No task prompt, data or approval rule was edited.

🤖 Generated with Claude Code

LucasWaki and others added 30 commits September 8, 2026 17:27
…ce, regrade revisions, runtime manifests, Studio with product graphs, offline Enterprise adapter; unblock plan of 8 Sep

Offline foundation work of feature 007 and the feature 011 checkpoints, as recorded in
specs/007-lab-foundation/implementation.md and specs/011-monarch-runtime-integration/.
Paid dispatch stays gated in the CLI. Frozen task sets untouched. Full suite: 1091 passed,
3 skipped.

Also: docs/AI-LABS-UNBLOCK-PLAN-2026-09-08.md (the plan settled with Lucas), the lab
direction and Studio review notes, research/ scaffolding (ledger database ignored),
.gitignore entries for the unrelated files that landed in this checkout (ignored by
name, not deleted).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… plan files, hashed when not the default

Unblock plan M2 T2.3. The default keeps every existing plan hash, the same rule
retry_on_fail follows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lder on a volume, host flag; Dockerfile and .railwayignore for Railway

Unblock plan D4 and Lucas's request of 8 Sep (web app on Railway behind a login).
STUDIO_AUTH_USER/PASSWORD turn on HTTP Basic Auth; STUDIO_PUBLIC_HOSTS names the
hosted host; STUDIO_DATA_DIR moves jobs and the weekly ledger to the mounted volume.
Binding to a non-local address without the auth pair is refused. 28 Studio tests pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…er API request, attempt cap, round admission, Monarch attempts reserve their ceiling

Milestone M3 of the unblock plan, steps T3.1 and T3.2. The CLI's API loop
reserves every provider request for its rate-card maximum, claims it, sends
it and settles from the usage receipt; an unreadable receipt keeps the hold.
Reservation ids are stable per run, attempt, invocation and turn, so a retry
or a resume never reserves a settled request again. Plan field
attempt_cap_usd (default 3.00, hashed only when set) ends an attempt as
infra:attempt_cap, which resume treats as final. An exhausted week ends the
attempt as infra:weekly_budget and stops the run with stop_reason
weekly_budget, resumable when the week has room. Round admission refuses a
round whose maximum liability (API attempts x cap + Monarch attempts x the
Monarch ceiling, capped by cost_ceiling_usd) exceeds the week's capacity,
naming the shortfall, before the run row exists; resume counts only the
attempts left. Monarch attempts reserve MONARCH_ATTEMPT_CEILING_USD per
attempt through the same ledger and settle from Langfuse; the ceiling
constant moves to wb_arms.monarch and the Studio re-imports it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ly Railway branch, seeds pin the front-door host, hosted Studio

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…onstructed catalog it descends from

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…of the blanket refusal, doctor through the ledger, wb budget reconcile

Milestone M3, steps T3.3 to T3.5. wb_orchestrator/approvals.py: the
launcher is WB_OPERATOR (required for any paid launch); approvers default to
lucas (WB_APPROVERS overrides). Above smoke scale an approver's wb run runs
at once under an approved record (new results-store table
approval_requests); anyone else's writes a pending request, prints
"<id> awaiting approval" and exits 0. wb approve / wb deny (approvers only),
wb approvals, and wb run --request <id> (any operator, config hash must
match, single use). approved_by in plan files is optional and ignored with a
one-line notice; the resolve-time gate is gone; wb monarch recipes proceeds
on --yes only.

approvals.launch_readiness replaces _paid_launch_blocked: API-loop
competitors launch with an operator and the ledger (top-level wb --ledger,
default research/budget.sqlite3); Monarch competitors, wb monarch recipes
and wb doctor --monarch-probe are refused with "Monarch instance not
verified: milestone M5", native competitors with "native runtime not
verified: milestone M7". The doctor's provider probes reserve every request
through the ledger. wb budget reconcile --week --provider --csv compares a
week's normalized usage rows with the ledger's settled total per provider,
writes research/reconciliation/<week>.md and .json, and marks
historical_billing_verified for that week only when every provider with
spend is within 5 %; wb budget status reports it per week with the
capability matrix.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rk skill; config README on the attempt cap, approvals, the ledger and wb budget reconcile

Lucas approves paid rounds; a launch by Carlos creates a request and waits;
the weekly ledger is the spending gate. The README explains attempt_cap_usd,
the ignored approved_by key, the approval commands, round admission, and how
to export the usage rows wb budget reconcile reads from the Anthropic
Console, the OpenAI usage page, Fireworks, Google and Langfuse (for
Monarch). The shipped plan files' approved_by comments now say the key is
ignored (comments only; no hash moves).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s manifest; the achievable-50 id list (unblock plan M2, T2.1 and T2.2)

The command copies each corpus task unchanged into the set's folder and writes <name>-manifest.yaml beside it (selection rule from the id list's comment lines, why the set exists, suite revision from the installed automation-bench, cut points reused from the tier manifest, count per domain, one row per task with domain, score, tier and hash). It refuses, naming every offender, when an id is missing from the corpus, has no approval rule, does not match its own hash, or already sits in a frozen set; --refreeze keeps the ids and refreshes the copies; --allow-frozen-overlap records an overlap instead of refusing.

tasks/achievable-50-ids.txt lists the 50 ids of the ApplicationBench achievable50 slate with the rule and source above them. Running the command on it refuses today: operations.docusign_prospect_nda sits in tier-complex and support.reamaze_cross_platform_dedup in random-10 (drawn 4 Sep, seed 20260904), so the set is not frozen until Lucas decides.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…request) and achievable-50-workflow (track create-run) (unblock plan M2, T2.4)

Same task set, mode, size, competitors (answer key, claude-opus-5/api as baseline, monarch-stock, monarch-lab), ceiling US$ 220 and internal audience; they differ only in name, description and track. The Monarch harness files arrive with M5, so the plans load but do not resolve yet; the claude-code competitor joins with M7. Tested through config.load_plan.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rack) and EXP-2026-002 (workflow track), ledger lines, and the two round docs (unblock plan M2, T2.5 and T2.6)

Control claude-opus-5/api; treatment monarch-lab (secondary monarch-stock); primary comparison lab against bare on first-attempt strict pass rate, paired by task, McNemar as wb report computes; prediction at least 5 points from the 26 Aug 2026 record (403/600 against 348/600); ceiling US$ 220 reserved in the weekly ledger; promote to a replication on a fresh draw only with p < 0.05 and no integrity flag. Status hypothesis. Round docs in the shape of the tier docs; status: set listed, freeze pending Lucas's decision on two ids already in frozen tier sets, re-freeze pending the evalrepair.10 import, Monarch harness files pending M5.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rlap with two tier ids recorded); re-freeze pending the evalrepair.10 import

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…y) on both tracks, runnable before the Monarch instances exist

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…orld; vendor script with a version check and a provenance record

scripts/vendor_automation_bench.py copies a source tree into vendor/automation-bench, refuses any version other than --expect-version, leaves .venv, caches, logs and .git behind, and writes VENDORED-FROM.txt (source, version, pyproject hash, file count, content hash, tree id as given, date). pyproject pins automation-bench==1.0.6+evalrepair.10; uv.lock moved on that one package (uv 0.9.10 also rewrote two platform markers). Milestone M1 of docs/AI-LABS-UNBLOCK-PLAN-2026-09-08.md, T1.1.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… wb run refuses a set imported under another world; reports never pool suites

wb corpus import-ab --revision LABEL --out DIR stamps info.world (package, installed version, label; hashed) and writes DIR/MANIFEST.yaml; wb corpus manifest rewrites it with per-domain counts, usable counts and the tasks without a rule. suite_id(tasks) gives workflowbench-synthetic@<version> for such a set and keeps @0.1 for the old ones; the orchestrator and the Studio record it on runs and rows, resume refuses a suite drift, and wb summary refuses rounds of different suites. config.resolve refuses a task set whose recorded world (1.0.6 when none) is not the installed package, naming both. seeded_services() restores what 'services seeded' meant now that the repaired world spells out all 48 apps' empty defaults in every scored task: declare's side-effect allowance, the difficulty measure and the two product service checks use it. Milestone M1, T1.2 to T1.4.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lrepair10 (1.0.6+evalrepair.10), approval rules derived, manifest written

800 tasks (simple 200; finance, hr, marketing, operations, sales, support 100 each), 0 unmapped assertion types, 800 usable. Every task carries info.world; corpus/ (upstream 1.0.6) is untouched as the record of the old world. Milestone M1, T1.2.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Source, versions, hashes, commands, test counts before and after, the world's effect on the bench (materialized defaults, the 48th app, the no-op validator) and what was left for M2 and M5. Force-added: .gitignore line 64 (commit e033c90) ignores /specs/007-lab-foundation/, which T0.1 should undo.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…th descriptions from the PG-Waki catalog through an explicit table

Unblock plan of 8 Sep 2026, M6 T6.3 groundwork. `wb monarch knowledge
--knowledge <catalog.json> --out <dir>` generates the stock seeds and
enriches `business_action.description` of every matched action with the
reviewed catalog's purpose, non-effects, idempotency, record location (only
where the seed's own response schema reaches it) and argument meaning (only
for parameters the seed has); each product's paragraph opens its first action
prefixed "Product:" since `_meta.json` has exactly five keys. Ids, verbs,
url templates, parameters, extracts, schemas and `_meta.json` are
byte-identical to the stock set; Monarch's validator reports the same 0
errors and 170 warnings on both.

The catalog's ids are AutomationBench Zapier tools (`zapier:<app>_<tool>`)
and the seeds' are REST routes (`bench-<app>:<verb>:<object>`); no rule
derives one from the other, so the mapping is the explicit table
`config/products/simulated-apps.knowledge-map.yaml` (257 rows, 78 notes),
read from the data product by product. `<dir>/KNOWLEDGE-MAPPING.yaml` lists
the catalog's sha256, the rule, the counts, every pair, the 16 entries
without a row (with reasons) and the 462 actions without an entry; `ok.txt`
carries `knowledge_sha256` and `knowledge_source` next to the seed version.
Same inputs, same bytes. `wb monarch setup --knowledge` runs the same step
before validating, checking and importing, and records the two fields in
the knowledge-base hash file. Nothing is imported anywhere by this change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… lab seed set

`KNOWLEDGE-MAPPING.yaml` as written by `wb monarch knowledge` on 8 Sep 2026
from catalog sha256 7aea3d99... and the shipped table: 273 entries, 257
matched onto 224 of the 686 bench actions, 16 catalog-only (each with its
reason), 462 bench-only, 43 product paragraphs applied, 4 products without
one. Seed folder digest dd03a8698bdb...; front door pinned to
http://host.docker.internal:9105 for this offline run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…authoring backstops) into 007-benchmark-foundations

Resolved: pilot plan ceiling US$ 5 with the approval note; deferred.md keeps both additions;
orchestrator attaches the journal and sets the artifacts folder; the Monarch arm keeps the
stream-then-poll follower and gains the no-writes verdict; the engine-terminal check is renamed
_run_terminal so it no longer shadows the authoring terminal-frame handler; the orphan-deletion
attribute is initialised in __init__; the SSE reader takes origin's raw-socket loop under the
generic stream label. Monarch, Studio Enterprise, recipes and run-only tests: 114 passed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…stray files on 8 Sep)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…air10; world revision on every task; reports never pool suites) into 007-benchmark-foundations

Resolved: the run start keeps the ledger admission and uses the set's own suite id.
Vendored copy replaced with scripts/vendor_automation_bench.py; installed package
automation-bench 1.0.6+evalrepair.10.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d table, mapping report) into 007-benchmark-foundations

Resolved: .gitignore and deferred.md keep both sides' additions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… (same 50 ids)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… flow, evalrepair.10 done; what is ready to trigger and what is not

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…anifest (re-freeze on another revision)

Full suite on the merged tree: 1251 passed, 3 skipped, this one test failed and is fixed; test_slate 14 passed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
LucasWaki and others added 9 commits September 8, 2026 20:13
… hosted front door; the three steps left for the Monarch arms

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d instance admits Monarch competitors for two hours

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ont door (47 apps, 8 Sep night)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…COMMIT) freezes its manifest on a hosted bench

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ilot monarch-pilot-001; how Monarch rounds launch

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…omous scientist, identity file (features 012-020)

Studio: one token sheet over Radix Colors with IBM Plex and Lucide vendored,
measures computed on the server (Wilson intervals, paired deltas, cost),
run and round reports with caveats from data and a historical Bare baseline,
the live product graph read from Feature Discovery over GET only, a Graph view
per bench version, the streaming run page without duplication, the sticky
inspector, the weekly budget chip, and the browser suite (27 checks).

Genesis (feature 019): cards as inputs worked by a watcher under per-card and
daily ceilings, a daily Monarch code index with read-only code tools, a bounded
three-tier memory with nightly consolidation, a Trello-like board, a fixed
library topic list, and the daily scheduler.

Feature 020, first item: SOUL.md, the identity file. A starter text on first
start, edited only by a person from the Memory tab, the first block of every
prompt, out of reach of every Genesis tool.

Also: Windows long-path handling in evidence writes and a short pytest temp
root; wb studio port from the environment; research log, plan and handoff
docs of 8 and 9 Sep. Ignored: local scratch, the Monarch checkout snapshot
under artifacts, and Genesis test sessions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… the public front door) into 007-benchmark-foundations
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant