Repository navigation
Docs: cold-reader audit of README — examples that fail as pasted, stale claims, canvas_bind re-keying gap #227
Description
Context
We ran a cold-reader audit on README.md (2026-10-23): a fresh LLM agent whose only allowed source was the README answered 12 verifiable tasks (install command, bind semantics, grid placement, READY rules, CLI exit codes, etc.) and then flagged every passage it could not extract an actionable meaning from. All 12 tasks were answered correctly — the README is operable — but the audit surfaced real defects. Each finding below was then verified against the source by a second pass; verified ones cite the code. Line numbers refer to README.md at v2.2.3.
This issue is the work list for a docs pass. Please re-verify each item against the current source before applying. The one code defect the audit surfaced has its own issue: #228 (batch_design parser rejects multi-line ops and # comment lines — the README's own examples fail as pasted). Its doc follow-through lives there; don't touch line 262 or the batch_design examples here until #228 decides parser-vs-reformat.
Verified findings
1. The README's own multi-line batch_design examples fail when pasted literally → #228
- README line 262 says "Operations are line-separated strings", yet the chart (300–306), donut (311–317), glassmorphism (1026–1030) examples span multiple lines, and the intro example (264–284) uses
# commentlines and the chart example uses trailing// actualcomments. Verified executable: the donut example returnsUnmatched parentheses;# commentlines returnInvalid operation. - Tracked in batch_design parser rejects multi-line ops and # comment lines — the README's own examples fail as pasted #228 with a proposed parser fix (preferred over reformatting) and its doc follow-through. Listed here only so this audit is complete.
2. "across all five presets" — there are four
- Line 582 claims every page scaffold is regression-tested "across all five presets"; line 41 says "across every theme".
- Verified:
src/presets.tsdefines exactly four (dark,light,material,minimal), matching lines 514 and 570. - Fix the count (and check whether the regression test itself says four or five).
3. canvas_bind never mentions ID re-keying and returns no IDs
- The
canvas_bindsection (235–258) does not say that binding re-keys every project/canvas ID torepo-*. That fact lives only in theinitsection (line 172). A reader who runs barecanvas_bindkeeps using dead IDs — the cold reader named this as the first mistake it would make. - Verified:
src/bind.tsre-keys (repo-proj-…,repo-ws-…) and its return is{ ok, root, dir, workspace, projects: <count>, migrated: <count> }— counts, not the new IDs. There is no documented way to get the new IDs after a bare bind other than re-listing. - Doc fix: state the re-keying in the
canvas_bindsection and say how to recover the new IDs (project_list/canvas_list, or preferinit). Product option worth considering: havecanvas_bindreturn the old→new ID map, likecanvas_add_variantandcopy_nodesalready do. - Related: the Usage Example (line 1154) passes
projectId: "default-project", an ID never defined anywhere in the README and presumably not valid after a bind.
4. "all advisory" vs "cliché tells block even at info severity"
- Line 808 (cliché category row): "all advisory (warning/info)". Line 784: "Cliché tells block even at info severity — they're the slop signal".
- A reader who only reads the categories table concludes tells don't block — backwards. Reword the table row (e.g. "warning/info severity, but tells always block the directive").
5. Whether a failing LLM critique blocks presenting is undefined
- With no API key, lines 836/1229 are explicit: the heuristic directive alone decides. With a key, if
canvas_evaluate({ mode: "llm" })returnsneedsRevision: truewhile the heuristic directive is READY, the README never says whether the agent may present. Line 1229 calls the critique loop "the standard last step, not an optional extra", which implies no — but it's an inference. - Decide the contract and state it in both the
canvas_evaluatellm section and the Workflow step 7.
6. Unqualified "all canvases persist to ~/.framesmith/canvases/"
- Lines 77 and 1217 state it universally; line 256 says bound canvases live in the repo
.framesmith/and "a canvas is either repo-bound or global, never both". Line 1204 says the viewer watches~/.framesmith/canvases/(reconciled only by the registry-mirror sentence at 258). - Qualify the global-store sentences ("unbound canvases…" / "…plus registered repos via
registry.json").
Reported by the audit, lower stakes (verify before applying)
- The
batch_designProperties list (line 291) omits properties used elsewhere in the README:name(1126),responsive(325),pulse(289),highlight/barGradient(319),area(302). batch_designhas no param table —canvasId/operationsappear only in usage snippets (866, 1157);M("nodeId", "newParentId", 0)'s third argument is unlabeled (index).canvas_evaluatecategory weights (801–810) sum to 145; the README doesn't say how they normalize to the 0–100 score.- "Phase 28" (308, 583) and "Phase 13" (836, 961) are internal roadmap jargon, undefined in the README.
- Viewer port: line 61 says "default port 3001", line 1249 says "auto-picks from 3001".
- Line 325: "Unsafe
gridColumns/gridColumn/gridRowvalues" — "unsafe" is never defined (the code's rule isSAFE_GRID_PLACEMENTinsrc/renderer.ts).
What the audit did NOT flag (for calibration)
The dense-but-precise passages — the canvas_version hashing paragraph (706), the inherited-token contract (455–461), the usability row (810) — were all judged decodable on one careful read. No action needed there.
Method note
Cold-reader test: one Sonnet agent, README-only (no code access, no prior project knowledge), 12 verifiable tasks + an "I genuinely can't tell what this means" audit, findings then verified against source. ~130k tokens, ~80s. The likeliest-first-mistake ranking (bind IDs > gridColumn-as-span > multi-line ops > variant scaffolding sequence) is the part worth keeping as a doc-priority order.
Activity
Merge pull request #230 from vicmaster/docs-readme-audit-227