Skip to content

Docs: cold-reader audit of README — examples that fail as pasted, stale claims, canvas_bind re-keying gap #227

Closed

Description

@vicmaster

Context

We ran a cold-reader audit on README.md (2026-10-23): a fresh LLM agent whose only allowed source was the README answered 12 verifiable tasks (install command, bind semantics, grid placement, READY rules, CLI exit codes, etc.) and then flagged every passage it could not extract an actionable meaning from. All 12 tasks were answered correctly — the README is operable — but the audit surfaced real defects. Each finding below was then verified against the source by a second pass; verified ones cite the code. Line numbers refer to README.md at v2.2.3.

This issue is the work list for a docs pass. Please re-verify each item against the current source before applying. The one code defect the audit surfaced has its own issue: #228 (batch_design parser rejects multi-line ops and # comment lines — the README's own examples fail as pasted). Its doc follow-through lives there; don't touch line 262 or the batch_design examples here until #228 decides parser-vs-reformat.

Verified findings

1. The README's own multi-line batch_design examples fail when pasted literally → #228

  • README line 262 says "Operations are line-separated strings", yet the chart (300–306), donut (311–317), glassmorphism (1026–1030) examples span multiple lines, and the intro example (264–284) uses # comment lines and the chart example uses trailing // actual comments. Verified executable: the donut example returns Unmatched parentheses; # comment lines return Invalid operation.
  • Tracked in batch_design parser rejects multi-line ops and # comment lines — the README's own examples fail as pasted #228 with a proposed parser fix (preferred over reformatting) and its doc follow-through. Listed here only so this audit is complete.

2. "across all five presets" — there are four

  • Line 582 claims every page scaffold is regression-tested "across all five presets"; line 41 says "across every theme".
  • Verified: src/presets.ts defines exactly four (dark, light, material, minimal), matching lines 514 and 570.
  • Fix the count (and check whether the regression test itself says four or five).

3. canvas_bind never mentions ID re-keying and returns no IDs

  • The canvas_bind section (235–258) does not say that binding re-keys every project/canvas ID to repo-*. That fact lives only in the init section (line 172). A reader who runs bare canvas_bind keeps using dead IDs — the cold reader named this as the first mistake it would make.
  • Verified: src/bind.ts re-keys (repo-proj-…, repo-ws-…) and its return is { ok, root, dir, workspace, projects: <count>, migrated: <count> } — counts, not the new IDs. There is no documented way to get the new IDs after a bare bind other than re-listing.
  • Doc fix: state the re-keying in the canvas_bind section and say how to recover the new IDs (project_list / canvas_list, or prefer init). Product option worth considering: have canvas_bind return the old→new ID map, like canvas_add_variant and copy_nodes already do.
  • Related: the Usage Example (line 1154) passes projectId: "default-project", an ID never defined anywhere in the README and presumably not valid after a bind.

4. "all advisory" vs "cliché tells block even at info severity"

  • Line 808 (cliché category row): "all advisory (warning/info)". Line 784: "Cliché tells block even at info severity — they're the slop signal".
  • A reader who only reads the categories table concludes tells don't block — backwards. Reword the table row (e.g. "warning/info severity, but tells always block the directive").

5. Whether a failing LLM critique blocks presenting is undefined

  • With no API key, lines 836/1229 are explicit: the heuristic directive alone decides. With a key, if canvas_evaluate({ mode: "llm" }) returns needsRevision: true while the heuristic directive is READY, the README never says whether the agent may present. Line 1229 calls the critique loop "the standard last step, not an optional extra", which implies no — but it's an inference.
  • Decide the contract and state it in both the canvas_evaluate llm section and the Workflow step 7.

6. Unqualified "all canvases persist to ~/.framesmith/canvases/"

  • Lines 77 and 1217 state it universally; line 256 says bound canvases live in the repo .framesmith/ and "a canvas is either repo-bound or global, never both". Line 1204 says the viewer watches ~/.framesmith/canvases/ (reconciled only by the registry-mirror sentence at 258).
  • Qualify the global-store sentences ("unbound canvases…" / "…plus registered repos via registry.json").

Reported by the audit, lower stakes (verify before applying)

  • The batch_design Properties list (line 291) omits properties used elsewhere in the README: name (1126), responsive (325), pulse (289), highlight / barGradient (319), area (302).
  • batch_design has no param table — canvasId / operations appear only in usage snippets (866, 1157); M("nodeId", "newParentId", 0)'s third argument is unlabeled (index).
  • canvas_evaluate category weights (801–810) sum to 145; the README doesn't say how they normalize to the 0–100 score.
  • "Phase 28" (308, 583) and "Phase 13" (836, 961) are internal roadmap jargon, undefined in the README.
  • Viewer port: line 61 says "default port 3001", line 1249 says "auto-picks from 3001".
  • Line 325: "Unsafe gridColumns/gridColumn/gridRow values" — "unsafe" is never defined (the code's rule is SAFE_GRID_PLACEMENT in src/renderer.ts).

What the audit did NOT flag (for calibration)

The dense-but-precise passages — the canvas_version hashing paragraph (706), the inherited-token contract (455–461), the usability row (810) — were all judged decodable on one careful read. No action needed there.

Method note

Cold-reader test: one Sonnet agent, README-only (no code access, no prior project knowledge), 12 verifiable tasks + an "I genuinely can't tell what this means" audit, findings then verified against source. ~130k tokens, ~80s. The likeliest-first-mistake ranking (bind IDs > gridColumn-as-span > multi-line ops > variant scaffolding sequence) is the part worth keeping as a doc-priority order.

Activity

  1. added a commit that references this issue on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions