[adewale]

Summary

Adds the repository's first check that executes the skill's runtime claims. A probe Worker runs under real workerd (pywrangler dev, local D1, nothing deployed), and scripts/check_runtime_claims.py compares 12 observations with values recorded from that runtime.

The first run found that the skill's central D1 advice is wrong on the current runtime. With workers-runtime-sdk 1.9.2 and wrangler 4.147.0, at compatibility dates 2025-12-01 and 2026-09-01:

• results.results is a Python list of dict-like workers.rpc.JsDict rows, not a JsProxy.
• row["id"] works. The code in eval 2, which the skill says raises TypeError, runs correctly.
• The fix the skill recommends, results.results.to_py(), raises AttributeError: 'list' object has no attribute 'to_py'.
• .bind(None) stores SQL NULL, the same as .bind(jsnull). Gotcha Synchronous http libraries work fine #3 calls bind(None) WRONG.

The skill text is not changed here; see "Not in this PR".

Why

Leads from the earlier survey: "eval oracles that any answer passes" and "code samples in the skill that are never executed". On main, CI runs Ruff (one Python file) and the install-boundary check. Open PR #10 adds parsing of code blocks, but nothing runs any example against the runtime. tests/evals.json has prose expected_output and no grader, and eval 2's expected output rewards the .to_py() answer that now crashes.

Changes

• tests/runtime/ (new): a probe Worker (src/entry.py, wrangler.jsonc at compat date 2025-12-01, pyproject.toml, uv.lock). It reports types and outcomes; it asserts nothing itself.
• scripts/check_runtime_claims.py (new):
  • starts the probe on a free port and fetches its observations;
  • compares every key with EXPECTED, failing on missing or unexpected keys too;
  • tags each expected value with the skill section that makes the claim and whether the runtime confirms (7) or contradicts (5) it;
  • exits 0 when everything matches, 1 on a mismatch or a Worker HTTP error (terminal, no retry), and 2 when the probe never starts.
• .github/workflows/runtime-claims.yml (new): runs on PRs touching skills/**, the probe, the script or the workflow, plus workflow_dispatch. It is path-filtered because pywrangler runs npx --yes wrangler (unpinned), so a new wrangler release must not turn unrelated PRs red. Measured locally: about 14 s with warm caches.
• .gitignore: the probe's local build outputs.

Audit of the existing gates (no change needed):

• install-boundary can fail: planting skills/python-workers/tests/x.py gave exit 1, and so did planting skills/python-workers/.DS_Store.
• Ruff has a non-empty target.
• There are no skipped or focused tests, || true or continue-on-error.

Validation

All runs were local, on the final tree b8bcf7d in a detached worktree:

Command	Exit
uvx ruff@0.16.0 check .	0 (All checks passed!)
python3 scripts/check_install_boundary.py	0
python3 scripts/check_runtime_claims.py	0 (OK: 12 runtime observations match (5 contradict the skill text))

Baseline on main (637fb40): ruff 0, install-boundary 0. CI history on main: the last run of both jobs, on 637fb40, succeeded.

Planted bugs (mutate.sh in a detached worktree; each file was restored from HEAD and checked with git diff --quiet HEAD). "Old" is main's CI (ruff + install-boundary); "new" is the runtime check:

Planted bug	Old	New
M1: drop dict_converter=Object.fromEntries (gotcha #5)	0 (missed)	1: to_js_dict_fromentries_constructor: expected 'Object', observed 'LiteralMap'
M2: apply the skill's advice rows = results.results.to_py()	0 (missed)	1: the Worker returns HTTP 500 and the observations are missing
M3: drop the python_workers compatibility flag	0 (missed)	2: the probe never starts
M4: bind(jsnull) → bind(None)	0	0 (survives)

M4 survives because the runtime treats the two the same. That is evidence that gotcha #3's D1 warning has no observable contract on this runtime, not a gap in the check.

How the runtime double was checked: the skill text is the "double" here, and every expected value came from workerd plus miniflare D1. One run at compatibility date 2026-09-01 and one at 2025-12-01 gave identical D1 behaviour.

Not in this PR

• Rewrite the skill's D1 guidance (high). The stale text is in:

  • gotchas.md Add install boundary CI check #8, and the read-back part of Synchronous http libraries work fine #3 ("JS null arrives as jsnull": D1 rows read it back as None);
  • patterns.md "D1 Row Conversion" (results.results.to_py());
  • api.md lines 193 and 284;
  • SKILL.md lines 66, 88 and 123–127, and the BEST_PRACTICES.md mirror. PR Check skill examples: parse every code block and enforce last_verified in CI; check runtime claims in workerd by hand #10 owns those two files.

  Reproduce with python3 scripts/check_runtime_claims.py. Before rewriting, the owner should decide whether legacy Workers (an older SDK, or bindings reached outside WorkerEntrypoint) still get raw JsProxy, and document both cases if so. When the text changes, flip the contradicts entries in EXPECTED to confirms.

• Eval 2's expected_output in tests/evals.json rewards .to_py(). Fix it together with the doc rewrite. The evals still have no grader, so any answer "passes"; a deterministic must-mention / must-not-claim grader with a known-good and a known-bad answer is a follow-up.

• Running the skill's own snippets verbatim in workerd rather than in a hand-written probe.

• Pinning wrangler, so the runtime check could run on every PR.

Merge readiness

• Local: ruff, install-boundary and runtime-claims all exit 0 on b8bcf7d.
• CI: not yet run on this branch at the time of writing. GitHub Actions runs on this public repo; runtime-claims will trigger, since this PR touches its paths.
• Conflicts: no file overlaps with open PR Check skill examples: parse every code block and enforce last_verified in CI; check runtime claims in workerd by hand #10 (comm -12 is empty), and git merge-tree with PR Check skill examples: parse every code block and enforce last_verified in CI; check runtime claims in workerd by hand #10's head is clean.
• Open findings: the D1 doc drift above, left to the owner.
• Not verified: the workflow on GitHub-hosted runners (it needs setup-uv to provide uv ≥ 0.12.3; locally uv 0.8.17 was too old and I used uv 0.12.22).
• No independent reviewer was started: no production code or skill text changed (small-mode rule).

Lessons for testing-best-practices

[
 {"id": "python-workers-skill-d1-advice-crashes", "repo": "adewale/python-workers-skill", "tag": "double-fidelity", "value": "bug", "severity": "high",
  "finding": "The skill's D1 advice (call results.results.to_py()) raises AttributeError on the current runtime, where D1 rows already arrive as a Python list of dict-like rows. PR #10's parse-only check could not see this; one run in workerd did.",
  "evidence": {"files": ["skills/python-workers/references/python-workers/gotchas.md:197", "skills/python-workers/references/python-workers/patterns.md:162", "scripts/check_runtime_claims.py"], "planted_bug": "probe applies the skill's advice rows = results.results.to_py()", "old": "ruff+boundary exit 0", "new": "runtime-claims exit 1 (HTTP 500, observations missing)", "commit": "b8bcf7d"},
  "skills": {"testing_best_practices": {"verdict": "covered", "passage": "Take a double's expected values from the real service (recorded or locally emulated), never from the mock author's belief", "path": "testing-best-practices/SKILL.md"},
             "test_audit": {"verdict": "missing", "passage": ""}},
  "prompted_by": "prompt", "references_loaded": ["SKILL.md"],
  "suggested_edit": {"type": "add", "file": "testing-best-practices/SKILL.md", "section": "Prefer real behavior over mocks", "text": "Docs and agent-skill examples that claim runtime behaviour are doubles of that runtime: run their claims once against a local emulation; parsing or compiling the snippets does not check them."},
  "counterexample": "Pure-language docs (syntax, stdlib) where compiling is the whole claim, or runtimes with no local emulator.",
  "eval_seed": {"setup": "A skill doc says 'D1 results are JsProxy; call .to_py()' and CI parses the code blocks.", "bad": "Add more parsing or text assertions on the snippet.", "good": "Run a probe Worker under wrangler dev and record the type of results.results.", "check": "Does the response execute the claim in the real runtime?"},
  "cost_minutes": 20},
 {"id": "python-workers-skill-eval-rewards-wrong-answer", "repo": "adewale/python-workers-skill", "tag": "eval-oracle", "value": "bug", "severity": "medium",
  "finding": "tests/evals.json has prose expected_output and no grader, so any answer passes. Eval 2's expected output rewards the .to_py() fix that now crashes, so even a human grader following it would reward a wrong answer.",
  "evidence": {"files": ["tests/evals.json:13"], "planted_bug": "", "old": "no grader exists", "new": "not changed; follow-up", "commit": "b8bcf7d"},
  "skills": {"testing_best_practices": {"verdict": "covered", "passage": "Every custom oracle, linter, validator, fixer, or eval grader needs a known-good input that passes and a known-bad input that fails", "path": "testing-best-practices/SKILL.md"},
             "test_audit": {"verdict": "missing", "passage": ""}},
  "prompted_by": "prompt", "references_loaded": ["SKILL.md"],
  "suggested_edit": {"type": "clarify", "file": "testing-best-practices/SKILL.md", "section": "Quality beats coverage", "text": "An eval's expected answer is itself a claim about the system; check it against the real runtime before using it as an oracle."},
  "counterexample": "Evals about style or prose quality, with no runtime fact to check.",
  "eval_seed": {"setup": "An eval's expected_output says 'shows the fix with .to_py()'.", "bad": "Add an LLM grader that matches '.to_py()'.", "good": "Verify the expected fix in the runtime first, then write a grader with a known-good and a known-bad answer.", "check": "Was the expected answer itself verified?"},
  "cost_minutes": 5},
 {"id": "python-workers-skill-learning-test-oracle", "repo": "adewale/python-workers-skill", "tag": "observed-output-oracle", "value": "none", "severity": "low",
  "finding": "The runtime check's expected values are observed output from workerd. This is deliberate for a learning test of a third-party runtime: each value is tagged with the doc claim it confirms or contradicts, so a change is a prompt to update the docs, not to re-record silently.",
  "evidence": {"files": ["scripts/check_runtime_claims.py:37"], "planted_bug": "M1 drop dict_converter", "old": "exit 0", "new": "exit 1", "commit": "b8bcf7d"},
  "skills": {"testing_best_practices": {"verdict": "covered", "passage": "use a learning test for third-party APIs", "path": "testing-best-practices/SKILL.md"},
             "test_audit": {"verdict": "covered", "passage": "When the test claims dependency-backed behavior, inspect the dependency source or types directly."}},
  "prompted_by": "skill", "references_loaded": ["SKILL.md"],
  "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""},
  "counterexample": "",
  "eval_seed": {"setup": "Recording the real runtime's behaviour as expected values.", "bad": "Pin the observations with no link to the claim they check.", "good": "Tag each expected value with its source claim and verdict.", "check": "Can a failure message say which doc to fix?"},
  "cost_minutes": 5},
 {"id": "python-workers-skill-surviving-mutant-is-doc-signal", "repo": "adewale/python-workers-skill", "tag": "real-engine", "value": "doc", "severity": "medium",
  "finding": "Planting the bug the skill warns about (bind(None) instead of bind(jsnull)) changed nothing observable in real D1. The surviving mutant shows the warning has no contract on this runtime; it is not a weak test.",
  "evidence": {"files": ["skills/python-workers/references/python-workers/gotchas.md:81", "tests/runtime/src/entry.py"], "planted_bug": "bind(jsnull) -> bind(None)", "old": "exit 0", "new": "exit 0 (survives; both read back NULL)", "commit": "b8bcf7d"},
  "skills": {"testing_best_practices": {"verdict": "missing", "passage": "", "path": "testing-best-practices/SKILL.md"},
             "test_audit": {"verdict": "missing", "passage": ""}},
  "prompted_by": "prompt", "references_loaded": ["SKILL.md"],
  "suggested_edit": {"type": "add", "file": "testing-best-practices/references/mutation-testing.md", "section": "Survivors", "text": "Before strengthening a test to kill a survivor, check in the real engine whether the mutant is equivalent; an equivalent mutant of documented 'wrong' code means the documentation is wrong."},
  "counterexample": "When the engine under test is a double, not the real runtime.",
  "eval_seed": {"setup": "A mutation that swaps a documented-correct call for the documented-wrong one survives.", "bad": "Add assertions until the mutant dies.", "good": "Confirm with the real engine that both calls behave the same, then flag the doc.", "check": "Did the response check equivalence in the real engine?"},
  "cost_minutes": 5},
 {"id": "python-workers-skill-boundary-gate-live", "repo": "adewale/python-workers-skill", "tag": "gate-liveness", "value": "none", "severity": "low",
  "finding": "The install-boundary gate goes red on planted artifacts (a tests/ dir, a .DS_Store) inside the skill directory, and its last result on main is success.",
  "evidence": {"files": ["scripts/check_install_boundary.py"], "planted_bug": "skills/python-workers/tests/x.py; skills/python-workers/.DS_Store", "old": "exit 1 for both", "new": "unchanged", "commit": "637fb40"},
  "skills": {"testing_best_practices": {"verdict": "covered", "passage": "Is it running, and can it fail?", "path": "testing-best-practices/SKILL.md"},
             "test_audit": {"verdict": "missing", "passage": ""}},
  "prompted_by": "skill", "references_loaded": ["SKILL.md"],
  "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""},
  "counterexample": "",
  "eval_seed": {"setup": "A CI script that rejects files in a directory.", "bad": "Assume it works because CI is green.", "good": "Plant a forbidden file and see exit 1.", "check": "Was a red run observed?"},
  "cost_minutes": 2},
 {"id": "python-workers-skill-unpinned-runtime-lane", "repo": "adewale/python-workers-skill", "tag": "campaign-process", "value": "cost", "severity": "low",
  "finding": "pywrangler needs uv >= 0.12.3 (the container had 0.8.17; I installed 0.12.22 into scratch) and runs an unpinned `npx --yes wrangler`. The new lane is therefore path-filtered rather than run on every PR, so unrelated PRs stay green when wrangler changes.",
  "evidence": {"files": [".github/workflows/runtime-claims.yml"], "planted_bug": "", "old": "", "new": "local run 14 s with warm caches", "commit": "b8bcf7d"},
  "skills": {"testing_best_practices": {"verdict": "covered", "passage": "Local emulation of the real runtime (workerd/vitest-pool-workers, Pyodide in Node, `wrangler dev`, SQLite loaded from migrations).", "path": "testing-best-practices/SKILL.md"},
             "test_audit": {"verdict": "missing", "passage": ""}},
  "prompted_by": "prompt", "references_loaded": ["SKILL.md"],
  "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""},
  "counterexample": "",
  "eval_seed": {"setup": "Adding a CI job that downloads an unpinned toolchain.", "bad": "Run it on every PR.", "good": "Pin it, or path-filter it to the files it checks.", "check": "Can an unrelated PR turn red?"},
  "cost_minutes": 5},
 {"id": "python-workers-skill-chained-push-denied", "repo": "adewale/python-workers-skill", "tag": "campaign-process", "value": "none", "severity": "low",
  "finding": "I appended a stray `git push` of another PR's branch to a read-only overlap-check command. The permission classifier denied the whole command, and nothing was pushed. Keep verification commands free of writes.",
  "evidence": {"files": [], "planted_bug": "", "old": "", "new": "denied; rerun without the push", "commit": ""},
  "skills": {"testing_best_practices": {"verdict": "missing", "passage": "", "path": ""}, "test_audit": {"verdict": "missing", "passage": ""}},
  "prompted_by": "neither", "references_loaded": ["SKILL.md"],
  "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""},
  "counterexample": "",
  "eval_seed": {"setup": "", "bad": "", "good": "", "check": ""},
  "cost_minutes": 1}
]

References read: testing-best-practices SKILL.md only (no references/ file), and openclaw test-audit/SKILL.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F2Hz94JHpV9RvpukfHvtjA

---

Generated by Claude Code

[adewale]

Folded into #10 at the owner's request; its commits are in #10's history.

---

Generated by Claude Code