Skip to content

Test audit: unbreak main (pin fastapi below 0.142), byte-level issue 2 oracle, wire-level issue 5 oracle - #4

Closed
adewale wants to merge 3 commits into
mainfrom
claude/test-audit-campaign
Closed

adewale wants to merge 3 commits into
mainfrom
claude/test-audit-campaign

Conversation

@adewale

@adewale adewale commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Product fix (separate commit): main is currently broken. FastAPI 0.142.0 shipped on 2026-09-29 and depends on opentelemetry-api. That package calls uuid4() at import time, and Python Workers forbid randomness during startup. Because pywrangler vendors the newest unpinned fastapi, the issue 2 and issue 4 Workers no longer start. This PR pins fastapi<0.142 in both directories.
  • test_2_fastapi_r2_streaming: checks the bytes of GET /read, not just the length. The seed is now non-periodic, so a reordered or repeated chunk changes the bytes.
  • test_5_sync_http_libraries: asserts on what a test-owned loopback echo server actually received, instead of on the Worker's own report. It also keeps one live call to httpbin.org, and asserts that the test-only ?echo= override rejects non-loopback hosts.

Why

  • Baseline on main: CI=1 uv run pytest -vv -rxXs → exit 1. test_2 ERRORs with Server failed to start within 300 seconds; the dev log shows OSError: [Errno 29] Randomness is not allowed while a Worker is starting, raised from fastapi/telemetry/_api.py → opentelemetry/context/__init__.py → uuid4().
    • pywrangler vendored fastapi 0.142.2 into python_modules, while the directory's uv.lock says 0.133.0. pywrangler resolves from pyproject.toml, not from that lock.
    • CI last ran on main on 2026-05-02: run 25253570446, job test, success. That predates the breakage, so nothing reported it.
  • Issue 2 oracle: /read was checked by length only. The old seed bytes(range(256)) * 512 repeats every 256 bytes and R2 chunks are 4096 bytes, so even an exact byte comparison could not see reordered chunks. This shows up in the planted-bug table below.
  • Issue 5 oracle:
    • The expected values came from the Worker's own headers_sent field, and the observed values came from the Worker's report of httpbin's echo.
    • So a Worker that skipped urllib3 and copied the requests result stayed green.
    • So did a Worker that never sent the headers but reported HEADERS as received.

Changes

Commit Files What
1f5633c 2-fastapi-r2-streaming/pyproject.toml, uv.lock; 4-r2-large-binary-roundtrip/pyproject.toml, uv.lock fastapi<0.142, with a comment explaining why. Lockfiles were regenerated by uv lock; only the requires-dist specifier changed.
b0ffcb4 2-fastapi-r2-streaming/src/worker.py /seed writes 32768 big-endian uint32 words, where word i holds i. It is still 131072 bytes.
tests/test_examples.py (test_2) Asserts that no word is misplaced and reports the first misplaced word. A bare bytes == made pytest spend 453 s building the failure diff; this assertion fails in 14 s.
tests/echo_server.py (new) Loopback echo server that answers like httpbin /headers (Title-Cased names) and records every request it receives.
5-sync-http-libraries/src/entry.py, README.md Accepts a loopback-only ?echo= override and tags each request with ?client=requests or ?client=urllib3.
tests/test_examples.py (test_5) Asserts that the echo server recorded exactly one request per library, each carrying both headers. Expected values are literals, not the Worker's headers_sent.
4ac741a (review fixes) 5-sync-http-libraries/src/entry.py, README.md, tests/test_examples.py Closes a parser-differential bypass in the ?echo= check: http://evil.com\@127.0.0.1/ passed urllib.parse, but requests/urllib3 send that request to evil.com. The Worker now rebuilds the URL from its parsed host, port and path, and answers 400. test_5 asserts the 400 and calls the default httpbin.org path once, so a real HTTPS host stays covered.

Tiers: nothing was deleted or downgraded. test_2 and test_5 still run real workerd through pywrangler dev. test_5 gained a local wire-level oracle and kept its live HTTPS call.

Validation

All runs were local in this container. GitHub Actions did not run.

Exact CI commands on the final tree (4ac741a), starting with every python_modules and .venv-workers removed:

uv sync --dev && (cd 2-fastapi-r2-streaming && uv run pywrangler dev --help)   # exit 0
CI=1 nice -n 10 uv run pytest -vv -rxXs                                         # exit 0
  test_2 PASSED, test_3 XFAIL, test_5 PASSED, test_4a/4b/4c SKIPPED (deploy-only)
  2 passed, 3 skipped, 1 xfailed
uv run pytest --collect-only -q                                                  # 6 tests collected (same as main)

Lint. CI does not run lint.

Red → green for the product fix:

Command Result
Red, issue 2 main: CI=1 uv run pytest -vv -rxXs exit 1, test_2 ERROR (fastapi 0.142.2 vendored, Errno 29)
Red, issue 4 main's 4-*/pyproject.toml, python_modules removed, uv run pywrangler dev fastapi 0.142.2 vendored, OSError: [Errno 29]
Green, issue 2 CI=1 uv run pytest -k test_2 exit 0 (fastapi 0.141.1)
Green, issue 4 uv run pywrangler dev Ready on, GET / serves (fastapi 0.141.1)

Bypass control: the new test_5 against the b0ffcb4 Worker gives exit 1 (assert 500 == 400). Against 4ac741a it gives exit 0.

Double fidelity: the same 5-sync-http-libraries Worker under wrangler dev returned identical results for requests and urllib3 whether it called httpbin.org or tests/echo_server.py. httpbin also Title-Cases lower-cased header names, as the echo server does.

Planted bugs. The script plants each bug in the Worker, runs the old tests (origin/main:tests/test_examples.py) and the new tests with CI=1 uv run pytest -k <test>, then restores the file with git checkout HEAD -- <file>. After each run, git diff --quiet HEAD was clean and git status was empty.

Planted bug Old tests (main) New tests
M1 /seed writes zeros (same length) survived (exit 0) killed (exit 1, word 1 holds 0)
M2 /read joins chunks in reverse order survived (exit 0) killed (exit 1, word 0 holds 31744). With the old periodic seed, the new byte check also survived.
M3 /read drops the last chunk killed (exit 1) killed (exit 1)
M4 urllib3 never called; result copied from requests survived (exit 0) killed (exit 1, ['requests'] == ['requests', 'urllib3'])
M5 requests sends no custom headers; Worker reports HEADERS as received survived (exit 0) killed (exit 1, 'python-requests/2.34.2' == 'sync-repro/1.0')
M6 urllib3 drops X-Custom killed (exit 1) killed (exit 1)

Independent review. Two read-only reviewer subagents checked the change: one for test effectiveness, one for production correctness.

  • Fixed:
    • the ?echo= host bypass, now a 400, with a test;
    • the lost live-HTTPS contract;
    • the 500 on a rejected override.
  • Accepted as residual risk:
    • The client tag is written by the Worker, so a Worker that used requests for both calls and labelled one urllib3 would pass. Telling the libraries apart would mean pinning their default headers.
    • The echo server collapses duplicate headers where httpbin would join them.
    • from echo_server import … relies on pytest's default prepend import mode.

Not in this PR

These follow-ups mostly sit in files or hunks that open PRs own.

Merge readiness

  • Local: the exact CI commands exit 0 on 4ac741a. Pre-commit is clean on the changed lines; the only failure is pre-existing (above).
  • CI: not run. Per the campaign brief, GitHub Actions has not run on the owner's private repos since 2026-09-04 (spending limit). The last semgrep run on main was 2026-07-02, and the last CI run on main was 2026-05-02 (test: success).
  • Conflicts: none.
  • Open findings: none blocking. Residual risks are listed under Validation.
  • Not verified:
    • deployed tests 4a, 4b and 4c (no deploys were made);
    • GitHub Actions;
    • the pin's effect on a deployed issue 4 Worker. Only pywrangler dev startup was checked.

Lessons for testing-best-practices

[
{"id": "python-workers-issues-unpinned-vendored-dep", "repo": "adewale/python-workers-issues", "tag": "gate-liveness", "value": "bug", "severity": "high", "finding": "main broke on 2026-09-29 with no code change: pywrangler vendored fastapi 0.142.2 (unpinned in pyproject; the per-directory uv.lock pinning 0.133.0 is not what pywrangler installs), whose opentelemetry import calls uuid4() at Worker startup. CI last ran 2026-05-02, so nothing noticed.", "evidence": {"files": ["2-fastapi-r2-streaming/pyproject.toml:11", "4-r2-large-binary-roundtrip/pyproject.toml:11"], "planted_bug": "none (real upstream drift); red control = main's pyproject with python_modules removed", "old": "CI=1 uv run pytest -vv: exit 1, test_2 ERROR 'Server failed to start within 300 seconds' (OSError Errno 29 in dev log)", "new": "exit 0, 2 passed / 1 xfailed / 3 skipped", "commit": "1f5633c"}, "skills": {"testing_best_practices": {"verdict": "covered", "passage": "The artifact under test must be the artifact deployed: lockfile parity,", "path": "references/gate-integrity.md"}, "test_audit": {"verdict": "covered", "passage": "a retained test that fails on the baseline: treat it as a possible product\n  bug, reproduce it, and repair the owner rather than deleting it."}}, "prompted_by": "prompt", "references_loaded": ["SKILL.md", "references/gate-integrity.md", "references/antipatterns.md", "references/python.md", "references/vcr-cassettes.md", "references/differential-testing.md"], "suggested_edit": {"type": "clarify", "file": "references/gate-integrity.md", "section": "5. Test the tester, and trust the artifact", "text": "Lockfile parity means the lockfile the bundler or vendoring step actually reads: some tools (pywrangler's python_modules) re-resolve from pyproject and ignore a present uv.lock, so pin runtime-critical dependencies where the bundler looks."}, "counterexample": "Libraries meant to float on dependency ranges should not pin; there the fix is a scheduled or lower/upper-bound CI matrix, not a pin.", "eval_seed": {"setup": "Worker dir with pyproject 'fastapi' unpinned and a uv.lock at an old version; bundler vendors latest from PyPI.", "bad": "Declare the lockfile as proof of reproducibility and call the test failure a flake.", "good": "Inspect the vendored dist-info version, find the breaking release, pin in pyproject, show red on old pin and green on new.", "check": "Does the response identify which file the bundler reads and produce red/green evidence?"}, "cost_minutes": 25},
{"id": "python-workers-issues-periodic-fixture-hides-reorder", "repo": "adewale/python-workers-issues", "tag": "vacuous-pass", "value": "cannot-fail-test", "severity": "medium", "finding": "The issue 2 seed was bytes(range(256))*512; R2 delivers 4096-byte chunks, so every chunk is identical and even an exact byte comparison could not detect reordered or repeated chunks. The planted 'reverse chunks' bug survived the new byte check until the seed became 32768 big-endian words with word i == i.", "evidence": {"files": ["2-fastapi-r2-streaming/src/worker.py:55", "tests/test_examples.py:24"], "planted_bug": "/read: b''.join(reversed(parts))", "old": "exit 0 (survived)", "new": "with periodic seed: exit 0 (survived); with word-index seed: exit 1 'word 0 holds 31744'", "commit": "b0ffcb4"}, "skills": {"testing_best_practices": {"verdict": "missing", "passage": "", "path": "references/test-data-builders.md"}, "test_audit": {"verdict": "covered", "passage": "names or fixtures that promise more than the input exercises"}}, "prompted_by": "prompt", "references_loaded": ["SKILL.md", "references/gate-integrity.md", "references/antipatterns.md", "references/python.md"], "suggested_edit": {"type": "add", "file": "SKILL.md", "section": "Write mode", "text": "For ordering or integrity checks on chunked/streamed data, make the payload non-periodic at the chunk size (e.g. each word encodes its own offset); a repeating pattern makes reordered or duplicated chunks byte-identical."}, "counterexample": "Tests that only care about length or throughput (e.g. memory-limit probes) gain nothing from position-encoded payloads.", "eval_seed": {"setup": "Endpoint streams 128 KB from storage in 4 KB chunks; seed is bytes(range(256))*512.", "bad": "Add assert body == bytes(range(256))*512 and claim reordering is covered.", "good": "Change the seed so each 4-byte word encodes its index, assert the first misplaced word, and show a reverse-chunks mutant is killed.", "check": "Does a reversed-chunks mutant fail the proposed test?"}, "cost_minutes": 15},
{"id": "python-workers-issues-pytest-bytes-diff-cost", "repo": "adewale/python-workers-issues", "tag": "cost", "value": "cost", "severity": "medium", "finding": "A plain `assert body == expected` on 128 KB of bytes made pytest's assertion rewriting compute a diff for 7.5 minutes when it failed (CI runs -vv). Reporting the first mismatching word instead failed in 14 s.", "evidence": {"files": ["tests/test_examples.py:24"], "planted_bug": "seed written as zeros", "old": "n/a", "new": "bytes ==: exit 1 after 453 s; first-mismatch assertion: exit 1 after 13.7 s", "commit": "b0ffcb4"}, "skills": {"testing_best_practices": {"verdict": "missing", "passage": "", "path": "references/python.md"}, "test_audit": {"verdict": "missing", "passage": ""}}, "prompted_by": "neither", "references_loaded": ["SKILL.md", "references/python.md"], "suggested_edit": {"type": "add", "file": "references/python.md", "section": "Assertions", "text": "Do not compare large bytes/str/sequences with a bare `==` under pytest: the failure explanation can take minutes. Assert lengths, then report the first differing offset."}, "counterexample": "Small payloads (a few KB) are fine with bare ==, and the full diff is more useful there.", "eval_seed": {"setup": "Test compares a 128 KB HTTP body to an expected bytes literal under pytest -vv.", "bad": "assert resp.content == expected", "good": "assert len(...) == N, then find the first mismatching offset and assert it is None with a message", "check": "Time the failing run with a corrupted body."}, "cost_minutes": 10},
{"id": "python-workers-issues-self-reported-oracle", "repo": "adewale/python-workers-issues", "tag": "observed-output-oracle", "value": "cannot-fail-test", "severity": "medium", "finding": "test_5 took its expected headers from the Worker's own headers_sent field and its observations from the Worker's report of httpbin's echo; a Worker that skipped urllib3 or reported HEADERS without sending them stayed green. A test-owned loopback echo server makes the wire the oracle.", "evidence": {"files": ["tests/test_examples.py:78", "tests/echo_server.py:20"], "planted_bug": "M4 urllib3 never called, result copied; M5 requests sends no headers, Worker reports HEADERS", "old": "exit 0 / exit 0 (both survived)", "new": "exit 1 / exit 1 (both killed)", "commit": "b0ffcb4"}, "skills": {"testing_best_practices": {"verdict": "covered", "passage": "Do not recompute the expectation with the SUT's own logic or constants, because the test can share its bug and stay green.", "path": "SKILL.md"}, "test_audit": {"verdict": "covered", "passage": "expected values produced by the helper or renderer under test;"}}, "prompted_by": "skill", "references_loaded": ["SKILL.md", "references/antipatterns.md"], "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""}, "counterexample": "", "eval_seed": {"setup": "Worker calls an echo service and returns {'headers_sent': H, 'received': echo}; test asserts received == headers_sent.", "bad": "Keep the self-report comparison and only add more fields.", "good": "Run a local echo server the test owns, point the Worker at it, assert on what the server recorded with literal expected values.", "check": "Does a Worker that fabricates 'received' fail?"}, "cost_minutes": 30},
{"id": "python-workers-issues-echo-double-fidelity", "repo": "adewale/python-workers-issues", "tag": "double-fidelity", "value": "none", "severity": "low", "finding": "The local echo server stands in for httpbin.org. Checked once through the real Worker under wrangler dev: both produced identical results for requests and urllib3, and both Title-Case lower-cased header names.", "evidence": {"files": ["tests/echo_server.py:18"], "planted_bug": "", "old": "httpbin.org results", "new": "local echo results; equal: True", "commit": "b0ffcb4"}, "skills": {"testing_best_practices": {"verdict": "covered", "passage": "Take a double's expected values from the real service (recorded or locally emulated), never from the mock author's belief;", "path": "SKILL.md"}, "test_audit": {"verdict": "missing", "passage": ""}}, "prompted_by": "prompt", "references_loaded": ["SKILL.md", "references/vcr-cassettes.md"], "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""}, "counterexample": "", "eval_seed": {"setup": "Replace a public echo API with a local echo server in a test.", "bad": "Write the echo server from memory and never compare it to the real API.", "good": "Run the same client against both once and diff the parsed results, then note the check in the PR.", "check": "Is there recorded evidence the double and the real service agree on the asserted fields?"}, "cost_minutes": 5},
{"id": "python-workers-issues-lost-https-contract", "repo": "adewale/python-workers-issues", "tag": "lost-check", "value": "lost-check", "severity": "medium", "finding": "Moving test_5 to the loopback echo server silently dropped its only proof that sync requests/urllib3 reach a real HTTPS host (TLS, DNS) from a Worker, the behavior the issue documents. The reviewer caught it; test_5 now also calls the default httpbin.org path once.", "evidence": {"files": ["tests/test_examples.py:101"], "planted_bug": "(contract) default ECHO_URL path unreached by new test", "old": "covered", "new": "b0ffcb4: not covered; 4ac741a: covered again", "commit": "4ac741a"}, "skills": {"testing_best_practices": {"verdict": "covered", "passage": "Never silently downgrade a tier: list removed or replaced tests by tier,", "path": "SKILL.md"}, "test_audit": {"verdict": "covered", "passage": "Each contract has\n   one primary test owner at the strongest boundary;"}}, "prompted_by": "reviewer", "references_loaded": ["SKILL.md"], "suggested_edit": {"type": "clarify", "file": "SKILL.md", "section": "Upgrade mode", "text": "Replacing a live external call with a local double is a tier change too: keep one real call for the transport contract (TLS/DNS/host) unless it is proven elsewhere."}, "counterexample": "Where the live service is paid, rate-limited or unavailable in CI, keep the real call as an opt-in or scheduled check instead.", "eval_seed": {"setup": "Test hits httpbin.org over HTTPS; agent replaces it with a loopback http echo server.", "bad": "Remove all live calls and claim equal coverage.", "good": "Keep one assertion on the real HTTPS path, or name the lost transport contract as a known gap.", "check": "Would a typo in the production URL or an HTTPS-only regression still fail a test?"}, "cost_minutes": 10},
{"id": "python-workers-issues-test-seam-parser-differential", "repo": "adewale/python-workers-issues", "tag": "other", "value": "bug", "severity": "medium", "finding": "The new loopback-only ?echo= test seam was bypassable: urllib.parse read http://evil.com\\@127.0.0.1/ as host 127.0.0.1 while requests/urllib3 sent it to evil.com. Fixed by rejecting userinfo/backslash/query/fragment, rebuilding the URL from parsed parts, returning 400; test_5 asserts the 400.", "evidence": {"files": ["5-sync-http-libraries/src/entry.py:26"], "planted_bug": "previous validator (b0ffcb4)", "old": "new test_5 vs b0ffcb4 Worker: exit 1 'assert 500 == 400'", "new": "exit 0", "commit": "4ac741a"}, "skills": {"testing_best_practices": {"verdict": "missing", "passage": "", "path": "references/go.md"}, "test_audit": {"verdict": "covered", "passage": "4. Does it need a production seam (export, flag, wrapper, injection hook) that no\n   production caller needs? If yes, move the test to the real boundary instead."}}, "prompted_by": "reviewer", "references_loaded": ["SKILL.md"], "suggested_edit": {"type": "add", "file": "SKILL.md", "section": "Prefer real behavior over mocks", "text": "A test seam reachable in a deployed artifact (a URL override, header, flag) is attack surface: validate it with the same parser the consumer uses, and test the rejection."}, "counterexample": "Seams compiled out of production builds (build tags, test-only modules) do not need hostile-input tests.", "eval_seed": {"setup": "Worker accepts ?target= restricted to localhost via urllib.parse hostname, then calls requests.get(target).", "bad": "Check hostname in ('127.0.0.1','localhost') and stop.", "good": "Reject userinfo/backslash, rebuild the URL from parsed parts, test a parser-differential payload returns 400.", "check": "Does http://evil.example\\@127.0.0.1/ get rejected?"}, "cost_minutes": 15},
{"id": "python-workers-issues-imperative-xfail", "repo": "adewale/python-workers-issues", "tag": "always-green", "value": "cannot-fail-test", "severity": "high", "finding": "Lead verified: resolved issue 2 still calls pytest.xfail() on truncation, and tests 3/4a/4b use imperative xfail, so a regression or an upstream fix stays green. Not changed here because open PR #3 owns those lines and fixes it with strict markers.", "evidence": {"files": ["tests/test_examples.py:41", "tests/test_examples.py:65"], "planted_bug": "none here (PR #3 shows red/green)", "old": "", "new": "", "commit": ""}, "skills": {"testing_best_practices": {"verdict": "covered", "passage": "Use `@pytest.mark.xfail(strict=True, reason=...)` with the bug condition as the assertion, so an upstream fix turns the run red and prompts cleanup.", "path": "references/python.md"}, "test_audit": {"verdict": "missing", "passage": ""}}, "prompted_by": "skill", "references_loaded": ["SKILL.md", "references/python.md", "references/gate-integrity.md"], "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""}, "counterexample": "", "eval_seed": {"setup": "", "bad": "", "good": "", "check": ""}, "cost_minutes": 2},
{"id": "python-workers-issues-startup-spin-on-dead-server", "repo": "adewale/python-workers-issues", "tag": "cost", "value": "cost", "severity": "low", "finding": "When pywrangler dev exited at once (the fastapi failure), the conftest kept reading EOF until its 300 s CI budget expired, so the baseline took 5m42s to report a 10 s failure; each test also leaves ~7 wrangler/workerd processes behind. Both are fixed in open PR #3's conftest; not touched here.", "evidence": {"files": ["tests/conftest.py:81"], "planted_bug": "", "old": "ERROR after 300 s", "new": "(PR #3)", "commit": ""}, "skills": {"testing_best_practices": {"verdict": "covered", "passage": "helper-level retry loops; raised global timeouts or `isCI ? a : b` budgets.", "path": "SKILL.md"}, "test_audit": {"verdict": "missing", "passage": ""}}, "prompted_by": "neither", "references_loaded": ["SKILL.md"], "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""}, "counterexample": "", "eval_seed": {"setup": "", "bad": "", "good": "", "check": ""}, "cost_minutes": 5},
{"id": "python-workers-issues-merge-tree-vacuous", "repo": "adewale/python-workers-issues", "tag": "campaign-process", "value": "none", "severity": "medium", "finding": "My first overlap check reported 'no shared files, merge-tree clean' because the temporary commit contained only the staged new file (a stash round-trip unstaged the edits). The repeat with the full change set showed tests/test_examples.py shared with PR #3 (merge-tree still clean). Overlap checks need a non-vacuity check: the file list must match `git diff --name-only origin/main...` of the real change.", "evidence": {"files": [], "planted_bug": "", "old": "shared: (empty)", "new": "shared: tests/test_examples.py; merge-tree clean", "commit": "4ac741a"}, "skills": {"testing_best_practices": {"verdict": "covered", "passage": "Assert the precondition (non-empty, HTTP 200, font loaded, n > 0) before\nthe universal claim and print the denominator.", "path": "references/antipatterns.md"}, "test_audit": {"verdict": "missing", "passage": ""}}, "prompted_by": "neither", "references_loaded": ["references/antipatterns.md"], "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""}, "counterexample": "", "eval_seed": {"setup": "", "bad": "", "good": "", "check": ""}, "cost_minutes": 5},
{"id": "python-workers-issues-shared-tree-mutation", "repo": "adewale/python-workers-issues", "tag": "campaign-process", "value": "none", "severity": "low", "finding": "The mutation script edits Worker sources in the shared checkout; the test reviewer saw it could collide with concurrent runs and declined to run tests. Separately, `pkill -f wrangler` from the agent's shell killed that shell, because its own command line matched. The prompt could say: run mutants in a git worktree, and reap dev servers by binary path, not by pattern.", "evidence": {"files": [], "planted_bug": "", "old": "", "new": "", "commit": ""}, "skills": {"testing_best_practices": {"verdict": "covered", "passage": "A hand-seeded fault in a scratch copy or worktree, never the shared tree:", "path": "references/gate-integrity.md"}, "test_audit": {"verdict": "missing", "passage": ""}}, "prompted_by": "reviewer", "references_loaded": ["references/gate-integrity.md"], "suggested_edit": {"type": "none", "file": "", "section": "", "text": ""}, "counterexample": "", "eval_seed": {"setup": "", "bad": "", "good": "", "check": ""}, "cost_minutes": 5}
]

🤖 Generated with Claude Code

https://claude.ai/code/session_01SNmM8CmBcqJZ87FWEH5DiF

claude added 3 commits October 3, 2026 03:38
fastapi 0.142.0 (2026-09-29) depends on opentelemetry-api, whose context
module calls uuid4() at import time. Python Workers forbid randomness
during startup, so `pywrangler dev` fails with
"OSError: [Errno 29] Randomness is not allowed while a Worker is starting".
pywrangler resolves the unpinned "fastapi" requirement to the newest
release (the directory's uv.lock is not used for python_modules), so main
broke without a code change.

Red: on main, CI=1 uv run pytest -vv errors at setup of
test_2_fastapi_r2_streaming ("Server failed to start within 300 seconds"),
and `pywrangler dev` in 4-r2-large-binary-roundtrip vendors fastapi
0.142.2 and dies with the same OSError.
Green: with the pin, fastapi 0.141.1 is vendored, test_2 passes and the
issue 4 Worker serves GET /.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SNmM8CmBcqJZ87FWEH5DiF
…eports

test_2_fastapi_r2_streaming checked only the length of GET /read. The
seed was bytes(range(256)) * 512, which repeats every 256 bytes, so even a
byte check could not see reordered 4 KB R2 chunks. /seed now writes 32768
big-endian words where word i holds i, and the test reports the first
misplaced word (a plain bytes == made pytest diff 128 KB for 7.5 minutes
on failure).

test_5_sync_http_libraries took its expected headers from the Worker's
own "headers_sent" field and its observations from the Worker's report of
what httpbin.org echoed. A Worker that skipped a library or reported
HEADERS without sending them stayed green. The Worker now accepts a
loopback-only ?echo= URL; the test runs tests/echo_server.py, a local
httpbin-style echo server, and asserts on the requests that server
recorded: one per library, each with both headers. It no longer needs
outbound network. Against the same Worker, httpbin.org and the local echo
server produced identical results.

Planted bugs, old tests (main) vs new tests:
  seed zeroed                      survived / killed
  /read chunks reversed            survived / killed
  /read drops last chunk           killed   / killed
  urllib3 skipped, result copied   survived / killed
  requests sends no headers,
    Worker reports HEADERS         survived / killed
  urllib3 drops X-Custom           killed   / killed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SNmM8CmBcqJZ87FWEH5DiF
Review findings on the previous commit:

- `?echo=http://evil.com\@127.0.0.1/` passed the loopback check, because
  urllib.parse reads the host as 127.0.0.1 while requests/urllib3 send the
  request to evil.com. The Worker now rejects userinfo, backslashes,
  queries and fragments, rebuilds the URL from the parsed host, port and
  path, and answers 400 instead of raising a 500. test_5 asserts the 400
  (red on the previous Worker: 500 == 400 fails).
- Moving test_5 to the local echo server dropped coverage of the default
  path: sync requests/urllib3 to a real HTTPS host. test_5 calls it once
  more and checks both headers; the wire-level assertions stay on the
  local echo server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SNmM8CmBcqJZ87FWEH5DiF
@adewale adewale changed the title Test audit: unbreak main (fastapi<0.142), byte-level issue 2 oracle, wire-level issue 5 oracle Test audit: unbreak main (pin fastapi below 0.142), byte-level issue 2 oracle, wire-level issue 5 oracle Oct 3, 2026
adewale pushed a commit that referenced this pull request Oct 5, 2026
adewale pushed a commit that referenced this pull request Oct 5, 2026
PR #4 left this as a follow-up because PR #3 rewrote the /stream lines.
With both folded together, assert the streamed body against the
word-index seed the same way /read is checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012E6xQEfTtV2ws3dCSMnoCo

adewale commented Oct 5, 2026

Copy link
Copy Markdown
Owner Author

Folded into #3 at the owner's request; its commits are in #3's history.


Generated by Claude Code

@adewale adewale closed this Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants