Repository navigation
Real-server HTTPS sprint: streaming deframer + first live github.com/browserleaks.com GETs (HTTP 200) - #123
Merged
Merged
Conversation
Groundwork for the "real public HTTPS server at 48 MHz" sprint. No 6502 code changes; both additions are host-side measurement tools. tools/probe_server_tls.py (new) Sends a byte-faithful replica of the ClientHello src/tls_handshake.s emits and reports the three things that decide reachability: whether the server accepts our very narrow offer (one cipher suite, one group, ECDSA only), how large its records are against the 548 B TLS_REC_BUF_MAX, and how long it tolerates a silent client. Measured 2026-08-15: github.com 529 B / 90 s, browserleaks.com 529 B / 45 s, en.wikipedia.org 529 B / >100 s (leaf 1636 B exceeds cert_buf), cloudflare.com 2714 B / 15 s. Normal OpenSSL servers DO honor max_fragment_length, so their records already fit; Cloudflare ignores it AND enforces a hard 15 s connect-to-first-request deadline, which is why it is out of scope. tools/uci/rig_https_local.py Opt-in phase timing via PHASE_TIMING=1 (PHASE_POLL tunes the interval); default behaviour is unchanged. Writes phase_timing.txt to the artifact dir alongside a per-marker cost rollup. Deliberately does NOT reuse tests/rig_vice_https_macos.py's "latest progress needle on screen" heuristic: measured here it yielded 3 transitions across a 42 s handshake, because the screen scrolls and the markers repeat once per encrypted record. Taking the last non-empty line is immune to both. tls_state ($A84A) is not a usable alternative either — it sits under BASIC ROM, so a DMA read returns ROM bytes. W0 result (U64E @ 48 MHz, comb, local listener, PASS/200): client Finished at 31.1 s, of which the server sees 24.7 s of silence — 3.6x headroom against github.com, 45% against browserleaks.com. X25519 is now co-dominant with the verify (8.0 s vs 16.3 s), not a minor term. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The HTTPS GET target (host data, SNI copy, DNS lookup) was hardcoded to www.foo.bar in src/boot.s. Add `make HTTPS_HOST=<host>` mirroring the HTTPS_PORT idiom. ca65's -D is numeric-only, so the string travels via a generated build/https_host.inc (already on the -I build path), with a content-compared FORCE rule so a changed host rebuilds boot.o without `make clean` — on a change the rule also deletes boot.o, because Make 3.81's 1-second mtime resolution otherwise leaves the previous host's object counting as up to date (measured). The link step keeps the repo-wide same-second caveat; comment documents it. Default build proven byte-identical (b574b344cdca...) with the default string; en.wikipedia.org/github.com overrides verified to embed and to revert cleanly; UCI comb variant links. A .assert refuses hosts longer than the 63-char SNI copy guard at build time. http_host_foo renamed to http_host_target (no external consumers). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The W0 phase-timing commit (13b5063) inserted an 'if phase_timing:' block between the sentinel poll loop and its 'else:', silently rebinding the else from the while (runs only on exhaustion) to the if. Result at 13b5063: a successful default-mode run (PHASE_TIMING unset) reported TIMEOUT and exited 1, and a PHASE_TIMING=1 run that genuinely timed out skipped the TIMEOUT diagnostics entirely. Track the break explicitly (sentinel_seen) and route the TIMEOUT path off that flag, independent of phase timing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ustness Real web servers (github.com, browserleaks.com, en.wikipedia.org) send responses the local listener never did: Transfer-Encoding: chunked with no Content-Length, and header blocks far larger than http_resp_buf that span many TLS records. Previously chunk framing landed raw in the body buffer. - check_transfer_encoding: case-insensitive 'Transfer-Encoding: chunked' header match (same single-SP-after-colon shape as check_content_length), sets http_chunked. - http_state_body_chunked: strips chunk framing byte-by-byte with persistent sub-state (hex size lines incl. chunk extensions, per-chunk CRLF, terminal 0/trailers/blank-line), copying only payload into http_resp_buf. Terminates C=0 on the terminal chunk exactly like the Content-Length match path. - Truncate-at-capacity: past 512 B payload is consumed and discarded (http_resp_len caps at the captured prefix) and the parser still finds the terminal chunk, so oversized bodies end cleanly instead of falling to the ~65k-poll timeout. - Header scanning was already incremental (persistent line-state across records, 31-byte line cap); now proven against >2.5 KB header blocks split across 14 spans by new tests. - Memory: body-state handler moved out of CODE into new HTTP_AUX_CODE segment (UCI: LOADER, 588 B free pre-W4; ip65: CRYPTO_OVERLAY — ip65 LOADER had only 8 B free). CODE shrank 64 B on both backends. New BSS: 4 B (http_chunked, http_chunk_state, http_chunk_rem). PRG sizes unchanged (47,105 B ip65 / 62,977 B uci-comb). - tools/test_http.py: span-mode drive (the TLS input path) + 5 new vector groups: chunked single-span, 5-span chunked w/ split header name + split chunk data + uppercase hex + chunk extension + split terminal CRLF, 700 B chunked truncation, 2.6 KB headers w/ buried Content-Length, chunked + large headers. 48/48 (was 27). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TLS 1.3 handshake + GET against a real public server (HTTPS_TARGET, default github.com) through the UCI backend. No inline listener, no repo certs; pass criteria are C64-side only: sentinel + http_get C=0 + http_status == EXPECT_STATUS (default 200). Follows every established hardware convention: turbo set BEFORE reset/run_prg with the read-skip-write pattern, preflight_reu() under the DeviceLock right after enable_uci (the comb PRG claims REU bank 2), all DMA addresses from build_policy_and_arbiter_with_overlay_carveout(), and a rig_ name so pytest never collects it. C64_INIT_WAIT defaults to 75 s on a comb build (ec_precompute_256 boot pass), detected from build/labels.txt via the banks equate + ec_precompute symbol union. PHASE_TIMING defaults ON here (opt-in in the local rig) so live runs produce the W0 phase table; artifacts are kept on PASS by default. The target reaches the C64 via http_host_ptr DMA -> http_get, whose prologue copies the hostname into tls_hostname for SNI, so the rig works with today's PRG and does not depend on W3's HTTPS_HOST build define. en.wikipedia.org is selectable but warned about: its 1636 B leaf exceeds the 1536 B cert_buf. Hardware-free coverage: --selfcheck exercises env handling, the comb init-wait derivation, and the trampoline's opcode structure (verified against both synthetic labels and a real comb build's labels.txt). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New cert profile 'p256-chain' (kwarg or HTTPS_LISTENER_CERT_PROFILE=, mirroring the p384 selector): the existing P-256 test leaf, unchanged, plus two padded throwaway intermediates minted by chain_certs.py with the stdlib-only gen_certs toolkit. Chain total ~3.0 KB (real targets measured 2026-08-21: github 2719 B, browserleaks 3430 B, wikipedia 4114 B), which forces the server flight into 11 records of <=529 B against the C64's ClientHello — the real-record-count overhead becomes benchable with no internet egress, while the C64 still verifies the same leaf. Padding is one non-critical private-OID extension per intermediate (CHAIN_PAD_BYTES tunes it); linkage is deliberately fake — nothing on either side parses the intermediates, byte/record count is what is under test. The chain file regenerates whenever the leaf does. Host-side selftest (chain_certs.py --selftest, loopback, no hardware): stdlib ssl client confirms TLS 1.3, all 3 certs served, leaf identical to server.pem, HTTP 200; then the byte-faithful C64 ClientHello replica (probe_server_tls) confirms MFL is honored — max record 529 B, 11-record flight. PASSES on this host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The W0 phase-timing commit (13b5063) inserted an 'if phase_timing:' block between the sentinel poll loop and its 'else:', silently rebinding the else from the while (runs only on exhaustion) to the if. Result at 13b5063: a successful default-mode run (PHASE_TIMING unset) reported TIMEOUT and exited 1, and a PHASE_TIMING=1 run that genuinely timed out skipped the TIMEOUT diagnostics entirely. Track the break explicitly (sentinel_seen) and route the TIMEOUT path off that flag, independent of phase timing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The three encrypted-handshake handlers hardcoded tls_rec_buf as the message base (lda tls_rec_buf,y / tls_rec_buf+N in tls_cert.s, the tls_rec_buf+4 verify_data compare in tls_keyschedule.s). Introduce tls_hs_ptr at $3E-$3F — inside ZP_CRYPTO but past the ECDSA bignum claim ($22-$3C) and nistcurves_zp_ptr2 ($3D), so it survives across ecdsa_verify inside tls_handle_cert_verify — and read the message through (tls_hs_ptr),y instead. Behavior is identical: each handler resets the pointer to tls_rec_buf at entry via the new tls_hs_ptr_reset (CRYPTO_CODE — the NET_CODE region hosting TLS_CODE under UCI has ~8 B of slack), which also keeps the direct-call test drivers (test_finished_verify.py, test_tls_p384_negotiation.py) working unchanged. tls_verify_finished moves to CRYPTO_CODE for the same budget reason; TLS_CODE nets out 38 bytes smaller. This is the mechanical precursor to the W1 streaming deframer, which will point tls_hs_ptr at tls_rec_buf+offset (in-place dispatch) or at a carry buffer. Verified: UCI comb, ip65 default and ip65 onchip all link; VICE regression on the ip65 build: run_all_tests 266/266, test_finished_verify 18/18, test_tls_handshake 21/21, test_tls_p384_negotiation PASS, test_x509 PASS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real servers honoring max_fragment_length deliver the encrypted flight
as 11-14 records where handshake messages no longer align with records:
Certificate spans ~6 records, and CertificateVerify + Finished share
the tail record. The old tls_recv_encrypted dispatched exactly one
message per record from tls_rec_buf[0].
New src/tls_deframe.s tracks the current message type, its 24-bit
declared length and bytes consumed, and per decrypted record feeds the
plaintext through:
- message wholly inside the record -> dispatched IN PLACE, zero copy
(tls_hs_ptr = tls_rec_buf + offset). This covers 100% of the local
listener's traffic, so the existing e2e is the control.
- small message spanning records -> assembled into a 280 B carry
buffer, dispatched via the pointer. Oversize spanning messages are
an explicit error, never an overflow.
- Certificate spanning records -> rejected for now; the W2 streaming
consumer lands next.
Transcript discipline now holds per MESSAGE rather than per record:
when a header completes the running hash is snapshotted (so CertVerify
and Finished verify against the transcript excluding their own
message), then the header and every body chunk are folded as consumed.
Fold-as-you-go is what the streaming Certificate will require; for the
other paths it is equivalent to the old fold-after-dispatch.
tls_recv_encrypted still returns after ONE dispatched message, so the
tls_connect state machine and its screen markers are unchanged; a
record's leftover bytes are consumed on the next call before any new
record is fetched.
Placement: code in a new TLS_DEFRAME_CODE segment routed to
CRYPTO_OVERLAY by both UCI cfgs (CRYPTO_HOT is 446 B short of holding
it under the plain-uci REU archive); resident there, mutually
exclusive with the overlay-embed flags exactly as
LIB_NISTCURVES_MUL_CODE already is. State + carry buffer (300 B) in
NET_BSS_TAIL. Gated on TLS_STREAM_DEFRAME - default ON under
BACKEND=uci, OFF under ip65 (no code or BSS headroom there); the ip65
PRG is byte-identical across this commit (f557658a...).
New VICE coverage: tools/test_tls_deframe.py (BACKEND=uci build,
DMA-driven) - 18/18: multi-message records, splits at body/header/
header-boundary offsets, split server Finished accepted AND a
corrupted split Finished rejected (per-message transcript proof),
unknown-type and oversize-spanning error paths, transcript compared
against a Python SHA-256 of the exact fed bytes after every scenario.
All three UCI profiles (plain / onchip / comb) link with the deframer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lane D of the real-server sprint. Independent acceptance gate for the streaming handshake-message deframer (W1/W2), written without sight of the implementation — expectations derive from RFC 8446 reassembly. tools/test_tls_deframer.py (new): Drives the public tls_recv_encrypted seam over genuinely AEAD-encrypted records primed into the TCP receive ring (net_poll patched to RTS). Oracles are implementation-independent: leaf pubkey extraction proves a Certificate was fully reassembled+parsed; the running transcript proves per-message folding; a cert_buf+1536 sentinel proves the oversized-leaf guard fires instead of overflowing. Current build: plumbing 2/2 PASS (rig proven), deframer 0/9 (xfail as expected pre-deframer). Exit 2 = gate not yet met, plumbing OK. Covers: two-messages-per-record, message split across records incl. at each of the 4 header-byte offsets, mid-body, exact boundary, many-record cert, and oversized-leaf clean error. tools/https_e2e/evil_listener.py: Add RECORD_FRAME re-framing (onepermsg|mfl512|pathological), orthogonal to good/bad_finished. onepermsg kept byte-for-byte (rig_https_bad_finished unaffected). --selftest now covers all 6 combos: OpenSSL accepts every framing in good and rejects bad_finished, proving the wire bytes are valid TLS and framing is the only variable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A real server's Certificate message (~2.7 KB chain) spans ~6 records
and cannot be dispatched from a buffer. The deframer now parses it
incrementally as its bytes arrive: certificate_request_context (must
be 0), certificate_list length, then per CertificateEntry the 24-bit
cert_data length, the data, and the extensions. Only the FIRST entry
(the leaf) is copied — into cert_buf ($A000, 1536 B) — and the moment
it completes the public key is extracted through the new
tls_cert_stream_finish (src/tls_cert.s), which runs the same
x509_extract_pubkey the in-place path uses against the staged copy and
sets cert_buf_len. Every later chain entry is counted and discarded
without buffering. Message bytes are folded into the running
transcript in bulk per record chunk (fold-as-you-go — nothing is
retained to fold later).
A Certificate wholly inside one record still dispatches in place
through tls_handle_certificate, untouched — the local-listener e2e
path is byte-for-byte the same.
Error paths are explicit, never overflows:
- DF_ERR_CERT_TOO_BIG ($06): leaf > 1536 B (en.wikipedia.org's leaf
is 1636 B). Checked when the entry length completes, BEFORE any
copy — cert_buf is not written at all (proven by test).
- DF_ERR_CERT_FMT ($05): non-zero request context, entry >= 64 KB,
truncated message, or a complete leaf with no usable EC key.
Coverage: tools/test_tls_deframe.py grows to 41/41 — streamed 2-entry
chain split at awkward offsets (leaf staged byte-exact in cert_buf,
pubkey == the generated cert's coordinates, transcript == Python
SHA-256 of the fed bytes), streamed chain followed by a trailing
message in the tail record, in-place single-record Certificate,
oversize/1536-boundary/bad-context/junk-leaf error paths. Lane D's
independent RFC-derived acceptance gate (tools/test_tls_deframer.py,
on the sprint branch) passes 8/9 plumbing+deframer scenarios; the one
miss is a fixture artifact — its overflow sentinel at cert_buf+1536
sits on tls_rec_buf[0] ($A600), which the record layer overwrites
legitimately on any build.
ip65 remains gated off and its PRG byte-identical (f557658a...); all
three UCI profiles link.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The overflow sentinel at cert_buf+1536 (=$A600) is tls_rec_buf[0] under both cfgs — the record layer legitimately writes it on every flight, so no implementation could pass that check (found by the deframer lane's first run against the gate). Replace it with a strictly stronger, collision-free oracle: fill cert_buf itself with the sentinel and require it untouched — the CertificateEntry length field precedes the data, so a correct guard rejects an oversized leaf before copying anything. Also: build BACKEND=uci by default (the deframer is TLS_STREAM_DEFRAME, UCI-only; ip65 xfails by design — DEFRAMER_BUILD_BACKEND=ip65 overrides), and early-exit the pump loop once the ring is drained and all expected messages dispatched, so surplus calls no longer spin the 65,536-poll timeout loop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cert_buf+1536 is $A600 = tls_rec_buf[0] under both cfgs, so the record layer clobbers the past-the-end sentinel on any build — the gate's oversized-leaf case failed against a correct deframer. Replace it with a full-buffer sentinel fill: W2's guard rejects before any copy, so the whole buffer staying intact is the stronger, collision-free oracle (it also catches partial copies). Analysis credit: Lane A's report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-sprint-w0 # Conflicts: # tools/test_tls_deframer.py
…eal-server shape) The inner-type extraction in tls_record_decrypt indexed the decrypted plaintext with a two-page dispatch: high byte 0 -> tls_rec_buf,y, any other -> tls_rec_buf+256,y. A record carrying 512 content bytes puts the inner type at index 512 (page 2), so the code returned tls_rec_buf[256] — a mid-payload byte — and tls_recv_encrypted aborted on a bogus content type. Exactly-512-content records are what max_fragment_length-honoring servers send; every local fixture topped out one byte short, so this latent (pre-sprint) bug was reachable only live. Found on the first github.com attempt: abort in CERTIFICATE state, tls_rec_type=$08 = plaintext[256] of the first Certificate record (verified by offline decrypt of the captured ring against the DMA-dumped handshake key). Add the missing page-2 arm (legal indices reach $0213) and a gate vector shaped like the real flight — a 512-byte zero-padded EE message so the misread byte is $00 by construction, never $16 by luck (the first draft of the vector passed on the buggy build exactly that way). Mutation-proven in VICE: buggy build 9/10 failing only this scenario, fixed build 10/10. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…) + live-run findings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fw <= 3.14d leaks one /Temp attachment per REST body (run_prg's ~63 KB PRG + a 3 B trigger POST per rig run, verified by FTP listing) and never collects them; ~15 runs fill /Temp and wedge REST + the UCI bridge together, recoverable only by power cycle. Upstream fixed it in GideonZ/1541ultimate#686 (auto-cleanup, keep youngest 10) but no released firmware carries it yet. _temp_gc.py mirrors that policy host-side over FTP: delete managed temp<digits> files oldest-first, keep the youngest 2, touch nothing else. Wired into rig_https_live.py right after enable_uci (under the DeviceLock); best-effort, C64_SKIP_TEMP_GC=1 bypasses. Verified live on the U64E. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… reference sections Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same generated-include mechanism; default '/' behaves identically. Needed for the Wikipedia-article stretch goal (action=raw endpoint). Build-time assert caps the path at 100 chars for the request builder. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Host-side rig for the ~122 KB Commodore_64 wikitext fetch into the REU
at $10:0000 with the on-screen viewer left live for the human. Based on
rig_https_live.py; written against the 6502 lanes' contract exports
(http_body_total 3B LE / http_body_sink / http_resp_buf 512 B prefix /
viewer_enter) before those lanes land — main() refuses a PRG missing
them and names the build line.
Differences from the live rig:
- REU forced to (Enabled, 16 MB) under the DeviceLock before reset,
via the harness's device-generation-aware set_reu(); read first,
write only on mismatch, settle after a real write, revert-on-power-
cycle printed. preflight_reu() kept (presence, not size).
- 600 s sentinel budget (WIKI_TIMEOUT), http_body_total polled at
1 Hz as the progress signal, printed at most every 10 s.
- Verification: http_status==200 AND a host-side stdlib-ssl reference
fetch of the same URL with C64-faithful headers + a User-Agent;
exact size match + byte-exact 512 B prefix = PASS, size drift with
matching prefix = WARN (both sizes printed), prefix mismatch or an
unavailable reference = FAIL.
- On PASS the machine is NOT reset and UCI stays enabled; the lock-
release caveat (next queued job resets the device) is printed.
- Trampoline JSRs viewer_enter carry-gated, after the sentinel write.
Recon baked into the header (2026-08-21): wikipedia still MFL-honoring
(529 B max record, 14-rec/4,659 B flight, ~120 s tolerance), ECDSA leaf
still 1636 B DER; action=raw is Transfer-Encoding: chunked (NO
Content-Length), 4 chunks <=32 KB, body 125,235 B, byte-stable; and a
UA-less request — what src/http.s sends today — gets HTTP 403.
Selfcheck: 26/26 offline+live; tools/test_pytest_boundary.py green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t-Length The W4 body-termination gap found live against browserleaks.com: once the stored copy truncated at the 512 B http_resp_buf cap, http_resp_len froze, the Content-Length match could never fire, and only the connection-close / poll-timeout fallback ended the read — a server that holds the connection open left http_get never returning. - New 24-bit consumed-byte counter http_body_total (exported, LE), incremented in the new shared http_body_append for every body byte, stored or not; bytes past the 512 B cap are drained and discarded instead of ending the parse. - http_content_length widened to 24 bits (real bodies exceed 64 KB); the $FFFF magic 'absent' sentinel is replaced by a separate http_cl_valid flag byte that cannot collide with a real length. - Termination compares consumed count against Content-Length; with the header absent the legacy 512 B cap behaviour is preserved (chunked/streaming fallback). - http_resp_len keeps its contract: min(512, total), first 512 bytes. - The body machinery moved to a new HTTP_AUX_CODE segment: LOADER slack under UCI, CRYPTO_OVERLAY under ip65 (LOADER is 8 B from full there; the grown 24-bit parser no longer fits the NET_CODE tail on either backend). All five build profiles link. - tools/test_http.py: three new span-driven vectors (700 B body with CL — the regression; 24-bit CL parse; CL-absent legacy path); 38/38 PASS in VICE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lane H's live recon (2026-08-21): en.wikipedia.org returns HTTP 403 to the exact request c64-https sends today because it carries no User-Agent; with one it returns 200. Emit a short honest UA line, 'User-Agent: c64-https/0.1', between Host and Connection. Request is now 87 B for the test vector (256 B http_req_buf; the 42-char article path leaves ample headroom). String rides RODATA; only the ~20 B copy loop lands in LOADER CODE. tools/test_http.py asserts the exact UA line and the updated full request bytes; 39/39 PASS in VICE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tch goal New src/viewer.s (UCI-only, .ifdef BACKEND_UCI — the ip65 PRGs are byte-identical with and without this change, verified by hash on both ip65 profiles): renders a large ASCII document stored in the REU at HTTP_REU_BODY_BASE (default $10:0000, -D-overridable; length in the 24-bit http_body_total) on the 40x25 screen, writing screen RAM directly with the ascii_chrout strategy-B character mapping. - 24 content rows + status row 0 (hex offset/total + Q=QUIT) - hard wrap at 40 cols, LF newline (one LF after a wrap is consumed), CR ignored, TAB -> space; clamped at both ends; EMPTY screen for total = 0 - keys: CRSR-DOWN/UP one row, SPACE/F1 page, HOME top, Q quit - window model: 3,584 B REU->C64 DMA into the dead tcp_recv_buf ring (1 KB back-coverage); row-start table + REU shadow-register save slots live in the ring tail, so viewer BSS is 20 B in NET_BSS_TAIL - scroll-up derives the previous display-row start from a backward LF scan (P + 40*floor((n-1)/40)), matching forward rendering exactly - REU shadow registers ($DF02-$DF0A) are saved/restored around every DMA-using entry so reu_fetch_mul_row's latched state survives VIEWER_CODE segment: CRYPTO_HOT in the comb cfg (265 B still free), CRYPTO_OVERLAY in the plain UCI cfg (CRYPTO_HOT there has only ~69 B of slack at the v0.11.2 pin). Code+rodata 1,189 B. boot.s hook (BACKEND_UCI only): after http_recv_body, a non-zero http_body_sink routes to viewer_enter instead of print_resp_body. viewer.s carries .ifndef-guarded provisional definitions of http_body_sink / http_body_total until the body-sink lane's land (-D VIEWER_EXTERNAL_SINK=1 flips them to imports). tools/test_viewer.py (VICE, default_vice_config): 7/7 PASS — exact 25x40 screen-code assertions for LF handling, wrap + wrap-LF consume, lowercase folding, TAB/CR, odd bytes, mid-document and arbitrary offsets, EOF clamp, offset==total, EMPTY, plus 9 scroll-up vectors against a forward-rendered row-start list. Test image builds with make BACKEND=uci HTTP_REU_BODY_BASE=196608 VIEWER_TEST_HELPERS=1. The interactive key loop is a thin dispatch over these tested primitives and is deliberately not driven from VICE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-sprint-w0 # Conflicts: # Makefile
…mination Wikipedia — the stretch-goal target — serves the article CHUNKED with NO Content-Length (lane H live recon 2026-08-21: 4 data chunks of 26,931-32,768 B, de-chunked body 125,235 B), so on that path the terminal 0-chunk, not a Content-Length match, is what must end the body read. - 'Transfer-Encoding: chunked' matched case-insensitively per header line (check_header_line now runs both matchers); http_te_chunked routes the end-of-headers transition to the new chunk machine. - parse_state 3..6: chunk-size line (hex, case-insensitive, chunk extensions skipped), chunk payload, closing CRLF, trailer section; the empty trailer line after the 0-chunk completes the body (C=0). - Only payload bytes reach http_body_append, so http_body_total counts de-chunked payload — never framing — and the first-512/http_resp_len contracts hold unchanged on the chunked path. - http_chunk_rem is 16-bit; chunks >= 64 KB would mis-parse (largest observed anywhere is Wikipedia's 32,768 B) — documented at the declaration. - New HTTP_CHUNK_CODE segment: CRYPTO_RESIDENT tail under ip65 (the onchip profile now has 2 B free there — the region is at capacity), CRYPTO_OVERLAY under UCI. chunk_hex_digit rides HTTP_AUX_CODE for exactly that fit. All five profiles link. - tools/test_http.py: two chunked vectors (700 B across 3 chunks with an extension + uppercase hex + trailer; small 2-chunk body with case-folded TE header); 47/47 PASS in VICE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…af fits CERT_BUF_SIZE is now a per-backend equate in net_tuning.inc (2048 UCI / 1536 ip65); der_decode.s, the deframer's leaf cap (DF_CERT_BUF_SIZE) and the tests all derive from it instead of a literal. src/exports.s exports it as the absolute symbol `cert_buf_size` so tests/rigs read the real capacity from build/labels.txt. Under UCI, CERT_BUF_BSS moves from CRYPTO_COLD_SHADOW into CRYPTO_OVERLAY (after TLS_DEFRAME_CODE; plain/onchip end at $4F62 with 4,253 B free, comb through ~$5BD2 with ~1.1 KB free — still room for the rigs' DMA scratch, proven via the _memory_policy arbiter + rig_https_live --selfcheck). Moving it removed the packing that landed the shared sqtab at its baked $BC00 anchor, so both UCI cfgs now pin TABLES_BSS with start=$BA00; the Makefile's post-link check passes on all UCI profiles. Under ip65 nothing moves: cert_buf stays 1536 B at $A000 in the #68 SCRATCH_UNION, and both ip65 PRGs are byte-identical across this change. Casualty, accepted: USE_X25519_SIBLING=1 no longer links under UCI (X25519_BSS overflows CRYPTO_OVERLAY by 1,280 B; it linked pre-growth). EMBED_P256_OVERLAY was already broken before this change (labels.txt ordering defect) and is unchanged. Documented in CLAUDE.md. Tests: tools/test_tls_deframe.py gains cap-relative boundary vectors (cap accepted, cap+64 rejected, 1636 B accepted on UCI) — 42/42; tools/test_tls_deframer.py gains a wikipedia_sized_leaf_accepted gate (real minted 1636 B P-256 cert, pubkey extraction as oracle) — 13/13. The acceptance rig now calls tls_deframe_init per scenario, mirroring tls_connect's per-connection reset (the oversize-abort scenario used to leak DF_MODE_STREAM state into whatever ran after it). ip65 regression: run_all_tests 287/287, test_x509 11/11, test_tls_handshake 21/21. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…raint With CERT_BUF_BSS resident in CRYPTO_OVERLAY, a runtime overlay swap-in would clobber the staged leaf certificate mid-handshake — live data, not just deframer code. Note it in both UCI cfgs (raised by the deframer lane's review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With http_body_sink=1 the 512 B http_resp_buf becomes a bounce buffer: http_body_append fills it, and each fill is REU-DMA-STASHed (HTTP_SINK_CODE, register programming mirrors boot.s::reu_mul_init) to http_reu_body_base + a 24-bit http_reu_cursor. At completion http_body_finish (idempotent via a latch, also invoked from the poll-timeout fallbacks) blits the final partial bounce, then FETCHes the FIRST min(512, total) body bytes back into http_resp_buf — the host-side verification contract: first-512 prefix + http_body_total + REU contents. Works under both identity and chunked bodies (the sink sits under the shared http_body_append). - Base: bank 16 ($10:0000, 16 MB REU — the stretch-goal config), clear of mul-table banks 0-1, comb's Lim-Lee bank 2, reserved banks 6-7. HTTP_REU_BODY_BASE equate in constants.inc, overridable with `make HTTP_REU_BODY_BASE=<decimal>`; the runtime copy http_reu_body_base (3 B LE, RODATA = RAM) can also be retargeted by DMA — VICE's 512 KB REU cannot hold bank 16, so the harness pokes bank 3 ($03:0000) at runtime. - New knob HTTPS_BODY_TO_REU=1: boot.s's do_https_get sets http_body_sink=1 before http_recv_body (the only boot.s change). - UCI-ONLY: every ip65 code pool is within ~30 B of full after the W4 termination + chunked work, so ip65 builds get 3-byte inert stubs (sanctioned by the sprint brief; the knob is UCI-only anyway) and http_body_sink is a dead flag there. All five profiles link; the knob+base-override build links too. - http_body_finish re-latches the reu_fetch_mul_row register state (mul_dma_lo target, 512 B length) that the sink clobbers, so a second handshake on a REU-profile build still fetches multiply rows correctly (guard mirrors boot.s's). - tools/test_http.py: sink vectors (chunked 700 B wiki-shape + identity/CL 600 B) verify body_total, the first-512 restore, and the REU contents via a C64-side DMA FETCH stub; the suite now uses default_vice_config (mandatory -reu). On an ip65 build the sink vectors report loudly that they did NOT run (C64_EXPECT_SINK=1 makes absence a failure). 47/47 PASS on ip65, 52/52 on uci. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lane E's landed contract sets http_body_sink=1 inside do_https_get (the menu 'G' path), but the rig's trampoline JSRs http_get directly and never crosses it — without the poke the body would not be sunk. Poking the exported flag by DMA before http_get is the supported alternative per lane E. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…print tip My four W4 commits were built from a stale base (pre-2158037). This merge keeps Lane B's github-validated chunked decoder as the base and ports the Lane E deltas onto it: - ONE chunked decoder survives: Lane B's http_state_body_chunked (with its public names http_chunked / http_chunk_state / http_chunk_rem and its large-header robustness). My parallel state-3..6 machine, HTTP_CHUNK_CODE segment, and duplicate check_transfer_encoding are gone. - Lane E deltas ported onto it: 24-bit Content-Length + http_cl_valid ($FFFF sentinel retired in http_hdr_init), 24-bit consumed-count termination (http_body_total, maintained in the shared body_append — de-chunked payload only), truncate-at-capacity now counts past the cap on the identity path too (the browserleaks fix), User-Agent header, and the BACKEND=uci REU body sink (bounce blits + http_body_finish first-512 restore + reu_fetch_mul_row latch restore) hooked into both body paths via sink_finish_done. - viewer.s: Lane G's provisional .ifndef VIEWER_EXTERNAL_SINK block deleted as its header prescribed; http_body_total / http_body_sink are canonically defined + exported in src/data.s. - Makefile: HTTP_REU_BODY_BASE pass-through deduped (Lane G's block kept, comment widened); HTTPS_BODY_TO_REU added, honored only under BACKEND=uci. - cfgs: tip's CERT_BUF_BSS / VIEWER_CODE / deframer lines kept. New segments: HTTP_SINK_CODE (UCI: CRYPTO_OVERLAY; ip65: 3-byte stub in NET_CODE) and HTTP_AUX_CODE2 (jsr-only helpers + patterns split out of HTTP_AUX_CODE: ip65 CRYPTO_RESIDENT / UCI CRYPTO_OVERLAY — ip65's CRYPTO_OVERLAY and UCI's CRYPTO_HOT could not absorb the growth whole). Comb overlay keeps ~700 B tail free for the rig MemoryArbiter (rig_https_live.py --selfcheck PASS). - tools/test_http.py: UNION suite — Lane B's 48-vector set (incl. the 5-span github-shape groups) + Lane E's vectors (identity>512/CL regression, 24-bit CL, CL-absent legacy, REU sink with DMA readback, UA line, body_total assert in the chunked-truncation group). 61/61 on ip65, 66/66 on BACKEND=uci in VICE. All five build profiles link. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…O_COLD_SHADOW) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
temp0009 is followed by temp000A (observed live on the U64E); the \d+ pattern silently skipped lettered names forever. Match and sort hex. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ckets on rig abort Two findings from the U64E wikipedia attempts (2026-08-21): 1. UCI firmware ACK autonomy ends at its ~4.3 KB socket buffer — bytes beyond are silently DROPPED, not flow-controlled. wikipedia's 4,659 B flight is the first to exceed it: with 16 drain polls the ~8 s X25519 stall let the buffer overflow, leaving a permanent hole at record 9 (tls_read_seq=8, mid-header — measured twice). github's 3.2 KB flight squeaked under. Raise the post-SH drain budget to 8x16=128 polls (~5.2 s), which moves ~4 KB firmware->ring before the stall. Client silence grows ~30 s vs 45/90/120 s tolerances. 2. Abandoning an ACTIVE firmware socket (rig timeout with the C64 parked in http_get) is the leading suspect for the recurring DHCP-0.0.0.0 poisoning that survives machine:reboot. The wiki rig's timeout path now saves the socket id, resets the C64 (PRG stays in RAM), and SYS-trampolines net_tcp_close so the firmware sees a real SOCKET_CLOSE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nnect host Lets the C64 dial a local TCP relay while the ClientHello names the real origin (wikipedia rejects absent/mismatched SNI — verified). Built for the wikipedia-stall forensics: the instrumented relay proved the full 4,658 B flight is delivered into the firmware connection with zero backpressure and the client stalls anyway, killing the delivery/ window theories. Off unless HTTPS_SNI= is set; ip65 unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
net_poll always requested a fixed 512 B from firmware, but the fill loop drops any byte past the ring's current free space and the response drain discards the remainder — bytes the firmware counts as delivered. Once the ~4 KB ring wraps (any TLS flight over ring capacity, i.e. en.wikipedia.org's 4.7 KB cert flight), the stream gets a permanent hole and the client parks forever at that record. Found by decrypting the stalled run's captured ring (wrap-aware, per- record seq brute-force): the hole sits exactly one ring-capacity past the wrap origin. The relay experiment had already excluded delivery/ window causes (full flight handed to the firmware, zero backpressure, client stalled anyway) — this is the read side. Fix: request min(ring_free - 1, 512) and skip the read entirely when nothing fits; unread bytes wait in the firmware for the next poll. Adds a wikipedia-shaped gate scenario (multi-cert chain, real 1636 B leaf) that proves the CONSUMER handles the shape — but note in-code: the gate primes the ring directly with net_poll stubbed, so it CANNOT exercise this adapter fill path. The fix is validated on hardware, not in VICE. Gate 12/12, viewer 7/7, http 61/61. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
⭐ Stretch goal achieved — Commodore 64 reads its own Wikipedia article over real-internet TLS 1.3Confirmed on U64E hardware @ 48 MHz (commit d9cd021):
The bug that blocked it (d9cd021)
Fix: request Found by decrypting the stalled run's captured ring (wrap-aware) after a TCP-relay experiment excluded delivery/window causes (full flight handed to firmware, zero backpressure, client stalled anyway). Three real public sites now verified end-to-end: github.com, browserleaks.com, lwn.net (all HTTP 200), plus the full 122 KB wikipedia article into REU with an on-C64 scroll viewer. |
…DE.md README: milestone callout in the intro, two Project Status entries (real public-internet HTTPS on UCI/comb; wikipedia article into REU + viewer), and refreshed the two stalest Known Issues (real-server reach/limits; the now-fixed wikipedia ring-drop). CLAUDE.md: rewrote the End-to-end HTTPS status milestone block for the final state — 4 live sites at HTTP 200 incl. wikipedia's 125,235 B article to REU + viewer; the three live-only bugs fixed (512-content inner-type, large-body consumed-count termination, the net_poll ring-drop stall d9cd021); and the two distinct U64E device gotchas (lease-poisoning via reset-with-live-socket vs the writemem/temp wedge). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone
First HTTPS GETs from the C64 to real public internet servers — U64E @ 48 MHz, comb profile, 2026-08-21:
http_resp_bufServer-visible silence ~26 s against 90 s / 45 s measured tolerances.
What's in here
src/tls_deframe.s, gatedTLS_STREAM_DEFRAME, UCI-on/ip65-off): in-place zero-copy dispatch via a ZP base pointer (tls_hs_ptr=$3E, own mechanical commit), 280 B carry buffer for small spanning messages, per-MESSAGE transcript discipline.cert_buf, intermediates discarded unbuffered,DF_ERR_CERT_TOO_BIGrejects before any copy.make HTTPS_HOST=<host>(generated include; default build proven byte-identical).src/http.s(new 367 BHTTP_AUX_CODEsegment;CODEshrank 64 B).tools/uci/rig_https_live.py+ padded 3-cert-chain listener profile (github-shaped 11-record flight, no egress).tools/test_tls_deframer.py(10/10 + 2/2 plumbing), Lane-A suitetools/test_tls_deframe.py(41/41),evil_listener.pyRECORD_FRAME re-framing modes (selftest 6/6 vs OpenSSL),tools/test_http.py48/48 (was 27)./TempGC (tools/uci/_temp_gc.py): userspace mirror of Add automatic cleanup of Temp folder GideonZ/1541ultimate#686 defusing the writemem exhaustion wedge (harness adoption: c64-test-harness#153).Verification
run_all_tests.py287/287; deframer gate 10/10 (mutation-proven); Lane-A suite 41/41; re-framed server-Finished checks on hardware: good controls PASS (mfl512 + pathological), forged Finished → clean abort at FINISHED.Known follow-ups (deliberately not in this PR)
cert_buf1536 B vs wikipedia's 1636 B leaf (reroute to CRYPTO_OVERLAY under UCI, backend-conditional size).🤖 Generated with Claude Code