Repository navigation
http_get reports success (carry=0, HTTP 200) on a truncated body — silent data loss #211
Description
The defect is the false success, not the byte count. http_get returns carry=0 with http_status=200 after consuming a body tens of kilobytes short of the Content-Length it was given. A caller cannot distinguish a complete body from a 51% one, which makes HTTPS_BODY_TO_REU fetches untrustworthy rather than merely slow.
What this evidence can and cannot support. Every figure below is http_body_total, a 24-bit consumed count on the C64 side. None of them observes what the server sent, except in the offline runs where the listener's Content-Length is known by construction. They establish that the fetch is short and whether the shortfall is stable; they cannot by themselves distinguish "we stopped asking" from "we dropped what arrived".
Offline reproduction — local TLS listener, Content-Length, no chunking, no internet. PRG 1c828c4adbb7f9f58efa538045a170ebeb9b345d47eb4c4f01b2cd7247425807 (e4fda6d, BACKEND=uci USE_NISTCURVES_ONCHIP=1 HTTPS_SNI=www.foo.invalid), U64E @ 48 MHz:
| served | http_body_total |
delta | net_last_error |
carry |
|---|---|---|---|---|
| 40,000 | 40,000 | 0 | $00 |
0 |
| 160,000 | 78,293 | 81,707 | $86 |
0 |
| 160,000 | not captured | — | $00 |
0 |
| 160,000 | 160,000 | 0 | $00 |
0 |
It is intermittent, not size-dependent. The same 160,000 B body on the same PRG both failed and succeeded within minutes. An earlier reading of these numbers as a size threshold was wrong: the two runs differed in time as well as size, and time was the variable that mattered. A size threshold inferred from two runs at different sizes is unfalsifiable until one of them is repeated.
Intermittent silent truncation is worse than deterministic truncation, because no fixed-size test catches it reliably.
Live (wikipedia, chunked) — same-day curl anchor 125,703 B at 06:16:
| tree | PRG | consumed |
|---|---|---|
master 48657f5 |
012c7848… |
117,192 B |
e4fda6d |
0712def5… |
89,526 B |
e4fda6d (n=2) |
0712def5… |
73,720 B |
The anchor is a moving target — the article is user-editable — which is why it is quoted with its capture time. It does not weaken the finding: a 36 KB shortfall is not an edit, though a few-hundred-byte difference between runs may be. Two runs of the same PRG differ by 15,806 B, so these live counts characterise variance, not any difference between trees.
Both trees. The live wikipedia fetch truncated on master 48657f5 (PRG 012c7848…, 117,192 B against a same-day 125,703 B anchor) and on e4fda6d (PRG 0712def5…, 89,526 B and 73,720 B). The offline reproduction was run on e4fda6d only. Because the failure is intermittent, a single offline run on master would not have discriminated — a complete run there would not show absence — so it was deliberately not used as evidence. The supported statement is: the defect is observed on both trees, it is intermittent, and there is no evidence the bump introduced it.
Leads, recorded as leads — no cause is attributed:
- Chunked desync above 64 KB (CLAUDE.md,
http_recv_response). Excluded for the offline case: the local listener sendsContent-Length, so no chunking is involved. Negative evidence for the live case too — neither live plateau sits at the boundary (89,526 is 24,010 past 65,536; 117,192 is 51,656 past). - The
net_pollring clamp thatd9cd021fixed.tcp_recv_overflow($A248) is the one-byte discriminator:src/net/uci/net.s:387sets it when the ring fills and then drops the delivery, commenting "The clamp at @do_poll makes this unreachable in practice … If it latches, the request/free-space arithmetic has regressed — that is the wikipedia-stall bug's signature." Not captured on any run above; instrumented now. net_last_error = $86correlates with the one captured short run and$00with every complete one, but$86is documented as appearing on passing runs (CLAUDE.md, github.com 2026-08-28), and the one$00run at 160,000 B is the run whose count was not captured. Candidate, not cause.
State at the stall on the short run: tcp_recv_head == tcp_recv_tail == $0ACB (ring empty), uci_read_hdr = $FFFF (no-data sentinel), uci_status = '02,NO DATA: 11', tls_state=$00, tls_recv_progress=$05, tls_recv_sub_progress=$02.
Why this survived: the only rig exercising this path could not go red on it — see the companion issue on rig_https_banner.py.
Two method traps for whoever picks this up:
- Reading
http_body_totalafter the rig returns is unreliable. A post-hoc read returned all zeros includinghttp_status, on a run the rig itself had read as 200. Wrap the rig's_dump_diagso the read happens while the transport is live. - The listener's post-send
time.sleep(1.0)is not the cause. Checked and cleared: TCP orders FIN after queued data andsendallblocks on the C64's window.
Activity
JC-000 commentedon Sep 7, 2026
tcp_recv_overflowresult — the ring is excluded, and$86is excluded as the causeThree further 160,000 B offline repeats, same PRG
1c828c4adbb7f9f58efa538045a170ebeb9b345d47eb4c4f01b2cd7247425807(e4fda6d), same local listener withContent-Lengthand no chunking, each capturinghttp_body_total,tcp_recv_overflowandnet_last_errorin one live-transport dump:http_body_totaltcp_recv_overflownet_last_erroruci_read_hdr$86$0016$0E54=$0E54$00$FFFF$00B3/$00CB$00$FFFF$00B6=$00B6All three returned
carry=0,http_status=200.Lead 2 (the
net_pollring clamp) is excludedtcp_recv_overflowreads 0 on both truncating runs.src/net/uci/net.s:387sets that byte when the ring fills and then drops the rest of a delivery; it never latched. So the bytes were not dropped on arrival — we stopped asking.d9cd021's clamp has not regressed and the request/free-space arithmetic is not the fault. The comment's claim that the overflow path is "unreachable in practice" is upheld, not falsified.Both leads recorded in the issue body are now dead: chunked desync was already excluded for the offline case (the listener sends
Content-Length), and the ring is excluded here. The loss is upstream of the ring, in the poll / request / termination path.$86is excluded as the causeRun 3 truncated by 68,395 B with
net_last_error = $00. The earlier 3-for-3 correlation between$86and short runs is broken by a counterexample, which is what it needed.$86remains a real signal when present — it has exactly one setter in the FPGA — but it does not explain this defect.One observation worth chasing, not a claim
Run 1 stopped at 25,557 B with
uci_read_hdr = $0016anduci_status = '00,OK'— a 22-byte read header and an OK status, not the$FFFFno-data sentinel seen on runs 2 and 3. At the moment of the dump the firmware had data available and an OK status, and the client had already returned success. That is consistent with the client deciding the body was complete while the transport still had bytes to give, but the dump is taken afterhttp_getreturns, so it is a lead and not a measurement of the stall state itself.Range now observed
Offline at a constant 160,000 B: 25,557 / 78,293 / 91,605 / 160,000 B. The shortfall varies by more than 130 KB across runs of one image against one listener. That reinforces the intermittency finding and makes any fixed-threshold test unreliable as a detector.
docs(readme): v0.14.0 pin, the P-384 link gate, and the unverified body
docs(readme): name the artifacts, and tie the cfg move to this branch
JC-000 commentedon Sep 7, 2026
Correction to my previous comment, plus the
tcp_recv_overflowresultRetracting a claim I made above: "
$86is excluded as the cause" was wrong. I based it on a run that truncated by 68,395 B carryingnet_last_error = $00, and treated that as the counterexample the correlation needed. That run did not complete — it hung, and its sentinel was never set (TIMEOUT: sentinel not set after 900s, progress=0x03). Itscarry=0and its byte count are uninitialised values read after a timeout, not the results of a finished fetch. It is not a counterexample to anything.Method note, which is the reusable part:
carryandhttp_body_totalread from a run whose sentinel never fired are stale and must not be quoted as a measurement. Every offline run below was audited forsentinel setversusTIMEOUTbefore being included.The measurement
Offline, PRG
1c828c4adbb7f9f58efa538045a170ebeb9b345d47eb4c4f01b2cd7247425807(e4fda6d), local TLS listener, 160,000 BContent-Length, no chunking, U64E @ 48 MHz:net_last_errortcp_recv_overflow$86carry=0, HTTP 200$00carry=0$00The
d9cd021ring-clamp lead is excludedtcp_recv_overflow = 0on a run short by 134 KB.src/net/uci/net.s:387sets that byte when the ring fills and then drops the rest of a delivery. It never latched, so the bytes were not dropped on arrival — we stopped asking. The clamp has not regressed, and the comment there asserting the overflow path is "unreachable in practice" is upheld, not falsified.Both leads recorded in the issue body are now excluded: chunked desync was already out for the offline case (the listener sends
Content-Length), and the ring is out here. The loss is upstream of the ring, in the poll / request / termination path.$86survived a test that could have killed itAmong completed runs only, the correlation is 5/5:
$86), 25,557 B ($86)$00), 160,000 B ($00), 160,000 B ($00)Still recorded as a candidate, not a cause — CLAUDE.md documents
$86appearing on passing runs (github.com, 2026-08-28), so a known counterexample class exists. But my earlier exclusion of it was based on a run that had not completed, and I withdraw it.A second failure mode, not described in the issue body
Run 3 consumed 91,605 B and never returned within 900 s. That is distinct from the silent-success truncation this issue describes:
http_getcan also hang outright rather than return a short body withcarry=0. Both are reachable on the same image against the same listener at the same body size.Range now observed
At a constant 160,000 B offline: 25,557 / 78,293 / 91,605 (hung) / 160,000. The shortfall varies by over 130 KB across runs of one image against one listener, which is why no fixed-size or fixed-threshold test is a reliable detector.
docs(claude.md): the wikipedia body is not byte-verified (#211)
Merge pull request #219 from JC-000/fix/211-body-truncation
Merge pull request #238 from JC-000/fix/230b-uci-reply-valid-wait
README: delete the remaining #211-as-open text and "no trust store"
Merge pull request #287 from JC-000/docs/readme-v0.5.0