FUG-133: HITL gate — wss:443 handshake + cert-page GET under worst-case load

[issuefleet]

FUG-133 — HITL gate: wss:443 handshake under worst-case heap load

An asserting HITL gate for the PR #114 failure class: on a heap already loaded with a
resident map + a texture-sampling effect + a streamed texture keyframe, mbedTLS could
not allocate its TLS session and fresh wss:443 handshakes / the HTTPS cert page failed
until reboot. rename_wss and tls_churn exercise the wss re-issue / connection-churn
paths but only on a clean device, so neither reproduces the OOM-during-handshake on a
loaded heap. This gate does.

Rebased + reworked for the netstack transport

This PR was originally written (2026-08-20) against the vendor two-mbedTLS-slot wss
server (main.cpp wss_start, now behind #if !defined(LM_NETSTACK) and deleted from the
shipping image). The rigs now run the netstack transport (netstack_transport.cpp),
a single-connection TLS server that RST-sheds concurrent SYNs. The original step-2
assertion — hold a wss session open and force a concurrent HTTPS GET / → 200 ("the
second ~28 KB session") — is incompatible with that model and would false-FAIL on a
refused/reset connection. The gate has been reworked to the single-connection reality:

• Sequential single-connection model. LOAD the heap to worst case (full LM_MAX_LEDS
  map + activated texture effect + resident texture keyframe) over wss, drop that
  socket so its TLS session frees while the map/effect/texture stay RAM-resident, then run
  N sequential rounds — each a fresh wss:443 handshake + welcome on the loaded heap
  (the exact Fix device connection reliability: heap headroom, JIT-on, and web (re)connect flow #114 alloc), followed by a sequential cert-page GET / (not concurrent),
  then a recovery probe.
• Gate on RECOVERY, reusing tls_churn_core's unit-tested verdict (baseline unreachable
  ⇒ SKIP, never FAIL; recovery of the final loaded-heap round is the anti-wedge gate; a
  serial crash marker fails an otherwise-PASS run). A strict single-shot "handshake +
  GET==200" assert red-lines the genuine transient shed-then-recover, which is graceful
  degradation on a heap-tight board, not the regression (the FUG-136 lesson). On unfixed
  firmware this gate can legitimately go red on the lane — that is the wedge reproducing,
  not a test defect.

video_stream/LOAD-leg flake — root cause + fix

The hitl_tests check was red because of the LOAD leg. Root causes, all fixed:

1. Build error (primary red). The target referenced //firmware/player_app:esp32c6_flashbundle,
   which no longer exists — the vendor image was deleted; the shipping image is
   esp32c6_netstack_flashbundle. A stale data dep fails the tag-discovered
   bazel test → the whole lane reds. Target now ships the netstack bundle and reads
   HITL_BUNDLE_RUNFILE.
2. Oversized TLS records. map_upload_core.CHUNK_BYTES defaults to 4096, which
   overflows the netstack TLS record buffer → mbedTLS alloc failure mid-map-upload. The
   target now sets HITL_CHUNK_BYTES=1024 (as the other netstack drivers do).
3. NVS-join reboot stranded the DUT. The netstack keeps WiFi creds in RAM (no NVS),
   so the old post-provision reboot lost them and the board never rejoined. The driver now
   tests the just-provisioned link directly (mirrors tls_churn --skip-nvs-reboot).
4. Tight timeouts. Hardcoded 8 s/15 s reintroduced the driver→tailnet→rig→ssh -L→DUT
   jitter flake; the driver now uses the shared hitl_ws tolerances.
5. --led-count default 768 > the firmware cap (512). A 768-LED map was rejected; the
   default is now 512 (LM_MAX_LEDS).

The serial OOM scan now matches the strings the netstack actually prints ([heap] alloc FAILED, TLS handshake err -0x7f00); the vendor esp_tls_create_server_session string is
never emitted. That scan is informational (a shed-line count for the reviewer), never
the gate.

Verification

• bazel test //pi/hitl/tests:hitl_test — pure verdict + fixtures + serial-scan, green.
• bazel build --platforms=@embedded//platforms:esp32c6 //firmware/player_app:esp32c6_netstack_flashbundle — green.
• On real hardware (esp32c6 on a Pi 5 rig, flashed with the netstack bundle + provisioned
  over Improv):

  [load] map: 30658 B -> 30 window(s)            # 512-LED map, sharded at 1024 B/record
  [load] resident: 512-LED map + effect '__fug133' + 40x40 texture (3200 B frame)
  [round 1] handshake OK under load … cert-page GET / -> 200 … wss RECOVERED 2.2s
  [round 2] handshake OK under load … cert-page GET / -> 200 … wss RECOVERED 2.3s
  [round 3] handshake OK under load … cert-page GET / -> 200 … wss RECOVERED 2.2s
  RESULT rounds=3 baseline=ok recovered=3/3 served_total=3 cert_ok=3/3 crashed=no verdict=PASS
  

  The LOAD leg that was flaking now completes cleanly (30 × ~1024 B windows over wss on the
  netstack), every fresh handshake on the loaded heap succeeds, and each sequential cert
  GET / returns 200. (The Pi 3 rig is fenced from the gating lane for unrelated USB-bus
  contention, so this was validated on a Pi 5.)

The gate joins the HITL lane automatically (tag hitl); it is manual so it stays out of
bazel test //.... --device-ws wss://<ip>/ws hits a reachable board directly.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-04 18:24 UTC