Repository navigation
FUG-74: large-map upload — arena fix + sharded, flash-streamed uploads + HITL test - #46
Merged
Merged
Conversation
…logy A topology upload for a ~150-LED scan failed with map_too_large. That error is ArenaFull from the firmware's 16 KB static decode arena (player_app/ffi.rs ARENA_BYTES) — not a transport limit (the C++ rx buffer is 32 KB and the store streams via ChunkedReader without buffering the whole upload). The live map+topology footprint for 150 LEDs is only ~10 KB; it overflowed because the growable topology lists (associations, per-segment polylines, branch points, segments) grew by doubling and LEAKED the old region each time (a bump arena cannot free), peaking at ~19.5 KB of churn. The arena is deliberately trimmed to 16 KB for TLS-handshake heap headroom, so enlarging it isn't viable. Fix: ArenaVec now extends its region IN PLACE when it still owns the arena tail (its end sits on the bump cursor) via Arena::try_grow_tail_in_place — a cursor bump, no copy, no leak. The big topology lists are each built at the tail, so this removes essentially all churn: peak drops to ~11.7 KB, comfortably inside 16 KB. Growth only falls back to realloc-and-leak when a later allocation sits after the vec. No proto / web / transport changes; arena budget preserved. Tests: - arena: in-place growth doesn't move the base or churn; non-tail and would-overflow regions correctly refuse in-place growth (fall back / ArenaFull). - store: a realistic ~150-LED topology (150 associations, 12 segments x 20 polyline points, 12 branch points) decodes into a 16 KB arena beside its map. Verified to fail with ArenaFull on the old leak path and pass with the fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FUG-74 Topology Editor: failed upload with map_too_large
When trying to upload a topology map for a larger scan (~150 leds), I get: This should not happen. If necessary, the upload should be batched (append-mode proto) such that it can all be uploaded into the storage. My guess is that it's not too big to fit into memory at actual runtime, it's just that the buffer for storing it at transport time when loading into flash is not big enough (and it could be made smaller by doing batched uploading). |
Contributor
|
A ~150-LED submit_map is one ~15 KB TLS record, and the C6 can't allocate that
contiguous block on its fragmented heap (min free ~7.7 KB) — so the wss upload
dropped mid-transfer ("Send failed: socket closed"; alloc(15011) failed).
Shard it. New UploadChunk/ChunkAck arms carry the encoded submit_map /
submit_topology frame as <=4 KB windows; the client (sendChunked) sends one per
awaited chunk_ack, so the browser flushes one small TLS record at a time instead
of re-coalescing into a big one. The device never needs more than ~4 KB for a
record. The wire is opaque byte-slices, decoded by the unchanged store decoder.
Then stream, don't buffer: each window's bytes append to /lfs/upload.tmp; the
last one decodes straight off flash via a block-streaming reader (store
BlockReader / lm_decode_upload_stream) and is renamed into place — so no whole
frame is ever resident and persistence is free (the file IS the frame). rx
shrank 32 KB -> 12 KB, lifting min free heap from ~7.7 KB to ~42-55 KB (~5.5x).
Both transports shard; boot replay streams the map/topology off flash.
Verified on an ESP32-C6: the exact payload that OOMed now uploads and
round-trips (map + topology), with min heap ~5.5x higher and persistence intact.
Tests: store BlockReader + sharded reassembly (map + topology) +
parse_upload_chunk guards; ffi/player arm coverage; web proto round-trip + a wss
sharding client test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reserve a rig (or --device-ws a reachable board), shard a 256-LED submit_map (~15 KB, the firmware cap and the OOM size) plus a multi-KB submit_topology into UploadChunk windows, and assert a chunk_ack per non-final window and a result_ready for the last. Then get_stored_map round-trips the MappingBundle and checks the led_count / segment / association counts survive the whole path (upload -> streamed decode -> arena -> dump) — catching a mis-reassembled or truncated upload, not just a dropped socket. The regression guard for the wss "socket closed" OOM. The window-plan + fixtures (map_upload_core) are unit-tested with no hardware in //pi/hitl/tests, pinning the chunking contract to the client + firmware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fughilli
changed the title
issuefleet Bot
pushed a commit
that referenced
this pull request
Aug 6, 2026
…ploads) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issuefleet Bot
pushed a commit
that referenced
this pull request
Aug 7, 2026
…ploads) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issuefleet Bot
pushed a commit
that referenced
this pull request
Aug 9, 2026
…ploads) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Uploading a real ~150-LED map failed at two ceilings, one after the other:
map_too_large— the 16 KB decode arena overflowed because the growable topology lists doubled-and-leaked while growing (a bump arena can't free).Send failed: socket closed— a wholesubmit_mapis one ~15 KB TLS record, and the C6 can't allocate that contiguous block on its fragmented heap (min free ~7.7 KB), so the wss connection dropped mid-transfer (alloc(15011) failed).Fix (three parts)
ArenaVecgrows its region in place when it still owns the arena tail — no copy, no leak. Peak drops ~19.5 KB → ~11.7 KB, inside the 16 KB arena.UploadChunk/ChunkAckarms carry the encoded frame as ≤4 KB windows; the client (sendChunked) sends one per awaitedchunk_ack, so the browser flushes one small TLS record at a time instead of re-coalescing into a big one. The device never needs more than ~4 KB for a record. Wire is opaque byte-slices → the store decoder is unchanged./lfs/upload.tmp; the last one decodes straight off flash via a block-streaming reader (BlockReader/lm_decode_upload_stream) and is renamed into place — so no whole frame is ever resident and persistence is free (the file is the frame).rxshrank 32 KB → 12 KB. Both transports shard; boot replay streams the map/topology off flash.Verified on an ESP32-C6
streamedoff flash; norename failed(LittleFSrenameworks).//pi/hitl/harness:map_upload): shards a 256-LED map + topology, assertschunk_ack/result_ready, and round-trips viaget_stored_map(led_count / segments / associations). Pure-logic window-plan + fixtures unit-tested in//pi/hitl/tests.Tests
store
BlockReader+ sharded reassembly (map + topology) +parse_upload_chunkguards; ffi/player arm coverage; web proto round-trip + a wss sharding client test; HITL parity test. Firmware builds-c opt; all host + web suites green.🤖 Generated with Claude Code