Custom Edition maps a host does not send downloaded from the community's archives

[FractumSeraph]

Custom Edition maps that a host does not send are now downloaded from the community's archives (HaloNet, then HaloMaps, then a web seed). Each file is checked against an index of the maps before the game uses it. Map sharing's downloads from the game's players also take pieces from the web seed, so a host with a slow upload doesn't hold its players up.

This builds on #333 and #334, and its branch contains their commits until they are merged; after that I'll rebase it so only its own two commits remain: "Custom Edition maps a host does not send downloaded from the community's archives" and "Map sharing takes pieces from the web seeds too". #333 gives a downloaded map its name (<name>_<hash>.map, which the game finds without renaming), and #334 gives the download screen its drawing.

This replaces #315, which added a BitTorrent client and reverted map sharing, and no longer merged with main (the web build among it). This one is built on main as it is, network version 28, and keeps map sharing as it is. It has no BitTorrent client and changes nothing on the wire.

When it downloads

Map sharing downloads a missing map from the game's players. When it cannot, the game downloads the map instead:

• the host declines (SHARE MAPS off);
• the level is named without its hash (a host of another build);
• a game in the games list is joined without its map.

A map comes from the zip on HaloNet's repository or on HaloMaps (maps.archives), taken out of the zip, and otherwise from a web seed (maps.web_seeds).

Every file is checked against the index (port/assets/network/map_index.txt: 5,829 maps plus the three resource maps) by the SHA-1 info hash of its single-file torrent. A file that fails is thrown away and the next source is tried.

Halo's loading screen covers the menus while a map downloads, and B stops the download. From the games list, the join goes ahead by itself once the map is there. When the host declined, the player is told to join again.

Details are in docs/map_downloads.md.

Web seeds for map sharing

When map sharing downloads a map from the game's players, the web seed sends pieces of it too:

• It is used when the index has the file at the size the host offered: the map by its name and the host's header checksum, a resource map by its name.
• It is asked for 16 pieces (4 MiB) at a time, pieces no player has been asked for, with an HTTP range request on a thread of its own (update_request_range on Linux and Windows).
• Each piece is checked against its hash, as a player's piece is. A piece that isn't the file's ends the web seed for that file; three failed requests end it for the download, with a pause after each.
• A range reply must be 206 for exactly that range (Content-Range) and its full length. A server that sends the whole file instead is refused unread.
• No setup: by default it's the seed box's folder of every map in the index. It is off with maps.downloads = false or an empty maps.web_seeds.

Nothing of this goes over the game's connections, and players of other builds are unaffected.

No map file is ever replaced or renamed

• A downloaded map goes into custom_maps as <name>_<digits>.map, the same name map sharing gives that version, beside any other version.
• A resource map is written only when none is there.
• On POSIX, the name is taken with O_EXCL before the rename.
• When a host names its level without the hash and the player's <name>.map is another version, the client plays the version beside it whose header checksum is the host's.

Platforms

• Linux and Windows download, using the self-updater's HTTPS. update_request adds a POST, a user agent and a stop flag to posix_update.c and win32_update.c, plus update_request_range; its download cap is 1 GiB (the largest map in the index is about 760 MB).
• Android has no HTTPS in its platform layer, so it doesn't download from the archives or the web seed. Map sharing works as before.
• The web build leaves the downloads out (port/web/src/web_map_downloads.c).

Network version

Unchanged (28). Nothing new goes over the game's connections.

The web seed and torrents

The default web seed, https://halomaps.fractumseraph.net/maps/, is my seed box's folder of every map in the index. The index's info hashes are those of the torrents it seeds, so anyone can seed the same files with any BitTorrent client. A BitTorrent source could be added later as another entry in maps.archives, without changing the index.

Checked

• The whole branch (all three commits) was tested together; each commit also builds on its own.
• Linux CI steps in Docker (tools/ci_build.py linux release): the test step's pytest list with tools/test_map_index.py and the new tools/test_map_download_check.py (335 passed; each commit also builds on its own), and test_light_storage.py.
• Web CI steps in Docker (emsdk 6.0.10): ci_build.py web release, tools/test_web_build.py, node --test port/web/tests (16 passed).
• Windows: ci_build.py windows release built with clang-cl and MSVC 2019's libraries. Then run on Windows:
  • A host with map sharing off and a client without the map. The map came from the web seed over WinHTTP, and in a second run from HaloMaps alone. It went in as ChaosGulch_7cac43f9.map and was played on the second join; the host's chaosgulch.map was unchanged.
  • A host sharing at 16 KB/s: 13.4 of the 13.9 MB came from the web seed in WinHTTP ranges, and the client played the map.
• End to end on Linux in Docker, on the loopback, with OpenCE build-201 and a ChupathingyCE-based dedicated server (version 28, with plain level names when set to):

Case	Result
This build hosts; build-201's client has no map	Map sharing sends it under the hash name; the host's chaosgulch.map keeps its name and bytes
Build-201 hosts; this build's client has no map	Map sharing sends it; the second join plays the downloaded copy
Host with SHARE MAPS off	The client is told to join again; the map comes from the web seed (chaosgulch), and from HaloNet's zip (battlecity_lvl1) beside the client's own other version; each is the host's byte for byte, and plays on the second join
Server sends custom_maps\chaosgulch (no hash); the client has another version	The host's version comes from HaloMaps's zip as ChaosGulch_7cac43f9.map; the second join plays it, chosen by header checksum; the client's own copy is unchanged
An index whose info hash is not the file's	The download is refused and nothing is written
Server sends custom_maps\chaosgulch; the client has two other versions named for their hashes and no chaosgulch.map	The host's version comes from the web seed as ChaosGulch_7cac43f9.map; the second join plays it, chosen by header checksum; the client's two files are unchanged
Map sharing with the host's upload limited to 16 KB/s	13.4 of the 13.9 MB came from the web seed; the map is the host's byte for byte, and the client played
Map sharing with a web seed that answers 404	Three failed requests, then the web seed is left alone; the host sent the map
Map sharing with maps.downloads off	No web seed asked; the host sent the map
A second run of a host	map_hashes.txt is used and nothing is hashed again
Loading screen, recorded on a virtual display	The light sweeps over the Xbox picture as paradis_perdu (126 MB) arrives from the web seed, with "from the web seed", the amount counting up, and B=STOP

No map file was renamed or changed by this build in any run.

Found along the way and left alone (they are in build-201 too): with debug.network_test, a client that joins a lobby game whose host starts it mid-download stops responding (so the two map sharing rows above use debug.network_test_start to give the download time); and the Windows build does not exit at debug.exit_after.

🤖 Generated with Claude Code