Skip to content

Custom Edition maps downloaded before joining (archives + BitTorrent), keeping network version 27 - #315

Closed
FractumSeraph wants to merge 3 commits into
OpenCommunityEdition:mainfrom
FractumSeraph:map-torrents-pr
Closed

Custom Edition maps downloaded before joining (archives + BitTorrent), keeping network version 27#315
FractumSeraph wants to merge 3 commits into
OpenCommunityEdition:mainfrom
FractumSeraph:map-torrents-pr

Conversation

@FractumSeraph

@FractumSeraph FractumSeraph commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Custom Edition maps a player lacks are now downloaded before the join, over BitTorrent, from the map archive's seeds, its web seed and other players. This replaces the host-to-player map sharing of builds 189 and 190. The Xbox loading screen now draws, and covers the menus while a map downloads.

What it does

  • Download before joining. Joining a game (LAN, Direct Link or the Server Browser) on a Custom Edition map this machine lacks looks the map up in an index of the archive's torrents (port/assets/network/map_torrents.txt, 5,829 maps plus bitmaps.map, sounds.map and loc.map). The lookup uses the file name and the header checksum the host already sends as the map version. The map downloads, and the join goes ahead by itself once it is there. Resource maps that are missing come the same way. If the host changes map after the join, the map downloads and the player is told to join again.
  • No .map file is ever replaced, renamed or changed. When the host's map is another version of one the player has, the host's version goes beside it, under its own name, in custom_maps/versions/<checksum>/<name>.map. A client in that game plays that copy, and the player's own copy stays as it was. The map lists show only custom_maps itself, so hosting still uses the player's own copy.
  • Seeding. The host seeds the map it is playing at no more than maps.upload_limit (512 KB/s), so the game's traffic comes first. With maps.seed = "all", clients seed too.
  • Loading screen. Halo's own loading screen (interface/progress_bar.c) drew nothing before: it uses the fixed-function pipeline, which d3d8_gl.c did not have. It now draws, including the light's trailing glow, which reads the previous frame through GetBackBuffer(-1).
    • It covers the menus while a map downloads, with the map's name and progress. B stops the download.
    • Its picture is the Xbox's own (the ring and LOADING...). The retail disc has no file of it; default.xbe holds it. Importing a disc image now writes it to maps/loading.tga: on desktop (xiso.c), Android (XisoExtractor.java) and the web page (xiso-worker.js). Without a disc image's picture, the light sweeps over clouds the game generates. No game data is in the repository.
  • The client is the port's own, in C (port/linux/src/torrent*.c, about 4,000 lines on the platform layer's sockets and threads). It speaks BEP 3, 9, 10, 11, 15 and 19 (peers, metadata, PEX, UDP and HTTP trackers, HTTP web seeds). It does not speak the DHT, uTP, encryption or HTTPS. The web build leaves it out: a page has no TCP or UDP sockets and plays no Custom Edition maps.

Design, settings and the archive's servers are described in docs/map_torrents.md. The [maps] settings are in port/linux/README.md.

Network version

This PR does not add a network version. It removes the map sharing commits (a2aad18, 71ceb36), which were the only change in version 27. The torrent downloads send nothing over the game's connections. So the protocol is version 26's again, and HALO_PORT_NETWORK_VERSION goes back to 26:

  • These builds play with every build up to 188.
  • Builds 189 and 190 (version 27) are told to update, as with any version mismatch.

NETCODE.md says so.

Servers

The defaults point at the archive I run:

  • tracker: opentracker on halovps.fractumseraph.net:6969, UDP and HTTP. It takes only the index's torrents.
  • web seed: http://halomaps.fractumseraph.net/maps/.
  • seed box: a libtorrent seeder of all 5,800+ torrents.

The settings maps.trackers, maps.web_seeds and maps.torrent_index can point anywhere, and anyone can seed the .torrent files.

Checked

  • The Linux CI steps in Docker (archlinux, tools/ci_build.py linux release):
    • the test step's pytest list, with tools/test_map_torrents.py and the new tools/test_torrent_check.py added;
    • tools/test_light_storage.py.
  • The web CI steps in Docker (emsdk 6.0.10):
    • tools/ci_build.py web release;
    • node --test "port/web/tests/*.test.js";
    • tools/test_web_build.py.
  • New tests:
    • tools/test_torrent_check.py: ninja linux now also builds build/linux/torrent_check. One copy seeds a made-up file on the loopback and another downloads it, which must give the file byte for byte. A peer with the wrong data gives nothing.
    • port/web/tests/xiso.test.js: the loading picture taken out of made-up executables.
    • The Java decoder gives the same bytes as the JavaScript one.
  • End to end on Linux in Docker: a host on chaosgulch, and a client whose custom_maps/chaosgulch.map is another version (header checksum changed).
    • The first join downloaded the host's version into custom_maps/versions/7B309554/ (same bytes as the host's); the client's own copy was untouched.
    • The second join loaded the map from the versions folder and played.
  • Not built here: Android and Windows. Their build scripts copy map_torrents.txt as they copy brokers.txt. The client and the hooks are the same code the Linux build compiles.

🤖 Generated with Claude Code

FractumSeraph and others added 3 commits October 12, 2026 01:31
@FractumSeraph @claude
Remove Custom Edition map sharing, keeping network version 27
e71edb0 
Reverts a2aad18 ("Custom Edition map sharing") and 71ceb36 ("Players who
already have the map no longer wait for a machine that is still downloading
it."), which this pull request replaces: a Custom Edition map a client
lacks is downloaded before it joins, from the community's map archives or
over BitTorrent (the commits that follow), and no map a player has is ever
renamed. Gone with it: the map sharing transfers, the players' sessions
with each other for it, the renaming of maps for their hashes, the MAP
SHARING settings screens and the network.map_* settings.

HALO_PORT_NETWORK_VERSION stays 27, and builds 189 and 190 (version 27,
with map sharing) play with these as with a version 27 machine that does
not share maps (port/linux/NETCODE.md, version 27):

- a request for the game's map (distributed message 83) is answered with
  version 27's own "off" offer (port/linux/game/map_share_decline.c; once
  in five seconds a machine, at most), so that player is told at once that
  the host does not share its maps; the
  other map sharing messages (83 to 86) are dropped, as any message of an
  unknown type already is, and the numbers stay reserved;
- the maps those builds renamed in a player's custom_maps
  (<name>_<hash>.map) stay as they are and play, and a version 27 host's
  hash-named levels are found by their names: the map downloads' commit
  (custom_edition_cache.c, custom_edition_maps.c, map_torrents.c).

Internet play's signalling is byte for byte the same (27 only reused the
join and accept messages to let players reach each other for maps).

Checked, on the loopback in Docker against OpenCE's build-190 release and
ChupathingyCE 0.7.3e's dedicated server (both version 27): build-190's
host and client play with these builds' on a stock map both ways; these
builds' client joins the 0.7.3e server; build-190's client without the map
asks this build's host for a hash-named map and is declined at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FractumSeraph @claude
Halo's loading screen draws, with the Xbox's own picture or clouds
80eeb39 
The loading screen (interface/progress_bar.c: a light sweeping from left to
right over a picture, with a glowing trail) drew nothing: it uses the
fixed-function pipeline (SetVertexShader with 0), which d3d8_gl.c did not
have, and its picture, loading.tga, is on no retail disc.

- d3d8_gl.c draws fixed-function immediate-mode draws with a vertex shader
  of its own (nv2a_fixed_function_glsl), positions transformed by the
  world, view and projection matrices; and keeps a front buffer (a 640x480
  render target taking the middle of the frame at BlockUntilVerticalBlank)
  for GetBackBuffer with -1, the frame before, whose blur makes the light's
  trail. The screen is drawn in the middle 640 columns, as the menus are on
  a wide screen.
- After it, the rasterizer's caches of the device's states and of its
  vertex shader are made the device's again (rasterizer_forget_hardware_state,
  rasterizer_vertex_shader_runtime_forget): it sets them itself; and
  SelectVertexShader (a packed program) always leaves the fixed-function
  pipeline.
- On OpenGL ES (Android, the web) the fixed-function shader turns
  Direct3D's clip space as the others do (HALO_GLES); without OpenGL
  (debug.null_renderer) it is not compiled.
- The picture is the Xbox's own: the retail default.xbe holds it (320 by
  240, runs of 4-bit greys after its width and height), and importing a disc
  image now writes it into the maps folder as loading.tga: port/linux/src/
  xiso.c, the Android app's XisoExtractor.java and the web page's
  xiso-worker.js. Without one (a maps folder copied by hand) the light
  sweeps over clouds the game makes. No game data is in the repository.
- progress_bar.c also has the mode the next commit's download screen uses
  (a share given, from 0, drawn over the menus, and let go of).
- A picture that cannot be written whole is not left cut short (the
  loading screen would read past it).

Checked: port/web/tests/xiso.test.js takes the picture out of made-up
executables (and an image's default.xbe); the Java decoder gives the same
bytes as the JavaScript one; xiso.c's picture from a Rev 2 disc image
matched a separate decoder's byte for byte; the Linux and web builds in
Docker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FractumSeraph @claude
Custom Edition maps downloaded before joining, from the archives or o…
6466596 
…ver BitTorrent

A machine joining a game on a Custom Edition map it lacks (or has another
version of, or without Custom Edition's resource maps) downloads it before
the join, and joins by itself once it is there; the host seeds the map it
plays. Design and details in docs/map_torrents.md.

- The index of the maps' torrents (port/assets/network/map_torrents.txt,
  tools/map_torrents.py): 5,829 maps and the three resource maps, by file
  name and header checksum (the map version the host already sends), and
  for 5,826 of them the community archives' zips that hold them.
- The archives first (port/linux/src/map_archives.c, maps.archives):
  HaloNet's zip by its URL, HaloMaps's as its Download button fetches it,
  over HTTPS with the self-updater's download (update_request in update.h:
  a POST, a user agent and a stop flag added to posix_update.c and
  win32_update.c), the map taken out of the zip (map_archives_zip.c, every
  offset checked, its CRC too). What they give is added to the BitTorrent
  client as a seed, which hashes it against the info hash; what they do
  not, or what is not the index's, comes over BitTorrent.
- The BitTorrent client (port/linux/src/torrent*.c, on the platform layer's
  sockets and threads: BEP 3, 9, 10, 11, 15, 19), its downloads and seeding
  (map_torrents.c), the [maps] settings, and the hooks: ui_widget_port_join
  waits for the map (menu_functions.c, which posts the join again),
  cache_files_map_present downloads a map the host changed to, the server's
  listing and the client's settings seed the map played.
- No map a player has is replaced, renamed or changed: another version of
  one goes in custom_maps\versions\<checksum>\<name>.map, which a client of
  a game on that version plays (custom_edition_cache.c,
  custom_edition_map_version_forget once it leaves); the map lists show
  only custom_maps itself.
- The download screen (port/linux/game/download_screen.c,
  maps.download_screen): Halo's loading screen over the menus at the
  download's share, the map's name over it, B to stop it; the menus go on
  behind it (nothing drawn without a renderer, debug.null_renderer).
- The downloads and seeding are polled each frame from main.c's loop, a
  window or not (a server, debug.null_renderer).
- The builds put map_torrents.txt beside the game (and in the APK); the web
  build leaves the client out (port/web/src/web_map_torrents.c).
- tools/torrent_check.c runs the client and the unzip alone; ninja linux
  builds it.

- Network version 27's hosts and players (builds 189 and 190): a host's
  level name with the map's hash after it (custom_maps\<name>_<hash>) is
  found as <name>.map (custom_edition_cache.c, map_torrents.c), and the
  maps those builds renamed for their hash in a player's custom_maps stay
  as they are, found for a host's custom_maps\<name> when their checksum
  is the host's and listed without the hash (custom_edition_maps.c).

Nothing goes over the game's connections: no network version of its own.

The client is careful with what peers send: a metadata piece only as asked
of the peer, of the size it told (one stalled at another size let go of);
a block only as asked; a peer whose blocks fail a piece's hash banned (at
once when it sent the piece alone, else at its third) and not connected to
again; requests served of
two blocks at most, their offsets checked without overflow; four incoming
connections from one address at a time. Names are looked up on a thread of
their own and the game's thread reads a torrent's status without waiting
for the client's (whose lock a lookup or a file's hashing holds); one HTTPS
request at a time over Mbed TLS (no threading support in it); a file moved
into place never replaces one there (on POSIX, whose rename would, the name
is taken first with O_EXCL); a download's file is not opened through a
link; the lists of files to fetch are tab-separated (some maps' names have
commas); an index entry of checksum 0 is that version, not any; a map
name with a pattern's characters (* ?) is never listed for; a web seed's
server is looked up before a piece is taken for it; files are hashed 4 MB a
tick under the client's lock.

Checked: the Linux CI steps in Docker (301 passed, test_profile's
ThreadSanitizer check failing only as Docker forbids it, with the new
tools/test_torrent_check.py: a file seeded and downloaded on the loopback,
a peer with other data refused, maps taken out of made-up zips and damaged
ones refused; tools/test_map_torrents.py with the sources); the web build
and its tests; the Windows files' compile (syntax, MSVC's and the SDK's
headers). End to end on Linux in Docker, two machines on the loopback: a
client with another version of chaosgulch got the host's over BitTorrent
into custom_maps\versions\7B309554 (its own copy untouched) and played it
on the next join; with no trackers or web seeds, a client without
chaosgulch got it out of HaloMaps's zip, and one with another version of
battlecity_lvl1 got the host's out of HaloNet's into the versions folder
and played it; each the same as the host's, byte for byte. Against
build-190 (network version 27): its host's hash-named chaosgulch was
downloaded by its name and played; a client holding only a map build-190
renamed played it for a host's plain chaosgulch (no download, the file
untouched); this build hosting a renamed map, build-190's client was
declined, this build's client downloaded it by its name and played, and
build-190's client holding it played.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FractumSeraph FractumSeraph mentioned this pull request Oct 13, 2026
Open
@FractumSeraph FractumSeraph changed the title Custom Edition maps downloaded before joining (archives + BitTorrent), keeping network version 27 Custom Edition maps a host does not send downloaded from the community's archives Oct 13, 2026
@FractumSeraph FractumSeraph mentioned this pull request Oct 13, 2026
Open
@FractumSeraph FractumSeraph changed the title Custom Edition maps a host does not send downloaded from the community's archives Custom Edition maps downloaded before joining (archives + BitTorrent), keeping network version 27 Oct 13, 2026
@FractumSeraph

FractumSeraph commented Oct 13, 2026

Copy link
Copy Markdown
Author

Closing in favour of #335, which downloads the maps from the archives and a web seed on top of main as it is (network version 28, map sharing kept), without a BitTorrent client. It builds on #333 (maps keep their names) and #334 (the loading screen). Thanks, and thanks Pablo for flagging the web build merge problem.

FractumSeraph pushed a commit to FractumSeraph/OpenCE that referenced this pull request Oct 16, 2026
@MrMilenko
Merge pull request OpenCommunityEdition#315 from ChupathingyCE/readme…
5d789e8 
…-refresh

README: badges and a Builds section
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@FractumSeraph