feat: page hole purging (Bun parity P7b)

[zackees]

Stacked on #299 (Phase 7a). Base branch is bun-parity/p7-scavenger, not main.
Closes the second clause of mi_on_thread_idle's contract.

imported from oven-sh/mimalloc @ 942b8342, MIT. Refs #272, #264, #10.

What this is

Upstream mimalloc returns a page's memory to the OS only when every block in it is
free, so one long-lived object pins a whole 64 KiB/512 KiB page. That is oven-sh/bun#39844
(heap peaks 1.4–2.6× Node). Hole purging discards the memory of the free blocks inside a
still-used page, in OS-page units, at each mi_on_thread_idle / park point.

The unit is an OS page because that is what madvise/MEM_RESET works in. page->purged
is a bitmap over the OS pages of the block area; an OS page is discarded only when every
block overlapping it is free, and a purged block is held off every free list (mimalloc
threads its free list through the free blocks themselves, so a discarded block cannot carry
a next pointer). _mi_os_discard never touches commit state — the arena tracks commit per
64 KiB slice and cannot represent a sub-slice hole.

Layout (CLAUDE.md rule 6)

Bun puts all of this in src/page.c (+1038) plus the sweep drivers in src/theap.c. Here
the whole engine is in a new src/page-holes.c and src/page.c carries exactly five
hook calls, each marked HOOK n/5 in the source:

#	site	call
1	mi_page_free_collect_ex	_mi_page_unpurge_run when the free list is empty but holes exist
2	mi_page_free_collect_ex	the _mi_page_free_collect_no_unpurge split (inspection must not mutate)
3	_mi_page_abandon	→ _no_unpurge
4	mi_page_extend_free	_mi_page_unpurge_unformed_upto before the first free-list pointer is written
5	_mi_page_init	_mi_page_purged_reset

plus two debug-only assertion lines (the conservation invariant, and
_mi_page_holes_assert_valid). arena.c gets _mi_page_unpurge_all at the single choke
point where a page returns to the arena, and the abandoned-page sweep/report walkers (they
need the arena bitmaps and the claim protocol). scavenger.c gets the _mi_purge_holes_of
phase and the purge_holes_min_interval pacing _mi_theap_sweep_parked was already shaped
for. theap.c counts purged blocks as free in the block visitor.

Public API: mi_purge_holes_stats_t / mi_purge_holes_stats_get / mi_purge_holes_report,
a mi_stats_print block, and purge_holes_stats() in the Rust crate. Options appended after
scavenger (no slot renumbered): purge_holes=1, purge_holes_eager_zero=0,
purge_holes_min_interval=100 ms, purge_holes_full_every=64.

Numbers

Peak RSS, Bun's workload shape (400k × 512 B, survivors every 2 OS pages, mi_on_thread_idle
between 12 rounds; min of 5, Release, Linux x64):

	peak RSS
MIMALLOC_PURGE_HOLES=1	105.0 MB
MIMALLOC_PURGE_HOLES=0	210.0 MB

Peak RSS, the gate + stress suite (min of 5). These never call mi_on_thread_idle, so
the engine is inert in them and the numbers only show it costs nothing. Same-binary
run-to-run spread on test-stress-heaps was 785.5 → 828.1 MB (5.4%) across two independent
min-of-5 rounds, which is larger than every delta below:

workload	holes=1	holes=0	7a base	this branch
test-memory-gate	27.1 MB	28.3 MB	—	—
test-stress	264.4 MB	259.5 MB	259.6 MB	253.8 MB
test-stress-heaps	785.5 MB	770.3 MB	794.3 MB	828.1 MB
test-process-rss	137.4 MB	137.4 MB	—	—

ci/memory_gate.py check against a fresh Release build: PASS, peak_rss 19.7 MB vs
baseline 22.3 MB (allowed 25.6) — no re-baselining needed, it went down.

Latency (#154 methodology: single-threaded alloc/free pairs over 8 small sizes,
N=4e6, best-of-5, three independent rounds, Release):

	7a base	this branch
plain alloc/free (ns)	25.91 / 27.18 / 25.34	25.44 / 23.86 / 25.99
against a heap that has holes (ns)	24.29 / 24.42 / 24.01	23.77 / 24.11 / 24.58

Within noise in both shapes, which is what the design predicts: the free path gains no
code at all in a Release build. The only new free.c line is inside MI_CHECK_DOUBLE_FREE
(debug/secure), because a block being freed was live and therefore was never inside a
discarded OS page. Posted on #10.

The one real cost: sizeof(mi_page_t) 144 → 192 bytes, all of it appended at the tail,
behind Bun's static assert that the hot fields still fit the first cache line. With
MI_PAGE_META_IS_SEPARATED that is +768 KB of arena page-meta per 1 GiB arena.

Profiler interaction (#272 design points 1–4), designed and tested

1. A discard never covers a sampled block. _mi_prof_debug_assert_no_records_in
   (src/profile.c, MI_DEBUG only) is called from src/page-holes.c before every
   _mi_os_discard and asserts that no record's block and no record struct lies in the
   range — records come from the profiler's raw-OS arena (rule 4), never from a page. It
   try-acquires prof_lock and skips rather than blocking: the sweep holds
   tld->theaps_lock and mi_prof_visit holds prof_lock across a user callback that may
   delete a heap, which would be an ABBA cycle.
2. No inspection path hands out a discarded block. _mi_theap_area_visit_blocks marks
   purged blocks in its free map, and every inspection collect became _no_unpurge.
3. A parked thread's mi_tld_t.profiler is untouched — nothing in the sweep reads or
   writes it; 7a's assertion around _mi_thread_idle_work still holds.
4. See "Two things Bun parity P7: background scavenger, page hole purging, and mi_on_thread_idle #272 got wrong" below.

test-profile-race.c gains a fifth scenario for 1 and 2: two owner-sweeping threads, two
parked threads swept by the scavenger, and a walker running mi_prof_visit,
mi_prof_snapshot_visit and mi_theap_visit_blocks over pages it is itself having swept.
Every pointer either walk yields is checked with mi_page_block_is_purged and dereferenced.
It asserts it was not vacuous: ~38k discards / ~270 MB and ~94 live records per run. A one-off
negative control (mi_assert_internal(false) inside _mi_prof_debug_assert_no_records_in)
confirmed that assert does fire on this workload.

Two things #272 got wrong

• purge_holes_eager_zero is not zero-tracking. It is a _mi_memzero performed before
  _mi_prim_discard, so a mis-scoped discard corrupts visibly on macOS's lazy
  MADV_FREE_REUSABLE instead of silently. It makes discarding more expensive. (First
  spotted by the 7a agent on Bun parity P7: background scavenger, page hole purging, and mi_on_thread_idle #272; confirmed against src/os.c:734-745 at 942b8342.)
• mi_option_purge_zeroes has no implementation to unify with. It is dead in this tree
  since the Bump the upstream pin from 579f8c0e to the dev3 tip #80 pin bump (Phase D: zero-tracking (zalloc skips memset after zero-purge) #67). The slot is kept and never renumbered, MIMALLOC_PURGE_ZEROES
  still parses, and mimalloc.h now says so explicitly rather than the option being silently
  dropped. test-purge-zero.c's header records that despite its name it does not test that
  option either.

Also, contrary to the issue's rule-6 line: there is no mi_page_can_purge_holes call on the
free fast path, and _mi_page_unpurge_run is not a mi_page_queue_find_free_ex hook — it
lives in _mi_page_free_collect, which that function calls.

Tests

• test/test-purge-holes.c — Bun's own 1705-line, 27-case test, imported and passing
  unmodified apart from two adaptations noted in its header (the src/page.c file
  references become src/page-holes.c; the one open-coded sweep state calls the shared
  mi_page_sweep_state() inline). Registered twice: as itself, and as test-purge-holes-off
  with MIMALLOC_PURGE_HOLES=0, where every data-integrity check still runs and nothing may
  be discarded.
• test/test-purge-zero.c — verbatim.
• test/test-park-handoff.c — both Phase-7b caveats removed: the two
  purge_holes_min_interval cases are live (in-window park observed swept at 96 ms of a
  100 ms window, vs the 30 s safety net), a new test-park-handoff-eager variant runs the
  file with MIMALLOC_PURGE_HOLES_MIN_INTERVAL=0, and the run asserts hole memory was
  actually discarded (1902 discards / 19 MB here; exactly 0 with the option off).
• test/test-profile-race.c — scenario 5 above.

Existing suite: every one of the 55 debug-full tests passes with purge_holes default-on and
purge_holes_eager_zero forced on by MI_DEBUG>1, which means a mis-scoped discard would
have corrupted visibly. _mi_page_holes_assert_valid and the conservation invariant in
mi_page_is_valid_init make each of those a test of the purge machinery.

Local verification (Linux x64, gcc)

config	result
Debug + MI_DEBUG_FULL + MI_PPROF=ON	55/55, 10/10 consecutive runs of the five hole/park/race tests
Release + MI_PPROF=ON	46/46
Release + MI_PPROF=OFF	36/36
gcc ASan (MI_TRACK_ASAN=ON, debug-full)	54/54
g++ -x c++ syntax check of src/static.c (the native MSVC gate compiles as C++)	clean
cargo test -p mimalloc-pprof	green
ci/check_internal_state.py, ci/check_doc_snippets.py, ci/memory_gate.py check	green

uv run ci/verify_local.py --keep-going results are in a follow-up comment.

Known / stated, not fixed here

• _mi_os_discard's failure path calls _mi_warning_message, which can allocate; on the
  scavenger thread that would create a theap and trip 7a's "no theap on the scavenger"
  invariant. Bun accepts this and so does this PR — it is the ENOMEM path only.
• macOS: compile-verified only for MADV_FREE_REUSABLE; arm64 via the cross toolchain,
  x64 pending the golden image. The macOS CI lanes are the real check.
• _mi_deferred_free in this tree lacks Bun's owner-only guard
  (tld->thread_id != _mi_thread_id() early return), so the scavenger's mi_theap_collect
  for a parked thread can invoke a user deferred-free callback on the scavenger thread. That
  is pre-existing on 7a, not introduced here; reported on feat: background scavenger thread and mi_on_thread_idle (Bun parity P7a) #299 rather than fixed on a branch
  another agent owns.
• cargo fmt --check flags rust/mimalloc-pprof/build.rs — verified pre-existing on
  origin/bun-parity/p7-scavenger, untouched here.

Do not merge before #299.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YT4jVokb2gdT8ngFrH8i8S

[zackees]

Update after the first CI round

Four lane-only problems, all fixed and each verified locally against the lane's exact configure line. None of them touched the engine.

lane	symptom	cause	fix
ctest-shared (linux + windows), *-x64-shared bundles	undefined reference to _mi_os_page_size / __mi_page_map	MI_BUILD_STATIC=OFF hides the internal symbols; test-purge-holes.c needs them throughout, test-profile-race.c's new scenario 5 for two assertions	test-purge-holes is now static-lane-only (its whole add_executable is inside the MI_BUILD_STATIC branch); scenario 5's assertions are behind MI_TEST_LINKS_STATIC and its page size falls back to sysconf/GetSystemInfo, so the scenario still runs everywhere as the concurrency test it also is
build-windows-msvc (all), ctest* (windows-latest)	lld-link: undefined symbol: _mi_os_page_size even against the static library	those lanes build the library as C++ (MI_USE_CXX), and mimalloc/internal.h — unlike mimalloc.h — is not extern "C", so the internal symbols are mangled	both tests joined CMakeLists' existing "compile as C++ when the library is" list, next to test-api.c which is there for the same reason; also MI_STATIC_LIB on the static branch, as mimalloc-test-tls-controls already needs
ctest-shared-win-gnu	unknown type name 'SYSTEM_INFO'	the shared-lane page-size fallback sits above the file's threading shim, which is where <windows.h> came in	include it where it is used
ctest-guarded pass 2 (MIMALLOC_GUARDED_SAMPLE_RATE=1)	page ... is not exclusively ours: used=8 but we hold 7 in 6 of 25 cases	guarding every allocation makes each one oversized, guard-page-backed and returned as an interior pointer, so a test can neither derive a block's index in its page nor keep a page to itself	those six cases (report-*, both unformed-tail*, sweep-skips-unchanged-pages) skip loudly when the sample rate is 1; layout_is_predictable() in the test says why. The engine is not guard-sensitive — a guarded block is an ordinary free-listed block to the sweep — and the lane's first pass still runs all 25

Locally verified with each lane's own configure line: shared 41/41 · MI_USE_CXX=ON links and 5/5 on the purge + race tests · MI_GUARDED=ON green in both passes.

One open decision: the memory gate

memory-gate (ubuntu-latest) is red and I have not re-baselined it. Full evidence and the three options are on #272 (#272 (comment)). Summary:

• 7a base on CI: 25.3 MB, allowed 25.6 (passes with 1.2% headroom). 7b: 26.5 MB.
• Locally, interleaved, 4-CPU-pinned, min of 10 alternating runs: 7a 21.9 MB, 7b 24.0 MB with purge_holes=1, 24.1 MB with purge_holes=0.
• The option makes no difference, so this is not the sweep — it is sizeof(mi_page_t) 144 → 192 B, which mi_arena_info_slices_needed multiplies by the arena's slice count and commits up front: +0.75 MiB of page-meta per 1 GiB arena. Both builds reserve exactly one 1 GiB arena (MIMALLOC_VERBOSE=1). The remaining ~1 MB of the min-of-N delta is unattributed; the medians are 25.1 vs 25.3 MB and the gate's own output warns its spread (38.6% locally, 21.3% on 7a's CI run) exceeds its 15% tolerance.
• MI_PAGE_PURGE_BITS cannot drop to 128 (a 512 KiB page on a 4 KiB OS page needs 129 bits, so every medium page would become ineligible — most of the benefit), and shrinking swept_state to 32 bits saves nothing after alignment. Bun pays the same 48 bytes.

Re-baselining is the option I'd pick, from artifacts of a green run on all three platforms, but it is a reviewed act and the alternative (lazy pages_meta commit) is an arena change that does not belong in a parity import. Your call.

Two 7a findings, reported not fixed

Both on #299, neither touched on either branch:

• _mi_deferred_free is missing Bun's owner-only guard — the scavenger's mi_theap_collect for a parked thread writes another thread's plain heartbeat/recurse fields and can invoke the embedder's deferred-free handler on the scavenger thread.
• _mi_theap_can_touch reads the 32-bit park_state through the 64-bit MSVC atomic primitive (C4133 in the windows-msvc build log) — __iso_volatile_load64 on an _Atomic(uint32_t), so the compare also sees park_reclaim in the high half. Not caught by CI because test-park-handoff.c is #ifdef _WIN32-skipped, i.e. the park protocol has no Windows execution coverage at all today.

[zackees]

Final state

Head is a1c96855, which includes a merge of bun-parity/p7-scavenger at 8885a8af (the Windows exit-hang fix and the merge of main).

The memory gate passes; my earlier escalation on #272 is withdrawn. On this head, memory-gate (ubuntu-latest) reports 25.2 MB against 7a's own 25.3 MB, allowed 25.6 — 7b is marginally below the base branch. The 26.5 MB that prompted the escalation was one unlucky min-of-8 on a shared runner whose spread that same run reported as 24.2%, above the gate's own 15% tolerance. memory-gate (windows-latest) passes too. No re-baseline needed; ci/memory-baselines/*.json is untouched.

Two 7a interactions audited, no code change needed (da43fabc records the reasoning in src/page-holes.c):

• After _mi_scavenger_stop sets _mi_scavenger_shutdown, no sweep can start on the scavenger — the run loop exits on _mi_scavenger_running == 0, the stop joins it, _mi_scavenger_start_lazy refuses to restart, so mi_on_thread_idle_start hands nothing off. A direct mi_on_thread_idle() after that sweeps on the calling thread over its own theaps, safe at any point.
• mi_heap_delete/_destroy calls _mi_park_leave on every parked owner before _mi_heap_detach_theaps, and that call does not return until MI_PARK_SWEEPING clears — so no theap is detached under a running sweep. It terminates against this sweep because every phase re-reads tld->park_reclaim: between heaps (_mi_purge_holes_of), between pages (mi_theap_page_purge_holes), between abandoned pages (mi_arena_page_purge_holes_at). The wait is bounded by one page's walk, and tld->theaps_lock is always released before the deleter needs it.

No new atomics of the C4133 class (a1c96855 writes the rule into types.h next to the fields). This branch adds zero _Atomic fields: all six mi_tld_t hole-sweep fields and all four mi_page_t hole fields are plain, because only the single thread performing a sweep touches them. The process-wide counters in src/page-holes.c are int64_t reached through mi_atomic_addi64_relaxed — the 64-bit primitive on a 64-bit field, as mi_stat_counter_t does. My reads of 7a's park_state/park_reclaim go through mi_atomic_load_* and become correct automatically when those fields widen to _Atomic(size_t); I will re-verify at the next merge.

uv run ci/verify_local.py --keep-going on this head

config	result
release, off, debug-full, guarded, shared, bundle, memory-gate, diag, rust, lint	PASS (10/10)
asan	FAIL — environmental only: MI_TRACK_ASAN did not survive configure (no clang on this box). Run by hand with gcc instead: MI_TRACK_ASAN=ON debug-full, 54/54.

guarded covers both passes, the second with MIMALLOC_GUARDED_SAMPLE_RATE=1. Separately, 10/10 consecutive runs of test-purge-holes, test-purge-holes-off, test-purge-zero, the three test-park-handoff variants and test-profile-race on the debug-full tree.

CI

Everything green except two, neither of them this branch's:

• run-macos-x64 (dockurr/macos on Linux) — No golden macOS disk under cache key 'macos-x64-golden-ventura-v1'. Fails identically on the base branch feat: background scavenger thread and mi_on_thread_idle (Bun parity P7a) #299; macOS x64 needs the golden image published before any PR can exercise it. macOS arm64 is green (build-macos (macos-arm64-{release,debug-full,leak})), so MADV_FREE_REUSABLE is run-verified there.
• test (ubuntu-latest) (rust-native) — error: could not compile 'proc-macro2' (build script) ... exit status: 255 under the soldr rustc shim, with zccache ... errors=1. A toolchain/cache failure with no relation to the diff; the same job passed on the previous head, cargo test is green locally, and the other five rust-native targets (linux-musl, both darwin, both windows) pass on this head. Needs a re-run.

All Windows lanes are green, including the ones that matter for MEM_RESET/VirtualUnlock: ctest-debug-full (windows-latest) (MSVC Debug, where MI_DEBUG>1 forces purge_holes_eager_zero so every discard memzeroes its range first), ctest (windows-latest), ctest-shared (windows-latest), all four windows-msvc bundles and all four windows-gnu bundles.

[zackees]

CI settled: 60 pass / 2 fail, and neither failure is this branch's

check	verdict
ctest-debug-full (ubuntu-latest)	pre-existing race on the base branch. test-park-handoff trips mi_theap_visit_pages's count == total assertion inside test_park_stress, on the scavenger, walking a parked thread's theap. Reproduced on bun-parity/p7-scavenger @ 2fcac37b at 3/120 runs versus 2/120 on this branch, with the same assertion and the same stack. Full evidence, instrumented-probe results and reproduction recipe on #299.
run-macos-x64 (dockurr/macos on Linux)	No golden macOS disk under cache key 'macos-x64-golden-ventura-v1'. Infrastructure; fails identically on #299.

On the first one, the numbers that matter:

branch	config	failures / 120 runs
bun-parity/p7-scavenger @ 2fcac37b	default	3
this branch @ a1c96855	default	2
this branch @ a1c96855	MIMALLOC_PURGE_HOLES_MIN_INTERVAL=0	2

Pinned to 4 CPUs (taskset -c 0-3), debug-full; unpinned it does not reproduce. The hole sweep gives the parked-thread walk more to do per park, so it is a marginally better amplifier of the race — but the base branch's rate is higher, and forcing a sweep on every park (min_interval=0) does not change it, so sweep frequency is not the mechanism. At ~2.5% per run this lane is green most of the time on both branches; #299's own ctest-debug-full being green is luck rather than evidence.

I have not worked around it here — no assertion relaxed, no test skipped, no ctest variant dropped. It is #299's to fix, and 7b will inherit the fix.

Everything else is green, including every Windows lane (ctest-debug-full (windows-latest) is the MSVC Debug one where MI_DEBUG>1 forces purge_holes_eager_zero, so every discard memzeroes its range before MEM_RESET + VirtualUnlock), macOS arm64, both memory gates, bundle-roundtrip, address sanitizer, fuzz, amalgamation-drift and all six rust-native targets.

[zackees]

Merge order, to be explicit: ctest-debug-full (ubuntu-latest) will stay red on this PR until #299 fixes the parked-sweep race. That is not a 7b defect — it reproduces at a higher rate on the base branch (3/120 vs 2/120), including on the head that already carries the park-field atomic-width fix — but it does mean this PR should not be merged before #299 lands the fix, and it will go green here on its own once it does. Do not merge.

[zackees]

Opus review (two rounds): APPROVE. Hole engine in src/page-holes.c with 5 hooks in page.c; free list rebuilt before discard; zero contract preserved; profiler records off-page with a per-discard MI_DEBUG assert; owner-driven sweeps paced; MinGW test-profile flake root-caused (fixed 12 ms spin window lost the workers' quantum on Windows; now sized by allocation, 6606336e). Memory gate: runner read 30.5 MB with 21.6% spread (#298); pinned local A/B on identical hardware, min-of-8: main 22.7 MB vs this head 24.1 MB (+1.4 MB, within tolerance; the per-page bitmap). Churn RSS after idle 230.7→73.2 MB (README bench), latency neutral. Merging; closes #272.