PyAV 17 wheel: ~5x more resident decode memory mappings vs PyAV 16 (same bundled FFmpeg 8.0.1)

[jacob-aegis]

Summary

Upgrading the av wheel from 16.1.0 → 17.x increases resident memory of a
many-concurrent-decoder H.264 workload by ~12% RSS, driven by a ~5×
increase in the number of large anonymous memory mappings per decoder. The
increase reproduces with the bundled FFmpeg version held constant at 8.0.1
(both av==16.1.0 and av==17.0.1 ship FFmpeg 8.0.1), so it is not an
FFmpeg-version change — it is something in the PyAV 17 wheel (the Cython
layer and/or how the wheel builds its bundled FFmpeg).

Environment

• Platform: Linux aarch64 (AWS Graviton), CPython 3.12, manylinux av wheels
• Allocator: jemalloc via LD_PRELOAD
  (narenas:4,dirty_decay_ms:1000,muzzy_decay_ms:1000,oversize_threshold:20MB)
• Workload: one process decoding ~13–14 concurrent RTSP H.264 1280×720
  streams, software decode, codec_context.thread_count = 1, ~4 reference
  frames. Per frame: container.demux(stream) → packet.decode() →
  frame.to_ndarray(format="bgr24").

Controlled measurement (single variable = the av wheel)

Same site, same cameras, same process layout; only the av pin changed.
Per-process (~14 decoders), steady state:

av	PyAV	bundled FFmpeg	large anon mappings / process	je_alloc	RSS
13.1.0	13	7.0.2	~300	~610 MB	baseline
16.1.0	16	8.0.1	~270	~620 MB	baseline
17.0.1	17	8.0.1	~1550	~750 MB	+~12%
17.1.0	17	8.1.1	~1900	~680 MB	+~12%

That is ~19 large mappings/decoder on PyAV 16 → ~110/decoder on PyAV 17 (~5.5×).

Decisive comparison: 16.1.0 vs 17.0.1 — identical bundled FFmpeg (8.0.1),
large-mapping count jumps ~270 → ~1550. The only variable is PyAV 16 → 17.

Ruled out

• FFmpeg version — 8.0.1 is fine under PyAV 16, regressed under PyAV 17;
  8.0 → 8.1 makes no difference.
• Frame threading — thread_count=1; no per-thread DPB/picture multiplication.
• numpy / opencv — unchanged (numpy 2.4.6, opencv-python-headless 4.13.0.92)
  across all rows.
• Allocator fragmentation — jemalloc tuning (narenas:2, oversize_threshold
  down to 128 KB, eager dirty/muzzy_decay) does not reclaim it. The extra
  mappings behave as distinct live allocations, not reclaimable dirty/muzzy
  pages → it is an allocation-count increase from libav/PyAV, not allocator behavior.
• Single-decoder isolation looks normal — the effect only emerges at N
  concurrent decoders; one decoder in isolation is comparable across versions.
  Suggests per-decoder buffer pools that no longer share/reuse, fragmenting
  the arena at scale.

Suspected root: the v17 zero-copy buffer-ownership change

PyAV 17.0.0 introduced a zero-copy refactor that changed buffer/memory ownership:

• no-copy VideoFrame.to_ndarray() for padded frames (by @lgeiger)
• zero-copy Packet init from buffer data (by @WyattBlue)

This same change is the documented root cause of #2223 "Writing packets to a
data stream broken in av>=17" —
where a packet ends up referencing memory whose ownership/lifetime changed
(absent in v16). Our memory regression appears to be another manifestation in
the decode path: if returned arrays/packets now hold references to
decoder-owned buffers, the decoder's buffer pool cannot recycle and allocates
fresh buffers each frame → the ~5× large-mapping growth observed.

Honest caveat: our pipeline calls to_ndarray(format="bgr24"), which converts
YUV→BGR and therefore cannot be strictly zero-copy — so the @lgeiger
padded-frame optimization may not hit us directly. But the broader v17
buffer-ownership refactor that broke #2223's packet path is the same suspect.

Unaddressed on main (verified)

main is 28 commits ahead of v17.1.0 (verified via the compare API). None
touches buffer-pool reuse, frame-buffer retention, refcounting, or a
memory/leak fix in the decode path. The only to_ndarray change adds a new
format (yuv420p10le); the two main commits mentioning "zero-copy" are new
DLPack export/import features (extending the zero-copy approach, not reverting
it); "Keep decoded frames when a packet ends with undecodable bytes" retains
more frames. So the regression is live on both the latest release (17.1.0)
and main, and #2223's zero-copy root is also still unfixed there.

Open question

Whether the regression is (a) PyAV 17's Cython frame/buffer code, or (b)
the av-17 wheel compiling its bundled FFmpeg differently from the av-16 wheel.
Clean split: build PyAV 17 from source against a self-built FFmpeg (same
version) and compare to the wheel — if source-built is fine, it's the wheel's
FFmpeg build; if still regressed, it's the Cython layer.

How the regression was established

The figures above were measured in a production deployment via jemalloc / /proc/<pid>/smaps (large-anonymous-mapping counts), by changing only the av wheel pin and holding the rest of the stack constant. The decisive single-variable comparison is av==16.1.0 vs av==17.0.1 — identical bundled FFmpeg (8.0.1), ~270 vs ~1550 large mappings/process.

I do not yet have a confirmed minimal standalone reproduction. In particular it is unverified whether the increase requires the decoded frames/arrays to be retained downstream (as the full pipeline does) to manifest, or whether N concurrent decoders alone is sufficient — earlier small-scale/single-decoder isolation did not show a difference. Happy to put together and confirm a minimal case if that would help triage.

Workaround

Pinning av~=16.1.0 (FFmpeg 8.0.1) avoids the regression. FFmpeg 8.0.1 still carries the fixes that motivated moving off 7.x — the HEVC output_corrupt/showall handling (commit bc1a3bfd2) and CVE-2025-1816 (both shipped in FFmpeg 8.0) — so it is a clean pin for our use.

Questions

1. Does the v17 zero-copy change (to_ndarray padded-frame / Packet buffer
   init) cause decoded-frame or packet buffers to be retained longer than v16,
   preventing decoder buffer-pool reuse?
2. Is there a way to opt back into copy semantics on the decode path without
   pinning to v16?
3. Is this on the radar alongside Writing packets to data stream broken in av>=17 #2223 (same zero-copy root), given it is
   unaddressed on main?

[WyattBlue]

Slop

[jacob-aegis]

@WyattBlue, sorry for the slop in the original post. I was able to pinpoint where this is coming from. PyAV 17.0 changed VideoReformatter._reformat from the classic sws_getCachedContext() + sws_scale() API to sws_alloc_context() + sws_scale_frame().

Under the bundled FFmpeg 8.x, sws_scale_frame() engages the new swscale implementation, which retains around 15 MB of internal buffers per SwsContext for the context's lifetime.

The effect of this became noticeable at scale on higher resolution RTSP streams. I hold many long-lived RTSP decode+convert contexts concurrently.

The following snippet reproduces the issue:

# 12 concurrent decoders over an H264 file; each converts frames via
# to_ndarray and retains the last 40 arrays; prints RSS + large-mapping count.
# Generate input: ffmpeg -f lavfi -i "testsrc2=size=1280x720:rate=5" -t 30 \
#   -c:v libx264 -pix_fmt yuv420p -g 10 sample.mp4
import os, sys, time, threading, collections
import av, numpy as np

N, K, SECS, SRC = 12, 40, 60, sys.argv[1]
MODE = os.environ.get("MODE", "bgr24")   # bgr24 | native | none
retained = [collections.deque(maxlen=K) for _ in range(N)]
stop = threading.Event()

def worker(i):
    with av.open(SRC) as c:
        vs = c.streams.video[0]
        vs.codec_context.thread_count = 1
        while not stop.is_set():
            for frame in c.decode(vs):
                if MODE == "bgr24":
                    retained[i].append(frame.to_ndarray(format="bgr24"))
                elif MODE == "native":
                    retained[i].append(frame.to_ndarray())
                if stop.is_set():
                    return
                time.sleep(0.01)
            c.seek(0)

def big_maps():
    n = 0
    with open("/proc/self/smaps") as f:
        anon = False
        for line in f:
            if line[:1].isalnum() and "-" in line.split()[0]:
                parts = line.split()
                anon = len(parts) == 5
            elif line.startswith("Size:") and anon:
                if int(line.split()[1]) >= 1024:
                    n += 1
    return n

def rss():
    with open("/proc/self/status") as f:
        for line in f:
            if line.startswith("VmRSS"):
                return int(line.split()[1]) // 1024

ts = [threading.Thread(target=worker, args=(i,), daemon=True) for i in range(N)]
[t.start() for t in ts]
t0 = time.time()
while time.time() - t0 < SECS:
    time.sleep(15)
    print(f"t={int(time.time()-t0)}s av={av.__version__} mode={MODE} "
          f"rss_mb={rss()} big_anon_maps={big_maps()}", flush=True)
stop.set()

Example execution would be MODE=bgr24 python repro.py sample.mp4 with av 16 then av 17. Av 16.1.0 settles at 98 large anonymous mappings while av 17.0.1 settles at 266 (+14 per decoder, 8–16 MB each). Decode without conversion is identical between versions, and building 17 from source against a distro FFmpeg 8.0.1 reproduces it, which points to the API change, not the wheel or the FFmpeg version.

FFmpeg 9 replaces the legacy swscale implementation entirely, so I wanted to flag this while you're adding 9 support.

Freeing or shrinking the graph state between calls, or sharing contexts across same-geometry conversions, would presumably address it.