3.12: FormatReader deadlocks in deinit when released on the demux thread, so files stay open and drives can't be ejected (#188) #214

Summary

With 3.12, some files thumbnailed by Finder stay open forever in both QLVideo Formats and AudiovisualThumbnailExtension, so external drives can't be ejected ("dissented by AudiovisualThumbnailExtension"). This looks like the #188 symptom again, but the cause is new in 3.12: FormatReader.deinit can run on the demux thread, and then demuxer.join() waits for that same thread to exit.

The thumbnail itself is generated fine (0.3–3 s). The file handle leaks afterwards.

Environment

• QuickLook Video 3.12 (Homebrew cask quicklook-video)
• macOS 27.0.1 (26A434), Apple M1 Pro
• Files on exFAT USB drives

Reproduction

I called QLThumbnailGenerator.generateBestRepresentation after MTRegisterProfessionalVideoWorkflowFormatReaders()/VTRegisterProfessionalVideoWorkflowVideoDecoders(), which takes the same path as Finder, then ran lsof on the drive.

• Batch of 261 videos (MKV/WebM/AVI) on a USB drive, 8 concurrent requests: all 261 thumbnails were generated, but 1 file was still open in both processes afterwards and stayed open.
• That file (a 55 MB 1080p H.264 + Opus MKV) requested 10 times in a row at uncached sizes: 3 of the 10 requests leaked a handle.

Evidence

sample "QLVideo Formats" shows one thread per leaked file, all identical:

Thread  DispatchQueue: uk.org.marginal.qlvideo.formatreader  (serial)
  _dispatch_call_block_and_release
    ???  (in QLVideo Formats)  + 0x15608
      ???  (in QLVideo Formats)  + 0x1e920
        ???  (in QLVideo Formats)  + 0x11210
          ???  (in QLVideo Formats)  + 0x11144
            _dispatch_group_wait_slow
              _dlock_wait
                __ulock_wait

Mapped against the shipped 3.12 arm64 binary:

So when AVFoundation drops its last reference while demuxLoop is mid-iteration, the loop's local format becomes the last strong reference. Releasing it at the end of the iteration runs FormatReader.deinit on the demux thread. deinit then calls join(), which waits for demuxLoop to leave demuxGroup, but demuxLoop is further up the same stack. The thread blocks forever.

As a result, avformat_close_input() never runs and byteSource is never released. The host side (AudiovisualThumbnailExtension) therefore keeps the file open too. Killing QLVideo Formats alone doesn't release AudiovisualThumbnailExtension's handle; you have to kill both, as you noted in #188.

The join() call was added in 588c564 ("Wait for demux thread to exit before deinitting FormatReader"), first released in 3.12.

Proposed fix

When join() is called on demuxQueue itself, skip the wait. It's safe there: the release happens after withExtendedLifetime(format) { ... } has returned, so demuxLoop isn't inside av_read_frame and doesn't hold stateLock. stop() has just set stopping, so as soon as deinit returns the loop goes back to its top, sees stopping and exits without touching fmt_ctx again. PacketDemuxer itself stays alive until then because the async closure retains it.

diff --git a/formatreader/packetdemuxer.swift b/formatreader/packetdemuxer.swift
--- a/formatreader/packetdemuxer.swift
+++ b/formatreader/packetdemuxer.swift
@@ -195,6 +195,7 @@ final class PacketDemuxer {
     private let stateLock = NSLock()
     private let demuxGroup = DispatchGroup()  // prevent deinit until demuxLoop has exited
     private let demuxQueue = DispatchQueue(label: "uk.org.marginal.qlvideo.formatreader", qos: .default)
+    private let demuxQueueKey = DispatchSpecificKey<Void>()  // identifies code running on demuxQueue
     private let demuxSem = DispatchSemaphore(value: 0)  // wake demuxLoop when paused
 
     init(format: FormatReader) throws {
@@ -253,6 +254,7 @@ final class PacketDemuxer {
         guard ret >= 0 else {
             throw AVERROR(errorCode: ret, context: "avformat_seek_file(min)")  // If we can't seek to start we can't demux
         }
+        demuxQueue.setSpecific(key: demuxQueueKey, value: ())
         startDemuxLoop()
     }
 
@@ -294,6 +296,10 @@ final class PacketDemuxer {
     }
 
     func join() {
+        // If demuxLoop held the last reference to FormatReader then FormatReader.deinit runs here on demuxQueue.
+        // demuxLoop can't then be inside av_read_frame, and it exits as soon as deinit returns since we're stopping.
+        // So don't wait for it - that would deadlock and leak the file handle.
+        if DispatchQueue.getSpecific(key: demuxQueueKey) != nil { return }
         demuxGroup.wait()
     }

The patch applies cleanly to master (987a2c2).

Testing

I built rel-312 with and without the patch (Xcode 27.1, arm64). Then I ran the real FormatReader/PacketDemuxer code in-process with a stand-in MEByteSource subclass that reads the file with pread. Each iteration opens the file, calls generateSampleCursor(atPresentationTimeStamp: QLThumbnailTime), loads one sample buffer and drops every reference, much as the thumbnailer does. An iteration counts as leaked if the byte source is still alive 3 s later.

• I also ran an instrumented copy of the patched build 20 more times on the first file. It took the "deinit on demux thread" path 15 times and finished normally each time.
• I ran the patched runs with MallocScribble, MallocPreScribble and MallocGuardEdges, and there were no crashes.
• Slower reads make the leak more likely, which fits it showing up mostly on external drives.

I couldn't test the patched extension in Finder itself. AMFI refuses an ad-hoc-signed appex with com.apple.developer.mediaextension.formatreader (error -424, "adhoc signed but contains restricted entitlements"), and I don't have a team that can provision that entitlement.

Workaround until a fix ships

killall -9 "QLVideo Formats"; killall AudiovisualThumbnailExtension

Both get relaunched on demand, and after that the drive ejects.

Side note for triage

On macOS 27, qlmanage -t isn't a useful repro for these reports. It uses the legacy Movie.qlgenerator in-process, which doesn't opt in to Media Extensions, so it can't open MKV. It then hangs without exiting, for any file the generator fails on (even random bytes saved as .mov). So reports of "qlmanage times out on MKV" are probably unrelated to the plugin.