Add Custom Output (FFmpeg) mode for alpha channel recording

[IsAvaible]

Add Custom Output (FFmpeg) mode for alpha channel / transparency recording

Closes #162
Closes #151

Overview

Currently, obs-source-record routes recordings through the standard OBS encoder pipeline (obs_encoder_t + ffmpeg_muxer). Standard OBS video encoders are strictly limited to opaque pixel formats (such as NV12 and I420) and drop any alpha transparency before frames reach the muxer. Consequently, recording sources with transparent backgrounds (e.g., green screen chroma keying, overlays, VTuber avatars, stinger transitions) into codecs like ProRes 4444 or QuickTime RLE loses the alpha channel.

This PR introduces a Custom Output (FFmpeg) mode for recording, routing the source frames directly through OBS's built-in ffmpeg_output (the same backend used by OBS Studio's native Settings -> Output -> Custom Output (FFmpeg)). Because ffmpeg_output manages its own libavcodec encode and muxing, it directly preserves alpha channels and supports transparent pixel formats (such as yuva444p10le, argb, etc.).

---

Key Changes

1. Custom Output (FFmpeg) Recording Pipeline:

   • Added an optional checkable Custom Output (FFmpeg) settings group in the filter properties.
   • When enabled, creates an ffmpeg_output output instance instead of ffmpeg_muxer.
   • Audio is routed via obs_output_set_media() and obs_output_set_mixers() respecting track selection (different_audio).
   • Container formats (mov, mkv, webm, mp4, etc.) are mapped to libavformat container identifiers.
   • For fragmented_mov and fragmented_mp4, fragmented muxer flags are automatically appended for crash-resilient recordings.

2. Alpha Channel Preservation:

   • Added an Alpha Channel toggle (ff_alpha).
   • When enabled, configures the filter render view format to VIDEO_FORMAT_BGRA.
   • ffmpeg_output's internal format selection detects the BGRA input and negotiates alpha pixel formats (e.g. yuva444p10le for ProRes 4444).
   • If alpha settings change during runtime, the source view dynamically recreates with the requested format and restarts cleanly.

3. Encoder Presets & Safe Defaults:

   • Added presets for transparent encoders:
     • Apple ProRes 4444 (prores): Fast Apple ProRes encoder with alpha (profile=4).
     • Apple ProRes 4444 KS (prores_ks): Costlier reference encoder.
     • QuickTime Animation RLE (qtrle): Lossless QuickTime animation with alpha.
     • FFV1 (ffv1): Lossless video in MKV.
     • VP9 (libvpx-vp9): WebM alpha video.
     • Also supports standard libavcodec encoders like libx264.
   • Strips unsupported manual pix_fmt options from options string (since ffmpeg_output dictates the pixel format).
   • Automatically caps default encoder thread count based on logical cores to prevent CPU oversubscription.
   • Only configures scale_width/scale_height when the user actually requests downscaling/upscaling, avoiding unnecessary swscale passes.

4. Crash Prevention & Async Stop Hardening:

   • Asynchronous Output Teardown: Stopping outputs is offloaded via background threads (stop_output_async) with reference counting and weak source tracking (obs_weak_source_t), eliminating UI lockups and race conditions when encoders take time to drain.
   • Encoder Re-use Protection: Checks whether existing encoders are still actively draining before starting or assigning new outputs, preventing use-after-free conditions.
   • Loop / Folder Spam Mitigation: Halts repetitive restart attempts after 10 consecutive failures to prevent creating hundreds of empty folders when a configuration is invalid.

---

Video

You can find a short video on how the feature works in practice here:
https://youtu.be/_agfOjppnxw

---

How to Test

1. Add a Chroma Key or Color Key filter to an image or camera source.
2. Add a Source Record filter below the chroma key in the filter list.
3. In Source Record settings:
   • Set Rec Format to mov.
   • Check Custom Output (FFmpeg).
   • Choose Apple ProRes 4444 (alpha) (prores) or QuickTime RLE (alpha) (qtrle).
   • Check Alpha Channel.
4. Record a clip and import into DaVinci Resolve, Premiere Pro, Final Cut Pro, or After Effects:
   • The background will be fully transparent without needing a chroma key or matte in post-production.

[IsAvaible]

Also probably fixes #99, #48, #73, #188 as I had to harden the pipeline to get this working.
With some small adjustments this new pipeline could also be used to support #187.