fix(linux/pipewire): restore dummy buffer for software encoders and fix buffer ownership

[luanweslley77]

Description

the problem (software encoder + KWin/portal crashes at startup with "Couldn't scale frame"; and SIGABRT during teardown), the cause (regression from PR #5360, which removed the dummy_img allocation and added a destructor with delete[] despite lacking ownership), and what the PR does (restores the dummy buffer with data_owned; adds NV12 support to the software encoder; adds a unit test).

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

• feat: New feature (non-breaking change which adds functionality)
• fix: Bug fix (non-breaking change which fixes an issue)
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
• refactor: Code change that neither fixes a bug nor adds a feature
• perf: Code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit
• BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

AI Usage

See our AI usage policy.

• None: No AI tools were used in creating this PR
• Light: AI provided minor assistance (formatting, simple suggestions)
• Moderate: AI helped with code generation or debugging specific parts
• Heavy: AI generated most or all of the code changes

[luanweslley77]

Thanks for the review — and agreed on both the destructor ownership and the
fill_img() points. For the dummy image, I dug into what you linked and I think
I can show where the root cause actually is, because it ends up supporting your
preference to keep dummy_img() as a no-op.

The reason kmsgrab and wlgrab work with a no-op dummy_img() is not the dummy
itself — it's that their alloc_img() already hands out a buffer:

// kmsgrab.cpp:1702
img->data = new std::uint8_t[height * img->row_pitch];

// wlgrab.cpp:351
img->data = new std::uint8_t[height * img->row_pitch];

So when validate_config() runs alloc_img() → dummy_img() → convert(),
the image already has valid data. PipeWire's alloc_img() is the odd one out:

// pipewire.cpp:939
img->data = nullptr;

That's why the same no-op dummy breaks software encoding only on PipeWire
captures (kwin/portal): the software encoder's sws_scale_frame() gets a null
pointer and fails with EINVAL. The ValidateEncoder/software gtest reproduces
it — it fails on master and passes with the fix.

To keep dummy_img() a no-op everywhere (which I agree is the right thing for
maintenance), the fix would move into alloc_img() instead: allocate the
buffer there like kmsgrab/wlgrab, and have fill_img()'s memory path copy the
frame into that owned buffer instead of pointing img->data at the staging
vector. That removes the front_buffer aliasing entirely (no heap corruption
by construction), and there's no leak (the buffer is owned by the image,
recycled through the pool, and freed by the destructor).

One trade-off to be transparent about: that means one extra memcpy per frame
in the memory-buffer path (the staging copy already exists in on_process).
For 1080p60 NV12 that's roughly 180 MB/s of additional copying — a couple
percent of CPU on software encoding. If that cost is acceptable, I'll rework
the PR along those lines. If you'd rather keep the allocation in
dummy_img() as it is now (which doesn't add per-frame copies and doesn't
leak), the current version already addresses the crash on master.

---

[Kishi85]

Thanks for the review — and agreed on both the destructor ownership and the fill_img() points. For the dummy image, I dug into what you linked and I think I can show where the root cause actually is, because it ends up supporting your preference to keep dummy_img() as a no-op.

The reason kmsgrab and wlgrab work with a no-op dummy_img() is not the dummy itself — it's that their alloc_img() already hands out a buffer:

// kmsgrab.cpp:1702
img->data = new std::uint8_t[height * img->row_pitch];

// wlgrab.cpp:351
img->data = new std::uint8_t[height * img->row_pitch];

So when validate_config() runs alloc_img() → dummy_img() → convert(), the image already has valid data. PipeWire's alloc_img() is the odd one out:

// pipewire.cpp:939
img->data = nullptr;

That's why the same no-op dummy breaks software encoding only on PipeWire captures (kwin/portal): the software encoder's sws_scale_frame() gets a null pointer and fails with EINVAL. The ValidateEncoder/software gtest reproduces it — it fails on master and passes with the fix.

To keep dummy_img() a no-op everywhere (which I agree is the right thing for maintenance), the fix would move into alloc_img() instead: allocate the buffer there like kmsgrab/wlgrab, and have fill_img()'s memory path copy the frame into that owned buffer instead of pointing img->data at the staging vector. That removes the front_buffer aliasing entirely (no heap corruption by construction), and there's no leak (the buffer is owned by the image, recycled through the pool, and freed by the destructor).

One trade-off to be transparent about: that means one extra memcpy per frame in the memory-buffer path (the staging copy already exists in on_process). For 1080p60 NV12 that's roughly 180 MB/s of additional copying — a couple percent of CPU on software encoding. If that cost is acceptable, I'll rework the PR along those lines. If you'd rather keep the allocation in dummy_img() as it is now (which doesn't add per-frame copies and doesn't leak), the current version already addresses the crash on master.

Thanks for clarifying the differences. I wasn't aware that both kmsgrab and wlgrab handle the alloc earlier already.

In that case I'm good with your PR changes on pipewire.cpp the way they are right now. No need to rework that especially when that would add unnecessary overhead.

Also feel free to mark my earlier comments as resolved (for some reason github won't allow me to do that myself due to missing permissions).

P.S.: I'll apply this PR to my personal setup build and do some testing

UPDATE: No issues on my system running this so far.

[Kishi85]

@ReenigneArcher This is (at least according to my testing) good to merge and will restore using software encoding for pipewire-based methods

[codecov]

Bundle Report

Bundle size has no change ✅

[ReenigneArcher]

Looks like it's segfaulting during unit tests.

[codecov]

Codecov Report

❌ Patch coverage is 76.81159% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.45%. Comparing base (c8763b6) to head (8ec3d60).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/platform/linux/pipewire.cpp	0.00%	16 Missing ⚠️
src/video.cpp	86.66%	16 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5495      +/-   ##
==========================================
+ Coverage   31.59%   32.45%   +0.85%     
==========================================
  Files         102      104       +2     
  Lines       24621    24968     +347     
  Branches    10827    11084     +257     
==========================================
+ Hits         7780     8103     +323     
+ Misses      15783    13934    -1849     
- Partials     1058     2931    +1873     

Flag	Coverage Δ
Archlinux	0.00% <0.00%> (ø)
FreeBSD-amd64	18.28% <64.49%> (+0.11%)	⬆️
Homebrew-macos-14	28.12% <62.16%> (+0.12%)	⬆️
Homebrew-macos-15	28.83% <71.17%> (+0.12%)	⬆️
Homebrew-macos-26	28.95% <71.17%> (+0.12%)	⬆️
Homebrew-ubuntu-24.04	18.95% <60.33%> (+0.11%)	⬆️
Linux-AppImage	18.37% <60.33%> (+0.11%)	⬆️
Windows-AMD64	23.76% <67.61%> (?)
Windows-ARM64	20.50% <72.72%> (+0.16%)	⬆️
macOS-arm64	25.38% <63.63%> (+0.15%)	⬆️
macOS-x86_64	26.02% <63.63%> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/video.h	66.40% <100.00%> (+2.91%)	⬆️
src/platform/linux/pipewire.cpp	0.16% <0.00%> (-0.01%)	⬇️
src/video.cpp	54.37% <86.66%> (+0.60%)	⬆️

... and 56 files with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c8763b6...8ec3d60. Read the comment docs.

[github-actions]

Build artifacts

Artifacts from workflow run 32147892152:

• build-Archlinux
• build-Docker-debian-trixie
• build-Docker-ubuntu-22.04
• build-Docker-ubuntu-24.04
• build-Docker-ubuntu-26.04
• build-FreeBSD-amd64
• build-Homebrew
• build-Linux-AppImage
• build-Linux-Flatpak-aarch64
• build-Linux-Flatpak-x86_64
• build-macOS-arm64
• build-macOS-x86_64
• build-Windows-AMD64
• build-Windows-ARM64

_{You must be signed in to GitHub to download workflow artifacts.}

[github-actions]

Last Updated	2026-08-18 15:58:40 UTC
Source Run	CI Run #5017
Commit	8ec3d60e27f661c1c4fa4f47ff52fe893ea647d7

Screenshot Comparison

PR #5495 screenshots vs screenshots baseline.

Matrix: AppImage

Image	Baseline	PR
sunshine_tray_initial.png
sunshine_tray_menu.png
sunshine_tray_pairing_request.png
sunshine_tray_paused.png
sunshine_tray_stopped.png
sunshine_tray_streaming.png
sunshine_tray_tooltip.png

Matrix: Windows-AMD64

Image	Baseline	PR
sunshine_tray_initial.png
sunshine_tray_menu.png
sunshine_tray_pairing_request.png
sunshine_tray_paused.png
sunshine_tray_stopped.png
sunshine_tray_streaming.png
sunshine_tray_tooltip.png

Matrix: Windows-ARM64

Image	Baseline	PR
sunshine_tray_initial.png
sunshine_tray_menu.png
sunshine_tray_pairing_request.png
sunshine_tray_paused.png
sunshine_tray_stopped.png
sunshine_tray_streaming.png
sunshine_tray_tooltip.png

Matrix: macOS-arm64

Image	Baseline	PR
sunshine_tray_initial.png
sunshine_tray_menu.png
sunshine_tray_pairing_request.png
sunshine_tray_paused.png
sunshine_tray_stopped.png
sunshine_tray_streaming.png
sunshine_tray_tooltip.png

Matrix: macOS-x86_64

Image	Baseline	PR
sunshine_tray_initial.png
sunshine_tray_menu.png
sunshine_tray_pairing_request.png
sunshine_tray_paused.png
sunshine_tray_stopped.png
sunshine_tray_streaming.png
sunshine_tray_tooltip.png

[luanweslley77]

On the unit test segfault in CI:

I asked my agent to analyze it, and it found something that seems to explain what's happening: a possible double-free in the new test. set_frame() takes ownership of the frame (per its docs), but the test also calls av_frame_free(&frame) — which would mean the frame gets freed twice. The failing run (c9e11a8) does point at SoftwareEncoderConversion.Bgr0AndNv12 on every platform (Linux: Aborted (core dumped); Windows: exit 127), and that would also explain the coverage jobs/codecov failing downstream. So this seems like a likely explanation, but we're still validating it.

As for why I didn't see this locally: the test never crashed on my machine, probably because glibc's tcache hides double-frees (the freed chunk stays reusable), while the CI allocators detect them and abort. It's reproducible locally with GLIBC_TUNABLES=glibc.malloc.tcache_count=0 → SIGABRT, which matches.

What I'm considering: not freeing the frame in the test (the device owns it), and using the real class from video.h instead of the mirror declaration in the test (I think it was an ODR violation — apply_colorspace was missing). While testing I also noticed the row_pitch == width NV12 detection could break with padded strides, and that reinit_sws() may drop the colorspace set by apply_colorspace()

@ReenigneArcher I'm confident everything is in order now. Could you please trigger the CI pipeline once more?

[ReenigneArcher]

• https://github.com/LizardByte/Sunshine/actions/runs/32138214340/job/95715577783?pr=5495#step:11:1669
• https://github.com/LizardByte/Sunshine/actions/runs/32138214340/job/95715578012?pr=5495#step:12:2397

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud