Linux: software encoder fails on GNOME portal/PipeWire BGRA (scale Invalid argument / imgutils ABRT) when no HW encode available

Is there an existing issue for this?

I have searched the existing issues

Related strings appear elsewhere but are not the same setup:

Error: Couldn't scale frame: Invalid argument #5309 / NvENC not loading: Error: Couldn't scale frame: Invalid argument #2827 — "Couldn't scale frame" often with NVENC / other capture stacks
This report is forced encoder = software (libx264) + capture = portal + GNOME Wayland virtual monitor (Meta-0 / headless remote) + GPU with VAAPI decode only (no encode)

Is your issue described in the documentation?

I have read the documentation

Docs cover portal capture and software encode; they do not document this failure path (portal BGRA → software scale fails, then fatal “no display/encoder” even though a monitor and PipeWire stream exist).

Is your issue present in the latest beta/pre-release?

This issue is present in the latest pre-release

Tested:

Nightly 0.0.5368 (commit: 4eb2db5f4884135ef593ec375e003523b019800d, debian-trixie-amd64)
Release 2026.516.143833 (commit: 14ffa6fdaa53f7b51512be2b3d24f3939695403c)

Describe the Bug

On Debian 13 (trixie) + GNOME Wayland, with a virtual/remote monitor Meta-0 1920×1080 (GNOME Remote Desktop headless-style session), Sunshine is configured with:

capture = portal
encoder = software (required: host GPU has no H.264 encode entrypoint)

Portal capture succeeds and PipeWire delivers format 12 (BGRA) @ 1920×1080, using memory buffers.

Software encode then fails:

Build	Behavior
Nightly 0.0.5368	During encoder probe: `Error: Couldn't scale frame: Invalid argument` → `Encoder [software] failed` → nvenc/vulkan/vaapi all fail → `Fatal: Unable to find display or encoder during startup` / `Video failed to find working encoder`
2026.516.143833	Encoder probe can pass; stream starts; Frame 1 IDR is produced; ~1s later process aborts on FFmpeg assert: `src_linesize … >= bytewidth failed at libavutil/imgutils.c:350` (`SIGABRT`). Client may see ~one frame then a dead session.

Misleading outcome: logs clearly show Wayland Meta-0, successful portal RemoteDesktop Start, and a PipeWire stream, yet the final fatal message blames a missing display/encoder.

system_tray = disabled is set here so Qt tray does not take down the process; tray is unrelated to the scale failure.

Expected Behavior

Portal BGRA frames should convert/scale cleanly into libx264, or
Fail with a clear, specific error (unsupported pixel format / stride / swscale), without:
- aborting on FFmpeg asserts, or
- claiming “no display” when Meta-0 + portal/PipeWire are already working

Steps To Reproduce

Debian 13, GNOME Wayland session with virtual monitor (e.g. GNOME Remote Desktop), output Meta-0 1920×1080.
GPU with no hardware H.264 encode (VAAPI VLD/decode only), so software encode is required.
Sunshine config essentials:

encoder = software
capture = portal
hevc_mode = 0
av1_mode = 0
stream_audio = disabled
system_tray = disabled
min_log_level = debug

Ensure XDG portal RemoteDesktop restore token / screen-share permission works (SelectSources/Start succeed; PipeWire node appears).
Start Sunshine; observe encoder probe logs (nightly fails at probe). Optionally connect Moonlight Desktop stream on 2026.516 to hit the later imgutils abort.

Host Operating System

Linux

Operating System Version

Debian GNU/Linux 13 (trixie), kernel 6.12.95+deb13-amd64

Architecture

amd64/x86_64

Package

Linux - .deb (LizardByte debian-trixie-amd64 / packaged 2026.516.143833 and nightly 0.0.5368)

GPU Type

AMD

GPU Model

AMD CAICOS PRO (Radeon HD 7450) — DRM radeon, Mesa Gallium

GPU Driver/Mesa Version

Mesa 25.0.7-2+deb13u1 (LLVM 19.1.7); libva 1.22 / 2.22.0

Capture Method

XDG Portal (PipeWire)

Apps

(default / Desktop stream; not app-specific)

Log output

Portal + PipeWire (common):

Screencasting with XDG portal
[wayland] Name: Meta-0
[wayland] Resolution: 1920x1080
[pipewire] Video format: 12
[pipewire] Size: 1920x1080
[pipewire] Framerate (from compositor): 0/1 (variable rate capture)
[pipewire] using memory buffers

Nightly 0.0.5368 (software probe):

Trying encoder [software]
Creating encoder [libx264]
[libx264] profile Constrained Baseline, level 4.2, 4:2:0, 8-bit
Error: Couldn't scale frame: Invalid argument
Creating encoder [libx264]
Error: Couldn't scale frame: Invalid argument
Encoder [software] failed
Error: Couldn't find any working encoder matching [software]
… nvenc / vulkan / vaapi fail …
Fatal: Unable to find display or encoder during startup.
Fatal: Please check that a display is connected and powered on.
Error: Video failed to find working encoder

2026.516.143833 (after stream start):

Creating encoder [libx264]
Frame 1: IDR Keyframe (AV_FRAME_FLAG_KEY)
New streaming session started [active sessions: 1]
Error: Assertion ((src_linesize) >= 0 ? (src_linesize) : (-(src_linesize))) >= bytewidth failed at libavutil/imgutils.c:350
# process SIGABRT (e.g. systemd status=6/ABRT)

VAAPI (expected on this GPU — decode only):

vaapi vendor: Mesa Gallium driver … for AMD CAICOS …
[h264_vaapi] No usable encoding entrypoint found for profile VAProfileH264High
Could not open codec [h264_vaapi]: Function not implemented
Encoder [vaapi] failed

Online logs

No response

Additional Context

Suspect portal/PipeWire BGRA memory buffer layout (stride/planes) vs FFmpeg sws / av_image_copy on the software path; imgutils assert matches src_linesize < bytewidth for the assumed format.
Headless Meta-* virtual outputs + GNOME portal matter for remote/server-style hosts; software encode is the only path when VAAPI has no encode entrypoint.
Happy to attach full debug logs, vainfo, and portal/PipeWire node details if useful.

   text                                                                        
   1 │ Trying encoder [software]                                               
   2 │ Creating encoder [libx264]                                              
   3 │ Error: Couldn't scale frame: Invalid argument                           
   4 │ Encoder [software] failed

Then nvenc / vulkan / vaapi all fail (expected here), and startup ends
with:

   text                                                                        
   1 │ Fatal: Unable to find display or encoder during startup.                
   2 │ Error: Video failed to find working encoder

So this pre-release does not fix the portal BGRA → software (libx264) scale still fails on my setup.

Same issue here, but with KWin Screencast instead of the portal. Fedora 44, KDE Plasma 6
Wayland, Sunshine 2026.808.164219, capture = kwin + encoder = software (GPU has no
hardware encode).

Startup dies with:

Info: Screencasting with KWin ScreenCast
Info: [pipewire] Video format: 12
Info: [pipewire] using memory buffers
Info: Creating encoder [libx264]
Error: Couldn't scale frame: Invalid argument
Info: Encoder [software] failed
Fatal: Unable to find display or encoder during startup.

I traced it to dummy_img() in src/platform/linux/pipewire.cpp. During startup
validation the encoder takes a dummy image and feeds it straight into sws_scale_frame(),
and since PR #5360 changed dummy_img() to just return 0 (no buffer), img->data is
null and sws bails with EINVAL. The PR's comment says it mirrors kmsgrab/wlgrab, but those
allocate in alloc_img() — the PipeWire one doesn't.

Same PR also added a ~img_descriptor_t() that delete[]s img->data unconditionally.
With the memory-buffer path (which is what PipeWire delivers to software encoders)
fill_img() points img->data at PipeWire's own staging buffer, so the destructor frees
memory it doesn't own — heap corruption and SIGABRT on session teardown once streaming
actually starts.

Locally I restored the dummy allocation and added a data_owned flag so the destructor
only frees what belongs to the image. With that, startup validation passes and sessions
tear down cleanly.

Navigation Menu

Repository navigation

Linux: software encoder fails on GNOME portal/PipeWire BGRA (scale Invalid argument / imgutils ABRT) when no HW encode available #5430

Is there an existing issue for this?

Is your issue described in the documentation?

Is your issue present in the latest beta/pre-release?

Describe the Bug

Expected Behavior

Steps To Reproduce

Host Operating System

Operating System Version

Architecture

Package

GPU Type

GPU Model

GPU Driver/Mesa Version

Capture Method

Apps

Log output

Online logs

Additional Context

Related

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Linux: software encoder fails on GNOME portal/PipeWire BGRA (scale Invalid argument / imgutils ABRT) when no HW encode available #5430

Description

Is there an existing issue for this?

Is your issue described in the documentation?

Is your issue present in the latest beta/pre-release?

Describe the Bug

Expected Behavior

Steps To Reproduce

Host Operating System

Operating System Version

Architecture

Package

GPU Type

GPU Model

GPU Driver/Mesa Version

Capture Method

Apps

Log output

Online logs

Additional Context

Related

Activity

ReenigneArcher commented on Jul 19, 2026

imzlh commented on Jul 19, 2026

ReenigneArcher commented on Jul 19, 2026

imzlh commented on Jul 19, 2026

luanweslley77 commented on Aug 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions