fix(linux/pipewire): restore dummy buffer for software encoders and fix buffer ownership by luanweslley77 · Pull Request #5495 · LizardByte/Sunshine

luanweslley77 · 2026-08-11T03:22:23Z

Description

the problem (software encoder + KWin/portal crashes at startup with "Couldn't scale frame"; and SIGABRT during teardown), the cause (regression from PR #5360, which removed the dummy_img allocation and added a destructor with delete[] despite lacking ownership), and what the PR does (restores the dummy buffer with data_owned; adds NV12 support to the software encoder; adds a unit test).

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

Checklist

Code follows the style guidelines of this project
Code has been self-reviewed
Code has been commented, particularly in hard-to-understand areas
Code docstring/documentation-blocks for new or existing methods/components have been added or updated
Unit tests have been added or updated for any new or modified functionality

AI Usage

See our AI usage policy.

None: No AI tools were used in creating this PR
Light: AI provided minor assistance (formatting, simple suggestions)
Moderate: AI helped with code generation or debugging specific parts
Heavy: AI generated most or all of the code changes

luanweslley77 · 2026-08-11T12:36:59Z

Thanks for the review — and agreed on both the destructor ownership and the
fill_img() points. For the dummy image, I dug into what you linked and I think
I can show where the root cause actually is, because it ends up supporting your
preference to keep dummy_img() as a no-op.

The reason kmsgrab and wlgrab work with a no-op dummy_img() is not the dummy
itself — it's that their alloc_img() already hands out a buffer:

// kmsgrab.cpp:1702
img->data = new std::uint8_t[height * img->row_pitch];

// wlgrab.cpp:351
img->data = new std::uint8_t[height * img->row_pitch];

So when validate_config() runs alloc_img() → dummy_img() → convert(),
the image already has valid data. PipeWire's alloc_img() is the odd one out:

// pipewire.cpp:939
img->data = nullptr;

That's why the same no-op dummy breaks software encoding only on PipeWire
captures (kwin/portal): the software encoder's sws_scale_frame() gets a null
pointer and fails with EINVAL. The ValidateEncoder/software gtest reproduces
it — it fails on master and passes with the fix.

To keep dummy_img() a no-op everywhere (which I agree is the right thing for
maintenance), the fix would move into alloc_img() instead: allocate the
buffer there like kmsgrab/wlgrab, and have fill_img()'s memory path copy the
frame into that owned buffer instead of pointing img->data at the staging
vector. That removes the front_buffer aliasing entirely (no heap corruption
by construction), and there's no leak (the buffer is owned by the image,
recycled through the pool, and freed by the destructor).

One trade-off to be transparent about: that means one extra memcpy per frame
in the memory-buffer path (the staging copy already exists in on_process).
For 1080p60 NV12 that's roughly 180 MB/s of additional copying — a couple
percent of CPU on software encoding. If that cost is acceptable, I'll rework
the PR along those lines. If you'd rather keep the allocation in
dummy_img() as it is now (which doesn't add per-frame copies and doesn't
leak), the current version already addresses the crash on master.

Kishi85 · 2026-08-11T13:22:26Z

Thanks for the review — and agreed on both the destructor ownership and the fill_img() points. For the dummy image, I dug into what you linked and I think I can show where the root cause actually is, because it ends up supporting your preference to keep dummy_img() as a no-op.

The reason kmsgrab and wlgrab work with a no-op dummy_img() is not the dummy itself — it's that their alloc_img() already hands out a buffer:
// kmsgrab.cpp:1702
img->data = new std::uint8_t[height * img->row_pitch];

// wlgrab.cpp:351
img->data = new std::uint8_t[height * img->row_pitch];
So when validate_config() runs alloc_img() → dummy_img() → convert(), the image already has valid data. PipeWire's alloc_img() is the odd one out:
// pipewire.cpp:939
img->data = nullptr;
That's why the same no-op dummy breaks software encoding only on PipeWire captures (kwin/portal): the software encoder's sws_scale_frame() gets a null pointer and fails with EINVAL. The ValidateEncoder/software gtest reproduces it — it fails on master and passes with the fix.

To keep dummy_img() a no-op everywhere (which I agree is the right thing for maintenance), the fix would move into alloc_img() instead: allocate the buffer there like kmsgrab/wlgrab, and have fill_img()'s memory path copy the frame into that owned buffer instead of pointing img->data at the staging vector. That removes the front_buffer aliasing entirely (no heap corruption by construction), and there's no leak (the buffer is owned by the image, recycled through the pool, and freed by the destructor).

One trade-off to be transparent about: that means one extra memcpy per frame in the memory-buffer path (the staging copy already exists in on_process). For 1080p60 NV12 that's roughly 180 MB/s of additional copying — a couple percent of CPU on software encoding. If that cost is acceptable, I'll rework the PR along those lines. If you'd rather keep the allocation in dummy_img() as it is now (which doesn't add per-frame copies and doesn't leak), the current version already addresses the crash on master.

Thanks for clarifying the differences. I wasn't aware that both kmsgrab and wlgrab handle the alloc earlier already.

In that case I'm good with your PR changes on pipewire.cpp the way they are right now. No need to rework that especially when that would add unnecessary overhead.

Also feel free to mark my earlier comments as resolved (for some reason github won't allow me to do that myself due to missing permissions).

P.S.: I'll apply this PR to my personal setup build and do some testing

UPDATE: No issues on my system running this so far.

Kishi85 · 2026-08-15T07:42:21Z

@ReenigneArcher This is (at least according to my testing) good to merge and will restore using software encoding for pipewire-based methods

codecov · 2026-08-15T13:40:23Z

Bundle Report

Bundle size has no change ✅

ReenigneArcher · 2026-08-15T17:42:49Z

Looks like it's segfaulting during unit tests.

codecov · 2026-08-15T19:52:54Z

Codecov Report

❌ Patch coverage is 76.81159% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.45%. Comparing base (c8763b6) to head (8ec3d60).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/platform/linux/pipewire.cpp	0.00%	16 Missing ⚠️
src/video.cpp	86.66%	16 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5495      +/-   ##
==========================================
+ Coverage   31.59%   32.45%   +0.85%     
==========================================
  Files         102      104       +2     
  Lines       24621    24968     +347     
  Branches    10827    11084     +257     
==========================================
+ Hits         7780     8103     +323     
+ Misses      15783    13934    -1849     
- Partials     1058     2931    +1873

Flag	Coverage Δ
Archlinux	`0.00% <0.00%> (ø)`
FreeBSD-amd64	`18.28% <64.49%> (+0.11%)`	⬆️
Homebrew-macos-14	`28.12% <62.16%> (+0.12%)`	⬆️
Homebrew-macos-15	`28.83% <71.17%> (+0.12%)`	⬆️
Homebrew-macos-26	`28.95% <71.17%> (+0.12%)`	⬆️
Homebrew-ubuntu-24.04	`18.95% <60.33%> (+0.11%)`	⬆️
Linux-AppImage	`18.37% <60.33%> (+0.11%)`	⬆️
Windows-AMD64	`23.76% <67.61%> (?)`
Windows-ARM64	`20.50% <72.72%> (+0.16%)`	⬆️
macOS-arm64	`25.38% <63.63%> (+0.15%)`	⬆️
macOS-x86_64	`26.02% <63.63%> (+0.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/video.h	`66.40% <100.00%> (+2.91%)`	⬆️
src/platform/linux/pipewire.cpp	`0.16% <0.00%> (-0.01%)`	⬇️
src/video.cpp	`54.37% <86.66%> (+0.60%)`	⬆️

... and 56 files with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c8763b6...8ec3d60. Read the comment docs.

github-actions · 2026-08-15T19:53:07Z

Build artifacts

Artifacts from workflow run 32147892152:

_{You must be signed in to GitHub to download workflow artifacts.}

github-actions · 2026-08-15T19:53:12Z


Last Updated	2026-08-18 15:58:40 UTC
Source Run	CI Run #5017
Commit	`8ec3d60e27f661c1c4fa4f47ff52fe893ea647d7`

Screenshot Comparison

PR #5495 screenshots vs screenshots baseline.

Matrix: `AppImage`

Image	Baseline	PR
`sunshine_tray_initial.png`
`sunshine_tray_menu.png`
`sunshine_tray_pairing_request.png`
`sunshine_tray_paused.png`
`sunshine_tray_stopped.png`
`sunshine_tray_streaming.png`
`sunshine_tray_tooltip.png`

Matrix: `Windows-AMD64`

Image	Baseline	PR
`sunshine_tray_initial.png`
`sunshine_tray_menu.png`
`sunshine_tray_pairing_request.png`
`sunshine_tray_paused.png`
`sunshine_tray_stopped.png`
`sunshine_tray_streaming.png`
`sunshine_tray_tooltip.png`

Matrix: `Windows-ARM64`

Image	Baseline	PR
`sunshine_tray_initial.png`
`sunshine_tray_menu.png`
`sunshine_tray_pairing_request.png`
`sunshine_tray_paused.png`
`sunshine_tray_stopped.png`
`sunshine_tray_streaming.png`
`sunshine_tray_tooltip.png`

Matrix: `macOS-arm64`

Image	Baseline	PR
`sunshine_tray_initial.png`
`sunshine_tray_menu.png`
`sunshine_tray_pairing_request.png`
`sunshine_tray_paused.png`
`sunshine_tray_stopped.png`
`sunshine_tray_streaming.png`
`sunshine_tray_tooltip.png`

Matrix: `macOS-x86_64`

Image	Baseline	PR
`sunshine_tray_initial.png`
`sunshine_tray_menu.png`
`sunshine_tray_pairing_request.png`
`sunshine_tray_paused.png`
`sunshine_tray_stopped.png`
`sunshine_tray_streaming.png`
`sunshine_tray_tooltip.png`

luanweslley77 · 2026-08-15T20:47:37Z

On the unit test segfault in CI:

I asked my agent to analyze it, and it found something that seems to explain what's happening: a possible double-free in the new test. set_frame() takes ownership of the frame (per its docs), but the test also calls av_frame_free(&frame) — which would mean the frame gets freed twice. The failing run (c9e11a8) does point at SoftwareEncoderConversion.Bgr0AndNv12 on every platform (Linux: Aborted (core dumped); Windows: exit 127), and that would also explain the coverage jobs/codecov failing downstream. So this seems like a likely explanation, but we're still validating it.

As for why I didn't see this locally: the test never crashed on my machine, probably because glibc's tcache hides double-frees (the freed chunk stays reusable), while the CI allocators detect them and abort. It's reproducible locally with GLIBC_TUNABLES=glibc.malloc.tcache_count=0 → SIGABRT, which matches.

What I'm considering: not freeing the frame in the test (the device owns it), and using the real class from video.h instead of the mirror declaration in the test (I think it was an ODR violation — apply_colorspace was missing). While testing I also noticed the row_pitch == width NV12 detection could break with padded strides, and that reinit_sws() may drop the colorspace set by apply_colorspace()

@ReenigneArcher I'm confident everything is in order now. Could you please trigger the CI pipeline once more?

ReenigneArcher · 2026-08-18T13:28:32Z

…e encoder The software encoder assumed every captured frame is BGR0 (4 bytes/pixel, single plane). KWin screencast / XDG portal PipeWire captures deliver NV12 (1 byte/pixel row pitch, two planes). Detect the real format from img.row_pitch and recreate the sws context once when needed.

…ership dummy_img() returned an image with data == nullptr, which makes sws_scale_frame fail with EINVAL in the software encoder during startup validation ('Couldn't scale frame' -> 'Unable to find display or encoder'). KMS capture works because its alloc_img() allocates a real buffer; allocate a black new[] buffer here, marked as owned. Also fix buffer ownership in img_descriptor_t: the memory-buffer capture path points img->data at the PipeWire staging vector (front_buffer), owned by pipewire_t. The destructor freed it with delete[], corrupting the heap (SIGABRT) once the software-encoder path is reachable in real streams. data_owned now tracks whether the image owns its buffer.

…frames Validates the source-format detection in avcodec_software_encode_device_t: BGR0 (4 bytes per pixel, KMS/DMABUF layout) and NV12 (1 byte per pixel row pitch, PipeWire captures such as KWin screencast / portal).

sonarqubecloud · 2026-08-18T14:35:08Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

luanweslley77 force-pushed the fix/5430-pipewire-software-encoder branch from e9388b2 to c70c430 Compare August 11, 2026 03:35

Kishi85 reviewed Aug 11, 2026

View reviewed changes

Comment thread src/platform/linux/pipewire.cpp

Kishi85 reviewed Aug 11, 2026

View reviewed changes

Comment thread src/platform/linux/pipewire.cpp

Kishi85 reviewed Aug 11, 2026

View reviewed changes

Comment thread src/platform/linux/pipewire.cpp

ReenigneArcher force-pushed the fix/5430-pipewire-software-encoder branch from c70c430 to c9e11a8 Compare August 15, 2026 13:20

luanweslley77 force-pushed the fix/5430-pipewire-software-encoder branch 4 times, most recently from 957b086 to 670578e Compare August 16, 2026 16:46

ReenigneArcher force-pushed the fix/5430-pipewire-software-encoder branch from 670578e to ec06b9e Compare August 18, 2026 12:41

luanweslley77 added 3 commits August 18, 2026 11:21

luanweslley77 force-pushed the fix/5430-pipewire-software-encoder branch from ec06b9e to 8ec3d60 Compare August 18, 2026 14:22

ReenigneArcher added this to the pipewire milestone Aug 18, 2026

ReenigneArcher merged commit 6ad5f97 into LizardByte:master Aug 18, 2026
103 of 107 checks passed

luanweslley77 mentioned this pull request Aug 28, 2026

feat(video,platform): declare explicit capture pixel formats #5568

Open

21 tasks

This was referenced Sep 7, 2026

chore: Update LizardByte/Sunshine to v2026.906.222525 LizardByte/pacman-repo#99

Merged

chore: Add blog post for Sunshine release v2026.906.222525 LizardByte/LizardByte.github.io#446

Merged

LizardByte-bot mentioned this pull request Sep 7, 2026

chore: Update dev.lizardbyte.app.Sunshine to v2026.906.222525 flathub/dev.lizardbyte.app.Sunshine#195

Merged

Navigation Menu

Uh oh!

fix(linux/pipewire): restore dummy buffer for software encoders and fix buffer ownership - #5495

fix(linux/pipewire): restore dummy buffer for software encoders and fix buffer ownership#5495
ReenigneArcher merged 3 commits into
LizardByte:masterfrom
luanweslley77:fix/5430-pipewire-software-encoder

luanweslley77 commented Aug 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

luanweslley77 commented Aug 11, 2026 •

edited

Loading

Uh oh!

Kishi85 commented Aug 11, 2026 •

edited

Loading

Uh oh!

Kishi85 commented Aug 15, 2026

Uh oh!

codecov Bot commented Aug 15, 2026 •

edited

Loading

Uh oh!

ReenigneArcher commented Aug 15, 2026

Uh oh!

codecov Bot commented Aug 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 15, 2026 •

edited

Loading

Uh oh!

luanweslley77 commented Aug 15, 2026 •

edited

Loading

Uh oh!

ReenigneArcher commented Aug 18, 2026

Uh oh!

sonarqubecloud Bot commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

luanweslley77 commented Aug 11, 2026

Description

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

Checklist

AI Usage

Uh oh!

Uh oh!

Uh oh!

Uh oh!

luanweslley77 commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kishi85 commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kishi85 commented Aug 15, 2026

Uh oh!

codecov Bot commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle Report

Uh oh!

ReenigneArcher commented Aug 15, 2026

Uh oh!

codecov Bot commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build artifacts

Uh oh!

github-actions Bot commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Screenshot Comparison

Matrix: AppImage

Matrix: Windows-AMD64

Matrix: Windows-ARM64

Matrix: macOS-arm64

Matrix: macOS-x86_64

Uh oh!

luanweslley77 commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ReenigneArcher commented Aug 18, 2026

Uh oh!

sonarqubecloud Bot commented Aug 18, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

luanweslley77 commented Aug 11, 2026 •

edited

Loading

Kishi85 commented Aug 11, 2026 •

edited

Loading

codecov Bot commented Aug 15, 2026 •

edited

Loading

codecov Bot commented Aug 15, 2026 •

edited

Loading

github-actions Bot commented Aug 15, 2026 •

edited

Loading

github-actions Bot commented Aug 15, 2026 •

edited

Loading

Matrix: `AppImage`

Matrix: `Windows-AMD64`

Matrix: `Windows-ARM64`

Matrix: `macOS-arm64`

Matrix: `macOS-x86_64`

luanweslley77 commented Aug 15, 2026 •

edited

Loading