fix(video): avoid UAF crashes during video reinit (reproducible on Vulkan)

[psyke83]

Description

On Linux with the Vulkan encoder, reinit triggered by display switching or mode changes can crash Sunshine due to packets in flight after encoder teardown.

With extra debug logging, this pattern was occurring:

    Jun 26 23:57:21 archlinux sunshine[263621]: [2026-06-26 23:57:21.625]: Warning: [crash] encode_avcodec session=0x7fd8c89ad480 thread=140569594554048 avctx=0x7fd8c8c7f980 hw_frames=0x7fd8c883a940 FFmpeg frames ctx=0x7fd8c8a6b200
    Jun 26 23:57:21 archlinux sunshine[263621]: [2026-06-26 23:57:21.628]: Warning: PACKET DESTROY 0x7fd8c8b76d00
    Jun 26 23:57:21 archlinux sunshine[263621]: [2026-06-26 23:57:21.628]: Warning: [crash] SESSION DESTROY START 0x7fd8c89ad480
    Jun 26 23:57:21 archlinux sunshine[263621]: [2026-06-26 23:57:21.628]: Warning: PACKET DESTROY 0x7fd8e8bfd490
    Jun 26 23:57:21 archlinux sunshine[263621]: [2026-06-26 23:57:21.628]: Warning: [crash] Calling avcodec_ctx.reset()
    Jun 26 23:57:21 archlinux sunshine[263621]: [2026-06-26 23:57:21.628]: Warning: [crash] SESSION DESTROY AFTER RESET 0x7fd8c89ad480
    Jun 26 23:57:21 archlinux sunshine[263621]: [2026-06-26 23:57:21.628]: Warning: [crash] Calling device.reset()
    Jun 26 23:57:21 archlinux sunshine[263621]: [2026-06-26 23:57:21.629]: Warning: PACKET DESTROY 0x7fd8c882de40
    Jun 26 23:57:21 archlinux kernel: rtsp::handler[263877]: segfault at 7fd8a0af7e20 ip 00007fd8a0af7e20 sp 00007fd8ebffd638 error 14 likely on CPU 11 (core 3, socket 0)
    Jun 26 23:57:21 archlinux kernel: Code: Unable to access opcode bytes at 0x7fd8a0af7df6.
    Jun 26 23:57:21 archlinux systemd-coredump[263973]: Process 263621 (sunshine) of user 1000 terminated abnormally with signal 11/SEGV, processing...
    Jun 26 23:57:21 archlinux systemd[1]: Started Process Core Dump (PID 263973/UID 0).
    Jun 26 23:57:21 archlinux systemd[1]: Started Pass systemd-coredump journal entries to relevant user for potential DrKonqi handling.
    Jun 26 23:57:22 archlinux systemd-coredump[263974]: [🡕] Process 263621 (sunshine) of user 1000 dumped core.
    
                                                        Stack trace of thread 263877:
                                                        #0  0x00007fd8a0af7e20 n/a (libvulkan_radeon.so + 0xf7e20)
                                                        #1  0x00007fd8a0bcd8cf n/a (libvulkan_radeon.so + 0x1cd8cf)
                                                        #2  0x00005636d70c0ea4 ff_vk_unmap_buffers (sunshine + 0x45dea4)
                                                        #3  0x00005636d70c100b ff_vk_unmap_buffer (sunshine + 0x45e00b)
                                                        #4  0x00005636d70c102d free_data_buf (sunshine + 0x45e02d)
                                                        #5  0x00005636d7198a17 buffer_pool_flush (sunshine + 0x535a17)
                                                        #6  0x00005636d7198c26 buffer_replace (sunshine + 0x535c26)
                                                        #7  0x00005636d704596e av_packet_unref (sunshine + 0x3e296e)
                                                        #8  0x00005636d70459da av_packet_free (sunshine + 0x3e29da)
                                                        #9  0x00005636d6eb51a8 _ZNSt10unique_ptrIN5video12packet_raw_tESt14default_deleteIS1_EED2Ev.lto_priv.0 (sunshine + 0x2521a8)
                                                        #10 0x00005636d6ea6b69 _ZN6stream20videoBroadcastThreadERN5boost4asio21basic_datagram_socketINS1_2ip3udpENS1_15any_io_executorEEE (sunshine + 0x243b69)
                                                        #11 0x00005636d7f4f749 execute_native_thread_routine (sunshine + 0x12ec749)
                                                        #12 0x00007fd938a97739 n/a (libc.so.6 + 0x97739)
                                                        #13 0x00007fd938b1bedc n/a (libc.so.6 + 0x11bedc)

Since an in-flight packet's packet_raw_avcodec destructor was calling av_packet_free(..) after device.reset(), a UAF-type SEGFAULT would manifest. Moving the breakout check of the encoding loop to check immediately before encode() resolves the issue on my system via repeated stress testing. It may be possible that in-flight packets can still be active, which may necessitate future refcounting of packets in flight and waiting for it to drop to zero before breaking out of the encode loop, but for now, this minimal change fully resolves the issue on my system.

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

• feat: New feature (non-breaking change which adds functionality)
• fix: Bug fix (non-breaking change which fixes an issue)
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
• refactor: Code change that neither fixes a bug nor adds a feature
• perf: Code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit
• BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

AI Usage

• None: No AI tools were used in creating this PR
• Light: AI provided minor assistance (formatting, simple suggestions)
• Moderate: AI helped with code generation or debugging specific parts
• Heavy: AI generated most or all of the code changes

[psyke83]

Pinging @Kishi85; to be clear, I'm also stress testing with your #5257 applied on my local build to avoid the non-UAF type segfault that can also occur.

[psyke83]

@Kishi85 worth nothing: stress testing this PR brought to my attention a memory leak issue that occurs on reinit (display or mode change initiated) with either Vulkan or VAAPI. It affects master branch, so it's not caused by this PR or your #5257.

My mode change script can trigger the memory leak on my system. Each invocation (100 cycles) roughly doubles Sunshine's memory usage, and display mode switching may leak even faster. I'll investigate that later, but keep in mind that if it also happens for you, it may influence long-term/extended stress testing of this PR and #5257.

Edit: quick testing confirms that it's mostly an issue with pipewire capture, but kmsgrab may also have a slight leak (100 mode change cycles increases memory by ~10%).

[Kishi85]

Good catch on the UAF and your fix looks reasonable to me as it keeps it rather simple yet seems to cover the issue (and if it for some reason doesn't in all cases you've already mentioned a solution by adding packet refcounting).

A memory leak now becoming visible is not that surprising as it was masked by crashes from this UAF and the nullptr deref from #5257. I've tried to run sunshine (master + #5257 + this PR) with valgrind to dig into the leak but I've had no luck getting that to run so far.

EDIT: Forgot to say that I can also confirm the leak as you described. It's adding around the same size of sunshine's initial memory footprint per 100 iterations of your testscript (with timeout=0)

[Kishi85]

Here's a diff for pmap $(pidof sunshine) > a/b for exactly one capture re-init (via single display switch):

--- a   2026-06-27 08:18:29.614814986 +0200
+++ b   2026-06-27 08:18:43.653875790 +0200
@@ -9,2 +9,2 @@
-00007f2d50000000  31980K rw---   [ anon ]
-00007f2d51f3b000  33556K -----   [ anon ]
+00007f2d50000000  40744K rw---   [ anon ]
+00007f2d527ca000  24792K -----   [ anon ]
@@ -20 +20 @@
-00007f2d5f816000   8236K rw---   [ anon ]
+00007f2d5f02c000  16340K rw---   [ anon ]
@@ -48,2 +48,2 @@
-00007f2d98000000   1348K rw---   [ anon ]
-00007f2d98151000  64188K -----   [ anon ]
+00007f2d98000000   1352K rw---   [ anon ]
+00007f2d98152000  64184K -----   [ anon ]
@@ -1282 +1282 @@
- total          3317952K
+ total          3326056K

The increase seems to be around 2-15MB per full capture re-init for kwingrab/portalgrab and the value is fluctuating within that range.

If using KMS I cannot see a memory leak with pmap (total is not increasing as it is missing from the diff):

--- a   2026-06-27 08:24:04.236740409 +0200
+++ b   2026-06-27 08:24:22.173059455 +0200
@@ -39,2 +39,2 @@
-00007fca48000000  43772K rw---   [ anon ]
-00007fca4aabf000  21764K -----   [ anon ]
+00007fca48000000  51512K rw---   [ anon ]
+00007fca4b24e000  14024K -----   [ anon ]

So the memory leak should be somewhere within the code related to pipewire-based capture methods (pipewire.cpp/portalgrab.cpp/kwingrab.cpp) and yet another separate issue.

[codecov]

Bundle Report

Bundle size has no change ✅

[codecov]

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 27.53%. Comparing base (fcc4395) to head (64fcac3).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/video.cpp	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5346      +/-   ##
==========================================
+ Coverage   27.51%   27.53%   +0.01%     
==========================================
  Files         113      113              
  Lines       25593    25593              
  Branches    11237    11237              
==========================================
+ Hits         7043     7048       +5     
- Misses      14599    15513     +914     
+ Partials     3951     3032     -919     

Flag	Coverage Δ
Archlinux	0.00% <0.00%> (ø)
FreeBSD-amd64	13.31% <0.00%> (+<0.01%)	⬆️
Homebrew-macos-14	21.02% <0.00%> (ø)
Homebrew-macos-15	21.19% <0.00%> (+<0.01%)	⬆️
Homebrew-macos-26	21.30% <0.00%> (ø)
Homebrew-ubuntu-24.04	13.06% <0.00%> (ø)
Linux-AppImage	12.39% <0.00%> (ø)
Windows-AMD64	15.26% <0.00%> (ø)
Windows-ARM64	13.29% <0.00%> (ø)
macOS-arm64	19.25% <0.00%> (ø)
macOS-x86_64	18.71% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/video.cpp	52.68% <50.00%> (ø)

... and 30 files with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcc4395...64fcac3. Read the comment docs.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud