[RFC]: Surviving Thunderbolt eGPU surprise-removal: a hardware-validated open-tree patch series

[atassis]

Hi! I've been debugging Thunderbolt eGPU hot-removal on Wayland and ended up with a small,
hardware-validated patch series against 610.43.02. Per CONTRIBUTING.md I'm raising it here before
sending a PR, to check scope and how you'd want it structured.

The problem. Hot-removing a TB eGPU (an enclosure power brownout, or pulling the Type-C cable)
hangs the whole desktop for minutes, unkillable, hard-reset only. The driver keeps running waits and
teardown steps on a GPU that is already off the bus, while holding the RM API and GPU locks. Through
KMS those locks reach the compositor's main thread, so every GPU client lands in uninterruptible
D-state and the session freezes.

The approach. The driver already tracks this state (PDB_PROP_GPU_IS_LOST,
API_GPU_ATTACHED_SANITY_CHECK) and already short-circuits on it in places like kgspRpcSanityCheck().
The series just extends that to the wait and teardown sites that don't check it yet. Every guard is a
no-op when the GPU is present, so the normal path is unchanged, and Hopper (no Falcon Booter teardown)
is untouched by design.

The result. The same removal becomes a clean detach: the compositor migrates to the iGPU, the
kernel stays responsive, no hung_task, no Oops, no hard reset. Tested on a GA102 TB3 eGPU (Intel Titan
Ridge enclosure, AMD Strix Halo host, kernel 6.19) with both a PSU brownout and a cable unplug under
an active session.

Six commits, about +390/-7, off tag 610.43.02:

• Branch: https://github.com/atassis/open-gpu-kernel-modules/tree/egpu/device-lost-bail-upstream
• Compare: atassis/open-gpu-kernel-modules@610.43.02...egpu/device-lost-bail-upstream

The three hang sites, with file:line

1. Display notifier waits. nvEvoMakeRoom() and EvoCheckNotifier() in
   src/nvidia-modeset/src/nvkms-dma.c spin on an EVO FIFO GET pointer that never advances once the
   GPU is gone, holding nvkms_lock. Two variants: link-up engine-dead hits nvEvoMakeRoom, link-down
   removal hits EvoCheckNotifier.
2. GSP and Falcon teardown. On removal, kgspTeardown_TU102()
   (gsp/arch/turing/kernel_gsp_tu102.c) runs FWSEC-SB and Booter-Unload. Each s_dmaPoll_GA102()
   Falcon DMA poll (gsp/arch/ampere/kernel_gsp_falcon_ga102.c) waits out its full 4-second timeout,
   and 10 to 15 stack up. threadStateResetTimeout() at the top of teardown re-arms the deadline, so
   the existing per-wait lost detection never collapses them; they run to completion holding the RM API
   and GPU locks for minutes.
3. GC6 refcount and teardown accounting. RmGc6BlockerRefCntDec NULL-derefs a removed device,
   externalKernelClientCount underflows, and outputs are not shut down before
   drm_mode_config_cleanup (leaks DP-MST state).

For the link-up engine-dead case, where RM never detects loss because the BAR stays mapped, the nvkms
waiters use a "GET pointer has not advanced in 30 seconds" backstop rather than a wall-clock timeout,
so a slow but healthy GPU won't trip it.

The commits: nvkms EVO push-buffer bail (plus a pushbuffer-overrun guard from my own review); tolerate
external-client-count underflow on a lost GPU; honor drm_dev_unplug() in the atomic check path; shut
down outputs before mode-config cleanup; guard the GC6 blocker refcount dec against a removed device;
skip Falcon teardown ucodes when the GPU is lost.

What I'd like to know:

1. Scope. Is "don't wedge the system on GPU surprise-removal" something you'd want here, or do you
   treat it as out of scope?
2. Structure. If you'd take it, how would you like it shaped (target branch, any splitting or
   squashing, one series vs one PR per subsystem)?
3. One honest note. The GSP/Falcon teardown hang is in shared RM code, so it likely affects the
   proprietary driver too; it isn't open-specific. That's why I'm in Discussions rather than the bug
   form. Is discussion plus a PR the right path, or do you have an internal route you'd prefer?

Happy to share full backtraces and test logs. Thanks!

[charliejuc]

Thanks for this series — I'm hitting what looks like the same hang on a different platform, compute-only, and I have a way to trigger the surprise-removal on demand, so I can help test.

System

• Host: Lenovo ThinkPad P16s Gen 2 AMD (21K9), Ryzen 7 PRO 7840U (Phoenix), BIOS 1.51 (R2FET71W), EC 1.35
• eGPU: Gigabyte AORUS RTX 5060 Ti AI BOX (GB206, 16 GB), Thunderbolt 5 enclosure (Intel Barlow Ridge, 8086:5786) on the host's USB4 port
• Fedora 44, kernel 7.2.8-200.fc44, NVIDIA open kernel modules 615.71.09 (RPM Fusion akmod), Secure Boot on
• Compute only: nvidia + nvidia_uvm loaded; nvidia_modeset/nvidia_drm NOT loaded; NVreg_DynamicPowerManagement=0x00. Workload: llama.cpp (CUDA) server, idle at the time.

What happens
The USB4 link to the enclosure drops (platform/firmware side, see below). Within milliseconds the driver detects the loss, but then the system freezes and can only be recovered with a forced power-off:

12:05:33.3019 thunderbolt 0000:66:00.6: acking hot unplug event on 0:2
12:05:33.3028 pcieport 0000:00:04.1: pciehp: Slot(0): Link Down
12:05:33.3099 NVRM: Xid (PCI:0000:06:00): 79, GPU has fallen off the bus.
12:05:33.3101 NVRM: krcRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79.
...
12:05:33.3237 NVRM: GPU0 mmuWalkUnmap: Failed to unmap VA Range 0x51b700000 to 0x51b76ffff. Status = 0x00000040
12:05:33.3237 NVRM: GPU0 kgmmuInvalidateTlb_GM107: TLB invalidation failed waiting for prior invalidate (status=0x0000000f)
12:05:33.3237 NVRM: GPU0 virtmemDestruct_IMPL: VirtualMemory memmgrFree failed, client: c1e00007, hVASpace: 9, gpu: 0
12:05:33.3244 NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 2063!
12:05:33.3244 NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1e00007; hObject=0x00000009; status=0x0000000f
12:05:33.3245 NVRM: GPU0 _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
12:05:33.3245 NVRM: GPU0 scrubberDestruct:  Timed out when waiting for the scrub to complete the pending work .
12:05:37.4997 NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from _intrServiceStallCommonCheckBegin(pGpu, pIntr, &pOldContext) @ intr.c:1631
12:05:37.4997 NVRM: GPU0 _intrServiceStallCommonCheckBegin: Failed GPU reg read : 0xffffffff. Check whether GPU is present on the bus
[the last two lines repeat; nothing else is logged; the machine stays frozen]

Next boot: x86/amd: Previous system reset reason [0x00200800]: ACPI power state transition occurred (i.e. the forced power-off).

So it looks like the client teardown path (the CUDA process freeing its VA space and RM objects via GSP RPCs) keeps running against a GPU that is already gone. An earlier occurrence ended in a kernel panic (drm_panic QR) instead of a hang; I didn't capture its text.

Reproducer on demand
On this laptop, any connect/disconnect on the other USB-C port makes the USB4 link to the enclosure drop 20–45 s later (a platform firmware issue I'm reporting separately; it also happens without the NVIDIA driver loaded). That gives a reliable surprise-removal with the driver loaded and a CUDA client active, which should exercise exactly the paths your series guards.

Questions

1. Does the series cover the compute-only path (no nvkms), i.e. RM client/VA-space teardown and GSP RPCs with a UVM/CUDA client alive, or only the display and Falcon-teardown sites?
2. Would you be open to a rebase on 615.71.09? I can build and sign it locally and report results with this reproducer.

[atassis]

I am a bit busy with making a thorough read of your text (sorry), but still make another suggestion to decrease pcie version to older ones. I was able to achieve stable work on 5xxx gpu only with older pcie 2.0
Pcie 3.0 has some kind of pcie event storm or smth, which makes GSP to work unstable and break

[charliejuc]

Thanks for the suggestion! In my case Gen3 turned out to be fine: the instability was host-side, not GSP-side. Every drop started with a USB4 lane hot-unplug before any GPU error.

What fixed it was disabling the BIOS option "CPU Power Management" (ThinkPad, Config → Power). With it enabled I got a spontaneous drop within ~75 minutes; with it disabled, about three days stable on a Gen3 x4 link, under both idle and sustained load. Small sample, but consistent so far.

When the link does drop, the driver still hangs the system, so I'm still interested in the actual fix.