Why does IOMMU needs to be OFF ?

[ArtificialEU]

NVIDIA Open GPU Kernel Modules Version

X

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

X

Kernel Release

X

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

NVIDIA RtX 4090

Describe the bug

Hi,

I'm using my GPU Cluster inside a proxmox VM and i'm passing through my devices, so IOMMU off is kinda not possible, what's the reason behind it and the consequences ?

Thanks

To Reproduce

X

Bug Incidence

Once

nvidia-bug-report.log.gz

X

More Info

No response

[legraphista]

Through my troubleshooting, I found out that P2P can be enabled with IOMMU on, but it may not work as intended.
When moving data between GPUs via P2P, I was getting IO_PAGE_FAULT and the copied tensor would be allocated but zeroed out.

Disabling IOMMU fixed this behavior in my case.

Note: my SR-IOV was disabled, your mileage may vary

[cunfate]

It's not neccessary to disable IOMMU for P2P driver.
Just set iommu=pt to avoid IOVA allocated for your GPU memory.

[erikboberg]

Hi!
I know this is an old thread, but I am trying to implement P2P on my proxmox VM with 2 RTX 3090:s passed through. I was not able to get it working yet, the testing with nvidia-smi topo -p2p r just shows NS, despite the install script running without errors. Can you give me some hints on how to get it working?