Tools: Configuring GPU Pass-Through for AMD cards on openSUSE Kalpa - Guide
Prerequisites
Configuring the host
Verify the host environment
Enable IOMMU
Configure VFIO and isolate the GPU used for pass-through
Load the VFIO driver
Including the driver in the initrd file
Configuring the guest
Requirements for the guest configuration Verify that the host support VT-d technology and that it is already enabled in the firmware settings: IOMMU is disabled by default. You need to enable it at boot time in the /etc/kernel/cmdline configuration file. 1) For Intel CPU add this line to file: 2) For AMD CPU add this line instead: 3) Finally, when you cat the file, you will see content like this: 1) Find the card vendor and model IDs: 2) Create the file /etc/modprobe.d/vfio.conf with the following content: 1) Create the file /etc/dracut.conf.d/gpu-passthrough.conf and add the following content (mind the leading whitespace): 2) Re-generate the initrd file: 3) Reboot the host machine For most of the changes in the above steps to take effect, you need to reboot the host machine: 4) Verify IOMMU is Active After your Kalpa host boots back up, open a terminal and check the kernel logs to confirm that the hardware isolation is working: This section describes how to configure the guest virtual machine so that it can use the host's GPU. This article was inspired by Configuring GPU Pass-Through for NVIDIA cards Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuseCommandCopy# dmesg | grep -e "Directed I/O"
[ 0.283295] [ T1] DMAR: Intel(R) Virtualization Technology for Directed I/O
# dmesg | grep -e "Directed I/O"
[ 0.283295] [ T1] DMAR: Intel(R) Virtualization Technology for Directed I/O
# dmesg | grep -e "Directed I/O"
[ 0.283295] [ T1] DMAR: Intel(R) Virtualization Technology for Directed I/O
intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
amd_iommu=on iommu=pt rd.driver.pre=vfio-pci
amd_iommu=on iommu=pt rd.driver.pre=vfio-pci
amd_iommu=on iommu=pt rd.driver.pre=vfio-pci
# cat /etc/kernel/cmdline root=/dev/nvme0n1p2 splash=silent swapaccount=1 systemd.show_status=1 mitigations=auto quiet security=selinux selinux=1 intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
# cat /etc/kernel/cmdline root=/dev/nvme0n1p2 splash=silent swapaccount=1 systemd.show_status=1 mitigations=auto quiet security=selinux selinux=1 intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
# cat /etc/kernel/cmdline root=/dev/nvme0n1p2 splash=silent swapaccount=1 systemd.show_status=1 mitigations=auto quiet security=selinux selinux=1 intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
# lspci -nn | grep -i "AMD"
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] [1002:7480] (rev cf)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
# lspci -nn | grep -i "AMD"
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] [1002:7480] (rev cf)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
# lspci -nn | grep -i "AMD"
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] [1002:7480] (rev cf)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
options vfio-pci ids=1002:7480,1002:ab30
options vfio-pci ids=1002:7480,1002:ab30
options vfio-pci ids=1002:7480,1002:ab30
add_drivers+=" vfio vfio_iommu_type1 vfio_pci "
add_drivers+=" vfio vfio_iommu_type1 vfio_pci "
add_drivers+=" vfio vfio_iommu_type1 vfio_pci "
# sdbootutil list-snapshots
1 first root filesystem
2 Snapshot Update of #1
3 Snapshot Update of #2
4 Snapshot Update of #2
5 Snapshot Update of #4
6 Snapshot Update of #5
7 Snapshot Update of #4 # sdbootutil list-kernels 7
ok /lib/modules/6.19.5-2-default/vmlinuz -> opensuse-microos-6.19.5-2-default-7.conf # sdbootutil -weight: 500;">update-all-entries 7
# sdbootutil mkinitrd 7
# sdbootutil list-snapshots
1 first root filesystem
2 Snapshot Update of #1
3 Snapshot Update of #2
4 Snapshot Update of #2
5 Snapshot Update of #4
6 Snapshot Update of #5
7 Snapshot Update of #4 # sdbootutil list-kernels 7
ok /lib/modules/6.19.5-2-default/vmlinuz -> opensuse-microos-6.19.5-2-default-7.conf # sdbootutil -weight: 500;">update-all-entries 7
# sdbootutil mkinitrd 7
# sdbootutil list-snapshots
1 first root filesystem
2 Snapshot Update of #1
3 Snapshot Update of #2
4 Snapshot Update of #2
5 Snapshot Update of #4
6 Snapshot Update of #5
7 Snapshot Update of #4 # sdbootutil list-kernels 7
ok /lib/modules/6.19.5-2-default/vmlinuz -> opensuse-microos-6.19.5-2-default-7.conf # sdbootutil -weight: 500;">update-all-entries 7
# sdbootutil mkinitrd 7
dmesg | grep -i "Adding to iommu group"
[ 0.283068] [ T1] pci 0000:00:00.0: Adding to iommu group 0
[ 0.283078] [ T1] pci 0000:00:01.0: Adding to iommu group 1
[ 0.283091] [ T1] pci 0000:00:14.0: Adding to iommu group 2
[ 0.283097] [ T1] pci 0000:00:14.2: Adding to iommu group 2
[ 0.283106] [ T1] pci 0000:00:15.0: Adding to iommu group 3
[ 0.283115] [ T1] pci 0000:00:16.0: Adding to iommu group 4
[ 0.283122] [ T1] pci 0000:00:17.0: Adding to iommu group 5
[ 0.283137] [ T1] pci 0000:00:1a.0: Adding to iommu group 6
[ 0.283147] [ T1] pci 0000:00:1c.0: Adding to iommu group 7
[ 0.283155] [ T1] pci 0000:00:1c.3: Adding to iommu group 8
[ 0.283163] [ T1] pci 0000:00:1c.4: Adding to iommu group 9
[ 0.283172] [ T1] pci 0000:00:1d.0: Adding to iommu group 10
[ 0.283188] [ T1] pci 0000:00:1f.0: Adding to iommu group 11
[ 0.283195] [ T1] pci 0000:00:1f.3: Adding to iommu group 11
[ 0.283201] [ T1] pci 0000:00:1f.4: Adding to iommu group 11
[ 0.283208] [ T1] pci 0000:00:1f.5: Adding to iommu group 11
[ 0.283216] [ T1] pci 0000:01:00.0: Adding to iommu group 12
[ 0.283224] [ T1] pci 0000:02:00.0: Adding to iommu group 13
[ 0.283235] [ T1] pci 0000:03:00.0: Adding to iommu group 14
[ 0.283245] [ T1] pci 0000:03:00.1: Adding to iommu group 15
[ 0.283260] [ T1] pci 0000:04:00.0: Adding to iommu group 16
[ 0.283268] [ T1] pci 0000:06:00.0: Adding to iommu group 17
[ 0.283275] [ T1] pci 0000:07:00.0: Adding to iommu group 18
dmesg | grep -i "Adding to iommu group"
[ 0.283068] [ T1] pci 0000:00:00.0: Adding to iommu group 0
[ 0.283078] [ T1] pci 0000:00:01.0: Adding to iommu group 1
[ 0.283091] [ T1] pci 0000:00:14.0: Adding to iommu group 2
[ 0.283097] [ T1] pci 0000:00:14.2: Adding to iommu group 2
[ 0.283106] [ T1] pci 0000:00:15.0: Adding to iommu group 3
[ 0.283115] [ T1] pci 0000:00:16.0: Adding to iommu group 4
[ 0.283122] [ T1] pci 0000:00:17.0: Adding to iommu group 5
[ 0.283137] [ T1] pci 0000:00:1a.0: Adding to iommu group 6
[ 0.283147] [ T1] pci 0000:00:1c.0: Adding to iommu group 7
[ 0.283155] [ T1] pci 0000:00:1c.3: Adding to iommu group 8
[ 0.283163] [ T1] pci 0000:00:1c.4: Adding to iommu group 9
[ 0.283172] [ T1] pci 0000:00:1d.0: Adding to iommu group 10
[ 0.283188] [ T1] pci 0000:00:1f.0: Adding to iommu group 11
[ 0.283195] [ T1] pci 0000:00:1f.3: Adding to iommu group 11
[ 0.283201] [ T1] pci 0000:00:1f.4: Adding to iommu group 11
[ 0.283208] [ T1] pci 0000:00:1f.5: Adding to iommu group 11
[ 0.283216] [ T1] pci 0000:01:00.0: Adding to iommu group 12
[ 0.283224] [ T1] pci 0000:02:00.0: Adding to iommu group 13
[ 0.283235] [ T1] pci 0000:03:00.0: Adding to iommu group 14
[ 0.283245] [ T1] pci 0000:03:00.1: Adding to iommu group 15
[ 0.283260] [ T1] pci 0000:04:00.0: Adding to iommu group 16
[ 0.283268] [ T1] pci 0000:06:00.0: Adding to iommu group 17
[ 0.283275] [ T1] pci 0000:07:00.0: Adding to iommu group 18
dmesg | grep -i "Adding to iommu group"
[ 0.283068] [ T1] pci 0000:00:00.0: Adding to iommu group 0
[ 0.283078] [ T1] pci 0000:00:01.0: Adding to iommu group 1
[ 0.283091] [ T1] pci 0000:00:14.0: Adding to iommu group 2
[ 0.283097] [ T1] pci 0000:00:14.2: Adding to iommu group 2
[ 0.283106] [ T1] pci 0000:00:15.0: Adding to iommu group 3
[ 0.283115] [ T1] pci 0000:00:16.0: Adding to iommu group 4
[ 0.283122] [ T1] pci 0000:00:17.0: Adding to iommu group 5
[ 0.283137] [ T1] pci 0000:00:1a.0: Adding to iommu group 6
[ 0.283147] [ T1] pci 0000:00:1c.0: Adding to iommu group 7
[ 0.283155] [ T1] pci 0000:00:1c.3: Adding to iommu group 8
[ 0.283163] [ T1] pci 0000:00:1c.4: Adding to iommu group 9
[ 0.283172] [ T1] pci 0000:00:1d.0: Adding to iommu group 10
[ 0.283188] [ T1] pci 0000:00:1f.0: Adding to iommu group 11
[ 0.283195] [ T1] pci 0000:00:1f.3: Adding to iommu group 11
[ 0.283201] [ T1] pci 0000:00:1f.4: Adding to iommu group 11
[ 0.283208] [ T1] pci 0000:00:1f.5: Adding to iommu group 11
[ 0.283216] [ T1] pci 0000:01:00.0: Adding to iommu group 12
[ 0.283224] [ T1] pci 0000:02:00.0: Adding to iommu group 13
[ 0.283235] [ T1] pci 0000:03:00.0: Adding to iommu group 14
[ 0.283245] [ T1] pci 0000:03:00.1: Adding to iommu group 15
[ 0.283260] [ T1] pci 0000:04:00.0: Adding to iommu group 16
[ 0.283268] [ T1] pci 0000:06:00.0: Adding to iommu group 17
[ 0.283275] [ T1] pci 0000:07:00.0: Adding to iommu group 18 - You need an additional display card on the host (eg. 1 igpu + 1 dedicate gpu or 2 dedicate gpu).
- GPU pass-through is supported on the AMD64/Intel 64 architecture only. - Use Q35 chipset if possible.
- Install the guest VM using UEFI firmware.
- Add the following emulated Graphic: Spice or VNC
- Add the following emulated Device: qxl, VGA or Virtio
- Add the host PCI device (03:00.0 in our example) to the guest.
- For the best performance, we recommend using virtio drivers for the network card and storage.
# dmesg | grep -e "Directed I/O"
[ 0.283295] [ T1] DMAR: Intel(R) Virtualization Technology for Directed I/O
# dmesg | grep -e "Directed I/O"
[ 0.283295] [ T1] DMAR: Intel(R) Virtualization Technology for Directed I/O
# dmesg | grep -e "Directed I/O"
[ 0.283295] [ T1] DMAR: Intel(R) Virtualization Technology for Directed I/O
intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
amd_iommu=on iommu=pt rd.driver.pre=vfio-pci
amd_iommu=on iommu=pt rd.driver.pre=vfio-pci
amd_iommu=on iommu=pt rd.driver.pre=vfio-pci
# cat /etc/kernel/cmdline root=/dev/nvme0n1p2 splash=silent swapaccount=1 systemd.show_status=1 mitigations=auto quiet security=selinux selinux=1 intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
# cat /etc/kernel/cmdline root=/dev/nvme0n1p2 splash=silent swapaccount=1 systemd.show_status=1 mitigations=auto quiet security=selinux selinux=1 intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
# cat /etc/kernel/cmdline root=/dev/nvme0n1p2 splash=silent swapaccount=1 systemd.show_status=1 mitigations=auto quiet security=selinux selinux=1 intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
# lspci -nn | grep -i "AMD"
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] [1002:7480] (rev cf)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
# lspci -nn | grep -i "AMD"
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] [1002:7480] (rev cf)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
# lspci -nn | grep -i "AMD"
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600] [1002:7480] (rev cf)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
options vfio-pci ids=1002:7480,1002:ab30
options vfio-pci ids=1002:7480,1002:ab30
options vfio-pci ids=1002:7480,1002:ab30
add_drivers+=" vfio vfio_iommu_type1 vfio_pci "
add_drivers+=" vfio vfio_iommu_type1 vfio_pci "
add_drivers+=" vfio vfio_iommu_type1 vfio_pci "
# sdbootutil list-snapshots
1 first root filesystem
2 Snapshot Update of #1
3 Snapshot Update of #2
4 Snapshot Update of #2
5 Snapshot Update of #4
6 Snapshot Update of #5
7 Snapshot Update of #4 # sdbootutil list-kernels 7
ok /lib/modules/6.19.5-2-default/vmlinuz -> opensuse-microos-6.19.5-2-default-7.conf # sdbootutil -weight: 500;">update-all-entries 7
# sdbootutil mkinitrd 7
# sdbootutil list-snapshots
1 first root filesystem
2 Snapshot Update of #1
3 Snapshot Update of #2
4 Snapshot Update of #2
5 Snapshot Update of #4
6 Snapshot Update of #5
7 Snapshot Update of #4 # sdbootutil list-kernels 7
ok /lib/modules/6.19.5-2-default/vmlinuz -> opensuse-microos-6.19.5-2-default-7.conf # sdbootutil -weight: 500;">update-all-entries 7
# sdbootutil mkinitrd 7
# sdbootutil list-snapshots
1 first root filesystem
2 Snapshot Update of #1
3 Snapshot Update of #2
4 Snapshot Update of #2
5 Snapshot Update of #4
6 Snapshot Update of #5
7 Snapshot Update of #4 # sdbootutil list-kernels 7
ok /lib/modules/6.19.5-2-default/vmlinuz -> opensuse-microos-6.19.5-2-default-7.conf # sdbootutil -weight: 500;">update-all-entries 7
# sdbootutil mkinitrd 7
dmesg | grep -i "Adding to iommu group"
[ 0.283068] [ T1] pci 0000:00:00.0: Adding to iommu group 0
[ 0.283078] [ T1] pci 0000:00:01.0: Adding to iommu group 1
[ 0.283091] [ T1] pci 0000:00:14.0: Adding to iommu group 2
[ 0.283097] [ T1] pci 0000:00:14.2: Adding to iommu group 2
[ 0.283106] [ T1] pci 0000:00:15.0: Adding to iommu group 3
[ 0.283115] [ T1] pci 0000:00:16.0: Adding to iommu group 4
[ 0.283122] [ T1] pci 0000:00:17.0: Adding to iommu group 5
[ 0.283137] [ T1] pci 0000:00:1a.0: Adding to iommu group 6
[ 0.283147] [ T1] pci 0000:00:1c.0: Adding to iommu group 7
[ 0.283155] [ T1] pci 0000:00:1c.3: Adding to iommu group 8
[ 0.283163] [ T1] pci 0000:00:1c.4: Adding to iommu group 9
[ 0.283172] [ T1] pci 0000:00:1d.0: Adding to iommu group 10
[ 0.283188] [ T1] pci 0000:00:1f.0: Adding to iommu group 11
[ 0.283195] [ T1] pci 0000:00:1f.3: Adding to iommu group 11
[ 0.283201] [ T1] pci 0000:00:1f.4: Adding to iommu group 11
[ 0.283208] [ T1] pci 0000:00:1f.5: Adding to iommu group 11
[ 0.283216] [ T1] pci 0000:01:00.0: Adding to iommu group 12
[ 0.283224] [ T1] pci 0000:02:00.0: Adding to iommu group 13
[ 0.283235] [ T1] pci 0000:03:00.0: Adding to iommu group 14
[ 0.283245] [ T1] pci 0000:03:00.1: Adding to iommu group 15
[ 0.283260] [ T1] pci 0000:04:00.0: Adding to iommu group 16
[ 0.283268] [ T1] pci 0000:06:00.0: Adding to iommu group 17
[ 0.283275] [ T1] pci 0000:07:00.0: Adding to iommu group 18
dmesg | grep -i "Adding to iommu group"
[ 0.283068] [ T1] pci 0000:00:00.0: Adding to iommu group 0
[ 0.283078] [ T1] pci 0000:00:01.0: Adding to iommu group 1
[ 0.283091] [ T1] pci 0000:00:14.0: Adding to iommu group 2
[ 0.283097] [ T1] pci 0000:00:14.2: Adding to iommu group 2
[ 0.283106] [ T1] pci 0000:00:15.0: Adding to iommu group 3
[ 0.283115] [ T1] pci 0000:00:16.0: Adding to iommu group 4
[ 0.283122] [ T1] pci 0000:00:17.0: Adding to iommu group 5
[ 0.283137] [ T1] pci 0000:00:1a.0: Adding to iommu group 6
[ 0.283147] [ T1] pci 0000:00:1c.0: Adding to iommu group 7
[ 0.283155] [ T1] pci 0000:00:1c.3: Adding to iommu group 8
[ 0.283163] [ T1] pci 0000:00:1c.4: Adding to iommu group 9
[ 0.283172] [ T1] pci 0000:00:1d.0: Adding to iommu group 10
[ 0.283188] [ T1] pci 0000:00:1f.0: Adding to iommu group 11
[ 0.283195] [ T1] pci 0000:00:1f.3: Adding to iommu group 11
[ 0.283201] [ T1] pci 0000:00:1f.4: Adding to iommu group 11
[ 0.283208] [ T1] pci 0000:00:1f.5: Adding to iommu group 11
[ 0.283216] [ T1] pci 0000:01:00.0: Adding to iommu group 12
[ 0.283224] [ T1] pci 0000:02:00.0: Adding to iommu group 13
[ 0.283235] [ T1] pci 0000:03:00.0: Adding to iommu group 14
[ 0.283245] [ T1] pci 0000:03:00.1: Adding to iommu group 15
[ 0.283260] [ T1] pci 0000:04:00.0: Adding to iommu group 16
[ 0.283268] [ T1] pci 0000:06:00.0: Adding to iommu group 17
[ 0.283275] [ T1] pci 0000:07:00.0: Adding to iommu group 18
dmesg | grep -i "Adding to iommu group"
[ 0.283068] [ T1] pci 0000:00:00.0: Adding to iommu group 0
[ 0.283078] [ T1] pci 0000:00:01.0: Adding to iommu group 1
[ 0.283091] [ T1] pci 0000:00:14.0: Adding to iommu group 2
[ 0.283097] [ T1] pci 0000:00:14.2: Adding to iommu group 2
[ 0.283106] [ T1] pci 0000:00:15.0: Adding to iommu group 3
[ 0.283115] [ T1] pci 0000:00:16.0: Adding to iommu group 4
[ 0.283122] [ T1] pci 0000:00:17.0: Adding to iommu group 5
[ 0.283137] [ T1] pci 0000:00:1a.0: Adding to iommu group 6
[ 0.283147] [ T1] pci 0000:00:1c.0: Adding to iommu group 7
[ 0.283155] [ T1] pci 0000:00:1c.3: Adding to iommu group 8
[ 0.283163] [ T1] pci 0000:00:1c.4: Adding to iommu group 9
[ 0.283172] [ T1] pci 0000:00:1d.0: Adding to iommu group 10
[ 0.283188] [ T1] pci 0000:00:1f.0: Adding to iommu group 11
[ 0.283195] [ T1] pci 0000:00:1f.3: Adding to iommu group 11
[ 0.283201] [ T1] pci 0000:00:1f.4: Adding to iommu group 11
[ 0.283208] [ T1] pci 0000:00:1f.5: Adding to iommu group 11
[ 0.283216] [ T1] pci 0000:01:00.0: Adding to iommu group 12
[ 0.283224] [ T1] pci 0000:02:00.0: Adding to iommu group 13
[ 0.283235] [ T1] pci 0000:03:00.0: Adding to iommu group 14
[ 0.283245] [ T1] pci 0000:03:00.1: Adding to iommu group 15
[ 0.283260] [ T1] pci 0000:04:00.0: Adding to iommu group 16
[ 0.283268] [ T1] pci 0000:06:00.0: Adding to iommu group 17
[ 0.283275] [ T1] pci 0000:07:00.0: Adding to iommu group 18 - You need an additional display card on the host (eg. 1 igpu + 1 dedicate gpu or 2 dedicate gpu).
- GPU pass-through is supported on the AMD64/Intel 64 architecture only. - Use Q35 chipset if possible.
- Install the guest VM using UEFI firmware.
- Add the following emulated Graphic: Spice or VNC
- Add the following emulated Device: qxl, VGA or Virtio
- Add the host PCI device (03:00.0 in our example) to the guest.
- For the best performance, we recommend using virtio drivers for the network card and storage.