Proxmox VM Passthrough

PCIe Pass-Through

This is a quick outline of the configuration changes that must be made to allow PCIe pass through for VM's for proxmox out-of-box.

Enable IOMMU features by editing the default grub file, by adding the following to the "GRUB_CMDLINE_LINUX_DEFAULT" options in /etc/default/grub Note: On newer systems, your install may be using "proxmox-boot-tool". If your is then the following must be appended to /etc/kernel/cmdline

intel_iommu=on or for AMD amd_iommu=on

Then run update-grub if using grub or proxmox-boot-tool refresh to push this file change to the grub boot loader

Note: The config file on this install, of PVE 8.0 had a ' at the beginning that was causing the update-grub command to fail.

Now reboot the system and check that the system supports IOMMU physically, by checking for different IOMMU groups in the output. It should look something like the following:

pvesh get /nodes/proxymoxy2/hardware/pci --pci-class-blacklist ""

Note the values are different between devices, if there are some with the same it means they are on the same device and share a "pcie hub"

If you see -1 as the IOMMU value on all the devices it means IOMMU is not initialized.

If you have some devices sharing IOMMU groups, that should not try adding pcie_acs_override=downstream to the grub file aswell. This will breakup the IOMMU group's even more on some hardware.

If this is for GPU devices, make sure to blacklist the drivers for the device. Note this may cause issues if your integrated graphics use the same driver:

AMD:

echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
Nvidia:

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf 
echo "blacklist nvidia*" >> /etc/modprobe.d/blacklist.conf 
Intel:

echo "blacklist i915" >> /etc/modprobe.d/blacklist.conf

Now that these drivers are blacklisted on the host, reboot one more time, which universalizes the drivers. Now once back up, connect to the web GUI of the server, and create a new VM. From there go to hardware settings > top left to Add > PCI device > and select a device from the drop down list:

Here we can see the 1080ti installed in this IBM server

Now we have access to the PCI devices in use on the system. In this configuration with IOMMU, the device is elusive to the system. There are ways to bypass IOMMU and attempt the device anyways, but if the host system tries to access the device, it will cause a hard system crash, and is highly unstable.