# Compute and GPU

## **PCI Passthrough**

## [**\#**](https://xcp-ng.org/docs/compute.html#_0-prerequisites)**0. Prerequisites**

**WARNING**

Ensure VT-d/IOMMU Support Is Enabled

In order to use PCI passthrough your host system must have VT-d/IOMMU functionality enabled. This should be more commonly enabled by default on enterprise hardware than on consumer hardware. It can be enabled in the BIOS/UEFI of systems with CPUs and chipsets which support it. For Intel platforms the feature is typically referred to as VT-d (Intel Virtualization Technology for Directed I/O); on AMD platforms it is typically listed as IOMMU or AMD-Vi. Please note that this is not the same as VT-x/AMD-v virtualisation support, and so these options are often listed separately.

Consult your system or motherboard manual for instructions on where to find the setting in your BIOS/UEFI. In addition, system BIOS updates may reset the feature to its default state, which may require you to re-enable it.

If you attempt to perform PCI passthrough on a system which does not have VT-d/IOMMU enabled, you may encounter the following error when you start the target virtual machine:

```plaintext
Internal error: xenopsd internal error: Device.PCI.Cannot_add(_, _)

```

<mark class="pen-red">**WARNING**</mark>

You may not be able to passthrough USB controllers

```plaintext
Internal error: xenopsd internal error: Cannot_add(0000:00:1d.0, Xenctrlext.Unix_error(30, "1: Operation not permitted"))

```

and an error in `/var/log/xen/hypervisor.log`

```plaintext
[2020-08-22 10:09:03] (XEN) [  297.542134] [VT-D] It's disallowed to assign 0000:08:00.0 with shared RMRR at 7ba77000 for Dom32753.
[2020-08-22 10:09:03] (XEN) [  297.542136] d[IO]: assign (0000:08:00.0) failed (-1)

```

This indicates that your device is using [RMRR (opens new window)](https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf). Intel [IOMMU does not allow DMA to these devices (opens new window)](https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt)and therefore PCI passthrough is not supported.

### [**\#**](https://xcp-ng.org/docs/compute.html#_1-find-your-devices-id-b-d-f-on-the-pci-bus-using-one-of-the-following-methods)**1. Find your devices ID (**[**B/D/F (opens new window)**](https://en.wikipedia.org/wiki/PCI_configuration_space#BDF)**) on the PCI bus using one of the following methods:**

**Method 1: List PCI Devices with** `<strong>lspci</strong>`

*This method is the easiest way to find devices.*

```plaintext
[root@xen ~]# lspci
...
04:01.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)

```

**Method 2: List System Device Classes with** `<strong>find</strong>`

*This method works best for finding the device ID by class. The example below the class is* `<em>net</em>` *and shows how to find the device ID of a specific network interface.*

```plaintext
[root@xen ~]# find /sys/class/net -exec readlink {} +
../../devices/virtual/net/lo
../../devices/pci0000:00/0000:04:01.0/net/eth1

```

### [**\#**](https://xcp-ng.org/docs/compute.html#_2-tell-xcp-ng-not-to-use-this-device-id-for-dom0)**2. Tell XCP-ng not to use this device ID for Dom0**

Add the `<strong>xen-pciback.hide</strong>` parameter to the kernel boot parameters:

```plaintext
/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:04:01.0)"

```

> You can hide multiple devices. If you wanted to add another device at `00:19.0` just append it to the parameter.
> 
> `/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:04:01.0)(0000:00:19.0)"`

To remove any passthrough devices from dom0:

```plaintext
/opt/xensource/libexec/xen-cmdline --delete-dom0 xen-pciback.hide

```

> **TIP**
> 
> This kernel parameter is not retained when you upgrade an XCP-ng host [using the installation ISO](https://xcp-ng.org/docs/upgrade.html#upgrade-via-installation-iso-recommended). Remember to re-do this step after the upgrade.

### [**\#**](https://xcp-ng.org/docs/compute.html#_3-reboot-the-xcp-ng-host)**3. Reboot the XCP-ng host**

`[root@xen ~]# reboot`

### [**\#**](https://xcp-ng.org/docs/compute.html#_4-check-with-xl-pci-assignable-list-on-cli)**4. Check with** `<strong>xl pci-assignable-list</strong>` **on CLI**

```plaintext
[root@xen ~]# xl pci-assignable-list
0000:04:01.0

```

### [**\#**](https://xcp-ng.org/docs/compute.html#_5-put-this-pci-device-into-your-vm)**5. Put this PCI device 'into' your VM**

`[root@xen ~]# xe vm-param-set<strong>other-config:pci=0/0000:04:01.0</strong>uuid=<vm uuid>`

> You can also pass through multiple devices. If you wanted to pass through another device at `00:19.0` just append it to the parameter.
> 
> `[root@xen ~]# xe vm-param-set<strong>other-config:pci=0/0000:04:01.0,0/0000:00:19.0</strong>uuid=<vm uuid>`

### [**\#**](https://xcp-ng.org/docs/compute.html#_6-start-your-vm-and-be-happy)**6. Start your VM and be happy 😃**

`[root@xen ~]# xe vm-start uuid=<vm uuid>`

## [**\#**](https://xcp-ng.org/docs/compute.html#gpu-passthrough)**GPU Passthrough**

To passthrough a complete graphics card to a VM (not virtualize it into multiple virtual vGPUs, which is different, see the vGPU section below), just follow the regular PCI passthrough instructions, no special steps are needed. Most Nvidia and AMD video cards should work without issue.

> **TIP**
> 
> Previously, Nvidia would block the use of gaming/consumer video cards for passthrough (the Nvidia installer would throw an **Error 43** when installing the driver inside your VM). They lifted this restriction in 2021 with driver R465 and above, so be sure to use the latest driver. [Details from Nvidia here.(opens new window)](https://nvidia.custhelp.com/app/answers/detail/a_id/5173/)

## [**\#**](https://xcp-ng.org/docs/compute.html#vgpu)**vGPU**

### [**\#**](https://xcp-ng.org/docs/compute.html#nvidia-vgpu)**NVIDIA vGPU**

**WARNING**

Due to a proprietary piece of code in XenServer, XCP-ng doesn't have (yet) support for NVIDIA vGPUs.

### [**\#**](https://xcp-ng.org/docs/compute.html#mxgpu-amd-vgpu)**MxGPU (AMD vGPU)**

AMD GPU are trivial using industry standard.  
Version 2.0 of the mxgpu iso should work on any 8.X version of XCP-ng

1. Enable SR-IOV in the server's BIOS
2. Install XCP-ng
3. Download Citrix XenServer from AMD's Drivers &amp; Support page. (Currently version 2.0.0 for XenServer 8.1)
4. Copy the `mxgpu-2.0.0.amd.iso` to the host
5. Install the supplemental pack:

```plaintext
cd /tmp
xe-install-supplemental-pack mxgpu-2.0.0.amd.iso
```

1. Reboot the XCP-ng
2. Assign an MxGPU to the VM from the VM properties page. Go to the GPU section. From the Drop down choose how big of a slice of the GPU you want on the VM and click OK

Start the VM and log into the guest OS and load the appropriate guest driver from AMD's Drivers &amp; Support page.

> Known working cards:

- S7150x2

## [**\#**](https://xcp-ng.org/docs/compute.html#usb-passthrough)**USB Passthrough**

**TIP**

There's no need to alter any files manually as some older guides suggest

It's fairly easy using the `xe` CLI. First use `xe pusb-list` to list all the physical USB devices on your host available for passthrough:

```plaintext
[root@xenserver ~]# xe pusb-list
uuid ( RO)            : 10fbec89-4472-c215-5d55-17969b473ee6
            path ( RO): 2-1.1
       vendor-id ( RO): 0781
     vendor-desc ( RO): SanDisk Corp.
      product-id ( RO): 5591
    product-desc ( RO):
          serial ( RO): 4C530001151223117134
         version ( RO): 2.10
     description ( RO): SanDisk Corp._4C530001151223117134

```

Find your USB device there, and note the `uuid`. Then use that uuid to enable passthrough for it:

```plaintext
[root@xenserver ~]# xe pusb-param-set uuid=10fbec89-4472-c215-5d55-17969b473ee6 passthrough-enabled=true

```

This will create a `usb-group` containing this USB device. We need to find the uuid of that group, so we use the `usb-group-list` command, specifying the physical USB uuid we got in step one:

```plaintext
[root@xenserver ~]# xe usb-group-list PUSB-uuids=10fbec89-4472-c215-5d55-17969b473ee6
uuid ( RO)                : 1f731f6a-6025-8858-904d-c98548f8bb23
name-label ( RW): Group of 0781 5591 USBs
name-description ( RW):

```

Note the uuid of this usb-group, then use it in the following command to attach this USB device to your desired VM. Remember to first shut down the target VM as hot-plug for USB passthrough is not supported:

```plaintext
xe vusb-create usb-group-uuid=<usb_group_uuid> vm-uuid=<vm_uuid>

```

So using the examples above, it would look like:

```plaintext
xe vusb-create usb-group-uuid=1f731f6a-6025-8858-904d-c98548f8bb23 vm-uuid=4feeb9b2-2176-b69d-b8a8-cf7289780a3f

```

Finally, start the target guest VM:

```plaintext
[root@xenserver ~]# xe vm-start uuid=<vm_uuid>

```

**Note:** If you get a message containing `internal error` when trying to start the VM after assigning it a USB device, try the following command to ensure its `platform:usb` parameter is set correctly:

```plaintext
xe vm-param-set uuid=<vm_uuid> platform:usb=True

```

In the future if you ever need to unplug the virtual USB device from your VM, or remove and unassign it completely, find the uuid of the virtual USB device by running `xe vusb-list`. Then use the uuid of the virtual USB device in one or both of the following commands:

```plaintext
xe vusb-unplug uuid=<vusb_uuid>
xe vusb-destroy uuid=<vusb_uuid>

```

## [**\#**](https://xcp-ng.org/docs/compute.html#passing-through-keyboards-and-mice)**Passing through Keyboards and Mice**

xcp-ng host uses usb-policy.conf at `/etc/xensource/usb-policy.conf` with ALLOW and DENY rules for different classes of usb devices. The default file contains Mice and Keyboards with DENY rules. You can edit this file to allow these devices (and any other ones similarly).

Once edited, run the following command to refresh:

```plaintext
/opt/xensource/libexec/usb_scan.py -d

```

Then run

```plaintext
xe pusb-scan host-uuid=<host_uuid>
```