PCI is a local computer bus for attaching hardware devices in a computer, and is the main peripheral bus on modern x86 systems. As such, having a proper way to emulate it is crucial for Xen to be able to expose both fully emulated devices or passthrough devices to guests. This talk will focus on the current status of PCI emulation in Xen, how and where it is used, what are its main limitations and future plans to improve it in order to be more robust and modular.

1. Status of PCI emulation in Xen
Roger Pau Monn´e
roger.pau@citrix.com
Chicago – July 9th, 2019

2. PCI bus PCI-passthroughon Xen Moving forward
PCI bus
Allows attaching hardware devices in a computer.
First specification developed by Intel in 1992.
Superseded VESA, MCA, EISA, NuBus...
Two standards:
PCI local bus.
PCI Express.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 2 / 20

3. PCI bus PCI-passthroughon Xen Moving forward
PCI slots on a motherboard
Obtained from wikipedia author snickerdo.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 3 / 20

4. PCI bus PCI-passthroughon Xen Moving forward
PCI card
Chicago – July 9th, 2019 Status of PCI emulation in Xen 4 / 20

5. PCI bus PCI-passthroughon Xen Moving forward
PCI configuration space
The PCI configuration space provides 256bytes or 4096bytes
of configuration space to each device.
Devices are identified by a 8bit bus, 5bit device and 3bit
function integers.
First 64bytes is standardized, the rest is device dependent
(contains capability structures).
Allows for easy discovery of devices. OS can scan the whole
bus in order to detect present devices.
On x86 can be accessed from IO space (legacy) or memory
(enhanced).
Chicago – July 9th, 2019 Status of PCI emulation in Xen 5 / 20

6. PCI bus PCI-passthroughon Xen Moving forward
Legacy PCI configuration access
Indirect access using the IO address space.
Address port at 0xcf8:
012781011151623243031
E RSV Bus Device Func Register RSV
Data port at 0xcfc.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 6 / 20

7. PCI bus PCI-passthroughon Xen Moving forward
Enhanced PCI configuration access
Maps the full config space registers into memory space.
0111214151920272831
Base Bus Device Func Register
If on 64bit mode bits from 63-32 also contain the base
address.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 7 / 20

8. PCI bus PCI-passthroughon Xen Moving forward
PCI header
0781516232431
Status Command 04h
Base Address 0 16h
Base Address 1 20h
Base Address 2 24h
Base Address 3 28h
Base Address 4 32h
Base Address 5 36h
CardBus CIS Pointer 40h
Subsystem ID Subsystem Vendor ID 44h
Expansion ROM Base Address 44h
Chicago – July 9th, 2019 Status of PCI emulation in Xen 8 / 20

9. PCI bus PCI-passthroughon Xen Moving forward
MSI capability
0781516232431
Message control Next pointer Capability ID 00h
Message Address [31, 0] 04h
Message Address [63, 32] 08h
Reserved Message Data 12h
Mask Bits 16h
Pending bits 20h
Chicago – July 9th, 2019 Status of PCI emulation in Xen 9 / 20

10. PCI bus PCI-passthroughon Xen Moving forward
MSI-X capability
0781516232431
Message control Next pointer Capability ID 00h
MSI-X Table Offset BIR 04h
PBA Offset BIR 08h
0
31
32
63
64
Vector Control Message Data
Upper Address Lower Address
Entry 0
...
...
Vector Control Message Data
Upper Address Lower Address
Entry N
Chicago – July 9th, 2019 Status of PCI emulation in Xen 10 / 20

11. PCI bus PCI-passthroughon Xen Moving forward
PCI handling in Xen
PV privileged domain (dom0) gets almost unlimited access to
the PCI config space:
Xen controls the MSI(-X) mask bits in order to keep a
coherent state when doing PCI-passthrough to HVM guests.
Read only access is allowed to the MSI-X table and the MSI
data and address registers.
Passthrough of PCI devices to unprivileged guests:
PV guests can access the PCI config space using a Xen PV
specific protocol (pciif).
HVM guests can access the PCI config space emulated by a
device model (QEMU).
PVH guests have no PCI-passthrough support yet.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 11 / 20

12. PCI bus PCI-passthroughon Xen Moving forward
PCI-passthrough for domUs
Hardware
Xen
Control Domain (VM0)
PV1 HVM1
user-space
kernel
QEMU
evtchn devpciback
Chicago – July 9th, 2019 Status of PCI emulation in Xen 12 / 20

13. PCI bus PCI-passthroughon Xen Moving forward
PCI-passthrough for domUs
PV domU communicates directly with pciback using a shared
memory ring and a Xen specific protocol.
Passthrough to HVM domUs is handled by QEMU, much like
emulated devices:
PCI config space accesses are forwarded by Xen to QEMU
using ioreqs.
QEMU emulates or forwards those accesses to the underlying
device.
Xen directly handles guest writes to the MSI-X mask bits for
performance reasons.
Device MMIO regions (BARs) are directly mapped to the
guest physmap, except for the MSI-X region if present.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 13 / 20

14. PCI bus PCI-passthroughon Xen Moving forward
PV dom0
Has almost unlimited read/write access to the configuration
space except for certain parts of the MSI(-X) capabilities.
Has to use hypercalls to deal with certain capabilities:
MSI/MSI-X.
Is fully trusted to not misbehave.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 14 / 20

15. PCI bus PCI-passthroughon Xen Moving forward
PVH dom0
PVH is a HVM guest from Xen’s point of view.
HVM-like access to the configuration space:
Transparent access to the MSI/MSI-X capabilities.
Transparent mapping of BARs into the physmap and handling
of writes to the BAR registers.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 15 / 20

16. PCI bus PCI-passthroughon Xen Moving forward
PVH dom0
Current PCI-passthrough code for HVM is in QEMU.
Impossible to use QEMU for PVH dom0.
No re-use of the QEMU PCI-passthrough code: would need
heavy modifications that would make sharing changes very
difficult.
Added a PCI config space mediator to the hypervisor: vPCI.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 16 / 20

17. PCI bus PCI-passthroughon Xen Moving forward
PCI-passthrough mediators in Xen
QEMU (user-space) and Xen for MSI-X mask bits for HVM
domUs.
pciback (hardware domain OS) for PV domUs.
Direct access / hypercalls for PV dom0.
vPCI (hypervisor) for PVH dom0.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 17 / 20

18. PCI bus PCI-passthroughon Xen Moving forward
Shortcomings
No support for VFIO/MDEV on Xen:
Threatening support for vGPU/XenGT in future releases.
3 different code bases to deal with PCI config space accesses:
More maintainership work.
Non uniform behaviour across different guests types.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 18 / 20

19. PCI bus PCI-passthroughon Xen Moving forward
Future items
Re-work vPCI so it can be used both inside the hypervisor and
in user-space.
Could be used by HVM and PVH guests as a standalone
PCI-passthrough utility.
Unify PCI-passthrough for HVM and PVH both domU and
dom0 into a single code-base.
Add support for the extended config space to HVM domUs:
allow to passthrough PCIe capabilities.
Add support to passthrough SRIOV capability to vPCI, for
PVH dom0.
Chicago – July 9th, 2019 Status of PCI emulation in Xen 19 / 20

20. PCI bus PCI-passthroughon Xen Moving forward
Q&A
Thanks
Questions?
Chicago – July 9th, 2019 Status of PCI emulation in Xen 20 / 20