Part 1: Reworking the ARM GIC Emulation The ARM Generic Interrupt Controller (GIC) provides some level of virtualization support in hardware. This still requires emulation of the distributor part, which has to integrate with the virtualization feature. Doing this in a performing and readable way is not trivial, especially the locking strategy tends to be complicated. While extending the existing virtual GIC support in Xen to cover support for MSIs, some issues have been discovered which ask for some significant changes in the existing code. The presentation will briefly describe the existing VGIC design and the issues we faced when trying to extend it. Based on this the changes will be presented and how they improve and ideally simplify the code. Part 2: Xen Challenges in the ARM ITS Emulation For being able to use MSIs on ARM systems in Xen domains we need to emulate the ARM GICv3 ITS controller. Its design is centered around a command queue located in normal system memory. Emulating this in the Xen hypervisor brings some interesting challenges, ranging from safely accessing the guest memory and dealing with possible propagation of commands, to possible DOS attacks by domains keeping the emulation code busy. The presentation outlines the main problems and how we hit Xen limits in emulating this correctly and efficiently. Also it presents our temporary workarounds and their drawbacks.

1. ARM®
Interrupt Virtualization
Andre Przywara <andre.przywara@arm.com>
ARM Ltd., Cambridge, UK
Xen summit 12/07/2017
© ARM 2017

2. ARM interrrupt virtualization agenda
GICv2 and virtualization overview
Xen vGIC implementation
Rework plans
GICv3/ITS architecture changes
ITS emulation challenges
2 © ARM 2017

3. GICv2 architecture
Device
CPU interface CPU interface CPU interface
Device Device
Core 0 Core 1 Core n....
AXI bus
IRQ lines (SPIs)
GICv2
....
SGIs
Distributor
PPIs
IRQ/FIQ
3 © ARM 2017

4. ARM GICv2 at a glance
”distributor” is the central component
supports up to 8 CPUs and 1020 interrupts (988 shared + 32 private)
has an input pin for each wired interrupt (SPIs)
connects to separate CPU interfaces (one per core)
has per-core input pins for private interrupts (PPIs)
handles inter-processor interrupts internally (SGIs)
Programmed via MMIO accesses
Configuration affects always a group of interrupts (32-bit registers)
Some registers are banked per CPU (at the same memory address)
4 © ARM 2017

5. Virtualization support in GICv2
Virtual CPU interface allows IRQ ACKs and EOIs without exiting the guest
Hypervisor sets up virtual IRQs in List Registers (LRs)
In the guest the (virtual) CPU interface relates to these
Allows connecting a physical interrupt to a virtual one
Physical IRQs gets EOIed at the same time
No need to trap or monitor EOI in this case anymore
Distributor
IRQ lines (SPIs)
GICCGICV
LRs
GICH
CPU interface
5 © ARM 2017

6. Xen vGIC implementation
Lives in xen/arch/arm (shared between arm(32) and arm64)
Distributor is emulated (mostly in vgic-v2.c)
Pending interrupts are written into the list registers (LRs)
Interrupt acknowledgment and EOI is handled without
exiting the guest (by the GIC hardware)
After guest run distributor emulation syncs back from
the LRs
Virtual IRQ configuration is stored in ranks (groups of
IRQs)
GICV
LRs
CPU interface
Distributor
(emulated)
GICCGICH
virtual IRQs
traps
6 © ARM 2017

7. vGIC implementation design
Code design is centered around MMIO IRQ grouping (ranks)
One lock per VCPU and one lock per rank
Sophisticated lockless tricks to avoid excessive rank locking
IRQ 32
IRQ 33
ISENABLER:
. . . .
IRQ 63
IRQ 62
IRQ 61
031
IRQ 34
ITARGETSR:
IRQ 32IRQ 33IRQ 34IRQ 35
0781516232431
BUT:
7 © ARM 2017

8. Implementation issues and limitations
Distributor accesses are rare (during IRQ configuration only)
VCPU entry/exit path is much more frequent and sensitive
State is held in bitmaps and bytemaps covering contigious IRQ numbers
proves problematic with sparse LPI allocation (later)
Handling of level vs. edge triggered interrupts
8 © ARM 2017

9. vGIC rework
To bring ITS/LPI support closer to the existing code base:
Store all virtual IRQ properties in struct pending_irq
Protect with one lock per IRQ
Get rid of the ranks and their locks at all
Simplify IRQ list management (remove lr_queue list)
Introduce reference counting for dynamic pending_irq’s
Model level-triggered interrupts correctly
9 © ARM 2017

10. GICv3 architecture
Device Device Device Device Device
....
AXI bus
IRQ lines (SPIs)
....
SGIs
Distributor
PPIs
GICv3
Redistributor Redistributor Redistributor
Core 0 Core 1 Core n
CPU interface CPU interface CPU interface
ITS
Memory
IRQ/FIQ
MSI MSI
10 © ARM 2017

11. GICv3 changes
System register access to CPU interface (drops banked MMIO)
IRQ routing allows millions of cores
Lifts the 8-CPU limit of GICv2
Uses MPIDR based values to specify one target core per IRQ
Splits distributor to separate private and shared IRQs
New class of interrupts (LPI) via an Interrupt Translation Service (ITS)
Allows MSI/MSI-X support
Supports indirections for target cores (via collections)
Introduces device ID sampled from the bus
New IRQ class with possibly thousands of LPIs and probably sparse allocation
Tables are held in physical memory
11 © ARM 2017

12. GICv3/ITS Xen implications
Potentially large, sparsely allocated LPIs spoil vGICv2 ranks
Requires dynamic allocation (and de-allocation → ref-counting)
Motivates storing all IRQ property in struct pending_irq
Brings up locking issues (solution: per-IRQ lock)
ITS data structures are held in guest physical memory
Need to reach into domain memory (fortunately cheap on ARM64)
Creates security question (we can’t trust this memory)
Provides natural resource limitation (more LPIs → more memory)
Caching is architected and common in hardware too
12 © ARM 2017

13. ITS tables (simplified)
device
command
prop pend
coll
13 © ARM 2017

14. ITS challenges
Security and DOS challenges:
Untrusted guest memory holds internal information
Need to sanitise information on every access
(sometimes not feasible)
Accessing guest memory is not for free
Looking at getting rid of storing information there
Guest could fill (potentially large) emulated command queue
Architecture allows defered execution
But we need to schedule this!
Some guest actions may require host action
Disabling or enabling interrupts is one case
To avoid IRQ storms we want to disable IRQs on the hardware
Requires queueing an ITS command to the host command queue
May open DOS vector when a domain often disables/enables LPIs
14 © ARM 2017

15. Preliminary fixes
ITS emulation only for Dom0 → trust the guest
Untrusted guest memory holds internal information
Sanitise as much as possible
Trust the guest!
Guest could fill (potentially large) emulated command queue
Real World Dom0s only schedule two or three commands
Trust the guest!
Some guest actions may require host action
Keep interrupts enabled on the host
Detect guest disabled IRQs early
Ignore IRQ storms for now (trust the guest!)
15 © ARM 2017

16. Xen questions
Accessing shared host resources on behalf of guests
Could introduce deferred host command queueing
How to schedule commands?
Suspend vCPUs until commands have been executed?
More costly emulation on behalf of a guest
Return early when timeslice expires!
But when to return to continue the work?
Deschedule guest while in an MMIO emulation?
Schedule tasklet on account of a guest?
16 © ARM 2017

17. The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM limited
(or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be
trademarks of their respective owners.
Copyright © 2016 ARM Limited
© ARM 2017