Argo
Hypervisor-Mediated data eXchange
Christopher Clark
OpenXT Project
Xen Design and Developer Summit, 10th
July, 2019

Argo
● What it is
○ Overview
○ Origin
● Why
○ Key properties
○ What is different, distinctive about it
● How it relates to other hypervisors
○ HMX in other hypervisors
● A bit more about what it is
○ Details of the interface
○ Examples of it in use
● References

Argo : an interdomain communication transport
● An OpenXT technology, originally developed for XenClient.
○ Derived from v4v deployed in uXen in Bromium’s vSentry on millions of machines.
● Hypervisor-mediated data copies via private ring buffers with notifications.
○ Prioritizes isolation, access control and mandatory conformance to the transfer protocol.
○ Simple but powerful primitive to build upon.
● Included in Xen 4.12
○ Set the KConfig option CONFIG_ARGO=y to build it in,
then turn it on at the Xen boot command line: argo=1,mac_permissive=1

Argo : an Odyssey
The longest patch upstreaming process in Xen’s history. 2011 - 2019.

Argo : an Odyssey
The longest patch upstreaming process in Xen’s history. 2011 - 2019.
● 2010: XenClient project begins
● 2011: Ross reserves the hypercall number (39) in Xen. ← The timer starts here with the first patch integrated.
● 2012: Jean’s v4v patch series posted
● 2013: Phil’s Xen Summit presentation, Ross’s v4v patch series posted
● 2014: OpenXT project begins
● 2016: Clark presentation at Xen Summit
● 2017: HP ships laptops with Bromium micro-virtualization for Hyper-V and Windows 10 VBS
● 2018 - 2019: Argo patch series submitted, iterated, reviewed and accepted into Xen
● 2019: Design activity on Argo’s next development
HMX:
● 2019: Microsoft ships Windows Sandbox and SystemGuard in Windows 10
● 2019: IBM/RedHat microvm QEMU machine type inspired by Intel NEMU and AWS Firecracker

Argo : an Odyssey
One new hypercall, with four operations.
○ Register ring
○ Unregister ring
○ Sendv
○ Notify

Argo : Key Properties
● No memory is shared between guest VMs.
● Data is copied from the source to the destination.
○ Maintains spatial isolation between guest VMs.
● The Hypervisor performs the data transfer into guest-accessible memory.
○ Ensures that the transfer protocol in the ring structure is strictly adhered to.
○ Enables enforcement of mandatory access control policy by the hypervisor.
These properties will be maintained.

MILS Architecture Foundational Security Principals (2005)
● Data Isolation
Information in a partition is accessible only by that partition and private data remains private.
● Control of Information Flow
Information flow between partitions is from an authenticated source to authenticated recipients;
the source of information is authenticated to the recipient, and information only goes where intended.
● Periods Processing / Temporal Separation
Resources may be used by different components by time-slicing, where the system enforces that the
resource is cleaned to remove any trace of its previous use before being reassigned.
● Fault Isolation
Failure within a partition is prevented from cascading to any other partition.
Failures are detected, contained and recovered locally.

Argo : an interdomain communication transport
● Strong isolation between communicating domains
○ No memory is shared between VMs.
○ Does not use or require grant tables or XenStore.
● Strong mechanism for data transfer
○ Hypervisor performs the data movement. Memory accesses conform to the protocol.
● High performance suitable for sustained throughput
● A clean mapping to Linux and Windows native I/O primitives
● Clear separation from guest Operating System networking stacks
● A foundation for the future work that we intend to do

HMX : Hypervisor Mediated data eXchange
A term to describe:
Asynchronous authenticated message passing between VM partitions,
performed by the hypervisor.
Channels use no shared memory between the source and the receiver.
Enable enforcement of Control of Information Flow between domains
and preserve Data Isolation.

HMX: pattern for data delivery
Hypervisor
Message
Data
Receive memory
buffer
Delivery performed by the hypervisor:
● data delivered with context (size, origin)
● writes to the receive buffer, will conform to protocol / structure
VM : Sender VM : Receiver
DataContext
Hypervisor invoked
to send message

Inter-partition Communication: Xen (Grants)
Traditional PV communication channels using Xen’s primitives: Grants + Events
eg: PV network front and back ends.
Grants are most commonly used to share or copy pages between domains.
● Grant copy operations can ask hypervisor to write to a remote page, but:
○ Conformance to a ring access protocol is not enforced.
○ Hypervisor does not provide message header with metadata about the source.
=> HMX: no.
Aside: Grant table code has excessive complexity: Attributed responsibility for 5% of Xen’s XSA/CVEs.
Most recent grant table XSA was issued... yesterday.
Two API versions, switchable at runtime plus legacy operations... so OpenXT adds KCONFIGs to reduce it.

Inter-partition Communication: Hyper-V
VMBus : HvPostMessage
Messages are copied by the hypervisor into a private per-receiver message buffer
in hypervisor-owned private memory.
For delivery, individual messages are then copied by the hypervisor out into a
Message Page shared between the hypervisor and the receiver partition, when free
slots within the page are available.
=> HMX: yes.
since at least 2006.

Inter-partition Communication: uXen
Bromium vSentry uses uXen, which uses v4v.
● Open Source
● Has no legacy PV interfaces.
○ no grant tables
○ no XenStore
● All virtual devices use the simple, copy-based primitive.
○ No memory sharing of any kind.
Demonstrates that Xen’s problematic legacy interfaces can be replaced with an
approach built upon Argo.
=> HMX: yes.

Inter-partition Communication: Xen (Argo, since 4.12)
=> HMX : yes.

Argo: ring registration
Hypercall operations to establish and tear down a communication channel:
Register ring:
● Guest supplies memory to be used for receiving incoming messages.
● Specifies either: a specific other domain as sender, or receive from anyone.
● Access control policy check: is this channel allowed?
● Ring is mapped into the hypervisor address space.
Unregister ring
● Removes the hypervisor state. Message delivery ceases.

Argo: sendv
Sendv:
● Guest indicates the destination ring it wants to send a message to
○ and provides the data to be sent
● Access control check: is this transmission allowed?
● Is there sufficient space in the destination ring?
○ If not, interest is recorded and a notification will occur when sufficient space is free.
● Hypervisor synchronously copies the data into the ring.
○ Message header is written by the hypervisor, indicating the size and source.

Argo: notify
Notify:
● Queries for space availability in remote rings
○ Access control check: can this domain send to the specified ring?
○ Returns ring state indicator flags:
■ Exists
■ Shared
■ Empty
■ Sufficient space exists for the specified message size
■ Busy
○ Register sender’s interest: notification issued when available space becomes sufficient
● Initiates checking of space within own rings to trigger pending notifications

Argo: XSM/Flask controls
● enable
○ governs whether a domain may access any Argo operations at all
● ring registration
○ single-sender rings: allow / disallow registration of rings with the specified sender
○ any-sender rings: allow / disallow a domain to register any-sender rings
● send
○ allow sending a message to a specified domain
○ if the send permission is not granted, notify query operations will also be disallowed

Argo: use cases in OpenXT
Examples:
● User interface export from the control domain to the User Interface renderer VM
● Stubdomain toolstack helper services
● Interdomain DBUS service
○ Enables use of existing components maintained by other Open Source
communities, deployed in a multi-domain configuration
■ eg. NetworkManager and its user interface
● VM provisioning services for remote management
● Debugging support: ssh access to remote VM terminals

Argo: simple to use: ssh example
eg. Using the Linux device driver and userspace library to communicate between VMs:
# VM1: load the Argo kernel module:
insmod /lib/modules/*/extra/xen-argo.ko
# use the interposer to run the ssh server on port 8022
export INET_IS_ARGO=1
LD_PRELOAD=/usr/lib/libargo-1.0.so.0.0.0/usr/sbin/sshd -p 8022
# VM2: load the Argo kernel module:
insmod /lib/modules/*/extra/xen-argo.ko
# use the interposer to run the ssh client to connect to the server
export INET_IS_ARGO=1
LD_PRELOAD=/usr/lib/libargo-1.0.so.0.0.0/usr/bin/ssh 1.0.0.1 -p 8022

ssh example
ssh server process
libargo interposer library
process env: INET_IS_ARGO=1
Linux kernel driver: xen-argo.ko
/dev/argo_stream
Xen hypervisor
System calls using the Linux Argo device
Hypercalls: argo_op : REGISTER_RING,
SENDV, NOTIFY
Argo VIRQ
ssh client process
libargo interposer library
process env: INET_IS_ARGO=1
Linux kernel driver: xen-argo.ko
/dev/argo_stream
System calls using the Linux Argo device
Hypercalls: argo_op: SENDV, NOTIFY
Argo VIRQ

ssh example
Xen
Message
Data
Receive memory
buffer
Delivery performed by Argo in Xen:
● data delivered with context (size, origin)
● writes to the receive buffer, will conform to protocol / structure
VM with ssh client VM with ssh server
DataContext
Hypervisor invoked
to send message

Development items
● Sender domain context: hypervisor to provide metadata to message recipient
○ message header conveys sender’s XSM sid and bits indicating stubdomain relationship state
● Connection state: enforce and track ring pairings for bidirectional communication
○ Tracking bidirectional connections enables fine-grained access policy control
● Access control: run-time configurable firewall
○ Uses sender context and connection state. Replaces OpenXT’s existing v4vtables port.
● Nesting data transport
○ Sending data between VMs at different levels of nesting, with firewall policy enforcement
● Wildcard rings: protection of client forward progress
○ Increases resilience of the clients of any-sender rings to DoS by other clients
● Shutter rings: mechanism and policy controls for reducing hypervisor mappings

Research items
● Accelerated transport options leveraging available hardware
● Asynchronous message send primitive
● Reducing scope of notify wakeups to replace existing broadcast
● Integration with memory encryption technologies

References
Ian Pratt, Founder of the Xen Project, CTO of Bromium, at PSEC 2018: Hypervisor Security : Lessons Learned
Essential viewing. Strongly recommended.

References
● PSEC 2018: Hypervisor-Mediated data eXchange (HMX): Primitives for Authentic Communication
● Xen Summit 2013: XenClient Mechanisms to Strengthen Guest Separation
● Microsoft Hyper-V: Hypercall APIs Explained WinHEC 2006 presentation, slide 45 describes HvPostMessage.
○ Hypervisor Top Level Functional Specification, v5.0C, see HvPostMessage references, inc. section 11.11.
● OpenXT Summit 2016: Secure Virtual Platform
● Argo source code:
○ Hypervisor: argo.c in Xen 4.12
○ Linux drivers: current driver, and under-development vsock-based driver
○ Linux userspace: libargo, including LD_PRELOAD interposer enabling transparent use by socket applications
○ For a Windows driver, a port from v4v is required: see the xenv4v source code in OpenXT as a starting point.
● Argo documentation: argo.pandoc in Xen 4.12
● Argo next development: OpenXT wiki on Argo development items
● OpenXT: about the Project, project events, v4v documentation, v4vtables port for basic access control, with tooling
● uXen: v4v.c in uXen 4.1.6. uXen hypervisor source code repository: https://github.com/OpenXT/uxen/
Any questions? Please get in touch: #OpenXT on freenode IRC