Resource: LCE13 Name: LAVA Multi-Node Testing Date: 11-07-2013 Speaker: Video: https://www.youtube.com/watch?v=jqGcYwVxILI

1. LAVA Multi-node Testing
Antonio Terceiro, Senthil Kumaran,
Neil Williams, James Tunnicliffe & Fu Wei
LCE13 - Dublin, July 2013
Support for testing use cases across multiple
devices

2. DISCLAIMER: none of the
stuff you are about to see
is in production yet.
→You get an Unicode cat, though

3. ● Scope and use cases
● Proposed API
● Demo (???)
● Status and Future Plans
● Feedback - open discussion
Session Outline

4. Scope and use cases

5. ● test jobs involving multiple devices
● devices may be of same type, or not - it
doesn't matter
● devices get roles assigned to them
● each device can be flashed with a different
test image
Scope (1)

6. ● test writers can write any test they want.
● LAVA provides information/synchronization
primitives
● each device runs tests as it would normally
run, plus they can synchronize with each
other using the API
● in the test results are aggregated into a
single results bundle.
Scope (2)

7. ● Requests 2 devices
● Role 1: server
● Role 2: client
● Server:
○ install/start server application
● Client:
○ install client application
○ hits server, does benchmarks
Example 1: client/server interaction

8. ● Requests 6 devices. Roles:
○ database-server x 1
○ web-backend x 2
○ load-balancer x 1
○ client x 2
● wiring up the application together is test
writer's job
● provisioning/orchestration tools can be used
since tests are shell scripts
○ salt
○ puppet
○ chef
○ etc
Example 2: multi-tier application

9. ● the test suite for toolchain component X can
be cleanly split in say 3 components, a, b, c
● Request 3 devices with roles, a, b, c
● Node with role ${label} will download pre-
built binaries and run the test suite
component labelled with ${label}
Example 3: parallel build/test run

10. ● Physical setup:
○ Phone connected through USB to x86 host
○ Host configured to pass through the USB device to a
KVM device
● Request one phone device and one KVM
device
● Deploy latest Android build to phone
● Boot Ubuntu build to KVM
● Test that you can do ADB over USB
operations from Ubuntu to the phone
Example 4: ADB connectivity

11. ● The LAVA team will provide the
infrastructure and support
● Other teams can come up with whatever
tests they need
○ that are physically possible ;-)
● use cases that require specialized
hardware setups require lab work and will
be handled in a case-by-case basis.
Example n ...

12. Proposed API

13. device_group: [
{
"device_type": "highbank",
"role": "server",
"count": 1,
},
{
"device_type": "panda",
"role": "server",
"count": 1,
},
]
JOB submission (1)

14. device_group: [
{
"device_type": "galaxy-nexus",
"role": "phone",
"count": 1, "tags": "usb-pair-01"
},
{
"device_type": "kvm",
"role": "x86-box",
"count": 1, "tags": "usb-pair-01"
},
]
JOB submission (1.1)
Requesting a specific hardware setup

15. actions: [
{
"command": "deploy_linaro_image",
"parameters": {
"image": "url://to/images/server.img"
},
"role": "server"
},
[...]
]
JOB submission (2)
only applies to the
devices
with the "server" role

16. actions: [
// ...
{
"command": "lava_test_shell",
"parameters": {
"testdef_urls": [
"url://to/testdef.yaml"
]
}
},
// ...
]
JOB submission (3)
THIS DID NOT CHANGE

17. # testdef.yaml
metadata:
name: lava-test-shell-simple
description: "description of this testdef"
version: 1.0
format: "Lava-Test-Shell Test Definition 1.0"
run:
steps:
- lava-test-case test1 --result pass
- lava-test-case test2 --result fail
- lava-test-case test3 --shell ./test-script
- [...]
LAVA-Test-Shell Review (1)

18. # testdef.yaml
metadata:
name: lava-test-shell-simple
description: "description of this testdef"
version: 1.0
format: "Lava-Test-Shell Test Definition 1.0"
run:
steps:
- ./run-my-test-suite
LAVA-Test-Shell Review (2)

19. Set of programs in $PATH that are available for
your LAVA-Test-Shell tests to call
lava-test-case
lava-test-case-attach
lava-test-run-attach
lava-os-build
lava-installed-packages
lava-install-packages
The LAVA Test Shell API

20. An extension of the existing LAVA Test Shell
API that adds:
● Commands that provide contextual
information
● Commands for communication and
synchronization
The LAVA multi-node API

21. lava-role
Prints the role of the current device
# testdef.yaml
run:
steps:
- ./run-$(lava-role).sh

22. lava-group
Prints the hostnames and roles in the group
$ lava-group
panda01 client
highbank01 backend
highbank02 backend
highbank03 load-balancer

23. lava-sync
Global synchronization across the group
# testdef.yaml
run:
steps:
- ./setup-$(lava-role).sh
- lava-sync setup-done
- ./run-$(lava-role).sh

24. lava-send <msg-id> [k1=v1 k2=v2 ...]
Sends a message to the group
lava-wait <msg-id>
Waits for <msg-id> from some other node
lava-wait-all <msg-id>
Waits for <msg-id> from all other nodes
Other building blocks

25. Demo

26. Status and Future Plans

27. ● Implementation in advanced stage
● Started initial testing
● Should be available in production at 2013.
XX
Status

28. ● Exclusive lock on groups of devices
○ "this 10 devices share a network bus. Even if I am
only using 5, I don't want the other 5 to be used
simultaneously because that may affect the results
of my networking benchmarks
● [...]
Future work

29. Feedback
(Open discussion)

30. ● Tell us about your testing use cases
● Does lava-send <msg-id> k1=v1 k2=v2
look useful?
● "What if I need a specific hardware
configuration for my tests?"
● ...
Seed questions