Resource: LCE13 Name: Advanced LAVA Lab configuration Date: 11-07-2013 Speaker: Video: https://www.youtube.com/watch?v=KgNSoe2WosY

1. Advanced LAVA Lab
Configuration
Tyler Baker & Georgy Redkozubov
LCE13 - Dublin, July 2013
Taking advantage of advanced configuration
options to optimize your LAVA Lab

2. Education
● Overview of the Cambridge Lab Deployment
● Multi-Machine Deployment
● Guided Tour: LAVA Dispatcher
○ Integrating a Test
○ Jenkins, Metadata, Filters, Image Reports
○ Override Boot Commands
○ Interactive Boot Commands
Ideas
● Remote Workers Improvements
Feedback
● LAVA Configuration Improvements
Community
Overview

3. Cambridge Lab Deployment
validation.linaro.org:
● control
● dispatcher01
● fastmodels01
● fastmodels02
● fastmodels03
● fastmodels04
● fastmodels05
● fastmodels06
● leg01
● leg02
● lng01
● lng02
● 75 Devices
● 4 Serial Console
Servers
● 14 Power
Distribution Units
● 5 Switches
● Constantly adding
new hardware to
keep up with
demand
● Mixture of VMs and
Bare metal

4. Cambridge Lab Deployment
How do we keep all the LAVA servers running
the same software?
● Salt Stack - https://github.com/saltstack/
○ Salt provides fast, flexible, and easy to use
configuration management.
○ LAVA team uses Salt to manage:
■ Deployment of LAVA code into production
● http://validation.linaro.org
■ Installed packages
■ Device configurations
■ User accounts and access control
■ Deployment of Fastmodels
■ Various helper scripts

5. Cambridge Lab Deployment
LAVA Lab Configuration
● It's open source!
○ bzr branch lp:lava-lab
● Scripts for managing our production LAVA instance
○ lava-lab/scripts
● This is great resource for anyone interested in sample
device configurations
○ lava-lab/lava/devices
● Want to know how our power cycling scripts work?
○ lava-lab/lava/lab-scripts

6. Cambridge Lab Deployment
There is a salt state for that!
● As an example lets say you want the latest version of
adb and fastboot installed.
android-tools-adb:
pkg:
- latest
android-tools-fastboot:
pkg:
- latest

7. Cambridge Lab Deployment
Salt is what keeps all the LAVA remote workers
in sync from code to configuration. Now lets
look at a multi machine LAVA deployment.

8. Cambridge Lab Deployment
DEMO

9. Multi-Machine LAVA Deployment
LAVA Remote Workers
● Master Instance
○ Typically host the postgres database
○ Can use a remote postgres database if required
● Remote Worker Instance
○ Connects remotely to postgres database
○ Mounts the master instance media directory using
sshfs
○ /srv/lava/instances/<instance>/var/lib/lava-
server/media
● Room for improvement

10. LAVA Master Instance
Multi-Machine LAVA Deployment
Database Logs
Worker Instance
D
U
T
D
U
T
D
U
T
Worker Instance
D
U
T
D
U
T
D
U
T
Worker Instance
D
U
T
D
U
T
D
U
T
...
User Interaction

11. Multi-Machine LAVA Deployment
● Master Instance
○ Is provisioned with salt to provide a user account for
LAVA management
■ instance-manager
○ LAVA is installed with LAVA Deployment Tool
■ become-instance-manager
■ ./lava-deployment-tool setup
■ export LAVA_BUILDOUT_CFG=buildout-production.cfg
■ export LAVA_DB_ALLOWREMOTE=yes
■ ./lava-deployment-tool install production

12. Multi-Machine LAVA Deployment
● Remote Worker Instance
○ Is provisioned with salt to provide a user account for
LAVA management
■ instance-manager
○ LAVA is installed with LAVA Deployment Tool
■ become-instance-manager
■ ./lava-deployment-tool setup
■ export LAVA_BUILDOUT_CFG=buildout-production.cfg
■ ./lava-deployment-tool installworker production
○ Salt managed files
■ Device configurations
■ PDU Scripts

13. Multi-Machine LAVA Deployment
DEMO

14. Guided Tour: Integrating a Test
LAVA checks return codes or parses stdout to
generate test results
● This is very powerful, as test written in ANY language
can be easily integrated.
● What do you need to know?
○ Where the test lives
○ How to compile and run the tests
○ How to interpret the test output
○ Write a regular expression
○ Submit a job

15. Guided Tour: Integrating a Test
metadata:
name: mauve
format: "Lava-Test-Shell Test Definition 1.0"
description: "Java Mauve Tests for Open Embedded"
version: 1.0
install:
git-repos:
- git://git.linaro.org/qa/test-definitions.git
- git://git.linaro.org/people/andrew.mcdermott/mauve/aarch64/mauve.git
run:
steps:
- 'DISPLAY=:10'
- 'export DISPLAY'
- 'Xvfb $DISPLAY &'
- 'XVFBPID=$!'
- 'PATH=$PATH:/lava/tests/$TESTRUN_ID/test-definitions/openembedded/scripts'
- 'rm -rf $HOME/mauve'
- 'ln -s /lava/tests/$TESTRUN_ID/mauve $HOME/mauve'
- 'cd $HOME/mauve'
- 'lava-test-case java.lang --shell mauve-test-pkg java.lang -showpasses'
parse:
pattern: "(?P<result>(PASS|FAIL|SKIP)):s+(?P<test_case_id>(.*))"
fixupdict:
PASS: pass
FAIL: fail
SKIP: skip

16. Guided Tour: Integrating a Test
SKIP: java.lang.UnsupportedOperationException.classInfo.getMethods
SKIP: java.lang.VerifyError.classInfo.getConstructors
SKIP: java.lang.VerifyError.classInfo.getDeclaredConstructors
SKIP: java.lang.VerifyError.classInfo.getDeclaredFields
SKIP: java.lang.VerifyError.classInfo.getDeclaredMethods
SKIP: java.lang.VerifyError.classInfo.getFields
SKIP: java.lang.VerifyError.classInfo.getMethods
[dix] Could not init font path element /usr/share/fonts/X11/misc/, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/TTF/, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/OTF/, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/Type1/, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/100dpi/, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/75dpi/, removing from list!
PASS: java.lang.reflect.Modifier.toString
PASS: java.lang.reflect.Modifier.toString12
PASS: java.lang.reflect.Field.promotion
PASS: java.lang.reflect.Field.toString
PASS: java.lang.reflect.Field.access
PASS: java.lang.reflect.Array.newInstance
PASS: java.lang.reflect.Array.set
PASS: java.lang.reflect.Method.invoke
PASS: java.lang.reflect.Method.equals
PASS: java.lang.reflect.Method.toString
PASS: java.lang.reflect.Proxy.ExceptionRaising
PASS: java.lang.reflect.Proxy.ToString
PASS: java.lang.reflect.Proxy.check13
PASS: java.lang.reflect.Proxy.DeclaringClass
PASS: java.lang.reflect.InvocationTargetException.Chain
PASS: java.lang.reflect.Constructor.newInstance
parse:
pattern: "(?P<result>(PASS|FAIL|SKIP)):s+(?P<test_case_id>(.*))"
fixupdict:
PASS: pass
FAIL: fail
SKIP: skip

17. Guided Tour: Integrating a Test
Many tests already integrated
● http://git.linaro.org/gitweb?p=qa/test-definitions.git;a=tree
● Folder for distribution specific tests
○ Ubuntu
○ Fedora
○ OpenEmbedded
○ Android
○ Contribution is encouraged. Please feel free to
help us integrate!

18. Guided Tour: Jenkins, Metadata, Filters, Image Reports
DEMO

19. Guided Tour: Override Boot Commands
● When might I need to override the default
boot commands?
○ FDT
○ Android
○ OE
○ Ubuntu
○ TFTP
○ PXE
○ UEFI

20. Guided Tour: Override Boot Commands
● Need to add the boot command options to
your device-type configuration or device
configuration
boot_options =
boot_cmds
[boot_cmds]
default = boot_cmds

21. Guided Tour: Override Boot Commands
● Now add your new boot commands to the
device-type configuration or device
configuration
boot_cmds_pxe = usb start,
dhcp,
pxe boot

22. Guided Tour: Override Boot Commands
● Simply add your new boot options to the job
file to override the boot commands.
{
"command": "boot_linaro_image",
"parameters": {
"options": ["boot_cmds=boot_cmds_pxe"]
}
},

23. Guided Tour: Interactive Boot Commands
● Allows you to specify boot commands in the
job file
○ sendline/expect stanzas
■ “sendline foo” - This instructs LAVA to send an
“foo” to the bootloader prompt.
■ “expect bar” - This instructs LAVA to expect a
“bar” from the bootloader prompt.
○ Traditional boot commands
■ This will simply send the commands to the
bootloader and proceed.
○ Rolling out to all devices very shortly.

24. Guided Tour: Interactive Boot Commands
● How do I use them?
○ First set the parameter "interactive_boot_cmds" to
true.
○ Add your sendline/expect stanza to the options field.
■ Each line is enclosed in quotations.
■ All lines are comma separated.
■ No need to quote your kernel arguments.
■ Any backslash must be escaped by ""

25. Guided Tour: Interactive Boot Commands
{
"command": "boot_linaro_image",
"parameters": {
"interactive_boot_cmds": true,
"options": ["sendline a",
"expect Choice:",
"sendline 1",
"expect Select the Boot Device:",
"sendline 2",
"expect File path of the EFI Application or the kernel:",
"sendline uImage",
"expect [a/g/l]",
"sendline l",
"expect Add an initrd: [y/n]",
"sendline y",
"expect File path of the initrd:",
"sendline uInitrd",
"expect Arguments to pass to the binary:",
"sendline console=ttyAMA0,38400n8 root=/dev/mmcblk0p2 rootwait ro mem=1024M
earlyprink",
"expect File path of the local FDT:",
"sendline rtsmrtsm_ve-ca15x4-ca7x4.dtb",
"expect Description for this new Entry:",
"sendline Test Image",
"expect Choice:",
"sendline 5",
"expect Start:",
"sendline 2"]
}
},

26. Community: Open Distributed Hardware Lab
Idea: Provide hardware as a service to the
community
● community.validation.linaro.org
○ Will be open to anyone.
○ Connecting board farms
from around the world.
○ Community donated
hardware (where you are)
● Want to help out?
○ tyler.baker@linaro.org
LAVA Master Instance
Databas
e
Log
s
Worker Instance
D
U
T
D
U
T
D
U
T
Worker Instance
D
U
T
D
U
T
D
U
T
Worker Instance
D
U
T
D
U
T
D
U
T

27. LAVA Master Instance
Multi-Machine LAVA Deployment
Database Logs
Worker Instance
D
U
T
D
U
T
D
U
T
Worker Instance
D
U
T
D
U
T
D
U
T
Worker Instance
D
U
T
D
U
T
D
U
T
...
User Interaction

28. Discussion: Better LAVA Scalability
● LAVA Dispatcher
○ Proxy all commands over ssh to node.
○ Can use ARM boards as dispatcher proxies.
● TODO: More discussion items.

29. Discussion: Remote Worker Improvements
● The current issues with the remote workers
○ If the remote worker goes offline, the boards appear
online in LAVA and submitted jobs are never
scheduled. This is very confusing for the LAVA user.
○ If a remote worker goes offline, LAVA should detect
this and offline all associated boards.
○ There needs to be a visual way to know which remote
workers are connected or disconnected.
○ Writing log files over sshfs is hacky, there must be a
better way.
○ Remote database connection security.

30. Feedback: LAVA Configuration Improvements
● How can we make LAVA easier to
configure?
○ Installation from packages
■ apt-get install lava-server
● apt-get install lava-worker
■ yum install lava-server
● yum install lava-worker
■ Effort in this area underway
● How can we make LAVA easier to
distribute?
○ Use SSH to tunnel database connection?
● What else can be done to make a large
LAVA lab deployment easier?