Drew Moseley is an embedded solutions architect at Mender.io, which provides an open source over-the-air update manager for embedded Linux devices. He discussed challenges with traditional update methods, requirements for secure and robust OTA updates, and how Mender addresses these challenges through its dual A/B partitioning, rollback capability, and support for full system and application updates across many embedded environments and device types. He then demonstrated Mender and discussed its architecture, community, and commercial support offerings.

1. Drew Moseley
Embedded Solutions Architect
Mender.io

2. 1. Challenges and motivations
2. Requirements
a. Non-functional
b. Installer strategies
3. Mender
a. Demo and explanations
b. Device integration
c. Testing
d. Community overview
Session overview

3. About me
Drew Moseley
○ 10 years in Embedded Linux/Yocto development.
○ Longer than that in general Embedded Software.
○ Project Lead and Solutions Architect.
drew.moseley@mender.io
https://twitter.com/drewmoseley
https://www.linkedin.com/in/drewmoseley/
https://twitter.com/mender_io
Mender.io
○ Over-the-air update manager for
embedded Linux
○ Open source (Apache License, v2)
○ System updates with dual A/B rootfs layout
(client)
○ Application updates (docker, deb, files, etc.)
○ Remote deployment management (server)

4. The Challenge without OTA updates
The update process was:
● Insecure: No authenticity checks
● Manually-intensive: No remote updates
Vehicles had to be updated by one of two ways:
● Bringing the car into the dealership
● Customers mailed a USB drive
A secure and robust OTA update manager for
the entire fleet can automate this

5. The Challenge with OTA updates
The endless reboot issue affected cars equipped
with the 2017 and 2018 versions of UConnect
Customers lost access to:
● Audio controls
● Heat and A/C
● Heated seats
● Rear-view cameras
Customers reported car batteries were drained
due to the endless reboot
No rollback option or sanity checking to ensure
the successful installation of an update

6. The Challenge with OTA updates
An open source community with more eyes
reduces the number of vulnerabilities

7. The embedded environment
● Remote
○ Expensive to reach physically
● Power
○ Battery
○ Unclean shutdown
● Hostile deployment environment
● Product lifetime
○ Some markets 5 - 10 years
● Network
○ Intermittent connectivity
○ Low bandwidth
○ Insecure

8. Requirements
1. Robust and secure
a. Rollback
b. Signed/trusted images
c. Integrity/compatibility checks
2. Atomic updates
3. Support kernel, apps, libraries, DTBs
4. Integrates with existing environments
a. Easy to get started
5. Bandwidth consumption
6. Downtime during update

9. Installer strategies
1. In-place
2. Maintenance mode
3. Dual A/B
4. Proxy
Bootloader
Kernel, initramfs
User space
Updater

10. 1. In-place
2. Maintenance mode
3. Dual A/B
4. Proxy
Bootloader
Kernel, initramfs
User space
Updater
Installer strategies

11. 1. In-place
2. Maintenance mode
3. Dual A/B
4. Proxy
Bootloader
Kernel,
initramfs A
User space A
Updater A
User space B
Kernel,
initramfs B
Updater B
Installer strategies

12. 1. In-place
2. Maintenance mode
3. Dual A/B
4. Proxy
Gateway
Remote device
(sensor, ECU, etc.)
Updater
Installer strategies

13. What is Mender?
Robust and secure end-to-end
over-the-air (OTA) software
update manager for embedded IoT devices

14. Mender Customers
Company and Team
● Company behind Mender.io (Northern.tech) was founded in 2008
● Extensive experience ensuring security of embedded systems at scale
● Dedicated to bringing a secure best-in-class OTA manager to world’s connected devices
● Highly technical team dedicated in solving customers’ problems in the IoT space
● Relationships with the world’s largest brands
● Offices in San Francisco, USA and Oslo, Norway
Who we are
Mender Partners

15. Mender is Open Source
Most companies write their own homegrown updater from scratch (!)
Existing free and open source tools can help you
Save time and avoid stress (where you can)
○ Initial development
■ It looks so easy! Is it easy?
○ Ongoing maintenance
■ 5 years? 10? 20?
■ 1 product? 5? 10?

16. Key Features and Benefits - System Updates
Atomic rollback support with a dual A/B partitioning layout
Allows recovery and keeping device operational when updates fail for any reason
Consistent full image atomic updates
Avoids inconsistencies from partial updates caused by interrupted package installations
Secure TLS client-server communication with code signing
Ensures security with end-to-end signing and verification of artifacts for
authenticity and integrity
Increased fleet robustness
Reproducible deployments means your fleet is running identical code to your test
systems. Reduced testing matrix due to package combinatorics.

17. Key Features and Benefits - Application Updates
IoT Device
type = deb
file = web1.2.deb
Management Server
Software
Operating System
Update Manager
Update Module: deb
Mender Client
Match package type to
correct Update Module
Application
Install
Application updates
Mender supports “all” kinds of updates
● Packages
● Containers
● Files and directories
● Proxy device deployment
● Full control and customizability
It is very easy to install application-based updates on Debian, Ubuntu
and Raspbian OSes with Mender

18. Challenges with Applications updates
Managing individual packages quickly
becomes a logistical nightmare
○ Testing of many combinations of
installed packages
○ Installation order not strictly
enforced.
○ Partially-installed packages can
block installs of new fixed
packages

19. Generic embedded update workflow
Detect update
(secure channel)
Download
(secure channel)
Integrity
(e.g. checksum)
Authenticate
(e.g. signature)
DecryptExtract
Install Failure recovery
(e.g. roll back)
Compatibility
check
Sanity checks
Post-install
actions
Pre-install
actions
Must-have
Environment-specific
(Re)Start*
*E.g. reboot, restart service, start container

20. Why Mender
Mender is the only end-to-end open source OTA manager with
both management server and client
○ No vendor lock-in
○ Permissive Apache 2.0 license
○ A healthy peer review mechanism with more eyes on the code to
increase its security
Less reliance on tribal knowledge; less expensive than a
homegrown system
○ Homegrown systems have unpredictable scope, which leads to more
cost and time
○ Reliance on tribal knowledge: if a key person leaves, could cause
serious issues
Mender Hub is the only online community dedicated to enable
OTA on any device and OS
○ Future-proof your product roadmap

21. Mender Devices
Users
API Gateway
TCP 443
DeviceAuth
UserAdm
Inventory
Deployments
GUI
Workflows
Storage
Proxy
TCP 9000
Minio (optional)
MongoDB
ElasticSearch
Redis
Filesystem
external clients stateless application layer persistent storage
● Microservices
● Only port 443 and 9000
● RESTful API
○ Device API
○ Management API
/api/management/v2/devauth
/api/management/v1/inventory
/api/management/v1/deployments
….
https://docs.mender.io/apis/overview
Mender Server Architecture

22. Managed mode vs standalone mode
Update types:
○ Full image, and delta of full image
○ Incremental (package, file, tarball, container)
○ Proxy
○ others
Partitions: Boot (optional), Rootfs A, Rootfs B, Data, others
Bootloader integration:
○ Controls boot process, A or B
○ Grub (BIOS/UEFI)
○ U-boot w/ auto patching
○ U-boot w/ manual patching
Linux runtime integration:
○ Supports eMMC/SD/block device & UBI volumes
○ Mender client application (as daemon for managed mode)
Target OSes supported:
○ Yocto/OpenEmbedded (OOB)
○ Debian, Raspbian, Ubuntu and other binary distros
○ Buildroot (somewhat)
○ OpenWRT (somewhat)
https://docs.mender.io/devices
Mender | System Integration Requirements

23. Automatic Rollback With A/B Image Update
Bootloader
Kernel
Persistent data
Application
Device
OS A (active) OS B (inactive)
Kernel
Application

24. Mender Architecture With Update Modules
● Any type of update
● Update actions implemented in
modules
Mender
Artifact
Mender
Client
2.x
File system
Image
File(s)
Containers
Update
Module
Proxy
firmware
OS A/B
image
● Popular application update types
supported out-of-the-box

25. Mender - State Scripts

26. Mender - State Scripts

27. Mender - State Scripts

28. Mender - State Scripts

29. Demo time

30. Mender is built to scale
Designed for low
infrastructure load
default: update check
every 30 min per device
exponential backoff
separate storage service
for transferring updates
Microservices architecture
for horizontal scaling
application and storage layer
separation
scale individual services
native support for docker and
Kubernetes
Tested with more than
120,000 devices
3 machines, 2 CPUs, 8 GB RAM
demo available online

31. Open source and commercial offering
● Mender will always support the open source edition
● Follows the “open core” business model
● Basic features relating to adoption and basic usage will always be open source
○ E.g. Board support, OS support, end-to-end update process, client customizability
● Scalability, risk management and security features are commercial offerings
○ E.g. Phased rollout, schedule deployment, permissions, user account management

32. Mender Plans

33. Professional Services
Mender-specific
- Integrating the Mender client with your device
- Design of device partition layouts
- Guidance on a scalable Mender backend setup
- Integrate Mender with CI/CD pipeline
- Develop custom use-case specific update scripts
Embedded Linux/Yocto project consulting
- Embedded Linux system design architecture
- Yocto build customizations
- Yocto platform layer development
- Yocto recipe development
- Custom automated build setup

34. Learn More
Get started now:
https://docs.mender.io/2.0/getting-started
Join the Mender Hub community:
https://hub.mender.io
Mender on Github:
https://github.com/mendersoftware/
contact@mender.io @mender_io /company/mender.iohttps://mender.io/

35. Questions?
Thank you!
@drewmoseley
https://mender.io
drew.moseley@mender.io