This document discusses immutable image-based operating systems. It begins with definitions, explaining that immutable systems have read-only critical components and use separate mechanisms for user data and applications. The conceptual architecture is described as updating the entire OS image rather than individual packages. Benefits include atomic updates and reproducibility, while drawbacks include less flexibility and required reboots. Technologies like libostree and Btrfs snapshots are discussed. Examples of desktop and embedded immutable systems are provided, along with references for further information.

1. Immutable Image-Based
Operating Systems
Presented by
Drew Moseley
Technical Solutions Architect
Toradex

2. WHAT WE’LL
COVER TODAY…
• Definitions
• Architecture
• Benefits
• Desktop Distro
• Embedded OS Architecture
• Demo(?)
AGENDA

3. WHAT WE DO
RELIABLEAND EASY-TO-USE EMBEDDED
SOLUTIONS FOR YOU
Arm® System on Modules
Reliable
Long-Term Maintenance
Scalable
From Stock
Production-Ready Software
Yocto-Based Linux
Windows Embedded Compact
Development Tools
Long-Term Maintenance
Ease-of-Use
Support
Ecosystem

4. Definitions
• Immutable1: not capable of or susceptibleto change
› Critical portions of the system are "read-only"
› Updates are performed with only well-defined mechanisms
› User data stored separately
› Applications generally use a different mechanism
1
https://www.merriam-webster.com/dictionary/immutable
2
https://www.merriam-webster.com/dictionary/image
• Image2: exact likeness
› Updating the entire "Operating System"
› Updating individual packages or applications "not supported"
3
https://www.merriam-webster.com/dictionary/atomic
• Atomic3: of, relating to, or concerned with atoms
› Incapable of being subdivided
› No chance of partially installed updates
Other names: Layered OS, Reprovisionable, Anti-hysteresis

5. Sidebar: Pets vs Cattle
• Coined by Randy Bias1
› Originally from Enterprise Computing
Space
• Desktop/Server:
› Pets - Individual laptops
› Cattle - Servers managed as code
• In Embedded:
› Pets - Weekend projects,
home automation
› Cattle - Large fleets of identical devices.
1 http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/

6. Conceptual Architecture
System
Operating System
(Image v1)
Bootloader
Kernel/DTB/Initramfs
"OS" Packages
User Data
User Applications

7. Conceptual Architecture
System
Operating System
(Image v2)
Bootloader
Kernel/DTB/Initramfs
"OS" Packages
User Data
User Applications
Operating System
(Image v2)
Bootloader
Kernel/DTB/Initramfs
"OS" Packages

8. Benefits
• Atomic versioning and updates of critical system components
› No more `apt --fix-missing --install` or related commands
• User components separately managed
› Better isolation of dependencies (ie containers)
› Fewer conflicts based on OS installed package versions
• Reproducibility
› The OS image is deterministic
› No configuration drift
• Better testing
› Exactly matching software on test and productiondevices
• Rollback capability
• More secure? Arguable

9. Drawbacks
• New/unfamiliar workflows
• Less flexible than traditional distros
• Do all your applications run in the
sandbox?
• Reboot required for any updates
› Mitigated by the app packaging system
• Is it really appropriate for desktop/laptop
use?

10. Technologies and Concepts
• libostree (https://ostreedev.github.io/ostree/)
› "Git for filesystems"
› Content-addressable objectstorage + hard links
• Multiple partitions
› Usually mounted read-only
› Symlinks for mutable config files
• Btrfs snapshots
• Declarative configuration
• Layering: https://coreos.github.io/rpm-ostree/

11. WHAT IS libostree?
"libostree is both a shared library and suite of command line tools
that combines a “git-like” model for committing and downloading
bootable filesystem trees, along with a layer for deploying them and
managing the bootloader configuration." 1
"git-like"
model
bootable
filesystem
trees
Bootloader
configuratio
n
1 https://github.com/ostreedev/ostree#libostree

12. OSTree BASICS
• File-based (!)
• Relies on non-root mount/“bind-mount”
- Normally the root of a file system is mounted as “the root”
- Linux allows to bind mount a subdirectory
• Initramfs mounts OSTree
- Pivot into bind mount/sub-directory
• Hardlinks are used to speed-up deployment and
minimize space usage
Source: https://medium.com/@1154_75881/what-is-the-difference-between-a-hard-link-and-a-symbolic-link-14db61df7707

13. Libostree filesystem layout
(Simplified)

14. "File system based on the copy-on-write principle
using B-trees, developed at Oracle since 2007"1
• Declared stable in Linux in 2013
• Subvolumes
• Atomically writable snapshots
• Cloning (multiple inodes pointing to the same
disk blocks)
BTRFS Snapshots
1
https://en.wikipedia.org/wiki/Btrfs

15. Applications
Containers: https://www.docker.com/ or https://podman.io/
Flatpak: https://www.flatpak.org/
Appimage: https://appimage.org/
Snaps: https://snapcraft.io/
Bundled with dependencies
"Distro-independent" Linux packages
Sandboxed from the host OS and other packages

17. Torizon Demo

18. Universal Blue
Based on Fedora Silverblue
"Cloud Native Linux Desktop Model"
• Base images generated by OCI containers
o RPM-OSTree
o BTRFS (snapshots?)
o Applications normally use Flatpak
Distrobox (https://distrobox.it/)
Linuxbrew (https://docs.brew.sh/Homebrew-on-Linux)
Many variants:
• Bluefin: GNOME Desktop
• Bluefin-DX: Bluefin + Cloud developer tools
• Built-in GPU drivers

19. Universal Blue Demo

20. NixOS
Reproducible
Declarative
Reliable
Package Manager
or
Full blown OS

21. VanillaOS
• Ubuntu Desktop based
• Dual A-B partitions

22. Survey of available systems
Desktop/Server
• Debian: Endless OS
• Ubuntu: VanillaOS
• Fedora: Silverblue
• Universal Blue
• NixOS
• GNU Guix
• Clear Linux
• Fedora CoreOS
• openSUSE Aeon (Gnome)
• openSUSE Kalpa (KDE)
• Flatcar Linux
• Bottlerocket OS
• Talos Linux (k8s)
• ChromeOS
Embedded
• Torizon
• Ubuntu Core
• Linux microPlatform
• BalenaOS
• SteamOS

23. References
• https://github.com/castrojo/awesome-immutable
• https://discord.gg/N4mswFw6ds
• https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-anti-hysteresis/
• https://www.torizon.io/
• https://www.torizon.io/open-source-community
• https://universal-blue.org/

24. THANK YOU
FOR YOUR INTEREST
www.toradex.com | www.torizon.io | developer.toradex.com
community.toradex.com | labs.toradex.com