Each processor/chip architecture has its own procedure to boot the kernel. It works with desgined partition layout and vendor specific firmwares/bootloaders in the boot partition. We can learn the related knowledge from the Raspbian image for Raspberry Pi, which is the board we can obtain easily. However, the diversity between the special booting procedures with specific firmwares/bootloaders increases the complexity for distribution maintainers. It will be great if there is a way to make it more generic that can be applied to most of the chip architectures/boards to boot up the system. After referring to some Linux distributions, we learned U-Boot may play a role in the solution. It splits the booting procedure into hardware specific and generic system parts. This helps distribution maintainers deploy the generic system with OSTree, including device trees. Let’s deep dive into this magic booting procedure!

1. A Journey to Boot Linux
on Raspberry Pi
Jian-Hong Pan (StarNight)
@ COSCUP 2020

2. Who am I
潘建宏 / Jian-Hong Pan (StarNight)
Endless OS Foundation
You can find me at
● http://www.slideshare.net/chienhungpan/
● GitHub: starnight
● Email:
jian-hong [AT] endlessm.com
starnight [AT] g.ncu.edu.tw

3. Outline
● History
● OSTree
● How does Raspberry Pi boot
● Device Tree
● Deploy Device Tree with OSTree
● Why U-Boot
● Wrap Up! Boot Endless OS on Raspberry Pi
● Demo
● Reference

4. History
Endless OS Foundation focuses on enabling people to harness the
power of computing everywhere.
● Develops a Linux-based operating system Endless OS as the payload
and pre-loads a lot of knowledge content and apps.
● The preloaded content and apps help people self-learning, especially
for the area without internet.
● Uses OSTree and flatpak as the key technology for system
deployment and update.
● Endless OS runs on general x86_64 computers, just like other Linux
distributions.
● Raspberry Pi is the wide spread low cost ARM based computer in
recent years. It is a good tool for education.

5. OSTree
Description from OSTree
Is both a shared library and suite of command line tools that combines a
"git-like" model for committing and downloading bootable filesystem
trees, along with a layer for deploying them and managing the
bootloader configuration.
● Transactional upgrades and rollback for the system
● Replicating content incrementally over HTTP via GPG signatures and
"pinned TLS" support
● Support for parallel installing more than just 2 bootable roots
● Binary history on the server side (and client)
● Introspectable shared library API for build and deployment systems
● Flexible support for multiple branches and repositories, supporting
projects like flatpak which use libostree for applications, rather than
hosts.
Another reference: The Fundamentals of OSTree

6. OSTree Example
$ ostree admin status
* eos 4c59454f03dbfe2c86858b0ade89caf36db3fba25233971484a4796a39e02d02.0
Version: 3.9.0~200702-230029
origin refspec: eos:os/eos/arm64/master
GPG: Signature made Fri 03 Jul 2020 07:50:23 AM CST using RSA key ID 9E08D8DABA02FC46
GPG: Good signature from "EOS OSTree Signing Key 1 <maintainers@endlessm.com>"
GPG: Key expires Sun 01 Jul 2029 12:18:11 AM CST
GPG: Signature made Fri 03 Jul 2020 07:50:26 AM CST using RSA key ID FCF17B17F1F8E157
GPG: Good signature from "EOS Flatpak Signing Key 1 <maintainers@endlessm.com>"
GPG: Key expires Wed 10 Jun 2026 01:47:38 AM CST
eos 1432c89ac02b1186ecbc0c3b6f1af2677aa09d4812cb97c4bd0fa46721342227.0 (rollback)
Version: 3.9.0~200621-230039
origin refspec: eos:os/eos/arm64/master
GPG: Signature made Mon 22 Jun 2020 07:56:54 AM CST using RSA key ID 9E08D8DABA02FC46
GPG: Good signature from "EOS OSTree Signing Key 1 <maintainers@endlessm.com>"
GPG: Key expires Sun 01 Jul 2029 12:18:11 AM CST
GPG: Signature made Mon 22 Jun 2020 07:56:57 AM CST using RSA key ID FCF17B17F1F8E157
GPG: Good signature from "EOS Flatpak Signing Key 1 <maintainers@endlessm.com>"
GPG: Key expires Wed 10 Jun 2026 01:47:38 AM CST

7. $ ls -l /ostree/deploy/eos/deploy/
total 16
drwxr-xr-x 14 root root 4096 Jun 22 08:08 1432c89ac02b1186ecbc0c3b6f1af2677aa09d4812cb97c4bd0fa46721342227.0
-rw-r--r-- 1 root root 41 Jun 22 08:08 1432c89ac02b1186ecbc0c3b6f1af2677aa09d4812cb97c4bd0fa46721342227.0.origin
drwxr-xr-x 14 root root 4096 Jul 6 16:13 4c59454f03dbfe2c86858b0ade89caf36db3fba25233971484a4796a39e02d02.0
-rw-r--r-- 1 root root 41 Jul 6 16:13 4c59454f03dbfe2c86858b0ade89caf36db3fba25233971484a4796a39e02d02.0.origin
$ ls -l /ostree/deploy/eos/deploy/4c59454f03dbfe2c86858b0ade89caf36db3fba25233971484a4796a39e02d02.0
total 48
lrwxrwxrwx 5 root root 7 Jun 22 08:01 bin -> usr/bin
drwxr-xr-x 2 root root 4096 Jan 1 1970 boot
drwxr-xr-x 2 root root 4096 Jan 1 1970 dev
drwxr-xr-x 115 root root 4096 Jul 6 17:07 etc
lrwxrwxrwx 3 root root 9 Jun 22 08:01 home -> /var/home
lrwxrwxrwx 5 root root 7 Jun 22 08:01 lib -> usr/lib
lrwxrwxrwx 5 root root 9 Jun 22 08:01 lib64 -> usr/lib64
lrwxrwxrwx 3 root root 10 Jun 22 08:01 media -> /run/media
drwxr-xr-x 2 root root 4096 Jan 1 1970 mnt
lrwxrwxrwx 3 root root 8 Jun 22 08:01 opt -> /var/opt
lrwxrwxrwx 3 root root 15 Jun 22 08:03 ostree -> /sysroot/ostree
drwxr-xr-x 2 root root 4096 Jan 1 1970 proc
lrwxrwxrwx 3 root root 13 Jun 22 08:06 root -> /var/roothome
drwxr-xr-x 9 root root 4096 Jan 1 1970 run
lrwxrwxrwx 5 root root 8 Jun 22 08:03 sbin -> usr/sbin
drwxr-xr-x 2 root root 4096 Jan 1 1970 srv
drwxr-xr-x 2 root root 4096 Jan 1 1970 sys
drwxr-xr-x 2 root root 4096 Jan 1 1970 sysroot
drwxrwxrwt 2 root root 4096 Jan 1 1970 tmp
drwxr-xr-x 12 root root 4096 Jan 1 1970 usr
drwxr-xr-x 11 root root 4096 Jan 1 1970 var
Filesystem Hierarchy Standard (FHS)

8. Thanks to Carl’s talk yesterday:
rootfs 淺談

9. Bootloader Configuration from OSTree
$ cat /boot/loader/uEnv.txt
kernel_image=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f94
acea/vmlinuz-5.6.0-7-generic
ramdisk_image=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f9
4acea/initramfs-5.6.0-7-generic.img
fdt_file=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f94acea/d
evicetree-5.6.0-7-generic
bootargs=rw splash plymouth.ignore-serial-consoles quiet loglevel=0
ostree=/ostree/boot.0/eos/e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f94a
cea/0
$ ls /boot/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f94acea
devicetree-5.6.0-7-generic initramfs-5.6.0-7-generic.img vmlinuz-5.6.0-7-generic

10. Start from Raspberry Pi 3
We started to enable Endless OS on Raspberry Pi 3 (RPi 3) first.
Key points:
● We want 64 bits system
● U-boot bootloader (RPi 3 & 4 ready)
● Linux mainline kernel including Device Tree
○ RPi 3 ready
○ RPi 4 boots up, but many parts are still under development
● OSTree
○ Generic for arm64 platforms

11. Boot Raspberry Pi 3
Understanding the Raspberry Pi Boot Process (Raspbian/Raspberry Pi
OS (Official), Arch Linux ARM, Fedora ARM, Ubuntu)
● Understanding the Raspberry Pi Boot Process
● The boot folder
● Boot sequence
1st stage
Boot from GPU with
firmware in ROM
and loads
bootcode.bin in FAT
2nd stage
Bootcode.bin
loads start.elf ...
3rd stage
Start.elf loads
DT, kernel …
and starts CPU
4th stage
Kernel boots
FAT boot partition
EXT4
partition
Raspbian for example

12. Device Tree
Description from devicetree.org
● The devicetree is a data structure for describing hardware.
● Rather than hard coding every detail of a device into an operating
system, many aspects of the hardware can be described in a data
structure that is passed to the operating system at boot time.
● The devicetree is used by OpenFirmware (OF)
Important events:
● Re: [GIT PULL] omap changes for v2.6.39 merge window on LKML
● Device trees I: Are we having fun yet? on LWN.net

13. Config
Data
Code

14. Config Data
Code
Device Tree
OF

15. drivers/bluetooth/hci_bcm.c
static int bcm_of_probe(struct bcm_device *bdev)
{
device_property_read_u32(bdev->dev, "max-speed", &bdev->oper_speed);
device_property_read_u8_array(bdev->dev, "brcm,bt-pcm-int-params",
bdev->pcm_int_params, 5);
bdev->irq = of_irq_get_byname(bdev->dev->of_node, "host-wakeup");
return 0;
}
…
static const struct of_device_id bcm_bluetooth_of_match[] = {
{ .compatible = "brcm,bcm20702a1" },
{ .compatible = "brcm,bcm4329-bt" },
{ .compatible = "brcm,bcm4345c5" },
{ .compatible = "brcm,bcm4330-bt" },
{ .compatible = "brcm,bcm43438-bt", .data = &bcm43438_device_data },
{ .compatible = "brcm,bcm43540-bt", .data = &bcm4354_device_data },
{ .compatible = "brcm,bcm4335a0" },
{ },
};
MODULE_DEVICE_TABLE(of, bcm_bluetooth_of_match);
arch/arm/boot/dts/bcm2711-rpi-4-b.dts
bluetooth {
compatible = "brcm,bcm43438-bt";
max-speed = <2000000>;
shutdown-gpios = <&expgpio 0 GPIO_ACTIVE_HIGH>;
};

16. Device Tree Blobs are Deployed like
$ ls /usr/lib/firmware/5.6.0-7-generic/device-tree/
actions amlogic broadcom intel qcom sprd
allwinner apm cavium lg realtek synaptics
altera arm freescale marvell renesas ti
amd bitmain hisilicon mediatek rockchip xilinx
$ ls /usr/lib/firmware/5.6.0-7-generic/device-tree/broadcom/
bcm2711-rpi-4-b.dtb bcm2837-rpi-3-b-plus.dtb stingray
bcm2837-rpi-3-a-plus.dtb bcm2837-rpi-cm3-io3.dtb
bcm2837-rpi-3-b.dtb northstar2

17. Single Device Tree? FIT for Generic arm64 OSTree
OSTree can deploy only one device tree file originally, so we gather all of
the boards’ DTBs into a single FIT-uImage as the deployment target.
/{
description = "Device Tree List";
#address-cells = <1>;
images {
qcom_msm8994-angler-rev-101.dtb {
description = "qcom_msm8994-angler-rev-101.dtb";
data = /incbin/("/boot/dtb/qcom/msm8994-angler-rev-101.dtb");
type = "flat_dt";
arch = "arm64";
compression = "none";
};
qcom_apq8096-db820c.dtb {
...
};
};
Make U-Boot load the FIT-uImage, then pick the designated device tree
in the FIT-uImage.

18. OSTree Supports Device Tree Directory Now
● OSTree can deploy only one device tree file originally
fdt_file=/ostree/eos-8e2b71f01721f41d806e96d158390aa8a52c0f2ddb
dfe2f6e7f043ce9ec3a2d7/devicetree-5.3.0-12-generic
● After opened the feature request ticket, OSTree can support Device
Tree directory deployment now! commit 5f08649f51fe (“deploy:
support devicetree directory”)
fdtdir=/ostree/eos-8a4f394fb3b8d81f37e1fdb62376335955fc807dc021
c9af6c0d59fa96a21dae/dtb
We have two ways to deploy the Device Trees now:
1. A single FIT-uimage → Endless OS chooses this way currently
2. Device Tree directory.

19. $ ls /boot/ostree/eos-8a4f394fb3b8d81f37e1fdb62376335955fc807dc021c9af6c0d59fa96a21dae/dtb/*
/boot/ostree/eos-8a4f394fb3b8d81f37e1fdb62376335955fc807dc021c9af6c0d59fa96a21dae/dtb/amlogic:
meson-a1-ad401.dtb meson-gxbb-p201.dtb meson-gxl-s905x-hwacom-amazetv.dtb
meson-axg-s400.dtb meson-gxbb-vega-s95-meta.dtb meson-gxl-s905x-khadas-vim.dtb
meson-g12a-sei510.dtb meson-gxbb-vega-s95-pro.dtb meson-gxl-s905x-libretech-cc.dtb
meson-g12a-u200.dtb meson-gxbb-vega-s95-telos.dtb meson-gxl-s905x-nexbox-a95x.dtb
meson-g12a-x96-max.dtb meson-gxbb-wetek-hub.dtb meson-gxl-s905x-p212.dtb
meson-g12b-a311d-khadas-vim3.dtb meson-gxbb-wetek-play2.dtb meson-gxm-khadas-vim2.dtb
meson-g12b-odroid-n2.dtb meson-gxl-s805x-libretech-ac.dtb meson-gxm-nexbox-a1.dtb
meson-g12b-s922x-khadas-vim3.dtb meson-gxl-s805x-p241.dtb meson-gxm-q200.dtb
meson-g12b-ugoos-am6.dtb meson-gxl-s905d-libretech-pc.dtb meson-gxm-q201.dtb
meson-gxbb-kii-pro.dtb meson-gxl-s905d-p230.dtb meson-gxm-rbox-pro.dtb
meson-gxbb-nanopi-k2.dtb meson-gxl-s905d-p231.dtb meson-gxm-s912-libretech-pc.dtb
meson-gxbb-nexbox-a95x.dtb meson-gxl-s905d-phicomm-n1.dtb meson-gxm-vega-s96.dtb
meson-gxbb-odroidc2.dtb meson-gxl-s905w-p281.dtb meson-sm1-khadas-vim3l.dtb
meson-gxbb-p200.dtb meson-gxl-s905w-tx3-mini.dtb meson-sm1-sei610.dtb
/boot/ostree/eos-8a4f394fb3b8d81f37e1fdb62376335955fc807dc021c9af6c0d59fa96a21dae/dtb/broadcom:
bcm2711-rpi-4-b.dtb bcm2837-rpi-3-b.dtb bcm2837-rpi-cm3-io3.dtb stingray
bcm2837-rpi-3-a-plus.dtb bcm2837-rpi-3-b-plus.dtb northstar2
Example: OSTree Deploys Device Tree Directory

20. 4th stage
Kernel boots
Boot Raspberry Pi 3 with Endless OS
Raspbian’s firmwares
4th stage
U-Boot loads kernel, initramfs and
flattened device tree (FDT) in OSTree
and boots kernel with boot script
Reference:
● Boot loaders in Raspbian
● The boot folder
● Boot options in
config.txt
Note:
Start.elf loads kernel8.img,
which is the entry file for
arm64.
Most of other distributions
make it as the kernel
directly. Endless OS makes
it as U-Boot for some
reasons.
1st stage
Boot from GPU with
firmware in ROM
and loads
bootcode.bin in FAT
2nd stage
Bootcode.bin
loads start.elf ...
3rd stage
Start.elf loads
DT, kernel …
and starts CPU
U-Boot

21. Why U-Boot?
1. Endless OS comes with OSTree which includes kernel, initramfs and device
tree. They will be changed with different commits / revisions.
2. Because of OSTree, the booting will become failed to find the init daemon, if
the kernel image has no related built-in modules and there is no initramfs.
[ 2.939544] ---[ end Kernel panic - not syncing: No working init found. Try passing init=
option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---
[ 13.350160] WARNING: CPU: 0 PID: 10 at kernel/irq_work.c:102
irq_work_queue_on+0xe8/0x108
[ 13.360149] Modules linked in:
[ 13.365009] CPU: 0 PID: 10 Comm: rcu_preempt Not tainted 4.19.76-v8+ #1
3. Raspberry Pi’s firmware loads the Device Tree and passes it to the booting file
entry. However, it is not the Device Tree used by Linux mainline kernel.
So, Endless OS needs U-Boot as the middle step to boot kernel with the initramfs
and designated device tree. We prepared an U-Boot script to load device tree,
initramfs and kernel, then boot with some arguments.

22. U-Boot Tells Us
U-Boot> print
...
board=rpi
board_name=4 Model B
...
boot_prefixes=/ /boot/
...
boot_targets=mmc0 mmc1 pxe dhcp
...
cpu=armv8
...
fdt_addr_r=0x02600000
...
fdtfile=broadcom/bcm2711-rpi-4-b.dtb
initrd_high=ffffffffffffffff
kernel_addr_r=0x00080000
...
ramdisk_addr_r=0x02700000
...
scriptaddr=0x02400000

23. /boot/uEnv.txt Provided by OSTree
kernel_image=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950
071aeb1ff61cb914f94acea/vmlinuz-5.6.0-7-generic
ramdisk_image=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a9
50071aeb1ff61cb914f94acea/initramfs-5.6.0-7-generic.img
fdt_file=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071ae
b1ff61cb914f94acea/devicetree-5.6.0-7-generic
bootargs=rw splash plymouth.ignore-serial-consoles quiet loglevel=0
ostree=/ostree/boot.0/eos/e5ab3e8095af17c26f85814919ea5d55faa36a950
071aeb1ff61cb914f94acea/0

24. /boot/boot.scr
platform_bootargs=cma=256M@512M
echo "Load u-Boot environment ..."
ext4load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} /boot/uEnv.txt
env import -t ${scriptaddr} ${filesize}
echo "Load device tree ..."
fdt_load_addr_r=${ramdisk_addr_r}
ext4load ${devtype} ${devnum}:${distro_bootpart} ${fdt_load_addr_r} /boot/${fdt_file}
fdt addr ${fdt_load_addr_r}
setexpr fdt_notation gsub / _ ${fdtfile}
fdt get addr fdt_start_addr_r /images/${fdt_notation} data
fdt get size fdt_size /images/${fdt_notation} data
fdt move ${fdt_start_addr_r} ${fdt_addr_r} ${fdt_size}
fdt addr ${fdt_addr_r}
echo "Load kernel and unzip it ..."
kernel_load_addr_r=${ramdisk_addr_r}
ext4load ${devtype} ${devnum}:${distro_bootpart} ${kernel_load_addr_r} /boot/${kernel_image}
unzip ${kernel_load_addr_r} ${kernel_addr_r}
echo "Load RAM disk ..."
ext4load ${devtype} ${devnum}:${distro_bootpart} ${ramdisk_addr_r} /boot/${ramdisk_image}
echo "Boot ..."
setenv bootargs root=LABEL=ostree ${bootargs} ${platform_bootargs}
booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}
Extract the DTB from FIT-uImage

25. Storage Layout
ROM on SoC
SD Boot Partition (FAT)
Boot loaders (bootcode.bin, start.elf ...)
u-boot
SD OSTree Partition (ext4)
● /boot/
○ boot.scr
○ uEnv.txt
○ ostree/ … /
■ <kerne>
■ <initramfs>
■ <devicetree>
● /ostree/…
● ...
Power on

26. 4th stage
Kernel boots
Boot Raspberry Pi 4 (new SoC) with Endless OS
Raspbian’s firmwares
4th stage
U-Boot loads kernel, initramfs and
flattened device tree (FDT) in OSTree
and boots kernel with boot script
1st stage
Boot from GPU with firmware in ROM
including bootcode.bin job in Second stage:
loads start4x.elf ... in FAT
3rd stage
Start4.elf loads DT,
kernel …
and starts CPU
U-Boot

27. But U-Boot always boots Linux
kernel failed on Raspberry Pi 4B !???

28. arch/arm/boot/dts/bcm2711-rpi-4-b.dts
/ {
compatible = "raspberrypi,4-model-b", "brcm,bcm2711";
model = "Raspberry Pi 4 Model B";
chosen {
/* 8250 auxiliary UART instead of pl011 */
stdout-path = "serial1:115200n8";
};
/* Will be filled by the bootloader */
memory@0 {
device_type = "memory";
reg = <0 0 0>;
};
...
WHAT ???!

29. [U-Boot] rpi4: fix dram bank initialization (Denied)
● Found the memory → reg must be fixed up for the DRAM bank in
Raspberry Pi 4B’s Device Tree. Raspberry Pi 4B’s memory could be 1,
2, 4 and 8 GB. So, the reg should not be fixed values. The values
must be filled by someone dynamically.
● Raspberry Pi’s boot firmware must know how to fix the reg’s values.
Although, U-Boot does not know the values directly, it can be
launched by Raspberry Pi’s firmware. And, Raspberry Pi’s firmware
already puts the correct values in the memory stack during the
execution of U-Boot. I sent the patch, but it is denied.
● Because, there is already a better and generic way which is good!!!
○ commit 7a3f15e7 ("dm: Fix default address cells return value")
○ commit 8076fc29 ("libfdt: Allow #size-cells of 0")
○ commit 0ba41ce1 ("libfdt: return correct value if #size-cells property is not present")
○ commit ce2dae3a ("libfdt: fdt_address_cells() and fdt_size_cells()")

30. Enable
CONFIG_ARCH_FIXUP_FDT_MEMORY
in U-Boot for Raspberry Pi 4B

32. Reference
● Raspberry Pi 4 Preview Release | Endless OS 3.7.7~beta1
● Upstream Raspberry Pi 4 B support
● Raspberry Pi → Documentation → The boot folder
● RPi U-Boot
● OSTree
● Device Tree
● U-Boot