Optimize Linux Boot Times

Andrea Righi - andrea@betterlinux.com
Ottimizzare i tempi di boot di Linux

Agenda
● Overview
● Case study: Raspberry Pi
● Kernel optimizations
● rootfs optimizations
● Q/A

Overview of boot phases
●
Firmware (boot-loader)
● Hardware probing
● Hardware initialization
● Kernel load and decompression
●
Kernel execution
● Core init (start_kernel)
● Driver init (initcalls)
●
User-space init
● /sbin/init
● RC scripts
●
Application start (first user impression)

Generic concepts
● Identify boot time functionalities
● Measure boot time of each functionality
● Remove unnecessary functionalities
● Optimize required functions
● Defer loading of less-important features
(modularization - LKM)
● Re-order initialization / parallelization
● Asynchronous initialization

Generic concepts
● Identify boot time functionalities
● Measure boot time of each functionality
● Remove unnecessary functionalities
● Optimize required functions
● Defer loading of less-important features
(modularization - LKM)
● Re-order initialization / parallelization
● Asynchronous initialization
Premature optimization is the root of all evil.
-Donald Knuth

Optimization
● Remove features
● The fastest code is the code you don't run!
● Defer loading of less important features
● Modularization (LKM)
● Parallelization
● async initialization (kernel/async.c)
● Reduce probing delays
● Filesystem tricks

Case study
● Raspberry Pi
● Boot time functionality
● Reponsive ssh login
● Tools and techniques
used to improve boot
time
● Focusing on cold-
boot optimizations

Boot steps
● RPi boot steps:
● bootcode.bin: 1st-stage boot loader from SoC
BCM2835 ROM
● bootloader.bin: 2nd-stage boot loader from external
SD card
● kernel.img: Linux
● rootfs: external SD card
● application: ssh (dropbear)

Boot steps
● Rpi boot steps:
● bootcode.bin: 1st-stage boot loader from SoC
BCM2835 ROM
● bootloader.bin: 2nd-stage boot loader from external
SD card
● kernel.img: Linux
● rootfs: external SD card
● application: ssh (dropbear)

Measure boot-time functionality
(kernel)
● CONFIG_PRINTK_TIME
● Configure it in “Kernel hacking” section
● Adds timing informations to kernel messages
(dmesg)
# dmesg
...
[ 0.022030] initcall bcm_mbox_init+0x0/0x38 returned 0 after 0 usecs
[ 0.022054] calling bcm_power_init+0x0/0xa4 @ 1
[ 0.022065] bcm_power: Broadcom power driver
[ 0.022080] bcm_power_open() -> 0
[ 0.022090] bcm_power_request(0, 8)
[ 0.522766] bcm_mailbox_read -> 00000080, 0
[ 0.522783] bcm_power_request -> 0
[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs

Kernel initcall tracer
● Introduced in Linux 2.6.28
● Add “initcall_debug” to the kernel boot options
● Allows to record the timings of each initcall in
dmesg
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8 | tail -5
[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs
[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs
[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs
[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs
[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs

Kernel initcall tracer (example)
● dmesg > dmesg.log
● cat dmesg.log | perl
scripts/bootgraph.pl >
/boot/dmesg.svg

Raspbian
● http://www.raspbian.org
● General-purpose distro based on Debian
● Optimized for the Raspberry Pi
● compilation settings adjusted to produce hard-float
code
● 2013-05-25-wheezy-raspbian.img

Raspbian: boot time
# dmesg
...
[ 3.107163] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1,
smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24
[ 5.664780] EXT4-fs (mmcblk0p2): recovery complete
[ 5.677326] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 5.689427] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 5.700523] devtmpfs: mounted
[ 5.706028] Freeing init memory: 128K
[ 6.257687] init start
[ 7.337703] udevd[154]: starting version 175
[ 8.511002] Registered led device: led0
[ 16.041293] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 25.553157] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
[ 28.672178] Adding 102396k swap on /var/swap. Priority:-1 extents:2 across:507900k SS
[ 30.436046] init done
- Shell after ~6.3sec
- SSH after ~30.5sec

sysv-rc-conf
● Run-level configuration for SysV like init scripts
● disable npt
● disable plymouth
● disable rsync
● disable x11-common and lightdm
● disable alsa-utils
● disable swap (dphys-swapfile)
● disable checkfs
● disable ntp

Static IP vs DHCP
● Assign a static IP to the board (save the time to
get an IP via DHCP)
● Disable CONFIG_IP_PNP in the kernel .config
● Used to mount rootfs via NFS

Raspbian: boot time after simple
optimizations
# dmesg
...
[ 3.132731] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1,
smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24
[ 5.730131] EXT4-fs (mmcblk0p2): recovery complete
[ 5.742783] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 5.754881] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 5.765958] devtmpfs: mounted
[ 7.389321] udevd[154]: starting version 175
[ 8.536932] calling leds_init+0x0/0x60 [led_class] @ 229
[ 8.537065] initcall leds_init+0x0/0x60 [led_class] returned 0 after 82 usecs
[ 8.543802] calling gpio_led_driver_init+0x0/0xc [leds_gpio] @ 229
[ 8.544067] Registered led device: led0
[ 8.545242] initcall gpio_led_driver_init+0x0/0xc [leds_gpio] returned 0 after 1344 usecs
[ 24.493346] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
[ 24.991030] init done
About 25 sec to login via ssh, instead of 30 sec (16% faster)

Build a minimal distro from scratch:
requirements
● build toolchain (gcc, glibc, binutils)
● git://github.com/raspberrypi/tools.git
● HINT: use hard-float toolchain (arm-bcm2708hardfp-linux-
gnueabi)
● Linux (kernel)
● git://github.com/raspberrypi/linux.git
● busybox (rootfs)
● git://busybox.net/busybox.git
● dropbear (ssh server)
● https://matt.ucc.asn.au/dropbear/

Kernel initcalls (before optimization)
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8
...
[ 0.810927] initcall iscsi_transport_init+0x0/0x154 returned 0 after 446 usecs
[ 1.225990] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 451 usecs
[ 1.232472] initcall pm_qos_power_init+0x0/0xac returned 0 after 713 usecs
[ 1.230686] initcall sysctl_ipv4_init+0x0/0x98 returned 0 after 755 usecs
[ 0.810406] initcall vchiq_init+0x0/0x1dc returned 0 after 1565 usecs
[ 1.228797] initcall sdhci_drv_init+0x0/0xc returned 0 after 1829 usecs
[ 0.560116] initcall inet_init+0x0/0x260 returned 0 after 2739 usecs
[ 0.808747] initcall loop_init+0x0/0x11c returned 0 after 4852 usecs
[ 0.803722] initcall brd_init+0x0/0x1c0 returned 0 after 8230 usecs
[ 0.540586] initcall genhd_device_init+0x0/0x84 returned 0 after 9765 usecs
[ 0.556774] initcall chr_dev_init+0x0/0xd8 returned 0 after 12234 usecs
[ 0.538982] initcall param_sysfs_init+0x0/0x1d4 returned 0 after 19531 usecs
[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs
[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs
[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs
[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs
[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs

Kernel optimizations
(initcalls)
● disable kprobes (CONFIG_KPROBES): ~95ms
● reduce the number of PTYs: ~90ms
● CONFIG_LEGACY_PTY_COUNT=256 =>
CONFIG_LEGACY_PTY_COUNT=1
● disable framebuffer: ~33ms
● remove loop device: ~5ms

Kernel initcalls (after optimization)
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8
...
[ 0.570870] initcall tun_init+0x0/0x8c returned 0 after 259 usecs
[ 0.568035] initcall des_generic_mod_init+0x0/0x10 returned 0 after 341 usecs
[ 0.975476] initcall bcm2835_cpufreq_module_init+0x0/0xc returned 0 after 358 usecs
[ 0.569678] initcall pty_init+0x0/0x194 returned 0 after 375 usecs
[ 0.561542] initcall vc_mem_init+0x0/0x1b4 returned 0 after 393 usecs
[ 0.974729] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 393 usecs
[ 1.018751] initcall deferred_probe_initcall+0x0/0x6c returned 0 after 395 usecs
[ 0.561078] initcall bcm2708_gpio_init+0x0/0xc returned 0 after 405 usecs
[ 1.018028] initcall pm_qos_power_init+0x0/0x60 returned 0 after 506 usecs
[ 0.559402] initcall inet_init+0x0/0x260 returned 0 after 1540 usecs
[ 1.021006] initcall net_secret_init+0x0/0x1c returned 0 after 2145 usecs
[ 1.024343] initcall initialize_hashrnd+0x0/0x1c returned 0 after 3052 usecs
[ 0.557293] initcall chr_dev_init+0x0/0xd8 returned 0 after 10978 usecs
[ 0.541305] initcall param_sysfs_init+0x0/0x19c returned 0 after 19531 usecs
[ 1.016084] initcall sdhci_drv_init+0x0/0xc returned 0 after 39250 usecs
[ 0.973739] initcall dwc_otg_driver_init+0x0/0xc4 returned 0 after 392536 usecs
[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs

Kernel optimizations
(other optimizations)
● preset loops_per_jiffy
● At each boot the kernel calibrates a delay loop (used later by the udelay()
function)
● 1 jiffy = time between 2 timer interrupts
● CONFIG_HZ=100 => 250ms!!!
● loop
● disable console output
● remove “console=xxx” and add “quiet” to the kernel boot parameters
● use LZO kernel decompression (CONFIG_KERNEL_LZO)
● LZO is a compression algorithm that is much faster than gzip, at the cost of a
slightly degrade compression ratio (+10%)
● reduce kernel size...
● A smaller kernel is faster to load, less code also means smaller working set
(good for caches)

Kernel image (before => after)

rootfs
● Generated using busybox and dropbear
● Hints about busybox (reduce fork()s):
● CONFIG_FEATURE_SH_STANDALONE: use applets instead of fork/exec/wait
external binaries
● CONFIG_FEATURE_SH_NOFORK: call <applet>_main() directly without
spawning another task
● Build everything with -Os
● Use mklibs to strip system libraries (rootfs is mounted to /mnt)
● mklibs -v -D -d lib2 -L /mnt/lib --ldlib lib/ld-linux.so.3 --target=arm-bcm2708-
linux-gnueabi /mnt/bin/* /mnt/sbin/* /mnt/usr/sbin/* /mnt/usr/bin/*; rm -rf lib; mv
lib2 lib
● After all these stops total rootfs size is ~3.2MB

filesystem optimizations
● Split the filesystem into read-only portion and
read/write portion
● Read-only filesystem mounts faster
● Use Squashfs + tmpfs (or aufs

root filesystem: boot time

Kernel image (before => after)
1.5GB
3.2MB

Demo: custom kernel+rootfs boot time
...
[ 1.249067] waiting eth0 to come up
[ 1.345882] usb 1-1: new high-speed USB device number 2 using dwc_otg
[ 1.346059] Indeed it is in host mode hprt0 = 00001101
[ 1.556942] hub 1-1:1.0: USB hub found
[ 1.557074] hub 1-1:1.0: 3 ports detected
[ 1.836029] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[ 1.940066] smsc95xx v1.0.4
[ 1.951858] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx
USB 2.0 Ethernet, b8:27:eb:a9:d6:24
[ 2.086695] init done
[ 3.406004] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
- Shell after ~1.2 sec
- SSH after ~3.4 sec

Conclusion
● Why a faster boot?
● Consumer electronics products require very fast
boot times (digital camera, mobile phone, etc.)
● Fast recovery upon system failures (crashes)
● You can show to your friends your new awesome
board booting in a couple of seconds ;-)

Optimize Linux Boot Times

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Optimize Linux Boot Times

Similar to Optimize Linux Boot Times (20)

More from Andrea Righi

More from Andrea Righi (6)

Recently uploaded

Recently uploaded (20)

Optimize Linux Boot Times