Describe the log buffer and consoles in Liunx kernel on openSUSE

1. Kernel debug log and
console on openSUSE
August, 2019, Taipei, COSCUP
Joey Lee
SUSE Labs Taipei

2. 2
Agenda
• Kernel log
• Consoles
• Q&A

3. 3
Kernel log buffer and consoles

4. Kernel log

5. 5
Kernel log

6. 6
Kernel log buffer
• SLE15 / openSUSE Leap 15.1
‒ CONFIG_LOG_BUF_SHIFT=18
‒ CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
‒ 2^18 = 256KB
• log_buf_len=n[KMG]
‒ Sets the size of the printk ring buffer,
‒ in bytes. n must be a power of two and greater than the
minimal size. The minimal size is defined by
LOG_BUF_SHIFT kernel config parameter. There is also
CONFIG_LOG_CPU_MAX_BUF_SHIFT config parameter
that allows to increase the default size depending on the
number of CPUs. See init/Kconfig for more details. [1]

7. 7
Kernel log buffer (cont.)
• The kmsg is an ring buffer
static char __log_buf[__LOG_BUF_LEN]
__aligned(LOG_ALIGN);
• Protected by spinlock
DEFINE_RAW_SPINLOCK(logbuf_lock);
• /dev/kmsg
[11] = { "kmsg", 0644, &kmsg_fops, 0 },
• dmesg utility accesses /dev/kmsg

8. 8
dmesg
• Print or control the kernel ring buffer
• Default print all /dev/kmsg messages
• Restrict message level
‒ dmesg -l err,warn
‒ dmesg -l info,debug
• Set console level
‒ dmesg -n 8
• Wait for new messages
‒ dmesg -w
• Clear ring buffer
‒ sudo dmesg -C

9. 9
loglevel
• loglevel=0..7
• Console loglevel
• All Kernel Messages with a loglevel smaller than the
console loglevel will be printed to the console.
suppress_message_printing(msg→level)
static bool suppress_message_printing(int level)
{
return (level >= console_loglevel && !ignore_loglevel);
}

10. 10
loglevel defined
• The loglevels are defined as follows:
0 (KERN_EMERG) system is unusable
1 (KERN_ALERT) action must be taken immediately
2 (KERN_CRIT) critical conditions
3 (KERN_ERR) error conditions
4 (KERN_WARNING) warning conditions [quiet]
5 (KERN_NOTICE) normal but significant conditions
6 (KERN_INFO) informational
7 (KERN_DEBUG) debug-level messages [default]
10 (CONSOLE_LOGLEVEL_DEBUG)

11. 11
Kernel parameters for loglevel
• ignore_loglevel
‒ Ignore loglevel setting - this will print /all/ kernel messages to
the console. Useful for debugging. We also add it as printk
module parameter, so users could change it dynamically,
usually by
/sys/module/printk/parameters/ignore_loglevel. [1]
• debug
‒ [KNL] Enable kernel debugging (events log level).
‒ console_loglevel = CONSOLE_LOGLEVEL_DEBUG; // 10
• quiet
‒ [KNL] Disable most log messages
‒ console_loglevel = CONSOLE_LOGLEVEL_QUIET; // 4

12. 12
rsyslog
• The rsyslog overwrites loglevel when booting
• systemctl status rsyslog
• /etc/rsyslog.conf
# set log level 1 (same as in /etc/sysconfig/syslog).
$klogConsoleLogLevel 1
• # cat /proc/sys/kernel/printk
1 4 1 7

13. 13
printk sysfs
• cat /proc/sys/kernel/printk
1 4 1 7
• console_loglevel: messages with a higher priority
than this will be printed to the console
• default_message_loglevel: messages without an
explicit priority will be printed with this priority
• minimum_console_loglevel: minimum (highest)
value to which console_loglevel can be set
• default_console_loglevel: default value for
console_loglevel

14. 14
Default loglevel of SLE/openSUSE
• Kernel config: loglevel=7
‒ CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
• Grub2 config (grub.cfg): loglevel=4
‒ Kernel parameter: quiet
‒ GRUB_CMDLINE_LINUX_DEFAULT="resume=/dev/sda1
splash=silent quiet showopts"
• rsyslog config (/etc/rsyslog.conf): loglevel=1
‒ $klogConsoleLogLevel 1
• Conclusion
‒ Before rsyslog start: console_loglevel = 4
‒ After rsyslog start: console_loglevel = 1
‒ Please add loglevel=9 or debug, and also “dmesg -n 8” for debugging

15. 15
Dynamic debug
• ddebug_query
‒ [KNL,DYNAMIC_DEBUG] Enable debug messages at
early boot time. See Documentation/dynamic-debug-
howto.txt for details. [1]
‒ Dynamic debug is designed to allow you to dynamically
enable/disable kernel code to obtain additional kernel
information. Currently, if ``CONFIG_DYNAMIC_DEBUG``
is set, then all ``pr_debug()``/``dev_dbg()`` and
``print_hex_dump_debug()``/``print_hex_dump_bytes()``
calls can be dynamically enabled per-callsite. [2]

16. 16
Dynamic debug (cont.)
• Kernel parameter:
'ddebug_query=file drivers/firmware/efi/efi-secret-key.c +p'
'ddebug_query=file kernel/module_signing.c +p; file
crypto/asymmetric_keys/pkcs7_trust.c +p'
• Userland command:
# echo -n 'file drivers/base/firmware_class.c +p' >
/sys/kernel/debug/dynamic_debug/control
# cat /sys/kernel/debug/dynamic_debug/control

17. 17
pr_debug() / dev_dbg()
#if defined(CONFIG_DYNAMIC_DEBUG)
#include <linux/dynamic_debug.h>
#define pr_debug(fmt, ...)
dynamic_pr_debug(fmt, ##__VA_ARGS__)
#elif defined(DEBUG)
#define pr_debug(fmt, ...)
printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)
#else
#define pr_debug(fmt, ...)
no_printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)
#endif

18. Consoles

19. 19
Consoles

20. 20
earlyprintk
• Serial port: earlyprintk=ttyS0,115200
‒ EFI framebuffer: earlyprintk=efi
‒ VGA: earlyprintk=vga
‒ USB2 debug port: earlyprintk=dbgp
‒ USB3 debug port: earlyprintk=xdbc
• earlyprintk is useful when the kernel crashes before the
normal console is initialized. It is not enabled by default
because it has some cosmetic problems.
‒ It’s a OLD parameter in kernel
• Only one of vga, efi, serial, or usb debug port can be used
at a time.
• early_printk() callers: e820, kaslr, mm(idt), perf

21. 21
earlycon
• Output early console device and options.
• CONFIG_SERIAL_EARLYCON
‒ v3.16 kernel
‒ Support for early consoles with the earlycon parameter. This enables the console
before standard serial driver is probed. The console is enabled when early_param is
processed.
• earlycon
‒ [X86] ACPI SPCR (Serial Port Console Redirection Table)
‒ [ARM64] Device Tree
• earlycon=uart[8250],io,<addr>[,options]
=uart[8250],mmio,<addr>[,options]
...
‒ earlycon=uart,io,0x3f8,115200
• EARLYCON_DECLARE() or OF_EARLYCON_DECLARE() in kernel
‒ Set initial function to __earlycon_table section

22. 22
console
• Output console device and options.
• tty<n>
ttyS<n>[,options]
ttyUSB0[,options]
‒ Serial: console=ttyS0,115200n8
‒ USB: console=ttyUSB0,115200n8
‒ ARM: console=ttyAMA0,115200n8
• Maximum 8 consoles
‒ console_cmdline[MAX_CMDLINECONSOLES]
‒ #define MAX_CMDLINECONSOLES 8

23. 23
Reading messages from consoles
• screen command
‒ # screen /dev/ttyS0 115200
‒ # screen -h 10000 /dev/ttyUSB1 115200
• minicom
‒ # minicom -b 115200 -D /dev/ttyUSB0

24. 24
When can earlcon be enabled?
No console before here

25. 25
The debug log in early booting stage
• For booting
‒ In EFI stub, the efi_printk() log is only for expected error.
‒ In kernel, earlyprintk= or earlycon= only be enabled after
parse_early_param().
‒ “KASLR using” message should be exposed with
earlyprintk= or earlycon=
‒ KASLR (Kernel Address Space Layout Randomization)
• For problems that occur before the serial console
and EFI framebuffer are initialized [4]
‒ Debugging a mysterious reset with an infinite loop
‒ Debugging a hang by triggering a reset

26. 26
DEBUG_LL and ARM
• Kernel low-level debugging functions
• CONFIG_DEBUG_LL
‒ ARM, Unicore32
‒ Say Y here to include definitions of printascii, printch, printhex in the kernel.
This is helpful if you are debugging code that executes before the console is
initialized.
‒ Note that selecting this option will limit the kernel to a single UART definition,
… this option should not be enabled for kernels that are intended to be
portable.
‒ CONFIG_DEBUG_BCM2835 for Raspberry Pi 1
‒ DEBUG_UART_PHYS = 0x20201000, DEBUG_UART_VIRT = 0xf0201000
‒ CONFIG_DEBUG_BCM2836 for Raspberry Pi 2
‒ DEBUG_UART_PHYS = 0x3f201000, DEBUG_UART_VIRT = 0xf0201000
• Should works with CONFIG_EARLY_PRINTK
‒ Enable by earlyprintk kernel parameter

27. 27
netconsole
• This module logs kernel printk messages over UDP
allowing debugging of problem where disk logging fails
and serial consoles are impractical. [3]
• As a built-in, netconsole initializes immediately after NIC
cards and will bring up the specified interface as soon as
possible. While this doesn't allow capture of early kernel
panics, it does capture most of the boot process. [3]
• CONFIG_NETCONSOLE=y
‒ NIC driver must also be build-in
• CONFIG_NETCONSOLE=m
‒ /etc/modules-load.d/netconsole.conf
‒ /etc/modprobe.d/netconsole.conf

28. 28
netconsole (cont.)
• Sender: netconsole=[+][src-port]@[src-ip]/[<dev>],
[tgt-port]@<tgt-ip>/[tgt-macaddr]
netconsole=6665@192.168.1.83/eth0,6666@192.168.1.11
/ab:cd:23:b0:14:b6
netconsole=@192.168.1.83/,@192.168.1.11/
• Receiver: netcat
netcat -k -l -u 192.168.1.11 6666

29. Q&A

30. 30
Reference
• [1] Documentation/admin-guide/kernel-
parameters.txt, Linux Kernel
• [2] Documentation/admin-guide/dynamic-debug-
howto.rst, Linux Kernel
• [3] Documentation/networking/netconsole.txt, Linux
Kernel
• [4] Early x86 Linux boot debug tricks, Matt Fleming
‒ http://www.codeblueprint.co.uk/2015/04/15/early-x86-linux-
boot-debug-tricks.html

31. Thank you.
31
Feedback to
jlee@suse.com

33. Corporate Headquarters
Maxfeldstrasse 5
90409 Nuremberg
Germany
+49 911 740 53 0 (Worldwide)
www.suse.com
Join us on:
www.opensuse.org
33

34. Unpublished Work of SUSE. All Rights Reserved.
This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE.
Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of
their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated,
abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE.
Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General Disclaimer
This document is not to be construed as a promise by any participating company to develop, deliver, or market a
product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making
purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document,
and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose.
The development, release, and timing of features or functionality described for SUSE products remains at the sole
discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at
any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in
this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All
third-party trademarks are the property of their respective owners.