COSCUP 2020 Talk - 1-Aug-2020

1. Linux Integrity Mechanisms -
Protecting Container Runtime as
an Example
Clay Chang
clay.chang@gmail.com
1-Aug-2020, COSCUP’20

2. Who Am I?
• Master Technologist (Linux) at Hewlett Packard Enterprise (HPE)
• Researcher at NTU
• COSCUP’17 - Memory-Style Storage in Linux
• COSCUP’19 - Remote Memory
This talk is a view of my own!

3. Contents
• Introduction to Container Technology
• Motivating Example - CVE-2019-5736
• Background of Linux Integrity Measurement Architecture (IMA)
• Protecting Container Runtime Executables with IMA

4. Container Technology
Docker Engine
(dockerd)
Podman
containerd
runc
Docker CLI Podman CLI
CRI-O
kubelet

5. CVE-2019-5736 aka Container Escape
• Description: … allows attackers to overwrite the host runc binary
(and consequently obtain host root access) by leveraging the ability
to execute a command as root within one of these types of
containers: (1) a new container with an attacker-controlled image, or
(2) an existing container, to which the attacker previously had write
access, that can be attached with docker exec. This occurs
because of file-descriptor mishandling, related to /proc/self/exe.
https://blog.dragonsector.pl/2019/02/cve-2019-5736-escape-from-docker-and.html

6. # git clone https://github.com/q3k/cve-2019-5736-poc
# docker build -t cve .
# strings /usr/bin/runc | tail -n 2
.gnu_debuglink
.gnu_debugdata
# docker run cve
Emulate Docker CLI using podman. Create /etc/containers/
nodocker to quiet msg.
HAX2: argv: /proc/self/fd/3
HAX2: fd: 4
HAX2: res: 13, 0
# strings /usr/bin/runc | tail -n 2
.debug_gdb_scripts
cve-2019-5736

7. Integrity Measurement Architecture
(IMA)
• Since v2.6.30 (9 Jun 2009)
• Key Features of IMA
• Detect if files have been altered accidentally or maliciously, both
remotely or locally
• Appraise a file’s measurement against a “good” value stored as
an extended attributes
• Enforce local file integrity

8. IMA Characteristics
• Policy-based rules
• Actions: measure / dont_measure / appraise / dont_appraise / …
• Conditions:
• File-based operations: BPRM_CHECK / MMAP_CHECK / FILE_CHECK / …
• Masks: MAY_EXEC / MAY_READ / MAY_WRITE / …
• Specific fsmagic / fsuuid / uid / …
• Supports LSM (Linux Security Module, e.g. SELinux): obj_type / obj_user / obj_role / …
• Options: appraise_type / keyring / …

9. # PROC_SUPER_MAGIC
dont_measure fsmagic=0x9fa0
dont_appraise fsmagic=0x9fa0
# SYSFS_MAGIC
dont_measure fsmagic=0x62656572
dont_appraise fsmagic=0x62656572
# DEBUGFS_MAGIC
dont_measure fsmagic=0x64626720
dont_appraise fsmagic=0x64626720
# TMPFS_MAGIC
dont_measure fsmagic=0x01021994
dont_appraise fsmagic=0x01021994
# RAMFS_MAGIC
dont_appraise fsmagic=0x858458f6
# DEVPTS_SUPER_MAGIC
dont_measure fsmagic=0x1cd1
dont_appraise fsmagic=0x1cd1
# BINFMTFS_MAGIC
dont_measure fsmagic=0x42494e4d
dont_appraise fsmagic=0x42494e4d
# SECURITYFS_MAGIC
dont_measure fsmagic=0x73636673
dont_appraise fsmagic=0x73636673
# SELINUX_MAGIC
dont_measure fsmagic=0xf97cff8c
dont_appraise fsmagic=0xf97cff8c
# CGROUP_SUPER_MAGIC
dont_measure fsmagic=0x27e0eb
dont_appraise fsmagic=0x27e0eb
# NSFS_MAGIC
dont_measure fsmagic=0x6e736673
dont_appraise fsmagic=0x6e736673
measure func=BPRM_CHECK
measure func=FILE_MMAP mask=MAY_EXEC
measure func=FILE_CHECK mask=MAY_READ uid=0
appraise fowner=0
Measure all executables in
bprm_check
3
1
2
4
1
Measure all files mapped
executable in file_mmap
2
Measure all files open for read
by root in do_filp_open
3
Appraise all files owned by root4
IMA Policy
Example

10. # cat policy > /sys/kernel/security/ima/policy
# cat /sys/kernel/security/ima/ascii_runtime_measurements
-bash: /usr/bin/cat: Permission denied
# dmesg
-bash: /usr/bin/dmesg: Permission denied
# ls
-bash: /usr/bin/ls: Permission denied
# reboot
-bash: /usr/bin/ls: Permission denied

11. # cat policy2 > /sys/kernel/security/ima/policy
# cat /sys/kernel/security/ima/ascii_runtime_measurements
[root@goldenarmor ima-policy]# cat /sys/kernel/security/ima/ascii_runtime_measurements
10 1d8d532d463c9f8c205d0df7787669a85f93e260 ima-ng sha1:0000000000000000000000000000000000000000 boot_aggregate
10 0000000000000000000000000000000000000000 ima-ng sha1:0000000000000000000000000000000000000000 /tmp/sh-thd.5cBnGN
10 d8ad76de60ab2242cf08bdc1b8bfd0ede36bc983 ima-ng sha1:a8c05f516f368a0654e9e3ae3b3217b6dd40e491 /tmp/sh-thd.5cBnGN
10 0000000000000000000000000000000000000000 ima-ng sha1:0000000000000000000000000000000000000000 /tmp/sh-thd.ccEXk2
10 fb5a5e532147ef93b494ae391022620cf8e2c78b ima-ng sha1:a8c05f516f368a0654e9e3ae3b3217b6dd40e491 /tmp/sh-thd.ccEXk2
10 37a4d23f6ba7f2d5c13747f463f9ff1a2f155be8 ima-ng sha1:359819795e404a69d8a4a52dddccdf2bf1d3859e /usr/bin/gawk
10 a791d8db0076287a372053cbf0882feeea70b57e ima-ng sha1:cf13787cf9f3961d8c94eee05b75727c862cc986 /usr/lib64/ld-2.28.so
10 032d456298ce54f8f4bcd50bd3ddfbe73a16abb7 ima-ng sha1:e337d4fefc5c23e89c57da45bdc48cefed10d547 /etc/ld.so.cache
10 d4bc023942ec7eb0f9093502b7d2234252b2f143 ima-ng sha1:8cb57415f34031b50a7193c6a0176f1a74f2d03c /usr/lib64/libsigsegv.so.2.0.4
10 b7e0667a05d829ac3c3798d9fecc2b8c1921f059 ima-ng sha1:bb4c4b67723c732f12352e5449da1c1f0d2123d7 /usr/lib64/libreadline.so.7.0
10 d085818f8e9e0e7a8d2c24bc2e2b2e5af2d7c426 ima-ng sha1:e028f33771036ece3505a586b4f562992e188996 /usr/lib64/libtinfo.so.6.1
10 fc48f8598aae7723d0c9f7e3035f2eb900d772f0 ima-ng sha1:9403913e901469a4c97219ac9daf2fdb701bb483 /usr/share/locale/locale.alias
10 69f01172cf5cb9c2f6d5c17f3f065f2de924fee7 ima-ng sha1:692a1e79d6e453c99714ec37bb4999abb57f5b44 /usr/lib/locale/en_US.utf8/LC_CTYPE
10 77a0b37e3bc8d8fe55eb1ad3a8fa6f108be73bab ima-ng sha1:a82fb20c382fc9e5dec432ca5ee2f04684536e94 /usr/lib64/gconv/gconv-modules.cache
10 1393afae5534259ab7ad4ee7a759643ccccafc5c ima-ng sha1:aa5248b0ac085665477b2feb1266205b3bed557d /usr/lib/locale/en_US.utf8/LC_ADDRESS
10 eb7c76027b0b21e55693619d884995916d140058 ima-ng sha1:a1a2eabc4a2a2da47d7a90d36b4d0826aac03c73 /usr/lib/locale/en_US.utf8/LC_NAME
...
10 d02d14d189677572ce53d868447b1c9961b9faa6 ima-ng sha1:c5b36af5bfba8854a4209bb6e905c1a7eb47caad /usr/lib64/libidn2.so.0.3.6
10 3477e8cb1563071d5b09695709e7e2d31f567e90 ima-ng sha1:e6dc3965b93be37385e783727fbb0dfdabd4c7c4 /usr/lib64/libattr.so.1.1.2448
10 ce4a6d8b052af0c8d9cff507552d3f5839d9871f ima-ng sha1:47dab9dcb38148a217773302a9384bcfa21ba974 /usr/lib64/libdevmapper.so.1.02
10 f0289ee791a681cf00f512a66351fc295c1d0507 ima-ng sha1:f7259220100a2377ea7fb6ccda3c3037c63debfe /usr/lib64/libssl.so.1.1.1c
10 942fe25b8dd8b66c0f906af71b4c986fb96a67b6 ima-ng sha1:1b533c9e2826d8652e133c979dab4906ec37fdce /usr/lib64/libcrypto.so.1.1.1c
10 0ca2d280e7e3b714caf850203322d7eaa8fae9b8 ima-ng sha1:e6f458475bb286dfdcd98171550fbb0c38917cc6 /usr/lib64/libjson-c.so.4.0.0
10 0e964e27ce222afe78a3e4a030a06ff7eca96048 ima-ng sha1:d3fa4ee3c0ef05ac5b3bf4126facceb7ca59dced /usr/lib64/libpcap.so.1.9.0
10 323e40c5dd45055ece2032058ace3d604e6bc4f8 ima-ng sha1:340c7b865b132a0ef0ea51985435861e347fa2ab /usr/lib64/libunistring.so.2.1.0
10 a042f42bc53ec00abaf8de4efe536b4e50ac3ea9 ima-ng sha1:dec7b77de0dab91f28e9d9f265775c04642d70d5 /usr/lib64/libsepol.so.1
10 8820e6b8ea6d401c8e287b66a8e11bf4b50f435c ima-ng sha1:1802538af67a8306fbf85e12a1a9e18589de475b /usr/lib64/libudev.so.1.6.11
10 7e6a73ee70e9829795945ad3d02d94048cc64f5c ima-ng sha1:442d6bbeb51f089ccb9a0f31e1cf79d9eb8cb42c /usr/lib64/libz.so.1.2.11
10 0000000000000000000000000000000000000000 ima-ng sha1:0000000000000000000000000000000000000000 /tmp/sh-thd.YUyyzn
10 cfb6053df14de2785ef1caf7a77d8e84c7a2406d ima-ng sha1:5f30a9ee5335c8ef0aa17876ddc5d622daec8b15 /tmp/sh-thd.YUyyzn
10 0f60e2884367dee70864927ba04f2e657209355e ima-ng sha1:6207eaf514d465da23059e66759456dfe162349d /usr/bin/cat

12. The Leverage & Solution
• container-selinux: SELinux Policy for Container Runtimes
• New IMA rule:
# ls -Z /usr/bin/runc
system_u:object_r:container_runtime_exec_t:s0 /usr/bin/runc
appraise func=BPRM_CHECK obj_type=container_runtime_exec_t appraise_type=imasig

13. What is appraise_type=imasig?
• Use digital signature of the hash of a file as the “good” value
• Store the digital signature in the xattr (extended attribute) of the file
# evmctl ima_sign -a sha256 -k privkey.pem /usr/bin/runc
# getfattr -m "." -d /usr/bin/runc
getfattr: Removing leading '/' from absolute path names
# file: usr/bin/runc
security.ima=0sAwIEH+VEdQEAkN3YSXNhBLf...UJQ8tk7...LhYUZVy5wmEmtBXSkBHP9yTZFg==
security.selinux="system_u:object_r:container_runtime_exec_t:s0"

14. How IMA appraise the file with imasig?
# keyctl describe %keyring:.ima
574740493: -lswrv--swrv------------ 0 0 keyring: .ima
# evmctl import x509_ima.der 574740493
# keyctl show %keyring:.ima
Keyring
574740493 ---lswrv 0 0 keyring: .ima
4225879 --als--v 0 0 _ asymmetric: IMA Signing Key:
e52f68af018134bfa644cbda02b5c8ef1fe54475
• Load the public key onto the .ima keyring (*)
• IMA uses the public keys loaded onto the .ima keyring to verify the
digsig (security.ima) of the hash of the file being appraised
(*) The public key must be signed by the trusted certificate in the kernel

15. # docker run cve
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
HAX2: argv: /proc/self/fd/3
HAX2: fd: 4
HAX2: res: 13, 0
# strings /usr/bin/runc | tail -n 2
.debug_gdb_scripts
cve-2019-5736
# docker run cve
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
ERRO[0000] Error removing container
6ed3c7ab94169e53352b3df9e461d4406d7851389948c4de39cc2cd0df7b4909 from runtime after
creation failed
Error: container create failed (no logs from conmon): EOF

16. References
• https://blog.dragonsector.pl/2019/02/cve-2019-5736-escape-from-docker-
and.html
• Integrity Verification of Docker Containers for a Lightweight Cloud
Environment, Future Generation Computer System 2019
• Leveraging Kernel Security Mechanisms to Improve Container Security: a
Survey, ARES’19
• Linux Integrity Subsystem Update by Mimi Zohar, Linux Security Summit 2019
• The Keylime Project, https://keylime.dev

17. Q/A

18. Backups

19. BPRM_CHECK
execve syscall
-> exec_binprm (fs/exec.c)
-> search_binary_handler (fs/exec.c)
-> security_bprm_check (security/security.c)
-> ima_bprm_check (security/integrity/ima/ima_main.c)
MMAP_CHECK
mmap_pgoff sys call
-> vm_mmap_pgoff (mm/util.c)
-> security_mmap_file (security/security.c)
-> ima_file_mmap (security/integrity/ima/ima_main.c)
FILE_CHECK
openat2 sys call
-> do_filp_open (fs/namei.c)
-> path_openat (ds/namei.c)
-> do_open (fs/namei.c)
-> ima_file_check (security/integrity/ima/ima_main.c)

20. IMA Keyring: .crs or _crs?
• Kernel config CONFIG_INTEGRITY_TRUSTED_KEYRING determines whether .ima
(trusted keyring) or _crs (regular keyring) is used for IMA
• If .ima was used, only those keys signed by a “kernel-trusted certificate” can be
loaded onto the .ima keyring for appraisal
• We can use CONFIG_SYSTEM_EXTRA_CERTIFICATE to reserve space in the
kernel for embedding own certificate to the kernel
• Then use our own certificate to sign the IMA signing public key
• Details refer to the “Generate trusted keys” in http://linux-ima.sourceforge.net/evmctl.1.html
• Use scripts/insert-sys-certs to insert the certificate into the kernel