Dynamically Hacking the Kernel with Containers: FreeBSD on Linux as an example.

1. Dynamically Hacking the Kernel
with Containers
CAT System Workshop
2016.09.20
Quey-Liang Kao
NTHUCS

2. About Myself
● PhD Student
– Adviser: Che-Rung Lee
● Research topics
➢ HPC (numerical), Heterogeneous computing
➢ High-end hardware virtulization (InfiniBand, GPGPU)
➢ Container technology

3. About Myself - the other side
● Archoholic
– Archwiki
– AUR packager
● runc-git, openscap
● kpatch
● GNOME
● BICIB
– https://github.com/NonerKao/BICIB

4. What is this?

5. Hacking the Kernel ...

6. Hack
...
Computers.
● to modify (a computer program or electronic device) or write
(a program) in a skillful or clever way.
● to circumvent security and break into (a network, computer,
file, etc.), usually with malicious intent: Criminals hacked the
bank's servers yesterday.
http://www.dictionary.com/browse/hacking

7. Hack
...
Computers.
● to modify (a computer program or electronic device) or
write (a program) in a skillful or clever way.
● to circumvent security and break into (a network, computer,
file, etc.), usually with malicious intent: Criminals hacked the
bank's servers yesterday.
http://www.dictionary.com/browse/hacking

8. Hacking the Kernel ... [ OK ]
Dynamically ...

9. 3 ways to modify the
kernelspace
Modules,
Live patches,
and Kernel detouring

10. Kernel Module: Loading
Kernel
ext.ko
Kernel Space
User Space
Process:
insmod ext.ko
syscall: init_module

11. Kernel Module: Using
some_ext.ko
Kernel Space
User Space
Process:
cat /dev/some_cdev
syscall: read
Kernel

12. Kernel
Live Patching: Building
fix.ko
Kernel Space
User Space
Process:
build fix.patch
syscall: init_module
File:
fix.patch
Current
kernel source
Bug()
Bug-fixed()

13. Kernel
Live Patching: Applying
fix.ko
Kernel Space
User Space
Process:
insmod fix.ko
syscall: init_module
Bug()
Bug-fixed()

14. Kernel
Live Patching: Applying
fix.ko
Kernel Space
User Space
Bug() Bug-fixed()
ftrace
Normal
Process

15. Kernel
Kernel Detouring
detour.ko
Kernel Space
User Space
Normal
Process
Special
Process
func() func()

16. Hacking the Kernel ... [ OK ]
Dynamically ... [ OK ]
with Containers ...

17. Virtual Machines ( KVM )
Kernel
User Space
Kernel Space
kvm.ko
VM 2VM 1
Proc 1
Libs / Conf
Guest Kernel
Proc 3
Libs /
Conf
Guest
Kernel
Proc 2
hypercall

18. Processes
Kernel
Process
1
User Space
Kernel Space
Libraries / Configuration
Process
2
Process
3

19. Container 2Container 1
Containers
Kernel
Process
1
User Space
Kernel Space
Libs / Conf
Process
2
Process
3
Libs /
Conf
Namespaces

20. Kernel
Kernel Detouring
detour.ko
(namespace-
aware)
Kernel Space
User Space
Normal
Process
Container
func() func()

21. Hacking the Kernel ... [ OK ]
Dynamically ... [ OK ]
with Containers ... [ OK ]

22. Why?

23. The short story
-- Just for fun!

24. The long story:
1. Impact
http://alaskarobotics.com/2009/11/30/gallery-walk/
+
=

25. The long story:
2. Trend
SaaS
PaaS
IaaS
CaaS
?

26. VM
Process
Libs /
Conf
Guest
Kernel
Container
The long story:
3. Towards higher isolation
Isolation++
Performance++
Process
Libs /
Conf
kvm
K-Container
Process
Libs /
Conf
Detour

27. VM
Process
Libs /
Conf
Guest
Kernel
Container
Side Note:
For better performance
Isolation++
Performance++
Process
Libs /
Conf
kvm
HyperV
Container
Process
Libs /
Conf
Windows
Server 10
Thin
Kernel

28. Where?

29. Possible Use Cases
● Experimental module/patch test bed
● Environment for other OSes
● Educational purpose

30. FreeBSD binary on Linux
01000110011100100110010101100101010000100101001101000100

31. Why it is not possible: Scene 1
Kernel
mov 0x1, %rax
syscall
(translate: I want to exit)

32. Why it is not possible: Scene 2
Kernel
...you want to write?
nothing specialsys_write()

33. Why it is not possible: Scene 3
Kernel
(Eventually)
SIGSEGV
End of life but no exit
...what’s wrong with this process?

34. Kernel
Recall: Kernel Detouring
detour.ko
(namespace-
aware)
Kernel Space
User Space
Normal
Process
Container
func() func()

35. Why it IS possible now: Scene 1
Kernel
mov 0x1, %rax
syscall
(translate: I want to exit)
Container

36. Container
Why it IS possible now: Scene 2
Kernel detour.ko
(namespace-
aware)
system call table
system call table
for FreeBSD
#1 is sys_write, so …
...wait, this is a FreeBSD container!

37. Container
Why it IS possible now: Scene 3
Kernel detour.ko
(namespace-
aware)sys_exit()
system call table
for FreeBSD

38. Specific Challenges ( FreeBSD )
● Corresponding system calls
– Flag numbers are not portable
– different calling/exiting conventions
● Unique system calls
– Re-implementation

39. General Challenges
● Insufficient isolation
● Limitation of development
– live patching should only be a temp. solution

40. Other Binary Compatibility Work
● Wine
– Special loader for PEs/DLLs
● FreeBSD, Windows 10
– Kernel built-in compatibility layer for Linux binary
– System call remapping/re-implementation

41. How?

42. Step 0: Setup
● Environment ( x86_64 machine )
– Linux 4.6.2
– FreeBSD 10.2
● Tools
– kpatch: A tool for kernel livepatch
– docker

43. Step 1: From LivePatching to
Detouring
kernel/livepatch/core.c.orig:klp_ftrace_handler
klp_arch_set_pc(regs, (unsigned long)func->new_func);
kernel/livepatch/core.c:klp_ftrace_handler
klp_arch_set_pc(regs, (is_freebsd_container()) ?
(unsigned long)func->new_func :
(unsigned long)ip+5);

44. Ftrace in LivePatching
old_func()
0x1230 call __fentry__
0x1235 push %rbp
0x1236 mov %rsp, %rbp
0x1239 push %r15
...
new_func()
0x2230 call __fentry__
0x2235 push %rbp
0x2236 mov %rsp, %rbp
0x2239 push %r14
...
ftrace
handler
IP
func->new_func

45. Ftrace in Detouring
old_func()
0x1230 call __fentry__
0x1235 push %rbp
0x1236 mov %rsp, %rbp
0x1239 push %r15
...
new_func()
0x2230 call __fentry__
0x2235 push %rbp
0x2236 mov %rsp, %rbp
0x2239 push %r14
...
ftrace
handler
IP
IP+5
func->new_func

46. Step 2: Detour-able Entry Point
arch/x86/entry/entry_64.S.orig:
ENTRY(entry_SYSCALL_64)
…
call *sys_call_table(, %rax, 8)
void detour_entry()
{
…
asm("jmp *sys_call_table(,
%rax, 8)");
}
arch/x86/entry/entry_64.S:
ENTRY(entry_SYSCALL_64)
…
call detour_entry
Assembly file is NOT
detour-able!
In some C source file

47. Step 3: Detoured Entry Point
void detour_entry()
...
asm("jmp *sys_call_table(, %rax, 8)");
void detour_entry()
...
asm("mov fbd_table(, %rax, 8), %rax");
asm("jmp *sys_call_table(, %rax, 8)");
asmlinkage const long long fbd_table[] = {0,60,57,...};
EXPORT_SYMBOL(fbd_table);
remapping syscall#

48. The work flow
1. Launch a normal container
2. Run a init script
1. which enables the specific detour modules
3. A FreeBSD environment in the container
4. On exit
1. disable detour modules

49. DEMO!

50. Conclusion
● The kernel detouring demo attempts to
indicate the possibility of the development of
OS containers
– as a proof-of-concept
– kpatch as a temp. solution
● Future direction
– Make more fun

51. Q & A

52. Behind the scenes
● How to implement the is_freebsd_container()
function?
● How was the ContainerCon Japan?