The talk introduces the need of live kernel patching. Further, it explains what is kGraft, how it works, what are its limitations, and our plans with the implementation in the future. The presentation includes also a live demo if stars constellation allows. Jiri Slaby, SUSE

1. kGraft
Live patching of the Linux kernel
Jiˇrí Kosina, Petr Mládek, Vojt ˇech Pavlík, Jiri Slaby
SUSE Labs
September 25th 2014
Paris, France
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 1 / 23

2. Why Live Patching?
1000 machines & severe security problem
Needs fixing now!
Rebooting the machines
Is not a quick way to fix an issue
Has a risk of not coming up
Live patching
Allows quick response
Leaves an actual update to a scheduled downtime window
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 2 / 23

3. Where is Live Patching Useful?
Common tiers of change management
1 Incident response – we are exploited
2 Emergency change – we could be exploited (we are vulnerable)
3 Scheduled change – time is not critical, we are safe
Live patching fits in with 1 and 2
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 3 / 23

4. Presentation Outline
1 KGRAFT
2 KGRAFT Internals
3 Live Demo
4 Conclusion
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 4 / 23

5. Section 1
KGRAFT
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 5 / 23

6. KGRAFT
Research project
Live patching technology
Developed by SUSE Labs
Specifically for the Linux kernel
Based on modern Linux technologies
INT3/IPI-NMI self-modifying code
Lazy update mechanism
fentry-based NOP space allocation
Standard kernel module loading/linking mechanisms
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 6 / 23

7. Advantages of KGRAFT
Does not require stopping the kernel
Ever!
Not even for short time periods
Unlike competing technologies
Allows code review on KGRAFT patch sources
Patches can be built from C source directly
No need for object code manipulation
Only an alternative: object code based automated patch generation
kGraft is lean
Small amount of code
Leveraging other Linux technologies
No complex instruction en/decoders or such
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 7 / 23

8. How does KGRAFT work?
A kGraft patch is a .ko kernel module
The .ko is inserted into the kernel using insmod
All linking (incl. the fix) is done by kernel
KGRAFT replaces whole functions in the kernel
Even while those functions may be executed
An updated KGRAFT module can replace an existing patch
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 8 / 23

9. Limitations
kGraft is designed for fixing critical bugs
Primarily for simple changes
Changes in kernel data structure layout require special care
Depending on the size of the change, reboot may be needed
Same as with other live patching techniques
KGRAFT depends on a stable build environment
Having history of built kernels
Best suited for
Linux distributions
Their customers
Anyone who builds their own kernels
Not good for 3rd party support
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 9 / 23

10. Section 2
KGRAFT Internals
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 10 / 23

11. KGRAFT and fentry
KGRAFT needs some space at the start of a function
To insert a jump to a patched function
The space can be provided by GCC profiling
-pg -mfentry
KGRAFT uses this
fentry call instructions
Patched out at boot
Replaced with 5-byte NOPs
kernel_func
CALL fentry
kernel_func
5-byte NOP
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 11 / 23

12. Using 5-byte NOPs Space
KGRAFT uses the ftrace infrastructure to perform patching
INT3 handler is installed with a JMP to the destination address
1 First byte of NOP is replaced by INT3
2 Remaining bytes are replaced by address
3 First byte is replaced by JMP
4 NMI IPIs are used to flush instruction decoders on other CPUs
kernel_func
kernel_func
5-byte NOP
INT3 | xxxx
INT3 handler
JMP addr
kernel_func
INT3 | addr
kernel_func
JMP addr
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 12 / 23

13. Patching a Function
Patching during runtime, no stop_kernel();
Callers are never patched
Rather, callee’s NOPs are replaced by a JMP to the new function
JMP remains forever
But this takes care of function pointers, including in structures
Like indirect calls (handler->function())
Does not require saving any old data in case we want to un-patch
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 13 / 23

14. Patching a Function in Pictures
kernel_func
buggy_func();
buggy_func
JMP fixed
fixed_func
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 14 / 23

15. Issue: Non-consistency
What happens when
Replaced function changes semantics and subsequent calls rely
on each other?
It is called recursively?
kernel_func
buggy_func();
buggy_func();
BOOM!
buggy_func
KGRAFT patch comes in
fixed_func
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 15 / 23

16. Cure: RCU-like Replacement
We need to provide a consistent “world-view” to each thread
User processes
Kernel processes
Interrupts
Solution: “reality check” trampoline
Per-thread flag set on each kernel entry/exit
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 16 / 23

17. RCU-like Replacement
kernel_func
heavy work
buggy_func();
reality_check
which universe
are you
coming from?
buggy_func
fixed_func
Userspace
Kernelspace
Welcome to
the new universe!
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 17 / 23

18. Lazy Replacement
All processes must wake up or execute a syscall
Sometimes this requires a signal to be sent (like for getty’s)
Once all processes have the "new universe" flag set
Patching is complete
Trampolines can be removed
Files to check
/proc/<pid>/kgr_in_progress
/sys/kernel/kgraft/kgr_in_progress
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 18 / 23

19. Lazy Replacement
kernel_func
heavy work
buggy_func();
buggy_func
fixed_func
Userspace
Kernelspace
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 19 / 23

20. Get It
Upstreaming
KGRAFT was submitted and reviewed upstream
There are other groups working on competing technologies
KPATCH, KSPLICE, criu-based aproach, . . .
SUSE will work together with them
Expectations: common standard kernel live patching
Publishing
Part of SLE12 kernel tree
GIT repository upstream
http://git.kernel.org/pub/scm/linux/kernel/git/jirislaby/
kgraft.git
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 20 / 23

21. Maintenance
KGRAFT patch is an RPM package
Once installed, always protected
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 21 / 23

22. Live Demo
Kernel with a security vulnerability
Exploit program
kGraft patch
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 22 / 23

23. Conclusion
SUSE provides
Demanded live Linux kernel patching
Dubbed KGRAFT
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 23 / 23