This talk will start with a deep dive and hands on examples of BPF, possibly the most promising low level technology to address challenges in application and network security, tracing, and visibility. We will discuss how BPF evolved from a simple bytecode language to filter raw sockets for tcpdump to the a JITable virtual machine capable of universally extending and instrumenting both the Linux kernel and user space applications. The introduction is followed by a concrete example of how the Cilium open source project applies BPF to solve networking, security, and load balancing for highly distributed applications. We will discuss and demonstrate how Cilium with the help of BPF can be combined with distributed system orchestration such as Docker to simplify security, operations, and troubleshooting of distributed applications.

1. CILIUM: NETWORK AND
APPLICATION SECURITY
WITH BPF AND XDP
Thomas Graf
Co-founder & CTO
Covalent

2. Who is this guy?

3. Helped build the
biggest monolith
ever …
Who is this guy?

4. Helped build the
biggest monolith
ever …
Who is this guy?

5. Time to rethink the kernel

6. syscalls syscalls
Net IOBlock IO
Time to rethink the kernel

7. Time to rethink the kernel
From monolith to “microkernel” with BPF
syscalls syscalls
BPF
BPF
BPF
BPF
Net IOBlock IO

8. Time to rethink the kernel
From monolith to “microkernel” with BPF
syscalls syscalls
BPF
BPF
BPF
BPF
BPF
BPF
Security
Networking
Net IOBlock IO

9. BPF is revolutionizing…
• Tracing / Profiling

10. BPF is revolutionizing…
• Tracing / Profiling
Container Performance
Analysis
Brendan Gregg
Wed 1:30pm “Black Belt”

11. BPF is revolutionizing…
• Tracing / Profiling
• Networking
Container Performance
Analysis
Brendan Gregg
Wed 1:30pm “Black Belt”

12. BPF is revolutionizing…
• Tracing / Profiling
• Networking
• Security
Container Performance
Analysis
Brendan Gregg
Wed 1:30pm “Black Belt”

13. Application
Architectures
Delivery Frequency
Operational
Complexity
Single Server
App
Yearly
Low
Delivery Frequency

14. Application
Architectures
Delivery Frequency
Operational
Complexity
Single Server
App
Yearly
Low
3-Tier App
Monthly
Moderate
Delivery Frequency

15. Application
Architectures
Delivery Frequency
Operational
Complexity
Single Server
App
Yearly
Low
Distributed
Microservices
10-100 x’s / day
Extreme
3-Tier App
Monthly
Moderate
Delivery Frequency

16. Network Security
has not evolved
$ iptables -A INPUT -p tcp
-s 15.15.15.3 --dport 80
-m conntrack --ctstate NEW
-j ACCEPT
The world still runs on iptables
matching IPs and ports:

17. Your HTTP ports be like …

18. Network Security
for Microservices
Example
Gordon is
looking for
a job…

19. Gordon Job Postings
Example: Security for Microservices

20. GET /healthz
GET /jobs/{id}
PUT /jobs/{id}
POST /jobs
API
Gordon Job Postings
Example: Security for Microservices

21. GET /healthz
GET /jobs/{id}
PUT /jobs/{id}
POST /jobs
API
GET /jobs/331
Gordon Job Postings
Example: Security for Microservices

22. L3/L4
GET /healthz
GET /jobs/{id}
PUT /jobs/{id}
POST /jobs
API
iptables -s 10.1.1.1
-p tcp --dport 80
-j ACCEPT
GET /jobs/331
Gordon Job Postings
Example: Security for Microservices

23. L3/L4
GET /healthz
GET /jobs/{id}
PUT /jobs/{id}
POST /jobs
API
exposed
exposed
exposed
GET /jobs/331
Gordon Job Postings
Example: Security for Microservices
iptables -s 10.1.1.1
-p tcp --dport 80
-j ACCEPT

24. Not exactly
least privilege
Security team is
not amused

25. GET /healthz
GET /jobs/{id}
PUT /jobs/{id}
POST /jobs
API
GET /jobs/331
Gordon Job Postings
Example: Security for Microservices

26. L3/L4
GET /healthz
GET /jobs/{id}
PUT /jobs/{id}
POST /jobs
API
FROM Gordon
ALLOW GET /jobs/.*
GET /jobs/331
Gordon Job Postings
Example: Security for Microservices

27. We demand
a demo

28. BPF - The
Superpowers
inside Linux

29. SANDBOX
BPF
GET /foo
BPF: Transparent redirection into proxy

30. SANDBOX
BPF
Proxy
GET /foo
redirect
rules
sk
BPF: Transparent redirection into proxy

31. SANDBOX
BPF
Proxy
GET /foo
redirect
rules
sk
Shared State
• Orig Dest IP
• Identity
BPF: Transparent redirection into proxy

32. SANDBOX
BPF
Proxy
GET /foo
redirect
reinject
rules
sk sk
Shared State
• Orig Dest IP
• Identity
BPF: Transparent redirection into proxy

33. SANDBOX
BPF
Proxy
GET /foo
rules
BPF: Transparent redirection into proxy
sk sk
403
Access
Denied

34. So what is BPF exactly?

35. .insns = {
BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
BPF_LD_MAP_FD(BPF_REG_1, 0),
BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
BPF_MOV64_REG(BPF_REG_1, BPF_REG_10),
BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -152),
BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0),
BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_1, 0),
BPF_ST_MEM(BPF_DW, BPF_REG_3, 0, 42),
BPF_EXIT_INSN(),
}
What is
BPF?
Learn more about BPF: docs.cilium.io

36. BPF: Toolchain – from user to kernel
USER SPACE
SOURCE CODE [C]
</>

37. BPF: LLVM compiles program code to bytecode
USER SPACE
SOURCE CODE [C]
</>
BYTE CODE [BPF]
</>

38. BPF: Bytecode is loaded and verified into kernel
USER SPACE
KERNELVERIFIER +
JIT
SOURCE CODE [C]
</>
BYTE CODE [BPF]
</>

39. BPF: Bytecode runs inside safe kernel sandbox
USER SPACE
KERNELVERIFIER +
JIT
SOURCE CODE [C]
</>
BYTE CODE [BPF]
</>
SANDBOX
BPF

40. BPF: Program is attached to event (packet-in)
USER SPACE
KERNELVERIFIER +
JIT
SOURCE CODE [C]
</>
BYTE CODE [BPF]
</>
SANDBOX
BPF

41. BPF: Program can redirect to netns & sockets
USER SPACE
KERNELVERIFIER +
JIT
SOURCE CODE [C]
</>
BYTE CODE [BPF]
</>
SANDBOX
BPF

42. BPF – An opportunity
to rethink security
policy enforcement

43. Status Quo: Policy Enforcement
connect()

44. Status Quo: Policy Enforcement
connect()
TCP

45. Status Quo: Policy Enforcement
connect()
TCP
Network
packets

46. Status Quo: Policy Enforcement
connect()
TCP
Network
packets
veth
veth
namespace boundary

47. Status Quo: Policy Enforcement
connect()
TCP
Network
packets
iptables
veth
veth
namespace boundary

48. Status Quo: Policy Enforcement
connect()
drop
TCP
Network
packets
iptables
veth
veth
namespace boundary

49. Status Quo: Policy Enforcement
connect()
drop
TCP
Network
packets
ETIMEDOUT
iptables
veth
veth
namespace boundary

50. Status Quo: Policy Enforcement
connect()
drop
TCP
Network
packets
ETIMEDOUT/
ECONNREFUSED
iptables
RST
veth
veth
namespace boundary

51. Can we do better?
connect()

52. BPF: Leverage user space tool chain
USER SPACE
KERNEL
connect()
VERIFIER +
JIT
SOURCE CODE [C]
</>
BYTE CODE [BPF]
</>

53. BPF: Attach program to connect() syscall (LSM)
USER SPACE
KERNEL
connect()
VERIFIER +
JIT
SOURCE CODE [C]
</>
BYTE CODE [BPF]
</>
BPF
LSM Hook

54. BPF: Return EACCESS – No packets created at all
USER SPACE
KERNEL
connect()
EACCESS
VERIFIER +
JIT
SOURCE CODE [C]
</>
BYTE CODE [BPF]
</>
BPF
LSM Hook

55. XDP/BPF – The
software loadbalancer
of the future

56. WHAT IF I TOLD YOU
XDP allows for 10x
IPVS performance

57. Source: https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf
FB moves from IPVS to BPF/XDP for L3/L4 LB
XDP throughput
IPVS throughput
Source:

58. Regular BPF mode
BPF
Driver Software Stack

59. XDP [Express Data Path] mode
BPF
Driver
Run BPF Program inside network
driver with access to DMA buffer
Software Stack

60. XDP [Express Data Path] mode
BPF
Driver
Can drop millions of packets per
Second while under DDoS
Software Stack
drop

61. XDP [Express Data Path] mode
BPF
Driver
Can pass packets to network stack
Software Stack
drop
Stack

62. XDP [Express Data Path] mode
BPF
Driver
Can perform loadbalancing and
transmit out the wire again
Software Stack
drop
LB & TX
Stack

63. How can I use BPF
with Docker?

65. Cilium Architecture
Cilium
Agent

66. Cilium Architecture
Cilium
Agent
Plugins

67. Cilium Architecture
BPF
Cilium
Agent
Plugins

68. Cilium Architecture
BPF
BPF
Cilium
Agent
Plugins

69. Cilium Architecture
BPF
BPF
Cilium
Agent
Plugins

70. Cilium Architecture
BPF
BPF
BPF
Cilium
Agent
Plugins

71. Cilium Architecture
BPF
BPF
BPF
Cilium
Agent
Plugins

72. Cilium Architecture
BPF
BPF
BPF
Cilium
Agent
CLI Monitor Policy
Plugins

73. Project Status
• Initial release two weeks ago
• Docker & Kubernetes integration
• Looking for feedback and
contributions

74. Getting Started
• Play with our vagrant box:
$ git clone https://github.com/cilium/cilium
$ cd cilium/examples/getting-started
$ vagrant up

75. Summary

76. Summary
• Never underestimate the
Jedi

77. Summary
• Never underestimate the
Jedi
• Traditional L3/L4 network
policies are insufficient for
microservices. Least
privilege requires HTTP /
API / Function awareness.

78. Summary
• BPF/XDP will drive the
future of software based
networking on Linux.
• Never underestimate the
Jedi
• Traditional L3/L4 network
policies are insufficient for
microservices. Least
privilege requires HTTP /
API / Function awareness.

79. Summary
• Never underestimate the
Jedi
• Traditional L3/L4 network
policies are insufficient for
microservices. Least
privilege requires HTTP /
API / Function awareness.
• BPF/XDP will drive the
future of software based
networking on Linux.
• Cilium brings BPF/XDP
and L7 policies to
containers and
microservices.

80. Thank You!
github.com/cilium/cilium
http://cilium.io/
@ciliumproject
Want to chat? DM me! @tgraf__
Don’t forget
to vote and
grab a shirt
on the way
out!

81. 75
140
205
240
325
365 370 365
410 412 425
445 450 460 460
490 495 505 515 525
545
565
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
BPF redirect() performance
[GBit per core]
Intel Xeon 3.5Ghz Sandy Bridge, 24 Cores,
(1 TCP GSO flow per core, netperf -t TCP_SENDFILE, 10K policies)