This talk explains what what Pod Security Policy is and it's importance in Kubernetes Security. The talk also takes a look at the current situation of docker hub's popular images and helm charts repository. This talk stresses on the fact that having PSP enabled the right way is absolutely necessary for the real security of the cluster. Link to the demos: What is Pod Security Policy? https://www.youtube.com/watch?v=nrWRMP94vqc Kubernetes Hostpath exploit thrawted with Pod Security Policy https://www.youtube.com/watch?v=APS0CfD6DsE

1. Hardening Kubernetes
by Securing Pods
Suraj Deshmukh

2. Hi, I’m Suraj Deshmukh
suraj.io
surajd_
surajssd

3. What is Kubernetes?
● Container orchestrating system.
● Project initiated by Google.
● Has robust API system, scheduler to schedule workloads onto Nodes.
● Uses etcd to store cluster state.

4. Components of Kubernetes
Image Source:: https://kubernetes.io/docs/concepts/architecture/cloud-controller/

5. Basic unit of workload - Pod
● Its a group of one or more containers, with shared
storage/network, and a specification for how to run
the containers.
● Pod’s contents are always co-located and
co-scheduled, and run in a shared context.
Image Source:: https://kubernetes.io/docs/concepts/workloads/pods/pod/

6. Threat Models in Kubernetes
● External attacks
● Compromised containers/nodes OR attack from inside
● Compromised credentials
● Misuse of Legitimate privileges

7. Threat Models in Kubernetes
● External attacks
● Compromised containers/nodes OR attack from inside
● Compromised credentials
● Misuse of Legitimate privileges

8. What is attack from inside?

9. We trust our developers
Image Source:: http://turnoff.us/geek/the-depressed-developer-15/

10. ● Multi-tenant setup where your clients are sharing resources like nodes on the same
cluster.
● An attacker gains access to the container’s shell and what they can do from there.

11. State of Container and
Kubernetes Security

12. Secure defaults
There are many ways to deliver an “out of the box” experience for users. However, by default,
the experience should be secure, and it should be up to the user to reduce their security – if they
are allowed. It is imperative for the software environment to have default secure settings which
may be opted out of by the user or other options which may be opted into (commonly known as
Opt-in and Opt-out).
- Open Web Application Security Project

13. ● Some of the bad practices we see are
because of the security being opt-in
than opt-out in most of these
systems.
● People are running root in the
container.
● User namespaces are very new.
● Not so secure by default design.
Image Source:: http://www.commitstrip.com/en/2016/10/14/good-old-adminpassword/

14. uid0 inside container

15. What is uid0 in container?
FROM registry.fedoraproject.org/fedora:30
ENTRYPOINT ["sleep", "infinity"]
FROM registry.fedoraproject.org/fedora:30
USER 1000
ENTRYPOINT ["sleep", "infinity"]

16. Containers don’t contain - Dan Walsh
● Containers are just set of linux kernel technologies that work in conjunction to form the
isolation.
● These technologies are comparatively new in Linux kernel and will mature.
● All these technologies are not battle tested like VM which provides actual isolation.
● There are always ways for a root inside container to break out and do some nasty stuff.

17. CVE-2019-5736
● If a process is running with UID0 inside the container, it could replace the runc binary
on the host and potentially can gain root on host
● This could have been clearly mitigated if root inside container is restricted by default.

19. Solution to enforce non-root containers?

20. Enter Pod Security Policy!

21. What are PSP?
● It’s a cluster wide Kubernetes resource.
● It helps you assign secure defaults.
● You define various aspects of a pod security context & container security context.
● Define what UID, GID is allowed, capabilities a container can have inside container

22. PSP Demo

23. PSP attack Demo
rm -rf /

24. Where does PSP sit in?
Image Source:: https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/

25. State of PSPs

26. On Managed Kubernetes offering

27. In helm charts

28. Improving this state
● Educating folks of this awesome feature is the way to go
● The current state we are in because security has been an afterthought.
● Use secure practices from day1 of the development phase.
● The docker images and helm charts need a revamp.

29. Secure Software Development Lifecycle S-SDLC
Security should be given a iterative approach and not a waterfall one.
- Cindy Blake

30. Defense in depth - PSPs are not enough
● Network Policy
● Secure image building practices
● Audit Logging
● Avoid mounting service accounts
● Permissions on demand in RBAC
● Use containers that actually contain, e.g. katacontainers, kubevirt, gvisor, etc.
● Use admission plugin DenyEscalatingExec

31. ● Containers don’t contain by Dan Walsh https://www.youtube.com/watch?v=a9lE9Urr6AQ
● Kubernetes Deployment and Security Patterns https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/
● GKE Using PodSecurityPolicies https://cloud.google.com/kubernetes-engine/docs/how-to/pod-security-policies
● EKS support for PSP https://github.com/aws/containers-roadmap/issues/174
● Hardening your cluster's security https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster
● Securing a Cluster https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/
● Runc and CVE-2019-5736 https://kubernetes.io/blog/2019/02/11/runc-and-cve-2019-5736/
● CVE-2019-5736 Detail https://nvd.nist.gov/vuln/detail/CVE-2019-5736
● Kubernetes Security - Michael Hausenblas, Liz Rice https://www.oreilly.com/library/view/kubernetes-security/9781492039075/
● Kubernetes logo https://github.com/kubernetes/kubernetes/blob/master/logo/logo.png
● TheNewStack's - Kubernetes Deployment and Security Patterns
https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/
● How to Secure Your Kubernetes Clusters - Cindy Blake https://youtu.be/M6db_dK0HF4
● Images running root stats https://github.com/surajssd/container-image-stats
● Running with Scissors - Liz Rice https://www.youtube.com/watch?v=ltrV-Qmh3oY
● Dilbert comic about firewall https://dilbert.com/strip/2013-04-07
● AWS EKS Pod Security Policy support PR https://aws.amazon.com/blogs/opensource/using-pod-security-policies-amazon-eks-clusters/
● OWASP secure defaults https://www.owasp.org/index.php/Establish_secure_defaults
References

32. Thank You