Slides from The Secure Developer webinar hosted on 9/26/2019 https://www.thesecuredeveloper.com/events/the-state-of-kubernetes-security

1. The State of Kubernetes Security

2. Hello! I’m Jimmy.

3. - What are Containers?
- What is Kubernetes?
- Kubernetes Security Considerations
- Kubernetes: Self Assessment
- The Future of K8S Security

4. Containers

5. Containers are an application-
layer construct which rely on
a shared kernel.

6. Containers are not
“lightweight VMs.”

8. Containers rely on Linux kernel
features such as namespaces
and cgroups to isolate a given
process (AKA container).

10. Container Breakout occurs when the
container isolation mechanisms have
been bypassed and additional privileges
have been obtained on the host.

11. Containers can do bad things:
- Mount volumes & directories
- Disable security features
- Run as root
- Share the host namespace

12. Exploits

13. Dangerous Mountpoints

14. Containers typically restrict the
number of powerful Linux capabilities
granted to it.

15. Privileged containers bypass those
controls, granting dangerous
capabilities to the container.

16. Demo: Privileged vs. Non-Privileged

17. Image Integrity

18. Kubernetes

19. Kubernetes is an open-source
platform built to automate
deployment, scaling, and
orchestration of containers.

20. ”Kubernetes is a
pile of Linux goop.”

21. I thought Kubernetes
was secure by default?

22. Kubernetes optimizes for
flexibility over security.
…And that’s ok!

23. Kubernetes is a living, breathing
system. Security is never “done”.

26. https://medium.com/jw-player-engineering/how-a-cryptocurrency-miner-made-its-way-onto-our-internal-kubernetes-clusters-9b09c4704205

28. Kubernetes Threat Model
28
User Compromise and Insider Threats
• Cluster admin account compromise
• Rogue Employee
• Build system compromised
Application Vulnerabilities
• Lack of authentication and authorization, both k8s internal and external
• Weak or incorrect usage of cryptography
• Application and API vulnerabilities - remote code execution (RCE), web
vulnerabilities (XSS, CSRF, SSRF, SQL Injection etc.)
• Insecure third-party components

29. Kubernetes Threat Model
29
Network and Infrastructure
• Network snooping, ARP spoof attacks
• Compromising infrastructure services (etc. NTP, DNS, SSH)
• Kernel and other operating system vulnerabilities
Application Containers
• Container breakout and unauthorized access control plane and other
containers
• Denial of Service - resource hogging, eating up CPU/Mem/Disk/IO to
impact or even crash other containers
• Compromised or malicious image or pipeline

30. Kubernetes Threat Model
30
Misconfiguration
• Insecure default configurations - unused open ports,
services, not enforcing system/application limits, failing to
implement security features
• Misuse of passwords, passphrases, TLS private keys
(*cough* checking them into git *cough*. Bad handling
include key reuse, insecure handling of keys, no key
rotation, weak passwords, not using MFA etc.
• Lack of network segmentation - exposing critical systems to
various network attacks

31. 31

32. Source: Kubernetes Security - Operating Kubernetes Clusters and Applications Safely
Access via Kubelet API

33. Defenses

34. RBAC
Container and Pod Permissions
Pod Security Policies
Dynamic Admission Control
OPA
Sandboxing
Node Protection

35. Role-Based Access Control
(RBAC) is how we regulate
access to Kubernetes resources.

36. Users
you@email.com
Service account
API Resources
Namespaces
Pod
Service
Secrets
…
Operations
Get
List
Delete
Patch

37. role.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: development
name: pod-reader
rules:
- apiGroups: [""]
resources: [”pods"]
verbs: ["get", "list"]

38. Containers may request elevated
privileges such as running as root,
mounting sensitive volumes, or
requesting access to specific ports.

39. Pod specifications may declare to
access devices on the host using
privileged mode.

40. Pod Security Policies give
administrators the ability to
validate requests to the cluster
based on security requirements.

41. psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: my-psp
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: MustRunAsNonRoot
volumes:
- 'configMap'
- 'emptyDir’
- 'secret’
- 'persistentVolumeClaim'

42. Dynamic Admission Control allows
teams to build custom security
checks by intercepting requests to
the Kubernetes API server prior to
scheduling the object.

43. https://github.com/kelseyhightower/denyenv-validating-admission-webhook

44. Open Policy Agent (OPA) acts as
a middleware to help enforce
cluster-wide security policies.

45. https://github.com/open-policy-agent/gatekeeper

46. Gatekeeper Examples
46
Require Specific Labels upon object creation
Audit Cluster for violations of policy
Namespace must have “Owner” label
Containers must have resource limits defined

47. Always ensure images come
from a known-good source
and the integrity has been
verified.

48. Tools such as gVisor and Kata
Containers can help further isolate
and sandbox containers that are
running untrusted workloads
inside of Kubernetes.

49. Remember, Kubernetes is just
running servers under the hood.
Our regular old OS hardening and
network protections apply.

50. Kubernetes can be secure,
but it is far from default.

51. Take Home Assignment

52. 52
• Can containers run as root?
• Can containers mount sensitive volumes / directories? Read or Read / Write?
• Can Pods run in “Privileged” mode?
• What policies (PSP, custom, OPA) are in place and for who?
• How is authentication handled?
• Is RBAC enforcing the principle of least privilege?
• How are secrets being stored and retrieved? Rotated? Revoked?
• Where do container images come from? Are images being validated?
• How is network security being enforced? Can you audit these rules?
• Are your hosts hardened? Monitoring in place?
• Are you using Kubernetes Audit? Where are logs sent?
• Ingress / LB inventory in place? What external IP addresses are available?
• What happens if / when your application has an SSRF bug?
• Have you performed a proper threat model of Kubernetes environments?
• Third party products, tools, helpers? Are they secure?

54. Beware of blind spots

55. Embrace a beginners mindset mindset

56. Adapt and evolve

57. Give back to the community

58. 58
• Flexibility > Security will be our reality
• Choose your Own Security Adventure
• More tooling
• Tighter Cloud integrations
• Overall Kubernetes maturity
• Increasing target for attack
The Future?

59. https://github.com/RhinoSecurityLabs/cloudgoathttps://github.com/ksoclabs/kube-goat
https://www.owasp.org/index.php/OWASP_Serverless_Goat
Practice

60. @jimmesta
jimmy[at]ksoc.com