Kubernetes Security 101
http://tremolo.io

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Who Am I?
● CTO Tremolo Security
● Identity Management expert for 20+ years
● Experience in commercial and federal agencies
● Kubernetes since 2015
● Co-Author Kubernetes an Enterprise Guide: 2nd Ed

Agenda
● Part I - Cluster Security
○ What is Kubernetes?
○ How users access the cluster
○ How pipelines access the cluster
○ Dashboards
○ NetworkPolicies
○ Secrets
○ Node Security
○ Defense-in-Depth
○ Multi-tenancy
https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org

Agenda
● Part II - Container Security
○ How your application can be be a security risk
○ Basics of a secure image
○ SBOMs
○ Farm-to-Table Supply Chain Security
https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
What is Kubernetes?
● Scheduler - Determines what containers should run and where
● Kubelet - Works with the scheduler to run and manage containers
● API Server - API for interacting with the Kubelet and Scheduler

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster
● Kubernetes supports multiple authentication mechanisms:
○ Certificates
○ OpenID Connect
○ Impersonating proxy
○ TokenRequest API
○ Custom Webhooks

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - Certificates
● “Break Glass” authentication
● Can’t be revoked
● Hardware certificates aren’t supported
● External CAs not supported
● Groups only supported through static subject mapping
● Requires point-to-point connectivity between the client and server

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - OpenID Connect
● Use of a JSON Web Token (JWT)
● Bearer token
● Easily maps groups
● Should be short lived (1-2 minutes with clock skew)
● Allows for network segmentation
● Can not be revoked
● JWT Demo

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - Impersonating Proxy
● Reverse Proxy authenticates users
● Reverse proxy sends headers with
the user’s request to tell the API
server who the user is
● Impersonation proxy is responsible
for authorizing inbound
impersonation
● ServiceAccount for Impersonating
Proxy is a privileged account
● Important to tie API access back to
original request
● Impersonation Demo

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - TokenRequest API
● JWT issued by API Server
● Intended for identifying workloads to the API Server
● Intended for identifying workloads to external services
● Not meant for use from outside of the cluster
● Demo of a container talking to the API server

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - Custom Webhooks
● If you’re not a cloud provider, don’t do it

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - RBAC
● How the API server authorizes access
● Two Scopes
○ Cluster - Objects that impact everyone
○ Namespaced - Objects that are contained inside of a Namespace
■ Namespaces are Cluster scoped
● Roles/ClusterRoles → Define permissions
● RoleBindings/ClusterRoleBindings → Assign permissions
● Rights are enumerated
● No “negative” rights
● Don’t use “*”
● Bindings - Groups, not Users

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How users access the cluster - RBAC Aggregate
Roles
● Create large Roles & ClusterRoles without maintaining large objects
● Uses labels to assemble a Role/ClusterRole from smaller Roles/ClusterRoles
● Examples are admin, editor
● How to let an admin create a new instance of a CRD

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How pipelines access the cluster
● Don’t use long lived tokens
● Use a local identity to get short lived
token
● Spire (project) & SPIFFE (standards)

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Dashboards
● Centralized
○ Can be secured via reverse proxy and impersonation
○ Should never have a privileged identity
○ Rely on the user’s identity
● Local
○ Uses user’s kubectl configuration
○ Opens a local port on loop-back with no TLS or authentication

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
NetworkPolicies
● “Firewalls” of the cluster
● Control both inbound and outbound traffic
● Dependent on CNI implementation
● Not on by default
● Important to add
● Demo of network policy

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Secrets
● Stored in etcd as base64 encoded string
○ Encoded to preserve binary data
● Threat model your Secrets
○ Plain Kubernetes Secrets are Fine - https://bit.ly/3K1nEcU
● Externalize Secrets
○ Mount directly to Pods
○ Synchronize Secrets
● DON’T EVER STORE SECRETS IN GIT OR HELM, EVEN WHEN
ENCRYPTED

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Node Security
● Don’t run containers as root
○ Unless you need to run a container as root
○ Init containers
● Limit, or drop all, capabilities
● You don’t need a port under 1024
● Admission Controller Webhooks - Enforce rules
● Mutating Webhooks - Enable sane defaults
● Common Tools
○ Pod Security Standards
○ OPA/GateKeeper
○ Kyverno
○ JSPolicy

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Defense-in-depth
● Don’t rely on a single layer
● Strong Identity+RBAC+Policy Enforcement
● Multi-factor Authentication
● Short lived tokens
● Self Service

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Multitenancy
● Better utilization of resources
● Manage sprawl
● “Something” needs to be multitenant
● Self Service
● Combination of all of the above
● Virtual clusters

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Part II

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
How Your Application Can Be A Security Risk
● ServiceAccount token mounted to your Pod
● RCE from an app can lead to access to your Pod’s tokens
● Lack of sane defaults and policies lead to a container breakout
● Leaking bearer tokens
● Debug tools
○ “Distroless” containers
○ Ephemeral Containers

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Basics of a Secure Image
● Switch from root
● Assume no Linux userid
● Write only to volumes
○ emptyDir for “scratch space”
● Rebuild often
● Be mindful of where you get base images from
● Take contracts and policies into consideration

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Software Bill of Materials
● Multiple standards
● Stored with containers
● Signed?

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Farm-to-table Supply Chain Security
Container
registry
Proof

https://openunison.github.io / @mlbiam / @mlbiam@fosstodon.org
Farm-to-table Supply Chain Security
Test Lab Build Infrastructure

Connect with us
● Web - https://openunison.github.io / http://tremolo.io
● Twitter - @tremolosecurity / @mlbiam
● Masstodon - @mlbiam@fosstodon.org / @tremolo@hachyderm.io
● GitHub - http://github.com/tremolosecurity/ / http://github.com/openunison/