While there have been many improvements around securing containers, there is still a large gap in monitoring the behavior of containers in production. That is the reason we created Falco, the open source behavioral activity monitor for containerized environments. Falco can detect and alert on anomalous behaviour at the application, file system and network level. In this session we get a deep dive into Falco and explain the following points: * How does behavioral security differ from existing security solutions like image scanning, seccomp, SELinux or AppArmor? * How Falco does its magic? * What Falco can detect? Creating your own rules and customize the existing ones for your Kubernetes applications. * How to deploy Falco in your Kubernetes cluster? * Reacting to security incidents, what we can do to stop the attackers in real-time? * Post-mortem analysis and forensics on containers with Sysdig Inspect. Even when containers does not exist anymore!

1. Néstor Salceda, Integrations Engineer
LibreCon Bilbao, Nov 22th 2018
Securing your
Kubernetes
applications

2. @nestorsalceda
• Open Source enthusiast
• Security and Monitoring passionate
• I work at Sysdig
• Daddy of twins
• Kubernetes member: Maintainer of Sysdig and
Falco Helm charts
• Top 3 Contributor to Falco
• Judo, Aikido and other Gendai Budo martial arts
lover and practicioner

3. Anomaly Detection in run-time: Falco
Active Security: Kubernetes Response Engine
Forensics: Sysdig Inspect
Current challenges of Container Security
Agenda
Layers of Container Security

4. •
Container
Security
Challenges

5. Breaches may extend for days or weeks before
detected
Attacks are changing to abuse activities rather than data
exfiltration (crypto mining)
Ephemeral nature of containers means that in the event
of a security breach you may never know
Many security paradigms are still reactive
Main Challenges

6. •
Layers of
Container
Security

7. Which layers?
Runtime
Build
Infrastructure

8. Networking: Filtering, Istio, Calico ...
Cluster Security: RBAC, Audit Events, Security Policies,
Affinity, Network Policies ...
Container Runtime: SELinux, AppArmor, CIS
Benchmarks, InSpec ...
Host Security: SecComp, SELinux, AppArmor, Resource
Constraints ...
Infrastructure

9. Vulnerability Management:
● Image Scanning: Sysdig Secure, Anchore, Clair
● Upstream OS
● Application Vulnerabilities
Image / Software Origin:
● Signed Images / Layers
● Artifact Signing
● Trusted Registries
Build

10. Secure Secrets: How secrets are stored or used?
Anomaly Detection: Someone altered my runtime
environment?
Forensics: What happened if compromised?
Service / Container Admission: What is allowed to run?
Runtime

11. Processes are “scoped” as to what’s expected
Container images are immutable, runtime
environments often aren’t
How do you detect abnormal behavior?
See containers like isolated processes
Anomaly Detection

12. Containers are highly volatile: Imagine Grisom doing
CSI stuff without the corpse
What did happen inside the container?
When a security incident has already happened
Forensics

13. What is Falco?

14. • Detects suspicious activity
defined by a set of rules
• Uses Sysdig’s flexible and
powerful filtering
expressions
Behavioral
Activity
Monitor
• Uses Sysdig’s container
and orchestrator support
Full Support of
Containers
Orchestration
Flexible
Notification
Methods
Open
Source
Software
• Files
• STDOUT
• Syslog
• Execute other programs
• And more ...
• CNCF Sandbox Project
• Welcome contributions
• Transparency &
Governance

15. falco_probe
Kernel
Module
Kernel
User
Syscalls
Sysdig Libraries
Events
Alerting
Falco Rules
Suspicious
Events
File
Syslog
Stdout
Filter Expression
Shell

16. Filter expressions
A shell is run in a container container.id != host and proc.name = bash
Overwrite system binaries
fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
and write
Container namespace change
evt.type = setns and not proc.name in
(docker, sysdig)
Non-device files written in /dev
(evt.type = create or evt.arg.flags contains O_CREAT) and proc.name
!= blkid and fd.directory = /dev and fd.name != /dev/null
Process tries to access camera
evt.type = open and fd.name = /dev/video0 and not proc.name
in (skype, webex)

17. Rules
- macro: bin_dir
condition: fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
- list: shell_binaries
items: [bash, csh, ksh, sh, tcsh, zsh, dash]
- rule: write_binary_dir
desc: an attempt to write to any file below a set of binary directories
condition: bin_dir and evt.dir = < and open_write and not package_mgmt_procs
output: "File below a known binary directory opened for writing
(user=%user.name command=%proc.cmdline file=%fd.name)"
priority: WARNING

18. More rules implemented in draios/falco-extras repository:
● Traefik
● Redis
● Nginx
● PostgreSQL
● ...
Falco ships with a nice default ruleset for best practices:
● Writing files in bin or etc
● Reading sensitive files
● Terminal spawning in a container
● ...
Batteries included

19. Requests made by anonymous user
Attach to cluster-admin Role
Service Account Created in Kube Namespace
Create / Modify ConfigMaps which exposes secrets
K8s Audit Events Support

20. Try it out!
$ helm install --name sysdig-falco-1
--set fakeEventGenerator.enabled=true
stable/falco

21. Active
Security

23. See it in action!

24. Start a capture
Network isolate
Demisto/Phantom integration
Delete a pod
Playbooks Available
Forbid that a node schedules more pods
Slack notification

25. Forensics

26. Correlate events to reconstruct the attack
Blameless Post-Mortem incident report
Capture system calls using Sysdig
Forensics

27. The ephemeral nature of containers changes the rules
Security offers us an opportunity to be proactive
Containers add more infrastructure, layers and risks. But
we have seen same security threats before: DDoS,
Injections ...
Just a quick summary

28. Do you want work with me?
Monitoring / Security Open Source Remote

29. Blog
https://www.sysdig.com/blog/tag/falco
Sysdig Secure
https://www.sysdig.com/product/secure
Website
https://www.sysdig.com/opensource/falco
https://falco.org
Join the community
Public Slack
https://slack.sysdig.com
https://slack.sysdig.com/messages/falco

30. Docker Hub
https://hub.docker.com/r/falcosecurity/falco
GitHub
https://github.com/falcosecurity/falco
Learn more
Wiki
https://github.com/falcosecurity/falco/wiki
Sysdig Docker Usage Report 2018
https://sysdig.com/blog/2018-docker-usage-report

31. Eskerrik Asko
Questions?
nestor.salceda@sysdig.com
@nestorsalceda