As presented at LinuxCon/ContainerCon 2016:
Cyber threats consistently rank as a high priority for data center operators and their reliability teams. As increasingly sophisticated attacks mount, the risk associated with a zero-day attack is significant. Traditional responses include perimeter monitoring and anti-malware agents. Unfortunately, those techniques introduce performance and management challenges when used at large VM densities, and may not work well with containerized applications.
Fortunately, the Xen Project community has collaborated to create a solution which reduces the potential of success associated with rootkit attack vectors. When combined with recent advancements in processor capabilities, and secure development models for container deployment, it’s possible to both protect against and be proactively alerted to potential zero-day attacks. In this session, we’ll cover models to limit the scope of compromise should an attack be mounted against your infrastructure. Two attack vectors will be illustrated, and we’ll see how it’s possible to be proactively alerted to potential zero-day actions without requiring significant reconfiguration of your datacenter environment.
Technology elements explored include those from Black Duck, Bitdefender, Citrix, Intel and Guardicore.
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Using hypervisor and container technology to increase datacenter security posture
1. Using Hypervisor and
Container Technology to
Increase Datacenter
Security Posture
LinuxCon North America 2016 – Toronto Canada
2. #whoami – Tim Mackey
Current roles: Senior Technical Evangelist; Occasional coder
• Former XenServer Community Manager in Citrix Open Source Business Office
Cool things I’ve done
• Designed laser communication systems
• Early designer of retail self-checkout machines
• Embedded special relativity algorithms into industrial control system
Find me
• Twitter: @TimInTech ( https://twitter.com/TimInTech )
• SlideShare: slideshare.net/TimMackey
• LinkedIn: www.linkedin.com/in/mackeytim
9. Knowledge is Key. Can You Keep Up?
glibc
Bug
Reported
July 2015
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based
buffer overflow
10. Knowledge is Key. Can You Keep Up?
glibc
Vuln
Introduced
May 2008
glibc
Bug
Reported
July 2015
CVE-2015-
7547
CVE
Assigned
Feb 16-2016
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based
buffer overflow
11. Knowledge is Key. Can You Keep Up?
glibc
Vuln
Introduced
May 2008
CVE-2015-
7547
CVE
Assigned
Feb 16-2016
glibc
Bug
Reported
July 2015
National
Vulnerability
Database
Vuln
Published
Feb 18-2016
Moderate Security Risk
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based
buffer overflow
12. Knowledge is Key. Can You Keep Up?
glibc
Vuln
Introduced
National
Vulnerability
Database
Vuln
Published
You
Find It
May 2008
CVE-2015-
7547
CVE
Assigned
Feb 16-2016 Feb 18-2016
glibc
Bug
Reported
July 2015
Patches
Available
You
Fix It
Highest Security Risk
Moderate Security Risk
Low Security Risk
Vuln: CVE-2015-7547: glibc getaddrinfo stack-based
buffer overflow
23. Virtual Switches as Local Edge Protection – Silent Block
Guest
VM
SSL access
Attack silently blocked
Virtual Switch Rules
Ingress:
HTTPS public
Egress:
Dynamic port to origin
MySQL internal
Private CIDR internal
Port 22 access
24. Virtual Switches as Local Edge Protection – Traffic Monitor
Guest
VM
SSL access
Attack blocked with traffic log
Virtual Switch Rules
Ingress:
HTTPS public
Egress:
Dynamic port to origin
MySQL internal
Private CIDR internal
Port 22 access
ovs Controller
Log SSH Port 22 access
Create port mirror for attacker
Traffic
Monitor
Virtual Switch Rules
Ingress:
HTTPS public
Egress:
Dynamic port to origin
MySQL internal
Private CIDR internal
Mirror:
Port 22 to Traffic Monitor
All attacker traffic to monitor
25. Guest
VM
Virtual Switches as Local Edge Protection – Quarantine
Guest
VM
SSL access
Attack quarantined with full log
Virtual Switch Rules
Ingress:
HTTPS public
Egress:
Dynamic port to origin
MySQL internal
Private CIDR internal
Port 22 access
ovs Controller
Log SSH Port 22 access
Create port mirror for attacker
Quarantine VM for attacker use
Trigger replacement VM for farm
Traffic
Monitor
Virtual Switch Rules
Ingress:
HTTPS attacker
Egress:
Dynamic port to origin
Mirror:
Port 22 to Traffic Monitor
All attacker traffic to monitor
29. Container Use Cases
Application containers
• Hold a single application
• Can follow micro-services, cloud native design pattern
• Starting point for most container usage
• Short lifespan, many per host
System containers
• Proxy for a VM
• Insulate against core operating system
• Perfect for legacy apps
• Long lifespan, few per host
MySQL
Tomcat
nginx
Kernel
MySQL
Tomcat
nginx
Kernel
31. Trust Container Source
Atomic Host
AtomicApp
AtomicApp
AtomicNulecule
AtomicNulecule
RedHat Registry
MySQL
Redis
Jenkins
Docker Hub
DockerContainer
DockerContainer
DockerContainer
DockerContainer
DockerContainer
Third Party and Custom
Problem: Who to trust, and why?
• Trusted source?
• Unexpected image contents
• Locked application layer
versions (e.g. no yum update)
• Layer dependencies
(monolithic vs micro-services)
• Validated when?
32. Determine Who Can Launch A Container
Container default is root access
• RBAC/ABAC is orchestration specific
Docker Datacenter
• Universal Control Plane
• RBAC – LDAP/AD/local users
• Full/Restricted/View/None
Kubernetes
• Authorization modules
• Admission controllers
33. Define Sensible Container Network Policies
Docker default network is Linux Bridge
Access policy defined in iptables
• Based on Docker daemon startup
External communication on by default
• -- iptables=off to disable iptables modification
Inter container communication on by default
• -- icc=false to disable inter container communication
• -- link=CONTAINER_NAME_or_ID:ALIAS with EXPOSE ports from Docker file
• All inter-container/cross host communication is external
`docker network` command simplifies aspects of network design
• Create user defined networks, including overlay networks
• docker network create --driver bridge sql
35. Host
Kubernetes Networking - Example
Kubernetes Network
eth0/10.204.136.20
Pod
Container
Pause
Container
Container
veth0/10.204.136.21
Pod
Container
Pause
Container
Container
veth0/10.204.136.22
Host
Kubernetes Network
eth0/10.204.136.10
Pod
Container
Pause
Container
Container
veth0/10.204.136.11
PodContainer
Pause
Container
Container
veth0/10.204.136.12
36. Limit the Scope of Compromise
• Enable Linux Security Modules
• SELinux
• --selinux-enabled on Docker engine, --security-opt=“label:profile”
• AppArmor
• -- security-opt=“apparmor:profile”
• Apply Linux kernel security profiles
• grsecurity, PaX and seccomp protections for ALSR and RBAC
• Adjust privileged kernel capabilities
• Reduce capabilities with --cap-drop
• Beware –cap-add and –privileged=false, and CAP_SYS_ADMIN
• Use a minimal Linux Host OS
• Atomic host, CoreOS, RancherOS
• Reduce impact of noisy neighbors
• Use cgroups to set CPU shares and memory
38. Risk Mitigation Shrinks Scope of Compromise
Open source license compliance
• Ensure project dependencies are understood
Use of vulnerable open source components
• Is component a fork or dependency?
• How is component linked?
Operational risk
• Can you differentiate between “stable” and “dead”?
• Is there a significant change set in your future?
• API versioning
• Security response process for project
39. 7 of the top 10
Software Companies
(44 of the top 100)
6 of the top 8
Mobile Handset Vendors
6 of the top 10
Investment Banks
24
Countries
250+
Employees
1,800Customers
Who is Black Duck Software?
27Founded
2002
40. 8,500
WEBSITES
350
BILLION LINES OF CODE
2,400
LICENSE TYPES
1.5
MILLION PROJECTS
76,000
VULNERABILITIES
• Largest database of open source project
information in the world.
• Vulnerabilities coverage extended through
partnership with Risk Based Security.
• The KnowledgeBase is essential for identifying
and solving open source issues.
Comprehensive KnowledgeBase
41. Black Duck Hub Security Architecture
Hub Scan1 File and Directory Signatures2 Open Source
Component Identified
3
Hub Web Application
Black Duck
KnowledgeBase
On Premises Black Duck Data Center
42. We Need Your Help
Knowledge is power
• Know what’s running and why
• Define proactive vulnerability response process
• Don’t let technology hype cycle dictate security
Invest in defense in depth models
• Don’t rely on perimeter security to do heavy lifting
• Do look at hypervisor & container trends in security
• Make developers and ops teams part of the solution
• Focus attention on vulnerability remediation
Together we can build a more secure data center
Editor's Notes
http://www.istockphoto.com/photo/computer-crime-concept-gm516607038-89059287?st=9174601
http://www.verizonenterprise.com/verizon-insights-lab/dbir/2016/
Every year since 2008, Verizon have published a report on the attempted data breaches occurring within their data centers. For 2015, they found close to 90% of them had either a financial or espionage component to them. This report is well worth the read, and there are a few key findings in this report we should all be aware of.
Costs of data breaches are heavily skewed towards legal consultation and forensics, and not to the public components of credit monitoring or lawsuits
Despite some vulnerabilities having been public for years, there remain vulnerable components in use
Some of those components simply may not have a patch forthcoming for a variety of reasons.
Despite years of organizations spending energy protecting against attacks, it remains up to the attacker to define what’s valuable. Consider the case of ransomware. A police department in the town next to where I live was subjected to a raonsomeware attack. For roughly 500 USD in bitcoin, the attackers would decrypt the booking and evidence records they had just crypto locked. As an attacker, they likely had no knowledge of who they had attacked or what they had locked up. What mattered was the ransom, and that they had a police organization’s files didn’t factor into the equation.
Big Idea: According to SAP research, most cyber attacks target the application layer, yet most security investment has been at the network layer. To best protect themselves from security breaches most IT organizations don’t necessarily need to spend more. They just need to spend smarter, investing in the areas that constitute the greatest risk.
Question: How does your organization allocate IPSec spending? What % of your budget goes to application security v. network security? Why?
Let’s take a little bit of time and look at how an attack is created. Potential attackers have a number of tools at their disposal, and use a number of different tactics. In this case, the attacker wishes to create an attack on a given component. In order to be effective, they have two primary models. First they can actively contribute code in a highly active area of the component with an objective of planting a back door of some form. The hope being that their code will fail to be recognized as suspect given how quickly the area of code is evolving.
Second they can look for areas of code which are stable, and the longer they’ve bene stable, the better. The reason for this is simple, old code is likely written by someone who isn’t with the project any longer, or perhaps doesn’t recall all assumptions present at the time the code was written. After all, its been long understood that even with the best developers, assumptions change and old code doesn’t keep up.
The goal in both cases being to create an attack against the component, so they test, and fail, and iterate against the component until they’re successful or move on. Assuming they’re successful, they create a deployment tool and document the tool for others. Of course, given the publicity received by some recent vulnerabilities, a little PR goes a long way.
Now there are responsible researchers who follow a similar workflow, and they legitimately attempt to work with component creators to disclose vulnerabilities. They too will publish results, but are less interested in creating the an attack beyond a proof of concept.
http://www.istockphoto.com/photo/person-in-hooded-sweater-using-a-laptop-on-wooden-table-gm464503138-58544934?st=cf78f31
http://www.istockphoto.com/photo/cloud-computing-gm518556682-90104967
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html
https://sourceware.org/bugzilla/show_bug.cgi?id=18665
https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT)
http://cve.mitre.org/cve/cna.html
https://openclipart.org/detail/200681/primary-patch
https://www.youtube.com/watch?v=hkryI6eapOA
Imagine a situation where you understand what the world looks like. Everything is green and wonderful, and then you wake up one morning to find that the world was very different.
Imagine a situation where you understand what the world looks like. Everything is green and wonderful, and then you wake up one morning to find that the world was very different.
Image taken by Tim Mackey
Good reference for why this matters: http://www.rekall-forensic.com/posts/2014-10-03-vms.html
The key to this technology is using the hypervisor, which already knows how to isolate VMs from one another in a performant way, to provide a virtual appliance, provided by a security vendor with the context required to monitor critical memory activity inside the VMs it’s protecting.
Just a little bit different….
Containers are in Pods, and all containers in the pod have equal access to each other as if they are on the same host. Effectively, this means that each container in a pod can speak to another using localhost with some port. In the event the container needs to speak externally, then all containers in the pod have the same address. How is this accomplished?
A kubernetes specific container named “pause” is created. This special container by default presents the configured network to the outside world and is used to ensure the network namespace is intact. It’s not a proxy, nor a gateway, but simply exists to ensure the subnet is present on whatever network topology you’ve chosen to implement.
If you assume from the outset your containers will be compromised, what would you do differently? What could you do to make life much harder to mount an attack from a compromised container?
The first item everyone should have on their list is to enable the SELinux and AppArmor security modules for the distro you’re using as a base. For SELinux, you specify that it’s enabled on the command line for Docker Engine, and then on the container you specify a security option representing the profile you want to use.
Now that you have some security modules in place, you’re in better shape, but you can still make it much harder to mount a proper attack. Consider for example an attacker who recognizes they’re in a container, and assumes that means there probably are multiple containers with the same profile. Armed with that knowledge, one of the easiest ways to make the life of an attacker harder is to randomize the memory load location. That’s where kernel security profiles come into play. Grsecurity, PaX and seccomp enable roles based access and address randomization upon load capabilities. Net of this, where the executable code lives in memory changes, and that makes it harder to know if you’ve created a viable attack or not.
Now the Docker people have done a great job over the years in locking down the level of privileged access you get from within a container. In part, this is done using what’s known as kernel capabilities. Kernel capabilities offer a granular level of control over the types of operations you want the kernel to perform. Consistent with the concept of least privilege, you don’t want to ask for more rights than you need. If you find the defaults are providing more access than your application requires, you can pare things back using the –cap-drop option. Of course, it’s entirely possible you might need more rights, but if you find you need to disable priviledged access, or want to set the CAP_SYS_ADMIN flag, beware you’re effectively giving the container the equivalent of root access.
Lastly, from a security perspective, you can choose to use a minimal Linux distro such as Atomic or CoreOS, but you still will want to pay attention to the options I’ve just outlined.
So now that you’ve limited the access a potential attacker has to system services, you still have to contend with other types of havoc. A perfect example of this is the concept of a noisy neighbor. Most of us have had the experience of having someone in a neighboring space behave in an annoying manner. In the case of computing, the annoying neighbor can be one which consumes excessive memory or processing time. Limiting the scope of interference is very easy. All you need to do is define some CPU shares and memory limits for the container and set them during launch.
Hub is based on a 3 part architecture:
Scan Client – scans directories and artifact files creating “code prints” that uniquely but confidentially identify the files & directories contained in them
Web Application – This is the main user interface and logic center for Hub.
KnowledgeBase – This is a repository for open source component, license, and vulnerability information
The code prints recoreded by the scan client are compared to reference code prints in the KnowledgeBase to identify open source components, versions, and origins. No source or binary code is ever uploaded.