Architecting Container Infrastructure for Security and Compliance - CON406 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Architecting Container Infrastructure
for Security and Compliance
M i t c h B e a u m o n t , S o l u t i o n s A r c h i t e c t , A W S
K e l v i n Z h u , P r o d u c t i v i t y T e a m L e a d D e v e l o p e r , O k t a
C O N 4 0 6
N o v e m b e r 3 0 , 2 0 1 7

“If everything seems under control,
you're not going fast enough.”
—Mario Andretti
Picture: 1990 Indy 500 / DoctorIndy / License

Mario Andretti crashed in the 1992 Indianapolis 500. The reason?
He lost control of his car under acceleration and smashed into the wall.
Mario Andretti crashed in the 1992 Indianapolis 500.
The reason? He lost control of his car under
acceleration and smashed into the wall.
Picture: F1 Collision / TMWolf / License© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Image Security Credentials and Secrets
Kernel Security Denial of Service Container Breakouts
Architecting a secure infrastructure
Runtime Security
503

Kernel and Host
Security
Denial of
Service
Container
Breakouts
Image
Security
Secrets Runtime

Containers as opposed to VMs
Server
Host OS
Hypervisor
Guest OS Guest OS
Bins/Libs Bins/Libs
Cats App Cats App
Server
Host OS
Container Engine
Bins/Libs
CatsApp
CatsApp
Bins/Libs
DogsApp
DogsApp
Virtual Machines Containers

Containers and VMs
Server
Host OS
Container Engine
Bins/Libs
CatsApp
CatApp
Bins/Libs
DogsApp
DogsApp
Hypervisor

Amazon Machine Image builds
EC2 instance
ECS Optimized
AMI
• ECS Optimised Amazon Linux
• RHEL
• Ubuntu
• Container Centric OS

Amazon Machine Image builds
EC2 instance EC2 instance
Foundational AMI
• Security best
practices
• Provisioners
• Loggers
• Config, and so on
ECS Optimized
AMI
• RHEL
• Ubuntu

Amazon Machine Image Builds
EC2 instance
• RHEL
• Ubuntu
EC2 instance
Foundational
AMI
• Frameworks
• Applications
EC2 instance
Full Stack AMI
• Security best
practices
• Provisioners
• Loggers
• Config, and so on
ECS Optimized
AMI

Building an AMI pipeline
DevOps
Persona
Code
Repository

DevOps
Persona
Code
Repository
CI/CD
Pipeline

DevOps
Persona
Code
Repository
CI/CD
Pipeline
Base AMI
AMI
Builder
ECS
Optimized
AMI
Public
Catalog

DevOps
Persona
Code
Repository
CI/CD
Pipeline
Base AMI
AMI
Builder
ECS
Optimized
AMI
Public
Catalog
Customer
AMI

DevOps
Persona
Code
Repository
CI/CD
Pipeline
Base AMI
AMI
Builder
ECS
Optimized
AMI
Public
Catalog
CloudFormation
Customer
AMI

Limiting resources
E C S T a s k
C a t s
C o n t a i n e r
C o n t a i n e r I n s t a n c e
S e c u r i t y G r o u p
E C S T a s k
D o g s
C o n t a i n e r

Limiting resources
• Define your resource limits up front
• It’s not just memory and CPU.
• Monitor usage
• Leverage Auto Scaling

Segmentation
E C S T a s k
C a t s
C o n t a i n e r
E C S T a s k
D o g s
C o n t a i n e r

Segmentation – Placement constraints
aws ecs put-attributes --cluster mycatsanddogscluster --attributes
“name=CDE,value=true,targettype=container-instance,targetId=<blahblahblah>”

Segmentation – Placement constraints
aws ecs put-attributes --cluster mycatsanddogscluster --attributes
“name=CDE,value=true,targettype=container-instance,targetId=<blahblahblah>”
"placementConstraints": [
{
"expression":"attribute:CDE==true",
"type": "memberOf"
}]
Task Definition
Container
Instance

Segmentation- Placement constraints
E C S T a s k
C a t s
C o n t a i n e r
C D E
E C S T a s k
C a t s
C o n t a i n e r
E C S T a s k
D o g s
C o n t a i n e r
N o n - C D E
E C S T a s k
D o g s
C o n t a i n e r

Segmentation- Placement Constraints
E C S T a s k
C a t s
C o n t a i n e r
C D E
E C S T a s k
C a t s
C o n t a i n e r
E C S T a s k
D o g s
C o n t a i n e r
N o n - C D E
E C S T a s k
D o g s
C o n t a i n e r
This doesn’t
need to happen!

Segmentation – Task ENI
E C S T a s k
C a t s
C o n t a i n e r
S e c u r i t y
G r o u p
E C S T a s k
C a t s
C o n t a i n e r
S e c u r i t y
G r o u p
E C S T a s k
D o g s
C o n t a i n e r
S e c u r i t y
G r o u p
E C S T a s k
D o g s
C o n t a i n e r
S e c u r i t y
G r o u p

1. Pre ENI Attachment: The primary ENI
(eth0) is in the default namespace
Default/Root Global Namespace
docker0
eth0
lo

2. ENI Attached: The new ENI (eth1) is in
the default namespace.
Default/Root Global Namespace Default/Root Global Namespace
docker0 docker0
eth0
lo lo
eth0
eth1

Default/Root Global Namespace Default/Root Global Namespace
Default/Root Global Namespace
docker0 docker0
eth0
lo lo
eth0
eth1
lo
eth0
docker0
ecs0
eth1
lo ve-
c1
3. ENI Provisioned: The ECS Agent
invokes CNI plugins to move the new ENI
into a new namespace and configure it
with the addresses and routes.
2. ENI Attached: The new ENI (eth1) is in
the default namespace.

Capabilities
Kernel
Server
Cats Container Dogs Container
NET_BIND_SERVICE CHOWN
• AUDIT_CONTROL
• BLOCK_SUSPEND
• DAC_OVERRIDE
• MKNOD
• …
• …

”LinuxParameters": {
”capabilities": {
add: [“AUDIT_CONTROL”, ...],
drop: [“MKNOD”, ...]
}
}
docker run --cap-drop ALL --cap-add ..
Dropping and adding capabilities

Do you really know your image?

Layers, binaries, and dependencies, oh my!
/usr/bin/passwd
/usr/bin/gpasswd
/usr/bin/chsh
/usr/bin/chfn
/usr/bin/newgrp
/bin/su
/bin/umount
/bin/mount
$ find / -user root -perm -4000 -print
Do I need binaries with SUID flag?

Layers, binaries, and dependencies, oh my!
IMAGE CREATED CREATED BY SIZE
a8bdc7fdaa4f 2 weeks ago /bin/sh -c #(nop) CMD ["httpd-foreground"] 0B
<missing> 2 weeks ago /bin/sh -c #(nop) EXPOSE 80/tcp 0B
<missing> 2 weeks ago /bin/sh -c #(nop) COPY file:761e313354b918... 133B
<missing> 2 weeks ago /bin/sh -c set -eux; buildDeps=" bzip2... 9.72MB
<missing> 2 weeks ago /bin/sh -c #(nop) ENV APACHE_DIST_URLS=ht... 0B
<missing> 2 weeks ago /bin/sh -c #(nop) ENV HTTPD_PATCHES= 0B
<missing> 2 weeks ago /bin/sh -c #(nop) ENV HTTPD_SHA256=777753... 0B
<missing> 2 weeks ago /bin/sh -c #(nop) ENV HTTPD_VERSION=2.4.29 0B
<missing> 4 weeks ago /bin/sh -c apt-get update && apt-get inst... 44.2MB
<missing> 4 weeks ago /bin/sh -c { echo 'deb http://deb.debian... 161B
<missing> 4 weeks ago /bin/sh -c #(nop) ENV OPENSSL_VERSION=1.0... 0B
<missing> 4 weeks ago /bin/sh -c #(nop) ENV NGHTTP2_VERSION=1.1... 0B
<missing> 4 weeks ago /bin/sh -c #(nop) WORKDIR /usr/local/apache2 0B
<missing> 4 weeks ago /bin/sh -c mkdir -p "$HTTPD_PREFIX" && ch... 0B
<missing> 4 weeks ago /bin/sh -c #(nop) ENV PATH=/usr/local/apa... 0B
<missing> 4 weeks ago /bin/sh -c #(nop) ENV HTTPD_PREFIX=/usr/l... 0B
<missing> 4 weeks ago /bin/sh -c echo 'deb http://deb.debian.org... 55B
<missing> 4 weeks ago /bin/sh -c #(nop) CMD ["bash"] 0B
<missing> 4 weeks ago /bin/sh -c #(nop) ADD file:55b071e2cfc3ea2... 123MB

Best practices
• Signing container images (Docker content trust)
• Set filesystems to be read-only (readonlyRootFilesystem)
• Remove setuid/setgid binaries from images (defang)
• Set containers to run as non-root user
• Consider running static binaries

Image security
Trusted Images
• ECR
• docker-hub-enterprise
• elastic.io/running-a-
docker-private-registry-
on-ec2
Minimal OS Base
Images
• Minimum OS (alpine etc)
• Docker cis-docker-bench
• Image signing
Container
Vulnerability
Scanning
• TwistLock
• Clair
• NueVector

DevSecOps container pipeline
Developers Security Engineers Ops Engineers
AWS CodeCommit
Task Definition
Dockerfile
FROM centos:centos7
MAINTAINER cb@demo.com
RUN yum -y update
RUN yum -y install openssh-
server U
SER sshduser
EXPOSE 5432
ENTRYPOINT sshd

Docker image
AWS
CodeBuild
Task Definition
AWS CodeCommit
Dockerfile
FROM centos:centos7
RUN yum -y update
server U
SER sshduser
EXPOSE 5432
ENTRYPOINT sshd

Docker image
AWS
CodeBuild
Validate Configuration > Merge >
 python ./check_dockerfile.py
./examples/Dockerfile-demo
|jq ".warnings.warnings[].message"
"yum clean all is not used"
"installing SSH in a container is not recommended"
"No 'USER' instruction"
AWS CodeCommit
Task Definition
Dockerfile
FROM centos:centos7
RUN yum -y update
server U
SER sshduser
EXPOSE 5432
ENTRYPOINT sshd

Docker image
AWS
CodeBuild
AWS CodeCommit
0
50
100
150
DEV INT
TEST QA
PROD
Vulnerabilities
Low Medium High
Scan Docker Image > Publish >
Task Definition
Dockerfile
FROM centos:centos7
RUN yum -y update
server U
SER sshduser
EXPOSE 5432
ENTRYPOINT sshd

Amazon ECR
Docker image
AWS
CodeBuild
AWS CodeCommit
0
50
100
150
DEV INT
TEST QA
PROD
Vulnerabilities
Low Medium High
Scan Docker Image > Publish >
Task Definition
Dockerfile
FROM centos:centos7
RUN yum -y update
server U
SER sshduser
EXPOSE 5432
ENTRYPOINT sshd

Storing secrets in environment variables
• Suggested by 12-factor apps (III. Config)
• Environment variables can be seen in too many
places
• Linked containers
• ECS API calls
• Docker inspect
• Can’t be deleted
https://12factor.net/

Protecting secrets using IAM roles for
tasks
Benefits
• Simplify usage of AWS SDKs in containers
• Credential isolation between tasks/container
• Authorization per task/container
• Auditability in Amazon CloudTrail with taskArn

IAM EC2 roles
Cats
Contai ner
Dogs
Contai ner
E C S T a s k E C S T a s k
C a t s B u c k e t
D o g s
B u c k e t

IAM task roles
Cats
Contai ner
Dogs
Contai ner
E C S T a s k E C S T a s k
C a t s B u c k e t
D o g s
B u c k e t

IAM roles for tasks, explained
1. ECS agent periodically queries ECS control plane
2. Control plane generates ID token
 Auto-rotated
3. ECS agent:
 Constructs HTTP URL for each container
 Sets AWS_CREDENTIALS_ENDPOINT in HostConfig
4. AWS SDK extracts URL

Amazon S3-based secrets storage
• Secrets stored in S3 bucket
• Accessed via IAM roles for EC2 or IAM roles for tasks
• Enforce encryption at rest and flight via IAM policies and KMS
• Use VPC endpoint for S3 to lock down access from certain VPCs

Amazon EC2 systems manager parameter
store
• Secrets stored in SSM parameter store
• Accessed via IAM roles for EC2 or IAM roles for tasks
• Enforce encryption at rest and flight via IAM policies and KMS
• Govern permission to decrypt using specific KMS through IAM policy

Using IAM roles to retrieve secrets
Cats
Contai ner
E C S T a s k
Parameter Store
secretStringAWS KMS

VPC flow logs and Task ENI
E C S T a s k
C a t s
C o n t a i n e
r
S e c u r i t y
G r o u p
E C S T a s k
D o g s
C o n t a i n e
r
S e c u r i t y
G r o u p
630247214269 eni-0123456a 10.0.1.221
10.76.2.101 27039 22 6 5 268 1466491141
1466491200 REJECT OK
22
StopTask: {
task: “52c…”
}
$ docker diff / inspect

Access Control Cloud Native
Firewalling
Runtime Defense
Compliance Vulnerability
Management

AWS re:INVENT
Container Security at Okta
K e l v i n Z h u – P r o d u c t i v i t y T e a m L e a d D e v e l o p e r

Topics
• How we use containers
• How we secure:
• Hosts
• Containers
• Images

What security is to us

Keeping it reliable
• Security breaches have the potential to bring down our infrastructure
• Improper use of secrets can cause unexpected and damaging effects
• Bad security results in costly problems! But with automation it doesn’t have to
be hard!
• We’ll focus on a few examples of how we apply security principles to our Docker
containers

How we use containers

High-level flow

High level flow
• Developer writes code and commits to GitHub
• Bacon Web application receives commit and processes into test runs
• Test runs inserted as messages to an AWS SQS queue
• ECS cluster of instances pick up messages from SQS queue and run them on
Docker containers
• Test results reported back to Bacon Web application

Container generation and usage

Container generation and usage
• Individual runs generate artifacts, some of which are Docker artifacts
• Docker images uploaded to repositories
• Individual Amazon ECS clusters brought up by AWS CloudFormation for either CI
or application images
• Application images deployed to Amazon ECS services

Deployment scale
• Two different types of deployments:
• CI scales up to thousands and down to just a few containers according to
messages in queue
• One time use containers that finish a task within an hour
• Almost no containers necessary overnight and during weekends
• Applications scale up and down according to web traffic
• Longer lived containers

Container host security

Components of host security
• AMI
• Accessibility
• Ease of change

Locking down the AMI
• The very packages we install are potential threats!
• Viruses
• Backdoors
• Creation of a hardened AMI
• Even if an installed package is unused, if a vulnerability is discovered
patching is required!
• AMI with only minimum necessary to run application
• Follow Center of Internet Security (CIS) checklists to ensure generated
AMIs are secure

Locking down the AMI
Steps to build, test, and certify an AMI
• Define packer template
• Test: run CIS
• Share foundation AMI to DMZ account
• Share foundation AMI to other AWS
accounts
• Delete foundation AMI from originating
AWS account

Preventing unauthorized access

Preventing unauthorized access
• All hosts behind the VPC
• Only users with VPN can communicate with hosts
• Security groups to limit approved traffic between hosts
• SSH keys given only to trusted set of users
• Regular rotation to prevent leaking

Ease of change
• Always prepare for the worst case
• Zero days!
• Time is of the essence in minimizing damage
• To enable easy patching, AMI creation is hooked into the CI system
aws ec2 run-instances --count 1
aws ec2 wait instance-status-ok --instance-id ${INSTANCE_ID}
aws ec2 create-image --instance-id ${INSTANCE_ID}
aws ec2 wait image-available --image-ids ${IMAGE_ID}
aws ec2 terminate-instances --instance-ids ${INSTANCE_ID}

Malicious users

Malicious users
UpdatePolicy:
AutoScalingRollingUpdate
MaxBatchSize: 5
MinInstancesInService: 5
PauseTime: “PT10M”

Ease of change
• AWS CloudFormation for each Amazon ECS service
• Auto Scaling used to reduce reliance on individual hosts staying alive
• If a host becomes compromised or starts failing, terminate it
and allow a new one to come up
• Auto replacement of all hosts every day ensures manual
patches do not occur

Container security

Credentials on containers

Credentials on containers
• Industry standards
• Pull credentials from external protected storage encrypted both on disk
and in transit into flat file available within container
• Pull credentials from external protected storage encrypted both on disk
and in transit into memory (slightly better)
• Both approaches allow anyone with access to code run in container to retrieve the
secrets!

Protecting creds
• The next step up!
• Containers run in protected networks to prevent unauthorized direct
access
• Standard and parameterized code that is run on containers and
modification disallowed
• Sanitize logs and output files for any retrieved secrets
• Prevents ability to sniff out secrets and prevents leaks

Service access

Minimizing service access
• Containers need access to services, but only the minimal set they actually need!
• Task definitions associated with each Amazon ECS task/service type given their
own task IAM role
• Avoid use of “*” in access policies!
• Access advisor – Remove access to unnecessary services
• CloudTrail logs – More granular view into exactly which resources are
accessed

Image security

Docker within Docker!
• By mounting Docker from the host to the container, we can run Docker and
Docker Compose inside the container
• -v /usr/bin/docker:/usr/bin/docker
• Utilized in the CI system to test Docker images the same way the are deployed
• Allows repeatable building of Docker images in ECS within immutable containers

Image creation and usage

Image creation and usage
• No repository code built into images
• First step: pull down credentials (locked behind IAM roles)
• Second step: clone code or artifacts to actually run onto images
• Prevents leakage of information if images are stolen
• Even if image stolen, running the image will fail if user does not have the
right IAM role to get credentials

Packages and registries
• Images build off of internal base image configured to only pull from internal
package repository
• Packages in internal repository vetted by security team
• Installed packages pinned to specific versions
• Amazon ECR
• More performant so utilize as storage for security approved images that
are used on clusters
• Artifactory
• Storage for intermediate images and artifacts for longer-term storage

Key takeaways

Key takeaways
• Protect from unauthorized access
• VPC and security groups to limit approved network traffic
• IAM roles to limit per task access to creds and services
• Minimize surface area for problems
• Limit packages installed
• Be prepared to fix issues
• Constant patching necessitates agile change infrastructure

Thank you!

Architecting Container Infrastructure for Security and Compliance - CON406 - re:Invent 2017

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Architecting Container Infrastructure for Security and Compliance - CON406 - re:Invent 2017

Semelhante a Architecting Container Infrastructure for Security and Compliance - CON406 - re:Invent 2017 (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Architecting Container Infrastructure for Security and Compliance - CON406 - re:Invent 2017