Slides from my talk at AWS Tech Community Days 2017 in Köln, Germany

1. AWS + KUBERNETES = ❤
Contentful Engineering
Anthony Stanton

2. Anthony Stanton
Infrastructure Developer
Product Owner for
Infrastructure and Reliability
Joined 2014, previously worked
at a Managed Service Provider
(Networks & Hosting)

3. Like a CMS… without the bad bits.
Contentful is a content management developer
platform with an API at its core.

4. Content infrastructure for projects of any size

5. Microservices architecture with a global CDN

6. Contentful Engineering
Contentful: ~ 130 in Berlin and San Francisco
Product and Engineering: ~ 50 in 10 cross-functional
teams, all in Berlin
Infrastructure Team: 5 (also doing some product dev)

7. Agenda

8. Why should you listen to this talk?
We're a small org developing a "cloud native" application
We've made a bet on Kubernetes as a deployment platform
We want to share our experience!

9. Agenda
1. Why we chose to move to Kubernetes
2. Kubernetes in AWS
3. Migrating
4. Lessons Learned

10. Before Kubernetes

11. Previous (2015-2017) Platform
Autoscaling Groups of EC2 instances
Service Discovery via EC2 tags, some Lambda functions for interacting
with them
Chef Solo deploying applications as Debian packages
Managed datastores: RDS, ElastiCache, S3
All AWS resources progressively added to Terraform

12. Terraform
Declarative cloud infrastructure
Same space as AWS CloudFormation
Works with multiple cloud providers
● AWS
● Fastly
● NS1
● Librato
● Sentry

14. Why change at all?
Feature development velocity and time to market are key
Our development teams were growing
We wanted a delivery-focused platform that didn't require deep
knowledge of Chef for developers to be productive

15. Why containers?
Containers were already in heavy use in development and continuous
integration
They provide a useful operational model for delivering applications in a
standardised way

16. Why Kubernetes?

17. Kubernetes advantages
Feature set focused on enabling application delivery
Open source development model
Rate of change (updates and improvements)
Community and market share
But still on AWS!

18. Kubernetes on AWS

19. Kubenet Networking
Kubenet: simplest networking model available (No overlay network with
more complex SDN)
Each cluster has its own routing table (max 200 per VPC)
Each worker node is assigned a /24 CIDR block for Pods and Services
So each worker node has a routing table entry (max 50 per RT)

21. Cluster Management: Kops
Kubernetes Operations

22. Cluster Management: Kops
Manages Kubernetes clusters
EC2 autoscaling groups for, IAM policies, routing
Handles cluster creation, configuration updates and rolling upgrades
Can effect changes directly to AWS resources, or produce Terraform
output

23. Kops + Terraform
With Terraform you can override
machine-generated configuration with
"overrides"
This allows us to customise what Kops generates
Apply additional security groups or IAM policies
Change the AMI used

24. File-based configuration for Kops
Declarative
Easiest way to deal with all the things you want to customise
Works well with our workflows using git and code review

26. CoreOS Container Linux
Minimise attack surface
Simplify container host OS management
Auto-updates for security patches

27. AWS Services Kubernetes uses
ECR for private container registry
EBS for StatefulSets (persistent storage for applications)
ELB for LoadBalancer Services

28. Unlock ELB magic with Annotations
Source: https://gist.github.com/mgoodness/1a2926f3b02d8e8149c224d25cc57dc1

29. What about IAM profiles?
ECS can use IAM profiles for AWS API access, so cool!
Kube2iam lets you map those onto Kubernetes objects

30. kube2iam example

31. Migrating

32. Service Discovery
We had no real Service Discovery in the original setup
Services migrated to Kubernetes exposed via LoadBalancer services
(implemented by ELB)
Announced via the VPC's Route53 Private DNS zone

33. debian
package
Deployment
commit
docker
image
code
repository
legacy
platform
package
archive
docker
image
debian
package
kubernetes
cluster
image
registry
deployment
artifact
promotion
continous
integration

34. Lessons Learned

35. All this is very new and changing
Kubernetes itself releases pretty frequently
Lots of times improvements and bug fixes are backported to the stable
release
Kops releases sometimes lag a little behind Kubernetes
Stay up to date!

36. Kops defaults to creating a new VPC
Extra elements required for every cluster
New VPC peering, new routes, new CIDRs to add to Security Groups
We rely on Route53 Private DNS zones for Service Discovery, so entries
must be replicated in every Private DNS zone
networkId parameter lets you specify an existing VPC, life is much easier
We are now deploying all clusters in the same VPC

37. Kops doesn't know about your other AWS
But Kops + Terraform = ♥
Terraform overrides let you customise Kops-generated resources and refer
to them in other resources (Route53 records, etc)

38. Logging
Kubernetes generates logs
More logs
Still more logs!
Expect your log volume to grow considerably,
use logLevel in Kops to reduce the default
verbosity of Kubernetes components

39. Monitoring
Kubernetes Heapster captures CPU, RAM and disk usage metrics per Pod
More metrics
Still more metrics!
This is currently overloading our application metrics visualisation tool,
Librato

40. Configuration and Secrets Management
Make our stateless apps as "12 factor" as possible - environment variables
vs. config files
Some shared configuration is inevitable as we migrate services over
Hashicorp Vault?
Chamber + AWS Parameter Store?

42. Wrap-up

43. Where Contentful is today
Our main API is 100% deployed on Kubernetes (few dependencies)
Work is ongoing to adapt more of our components to Kubernetes
We plan to complete the migration by Q1 2018

44. AWS + Kubernetes = ❤
For small organisations we can recommend this:
● Use Kops (Terraform optional but recommended)
● Use kubenet networking in the same VPC
● Use kube2iam to leverage IAM profiles, if your apps are using AWS
services
● (Parameter Store!)

45. Thank you!
Anthony Stanton
anthony@contentful.com
@elblivion