1. Continuous Integration and Deployment Best
Practices on AWS - ARC307
Leo Zhadanovsky, Senior Solutions Architect, AWS
@leozh
JP Schneider, DevOps / Internet Jedi, Mozilla Foundation
@jdotp
November 13th, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. Who Am I?
I work for AWS
I worked for the DNC 2009-2012
I was embedded in the
DevOps Team @ OFA
AWS does not endorse
political candidates

3. Who Am I?
JP, DevOps for Mozilla Foundation
Previous gig DevOps at OFA 2012
Before that, Ops at Threadless
@jdotp
Mozilla Foundation does
endorse animated cats

4. CONTINUOUS
INTEGRATION

5. What is Continuous Integration?
Changes to code automatically deployed to mainline branch
• After passing unit and mock tests
Makes changes to code and deployments iterative, not
monolithic
Bugs are detected quickly
Allows rapid development
Helps automate deployments

6. DEVELOPER

7. SOURCE CODE
REPOSITORY

8. SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER

9. SOURCE CODE
REPOSITORY
CONTINUOUS
INTEGRATION SERVER
PROJECT MANAGEMENT
SERVER

10. SOURCE CODE
REPOSITORY
CONTINUOUS
INTEGRATION SERVER
PROJECT MANAGEMENT
SERVER
PICK
TASKS

11. SOURCE CODE
REPOSITORY
SUBMIT
CODE
CONTINUOUS
INTEGRATION SERVER
PROJECT MANAGEMENT
SERVER

12. SOURCE CODE
REPOSITORY
CONTINUOUS
INTEGRATION SERVER
SCHEDULE
BUILD
PROJECT MANAGEMENT
SERVER

13. SOURCE CODE
REPOSITORY
CONTINUOUS
INTEGRATION SERVER
RECURRENT
BUILDS
PROJECT MANAGEMENT
SERVER

14. SOURCE CODE
REPOSITORY
CONTINUOUS
INTEGRATION SERVER
CODE
FETCH
PROJECT MANAGEMENT
SERVER

15. SOURCE CODE
REPOSITORY
CONTINUOUS
INTEGRATION SERVER
CODE QUALITY
TESTS
PROJECT MANAGEMENT
SERVER
TEST
RESULTS

16. SOURCE CODE
REPOSITORY
CONTINUOUS
INTEGRATION SERVER
BUILD OUTPUT
PROJECT MANAGEMENT
SERVER

17. SOURCE CODE
REPOSITORY
DOCS
CONTINUOUS
INTEGRATION SERVER
BINARIES
& PACKAGES
PROJECT MANAGEMENT
SERVER
DEV FACING
NOTIFICATIONS

19. SOURCE CODE
REPOSITORY
BUILDS
CONTINUOUS
INTEGRATION SERVER
DNS
PROJECT
MANAGEMENT SERVER

20. PAIN POINTS
•
•
•
•
•
UNIT TESTS INCOMPLETE
MOCK TESTS MAINTENANCE
EXPENSIVE TEST ENVIRONMENT
TEST ENVIRONMENT ≠ PRODUCTION
DEPLOYMENT CYCLES

21. ON-DEMAND
ELASTIC
PAY AS YOU GO

22. =
PROGRAMMABLE
PLATFORM

26. IF YOU CAN PROGRAM IT
YOU CAN AUTOMATE IT

27. A lot of options…
Configuration Management Systems
• Puppet
• Chef
• Saltstack
Deployment Frameworks
•
•
•
•
•
AWS Elastic Beanstalk
AWS OpsWorks
Ansible
Fabric
Capistrano
Infrastructure Management
• AWS CloudFormation

28. Bootstrapping
Bake an AMI
Time consuming
configuration (startup time)
Static configurations (less
change management)
Configure dynamically

29. Bootstrapping
Bake an AMI
Configure dynamically
Continuous deployment
(latest code)
Environment specific (devtest-prod)

30. Obama for America
awsofa.info

31. Web-Scale Applications

32. 500k+ IOPS DB Systems

33. Services API

35. Typical Charts

37. How?

38. Ingredients
Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic
EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3
ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire
nagios Paypal CentOS CloudSearch levelDB mongoDB python
securitygroups Usahidhi PostgresSQL Github apache bootstrap
SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS
Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch
ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS
Objective-C DynamoDB

39. Infrastructure, Configuration
Management & Monitoring
Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic
EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3
ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire
nagios Paypal CentOS CloudSearch levelDB mongoDB python
securitygroups Usahidhi PostgresSQL Github apache bootstrap
SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS
Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch
ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS
Objective-C DynamoDB

40. Configuration Management: Puppet
In mid-2011, we looked at options for configuration
management and chose Puppet
We needed to make it scale, and to get it to work with stateless, horizontally scalable infrastructure
How did we do this?

41. Bootstrapping Puppet with CloudInit
CloudInit is built
into Ubuntu and
Amazon Linux
• Allows you to
pass bootstrap
parameters in
Amazon EC2
user-data field, in
YAML format

42. Bootstrapping Puppet with CloudInit
Don’t store creds in puppet manifests, store them in private
Amazon S3 buckets
Either pass Amazon S3 creds through CloudInit:
Even better – avoid this by using AWS Identity and Access
Management (IAM) roles and AWS Unified CLI’s S3 client

43. Bootstrapping Puppet with CloudInit
Built-in puppet support
Use certname with %i for instance id to name the node
Puppetmaster must have auto sign turned on
• Use security groups and/or NACLs for network-level security
In nodes.pp, use regex to match node names

44. Puppet Tips
Use a base class to define your standard install

45. Puppet Tips
Use runstages
Don’t store credentials in puppet, store them in private Amazon S3
buckets
• Use AWS IAM to secure the credentials bucket/folders within that bucket

46. Puppet Tips
Use puppet only for configuration files and what makes your
apps unique
For undifferentiated parts of apps, use Amazon S3 backed
RPM/Debian repositories
• Can be either public or private repos, depending on your needs
• Amazon S3 Private RPM Repos: http://git.io/YAcsbg
• Amazon S3 Private Debian Repos: http://git.io/ecCjWQ

47. Puppet Tips
By using packages for applications deploys, you can set ensure
=> latest, and just bump the package in the repo to update
Log everything with rsyslog/graylog/loggly/NewRelic/splunk

48. Scaling the Puppet Masters
Use an Auto Scaling group for puppet masters
• Min size => 2, use multiple Availability Zones
Either have them build themselves off of existing puppet
masters in the group or off packages stored in Amazon S3 and
bootstrapped through user-data
Auto-sign must be on

49. One thing that is difficult to prepare for…

50. They had this built for the previous 3 months, a
on the East Coast.

51. They had this built for the previous 3 months, a
on the East Coast.
We built this part in
9 hours to be safe.
AWS +
Puppet +
Netflix Asgard +
WAN Optimization Software +
DevOps =
Cross-Continent FaultTolerance On-Demand

52. ARC205 Thursday @ 3:00PM Lando 4303
Deploying the ‘League of Legends’ Data
Pipeline with Chef

53. Mozilla Foundation

54. Webmaker.org circa 2012
• Included Apps, non-SOA: Thimble, Popcorn, Goggles
• ~20 pushes of new software in 2012
• Operations and Development interacted mostly through bugzilla
tickets for deploys
• Hosting in physical datacenter at Mozilla
Webmaker.org circa early 2013
• Deciding to go 12-factor, SOA in app layer
• Weekly pushes of Popcorn on train model
• Operations and Development interacted mostly through bugzilla
tickets for deploys

55. April 2013
Webmaker begins rebuilding entire platform
SOA, 12-factor in node.js exclusively
Moving apps into AWS and DevOps / CI

56. Since April 2013….
Openbadges, Webmaker combine for:
Pushes Per Day (Staging and Prod)
Pushes Per Day to Staging / Prod
1339 Pushes

57. Who?
•
•
•
•
•
~30 Paid Developers
Hundreds of Students
Thousands of Contributors
One DevOps / Internet Jedi
Multiple Teams
How?
• Puppet, Jenkins, Fabric
• Tight feedback loops:
Newrelic, Opsview
• Culture Shift
• Staging Envs
• Brave devs iterate,
keeping work in-context
• Visible Ops
• Cross-train developers in
operations

58. What changed?

59. 1) Know How You Were Doing Before
2) Know What Changed When
3) Know How You Are Now Doing
=
The confidence to try more things
and try them faster, with minimum
viable planning.

60. Deployment
Pipeline

61. ARC312 Friday @ 1:30PM Lando 4206
SmugMug’s Zero-Downtime Migration to
AWS

62. AWS
CLOUDFORMATION
STACK-BASED DEPLOYMENT
SERVICE

63. AWS CLOUDFORMATION
TEMPLATE

65. {
}
"Description" : "Create RDS with username and password",
"Resources" : {
}
"MyDB" : {
"Type" : "AWS::RDS::DBInstance",
"Properties" : {
"AllocatedStorage" : "500",
"DBInstanceClass" : "db.m1.small",
"Engine" : "MySQL",
"EngineVersion" : "5.5",
"MasterUsername" : "MyName",
"MasterUserPassword" : "MyPassword"
}
}

66. "AWS::CloudFormation::Init" : {
}
"config" : {
"packages" : {
"yum" : {
"mysql"
: [],
"mysql-server" : [],
"httpd"
: [],
"php"
: [],
"php-mysql"
: []
}
},
"sources" : {
"/var/www/html" :
"https://s3.amazonaws.com/my-builds/build-v4.zip"
}

67. {
}
"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2
KeyPair to enable SSH
access to the instance",
"Type" : "String"
}
},

68. PROCEDURAL
DEFINITION
KNOWN
CONFIGURATION
Create it programmatically
Store stack configuration in
source control
CLOUDFORMATION
TEMPLATE
PARAMETER
DRIVEN
Dynamic and user-driven
templates
COLLABORATION
Share templates with ease
as just files

70. APPLICATION
VERSIONS
+
INFRASTRUCTURE
VERSIONS

71. AWS
CLOUDFORMATION
TEMPLATE

72. ARC203 Wednesday @ 4:15 Lando 4303
How Adobe Deploys
Refreshing the Entire Stack Every Time

74. Mars Rover Landing by the #’s
NASA TV = HD stream, 1080p, ~1 mb/s per viewer
Expecting peak of ~1m viewers
All playback devices (iOS, Android, Flash, HTML5, blah)
Once in a lifetime moment in history (no crashy crashy)
FUN FUN FUN

75. NASA TV = HD stream, 1080p, ~1 mb/s per viewer
Expecting peak of ~1m viewers
All playback devices (iOS, Android, Flash, HTML5, blah)
Once in a lifetime moment in history (no crashy crashy)
NASA says we can’t use their live stream setup
It’s 6 days before the landing
It’s the same week as the Olympics
Available technical resources from JPL: Brett and Khawaja
NOT FUN NOT FUN NOT FUN
Mars Rover Landing by the #’s

76. The sticky wicket:
manifest.f4m
size = 4kb
New every 4 sec
caching difficult
HD Video stream
size = 4mb
never changes
easy to cache

77. The plan
Design a solution around our limits
•
•
•
•
Max connections to origin = 6
Max streams per cache node = 20
Local Latency = critical
US-WEST-1 capacity reserved for S3 static images
Test the snot out of it
Hang on!

79. ok, so one of them is a rocket scientist..

80. “The 42 pack”

81. LOAD TESTING

84. Benchmarking

85. US-East Cache Node Performance
25.3 Gbps

86. Impact on US-East FMS Origin Servers
Only ~42Mbps

88. ARC303 Friday @ 1:30PM Delfino 4003
Unmeltable Infrastructure at Scale:
Using Apache Kafka, Twitter Storm and
Elastic Search on AWS

89. CONTINUOUS
DEPLOYMENT
SMALL, FREQUENT CHANGES
CONSTANTLY INTEGRATING INTO
PRODUCTION

90. KEY = ITERATION

91. ITERATION
=
MODIFY THE SYSTEM TO BETTER
MEET THE EXPECTATIONS OF
YOUR USERS

92. DEPLOYMENTS AT
AMAZON.COM
11.6s
1,079
10,000
30,000
Mean time
between
deployments
(weekday)
Max number of
deployments in a
single hour
Mean number of
hosts
simultaneously
receiving a
deployment
Max number of
hosts
simultaneously
receiving a
deployment

93. SOFTWARE DEPLOY
≠
PRODUCT LAUNCH

95. 1.5 BILLION PAGE VIEWS
$83 MILLION IN TRANSACTIONS
4.2 MILLION ITEMS SOLD
OCTOBER 2012

96. 30 DEPLOYS PER DAY
1 DEPLOY EVERY 20 MINUTES

97. "Production is truly the only place you
can validate your code."

102. AWS OPSWORKS
INTEGRATED APPLICATION
MANAGEMENT
DMG 304 Thursday @ 3:00PM Murano 3206
AWS OPSWORKS UNDER THE HOOD

104. 14 BILLION REQUESTS/MONTH
50 000 DATABASE UPDATES / SEC
NO CACHE

105. DATA-DRIVEN
ARCHITECTURES

106. METRICS @ETSY

107. METRICS @OBAMA FOR AMERICA

108. COST-ORIENTED
ARCHITECTURES

109. PHP+APACHE+VARNISH
NGINX+NODEJS

111. CONTINUOUS
DEPLOYMENT
CONTINUOUS
INTEGRATION

112. CONTINUOUS
DEPLOYMENT
=
CONTINUOUS
EXPERIMENTATION

113. CONTINUOUS
DEPLOYMENT
=
CONTINUOUS
IMPROVEMENT

114. INNOVATE

115. « Want to increase innovation?
Lower the cost of failure »
Joi Ito

116. SPEED AND AGILITY
“ON-PREMISES”
Experiment
Infrequently
Experiment
Often
Failure is
expensive
Fail quickly at
a low cost
Less
Innovation
More
Innovation

117. AWS re:Invent Pub Crawl
Join the AWS Startup Team this evening at the AWS Pub Crawl
When: Wednesday November 13, 5:30pm - 7:30pm
Where: Canaletto at The Venetian, 2nd Floor
Who Will Be There: Startups, The AWS Startup Team,
Startup Launch Companies and
AWS re:Invent Hackathon winners

118. Startup Spotlight Sessions with Dr. Werner Vogels
Thurs. Nov 14, Marcello Room 4406
SPOT 203 - Fireside Chats – Startup Founders, 1:30-2:30pm
– Eliot Horowitz, CTO of MongoDB
– Jeff Lawson, CEO of Twilio
– Valentino Volonghi, Chief Architect of AdRoll
SPOT 204 - Fireside Chats – Startup Influencers, 3:00-4:00pm
– Albert Wegner, Managing Partner at Union Square Ventures
– David Cohen, Founder and CEO of TechStars
SPOT 101 - Startup Launches, 4:15-5:15pm
– 5 companies powered by AWS launching at AWS re:Invent 2013

119. Please give us your feedback on this
presentation
ARC 307
As a thank you, we will select prize
winners daily for completed surveys!