AWS Re:Invent 2012 presentation from Netflix which covers how to optimize cost and usage of your AWS resources. Areas of focus are Autoscaling EC2 instances, batch access of SQS, and improved S3 usage.

1. Optimizing Costs with AWS

2. Source: http://ir.netflix.com

3. • Rationale and High-level Methodology
• AWS resource-specific optimizations
• Performance Testing
• Results
• Q&A

5. Rationale
• Applications operate at massive scale
• Across three regions and multiple zones per region
• Service oriented architecture
• Many moving parts (teams)
• Unconstrained deployment capabilities
• “Freedom and Responsibility” culture

6. Rationale, cont.
• Improve availability
• Avoid saturation of key resources
• Dynamically adjust capacity to meet workload demands
• Plan for increasing workloads
• Less focus on reducing current demand
• Maximize efficiency
• Balance OLTP and batch demands
• “That which is measured improves”

7. • Asgard framework enables turnkey deployment (Netflix open-sourced)
• All engineers have full access
• Real-time reservation capacity
• Unconstrained ASG size limits

9. • Birds-eye view of usage
• Near real-time data
• Open sourcing plans for tool
• Decomposes by application

15. Unused Reservation Instance Hours *
2,000
1,500 Need to
use this
1,000 capacity
500
0
Mon Tue Wed Thu Fri Sat
* - fictitious volumes

19. Healthy
Thrashing
Double-Jump
Y-axis = number of instances in ASG

22. Adopted batch delete
Requests/day
Started batch send
adoption
Batch capabilities
Adoption complete
Time

28. Legend
Github / Techblog Priam Exhibitor
Servo and Autoscaling Scripts
Apache Cassandra as a Service Zookeeper as a Service
Contributions
Astyanax Curator Honu
Techblog Post Only
Cassandra client for Java Zookeeper Patterns Log4j streaming to Hadoop
Coming Soon
CassJMeter EVCache Circuit Breaker - Hystrix
Cassandra test suite Memcached as a Service Robust service pattern
Cassandra Multi-region EC2 Eureka / Discovery Asgard - AutoScaleGroup
datastore support Service Directory based AWS console
Aegisthus Archaius Chaos Monkey
Hadoop ETL for Cassandra Dynamics Properties Service Robustness verification
Edda
Explorers Latency Monkey
Queryable config history
Governator - Library lifecycle Server-side latency/error
Janitor Monkey
and dependency injection injection
Odin
REST Client + mid-tier LB Bakeries and AMI
Workflow orchestration
Blitz4j - Async logging Configuration REST endpoints Build dynaslaves

29. Netflix at 2012 re:Invent
Date/Time Presenter Topic
Wed 8:30-10:00 Reed Hastings Keynote with Andy Jassy
Wed 1:00-1:45 Coburn Watson Optimizing Costs with AWS
Wed 2:05-2:55 Kevin McEntee Netflix’s Transcoding Transformation
Wed 3:25-4:15 Neil Hunt / Yury I. Netflix: Embracing the Cloud
Wed 4:30-5:20 Adrian Cockcroft High Availability Architecture at Netflix
Thu 10:30-11:20 Jeremy Edberg Rainmakers – Operating Clouds
Thu 11:35-12:25 Kurt Brown Data Science with Elastic Map Reduce (EMR)
Thu 11:35-12:25 Jason Chan Security Panel: Learn from CISOs working with AWS
Thu 3:00-3:50 Adrian Cockcroft Compute & Networking Masters Customer Panel
Thu 3:00-3:50 Ruslan M./Gregg U. Optimizing Your Cassandra Database on AWS
Thu 4:05-4:55 Ariel Tseitlin Intro to Chaos Monkey and the Simian Army

30. We are sincerely eager to
hear your FEEDBACK on this
presentation and on re:Invent.
Please fill out an evaluation
form when you have a
chance.
Contact: cwatson@netflix.com