Neil Hunt and Yury Izrailevsky talk 2012 AWS re:invent conference about embracing the cloud

1. Netflix: Embracing the Cloud
Neil Hunt, CPO / Yury Izrailevsky, VP Engineering

3. Netflix – Service Unavailable – Database Crashed
Rest assured that the right people
are losing sleep to fix this problem!
We expect to resume service in approximately 72h
12 Aug 2008 03:12am

5. Availability
4 x nines
Scale Performance
Unconstrained Unlimited
horizontal scaling compute

9. • Experimented with both
• Ended up with NoSQL for almost everything important

12. Transitional Infrastructure: “Roman Riding”

17. Phase Components Data & Prerequisites
Trial (2009) Streaming Player Content keys (RO)
Membership status (RO)
Development Member product Content catalog (RW)
(2010-11) pages and APIs Personalization data
(RW) & recs algorithms
AB Test data (RW)
Followthrough Account and Membership data (RW)
(2011-12) membership
Final (2013) Payments PCI and SOX data

20. Availability
4 x nines
Scale Performance
Unconstrained Unlimited
horizontal scaling compute

21. Scalability Performance Availability

22. Scalability Performance Availability

23. 1/4/2009
2/4/2009
3/4/2009
4/4/2009
5/4/2009
6/4/2009
7/4/2009
8/4/2009
9/4/2009
10/4/2009
11/4/2009
12/4/2009
1/4/2010
2/4/2010
3/4/2010
4/4/2010
5/4/2010
6/4/2010
7/4/2010
8/4/2010
9/4/2010
10/4/2010
11/4/2010
12/4/2010
1/4/2011
2/4/2011
3/4/2011
4/4/2011
5/4/2011
6/4/2011
7/4/2011
8/4/2011
9/4/2011
10/4/2011
11/4/2011
12/4/2011
1/4/2012
2/4/2012
3/4/2012
4/4/2012
5/4/2012
6/4/2012
7/4/2012
8/4/2012
Scaling Netflix Streaming Service: Weekly Streaming Starts
23

24. Netflix Cross-Regional Cloud Architecture

25. Goal: Regional Failover

26. Building Global Netflix Streaming Product

27. Scalability Performance Availability

28. Weekly Cloud Cost Per Streaming Start (last 12 months)
28

29. Simian Army: Cloud Efficiency Automation
 Janitor Monkey
 Regularly scrape unused capacity
 Clean up instances, ASGs, ELBs, SGs, etc.
 Efficiency Monkey
 AI-based resource under-usage detection
(CPU, memory, etc.)
 Automated Deletion of Old Data
 TTL for S3 (using ObjectExpiration)
29

30. Cyclical Streaming Usage Pattern
30

31. Load-Based Auto Scaling
50%+ Cost Saving
Scale up/down
by 70%+
Move to Load-Based Scaling
31
31

32. Scalability Performance Availability

33. A Truly Great Service… Has To Just Work!
Availability Goal: 99.99%
(30 secs/week at peak traffic)
33

34. 7/17/2011
7/24/2011
7/31/2011
8/7/2011
8/14/2011
8/21/2011
8/28/2011
9/4/2011
9/11/2011
9/18/2011
9/25/2011
10/2/2011
10/9/2011
10/16/2011
10/23/2011
10/30/2011
11/6/2011
11/13/2011
11/20/2011
11/27/2011
12/4/2011
12/11/2011
12/18/2011
12/25/2011
1/1/2012
1/8/2012
1/15/2012
1/22/2012
1/29/2012
2/5/2012
2/12/2012
2/19/2012
2/26/2012
3/4/2012
3/11/2012
3/18/2012
3/25/2012
4/1/2012
4/8/2012
4/15/2012
4/22/2012
Other AWS Outages
4/29/2012
5/6/2012
5/13/2012
5/20/2012
5/27/2012
6/3/2012
6/10/2012
6/17/2012
6/24/2012
7/1/2012
Historical Streaming Availability (13wkMA)
7/8/2012
Outage
7/15/2012
7/22/2012
7/29/2012
8/5/2012
8/12/2012
AWS / Netflix
8/19/2012
8/26/2012
June 29th, 2012
9/2/2012
9/9/2012
9/16/2012
9/23/2012
9/30/2012
10/7/2012
14-Oct
10/21/2012
10/28/2012
Using Redundancy in AWS Infrastructure to Survive Failures
11/4/2012
11/11/2012

35. Cascading Failures
API
Instant
Queue
SimpleDB
35

36. Netflix Cloud Architecture
36

37. Cascading Failures
X …
99% Availability 99% Availability 99% Availability
300
99% = 4.90% 37

38. Strategies to Improve Availability
Graceful
Degradation Redundancy
38

39. Graceful Degradation
39

40. Redundancy
A B C
Zone Zone Zone Cassandra
A B C
S3 Backup
Redundancy
Across Availability Secure Cloud
Zones Backup
Storage Redundancy
Across
40
Regions, Vendors

41. Testing Fault Tolerance: Simian Army
Chaos Monkey Latency Monkey Chaos Gorilla
4

42. Open Source Portal at http://netflix.github.com

43. Superstorm Sandy
AWS Infrastructure Held Up
>2x Netflix Streaming Usage
in East Coast Markets
 Boston
 New York
 Philadelphia
 Baltimore
 D.C.

44. Focus on Building a Great Streaming Product
44

45. Netflix at 2012 re:Invent
Date/Time Presenter Topic
Wed 8:30-10:00 Reed Hastings Keynote with Andy Jassy
Wed 1:00-1:45 Coburn Watson Optimizing Costs with AWS
Wed 2:05-2:55 Kevin McEntee Netflix’s Transcoding Transformation
Wed 3:25-4:15 Neil Hunt / Yury I. Netflix: Embracing the Cloud
Wed 4:30-5:20 Adrian Cockcroft High Availability Architecture at Netflix
Thu 10:30-11:20 Jeremy Edberg Rainmakers – Operating Clouds
Thu 11:35-12:25 Kurt Brown Data Science with Elastic Map Reduce (EMR)
Thu 11:35-12:25 Jason Chan Security Panel: Learn from CISOs working with AWS
Thu 3:00-3:50 Adrian Cockcroft Compute & Networking Masters Customer Panel
Thu 3:00-3:50 Ruslan M./Gregg U. Optimizing Your Cassandra Database on AWS
Thu 4:05-4:55 Ariel Tseitlin Intro to Chaos Monkey and the Simian Army

46. We are sincerely eager to
hear your feedback on this
presentation and on re:Invent.
Please fill out an evaluation
form when you have a
chance.

47. We are sincerely eager to
hear your feedback on this
presentation and on re:Invent.
Please fill out an evaluation
form when you have a
chance.