Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2wt5mgs. Tim Bozarth shares how Netflix is enabling engineers to go from "zero" to "production ready" in minutes, incorporating best-practices learned through years in the cloud. He shares the story of transitioning from their home-grown RPC machinery to open-source standards and how their new approach is improving team velocity across Netflix engineering. Filmed at qconnewyork.com. Tim Bozarth is Director of Engineering at Netflix.

1. From Zero to
Production-Ready
in Minutes
Tim Bozarth
@timbozarth

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-deployment-practices

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Dev Experience:
Level up your Eng
Effectiveness

5. Agenda
1. It was the best of times...
2. Best practices made easy
3. Goodbye hand-written clients
4. From NIH to OSS

6. It was the best of
times…
(ie: The story of the skeletons in our closet)
1

7. About Netflix..
100m+ members
1000+ developers
190+ countries
1/3 US download traffic
500+ microservices
Over 100,000 VMs

9. Runtime Platform
Enable developers to productively create and integrate
software in the Netflix ecosystem.

10. Major Investments in
Platform

11. High Availability = Winning
Moments of
Truth

12. High Availability =

15. Challenges:
Hard to take advantage of
evolving best practices
Owning client-side logic is complex
and stressful
Non-Java experience is hard

16. Challenges:

18. Productivity++
(availability is table stakes)

19. Complexity is the
mind killer.

21. Runtime Platform
Enable developers to productively create and integrate
software in the Netflix ecosystem.

22. Best-practices
made easy
(Better living through less complexity)
2

23. Generators

24. Generators
What:
Gives you a deployed app
on the “paved road” in
minutes.

25. Generators
Why:
To make it easy to adopt,
understand, and build
production-ready apps.

26. +
Best
Practices

27. Historically:
“Let’s go!”

31. With Generators:
“Let’s go!”

34. +
+
+
+
+
=

35. But wait! There’s
more!
(Consistency)

36. Components != PaaS

37. Goodbye
hand-written
client libraries
3

39. @Netflix
every service owner
is responsible for a
client

40. Clients defend
themselves from
failure
(and the foundation to much of Netflix’s micro-service success)

41. Your service
Their service
Your
Client

42. RPC Internals
Platform
Integration
Serialization &
Deserialization
Bespoke business logic
Your
Client
Your service
Includes integration with Metrics,
Caching, Discovery, Fallbacks, etc...

43. Your Service
RPC Internals
Dependencies
RPC Internals
Platform
Integration
Serialization &
Deserialization
Bespoke business logic
Server Logic
Platform Integration
Serialization &
Deserialization
Your
Client
Their
Server

44. Problems
Server-API changes are a nightmare
So much hand-written RPC-related
code
No cross-language client story

45. These are solvable
problems

46. +

49. 2 big wins:
Code Generation
New Abstraction Layer

50. PROTO

51. Your Service
gRPC generated interfaces
Depend
encies
gRPC Generated Client
Bespoke business logic (please no)
Server Logic
Your
Client
Their
Server
Service
Proto

52. aching, Circuit-breake
Fallbacks, Failure
Injection, Discovery,
equest-context-tracin
etrics, Retries, Hedge

53. Interceptors
encapsulate
common patterns
(outside the user’s typical concern domain)

54. Client Defense Examples:
• Fallbacks
• Advanced Caching
• Retries
• Failure Injection
• Hedged Requests
• Circuit Breakers (Hystrix)
• Common analytics & event-logs
• ... and much more

55. Complex,
multi-tier
caching
took a lot of
code.

56. (In proto)
(In client
config)

57. gRPC ❤ languages!

58. NIH → OSS
4

59. Value
Effort

60. Value
Effort

61. With every step
comes the decision
to take another.

62. Inertia is a powerful
force, and a terrible
strategy.

63. Favor commodity
when it’s not our
core competency
(oh right! AWS!)

64. Wrapping up…
Ω

65. Everything discussed is done
gRPC = 10%+ of Netflix RPC
800+ projects made with generators
100+ services currently deployed from
generators
This stuff = Default for 6-12 months

66. Code generation is the short &
long term solution
IDLs = micro-services’ best
friend
Don’t build stuff you don’t
need to

67. <Appendix>

69. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-deployment-practices