Refactoring Organizations - A Netflix Study (QCon NYC 2017)

1. Josh Evans – (Former Netflix) Engineering Leader at Large June 26, 2017 Refactoring Organizations A Netflix Study

2. 2009

3. Devices …

4. Queue Reader UX

5. DVD Foundation

6. Customer Device Netflix Data Center NCCP Electronic Delivery LoadBalancer Netflix App Security Activation Playback Platform (NRDP) UI XML/RPC Ticket-based security Custom responses DB HTTP/S DVD Legacy DVD Legacy

7. Netflix API

8. Let 1000 flowers bloom!

9. Netflix Data Center API Netflix API LoadBalancer REST API JSON schema HTTP response codes Oauth Content Metadata Application HTTP/S

10. Customer Device Netflix Data Center API Proposal LB Netflix App Security Activation Playback Platform (NRDP) UI Content Metadata NCCP ED LB

11. Pros Separation of concerns Bandwidth Increased innovation Cons API reliability Heterogeneous architecture Lack of domain knowledge Assessment

12. Neil Hunt, CPO

13. What was the reaction?

14. Concern Anger

15. Tribalism

16. Conway’s Law

17. If you have four teams working on a compiler you will end up with a four pass compiler

18. Today’s Premise Conway’s Law describes dysfunction We must embrace architecture before organization Technical analogs drive better organizational solutions

19. Selfless Leadership Company Team You In that order!

20. Introductions Framework Scaling Teams IQ v EQ Conway’s Revenge Today’s Program

22. 1999 – 2009 Ecommerce (DVD  Streaming) 2009 – 2013 Streaming Infrastructure 2013 - 2016 Operations Engineering 2017 Time off, exploring options Josh Evans – Engineering Leader at Large @ops_engineering @

23. Global leader in subscription internet TV Growing slate of original content 100 million members 190 countries, 10s of languages 1000s of device types Microservices on AWS Unique company culture

25. Why do we refactor?

26. Functionality Engineering velocity Functional and operational quality As we scale! We refactor to improve or sustain

27. The ability to enhance a system by adding new functionality at minimal effort Functional Scalability The ease with which a system or component can be modified, added, or removed, to accommodate changing load Load Scalability

28. The ability for an organization to easily add people and domain responsibilities in response to increased work and complexity The ease with which an organization or team can adapt to shifts in business strategy Organizational Scalability

29. Common tasks are difficult Strategic efforts are impractical or impossible When do we refactor?

30. How do we refactor?

31. Technical Patterns Object-oriented design Micro-service architecture Systems engineering

32. Example: Organizational Polymorphism

33. With the right people Instead of a culture process adherence We have a culture of creativity and self discipline, freedom and responsibility Netflix Culture

34. You build it You run it

36. 2009

37. Devices in Production …

38. Key Platforms in Progress

39. Anthony Park

40. Surprise! John Funge

41. Big Picture Device Ubiquity Product Innovation Cloud Migration Internationalization Service Reliability

42. How many engineers? 6

43. What would you do?

44. Prioritize & Queue … … Task Queue Completed Tasks Thread Pool … …

45. Prioritize Service availability Game consoles Downloadable apps CE expansion Mobile … Queue Audio & subtitles International support New codecs … Prioritize & Queue

46. Scale Up Work profile Roles & throughput Team structure

47. Manager Engineer Engineer Engineer Engineer Engineer Test/Ops Monolithic Team One leader Undifferentiated roles Ad hoc responsibilities

48. Monolithic Decomposition Distinct modules & services Workload partitioning Dependency awareness Loose coupling

49. Server NRDP Features Protocols Security Bootstrap Key Platforms Device integration Device launch Streaming Infrastructure Device & partner-oriented load balancing

50. On Call Overload

51. Vicious Cycle Philip was great at Development Test infrastructure Project management Troubleshooting Philip Fisher-Ogden

52. You build it You run it

53. Risks Burnout Slow progress on key initiatives Philip Fisher-Ogden

54. Thread Starvation … … Shared exclusive resource High priority/frequency Other - blocked Tasks

55. Context Switching Process 1 Process 2 OS Interrupt or system call Save state - pcb1 .. Get state – pcb2 Interrupt or system call Save state – pcb2 Get state – pcb1 .. Executing Executing Idle Executing Idle Idle

56. Thread Pool Isolation Partition pools & locks Distribute problematic workloads … … … … … … … …

57. Organizational Solution Deepen troubleshooting skills Distribute escalations Engineer operations

58. Key Platforms Device integration Device launch Server NRDP Protocols Security Bootstrap Insight/Tools Delivery Dashboards Performance Operational tools Consolidating Operations Engineering

59. Cloud Migration

60. Rapid iteration v. systematic, long-cycle execution Cloud v. Product

61. S S S S. . . DB DB DB DB. . . . . . . . . Member Traffic Batch Processes Heterogeneous Workloads

62. Batch S S S S. . . DB DB DB DB. . . . . . . . . Member Path Member Path Member Path Batch Batch Interference

65. . . . Batch S S S S. . . DB DB DB DB. . . . . . Member Path Member Path Member Path Batch Batch Interference X

66. Batch S S S S. . . DB DB DB DB. . . . . . Member Path Member Path Member Path Batch Batch Partitioning Online Offline . . .

67. Silverlight Migration

68. Partitioning & Domain Portability Streaming Infrastructure Platform Engineering Engineer as a Library Ranjit Ranjit

69. Partitioning & Domain Portability Platform Engineering Engineer as a Library Streaming Infrastructure

70. Systems Cloud migration Key Platforms Device integration Device launch Server NRDP Protocols Security Bootstrap Insight/Tools Delivery Dashboards Performance Ops tools Streaming Infrastructure

71. Staffing

72. 6  24

73. Bottleneck

74. Systems Cloud migration Viewing history Viewing sessions Key Platforms Device integration Device launch Server NRDP Protocols Security Bootstrap Insight/Tools Delivery Dashboards Performance Ops tools Cloning & Parallel Processing

75. By 2012 cloud migration Canada, Latin America, UK massive device expansion major product improvements Netflix CDN

77. IQ Task-oriented Logical Literal Detached Autocratic EQ Feeling-oriented Emotional Social Empathetic Democratic Bimodal Thinking

78. IQ Design Evaluation Implementation EQ Inception Socialization Overcoming Tribalism

79. Flawed Inception

81. 2012

82. Customer Device Netflix Data Center API This… LB Netflix App Security Activation Playback Platform (NRDP) UI Content Metadata NCCP ED LB

83. ELB NCCP API …has become this Zuul

84. ELB …and this

85. Growing complexity Duplication of effort Engineering tax

86. Raising the Stakes Playback start in 500ms More UI/Playback scenarios Faster rate of innovation Better service reliability

87. Common tasks are difficult Strategic efforts are impractical or impossible When do we refactor?

88. If you have four teams working on a compiler you will end up with a four pass compiler Conway’s Revenge! We had two teams and a two-service edge architecture

89. Mature API team Robust API platform Strong operational focus Trust & respect A Better Foundation Daniel Jacobson

90. Josh: what’s the right architectural solution? Peter: do you care about the organizational implications? Moment of Truth

91. Selfless Leadership

92. Josh: what’s the right architectural solution? Peter: do you care about the organizational implications? Moment of Truth Josh: no – we’ll figure that out later

93. ELB NCCP API Before Zuul

94. After Integrated architecture Distributed functionality Shared services Common practices

95. Edge Services Zuul API Server Playback Services Features Security Data Systems Platform Insight/Tools Edge Services Shared services Organized around microservices, functionality, shared services

96. Takeaways Put architecture first Leverage technical analogs Know when to use IQ v EQ Be selfless

97. www.linkedin.com/in/jevansnflx Where to find me

98. ? Refactoring Organizations

Refactoring Organizations - A Netflix Study (QCon NYC 2017)

Esté a ler uma amostra.

Confira estes a seguir

Refactoring Organizations - A Netflix Study (QCon NYC 2017)

Mais Conteúdo rRelacionado

Diapositivos para si

Semelhante a Refactoring Organizations - A Netflix Study (QCon NYC 2017)

Em destaque

Livros relacionados

Refactoring Organizations - A Netflix Study (QCon NYC 2017)

Notas do Editor