At Netflix we run hundreds of A/B tests every year. Maintaining multivariate experiences quickly adds strain to any UI engineering team. In this talk, Alex Liu and Micah Ransdell explore the patterns we’ve built in Node.js to tame this beast - ultimately enabling quick feature development and rapid test iteration on our service used by over 50 million people around the world.

1. Scaling A/B testing
on Netflix.com with

2. welcome
Alex Liu
@stinkydofu @mjr578
Micah Ransdell

3. a/b testing
data-driven
product development

9. Test 1
A
B

10. Test 1
A
B
C
D
E
F
G

11. Test 1
A
B
C
D
E
F
G
Test 2
A
B
C
D
E
F
G
Test 3
A
B
C
D
E
F
G
Test 4
A
B
C
D
E
F
G
Test 5
A
B
C
D
E
F
G
Test 6
A
B
C
D
E
F
G
Test 7
A
B
C
D
E
F
G

12. Test 1
A
B
C
D
E
F
G
Test 2
A
B
C
D
E
F
G
Test 3
A
B
C
D
E
F
G
Test 4
A
B
C
D
E
F
G
Test 5
A
B
C
D
E
F
G
Test 6
A
B
C
D
E
F
G
Test 7
A
B
C
D
E
F
G

13. 2,097,152
unique experiences across
seven tests

14. hundreds
of new A/B tests per year

15. 984545041164535
824477853104281
030346308437926
36618855...

16. 2,105 566 685
CSS JSTemplates

17. 2.5 million
unique css/js packages
per push cycle

18. <html/>
<link/>
<script/>

19. problem:
conditional dependencies

21. ➔ Iterate Quickly
➔ Complex Concepts, Minimal Code
➔ Give the Power to the People
Why Node?

22. ➔Templating
Packaging
Bonus Round

23. Templating

25. Payment
Test
Testing the addition of
Direct Debit

26. Payment
Test
Testing the addition of
Direct Debit

27. Payment
Test
Testing the addition of
Direct Debit

28. Payment
Test
Testing the addition of
Direct Debit

29. Payment
Test
Testing the addition of
Direct Debit

31. payment template
Control Cell 2 Cell 3 Cell 4 Cell 5
if if ifif if

32. payment templatepayment template
Control Cell 2 Cell 3 Cell 4 Cell 5
if if ifif if

34. payment template
Control Cell 2 Cell 3 Cell 4 Cell 5
if if ifif if
payment_method_cc payment_method_dd

35. payment templatepayment template
Control Cell 2 Cell 3 Cell 4 Cell 5
if if ifif if
payment_method_cc payment_method_dd

36. payment template
Control Cell 2 Cell 3 Cell 4 Cell 5
payment_method_cc payment_method_dd
if if if if if

37. payment template
Control Cell 2 Cell 3 Cell 4 Cell 5
payment_method_cc payment_method_dd
if if if if if

39. payment
template
Control Cell 2 Cell 3 Cell 4 Cell 5
payment_method_cc payment_method_dd
?

40. payment
template
Control Cell 2 Cell 3 Cell 4 Cell 5
payment_method_cc payment_method_dd
payment_method.json

41. payment
template
Control Cell 2 Cell 3 Cell 4 Cell 5
payment_method_cc payment_method_dd
payment_method.json

45. require(‘nf-template-resolver’)

46. Resolver Rules
Template Resolver
Mappings

47. require(‘nf-template-loader’)

48. Resolver
Template
Inclusion
Template Loader
Template
Cache

49. ➔ Developers can combine rules
➔ Improves template legibility
➔ Increases template reuse
Benefits

50. Templating
➔Packaging
Bonus Round

51. Packaging

52. everything
is a module

55. app.js
oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep

56. app.js
oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep

57. oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
app.js

58. app.js

59. oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
app.js

60. oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
app.js

61. 685
files...?

62. oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
app.js

63. oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
app.js

64. oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
app.js

65. oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
app.js

66. oldSearch newSearch
dep1 dep2 dep3 dep4 dep5 dep6
sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
app.js

67. problem:
conditional dependencies

69. RequestBuild

70. require(‘nf-include-when’)

71. oldSearch.js

72. newSearch.js

73. anatomy of a rule

74. require(‘nf-asset-registry’)

75. app.js

76. app.js
oldSearch.js
newSearch.js
jquery
registry

77. what’s in the
registry?

79. nf-include-when

80. require(‘nf-packager’)

82. Step 1: Get the full
dependency tree for the
requested package from the
registry.

83. Step 2: Run the rules.

84. Step 3: Filter out all deps
that resolved false.

85. Step 3: Filter out all deps
that resolved false.

86. Step 3: Filter out all deps
that resolved false.
✓

87. ✓
Step 4: Filter out all
extraneous sub-deps.

88. Step 5: Concatenate the
files.

89. registry
javascript / css
Build

90. Request registry
rulespackage

91. ➔ Leverage build time tools
➔ Leverage the server when you can
➔ Divide and conquer with modules
Take aways

92. Templating
Packaging
➔Bonus Round

93. Bonus Round

94. be creative
with the registry

95. dependency
counting
dependency
pruning
file
sizes

96. CSS modules
@import

97. CSS
analysis

98. and now, the
coolest part...

100. templates
mappings
css
javascript
templates
mappings
css
javascript

101. templates
mappings
css
javascript
UI Bundle

102. UI BundleUI Bundle
UI Bundle
UI Bundle
UI BundleUI Bundle
UI BundleUI Bundle UI Bundle

104. anytime
deploy UI bundles

105. we never touch
the file system

106. < 5ms
average response time

107. ➔ Static analysis FTW
➔ Independent UI deployments
➔ Requests never touch the file system
➔ Fast package response times
Wins

108. final thoughts

109. learn by doing

110. move faster
fail fast

111. I have not failed.
I’ve just found 10,000 ways
that won’t work.”
Thomas Edison
“

112. simplify

113. thank you

114. questions?
Alex Liu
@stinkydofu @mjr578
Micah Ransdell