Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/162zy81. Nathan Marz shares lessons learned building Storm, an open-source, distributed, real-time computation system. Filmed at qconnewyork.com. Nathan Marz is currently working on a new startup. He was the lead engineer at BackType before being acquired by Twitter in 2011. At Twitter, he started the streaming compute team which provides and develops shared infrastructure to support many critical real-time applications throughout the company. Nathan is the creator of many open source projects, including projects such as Cascalog and Storm.

1. Lessons learned building Storm
Nathan Marz
@nathanmarz !1

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/storm-lessons

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Software development in theory

5. Software development in practice

6. Storm
Widely used stream processing system

7. Storm
Fully distributed and scalable
Strong processing guarantees
High performance
Multi-tenant

8. Storm architecture

9. Storm architecture
Master node (similar to Hadoop JobTracker)

10. Storm architecture
Used for cluster coordination

11. Storm architecture
Run worker processes

12. Lesson #1:
No such thing as a long-lived process

14. If JobTracker dies, all jobs die

15. =

17. Your processes will crash

18. Your code is not correct

19. Your code is not correct

20. Solution:
Design system to be fault-tolerant
to process restart

21. Implications
Program can be kill -9’d at any point
All state must be external to process
State modification might be aborted at any point

22. Other benefits for Storm
Easy to reconfigure
Can upgrade Nimbus without touching apps (e.g. with bug fixes)

23. Lesson #2:
Use state machines to express
intricate behavior

24. Nimbus has intricate behavior

25. Killing a topology
1. Stop emitting new data into topology
2. Wait to let topology finish processing in-transit messages
3. Shutdown workers
4. Cleanup state

26. Killing a topology
Asynchronous
Must be process fault-tolerant
Don’t allow activate/deactivate/rebalance to a killed topology
Should be able to kill a killed topology with a smaller wait time

27. Originally lots of (buggy)
conditional logic

28. Rewrote Nimbus to be an asynchronous,
process fault-tolerant state machine

30. Example of general solution being easier
to understand than the specific solution

31. Lesson #3:
Every feature will be abused

32. Example: Logging

33. One rogue topology uses up all
disk space on the cluster

34. Solution:
Switch from log4j to logback so
that size of logs can be limited

35. Example:
Storm’s “reportError” method

36. Used to show errors in the Storm UI

37. Error info is stored in Zookeeper

38. What happens when a user deploys code like this?

39. Denial-of-service on Zookeeper
and cluster goes down

40. Solution:
Rate-limit how many errors/sec
can be written to Zookeeper

41. This is a general principle

42. Lesson #4:
Isolate to avoid cascading failure

43. Originally one giant shared Zookeeper
cluster for all services within Twitter

44. If one service abused ZK, that could
lead to cascading failure

45. Zookeeper is not a multi-tenant system

46. Solution:
Storm got a dedicated Zookeeper cluster

47. Lesson #5:
Minimize dependencies

48. Your code is not correct

49. Other people’s code is not correct

50. Fewer
dependencies
Less possibility
for failure=

51. Example:
Storm’s usage of Zookeeper

52. Worker locations stored in Zookeeper

53. All workers must know locations of
other workers to send messages

54. Two ways to get location updates

55. 1. Poll Zookeeper

56. 2. Use Zookeeper “watch”
feature to get push notifications

57. Method 2 is faster but
relies on another feature

58. Storm uses both methods

59. If watch feature fails, locations
still propagate via polling

60. Eliminating dependence justified
by small amount of code required

61. Lesson #6:
Monitor everything

63. Monitoring is a prerequisite to
operating robust software

64. Storm makes monitoring
topologies very easy

65. Lots of stats monitored automatically

66. Simple API to monitor custom stats

67. Automatically integrates with
visualization stack

68. Lesson #7:
How we solved the resource
management problem

69. Resource management
Multi-tenancy: topologies don’t affect each other
Capacity management: converting $$$ into topologies

70. Initially treated these as
separate problems

71. Multi-tenancy attempt #1
Resource isolation
using Mesos

72. Machine Machine Machine Machine
Mesos
Storm

73. Each machine runs workers
from many topologies

74. Resource isolation is an
extraordinary claim

75. Extraordinary claims require
extraordinary evidence!

76. Ran into massive variance problems

77. At-least resource model of Mesos made
capacity measurement impossible

78. Almost went down route of resource
isolation with resource capping

79. But what about hardware threading
and caching?

80. Conclusion:
Sharing a single machine for
independent applications is
fundamentally complex

81. Capacity management attempt #1
1. Provide shared Storm cluster
2. Measure capacity usage in aggregate
3. Always have some % of cluster free
4. Grow cluster as needed according to usage

83. People would deploy topologies with
more workers than slots on cluster

84. People only care about getting their
application working and will twist any
knob possible

85. Problems
Production topologies starved
Bloated resource usage because no incentive to optimize
No process for making $$$ decisions

86. Requirements
1. Production topologies get priority to resources
2. One topology cannot affect the performance of another topology
3. Incentives for people to optimize resource usage
4. Process for making $$$ decisions on machines
5. Ability to measure how much capacity a topology needs for 3 and 4

87. The more complex the problem,
the simpler the solution must be

88. Solution
Isolation scheduler

89. Isolation scheduler

90. Isolation scheduler
Nimbus configuration

91. Isolation scheduler
Configurable only by cluster administrator

92. Isolation scheduler
Map from topology name to # of machines

93. Isolation scheduler
Topologies listed are production topologies

94. Isolation scheduler
These topologies guaranteed dedicated
access to that # of machines

95. Isolation scheduler
Remaining machines used for failover and
for running development topologies

96. Benefits
Resource contention issue of Mesos completely avoided
Takes advantage of process fault-tolerance of Nimbus
Simple to use and understand
Easy to do capacity measurements
Distinguishes production from in-development

97. Topology productionization process
1. Test topology on cluster as a development topology
2. When ready, work with admins to do capacity measurement
3. Submit capacity proposal for approval by VP
4. Allocate machines immediately from failover machines
5. Backfill capacity when machines arrive 4-6 weeks later

98. Benefits
Incentives to optimize resource usage
Backfill allows immediate productionization
Human process integrated with technical solution

99. Questions?

100. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/storm-
lessons