The architectural principles behind building systems that scale to vast amounts of data and operate on that data in realtime. Presented at POSSCON '11.

1. The Secrets of Building
Realtime Big Data
Systems
Nathan Marz
@nathanmarz

2. Who am I?

3. Who am I?

4. Who am I?

5. Who am I?
(Upcoming book)

6. BackType
• >30 TB of data
• Process 100M messages / day
• Serve 300 requests / sec
• 100 to 200 machine cluster
• 3 full-time employees, 2 interns

7. Built on open-source
Thrift
Cascading
Scribe
ZeroMQ
Zookeeper
Pallet

8. What is a data system?
View 1
Raw data
View 2
View 3

9. What is a data system?
# Tweets /
URL
Tweets
Influence
scores
Trending
topics

10. Everything else: schemas, databases,
indexing, etc are implementation

11. Essential properties of
a data system

12. 1. Robust

13. 1. Robust
to machine failure

14. 1. Robust
to machine failure
and human error

15. 2. Low latency reads
and updates

16. 3. Scalable

17. 4. General

18. 5. Extensible

19. 6. Allows ad-hoc
analysis

20. 7. Minimal maintenance

21. 8. Debuggable

22. Layered Architecture
Speed Layer
Batch Layer

23. Let’s pretend temporarily that
update latency doesn’t matter

24. Let’s pretend it’s OK for a view to
lag by a few hours

25. Batch layer
• Arbitrary computation
• Horizontally scalable
• High latency

26. Batch layer
Not the end-all-be-all of batch
computation, but the most general

27. Hadoop
Distributed Distributed
Filesystem Filesystem
Input files Output files
MapReduce
Input files Output files
Input files Output files

28. Hadoop
• Express your computation in terms of
MapReduce
• Get parallelism and scalability “for free”

29. Batch layer
• Store master copy of dataset
• Master dataset is append-only

30. Batch layer
view = fn(master dataset)

31. Batch layer
MapReduce Batch
Master dataset
View 1
MapReduce Batch
View 2
Batch
View 3
MapReduce

32. Batch layer
• In practice, too expensive to fully
recompute each view to get updates
• A production batch workflow adds
minimum amount of incrementalization
necessary for performance

33. Incremental batch layer
Batch
View 1
New data
Batch View Batch
View 2
maintenance
workflow
Query Append Batch
View 3
All data

34. Batch layer
Robust and fault-tolerant to both machine
and human error.
Low latency reads.
Low latency updates.
Scalable to increases in data or traffic.
Extensible to support new features or related
services.
Generalizes to diverse types of data and requests.
Allows ad hoc queries.
Minimal maintenance.
Debuggable: can trace how any value in the
system came to be.

35. Speed layer
Compensate for high latency
of updates to batch layer

36. Speed layer
Key point: Only needs to compensate for
data not yet absorbed in serving layer

37. Speed layer
Key point: Only needs to compensate for
data not yet absorbed in serving layer
Hours of data instead of years of data

38. Application-level Queries
Batch Layer Query
Merge
Speed Layer Query

39. Speed layer
Once data is absorbed into batch layer, can
discard speed layer results

40. Speed layer
• Message passing
• Incremental algorithms
• Read/Write databases
• Riak
• Cassandra
• HBase
• etc.

41. Speed layer
Significantly more complex
than the batch layer

42. Speed layer
But the batch layer eventually
overrides the speed layer

43. Speed layer
So that complexity is transient

44. Flexibility in layered
architecture
• Do slow and accurate algorithm in batch
layer
• Do fast but approximate algorithm in speed
layer
• “Eventual accuracy”

45. Data model
Every record is a single, discrete
fact at a moment in time

46. Data model
• Alice lives in San Francisco as of time 12345
• Bob and Gary are friends as of time 13723
• Alice lives in New York as of time 19827

47. Data model
• Remember: master dataset is append-only
• A person can have multiple location
records
• “Current location” is a view on this data:
pick location with most recent timestamp

48. Data model
• Extremely useful having the full history for
each entity
• Doing analytics
• Recovering from mistakes (like writing
bad data)

49. Data model
Reshare: true
Gender: female
Property
Tweet: 456
Property
Reaction
Reactor Reactor
Tweet: 123
Alice
Bob
Property
Property
Content: RT @bob Content: Data is fun!
Data is fun!

50. Questions?
Twitter: @nathanmarz
Email: nathan.marz@gmail.com
Web: http://nathanmarz.com