1. The Data Driven Network
Kapil Surlaker
Director of Engineering
Powering the Data Driven Network
Kapil Surlaker and Shirshanka Das
Hadoop Summit 2015

2. 2

5. How does PYMK work?
5

7. Houston
we have a problem

8. Step 1
Central transport pipeline

9. Still have
a problem

10. Hadoop Ingest Pipeline
Complexity

11. Step 2
Central
Ingestion
Framework
11

12. Requirements
Source
Diversity
Batch
and
Streaming
Data
Quality

13. Gobblin Architecture

14. 14
Source
Work
Unit
Work
Unit
Work
Unit
Extract
Extract
Extract
Convert
Convert
Convert
Quality
Quality
Quality
Write
Write
Write
Data
Publish
Task
Task
Task

15. Taming Source Diversity
REST
SFTP
JDBC
Protocol
Config
Source Extractor
checkpoint

16. Solving for real-time
Inefficiencies in batch
YARN based
Apache Helix
Continuous
Auto-scaling
YARN
Helix
Executor 1
Executor 2
Executor 3
HDFS
Stream Source

17. Data Quality
Per record, per task, or per
job
Composable quality checkers
Schema compatibility
Audit check
Sensitive fields
Unique key
Policy driven
Record
WriterJob
Task
Quality
Checker
FailQuarantine
Policy
Checker

18. Current Activity
Open source @ github.com/linkedin/gobblin
In production @ LinkedIn
Tens of TB per day
Hundreds of datasets
~20 different sources
Gobblin on YARN

20. Transformation: No one size fits all

21. Cubert: Converting hours to minutes
http://github.com/linkedin/cubert
Physical language
Block organization
Specialized operators

23. Got Diversity?

24. Where is the billings data?
How did it get here?
What data is used to create inferred
skills data?
Who owns that flow?
When will the latest profile data
show up? 24

25. 25

26. Where is my data?
How did it get here?
….
26

27. Where is my data?
How did it get here?
….
WhereHows
26

28. WhereHows architecture

29. 28

30. 29

32. 31

34. Lineage

35. WhereHows: Roadmap
Streaming ecosystem integration
Kafka, Samza
Recommendations for Datasets, Metrics
Exploring Open Source

37. Real-time. Interactive.

38. Slice and Dice metrics

39. Precompute!
Device Geo View
Android US 1
Android IN 1
iOS US 1
Dimension View
Android 2
iOS 1
US 2
IN 1
Android,US 1
iOS,US 1
Android,IN 1

40. More dimensions!
Device Geo Carrier View
Android US ATT 1
Android IN Reliance 1
iOS US Verizon 1
Dimension View
Android 2
iOS 1
US 2
IN 1
ATT 1
Reliance 1
Verizon 1
Android,US 1
... ...

41. Challenges
Horizontally scalable
Low latency
Data freshness
Fault tolerance
OLAP features

42. Introducing Pinot

43. Key features
SQL-like
interface
Columnar
storage and
indexing
Real-time
data load

44. (S)QL: Filters and Aggs
SELECT count(*)
FROM companyFollowHistoricalEvents
WHERE entityId = 121011 AND
'day' >= 15949 AND 'day' <= 15963 AND
paid = 'y’ AND
action = 'stop'

45. (S)QL: Group By
SELECT count(*)
FROM companyFollowHistoricalEvents
WHERE entityId = 121011 AND
'day' >= 15949 AND 'day' <= 15963 AND
paid = 'y’
GROUP BY action

46. (S)QL: ORDER BY and LIMIT
SELECT *
FROM companyFollowHistoricalEvents
WHERE entityId = 121011 AND
entityId = 1000 AND
action = 'start'
ORDER BY creationTime DESC LIMIT 1

47. Columnar Storage

48. Forward Index

49. Broker Helix
Real
time Historical
Kafka Hadoop
Pinot
Architecture
Queries
Raw
Data Samza

50. Fast but needs a ton of RAM

51. To pre-compute or not?

52. Data aware
pre-computation

53. Pinot@LinkedIn
Site-­‐facing
Apps Reporting
dashboards Monitoring

54. Breaking the cycle

55. Breaking the cycle

56. Breaking the cycle
Form hypothesis
Query
Repeat

57. Breaking the cycle
Form hypothesis
Query
Repeat
OR …

58. Hmm... whats up with portugese and
spanish speaking countries?

59. Brazil?

60. 56

61. 57
Holidays in Brazil 2015

63. Pinot Roadmap
Pinot is
Open Source !!!
github.com/linkedin/pinot
59

64. Kapil Surlaker
@kapilsurlaker
github.com/linkedin/
60
gobblin
cubert
pinot
Shirshanka Das
@shirshanka
Thanks!