1. The document discusses architecting data science platforms for a dating product using an event-driven architecture that stores all data as a stream of events. 2. Key aspects of the architecture include an event history repository that stores real-time event streams, a Solr search index for querying events, and using the event stream for both online and offline machine learning. 3. The architecture aims to enable fast experimentation cycles by using the same code and data for production, development, and training machine learning models.

1. Architecting for Data Science
johann@ifwe.co@jssmith github.com/ifwe
Johann Schleier-Smith
CTO, if(we)
O’Reilly Software Architecture Conference
Boston, March 19, 2015

2. Data Science

4. value from data

5. Alternative Definitions
extraction of knowledge from data
making discoveries in the world of big data
statistics + machine learning + scalable
computation + visualization + computer science +
business acumen + skilled communication

6. Related and Alternative Language
business intelligence
statistics
data mining
forecasting
business reporting
predictive modeling
analyticsknowledge extraction

7. value

8. Types of Value
understanding
revenue
product improvements
projections
new inspirations
predictions
customer satisfaction

9. Today’s Examples

10. • >10 million candidates to draw from
• >1000 updates/sec
• Must be responsive to current activity
• Users expect instant query results
Recommendation engine
for dating product

11. • Real-time is challenging
• Human behavior is
complicated, especially in
social context
• Previous interactions are
perhaps our best hope for
predicting future interactions

12. • Human connections
• User engagement ecosystem
• Subscription and other revenues
Value
♥

13. Kaggle competition
with Best Buy data
https://www.kaggle.com/c/acm-sf-chapter-hackathon-small

14. Kaggle competition
with Best Buy data

18. “outgoing and social (heavy messaging ---
especially distant recipients and opposite gender,
many outgoing comments, many friend requests to
distant people), doesn’t play Pets much”
“receives many messages, active user,
views many profiles, doesn't use meet me,
sends many messages to distant people”
Heavy user overall, (pets, meet me, messaging)!
“heavy user overall, (pets, meet me, messaging)”

19. value

20. data

21. • Vote history
• Social interaction history
• Profile information
Dating product data

22. {
“timestamp” : “2011-10-31 09:48:46”,
“query” : “Assassin’s Creed”,
“skuSelected” : “2670133”
}
product views

23. {
“sku” : “1032361”,
“regularPrice” : “19.99”,
“name” : “Need for Speed: Hot Pursuit”,
“description” : “Fasten your seatbelt and
get ready to drive like your life depends
on it...”
...
}
product updates

24. Formats for Data
log files
web services
relational databases
unstructured documents
spreadsheets
xml files

25. Types of Data
technical data
government data
usage records
sensor data
academic data
reference data
yet uncollected data

26. Vasant Dhar. 2013. Data science and prediction. Commun. ACM 56, 12 (December 2013), 64-73.
And International Telecommunication Union (ITU) and United Nations Population Division via www.internetlivestats.com/internet-users/

27. Trends
data quantity
machine learning maturing
data variety
data velocity

28. Machine Learning
classification
decision trees
supervised methods
unsupervised methodsclustering
what matters most is mapping of data

29. Machine Learning Techniques
• Classification - (Logistic Regression, Decision
Trees, Random Forests)
• Prediction - (Generalized Linear Models, Support
Vector Regression, …)
• Clustering - (K-Means, Hierarchical, Latent
Dirichlet Allocation, …)
• Collaborative filtering, …
Features often matter more than choice of algorithm

30. data

31. tools of
the
trade

32. • Created in 1993
• Implementation of S language but also
inherits from Scheme
• Object oriented code is possible but
not encouraged
• Vast high-quality package ecosystem
• Data is vectors and data frames

33. Demo

35. • Statistics
• Visualization
• Machine learning
• REPL, scripts, interactive IDE
• In-memory data sets

37. http://scikit-learn.org/stable/auto_examples/linear_model/plot_iris_logistic.html

38. • More of a general purpose language than R
• Arrays and matrices as basic data structures
• Supports data frames through Pandas
• Sophisticated machine learning libraries
• Generally limited to in-memory data sets

39. • Leverages commodity hardware to
store large data sets at low cost
• Vibrant and diverse ecosystem
• Popular but not always best solution
• Probably best viewed as marketing
terminology, as opposed to technology

40. https://hadoopecosystemtable.github.io/
Category
Number of
projects
Distributed Filesystem 7
Distributed Programming 18
NoSQL Database 4
Document Data Model 3
Stream Data Model 1
Key-Value Data Model 4
Graph Data Model 3
NewSQL 9
SQL-On-Hadoop 11
Data Ingestion 11
Service Programming 7
Scheduling 3
Machine Learning 6
Benchmarking 5
Security 3
System Deployment 12
Applications 5
Development Frameworks 2
Categorize Pending 16
130 freely licensed open source
projects listed in the Hadoop
Ecosystem Table

41. Hadoop for Data Scientists
• Pulling data from repository (SQL, Hive)
• MapReduce programming (Java, Scala,
Pig, Python)
• Spark in-memory framework is gaining
adoption rapidly

42. tools rarely used in data science

43. version control
automated testing
automated deployment
shared code
agile methodology
code review
software architecture

44. the cycle
of data science

45. Data ProcessingData Collection
Models, Algorithms
Data-driven Product
Features
Data Analysis &
Understanding
Reports &
Visualizations
Product Improvements

46. data science
at

47. • Profitable startup actively pursuing big
opportunities in social apps
• Millions of users on existing products
• Thousands of social contacts per second

48. what it should look like

49. 1. Gain understanding of the product usage
2. See opportunity to make the product better
3. Create training data
4. Train predictive models
5. Put models in production
6. See improvements

50. what it often looks like

51. 1. Gain understanding of the product usage
2. See opportunity to make the product better
3. Pull records from relational database to create
interesting features (usually aggregates)
4. Train predictive models
5. Go implement models for production
6. See improvements

52. 1. Gain understanding of the product usage
2. See opportunity to make the product better
3. Pull records from relational database to create
interesting features (usually aggregates)
4. Train predictive models
5. Go implement models for production
6. See improvements
3-6
months

53. 1. Gain understanding of the product usage
2. See opportunity to make the product better
3. Pull records from relational database to create
interesting features (usually aggregates)
4. Train predictive models
5. Go implement models for production
6. See improvements Cool!
Was it worth it?

54. implementation
pain points

55. • Data scientist hands model description to
software engineer
• May need to translate features from SQL to Java
• Aggregate features require batch processing
• May need to adjust features and model to achieve
real-time updates
• Fast scoring requires high-performance in-
memory data structures

56. new thinking

57. new architecture

58. one right way to data

59. event history
one right way to data

60. everything is an event

61. Bob registers
Alice registers
Alice updates profile
Bob opens app
Bob sees Alice in recommendations
Bob swipes yes on Alice
Alice receives push notification
Alice sees Bob swiped yes
Alice swipes yes
Alice sends message to Bob

62. architecture
comparison

63. Database'
Applica-on'
Web'API'
Ranking' Solr'Search'
Indexing'
Service'
Change'logs'
Occasional'
index'rebuilds'
Change'logs'
Produc'on)
Development)
Exploratory'
Analysis'
Training'&'
Backtes-ng'

64. Database'
Applica-on'
Web'API'
Ranking' Solr'Search'
Indexing'
Service'
Change'logs'
Occasional'
index'rebuilds'
Change'logs'
Produc'on)
Development)
Exploratory'
Analysis'
Training'&'
Backtes-ng'

65. Database'
Applica-on'
Web'API'
Ranking' Solr'Search'
Indexing'
Service'
Change'logs'
Occasional'
index'rebuilds'
Change'logs'
Produc'on)
Development)
Exploratory'
Analysis'
Training'&'
Backtes-ng'

66. Database'
Applica-on'
Web'API'
Ranking' Solr'Search'
Indexing'
Service'
Change'logs'
Occasional'
index'rebuilds'
Change'logs'
Produc'on)
Development)
Exploratory'
Analysis'
Training'&'
Backtes-ng'

67. Database'
Applica-on'
Web'API'
Ranking' Solr'Search'
Indexing'
Service'
Change'logs'
Occasional'
index'rebuilds'
Change'logs'
Produc'on)
Development)
Exploratory'
Analysis'
Training'&'
Backtes-ng'

68. Database'
Applica-on'
Web'API'
Ranking'
Event'History'Repository'
Solr'Search'
Real=-me'
State'Updates'
Change'logs'Real=-me'events'
Occasional'
index'rebuilds'
Exploratory'
Analysis'&'
Visualiza-on'
Produc'on)
Development)
Training'&'
Backtes-ng'
State'Updates'
Monitoring'

69. Database'
Applica-on'
Web'API'
Ranking'
Event'History'Repository'
Solr'Search'
Real=-me'
State'Updates'
Change'logs'Real=-me'events'
Occasional'
index'rebuilds'
Exploratory'
Analysis'&'
Visualiza-on'
Produc'on)
Development)
Training'&'
Backtes-ng'
State'Updates'
Monitoring'

70. Database'
Applica-on'
Web'API'
Ranking'
Event'History'Repository'
Solr'Search'
Real=-me'
State'Updates'
Change'logs'Real=-me'events'
Occasional'
index'rebuilds'
Exploratory'
Analysis'&'
Visualiza-on'
Produc'on)
Development)
Training'&'
Backtes-ng'
State'Updates'
Monitoring'

71. Database'
Applica-on'
Web'API'
Ranking'
Event'History'Repository'
Solr'Search'
Real=-me'
State'Updates'
Change'logs'Real=-me'events'
Occasional'
index'rebuilds'
Exploratory'
Analysis'&'
Visualiza-on'
Produc'on)
Development)
Training'&'
Backtes-ng'
State'Updates'
Monitoring'

72. Database'
Applica-on'
Web'API'
Ranking'
Event'History'Repository'
Solr'Search'
Real=-me'
State'Updates'
Change'logs'Real=-me'events'
Occasional'
index'rebuilds'
Exploratory'
Analysis'&'
Visualiza-on'
Produc'on)
Development)
Training'&'
Backtes-ng'
State'Updates'
Monitoring'

73. Event History API
trait EventHistory {
def publishEvent(e: Event)
"
def getEvents(
startTime: Date,
endTime: Date,
eventFilter: EventFilter,
eventHandler: EventHandler
)
}

74. Event History API
trait EventHistory {
def publishEvent(e: Event)
"
def getEvents(
startTime: Date,
endTime: Date,
eventFilter: EventFilter,
eventHandler: EventHandler
)
}

75. Event History API
trait EventHistory {
def publishEvent(e: Event)
"
def getEvents(
startTime: Date,
endTime: Date,
eventFilter: EventFilter,
eventHandler: EventHandler
)
}
+∞ for
real-time
streaming

76. training data
comparison

77. Database'
Applica-on'
Web'API'
Ranking' Solr'Search'
Indexing'
Service'
Change'logs'
Occasional'
index'rebuilds'
Change'logs'
Produc'on)
Development)
Exploratory'
Analysis'
Training'&'
Backtes-ng'

78. Events'State'Snapshots''
Training'
Features'
Training'
Outcomes'
Time'

79. Database'
Applica-on'
Web'API'
Ranking'
Event'History'Repository'
Solr'Search'
Real=-me'
State'Updates'
Change'logs'Real=-me'events'
Occasional'
index'rebuilds'
Exploratory'
Analysis'&'
Visualiza-on'
Produc'on)
Development)
Training'&'
Backtes-ng'
State'Updates'
Monitoring'

80. Updates(
Training(
features(
Training(
outcome(
Time(
Online(State(
Events(
Training(Data(

82. 1. Gain understanding of the product usage
2. See opportunity to make the product better
3. Create training data
4. Train predictive models
5. Put models in production
6. See improvements
Fast cycles!!

83. Live Demo
https://www.kaggle.com/c/acm-sf-chapter-hackathon-small
https://github.com/ifwe/antelope

84. • Open source implementation derived from if(we)’s
proprietary platform
• Not ready scale or production, but useful for
demonstration purposes
• Seeking collaborators

85. product update events
{
“timestamp” : “2012-05-03 6:43:15”,
“eventType” : “ProductUpdate”,
“eventProperties” : {
“sku” : “1032361”,
“regularPrice” : “19.99”,
“name” : “Need for Speed: Hot Pursuit”,
“description” : “Fasten your seatbelt and
get ready to drive like your life depends
on it...”
...
}
}

86. product view events
{
“timestamp” : “2011-10-31 09:48:46”,
“eventType” : “ProductView”,
“eventProperties” : {
“query” : “Modern warfare”,
“skuSelected” : “2670133”
}
}

87. demo
Try it yourself, code and instructions at:
https://github.com/ifweco/antelope/blob/master/doc/demo.md

91. new
tools :)

92. Architecture for
Data Science

93. data warehousing

94. Database'
Applica-on'
Web'API'
Ranking' Solr'Search'
Indexing'
Service'
Change'logs'
Occasional'
index'rebuilds'
Change'logs'
Produc'on)
Development)
Exploratory'
Analysis'
Training'&'
Backtes-ng'

95. Data$
Warehouse$
Extract$
Load$
Transform$
Opera7onal$
Data$Store$
Staging$

97. log transform use
extract transform load

98. Data Architecture
dimensional modeling
relational modeling

99. Warehouse Design
normalization
slowly changing
dimensions
denormalization
star schema

100. slowly changing dimensions

101. type 1: overwrite the old data
type 2: multiple rows with versioning
type 3: extra columns for older versions

102. slowly changing dimensions
event history

104. Database'
Applica-on'
Web'API'
Ranking'
Event'History'Repository'
Solr'Search'
Real=-me'
State'Updates'
Change'logs'Real=-me'events'
Occasional'
index'rebuilds'
Exploratory'
Analysis'&'
Visualiza-on'
Produc'on)
Development)
Training'&'
Backtes-ng'
State'Updates'
Monitoring'

105. Event&
Indexing&
Applica1on&
Web&API&
Ranking&
Event&History&Repository&
Solr&Search&
Real>1me&
State&Updates&
Change&logs&Real>1me&events&
Exploratory&
Analysis&&&
Visualiza1on&
Produc'on)
Development)
Training&&&
Backtes1ng&
State&Updates&
Monitoring&

106. trait EventHistory {
def publishEvent(e: Event)
"
def getEvents(
startTime: Date,
endTime: Date,
eventFilter: EventFilter,
eventHandler: EventHandler
)
}

107. event history design

108. Data Architecture
ok to denormalize
log a lot
think about the types

109. • Make sure that events are simple facts
• Files are ok for event history, don’t really need a database
• Use an object hierarchy to model events in code
• Use online features that are efficient to update incrementally
• Write efficient implementations before than scaling out
• Functional style makes it easier
• Encourage reactive processing

112. Data Quality
• Matters more than transformations, more than algorithms
• Data that doesn’t make sense often indicates an application bug
• Do assertions, e.g., make sure things aren’t happening out of order

114. • All data in form of events – no exceptions!
• Same feature code in production and development
• Emphasis on creative feature engineering
• Quick cycles between ideas and production
github.com/ifwe/antelope
@jssmith
Try the Antelope Demo:
https://github.com/ifwe/antelope/blob/master/doc/demo.md