Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2dxPLlz. Simon Chan shares the on-going challenges ML SaaS platforms face when developers try to build customized large-scale predictive applications on production environments. He talks about the ML platform product design dilemma, the difference between building A.I. in a lab and on production, and the steps to build an end-to-end Machine Learning solution, from data sourcing to live monitoring etc. Filmed at qconnewyork.com. Simon Chan is a serial entrepreneur and product innovator. Recently, he joined Salesforce as a Senior Director. He was the founding CEO of PredictionIO - an open source Machine Learning Server that empowers programmers and data engineers to build smart applications. PredictionIO has become the most popular Spark-based machine learning project on Github.

1. Building an A.I. Cloud
What We Learned from PredictionIO

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
machine-learning-saas

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Simon Chan
Sr. Director, Product Management, Salesforce
Co-founder, PredictionIO
PhD, University College London
simon@salesforce.com

5. The A.I. Developer Platform Dilemma
FlexibleSimple

6. “ Every prediction problem is unique.

7. 3 Approaches to Customize Prediction
GUIAutomation
Custom
Code & Script
FlexibleSimple

8. 10 KEY STEPS
to build-your-own A.I.
P.S. Choices = Complexity

9. One platform, build multiple apps. Here are 3 examples.
1. E-Commerce
Recommend
products
2. Subscription
Predict churn
3. Social Network
Detect spam

10. Let’s Go Beyond Textbook Tutorial
// Example from Spark ML website
import org.apache.spark.ml.classification.LogisticRegression
// Load training data
val training = sqlCtx.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")
val lr = new LogisticRegression().setMaxIter(10).setRegParam(0.3).setElasticNetParam(0.8)
// Fit the model
val lrModel = lr.fit(training)

11. Define the Prediction Problem
Be clear about the goal
1

12. Define the Prediction Problem
Basic Ideas:
◦ What is the Business Goal?
▫ Better user experience
▫ Maximize revenue
▫ Automate a manual task
▫ Forecast future events
◦ What’s the input query?
◦ What’s the prediction output?
◦ What is a good prediction?
◦ What is a bad prediction?

13. Decide on the Presentation
It’s (still) all about human perception
2

14. Decide on the Presentation
NOT SPAM SPAM
NOT SPAM True Negative False Negative
SPAM False Positive True Positive
Actual
Predicted
Mailing List and Social Network, for example, may tolerate false predictions differently

15. Decide on the Presentation
Some UX/UI Choices:
◦ Toleration to Bad Prediction?
◦ Suggestive or Decisive?
◦ Prediction Explainability?
◦ Intelligence Visualization?
◦ Human Interactive?
◦ Score; Ranking; Comparison; Charts; Groups
◦ Feedback Interface
▫ Explicit or Implicit

16. Import Free-form Data Source
Life is more complicated than MNIST and MovieLens datasets
3

17. Import Free-form Data Sources
Some Types of Data:
◦ User Attributes
◦ Item (product/content) Attributes
◦ Activities / Events
Estimate (guess) what you need.

18. Import Free-form Data Sources
Some Ways to Transfer Data:
◦ Transactional versus Batch
◦ Batch Frequency
◦ Full data or Changed Delta Only
Don’t forget continuous data sanity checking

19. Construct Features & Labels
from Data
Make it algorithm-friendly!
4

20. Construct Features & Labels from Data
Some Ways of Transformation:
◦ Qualitative to Numerical
◦ Normalize and Weight
◦ Aggregate - Sum, Average?
◦ Time Range
◦ Missing Records
Different algorithms may need different things
Label-specific:
◦ Delayed Feedback
◦ Implicit Assumptions
◦ Reliability of Explicit Opinion

21. Construct Features & Labels from Data
Qualitative to Numerical
// Example from Spark ML website - TF-IDF
import org.apache.spark.ml.feature.{HashingTF, IDF, Tokenizer}
val sentenceData = sqlContext.createDataFrame(Seq(
(0, "Hi I heard about Spark"),
(0, "I wish Java could use case classes"),
(1, "Logistic regression models are neat")
)).toDF("label", "sentence")
val tokenizer = new Tokenizer().setInputCol("sentence").setOutputCol("words")
val wordsData = tokenizer.transform(sentenceData)
val hashingTF = new HashingTF()
.setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(20)
val featurizedData = hashingTF.transform(wordsData)
val idf = new IDF().setInputCol("rawFeatures").setOutputCol("features")
val idfModel = idf.fit(featurizedData)
val rescaledData = idfModel.transform(featurizedData)

22. Set Evaluation Metrics
Measure things that matter
5

23. Set Evaluation Metrics
Some Challenges:
◦ How to Define an Offline Evaluation that
Reflects Real Business Goal?
◦ Delayed Feedback (again)
◦ How to Present The Results to Everyone?
◦ How to Do Live A/B Test?

24. Clarify “Real-time”
The same word can mean different things
6

25. Clarify “Real-time”
Different Needs:
◦ Batch Model Update, Batch Queries
◦ Batch Model Update, Real-time Queries
◦ Real-time Model Update, Real-time Queries
When to Train/Re-train for Batch?

26. Find the Right Model
The “cool” modeling part - algorithms and hyperparameters
7

27. Find the Right Model
Example of model hyperparameter selection
// Example from Spark ML website
// We use a ParamGridBuilder to construct a grid of parameters to search over.
val paramGrid = new ParamGridBuilder()
.addGrid(hashingTF.numFeatures, Array(10, 100, 1000))
.addGrid(lr.regParam, Array(0.1, 0.01)).build()
// Note that the evaluator here is a BinaryClassificationEvaluator and its default metric
// is areaUnderROC.
val cv = new CrossValidator()
.setEstimator(pipeline).setEvaluator(new BinaryClassificationEvaluator)
.setEstimatorParamMaps(paramGrid)
.setNumFolds(2) // Use 3+ in practice
// Run cross-validation, and choose the best set of parameters.
val cvModel = cv.fit(training)

28. Find the Right Model
Some Typical Challenges:
◦ Classification, Regression, Recommendation
or Something Else?
◦ Overfitting / Underfitting
◦ Cold-Start (New Users/Items)
◦ Data Size
◦ Noise

29. Serve Predictions
Time to Use the Result
8

30. Serve Predictions
Some Approaches:
◦ Real-time Scoring
◦ Batch Scoring
Real-time Business Logics/Filters is often added
on top.

31. Collect Feedback for Improvement
Machine Learning is all about “Learning”
9

32. Collect Feedback for Improvement
Some Mechanism:
◦ Explicit User Feedback
▫ Allow users to correct, or express opinion
on, prediction manually
◦ Implicit Feedback
▫ Learn from subsequence effects of the
previous predictions
▫ Compare predicted results with the
actual reality

33. Keep Monitoring & Be Crisis-Ready
Make sure things are still working
10

34. Keep Monitoring & Be Crisis-Ready
Some Ideas:
◦ Real-time Alert (email, slack, pagerduty)
◦ Daily Reports
◦ Possibly integrate with existing monitoring tools
◦ Ready for production rollback
◦ Version control
For both prediction accuracy and production
issues.

35. Summary: Processes We Need to Simplify
Define the Prediction Problem
Decide on the Presentation
Import Free-form Data Sources
Construct Features & Labels from Data
Set Evaluation Metrics
Clarify “Real-Time”
Find the Right Model
Serve Predictions
Collect Feedback for Improvement
Keep Monitoring & Be Crisis-Ready

36. The Future of A.I.
is the automation of A.I.

37. Thanks! Any Questions?
WE ARE HIRING.
simon@salesforce.com
@simonchannet

38. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
machine-learning-saas