1.
Scaling and Unifying
Scikit Learn and Spark
Pipelines using Ray
Raghu Ganti
Principal Research Staff Member
IBM T J Watson Research Center
Team (IBM & Red Hat):
Michael Behrendt, Linsong Chu, Carlos
Costa, Erik Erlandson, Mudhakar Srivatsa

2.
So many pipelines…
And many more…

3.
Ray.IO
§ Can we do pipelines on
Ray?
§ Can we scale popular
AI/ML pipelines on Ray?
§ Can we unify scikit learn
and Spark pipelines?

4.
Current pipeline API
• Focus on scikit learn and Spark pipelines
• Scikit learn missing scaling; Spark focus on data parallel
scaling
Transform
Fit
X
X
y
X’
Fitted model

5.
Scaling Pipelines: I/O as List of Objects
Transform
Fit
[X1, X2, … XN]
[X1, X2, … XN]
[y1, y2, … yN]
[X1’, X2’, …, XN’]
[FM1, FM2, … FMN]

6.
Scaling Pipelines: AND/OR Graphs
And node
X1
X2
XN
X1’
X2’
XM’
Or node
X
Step1
Step2
StepN
X’
X’
X’

7.
Key Features
▪ Python function as
unit of compute
▪ Intuitive for data
scientist
▪ Follows transformer
APIs
▪ MPI-style scaling
▪ Object references
as I/O for unit of
compute
▪ Sharing of objects
using Plasma store
▪ Enables zero-copy
object sharing
• List of objects as I/O
• Function as unit of
compute
▪ Scikit learn typically
in Python
▪ Ray.IO with RayDP
enables efficient
data exchange
• Cross environment
▪ Enriched DAGs from
plain pipelines
▪ OR nodes for fan-
out expressions
▪ AND nodes for
arbitrary lambdas
• AND/OR Graphs

8.
Illustrative Example
8
Preprocess
Random
Forest
Gradient
Boost
Decision
Tree
Sample Pipeline
Scikit learn Pipeline
Our Pipeline

9.
Pipelines Galore…
Airflow Kubeflow Scikit learn
Spark
Pipeline
Our
pipeline
Task
parallelism
✓ ✓ ✗ ✓ ✓
Data
parallelism
✗ ✗ ✗ ✓ ✓
And/Or Graphs ✓ ✓ ✗ ✗ ✓
Computational
unit
Container Container
Python
function
Python/Java
function
Python/Java
function
Mutability of
DAG
✗ ✗ ✓ ✓ ✓

10.
What to expect?
• Execution strategies based on graph traversals
• Early stopping criteria
• Mutability of execution pipelines
• Current status: Proposal discussion with Ray and OSS
community

11.
Q&A
Contacts:
Raghu Ganti (rganti@us.ibm.com)
Michael Behrendt (michaelbehrendt@de.ibm.com)
Linsong Chu (lchu@us.ibm.com)
Carlos Costa (chcost@us.ibm.com)
Erik Erlandson (eerlands@redhat.com)
Mudhakar Srivatsa (msrivats@us.ibm.com)

12.
Feedback
Your feedback is important to us.
Don’t forget to rate and review the sessions.