This document discusses TensorFrames, which bridges Spark and TensorFlow to enable data-parallel model training. TensorFrames allows Spark datasets to be used as input to TensorFlow models, and distributes the model training across Spark workers. The workers train on partitioned data in parallel and periodically aggregate results. This combines the benefits of Spark's distributed processing with TensorFlow's capabilities for neural networks and other machine learning models. A demo is provided of using TensorFrames in Python and Scala to perform distributed deep learning on Spark clusters.
(Genuine) Escort Service Lucknow | Starting âš,5K To @25k with A/C đ§đ˝ââ¤ď¸âđ§đť 89...
Â
Tallinn Estonia Advanced Java Meetup Spark + TensorFlow = TensorFrames Oct 24, 2016
1. TENSORFLOW + SPARK DATAFRAMES
=
TENSORFRAMES
Tallinn Advanced Java Meetup
Oct 24, 2016
Chris Fregly
Research Scientist @ PipelineIO
Thank You for Hosting, Planet OS!!
2. WHO AM I
Chris Fregly
⢠Currently
Research Scientist @ PipelineIO (http://pipeline.io)
Contributor @ Apache Spark
Committer @ Netflix Open Source
Founder @ Advanced Spark andTensorFlow Meetup
Author @ Advanced Spark (http://advancedspark.com)
Creator @ PANCAKE STACK (http://pancake-stack.com)
⢠Previously
Streaming Data Engineer @ Netflix, Databricks,IBM Spark
3. ADVANCED SPARK AND TENSORFLOW
MEETUP
4,600 Members+
Top 4 Spark Meetup!
Github Repo Stars + Forks
DockerHub Repo Pulls
4. CURRENT PIPELINE.IO RESEARCH
⢠Model Deploying andTesting
⢠Model Scaling and Serving
⢠Online ModelTraining
⢠Dynamic Model Optimizing
8. WHAT ARE NEURAL NETWORKS?
⢠Like All Machine Learning, Goal is to Minimize Loss (Error)
⢠Mostly Supervised Learning Classification
⢠Many labeled training samples exist
⢠Training
⢠Step 1: Start with Random Guesses for Input Weights
⢠Step 2: Calculate ErrorAgainst Labeled Data
⢠Step 3: Determine Gradient Amount and Direction (+ or -)
⢠Step 4: Back-propagate Gradient to Update Each Input Weight
⢠Step 5: Repeat Step 1 until Convergence or Max Epochs Reached
10. CONVOLUTIONAL NEURAL NETWORKS
⢠Apply Many Layers (aka. Filters) to Input
⢠Each Layer/Filter Picks up on Features
⢠Features not necessarily human-grokkable
⢠Brute Force âTry Diff numLayers & layerSizes
⢠Filter Examples
⢠3 Color Filters: RGB
⢠Moving AVG for Time Series
11. MY FAVORITE USE CASE â STITCH FIX
StitchFix Strata Conf SF 2016:
Using Deep Learning to Create New Clothing Styles!
17. MINIMIZE DATA DEPENDENCIES
⢠More natural for structured,independent data
⢠Tasks perform identical instructions in parallel on same-structured data
⢠Reduce data dependencies as they limit parallelism
Previous Instruction Previous Loop Iteration
21. WHAT IS TENSORFLOW?
⢠Google Open Source General Purpose Numerical Computation Engine
⢠Happens to be Good for Neural Networks!
⢠Tooling
⢠Tensorboard (port 6006 == `goog` upside down!) Ă
⢠DAG-based like Spark!
⢠Computation graph is logical plan
⢠Stored in Protobufâs
⢠Tensorflow converts logical to physical plan
⢠Lots of Libraries
⢠TFLearn (Tensorflowâs Scikit-learn Impl)
⢠Tensorflow Serving (Prediction Layer) Ă
25. WHAT ARE TENSORFRAMES?
⢠Bridge between Spark (JVM) and Tensorflow (C++)
⢠Python and Scala Bindings for Application Code
⢠Uses JavaCPP for JNI-level Integration
⢠Must Install TensorFrames C++ Runtime Libs on All Spark
Workers
⢠Developed by Old Co-worker @ Databricks,Tim Hunter
⢠PhD inTensors â Heâs âMr..Tensorâ
26. WHY TENSORFRAMES?
⢠Why Not?!
⢠Best of BothWorlds: Legacy Spark Support +Tensorflow
⢠Mix and Match Spark ML + Tensorflow AI on Same Data
⢠Tensorflow is DAG-based Similar to Spark
⢠Enables Data-Parallel Model Training
27. DATA-PARALLEL MODEL TRAINING
⢠Large Dataset are Partitioned Across HDFS Cluster
⢠Computation Graph (Logical Plan) Passed to SparkWorkers
⢠Workers Train on Each Data Partition in Parallel
⢠Workers Periodically Aggregate (ie.AVG) Results
⢠Aggregations happen in âParameter Serverâ
⢠Spark Master/Driver is Parameter Server
28. TENSORFLOW + MULTIPLE HOSTS/GPUS
Multi-GPU,Data-ParallelTraining
Step 1: CPU transfers model replica and (initial) gradients to each GPU
Step 2: CPU synchronizes and waits for all GPUs to process batch
Step 3: CPU copies all training results (gradients) back from GPU
Step 4: CPU averages gradients from all GPUs
Step 5: Repeat Step 1 with (new) gradients
Code
https://github.com/tensorflow/tensorflow/blob/master/
tensorflow/models/image/cifar10/
cifar10_multi_gpu_train.py
29. TENSORFRAME PERFORMANCE
⢠Depends on Algorithm and Dataset, of course!
⢠TensorFrames Require Extra Serialization JVM <-> C++
⢠What about Python Serialization from Python Bindings?
⢠Should be minimal unless using Python UDFs
⢠PySpark keeps small logical plan in Python layer
⢠Physical operations happen in JVM (except Python UDFs!)