XebiCon'17 : Faites chauffer les neurones de votre Smartphone avec du Deep Learning on-device - Qian Jin, Yoann Benoit et Sylvain Lequeux

Android Wear 2.0 Smart Reply
4
Learned
Projection
Model
https://research.googleblog.com/2017/02/on-device-machine-intelligence.html

https://en.wikipedia.org/wiki/Moore%27s_law

Source: https://www.qualcomm.com/news/snapdragon/2017/01/09/tensorflow-machine-learning-now-optimized-snapdragon-835-and-hexagon-682

Credit:
7
Source: https://9to5google.com/2017/01/10/qualcomm-snapdragon-835-machine-learning-tensorflow/

On-device Intelligence enables edge
devices to provide reliable execution with
or without network connection.
8

9
Image credit: https://www.andertoons.com

#datamobile
Chat History of the Slack channel

Magritte
Ceci n’est pas une pomme.

Build TensorFlow
Android Example With Bazel

Android Developer
Deep Learning Noob

WE CAN RECOGNIZE ALL
THE THINGS!

I THOUGHT THERE WERE
MODELS FOR EVERYTHING...

Prediction on an image - Inference
23

24

25
Apple: 0.98
Banana: 0.02

Back Propagation
29
Apple: 0.34
Banana: 0.66

Apple: 0.34
Banana: 0.66
Back Propagation
30
Prediction
Error

Apple: 0.34
Banana: 0.66
Back Propagation
31
Prediction
Error

Apple: 0.34
Banana: 0.66
Back Propagation
32
Prediction
Error

Back Propagation
33
Apple: 0.87
Banana: 0.13

Back Propagation
34
Banana: 0.93
Apple: 0.07

Deep Convolutional Neural Network
& Inception Architecture
Credit: http://nicolovaligi.com/history-inception-deep-learning-architecture.html

Deep Convolutional Neural Network
36
Image Credit: https://github.com/tensorflow/models/tree/master/research/inception
Visualisation of Inception v3
Model Architecture
Edges Shapes
High Level
Features
Classifiers

Source: CS231n Convolutional Neural Networks for Visual Recognition http://cs231n.stanford.edu/

Source: https://code.facebook.com/posts/1687861518126048/facebook-to-open-source-ai-hardware-design/

Transfer Learning
• Use a pre-trained Deep Neural Network
• Keep all operations but the last one
• Re-train only the last operation to specialize your network to your classes
Keep all weights identical
except these ones
40

42
Retrain a Model
Source: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/
python -m tensorflow/examples/image_retraining/retrain.py
--bottleneck_dir=tf_files/bottlenecks
--how_many_training_steps=500
--model_dir=tf_files/models/
--summaries_dir=tf_files/training_summaries/
--output_graph=tf_files/retrained_graph.pb
--output_labels=tf_files/retrained_labels.txt
--image_dir=tf_files/fruit_photos

Obtain the Retrained Model
•2 outputs:
• Model as protobuf file: contains
a version of the selected network
with a final layer retrained on
your categories
• Labels as text file
43
model.pb label.txt

44
public class ClassifierActivity extends CameraActivity implements
OnImageAvailableListener {
private static final int INPUT_SIZE = 224;
private static final int IMAGE_MEAN = 117;
private static final float IMAGE_STD = 1;
private static final String INPUT_NAME = "input";
private static final String OUTPUT_NAME = "output";
private static final String MODEL_FILE = "file:///android_asset/
tensorflow_inception_graph.pb";
private static final String LABEL_FILE = "file:///android_asset/
imagenet_comp_graph_label_strings.txt";
}
Source: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/ClassifierActivity.java

java.lang.UnsupportedOperationException:
Op BatchNormWithGlobalNormalization is
not available in GraphDef version 21.
45

Unsupported Operations
• Only keep the operations dedicated to the inference step
• Remove decoding, training, loss and evaluation operations
46

47
Optimize for Inference
python -m tensorflow/python/tools/optimize_for_inference
--input=tf_files/retrained_graph.pb
--output=tf_files/optimized_graph.pb
--input_names="input"
--output_names="final_result"

Data Scientist
Android Development Noob

CLICK 7 TIMES ON BUILD NUMBER 49

Pre-Google I/O 2017
• Use nightly build
• Library .so
• Java API jar
android {
//…
sourceSets {
main {
jniLibs.srcDirs = ['libs']
}
}
}
51

Post-Google I/O 2017
Source: Android Meets TensorFlow: How to Accelerate Your App with AI (Google I/O '17) https://www.youtube.com/watch?v=25ISTLhz0ys
52
Currently: 1.4.0

Model Size
All weights are stored as they are (64-bit floats) => 80MB
57

~80MB -> ~20MB
58
Weights Quantization
6.372638493746383 => 6.4
Source: https://www.tensorflow.org/performance/quantization

59
Quantize Graph
python -m tensorflow/tools/quantization/quantize_graph.py
--input=tf_files/optimized_graph.pb
--output=tf_files/rounded_graph.pb
--output_node_names=final_result
--mode=weights_rounded

MobileNet
Mobile-first computer vision models for TensorFlow
60
Image credit : https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md

Inception V3 v.s. MobileNet
61
Inception V3
78%
Accuracy*
85MB
MobileNet
(Largest configuration)
70.5%
Accuracy*
19MB
*: accuracy on ImageNet images

~80Mb => ~20Mb => ~1-5Mb
Source: https://research.googleblog.com/2017/06/mobilenets-open-source-models-for.html

Android SDK (Java) Android NDK (C++)
Classifier
Implementation
TensorFlow
JNI wrapper
Image (Bitmap)
Trained Model
top_resultsClassifications + Confidence
input_tensor1 2
34
Camera
Preview
Ref: https://jalammar.github.io/Supercharging-android-apps-using-tensorflow/
Overlay
Display

Image Sampling
Get Image from
Camera Preview
Crop the center square Resize Sample Image
67

Converts YUV420 (NV21) to ARGB8888
68
public static native void
convertYUV420ToARGB8888(
byte[] y,
byte[] u,
byte[] v,
int[] output,
int width,
int height,
int yRowStride,
int uvRowStride,
int uvPixelStride,
boolean halfSize
);

69
/**
* Initializes a native TensorFlow session for classifying images.
*
* @param assetManager The asset manager to be used to load assets.
* @param modelFilename The filepath of the model GraphDef protocol buffer.
* @param labels The list of labels
* @param inputSize The input size. A square image of inputSize x inputSize
is assumed.
* @param imageMean The assumed mean of the image values.
* @param imageStd The assumed std of the image values.
* @param inputName The label of the image input node.
* @param outputName The label of the output node.
* @throws IOException
*/
public static Classifier create(
AssetManager assetManager,
String modelFilename,
List<String> labels,
int inputSize,
int imageMean,
float imageStd,
String inputName,
String outputName) {
}

70
@Override
public List<Recognition> recognizeImage(final Bitmap bitmap) {
// Preprocess bitmap
bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(),
bitmap.getHeight());
for (int i = 0; i < intValues.length; ++i) {
final int val = intValues[i];
floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd;
floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd;
}
// Copy the input data into TensorFlow.
inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3);
// Run the inference call.
inferenceInterface.run(outputNames, logStats);
// Copy the output Tensor back into the output array.
inferenceInterface.fetch(outputName, outputs);
(continue..)
Preprocess Bitmap / Create Tensor

71
@Override
}
(continue..)
Feed Input Data to TensorFlow

72
@Override
}
(continue..)
Run the Inference Call

73
@Override
}
(continue..)
Fetch the Output Tensor

74
(continue..)
// Find the best classifications.
PriorityQueue<Recognition> pq =
new PriorityQueue<>(
3,
(lhs, rhs) -> {
// Intentionally reversed to put high confidence at the head of the queue.
return Float.compare(rhs.getConfidence(), lhs.getConfidence());
});
for (int i = 0; i < outputs.length; ++i) {
if (outputs[i] > THRESHOLD) {
pq.add(
new Recognition(
"" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i], null));
}
}
//...
return recognitions;
}
Find the Best Classification

Adding a New Model
77
2 * 20 MB = 40 MB

Model Fusion
• Start from previous model to keep all specific operations in the graph
• Specify all operations to keep when optimizing for inference
78
graph_util.convert_variables_to_constants(sess, graph.as_graph_def(),
[“final_result_fruits”, “final_result_vegetables”]

Continuous Training Pipeline
81
Source: https://www.tensorflow.org/serving/

TensorFlow Serving
Hosts the model
provides remote access to it
82
Source: https://www.tensorflow.org/serving/

model.pb label.txt
Continuous Training Pipeline
83

model.pb label.txt
Dispensing Model
84

Currently with Project Magritte…
Training
• Model debug done by overheating a laptop
• Model built on personal GPU
• Files uploaded manually
Model dispensing
• API available
• Deployment on AWS, currently migrating on Google Cloud
85

Android
FilesDir model.pb
Labels
model.pb label.txt

public TensorFlowInferenceInterface(AssetManager assetManager, String model) {
prepareNativeRuntime();
this.modelName = model;
this.g = new Graph();
this.sess = new Session(g);
this.runner = sess.runner();
final boolean hasAssetPrefix = model.startsWith(ASSET_FILE_PREFIX);
InputStream is = null;
try {
String aname = hasAssetPrefix ? model.split(ASSET_FILE_PREFIX)[1] : model;
is = assetManager.open(aname);
} catch (IOException e) {
if (hasAssetPrefix) {
throw new RuntimeException("Failed to load model from '" + model + "'", e);
}
// Perhaps the model file is not an asset but is on disk.
try {
is = new FileInputStream(model);
} catch (IOException e2) {
throw new RuntimeException("Failed to load model from '" + model + "'", e);
}
}
}
Source: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/android/java/org/tensorflow/contrib/android/TensorFlowInferenceInterface.java

92
Optimize for Mobile
> IMAGE_SIZE=224
> ARCHITECTURE="mobilenet_0.50_${IMAGE_SIZE}"

93
Optimize for Mobile
python -m tensorflow/examples/image_retraining/retrain.py
--bottleneck_dir=tf_files/bottlenecks
--how_many_training_steps=500
--model_dir=tf_files/models/
--summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}"
--output_graph=tf_files/retrained_graph.pb
--output_labels=tf_files/retrained_labels.txt
--architecture="${ARCHITECTURE}"
—image_dir=tf_files/fruit_photos

Model Inception V3
Optimized & Quantized

Source: https://developer.android.com/ndk/guides/neuralnetworks/index.html
• Android C API (NDK)
• Functionalities for high-level
machine learning frameworks
(TensorFlow Lite, Caffe2, others)
• Available on Android 8.1 and plus
(API 27+)
Android Neural Networks API

On-device Inferencing
• Latency: You don’t need to send a request over a network connection and wait for a
response. This can be critical for video applications that process successive frames
coming from a camera.
• Availability: The application runs even when outside of network coverage.
• Speed: New hardware specific to neural networks processing provide significantly
faster computation than with general-use CPU alone.
• Privacy: The data does not leave the device.
• Cost: No server farm is needed when all the computations are performed on the
device.
103
Source: https://developer.android.com/ndk/guides/neuralnetworks/index.html

Source: https://www.tensorflow.org/mobile/tflite/
TensorFlow Lite
• New model file format: based on
FlatBuffers, no parsing/unpacking
step & much smaller footprint
• Mobile-optimized interpreter: uses a
static graph ordering and a custom
(less-dynamic) memory allocator
• Hardware acceleration

Federate Learning
Collaborative Machine Learning without Centralized Training Data
105
Source: https://research.googleblog.com/2017/04/federated-learning-collaborative.html

Resources
• Artificial neural network: https://en.wikipedia.org/wiki/Artificial_neural_network
• Deep Learning: https://en.wikipedia.org/wiki/Deep_learning
• Convolutional Neural Network: https://en.wikipedia.org/wiki/Convolutional_neural_network
• TensorFlow for Poets: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/
• TensorFlow for Poets 2: Optimize for Mobile: https://codelabs.developers.google.com/
codelabs/tensorflow-for-poets-2/
• TensorFlow Glossary: https://www.tensorflow.org/versions/r0.12/resources/glossary
• Magritte project blog: http://blog.xebia.fr/2017/07/24/on-device-intelligence-integrez-du-
deep-learning-sur-vos-smartphones/
107

XebiCon'17 : Faites chauffer les neurones de votre Smartphone avec du Deep Learning on-device - Qian Jin, Yoann Benoit et Sylvain Lequeux

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a XebiCon'17 : Faites chauffer les neurones de votre Smartphone avec du Deep Learning on-device - Qian Jin, Yoann Benoit et Sylvain Lequeux

Semelhante a XebiCon'17 : Faites chauffer les neurones de votre Smartphone avec du Deep Learning on-device - Qian Jin, Yoann Benoit et Sylvain Lequeux (20)

Mais de Publicis Sapient Engineering

Mais de Publicis Sapient Engineering (20)

Último

Último (20)

XebiCon'17 : Faites chauffer les neurones de votre Smartphone avec du Deep Learning on-device - Qian Jin, Yoann Benoit et Sylvain Lequeux