Machine Learning with Neural Networks, TensorFlow and Keras

1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Machine Learning …
… with Neural Networks, TensorFlow and Keras
Christoph Pletz

2. Agenda
Name Presentation2 9/26/2018
1. Machine Learning Basics
2. Neural Networks
3. Using TensorFlow and Keras

3. Machine Learning with Neural Networks, TensorFlow and Keras3 9/26/2018
Machine Learning Basics

4. Types of Machine Learning
Machine Learning with Neural Networks, TensorFlow and Keras4 9/26/2018
Regression
– How much? How many?
Classification
– Which class does this belong to?
Clustering
– Are there different groups? To which group does an item belong?
Anomaly Detection
– Is this weird?
Recommendation
– Which option to choose?
supervisedunsupervised

5. Simplest Example – housing prices
Machine Learning with Neural Networks, TensorFlow and Keras5 9/26/2018
Living Space
Price
Living
Space
Price
95 1000000
100 950000
120 1200000
… …
250 3300000
Given a house with a certain living space,
what is the expected price?

6. Terminology: Features and Labels
Machine Learning with Neural Networks, TensorFlow and Keras6 9/26/2018
Living
Space
Price
95 1000000
100 950000
120 1200000
… …
250 3300000
Feature(s)
x
Label(s)
y

7. Hypothesis Function / Model
Machine Learning with Neural Networks, TensorFlow and Keras7 9/26/2018
𝑦 = ℎ(𝑥)
hypothesis
model
𝑥 predict
𝑦
𝑦
𝑦 Actual label value
Predicted value
𝑥 𝑦 𝑦
95 1000000 998734
100 950000 1043576
120 1200000 1236556
… …

8. Second Simplest Example – Iris classification
Machine Learning with Neural Networks, TensorFlow and Keras8 9/26/2018
CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=170298
Iris setosa Iris versicolor Iris virginica
Data collected by Edgar Anderson and published
by Ronald Fisher in 1936 (150 samples)

9. Iris Data Set
Machine Learning with Neural Networks, TensorFlow and Keras9 9/26/2018
https://upload.wikimedia.org/wikipedia/commons/thumb/5/56/Iris_dataset_scatterplot.svg/749px-Iris_dataset_scatterplot.svg.png

10. Iris Data Set (classification)
Machine Learning with Neural Networks, TensorFlow and Keras10 9/26/2018
Sepal
length
Sepal
width
Petal
length
Petal
width
Species
5.9 3.0 4.2 1.5 versicolor
6.9 3.1 5.4 2.1 virginica
5.1 3.3 1.7 0.5 setosa
… … … … …
Species
code
1
2
0
…
Is
Setosa
Is
versicolor
Is
virginica
0 1 0
0 0 1
1 0 0
… … …
One hot encoding
Features
x
Labels
y

11. A bit more challenging: Image Recognition
Machine Learning with Neural Networks, TensorFlow and Keras11 9/26/2018
Lets assume we want to classify the iris species from images …
E.g. 512 * 512 pixels and 3 color channels  786432 features, still 3 classes
Imagenet (http://www.image-net.org/) provides pictures for 1000 classes
model
predict 𝑦

12. A Linear Model for Housing Prices
Machine Learning with Neural Networks, TensorFlow and Keras12 9/26/2018
x
y
Linear Model
Parameters
(b = 100000, w = 0.5)
𝑦 = 𝑤 ∗ 𝑥 + 𝑏

13. What is a Machine Learning Model?
Machine Learning with Neural Networks, TensorFlow and Keras13 9/26/2018
Machine Learning Model
predict
Architecture
(Linear, Polynomial, Decision Tree, SVM, NN, …)
Parameters Hyper Parameters
(model config, training)
Training
Training data:
features + labels

14. Cost Function
Machine Learning with Neural Networks, TensorFlow and Keras14 9/26/2018
How good is our model in predicting labels for a given data set?
Calculate the distance from the predicted to the actual labels for all values of x and
sum them up “somehow”
– For regression often the “mean squared error” (MSE) is used
Model
Parameters

15. Minimize Cost Function with “Gradient Decent”
Machine Learning with Neural Networks, TensorFlow and Keras15 9/26/2018
x
y
Constant Model
Parameters
(b = 2000000)
𝑦 = 𝑏
b
J(b)
𝑏 𝑛+1 = 𝑏 𝑛 − 𝛼
𝜕
𝜕𝑏 𝑛
𝐽(𝑏 𝑛)
Learning
Rate
𝑏

16. Gradient Decent & Friends with 2 Parameters
Machine Learning with Neural Networks, TensorFlow and Keras16 9/26/2018
http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html

17. Under- and Over-Fitting, Validation Set
Machine Learning with Neural Networks, TensorFlow and Keras17 9/26/2018
x
y
Linear
Constant
Super-duper
Put some data aside for validation
Training
Set (~80%)
Validation
Set (~20%)

18. Machine Learning with Neural Networks, TensorFlow and Keras18 9/26/2018
Neural Networks

19. Machine Learning with Neural Networks, TensorFlow and Keras19 9/26/2018
𝑥1
𝑥2
𝑥3
𝑥4
∗ 𝑤1
1
∗ 𝑤2
1
∗ 𝑤3
1
∗ 𝑤4
1
𝑏1
𝑧1
∗ 𝑤1
2
∗ 𝑤2
2
∗ 𝑤3
2
∗ 𝑤4
2
𝑏2
𝑧2
∗ 𝑤1
3
∗ 𝑤2
3
∗ 𝑤3
3
∗ 𝑤4
3
𝑏3
𝑧3
Softmax(ActivationFunction)
𝑦1
𝑦2
𝑦3
Neuron 1
Neuron 2
Neuron 3 𝑦𝑖 = 1
Use with
“categorical crossentropy”
cost function

20. Neural Network Layers
Machine Learning with Neural Networks, TensorFlow and Keras20 9/26/2018
input
layer
output
layer
Softmax
Simpler representation:

21. Hidden Layers
Machine Learning with Neural Networks, TensorFlow and Keras21 9/26/2018
SoftmaxReLU
ReLU

22. Activation Functions
Machine Learning with Neural Networks, TensorFlow and Keras22 9/26/2018
https://medium.com/@shrutijadon10104776/survey-on-activation-functions-for-deep-learning-9689331ba092

23. Training
Machine Learning with Neural Networks, TensorFlow and Keras23 9/26/2018
activation
activation
activation
activation
X YY
Cost func
Forward pass
Backward pass
Backward pass:
• calculate gradients for all parameters (weights, biases)
• Perform one step of gradient decent

24. Machine Learning with Neural Networks, TensorFlow and Keras24 9/26/2018
Using TensorFlow and Keras

25. TensorFlow / Keras
Machine Learning with Neural Networks, TensorFlow and Keras25 9/26/2018
What is a Tensor?
– Generalization of Scalars  Rank 0, Vectors  Rank 1, Matrixes  Rank 2, …
TensorFlow
– Framework to perform mathematical calculations with tensors
– You build graphs of expression and execute them in a TensorFlow session
– Written in C++, language bindings to Python and other languages
Keras
– High level API over TensorFlow, CNTK and Theano
– More declarative
– Written in Python

26. Building a neural network with Keras
Machine Learning with Neural Networks, TensorFlow and Keras26 9/26/2018
def build_model():
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.models import Model
input = Input(shape=(4,), name="input_layer")
x = Dense(10, activation='relu', name="hidden_layer_1")(input)
x = Dense(8, activation='relu', name="hidden_layer_2")(x)
x = Dense(6, activation='relu', name="hidden_layer_3")(x)
output = Dense(3, activation='softmax', name="output_layer")(x)
model = Model(inputs=input, outputs=output)
return model

27. Setting up Training
Machine Learning with Neural Networks, TensorFlow and Keras27 9/26/2018
def compile_model(model):
from tensorflow.keras.optimizers import Adam
optimizer = Adam(lr=0.04, decay=1e-7)
lossfunction = 'categorical_crossentropy'
model.compile(optimizer,
lossfunction,
metrics=['accuracy'])

28. Training the model
Machine Learning with Neural Networks, TensorFlow and Keras28 9/26/2018
model.fit(training_dataset,
steps_per_epoch=TRAINING_STEPS,
epochs=50)
model.save(".modeliris.h5")

29. Making Predictions
Machine Learning with Neural Networks, TensorFlow and Keras29 9/26/2018
import numpy as np
from tensorflow.keras.models import load_model
code_to_name = {0: "setosa", 1: "versicolor", 2: "virginica"}
model = load_model(".modeliris.h5")
def predict(features):
x = np.array([features])
y = model.predict([x])
print("Model output: {}".format(y))
code = np.argmax(y)
return code_to_name[code]
prediction = predict([5.9, 3, 4.2, 1.5])
print("Prediction: {}".format(prediction))

30. Useful Tools
Machine Learning with Neural Networks, TensorFlow and Keras30 9/26/2018
Lutz Roeder’s Netron
– Visualizes saved models
– https://github.com/lutzroeder/Netron
TensorBoard
– Visualizes output from the
training process
– Part of TensorFlow

31. Using the Model in a .NET Application
Machine Learning with Neural Networks, TensorFlow and Keras31 9/26/2018
Save the model as a TensorFlow checkpoint
Freeze the saved model, save it as a .pb file
– Using the script: tensorflowpythontoolsfreeze_graph.py
Use TensorFlowSharp to load the model and make predictions
– https://github.com/migueldeicaza/TensorFlowSharp
from tensorflow.keras import backend as K
saver = tf.train.Saver()
saver.save(K.get_session(), '.modelkeras_model.ckpt')

32. Using the Model in a .NET Application
Machine Learning with Neural Networks, TensorFlow and Keras32 9/26/2018
public static IrisPrediction Predict(IrisFeatures features)
{
var data = features.ToArray();
var tensor = TFTensor.FromBuffer(new TFShape(1, data.Length), data, 0, data.Length);
using (var graph = new TFGraph())
{
graph.Import(File.ReadAllBytes("keras_frozen.pb"));
var session = new TFSession(graph);
var runner = session.GetRunner();
runner.AddInput(graph["input_layer"][0], tensor);
runner.Fetch(graph["output_layer/Softmax"][0]);
var output = runner.Run();
TFTensor result = output[0];
float[] p = ((float[][])result.GetValue(true))[0];
return IrisPrediction.FromArray(p);
}
}

33. Last Words
Machine Learning with Neural Networks, TensorFlow and Keras33 9/26/2018
Know the ML terminology
There are many ways to do things
– Start with one
This was only an introduction
– There are much more possibilities
Reuse what has been done by others
– Network designs
– Transfer learning

34. Name Presentation34 9/26/2018
Sample Code: https://github.com/cpletz/TFKerasTEDemo

35. Session Feedback – now
TechEvent September 201835 14.09.2018
Please use the Trivadis Events mobile app to give feedback on each session
Use "My schedule" if you have registered for a session
Otherwise use "Agenda" and the search function
If the mobile app does not work (or if you have a Windows smartphone), use your
smartphone browser
– URL: http://trivadis.quickmobileplatform.eu/
– User name: <your_loginname> (such as "svv")
– Password: sent by e-mail...

36. Christoph Pletz
Principal Consultant
Tel. +41 79 457 97 31
christoph.pletz@trivadis.com
9/26/2018 Name Presentation36