TensorFlow Dev Summit 2018 Extended: TensorFlow Eager Execution

1. Extended

2. Eager execution
+Alexandre Passos Taegyun Jeon+

3. 오늘 에 대해 전해드릴 내용
● 왜 가 좋은가
●
● 으로 디버깅
●
● 과 같이 사용하기
● 사용자 가이드 예제

4. TensorFlow: A graph
execution engine for ML

5. TensorFlow: A graph
execution engine for ML?

6. Why graphs?
●
●
●
●
●

7. Why eager execution?
●
●
●
●

8. Eager in practice

9. Enabling eager execution
import tensorflow as tf
tf.enable_eager_execution()

10. import tensorflow as tf
tf.enable_eager_execution()
a = tf.constant([[2.0, 3.0], [4.0, 5.0]])
print(tf.matmul(a, a))
# => tf.Tensor(
# [[ 16. 21.]
# [ 28. 37.]], shape=(2, 2), dtype=float32)
Enabling eager execution

11. x = tf.constant(3.0)
with tf.GradientTape() as g:
g.watch(x)
y = x * x
grad = g.gradient(y, [x])[0]
# y = x^2
# dy_dx = 2x
# The gradient at x = 3.0
# Will compute to 6.0
GradientTape Record operations for
automatic differentiation

12. def line_search_step(f, init_x, rate=1.0):
with tf.GradientTape() as t:
tape.watch(init_x)
value = f(init_x)
grad, = t.gradient(value, init_x)
grad_norm = tf.reduce_sum(grad * grad)
init_value = value
while value > init_value - rate * grad_norm:
x = init_x - rate * grad
value = f(x)
rate /= 2.0
return x, value
Dynamic control flow

13. import tensorflow as tf
import tensorflow.contrib.eager as tfe
tf.enable_eager_execution()
Enabling eager execution
Useful symbols which work
with eager and graph

14. w = tfe.Variable([[1.0]])
loss = w * w # hopefully yours is smarter
dw, = tf.gradients(loss, [w])
Gradients

15. Gradients with eager
w = tfe.Variable([[1.0]])
with tf.GradientTape() as tape:
loss = w * w # hopefully yours is smarter
dw, = tape.gradient(loss, [w])
print(dw) # tf.Tensor([[ 2.]], shape=(1, 1), dtype=float32)
Record operations for
automatic differentiation

16. dataset = tf.data.Dataset.from_tensor_slices((data.train.images,
data.train.labels))
for images, labels in dataset.make_one_shot_iterator():
with tf.GradientTape() as tape:
prediction = model(images, training=True)
loss_value = loss(prediction, labels)
tf.contrib.summary.scalar('loss', loss_value)
grads = tape.gradient(loss_value, model.variables)
optimizer.apply_gradients(zip(grads, model.variables))
Training loop

19. New APIs

20. @tf.custom_gradient
def clip_gradient_by_norm(x, norm):
y = tf.identity(x)
def grad_fn(dresult):
return [tf.clip_by_norm(dresult, norm), None]
return y, grad_fn
Customizing Gradients

21. with tf.device("gpu:0"):
v = tfe.Variable(tf.random_normal([1000, 1000]))
v = None # Now v no longer takes up GPU memory
# Note: keras layers already treat variables as objects
# Note: use tfe.Variable when building graphs too
Variables are objects
Create, Modify,
and Remove

22. class Model(tf.keras.Model):
def __init__(self):
self.l0 = tf.layers.Dense(10)
self.l1 = tf.layers.Dense(10)
def call(self, x):
x = self.l0(x)
x = self.l1(x)
x = self.l0(x) # reuses the variables of layer 0
return x
Variable sharing
Easy sharing
and reuse

23. m = tfe.metrics.Mean("my_metric")
m([2])
m([8])
m.result() # 5.0
Object-oriented metrics
1) Update
2) Return

24. model = MyModel()
optimizer = tf.AdamOptimizer(0.001)
checkpoint_directory = ‘/home/apassos/my_model’
checkpoint_prefix = os.path.join(checkpoint_directory, "ckpt")
root = tfe.Checkpoint(
optimizer=optimizer, model=model,
optimizer_step=tf.train.get_or_create_global_step())
root.restore(tf.train.latest_checkpoint(checkpoint_directory))
# or
root.save(file_prefix=checkpoint_prefix)
Object-based saving

25. But… is it fast?

26. Benchmarks: Single GPU
●
●
○
○
●
●
●

27. Benchmarks: Overheads
●
●
○
●
●

28. Interoperating with graphs

29. def apply_resnet(x):
return resnet50.Resnet50(x)
t = tfe.make_template("f", apply_resnet, create_graph_function_=True)
print(t(image))
Force graph: tfe.make_template
Runs with graphs!
tf.keras.applications.resnet50

30. def my_py_func(x):
x = tf.matmul(x, x) # You can use any tf op, and GPUs
print(x) # but it's eager!
return x
with tf.Session() as sess:
x = tf.placeholder(dtype=tf.float32)
# Call eager function in graph!
pf = tfe.py_func(my_py_func, [x])
sess.run(pf, feed_dict={x: [[2.0]]}) # [[4.0]]
Force eager: tfe.py_func

31. ● Most finished models work in both eager and graph
● So write, iterate, debug in eager...
● ...then import on graph for deployment
● (checkpoints are compatible)
● Browse the example models in tensorflow/contrib/eager/python/examples
Compatible code

32. Practical advice

33. Write eager-compatible code
●
●
●
●
●
●

34. Why you should enable_eager_execution
●
●
●
●
●

35. Also, it’s fun.

36. Next steps
●
●
● 한국어 가이드 https://github.com/tgjeon/TF-Eager-Execution-Guide-KR
●
●

37. Thank you!
+Alexandre Passos
+Taegyun Jeon
Extended