My presentation at iMLSE 2018

1. Guaranteeing Deep Neural Network
Outputs in a Feasible Region
Hiroshi Maruyama
PFN Fellow

2. 2
What is “invariance” in ML?

3. 3
Fundamental Limitation of Machine Learning :
It’s Statistics!
Original Distribution
i. i. d.
Training Data Set
Trained Model
Random
Sampling !!
No guarantee of “100% correctness”

4. 4
In deep learning, any point in the Rn is possible as output
Input
Output：A Point in Rn
For any point P in Rn, there is a combination of
the input, training data set, hyper-parameters,
and random-number seed that generates P
• Training Data Set
• Hyper parameters
• Random # seeds
• … and program itself

5. 5
Example: Controlling a Drone
DL ModuleSensor Input
Reference Point
How to Guarantee that the reference point is always in the region?

6. 6
Definition: Feasible Region and Non-Feasible Solutions
Feasible Region
Non-feasible
solutions
We assume the feasible region is convex

7. DNN
Policy
Filter
Simple Solution: Policy Filter
Remove (no output) Snap to the boarder

8. Proposed Solution: Transformation of Output Space
Rn → Rn Space
Transformation
Select an interior point (called pivot)

9. Step 1 (Bounding) : Transform Rn to n-dimensional
hypercube
9
Sigmoid Function
Apply Sigmoid on each dimension
Move the pivot to the origin of the hypercube

10. Step 2: Shrink / Extend every point towards the origin
10

11. 11
Step 3: Finally move the pivot to the original position
For any combination of the input, training data set, hyper parameters, and
random number seed, the output is guarantted to be feasible

12. Proposed transformation works for any “star-shaped” space
12
Make this x0 the pivot
Set S is Star-shaped iff there is x0 s.t. for any interior point x,
the line segment xx0 ∈ S

13. Teacher signals can be given in the transformed space
Original DNN
(parameters to be
trained)
Transformation to
Feasible Region
(fixed parameters)
Back propagation
Rn Space
Input
Feasible Region
Teacher
Signal
loss

14. 14
Thank You
Twitter: @maruyama