Presentation slide for ICML 2018

1. Functional Gradient Boosting based on
Residual Network Perception
• 1 : 2 :
6 6 2 2 2B6 2 6 2 : 2 6 2 2B6
2 , . A &
6B) B
!( : 62 2BB: 6"( 62 D 6 6 2 :
#$ #$ #$
6B,6 ( 0 6 2 6 .D & 1
2 2 :B : D : :B 2 B 6 6
: 62 B6 2 2 6 D 6B) B
6B)

2. ( 1 1 1 1 0 ) + 7 21
ℎ" = $"(&) (" , ) ,, .

3. ( 1 1 1 1 0 ) + 7 21
ℎ" = $"(&) (" , ) ,, .
ℎ*+, = ℎ* + (*(ℎ*)
) ./ℎ*+,, 1 = ) ./(ℎ* + (* ℎ* ), 1

4. ( 1 1 1 1 0 ) + 7 21
ℎ" = $"(&) (" , ) ,, .
ℎ*+, = ℎ* + (*(ℎ*)
) ./ℎ*+,, 1 = ) ./(ℎ* + (* ℎ* ), 1
= ) ./ℎ*, 1 + (* ℎ*
/234
) ./ℎ*, 1 + 5 (* ℎ*

5. ( 1 1 1 1 0 ) + 7 21
ℎ" = $"(&) (" , ) ,, .
ℎ*+, = ℎ* + (*(ℎ*)
) ./ℎ*+,, 1 = ) ./(ℎ* + (* ℎ* ), 1
= ) ./ℎ*, 1 + (* ℎ*
/234
) ./ℎ*, 1 + 5 (* ℎ*
= ) ./ℎ6, 1 + 7
"86
*
(" ℎ"
/239
) ./ℎ", 1 + 5 (" ℎ" .

6. ( 1 1 1 1 0 ) + 7 21
ℎ" = $"(&) (" . ) . ,
ℎ*+, = ℎ* + (*(ℎ*)
) ./ℎ*+,, 1 = ) ./(ℎ* + (* ℎ* ), 1
= ) ./ℎ*, 1 + (* ℎ*
/234
) ./ℎ*, 1 + 5 (* ℎ*
= ) ./ℎ6, 1 + 7
"86
*
(" ℎ"
/239
) ./ℎ", 1 + 5 (" ℎ" .
(" ℎ" ∼ −239
)(./ℎ", 1) .. .
. ℎ" . −239
)(./ℎ", 1) , , , .

7. 0 + ) .(0 1
ℎ" = $"(&) (" . ) . ,
ℎ*+, = ℎ* + (*(ℎ*)
) ./ℎ*+,, 1 = ) ./(ℎ* + (* ℎ* ), 1
= ) ./ℎ*, 1 + (* ℎ*
/234
) ./ℎ*, 1 + 5 (* ℎ*
= ) ./ℎ6, 1 + 7
"86
*
(" ℎ"
/239
) ./ℎ", 1 + 5 (" ℎ" .
(" ℎ" ∼ −239
)(./ℎ", 1) .. .
. ℎ" . −239
)(./ℎ", 1) , , , .
2 0 0 2 + ) 7

8. • . , , , , , . , ,
• , , . ! . , "
• , , , , ,
• , , , ,.
., , , , , , , ,
• , . , . ,
• , , . . ,
• , , , ,

9. • Problem setting: Multiclass Classification
min
$,&
ℒ( ) = +,-
. )(0), 2 () = 34
5, 6(: 89:. <=>= <?@>. ).
Set ℛ( 5 = min
&
ℒ( 34
5 +
C
D
3 D
D
.

10. • Problem setting: Multiclass Classification
min
$,&
ℒ( ) = +,-
. )(0), 2 () = 34
5, 6(: 89:. <=>= <?@>. ).
Set ℛ( 5 = min
&
ℒ( 34
5 +
C
D
3 D
D
.
• Functional Gradient EFGH F in IJ KH,L
ℎN = 5N(O)

11. • Problem setting: Multiclass Classification
min
$,&
ℒ( ) = +,-
. )(0), 2 () = 34
5, 6(: 89:. <=>= <?@>. ).
Set ℛ( 5 = min
&
ℒ( 34
5 +
C
D
3 D
D
.
• Functional Gradient EFGH F in IJ KH,L
ℎN = 5N(O) .
∇$ℛ( 5N (O) = QRS
.(34
5N(O), T)

12. • Problem setting: Multiclass Classification
min
$,&
ℒ( ) = +,-
. )(0), 2 () = 34
5, 6(: 89:. <=>= <?@>. ).
Set ℛ( 5 = min
&
ℒ( 34
5 +
C
D
3 D
D
.
• Functional Gradient EFGH F in IJ KH,L
ℎN = 5N(O) ..
∇$ℛ( 5N (O) = QRS
.(34
5N(O), T) 5N(O) − VW∇$ℛ( 5N (O)
. . XYH
F
GH F .

13. • Problem setting: Multiclass Classification
min
$,&
ℒ( ) = +,-
. )(0), 2 () = 34
5, 6(: 89:. <=>= <?@>. ).
Set ℛ( 5 = min
&
ℒ( 34
5 +
C
D
3 D
D
.
• Functional Gradient EFGH F in IJ KH,L
ℎN = 5N(O) ..
∇$ℛ( 5N (O) = QRS
.(34
5N(O), T) 5N(O) − VW∇$ℛ( 5N (O)
. XYH
F
GH F . .

14. • . .
!" #, #%
&'(,)∇+ℛ) -" .
&'(,)∇+ℛ) -" / =
1
2
3
456
)
∇+ℛ) -" #4 !" #4, / .
, ,
. , .

15. • . , , ,
. !" #, #%
. , &'(,)∇+ℛ) -" .
&'(,)∇+ℛ) -" / =
1
2
3
456
)
∇+ℛ) -" #4 !" #4, / .
&'(,)∇+ℛ) -" . ,

16. • !": ℝ%
→ ℝ'
(" ), )+
= !" -" )
.
!" -" )′

17. • !": ℝ%
→ ℝ'
. ,
(" ), )+
= !" -" )
.
!" -" )′
• Kernel Assumption ∃1, 2 > 0 !" -" ) 5
≤ 2 and
• 1 ∇8ℛ: -" ;<
=
>?,@
5
≤ ∇8ℛ: -" , ABC,:∇8ℛ: -" ;D
=
>?,@
.
, , . - ,., , ,
, ABC,:∇8ℛ: -" . , , ,

18. • !": ℝ%
→ ℝ'
. .( .
(" ), )+
= !" -" )
.
!" -" )′
• Kernel Assumption ∃1, 2 > 0 !" -" ) 5
≤ 2 and
• 1 ∇8ℛ: -" ;<
=
>?,@
5
≤ ∇8ℛ: -" , ABC,:∇8ℛ: -" ;D
=
>?,@
.
• Specific kernel choice
!": ℝ%
→ ℝ%
) . . .
!" -" ) ∼ ∇8ℛ: -" ())/ ∇8ℛ: -" ())
5

19. • ) !: ℝ$
→ ℝ&
) ) )((, ( )
'( ), )+
= ! -( )
.
! -( )′
• Kernel Assumption ∃1, 2 > 0 ! -( ) 5
≤ 2 and
• 1 ∇8ℛ: -( ;<
=
>?,@
5
≤ ∇8ℛ: -( , ABC,:∇8ℛ: -( ;D
=
>?,@
.
• Specific kernel choice
) !: ℝ$
→ ℝ$
) .. ) ( , , , )( )
! -( ) ∼ ∇8ℛ: -( ())/ ∇8ℛ: -( ())
5
) ABC,:∇8ℛ: -( J = K(! -( J , K(: L, L MNOPQ)

20. • ) !: ℝ$
→ ℝ&
) ) )((, ( )
'( ), )+
= ! -( )
.
! -( )′
• Kernel Assumption ∃1, 2 > 0 ! -( ) 5
≤ 2 and
• 1 ∇8ℛ: -( ;<
=
>?,@
5
≤ ∇8ℛ: -( , ABC,:∇8ℛ: -( ;D
=
>?,@
.
• Specific kernel choice
) !: ℝ$
→ ℝ$
) .. ) ( , , , )( )
! -( ) ∼ ∇8ℛ: -( ())/ ∇8ℛ: -( ())
5
) ABC,:∇8ℛ: -( J = K(! -( J , K(: L, L MNOPQ)
∇8ℛ: -( ()R) () ) ( -(()R)

24. . ( )(. ( . ( .
!"#
, $% = '%()
*
+% ∇-ℒ/ $
( ( , . ,
0 ≤ 1/!"#
4 , '%
*
'% ≽ ∃78 > 0 ,
1
;
<
%=>
?@)
∇-ℒ/ $% AB
C DE,G
8
≤
2ℛ/ +>
0J78;
.
,
, min
%
∇-ℒ/ $% AB
C
DE,G
8

25. . ( )(. ( . ( .
!"#
$% = '%()
*
+% ∇-ℒ/ , . $
( ( . , .
# 0 ≤ 1/!"#
4 , , , . '%
*
'% ≽ ∃78 > 0
1
;
<
%=>
?@)
∇-ℒ/ $% AB
C DE,G
8
≤
2ℛ/ +>
0J78;
.
Bounding the margin distribution
L- M, N = $O M − max
OTUO
$OT M V . W: . , . . , , .
ℙDE
L- Z, [ ≤ 0 ≤ 2 V ∇-ℒ/ $
AB
C
(DE,G)
.

26. ( . . .
!"#
$% = '%()
*
+% ∇-ℒ/ , . $
. , .
# 0 ≤ 1/!"#
4 , , , . '%
*
'% ≽ ∃78 > 0
1
;
<
%=>
?@)
∇-ℒ/ $% AB
C DE,G
8
≤
2ℛ/ +>
0J78;
.
Bounding the margin distribution
L- M, N = $O M − max
OTUO
$OT M V . W: . , . . , , .
ℙDE
L- Z, [ ≤ ≤ 1 + 1/ exp − V ∇-ℒ/ $
AB
C
(DE,G)
.
( ( . ( . ) . . )

27. ) )( (
ℱ = #$(&') , - )* ( ( , - ,
$: ,- . : ( ( . , )

28. ) ) )
ℱ = #$(&') : ) : ) : - :
$: +, ( :- A - - . : ,
(. . > 0. ' 2 ≤ Λ5 : .
:: +6 : - 78
9
, &8, ;8<8#8 , . . , Λ=, Λ, Λ8
>
∀@ > 0 1 − @ - - C ∀D : :B :
ℙF GH I, J ≤ 0 ≤
2LM
Λ5Λ=
. N
O
8PQ
RS2
1 + ΛΛ8
>
+, +
1
2N
log
1
@
+ 1 + 1/ exp −. L ∇Hℒ^ D
_`
a (Fb,c)
.
) ) )
) )

29. . ( )(. ( . ( (.( .
ℱ = #$(&') ) ) -
$: +, ( : - - - . ,
( ( A . > 0. ' 2 ≤ Λ5 .
+6 - 78
9
, &8, ;8<8#8 , . . ,B Λ=, Λ, Λ8
>
∀@ > 0 : 1 − @ - - C ∀D
ℙF GH I, J ≤ 0 ≤
2LM
Λ5Λ=
. N
O
8PQ
RS2
1 + ΛΛ8
>
+, +
1
2N
log
1
@
+ 1 + 1/ exp −. L ∇Hℒ^ D
_`
a (Fb,c)
.
- - - .
- - . ,

30. . ( )(. ( . ( (.( .
ℱ = #$(&') ) ) -
$: +, ( : - - - . ,
( ( A . > 0. ' 2 ≤ Λ5 .
+6 - 78
9
, &8, ;8<8#8 , . . ,B Λ=, Λ, Λ8
>
∀@ > 0 : 1 − @ - - C ∀D
ℙF GH I, J ≤ 0 ≤
2LM
Λ5Λ=
. N
O
8PQ
RS2
1 + ΛΛ8
>
+, +
1
2N
log
1
@
+ 1 + 1/ exp −. L ∇Hℒ^ D
_`
a (Fb,c)
.
- - - .
- - . ,
( (

31. ! > 0. % & ≤ Λ) , ,
* +, % &
≤ - ., ∗0 1 , , Λ2 ∑4 (6,)∗4 & ≤ Λ88
, ∃:, ∀= > 0 1 − = ∀@AB1 , , ,
ℙD EFGHI
J, K ≤ 0 ≤
2MN
Λ)Λ2
! O
1 +
C
R − 1
S
,TU
AB&
V, ∇FℒY @, ZI
[
D,]
AB1
+
2MN
Λ^_Λ)Λ2
! O
+
1
2O
− log = + c RlogR + 1 +
1
exp −!
M ∇FℒY @AB1 ZI
[ (D,])
.
, , , , , ., . ,
. .

32. ! > 0. % & ≤ Λ) , ,
* +, % &
≤ - ., ∗0 1 , , Λ2 ∑4 (6,)∗4 & ≤ Λ88
, ∃:, ∀= > 0 1 − = ∀@AB1 , , ,
ℙD EFGHI
J, K ≤ 0 ≤
2MN
Λ)Λ2
! O
1 +
C
R − 1
S
,TU
AB&
V, ∇FℒY @, ZI
[
D,]
AB1
+
2MN
Λ^_Λ)Λ2
! O
+
1
2O
− log = + c RlogR + 1 +
1
exp −!
M ∇FℒY @AB1 ZI
[ (D,])
.
, , , , , ., . ,
.

33. ! > 0. % & ≤ Λ) , ,
* +, % &
≤ - ., ∗0 1 , , Λ2 ∑4 (6,)∗4 & ≤ Λ88
, ∃:, ∀= > 0 1 − = ∀@AB1 , , ,
ℙD EFGHI
J, K ≤ 0 ≤
2MN
Λ)Λ2
! O
1 +
C
R − 1
S
,TU
AB&
V, ∇FℒY @, ZI
[
D,]
AB1
+
2MN
Λ^_Λ)Λ2
! O
+
1
2O
− log = + c RlogR + 1 +
1
exp −!
M ∇FℒY @AB1 ZI
[ (D,])
.
.. . , ., , , V = c 1/Ri .,
, , , 1/RN(1Bi)/& .
c
1
Y
exp(R(1Bi)/&
) + log(1/=) +
ℛ kl
A(IHm)/n
.
o p

34. . . .
# # #

35. 1 6 1
• 05 5 1 =5 1 5=B 2 B = 5B8 .5 -5B
5F 2 5 B 5 B8 1= 5 I5 1 1 5B G1B = 2 5
• 5=5 1 G1B = 2 = = = B =1 1 5=B = 5 5 2B1 =5
85 5B 1 1 1=B55 = G = = B =1 1 5=B =
• ,= 5F 5 5=B B85 5 5 1= 5 B85 5B8 1 5 5
( ) 6 #
!
"
#
exp(((")*)/-
) + log(1/3) +
ℛ5 67
8(9:;)/<
.5 :
61 1 1 1
=> => =>