This slideshow describes a mathematics topic on quadratic form in English. I made the slideshow to understand this topic on a deeper way.　 I like linear algebra very much!!

1. Quadratic Form and Functional
Optimization
9th June, 2011 Junpei Tsuji

2. Optimization of multivariate
quadratic function
𝑥1 1 3 1 𝑥1
𝐽 𝑥1 , 𝑥2 = 1.2 + 0.2, 0.3 𝑥2 + 𝑥1 , 𝑥2 𝑥2
2 1 4
3 2
= 1.2 + 0.2𝑥1 + 0.3𝑥2 + 𝑥1 + 𝑥1 𝑥2 + 2𝑥2 2
2
𝑥1 , 𝑥2 , 𝐽 = 0.045, 0.064, 1.1881

3. Quadratic approximation
1
𝑓 𝒙 ≈ 𝑓 ̅ + 𝑱̅ ∙ 𝒙 − � +
𝒙 𝒙−�
𝒙 𝑇�
𝑯 𝒙−�
𝒙
By Taylor's expansion
2
constant linear form quadratic form
𝒙 ∶= 𝑥1 , 𝑥2 , ⋯ 𝑥 𝑝
𝑇
where
𝑓 ̅ ∶= 𝑓 �
𝒙
•
𝑱̅: = , ,⋯,
𝜕𝑓 𝜕𝑓 𝜕𝑓
•
𝜕𝑥1 𝜕𝑥2 𝜕𝑥 𝑝
�
𝒙=𝒙
• Jacobian (gradient)
⋯
𝜕2 𝑓 𝜕2 𝑓
𝜕𝑥1 𝜕𝑥1 𝜕𝑥1 𝜕𝑥 𝑝
• � ∶=
𝑯 ⋮ ⋱ ⋮
⋯
𝜕2 𝑓 𝜕2 𝑓
Hessian (constant)
𝜕𝑥 𝑝 𝜕𝑥1 𝜕𝑥 𝑝 𝜕𝑥 𝑝
�
𝒙=𝒙

4. Completing the square
1
𝑓 𝒙 = 𝑓 ̅ + 𝑱̅ ∙ 𝒙 − � +
𝒙 𝒙−�
𝒙 𝑇�
𝑯 𝒙−�
𝒙
2
• Let � = 𝒙∗ where 𝑱 𝒙∗ 𝑇 = 𝟎 then
𝒙
1
𝑓 𝒙 = 𝑓∗ + 𝒙 − 𝒙∗ 𝑇 𝑯∗ 𝒙 − 𝒙∗
2
constant quadratic form

5. Completing the square
1 𝑇
𝑓 𝒙 = 𝑐 + 𝒃 𝒙 + 𝒙 𝑨𝑨
𝑇
2
1
𝑓 𝒙 = 𝑑+ 𝒙 − 𝒙0 𝑇 𝑨 𝒙 − 𝒙0
2
1 1 1
= 𝑑 + 𝒙0 𝑇 𝑨𝒙0 − 𝒙0 𝑇 𝑨 + 𝑨 𝑇 𝒙 + 𝒙 𝑇 𝑨𝑨
2 2 2
𝒃𝑇 =− 𝒙0 𝑇 𝑨 + 𝑨 𝑇
1
2
𝒙0 𝑇 = −2𝒃 𝑇 𝑨 + 𝑨 𝑇 −1
•
𝒙0 = −2 𝑨 + 𝑨 𝑇 −1 𝒃
𝑐= 𝑑+ 𝒙0 𝑇 𝑨𝒙0
1
1 𝑇
2
𝑑= 𝑐− 𝒙0 𝑨𝒙0 = 𝑐 − 2𝒃 𝑇 𝑨 + 𝑨 𝑇 𝑨 𝑨+ 𝑨𝑇 𝒃
•
−1 −1
2
𝑓 𝒙 = 𝑐 − 2𝒃 𝑇 𝑨 + 𝑨 𝑇 −1 𝑨 𝑨+ 𝑨𝑇 −1 𝒃
Therefore,
1
+ 𝒙 + 2 𝑨 + 𝑨 𝑇 −1 𝒃 𝑇 𝑨 𝒙 + 2 𝑨 + 𝑨 𝑇 −1 𝒃
2
If 𝑨 was symmetric matrix,
1 𝑇 −1 1
𝑓 𝒙 = 𝑐− 𝒃 𝑨 𝒃+ 𝒙 + 𝑨−1 𝒃 𝑇 𝑨 𝒙 + 𝑨−1 𝒃
•
2 2

6. Quadratic form
𝑓 𝒙𝒙 = 𝒙𝒙 𝑇 𝑺𝑺𝑺
• 𝑺 is symmetric matrix.
where

7. Symmetric matrix
• Symmetric matrix 𝑺 is defined as a matrix that satisfies the
𝑺𝑇 = 𝑺
following formula:
• Symmetric matrix 𝑺 has real eigenvalues 𝜆 𝑖 and
eigenvectors 𝒖 𝑖 that consist of normal orthogonal base.
𝑺𝒖 𝑖 = 𝜆 𝑖 𝒖 𝑖
where
𝜆1 ≥ 𝜆2 ≥ ⋯ ≥ 𝜆 𝑝
𝒖 𝑖 , 𝒖 𝑗 = 𝛿 𝑖𝑖
𝛿 𝑖𝑖 is Kronecker's delta

8. Diagonalization of symmetric matrix
• We define an orthogonal matrix 𝑼 as follows:
𝑼 = 𝒖1 , 𝒖2 , ⋯ , 𝒖 𝑝
• Then, 𝑼 satisfies the following formulas:
𝑼𝑇 𝑼= 𝑰
∴ 𝑼−1 = 𝑼 𝑇
• where 𝑰 is an identity matrix.
𝑺𝑺 = 𝑺 𝒖1 , 𝒖2 , ⋯ , 𝒖 𝑝 = 𝑺𝒖1 , 𝑺𝒖2 , ⋯ , 𝑺𝒖 𝑝
𝜆1
= 𝜆1 𝒖1 , 𝜆2 𝒖2 , ⋯ , 𝜆 𝑝 𝒖 𝑝 = 𝒖1 , ⋯ , 𝒖 𝑝 ⋱
𝜆𝑝
= 𝑼 𝐝𝐝𝐝𝐝 𝜆1 , 𝜆2 , ⋯ , 𝜆 𝑝
∴ 𝑺 = 𝑼 𝐝𝐝𝐝𝐝 𝜆1 , 𝜆2 , ⋯ , 𝜆 𝑝 𝑼𝑇

9. Transformation to principal axis
𝑓 𝒙′ = 𝒙′ 𝑇 𝑺𝑺′
• Then, we assume 𝒙𝒙 = 𝑼 𝑇 𝒛, where 𝒛 =
𝑧1 , 𝑧1 , ⋯ , 𝑧 𝑝 .
𝑓 𝑼 𝑇 𝒛 = 𝑼 𝑇 𝒛 𝑇 𝑺 𝑼 𝑇 𝒛 = 𝒛 𝑇 𝑼𝑺𝑼 𝑇 𝒛
= 𝒛 𝑇 𝐝𝐝𝐝𝐝 𝜆1 , 𝜆2 , ⋯ , 𝜆 𝑝 𝒛
𝑝
∴ 𝑓 𝒛 = � 𝜆 𝑖 𝑧 𝑖2
𝑖=1

10. Contour surface
• If we assume 𝑓 𝒛 equals constant 𝑐,
𝑝
𝑓 𝒛 = � 𝜆 𝑖 𝑧 𝑖2 = 𝑐
𝑖=1
• When 𝑝 = 2,
– a locus of 𝒛 illustrates an ellipse if 𝜆1 𝜆2 > 0.
– a locus of 𝒛 illustrates a hyperbola if 𝜆1 𝜆2 < 0.

11. Contour surface
𝑧2 2
𝑓 𝒛 = � 𝜆 𝑖 𝑧 𝑖 2 = 𝑐𝑐𝑐𝑐𝑐.
𝑖=1
𝜆1 𝜆2 > 0
𝑧1
maximal or minimal point
𝑓 𝑥1 , 𝑥2 = −𝑥1 2 − 2𝑥2 2 + 20.0

12. Transformation to principal axis
𝑥𝑥2
𝑓 𝒙𝒙 = 𝑐𝑐𝑐𝑐𝑐.
𝑥𝑥1
𝒙𝒙 = 𝑼 𝑇 𝒛
∴ 𝒛 = 𝑼𝒙′
Transformation to principal axis

13. Parallel translation
𝑥𝑥2
𝑥2
�
𝒙 𝑥𝑥1
𝑓 𝒙 = 𝑐𝑐𝑐𝑐𝑐.
𝑥1
𝒙𝒙 = 𝒙 − �𝒙

14. 1
Contour surface of quadratic function
𝑓 𝒙 = 𝑓 +
∗
𝒙 − 𝒙∗ 𝑇
𝑯∗ 𝒙 − 𝒙∗
2
𝑥2
�
𝒙
𝑓 𝒙 = 𝑐𝑐𝑐𝑐𝑐.
𝑥1

15. Contour surface
𝑧2
2
𝑓 𝒛 = � 𝜆 𝑖 𝑧 𝑖 2 = 𝑐𝑐𝑐𝑐𝑐.
𝑖=1
𝜆1 𝜆2 < 0
𝑧1
saddle point
𝑓 𝑥1 , 𝑥2 = 𝑥1 2 − 𝑥2 2

16. Stationary points
𝑓 𝑥1 , 𝑥2 = 𝑥1 3 + 𝑥2 3 + 3𝑥1 𝑥2 + 2
maximal point
saddle point

17. Stationary points
1 3
𝑓 𝑥1 , 𝑥2 = exp − 𝑥1 + 𝑥1 − 𝑥2 2
3
saddle point
maximal point

19. Newton-Raphson method
𝑓𝑓 𝒙 = 𝟎 where 𝑓 𝒙 is 𝑁-th polynomial by
• Newton’s method is an approximate solver of
using a quadratic approximation.
𝑓 𝒙
quadratic approximation of 𝑓 𝒙 in 𝒙
1
𝑓 𝒙 + Δ𝒙 ≈ 𝑓 𝒙 + 𝑱 𝒙 ∙ Δ𝒙 + Δ𝒙 𝑇 𝑯 𝒙 Δ𝒙
2
𝜕𝑓 𝒙 + Δ𝒙
𝑓𝑓 𝒙∗ = 𝟎 𝜕 Δ𝒙
= 𝑱 𝒙 𝑇 + 𝑯 𝒙 Δ𝒙
𝒙∗ 𝒙 + 𝚫𝒙 𝒙
𝒙

20. Algorithm of Newton’s method
Procedure Newton (𝑱 𝒙 , 𝑯 𝒙 )
1. Initialize 𝒙.
2. Calculate 𝑱 𝒙 and 𝑯 𝒙 .
equation and giving ∆𝒙 :
𝑱 𝒙 𝑇 + 𝑯 𝒙 ∆𝒙 = 𝟎
3. Solve the following simultaneous
4. Update 𝒙 as follows:
𝒙 ← 𝒙 + ∆𝒙
5. If ∆𝒙 < 𝛿 then return 𝒙 else go
back to 2.

21. Linear regression
𝑝
𝑦
𝑦 = 𝑓 𝒙 = 𝛽0 + � 𝛽 𝑗 𝑥 𝑗
𝑁 samples
𝒙 𝑖, 𝑦 𝑖
𝑗=1
𝒙 𝑝-th dimensional space
We would like to find 𝜷∗ that minimizes the residual sum of square (RSS).

22. Linear regression
min RSS 𝜷
𝜷
2
𝑁 𝑁 𝑝
• where
RSS 𝜷 = � 𝑦 𝑖 − 𝑓 𝒙 𝑖 2 = � 𝑦𝑖 − 𝛽0 + � 𝛽 𝑗 𝑥 𝑖𝑖
𝑖=1 𝑖=1 𝑗=1
• Given 𝑿, 𝒚, 𝜷 as follows:
𝑥11 ⋯ 𝑥1𝑝 1 𝑦1 𝛽1
𝑿= ⋮ ⋱ ⋮ ⋮ , 𝒚= ⋮ , 𝜷= ⋮
𝑥 𝑁𝑁 ⋯ 𝑥 𝑁𝑁 1 𝑦𝑁 𝛽𝑝
∴ RSS 𝜷 = 𝒚 − 𝑿𝜷 2

23. Linear regression
RSS 𝜷 = 𝐽 𝜷 = 𝒚 − 𝑿𝜷 2 = 𝒚 − 𝑿𝜷 𝑇 𝒚 − 𝑿𝜷
= 𝒚 𝑇 𝒚 − 𝜷 𝑇 𝑿 𝑇 𝒚 − 𝒚 𝑇 𝑿𝜷 + 𝜷 𝑇 𝑿 𝑇 𝑿𝜷
𝒂𝑇 𝜷 = 𝒂
𝜕
𝜕𝜷
𝜷𝑇 𝒂 = 𝒂
•
𝜕
𝜕𝜷
𝜷 𝑇 𝑨𝜷 = 𝑨
•
𝜕
𝜕𝜷
𝜕𝐽
𝐽′ 𝜷 = = −2𝑿 𝑇 𝒚 + 2𝑿 𝑇 𝑿𝜷
•
𝜕𝜷

24. Linear regression
Given 𝜷∗ that satisfies 𝐽′ 𝜷∗ = 𝟎,
𝑿 𝑇 𝒚 = 𝑿 𝑇 𝑿𝜷∗
𝒚 𝑇 𝑿 = 𝜷∗ 𝑇 𝑿 𝑇 𝑿
∴ 𝜷∗ = 𝑿𝑇 𝑿 −1
𝑿𝑇 𝒚
∴ 𝐽 𝜷 = 𝒚 𝒚 − 𝜷 𝑿 𝑿𝜷 − 𝜷 𝑿 𝑇 𝑿𝜷 + 𝜷 𝑇 𝑿 𝑇 𝑿𝜷
𝑇 𝑇 𝑇 ∗ ∗𝑇
∴ 𝐽 𝜷
= 𝒚 𝑇 𝒚 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷∗ + 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷∗ − 𝜷 𝑇 𝑿 𝑇 𝑿𝜷∗
− 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷 + 𝜷 𝑇 𝑿 𝑇 𝑿𝜷
∴ 𝐽 𝜷 = 𝒚 𝑇 𝒚 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷∗ + 𝜷 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿 𝜷 − 𝜷∗
completing the square

25. Linear regression
𝐽 𝜷 = 𝒚 𝑇 𝒚 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿𝜷∗ + 𝜷 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿 𝜷 − 𝜷∗
= 𝒚 − 𝑿𝜷∗ 2 + 𝜷 − 𝜷∗ 𝑇 𝑿 𝑇 𝑿 𝜷 − 𝜷∗
1
= 𝐽 𝜷 +
∗
𝜷 − 𝜷 ∗ 𝑇 𝑯 𝜷 − 𝜷∗
2
Residual sum of squares (RSS) quadratic form
𝛽2 𝐽 𝜷 = 𝑐𝑐𝑐𝑐𝑐.
by Linear Regression
𝜷∗
𝜷∗ = 𝑿 𝑇 𝑿 −1 𝑿 𝑇 𝒚
𝑯 = 2𝑿 𝑇 𝑿
𝛽1

26. Hessian
• 𝑯≔ = 2𝑿 𝑇 𝑿
𝜕2 𝐽
𝜕𝛽 𝑖 𝜕𝛽 𝑗
• 𝑯 has the following two features:
𝑯𝑇 = 𝑯
∀ 𝒙 ≠ 𝟎, 𝒙 𝑇 𝑯𝑯 > 0
– symmetric matrix:
– positive-definite matrix:
Therefore, 𝜷∗ = 𝑿𝑇 𝑿 −1
𝑿 𝑇 𝒚 is the minimum
of 𝐽 𝜷 .

27. Analysis of residuals
𝒚∗ = 𝑿𝜷∗
• Then, we substitute 𝜷∗ = 𝑿 𝑇 𝑿 −1
𝑿 𝑇 𝒚 in the above,
𝒚∗ = 𝑿𝜷∗ = 𝑿 𝑿 𝑇 𝑿 −1
𝑿𝑇 𝒚
∴ 𝒚∗ = ℋ𝒚 (Hat matrix)
• the vector of residuals 𝒓 can be expressed by follows:
𝒓 = 𝒚 − 𝒚∗ = 𝒚 − ℋ𝒚 = 𝑰 − ℋ 𝒚
𝑉𝑉𝑉 𝒓 = 𝑉𝑉𝑉 𝑰 − ℋ 𝒚 = 𝑰 − ℋ 𝑉𝑉𝑉 𝒚 𝑰 − ℋ 𝑇

28. Analysis of residuals
ℋ = 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇
The hat matrix ℋ is a projection matrix, which
1. Projection: ℋ 2 = ℋ
satisfies the following equations:
ℋ 2 = ℋ ∙ ℋ = 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇 ∙ 𝑿 𝑿 𝑇 𝑿 −1
𝑿𝑇
= 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇
= 𝑿 𝑿 𝑇 𝑿 −1 𝑿 𝑇 = ℋ
2. Orthogonal: ℋ 𝑇 = ℋ

29. Analysis of residuals
𝑥11 ⋯ 𝑥1𝑝 1 𝛽1 ∗
𝑦1 ∗ ⋮
⋮ = ⋮ ⋱ ⋮ ⋮
𝛽𝑝 ∗
𝑦 𝑁∗ 𝑥 𝑁1 ⋯ 𝑥 𝑁𝑁 1
𝛽0 ∗
𝑥11 𝑥1𝑝 1
= 𝛽1 ∗ ⋮ + ⋯ + 𝛽 𝑝∗ ⋮ + 𝛽0 ⋮
∗
𝑥 𝑁1 𝑥 𝑁𝑁 1
𝒙1 𝒙𝑝 𝒙 𝑝+1 = 𝟏
linear combination in 𝑝 + 1 -th vector space

30. Analysis of residuals
𝒚
𝒚∗ = ℋ𝒚 (Projection)
𝒙𝑝
𝒚∗
𝒙𝑗
𝑝 + 1 -th dimensional super surface
𝑁-th dimensional space

31. Analysis of residuals
𝒚 = 𝑿𝜷
• 𝜷 = 𝑿−1 𝒚, where 𝑿−1 is M-P generalized inverse.
𝑝= 𝑁
𝑝> 𝑁
1. Unique solution:
𝑝< 𝑁
2. Many solutions:
𝑿−1
3. No solution:
𝑿 −1
=� 𝑿𝑿 𝑿𝑿𝑿 −1 𝜷 = 𝑿−1 𝒚 is min in 𝜷
𝑿𝑿𝑿 −1 𝑿𝑿 𝒚 − 𝑿𝜷 2 is min
•