1. C OMPUTER V ISION :
S INGULAR VALUE D ECOMPOSITION
IIT Kharagpur
Computer Science and Engineering,
Indian Institute of Technology
Kharagpur.
(IIT Kharagpur) Minimization Jan ’10 1 / 11

2. Singular Value Decomposition SVD
An m × n matrix of rank r maps an r dimensional unit hypersphere
in rowspace (A) into an r -dimensional hyper-ellipse in range (A).
The rank 2 matrix
 √ √ 
 3 3
1 


b = Ax
 
A = √  −3

 3 


2 1
 
1

transforms the unit circle on the plane into an ellipse embedded in
three-dimensional space.
(IIT Kharagpur) Minimization Jan ’10 2 / 11

3. Two diametrically opposite points on the unit circle are mapped
into the two endpoints of the major axis of the ellipse, and two
other diametrically opposite points on the unit circle are mapped
into the two endpoints of the minor axis of the ellipse.
The lines through these two pairs of points on the unit circle are
always orthogonal. This result can be generalized to any m × n
matrix.
(IIT Kharagpur) Minimization Jan ’10 3 / 11

4. SVD
If A is a real m × n matrix then there exist orthogonal matrices U, V
U = [u1 , u2 , . . . , um ] ∈ Rm×m
V = [v1 , v2 , . . . , vn ] ∈ Rn×n
such that
T
U AV = Σ = diag(σ1 , . . . , σp ) ∈ Rm×n
where p = min(m, n) and σ1 ≥ σ2 ≥ . . . σp ≥ 0.
Equivalently A = UΣV T
Vectors v are the right singular vectors
Vectors u are the left singular vectors
Σ has real non-negative diagonal entries called as singular values.
(IIT Kharagpur) Minimization Jan ’10 4 / 11

5. (IIT Kharagpur) Minimization Jan ’10 5 / 11

6. SVD
SVD is one of the most useful matrix decompositions, particularly
for numerical computations.
Its most common application is in the solution of over-determined
systems of equations.
Generally the decomposition is carried out in such a way that the
diagonal entries of Σ are in descending order.
Since matrices U and V are orthogonal, they have the following
properties:
||Ux|| = ||x|| for any vector x
T
U U = In×n
In general UU T is not the identitiy unless m = n.
(IIT Kharagpur) Minimization Jan ’10 6 / 11

7. SVD
One can also define SVD for matrices with more columns than
rows, but generally this will not be of interest to us. It is
appropriate to extend A by adding rows of zeros to obtain a square
matrix, and then take the SVD.
Common implementations of SVD assume that m ≥ n. In this
case matrix U has the same dimension m × n as the input, matrix
A may be over-written by the output matrix U.
Since matrices U and V are orthogonal, they have the following
properties:
||Ux|| = ||x|| for any vector x
T
U U = In×n
In general UU T is not the identitiy unless m = n.
(IIT Kharagpur) Minimization Jan ’10 7 / 11

8. Fitting a line
Let pi = (xi , yi )T be a set of m ≥ 2 points on a plane.
Consider the line ax + by − c = 0
The normal to the line is given by the vector n = (a, b)T . We can
assume that ||n|| = a2 + b2 = 1, thus n is a unit vector.
The distance of this line to the origin is |c|, and the distance
between the line n and the point pi is
di = |axi + byi − c| = |piT n − c|
The best fit line minimizes the sum of squared distances.
min ||d||2 = min ||Pn − c1||2
||n||=1 ||n||=1
where P = (p1 , . . . , pm )T , d = (d1 , . . . , dm ), 1 is a vector of m ones.
(IIT Kharagpur) Minimization Jan ’10 8 / 11

9. Fitting a line
min ||d||2 = min ||Pn − c1||2
||n||=1 ||n||=1
Minimization w.r.t to c is done by
∂||d||2
=0 gives c = pT n
∂c
1 T
where p is the mean p = mP 1
Substituting for c, the minimization problem w.r.t n is:
min ||d||2 = min ||Pn − 1pT n||2 = min ||Qn||2
||n||=1 ||n||=1 ||n||=1
The solution is n = v2 where v2 is the second column of the
matrix V .
(IIT Kharagpur) Minimization Jan ’10 9 / 11

10. function [ l, residue ] = linefit(P)
[ m n ] = size(P);
if n = 2, error(’matrix P must be m x 2’), end
if m < 2, error(’Need at least two points’), end
one = ones(m, 1);
p = (P’ * one) / m;
Q = P - one * p’;
[U D V] = svd(Q);
n = V(:, 2);
l = [n ; p’ * n];
residue = D(2, 2);
(IIT Kharagpur) Minimization Jan ’10 10 / 11

11. Singular Values and Eigen Values
(IIT Kharagpur) Minimization Jan ’10 11 / 11