Normalized Graph Cut For Advanced Image Segmentation based on the research carried out by DR sha and DR malik

1. Image Segmentation using Normalized Graph Cut
By W A T Mahesh Dananjaya
110089M
Abstract: Image Processing is becoming paramount important technology to the modern world since it
is the caliber behind the machine learning and so called artificial intelligence. Image segmentation is
one of the major area of the modern image processing and computer vision. Many computer vision
researches that have been carrying out emerge the important of pattern analysis and perceptual vision of
a scene and regression of features. Image segmentation is inherent strength of image processing
techniques for pattern recognition and regression.
Normalized Graph cut method was proposed by Jianbo Shi and Jitendra Malik to address the
problem of perceptual grouping and organization in vision. Proposed method has provided a powerful
way of extracting and understanding of the global impression of the image rather than focusing heavily
on the local features and their consistencies in the image data. In this approach image is considered as
a Graph of image nodes and apply the solution in the way of graph partitioning problem. This method
was really address the drawbacks of exist image segmentation method using minimum cut. This
methodology has provided the new measure of graph partitioning called disassociation measure the
normalized cuts by taking total edge connection to all the nodes in the graph to compute the cut cost.
Normalized cut method is not only taking total dissimilarity between the different groups, but also it
takes total similarities within a group for the graph partitioning process. This method has come up with
advanced algorithm comprising with solution of generalized eigenvalue problem for optimal graph
partitioning including prevailing knowledge on image clustering grouping and segmentation.
1. Introduction
1.1 Minimum Cut
Although there were boundary based and region based methodologies for image clustering those
techniques do not provide a global impression for the computer vision based applications. Graph
partitioning is the most available and sustainable way to absorb global perception of an image rather
than focusing on local properties of the image. Therefore Graph theories and Eigen vector based methods
are used to achieve an optimal solution to image segmentation previously. Wu and Leahy proposed an
image clustering methodology based on minimum cut criteria compromising the optimal graph
portioning. In their method, they partition a graph into k-sub graphs in a way that the maximum cut
across the sub graph is minimized. This globally optimal criterion can be used to produce good
segmentation on some of the images. But this minimum cut is not always favorable. This criteria favors
cutting small sets of isolated nodes in the graph and gives bad partitioning when cost (weight) function
is inverse to the distance (Similarity) between nodes.

2. In the graph partitioning approach first the given image is taken as graph G= (V, E) and then segmented.
The Minimum cut method sometimes gives bad partitioning because this method is only looking at the
value of total edge weight connecting the two partition and no measure or parameter is used to indicate
the number of edge connection as a fraction of total connection to all the nodes in the graph.
In usual graph partitioning graph G= (V, E) is partitioned into two disjoint sub sets, A, B such that
A𝐴 ∪ 𝐵 ≡ 𝑉 𝑎𝑛𝑑 𝐴 ∩ 𝐵 ≡ ∅ by simply removing edges connecting the two parts. The degree of
dissimilarity between these two pieces can be computed as total weight of the edges that have been
removed. Cut is defined as:
𝐶𝑢𝑡 (𝐴, 𝐵) = 𝑤(𝑢, 𝑣)𝑢∈𝐴,𝑣∈𝐵 (1)
In minimum cut criteria the optimal partitioning of the graph is the cut that minimizes the total cost or
weight of the cut as the cost is inversely proportional to the distance or similarity between nodes. This
emerges the inherent drawback of the minimum cut criteria. Therefore the problem remains with the
isolated point in an image segmentation and clustering when we use minimum cuts. Requirement for the
novel method with a new measure of association of graph node is accomplished by the normalized cut
technique.
1.2 Normalized Cut
In this new method new measure to defined to calculate the association between nodes for a
graph cut. This usually known as disassociation measure the normalized cut.
𝑁𝑐𝑢𝑡 (𝐴, 𝐵) =
𝑐𝑢𝑡(𝐴,𝐵)
𝑎𝑠𝑠𝑜𝑐(𝐴,𝑉)
+
𝑐𝑢𝑡(𝐴,𝐵)
𝑎𝑠𝑠𝑜𝑐(𝐵,)
(2)
Where 𝑎𝑠𝑠𝑜𝑐(𝐴, 𝑉) = 𝑤(𝑢, 𝑡)𝑢∈𝐴,𝑡∈𝑉 is the total connection from node in A to all nodes in the graph
and 𝑎𝑠𝑠𝑜𝑐(𝐵, 𝑉) = 𝑤(𝑣, 𝑡)𝑣∈𝐵,𝑡∈𝑉 is the total connection from node in B to all nodes in the graph.
In this new way of defining disassociation between groups, the isolated points are no longer appear and
no longer have small Ncut values. Because of the assoc value is much smaller and provide large Ncut
value. And also we can define a measure for total normalized association within group for a given
partition.

3. 𝑁𝑎𝑠𝑠𝑜𝑐(𝐴, 𝐵) =
𝑎𝑠𝑠𝑜𝑐(𝐴,𝐴)
𝑎𝑠𝑠𝑜𝑐(𝐴,𝑉)
+
𝑎𝑠𝑠𝑜𝑐(𝐵,𝐵)
𝑎𝑠𝑠𝑜𝑐(𝐵,𝑉)
(3)
And also we can derived an important property between Ncut and Nassoc.
𝑁𝑐𝑢𝑡(𝐴, 𝐵) = 2 − 𝑁𝑎𝑠𝑠𝑜𝑐(𝐴, 𝐵) (4)
Therefore this graph partitioning algorithm minimize the disassociation
between the groups and maximize the association within the groups. Therefore it is essential to
have a normalized cut as the partitioning criteria.
2. Methodology
2.1 Optimal Partition and Generalized Eigenvalue Problem
Let 𝑑(𝑖) = 𝜔(𝑖, 𝑗)𝑗 be the total number of connection from node I to all other nodes. D
be the N× 𝑁 diagonal matrix with 𝑑(𝑖) on its diagonal.
Let W be a 𝑁 × 𝑁 symmetric matrix with𝑊(𝑖, 𝑗) = 𝑤(𝑖, 𝑗).
By complex mathematical calculation we can show that this optimal partitioning problem
is converged to the following expression.
𝑚𝑖𝑛 𝐴,𝐵 𝑁𝑐𝑢𝑡(𝐴, 𝐵) = 𝑚𝑖𝑛 𝑦
𝑦 𝑇(𝐷−𝑊)𝑦
𝑦 𝑇 𝐷𝑦
(5)
And if y is relaxed y is relaxed to take real value, we can minimize (5) by solving generalized
eigenvalue problem.
(𝐷 − 𝑊)𝑦 = 𝛾𝐷𝑦 (6)
2.2 Design steps
Step 1: Given an image sequence I and N=number of pixels
I – Image Sequence
Step 2: Construct the Weighted Graph G= (V, E) where each node represent a pixel in the
image I. N= |V| = number of nodes (pixels) in the graph
𝐺 = (𝑉, 𝐸)
N= |V|

4. Step 3: W-Compute the N  N weighted matrix (Similarity) as
𝑒𝑥𝑝
− 𝑋(𝑖)−𝑋(𝑗) 2
2
𝜎 𝑋
2
𝑖𝑓 𝑋(𝑖) − 𝑋(𝑗) 2
< 𝑟
𝑤(𝑖, 𝑗) = 𝑒𝑥𝑝
− 𝐹(𝑖)−𝐹(𝑗) 2
2
𝜎𝐼
2
× 0 else
Where 𝑋 ( 𝑖 ) is the spatial location of node I, i.e. the coordinate in the original image I
𝑎𝑛𝑑 𝐹(𝑖) is a feature vector defined as:
 𝐹(𝑖) = 1 for segmentation point set
 𝐹(𝑖) = 𝐼(𝑖) the intensity value for segmentation brightness (gray scale) images
 𝐹(𝑖) = [𝑣, 𝑣. 𝑠. 𝑠𝑖𝑛(ℎ), 𝑣. 𝑠. 𝑐𝑜𝑠(ℎ)](𝑖) Where h, s, v are the HSV values for
color segmentation
 𝐹(𝑖) = [ 𝐼 ∗ 𝑓1 , … , 𝐼 ∗ 𝑓𝑛 ](𝑖) 𝑤ℎ𝑒𝑟𝑒 𝑡ℎ𝑒 𝑓𝑖 are DOOG filter at various scale
and orientation
Let 𝑑(𝑖) = 𝑤(𝑖, 𝑗)𝑗 be the total connection from node I to all other node.
Step 4: D-Construct an N*N diagonal matrix D with d on its diagonal
Step 5: Model and Solve generalized Eigenvalue System
(𝐷 − 𝑊)𝑦 = 𝛾𝐷𝑦
Solve and get the eigenvector with the second smallest eigenvalue.
Step 6: Bi-Partitioning the graph
By using the eigenvector to bipartition the graph. In ideal case eigenvector only take on
two discrete values and the signs tell us how to partition the graph
( 𝐴 = 𝑉𝑖 𝑌𝑖 > 0 ), 𝐵 = 𝑉𝑖 𝑌𝑖 ≤ 0
However y is relaxed to take real value other than discrete values. So it is essential to select
splitting point. We can use various methods choose splitting point.
 0 value
 Median value
 Search a splitting point which result in that 𝑁𝑐𝑢𝑡 (𝐴, 𝐵) is minimized.
That will satisfy the equation that we derived earlier
𝑚𝑖𝑛 𝐴,𝐵 𝑁𝑐𝑢𝑡(𝐴, 𝐵) = 𝑚𝑖𝑛 𝑦
𝑦 𝑇(𝐷−𝑊)𝑦
𝑦 𝑇 𝐷𝑦
By minimizing
𝑦 𝑇(𝐷−𝑊)𝑦
𝑦 𝑇 𝐷𝑦

5. For (5) and (6) we have to some mathematical calculations which are not going to be
covered in this paper. There we put a substitution.
𝑌 = (1 + 𝑥) − 𝑏(1 − 𝑥) Where 𝑏 = 𝑘/(1 − 𝑘)
𝑘 =
𝑑𝑖𝑥 𝑖>0
𝑑𝑖𝑖
To find the minimum Ncut, we need to try different values of splitting points. The
optimal splitting point is generally around the mean value of the obtained eigenvector.
Step 7: Recursive Repartition
Repeat bipartition process recursively until some constraint will be acquired by the
program. We can stop if Ncut value is larger than a pre specified threshold value. There
is another constraint. We can stop if the partition area is smaller than a pre specified
threshold value. Those are little advancement for the original program.
3. ADVANCEMENTS & DEVELOPMENTS
Apart from the original concept of the research carried out by Shi I have added some
more features in to the algorithm based on the previous development of the same
concept. I have inserted another two parameters to restrict the unwanted processing
power.
SNV -> Smallest Normalized-Cut Value
SNA -> Smallest Normalized-Cut Area
SNV is used to stop the process if the Ncut value of the partition is smaller than pre
specified SNV (threshold) value.
SNA is used to stop the process if the total number of nodes in the partition (Area)is
smaller than the pre specified threshold (SNA)value.
These two Threshold values terminate the unwanted recursive bipartition of out
grouping algorithm.

6. 4. Experimental Results
4.1 Gray Scale Image Segmentation
First image is the original one of size 100*67.Image intensity is normalized to lie
within 0 and 255. Subplots shows the component of the partition. Here I used
𝑆𝐼 = 5.0 𝑆𝑋 = 4.0 𝑅 = 1.5 𝑆𝑁𝑉 = 0.14 𝑆𝑁𝐴 = 220
Values were selected based on the prior literature on graph partitioning. Image with
a Low resolution is used to reduce the calculation time.
𝑅𝑂𝑂𝑇 − 𝐴. 𝑁𝑐𝑢𝑡 = 0.119640
𝑅𝑂𝑂𝑇 − 𝐵 − 𝐴. 𝑁𝑐𝑢𝑡 = 0.086190
𝑅𝑂𝑂𝑇 − 𝐵 − 𝐵 − 𝐴 − 𝐴. 𝑁𝑐𝑢𝑡 = 0.134254
𝑅𝑂𝑂𝑇 − 𝐵 − 𝐵 − 𝐴 − 𝐵. 𝑁𝑐𝑢𝑡 = 0.109798
𝑅𝑂𝑂𝑇 − 𝐵 − 𝐵 − 𝐵 − 𝐴. 𝑁𝑐𝑢𝑡 = 0.124375
𝑅𝑂𝑂𝑇 − 𝐵 − 𝐵 − 𝐵 − 𝐵. 𝑁𝑐𝑢𝑡 = 0.113683
𝐸𝑙𝑎𝑝𝑠𝑒𝑑 𝑡𝑖𝑚𝑒 𝑖𝑠 2.351473 𝑠𝑒𝑐𝑜𝑛𝑑𝑠

7. 3.2 Color Image Segmentation
First image is the original one of size 100*67.Image RGB intensity is normalized to
lie within 0 and 255. Subplots shows the component of the partition. Here I used
𝑆𝐼 = 5.0 𝑆𝑋 = 6.0 𝑅 = 1.5 𝑆𝑁𝑉 = 0.21 𝑆𝑁𝐴 = 120
Values were selected based on the prior literature on graph partitioning. Image with
a Low resolution is used to reduce the calculation time.
𝑅𝑂𝑂𝑇 − 𝐴 − 𝐴 − 𝐴. 𝑁𝑐𝑢𝑡 = 0.161737
𝑅𝑂𝑂𝑇 − 𝐴 − 𝐴 − 𝐵. 𝑁𝑐𝑢𝑡 = 0.203867
𝑅𝑂𝑂𝑇 − 𝐴 − 𝐵. 𝑁𝑐𝑢𝑡 = 0.098142
𝑅𝑂𝑂𝑇 − 𝐵 − 𝐴. 𝑁𝑐𝑢𝑡 = 0.140840
𝑅𝑂𝑂𝑇 − 𝐵 − 𝐵. 𝑁𝑐𝑢𝑡 = 0.002487
𝐸𝑙𝑎𝑝𝑠𝑒𝑑 𝑡𝑖𝑚𝑒 𝑖𝑠 2.439764 𝑠𝑒𝑐𝑜𝑛𝑑𝑠.

8. 5. Comparison
The results are compared with the original implementation of Dr.Shi. According to
Resuts it took nearly minutes for the calculation for the Dr.Shi.
But My program is work much better and took only 10 seconds to complete the
work.
𝑅𝑂𝑂𝑇 − 𝐴 − 𝐴. 𝑁𝑐𝑢𝑡 = 0.119519
𝑅𝑂𝑂𝑇 − 𝐴 − 𝐵 − 𝐴. 𝑁𝑐𝑢𝑡 = 0.027895
𝑅𝑂𝑂𝑇 − 𝐴 − 𝐵 − 𝐵. 𝑁𝑐𝑢𝑡 = 0.000000
𝑅𝑂𝑂𝑇 − 𝐵 − 𝐴. 𝑁𝑐𝑢𝑡 = 0.026061
𝑅𝑂𝑂𝑇 − 𝐵 − 𝐵. 𝑁𝑐𝑢𝑡 = 0.020161
𝐸𝑙𝑎𝑝𝑠𝑒𝑑 𝑡𝑖𝑚𝑒 𝑖𝑠 10.907697 𝑠𝑒𝑐𝑜𝑛𝑑𝑠.

9. 6. REFERENCES
[1] Jianbo Shi and Jitendra Malik, "Normalized Cuts and Image
Segmentation," IEEE Transactions on PAMI, Vol. 22, No. 8, Aug. 2000.
http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf
[2] Graph Based Image Segmentation Tutorial
http://www.cis.upenn.edu/~jshi/GraphTutorial/
[3] MATLAB Normalized Cuts Segmentation Code
http://www.cis.upenn.edu/~jshi/software/
[4] D. Martin and C. Fowlkes and D. Tal and J. Malik, "A Database of
Human Segmented Natural Images and its Application to Evaluating
Segmentation Algorithms and Measuring Ecological Statistics",
Proc. 8th Int'l Conf. Computer Vision, vol. 2, pp. 416-423, July 2001.
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/
7. APPENDIX
calcNcutSegment.m
function Iseg = calcNcutSegment(I, SI, SX, R, SNV, SNA)
% NcutImageSegment - Normalized Cuts and Image Segmentation [1]
% [SegI] = NcutImageSegment(I, SI, SX, r, SNV, SNA)
% Parameters
% I NR*NC*C Image Matrix
% SI Coefficient Related to Feature
% SX Coefficient related to Position
% R Coefficient used to compute similarity (weight) matrix
% SNV Smalllest Normalized-Cut Value(Threashold)
% SNA Smallest Normalized-Cut Area (Threshols)
% SegI cell array of segmented images of NR*NC*C
[NR, NC, C] = size(I);
% Step 1 : Calculate |V|=N
N = NR * NC;
% Step 2: Construcyt Nodes (Vertices) of the Grapg G=(V,E)
V = reshape(I, N, C);
% Step 3. Compute weight matrix W and D
W = calcNcutW(I, SI, SX, R);
% Step 7. recursively repartition
Seg = (1:N)'; % the first segment has whole nodes. [1 2 3 ... N]'
[Seg Id Ncut] = calcNcutPartition(Seg, W, SNV, SNA, 'ROOT');
% Convert node ids into images
for i=1:length(Seg)
subV = zeros(N, C); %ones(N, c) * 255;
subV(Seg{i}, :) = V(Seg{i}, :);
Iseg{i} = uint8(reshape(subV, NR, NC, C));
fprintf('%s. Ncut = %fn', Id{i}, Ncut{i});
end

10. end
calcNcutW.m
function W = calcNcutW(I, SI, SX, R);
% NcutComputeW - Compute a similarity (weight) matrix
[NR, NC, C] = size(I);
N = NR* NC;
W = sparse(N,N);
% Feature Vectors
if C == 3
F = F3(I);
else
F = F2(I);
end
F = reshape(F, N, 1, C);
X = cat(3, repmat((1:NR)', 1, NC), repmat((1:NC), NR, 1));
X = reshape(X, N, 1, 2);
for ic=1:NC
for ir=1:NR
% This range satisfies |X(i) - X(j)| <= r (block distance)
jc = (ic - floor(R)) : (ic + floor(R)); % vector
jr = ((ir - floor(R)) :(ir + floor(R)))';
jc = jc(jc >= 1 & jc <= NC);
jr = jr(jr >= 1 & jr <= NR);
jN = length(jc) * length(jr);
% index at vertex. V(i)
i = ir + (ic - 1) * NR;
j = repmat(jr, 1, length(jc)) + repmat((jc -1) * NR, length(jr),
1);
j = reshape(j, length(jc) * length(jr), 1); % a col vector
% spatial location distance (disimilarity)
XJ = X(j, 1, :);
XI = repmat(X(i, 1, :), length(j), 1);
DX = XI - XJ;
DX = sum(DX .* DX, 3); % squared euclid distance
%DX = sum(abs(DX), 3); % block distance
% square (block) reagion may work better for skew lines than circle
(euclid) reagion.
% |X(i) - X(j)| <= r (already satisfied if block distance
measurement)
constraint = find(sqrt(DX) <= R);
j = j(constraint);
DX = DX(constraint);
% feature vector disimilarity
FJ = F(j, 1, :);
FI = repmat(F(i, 1, :), length(j), 1);

11. DF = FI - FJ;
DF = sum(DF .* DF, 3); % squared euclid distance
%DF = sum(abs(DF), 3); % block distance
% Hint: W(i, j) is a col vector even if j is a matrix
W(i, j) = exp(-DF / (SI*SI)) .* exp(-DX / (SX*SX)); % for squared
distance
%W(i, j) = exp(-DF / SI) .* exp(-DX / SX);
end
end
end
% F1 - F4: Compute a feature vector F
% F = F1(I) % for point sets
% F = F2(I) % intensity
% F = F3(I) % hsv, for color
% F = F4(I) % DOOG
%
function F = F1(I);
F = (I == 0);
end
function F = F2(I);
% intensity, for gray scale
F = I;
end
function F = F3(I);
F = I; % raw RGB
end
function F = F4(I);
% DOOG, for texture
% Future
End
calcNcutPartition.m
function [Seg Id Ncut] = calcNcutPartition(I, W,SNV,SNA, ID)
% Compute D
N = length(W);
d = sum(W, 2);
D = spdiags(d, 0, N, N); % diagonal matrix
% Step 2 and 3. Solve generalized eigensystem (D -W)*S = S*D*U (12).
warning off; % let me stop warning
[U,S] = eigs(D-W, D, 2, 'sm');
% 2nd smallest (1st smallest has all same value elements, and useless)
U2 = U(:, 2);
% T = abs(U2); m = max(T); T = T / m * 255; imshow(uint8(reshape(T, 15,
20)));
% Step 3. Refer 3.1 Example 3.
% (1). Bipartition the graph at 0 (hopefully, eigenvector can be
% splitted by + and -). % This did not work well.
%A = find(U2 > 0);

12. %B = find(U2 <= 0);
% (2). Bipartition the graph at median value.
%t = median(U2);
%A = find(U2 > t);
%B = find(U2 <= t);
% (3). Bipartition the graph at point that Ncut is minimized.
t = mean(U2);
t = fminsearch('NcutValue', t, [], U2, W, D);
A = find(U2 > t);
B = find(U2 <= t);
% Step 4. Decide if the current partition should be divided
% if either of partition is too small, stop recursion.
% if Ncut is larger than threshold, stop recursion.
ncut = calcNcutValue(t, U2, W, D);
if (length(A) < SNA || length(B) < SNA) || ncut > SNV
Seg{1} = I;
Id{1} = ID; % for debugging
Ncut{1} = ncut; % for duebuggin
return;
end
% Seg segments of A
[SegA IdA NcutA] = NcutPartition(I(A), W(A, A), SNV, SNA, [ID '-A']);
% I(A): node index at V. A is index at the segment, I
% W(A, A); % weight matrix in segment A
% Seg segments of B
[SegB IdB NcutB] = NcutPartition(I(B), W(B, B), SNV, SNA, [ID '-B']);
% concatenate cell arrays
Seg = [SegA SegB];
Id = [IdA IdB];
Ncut = [NcutA NcutB];
end
calcNcutValue.m
function ncut = calcNcutValue(t, U2, W, D);
% NcutValue - Computing the Optimal Partition Ncut
x = (U2 > t);
x = (2 * x) - 1;
d = diag(D);
k = sum(d(x > 0)) / sum(d);
b = k / (1 - k);
y = (1 + x) - b * (1 - x);
ncut = (y' * (D - W) * y) / ( y' * D * y );
end