1. 1/39
An Algorithm for Incremental
Unsupervised Learning and
Topology Representation
Shen Furao
Hasegawa Lab
Department of Computational
Intelligence and Systems Science

2. 2/39
Contents
 Chapter 1: Introduction
 Chapter 2: Vector Quantization
 Chapter 3: Adaptive Incremental LBG
 Chapter 4: Experiment of adaptive
incremental LBG
 Chapter 5: Self-organizing incremental
neural network
 Chapter 6: Experiment with artificial data
 Chapter 7: Application
 Chapter 8: Conclusion and discussion

3. 3/39
Introduction
 Clustering: Construct decision boundaries
based on unlabeled data.
 Topology learning: find a topology
structure that closely reflects the topology
of the data distribution
 Online incremental learning: Adapt to new
information without corrupting previously
learned information

4. 4/39
Vector Quantization
 Targets
 To minimize the average distortion through a
suitable choice of codewords
 Application
 Data compression, speech recognition
 Separate the data set to Voronoi regions, find the
centroid of the Voronoi regions
 LBG method (Linde, Buzo & Gray, 1980)
 Dependence on initial starting conditions
 Tendency to result in local minima

5. 5/39
Adaptive incremental LBG
(Shen & Hasegawa, 2005)
 To solve the problem caused by poorly chosen
initial conditions
 independent of initial conditions
 With fixed number of codewords, to find a suitable
codebook to minimize the distortion error MQE.
 It can work better than or same as ELBG (Patane &
Russo, 2001)
 With fixed distortion error, to minimize the number
of codewords and find a suitable codebook.
 Meaning: To get the same reconstruction quality for
different vector set, the codebook will have different size
and thus can save plenty of storage.

6. 6/39
Test Image
 Lena (512*512*8) is
separated to 4*4 blocks. Such
blocks are the input vectors.
There are totally 16384
vectors.
 Peak Signal to Noise Ratio
(PSNR) is used to evaluate the
resulting images after the
quantization process.
2552
PSNR  10 log10
1

N
i 1
( f (i )  g (i )) 2
N Lena (512*512*8)

7. 7/39
Improvement I:
Incrementally inserting codewords
 The optimal
solution of k-
clustering
problem can
be reachable
from the (k-
1)-clustering
problem.

8. 8/39
Improvement II:
Distance measure function
 Within cluster
distance must be
significantly less
than between
cluster distance.
l
d ( x, c)  ( ( xi  ci ) 2 ) p
i 1
p  log10 q  1

9. 9/39
Improvement III:
Delete and insert codeword
Delete codeword
with lowest local
distortion error
Insert codeword
near the codeword
with highest local
distortion error

10. 10/39
Experiment 1
PSNR
Number of
codewords LBG (Linde Mk (Lee et ELBG(Pata
AILBG
et al.,1980) al., 1997) ne, 2001)
256 31.60 31.92 31.94 32.01
512 32.49 33.09 33.14 33.22
1024 33.37 34.42 34.59 34.71
Meaning: With the same number of codewords, proposed
method can get highest PSNR, i.e., with the same compression
ratio, proposed method can get best reconstruction quality.

11. 11/39
Experiment 2
Number of codewords
PSNR ELBG (Patane,
AILBG
2001)
31.94 256 244
33.14 512 488
34.59 1024 988
Meaning:
• With a predefined reconstruction quality, proposed method can
find a good codebook with reasonable number of codewords.

12. 12/39
Experiment 3: Original Images
Boat Gray21

13. 13/39
Results of experiment 3
PSNR Number of codewords
(dB) Gray21 Lena Boat
28.0 9 22 54
30.0 12 76 199
33.0 15 454 1018
Meaning:
1. For different images, with the same PSNR, number of codewords will be different.
2. Proposed method can be used to set up an image database with same
reconstruction quality (PSNR)

14. 14/39
Unsupervised learning
 Clustering
 K-means (King, 1967), ELBG (Patane, 2001), Global k-means (Likas, 2003),
AILBG (Shen, 2005)
 Determine the number of clusters k in advance
 data sets consisting only of isotropic clusters
 Single-link (Sneath, 1973), complete-link (King, 1967), CURE (Guha, 1998)
 Computation overload, much memory space
 Unsuitable for large data sets or online data
 Topology Learning: Reflects topology of high-dimension data distribution
 SOM (Kohonen, 1982): predetermined structure and size
 CHL+NG (Martinetz, 1994): a priori decision about the network size
 GNG (Fritzke, 1995): permanent increase in the number of nodes
 Online Learning
 GNG-U (Frutzke, 1998): destroy learned knowledge
 LLCS (Hamker, 2001): supervised learning

15. 15/39
Self-organizing incremental neural
network (Shen & Hasegawa, 2005)
1. To process the on-line non-stationary data.
2. To do the unsupervised learning without any priori
condition such as:
• suitable number of nodes
• a good initial codebook
• how many classes there are
3. Report a suitable number of classes
4. Represent the topological structure of the input probability
density.
5. Separate the classes with some low-density overlaps
6. Detect the main structure of clusters polluted by noises

16. 16/39
The Proposed algorithm
First Layer Second Layer
Input Growing First Growing Second
pattern Network Output Network Output
Insert Delete
Classify
Node Node

17. 17/39
Algorithms
 Insert new nodes
 Criterion: nodes with high errors serve as a criterion to
insert a new node
 error-radius is used to judge if the insert is successful
 Delete nodes
 Criterion: remove nodes in low probability density
regions
 Realize: delete nodes with no or only one direct topology
neighbor
 Classify
 Criterion: all nodes linked with edges will be one cluster

18. 18/39
First-layer Second-layer
Input signals==
Initialize multiple of 
Input signal Within-class
Insertion
Find winner Judge if insertion
and second winner is successful
Delete overlap and
Y Between-class noise nodes
Insertion
N N Input signals==
Connect winner multiple of LT
and second winner
Y
Update weight of First-layer Y
winner and neighbor
N
Output results

19. 19/39
Experiment
Environment
I II III IV V VI VII
A 1 0 1 0 0 0 0
B 0 1 0 1 0 0 0
C 0 0 1 0 0 1 0
D 0 0 0 1 1 0 0
E1 0 0 0 0 1 0 0
E2 0 0 0 0 0 1 0
Original Data Set E3 0 0 0 0 0 0 1

20. 20/39
Experiment:
Stationary environment
Original Data Set GNG (Fritzke, 1995)

21. 21/39
Experiment:
Stationary environment
Proposed method: first layer Proposed method: final results

22. 22/39
Experiment:
Non-stationary environment
GNG (Fritzke, 1995) GNG-U (Fritzke, 1998)

23. 23/39
Experiment:
Non-stationary environment
Proposed method: first layer

24. 24/39
Experiment:
Non-stationary environment
Proposed method: first layer

25. 25/39
Experiment:
Non-stationary environment
Proposed method: first layer

26. 26/39
Experiment:
Non-stationary environment
Proposed method: first layer Proposed method: Final output

27. 27/39
Application: Face recognition
(ATT_FACE)
Facial Image
(a) 10 classes
(b) 10 samples of class 1

28. 28/39
Face recognition: Feature Vector
Vector of (a)
Vector of (b)

29. 29/39
Face Recognition: results
10 clusters
Stationary
Correct
Recognition
Ratio: 90%
Non-Stationary
Correct
Recognition
Ratio: 86%

30. 30/39
Application: Vector Quantization
Stationary Environment: Decoding
Original Lena (512*512*8)
image, 130 nodes, 0.45bpp,
PSNR = 30.79dB

31. 31/39
Vector Quantization:
Compare with GNG
Stationary Environment
Number of
bpp PSNR
Nodes
First-layer 130 0.45 30.79
GNG (Fritzke,
130 0.45 29.98
1995)
Second-layer 52 0.34 29.29
GNG 52 0.34 28.61

32. 32/39
Vector Quantization:
Non-stationary Environment
First-layer: 499 nodes, 0.56bpp, Second-layer: 64 nodes, 0.375bpp,
PSNR = 32.91dB PSNR = 29.66dB

33. 33/39
Application: Handwritten
character recognition
 Optical Recognition of Handwritten Digits
database (optdigits) (UCI repository, 1996)
 10 classes (handwritten digits) from a total of 43
people
 30 contributed to the training set, 3823 samples
 Different 13 to the test set, 1797 samples
 Dimension of the samples is 64
 Method:
 Train: A separate SOINN to describe each class of data
 Test: Classify an unknown data point according to
whichever model gives the best match (nearest
neighbor)

34. 34/39
Optdigits: Comparison with 1-NN
Proposed method
1-NN
(1) (2) (3) (4)
Recognition
98% 98.5% 97.1% 96.5% 96.0%
ratio
No. of
3823 845 544 415 334
prototype
Speed up
1 4.53 7.02 9.21 11.45
(times)
Memory 100% 22.1% 14.2% 10.8% 8.7%

35. 35/39
Optdigits: Comparison with SVM
Improved SVM
Traditional SVM
(Passerini, 2002) Proposed
method
One-vs-All All-pairs One-vs-All All-pairs
Recog
nition 97.2 97.4 98.2 98.1 98.5
ratio
Gaussian Kernel

36. 36/39
Application: others
 Humanoid robot
 Scene recognition
 Texture recognition
 Semi-supervised learning

37. 37/39
Journal papers (2003~2005)
1. Shen Furao & Osamu Hasegawa, “An adaptive incremental LBG
for vector quantization,” Neural Networks, accepted.
2. Shen Furao & Osamu Hasegawa, “An incremental network for on-
line unsupervised classification and topology learning,” Neural
Networks, accepted.
3. Shen Furao & Osamu Hasegawa, Fractal image coding with
simulated annealing search, Journal of Advanced Computational
Intelligence and Intelligent Informatics, Vol.9, No.1, pp.80-88,
2005.
4. Shen Furao & Osamu Hasegawa, A fast no search fractal image
coding method, Signal Processing: Image Communication, vol.19,
pp.393-404, (2004)
5. Shen Furao & Osamu Hasegawa, A growing neural network for
online unsupervised learning, Journal of Advanced Computational
Intelligence and Intelligent Informatics, Vol.8, No.2, pp.121-129,
(2004)

38. 38/39
Refereed International
Conference (2003~2005)
1. Shen Furao, Youki Kamiya & Osamu Hasegawa, “An incremental neural network for online
supervised learning and topology representation,” 12th International Conference on Neural
Information Processing (ICONIP 2005), Taipei, Taiwan, October 30 - November 2, 2005, accepted.
2. Shen Furao & Osamu Hasegawa, “An incremental k-means clustering algorithm with adaptive
distance measure,” 12th International Conference on Neural Information Processing (ICONIP
2005), Taipei, Taiwan, October 30 - November 2, 2005, accepted.
3. Shen Furao & Osamu Hasegawa, “An on-line learning mechanism for unsupervised classification
and topology representation,” IEEE Computer Society International Conference on Computer
Vision and Pattern Recognition (CVPR 2005), San Diego, CA, USA, June 21-26, 2005.
4. Shen Furao & Osamu Hasegawa, “An incremental neural network for non-stationary unsupervised
learning,” 11th International Conference on Neural Information Processing (ICONIP 2004), Calcutta,
India, November 22-25, 2004.
5. Shen Furao & Osamu Hasegawa, “An effective fractal image coding method without search,” IEEE
International Conference on Image Processing (ICIP 2004), Singapore, October 24-27, 2004.
6. Youki Kamiya, Shen Furao & Osamu Hasegawa, “Non-stop learning : a new scheme for continuous
learning and recognition,” Joint 2nd SCIS and 5th ISIS, Keio University, Yokohama, Japan,
September 21-24, 2004.
7. Osamu Hasegawa & Shen Furao, “A self-structurizing neural network for online incremental
learning,” CD-ROM SICE Annual Conference in Sapporo, FAII-5-2, August 4-6, 2004.
8. Shen Furao & Osamu Hasegawa, “A self-organized growing network for on-line unsupervised
learning,” 2004 International Joint Conference on Neural Networks (IJCNN 2004), Budapest,
Hungary, CD-ROM ISBN 0-7803-8360-5, Vol.1, pp.11-16, 2004.
9. Shen Furao & Osamu Hasegawa, “A fast and less loss fractal image coding method using
simulated annealing,” 7th Joint Conference on Information Science (JCIS 2003), Cary, North
Carolina, USA, September 26-30, 2003.