1. Neural Networks NN 1 1
Neural Networks
Teacher:
Elena
Marchiori
R4.47
elena@cs.vu.nl
Assistant:
Marius Codrea
S4.16
mcodrea@few.vu.nl

2. Neural Networks NN 1 2
Course Outline
The course is divided in two parts: theory and
practice.
1. Theory covers basic topics in neural
networks theory and application to
supervised and unsupervised learning.
2. Practice deals with basics of Matlab and
application of NN learning algorithms.

3. Neural Networks NN 1 3
Course Information
• Register for practicum: send email to
mcodrea@few.vu.nl with:
1. Subject: NN practicum
2. Content: names, study numbers, study directions
(AI,BWI,I, other)
• Course information, plan, slides and links to
on-line material are available at
http://www.cs.vu.nl/~elena/nn.html

4. Neural Networks NN 1 4
Course Evaluation
• Course value: 6 ects
• Evaluation is based on the following two
parts:
– theory (weight 0.5): final exam at the end of the
course consisting of questions about theory part.
(Dates to be announced)
– practicum(weight 0.5): Matlab programming
assignments to be done in couples (Available
during the course at
http://www.few.vu.nl/~codrea/nn)

5. Neural Networks NN 1 5
What are Neural Networks?
• Simple computational elements forming a
large network
– Emphasis on learning (pattern recognition)
– Local computation (neurons)
• Definition of NNs is vague
– Often | but not always | inspired by biological brain

6. Neural Networks NN 1 6
History
• Roots of work on NN are in:
• Neurobiological studies (more than one century ago):
• How do nerves behave when stimulated by different magnitudes
of electric current? Is there a minimal threshold needed for
nerves to be activated? Given that no single nerve cel is long
enough, how do different nerve cells communicate among each
other?
• Psychological studies:
• How do animals learn, forget, recognize and perform other types
of tasks?
• Psycho-physical experiments helped to understand how individual
neurons and groups of neurons work.
• McCulloch and Pitts introduced the first mathematical model of
single neuron, widely applied in subsequent work.

7. Neural Networks NN 1 7
History
Prehistory:
• Golgi and Ramon y Cajal study the nervous system and discover
neurons (end of 19th century)
History (brief):
• McCulloch and Pitts (1943): the first artificial neural network with
binary neurons
• Hebb (1949): learning = neurons that are together wire together
• Minsky (1954): neural networks for reinforcement learning
• Taylor (1956): associative memory
• Rosenblatt (1958): perceptron, a single neuron for supervised
learning

8. Neural Networks NN 1 8
History
• Widrow and Hoff (1960): Adaline
• Minsky and Papert (1969): limitations of single-layer perceptrons (and
they erroneously claimed that the limitations hold for multi-layer
perceptrons)
Stagnation in the 70's:
• Individual researchers continue laying foundations
• von der Marlsburg (1973): competitive learning and self-organization
Big neural-nets boom in the 80's
• Grossberg: adaptive resonance theory (ART)
• Hopfield: Hopfield network
• Kohonen: self-organising map (SOM)

9. Neural Networks NN 1 9
History
• Oja: neural principal component analysis (PCA)
• Ackley, Hinton and Sejnowski: Boltzmann machine
• Rumelhart, Hinton and Williams: backpropagation
Diversification during the 90's:
• Machine learning: mathematical rigor, Bayesian methods,
infomation theory, support vector machines (now state of the
art!), ...
• Computational neurosciences: workings of most subsystems of the
brain are understood at some level; research ranges from low-level
compartmental models of individual neurons to large-scale brain
models

10. Neural Networks NN 1 10
Course Topics
Learning Tasks
Supervised Unsupervised
Data:
Labeled examples
(input , desired output)
Tasks:
classification
pattern recognition
regression
NN models:
perceptron
adaline
feed-forward NN
radial basis function
support vector machines
Data:
Unlabeled examples
(different realizations of the
input)
Tasks:
clustering
content addressable memory
NN models:
self-organizing maps (SOM)
Hopfield networks

11. Neural Networks NN 1 11
NNs: goal and design
– Knowledge about the learning task is given in the
form of a set of examples (dataset) called training
examples.
– A NN is specified by:
• an architecture: a set of neurons and links connecting
neurons. Each link has a weight,
• a neuron model: the information processing unit of the
NN,
• a learning algorithm: used for training the NN by
modifying the weights in order to solve the particular
learning task correctly on the training examples.
The aim is to obtain a NN that generalizes well, that
is, that behaves correctly on new examples of the
learning task.

12. Neural Networks NN 1 12
Example: Alvinn
Autonomous
driving at 70 mph
on a public
highway
Camera
image
30x32 pixels
as inputs
30 outputs
for steering 30x32 weights
into one out of
four hidden
unit
4 hidden
units

13. Neural Networks NN 1 13
Face Recognition
90% accurate learning head pose, and recognizing 1-of-20 faces

14. Neural Networks NN 1 14
Handwritten digit recognition

15. Neural Networks NN 1 15
Dimensions of a Neural
Network
• network architectures
• types of neurons
• learning algorithms
• applications

16. Neural Networks NN 1 16
Network architectures
• Three different classes of network architectures
– single-layer feed-forward neurons are organized
– multi-layer feed-forward in acyclic layers
– recurrent
• The architecture of a neural network is linked with the
learning algorithm used to train

17. Neural Networks NN 1 17
Single Layer Feed-forward
Input layer
of
source nodes
Output layer
of
neurons

18. Neural Networks NN 1 18
Multi layer feed-forward
Input
layer
Output
layer
Hidden Layer
3-4-2 Network

19. Neural Networks NN 1 19
Recurrent Network with hidden neuron: unit delay
operator z-1
is used to model a dynamic system
z-1
z-1
z-1
Recurrent network
input
hidden
output

20. Neural Networks NN 1 20
The Neuron
Input
values
weights
Summing
function
Bias
b
Activation
functionLocal
Field
v
Output
y
x1
x2
xm
w2
wm
w1
∑ )(−ϕ
………….

21. Neural Networks NN 1 21
Input Signal and Weights
Input signals
An input may be either a
raw / preprocessed signal or
image. Alternatively, some
specific features can also be
used.
If specific features are used
as input, their number and
selection is crucial and
application dependent
Weights
Weights are connected
between an input and a
summing node. These affect to
the summing operation.
The quality of network can be
seen from weights
Bias is a constant input with
certain weight.
Usually the weights are
randomized in the beginning

22. Neural Networks NN 1 22
The Neuron
• The neuron is the basic information processing unit of
a NN. It consists of:
1 A set of links, describing the neuron inputs, with
weights W1, W2, …, Wm
2 An adder function (linear combiner) for computing
the weighted sum of
the inputs (real numbers):
3 Activation function (squashing function) for
limiting the amplitude of the neuron output.
∑=
=
m
1
jjxwu
j
ϕ
)(uy b+= ϕ

23. Neural Networks NN 1 23
Bias of a Neuron
• The bias b has the effect of applying an affine
transformation to the weighted sum u
v = u + b
• v is called induced field of the neuron
x2x1u −=x1-x2=0
x1-x2= 1
x1
x2
x1-x2= -1

24. Neural Networks NN 1 24
Bias as extra input
Input
signal
Synaptic
weights
Summing
function
Activation
functionLocal
Field
v
Output
y
x1
x2
xm
w2
wm
w1
∑ )(−ϕ
w0
x0 = +1
• The bias is an external parameter of the neuron. It can be
modeled by adding an extra input.
bw
xwv j
m
j
j
=
= ∑=
0
0
…………..

25. Neural Networks NN 1 25
Activation Function
There are different activation functions used in different applications. The
most common ones are:
Hard-limiter Piecewise linear Sigmoid Hyperbolic tangent
( )



<
≥
=
00
01
vif
vif
vϕ ( )





−≤
−≥≥
≥
=
210
2121
211
vif
vifv
vif
vϕ ( )
)exp(1
1
av
v
−+
=ϕ
( ) ( )vv tanh=ϕ
ϕ

26. Neural Networks NN 1 26
Neuron Models
• The choice of determines the neuron model. Examples:
• step function:
• ramp function:
• sigmoid function:
with z,x,y parameters
• Gaussian function:
ϕ













 −
−=
2
2
1
exp
2
1
)(
σ
µ
σπ
ϕ
v
v
)exp(1
1
)(
yxv
zv
+−+
+=ϕ





−−−+
>
<
=
otherwise))/())(((
if
if
)(
cdabcva
dvb
cva
vϕ



>
<
=
cvb
cva
v
if
if
)(ϕ

27. Neural Networks NN 1 27
Learning Algorithms
Depend on the network architecture:
• Error correcting learning (perceptron)
• Delta rule (AdaLine, Backprop)
• Competitive Learning (Self Organizing Maps)

28. Neural Networks NN 1 28
Applications
• Classification:
– Image recognition
– Speech recognition
– Diagnostic
– Fraud detection
– …
• Regression:
– Forecasting (prediction on base of past history)
– …
• Pattern association:
– Retrieve an image from corrupted one
– …
• Clustering:
– clients profiles
– disease subtypes
– …

29. Neural Networks NN 1 29
Supervised learning
Non-linear classifiers
Linear classifiers
Perceptron Adaline
Feed-forward networksRadial basis function networks
Support vector machines
Unsupervised learning
Clustering
Content addressable memories
Optimization Hopfield networks
Self-organizing maps K-means

30. Neural Networks NN 1 30
Vectors: basics
(slides by Hyoungjune Yi: aster@cs.umd.edu)
• Ordered set of
numbers: (1,2,3,4)
• Example: (x,y,z)
coordinates of pt in
space. runit vectoais,1If
1
2
),,
2
,
1
(
vv
n
i
i
xv
n
xxxv
=
=
=
=
∑


31. Neural Networks NN 1 31
Vector Addition
),(),(),( 22112121 yxyxyyxx ++=+=+ wv
vv
ww
V+wV+w

32. Neural Networks NN 1 32
Scalar Product
),(),( 2121 axaxxxaa ==v
vv
avav

33. Neural Networks NN 1 33
Operations on vectors
• sum
• max, min, mean, sort, …
• Pointwise: .^

34. Neural Networks NN 1 34
Inner (dot) Product
vv
ww
αα
22112121 .),).(,(. yxyxyyxxwv +==
The inner product is aThe inner product is a SCALAR!SCALAR!
αcos||||||||),).(,(. 2121 wvyyxxwv ⋅==
wvwv ⊥⇔= 0.

35. Neural Networks NN 1 35
Matrices
















=×
nmnn
m
m
m
mn
aaa
aaa
aaa
aaa
A





21
33231
22221
11211
mnmnmn BAC ××× +=
Sum:Sum:
ijijij bac +=
A and B must have the sameA and B must have the same
dimensionsdimensions

36. Neural Networks NN 1 36
Matrices
pmmnpn BAC ××× =
Product:Product:
∑=
=
m
k
kjikij bac
1
A and B must haveA and B must have
compatible dimensionscompatible dimensions
nnnnnnnn ABBA ×××× ≠
Identity Matrix:
AAIIAI ==












=
100
010
001





37. Neural Networks NN 1 37
Matrices
mn
T
nm AC ×× =
Transpose:Transpose:
jiij ac = TTT
ABAB =)(
TTT
BABA +=+ )(
IfIf AAT
= A is symmetricA is symmetric

38. Neural Networks NN 1 38
Matrices
IAAAA nnnnnnnn == ××
−
×
−
×
11
Inverse:Inverse: A must be squareA must be square






−
−
−
=





−
1121
1222
12212211
1
2221
1211 1
aa
aa
aaaaaa
aa

39. Neural Networks NN 1 39
2D Translation
tt
PP
P’P’

40. Neural Networks NN 1 40
2D Translation Equation
PP
xx
yy
ttxx
ttyy
P’P’
tt
tPP +=++= ),(' yx tytx
),(
),(
yx tt
yx
=
=
t
P

41. Neural Networks NN 1 41
2D Translation using Matrices
PP
xx
yy
ttxx
ttyy
P’P’
tt
),(
),(
yx tt
yx
=
=
t
P










⋅





=





+
+
→
1
1
0
0
1
' y
x
t
t
ty
tx
y
x
y
x
P
tt PP

42. Neural Networks NN 1 42
Scaling
PP
P’P’

43. Neural Networks NN 1 43
Scaling Equation
PP
xx
yy
s.xs.x
P’P’
s.ys.y
),('
),(
sysx
yx
=
=
P
P
PP ⋅= s'






⋅





=





→
y
x
s
s
sy
sx
0
0
'P
SPSP ⋅='