2. A Survey of Convolutional Neural
Networks:
Analysis, Applications, and Prospects
Zewen Li, Wenjie Yang, Shouheng Peng, Fan Liu, Member, IEEE
3. Introduction to Convolution Neural Network (CNN)
•Applications using CNN –
•Face recognition
•Autonomous vehicles
•Self-service supermarket
•Intelligent medical treatment
4. Emergence of CNN
• McCulloch and Pitts – First mathematical MP model of
neurons
• Rosenblatt - Added learning capability to MP model
• Hinton – Proposed multi-layer feedforward network trained
by the error Back Propagation – BP network
• Waibel - Time Delay Neural Network (TDNN) for speech
recognition
• LeCun – First convolution network (LeNet) to recognize
handwritten text
5. Overview of CNN
• Feedforward neural network
• Extracts features from data from convolution structures
• Architecture inspired by visual perception
• Biological neuron corresponds to an artificial neuron
• CNN kernels represent different receptors that can respond to various
features
• Activation function transmit signal to next neuron if it exceeds certain
threshold
• Loss functions and optimizers teach the whole CNN system to learn
6. Advantages of CNN
• Local connections – Each neuron connected to not all but
small no. of neurons. Reduces parameters and speed up
convergence.
• Weight sharing - Connections share same weights
• Down-sampling dimensionality reduction.
• These characteristics make CNN most representative
algorithms
7. Components of CNN
• Convolution - pivotal step for feature extraction. Output is feature
map
• Padding - introduced to enlarge the input with zero value
• Stride – Control the density of convolution
• Pooling - Obviate redundancy or down sampling
8. LeNet - 5
• Composed of 7 trainable layers containing 2 convolutional layers, 2
pooling layers, and 3 fully-connected layers
• NN characteristics of local receptive fields, shared weights, and spatial
or temporal subsampling, ensures shift, scale, and distortion
• Used for handwriting recognition
9. AlexNet
• Has 8 layers, containing 5 convolutional layers and 3 fully-connected
layers
• uses ReLU as the activation function of CNN to solve gradient
vanishing
• Dropout was used in last few layers to avoid overfitting
• Local Response Normalization (LRN) to enhance generalization of
model
10. AlexNet
• Employ 2 powerful GPUs, two feature maps generated by two GPUs
can be combined as the final output
• Enlarges dataset and calculates average of their predictions as final
result
• Principal Component Analysis (PCA) to change the RGB values of
training set
11. VGGNet
• LRN layer was removed
• VGGNets use 3 × 3 convolution kernels rather than 5 × 5 or 5 × 5
ones, since several small kernels have the same receptive field and
more nonlinear variations compared with larger ones.
12. GoogLeNet - Inception v1
• CNN formed by stacking with Inception modules
• Inception v1 deploys 1 × 1, 3 × 3, 5 × 5 convolution kernels to
construct a “wide” network
• Convolution kernels with different sizes can extract the feature maps
of different scales of the image
• 1 × 1 convolution kernel is used to reduce the number of channels,
i.e., reduce computational cost
13. GoogLeNet - Inception v2
• Output of every layer is normalized to increase the robustness of
model and train it with high learning rate
• Single 5 × 5 convolutional layers can be replaced by two 3 × 3 ones
• One n x n convolutional layer can be replaced byone 1 x n and one n x
1 convolutional layer
• Filter banks expanded wider to improve high dimensional
representations
14. ResNet
• Two layer residual block constructed by the shortcut connection
• 50-layer ResNet, 101-layer ResNet, and 152-layer ResNet utilize three-
layer residual blocks
• Three-layer residual block is also called the bottleneck module
because the two ends of the block are narrower than the middle
• Can mitigate the gradient vanishing problem since the gradient can
directly flow through shortcut connections
•
15. DCGAN
• GAN has generative model G and a discriminative model D
• The model G with random noise z generates a sample G(z) that
subjects to the data distribution data learned by G.
• The model D can determine whether the input sample is real data x
or generated data G(z).
• Both G and D can be nonlinear functions. The aim of G is to generate
real data, the aim of D is to distinguish fake data generated by G from
the real data
16. MobileNets
• lightweight models proposed by Google for embedded devices such
as mobile phones
• depth-wise separable convolutions and several advanced techniques
to build thin deep neural networks.
17. ShuffleNets
• Series of CNN-based models to solve the problem of insufficient
computing power of mobile devices
• Combine pointwise group convolution, channel shuffle, which
significantly reduce the computational cost with little loss of accuracy
18. GhostNet
• As large amounts of redundant features are extracted by existing
CNNs for image cognition, GhostNet is used to reduce computational
cost effectively
• Similar feature maps in traditional convolution layers are called ghost
• Traditional convolution layers divided into two parts
• Less convolution kernels are directly used in feature extraction
• These features are processed in linear transformation to acquire
multiple feature maps. They proved that Ghost module applies to
other CNN models
19. Activation function
• In a multilayer neural network, there is a function between two layers
which is called activation function
• Determines which information should be transmitted to the next
neuron
• If no activation function, input layer will be linear function of the
output
• Nonlinear functions are introduced as activation functions to enhance
ability of neural network
20. Types of activation function
• Sigmoid function can map a real number to (0, 1), so it can be used
for binary classification problems.
• Tanh function maps a real number to (-1, 1), achieves normalization.
This makes the next layer easier to learn.
• Rectified Linear Unit (ReLU), when x is less than 0, its value is 0; when
x is greater than or equal to 0, its value is x itself. Speeds up learning.
• ELU function has a negative value, so the average value of its output is
close to 0, making the rate of convergence faster than ReLU.
21. Loss/Cost function
• Calculates the distance between the predicted value and the actual
value
• Used as a learning criterion of the optimization problem
• Common loss functions Mean Absolute Error (MAE), Mean Square
Error (MSE), Cross Entropy
22. Rules of Thumb for Loss Function Selection
• CNN models for regression problems, choose L1 loss or L2 loss as the
loss function.
• For classification problems, select the rest of the loss functions
• Cross entropy loss is the most popular choice, with a softmax layer in
the end.
• The selection of loss function in CNNs also depends on the
application scenario. For example, when it comes to face recognition,
contrastive loss and triplet loss are turned out to be the commonly-
used ones nowadays.
23. Optimizer
• In convolutional neural networks, need to optimize non-convex
functions.
• Mathematical methods require huge computing power, so optimizers
are used in the training process to minimize the loss function for
getting optimal network parameters within acceptable time.
• Common optimization algorithms are Momentum, RMSprop, Adam,
etc.
24. Applications of one-dimensional CNN
• Time Series Prediction
• Electrocardiogram (ECG) time series, weather forecast, and traffic flow
prediction, highway traffic flow prediction
• Signal Identification
• ECG signal identification, structural damage identification, and system fault
identification
25. Applications of two-dimensional CNN
• Image Classification
• medical image classification, traffic scenes related classification, classify
breast cancer tissues
• Object Detection
• Image Segmentation
• Face Recognition
27. Conclusion
• Due to the advantages of convolutional neural networks, such as local
connection, weight sharing, and down-sampling dimensionality reduction,
they have been widely deployed in both research and industry projects
• First, we discussed basic building blocks of CNN and how to construct a
CNN-based model from scratch
• Secondly, some excellent CNN networks
• Third, we introduce activation functions, loss functions, and optimizers for
CNN
• Fourth, we discuss some typical applications of CNN
• CNN can be refined further in terms of model size, security, and easy
hyperparameters selection. Moreover, there are lots of problems that
convolution is hard to handle, such as low generalization ability, lack of
equivariance, and poor crowded-scene results, so that several promising
directions are pointed.
