1. Gender and Ethnic Classification From Frontal Face
Images
An Investigation Into Methods of Feature Extraction and Classification for Estimating
Gender and Ethnicity From Face Images
Presented by:
David Beretta
Prepared for:
Dr. F. Nicolls
Department of Electrical Engineering
University of Cape Town
Submitted to the Department of Electrical Engineering at the University of Cape Town in
partial fulfilment of the academic requirements for a Bachelor of Science degree in Electrical
Engineering.
October 16, 2012
Keywords: Machine Learning, Facial Recognition, Computer Vision, Eigenface, Fisherface,
PCA, LDA, QDA

3. Abstract
This work involves investigating various methods of feature extraction and classification tech-
niques for the particular problem of determining a person’s gender and ethnicity from an image
of their face. Different methods of feature extraction and classification are tested and compared
in constrained and unconstrained environments to determine the best algorithms for gender and
ethnic classification. Ethnic classification is simplified to a two class problem that only involves
differentiating between African and Caucasian faces. Various parameters such as image reso-
lution, number of training images and number of features are varied to gauge their effect on
performance. The most effective classifier for gender is found to be a Quadratic Discriminant
classifier using PCA with 100 eigenfaces, achieving an accuracy of 97.6 ± 1.6%. The best clas-
sifier for estimating 2-class ethnicity is found to be a Quadratic Discriminant classifier using
50 eigenfaces and achieving an accuracy of 98.2 ± 1.8%. A real-time face classification system
is also implemented using the Viola-Jones face detector, a Quadratic Discriminant classifier
with PCA and a webcam. It is found that while illumination variations impede the success of
real-time face classification, gender and 2-class ethnic classification can be achieved with high
success in conditions of controlled illumination.
This work presents results using the MUCT face database which contains an ethnically diverse
set of faces and has not been used in other literature on gender and ethnic classification.
iv

4. Contents
1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Plan of Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Literature Review 4
2.1 Facial Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.1 Facial Recognition in Humans . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.2 Facial Recognition in Machines . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.3 Face Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.4 Feature Extraction for Facial Recognition . . . . . . . . . . . . . . . . . . 7
2.1.5 Classifiers for Facial Recognition . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.6 Limitations in Machine Facial Recognition . . . . . . . . . . . . . . . . . . 10
2.2 Methods of Gender and Ethnic Classification . . . . . . . . . . . . . . . . . . . . 10
2.2.1 Appearance-Based Methods using Holistic Features . . . . . . . . . . . . . 10
v

5. 2.2.2 Geometric and Local Feature Methods . . . . . . . . . . . . . . . . . . . . 12
3 Face Detection 13
3.1 Introduction to the Viola-Jones Face Detector . . . . . . . . . . . . . . . . . . . . 13
3.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Learning Best Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 Cascaded Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5 Final Detector Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4 Feature Extraction 18
4.1 Image Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2 Feature Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.3 Principal Component Analysis (PCA) . . . . . . . . . . . . . . . . . . . . . . . . 20
4.3.1 Finding Principal Components . . . . . . . . . . . . . . . . . . . . . . . . 20
4.3.2 Eigenfaces as Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.4 Fisher’s Linear Discriminant (FLD) . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.4.1 Finding Features with FLD . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4.2 Problems with FLD in Facial Recognition . . . . . . . . . . . . . . . . . . 27
4.4.3 Fisherfaces as Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5 Face Classification 29
5.1 Bayes Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.2 Quadratic Discriminant Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
vi

6. 5.3 Mahalanobis Distance/Linear Discriminant Classifier . . . . . . . . . . . . . . . . 33
5.4 Euclidean Distance or Nearest Mean Classifier . . . . . . . . . . . . . . . . . . . . 34
5.5 Na¨ıve Bayes Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.6 k-Nearest Neighbour Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.7 Support Vector Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.7.1 Linear SVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.7.2 Kernel SVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6 Implementation and Experimental Philosophy 39
6.1 Testing Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.1.1 Hardware and Operating System . . . . . . . . . . . . . . . . . . . . . . . 39
6.1.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.2 Software Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.2.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.2.2 Classifier Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.3 Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.3.1 Classification Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.3.2 Processing Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.4 Cross-validation of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.4.1 k-Fold Cross-validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.5 Gender and Ethnic Categorisation . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.5.1 Classing Gender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
vii

7. 6.5.2 Classing Ethnicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.6 Experimental Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.6.1 Training and Testing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.6.2 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.6.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6.6.4 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
7 Results 48
7.1 Extracting Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7.1.1 Using Eigenfaces as Features . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.1.2 Using Fisherfaces as Features . . . . . . . . . . . . . . . . . . . . . . . . . 55
7.2 Classification Under Controlled Conditions . . . . . . . . . . . . . . . . . . . . . 58
7.2.1 Comparing Feature Extraction Methods and Classifiers . . . . . . . . . . 58
7.2.2 Effect of Varying Parameters . . . . . . . . . . . . . . . . . . . . . . . . . 61
7.3 Classification Under Uncontrolled Conditions . . . . . . . . . . . . . . . . . . . . 71
7.3.1 Static Image Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
7.3.2 Real-time Video Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
8 Conclusions 77
8.1 Conclusions from Testing in Controlled Conditions . . . . . . . . . . . . . . . . . 77
8.1.1 Eigenfaces and Fisherfaces Useful Features for Gender and Ethnic Clas-
sification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
8.1.2 Quadratic Classifier Superior for Gender and Ethnic Classification . . . . 78
viii

8. 8.1.3 The Effects of Varying Different Parameters . . . . . . . . . . . . . . . . . 78
8.2 Conclusions from Testing in Uncontrolled Conditions . . . . . . . . . . . . . . . . 79
8.2.1 Extreme Lighting Variations and Non-constant Backgrounds Greatly Af-
fect Classification Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8.3 Conclusion on Classifying Gender . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8.4 Conclusion on Classifying Ethnicity . . . . . . . . . . . . . . . . . . . . . . . . . . 80
9 Recommendations and Future Research 81
9.1 Recommendations to Improve Feature Extraction . . . . . . . . . . . . . . . . . . 81
9.1.1 Use Feature Subset Selection with PCA . . . . . . . . . . . . . . . . . . . 81
9.1.2 Use Null-space Methods for Computing Better MDF’s with FLD . . . . . 82
9.1.3 Investigate Other Feature Extraction Methods . . . . . . . . . . . . . . . 82
9.2 Recommendations to Improve Classification . . . . . . . . . . . . . . . . . . . . . 82
9.2.1 Use a Combination of Different Training Databases . . . . . . . . . . . . . 82
9.2.2 Discard Bad Training Images . . . . . . . . . . . . . . . . . . . . . . . . . 82
9.2.3 Investigate SVM with Different Kernels . . . . . . . . . . . . . . . . . . . 83
9.3 Suggestions for Mitigating Effects of Varying Illumination . . . . . . . . . . . . . 83
9.3.1 Train Classifiers for Illumination Invariance . . . . . . . . . . . . . . . . . 83
9.3.2 Use Illumination Compensation Methods . . . . . . . . . . . . . . . . . . 83
9.4 Suggestions for Mitigating Effects of Inconsistent Backgrounds . . . . . . . . . . 83
Bibliography 86
A Extra resources 87
ix