Face annotation for personal photos using collaborative face recognition in online social networks.

1. Face Annotation for Personal Photos
Using Collaborative Face Recognition
in Online Social Networks
16th International Conference on Digital Signal Processing
July 2009, Santorini, Greece
Jae Young Choi, Wesley De Neve,
Yong Man Ro, Konstantinos N. Plataniotis
jygchoi@kaist.ac.kr
Image and Video Systems Lab
Department of Electrical Engineering
Korea Advanced Institute of Science and Technology
Daejeon, Korea

2. -2/19-
Outline of Presentation
 Research motivation
 Research background
 Collaborative face recognition
 Evaluation
 Conclusions

3. -3/19-
Research Motivation (1/2)
Increasing consumption of personal photos on
online social networks
Management
Storing Sharing
Photo
acquisition
Photo collection
Digital camera /camcorder Social network site
Mobile phone
MP3 / PMP
…

4. -4/19-
Research Motivation (2/2)
Most users want
to manage photo
collections based
on people
Automatic face tagging
becomes key marketing
techniques
 Google’s Picasa Web Albums
 Face recognition in Riya
 Face search in Apple’s iPhoto

5. -5/19-
Research Background
Basic idea
 Social context: Strong tendency that people capture photos with
friends, family members, and co-workers
 Customization of FR engine: A personalized FR engine is the most
suitable for indexing face images of the corresponding user
 We expect that FR engines will be built using a high number of training face
images of their owner and the (close) contacts of the owner
 Decentralization of FR engines: An online social network facilitates
collaboration among multiple personalized FR engines
Face images of
the owner of the
FR engine and
his/her close
contacts
Illustration of social context Personalized FR engine Decentralization of
FR engines

6. -6/19-
Research Question
How to make use of personalized and
distributed FR engines to improve the
annotation accuracy?
Each FR engine is
specialized for a particular
community member and
his/her close contacts
FR engines are distributed
over an online social network
and can be accessed by
community members

7. -7/19-
Collaborative Face Recognition (1/6)
Goal
 To make use of distributed FR engines in a
collaborative way
Two important research topics
 How to select multiple FR engines from an online
social network?
 Use of online social context
 Use of social context available in personal photo collections
 How to make use of the collected FR engines for FR
purposes?
 Fusion of multiple FR results obtained from the selected FR
engines

8. -8/19-
Collaborative Face Recognition (2/6)
Proposed system for collaborative face recognition
and face annotation
The i th community member’s face annotation system
Photo collection Pli to be annotated
Distributed FR databases
Social relationship
model
Social context
data FR database Ωl2
FR database Ω l1
S li
FR database
Face detection
selection
(k )
Query face images Fquery
FR database Ωl
j
FR database Ω l5
Feature Feature Feature
FR database Ω l
extractor 1 extractor 2 extractor j 3
Collected FR databases  col
Classifier 1 Classifier 2 Classifier j
FR database Ω lM
FR database Ω l
4
Multiple evidences fusion based collaborative FR
Face annotation result

9. -9/19-
Collaborative Face Recognition (3/6)
How to select multiple FR engines
 Online social context can be drawn from the contact list
of a community member
Roguer’s FR
engine
Venning’s FR
Name of friends G  N, E, W engine
or families
n4
Construction of social
w1, 4 Collaborative
n3
graph related to i-th face recognition
n1 w1,3
community member
Woodham’s FR
n2
w1, 2 engine
w1,5
Bolland’s FR
w1,6 n5
engine
n6
n7
Cho’s FR engine
Profile of certain community member in
“Facebook”

10. -
10/19-
Collaborative Face Recognition (4/6)
How to make use of the selected FR engines?
Selected FR databases for collaborative FR
FR database 1 FR database 2 FR database 3 FR database M
Ω3  {3 , u3 , G 3} Ω M  { M , uM , G M }
Ω1  {1, u1, G1} Ω 2  { 2 , u 2 , G 2 }
Φ col  {1 ,  2 , ,  M }
U col  {u1 , u2 ,, u M }
G col  {G1 , G 2 ,, G M }
i : Feature extractor of i-th FR
engine Selected feature extractors,
ui : Classifier of i-th FR engine classifiers, and gallery sets are
Ω col  {Φ col , U col ,, G col } used for the purpose of
Gi : collaborative FR
Gallery set of i-th FR engine

11. -11/19-
Collaborative Face Recognition (5/6)
Collaborative FR using measurement-level fusion Collected
gallery set (Ω col )
Query face
Selected
feature
extractors
Feature extractor 1 Feature extractor 2 Feature extractor M Using Φ col
Nearest neighbor Nearest neighbor Nearest neighbor Using U col
Classifier u1 Classifier u 2 Classifier u M
Normalization
Normalization
and confidence
Normalization
and confidence and confidence Selected
transformation
transformation transformation
classifiers
Posterior probability Posterior probability Posterior probability
estimation estimation estimation
Probability summation
Annotation result

12. -12/19-
Collaborative Face Recognition (6/6)
Collaborative FR using confidence-based majority
voting Collected
gallery set (Ω col )
Query face
Selected
feature
extractors
Feature extractor 1 Feature extractor 2 Feature extractor M Using Φ col
Nearest neighbor Nearest neighbor Nearest neighbor Using U col
Classifier u1 Classifier u 2 Classifier u M
Selected
N vote ( n) Compute confidence Cconf ( n)
Compute the number of votes
classifiers
Combination of the number of votes
and corresponding confidence value
Annotation result

13. -13/19-
Experimental Settings (1/3)
Photo databases
 1,120 photos provided by MPEG-7 VCE3
 4,120 realistic web photos collected from ‘Flickr’ and ‘Myspace’
 Ground-truth data sets
 Subjects are used that appear at least 15 times in the photo collections
 MPEG-7 VCE3: 1,345 face images of 54 subjects
 Web photo dataset: 8,610 face images of 420 subjects
Uncontrolled image
acquisition conditions:
illumination, pose, heavy
make-up, and even
occlusion

14. -14/19-
Experimental Settings (2/3)
Annotation performance metrics
 Person identification (or classification)
 Assigns a query face to the correct person
 Person-based photo retrieval
 When a user enters a name, photos are retrieved containing the
person with the given name
Query photos
Personal user
Enter name JaeYoung
Person classes Retrieve photos in which
Bang-Sil Yong Man
‘JaeYoung’ appears
JaeYoung Yune Choi
Photo annotation system

15. -15/19-
Experimental Settings (3/3)
 Experimental protocol
Collaborative FR Centralized FR
Total query set Total query set
Q1 Q2 QN Q1 Q2 QN
A single large collection
T1 ,, TN 
T1 T2 TN
Independent FR
Total query set Total query set Total query set
Q1 Q2 QN Q1 Q2 QN Q1 Q2 QN
T1 T2 TN

16. -16/19-
Annotation Performance (1/3)
For MPEG-7 VCE3 Each FR engine is trained with 180
images of 18 subjects (10
samples/subject)
Each FR engine
is trained with 90
images of 9
(a) When using 3 collected FR engines subjects (10
samples/subject)
Collaborative FR
significantly
outperforms
independent FR,
regardless of the
number of selected
(b) When using 6 collected FR engines FR engines

17. -17/19-
Annotation Performance (2/3)
Web photo collection
 8,610 face images of 420 subjects obtained from 4,120 web
photos
 Using random partitioning, 4,200 images of 420 subjects
are used for training the FR engines, and the remaining
images are used to create a query set
 The number of selected FR engines for collaborative FR
No. of selected FR engines 5 12 20
No. of subjects in each FR engine 84 35 21
No. of samples/subject 10 10 10
To verify the robustness of the proposed
collaborative FR against changes in the number
of selected FR engines

18. -18/19-
Annotation Performance (3/3)
 Web photo collection
0.75
0.8
0.75 0.7
0.7
Collaborative FR 1
Precision
0.65
Recall
Collaborative FR 2
0.65 Independent FR
Centralized FR
0.6
0.6
Collaborative FR 1
Collaborative FR 2
0.55 Independent FR
0.55
Centralized FR
5 12 20 5 12 20
No. of collected feature extractors used for collaboration No. of collected feature extractors used for collaborative FR
(a) Bayesian
0.8
0.7
0.65
0.7
Collaborative FR 1 0.6
Collaborative FR 2
0.6 0.55
Independent FR Collaborative FR 1
Precision
Centralized FR 0.5 Collaborative FR 2
Recall
0.5 0.45 Independent FR
Centralized FR
0.4
0.4
0.35
0.3
0.3
0.25
0.2 0.2
5 12 20 1 12 20
No. of collected classifiers used for collaborative FR No. of collected classifiers used for collaborative FR
(b) FLDA

19. -19/19-
Conclusions
The accuracy of collaborative FR is considerably
better than the accuracy of independent FR
 Performance improvement is generally independent of the
number of selected FR engines
 The effect of collaboration becomes more significant when
the classification capability of an individual FR engine is
degraded
The accuracy of collaborative FR is comparable to
the accuracy of centralized FR (using a single and
larger set of training face images)