Overview presentation of Verifying Multimedia Use task by Christina Boididou and Stuart Middleton at MediaEval workshop, Hilversum, Amsterdam.

1. Verifying Multimedia Use at
MediaEval 2016
Christina Boididou1, Stuart E. Middleton5, Symeon
Papadopoulos1, Duc-Tien Dang-Nguyen2,3, Giulia
Boato2, Michael Riegler4 & Yiannis Kompatsiaris1
1 Information Technologies Institute (ITI), CERTH, Greece
2 University of Trento, Italy.
3 Insight Centre for Data Analytics at Dublin City University, Ireland.
4 Simula Research Lab, Norway.
5 University of Southampton IT Innovation Centre, UK.

2. REAL OR FAKE
The verification problem
1

3. Real photo
Captured in Dublin’s
Olympia Theatre
A photo of Eagles of Death Metal in concert
But
Mislabeled on social
media as showing
the crowd at the
Bataclan theatre
just before gunmen
began firing.

4. A TYPOLOGY OF FAKE: REPOSTING OF REAL
Photos from past events reposted as being
associated to current event
‘Eiffel Tower lights up in
solidarity with Pakistan’
‘Syrian refugee girl
selling gum in Jordan’

5. A TYPOLOGY OF FAKE: PHOTOSHOPPING
Digitally manipulated photos / Tampered
‘Sharks in New York
during Hurricane Sandy’
‘Sikh man is a suspect of
Paris attacks’

6. TASK DEFINITION2

7. MAIN TASK
POST
IMAGE
MEDIAEVAL
SYSTEM
FAKE
REAL
AUTHOR
(PROFILE)
‘Given a post (image + metadata),
return a decision (fake, real, unknown)
on whether the information presented
by the post reflects the reality’

8. SUB-TASK
Given an image, return a decision
(tampered, non-tampered, unknown)
on whether the image has been
digitally modified or not.
IMAGE
MEDIAEVAL
SYSTEM
TAMPERED
NON
TAMPERED

9. VERIFICATION CORPUS3

10. GROUND TRUTH GENERATION
Multimedia cases were labeled as fake/real after
consulting online reports (articles, blogs)
Data (post) collection associated to these cases
performed using Topsy (historic events) or using
streaming and search API (real-time events)
Post set expansion: Near-duplicate image search +
journalist debunking reports + human inspection was
used to increase the number of associated posts
Crowdsourcing campaign carried out with
microWorkers platform; each worker asked to provide
three cases of multimedia misuse

11. DEVELOPMENT SET
Event
Fake Real
Multimedia Posts Multimedia Posts
Hurricane Sandy 62 5,559 148 4,664
Boston Marathon bombing 35 189 28 344
Sochi Olympics 26 274 - -
MH370 Flight 29 501 - -
Bring Back Our Girls 7 131 - -
Columbian Chemicals 15 185 - -
Passport hoax 2 44 - -
Rock Elephant 1 13 - -
Underwater bedroom 3 113 - -
Livr mobile app 4 9 - -
Pig fish 1 14 - -
Solar Eclipse 6 137 4 140
Samurai with girl 4 218 - -
Nepal Earthquake 21 356 11 1004
Garissa Attack 2 6 2 73
Syrian boy 1 1786 - -
Varoufakis 1 61 - -
Total 220 9596 193 6225

12. TEST SET
Event
Fake Real
MM Posts MM Posts
American Soldier
Quran
1 17 - -
Airstrikes 1 24 - -
Attacks in Paris 3 44 22 536
Ankara Explosions - - 3 19
Bush book 1 27 - -
Black Lion 1 7 - -
Boko Haram 1 31 - -
Bowie David 2 24 4 48
Brussels Car Metro 3 41 - -
Brussels Explosions 3 69 1 9
Burst in KFC 1 25 - -
Convoy Explosion
Turkey
- - 3 13
Donald Trump
Attacker
1 25 - -
Eagle Kid 1 334 - -
Five Headed Snake 5 6 - -
Fuji Lenticular
Clouds
1 123 1 53
Total 66 1230 64 998
Event
Fake Real
MM Posts MM Posts
Gandhi Dancing 1 29 - -
Half of
Everything
9 39 - -
Hubble
Telescope
1 18 - -
Immigrants’ fear 5 33 3 18
ISIS children 2 3 - -
John Guevara 1 33 - -
Mc Donalds’ Fee 1 6 - -
Nazi Submarine 2 11 - -
North Korea 2 10 - -
Not Afraid 2 32 3 35
Pakistan
Explosion
1 53 - -
Pope Francis 1 29 - -
Protest 1 30 10 34
Refugees 4 35 13 33
Rio Moon 1 33 - -
Snowboard Girl 2 14 - -
Soldier Stealing 1 1 - -
Syrian Children 1 12 1 200
Ukrainian Nazi 1 1 - -
Woman 14
children
2 11 - -

13. EVALUATION & RESULTS4

14. Main task
Target class: Fake
TASK EVALUATION
Sub-task
Target class: Tampered
Classic IR metrics
Precision
Recall
F1-score -> main evaluation metric
Participants were allowed to mark a case
as “unknown” (expected to result in
reduced recall)

15. TASK SUBMISSIONS
10 submissions for the main task
2 submissions for the sub-task
(just one team)
3 teams submitted
(+1 the organizers)

16. TRENDS IN APPROACHES
Features being used
- Text features (most common)
- Post and user metadata
- Image forensics
- Video quality metadata
- Topics of post
- Text similarity of posts (per image case)
- Trusted sources attributed in text
- Mentioned online external sources

17. RESULTS: MAIN TASK
Team Run Recall Precision F1-Score
Linkmedia
run1TextKnn 0.9227 0.6397 0.7556
run2CBIR1 0.3406 0.4917 0.4024
run3Sources 0.9463 0.9030 0.9241
run4Fusion 0.9121 0.7525 0.8246
MMLAB@DISI
RUN1 0.5487 0.7060 0.6175
RUN2 0.9365 0.8135 0.8707
RUN3 0.9398 0.7405 0.8283
MCGICT
hybrid 0.6097 0.7637 0.6781
image 0.5138 0.6975 0.5917
text 0.6292 0.7471 0.6831
VMU
Run1 0.8512 0.9812 0.9116
Run2 0.9056 0.7709 0.8328
Run3 0.8869 0.9882 0.9348
Run4 0.8739 0.9799 0.9239
Run5 0.9951 0.5873 0.7386
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants and organizers F1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants F1

18. RESULTS: MAIN TASK
Team Run Recall Precision F1-Score
Linkmedia
run1TextKnn 0.9227 0.6397 0.7556
run2CBIR1 0.3406 0.4917 0.4024
run3Sources 0.9463 0.9030 0.9241
run4Fusion 0.9121 0.7525 0.8246
MMLAB@DISI
RUN1 0.5487 0.7060 0.6175
RUN2 0.9365 0.8135 0.8707
RUN3 0.9398 0.7405 0.8283
MCGICT
hybrid 0.6097 0.7637 0.6781
image 0.5138 0.6975 0.5917
text 0.6292 0.7471 0.6831
VMU
Run1 0.8512 0.9812 0.9116
Run2 0.9056 0.7709 0.8328
Run3 0.8869 0.9882 0.9348
Run4 0.8739 0.9799 0.9239
Run5 0.9951 0.5873 0.7386
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants and organizers F1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants F1

19. RESULTS: MAIN TASK
Team Run Recall Precision F1-Score
Linkmedia
run1TextKnn 0.9227 0.6397 0.7556
run2CBIR1 0.3406 0.4917 0.4024
run3Sources 0.9463 0.9030 0.9241
run4Fusion 0.9121 0.7525 0.8246
MMLAB@DISI
RUN1 0.5487 0.7060 0.6175
RUN2 0.9365 0.8135 0.8707
RUN3 0.9398 0.7405 0.8283
MCGICT
hybrid 0.6097 0.7637 0.6781
image 0.5138 0.6975 0.5917
text 0.6292 0.7471 0.6831
VMU
Run1 0.8512 0.9812 0.9116
Run2 0.9056 0.7709 0.8328
Run3 0.8869 0.9882 0.9348
Run4 0.8739 0.9799 0.9239
Run5 0.9951 0.5873 0.7386
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants and organizers F1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants F1

20. RESULTS: MAIN TASK
Team Run Recall Precision F1-Score
Linkmedia
run1TextKnn 0.9227 0.6397 0.7556
run2CBIR1 0.3406 0.4917 0.4024
run3Sources 0.9463 0.9030 0.9241
run4Fusion 0.9121 0.7525 0.8246
MMLAB@DISI
RUN1 0.5487 0.7060 0.6175
RUN2 0.9365 0.8135 0.8707
RUN3 0.9398 0.7405 0.8283
MCGICT
hybrid 0.6097 0.7637 0.6781
image 0.5138 0.6975 0.5917
text 0.6292 0.7471 0.6831
VMU
Run1 0.8512 0.9812 0.9116
Run2 0.9056 0.7709 0.8328
Run3 0.8869 0.9882 0.9348
Run4 0.8739 0.9799 0.9239
Run5 0.9951 0.5873 0.7386
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants and organizers F1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants F1

21. RESULTS: MAIN TASK
Team Run Recall Precision F1-Score
Linkmedia
run1TextKnn 0.9227 0.6397 0.7556
run2CBIR1 0.3406 0.4917 0.4024
run3Sources 0.9463 0.9030 0.9241
run4Fusion 0.9121 0.7525 0.8246
MMLAB@DISI
RUN1 0.5487 0.7060 0.6175
RUN2 0.9365 0.8135 0.8707
RUN3 0.9398 0.7405 0.8283
MCGICT
hybrid 0.6097 0.7637 0.6781
image 0.5138 0.6975 0.5917
text 0.6292 0.7471 0.6831
VMU
Run1 0.8512 0.9812 0.9116
Run2 0.9056 0.7709 0.8328
Run3 0.8869 0.9882 0.9348
Run4 0.8739 0.9799 0.9239
Run5 0.9951 0.5873 0.7386
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants and organizers F1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Participants F1

22. RESULTS: SUB-TASK
Team Run Recall Precision F1-Score
MMLAB@DISI
RUN1 0.5 0.4827 0.4912
RUN2 0.9285 0.4906 0.6420
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7

23. FUTURE PLANS
Reconsider the fake/real distinction
Think about different evaluation
metrics
Use posts from other social media

24. Thanks for your attention!
ANY QUESTIONS?
Get in touch at
boididou@iti.gr (@cmpoi)
papadop@iti.gr (@sympap)
www.revealproject.eu
@RevealEU