Media hyperlinking consists in linking together media based on their content, uncovering the relation between them. It is becoming an important functionality for providing users with a way to navigate between video entities illustrating various related facets of the original topic and satisfy their information needs. Thanks to such approaches, multimedia search can often be replaced by recommendation. A particular usage of hyperlinking is to provide, through a second screen application, extra information or content about the video watched on a main screen (TV). In this talk, we will focus on event-centric media hyperlinking: the task at hand consists in locating and identifying relevant media items, and displaying them on the second screen. The related material is selected based on underlying events that will be detected through web services: events are seen as structuring elements, defined in terms of date, location, intent and attendance. Two approaches for event-based mining of such additional and related information will be presented. Each of them satisfying a different user information need and addressing different usage scenario; static vs dynamic second screening. In both cases, the event-centric organization of the multimedia material enables to rapidly and effectively visualise what matters about topics of interest.

1. Event-centric HyperLinking of
Socially Contributed
Multimedia Content
Mathilde Sahuguet, Xueliang Liu, Benoit Huet
EURECOM, Sophia Antipolis, France

2. Our motivation…
 Storytelling is the conveying of events in
words, and images… videos…
 Hyperlinks are conveying additional or related
information
 Related content (Media) would benefit from
being organized around events
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 2

3. Audiovisual Explosion
 EU alone hosts 500+ online video platforms
 42.7m hrs of footage in online archives of
broadcasters and producers (61% of archive
footage is online)
 UGC on the advance:
YouTube receives 60 hrs of video/minute
 Internet video is now 40 percent of consumer
Internet traffic, and will reach 62 percent by the
end of 2015
 Media Retrieval Issues
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 3

4. Hyperlinking and Second Screen
 Nielsen 12/2012: 85 Percent Of Tablet And
Smartphone Owners Use Devices As “Second
Screen” Monthly, 40 Percent Do So Daily
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 4

5. Hyperlinking and Second Screen
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 5

6. Hyperlinking at the fragment level
 MediaMixer:
“set up and sustain a community of video producers,
hosters and redistributors…”
“who will be supported in the adoption of semantic
multimedia technology…”
“to build an European market for media fragment re-
purposing and re-selling.”
01/04/14 - SoMuS 2014 / ICMR 2014
http://community.mediamixer.eu
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 6

7. Hyperlinking and Second Screen
01/04/14 - SoMuS 2014 / ICMR 2014
 LinkedTV: linking television to
Web content
 Second screen scenario for
enriching television content
and achieving interaction
between user and content
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 7

8. Enrichment framework
01/04/14 - SoMuS 2014 / ICMR 2014
Media Annotation
Web Media Search
Segment annotation Media
items
Annotated
set of media
items
Multimedia MiningInput video
Set of relevant
media with cues
Personalization,
Display, etc
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 8

9. Topic Enrichment Hyperlinking
 Need to enrich given topics:
based on terms/topics appearing in video (passive or
active)
based on user query (active)
 Select and organize media items
Different approaches for searching for information
Diverse way to render information
 What are people interested to see ?
 we propose and investigate:
Event-centric HyperLinking of Socially
Contributed Multimedia Content
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 9

10. Publish content
Socially Aware Enrichments
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 10

11. Query content
Socially Aware Enrichments
01/04/14 - SoMuS 2014 / ICMR 2014
Search term
Google
Trends
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 11

12. Event-centric Enrichments
 Events are natural structuring elements for
humans
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 12

13. Outline
 Related work
 Framework
Enrichments for Passive Second Screen
 Query Formulation
 Static Event visualisation
Enrichments for Active Second Screen
Trends mining
Focused search
Video selection
Evaluation and Results
 Conclusion
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 13

14. Related work
 Chieu et al. (ACM SIGIR 2004)
 Query based event extraction along a timeline
 Yan et al. (CIKM2012)
 Visualizing timelines: evolutionary summarization via iterative reinforcement
between text and image streams.
 Christiansen et al. (WIDM '12)
 Modeling topic trends on the social web using temporal signatures
 Chen et al. (Knowledge and Data Engineering 2007)
 Hot Topic Extraction Based on Timeline Analysis and Multidimensional
Sentence Modeling
 Tan et al. (ACM MM’10)
 Topic-based summarization of web videos by visual-text time-dependent
alignment.
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 14

15. Our Contributions
 Event-based methodology
4W paradigm (Who, What, Where, When)
 Mining events from user contributed data and
search trends
People-based point of view (not editorial choice)
Event discovery
 Event illustration
Mining from a pool of potential media items
Choice of illustrative media item(s)
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 15

16. Pre-Processing Media
 Search
Query provided by the user
 Hyperlinking
The content is the query
Content analysis is required!
Legacy metadata / Subtitles
Video Segmentation
Automatic Speech Recognition
Visual Concept Detection
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 16

17. Passive Second Screen Framework
query eventslocation
time
topic
01/04/14 - SoMuS 2014 / ICMR 2014
Step1
Step2
Step3
 Aim:
Enrich & Illustrate the Event on the TV Screen
Step1: parse the semantic from query string
Step2: Extract events from Social News site
Step3: Illustrating with tag cloud and image collage
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 17

18. Event Querying
 Events are queried from Digg.com
a news aggregator with an editorially driven front page
01/04/14 - SoMuS 2014 / ICMR 2014
Two advantages:
1. Save storage and
computation
2. Leverage from
collective intelligence
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 18

19. Textual Summarization
 Source: Full textual search on Twitter;
query words = event title
filter by time = event occurrence date
 Representation: Tag Cloud
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 19

20. Visual Summarization
 Source: Full textual search on Google Images
Query words = event title
Sorted by the similarity of image and event title
 Representation: Visual Collage
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 20

21. Event Summary
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 21

22. Event Summary
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 22

23. 01/04/14 - SoMuS 2014 / ICMR 2014
Active Second Screen Framework
 Aim:
Propose Enrichments which Illustrate the Event
on the TV Screen
Interaction with the viewer
Multiple choices
Dynamic rendering
TV / Main screen might need to be paused
Leverage on user generated content and collective
intelligence
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 23

24. 01/04/14 - SoMuS 2014 / ICMR 2014
Active Second Screen Framework
 Goal = automatically create and illustrate a
timeline of events on a given topic
 Event: occurrence of abnormal activity on a
limited time segment, that captured a lot of
interest and triggered massive web search
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 24

25.  Bursts = peaks: sudden increase in the search
volume => unexpected event, massive search
behavior
 Long-term events: abnormal high interest
spread over several time units
Trends mining
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 25

26. Trends mining
 Granularity:
 Trend value: Week (imposed by Google trends)
 Trend keywords: Monthly
 Feature:
 Burst value= left derivative of the trends values
 Threshold to select bursts:
 Adaptive threshold
 mean + standard deviation
 Output:
 Set of week dates
 Objective:
 link trendy time-segment to some event in the real world
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 26

27. Focused search on time segments
 Goal: search for items relevant to the event on
the extracted dates
 Extracting keywords for a focused search:
Input: initial search term + dates (time segment)
Query to Google trends
Output: list of associated rising search terms
 Focused media search:
Private / Public Media repositories
Social Media platform
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 27

28. Video Selection
01/04/14 - SoMuS 2014 / ICMR 2014
 Goal: represent each event with a media item
 Features:
Using text (from media title + description)
Discard non-English content
TF-IDF with cosine distance
 Media Item Selection
Item with average similarity to all other media items
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 28

29. Example query: Snowden
1. Get trends
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content01/04/14 - SoMuS 2014 / ICMR 2014 - p 29

30. Example query: Snowden
2. Mine time segments
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
01/04/14 - SoMuS 2014 / ICMR 2014 - p 30

31. Example query: Snowden
3. Extract associated keywords
(month granularity)
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
01/04/14 - SoMuS 2014 / ICMR 2014 - p 31

32. Example query: Snowden
3. Focused media search:
Tag cloud of associated text
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
01/04/14 - SoMuS 2014 / ICMR 2014 - p 32

33. 4. Illustrative video selection
Example query: Snowden
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
01/04/14 - SoMuS 2014 / ICMR 2014 - p 33

34. 5. Ground truth
Example query: Snowden
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
Snowden identity was
revealed on the 9th
Snowden flied to Russia on
the 23rd
01/04/14 - SoMuS 2014 / ICMR 2014 - p 34

35. Experiments
 Focus on people: automatic event-based
timeline generation of 5 celebrities
 “Ground truth” extracted from expert
biographies and Wikipedia (from 2004)
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
Mark
Zuckerberg
27 events
Oscar
Pistorius
9 events
Batman
9 events
Beyoncé
Knowles
28 events
Edward
Snowden
43 events
01/04/14 - SoMuS 2014 / ICMR 2014 - p 35

36. Event extraction Evaluation
 Performance of our approach to detect
important events
 TP: true positive, TP: false positive, FN: false negative, DE: discovered event
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
query # bursts TP FP FN DE
Beyoncé K. 9 7 1 21 1
M.Zuckerberg 6 2 4 25 0
O.Pistorius 1 1 0 8 0
Batman 6 2 2 7 2
E. Snowden 2 2 0 41 0
01/04/14 - SoMuS 2014 / ICMR 2014 - p 36

37. Event discovery: Beyoncé
 Burst week: 22nd to 28th July, 2013
 Nothing in the ground truth
 Associated keywords:
Beyonce fall video
Beyonce falling
Beyonce falls
Beyonce orlando
Beyonce stairs
Caida beyonce
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content01/04/14 - SoMuS 2014 / ICMR 2014 - p 37

38. Video representative evaluation
 Challenge: choice of an illustrative video
 Event Relatedness ER (mean over all events):
evaluation of the content of the video (not the quality)
 Based on description provided by users
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
Query # events Event
Relatedness
Beyoncé K. 8 0,81
M. Zuckerberg 2 1
O. Pistorius 1 0
Batman 4 1
E. Snowden 2 1
01/04/14 - SoMuS 2014 / ICMR 2014 - p 38

39.  Week: 10th to 16th February, 2013
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 39

40.  Week: 10th to 16th February, 2013
 Associated keywords:
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 40

41.  Week: 10th to 16th February, 2013
 Associated keywords:
 DataSet tag cloud:
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 41

42.  Week: 10th to 16th February, 2013
 Associated keywords:
 Dataset tag cloud:
 Illustrative video:
 Nothing about Pistorius himself or the murder case
 One of his opponents
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
- p 42

43.  Week: 10th to 16th February, 2013
 Associated keywords:
 Dataset tag cloud:
 Illustrative video:
 Why: associated text
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
BBC News -Paralympic Star Oscar Pistorius 'shoots girlfriend' 2013 Inscreva-se em
nosso canal. Subscribe yourself to our channel. Suscríbase a nuestro canal, News for
Oscar Pistorius girlfriend shot dead Oscar Pistorius charged with murder after
girlfriend shot dead The Guardian ?- 17 hours ago South African athlete will appear in
court on Friday after Reeva Steenkamp was found dead at his home in Pretoria.
Pistorius' girlfriend killed on Valentine's Day she was looking forward to CNN? - 15
hours ago South African sprinter Oscar Pistorius charged with murder after girlfriend,
Reeva Steenkamp, shot dead in his home New York Daily News? - 6 hours ago
Pistorius 'shot, killed' girlfriend - The Courier-Mail www.couriermail.com.au/.../
- p 43

44. Conclusion
 Socially Aware Hyperlinking Approaches
 Automated Event-centric Illustration of topics
 Creation of an Event-centric media-illustrated
timeline related to a topic
 “new” Events Discovery = non-official but
triggered massive attention/searches
 Future work:
Work with visual features + ASR.
Detect and identify people
Work on long-term events
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 44

45. References
 http://www.linkedtv.eu
 http://www.mediamixer.eu
 B. Safadi, M. Sahuguet, B. Huet, “When Textual and Visual Information join forces
for Multimedia Retrieval”, ICMR 2014, ACM International Conference on
Multimedia Retrieval 2014, April 1-4, Glasgow, Scotland
 M. Sahuguet and B. Huet, “Mining the Web for Multimedia-based Enriching”,
Multimedia Modeling 2014, Dublin, Ireland
 X. Liu and B. Huet “Event Representation and Visualization from Social Media”,
Pacific Rim Conference on Multimedia 2013, Dec 13-16, Nanjing, China
 M. Sahuguet and B. Huet, “Socially motivated multimedia topic timeline
summarization”, 2nd ACM International Workshop on Socially-Aware
Multimedia, In conjunction with ACM Multimedia 2013, 21 October 2013,
Barcelona, Spain
 X. Liu and B. Huet “On the automatic online collection of training data for visual
event modeling” Multimedia Tools and Applications, February 2013, ISSN:
1380-7501
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 49

46. Thank you for your attention
 Questions?
Benoit Huet
EURECOM tel: +33 (0)493008179
Campus SophiaTech, 450 route des Chappes fax: +33 (0)493008200
06410 Biot Sophia Antipolis email: Benoit.Huet@eurecom.fr
France http://www.eurecom.fr/en/people/huet-benoit
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 50