Media hyperlinking consists in linking together media based on their content, uncovering the relation between them. It is becoming an important functionality for providing users with a way to navigate between video entities illustrating various related facets of the original topic and satisfy their information needs. Thanks to such approaches, multimedia search can often be replaced by recommendation. A particular usage of hyperlinking is to provide, through a second screen application, extra information or content about the video watched on a main screen (TV). In this talk, we will focus on event-centric media hyperlinking: the task at hand consists in locating and identifying relevant media items, and displaying them on the second screen. The related material is selected based on underlying events that will be detected through web services: events are seen as structuring elements, defined in terms of date, location, intent and attendance. Two approaches for event-based mining of such additional and related information will be presented. Each of them satisfying a different user information need and addressing different usage scenario; static vs dynamic second screening. In both cases, the event-centric organization of the multimedia material enables to rapidly and effectively visualise what matters about topics of interest.
2. Our motivation…
Storytelling is the conveying of events in
words, and images… videos…
Hyperlinks are conveying additional or related
information
Related content (Media) would benefit from
being organized around events
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 2
3. Audiovisual Explosion
EU alone hosts 500+ online video platforms
42.7m hrs of footage in online archives of
broadcasters and producers (61% of archive
footage is online)
UGC on the advance:
YouTube receives 60 hrs of video/minute
Internet video is now 40 percent of consumer
Internet traffic, and will reach 62 percent by the
end of 2015
Media Retrieval Issues
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 3
4. Hyperlinking and Second Screen
Nielsen 12/2012: 85 Percent Of Tablet And
Smartphone Owners Use Devices As “Second
Screen” Monthly, 40 Percent Do So Daily
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 4
5. Hyperlinking and Second Screen
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 5
6. Hyperlinking at the fragment level
MediaMixer:
“set up and sustain a community of video producers,
hosters and redistributors…”
“who will be supported in the adoption of semantic
multimedia technology…”
“to build an European market for media fragment re-
purposing and re-selling.”
01/04/14 - SoMuS 2014 / ICMR 2014
http://community.mediamixer.eu
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 6
7. Hyperlinking and Second Screen
01/04/14 - SoMuS 2014 / ICMR 2014
LinkedTV: linking television to
Web content
Second screen scenario for
enriching television content
and achieving interaction
between user and content
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 7
8. Enrichment framework
01/04/14 - SoMuS 2014 / ICMR 2014
Media Annotation
Web Media Search
Segment annotation Media
items
Annotated
set of media
items
Multimedia MiningInput video
Set of relevant
media with cues
Personalization,
Display, etc
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 8
9. Topic Enrichment Hyperlinking
Need to enrich given topics:
based on terms/topics appearing in video (passive or
active)
based on user query (active)
Select and organize media items
Different approaches for searching for information
Diverse way to render information
What are people interested to see ?
we propose and investigate:
Event-centric HyperLinking of Socially
Contributed Multimedia Content
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 9
10. Publish content
Socially Aware Enrichments
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 10
11. Query content
Socially Aware Enrichments
01/04/14 - SoMuS 2014 / ICMR 2014
Search term
Google
Trends
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 11
12. Event-centric Enrichments
Events are natural structuring elements for
humans
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 12
13. Outline
Related work
Framework
Enrichments for Passive Second Screen
Query Formulation
Static Event visualisation
Enrichments for Active Second Screen
Trends mining
Focused search
Video selection
Evaluation and Results
Conclusion
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 13
14. Related work
Chieu et al. (ACM SIGIR 2004)
Query based event extraction along a timeline
Yan et al. (CIKM2012)
Visualizing timelines: evolutionary summarization via iterative reinforcement
between text and image streams.
Christiansen et al. (WIDM '12)
Modeling topic trends on the social web using temporal signatures
Chen et al. (Knowledge and Data Engineering 2007)
Hot Topic Extraction Based on Timeline Analysis and Multidimensional
Sentence Modeling
Tan et al. (ACM MM’10)
Topic-based summarization of web videos by visual-text time-dependent
alignment.
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 14
15. Our Contributions
Event-based methodology
4W paradigm (Who, What, Where, When)
Mining events from user contributed data and
search trends
People-based point of view (not editorial choice)
Event discovery
Event illustration
Mining from a pool of potential media items
Choice of illustrative media item(s)
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 15
16. Pre-Processing Media
Search
Query provided by the user
Hyperlinking
The content is the query
Content analysis is required!
Legacy metadata / Subtitles
Video Segmentation
Automatic Speech Recognition
Visual Concept Detection
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 16
17. Passive Second Screen Framework
query eventslocation
time
topic
01/04/14 - SoMuS 2014 / ICMR 2014
Step1
Step2
Step3
Aim:
Enrich & Illustrate the Event on the TV Screen
Step1: parse the semantic from query string
Step2: Extract events from Social News site
Step3: Illustrating with tag cloud and image collage
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 17
18. Event Querying
Events are queried from Digg.com
a news aggregator with an editorially driven front page
01/04/14 - SoMuS 2014 / ICMR 2014
Two advantages:
1. Save storage and
computation
2. Leverage from
collective intelligence
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 18
19. Textual Summarization
Source: Full textual search on Twitter;
query words = event title
filter by time = event occurrence date
Representation: Tag Cloud
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 19
20. Visual Summarization
Source: Full textual search on Google Images
Query words = event title
Sorted by the similarity of image and event title
Representation: Visual Collage
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 20
21. Event Summary
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 21
22. Event Summary
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 22
23. 01/04/14 - SoMuS 2014 / ICMR 2014
Active Second Screen Framework
Aim:
Propose Enrichments which Illustrate the Event
on the TV Screen
Interaction with the viewer
Multiple choices
Dynamic rendering
TV / Main screen might need to be paused
Leverage on user generated content and collective
intelligence
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 23
24. 01/04/14 - SoMuS 2014 / ICMR 2014
Active Second Screen Framework
Goal = automatically create and illustrate a
timeline of events on a given topic
Event: occurrence of abnormal activity on a
limited time segment, that captured a lot of
interest and triggered massive web search
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 24
25. Bursts = peaks: sudden increase in the search
volume => unexpected event, massive search
behavior
Long-term events: abnormal high interest
spread over several time units
Trends mining
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 25
26. Trends mining
Granularity:
Trend value: Week (imposed by Google trends)
Trend keywords: Monthly
Feature:
Burst value= left derivative of the trends values
Threshold to select bursts:
Adaptive threshold
mean + standard deviation
Output:
Set of week dates
Objective:
link trendy time-segment to some event in the real world
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 26
27. Focused search on time segments
Goal: search for items relevant to the event on
the extracted dates
Extracting keywords for a focused search:
Input: initial search term + dates (time segment)
Query to Google trends
Output: list of associated rising search terms
Focused media search:
Private / Public Media repositories
Social Media platform
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 27
28. Video Selection
01/04/14 - SoMuS 2014 / ICMR 2014
Goal: represent each event with a media item
Features:
Using text (from media title + description)
Discard non-English content
TF-IDF with cosine distance
Media Item Selection
Item with average similarity to all other media items
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 28
29. Example query: Snowden
1. Get trends
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content01/04/14 - SoMuS 2014 / ICMR 2014 - p 29
30. Example query: Snowden
2. Mine time segments
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
01/04/14 - SoMuS 2014 / ICMR 2014 - p 30
31. Example query: Snowden
3. Extract associated keywords
(month granularity)
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
01/04/14 - SoMuS 2014 / ICMR 2014 - p 31
32. Example query: Snowden
3. Focused media search:
Tag cloud of associated text
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
01/04/14 - SoMuS 2014 / ICMR 2014 - p 32
33. 4. Illustrative video selection
Example query: Snowden
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
01/04/14 - SoMuS 2014 / ICMR 2014 - p 33
34. 5. Ground truth
Example query: Snowden
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
9th to 15th June 2013
23rd to 29th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
Snowden identity was
revealed on the 9th
Snowden flied to Russia on
the 23rd
01/04/14 - SoMuS 2014 / ICMR 2014 - p 34
35. Experiments
Focus on people: automatic event-based
timeline generation of 5 celebrities
“Ground truth” extracted from expert
biographies and Wikipedia (from 2004)
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
Mark
Zuckerberg
27 events
Oscar
Pistorius
9 events
Batman
9 events
Beyoncé
Knowles
28 events
Edward
Snowden
43 events
01/04/14 - SoMuS 2014 / ICMR 2014 - p 35
36. Event extraction Evaluation
Performance of our approach to detect
important events
TP: true positive, TP: false positive, FN: false negative, DE: discovered event
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
query # bursts TP FP FN DE
Beyoncé K. 9 7 1 21 1
M.Zuckerberg 6 2 4 25 0
O.Pistorius 1 1 0 8 0
Batman 6 2 2 7 2
E. Snowden 2 2 0 41 0
01/04/14 - SoMuS 2014 / ICMR 2014 - p 36
37. Event discovery: Beyoncé
Burst week: 22nd to 28th July, 2013
Nothing in the ground truth
Associated keywords:
Beyonce fall video
Beyonce falling
Beyonce falls
Beyonce orlando
Beyonce stairs
Caida beyonce
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content01/04/14 - SoMuS 2014 / ICMR 2014 - p 37
38. Video representative evaluation
Challenge: choice of an illustrative video
Event Relatedness ER (mean over all events):
evaluation of the content of the video (not the quality)
Based on description provided by users
Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
Query # events Event
Relatedness
Beyoncé K. 8 0,81
M. Zuckerberg 2 1
O. Pistorius 1 0
Batman 4 1
E. Snowden 2 1
01/04/14 - SoMuS 2014 / ICMR 2014 - p 38
39. Week: 10th to 16th February, 2013
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 39
40. Week: 10th to 16th February, 2013
Associated keywords:
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 40
41. Week: 10th to 16th February, 2013
Associated keywords:
DataSet tag cloud:
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 41
42. Week: 10th to 16th February, 2013
Associated keywords:
Dataset tag cloud:
Illustrative video:
Nothing about Pistorius himself or the murder case
One of his opponents
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
- p 42
43. Week: 10th to 16th February, 2013
Associated keywords:
Dataset tag cloud:
Illustrative video:
Why: associated text
Example query: Pistorius
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
BBC News -Paralympic Star Oscar Pistorius 'shoots girlfriend' 2013 Inscreva-se em
nosso canal. Subscribe yourself to our channel. Suscríbase a nuestro canal, News for
Oscar Pistorius girlfriend shot dead Oscar Pistorius charged with murder after
girlfriend shot dead The Guardian ?- 17 hours ago South African athlete will appear in
court on Friday after Reeva Steenkamp was found dead at his home in Pretoria.
Pistorius' girlfriend killed on Valentine's Day she was looking forward to CNN? - 15
hours ago South African sprinter Oscar Pistorius charged with murder after girlfriend,
Reeva Steenkamp, shot dead in his home New York Daily News? - 6 hours ago
Pistorius 'shot, killed' girlfriend - The Courier-Mail www.couriermail.com.au/.../
- p 43
44. Conclusion
Socially Aware Hyperlinking Approaches
Automated Event-centric Illustration of topics
Creation of an Event-centric media-illustrated
timeline related to a topic
“new” Events Discovery = non-official but
triggered massive attention/searches
Future work:
Work with visual features + ASR.
Detect and identify people
Work on long-term events
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 44
45. References
http://www.linkedtv.eu
http://www.mediamixer.eu
B. Safadi, M. Sahuguet, B. Huet, “When Textual and Visual Information join forces
for Multimedia Retrieval”, ICMR 2014, ACM International Conference on
Multimedia Retrieval 2014, April 1-4, Glasgow, Scotland
M. Sahuguet and B. Huet, “Mining the Web for Multimedia-based Enriching”,
Multimedia Modeling 2014, Dublin, Ireland
X. Liu and B. Huet “Event Representation and Visualization from Social Media”,
Pacific Rim Conference on Multimedia 2013, Dec 13-16, Nanjing, China
M. Sahuguet and B. Huet, “Socially motivated multimedia topic timeline
summarization”, 2nd ACM International Workshop on Socially-Aware
Multimedia, In conjunction with ACM Multimedia 2013, 21 October 2013,
Barcelona, Spain
X. Liu and B. Huet “On the automatic online collection of training data for visual
event modeling” Multimedia Tools and Applications, February 2013, ISSN:
1380-7501
01/04/14 - SoMuS 2014 / ICMR 2014 Benoit HUET - Event-centric HyperLinking of Socially Contributed Multimedia Content - p 49
46. Thank you for your attention
Questions?
Benoit Huet
EURECOM tel: +33 (0)493008179
Campus SophiaTech, 450 route des Chappes fax: +33 (0)493008200
06410 Biot Sophia Antipolis email: Benoit.Huet@eurecom.fr
France http://www.eurecom.fr/en/people/huet-benoit
01/04/14 - SoMuS 2014 / ICMR 2014
Benoit HUET - Event-centric HyperLinking of
Socially Contributed Multimedia Content - p 50