1. Television linked to the web:
the case for audiovisual archives
Johan Oomen and Lotte Belice Baltussen
@johanoomen and @lottebelice
Netherlands Institute for Sound and Vision
FIAT/IFTA | Dubai, United Arab Emirates – 27 October 2013
www.linkedtv.eu

2. www.linkedtv.eu
multimedia
content
analysis
new distribution
platforms
lots of archive
content
2

3. www.linkedtv.eu
multimedia
content
analysis
new distribution
platforms
where the magic
happens!
lots of archive
content
2

4. Where the magic happens
www.linkedtv.eu
multimedia
content
analysis
new distribution
platforms
lots of archive
content
3

6. Audiovisual
broadcasts

7. Audiovisual
broadcasts

8. Audiovisual
broadcasts
Professional annotation

9. Audiovisual
broadcasts
Professional annotation
Search engine

10. Audiovisual
broadcasts
Professional annotation
Television
makers
Search engine

11. Audiovisual
broadcasts
Professional annotation
Academics
Television
makers
Search engine

12. Audiovisual
broadcasts
General
public
Professional annotation
Academics
Television
makers
Search engine

13. Audiovisual
broadcasts
General
public
Professional annotation
Machine analysis
Academics
Data gathering
Television
makers
Search engine

14. multimedia content analysis
Data gathering
Crowdsourcing
Twitter
News
Importer
Subtitles
Wikis

15. Data Gathering:
Pros and Cons

16. Data Gathering:
Pros and Cons
•
Large-scale data for
some programs
•
Noisy data
(sometimes)
•
Access to opinions
about television
•
Not created for the
archive
•
Puts programs in
larger national context
•
•
Unclear how the data
can be used
(sometimes)
Affordable

17. multimedia content analysis
Machine Analysis
Multimedia
content analysis
Low-level features

18. multimedia content analysis
Machine Analysis
Concept detectors
Audio transcripts
today sniper fire disrupted the funeral of an
eleven year old ethnic albanian boy he was
killed yesterday while while chopping wood
his family blames serb police before his death
louisiana state police now say six workers
were killed after a natural gas well exploded
and caught fire about forty five miles east of
shreveport four others were injured in
yesterday's blast a police spokesman says the
derek started to melt in the intense heat the
Speaker recognition
Multimedia
content analysis
Low-level features
Face recognition

19. Machine Analysis:
Pros and Cons

20. •
Machine Analysis:
Pros and Cons
Extremely detailed,
millisecond-level
descriptions
•
Directly describe video
content
•
•
Affordable
Large scale!
•
•
Noisy
•
No summarisation of
what is important
•
Confusing for users
Limited to what the
machine “knows”

21. Media fragments
www.linkedtv.eu
10
linkedtv.eu

22. Where the magic happens
www.linkedtv.eu
multimedia
content
analysis
new distribution
platforms
lots of archive
content
11

23. Context guestimates
www.linkedtv.eu
thousands of hrs of new television content produced by
Dutch public broadcasters every year
Hours of archival content of Sound
and Vision used in this new content
12
linkedtv.eu

24. How do broadcast professionals use
archive content?
www.linkedtv.eu
Number of orders
§ Query log analysis.* (50k user
sessions, 264k queries)
4000
3500
3000
2500
2000
1500
1000
500
0
broadcast order
story order
fragment order
>30
29-30
28-29
27-28
26-27
25-26
24-25
23-24
22-23
21-22
20-21
19-20
18-19
17-18
16-17
15-16
14-15
13-14
12-13
11-12
10-11
9-10
8-9
7-8
6-7
5-6
4-5
3-4
2-3
1-2
0-1
Order duration in minutes
§
§
§
33% programs
17% stories, and fragments
50% audiovisual fragments with a start and end time
specified by the users themselves.
13
* Paper: Search Behavior of Media Professionals at an Audiovisual Archive: ATransaction Log Analysis. By Bouke Huurnink

25. Preference for new material
www.linkedtv.eu
§ 46 % of the orders are for footage
broadcast more than 1 year before the
order date
§ 12% of orders are for material that is
over 10 years old.
§ => a lot of high quality content is not
used.
14

26. Context guestimates
www.linkedtv.eu
thousands of hrs of new television content produced by
Dutch public broadcasters every year
Hours of archival content of Sound
and Vision used in this new content
15
linkedtv.eu

27. Context – the potential
www.linkedtv.eu
Total av-collection Sound and Vision
750k of hours (& growing daily)
16
linkedtv.eu

28. Context – the potential: on-demand
personalised services
Total av-collection Sound and Vision
750k hours (& growing daily)
17
linkedtv.eu
www.linkedtv.eu

29. Where the magic happens
www.linkedtv.eu
multimedia
content
analysis
new distribution
platforms
lots of archive
content
18

30. Web and TV exist in parallel
www.linkedtv.eu
§
86% of TV viewers surf the Web at the same time
(Yahoo!, USA)
§
88% of USA consumers use mobile as second
screen (Business Insider, May 2013)
§
Manual efforts to link TV program to additional
content (Shazam, IntoNow)
19
linkedtv.eu

31. Samsung advert
www.linkedtv.eu
20
http://www.youtube.com/watch?v=UwmoBmlkaGQ

32. Examples – BBC Interactive Video Player
www.linkedtv.eu
21
linkedtv.eu

33. Examples – Hulu Face Match
www.linkedtv.eu
22
linkedtv.eu

34. The LinkedTV vision
www.linkedtv.eu
LinkedTV: interweaving Web and TV
into a single experience
§ Mixing lean back and lean forward
§ how to maintain immersion while
removing fourth wall illusion?
§ will this work across genres and
audiences?
§ Does second screen help or hinder?
23
linkedtv.eu

35. www.linkedtv.eu
Research Organisations
SME Technology Providers
User Interface and Scenario Partners
Technical coordinator
Studies
24
Market
linkedtv.eu

36. LinkedTV – Use cases
www.linkedtv.eu
Hyperlinked Documentary:
Antiques Interactive
§ Program watched
predominantly by 50+ers
§ AVRO:
Tussen Kunst en Kitsch (TKK)
§ Objects, locations and
experts provide anchors to
related materials (targets).
25
Interactive News
§ Professional news content
produced by RBB for general
audience
§ local news show rbb Aktuell
§ People, locations and events
can be linked to previous
events.

37. Use case – Antiques Interactive
Bert and Anne
www.linkedtv.eu
“I like to relax and sit back when I
watch Antiques Road Show. It’s
always exciting when something’s
worth a lot. In some cases, I like to
look up extra info about an object or
art style after the show on the PC,
but if something doesn’t interest me,
I’ll just read a magazine.”
“As an antiques shop owner, by work is
also my hobby. I’m always looking up
information about art objects on my PC
or tablet. I like watching my colleagues
talking about are on Antiques Road Show
and I use my tablet to bookmark specific
objects or other information that I like. I
also do this for Anne, and send her
overviews.”
26

38. How LinkedTV helps Bert and Anne
www.linkedtv.eu
Video Object and
word detection
Connection to
concepts
Selecton of related
concepts
Selection of related
content
Presentation
engine
27
linkedtv.eu

39. How LinkedTV helps Bert and Anne
www.linkedtv.eu
Video Object and
word detection
Connection to
concepts
Selecton of related
concepts
Selection of related
content
“...painting of Jan Sluijters....”
27
Presentation
engine
linkedtv.eu

40. How LinkedTV helps Bert and Anne
www.linkedtv.eu
Video Object and
word detection
Connection to
concepts
Selecton of related
concepts
Selection of related
content
“...painting of Jan Sluijters....”
dbpedia.org/resource/Jan_Sluyters
27
linkedtv.eu
Presentation
engine

41. How LinkedTV helps Bert and Anne
www.linkedtv.eu
Video Object and
word detection
Paintings by Jan Sluijters
Connection to
concepts
Selecton of related
concepts
Selection of related
content
“...painting of Jan Sluijters....”
dbpedia.org/resource/Jan_Sluyters
27
linkedtv.eu
Presentation
engine

42. How LinkedTV helps Bert and Anne
www.linkedtv.eu
Video Object and
word detection
Paintings by Jan Sluijters
Connection to
concepts
Selecton of related
concepts
Selection of related
content
“...painting of Jan Sluijters....”
dbpedia.org/resource/Jan_Sluyters
27
linkedtv.eu
Presentation
engine

43. How LinkedTV helps Bert and Anne
www.linkedtv.eu
Video Object and
word detection
Paintings by Jan Sluijters
Connection to
concepts
Selecton of related
concepts
Selection of related
content
“...painting of Jan Sluijters....”
dbpedia.org/resource/Jan_Sluyters
27
linkedtv.eu
Presentation
engine

44. From one screen...
www.linkedtv.eu
28
http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/
domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2

45. From one screen...
www.linkedtv.eu
28
http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/
domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2

46. From one screen...
www.linkedtv.eu
28
http://www3.springfieldwebtv.nl/view.html?path=/view.html&&presentation=/
domain/springfieldwebtv/user/linkedtv/collection/1/presentation/2

47. §
www.linkedtv.eu
29

48. ...to paper prototype...
www.linkedtv.eu
30
linkedtv.eu

49. www.linkedtv.eu
31

50. ...to second screen demo
www.linkedtv.eu
32
linkedtv.eu

51. How do we give broadcasters some control?
www.linkedtv.eu
Hypervideo
analysis
Paintings by Jan Sluijters
Generation of
media fragments
Named Entity
Recognition
Enriching NEs with
web content
Editor tool
“...painting of Jan Sluijters....”
dbpedia.org/resource/Jan_Sluyters
33
linkedtv.eu
Presentation
engine

52. [Useful tool example] editor tool
www.linkedtv.eu
§
§
§
Broadcasters want to have some control over noisy data.
Editors have the expertise to determine relevance +
The user interface can only show a limited amount of links, and there are many
§
Research: sample 805 videos, average 55 NEs per video. 55 NEs = many, many
possible links depending on the size of the white list [Li et al., 2013]
t1
w
Link
NE - person: Jan Sluijters
, Roeland Ordelman1
hite l
is
Link white li
Link white li
st 2
st 3
Link white list 4
NE – art syle: Fauvism
Link white list 5
whi
Link
NE - person: Emiel Aardewerk
te list
Link white li
Link white li
34
6
st 7
st etc etc

53. Editor tool overview – selecting entities and
enrichments
35
www.linkedtv.eu

54. Editor tool overview – selecting entities and
enrichments
36
www.linkedtv.eu

55. Editor tool overview – editor‘s choices
www.linkedtv.eu
37

56. Future work in LinkedTV
www.linkedtv.eu
The project ends in May 2015. The coming year, we’ll focus on:
•Improving noisy data from automatic enrichments
•
eg. through aggregating multiple entities and visual cues
•Multiple user studies and iterative development of front-end
functionalities and interface based on feedback
•Making enrichments more relevant for users by further refining
personalisation technologies
•Further Market studies and business model development
38

57. Online resources!
www.linkedtv.eu
39

58. Do stuff now!
Media Fragment Enricher
, Roeland Ordelman1
40
Eurecom’s tool: http://mfe.synote.org/mfe/
www.linkedtv.eu

59. Lets explore!
www.linkedtv.eu
multimedia
content
analysis
new distribution
platforms
lots of archive
content
41

60. Lets explore!
www.linkedtv.eu
multimedia
content
analysis
new distribution
platforms
where the magic
happens!
lots of archive
content
41

61. Thanks!
www.linkedtv.eu
Find me here:
@johanoomen| joomen@beeldengeluid.nl
@lottebelice lbbaltussen@beeldengeluid.nl
Credits:
§All LinkedTV partners
§Slide hat-tip to: Lynda Hardman (CWI), Lyndon Nixon (MODUL), Raphaël Troncy
, Roeland Ordelman1
(Eurecom), Bouke Huurnink (Sound and Vision)
§
§
§
LinkedTV overview: From SmartTV to LinkedTV. slideshare.net/linkedtv/linked-tv-introduction
The importance of Linked Media to the Future Web. slideshare.net/linkedtv/www-linked-mediakeynote
Describing media assets: media fragment specification and description. slideshare.net/troncy/
describing-media-assets-media-fragment-specification-and-description
§Heaps of thanks to AVRO for so generously letting LinkedTV use their content.
Links:
§www.linkedtv.eu. See demo’s, tools, research, scenario’s and more!
§twitter.com/BenGlabs
§www.avro.nl
42