LinkedTV project results at the end of year 2

1. Television Linked To The Web
LinkedTV results at
the end of the 2nd year
Lyndon Nixon, MODUL University
27/28 November 2013, 2nd Project Review, Hilversum
www.linkedtv.eu

2. Overview
www.linkedtv.eu
- What is special about LinkedTV?
- Scenarios at end of year 2
- How LinkedTV works (at M24)
- Thanks
2

3. What is special about LinkedTV?
www.linkedtv.eu
 There is a trend towards
„interactive images“ on the Web
 There is a trend towards
„interactive video“ (or „hypervideo“)
on the Web
ThingLink can
embed clickable
images in
Facebook
WireWax is
sharing clickable
video on YouTube
 We develop a new solution for
interactive media in a TV-integrated
environment: „LinkedTV“
„... a true TV ecosystem must functionally integrate the apps with the
television programming service and content being offered to the consumer.“
TVs can't be smart. Stop trying to make it happen. - WIRED Opinion, Gary Myer, Oct 2013
3

4. What is special about LinkedTV?
www.linkedtv.eu
Build upon Web and broadcast specifications and standards
Make sure LinkedTV is part of the whole industry, not another proprietary fragment
HTML5, Linked Data, W3C Media, hbbTV, Open IPTV Forum
Reuse the rich wealth of data and services on the open Web
Give added value at low cost
Extendible to closed data and services for specialised offers
Automate the annotation and enrichment as far as possible
Broadcasters barrier to entry is the complexity and effort to make hypervideo
Current approaches will never scale to the amount of video being produced
4

5. Scenarios at end of year 2
www.linkedtv.eu
Scenario 1:
Interactive News
Show
 Professional news
content produced
by RBB
 Seed content: local
news show "rbb
Aktuell"
5
Scenario 2:
Hyperlinked Documentary
 Cultural heritage
content from S&V
 Seed content:
"Tussen Kunst &
Kitsch" from AVRO
(Antiques
Roadshow format)
Scenario 3:
Media Arts
 MEDIADROM
competition for
defining new ideas
for linked television
 Concepts will be
promoted at Mons:
European Capital of
Culture 2015.

6. Scenarios at end of year 2
www.linkedtv.eu
Scenarios have been realised on top
of the full end to end workflow of the
LinkedTV Platform
Main and companion screen
experience with the LinkedTV Player
Video demos are online
6

7. How LinkedTV works (at M24)
www.linkedtv.eu
Video object and
text detection
Examples of Paintings by Jan Sluijters
… and their digital images
Connection to
concepts
Expansion of
related concepts
Linking to
related content
“...schilderij van Jan Sluijters....”
dbpedia.org/resource/Jan_Sluyters
7
Personalisation
engine

8. Concept detection
www.linkedtv.eu
Video object and
text detection
LinkedTV works on services using &
combining different forms of audiovisual
content analysis:
– Face detection
– Textual analysis (subtitles,
transcripts)
– Keyword extraction
– Multilingual automatic speech
recognition
– Speaker identification
– Concept detection in video
– Object re-detection
– Shot segmentation
This provides us with identification of
“...schilderij van Jan Sluijters....” meaningful media fragments and
summaries of their content analysis
8

9. Concept connections
www.linkedtv.eu
Connection to
concepts
Keywords and text fragments can be connected to
uniquely identified, distinct named entities
using online entity extraction services (NERD,
http://nerd.eurecom.fr)
– Media annotations are represented in RDF
and stored in LinkedTV platform
– Re-use of existing Web vocabularies
e.g. schema.org, NERD, LSCOM, DBpedia
– Named Entity Recognition uses
statistical & knowledge-based approaches
(aiming at entity meaning disambiguation)
– Extended by complementary visual
analyses (e.g. a persons name in text with
a face detected in a video fragment)
dbpedia.org/resource/Jan_Sluyters
9

10. Concept expansion
www.linkedtv.eu
Paintings by Jan Sluijters
Expansion of
related concepts
10
The use of Linked Data in the identification
of concepts in LinkedTV means we can
expand concepts along different facets, i.e.
allow users to explore in terms of their
different interests in a given concept.
For example, for an artist like Jan Sluijters,
LinkedTV can link into:
 Additional information, e.g.
 biography of artist
Leo Gestel
has art style
 style of painting
 Related information, e.g.
luminism
has art style
has art style
 artists from same
Piet Mondriaan
Jan Sluijters
period
 paintings in similar style
 related styles

11. Linking to related content
www.linkedtv.eu
… and their digital images
Linking to
related content
11
LinkedTV provides enrichment services
which provide recommendations for
related online content (Web pages,
images, audio, video) pertaining to the
concepts in the TV program:
– Fresh Social Web content coming
from e.g. Twitter and Facebook
– User Generated content coming
from e.g. Flickr and YouTube
– Whitelist content coming from
partner Websites like public
broadcasters in Germany or cultural
heritage archives in the Netherlands
– Extendable and configurable by
source and media type

12. Personalisation engine
www.linkedtv.eu
Personalisation
engine
12
The LinkedTV Platform pushes annotation and
enrichment information to the client (LinkedTV
Player)
– Both annotations and enrichments include
further metadata about the topics they relate
to, e.g. Jan Sluijters + Amsterdam, BER
Airport + Berlin
– A Personalisation engine on the Player
filters & ranks the annotations and
enrichments according to a locally stored
user model, e.g. interested in Amsterdam vs
interested in Berlin
– The user model can be both manually edited
online (LUME) or learnt by the LinkedTV
system (attention tracker, event tracking)
– All personalisation is opt in and private,
respecting ethical issues

13. Packaged into the LinkedTV Platform
www.linkedtv.eu
The LinkedTV Platform
encapsulates the
functionality of the
previous 4 steps.
The main core of the
platform is an
Application Service Bus
which monitors and
triggers events across
the components,
realising the end to end
workflow from a video
ingestion to enriched
playout.
Communication between platform components and between platform and
client is realised using RESTful APIs.
13

14. Playout on the LinkedTV Player
www.linkedtv.eu
LinkedTV enriched video playout needs a UI for Linked Television
 The Springfield multiscreen platform abstracts the video and its
enrichments from the device where they are displayed
 Viewers can browse the topics of the program
along „Who“, „Where“, „What“ axes
 Enrichments can be browsed to, or shared,
or bookmarked for later
Current year 2 interface
14
Future user interface mockups

15. Thanks to...
www.linkedtv.eu
15