Television Linked to the Web
Our vision of future Television Linked To The Web (LinkedTV) is of a ubiquitously online cloud of Networked Audio-Visual Content decoupled from place, device or source. Accessing audio-visual programming will be “TV” regardless whether it is seen on a TV set, smartphone, tablet or personal computing device, regardless of whether it is coming from a traditional or new media broadcaster, a Web video portal or a user-sourced media platform.
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
LinkedTV: Television Linked to the Web, June 2013
1. Television Linked To The Web
www.linkedtv.eu
Building the future of television
Lynda Hardman
Centrum Wiskunde & Informatica
2. 2
www.linkedtv.eu
Web and TV exist in parallel
86% of TV viewers surf the Web at the same time
(Yahoo!, USA)
88% of USA consumers use mobile as second
screen (Business Insider, May 2013)
Manual efforts to link TV program to additional
content (Shazam, IntoNow)
LinkedTV: interweaving Web and TV into a
single experience
Mixing lean back and lean forward
Want to maintain immersion while removing fourth
wall illusion
Works only for certain genres and audiences
Does second screen help or hinder?
The LinkedTV vision
linkedtv.eu
3. 3
www.linkedtv.eu
Scenarios and content
Enriched News
Professional news content
produced by RBB for general
audience
local news show rbb Aktuell
People, locations and events
can be linked to previous
events.
Antiques Edutainment
Program watched
predominantly by 50+ers
AVRO:
Tussen Kunst en Kitsch
Presenters, locations and
objects provide hooks to
related materials.
linkedtv.eu
5. 5
www.linkedtv.eu
dbpedia.org/resource/Jan_Sluyters
How will it work?
linkedtv.eu
Paintings by Jan Sluijters
“...schilderij van Jan Sluijters....”
Video object and
word detection
Presentation
engine
Selection of
related concepts
Selection of
related content
Connection to
concepts
6. 6
www.linkedtv.eu
Object detection in video
Face detection
Text analysis, keyword extraction
Multilingual automatic speech recognition
Combining results from different modality
analyses
Speaker identification
Object re-detection
Output in XML using OpenSource tool:
EXMARaLDA
http://www.exmaralda.org/
Concept Detection
linkedtv.eu
“...schilderij van Jan Sluijters....”
Video object and
word detection
7. 7
www.linkedtv.eu
Annotations represented in RDF
stored in LinkedTV platform
Use existing vocabularies
e.g. schema.org, NERD, LSCOM, DBpedia
Named Entity Recognition using
statistical & knowledge-based approaches
Enrichment based on textual and visual
analyses
Concept connections
linkedtv.eu
Connection to
concepts
dbpedia.org/resource/Jan_Sluyters
9. 9
www.linkedtv.eu
Additional information, e.g.
biography of artist
style of painting
Related information, e.g.
artists from same period
paintings in similar style
related styles
Concept selection
linkedtv.eu
Paintings by Jan Sluijters
Selection of
related concepts
Jan Sluijters
has art style
luminism
Leo Gestel
Piet Mondriaan
has art style
has art style
10. 10
www.linkedtv.eu
Selection based on context of scene,
e.g. system knows which topics appear
next
User preferences, e.g.
closely related information to
uninteresting topic is less useful
We need to understand user
information needs (per genre), e.g.
don’t miss important news because of
personalisation
Broadcasters don’t want users to drift
away from program
Content selection
linkedtv.eu
Selection of
related content
11. 11
www.linkedtv.eu
Example user information needs for news
LinkedTV Berlin, 2013
MORE IN DEPTH SPECIFIC INFORMATION
I was wondering just how expensive it is to make all those
vaccines, you know? I was just wondering if it is a good idea to
even make a lot of it or how hard it is to make.
LOCAL EFFECT OF GLOBAL NEWS – What is the impact
in my life?
Yes then I would look it up, what is the effect in the
Netherlands… (of the bird flu virus)
OVERVIEW OF PAST RELATED EVENTS
My search would be something like "South Korea state of war
ramp up” (…)
I would like to know, what happened two years ago and how
they solved it…
12. 12
www.linkedtv.eu
User information needs for news
LinkedTV Berlin, 2013
Information about how the story ends
Updates when important changes occur
More information about the people involved
Where is that location?
Consult related publications and documentaries
Opinions from different experts
Access related info selected by experts and suggested by broadcasters
13. 13
www.linkedtv.eu
Many users watch TV news during dinner!
Use second screen to
give user more control
retain sense of immersion via TV screen
Need to resolve issues of pausing when in
group situation
There is no single best solution
Content display
linkedtv.eu
Presentation
engine
14. 14
www.linkedtv.eu
How will it work?
linkedtv.eu
Paintings by Jan Sluijters
“...schilderij van Jan Sluijters....”
Video object and
word detection
Presentation
engine
Selection of
related concepts
Selection of
related content
Connection to
concepts
dbpedia.org/resource/Jan_Sluyters
16. 16
www.linkedtv.eu
LinkedTV platform & player
TV Set
LinkedTV
Server
• Video Analysis
• Enrichment
• Personalization
• Delivery
Browser
CE-HTML
Platform ClientExternal
Broadcast
HTTP
Browser
Variant
HbbTV
Variant
HTTP
Repository
• Annotations
• Links to
external
Resources
Editor
LoD
Cloud
Videos
from
Scenarios
Browser
HTML 5
PC / Tablet
Network
Material
selection,
Annotation
adjustment
A clearly defined and modularized
platform architecture, deployment
decisions made, initial REST API
An interactive video player based on
Web specifications (HTML5) realising
intuitive interaction & enrichment layers
linkedtv.eu
18. 18
www.linkedtv.eu
Individual video analysis, annotation, hyperlinking, UI/UX design and
personalisation components integrated into a LinkedTV platform with
end-to-end content workflow
Provide player for desktop, TV and second screen
Create interactive scenarios (rbbAktuell and Tussen Kunst en Kitsch) to
engage with users to further understand their interaction and information
needs
LinkedTV in 2013
linkedtv.eu
Not just TV, but interaction with online media (archives).
(1) Big Q: Do you ever have a question about what you are watching? Would you like related information about something you have seen? Something someone mentioned? One option is to pause the TV, go find your smart phone or pad and start some google search. When you carry out the search, your device doesn't know what you were watching, nor what the specific scene was about. Wouldn't it be nice to have a system that knows what you are watching, knows what you like and that presents links to extra pieces of information to you? Technology only likely to be useful when audience , content and program genre match. (2) slide
(1) Currently developing tools using 2 program genres
(1) What do we think it will look like? Not like this! Select video. Link to video!
Developing individual technologies, using content from 2 program genres, with the aim of creating an end2end prototype. I’ll go through each of these stages. Don’t explain anything about each part.
(1) Lower the cost of preparing seed video for enrichment with other content Objects are detected in the video, e.g. painting, face Words are also detected in the speech, or closed captions (2) Slide: Web aspects: using and cnotributing to W3C recommendations, much of the s/w open source
Linking object or word to concept (painting, face, painter ) or Named Entity (Jan_Sluijters, Piet_Mondrian ) in Linked Open Data cloud. Each circle is a data set, representing thousands or millions of concepts. Familiar with wikipedia? DBpedia is machine-readable form? RDF is the language used on the web for expressing concepts (similar to XML for document structure)
A number of tools are available, e.g. allowing the conversion of different sources of metadata to be converted to RDF, Named Entity Recognition.
Once we have the concepts & named entities in the Linked Open Data cloud, we can find other content (web pages or audiovisual material) by tracing paths through this cloud.. Go through the bullets click through the links Could also link to concepts or entities in other sets, such as birthplace in GeoNames.
Large numbers of related concepts can be found very quickly. Selection based on connections within video topics and user preferences Need to find suitable video/information sources for users. If suitable material is scarce then can increase complexity of search (length of path in linked open data cloud). Broadcasters need to be sure of high quality sources, e.g., wikipedia, Artcyclopedia Want structure that ensures that user "stays with the program" but has access to related information.
Team@CWI is investigating user info needs specific to narrow goals of LinkedTV. Working together with Beeld Geluid on this.
Skip if time short
Once we have chosen related content, we need to display it to the end user. Explore interaction modalities to browse the enrichment content. We need to understand how involved user wishes to be. Playout of enriched video on multiple devices and cross-device, e.g. second screen Providing requirements for first implementation of the presentation layer based on User Interface mock-ups of scenarios Carrying out user focus groups for second round requirements
I hope that this overview of the project makes more sense now!
Project is making underlying technology available. Daniel Ockeloen, Noterik, presenting later this afternoon. HTML5 for cross-device Web application development
In addition to the technology development, also collecting annotations of resources in project database.