If the future Web will be able to fully leverage the scale and quality of online media, a Web scale layer of structured, interlinked media annotations is needed, which we will call Linked Media, inspired by the Linked Data movement for making structured, interlinked descriptions of resources better available online. Mobile and tablet devices, as well as connected TVs, introduce novel application domains that will benefit from broad understanding and acceptance of Linked Media standards. In this talk, I will provide an overview of current practices and specification efforts in the domain of video and Web content integration, drawing from the LinkedTV and MediaMixer projects. From this, I will present a vision for a Linked Media layer on the future Web will can empower new media-centric applications in a world of ubiquitous online multimedia.

1. The importance of Linked Media to theThe importance of Linked Media to the
Future WebFuture Web
Lyndon NixonLyndon Nixon
lyndon.nixon@sti2.orglyndon.nixon@sti2.org
@lyndonjbnixon@lyndonjbnixon

2. The Future Web?The Future Web?
Our first assumption is that the FutureOur first assumption is that the Future
Media Internet will involve:Media Internet will involve:
•
Ubiquituous online, high bandwidthUbiquituous online, high bandwidth
access via multiple devicesaccess via multiple devices
•
All our media is digital and stored in andAll our media is digital and stored in and
accessed from the cloudaccessed from the cloud
•
Media access is seamless and abstractedMedia access is seamless and abstracted
from its sourcefrom its source
•
Consumers expect interactivity in andConsumers expect interactivity in and
integration of mediaintegration of media
Image from www.sics.se/efraim used without permission.
The EFRAIM project aims to „identify and develop distributed network
solutions for media distribution“

3. Tom sees a street posterTom sees a street poster
•
An app with text recognition capability extracts theAn app with text recognition capability extracts the
artist name and concert location and via an eventartist name and concert location and via an event
calendar can determine the event date.calendar can determine the event date.
•
Tom sees video clips of the artist's last concert.Tom sees video clips of the artist's last concert.
•
A map is overlaid showing the location of theA map is overlaid showing the location of the
advertised concert. Photos show Tom that inside itadvertised concert. Photos show Tom that inside it
has a nice industrial architecture.has a nice industrial architecture.
•
It's a future event, so the ticket can still be boughtIt's a future event, so the ticket can still be bought
online.online.
•
As he leaves, he can stream audio of the artist'sAs he leaves, he can stream audio of the artist's
songs.songs.

4. Artist Venue
Event
Video (fragments)
Event Map
Photos
Audio
Ticketing
service
Date Time
Image (Poster)
Spatial regions
earlier than

5. Is it possible today?Is it possible today?
•
Concept identifiersConcept identifiers
•
Web APIs using labels (strings) OR internal identifiers, Scarce sharedWeb APIs using labels (strings) OR internal identifiers, Scarce shared
identifiers across data and media sourcesidentifiers across data and media sources
•
Heterogeneous APIsHeterogeneous APIs
•
No machine processable descriptions of online APIs, Programmers needNo machine processable descriptions of online APIs, Programmers need
to define API calls, can't be made on the flyto define API calls, can't be made on the fly
•
Media assets are hidden from computer agentsMedia assets are hidden from computer agents
•
Media descriptions non-existent / limited / non-standardised in bothMedia descriptions non-existent / limited / non-standardised in both
data model and vocabulary used, not re-using our concept identifiersdata model and vocabulary used, not re-using our concept identifiers

6. Linked Media PrinciplesLinked Media Principles
1.1. Web media descriptions need a common representation ofWeb media descriptions need a common representation of
media structuremedia structure
2.2. Web media descriptions need a common representation ofWeb media descriptions need a common representation of
mediamedia contentcontent
3.3. Web media descriptions need to use a media ontology whichWeb media descriptions need to use a media ontology which
supportssupports description of both the structure and content of mediadescription of both the structure and content of media
4.4. The descriptions of media in terms of common representationsThe descriptions of media in terms of common representations
ofof structure and content are the basis for deriving links acrossstructure and content are the basis for deriving links across
media on the Web (Linked Media)media on the Web (Linked Media)

7. State of the ResearchState of the Research

8. Media descriptionMedia description
•
Metadata's main purpose for media isMetadata's main purpose for media is
indexing and retrievalindexing and retrieval
•
Descriptive metadata for mediaDescriptive metadata for media
characteristicscharacteristics
•
Provenance metadata to credit source andProvenance metadata to credit source and
specify rightsspecify rights
•
Conceptual metadata to reflect what mediaConceptual metadata to reflect what media
is perceived to representis perceived to represent

9. Media description modelsMedia description models
•
Highly heterogeneous: varying coverageHighly heterogeneous: varying coverage
depending on originating communitydepending on originating community
•
W3C Media OntologyW3C Media Ontology defines a shareddefines a shared
subset of all metadata properties withsubset of all metadata properties with
(rough) mappings(rough) mappings
•
More a basis for metadataMore a basis for metadata
interoperability than a metadata schemainteroperability than a metadata schema
to be used directlyto be used directly

10. Conceptual representation of mediaConceptual representation of media
•
Conceptual annotation is limited in media metadataConceptual annotation is limited in media metadata
•
How much was the Semantics DS used in MPEG-7?How much was the Semantics DS used in MPEG-7?
•
Typically whole media items are annotated with labels (strings) or aTypically whole media items are annotated with labels (strings) or a
local controlled vocabulary (examples from EPG data: title, description,local controlled vocabulary (examples from EPG data: title, description,
tags, categories, genres)tags, categories, genres)
•
Media Ontology has only the „keyword“ property to connect media toMedia Ontology has only the „keyword“ property to connect media to
concepts that it describes & does not enforce any means to identify theconcepts that it describes & does not enforce any means to identify the
concepts referred toconcepts referred to

11. Concept spaceConcept space
•
Globally unambiguous identifiers forGlobally unambiguous identifiers for
any concept neededany concept needed
•
Ideally more information about theIdeally more information about the
concept itself is available via theconcept itself is available via the
identifieridentifier
•
Use of Web URIs →Use of Web URIs →Linked DataLinked Data
concept spaceconcept space
•
Would allow for inference of conceptWould allow for inference of concept
type and relationship to othertype and relationship to other
conceptsconcepts

12. Media fragmentsMedia fragments
•
The subject of media description isThe subject of media description is
typically the whole media assettypically the whole media asset
•
Parts of media can represent distinctParts of media can represent distinct
concepts, but how to refer to them?concepts, but how to refer to them?
•
MPEG-7 provided for mediaMPEG-7 provided for media
segmentation but wasn't well definedsegmentation but wasn't well defined
(how does an external application refer(how does an external application refer
to a media segment?)to a media segment?)
•
W3C Media Fragment URI SpecificationW3C Media Fragment URI Specification
defines an agreed syntax for spatial anddefines an agreed syntax for spatial and
temporal parts of audio-visualtemporal parts of audio-visual
materials on the Webmaterials on the Web

13. Linked Media ontologiesLinked Media ontologies
Annotation
Concept
Keyword
BBC Ontology +
SchemaDotOrgTV
ANALYSIS RESULTS (Support for segmentation)
Ontology for
Media Resources
(W3C)
Ontology for Media
Resources (W3C)
Open Annotation
Core Data Model
EXTERNAL DATASETS
Entity
NERD
Provenance
Ontology for
Provenance
Management
Programme
Brand
Series
Episode
Version Broadcast
ServiceBroadcast Channel
Scene
Shot
MediaFragment
Face
http://data.linkedtv.eu/ontologies/core http://www.connectme.at/ontology

14. Media analysisMedia analysis
•
Video analysis can be used to detectVideo analysis can be used to detect
distinct shots or objects in the videodistinct shots or objects in the video
•
Combining other feature analysisCombining other feature analysis
can help improve detectioncan help improve detection
results (results (hybrid approacheshybrid approaches))
•
Defining what constitutes aDefining what constitutes a
„conceptually“ distinct fragment„conceptually“ distinct fragment
is a challenging taskis a challenging task

15. Entity recognitionEntity recognition
•
Metadata in textual form can beMetadata in textual form can be
used in extracting distinct namedused in extracting distinct named
entities, cf.entities, cf. NERDNERD
•
Subtitles, transcripts, OCR,Subtitles, transcripts, OCR,
speech to text...speech to text...
•
Entity disambiguation is anEntity disambiguation is an
important contributionimportant contribution
Screenshot of NERD http://nerd.eurecom.fr

16. Media metadata publicationMedia metadata publication
•
We need to be able to publishWe need to be able to publish
Linked Media following the sameLinked Media following the same
principles as Linked Dataprinciples as Linked Data
•
The media asset has a globallyThe media asset has a globally
(Web-wide) unique identifier(Web-wide) unique identifier
•
Metadata about the media assetMetadata about the media asset
can be accessed via its identifiercan be accessed via its identifier
•
Media identifer can't be = mediaMedia identifer can't be = media
locator !locator !
TV Set
LinkedTV
Server
• Video Analysis
• Enrichment
• Personalization
• Delivery
Browser
CE-HTML
Platform ClientExternal
Broadcast
HTTP
Browser
Variant
HbbTV
Variant
HTTP
Repository
• Annotations
• Links to
external
Resources
Editor
LoD
Cloud
Videos
from
Scenarios
Browser
HTML 5
PC / Tablet
Network
Material
selection,
Annotation
adjustment
LinkedTV Platform & REST API
http://data.linkedtv.eu

17. DerivingDeriving LinkedLinked MediaMedia
•
If different media assets share onlineIf different media assets share online
descriptions of their conceptualdescriptions of their conceptual
representations using Linked Data...representations using Linked Data...
•
Collecting media from differentCollecting media from different
sources which represent the samesources which represent the same
concept is only the first stepconcept is only the first step
•
Link media together in differentLink media together in different
relationships based on therelationships based on the
conceptual relationship between theconceptual relationship between the
concepts they representconcepts they represent

18. Linked Media presentationLinked Media presentation
•
Relationships between media couldRelationships between media could
(should?) be reflected in the presentation(should?) be reflected in the presentation
layer of the media applicationlayer of the media application
•
Spatial (map) and temporal (timeline)Spatial (map) and temporal (timeline)
presentation is well knownpresentation is well known
•
WhichWhich conceptualconceptual relations could berelations could be
visualised and how?visualised and how?
•
Would it be intuitive to the mediaWould it be intuitive to the media
application user?application user?

19. Where do we go from here?Where do we go from here?
We need more online media descriptions which reflect Linked Media principlesWe need more online media descriptions which reflect Linked Media principles
•
Consider meaningfully distinct fragments of atomic media assetsConsider meaningfully distinct fragments of atomic media assets
•
Describe what a person perceives in the fragment using (Linked Data) conceptsDescribe what a person perceives in the fragment using (Linked Data) concepts
•
Use more descriptive properties to reflect how they represent a conceptUse more descriptive properties to reflect how they represent a concept
•
Publish the descriptions in a structured & semantic modelPublish the descriptions in a structured & semantic model
•
Follow Linked Data principles for applications to access and process media metadataFollow Linked Data principles for applications to access and process media metadata
Generate new links between online media based on related conceptual descriptionsGenerate new links between online media based on related conceptual descriptions
•
Come up with presentation ideas to visualise intuitively the links between mediaCome up with presentation ideas to visualise intuitively the links between media

20. LinksLinks
LinkedTV Project websiteLinkedTV Project website http://www.linkedtv.euhttp://www.linkedtv.eu
MediaMixer Project communityMediaMixer Project community http://community.mediamixer.euhttp://community.mediamixer.eu
ConnectME Annotation tool:ConnectME Annotation tool: http://annotator.connectme.athttp://annotator.connectme.at
ConnectME Hypervideo Player:ConnectME Hypervideo Player: http://player.connectme.athttp://player.connectme.at
CMF (ConnectME Framework):CMF (ConnectME Framework): http://connectme.salzburgresearch.athttp://connectme.salzburgresearch.at
LSI (Linked Services Infrastructure):LSI (Linked Services Infrastructure): http://production.sti2.org/lsihttp://production.sti2.org/lsi