Today, media material is increasingly digital and shifting to delivery via IP and the Web,
including cultural artifacts or broadcast television. This opens up the possibility of new
services deriving added value from such material by combining it with other material
elsewhere on the Web which is related to it or enhances it in a meaningful way, to the
benefit of the owner of the original content, the providers of the content enhancing it and
the end consumer who can access and interact with these new services. Since the
services are built around providing new experiences through connecting different related
media together, we consider such services to be Connected Media Experiences
(ConnectME).
In particular, much rich depth of information and service functionality associated to
content in video is not derived today due to a lack of suitably granular description of
video, including linking of video objects to the concepts they represent. For example,
news reports about local tourism and events are not linked to tourist information and
event–related services which a viewer may (spontaneously) wish to access and make
use of.
The technological result of the ConnectME project will be an end-to-end service platform
to host those added-value services over different networks, providing the common
required functionality of each service: multimedia annotation and subsequent enrichment
with related content from the Web, combined with the packaging and delivery of
synchronized multimedia presentations to the end device. At the device, intuitive user
interfaces must be developed so that the selection of on-screen objects and the
browsing of the associated content can be done in a non-disruptive and intuitive fashion.
As a result, ConnectME facilitates a new interactive media experience built on top of the
convergence of TV/video and the Web.
Dev Dives: Streamline document processing with UiPath Studio Web
ConnectME: connecting content for future TV & video
1. ConnectME: connecting content
for future online TV & video
Lyndon Nixon
lyndon.nixon@modul.ac.at
Last update: September 25,2013
More at www.connectme.at
Contact : info@connectme.at
2. Connected Media Experiences
• New services deriving added value for video by
combining it with complementary content from
the Web
• Linking of video objects to the concepts they
represent, and hence related content about
those concepts
• Provisioned as an enhanced in-video experience
• Delivered via an end-to-end service platform
hosting the added value services over different
networks
3. Growth of Internet on the TV
Below: Boxee, ConnectedTV, Samsung Smart TV (http://thenextweb.com/gadgets/files/2011/05/5msamsung-apps-1.jpeg)
5. Growth of mobile video
Video streaming accounts
for 37% of all mobile traffic
Of all video streaming
traffic, YouTube accounts
for 45%
A Cisco study* on mobile
traffic growth expects
• 66% of all traffic by 2014
will be video
• having increased 66-fold
from 2009 to 2014
* Cisco‘s Visual Networking Index Global Mobile Data Forecast
2010-2015
9. What could be a
connected media experience?
Where is Obertauern?
Which peak is that
Can I ski there? Ho
What is the buildin
If it is a hotel, how
the accommodat
11. The business case for hypervideo
• Offering an added value service
– Solely to win more visitors & keep them using the
service
– Encourage upgrades to premium packages (by
content provider, by end consumer)
• Monetarization of the links
– E-commerce (drive traffic to electronic purchases)
– Content partnerships (drive traffic to others‘ content)
– Ad inserts
12. The technical case for
connected media
• Channel annotation, enrichment and delivery
are all services of the ConnectME framework
13. Content workflow
• Upload video in partner CMS
• Fill in basic information and „commit“ video to
the ConnectME Framework (CMF)
• CMF includes video instance and preannotates it from the basic information and
video analysis components
• Annotation tool provides preview of preannotation and means to finalise annotation
• CMF handles links to content enrichments
• Preview in ConnectME player with widgets
rendering different content enrichments
14. Relationship Linked media framework (LMF) <->
Connected media framework (CMF)
• CMF wraps the LMF reusing its inherent media & metadata
handling functionality and realising the ConnectME workflow
on top.
Analysis, annotation
Visual analysis
LMF
Speech to text
Workflow
Linked
Media
Store
Concept
extraction
REGISTRATION
Interlinking, enrichment
Annotation
views
ConnectME ontology based
Linked Media
engine
rul
es
LOD
REST
Annotation
tool
Interlinked media generation
Presentation
engine
Enrichment
playout
Presentation, playout
REST
AUTENTICATION
Hypervideo
browser
15. Channel annotation
• Grouping of videos into a „channel“
• Easy to use tools for content owners or hosts
to annotate video segments with links to
semantic concepts
– Manual annotation tool
– Supported by
• automatic classifiers over visual features, audio,
subtitling/transcripts, or other annotations incl.
crowdsourced tagging
• tool tips for choosing concepts from pre-selected
sources
16. Yoovis use case: ConnectME
channel
Courtesy Yoovis GmbH, www.yoovis.tv
18. Pre-annotation in Yoovis use case
• Yoovis CMS calls a Web service on CMF which
accepts the mediaRSS format
– „registers“ a video instance in the CMF
– Includes video metadata directly extractable from
the mediaRSS, e.g. locator(s)
– Generates video annotations from title,
description and tags from the author
• Attachs these „pre-annotations“ as 'dct:subjects' in the
video metadata with local Yoovis terms, mapped where
possible to LOD such as DBPedia instances
19. Pre-annotation in the PS Media
use case
• PS Media CMS pushes to CMF a RDF
description generated in Drupal
– RDF Module maps Drupal node data to RDF
– LMF Module pushes this RDF into our server
– Generates video annotations from title,
description and subtitles from the EPG
• Attachs the „pre-annotations“ as in Yoovis use case
• Entity extraction from subtitles can associate entities
already to a temporal segment (store as annotation)
20. Pre-annotation (generic)
• Any client could directly call a Web service on
CMF which takes the parameters title,
description, URL & subtitles
– „registers“ a video instance in the CMF
– Includes locator(s) from URL parameter
– Generates video annotations from title,
description and subtitles
• Attachs the „pre-annotations“ as in Yoovis use case
• Entity extraction from subtitles can associate entities
already to a temporal segment (store as annotation)
21. Annotation suite
- HTML 5, CSS 3,
responsive layout, based
on Twitter Bootstrap
- jQuery, JS based UI
- Timeline based interface
- Allows adding/editing/
deleting annotations in/to
the CMF
- Allows registering videos
to and loading video
annotations from the CMF
- Multilanguage support
(currently EN, DE)
- Access via OpenID
22. Annotation suite
- Annotations are RDF and
concepts are using LOD
identifiers
- Annotation export in N3,
RDF/XML and Turtle
- Manual annotation with LOD
concepts (URIs) and HTML
content (URLs)
- Currently Plugins for
dbpedia and geonames
search APIs
- Plugin based architecture
allows adding further LOD
sources easily
23. Annotation model
published at www.connectme.at/ontology
Re-uses a subset of W3C
Media Ontology and
Open Annotation Model
Extends
mao:hasKeyword
to subproperties of
explicitly/implicitly
seeing/mentioning
concepts.
Pre-annotations attached
to video using
dct:subject and
skos:Concepts
cmo:preferredLabel
overrides concept labels
cmo:hasContent points
to static Web content
26. Annotation via the CMF
• Video metadata is turned into ConnectME
annotations on the CMF
– Ingestion services generate top level metadata
such as video title, description, author, license...
• Analysis of textual metadata to extract
keywords a task of the Apache Stanbol
Enhancer
– A generic enrichment chain for entities->DBPedia
– Specific enrichment chains for Yoovis and PS
Media (entities->keywords(->LOD))
27. Enrichment via the CMF
• Each concept needs to be linked to related
content items
– The player widgets could query this directly but
– The CMF can cache relevant links so that the
player can make simple queries for content
• Common data model for links regardless of
source, type etc.
– See the Media Ontology for Linked Media
28. Direct & indirect links
Direct links in the concept metadata to related
content can be transformed to the common
model by extending CMF reasoning rules
(„Annotation views“)
($c http://xmlns.com/foaf/0.1/depiction $m) -> ($m
http://connectme.at/ontology#explicitlyShows $c)
- Indirect links must be derived for types of concept
using other Web based content sources (APIs)
29. Linked Services Infrastructure
• http://production.sti2.org/lsi
– Web interface and REST API
– Input: a LOD URI, output: RDF for matching media
resources
30. Linked Services Infrastructure
Middleware to different Web APIs (YouTube, Flickr, Ookaboo, Foursquare
Venues, Instagram)
» „lowers“ the input concept to query parameters such as label, long/lat ...
» „lifts“ the API response to RDF links for the CMF, where a content item is
described and connected to the original concept by ma:hasKeyword
LMF
API query
LSI module
RDF answer
Linked
Media
Store
Linked Services
Infrastructure
New module
To-beinterpreted
answer
(wrapped)
query
„lifting“ and „lowering“
handled by LSI
Some online
content source
31. Linked Services Infrastructure
RDF response format based on
W3C Media Ontology
Types are e.g. Image, VideoTrack
Locator points to the actual
resource URL
hasKeyword links the resource to
LOD URIs related to the resource
We can include further metadata
as provided from the Web API
(e.g. when uploaded, no of
views, duration, size...)
32. Channel enrichment
• ConnectME player with optional styling
(customer branding)
• Pre-selection of content sources and
presentation (widgets) by administrator
(administration interface)
• On-the-fly video enrichment by selected
widgets at play-out (user interface)
• Administrator access to player analytics –
playback, click-through, in-widget actions
33. Hypervideo playout & browsing
Based on open source
LIME player project
HTML5 & CSS3 based
Video format support as
per browser support
- MPEG-4 (IE, Safari,
Android)
- WebM and OGV
(Firefox, Chrome,
Android, Opera)
Production player http://player.connectme.at
Development player http://tkurz.github.io/lime
Software project http://github.com/tkurz/lime (Apache 2.0 license)
34. Player implementations
• Desktop
– Administrative & user interface
– Widgets as extra webpage
regions („standard“ player)
– Fullscreen (with widgets as
overlays)
• TV
– GoogleTV (fullscreen, no video
pause, supports remote control)
• Mobile
– Tablet version (fullscreen
adapted to touch, min 7“
screen)
„The“ player incorporates both
standard and fullscreen impl.
35. Player widget model
Standard widget model
is tiles-based* and abstracts the
Interaction model from
• Desktop: point and click
• Mobile: touch
• TV: remote control
* A tile is 300x300px and a typical
widget is max 600x600px
37. „Extended“ player widgets
Extended widget model
needs definition of:
- Annotation trigger (concept type)
- Content to display (query CMF)
- How to display it (tiling)
38. „Live TV“ player widgets
PS Media use case
defines these additional widgets
- Actor
- Character
- Director
39. Demos
Production workflow has demos of ConnectME,
at http://player.connectme.at
IPTV example:
Choose „Fahrenheit 9/11“ and TV player
Mobile tourism example:
Choose „Schladming“ and tablet player
Editor's Notes
WireWax on iPad video: http://www.youtube.com/watch?v=KmILrnoc8bk
Dev CMF: http://connectme.salzburgresearch.at:8080/CMF
Prod CMF: http://connectme.salzburgresearch.at/CMF
ConnectME videos: Login to http://www.yoovis.tv/administrator/video/as connectme@yoovis.tv - password: connectme. Currently there are 18 videos.
CURRENTLY NEITHER PLAY (IN PLAYER) OR ANNOTATE (IN ANNOTATION TOOL) WORKS
A sample mediaRSS file:
What can be created in CMF as video metadata:
(NOTES: we should be able to include video creator, owner, license...)
A sample TV Anytime file:
What can be created in CMF as video metadata:
API: http://connectme.salzburgresearch.at/CMF/video/resource?url=
Doc: see http://connectme.salzburgresearch.at/CMF/connectme/doc/rest/video/resource/url/index.html
accepts video url, title, description, keywords
HOW TO INCLUDE MULTIPLE LOCATORS
HOW TO INCLUDE SUBTITLES FILE
HOW TO INCLUDE OTHER METADATA, e.g. owner, creator, license
Getting started:
Open video → get a list of videos from the CMF the tool is configured to use
„Enter a video URL“ option from „Open video“ also uses the Web service to add a new video instance to the CMF
From Yoovis, the tool is opened directly using a special URL, e.g.
http://annotator.connectme.at/yoovis.php?channel=1802&video=7112&partner=yoovis&user=519
From PS Media, TBD
If video is pre-annotated, these are loaded into the add new annotation interface
If video is annotated, these are loaded into the timeline interface
Annotation interface:
- enter a concept or thing uses DBPedia
- enter a place uses GeoNames
- choose a suggestion uses pre-annotations (if any)
Published at http://connectme.at/ontology
TO BE CORRECTED; USING mao:locator
Both tools are STABLE and production is FROZEN
CMF can be configured to use different Stanbol enrichment chains
WHAT IS IMPL IN PRODUCTION AND WHAT IS IMPL IN DEVELOPMENT
CURRENTLY
- for ingestion via Web service
the default is DBPedia entity extraction using a local DBPedia dump (subset) from the title, descriptions and keywords
Same for subtitle files, but generates annotations NOT pre-annotations
- for ingestion from Yoovis CMS (mediaRSS)
The default is keyword extraction using a local specific taxonomy of SKOS concepts
(http://connectme.salzburgresearch.at/CMF/skos/index.html, open http://yoovis.tv/resource/context)
Yoovis taxonomy looks completely flat, intended? Where are the mappings to LOD?
OPTIONALLY
I still need to see evidence of use of Stanbol giving us the best (pre-)annotations
Online NER services (like NERD) could be used, NERD also supports subtitle files (SRT)
AFAIK no annotation views are programmed in any CMF
(widgets encapsulate this functionality if required by SPARQL querying cached LOD)
LSI integration has been made in the DEVELOPMENT CMF?