Second screen, multi screen using XMPP. What is second screen, why is this popular both for the consumers and for the streaming companies? Why should you use XMPP for doing the second screen experience and why not DIAL or other protocols.
2. <STREAM> ABOUT ME
• Developing XMPP since 2007, MSc. Copenhagen Uni. - developed
pidginVOIP/Video plugin, XMPP for signalling
• Member of the XMPP Standard Foundation (XSF) since 2009 -
now also on the editorial team
• Developed software since 1998 for various companies: telcos,
streaming companies etc.
• Have my own consulting company (BrainTrust) - Developing
software and doing some startups!
3. RELATED PROJECTS
• I have been working with:
• Open source:Tigase server, components, Strophe.js plugins etc.
• Various chat applications, for video playback commentator, support tools etc.
• XMPP over satellites - standardised maritime satellite project (MSDS) for
maritime safety service (advanced group chat)
• XMPP in gaming and betting industry
• XMPP forTV presence for various telcos and cable operators in scandinavia
and Europe (first one in 2008!)
I’ve done
4. AGENDA
• Intro to Second Screen
• Brief intro to XMPP
• Second Screen in a XMPP environment
(no demo - sorry!)
• Beyond Second Screen
5. SECOND SCREEN?
Second screen, sometimes also referred to as “companion device” (or
“companion apps” when referring to software applications), is a term that refers
to an additional electronic device (e.g. tablet, smartphone) that allows a
content consumer to interact with the content they are consuming, such
as TV shows, movies, music, or video games. Extra data is displayed on a
portable device synchronised with the content being viewed on television.
-wikipedia
9. AIRPLAY
• Proprietary protocol (formerly known as AirTunes)
• Allows wireless streaming of audio, video and photos
• Troublesome license..Audio part can be licensed
• Screen casting
• Discover Airplay devices on same network
• Request device control such as trick play (play, pause, ff, rev, stop etc.)
• Streams can only be decrypted by using apple keys (no other DRM systems)
11. CHROMECAST
• An implementation of DIAL (netflix)
• DIscover And Launch - Simple! - Multicast for discover, HTTP/
REST to open and launch apps (urls).
• Not “locked” to any device, but can discovered on the same
network
• Gives you playback of Netflix,Youtube etc. on a simple cheap stick
• Supports screen casting from chrome etc.
12. CHROMECAST
• Libraries for IOS and Android only (no windows)
• Browser to browser support, JS (Chrome only)
• Supports Play Ready and Encrypted Media Extensions (DRM)
• Uses Smooth Streaming and MPEG-Dash - No HLS? WTF!
• Uses Google DNS.. :-(
• White listings of AppIds
13. SECOND SCREEN?
• An offer to interact through a secondary device than the one you are
using as primary screen
• Some call it:“Multi screen”,“Companion devices”,“Beaming”
• The secondary / companion device: smart phone, tablets and laptops
etc
• Primary:TV, set-top box, xbox, play station etc.
• Second screen for implemented: kiosk systems, remote controlling,
multimedia trickplay,“flick-to-screen”.. which I am going to talk about
16. TV IS DEAD, LONG LIVETV!
“The ways we access television content is changing and evolving, and the multitude of devices on
which we can viewTV is ever growing; withTV everywhere on your tablet, mobile or PC, set top
boxes and connectedTV’s. How do broadcasters and content providers target audiences
appropriately, and where should they be focussing their attentions?”
• Usage is different now than before the smart devices,
like iPhone,Android and tablets - the want to reach
the new audience!
!
• second screen for better experience. Some “Smart
TVs” and STB are horrible UI and slow on rendering.
!
• Make theTV/STB device a stream device, make the UI
on second screen
19. KNOWN PROTOCOLS
• DIAL - DIscovery And Launch protocol (Google chromecast) - 2012
• Apple AirPlay
• MQTT - weak security (why not use SASL?), pub/sub only in
application layer
• COAP (HTTP / REST based)
• APIs and REST? (many IoT companies does this!)
• Others
20. REQUIREMENTS
• Support for many devices!
• Routable over the Internet
• Secure with authorisation and authentication (because
it over the Internet!)
• Transport agnostic (diverse devices, software stacks etc)
• Do not want to reinvent the wheel!
21. REQUIREMENTS
• Easy Discovery & pairing - how do they find each other?
• Roles, authorisation and authentication (diff. devices and users)
• Single user or multi - let others do remote stuff.
• Control over the infrastructure (no dependencies to silos)
• Streaming media or signalling? - most want just signalling
• A protocol instead of an API
22. API’SVS
PROTOCOLS
•Many IoT vendors only
focus on API
!
•Via an open standard
protocol we would be
able to communicate a
cross vendors and
devices.
23. DISCOVERY
• Scan QR code, click download, wait, launch app =
tedious!
• Open web browser, type in URL
• OpenWRT, DDWrt and others: connect to SSID etc
• Have a protocol doing discovery and signalling for
you
24. • Roles - user login / auth or simple devices with
no login?
• provisioning / re-provisioning devices.
• Single user / Multiuser
• Do you owe all of the device stack???.. login vs.
kiosk system (no login, presentation/buying etc)
ROLESVS PROVISIONING
25. TO CAST ORTO MIRROR?
• Mirror: to mirror your display and to transcode it on the
second screen device
• Cast: to send play and trick play “commands” from the
second screen to first screen and let 1’st screen decode etc.
• Stream vs. signalling. Content owners have a hard time
accepting streaming (airplay mirroring.. transcending on the
sending device etc)
26. SECOND SCREEN ENVIRONMENTS
AND LIMITATIONS
• To be used on aTV or manyTVs!..
• Many people in a household!
• Some native, some are browser based! (only a small limited numbers are native of STB)..
a diversity of clients and hardware. need a protocol!
• DRM
• Easy to use.. people normally have a phone and have a special application installed where
they can watchTV and seVOD (video on demand) - no need for discovery of the app!
• Pairing should only be done if another customer wants to take control of yourTV
experience!
27. • An Open Standard! yay!
• Secure & proven over many
years! (many implementations,
open source / commercial)
• Many routing mechanisms
• Federated! (setup your own)
• Flexible - beyond Second
Screen!
28. • Transport agnostic
• TCP, BOSH - XMPP over HTTP, Websockets
• Diff. backends (many servers support native MySQL,
REST or own plugins) for login and authentication
• Easy to plugin your own business logic and extensions
• Many extensions! - for routing possibilities, and others!
29. • Started as a chat protocol - Google, Facebook,
WhatsApp etc.
• Also used in gaming (e.g chess, Bingo),
synchronisation software, white boarding and
collaboration software, taxi hailing etc.
• Now used in many other places!
30. WHY?
• Can route over the internet (you do not need to be home
to use your second-screen)
• Can be used locally (link local - bonjour) as well (instead of
client/server model)
• Versatile protocol - and not just yet another API!
• Open and can federate your data. Move it if you like to
your own domain and server
32. • Normal XMPP architecture involves a server
and a lot of clients.
• A user can login with several clients
• Servers can be federated pr. domain
• Server less mode - Link Local (Bonjour)
XMPP ARCHITECTURE
33. XMPP CONNECTION
LIFECYCLE
• Connect with your user and cred. or anonymous login (authorisation)
• Get resource from server (or given from client)
• Send initial presence and capabilities
• Get presence from your other clients if any
• Get roster
• Communicate with communication primitives (persistent connection)
34. XMPP ADDRESSING AND
ROUTING
• Domain routing like mail (SMTP)
• Many diff. mechanisms that can affect a route
• bare JID vs. full JID
• me@example.com vs me@example.com/Home
• extensions like: Pub/Sub, MUC, etc.
36. <STANZAS>
• You can define the payload it self.
• Make an extension!
• Define your own <tag> with your own name
space.
• Embed and use JSON containers or other as
payload. easier for browsers to understand
41. RELEVANT EXTENSIONS
• Event Logging over XMPP - XEP-0337
• IoT: Sensor Data
• IoT: Provisioning
• IoT: Control
• IoT: Concentrators
42. USING XMPP IN
SECOND SCREEN
• Almost all OTT (over-the-top) devices like smartTVs and
set-top boxes (STB) are browser based.
• We have STBs that support HTTP/1.0!
• Devices that connect nativelyTCP (like IOS,Android,
Microsoft etc)
• Customers can use the website to watchTV/Video as well..
that is also connected (via. websockets)
customer scenario
43. • You connect with. second screen (IOS):
• me@example.com/IOS
• Turn on (smart)TV or STB and it connects auto.
• me@example.com/TV1
• me@example.com/TV2
CONNECTION AND
DISCOVERY
45. USAGES
• Second Screen stuff (trick play etc) - sending play/stop,ff commands!
• Capabilities and service discovery pr. device
• Use presence for watching the number of concurrent clients and
throttle it (DRM purposes)
• Use presence for watching user interaction (channel changes,VOD,
follow me etc)
• Push notifications via. full JID, bare JID, Pub/Sub (groups) and broadcasts
to all
46. • Remote logging / control the device (customer
support)
• Let your devices talk together in many topologies -
one-to-one, many-to-one , many-to-many
• Remote recordings on local storage (PVR)
• Exchange bandwidth between devices
BEYOND SECOND SCREEN
47. BEYOND SECOND SCREEN
• Instant Context Switching
• Instant statistics for online devices (second screen as well asTV etc)
• InterconnectedTV with your IoT home! (GW for MQTT, zigbee
etc)
• Connect with your alarm, turn off theTV etc.
• The living room devices gives away a lot of data for
recommendation
48. XMPP AND IOT
• We can connect a lot of diverse devices. No matter transport
• Create your own gateways if you want to connect to other APIs or
protocols
• I’ve created a Pub/Sub to MQTT gateway to connect to MQTT
connected devices
• WebRTC - federation and discovery
• UPnP cloud - uses XMPP standard for interconnecting devices
• Usage stats! - delivery of a huge amount of real-time data == BIG DATA!
49. UPNP CLOUD
“Cloud support in the UPnP architecture will enable
IoT and internet-wide usage by allowing IP-based
devices and servers to discover each other and
communicate independent of the network they
are connected with.”
50. QUESTIONS? </STREAM>
Email / XMPP:
slarsen@braintrust.dk
Twitter: @zooldk thanks to t.christensen from nordija
for inspiration to the slides.
