Presentation of the research activities of the GPAC team of Telecom ParisTech during the plenary session of the "Réseau Thématique 4" of the Mines-Telecom Institute

1. GPAC Team
Séminaire RT4
Cyril Concolato
08/04/2013
Institut Mines-Télécom

2. People
C. Concolato J.-C. Dufourd J. Le Feuvre J.-C. Moissinac
 + PhD Students, Post-Docs, S/W Engineers
• N. Bouzakaria, R. Bouazizi, V. Gros, M. Sarkis
• J. Gorin, A. Shafiei, V. Nguyen
• & many former students/engineers
 Part of the TPT / TSI / Multimedia Group
2 Institut Mines-Télécom

3. Activities
 Current Research Themes
• Multimedia Adaptation and Multimodal Interfaces
• Multimedia Transport and Delivery
• Multimedia Services and Transmedia
 Teaching
• Multimedia Courses (Master, Mastère, Lifelong
learning)
 Our specificities
• Open Source Software: GPAC
• Standardization: MPEG, W3C, …
3 Institut Mines-Télécom

4. Current Research Projects
 ANR HybRadio
 FUI Austral
 FUI 4Ever
 FUI OptisaT2
 FSN M44G
 ANR ILOT
 EU COLTRAM
 EU KIC ICT ANSFM
 Industrial partnerships
4 Institut Mines-Télécom

5. Multimedia Adaptation and Multimodal
Interfaces
 Problems
• Adapt the multimedia consumption to the user context
(preferences, network, device constraints)
• Find representations and architectures for adaptations
 Latest work (in progress)
• Usage of semantic information for adaptation and content
generation
• V. Gros’s PhD – “Digital Book Publishing”
 Some other results
• W3C recommendation: "Multimodal Architecture and
Interfaces"
• Multimodality: B. Rodriguez’s PhD.
• Adaptation: B. Pellan’s PhD
5 Institut Mines-Télécom

6. Multimedia Adaptation and Multimodal
Interfaces: Ontologies and Semantic
- Interactions Description
- Ontologies to describe
- How/Where/Which device
you interact with,
- Tight relationship with
W3C MMI group
- Description of
multimedia adaptation
engines
6 Institut Mines-Télécom

7. Multimedia Adaptation and Multimodal
Interfaces: Adaptation Engine Ontologies
WSML
example
Displayed with
Web Services
Modeling
Toolkit
Now:
RDF+OWL+OWL-S
7 Institut Mines-Télécom

8. Multimedia Transport and Delivery:
Context
 Internet Video Growth
• Cisco’s forecast
• Netflix Example: 30%
of US traffic during
peak hours
 Mobile Video
• >= 50% of traffic
8 Institut Mines-Télécom

9. Multimedia Transport and Delivery:
Requirements
 Basic constraints are satisfied
• Audio/Video Synchronization
• Handles Loss and Corruption
• Fast start-up
• Limited rebuffering
• Compatible with Live, VoD, trick modes, …
 Additional constraints
• Adaptive to varying bandwidth conditions (seamlessly,
granularity)
• IP-compatibility (proxies, firewalls, web servers,
caches)
• Efficiency
9 Institut Mines-Télécom

10. Multimedia Transport and Delivery:
« HTTP Streaming » Principles
 Use a playlist
• Continuously download small media files via HTTP
─ Selection btw. alternative files based on bandwidth
─ Constant duration files, cacheable
 Properties
• Time continuity btw. files across quality to guarantee
seamless playback and switching
• Compatible with live and VoD
 Some Limitations
• Latency for live events
• Some overhead (e.g. MPEG-2 TS / HTTP)
 Proprietary Solutions: Apple, Microsoft, Adobe
 Standards: MPEG, 3GP, OIPF, W3C, …
10 Institut Mines-Télécom

11. Multimedia Transport and Delivery:
HTTP Streaming Research Aspects
 Adaptation strategies and Quality of Experience
 Cache optimizations
 Energy efficient streaming (mobile)
 Low latency live streaming
 Scalable media streaming
 Hybrid Broadcast Broadband
 Multipath streaming
 …
 PhD Student: N. Bouzakaria
11 Institut Mines-Télécom

12. Multimedia Transport and Delivery:
Application Integration
 How to repurpose broadcast data?
 How to design HTML5 streaming-aware applications?
 How to deliver new media data (HEVC, subtitling …)?
 How to synchronize broadcast/broadband events?
Audio
Concolato et al., Usages of DASH
for rich media services. MMSys
Data carousel (500 ms) '11.
DOI=10.1145/1943552.1943587
Data carousel (2 s)
Live data
12 Institut Mines-Télécom

13. Multimedia Services and Transmedia:
Research Topics
 New types of services
• Interactive Multimedia
─ Digital TV (HBBTV, …)
─ Digital Radio
─ Web (HTML5, SVG)
• Stereoscopic Displays
• Second Screens
13 Institut Mines-Télécom

14. Multimedia Services and Transmedia:
Second Screens
 User Interfaces are evolving
• Integrate advanced rich media
─ 2D/3D, animations, video/audio clips
• Possible to have a common
user interfaces on many devices
─ TV, Phone, Desktop, Web page
 Designed around
dedicated small applications
⇒ Widgets
14 Institut Mines-Télécom

15. Multimedia Services and Transmedia:
Transmedia & Widgets
Widget Web Server
widget
2) Download
& activation Internet
Home Network
1) Discovery
3) UPnP Control
messages
4) UPnP Notification
messages DLNA Blu-Ray Player
4) Data exchange
DLNA TV
16 Institut Mines-Télécom

16. Multimedia Services and Transmedia:
Our Approach
 MPEG-U (Rich Media User Interface)
• Lead by Telecom ParisTech
─ Collaboration with Samsung R&D HQ
• MPEG Media Type
• Simplified representation
• Widget Communication Communicating and migratable interactive
multimedia documents, MTAP 2011
 COLTRAM EU Project
• Collaboration with Fraunhofer FOKUS
• Use of Web Technologies
─ PhD Student: R. Bouazizi
• Authoring of Migratable Web Apps
─ PhD Student: M. Sarkis
17 Institut Mines-Télécom

17. Standardization activities
 MPEG
• Since 1996
• Mainly MPEG Systems activities and standards
─ Interactivity: MPEG-4 BIFS, MPEG-4 LASeR
─ File Format: MP4, AVC, SVC, HEVC
─ Streaming: MPEG-2 TS, MPEG DASH
• Editors of several standards (+ conformance
sequences, reference software, utility software)
 W3C
• Since 2004
• Working Groups: SVG, HTML, MMI, WebApps, …
18 Institut Mines-Télécom

18. The GPAC Open Source Project
 GPAC (http://gpac.wp.institut-telecom.fr)
• Multimedia player, encoder, and delivery tools
• Distributed Open Source under LGPL License
 Goals
• Platform for research and academic purposes
• Dissemination of multimedia standards
 Development Context
• Within our research activities
• In sync with industry standards (MPEG, W3C, DVB, ...)
• Using academic resources (student projects,
internships)
19 Institut Mines-Télécom

19. The GPAC Open Source Project:
Highlights
 13 years-old project!
 ~ 500 000 lines of code
• Cross-platform, ~ANSI C
• CPU architectures:
─ x86, ppc, arm, arc, sh4 / 32-/64-bits
• OS:
─ Windows (desktop and mobile), Linux, Mac OS X, various UNIXes
(BeOS, Solaris, …), Symbian, iOS, Android.
 Good visibility/community
• 5000 downloads per month
• Research community
─ 2nd Best OSS @ ACM Multimedia 2007
• End-User community
─ Mostly uses MP4Box
20 Institut Mines-Télécom

20. The GPAC Open Source Project:
MP4Box
 Swiss-army knife for the ISO Base Media File Format and
derivatives:
• MP4,3GP,OMA,MJ2K,MP21, PSP...
• Major MP4 packager on the web
 Media track manipulations
• Importers and exporters
─ audio, video, subtitling in different formats
• Grouping, disabling, tagging ...
 Delivery preparation: offline/live
• Interleaving for HTTP download
• RTP Hinting for 3GP/OMA streaming servers
• DASH (segmentation, MPD generation)
• MPEG-2 TS generation
 Encryption/Decryption
 File analysis
 Scene Description encoding
21 Institut Mines-Télécom

21. The GPAC Open Source Project:
Basic Playback Features
 Classic Audio/Video player
• Similar to Quicktime, VLC
• Integrated with all browsers (IE, Safari*, Chrome, Firefox,
Opera)
• Available with different UI: command-line, GUI (Windows,
SDL, Qt, ...)
 Various media codecs
• Including HEVC, H.264|AVC, SVC, AAC …
• Based on third party libraries
 Various network stacks
• Web protocols: HTTP, IceCast, DASH
• Streaming protocols: RTP, RTSP
• File Formats: MP4, MPEG-2 TS...
• Home Networking protocols: UPnP/DLNA, ...
 Extensible through plug-in architecture
22 Institut Mines-Télécom

22. The GPAC Open Source Project:
Advanced Playback Features
 Enhanced Multimedia Player
• Very similar to a Web Browser
• Support for several document formats
 Synthetic Graphics
• 2D: SVG, MPEG-4 BIFS, Flash
• 3D: VRML, X3D
• With some acceleration (OpenGL / OpenGL ES / TinyGL)
 Advanced Composition
• Mixing of languages
• Animation & Synchronization
• Composition & Synchronization
of multiple media elements
 Interactivity
• Support for JavaScript, including AJAX
23 Institut Mines-Télécom

23. Merci!
24 Institut Mines-Télécom