Paper presentation by Celso A. S. Santos in the XXI Brazilian Symposium on Multimedia and the Web (October 27-30, 2015, Manaus)
Authors: Estêvão B. Saleme and Celso A. Saibel Santos
DOI: http://dx.doi.org/10.1145/2820426.2820450
PlaySEM demo video is available at: http://youtu.be/aka4jBZcoKk
Another application using SER: http://youtu.be/a5QPsB2zKpk
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
PlaySEM: a Platform for Rendering MulSeMedia Compatible with MPEG-V
1. PlaySEM: a Platform for Rendering
MulSeMedia Compatible with MPEG-V
Estêvão SALEME & Celso. A. S. SANTOS
Manaus, AM, Brazil – 28/10/2015
Networks and Multimedia Research Lab – LPRM
Federal University of Espírito Santo – UFES
2. “Traditional” multimedia (video, audio, text, ...)
MulSeMedia - stimulate other human senses:
◦ Mechanoreceptors, Thermoreceptors, Chemoreceptors
Subjective experiments -> QoE improvement compared
to Media without sensory effects (WALTL et. al, 2009)
WALTL, M.; TIMMERER, C.; HELLWAGNER, H. A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects.
In: First International Workshop on Quality of Multimedia Experience (QoMEX 2009). [s.n.], 2009.
Introduction
3. How it works?
WALTL, M.; TIMMERER, C.; HELLWAGNER, H. A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects.
In: First International Workshop on Quality of Multimedia Experience (QoMEX 2009). [s.n.], 2009.
Sensory Effects
(WALTL et. al, 2009)
4. Problem
• Are there any
integration or
interoperability
issues with other
applications?
• The development of
interoperable
solutions (MPEG-V)
is required.
Heterogeneous
solution
• Is it not feasible to
reutilize the sensory
effects processor?
• There is a need to
reutilize solutions
in different ways,
such as Media
Players, TV Digital,
games, etc.
Strong
coupling
5. New approach for simulating and rendering
sensory effects, independently from Media Players,
communicating through UPnP that is compatible
with MPEG-V
It provides:
◦ SE Video Player
◦ MPEG Metadata library
◦ Simple platform for simulating and rendering sensory effects (SE
Video Player + SE Renderer)
◦ Open-source, fuel for future researches on the sensory effects area
Objective of our work
6. Use embedded sensory effect renderers (guide
virtual x real world):
i. SEMP (WALTL et al., 2013)
ii. SESim (WALTL et al., 2013)
iii. Sensorama (CHO, 2010)
iv. Sensible Media Simulator (KIM; JOO, 2014)
WALTL, M. et al. An end-to-end tool chain for sensory experience based on mpeg-v. Sig. Proc.: Image Comm., v. 28, n. 2, p. 136–150, 2013.
KIM, S.-K.; JOO, Y. S. Sensible media simulation in an automobile application and human responses to sensory effects. ETRI Journal, ETRI, v. 35, n. 6, p. 1001–1010, 2014. ISSN 1225-6463.
CHO, H.-Y. Event-Based control of 4D effects using MPEG RoSE. 2010.
Sensory Effects Media Players
7. SEMP
WALTL, M. et al. An end-to-end tool chain for sensory experience based on mpeg-v. Sig. Proc.: Image Comm., v. 28, n. 2, p.
136–150, 2013.
(WALTL et. al, 2013)
8. SESim
WALTL, M. et al. An end-to-end tool chain for sensory experience based on mpeg-v. Sig. Proc.: Image Comm., v. 28, n. 2, p.
136–150, 2013.
(WALTL et. al, 2013)
10. Sensible Media Simulation
KIM, S.-K.; JOO, Y. S. Sensible media simulation in an automobile application and human responses to sensory effects. ETRI
Journal, ETRI, v. 35, n. 6, p. 1001–1010, 2014. ISSN 1225-6463.
(KIM; JOO, 2014)
11. Sensory Effect Renderer is decoupled from the
presentation interface
◦ It can be used with another Media Player or
application
It is not restricted to timeline based applications
Multi-platform (coded in Java)
What is different about PlaySEM?
13. SE Video Player is responsible for playing video and
reading MPEG-V SEM. Main functionalities include:
◦ open a video and related SEM
◦ control media state (play, pause and stop)
◦ position the video at a specific time in the timeline
◦ switch to full screen mode
◦ locate SER devices
◦ obtain information regarding device capabilities and
configure a variety of application parameters
SE Video Player - functionalities
15. SE Renderer is responsible for transforming SEM into
commands and for driving devices
Central component and can be reutilized in other applications.
Its services include:
SE Renderer - services
18. No approaches were found in literature that
decouples sensory effects renderer from media
players, so:
Would it cause any relevant delays or
synchronization loss during the playback of sensory
effects at the user environment?
Open issue
19. Data traffic over a local network
does not introduce significant
delays
H1
SE Renderer architecture has little
impact on synchronization loss
H2
Hypotheses
20. Latency was measured by capturing network packages http
Stats parameter (SE Video Player):
Execution time = T3 (T4 for SetSem) - T1
Data from 5 test rounds was collected and then the average
time was calculated
ID Description
T1 Before command execution
T2 After command execution
T3 SE Renderer processing confirmation
T4 SEM processing confirmation
H1 - Network latency
22. Time readings came from two clocks: C1, from SE
Video Player, and C2, from SE Renderer
The VLCJ clock has a 300ms precision. Another
clock with a better precision was created
Times were recorded on 5 test rounds
Synchronization loss = time elapsed between C2
and C1
H2 - Synchronization loss
24. The PlaySEM platform is based on parts 2, 3, 6 and 7
of the MPEG-V standard
Flexibility for replacing “components”, taking
advantage of its architecture
Multi-platform support
Network Latency:
◦ Basic control functions (Play, Pause, Stop): average time <
16ms
◦ Average latency of all services < 25ms (except SetSem)
◦ Delay exclusion would bring no significant performance
gains
Conclusion
25. Synchronization loss:
◦ There were periods when the video playback was ahead
or behind the rendering of sensory effects, although
never beyond 34 milliseconds
The synchronization loss can be considered negligible,
although it does not account for the time that the hardware
was actually working
Effect type Acceptable time
Haptic 1s behind video
Air flow 5s ahead of or 3s behind video
(YUAN et. al, 2015)
Conclusion
YUAN, Z. et al. Perceived synchronization of mulsemedia services. Multimedia, IEEE Transactions on, PP, n. 99, p. 1–1, 2015.
ISSN 1520-9210.
No modo tradicional da multimedia visao e audicao, seja lendo um livro, escutando musica, vendo um filme ou visualizando imagens
Porque não estimular outros sentidos?
Imersão como uma experiência realista, vista como condição necessária para criação de "presença“.
Imersão como um estado psicológico caracterizado por perceber-se envolvido, incluído e interagindo com um ambiente que fornece um fluxo contínuo de estímulos e experiências.
Presença refere-se a experimentar o ambiente gerado por computador, em vez da localidade física real (WITMER; SINGER, 1998).
Não pretendo processar mídia pois quero que outros tipos de aplicações reusem o renderizador
Dispositivo que não pode interoperar com outros produtos que o cliente espera interoperar perde valor*
Capacidade de diferentes produtos interoperar entre eles permite aos consumidores uma escolha
MPEG-V = interoperabilidade mundo virtual (ex. jogos, simuladores e aplicações) e mundo real (ex. sensores e atuadores)
Acoplamento dificulta a adoção de múltiplos efeitos em outras áreas como medicina, jogos e educação, que geralmente são aplicativos baseados em eventos
Preço do amBX é atualmente 800 libras esterlinas (kit premium)
Renderizar significa entregar alguma coisa a alguém, como por exemplo, um serviço
Porque é relevante? Permite que o renderizador de efeitos sensoriais seja reaproveitado para diferentes tipos de aplicação, não somente baseadas em linha de tempo
Exemplo: 1) Peças teatrais, onde o ator proporciona efeitos sensoriais para os expectadores; 2) Jogos de vídeo games, podem se comunicar com o renderizador para promover efeitos sensoriais; 3) Aplicações na área de educação para ensinar o frutas e vegetais para crianças com associação de aroma.
A abordagem do SE Renderer desacoplado do SE Video Player para o processamento do SEM, o torna capaz de ser utilizado em qualquer Media Player
AL shows messages network Average Latency, (standard deviation) indicates the time variation on AL in order to project how higher it could grow, from a statistical standpoint.
VC indicates the Variation Coecient, as a percentage, projected in AMI, Average Minimum Latency, on the lower bound and AMA, Average Maximum Latency, on the upper bound. APT indicates the Average Processing Time on SER, deducting latency time from execution time. AET indicates the Average Execution Time for each of the service provided by SER.