Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Interactive Socio-Mobile Systems for Active Music Experience
1. Interactive Socio-Mobile Systems for Active
Experience of Audiovisual Content
Antonio Camurri and Alberto Massari
{toni, alby}@infomus.org
Casa Paganini – InfoMus Research Centre
University of Genoa, Italy
www.infomus.org
www.siempre.infomus.org
2. Summary
• The Casa Paganini – InfoMus Research
Centre of University of Genoa
• How S&T research may benefit from the arts
• The European Project SIEMPRE
• The EyesWeb software platform
• Research perspectives
3. Real-time measure of expressive gesture and non
verbal social signals (entrainment, leadership).
The Virtual Binocular to explore AV content
(Socio-mobile) active music listening
…Research
4. Examples of scientific projects at Casa Paganini - InfoMus
based on cross-fertilization of ICT and art
From artistic project… …to S&T research
Music Theatre Opera “Outis”
Luciano Berio
Teatro Alla Scala di Milano (1996)
Invisible interfaces for on-stage interaction
and synchronization with audio
Music Theatre Opera “Cronaca del Luogo”,
Luciano Berio, opening
Salzburg Festival (July 1999)
Real-time analysis of full-body movement,
non-verbal expressive behaviour qualities.
EyesWeb software platform.
Music Theatre Opera “Un Avatar del
Diavolo”, Roberto Doati,
La Biennale Venezia (2005)
Tangible acoustic interfaces: give the
sense of touch to everyday objects
Museum “Enrico Caruso”,
permanent interactive installation “Sala
della Musica”, Firenze (2011-)
Non-verbal behaviour analysis for
individual and group interaction with
cultural heritage content
EU FET11 Closing Performance: TanGO
Touching Music” (6 May 2011)
Performance built upon scientific results of
the European ICT FET SIEMPRE Project.
Study of music joint performance: string
quartets, orchestra sections with
conductor, audience response
S&T research in EU ICT FET SIEMPRE
Project
5. Casa Paganini – InfoMus Research Centre
The monumental building of Santa Maria delle Grazie La Nuova
6. The European Project SIEMPRE
Social Interaction and Entrainment
using Music PeRformance Experimentation
May 2010 – June 2013
Consortium:
Casa Paganini - InfoMus, Univ of Genoa (Coordinator)
Queens University of Belfast
Swiss Center for Affective Sciences, University of Geneve
Pompeu Fabra University, Barcelona
Italian Institute of Technology, Genoa
INCO (non EU) Partners:
Stanford University, USA
Waseda University, Tokyo, Japan
Virginia Tech, USA
7. SIEMPRE: experimentation centered
on music
• Music as a window into complex integrative brain
processes: exploited to study brain mechanisms
• String quartet representative of the Self-Managed
Team (studies in economy and management) in an
ecologically valid framework: e.g., as a proxy for a
group of employees having a total responsibility for a
defined project.
• Orchestra: a scenario characterized by social sub-
groups (e.g., first and second violins sections) with a
leader (the conductor). An orchestra is an example
of inter-team cooperation inside an organization.
• Audience exposed to artistic stimuli.
8. SIEMPRE research challenges
• What factors drive interpersonal synchronisation? How do they
vary according to physical, emotional and social contexts?
• Can specific roles (e.g., leadership) be identified inside a
group? Can general principles be discerned concerning the
influence of individuals over others?
• What are the factors that determine feelings of group cohesion
or a sense of shared meaning? Can the validity of such reports
be supported or confirmed by expressive Movement Audio and
Physiological (eMAP) measures?
• How does social context affect individual intrapersonal
synchronisation of eMAP and vice-versa? How does the
emotion of an individual affect a collaborative creative product?
• Which are the neurophysiological foundations of creative group
communication?
www.siempre.infomus.org
9. SIEMPRE: collaborations with
world-level artists
• Conductors: Riccardo Muti and his
«Youth Orchestra Luigi Cherubini»,
Pietro Borgonovo, Sera Tokay
• Renaud Capuçon, violinist
• Quartetto di Cremona, string quartet
Experiments also with music students
from Music Conservatories
10. String Quartet experiments
“Self-Managed Team”: all musicians contribute equally to
the task [Gilboa & Tal-Shmotkin 2010]
Picture: Quartetto di Cremona during an experiment.
11. SIEMPRE string quartet scenario: research questions
• Which multimodal cues explain the difference between
playing alone or in ensemble ?
12. SIEMPRE string quartet scenario: research questions
• Which multimodal cues explain who is the leader ?
• Which multimodal cues explain the difference between
a performance engaging an audience and a simply
correct (but «cold») performance ?
14. Orchestra scenario: a study
• Studying the effectiveness of an established
leader through a group measure of entrainment
• Defining a metric for quantifying such
effectiveness
• Validating the metric with raters (experts)
Boerner et al. (2004) model of leadership in orchestra:
• conductor’s leadership style is not participative or delegative, but directive
• Goal-directed leadership: goal is artistic quality
15. Soft entrainment
• beauty in music lies largely in artistic deviation
from the exact or rigid (Seashore)
• Expressivity is characterized by such deviations
• In joint music performance, players do not reach
and maintain the maximum level of entrainment
• Soft-entrainment: alternation of high and low
entrainment (Yoshida):
16. Orchestra: Experiment
• Recordings of Motion Capture and audio on two
string sections and conductor
• Focus on the z-component of bow trajectory of
string sections (z-direction assumed as the
predominant direction of bow movement)
• Segmentation of music phrases
17. The obtained measures
-> Synchronization Index
-> Effectiveness of Leadership, in terms of a
weighted combination of entrainment and
soft-entrainment
18. The obtained measures
Synchronization Index and Effectiveness of
Leadership seem to explain the rating of
expert listeners, asked to judge the quality
of performance of the same music
fragments played by the same orchestra
but with different conductors
20. The SIEMPRE EyesWeb Platform
Supports the synchronous recording of multimodal channels
21.
22. Recent publications
• www.infomus.org www.eyesweb.org
• G.Varni, G.Volpe, A.Camurri (2010) A System for Real-Time Multimodal
Analysis of Nonverbal Affective Social Interaction in User-Centric Media.
IEEE Transactions on Multimedia, Vol.12, No.6, pp.576-590.
• G.Varni, M.Mancini, G.Volpe, A.Camurri (2010) A system for mobile active
music listening based on social interaction and embodiment, ACM Mobile
Networks and Applications Intl Journal (ACM MONET), 16(3).
• D.Glowinski, N.Dael, A.Camurri, G.Volpe, M.Mortillaro, K.Scherer (2011)
Towards a Minimal Representation of Affective Gestures. IEEE
Transactions on Affective Computing, March 2011, 2(2).
• G.Volpe, A.Camurri (2011) A system for embodied social active listening
to sound and music content. ACM Journal on Computing and Cultural
Heritage, 4(1).
• Research team: Antonio Camurri, Gualtiero Volpe, Corrado
Canepa, Paolo Coletta, Nicola Ferrari, Simone Ghisio, Giorgio Gnecco,
Donald Glowinski, Maurizio Mancini, Alberto Massari, Francesca Odone,
Stefano Piana, Marcello Sanguineti, Alessandra Staglianò, Giovanna Varni
23. 4th EyesWeb Week
• A Tutorial and Workshop on EyesWeb: how
to design interactive experiments? How to
master the EyesWeb platform?
• Casa Paganini, spring 2014
• See www.infomus.org for updates
• www.youtube.com/InfoMusLab
• www.siempre.infomus.org
24. future research directions ?
• aesthetic appreciation is one of the most
intangible aspects of higher cognition
• music as a window into complex integrative
brain processes: arts exploited to study brain
mechanisms (Zeki & Lamb)
• exploring the rules governing aesthetic
experience has great relevance for
neuroscience and future ICT
• social aesthetics, quality of experience,
the perceived quality of music performance
Notas do Editor
joint performance of music offers a test bed for ecologically valid investigations of the way non-verbal behavior facilitates joint action. Here we compare the expressive movement of violinists when playing solo Vs. in the string quartet music ensemble. promising context for investigating expressive and adaptive interactions [6], [9] as all four musicians contribute equally to the performance and no explicit asymmetry as it can be observed in the orchestra case computing a set of non-verbal expressive cues to characterize individual performance characteristics (e.g., entropy) and hence giving an insight on why some musician succeed in causing other’s behaviour
SIEMPRE feasibility study (May 2009): the preparation of the preliminary experimental setup of the “string quartet scenario”, in one of the ecological settings (Casa Pa- ganini at UNIGE) available during the whole duration of the project for experiments. Part- ners involved here were UNIGE and QUB. The red circles show two high-speed and high- resolution video-cameras used for movement analysis. Audio recording: two environmental microphones, and one contact microphone on each instrument for separate musical instru- ments recordings (for audio analysis).
SIEMPRE feasibility study (May 2009): the preparation of the preliminary experimental setup of the “string quartet scenario”, in one of the ecological settings (Casa Pa- ganini at UNIGE) available during the whole duration of the project for experiments. Part- ners involved here were UNIGE and QUB. The red circles show two high-speed and high- resolution video-cameras used for movement analysis. Audio recording: two environmental microphones, and one contact microphone on each instrument for separate musical instru- ments recordings (for audio analysis).
Support for fine-grain synchronized recording and analysis of different multimodal chanelsTime stampQualysis used for MoCap data and analog sensors (EMG)EW XMI used for synchronizationRepoVizz: web app and repository to browse and reuse the SIEMPRE archive of multimodal dataIllustration of the synchronized audio-video material used for the perceptual experiment. On left side, a view of the Multimodal Setup for the recording of the Quartetto di Cremona at Casa Paganini (University of Genoa). The selected material consisted in (i) audio from piezo-electric microphone attached to the first violinist instrument (ii) High-quality video centered on the first violinist (iii) smtpe time code managed through the EyesWeb software platform to synchronize the audio-video streams.