O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Sonification of virtual and real surface tapping:
evaluation of behaviour changes, surface perception
and emotional indice...
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 2
Dr. Nadia Berthouze,
Professor i...
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 3
Dr. A. Tajadura-Jiménez,
Researc...
What does Sonification mean?
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 4
Three aspects
Auditory displays
• Systems that systematically transform data into
sounds that can be manipulated by human ...
Why is it important?
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 6
Why is it important?
• Interaction with objects is increasingly mediated
through their digital representation.
• Interacti...
The research
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 8
Our aim
To explore how the sounds resulting from tapping on a
surface inform :
1. The applied strength when tapping.
2. Th...
1. Tapping behaviour
Changes might occur because of the auditory action loop.
• The effect for which users try to adjust t...
2. User’s ability to tap
Users’ ability to tap might be affected because of the
sense of body can be altered by sound feed...
2. User’s emotional states
Body movement is a medium to express and to regulate
one’s own emotions.
• We expected changes ...
3. Physical features of the material
Changing audio feedback can result in changes in an
object perceived material propert...
3. Physical features of the material
Changing audio feedback can result in changes in an
object perceived material propert...
The prototype
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 15
tapping on “real” surface
tapping on “virtual” surface
The tapping action triggers, in real-time, the presentation of pre-...
The measures
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 17
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 18
Design & procedure
Virtual
Real...
1. Measuring tapping behaviour
The accelerometer was used to measure the tapping
behaviour.
• Acceleration = tap’s strengt...
Galvanic skin response (GSR)
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 20
2....
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 21
2. Measuring emotional experien...
To measure perceived surface hardness we used a 7-point Likert scale.
24/10/2015 Enrico Furfaro, We Love Digital | Sonific...
The results
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 23
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 24
Results
Real vs. Virtual surfac...
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 25
Takeaways
When measuring and de...
Implications for design
1. To complement when interacting with digitally
represented objects.
Digital representation of ob...
THANKS!
QUESTIONS?
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 27
enrico@welov...
Sonification of Surface tapping original paper
• Tajadura-Jimenez, A; Bianchi-Berthouze, N; Furfaro, E; Bevilacqua, F; (20...
Fruit Ninja to detect players’ emotional states
• Y. Gao, N. Bianchi-Berthouze, and H. Meng, “What does touch tell us abou...
Próximos SlideShares
Carregando em…5
×

UCD15 Talk - Enrico Furfaro - Sonification of Virtual and Real Surface Tapping: Evaluation of Behaviour Changes, Surface Perception and Emotional Indices

1.310 visualizações

Publicada em

When interacting with physical objects, we receive sensory feedback about the characteristics of these objects. Specifically, the sounds resulting from touching a surface provide information about the material of the surface and about one's own touching behaviour. Current developments in interactive systems are opening up new avenues in the use and design of both physical and virtual objects. These developments have the potential to change people’s behaviours, perceptions and emotions – elements of user experience that can be measured.

This talk presents the results of a study on auditory feedback as example of how research methods from cognitive psychology and human computer interaction can be used to evaluate these potential changes. This research may inform the design of physical and virtual objects and help enhance everyday user experiences.

Publicada em: Design
  • Seja o primeiro a comentar

UCD15 Talk - Enrico Furfaro - Sonification of Virtual and Real Surface Tapping: Evaluation of Behaviour Changes, Surface Perception and Emotional Indices

  1. 1. Sonification of virtual and real surface tapping: evaluation of behaviour changes, surface perception and emotional indices Enrico Furfaro UX Architect @ We Love Digital 24th October 2015 Ana Tajadura-Jiménez, Research Fellow, UCL Nadia Berthouze, Professor in Affective Computing, UCL Frédéric Bevilacqua, Head Researcher, IRCAM UCD 2015 - Humanity in digital landscapes: Mind, Cognition and Psychology
  2. 2. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 2 Dr. Nadia Berthouze, Professor in Affective Computing, UCL Frédéric Bevilacqua, Head Researcher, IRCAM Enrico Furfaro UX Architect Dr. A. Tajadura-Jiménez, Research Fellow and Principal Investigator
  3. 3. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 3 Dr. A. Tajadura-Jiménez, Research Fellow and Principal Investigator Dr. Nadia Berthouze, Professor in Affective Computing, UCL Frédéric Bevilacqua, Head Researcher, IRCAM Enrico Furfaro UX Architect
  4. 4. What does Sonification mean? 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 4
  5. 5. Three aspects Auditory displays • Systems that systematically transform data into sounds that can be manipulated by human users through an interface. Sonification • Sonification is this transformation: data into sounds to convey information. Sonic interaction design • It is the study and exploitation of sound as one of the principal channels conveying information. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 5
  6. 6. Why is it important? 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 6
  7. 7. Why is it important? • Interaction with objects is increasingly mediated through their digital representation. • Interaction with object is multisensory. • Touch, vision and hearing contributing and interacting with each other To what extent do we make use of the information conveyed by auditory feedback? How do we measure it? 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 7 Google’s Project Soli
  8. 8. The research 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 8
  9. 9. Our aim To explore how the sounds resulting from tapping on a surface inform : 1. The applied strength when tapping. 2. The user’s ability to tap & changes in emotional states. 3. The physical features of the surface material (hardness). And to test a multidimensional measurement approach to evaluate user experience related to these three dimensions. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 9 Me tapping on a table
  10. 10. 1. Tapping behaviour Changes might occur because of the auditory action loop. • The effect for which users try to adjust their actions based on the auditory feedback they receive. Example: the audio feedback in real time helped elite rowers to adjust their strokes and improved their performance. (Schaffert et al.’10). 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 10 Listening to the sound of the boat motion increases boat velocity (Schaffert et al.’10) Audio Feedback Motor Behaviour
  11. 11. 2. User’s ability to tap Users’ ability to tap might be affected because of the sense of body can be altered by sound feedback . • Sense of body = the mental representation of body. • Crucial for our interaction with our environment. • Obtained and continuously updated through sensing and acting. Example - the perceived length of participants' arm can be altered by presenting real-time sound feedback suggesting an altered distance. (Tajadura-Jiménez et al.’12). 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 11 Tajadura-Jiménez et al.’12
  12. 12. 2. User’s emotional states Body movement is a medium to express and to regulate one’s own emotions. • We expected changes in behaviour and emotions. For example, the strokes in ‘Fruit Ninja’ were successfully used to detect players’ emotional states (frustrated, excited, relaxed, bored) during the gameplay. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 12 Gao et al.’12
  13. 13. 3. Physical features of the material Changing audio feedback can result in changes in an object perceived material properties. • Both in case of natural surfaces and virtual haptic surfaces. Example: • Reduced high frequencies led to biased perception of sand paper’s texture (smoother). (Guest, et al. 2002) 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 13 Sandpaper
  14. 14. 3. Physical features of the material Changing audio feedback can result in changes in an object perceived material properties. • Both in case of natural surfaces and virtual haptic surfaces. Examples: • Attenuating high frequencies led to biased perception of sand paper (smoother). (Guest, et al. 2002) • higher frequencies or louder sound while biting crisps led to biased perception of crispness and freshness (Zampini & Spence, 2003) 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 14 Kermit the frog eating crisps
  15. 15. The prototype 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 15
  16. 16. tapping on “real” surface tapping on “virtual” surface The tapping action triggers, in real-time, the presentation of pre-recorded tapping sounds Surfaces & Components 24/10/2015 Enrico Furfaro, WLD | Sonification of surface tapping | UCD 2015 16 Two modes: “real” and “virtual”.
  17. 17. The measures 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 17
  18. 18. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 18 Design & procedure Virtual Real Six conditions for each participant “Strong” sound feedback “Medium” sound feedback “Weak” sound feedback Design 2 surface type x 3 sound level strength Timeline for each condition 0 s 10 s baseline1 70 s 80 s baseline2Audio-feedback phase time Questionnaire x
  19. 19. 1. Measuring tapping behaviour The accelerometer was used to measure the tapping behaviour. • Acceleration = tap’s strength • Interval between taps = frequency of taps 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 19
  20. 20. Galvanic skin response (GSR) 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 20 2. Measuring emotional experience and ability to tap Self Assessment Manikin (SAM) Valence Arousal Dominance
  21. 21. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 21 2. Measuring emotional experience and ability to tap 7-point Likert scales: physical strength, ability to tap, aggressiveness. Now please, draw an horizontal line through the vertical scale to indicate how much mental effort you feel you had to invest to complete the tapping task. Subjective Mental Effort Questionnaire (SMEQ)
  22. 22. To measure perceived surface hardness we used a 7-point Likert scale. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 22 3. Measuring surface hardness
  23. 23. The results 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 23
  24. 24. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 24 Results Real vs. Virtual surface • Harder surface, larger perceived strength and ability to tap, less stressed & more in control. Surface hardness • Virtual surface perceived softer when ‘weak’ feedback. Emotions & ability to tap • Audio motor incongruences led to unpleasant arousing experiences. • Real surface. More negative valence when weak sound feedback and less able to tap. • Virtual surface. Weak sound led to higher GSR values. Behaviour • Real & virtual surface. introducing the sound feedback sped up movements and decreased acceleration. • Real surface. Weak sounds led to higher maxima acceleration
  25. 25. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 25 Takeaways When measuring and designing digital representation of objects, consider that audio feedback plays a fundamental role in complementing or substituting haptic feedback. 1. it informs about physical properties of objects. 2. it can be used to manipulate and/or lead motor behaviour. 3. if congruent to the action, it leads to a more pleasant user experience. 4. it might affect the sense of body which relates to perceived body characteristics and self-esteem Example of measures: • Self Assessment Manikin (SAM) • 7-point Likert scales • Subjective Mental Effort Questionnaire (SMEQ) • Bandura’s self-efficacy test • GSR • Sensors (for example accelerometer and piezo) to capture data and for sonification of objects.
  26. 26. Implications for design 1. To complement when interacting with digitally represented objects. Digital representation of objects characterized by a limited amount of haptic & visual feedback: • when extreme precision in applied strength is critical e.g. touch-less surgery or dismantling bombs. • Shopping on-line (tactile-sensory substitution) perceived material properties & emotion responses are important . 2. To guide motor behaviour. Motor behaviour changes in a self-controlled way: • Physical rehabilitation (VR/augmented reality). • Games promoting health- or fun-movement (rehabilitation, education, entertainment). 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 26 Nicole Stenger wearing one of the first VR prototypes in the early 90ies
  27. 27. THANKS! QUESTIONS? 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 27 enrico@welovedigital.com @fasteedio
  28. 28. Sonification of Surface tapping original paper • Tajadura-Jimenez, A; Bianchi-Berthouze, N; Furfaro, E; Bevilacqua, F; (2015) Sonification of surface tapping changes behavior, surface perception, and emotion. IEEE Multimedia , 22 (1) 48 - 57. • Furfaro, E., Berthouze, N., Bevilacqua, F., Tajadura-Jiménez, A. (2013) Sonification of surface tapping: Influences on behaviour, emotion and surface perception. Interactive Sonification Workshop (ISon 2013), Bielefeld, 9th-10th December, 2013. Sonification and Sonic Interaction design • T. Hermann, A. Hunt, and J.G. Neuhoff, The sonification handbook. Berlin:Logos Publishing House, 2011. • Franinović, Karmen, and Stefania Serafin. Sonic interaction design. Mit Press, 2013. Sonification of boat motion improves athletes’ movements • N. Schaffert, K. Mattes, and A.O. Effenberg, “Listen to the boat motion: acoustic information for elite rowers”, in Proc. Interaction Sonification workshop (ISon), Stockholm, Sweden, 2010 Sound feedback affects arm’s length perception • A. Tajadura-Jiménez, A. Väljamäe, I. Toshima, T. Kimura, M. Tsakiris and N. Kitagawa, “Action sounds recalibrate perceived tactile distance”. Current Biology, 22(13), 2012, pp. R516-R517. 24/10/2015 Enrico Furfaro, WLD | Sonification of surface tapping | UCD 2015 28 References
  29. 29. Fruit Ninja to detect players’ emotional states • Y. Gao, N. Bianchi-Berthouze, and H. Meng, “What does touch tell us about emotions in touchscreen-based gameplay?”, in ACM Transactions on Computer-Human Interaction 19(4), pp. 31, 2012. Body movements and emotions • N. Bianchi-Berthouze, “Understanding the role of body movement in player engagement”. Human Computer Interaction 28(1), 2012, pp. 42-75. • A. Kleinsmith, R. De Silva, and N. Berthouze, “Grounding affective dimensions into postures description”, Lecture Notes in Computer Science, Berlin / Heidelberg: Springer, 3784, 2005, pp.263-270. Sound and perception of objects’ characteristics • Guest, S., Catmur, C., Lloyd, D. & Spence, C. (2002). Audiotactile interactions in roughness perception. Experimental Brain Research, 146(2), 161-171 • D. Merrill, and H. Raffle, “The sound of touch”, in Proc. ACM Conference on Human Factors in Computing Systems 2007, ACM Press, 2007, pp. 2807-2812. • Zampini, M., & Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and staleness of potato chips. Journal of sensory studies,19(5), 347-363 24/10/2015 Enrico Furfaro, WLD | Sonification of surface tapping | UCD 2015 29 References

×