1. By: Michael Pouris, BSc, Deborah I. Fels, PhD, P.Eng
IMDC, Ryerson University, Toronto, Canada

2. 1. Background information
2. Research and Model
 Purpose of the visualization
 Past Industry attempts
 The priorities
 Psychological model
3. The implementation
 Translating to a visual medium
4. Study results and discussion

3.  Music is a major art form
 Present in all cultures worldwide
 Transcends cultural and language boundaries
 Portal to a cultural shared experience
(McDermott, 2004)
 Hard of hearing and deaf have limited access to
a shared hearing experience

4.  Serve as a tool that depicts music visually for
hard-of-hearing or deaf audiences
 Sensory substitution is possible (Nanayakkara,
Taylor, Wyse and Ong, 2009)
 Research questions:
 Can sensory substitution models be used to provide
access to music using the visual channel?
 What are user reactions to MusicViz?

5.  Industry attempts have failed
 Does not use justified psychological model
which describes auditory mappings to visual
system

6.  Three primary goals are to develop a
visualization that is:
1. Aesthetically pleasing
2. Emotionally moving
3. Adaptable

7.  Interpretation is to explain the meaning of
something with subjective bias
 Interpreting feelings of the music is not
feasible
 There is not one “feeling” for a musical piece
 Everyone interprets music differently
 Goal: Visualization is open to each individuals
interpretation

8.  Translation is to change or convert to another
form
 Translate auditory cues to visual medium
 Give enough information
 Goal: Individual can interpret information
that is translated

9. Auditory Visual
Auditory System Visual System
Pitch height and Pitch differences • Higher pitches = smaller objects
• Lower pitches = larger objects
• Pitches = altitude
Volume changes • Bigger objects = more prominent
Tempo and Beats • Indicate BPM and rhythm (repeated
pattern)
 Based on Ilie & Thompson (2006)

10.  Issue between pitch and volume in visual
system:
 Higher pitch  smaller object
 Higher amplitude (volume)  larger object
 Problem: amplitude and frequency are
integrated within visual system
 Solution:
Auditory System Visual System
Pitch height and Pitch differences • Associate with altitude
Volume changes • Associate with size

11.  Implemented using JOGL 2.0 (Java OpenGL)
 Is a wrapper class for C OpenGL calls
 Uses GLUEGEN to bind Java to C
X-Axis: Instrument arrangement
Y-Axis: Pitch scale
Z-Axis: Time progression

12.  Pitch: Pitch height determines height of the pipe
 Timbre: Different “sound families” are defined through color

13.  Volume: Depicted through using thickness of pipes
 Tempo: Clusters of activity

14.  Rhythm: Visual depiction of beats and interactivity

16.  12 participants
 3 hearing, 3 hard of hearing and 6 deaf ASL users
 Procedure:
 6 one minute-songs, each from different genres in
random order
 Genres: classical, country, jazz, pop, rap/hip-hop/R&B
(RHRB), rock
▪ Commonly used groupings in industry
 Pre-study, post-song, post-study questionnaires
 Eye tracking data recorded with FaceLab 5
 Data analysed with repeated measures ANOVA

17.  Feedback on enjoyment and emotional experience
 Enjoyment:
 5 point Likert scale (1-not enjoyable at all to 5-enjoyable)
 Emotions:
1. Valence and arousal model
(Russell, 1972), overbearingness added
2. Discrete model (happy, sad, anger, fear) (Ekman, 1972)
 Level of focus

18.  No statistical difference between genres
 Descriptive analysis:
 All genres are rated between 4-somewhat enjoyable and
5-enjoyable
 Rock is most enjoyable (M=4.5, SD=0.85)
 RHRB (M=4.0, SD=1.054) and classical (M=4.0, 1.247) are
least enjoyable
▪ Initially hypothesized to be the most enjoyable due to bass

19.  Overbearingness: 1-subtle to 9-overbearing:
 Between pop (M=6.42, SD=1.084) and RHRB
(M=4.92, SD=1.564)
 Between pop and country (M=4.25, SD=1.603)
 Trend: an overbearing song can still be happy.
 Pop is most overbearing (M=6.42, SD=1.084), highest
arousal (M=6.08, SD=2.021) and happiest
(M=6.75, SD=2.179)
 Valence from 1-unhappy to 9-happy: not significant
 Arousal: 1-calm to 9-excited: not significant

20.  Four 7-point scales (1-weak to 7-strong) for:
1. Happiness 3. Anger
2. Sadness 4. Fear
 No significant difference between genres
 A song may have multiple emotions and people were
not able to choose one with limited set of words

21.  5-point likert scale: “1-my mind wondered a lot” to
“5-I was always focused on the visualization”
 No significant difference between genres
 Trends:
 Paid most attention to rock (M=4.91, SD=0.302), country
(M=4.73, SD=1.206) and jazz (M=4.64, SD=0.505)
 Paid least attention to pop (M=4.09, SD=1.378), RHRB
(M=4.09, SD=1.578)

22.  Major limitation is the lack of participants
 Additional participants are needed to determine
whether trends observed are actual statistical
differences
 Additional participants are needed to explore the
differences between the hearing statuses
 Participants showed preference for rock, pop and
classical genres
 MusicViz provided enjoyment and information

23.  Funding provided by GRAND, CFI, NSERC
 Co-workers at the IMDC
 Study participants

24.  Questions?
Michael Pouris, Deborah Fels
Ryerson University, Toronto, Ontario, Canada
mpouris@ryerson.ca, dfels@ryerson.ca