1. Music and the Brain
Structured neural responses to natural stimuli
Princeton University
Uri Hasson
Psychology department and the Neuroscience Institute

2. David Heeger
Morwaread Farbood
Gary Marcus
Lab members Collaborators
Yulia Lerner
Chris Honey
Greg Stephens
Janice Chen
Erez Simony
Lauren Silbert
Mor Regev
Who did the work

3. Complex natural stimuliControl simplified stimuli
Sensory Coding and the Natural Environment
Few well characterized dimensions
linear properties, simple math
Multidimensional
and Messy
Parameterized Scientific investigation

4. Complex natural stimuliControl simplified stimuli
Real life
Complex
Structured
Movies/Stories/Music
Sensory Coding and the Natural Environment

5. Memory formation of real world events
Hasson et al. Neuron 2008
Neural responses to natural complex stimuli
Hasson et al. Science 2004
Scan paths of real life events
Hasson et al. In-press
Disruption of brain responses in autism
Hasson et al. Autism Research 2009
Time scale of processing
Hasson et al. J neuroscience 2008
Using movies as a research tool in
cognitive neuroscience
Social communication
Stephan et al. PNAS 2010

6. Subjects
Experiment / measurements

8. Inter-subject correlations
Talairach coordinates
Subject 1/Run 1 Subject 2/Run 2

9. Idiosyncratic responsesSharedsignal Idiosyncratic signal
Intra-SC
Inter-SC

10. Sharedsignal Idiosyncratic signal
Shared responses
Intra-SC
Inter-SC

11. Inter-subject correlations
Hasson et al, Science, 2004

12. So I’m banging out my story and I know it’s good, and then I start to make it better
by adding an element of embellishment. Reporters call this“making shit up”.
And they recommend against crossing that line.
But I had just seen the line crossed between a high-powered dean and an assault with a
pastry, and I kind of liked it.
Real-Life Story Stimulus
“Pie Man”
Story

13. Early Auditory Cortex
r = 0.55
r = 0.72
Individual subjects
Average subject
Brain Responses to Real-Life Story
“Pie Man”
Story

14. PrecuneusEarly Auditory Cortex
Angular GyrusInferior Frontal Gyrus
r = 0.66
r = 0.6
r = 0.72
r = 0.41
r = 0.56r = - 0.18
r = 0.1
Brain Responses to Real-Life Story
“Pie Man”
Story

15. medial view
Left
Hemisphere
Right
Hemisphere
lateral view lateral view
A P P A
r=0.11
0.45
N=11
Brain Responses to Real-Life Story
“Pie Man”
Story

16. Brahms
Piano Concerto No.1 in D
minor
Brain Responses to Real-Life Music
A1
M1
M1
STG STG
A1
BG/ThalamusBG/Thalamus

17. The extent of neural overlap between language-related and music-
related processes

18. Intermediate summary
Real life stimuli, as movies, stories and music can exert considerable control over
the responses of many brain areas, evoking a similar time course of activity across
all viewers.

19. How does the brain process such complex and rich
temporal structures?

20. So I’m banging out my story and I know it’s good, and then I start to make it better
by adding an element of embellishment. Reporters call this“making shit up”.
And they recommend against crossing that line.
But I had just seen the line crossed between a high-powered dean and an assault with a
pastry, and I kind of liked it.
1:43 1:45 1:47
1:50 1:54
1:58
2:05
0:55
Time-scales of Information in a Narrated Story
So

21. medial view
Left
Hemisphere
Right
Hemisphere
lateral view lateral view
A P P A
r=0.11
0.45
N=11
“Pie Man”
Story
Different Processing Timescales in Different Regions?

22. Scrambled past
Word Word
present
Coherent pastScrambled past
Extent of past information needed to evoke reliable responses in the present
Entire story
Neural responses
at the present
Coherent past
Paragraph Paragraph
present
Coherent pastScrambled past
Sentence Sentence
present
Coherent pastScrambled past
present
Long
memory
No
memory
Intermediate

23. Words
Parametric variation of the temporal structure of a verbal
monologue
Paragraphs
Backward
Sentences
Intact story
Temporal rate is fixed
Each 7 minutes condition is
composed of the exact same basic
units

24. A P P
Reverselateral
medial
AP P
LH RH
q<0.05 (FDR)
N = 11
A
Inter-subject Correlation During a Narrated Story
Words
Sentences
Paragraphs

25. medial view
Left
Hemisphere
Right
Hemisphere
lateral view lateral view
A P P A
AP P
N = 11
sent paragrev words
A Hierarchy of Processing Timescales

26. A1+ TPJ
Temporo-Parietal Axis
FS P S W R
correlation
sent (S) parag (P)rev (R) words (W)
FS P S W R FS P S W R
N = 11
1
2
3
4
5
FS P S W R
A Hierarchy of Processing Timescales

27. auditory story
+
silent movie
short mid long overlap
=
Processing Time-Varying Information About the World

28. Bars Phrases
Sections
Intact
Short temporal scales Mid temporal scales Long temporal scales
Reversed
Parametric
varia+on
of
the
coherent
temporal
structure
within
a
music
piece
Brahms
Piano
Concerto
No.1
in
D
minor

29. Musical Temporal Receptive Windows
A1+

30. Musical vs. Linguistic based Temporal Receptive Windows

31. Musical vs. Linguistic based Temporal Receptive Windows

32. Word/bar
present
Short temporal integration window
Extent of past information needed to evoke reliable responses in the present
Entire story/musical piece
present
Long temporal integration window
Paragraph/section
present
Intermediate temporal integration window
Sentence/phrase
present
Intermediate temporal integration window
present
Long
memory
No
memory
Intermediate

33. Spatial Scale Temporal Scale
Hubel & Wiesel (1959)
J Physiol
Gross et al (1972)
J Neurophysiol
Ungerleider & Mishkin (1982)
Analysis of Behavior
Functional Hierarchy
A Hierarchy of Temporal Receptive Windows

34. Accumulating information over space
and time
Space
Time
Electrode in IT cortex
Temporal Receptive Window in IT cortex
A Hierarchy of Temporal Receptive Windows

35. Integrate Real-World Information
Baddeley & Hitch (1974)
Classical
Working Memory
Model
Hierarchy of Temporal Receptive Windows
Limited Capacity Bottleneck
Processing Time Scales and Working Memory
Maintain Discrete Units of Information

36. The end

38. Is Integration Temporal or Ordinal?
Essential Role of Time Essential Role of Semantic Units
Or Both?
See also Howard & Eichenbaum (in press) JEP General
versus

39. Temporal units and the information units are easily dissociated in real-life speech
100%
75%
50%
150%
200%
Time

40. Yulia Lerner
Our speech perception is invariant to changes in rate

41. Is Integration Really Temporal or Just Ordinal?
Essential Role of Time Essential Role of Semantic Units
Speech intelligibility recovered
by insertion of pauses
Rescaling of neural responses
throughout the brain
Memory for absolute tempo
in musical sequences
Integration of information over time
much easier for meaningful speech
Neurophysiology has intrinsic timescales
Or Both?
Ghitza & Greenberg (2009)
Lerner et al (submitted)
See also Howard & Eichenbaum (in press) JEP General
“Time”cells in hippocampus
Naya & Suzuki (2011)
Macdonald et al (2011)
Behavioral invariance
to moderate changes in stimulus rate
Levitin & Cook (1996)

43. Time NOWAccessibility
low high
Timescales of Information

44. Integrate Real-World Information
Cowan (1999)
Embedded
Processes Model
Working Memory =“Activated”Memory Traces
Activated
Memory
Central
Executive
Long Term Memory
Attended
Processing Time Scales and Working Memory
Hebb (1949)
Activated
Cell Assemblies
Hierarchy of Temporal Receptive Windows

45. Time vs. accumulation of information over time

46. Integrate Real-World Information
Persistent Neuronal Activity
Activated
Memory
Central
Executive
Long Term Memory
Attended
Funahashi et al (1989)
Gnadt & Anderson (1988)
Goldman-Rakic (1996)
Processing Time Scales and Working Memory
Hierarchy of Temporal Receptive Windows

47. Hypothesis
A Hierarchy of Timescales in Brain Dynamics
Hierarchy of Temporal Receptive Windows

48. Open Questions
Is the hippocampus required to sustain the long temporal receptive windows?
What happens in the hierarchy at (macro & micro) event boundaries?
c.f. Ranganath & Ritchie (2012) Nat Rev Neurosci

49. Lab Questions
Should I shorten the title?
Present larger questions at the beginning or at the end?
Emphasize /time/ or mental context
Narrative style or argument style?

50. Different Processing Timescales in Different Regions?
Criterion One
minimum prior duration of coherent
information required for a response

51. responses invariant to changes
beyond a maximum duration
Criterion One
Criterion Two
minimum amount of coherent
information required for a response
Different Processing Timescales in Different Regions?
equals

52. Study of Naturalistic Perception
Costs Benefits
Poorer experimental control
Poorer experimental control

53. Why unrelated materials?
Why under conditions of distraction?
Working Memory Experiments

54. Memory systems are organized to represent the real world.
We may look into that window on the mind as through a glass darkly,
but what we are beginning to discern there looks very much like a
reflection of the world.
Roger Shepard (1990) Mind Sights
Perceptual systems are organized to represent the real world.
Anderson & Schooler (1991) Psych Science
Bartlett (1932) Remembering
Neisser (1978) Practical Aspects of Memory

55. Stimulus locked Circuit Dynamics using BOLD

56. Mary Potter
fast semantics
stabilization idea
ISC?
Van Dijk & Kintsch
unavaoidable semantics
the log was on the tutrlte
McLelland and Rumelhart
Bransford and Johnson
effects on memory
Stabiliza

57. Inter-subject Correlation during Movie Viewing
Single Subjects (N=9)
Mean Timecourse
medial view
Left
Hemisphere
Right
Hemisphere
lateral view lateral view
A P P A
r=0.15
0.55

58. Single Subjects (N=9)
Mean Timecourse
medial view
Left
Hemisphere
Right
Hemisphere
lateral view lateral view
A P P A
r=0.15
0.55
Inter-subject Correlation during Movie Viewing

59. Processing Time Scales and Working Memory
Working memory is the
“ability to keep a representation active,
particularly in the face of interference and distraction”.
Engle et al (1999) JEP:General

60. Youssef Ezzyat, Lila Davachi (NYU): Neural mechanisms supporting the temporal organization of episodic long-term
memory
Discussant: Per Sederberg (Ohio State)
Christopher J. Honey, Janice Chen, Erez Simony, Olga Lositsky, Daniel Toker, Kenneth A. Norman, Uri
Hasson (Princeton): Temporal receptive windows in natural perception: a topographic map of mental context
Discussant: Ryan Canolty (UC Berkeley)
Gregory J. Koop, Amy H. Criss (Syracuse): Response dynamics as a measure of bias and strength in recognition
memory
Discussant: Adam Osth (Ohio State)
Isabel A. Muzzio (Penn): Effects of emotion on hippocampal contextual representations
Discussant: Sam Gershman (MIT)
Robert M. Nosofsky, Christopher Donkin, Jason M. Gold, Richard M. Shiffrin (Indiana University): Discrete-slots
models of visual working memory response times
Discussant: Michael Lee (UC Irvine)
Sean M. Polyn (Vanderbilt): Incorporating neural signals into computational models of memory search
Discussant: Jeremy Manning (Princeton)
Alison R. Preston (University of Texas): Building new knowledge through memory integration
Discussant: Marc Howard (Boston University)
Maureen Ritchey, Andrew P. Yonelinas, Charan Ranganath (UC Davis): Medial temporal lobe subregions interact
with functionally distinct systems
Discussant: Ken Norman (Princeton)
Karthik Shankar, Marc W. Howard (Boston University): Optimally fuzzy memory
Discussant: Sue Becker (McMaster University)
Geoff Ward, Cathleen Cortis, Rachel Grenfell-Essam, Jessica Spurgeon, Lydia Tan (University of Essex): Why do
participants initiate their immediate free recall of short lists of words with the first list item?
Discussant:Karl Healey (Penn)
26 minutes for primary speaker; 13 minutes for discussant; 6 for questions

61. “whole” “scrambled”“grid”
Scrambling Objects in Space
completion no completion
Lerner 2003

62. “whole” “scrambled”“grid”
1
2
212
2
1
1 2 1
Lerner 2003
Scrambling Objects in Space
1
2