Thinking with your body and other things

Thinking with your Body
and Other Things
David Kirsh
Cognitive Science
UCSD

HOW DO WE
MOVE THOUGHT FORWARD?

How do we use our …
Eyes
Hands
Bodies
Tools
Nearby objects
To help
us think?

Which square becomes the cube’s top?
XD
C
B
AF
E

D is the top square
DD
Mental Simulation
D
C
B
AF
E

Some thinking involves
mental simulation while looking

Projection: another element of
visual thinking
See things that aren’t there

Move 11 to where 15 is
Initially visual search
to locate 11 and 15
1 1
1 5

We project faint mental
images onto external
or internal visual anchors
or indices (FINSTs)
11 to 15

How many moves can you
reliably project?
1
1 1
Now 1 to 11

Visual search
Projection
Manipulation
then Create change
Project
Project  Create Interactive strategy

Thinking often involves
making models or simulations
through ‘active’ perception,
projection and manipulation.

Thinking often involves
manipulating external
structures (as well as internal)
projections.

Agenda
Main Theoretical Claims
Thinking through the Body
Study of Dance Making
Thinking through Instruments
Action vs. Observation
Thinking through Ordinary Objects,
Bottles, Representations, …

ONE: THINKING IS OFTEN
INTERACTIVE
Claim

How do we use our …
Eyes
Hands
Bodies
Tools
Nearby objects
To
Think

Prove:
All three medians of a triangle
always intersect at a single point.

Can you do it
just in your head?
Do the medians intersect at a single point?

Conceive then Imagine
Do the medians intersect at a single point?
Triangle
Medians
Intersect
Imagination is fallible

All three medians intersect at a point.
Easier if you create an external representation
Conceive then
Externalize

Create 3rd median
Still too hard to project reliably so create

 THINKING IS OFTEN INTERACTIVE
Claim One
PROJECT  CREATE INTERACTIVE
STRATEGY

TWO: PROJECTION CAN BE IN
ANY MODALITY
Claim

Look and use your hands to feel its width
Kinesthetic projection

Look and use your hands to feel its width
Kinesthetic projection
Anchored on
visually perceived
object

She sells seashells by the seashore
Read this silently

Read this silently
She sells seashells by the seashore
Auditory projection onto words

 WE PROJECT IN MANY
MODALITIES
Claim Two
KINESTHETIC, SOUND, VISION . . .

THREE: WE CREATE STRUCTURE
TO
PROJECT ONTO
IN MANY MODALITIES
Claim

Thinking through Gesture Mental Abac
Source: Brooks et al, Gesture in Mental Abacus Calculation, 2012
Visual projection onto hand movement
or
Kinesthetic projection onto hands

Numbers are encoded visually rather than as
sounds or words

Thinking through Body Movement

Work done with Paul Maglio 1994
Where should this
piece go?
Thinking by
moving objects

Physical Rotation
is faster than
Mental Rotation

Physical Rotation
reveals different
shapes

We project the location

Solving it in the world is more cost-effective
• More reliable
• Usually faster
• Less effortful
• Scales up better
Oops

 WE CREATE STRUCTURE TO PROJECT
ONTO
IN MANY MODALITIES
Claim Three
HAND MOTION, SOUND, MOVING OBJECTS . . .

Interim Summary
We think interactively through
projection  creation
Thinking often involves building models or
simulations
– to make sense of things or
– to think things through

Interim Summay
We think interactively through
projection  creation
Thinking often involves building models or
simulations
– to make sense of things or
– to think things through

Interim Summary
This interactive strategy is multimodal
Gestures + Mental Imagery
(projection)
Body Movement + Mental Imagery “
Manipulate objects + Mental Imagery “
Writing + Mental Imagery
Drawing + Mental Imagery “

FOUR: INTERACTIVE THINKING REQUIRES A TIGHT
TEMPORAL COUPLING BETWEEN INTERNAL
AND EXTERNAL PROCESSES
Claim

Hand - ‘Mind’ Coordination
Tight Temporal Coupling
Only useful for moments

Kinesthetic
imagery -
projection
Thinking involves tight temporal coupling
between components
Executes this
Thinking this
helps shape performance
Dancer: Noah Gelber

Tight’ish temporal coupling
• ‘Read these words’
– inner speech is tightly coupled
• Write what you’re thinking
– to build on your writing interactively inner processes cannot
decouple from outer ones
• write  read  write
• Conversation – joint activity breaks down if delays are
extended

 TIGHT TEMPORAL COUPLING BETWEEN
INTERNAL AND EXTERNAL
Claim Four
MENTAL IMAGERY  MOVEMENT

FIVE: WHEN WE EXTERNALLY MODEL WE
MENTALLY PROJECT TO SOME ASPECT
OR
DIMENSION OF THE TARGET THING
Claim

When making a caricature you think of some aspect of the
real thing

Models are simpler than the thing modeled

 WE MENTALLY PROJECT TO SOME ASPECT
Claim Five
LOOK AT IMAGE  SEE JOWLS, DETERMINATION

Agenda
✔ Main Theoretical Claims
Thinking through the Body
Thinking through Instruments
Thinking through Ordinary Objects,
Bottles, Representations, …

Case Study
Thinking with our body
Body Mediated Cognition

Timeline – ‘Dyad’ 2009
Wayne McGregor
Jan 26th Feb 13th
UCSD
Aug 25th Sept 7th
London
Oct 13th
Premiere
Total = 6 weeks
Random Dance

Timeline – ‘Far’ 2010
Sept 13th Nov 13th
London
Nov 17th
Premiere
Total = 6 weeks

Timeline – ‘Undance’ 2011
Aug 30th Oct 23rd
London
Nov 30th
Premiere
Total = 6 weeks

Timeline – ‘Atomos’ 2013
July 21st Oct 8th
London
Oct 9th
Premiere
Total = 11 weeks

Observation Set-Up - UCSD
Rear
2
Rear
1
W
Cam
Daily
2
Daily
1
Mandeville
Observers
 Mandeville Auditorium  Cal-IT2
Daily2 W
Cam
(b)
Daily1(a)
Rear1(b)
Rear1(c)
Rear2
Cal-IT2
WCam(a)
Artist
Entrance
Rear1(a)
Daily1(b)
Rear1(a)
Student's
Entrance

Field Notes
Observations 5 hours/day 6 -11 weeks
Time, Activity code, Description of
activity

Interviews
Before and after session
2 hours per day, for 23 days

Interviews
After session each day usually in two’s

Data Collected at Each Capture
• 20 TB of video of dancers and choreographer
• Dozens of interviews with choreographer and
dancers
• Still images
• Dancer notes
• Associate choreographer’s notes
• Student notes of ongoing activity
• Music used

MARKING in dance
Thinking with the body

What is Marking?
A dance phrase is practiced, explored or reviewed in a less
energetic manner than doing it ‘full-out’.
Small marking
Marking for time

Marking
• Imperfect model of real phrase
• a sketch, abstraction
• Attend to specific aspect of a movement
• Dancers often practice real phrases by marking

Marking: a universal phenomenon
• Tennis swing – work on aspect
• Cello – on the arm
• Staged Plays – an Italian run-through
• Imperfect modeling – aspectual – as a
learning/practice technique

Teachers have students swing without hitting a ball in
order to grasp shapes or make small adjustments
Not even a tennis racket

Planning grips and placements in rock
climbing by marking when on the ground
Similar Phenomena

Planning a downhill run when skiing moguls
Similar Phenomena

Agnes: Full vs. Marked
I’m not using energy
but I’m thinking
what’s my pathway

Agnes: Large vs. Small marking
Tombe Pas de Bouree

Experiment to explore value of
marking

Which condition facilitate learning most?
MarkFull-Out Simulate in the head

Hypothesis
Mark
Full-Out
Simulate in the head
>
>

Experimental Design
MEASUREMEASURE PRACTICETEACH

Experimental Design
Mark
Lie on floor
Full-Out
Trial 2
Trial 3
Trial 1
Mark
Lie on floor
Full-Out Mark
Lie on floor
Full-Out
Group 1 Group 2 Group 3

Performance Measures
1 2 3 4
Technicality Memory Dynamics
Precision of positions Completeness of detail Speed, Force, Acceleration
Timing
Tempo, duration

Higher is better
P = .018
Marking was best!

Measure Mark >Full Full>Mark Mark>Sim Full>Sim
Memory .7334 < .0001 < .0001
Technicality .0029 < .0001 .0005
Timing .0194 < .0001 < .0001
Dynamics - .145 .0003 < .0001
Mem, Tech, Timing .0189 - < .0001 < .0001
Higher is better
P Values
Results by Measured Dimension

Upshot: Marking Study
• Marking is a movement reduction
system
• Other movement reduction systems that
might facilitate thought:
• Whispering or Subvocalizing
• Gesturing
• Physical miming with objects

Marked aspect anchors
projection
Marked phrase
‘Imagery”
In world
In mind
Movement
Projection in mind

Projection vs. Imagination
Reality oriented
See what is present
Perception
Augment reality
Anchored Imagery
Virtual Reality
Imagery has no size
or location
Projection Imagination

Projection: Claims
1. Mental Projection is more powerful than
mental imagery alone
2. We can project beyond what we can
readily imagine
3. External structure helps us

Experiment to explore projection

Blank
Sheet
Imagination Condition

Tic Tac Toe experiment – 3 by 3
Projection conditions
Blank
Sheet
Imagination Condition
Blank Table Table + X O
Experimental Conditions
X O

Blank
Sheet
Within Subject Design, with Practice Phase
Imagination ConditionPractice
Blank
X O
Projection conditions
Table Table + X O

Projection ≠ Memory Offload of State
Board remains the same over time
Memory not offloaded because no persistent structure
X
O

4,8
4,9
4,6
4,0
4,5
5,0
5,5
6,0
Blank Table Table + XO
Overall Means
Not significant N = 25
Secs
X O
Lower is better
Results 3 by 3
4.9s ≈ 4.8s
Surprise!
≈
Table is no faster than blank

Results 3 by 3: Individual Differences
4,4
5,8
4,7
4,0
4,5
5,0
5,5
6,0
Blank Table Table + XO
Significant p = .002
X O More than half were
significantly better using
Imagination alone
Secs

X
Anchoring Costs
Anchoring = Spatializing a component of mental image
X
Placement costs

Do benefits ever overcome
anchoring costs?

If the imagery task is hard
enough everyone will benefit
Conjecture:

4 by 4 Board
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16

4 by 4 Experiment: harder imagery task
Imagination Projection
Practice
Blank Table Table + X O
3 Conditions
Blank
Page
X O
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16

Results 4 by 4
Table is worth
the cost of coordination
If task is hard enough
Table faster than XO p=.01
N = 25
4 by 4
Mean Time per move
p = .002
> p = .002
9.1s 10.7s

Is it better for everyone?
Bad visualizers are helped much more!
Strong
grid > blank
p = .11
Weak
grid > blank
p = .01
*
Lower is better*
Weak 10.4 < 14.1
Strong 9.3 < 10.1
>
Strong trending to benefit
Blank XO Grid Blank XO Grid Blank XO Grid

Upshot: Projection
• Projection: real process
– distinct from perception and imagination
• As problems get harder we cannot easily
imagine the answer
• So we rely on creating external structures
to project onto and scaffold imagination

SIX: IF WE CAN THINK WITH OUR BODIES
THEN WE CAN THINK WITH THINGS
THAT WE ARE HIGHLY PRACTICED
WITH
Claim

Agenda
1. Thinking with the Body
Thinking with Tools
Thinking with Ordinary Objects,
Representations …
Conclusions

Thesis
Eyes
Hands
Bodies
Tools
Nearby objects
To help us think
We Use our

Players can Mark with their Instrume
As if it is their body

So ... Experts can think with
their
instruments

Can they have the same
thoughts without?

Why an expert might have the
same thought without playing

Super Experts
• Play enough, you eventually simulate playing without an
instrument in hand - i.e. internalization
– Instrument-mediated cognition
is possible without the instrument
• Perlman’s auditory perception
is so expert when he listens
it is as if he is physically playing

Neural support that observing is
enough
• Motor resonance, mirror neuron, action observation
network
– Watching or listening stimulates covert actions as if
dancing or playing
• Covert actions involve motor planning, just as overt
action does
• The real difference is that covert action does not
activate muscular control
• So experts simulate playing

Support from Enactive Perception
• Enactive account: observer sees the counterfactual
futures in the present – what it would look like if I
moved, turned it, played it …
• Observer’s phenomenological experience includes
possible ways a phrase may be continued

So in auditory perception agents
enact a possible future

Do mirror neurons simulate at a fine
enough resolution?

Listening is not Performing
physicality of the piece

Performer Situation ≠ Observer
Situation
• Performer has responsibility to succeed
• observer doesn’t
• Performer must decide: how to attack a note, its mood,
emotionality
• These concepts are ad hoc, situated, and embodied
• unavailable even to an an expert observer.
• Hence agents project a future that is conceptually and
experientially richer than the future projected by an
observer
• Simulation is a lower resolution than actual playing

With a violin in hand musical engagement is
mediated differently than in imagination and
active listening
Instrument mediated conclusion

Agenda
1. Thinking with the Body
Thinking with Instruments
Thinking with Ordinary Objects,
Representations …
Conclusions

Thinking with Illustrations by deforming
them on a stretchy surface
D’Arcy Thompson’s famous fish geometry
Distorting the plane equals distorting growth rates – explains different forms
Stretching, twisting, shearing

Solving it in the world is more cost-effective
• More reliable
• Usually faster
• Less effortful
• Scales up better

Cognition flows to wherever costs
are lower
Internal Processes
External Processes
In closely coupled system, process and structure migrates
to wherever computational and physical costs are lowest
External Processes

Physical is cheaper
Here processes flow to the physical

Thinking by interacting with 3D models
We use the model as tool for visualization – focusing
attention on things that suddenly make sense

Driving thought forward
Objects in motion can do work for us

We can think causally with objects
If we interact we can:
• Compare rotation speeds
• Try out speeds
• Get a feel for force
• Visualize effects

THINKING IS MULTI-MODAL
Bodies & Things
Eyes & Hands
Bodies & Kinesthetics

HIGHLY PERCEPTUALVisual Probing
Kinesthetic Probing

INSIDE AND OUTSIDE
PROCESSES ARE TIGHTLY
COUPLED

OFTEN INVOLVES PHYSICAL OR
MENTAL SIMULATION OR
CONSTRUCTION

SIMULATION AND PROJECTION ARE
HARD SO INTERACTING IS USUALLY
PART OF THINKING

Eye hand mediated cognition
Body mediated cognition
Tool mediated cognition
External Rep’n mediated Cognition
Object mediated Cognition

WITH THE RIGHT STRUCTURES AND
MOVEMENTS WE CAN PUSH
THOUGHT FORWARD FASTER

Acknowledgements
Thanks to
Dafne Muntanyola
Wayne McGregor
Odette Hughes
Random Dance Company
Funding:
NSF grant: Distributed Creative Cognition in Dance
UCSD core grant
R-Research, London
UCSD classes

Thinking with your body and other things

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (16)

Semelhante a Thinking with your body and other things

Semelhante a Thinking with your body and other things (20)

Mais de Alexandre Monnin

Mais de Alexandre Monnin (20)

Último

Último (20)

Thinking with your body and other things