The document discusses supporting human communication through human-computer interaction designs. It describes how communication involves not just transmitting data across distances, but also understanding between people despite psychological barriers. While technologies like video conferencing and machine translation aim to help, they don't always work effectively if they don't consider the nature of human communication. The document argues that designs should be grounded in research on the features and constraints of successful and unsuccessful communication. The goal is for technologies to help bridge psychological distances and truly connect human minds.

1. On Grounding Human Communication with
Human-Computer Interaction Designs
Hao-­‐Chuan
Wang
.
王浩全
Department
of
Computer
Science
Ins3tute
of
Informa3on
Systems
and
Applica3ons
Na3onal
Tsing
Hua
University
h-p://www.cs.nthu.edu.tw/~haochuan
May
21,
2014
@
NTHU-­‐CS
Seminar

2. Wang
A Quick Overview of Human-Computer
Interaction (HCI)
2

3. The
two
“senses”
of
Human-­‐Computer
Interac7on:
From
interface
…
“Interacon”
in
the
sense
of
computers
listening
and
responding
to
people’s
input

4. …
to
problem
solving
and
value
crea7on
in
the
real
world
“Interacon”
in
the
sense
of
designing
technologies
based
on
user
needs,
goals,
constraints,
and
characteriscs.
UCD:
User-­‐Centered
Design.
Iden7fying
fixing
usability
problems
Technology
supported
educa7on
Persuasive
(behavioral
change)
compu7ng

5. Wang
HCI: Studying Existing and Possible Relationships between
Computers and People
5
ACM
SIGCHI
Curricula
1996

6. Wang
30 Years of the HCI Community
6
ACM
SIGCHI:
9
Turing
Award
Winners
/
188
ACM
Fellows
http://dl.acm.org/sig.cfm?id=SP923

7. Wang
What’s Changing in HCI Today?
Big
picture
is
sll
there,
but:
• More
emphasis
is
on
use
contexts
and
applicaons.
• Computers
are
of
many
forms,
doing
all
sort
of
things.
• Compung
is
not
necessarily
done
by
silicon
chips
computers.
-­‐ Input
and
output
are
versale.
Not
necessarily
“keyboard
and
mouse”,
“text,
speech
or
graphics”
-­‐ Collaboraon
and
social.
Not
necessarily
“one
human,
one
computer”.
7

8. Wang
Computer-Mediated Communication (CMC)
8

9. Wang
What’s
the
longest
distance
in
the
world?
世界上最遠的距離是什麼？
9

10. Wang 10

11. Wang
Supporting Human Communication
Communicaon
in
the
sense
of
data
transmission
across
physical
distance
is
not
that
hard
today
• Wired
and
wireless
computer
networking,
internet
etc.
Communicaon,
in
the
sense
of
understanding
each
other,
or
crossing
the
“psychological
distance”
between
people
remains
hard
• Difficules
in
expressing
or
understanding
thoughts
• Barriers
between
generaons,
genders,
professions,
languages,
and
cultures.
Supporng
human
communicaon
connues
to
be
a
challenging
yet
worth-­‐of-­‐pursuing
topic
in
HCI.
11

12. Wang
Supporting Human Communication
Communicaon
in
the
sense
of
data
transmission
across
physical
distance
is
not
that
hard
today
• Wired
and
wireless
computer
networking,
internet
etc.
Communicaon,
in
the
sense
of
understanding
each
other,
or
crossing
the
“psychological
distance”
between
people
remains
hard
• Difficules
in
expressing
or
understanding
thoughts
• Barriers
between
generaons,
genders,
professions,
languages,
and
cultures.
Supporng
human
communicaon
connues
to
be
a
challenging
yet
worth-­‐of-­‐pursuing
topic
in
HCI.
12

13. Wang
Ultimate Goal? Mind-Connecting!
13

14. Wang
Lost in Technologies
However,
technology
development
does
not
always
approach
the
goal
effecvely.
For
example:
Video
conferencing
• Bandwidth-­‐demanding.
Video
lagging
that
disrupts
conversaon
• Adopon
is
not
guaranteed
.
Privacy
and
other
social
concerns
Machine
translaon
• Quality
concern
• Influent
second
language
can
beat
MT
(cf.
Yamashita
Ishida,
2006).
14

15. Wang
Observation
Designs
of
CMC
can
work
be-er
when
features
and
constraints
of
human
communicaon
are
invesgated
and
considered.
Ex.
Awareness
indicator
that
makes
“typing”
visible
in
instant
messaging.
Basic
research
stays
relevant!
What
are
the
features
of
successful
and
unsuccessful
communicaon?
What’s
the
nature
of
“understanding”?
15

16. Wang
Grounding Communication
16

17. Wang
Language as a Cognitive Inquiry
17
Noam
Chomsky

18. Wang
Language as a Social Inquiry
18
Pragmacs
(linguiscs),
language
use
(psychology,
communicaon)

19. Wang
How Would You Describe…
Where
you
live
in
Hsinchu?
Where
you
lived
when
you
were
in
U.S.?
19

20. Wang
My Answer
Where
you
live
in
Hsinchu?
Near
清大後門.
Where
you
lived
when
you
were
in
U.S.?
In
Ithaca,
a
college
town
in
the
middle
of
New
York
state
if
you
know
where
it
is.
It’s
where
Cornell
University
is
located.
20

21. Wang
My Answer
Where
you
live
in
Hsinchu?
Near
清大後門.
Where
you
lived
when
you
were
in
U.S.?
In
Ithaca,
a
college
town
in
the
middle
of
New
York
state
if
you
know
where
it
is.
It’s
where
Cornell
University
is
located.
Do
you
see
the
general
difference?
Why?
21

22. Wang
My Answer
Where
you
live
in
Hsinchu?
Near
清大後門.
Where
you
lived
when
you
were
in
U.S.?
In
Ithaca,
a
college
town
in
the
middle
of
New
York
state
if
you
know
where
it
is.
It’s
where
Cornell
University
is
located.
Do
you
see
the
general
difference?
Why?
The
amount
of
knowledge
that
we
shared.
22

23. Wang
Common Ground
23
Knowledge,
beliefs,
aitudes
we
share,
and
know
that
we
share,
and
know
that
we
know
that
we
share,
influence
how
we
use
language
to
communicate.
Grounding:
Interacve
process
by
which
communicators
exchange
evidence
of
their
understanding
to
arrive
at
the
state
of
common
ground.
Herbert
Clark

24. Wang
Evidence of Common Ground
Physical
co-­‐presence
(being
co-­‐located)
• “close
that
door”
Shared
community
membership
• “Let’s
meet
at
小七”
Linguisc
co-­‐presence
(can
access
same
u-erances)
24

25. Wang
Evidence of Common Ground
Physical
co-­‐presence
(being
co-­‐located)
• “close
that
door”
Shared
community
membership
• “Let’s
meet
at
小七”
Linguisc
co-­‐presence
(can
access
same
u-erances)
25
“What’s
this?”

26. Wang
Grounding is a Collaborative Process
26

27. Wang
The Role of Media: Affordances
An
influenal
HCI-­‐rooted
concept,
which
roughly
means
“acon-­‐permiing
properes”
of
objects
that
people
see
• Chair
affords
siing
• Door-­‐knob
affords
door-­‐opening
• Virtual
keyboard
affords
typing
(but
is
this
trivial?)
27
Don
Norman

28. Wang
Affordances of Communication Media
28

29. Wang
Technology Changes Grounding
Affordances
of
media
constrain
how
people
may
interact
with
one
another
• E.g.,
if
no
visibility,
impossible
to
use
head-­‐nodding
as
a
technique
for
grounding
People
may
learn
to
adapt
their
grounding
behaviors
(this
happens.
E.g.,
emocons
in
IM)
or
Design
new
CMC
tools
with
useful
proper7es
to
support
grounding
and
communica7on.
29

30. 30

31. 使用體感裝置探討在電腦中介傳播下之手勢使用行為
使用電腦作為訊息傳遞媒介進行人與人間的溝通已經是一個普遍的現象，我們亟需瞭解以電腦為中介
之溝通模式與面對面溝通的模式之間到底有那些差異，對於人際溝通的影響為何。過去這方面的研究
多著重在媒介的性質對於信任及語言使用的影響。對於非語言的溝通行為，例如溝通手勢的使用則探
討有限，其中一個原因在於缺少可快速有效量測細微手勢的方法。本論文提出一個應用技術，利用體
感裝置Microsoft Kinect來捕捉人與人溝通時肢體動作細微的變化。透過對肢體移動速度的分析以及多
重特徵值的截取，我們得以實驗比較面對面溝通(Face-to-Face)、視訊通訊(Video)與音訊通訊(Audio)三
種不同媒介對於溝通手勢行為所產生的影響，包括了手勢使用的程度以及兩個溝通者間行為的相似度。
此運用體感裝置作為行為科學量測工具的方法可用於快速評估新設計之線上溝通介面對於溝通行為的
影響，亦可用於傳播理論研究之發展與探討。在設計上，所提出之資料收集與分析方法亦可能作為未
來電腦中介傳播工具設計的基礎。
Microsoft Research Asia UR Project: FY13-RES-OPP-027
Wang, H-C., Lai, C-T. (accepted). Kinect-taped Communication: Using Motion Sensing to Study Gesture Use
and Similarity in Face-to-Face and Computer-Mediated Brainstorming. ACM Conference on Human Factors in
Computing Systems (CHI) 2014. Full paper. [Acceptance rate: 22.8%]

32. Kinect-taped Communication:
Using Motion Sensing to Study Gesture Use
and Similarity in Face-to-Face and
Computer-Mediated Brainstorming
Hao-Chuan Wang, Chien-Tung Lai
National Tsing Hua University, Taiwan

33. [cf.
Bos
et
al.,
2002;
Setlock
et
al.,
2004;
Scissors
et
al.,
2008,
Wang
et
al.,
2009]
Computer-mediated communication (CMC) tools are
prevalent, but are they all equal?
• Ex. Video vs. Audio
Media properties influence aspects of communication
differently
• Task performance, grounding, styles, similarity of
language patterns, social processes and outcomes
etc.
How media influence communication?

34. Communication could be more than
speaking.
Both verbal and non-verbal channels are
active during conversations.
Facial
expression
Gesture
[cf.
Goldin-­‐Meadow,
1999;
Giles
Coupland,
1991
]
The (missing) non-verbal aspect in
CMC research

35. Studying gesture use in
communication
Current methods:
• Videotaping with manual coding.
• Giving specific instructions to participants
(e.g., to gesture or not).
• Using confederates etc.
Problems to solve:
• High cost. Labor-intensiveness.
• Resolution of manual analysis-
Hard to recognize and reliably label small
movements.
• Scalability-
Hard to study arbitrary communication in the wild.

36. “Kinect-taping”method
Like videotaping, we use motion sensing devices,
such as Microsoft Kinect, to record hand and body
movements during conversations.
• Detailed, easier-to-process representations.
• Behavioral science instrument (“microscope”) to
study non-verbal communication in ad hoc
groups.
• Low cost if automatic measures are satisfactory.

37. Re-appropriating motion sensors in HCI:
Sensing-aided user research for
future designs
From sensors as design elements to sensors as
research instruments to help future designs.
!
(a)!Face(to(face!(F2F)!communication!
[cf.
Mark
et
al.,
2014]

38. A media comparison study
Investigate how people use gestures during face-
to-face and computer-mediated brainstorming
Compare three communication media
• Face-to-Face
• Video
• Audio
!
(a)!Face(to(face!(F2F)!communication!
!
(b)!Video(mediated!communication!
Figure'1.'A'sample'study'setting'that'compares'(a)'F2F'to'(b)'videomediated'communication'
by'using'Kinect'as'a'behavioral'science'instrument.'
!

39. Hypotheses
H1. Visibility increases gesture use
Proportion of gesture
Face-to-Face Video Audio
H2. Visibility increases accommodation
Similarity between group members’ gestures
Face-to-Face Video Audio
Also explore how gesture use, level of
understanding, and ideation productivity correlate.
[cf.
Clark
Brennan,
1991]
[cf.
Giles
Coupland,
1991]

40. Experimental design
36 individuals, 18 two-person
groups
Kinect-taped group brainstorming sessions
Face-to-Face
Video
Audio
Three
trials
(15
min
each)
in
counterbalanced
order
Data analysis
Amount and similarity of gestures,
Level of understanding, Productivity

41. How to quantify gestures?
How many gestures are there in a 15 min talk?

43. moving
not moving

44. Two unit motions with speed threshold 0

45. Three unit motions with speed threshold 2

46. Choose the thresholds
(m/s)

47. Choose the thresholds
Too
few
signals
Almost
everything
Data
points
of
interest
(m/s)

48. How to measure similarity
between unit motions?

49. Feature extraction and representation
Unit motions are represented as feature vectors
• Time length, path length, displacement,
velocity, speed, angular movement etc.
• Features extracted for both hands and both
elbows.
73 features extracted for each unit motion.
Similarity between unit motions: Cosine value
between the two vectors.

50. Validating the similarity metric
1
2
3
Machine
Ranking
Human
Ranking
1
2
3
Randomly select
motion queries
Retrieve similar and
dissimilar motions
Kinect-taped motion
database

51. Count
Human Rank
R1
R2
R3
Machine
Rank
R1
29
2
5
R2
7
27
2
R3
0
7
29
x2=107.97,
p.001
Validating the similarity metric
Contingency analysis

52. H1: Amount of gesture use
H2: Similarity between group members
Associations
• Amount of gesture and understanding
• Amount of gesture and ideation productivity
• Gesture similarity and ideation productivity
Key Results

53. Visibility on proportion of gesture use
0
2
4
6
8
10
12
14
16
Face-to-face Video Audio
ProportionofGestureUse(%)
H1 not supported. Media did not influence percentage of
gesture.
People gesture as much in Audio as in F2F and Video.

54. Association between self-gesture
and level of understanding
ModelPredicted,UnderstandingModelPredicted,Numb
Propor9on,of,Individual’s,Own,Gesture,Use,(%)
Audio
F2F
Video
Individual’s Own Gesture Use (%)
Non-communicative
function of gesture.
Understanding
correlates with
self-gesture but not
partner-gesture
Stronger correlation
with reduced or no
visibility.

55. Similarity between group members
0.46
0.47
0.48
0.49
0.5
0.51
0.52
0.53
0.54
0.55
Face-to-face Video Audio
Between-participantGestural
Similarity
H2 supported. Similarity F2F Video Audio.
People gesture more similarly when they can see each other.

56. Summary and implications
Media
Comparison
Study
Kinect-
taping
Method

57. Motion sensing for
studying non-verbal
behaviors in CMC.
Summary and implications
Media
Comparison
Study
Kinect-
taping
Method
Visibility influences
similarity but not
amount of gesture.
Only self-gesture
correlates with
understanding.
Gesture doesn’t
seem to convey
much meaning to
the partner. Seeing
the partner is not
crucial to
understanding.

58. Study
communication of
ad hoc groups
in the wild.
Distributed
deployment
study of CMC tools.
Cross-lingual and
cross-cultural
communication.
Summary and implications (cont.)
Media
Comparison
Study
Kinect-
taping
Method
The value of video
may be relatively
limited to the social
and collaborative
aspect (similarity
etc.).
Feedback that
promotes self-
gesturing may help
understanding.

59. Wang
Key Messages
Supporng
human
communicaon
connues
to
be
an
important
topic
in
HCI,
both
to
research
and
design
pracce.
• Focusing
on
how
to
shorten
the
“psychological
distance”
between
people.
“Mind-­‐connecng”!
Basic
and
applied
behavioral,
cognive
and
social
sciences
helps
to
understand
the
features
of
successful
and
unsuccessful
communicaon
• Insight
that
we
should
focus
on
CMC
affordances
as
much
as
technicality.
Interdisciplinary
work
can
benefit
both
sides:
Social
and
behavioral
sciences
help
technology
design,
and
vice
versa.
59

60. Wang
Ultimate Goal? Mind-Connecting!
60

61. Wang 61
國立清華大學人機合作與社群運算實驗室
NTHU
Collaborave
and
Social
Compung
Lab
(CSC
Lab)
Acknowledgement
for
Support
from
Naonal
Science
Council,
Taiwan
Google
Inc.
Microsou
Research
Asia
微軟亞洲研究院
Industrial
Technology
Research
Instute
(ITRI)
工業技術研究院
Delta
Corp
台達電子公司
Naonal
Science
Foundaon,
USA