Designing Social Interactions in a Teachable Agent

Erin Walker
School of Computing, Informatics, and Decision Systems
Engineering
Arizona State University
Designing Social Interactions in
Teachable Agents

Spoken Dialogue Systems
Spoken dialogue systems are
becoming increasingly prevalent
(Siri, Alexa, Google Now).
These systems have to sustain
interactions with people over longer
periods of time.
It matters what these systems say,
but also how they say it.

Teachable agents
Betty’s Brain -- Leelawong & Biswas, 2008

Cognitive Factors
Novices benefit from helping others
 Increased attention
 Increased reflection
 Elaboration of knowledge
 Co-construction of knowledge
Roscoe & Chi, 2007; Duran & Monero, 2005; Webb, 2013
These benefits transfer to teachable agents.

Social Factors
Less well understood in learning by teaching
 Students experience a protégé effect Chase et al., 2009
 Students build self-efficacy through learning by
teaching Medway & Barron, 1977
 Feelings of rapport for one’s collaborating partner
may enhance the effects of learning by teaching
Ogan et al., 2012

Research Questions
1. How do social factors enhance benefits of
learning by teaching?
2. How can we design a teachable agent
respond socially to a student in order to
enhance the benefits of learning by teaching?

Goal: examine social interactions in
teachable agent environments.
How do social factors influence
learning by teaching?
How can social factors be leveraged
by an intelligent agent?
Implications and future directions

Two examples
Prior knowledge, social
engagement, & a robotic
teachable agent.
Rapport & peer tutoring

Two examples
teachable agent.
Rapport & peer tutoring
Kasia Muldner Victor Girotto
Cecil Lozano Win Burleson

Students teach a robot about plotting a point on
a graph.
rTAG Project

11
Problem: Plot the point (3,1)
In the coordinate (3,1), the
number 1 corresponds to
the y coordinate, right?
Move 3
Projected Coordinate System Mobile Interface

12
We got that
right because
we tried hard!
Problem: Plot the point (3,1)
Projected Coordinate System Mobile Interface

3 Conditions
Does the use of the robotic learning environment increase social
engagement over the use of a virtual agent?
Does the increased social engagement lead to increased learning?
rTAG eTAG vTAG

Participants and Procedure
35 participants, 4th-6th grade, 18 male, 17 female.
Procedure
 5 minute geometry review
 15 minute pretest
 20 minutes of training
 45 minutes of problem-solving
 15 minute posttest
 Self-report questionnaire

Measures
Learning:
 Isomorphic, counterbalanced pre and posttest forms,
11 questions
Social Perceptions Bartneck et al., 2009
 Animacy
 Likability
 Intelligence
 Trustworthiness

Results
High prior knowledge students had higher social
perceptions of vTAG, while low prior knowledge
students had higher social perceptions of rTAG.
Effects of prior knowledge on social perceptions
vTAG rTAG
High prior
knowledge + -
Low prior
knowledge - +

Results: Deeper Dive
While not significant, in rTAG and eTAG, better
social perceptions were negatively correlated
with learning, while in vTAG, social perceptions
were positively correlated with learning.
Effects of social perceptions on learning
vTAG rTAG
Better social
perceptions + -
Worse social
perceptions - +

Discussion
While low prior knowledge students responded more
positively to rTAG, those students that responded more
positively learned less.
 Low prior knowledge students may have learned less in
general
 Low prior knowledge students may have been more
easily distracted by the additional features of rTAG
Limitations: small sample size, short term interaction
Important to socially engage students without creating
distracting elements.

Two examples
teachable agent.
Rapport & peer tutoring Amy Ogan Samantha Finkelstein
Ryan Carlson Justine Cassell

Peer Tutoring Context
8th-10th grade students, literal equation solving
54 friend dyads, 6 stranger dyads
Procedure
 20 minute pretest
 20 minute individual work
 60 minute computer-mediated tutoring (students were
randomly assigned to role)
 20 minute posttest

Coding Scheme
4 conversational factors
 Playfulness to lighten the mood or mitigate negativity
 Face-threat remarks directed toward the partner
 Attention-getting to draw the partner back on task
 Emphasis to add emotive features
2 social functions
 Positivity (e.g., politeness, empathy, praise,
reassurance)
 Impoliteness (e.g., cooperative and uncooperative
rudeness)

Results
Friends learn more than strangers.
In friends, the combination of face threat and
positivity improved learning.
 When a tutee exhibits face threat, a tutor will focus the
conversation on task-related features
 Positivity enhances the learning benefits of these face-
threatening acts
In strangers, face threat was a negative predictor
of learning gains.

Discussion
rTAG: Students socially engaged more with the
robot, but this decreased their learning.
Rapport and tutoring: If collaborating partners felt
rapport, they appeared to challenge each other
more and reflect better on the domain.
Social engagement interacts with the context to
produce positive or negative effects.

Entrainment
Acoustic-Prosodic Entrainment: Two speakers adapt
their acoustic-prosodic features including their tone,
intensity, and speaking rate to mirror to one another.
Levitan et al. 2012
Correlated with
 Communicative success Borrie & Liss, 2014
 Conversational flow Nenkova, Gravano, & Hirschberg, 2008
 Rapport Lubold & Pon-Barry, 2014
Not clear how to produce acoustic-prosodic
entrainment within a dialogue system.

Iterative process
Prototype different
methods of pitch
adaptation
Evaluate the effects of
pitch adaptation in a
learning context
Nichola Lubold
Heather Pon-Barry

Dialogue System
Domain: Literal equation
solving
Students walk the agent,
Quinn, verbally through
steps displayed on the
screen.
S: We will divide both sides by negative
six
Q: Can you explain why we divide?
S: On the left hand side, we have
negative 6y. We need to have it
equal just y so we need to get rid of
the negative six. The easiest way is to
divide.
Q: Thank you for explaining! I get it now.
So we divide. Then what?

Explore 4 methods of pitch-
adaptation
Entrainment: low high medium medium
Naturalness: high mdium high low
Rapport: low high high high

Data Collection
Collected 32 dialogues from four college
students.
Gender of Quinn’s voice chosen to match the
gender of the student.
Each student encountered all forms of pitch
adaptation (order counterbalanced).
5.4 minutes per dialogue (30 turns per dialogue).

AMT Evaluation
Selected 10 exchanges per student in each
condition (160 possible exchanges).
174 AMT workers listened to each exchange and
answer questions regarding the speakers
 Naturalness: Quality of the voice from very poor to
completely natural (Likert scale)
 Rapport: Understanding & closeness between two
speakers
 E.g., “Alex and Quinn understood each other.”
 From Gratch et al., 2007

Results
Shift + contour is more natural than other
methods, equivalent to control (p < 0.001).
Different properties of exchanges:
 Quinn speaks first or second
 Quinn speaks socially
However, when Quinn speaks second for social
exchanges, shift+contour produced more rapport
than the other three methods (p < 0.001).

Summary
Prototyped four different entrainment methods.
Findings
 Shift+contour appeared to be the best way of
mimicking pitch-based entrainment.
 A less effective method of entrainment was perceived
as worse than doing nothing at all.
 Perceived rapport depended on when entrainment
was used.

Iterative process
Prototype different
methods of pitch
adaptation
Evaluate the effects of
pitch adaptation in a
learning context
Nikki Lubold
Heather Pon-Barry

Next step
Two factors
 Shift+contour pitch-based entrainment
 Social turns in the dialogue system
How does acoustic-prosodic entrainment and
social content influence:
1. Social variables like mutual attention and rapport
2. Learning

Three Conditions
Control
Social
Voice plus
social
Students explain problem steps to
Quinn, and Quinn responds.

Conditions
Control
Social
Voice plus
social
Quinn responds with social dialogue moves
in addition to task-related dialogue moves.
Social responses occur 15-20% of the time.

Conditions
Control
Social content
Voice plus
social
Quinn responds socially to the
student, and shifts pitch to match
the student’s pitch.

Social Dialogue & Voice Adaptation
Social dialogue
with voice
adaptation
“We factor? Ok. Math must be
your forte, you are so good at this”
Non-social, non-
adaptive
“Oh, are you saying we factor out
the two? Then we can use it to
help simplify? That makes more
sense”

Method
43 undergraduate students explained 6 problems
to Quinn
 16 voice plus social, 14 social, 13 control
 24 female, 24 male
Collected self-report responses relating to rapport
and mutual attention.
Coded dialogue for rapport-building and rapport-
hindering behaviors.

Results: Mutual Attention
Condition has a significant
effect on mutual attention (p =
0.02).
Social condition is significantly
lower than the other two
conditions (p = 0.02).
Effect is driven by the male
participants (p < 0.01).
5.5
4.9
5.57
Mutual Attention
Control Voice+SocialSocial

Results: Self-Reported Rapport
No significant effects of
condition on rapport.
But, there were gender effects:
Females feel more rapport than
males (p = .006).
4.7
5.6
Rapport (p < 0.01)
Males Females

Rapport-Building Dialogue
0
20
40
60
80
100
120
140
Males
Females
Males responded best to the
voice+social condition (as
expected), while females
responded best to the social
condition (p < 0.05).
Some evidence that differences
between males and females are
due to the expectations regarding
entrainment.

Discussion
Condition and gender influences social
engagement, rapport-building and rapport-
hindering behaviors.
 Males responded well to the voice+social condition.
 Females appeared to perceive it as unnatural.
Next step: Entrain more dynamically, adapt
based on gender.

Summary
Goals: Explore the intersection between agent
social responses, human social responses, and
learning by teaching interactions.
Three related areas of work:
 Designing a teachable robotic agent
 Study of human-human tutoring dialogues
 Research program to build a teachable agent that
produces rapport through verbal and non-verbal cues

Summary
Social engagement is not always good.
 Robotic form improves social perceptions over a virtual
agent, but only for those with low prior knowledge.
 Improved social perceptions of the robot lead to less
learning.
In human-human dialogues, higher rapport -> more
learning, due to playful challenges in dialogue.
Attempts to create rapport using entrainment and
social dialogue yielded mixed results.
 Dependent on context
 Dependent on gender

Discussion
Social factors are hard to get right, easy to get
wrong.
 For some students, learning from the robot may have
been distracting.
 In the Mechanical Turk study, the least natural form of
entrainment was worse than the control.
 For men, social condition without pitch adaptation
appeared worse than control.
 For women, social condition with pitch adaptation
was worse than the control.

Discussion
Context is important
 Individual students may be more or less receptive to
these kinds of interventions (gender, prior knowledge)
 Social engagement can be influenced by subtle cues
We need to think deeply about strategies of
adaptation as they interact with context and
individual differences.

Conclusions & Future Work
Are social adaptations worth it?
 We know they influence cognitive factors
 May be more critical as users interact with technology
across contexts and over time
 May be more critical for disengaged students
 Choices we make might have unanticipated social
impacts
Important to understand how the choices we make in
technology design interact with individual differences
and social perceptions to influence learning.

Thanks!
Collaborators: Victor Girotto, Amy Ogan,
Samantha Finkelstein, Ryan Carlson, Justine
Cassell, Nichola Lubold, Heather Pon-Barry, Kasia
Muldner, Cecil Lozano, Win Burleson, Ruth Wylie

Designing Social Interactions in a Teachable Agent

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Semelhante a Designing Social Interactions in a Teachable Agent

Semelhante a Designing Social Interactions in a Teachable Agent (20)

Mais de diannepatricia

Mais de diannepatricia (13)

Último

Último (20)

Designing Social Interactions in a Teachable Agent