Holding and Promoting Beliefs: A Develpmental View.
ODR 2013 SDSkills dashboard umass
1. An Online Deliberation
Facilitators'
Dashboard: Visualizations and
Text analysis to support quality
dialogues
Wing, L., Murray, T., Woolf, B., Katsh, E.
The Twelfth International Online Dispute Resolution Forum,
Montreal, June 2013
2. “The Fourth Party: Improving Computer-Mediated
Deliberation through Cognitive, Social and
Emotional Support”
• 3-Year NSF Social Computing grant, started
Fall 2010
• Description at
www.socialdeliberativeskills.com
2
3. Project collaborators
• Beverly Woolf: CompSci, PI (intelligent and collaborative
educational systems)
• Tom Murray: CompSci; project manager/co-PI, principal
visionary and instigator (ed-tech, cog-psych & D&D)
• Ethan Katsh (ODR), Legal Studies, co-PI
• Leah Wing (social justice and ODR), Legal Studies, co-PI
• Linda Tropp, Psychology of Peace and Violence, advisor
(intergroup relations/conflict)
• Zan Goncalves, New England Center for Civic Life
(teaching, practice and study of deliberative democracy)
• Idealogue Inc.; iCohere. Inc. software
platforms(Advanced dialogue)
3
4. Thanks for consultation
and/or data from:
• DemarsAssociates.com/PayPal/ebay (e-commerce)
• Juripax.com (online workplace and divorce
settlements)
• National Mediation Board (transportation
management/labor disputes)
• Modria.com (e-commerce plus)
• Idealogue.com (depth-oriented online dialogue
platform)
• iCohere.com (online communities and work groups)
• Mass Dept. of Dispute Resolution (civic engagement)
• New England Center for Civic Life (teaching, practice
and study of deliberative democracy)
4
5. Social Deliberative Skills:
Social/Emotional/Reflective
Perspective taking & cognitive empathy
Perspective seeking (curiosity/inquiry)
Self-reflection: on one's biases, intentions,
emotional state
Meta-dialog: Reflect on the quality of the
dialog
Epistemic skill: e.g. treating facts/data
differently from opinions/hypotheses
Tolerance for uncertainty, ambiguity,
disagreement, paradox
…
5
7. Example: Topics Chosen by Students—
set context in Spring ’12
• Week 1: Discuss the pros and cons of legalizing
marijuana
• Week 2: Sex – what's the big deal? What values are
most important in making sexual choices?
• Week 3: Discus the pros and cons of the death penalty
(capital punishment)
7
8. [CURRENT] WEEK 1: Discuss the pros and cons of leg...
UPDATE PROFILE
LOG OUT
HOME
Logged in as tomm
[CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana.[CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana.
To focus the conversation, we invite you to assume you are on an advisory panel for the state
legislature, having some preliminary conversations online, and you will eventually be drafting
a group recommendation. Consider not only your own preferences but what is best for the
state (or society).
edit delete
CONTRIBUTE YOUR THOUGHTS
14:53 EDT Sunday, November 13 by tomm
tomm has joined the conversation
23:53 EDT Saturday, November 12 by ines- v
ines-v added a resource: 'Getting a Fix'
23:52 EDT Saturday, November 12 by ines- v
I have to disagree with your third point that marijuana is a gateway drug. Of
all the people I know that smoke marijuana, they do not do any hard drugs.
I do agree that gateway drugs exist, however I feel like that typically
happens from one hard drug to another when one doesn't seem to be
enough. But if you want to talk about gateway drugs we would also have to
mention alcohol and cigarettes which many people consume and smoke.
Alcohol and cigarettes are also drugs and often considered gateway drugs.
They are both legal so that option is void in regards to marijuana.
You also mentioned cancer and other lung related issues. Marijuana is a
natural plant. Cigarettes are made up of extremely harmful chemicals that
cause lung related issues and cancer much faster than marijuana ever could.
Yet, they are still legal. If anything, cigarettes should be illegal when
considering public health. Marijuana is a lot safer than cigarettes.
I do appreciate you playing Devil's advocate though!
I'd like to explain how I see it differently (ines-v)
18:26 EDT Friday, November 11 by arthur- x
It seems like the vast majority is supportive of the legalization of marijuana,
so I'm going to play devil's advocate in order to bring the opposition's side
to the table.
First off, research has demonstrated that marijuana use reduces learning
ability by limiting the capacity to absorb and retain information. A 1995
study of college students discovered that the inability of heavy marijuana
users to focus, sustain attention, and organize data persists for as long as 24
hours after their last use of the drug. Earlier research, comparing cognitive
abilities of adult marijuana users with non-using adults, found that users fall
short on memory as well as math and verbal skills. Although it has yet to be
proven conclusively that heavy marijuana use can cause irreversible loss of
intellectual capacity, animal studies have shown marijuana-induced
ines-v
arthur-x
joseph-t
laura-t
rtwells
matthew-s
tomm
DIALOGUE TABLE
Everyone (no demographics set)
8
10. Experimental Conditions
Exp Group N Gender Grade
Vanilla 8 (5 Female,
3 Male)
4 soph, 4 juniors,
0 seniors
Reflective Tools 8 (5 Female,
3 Male)
4 soph, 2 juniors,
2 seniors
(Sliders) 8 (Group omitted due to interaction issues)
10
• V&R groups: 241 posts and 516 segments (average
of 15.06 (SD = 7.45) posts/student)
• Mean words/post = 54 (SD = 42); mean
characters/post = 299 (SD = 242)
11. Text Coding
11
Soc. DELIBERATION
Skill Evidence
MISC CODES ACTION
NEGOTIATION
ARGUMENT CODES
SELF_REFLection
_INTERSUBictive
Q_INTERLocutor
REF_INTERLocutor
PERSPECTIVE_taki
ng
_META_Dialog
MEDIATE
META_CONS
META_CONFL
META_SUM
META_CHECK
_META_TOPIC
WEIGH
SYSTEMs_thinking
FACT_cite_SouRCe
SOURCE_REFerence
APPRECiation
Q_TOPIC
CHANGE_mind
UNCERtainty
OTHERS_THNK
APOLOGY
HELP
REQ_HELP
PROCESS
AGREE
DISAGREE
_NEGative-emotion
NEGEMO_INTer
locutor
NEGEMO_Topic
_OFFTOPIC
TECHnical
SOCIAL
(External actions)
ActRequest
ActPropose
ActAccept
ActDecline
ActNegot
(Dialogue_Actions)
DI_ActRequest
DI_ActPropose
DI_ActAccept
DI_ActDecline
DI_ActNegot
(Facilitators only)
WELCOMING
PROC_EXPL
MOTIVATE
_ARGument_GENeric
FACT_NOSRC
GENERAL_SOLUTN
EXPER_OBSERV
ARG_OPINION
OPINION_ONLY
OVER_GEN
SUPPORT
SUM_MY-argumt
EXAMPLE
ELAB
12. Main Effect
Exp. Group Total_
SD_Skill
Intersubjective
speech acts
Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09)
Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08)
12
• A significant difference and main effect between
Total-SD-Score and grouping, F(1, 14) = 6.89, p =
0.02*, d = 1.46 (a large effect) in favor of the
Reflective Tools group
• A significant relationship between Intersub and
grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large
effect) in favor of the Reflective Tools group
16. Applicability and Use
Facilitators
• To identify individual and group participation
levels for assessing useful interventions
(ie: to stimulate more involvement, more
constructive engagement, etc.)
• To identify patterns of ‘silence’ or intensity
(ie: to ID topics/relationships to attend to;
use as clues to search text analysis for
conflicts or breakthroughs)
17. Applicability and Use
Participants
• For self/group assessment of engagement,
topical, and relationship patterns re:
participation, intensity, silence
• For clues to search text analysis for
conflicts or breakthroughs
22. Future Applications
Common problems encountered in online facilitation
• Low or no participation of individuals or groups, or
silences or lulls on the part of individuals, the entire
group, or sub-groups
• Conversation domination by an individual or group
• Inappropriate or disrespectful behavior
• Off-topic conversation
• Tension-filled disagreements, or high emotional
content
• Too much agreement or politeness
• Misunderstanding due to missing communication skills
normally available in face-to-face communication
• Violation of rules (e.g. confidentiality, no advertising,
etc.)
26. Automated Text Analysis
LIWC (Pennebaker et al.) – Dictionary-based (linguistic
inquiry word count)
– 4,500 words/STEMS; 80 word categories
– we focus on 19 of them
– 80 >> 4 general descriptor categories (word count, words per sentence, % of words captured, and %
of words >6 letters), 22 standard linguistic dimensions (e.g., % pronouns, articles, auxiliary
verbs, etc.), 32 psychological constructs (e.g., affect, cognition, biological processes), 7 personal
concern categories (e.g., work, home, leisure activities), 3 paralinguistic dimensions
(assents, fillers, nonfluencies), and 12 punctuation categories (periods, commas, etc).
– Relate the categories to things like aggression, used in theraputic contexts, to ID lying,
• Coh-Metrix (Graesser et al.)
– syntax, referential cohesion, semantic cohesion, rhetorical composition…
– 100 measurements categories
– We focus on 4 composite measurements (or major
factors): Narrativity, Referential Cohesion, Syntactic
Simplicity, and Word Concreteness
29. Total Skill score adds:
• Appreciation (Gratitude, affirmation of another's idea
or situation)
• Apology
• Fact--sourced (stating a fact and noting the source in
the same post)
• Source Reference (Mentioning a source, with a
reference or description; without a fact)
• Intersubjectivity: perspective taking or question asking
• Meta-dialogue, discussing the quality of the dialogue
• Meta-Topic: Birds eye or systemic view of the topic
29
30. Next: Linked Representations
• clicking on the name of an individual or group in a chart or
network diagram will focus (or filer or highlight) all tools on
that individual or group;
• clicking on a link in the network diagram will show posts
between the relevant interlocutors;
• clicking on a word in the word cloud will highlight posts
including that word;
• clicking on a location in the time-axis of a trend line will
navigate to posts in the Timeline at that time;
• hovering over an agent trigger will show the Advice or Alert
associated with that event; and clicking on Advice or Alerts
will navigate to the place in the dialogue timeline for the
triggering event(s).
31. Text Coding
31
Soc. DELIBERATION
Skill Evidence
MISC CODES ACTION
NEGOTIATION
ARGUMENT CODES
SELF_REFLection
_INTERSUBictive
Q_INTERLocutor
REF_INTERLocutor
PERSPECTIVE_taki
ng
_META_Dialog
MEDIATE
META_CONS
META_CONFL
META_SUM
META_CHECK
_META_TOPIC
WEIGH
SYSTEMs_thinking
FACT_cite_SouRCe
SOURCE_REFerence
APPRECiation
Q_TOPIC
CHANGE_mind
UNCERtainty
OTHERS_THNK
APOLOGY
HELP
REQ_HELP
PROCESS
AGREE
DISAGREE
_NEGative-emotion
NEGEMO_INTer
locutor
NEGEMO_Topic
_OFFTOPIC
TECHnical
SOCIAL
(External actions)
ActRequest
ActPropose
ActAccept
ActDecline
ActNegot
(Dialogue_Actions)
DI_ActRequest
DI_ActPropose
DI_ActAccept
DI_ActDecline
DI_ActNegot
(Facilitators only)
WELCOMING
PROC_EXPL
MOTIVATE
_ARGument_GENeric
FACT_NOSRC
GENERAL_SOLUTN
EXPER_OBSERV
ARG_OPINION
OPINION_ONLY
OVER_GEN
SUPPORT
SUM_MY-argumt
EXAMPLE
ELAB
32. Social Deliberative Skills:
Social/Emotional/Reflective
• 1. Social perspective taking
(cognitive empathy, reciprocal role
taking...)
• 2. Social perspective seeking (social
inquiry, question asking skills...)
• 3. Social perspective monitoring
(self-reflection, meta-dialogue...)
• 4. Social perspective weighing
(reflective reasoning; comparing and
contrasting views...)
32
35. Social Deliberative Skill:
application of HOSs to me/you/we
Higher Order Skills
• argumentation
• critical thinking
• explanation & clarification
• inquiry/curiosity
(question asking & investigation)
• reflective judgment
• meta-cognition
• epistemic reasoning
Apply these skills, not to
EXTERNAL REALITY (“IT”/problem
domain) but to the
INTERSUBJECTIVE domain
Higher Order Skills applied to:
SELF
goals; level of certainty;
feelings, values, assumptions…
YOU
goals, assumptions, feelings,
values; perspective taking;
"believing" & cognitive empathy…
WE
agreements, goals; quality of
the discourse/collaboration;
differences and similarities in
values, beliefs, goals, power, roles…
PCP, e-democracyhttp://www.franklinpierce.edu/institutes/neccl/IF: “expand the scope and health of our public discussions”
Students who posted fewer than 5 times for both topics combined are excluded ;One student failed to follow instructions (did not use the sliders). This student dominated the discussion, contributing over a third of the total posts. This student’s posts were longer than average, constituting 41% of the total length of the conversation of this group, as gauged by the total number of characters typed. Two other students in this group did not post enough to be included in the analysis. One student wrote a note to the facilitator claiming that one student in this group seemed overly critical and not respectful, which affected her feeling of safety. The tension here may have put a damper on the entire group
Features that are predictive for the classification task are unknown from the literature -- we are the first to explore this research territory of identifying social deliberative skills. Social deliberative skills have a hierarchical structure, which implies that we are tasked with doing hierarchical classification with multi-classes. Our data annotated with social deliberative skills are highly skewed, because some skills are used much more often than others.
