Paper for the First Workshop on Argumentation Mining at the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, Maryland, June 26 2014
Abstract:
Argumentation mining, a relatively new area of discourse analysis, involves automatically identifying and structuring arguments. Following a basic introduction to argumentation, we describe a new possible domain for argumentation mining: debates in open online collaboration communities. Based on our experience with manual annotation of arguments in debates, we envision argumentation mining as the basis for three kinds of support tools, for authoring more persuasive arguments, finding weaknesses in others’ arguments, and summarizing a debate’s overall conclusions.
Full paper:
http://jodischneider.com/pubs/aclargmining2014.pdf
Proceedings with links:
http://acl2014.org/acl2014/W14-21/index.html
Workshop homepage:
http://www.uncg.edu/cmp/ArgMining2014/
Continued citation of bad science and what we can do about it--2021-02-19jodischneider
Mais conteúdo relacionado
Semelhante a Envisioning argumentation and decision making support for debates in open online collaboration communities--acl arg mining workshop 2014-06-26
Semelhante a Envisioning argumentation and decision making support for debates in open online collaboration communities--acl arg mining workshop 2014-06-26 (20)
2. Argumentation mining today
• No unified vision of the field. Multiple:
– Interrelated problems
– Application domains
– Tools handling one aspect of annotation
• Few corpora
• Need for
– Common definition(s) of argumentation
– "Challenge problems"
– Shared corpora
– Applications
3. Argumentation mining today
• No unified vision of the field. Multiple:
– Interrelated problems
– Application domains
– Tools handling one aspect of annotation
• Few corpora
• Need for
– Common definition(s) of argumentation
– "Challenge problems"
– Shared corpora
– Our Application: debates in online collaboration
4. Application: Debates in Open Online
Collaboration
• Wikipedia
• HTML5
• OpenStreetMap
• Project Gutenberg
• Apache projects
• Mozilla Firefox
• …
8. Argument-based support
• How can I win an argument?
Which arguments sway the community?
• Why were previous decisions made?
• Which ongoing debates need more comments?
9. Argument-based support
• How can I win an argument?
Which arguments sway the community?
• Why were previous decisions made?
• Which ongoing debates need more comments?
10. Corpus: 72 discussions started on 1 day
• Each discussion has:
3-33 messages
2-15 participants
• 741 messages contributed by 244 users.
Each message has 3-350+ words.
• 98 printed A4 sheets
11. Approach
• Compare two argumentation theories
• Iterative annotation with multiple annotators
– Refine to get good inter-annotator agreement
• 4 rounds of annotation
– Rounds 1-2 by me
– Rounds 3-4 by 2 assistants
12. We used two argumentation theories
• Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
– Informal argumentation
(philosophical & computational argumentation)
– Identify & prevent errors in reasoning (fallacies)
– 60 patterns
• Factors/Dimensions Analysis
(Ashley 1991; Bench-Capon and Rissland, 2001)
– Case-based reasoning
– E.g. factors for deciding cases in trade secret law,
favoring either party (the plaintiff or the defendant).
13. Walton’s Argumentation Schemes
Example Argumentation Scheme:
Argument from Rules – “we apply rule X”
Critical Questions
1. Does the rule require carrying out this type of action?
2. Are there other established rules that might conflict
with or override this one?
3. Are there extenuating circumstances or an excuse for
noncompliance?
Walton, Reed, and Macagno 2008
14. Example: "Stop at a red light"
1. Does the rule require carrying out this type of action?
Were you driving a vehicle?
2. Are there other established rules that might conflict
with or override this one?
Did a police officer direct you to continue without
stopping?
3. Are there extenuating circumstances or an excuse for
noncompliance?
Were you driving an ambulance with its siren on?
Critical Questions from Argument from Rules based on Walton, Reed, and Macagno 2008
16. How to win an argument with a
Wikipedian?
• Argument from Evidence to Hypothesis (19%)
• Argument from Rules (17%)
17. How to win an argument (Arucaria)?
Classifying Arguments by Scheme. Vanessa Wei Feng. Master's thesis, Toronto, 2010.
18. Experts vs. Novices
• Experts were more likely to use
– Argument from Precedent
• Novices were more likely to use
– Argumentation from Values
– Argumentation from Cause to Effect
– Argument from Analogy
19. Unsuccessful arguments from novices
• Emsworth Cricket Club is one of the oldest
cricket clubs in the world, and this really is
worth a mention. Especially on a website,
where pointless people … gets a mention.
• Why just because it is a small team and not
major does it not deserve it’s (sic) own page
on here?
21. Factors/Dimensions Analysis
• Factors (case-based reasoning)
– All or nothing
• Either present ("applicable") or absent
• When present, a factor always favors the same side
• Dimensions
– More complex/subtle
• Can be applicable to a varying degree ("sliding scale")
• Favor plantiff on one extreme; defendant on the other
Ashley 1991; Bench-Capon and Rissland, 2001
23. Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussions
“Other” as 5th catchall
24. Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussions
“Other” as 5th catchall
Factor Example (used to justify `keep')
Notability Anyone covered by another
encyclopedic reference is
considered notable enough for
inclusion in Wikipedia.
Sources Basic information about this
album at a minimum is certainly
verifiable, it's a major label
release, and a highly notable
band.
Maintenance …this article is savable but at its
current state, needs a lot of
improvement.
Bias It is by no means spam (it does
not promote the products).
**Other I'm advocating a blanket
"hangon" for all articles on
newly-drafted players
25. Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussions
“Other” as 5th catchall
Factor Example (used to justify `keep')
Notability Anyone covered by another
encyclopedic reference is
considered notable enough for
inclusion in Wikipedia.
Sources Basic information about this
album at a minimum is certainly
verifiable, it's a major label
release, and a highly notable
band.
Maintenance …this article is savable but at its
current state, needs a lot of
improvement.
Bias It is by no means spam (it does
not promote the products).
**Other I'm advocating a blanket
"hangon" for all articles on
newly-drafted players
27. Comparison of Annotation
• Cohen’s kappa (Cohen, 1960)
.48 for Walton’s argumentation schemes
.64-.82 for factors, depending on the factor
• Potential for task support
– Argumentation schemes
• Write effective arguments
• Ask critical questions to check others' arguments
– Factors
• Summarize debates
28. Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide personalized feedback on drafts
– Requires: knowing which arguments are accepted;
identifying argumentation in a drafts
• Find weaknesses in others’ arguments
– How: suggest & instantiate relevant critical questions
– Requires: identifying argumentation schemes
• Summarize the overall conclusions of the debate
– How: identify the winning and losing rationales
– Requires: identifying rationales and contradictions
29. Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide personalized feedback on drafts
– Requires: knowing which arguments are accepted;
identifying argumentation in a drafts
• Find weaknesses in others’ arguments
– How: suggest & instantiate relevant critical questions
– Requires: identifying argumentation schemes
• Summarize the overall conclusions of the debate
– How: identify the winning and losing rationales
– Requires: identifying rationales and contradictions
30. Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide personalized feedback on drafts
– Requires: knowing which arguments are accepted;
identifying argumentation in a drafts
• Find weaknesses in others’ arguments
– How: suggest & instantiate relevant critical questions
– Requires: identifying argumentation schemes
• Summarize the overall conclusions of the debate
– How: identify the winning and losing rationales
– Requires: identifying rationales and contradictions
31. Argumentation Mining papers
Arguing on Wikipedia
• “Arguments about Deletion: How Experience Improves the Acceptability of Arguments
in Ad-hoc Online Task Groups” CSCW 2013.
• “Deletion Discussions in Wikipedia: Decision Factors and Outcomes” WikiSym2012.
Arguing in Social Media
• “Dimensions of Argumentation in Social Media" EKAW 2012
• “Why did they post that argument? Communicative intentions of Web 2.0 arguments.”
Arguing on the Web 2.0 at ISSA 2014
Arguing in Reviews
• “Identifying Consumers' Arguments in Text” SWAIE 2012
• “Semi-Automated Argumentative Analysis of Online Product Reviews" COMMA 2012
• “Arguing from a Point of View” Agreement Technologies 2012
Structuring Arguments on the Social Semantic Web
• “A Review of Argumentation for the Social Semantic Web” Semantic Web –
Interoperability, Usability, Applicability, 2013.
• “Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration
Systems" 2013 Thesis: purl.org/jsphd
• “Modeling Arguments in Scientific Papers” at ArgDiaP 2014
http://jodischneider.com/jodi.html
38. Open collaboration based on
• Technological infrastructure
• People
• Social structures: rules, policies, procedures,…
39. Open collaboration based on
• Technological infrastructure
• People
• Social structures:
– joint decision-making
– Importance of rationales: reasons for opinions
43. Results: Important tasks
for consensus discussions
1. Determine one’s personal position
2. Express one’s personal position in accordance
with community norms
3. Determine the consensus
44. Related work
• Dissent and rhetorical devices in bug reporting
(Ko and Chilana, 2011)
• how Python listservs select enhancement
proposals (Barcellini et al., 2005).
– role of a participant is related to the kinds of
message they quote (Syntheses, Disagreements,
Proposals, or Agreements)
– Syntheses and Disagreements are the most
quoted
Notas do Editor
12-12:30
20 minutes + questions
4th paper in 90 minute session 11-12:30
Paper:
http://jodischneider.com/pubs/aclargmining2014.pdf
Workshop homepage:
http://www.uncg.edu/cmp/ArgMining2014/
Proceedings with links:
http://acl2014.org/acl2014/W14-21/index.html
Abstract:
Argumentation mining, a relatively new area of discourse analysis, involves automatically identifying and structuring arguments. Following a basic introduction to argumentation, we describe a new possible domain for argumentation mining: debates in open online collaboration communities. Based on our experience with manual annotation of arguments in debates, we envision argumentation mining as the basis for three kinds of support tools, for authoring more persuasive arguments, finding weaknesses in others’ arguments, and summarizing a debate’s overall conclusions.
“people form ties with others &
create things together”
(Forte and Lampe 2013)
Factors provide a good way to organize the debate;
Filtering discussions based on each factor can show the rationale topic by topic, which supported decision making in a pilot user-based evaluation
16 of 19 participants (84%) preferred
See (Schneider et al., WikiSym 2012) and (Schneider et al., CSCW 2013)
When the argumentation scheme used in a draft message is not generally accepted, the author could be warned that their message might not be persuasive, and given personalized suggestions
Listing these questions in concrete and contextualized form (drawing on the premises, inference rules, and conclusions to instantiate and contextualize them) would encourage participants to consider the pos- sible flaws in reasoning and might prompt partici- pants to request answers within the debate.
Macro- argumentation, such as the factors analysis de- scribed above, would be a natural choice for sum- marization, as it has already proven useful for fil- tering discussions. A more reasoning-intensive approach would be to calculate consistent out- comes (Wyner and van Engers, 2010), if debates can be easily formalized.
When the argumentation scheme used in a draft message is not generally accepted, the author could be warned that their message might not be persuasive, and given personalized suggestions
Listing these questions in concrete and contextualized form (drawing on the premises, inference rules, and conclusions to instantiate and contextualize them) would encourage participants to consider the pos- sible flaws in reasoning and might prompt partici- pants to request answers within the debate.
Macro- argumentation, such as the factors analysis de- scribed above, would be a natural choice for sum- marization, as it has already proven useful for fil- tering discussions. A more reasoning-intensive approach would be to calculate consistent out- comes (Wyner and van Engers, 2010), if debates can be easily formalized.
When the argumentation scheme used in a draft message is not generally accepted, the author could be warned that their message might not be persuasive, and given personalized suggestions
Listing these questions in concrete and contextualized form (drawing on the premises, inference rules, and conclusions to instantiate and contextualize them) would encourage participants to consider the pos- sible flaws in reasoning and might prompt partici- pants to request answers within the debate.
Macro- argumentation, such as the factors analysis de- scribed above, would be a natural choice for sum- marization, as it has already proven useful for fil- tering discussions. A more reasoning-intensive approach would be to calculate consistent out- comes (Wyner and van Engers, 2010), if debates can be easily formalized.
Envisioning argumentation and decision-making support for debates in open online collaboration communities
“people form ties with others &
create things together”
(Forte and Lampe 2013)
The prototypical open collaboration system is an online environment that
supports the collective production of an artifact
through a technologically mediated collaboration platform
(3) that presents a low barrier to entry and exit, and
(4) supports the emergence of persistent but malleable social structures.
"open collaboration system" = Artifact + Technology + Ad hoc community + Social Structures
Image from http://breakingenergy.com/2013/04/08/energy-legal-work-takes-the-number-two-spot-growth-forecast/
***Picture conveying social norms/policies – e.g. growth over time from Butler (if there’s some image of that)
“people form ties with others &
create things together”
(Forte and Lampe 2013)
The prototypical open collaboration system is an online environment that
supports the collective production of an artifact
through a technologically mediated collaboration platform
(3) that presents a low barrier to entry and exit, and
(4) supports the emergence of persistent but malleable social structures.
"open collaboration system" = Artifact + Technology + Ad hoc community + Social Structures
Image from http://breakingenergy.com/2013/04/08/energy-legal-work-takes-the-number-two-spot-growth-forecast/
***Picture conveying social norms/policies – e.g. growth over time from Butler (if there’s some image of that)
The organizational relevance of these open decision making discussions in collaborative communities makes them a promising target for support, and argumentation mining technology is an appropriate tool to deploy towards that end.