Lecture Polimi April2021

Deliberation Technologies: State of the Art, Current Limitations,
and Future Research to enable Contested Collective Intelligence.
idea.kmi.open.ac.uk
Dr. Anna De Liddo
Senior Research Fellow

Anna De Liddo
Research Fellow
Lucia Lupi
PhD Student Urban
Informatics
Alberto Ardito
Web Developer
Retno Lasarti
PhD Student Explainable AI
Lucas Anastasiou
PhD Student
Riccardo Pala
Web Developer
Michelle Bachler
Senior Project Officer

Deliberation in Panning Practices
vThe actual idea of deliberative democracy and citizen
involvements in planning practices has deep roots in
planning theory.
vIt developed and evolved from one theory to another
changing the emphasis given to different aspects and
issues related to the problem of participation in planning
practices

“Planning and Design as making sense together in
practical conversations” Forester, J. (1984) "Designing: Making Sense
Together in Practical Conversations." Journal of Architectural Education (1984-), Vol. 38, No. 3
(Spring, 1985), pp. 14-20 38(3): 14-20.
vplanning as sensemaking aims at building mutual
understanding through a process of deliberation involving
diverse expertise, organizations, interests groups and
enlarged community members.

and “planning as making sense together in practical
conversations” Forester
vThese conversations are highly bounded by organizational,
political and cultural matters and are practical in the sense
of being compelled by contingent issues and case oriented
topics.
vDuring these conversations, participants make sense and
interpret planning problems and discuss planning solutions

Why Discussion and Deliberation?
vdiscussions put sentences in context and make this context
explicit. Thus enabling “Reading Problem Context and
Desire”
v“is not simply a matter of instrumental problem solving it is
a matter of altering, respecting, reflecting, acknowledging,
and shaping”
v“design activities evolves through the communicative
actions of participants in practical conversations”

vlearning opportunities happen when “contra-dictions”
occur. Contradictory sensemaking in conversation is a local
manifestations of structural problems
vConversations are charged with words and contents which
capture the political, historical, organizational and social
context in which design deliberation takes place
vspeakers may take multiple “roles” in a planning
conversation, and therefore are reproductive and social-
identity-shaping as well
Why Discussion and Deliberation?

and “planning as making sense together in practical
conversations” Forester, J. (1984)
vAware of biases and limitations of any design in terms of
comprehensive and democratic representation of all
interests
vdeliberation as a possible way to enter in the very
“reasons” behind the inevitable design biases, as a way to
study the character, motivations and implication of the
bounded rationality followed in the design practice

How can we Enable Large-Scale Public Deliberation?
Can Collective Intelligence emerge by Enabling
Structured Collective Dialogue, Debate and
Deliberation?
Technology Mediated Deliberation

Deliberation
vDeliberation is the careful discussion before decision, and
it can be defined as the thorough dialogical assessment of
the reasons for and against a measure before a decision is
made.
vWhen teams are geographically distributed, decision
making is made more difficult by the fact that these
thorough conversations cannot happen face-to-face, with
people sitting in the same room.

Online Deliberation
vDeliberation carried out online, with social media and
online discussion technologies that are
• generally limited in features,
• not designed to support decision making, and
• often produce polarisation, division and conflict
(Sunstein, 2018; Golbeck et al., 2017; Matias et al., 2015,
Binder et al., 2009).

vare rudimental in the way
they structure data
vlack of content quality and
variety
vscarcely support evidence-
based reasoning
vlack features to enhance
personal understanding
and situational awareness
The discussion spaces that we see on the Web today are flawed
Setting the Problem: Technical Dimension

… and produce well known negative socio-technical
effects, such as
ü ‘echo chambers” and the activation of biased information
dynamics (Ditto & Lopez, 1992; Taber & Lodge, 2006)
ü Homophily and Lack of content variety (Huckfeldt & Sprague, 1995,
Mutz & Martin, 2001).
ü polarisation, division and conflict (Sunstein, 2018; Golbeck et al., 2017;
Matias et al., 2015, Binder et al., 2009)
ü degrade the quality, balance and safety (Golbeck, 2017; Guntuku, 2017)
of online discourse, up to undermining social tolerance (Mutz 2002)
Setting the Problem: Social/Organisational Dimension

Current solutions
Can be classified in three class of Systems:
ü Time-Centric
ü Question-centric
ü Issue-Centric (Klein 2012).

Flat listing of
posts and no
insight into
the logical
structure of
ideas and
arguments:
such as
coherence or
evidential
basis of an
argument.
• highly
scattered
content,
• low signal-to-
noise ratios
(due to large
levels of
repeated and
irrelevant
content)
Time Centric Systems: email, chat rooms, forums and
common social media

• Poor Debate: No tools to identify were ideas contrast, where
people disagree and why...
Reward popularity vs critical thinking
toxic dynamics (where the discussion is dominated by trolling and repetitive “flame wars”), and
dysfunctional argumentation (where bias and rumor dominates over clear reasoning and well-founded data)

No support for idea refinement and improvement
These tools are increasingly used to
support online debate and
facilitate citizens’ engagement in
policy and decision-making. These
are fundamentally chronological
views which offer:
• No support for idea refinement
and improvement
solo ideation (where most
contributions are relatively simple
single-user ideas and arguments
rather than more refined,
collaboratively-developed
structures)

Question-Centric Systems: Community Question
Answering such as Stackoverflow, Quora, Spigit etc
ü individuals post questions, and other individuals post
answers for these questions as well as rate the answers that
others have provided.
ü Such tools have proven highly effective at answering
“eureka” questions whose correctness is easy to verify (e.g.
questions about how to achieve a given capability in
javascript)

No ways to assess the quality of any given idea
LINK to QUORA:
http://www.quora.com/Physics/Do-
wormholes-always-have-black-holes-at-
the-beginning#answers

Community Question Answering such as
Stackoverflow, Quora, Spigit etc – Question Centric
Systems
ü Are not effective for complex open ended questions
(Mamykina et al. 2011).
ü elicit large numbers of shallow and overlapping single-user
ideas,
ü result in impractically high filtering costs.

Limitations of current solutions
ü highly scattered content (large levels of repeated and irrelevant
content),
ü platform island (reinforcement of initial biases)
ü toxic dynamics (trolling and “flame wars”)
ü solo ideation (simple single-user ideas)
ü and dysfunctional argumentation (bias and rumor dominates
over clear reasoning and well-founded data) (Klein 2012).

Issue-centric systems: A different class of Online
Deliberation Platforms
That make the structure and status of a dialogue or debate visible
Coming from research on Argumentation and CSAV, these tools make
visually explicit users’ lines of reasoning and (dis)agreements.
Deliberatorium, Debategraph, Cohere,
CoPe_it!, YourView, The Evidence Hub
etc
Curated by Canonical Debate Lab, review of:
168 Argumentation
technologies
https://docs.google.com/spreadsheets/d/1w
QShF0J3lGmIFACTvt5FLQWnKc1YdRaaCEDvRJl
cUT8/edit#gid=0

A Common Data Model: simplified IBIS
IBIS adds a simple semantic structure to the online conversation and has demonstrated
to be usable by lay people in different public debates (Iandoli et al. 2009, Klein 2012).

Some Notable Examples of Deliberation
Technologies / Argumentation-based Discussion
Tools

Kialo -
https://www.kia
lo.com/
it hosts
argumentation
based
discussion with
thousands of
users

Considerit –
consider.it
Strategic
Planning, Public
Engagement
10,000+
opinions
1,500+
participants
400+
ideas deliberated

Pol.is –
(https://pol.is/de
mo/2demo)
used all over the
world by
governments,
academics,
independent
media and
citizens (open
source)

Advantages of Argumentation-based Deliberation
Systems
ü help communities be much more systematic and complete in
their deliberations about complex and contentious topics (Iandoli
et al 2010),
ü enhance evidence based dialogue, build common ground, favour
constructive rather than confrontational discourse (Kriplean et al.,
2014),
ü support the development of shared understanding of complex
problems (Conklin and Begeman 1988), and
ü improve the quality of online argumentation (De Liddo et al.
2012; Arniani et al. 2016; Kaye 2002; Hathorn and Ingram 2002).

Current limitations of Argumentation-based
Deliberation Systems
ü lack intuitive interfaces for contributing to online dialogue
without hindering the natural flow of a conversation,
ü require a cognitive leap from users to participate and change
of mindset toward evidence-based reasoning
ü Fail to scale
ü requires a level of idea formalisation that comes at extra
costs for the users (Shipman and Marshall 1999).
ü Not interoperable with existing discussion media
ü Centralised data storage

Contested Collective Intelligence
(De Liddo 2012)

Collective Intelligence
defined as the capability to collectively solve
complex problems
vCI research seek to develop the conceptual foundations
and sociotechnical infrastructures to improve the ability of
small to large groups to act more intelligently that any
person or machine would do in isolation.

Collective Intelligence
Aggregation Approach
vCI generated by machine aggregation of networked but
isolated human intelligence
va wider challenge or work task is parceled in micro-tasks
that are then allocated to a crowd.
vCrowds work in isolation and the system meaningfully
aggregates contributions
vCrowdsorucing, Croudfunding, Prediction Markets,
ideation systems

Aggregation Approaches to CI provide
vdo not require any group awareness or collective understanding
of the problems at hand
vdo not support social interaction and communication
vno improvement of users’ activity or personal learning
therefore are less suitable
vTo improve societal awareness and civic intelligence [De Liddo
et al.2012, Schuler et al 2018];
vWhen decision-makers need to share information and move
toward consensual decisions [Romero et al. 2015].

When tackling complex and contested problems:
In contexts such as public policy or business strategy there will
almost always be contention over the right answers.
vthere may not be one worldview, or clear option
vevidence can be ambiguous or of dubious reliability requiring the
construction of plausible, possibly competing narratives;
vgrowth in intelligence results from learning, which is socially
constructed through different forms of discourse, such as dialogue
and debate.
Contested Collective Intelligence
(De Liddo 2012)

Contested Collective Intelligence Spectrum
(Argumenation-based CI)
Model of Collective Intelligence (CI):
from sensing the environment, to interpreting it, to generating good
options, to taking decisions and coordinating action...
Collec&ve(
Ac&on(
Collec&ve(
Decision(
Collec&ve(
Idea&on(
Collec&ve(
Sensemaking(
Collec&ve(
Sensing((
(

Collec&ve(
Ac&on(
Collec&ve(
Decision(
Collec&ve(
Idea&on(
Collec&ve(
Sensemaking(
Collec&ve(
Sensing((
(
Contested Collective Intelligence Spectrum
(Argumenation-based CI)

Collaborative Web Annotation and
Knowledge Mapping
http://litemap.open.ac.uk
Over 2000 users,10 different countries, 100 community groups
ü Local Area Coordinators in Leicester, to improve agency, promote digital
skills
ü Brazilian community of 1300 teachers for collaborative online learning and
collective inquiries.
Incremental Formalization

Internationalization to
English and German
Connect and Map out the
key issues and arguments
visually with LiteMap
Get the LiteMap
bookmarklet
Harvest, annotate and classify
contributions from the Utopia’s
discussion forum
1
2
3

Since its first launch in 2015, has been used
• By over 2000 users
• in 10 different countries,
• Over 100 community groups
• 560 Maps to confirm an emerging public and education impact.
• Local Area Coordinators in Leicester, LiteMap has proved to improve
agency, promote digital skills
• a Brazilian community of 1300 teachers carry out collaborative work
and coordinate online course activities with, LiteMap improve
collaborative online learning and collective inquiries.

Collective Intelligence Spectrum
Collec&ve(
Ac&on(
Collec&ve(
Decision(
Collec&ve(
Idea&on(
Collec&ve(
Sensemaking(
Collec&ve(
Sensing((
(

Structured Online Discussion and
Argumentation based Decision Making
debatehub.net
DebateHub is an online discussion tool which goes beyond
simple commenting and facilitates activities such as: collective
ideation, structured debate, and collective decision making.

v Facilitation features such as
merge, move and split ideas to
avoid duplication, redundancy
and improve idea structuring
v Analytics and Visualizations to
help sense making of the
debate
v A Phased Deliberation Process
in which online communities
can alternate ideation,
discussion and voting to
support idea selection and
decision making.

Phased, dialogue based decision making
Collective reach faster agreement when they reflect on what they
hate rather that what they like.
Uses the bag of lemons/bag of stars method (Klein and Garcia
2014)

Since its first launch in 2015, has been used
has been mostly used in the social innovation sector
• (OuiShare, Wisdom Hacker, DS&NY, CSPC, UTOPIA, I4P)
and two Urban Community Networks for democratic decision
making
• (Ganemos Madrid and AutoConsulta Ciudadana) Spain.

Not all deliberation processes are facilitated
Online
Live Instant Interactions and Hypervideo
vHow can we harness audience reaction to live events?
vHow can we analyse and visualise audience reactions to
improve collective reflection and sensemaking of Political
Debate?

New Modes of Engagement with Televised Political
Debate through Audience Feedback
democraticreflection.org
Minimal instant feedback

The present…
The past….
?The Future?

Research Questions:
• Is this new “participation experience” really informative? And to what
extent does it improve citizens’ confidence about the issues discussed?
• Do social media voices truly capture the richness of citizens’ reactions to
political debates?
• What could we learn about the audience of political election debate, and
about the debate as media event, if we had better analytical tools to
scrutinize audience’s understanding and reactions?

Real Time Audience Feedback Objectives
• promoting active engagement by enabling the audience to react to
the televised debates in new unitrusive, yet expressive, and timely
manner;
• harnessing and analysing viewers’ reactions to better understand
the audience and their debate experience;
• Enabling self and collective reflection, sensemaking and learning
through advance analytics and visualisations
• providing new metrics to assess the debate as media event in
terms of its capability to engage the audience emotionally,
intellectually, critically and democratically.

A New Method to Harness Audience Reactions
• Instant
• Nuanced meaning
• Discourse-based: Provided in form
of discourse elements
• Voluntary and non-intrusive
• Enabling analytics and
visualisations
‘Soft’ Feedback:

From Paper Prototype to an Instant Audience
Feedback Web App
• For citizens/users at large
• For analysts (political analysts, digital journalists)
• For domain experts (Politicians, Media Broadcasters)
Check it out at:
democraticreflection.org

2015 Election Debate
• Panel of 400 people
• Experiment in the wild

2017 Election Debate
• Mobile Application
• First analytics interface
• New feedback intensity interaction
• 2 panels of 20 people
• Experiment in the wild

Visual Analytics
• Personal/Self reflection Analytics
• Collective Analytics
to be viewed:
- during the live event or replay,
- Post hoc
- both static and dynamic visualisations

Advantages of the Real Time Audience
Feedback Method
The instant, nuanced feedback method we propose
provides:
• similarly powerful insights on the audience
• while preserving the accountability of the results and
addressing issues of scale
• Enables new mechanisms of civic learning and collective
sensemaking

Democratic Reflection for Peace Building Education

Advanced Visual Interfaces to improve
Sensemaking of Political Debate

new hypervideo technology for sensemaking of political
debates and enable citizens to
• detect and make sense of political manipulations,
• check facts versus speculations,
• gain new insights, and
• confidently inform their political choices.
Democratic Replay

Lessons Learned from Users Testing of Democratic
Replay comparison with BBC replay
Democratic Replay enables the main sensemaking capabilities:
• “unexpected insights on the debaters and on what they
said,”
• To “reflect on the debate in a deeper way”
• significantly better “ways to evaluate facts and evidence
• “focusing on different aspects of the debate” and
• “reconstructing the arguments that the speakers made.”
• “Assessing personal assumption” and “changing some initial
assumptions had before the debate.”

Lessons Learned from
Real World Deployments

Technical Lessons Learned
v Our CI model Works
v Success in sharing data between
different components and data
models
v Hard getting large-scale
community testing off the
ground - need to tackle
integration with existing
communities’ platforms
v paramount importance of user
interface work: CI works best
when it is transparent

• If we want to support people’s capability to question
assumptions and think critically, we need to design
spaces for personal reflection and sensemaking.
• Individual sensemaking processes need human–
machine support.
• New tools are needed to bridge political debate across
community platforms: a visual analytics and data
science approach
Lessons Learned from Users Testing of
Democratic Replay

Key Risks of Technological Enhancements
• Powerful analytical tool are often used as persuasive tools but
the same tools can be used for improving civic engagement and
learning
• Users profiling is more and more used by big corporations to
target people but it can be also used by government to provide
better services and to design effective civic learning experience
• How to we design for this second class of applications and try
preventing misuse of technology?

vSocial Media companies own the most accessible and widespread
discussion technologies available on the Web
vbusiness and organisations have to pay substantial sums to
access technologies for online discussion, on which they have no
control, and cannot be customised to organisational
needs/values
vto be granted access to these technologies organisations “trade”
their data rights, which is highly problematic; because online
dialogue systems often contain highly sensitive data, and if well-
structured and processed can be misused.
Existing dialogue technologies create economic dependence, and
lack data ethics

vThis motivates the importance to design new tools for accessible
online discussions that are decentralised.
The centralisation of dialogue technologies produces economic
dependence from big social media companies and is highly
problematic in terms of data ethics and privacy

BCAUSE Project
(bcause.kmi.open.ac.uk)
• unleashes the power of collective reasoning and sense making to improve
critical thinking and enable sounder judgements in public deliberation
• STRUCTURED DECENTRALISED DELIBERATION SYSTEMS for collective
sense/decision making
ONR - Office of Naval
Research Global (US
grant) - individual
fellowship 3 years

BCAUSE Project
REASONING FOR CHANGE
A Structured and Decentralised Discussion System
for Distributed Decision Making
BCAUSE will develop a new online discussion platform for public
deliberation around scholarly and media articles, which is equally
accessible, but more structured, decentralised, and higher data quality
than common social media.

Accessibility
Our goal is to develop new simple and intuitive User
Interfaces for structured online discussion
ü providing the many benefits of large-scale argumentation
systems while avoiding the “cognitive leap” barrier.
ü developing tools and incentive structures that help users to
contribute to “apparently” unstructured discussions that can
then lead into structured deliberation data and processes.

Advanced Visual Analytics
Our goal is to improve sensemaking of the online
discussion and transparency of analytical processes
We envision this being achieved by sophisticated analytics and
visualization tools (Ullmann and De Liddo 2018) (Klein 2015) (De
Liddo and Buckingham Shum 2014) that can summarize large
deliberation data as well as automatically generate personalized
alerts that help users learn and contribute as effectively as
possible.

Decentralisation
Transparency and accountability are essential capabilities
for social acceptance of Deliberation Systems
Key requirements for online discussion/deliberation systems to truly
promote balance, quality, and safety of online discourse :
ü decentralisation (freedom from central system/org dependence)
ü building on real people and not susceptible to bot-driven propaganda
ü enabling customisable anonymity (different level of user defined
confidentiality)
ü allowing tracking of users reputations

Decentralisation
Our goal is adopt decentralised anonymous reputation
protocols, that incentivize accountable online
discussion.
ü Blockchain and privacy-with-accountability systems (Ford, 2018)
(Zhai et al. 2016)
ü Holochain (https://holochain.org/) take an agent centric (rather
than data centric) approach

Citizen Consultation:
Government agencies as well as major project developers (e.g. for
major infrastructure projects) are increasingly motivated or even
legally required to gather and account for public input before
proposed policies or projects can be executed.
Our solution can support successful large-scale citizen engagement
and consultations, accessible interfaces for people to engage, a
decentralised data structure to preserve accountability of
opinions, intuitive visual analytics to inform decision-making.
Impact on critical real-world challenges

Communities of Practices and Virtual Professional Networks
With the increase of mobile and agile work, Communities of
Practices and professional networks need to virtually stay in touch
and formally and informally discuss within the community. These
networks need to carry on task oriented discussions that are often
technical, in depth and aimed to inform practical decisions.
Our solution will provide alternative platforms for
organisational/issue driven online discussions aimed at supporting
effective operational/organisational decision making.

Online Media Commenting:
Media companies are increasingly shutting down online
commenting sections due to the lack of existing discussion
platforms that can effectively respond to the requirements of civil,
accountable discussion.
Our solution will provide a tool to facilitate and monitor healthy
discussions, and mechanisms for users to stay anonymous but
accountable. This way to manage online identity will promote
civilised yet free speech in online media commenting.

How to Enable Very Large Scale Public
Deliberation?
a pervasive challenge for scaling up adoption of such
novel deliberation technologies is:
v Enabling collective sensemaking across community
platforms
v Defining the architecture of effective participation
v Moving from discussion-based ideation to collective
decision making - Closing up the decision making to
action cycle

Interfaces for Minimal Meaningful Participation
Real Time Analytics, Argument Mining, Fact Checking and Human
Machine Annotation
a pervasive challenge for building CI platforms is
balancing a critical tension between:
• The need to structure and curate contributions from
many people in order to maximise the signal-to-noise-
ratio and provide more advanced CI services
• versus permitting people to make contributions with
very little useful indexing or structure

How can artificial intelligence enhance and scale
collective intelligence?

First Demo
Machine Learning and NLP tools for Argument
Mining can be used to highlight “in context”
claims and evidence extracted on
debatehub.net, as a way to reflect and annotate
the discussion.
Discussion facilitators can use these machine
generated annotations to better structure the
discussion content, by: merging duplicate ideas,
splitting ideas and comments which include
more than one claim and evidence in it.
Such a machine support can reduce idea
duplication and improve content structure in the

Interfaces for Explicability and Conversational
Intelligence – to improve Trust and Accountability of
Machine Predictions
CI works best when it is transparent
• participants want to understand how their
contributions are integrated and must be given access
to visible expressions of analytics processes.
• On the other hand, the complexity of the underlying
process can also scare participants away, and much raw
data from analytics is hard to interpret without training

OUR ULTIMATE RESEARCH GOAL:
…enabling ideation and deliberation at unprecedented
scales while allowing many voices to contribute to
effective, unbiased, democratic conversations that lead
to intelligent group behaviors and positive social change.

THE VALUE WE CARE
FOR:
v AWARENESS,
Transparency and
EXPLICABILITY
v USERS’
Engagement,
Interaction,
EMPOWERMENT

DISCOURSE-BASED
EXPLAINABLE AI
VISUAL ANALYTICS SENSE MAKING
HUMAN-MACHINE
DIALOGUE
ONLINE DELIBERATION
COLLECTIVE INTELLIGENCE

Collective Intelligence For the Common Good
Community - ci4cg.org
Several international
workshops and 2 Special issues

Thank you for listening!
Please fell free to contact me at anna.deliddo@open.ac.uk
to know more about our work please visit the research group
website at:
idea.kmi.open.ac.uk

Lecture Polimi April2021

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Lecture Polimi April2021

Semelhante a Lecture Polimi April2021 (20)

Mais de Anna De Liddo

Mais de Anna De Liddo (16)

Último

Último (20)

Lecture Polimi April2021