1. Persuasiveness and Audience
Reactions in Political Speeches
Marco Guerini

2. Contributors
Marco Guerini
Carlo Strapparava
Oliviero Stock
Danilo Giampiccolo
Rachele Sprugnoli
Giovanni Moretti

3. INTRODUCTION

4. Persuasive NLP
• Persuasion is becoming a hot topic in
Natural Language Processing.
• Automatic analysis and recognition of the
persuasive impact of communication.
• Address the various effects which
persuasive communication can have in
different contexts on different audiences.

5. Approaches
• Knowledge-based: Starting from theory.
• Data-driven: Starting from linguistic data.
Linguistic data should be possibly augmented
with annotation of various audience reactions.

6. Resources Examples
• Long texts: Political Speeches
• Short texts: Posts on Social networks
• Short sentences: Advertising Slogans
• Single words: Evocative Brand Names

7. CORPUS
DESCRIPTION

8. Characteristics & Collection
• CORPS: CORpus of tagged Political Speeches
• Hypothesis: tags about audience reaction,
such as APPLAUSE, are indicators of hot-
spots, where persuasion attempts succeeded
• Collection: Annotated speeches from various
web sources
• Normalization: Metadata insertion (speaker,
date, title, etc.) and Semi-automatic conversion
of tags names to make them homogeneous

9. Characteristics & Collection
CORPS - Main Statistics
Total number of speeches: ~ 3,600
Total number of speakers: ~ 200
Total number of words: ~8M
Total number of tags: ~ 66,000
Temporal range: from 1917 to 2010

10. CorpsFormatConverter
• Four annotators
have been trained.
• Annotation
supported by an
ad-hoc standalone
application.
• The tool facilitates
the extraction of
the speech text
and metadata from
the Web sources.
• The tool
automatically
converts the most
frequent tags.

11. Example of Tags and Conversion

12. Document Structure ex. - JFK
{title} Ich bin ein Berliner {/title}
{event} ----- {/event}
{speaker} John F. Kennedy {/speaker}
{date} 26 June 1963 {/date}
{source} americanspeech.com {/source}
{description} ----- {/description}
{speech} … {/speech}

13. Speech Fragment ex. - JFK
Freedom has many difficulties and democracy is
not perfect. But we have never had to put a wall up
to keep our people in, to prevent them from leaving
us. {APPLAUSE ; CHEERS}
I want to say on behalf of my countrymen who live
many miles away on the other side of the Atlantic,
who are far distant from you, that they take the
greatest pride, that they have been able to share
with you, even from a distance, the story of the last
18 years. I know of no town, no city, that has been
besieged for 18 years that still lives with the vitality
and the force, and the hope, and the determination
of the city of West Berlin. {APPLAUSE ; CHEERS}

14. Speeches Distribution
Number of speeches per Speaker.

15. Speeches Distribution
Temporal distribution of the
Speeches

16. Tags Count
SINGLE TAGS
{APPLAUSE} 46310
{LAUGHTER} 14055
{AUDIENCE} 1803
{BOOING} 756
{SPONTANEOUS-DEMONSTRATION} 313
{CHEERS} 234
{SUSTAINED APPLAUSE} 97
{STANDING-OVATION} 51
MULTIPLE TAGS
{LAUGHTER ; APPLAUSE} 1579
{CHEERS ; APPLAUSE} 837
OTHERS 47
SPECIAL TAGS
{AUDIENCE-MEMBER} 999
{COMMENT} 787
{OTHER-SPEAK} 404

17. Audience Tags - Count
Tag Count
{AUDIENCE} Yes! {/AUDIENCE} 482
{AUDIENCE} No! {/AUDIENCE} 390
{AUDIENCE} Four more years! Four more years! {/AUDIENCE} 346
{AUDIENCE} Yes, sir {/AUDIENCE} 87
{AUDIENCE} U.S.A.! U.S.A.! U.S.A.! {/AUDIENCE} 41
{AUDIENCE} All right {/AUDIENCE} 39
{AUDIENCE} Flip-flop! Flip-flop! Flip-flop! {/AUDIENCE} 39
{AUDIENCE} Hooah. {/AUDIENCE} 38
{AUDIENCE} Reagan! Reagan! Reagan! {/AUDIENCE} 37
… …
{AUDIENCE} Hooah! {/AUDIENCE} 24
{AUDIENCE} Tell it {/AUDIENCE} 23
… …

18. Comment Tags - Count
Tag Frequencies
{COMMENT="Inaudible"} 257
{COMMENT="A toast is offered"} 30
{COMMENT="The bill is signed"} 30
{COMMENT="The medal was presented"} 26
{COMMENT="The medal was awarded"} 24
{COMMENT="Recording interrupted"} 18
{COMMENT="The citation is read"} 18
{COMMENT="The citation was read"} 16
{COMMENT="Interruption"} 9
{COMMENT="A moment of silence was observed"} 8
… …

19. Audience Reactions Typologies
• Positive-Focus: a persuasive attempt that sets a positive
focus in the audience. Tags considered:
{APPLAUSE} , {STANDING-OVATION} ,
{SUSTAINED-APPLAUSE} , {CHEERING} , etc.
• Negative-Focus: a persuasive attempt that sets a negative
focus in the audience. Negative focus set towards the
object of the speech not on the speaker.
{BOOING} , {AUDIENCE} No! {/AUDIENCE}
• Ironical: Indicate the use of ironical devices in persuasion.
Tags considered:
{LAUGHTER} and multiple tags containing laughter.

20. Audience Reactions Typologies
• These 3 groups represents different effects
which political communication can have in
different contexts on different audiences.
Reaction Typology Count Percentage
POSITIVE-FOCUS TAGS 49275 0.74
IRONICAL TAGS 15660 0.24
NEGATIVE-FOCUS TAGS 1147 0.02

21. MACRO ANALYSIS

22. Tag Density
• How much “persuasive” is, on average,
a speech or group of speeches?
• Compute how many audience reaction
tags are present in a speech (normalize
according to speech length).

23. Tag Density
• Given a set of speeches - e.g. Democrats’ speeches -, tag
density can be computed in two different ways:
– Micro-averaged tag density (µ) - counting all tag occurrences
in the set and dividing the result for the total number of words.
– Macro-averaged tag density (M) - computing the tag density
for each category (e.g. each Democrat speaker) and then
averaging over the results of each speaker.
• µ gives the “real” tag density of the dataset, while M
avoids over-representation of unbalanced classes (e.g. a
vast majority of Bill Clinton’s speeches).

24. set of n speeches S, where aasingle speech is is repre-
set of n speeches S, where single speech repre-
|tii|| represents the number of tags inin a given speech
|t represents the number of tags a given speech
Tag Densitywe can define µ
number of words in the same speech; we can define µ
umber words in the same speech;
A set of n speeches,
n
n
i=1 |tii |
i=1 |t
| |ti| represents the number of
µ = n
n tags in a given speech/ (1) (1)
i=1 |wii |
i=1 |w
|
category
be defined as:
be defined as:
|wi| represents the number of
|C| |ti |
|C| |ti | words in the speech/category
i=1 |wi |
M= i=1 |wi | |C| represent the number (2)
of
M= |C| categories (speakers) in the(2)
|C|
number of categories (speakers) speeches.of speeches,
set of in the set
number of categories (speakers) in the set of speeches,
the total number of tags and words for the category.
he total number of tags and words for the category.

25. Tags Density - Corpus
Overall Tag density (μ): 0.0084
PF-density (μ): 0.0062
I-density (μ): 0.0020
NF-density (μ): 0.0002

26. Tags Density – Main Speakers
Speaker Speeches Tag-Density PF-density I-density NF-density
Bill Clinton 889 0.007 0.005 0.002 0.00001
George W. Bush 427 0.015 0.012 0.002 0.00005
Ronald Reagan 388 0.004 0.001 0.003 0.00044
Dick Cheney 356 0.011 0.008 0.002 0.00061
Barack Obama 347 0.01 0.008 0.003 0.00007
John F. Kennedy 316 0.009 0.008 0.001 0
Michelle Obama 107 0.009 0.005 0.003 0.00001
Margaret Thatcher 102 0.005 0.004 0.001 0.00001
Laura Bush 93 0.015 0.014 0.001 0
Richard M. Nixon 61 0.006 0.005 0 0.00008
Al Gore 53 0.007 0.005 0.002 0.00004
Alan Keyes 51 0.004 0.003 0.001 0.00007

27. Tags Density – Party and Gender
Party Corpus-Cover. Tag-Density PF-density I-density NF-density
Democrats 0.45 0.0075 0,0055 0,0019 0,000027
Conservatives 0.55 0.0097 0,0072 0,0022 0,000309
Gender Corpus-Cover. Tag-Density PF-density I-density NF-density
Females 0.11 0.0085 0.0067 0.0018 0.000007
Males 0.89 0.0083 0.0062 0.0020 0.000158
Micro-averaged
densities (μ)

28. Tags Density – Party and Gender
Party Corpus-Cover. Tag-Density PF-density I-density NF-density
Democrats 0.45 0.0076 0.0056 0.0019 0.000036
Conservatives 0.55 0.0094 0.0076 0.0017 0.000199
Gender Corpus-Cover. Tag-Density PF-density I-density NF-density
Females 0.11 0.0068 0.0055 0.0013 0.0000007
Males 0.89 0.0070 0.0052 0.0017 0.0000444
Macro-averaged
densities (M)

29. Tags Density – Party and Gender
• While the Democrats/Conservatives partition is well
balanced (0.45 vs. 0.55), the Males/Females partition is
unbalanced (0.89 vs. 0.11).
• Tag density is slightly higher for Conservative speakers
(the same holds for positive-focus tags), while the ironical-
focus tags have almost the same density in both groups.
• Analysis ex. Negative-focus tags (representing a more
“aggressive” kind of rhetoric): density in the Conservative
group is 11 times higher than the in Democrats. A similar
consideration for the male/female distinction: while other
tag densities are almost the same, for the negative-focus
tags we have a density 60 times higher for male speakers.

30. Tag Density - Temporal Distribution

31. Language and
Micro Analysis

32. Language Persuasiveness
• Are there words, linguistic expressions
that are more “persuasive” than others?
• In a speech not all text fragments have
the same importance. Consider
audience reaction tags.

33. Possible Uses
• Persuasive expression mining. recognition and classification of
phenomena such as audience reactions, speaker vocal effort can
improve information retrieval (Bertoldi et al. 2002; Hu et al., 2008).
New approaches for extracting relevant linguistic material, e.g.
words persuasive impact (pi), see (Guerini et al., 2008).
• Automatic analysis of political communication. Computational
linguistics to automatize analysis on politicians’ rhetoric.
Considering audience’s reactions new rhetorical phenomena
discovered (vs. traditional approaches based on words counting).
• Prediction of text impact. Machine learning for predicting the
persuasive impact of novel speeches (Strapparava et al., 2008).
• Persuasive natural language generation. Eg. lexical choice: on
the basis of lemma impact rather than lemma use.

34. Approach
• In analyzing CORPS, we focused on the
lexical level.
• We considered:
– Windows of different width wn of terms
preceding audience reactions tags.
– The typology of audience reaction.

35. Approach ex. Fragment from JFK
Freedom has many difficulties and democracy is not
perfect. But we have never had to put a wall up to
keep our people in, to prevent them from leaving us.
{APPLAUSE ; CHEERS}
I want to say on behalf of my countrymen who live
positive-focus
many miles away on the other side of the Atlantic,
who are far distant from you, that they take the
wn = 15
greatest pride, that they have been able to share
with you, even from a distance, the story of the last
18 years. I know of no town, no city, that has been
besieged for 18 years that still lives with the vitality
and the force, and the hope, and the determination
of the city of West Berlin. {APPLAUSE ; CHEERS}

36. Valence and Persuasion
The phase that leads -
to audience reaction,
if it presents valence
dynamics, is
characterized by a
valence crescendo

37. Words persuasive impact
• Basic idea: a word is more persuasive if at
the same time its occurrences appear
close to audience reactions tags and they
do not appear far from them.
• We extracted “persuasive words” by using
a coefficient of persuasive impact (pi)
based on a weighted tf-idf (pi = tf × idf).

38. Words persuasive impact (cont’d)
• We created a “virtual document” by collecting terms inside
windows preceding audience reactions tags (wn = 15).
• |D| = number of speeches in the corpus (included the
virtual document)
• n = number of times the term (word) ti appears in the
i
virtual document
• Σn s = sum of word scores (closer to the tag, higher score)
i i
• Σ n = number of occurrences of all words in the virtual
k k
document = wn × |tags number|
• |{d : d ∋ t }| = number of documents where the term t
i i
appears (we made a hypothesis of equidistribution)

39. Corpus Pre-processing
• POS-tagged all the speeches to reduce
data sparseness, e.g.
– win, won, wins  win#v
– war, wars  war#n

40. Topmost Persuasive Words

41. Advantages
For persuasive political communication
the approach using the persuasive impact
(pi) of words is much more effective than
simple word count.

42. Examples of Use - Reagan
Many qualitative researches on Reagan’s (aka “the great
communicator”) rhetorics: conversational style, irony, etc.
• Great Communicator? 32 Reagan’s speeches, mean tag
density 1/2 of the whole corpus (t-test; α 0.001). Being a “great
communicator” not bound to “firing up” rate.
• Reagan’s style: “simple and conversational”. Hp: conversational
style more polysemic than a “cultured” style (richer in technical,
less polysemic, terms). No statistical diff. between mean
polysemy of Reagan’s words and whole corpus. But mean
polysemy of Reagan persuasive words is double of the whole
corpus (t-test; α 0.001).
• Use of irony: Density of ironical tags in Reagan’s speeches
almost double as compared to the whole corpus (t-test; α
0.001). In Reagan’s speeches the mean ironical-tags ratio (mtri)
is about 7.5 times greater than the mtri of the whole corpus (t-
test; α 0.001).

43. Examples of Use – Bush and 9/11
• How do political speeches change after
key historical events? Bush’s speeches
before and after 9/11 (70 + 70 speeches)
– While words positive valence remains unvaried, the
negative increases by 15% (t-test; α 0.001).
– Words counts only partially reflects word impact…

44. Lemma pi before pi after Count before Count after
win#v 112 7 27 52
justice#n x 9 15 111
military#n 197 36 23 29
defeat#v x 16 1 44
right#r x 25 94 55
victory#n 826 65 9 26
evil#a - 129 0 44
death#n 4 450 65 32
war#n 36 x 80 258
soldier#n 70 296 20 47
tax#n x 93 702 81
drug-free#a 87 x 9 3
leadership#n 81 261 40 75
future#n 83 394 54 51
dream#n 99 321 77 30
Notes. In the second and third column, the number represents the rank in the list of persuasive words; an “x” indicates a pi = 0; an
“–” indicates the word is not present in the corpus at all. In the fourth and fifth columns the total number of occurrences.

45. Bush and 9/11- Analysis Example
• For every word, we can record an increase or decrease of use
(word count) compared with an increase or decrease of
persuasiveness (pi).
• Let us consider the words military#n or treat#v. Both words are
used almost the same number of times before and after 9/11.
So their informativeness, based on number of occurrences, is
null. But considering the persuasiveness score, we see that their
impact varies (respectively from 197 to 36 and from 54 to 473).
• Let us also consider the word war#n; if we consider only the
number of occurrences, we could infer that after 9/11 this topic
was much more “felt” (mentioned three times more after 9/11),
but if we look at persuasiveness we see that before 9/11 the
word war#n was very “popular” (position 36) while after it never
got audiences’ reactions.

46. PREDICTION OF PERSUASIVE
EFFECTS

47. Experiments
• Using machine learning for predicting the
persuasive impact of novel discourses.
– Distinguishing Democrats from Republicans
– Predicting the passages that trigger a positive
audience reaction
– Cross classification (training made on adverse
party speeches, and test on the others)
– Experimenting the classifiers on plain and
typical non-persuasive texts taken from British
National Corpus and on speeches from the
Obama-McCain political campaign.

48. Framework and Dataset
• We used the Support Vector Machines (SVM) framework.
• Dataset preprocessing: to reduce sparseness, used lemma#pos
instead of tokens.
• We did not make any frequency cutoff or feature selection.
• All the speeches divided into fragments of about four sentences
(if a tag is present in the fragment the chunk ends at that point).
• Obtained chunks are then labeled as Neutral (i.e., no tag), and
Positive-ironical (i.e., all positive-focus and ironical tags). We did
not consider the negative-focus tags, since they are only a few.
• A total of ~38000 four-sentence chunks, roughly equally
partitioned into the two considered labels.
• This accounts for a baseline of 0.5 in distinguishing between
Neutral and Positive-ironical chunks. In all the experiments we
randomly split the corpus in 80% training and 20% test.

49. Democrats vs. Republican
Precision Recall F1
Democrats 0.842 0.756 0.797
Republicans 0.773 0.854 0.811
Average (μ) 0.804 0.804 0.804

50. Positive vs. Neutral
• Whole Corpus
Precision Recall F1
Positive-Ironical 0.646 0.683 0.664
Neutral 0.676 0.641 0.658
Average (μ) 0.660 0.660 0.660

51. Positive vs. Neutral
• Republican only
Precision Recall F1
Positive-Ironical 0.660 0.766 0.709
Neutral 0.663 0.549 0.601
Average (μ) 0.661 0.661 0.661
• Democrat Only
Precision Recall F1
Positive-Ironical 0.666 0.674 0.670
Neutral 0.686 0.680 0.683
Average (μ) 0.676 0.676 0.676

52. Cross Classification
• Training on Democrats, Test on Republicans
Precision Recall F1
Positive-Ironical 0.642 0.632 0.637
Neutral 0.579 0.599 0.589
Average (μ) 0.612 0.612 0.612
• Training on Republicans, Test on Democrats
Precision Recall F1
Positive-Ironical 0.625 0.660 0.642
Neutral 0.658 0.626 0.641
Average (μ) 0.641 0.641 0.641

53. Untagged texts Classification
• Typical non- Total chunks 7243
persuasive texts from Positive-Ironical 784
BNC (A00 to A0H)  Neutral 6459
Supposing all chunks Prec/Rec/F1 0.892
are neutral
• Typical persuasive Obama McCain
texts from the last Positive-Ironical 2372 2360
Obama-McCain Neutral 68 80
presidential campaign Total chunks 2440 2440

54. Conclusions
• We have presented a resource and some
approaches for persuasive NLP:
– a Corpus of tagged Political Speeches (CORPS)
and a method for extracting persuasive words.
– a measure of persuasive impacts of words

55. Future Work
• Consider also persuasive rhetorical pattern
extraction from CORPS.
• Consider windows width (wn) based on
sentences rather than tokens.
• …

56. Some References
• Marco Guerini, Danilo Giampiccolo, Rachele Sprugnoli,
Giovanni Moretti and Carlo Strapparava. The New Release of
CORPS: Tagged Political Speeches for Persuasive
Communication Processing, to appear.
• Marco Guerini, Carlo Strapparava and Oliviero Stock. CORPS:
A corpus of tagged political speeches for persuasive
communication processing. Journal of Information
Technology Politics 5 (1), 19-32, 2008.
• Marco Guerini, Carlo Strapparava and Oliviero Stock. Audience
Reactions for information extraction about persuasive
language in political communication. In M. Maybury (ed.)
Multimodal Information Extraction, to appear.
• Carlo Strapparava, Marco Guerini and Oliviero Stock.
Predicting Persuasiveness in Political Discourses. In
Proceedings of LREC2010.