https://dbsec2019.cse.sc.edu/Keynote.html
Abstract: As social media permeates our daily life, there has been a sharp rise in the misuse of social media affecting our society in large. Specifically, harassment and radicalization have become two major problems on social media platforms with significant implications on the well-being of individuals as well as communities. A 2017 Pew Research survey on online harassment found that 66% of adult Internet users have observed online harassment and 41% have personally experienced it. Nearly 18% of Americans have faced severe forms of harassment online such as physical threats, harassment over a sustained period, sexual harassment or stalking. Moreover, malicious organizations (e.g., terrorist groups, white nationalists not classified legally as terrorists but as a group with extreme ideology) have been using social media for sharing their propaganda and misinformation to persuade individuals and eventually recruit them to propagate their ideology. These communications related to harassment and radicalization are complex concerning their language and contextual characteristics, making recognition of such narratives challenging for researchers as well as social media companies. As most of the existing approaches fail to capture fundamental nuances in the language of these communications, two prominent challenges have emerged: ambiguity and sparsity. Sole data level bottom-up analysis has been unsuccessful in revealing the actual meaning of the content. Considering the significant sensitivity of these problems and its implications at individual and community levels, a potential solution requires reliable algorithms for modeling such communications.
Our approach to understanding communications between source and target requires deciphering the unique language, semantic and contextual characteristics, including sentiment, emotion, and intention. This context-aware and knowledge-enhanced computational approach to the analysis of these narratives breaks down this long-running and complex process into contextual building blocks that acknowledge inherent ambiguity and sparsity. Based on prior empirical and qualitative research in social sciences, particularly cognitive psychology, and political science, we model this process using a combination of contextual dimensions -- e.g., for Islamist radicalization: religion, ideology, and hate -- each elucidating a degree of radicalization and highlighting independent features to render them computationally accessible.
Understanding Online Socials Harm: Examples of Harassment and Radicalization
1. Understanding Online Socials Harm:
Examples of Harassment and Radicalization
Prof. Amit Sheth
Founding Director, AI Institute
University of South Carolina
AI @
UofSC
33rd Annual IFIP WG 11.3 Conference on Data and Applications Security
and Privacy (DBSec'19)
Charleston, SC, USA
July 15 -17, 2019
Icons by thenounproject
Slides by SlideModel
2. 2
The youngest adults stand out in their social media consumption
88% of 18- to 29-year-olds indicate that they use any form of social media.
By Pew Research Center “Social Media Use Report 2018”
3. 3
Social Good and Social Harm on Social Media
A spectrum to demonstrate the variety of social good to social harm
Adapted from : Purohit, Hemant & Pandey, Rahul. (2019). Intent Mining for the Good, Bad, and Ugly Use of Social Web: Concepts, Methods, and
Challenges. 10.1007/978-3-319-94105-9_1.
Zika Virus
Monitoring
Help
Fighting
depression
Disaster
Relief
Opioid Usage
Monitoring
Joking
Marketing
Sensationalizing
Harassment
Accusing
Rumouring
Deceiving
Fake News
Radicalization
Illicit Drugs
Social HarmSocial Good
Positive Effects Negative Effects
4. 4
Fake-porn videos are being weaponized to harass and
humiliate women: ‘Everybody is a potential target’
‘Deepfake’ disturbingly realistic, computer-generated videos with photos taken from
the Web, and ordinary women are suffering the damage
5. 5
Different meanings of
diagnostic terms
Ambiguity
Different perceptions
of same concepts.
Subjectivity
Low prevalence of
relevant content
Sparsity
Nature of content with
more than one
context.
Multi
Dimensionality
Significant implications
in a big scale
application.
False Alarms
Knowledge Graphs and Knowledge Networks: The Story in Brief - IEEE Internet
Computing Magazine 2019
Multimodal
Content
Different modalities of
data
Challenges --Complex Problems
7. 7
Severity of online harm can differ
based on several criteria
It can span for more than a decade in
one’s life
Or it can lead to teenage suicides
Police accuse two students, age
12, of cyberbullying in suicide
By Jamiel Lynch, CNN
Teenage cyber bullying victims are
Use-cases of Online Harassment
8. 9
Existing Approaches
Tweet Content
Binary
Classifier
Harassing Non-Harassing
People
Network
Our Approach (Incorporating context)
Tweet Content
Multiclass
Classifier
Sexual
harassment Appearance
related
Harassment
Traditional Approach vs Recent Advances
9. 10
Problem Definition & Sparsity
Dataset # of Tweets Classes (%)
Waseem et al.
(2016)
16093
Racism (12%),
Sexism (19.6%),
Neither (68.4%)
Davidson et al.
(2017)
24,802
Hate (5%), Non-
hate (95%)
Zhang et al.
(2018)
2,435
Hate (17%),
Non-hate (83%)
Mostly binary classifications
Datasets have small percentages of positive
(harassing) instances
A Quality Type-aware Annotated Corpus And
Lexicon For Harassment Research [Rezvan et
al.]
This paper provides both a quality annotated
corpus and an offensive words lexicon
capturing different types of harassment
content:
(i) sexual (7.4%)
(ii) racial (22.5%)
(iii) appearance-related (21.8%)
(iv) intellectual (26%)
(v) political (22.4%)
Harassing (12.9%)
Non-Harassing
(87.1%)
Challenges in Online Harassment
10. 12
According to Pew research center (2017)
Subjectivity
“pEoPlE dO iT tO tHeMsElVeS”
shut the f**k up
you know what im not gonna
argue anymore u guys are all so
f**king ignorant when it comes to
addiction so PLEASe stop f**king
speaking on it
all the girls who i have beef w are
little ass girls who think they can
say the n word but get scared
when black guys come around 🤔 🤔
on that note, gn twit
An interaction example from
Highschool students’ tweet corpus
Challenges in Online Harassment
11. “Language used to express hatred
towards a targeted individual or group, or
is intended to be derogatory, to
humiliate, or to insult the members of the
group, on the basis of attributes such as
race, religion, ethnic origin, sexual
orientation, disability, or gender is hate
speech” - Founta et al. 2018
13
Challenges in Online Harassment Detection
Ambiguity
Researchers have defined harassment using jargon that overlaps, causing ambiguity in
annotations
“Profanity, strongly impolite, rude or
vulgar language expressed with fighting
or hurtful words in order to insult a
targeted individual or group is
offensive language ” - Founta et al. 2018
Ex. 1: @user_name nah you just a dumb
hoe who doesn’t know her place 😂 😂
This tweet belongs to hate speech and
offensive language based on above
definitions
Examples from Highschool students’
tweet corpus
Challenges in Online Harassment
Ex. 2: IS THAT A MICROAGRESSION AGAINST
MEXICANS BY STEREOTYPING THEM AS
ILLEGALS?!? only if you were vegan you wouldn’t
be such a racist pig
This tweet falls into the category of
hate speech but not necessarily
offensive language
12. 14
What? How? Who?
What causes online
harassment? [REASON]
How can online
harassment happen?
[METHOD]
What are the effects of
online harassment?
[RESULT]
Victim
Appearance/
Religion/
Race etc.
Harasser
Xenophobia/
Homophobia/
Intolerance
etc.
Victim feels:
~ Offended
~ Discriminated
~ Afraid of losing life or
losing social capital
~ Depressed
~ Suicidal
Direct vs. Indirect
Where?
~ Flaming (the act of posting or
sending offensive messages over the
Internet)
~ Doxxing (broadcasting private
or identifying information of
individuals)
~ Dogpiling (several people in
twitter addressing someone, usually
negatively in a short period of time)
~ Impersonation
~ Public shaming
~ Threats
[ACTORS]
~ Harasser
~ Victim
~ Bystanders:
a. Aggravators
(People who try to
fuel a harassing
situation indirectly,
for example by
retweeting
harassing tweets)
a. Empathizers
(People who
empathize with the
victim, providing
support)
~ Social Media
(Twitter,
Facebook,
Instagram)
~ Discussion
Boards
(Reddit, 4chan)
~ Email
~ Private
Messaging
~ Online
Gaming
* Policies of
platforms
[PLATFORM]
Frequently vs. One-
time
Cyber Bullying vs. Cyber
Aggression
Online Harassment - Dimensions
13. 16
Resolving Data Scarcity Issue
Using Generative adversarial networks (GANs) to generate text,
increasing the positive(harassing) examples in a dataset
Changing the generator objective function in the GAN to incorporate
domain specific knowledge
Multiclass classification of harassing tweets
Harassment type prediction was done using a multiclass classifier
The process of tweet vectorization leveraged domain knowledge in the
form of an offensive words lexicon
Current Research Directions Pursued
14. 17
Key Takeaways
● In spite of recognition of importance, the problem is
still not well-understood, and not well-defined.
● Use cases and data show that the problem is far more
complex and nuanced. Understanding context is
critical.
● Current social media platforms appear to have little or
no automated processes for detection and prevention.
Oversimplification of problem definition largely
relying on machine learning without significant domain
specific knowledge has rendered solutions practically
useless.
16. 19
Challenges in Online Harassment Detection
Efforts by High-Tech Companies
Capabilities of social media
companies (Twitter, Facebook
and Google) are inadequate and
ineffective.
Governments insisted that the
industry had a ‘social
responsibility’ to do more to
remove harmful content.
If unsolved, social media
platforms will continue to
negatively impact the society.
Unsolved: Detection of Extremism on Social Media
17. 20
Challenges in Online Harassment Detection
● One thousand Americans
between 1980 and 2011.
300 Americans since 2011
attempted or traveled.
● > 5 thousand individuals from
Europe traveled to Join
Extremist Terrorist Groups
(ISIS, Al-Qaeda) abroad
through 2015,
● Most inspired and persuaded
online.
“The Travelers”
*George Washington University, Program on Extremism
18. ● 24 year old college student from Alabama became
radicalized on Twitter. After a year, moved to Syria
to join ISIS.
● Self-taught, she read verses from the Qur’an, but
interpreted them with others in the extremist
network.
● Persuaded that when the true Islamic State is
declared, it is obligatory to do hijrah, which they
see as the pilgrimage to ’the State’.
21
Illustrative Case
*New York Times: “Alabama Woman Who Joined ISIS Can’t Return Home, U.S. Says”
19. 22
Challenges in Online Harassment DetectionRadicalization Scale (Achilov et al.)
0
None
Mainstream
religious
views and
orientations
Indicator:
Islam; Allah;
jihad (self
struggle);
halal;
democracy,
islam, salah,
fatwa, hajj.
1
Low
Attitudinal
support for
politically
moderate
Islamism
Indicator:
Hadith;
Caliphate
(Khilafah)
justified;
Sharia better
(than secular
law);
Hypocrisy
west.
2
Elevated
Emergent
support for
exclusive rule
of the Shari’a
law
Indicator:
Shariah best;
revenge
(justified);
jihad (against
West); justify
Daesh (ISIS)
3
High
Support for
extremist
networks and
travel to
“Darul Islam”
Indicator:
Kafir; infidel;
hijrah to Darul-
Islam;
(supporting)
fatwa Al-
Awlaki;
mushrikeen.
4
Severe
Call for action
to join the
fight and the
use of
violence.
Indicator:
apostate;
sahwat;
taghut; kill;
kafir; kuffar;
murtadd;
tawaghit;
al_baghdadi;
martyrdom
khilafah
20. 23
Challenges in Online Harassment DetectionRadicalization Process over time
Ultimately, analysis of content in context will provide better
finer-granular understanding the underlying factors in the
radicalization process.
Non-extremist
ordinary
individual
Radicalized
extremist
individual
0 1 2 4
SevereHighLowNon
e
Elevated
3
21. Islamist Extremism on Social Media
(e.g., recruiter, follower) with
respect to different stages of
radicalization.
Modeling users
psychological process
over a time period.
Persuasive
relevant to Islamist
extremism.
Domain Knowledge
of the context (“jihad” has
different meaning in
different context)
Multidimensionality
Radicalization
22. Security Implications
Specifically, unfair
classification of non-
extremist individuals as
extremist.
False alarm might potentially
impact millions of innocent
people.
25
Local and Global security implications,
while predicting online terrorist activities
and involved individuals.
23. 26
Multidimensionality of Extremist Content
● Dimensions to define the context:
○ Based on literature and our empirical study of the
data, three contextual dimensions are identified:
Religion, Ideology, Hate
● The distribution of prevalent terms (i.e., words, phrases,
concepts) in each dimension is different.
● These terms should be represented in different
dimensions, to disambiguate especially diagnostic
terms (e.g., jihad): .
25. 28
“Reportedly, a number of
apostates were killed in
the process. Just
because they like it I
guess.. #SpringJihad
#CountrysideCleanup”
“Kindness is a language
which the blind can see
and the deaf can hear
#MyJihad be kind
always”
“By the Lord of Muhammad (blessings and peace be upon
him) The nation of Jihad and martyrdom can never be
defeated”
“Jihad” can appear in tweets with different meanings in different dimensions
of the context.
H
I
R
Example Tweets with “Jihad”
26. 29
Challenges in Online Harassment Detection
● Same term can have
different meanings for each
dimensions.
● Example:
“Meaning of Jihad”
is different for extremists
and non-extremists.
○ For extremists, meaning closer
to “awlaki”, “islamic state”,
“aqeedah”
For non-extremists, closer to
“muslims”, “quran”, “imams”
Ambiguity of Diagnostic terms/phrases
ExtremistsNon-Extremists
27. ● Different Contextual Dimensions
incorporating:
○ Knowledge Graphs
○ Dimension Corpora
● Utilization of Deep Learning models,
generate knowledge-enhanced
representations
● KG creation:
Religion: Qur’an, Hadith
Ideology: Books, lectures of
ideologues
[Not KG: Hate: Hate Speech Corpuss
(Davidson et al. 2017)]
● Can be applied over many social
problems.
30
Modeling
Modeling
Modeling
Dimension 1
Dimension 2
Dimension 3
DimensionDimensionDimension
Dimension Modeling
Process
Dimension based
Knowledge
enhanced
Representation
Contextual Dimension Modeling
28. (Hate)
Capturing similarity:
● Learning word similarities from a substantial knowledge
graph
● A solution via distance between concepts in the knowledge
graph.
Modeling
31
Using a Knowledge Graph
“You shall know a word by the company it keeps” (J. R. Firth 1957: 11)
29. Capturing similarity (and resolving ambiguity):
● Learning word similarities from a large corpora.
● A solution via distributional similarity-based
representations.
Modeling
32
(Hate)
Using a Corpus
“You shall know a word by the company it keeps” (J. R. Firth 1957: 11)
30. 33
● Found two distinct groups
employing different contexts with
different density.
● Religion and Hate are usually
mixed, suggesting that extremists
might employ different hate
tactics.
● A small group of users employ
ideological context far more often
than others, suggesting these
users might be disseminators of
ideologically intense content.
Density of Dimensions in Extremist Content
31. 34
● Tri-dimension model
performs best.
● Precision used as
metric, to emphasize
reduction on
misclassification of
non-extremist
content.
● Implications in a large
scale application.
Results
32. ● False alarms: significantly reduced via incorporation of three
specific dimensions of context.
● Extremist users employ religion along with hate, suggesting they
employ different hate tactics for their targets.
● Inclusion of all three contextual dimensions significantly reduces
the likelihood of an unfair mistreatment towards non-extremist
individuals, in a real world application.
● Each dimension plays different roles in different levels of
radicalization, capturing nuances as well as linguistic and
semantic cues better throughout the radicalization process.
35
Key Insights
33. 36
Public/
Society
Social
Interactions
Cognitive
Neuro
Cognitive
Process
● Human brain processes information
from extremist narratives on social
media, that includes different
contexts, emotions, sentiment, etc.
● Individuals change behavior, make
choices in consuming/sharing
content with an intent.
● Coordination, information flow and
diffusion on social networks.
● Outcomes/impact on society through
events and collective actions (eg,
civil war or result of an election).
Our Highly Multidisciplinary Approach
35. 38
Context-Aware Harassment Detection on Social Media(wiki link)
is an interdisciplinary project among the Ohio Center of
Excellence in Knowledge-enabled Computing (Kno.e.sis), the
Department of Psychology, and Center for Urban and Public
Affairs (CUPA) at Wright State University.
We are supported by the NSF Award#: CNS 1513721
Supporting Grants
37. 1. Hinduja, S. and Patchin, J.W., 2010. Bullying, cyberbullying, and suicide. Archives of suicide research,
14(3), pp.206-221
2. Rezvan, M., Shekarpour, S., Balasuriya, L., Thirunarayan, K., Shalin, V.L. and Sheth, A., 2018, May. A
Quality Type-aware Annotated Corpus and Lexicon for Harassment Research. In Proceedings of the
10th ACM Conference on Web Science (pp. 33-36). ACM.
3. Zeerak Waseem. Are you a racist or am i seeing things? annotator influence on hate speech detection
on twitter. In Proc. of the Workshop on NLP and Computational Social Science, pages 138–142.
Association for Computational Linguistics, 2016.
4. Thoams Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech
detection and the problem of offensive language. In Proceedings of the 11th Conference on Web and
Social Media. AAAI, 2017.
5. Zhang, Ziqi & Luo, Lei. (2018). Hate Speech Detection: A Solved Problem? The Challenging Case of
Long Tail on Twitter. Semantic Web. Accepted. 10.3233/SW-180338.
40
References
Notas do Editor
In all, Facebook has added 60,100 data center-specific jobs since 2010. This equates to roughly 8,600 jobs each year.
General info on social media, on how often they are used by people. Give stats on the use of social media. % in US, China etc.
Social media is enabler of instant communication, further enabling good or bad outcomes.
Put a picture.
Interacting technical (on the left) and usability (on the right) challenges to the exploitation of the new data environment.
Circle the four of these that we address.
(i) Appropriate incorporation of multimodal data in the views of person, content and network, (ii) Ambiguity in the meaning of significant concepts in the content, (iii) Sparsity of important lexical and semantic cues in the domain-specific corpus, (iv) Noisy nature of social media data, that threatens performance of learning process, (v) Imbalance in a training dataset
Online harassment example (prolonged): https://www.theguardian.com/society/2018/aug/03/harassed-online-for-13-years-the-victim-who-feels-free-at-last
Online harassment(teenagers death) - https://www.cnn.com/2018/01/23/us/florida-cyberstalking-charges-girl-suicide/index.html
-DeepFake example
This slide explains the following 2 things:
Online harassment can be severe due to its prolonged nature: the example is the guardian news article (being harassed for 13 years)
Online harassment can increase the risk of suicide attempts in cyberbullying victims by 2 times
The existing online harassment detection approaches only focus on the content of the tweet to classify the tweet into harassing or non-harassing categories (just binary classification)
Our approach suggests to incorporate context in the form of People (user characteristics) and Network (follower followee characteristics) in addition to the content of the tweet to do multiclass classification of harassment. So the harassing class can be subdivided into several subclasses of harassment. Ex: sexual, political, racial, intellectual, appearance-related
This slide depicts a challenge in online harassment detection; which is data sparsity. This sparsity of data can be present in the research landscape in two forms:
The positive cases (harassing examples) are generally low in datasets. Ex: Davidson et al. only 5% of the entire dataset is tagged as harassing
The number of subclasses inside harassment is low. Ex: Waseem et al. has only two subclasses inside harassments class.
Saeedeh’s work in the slide show that we have tried to increase the identified subclasses of harassment
References for this slide:
A Quality Type-aware Annotated Corpus and Lexicon for Harassment Research. Web Science, WebSci 2018, Amsterdam, The Netherlands, May 27-30, 2018
[1] Zeerak Waseem. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proc. of the Workshop on NLP and Computational Social Science, pages 138–142. Association for Computational Linguistics, 2016.
[2] Thoams Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th Conference on Web and Social Media. AAAI, 2017.
[3] Zhang, Ziqi & Luo, Lei. (2018). Hate Speech Detection: A Solved Problem? The Challenging Case of Long Tail on Twitter. Semantic Web. Accepted. 10.3233/SW-180338.
What is meant by subjectivity in this slide: It means that if annotators are given a set of tweet like above (right hand side of the slide) one annotator would find the the tweets harassing but another annotator would find it non harassing. Maybe the last tweet in that interaction would be tagged as toxic (harassment) by a female annotator whereas a male annotator would find it non-harassing.
This suggests that online harassment is a phenomenon that is subjective in nature. Different individuals perceive it differently
Online harassment can be ambiguous. The meaning of ambiguity in this slide is as follows:
Once a tweet is annotated as harassment, if we want to annotate further to reflect the subclass of harassment, we would give definitions of each subclass to annotators and ask them to annotate the tweets accordingly.
So if you look at the definitions for offensive language and hate speech on the left-hand side they overlap with each other and the same tweet can be annotated by two annotated differently; falling into different subclasses. Look at the example on the right-hand side of the slide.
This table depicts the following:
We can look at online harassment from four major dimensions.
The first dimension is the what dimension of online harassment. We can approach this dimension via 2 routes.
The first route is to explore the causes of online harassment. This can also be identified as reasons for online harassment. For an example a victim of online harassment could be harassed due to his/her ethnicity or religious beliefs (Saeedeh’s types of harassment is falling into this; victims are harassed because of their sexuality, political standpoint, appearance, level of intellect). Also a harasser could harass a victim because the harasser is xenophobic, homophobic, etc.
The second route is to explore the effects of online harassment felt by the victims. Victim would feel offended, discriminated, depressed or in extreme cases even suicidal.
The second dimension is the how of online harassment. This can also be identified as the method of harassment. These methods could be again divided to several aspects. One such aspect is direct online harassment vs. indirect online harassment. Another aspect is based on the frequency of harassment. For an example: prolonged online harassment can be identified as cyber bullying where as one-time online harassment can be identified as cyber aggression. There are several identified methods of online harassment. Harassers could use one or a mixture of few. Such methods are flaming, doxxing, dogpiling, impersonation, public shaming, threatening, spreading rumours…
The third dimension is related to the who of online harassment. This dimension explores the actors involved in online harassment. The main two actors are harassers and victims. Apart from these major players, we also can identify bystanders who either aggravate the harassment process or conciliate it. Those who aggravate the situation can be identified as aggravators and those who conciliate can be identified as empathizers.
The final dimension worth exploring is the where of online harassment. This basically explores the platforms where online harassment happens. These could be social networking sites, online discussion boards, private messaging applications and online gaming platforms. The policies in these platforms also could help increase or decrease the act of harassment.
This slide connects to the previous slides that we talk about the challenges in online harassment detection.
One challenge was the data sparsity. To overcome this challenge we, in our group is working on text generation of positive harassing examples using GANs. The novelty is that the generator objective function of the GAN is being modified to incorporate domain specific knowledge in the form of a tweet corpus used for online harassment detection
The other challenge was identifying subclasses of harassment. The solution is to use a tweet representation that leveraged the domain knowledge of harassment in the form of an offensive words lexicon. These tweet representations are then fed into multiclass classifiers for classifications
Religion may play more role first, hate later
Two types of resources: 1-Structured --KG 2-Unstructured --Corpora
The surrounding words will represent the words in bold and italic.
The surrounding words will represent the words in bold and italic.
Why use precision? How 1% misclassification would translate into a big number of people being affected.
Weaponization
Social media is weaponizing the players with malignant intentions, causing harm on individuals and society.
Examples of radicalization, online harassment leading to teenage suicides.
Ambiguity
Defining What harm constitutes is hard.
Technical solution: data-driven solutions won’t work well. KG and domain specificity needs to be incorporated.
Sparsity
Even though these issues are serious and result in grave consequences, data is not abundantly available. For an example, the positive examples for online harassment was quite low in datasets that are being used for research. Data for different types of harassment is yet another hurdle.
Complexity
There are multiple dimensions that are affecting the dynamics of online harassment and online radicalization. These are interwoven and sometimes can depend on each other. Also These dimensions can be volatile in nature.
Therefore carefully modeling the dimensions is a must.