Real-Time Processing of Social Media Content for Social Good
•
1 gostou•647 visualizações
This tutorial was given at the Data Science workshop organized by the Higher Education Commission (HEC) Pakistan in 2017.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Real-Time Processing of Social Media Content for Social Good
Semelhante a Real-Time Processing of Social Media Content for Social Good (20)
Mais de Muhammad Imran
Mais de Muhammad Imran (13)
Último
Último (20)
Real-Time Processing of Social Media Content for Social Good
- 2. Outline • P1: Background of Humanitarian CompuBng (10%) – Sudden-onset emergencies, Time-cri,cal situa,ons – Social Good factors – Aid and informa,on needs • P2: The Role of Social Media for Social Good (20%) – Par,cular focus on micro-blogging plaKorms – Availability of various types of informa,on and opportuni,es • P3: The Role of ArBficial Intelligence for Social Good (70%) – How AI is useful in crisis response – Various AI techniques, approaches, and tools – Work of crisis compu,ng group at QCRI – Ongoing research – Future direc,ons
- 3. Aid Needs, InformaBon Needs, and Gaps Info. Info. Info. Disaster event (earthquake, flood) Urgent needs of affected people InformaBon gathering Humanitarian organizaBons and local administraBon InformaBon gathering, especially in real-Bme, is the most challenging part Relief operaBons - Food, water - Shelter - Medical assistance - DonaBons - Service and uBliBes
- 4. Aid Needs, InformaBon Needs, and Gaps Info. Info. Info. Disaster event (earthquake, flood) Urgent needs of affected people InformaBon gathering Humanitarian organizaBons and local administraBon InformaBon gathering, especially in real-Bme, is the most challenging part Relief operaBons - Food, water - Shelter - Medical assistance - DonaBons - Service and uBliBes --Informa,on Bestows Power-- Will access to informaBon solve the problem?
- 5. Decision-Making and Response Department of Community Safety, Queensland Govt. & UNOCHA, 2011 - Delayed decision-making - Delayed crisis response - High community harm - Early decision-making - Rapid crisis response - Low community harm Target
- 6. Decision-Making and Response Department of Community Safety, Queensland Govt. & UNOCHA, 2011 - Delayed decision-making - Delayed crisis response - High community harm - Early decision-making - Rapid crisis response - Low community harm Target --Need Early Informa,on-- How early do we need it?
- 9. InformaBon Types and Needs • Reports of Injured or dead people • Infrastructure damage (e.g., buildings, bridges, Roads) • Urgent needs of affected people (e.g., food, water, shelter) • Dona,on offers and requests (e.g., money, volunteers) • Medical Emergencies • Disease symptoms reports • Disease treatment reports and ques,ons • …
- 16. MERS Outbreak: Twiaer Data Analysis Middle East Respiratory Syndrome (MERS) Twicer data analysis from: 2014-04-27 to 2014-07-14 QualitaBve analysis categories: Reports of symptoms Affected people reports Death reports Disease transmission reports Preven,on ques,ons Treatment ques,ons Reports of signs or symptoms such as fever, cough or ques,ons Reports of affected people due to the MERS disease Reports of deaths due to the MERS disease Ques,ons or sugges,ons related to the preven,on of disease Reports or ques,ons related to the transmission of the disease Ques,ons or sugges,ons regarding the treatment of the disease
- 21. Types of InformaBon on Twiaer - Twicer data from 13 recent crises - Over 100,000 tweets - InformaBon types - Types of sources Source: Qatar Compu,ng Research Ins,tute - Published in World Humanitarian Data and Trends 2014 (UN OCHA)
- 22. 2013 Pakistan Earthquake September 28 at 07:34 UTC 2010 HaiB Earthquake January 12 at 21:53 UTC Data and OpportuniBes Social Media Plaiorms Availability of Immense Data: Around 16 thousands tweets per minute were posted during the hurricane Sandy in the US. OpportuniBes: - Early warning and event detecBon - SituaBonal awareness - AcBonable informaBon extracBon - Rapid response - EffecBve communicaBons Disease outbreaks
- 24. Big Data Challenges – 4Vs (Under Time-criBcal SituaBons) • Volume Scale of data (e.g., millions of tweets aber an event) • Velocity High-velocity streams (e.g., thousands of tweets/min) • Variety Different forms/types of data (informa,on types) • Veracity Uncertainty of data
- 25. Data AcquisiBon
- 26. Twiaer Data CollecBon • REST APIs – Provides programma,c access to post a new tweet, read profile, and followers. • Streaming APIs – Receive live updates on the latest tweets matching a search query. • Ads API, MoPub, and Gnip – Twicer adver,sing management, MoPub is a mobile ad exchange and ad server. – Gnip provides commercial-grade access to real-,me and historical Twicer data.
- 28. ProperBes of Social Media Data • Mostly SM data is publicly available • Near Real-Bme access • 1% to 3% geo-tagged • Highly informal, oben brief, and non- structured • Wricen by different people in many languages • Contains rumors and misinforma,on
- 29. Slangs and Shortened forms • Single-word slangs: pls (please), srsly (seriously) • MulB-word slangs: imo (in my opinion) • Misspellings: missin (missing), ovrcme (overcome) • PhoneBc subsBtuBon: 2morrow (tomorrow) • Word without spaces: prayfornepal (pray for nepal) Can you guess? “r u ok m8” ?? >> “Are you OK, mate?”
- 30. Data Velocity and Volume High velocity • 2012 Hurricane Sandy: 18,000 tweets/min • 2013 Boston bombings: 54,000 tweets/min • 2011 Japan earthquake: 66,000 tweets/min High volume • 2012 Hurricane Sandy: 20 million tweets in 5 days Batch Periodic Near real-,me Real-,me Stream Increase in Data Velocity KB MB GB TB PB Increase in Data Volume File system -- MySQL -- Postgres – MongoDB -- Apache Cassandra -- Redis
- 31. Data Processing
- 34. Data Stream Processing 1. Data items arrive online 2. Streams have infinite length and unbounded in size 3. No control over the order in which data items arrive 4. Processed items are either discarded or archived 5. No retrieval unless stored in memory (oben small size) Credit Card fraud detecBon Sensor data classificaBon Social media streams mining Data stream
- 35. TradiBonal vs. Stream Processing Property TradiBonal System Stream Processing System Number of passes Mul,ple Single Memory availability Unlimited Restricted Processing ,me Unlimited Restricted Results availability Delayed Real-,me Results reliability Accurate Improvable
- 36. Pure Stream Processing and Issues • Rely en,rely on automated algorithms • SM data streams can be imprecise, highly variable, and oben unseen – Concept-dri_: happens due to slow changes in the concepts – Concept-evoluBon: happens due to the presence of unknown classes Aurora Stream Processing (Brown University) Flu pandemic 2009
- 37. Crowdsourced Stream Processing (CSP) In cases where cri1cal—in terms of cost, 2me or reliability—decision-making needs to take place in real-1me, based on data streams that are poten2ally noisy and unseen, fully automated stream processing systems do not meet the needs. Stream processing systems (SPs) Crowdsourcing systems (Cs) Crowdsourced stream processing systems (CSPs) Human processing role Automatic processing role Compostion Binary classification N-ary classification Open-ended Computation Filtering Task-generation Task-assignment Task-aggregation Serial Parallel Complex Hierarchal taxonomy Faceted taxonomy System Ref. Imran, Muhammad, Ioanna Lykourentzou, Yannick Naudet, and Carlos Cas2llo. "Engineering crowdsourced stream processing systems." arXiv preprint arXiv:1310.5463.
- 38. hcp://aidr.qcri.org/ AIDR —Ar,ficial Intelligence for Disaster Response— is a free, open, and easy-to-use plaKorm to automa,cally filter and classify relevant tweets posted during humanitarian crises. 1 2 3 Collect Curate Classify Grand Prize Winner from the Open Source So_ware World Challenge 2015
- 39. Data collec,on 1 2 Human annota,ons Machine training 3 Classifica,on 4 ONLINE APPROACH DATA COLLECTION H A Learning-1 CLASSIFICATION OF DATA & DECISION MAKING PROCESS Learning-2 Learning-3 … Learning-n Human annota,on - 1 Human annota,on - 2 Human annota,on - 3 … Human annota,on - n First few hours Near Real-Bme Processing
- 42. AIDR Architecture Tweets collector Twitter streaming API Features extractor ClassifierP/S Task generator Q P/S Annotator model parameters Learner Output adapters Q Q load shedding load shedding query tweets 〈tweet〉 〈tweet, features〉 〈task〉 〈task, label〉 〈tweet, label, confidence〉 Redis channel Redis queue Human-in-the-loop (crowdsourcing) - Uni-grams - Bi-grams - InformaBon gain Random Forest (decision trees) - Task selecBon - Task prioriBzaBon Database: Postgres ApplicaBon layer: Java EE, RESTFul services, Weka machine learning library Data flow and control flow: Redis Front-end: ExtJS (JavaScript library)
- 53. Quality vs. Cost in AIDR hcp://aidr.qcri.org/ Goals: Maximize quality – Minimize cost • Quality • Classifica,on accuracy • Precision/AUC • Cost to obtain labeled data • Monetary in case of paid-workers • Time in case of volunteers
- 57. Data Stream Processing 1. Data items in the stream arrive online 2. Streams have infinite length and unbounded in size 3. No control over the order in which data items arrive 4. Processed items are either discarded or archived 5. No retrieval unless stored in memory (oben small size) Credit Card fraud detecBon Sensor data classificaBon Social media streams mining Data stream
- 65. Social Media Data Streams ClassificaBon Two major issues in the supervised classifica,on of social media streams: 1. How to keep the categories used for classificaBon up-to-date? 2. While adding new categories, how to maintain high classificaBon accuracy? by crowd Automatic processing Automatic processing output output Performing verification Providing training data a: Split automatic/manual processing b: Detect-verify paradigm Automatic processing Automatic processing output c: Improving quality through active learning input input input
- 66. IdenBficaBon of Novel Categories Classes. - Injured people - Infrastructure damage - Shelter needs - Dona,on requests - Missing or stranded people - Different health issues - Novel urgent needs like - Blankets - Medicine - Schools shut - Airport closed/open - … Pre-defined classes Unseen classes (Miscellaneous) Keep in mind we have a new class “Miscellaneous”
- 67. Expert-Machine-Crowd Sesng Constraints Outlier DetecBon (COD-Means): 1. Constraints forma,on using classified items 2. Clustering using COD-Means 3. Labeling errors iden,fica,on (using outlier detec,on) List of categories documents stream Supervised Learning System Novel Categories Detector Using COD-Means Crowdsourcing task generator Emerging novel categories Crowdsourcing tasks to be labeled by crowd An expert Crowd workers Crowd/machine classified items. (Machine classified items with confidence score >= 0.90) Incoming uncategorized documents stream Machine categorized items (item, category and machine confidence score) triplet Refined training set Human labels Labels 1 2 3 4
- 68. Input and Output Category A Category B Category C Miscellaneous Z Category A’ Category B’ Category C’ Z1 Z2 Z’ INPUT OUTPUT
- 69. Constraints FormaBon 1. Items in same category have Must-link constraints 2. Items belonging to different categories have Cannot-link constraints Category A Category B Category C Category Z Must-link Cannot-link Note: Items in Z do not have any constraints
- 73. Dataset and Experiments 1. Are the new clusters iden,fied by the COD-Means algorithm genuinely different and novel? 2. What is the nature of outliers (labeling errors) discovered by the COD-Means algorithm? Are they genuine outliers? 3. What is the impact of outlier on the quality of clusters generated by COD-Means? 4. Once refined clusters (without labeling errors) used in the training process, does the overall accuracy improves? 8 disaster-related datasets were used from Twiaer
- 75. Data Improvements EvaluaBon Affected individuals Caution and advice Donations and volunteering Infrastructure and utilities Sympathy and support Misc. to other categories Precision 0 0.25 0.5 0.75 1 Precision 0 0.25 0.5 0.75 1 2012 Colorado Wildfires 2013 Alberta Floods 2013 Boston Bombings 2013 Colorado Floods 2013 Train Crash 2013 Australia Bushfire 2013 Queensland Floods 2013 West Texas Explosion Precision 0 0.25 0.5 0.75 1 Precision 0 0.25 0.5 0.75 1 Affected individuals Caution and advice Donations and volunteering Infrastructure and utilities Sympathy and support Misc. to other categories Precision 0 0.25 0.5 0.75 1 Precision 0 0.25 0.5 0.75 1 Precision 0 0.25 0.5 0.75 1 Precision 0 0.25 0.5 0.75 1 1. Labeling errors in non-miscellaneous categories 2. Items incorrectly labeled as miscellaneous
- 86. Infrastructure Damage Assessment • Three-class classifica,on – Categories: severe, mild & licle-to-none • Dis,nc,on between categories is ambiguous. • Agreement among human annotators is low. – in par,cular for mild category • Fine-tuning a pre-trained CNN (e.g., VGG16)
- 91. Media Coverage
- 94. Same Domain Learning Training data Machine learning model Tes,ng data infer predict Apples Apples Apples Oranges Different shapes, colors, skins, tastes, etc. Source domain Target domain Oranges Oranges BUT
- 96. Domain AdaptaBon
- 97. Model AdaptaBon EvaluaBon • Model adapta,on using single source – Using both: in-domain and cross-domain • Model adapta,on using mulBple sources – In-domain – Mul,ple source events without the target – Mul,ple source events with the target • Model adapta,on in special cases – Same languages – Similar languages
- 98. Transfer Learning Differences in classificaBon tasks: • Different classifica,on tasks • Different types of disasters, stakeholders, informa,on needs Task: • Learn from source to classify target • Seman,c similarity between tasks • Zero-shot learning (no training examples) • One-shot learning (few training examples)
- 99. SummarizaBon and PrioriBzaBon of AcBonable InformaBon InformaBon needs & problem: • Different stakeholders • Different goals, requirements, and info. needs General situaBonal awareness vs. Target situaBonal awareness • High-level general updates from an event • Specific updates (infrastructure damages)
- 100. InformaBon SummarizaBon In Real-Time Class A Class B Class C Class D Summary Summary Summary Summary Classified documents stream
- 102. Towards Standard Baselines and Datasets CrisisNLP.qcri.org - Access to 52 million tweets - Around 50k labeled tweets into humanitarian categories - Largest word2vec embeddings trained on 52m crisis-related tweets - Out-of-vocabulary dic,onaries - Tweets downloader
- 103. ACM CompuBng Survey Processing Social Media Messages in Mass Emergency: A Survey [Imran et al. 2015]
- 104. 27 Free Data Mining Books hap://www.datasciencecentral.com/profiles/blogs/27-free-data-mining-books
- 106. Conclusions • InformaBon bestows power for disaster response – People need informa,on as much as water, shelter, and food – Disasters are unavoidable, but planning can lessen their effects • Social media as Bme-criBcal informaBon source – Early warnings, event detec,on, event monitoring – Availability of informa,on opens new opportuni,es • ArBficial Intelligence for Social Good – Applied research at its best – AI + humans-in-the-loop can enable rapid crisis response – AI techniques useful for: • Situa,onal awareness • Ac,onable informa,on extrac,on • Summariza,on