1. Gunther Eysenbach MD MPH Gunther Eysenbach MD MPH Pandemics in the Age of Twitter: A Case Study of Infodemiology and Infoveillanceof Social Media as New Methods for Knowledge Translation Research and Syndromic Surveillance Director, Consumer Health & Public Health Informatics Lab Professor Department ofHealthPolicy, Management and Evaluation, University of Toronto; Senior Scientist, Centrefor Global eHealth Innovation,Division of Medical Decision Making andHealth Care Research; Toronto General Research Institute ofthe UHN, Toronto General Hospital, Canada geysenba@gmail.com

2. Economists have something public health practitioners don’t have: Real-time indices to track behavior and emotions S&P 500 Stock Index Fear Index

3. The premise “The Internet has made measurable what was previously immeasurable: The distribution of health information in a population, tracking (in real time) health information trends over time, and identifying gaps between information supply and demand. “ Eysenbach G. Infodemiology: Tracking Flu-Related Searches on the Web for Syndromic Surveillance. Proc AMIA Fall Symp2006: 244-249

4. Eysenbach G. Infodemiology: Tracking Flu-Related Searches on the Web for Syndromic Surveillance. Proc AMIA Fall Symp2006: 244-249

5. Infodemiology: Tracking Flu-Related Searches on the Web for Syndromic Surveillance(Eysenbach G, 2006) Google searches vsFlu Cases 2004/05 Sentinel Physician ILI reports vs Flu Cases 2004/05 Eysenbach G. Infodemiology: Tracking Flu-Related Searches on the Web for Syndromic Surveillance. Proc AMIA Fall Symp2006: 244-249

6. Idea was 3 yrs later confirmed (Ginsberg, 2009) and commercialized by Google

8. “epidemic intelligence” (Brownstein)

9. Infometrics, webometricshttp://www.youtube.com/watch?v=Xe27-mrQibQ

10. Infodemiology * / InfoveillanceApplication Areas Syndromic surveillance detecting and quantifying disparities in health information availability identifying and monitoring of public health relevant publications on the Internet (e.g. anti-vaccination sites) measure information diffusion and knowledge translation, and tracking the effectiveness of health marketing campaigns. Monitoring health-related behavior * science of distribution and determinants of information in an electronic medium, specifically the Internet, or in a population, with the ultimate aim to inform public health and public policy.

11. Infodemiology Research Program @ Centre for Global eHealth Innovation, Eysenbach Lab Developing innovative tools & methods to measure/track health-related attitudes, knowledge, emotions, public attention, behavior in real time for public healthusing textual data from the Internet & Social Media Investigate how the public is using social media during a pandemic, and how social media can be used to engage the public

12. Studying information patterns in the era of user-generated information (Web 2.0) enables us to measure user attitudes, behavior, awareness, knowledge, attention, information needs etc. Image Source: http://web2.wsj2.com/ Gunther Eysenbach MD, MPH, www.medicine20congress.com

13. Infovigilan open source infoveillance prototypeCentre for Global eHealth Innovation, Toronto Filter Keywords / Concepts of Interest User-submitted information Infovigil Aggregator/ Datamining/ Vizualisation Websites Public, Clinicians, Epidemiologists

15. “H1N1 marks the first instance in which a global pandemic has occurred in the age of Web 2.0 and presents a unique opportunity to investigate the potential role of these technologies in public health emergencies.” Chew C, Eysenbach G (2010) Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE 5(11): e14118

17. Infoveillance Predicting/tracking public-health relevant events (syndromic surveillance for outbreaks, “all-hazards” surveillance), Tracking changes in behavior, attitudes, knowledge (e.g. as a result of public health messages or interventions) Situational awareness regarding current concerns, issues, questions, emotions, of the public Eysenbach G. Infodemiology and Infoveillance: New Methods…. J Med Internet Res 2009: e11http://www.jmir.org/2009/1/e11

18. Absolute and relative number of H1N1/Swine-Flu Tweets % both % H1N1 % swine flu The relative proportion of tweets using “H1N1” increased from 8.8% to 40.5% in an almost linear fashion (R2= .788; p < .001), indicating a gradual adoption of the WHO-recommended H1N1 terminologyas opposed to “Swine Flu”

19. 53 % 23 % 14 % 8 % 1 % 2 % Chew C, Eysenbach G (2010) Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE 5(11): e14118

21. [P] Obama declares national emergency

22. [B] Harry Potter actor Rupert Grint has Swine FluChew C, Eysenbach G (2010) Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE 5(11): e14118

24. also social media campaigns show some effect ([G] #oink campaign of farmers)Chew C, Eysenbach G (2010) Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE 5(11): e14118

25. “Happiness / Humor / Mood Index”: Smileys : Frowneys Ratio

26. Question IndexNumber of tweets with ? : Total Tweets Chew C, Eysenbach G (2010) Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE 5(11): e14118

27. Prayer IndexNumber of tweets with “pray” : Total Tweets H1N1 Hospitalizations / Deaths

28. Personal Experiences Chew C, Eysenbach G (2010) Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE 5(11): e14118

29. Number of tweets with “personal experiences” correlates to H1N1 incidence Chew C, Eysenbach G (2010) Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE 5(11): e14118

30. Some newer findings on vaccination sentiment

31. Vaccine / Vaccination Mentionings Chew C, Eysenbach G (2010) Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE 5(11): e14118

32. Sentiment Categories

33. Manual Analysis: Anti-Vaccination Concerns

34. Manual Sentiment Analysis Green = Pro-vaccination Red = Anti-vaccination

35. % of H1N1 Pro- and Anti-Vaccination Tweets over 2009(keyword based) Red = anti Green = pro

36. Pro-Vaccination Tweets (keyword-based) Subconcepts: Green = support/promotion; Red = positive intent/behaviour; Blue = positive emotion; Yellow = rollout satisfaction 1 = Jun 18: “Vaccinate kids to control H1N1 flu: researchers” 2 = Jul 13: “WHO says health workers priority for H1N1 vaccine” 3 = Aug 22: “Swine flu vaccine seems safe in early trials” 4 = Sep 16: “FDA approves H1N1 vaccines”

37. Anti-Vaccination Tweets (keyword-based) Subconcepts: Green = vaccine health risks; Blue = conspiracy; Red = rollout dissatisfaction; Yellow = negative intent/behavior; Purple = negative emotion; Navy Blue = vaccine downplay/dissuasion 1 = Jul 8: “Canadian Doctor: H1N1 vaccination a eugenics weapon for mass extermination” 2 = Aug 16: “Swine flu jab linked to killer nerve disease: leaked letter reveals concerns of neurologists” 3 = Sep 27: “Doctor admits vaccine is more deadly than the H1N1 virus and wouldn’t give it to his children”

38. Another analysis: Number of retweets as measure of influence

39. Who was the most RT’ed? Retweets (RT) may be a measure of influence, reach, or interest

40. Other Heavily RT’ed Accounts Surprising Finding: With few exceptions, these are mainly the traditional “intermediaries”

41. Dr. Mercola, author of “The Great Bird Flu Hoax”

42. Conclusions Infoveillance: New methodology, offers wealth of quantitative + qualitative data, complementary to traditional survey methods, more timely and inexpensive Twitter is a rich source of opinions and experiences, which can be used for near real-time content and sentiment analysis, knowledge translation research, and potentially as a syndromic surveillance tool, allowing health authorities to become aware of and respond to real or perceived concerns/issues raised by the public Social media appeared underused by Canadian public health authorities during the H1N1 pandemic

43. Future work Natural language processing / machine learning approaches Grant proposal pending – re-engineer Infovigil to make it usable for other groups

44. “In the era of the 24-hour news cycle, the traditional once-a-day press conference featuring talking heads with a bunch of fancy titles has to be revamped and supplemented with Twitter posts, YouTube videos and the like. The public needs to be engaged in conversations and debate about issues of public health, they don’t need to be lectured to.”-Andre Picard Picard A (2010) Lessons of H1N1: Preach less, reveal more. Globe and Mail. Available: http://www.webcitation.org/5qYZly99e.

45. Principal Investigator:Gunther Eysenbach MD MPHDirector, Consumer Health & Public Health Informatics LabCentre for Global eHealth Innovationgeysenba@gmail.com Acknowledgements Thanks to CIHR & Reviewers Cynthia Chew (MSc Student): Coding & Qualitative Analysis of Tweets Latifa Mnyusiwalla (MHI Student): Vaccination Sentiment Analysis Marina Sokolova PhD, CHEO Ottawa: Natural Language Processing Phil Cairns: Developer

46. Appendix For Q & A

47. Challenges Data are noisy - advanced NLP (natural language processing) algorithms required not necessarily representative for the general population (young + educated) Spamming / twitter bots Geographic origin difficult to determine Privacy issues?

48. http://www.jmir.org/2009/1/e11

49. The science of distribution and determinants of disease in populations Epidemiology, Polls, Focus groups Public Health Interventions Policy Decisions Public Health Professionals Policy Makers Traditional Knowledge Translation Circle Population Behaviour, Attitudes, Health Status PR / Media Campaigns

50. Information & Communication patterns The science of distribution and determinants of disease in populations Epidemiology, Polls, Focus groups Public Health Interventions Policy Decisions Public Health Professionals Policy Makers Traditional Knowledge Translation Circle Population Behaviour, Attitudes, Health Status PR / Media Campaigns Web 1.0: Webpages, News Web 2.0: User generated content, social media Searches, Navigation, Clicks

51. Information & Communication patterns The science of distribution and determinants of disease in populations Epidemiology, Polls, Focus groups Public Health Interventions Policy Decisions Public Health Professionals Policy Makers Traditional Knowledge Translation Circle Population Behaviour, Attitudes, Health Status Infoveillance PR / Media Campaigns “Infodemiology”the epidemiology of informationAnalyzing information & communication patterns (on the web) Web 1.0: Webpages, News Web 2.0: User generated content, social media Searches, Navigation, Clicks Metrics