O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

Analytics Academy 2017 Presentation Slides

Carregando em…3

Confira estes a seguir

1 de 180 Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Quem viu também gostou (15)


Semelhante a Analytics Academy 2017 Presentation Slides (20)

Mais de HarvardComms (18)


Mais recentes (20)

Analytics Academy 2017 Presentation Slides

  2. 2. A/B Testing and Creating a Culture of Experimentation
  3. 3. A/B experiments show users different versions of your site and then compare results
  4. 4. TEST FIRST: FAST ● Can often mock up a feature in the testing tool first, without involving a tech queue ● Measuring results isn’t affected by seasonality, or other marketing efforts, or changes to the consumer mood because you are testing one group randomly divided, so all these factors are controlled for ● Results are statistically tested and validated BUILD FIRST: SLOW ● Higher up-front investment: Have to invest in building the feature without knowing if it will work ● Hard to measure results: you roll out the feature and compare conversion before (5-7% for last two months) to after (5.5-6.7% for month after). Is this an improvement or normal variation? Is it affected by seasonality? Did the email campaign that went out last week affect this rate? Benefits of A/B testing
  5. 5. Experiment 1: Tools in Store Hypothesis: ● Replacing "Tools" with a more learning-centered phrase will produce more click throughs What’s the result?
  6. 6. Experiment 1: Tools in Store Hypothesis: ● Replacing "Tools" with a more learning-centered phrase will produce more click throughs What’s the result? ● Learning Tools click rate up 16% ● Teach Yourself click rate up 27%
  7. 7. Hypothesis: ● Changing to one year would have no negative effect on subscription conversion What’s the result? ● What we think ● Experiment 2: One year vs 10 issues
  8. 8. Hypothesis: ● Changing to one year would have no negative effect on subscription conversion What’s the result? ● One year variation subscription rate up 9% What we think ● People understand the value of a year more than number of issues Experiment 2: One year vs 10 issues
  9. 9. Experiment 3: Formatting on product page Hypothesis: ● Improving formatting would increase product purchase What’s the result? ● What we think ●
  10. 10. Experiment 3: Formatting on product page Hypothesis: ● Improving formatting would increase product purchase What’s the result? ● No statistical difference in product purchase What we think ● Was formatted description too long? Should we have short text and preview page?
  11. 11. A/B testing can inspire cultural change ● Practice with A/B tests builds experimentation muscles ○ People practice the steps to build a good experiment so they start to feel obvious ○ A/B tests require good methodology: you are forced to pick a goal to measure; you automatically have a control group; the software collects and reports on the results ○ The benefits of the speed/clarity from these experiments increase demand for similar speed/clarity in areas outside the website Ideal state for all business stakeholders for all questions: always ask, “Can this be an experiment?”
  12. 12. Basic human nature makes this hard ● Short term, it feels easier to make a decision based on gut feel, or defer to highest paid person’s opinion (HiPPO), or just try something and see what happens without formalizing a hypothesis or measuring the result (but still call it an experiment) What makes it hard to experiment?
  13. 13. In the long run, it is actually a LOT easier to run an experiment ● Fail fast: have an idea? Experiment with a minimum viable product to see if the idea deserves further development--or not ● Decisions vs more discussion: The organization can move a lot faster when there’s certainty around a course of action. When there’s uncertainty, healthy discussion can sometimes sour into multiple meetings and prolonged debates, or, perhaps worse, unspoken doubts sap the momentum for the group moving forward Experimentation done well becomes self-reinforcing, as people see how much easier/faster they can work A/B testing shows experiments are easier
  14. 14. ● New product development: set out hypotheses about the market (e.g. “managers want to buy HBP materials to help their direct reports”) and then test with customers (e.g. customer interviews where we learn that there’s an equally large market from coaches). Key is to build and test in stages, so you validate hypotheses along the way ● Email testing: split list as randomly as possible and send different emails to the two groups ● Before and after testing: Create an insider newsletter and compare subscriber engagement before and after ● A/B testing: use a formal A/B testing platform on the website Some ways that HBR experiments
  15. 15. Change is as good as a rest: Sometimes a change tests well just because it is a change and gets people’s attention. Change a button from red to blue, you may get higher click throughs; effect diminishes over time, then six months later change it back to red and get higher click throughs Focus on the big picture: don’t look at a change in isolation; look at the total impact. Adding a newsletter widget that gets clicks is good, but does it increase the total newsletter signups (or just cannibalize the clicks you are getting from other widgets)? Does the additional visual clutter lower overall engagement (higher bounce, lower time on site)? Follow some best practices
  16. 16. But remember we are not a lab Having a culture of experimentation does not mean that your group transforms into a medical lab where we need 98% certainty and huge sample sizes to make a decision Perfect is the enemy of the good: It’s better to have 20 good experiments than 3 perfect ones Hurdle: is the experimental results better information that what you would have used otherwise? (e.g. better than gut instinct?)
  17. 17. Questions?
  18. 18. Resources Google analytics experiments: FREE! (but anecdotally pretty hard to use) Optimizely: easy interface, great training resources to help you get acquainted with testing (we started here) VWO: may be cheaper than Optimizely Adobe Target: more robust, integrates with Adobe Analytics in a very powerful way (we moved here in January)
  19. 19. Facebook Ads: Audiences and Impact Joseph Casciano, Harvard Public Affairs and Communications
  20. 20. How much to spend? Why are we spending it?
  21. 21. ● Low-value objective ● Larger audience ● High-value objective ● Smaller audience
  22. 22. Problems with Interest Targeting ● Often inaccurate, so serves your ad to the wrong people. ● Hard to get a precise, smaller, well-targeted audience. ● To big audiences, Facebook serves your ad to whoever is cheap and easy.
  23. 23. Business Manager ↓ Assets ↓ Audiences ↓ Create Audience ↓ Custom Audience
  24. 24. Metric Time
  25. 25. Relevance Score
  26. 26. CPM (cost per 1,000 impressions)
  27. 27. Results and Reach
  28. 28. Custom audiences make metrics matter ● Don’t provide contextless numbers about faceless masses. ● Tell concrete, true stories about your valued audience. ● So: “We reached half of our email subscribers on Facebook, half of who watched the video for more than three seconds.” ● Not: “We reached 132,674 Malaysian bots who couldn’t even theoretically fly into Cambridge for our symposium.”
  29. 29. ● Use custom audiences ○ To guide your budget ○ To make metrics matter
  30. 30. Questions? joseph_casciano@harvard.edu
  31. 31. Google Analytics Tips Tricks Elizabeth Brady, EWB Analytics
  32. 32. Intro Elizabeth Brady, Founder & Principal Web Analyst - EWB Analytics LLC - launched March 2010 Specialties: Google Analytics and Google Tag Manager implementations, site audits, web analytics support during site re-launch, measurement strategy and ongoing analysis Harvard groups I have collaborated with since 2012: Digital Communications, Harvard Alumni, Harvard Admissions, Harvard Innovation Lab, Kennedy School, Harvard Library, Harvard Learning Portal, HWPI, Ash Center Contact: elizabeth@ewbanalytics.com
  33. 33. Keep It Clean
  34. 34. Filters Can Help Prevent Internal Traffic m 1 Maintain ‘exclude’ filter of known internal IP addresses Ghost Spam Maintain ‘include’ filter of valid site hostnames Crawler Spam Maintain ‘exclude’ filter of list of known spam referrers
  35. 35. Filter: Exclude Internal Traffic by IP Address Internal traffic inflates conversions & conversion rates. Check current IP address by visiting whatismyip.com IP Addresses Use regular expressions for a range of IP addresses (ask IT for office IP range) Dynamic IP Addresses (residential) Verify/update regularly Test Activity Use custom dimensions to track test users even when not on internal network
  36. 36. Filter: Test Accounts by Custom Dimension ● Use a custom dimension to identify a test visitor who visits a specific internal page (webadmin, test, etc) ● Set the custom dimension at the ‘user’ level ● Create a filter for any traffic with that custom dimension value
  37. 37. Ghost referrer spam: ● Never actually visits your site ● Sends data via the ‘measurement profile’ randomly to your GA account (became an issue only with Universal Analytics) ● Sends data with a missing (not set) or inaccurate hostname ● Can be prevented with a valid ‘hostname’ (include) filter Prevent Ghost Referrer Spam
  38. 38. Filter: Valid Hostname(s)
  39. 39. Crawler spam: ● Actually crawls/visits your site so the traffic appears legitimate ● Filter this traffic by filtering on ‘campaign source’ ● Sample ‘exclude’ filter for known spam crawlers and domains referenced as referrals from spam crawlers: semalt|anticrawler|best-seo-offer|best-seo-solution|buttons-for-website|buttons-for-your-website|7makemoneyonline|-musicas*-grat is|kambasoft|savetubevideo|ranksonic|medispainstitute|offers.bycontext|100dollars-seo|sitevaluation|dailyrank ● Full set of 4 filters for crawler spam can be found here: http://help.analyticsedge.com/spam-filter/definitive-guide-to-removing-google-analytics-spam/ Filter: Crawler Spam
  40. 40. Language Spam - New Spam in 2016 Language Spam ● Rather than referrers, the spamming sites inserted spam messages into the ‘language’ reports ● Most do not use valid hostnames so this would also be prevented with a ‘hostname’ include filter ● Additional exclude filters can be added to address language spam
  41. 41. Lowercase Filters ● Especially when starting a new view, lowercase filters can avoid capitalization inconsistencies ● Recommended for - page, campaign (medium/source/campaign), search term (on site search)
  42. 42. Query String Cleanup ● Google Analytics includes any query string parameters (after the ‘?’ as part of the URL) ● Leads to multiple versions of the same ‘page’ and a challenge aggregating data ● Parameters to exclude can be identified in a list in the view settings, or you could ‘go nuclear’ and exclude them all with this filter on the right ● Full URL with query strings (or just query strings) can be captured as a custom dimension to be viewed when needed
  43. 43. Check Your Setup
  44. 44. Referrer Exclusion List (Property Level) ● Make sure your site subdomain/s is included (new properties set up with Universal Analytics will have this set up on creation but any older site rolled over to Universal Analytics did not automatically have this configured) ● Include any off-site flows (login validation, back-end sites like pin1.harvard.edu) to prevent triggering a new session ● Do not set up harvard.edu in the exclusion list (that will prevent any other Harvard sites showing up as ‘referrals’) - they will be ‘direct traffic’
  45. 45. Check ‘Exclude Bots & Spiders’ (View) ● Excludes traffic from sites on the IAB (Interactive Advertising Bureau) list of known bots & spiders ● Sometimes these can be contracted services like site response time (like Gomez) that execute javascript and would otherwise show up in reports ● It is recommended to leave this unchecked for the unfiltered view
  46. 46. Link Google Search Console (Property) for Organic Search Trends ● Google Analytics no longer has much insight into Google organic search keywords ● Link your site’s Search Console (formerly ‘Webmaster Tools’) account for impressions/clicks on Google
  47. 47. Remember Campaign ‘Timeout’ (Property) is Configurable ● Standard campaign setting is 6 months (sessions and conversions will be credited to the last campaign in the past 6 months) ● This explains why you may see traffic/conversion for ‘old’ campaigns ● Your business group may decide you need a shorter or longer campaign timeout
  48. 48. Data Import (File Upload) Can Extend Analysis ● Data import lets you append data to any dimension (standard or custom) you collect ● The actual import can be a simple text file upload ● Some uses might be to add details around campaigns, add authors or other details to content pages, or group information differently than they way it is grouped in Google Analytics
  49. 49. Reporting & Analysis
  50. 50. Custom Reports Don’t dig for your data! 404s Social Media Details Top Pages by Type Top Events by Type Deep dive into a certain source of traffic (ex: email campaigns)
  51. 51. Tracking 404’s ● No extra tagging needed ● Report Filter: Page title contains ‘Page Not Found’ ● Dimensions: URL (page requested), Previous Page (might be entrance), source/medium (more important for entrance pages to understand source of traffic)
  52. 52. 404 Error Report ● Monitor 404 volume over time ● Monitor broken links ● Set up 301 redirects where needed ● This report is very helpful after a site re-launch
  53. 53. Social Media Details ● Report Filter: Channel = Social ● Dimensions: Medium, Social Network (or Source) ● ‘Social’ = tagged social campaigns, ‘referral’ = organic social traffic (no camppaign tags)
  54. 54. Pages by Type ● Report Filter: Page contains <URL identifier for type of content> ● Dimensions: Page Possible content: blogs, story pages, article pages, FAQ’s
  55. 55. Events by Type ● Report Filter: Event category = ______________ ● Dimensions: Event label, event action Possible events: document downloads, offsite links, navigation links, carousel clicks
  56. 56. ‘Unique’ Metrics - Pageviews ● To report the number of sessions that viewed a page, use ‘unique pageviews’ ● GOTCHA - do NOT combine page with sessions as a custom report (GA WILL let you set this up, but sessions are ONLY associated with the entry page)
  57. 57. ‘Unique’ Metrics - Events ● To report the number of sessions that recorded a certain event, use ‘unique dimension combinations’ ● This shows the sessions with the event for whatever dimension combination is presented in the report
  58. 58. Custom Segments ● Create a custom segment to filter any report by sessions/users meeting specific criteria ● Some common segments include: ○ Sessions from a specific campaign ○ Sessions that viewed a specific page ○ Sessions that registered a specific ‘event’ ● Then apply a segment to a basic or custom report, for example: ○ Geographic reports ○ Traffic (source/medium) reporting ○ Technology: device/browser/OS reporting
  59. 59. Custom Segment - Viewed the homepage ●
  60. 60. Custom Channel Groupings ● CUSTOM channel groupings give you the flexibility to roll up the data the way you want to see it ● They are retroactive, but are specific to the user account where they are created but can be shared like other assets ● For the Gazette, we break out Harvard.edu referrals, other Harvard referrals, and non-Harvard referrals as separate ‘Channels’
  61. 61. Custom Channel Groupings ● CUSTOM channel groupings are a view-level setting - be sure to find the ‘custom’ groupings rather than the ‘channel groupings’ (changes to the core channel groupings will not be retroactive and will only impact data collection moving forward)
  62. 62. Helpful Toolkit
  63. 63. Google Tag Assistant Chrome Extension ● Quickly check the status of Google Analytics and Tag Manager code on any page ● Red/yellow warnings identify tagging problems
  64. 64. EditThisCookie Chrome Extension ● View cookies set on a site ● Delete selected, or all, cookies on the site without having to clear all of your cookies for other sites
  65. 65. Google Data Studio Google’s new dashboard Tool now offers free, unlimited dashboards, with great integration with Google Analytics and other Google products Features: interactive filters, flexible formatting, multi-page dashboards datastudio.google.com
  66. 66. Takeaways - Top Tip From Each Topic! 1. Filters - maintain data integrity by collecting the cleanest data you can in your production a view (a non-filtered view should also exist), slides 4-12. 2. Settings - key settings to check include referral exclusions (include your own harvard SUBdomain/s) and make sure bots/spiders are excluded. 3. Reporting - remember to use ‘unique’ metrics when reporting the number of sessions with a specific page/event. 4. Tools - download ‘Google Tag Assistant’ for very user-friendly feedback on tag set-up and data collection. elizabeth@ewbanalytics.com - Feel free to reach out with specific questions!
  67. 67. Copyright © President & Fellows of Harvard College Responding to Analytics with SEO Marcus Dandurand - March 30th , 2017
  68. 68. How to Measure Traffic from Search Engines In Google Analytics: Source/Medium = Google/Organic In Adobe Analytics: Marketing Channel = Natural Search 2
  69. 69. Are we getting enough Search traffic? 3
  70. 70. Optimization is never done! 4 ➢ Benchmark against yourself ➢ Compare traffic Year-Over-Year ➢ Try optimizing existing pages ➢ Create new pages to target new keywords ➢ Think beyond “branded” keywords
  71. 71. Search Traffic Year-Over-Year 5
  72. 72. Wait… What?! Traffic is down!!! Is it something we did? Did Google change its algorithm? Will it fix itself? What can we do? 6
  73. 73. First, a few SEO myths... 7 1. We don’t know what Google wants 2. The algorithm changes too often 3. SEO is an attempt to “game the system” 4. SEO is a job for IT 5. My CMS has SEO built-in
  74. 74. What do search algorithms care about? 8 Relevance Performance Authority
  75. 75. What is page “Relevance”? Your page content closely matches a keyword search phrase. 9
  76. 76. A few ways to improve a page’s relevance: 10 ➢ Keyword Research - Relabel content using “outside voice” ➢ Breakup Content - Each important idea should have its own landing page ➢ Accurately describe all page components ∙ Page Titles/Meta data ∙ Navigation links ∙ Section Headers ∙ Etc.
  77. 77. What is page “Performance”? Pages are useful, Pages load fast, Site is accessible to humans & robots. 11
  78. 78. A few ways to improve a page’s performance: 12 ➢ Reduce page load time ➢ Make sites mobile friendly ➢ Improve click-through rate in search engine results ➢ Make sure websites can be crawled/indexed properly by search engines
  79. 79. What is page “Authority”? Every link acts as an endorsement of a page’s credibility. Both External and Internal links! 13
  80. 80. Authority began as Google “PageRank” 14
  81. 81. A few ways to improve a page’s authority: 15 ➢ Create resources that people will share (inbound linking) ➢ Use 301 redirects (site cleanup) ➢ Restructure website navigation to distribute authority to your most important pages Distributing Authority: Are the important links on your page 1/10 or 1/100?
  82. 82. Working Knowledge Website Updates: SEO lessons learned the hard way. 16
  83. 83. Mid August 2015 17
  84. 84. Working Knowledge has a search traffic problem! 18
  85. 85. Working Knowledge search traffic improves! 19
  86. 86. Working Knowledge search traffic improves! 20 ➢ Sept – Nov 2015: down 43% vs. Prior Year ➢ Sept – Nov 2016: up 46% vs. Prior Year
  87. 87. So, what did we fix? 21
  88. 88. 1) New dynamic landing pages 22 ➢ Hundreds of Topic landing pages were not indexed by Google. New browse page was behaving like a single dynamic page. (Performance) ➢ Each page had the same Title & Meta Description (Relevance)
  89. 89. 2) Deleted the Working Knowledge Archive 23 ➢ Several articles were still very popular for search traffic. (Relevance) ➢ Many pages were still cited and linked to by important sources. (Authority) Without proper redirects, the authority passed back to the home page was lost.
  90. 90. Other optimization for Working Knowledge 24 ➢ Created customized Titles & Descriptions for each page in the CMS. (Relevance)
  91. 91. Other optimization for Working Knowledge 25 ➢ Created new “display descriptions” visible on-page. (Relevance)
  92. 92. Are You Making A Major Website Update? Please consider the following... 26
  93. 93. #1) Navigation Links Transfer Page Authority 27 ➢ Primary navigation create backlinks from every page on your site. ➢ Try to put your important pages in your primary navigation (but only if it makes sense). ➢ Try to remove links that are nice to have, but not critical. ➢ Adding new links will dilute the authority of existing links.
  94. 94. #2) All URL Changes Need 301 Redirects 28 ➢ 404 errors create a bad user experience and waste page authority. ➢ 302 redirects are “temporary,” so Google keeps the old page in the index. No authority is passed! ➢ 301 redirects are “permanent,” so the authority of old pages are passed.
  95. 95. #3) Avoid Duplicate Content 29 ➢ Each page should have a unique Title & Description. ➢ You should not be able to see the same page via two different URLs.
  96. 96. Pop Quiz: 30 Which of the following URLs below are exactly the same as: www.hbs.edu/mba A. www.mba.hbs.edu B. hbs.edu/mba C. http://www.hbs.edu/mba D. https://www.hbs.edu/mba E. All of the above
  97. 97. Free Tools for SEO Analytics 31
  98. 98. Free Tools for SEO Analytics 32 ➢ MozBar (Browser Extension) ● Page Authority, Inbound links ➢ SiteImprove - Free through HUIT! ● Missing Page Titles, Descriptions, Broken Links, 302 redirects ➢ Google Search Console - Formerly “Google Webmaster” ● 404 errors, Organic Keywords, XML sitemap, Mobile issues ➢ Link Redirect Trace (Browser Extension) ● Follow the path of multiple redirects
  99. 99. Thank You! 33
  100. 100. From Big Data to Insights in Massive Open Online Courses A Traveler’s Guide Daniel Seaton Harvard University Sr. Research Scientist VPAL Research Team
  101. 101. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 • Consortium of Institutions creating MOOCs • Maintain Open-Source Platform • Host Courses/Content • Lead Outreach • Maintain “https://www.edx.org” • Partners from Higher Ed / Industry / Government / High Schools • Create Courses/Content • Manage Courses in Open Online (MOOC) and On-Campus (residential) settings. • Perform Research into Teaching and Learning Consortium Members What is edX?
  102. 102. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Data from Dec. 2016 http://harvardx.harvard.edu/ Harvard University’s MOOC Organization: • Partners with faculty to create open online courses • Supports initiatives to use MOOC content beyond open online models
  103. 103. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Data from Dec. 2016 Harvard University’s MOOC Organization: • Partners with faculty to create open online courses • Supports initiatives to use MOOC content beyond open online models http://harvardx.harvard.edu/
  104. 104. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Research Timeline and Perspective 2012 What are learners doing? Who and where are our learners? Besides open online, how else can we use MOOC platforms and content? Why are learners taking courses? 2013 2014 2015 2016 Single MOOC Transforming Advanced Placement High School Classrooms Through Teacher-Led MOOC Models Seaton, Hansen, Goff, Houck, Sellers Many MOOCs Context around MOOC enrollments Alternative MOOC Models
  105. 105. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Single MOOC
  106. 106. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 • 6.002x: Circuits and Electronics (first MOOC from MITx - now edX) • Over 100K enrollees • Over 7K certified users • Over 100GB of data from clickstream • Limited profile information • MITx now a member of edX: ~ 100 open access courses
  107. 107. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 We started with “what” people are doing in 6.002x Transition between resources Nodes are resources (size ~ time spent) Edges are transitions (size ~ weight) Who does what in a Massive Open Online Course? Seaton, Bergner, Mitros, Chuang, Pritchard ( Comm. of the ACM - 2014) Analyzed learner interactions with all aspects of 6.002x. Particular focus on time-on-task and resource-use during problem solving. Measurements • Time-on-Task • Resource Interactions • Daily/Weekly Progress • Transitions between resources during problem solving
  108. 108. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Many MOOCs + Context Around Enrollments
  109. 109. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 • HarvardX and MITx Working Paper #1 • Now had access to all course data from MITx and HarvardX • Addressed “what” people were doing, and “who” they are, across 17 MITx and HarvardX courses Key Takeaways: 1. Courses are very different. 2. Registrant diversity is immense compared to residential. 3. Participation greatly varies. Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
  110. 110. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 What are learners doing across MITx and HarvardX? %Grade % Chapters Accessed0 100 100 Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
  111. 111. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 What are learners doing across MITx and HarvardX? Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
  112. 112. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Cross course surveys launched in 2014 addressing broad issues across MITx, but teaching experience was central issue. Results from 11 spring 2014 MITx MOOCs: • 28.0% (9451) self-identify as past or present teachers (navy). • 8.7% (2847) current teachers (orange). • 5.9% (1871) teach/taught the topic (gray). On average across courses, ~ 8% (1 in 12) of comments are from current teachers. For teachers that teach/taught the topic, the average across courses is ~6% (1 in 16). Percent Comments in Forum Did not take survey Surveyed Surveyed Teachers Non-Teachers 43.8% 22.4% 33.8%
  113. 113. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Why are we still not talking about course structure/design? + Visualizing Course Design
  114. 114. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Goals and Motivations • Support HarvardX by collecting relevant stats on course structure. • From a research perspective, identify canonical patterns in course development and better understand how those patterns affect behavior and outcomes. Human beings, viewed as behaving systems, are quite simple. The apparent complexity of our behavior over time is largely a reflection of the complexity of the environment in which we find ourselves. - Herbert Simon, “The Science of the Artificial” Practical Motivation Abstract Motivation http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research
  115. 115. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Visualizing Course Design Key point: Use these visualizations to look across courses.
  116. 116. daniel_seaton@harvard.eduAnalytics Academy - March, 2017
  117. 117. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research
  118. 118. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Alternative MOOC Models AP High School Content/Courses
  119. 119. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 • Boston Public School students • Take MOOC online during school • Commute to BU weekly for labs/recitations with TAs/Faculty Of 34 regular and charter schools serving 16,165 students, 2 high schools offer algebra based AP® Physics 1. Only 60 BPS students took the AP® Physics 1 exam during the 2014-2015 school year. BU Project Accelerate • Open-online and teacher-led/flipped • All content open on edx.org • Special instances for teachers to use content in classrooms • Showed 0.08 added to AP exam score per hour usage above class average http://vpal.harvard.edu/blog/complementary-models-mooc-instruction-advanced-placement%C2%AE-high-school-courses
  120. 120. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Davidson Next - AP content for use by teachers and students Program at Davidson College: • Supplemental content for 14 Challenging Concepts in each AP subject. • Challenging concepts determined using College Board exam data from 2011 to 2013. Piloted with Charlotte-Mecklenburg School System in 2014-2015 school year. • Modules designed for each concept meant to facilitate use both in classrooms, and open online. Real AP Teachers from developed content with Davidson faculty. • Courses released on edX.org and through a new Custom Course tool (CCX).
  121. 121. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Bubble Charts for Detecting Daily Activity • Made these to help monitor teacher use of Davidson Next in Charlotte High Schools • Full time assessment coordinator worked with teachers on implementation and efficacy of content (collected district data and AP exam scores). Transforming Advanced Placement High School Classrooms Through Teacher-Led MOOC Models Seaton, Hansen, Goff, Houck, Sellers (MIT LINC Conference - May 2016) Pilot program in North Carolina High Schools Massive Open Online Courses via edX.org Exam score residuals are then correlated with student usage relative to class median indicating 0.08 points per hour spent (p<0.05).
  122. 122. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Building Community Around Analytics…
  123. 123. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Open Source Tools for edX Data • Harvard and MIT already share resources and code for analytics • https://github.com/mitodl/edx2bigquery • https://github.com/mitodl/xanalytics • Open-Source Repos • Python + Google BigQuery for aggregation of edX data. • Dashboard via Google App Engine
  124. 124. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 edX Data Workshop Summer 2016 Meeting of data analysts and engineers in institutional roles responsible for edX data; 16 attendees from 11 institutions. Goals for meeting: • Discuss broader aspects of data sharing and analytics. • Standup the Harvard/MIT edX Data Pipeline. • Happy to report that each participant completed this task. • Next workshop summer 2017? • Hoping to broadly release workshop documentation in the spring. http://news.harvard.edu/gazette/story/2016/07/moocs-ahead/
  125. 125. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Visit our blog: http://vpal.harvard.edu/blog
  126. 126. daniel_seaton@harvard.eduAnalytics Academy - March, 2017 Many collaborators to thank before dicussion! HarvardX Andrew Ho, Dan Levy, Jim Waldo, John Hansen, Sergiy Nesterko, Justin Reich, Tommy Mullaney Miki Goyal, Gabe Mulley, Carlos Rocha, Victor Schnayder, Olga Stroilova, Brian Wilson Julie Goff, Aaron Houck, Kristen Eshleman, Pat Sellers, Noelle Smith Yoav Bergner, Cody Coleman, Isaac Chuang, Curtis Northcutt, David Pritchard, Saif Rayyan VPAL Research Team Andrew Ang, Glenn Lopez, Brooke Pulitzer, Yigal Rosen, Dustin Tingley, Selen Turkay, Jacob Whitehill, Joseph Williams Lydia Snover, Jon Daries, Mark Hansen Research and Analytics
  127. 127. Ditch the spreadsheet and tell the story Katie Hammer Office for Sustainability and Harvard Public Affairs and Communications
  128. 128. How do you get your team to understand your analytics story?
  129. 129. MATERIALS AND CONTENT CREATED: GHG Landing page (OFS) 4 Page Climate Report PDF Community-wide message from President Faust Wide-format Gazette Story and Graphics 8 #HarvardClimateStories instagram profiles Video targeted at social media Custom social graphics for 12 Schools + departments harvard.edu/climate modules
  130. 130. Community-wide email sent by President Faust 52% open rate 867 clicks Gazette Story in the Daily Gazette 22% open rate 637 clicks Social Promotion (Harvard & OFS) ● Video with paid boost ● Twitter & Twitter Moment ● #HarvardClimateStories Instagram Campaign ● 108,000+ video views ● Almost all Schools shared news w/ graphics ● 17,694 likes; 84 comments Inclusion in OFS December email newsletter 25% open rate 507 clicks Feature on Harvard.edu 3,010 clicks External Press (Crimson, Harvard Magazine Story,NY Times, Boston Globe, WGBH etc.) DISTRIBUTION EFFORTS:
  131. 131. OFS Goal Page: 335 Gazette Story: 210 HUCE Site: 301 Clicks
  132. 132. Try something new? Flaunt it.
  133. 133. FACEBOOK AD CAMPAIGN: OFS OFS Ad Spend: $170 Duration: December 8 - 12 Audience: Targeted students and alumni (where Harvard was listed as School and age was 18+); OFS email list; People who liked our Facebook page Reach (number of people that saw the post): ● 42,019 total people reached ● 16,624 people reached as a result of paid ● 17,000 total video views; ● Cost per 1,000 people reached $10.23 ● $.05 per 10 second video view Engagement (reactions, comments, shares): ● 7,696 total actions ● $0.27 per engagement ● 23 link clicks ● Post generated 55 new GreenHarvard Facebook page likes Context: ● Typical GreenHarvard video is viewed ~500 times Note: Many comments did include mention of divestment; however about half were positive and congratulatory.
  134. 134. Qualitative data matters too.
  135. 135. TWITTER STRATEGY: ● Worked with Facility teams to create custom graphics optimized for social for Schools to use ● Partnered w/ HPAC to share on the @Harvard accounts ● Outreach in advance to all digital counterparts at Schools/Depts ● Created a Twitter Moment to capture various influencer and School tweets about announcement RESULTS: ● Initial tweet: Retweeted 66 times; Liked 102 times, Clicked 34 times ● Moment tweet: Retweeted 33 times, Liked 92 times, Clicked 90 times ● Retweets and original tweets from internal “influencers” like HBS, HSPH, HAA ● Almost all 12 Schools promoted us in some way, in addition to the Museums, Libraries, and various departments
  136. 136. And they all lived happily ever after...
  137. 137. Lessons learned and opportunities ● Targeted outreach to Schools and Depts works; Schools/Depts find easier to promote when can link data/anecdotes back to them (social graphics received well). ● While School/Dept outreach worked and we did have some external influencer tweets (Climate Registry, USGBC), we should develop a more solid plan for faculty and social influencers in the future. ● #HarvardClimateStories campaign a success; 8 profiles in a month was ambitious; for future campaigns could start earlier and conduct interviews and shoots further in advance. ● Important to consider different outreach methods for different audiences; for example the social video was extremely brief but gave an external audience the message they needed “Harvard set an ambitious goal and they met it.” ● We should consider allocating time more evenly across a wide range of projects, considering goals, audiences, and reach (PDFs, videos, social campaigns). For example, though PDF a considerable amount of our time, the social/web reach was minimal.
  138. 138. vs.
  139. 139. Keys to a telling a good story Format Choose a vehicle that’s relatable (even if that means powerpoint). Keep it simple. Style Use language that seems right for your story (and for your client). Setting Set your story by bringing in context to explain the why. Remember you control this! Themes Let the themes of your data shine by weaving them throughout your story. Illustrations Images, examples, and visual cues only add to your story. Conclusion End your story with lessons learned and opportunities that leave the reader ready for your next story!
  140. 140. Thanks! Any questions ? ◉ kate_hammer@harvard.edu
  141. 141. Strategize, Synthesize, and JAZZERCIZE® your Analytics Dashboards and Reports with your host Aaron David Baker
  142. 142. Remember Jazzercize® ? Problem: Non-dancers are taking Jazz Dance classes because it is a great workout but aren’t interested in all the work on form and technique. They just wanna have fun while exercising. Solution: Create a fun jazz dance-style fitness class that’s interesting and fun!
  143. 143. The power of FUN
  144. 144. The problem was neither exercise nor jazz dance class Exercise is a chore; we make it fun to get it done.
  145. 145. It’s the same with creating/consuming analytics data
  146. 146. Give them what they want! The people who consume your reports want them to be engaging. They don’t always have to contain charts and graphs. Good document design also conveys professionalism. Don’t just report on the content you produce or see, share screenshot highlights to give context.
  147. 147. Great reports are persuasive and can change attitudes
  148. 148. Introducing Scoop HPAC’s analytics dashboard, built around public APIs to Google Analytics, Facebook, Silverpop, and more to come Presents consistent, up-to-date performance data per story, post, and mailing Makes comparative metrics possible with benchmarking and visualizations
  149. 149. Introducing Scoop We have a Strategy ● Identify and capture metrics that matter in one convenient place We have Synthesis ● Stats from many different platforms in one place for easy reporting We are working on that Jazz ● Make it beautiful and fun
  150. 150. Strategize
  151. 151. The official Harvard Style Guidelines & Best Practices site has an updated Analytics with resources and setup information. harvard.edu/guidelines Use these best practices to ensure your site is up-to-date with the latest analytics code and tracking practices. First, check that everything is in order
  152. 152. Second, define key stats (these are just some examples) Health stats are these ● Users/Sessions/Pageviews ● % New Users over time ● Page speed ● 404s Strategic metrics are these ● Content performance (pageviews) ● Content engagement ○ Time on page ○ Scroll depth ● Users by ○ New/returning ○ Geolocation ○ Content sections they visit ○ Frequency of visits ● Content dimensions ○ Content category/section/tag ○ Content length ● Acquisition paths ○ Search keywords
  153. 153. Choose goals and metrics that make sense for your organization!
  154. 154. Report on what’s exceptional and on what’s important What’s exceptional ● Top content in terms of pageviews and/or engagement metrics (time and scroll) ● Large number of people reached or high number of impressions ● Social activity (likes, comments, shares, retweets) ● Unusual spikes in traffic* or unknown sources What’s important ● Key initiatives ○ President Faust’s priorities ○ Special Gazette features ● Things you spent money on ○ Paid social, AdWords, etc. ○ Extra money gives you extra metrics ● Experiments ○ A/B testing ○ SEO ● Conversions
  155. 155. Here we’ve chosen to highlight pageviews by channel and accumulated pageviews over time ● Helps us visualize content distribution and sources of traffic. ● Benchmarking is our own comparative metric—average daily pageviews of all stories in Scoop. Scoop Example
  156. 156. Synthesize
  157. 157. The chore of reporting ● Go here ● Find stats ● Copy/paste into doc ● Go there ● Find stats ● Copy/paste into doc ● …
  158. 158. Wouldn’t it be nice if the stats gathered themselves?
  159. 159. Scoop pulls in stats from ● Google Analytics ● Facebook ● Silverpop With plans to incorporate ● Twitter ● Instagram ● YouTube ● Etc. Anything with an API can be consumed. Scoop Example
  160. 160. Synthesis isn’t just about automation Automation is nice and does save a lot of time. Linking things by a common element (like URL or topic) can make finding the stats easier. Synthesis is really about telling the whole story. Automated reporting cannot speak for you. Talk about the why in your reports.
  161. 161. My Weekly Report Ultimate synthesis of what happened last week Mostly highlighting what’s exceptional Occasionally mentioning what is bizarre Always as interactive as possible ● links go to actual online posts or to Scoop itself
  162. 162. Jazzercise!
  163. 163. Back to Jazzercize® Our reports are working, and people come to us for information, but how can we analytics reports more enjoyable?
  164. 164. Design and data visualizations: ● Beautiful, clean, contemporary, and inviting design ● Display visually our data’s trends, patterns, and correlations ● Provide content creators and distributors with intuitive, at-a-glance insights about performance of published work ● Be designed with interactive development in mind The plan for Scoop
  165. 165. Your turn to talk: How do you add jazz to your reports?
  166. 166. Thanks y’all!