Analytics Academy 2017 Presentation Slides

A/B Testing and Creating a
Culture of Experimentation

A/B experiments show users different
versions of your site and then compare results

TEST FIRST: FAST
● Can often mock up a feature in the testing
tool first, without involving a tech queue
● Measuring results isn’t affected by
seasonality, or other marketing efforts, or
changes to the consumer mood because
you are testing one group randomly
divided, so all these factors are controlled
for
● Results are statistically tested and
validated
BUILD FIRST: SLOW
● Higher up-front investment: Have to invest
in building the feature without knowing if it
will work
● Hard to measure results: you roll out the
feature and compare conversion before
(5-7% for last two months) to after
(5.5-6.7% for month after). Is this an
improvement or normal variation? Is it
affected by seasonality? Did the email
campaign that went out last week affect
this rate?
Benefits of A/B testing

Experiment 1: Tools in Store
Hypothesis:
● Replacing "Tools" with a more
learning-centered phrase will produce
more click throughs
What’s the result?

Experiment 1: Tools in Store
Hypothesis:
● Replacing "Tools" with a more
learning-centered phrase will produce
more click throughs
● Learning Tools click rate up 16%
● Teach Yourself click rate up 27%

Hypothesis:
● Changing to one year would have
no negative effect on subscription
conversion
●
What we think
●
Experiment 2: One year vs 10 issues

Hypothesis:
● Changing to one year would have
no negative effect on subscription
conversion
● One year variation subscription
rate up 9%
What we think
● People understand the value of a
year more than number of issues
Experiment 2: One year vs 10 issues

Experiment 3: Formatting on product page
Hypothesis:
● Improving formatting would
increase product purchase
●
What we think
●

Experiment 3: Formatting on product page
Hypothesis:
● Improving formatting would
increase product purchase
● No statistical difference in
product purchase
What we think
● Was formatted description too
long? Should we have short text
and preview page?

A/B testing can inspire cultural change
● Practice with A/B tests builds experimentation muscles
○ People practice the steps to build a good experiment so they start to feel obvious
○ A/B tests require good methodology: you are forced to pick a goal to measure; you
automatically have a control group; the software collects and reports on the results
○ The benefits of the speed/clarity from these experiments increase demand for similar
speed/clarity in areas outside the website
Ideal state for all business stakeholders for all questions: always
ask, “Can this be an experiment?”

Basic human nature makes this hard
● Short term, it feels easier to make a decision based on gut feel, or defer to
highest paid person’s opinion (HiPPO), or just try something and see what
happens without formalizing a hypothesis or measuring the result (but still call
it an experiment)
What makes it hard to experiment?

In the long run, it is actually a LOT easier to run an experiment
● Fail fast: have an idea? Experiment with a minimum viable product to see if
the idea deserves further development--or not
● Decisions vs more discussion: The organization can move a lot faster when
there’s certainty around a course of action. When there’s uncertainty, healthy
discussion can sometimes sour into multiple meetings and prolonged
debates, or, perhaps worse, unspoken doubts sap the momentum for the
group moving forward
Experimentation done well becomes self-reinforcing, as people see how much
easier/faster they can work
A/B testing shows experiments are easier

● New product development: set out hypotheses about the market (e.g.
“managers want to buy HBP materials to help their direct reports”) and then
test with customers (e.g. customer interviews where we learn that there’s an
equally large market from coaches). Key is to build and test in stages, so you
validate hypotheses along the way
● Email testing: split list as randomly as possible and send different emails to
the two groups
● Before and after testing: Create an insider newsletter and compare subscriber
engagement before and after
● A/B testing: use a formal A/B testing platform on the website
Some ways that HBR experiments

Change is as good as a rest: Sometimes a change tests well just because it is a
change and gets people’s attention. Change a button from red to blue, you may
get higher click throughs; effect diminishes over time, then six months later
change it back to red and get higher click throughs
Focus on the big picture: don’t look at a change in isolation; look at the total
impact. Adding a newsletter widget that gets clicks is good, but does it increase
the total newsletter signups (or just cannibalize the clicks you are getting from
other widgets)? Does the additional visual clutter lower overall engagement
(higher bounce, lower time on site)?
Follow some best practices

But remember we are not a lab
Having a culture of experimentation does not mean that your group transforms into
a medical lab where we need 98% certainty and huge sample sizes to make a
decision
Perfect is the enemy of the good: It’s better to have 20 good experiments than 3
perfect ones
Hurdle: is the experimental results better information that what you would have
used otherwise? (e.g. better than gut instinct?)

Resources
Google analytics experiments: FREE! (but anecdotally pretty hard to use)
Optimizely: easy interface, great training resources to help you get acquainted
with testing (we started here)
VWO: may be cheaper than Optimizely
Adobe Target: more robust, integrates with Adobe Analytics in a very powerful
way (we moved here in January)

Facebook Ads:
Audiences and Impact
Joseph Casciano, Harvard Public Affairs and Communications

How much to spend?
Why are we spending it?

● Low-value objective
● Larger audience
● High-value objective
● Smaller audience

Problems with Interest Targeting
● Often inaccurate, so serves
your ad to the wrong people.
● Hard to get a precise, smaller,
well-targeted audience.
● To big audiences, Facebook
serves your ad to whoever is
cheap and easy.

Business Manager
↓
Assets
↓
Audiences
↓
Create Audience
↓
Custom Audience

CPM (cost per 1,000 impressions)

Custom audiences make metrics matter
● Don’t provide contextless numbers about faceless masses.
● Tell concrete, true stories about your valued audience.
● So: “We reached half of our email subscribers on Facebook, half of who
watched the video for more than three seconds.”
● Not: “We reached 132,674 Malaysian bots who couldn’t even theoretically
fly into Cambridge for our symposium.”

● Use custom audiences
○ To guide your budget
○ To make metrics matter

Questions?
joseph_casciano@harvard.edu

Google Analytics Tips Tricks
Elizabeth Brady, EWB Analytics

Intro
Elizabeth Brady, Founder & Principal Web Analyst - EWB Analytics LLC - launched
March 2010
Specialties: Google Analytics and Google Tag Manager implementations, site audits,
web analytics support during site re-launch, measurement strategy and ongoing
analysis
Harvard groups I have collaborated with since 2012: Digital Communications, Harvard
Alumni, Harvard Admissions, Harvard Innovation Lab, Kennedy School, Harvard
Library, Harvard Learning Portal, HWPI, Ash Center
Contact: elizabeth@ewbanalytics.com

Filters Can Help Prevent
Internal Traffic
m 1
Maintain ‘exclude’ filter of
known internal IP
addresses
Ghost Spam
Maintain ‘include’ filter of
valid site hostnames
Crawler Spam
Maintain ‘exclude’ filter of
list of known spam
referrers

Filter: Exclude Internal Traffic by IP Address
Internal traffic inflates conversions & conversion rates. Check
current IP address by visiting whatismyip.com
IP Addresses Use regular expressions for a range of IP
addresses (ask IT for office IP range)
Dynamic IP Addresses
(residential)
Verify/update regularly
Test Activity Use custom dimensions to track test users
even when not on internal network

Filter: Test Accounts by Custom Dimension
● Use a custom dimension to
identify a test visitor who
visits a specific internal page
(webadmin, test, etc)
● Set the custom dimension at
the ‘user’ level
● Create a filter for any traffic
with that custom dimension
value

Ghost referrer spam:
● Never actually visits your site
● Sends data via the ‘measurement profile’
randomly to your GA account (became an
issue only with Universal Analytics)
● Sends data with a missing (not set) or
inaccurate hostname
● Can be prevented with a valid ‘hostname’
(include) filter
Prevent Ghost Referrer Spam

Language Spam - New Spam in 2016
Language Spam
● Rather than referrers, the spamming
sites inserted spam messages into the
‘language’ reports
● Most do not use valid hostnames so
this would also be prevented with a
‘hostname’ include filter
● Additional exclude filters can be added
to address language spam

Lowercase Filters
● Especially when starting a new
view, lowercase filters can avoid
capitalization inconsistencies
● Recommended for - page,
campaign
(medium/source/campaign),
search term (on site search)

Query String Cleanup
● Google Analytics includes any query string
parameters (after the ‘?’ as part of the URL)
● Leads to multiple versions of the same ‘page’
and a challenge aggregating data
● Parameters to exclude can be identified in a
list in the view settings, or you could ‘go
nuclear’ and exclude them all with this filter
on the right
● Full URL with query strings (or just query
strings) can be captured as a custom
dimension to be viewed when needed

Referrer Exclusion List (Property Level)
● Make sure your site subdomain/s is
included (new properties set up with
Universal Analytics will have this set
up on creation but any older site rolled
over to Universal Analytics did not
automatically have this configured)
● Include any off-site flows (login
validation, back-end sites like
pin1.harvard.edu) to prevent triggering
a new session
● Do not set up harvard.edu in the
exclusion list (that will prevent any
other Harvard sites showing up as
‘referrals’) - they will be ‘direct traffic’

Check ‘Exclude Bots & Spiders’ (View)
● Excludes traffic from sites on the IAB
(Interactive Advertising Bureau) list of
known bots & spiders
● Sometimes these can be contracted
services like site response time (like
Gomez) that execute javascript and
would otherwise show up in reports
● It is recommended to leave this
unchecked for the unfiltered view

Link Google Search Console (Property) for Organic Search
Trends
● Google Analytics no longer has much
insight into Google organic search
keywords
● Link your site’s Search Console
(formerly ‘Webmaster Tools’) account
for impressions/clicks on Google

Remember Campaign ‘Timeout’ (Property) is Configurable
● Standard campaign setting is 6 months
(sessions and conversions will be
credited to the last campaign in the
past 6 months)
● This explains why you may see
traffic/conversion for ‘old’ campaigns
● Your business group may decide you
need a shorter or longer campaign
timeout

Data Import (File Upload) Can Extend Analysis
● Data import lets you append
data to any dimension (standard
or custom) you collect
● The actual import can be a
simple text file upload
● Some uses might be to add
details around campaigns, add
authors or other details to
content pages, or group
information differently than
they way it is grouped in Google
Analytics

Custom Reports
Don’t dig for your data!
404s
Social Media Details
Top Pages by Type
Top Events by Type
Deep dive into a certain source of
traffic (ex: email campaigns)

Tracking 404’s
● No extra tagging needed
● Report Filter: Page title contains ‘Page Not Found’
● Dimensions: URL (page requested), Previous Page (might be entrance),
source/medium (more important for entrance pages to understand source of
traffic)

404 Error Report
● Monitor 404 volume over time
● Monitor broken links
● Set up 301 redirects where needed
● This report is very helpful after a site re-launch

Social Media Details
● Report Filter: Channel = Social
● Dimensions: Medium, Social Network (or Source)
● ‘Social’ = tagged social campaigns, ‘referral’ = organic social traffic (no camppaign
tags)

Pages by Type
● Report Filter: Page contains <URL identifier for type of content>
● Dimensions: Page
Possible content: blogs, story
pages, article pages, FAQ’s

Events by Type
● Report Filter: Event category = ______________
● Dimensions: Event label, event action
Possible events: document
downloads, offsite links, navigation
links, carousel clicks

‘Unique’ Metrics - Pageviews
● To report the number of sessions that
viewed a page, use ‘unique pageviews’
● GOTCHA - do NOT combine page
with sessions as a custom report (GA
WILL let you set this up, but sessions
are ONLY associated with the entry
page)

‘Unique’ Metrics - Events
● To report the number of sessions that recorded a certain event, use ‘unique
dimension combinations’
● This shows the sessions with the event for whatever dimension combination is
presented in the report

Custom Segments
● Create a custom segment to filter any report by sessions/users meeting specific
criteria
● Some common segments include:
○ Sessions from a specific campaign
○ Sessions that viewed a specific page
○ Sessions that registered a specific ‘event’
● Then apply a segment to a basic or custom report, for example:
○ Geographic reports
○ Traffic (source/medium) reporting
○ Technology: device/browser/OS reporting

Custom Segment - Viewed the homepage
●

Custom Channel Groupings
● CUSTOM channel groupings give you
the flexibility to roll up the data the way
you want to see it
● They are retroactive, but are specific to
the user account where they are created
but can be shared like other assets
● For the Gazette, we break out
Harvard.edu referrals, other Harvard
referrals, and non-Harvard referrals as
separate ‘Channels’

Custom Channel Groupings
● CUSTOM channel groupings are a
view-level setting - be sure to find the
‘custom’ groupings rather than the
‘channel groupings’ (changes to the core
channel groupings will not be retroactive
and will only impact data collection
moving forward)

Google Tag Assistant
Chrome Extension
● Quickly check the status of Google
Analytics and Tag Manager code on
any page
● Red/yellow warnings identify tagging
problems

EditThisCookie
Chrome Extension
● View cookies set on a site
● Delete selected, or all, cookies on the site
without having to clear all of your cookies
for other sites

Google Data Studio
Google’s new dashboard Tool
now offers free, unlimited
dashboards, with great
integration with Google
Analytics and other Google
products
Features: interactive filters,
flexible formatting, multi-page
dashboards
datastudio.google.com

Takeaways - Top Tip From Each Topic!
1. Filters - maintain data integrity by collecting the cleanest data you can in your
production a view (a non-filtered view should also exist), slides 4-12.
2. Settings - key settings to check include referral exclusions (include your own
harvard SUBdomain/s) and make sure bots/spiders are excluded.
3. Reporting - remember to use ‘unique’ metrics when reporting the number of
sessions with a specific page/event.
4. Tools - download ‘Google Tag Assistant’ for very user-friendly feedback on tag
set-up and data collection.
elizabeth@ewbanalytics.com - Feel free to reach out with specific questions!

Copyright © President & Fellows of Harvard College
Responding to Analytics with SEO
Marcus Dandurand - March 30th
, 2017

How to Measure Traffic from Search
Engines
In Google Analytics: Source/Medium = Google/Organic
In Adobe Analytics: Marketing Channel = Natural Search
2

Are we getting enough Search traffic?
3

Optimization is never done!
4
➢ Benchmark against yourself
➢ Compare traffic Year-Over-Year
➢ Try optimizing existing pages
➢ Create new pages to target new keywords
➢ Think beyond “branded” keywords

Search Traffic Year-Over-Year
5

Wait… What?!
Traffic is down!!!
Is it something we did?
Did Google change its algorithm?
Will it fix itself?
What can we do?
6

First, a few SEO myths...
7
1. We don’t know what Google wants
2. The algorithm changes too often
3. SEO is an attempt to “game the system”
4. SEO is a job for IT
5. My CMS has SEO built-in

What do search algorithms care about?
8
Relevance
Performance
Authority

What is page “Relevance”?
Your page content closely matches a keyword search phrase.
9

A few ways to improve a page’s relevance:
10
➢ Keyword Research - Relabel content using “outside voice”
➢ Breakup Content - Each important idea should have its own landing page
➢ Accurately describe all page components
∙ Page Titles/Meta data
∙ Navigation links
∙ Section Headers
∙ Etc.

What is page “Performance”?
Pages are useful,
Pages load fast,
Site is accessible to humans & robots.
11

A few ways to improve a page’s performance:
12
➢ Reduce page load time
➢ Make sites mobile friendly
➢ Improve click-through rate in search engine results
➢ Make sure websites can be crawled/indexed properly by search engines

What is page “Authority”?
Every link acts as an endorsement of a page’s credibility.
Both External and Internal links!
13

Authority began as Google “PageRank”
14

A few ways to improve a page’s authority:
15
➢ Create resources that people will share (inbound linking)
➢ Use 301 redirects (site cleanup)
➢ Restructure website navigation to distribute authority to your
most important pages
Distributing Authority:
Are the important links on your page 1/10 or 1/100?

Working Knowledge Website Updates:
SEO lessons learned the hard way.
16

Working Knowledge has a search traffic problem!
18

Working Knowledge search traffic improves!
19

Working Knowledge search traffic improves!
20
➢ Sept – Nov 2015: down 43% vs. Prior Year
➢ Sept – Nov 2016: up 46% vs. Prior Year

1) New dynamic landing pages
22
➢ Hundreds of Topic landing pages
were not indexed by Google. New
browse page was behaving like a
single dynamic page. (Performance)
➢ Each page had the same Title & Meta
Description (Relevance)

2) Deleted the Working Knowledge Archive
23
➢ Several articles were still very
popular for search traffic.
(Relevance)
➢ Many pages were still cited and
linked to by important sources.
(Authority)
Without proper redirects, the
authority passed back to the home
page was lost.

Other optimization for Working Knowledge
24
➢ Created customized Titles & Descriptions for each page
in the CMS. (Relevance)

Other optimization for Working Knowledge
25
➢ Created new “display descriptions” visible on-page.
(Relevance)

Are You Making A Major Website Update?
Please consider the following...
26

#1) Navigation Links Transfer Page Authority
27
➢ Primary navigation create backlinks from every page on your site.
➢ Try to put your important pages in your primary navigation (but only
if it makes sense).
➢ Try to remove links that are nice to have, but not critical.
➢ Adding new links will dilute the authority of existing links.

#2) All URL Changes Need 301 Redirects
28
➢ 404 errors create a bad user experience and waste page authority.
➢ 302 redirects are “temporary,” so Google keeps the old page in the
index. No authority is passed!
➢ 301 redirects are “permanent,” so the authority of old pages are
passed.

#3) Avoid Duplicate Content
29
➢ Each page should have a unique Title & Description.
➢ You should not be able to see the same page via two different URLs.

Pop Quiz:
30
Which of the following URLs below are exactly the same as:
www.hbs.edu/mba
A. www.mba.hbs.edu
B. hbs.edu/mba
C. http://www.hbs.edu/mba
D. https://www.hbs.edu/mba
E. All of the above

Free Tools for SEO Analytics
31

Free Tools for SEO Analytics
32
➢ MozBar (Browser Extension)
● Page Authority, Inbound links
➢ SiteImprove - Free through HUIT!
● Missing Page Titles, Descriptions, Broken Links, 302 redirects
➢ Google Search Console - Formerly “Google Webmaster”
● 404 errors, Organic Keywords, XML sitemap, Mobile issues
➢ Link Redirect Trace (Browser Extension)
● Follow the path of multiple redirects

From Big Data to Insights in Massive Open Online Courses
A Traveler’s Guide
Daniel Seaton
Harvard University
Sr. Research Scientist
VPAL Research Team

daniel_seaton@harvard.eduAnalytics Academy - March, 2017
• Consortium of Institutions
creating MOOCs
• Maintain Open-Source Platform
• Host Courses/Content
• Lead Outreach
• Maintain “https://www.edx.org”
• Partners from Higher Ed /
Industry / Government / High
Schools
• Create Courses/Content
• Manage Courses in Open Online
(MOOC) and On-Campus
(residential) settings.
• Perform Research into Teaching
and Learning
Consortium Members
What is edX?

Data from Dec.
2016
http://harvardx.harvard.edu/
Harvard University’s MOOC Organization:
• Partners with faculty to create open online courses
• Supports initiatives to use MOOC content beyond open
online models

Data from Dec.
2016
Harvard University’s MOOC Organization:
• Partners with faculty to create open online courses
• Supports initiatives to use MOOC content beyond open
online models
http://harvardx.harvard.edu/

Research Timeline and Perspective
2012
What are learners doing?
Who and where are our learners?
Besides open online, how
else can we use MOOC
platforms and content?
Why are learners taking courses?
2013 2014 2015 2016
Single MOOC
Transforming Advanced
Placement High School
Classrooms Through
Teacher-Led MOOC Models
Seaton, Hansen, Goff, Houck, Sellers
Many MOOCs
Context around
MOOC
enrollments
Alternative MOOC
Models

Single MOOC

• 6.002x: Circuits and Electronics (first MOOC
from MITx - now edX)
• Over 100K enrollees
• Over 7K certified users
• Over 100GB of data from clickstream
• Limited profile information
• MITx now a member of edX: ~ 100 open
access courses

We started with “what” people are doing in 6.002x
Transition between resources
Nodes are resources
(size ~ time spent)
Edges are transitions
(size ~ weight)
Who does what in a Massive Open Online Course?
Seaton, Bergner, Mitros, Chuang, Pritchard ( Comm. of the ACM - 2014)
Analyzed learner interactions with
all aspects of 6.002x. Particular
focus on time-on-task and
resource-use during problem
solving.
Measurements
• Time-on-Task
• Resource Interactions
• Daily/Weekly Progress
• Transitions between resources
during problem solving

Many MOOCs
+
Context Around Enrollments

• HarvardX and MITx Working Paper #1
• Now had access to all course data
from MITx and HarvardX
• Addressed “what” people were doing,
and “who” they are, across 17 MITx
and HarvardX courses
Key Takeaways:
1. Courses are very different.
2. Registrant diversity is immense
compared to residential.
3. Participation greatly varies.
Ho, et al. (2014). HarvardX and MITx: The first year of open online courses
(HarvardX and MITx Working Paper No. 1).

What are learners doing across MITx and HarvardX?
%Grade
% Chapters Accessed0
100
100

What are learners doing across MITx and HarvardX?

Cross course surveys launched in 2014
addressing broad issues across MITx, but
teaching experience was central issue.
Results from 11 spring 2014 MITx MOOCs:
• 28.0% (9451) self-identify as past or
present teachers (navy).
• 8.7% (2847) current teachers (orange).
• 5.9% (1871) teach/taught the topic (gray).
On average across courses, ~ 8% (1 in 12) of
comments are from current teachers.
For teachers that teach/taught the topic, the
average across courses is ~6% (1 in 16).
Percent
Comments
in Forum
Did not
take
survey
Surveyed
Surveyed
Teachers
Non-Teachers
43.8%
22.4%
33.8%

Why are we still not talking about course structure/design?
+
Visualizing Course Design

Goals and Motivations
• Support HarvardX by collecting relevant stats on course structure.
• From a research perspective, identify canonical patterns in course development and
better understand how those patterns affect behavior and outcomes.
Human beings, viewed as behaving systems, are quite simple. The
apparent complexity of our behavior over time is largely a reflection
of the complexity of the environment in which we find ourselves.
- Herbert Simon, “The Science of the Artificial”
Practical
Motivation
Abstract
Motivation
http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research

Visualizing Course Design
Key point: Use these
visualizations to look
across courses.

http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research

Alternative MOOC Models
AP High School Content/Courses

• Boston Public School students
• Take MOOC online during school
• Commute to BU weekly for
labs/recitations with TAs/Faculty
Of 34 regular and charter schools serving 16,165 students, 2 high schools offer algebra based AP® Physics 1.
Only 60 BPS students took the AP® Physics 1 exam during the 2014-2015 school year.
BU Project Accelerate
• Open-online and teacher-led/flipped
• All content open on edx.org
• Special instances for teachers to
use content in classrooms
• Showed 0.08 added to AP exam score per
hour usage above class average
http://vpal.harvard.edu/blog/complementary-models-mooc-instruction-advanced-placement%C2%AE-high-school-courses

Davidson Next - AP content for use by teachers and
students
Program at Davidson College:
• Supplemental content for 14 Challenging Concepts in each AP subject.
• Challenging concepts determined using College Board exam data from 2011 to
2013. Piloted with Charlotte-Mecklenburg School System in 2014-2015 school
year.
• Modules designed for each concept meant to facilitate use both in classrooms,
and open online. Real AP Teachers from developed content with Davidson
faculty.
• Courses released on edX.org and through a new Custom Course tool (CCX).

Bubble Charts for Detecting Daily Activity
• Made these to help monitor teacher use of Davidson Next in Charlotte High
Schools
• Full time assessment coordinator worked with teachers on implementation and
efficacy of content (collected district data and AP exam scores).
Transforming Advanced Placement High School
Classrooms Through Teacher-Led MOOC Models
Seaton, Hansen, Goff, Houck, Sellers (MIT LINC Conference - May 2016)
Pilot program in
North Carolina High Schools
Massive Open Online Courses via edX.org
Exam score residuals
are then correlated
with student usage
relative to class
median indicating
0.08 points per hour
spent (p<0.05).

Building Community Around Analytics…

Open Source Tools for edX Data
• Harvard and MIT already share resources and code for analytics
• https://github.com/mitodl/edx2bigquery
• https://github.com/mitodl/xanalytics
• Open-Source Repos
• Python + Google
BigQuery for
aggregation of edX data.
• Dashboard via Google
App Engine

edX Data Workshop Summer 2016
Meeting of data analysts and engineers
in institutional roles responsible for edX
data; 16 attendees from 11 institutions.
Goals for meeting:
• Discuss broader aspects of data
sharing and analytics.
• Standup the Harvard/MIT edX Data
Pipeline.
• Happy to report that each
participant completed this task.
• Next workshop summer 2017?
• Hoping to broadly release workshop
documentation in the spring.
http://news.harvard.edu/gazette/story/2016/07/moocs-ahead/

Visit our blog: http://vpal.harvard.edu/blog

Many collaborators to thank before dicussion!
HarvardX
Andrew Ho, Dan Levy, Jim Waldo,
John Hansen, Sergiy Nesterko,
Justin Reich, Tommy Mullaney
Miki Goyal, Gabe Mulley,
Carlos Rocha, Victor Schnayder,
Olga Stroilova, Brian Wilson
Julie Goff, Aaron Houck,
Kristen Eshleman, Pat Sellers,
Noelle Smith
Yoav Bergner, Cody Coleman,
Isaac Chuang, Curtis Northcutt,
David Pritchard, Saif Rayyan
VPAL Research Team
Andrew Ang, Glenn Lopez, Brooke Pulitzer,
Yigal Rosen, Dustin Tingley, Selen Turkay,
Jacob Whitehill, Joseph Williams
Lydia Snover, Jon Daries, Mark Hansen
Research
and
Analytics

Ditch the spreadsheet
and tell the story
Katie Hammer
Office for Sustainability and Harvard Public Affairs and Communications

How do you get your
team to understand
your analytics story?

MATERIALS AND CONTENT CREATED:
GHG Landing page (OFS)
4 Page Climate Report PDF
Community-wide message from
President Faust
Wide-format Gazette Story and
Graphics
8 #HarvardClimateStories
instagram profiles
Video targeted at social media
Custom social graphics for 12
Schools + departments
harvard.edu/climate modules

Community-wide email sent by
President Faust
52% open rate
867 clicks
Gazette Story in the Daily Gazette 22% open rate
637 clicks
Social Promotion (Harvard & OFS)
● Video with paid boost
● Twitter & Twitter Moment
● #HarvardClimateStories
Instagram Campaign
● 108,000+ video views
● Almost all Schools shared
news w/ graphics
● 17,694 likes; 84 comments
Inclusion in OFS December email
newsletter
25% open rate
507 clicks
Feature on Harvard.edu 3,010 clicks
External Press (Crimson, Harvard
Magazine Story,NY Times, Boston
Globe, WGBH etc.)
DISTRIBUTION EFFORTS:

OFS Goal Page: 335
Gazette Story: 210
HUCE Site: 301
Clicks

FACEBOOK AD CAMPAIGN: OFS
OFS Ad Spend: $170
Duration: December 8 - 12
Audience: Targeted students and alumni (where
Harvard was listed as School and age was 18+); OFS
email list; People who liked our Facebook page
Reach (number of people that saw the post):
● 42,019 total people reached
● 16,624 people reached as a result of paid
● 17,000 total video views;
● Cost per 1,000 people reached $10.23
● $.05 per 10 second video view
Engagement (reactions, comments, shares):
● 7,696 total actions
● $0.27 per engagement
● 23 link clicks
● Post generated 55 new GreenHarvard
Facebook page likes
Context:
● Typical GreenHarvard video is viewed ~500
times
Note: Many comments did include mention of divestment;
however about half were positive and congratulatory.

TWITTER STRATEGY:
● Worked with Facility teams to create custom graphics optimized for
social for Schools to use
● Partnered w/ HPAC to share on the @Harvard accounts
● Outreach in advance to all digital counterparts at Schools/Depts
● Created a Twitter Moment to capture various influencer
and School tweets about announcement
RESULTS:
● Initial tweet: Retweeted 66 times; Liked 102 times, Clicked 34 times
● Moment tweet: Retweeted 33 times, Liked 92 times, Clicked 90 times
● Retweets and original tweets from internal “influencers” like HBS,
HSPH, HAA
● Almost all 12 Schools promoted us in some way, in addition to the
Museums, Libraries, and various departments

And they all lived
happily ever after...

Lessons learned and opportunities
● Targeted outreach to Schools and Depts works; Schools/Depts find easier to promote when can link
data/anecdotes back to them (social graphics received well).
● While School/Dept outreach worked and we did have some external influencer tweets (Climate
Registry, USGBC), we should develop a more solid plan for faculty and social influencers in the
future.
● #HarvardClimateStories campaign a success; 8 profiles in a month was ambitious; for future
campaigns could start earlier and conduct interviews and shoots further in advance.
● Important to consider different outreach methods for different audiences; for example the social
video was extremely brief but gave an external audience the message they needed “Harvard set an
ambitious goal and they met it.”
● We should consider allocating time more evenly across a wide range of projects, considering goals,
audiences, and reach (PDFs, videos, social campaigns). For example, though PDF a considerable
amount of our time, the social/web reach was minimal.

Keys to a telling a good story
Format
Choose a vehicle that’s
relatable (even if that
means powerpoint).
Keep it simple.
Style
Use language that
seems right for your
story (and for your
client).
Setting
Set your story by
bringing in context to
explain the why.
Remember you control
this!
Themes
Let the themes of your
data shine by weaving
them throughout your
story.
Illustrations
Images, examples, and
visual cues only add to
your story.
Conclusion
End your story with
lessons learned and
opportunities that leave
the reader ready for
your next story!

Thanks!
Any questions ?
◉ kate_hammer@harvard.edu

Strategize, Synthesize, and
JAZZERCIZE®
your Analytics Dashboards and Reports
with your host Aaron David Baker

Remember Jazzercize®
?
Problem:
Non-dancers are taking Jazz Dance
classes because it is a great workout
but aren’t interested in all the work
on form and technique. They just wanna
have fun while exercising.
Solution:
Create a fun jazz dance-style fitness
class that’s interesting and fun!

The problem was neither exercise nor jazz dance class
Exercise is a chore; we make it fun to get it done.

It’s the same with creating/consuming analytics data

Give them what they want!
The people who consume your
reports want them to be
engaging.
They don’t always have to
contain charts and graphs.
Good document design
also conveys professionalism.
Don’t just report on the
content you produce or see,
share screenshot highlights
to give context.

Great reports are persuasive and can change attitudes

Introducing Scoop
HPAC’s analytics dashboard,
built around public APIs to
Google Analytics, Facebook,
Silverpop, and more to come
Presents consistent, up-to-date
performance data per story,
post, and mailing
Makes comparative metrics
possible with benchmarking and
visualizations

Introducing Scoop
We have a Strategy
● Identify and capture metrics
that matter in one convenient
place
We have Synthesis
● Stats from many different
platforms in one place for
easy reporting
We are working on that Jazz
● Make it beautiful and fun

The official Harvard Style
Guidelines & Best Practices site
has an updated Analytics with
resources and setup information.
harvard.edu/guidelines
Use these best practices to ensure
your site is up-to-date with the
latest analytics code and tracking
practices.
First, check that everything is in order

Second, define key stats (these are just some examples)
Health stats are these
● Users/Sessions/Pageviews
● % New Users over time
● Page speed
● 404s
Strategic metrics are these
● Content performance (pageviews)
● Content engagement
○ Time on page
○ Scroll depth
● Users by
○ New/returning
○ Geolocation
○ Content sections they visit
○ Frequency of visits
● Content dimensions
○ Content category/section/tag
○ Content length
● Acquisition paths
○ Search keywords

Choose goals and
metrics that make
sense for your
organization!

Report on what’s exceptional and on what’s important
What’s exceptional
● Top content in terms of
pageviews and/or engagement
metrics (time and scroll)
● Large number of people reached
or high number of impressions
● Social activity (likes,
comments, shares, retweets)
● Unusual spikes in traffic*
or
unknown sources
What’s important
● Key initiatives
○ President Faust’s priorities
○ Special Gazette features
● Things you spent money on
○ Paid social, AdWords, etc.
○ Extra money gives you extra
metrics
● Experiments
○ A/B testing
○ SEO
● Conversions

Here we’ve chosen to highlight
pageviews by channel and
accumulated pageviews over time
● Helps us visualize content
distribution and sources of
traffic.
● Benchmarking is our own
comparative metric—average
daily pageviews of all
stories in Scoop.
Scoop Example

The chore of reporting
● Go here
● Find stats
● Copy/paste into doc
● Go there
● Find stats
● Copy/paste into doc
● …

Wouldn’t it be nice if
the stats gathered
themselves?

Scoop pulls in stats from
● Google Analytics
● Facebook
● Silverpop
With plans to incorporate
● Twitter
● Instagram
● YouTube
● Etc.
Anything with an API can be
consumed.
Scoop Example

Synthesis isn’t just about automation
Automation is nice and does
save a lot of time.
Linking things by a common
element (like URL or topic)
can make finding the stats
easier.
Synthesis is really about
telling the whole story.
Automated reporting cannot
speak for you.
Talk about the why in your
reports.

My Weekly Report
Ultimate synthesis of
what happened last week
Mostly highlighting
what’s exceptional
Occasionally mentioning
what is bizarre
Always as interactive as
possible
● links go to actual
online posts or to
Scoop itself

Back to Jazzercize®
Our reports are working,
and people come to us
for information, but how
can we analytics reports
more enjoyable?

Design and data visualizations:
● Beautiful, clean, contemporary,
and inviting design
● Display visually our data’s
trends, patterns, and
correlations
● Provide content creators and
distributors with intuitive,
at-a-glance insights about
performance of published work
● Be designed with interactive
development in mind
The plan for Scoop

Your turn to talk:
How do you add jazz
to your reports?

Analytics Academy 2017 Presentation Slides

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (15)

Semelhante a Analytics Academy 2017 Presentation Slides

Semelhante a Analytics Academy 2017 Presentation Slides (20)

Mais de HarvardComms

Mais de HarvardComms (18)

Último

Último (20)

Analytics Academy 2017 Presentation Slides