All Possible Worlds iab mx_public

All Possible Worlds:
Advances in Digital Measurement

P.A Pellegrini, Ph.D. VP International Research
comScore, Inc.

Agenda

• Strategy for Digital Audience Measurement

• Current Methodology and Enhancements
• Impact of Mobile Devices and Tablets
• Fragmentation of Platforms and Consumption
• Towards a Digital Measurement Ecosystem
• User Panels, Site-Centric Tag Measurement, Network-Centric Logs,
Dynamic Empanelment

• Establishing Next Generation Best Practices

• Probabilistic Reference Point and Validation

© comScore, Inc. Proprietary. 2

comScore Digital Measurement Strategy


The Challenge: Devices to be tracked


The Challenge: Formats to be tracked


The Challenge: Data sources to integrate


The Challenge: Processes to be linked


The scope of the challenge for Digital Media Measurement


Digital Requirements

 REQUIRE Depth & Representivity
– Endless scope of possible behaviors
– Engagement profiles dominated by heaviest users
– Must capture micro audiences
 REQUIRE Integration within broad Media Research environment
– Media ecosystem is bigger than just Digital
– Methods & Practices must synch with those used in other media
 REQUIRE full view into the digital consumer across all platforms
– Web-only view no longer represents the digital consumer
 REQUIRE ability to integrate methodologies
– Methods that work on one platform may fail on another
– Must integrate ‘best-of-breed’ methodologies for complete measurement


Key UDM Lessons

 Panel Requirements Staggeringly high
– Key segments difficult to represent: Work, Tech Professionals
– Systemic under-measurement of certain behaviors: news, developer networks
– .. efforts to date to build a “perfect panel” have fallen short

 Publishers willing to provide Site-centric measurement
– Better measurement is valued. Outweighs cost of tagging burden.
– Ability to ‘close-the-loop’ to reconcile sources reduces transaction costs

 Some Ecosystem elements best measured by tag-based collection
– Ad networks, Video & Mobile
– Tags allow for rich event attribution (Ad Net reach, YouTube, Hulu)
– Coverage across all platforms

 UDM must evolve in response to feedback
– Simple assumptions must yield to more robust methods (cookie/person est.)
– UDM highlights key gaps in panel methodology (large deltas, apples/oranges)
– Must integrate Site-centric measurements into panel methodology


Digital Measurement Outlook

 Drive next-generation Panel Methodology
– Leverage Site-census measurements
 Integrate Digital Measurement into broader media ecosystem
– Adopt industry standards: enumeration, demography, sampling
– Deliver transparency through public audit reporting
– Integrate digital into Agency/Planning infrastructure & 3rd party tools
 Move Digital beyond the Web
– Mobile matters. So do Tablets… whether we can build panels or not
– It is our responsibility to deliver measurement
 Develop Cross-Platform integration techniques
– Anchor methods to site-centric observations across platforms
– Accommodate platform-specific methods to deliver a Unified view


Current Methodology and Enhancements


Building Blocks of Unified Digital Measurement
Establishment Surveys
Population Targets

Enumeration

Unified Digital Measurement Online Recruitment
UVs and Page Views Panel
Unification
Recruitment

comScore
Unified Digital
Measurement

Passive Meter Data Collection
Traffic Data Session Assignment
Dictionary
Site Hierarchy Allocation Collection
Site Centric tagging

Panel Bias
Correction

Weighting, Preventing Bias,


Enumeration

Enumeration

Panel
Unification
Recruitment

comScore
Unified Digital
Measurement

Traffic Data
Allocation Collection

Bias
Correction


Defining the Universe: The Establishment Survey

 Accurate estimates of the size and composition of the target populations
are critical to audience measurement.
 High quality establishment survey enables comScore to accurately
estimate the target population sizes for in-market age and gender
demographics, penetration, and frequency of use in each market.
 comScore currently uses proprietary surveys, high quality government
sources (eg, Eurostat), or the local ‘gold standard’ survey.
 For expanded demographics and to avoid mobile-only issues, comScore
has agreed to partner or license gold standard surveys like the NRS in the
UK, PMB Print Measurement Bureau in Canada, The Readership Works
by Ipsos in AU, and The EGM survey in Spain.

comScore currently measures & reports persons age 6+ who accessed the
Internet from either a home or a work computer in the past 30 days.


Panel Recruitment

Enumeration

Panel
Unification
Recruitment

comScore
Unified Digital
Measurement

Traffic Data

Bias
Correction


Recruitment in the Digital Age

 Given the issues of:
– Scalability (the need for panel sizes in the hundreds of thousands to address the long
tail of digital media and fragmentation)
– Declining efficacy of RDD recruiting

 comScore recruits panelists online, in two ways:
– The Affiliate Program (“banners-plus-Permission Research”)
– Third Party Application Providers (TAP)

 Pioneers of large, non-probability panels
– Panel Dashboard, Jackknife Replication


Online Recruitment from Two Approaches…

Affiliate Program Third-Party Application Provider (TAP)

 Affiliate network comprised of web  comScore partners with application
entities which meet comScore's providers who offer visitors a “quid
quality criteria pro quo”
 On these sites, panelists are  The web user is offered something
recruited via banner ads free (software, applications, utilities
etc.) in exchange for exposure to
 Appeals are targeted through a
our panel solicitation
broad array of smaller web entities
 The user can generally get the free
 Respondents are directed to our
app without joining the panel
online intake entity
www.permissionresearch.com  New “Value Proposition” (in Affiliate
and TAP): Trees for Knowledge


Permission Research Panel Recruitment – Creative Examples


TAP Recruiting Experience

 User is on the Internet
searching for software
applications, or some
other download
 This user is looking for
“photo animation
software”


TAP

 User gets desired search result and clicks through to site


TAP – Partner Site

 Once on the site, user
decides to initiate the free
download


TAP – Installation Flow

 Begin Installation process


TAP – Installation Flow

 Accept the License
Agreement of the value
proposition software (this
is the partner’s
agreement)


TAP – User Acceptance

 User is also asked to join
the comScore panel.

 This is a separate step
from the partner’s
software acceptance

 User must take a positive
action (not pre-selected)


TAP – Partner-side Demo Collection

 Take a short survey where
we collect basic demos


Value Propositions from TAP Recruitment

A sampling of the many value propositions offered….
PC Utilities and Productivity Desktop Personalization and
Digital Media Applications Games and Entertainment
Tools Appearance
•Alarm Clock •Screensavers •Audio/Video Converters •Arcade Games
•Application Launcher •Icons •Audio Editor •Board Games
•Checklist Software •Images •Audio Extractor •Fantasy Games
•Download Accelerator •Smileys •CD and DVD Burners & •Global Radio & TV Stations
•Download Manager •Emoticons Rippers •Lyric finder
•Media Streamer •Video Converter for •Media Players •Music Downloads
•PC Access Control Screensavers •Media Downloaders •Photo Album
Software •Icon File converter •Media Search Tools •Photo Morphing Software
•PC Customizable Shutdown •Wallpapers / Themes •Media Library/Organizer •Quest Games
•PC DVR •Screen Pen •Ringtones
•TV Optimization Software •Sports Games
•Windows Application/PC •Strategy Games
Lock •Online Games
•File Protection •War Games
•PC Power Saver
•Online Disk Storage
•History Cleaner
•Internet Usage and
Performance Stats
•Unit Converter/Metric
Converter
•Currency
Calculator/Converter
•Customizable Binary
Converter


Trees for Knowledge

 In 2008 comScore launched a new recruitment appeal, for both
Permission Research and TAP panelists
– Under our Trees for Knowledge program, comScore has partnered with
Trees for the Future to plant a tree for new members that join the panel
– comScore’s initial donation will support the planting of one million trees in
developing communities throughout the world
 Creative made available to affiliates to promote campaign

 Permission Research offer: plant a tree at installation; then one tree for
each month the panelist stays in panel

28

Passive Data Collection

Enumeration

Panel
Unification
Recruitment

comScore
Unified Digital
Measurement

Traffic Data

Bias
Correction


comScore’s Privacy Protected Proprietary Collection Software with
Explicit Panelist Permission

comScore’s cProxy meter allows us to “see” user activity at the device or screen
side (user experience as opposed to site-centric) without using cookies

■ comScore captures:
– URL
– Engagement (active versus passive)
– Keystroke and mouse activity and intensity
– Information passing between the user and the Internet entity
– Ads delivered, whether clicked or not
– Application installation and usage (Excel, Word etc.)

■ Data capture tiers
– HTTP: comScore can capture just about anything over the HTTP protocol (including HTTP and
HTTPS)
– Proprietary technology for real time measurement of advanced protocols like streaming
– Measurement of AOL proprietary and IM engagement


Session Assignment Technology (SAT)


Who is Using the Device?

 In the digital age, tracking the behavior of devices is easier
than ever before.
 Moving from devices (which don’t buy products or view
advertisements) to people still poses unique challenges.
 In panel measurement, the ideal is passive observation over
an extended time period.
 One approach: use “who are you?” pop-ups.
– Requiring respondents to self-identify each user session
(similar to traditional media construct used in TV People Meter)
– Intrusive
– Fatiguing
– Biasing

 Intrusiveness drives panel turnover to unmanageable levels.


Who is using the device? – Passive Biometric Observation

 comScore develops a unique biometric signature for each panelist.
– Session Assignment Technology (SAT): a proprietary, patented technology
that creates biometric “fingerprints” for each person using the device.

Family Roster: Mouse Activity:
Email, Age, Sex… Double Click Speed,
Unique Profile of Each
Mouse Movement
User in a Household
Sharing a Computer

Online Accounts: Keyboard Activity:
Cadence tracking of
100+ keywords


Bias Correction

Enumeration

Panel
Unification
Recruitment

comScore
Unified Digital
Measurement

Traffic Data

Bias
Correction


Preventing Panel Recruitment Bias

 At the recruitment stage, this is handled by 3 strategies:
– Hundreds of diverse incentives
– Over 40 different partners, no access panelists or ‘take a survey’ partners
– Multiple advertising links to partners provide huge reach over each campaign
 Provides extensive and broad reach across online universe to minimize
potential recruitment bias.
 Additionally, both demographic AND behavioural weighting (based on
census information); the latter removes recruitment bias.

Ultimately, the controls in UDM are census based rather than sample based,
and therefore they overcome any recruitment bias and sampling error of
either a calibration panel or probability sample.


The Recruitment Dashboard: Online Recruitment Balancing

 Recruitment targeting allows more control over marginal recruitment via
development of partner dashboard.
 The dashboard uses reports to track partner yield by demographic group,
over time
Current Month Index Last Month Index 2 Months Ago Index Partner1
Target In Tab vs UE In Tab vs UE In Tab vs UE Target In Tab % UE
All_2-5 2.3 2.8 2.7 All_2-5 50 0.8% 2.0%
All_6-11 4.4 5.3 4.9 All_6-11 50 0.8% 8.1%
All_12-14 0 0.0% 5.7%
All_12-14 15.1 43.9 42.7
All_15-17 0 0.0% 6.0%
All_15-17 114.7 154.4 151.6 All_18-24 1,600 26.7% 13.9%
All_18-24 203.4 193.4 188.0 All_25-34 2,000 33.3% 18.0%
All_25-34 122.4 118.0 117.7 All_35-44 1,800 30.0% 18.1%
All_35-44 91.1 92.0 91.6 All_45-49 500 8.3% 8.8%
All_50-54 0 0.0% 6.7%
All_45-49 81.5 79.7 80.8
All_55-64 0 0.0% 7.6%
All_50-54 89.4 78.5 82.3 All_65+ 0 0.0% 5.0%
All_55-64 96.3 82.5 84.4
Total 6,000 100.0% 100.0%
All_65+ 89.8 83.0 90.5


Traffic Allocation

Enumeration

Panel
Unification
Recruitment

comScore
Unified Digital
Measurement

Traffic Data

Bias
Correction


Web visitation and traffic allocated based on 6 levels of site hierarchy


Client Focus Dictionary (CFD) – Media Entities

 Traffic allocated across
– 118 content categories and sub-categories
– Six levels of hierarchy supporting a “parent/child”
relationship with intelligent grouping of properties
– Ad Networks providing potential and actual reach
– All platforms including internet and mobile

 Building block for each level in the hierarchy is a
URL pattern
– Example: %yahoo.com%sports%cricket%

 Allows calculation of key inputs to Unified
Digital Measurement at as granular a level as a
URL pattern resulting in
– Accurate estimates of crucial metrics like cookie
deletion, audience overlap etc
– Bias elimination created by grouping different
types of sites into one broad bucket


Unification

Enumeration

Panel
Unification
Recruitment

comScore
Unified Digital
Measurement

Traffic Data

Bias
Correction


Fundamental Problem of Site Centric Measurement:
No Unique User ID

Web Analytics Approximation
Unique User = Cookie ID (if Cookies can be set) or
IP Address + User Agent

More Problems…

• Cookies are deleted, and when they are, the same user can be counted multiple
times

• IP Addresses change all the time causing inflation of user counts

• In any case, a UU is an attempt to identify a machine (or a browser), which can
represent multiple people or a fraction of the usage of a single person

• Some 56% of all machines had a session on more than one browser, and since
each browser sets its own cookie on a machine, they can’t de-dup.


The Impact of Cookie Deletion:
Repeat Visitors are counted Multiple Times

The site reads THREE distinct cookies, which means
this ONE visitor is counted THREE times.


Unified Unique Visitors

 UDM combines cookies at census level with panel insights to report unique
visitors for the combined home and work audience.
 Methodology:
– Collect 3rd party cookies
– Filter cookie data to include:
User generated calls by excluding bot and spider traffic (IAB lists)
Country specific traffic by excluding traffic outside reportable market
Home, work, mobile and commercial shared use audience
– Use panel to understand cookies per person (explained in the next slide)
Based on entity-specific user behavior
– Some visitors to a entity may not receive a comScore cookie due to non-
tagged portion of the entity
comScore accounts for these visitors using the panel.
– Visitors are then de-duplicated across home and work for both unique visitors
with and without cookies.


Cookies per Person

 Advertisers demand people measurement, not cookies or browsers
 Cookies are not people due to duplication at the cookie level caused by
several factors including:
– Users deleting or blocking cookies
– Multiple browsers per device This is what differentiates UDM
from other ‘hybrid’ methodologies;
– Multiple devices per location consistent with Global Industry
– Multiple users on the same device recommendations

 comScore calculates cookies per person each month at the site level by
country using panel observations
– This factor is applied to cookies observed for the entity to remove duplication and
derive Unique Persons
 Cookies per person may be seen as a function of:
– Site usage (see above)
– Visitation frequency
– Usage intensity

Total Universe: Unique Visitor Calculation


UDM Traffic collection

We collect traffic (tag counts and cookies) from various
locations, devices and content types that are tagged

• Home
• Work
• Libraries
Locations • Universities
• Internet Cafes
• Other

• PC
• Mac
Devices • Mobile phones
• Tablets
• Other

• PC based web
Content • Mobile web
Types • Apps
• Other


Total Universe Page Views

Segment Traffic by Total Universe
Location / Device Page Views

Mobile Phones

Shared
Machines

Work

Home

Filtered
Census Traffic


Traffic Allocation
Segment Traffic by Total Universe
Location / Device Unique Visitors

Home Work

Home +
Work

Mobile Other

Multiple
Shared
Filtered Web Apps Schools use
machines
Census Traffic
Libraries
& Other


Impact of Mobile Devices and Tablets


Device Fragmentation beyond the PC is starting to
define Digital Media


Device Fragmentation

Connected Devices per HH
60%
% of Households 50%

40%
PC Devices
30%
All Devices
20%

10%

0%
1 2 3 4+
# of Devices in HH

Mobile Devices & Tablets are Driving trend in Multi-device Access


Device Fragmentation

More than 50% Connected Devices per HH
of Internet 60%
Households still
have a Single % of Households 50%
PC
40%
PC Devices
30%
.. But Nearly All Devices
40% of 20%
Internet
Households 10%
have more
than 3 0%
Connected 1 2 3 4+
Devices # of Devices in HH

Mobile Devices & Tablets are Driving trend in Multi-device Access


Behaviors are different across devices.

PC behaviors do not represent activities across all
devices


iPhone & iPad User Incremental Usage vs. PC

iPhone & iPad Users: The Platform is
Incremental Reach over PC
the Message:
60.0% 56.8%

50.0%
40.0% Behaviours vary
28.9%
30.0% 26.8%
23.6% 21.7% by platform
20.0% 12.5%
8.7%
10.0% 6.9%
0.2%
0.0%

Information

e-mail
Maps

Business/

Networking
Total Internet

Retail
Sports

Navigation
Finance

Search/
News/

Social
Incremental
2.0x 9.2x 1.6x 1.2x 1.9x 1.3x 2.8x 2.5x 1.4x
Duration

Source: comScore custom research; experimental iPhone/PC Overlap panel


Trend toward a mutli-platform digital world is just
beginning


Platform Explosion is just getting started

40%
US Smartphone Penetration: 33.5%
35%

30%
Smartphone Population
25%
Growing Strong
20%
50%+ Annual Growth
15%

10%
US iPad Penetration: 4.4%
5%

0% Nov-2010

Feb-2011
Feb-2010
Mar-2010

Mar-2011
May-2010

Jul-2010
Aug-2010
Sep-2010

Dec-2010
Apr-2010

Oct-2010

Apr-2011
Jan-2010

Jun-2010

Jan-2011

Jun-2011
May-2011
Source: comScore Mobilens US June 2011


Platform Explosion is just getting started

100%
90%
66% of
80%
Americans
70% DO NOT yet
60% have a
Smartphones
50%
40%
US Smartphone Penetration: 33.5%
30%
20%
US iPad Penetration: 4.4% 95% of
10%
Americans
0% Nov-2010 DO NOT yet

Feb-2011
Feb-2010
Mar-2010

Mar-2011
May-2010

Jul-2010
Aug-2010
Sep-2010

Dec-2010
Apr-2010

Oct-2010

Apr-2011
Jan-2010

Jun-2010

Jan-2011

Jun-2011
May-2011
have an iPad

Source: comScore Mobilens US June 2011


Unified Digital Measurement™ (UDM)

Global PERSON Global DEVICE
Measurement Measurement

PANEL CENSUS

Unified Digital Measurement (UDM)
Patent-Pending Methodology
Adopted by 80% of Top 100 US Media Properties & 60% in U.K.

© comScore, Inc. Proprietary. 58 V0910

Digital Measurement Ecosystem

Network Logs

User Site
Panels Tags

Household Application
Panels Tags


Technical Measurement Review

Platform Network Publisher Format Demo
Coverage Coverage Coverage Coverage Availability

Metered Panel Poor Very Good Moderate Moderate Very Good

Publisher Tags Very Good Very Good Poor Moderate Poor

Network Census Good Poor Very Good Good Poor







Site Tags provide foundation for full cross platform view.
Drives de-duplication methodology






Panels provide foundation for person-centric
measurement & demographics





Site Tags Very Good Very Good Poor Moderate Poor


Publisher coverage remains a challenge. Site tags will
remain most accurate measurement


Establishing Next Generation Best Practices


A Next Generation Digital Measurement System

Industry Oversight Define Universe & Control
& Audits by Independent for Demographic Bias
Agents
High Quality
Enumeration
Survey

Active Panel
Census Panel
Calibration Partner Control
Control Dashboard
Panel
Behavioral In-tab Monthly Composition
Bias Sample
Probability Enhanced
Validation Sample
Panel Selection

Probability Jackknife
Validation Replication
Panel Study

Panel Stability & Calculate
Margin of Error


Probability Validation Panel: Concept & Explanation

• Non-probability sampling in digital user panels
• Sampling challenges
• Statistical efficiency and effective sample size

• Jackknife replicates, probability and validation

• Empirical method to examine variance in sample data
• Provides standard error, design effect, stability over sample sources
• Assumes EDF is good estimator of PDF, very large samples,…
• PVP triangulates on bias and reliability questions

• Census as a source for behavioral weights

• Panelist cookie consumption versus universe cookie consumption


Standard Error Study using Jackknife replication

 Jackknife replication, related to the more general Bootstrap, is used to
assess the variability of a statistic by examining the variation within the
sample data (vs parametric assumptions), especially with complex
sampling (Efron & Tibshirani, 1994).
 Variance of standard comScore key measures estimators (for monthly,
annual, including top 100 properties & associated entities):
– Unique Visitors
– Page Views
– Duration(Minutes)
 Purpose is to understand the variance of the sample design, controlling for
comScore’s methodology and processes for any monthly delivery.
 Results of the study will be utilized to create a general functional form for
estimating margin of error for any reportable data.


Unique Visitor replication results: Smoothed polynomial showing
consistency of UV estimates month over month


Page View replication results: Smoothed polynomial showing
consistency of PV estimates month over month


Jackknife Validation continued…

 Replication study shows consistency of estimates across properties of
various sizes and categories, and across time periods.
 Provides empirical data to develop a polynomial function for estimating
margin of error for any reportable data.
 Addresses key concern/question about variance estimates from non-
probability samples and their stability over time, given the fluctuation of
sample sources (partners, affiliate sites).
 Also powerful to validate bias correction using device census calibration –
there are important assumptions, however:
– But Jackknife assumes EDF is representative of the PDF. While this is
mitigated by large sample sizes, it requires validation…hence, PVP.


Summary

• Digital Measurement Challenges are unprecedented
• Fragmentation of devices, consumption, attention
• Multi-source and mixed methods are required

• Measurement across platforms and locations

• Site tagging, OS and IP information plus user panels
• Duplication issue across multiple access points
• Differences in patterns of duplication across content types, and demos
• Industry standards must be set; industry participation is crucial

• Probabilistic reference for multi-source measurement

• Guide to industry accepted methodology and best practices


All Possible Worlds iab mx_public

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Semelhante a All Possible Worlds iab mx_public

Semelhante a All Possible Worlds iab mx_public (20)

Mais de IAB México

Mais de IAB México (20)

Último

Último (20)

All Possible Worlds iab mx_public