SlideShare uma empresa Scribd logo
1 de 27
Baixar para ler offline
EVERYONE CARES ABOUT SAMPLE
QUALITY, BUT NOT EVERYONE VALUES
IT!
A review of responsibilities and techniques you can implement to protect
your online research and beyond
REAL PEOPLE, QUALITY
DATATM
DATA QUALITY SOFTWARE
Lisa Wilding-Brown
Chief Research Officer
Mark Menig
Chief Executive Officer
Agenda
■ Quality through the years (brief overview of where we’ve been and where we are going)
■ Current landscape i.e., bots, hijackers, foreign click shops in China etc.
■ Challenges & costs associated with today’s online fraud and how it impacts data quality
■ Implementing an effective solution (multi-layered approach)
– Technical approaches: Digital fingerprinting (when and where); Respondent validation;
algorithmic solutions over a member’s lifetime, other 3rd-party techniques, etc.
– Behavioral approaches: Knowledge question design (red-herrings); Pre-survey screening; smart
survey design (do’s and don’ts)
■ The Path Forward: Responsibility, Accountability, & Collaboration
3
Care About vs.Value
■ When you care about something, you simply have even minimal regard for someone
or something.
■ When you VALUE something, you consider it important and worthwhile. ...As a verb,
it means "holding something in high regard," (like "I value our friendship") but it can
also mean "determine how much something is WORTH," like a prize valued at $200.
4
QUALITY
means doing it right when no
one is looking
5
2000 2006 2008 2012 2016 2020
The industry
rapidly
becomes
enamored
with the
speed and
cost savings
of moving to
online
Industry
associations
launch major
initiatives to
investigate
and restore
online
research
quality
Fraud
continues to
morph and
evolve with
the
emergence
of new
threats
P&G speaks
out about
online data
quality issues
at the Client
Summit
sparking
industry-
wide
discourse
Rapid
evolution and
diversification
of devices and
engaging
respondents
migrates from
a proximity-
fixed
experience to
a portable
experience
The only
constant is
change!
Continual
innovation is
required in
order to stay
ahead;
recognizing
the battle is
never over
Current Landscape
Dr. Liz Nelson, co-founder of
TNS, advisor to the board of Fly
Research and a fellow of the
Market Research Society, talks
about how the need for speed is
affecting the quality of
research.
Research Live – November 24, 2016
7
“I would say immediately that the emphasis on speed is what’s happening now. Clients demand
immediate results with the survey in field on Friday, and 2000 results the next day. I think the sad bit is
that quality suffers”
Current Landscape
 Recent advances in big data and artificial intelligence are
now making it possible to teach a machine to understand
and speak to humans.
 It's very difficult to simply look at the data provided by
some of the more sophisticated bots and identify what to
remove, because it's all gray goo inside, just like a real
brain, and may be indistinguishable from real data.
 Need a real world example? Take out your iPhone and ask
Siri a question.
 Forums like the one to the left abound online with users
looking for and sharing information about how to utilize
tools to create/mimic bots and automate the process of
filling in surveys.
8
Current Landscape
“Here is survey bot attempting to
complete a survey with no given
information.The creator ran this on 6
surveys a day for two weeks (fully
automated of course) and got the total
sum of £14.95p, with no user
interaction what so ever!”
That was 10 questions completed in
under 17 seconds in case you lost
count!
9
Current Landscape
“Create a fake whatever you need”
10
Current Landscape
 TheTor software protects users by bouncing their
communications around a distributed network of relays run
by volunteers all around the world.
 TheTor Browser gives access toTor onWindows, Mac OS X,
or Linux without needing to install any software.
 Survey Click Shops are popping up around the globe
 Comprised of many “unique” devices in a single location
being utilized by a group of fraudsters to game surveys and
generate incentives
11
Current Landscape
 Device Emulators. In computing, an emulator is hardware
or software that enables one computer system (called the
host) to behave like another computer system (called the
guest).
 This threat will only get worse as computers and global
computer networks continued to advance and emulator
developers grow more skilled in their work.
 Datacenters,VPNs,Anonymous Proxies, etc. are favorite
tools for fraudsters because they allow them to spoof their
device to appear to be coming from a different country on a
case by case basis as needed based on the requirements of
a given survey.
12
Challenges & Costs
Timeliness
of fielding
Purchase
process
Ease of
accessing
panel
Customer
service
Quantity of
respondents
Cost of panel Quality of
Respondents
Not at all satisfied 2% 2% 2% 3% 5% 5% 7%
Slightly satisfied 11% 8% 12% 10% 17% 15% 26%
Moderately satisfied 33% 37% 36% 39% 41% 46% 42%
Very satisfied 44% 44% 42% 40% 31% 30% 23%
Completely satisfied 9% 9% 8% 8% 5% 5% 3%
Top 2 box 54% 53% 50% 49% 36% 34% 26%
2016 GRIT Report
13
Challenges & Costs
“Technology, or lack thereof, is the prime culprit for sample
getting worse: from bots, to survey design, to mobile enabled
surveys, all these are driving sample quality down. Many folks
have a strong sense that there are only professional survey
takers and fraudulent bots that are taking all the surveys
because there is a race to the bottom in terms of cost.”
“Sample providers should only actively communicate on
issue of representativeness, not quality or design.”
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Insights Buyer or Client
Insights Providers or Supplier
Sample Quality by Buyers vs. Suppliers
Better Worse Stay the Same Not Sure
2016 GRIT Report
14
Implementing an Effective Solution
Technical Approaches
Most Adopted Fraud DetectionTools
67%
51%
32%
11%
13%
17%
0 10 20 30 40 50 60 70 80 90
Identity/Address Validation
IP Geo Location Information
Device Fingerprinting
Currently Using Planning New Implementation 2016 Fraud Report
15
Implementing an Effective Solution
Technical Approaches
DEVICE FINGERPRINT A device fingerprint or machine fingerprint or browser
fingerprint is information collected about a remote
computing device for the purpose of identification.
Fingerprints can be used to fully or partially identify
individual users or devices even when cookies are turned off.
Motivation for the device fingerprint concept stems from
the forensic value of human fingerprints. In the "ideal" case,
all web client machines would have a different fingerprint
value (diversity), and that value would never change
(stability). Under those assumptions, it would be possible to
uniquely distinguish between all machines on a network,
without the explicit consent of the users themselves.
16
Implementing an Effective Solution
Technical Approaches
IDENTITY VALIDATION
Identity validation solutions allow for the evaluation of
names, postal addresses, and/or email addresses against
third-party consumer databases to determine if they're
legitimate and correspond with one another. They provide
confidence in knowing that a participant is who they say they
are and lives where they say they live. Also allows for the
removal of duplicates within and across sources.
Layering in a Geo-Location Distance Check adds additional
fraud detection by calculating the distance (in miles across the
surface of a sphere) between the latitude/longitude
coordinates of the postal address and the latitude/longitude
coordinates that the user’s IP address resolves to.
17
Implementing an Effective Solution
Technical Approaches
FRAUD DETECTION
At the device level, there are key markers that can be identified
to indicate the risk of first time user fraud:
 Language Check
 Geo-Browser Language Check
 Geo-OS Language Check
 Geo-Time Zone Check
 Geo-Off Hours Check
 Geo-Country Check
 Multi-Device Check
 Bot Check
 Anonymous Check
 Blacklist Check
 Browser Status Check
18
Implementing an Effective Solution
Technical Approaches
SURVEY VALIDATION
A respondent can be flagged as unengaged in the survey if he or
she speeds on at least X% of the pages they saw in the survey.The
norms and standard deviations of the times for each page should
be calculated in real-time as the page submissions from the
respondents are received by the survey platform.
It can also be useful to consider the response patterns that are
being submitted as another key indicator. Respondents who
provide undesirable response patterns on more than X% of pages
can also be classified as unengaged for the survey.
Good ResponseValidation tools leverage real-time Bayesian
statistical models/analysis to determine engagement.
19
Implementing an Effective Solution
Behavioral Approaches
There are three channels to address
in order to ensure superior data
quality in your study:
 Sample Design & Management
 Survey Design
 Member Management
20
Implementing an Effective Solution
Behavioral Approaches – Sample Design & Management
 Vendor selection is key. Understand how your vendor’s sample is sourced,
managed and incentivized.
 Ask the tough questions! How is sample outgo balanced? What measures are
implemented to ensure the highest quality sample is provided?
 Demographic balance
 Activity & tenure balance
 Survey field time
 Invitation/introductory language
 Competing survey inventory
 Survey frequency & variation
 Routing/project prioritization 21
Implementing an Effective Solution
Behavioral Approaches – Survey Design
 Question design is key!
 Use non-leading wording
 Provide an out for all respondents
 Use open-ends sparingly
 Avoid yes/no format
22
Implementing an Effective Solution
Behavioral Approaches – Survey Design
 Avoid burdensome question formats (i.e., extensive grids and lists longer
than 10-15 attributes).
 Strive to keep your survey short and simple.
 Clear, concise wording – write for a 5th grader!
 Avoid multiple questions on one screen – visual clutter will result in
respondent fatigue.
 Mobile-compatible and mobile-friendly are two different things!
23
Implementing an Effective Solution
Behavioral Approaches – Member Management
 Trap Questions
 Honey Pots
 Algorithmic solutions
 Tracking activity over time (LOI completions & invalids)
 Profiling & third-party data validation sources
 Demo consistency checks
 Quality exists across a wide spectrum; lifetime
management is critical 24
Implementing an Effective Solution
Behavioral Approaches – Trap Questions Do’s & Don’ts
 Not all trap questions are effective! Trap questions shouldn’t be too simple or too complex.
 Types:
 Instructional (i.e., Select the image which shows a book.)
 Skill-based (i.e., 2+2 = ?)
 Honesty-based (i.e., What brand(s) are you aware of? What activities have you done in the
last 12 months?)
 Implement multiple measures to assess quality, never rely on a singular question within the
survey to dictate quality.
 Be mindful of question position within the survey i.e., adding your trap question at minute 45
will yield false positives that arguably are a result of a lengthy survey NOT a poorly-behaving
respondent.
25
Implementing an Effective Solution
Applying Our Learnings to B2B Research
 Know thy sample source!
 Always use multiple knowledge-based trap questions (.i.e.,
looking for experts in cloud-computing? Test their knowledge
on various storage products vs. the color of the sky).
 Implement multiple measures to assess quality (inclusive of
technical and behavioral approaches).
 When possible, leverage 3rd party data sources to validate
member data.
 Never become complacent – your research will always be a hot
target for fraud. Stay protected! 26
The Path Forward: Responsibility,
Accountability, & Collaboration
 Every company up and down the supply chain involved in the execution of online
research has a role/responsibility as it relates to data quality/fraud detection. What
you are responsible for depends on which part of the research process you have
operational control over (i.e. you can’t just push responsibility down to the
operational layer below you, everyone has to do their part, or the whole system
suffers).
 There is no silver bullet solution. Effective solutions require a layered
technique/approach that incorporates redundancies and failsafe mechanisms.
 It’s not enough to simply care about data quality and fraud detection, you must
VALUE it!
27

Mais conteúdo relacionado

Mais procurados

Engaging with Users on Public Social Media
Engaging with Users on Public Social MediaEngaging with Users on Public Social Media
Engaging with Users on Public Social MediaJeffrey Nichols
 
Digital analytics: Analytics problems (Lecture 9)
Digital analytics: Analytics problems (Lecture 9)Digital analytics: Analytics problems (Lecture 9)
Digital analytics: Analytics problems (Lecture 9)Joni Salminen
 
Social Listening and Intelligence is Predictive! Now What?
Social Listening and Intelligence is Predictive!  Now What?Social Listening and Intelligence is Predictive!  Now What?
Social Listening and Intelligence is Predictive! Now What?Rob Key
 
29 Revenue Model Options for Industrial enterprises (curated by @arnevbalen -...
29 Revenue Model Options for Industrial enterprises (curated by @arnevbalen -...29 Revenue Model Options for Industrial enterprises (curated by @arnevbalen -...
29 Revenue Model Options for Industrial enterprises (curated by @arnevbalen -...Board of Innovation
 
Data-Driven Design (MX '10)
Data-Driven Design (MX '10)Data-Driven Design (MX '10)
Data-Driven Design (MX '10)Ryan Freitas
 
Mobile Sentiment Presentation Ver 3.0
Mobile Sentiment Presentation Ver 3.0Mobile Sentiment Presentation Ver 3.0
Mobile Sentiment Presentation Ver 3.0Darrin Helsel
 
The UX Lexicon is Born – clear communication and understanding for all resear...
The UX Lexicon is Born – clear communication and understanding for all resear...The UX Lexicon is Born – clear communication and understanding for all resear...
The UX Lexicon is Born – clear communication and understanding for all resear...Ray Poynter
 
The Right Research Method For Any Problem (And Budget)
The Right Research Method For Any Problem (And Budget)The Right Research Method For Any Problem (And Budget)
The Right Research Method For Any Problem (And Budget)Leah Buley
 
A day in the life of a data scientist in an AI company
A day in the life of a data scientist in an AI companyA day in the life of a data scientist in an AI company
A day in the life of a data scientist in an AI companyFrancesca Lazzeri, PhD
 
The Gamification of Smart Devices: Some Preliminary Thoughts on Concepts and ...
The Gamification of Smart Devices: Some Preliminary Thoughts on Concepts and ...The Gamification of Smart Devices: Some Preliminary Thoughts on Concepts and ...
The Gamification of Smart Devices: Some Preliminary Thoughts on Concepts and ...Donna Hoffman
 
OTO: Online Trust Oracle for User-Centric Trust Establishment, at CCS 2012
OTO: Online Trust Oracle for User-Centric Trust Establishment, at CCS 2012OTO: Online Trust Oracle for User-Centric Trust Establishment, at CCS 2012
OTO: Online Trust Oracle for User-Centric Trust Establishment, at CCS 2012Jason Hong
 
Data Driven Design
Data Driven DesignData Driven Design
Data Driven DesignTanya M.
 
MIT C-Brief Closing the CX Gap with Digital-Performance Management
MIT C-Brief Closing the CX Gap with Digital-Performance ManagementMIT C-Brief Closing the CX Gap with Digital-Performance Management
MIT C-Brief Closing the CX Gap with Digital-Performance ManagementSteve Trimbo
 

Mais procurados (15)

Engaging with Users on Public Social Media
Engaging with Users on Public Social MediaEngaging with Users on Public Social Media
Engaging with Users on Public Social Media
 
Digital analytics: Analytics problems (Lecture 9)
Digital analytics: Analytics problems (Lecture 9)Digital analytics: Analytics problems (Lecture 9)
Digital analytics: Analytics problems (Lecture 9)
 
Social Listening and Intelligence is Predictive! Now What?
Social Listening and Intelligence is Predictive!  Now What?Social Listening and Intelligence is Predictive!  Now What?
Social Listening and Intelligence is Predictive! Now What?
 
CUTGroup 13 - mRelief Final Report
CUTGroup 13 - mRelief Final ReportCUTGroup 13 - mRelief Final Report
CUTGroup 13 - mRelief Final Report
 
29 Revenue Model Options for Industrial enterprises (curated by @arnevbalen -...
29 Revenue Model Options for Industrial enterprises (curated by @arnevbalen -...29 Revenue Model Options for Industrial enterprises (curated by @arnevbalen -...
29 Revenue Model Options for Industrial enterprises (curated by @arnevbalen -...
 
Data-Driven Design (MX '10)
Data-Driven Design (MX '10)Data-Driven Design (MX '10)
Data-Driven Design (MX '10)
 
CUTGroup 12 Roll with Me Final Report
CUTGroup 12 Roll with Me Final ReportCUTGroup 12 Roll with Me Final Report
CUTGroup 12 Roll with Me Final Report
 
Mobile Sentiment Presentation Ver 3.0
Mobile Sentiment Presentation Ver 3.0Mobile Sentiment Presentation Ver 3.0
Mobile Sentiment Presentation Ver 3.0
 
The UX Lexicon is Born – clear communication and understanding for all resear...
The UX Lexicon is Born – clear communication and understanding for all resear...The UX Lexicon is Born – clear communication and understanding for all resear...
The UX Lexicon is Born – clear communication and understanding for all resear...
 
The Right Research Method For Any Problem (And Budget)
The Right Research Method For Any Problem (And Budget)The Right Research Method For Any Problem (And Budget)
The Right Research Method For Any Problem (And Budget)
 
A day in the life of a data scientist in an AI company
A day in the life of a data scientist in an AI companyA day in the life of a data scientist in an AI company
A day in the life of a data scientist in an AI company
 
The Gamification of Smart Devices: Some Preliminary Thoughts on Concepts and ...
The Gamification of Smart Devices: Some Preliminary Thoughts on Concepts and ...The Gamification of Smart Devices: Some Preliminary Thoughts on Concepts and ...
The Gamification of Smart Devices: Some Preliminary Thoughts on Concepts and ...
 
OTO: Online Trust Oracle for User-Centric Trust Establishment, at CCS 2012
OTO: Online Trust Oracle for User-Centric Trust Establishment, at CCS 2012OTO: Online Trust Oracle for User-Centric Trust Establishment, at CCS 2012
OTO: Online Trust Oracle for User-Centric Trust Establishment, at CCS 2012
 
Data Driven Design
Data Driven DesignData Driven Design
Data Driven Design
 
MIT C-Brief Closing the CX Gap with Digital-Performance Management
MIT C-Brief Closing the CX Gap with Digital-Performance ManagementMIT C-Brief Closing the CX Gap with Digital-Performance Management
MIT C-Brief Closing the CX Gap with Digital-Performance Management
 

Destaque

Presentación1
Presentación1Presentación1
Presentación12QJOSE
 
Live Icons on iOS
Live Icons on iOSLive Icons on iOS
Live Icons on iOSMartin Kiss
 
Touristquiz- Consultoria Calidad Sector Turismo
Touristquiz- Consultoria Calidad Sector TurismoTouristquiz- Consultoria Calidad Sector Turismo
Touristquiz- Consultoria Calidad Sector TurismoGuadalupe Chamorro
 
5 stylish Shoes according to Victoria’s Secret Models!!
5 stylish Shoes according to Victoria’s Secret Models!!5 stylish Shoes according to Victoria’s Secret Models!!
5 stylish Shoes according to Victoria’s Secret Models!!Charu Fashions
 
Cùng tủ bếp xinh đón tết đoàn viên
Cùng tủ bếp xinh đón tết đoàn viênCùng tủ bếp xinh đón tết đoàn viên
Cùng tủ bếp xinh đón tết đoàn viênluyen ngo
 
История компьютеров в числовых ребусах. Альманах
История компьютеров в числовых ребусах. АльманахИстория компьютеров в числовых ребусах. Альманах
История компьютеров в числовых ребусах. АльманахАнатолий Мячев
 
Quiero volver a lo de antes
Quiero volver a lo de antesQuiero volver a lo de antes
Quiero volver a lo de antesJose Iglesias
 
Parejas disparejas
Parejas disparejasParejas disparejas
Parejas disparejaspolinchis
 

Destaque (13)

Presentación1
Presentación1Presentación1
Presentación1
 
Live Icons on iOS
Live Icons on iOSLive Icons on iOS
Live Icons on iOS
 
Los caminos de la vida
Los caminos de la vida Los caminos de la vida
Los caminos de la vida
 
A mi edad
A mi edadA mi edad
A mi edad
 
Food Processing Sector Reports November-2016
Food Processing Sector Reports November-2016Food Processing Sector Reports November-2016
Food Processing Sector Reports November-2016
 
Touristquiz- Consultoria Calidad Sector Turismo
Touristquiz- Consultoria Calidad Sector TurismoTouristquiz- Consultoria Calidad Sector Turismo
Touristquiz- Consultoria Calidad Sector Turismo
 
5 stylish Shoes according to Victoria’s Secret Models!!
5 stylish Shoes according to Victoria’s Secret Models!!5 stylish Shoes according to Victoria’s Secret Models!!
5 stylish Shoes according to Victoria’s Secret Models!!
 
Chromium Sandbox
Chromium SandboxChromium Sandbox
Chromium Sandbox
 
Cùng tủ bếp xinh đón tết đoàn viên
Cùng tủ bếp xinh đón tết đoàn viênCùng tủ bếp xinh đón tết đoàn viên
Cùng tủ bếp xinh đón tết đoàn viên
 
История компьютеров в числовых ребусах. Альманах
История компьютеров в числовых ребусах. АльманахИстория компьютеров в числовых ребусах. Альманах
История компьютеров в числовых ребусах. Альманах
 
Quiero volver a lo de antes
Quiero volver a lo de antesQuiero volver a lo de antes
Quiero volver a lo de antes
 
CV updated-Malik Akhtar Ali.
CV updated-Malik Akhtar Ali.CV updated-Malik Akhtar Ali.
CV updated-Malik Akhtar Ali.
 
Parejas disparejas
Parejas disparejasParejas disparejas
Parejas disparejas
 

Semelhante a Webinar: Everyone cares about sample quality but not everyone values it!

How to manage and optimize mobile marketing using webanalytics - Remi van Bee...
How to manage and optimize mobile marketing using webanalytics - Remi van Bee...How to manage and optimize mobile marketing using webanalytics - Remi van Bee...
How to manage and optimize mobile marketing using webanalytics - Remi van Bee...StormMC
 
World Paper Free Day
World Paper Free DayWorld Paper Free Day
World Paper Free DayJustin Phipps
 
Putting data science into perspective
Putting data science into perspectivePutting data science into perspective
Putting data science into perspectiveSravan Ankaraju
 
CPCU 2016 future of underwriting insurtech
CPCU 2016   future of underwriting insurtechCPCU 2016   future of underwriting insurtech
CPCU 2016 future of underwriting insurtechintellectseec
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in FocusRay Poynter
 
Digital Experiences Using a Conversational Interface
Digital Experiences Using a Conversational InterfaceDigital Experiences Using a Conversational Interface
Digital Experiences Using a Conversational InterfaceBala Iyer
 
PagerDuty: Optimizing Incident Response to Deliver Amazing Digital Experiences
PagerDuty: Optimizing Incident Response to Deliver Amazing Digital ExperiencesPagerDuty: Optimizing Incident Response to Deliver Amazing Digital Experiences
PagerDuty: Optimizing Incident Response to Deliver Amazing Digital ExperiencesPagerDuty
 
Opportunities with data science
Opportunities with data scienceOpportunities with data science
Opportunities with data scienceAshiq Rahman
 
Big Data Meetup by Chad Richeson
Big Data Meetup by Chad RichesonBig Data Meetup by Chad Richeson
Big Data Meetup by Chad RichesonSocietyConsulting
 
Artificial Intelligence Primer
Artificial Intelligence PrimerArtificial Intelligence Primer
Artificial Intelligence PrimerImam Hoque
 
Living Up to Employee & Consumer Expectations
Living Up to Employee & Consumer ExpectationsLiving Up to Employee & Consumer Expectations
Living Up to Employee & Consumer ExpectationsInsight
 
Becoming a Customer Centric Bank
Becoming a Customer Centric BankBecoming a Customer Centric Bank
Becoming a Customer Centric BankNG DATA
 
Data-Driven Design for User Experience
Data-Driven Design for User Experience Data-Driven Design for User Experience
Data-Driven Design for User Experience Emi Kwon
 
GTC West (AM): Technology As A Tool For Innovation
GTC West (AM): Technology As A Tool For InnovationGTC West (AM): Technology As A Tool For Innovation
GTC West (AM): Technology As A Tool For InnovationDustin Haisler
 
LSI Spring Agent Open House 2014
LSI Spring Agent Open House 2014LSI Spring Agent Open House 2014
LSI Spring Agent Open House 2014Ashlie Steele
 
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...InsightInnovation
 
Connecting the "dots" around your Consumers
Connecting the "dots" around your ConsumersConnecting the "dots" around your Consumers
Connecting the "dots" around your ConsumersDigital Air Strike
 
How to use Online Marketing Technology to Improve Campaign Performance - Lowe...
How to use Online Marketing Technology to Improve Campaign Performance - Lowe...How to use Online Marketing Technology to Improve Campaign Performance - Lowe...
How to use Online Marketing Technology to Improve Campaign Performance - Lowe...Online Marketing Summit
 

Semelhante a Webinar: Everyone cares about sample quality but not everyone values it! (20)

How to manage and optimize mobile marketing using webanalytics - Remi van Bee...
How to manage and optimize mobile marketing using webanalytics - Remi van Bee...How to manage and optimize mobile marketing using webanalytics - Remi van Bee...
How to manage and optimize mobile marketing using webanalytics - Remi van Bee...
 
World Paper Free Day
World Paper Free DayWorld Paper Free Day
World Paper Free Day
 
Putting data science into perspective
Putting data science into perspectivePutting data science into perspective
Putting data science into perspective
 
CPCU 2016 future of underwriting insurtech
CPCU 2016   future of underwriting insurtechCPCU 2016   future of underwriting insurtech
CPCU 2016 future of underwriting insurtech
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
 
Digital Experiences Using a Conversational Interface
Digital Experiences Using a Conversational InterfaceDigital Experiences Using a Conversational Interface
Digital Experiences Using a Conversational Interface
 
PagerDuty: Optimizing Incident Response to Deliver Amazing Digital Experiences
PagerDuty: Optimizing Incident Response to Deliver Amazing Digital ExperiencesPagerDuty: Optimizing Incident Response to Deliver Amazing Digital Experiences
PagerDuty: Optimizing Incident Response to Deliver Amazing Digital Experiences
 
Opportunities with data science
Opportunities with data scienceOpportunities with data science
Opportunities with data science
 
Big Data Meetup by Chad Richeson
Big Data Meetup by Chad RichesonBig Data Meetup by Chad Richeson
Big Data Meetup by Chad Richeson
 
Artificial Intelligence Primer
Artificial Intelligence PrimerArtificial Intelligence Primer
Artificial Intelligence Primer
 
Ai in Society
Ai in SocietyAi in Society
Ai in Society
 
Living Up to Employee & Consumer Expectations
Living Up to Employee & Consumer ExpectationsLiving Up to Employee & Consumer Expectations
Living Up to Employee & Consumer Expectations
 
Becoming a Customer Centric Bank
Becoming a Customer Centric BankBecoming a Customer Centric Bank
Becoming a Customer Centric Bank
 
Data-Driven Design for User Experience
Data-Driven Design for User Experience Data-Driven Design for User Experience
Data-Driven Design for User Experience
 
GTC West (AM): Technology As A Tool For Innovation
GTC West (AM): Technology As A Tool For InnovationGTC West (AM): Technology As A Tool For Innovation
GTC West (AM): Technology As A Tool For Innovation
 
LSI Spring Agent Open House 2014
LSI Spring Agent Open House 2014LSI Spring Agent Open House 2014
LSI Spring Agent Open House 2014
 
Real Life Analytics
Real Life AnalyticsReal Life Analytics
Real Life Analytics
 
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
 
Connecting the "dots" around your Consumers
Connecting the "dots" around your ConsumersConnecting the "dots" around your Consumers
Connecting the "dots" around your Consumers
 
How to use Online Marketing Technology to Improve Campaign Performance - Lowe...
How to use Online Marketing Technology to Improve Campaign Performance - Lowe...How to use Online Marketing Technology to Improve Campaign Performance - Lowe...
How to use Online Marketing Technology to Improve Campaign Performance - Lowe...
 

Último

Digital Marketing Analytics: Driving Hotel Success (2016 May report)
Digital Marketing Analytics: Driving Hotel Success (2016 May report)Digital Marketing Analytics: Driving Hotel Success (2016 May report)
Digital Marketing Analytics: Driving Hotel Success (2016 May report)yaeyukimoto
 
A navigation of two creative processes Study
A navigation of two creative processes StudyA navigation of two creative processes Study
A navigation of two creative processes Studystuwilson.co.uk
 
Amazon Seller New Product Launch Guide - 2024.pptx
Amazon Seller New Product Launch Guide - 2024.pptxAmazon Seller New Product Launch Guide - 2024.pptx
Amazon Seller New Product Launch Guide - 2024.pptxWill Haire
 
Voltas turnaround strategy management case
Voltas turnaround strategy management caseVoltas turnaround strategy management case
Voltas turnaround strategy management caseAnkit Sarkar
 
Friends of Search '24 - Scaling SEO_ Lessons for All Types of Sites.pptx
Friends of Search '24 - Scaling SEO_ Lessons for All Types of Sites.pptxFriends of Search '24 - Scaling SEO_ Lessons for All Types of Sites.pptx
Friends of Search '24 - Scaling SEO_ Lessons for All Types of Sites.pptxGregory Edwards
 
Digital Marketing Services like SEO, SMM, SEM
Digital Marketing Services like SEO, SMM, SEMDigital Marketing Services like SEO, SMM, SEM
Digital Marketing Services like SEO, SMM, SEMNazal Digital
 
SVETLANA YONCHEVA Evolution of digital marketing.pdf
SVETLANA YONCHEVA Evolution of digital marketing.pdfSVETLANA YONCHEVA Evolution of digital marketing.pdf
SVETLANA YONCHEVA Evolution of digital marketing.pdfvikrs213
 
Linking Technical SEO to Performance Milestones PDF.pdf
Linking Technical SEO to Performance Milestones PDF.pdfLinking Technical SEO to Performance Milestones PDF.pdf
Linking Technical SEO to Performance Milestones PDF.pdfRasida Begum
 
Elevate Your Design Skills: Enroll in Pune's Premier UI/UX Design Course
Elevate Your Design Skills: Enroll in Pune's Premier UI/UX Design CourseElevate Your Design Skills: Enroll in Pune's Premier UI/UX Design Course
Elevate Your Design Skills: Enroll in Pune's Premier UI/UX Design Courseamirshaikhv21realtyp
 
Imposter Syndrome in Marketing & Why You're Not Alone
Imposter Syndrome in Marketing & Why You're Not AloneImposter Syndrome in Marketing & Why You're Not Alone
Imposter Syndrome in Marketing & Why You're Not AloneHerd
 
Digital Marketing PPT.pptx(PPT related the Digital marketing)
Digital Marketing PPT.pptx(PPT related the Digital marketing)Digital Marketing PPT.pptx(PPT related the Digital marketing)
Digital Marketing PPT.pptx(PPT related the Digital marketing)digitalwithalok1
 
Run more experiments with fewer resources
Run more experiments with fewer resourcesRun more experiments with fewer resources
Run more experiments with fewer resourcesVWO
 
A_B Testing Personalized Meditation Recommendations.pdf
A_B Testing Personalized Meditation Recommendations.pdfA_B Testing Personalized Meditation Recommendations.pdf
A_B Testing Personalized Meditation Recommendations.pdfVWO
 
SEO Trends in 2024: What You Need to Know to Succeed
SEO Trends in 2024: What You Need to Know to SucceedSEO Trends in 2024: What You Need to Know to Succeed
SEO Trends in 2024: What You Need to Know to SucceedMumbai Pixels
 
best digital marketing company in south delhi
best digital marketing company in south delhibest digital marketing company in south delhi
best digital marketing company in south delhifirstpointcreationsa
 
What's Happening with Your Microsoft Advertising's Search Campaigns? - Sophie...
What's Happening with Your Microsoft Advertising's Search Campaigns? - Sophie...What's Happening with Your Microsoft Advertising's Search Campaigns? - Sophie...
What's Happening with Your Microsoft Advertising's Search Campaigns? - Sophie...Sophie Logan
 
Increase Your Website Sales & Leads Webinar
Increase Your Website Sales & Leads WebinarIncrease Your Website Sales & Leads Webinar
Increase Your Website Sales & Leads WebinarSEO Optimizers
 
Unifying feature management with experiments - Server Side Webinar (1).pdf
Unifying feature management with experiments - Server Side Webinar (1).pdfUnifying feature management with experiments - Server Side Webinar (1).pdf
Unifying feature management with experiments - Server Side Webinar (1).pdfVWO
 
Digital Gravity - Full-Scale SEO Services Preview
Digital Gravity - Full-Scale SEO Services PreviewDigital Gravity - Full-Scale SEO Services Preview
Digital Gravity - Full-Scale SEO Services PreviewDigital Gravity
 
The Creative Marketing campaigns of WeRoad
The Creative Marketing campaigns of WeRoadThe Creative Marketing campaigns of WeRoad
The Creative Marketing campaigns of WeRoadFabio Bin
 

Último (20)

Digital Marketing Analytics: Driving Hotel Success (2016 May report)
Digital Marketing Analytics: Driving Hotel Success (2016 May report)Digital Marketing Analytics: Driving Hotel Success (2016 May report)
Digital Marketing Analytics: Driving Hotel Success (2016 May report)
 
A navigation of two creative processes Study
A navigation of two creative processes StudyA navigation of two creative processes Study
A navigation of two creative processes Study
 
Amazon Seller New Product Launch Guide - 2024.pptx
Amazon Seller New Product Launch Guide - 2024.pptxAmazon Seller New Product Launch Guide - 2024.pptx
Amazon Seller New Product Launch Guide - 2024.pptx
 
Voltas turnaround strategy management case
Voltas turnaround strategy management caseVoltas turnaround strategy management case
Voltas turnaround strategy management case
 
Friends of Search '24 - Scaling SEO_ Lessons for All Types of Sites.pptx
Friends of Search '24 - Scaling SEO_ Lessons for All Types of Sites.pptxFriends of Search '24 - Scaling SEO_ Lessons for All Types of Sites.pptx
Friends of Search '24 - Scaling SEO_ Lessons for All Types of Sites.pptx
 
Digital Marketing Services like SEO, SMM, SEM
Digital Marketing Services like SEO, SMM, SEMDigital Marketing Services like SEO, SMM, SEM
Digital Marketing Services like SEO, SMM, SEM
 
SVETLANA YONCHEVA Evolution of digital marketing.pdf
SVETLANA YONCHEVA Evolution of digital marketing.pdfSVETLANA YONCHEVA Evolution of digital marketing.pdf
SVETLANA YONCHEVA Evolution of digital marketing.pdf
 
Linking Technical SEO to Performance Milestones PDF.pdf
Linking Technical SEO to Performance Milestones PDF.pdfLinking Technical SEO to Performance Milestones PDF.pdf
Linking Technical SEO to Performance Milestones PDF.pdf
 
Elevate Your Design Skills: Enroll in Pune's Premier UI/UX Design Course
Elevate Your Design Skills: Enroll in Pune's Premier UI/UX Design CourseElevate Your Design Skills: Enroll in Pune's Premier UI/UX Design Course
Elevate Your Design Skills: Enroll in Pune's Premier UI/UX Design Course
 
Imposter Syndrome in Marketing & Why You're Not Alone
Imposter Syndrome in Marketing & Why You're Not AloneImposter Syndrome in Marketing & Why You're Not Alone
Imposter Syndrome in Marketing & Why You're Not Alone
 
Digital Marketing PPT.pptx(PPT related the Digital marketing)
Digital Marketing PPT.pptx(PPT related the Digital marketing)Digital Marketing PPT.pptx(PPT related the Digital marketing)
Digital Marketing PPT.pptx(PPT related the Digital marketing)
 
Run more experiments with fewer resources
Run more experiments with fewer resourcesRun more experiments with fewer resources
Run more experiments with fewer resources
 
A_B Testing Personalized Meditation Recommendations.pdf
A_B Testing Personalized Meditation Recommendations.pdfA_B Testing Personalized Meditation Recommendations.pdf
A_B Testing Personalized Meditation Recommendations.pdf
 
SEO Trends in 2024: What You Need to Know to Succeed
SEO Trends in 2024: What You Need to Know to SucceedSEO Trends in 2024: What You Need to Know to Succeed
SEO Trends in 2024: What You Need to Know to Succeed
 
best digital marketing company in south delhi
best digital marketing company in south delhibest digital marketing company in south delhi
best digital marketing company in south delhi
 
What's Happening with Your Microsoft Advertising's Search Campaigns? - Sophie...
What's Happening with Your Microsoft Advertising's Search Campaigns? - Sophie...What's Happening with Your Microsoft Advertising's Search Campaigns? - Sophie...
What's Happening with Your Microsoft Advertising's Search Campaigns? - Sophie...
 
Increase Your Website Sales & Leads Webinar
Increase Your Website Sales & Leads WebinarIncrease Your Website Sales & Leads Webinar
Increase Your Website Sales & Leads Webinar
 
Unifying feature management with experiments - Server Side Webinar (1).pdf
Unifying feature management with experiments - Server Side Webinar (1).pdfUnifying feature management with experiments - Server Side Webinar (1).pdf
Unifying feature management with experiments - Server Side Webinar (1).pdf
 
Digital Gravity - Full-Scale SEO Services Preview
Digital Gravity - Full-Scale SEO Services PreviewDigital Gravity - Full-Scale SEO Services Preview
Digital Gravity - Full-Scale SEO Services Preview
 
The Creative Marketing campaigns of WeRoad
The Creative Marketing campaigns of WeRoadThe Creative Marketing campaigns of WeRoad
The Creative Marketing campaigns of WeRoad
 

Webinar: Everyone cares about sample quality but not everyone values it!

  • 1. EVERYONE CARES ABOUT SAMPLE QUALITY, BUT NOT EVERYONE VALUES IT! A review of responsibilities and techniques you can implement to protect your online research and beyond
  • 2. REAL PEOPLE, QUALITY DATATM DATA QUALITY SOFTWARE Lisa Wilding-Brown Chief Research Officer Mark Menig Chief Executive Officer
  • 3. Agenda ■ Quality through the years (brief overview of where we’ve been and where we are going) ■ Current landscape i.e., bots, hijackers, foreign click shops in China etc. ■ Challenges & costs associated with today’s online fraud and how it impacts data quality ■ Implementing an effective solution (multi-layered approach) – Technical approaches: Digital fingerprinting (when and where); Respondent validation; algorithmic solutions over a member’s lifetime, other 3rd-party techniques, etc. – Behavioral approaches: Knowledge question design (red-herrings); Pre-survey screening; smart survey design (do’s and don’ts) ■ The Path Forward: Responsibility, Accountability, & Collaboration 3
  • 4. Care About vs.Value ■ When you care about something, you simply have even minimal regard for someone or something. ■ When you VALUE something, you consider it important and worthwhile. ...As a verb, it means "holding something in high regard," (like "I value our friendship") but it can also mean "determine how much something is WORTH," like a prize valued at $200. 4
  • 5. QUALITY means doing it right when no one is looking 5
  • 6. 2000 2006 2008 2012 2016 2020 The industry rapidly becomes enamored with the speed and cost savings of moving to online Industry associations launch major initiatives to investigate and restore online research quality Fraud continues to morph and evolve with the emergence of new threats P&G speaks out about online data quality issues at the Client Summit sparking industry- wide discourse Rapid evolution and diversification of devices and engaging respondents migrates from a proximity- fixed experience to a portable experience The only constant is change! Continual innovation is required in order to stay ahead; recognizing the battle is never over
  • 7. Current Landscape Dr. Liz Nelson, co-founder of TNS, advisor to the board of Fly Research and a fellow of the Market Research Society, talks about how the need for speed is affecting the quality of research. Research Live – November 24, 2016 7 “I would say immediately that the emphasis on speed is what’s happening now. Clients demand immediate results with the survey in field on Friday, and 2000 results the next day. I think the sad bit is that quality suffers”
  • 8. Current Landscape  Recent advances in big data and artificial intelligence are now making it possible to teach a machine to understand and speak to humans.  It's very difficult to simply look at the data provided by some of the more sophisticated bots and identify what to remove, because it's all gray goo inside, just like a real brain, and may be indistinguishable from real data.  Need a real world example? Take out your iPhone and ask Siri a question.  Forums like the one to the left abound online with users looking for and sharing information about how to utilize tools to create/mimic bots and automate the process of filling in surveys. 8
  • 9. Current Landscape “Here is survey bot attempting to complete a survey with no given information.The creator ran this on 6 surveys a day for two weeks (fully automated of course) and got the total sum of £14.95p, with no user interaction what so ever!” That was 10 questions completed in under 17 seconds in case you lost count! 9
  • 10. Current Landscape “Create a fake whatever you need” 10
  • 11. Current Landscape  TheTor software protects users by bouncing their communications around a distributed network of relays run by volunteers all around the world.  TheTor Browser gives access toTor onWindows, Mac OS X, or Linux without needing to install any software.  Survey Click Shops are popping up around the globe  Comprised of many “unique” devices in a single location being utilized by a group of fraudsters to game surveys and generate incentives 11
  • 12. Current Landscape  Device Emulators. In computing, an emulator is hardware or software that enables one computer system (called the host) to behave like another computer system (called the guest).  This threat will only get worse as computers and global computer networks continued to advance and emulator developers grow more skilled in their work.  Datacenters,VPNs,Anonymous Proxies, etc. are favorite tools for fraudsters because they allow them to spoof their device to appear to be coming from a different country on a case by case basis as needed based on the requirements of a given survey. 12
  • 13. Challenges & Costs Timeliness of fielding Purchase process Ease of accessing panel Customer service Quantity of respondents Cost of panel Quality of Respondents Not at all satisfied 2% 2% 2% 3% 5% 5% 7% Slightly satisfied 11% 8% 12% 10% 17% 15% 26% Moderately satisfied 33% 37% 36% 39% 41% 46% 42% Very satisfied 44% 44% 42% 40% 31% 30% 23% Completely satisfied 9% 9% 8% 8% 5% 5% 3% Top 2 box 54% 53% 50% 49% 36% 34% 26% 2016 GRIT Report 13
  • 14. Challenges & Costs “Technology, or lack thereof, is the prime culprit for sample getting worse: from bots, to survey design, to mobile enabled surveys, all these are driving sample quality down. Many folks have a strong sense that there are only professional survey takers and fraudulent bots that are taking all the surveys because there is a race to the bottom in terms of cost.” “Sample providers should only actively communicate on issue of representativeness, not quality or design.” 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Insights Buyer or Client Insights Providers or Supplier Sample Quality by Buyers vs. Suppliers Better Worse Stay the Same Not Sure 2016 GRIT Report 14
  • 15. Implementing an Effective Solution Technical Approaches Most Adopted Fraud DetectionTools 67% 51% 32% 11% 13% 17% 0 10 20 30 40 50 60 70 80 90 Identity/Address Validation IP Geo Location Information Device Fingerprinting Currently Using Planning New Implementation 2016 Fraud Report 15
  • 16. Implementing an Effective Solution Technical Approaches DEVICE FINGERPRINT A device fingerprint or machine fingerprint or browser fingerprint is information collected about a remote computing device for the purpose of identification. Fingerprints can be used to fully or partially identify individual users or devices even when cookies are turned off. Motivation for the device fingerprint concept stems from the forensic value of human fingerprints. In the "ideal" case, all web client machines would have a different fingerprint value (diversity), and that value would never change (stability). Under those assumptions, it would be possible to uniquely distinguish between all machines on a network, without the explicit consent of the users themselves. 16
  • 17. Implementing an Effective Solution Technical Approaches IDENTITY VALIDATION Identity validation solutions allow for the evaluation of names, postal addresses, and/or email addresses against third-party consumer databases to determine if they're legitimate and correspond with one another. They provide confidence in knowing that a participant is who they say they are and lives where they say they live. Also allows for the removal of duplicates within and across sources. Layering in a Geo-Location Distance Check adds additional fraud detection by calculating the distance (in miles across the surface of a sphere) between the latitude/longitude coordinates of the postal address and the latitude/longitude coordinates that the user’s IP address resolves to. 17
  • 18. Implementing an Effective Solution Technical Approaches FRAUD DETECTION At the device level, there are key markers that can be identified to indicate the risk of first time user fraud:  Language Check  Geo-Browser Language Check  Geo-OS Language Check  Geo-Time Zone Check  Geo-Off Hours Check  Geo-Country Check  Multi-Device Check  Bot Check  Anonymous Check  Blacklist Check  Browser Status Check 18
  • 19. Implementing an Effective Solution Technical Approaches SURVEY VALIDATION A respondent can be flagged as unengaged in the survey if he or she speeds on at least X% of the pages they saw in the survey.The norms and standard deviations of the times for each page should be calculated in real-time as the page submissions from the respondents are received by the survey platform. It can also be useful to consider the response patterns that are being submitted as another key indicator. Respondents who provide undesirable response patterns on more than X% of pages can also be classified as unengaged for the survey. Good ResponseValidation tools leverage real-time Bayesian statistical models/analysis to determine engagement. 19
  • 20. Implementing an Effective Solution Behavioral Approaches There are three channels to address in order to ensure superior data quality in your study:  Sample Design & Management  Survey Design  Member Management 20
  • 21. Implementing an Effective Solution Behavioral Approaches – Sample Design & Management  Vendor selection is key. Understand how your vendor’s sample is sourced, managed and incentivized.  Ask the tough questions! How is sample outgo balanced? What measures are implemented to ensure the highest quality sample is provided?  Demographic balance  Activity & tenure balance  Survey field time  Invitation/introductory language  Competing survey inventory  Survey frequency & variation  Routing/project prioritization 21
  • 22. Implementing an Effective Solution Behavioral Approaches – Survey Design  Question design is key!  Use non-leading wording  Provide an out for all respondents  Use open-ends sparingly  Avoid yes/no format 22
  • 23. Implementing an Effective Solution Behavioral Approaches – Survey Design  Avoid burdensome question formats (i.e., extensive grids and lists longer than 10-15 attributes).  Strive to keep your survey short and simple.  Clear, concise wording – write for a 5th grader!  Avoid multiple questions on one screen – visual clutter will result in respondent fatigue.  Mobile-compatible and mobile-friendly are two different things! 23
  • 24. Implementing an Effective Solution Behavioral Approaches – Member Management  Trap Questions  Honey Pots  Algorithmic solutions  Tracking activity over time (LOI completions & invalids)  Profiling & third-party data validation sources  Demo consistency checks  Quality exists across a wide spectrum; lifetime management is critical 24
  • 25. Implementing an Effective Solution Behavioral Approaches – Trap Questions Do’s & Don’ts  Not all trap questions are effective! Trap questions shouldn’t be too simple or too complex.  Types:  Instructional (i.e., Select the image which shows a book.)  Skill-based (i.e., 2+2 = ?)  Honesty-based (i.e., What brand(s) are you aware of? What activities have you done in the last 12 months?)  Implement multiple measures to assess quality, never rely on a singular question within the survey to dictate quality.  Be mindful of question position within the survey i.e., adding your trap question at minute 45 will yield false positives that arguably are a result of a lengthy survey NOT a poorly-behaving respondent. 25
  • 26. Implementing an Effective Solution Applying Our Learnings to B2B Research  Know thy sample source!  Always use multiple knowledge-based trap questions (.i.e., looking for experts in cloud-computing? Test their knowledge on various storage products vs. the color of the sky).  Implement multiple measures to assess quality (inclusive of technical and behavioral approaches).  When possible, leverage 3rd party data sources to validate member data.  Never become complacent – your research will always be a hot target for fraud. Stay protected! 26
  • 27. The Path Forward: Responsibility, Accountability, & Collaboration  Every company up and down the supply chain involved in the execution of online research has a role/responsibility as it relates to data quality/fraud detection. What you are responsible for depends on which part of the research process you have operational control over (i.e. you can’t just push responsibility down to the operational layer below you, everyone has to do their part, or the whole system suffers).  There is no silver bullet solution. Effective solutions require a layered technique/approach that incorporates redundancies and failsafe mechanisms.  It’s not enough to simply care about data quality and fraud detection, you must VALUE it! 27