The document provides an update on ad fraud trends in Q1 2016. Some key points:
- Overall, ad fraud has not improved year-over-year but individual companies are seeing reductions with greater prevention efforts.
- Bots remain the primary driver of all forms of ad fraud and are undermining analytics through polluted data.
- Viewability standards are now widespread but ad blocking has emerged as a new challenge.
- The author recommends moving beyond industry averages and assumptions to analyze individual site data and traffic patterns to better identify fraudulent bot activity.
State of Ad Fraud Ad Blocking Q1 2016 Update Augustine Fou
1. The State of Ad Fraud
Q1 2016 Update
April 2016
Augustine Fou, PhD.
acfou [at] mktsci.com
212. 203 .7239
2. April 2016 / Page 1marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Executive Summary
• Year-over-year, ad fraud has not improved overall; but
individual companies see improvements with greater effort
• Bots remain the key ingredient for all forms of ad fraud and
their actions are polluting analytics, making them unreliable
• Viewability standards are widespread now, but ad blocking
has emerged as the new “crisis du jour”
• Cybercriminals continue to easily avoid or defeat detection
technologies because they cheat and don’t play by any rules
• The proliferation of connected devices and IoT will increase
the capabilities of bots and opportunity for ad fraud
• 4D cybersecurity approaches are needed to fight ad fraud
3. April 2016 / Page 2marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Ad Fraud Background Update
4. April 2016 / Page 3marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Year-over-year fraud estimates continue to vary widely
WhiteOps/DCN (Oct 2015)WhiteOps/ANA (Dec 2015)
PublishersAdvertisers
Average Bots 3%
Range: 2% - 7%
Average Bots 11%
Range: 3% - 37%
“there are industry average bot% and very wide ranges;
but what really matters is the bot activity on YOUR site.”
5. April 2016 / Page 4marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Bad guys continue to focus on where the money is
Impressions
(CPM/CPV)
Clicks
(CPC)
Search
$18.8B
$43 billion
Display
$7.9B
Video
$3.5B
Mobile
$6.2B$6.2B
Leads
(CPL)
Sales
(CPA)
Lead Gen
$2.0B
Other
$5.0B
• classifieds
• sponsorship
• rich media
estimated fraud
not at risk
(86% of digital spend - 2014)
6. April 2016 / Page 5marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Two main types of fraud have the same 2 key ingredients
Impression
(CPM) Fraud
(includes mobile display, video ads)
1. Put up fake websites and
load tons of ads on the pages
Search Click
(CPC) Fraud
(includes mobile search ads)
2. Use bots to repeatedly load
pages to generate fake ad
impressions (hide the true
origins to avoid detection)
1. Put up fake websites and
participate in search networks
2. Use bots to type keywords
to cause search ads to load
and then to click on the ad
to generate the CPC revenue
(screenshots of
fraud sites)
7. April 2016 / Page 6marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Bots are the cause of ALL automated ad fraud
Headless Browsers
Selenium
PhantomJS
Zombie.js
SlimerJS
Mobile Simulators
35 listed
Bots are made from 1) malware
compromised PCs or 2) headless
browsers (no screen) in datacenters.
8. April 2016 / Page 7marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
“Ad fraud and ad blocking are
now related because of bots”
9. April 2016 / Page 8marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Bots of all kinds do all kinds of fraud
10. April 2016 / Page 9marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Different kinds of bots create different kinds of fraud
Malware (on PCs)Botnets (from datacenters)
Toolbars (in-browser)Javascript (on webpages)
11. April 2016 / Page 10marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Bad guys’ (advanced) bots are not on any industry list
10,000
bots observed
in the wild
user-agents.org
bad guys’ bots
3%
Dstillery, Oct 9, 2014_
“findings from two independent third parties,
Integral Ad Science and White Ops”
3.7%
Rocket Fuel, Sep 22, 2014
“Forensiq results confirmed that ... only 3.72% of
impressions categorized as high risk.”
2 - 3%
comScore, Sep 26, 2014
“most campaigns have far less; more in the
2% to 3% range.”
detect based on
industry bot list
“not on any list”
disguised as normal browsers –
Internet Explorer; constantly
adapting to avoid detection
12. April 2016 / Page 11marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Advanced bots need to be detected differently
Confirmed humans
• found page via search
• observed events (mouse click
with coordinates)
“Honest” (declared) bots
• search engine crawlers
• declare user agent honestly
• observed to be 1 – 5% of
websites’ traffic
Fraud bots
• come from data centers
• malware compromised PCs
• deliberately disguised user
agent as normal browsers
13. April 2016 / Page 12marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Any device with a chip and internet can be used as a bot
Traffic cameras turned into
botnet (Engadget, Oct 2015)
mobile devices
webcams
connected
traffic lights
connected cars
thermostat
connected fridge
14. April 2016 / Page 13marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Fraud Activities Mess Up Measurement
15. April 2016 / Page 14marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Bad guys hide fraud by passing fake parameters
Click thru URL
passing fake source
“utm_source=msn”
fake campaign
“utm_campaign=Olay_Search”
http://www.olay.com/skin-care-
products/OlayPro-
X?utm_source=msn&utm_medium=
cpc&utm_campaign=Olay_Search_D
esktop_Category+Interest+Product.P
hrase&utm_term=eye%20cream&ut
m_content=TZsrSzFz_eye%20cream_
p_2990456911
16. April 2016 / Page 15marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Bad guys fake KPIs, trick measurement systems
Bad guys have higher CTR Bad guys have higher viewability
AD
Bad guys stack
ads above the
fold to fake
100% viewability
Good guys have to
array ads on the
page – e.g. lower
average viewability.
17. April 2016 / Page 16marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Bad guys’ bots earn more money, more efficiently
Cash out sitesCookie collecting
Source: DataXu/DoubleVerify Webinar, April 2015
100 1x1
iFrames
18. April 2016 / Page 17marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Bad guys’ bots can fake quantity and quality metrics
click on links
load webpages tune bounce rate
tune pages/visit
19. April 2016 / Page 18marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Turn off 1 ad network to cut out 90% bot traffic (red)
• Bot volumes are flat across (red line at top; green volume bars at
bottom)
• Human traffic patterns are messy and irregular (blue line at top)
• Dark red area decreased after we turned off a single bad ad network
• Share blacklists through central data repository to increase impact
20. April 2016 / Page 19marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Google analytics view of traffic from single fraud domain
Despite losing all of the traffic from these fake/fraud sites, there was no
change to the number of pledges, during the same period of time.
102,231 sessions
0 sessions
goal events – no change
“ … because bots don’t make donations!”
21. April 2016 / Page 20marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
AppNexus example – cleaned up 92% of impressions
Increased CPM prices
by 800%
Decreased impression
volume by 92%
“AppNexus aggressively detected and discarded fraudulent
inventory; good for them and good for those who buy from
them. This should be an example for publishers.”
Source: http://adexchanger.com/ad-exchange-news/6-months-after-fraud-cleanup-appnexus-shares-effect-on-its-exchange/
22. April 2016 / Page 21marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
What normal (human) traffic patterns look like (5 + 2)
• Notice the normal 5+2 pattern of weekday versus weekend traffic
• Human traffic is lowest between 2a – 5a , because humans sleep
• Bot volumes are now seen to be roughly tuned to human pattern
5 weekdays 5 weekdays
(lower volume on 2 weekend days)
23. April 2016 / Page 22marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
End of month traffic and impressions fulfillment
Traffic surge
end of February
Impressions
surge on
Feb 29th
24. April 2016 / Page 23marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Bad acting mobile apps and .xyz domains (abnormal CTRs)
.xyz domainssuspicious mobile apps
25. April 2016 / Page 24marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Connection with Ad Blocking
26. April 2016 / Page 25marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Ad blocking impacts both CPM and CPC ads
Ad blocking ONNo ad blocking
Search Ads
Display Ads
27. April 2016 / Page 26marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Humans use adblock, fraud bots don’t block ads
Publisher 1
“We have long used ad block detection as a signal
for human users because humans use adblock.”
“Fraud bots that want the ads to load don’t block ads.”
Note that we focus on the % of ads that successfully loaded, not blocked.
80 – 85% loaded is a good rule of thumb
Publisher 2
28. April 2016 / Page 27marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
University websites (no ads)
University 1
University 3
University 2
29. April 2016 / Page 28marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Tech and Gaming (tech savvy users)
Site 1
Site 3
Site 2
30. April 2016 / Page 29marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Ad supported content sites see highest ad blocking rates
Publisher 1
Publisher 3
Publisher 2
31. April 2016 / Page 30marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Recommended Actions
32. April 2016 / Page 31marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Debunk ALL assumptions and discard all averages
• Ad Fraud / Bots
• Ad Blocking
• Measurement
• Bots are low; list-based detection catches very little
• Bots are malware on PCs; more and more from data centers
• Bots are simple (cant fake clicks, mouse, scrolling); they do
• Ad blocking is lower in mobile; mobile is less measurable
• Ad blocking is higher among younger users; not necessarily
• Ad blocking is still in the single digits; look at “ads served”
• Higher CTRs are better; not if the clicks are fake
• Higher viewability is better; not if the sites fake viewability
• Data is reliable; not so if you don’t know bot/blocking levels
33. April 2016 / Page 32marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
About the Author
34. April 2016 / Page 33marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Dr. Augustine Fou – Recognized Expert on Ad Fraud
2013
2014
SPEAKING ENGAGEMENTS / PANELS
4A’s Webinar on Ad Fraud
Digital Ad Fraud Podcast
Programmatic Ad Fraud Webinar
AdCouncil Webinar on Ad Fraud
TelX Marketplace Live
ARF Audience Measurement
IAB Webinar on Ad Fraud / Botnets
Publishers Forum
2015
35. April 2016 / Page 34marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
Harvard Business Review – October 2015
Excerpt:
Hunting the Bots
Fou, a prodigy who earned a Ph.D. from MIT at
23, belongs to the generation that witnessed
the rise of digital marketers, having crafted his
trade at American Express, one of the most
successful American consumer brands, and at
Omnicom, one of the largest global advertising
agencies. Eventually stepping away from
corporate life, Fou started his own practice,
focusing on digital marketing fraud
investigation.
Fou’s experiment proved that fake traffic is
unproductive traffic. The fake visitors inflated
the traffic statistics but contributed nothing to
conversions, which stayed steady even after the
traffic plummeted (bottom chart). Fake traffic is
generated by “bad-guy bots.” A bot is computer
code that runs automated tasks.