Native advertising is relatively young compared to the paid search and traditional display advertising markets which means that the tools available to marketers to measure the performance of their campaigns is immature. Marketers are often forced to use click-through-rate (CTR) to measure the performance of their campaign. In some scenarios it makes sense to define a conversion goal and measure conversion rate. However, this is not possible in all scenarios.

1. Measuring Engagement
in Native Advertising

2. Author:
Kristopher Kalish
Bidtellect
777 East Atlantic Ave, Suite 312
Delray Beach, FL 33483

3. Introduction//
Native advertising is relatively young compared to the paid
search and traditional display advertising markets which
means that the tools available to marketers to measure
the performance of their campaigns is immature. Market-
ers are often forced to use click-through-rate (CTR) to
measure the performance of their campaign. In some
scenarios it makes sense to define a conversion goal and
measure conversion rate. However, this is not possible in
all scenarios.
Content marketing is a form of marketing involving the
generation and sharing of valuable content and media to
acquire customers. If an advertiser is trying to drive traffic
to their content marketing efforts, there is no clear
conversion goal. Conversion goals are fundamentally tied
to discrete events, that is to say the event either occurs or
does not. However, with content marketing, progress is
more continually measurable. For instance, if an advertiser
drives traffic to their blog and a user spends 1 minute on
the blog, that is great, but if a user spends two minutes on
the blog, that is even better. This phenomenon makes it
difficult to pick a particular time on site as a conversion
goal because for any goal time on site there is always
another goal time that is better. The continuous variable
that marketers are seeking to measure is known as
engagement.
There has been a growing interest in measuring
customer engagement as shown by companies raising
millions of dollars in funding to research engagement in
2014, especially in the native space [2, 3]. There have
been large case studies by Nielsen and Turn to try to
figure out how customers engage with brands and how
it affects their buying decisions [7, 5]. Companies like
89 degrees provide case-by-case analysis of customer
engagement and publish white papers to build their
definition of engagement [1].
Native ad exchanges have been lagging to provide
simplified tools to advertisers for increasing customer
engagement. Bidtellect provides a measurement
capability known as an engagement score to judge the
success of their adver- tiser’s native advertising
campaigns. This enables our exchange optimization
engine to run native advertisements on publishers
which maximize the engagement score. This is a win for
all three parties. The advertiser makes effective use of
their budget, the ads are relevant to the publisher’s site
and the end user is shown messages they react to.
1

4. Related
Work//
Paid search marketers may be familiar with the “Quality
Score” on Google Ad Words and Bing Ads. These methods
are a function of your ad’s performance on the network
and the contextual relevancy of the landing page to the
user’s search query. Relevancy is established by crawling
the landing page and applying proprietary analysis to
assign keywords to it. This makes Quality Score
“exchange-centric” in that it is calculated from the
performance of the campaign on the exchange and
contextual relevancy is established from keywords
originating from the exchange. The engagement score
presented here differs in that it advertiser-centric; It only
captures the performance of your campaign once the
user has been acquired from the exchange.
2

5. The
Engagement
Score//
The engagement score is a number ranging from 0 to 10
where 0 corresponds to the least customer engagement
and 10 represents the highest level of engagement. It is
derived from four variables measured from the behavior of
users once they land on the advertiser’s site. The advertiser
places our JavaScript snippet on their landing page and
any other pages they wish to track. When a user clicks an
ad and lands on their site, this is considered a visit and a
page view. If the user clicks a second page, they have
contributed two page views and one visit. If the user closes
the browser and revisits that is considered another visit. If a
user visits the page and leaves within five seconds, that is
considered one visit, one page view, and one bounce. The
time on site is measured as the difference between the
first event and last event in a visit measured in minutes.
For example if a user clicks an ad, lands on the advertiser’s
page, reads it for 1 minute, then clicks a link to another
article and reads that for 30 seconds before leaving the
time on site for that visit is 1.5 minutes.
The engagement score is a function of the four variables
described above: time on site, page views, number of
visitors, and bounces. For reference the variables are
explained in Table 1.
Metric Abv. Description
Table 1: The four variables used to calculate the engagement score
The engagement score, e is the linear combination of three logistic functions to be:
t
In this work, all three functions ƒ, g, and h are logistic functions of the form:
e = 4 • ƒ ( ) + 2 • g ( ) + 4 • h ( )b
v
p
v
t
v
Time on site
Visits
Page views
Bounces
Total minutes spent on site
Number of browsing sessions
Number of pages loaded
Number of visits lasting less than 5 seconds
Here β controls the “center” of the logistic function, which is where the
function reaches the value 0.5. For example, setting β = 1.0 makes
Equation 2 reach a a value of 0.5 when x = 1.0. The second parameter,
α controls the steepness of the function, which is how quickly the
function transitions from values of zero to one. Larger values of α make
the logistic function behave more linearly near the “center” while smaller
values make the logistic function behave more like a step function.
Figure 1: In red is the function described in Equation (2) with β = 0. The
dashed function has β = −1, while the gray function has β = 1.
ƒ(x)
x
(1)
(2)
1
1+e
ƒ (x)=
x-β
α
v
p
b
3

6. In this work, a bounce is defined to be a visit to a website
that lasts less than 5 seconds. Let b represent the number
of bounces for a site, and v be the number of visits, then
the bounce rate is b/v. The bounce rate is a ubiquitous
metric of website performance that represents which
fraction of website visitors are leaving the website without
consuming any information. A common rule of thumb
among marketers and webmasters is that a bounce rate
below 50% is considered to be good. That implies that at
least have of visitors are engaging with the site. The
RocketFuel team built a histogram of bounce rates from
over 60 web properties and found that the rule of thumb
still holds as only half the sample had a bounce rate below
50% [6].
To make a function of bounce rate that rewards bounce
rates below 50%, we must set α to a negative number to
“flip” the function and encourage low values. We should
also set the center of the logistic function described in
Equation (2) to be at what we would consider a neutral
value. The median from the data presented by RocketFuel
is about 60% so we pick β to be 0.6. That gives the
following function of bounce rate:
The
Bounce Rate
Component//
Equation (3) shown in Figure 2.
1
1+e
(b/v)-0.6
0.08
ƒ( )
b
v
=
Figure 2: The contribution of bounce rate to engagement score as a function of
bounce rate. Bounce rate makes a full contribution to the engagement score
as it approaches zero.
ƒ(b/v)
b/v
0
0
0.2
0.2
0.4
0.4
0.5 0.6
0.6
0.8
0.8
1
1
3.1

7. Figure 3: The contribution of page views per visit to engagement score. It reaches a
maximum as page views per visit reaches > 3.
3.2The
Page
Views
Per Visit
Component//
Another major measurement that marketers and web
masters use to measure the performance of their website
is the number of page views per visit. Using the variables
defined earlier the number of page views per visit is
expressed as p/v. We wish to construct a function so that
when page views per visit is 1, the score is low and when it
reaches 2 the score is much higher. This behavior is
encapsulated in Equation (4).
which looks like:
g(p/v)
p/v
0
0
0.5
0.2
1
0.4
1.5 2
0.6
2.5
0.8
3
1
1
1+e
(p/v)-1.25
0.2
g( )p/v = (4)

8. Average time on site is a website performance metric that
indicates the average length of a users session. If user A
visits a website and stays for two minutes while user B
visits the same site and stays for one minute, the average
time on site is 1.5 minutes. The variable described earlier t
is not the average time on site, but the total time on site.
The average time on site is t/v. Microsoft researchers
determined that time on site is modeled by a Weibull
distribution. Furthermore, 98.5% of the sites sampled were
modeled by a Weibull distribution with shape parameter
k < 1. A Weibull with shape parameter below 1 is used to
model the failure time for parts in which the failure rate
decreases with age - also known as negative aging.
Microsoft researches attribute this to users using a
“screening” approach to web browsing where the user tries
to decide whether to stay or leave as quickly as possible.
The Microsoft researchers looked at the shape parameter
by site category. They found that the education category
has the smallest median shape parameter of 0.65. This
implies that the harshest screening occurs on this
category and that the user is likely to drop off very early on
these sites [4]. The probability density function (PDF) for a
Weibull distribution with shape parameter k = 0.65 and
scale parameter λ = 1 is shown in Figure 4.
It is not possible to be notified when a user leaves a web
page without the possibility of negatively impacting the
user experience. For example, JavaScript could be used to
prevent navigation away from a page until a tracking server
is notified. However, if the server goes down or is latent, the
user will be delayed from navigating to the next page.
Hence, it is ideal to sample the time on site by sending
“heartbeats” in the background. The delay between
hearbeat signals needs to be chosen carefully because
trivial schemes can introduce data anomalies. Consider a
The
Time
on Site
Component//
3.3
Figure 4: The Weibull PDF for education websites [4] as function of t.
p(t)
t
0
0
0.2
1
0.4 0.6
2
0.8 1
3

9. scenario where heartbeats are sent every 10 seconds.
Since most visitors drop off quickly, it would appear that
most users were on the site for 0 seconds. In reality, it
takes these users several seconds to screen the website
and leave.
Intuitively, it is understood that we should send heartbeats
at a fast rate initially and progressively less frequently the
longer a user is on a page. To mathematically formulate
this, we need to make a decision on exactly what is an
acceptable coarseness. In this work we decided that the
sampling frequency should bet done so that an equal part
of the browser population falls between hearbeats. This
prevents any part of the population from exerting more
influence of the time on site variable than another.
In order to break the population into equally sized intervals,
we must con- sider the cumulative distribution function
(CDF) of the Weibull distribution which is presented in
Equation (5).
To send heartbeats that capture equally sized groups we
must send heartbeats at equally spaced quantiles of the
CDF. Let ∆ [0,1] be the quantile size. We send a heartbeat
i at F−1(i∆) seconds from the beginning of a page
view.Picking the quantile size ∆ = 0.01 means approxi-
mately one percent of users will drop off between each
heartbeat. After sampling time on site, we must find a
function of time on site that controls its contribution to the
Figure 5: Average time on site’s contribution to engagement score. Notice
how it quickly climbs after exceeding the performance of most sites in the
Microsoft study by Liu et. al.
Equation (7) is plotted and shown in Figure 5.
t/v
0
0
0.5
0.2
1
0.4
1.5 2
0.6
2.5
0.8
3
1
h()
t
v
By taking the inverse we get:
(5)
(6)
F (x) = 1 − e (x/λ) k−
F = λ(−log(1 − x))(1/k)−1(x)
engagement score. Liu et. al. found that 80% of pages were
modeled by λ ≤ 70 [4]. The scale parameter, λ controls the
midpoint of the cumulative distribution function because
1 − e = 1 − e−1 ≈ 0.63. Hence Liu et. al. found that for
80% of sites, over half the users drop off before they view
the page for 70 seconds. Due to this finding, we choose
70/60 ≈ 1.16 minutes to be the midpoint of our time on
site engagement score component which can be seen in
Equation (7).
h( )
t
v
= (1/(1 + exp((x − 1.16)/ − 0.2)) (7)
−(λ/λ)k

10. This work presented an algorithm for measuring customer
engagement with a single number ranging from 0 to 10. It
described a way to accurately sample time on site based
on the Weibull distribution to control discretization error.
The Bidtellect platform is able to measure which suppliers
contributed to the score and selectively run advertising
campaigns on supply which maximizes the engagement
score. By optimizing for engagement score we satisfy
publishers and advertisers simultaneously. Publishers win
because we only run ads that their users like and engage
with. Advertisers win because the users they receive from
their campaign are interacting with their site.
Conclusion
and Future Work//
4

11. [1] 89 Degrees. Engagement scoring: Are you ready?
89degrees.com, 2013.
[2] Anthony Ha. Real-time analytics startup chartbeat adds data
for native ads,raises $3m more. Tech Crunch, May 2014.
[3] Anthony Ha. Simplereach raises $9m to measure content
marketing and native ads. Tech Crunch, July 2014.
[4] Chao Liu, Ryen W. White, and Susan Dumais. Understanding
web brows- ing behaviors through weibull analysis of dwell time.
In Proceedings of the 33rd International ACM SIGIR Conference on
Research and Development in Information Retrieval, SIGIR ’10,
pages 379–386, New York, NY, USA, 2010. ACM.
[5] Nielsen. 2013 nielsen national cross-media engagement study.
Newspaper Association of America, April 2013.
[6] RocketFuel. What’s the average bounce rate for a website?
http://www.gorocketfuel.com, February 2014.
[7] Turn. The new rules of engagement measuring the power of
social currency. Forbes Insights, 2012.
References//