Metrics and Analytics for new media

New Media Measurement
sean zehnder socialesque 09

Wednesday, April 29, 2009 1

Outline
old-school Media
New-School Interactive
The Interactive Revolution
(How) are video games different?
New‐School Measurement

www.socialesque.com info@socialesque.com www.socialesque.com/socialesque-blog


old-school Media

How do they size... Why do they size...

Newspaper Audiences
Radio Audiences
TV Audiences

What is the smallest unit of an ‘audience’ in each?

What does each metric imply about the behavior of the
audience member -- meaning what are they doing?



old-school Media
May 1996 May 1997 Oct 1997 June 1998

Portals Virtual Community New Measurement New Busines Opportunities

June 1999 July 1999 Sept 1999 Aug 1999

New Skills in Demand Disruptive Technologies Old Industries In Flux Dreaming of Mobile



old-school Media

Web 1.0
Most user’s didn’t have broadband
Flash existed, but it was mostly for animations
There was dynamic content, but it required a lot of
custom scripting and code to do relatively simple things
Discourse around ofﬂine vs. online



old-school Media
Web 1.0 Measurement
Impressions
Clicks
(later) Time on Screen
Google Analytics “breakthrough” -- ﬁnally it is easy to
gather and chart basic stats
Data was primarily HTTP/Referrer, Cookie, and
JavaScript based



old-school Media

page 61 of July 5, 1993 issue of The New Yorker, (Vol.69 (LXIX) no. 20)



April 2001 March 2004 Feb 2005 Nov 2005

Dealing w/ Massive Scale Wisdom of Crowds Open Source Mobile Revolution

Dec 2006 July 2006 April 2006 Aug 2006

Social Networks Emerge Games Grow Up Cult of Personality
User-Generated Content



Let’s say you have some data in an newspaper...

“The Chronicle” 1.0 Unit Metric
Ofﬂine Newspaper

AVERAGE RAINFALL (INCHES/MONTH)

A Sale # of Sales
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
San Fran 4.35 3.17 3.06 1.37 0.19 0.11 0.03 0.05 0.20 1.22 2.86 3.09
Seattle 5.35 4.03 3.77 2.51 1.84 1.59 0.85 1.22 1.94 3.25 5.65 6.00
Chicago 1.53 1.36 2.69 3.64 3.32 3.78 3.66 4.22 3.82 2.41 2.92 2.47
New York 3.17 3.02 3.59 3.90 3.80 3.65 3.80 3.41 3.30 2.88 3.65 3.42
Total Subscriptions
A Subscription
Miami 2.01 2.08 2.39 2.85 6.21 9.33 5.70 7.58 7.63 5.64 2.66 1.83
Total New Subscriptions

You might consider sprucing it up to make it more readable...

“The Chronicle” 2.0
Ofﬂine Newspaper

Rationale Costs ROI
Average Rainfall (inches/month)
Jan Feb Mar Apr May June July Aug Sep Oct Nov Dec

More Sales
San Francisco 3.17 3.06 1.37 0.19 0.03 0.06 0.05 0.20 1.22 2.86 3.09
4.35
You believe readability
Seattle 5.35 4.03 3.77 2.51 1.84 1.59 0.85 1.22 1.94 3.25 5.65 6.00
increases the attractiveness of Layout & Graphic
Chicago 1.53 1.36 2.69 3.64 3.32 3.78 3.66 4.22 3.82 2.41 2.92 2.47
your paper, and attractiveness Design
New York 3.17 3.02 3.59 3.90 3.80 3.65 3.80 3.41 3.30 2.88 3.65 4.42
More New
will drive sales.
Miami 2.01 2.08 2.39 2.85 6.21 9.33 5.70 7.58 7.63 5.64 2.66 1.83
Subscriptions



A new online paper opens in your market... Unit Metric

theHerald.com 1.0 A Sale # of Sales
Online Newspaper

A Subscription Total Subscriptions, Total New Subscriptions
Average Rainfall (inches/month)
4+ 2-3 1-2 <1
An Impression Time On-Screen , # of Impressions

San Francisco
Rationale Costs ROI
Seattle

Layout & Graphic
Chicago
They can reach a new audience (e.g. Did we reach a
Design
New York City
a younger one that gets their info younger demographic?
Miami
online).
Can we learn more
Website creation
They hope to learn more about their about our readers
‘readers’ by using new metrics. now?
Website hosting &
support
So you update your own look-and-feel...
3.0
“The Chronicle”
Ofﬂine Newspaper

Rationale Costs ROI

Layout & Graphic
Did we reach a younger
Design
You believe your readership is more demographic?
interested in getting a sense of how
the environment is changing, rather Are we keeping our
Updating your
than using raw rainfall data. subscribers or losing them
printing press to to the competitor?
do color graphics

Your notion of an the ‘average reader’ demographic has changed.
Note Your notion of what the ‘average reader’ is DOING has also changed.



So your competitor decides to make their graphic interactive...

theHerald.com 2.0 Unit Metric
Online Newspaper

A Sale # of Sales
Average Rainfall (inches/month) Choose cities...

A Subscription Total Subscriptions, Total New Subscriptions

An Impression Time On-Screen , # of Impressions
Seattle
Chicago
A Click Most Popular Cities
New York
A Drag Most Popular Months
San Francisco

An Error Number of times the city was changed

A Behavior (generally) Number of Times Shared w/ a Friend
Miami
An Event (generally) Number of Times the Server Crashed

Rationale Costs ROI

Can we track how often they share
Layout & Graphic this page with a friend?
Design
They believe that more interactive content is
more engaging for their ‘readers.’ Can we track how long they
typically spend with the content?
Flash Designer /
They believe that users will share the link to this
Scripter
website with friends if they ﬁnd the content really Can we tell if they spend more time
cool (or fun, or useful, etc.). with this content than they did
DB Engineer
reading our static table (from
Hosting & Support previous iteration)?

Take-Away: Behind Every Good Metric is a Model of Behavior


More Reasons to Measure

I measure... In order to...

Time

Assess Effectiveness of site on gaining and holding user’s attention
Impressions

System Behavior Save money. Stop bad behavior. Keep system out of user’s way.

User Behavior Ensure they are having good experience. Identify new business opportunities.



1. Games are *highly* interactive 2. Games involve simulations

Dynamic Interactive
Static Systems Model System Display
Systems Systems

3. Game Simulations enable Designed Experiences

Mechanics Dynamics Aesthetics

Mechanics - logic and data that makes the system work (e.g. code)

Dynamics - the rules of the game world (e.g. physics)

Aesthetics - the game as experienced/perceived by the user



4. Games require mastery

Perceives the Plans Some
Builds a Acts in the
The Player (Agent)... World Action
Theory World

Refines their idea of the world...

5. Since key behaviors differ across games, standardization can be extremely difficult.

Example: “Dead Reckoning”

An expert behavior common in an FPS where a
player moves in an arc relative to a target such that
it makes it harder for the target to hit the player
while the player is still able to fire on the target.



5. Many behaviors are emergent from this complex system.

For one, artiﬁcial intelligence is an increasingly
important aspect of video games. It is being
used to improve the ‘performances’ of the non-
player characters, and to simulate complex
dynamics in real-time such as crowd behavior.

Also, massively multiplayer games have
dramatically increased the SCALE of data
needing to be measured and processed.

In other words, behavior is emergent.



Eve Online



New-School Measurement

Will Wright -- Creator of The Sims franchise

“With Sims Online, we’re trying to take a lot of the
community dynamics that we’ve learned from The Sims
off-line and reinterpret them in an online world. And we
study the online community all the time. It’s a very
interesting community -- it’s over half female, which for
an online game is totally different. And it turns out that a
community of 55% females behaves very differently than
a community that’s 75% males...In fact we’re capturing
very detailed information. I can tell you how many people
are kissing more today than they were yesterday, or how
that’s correlated to other things.”
(From Laurel, B. 2003, p. 255).




Measurement at Massive Scale
Metrics that can’t often be standardized
New ways of thinking about engagement
New reasons to measure
Cross-Media Measurement



Socialesque’s “Rules of Engagement”

analyze
model instrument reﬁne

What do you expect will be What might fun look like in your Did people do what you What can be done to improve the target
engaging about your application? game? expected? behaviors?

What might frustrate users? What might frustration look like? Were there things that you notice Can the model of engagement be reﬁned
in the data that you didn’t think to based on the new understanding?
What do you hope they do while What measures might capture include in the model?
they use the application? engagement? Which items and features were most
Which measures are the strongest popular, and might some features need to
What is practical to measure in your model? be removed or improved to enhance
from this application? engagement?



Case Study: Attrition in Social Network
“A friend leaves the network”
model

expectations basic model
Social measures will be most
inﬂuential.
Amount of Social
Number of Friends Quality of Friendship
Interaction

+ +
+
The effect will be cumulative:
the more friends you have Game Satisfaction &
Enjoyment

lost, the more each loss is felt.
-

Will have a greater impact with Likelihood of Attrition

a closer friend.




model
Examination of Context
Aggregate User Behavior Social Network
SESSIONS
Number of Friends
How often do people play?
Amount of Interaction
Quality of Friendship
How long do people play?
Quality of Interaction
Network Density
What does a play session look like?
Network Centralization
USERS
Network Homogeneity
Financial Investment
In this example, the user behaviors are
seen as proxies of social network qualities,
Brand Loyalty or Favour
since we do not have direct measures for
those variables.
Time Investment




Statistical analyze

Filter Query
Analysis
if not in avatars Distributed
table Processing using
‘cloud’
computing
If they don’t have
friends
11 node Cluster

If they never 10 hour query
whispered reduced to 30
seconds

random reduced Survival 95% Confidence

sample processing time
Std. Error
Time Interval

Mean 76.32 0.69 74.97, 77.66
by ~100X



losing a friend *does* increase the analyze

however
chance that a user will leave.

this effect can be reduced when users
have strong social networks and
communicate frequently.
Regression Analysis of Social Factors in Game Attrition

Predictor B Std. Error Wald Sig. Exp(B)

Model 1

Friendship Size –.034 .004 64.381 .000 .966

Cohesiveness –.010 .005 5.229 .022 .990

Outgoing Whispers –.001 .001 5.269 .022 .999

Model 2

Friendship Size –.035 .004 66.599 .000 .966

Cohesiveness –.001 .001 4.604 .032 .999

Incoming Whispers –.010 .005 5.208 .022 .990




Was the data timely enough to effect change? refine

If users aren’t doing what you expected, should you
general be trying to redirect them or reward their actual
behavior?

After implementing a fix, did you observe the desired
outcome?

Additional user-level variables would improve
analysis: demographic, account creation/
cancellation, etc.

It is possible, from this type of model, to implement
Specific a monitoring system to alert Disney of high-risk
clusters or even individual users.

The content of the messages can also be an
important source of information about the network.



Case Study: Avenue Q Facebook Application
analyze



Case Study: Navigation in an FPS
analyze

Animated heatmap-style visualization

Contact Sheet for Frame-by-Frame Study

Useful for level balancing & helping users ﬁnd their way



Case Study: Navigation + Survey Data in an FPS
analyze

Pearson Correlations: Navigational Behaviors and Player Reactions
# Cells Visited Freq. of map use .188*
Max Time For A Cell Frustration .208*
Distance Traveled Understanding of Map .194*
# Cells Visited Understanding of Map .316**
Speed Understanding of Map .263**
# Cells Visited Frustration -.202*
Max Time For A Cell Understanding of Map -.311**
Dead Reckoning Count Intensity -.191*
**. Correlation is signiﬁcant at the 0.01 level (2-tailed)
*. Correlation is signiﬁcant at the 0.05 level (2-tailed)



Metrics and Analytics for new media

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (14)

Similar to Metrics and Analytics for new media

Similar to Metrics and Analytics for new media (20)

Recently uploaded

Recently uploaded (20)

Metrics and Analytics for new media