Paper from ESOMAR APAC with Luma Research.

1. Mobile research goes to the game
In the scrimmage moment with mobile?
James Burge, Managing Director, Asia-Pacific, Research Now
Sally Joubert, CEO, Luma Research
Georgia Phillips, Client Service Director, Luma Research
Introduction
The rapid advances in technology over the past decade have changed the way we interact.
Whether it is on a personal level or the interaction between brands and consumers,
technology has opened up so many more opportunities to reach out to each other.
One of the major advances has been the rise in the use and the capability of mobile phones
and this is particularly evident in the Asia Pacific region. APAC has the highest percentage of
smartphones globally (76%)1
and more APAC users turn on their mobile device throughout
the day than the global norm2
. And this is on the rise with two-thirds (68%) of APAC users
agree that compared to 12 months ago, they now use their mobile phone for more activities
other than just making calls and texting. Again this is above the global average of 63%3
.
These technological advances have had a major impact on the environment for brands.
Advertising is everywhere from traditional mass market TV, sport sponsorships, mobile,
digital, to fully immersed brand experiences. As consumers we have even shorter attention
spans so the competition to really engage is becoming increasingly more difficult. Brands
have so many different channels they can use but how do we know what will work best?
Businesses are seeking to answer this in order to facilitate advertising and media spend
decisions. Importantly they need to know this quickly and ahead of their competitors. Gone
are the days when businesses could afford to spend half of the research budget on long,
cumbersome tracking studies which might yield some interesting findings on a small number
of the channels weeks after the event.
2
http://www.sap.com/bin/sapcom/downloadasset.The%20Mobile%20Consumer%20In
sights%20on%20Asia%20Pacific%20Trends%20Impacting%20Mobile%20Momentu
m%20and%20Customer%20Engagement-pdf.bypassReg.html
3
http://www.sap.com/bin/sapcom/downloadasset.The%20Mobile%20Consumer%20In
sights%20on%20Asia%20Pacific%20Trends%20Impacting%20Mobile%20Momentu
m%20and%20Customer%20Engagement-pdf.bypassReg.html

2. A few years ago Luma and Research Now saw this as an opportunity for mobile research and
started investigating what it could offer in this space. Luma listened to the concerns of their
clients and is re-inventing advertising research to meet the needs of business managers in
this new environment. Research Now has lead the way in mobile research technology by
building and perfecting a best in class mobile platform, enabling researchers and advertisers
to better tap into the mobile audience.
This paper shares the story of our mobile research evolution from the early stages and
challenges through to the current state of play and opportunities. We begin by summarizing
our initial early trials and then focus on how we have applied and what we have learnt in
two highly competitive advertising spaces. The first is the State of Origin Rugby Series in
Australia and the second is the world’s most expensive advertising platform the Super Bowl
2014 in USA. We hope our findings spark questions and ideas within the industry about how
mobile research can deliver richer and more relevant insights for the future.
Early Testing
In November 2011, Luma and Research Now started parallel tests to determine whether the
online ad testing methodology add+impact® could be successfully adapted to a mobile
platform. The early testing explored the feasibility, practicality, reliability and value of mobile
ad testing to find out whether the data collection is as effective and reliable on mobile as it
is using an online format. The testing compared the collection of key advertising research
effectiveness measures (Attention to the creative and Bonding to the brand) across the two
platforms in both Australian and the UK4
. This initial study used static advertising and found
remarkable consistency across all the key Brand Bonding measures which evaluate how
people feel about the brand after exposure to the advertising campaign (see Figure 1). The
response and participation rates in this initial study showed that mobile was a great way to
reach people during a variety of activities through the day and night - 56% of the mobile
surveys were done out of working hours compared to 21% online.
Figure 1 – Brand Bonding Measures Across Platform
4
Research Goes Mobile – Findings from Initial Smartphone Applications Research C. Dubreuil and S Joubert
ESOMAR 3D Digital Dimensions Amsterdam 2012

3. The Attention measures which evaluate how people feel about the creative itself revealed
some differences in the level of cut-through for the static ads across the two platforms. At
the time it was concluded that this could have been due to the different screen sizes, the
potential distractions from being ‘on the go’ as opposed to sitting at a computer screen or
simply a result of the type of ad itself. To investigate this further we completed a second
round of testing in 2012 using video advertising for Kleenex Toilet Tissue and McDonalds.
This next round showed consistency across both key ad effectiveness measures suggesting
that the differences in the initial tests related to the type of ad rather than the platform
itself. See figure 2 below.
Figure 2 Attention and Bonding Scores (Phase Two)
These results were very encouraging and suggested that utlilising smartphone as a data
collection platform does not have a direct impact on key advertising effectiveness measures
and can therefore be used alongside traditional online surveys. This opens up enormous
opportunity to capitalize on the advantages of mobile as a research tool whilst still ensuring
continuity with traditional measures. Potentially mobile offers more cost-effective
advertising research because you can reach people more quickly and easily.
Next we wanted to explore how to make it even better.

4. Going to the Game…
To do this we have put mobile during two major sporting events where people are typically
hard to reach yet the need for quicker, reliable feedback is strong. Global Sponsorship
investment is significant (53.2 billion US dollars in 20135
), as are the opportunities, and
businesses need to know quickly whether it is worth it and how to make it better.
In 2013 in Australia we chose the State of Origin Rugby Series in Australia – it is the biggest,
most watched Australia sports series.
As researchers we wanted to see whether mobile could deliver this and importantly what it
could offer over traditional pre-post online studies. Specifically:
• Could we really interview avid sports fans in the moment?
• What could mobile add to traditional research?
• Does mobile improve the survey experience?
State of Origin 2013 - Australia
Method
Luma and Research Now undertook a five-part study to conduct our research into mobile
sponsorship research in August 2013. We utilised a traditional pre and post online survey
and a short mobile survey during each game to track any changes in brand feelings and
perceptions. We recruited N=431 respondents who were considering watching or attending
the State of Origin series. The composition of the sample surveyed remained reasonably
consistent across the three games in terms of location, age, team supporting and passion for
Rugby League, meaning there was very minimal sample skew when we moved from the
traditional PC-based survey to the mobile survey.
What we found
Interviewing in the Moment
Our study confirmed that we can interview “in the moment” with even the most avid
sporting fans. Participation rates and respondent engagement were high. 69% who
undertook the initial online survey participated in at least one of the mobile surveys. This is
higher than we would expect for a standard repeat survey.
What was even more encouraging is that 69% of respondents who participated in Game One
also participated in Game Two and 76% in Game Three.
5
http://www.statista.com/statistics/196898/global-sponsorship-spending-by-region-
since-2009/

5. The next interesting finding related to speed. Across the three games an average of over
60% of interviews were completed by the end of the first half of the game which again is
faster than traditional online surveys.
We were also encouraged by the level and depth of these mobile respondents’ engagement
– 38% uploaded a photo in Game One and 34% in Game Two. Given that these were avid
fans watching the game they love, this level of participation was astonishing. Figure 3 shows
the variety and quality of the pictures we received:
Figure 3
A Cross Section of Photos Uploaded for the State of Origin Mobile Survey
Source: State of Origin 2013, Mobile Research Goes to the Game, Research Now & Luma Research 2013
What Can Mobile add to traditional research?
The results from the traditional pre-post surveys on their own provided very little clue as to
how fans were feeling about the brands and whether or not the sponsorship drove a positive
return on investment (see Figure 4).

6. Figure 4
Brand Feelings Pre and Post State of Origin
Source: State of Origin 2013, Mobile Research Goes to the Game, Research Now & Luma Research 2013
With these results alone you could have been excused for thinking that sponsorship does
little for the brands, but this was not the whole story. While this traditional pre-post
approach was useful in identifying differences in feelings toward the sponsors, for example
our strong rugby league fans were much more positive towards all the State of Origin
sponsors than the non-fans, we wanted to know more. What else was happening? Were we
missing something?
The answer was “yes”. Whilst we saw an overall decline in brand feelings when comparing
pre and post results, the mobile game data provided additional insight that would have been
missed with the standard approach.
For instance, we found that feelings toward Holden remained consistent before, during and
after the series. However, both GIO and AAMI Insurers were generally lower during play
itself (See Figure 5). Our ‘QLD’ Rugby team supporters loved XXXX so much more than the
‘NSW’ supporters during Games 2 & 3, notably when they were winning (See Figure 6).
The level of initial data that mobile was able to deliver opened up a wealth of valuable
further research that could be explored in any manner of ways to ascertain the exact
reasons for these significant findings. For us the key takeout simply put, was we would not
have known any of this if we had just stuck the traditional survey method.
Figure 5
Brand Feelings Pre, During and after the Games

7. Source: State of Origin 2013, Mobile Research Goes to the Game, Research Now & Luma Research 2013
Figure 6
XXXX Brand Feelings by Team Supporters
Source: State of Origin 2013, Mobile Research Goes to the Game, Research Now & Luma Research 2013
Team NSW Team QLD
Pre-game
Game 1
Game 2
Game 3

8. Could Mobile add to the respondents’ experience?
We know that shorter and more engaging surveys are going to win over the hearts and
minds of respondents. So next we looked at whether we could replace wordy questions by
utlizing the mobile phones other features like photo uploading. Specifically we looked at
whether common ‘feelings’ questions (i.e. Which of the following words best describes how
you feel about this State of Origin game?...) with a simple image analysis from the photo
uploads. We were interested in this line of questioning as traditionally they have been fairly
limited. We know that people often cannot express how they feel or their answers to
questions like these reflect only a considered and rational response.
Not surprisingly we found that our rugby fans generally claimed they were happy, excited
and positive. When we scanned the uploaded images we came to the same conclusions by
simply and quickly observing the images (See Figure 7). Given the willingness to participate
in this survey and to upload photos this approach is certainly worth considering.
Figure 7
Feelings About the Game versus Image Analysis
Source: State of Origin 2013, Mobile Research Goes to the Game, Research Now & Luma Research 2013
The video uploads also added another layer of analysis and depth to the findings. We asked
respondents to upload a video of themselves talking about sponsorship. The depth and
diversity of the clips was incredible. One of the key themes that emerged was the desire to
find a sponsor who promoted healthy eating. Many felt that alcohol sponsors would be
better balanced or replaced with brands that linked to sport and health, be it healthy food,
exercise services or sport itself. Even those drinking a beer shared this same sentiment!

9. Overall our State of Origin Study showed us that mobile can help us deliver not only faster
and potentially cheaper but also better research:
• You can interview avid sports fans ‘in the moment’
• Mobile can add speed and depth when compared to a traditional approach
• Mobile (and all it’s features) can potentially add to respondents positive experience
and help to reduce survey length by asking less questions
Super Bowl 2014 - USA
This year we took this to the next level to see whether mobile could actually replace some of
the research we were doing? We chose the Super Bowl in the US as it is arguably the most
sort after advertising space being watched by over 111 million on TV and over 500,000
online. Here the stakes are high with marketers shelling out over $4million for a 30-second
ad and this is just the starting point. The majority take a multi-channel approach to
maximize their investment. Some ads are pre-released, some use teasers to build
expectation and others used crowd sourcing and pre-game competitions to maximize
engagement. Not surprisingly this event attracts the interest of the global advertising and
research industries and as a result the ads are possibly the most researched/talked about
ads in the world.
Given their Super Bowl 2014 investment advertisers want to know whether their ads are
effective and how they can drive post game viewing. We all know about the amazing Super
Bowl advertising success stories that have become almost folklore – the apple ad in 1984…
but what about the rest? Dozens of polls have tried to track the best and worst performers
and interestingly there are some ads which feature on all the tables and others that don’t.
We find Super Bowl advertising fascinating and want to find out what mobile can offer in this
space. Specifically:
• Do the findings from Rugby in Australia translate to Grid Iron in the US i.e. Can we
really interview avid sports fans in the moment?
• Can we use mobile to do smarter advertising research i.e. can we use quick in the
moment mobile surveys to replace some of the cumbersome tracking questions?
Method
Our Super Bowl 2014 Study had two parts, one mobile and one online. Stage One was a
short three minute mobile survey designed to capture ‘in the moment’ feelings about the
Super Bowl ads (cut-through, memorability and sharing). We pre-recruited fans who were
planning on watching the game and were willing to participate in a short mobile survey. Just
as the game finished they were sent the mobile survey and within the two hour window that
followed we had received n=500 completes. The majority of the sample were keen sports
viewers (47%) or occasional sports viewers (39%).

10. Using the results from Stage One we developed a follow up tracking study to measure the
effectiveness of 50 of the ads and to look at 20 of the most polarizing ads in detail (Stage
Two). A nationally representative sample of n=1000 sourced from the Research Now panel
completed the twenty minute online survey. The questionnaire covered traditional
advertising research measures recognition and brand link and key add+impact® creative
effectiveness measures.
What we found
Interviewing in the Moment
This study again confirmed that mobile gets you to the game. We achieved a 45% response
rate in the two hour window after the game which, although based on a pre-recruited
sample in this example, is significantly quicker than the typical 24 hour window we would
see for online. While the survey timing was different to the State of Origin it was
encouraging to see that fans are willing to participate in mobile research when they are ‘in
the moment’.
Stage One – Our Super Quick Super Bowl Survey
With only a two hour window it was amazing how quickly we obtained the ‘in the moment’
feelings about the ads. By Monday morning we had our own Super Quick results to over 50
ads. Firstly we could see which ads were generating the strongest positive feelings (See
Figure 8). We know from our previous work that brand feelings are the strongest predictor
of purchase intent7
so we are calling this our Top 10.
Figure 8
Super Bowl Top 10
Luma’s Super Quick Super Bowl Top 10
1 Budweiser ‘Puppy Love’
2 Budweiser ‘Hero’
3 Coke America ‘America the Beautiful’
4 Doritos ‘Time Machine’
5 Cheerios ‘Gracie’
6 Radio Shack ‘Phone Call’
7 Chrysler ‘America’s Import’
8 Wonderful Pistachios ‘Wonderful Pistachios’
9 Bud Light ‘Epic Night’
10 Dannon Oikos ‘Spill’
Source: Luma Research Now Mobile Study Feb 2014 Which ad made you feel the best?
7
Cramphorn, M and Phillips, G Global Advertising that Sells. ESOMAR Congress Cannes 2005

11. We also collected consumer predictions of the ads that would be most remembered and
most likely to be shared (see Figure 9 and 10)
Figure 9
Consumer Predictions of the Most Memorable Super Bowl Ads
Consumer Predictions of the
Most Memorable Super Bowl Ads
1 Budweiser ‘Puppy Love’
2 Budweiser ‘Hero’
3 Bud Light ‘Epic Night’
4 Audi ‘Doberhuahua’
5 Doritos ‘Time Machine’
6 Coke ‘America the Beautiful’
7 VW ‘Wings’
8 Chrysler ‘America’s Import’
9 Toyota ‘Epic Night’
10 Wonderful Pistachios ‘Wonderful Pistachios’
Source: Luma Research Now Mobile Study Feb 2014
Figure 10
Consumer Predictions of the Most Shared Super Bowl Ads
Consumer Predictions of the
Most Shared Super Bowl Ads
1 Budweiser ‘Puppy Love’
2 Budweiser ‘Epic Night’
3 Audi ‘Doberhuahua’
4 Doritos ‘Time Machine’
5 Budweiser ‘Hero’
6 Chrysler ‘America’s Import’
7 VW ‘Wings’
8 Coke America ‘America the Beautiful’
9 Radio Shack ‘Phone call’
10 Wonderful Pistachios ‘Wonderful Pistachios’
Source: Luma Research Now Mobile Study Feb 2014

12. From our short three minute mobile survey we found out a lot about the ads and here are a
few interesting findings:
• Budweiser “Puppy Love” was the overall “winner”. It was the strongest ad in terms
of generating the most positive feelings and consumers predicted it would be the
most memorable and shared ad.
• Budweiser “Hero” ad was also a strong performer featuring second on Luma’s Top
Ten and also predicted to be remembered and to a less extent shared.
• The third Budweiser ad “Epic Night” for Bud Light didn’t perform in quite the same
way – it didn’t feature on the Luma’s Top 10 but it was predicted to be remembered
and shared.
• Interestingly, the ads for Cheerios, Toyota and Dannon Oikos also appeared to work
in a different way – generating strong feelings but not necessarily predicted to be
remembered or shared.
And the list goes on but the main point here is that we were able to capture a lot of
feedback in a short space of time. However, it was important for us to know whether these
findings are useful and would predict what eventually happened.
To do this we looked at how the findings from our mobile study compared to the full
detailed online study we conducted post the Super Bowl and to other published
polls/surveys conducted by others. Specifically;
• How does our Top Ten compare to others?
• Could our mobile audience predict what ads were remembered?
• Did our audience predictions of sharing actually play out in market?
How Does Our Top Ten Perform relative to others?
Our initial results are encouraging. With our one simple question we predicted:
• Four out of the top five of the Super Bowl Ad Meter Results (published in USA Today)
which are based on thousands of voters. This Ad Meter is regarded as the ‘holy grail’
of Super Bowl Polls – many KPIs are won or lost on this basis of this list alone.
• Five out of the top six of the Super Bowl add+impact® Commitment Results. This is
the key effectiveness score from the add+impact® ad test which is used in over 50
countries globally. The Commitment score is calculated from two key measures
attention to the ad and bonding to the brand and is continually validated with in
market performance.

13. Figure 11
Ad Meter and add+impact® Top Ads Tables
Could our mobile audience predict what ads were remembered?
At this stage it looks like this is not as simple and may not be something that is as easily
predicted.
• We found almost no relationship between our audience predictions and the
recognition scores we obtained from the large scale follow up tracker See Figure 12

14. Figure 12
add+impact® Recognition versus Predicted Top Ten
Consumer Predictions of the
Most Memorable Super Bowl
Ads
Recognition from follow up
online survey
1 Budweiser ‘Puppy Love’ Puppy Love 81%
2 Budweiser ‘Hero’ Geico ‘Pig’ 79%
3 Bud Light ‘Epic Night’ Cheerios ‘Gracie’ 74%
4 Audi ‘Doberhuahua’ Go Daddy ‘Body Builder’ 67%
5 Doritos ‘Time Machine’ VW ‘Wings’ 64%
6 Coke ‘America the Beautiful’ H&M ‘David Beckham’ 64%
7 VW ‘Wings’ Honda ‘Hugfest’ 63%
8 Chrysler ‘America’s Import’ Subway ‘Crunch Time’ 61%
9 Doritos ‘Cowboy’ Doritos ‘Cowboy’ 60%
10 Wonderful Pistachios Toyota ‘Muppets’ 57%
However, it is worth noting that there was actually very little relationship between
Recognition and the Super Bowl Ad meter score or our other measures of ad effectiveness.
Did our audience predictions of sharing play out in market?
At the time of writing this paper it looks like this may also not be as straightforward and we
are still investigating to see whether consumers can predict this:
• According to Unruly Media the top most shared ads of the Super Bowl 2014 were:
o Budweiser ‘Puppy Love’ 1,309,403 shares
o Budweiser ‘Hero’ 202,556 shares
o Axe ‘Make Love, Not War’ 136,282 shares
o Jaguar ‘British Villians’ 125, 896 shares
o Bud Light ‘Epic Night’ 123,418 shares
While consumers predicted the Budweiser ads, AXE and Jaguar were not predicted from our
quick mobile survey.

15. To sum up….
Thank you for listening to our story. We recognize that this paper is not conclusive and there
are many more things we can explore in terms of the impact of mobile on market research.
However, we hope these examples inspire you with the doors mobile can open. With
sufficient groundwork on the technological capabilities and smartphone penetration, mobile
data collection offers a relevant and convenient means of participation for respondents.
Furthermore, the affinity people feel with their mobile device demonstrates a clear
opportunity for more detailed feedback and involvement in survey responses. People are
willing to share even when they are in the middle of a major ‘sporting moment’ and it looks
encouraging that mobile will help us to do better research.
By daring to combine this with the advertising research process, brands and businesses can
benefit from faster, better and cheaper way of harnessing their audiences’ perceptions and
motivations.