The document discusses engagement measurement frameworks for cultural institutions online. It proposes a user-centric engagement framework measuring awareness, actions, and advocacy. The framework is applied in a case study measuring engagement with an online game from Channel 4 beyond just website metrics. Results showed seeded blogs drove over 75% of traffic and engagement occurred across networks through gameplay, sentiment, discussions and links. The case study validated measuring total online experience rather than just website metrics.
'Stories & Numbers' - A Framework for Measuring Engagement
1. Learnings from the live web: ‘ Stories & Numbers’ A Framework for Measuring Engagement MCN 2009 Dr. Jason Ryan, iCrossing
2. Introduction This presentation 1. Background/context 2. What is engagement? 3. Framework for measurement 4. Case study 5. The ROI of engagement 6. Opportunities/challenges for Museums About me Working in digital for 14 years Joined Cogapp in 1998 - many projects with Museums & Galleries UX and consultancy work for cultural, government and commercial clients Left Cogapp in 2007 to join iCrossing, a global digital marketing agency
3.
4. Many organizations see their website as the sum total of their online existence – but every organization will always exist in a far broader network - the question is how we choose to listen to and engage with those networks The network view
6. Measurement models need to change Web analytics is about individual “eyeballs on page” metrics such as visits and time spent on site Analytics is often ignored – it needs to work with qualitative observations to tell a better story and understand what success online looks like But visitors complete actions such as commenting and watching videos that can not be tracked by simply calculating time on site And people interact with a site’s content by sharing and discussing it in the broader network - on blogs, forums, social networks and content streams and feeds. How do we evaluate visitor engagement that takes place off our owned website? What can we learn from that offsite engagement?
7. Measurement models need to change Recognising the user as active participant – and considering the total online experience – not just the onsite experience There is a paradigm shift underway from mass media measures (traffic) to engagement measures Active attention Deliberate actions Brand and content advocacy Development of a user-centric Engagement Framework which recognises: Awareness, Actions and Advocacy
8. Advocacy (Creators) Actions (Contributors) Awareness (Consumers) A user-centric framework Volume & Brand Control
12. Channel 4: Bow Street Runner Channel 4 is one of 4 terrestrial broadcasters in the UK It has a public service broadcasting remit – including education programming It was felt that education programming was not reaching the intended audience Channel 4 took the bold step of taking all education programming online - Including taking content to the audience iCrossing did a pilot project with Channel 4 to measure engagement with the educational flash game 'Bow Street Runner' Bow Street Runner is historical role playing game set in Georgian London – and explores the foundation of the first police force
14. Bow Street Runner The Game An episodic 5-part RPG built by Littleloud based on the 5 part TV series City Of Vice Features video shot in high-definition channel4.com/cityofvice The game was not marketed – awareness and traffic was driven solely by social media outreach – seeding content on relevant blogs Players are Bow Street Runners Level up from Runner to Chief Inspector Investigate crime > Collect evidence > Present case > Solve crime
15. To understand the landscape of the networks in which our potential audience exists Objectives To understand how users engaged with the game outside of the Channel 4 networks To measure the effectiveness of the social media seeding plan To measure how awareness spreads within the network To evaluate the Measuring Engagement framework
16.
17. Involvement: Are people coming to the site & playing the game? Interaction: How far do they get in the game, do they come back, do they interact with other users and Channel 4? Intimacy: What do they feel about the game, do they bookmark it? Influence: How far do users spread knowledge of the game through associated networks? A Method – Engagement Scorecard
18.
19.
20. Going beyond Web Analytics Web Analytics Engagement Metrics Visits Visits Time on Page Time on Page Bounce Rate Bounce Rate Comments on c4 forum Rank & Esteem data Return Visits Email Sign-up Comments on non-C4 sites Positive Sentiment Negative Sentiment # Tags/ Bookmarks Links # Influential Blogs Influential Blogs: Positive Sentiment Influential Blogs: Negative Sentiment Page Rank Average Search Engine Ranking City of Vice Viewing Figures
23. I finished the 3rd game but missed the esteem points at the end. I wanted to play 3 again but it had already saved my password for game 4 which doesn't exist yet. The password for 3 is gone. If I want to play it again, I have to start over… I’m not going to do that 1 Based on what I read, I thought the episodes were supposed to come out every Monday, but Ep. 2 took an entire month to show up. Hopefully… we'll be seeing a weekly release schedule from here on 2 I was hoping episode four would be up today, following the general weekly pattern lately, but seems not 4 Oh brother... on episode 2 with the nails in the hatch... whenever I try to pull them upwards I get my mouse outside the screen and I have to start all over!!! This happened twice now … this is sooooo frustrating 3 Makes me wish for programs of similar quality to be developed here for education. Imagine doing problem-based learning in this format… today’s kids would want something a bit edgier and more interactive. Who’d be up to the task? Discovery Learning? The History Channel? 1 Flash loading problems Release Dates Passcodes Loading Time Gameplay frustrations Negative Sentiment
24. The game is really well done and uses a pretty decent cast of actors… the actors are caught in full-motion and talk in real time 1 Historically accurate detective game where you have to investigate the locations in order to find clues and evidence 3 The game was created as a companion to the UK television series City of Vice and is not only well-acted, but also takes great pains to be historically accurate 2 Bow Street Runner … is a murder mystery that takes place in the seedier neighborhoods of Georgian London… you poke around at things, talk to people, and little by little the facts are uncovered 4 Gameplay Actors Plot Atmosphere Positive Sentiment
31. The importance of social tools to encourage Discussion and Advocacy Insight into the effect of decisions – (beta, delays emails) Sentiment fed into the design and allowed fixes to be prioritised Understanding how quickly the community take ownership of the product Seeding worked driving 75% of the traffic Outcomes Provided a benchmark for similar initiatives
32. Moved from a small Education pilot to contract to measure online engagement for the whole ‘Disarming Britain’ TV series and related online activity Highlighted the real possibility of an AGILE PLANNING FRAMEWORK – from seeding to creative execution Validation of the approach and framework – led to a Forrester Case Study – the first on Measuring Engagement Outcomes What if marketing and creative just got better over the course of a campaign?
34. Disarming Britain blog: Network growth Week Two Week Four Week Ten Week Eight Week Six External incoming link Blog Post/Page
35. Top Referring Sites to C4 MicroSite (you don’t always get what you want) Airgun Enthusiasts US Rifle Enthusiasts Anti-Knife Community Grime Fans Community Football Community Q&A Site Pro Gun Ex- Soldier Dutch Blog Gun Advertising Community Action Site
36. Disarming Britain Season – themes of discussion Themes of wider discussion Glamorisation of violent crime Issues relating to child discipline in school and in the home Mentioned in the news alongside the Home Office campaign and reports of fatal stabbings Media scaremongering
37. Channel 4 & Home Office Bebo friend networks Friends with one profile Contributed content to both profiles (Comment or Whiteboard post) Friends with both profiles
38. Which content drives traffic “ Spy shots of McLaren P11 supercar” “ McLaren P11 pictures surface” “ Nissan's Nuvu: electric, cartoonish” “ Aston Martin and Mercedes 'to strike £235m (€300m) deal‘” “ Test Drives Audi's new S4!”
39. The value of content Visit Value Link Value Total Value this month: The value of content…
40. Summary – Outcomes & Insights The user-centric framework is a useful tool for evaluating user engagement Ability to measure ROI for government and non-profit organisations in terms of social impact Potential to benchmark and index engagement scores by web traffic Always design-in measurement Don’t be limited by analytics tools – aggregate everything! “ Numbers & stories” – is a practical approach that allows analytics insights to ‘travel’ within organisations
41.
42. JASON RYAN VP, Head of User Experience, iCrossing UK +44 (0) 1237 828 146 [email_address] thank you.
43. What is engagement? Awareness: H ave people heard about you? How do you know? Conversation: What is depth and quality of the conversation? Advocacy: Who are the influencers? What are they saying about you? Actions: What actions are being taken? Sales, downloads, subscriptions, ratings? A relationship
Notas do Editor
Good afternoon everyone, Today I am going to talk about a measurement and evaluation framework – one that goes beyond visits and page views – to evaluate user engagement with your online content. I am going to outline the framework and the thinking behind it – and then illustrate it in action through some work that I did with the British broadcaster Channel 4 But before I dive into that – just a brief introduction about who I am – and why I am here today I have been working with digital technology for about 14 years now I joined Cogapp, who I’m sure many of you are familiar with, in 1998, and whilst there I havd the privelege of working with many Museums & Galleries – including MoMA, the British Museum and the Cleveland Museum of Art I helped set up a User Experience team at Cogapp and undertook digital consultancy work for a range of commerical and no-for-profit clients And in 2007, I left Cogapp to join iCrossing, as US based company, and one of the largest independent digital agencies – where I have got a good taste of the commericial world – working for clients including Toyota, Coca Cola and Bank of America – mostly on social media and change management programmes
So, first some context: the web has changed since I started working in 1995 - and it continues to change at an amazing rate. I think when we all started out on the web, we began by adapting our existing models - our marketing approach, digitising our catalogs, even projecting our internal structures on our audiences BUT somewhere along the way we started to realise the web was about more than replicating the real world - it represented a whole new set of behaviours, challenges, opportunities and discoveries - and I believe that all of us that work with digital technology are on our own journey of discovery in that respect It is the emergence of personal blog, along with media upload services like Flickr and YouTube; And the web start-ups began to build the tools for the next iteration of the web, connecting people and data. It is this movement (social web/web2.0), where the real disruption and innovation has come from over the last 3 or 4 years. Specifically there has been: an explosion of content creation and distribution tools our 'users' are increasingly becoming active participants rather than simply passive visitors, or consumers of our content they are increasingly able to build their own interfaces to and contexts for our content - through start pages, RSS, widgets, mobile devices; they are increasingly able to talk about us and our content, as well as interact with us, in their own networks - the ones that we cannot control. Like it or not the web IS social, and we are all having fun adapting
When I joined iCrossing – essentially a search and social media company, I got introduced to the concept of networks, networks of attention and influence around themes and topics We have a number of tools to map and visualize networks – and what you see here is a typical network map. Basically how it works is we look at which sites are most influential and visible for certain concepts through keywords, and generate visualizations based on backlinks to those sites. The maps are only snapshots of an incredibly dynamic ecosystem – but they do serve to explain how our site is only one authoritative node in this complex web of attention. And many organisations (commercial and non-commercial) do see their website and the sum total of their entire online existence. But every organisation will always exist in a far broader network The question is how we choose to listen and engage with those networks
Tech start up and native digital brands, like Digg for example, understand the web - and know how to build architectures that support consistent user experiences across websites, widgets, devices, social network applications, podcasts --- it all works together as part of a seamless but distributed brand experience
But many organisations till hold on to traffic or visits as the primary measure by which to evaluate the effectiveness of their online presence Clearly our measurement models need to change For example Web analytics is all about eyeballs on pages - But visitors complete important actions such as commenting and watching videos that cannot be simply be tracked And people interact with a sites content by sharing it – and discussing it in the broader network – on blogs, forums, social networks and content streams and feeds How do we evaluate visitor engagement takes place off of our own website? Do we care? What can we learn from that offsite engagement? - Analytics is often ignored – its just not sexy enough – it needs to work with qual observations to tell a better story about what success is
I believe we need Measurement models that recognise the user as an active participant – and that consider the total online experience – not just the onsite experience There is paradigm shift underway from MASS MEDIA measures (traffic) to engagement measures – like ACTIVE ATTENTION DELIBERATE USER ACTIONS ADVOCACY FOR THE CONTENT AND BRAND I have been involved in developing a user-centric ENGAGEMENT MEASUREMENT FRAMEWORK that recognises these three components of online engagement – AWARENESS, ACTIONS AND ADVOCACY
The foundations for this framework were born out a number if influences - The network maps that I explained earlier The balanced scorecard that Jack outlined in his presentation - Our increasing data literacy and visualisation of data thanks to free tools like GA - And by thinkers on Participation and engagement – including Ross Mayfield and his Power Law of Participation
We came across this groundbreaking Forrester paper by Brian Haven in early 2008 – that was completely in line with our AWARENESS, ACTIONS, ADVOCACY FRAMEWORK – and it was this validation that allowed us to push on and turn the Engagement Framework into a commercial reality last year. Basically, Haven talks about the 4 Is of Involvement, Interaction, Intimacy and Influence Involvement - how involved your users are with you – metrics like site visits, pages viewed etc. data provided by standard web analytics or logs Interaction – going beyond measuring touches of Involvement to measuring Actions (comments, reviews, sales) Intimacy –about the emotional reaction and connection they have with you and your content– measured though a new breed of social media and sentiment monitoring tools – looking at the volume and quality of the online conversation – and combined with more traditional methods like feedback forms and surveys Influence – where people have advocated our content – forwarded it, shared it, linked to it – and weighting those influence measures more heavily when known influencers in the network have taken these actions -And it was this Engagement Model that we applied to a piece of work for Channel 4 that I will briefly describe now
Balance scorecard Involvement: Are people coming to the site & playing the game? Interaction: How far do they get in the game, do they come back, do they interact with other users and Channel 4? Intimacy: What do they feel about the game, do they bookmark it? Influence: How far do users spread knowledge of the game through associated networks?
Channel 4 is one of terrestrial broadcasters in the UK It has a public service broadcasting remit – including education programming C4s education programming was not reaching the intended audience Channel 4 decided to take all of its education programming online and to the audience in their natural habitat rather than try to drive to the C4 website C4 wanted to be able to show that this approach and been successful – so they asked iCrossing to evaluate audience engagment (both an and offsite) for a pilot project - an educational flash game 'Bow Street Runner’ Bow Street Runner is historical role playing game set in Georgian London – and explores the foundation of the first police force
The game was not marketed – awareness and traffic was driven solely by social media outreach – seeding content on relevant blogs
C4 Objectives To understand the landscape of the networks in which our potential audience exists To understand how users engaged with the game outside of the Channel 4 networks To measure the effectiveness of the seeding plan No marketing – only way to drive traffic was by identifying influential sites and blogs in the network and approach them with details about the game 17 gaming sites were approached based on size of audience and relevancy To measure how awareness spreads within the network To evaluate the Measuring Engagement framework
This was very simple diagram to explain the that Engagement with the online content would bedistributed across networks And that multiple measurements at multiple points would be required to capture all interactions
Balance scorecard Involvement: Are people coming to the site & playing the game? Interaction: How far do they get in the game, do they come back, do they interact with other users and Channel 4? Intimacy: What do they feel about the game, do they bookmark it? Influence: How far do users spread knowledge of the game through associated networks?
This is the key metrcis C4 would monitor fir the onsite content And the engagement metrics are the extended set for this project
Seeding plan No marketing – only way to drive traffic was by identifying influential sites and blogs in the network and approach them with details about the game 17 gaming sites were approached based on size of audience and relevancy As content about the game and links to it were posted on these sites, we monitored additional sites that linked to this offofsite content – and the volumes of traffic delivered through bothe the seeded sites and the extended network - 35% of all traffic came from 17 seeded sites; over 75% of all traffic from extended network
And because we were dealing with a broadcaster, we converted some of the engagement metrics into comparative measures that the organisation would understand and that would demonstrate the success of the project
The blog seeding of BSR was very successful in attracting gamers - Seeding drove majority of traffic to game (>75%) Seeded sites also attracted links from other gaming websites - Seeded sites attracted nearly 1000 links between them Seeding was done too early - Traffic for the game never reached the same level after the initial seeding Seeding could also have targeted ‘gatekeeper’ and 14-19 year old demographic - The sites identified through direct response info, web stats, and desk research
Seeding worked driving 75% of the traffic The importance of social tools to encourage Discussion and Advocacy Insight into the effect of decisions – (beta, delays emails) Sentiment fed into the design and allowed fixes to be prioritised Understanding how quickly the community take ownership of the product Provided a benchmark for similar initiatives
Validation of the approach and framework – led to a Forrester Case Study – the first on Measuring Engagement Moved from a small Education pilot to contract to measure online engagement for the whole ‘Disarming Britain’ TV series and related online activity Highlighted the real possibility of an AGILE PLANNING FRAMEWORK – from seeding to creative execution What if marketing and creative just got better over the course of a campaign?
The next few examples are showing how some of the learnings from the C4 work are being applied more directly for site owners who rely on ad revenue. In this case an online publisher – for their car and driving magazine This diagram shows what specific onsite content is driving referring traffic, and from which sites over the course of a month. For example on the 19 th October – the ‘Spy shots of McLaren P11 supercar piece’ drove 7,500 visits from autoblog.com – and 3000 visits from leftlanenews.com
Here, the value of a specific piece of content has been calculated by adding the page visit value to the link value. For example: The link value calculated where Ad Inventory costs might be £10 per 1,000 unique page views (CPM) And the Link value calculated using a nominal value for each link – based on a standard value that might be paid when buying links Engagement measurement is appealing to media owners - because by showing advertisers that whole sites, or even specific content pages are more engaging than their competitors – they could charge a premium on standard ad placement rates
The user-centric framework is a useful tool for evaluating user engagement Ability to measure ROI for government and non-profit organisations in terms of social impact Potential to benchmark and index engagement scores by web traffic “ Numbers & stories” – is a practical approach that allows analytics insights to ‘travel’ within organisations Always design-in measurement Don’t be limited by analytics tools – aggregate everything!
Web visits is a metric that is ‘understandable’ to Museums – it is rooted in the real world How many Museums go beyond visits and page views as the primary measure and attempt to evaluate how engaged visitors are ? With more and more free quality content and media online, Museums, as well as commercial brands, face competition for attention Unlike many commerical ‘brands’ Museums have rich and potentially very engaging online content By extending their measurement models Museums can: drive more traffic provide better experiences, and even find better ways to monetise their content