1. Stats are Sexy!
Metrics and Testing
That’s HOT!
1

2. 2
“The only truth is what actually
happens.”
The number of people who did the
thing you wanted them to do. Give.
Act. Volunteer. Come to Get
Together.
2

3. 3
The data
Oh NO! Let’s take a closer look...
3

4. 4
What are all these metrics?
• It’s easy to get lost in the
result metrics, some
common calculations are:
• Delivery rate
• Open rate
• Click-through rate
• Conversion rate
• Tell-a-Friend rates
4

5. 5
Delivery Rate =
Deliveryemails Received/
Number of
Total emails Sent
Delivered Emails,
not yet opened
5

6. 6
Open Rate =
Opening the email members
Number of emailed
who opened at least 1 email/
Opened emailTotal Emails Sent
6

7. 7
Beware, Preview Pane
Affects Open Rates
7

8. 8
Opened email
8

9. 9
Click-Through
Click-through Rate =
Number of emailed members
who clicked on at least one link/
Total Emails Sent
9

10. 10
The Response
Response Rate =
Number of emailed members who
participated in the campaign /
Total emails Sent
10

11. 11
Tell-a-Friend Usage
Tell-a-Friend Usage Rate =
Number of emailed members
who told at least 1 friend/
Total emails Sent
11

12. 12
Tell-a-friend Recruitment
Tell-a-Friend Recruitment Rate =
Number of Friends who
joined a center/
Total Emails Sent
12

13. 13
Which Rates are
the most important?
• Conversion Rates are the best measure of a
campaign’s immediate success
• Click-through rates can be good indicators of
how members are interacting with your
messages
• Open rates are deceiving for short term data,
but if measured over several years can become
interesting. They also are good for normalizing
other data.
13

14. 14
So, what can you do with all these
statistics?
14

15. 15
This is a numbers game …
… do the math
• If you want 10 people to TAKE ACTION,
then:
• (at least) 20 people need to click to the
advocacy page
• (at least) 30 people need to open your e-mail
message
• (at least) 100 people need to receive your
message
15

16. 16
This is a numbers game … math
… do the
• If you want 10 people to DONATE, then:
• (at least) 50 people need to click to the
donation page
• (at least) 250 people need to open your e-
mail message
• (at least) 1,000 people need to receive your
message
• Yes, your results may vary…dramatically!
16

17. 17
It all starts with the list
• Good direct marketing is all about the
list
• It starts with the list
• It ends with the list
• It is all about the list
• But building a good list requires you
to be good at everything else: your
work, copy, layout, timing, etc.
17

18. 18
Typical response rates
• Nothing is typical, really
• Big lists usually = lower response…
• …and smaller lists usually ROCK
• But if you make us…
18

19. 19
Where can I find my
industry standards?
• Marketingsherpa.com
• Emarketer.com
• Opt-In News
• Your own campaigns!
19

20. 20
Making the most of the numbers to get the most out of
your online program…
Segmenting and Testing
20

21. 21
Segmenting is…
…dividing your e-mail list when running campaigns to
learn more, personalize, and generate higher results.
21

22. 22
Segmentation example
• Donors
Recipients Amt of # of Avg Resp.
Gifts Gifts gift rate
3,500 $2,290 40 $57.75 1.13%
• People on direct mail list with e-mail
Recipients Amt of # of Avg Resp.
Gifts Gifts gift rate
7,250 $3,385 92 $36.79 1.26%
• E-mail subscribers
Recipients Amt of # of Avg Resp.
Gifts Gifts gift rate
7,000 $775 26 $29.81 0.37%
22

23. 23
Before segmenting…
23

24. 24
After segmenting…
24

25. 25
Other segmentation strategies
• Giving level
• Campaign-based
• Donors who have taken action
• Donors who have not taken action
• Non-donors who have taken action
• Non-donor, non-action-takers
• Recency
• Source
• One-time and lapsed online donors
• Active and inactive non-donors 25

26. 26
Testing …
…Once you’ve determined how best to segment your
particular list, you can begin to test what works within
each of these segments.
26

27. 27
Testing
• Open rates
• Time of day, day of week
• Subject lines
• Straightforward subject lines vs suggestive ones.
• Your action will help a child eat today
vs.
• How can you help the orphans?
• To Mention The Ask Or Not?
• Looking ahead, more animals need your help in 2007
vs.
• Last chance for your tax-deductible gift to animals
• Subject lines that in different ways ask for a renewal of
commitment.
• Renew your commitment for 2007
vs.
• Your membership is expiring
vs.
• Renew today—and save even more puppies 27

28. 28
Testing, continued
• Click through rates
• Premiums
• Offer
• Signer
• Short or long copy
• Link placement
• Conversion rates
• Short vs. long page
• Images and insets
• First things first!
• Giving levels
28

29. 29
How to create a good test...
• Pick one variable you want to test
• Keep everything else the same
• Randomly 50/50 split the population of
the campaign
• Compare the results from each test
group using a “significance calculator”
• “Significant” results mean the results
can be repeated
29

30. 30
Testing Do’s and Don’t’s
• DO
• Ask a question and design a test to answer it
• Select a RANDOM sample of your list
• Test ONE thing at a time
• Keep everything the same except for the thing
you are testing
• Check for statistical significance
• Make sure your list segments are large
enough to produce meaningful results
• Understand that some test results are only
applicable to the message you were testing
30

31. 31
Testing Do’s and Don’t’s
• DON’T
• Assume that test results apply across lists
or list segments
• Bite off more than you can chew
• Read too much into the numbers
• Assume that every test will yield
statistically significant results
31

32. 32
Testing Gone Wrong
Click Response Better Statistical
Messages Through Unique Rate (unique Than Significance
Version Description Sent Open Rate Rate Signatures sigs) Control? ?
CONTROL Regular Email 21,059 20.90% 8.40% 1,395 6.62%
Masked URLs – used
hyperlinked text instead
Test 1 of spelled out URL 20,953 22.70% 10.60% 1,645 7.85% Y Y
In Line URLs –URLs are
in the line of text, not
Test 2 separate paragraphs 20,377 21.30% 8.30% 1,318 6.47% N N
Longer subject line –
Everything is same as
control, except the subject
Test 3 line is a longer version 20,419 20.30% 7.90% 1,262 6.18% N N
Header Images-
Everything is same as
control, except header
image does not have a
Test 4 picture of candidate 20,784 21.50% 8.30% 1,391 6.69% Y N
No Header Image –
Test 5 header image is removed 21,734 22.50% 8.80% 1,524 7.01% Y N
Large Button - take
action button in the upper
Test 6 right hand corner is larger 21,014 21.10% 8.50% 1,431 6.81% Y N
Move On Style- email is
stripped of header image
Test 7 and all graphics 21,591 22.60% 10.70% 1,804 8.36% Y Y
Short Email – middle
Test 8 paragraphs are removed 21,146 23.90% 8.90% 1,483 7.01% Y N
Greeting - lead in to the
email says “Dear Friend”
32
Test 9 instead of Dear First Name 20,702 21.30% 7.90% 1,328 6.41% N N
Total 209,779 21.82% 8.84% 14,581 6.95% Y N

33. 33
What Happened?
• Too many variables
• Tried to answer every question at
once
• The result:
• Hours of work to set up 10 segments
• Data was messy and difficult to decipher
• We didn’t learn much about our list
33

34. 34
What We Should Have Done
• Picked one variable
• Had a clear question to answer
• Split the list in half or thirds
instead of in tenths
34

35. 35
And the winner is….
• Subject Line Test
• What metric should be used to evaluate a subject
line?
• Which is the winner?
35

36. 36
And the winner is…
• Open Rate
• Two-way tie
Click
Open Through Response
Message Emails Sent Rate Rate Actions Rate
Bush's Budget: Guns
Not Butter 1,000 28.50% 13.00% 53 5.30%
What's At Stake 999 31.90% 15.50% 52 5.20%
Action Alert: Defend 100
Hours Agenda 999 27.70% 14.80% 49 4.90%
Action Alert: Organize
agaist Bush's budget 1,000 31.90% 18.20% 51 5.10%
Organize Against Bush's
Budget 1,000 28.80% 16.70% 54 5.40%
Total 4,998 30.60% 16.90% 259 5.18%
36

37. 37
And the winner is…
• Link placement test
• Which is the most important metric?
• Which segment is the winner?
Click
Messages Through Unique Response
Test Sent Open Rate Rate Signatures Rate
Below the first
paragraph 20,419 22.70% 10.60% 1,645 8.06%
Below the third
paragraph 20,567 21.30% 8.30% 1,318 6.41%
Below the fifth
paragraph 20,489 20.30% 7.90% 1,262 6.18%
37

38. 38
And the winner is…
• Click Through Rate
• Below the first paragraph
38

39. 39
And the winner is….
• Short Email vs. Long Email
• What metric(s) should you look at?
• Which message is the winner?
Click
Messages Through Response
Sent Open Rate Rate Rate
Long Email 95,917 20.90% 7.50% 5.62%
Short Email 96,953 21.10% 7.30% 5.48%
39

40. 40
And the winner is…
• Click through rates and response rates
• Trick question – the results are statistically
insignificant.
• You can’t always eyeball test results – be sure to
test for statistical significance
Click
Messages Through Response
Sent Open Rate Rate Rate
Long Email 95,917 20.90% 7.50% 5.62%
Short Email 96,953 21.10% 7.30% 5.48%
40

41. 41
And the winner is…
• Which campaign is more successful? Why?
Click
Messages Open Through Amount Average Response
Campaign Received Rate Rate Donations Donated Gift Rate
Issue #1 63,101 42.06% 0.49% 102 $6,578 $64.49 0.16%
Issue #2 62,653 41.48% 0.62% 125 $4,960 $39.68 0.20%
41

42. 42
And the winner is…
• Arctic campaign most likely to raise
money (fewer donors, but higher
average gift)
• Public lands campaign most likely to get
engage donors (more donors, lower
average gift).
• Depending on the goals of the campaign
(raise a lot of $, or get more people to
give), both are a success. 42

43. 43
And the winner is….
• From line testing – by list segment
• What should the organization do
moving forward?
43

44. 44
And the winner is…
• Even thought the inactive segment had
a lower open rate with the from line
“Kathy, Organization” all of the other
segments saw improved open rates.
• You could segment out the inactive
folks and continue to send them
messages with the old from line, but it’s
probably not worth the effort.
44

45. 45
Digging Deeper:
5 Questions you could ask
45

46. 46
Question 1
Which group of people on my
email list is the most likely to
take action, tell a friend, or
donate?
46

47. 47
Question 1, Sample answer
Source 1 Source 2
Members in Group 2,403 4,014
Activists* 59.84% 29.45%
Donors 5.45% 2.72%
TAF 35.41% 9.12%
Super activists 4.99% 2.99%
Members who opened an email 57.26% 55.23%
in 2005
Members who clicked 70.41% 54.53%
47

48. 48
Answering Question 1
• Two words: Source Codes
• Suffix on imports or on any link someone may
click to come on your list, starts with ‘?
source=’
• Advocacy campaign (?source=advo1)
• Viral Flash game or movie (?source=funny_movie)
• Banner ads (?source=yahoo_banner)
• Paid source (?source=care2)
• Donation campaign (?source=moneybags)
• Source codes are attached to a member so
you always know where they originally came 48
from

49. 49
Answering Question 1
• Each cell in the table is a member
select
• Query members who have the source
code you want AND have donated,
taken action, etc.
• The results can help you compare how
valuable names from various sources
are to your organization
49

50. 50
Question 2
How do I know how fast my
list is growing?
50

51. 51
Answering Question 2
• List Growth Rate: # new sign-ups/# of
deliverable addresses
• Shows % list growth over a period of time (month,
3-months, year)
• List Hurdle Rate: # removals/# of deliverable
addresses
• Shows % of list that leaves over a period of time
• # removals tells you how many new names you
need to get before you achieve net growth
51

52. 52
Question 3
How come my advocacy
response rates are so low?
52

53. 53
Question 4
How do I find out if I’m
sending too much email?
53

54. 54
Question 5
How valuable is an email
address?
54