DSPy a system for AI to Write Prompts and Do Fine Tuning
Predict 2014, SiriusDecisions Kerry Cunningham
1. Increasing the odds: The conditions and
correlates for predictive lead scoring
success
Kerry Cunningham
Research Director, SiriusDecisions
2. #predict2014
SiriusDecisions, Kerry Cunningham
Research Director, SiriusDecisions
– 20 years Lead Development &
Management
– Research methods and analytics
5 years social science research
– Propensity modeling behavior
• Spending/ Consumption
• Employee performance
• Personality correlates of well-being,
Book Chapter,
September, 2014
3. Agenda
• Why is predictive necessary?
• 4 Principles of Prediction
• 5 Building Blocks for Predictive lead scoring
success
• 4 Keys to Success with Predictive Lead
Scoring
8. #predict2014
Why Predictive Is Needed
Today, most of that
qualification involves
teleprospecting and
sales calls
9. The Future of B-to-B Lead Development
The Role of Data Science
Find clues that exist out in the
world, which reliably point to
qualifying criteria & buying signals
you would ask the decision-maker
about if you could get him/ her on
#predict2014
the phone?
15. #predict2014
Conditions For Good Predictions
Sirius Perspective: Predictions made by the model need to make a real
difference
Wins Against
Replacement
Player
3
18. #predict2014
Model
Use Case
Starting
Point
Entity
Predicted
Source of
Predictors
Data
Building a Model
19. #predict2014
Sirius
Data
Perspective:
Prediction begins
with data that is
related to the
outcomes that
are to be
predicted.
Digital artifacts 1
20. Data
Sirius Perspective: Modern data science can reach deeply into online digital artifacts to
unearth evidence of business problems and buying initiatives.
#predict2014
• Corporate websites
• Press releases
• Job postings
• Application signatures
1
21. #predict2014
Use Cases
Use Case
Find new
businesses that
have a high
propensity to buy
from me
2
22. #predict2014
Use Cases
Find new
businesses that
have a high
propensity to buy
from me
2
Score and prioritize
businesses already in my
database on their
propensity to buy from me
23. #predict2014
Use Cases
Score and prioritize
existing customers for their
propensity to buy other
products and services we
sell
Find new
businesses that
have a high
propensity to buy
from me
Score and prioritize
businesses already in my
database on their
propensity to buy from me
2
24. #predict2014
Starting Point
Historical Data
Prospects that:
• bought or not
• converted or not
• responded or not
Didn’t Become
Customers
Data that clearly distinguishes
the two groups
3
25. #predict2014
Starting Point
No Historical Data
Prospects that:
• Have a business problem
• the motivation and
resources to solve it
Fit the profile Don’t fit
Data that clearly distinguishes
the two groups
3
26. Sirius Perspective: What is likely to be most predictive may be at the contact or the account
level, and gleaning information from both is normally important.
#predict2014
Source of Predictors
Top down
Bottom up
The best models
typically include both
prospect and account
level predictors
4
27. Entity PredictedSirius Perspective: Data science can reach deep into a contact’s
world to determine who is most likely to be involved in a buying
cycle.
#predict2014
Company
Contacts
Job Role
Common Titles
5
28. #predict2014
Entity Predicted
Contacts | Accounts
Company Hiring
Tech Ecosystem
Job Role
Common Titles
Content Engagement
Prof. Communities
Social Media Interaction
MAP
PLS
5
33. #predict2014
Realistic Expectations: The Nature of Prediction
SiriusPerspective: Complexity is the (other) enemy of prediction, but
the reality in b-to-b selling.
B-to-b selling, like
human behavior, is
complex
34. #predict2014
Realistic Expectations
• Experience and expression
of emotions
• Personality & Well-being
• Spending and money
• Biological basis of
behavior
• Time perspectives
• Statistical modeling of
employee performance
and attrition
• Propensity modeling -
predictive lead scoring
39. Pilot teams should be
comprised of lead
recipients who can be
expected to perform
like typical end-users
#predict2014
Pilots | Champions
Pilot projects help
develop realistic
expectations, reveal
process flaws, and
develop in-role
champions.
40. #predict2014
Pilots | Champions
Lead recipients
normally have
compelling reasons to
find defects in lead
scoring, and they will.
41. #predict2014
Pilots | Champions
A strong pilot team will help
you socialize the project
effectively, in terms the
end-users understand and
trust.
46. #predict2014
Feedback | Iteration
Part of every SLA should include
specific, actionable feedback to
marketing, enabled via technology
not (just) word of mouth
47. Key Take-aways
• Current lead scoring does not account for enough of
the variance in lead conversion
• Modern data science can generate proxies for
questions your best salesperson would ask prospects
if he/she could reach them all
• It is possible to model contacts, accounts and even
existing customers
• Prediction requires good, relevant data related to
consistent processes and outcomes
• Unrealistic expectations will kill even the best
modeling efforts
• Pilot projects will establish realistic expectations and
develop internal project champions
Notas do Editor
So, first, I want to touch on some principles for making predictions of any sort, and I draw on my experience both in the b-to-b arena and from my academic research experience..
<<Advance slide>>
To begin with, the best data we have to predict the future is past behavior. But there are important limitations.
<<Advance slides>>
Another way to think of it is as the Muni problem…
I live in san Francisco and I take public transportation to work. Here, the subway system is notoriously unreliable with respect to schedule. Most subways run on time, because there are not many variables at play underground. They run on a schedule, and unless something breaks, there’s little to mess up the schedule. But here, the subway cars also run above ground for part of their journeys and are subject to the same traffic issues drivers face. So, there are a lot more variables at play, a lot more can go wrong, and so as any MUNI rider can tell you, these predictions are barely more than hopeful guesses…
So, the more variables there are impacting an outcome, the less predictable it will be. The more those variables change, the less predictable the outcomes will be.
<<Advance slide>>
In another recent modeling effort I ran, we were interested in predicting which small business owners were more likely to buy advertising from us.
As I already mentioned, we started with a data set consisting of two kinds of prospects we had tried to sell in the prior year – those who bought, and those who didn’t. Then we added data that we acquired from a third party, which told us whether those companies were listed on Yelp, Craig’s list, Angies list, etc. Those that were listed on one particular online destination and had lots of reviews were 23X more likely to have bought from us the prior year. In other words, if we had only called companies like that, we would have been 23X more successful. Most models product lift numbers in the low single digits, so that is an exceptional find. Your mileage will vary!
<<Advance slide>>
Predicting stage 4 from stage 3 is going to work better than predicting it from stage 1. More time allows for the introduction of more variables, more change, and so time is the enemy of good prediction. It is harder to predict a 2 years sales cycle than a 2 day sales cycle.
<<Advance slide>>
So, now that we understand a bit more of the mechanics of prediction, let’s talk about some of the considerations you should have in mind when thinking about whether and how to engage in a predictive lead scoring effort.
<<Advance Build >>
The use cases – when and why you would employ predictive lead scoring
What you will need as a starting point
Which entity you are predicting – is it a contact or a company?
And then, where the predictor variables, or clues, come from
<<Advance slide>>
But, for some of you, particularly those with large addressable markets, what matters most is finding the right accounts to be marketing to, and finding them at the right time.
<<Advance Build >>