This set of slides shared at PyCon 2017 Delhi talk about the need for visualizing black box models and the ways in which we can improve their understanding through interactions.
3. 3
TELECOM CHURN
“Churn of customers is a
particularly severe problem in
the telecom industry.
The challenge is to identify
the propensity of churn up to
a month in advance, even
before a customer moves out,
so that proactive
interventions can begin”
5. 5
Outgoing call
0 0 - 4 15+5-14
1
RECHARGE
AMT > $20
01
YN
> 1
RECHARGE
0
N Y
3.2% 3.6%
MISSED WASTED
4.0
COST PER CUST.
39%
IMPROVEMENT
Decision Tree
MODELS
6. 60.6% 2.5%
MISSED WASTED
2.21
COST PER CUST.
66%
IMPROVEMENT
SVM
MODELS
OK
WASTED
Marketing
cost
$1.8
MISSED
Acquisition
cost
$4.1
OK
No churn ChurnNochurnChurn
PredictionActual
7. PRICE FORECASTING FOR AN
ASIAN AGRICULTURAL ENTERPRISE
Problem Approach Outcome
A Gramener Advanced Analytics Case Study
A leading agricultural
enterprise wanted price
forecasts for their products in
order to plan inventory
release to optimise revenue.
Incorrect timing was leading
either to loss of revenue or
unsold inventory.
Gramener applied a suite of
price forecasting models
based on internal and
external factors.
The models were evaluated
on multiple test datasets to
select one that minimised
median absolute deviation.
The model was able to
forecast the price to an
accuracy of 88%.
Within the first quarter of
deploying the model, the
revenue uplift attributable
directly to pricing was +3.2%.
12. 12
SEGMENTING INDIA’S DISTRICTS BASED ON BEHAVIOUR
Previously, the client was treating contiguous regions as a
homogenous entity, from a channel content perspective.
To deliver targeted content, we divided India into 6
clusters based on their demographic behaviour.
Specifically, three composite indices were created based
on the economic development lifecycle:
• Education (literacy, higher education) that leads to...
• Skilled jobs (in mfg or services) that leads to...
• Purchasing power (higher income, asset ownership)
Districts were divided (at the average cut-off) by:
Offering targeted content to these clusters will reach a
more homogenous demographic population.
Skilled
Poorer Richer
Unskilled Skilled
Uneducated Educated Uneducated Educated
Unskilled
Purchasing power
Skilled jobs
Education
Poor Breakout Aspirant Owner Business Rich
Poor
Rural, uneducated agri
workers. Young population
with low income and asset
ownership. Mostly in Bihar,
Jharkhand, UP, MP.
Breakout
Rural, educated agri workers
poised for skilled labour.
Higher asset ownership. Parts
of UP, Bihar, MP.
Aspirant
Regions with skilled labour
pools but low purchasing
power. Cusp of economic
development. Mostly WB,
Odisha, parts of UP
Owner
Regions with unskilled labour
but high economic prosperity
(landlords, etc.) Mostly AP,
TN, parts of Karnataka,
Gujarat
Business
Lower education but working
in skilled jobs, and
prosperous. Typical of
business communities. Parts
of Gujarat, TN, Urban UP,
Punjab, etc
Rich
Urban educated
population
working in skilled
jobs. All metros,
large cities, parts
of Kerala, TN
The 6 clusters are
13. 13
How to classify clients by behaviour
Using customers’ ad spend patterns, categories of
purchase, periodicity, price points and impact,
Gramener accurately classified clients to
1. Offer personalised deals
2. Create new products
Big buyers across categories at
low price points
P&G
Cadbury
Reckitt
HUL
1
Big buyers across categories
with better price & viewership
Godrej L’Oreal
ITC GSK J&J
Amazon Coke
2
Mid-buyers across categories
with avg price & viewership
4
Heinz Apple
Future Group
LIC Ford Amul
Large clients
Medium clients
Small clients
Tiny clients
Size legend
Each box contains a
cluster of advertisers
with similar behaviour
FMCG
Auto
Telecom
E-commerce
Electronics
Retail
BFSI
Infrequent Hindi Movie ads in
regular slots at high price
5
Getit TVS Quickr
Lenovo HPAircel
Axis MRF
Microsoft ICICI Ceat
Motorola
Infrequent Hindi GEC advts
with high TVR/ very low price
6
Saavn Voltas PNB
Birla Sunlife
Jivraj Tea Pitambari
Summercool Home Appliances
Frequent regional channel ads
with low viewership
7
Pepperfry
Shoppers Stop
Bank of Maharashtra
Raja Biscuits
Cookme Spices
Pran Foods
Dipros
Metro Dairy
Koel Fashions
Meghbela
Big buyers across categories
with low regional advertising
3
Nestle
Maruti Airtel
OLX Samsung
Dabur
Occasional Hindi GEC advts at
moderate price points
United Biscuits
8
Expedia
BigBasket
Sulekha
Union Bank
Yes Bank Piaggio
BMW
Hitachi
Occasional regional and Hindi
GEC ads at high price
9
PayTM
Franklin Templeton
Duroflex
Mother’s Recipe
Anchor Electricals
Advertiser
Clustering
Transform variables to
minimize correlation
Cluster customers to
minimise overlap
Profile clusters to interpret
their characteristics
17. 17
68% correlation
between AUD & EUR
Plot of 6 month daily
AUD - EUR values
Block of correlated
currencies
… clustered
hierarchically
18. WHAT YOU SHOULD TAKE AWAY
BLACK-BOX MODELS ARE
INCREASINGLY ACCURATE
BLACK-BOX MODELS NEED
INTERPRETATION (EVEN
MORE)
BUILD VISUAL SUMMARIES TO
EXPLAIN MODELS
MOVE UP & DOWN THE
LADDER OF ABSTRACTION
TOOLS ARE LESS IMPORTANT THAN TECHNIQUE
19. 19
THIS IS GRAMENER’S STACK
PythonR
JavaScriptExcel
Pandas
Tornado
d3
lodash
Pivots
VBA
ggplot2
plyr
Analyse Communicate
External
Internal
We recruit across this stack
(and there’s a skill gap in the market in each of these)
Notas do Editor
A decision tree is a visual representation of choices, consequences, probabilities and opportunities. They are visual representations of the average outcome.
Applying the same fundamental to predict the churn handling we were able to calculate the cost per customer and improvements which were done.
On an average, whether an outgoing call was made from the phone. In case of a viable answer, we were able to fix 3 buckets of 0-4 days, 5 to 14 and more than 15 days. If the call has not been made for more than 15 days, there will no recharge voucher applied and the customer may likely leave the network. In cases where the call has been done within 15 days and the simultaneous recharge has been done only once, what has been the recharge amount. If amount is greater than 50, the loop starts from beginning and we establish that the consumer is engaged and spending will not be huge.
Earlier, the telecom operator for whom the design has been done was spending more. The decision tree helped them to save 62% of their costs with only 3.2% of cases on an overall basis.
So, what we did was put a variant of this visual together. On the right, you have a series of currencies like the Australian dollar, the Euro, the British pound, etc; some commodities like silver and gold; and some stock indices like Sensex, FTSE, and S&P.
The cells here have a number inside that indicates the pairwise correlation between a pair of securities. For example, the number 68 on the top left indicates a 68% correlation between the Australian dollar and the Euro. To the left of the Euro and just below the dollar (diagonally opposite to the 68), there’s a scatter plot that shows the daily prices of both these currencies. Each dot is one day’s data. The x-axis shows the Australian dollar value. The y-axis shows the Euro value. This helps identify what the pattern of movements of any two currencies is. From this, you can easily see visually that the Australian dollar and the Euro both tend to move together. Or, where there are strong correlations like the FTSE & S&P, the pattern is almost a straight line.
In some cases there are negative correlations. For instance, if you take the Sensex against the Japanese Yen, the correlation is -79%. The cells are coloured based on their correlation values. Greens indicate strong positive correlation. Reds indicate strong negative correlation.
These are also grouped hierarchically. On the left, we have a series of lines indicating clusters. The most similar securities are grouped together. So FTSE and S&P with a 98% correlation are very close. The ones that are less correlated are kept further away based on a tree-structure.
This leads to clustering of securities. For example, there is a green block in the center which has SGD, JPY, XAU, CHF and CNY. All of these are fairly well correlated. When any one currency in this block goes up, all the others go up as well. When any one goes down, all others go down as well.
Similarly, you have another block to its top left: S&P, FTSE, Sensex and to a certain extent, the Pakistani Rupee. These move together as a block as well.
But when this block goes up, all the currencies in the other block go down, as indicated by the red negative correlations between these two blocks.
This can be used very easily for decision making. For example, one client who was trading with Singapore and Japan looked at the strong correlation and decided to consolidate their holdings in Japanese Yen. They then moved up and down this column to find a good hedge. FTSE looked like a good hedge – it was the most negatively correlated with JPY at that time -- and they decided to place a third of their portfolio in FTSE.
A sheet like this improves people’s understanding of relatively complex data, and results in significantly increased trade volumes.