Claudia Perlich, Chief Scientist, Dstillery at MLconf NYC
There is a deeply symbiotic relationship between machine learning/predictive modeling and Big Data. Machine learning theory asserts that the more data the better. Empirical observations suggest that more granular data, a hallmark of Big Data, further improves performance. Predictive modeling is one of the core techniques that measurably delivers value across many industries and demonstrates the value of Big Data. However, there is a surprising paradox of predictive modeling: when you need models most, even all the data is not enough or just not suitable. The foundation of predictive modeling requires that you have enough training data with the respective outcomes, preferably IID. But often this data is not available: there are only so many people buying luxury cars online to inform my targeting models. I can never observe what happens BOTH when I treat you AND when I don’t – which is what I need to make causal claims and measure the impact of strategic decisions. To allocate sales resources I love to know what a customer’s budget is – but maybe even he does not know. So in the days and age of Big Data there remains an art to machine learning in situation where the right data is scarce. This talk will present a number of cases where enough of the right data is fundamentally not obtainable and how creative data science can still solve them.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais de MLconf
Mais de MLconf (20)
Último
Último (20)
Claudia Perlich, Chief Scientist, Dstillery at MLconf NYC
- 1. All the data and still not enough! Claudia Perlich Chief Scientist @claudia_perlich
- 2. Predictive Modeling: Algorithms that LearnFunctions
- 3. Income Age Buy 123,000 30 yes 51,100 40 yes 68,000 55 no 74,000 46 no 23,000 47 yes 100,000 49 no Data forPredictiveModeling Target Examples Features
- 4. ? yes yes no no yes no RulesforPredictiveModeling Target Examples Features Data should be: Large enough Independently Identically Distributed
- 5. Paradox of BigData: “Youneverhave thedatayouwant” Art of making due with second best
- 7. WalletisNEVERobserved We observe this in the data But we do not observe this IBM Sales to this Company Company Revenue (D&B) Wallet/Opportunity How can we make this a predictive modeling problem?
- 9. 9 REALISTICWalletsas quantiles Motivation Imagine 100 identical firms with identical IT needs Consider the distribution of the IBM sales to these firms Bottom firms should spend as much as the top Define wallet as high percentile of spending conditional on the customer attributes Frequency IBM Sales Wallet Estimate
- 11. QuantileRegressionoptimizing weightedabsoluteloss 10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 Firm Sales IBMRevenue Company Sales IBMRevenue Opportunity for C 2 Opportunity for C 1 C1 C2 10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 Firm Sales IBMRevenue Company Sales IBMRevenue Opportunity for C 2 Opportunity for C 1 C1 C2 10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 Firm Sales IBMRevenue Company Sales IBMRevenue Opportunity for C 2 Opportunity for C 1 C1 C2 10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 Firm Sales IBMRevenue Company Sales IBMRevenue Opportunity for C 2 Opportunity for C 1 C1 C2
- 13. © IBM Corporation 2008 Slide 13 Siemens: Computer-Aided Detection of Breast Cancer in Mammograms 1712 Patients 6816 Images 105,000 Candidates [ x1 , x2 , … , x117 ] Image feature vector Malignant ? MLO CC MLO CC
- 14. SiemensMedical:fMRIbreastcancerdata 245 Patients: 36% Cancer 414 Patients: 1% Cancer 1027 Patients 0% Cancer 18 Patients: 85% Cancer Model score Log of Patient ID Every point is a candidate Inessence,themostpredictivevariableisthepatientID
- 16. ModelingtheSources… Target Examples Features Source Cancer 1 yes 2 no 1 no 1 no 4 no 3 no
- 21. Multi-ArmedBandit: Explorationvs.exploitation Show some random ads to learn a good model Tradeoff between learning and using
- 27. ClickOptimization:Fumblingin theDark Top 10 Apps by CTR
- 28. How BigData andOptimizationis killingMetrics 90% of clicks are ‘accidental/non intentional’ 10% are meaningful, and changes can be measures Optimization can find structure in the other 90% You will end up with only non-intentional …
- 31. AdvertisingFraud
- 40. Buildtwoseparatemodelsand calculateimpactas thedifference Site Visit yes no no yes no no Site Visit yes no no yes no no Examples1 seenad Examples2 notseenad ExpectedImpact: p(SV|Ad)-p(SV|noad)
- 41. Usepredictivemodelstomeasureimpact Negative Test: wrong ad Positive Test: A/B comparison
- 42. Relationshipoforganicconversionrateand causalimpact -0.001000 0.000000 0.001000 0.002000 0.003000 0.004000 0.005000 0.006000 0.40% 0.50% 0.60% 0.70% 0.80% 0.90% 1.00% 1.10% 1.20% 1.30% 1.40% Organic conversion propensity Additivecasualimpact
- 45. Pleasingtheadvertisingoracle… Audience reports from matched populations in Facebook 68% of the ads where shown to females Makeup for 32% of ads The Oracle
- 46. Data forAudienceOptimization Target Examples Features Gender male female female male male female
- 47. WeightedLogisticRegressionon aggregated Target Examples Features Weight Gender 0.32 male 0.68 female 0.32 male 0.68 female 0.73 male 0.27 female
- 48. HyperlocalTargeting? Foursquare locations: very noisy…
- 51. Catalan traditions pop up everywhere….
- 52. Data forLocationReliabilityinAuction Target Examples Features Reliable? maybe no no maybe maybe no
- 53. Paradox of BigData: “Youneverhave thedatayouwant” Art of making due with second best