O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Solving a business problem in just two weeks
using machine learning
Lucas F Navarro and Pedro Suguimoto
lucas.navarro@getn...
GetNinjas’ Data Science Squad
Product Owner
Bs’ in Computer Science at UFSCar
Ms’ in Artificial Intelligence at
UFSCar
Dat...
Agenda
1. How does GetNinjas Work
2. The current Business Model and why we chose it
3. The big problem that has emerged
4....
How does GetNinjas work?
(Client)
1. Choose what kind
of service that you
need.
2. We indicate up to 4
pros at your region...
How does GetNinjas work?
(Pro)
1. Choose a service
request from the
store
2. Use credits to unlock
the contact of the clie...
OVERVIEW
2,000,000
services hired in the last year
200,000
professionals registered
R$ 210M+
in transactions concluded (GM...
We were performing a staged rollout of a new pay-per-lead model, with
significant improvements to the pros' experience
Old...
Despite offering higher flexibility to pros, the pay-per-lead model can leave some
clients unanswered
Next launch was in
t...
Despite offering higher flexibility to pros, the pay-per-lead model can leave some
clients unanswered
Next launch was in
t...
Despite offering higher flexibility to pros, the pay-per-lead model can leave some
clients unanswered
Next launch was in
t...
The problem gets bigger!
Each point is a “Mini
GetNinjas”, with
100 categories
300 cities
100 categories
x
30k pairs
which...
What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately...
What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately...
What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately...
How could we predict that a client request wouldn’t be purchased?
We had already figured out the reasons that a
service wa...
From R to Python, Fujin was Born!
We came up with a Neural Network with great performance:
○ With this specificity rate, j...
How fast this entire project was delivered?
To have a functional ML model to predict unpurchased requests we spent just ar...
In Its first month live, Fujin were able to predict 68% of the unpurchased requests
Unpurchased
Predicted/
Distributed
Con...
Fujin was also helping to activate Free Pros into Payees
*Numbers from our requests of November/2016
≈R$25k
in purchases f...
Towards to the Future
● The distribution algorithm
○ Pick the best pros to send free requests
○ Determine the amount of fr...
Touchpoints
● The non-bureaucratic/non-hierarchical organization of a StartUp company was
crucial for us to develop Fujin ...
Thanks for your attention!!!
Questions?
Próximos SlideShares
Carregando em…5
×

Solving a business problem in 2 weeks using machine learning — Lucas Fonseca Navarro and Pedro Suguimoto (Getninjas) @PAPIs Connect — São Paulo 2017

889 visualizações

Publicada em

Since Data Science became a very popular field, machine learning (ML) algorithms are being used more than ever in the industry. ML techniques are amazing to solve a bunch of complex business problems efficiently and also in a very fast manner with all of the available tools that we have nowadays. In this presentation, we will show you how we designed and implemented an ML application to solve an emergent problem in just two weeks. We want to show how a dynamic and pro-creativity environment of a StartUp company combined with machine learning can be powerful to create efficient solutions.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Solving a business problem in 2 weeks using machine learning — Lucas Fonseca Navarro and Pedro Suguimoto (Getninjas) @PAPIs Connect — São Paulo 2017

  1. 1. Solving a business problem in just two weeks using machine learning Lucas F Navarro and Pedro Suguimoto lucas.navarro@getninjas.com.br pedro.suguimoto@getninjas.com.br PAPIs Connect - São Paulo 2017
  2. 2. GetNinjas’ Data Science Squad Product Owner Bs’ in Computer Science at UFSCar Ms’ in Artificial Intelligence at UFSCar Data Scientist Mechatronics Engineer from USP/CAASO Data Scientist Bs’ in Applied Maths at UNICAMP PhD’ in Artificial Intelligence at USP/POLI Python Developer Bs’ in Information System at USP/EACH BackEnd Developer Computer Engineer from USP/POLI
  3. 3. Agenda 1. How does GetNinjas Work 2. The current Business Model and why we chose it 3. The big problem that has emerged 4. How we solved this problem in just two weeks 5. Results we achieved 6. Touchpoints
  4. 4. How does GetNinjas work? (Client) 1. Choose what kind of service that you need. 2. We indicate up to 4 pros at your region 3. Hire a Pro to solve your problem
  5. 5. How does GetNinjas work? (Pro) 1. Choose a service request from the store 2. Use credits to unlock the contact of the client 3. Get in touch with the client to be hired
  6. 6. OVERVIEW 2,000,000 services hired in the last year 200,000 professionals registered R$ 210M+ in transactions concluded (GMV) over the last 12 months 100+ categories ranging from electrician to english teacher to lawyer and so on...
  7. 7. We were performing a staged rollout of a new pay-per-lead model, with significant improvements to the pros' experience Old Model Subscription - Pro Pays Fixed Value Per Month - Receives up to 30 contacts New Model Bundle - Pro Purchases a Bundle of credits - Use credits to purchase contacts Vs First Launch: South (Sep/18) Second Launch: North + Center (Oct/23) At last: SP & RJ (2017) Main improvements: ● Pros can purchase how many contacts they want, when they want ● Clients will always receive interested Pros ● More feasible to balance Supply & Demand
  8. 8. Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered Next launch was in two weeks!!! 29% of the requests were not being sold We would have 43.500 clients/month without response (Around 0.5M per year) We made quantitative and qualitative analyses to figure out what causes this rate Some of the possible reasons: ○ Price ○ Location ○ Clients only estimating price ○ Requests lacking a good description ○ Miscategorized services ○ Not enough pros with credits How to solve all of this in two weeks? Our next launch was much bigger and we could not leave our clients unanswered.
  9. 9. Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered Next launch was in two weeks!!! 29% of the requests were not being sold We would have 43.500 clients/month without response (Around 0.5M per year) We made quantitative and qualitative analyses to figure out what causes this rate Some of the possible reasons: ○ Price ○ Location ○ Clients only estimating price ○ Requests lacking a good description ○ Miscategorized services ○ Not enough pros with credits How to solve all of this in two weeks? Our next launch was much bigger and we could not leave our clients unanswered.
  10. 10. Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered Next launch was in two weeks!!! 29% of the requests were not being sold We would have 43.500 clients/month without response (Around 0.5M per year) We made quantitative and qualitative analyses to figure out what causes this rate Some of the possible reasons: ○ Price ○ Location ○ Clients only estimating price ○ Requests lacking a good description ○ Miscategorized services ○ Not enough pros with credits How to solve all of this in two weeks? Our next launch was much bigger and we could not leave our clients unanswered.
  11. 11. The problem gets bigger! Each point is a “Mini GetNinjas”, with 100 categories 300 cities 100 categories x 30k pairs which we have to deal with that problem
  12. 12. What if we could predict that a client request wouldn’t be purchased? 1st option, tell the client the truth “Unfortunately we don’t have Pros available to solve your problem at this moment” 2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔ “GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url” 11% of the Free Pros were purchasing a bundle after seeing the services store at our App Only 4% of our subscribed pros were payees at that time We had the potential to more than double our payees base if we sustained that conversion rate
  13. 13. What if we could predict that a client request wouldn’t be purchased? 1st option, tell the client the truth “Unfortunately we don’t have Pros available to solve your problem at this moment” 2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔ “GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url” 11% of the Free Pros were purchasing a bundle after seeing the services store at our App Only 4% of our subscribed pros were payees at that time We had the potential to more than double our payees base if we sustained that conversion rate
  14. 14. What if we could predict that a client request wouldn’t be purchased? 1st option, tell the client the truth “Unfortunately we don’t have Pros available to solve your problem at this moment” 2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔ “GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url” 11% of the Free Pros were purchasing a bundle after seeing the services store at our App Only 4% of our subscribed pros were payees at that time We had the potential to more than double our payees base if we sustained that conversion rate
  15. 15. How could we predict that a client request wouldn’t be purchased? We had already figured out the reasons that a service was not sold, so we used it as features and started to test some machine learning models using R. Main R packages used: For data prep: ○ tidyr ○ dplyr ML algorithms: ○ monmlp (neural net) ○ e1071 (naivebayes) ○ party (decision trees) Utilities: ○ ROCR (ROC curve) ○ SDMTools (confusion matrix) ○ Boruta (feature selection) ○ Pool + RPostgreSQL (integration for our data warehouse - RedShift) Some of the features: ● Number of Pros with credits* ● Avg Distance from pros* ● Number of highly reviewed pros* ● Price ● Does service has a text description? Class: Was the request sold? (boolean) We used Rstudio Server remotely from a m4.large EC2 machine at Amazon’s Cloud *Just pros inside the request area (inside a radius)
  16. 16. From R to Python, Fujin was Born! We came up with a Neural Network with great performance: ○ With this specificity rate, just 30% of the unsold requests clients would remain without an answer ○ With this sensitivity rate, we would distribute only 10% of the good requests for free After we validated the model, we implemented it in python to go live: ○ We used the basic DS set (scikit-learn, pandas, numpy) ○ Alse we used Neurolab for the MLP algorithm ○ We used Flask to create the prediction microservice 70% of specificity First Model 79% of accuracy *Python is the main Data Science language at GetNinjas 90% of sensitivity We called the project Fujin!
  17. 17. How fast this entire project was delivered? To have a functional ML model to predict unpurchased requests we spent just around 2 weeks! *The first distribution algorithm was a random distribution to Free Pros
  18. 18. In Its first month live, Fujin were able to predict 68% of the unpurchased requests Unpurchased Predicted/ Distributed Contacted Hired 41% 28% 9% 5% Service Requests With north+center regions, unpurchased request rate grown from 29% to 41% due sparsity of S&D in this new regions With Fujin online, the number of unanswered clients were just 13% (instead of 41%) Fujin also helps to increase our Gross Merchandise Volume(GMV) through hiring rate *Numbers from our requests of November/2016
  19. 19. Fujin was also helping to activate Free Pros into Payees *Numbers from our requests of November/2016 ≈R$25k in purchases from Pros after they received a free client contact from Fujin 65% of this pros didn’t even had our App installed ⚠ Only 14% of the distributed leads were read by the Pros Already had our App installed Downloaded App post Fujin lead Didn`t Download
  20. 20. Towards to the Future ● The distribution algorithm ○ Pick the best pros to send free requests ○ Determine the amount of free pros to send the contact from a request given the Fujin’s feedback ● Enhance the model with more features ○ We have more service categories with distinct characteristics nowadays ○ We have SP and RJ state (higher density regions than before) ● Features to increase the visualization rate of free distributed requests ○ Such as adding a deeplink to our the contact screen on our App at the e-mail/sms sent to Pros
  21. 21. Touchpoints ● The non-bureaucratic/non-hierarchical organization of a StartUp company was crucial for us to develop Fujin in such short time ● ML is amazing, in just two weeks we had a running model to solve the biggest part of our problem ● ML is super cool, but most of the times we solve our problems with much simpler approaches ⚠ Do not create a super complicated solution and later adjust your problem to it, start from the problem, and climb to the complex solution as you need it.
  22. 22. Thanks for your attention!!! Questions?

×