O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Using language models to supercharge Monzo’s customer support

1.461 visualizações

Publicada em

Presentation @ PyTorch London, November 2019

Publicada em: Engenharia
  • Entre para ver os comentários

Using language models to supercharge Monzo’s customer support

  1. 1. Using language models to supercharge Monzo’s customer support @neal_lathia November 5, 2019
  2. 2. See this post by @jackkleeman https://monzo.com/blog/we-built-network-isolation-for-1-500-services
  3. 3. Many different machine learning problems!
  4. 4. Enabling swift, delightful customer service.
  5. 5. 1. Helping customers find the right answers to their queries in the app. // 2. Helping agents to diagnose and response to customer queries swiftly.
  6. 6. Two approaches we are working on
  7. 7. ⬅ Prompt for information ⬅ Recommendations (search results) ⬅ Self-selected chat closure ⬅ The chat first turn (this could be more than one message)
  8. 8. Formulation Article recommendations as a relevance problem: can we improve our search results?
  9. 9. General Approach The customer query is turned into a numerical representation (an embedding). We also keep all of the latest help article embeddings. The top recommendations are the help articles that have the most similar embeddings (Cosine distance ranking).
  10. 10. You can change your card PIN at any large bank (HSBC, Barclays, etc.) ATM in the UK by selecting PIN services ☺ You can see and download your bank statements through the Monzo app. How do I change my PIN? Question + Answer - Answer
  11. 11. Baseline Train an encoder-decoder model from scratch, using Monzo chat data only Challenger Use a pre-trained BERT encoder, and finetune it using Monzo Chat data Result! Challenger ⬆ self-service rate by ~9%
  12. 12. Experimental classification approach
  13. 13. From our ongoing analysis, we can identify topics that are: Eligible for self-service (e.g., PIN reset) Predictable (in our chat data) Valuable for customer service (i.e., accounts for a high volume of their work) ... perhaps we could approach this as a classification problem instead?
  14. 14. We were looking for an approach that is: 🎯 Accurate We were looking for a way where: 🚢 We could ship quickly to get some customer feedback 🏦 Use all the relevant state that is stored across many backend services
  15. 15. Approach Finetune a single binary classifier about a topic: is the customer asking about needing a replacement card? If they are (& a card hasn’t already been ordered for them), then give them the answer directly!
  16. 16. Offline validation Precision Out-of-sample precision Card replacement 0.86 0.69 Card not arrived 0.93 0.60 Update details 0.93 0.52 😫 ... ... ... We deployed a number of these models in shadow prediction mode. This allows it to make predictions on live data, but it’s not sending customers any answers.
  17. 17. What the 🤬? The main culprit was poorly tagged conversations. But the definition of “correctly” tagged was subjective!
  18. 18. Large set of noisy tags Small set of re-labelled examples Hi there! How can I transfer money into a savings pot?
  19. 19. F1 Score Fine tune on tags 69 Fine tune on re-labelled examples 78 “Ultra” fine-tune on both 85 Example result: Update details (48% of tagged chats labels changed when re-labelled)
  20. 20. Conclusion - how do we enable this work?
  21. 21. Orchestrator Classifier for A Classifier for B Classifier for C
  22. 22. Orchestrator Classifier for A Classifier for B Classifier for C 0.0% 8.0% 98.0% Topic: C
  23. 23. Orchestrator Classifier for A Classifier for B Classifier for C Winner! Rules that determine eligibility for topic C
  24. 24. Orchestrator Classifier for A Classifier for B Classifier for C Winner! Rules that determine eligibility for topic C Classifier for D
  25. 25. 1. Helping customers find the right answers to their queries (in the app) // 2. Helping agents to diagnose and response to customer queries swiftly.
  26. 26. Thanks! @neal_lathia October 8, 2019

×