O slideshow foi denunciado.

Rsqrd AI: How to Design a Reliable and Reproducible Pipeline

0

Compartilhar

Próximos SlideShares
What is MLOps
What is MLOps
Carregando em…3
×
1 de 42
1 de 42

Rsqrd AI: How to Design a Reliable and Reproducible Pipeline

0

Compartilhar

Baixar para ler offline

In this talk, David Aronchick, co-founder of Kubeflow and Microsoft's Head of Open Source ML, talks about designing reproducible and reliable ML pipelines. He speaks about the importance and impact of MLOps and use of metadata in pipelines. He also talks about a library he wrote to help with this problem, MLSpecLib.


**These slides are from a talk given at Rsqrd AI. Learn more at rsqrdai.org**

In this talk, David Aronchick, co-founder of Kubeflow and Microsoft's Head of Open Source ML, talks about designing reproducible and reliable ML pipelines. He speaks about the importance and impact of MLOps and use of metadata in pipelines. He also talks about a library he wrote to help with this problem, MLSpecLib.


**These slides are from a talk given at Rsqrd AI. Learn more at rsqrdai.org**

Mais Conteúdo rRelacionado

Semelhante a Rsqrd AI: How to Design a Reliable and Reproducible Pipeline

Livros relacionados

Gratuito durante 30 dias do Scribd

Ver tudo

Rsqrd AI: How to Design a Reliable and Reproducible Pipeline

  1. 1. Building Reproducible ML with MLOps and Metadata
  2. 2. SpeechVision Language Switchboard Switchboard cellular Meeting speech IBM Switchboard Broadcast speech 1993 20172000 2006 2010 5.1% Switchboard speech recognition test 96% RESNET vision test 152 layers 88.5% SQuAD reading comprehension test 69.9% MT research system 2016 Object recognition Human parity 2017 Speech recognition Human parity 2018 Machine reading comprehension Human parity 2018 Machine translation Human parity Microsoft ML breakthroughs
  3. 3. Microsoft 365 ML at Microsoft Research
  4. 4. But ML is HARD!
  5. 5. Building a model
  6. 6. Building a model Data ingestion Data analysis Data transformation Data validation Data splitting Trainer Model validation Training at scale LoggingRoll-out Serving Monitoring
  7. 7. Ok, but, like, I’m a data scientist. IDGAF I don’t care about all that.
  8. 8. Yes You Do!
  9. 9. Cowboys and Ranchers Can Be Friends! SRE/ML EngineersData Scientist • Quick iteration • Frameworks they understand • Best of breed tools • No management headaches • Unlimited scale • Reuse of tooling and platforms • Corporate compliance • Observability • Uptime
  10. 10. MLOps
  11. 11. MLOps = ML + DEV + OPS Experiment Data Acquisition Business Understanding Initial Modeling Develop Modeling Operate Continuous Delivery Data Feedback Loop System + Model Monitoring + Testing Continuous Integration Continuous Deployment ML
  12. 12. A Pipeline You Say?
  13. 13. Does My Model Actually Work? SRE/ML EngineersData Scientist Time to test out my model… Laptop The Cloud
  14. 14. Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud Looks good to me! To Production!
  15. 15. What is happening… Source Control Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud
  16. 16. A Small Example of Issues You Can Have… • Inappropriate HW/SW stack • Mismatched driver versions • Crash looping deployment • Data/model versioning [Nick Walsh] • Non-standard images/OS version • Pre-processing code doesn’t match production pre-processing • Production data doesn’t match training/test data • Output of the model doesn’t match application expectations • Hand-coded heuristics better than model [Adam Laiacano] • Model freshness (train on out-of-date data/input shape changed) • Test/production statistics/population shape skew • Overfitting on training/test data • Bias introduction (or not tested) • Over/under HW provisioning • Latency issues Or It Just Doesn’t Work! At All! • Permissions/certs • Failure to obey health checks • Killed production model before roll out of new/in wrong order • Thundering herd for new model • Logging to the wrong location • Storage for model not allocated properly/accessible by deployment tooling • Route to artifacts not available for download • API signature changes not propagated/expected • Cross-data center latency • Expected benefit doesn’t materialize (e.g. multiple components in the app change simultaneously) • Get wrong/no traffic because A/B config didn’t roll out • No CI/CD; manual changes untracked [Jon Peck] • Get too much traffic too soon (expected to canary/exponential roll out) • Outliers not predicted [MikeBSilverman] • Change was a good change, but didn’t communicate with the rest of the team (so you must roll back) • No dates! (date to measure impact/improvement against a pre-agreed measure; date scheduled to assess data changes) [Mary Branscombe] • LACK OF DOCUMENTATION!! (the problem, the testing, the solution, lots more) [Terry Christiani] • Successful model causes pain elsewhere in the organization (e.g. detecting faults previously missed) [Mark Round] • Lack of visibility into real-time model behavior (detecting data drift, live data distribution vs train data, etc) [Nick Walsh]
  17. 17. Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud Source Control Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment Nice. Nice. ✔
  18. 18. But I Can Do All These Manually…
  19. 19. No.
  20. 20. MLOps is a Platform and a Philosophy Even if: • Every data scientist trained... • And you had all the tools necessary... • And they all worked together... • And your SREs understood ML modeling... • And and and and ... You’d still need a permanent, repeatable record of what you did
  21. 21. That’s MLOps!
  22. 22. Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud Source Control Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment Nice. Nice. ✔ What goes here?
  23. 23. Metadata!
  24. 24. Metadata is ... A contract for the interface of a service A historical record of the outcome of a process 3. Structured data that allows for (more) reliable automated workflows 4. And much much more...
  25. 25. Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud Source Control Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment Nice. Nice. ✔
  26. 26. Haven’t Convinced You Yet?
  27. 27. What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer I’d Like a loan, please. Source Control
  28. 28. What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer No. Source Control
  29. 29. What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Ok, but why? Source Control
  30. 30. Source Control What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Uh oh. Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer LawyerLawyer
  31. 31. It’s Not Just About Explainability! • Yes, models are complicated • But, that’s not enough: • What data did you train on? • How did you transform/exclude outliers? • What are the data statistics? • Did anything change between code and production? • What model did you actually serve (to this person)? • Metadata can help!
  32. 32. What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Source Control Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment
  33. 33. 32c04681d7573 Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Source Control Immutable Metadata Store b151f8e65b32a c7f4e7607b4b7 0ef1d58921d89 e2e1e994c4251 786c8e57a6d51 9ce88802f0759 32c04681d7573
  34. 34. Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Source Control Immutable Metadata Store b151f8e65b32a c7f4e7607b4b7 0ef1d58921d89 e2e1e994c4251 786c8e57a6d51 9ce88802f0759 32c04681d7573 Why didn’t I get a loan? 32c04681d7573
  35. 35. What Did My Customers See? SRE/ML Engineers Front End Model Server Customer Immutable Metadata Store 32c04681d7573 32c04681d7573 Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment The Cloud Source Control b151f8e65b32a c7f4e7607b4b7 0ef1d58921d89 e2e1e994c4251 786c8e57a6d51 9ce88802f0759 32c04681d7573
  36. 36. Metadata Gives You a Repeatable Record • What data you trained on • How you transformed it for training • What the results of the training were • What kind of fairness tests you ran • How those results compared with previous results • How you rolled it out • Which version a customer saw • And, and, and ... All Automatically! (Mostly)
  37. 37. Ok, but you can’t possibly expect me to use YAML.
  38. 38. Introducing MLSpecLib A simple, Python-native library for using with schematized objects • Extends marshmallow (minimum rewriting) • Comes with some standard schemas in the box • It started with ML but it works for anything But wait there’s more! • Read/write serialized objects natively with Python (using dot notation and everything) - No YAML! No JSON! • User friendly, trivially extensible schema language - including importing from a remote store • “Lazy” enforcement (at load/save time only) • Code-gen for the REALLY lazy (like me)
  39. 39. ENOUGH TALK. GET TO THE DEMO.
  40. 40. Come Help!
  41. 41. me: David Aronchick (aronchick@gmail.com) twitter: @aronchick apps: http://mlops-github.com/ mlspec-lib on pypi: https://pypi.org/project/mlspeclib/ mlspec-lib on github: https://github.com/mlspec/mlspec-lib THANK YOU!

×