mabl's Machine Learning Implementation on Google Cloud Platform
•
0 gostou•148 visualizações
mabl uses machine learning on Google Cloud Platform to enable continuous testing at scale. It detects anomalies and automatically heals broken test scripts. mabl analyzes observations and measurements to detect state changes and recognize significant anomalies. It trains models using Cloud ML Engine and stores metadata in Datastore. mabl's cloud architecture is serverless, decoupled, and event-driven to harness Google Cloud capabilities and scale processing to millions of pages per month with a small team.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a mabl's Machine Learning Implementation on Google Cloud Platform
Semelhante a mabl's Machine Learning Implementation on Google Cloud Platform (20)
Último
Último (20)
mabl's Machine Learning Implementation on Google Cloud Platform
- 1. mabl’s ML Implementation Testing @ Scale with Google Cloud Platform Joseph Lust mabl engineer @lustcoder
- 2. About Me: Joseph Lust Engineer ■ Building the web for two decades ■ Cloud native since 2011 ■ Currently building mabl ML cloud on GCP ■ Co-organizer GDG Cloud Boston Meetup
- 3. Agenda ■ mabl’s problem statement ■ ML on Google Cloud ■ Cloud best practices at mabl ■ Q & A
- 5. Developing Quality Software is Slow Image Source: IBM SAGE ■ Software is complex ■ Testing software is slow ■ Humans have low clock speeds ■ Humans are expensive
- 6. Continuous Delivery breaks QA ■ Change happens infrequently ■ Weeks or months to write tests ■ Weeks to execute tests ■ Change is constant ■ Hours to write tests ■ Minutes to execute tests
- 7. The Doughnut Hole: Continuous Testing (CT) Source: Google Trends Hudson Released Facebook CD Article
- 8. Continuous Testing Roadmap ■ Highly parallel testing (fail fast) ■ Automatically heal broken scripts ■ Automatically generate test scripts Manual Testing Manually Scripted Testing Playback Testing Auto Test Generation Auto-healing Testing ✔ ✔ ✔ ✔
- 9. the ML @ mabl
- 10. Machine Learning Areas @ mabl ■ Anomaly detection ▲ Timing anomalies ▲ Visual change anomalies ■ Auto-healing tests ▲ Alternative element candidate election ▲ Multi-branch evaluation
- 14. Why anomaly detection isn’t quite that easy ■ Domain variability one app’s anomaly is another’s normal ■ Temporal variability today’s anomaly is tomorrow’s normal ■ Detection quality not all anomalies are bad (or important) ■ Initial quality what can we detect with minimal app-specific data
- 15. Approach ■ Modeling the app ▲ domain variability, quality ■ Incremental learning ▲ temporal variability, initial quality ■ User feedback ▲ use case variability, quality
- 16. Analysis pipeline Observations ▲ Raw data/events Measurements ▲ Calculated features State detections ▲ Aggregate measurements ▲ Apply detection models (abstraction) ▲ Integrate detections (higher abstraction) Insights ▲ Recognize significant state changes
- 17. Signals for anomaly detection ■ Visual appearance ■ Performance ■ Failures ■ Errors (non-fatal) ■ Page structure changes
- 18. visual anomalies
- 30. model training and feedback
- 31. Pieces of the puzzle ■ Cloud ML Engine ▲ Tensorflow machine learning models ▲ Managed model training ▲ Online prediction service ■ Dataflow ▲ Streaming data processing ▲ I/O connectors for PubSub, etc. ■ Datastore ▲ Augment trained model with mutable data ▲ Fast query by type and filter (e.g., by key)
- 32. Architecture of model feedback loops
- 33. harnessing google cloud for ML
- 34. mabl cloud design tenets ■ Serverless ■ Decoupled sub systems ■ Event driven architecture
- 35. Serverless just works ■ No Provisioning ■ Transparent Scaling ■ Event Driven ■ Pay only for Use 35
- 36. e.g. Kubernetes Engine ■ NoOps - developers’ containers in prod ■ Language agnostic ■ Massively scale ▲ running millions of containers a month ■ “Set it and forget it!” 36 Kubernetes Engine
- 37. Decoupled Architecture ■ Message passing ■ Polyglot development ■ Independent deployment ■ Minimize blast radius Cloud Pub/Sub Cloud Functions
- 38. Event Driven Architecture ■ No “cron jobs” or batch processes ■ Continuous pipelines ■ Surge buffers ■ 100s ms to 1-2s E2E handling 38 Cloud Dataflow Cloud Functions
- 39. Google Cloud’s Impact on mabl ■ From zero to alpha product in ~6mo with 8 developers ▲ Processing ~100M pages per month ■ Systems designed for scale on day one ■ No dreaded “rewrite” ■ Product families work very well together out of the box
- 40. Questions?