Building an Enterprise Data Platform with Azure Databricks to Enable Machine Learning and Data Science at Scale at Sam’s Club
•
3 likes•2,021 views
This document summarizes Walmart's transition to building an enterprise data platform on Azure Databricks to enable machine learning and data science at scale. Previously, Walmart had a complex and slow legacy technology stack. The new platform goals were to centralize data in the cloud, increase productivity with data science tools, and reduce costs. Key aspects of the new platform included using Azure and Databricks for data processing and machine learning, Airflow for orchestration, and building several machine learning models for applications like fraud detection and product recommendations. Challenges in the transition included optimizing performance and managing resources across the platforms.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building an Enterprise Data Platform with Azure Databricks to Enable Machine Learning and Data Science at Scale at Sam’s Club
Similar to Building an Enterprise Data Platform with Azure Databricks to Enable Machine Learning and Data Science at Scale at Sam’s Club (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Building an Enterprise Data Platform with Azure Databricks to Enable Machine Learning and Data Science at Scale at Sam’s Club
- 1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics
- 2. Andrew Ray Craig Covey Building an Enterprise Data Platform with Azure Databricks to Enable Machine Learning and Data Science at Scale #UnifiedAnalytics #SparkAISummit
- 3. Intro Andrew Ray Craig Covey 3#UnifiedAnalytics #SparkAISummit
- 4. The Problem 4#UnifiedAnalytics #SparkAISummit Dependent on Walmart Data Everywhere Old Tech Stack Slow Data Science
- 5. Previous State Technology managed by Walmart • 10+ Big Hadoop Clusters • Multiple large instances of Teradata • Data dispersed in operational DB’s – Oracle, MySQL, MS SQL, DB2, Informix, SAP 5#UnifiedAnalytics #SparkAISummit
- 6. Platform Build Goals 6#UnifiedAnalytics #SparkAISummit Cloud Focus Centralize Sam's Club Data Tools for data science and analysis Increase Productivity
- 8. 8#UnifiedAnalytics #SparkAISummit Current State Azure + Databricks + Spark + Airflow + Hadoop
- 9. Azure 9#UnifiedAnalytics #SparkAISummit Reduced infrastructure costs In the Microsoft ecosystem Spin up and destroy Managed Services Active Directory Integration
- 10. Current State 10#UnifiedAnalytics #SparkAISummit Databricks is central to everything we do! data airflow jobs apps ML analysis Azure Databricks 2 Regions 10 Workspaces 100+ Users 50+ Scheduled Jobs 1,000+ Notebooks Scores of ML Models
- 11. Use Cases 11#UnifiedAnalytics #SparkAISummit Fraud Engine Credit Cards & Account Takeover Fresh Sales Tool Fresh Food Production Plan Assortment What to Sell in Clubs Personalization Product Recommendations Finance Financial Forecasts
- 13. Airflow 13#UnifiedAnalytics #SparkAISummit 3 Airflow Environments (dev, stage, prod) Hosted in Walmart's internal cloud Code in GitHub Running on Docker 180+ DAGs 600,000,000,000+ rows 150+ TB 6 Sources Zero to Prod in ~2 Months
- 14. Airflow DAG Workflow 14#UnifiedAnalytics #SparkAISummit Create DAGs dynamically Various Hadoop jobs move the data Send data to Azure Blob to two regions in parallel Databricks notebooks verify source and Azure data are statistically nearly equal
- 15. Airflow DAG Setup 15#UnifiedAnalytics #SparkAISummit
- 16. Databricks Workspaces 16#UnifiedAnalytics #SparkAISummit Tables in Databricks are created with a specific path in DBFS that is mounted to a container in Azure Blob Storage Multiple Databricks Workspaces to segregate access Blob containers are mounted with read only SAS tokens to protect data Data lives in Blob Storage CREATE TABLE db1.tb1 USING delta LOCATION ‘/mnt/general/...’ Create Delta tables
- 17. Databricks Notebooks • Hybrid of Jupyter notebooks and Google Docs • Allows for collaborative editing – Easy technical support – Remote pair programming • Multiple languages in one notebook 17#UnifiedAnalytics #SparkAISummit
- 18. Databricks Delta 18#UnifiedAnalytics #SparkAISummit Performance Queries run fast! Transactional Updates No downtime or consistency issues during updates Change Data Capture MERGE will make it easy Next-gen format built on top of Spark for batch and streaming big data use cases 1+ hours! <6 seconds!
- 19. 19#UnifiedAnalytics #SparkAISummit Use Cases in Production Using Azure & Databricks
- 20. Finance 20#UnifiedAnalytics #SparkAISummit Purpose high-quality financial forecasts for sales and wage categories Machine Learning gradient boosting, statistical time-series forecasting algorithms Production Databricks to analyze data, train ML models, run production scoring jobs
- 21. Assortment 21#UnifiedAnalytics #SparkAISummit Purpose Select products and inventory levels to maximize profit/sales within finite shelf space Machine Learning Expectation-Maximization (EM), Catboost (boosting to estimate True Demand) Production Databricks to manipulate data, train ML models, run production scoring jobs
- 22. Fresh Sales Tool 22#UnifiedAnalytics #SparkAISummit Fresh Sales Tool Job Current System (Spark R, Spark Scala, Databricks Delta) Previous System (Hive Query Language, Hadoop) Fresh Forecasting 40 Minutes ~ 7 Hours Minimum Presentation Forecasting 3 Minutes ~ 1 Hour Ly 8-week sales Report 1 Minute ~ 20 Minutes Admin Dashboard report 5 Minutes ~ 4 Hours $1,000,000+ per year Initial projected cost ~$100,000 per year Job scoped clusters & code optimizations
- 23. 23#UnifiedAnalytics #SparkAISummit Future Plans To the infinity and beyond!
- 24. Access Control 24#UnifiedAnalytics #SparkAISummit Databricks Role Based Access Control PHASE 2 Sync groups for use with table access control CURRENT Sperate Workspaces for projects that need special data PHASE 1 Provision/deprovision users to workspaces with AD groups
- 25. Disaster Recovery Ú⌅[ åˇ &cã[ } Æ åæcæÆã•Æ æ|⌅^æåˆ Æ ⌅^] |ã&æc^å 25#UnifiedAnalytics #SparkAISummit Œ⌅&@ãç^Æc[ Æà|[ àÆ •c[ ⌅æ* ^K • ⇤ [ c^à[ [ • • ⇥ãà⌅æ⌅ã^• • R[ à• • Ô|ˇ •c^⌅• • W•^⌅Æåæcæ
- 26. JupyterHub on AKS 26#UnifiedAnalytics #SparkAISummit Deploy on Azure Kubernetes Service Multi-user hub that spawns Jupyter notebook servers Targeting non-distributed workloads Flexible Cloud Independent Data Source Connectivity Low Cost Open Source Preferred solution for non-distributed GPU needs Benefits Python & R Notebooks TensorFlow Easy access to data via Databricks
- 28. Challenges 28#UnifiedAnalytics #SparkAISummit Airflow is not easy Moving at scale Wrangling Airflow DAG code Managing DDLs across Databricks workspaces
- 29. Lessons Learned 29#UnifiedAnalytics #SparkAISummit Unoptimized ORC files with blob storage Rouge clusters on Databricks Azure blob storage performance is limited Most built in Airflow operators are useless
- 30. Thank You! Andrew.Ray@samsclub.com Dallas Bentonville Silicon Valley Austin https://careers.walmart.com/We are hiring!
- 31. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT