Early talk given at Fifth Elephant Winter Edition at Mumbai in Jan, 2019

1. Reducing Cost of Production ML:
Feature Engineering Case Study
Dr. Venkata Pingali
Scribble Data
1

2. Outline
© Scribble Data 2018
● Production ML is Complex
● Feature Engineering Overview
● Detailed Cost Drivers
● Indicative Quantitative Improvement
● Approaches for Each Cost Driver
2
Takeaway: A disciplined approach will deliver 10x improvement

3. Production ML - Complex and Expensive
© Scribble Data 2018https://eng.uber.com/michelangelo/
3
⇑
⇓

4. Distribution of Challenges
© Scribble Data 2018https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
4
“Only a small fraction of real-world ML systems is composed of the ML code, as shown by the
small black box in the middle. The required surrounding infrastructure is vast and complex.”
Paper from Google - NeurIPS 2015
Drivers: Speed, Correctness, Evolution, Scale

5. Feature Engineering
© Scribble Data 2018
5

6. Feature Engineering - Nature
© Scribble Data 2018
● Features are variables generated from data
○ Continuous process (Batch + Near Realtime + Realtime)
● Large in number (‘00s to ‘000s) & evolving
● Frequently executed
Customer SKU Name
17826162 0293192 Thai Dragon Fruit
Customer Premium Imported
17826162 15% of txns 5% of spend
Retail Customer
(X GB)
Features
(~X/1000)
6

7. Historical
1
N
Models
….
7
Near RT
RT
Multi-scale Features
(1 min, 1 day, 1 year)
Multi-Phase Feature Engineering

8. Emerging Area of Work
© Scribble Data 2018
● Conceptual
○ Functional Data Engg (Beauchemin, Lyft)
○ Positive/Negative Data Engg (Lowin, Prefect)
○ Design discussions: Michelangelo (Uber), Zipline (AirBnB), TFX (Google)
● Implementations
8
Data Science platforms
Inhouse Systems
ML Engineering Platforms

9. Feature Engineering Cost Drivers
© Scribble Data 2018
Cost Driver Explanation
Correctness & Reconciliation Confidence in output
Evolution of data/algorithms/ features Safely scaling
Development Best use of time
Operations incl Backfilling Controlled execution
All are forms of end-to-end (in)discipline & known
9

10. Usecase - Customer Segmentation
© Scribble Data 2018
Gain (2015 ⇒ 2018)
Data Volume 10x+
Feature Output Size 10x
Dev Time 2x
Server Cost 10x
Bonus
Auditability, Idempotency
Documentation, Notifications
10
Data from Tier I Retail (2018) & F&B (2015) chains w/ 1500 Stores

11. Gaining Confidence in Output
© Scribble Data 2018
● End-to-end auditability
○ All data and all runs
○ Metadata standardization
● Discovery and reuse
○ Pipelines, modules
○ Automatic doc & ownership tracking
● Early warning systems
○ Input/output quality checks
○ Note critical decisions
11

12. Scaling Deployments
© Scribble Data 2018
● Isolation: Multi-tenant
namespaces
○ File system, tables, S3
○ Runs
● Time: Versioned namespaces
○ Storage locations
○ Metadata
● Version tracking of all data/code
○ Link data and code
12
Prepare for pipelines, models, versions, & dataset proliferation
Functional Data Engineering - Maxime Beauchemin

13. Increasing Development Speed
© Scribble Data 2018
● Feature DSL
○ Reduced code and errors
● Reusable assets
○ GUI, preprocessing
○ Standard transforms
● Standardized development
○ Unit testing
○ Automated documentation
13

14. Control Execution
© Scribble Data 2018
● Execution management
○ Parameterization, over-rides
○ Managed data access
● Automated deployment
○ Rollout best practices
● Minimum service integration
○ Performance, background tasks,
scheduled execution
● Notifications
○ With callouts
14

15. © Scribble Data 2018
15
● Production ML is expensive
○ Cost drivers: Speed,Correctness, Evolution, Scaling
● Feature engineering important component
○ 10x productivity improvement possible
● Platforms required to improve speed & reduce risk
○ All major companies have platforms
Key Takeaways

16. THANK YOU
FOR YOUR TIME
DENVER BANGALORE
Littleton Indiranagar | HSR
pingali@scribbledata.io
16

17. Feature Engineering
© Scribble Data 2018
17

18. Feature Engineering - Key Activity
© Scribble Data 2018
Source:https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation
-most-time-consuming-least-enjoyable-data-science-task-survey-says/
Modeling = Statistics over Features
Typical
DE:DS = 2-3:1
18

19. © Scribble Data 2018
2015* 2018 Improvement (2015>2018)
Usecase Customer Segmentation Customer Segmentation
Organization Tier I National F&B Chain Tier I National Retail Chain
Raw Data 100GB Pos Txns 1.1TB PoS Txns 11x
Output 500K x 100 Features 1M x 500 Features 10x
Compute Engine Hive+Pandas Pandas+Itertoolz
Dev Timeframe 3 Months 1.5 Months 2x
Compute 6 Node Cluster x 3 hours 1 Server x 1.5 hour ~ 10x
Bonus
Auditability, Idempotency
Documentation
2018 Implementation strongly informed by 2016
19

20. Typical Feature Engineering Cycle
© Scribble Data 2018
Change From Past: In Production, Everyday
20
Catalog
Compute
Features
Test Algorithm
Preprocess
Data
Refine
Usecase
Deploy &
Operate
Dataset

21. Production Feature Engineering Challenges
© Scribble Data 2018
21
Growing Volume
& Complexity,
Changing
SchemaCatalog
Compute
Features
Test Algorithm
Preprocess
Data
Refine
Usecase
Deploy &
Operate
Dataset

22. © Scribble Data 2018
22
Production Feature Engineering Challenges
Data loss,
inconsistenciesCatalog
Compute
Features
Test Algorithm
Preprocess
Data
Refine
Usecase
Deploy &
Operate
Dataset

23. © Scribble Data 2018
23
Production Feature Engineering Challenges
Catalog
Compute
Features
Test Algorithm
Preprocess
Data
Refine
Usecase
Deploy &
Operate
Dataset
Growing
number, type,
& complexity,

24. © Scribble Data 2018
24
Production Feature Engineering Challenges
100s-1000s
Entity common,
growing
Catalog
Compute
Features
Test Algorithm
Preprocess
Data
Refine
Usecase
Deploy &
Operate
Dataset

25. © Scribble Data 2018
25
Production Feature Engineering Challenges
Continuously
Tuned, Growing
number
Catalog
Compute
Features
Test Algorithm
Preprocess
Data
Refine
Usecase
Deploy &
Operate
Dataset

26. © Scribble Data 2018
26
Production Feature Engineering Challenges
Manage versions,
outputs, debug
errors
Catalog
Compute
Features
Test Algorithm
Preprocess
Data
Refine
Usecase
Deploy &
Operate
Dataset

27. Feature Engineering
© Scribble Data 2018
27
Only a small fraction of real-world ML systems
is composed of the ML code, as shown by the
small navy blue box in the middle. The
required surrounding infrastructure is vast and
complex.ML
Code
Feature
Extraction
Data
Collection
Analysis
Tools
Data
Verification
Process Management Tools
Machine Resource Management
ServicingInfrastructure
Configuration
Monitoring