The ODAHU project is focused on creating services, extensions for third party systems and tools which help to accelerate building enterprise level systems with automated AI/ML models life cycle.

1. EPAM ML/AI Accelerator
Open Data Analytics Hub
ODAHU
Dmitrii Suslov
Vladislav Tokarev
January 2020

2. AGENDA
1 G E N E R A L M L /A I P R O D U C T L I F E C Y L E
2
3
6
M L /A I P R O D U C T S O P E R AT I O N A L I Z AT I O N
C H A L L E N G E S
O DA H U K E Y F E AT U R E S I N 1 . X R E L E A S E
O DA H U F O R I N T E R N A L E PA M P R O J E TC S
8 D E M O
9 Q & A
4 O DA H U A R C H I T E C T U R E
5 O DA H U R OA D M A P
7 C O O P E R AT I O N A N D C O N T R I B U T I O N
2

3. GENERAL ML/AI PRODUCT LIFECYCLE
Ideation
Phase
Data Preparation
Phase
Data Exploratory
Phase
ML Models Training &
Tuning Phase
AI Products
Development &
Integration Phase
AI Products
Production Phase
AI product feedback loop
ML model
selection & tuning loop
AI product development cycle
Define AI
product
Collect AI
product
requirements
Discover
available
datasets
Develop and
deliver data
ETL pipelines
Deliver data
product
Prepare
training and
testing data
sets
Evaluate and
choose right
ML algorithms
Train, test
and tune ML
model
Build binary
file with
trained
model
Build AI services
with trained ML
models wrapped
into RESTful service
and Docker
containers
Build AI products
with families of AI
services and
automation
pipelines
Deploy and test AI
services and
products
Deliver AI products in
production
- Service mesh
- A/B testing
- Traffic mirroring
- Req. & Resp.
logging
Monitoring &
Alerting
Collect feedback and
monitor prediction
accuracy
Automate ML
CI pipelines
3

4. COMMON ML/AI PRODUCTS OPERATIONALIZATION CHALLENGES
• CICD for ML models
• Dependency management
• ML training experiments evaluation
• Keeping track of data and models
• Packaging models for different target environments
• Scaling ML model training and runtime environment
• Enterprise level infrastructure: automated, secured, multi-tenant, scalable, manageable,
etc..
4

5. KEY FEATURES IN RELEASE 1.X
• Pluggable ML toolchains system and Mlflow support
• Kubernetes native services for training, packaging, deploying ML models with APIs in OpenAPI (ex.
Swagger) specification
• AI service catalog
• Connections manager
• SDKs generated from OpenAPI specifications and command line tool
• ML feedback loop components
• GPU for ML training loads in K8S
• Horizontal scaling with Knative for models deployed as services in kubernetes
• Advanced traffic routing schemas with Istio for ML models deployed as RESTful AI services
• Plugin for JupyterLab
• Plugin for Airflow
• SSO with OpenID Connect protocol
• System monitoring
• Deployment automation in major kubernetes platforms: GCP GKE, AWS EKS, Azure AKS
• Open source under Apache 2.0 https://github.com/odahu
• Open documentation https://docs.odahu.org/
5

6. HIGH-LEVEL LAYERED COMPONENTS VIEW ON ODAHU
Plugins for ML IDEsCommand line tools Plugins for workflow and CICD engines
SDKs SDKs (Python, Go and other languages) generated from ODAHU OpenAPI specifications (Ex. Swagger)
Core
Components
Training
ML models
Infrastructure deployment
automation
Packaging
ML models
External
Systems
Deploying
ML models
ML training clusters (K8S, Spark, HPC, others)
Infrastructure AWS Azure GCP
Feedback
loop
On-Premise
KMS SSO
AI runtime clusters (K8S, Spark, Hadoop, others)
ML frameworks (Mlflow, Sklearn, TensorFlow, others) Data sources (Object storages, DBs, File systems, others)
VSC (github gitlab, bitbucket, TFS, others)
Docker registries Package registries ETL CICD
Web control panel
ML scripts
ODAHU componentLegend: External component Custom scripts of ML project Logical group Depends
Connections
manager
Monitoring
Alerting
Logging
ML pipelines CICD pipelines
User Facing
Components
ML/AI Project
Components
ML/AI productsData pipelinesODAHU manifests
6

7. ODAHU SERVICE FOR ML MODEL TRAINING
Data Scientist IDEODAHU Command Line Tool
Core
Components Training ML Model Service
External
Systems
User Facing
Components
Connections ManagerAudit Service
orchestrate ML training jobs
get credentialssend audit info
get ML
scripts
get
data
send package with ML model
send log msg & metrics from cluster
send ML training metrics
Version Control System
Data Source
Compute cluster
Package repository
ML metrics tracking system
Cluster monitoring system
submit training
request with
ODAHU manifest
Workflow engine
ODAHU Plugin ODAHU Plugin
ODAHU componentLegend: External component Custom scripts of ML project Logical group Get
Send
Control
send req. to train ML model
ML scripts
Prepared datasets
ML training jobs
ML model archive
7

8. ODAHU SERVICE FOR ML MODEL PACKAGING
Data Scientist IDEODAHU Command Line Tool
Core
Components
Packaging ML model Service
External
Systems
User Facing
Components
Connections ManagerAudit Service
orchestrate ML training job
get credentialssend audit info
get archive
with ML model
send docker imagePackage registry Compute cluster
submit packaging
request with
ODAHU manifest
Workflow engine
ODAHU Plugin ODAHU Plugin
Registry
send log msg
& metrics
from cluster
Monitoring system
ODAHU componentLegend: External component ML project artifact Logical group Get
Control
send req. to package model for target platform
ML model archive ML model packaging job
ML model packaged for
target platform
8

9. ODAHU SERVICE FOR ML MODEL DEPLOYMENT
Data scientist IDEODAHU command line tool
Core
Components Deploying ML model service
External
Systems
ODAHU componentLegend: External component ML project artifact Logical group
User Facing
Components
Connections manager
Audit service
orchestrate model deployment job
get credentials
send audit info
submit packaging
request with
ODAHU manifest
Workflow engine
Control
Get
Send
ODAHU plugin ODAHU plugin
get docker
image and deployRegistry Compute cluster
send log msg & metrics
from cluster
Monitoring system
Service Catalog
send req. to deploy model on target platform
AI services or/and jobsAI services or/and jobs
9

10. Workflow engine with ODAHU plugin
ML model delivery pipeline
ML/AI PIPELINES EXAMPLE BASED ON ODAHU COMPONETS
ODAHU Core
Components
Deploying ML model
service
ODAHU componentLegend: External component ML project artifact Control
Send
load
data
transform
data
train & validate
ml model
validate
data
package model for target
platform and store it in registry
deploy
model
ML model inference pipeline
load
data
transform
data
request
inference
validate
data
deliver
inference
Training ML model
service
Packaging ML model
service
Feedback loop
components
log inference req. & resp.
send inference feedback message
Compute cluster
Inferencereq.,resp.,feedbackmessages
Data Storage
Historical
data
Inference
req., resp.
and
feedback
data
Inference
input data
Inference services or/and
batch jobs
send inference req. & resp.
ML
model’s
inference
consuming
system
deploy
model
send inference
Get
get
inferencesend ml training req. send ml training req. send ml packaging req.
10

11. KEY FEATURES IN ODAHU ROADMAP FOR NEXT RELEASES
• Web control panel
• Role-based access control
• Activity audit
• Advanced logging and alerting
• Support more platforms for ML model training loads
• Support more runtime platforms ML models
• Plugins for more IDEs
• Plugins for more workflow engines
• Additional services for ML/AI lifecycle
• Deployment automation for on-premise infrastructure with OpenShift and pure K8S
• Deployment from Cloud Marketplaces – Google, AWS and other clouds
• More and better documentation
11

12. ODAHU FOR INTERNAL EPAM PROJECTS
• Employee vs positions matching
• Employee attrition score
• Employee star score
• Automated language assessment
• Employee productivity model
• And more ….
12

13. COOPERATION AND CONTRIBUTION TO ODAHU
• ODAHU team helps with conducting demo for EPAM clients
• ODAHU team helps with ODAHU deployment and configuration for EPAM clients
• Contribution to ODAHU projects is welcome https://github.com/odahu
13

14. ODAHU DEMO
D L A B + O DA H U + 3 d p a r t y
s e r v i c e s
M L p r o j e c t s ex a m p l e s
a d a p t e d t o O DA H U
• Wine quality inference
(ODAHU + Mlflow + Scikit-
learn + Airflow DAG)
• Text classification (ODAHU+
Mlflow + Keras + TensorFlow)
• Image recognition (ODAHU +
Mlflow + Keras + TensorFlow)
• Data Scientist IDEs
orchestration with DLab
• Core ODAHU services
• 3d party systems
integrated with ODAHU
14

15. DEMO
15

16. Q&A
16