The document discusses Tevec Systems' approach to machine learning as a service (MLaaS). It describes establishing separate data science and software engineering teams to develop models and pipelines. The teams collaborate using an agile data science process of continuous experimentation. This involves designing models at small/medium scale, then large scale testing on production frameworks before deciding whether to deploy in production. Establishing interfaces and software architecture standards from the start helps speed deployment with consistent results. The process has improved team growth and model performance incrementally for customers.

1. Shortening the time from analysis to deployment
with ML-as-a-Service
TEVEC Systems
Luiz Augusto Canito Gallego de Andrade
Gabriel deBodt Sivieri

2. Time Series Forecasting
Brazil’s GDP
Indistrial Capacity
Sales
What will sales be like in the coming periods?

3. Time Series Forecasting
Some strategies to deal with the problem

4. Time Series Forecasting
Some strategies to deal with the problem
Embedding strategy
Feature engineering strategy

5. API Customer Story
The customer needs insights about his
data and to build value upon its database
1
The customer is thrilled with the results and
eagerly wants to deploy this new acquired
knowledge in his business processes
3
Data Science teams comes in the scene to
crunch data and deliver powerfull models
and insights about customer data
2
What are the requirements?
4
Customer SideConsulting Side

6. API Customer Story
API Service Level Cloud Standards Improved accuracy
over time
Fresh insights to
increase value
Code Standards and
release workflow
New variables from
public sources

7. API Customer Story
API Service Level Cloud Standards Improved accuracy
over time
Fresh insights to
increase value
Code Standards and
release workflow
New variables from
public sources
Some objectives/requirements
are extremely software related

8. API Customer Story
API Service Level Cloud Standards Improved accuracy
over time
Fresh insights to
increase value
Code Standards and
release workflow
New variables from
public sources
Others are Data Science related

9. Machine learning as a Service
Focus Groups strategies
Focus Group 1
Collaboration is hard
Problems are solved locally
Problem oriented
There is no long term strategy
Focus Group 2
Focus Group 4Focus Group 3

10. Machine learning as a Service
Product Oriented Strategy
Limited API problem range
Software problems become focus
”Distance from data”
”One size fits all”
Software engineering
Customer service
Data Science User Experience

11. Our view of the matter
Experimentation framework
Commonly used
frameworks and APIs
Model 1
Model 2
Model 3
Model 4
Pipelines
Document
based
database
Modelos
treinados
Production Structure REST
Continuous Data Science

12. What’s a pipeline?
Node
Node
Node
Node
Node
Target
By combining effective
software architecture and
state-of-the-art ML and DS
tools we are able to
quickly test and deploy a
fresh pipelines for
different problems

13. Experimenting (Agile Data Science)
ML engineering
Run Accuracy Report
Data Science
Subsamples datasets to
focus on an improvement
Data Science
Designing new models in
small/medium size scale
testing
Focus on Business metrics (MAPE, ROC).
Secondary use of ”math” metrics such as
RMSE or LogLoss
Accuracy is reported based in production
forecasts versus updated information
Cluster accuracy by dataset theme or key
statistical metrics
Use of TEVEC’s pipelining framework for
quick model design
Prototype using small scale testing in a
console application (JupyterHub)

14. Experimenting (Agile Data Science)
ML engineering
Run Accuracy Report
Data Science
Subsamples datasets to
focus on an improvement
Data Science
Designing new models in
small/medium size scale
testing
Data Science/ML Engineering
Large scale testing on
production framework using
production data
ML engineering
Push pipelines to production
and monitor operations
Business Decision
Analyze the accuracy report
and decide to push to
production
Experimenting structure is an actual
document in TEVEC’s ODM data structure
Experiment connects with pipelines and
applies it to a sequence of datasets
A/B Testing compares performance in
same format as Accuracy Report
Business has business-like inputs to decide
communicate expected results to customer
The new pipeline was validated
throughout the whole experiment.
It is safe to push to production.

15. Experimenting (Agile Data Science)
ML engineering
Run Accuracy Report
Data Science
Subsamples datasets to
focus on an improvement
Data Science
Designing new models in
small/medium size scale
testing
Data Science/ML Engineering
Large scale testing on
production framework using
production data
ML engineering
Push pipelines to production
and monitor operations
Business Decision
Analyze the accuracy report
and decide to push to
production
We try to repeat the
cycle every week

16. Experimenting (Agile Data Science)
Large Scale experimenting is an
inherent part of the system.

17. Conclusions
We achieved process stability once we separated our Data Science team from the Production Software Ecosystem
Through a collaboration between Data Science team and ML Engineers we were able to design a continuous
experimentation process
To care about standards and interface in experimentation stage is to save time in deployment. This also reduces
the risk of unexpected errors in production
Pipeline structure uses state-of-the-art packages and frameworks while enforcing interfaces and software
architecture, not coding standards. This saves time to focus on Data Science
We are still learning from this new ”continuous” DS process, but so far we have had excellent results in team
growing and incrementally improving our software

18. Luiz Augusto Canito Gallego de Andrade
+55 (11) 9 7163-2619
luiz.andrade@tevec.com.br
Gabriel Sivieri
+55 (11) 9 7191-3783
gabriel.sivieri@tevec.com.br