O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Pipelines for model deployment

223 visualizações

Publicada em

Onna presentation about how we deploy static models and dynamic models on production.

Publicada em: Dados e análise
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Pipelines for model deployment

  1. 1. Pipelines for model deployment Ramon Navarro Bosch, CTO Onna
  2. 2. What is • Startup Barcelona / San Francisco • SaaS / On-Premise • Connect sources of information, gather, analyze and offer a UI to search for information • Knowledge Management and apps for eDiscovery and contract management • API centric solution
  3. 3. “41 components architecture distributed with kubernetes (k8s) and deployed with jenkins CI/CD”
  4. 4. Static models • Calculate a model offline • SVM / CNN / RF / RNN / LSTM / Summary / … • Mostly tensorflow, sklearn
  5. 5. Static models Dataset Algorithm Test Pack Deploy dev stage prod tf… Train
  6. 6. Static models • Detect interesting content type • Get dataset - apply generic feature extraction • Train model … for example: SVM (binary) - versions of the models as contained files • Embed in processing container jenkinsfile / dockerfile • Build container and push version to registry • Adding to production stack • Continuous build on each commit
  7. 7. Arquitecture - k8s Canonical REST API Guillotina Processing engine … Account namespace Shared namespace
  8. 8. Queue system New document Pipeline RabbitMQ Feature Extraction Beat Beat Write Each component redirect messages processed to next exchange Director Exit
  9. 9. • Testing accuracy on production processing new messages. Plug to queue and don’t ack messages to get reprocessed • Scale the # of pods/beats for each one based on the size of the queue • Dead letter queues for not well processed • Delay queues for incremental processing • Experimental beats • REST API to our Beats engine to extract information about a resource Static models
  10. 10. “We allow users to create their own models”
  11. 11. Guillotina • Open Source scalable resource management framework (data plaform) • Provides an extensible Transactional Traversal REST API with distributed DB support • Event triggering • Security model • All async • PyData talk 21/5
  12. 12. Dynamic models • Distributed continuous training • Serve multi model live • Model storage • Direct access to distributed data • Inference and train
  13. 13. Model+Meta Worker ParameterServer Serving Algorithm DocumentsLabels Metadata EmbeddingVocabulary
  14. 14. Simplified example • User defines labels - ML - BUSINESS • Asks to train a classification model based on logistic regression • A model is saved on the DB (SavedModel) • User defines a Rule to apply the model (tf serving) - each time I have a new Mail from Gmail
  15. 15. tf.serving pods (c++) • gRPC from Canonical to inference with mini batch / model spec • SourceAdapter (Loader) connector for Canonical model loading (versions handling of the model) • ServableAdapter with sharing vocabulary and multiple loaded models (multiple models on a serving component) • k8s scaling and monitoring
  16. 16. Dynamic distributed flow • Allocate variables and workers pods dynamic with k8s python api (there are limits) • Packs an Experiment and the Estimator (with Keras model or direct TF) • input_fn is a feed generator that gets the data from the web socket API to tfRecord (Guillotina model to Protobuff model) • Re-train with new documents loading the model last check • Write the model by the main worker on the canonical • Another pod runs the validation on the saved canonical for each checkpoint • Dev can runs the tensorboard to the canonical model endpoint
  17. 17. Canonical Ecosystem Serving TrainingTrainingTrainingTrainingTraining
  18. 18. Preguntes ? Gràcies! Ramon Navarro Bosch ramon@onna.com