Talk from Kafka Summit San Francisco 2019 (https://kafka-summit.org/sessions/event-driven-model-serving-stream-processing-vs-rpc-kafka-tensorflow/). Video recording will be available for free on the Summit website.
Event-based stream processing is a modern paradigm to continuously process incoming data feeds, e.g. for IoT sensor analytics, payment and fraud detection, or logistics. Machine Learning / Deep Learning models can be leveraged in different ways to do predictions and improve the business processes. Either analytic models are deployed natively in the application or they are hosted in a remote model server. In the latter you combine stream processing with RPC / Request-Response paradigm instead of direct doing direct inference within the application. This talk discusses the pros and cons of both approaches and shows examples of stream processing vs. RPC model serving using Kubernetes, Apache Kafka, Kafka Streams, gRPC and TensorFlow Serving. The trade-offs of using a public cloud service like AWS or GCP for model deployment are also discussed and compared to local hosting for offline predictions directly “at the edge”.
Key takeaways
• Machine Learning / Deep Learning models can be used in different ways to do predictions. Scalability and loose coupling are important success factors
• Stream processing vs. RPC / Request-Response for model serving has many trade-offs – learn about alternatives and best practices for your different scenarios
• Understand the alternatives and trade-offs of model deployment in modern infrastructures like Kubernetes or Cloud Services like AWS or GCP
• See live demos with Java, gRPC, Apache Kafka, KSQL and TensorFlow Serving to understand the trade-offs
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka Streams, TensorFlow, gRPC
1. 1Apache Kafka and Machine Learning – Kai Waehner
Event-Driven Stream Processing and
Model Deployment with Kafka and TensorFlow
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
2. 2Apache Kafka and Machine Learning – Kai Waehner
Key Takeaways
• Model deployment can be completely separated from model training (process and technology)
• Two alternatives for model deployment: Model server vs. embedded into application
• Model embedding into Kafka applications is simple and creates a robust and performant environment
3. 3Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Machine Learning
• TensorFlow Ecosystem
• Stream Processing with Model Server and RPC
• Stream Processing with Embedded Models
• Deployment Options and Trade-Offs
• A Cutting Edge Cloud-Native ML Streaming Infrastructure
4. 4Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Machine Learning
• TensorFlow Ecosystem
• Stream Processing with Model Server and RPC
• Stream Processing with Embedded Models
• Deployment Options and Trade-Offs
• A Cutting Edge Cloud-Native ML Streaming Infrastructure
5. 6Apache Kafka and Machine Learning – Kai Waehner
Machine Learning (ML)
...allows computers to find hidden insights without being explicitly
programmed where to look.
Machine Learning
• Decision Trees
• Naïve Bayes
• Clustering
• Neural Networks
• Etc.
Deep Learning
• CNN
• RNN
• Autoencoder
• Etc.
6. 7Apache Kafka and Machine Learning – Kai Waehner
The First Analytic Models
How to deploy the models
in production?
…real-time processing?
…at scale?
…24/7 zero downtime?
7. 8Apache Kafka and Machine Learning – Kai Waehner
Hidden Technical Debt in Machine Learning Systems
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
8. 10Apache Kafka and Machine Learning – Kai Waehner
Scalable, Technology-Agnostic Machine Learning Infrastructures
https://www.infoq.com/presentations/netflix-ml-meson
https://eng.uber.com/michelangelo
https://www.infoq.com/presentations/paypal-data-service-fraud
What is this
thing used everywhere?
9. 11Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Machine Learning
• TensorFlow Ecosystem
• Stream Processing with Model Server and RPC
• Stream Processing with Embedded Models
• Deployment Options and Trade-Offs
• A Cutting Edge Cloud-Native ML Streaming Infrastructure
10. 12Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka - A Distributed Commit Log
Producers
Kafka
Cluster
Consumers
11. 13Apache Kafka and Machine Learning – Kai Waehner
Kafka Streams
Your
app
sinksource
KafkaConnect
KafkaConnect
Kafka Cluster
Apache Kafka includes Kafka Connect and Kafka Streams
12. 14Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka at Scale at Tech Giants
> 4.5 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data
13. 15Apache Kafka and Machine Learning – Kai Waehner
Confluent - Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log
aggregation, Splunk
replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)
14. 16Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka’s Open Source Ecosystem as Infrastructure for ML
15. 17Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams /
KSQL
Kafka
Connect
Rest Proxy
Schema Registry
Go/.NET /Python
Kafka Producer
KSQL
Kafka
Streams
Focus of this talk
16. 19Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Machine Learning
• TensorFlow Ecosystem
• Stream Processing with Model Server and RPC
• Stream Processing with Embedded Models
• Deployment Options and Trade-Offs
• A Cutting Edge Cloud-Native ML Streaming Infrastructure
17. 20Apache Kafka and Machine Learning – Kai Waehner
TensorFlow
TensorFlow is an open source software library for high
performance numerical computation. Its flexible architecture
allows easy deployment of computation across a variety of
platforms (CPUs, GPUs, TPUs), and from desktops to clusters of
servers to mobile and edge devices. Originally developed by
researchers and engineers from the Google Brain team within
Google’s AI organization, it comes with strong support for
machine learning and deep learning and the flexible
numerical computation core is used across many other scientific
domains.
https://www.tensorflow.org/
18. 21Apache Kafka and Machine Learning – Kai Waehner
TensorFlow Ecosystem
+ large community
+ integration with most 3rd party ML tools
+ support by all major cloud providers
Focus of
this talk
19. 23Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Machine Learning
• TensorFlow Ecosystem
• Stream Processing with Model Server and RPC
• Stream Processing with Embedded Models
• Deployment Options and Trade-Offs
• A Cutting Edge Cloud-Native ML Streaming Infrastructure
20. 24Apache Kafka and Machine Learning – Kai Waehner
Model Deployment with a Model Server
Open source vs. commercial
Self-managed vs hosted
Features (Versioning, A/B Testing, …)
Communication (RPC via HTTP or gRPC)
Serving
21. 25Apache Kafka and Machine Learning – Kai Waehner
TensorFlow Serving
• Serve multiple models or multiple versions
simultaneously
• gRPC and HTTP endpoints
• Model versioning without changing any client code
• Canarying and A/B testing
• Optimized for minimal latency to inference time
• Scheduler grouping individual inference requests into
batches for joint execution
• Supports many servables
• Tensorflow models
• Embeddings,
• Vocabularies,
• Feature transformations,
• Non-Tensorflow-based models
https://github.com/tensorflow/serving
22. 26Apache Kafka and Machine Learning – Kai Waehner
Stream Processing with External Model and RPC
Streams
Input Event
Prediction
Request
Response
Model Serving
TensorFlow Serving
gRPC / HTTP
Application
23. 27Apache Kafka and Machine Learning – Kai Waehner
TensorFlow + Kafka Streams
Filter
Map
2) Configure Kafka Streams Application
3) RPC to TensorFlow Serving (and catch Exceptions)
4) Start Kafka Streams App
1) Import Kafka and TensorFlow Serving API
Model
Server
Request
Response
24. 28Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Machine Learning
• TensorFlow Ecosystem
• Stream Processing with Model Server and RPC
• Stream Processing with Embedded Models
• Deployment Options and Trade-Offs
• A Cutting Edge Cloud-Native ML Streaming Infrastructure
25. 29Apache Kafka and Machine Learning – Kai Waehner
Stream processing with embedded model
Application
Input Event
Prediction
26. 30Apache Kafka and Machine Learning – Kai Waehner
Stream processing with embedded model
Streams
Input Event
Prediction
Stream Processing
Model
doPrediction()
return value
27. 31Apache Kafka and Machine Learning – Kai Waehner
Stream processing with embedded model
Input Event
Prediction
Kafka Client
REST
Client
Model
doPrediction()
return value
28. 32Apache Kafka and Machine Learning – Kai Waehner
TensorFlow + Kafka Streams
Filter
Map
2) Load TensorFlow Model
3) Configure Kafka Streams Application
4) Apply TensorFlow Model to Streaming Data
5) Start Kafka Streams App
1) Import Kafka and TensorFlow API
29. 33Apache Kafka and Machine Learning – Kai Waehner
What Models should be directly embedded into an App?
Examples
• Python Model à Slow
• H2O Java Class (e.g. Decision Tree) à Fast
• TensorFlow protobuf (small Neural Network, few Mb or less) à Quick load
• TensorFlow protobuf (large Neural Network, 100Mb and more) à Much memory
• Standards-based (e.g. XML / JSON, based on PMML, ONNX )
• Includes other steps like pre-processsing
• Often organizational challenges and technical limitations / constraints
• Model Performance à Faster is better
• Model Binary Format à Best case: Java Bytecode
• Model Size à Less MB and Memory is better
• Model Server Features à Out-of-the-box vs. build-yourself vs. not needed
30. 34Apache Kafka and Machine Learning – Kai Waehner
Rebuilding the Features of a Model Server is not that hard…
Application
Input Event
Prediction
- Start new version of application (e.g. K8s Pod)
- Send and consume model or weights via Kafka topic
- Load new version via API (e.g. Java TensorFlow API)
- Service Mesh (Envoy / Linkerd / Istio etc.) for A/B
Testing, Green / Blue Deployments / Dark Launch, …
Update Model?
Versioning?
A/B Testing?
Canary?
31. 35Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Machine Learning
• TensorFlow Ecosystem
• Stream Processing with Model Server and RPC
• Stream Processing with Embedded Models
• Deployment Options and Trade-Offs
• A Cutting Edge Cloud-Native ML Streaming Infrastructure
32. 36Apache Kafka and Machine Learning – Kai Waehner
Stream Processing with Model Server vs. Embedded Model
Why use a model server and RPC?
• Simple integration with existing technologies
and organizational processes
• Easier to understand if you come from non-
streaming world
• Later migration to real streaming is also
possible
• Model management built-in for different
models, versioning and A/B testing
• Monitoring built-in
Why embed model into streaming app?
• Better latency as remote call instead of local
inference
• Offline inference (devices, edge processing, etc.)
• No coupling of the availability, scalability, and
latency/throughput of your Kafka Streams
application with the SLAs of the RPC interface
• No side-effects (e.g., in case of failure), all
covered by Kafka processing (e.g., exactly once)
Application
Input Event
Prediction
33. 37Apache Kafka and Machine Learning – Kai Waehner
Separation of Model Training and Model Inference
Local
Predictions
Model Training in Cloud
Model Deployment On Premise
Analytic
Model
34. 38Apache Kafka and Machine Learning – Kai Waehner
Where to Deploy the Model?
• Public Cloud
• VM / Container
• Serverless
• On Premise
• Traditional Application
• Cloud-Native
• Edge
• Data Center
• Machine
• Device
VM
35. 39Apache Kafka and Machine Learning – Kai Waehner
Traditional Application Deployment
VM
Application
Input Event
Prediction
36. 40Apache Kafka and Machine Learning – Kai Waehner
Traditional Application Deployment – A Kafka Streams App
1) Java Code (Kafka Streams App + Unit Test)
2) Maven Build
3) Application Deployment of JAR File
https://github.com/kaiwaehner/kafka-streams-machine-learning-examples
Streams
37. 41Apache Kafka and Machine Learning – Kai Waehner
Cloud-Native Deployment leveraging Kubernetes
38. 42Apache Kafka and Machine Learning – Kai Waehner
Kubernetes
Cluster K8 NodeK8 NodeK8 Node
Replicator
Pod
C3 Pod SR Pod
K8 NodeConfluent
Operator
Kafka
Pod
ZK Pod
Persistent Volumes
(AWS EBS, GCE Persistent Disk, Local Persistent Volume, etc.)
External
Access
Load
Balancers
Configurations
ConfigMaps
Kafka Application Pod
Cloud-Native Deployment – Embedded Model
Kafka Streams App
(TensorFlow Model Embedded)
39. 43Apache Kafka and Machine Learning – Kai Waehner
Kubernetes
Cluster K8 NodeK8 NodeK8 Node
Replicator
Pod
C3 Pod SR Pod
K8 NodeConfluent
Operator
Kafka
Pod
ZK Pod
Persistent Volumes
(AWS EBS, GCE Persistent Disk, Local Persistent Volume, etc.)
External
Access
Load
Balancers
Configurations
ConfigMaps
Kafka Application Pod
Cloud-Native Deployment – Model Inference via Sidecar Pattern and RPC
Kafka
Streams App
TensorFlow
Serving
(Sidecar)
40. 44Apache Kafka and Machine Learning – Kai Waehner
Model Deployment at the Edge
Questions to discuss
• Edge Data Center vs. Edge Device /
Machine
• Kafka Cluster vs. 1 Kafka Broker vs. Kafka
Client at the Edge
• Powerful Client (KSQL, Java) vs.
Lightweight Client (C, JavaScript)
• Embedded vs. RPC Model Inference
• Local vs. Remote Training
• Legal and Compliance Implications
What is
“the Edge”?
41. 45Apache Kafka and Machine Learning – Kai Waehner
Example: Edge Computing in Telco Business
• StarlingX (https://www.starlingx.io/) - open source “cloud infrastructure software stack for the
edge used by the most demanding applications in industrial IOT, telecom, video delivery and
other ultra-low latency use cases”
• Wind River Titanium Cloud is „the industry’s highest-performing cloud infrastructure for the
network edge“ based on StarlingX
• Definition of “Edge Computing”: Ultra low latency with < 100ms end-to-end communication
Let’s always define
what you mean with
“Edge Computing”
42. 46Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Machine Learning
• TensorFlow Ecosystem
• Stream Processing with Model Server and RPC
• Stream Processing with Embedded Models
• Deployment Options and Trade-Offs
• A Cutting Edge Cloud-Native ML Streaming Infrastructure
43. Data Lake
Batch
Analytics
Event
Streaming
Platform
Batch
Integration
Real Time
Pre-
processing
Machine Sensors
Streaming Platform
Other Components
Real Time
Processing
(6b) All Data
(7) Potential Defect
(3)
Read Data
Optimization
/ Analytics
(5)
Deploy
Optimization
Model
(8b) Alert Person (e.g. Mobile App)
(2)
Preprocess
Data (6a) Consume machine data
Model
Standard-
based
Integration
(8a)
Stop Machine
(1)
Ingest Data
Real Time Edge
Computing
Model Lite
Real Time App
Model Server
RPC
PLC Proprietary-
based
Integration
Standard
Interface
Proprietary
Interface
(9) Manual user-based analytics
and reporting to find insights
and improve real time process
(4)
Train Model
44. 48Apache Kafka and Machine Learning – Kai Waehner
Time
Model BModel A
Producer
Distributed Commit Log
Direct streaming ingestion
for model training with
TensorFlow I/O + Kafka Plugin
(https://github.com/tensorflow/io).
(no additional data storage
like S3 or HDFS required!)
Streaming Ingestion and Model Training with TensorFlow IO
45. 49Apache Kafka and Machine Learning – Kai Waehner
Model Deployment with Apache Kafka, KSQL and TensorFlow
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id, detectAnomaly(sensor_values)
FROM machine_engine;“
User Defined Function (UDF)
46. Spark
Notebooks
(Jupyter)
Kafka
Cluster
Kafka
Connect
KSQL
Machine Sensors
Kafka Ecosystem
Other Components Real Time
Kafka Streams
Application
(Java / Scala)
(6b) All Data
(7) Potential Defect
(3)
Read Data
TensorFlow I/O
TensorFlow
(5)
Deploy Model
(2)
Preprocess
Data (6a) Consume machine data
TensorFlow
MQTT
File
HTTP
(8a)
Stop Machine
(1)
Ingest Data
Real Time Edge
Computing
(C / librdkafka)
TensorFlow Lite
Real Time Kafka
App
TensorFlow
Serving
HTTP /
gRPC
(4)
Train Model
PLC
Beckhoff
S7
Modbus
OPC-UA
PLC4X
Connector
Kafka Connect
Standard
Interface
Proprietary
Interface
(8b) Alert Person (e.g. Mobile App)
(9) Manual user-based analytics
and reporting to find insights
and improve real time process
47. 54Apache Kafka and Machine Learning – Kai Waehner
Machine Learning + Apache Kafka à Examples @ Github
https://github.com/kaiwaehner
48. 55Apache Kafka and Machine Learning – Kai Waehner
Key Takeaways
• Model deployment can be completely separated from model training (process and technology)
• Two alternatives for model deployment: Model server vs. embedded into application
• Model embedding into Kafka applications is simple and creates a robust and performant environment
49. 56Apache Kafka and Machine Learning – Kai Waehner
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!