2. Cover w/ Image
Agenda
● What is Spring Cloud Data Flow?
● Spring Batch
● Spring Task
● Spring Cloud Functions
● Spring Cloud Stream Applications
● Interview with a User
● Q+A
3. What is Spring Cloud Data Flow?
● Microservice based Enterprise
Application Integration.
○ Batch Jobs
○ Tasks
○ Streams
4. What is Spring Cloud Data Flow?
● Orchestration and Deployment
○ Tanzu Application Service /
Cloud Foundry
○ Kubernetes
○ Local
5. What is Spring Cloud Data Flow?
● Observability
○ Micrometer
○ Prometheus
○ WaveFront
○ InfluxDB
○ More…
12. Batch/Task Workflows
● DSL can define complex Batch
and or Task application
topologies in addition to a
single Batch or Task
application
● Each box is a Spring Batch or
Task application.
● Application Flow can split and
join, based on optional
conditional expressions
13. How did we build Spring Cloud Data Flow?
Spring Cloud
Data Flow
Spring Boot Spring Cloud
Spring Batch Spring Integration Spring AMQP Spring Kafka
Spring Security Spring Data
14. Spring Cloud Data Flow Architecture
● Data Flow UI, REST Client or Shell
● Data Flow Server
● Skipper Server
● Messaging Middleware
20. Data ingest
Data comes in via file or web
service
Once in the DB the batch process
can consume
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
25. Design
● How do you eat an elephant?
● Enable more real-time processing
● Improve the feedback loop for failures
● Use Java
● Functional programming paradigm
● How do you eat an elephant?
27. Stream processing
System is built to be event driven
Files are broken into individual
events
File and web service events are
processed by streaming
applications
All streaming apps are custom.
(Didn’t use any off the shelf
applications)
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
28. Stream processing
Failures can be handled on
individual events
Streaming apps are designed to be
idempotent
Load can be spread out much
better
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Error handler
29. Batch processing
System is built to be event driven
Files are broken into individual
events
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Spring
Batch
30. Idempotent processing
● At some point you must write to the DB (sink)
● Processes are designed to be idempotent
● The sink that persists is designed to cater for items being replayed
31. Developer experience
● Everything is a Spring Boot application (Our Devs love this)
● Changes can be tested locally
● Custom templates
● Dependency management (Solution BOM)
● Easy to understand applications that are small and
do one thing
● Unit and integration testing is easy
● Containers
Paving the road to production
33. Deployment
Data Flow deployment
● Deployed to Kubernetes (using Bitnami Helm Charts)
● Message broker is Apache Kafka
● Customized to use non-OSS database
● JDK17
Stream and Batch applications
● GitOps
○ Applications are defined in git
○ Stream definition (DSL) and deployment info is in git
○ Deployments via Dataflow REST API
34. Management and Observability
Monitoring
● Application metrics (uptime, resource utilization)
● Monitor topic lag
● Logs exported via sidecar application
Scaling
● Make sure JVM properties are set (Heap memory, CPU count, GC, etc)
● Single instance of most apps
● No dynamic scaling yet
Have you heard of Enterprise Integration Patterns? SCDF provides for the deployment and orchestration of batches, tasks and streams. These are deployed as microservice based Spring Boot Applications into Cloud Foundry / Tanzu Application Service, Kubernetes or as processes on a single machine.
These are deployed as microservice based Spring Boot Applications into Cloud Foundry / Tanzu Application Service, Kubernetes or as processes on a single machine.The platform manages the availability and scalability
Micrometer should be thought of as SLF4J for metrics.
Spring Cloud Data Flow use Spring Boot, Spring Security, Spring Data and Spring Cloud Deployer modules.
I would like to introduce you to one of the rarest sightings in Open source. Vincent Chegwidden works at Discovery using Spring Cloud Data Flow.
Q: How did you get to use Spring Cloud Data Flow?
Q: So I suppose you use or create Spring Cloud Stream applications…
Q: Do you use any Spring Batch functionality?
Q: Do any of the processors have side-effects?
Q: What does the development cycle look like for your custom stream apps?
Q: How do you deploy and manage SCDF and your solution?
Future: Boot 3 support for Stream apps, Task and Batch apps and observability improvements.
Data Flow interacting with multiple deployment environments. Environments may be connected….
Polyglot apps, Web Assembly, IoT, Edge.