1. for a Kafka streaming platform
with a bit of devops
All the Ops: DataOps with
GitOps

2. Presenters
Andrew Stevenson
CTO Lenses.io
Spiros Economakis
DevOps Cloud Lead Lenses.io

3. Delivering data intensity on Kafka
Lenses.io

4. Kafka ?
What is DataOps & Lenses.io
How to automate your data landscape
GitOps demo
Agenda

5. What is Kafka?
distributed commit log

6. Distributed commit log?
*Oreilly - Kafka definitive guide

7. Data platform
Kafka

8. What do we do?

9. Deploying Kafka isn’t interesting any more?

10. Well trodden path

11. Tech Intensity
Use commodity infrastructure
Don’t reinvent
Focus on building data landscape

12. If not kafka, what are we automating?
The important bit...
Data Intensity
Tech Intensity

13. How are you automating?
(at a high level)

14. Pushing
Are sure you want to give access?
Who resolves the desired state?
Who applies the desired state?

15. K8s operator for Kafka
Better but...

16. Observe git, apply desired state
GitOps

17. GitOps for DataOps

18. GitOps Benefits
Focus build reusable apps - config driven - less ops
Reduced attack surface to sensitive environments
Immutable
Auditable
Rollback
Separation of concerns
Use any Git server - Don’t need k8s
Operator enforces multi-tenancy and governance

19. GitOps for Kafka
Infrastructure deployment is now a commodity - Terraform, Cloudformation, etc.
Tech intensity
Repeatable, rapid deployment of the data landscape
Topics
Schemas
ACLs
Alert rules
Data policies
Applications
Data intensity

20. LEARN MORE
All-in-one Lenses.io DataOps
workspace & Kafka docker
container
Lenses.io workspace with Kafka
in the cloud
lenses.io/start
launchpass.com/lensesio
https://launchpass.com/lensesio

21. GitOps for Kafka applications
Demo

22. Demo architecture

24. Monitoring the Lenses operator
:3030/metrics
Visualization and
dashboards
Prometheus Alertmanager
fire alertsdatasource

25. Monitoring the Lenses operator
Metric Description
lenses_daemon_sync_duration_seconds Duration of git-to-Lenses synchronisation
lenses_repo_fetch_duration_seconds Duration of setting up repo and provided commit
lenses_repo_fetch_failed Number of failed repo fetches
lenses_daemon_sync_succeeded Number of successfully syncs to Lenses
lenses_daemon_sync_failed Number of failed syncs to Lenses

26. Manifests - App
apiVersion: processors/v1
kind: Processor
metadata:
name: high-speed-vessels
pipeline: filter-fast-vessels
spec:
runners: 1
sql: |-
SET autocreate=true;
INSERT INTO fast_vessel_processor
SELECT MMSI, Speed, Longitude AS Long, Latitude AS Lat, `Timestamp`
FROM sea_vessel_position_reports
WHERE Speed > 10;

27. Manifests - Topic
apiVersion: kafka/v1
kind: Topic
metadata:
name: sea_vessel_position_reports
pipeline:
- filter-fast-vessels
depends:
matchTags:
schemas: sea_vessel_position_reports
spec:
replication: 1
partitions: 3
configs:
compression.type: uncompressed
retention.bytes: "26214400"
segment.bytes: "8388608"

28. Manifests - Schema
apiVersion: schemas/v1
kind: Schema
metadata:
name: sea_vessel_position_reports-key
tags:
schemas: sea_vessel_position_reports
spec:
avroSchema: |-
{"type": "record",
"name": "Key",
"namespace": "io.lenses.ais",
"fields": [{
"name": "MMSI",
"type": "long",
"doc": "User ID (MMSI)"
}]
}

29. Building data intensity
DataOps with
GitOps

30. LEARN MORE
All-in-one Lenses.io DataOps
workspace & Kafka docker
container
Lenses.io workspace with Kafka
in the cloud
lenses.io/start
launchpass.com/lensesio
https://launchpass.com/lensesio