apidays LIVE India 2021 - Connecting 1.3 billion digital innovators May 20, 2021 REST the Events - REST APIs for Event-Driven Architecture Mark Teehan, Principal Solution Engineer at Confluent APAC

1. REST the Events - REST APIs for Event-
Driven Architecture
Mark Teehan - Principal Engineer - Confluent Singapore
API Days – India
teehan@confluent.io
linkedin

2. REST the Events - REST APIs for Event-
Driven Architecture
All things apache kafka in asia pacific
Teehan@confluent.io

3. What is Kafka
A modern
distributed
platform for
data streams
Confluent helps to build
• Event-driven apps
• Data in motion
• Mainframe offload (core banking)
• Shipping logs, metrics, traces

4. Java
Framework
etc
App() App()
Event
streaming
etc
Kafka
Connect
HTTP
Sink
REST
Proxy
AppHttp()
request /
response
AppHttp()
CDC
HTTP

5. Java
Framework
etc
App() App()
Event
streaming
etc
Kafka
Connect
HTTP
Sink
REST
Proxy
AppHttp()
AppHttp()
CDC
HTTP
This session will focus
on options
REST for Kafka clients
request /
response
1
2
3
4

6. Request-Response - http
Client sends request and waits for response
something that should happen
Image: The most useless Machine (imgur)
Low Latency
Point to point
Presumption of a response
Pre-defined API

7. Event Streaming - kafka
Messages received in a queue:
drop and process later
something that has happened
Image: reddit - oddly satisfying
Continuous Processing
Event-driven
No presumption of a response
General Purpose Events

8. Request-Response - http
Challenges
● Difficult to enforce standards across
services
● Does not scale if servers are synchronous
● Risky inter-service dependencies
● Services required to maintain state
● Complex management and version
compatibility—slows development
● Requires load balancing

9. Event Streaming - kafka
Challenges
● Complex Infrastructure
● Event thinking is hard
● Adopting the Kafka Way

10. So when do we want to
Use HTTP with Kafka?

11. So when do we want to
Use HTTP with Kafka?
● Management Plane
● Data Plane
● Other Reasons

12. ● Management Plane
● Data Plane
● Other Reasons
Cluster configuration: Management of Kafka topics, consumer groups, ACLs, etc.
CI/CD and DevOps integration:
Build delivery pipelines and to automate administration
HTTP APIs - Yes Python/Others - no

13. ● Management Plane
● Data Plane
● Other Reasons
Legacy apps: use of offbeat middleware
Usually good ETL options, but poor streaming options
HTTP/REST is sometimes the only protocol in common
Mobile apps: Natural fit for request response
Use of websockets and Server Sent Events
API Gateway: Most API management tools do not have native event streaming support
Common for integration with partner systems and service monetization
HTTP/REST is sometimes the only protocol in common
Other Cobol,Erlang, Kotlin, Node.js, PHP, Ruby, Rust; 0% ~ some% support
Languages:

14. ● Management Plane
● Data Plane
● Other Reasons
Familiarity: Quicker and easier than learning a new API
Technology
Lock In Avoid all Client dependencies
Security: Securing HTTP ports is easier (F5 proxies owned by Infosec)
Securing TCP is harder
Domain Use both:
Driven HTTP/REST for synchronous client/server
Design: Kafka protocol for decoupling
A service mesh to combine both is a common architecture.

15. So that was “When to use HTTP with Kafka”
Lets talk about three of the Kafka REST APIs

16. Three Kafka REST API :
Confluent REST Proxy
Confluent Broker REST
Confluent Cloud REST API

17. Confluent REST Proxy

18. a RESTful interface to an Apache Kafka® cluster
making it easy to
● produce and consume messages
● view the metadata of the cluster
● perform administrative actions
Confluent REST Proxy
Confluent Broker REST
Confluent Cloud REST API

19. Management Plane (metadata)
● Brokers
● Topics
● Consumer Groups
● ACLS
Data Plane (data)
● Produce
● Consume
Low throughput
Limited API calls
Continuous consumption
Clustering for throughput
Confluent REST Proxy
Confluent Broker REST
Confluent Cloud REST API

20. Confluent Broker REST

21. Brokers: Describe, List, Configure
Topics: create, delete, describe, list, configure
Consumer Groups: delete, describe, list
ACLs: create, delete, describe, list
Partition Reassignments: list
Runs on the broker - not a separate service
Confluent REST Proxy
Confluent Broker REST
Confluent Cloud REST API

22. Confluent Cloud REST API

23. Launched (early-access) on 17-Feb-2021
connectors | users | service accounts | environments
(coming soon: topics | acls | consumer lag)
Confluent REST Proxy
Confluent Broker REST
Confluent Cloud REST API

24. [Also] coming soon:
Terraform integration for
Confluent Cloud REST API

25. What about REST Gateways?
Mulesoft
Apigee
Kong
Anypoint
Tibco M
...

28. Mark Teehan - Principal Engineer - Confluent Singapore
teehan@confluent.io / LinkedIn

29. REST the Events - REST APIs for Event-
Driven Architecture
Mark Teehan - Principal Engineer - Confluent Singapore
teehan@confluent.io / LinkedIn