High level introduction to Confluent REST Proxy and Schema Registry (leveraging Apache Avro under the hood), two components of the Apache Kafka open source ecosystem. See the concepts, architecture and features.

1. Confluent REST PROXY and SCHEMA REGISTRY
The Rise of Apache Kafka as Streaming Platform
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

2. 2
Apache Kafka
A Distributed, Scalable Commit Log

3. 3
Apache Kafka
A Distributed, Scalable Commit Log

4. 4
Apache Kafka
The Rise of a Streaming Platform

5. 5
Orders Customers
Payments
Stock
Apache Kafka
Single Shared Source of Truth for (Micro)Services

6. 6
Confluent Ecosystem

7. 7
Confluent Platform - Reference Architecture
https://www.confluent.io/whitepaper/confluent-enterprise-reference-architecture/

8. REST Proxy
Produce and Consume Messages via HTTP(S)

9. 99
Apache Kafka Native Clients
Confluent Native Clients
Community Supported Clients
Kafka Clients

10. 1010
REST Proxy
Non-Java Applications
Native Kafka Java
Applications
Schema Registry
REST / HTTP(S)
Simplifies administrative
actions
Simplifies message
creation and consumption
Provides a RESTful
interface to a Kafka cluster
Talking to Non-native Kafka Apps and Outside the Firewall
Confluent REST Proxy

11. 1111
• Producers
• Accepts produce requests targeted at specific topics or partitions and routes them all
through a small pool of producers.
• Consumers
• Stateful consumption of messages. Simple Consumer or low-level read operations, for
example to retrieve messages at specific offsets
• Metadata
• Most metadata about the cluster – brokers, topics, partitions, and configs
• Data Formats
• JSON, raw bytes encoded with base64 or using JSON-encoded Avro. With Avro, schemas are
registered and validated against the Schema Registry.
REST Proxy Features

12. 1212
Produce and Consume Messages

13. 13
REST Proxy Clusters and Load Balancing
• Designed to support multiple instances running together to spread
load and can safely be run behind various load balancing mechanisms
(e.g. round robin DNS, discovery services, load balancers)
• Sticky Load Balancer needed for Consumers because consumers are
stateful. REST Proxy will store the offsets.

14. 14
Getting Started in Minutes…
https://docs.confluent.io/current/kafka-rest/docs/intro.html#quickstart

15. 15
REST Proxy Security Plugin (Enterprise)
REST Proxy
Non-Java Applications
Native Kafka Java
Applications
Schema Registry
REST / HTTPS
Simplifies administrative
security
Simplifies Security and
ACLs
Provides a Secure RESTful
interface to a Kafka cluster
Security Plugins for REST Proxy to pass through client credentials
to Kafka Broker (rather than making all clients look like one single Kafka client)

16. Schema Registry
Schema Management for Kafka Topics (including Schema Evolution)

17. 17
APIs are contracts between services

18. 18
But not all services talk to each other directly

19. 19
And naturally…

20. 20
Schema are APIs
We need specifications
We need to make changes to them
We need to detect breaking changes
We need versions
We need tools

21. 21
Enter Schema Registry

22. 22
What do Schema Registries do?
1. Store schemas – put / get
2. Link one or more schema to each event
3. Clients (Java, .NET, REST, …) that fetch & cache schemas
• Producers sending messages
• Consumers receiving messages
4. Enforcement of compatibility rules
5. Graphical browser

23. 23
Version compatibility?

24. 24
Forward compatibility
Forward compatibility means data written with
newer schema is readable with old schemas.

25. 25
Forward & Backward compatibility
Backward compatibility means data written with
older schema is readable with a newer schema.

26. 26
Confluent Schema Registry
• Define the expected fields for each Kafka topic
• Automatically handle schema changes (e.g. new fields)
• Prevent backwards incompatible changes
• Multiple schemas per topic
• Supports multi-datacenter environments
• Leverages Apache Kafka for Fail-Over and Schema
Management
• Allows CI/CD with Maven Integration

27. 27
Multiple Schemas per Topic
https://www.confluent.io/blog/put-several-event-types-kafka-topic/

28. 28
REST API

29. 29
Apache Avro
https://avro.apache.org/docs/current/

30. 30
Confluent Schema Registry
Elastic
HDFS
Example Consumers
Serializer
App 1
Serializer
App 2
!
Kafka
Topic
!
Schema Registry
Deserializer
Deserializer

31. 31
Producers contain Serializers
Define the serializers:
props.put("key.serializer", ”org.apache.kafka.serializers.StringSerializer");
props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", schemaUrl);
…
producer<String, LogLine> producer = new KafkaProducer<String, LogLine>(props);
Create a record:
ProducerRecord<String, LogLine> record =
new ProducerRecord<String, LogLine>(topic, event.getIp().toString(), event);
Send the record:
producer.send(record); Consumers deserialize the same way…

32. 32
Serializers cache schemas, register new schema … and serialize
serialize(topic, isKey, object):
subject = getSubjectName(topic, isKey)
schema = getSchema(record)
schemaIdMap = schemaCache.get(subject)
if (schemaIdMap.containsKey(schema):
id = schemaIdMap.get(schema)
else
id = registerAndGetId(subject, schema)
schemaIdMap.put(schema, id)
output = MAGIC_BYTE + id + avroWriter(schema, object)

33. 33
Schema Registry caches schemas and validates compatibility
register(schema, subject):
if (schemaIsNewToSubject):
prevSchema = getPrevSchema(subject)
level = getCompatibilityLevel(subject)
if (level == FULL):
validator =
new SchemaValidatorBuilder().mutualReadStrategy().validateLatest()
if (validator.isCompatible(schema, prevSchema))
register
else
throw
…

34. 34
Getting Started in Minutes…
https://docs.confluent.io/current/schema-registry/docs/intro.html

35. 35
Schema Registry UI in Confluent Control Center
• See the schema per topic for each key
and value
• Version history
• Compare previous and current schema
• Schema Management – create, edit,
delete (later in 2018)

36. 36
Schema Registry Security Plugin (Enterprise)

37. 37

38. 38
Confluent Cloud
• Apache Kafka™
as a Service
Freedom to use
Kafka in
any public cloud
Access the rich
Confluent ecosystem
Develop faster
using the leading
streaming
platform
Liberate yourself
from the
operations burden
Including REST Proxy
and Schema Registry

39. Questions?
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de