Confluent Platform 5.5 introduces several new features to simplify event streaming development including adding Protobuf and JSON schema support throughout the platform, providing exactly-once semantics for non-Java clients, introducing administrative functions to the REST Proxy, expanding the functionality of ksqlDB with new aggregates and data types, and adding a ksqlDB flow view to Confluent Control Center for increased visibility of streaming applications. The release is also based on the latest Apache Kafka 2.5 version.

1. What’s New in Confluent Platform 5.5
May 5th 2020
Kai Waehner
Technology Evangelist
kai.waehner@confluent.io
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

2. Poll 1
What Stack are you using Today in your Projects?
● Apache Kafka (Pub/Sub, Storage, Connect, Streams)
● Confluent Community (REST, Schema Registry, KSQL, non-Java Clients, …)
● Confluent Platform (Self-managed Deployment)
● Confluent Cloud (Fully-managed Service)
● I don’t use Kafka yet
www.kai-waehner.de | @KaiWaehner

3. Ways to Deploy
Confluent Platform
The enterprise distribution of
Apache Kafka
VM
Deploy on any platform
on-prem or cloud
Self Managed Software Fully Managed Software
Confluent Cloud
Apache Kafka re-engineered
for the Cloud
Available on the leading public clouds
Also via MarketplaceAlso via Marketplace

4. Confluent Platform
Dynamic Performance & Elasticity
Auto Data Balancer | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Global Resilience
Multi-region Clusters | Replicator
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient
Operations at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud ServiceSelf-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECTOPERATORDEVELOPER BUYER
Apache Kafka

5. Rapid Pace of Innovation to Enable Enterprises
April 2020
CP 5.5 (based on AK 2.5)
Data Compatibility
● Protobuf and JSON schema
support throughout platform
Multi-language Development
● Exactly-once semantics for non-
Java clients
● Admin functions for REST Proxy
(preview)
Event Streaming Database
● ksqlDB 0.7
● ksqlDB Flow View
Security
● AD/LDAP authentication for
clients
July 2019
CP 5.3 (based on AK 2.3)
Security
● Role-Based Access Control
(preview)
● Secret Protection
DevOps automation
● Kubernetes Operator
● Ansible Playbooks
Management & Monitoring
● Control Center redesigned
user interface
● New CLI
April 2019
CP 5.2 (based on AK 2.2)
Developers
● Free single-broker
developer license
● librdkafka and clients 1.0
KSQL
● New query expressions
● GUI enhancements
Replicator
● Schema migration to
CCloud
Control Center
● Dynamic broker
configuration
● Schema Registry
management
● Multi-cluster Connect &
KSQL
● Enhanced scalability
January 2020
CP 5.4 (based on AK 2.4)
Security
● Role-Based Access Control
● Structured Audit Logs
Resilience
● Multi-Region Clusters
Data Compatibility
● Schema Validation
Management & Monitoring
● Control Center
○ RBAC management
○ Replicator monitoring
Performance & Elasticity
● Tiered Storage (preview)
Stream Processing
● ksqlDB features (preview) 5

6. What’s New in
Confluent Platform 5.5

7. CP 5.5 makes Apache Kafka and event streaming
more broadly accessible to developers
Rich Pre-built
Ecosystem
Protobuf, JSON, and plug-in
schema support for Schema
Registry and throughout the
platform
7
Multi-language
Development
Exactly-once semantics for
non-Java clients
Admin functions for REST
Proxy (preview)
Event Streaming
Database
ksqlDB 0.7
ksqlDB Flow View within
Control Center

8. JSON Schema and Protobuf

9. Allow developers
and businesses to
adopt event
streaming broadly
Organizations no longer
need to adjust event
streaming apps to use Avro
to use Schema Registry
Applications with
other formats also
need to work with
Schema Registry
Plug-ins provide a way for
Kafka users to address the
“long tail” of data formats
Protobuf and
JSON are both very
prevalent data
formats
The two data formats were
the most commonly
requested by customers by
a significant margin
Why add Protobuf, JSON Schema, and plug-in
schema support to Schema Registry?
9
Rich Pre-built Ecosystem

10. 10
Schema Registry
expands the Kafka
ecosystem with
Protobuf, JSON,
and plug-in schema
support
Schema Registry
Custom Plug-
Ins
Supported Data Formats
Rich Pre-built Ecosystem
App 1
!
Schema
Registry
Kafka
topic
!
Serializer
App 1
Serializer

11. Kafka Connect
Converters for Protobuf and JSON
ksqlDB and Kafka Streams
Enrich messages in either data format
REST Proxy
Register and retrieve schemas
Control Center
Create topic schemas directly in the GUI
Schema Validation
Broker-side enforcement of schemas
11
Protobuf and JSON
Schema are also
supported across all
components of
Confluent Platform
Rich Pre-built Ecosystem

12. Exactly-Once Semantics

13. Why add exactly-once semantics to non-Java
clients?
13
Exactly-once semantics ensures
critical messages are neither
skipped nor repeated
For example, a message that could trigger a
financial trade cannot be produced or
consumed more than once
Feature parity between the
clients make event streaming
accessible to all developers
Organizations with many developer teams
need a broad range of clients to allow Kafka
to become their central nervous system
Multi-language Development

14. Confluent enhances
multi-language
development with
exactly-once
semantics for
librdkafka-based
clients
14
Clients with EOS
0 1 2 3 4 5 6 7 8
Producer
Write idempotently
Read exactly once
Consumer
Multi-language Development

15. REST Proxy v3

16. Why introduce Admin functions for REST Proxy?
16
Removes need to switch over
from REST Proxy to Java Admin
client, CLI, or Control Center to
run Admin functions
REST Proxy can now both perform
administrative tasks and produce and
consume messages
Feature parity between REST
Proxy and the clients provides
greater flexibility and more
choices to developers
Many developers prefer REST because of its
language-agnostic nature and simplicity
Multi-language Development

17. REST Proxy
introduces Admin
functions to
improve Kafka
connectivity
(Preview State)
17
REST Proxy
REST Proxy
Applications
Schema
Registry
REST / HTTP
Describe and list brokers
Create, delete, describe, and list topics
Describe and list configs for topics
New Admin Functions
Multi-language Development

18. ksqlDB

19. Kafka
producer/
consumer
Kafka
Streams
ksqlDB
The 3 stream processing modalities with Confluent

20. ConsumerRecords<String, String> records = consumer.poll(100);
Map<String, Integer> counts = new DefaultMap<String,
Integer>();
for (ConsumerRecord<String, Integer> record : records) {
String key = record.key();
int c = counts.get(key)
c += record.value()
counts.put(key, c)
}
for (Map.Entry<String, Integer> entry : counts.entrySet()) {
int stateCount;
int attempts;
while (attempts++ < MAX_RETRIES) {
try {
stateCount = stateStore.getValue(entry.getKey())
stateStore.setValue(entry.getKey(), entry.getValue() +
stateCount)
break;
} catch (StateStoreException e) {
RetryUtils.backoff(attempts);
}
}
}
The 3 stream processing modalities differ in
flexibility and ease of use
Kafka producer/consumer Kafka Streams ksqlDB
builder
.stream("input-stream",
Consumed.with(Serdes.String(), Serdes.String()))
.groupBy((key, value) -> value)
.count()
.toStream()
.to("counts", Produced.with(Serdes.String(), Serdes.Long()));
SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES;

21. Using external processing systems leads to
complicated architectures
DB CONNECTOR
CONNECTOR
APP
APP
DB
STREAM
PROCESSING
CONNECTOR APPDB

22. We can put it back together in a simpler way
DB
APP
APP
DB
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
STATE STORES
ksqlDB

23. Connect integration and pull queries enable end-to-end
streaming in just a few SQL statements
Serve lookups against
materialized views
Create
materialized views
Perform continuous
transformations
Capture data
CREATE STREAM purchases AS
SELECT viewtime, userid,pageid,
TIMESTAMPTOSTRING(viewtime, 'yyyy-MM-dd HH:mm:ss.SSS')
FROM pageviews;
CREATE TABLE orders_by_country AS
SELECT country, COUNT(*) AS order_count, SUM(order_total) AS order_total
FROM purchases
WINDOW TUMBLING (SIZE 5 MINUTES)
LEFT JOIN user_profiles ON purchases.customer_id = user_profiles.customer_id
GROUP BY country
EMIT CHANGES;
SELECT * FROM orders_by_country WHERE country='usa';
CREATE SOURCE CONNECTOR jdbcConnector WITH (
‘connector.class’ = '...JdbcSourceConnector',
‘connection.url’ = '...',
…);

24. Why expand the functionality of ksqlDB?
24
ksqlDB is becoming the tool of
choice to build event streaming
applications that leverage
stream processing
ksqlDB continues to be adopted widely to
create these modern applications
Expanding the feature set
further simplifies the process of
building event streaming
applications
Adding more aggregates, support for more
data types, etc.
Event Streaming Database

25. Confluent Platform
5.5 launches with
ksqlDB 0.7 to
further simplify the
development of
event streaming
applications
25
COUNT_DISTINCT Aggregate
Count distinct values within a field
COUNT_DISTINCT → 3
Support for more data types
Record keys can now be of type String, INT,
BIGINT, or DOUBLE
Higher availability of pull queries1
Reroute queries to replica table partitions if the
leader partition is down
Pull
Query
1 - Pull queries are still in preview
Event Streaming Database

26. Why add the ksqlDB Flow View to Confluent
Control Center?
26
Provides developers with a
simple mental model of how
tables, streams, and queries are
interacting
Applications often have many different
queries that are more easily understood
visually than reviewing the editor
Enables developers to quickly
discover and understand a
ksqlDB application that others
have built
ksqlDB applications are often built by many
different developers and teams, so
discoverability is crucial
Event Streaming Database

27. Control Center provides greater visibility and
discovery of ksqlDB applications
27
Visualize what
queries, streams,
and tables are
being used in a
ksqlDB application
ksqlDB Flow View
Application Overview
Event Streaming Database

28. Control Center provides greater visibility and
discovery of ksqlDB applications
28
Drill down into
table or stream
nodes to quickly
visualize their
schema and
messages
ksqlDB Flow View
Table / Stream Node Doubleclick
Event Streaming Database

29. Drill down into
query nodes to
view the code and
understand how
inputs are being
manipulated
Control Center provides greater visibility and
discovery of ksqlDB applications
29
ksqlDB Flow View
Query Node Doubleclick
Event Streaming Database

30. Confluent Platform 5.5 launches with
the latest Apache Kafka 2.5 version

31. https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
Zookeeper Removal (KIP-500)

32. Release Notes
● https://docs.confluent.io/current/release-notes/index.html
● https://downloads.apache.org/kafka/2.5.0/RELEASE_NOTES.html
… for all the details.

33. Poll 2
What Features from Confluent Platform 5.5 are the Most Exciting for You?
● Exactly-once Semantics for all Client APIs
● ksqlDB (including Connect Integration + Pull Queries)
● Improved UIs in Control Center
● New REST Proxy v3
● Support for JSON Schema / Protobuf for whole Confluent Platform
www.kai-waehner.de | @KaiWaehner

34. Kai Waehner
Technology Evangelist
kai.waehner@confluent.io
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de