Talk given at the LA Kafka Meetup at Ticketmaster HQ 02/20/2019

1. who am i
user ID Dani
Alias DTrapezoid
Bg Apache Proj
Dist Sys
f s Real time
event driven Sys
slaying legacy
How did we
get here
QqEg
3L ksqeipgggaqqm.o.mg
aknafakuatww.sn
what are we
doing
Y
KSQL Architecture
KSQL Use cases
3 KS Q performance considerations
4 When to use K SQL when to not
g introduce KST Reams

2. Ok let's learn more
about
you

3. What letter best describes your ego
on
T
select hope
for
your
cluster
select profit
I
l select
y
it
depends
select make
a wish

4. it
y.souyewieqsm
bzdr.my
give
select
eqsocluster
select x
smeary
it
qnaEg
depends
select
y
make
a
wish

5. Why does it matter to
think about a
Query
before you
eE
execute it p
P 77
27
n
p
Lots of
led sons

6. we do hit always have control over stuff
E
ear
to
I can r
jeng'd
s
g
FEI
a

7. WH
e
Iigigggqiioniaz
ebay.ki.g.de
RDBMS
ao.ge
ei
sa
initial use cases thinking Benefits
11
DB integration Broering
l Back pressure
message
bus
Decouples systems
Pub Sub Distributed
Highly Available
Lowly latent
0 Data governance
Event Driven
box wait there's
moire as a
streaming Platform
K2fk2 Streams API

8. Event Driven systems
Transform once use
many
haepwp
z
I customer
µIgsffIe etc
ftp.eamprocessing
23 2
1 a Ii
ieTfF
ii
ieIfF
KSQL KSTREAMS
ice creams

9. 0 Stream ProcessingQ.it
Ea
hEEyYyEFE'i
w Kafka
ftp.PEFFEEEF
abstracts ihIfEf
9 to
KSQL KSTREAMS
ffffgt.EE
CkafkaastEeamg
Let's talk about what KSQL is

10. what happens when you KSQL
CREATE STREAM AS SELECT
CREATE TABLE AS SELECT
14
Output
To P l C
Name Default same
2Tstream
table Or
stoma with KAFKA TOPIC prop WITH clause
partitions Default 4 or customise
with PARTITIONS prop WITH clause
Replication Factor Default RF topic L
customial with REPLICAS property WITH clause
Aggregations leverage embedded storage
engine to locally manage state
A compacted changelog topic persistsaggregation
state
compacted changelogtopics have the same
ofpartitions as the input stream Default L
replies

11. IS KSQL a better version of SQL
tmrw
0 KSQLO
O A
O 11
I
IT Aggregate
µ
IT
11111T t Ss
HT
11111T
Join
IT
Filter
HDMI
0 8 connect 0 7
Nfb 0 sink E
oTsource 1 0 connect
connector T O Eld s ti
a EEEtor lot FEt
jq.mggEonE0
4 f
Eia ago 53
o L.eeEiEfEEiEE.EE
tEEBtaaE
Golf connect
cassandr
Ksar is a
continuous Queryengine
t's a DBA's worst nightmare it's a
Query that
never

12. why would someone want to use
KSQL then
Mumia
to write
EE
Stream processing programs
simply
my Run SQL Queries against
date
2g nostoc of programming language
my works contrhously in Real time
Queries won't Quit until the messages
do
my Fault tolerant
my scales horizontally Vertically
f Distributed

13. what does the architecture of Ksa
look like
EE
ii
I I I
I
Ksai clients
if
EEE
SQL Engine
www.gesksQLstdtemeuts
Quenes
rest API
itow client aeu.esasffctohememra ee.me
384 tzfnkqgtopykosurtkso.ie Apps w confluentcontrol
Ksar server fEosgedneofsikEEEEIYEE.EEaEEa
ADD KSQL servers who Restarting APPS

14. https://www.confluent.io/blog/ksql-in-action-real-time-streaming-etl-from-oracle-
transactional-data
https://www.confluent.io/blog/ksql-in-action-enriching-csv-events-with-data-from-rdbms-into-AWS/
https://docs.confluent.io/current/ksql/docs/developer-guide/transform-a-stream-with-ksql.html
https://www.confluent.io/stream-processing-cookbook/ksql-recipes/detecting-abnormal-transactions
what are some cool KSQL use cases
if
t
my streaming E 14 RT monitoring
rders Kafka
CDC 1
4 4 69905 connect elastic
login Debezium ksypez.ms Elasticus KSQL
T Data Enrichment
card Eeo
Kwangyez qq.io
gEioE9oqzIksapfoko98cnr're'tflog 2
Egg53
Kafkaconnect
Debezium KSQL
my Data Transformation
change data format of message Wes
timestamp field
of partitions timestamp format
of replicas Kafkatopic name
F Anomaly Detection
create a of time to write to a ksactableftopic
collect anomalous behavior
WINDOW TUMBLING

15. {
"order_id": 1,
"customer_name": "Maryanna Andryszczak",
"date_of_birth": "1922-06-06T02:21:59Z",
"product": "Nut - Walnut, Pieces",
"order_total_usd": "1.65",
"town": "Portland",
"country": "United States"
}
ksql> CREATE STREAM purchases
(order_id INT, customer_name VARCHAR, date_of_birth
VARCHAR,
product VARCHAR, order_total_usd VARCHAR, town
VARCHAR, country VARCHAR)
WITH (KAFKA_TOPIC='purchases', VALUE_FORMAT='JSON');
Message
----------------
Stream created
----------------
SELECT * FROM PURCHASES LIMIT 5;
SELECT ORDER_ID, PRODUCT, TOWN, COUNTRY FROM PURCHASES WHERE
COUNTRY='Germany';

16. CREATE STREAM PUCHASES_GERMANY AS SELECT * FROM PURCHASES
WHERE COUNTRY='Germany';
ksql> LIST TOPICS;
Kafka Topic | Registered | Partitions | Partition
Replicas | Consumers | ConsumerGroups
-------------------------------------------------------------
-----------------------------------
_confluent-metrics | false | 12 | 1
| 0 | 0
PUCHASES_GERMANY | true | 4 | 1
| 0 | 0
purchases | true | 1 | 1
| 1 | 1
-------------------------------------------------------------
-----------------------------------
ksql>
https://www.confluent.io/stream-processing-cookbook/ksql-
recipes/data-filtering
Let's just get the German orders on
one
topic1 table
cool Now we have all the purchases
from Germany in a separate topic
Try it
if

17. https://docs.confluent.io/current/ksql/docs/capacity-
planning.html
HttRDWARET Performance
Considerations
MemoryCPU
Resource used by
0h heap
message
SQL to serialize 2 Processing
des endre messages off heap Aggregations
to KStreams ktables
then executequeries
Joins
4 cores min
32GB Min
Network
Disk
Aggregations
I join Always the
persist temp state today Biggest
bottle
100 GB SSD s min heck for good
throughput
I GBit Nlc
people
into Mr

18. Sizing
considerations
It may need to add more brokers
KSQL Queries consume I produce
from topics
Repartitioning
stateful Queries mean changelog
topics
Throughput
Decreases relative to every count due
to message sculcomplexity
Query Types
project joins AggregationsFilter
2X CPU
sumSELECT
COUNT
FROM ETC
WHERE

19. What about headless
TIP D Test in
deployment
indeYET.info
Yekssg
ii
JVM
I
af
s
E0ag
TDEEjagTOE.gg
path to confluent bin1kSqlnodequery file pathltolmyquery Sql
create Drop streams
start Stop Queries
Start Server nodes
Already using k8s
VC your workflow

20. When to use KSQL when to not
are you a Java team
How will
you deploy
what hardware do you have
access to
8
Do
you want to be overlord
maintainer

21. Kafka Streams
Processer API low level
Imperative customizable
streams API built in abstractions
Functional
KSTREAM
KTABLE
GLOBAL K TABLE
Stateless Stateful
transformations

22. RADHA
g
t gqIt
affstrapezoidFooloewarnmmore
D
g2 bout Kafka
my dog my losing
battle against
the girl
scouts of America

23. https://mockaroo.com/
EYE 99
earn more
j
F YEE FEELEY
more
Yekikes
l
get
Ksac
what if l want to use differentdats
https://docs.confluent.io/
current/ksql/docs/index.html
https://www.confluent.io/
stream-processing-cookbook/
https://docs.confluent.io/
current/streams/concepts.html