Apache Kafka is a popular, open-source technology for collecting, processing, and analyzing streaming data in real-time. Amazon MSK is a fully managed service that removes the complexities of managing Kafka clusters so that you can focus on building real-time applications. In this session, we provide an overview of Amazon MSK and then discuss how to get started. We then look at some best practices and top tips, and walk through how to decide whether to choose Amazon MSK, Amazon Kinesis, or a mix of both to address your data streaming use cases.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
A deep dive into Amazon MSK
Damian Wylie
Principal product manager
AWS
A D B 2 0 6
Vijay Kistampalli
Software development engineer
AWS

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
What is Amazon Managed Streaming for Kafka (Amazon MSK)?
Announcements
Getting started
Discuss tips
Compare and contrast Amazon MSK to Amazon Kinesis Data Streams
Open Q&A

3. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How Amazon
MSK works

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How pricing works
• On-demand, hourly pricing is prorated to the second
• Broker and storage pricing
• Broker pricing starts with kafka.m5.large at $0.21 per hour
• Storage pricing is $0.10 per GB-month
• Data transfer from replication within the cluster and ZooKeeper nodes are
included at no additional cost

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Where’s Apache Zookeeper?
Apache Zookeeper is under the hood
It is highly available, fully managed,
automatically provisioned, and included
with each cluster at no additional cost

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Launching now

8. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New!
New!
New!

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New security features
Encryption in transit via TLS
inCluster and clientBroker

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New security features
Mutual TLS authentication
Certificate-based authentication using AWS Certificate
Manager Private Certificate Authority (AWS PCA)
1. Create PCA with a root certificate within AWS ACM
2. Create Amazon MSK cluster with authentication enabled, selecting PCAs
3. Consumers and producers are configured with a certificate issued by the root CA and trust store
AWS Certificate Manager

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Using TLS certificates and ACLs together
DName of the client certificates = principal of ACL
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=<YourZookeeperString> --add --allow-principal "User:CN=Damian" --
operation Read --group=* --topic AWSKafkaTutorialTopic

12. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
HIPAA eligible
AWS CloudTrail for API auditing
AWS
CloudTrail
New compliance features

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New ease of use features
Custom configurations
For new clusters; support for updating existing clusters
coming soon
Cluster-wide storage scaling
Cluster tagging and tag-based IAM polices

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New ease of use features
Custom configurations (CLI only)
Console support coming in the next few weeks
auto.create.topics.enable
delete.topic.enable
group.initial.rebalance.delay.ms
group.max.session.timeout.ms
group.min.session.timeout.ms
log.cleaner.delete.retention.ms
log.cleaner.min.cleanable.ratio
log.flush.interval.messages
log.flush.interval.ms
log.retention.bytes
log.retention.hours
log.retention.minutes
log.retention.ms
log.roll.ms
log.segment.bytes
max.incremental.fetch.session.cache.slots
message.max.bytes
min.insync.replicas
num.partitions
offsets.retention.minutes
transaction.max.timeout.ms
unclean.leader.election.enable
zookeeper.connection.timeout.ms

16. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How performance meets cost

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Rule of thumb: Selecting size of Amazon MSK broker
• Work backward from expected storage requirement
• Cluster storage = hourly average ingest rate (TB) x hours of retention / storage
utilization
Start at a moderate utilization to allow for headroom to scale storage if needed
(e.g., 60% cluster storage utilization)
• # of brokers per availability zone = Cluster storage/16 TB
16 TB is the current storage max per broker

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Sizing your first cluster
Instance type Cluster throughput (MB/second)
Kafka.m5.large 4
Kafka.m5.xlarge 8
Kafka.m5.2xlarge 16
Kafka.m5.4xlarge 32
Kafka.m5.12xlarge 96
Kafka.m5.24xlarge 192

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Comparing Kinesis Data Streams to Amazon MSK
• AWS API experience
• Throughput provisioning model
• Seamless scaling
• Typically lower costs
• Deep AWS integrations
• Open-source compatibility
• Strong third-party tooling
• Cluster provisioning model
• Apache Kafka scaling isn’t
seamless to clients
• Raw performance
Kinesis Data Streams Amazon MSK

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Comparing costs
• Granular provisioning stabilizes $/GB
costs for Kinesis Data Streams
• Recommended 4 MB/sec per set of 3
kafka.m5.large
• Amazon MSK will likely cost less than
Kinesis Data Streams for high
utilization workloads with
>4 MB out for every 1 MB in

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Under the hood of Kinesis Data Streams

23. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Damian Wylie
wylied@amazon.com
Want updates?
LinkedIn: wyliedamian
Twitter: @DamianWylie
Vijay Kistampalli
kistampa@amazon.com