IBM MQ and Kafka, what is the difference?

1. TechCon 2020 IBM MQ and Kafka, what is the difference? David Ware IBM MQ Chief Architect

2. Messaging is essential for building fully connected, efficient and scalable solutions. More now than ever before Messaging covers a broad spectrum of requirements

3. Messaging Scalability Security Management High Availability There are many capabilities essential to every enterprise messaging solution IBM MQ and Kafka have both repeatedly been proven to tick these boxes…

7. Critical data exchange: work that needs to be done Critical applications demand assured asynchronous interactions Essential capabilities: End-to-end once and once only Deliveryü Fine grain messaging Messages typically represent commands, queries and operations The message is a way to pass control from the originator of the message to the consumer

8. Event Driven: building scalable microservices Microservices increases the need for communication. API-based interactions can build fragile and unscalable tight bonds between components Topics and Subscriptions Publishing and subscribing to events relaxes the coupling of microservices Events are messages that communicate that something has occurred Essential capabilities:

9. Event Streaming: the expanding need for messaging Event Streaming brings data together from disparate sources, enabling even more responsive and engaging experiences for a wider set of users Scalable Subscription Stream History Efficient cloud and analytics applications utilize local decoupled buffers of event data Essential capabilities:

10. The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History

14. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked

15. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7

16. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked

17. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked

18. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 … • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked

19. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER

20. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER

21. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER 0 1 23 45 67

22. © 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots

23. © 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots Kafka topic partitions…

25. © 2019 IBM Corporation Comparing queues and topics QUEUE 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER PRODUCER PRODUCER 0 1 23 45 6 7 TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER CONSUMER 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Queues Topics • Each messages goes to a single consumer from the queue • Each message goes to every subscriber to the topic

26. © 2019 IBM Corporation CONSUMERCONSUMER MQ and Kafka topics TOPIC PARTITION 0 1 2 3 4 5 6 7 CONSUMER GROUP CONSUMER GROUP PRODUCER PRODUCER CONSUMER CONSUMER TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER 1 2 3 4 5 6 70 1 2 3 4 5 6 70 IBM MQ Kafka

27. © 2019 IBM Corporation Queues and topics IBM MQ Kafka Queues Primary MQ concept Queues are administratively or dynamically created Queues can be partitioned for scale All messages are stored on queues, even publications n/a Topics Core MQ capability Topics are dynamically created as needed by applications Each subscribing application has access to a dedicated queue of to-be-consumed messages matching their subscription. Primary Kafka concept Rolling history of messages maintained by Kafka. Topics can be partitioned for scale… Consumer’s offset within a shared stream of messages is maintained by Kafka.

29. Horizontal scaling – workload balance the messages PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER IBM MQ Kafka PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER

30. Kafka Clusters • A Kafka system is typically made up of multiple active nodes • Topics can be partitioned to enable workload balancing across those nodes, improving scale and availability of the topics • Kafka workload balances messages across partitions, either randomly or based on application provided context to enable per-partition ordering • Each topic partition is active in one broker at a time • The messages held in a topic partition can be replicated to other brokers to ensure data integrity across failures and automatic recovery

31. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 Kafka partitions and consumer groups 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers

32. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers

33. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers

34. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups - p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers X

35. IBM MQ Clusters and workload balancing PRODUCE MESSAGES TO QUEUE A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER • A single running instance of MQ is a queue manager. • These can act independently or collectively to build horizontally scaled solutions. • An MQ Cluster is a dynamically provisioned set of queue managers with the ability to partition queues across them. • MQ will workload balance inbound messages from producing applications across each partition of the queue. • With MQ’s Uniform Cluster capability, MQ will workload balance multiple consuming applications across each queue manager, ensuring messages are consumed efficiently. • MQ does not provide simple message ordering in these topologies • MQ’s HA technologies provide data integrity across failures and automatic recovery

36. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances

37. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20, … 01,11, … X

38. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20 01,11 03,13,23

39. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20,40 01,11 03,13,23,43 CONSUMER CONSUMER CONSUMER CONSUMER 30,50,70 32,52 31 21

40. © 2019 IBM Corporation Scaling IBM MQ Kafka Consuming application is the bottleneck Scale out the consuming application Messaging system is the bottleneck Scale out the messaging system Increase the number of consuming applications from the existing queue/subscription. Increase the number of topic partitions to support a greater number of consuming applications in a consumer group. Increase the number of queue managers deployed and workload balance messages and consumers across them. Increase the number of Kafka broker nodes deployed and redistribute the topic partitions across them. †Good practice is to capacity plan for highest load and pre-define multiple topic partitions to support more consumers than you currently require, enabling more dynamic consumer scaling https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ † † † Unlike MQ, where new queue managers do not require copies of the existing messages, Kafka requires event histories to be copied to a new broker before it can be effective, slowing down dynamic scaling https://www.infoq.com/presentations/cloud-native-kafka-netflix/ ††

42. © 2019 IBM Corporation ConversationConversation ConversationConversation Fine grain communication? • MQ is designed for tens of thousands of lightweight queues and hundreds of thousands of dynamic topic strings per queue manager • This enables each conversation (e.g. reply flow) to be a dedicated queue or topic • Each consumer only sees its intended message • Dynamic create/deletion of queues and topic strings IBM MQ Equivalent Kafka model … • Kafka is designed for less, larger, configured topic partitions per broker • Recommendations are typically for 2-4k topic partitions per broker* • Consumers process every message in the topic from a point forward • This model is considered a Kafka anti-pattern. Recommendation is to rearchitect application to a stream processing pattern if possible** • https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions ** https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka PRODUCERPRODUCER PRODUCERPRODUCER CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … TOPIC 2 30 1 … …

44. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Any problem in the messaging system or the application may result in messages being lost Messages are not lost in the event of a failure, but any message may be duplicated Messages are not lost in the event of a failure and each message can be processed individually to prevent possible duplication An application can coordinate the processing of a message with a specific update to an external resource, such as a database

45. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications

46. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications Ideal for event streaming use case Ideal for critical data exchange use case

47. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Higher levels of message reliability typically reduce the need for applications to consider error scenarios. However, higher message reliability typically reduces overall message performance and can limit overall availability The trade off Event streaming criticaldata exchange Reliability Performance

48. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Kafka IBM MQ https://medium.com/@andrew_schofield/does-apache-kafka-do-acid-transactions-647b207f3d0e Sweet spots Default behavior. Some messages may be lost following an application or Kafka failure, but others persisted Default behavior. All messages of this type lost following an MQ failure by design Kafka well designed for this, producers can be configured to retry failed sends and consumers typically periodically update their offset Possible to design an application to behave like this through message batching and protocol optimizations, but not a key MQ scenario Possible within a closed Kafka system, but much harder with external systems. Technically possible to lock Kafka down to confirm every message for individual acknowledgment but loses benefits A natural fit for the destructive consume approach of MQ coupled with its core persistence model Supports Kafka-only transactions for exactly-once stream processing applications, but no external resource coordination A core MQ capability is as a resource manager (or coordinator) for global transactions Event streaming criticaldata exchange

50. © 2019 IBM Corporation Per-message availability IBM MQ Kafka Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes. PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9

51. © 2019 IBM Corporation Per-message availability IBM MQ Kafka PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 active/active partitioning of queues and topics, each partition is active/passive Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes.

54. The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History

55. Thank you David Ware IBM MQ Chief Architect — www.linkedin.com/in/dware1 © Copyright IBM Corporation 2020. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and ibm.com are trademarks of IBM Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available at Copyright and trademark information. 57TechCon 2020 © 2020 IBM Corporation

IBM MQ and Kafka, what is the difference?

Esté a ler uma amostra.

Confira estes a seguir

IBM MQ and Kafka, what is the difference?

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a IBM MQ and Kafka, what is the difference? (20)

Mais de David Ware (9)

Mais recentes (20)