IBM Message Hub is a new Bluemix service based on Apache Kafka for messaging in the cloud. It's ideal for linking together microservices to build a scalable, flexible application in the cloud. It's great for feeding data at speed into other services such as analytics. You can also use it to bridge securely from your enterprise MQ systems into the cloud.
2. 2
Please note
IBM’s statements regarding its plans, directions, and intent
are subject to change or withdrawal without notice at IBM’s
sole discretion.
Information regarding potential future products is intended to
outline our general product direction and it should not be relied
on in making a purchasing decision.
The information mentioned regarding potential future products
is not a commitment, promise, or legal obligation to deliver
any material, code or functionality. Information about potential
future products may not be incorporated into any contract.
The development, release, and timing of any future features
or functionality described for our products remains at our sole
discretion.
Performance is based on measurements and projections
using standard IBM benchmarks in a controlled environment.
The actual throughput or performance that any user will
experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in
the user’s job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no
assurance can be given that an individual user will achieve
results similar to those stated here.
3.
4. “Kafka adoption was accelerating…with production deployments of Kafka claiming
six of the top 10 travel companies, seven of the top 10 global banks, eight of
the top 10 insurance companies, and nine of the top 10 US telecom companies."
Kafka is: Everywhere
Source: http://www.techrepublic.com/article/an-inside-look-at-why-apache-kafka-adoption-is-exploding/
13. Building: Message Hub + Public Cloud
• Message Hub + OpenWhisk
• Monitor events received in Message Hub to trigger actions in OpenWhisk
• Blog post
• Message Hub + Watson IoT
• Push events from Watson IoT to Message Hub for use in real-time applications
• Documentation
• Message Hub + Object Storage
• Push events from Message Hub automatically into Object Storage for cost effective
long term storage
• Documentation
14. • Message Hub + MQ
• Consume events from MQ using our new bridge to allow other applications and services
to consume directly from Message Hub
• Documentation
• Message Hub + Integration Bus
• IIB supports native Kafka consumer and producer nodes, allowing you to produce to and
consume from a Message Hub instance
• Blog post
Building: Message Hub + Hybrid Cloud
17. Uses an MQI client connection
• Supports connection to multi-instance
queue managers
Offers at-least-once reliability
• There may be duplication in some
cases, but not loss
• Kafka users are used to at-least-once
reliability
Partitioning into Kafka topics
• By CorrelId, GroupId or round-robin
17
MQ Bridge details
Integrated into Message Hub
18. Message Hub + Object Storage
BRIDGE
• Take data from a Kafka topic and copies into an Object Storage container
• Output format intended to be easy for processing in Apache Spark
• Use cases:
• Cost-effective, long-term storage of data
• Feeding batch analytics systems
19. • Decide how to organize the
data in Object Storage
• according to Kafka offset
• according to date
• Decide how often to upload
new objects
• time-based policy
• size-based policy
19
Object Storage Bridge details
Data organized according to date
22. How do you deploy this in IBM Cloud?
22
IBM Cloud
Internet of Things
Platform
Message Hub
Data Science
ExperienceObject Storage Apache Spark
BRIDGE
23. Demo: the COSMOS Madrid Traffic Use Case
• Objective: Respond more quickly and efficiently to real-time traffic problems.
Improve customer satisfaction and reduce costs.
• Approach: Monitor data from up to 3000 sensors. React by rerouting buses,
modifying traffic lights, etc., based upon knowledge derived from historical data
26. 26
Why use Message Hub?
Run your own Kafka
• Deploy your own Kafka
cluster
• Deploy Zookeeper cluster to
support it
• Put systems in place to
operate the service 24x7
• Keep up-to-date with the
open source community
• Handle version-to-version
migration and updates
Use Message Hub
• One-click deploy of a fully
managed Kafka environment
• Additional custom integration
points with the IBM Cloud
Platform
27. 27
Public vs dedicated Message Hub
Public Dedicated
Cluster tenancy model Multi-tenant Single-tenant
Partition limit 100 per tenant 100 per environment
Log data per partition 1GB 10GB
Throughput quotas Per-tenant None
Bridges Object Storage (out)
MQ (in)
Could be deployed if
required
Billing Usage-based
• Partitions/month
• Messages sent/rcvd
Subscription
28. • Interesting links:
• kafka.apache.org
• IBM Message Hub
• Check out Paula’s blog on the Object Storage Bridge:
• https://www.ibm.com/blogs/bluemix/2017/03/end-to-end-iot-data-pipelines-
introducing-the-message-hub-object-storage-bridge/
• Try out Message Hub in our lab:
• Streaming Analytics in IBM Bluemix Using IBM Message Hub and Apache
Spark
• Mandalay Bay South, Level 3 - South Seas H
• Wed, 22-Mar 10:15 AM - 12:00 PM
Message Hub: Next steps
30. 30
Notices and disclaimers
continued
Information concerning non-IBM products was obtained from the
suppliers of those products, their published announcements or other
publicly available sources. IBM has not tested those products in
connection with this publication and cannot confirm the accuracy of
performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be
addressed to the suppliers of those products. IBM does not warrant the
quality of any third-party products, or the ability of any such third-party
products to interoperate with IBM’s products. IBM expressly disclaims
all warranties, expressed or implied, including but not limited to, the
implied warranties of merchantability and fitness for a particular,
purpose.
The provision of the information contained herein is not intended to, and
does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS,
Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document
Management System™, FASP®, FileNet®, Global Business Services®,
Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®,
IBM Social Business®, Information on Demand, ILOG, Maximo®,
MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower,
PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®,
PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®,
PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®,
SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®,
Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and
System z® Z/OS, are trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. Other product
and service names might be trademarks of IBM or other companies. A
current list of IBM trademarks is available on the Web at "Copyright and
trademark information" at: www.ibm.com/legal/copytrade.shtml.