You’ve decided to develop in Azure and need to make a decision on the messaging technology. Storage Queues, Service Bus, Event Grid, Event Hubs, etc. Which technology should you use? How do you pick the right one if they all deal with messages? This session will help you answer these questions.
12. Storage Queues are
Task Queues
• Coordinate work across compute
• VMs, App Service, Functions, etc.
• Messages contain data and senders have expectations
• Low cost
• Easy to use
• Unlimited number of queues
• Unlimited number of concurrent clients
• Maximum queue size 500TB
• Maximum message size 64KB (48KB with Base64 encoding)
• Limited features
13. What did get sacrificed?
• No headers/metadata
• No pub/sub
• No scheduling
• No dead-lettering
• No transactional guarantees
• Cannot peek more than 32 messages
• 7 days retention period
• Infinite (almost) lately, but careful
• and other compute-related features
14. Simple Queue in Action
• Sender sends message to queue
• Queue ACKs receipt
• Receiver connects to queue & retrieves message
• Receiver ACKs complete (or other action)
20. .NET Client
Helper
• Concurrent Message Pump
• RegisterMessageHandler()
• Full control over
• Concurrency
• Lock renewal
• Message completion
21. Premium Features
• Scale up (MUs)
• Geo-DR
• Availability Zones
• Virtual Network
Service Endpoints
22. Migrating from Standard to Premium
• No downtime
• Same connection string for Senders/Receivers as before
• Some messages need to be drained
• Up to 1000 entities
26. How Event Hubs is
different from queues
• Records a stream
• Recoding moves forward only
• You can play a stream over and over again
• Like a cassette tape it has channels
• When recording, all channels record
• But the data on each channel is different
• Event Hubs calls these channels “partitions”
29. How are Event Hubs and Kafka similar?
• Both are ingesting technologies, not a queue
• Both use partitioned consumer
• Partitions are independent
• Both use a client side cursor concept
• Both can scale to very high workloads
• Conceptually they are nearly the same
• Every Kafka user is close to being an Event Hubs user
30. Event Hubs for Apache Kafka™
Ecosystems
• Kafka 1.0 compatible endpoint can be backed by Event Hubs
• Use existing Kafka application & tools with Event Hubs
• Only need to change the connection string
31. Event Hubs Scaling
• Throughput Units (TUs)
• Variable reserved capacity (paid for) –Dedicated Tier
• 1 MB/second or 1,000 events/second ingress
• 2 MB/second or 2,000 events/second egress
• Overages are throttled (Server Busy Exception)
• A feature to help with it
32. Auto Inflation
• With insufficient TUs senders will be throttled
• Will go up (up to a cap), but won’t scale down
34. Event Hubs Capture
lets you go
fast and slow
• Automatically send Event Hub data into the Storage account
or Azure Data Lake
• Minimize your overhead: no code or other services to
configure
• Focus on data processing, not capturing
45. Event Grid Concepts
1. Event: what happened
2. Event source: where it took place
3. Topic: where publishers send events
4. Event Subscription: what events to handle
5. Event Handler: the app or service reacting to the event
47. Event Filtering
• Event type / subject / topic
• Event data (data.key)
• Values (numbers, strings, Boolean, array)
• NumberLessThan
• NumberLessThanOrEquals
• NumberGreaterThan
• NumberGreaterThanOrEquals
• NumberIn – the value for data.key equals a value in the set [0, 2.08, 3.14]
• NumberNotIn – the value for data.key is not in the set [1, 11, 112, 1124]
• StringContains – the value for data.key contains “the”
• StringIn – the value for data.key equals a value in the set [“small”, “brown”, “fox”]
• StringNotIn – the value for data.key equals a value in the set [“jumped”, “over”, “the”]
• StringBeginsWith – the value for data.key begins with “lazy”
• StringEndsWith – the value for data.key ends with “dog”
• BoolEquals
50. Why is Event Grid different?
• Cloud Native by design
• Serverless friendly
• Engineered for reliability and scale
• Supports CNCF Cloud Events v0.1
51. Cloud native
by design
• Always available (99.99% SLA)
• Near real-time event delivery (<1s e2e)
• At-least-once delivery
• Dynamic scale
• 10,000,000 events per second per region
• 100,000,000 subscriptions per region
• Platform agnostic (WebHook)
• Language agnostic (HTTP protocol)
52. Engineered for
scale and
reliability
• Defaults: 30 delivery attempts / 24 hours
• Retry intervals
• 10 seconds
• 30 seconds
• 1 minute
• 5 minutes
• 10 minutes
• 30 minutes
• 1 hour
• Once an hour up to 24 hours
53. Engineered for scale
and reliability
Dead-lettering
• Requires Storage account + container
• Dead-lettered events stored as blobs
56. Ubiquitous
• Today there are 10+ Azure publishers
https://docs.microsoft.com/en-us/azure/event-grid/event-sources
• At some point in time most Azure services will be publishers
59. How Event Grid composes with Queues and Streams
Messaging services can be publishers or subscribers to Event Grid
sub1
sub2
mytopic
Event Hubs
Storage queues / Service Bus
60.
61. Security and Authentication
• Validation Handshake (WebHook event delivery)
• Event of type Microsoft.EventGrid.SubscriptionValidationEvent
• With validation data
• Prove
• Echo back
{validationCode: “value”}
• Send GET to validationURL
62. EVENT DRIVEN PUB/SUB NOT ENTERPRISE
MESSAGING
SCALE THROUGHPUT
RETRIES DEAD-LETTERING CNCF CLOUD
EVENT SUPPORT
CROSS-CLOUD
Remember
And what about Azure SignalR?
It can send and broadcast message as well.
Does it mean it replaces Service Bus and Event Grid both? No! This is not the messaging we’re talking about in this session.
Azure SignalR is intended fo rnear real-time content updates. It’s a notification service. Notifications, if lost, are not as critical as messages that would be lost.
Some examples of real-time content updates:
- Apps that require high frequency updates from the server. Examples are gaming, voting, auction, maps, and GPS apps.
- Dashboards and monitoring apps. Examples include company dashboards and instant sales updates.
- Collaborative apps. Whiteboard apps and team meeting software are examples of collaborative apps.
- Apps that require notifications. Social networks, email, chat, games, travel alerts, and many other apps use notifications.
Remember, these are real-time web applications. The messages are not durable, meaning they are not stored anywhere and if fail delivery are gone.
With messaging market segmentation and product categorization I feel that I know what service to use when,
But when facing a real-world project, sometime the clear line between those becomes somewhat blurry.
So lets start looking at each service, it’s purpose, capabilities, strong and weak sides.
- A reference to MSMQ (when on-premises solutions roamed the world)
- Mention it's HTTP-based implementation
- Tasks queues coordinate work across compute (VMs, Functions, etc.)
- The messages in a queue are generally homogeneous
- Low cost
- Pay as you Go billing model
- Easy to use
- Have as many queues as you like (or connections)
- Maximum queue size is insane while message size is small
- mention claim check that's required to be implemented
- Very few features – next slide
Received message acknowledgement (ACK) is known as pop receipt
Scale up - 1, 2, 4 or 8 messaging units
Geo-DR - meta plane only, no data. Triggered manually, never automated.
Azure Zones – 3 “copies” in the same region to ensure HA
VNSE – traffic from between VNET and ASB service is kept secure from the public internet access
* features in Preview
Each consumer group provides a view into partitions for parallel work
Up-to 20 consumer group per EventHub (Consumer Group per app)
Number of Partitions directly relates to the # of concurrent reader instances within a consumer group
EventHubs is not intended to be used as a permanent data store.
Retention periods greater than 24 hours are intended for scenarios in which it is convenient to replay an event stream into the same systems; for example, to train or verify a new machine learning model on existing data.
If you need message retention beyond seven days, enabling Event Hubs Capture on your event hub pulls the data from your event hub into the Storage account or Azure Data Lake Service account of your choosing.
This method checks the sizes of the batch, the EventData object and the specified limit to determine if the EventData object can be added. It does not perform other validations on the current batch and the EventData object
EventProcessorHost stores offset in storage to scale out consumption
Event Grid is one of the latest additions to the messaging services that has recently GA-ed.
It’s an eventing backplane that enables event-driven, reactive programming based on a publish-subscribe model.
Traditionally, with queues or subscriptions, a message is sent and it needs to be RECEIVED. An application is responsible to poll for messages.
This requires a continuous execution of a process that checks for new messages.
But when we have applications that need to react to occasionally sent messages, we no longer need to have a 24/7 running process to check for new messages.
The polling model is not viable anymore. And with a rise of serverless options, it even became more apparent that some applications need a push model to react to changes.
And it all boils down to events. Let’s define what an event is.
Some example would be detecting specific objects in the image; generating thumbnails; etc.
And this is what’s possible today.
An event is the smallest amount of information that fully describes something that happened in the system. Every event has common information like: source of the event, time the event took place, and unique identifier. Every event also has specific information that is only relevant to the specific type of event. For example, an event about a new file being created in Azure Storage has details about the file, such as the lastTimeModified value. Or, an Event Hubs event has the URL of the Capture file. Each event is limited up to 1 MB of data (in 64 KB chunks).
A publisher is the user or organization that decides to send events to Event Grid. Microsoft publishes events for several Azure services. You can publish events from your own application. Organizations that host services outside of Azure can publish events through Event Grid.
An event publisher (aka source) is where the event happens. Each event source is related to one or more event types. For example, Azure Storage is the event source for blob created events. IoT Hub is the event source for device created events. Your application is the event source for custom events that you define. Event sources are responsible for sending events to Event Grid.
The event grid topic provides an endpoint where the source sends events. The publisher creates the event grid topic, and decides whether an event source needs one topic or more than one topic. A topic is used for a collection of related events. To respond to certain types of events, subscribers decide which topics to subscribe to.
System topics are built-in topics provided by Azure services. You don't see system topics in your Azure subscription because the publisher owns the topics, but you can subscribe to them. To subscribe, you provide information about the resource you want to receive events from. As long as you have access the resource, you can subscribe to its events.
Custom topics are application and third-party topics. When you create or are assigned access to a custom topic, you see that custom topic in your subscription.
When designing your application, you have flexibility when deciding how many topics to create. For large solutions, create a custom topic for each category of related events. For example, consider an application that sends events related to modifying user accounts and processing orders. It's unlikely any event handler wants both categories of events. Create two custom topics and let event handlers subscribe to the one that interests them. For small solutions, you might prefer to send all events to a single topic. Event subscribers can filter for the event types they want.
A subscription tells Event Grid which events on a topic you are interested in receiving. When creating the subscription, you provide an endpoint for handling the event. You can filter the events that are sent to the endpoint. You can filter by event type, or subject pattern.
An event handler is the place where the event is sent. The handler takes some further action to process the event. Event Grid supports multiple handler types. You can use a supported Azure service or your own webhook as the handler. Depending on the type of handler, Event Grid follows different mechanisms to guarantee the delivery of the event. For HTTP webhook event handlers, the event is retried until the handler returns a status code of 200 – OK. For Azure Storage Queue, the events are retried until the Queue service is able to successfully process the message push into the queue.
Event Grid provides security for subscribing to topics, and publishing topics. When subscribing, you must have adequate permissions on the resource or event grid topic. When publishing, you must have a SAS token or key authentication for the topic.
Here’s an example of an event generated by Storage Blob when a new blob is create.
Notice that there’s a topic, subject, and eventType that every event will have.
topic - full resource path to the event source. This field is not writeable. Event Grid provides this value. (here: Storage Account)
subject - Publisher-defined path to the event subject. (here: Blob Container)
eventType - One of the registered event types for this event source. (here: blob created event)
Each event is comprised of data. Data schema is defined by event publishers.
Note: that events can be bundled and subscribers could receive multiple events.
X
How is Event Grid is different from Azure Service Bus? It’s cloud native by design.
Built to address serverless issues with reactive nature of communication.
Which includes built in reliability and massive scale it can handle – cross data-center
It was designed for cloud scenarios, highly available with near real time delivery end-to-end
It has at-least-once delivery semantics.
The scale is incomparable to a service such as ASB.
It was designed in a way that can support various platforms. Yes, not just Azure.
And thanks to the HTTP protocol use it can be used from any platform.
Event Grid adds a small randomization to all retry intervals. After one hour, event delivery is retried once an hour.
By default, Event Grid expires all events that aren't delivered within 24 hours. You can customize the retry policy when creating an event subscription. You provide the maximum number of delivery attempts (default is 30) and the event time-to-live (default is 1440 minutes). [30=7 in the first hour + 23 with one-per-hour]
By default, Event Grid expires all events that aren't delivered within 24 hours. You can customize the retry policy when creating an event subscription. You provide the maximum number of delivery attempts (default is 30) and the event time-to-live (default is 1440 minutes). [30=7 in the first hour + 23 with one-per-hour]
CloudEvents simplifies interoperability by providing a common event schema for publishing, and consuming cloud based events.
This schema allows for uniform tooling, standard ways of routing & handling events, and universal ways of deserializing the outer event schema.
With a common schema, you can more easily integrate work across platforms.
Azure Logic Apps,
Azure Automation,
Azure Functions for EventGrid Trigger.
Only HTTPS is supported!
Summary
In the ideal world, we’d want to have a very clear instructions about what messaging service to use when
So lets have a look at these services to understand better when what services make sense by understanding their key features.
