Milwaukee ALT.NET Presentation on Webhooks, Asynchronous Web Applications and Push Notifications utilizing PubSubHubbub Presented on 02/20/2012

1. 1

2. I am the Application Architect at Montage Talent. My responsibilities at Montage
primarily include designing the next generation of the Montage video interviewing
solution. When I began with Montage, the service was a simple web application with
a single user interface. Since joining Montage I have led the effort to create a service
oriented application to allow clients to integrate our services using a web API and to
allow Montage to create additional web-based user interfaces including interfaces
intended to be used from PCs and mobile devices.
I have 10 years of professional development experience, 6 of which were in
consulting. Much of my experience has been in enterprise environments, and that
experience is what I have taken with me to Montage as I continue to help us build out
our service oriented architecture.
Lastly, I ultimately love my job because I get to help people solve problems, and
solving a specific problem is the basis for this presentation.
2

3. Montage Talent is The Leader in Video Interviewing Technology. The service that
Montage provides is the Montage Network which is a video interviewing solution sold
as multiple applications; Montage Interview & Montage View.
Montage Interview is the application that we are using to live stream and record this
presentation. Montage Interview is a live interviewing platform that allows for up to
16 live cameras to be connected in one live virtual interview room. Montage View is
our recorded interview application. As a candidate, a recruiter may respond to your
job inquiry asking you to compete a Montage View. During this process you will be
prompted to answer a series of questions using your webcam, and your answers are
then available for the hiring company to review.
Montage does more than just provide the technology to power video interviews. Our
solution allows our clients to elevate their brands by demonstrating to their
candidates that they understand the needs of today’s candidate and utilize the latest
technology to make the recruiting process as quick and seamless for the candidate as
possible.
Our solution also enhances the candidate experience by allowing our clients to reach
out to candidates, easily allowing candidates to apply for a job in any location around
the globe. Our solution is also highly beneficial to those candidates that can’t express
all of their qualifications in a traditional resume. By allowing candidates to use video,
candidates are able to demonstrate other soft skills on video, that may not translate
3

4. well to a written document.
3

5. Here is an overview of what tonight’s presentation will entail.
First I want to present what my intent is for speaking with you tonight and why I
thought this would be a useful topic to share. Then I will present the material in a
problem solution format discussing what caused me to stumble upon this solution
and discuss how I implemented the solution. Lastly, there should be time at the end
for any questions anyone might have or discussion around how the demonstrated
patterns and protocols might be useful in your environment.
4

6. I am going to work through this presentation by first introducing the problem space
that we faced in our business model that caused us to implement the technical
solution that my presentation is based on.
The intent of my presentation is not to tell you “this is how you should solve this
problem,” but rather use the problem space and my solution as a vehicle for a larger
discussion around “the real-time web”, push notifications/events and webhooks. I
have found the problem space to be a very common issue, but I had a difficulty
finding a common pattern to solve it. By the end of my presentation I hope that you
are able to identify this problem in your environment and are able to use concepts
presented and discussed during this session to solve your problem.
5

7. While I was designing the new API services to be implemented at Montage, I quickly
realized that Montage was unlike most other services I had worked with before. The
workflow that we needed to support through our services is one that make take days
to complete. Unlike the services I was used to working with, we are unable to give
our API consumers immediate results in some cases. The ultimate result of a request
may come back to the consumer days later, and during that time we can keep our
consumers up-to-date on the status of the request, but it’s a far cry from immediate
results.
Thus, we needed to develop a way to send status updates and the result to our
consumers over this extended period of time and herein lies the problem.
Our services are exclusively RESTful, so implementing any sort of proprietary protocol
or messaging was out of the question, especially since we don’t control our
consumers. If this were just an internal service-to-service problem any off-the shelf
messaging solution could have worked. But our messaging needed to support
HTTP/S at a bare minimum. As a nice to have, if the messaging solution could also
support other delivery protocols like SMTP, SMS, etc., that would be a plus.
Lastly, we needed to be sure that the solution we developed is reliable else our
consumers would never know the status of their requests. This meant we had to
either a) guarantee delivery b) provide a way to query for status updates.
6

8. I had in my mind that the solution I was looking for was essentially a post office. I
wanted to be able to utilize some service in which I could drop off a message
indicating delivery instruction, the payload and the receiver. This service would then
do its best to deliver the message and if the receiver could not be reached, would fail
and notify me of such failures for reconciliation.
Some of the solutions I looked into were various ESBs and Message queues.
• BizTalk
• RabbitMQ
However, at the time I wasn’t able to find a solution that was going to be able to
deliver messages reliably over HTTP/S, so I started poking around looking for anything
I could find to point me to a common solution to this problem.
I even asked a question about this on StackOverflow thinking that the thousands of
developers out there would be able to help point me toward a solution I was looking
for, but all I got were crickets. As you can see from the screenshot, I ultimately
answered my own question.
7

9. What I eventually found were WebHooks. Unfortunately the term WebHook is just
barely ubiquitous enough to make the term discoverable given the parameters I was
searching for when I didn’t know what I was looking for. The term WebHook was
coined by Jeff Lindsay, and he has a number of presentations available discussing
webhooks.
Essentially, a WebHook is an HTTP POST used to notify a subscriber of an event that
has occurred. It represents one way to begin enabling the real-time web. If a service
does not offer WebHooks or some other type of notification service, consumers are
forced to constantly poll the service looking for updates, which is sorely inefficient for
both the publisher and subscriber. Not only is it inefficient, it also isn’t real-time. It’s
only as timely as you are allowed to poll. Since WebHooks represent data that is
pushed from a publisher to a subscriber, the subscriber will receive real-time data
efficiently.
But what about RSS/ATOM some might say? With a feed I can subscribe to and
consume data from a publisher and those specifications are widely implemented.
The problem with feeds is that the publisher doesn’t send the feed data to a
subscriber. The publisher simply publishes the data to a well known location and
anyone interested in consuming it has to come fetch it. Thus, feed readers have to
implement polling to fetch the data. Again, this means that the reader is sorely
inefficient and you never get your data immediately after it has been made available.
8

10. Facebook’s realtime API uses webhooks and is loosely based on PubSubHubbub, the
solution protocol that I will dive into in a bit.
Stripe has just released a newly updated events API. Since their API is strictly JSON,
they have a different implementation than I am going to present on today, but the
concept is the same.
Twillio uses WebHooks to send SMS messages.
GitHub uses WebHooks to implement git push.
9

11. Just recently the term “WebHook” seems to be starting to catch-on. While the
concept isn’t anything really new, the usage of the term is. Also, there is no standard
implementation of WebHooks. There are quite a few implementations out there but
there is no standard pattern.
There have been a few attempts to create a standard. The webhooks.org wiki makes
an effort at creating a specification for a RESTful implementation, however their
attempt simply shows examples of how this can be done, and is hardly a
specification.
One specification comes from the XMPP Standards Foundation. XMPP maintains
standards for use in IM/real-time messaging scenarios. One of these specs is XMPP
PubSub. XMPP maintains “pure” standards. Implementing XMPP PubSub yourself
wouldn’t be a good use of most developer’s time since you would spend a lot of time
just implementing the protocol. Yes, finding a library to implement would ease the
pain, but then you have the other side of the fence, your consumers. Not only would
you have to implement the protocol, but so would your consumers.
That’s where PubSubHubbub comes into play. PuSH is designed to be a “pragmatic”
protocol. One that is much lighter weight and easier to implement than XMPP, but as
10

12. a result, isn’t as well defined on all fronts, which we will see shortly.
10

13. PuSH was developed by 2 Google engineers, Brett Slatkin and Brad Fitzpatrick. Brad
is best known for creating both LiveJournal and memcached.
PuSH is a simple to use, simple to implement, topic-based publish/subscribe protocol
based on ATOM/RSS. The goal of PuSH is to convert ATOM/RSS feeds into real-time
data by eliminating the traditional polling that occurs to consume most feeds. While
it’s not a RESTful protocol itself in-terms of the self-subscribe/unsubscribe
specification, publishing is HTTP based.
PuSH is simple. It consists of 3 participants:
• Publisher
• Subscriber
• Hub
Together, these 3 participants can be combined to create a real-time messaging
system that communicates over nothing but HTTP/S.
11

14. The link in this slide to the Subscription Flow, links to the PubSubHubbub’s project
site with a slide deck describing how the subscription process works.
The XML snippet shows an example of what the <link /> node would look like in an
ATOM feed that supports PuSH.
To subscribe, a subscriber makes an HTTP POST request to the link provided in the
hub node. The hub will then verify the subscriber and return an HTTP error code that
indicates if the subscription was successful or not. Hubs can verify subscriptions
either synchronously or asynchronously. If the hub is using synchronous verification it
will return a 204 “No Content.” If asynchronous, it will return a 202 “Accepted.” In
this case, the hub will do what it needs to verify the subscription and will send a HTTP
GET request to the subscriber indicating that the subscription has been accepted. If a
subscription request is denied, appropriate codes in the 4xx-5xx range will be
returned.
Callback authorization uses a challenge key to authorize the subscription
Subscribers can renew their subscription at anytime by re-subscribing
12

15. PuSH is designed so that all complexity exists in the hub. All a publisher has to
provide is a link in their feed to their hub and then they need to negotiate with their
hub on how they will notify the hub that they have updated content.
All the rest of the work is put on the hub. The hub is responsible for:
• accepting and verifying subscriptions
• managing the subscriptions
• handling update pings from the publishers
• extracting new and updated data from the publisher’s feed
• sending subscribers their new content
• DoS protections
Publishers may be their own hub, or they can use a commercially available hub. The
protocol allows for publishers and hubs to negotiate how they are going to
communicate. This means they get to choose the protocols and data formats they
will use internally. As a matter of fact, Montage has taken some liberties with this
flexibility in the protocol, which we will get into in a bit.
13

16. Thrift is a cross-language services development framework open sourced by
Facebook.
13

17. Superfeedr is the largest public PuSH hub. Since PuSH is a pragmatic protocol it
doesn’t define the implementation to a T. This allows hubs to take some liberties
with their implementations, and Superfeedr has taken advantage of this.
• They have added digest notifications which will give subscribers a digest of their
subscription, and they suggest it can be used as a heartbeat to ensure your
subscription is still active.
• Feed status provides information on feed such as how much data was fetched,
how many new entries there were and when the next fetch will occur.
• Virtual feeds allow subscribers to filter their feed at the hub.
http://boxcar.io/ - Instant Twitter, Facebook & Feed notifications to mobile devices
14

18. This diagram demonstrates the messaging infrastructure within the Montage
Network.
At Montage we are currently maintaining two APIs. One is our Core API and the
other is our Notifications API. The Notification API is nothing more than endpoints
that respond to GET requests with an ATOM feed.
Since the PuSH spec leaves the communication between the publisher and the hub
open for negotiation, we have taken some liberties that have allowed us to use push
notifications, even though our entire infrastructure has not yet been converted to use
real-time events. At this time, Montage’s main user interface has not yet been
converted to use the API, so any interaction with the system through that interface is
not setup to raise real-time notifications to the hub.
To compensate we have setup a polling service to pull data from the notification
feeds. This polling actually accomplishes 2 needs. One is the aforementioned issue
we have supporting a legacy application. The second, however, is that we use the
polling service as a catch-all for any missed notifications. We have made a decision to
centralize all complexity in the hub, which means the update pings from the Montage
API are not reliable. If one fails we don’t care and we won’t retry. The polling service
15

19. will eventually pickup the missed event.
15

20. You may potentially be wondering what this fetch update is for. If we have polling in
place, that will pull event data from the Montage API and if we support an update
ping from the Montage API, what is this doing there?
There are 2 ways we could have implemented our update pings. The way we are
doing it is that we use a ping that just tells the hub that new or updated data for a
given client is available on a feed. The hub then takes this information and fetches
the updated data. This allows the hub to control the amount of data that it’s fetching
and allows it to control how it gets the data. Thus it could fetch the data
immediately, or could wait if it wanted, but it has that flexibility.
The other way it could be implemented would be with a fat ping. The Montage API
could be configured to send a ping that includes the data being updated to the hub.
This yields the benefit of reducing the callback step and could help with an accidental
DoS on the Notification API if the hub got overloaded with pings that all required
callbacks to the same feed endpoint. These are all great benefits and decisions we
weighed and will continue to look at. However, for now our pattern is to yield full
control to the hub, which is one reason why we have chosen the latter type of ping.
16

21. As I mentioned at the beginning, I was initially looking for a solution that would allow
us to send notifications over HTTP/S, SMTP, SMS, etc. Since I was unable to find a
tool or service I realized that PuSH could actually enable us to do this quite easily.
Our solution is that we are creating our own subscribers that will translate the
notification POSTs into whatever protocol we ultimately want the notification
delivered over. For example, when a recruiter sets up a live interview, emails need to
be sent to the participants with the meeting link. What we do is we setup a
subscriber for that client with a URL to our Email Service. The Email Service accepts a
notification that a new interview has just been created, it merges the participants
with an email template and sends the email to the recipient.
With this pattern, we can support any outgoing protocol. In addition, we also gain
the benefit of allowing flexibility for our consumers. Some of our consumers would
rather send out communications themselves. They have the flexibility to setup their
own subscribers to do the translation and delivery of these messages if they desire.
17

22. As for what our endpoints look like, our pattern is to create an endpoint per
notification type in the Notification API. Data is segregated by client and can be
limited by the date/time of the notifications and can have count limits set by the take
and skip parameters . By using a feed that supports subscriptions, we give our
consumers flexibility in how they wish to consume notifications.
If they choose not to setup an endpoint that we can use to POST to for subscriptions,
the consumer is able poll their feed for updates (as long as they don’t DoS us).
However, as we would prefer they do, they may setup a subscription to any of the
notification endpoints and we will notify them of new events.
18

23. This diagram represents the internals of the Montage Notification Hub.
The Montage Notification Hub is a service that accepts update notifications from the
Montage Core API when events occur, fetches data that applies to these events and
forwards the data to anyone who has registered as a subscriber for that data. Our
hub maintains its own database of feeds and subscribers as well as any subscription
verification data it may need to accept new subscribers.
There is one part of this diagram which refers to something that as of yet, has not
been addressed, and that is message reliability. Just before I mentioned that we have
pushed all reliability to the hub, and this is where it comes into play. We will be
implementing a queuing system to give us message durability and reliability. The
addition of this queue will allow our subscribers to configure what to do in the case
of a failure. Will we try to resend? How many times will we try to resend, and how
often? What happens when all that fails? The PuSH spec addresses reliability, but
very loosely. “Hubs SHOULD retry notifications repeatedly until successful (up to
some reasonable maximum over a reasonable time period).” Obviously the spec isn’t
too helpful in this regard, so in our case, our intent is to leave the reliability
configuration up to the subscriber.
19

24. As mentioned, the PuSH spec is pragmatic, not pure, and definitely doesn’t define
every nuance of an implementation. One of those nuances is authentication.
The only thing close to authentication that PuSH addresses is transmission layer
security and message security. Transmission security is easy and obvious, HTTPS,
where possible. The spec does mention how to distribute content securely over an
open connection, which involves encrypting the payload of the message.
What the spec doesn’t address is what if the subscriber requests that the hub
authenticate with it before it will accept notifications? This may not seem like a big
deal when you are using a hub to retrieve blog postings, but when you are
transmitting sensitive data, authenticating with the subscriber isn’t a terrible thought.
As a result of our client’s needs, we have taken our own liberties with the spec and
have implemented the ability to authenticate with our subscribers. Our architecture
should allow us to authenticate with any subscriber in the manner in which they
would like us to authenticate.
HMAC Hashing
20

25. Ideally we would like to utilize a public hub or some other service to do this. While
we think that the solution to our problem has been well designed, maintaining a hub
and sending notifications isn’t our core business. We need it to support what we do,
but actually making it happen is a distraction from our core functionality. For us to go
to a service, the service is going to have to be flexible enough for us to accommodate
the varying needs of our consumers, or we will have to work with our consumers to
conform to the way that we send notifications.
While it’s ideal that we work with a service provider for our notifications, that doesn’t
seem like it’s in the cards anytime soon. As a result, we are planning to implement
queuing and are looking for both queuing tools and tools to manage reading items off
the queue. Managing reading the items off the queue, again, isn’t core to what we
do, so any package we can utilize to manage that will be in our best interest.
Self subscribe/unsubscribe will be added in the future allowing our clients to
maintain what notifications they would like to receive.
Lastly, for simplicity all of our messaging is HTTP/S within the Montage Network, as
well as the notifications that are sent to our consumers. A change we would like to
make is to a lighter faster protocol internally, especially between the publisher and
21

26. the hub.
21

27. 22

28. 23

29. 24

30. 25