ZendCon - Integration and Asynchronous Processing with ActiveMQ and Camel

© 2016 Rogue Wave Software, Inc. All Rights Reserved. 1
Integration and
Asynchronous
Processing with
ActiveMQ and
Camel
Justin Reock
Sr. Director and Lead Architect
Rogue Wave OSS Support

Who’s speaking to me?

Justin Reock
• Sr. Director and Lead Architect of our OSS Support Team
• Way too much experience with ActiveMQ and Camel
• Developed our formal instructor-led training
• Numerous professional services and training engagements
• Early adopter of JMS and Camel
• Major FL/OSS evangelist!

What are we going to solve?

Asynchronous processing
• No one likes the spinning wheel of doom hope!
• We’re going to talk about offloading work to a backend server
• We’re going to do it in a way that’s reliable, flexible, and guaranteed
• Built on industry standards and vetted against thousands of production
systems

What else?

Quick exercise…
• I’m going to show you three slides
• They all have something in common….
• See if you can figure out what it is!

The Chupacabra…

A magical unicorn…

BillingFulfillment
CRME-Commerce
A homogeneous enterprise
landscape…

Answer
 None of these things exist in real life!

I’m listening…

Meet: The solutions!
• We’ll be leveraging two related technologies, Apache ActiveMQ and
Apache Camel
• Apache ActiveMQ is a JMS implementation, and can help with both
federation of systems and asynchronous processing
• Apache Camel is a sophisticated message oriented middleware
language, with hundreds of integration components
• But let’s take a step back and answer a couple questions first…

What is JMS?

Messaging and JMS
• Applications often have a need to send information back and forth to one
another in a normalized fashion
• Before the advent of messaging systems, it was often difficult if not
impossible to “federate” applications written in disparate languages or
residing on heterogeneous platforms
• An early goal of Java was to present a language with the power of C which was
not hindered by cross-platform issues
• JMS, the Java Messaging Service, arose out of a growing need to federate
very different systems with each other
PHP Application AS400 Application

Messaging and JMS
• JMS 1.1 – Java Messaging Service
– Intent was to build vendor-neutral specification for sending enterprise
messages through Java
– JMS is just a specification
– Messaging consists of Messages, Queues, and Topics
– Transport endpoints are called Producers and Consumers
• JMS Provider
– These are the vendor-specific implementations of JMS
– ActiveMQ is a JMS Provider
– A JMS Provider must implement the JMS Spec
– A JMS Provider forms the framework for facilitating the use of JMS
concepts inside an application
– JMS is portable, but JMS providers may not be

What is Message Oriented
Middleware?

Message oriented middleware
• Technically, MOM is any platform that sends and receives messages
between distributed applications
• This allows for asynchronous processing, and normalization of data
exchanges
• Clients connect to a messaging provider, and send and receive
messages via that provider

What’s it good for?
• Data normalization makes it a lot easier for disparate applications to
trade information
• Just like in web services, heterogeneous systems can agree on a data
“contract” and send and receive data in that format
• All the application needs is a client compatible with the messaging
provider
• And since messages are allowed to queue up on the message provider,
it also makes it very easy to perform asynchronous processing
• Most message providers offer additional benefits, like guaranteed
delivery, options for traffic shaping, high availability, and scaling

What is ActiveMQ?

Apache ActiveMQ
• ActiveMQ is a Java-based multi-purpose messaging middleware solution,
called a message broker, designed for high throughput and scale
• It is useful in a number of situations, including aggregating large amounts of
data in an efficient and safe manner, federating disparate or heterogeneous
applications across an enterprise, and/or moving data between
geographically separated data centers or applications
• ActiveMQ supports a number of connection protocols, allowing it to provide
connection points between applications coded in different languages, such as
Java, C++, .NET, PHP, etc
• Customers commonly migrate to ActiveMQ from commercial middleware
such as WebSphereMQ, Oracle AQ, TIBCO EMS, and/or MSMQ

Why is it so popular?
• ActiveMQ is a mature and well-adopted platform, with thousands of
companies using it, and backed by a strong community
• It is by far the most flexible OSS message broker available, offering
integration points for dozens of application frameworks and languages
• ActiveMQ ships with Apache Camel, one of the most revolutionary
integration frameworks available, and shares developers and committers with
the Apache Camel product
• ActiveMQ can be tuned for massive scale, supporting large data sets, such
as those generated by the CERN large hadron collider
• Its configuration can support numerous traffic and implementation models
• ActiveMQ has a powerful statistics and metrics engine, exposing crucial
details about the health of the broker and the amount and size of message
traffic and client status

Where does it fit?
• ActiveMQ acts as integration middleware, sitting between various
enterprise applications, data sources, and APIs, and “federating” data
between the systems

Where does it fit?
• ActiveMQ can also be used as a reliable message buffer
• Applications can send messages into ActiveMQ and continue working,
knowing that ActiveMQ will guarantee delivery of the message
• Or, ActiveMQ can be used to broadcast messages to several different
systems
• ActiveMQ supports transactions between
the broker and the client, so that exceptions
can be caught and managed
• Using these constructs, you can easily create
asynchronous message patterns by sending
or receiving messages from ActiveMQ

Destination types
Queues
• Queues are FIFO buffers of messages
• Multiple listeners, or “consumers” can be attached to a single queue, and
can round-robin or otherwise balance processing load
• This is called “fanning out”
• Client libraries can scale up and down consumers as necessary, but only
one consumer will ever receive a single copy of a message
Topics
• These are message broadcast channels
• A single message can be broadcast to multiple, single-threaded
“subscribers”
• Subscribers can scale using “virtual topics” in ActiveMQ, or “shared
subscriptions” in JMS 2.0

Quick demo…
• Let’s integrate a PHP subscriber with a Java publisher
• These demos ship with ActiveMQ, and can be found in the “examples”
directory
• We’ll be using /examples/openwire/java and /examples/stomp/php
• A Java producer will send 10,000 messages into a topic called “event”, and
a PHP listener will receive those messages

Important concepts
• Durability – Describes whether a message will be available to consumers later if they are
currently disconnected from the broker
– ActiveMQ defaults all queues to durable, and supports durable topic subscriptions
• Persistence – Allows messages to be written to disk storage, so that they will be
available later if the broker should crash
– ActiveMQ offers persistence through a choice of high-speed journaling databases,
or RDBMS
• High Availability – A factor determining how often a service is available, in this case how
often the broker is up and able to process messages
– ActiveMQ includes many native solutions for HA including shared filesystem
options, or true replication through LevelDB/Zookeeper
• Horizontal Scaling – Load balancing across multiple brokers and/or message
consumers
– ActiveMQ supports highly configurable consumer scaling, along with a Network of
Brokers for chaining multiple brokers together to load balance

Working with PHP
• ActiveMQ supports a bevy of transport protocols

Working with PHP
• The native protocol for PHP is Stomp, which is a text-based protocol
very similar to HTTP
• The Stomp library can be downloaded via pear
– pear install pecl.php.net/stomp
• Once the Stomp library has been installed and linked, a Stomp object will
be available which can manage every aspect of communicating with
ActiveMQ:

Additional features
Message grouping
Scheduled messages
Security framework
Metrics and statistics
Network of Brokers
Inactivity monitoring
Transactions
Fault tolerance
Traffic Shaping
Plugin stack
Virtual destinations
Temporary destinations
Transformation
Load balancing
Client failover
Property encryption
Destination policies
Dispatch policies
Producer flow control
Message swapping
And of course… Camel!

What is Camel?

What’s Camel?
• Camel is an open source MOM language developed by Apache and based
on Enterprise Integration Patterns
• Book by Gregor Hohpe and Bobby Woolf
• Centers around 60+ common patterns found
in Enterprise integration projects
• Defines these common patterns in a vendor
neutral way
• Camel aims to provide a language which
implements these patterns, in a style
modeled after the UNIX pipeline

Integration patterns
• Content Based Router
– Where a message is routed to disparate endpoints based on message criteria

• Content Enricher
– In which a basic message enters the pattern, takes data from another
source, and then adds it to the message

• Recipient List
– A message is broadcast to n channels based on message criteria. When this message
is broadcast to all channels regardless, it becomes a Multicast pattern

• Scatter-Gather
– In which messages are broadcast out to disparate endpoints and results are
aggregated into a single message

How does it work?
• Camel uses a repeatable, normalized concatenation of “Processor” and
”Message” objects in a group called an Exchange
• There is an ”In” message, a “Processor”, and an “Out” message
Processor
In Out

How does it work?
• And, you can chain these Exchanges together – just like piping commands
through UNIX – and form a Camel Route
• The “Out” message of a previous Exchange becomes the “In” message of a
new Exchange:
P
In Out
P
In Out
P
In Out

Camel magic
• Now here’s the magical part – a Processor can be anything you want
• Components in Camel work by creating Processors that perform various
business functions
• The components can be chained together to build Integration Patterns
• For instance, you can have an “FTP” processor that takes the In message
and initiates an FTP session
• Or you can have an “SMTP” processor that takes the In message and
sends an email with it
• This is in addition to Camel’s many built-in patterns, such as LoadBalance,
Multicast, DataSets, etc.
• And there are a lot of components:
http://camel.apache.org/components.html

Embedded scripting languages
• Camel also contains a number of embedded script languages that can be
used for various purposes
• Simple: The Simple query language can be used to introspect message
headers, payloads, and properties, and is useful when routing messages
based on their content
• Xpath: The XPath query language will let you traverse XML structures
and query for attribute and entity values, which is great for routing
messages based on their XML content
• JSONPath: Just like XPath, except for JSON-structured messages
• Groovy: The Groovy scripting language can be used for more advanced
data introspection and manipulation
• JavaScript: Yes, you can do entire processors in JavaScript

Camel source
• Camel routes can be coded in Java POJO
• Or by using Camel Spring DSL XML files
from("activemq:foo").filter().
xquery("//foo").
to("activemq:bar")
<route xmlns="http://camel.apache.org/schema/spring" id="TimerClient">
<from uri=”activemq:topic:broadcastData"/>
<setBody>
<constant>This is a test JMS message.</constant>
</setBody>
<to uri="activemq:Test_ActiveMQ_Route.inQueue"/>
</route>

Camel deployment
• Camel can be deployed in a number of ways:
• It can be deployed as a Java Web App using Spring or POJO
• Standalone Camel routes can also be deployed using Maven and Camel
• Spring Boot can launch Camel routes
• Camel can be embedded within another Java application
• Or within a Java-based container such as Jetty, Tomcat, JBoss/Wildfly, and
others
• ActiveMQ contains the Camel core libraries and can launch and deploy
Camel routes
• And OSGI containers like Karaf or its Servicemix counterpart can launch
Camel routes
• Standalone Camel routes scale very nicely in microservice containers as
well

Camel development
• Major IDEs like Eclipse and IntelliJ can ease Camel development by
allowing routes to execute within the IDE
• As of Camel 2.16, comprehensive in-line tools are available which allow
auto-completion and even list every parameter of an available component

Camel development
• But HawtIO is my current favorite way to prototype with Camel
• HawtIO allows for visual debugging of routes, tracing of message
lifecycle, diagram driven development, as well as monitoring

Camel monitoring
• Camel has a robust set of JMX instrumentation beans which can be
looked at to gain more information on the route
• Any JMX-capable enterprise monitoring solution will be a good fit for
Camel

Demo: Content based router
• In this example, we will be using a very simple content based router
• In fact, we’ll be using the Simple query language to introspect a JMS
header in a message, and routing it to a destination based on the contents
of that header
• We will use Camel deployed directly into ActiveMQ, and we’ll write it from
scratch against a clean ActiveMQ install

Putting it all together

The big picture
• So, ActiveMQ provides a platform by which disparate clients can interact
with one another in a normalized way
• ActiveMQ’s wealth of clients, including the Stomp, REST, and WebSocket
connectors, means almost anything can be a client
• It’s queueing mechanisms make reliable asynchronous messaging a
breeze
• Camel can provide sophisticated routing, transformation, and component-
driven integration
• And Camel’s huge component library ensures that any application can
route through Camel

Demo: ActiveMQ, Camel, and PHP
• In this demo, we’ll take a simple web form, and process its data through a
synchronous web service as well as an asynchronous queue
• Camel will provide the web service endpoint, and the form will call it
synchronously
• PHP’s Stomp client will be used to create a message on a queue for the
asynchronous transaction
• Delay has been built in to the messaging flow for both workflows

Questions?
Feel free to reach out after, too!
justin.reock@roguewave.com

ZendCon - Integration and Asynchronous Processing with ActiveMQ and Camel

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to ZendCon - Integration and Asynchronous Processing with ActiveMQ and Camel

Similar to ZendCon - Integration and Asynchronous Processing with ActiveMQ and Camel (20)

More from Justin Reock

More from Justin Reock (8)

Recently uploaded

Recently uploaded (20)

ZendCon - Integration and Asynchronous Processing with ActiveMQ and Camel

Editor's Notes