Enterprise Integration Patterns with ActiveMQ

Enterprise Integration:
Patterns and Deployments with Apache ActiveMQ
Patterns and Deployments with Apache ActiveMQ

Rob Davies
CTO
FuseSource
A Progress Software Company
July 2011Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. A Progress Software Company
1 Copyright © 2010

Your presenter is: Rob Davies

 CTO at FuseSource - “the experts in open source integration and
messaging” http://fusesource.com

 Apache ActiveMQ, ServiceMix and Camel PMC member

 Co-creator of ActiveMQ, ServiceMix and Camel

 Co-author of ActiveMQ in Action:
 Chapter 5:ActiveMQ Message Store

 Chapter 10: Deploying ActiveMQ in the Enterprise

 Chapter 11: ActiveMQ Broker Features In Action

 Chapter 12: Advanced Client Options

 Chapter 13: Tuning ActiveMQ for Performance

Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. A Progress Software Company
2

FuseSource - the experts in open source integration and messaging

FuseSource Sponsors the Apache Software Foundation

3

FuseSource: the Team that Wrote the Code

Over 25 active committers on 11 Apache
projects

4

Enterprise Integration Patterns

 A Book by Gregor Hohpe and Bobby Woolf
 Patterns and Recipes for common integration problems
 Message Centric
 Used as the basis for all the major integration products
 Should be the the first thing to reference when starting
an integration project
 http://www.eaipatterns.com/

5

Why use Messaging ?

 Remote communication between applications
• different processes on the same machine
• different process on different machines in different locations
 Platform/language integration - heterogeneous support
 Asynchronous communication - de-coupling between
producer and consumer (loose coupling)
 location transparency
 Fault tolerant communication
• guaranteed delivery
• once and only once delivery

6

What is Apache ActiveMQ ?

 Top level Apache Software Foundation Project
 Wildly popular, high performance reliable message broker
• supports JMS 1.1, adding support for AMQP 1.0 and JMS 2.0
• Clustering and Fault Tolerance
• supports publish/subscribe, point to point, message groups, out of
band messaging and streaming, distributed transactions
 Myriad of Connectivity options
• Native Java, C++ and .Net
• STOMP protocol extends reach to Ruby, JS, Perl, Python,PHP,
ActionScript ...
 Embedded and standalone deployment options
• Pre-integrated with open source integration and application
frameworks
• Deep integration with Spring Framework and JEE

7

Enterprise Integration

Message Channels ...

8

Message Channels = JMS Destinations

9

Publish/Subscribe Channel (Topics)

10

Point-to-Point Channel (Queues)

11


Message Routing ...

12

Message Routing: Destinations - ActiveMQ supports
Wildcards

13

Message Routing - selectors

14

Message Routing: Selectors - usually headers only

15

ActiveMQ also supports content based routing:

16

Destinations or Selectors ?

 Overhead to both destinations and Selectors
 Queues use more resources than Topics
 Selectors are OK providing there’s always a consumer for a
filtered message (“Stuck Message” problem)
 In general use more destinations before using selectors
 If you need finer control, then use selectors

17


Types of Messages ...

18

Types of Message: Command

19

Types of Message: Document

20

Types of Message: Event

21


Message Integration -
Push, Pull or both ?

22

Push Model for Integration

 Typically uses a document message being sent as an event
• Information about a change (e.g. a price change) is an event
• Information about a change and the changed information - is an
event/document combination

23

Push Model Using ActiveMQ for Scalability - FanOut

24

Push Model Using ActiveMQ for Scalability - FanOut
configuration

ActiveMQ producer connection URI - will connect to all
brokers:
fanout:(static:(tcp://broker1:61616,tcp://broker2:61616))

ActiveMQ Consumers connection URI - will connect to
only one broker
failover:(tcp://broker1:61616,tcp://broker2:61616)

25

Pull Model for Integration

 Three message types used
• Event message - to notify observers of changes
• Command message: - to request updated information
• Document message: - details of the change

26

Which model to use - Push or Pull ? - it depends :)

 Push model is good when:
• when all consumers want details of change
• Information (Document part) isn’t too large
 Push model is bad when:
• lots of consumers - but only a few want updated require updated
information
 Pull model is good when:
• lots of consumers, only a few will be interested in the change
• flexibility in the implementation
 Pull model is bad when:
• need to reduce traffic - 3 messages vs 1 for push
• 2 Destinations vs 1 for push

27

A Bad use of Pull:

28


Request/Reply

29

Two way conversation: Request/Reply

30

Two way conversation: Request/Reply with JMS

 javax.jms has helper classes for Request/Reply pattern
• QueueRequestor
• TopicRequestor
 Limitations
• Requests have to be persistent
• Request can’t be transacted
• Requestor is synchronous
• Uses a temporary destination for response:
– there maybe a network outage - loose response
– you may want to load balance responses - so need a Queue

31

Two way conversation: With ActiveMQ

 Don’t use javax.jms has helper classes for Request/Reply
 Don’t use temporary Queues
 What Queues that should only live as long as a
connection ?
• use timeToLive on messages
• enable garbage collection on destinations

32


Job Processing

33

Messaging for scalable Job processing

 Distributes processing
 Scalable - multiple consumers of each queue
 Fault tolerant - processing can survive Processor outage

34


Deployment Pattern - Hub and
Spoke

35

Geographically Dispersed Hub and Spoke

36

Geographically Dispersed Hub and Spoke - Challenges

 Need immediate time two way communication
between remote office and head office
 remote office applications still need to work if head
office is not contactable
 Network outage, or unreliable networks, need to be
catered for

37

Geographically Dispersed Hub and Spoke - Solution

38

Best practices for ActiveMQ networks

 For large number of remote brokers - or when
firewalls are an issue - use duplex networks - the
remote broker establish the connection to the “head
office”
 Use network filters - to cut down network chatter
 Consider using static includes on networks - if
networks don’t need to be dynamic
 Use compression - especially over slow links
 Networks are fault tolerant - so can failover to a slave

39


Deployment Pattern - Failover
between data centres ...

40

ActiveMQ use case - geographically dispersed data
centres

41

ActiveMQ Networks - using failover

42

centres - redundant links

 Fully redundant inter-connected brokers
 If a network link dies - messages need to automatically
routed another way
 Needs to work for both Queues and Topics

43

centres - redundant links - continued

 Set networkTTL to 2
 Queues will automatically work
 Topics need extra configuration ...

44

ActiveMQ use case - geographically dispersed data centres -
redundant links - continued - support for topics
Enable duplicate subscriptions over the network:
<networkConnectors> <networkConnector uri="static:(tcp://brokerB:61617)" name="A-B"
networkTTL="2" suppressDuplicateTopicSubscriptions="false"> </networkConnector>
<networkConnector uri="static:(tcp://brokerC:61618)" name="A-C" networkTTL="2"
suppressDuplicateTopicSubscriptions="false">
</networkConnector> </networkConnectors>

Ensure every Topic message is only sent through one network
connection - the one with the highest priority:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" enableAudit="true">
<dispatchPolicy>
<priorityNetworkDispatchPolicy/>
</dispatchPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy

45


What about other Enterprise
Integration Patterns ?

46

Some Integration Patterns

47

What is Apache Camel ?

 A powerful open source Mediation Router/Integration
Framework
 Based on Enterprise Integration Patterns
 100 + integration components
 Designed to:
• have no container dependency
• but ... work very well with Apache ActiveMQ, ServiceMix and CXF
• Can integrate seamlessly with Spring
• Easily extendable

48

What is Apache Camel

 Camel and EIP

Apache WebSphereMQ
ActiveMQ

49


 Camel and EIP

16 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. A Progress Software Company
50


 Camel and EIP

from filter send to
A message B

51


 Camel and EIP

from(A) filter(predicate) to(B)

52


 Camel and EIP

from(A) .filter(isWidget) .to(B)

53


 Camel and EIP

from(A).filter(isWidget).to(B);

54


 Camel and EIP

isWidget = xpath("/quote/product = ‘widget’");


55


 Camel and EIP

Endpoint A = endpoint("activemq:queue:quote");
Endpoint B = endpoint("mq:quote");
Predicate isWidget = xpath("/quote/product = ‘widget’");


56


 Filter Route

public void configure() throws Exception {

}

57


 Filter Route - Java DSL

import org.apache.camel.builder.RouteBuilder;

public class FilterRoute extends RouteBuilder {

}
}

58


 Filter Route - Java DSL

import org.apache.camel.builder.RouteBuilder;

public class FilterRoute extends RouteBuilder {

from("activemq:queue:quote")
.filter().xpath("/quote/product =‘widget’")
.to("mq:quote");
}
}

59


Lets look at the Content based
filter ...

60


 Content Based Router

61


 Content Based Router - XML DSL

<camelContext>
<route>
<from uri="activemq:NewOrders"/>
<choice>
<when>
<xpath>/order/product = 'widget'</xpath>
<to uri="activemq:Orders.Widgets"/>
</when>
<otherwise>
<to uri="activemq:Orders.Gadgets"/>
</otherwise>
</choice>
</route>
</camelContext>
62


 Content Based Router - Java DSL

from("activemq:NewOrders")
.choice()
.when().xpath("/order/product = 'widget'")
.to("activemq:Orders.Widget")
.otherwise()
.to("activemq:Orders.Gadget");

63


 Endpoints as URIs
use file instead

from("file:inbox/orders")
.choice()
.otherwise()

64


 Endpoints as URIs
parameters

from("file:inbox/orders?delete=true")
.choice()
.otherwise()

65


Apache Camel and ActiveMQ
work well together ...

66

ActiveMQ with embedded Camel: Flexible and Performant

67

ActiveMQ with embedded Camel: import camel into
ActiveMQ broker config:

<beans>
   <broker brokerName="testBroker"  xmlns="http://activemq.apache.org/schema/core">
    <transportConnectors>
      <transportConnector uri="tcp://localhost:61616"/>
    </transportConnectors>
  </broker>
  <import resource="camel.xml"/>
</beans>

68

ActiveMQ with embedded Camel: Setup Camel Context in
usual way

<camelContext errorHandlerRef="errorHandler" xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="activemq:queue:test.queue"/>
<choice>
<when>
<xpath>$foo = 'bar'</xpath>
<to uri="activemq:topic:topic.bar"/>
</when>
<when>
<xpath>$foo = 'cheese'</xpath>
<to uri="activemq:topic:topic.cheese"/>
</when>
<otherwise>
<to uri="activemq:topic:topic.all"/>
</otherwise>
</choice>
</route>
</camelContext>

69


Lets look at some challenges
for messaging

70

Messaging Challenges

 Scalability
• Vertical scaling - how do we support a 100,000 destinations?
• Horizontal Scaling - how can we linear scale greater than
100k destinations ?
 Performance - everything needs to be faster -
ActiveMQ - should be the fastest open source
messaging
 Continuous availability (active active clustering)
 Protocol support - there’s a range of choices -
ActiveMQ should support them.

71

CERN Large Hadron Collider

 Worlds largest and most powerful particle accelerator
 Currently 10 different Collider Experiments
 Budget of $9 billion
 30G/s optical network
72

CERN Large Hadron Collider

 Produces 15 petabytes of data annually
 WLCG – 34 different countries
 Lots of Data – lots of destinations
 Requires next generation messaging to cope with information demand
73

Need new Messaging Architecture

 Extreme throughput
 Scalable to millions of
Dynamic Destinations
 Extensive protocol support
 Scale to x100k connections

74

Introducing ActiveMQ Apollo

 Scala based core for very fast, scalable dispatching
 modular design – independent lifecycle support
 Enhanced Queues
 More Protocols
 Richer REST based Management
 Intelligent Clustering

75

Hawtdispatch

 Based on Grand Central Dispatch (from OS X)
 Event based – supports both reactor and proactor
patterns
 Uses fixed size thread pool – to match number of cores
available
 Reduces Thread contention - tries to use same thread
and scales to very large connections
 Handles millions of messages per second

76

More protocols than OpenWire and STOMP

 MQTT
• IBM developed open protocol- supported by WebsphereMQ, Mosquitto and
now Apollo
• Publish/Subscribe and Queues (version 5 spec)
• Designed to be used from embedded devices all the way up to application
stacks
 Beanstalk
• Short lived tasks
• sender needs to take action if a job is not consumed
• sender needs to know job is on the Queue
• very scalable
 AMQP 1.0
• First Enterprise Version
• supports distributed transactions
• supports reliable messaging
• Flow Control should now work
77

Automatic Destination Partioning (clustering)

 Uniform load across multiple brokers
 Clients automatically connect to the correct broker(s)
 Massive scalability
 Reduce network hops

78

What about performance ?

79

FuseSource Subscription = Long Term Success

Support
•Enterprise-class 24x7 coverage
•Global organization
•Mission-critical
integration expertise Influence
•Updates and migration •Product roadmaps
assistance
•Planning processes
Tools •Conduit to Apache
•Development
•Operations
•Management
•Performance
•Documentation

80

FuseSource Knows How To Build Enterprise Apps

Help throughout the software development lifecycle…

Pilot /
Investigation Development Production
PoC

Get started Try it out Build right Deploy safely
•training videos •tech overviews •dev. subscription •prof. subscription
•webinars •training •arch. workshop •health check
•tutorials •project planning •best practices •perf. workshop
•documentation •pilot workshop •QoS •HA workshop
•white papers •pilot subscription development •training
•training
Available for Free FuseSource Paid Engagement

81

FuseSource Track Record

100’s millions+ Transactions every day running on Fuse

200,000+ Downloads per month

200+ Customers and growing rapidly

100%+ Rapid growth of FuseSource for past 3 years

24 hr Enterprise Support every day

25+ Apache Committers and Contributors

15+ Years of Enterprise Integration experience

#1
Number one provider of
Open Source Integration and Messaging
products and services

82

Any Questions?

No vendor lock-in
Free to redistribute
Enterprise class…….. A Progress Software Company
83

Enterprise ActiveMQ - More Information:

 http://fusesource.com/
 http://activemq.apache.org/
 http://camel.apache.org/
 http://activemq.apache.org/apollo/

84

Enterprise Integration Patterns with ActiveMQ

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Enterprise Integration Patterns with ActiveMQ

Similar to Enterprise Integration Patterns with ActiveMQ (20)

More from Rob Davies

More from Rob Davies (8)

Recently uploaded

Recently uploaded (20)

Enterprise Integration Patterns with ActiveMQ

Editor's Notes