Scalable Messaging is the the need of the enterprise messaging infrastructure. JBossA-MQ is one of the leading products for scalable messaging. It also covers some common concepts beween Apache Active MQ and JBoss A-MQ . Apache Camel as a integration framework supports the enterprise messaging greately and the presentation also contains some of its areas of adjunction to Scalable messaging with queues and topics.

1. 1

2. Scalable messaging with
JBoss A-MQ and Camel
Bhavani Ananth
@bhavani_an
in.linkedin.com/in/bhavaniananth/
2
Naveen Raj Balasubramaniam
@Naveenrajbala
in.linkedin.com/in/naveenraj/

3. Messaging And Integration
JBoss A-MQ
JBoss A-MQ – Protocols
JBoss A-MQ – Message Stores
Vertical and Horizontal Scalability
3
Camel in brief!
Agenda

4. Messaging And Integration
4

5. Scalable Solution
5
Large Retail Industry
Billing Go Online Vendor
Inventory Warehouse Logistics
Integration Infrastructure
MessagingMessaging
Messaging
Messaging
MessagingMessaging

6. Data Is Growing
Every Second –
Data Deluge
More Data Due to
Emergence of
Internet Of Things
Need For A
Scalable Solution

7. What do we have!
Red Hat JBoss A-MQ
7

8. JBoss A-MQ
8

9. What is JBoss A-MQ
9
JMS 1.1-compliant messaging system based on
Apache Active MQ.
Consists of a broker and client-side libraries enabling
remote communication among distributed client
applications.

10. Architecture and Features
10
JBoss-AMQ
Apache Karaf + Fuse Fabric
Monitoring
and
Management
Apache Active MQ
STOMP MQTT
Open Wire AMQP NMS
JMS

11. JBoss A-MQ – Protocols
11

12. Protocols Supported
12
Client API’s
JMS
NMS
CMS
Rest/Ajax
Wire
Protocols
MQTT
AMQP
STOMP
Open Wire
Transports
TCP/SSL
HTTP/HTTPS
WS/WSS

13. MQTT
MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol.
URI- mqtt://Host:[Port]?transportOptions
13
Facebook Messenger Smart Santander Features
Open
Simple
Works well in
constrained networks

14. AMQP
Features
 Multi-Vendor
interoperability
 Highly reliable
 Feature rich
AMQP (Advanced Message Queueing Protocol) is an openly published wire
specification for asynchronous messaging
URI: “amqp:// Host:[Port]?transportOptions
14

15. STOMP
STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol.
URI: “stomp://localhost:61613”
15
Broker
Producer
STOMP
Consumer
(Ruby,PHP)
TCP
Features
 Simple
 Used in scripting
languages

16. Openwire
Features
Default wire format used by ActiveMQ.
Efficient
Supports auto reconnect
Supports client failover
OpenWire is the cross language Wire Protocol to allow native access to ActiveMQ from
a number of different languages and platforms
16

17. JBoss A-MQ – Message
Stores
17

18. Message Stores
KahaDB message store
Distributed KahaDB message store
LevelDB message store – the best in terms of
scalability and performance
Journaled JDBC adapter
Non-journaled JDBC adapter
18

19. Message Broker
Message Cursors
Message Store and Message cursors in an Retail
Infrastructure
19
Vendor
Systems
System A
System B
Inventory
System
Message Store
App
Queue 2
Dispatch queue Pending Cursor
App
Queue
1
App
Queue 3
Queue impl
details

20. Message Broker
Message Cursors – Increase Scalability of
Message Stores
Store Based cursors for fast Consumers (Retail Scenario)
20
Dispatch
Queue
Pending
Cursor
Message Store
Inbound
Messages
Vendor
Systems
System A
System B

21. Store Based cursors for slow Consumers
Message Cursors – Increase Scalability of
Message Stores Contd…
21
Message Broker
Dispatch
Queue
Pending
Cursor
Message Store
Inbound
Messages
Vendor
Systems
System A
System B

22. VMCursors
Message Cursors – Increase Scalability of
Message Stores Contd…
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
<pendingQueuePolicy> <vmQueueCursor/> </pendingQueuePolicy>
</policyEntry>
22
Message Broker
Dispatch
Queue
Pending
Cursor
Message Store
Inbound
Messages
Vendor
Systems
System A
System B

23. File Based Cursors
•Does not scale well if the consumers are behind by a large margin
•Used for non persistent messages
Message Cursors – Increase Scalability of
Message Stores Contd…
23
Message Broker
Dispatch
Queue
Pending
Cursor
Message Store
Inbound
Messages
File Based Cursors
Temporary
Files
Vendor
Systems
System A
System B

24. Vertical and Horizontal
Scalability
24

25. Vertical Scalability
 Capacity of a single broker to support large numbers of connections from
consumers and producers.
 To Consider
 Thread Count
 CPU Usage
 Memory Usage
25

26. Vertical Scalability – Reducing Thread Count
NIO protocol
<transportConnectors>
<transportConnector name="nio" uri="nio://0.0.0.0:61616"/>
</<transportConnectors>
26

27. JBoss A-MQ – Vertical Scalability – Reducing
Thread Count
Disable Dedicated Task Runner
ACTIVEMQ_OPTS="-org.apache.activemq.UseDedicatedTaskRunner=false“
Use optimized dispatch for queues
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" optimizedDispatch="true">
...
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
27

28. Vertical Scalability – CPU
Enable tightEncodingEnabled
uri = "failover://(tcp://localhost:61616wireFormat.tightEncodingEnabled=false)";
Enabling compression
tcp://hostA:61617?jms.useCompression=true
28

29. Vertical Scalability – Memory
Allocate more memory to broker
• pass the –Xmx option to the JVM.
ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRunner=false“
Configure big enough memory usage
<systemUsage>
<memoryUsage>
<memoryUsage limit="1024 mb" />
</memoryUsage>
</systemUsage>
29

30. Horizontal Scalability – Network of Brokers
Concepts
Network Connector
Transport Connector
Store and Forward
30
Broker-1
Broker-2
Network
Connector
Producer
Transport
Connector
Consumer
Transport
Connector

31. Retail Example – Hub and Spoke Topology
31

32. Store and Forward-
Configuration
Store Store
Retail
Outlet
Ware
house
Fire
wall
<networkConnectors>
<networkConnector
uri="static:(tcp://warehouse:61617)"
userName="admin" password="admin“
duplex=“true”
conduitSubscriptions=“true”
decreaseNetworkConsumerPriorit
y=“false” />
</networkConnectors>

33. When there are a lot of producers
Brokers, A, B, and C, accept
connections from message producers.
Brokers, X and Y, accept connections
from message consumers.
Concentrator Topology - Producer Load
Balancing
33
A B C
YX
C1 C2 C3 C4

34. Some Other Topologies
34
 Tree Topology
Graph Topology
Mesh Topology

35. Retail Example-High
Availability
Shared
Store
Retail
Master
Broker
Retail
Slave
Broker
Wareh
ouse
Slave
Broker
Wareh
ouse
Master
Broker
Shared
Store
Network
Connection on
Failure
Network
Connection
<networkConnectors>
<networkConnector
uri="static:(tcp://retail-
master:61617, tcp://retail-
slave:61617)"
</networkConnectors>
<services>
<masterConnector
remoteURI=“tcp://retail-
slave:62001”
userName=“admin”
password=“admin”/>
</services>

36. Other Master Slave Topologies
Master
Broker
Slave
Broker
Store Store
Message
Shared Nothing Master Slave
Reliable
Only one slave possible
Downtime when master fails
Client
36

37. Shared Storage Master-Slave
Master
Broker with
file lock
Slave
Broker
Shared File
System or
SAN
Shared File system Master Slave
Recommended if SAN is
available
Easy to configure
No restriction on the slaves
Client
Slave
Broker
Slave polls for
lock
37

38. Hybrid Scaling
38
Client
Acc
Payable
Master
Acc
Payable
Slave
Accounts Payable
Information
Acc
Receivabl
e Master
Acc
Receivabl
e Slave
Accounts
Receivable
Information
 Client Complexity
 Less broker cross
talk

39. Apache Camel
A ride to solve your integration problems
39

40. Camel-What is it
Open source integration framework
Based on Enterprise Integration Patterns(EIPs)
Supports more than 120+ components
Supports DSL programming with Spring, Java, Scala and groovy
Flexible deployment
Can run standalone with maven
40
CBR Wire Tap TransformationEndpoint Translator

41. Camel-Architecture
41

42. Camel Coding Structure
import org.apache.camel.builder.RouteBuilder;
public class MyRouteBuilder extends RouteBuilder {
public void configure() {
from("file:SourceFolder?noop=true")
.to("file:TargetFolder");
}
}
<camelContext
xmlns="http://camel.apache.org/schema/spring">
<route id=“FileToFile">
<from uri="file:SourceFolder?noop=true""/>
<to uri="mock:inside"/>
</route>
</camelContext>
Java DSL
Spring DSL
42
Read a file from
SourceFolder and write it to
TargetFolder

43. 43
<route>
<from
uri="activemq:JUDCon?destination.consumer.exclusive=true&amp;destinati
on.consumer.prefetchSize=50"/>
<to uri="mock:2014"/>
</route>
Routes
Camel and Messaging
Configuring the Destination Options at the Endpoint in the
Routes

44. Camel and Messaging Contd…
<bean id="jmsConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://localhost:61616" />
</bean>
Beans
44
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig"/>
<property name="transacted" value="true"/>
<property name="cacheLevelName" value="CACHE_CONSUMER" />
</bean>
<bean id="pooledConnectionFactory"
class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
<property name= "maxConnections" value="8" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
<bean id="jmsConfig"
class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory"/>
<property name="concurrentConsumers" value="10“ />
</bean>

45. Camel – Scalability Techniques
45
Throttle
Load
Balancing
from("activemq:queueA").throttle(10).
to("activemq:queueB")
<from uri=" activemq:queueA " />
<throttle maximumRequestsPerPeriod="3“>
<to uri=" activemq:queueB " />
from(" activemq:queueA ").
loadBalance().roundRobin().
to(" activemq:queueB ",
" activemq:queueC",
" activemq:queueD");
<from uri=" activemq:queueA "/>
<loadBalance>
<roundRobin/>
<to uri=" activemq:queueB"/>
<to uri=" activemq:queueC "/>
<to uri=" activemq:queueD "/>
</loadBalance>

46. 46
Embedding Camel into Your Broker
• Reduces serialization cost
• Reduced networkWhy?
• Using Interceptors to your
brokerWhen?
• Add the camel route to
activemq.xmlHow?

47. Summary
• Messaging and Integration
• Significance of Asynchronous messaging
• How scalability is important to solve large flow of data
• Scalability – Vertical and Horizontal
• What's camel and Integration
47

48. Useful Links
References
• http://camel.apache.org/architecture.html
• http://camel.apache.org/jms.html
• http://activemq.apache.org/broker-camel-component.html
48

49. Thank You
Scale Your Enterprise ECO System
49
@bhavani_an @Naveenrajbala