Mais conteúdo relacionado Semelhante a Integrating Postgres with ActiveMQ and Camel (20) Integrating Postgres with ActiveMQ and Camel2. 2© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Justin Reock
Chief Architect
Rogue Wave Software
Justin has over 20 years’ experience
working in various software roles and is
an outspoken free software
evangelist, delivering enterprise
solutions and community education on
databases, integration work,
architecture, and technical leadership.
He is currently the Chief Architect at
Rogue Wave Software.
4. 4© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• I’m going to show you three slides
• They all have something in common….
• See if you can figure out what it is!
7. 7© 2019 Rogue Wave Software, Inc. All Rights Reserved.
BillingFulfillment
CRME-Commerce
8. 8© 2019 Rogue Wave Software, Inc. All Rights Reserved.
None of these things exist in real life!
10. 10© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Technically, MOM is any platform that sends and receives
messages between distributed applications
• This allows for asynchronous processing, and normalization of
data exchanges
• Clients connect to a messaging provider, and send and receive
messages via that provider
11. 11© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Data normalization makes it a lot easier for disparate applications to
trade information
• Just like in web services, heterogeneous systems can agree on a data
“contract” and send and receive data in that format
• All the application needs is a client compatible with the messaging
provider
• And since messages are allowed to queue up on the message provider,
it also makes it very easy to perform asynchronous processing
• Most message providers offer additional benefits, like guaranteed
delivery, options for traffic shaping, high availability, and scaling
13. 13© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• ActiveMQ is a Java-based multi-purpose messaging middleware solution,
called a message broker, designed for high throughput and scale
• It is useful in a number of situations, including aggregating large amounts of
data in an efficient and safe manner, federating disparate or heterogeneous
applications across an enterprise, and/or moving data between
geographically separated data centers or applications
• ActiveMQ supports a number of connection protocols, allowing it to provide
connection points between applications coded in different languages, such as
Java, C++, .NET, PHP, etc
• Customers commonly migrate to ActiveMQ from commercial middleware
such as WebSphereMQ, Oracle AQ, TIBCO EMS, and/or MSMQ
14. 14© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• ActiveMQ is a mature and well-adopted platform, with thousands of companies
using it, and backed by a strong community
• It is by far the most flexible OSS message broker available, offering integration
points for dozens of application frameworks and languages
• ActiveMQ ships with Apache Camel, one of the most revolutionary integration
frameworks available, and shares developers and committers with the Apache
Camel product
• ActiveMQ can be tuned for massive scale, supporting large data sets, such as
those generated by the CERN large hadron collider
• Its configuration can support numerous traffic and implementation models
• ActiveMQ has a powerful statistics and metrics engine, exposing crucial details
about the health of the broker and the amount and size of message traffic and
client status
15. 15© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• ActiveMQ acts as integration middleware, sitting between
various enterprise applications, data sources, and APIs, and
“federating” data between the systems
16. 16© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• ActiveMQ can also be used as a reliable message buffer
• Applications can send messages into ActiveMQ and continue
working, knowing that ActiveMQ will guarantee delivery of
the message
• Or, ActiveMQ can be used to broadcast messages to several
different systems
• ActiveMQ supports transactions between the broker and
the client, so that exceptions can be caught and managed
• Using these constructs, you can easily create asynchronous
message patterns by sending or receiving messages from
ActiveMQ
17. 17© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Message grouping
Scheduled messages
Security framework
Metrics and
statistics
Network of Brokers
Inactivity monitoring
Transactions
Fault tolerance
Traffic Shaping
Plugin stack
Virtual destinations
Temporary destinations
Transformation
Load balancing
Client failover
Property encryption
Destination policies
Dispatch policies
Producer flow control
Message swapping
And of course… Camel!
19. 19© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• So, data routing and transformation in the Enterprise lends itself to a
lot of what you would call “boilerplate” code
• Boilerplate text, in publishing, is text that can be used and reused in
many places
• Boilerplate code is the same concept
• How many times have you written code to open up a web server,
push a file to an ftp site or drop a message into a message broker?
• Have you ever felt that you’re just writing different versions of the
same thing over and over again?
• You’re not alone
• Enter the concept of an Enterprise Integration Pattern…
20. 20© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Camel is an open source MOM language developed by Apache and based
on Enterprise Integration Patterns
• Book by Gregor Hohpe and Bobby Woolf
• Centers around 60+ common patterns found in
Enterprise integration projects
• Defines these common patterns in a vendor
neutral way
• Camel aims to provide a language which
implements these patterns, in a style modeled
after the UNIX pipeline
21. 21© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Content Based Router
– Where a message is routed to disparate endpoints
based on message criteria
22. 22© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Content Enricher
– In which a basic message enters the pattern, takes data from
another source, and then adds it to the message
23. 23© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Recipient List
– A message is broadcast to n channels based on message criteria. When this message is
broadcast to all channels regardless, it becomes a Multicast pattern
24. 24© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Scatter-Gather
– In which messages are broadcast out to disparate endpoints and results are
aggregated into a single message
25. 25© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Camel is maintained by the Apache Camel community
• It is a top-level project hosted at http://camel.apache.org
• The project’s chief maintainer is Claus Ibsen, author of Camel in Action
• As of August of 2017, the project has had almost 400 distinct committers, spanning over
30,000 commits
• It is written primarily in Java, and also contains some Scala code
• The project is licensed under the ALv2 (Apache License v2.0), which is considered a permissive
license
• So, Camel has been folded in to many other projects, including the Mule ESB and the
Switchyard project
• Camel can be run standalone, packaged into a WAR, or deployed as an OSGI bundle
• A POJO syntax as well as a Spring DSL XML syntax are made available for developers
27. 27© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• ActiveMQ needs to be able to persist messages in
something called a persistence journal
• This way, if the broker crashes, it can replay unsent
messages when it comes back up
• And Camel is just plain hungry for endpoints, and
databases are no exception!
Postgres works great with ActiveMQ and Camel!
28. 28© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• ActiveMQ needs a
persistence journal to
manage High Availability
• The default is a NoSQL
database called KahaDB
• The default is fast, but
difficult to replicate leading
to a SPOF architecture
29. 29© 2019 Rogue Wave Software, Inc. All Rights Reserved.
When should I use…
Postgres/JDBC? KahaDB?
• I need true no-SPOF HA
• Throughput is less important
• I need to easily introspect
messages in the persistence
store
• I want familiar backup and
replication
• I don’t need true no-SPOF
HA
• Throughput is the most
important aspect
• I can lose messages or don’t
need to introspect them
• I can rely on storage backup
30. 30© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Quick reality check…
KahaDB
Postgres w/
Pooling
Postgres w/o
Pooling
31. 31© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• You can initialize Postgres as an ActiveMQ Persistence
Journal in a few simple steps:
– Configure Postgres with an ActiveMQ User and
Database
– Copy Postgres JDBC driver into ActiveMQ Classpath
– Configure ActiveMQ Persistence Adapter and Pooling
– Disable automatic schema creation going forward
32. 32© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Several components are used to connect…
ActiveMQ Code
Third-party
or
PGPooling
Postgres Code
33. 33© 2019 Rogue Wave Software, Inc. All Rights Reserved.
From a fresh install…
34. 34© 2019 Rogue Wave Software, Inc. All Rights Reserved.
pg_hba.conf
Adjust your trust settings accordingly for your environment…
35. 35© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• ActiveMQ’s default classpath is in the /lib directory of
the installation root
36. 36© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Create a Bean reference in /conf/activemq.xml
<bean id="postgres-ds"
class="org.postgresql.ds.PGPoolingDataSource"
destroy-method="close">
<property name="serverName" value="localhost"/>
<property name="databaseName" value="activemq"/>
<property name="portNumber" value="0"/>
<property name="user" value="activemq"/>
<property name="password" value="activemq"/>
<property name="dataSourceName" value="postgres"/>
<property name="initialConnections" value="1"/>
<property name="maxConnections" value="10"/>
</bean>
37. 37© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Set the <persistenceAdapter> to JDBC and reference the bean
<persistenceAdapter>
<jdbcPersistenceAdapter dataSource="#postgres-ds"
createTablesOnStartup=”true” />
</persistenceAdapter>
Remember that createTablesOnStartup setting…..
38. 38© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Fire up the broker and check the persistence…
Looks good, lets check out the schema …..
39. 39© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Lets see all tables…
40. 40© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Our ACK Table…
41. 41© 2019 Rogue Wave Software, Inc. All Rights Reserved.
Our Lock table…
42. 42© 2019 Rogue Wave Software, Inc. All Rights Reserved.
And our Messages table…
43. 43© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• To avoid an ugly (but benign) warning condition…
Turn off create tables on startup….
44. 44© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Demonstrate Postgres as a persistence store for
ActiveMQ
• Persist some messages into the broker
• Introspect the messages
• Crash the broker
• Restart the broker and watch as messages replay
• Bonus: Demonstrate the HA failover functionality
I thought he’d never get there….
46. 46© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Recall that Camel is a normalized routing engine meant to shuttle data
around
• Camel takes a component driven approach where components are
published for various endpoints
• One of those endpoints is the JDBC endpoint, meant to interact with JDBC-
compliant database drivers
• So, Camel can either read data from a database to enrich a message, or
produce data into a database from an external process
• Camel’s marshalling and transformation capabilities make it ideal for
building API abstraction in front of a Postgres database
Camel moves data around an enterprise…
47. 47© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Camel-JDBC – Most basic integration, uses explicit SQL to
execute queries against a JDBC-compliant driver
• Camel-JPA – For using Object Relational Mapping
solutions like OpenJPA and Hibernate
• Camel-SQL – Variant of camel-jdbc which uses Spring IoC
to provide query details
• As well as NoSQL-specific components like MongoDB,
Cassandra, and Hadoop
Camel provides a lot of options…
48. 48© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Camel uses a repeatable, normalized concatenation of
“Processor” and ”Message” objects in a group called an
Exchange
• There is an ”In” message, a “Processor”, and an “Out”
message
Processor
In Out
49. 49© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• And, you can chain these Exchanges together – just like piping
commands through UNIX – and form a Camel Route
• The “Out” message of a previous Exchange becomes the “In”
message of a new Exchange:
P
In Out
P
In Out
P
In Out
50. 50© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• Now here’s the magical part – a Processor can be anything you want
• Components in Camel work by creating Processors that perform various business
functions
• The components can be chained together to build Integration Patterns
• For instance, you can have an “FTP” processor that takes the In message and initiates an
FTP session
• Or you can have an “SMTP” processor that takes the In message and sends an email
with it
• And as we’ll see in a few minutes, there are several components designed specifically for
interacting with JDBC-compliant database drivers
• This is in addition to Camel’s many built-in patterns, such as LoadBalance, Multicast,
DataSets, etc.
• And there are a lot of components: http://camel.apache.org/components.html
51. 51© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• We will develop a simple RESTful API for writing and
reading to a table in Postgres
• GET requests to our API will output data in JSON
• POST requests will write the post body to the database
• We will do this using very little code, demonstrating
Camel’s propensity for rapid development of integration
logic
53. 53© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• To initialize our datasource, we will use a bean very similar to the one that we
used for ActiveMQ, since Camel is also powered by Spring:
<bean id="postgres-ds"
class="org.postgresql.ds.PGPoolingDataSource"
destroy-method="close">
<property name="serverName" value="localhost"/>
<property name="databaseName" value="cameljdbc"/>
<property name="portNumber" value=”5432"/>
<property name="user" value=”camel"/>
<property name="password" value="camel"/>
<property name="dataSourceName" value="postgres"/>
<property name="initialConnections" value="1"/>
<property name="maxConnections" value="10"/>
</bean>
54. <camelContext xmlns="http://camel.apache.org/schema/blueprint" id="JDBCDemos">
<dataFormats>
<json id="jack" library="Jackson"/>
</dataFormats>
<route>
<description>PostgresConf 2019 :: JDBC Marshalling</description>
<from uri="jetty:http://0.0.0.0:8080/jdbcWeb"/>
<when>
<simple>${headers.CamelHttpMethod} == "GET"</simple>
<setBody>
<constant>SELECT * FROM post_data</constant>
</setBody>
<to uri="jdbc:postgres-ds"/>
<marshal ref="jack"/>
</when>
<when>
<simple>${headers.CamelHttpMethod} == "POST"</simple>
<setBody>
<simple>
INSERT INTO post_data VALUES ('${date:now:yyyy-MM-dd'T'HH:mm:ss.SSSZ}','${body}’)
</simple>
</setBody>
<to uri="jdbc:postgres-ds"/>
</when>
</route>
</camelContext>
Yep, that’s all the code we need….
55. 55© 2019 Rogue Wave Software, Inc. All Rights Reserved.
• This one is a lot easier to just demonstrate
• We will write the code together, and I will deploy out to
ServiceMix
• Camel can be deployed in lots of different ways, but
ServiceMix is easy for prototyping
• Bonus: Add a parameter to allow searching for specific
data
Let’s watch it work!
56. 56© 2019 Rogue Wave Software, Inc. All Rights Reserved.
LinkedIn – Only Justin Reock in the world apparently!
Twitter – @jreock - But I do get a little political on
there….
Blog - http://blog.klocwork.com/author/justin-reock/
Email – justin.reock@roguewave.com
Feel Free to Reach Out – I Get Lonely…
Notas do Editor Establish right to play here by explaining background Honeywell