Learn more about the evolution of inter-system communication from direct database access to message-queue based solutions.
It's not a Holy Grail or Silver Bullet but might prevent people making mistakes already made by others on their behalf.
The journey starts with a quick tour of things we used to do and progresses through time ... ending with an overview of an almost-in-production Net::ActiveMQ.
No Aardvark were harmed in the making of this presentation.
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
YAPC::EU 2010 - Going Postal
1. Motivation
Our Solution
Issues
Summary
Going Postal
Chisel Wright
NET-A-PORTER
YAPC::EU 2010
Chisel Wright Going Postal
2. Motivation
Our Solution
Talking To Strangers
Issues
Summary
Motivation
Why Bother?
Chisel Wright Going Postal
3. Motivation
Our Solution
Talking To Strangers
Issues
Summary
A Long Time Ago
When There Weren’t Any Better Solutions
Direct, far away database writes
Need to know their dialect
They can’t easily change their schema
Tightly coupled
Chisel Wright Going Postal
4. Motivation
Our Solution
Talking To Strangers
Issues
Summary
A Long Time Ago
When There Weren’t Any Better Solutions
Direct, far away database writes
Need to know their dialect
They can’t easily change their schema
Tightly coupled
Chisel Wright Going Postal
5. Motivation
Our Solution
Talking To Strangers
Issues
Summary
A Long Time Ago
When There Weren’t Any Better Solutions
Direct, far away database writes
Need to know their dialect
They can’t easily change their schema
Tightly coupled
Chisel Wright Going Postal
6. Motivation
Our Solution
Talking To Strangers
Issues
Summary
A Long Time Ago
When There Weren’t Any Better Solutions
Direct, far away database writes
Need to know their dialect
They can’t easily change their schema
Tightly coupled
Chisel Wright Going Postal
7. Motivation
Our Solution
Talking To Strangers
Issues
Summary
Not quite so long ago
When There Were Options
TheSchwartz
resolved a slightly different issue
first step in the right direction
didn’t help with far away issue
perl only
Chisel Wright Going Postal
8. Motivation
Our Solution
Talking To Strangers
Issues
Summary
Not quite so long ago
When There Were Options
TheSchwartz
resolved a slightly different issue
first step in the right direction
didn’t help with far away issue
perl only
Chisel Wright Going Postal
9. Motivation
Our Solution
Talking To Strangers
Issues
Summary
Not quite so long ago
When There Were Options
TheSchwartz
resolved a slightly different issue
first step in the right direction
didn’t help with far away issue
perl only
Chisel Wright Going Postal
10. Motivation
Our Solution
Talking To Strangers
Issues
Summary
Not quite so long ago
When There Were Options
TheSchwartz
resolved a slightly different issue
first step in the right direction
didn’t help with far away issue
perl only
Chisel Wright Going Postal
11. Motivation
Our Solution
Talking To Strangers
Issues
Summary
Not quite so long ago
When There Were Options
TheSchwartz
resolved a slightly different issue
first step in the right direction
didn’t help with far away issue
perl only
Chisel Wright Going Postal
14. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
ActiveMQ
Java guys like it
Perl guys can interact with it
Reliable, well-used solution
Chisel Wright Going Postal
15. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
ActiveMQ
Java guys like it
Perl guys can interact with it
Reliable, well-used solution
Chisel Wright Going Postal
16. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
ActiveMQ
Java guys like it
Perl guys can interact with it
Reliable, well-used solution
Chisel Wright Going Postal
17. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Net::Stomp
STOMP is great for talking to ActiveMQ
Net::Stomp excellent for quickly interacting with ActiveMQ
Chisel Wright Going Postal
18. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Net::Stomp
STOMP is great for talking to ActiveMQ
Net::Stomp excellent for quickly interacting with ActiveMQ
Chisel Wright Going Postal
19. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::Stomp
Talking to a Queue
# send a message to the queue ’foo’
use Net::Stomp;
my $stomp = Net::Stomp->new({
hostname => ’localhost’,
port => ’61613’,
});
$stomp->connect({
login => ’hello’,
passcode => ’there’,
});
$stomp->send({
destination => ’/queue/foo’,
body => ’test message’,
});
$stomp->disconnect;
Chisel Wright Going Postal
20. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Receiving with Net::Stomp
Subscribing to a Queue
# subscribe to messages from the queue ’foo’
use Net::Stomp;
my $stomp = Net::Stomp->new({
hostname => ’localhost’,
port => ’61613’,
});
$stomp->connect({
login => ’hello’,
passcode => ’there’,
});
$stomp->subscribe({
destination => ’/queue/foo’,
’ack’ => ’client’,
’activemq.prefetchSize’ => 1,
});
Chisel Wright Going Postal
21. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Receiving with Net::Stomp
Plucking from the Queue
while (1) {
my $frame = $stomp->receive_frame;
warn $frame->body; # do something here
$stomp->ack( { frame => $frame } );
}
$stomp->disconnect;
Chisel Wright Going Postal
22. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Net::Stomp
Thoughts
Quickly and easily talk to ActiveMQ
Does what it promises
Low-level
Copy-and-paste coding
Error-handling, . . .
Chisel Wright Going Postal
23. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Net::Stomp
Thoughts
Quickly and easily talk to ActiveMQ
Does what it promises
Low-level
Copy-and-paste coding
Error-handling, . . .
Chisel Wright Going Postal
24. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Net::Stomp
Thoughts
Quickly and easily talk to ActiveMQ
Does what it promises
Low-level
Copy-and-paste coding
Error-handling, . . .
Chisel Wright Going Postal
25. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Net::Stomp
Thoughts
Quickly and easily talk to ActiveMQ
Does what it promises
Low-level
Copy-and-paste coding
Error-handling, . . .
Chisel Wright Going Postal
26. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Net::Stomp
Thoughts
Quickly and easily talk to ActiveMQ
Does what it promises
Low-level
Copy-and-paste coding
Error-handling, . . .
Chisel Wright Going Postal
28. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Net::ActiveMQ
Standardise our AMQ solution across apps
Lower barrier to entry
Chisel Wright Going Postal
29. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Net::ActiveMQ
Standardise our AMQ solution across apps
Lower barrier to entry
Chisel Wright Going Postal
30. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
What Is It?
Essentially:
Net::Stomp
Catalyst::Engine::Stomp
with a lovely ribbon and bow around it.
Message producers
Message consumers
all in one place.
Chisel Wright Going Postal
31. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Talking to a Queue
use Net::ActiveMQ::Producer;
my $producer = Net::ActiveMQ::Producer->new({
hostname => ’localhost’,
port => 61613
});
$producer->send(
’Some::Message’,
{ message => ’data’, goes => ’here’ }
);
Net::ActiveMQ::Producer - message type does not exist -
Some::Message
at /path/to/.../Class/MOP/Method/Wrapped.pm line 159
Chisel Wright Going Postal
32. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
A Producer
package Net::ActiveMQ::Producer::Some::Message;
use Moose;
with ’Net::ActiveMQ::Role::Producer’;
sub transform {
my ($self, $header, $data) = @_;
# make sure it goes somewhere
$header->{destination} ||= ’/queue/some-message’;
# the desired action from the consumer
$data->{’@type’} ||= ’action_method’;
# "transform" the data
$data->{process_time} = scalar localtime;
return ($header, $data);
}
1;
Chisel Wright Going Postal
33. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Validate What You Send
# This is completely optional!
sub message_spec {
# Data::Rx format
return {
type => ’//rec’,
required => {
message => ’//str’,
},
optional => {
’@type’ => ’//str’,
goes => ’//str’,
process_time => ’//str’,
},
};
}
Chisel Wright Going Postal
34. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Catalyst Model
# create your model class
./script/myapp_create.pl model
MyMQ
Net::ActiveMQ
localhost
61613
YES
# in your controller
$c->model(’MyMQ’)->send(
’Some::Message’,
{ message => { a => ’shiny’, hash => ’reference’ } }
);
Chisel Wright Going Postal
35. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Why did you do that?
sub message_spec
Used by proof-of-concept
Can catch your own mistakes
No compelling reason to remove it
Data::Rx
Nicer error messages
Slightly more agnostic
Chisel Wright Going Postal
36. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Why did you do that?
sub message_spec
Used by proof-of-concept
Can catch your own mistakes
No compelling reason to remove it
Data::Rx
Nicer error messages
Slightly more agnostic
Chisel Wright Going Postal
37. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Why did you do that?
sub message_spec
Used by proof-of-concept
Can catch your own mistakes
No compelling reason to remove it
Data::Rx
Nicer error messages
Slightly more agnostic
Chisel Wright Going Postal
38. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Why did you do that?
sub message_spec
Used by proof-of-concept
Can catch your own mistakes
No compelling reason to remove it
Data::Rx
Nicer error messages
Slightly more agnostic
Chisel Wright Going Postal
39. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Why did you do that?
sub message_spec
Used by proof-of-concept
Can catch your own mistakes
No compelling reason to remove it
Data::Rx
Nicer error messages
Slightly more agnostic
Chisel Wright Going Postal
40. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Why did you do that?
sub message_spec
Used by proof-of-concept
Can catch your own mistakes
No compelling reason to remove it
Data::Rx
Nicer error messages
Slightly more agnostic
Chisel Wright Going Postal
41. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Sending with Net::ActiveMQ
Why did you do that?
sub message_spec
Used by proof-of-concept
Can catch your own mistakes
No compelling reason to remove it
Data::Rx
Nicer error messages
Slightly more agnostic
Chisel Wright Going Postal
42. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Receiving with Net::ActiveMQ
Quick But Boring
$ CATALYST_DEBUG=1
> net_activemq_consumer_server.pl
[debug] Loaded engine "Catalyst::Engine::Stomp"
...
[debug] Loaded components:
.--------------------------------+----------.
| Class | Type |
+--------------------------------+----------+
| ...::Controller::Root | instance |
’--------------------------------+----------’
...
[info] Application powered by Catalyst 5.80016
Not very useful in this state
Fairly easy to add something useful
Chisel Wright Going Postal
43. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Receiving with Net::ActiveMQ
Quick But Boring
$ CATALYST_DEBUG=1
> net_activemq_consumer_server.pl
[debug] Loaded engine "Catalyst::Engine::Stomp"
...
[debug] Loaded components:
.--------------------------------+----------.
| Class | Type |
+--------------------------------+----------+
| ...::Controller::Root | instance |
’--------------------------------+----------’
...
[info] Application powered by Catalyst 5.80016
Not very useful in this state
Fairly easy to add something useful
Chisel Wright Going Postal
44. Motivation
ActiveMQ
Our Solution
Net::Stomp
Issues
Net::ActiveMQ
Summary
Receiving with Net::ActiveMQ
Consuming Some::Message
$ net_activemq_consumer_create.pl controller Some::Message
ActiveMQ
created "lib/Net/ActiveMQ/Consumer/Controller/Some"
created "t"
created "lib/Net/ActiveMQ/Consumer/Controller/Some/Message.pm"
created "t/controller_Some-Message.t"
Catalyst::Helper to do the hard work
Chisel Wright Going Postal
49. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Peer Review
Only recently been used by the rest of the team
Bugs
Handling errors
Lack of clear documentation
POD
How-To / “How do I. . . ?”
Chisel Wright Going Postal
50. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Peer Review
Only recently been used by the rest of the team
Bugs
Handling errors
Lack of clear documentation
POD
How-To / “How do I. . . ?”
Chisel Wright Going Postal
51. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Peer Review
Only recently been used by the rest of the team
Bugs
Handling errors
Lack of clear documentation
POD
How-To / “How do I. . . ?”
Chisel Wright Going Postal
52. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Peer Review
Only recently been used by the rest of the team
Bugs
Handling errors
Lack of clear documentation
POD
How-To / “How do I. . . ?”
Chisel Wright Going Postal
53. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Peer Review
Only recently been used by the rest of the team
Bugs
Handling errors
Lack of clear documentation
POD
How-To / “How do I. . . ?”
Chisel Wright Going Postal
54. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Peer Review
Only recently been used by the rest of the team
Bugs
Handling errors
Lack of clear documentation
POD
How-To / “How do I. . . ?”
Chisel Wright Going Postal
55. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Implementation
General
Some early features not fully phased out
@type => ’...’
JMSType
Chisel Wright Going Postal
56. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Implementation
General
Some early features not fully phased out
@type => ’...’
JMSType
Chisel Wright Going Postal
57. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Implementation
General
Some early features not fully phased out
@type => ’...’
JMSType
Chisel Wright Going Postal
58. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Implementation
Producers
Shared spec files implementation not complete
Still living in the ::Consumer:: namespace
Not easily used by ::Producer:: classes
message_spec vs build_message_spec
Chisel Wright Going Postal
59. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Implementation
Producers
Shared spec files implementation not complete
Still living in the ::Consumer:: namespace
Not easily used by ::Producer:: classes
message_spec vs build_message_spec
Chisel Wright Going Postal
60. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Implementation
Producers
Shared spec files implementation not complete
Still living in the ::Consumer:: namespace
Not easily used by ::Producer:: classes
message_spec vs build_message_spec
Chisel Wright Going Postal
61. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Implementation
Producers
Shared spec files implementation not complete
Still living in the ::Consumer:: namespace
Not easily used by ::Producer:: classes
message_spec vs build_message_spec
Chisel Wright Going Postal
62. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Implementation
Consumers
Consumers limited to Net::ActiveMQ namespace
Error-Handling still evolving
Chisel Wright Going Postal
63. Motivation
Our Solution Peer Review
Issues Implementation
Summary
Implementation
Consumers
Consumers limited to Net::ActiveMQ namespace
Error-Handling still evolving
Chisel Wright Going Postal
64. Motivation
Our Solution
Issues
Summary
Summary
Avoid direct database interaction
Use message queues
DRY
The Future
Evolution - final stages
We’re starting to use it in production
We hope to release to the CPAN soon
Chisel Wright Going Postal
65. Motivation
Our Solution
Issues
Summary
Summary
Avoid direct database interaction
Use message queues
DRY
The Future
Evolution - final stages
We’re starting to use it in production
We hope to release to the CPAN soon
Chisel Wright Going Postal
66. Motivation
Our Solution
Issues
Summary
Summary
Avoid direct database interaction
Use message queues
DRY
The Future
Evolution - final stages
We’re starting to use it in production
We hope to release to the CPAN soon
Chisel Wright Going Postal
67. Motivation
Our Solution
Issues
Summary
Summary
Avoid direct database interaction
Use message queues
DRY
The Future
Evolution - final stages
We’re starting to use it in production
We hope to release to the CPAN soon
Chisel Wright Going Postal
68. Motivation
Our Solution
Issues
Summary
Summary
Avoid direct database interaction
Use message queues
DRY
The Future
Evolution - final stages
We’re starting to use it in production
We hope to release to the CPAN soon
Chisel Wright Going Postal
69. Motivation
Our Solution
Issues
Summary
Summary
Avoid direct database interaction
Use message queues
DRY
The Future
Evolution - final stages
We’re starting to use it in production
We hope to release to the CPAN soon
Chisel Wright Going Postal
70. Motivation
Our Solution
Issues
Summary
Summary
Avoid direct database interaction
Use message queues
DRY
The Future
Evolution - final stages
We’re starting to use it in production
We hope to release to the CPAN soon
Chisel Wright Going Postal