Choosing the right way to process data might become a strategic and non trivial decision for many kind of applications. Especially in applications where an high percentage of the time is spent elaborating information behind the scenes. There are different message queuing systems designed to manage and process data asynchronously. Using simple messages, it becomes possible to define many types of queue patterns, from the straightforward solution to the more complex one like routing, publisher/subscriber and topic. The purpose of the talk is to show how to approach the different scenarios in php, adding value to your application.

1. A S Y N C H R O N O U S
D ATA
P R O C E S S I N G
A N D R E A G I U L I A N O
@bit_shark
# C A K E F E S T

2. T H E P R O B L E M

3. P R O D U C T C ATA L O G G E N E R AT O R
• 30k+ records in db representing products
• dispose records in pages (with a certain order)
• generate a 300+ pages PDF catalog
T H E P R O B L E M

4. H E AV Y
O P E R AT I O N S
T O D O H E R E
T H E P R O B L E M

5. U S E R S D O N ’ T WA N T T O WA I T
T H E P R O B L E M

6. U S E R S D O N ’ T WA N T T O WA I T
T H E P R O B L E M
A N D I F T H E Y H AV E T O
T H E Y D O N ’ T WA N T T O
G E T S T U C K !

7. T H E P R O B L E M
S Y N C H R O N Y
time
process A
process B
Request
Response
blocked

8. T H E N E E D S

9. P R E V E N T T I M E O U T
T H E N E E D S

10. E V E N T U A L LY D E L I V E RY
T H E N E E D S
“sooner or later your job will be processed.”

11. “ E V E R I T H I N G ’ S G O N N A B E A L R I G H T ”
N O T I F I C AT I O N
T H E N E E D S

12. T H E N E E D S
A S Y N C H R O N Y
time
process A
process B
Request

13. I N T E R O P E R A B I L I T Y
T H E N E E D S

14. A D VA N C E D M E S S A G E
Q U E U I N G P R O T O C O L
T H E S O L U T I O N

15. H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce Consumes
B R O K E R

16. H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce
Exchange
Consumes
B R O K E R

17. H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce
Exchange
Routes
Consumes
B R O K E R

18. H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce
Exchange Queue
Routes
Consumes
B R O K E R

19. T H R O U G H C O M P O S E R
I N S TA L L A M Q P L I B R A RY
{
"require":
{
"videlalvaro/php-­‐amqplib":
"@stable",
...
}
}
$ composer.phar install

20. D I F F E R E N T S C E N A R I O S

21. P R O D U C E R / C O N S U M E R
S C E N A R I O

22. S C E N A R I O
P R O D U C E R / C O N S U M E R
P R O D U C E R C O N S U M E R
Q U E U E

23. S C E N A R I O
P R O D U C E R / C O N S U M E R
use
PhpAmqpLibConnectionAMQPConnection;
use
PhpAmqpLibMessageAMQPMessage;
$connection
=
new
AMQPConnection(HOST,
PORT,
USER,
PASSWORD);
$channel
=
$connection-­‐>channel();
S E T T I N G U P
C O N N E C T I O N
C R E AT E A C H A N N E L

24. S C E N A R I O
P R O D U C E R / C O N S U M E R
P R O D U C E R
Q U E U E
passive
durable
exclusive
auto-­‐delete
$channel-­‐>queue_declare('catalog',
false,
false,
false,
false);
foreach
($catalog-­‐>getPages()
as
$page)
{
$message
=
new
AMQPMessage($page);
$channel-­‐>basic_publish($message,
'',
'catalog');
}
$channel-­‐>close();
$connection-­‐>close();

25. S C E N A R I O
P R O D U C E R / C O N S U M E R
C O N S U M E R
Q U E U E
$connection
=
new
AMQPConnection(HOST,
PORT,
USER,
PASSWORD);
$channel
=
$connection-­‐>channel();
$channel-­‐>queue_declare('catalog',
false,
false,
false,
false);
$callback
=
function($msg)
{
$msg-­‐>body-­‐>generatePdf();
};
$channel-­‐>basic_consume('catalog',
'',
false,
true,
false,
false,
$callback);
while(count($channel-­‐>callbacks))
{
$channel-­‐>wait();
}
$channel-­‐>close();
$connection-­‐>close();

26. S C E N A R I O
M U LT I P L E C O N S U M E R S
P R O D U C E R
C O N S U M E R
Q U E U E
C O N S U M E R
…

27. S C E N A R I O
M U LT I P L E C O N S U M E R S
P R O D U C E R
C O N S U M E R
Q U E U E
C O N S U M E R
…
parallelize work
easy scalability

28. C O N S U M E R S C A N D I E

29. M E S S A G E
A K N O W L E D G E M E N T

30. S A F E T Y
M E S S A G E A C K S
$callback
=
function($msg){
$msg-­‐>body-­‐>generatePdf();
$msg-­‐>delivery_info['channel']-­‐>basic_ack($msg-­‐>delivery_info['delivery_tag']);
};
!
$channel-­‐>basic_consume('catalog',
'',
false,
false,
false,
false,
$callback);
S W I T C H A C K O N

31. D O N ’ T F O R G E T
T O S E N D A C K S

32. T H E B R O K E R
C A N D I E

33. D U R A B I L I T Y

34. D U R A B I L I T Y
M A R K T H E Q U E U E A N D T H E C H A N N E L
$channel-­‐>queue_declare('catalog',
false,
true,
false,
false);
In order to achieve durability
$message
=
new
AMQPMessage($data,
#the
message
is
now
persistent
array('delivery_mode'
=>
2)
);
the queue must be declared durable
the message must be marked as persistent

35. Q O S P O L I T I C S

36. Q O S P O L I T I C A
B E FA I R
C O N S U M E R
C O N S U M E R
for certain instances of the Round Robin dispatching
B R O K E R
#1
#3
#5
#2
#4
#6

37. Q O S P O L I T I C S
B E FA I R
$channel-­‐>basic_qos(null,
1,
null);
C O N S U M E R
C O N S U M E R
the broker sends messages only when it receives acks
B R O K E R
#1
#3
#3 #2#4

38. P U B L I S H / S U B S C R I B E
S C E N A R I O

39. S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
FA N O U T E X C H A N G E
$connection
=
new
AMQPConnection(HOST,
PORT,
USER,
PASSWORD);
$channel
=
$connection-­‐>channel();
$channel-­‐>exchange_declare('chatRoom',
'fanout',
false,
false,
false);
Exchange
…
Setting up the connection and declare the fanout exchange

40. S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
Exchange
…
$data
=
getAMessageToSendInTheRoom();
$msg
=
new
AMQPMessage($data);
$channel-­‐>basic_publish($msg,
'chatRoom');
$channel-­‐>close();
$connection-­‐>close();
P R O D U C E R
Produce
Publishing a message to the exchange

41. N O W I T ’ S T H E
S U B S C R I B E R ’ S T U R N

42. S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
Q U E U E B I N D I N G
Exchange
…
//connection
setted
up
!
list($queue_name,
,)
=
$channel-­‐>queue_declare("",
false,
false,
true,
false);
$channel-­‐>queue_bind($queue_name,
'chatRoom');
Bind the Queue on the Exchange
amq.gen-A7d
bind
bind
amq.gen-Sb4

43. S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
Exchange
…
amq.gen-A7d
$channel-­‐>basic_consume($queue_name,
'',
false,
true,
false,
false,
'readMessage');
$channel-­‐>close();
$connection-­‐>close();
C O N S U M E R
Consumes
amq.gen-Sb4

44. R O U T I N G
S C E N A R I O

45. S C E N A R I O
R O U T I N G
chatRoom
Exchange
type=direct
…
amq.gen-A7d
amq.gen-Sb4
Consumer1
ConsumerN
P R O D U C E R
Produce
routing_keys = friends
routing_keys = friends, colleagues
$channel-­‐>exchange_declare('chatRoom',
'direct',
false,
false,
false);

46. S C E N A R I O
R O U T I N G
chatRoom
Exchange
type=direct
P R O D U C E R
Produce
Producing messages
//connection
setted
up
!
$channel-­‐>exchange_declare('chatRoom',
'direct',
false,
false,
false);
$data
=
getAMessageToSendInTheRoom('friends');
$msg
=
new
AMQPMessage($data);
$channel-­‐>basic_publish($msg,
'chatRoom',
'friends');
$channel-­‐>close();
$connection-­‐>close();

47. S C E N A R I O
R O U T I N G
//connection
setted
up
//exchange
setted
up
//$queue_name
is
a
system
generated
queue
name
!
$rooms
=
array('friends',
'colleagues');
foreach($rooms
as
$room)
{
$channel-­‐>queue_bind($queue_name,
'chatRoom',
$room);
}
$channel-­‐>basic_consume($queue_name,
'',
false,
true,
false,
false,
'readMessage');
$channel-­‐>close();
$connection-­‐>close();
Bind a Consumer on different routing keys and consuming messages
chatRoom
amq.gen-Sb4
Consumer
routing_keys = friends, colleagues

48. T O P I C
S C E N A R I O

49. G O F U R T H E R
T O P I C
$channel-­‐>exchange_declare('vehicle',
'topic',
false,
false,
false);
vehicle
Exchange
type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<type>.<vehicle>.<colour>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y

50. A Y E L L O W S P O R T C A R

51. T O P I C
A Y E L L O W S P O R T C A R
vehicle
Exchange
type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<sport>.<car>.<yellow>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y

52. T O P I C
A Y E L L O W S P O R T C A R
vehicle
Exchange
type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<sport>.<car>.<yellow>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y

53. A R E D R A C E M O T O R B I K E

54. T O P I C
A R E D R A C E M O T O R B I K E
vehicle
Exchange
type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<motorbike>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y

55. T O P I C
A R E D R A C E M O T O R B I K E
vehicle
Exchange
type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<motorbike>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y

56. A R E D R A C E C A R

57. T O P I C
A R E D R A C E C A R
vehicle
Exchange
type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<car>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y

58. T O P I C
A R E D R A C E C A R
vehicle
Exchange
type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<car>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y

59. A B L U E C I T Y VA N

60. T O P I C
A B L U E C I T Y VA N
vehicle
Exchange
type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<city>.<van>.<blue>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y

61. D I S C A R D E D

63. R E M E M B E R

64. P R O C E S S I N G Y O U R D ATA
A S Y N C H R O N O U S LY C A N B E D O N E E A S I LY

65. E V E N I N P H P

66. C A N M A K E Y O U R
( D E V ) L I F E H A P P I E R

67. Y O U R A P P L I C AT I O N
S T R O N G E R

68. Andrea Giuliano
@bit_shark
andreagiuliano.it

69. joind.in/11608
Please rate the talk!

70. C R E D I T S
• https://www.flickr.com/photos/rayofsun/9401226342
• https://www.flickr.com/photos/thereeljames/11376085194
• https://www.flickr.com/photos/ollily/359817111
• https://www.flickr.com/photos/legofenris/4004170937
• https://www.flickr.com/photos/bsabarnowl/10993445723
• https://www.flickr.com/photos/jpott/2984914512
• https://www.flickr.com/photos/kalexanderson/6231391820
• https://www.flickr.com/photos/a2gemma/1448178195
• https://www.flickr.com/photos/dubpics/5619966355
• https://www.flickr.com/photos/umbertofistarol/5747425870
• https://www.flickr.com/photos/streetcarl/6888965017
• https://www.flickr.com/photos/infomastern/12407730413
• https://www.flickr.com/photos/giuseppemilo/11817936944
• https://www.flickr.com/photos/avardwoolaver/7137096221