Developing highly scalable applications with Symfony and RabbitMQ

1. Developing highly scalable
applications with
Symfony and RabbitMQ
Alexey Petrov
alexhelkar@gmail.com

2. Symfony and RabbitMQ
Agenda
• Story 1: For those who not used queueing before
• Story 2: For those who started using queueing
• Story 3: For those who have to scale and distribute
• Story 4: Symfony2 Integration

3. Symfony and RabbitMQ
Story 1
For those who not used queueing before

4. Symfony and RabbitMQ
Request
Response
Time
P
R
O
C
E
S
S
I
N
G

5. Symfony and RabbitMQ
Time min
Muscules
(better hardware)
Brains
(better software)

6. Prepare
Response
Symfony and RabbitMQ
Request
Response
Time
Parsing
Request
P
R
O
C
E
S
S
I
N
G

7. Symfony and RabbitMQ
Request
Response
Time
Parsing
Request
Prepare
Response
P
R
O
C
E
S
S
I
N
G

8. Symfony and RabbitMQ
Request
Response
Time
Parsing
Request
Prepare
Response Task 1 Task 2

9. Symfony and RabbitMQ
What is asynchronous?

10. Symfony and RabbitMQ

11. Symfony and RabbitMQ
How to run asynchronous
process from PHP

12. Symfony and RabbitMQ
• popen
• proc_open
• shell_exec
• system
• exec

13. Symfony and RabbitMQ
Symfony Process

14. Symfony and RabbitMQ

15. Symfony and RabbitMQ

16. Symfony and RabbitMQ

17. Symfony and RabbitMQ
Problems?

18. Symfony and RabbitMQ
1 Request = X tasks
N Requests = N*X tasks

19. Symfony and RabbitMQ
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
Worker

20. Symfony and RabbitMQ
Database Cron Job
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
Worker

21. Symfony and RabbitMQ
Problems?

22. Symfony and RabbitMQ
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
Worker
Producer
Producer
Producer

23. Symfony and RabbitMQ
Producer Worker
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
T
A
S
K
Producer
Producer
Worker
Worker

24. Symfony and RabbitMQ
Solution:
USE RIGHT TOOLS!

25. Symfony and RabbitMQ
Use RabbitMQ

26. Symfony and RabbitMQ

27. Symfony and RabbitMQ
Story 2
For those who started using queueing

28. Symfony and RabbitMQ
Basic Examples
http://www.rabbitmq.com/getstarted.html

29. Symfony and RabbitMQ
Connection and Channels

30. Symfony and RabbitMQ
COOL STORY BRO
But we’re in PHP :(

31. Symfony and RabbitMQ
Exchanges To Exchanges

32. Symfony and RabbitMQ
Exchanges and Queues
Fanout
DB1
DB2
Msg Topic Stats
Email
SMS
Topic

33. Symfony and RabbitMQ
Libraries
• PhpAmqpLib
https://github.com/videlalvaro/php-amqplib
• PECL amqp
http://php.net/manual/pl/book.amqp.php
https://github.com/alanxz/rabbitmq-c
https://github.com/bkw/pecl-amqp-official/tree/master/stubs

34. Symfony and RabbitMQ
PECL amqp
https://bugs.php.net/bug.php?id=66661

35. Symfony and RabbitMQ
Durability

36. Symfony and RabbitMQ
Durability
• Exchanges
• Queues
• Messages

37. Symfony and RabbitMQ
Durability

38. Symfony and RabbitMQ
Transactions

39. Symfony and RabbitMQ
Transactions

40. Symfony and RabbitMQ
4 minutes to publish
10 000 messages

41. Symfony and RabbitMQ
Publisher Confirms

42. Symfony and RabbitMQ
Publisher Confirms

43. Symfony and RabbitMQ
Publisher Confirms

44. Symfony and RabbitMQ
Dead Letter Exchanges

45. Symfony and RabbitMQ
Dead Letter Exchanges
• Delayed Messages (Events)
• Retries
• Errors processing
• etc.

46. Symfony and RabbitMQ
Dead Letter Exchanges
• The message is rejected
(basic.reject or basic.nack)
with requeue=false
• The TTL for the message expires;
• The queue length limit is exceeded.

47. Symfony and RabbitMQ
Dead Letter Exchanges

48. Symfony and RabbitMQ
Dead Letter Messages
x-death header:
queue – name of the queue the message was in before
reason – rejected | expired | maxlen
time – timestamp the message was dead lettered
exchange – the exchange the message was published to
routing-keys – the routing keys the message was published with.
original-expiration – the original expiration message property

49. Symfony and RabbitMQ
Delayed Pattern
MSG Exchange
Q1
TTL = 60000
DLX Q2

50. Symfony and RabbitMQ
More Queues = Better
Queue lives in 1 Eralng process

51. Symfony and RabbitMQ
Per-Message TTL
vs
Per-Queue TTL

52. Symfony and RabbitMQ

53. Symfony and RabbitMQ
Story 3
For those who have to scale and distribute

54. Symfony and RabbitMQ
Clustering

55. Symfony and RabbitMQ
Clustering

56. Symfony and RabbitMQ
1 client = 1 socket
Cluster helps to scale!

57. Symfony and RabbitMQ
Mirrored queues

58. Symfony and RabbitMQ
Mirrored queues

59. Symfony and RabbitMQ
Network Partitions

60. Symfony and RabbitMQ
Network Partitions
• Ignore mode
• Pause-minority mode
• Autoheal mode

61. Symfony and RabbitMQ
Federation

62. Symfony and RabbitMQ
Federation

63. Symfony and RabbitMQ
Federated Exchanges

64. Symfony and RabbitMQ
Federated Queues

65. Symfony and RabbitMQ
Shovel

66. Symfony and RabbitMQ
The Symfony

67. Symfony and RabbitMQ
Story 4
Symfony2 Integration

68. Symfony and RabbitMQ
Installing Bundle

69. Symfony and RabbitMQ
Configuration

70. Symfony and RabbitMQ
Producer

71. Symfony and RabbitMQ
Consumer

72. Symfony and RabbitMQ
Questions?
Alexey Petrov
Senior PHP Developer / System Architect
Email: alexhelkar@gmail.com
Twitter: @alexhelkar