This document discusses best practices for optimizing RabbitMQ performance and stability based on experience running thousands of RabbitMQ clusters. Key recommendations include using short-lived, separate connections for publishers and consumers; limiting queue sizes; enabling lazy queues for predictable performance; splitting queues across cores; keeping connections and channels open; adjusting prefetch values; using a stable RabbitMQ version; disabling unused plugins; and deleting unused queues. Diagnostic tools are also recommended to check for issues like connection leaks, large queues, and improper exchange usage.

1. What we've learned from running thousands
of production RabbitMQ clusters
Lovisa Johansson
lovisa@cloudamqp.com

4. 3000 emails

5. ● Unstable RabbitMQ version
● Unoptimized configuration for a specific use case
➢ High availability
➢ High Performance
● Users (you?) are using RabbitMQ in a bad way
● Client libraries are using RabbitMQ in bad way
● Things are not done in an optimal way
● Customer use cases
● Configuration mistakes
● Common mistakes
Client side problems
Server side problems

6. What we've learned from running
thousands of production RabbitMQ
clusters

7. Lovisa Johansson
Marketing Manager
Support Engineer
RabbitMQ Engineer
Umeå, Sweden

8. 23000 running instances 7 clouds
Largest provider of managed RabbitMQ servers
75 regions
Headquarter
Stockholm Sweden

9. Don’t use too many connections or channels
● Keep connection/channel count low
● Each connection uses about 100 KB of RAM
● Thousands of connections can be a heavy burden on a RabbitMQ server
● Channel and connections leaks are among the most common errors that we see
Recommendation number 1.
CONNECTIONS AND CHANNELS

10. ● Long-lived connections.
● Don’t open a channel every
time you are publishing
Don’t open and close connections or channels repeatedly
● AMQP connections: 7 TCP packages
● AMQP channel: 2 TCP packages
● AMQP publish: 1 TCP package
● AMQP close channel: 2 TCP packages
● AMQP close connection: 2 TCP packages
Total 14-19 packages (+ acks)
Recommendation number 2.
CONNECTIONS AND CHANNELS

11. ● Our benchmarks show that the proxy is increasing publishing
speed with a magnitude or more.
● https://github.com/cloudamqp/amqproxy
● Some clients can’t keep long-lived connections
(looking at you PHP )
● Avoid connection churn by using a proxy that pools
connections and channels for reuse.
AMQProxy

12. Flow control: Might not be able to consume if the connection is in flow control
Back pressure: RabbitMQ can apply back pressure on the TCP connection when the
publisher is sending too many messages
Separate connections for publishers and consumers
Recommendation number 3.
CONNECTIONS AND CHANNELS

14. ● Less than 10 000 messages in one queue
● Heavy load on RAM usage
QUEUES
Recommendation number 4.
Don't have too large queues
○ In order to free up RAM, RabbitMQ starts page out messages to disk
○ Blocks the queue from processing messages
● Time-consuming to restart a cluster
● Limit queue size with TTL or max-length

15. ● Lazy queues was added in RabbitMQ 3.6
● Writes messages to disk immediately, thus spreading the work out over time instead of taking the
risk of a performance hit somewhere down the road
● More predictable and smooth performance curve
○ Messages are only loaded into memory when they are needed.
Enable lazy queues to get predictable performance
Recommendation number 5.
QUEUES
Enable lazy queues if…
● the publisher is sending many messages at once
● the consumers are not keeping up with the speed of the publishers all the time
Ignore lazy queues if..
● you require high performance
● queues are always short

16. The RabbitMQ management collects and calculates metrics for every queue, connection,
and channel in the cluster
● Slows down the server if you have thousands upon thousands of active queues or
consumers
Don’t set RabbitMQ Management statistics rate mode to detailed
Recommendation number 6.
QUEUES

17. Split queues over different cores, and route messages to multiple
queues
Recommendation number 7.1
QUEUES
● A queue is single threaded
○ 50k messages/s
● Queue performance is limited to one CPU core.
● All messages routed to a specific queue will end up
on the node where that queue resides.
Plugins
The consistent hash
exchange plugin
RabbitMQ sharding

18. Recommendation number 7.2
QUEUES
● Load-balance messages between queues
● Messages are consistently and equally distributed across many queues
● Consume from all queues
● https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange
The consistent hash exchange plugin

19. Recommendation number 7.3
QUEUES
RabbitMQ sharding
● Automatic partitioning of queues
● Queues are created on every cluster node and messages are sharded across them
● Shows one queue to the consumer, but it could be many queues running behind it in
the background
● https://github.com/rabbitmq/rabbitmq-sharding

20. Recommendation number 8.
QUEUES
Have limited use on priority queues
● Each priority level uses an internal queue on the Erlang VM, which takes up
resources.
● In most use cases it's sufficient to have no more than 5 priority levels.

21. Recommendation number 9.
QUEUES
Send persistent messages and durable queues
● Messages, exchanges, and queues that are not durable and persistent are lost
during a broker restart
● High performance - use transit messages and temporary, or non-durable queues

22. Recommendation number 10.1
PREFETCH
Adjust prefetch value
● Limits how many messages the client can receive before acknowledging a message
● RabbitMQ default prefetch value - unlimited buffer
● RabbitMQ 3.7
○ Option to adjust the default prefetch
○ CloudAMQP servers has a default prefetch of 1000

23. Recommendation number 10.2
PREFETCH
Prefetch - Too small prefetch value
RabbitMQ is most of the
time waiting to get
permission to send more
messages

24. Recommendation number 10.3
PREFETCH
Prefetch - Too large prefetch value

25. Recommendation number 10.4
PREFETCH
Prefetch
● One single or few consumers with short processing time
○ prefetch many messages at once
● About the same processing time and a stable network
○ estimated prefetch value by using the total round trip time divided by
processing time on the client for each message
● Many consumers, and short processing time
○ A lower prefetch value than for one single or few consumers
● Many consumers, and/or long processing time
○ Set prefetch count to 1 so that messages are evenly distributed among all
your workers
● The prefetch value have no effect if your client auto-ack messages

26. Recommendation number 11.
HiPE
HiPE
● HiPE increases server throughput at the cost of increased start-up time
○ increases throughput with 20-80%
○ increases start-up time about 1 to 3 minutes
● HiPE is recommended if you require high availability
● We don’t consider HiPE as experimental any longer

27. ● Pay attention to where in your consumer logic you’re acknowledging messages
● For the fastest possible throughput, manual acks should be disabled
● Publish confirm is required if the publisher needs messages to be processed at
least once
Recommendation number 12.
ACKS AND CONFIRMS
Acknowledgments and Confirms

28. Great improvements are made to RabbitMQ, all the time <3
● 3.7
○ Default prefetch
○ Individual vhost message stores
● 3.6
○ Lots of many memory problems, up to version 3.6.14
○ Lazy queues
● 3.5
○ Still may customers on 3.5.7
Recommendation number 13.
VERSION
Use a stable RabbitMQ version
Back compatibility is
really good in RabbitMQ

29. ● Some plugins are consuming lots of resources
● Make sure to disable plugins that you are not using
Recommendation number 14.
Plugins
Disable plugins you are not using

30. ● Unused queues take up some resources, queue index, management statistics etc
● Temporary queues should be auto deleted
Recommendation number 15.
Unused queues
Delete unused queues

31. ● Message loss on netsplits
● Needed to be able to upgrade without losing messages at CloudAMQP
Recommendation number 16.
VHOST
Enable HA-vhost policy on custom vhosts

32. Summary Overall
Server side problems
● Short queues
● Long lived connections
● Limited use of priority queues
● Use multiple queues and consumers
● Split your queues over different cores
● Stable Erlang and RabbitMQ version
● Disable plugins you are not using
● Channels on all connections

33. Summary Overall
Server side problems
● Separate connections for publishers and
consumers
● Management statistics rate mode
● Delete unused queues
● Temporary queues should be auto deleted

34. Summary High Performance
Server side problems
● Short queues
○ max-length if possible
● Do not use lazy queues
● Send transit messages
● Disable manual acks and publish
confirms
● Avoid multiple nodes (HA)
● Enable RabbitMQ HiPE

35. Summary High Availability
Server side problems
● Enable lazy queues
● RabbitMQ HA - 2 nodes
○ HA-policy on all vhosts
● Persistent messages, durable queues
● Do not enable HiPE

36. DIAGNOSTIC TOOL

37. DIAGNOSTIC TOOL
Diagnostics Tool
● RabbitMQ and Erlang version
● Queue length
● Unused queues
● Persistent messages in durable queues
● No mirrored auto delete queues
● Limited use of priority queues
● Long lived connections
● Connection and channel leak
● Channels on all connections
● Insecure connections
● Client library
● AMQP Heartbeats
● Channel prefetch
● Limited use of priority queues
● Management statistics rate mode
● Ensure that you are not using topic exchange as fanout
● Ensure that all published messages are routed
● Ensure that you have a HA-policy on all vhosts
● Auto delete on temporary queues
● Persistent messages in durable queues
● No transient messages in mirrored queues
● No mirrored auto delete queues
● Separate connections for publishers and consumers

38. It should be easier to do things
right!

39. Questions?
Visit www.cloudamqp.com blog site, documentation and FAQ for more info