Designing & architecting RabbitMQ engineered systems - Ayanda Dube @ London RabbitMQ Meetup

www.erlang-solutions.com
AYANDA DUBE
engineer @ESL
DESIGNING & ARCHITECTING
RABBITMQ
ENGINEERED SYSTEMS

FABRIC DESIGN

FABRIC DESIGN : DEFINITIONS
▸ Fabric is/are the internal (user-defined/created) elements within
a RabbitMQ instance
▸ Queues, Exchanges, Bindings, Policies, Parameters, Connections,
Channels, etc, internal to the broker
▸ Fabric design entails structuring all internal elements to precisely
meet/match your requirement
▸ Direct impact on overall performance
▸ Direct impact on overall resource utilization
RabbitMQ Node
Q1
Ex1
Q2
Q3
RK-1
RK-2
RK-3

FABRIC DESIGN : CONTROL
▸ Any interface that directly alters the internal structure of your
RabbitMQ instance
▸ Standard/built-in interfaces are:
▹ Configuration file(s)
▹ CLI
▹ Client Applications (AMQP, MQTT, STOMP, etc)
▹ Management UI/API
▹ Plugins

FABRIC DESIGN : OBJECTIVES
▸ Four main areas of focus:
1. THROUGHPUT (~100%)
2. MEMORY (MINIMAL)
3. LATENCY (MINIMAL)
4. ERGONOMICS

FABRIC DESIGN : FABRIC MATRIX
▸ Rough fabric matrix

THROUGHPUT

FABRIC DESIGN : THROUGHPUT
▸ Achieve highest attainable - 100% ideally!
▸ Efficiently!
▸ Some guarantees of high throughput come at a cost!
▸ e.g. Using transactions, longer TCP buffer sizes, etc
▸ Predictions i.e. model your solution (e.g. based on predicted
consumer-producer ratios)
▸ Run performance tests
▸ Capture results
▹ Observations
▹ Draw conclusions on design
▸ Deduce final design approximations based on quantitative
estimate results

RabbitMQ Node
Q1
Ex1
Q2
Q3
RK-1
RK-2
RK-2
P1
P2
P3
C1
C2
C3
VARYING NODE CONFIG(S)

RabbitMQ Node
Q1
Ex1
Q2
Q3
RK-1
RK-2
RK-2
P1
P2
P3
C2
C3
C4
VARYING CONFIG
C1
C’N

RabbitMQ Node
Q1
Ex1
Q2
Q3
RK-1
RK-2
RK-2
P2
P3
P4
C2
C3
C4
VARYING CONFIG
P1
P’N

My system model approaches 500% consumer-producer ratio
Message payloads less than 1KB
100% throughput attainable at 10KiB sendbuf/recvbuf
▸ Conclude on “near precise” configuration

MEMORY

FABRIC DESIGN : MEMORY
▸ High performance while conserving memory as much as possible
▸ Consequences & impact:
▹ Service
▹ Node terminations
▸ How much memory do I allocate for each node?
▸ Considerations from fabric element
▹ Node
▹ Queues
▹ Connections
▸ Precise RabbitMQ version selection (resolved memory leaks)
▸ Simulate extreme conditions

▸ Node memory
▸ With/without certain plugins?
▸ Apply persistent/long-standing/static configurations
▸ Capture memory utilization
Standalone Node Memory ~ 250MB

▸ Queues
▸ Consider worst case scenario - all queues on single node
1. Define a queue (varying types depending on use-case)
2. Publish max-number of anticipated messages
3. Capture/compare “rabbitmqctl list_queues name memory” = M
4. QueueMemoryLimit = M x ApproximateMaxCount
Total Max-Approximate Queue Memory

▸ Connections
▸ Consider worst case scenario - send/recv buffers full
▸ Consumers - assume sendbuf MAX ( e.g. + ~½ or ALL, confirming
in recvbuf)
▸ Producers - assume recvbuf MAX (~⅓ sendbuf for
confirmations)
▸ Add socket reader memory
▸ And more elements if possible

1. Default 192KiB sendbuf/recvbuf
2. MaxConsumerConnMemory = (192KiB + 62KiB) x N consumers
3. MaxProducerConnMemory = 192KiB x N producers
4. MaxSocketReaderMemory Approximation x Total Connections
Total Max-Approximate Connection Memory
ACKs

Total Max-Approximate Queue Memory
Total Max-Approximate Connection Memory
Total Max-Approximate Live Node Memory Utilization
Standalone Node Memory

▸ Defined queue types to be used (e.g. lazy?)
▸ Purchase appropriate server
▸ Define precise configurations
▹ VM High watermark limits
▹ VM High watermark paging ratio
▸ Same can be done for DISK durable fabric and persistent
messages
▸ And more …

LATENCY

FABRIC DESIGN : LATENCY
▸ Speed of traversal/message propagation time through fabric
▸ Main hindrances:
▹ Head of Line Blocking (HOL)
▹ Routing inefficiencies
▹ *Disk I/O
▹ *Topology Related (Mirroring, no of replicas, ),
RabbitMQ Node
Q1
Ex1
Q2
Q3
RK-1
RK-2
RK-2
P1 C1
LATENCY

▸ HOL
▸ How do we speed-up exit of messages from queues?
RabbitMQ Node
Ex1
Q2
RK-2
P1 C1
LATENCY
BLOCKED
Q1
Q3

▸ Routing inefficiencies
▸ How do we speed-up exit of messages from exchanges?
RabbitMQ Node
Ex1
Q2
RK-2
P1 C1
LATENCY
Q1
Q3
ROUTING
RK-1
RK-3

▸ Strategies for tackling HOL from fabric perspective
▹ Queue types (Default or Lazy?)
▹ Priorities
▹ Load sharing across multiple queues
▹ Queue-Length limits
▹ *Dead-lettering
▹ AE (if used with confirmations)
▸ Strategies for tackling Routing inefficiencies
▹ Avoiding slow exchange types (Headers)
▹ Shared bindings across multiple exchanges
▹ Plugins (sharding, consistent hashing)
▸ Disk I/O delays
▹ Disk types (SSD/HDD)
▹ Node-types (DISK/RAM)
▹ Durability & persistence when appropriate

ERGONOMICS

▸ User friendliness (support)
▸ Fabric state evolves and gets more complex
▸ Maintaining and monitoring can become a nightmare if, no
conventions were followed
▸ Designing fabric to be as easy to maintain and possible
▹ Naming conventions
▹ Policies
▹ Consistent bindings
▹ Precise use of exchanges
▹ Partitioning business logic by VHOST
FABRIC DESIGN : ERGONOMICS

Q1
Q2
Q3
Q4
Q5
Q6
EX1
EX2
EX3
EX4
EX5
EX6

Q1
Q2
Q3
Q1
Q2
Q3
EX1
EX1
vhost-1
vhost-2

That’s it,
for now

Designing & architecting RabbitMQ engineered systems - Ayanda Dube @ London RabbitMQ Meetup

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (19)

Semelhante a Designing & architecting RabbitMQ engineered systems - Ayanda Dube @ London RabbitMQ Meetup

Semelhante a Designing & architecting RabbitMQ engineered systems - Ayanda Dube @ London RabbitMQ Meetup (20)

Mais de Erlang Solutions

Mais de Erlang Solutions (7)

Último

Último (20)

Designing & architecting RabbitMQ engineered systems - Ayanda Dube @ London RabbitMQ Meetup