Distributed queues, threads and sockets with ZeroMQ Burgasconf 2017, Burgas, Bulgaria

1. Distributed queues with
ØMQ
A QUICK GUIDE

2. ØMQ – what it is and what it’s not
 ØMQ is:
 Broker-less
 Fast (Ø-latency)
 Distributed
 Library
 Open-source (LGPL)
 ØMQ is not/doesn’t have:
 Centralized message queue server
 Administration UI
Image credit: Randy Bias/cloudscaling.com

3. ØMQ – how to use it?
 Server (server.py)
import zmq
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:%s" % 5556)
while True:
message = socket.recv()
print "Received: ", message
socket.send(”Server ack”)

4. ØMQ – how to use it?
 Client (client.py)
import zmq
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:%s" % 5556)
socket.send("Greetings")
message = socket.recv()
print "Received reply: ", message

5. ØMQ – how to use it?
$ python server.py
$ python client.py
Received: Greetings
Received: Greetings
$ python client.py
Received reply: Server ack
Received reply: Server ack

6. ØMQ – Threads, synchronization
 I am OS Thread,
nice to meet you.
 OS threads vs. Green threads
 Mutex, semaphore
 Go away GIL, you’re drunk!
Image credit: https://community.eveonline.com/news/dev-blogs/carbonio-and-bluenet-next-level-network-technology-1/

7. ØMQ – Threads, synchronization
Image credit: https://community.eveonline.com/news/dev-blogs/carbonio-and-bluenet-next-level-network-technology-1/
Some people, when confronted
with a problem, think, "I know, I'll
use mutexes.”
Now they have
Some people, when confronted with a
problem, think "I know, I'll use
multithreading".
Nothhw tpe yawrve o oblems.

8. ØMQ – Threads, synchronization
Image credit: http://www.tivix.com/blog/lets-go-python/

9. ØMQ – socket types
 REQ-REP
Image credit: http://zguide.zeromq.org/

10. ØMQ – socket types
 PUSH-PULL
Image credit: http://zguide.zeromq.org/

11. ØMQ – socket types
 PUB-SUB
Image credit: http://zguide.zeromq.org/

12. ØMQ – socket types
 Advanced
PUB-SUB
XPUB-XSUB
Image credit: http://zguide.zeromq.org/

13. ØMQ – message structure
 Multi-frame
 Header
 Body
Image credit: http://zguide.zeromq.org/

14. ØMQ – socket types
 DEALER
 A.k.a. proxy
 Round-robin
distribution
 One-way
distribution
Image credit: http://zguide.zeromq.org/

15. ØMQ – socket types
 ROUTER
 Directed messages
 Can perform non-uniform
distribution, based on
message type
Image credit: http://zguide.zeromq.org/

16. ØMQ – socket types
 PAIR
 Sync between threads
 Uses inproc://
Image credit: http://zguide.zeromq.org/

17. ØMQ – socket types
 Supported socket transports
Image credit: http://zguide.zeromq.org/
socket = context.bind(”tcp://0.0.0.0:5555")
socket = context.bind(”inproc://endpoint")
socket = context.bind(”icp:///tmp/socket_descriptor")
socket = context.bind(”epgm://host:port ")
socket = context.bind(”tcp://eth0:5555")

18. ØMQ – POSIX socket basics
 Bind and connect
 The LOG collector example
 Who is the SERVER?
Image credit: http://zguide.zeromq.org/

19. ØMQ – POSIX socket basics
 select, poll, epoll
import zmq
context = zmq.Context()
receiver = context.socket(zmq.PULL)
receiver.connect("tcp://localhost:5557")
subscriber = context.socket(zmq.SUB)
subscriber.connect("tcp://localhost:5556")
subscriber.setsockopt(zmq.SUBSCRIBE, b”topic")

20. ØMQ – POSIX socket basics
poller = zmq.Poller()
poller.register(receiver, zmq.POLLIN)
poller.register(subscriber, zmq.POLLIN)
while True:
sockets = dict(poller.poll())
if receiver in sockets:
message = receiver.recv()
if subscriber in sockets:
message = subscriber.recv()

21. ØMQ – Examples
 Load Balancing Broker
Image credit: http://zguide.zeromq.org/

22. ØMQ – Examples
Image credit: http://zguide.zeromq.org/
 High-availability
Clone Server Pair

23. ØMQ – Examples
Image credit: http://zguide.zeromq.org/
 State replication

24. ØMQ – Features
Image credit: http://zguide.zeromq.org/
 Encryption
 http://curvezmq.org/
 Elliptic-curve cryptography

25. Thank you!
Tihomir Trifonov
https://bg.linkedin.com/in/tisho