Mais conteĂșdo relacionado
Semelhante a Data Migration at Scale with RabbitMQ and Spring Integration (20)
Mais de Alvaro Videla (20)
Data Migration at Scale with RabbitMQ and Spring Integration
- 1. Data Migration at Scale with
RabbitMQ and Spring Integration
By Ălvaro Videla and FĂ©lix LĂłpez
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 2. Ălvaro Videla
âą Developer Advocate at Pivotal / RabbitMQ
âą Co-Author of RabbitMQ in Action
âą Creator of the RabbitMQ Simulator
âą Blogs about RabbitMQ Internals: http://videlalvaro.github.io/
internals.html
âą @old_sound â alvaro@rabbitmq.com â github.com/videlalvaro
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
2
- 3. Ălvaro Videla
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
3
Co-authored
!
RabbitMQ in Action
http://bit.ly/rabbitmq
- 4. What is RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
4
- 5. RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
5
- 6. RabbitMQ
âą Multi Protocol Messaging Server
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
6
- 7. RabbitMQ
âą Multi Protocol Messaging Server
âą Open Source (MPL)
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
7
- 8. RabbitMQ
âą Multi Protocol Messaging Server
âą Open Source (MPL)
âą Polyglot
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
8
- 9. RabbitMQ
âą Multi Protocol Messaging Server
âą Open Source (MPL)
âą Polyglot
âą Written in Erlang/OTP
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
9
- 10. Multi Protocol
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
10
http://bit.ly/rmq-protocols
- 11. Polyglot
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
11
- 12. Polyglot
âą Java
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
12
- 13. Polyglot
âą Java
âą node.js
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
13
- 14. Polyglot
âą Java
âą node.js
âą Erlang
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
14
- 15. Polyglot
âą Java
âą node.js
âą Erlang
âą PHP
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
15
- 16. Polyglot
âą Java
âą node.js
âą Erlang
âą PHP
âą Ruby
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
16
- 17. Polyglot
âą Java
âą node.js
âą Erlang
âą PHP
âą Ruby
âą .Net
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
17
- 18. Polyglot
âą Java
âą node.js
âą Erlang
âą PHP
âą Ruby
âą .Net
âą Haskell
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
18
- 19. Polyglot
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
19
Even COBOL!!!11
- 21. Messaging with RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
21
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator
- 22. RabbitMQ Simulator
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
22
- 24. RabbitMQ Basic Qos Simulator
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
24
- 25. Prevent Unbound Buffers
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
25
λ = mean arrival time
Ό = mean service rate
if λ > Ό what happens?
- 26. Prevent Unbound Buffers
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
26
λ = mean arrival time
Ό = mean service rate
if λ > Ό what happens?
Queue length goes to infinity over time.
- 27. Prevent Unbound Buffers
λ = mean arrival time
Ό = mean service rate
if λ > Ό what happens?
Queue length goes to infinity over time.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
27
https://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-
with-rabbitmq/
- 28. Recommended Reading
Performance Modeling and
Design of Computer Systems:
Queueing Theory in Action
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
28
- 29. What you get out of the box
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
29
- 30. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
30
- 31. Some RabbitMQ Features
âą Persistent Messages
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
31
- 32. Some RabbitMQ Features
âą Persistent Messages
âą Publisher Confirms
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
32
- 33. Some RabbitMQ Features
âą Persistent Messages
âą Publisher Confirms
âą Message Acknowledgment
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
33
- 34. Some RabbitMQ Features
âą Persistent Messages
âą Publisher Confirms
âą Message Acknowledgment
âą Mirrored Queues
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
34
- 35. Some RabbitMQ Features
âą Persistent Messages
âą Publisher Confirms
âą Message Acknowledgment
âą Mirrored Queues
âą Message Ordering
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
35
- 36. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
36
- 37. Some RabbitMQ Features
âą Dead Letter Exchanges
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
37
- 38. Some RabbitMQ Features
âą Dead Letter Exchanges
âą Alternate Exchanges
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
38
- 39. Some RabbitMQ Features
âą Dead Letter Exchanges
âą Alternate Exchanges
âą Message and Queues TTLs
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
39
- 40. Some RabbitMQ Features
âą Dead Letter Exchanges
âą Alternate Exchanges
âą Message and Queues TTLs
âą Consumer Priorities
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
40
- 41. Some RabbitMQ Features
âą Dead Letter Exchanges
âą Alternate Exchanges
âą Message and Queues TTLs
âą Consumer Priorities
âą Federation
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
41
- 42. Some RabbitMQ Features
âą Dead Letter Exchanges
âą Alternate Exchanges
âą Message and Queues TTLs
âą Consumer Priorities
âą Federation
âą Shovel
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
42
- 43. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
43
- 44. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
44
âą SSL Connections
- 45. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
45
âą SSL Connections
âą User Authentication
- 46. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
46
âą SSL Connections
âą User Authentication
âą Per resource authorisation
- 47. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
47
âą SSL Connections
âą User Authentication
âą Per resource authorisation
âą Different kinds of users
- 48. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
48
âą SSL Connections
âą User Authentication
âą Per resource authorisation
âą Different kinds of users
âą Various authentication mechanisms via plugins (LDAP)
- 49. Management Interface / REST API
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
49
- 50. RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
50
- 51. RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
51
Flexibility
- 52. Questions?
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
52
- 53. Learn More. Stay Connected
@old_sound @flopezluis
@springcentral | spring.io/video
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 53