O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Advanced Messaging with
       Apache ActiveMQ


Bosanac Dejan
May 2011




                                              ...
About me

 Bosanac Dejan
 Senior Software Engineer at FUSESource - http://
  fusesource.com
 Apache ActiveMQ committer ...
What we are going to cover?




       What is ActiveMQ
       The Basics
       Flow control
       Scaling
       H...
What is ActiveMQ?




                                                                                                    ...
Apache ActiveMQ



 Apache ActiveMQ
     • Leading Open Source messaging platform
     • Supported Java Standards:
      ...
Background




       ActiveMQ started in 2005 at CodeHaus
       Moved to Apache Software Foundation in 2006
       1,...
The Basics




                                                                                                           ...
Messaging is




 Loosely coupled exchange of messages between
  applications
 Location transparency
 Can be persistent...
Topics




                                                                                                               ...
Topics




 One message goes to 0-to-many consumers
  based on the current subscribers
 Think like mailing lists or disc...
Queues




                                                                                                               ...
Queues




    Messages are load balanced across many consumers
    Each message goes to exactly one consumer
    Consu...
Challanges


 Create a general messaging platform
 Support variety of use-cases
     • Large number of clients
     • La...
Flow Control




                                                                                                         ...
Flow Control - Why?

 Dealing with deep queues
 Dealing with slow consumers
 We want to prevent broker from being floode...
Flow Control - How?




 Message Cursors
 Producer Flow Control




                                                    ...
Flow Control - Cursors




                                                                                               ...
Flow Control - Store-based Cursor




                                                                                    ...
Flow Control - Non-persistent cursor




                                                                                 ...
Flow Control - Limits

                        Per destination
     <destinationPolicy>
           <policyMap>
           ...
Flow Control - Producer Flow Control

 Throttling producer speed to the speed of consumers




                          ...
Flow Control - VM Cursor




     <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
       <pendingQueu...
Scaling




                                                                                                              ...
Scaling - Types




 Vertical scaling
 Horizontal scaling
 Traffic partitioning




                                   ...
Vertical Scaling




     Increase load capacity using a single broker can handle.



     Problems:
     • Thread count
 ...
Vertical Scaling - Threads




     What are threads used for
     • For Connections - Thread per Connection (blocking tra...
Vertical Scaling - Number of Connections




     Use non-blocking transport

     <transportConnectors>
       <transport...
Vertical Scaling - Number of Destinations




     Don’t use dedicated task runner

      ACTIVEMQ_OPTS="-Dorg.apache.acti...
Vertical Scaling - Memory


 Give broker enough memory

     ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedT...
Vertical Scaling - CPU




 Disable tight encoding

 It uses more CPU to create smaller packets

 uri = "failover://(tcp:...
Vertical Scaling - Conclusion




     There is a limit to the scalability a single machine can
                          ...
Horizontal Scaling


     Increase load capacity using networked brokers




  Concepts:
 Network of Broker




         ...
Horizontal Scaling - Network of Brokers




                                                                              ...
Horizontal Scaling - NOB Usage

                                                       Configuration

                <netw...
Horizontal Scaling - Conclusion




 More latency in processing messages
 Beware of complex topologies




             ...
Hybrid Scaling

     Partition traffic to more non-connected brokers




                                                 ...
Hybrid Scaling - Conclusion




 Pros
     • You can use all the tuning techniques used in Vertical scaling
     • Have b...
High Availability




                                                                                                    ...
High Availability




 Pure Master/Slave
 JDBC Master/Slave
 Shared File System Master/Slave




                      ...
Pure Master-Slave




                                                                                                    ...
Pure Master-Slave




 Shared nothing
 Fully replicated
     • All messages
     • All acknowledgements
     • All trans...
JDBC Master-Slave




                                                                                                    ...
JDBC Master-Slave




    Extreme reliability – but not as fast
    Recommended if already using an enterprise database
...
Shared Storage Master-Slave




                                                                                          ...
Shared Storage Master-Slave




    Recommended if you have a SAN
    No restriction on number of slaves
    Simple con...
Future




                                                                                                               ...
Future - ActiveMQ Apollo


 http://activemq.apache.org/apollo

 ActiveMQ 5.x reached scalability and performance
  limit...
Future - ActiveMQ Apollo




    Reactor Based Thread Model
    Scala 2.8 Implementation
    Protocol Agnostic
    RES...
Future - ActiveMQ Apollo Performance


                            http://hiramchirino.net/blog




                      ...
Conclusions




    Dynamic community
    Leading in terms of messaging innovation
    Built for Enterprise
    Scalab...
Questions?




 ActiveMQ Web sites:
     • http://activemq.apache.org/
     • http://fusesource.com/products/enterprise-a...
Próximos SlideShares
Carregando em…5
×
Próximos SlideShares
Messaging With ActiveMQ
Avançar
Transfira para ler offline e ver em ecrã inteiro.

Compartilhar

Advanced messaging with Apache ActiveMQ

Baixar para ler offline

Features of ActiveMQ message broker for dealing with high load, scaling and high-availability

Advanced messaging with Apache ActiveMQ

  1. 1. Advanced Messaging with Apache ActiveMQ Bosanac Dejan May 2011 A Progress Software 1 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  2. 2. About me  Bosanac Dejan  Senior Software Engineer at FUSESource - http:// fusesource.com  Apache ActiveMQ committer and PMC member  Co-author of ActiveMQ in Action A Progress Software 2 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  3. 3. What we are going to cover?  What is ActiveMQ  The Basics  Flow control  Scaling  High Availability  Future  Conclusion A Progress Software 3 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  4. 4. What is ActiveMQ? A Progress Software 4 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  5. 5. Apache ActiveMQ  Apache ActiveMQ • Leading Open Source messaging platform • Supported Java Standards: o JMS 1.1, J2EE 1.4, JCA 1.5 and XA  Reliable, high performance messaging • Out-performs many legacy proprietary message queues • Configurable for many different deployments  Multi-Protocol/Multi-Language Support A Progress Software 5 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  6. 6. Background  ActiveMQ started in 2005 at CodeHaus  Moved to Apache Software Foundation in 2006  1,117,537 lines of code  24 committers  Now the most widely used open source messaging system on the planet A Progress Software 6 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  7. 7. The Basics A Progress Software 7 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  8. 8. Messaging is  Loosely coupled exchange of messages between applications  Location transparency  Can be persistent or non-persistent  Can be transactional A Progress Software 8 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  9. 9. Topics A Progress Software 9 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  10. 10. Topics  One message goes to 0-to-many consumers based on the current subscribers  Think like mailing lists or discussion forums  Ideal for publishing business events  Distributed observer pattern  Allows one part of your system to notify anyone else who may be interested in an event A Progress Software 10 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  11. 11. Queues A Progress Software 11 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  12. 12. Queues  Messages are load balanced across many consumers  Each message goes to exactly one consumer  Consumers compete for messages  Its easy to browse and monitor queues  Ideal for grid style applications A Progress Software 12 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  13. 13. Challanges  Create a general messaging platform  Support variety of use-cases • Large number of clients • Large number of destinations • Slow consumers  Provide enterprise feaures • Security • High availability • Management • etc A Progress Software 13 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  14. 14. Flow Control A Progress Software 14 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  15. 15. Flow Control - Why?  Dealing with deep queues  Dealing with slow consumers  We want to prevent broker from being flooded with messages  We want to prevent broker running out of memory and other resources A Progress Software 15 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  16. 16. Flow Control - How?  Message Cursors  Producer Flow Control A Progress Software 16 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  17. 17. Flow Control - Cursors A Progress Software 17 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  18. 18. Flow Control - Store-based Cursor A Progress Software 18 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  19. 19. Flow Control - Non-persistent cursor A Progress Software 19 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  20. 20. Flow Control - Limits Per destination <destinationPolicy> <policyMap> <policyEntries> <policyEntry queue=">" memoryLimit="10mb"/> </policyEntries> </policyMap> </destinationPolicy> System settings <systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="256 mb" /> </memoryUsage> <storeUsage> <storeUsage limit="100 gb" /> </storeUsage> <tempUsage> <tempUsage limit="10 gb" /> </tempUsage> </systemUsage> </systemUsage> A Progress Software 20 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  21. 21. Flow Control - Producer Flow Control  Throttling producer speed to the speed of consumers A Progress Software 21 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  22. 22. Flow Control - VM Cursor <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"> <pendingQueuePolicy> <vmQueueCursor/> </pendingQueuePolicy> </policyEntry> A Progress Software 22 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  23. 23. Scaling A Progress Software 23 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  24. 24. Scaling - Types  Vertical scaling  Horizontal scaling  Traffic partitioning A Progress Software 24 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  25. 25. Vertical Scaling Increase load capacity using a single broker can handle. Problems: • Thread count • Memory usage • CPU usage A Progress Software 25 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  26. 26. Vertical Scaling - Threads What are threads used for • For Connections - Thread per Connection (blocking transport) • For Dispatching - Thread per Destination A Progress Software 26 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  27. 27. Vertical Scaling - Number of Connections Use non-blocking transport <transportConnectors> <transportConnector name="nio" uri="nio://0.0.0.0:61616"/> </<transportConnectors> Enables handling large number of clients A Progress Software 27 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  28. 28. Vertical Scaling - Number of Destinations Don’t use dedicated task runner ACTIVEMQ_OPTS="-Dorg.apache.activemq.UseDedicatedTaskRunner=false" Use optimized dispatch for queues <destinationPolicy> <policyMap> <policyEntries> <policyEntry topic=">" optimizedDispatch="true"> ... </policyEntry> </policyEntries> </policyMap> </destinationPolicy> A Progress Software 28 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  29. 29. Vertical Scaling - Memory  Give broker enough memory ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRunner=false"  Configure big enough memory usage <systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="1024 mb" /> </memoryUsage> ... </systemUsage> </systemUsage> A Progress Software 29 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  30. 30. Vertical Scaling - CPU  Disable tight encoding It uses more CPU to create smaller packets uri = "failover://(tcp://localhost:61616wireFormat.tightEncodingEnabled=false)"; A Progress Software 30 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  31. 31. Vertical Scaling - Conclusion There is a limit to the scalability a single machine can give A Progress Software 31 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  32. 32. Horizontal Scaling Increase load capacity using networked brokers Concepts:  Network of Broker A Progress Software 32 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  33. 33. Horizontal Scaling - Network of Brokers A Progress Software 33 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  34. 34. Horizontal Scaling - NOB Usage Configuration <networkConnector name="broker1-broker2" uri="static:(tcp://broker2:61617)" dynamicOnly="true" prefetchSize="1000" conduitSubscriptions="true" decreaseNetworkConsumerPriority="true" suppressDuplicateTopicSubscriptions="true" networkTTL="3"> </networkConnector> Connecting failover://(tcp://broker1:61616,tcp://broker2:61616)?randomize=true A Progress Software 34 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  35. 35. Horizontal Scaling - Conclusion  More latency in processing messages  Beware of complex topologies A Progress Software 35 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  36. 36. Hybrid Scaling Partition traffic to more non-connected brokers A Progress Software 36 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  37. 37. Hybrid Scaling - Conclusion  Pros • You can use all the tuning techniques used in Vertical scaling • Have better Horizontal scaleability than using Network Of Brokers (Less broker cross talk)  Cons • Added complexity required on the end user Application A Progress Software 37 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  38. 38. High Availability A Progress Software 38 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  39. 39. High Availability  Pure Master/Slave  JDBC Master/Slave  Shared File System Master/Slave A Progress Software 39 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  40. 40. Pure Master-Slave A Progress Software 40 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  41. 41. Pure Master-Slave  Shared nothing  Fully replicated • All messages • All acknowledgements • All transactions  Slave does not start any transports or network connections A Progress Software 41 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  42. 42. JDBC Master-Slave A Progress Software 42 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  43. 43. JDBC Master-Slave  Extreme reliability – but not as fast  Recommended if already using an enterprise database  No restriction on number of slaves  Simple configuration A Progress Software 43 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  44. 44. Shared Storage Master-Slave A Progress Software 44 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  45. 45. Shared Storage Master-Slave  Recommended if you have a SAN  No restriction on number of slaves  Simple configuration  N.B. – ensure file locking works – and times out – NFSv4 good! A Progress Software 45 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  46. 46. Future A Progress Software 46 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  47. 47. Future - ActiveMQ Apollo  http://activemq.apache.org/apollo  ActiveMQ 5.x reached scalability and performance limits with the current architecture  New broker core A Progress Software 47 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  48. 48. Future - ActiveMQ Apollo  Reactor Based Thread Model  Scala 2.8 Implementation  Protocol Agnostic  REST Based Management A Progress Software 48 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  49. 49. Future - ActiveMQ Apollo Performance http://hiramchirino.net/blog 10 producers/10 consumers single topic using Stomp 20 byte payload A Progress Software 49 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  50. 50. Conclusions  Dynamic community  Leading in terms of messaging innovation  Built for Enterprise  Scalable, Good Performance, Reliable A Progress Software 50 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  51. 51. Questions?  ActiveMQ Web sites: • http://activemq.apache.org/ • http://fusesource.com/products/enterprise-activemq/  Blog: • http://www.nighttale.net/  Twitter: • http://twitter.com/dejanb • http://twitter.com/fusenews A Progress Software 51 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  • jadeflon

    Oct. 24, 2017
  • AmarBabuKakumanu

    Oct. 12, 2017
  • EminBAHADIR

    Sep. 17, 2017
  • kunkunur

    Jun. 2, 2017
  • DariuszFurmaniak

    May. 6, 2017
  • hyh961

    Apr. 15, 2017
  • TatuMikkonen

    Mar. 11, 2017
  • sharmisthamona

    Feb. 16, 2017
  • rafaelcba

    Dec. 30, 2016
  • AlexMart001

    Nov. 1, 2016
  • MerkerXu

    Sep. 14, 2016
  • MuralimohanDintakurt

    Aug. 28, 2016
  • pvmsikrsna

    May. 12, 2016
  • PhaniTurlapati

    Apr. 14, 2016
  • jesusvegagonzalez

    Apr. 5, 2016
  • AlejandroFlores158

    Mar. 13, 2016
  • hyoungseunglee

    Mar. 2, 2016
  • ssuser52c883

    Feb. 22, 2016
  • moviesnmusic

    Dec. 16, 2015
  • AdeLucas1

    Nov. 18, 2015

Features of ActiveMQ message broker for dealing with high load, scaling and high-availability

Vistos

Vistos totais

40.603

No Slideshare

0

De incorporações

0

Número de incorporações

5.204

Ações

Baixados

1.150

Compartilhados

0

Comentários

0

Curtir

68

×