SlideShare uma empresa Scribd logo
1 de 26
sipXecs Architecture & Direction 
moderator: michael picher 
March 10, 2013 / Bentley University / Boston MA 
1
2 Intro of Presenters Team 
>Load Testing & Build Team: 
>Douglas Hubler 
>Ciprian Hacman 
>SIP Core Team: 
>Joegen Baclor 
>Daniel Tacalau
Agenda 3 
>sipXecs 4.6 Architecture Overview 
>Testing Procedure 
>Test Automation 
>Status of 4.6 
>Roadmap Near-Term / Long-Term
Typical sipXecs Cluster
Multi-master (Before… 4.4 and earlier)
Single Master - Now
Message Queuing Introduced 7 
>Needed better interprocess communications 
>Lightweight 
>Robust 
>Evaluated several 
>Build? 
>RabbitMQ 
>ZeroMQ
Message Queuing 8 
Publisher / Subscriber
Message Queuing 9 
Dealer / Worker
sipXecs 4.6 Architecture 10 
SQA
11 Testing Procedure 
>Manual smoke test 
>Basic tests that catch major issues 
>Manual sanity test 
>Detailed tests for each feature 
>Manual regression tests 
>Complex tests for features that are added/changed 
>Automated Load Tests 
>Deployment for a week on our Production System 
>Dog fooding…
Automated Load Tests 
sipxtest 12 
>Placing and receiving calls is the core feature and we want this to be stable 
>Basic testing cannot predict 
>How a server behaves over time 
>How a server behaves under stress 
>Call Load tests helps address both problems 
>Allows determination the performance of a Certain Server 
>All servers are different (physical, virtual) 
>Allows for determining how well sipXecs scales
sipxtest - Architecture 13 
>Simple install ‘yum install 
sipxtest’ 
>Pink – Files or Commands that test 
user can interact with. 
>Yellow – Generated files (you can 
edit these files, but know that 
sipxtest changes overwrite edits)
Load Test Numbers 
What do we do as part of build testing? 14 
>3 days of load testing for all major builds 
>15 calls per second 
>4 million calls total
sipXecs 4.6 Status 15 
>Running in house on production system since end of July 2012 
>Controlled release since August 2012 
>GA December 1, 2013 
>Update 1, February 5 
>Polycom Firmware Updates, New iptables capabilities, bug fixes. 
>Update 2, February 6 (small revert) 
>Update 3, March 13 
>fail2ban, bug fixes.
Roadmap – Near Term 16 
End of Q1 to End of Q2 
>openACD w/Supervisor & Agent Portals 
>Multiple Level Administrator 
>Multiple Time Zone 
>Polycom VVX 300/400 Support 
>sipXsbc 
>Session State Services – SSS (clean up RLS / XMPP link) 
>Improvements to HA (get rid of odd # of server requirement) 
>Call Queuing 
>Unite 2.0
Roadmap – Longer Term 17 
>openACD Reporting 
>Branch Office Solution 
>Will likely involve looking at User & System management differenly (i.e., more like a directory 
structure). 
>User Portal re-write 
>Browser based client, WebRTC. Zero Install Communications Solution. 
>New Admin GUI 
>Time to modernize a bit. The old interface is efficient but dated.
End 18
19 
>What is different as compared to traditional architectures? 
>What makes sipXecs an IT application? 
>High-level intro to sipXecs architecture (diagram) 
>Hardware independence: What does this mean? 
>Resulting deployment options: Focus on flexibility, global scale, redundancy 
>Redundancy, branch redundancy 
>Focus on our ‘secret sauce’. What makes this architecture better than all the 
others?
20 Status of the 4.6 Release 
>What is new? 
>Experience with 4.6 in the field 
>Test results and test methodology
21 Deployment Examples 
>Pick 2 to 3 examples. E.g. Axcess Finacial
22 Roadmap 
>Discuss near term and longer term roadmap 
>What is our goal?
23
24
3:00-4:00 sipXecs Architecture Moderator: Mike 
Participants: Douglas, Daniel, Joegen, Ciprian 
Engineering provided content: 
•Architecture overview (Mongo, SIP, XMPP, CFEngine high-level arch diagram). 
•Features and improvements delivered with 4.6 
•Test automation (how do we test?) 
•Status of 4.6 
•Deployment examples (distributed, virtualized, redundancy) 
•Roadmap – what to come next?
SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)

Mais conteúdo relacionado

Semelhante a SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)

Using Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at SplunkUsing Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at Splunk
Docker, Inc.
 
Software Engineering Practice - Configuration management
Software Engineering Practice - Configuration managementSoftware Engineering Practice - Configuration management
Software Engineering Practice - Configuration management
Radu_Negulescu
 
Maruthi_YH_resume
Maruthi_YH_resumeMaruthi_YH_resume
Maruthi_YH_resume
Maruthi YH
 
TechTalk_Cloud Performance Testing_0.6
TechTalk_Cloud Performance Testing_0.6TechTalk_Cloud Performance Testing_0.6
TechTalk_Cloud Performance Testing_0.6
Sravanthi N
 

Semelhante a SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS) (20)

SIPfoundry CoLab 2013 - Cloud virtualization of the sipXecs communications so...
SIPfoundry CoLab 2013 - Cloud virtualization of the sipXecs communications so...SIPfoundry CoLab 2013 - Cloud virtualization of the sipXecs communications so...
SIPfoundry CoLab 2013 - Cloud virtualization of the sipXecs communications so...
 
Recipes for the Oracle Cloud: Cooking with OneCloud in Your EPM Kitchen
Recipes for the Oracle Cloud: Cooking with OneCloud in Your EPM KitchenRecipes for the Oracle Cloud: Cooking with OneCloud in Your EPM Kitchen
Recipes for the Oracle Cloud: Cooking with OneCloud in Your EPM Kitchen
 
Load and Performance Testing for J2EE - Testing, monitoring and reporting usi...
Load and Performance Testing for J2EE - Testing, monitoring and reporting usi...Load and Performance Testing for J2EE - Testing, monitoring and reporting usi...
Load and Performance Testing for J2EE - Testing, monitoring and reporting usi...
 
ESP POC Findings
ESP POC FindingsESP POC Findings
ESP POC Findings
 
Renesas DevCon 2010: Starting a QT Application with Minimal Boot
Renesas DevCon 2010: Starting a QT Application with Minimal BootRenesas DevCon 2010: Starting a QT Application with Minimal Boot
Renesas DevCon 2010: Starting a QT Application with Minimal Boot
 
TestCon Vilnius 2017 - Stating the obvious
TestCon Vilnius 2017 - Stating the obviousTestCon Vilnius 2017 - Stating the obvious
TestCon Vilnius 2017 - Stating the obvious
 
What You Should Know About WebLogic Server 12c (12.2.1.2) #oow2015 #otntour2...
What You Should Know About WebLogic Server 12c (12.2.1.2)  #oow2015 #otntour2...What You Should Know About WebLogic Server 12c (12.2.1.2)  #oow2015 #otntour2...
What You Should Know About WebLogic Server 12c (12.2.1.2) #oow2015 #otntour2...
 
GWAVACon 2015: Microsoft MVP - Exchange Server Migrations & Updates
GWAVACon 2015: Microsoft MVP - Exchange Server Migrations & UpdatesGWAVACon 2015: Microsoft MVP - Exchange Server Migrations & Updates
GWAVACon 2015: Microsoft MVP - Exchange Server Migrations & Updates
 
Using Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at SplunkUsing Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at Splunk
 
Bots on guard of sdlc
Bots on guard of sdlcBots on guard of sdlc
Bots on guard of sdlc
 
Software Engineering Practice - Configuration management
Software Engineering Practice - Configuration managementSoftware Engineering Practice - Configuration management
Software Engineering Practice - Configuration management
 
Maruthi_YH_resume
Maruthi_YH_resumeMaruthi_YH_resume
Maruthi_YH_resume
 
TechTalk_Cloud Performance Testing_0.6
TechTalk_Cloud Performance Testing_0.6TechTalk_Cloud Performance Testing_0.6
TechTalk_Cloud Performance Testing_0.6
 
Continuous Delivery at Deli XL
Continuous Delivery at Deli XLContinuous Delivery at Deli XL
Continuous Delivery at Deli XL
 
Java 9 and Beyond
Java 9 and BeyondJava 9 and Beyond
Java 9 and Beyond
 
Oracle WebLogic: Feature Timeline from WLS9 to WLS 12c
Oracle WebLogic: Feature Timeline from WLS9 to WLS 12cOracle WebLogic: Feature Timeline from WLS9 to WLS 12c
Oracle WebLogic: Feature Timeline from WLS9 to WLS 12c
 
Knowledge sharing session Java 9
Knowledge sharing session Java 9Knowledge sharing session Java 9
Knowledge sharing session Java 9
 
GLOC 2014 NEOOUG - R12 Upgrade Downtime Reduction
GLOC 2014 NEOOUG - R12 Upgrade Downtime ReductionGLOC 2014 NEOOUG - R12 Upgrade Downtime Reduction
GLOC 2014 NEOOUG - R12 Upgrade Downtime Reduction
 
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
 
DockerCon Europe 2018 Monitoring & Logging Workshop
DockerCon Europe 2018 Monitoring & Logging WorkshopDockerCon Europe 2018 Monitoring & Logging Workshop
DockerCon Europe 2018 Monitoring & Logging Workshop
 

Mais de SIPfoundry

SIPfoundry CoLab 2013 - Communications-enabling Web and Mobile Applications
SIPfoundry CoLab 2013 - Communications-enabling Web and Mobile ApplicationsSIPfoundry CoLab 2013 - Communications-enabling Web and Mobile Applications
SIPfoundry CoLab 2013 - Communications-enabling Web and Mobile Applications
SIPfoundry
 

Mais de SIPfoundry (9)

SIPfoundry CoLab 2013 - Specific customer case studies for sipXecs and ROI an...
SIPfoundry CoLab 2013 - Specific customer case studies for sipXecs and ROI an...SIPfoundry CoLab 2013 - Specific customer case studies for sipXecs and ROI an...
SIPfoundry CoLab 2013 - Specific customer case studies for sipXecs and ROI an...
 
SIPfoundry CoLab 2013 - Video support and new system features and devices fo...
SIPfoundry CoLab 2013 -  Video support and new system features and devices fo...SIPfoundry CoLab 2013 -  Video support and new system features and devices fo...
SIPfoundry CoLab 2013 - Video support and new system features and devices fo...
 
SIPfoundry CoLab 2013 - WebRTC - A Communications Revolution
SIPfoundry CoLab 2013 - WebRTC - A Communications RevolutionSIPfoundry CoLab 2013 - WebRTC - A Communications Revolution
SIPfoundry CoLab 2013 - WebRTC - A Communications Revolution
 
SIPfoundry CoLab 2013 - Web Contact Center
SIPfoundry CoLab 2013 - Web Contact CenterSIPfoundry CoLab 2013 - Web Contact Center
SIPfoundry CoLab 2013 - Web Contact Center
 
SIPfoundry CoLab 2013 - Communications-enabling Web and Mobile Applications
SIPfoundry CoLab 2013 - Communications-enabling Web and Mobile ApplicationsSIPfoundry CoLab 2013 - Communications-enabling Web and Mobile Applications
SIPfoundry CoLab 2013 - Communications-enabling Web and Mobile Applications
 
SIPfoundry CoLab 2013 - Blair Pleasant Unified Communications Market Update
SIPfoundry CoLab 2013 - Blair Pleasant Unified Communications Market UpdateSIPfoundry CoLab 2013 - Blair Pleasant Unified Communications Market Update
SIPfoundry CoLab 2013 - Blair Pleasant Unified Communications Market Update
 
SIPfoundry CoLab 2013 - Solving the Bring Your Own Device BYOD problem with o...
SIPfoundry CoLab 2013 - Solving the Bring Your Own Device BYOD problem with o...SIPfoundry CoLab 2013 - Solving the Bring Your Own Device BYOD problem with o...
SIPfoundry CoLab 2013 - Solving the Bring Your Own Device BYOD problem with o...
 
SIPfoundry CoLab 2013 - Red Hat's Chris Stierle presents a case study about d...
SIPfoundry CoLab 2013 - Red Hat's Chris Stierle presents a case study about d...SIPfoundry CoLab 2013 - Red Hat's Chris Stierle presents a case study about d...
SIPfoundry CoLab 2013 - Red Hat's Chris Stierle presents a case study about d...
 
SIPfoundry Colab 2013 - Bill Haskins on the current state of the UC-enabled e...
SIPfoundry Colab 2013 - Bill Haskins on the current state of the UC-enabled e...SIPfoundry Colab 2013 - Bill Haskins on the current state of the UC-enabled e...
SIPfoundry Colab 2013 - Bill Haskins on the current state of the UC-enabled e...
 

Último

No Advance 8868886958 Chandigarh Call Girls , Indian Call Girls For Full Nigh...
No Advance 8868886958 Chandigarh Call Girls , Indian Call Girls For Full Nigh...No Advance 8868886958 Chandigarh Call Girls , Indian Call Girls For Full Nigh...
No Advance 8868886958 Chandigarh Call Girls , Indian Call Girls For Full Nigh...
Sheetaleventcompany
 
Chiulli_Aurora_Oman_Raffaele_Beowulf.pptx
Chiulli_Aurora_Oman_Raffaele_Beowulf.pptxChiulli_Aurora_Oman_Raffaele_Beowulf.pptx
Chiulli_Aurora_Oman_Raffaele_Beowulf.pptx
raffaeleoman
 
If this Giant Must Walk: A Manifesto for a New Nigeria
If this Giant Must Walk: A Manifesto for a New NigeriaIf this Giant Must Walk: A Manifesto for a New Nigeria
If this Giant Must Walk: A Manifesto for a New Nigeria
Kayode Fayemi
 

Último (20)

ICT role in 21st century education and it's challenges.pdf
ICT role in 21st century education and it's challenges.pdfICT role in 21st century education and it's challenges.pdf
ICT role in 21st century education and it's challenges.pdf
 
Re-membering the Bard: Revisiting The Compleat Wrks of Wllm Shkspr (Abridged)...
Re-membering the Bard: Revisiting The Compleat Wrks of Wllm Shkspr (Abridged)...Re-membering the Bard: Revisiting The Compleat Wrks of Wllm Shkspr (Abridged)...
Re-membering the Bard: Revisiting The Compleat Wrks of Wllm Shkspr (Abridged)...
 
Governance and Nation-Building in Nigeria: Some Reflections on Options for Po...
Governance and Nation-Building in Nigeria: Some Reflections on Options for Po...Governance and Nation-Building in Nigeria: Some Reflections on Options for Po...
Governance and Nation-Building in Nigeria: Some Reflections on Options for Po...
 
No Advance 8868886958 Chandigarh Call Girls , Indian Call Girls For Full Nigh...
No Advance 8868886958 Chandigarh Call Girls , Indian Call Girls For Full Nigh...No Advance 8868886958 Chandigarh Call Girls , Indian Call Girls For Full Nigh...
No Advance 8868886958 Chandigarh Call Girls , Indian Call Girls For Full Nigh...
 
Chiulli_Aurora_Oman_Raffaele_Beowulf.pptx
Chiulli_Aurora_Oman_Raffaele_Beowulf.pptxChiulli_Aurora_Oman_Raffaele_Beowulf.pptx
Chiulli_Aurora_Oman_Raffaele_Beowulf.pptx
 
Thirunelveli call girls Tamil escorts 7877702510
Thirunelveli call girls Tamil escorts 7877702510Thirunelveli call girls Tamil escorts 7877702510
Thirunelveli call girls Tamil escorts 7877702510
 
Introduction to Prompt Engineering (Focusing on ChatGPT)
Introduction to Prompt Engineering (Focusing on ChatGPT)Introduction to Prompt Engineering (Focusing on ChatGPT)
Introduction to Prompt Engineering (Focusing on ChatGPT)
 
Call Girl Number in Khar Mumbai📲 9892124323 💞 Full Night Enjoy
Call Girl Number in Khar Mumbai📲 9892124323 💞 Full Night EnjoyCall Girl Number in Khar Mumbai📲 9892124323 💞 Full Night Enjoy
Call Girl Number in Khar Mumbai📲 9892124323 💞 Full Night Enjoy
 
BDSM⚡Call Girls in Sector 97 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 97 Noida Escorts >༒8448380779 Escort ServiceBDSM⚡Call Girls in Sector 97 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 97 Noida Escorts >༒8448380779 Escort Service
 
Presentation on Engagement in Book Clubs
Presentation on Engagement in Book ClubsPresentation on Engagement in Book Clubs
Presentation on Engagement in Book Clubs
 
Busty Desi⚡Call Girls in Sector 51 Noida Escorts >༒8448380779 Escort Service-...
Busty Desi⚡Call Girls in Sector 51 Noida Escorts >༒8448380779 Escort Service-...Busty Desi⚡Call Girls in Sector 51 Noida Escorts >༒8448380779 Escort Service-...
Busty Desi⚡Call Girls in Sector 51 Noida Escorts >༒8448380779 Escort Service-...
 
Report Writing Webinar Training
Report Writing Webinar TrainingReport Writing Webinar Training
Report Writing Webinar Training
 
Air breathing and respiratory adaptations in diver animals
Air breathing and respiratory adaptations in diver animalsAir breathing and respiratory adaptations in diver animals
Air breathing and respiratory adaptations in diver animals
 
If this Giant Must Walk: A Manifesto for a New Nigeria
If this Giant Must Walk: A Manifesto for a New NigeriaIf this Giant Must Walk: A Manifesto for a New Nigeria
If this Giant Must Walk: A Manifesto for a New Nigeria
 
Dreaming Marissa Sánchez Music Video Treatment
Dreaming Marissa Sánchez Music Video TreatmentDreaming Marissa Sánchez Music Video Treatment
Dreaming Marissa Sánchez Music Video Treatment
 
ANCHORING SCRIPT FOR A CULTURAL EVENT.docx
ANCHORING SCRIPT FOR A CULTURAL EVENT.docxANCHORING SCRIPT FOR A CULTURAL EVENT.docx
ANCHORING SCRIPT FOR A CULTURAL EVENT.docx
 
My Presentation "In Your Hands" by Halle Bailey
My Presentation "In Your Hands" by Halle BaileyMy Presentation "In Your Hands" by Halle Bailey
My Presentation "In Your Hands" by Halle Bailey
 
BDSM⚡Call Girls in Sector 93 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 93 Noida Escorts >༒8448380779 Escort ServiceBDSM⚡Call Girls in Sector 93 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 93 Noida Escorts >༒8448380779 Escort Service
 
SaaStr Workshop Wednesday w/ Lucas Price, Yardstick
SaaStr Workshop Wednesday w/ Lucas Price, YardstickSaaStr Workshop Wednesday w/ Lucas Price, Yardstick
SaaStr Workshop Wednesday w/ Lucas Price, Yardstick
 
Dreaming Music Video Treatment _ Project & Portfolio III
Dreaming Music Video Treatment _ Project & Portfolio IIIDreaming Music Video Treatment _ Project & Portfolio III
Dreaming Music Video Treatment _ Project & Portfolio III
 

SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)

  • 1. sipXecs Architecture & Direction moderator: michael picher March 10, 2013 / Bentley University / Boston MA 1
  • 2. 2 Intro of Presenters Team >Load Testing & Build Team: >Douglas Hubler >Ciprian Hacman >SIP Core Team: >Joegen Baclor >Daniel Tacalau
  • 3. Agenda 3 >sipXecs 4.6 Architecture Overview >Testing Procedure >Test Automation >Status of 4.6 >Roadmap Near-Term / Long-Term
  • 7. Message Queuing Introduced 7 >Needed better interprocess communications >Lightweight >Robust >Evaluated several >Build? >RabbitMQ >ZeroMQ
  • 8. Message Queuing 8 Publisher / Subscriber
  • 9. Message Queuing 9 Dealer / Worker
  • 11. 11 Testing Procedure >Manual smoke test >Basic tests that catch major issues >Manual sanity test >Detailed tests for each feature >Manual regression tests >Complex tests for features that are added/changed >Automated Load Tests >Deployment for a week on our Production System >Dog fooding…
  • 12. Automated Load Tests sipxtest 12 >Placing and receiving calls is the core feature and we want this to be stable >Basic testing cannot predict >How a server behaves over time >How a server behaves under stress >Call Load tests helps address both problems >Allows determination the performance of a Certain Server >All servers are different (physical, virtual) >Allows for determining how well sipXecs scales
  • 13. sipxtest - Architecture 13 >Simple install ‘yum install sipxtest’ >Pink – Files or Commands that test user can interact with. >Yellow – Generated files (you can edit these files, but know that sipxtest changes overwrite edits)
  • 14. Load Test Numbers What do we do as part of build testing? 14 >3 days of load testing for all major builds >15 calls per second >4 million calls total
  • 15. sipXecs 4.6 Status 15 >Running in house on production system since end of July 2012 >Controlled release since August 2012 >GA December 1, 2013 >Update 1, February 5 >Polycom Firmware Updates, New iptables capabilities, bug fixes. >Update 2, February 6 (small revert) >Update 3, March 13 >fail2ban, bug fixes.
  • 16. Roadmap – Near Term 16 End of Q1 to End of Q2 >openACD w/Supervisor & Agent Portals >Multiple Level Administrator >Multiple Time Zone >Polycom VVX 300/400 Support >sipXsbc >Session State Services – SSS (clean up RLS / XMPP link) >Improvements to HA (get rid of odd # of server requirement) >Call Queuing >Unite 2.0
  • 17. Roadmap – Longer Term 17 >openACD Reporting >Branch Office Solution >Will likely involve looking at User & System management differenly (i.e., more like a directory structure). >User Portal re-write >Browser based client, WebRTC. Zero Install Communications Solution. >New Admin GUI >Time to modernize a bit. The old interface is efficient but dated.
  • 19. 19 >What is different as compared to traditional architectures? >What makes sipXecs an IT application? >High-level intro to sipXecs architecture (diagram) >Hardware independence: What does this mean? >Resulting deployment options: Focus on flexibility, global scale, redundancy >Redundancy, branch redundancy >Focus on our ‘secret sauce’. What makes this architecture better than all the others?
  • 20. 20 Status of the 4.6 Release >What is new? >Experience with 4.6 in the field >Test results and test methodology
  • 21. 21 Deployment Examples >Pick 2 to 3 examples. E.g. Axcess Finacial
  • 22. 22 Roadmap >Discuss near term and longer term roadmap >What is our goal?
  • 23. 23
  • 24. 24
  • 25. 3:00-4:00 sipXecs Architecture Moderator: Mike Participants: Douglas, Daniel, Joegen, Ciprian Engineering provided content: •Architecture overview (Mongo, SIP, XMPP, CFEngine high-level arch diagram). •Features and improvements delivered with 4.6 •Test automation (how do we test?) •Status of 4.6 •Deployment examples (distributed, virtualized, redundancy) •Roadmap – what to come next?

Notas do Editor

  1. I'm going to show you how mongo is integration into this open source project. Along the way I'm also going showing features of this project. I think it's really good project, I'm really proud of it and it's hard to think of a company that could not use it. Each box represents a running daemon each arrow is a network connection. This architecture is of course highly scalable and fairly unique to communications systems, open source and otherwise. What makes this system unique is that this entire system can be installed in minutes span multiple machines includes monitoring, security, backup, restore. highly customizable, partly due to architecture
  2. Hyper-focused on call redundancy at server level. When replication broke "split brain" - half calls went to voicemail Lot's of custom replication logic Recovery took a while with a lot of data and long down times.
  3. Much cleaner. We'll be getting rid of postgres eventually.
  4. publisher / watcher means that multiple apps could receive same event
  5. dealer / worker means that only one app can receive and consume the event