Presentation from OpenCms Days 2014. Running OpenCms within a private cloud-environment using the features of dynamic up- and downscaling is a challenge. Our expandable service architecture around OpenCms allows us to run the whole system without noticeable outages. We demonstrate how we have overcome the tight coupling of OCEE to achieve this goal and show which approaches we pursue to get OpenCms enterprise- & cloud-ready.

1. Interactive Thinking
OpenCms Cloud setup with the FI-TS
November 4, 2014
OpenCms Days 2014

2. Talking about
»Introduction
»A brieflookback
»Todaysclustersetup
»Cloud-readiness
»Dynamic Up-& Down-Scaling
2 10.11.2014 OEV Interactive Thinking

3. Introduction
Preconditions + Architecture

4. Preconditions
»High Availability(24x7, 99.5%)
»60 Sites (40 https), 250.000 Resources, 25GB
»8.000.000+ Requests/dayduringbusinesshours
»User-& Group-Management via LDAP (2000 Users, 250 Groups)
»~25 WebappsbesideOpenCms
»Connection to20 customerbackendsvia private network
»Workplaceonlyaccessiblevia private network
4 10.11.2014 OEV Interactive Thinking

5. Architecture (detailed)
5 10.11.2014 OEV Interactive Thinking
Internet
Loadbalancer+ SSL
Content
Database
OpenCms
Apache
App
Data
Service
ESB
Backend Gateway
LDAP
Static
exports
Service
Service Fast Lane
Service
Backend Fast Lane

6. Architecture (Bird‘s-Eye-View)
6 10.11.2014 OEV Interactive Thinking
Internet
Loadbalancer+ SSL
MySQL
OpenCms
Apache
MySQL
OpenCms
Apache
Backend Gateway
Customers
MySQL
OpenCms
Apache
publish
…
Workplace access
LDAP
Services
Services

7. Star of the Show
7 10.11.2014 OEV Interactive Thinking
Internet
Loadbalancer+ SSL
MySQL
OpenCms
Apache
MySQL
OpenCms
Apache
Backend Gateway
Customers
MySQL
OpenCms
Apache
publish
…
Workplace access
LDAP
Services
Services
OCEE

8. OpenCms Enterprise Extensions
»Accelerator: Reducesdatabasecalls
»VFS Doctor: Simplifiesdatabasemaintainingforwitnessedorduplicateresources
»LDAP Connector: self-explanatory
»Cluster Manager: manage Workplace-andNon-Workplace-Servers
»Replication: publishoverserverboundaries
8 10.11.2014 OEV Interactive Thinking
Combat tested and approved
Since 2006

9. The Past
-2013

10. The former provider
»Apache andJBosson different servers
»OpenCms7.5 andServices withinsame JVM
»Dedicatedhardware
»Contractabandonedat theend of2013
10 10.11.2014 OEV Interactive Thinking
Internet
Loadbalancer+ SSL
MySQL
JBoss
OpenCms
Services
Apache
Backend Gateway
MySQL
JBoss
OpenCms
Services
Apache

11. In search for a new provider…
… wefound:
»State-of-Art Datacenter
»Connectivity toall ourcustomers
»Cloud-System (IaaS)
»DetailedOpenCms-knowledge
»Long time developercollaboration
»Cloud experience
11 10.11.2014 OEV Interactive Thinking
Combining the best of two Worlds
The Content-Delivery-Triplet

12. 12 10.11.2014 OEV Interactive Thinking
Who is who in the Cloud-Ecosystem
Hardware
IaaS
PaaS
SaaS
BPaaS

13. Today
2014

14. After the migration
»Apache andJBosson thesame servers
»OpenCms7.5 andServices withinsame JVM
»VirtualizedEnvironment
»Network storage
»Self-Service-Management
»Application-andSystem-Monitoring
»Enhanced Alerting
14 10.11.2014 OEV Interactive Thinking
Internet
Loadbalancer+ SSL
MySQL
JBoss
OpenCms
Services
Apache
Backend Gateway
MySQL
JBoss
OpenCms
Services
Apache
Management & Monitoring

15. 15 10.11.2014 OEV Interactive Thinking
Nimbus

16. 16 10.11.2014 OEV Interactive Thinking
Zabbix

17. 17 10.11.2014 OEV Interactive Thinking
AppDynamics

18. 18 10.11.2014 OEV Interactive Thinking
Splunk

19. Experiences after 10 Month (PaaS)
»Setting up a 3rd cluster member within 24h
»Needs reconfiguring the workplace-server
»DB-cloning is the bootleneck
»+1 OCEE-License required
»Running perfect 
19 10.11.2014 OEV Interactive Thinking
Internet
Loadbalancer+ SSL
MySQL
JBoss
OpenCms
Services
Apache
Backend Gateway
MySQL
JBoss
OpenCms
Services
Apache
Management & Monitoring
MySQL
JBoss
OpenCms
Services
Apache

20. Experiences after 10 Month (IaaS)
»ElasticLoadbalancerbasedon nginx& haproxy
»Fast
»Setup sharedwithothercustomers
»SSL-Cipher-Ordering
»Header crippling
»noaccesstologfiles
»Self-Service-GUI canonlyhandle1 certificateper HTTPS => NoSNI
»Limited HTTP/1.1 support
»POST chunkendencoding
20 10.11.2014 OEV Interactive Thinking
Internet
Loadbalancer+ SSL
nginx
SSL-Termination
haproxy
Loadbalancing

21. Experiences after 10 Month (IaaS)
»Server sizing only as fixed packages
»S= 1CPU/4GB RAM
»M= 2CPUs/8GB RAM
»L= 4CPUs/16GB RAM
»XL= 8CPUs/32GB RAM
»Resizing requires Backup and Restore
»Only for Operating System (~16GB)
»Data-Storage untouched
21 10.11.2014 OEV Interactive Thinking

22. The Future
November 3, 2014 12:00 -

23. … arewereadyforit?
»Adapttoworkloadchangesusingperformanceindicators
»Anomaly-detection
»Tracking & tracingmicroservices
23 10.11.2014 OEV Interactive Thinking
Now that we have a cloud…
from
to
Where is it?

24. Usingautomationtoolslike puppet, cheforansiblefor
»Networking
»OS
»Apache
»Tomcat
»MySQL
24 10.11.2014 OEV Interactive Thinking
Setting up the base system

25. Architecture (Bird‘s-Eye-View)
25 10.11.2014 OEV Interactive Thinking
Internet
Loadbalancer+ SSL
MySQL
OpenCms
Apache
MySQL
OpenCms
Apache
Backend Gateway
Customers
MySQL
OpenCms
Apache
publish
…
Workplace access
LDAP
Services
Services

26. Withautomationtoolsyoucan
»DeployOpenCms
»DeployOCEE
26 10.11.2014 OEV Interactive Thinking
Setting up OpenCms + OCEE
But youcan‘t
»Aquirea licensekeytoactivateOCEE
»Dependson server.ethernet.address
»server.ethernet.addressmust beunique
Possibleworkarounds
»Crack thelicensemanager=> evil
»Stockpile licenses=> expensive
»Call Alkacon, installlicenseafter assignment=> slow
»@Alkacon: providea licensingwebservice, please=> nicetohave

27. Site Management
Workplaceonlyaccessiblevia private network:
»Usinga DNS-SchemelikeDevelopment http://d.cloud.privateIntegrationhttp://i.cloud.privateProductionhttp://r.cloud.private (r = Redaktion, germanforeditorsstaff)
Eachcustomerisprovidedwitha shorthandsymbolicname:
»OEVOnline Dienste = oevVersicherungskammer Bayern = vkb…
Symbolicname= OpenCmsSitename= Servername = Secure Servername:
»/sites/oevhttp://oev.d.cloud.privatehttps://oev.d.cloud.private/sites/vkbhttp://vkb.d.cloud.privatehttps://oev.d.cloud.private…
27 10.11.2014 OEV Interactive Thinking

28. Site Management
Symbolicname= OpenCmsSitename= prefixforServername
»/sites/oevhttp://oev.d.cloud.privatehttps://oev.d.cloud.private/sites/vkbhttp://vkb.d.cloud.privatehttps://oev.d.cloud.private
»Easy tosetupwithNameserver-Delegation
28 10.11.2014 OEV Interactive Thinking
Internet
DNS
Cloud
DNS
Client
nslookup
*.cloud.private
Everything else

29. Site Management
Doesnot workforproductionsystem, because
»/sites/oevhttps://www.oev-online.de≠https://oev.r.cloud.private/sites/vkbhttps://www.vkb.de≠https://vkb.r.cloud.private
NosecureServer-AliasesforHTTPS
»/sites/oevhttps://www.oev-online.de≠https://www.oevonline.de/sites/vkbhttps://www.vkb.de≠https://www.versicherungskammer.de
»Site Manager in OpenCms9 hasnooptiontodistinguishbetweenworkplace-andnon-workplace-servername
»Thereisonlyonesecureserverallowed=> noSNI
»Site-Management withinCluster Manager canonlyremovesitesfromnon-workplace- servers orclone(via compare) tothenon-workplace-servers
»Changingsitesrequiresreconfigurationoffrontsidewebservers
29 10.11.2014 OEV Interactive Thinking

30. Site Management (Workaround)
All changesin Site-Management arewrittentoopencms-system.xml immediately
»Watch thefileforchanges
»usingLinux inotify-interface
»useinotifywaitwithinshellscripts
30 10.11.2014 OEV Interactive Thinking
#!/bin/sh
while inotifywait -e modify opencms-system.xml; do
# do something useful
done

31. Site Management (Workaround)
do somethinguseful(examples)
»Create opencms-system.xml fornon-workplace-servers anddistributethem
»Usea servername-lookup-tablewithsitenameaskey
»Create fragmentsforApacheshttpd.conf
»Usingxsltotransformxml
»Populatingmap-Files forApache RewriteMapsdoesnot requirerestarting
»Apache 2.4 mod_macroisyourfriend
»Check-in changestoconfigurationmanagement
»Analyse changes
»automagicalypopulateyourconfigurationtemplates
»Distributetheneededconfigs
»Restartservicesasneeded(regardingavailabilityrequirements)
31 10.11.2014 OEV Interactive Thinking

32. Up-& Down-Scaling
Metricsretrievedfrom
»Zabbix
»Apache mod_status
»MySQL stats
»JMX
»JVM
»Tomcat
»OpenCms(extendedbyownJMX-lib)
»AppDynamics
Thresholds=> WIP
Trigger scaling, but don‘tbuildservers
32 10.11.2014 OEV Interactive Thinking

33. Conlusion

34. Conclusion
»Mass-hosting is possible with OpenCms
»Much easier with it‘s little helper: OCEE
»Unified naming for clarity
»External toolset needed
»Experienced knowledge of the inner workings
34 10.11.2014 OEV Interactive Thinking
Cloudreadywithminor drawbacks

35. OEV Online Dienste GmbH
Hansaallee 183
40549 Düsseldorf
oev-online.de
Tel.: 0211 / 4554-4287
mlinkenheil@oev.de
Thanks
For Your Attention.
Michael Linkenheil