This talk is about how to build a cluster to run a python or ruby (ruby on rails) application. We'll have a look at how the procedure of building such a cluster could look like and what you should take into consideration. We'll look at issues like: datacenter, networking, load balancing, storage, database replication, ....

1. Enterprise Hosting
for Ruby and Python
Julian Fischer
fischer@enterprise-rails.de
http://www.enterprise-rails.de

2. Introduction

3. Introduction
About me

4. About me
Julian Fischer
‣Twitter: http://www.twitter.com/railshoster
‣E-Mail: fischer@enterprise-rails.de

5. About me
Julian Fischer
‣Twitter: http://www.twitter.com/railshoster
‣E-Mail: fischer@enterprise-rails.de
‣ CEO of Avarteq GmbH

6. About me
Julian Fischer
‣Twitter: http://www.twitter.com/railshoster
‣E-Mail: fischer@enterprise-rails.de
‣ CEO of Avarteq GmbH
‣ Lecturer „Ruby on Rails“ @ HTWdS

7. About me
Julian Fischer
‣Twitter: http://www.twitter.com/railshoster
‣E-Mail: fischer@enterprise-rails.de
‣ CEO of Avarteq GmbH
‣ Lecturer „Ruby on Rails“ @ HTWdS
‣ Ruby und Ruby on Rails programmer

8. About me
Julian Fischer
‣Twitter: http://www.twitter.com/railshoster
‣E-Mail: fischer@enterprise-rails.de
‣ CEO of Avarteq GmbH
‣ Lecturer „Ruby on Rails“ @ HTWdS
‣ Ruby und Ruby on Rails programmer
‣ Entperise-Rails.de - Head of Hosting

9. Introduction
About Avarteq GmbH

10. About Avarteq GmbH
http://www.twitter.com/railshoster

11. About Avarteq GmbH
‣ Founded in Nov. 2008
from two existing companies.
http://www.twitter.com/railshoster

12. About Avarteq GmbH
‣ Founded in Nov. 2008
from two existing companies.
‣ Involvment of Key-Systems GmbH
manages ~2,5 * 10^6 domains for customers of 200+ countries.
http://www.twitter.com/railshoster

13. About Avarteq GmbH
‣ Founded in Nov. 2008
from two existing companies.
‣ Involvment of Key-Systems GmbH
manages ~2,5 * 10^6 domains for customers of 200+ countries.
‣ Team size: 14 people
8 full-time, 6 part-time/freelancer
http://www.twitter.com/railshoster

14. Introduction
Portfolio

15. About Avarteq GmbH
http://www.twitter.com/railshoster

16. About Avarteq GmbH
‣ Covers all stages of web development
http://www.twitter.com/railshoster

17. About Avarteq GmbH
‣ Covers all stages of web development
‣ Consulting
http://www.twitter.com/railshoster

18. About Avarteq GmbH
‣ Covers all stages of web development
‣ Consulting
‣ Conceptual ~ and screen design
http://www.twitter.com/railshoster

19. About Avarteq GmbH
‣ Covers all stages of web development
‣ Consulting
‣ Conceptual ~ and screen design
‣ Ruby&onplace. development.
In house in
Rails
http://www.twitter.com/railshoster

20. About Avarteq GmbH
‣ Covers all stages of web development
‣ Consulting
‣ Conceptual ~ and screen design
‣ Ruby&onplace. development.
In house in
Rails
‣ Ruby on Rails
hosting/servers/clusters
RailsHoster.de - Enterprise-Rails.de
http://www.twitter.com/railshoster

21. Enterprise Hosting

22. Enterprise Hosting
Situation

23. EH - Basic Idea
http://www.twitter.com/railshoster

24. EH - Basic Idea
‣ Given
http://www.twitter.com/railshoster

25. EH - Basic Idea
‣ Given
‣ Python / Ruby / Rails App
http://www.twitter.com/railshoster

26. EH - Basic Idea
‣ Given
‣ Python / Ruby / Rails App
‣ Needed
http://www.twitter.com/railshoster

27. EH - Basic Idea
‣ Given
‣ Python / Ruby / Rails App
‣ Needed
‣ Max. availability ideal: 100% uptime
http://www.twitter.com/railshoster

28. EH - Basic Idea
‣ Given
‣ Python / Ruby / Rails App
‣ Needed
‣ Max. availability ideal: 100% uptime
‣ Acceptable performance even for peaks, e.g.
<= 200 ms response time / request
http://www.twitter.com/railshoster

29. Enterprise Hosting
Obstacles

30. EH - Obstacles
http://www.twitter.com/railshoster

31. EH - Obstacles
‣ Single server = a lot of SPOFs
http://www.twitter.com/railshoster

32. EH - Obstacles
‣ Single server = a lot of SPOFs
‣ Multiple servers = a lot of work
http://www.twitter.com/railshoster

33. EH - Obstacles
‣ Single server = a lot of SPOFs
‣ Multiple servers = a lot of work
‣ Eliminating SPOFs is not easy/possible
for every service
http://www.twitter.com/railshoster

34. How to build a
fancy & rockin‘
cluster?

35. Cluster Building Blocks
Monitoring
DBApp
Backups
Storage
MA/SL
LB
Enterprise
Cluster

36. Cluster Building Blocks
Monitoring
DB MA/SL
Backups
Storage
LB
App
Enterprise
Cluster

37. Cluster Building Blocks
Monitoring
DB MA/SL
Backups
Storage
App
LB
Enterprise
Cluster

38. Cluster Building Blocks
Monitoring
DB MA/SL
Backups
Storage
App
LB
Enterprise
Cluster

39. Cluster Building Blocks
Monitoring
Backups
DBApp
Storage
MA/SL
LB
Enterprise
Cluster

40. Cluster Building Blocks
Monitoring
DBApp
Backups
Storage
MA/SL
LB
Enterprise
Cluster

41. Cluster Building Blocks
Monitoring
DBApp
Backups
Storage
MA/SL
LB
Enterprise
Cluster

42. Cluster Building Blocks
Monitoring
DBApp
Backups
Storage
MA/SL
LB
Enterprise
Cluster
✓

43. maybe it‘s not that easy :)

44. How to build a cluster
Requirements Analysis

45. How to build a cluster
http://www.twitter.com/railshoster

46. How to build a cluster
‣ desired response time which again depends on several
factors
http://www.twitter.com/railshoster

47. How to build a cluster
‣ desired response time which again depends on several
factors
‣ average load e.g. on a cloudy sunday afternoon
http://www.twitter.com/railshoster

48. How to build a cluster
‣ desired response time which again depends on several
factors
‣ average load e.g. on a cloudy sunday afternoon
‣ peak load e.g. after being a new trend on twitter
http://www.twitter.com/railshoster

49. How to build a cluster
‣ desired response time which again depends on several
factors
‣ average load e.g. on a cloudy sunday afternoon
‣ peak load e.g. after being a new trend on twitter
‣ ...
http://www.twitter.com/railshoster

50. How to build a cluster
http://www.twitter.com/railshoster

51. How to build a cluster
‣ which components are present? search server,
background processing, key/value store, replicating rdbms, ...
http://www.twitter.com/railshoster

52. How to build a cluster
‣ which components are present? search server,
background processing, key/value store, replicating rdbms, ...
‣ apptired programmers? & quality coded by super geeks or a
lot of
performance
http://www.twitter.com/railshoster

53. How to build a cluster
http://www.twitter.com/railshoster

54. How to build a cluster
‣ Do it yourself or full management?
http://www.twitter.com/railshoster

55. How to build a cluster
‣ Do it yourself or full management?
‣ admin know how present?
Is your team able to care of things like
http://www.twitter.com/railshoster

56. How to build a cluster
‣ Do it yourself or full management?
‣ admin know how present?
Is your team able to care of things like
‣ os updates
http://www.twitter.com/railshoster

57. How to build a cluster
‣ Do it yourself or full management?
‣ admin know how present?
Is your team able to care of things like
‣ os updates
‣ web server & load balancer setup
http://www.twitter.com/railshoster

58. How to build a cluster
‣ Do it yourself or full management?
‣ admin know how present?
Is your team able to care of things like
‣ os updates
‣ web server & load balancer setup
‣ db configuration & tuning
http://www.twitter.com/railshoster

59. How to build a cluster
‣ Do it yourself or full management?
‣ admin know how present?
Is your team able to care of things like
‣ os updates
‣ web server & load balancer setup
‣ db configuration & tuning
‣ monitoring, ...
http://www.twitter.com/railshoster

60. How to build a cluster
http://www.twitter.com/railshoster

61. How to build a cluster
‣ Willing to fight with a large number of
servers a lot of programmers don‘t want to do both: programming
& hosting
http://www.twitter.com/railshoster

62. How to build a cluster
‣ Willing to fight with a large number of
servers a lot of programmers don‘t want to do both: programming
& hosting
‣ Available budget
http://www.twitter.com/railshoster

63. How to build a cluster
‣ Willing to fight with a large number of
servers a lot of programmers don‘t want to do both: programming
& hosting
‣ Available budget
‣ more servers ... more reliability, more performance
http://www.twitter.com/railshoster

64. How to build a cluster
‣ Willing to fight with a large number of
servers a lot of programmers don‘t want to do both: programming
& hosting
‣ Available budget
‣ more servers ... more reliability, more performance
‣ Many other questions ...
http://www.twitter.com/railshoster

65. How to build a cluster
Choose your HW platform

66. Choose HW plattform
http://www.twitter.com/railshoster

67. Choose HW plattform
‣ Physical HW low budget or real server HW?
http://www.twitter.com/railshoster

68. Choose HW plattform
‣ Physical HW low budget or real server HW?
‣ Virtual HW Amazon EC2, Joyent, EY, ...
http://www.twitter.com/railshoster

69. Choose HW plattform
‣ Physical HW low budget or real server HW?
‣ Virtual HW Amazon EC2, Joyent, EY, ...
‣ Assume we use physical HW ...
http://www.twitter.com/railshoster

70. Choose HW plattform
http://www.twitter.com/railshoster

71. Choose HW plattform
‣ Compare TCO and performance/€
http://www.twitter.com/railshoster

72. Choose HW plattform
‣ Compare TCO and performance/€
‣ When is your next scale out?
http://www.twitter.com/railshoster

73. Choose HW plattform
‣ Compare TCO and performance/€
‣ When is your next scale out?
‣ Do you really need to scale within
hours? ... and is it worth to pay a lot of extra money?
Estimate your resource consumption grow rate e.g. using Munin.
http://www.twitter.com/railshoster

74. How to build a cluster
Cluster dimensions

75. Cluster dimensions
http://www.twitter.com/railshoster

76. Cluster dimensions
‣ How many servers needed? 2, 4, 6, 10, ...
http://www.twitter.com/railshoster

77. Cluster dimensions
‣ How many servers needed? 2, 4, 6, 10, ...
‣ Which systems components will be there?
App, MA DB, SL DB, BG-Jobs, Search-Server, Memcached, Storage, ...
http://www.twitter.com/railshoster

78. Cluster dimensions
‣ How many servers needed? 2, 4, 6, 10, ...
‣ Which systems components will be there?
App, MA DB, SL DB, BG-Jobs, Search-Server, Memcached, Storage, ...
‣ Which HW will be most suitable for each
individual role? DB Server = a lot of fast hdds, BG-Kob
Server = a lot of cpus, ...
http://www.twitter.com/railshoster

79. How to build a cluster
Cluster design

80. Cluster design
http://www.twitter.com/railshoster

81. Cluster design
‣ How to distribute components to servers?
http://www.twitter.com/railshoster

82. Cluster design
‣ How to distribute components to servers?
‣ App Server & DB Server together?
http://www.twitter.com/railshoster

83. Cluster design
‣ How to distribute components to servers?
‣ App Server & DB Server together?
‣ Dedicated DB Server?
http://www.twitter.com/railshoster

84. Cluster design
http://www.twitter.com/railshoster

85. Cluster design
‣ Where to put the search server? Sphinx,
Solr, ...
http://www.twitter.com/railshoster

86. Cluster design
‣ Where to put the search server? Sphinx,
Solr, ...
‣ Where to run background jobs?
http://www.twitter.com/railshoster

87. Cluster design
‣ Where to put the search server? Sphinx,
Solr, ...
‣ Where to run background jobs?
‣ Where to run message queues? See talk of
Paolo Negri.
http://www.twitter.com/railshoster

88. How to build a cluster
Cluster building blocks

89. Cluster building blocks
Network

90. Network
http://www.twitter.com/railshoster

91. Network
‣ Connection between cluster nodes
http://www.twitter.com/railshoster

92. Network
‣ Connection between cluster nodes
‣ Use a private (!) physical network
http://www.twitter.com/railshoster

93. Network
‣ Connection between cluster nodes
‣ Use a private (!) physical network
‣ Minimize communication latency distribution
might cost performance for low load scenarios
http://www.twitter.com/railshoster

94. Network
‣ Connection between cluster nodes
‣ Use a private (!) physical network
‣ Minimize communication latency distribution
might cost performance for low load scenarios
‣ >= 1even within the same datacenter hosters limit bandwith between
servers
GBit/s be aware that some
http://www.twitter.com/railshoster

95. Cluster building blocks
Application Server

96. Application Server
http://www.twitter.com/railshoster

97. Application Server
‣ Ruby / Ruby on Rails
http://www.twitter.com/railshoster

98. Application Server
‣ Ruby / Ruby on Rails
‣ Passenger our favorite.
http://www.twitter.com/railshoster

99. Application Server
‣ Ruby / Ruby on Rails
‣ Passenger our favorite.
‣ LB &&interesting. Thin and mongrel do their job.
Unicorn is
(Unicorn || Thin || Mongrel)
http://www.twitter.com/railshoster

100. Application Server
‣ Ruby / Ruby on Rails
‣ Passenger our favorite.
‣ LB &&interesting. Thin and mongrel do their job.
Unicorn is
(Unicorn || Thin || Mongrel)
‣ JRuby with Glassfish-Gem or servlet containers
like Glassfish-Server, Jetty, Tomacat, ...
http://www.twitter.com/railshoster

101. Application Server
‣ Ruby / Ruby on Rails
‣ Passenger our favorite.
‣ LB &&interesting. Thin and mongrel do their job.
Unicorn is
(Unicorn || Thin || Mongrel)
‣ JRuby with Glassfish-Gem or servlet containers
like Glassfish-Server, Jetty, Tomacat, ...
‣ others CGI, fastCGI, SCGI, ...
http://www.twitter.com/railshoster

102. Application Server
http://www.twitter.com/railshoster

103. Application Server
‣ Python
http://www.twitter.com/railshoster

104. Application Server
‣ Python
‣ mod_python run Python interpreter inside apache.
http://www.twitter.com/railshoster

105. Application Server
‣ Python
‣ mod_python run Python interpreter inside apache.
‣ mod_wsgi better speed & memory usage than mod_python.
http://www.twitter.com/railshoster

106. Application Server
‣ Python
‣ mod_python run Python interpreter inside apache.
‣ mod_wsgi better speed & memory usage than mod_python.
‣ others CGI, fastCGI, SCGI, ...
http://www.twitter.com/railshoster

107. Cluster building blocks
Load Balancer

108. Load Balancer
http://www.twitter.com/railshoster

109. Load Balancer
‣ Most likely not the bottleneck
http://www.twitter.com/railshoster

110. Load Balancer
‣ Most likely not the bottleneck
‣ Often a single point of failure (SPOF)
http://www.twitter.com/railshoster

111. Load Balancer
‣ Most likely not the bottleneck
‣ Often a single point of failure (SPOF)
‣ Two are better than one!
http://www.twitter.com/railshoster

112. Load Balancer
‣ Most likely not the bottleneck
‣ Often a single point of failure (SPOF)
‣ Two are better than one!
‣ What to do if one fails?
http://www.twitter.com/railshoster

113. Load Balancer
‣ Most likely not the bottleneck
‣ Often a single point of failure (SPOF)
‣ Two are better than one!
‣ What to do if one fails?
‣ Setup automatic IP failover not possible in every
datacenter
http://www.twitter.com/railshoster

114. Load Balancer
http://www.twitter.com/railshoster

115. Load Balancer
‣ Open Source Software LBs
http://www.twitter.com/railshoster

116. Load Balancer
‣ Open Source Software LBs
‣ Apache, NGINX, Pound, HA Proxy, ...
http://www.twitter.com/railshoster

117. Load Balancer
‣ Open Source Software LBs
‣ Apache, NGINX, Pound, HA Proxy, ...
‣ LB Appliances BalanceNG, ...
http://www.twitter.com/railshoster

118. Cluster building blocks
Storage

119. Storage
http://www.twitter.com/railshoster

120. Storage
‣ Use a shared filesystem to store user
uploads such as profile pictures, etc.
http://www.twitter.com/railshoster

121. Storage
‣ Use a shared filesystem to store user
uploads such as profile pictures, etc.
‣ Some storage technologies
http://www.twitter.com/railshoster

122. Storage
‣ Use a shared filesystem to store user
uploads such as profile pictures, etc.
‣ Some storage technologies
‣ NFS, sync or async, often a SPOF, lot of overhead
http://www.twitter.com/railshoster

123. Storage
‣ Use a shared filesystem to store user
uploads such as profile pictures, etc.
‣ Some storage technologies
‣ NFS, sync or async, often a SPOF, lot of overhead
‣ RSync, async, replication lag
http://www.twitter.com/railshoster

124. Storage
‣ Use a shared filesystem to store user
uploads such as profile pictures, etc.
‣ Some storage technologies
‣ NFS, sync or async, often a SPOF, lot of overhead
‣ RSync, async, replication lag
‣ iSCSI && (GFS || OCFS2), incredibly fast
http://www.twitter.com/railshoster

125. Storage
iSCSI

126. iSCSI Storage
http://www.twitter.com/railshoster

127. iSCSI Storage
‣ iSCSI = SCSI over TCP
http://www.twitter.com/railshoster

128. iSCSI Storage
‣ iSCSI = SCSI over TCP
‣ Exports a blockdevice
http://www.twitter.com/railshoster

129. iSCSI Storage
‣ iSCSI = SCSI over TCP
‣ Exports a blockdevice
‣ iSCSI = less overhead than NFS
http://www.twitter.com/railshoster

130. iSCSI Storage
‣ iSCSI = SCSI over TCP
‣ Exports a blockdevice
‣ iSCSI = less overhead than NFS
‣ Need to use a distributed FS like
http://www.twitter.com/railshoster

131. iSCSI Storage
‣ iSCSI = SCSI over TCP
‣ Exports a blockdevice
‣ iSCSI = less overhead than NFS
‣ Need to use a distributed FS like
‣ GFS
http://www.twitter.com/railshoster

132. iSCSI Storage
‣ iSCSI = SCSI over TCP
‣ Exports a blockdevice
‣ iSCSI = less overhead than NFS
‣ Need to use a distributed FS like
‣ GFS
‣ OCFS2
http://www.twitter.com/railshoster

133. Storage
iSCSI Storage Appliance

134. iSCSI Storage Appliance
http://www.twitter.com/railshoster

135. iSCSI Storage Appliance
‣ Think about using a storage appliance, e.g.
NetApp
http://www.twitter.com/railshoster

136. iSCSI Storage Appliance
‣ Think about using a storage appliance, e.g.
NetApp
‣ Has redundant HW
http://www.twitter.com/railshoster

137. iSCSI Storage Appliance
‣ Think about using a storage appliance, e.g.
NetApp
‣ Has redundant HW
‣ Out of the box solution
http://www.twitter.com/railshoster

138. iSCSI Storage Appliance
‣ Think about using a storage appliance, e.g.
NetApp
‣ Has redundant HW
‣ Out of the box solution
‣ Very low maintenance
http://www.twitter.com/railshoster

139. iSCSI Storage Appliance
‣ Think about using a storage appliance, e.g.
NetApp
‣ Has redundant HW
‣ Out of the box solution
‣ Very low maintenance
‣ Connect each app server via dedicated
GBit LAN connections
http://www.twitter.com/railshoster

140. Cluster building blocks
Database

141. Database
http://www.twitter.com/railshoster

142. Database
‣ RDBMS? There are alternatives CouchDB, MongoDB, ...
http://www.twitter.com/railshoster

143. Database
‣ RDBMS? There are alternatives CouchDB, MongoDB, ...
‣ Most likely MySQL, Postgres, ...
http://www.twitter.com/railshoster

144. Database
‣ RDBMS? There are alternatives CouchDB, MongoDB, ...
‣ Most likely MySQL, Postgres, ...
‣ Horizontal DB scale out is ugly
http://www.twitter.com/railshoster

145. Database
‣ RDBMS? There are alternatives CouchDB, MongoDB, ...
‣ Most likely MySQL, Postgres, ...
‣ Horizontal DB scale out is ugly
‣ Scaleout vertically, first
http://www.twitter.com/railshoster

146. Database
‣ RDBMS? There are alternatives CouchDB, MongoDB, ...
‣ Most likely MySQL, Postgres, ...
‣ Horizontal DB scale out is ugly
‣ Scaleout vertically, first
‣ CPU++, RAM++, a lot of fast HDDs
http://www.twitter.com/railshoster

147. Horizontal MySQL
scaling?

148. Database
MySQL NDB Cluster

149. Database
http://www.twitter.com/railshoster

150. Database
‣ NDB MySQL Cluster
http://www.twitter.com/railshoster

151. Database
‣ NDB MySQL Cluster
‣ Features look performance, ... automatic failover,
transactions, good write
very nice
http://www.twitter.com/railshoster

152. Database
‣ NDB MySQL Cluster
‣ Features look performance, ... automatic failover,
transactions, good write
very nice
‣ At failover advantages.
gain
least 3 physical maschines needed to
http://www.twitter.com/railshoster

153. Database
‣ NDB MySQL Cluster
‣ Features look performance, ... automatic failover,
transactions, good write
very nice
‣ At failover advantages.
gain
least 3 physical maschines needed to
‣ Certain joins.
such as large
statements are incredibly slow
http://www.twitter.com/railshoster

154. Database
‣ NDB MySQL Cluster
‣ Features look performance, ... automatic failover,
transactions, good write
very nice
‣ At failover advantages.
gain
least 3 physical maschines needed to
‣ Certain joins.
such as large
statements are incredibly slow
‣ DB sizestored in memory. RAM size since indexed
columns are
limited by
http://www.twitter.com/railshoster

155. Database
http://www.twitter.com/railshoster

156. Database
‣ NDB MySQL Cluster
http://www.twitter.com/railshoster

157. Database
‣ NDB MySQL Cluster
‣ Only suitable for certain situations
http://www.twitter.com/railshoster

158. Database
‣ NDB MySQL Cluster
‣ Only suitable for certain situations
‣ Joins avoided by using a search server
solr, sphinx, ...
http://www.twitter.com/railshoster

159. Database
‣ NDB MySQL Cluster
‣ Only suitable for certain situations
‣ Joins avoided by using a search server
solr, sphinx, ...
‣ Many write transactions but slow data grow rate
http://www.twitter.com/railshoster

160. Database
‣ NDB MySQL Cluster
‣ Only suitable for certain situations
‣ Joins avoided by using a search server
solr, sphinx, ...
‣ Many write transactions but slow data grow rate
‣ Maybe replication is an alternative ...
http://www.twitter.com/railshoster

161. Database
Replication

162. Database
http://www.twitter.com/railshoster

163. Database
‣ MA/MA replication
http://www.twitter.com/railshoster

164. Database
‣ MA/MA replication
‣ Possible but
http://www.twitter.com/railshoster

165. Database
‣ MA/MA replication
‣ Possible but
‣ Not conflicts
record
recommendable except your app resolves db
http://www.twitter.com/railshoster

166. Database
http://www.twitter.com/railshoster

167. Database
‣ MA/SL replication
http://www.twitter.com/railshoster

168. Database
‣ MA/SL replication
‣ Good starting point
http://www.twitter.com/railshoster

169. Database
‣ MA/SL replication
‣ Good starting point
‣ SL = realtime backup of MA DB but
http://www.twitter.com/railshoster

170. Database
‣ MA/SL replication
‣ Good starting point
‣ SL = realtime backup of MA DB but
‣ Can be used read-only
http://www.twitter.com/railshoster

171. Database
‣ MA/SL replication
‣ Good starting point
‣ SL = realtime backup of MA DB but
‣ Can be used read-only
‣ No automatica failover but manual failoverfor the
faster than reinstalling single server. But calculate time
is still
master reintegration!
http://www.twitter.com/railshoster

172. Software
Deployment

173. Deployment

174. Deployment
‣ Capistrano

175. Deployment
‣ Capistrano
‣ Adapt deployment recipe to server and server
roles

176. Deployment
‣ Capistrano
‣ Adapt deployment recipe to server and server
roles
‣ Initial deployment painful

177. Deployment
‣ Capistrano
‣ Adapt deployment recipe to server and server
roles
‣ Initial deployment painful
‣ Further deployments more or less painless

178. Backups

179. Backups

180. Backups
‣ Create automated backups!

181. Backups
‣ Create automated backups!
‣ Cover filesystem and database

182. Backups
‣ Create automated backups!
‣ Cover filesystem and database
‣ Create them, pack them, encrypt them
and upload them to a safe place. e.g. use
duplicity!

183. Backups
‣ Create automated backups!
‣ Cover filesystem and database
‣ Create them, pack them, encrypt them
and upload them to a safe place. e.g. use
duplicity!
‣ Test backups regularily. Broken backups = no backups.

184. Monitoring

185. „Just because you’re paranoid, doesn’t mean
they’re not out to get you.“

186. How does your admin know that something
is broken?

187. Monitoring
Nagios

188. Nagios

189. Nagios
‣ Nagios monitors key resources of a
system. At least it that how it should be configured.

190. Nagios
‣ Nagios monitors key resources of a
system. At least it that how it should be configured.
‣ Is some resource not responding or hits a
threshold: notify! Again, that‘s how it should be.

191. Nagios
‣ Nagios monitors key resources of a
system. At least it that how it should be configured.
‣ Is some resource not responding or hits a
threshold: notify! Again, that‘s how it should be.
‣ Also consider using monit for local proactive monitoring issues.

192. Nagios

193. Nagios
‣ What should be checked? List is incomplete.

194. Nagios
‣ What should be checked? List is incomplete.
‣ Swap. Lot of swap usage = not enough RAM = Sloooooow.

195. Nagios
‣ What should be checked? List is incomplete.
‣ Swap. Lot of swap usage = not enough RAM = Sloooooow.
‣ LOAD. More sysload than cores? Processes need to wait.

196. Nagios
‣ What should be checked? List is incomplete.
‣ Swap. Lot of swap usage = not enough RAM = Sloooooow.
‣ LOAD. More sysload than cores? Processes need to wait.
‣ HTTP. Webserver, are you still there?

197. Nagios
‣ What should be checked? List is incomplete.
‣ Swap. Lot of swap usage = not enough RAM = Sloooooow.
‣ LOAD. More sysload than cores? Processes need to wait.
‣ HTTP. Webserver, are you still there?
‣ APP. Does my app still serve my control string XY?

198. Nagios
‣ What should be checked? List is incomplete.
‣ Swap. Lot of swap usage = not enough RAM = Sloooooow.
‣ LOAD. More sysload than cores? Processes need to wait.
‣ HTTP. Webserver, are you still there?
‣ APP. Does my app still serve my control string XY?
‣ and a lot more!

199. Monitoring
Resource History

200. Resource History

201. Resource History
‣ Munin

202. Resource History

203. Resource History
‣ Stores the consumption of individual
resources over time.

204. Resource History
‣ Stores the consumption of individual
resources over time.
‣ Trend analysis. What happened after the last puplicity ploy
on our servers?

205. Resource History
‣ Stores the consumption of individual
resources over time.
‣ Trend analysis. What happened after the last puplicity ploy
on our servers?
‣ Scale out needed? Or maybe in a month?

206. Resource History
‣ Stores the consumption of individual
resources over time.
‣ Trend analysis. What happened after the last puplicity ploy
on our servers?
‣ Scale out needed? Or maybe in a month?
‣ Does my app leak?

207. Maintenance

208. Maintenance

209. Maintenance
‣ OS updates. updates.
Minor-, major-, distro

210. Maintenance
‣ OS updates. updates.
Minor-, major-, distro
‣ Take a server out of the cluster to perform bigger updates. Reintegrate
if afterwards. No or minor service impact.

211. Maintenance
‣ OS updates. updates.
Minor-, major-, distro
‣ Take a server out of the cluster to perform bigger updates. Reintegrate
if afterwards. No or minor service impact.
‣ Ruby Updates. Ruby, Rails, Gems, ...

212. Maintenance
‣ OS updates. updates.
Minor-, major-, distro
‣ Take a server out of the cluster to perform bigger updates. Reintegrate
if afterwards. No or minor service impact.
‣ Ruby Updates. Ruby, Rails, Gems, ...
‣ Rotation and monitoring of logs. Bad things
going on?

213. Maintenance
‣ OS updates. updates.
Minor-, major-, distro
‣ Take a server out of the cluster to perform bigger updates. Reintegrate
if afterwards. No or minor service impact.
‣ Ruby Updates. Ruby, Rails, Gems, ...
‣ Rotation and monitoring of logs. Bad things
going on?
‣ Check your backups!

214. Maintenance
‣ OS updates. updates.
Minor-, major-, distro
‣ Take a server out of the cluster to perform bigger updates. Reintegrate
if afterwards. No or minor service impact.
‣ Ruby Updates. Ruby, Rails, Gems, ...
‣ Rotation and monitoring of logs. Bad things
going on?
‣ Check your backups!
‣ ...

215. Fazit
Ok, nice to know but ...

216. Fazit
... who the hell should to do all those
things and additionally develop an
application?

218. Thank you
for your
attention!
Headquarter: Rails Enterprise Hosting:
http://www.avarteq.de http://www.enterprise-
rails.de
Blog:
http://ww.treibstofff.de Rails Hosting:
http://www.railshoster.de

219. Resources
http://www.twitter.com/railshoster

220. Resources
‣ Python Web Application Hosting
http://hostingfu.com/article/python-web-application-hosting
http://www.twitter.com/railshoster

221. Resources
‣ Python Web Application Hosting
http://hostingfu.com/article/python-web-application-hosting
‣ Apache and mod_wsgi
http://djangoinproduction.com/wiki/Apache_and_mod_wsgi
http://www.twitter.com/railshoster

222. Resources
‣ Python Web Application Hosting
http://hostingfu.com/article/python-web-application-hosting
‣ Apache and mod_wsgi
http://djangoinproduction.com/wiki/Apache_and_mod_wsgi
‣ MySQL Cluster
http://www.mysql.com/products/database/cluster/
http://www.twitter.com/railshoster

223. Resources
‣ Python Web Application Hosting
http://hostingfu.com/article/python-web-application-hosting
‣ Apache and mod_wsgi
http://djangoinproduction.com/wiki/Apache_and_mod_wsgi
‣ MySQL Cluster
http://www.mysql.com/products/database/cluster/
‣ Duplicity Backup Tool
http://duplicity.nongnu.org/
http://www.twitter.com/railshoster