1. Porting Rails apps for
High Availability
Systems
Marcelo Correia Pinheiro
@salizzar

2. $ whoami
Programmed with ASP, PHP, C#, Java, Python,
Ruby, etc etc
Twitter: @salizzar
http://salizzar.net/ (always under
construction)
Locaweb PaaS/IaaS Engineering Team
(Domain Registration)

3. Topics
Common Architecture
Application Modularity
Useful Gems
Database Replication
Session Management
App Distribution and Deployment
Configuration Management
Load Balancing
Conclusion

5. Common
Architecture
One machine running HTTP Server, App and Database
(sometimes in a dedicated server)
Production config files in application GIT repo (all of us
use git, alright?)
Deploy via Capistrano / Vlad / * ?
Sufficient for small apps with no perspective to grow
absurdly or have monitored growing
Grants HA? Hmmm... what do you do if your database
corrupts journaling? (MongoDB REAL CASE)

7. Common
Architecture
Think about it and the possibility of a
nightmare due for a bad db query, zombie
processes and unexpected errors that will
frozen your app
What to do? Look at your app with a
Distributed System perspective

8. Application
Modularity
Rails apps, in most cases, can be splitted to two
or more small gems or apps (web services,
checkout systems, asynch processing, etc)

9. Application
Modularity
You write good code?
If yes, creating gems with it is EASY.
How to detect: first DRY while splitting a app
How to create: $ bundle gem <asdfg>
How to learn more: Creating Your Own Gem in
www.rubygems.org
If not, refactoring!

10. Application
Modularity
Breaking your Rails app in smaller apps:
increase future improvements
first step to make distribution via packaging
more easy
consequently, more scalable
Look at ActiveController::Metal / sinatra /
rack-api / goliath / cramp

11. Application
Modularity
Be careful:
using multiple gems
coupling Rails features inside your models
(Rails.logger)

12. Useful Gems
Hypermedia
Asynchronous Processing (resque)
Webservers
Monitoring

13. Useful Gems
JSON for the win
Hypermedia-based REST API’s
ROAR: https://github.com/apotonick/roar
RestModel: https://github.com/rodrigues/
rest_model

14. Useful Gems
# -*- encoding: UTF-8 -*-
class Domain
attr_accessor :id, :creation_date, :expiration_date
end
module DomainRepresenter
include Roar::Representer::JSON,
Roar::Representer::Feature::Hypermedia
property :id, :creation_date, :expiration_date
end
request = RestClient.get 'http://an.app/domains/example.com'
domain = Domain.new.extend DomainRepresenter
domain.from_json request.body

15. Useful Gems
# -*- encoding: UTF-8 -*-
class Domain < RestModel
property :id, type: String
property :creation_date, type: Date
property :expiration_date, type: Date
property :provisioning_id, type: Integer
end
request = RestClient.get 'http://an.app/domains/example.com'
domain = Domain.from_source(request.body).first

16. Useful Gems
Asynchronous Processing
Resque
resque-scheduler
resque-retry
resque-lock
resque-logger

17. Useful Gems
resque-scheduler:
https://github.com/bvandenbos/resque-
scheduler
cron-based Resque extension

18. Useful Gems
# resque_scheduler.yml
appointment_notification_enqueuer:
cron: "0 4 * * *"
class: AppointmentNotificationEnqueuer
queue: appointment_notification_enqueuer
args:
email: true
sms: true de cron
Incluir config
description: Notify today appointments to customers
free_slot_monitor:
cron: "0/15 * * * *"
class: FreeSlotMonitor
queue: free_slot_monitor
description: Check for free slots in agenda

19. Useful Gems
# -*- encoding: UTF-8 -*-
class AppointmentNotificationEnqueuer
@queue = :appointment_notification_enqueuer
def self.perform(args)
appointments = Appointment.today
factory = NotificationFactory.new args
appointments.each do |appointment|
factory.enqueue appointment.id
end
end
end

20. Useful Gems
resque-retry:
https://github.com/lantins/resque-retry
redis backed
retry count times until reach a limit
retry on all or specific exceptions
retry functionality on Resque Web Server

21. Useful Gems
# -*- encoding: UTF-8 -*-
class EmailNotificationWorker
extend Resque::Plugins::Retry
@queue = :email_notification_enqueuer
@retry_limit = 4
@retry_delay = 300
def self.perform(args)
appointment = Appointment.find id: args['id']
mailer = MailNotification.new
mailer.notify_appointment appointment
end
end

22. Useful Gems

23. Useful Gems
resque-lock:
https://github.com/defunkt/resque-lock
Grants only one instance of a job running in
a time

24. Useful Gems
# -*- encoding: UTF-8 -*-
class SmsNotificationWorker
extend Resque::Plugins::Lock
def self.perform(args)
appointment = Appointment.find id: args['id']
sender = SmsNotification.new
sender.deliver_appointment appointment
end
end

25. Useful Gems
resque-logger:
https://github.com/salizzar/resque-logger
Provides a logger for each Resque worker
based on queue name
Log Driven Development, please!

26. Useful Gems
# config/initializers/resque.rb
log_path = Rails.root.join 'log'
config = {
folder: log_path,
class_name: Logger,
class_args: [ 'daily', 1.gigabyte ],
level: Logger::INFO,
formatter: Logger::Formatter.new,
}
Resque.logger = config

27. Useful Gems
# -*- encoding: UTF-8 -*-
class EmailNotificationWorker
extend Resque::Plugins::Retry, Resque::Plugins::Logger
@queue = :email_notification_enqueuer
@retry_limit = 4
@retry_delay = 300
def self.perform(args)
appointment = Appointment.find id: args['id']
logger.info(“trying to notify #{appointment.id}”)
mailer = MailNotification.new
mailer.notify_appointment appointment
end
end

28. Useful Gems
Web Servers:
Thin
https://github.com/macournoyer/thin
EventMachine powered
Killer combo with Nginx
Unicorn
http://unicorn.bogomips.org
Pre-fork based
Great with low-latency network

29. Useful Gems
Monitoring
God
https://github.com/mojombo/god
Several memory leaks at beginning, very stable today
Largely used to manage Resque workers and Thin clusters
Bluepill
https://github.com/arya/bluepill
More simple than God, less features
Some issues with signal handling

30. Useful Gems
If memory usage is a problem, use Monit.
http://mmonit.com/monit
VERY stable
HIGH expansible
EASY configuration

31. Database Replication
Relational Database Replication is not trivial
Be a friend of a sysadm
CAP Theorem: YOU NEED TO KNOW
No magic: look to your architecture
Traditional replication strategies:
master / slave for consistency
master for write / slave for read

32. Database Replication
My experience:
Relational Databases: no choice
MongoDB: replica set
Redis: master / slave

33. Database Replication
For MongoDB:
MongoMapper
https://github.com/jnunemaker/mongomapper
Largely used, stable
Mongoid
https://github.com/mongoid/mongoid
ActiveRecord-like

34. Database Replication
brotip #1342: check if your MongoDB client
detects when master server goes down
automagically (MongoMapper NOT does it)
Suggested keynote: MongoDB em Produção,
dá pra confiar?
http://www.slideshare.net/deferraz/
mongodb-em-producao

35. Session Management
Given a problem, use the right tool PLEASE
memcached
http://memcached.org
distributed memory object caching system
key/value storage
stores arbitrary data
can be clustered

36. Session Management
repcached
http://repcached.lab.klab.org
patch inside memcached to add data
replication
offers session sharing between memcached
servers

37. Session Management
available gems for memcache
memcache-client
https://github.com/mperham/memcache-client
First Ruby client, written by Mike Perham
memcached
https://github.com/evan/memcached
Ruby SWIG wrapper for libmemcached C library
dalli
https://github.com/mperham/dalli
Major refactoring from memcache-client

38. App Distribution and
Deployment
Capistrano? Hnmm... talk with a sysadm about
it. It works but lacks security severely (SSH, GCC
in production is a invite for attack)
Sysadms knows process monitoring tools,
software distribution and server configuration
better than us
Be a friend of a sysadm
Or LEARN A LOT to be one :)

39. App Distribution and
Deployment
Build self-packaged apps
$ man dh_make (for Debian based distros)
Hard or too boring? Maybe, but effort compensates
$ dh_make --native --simple --indep --
packagename <yourapp> --copyright blank
$ debuild
# dpkg -i <yourapp>_<version>_<arch>.deb

40. App Distribution and
Deployment
Generating a distro-based package with your app
grants fast update/rollback to sysadms
# apt-get update && apt-get install <yourapp>
# dpkg -i <your_app_package>
If you want to automate builds:
Setup a repository server
Use Bricklayer and be happy

41. Configuration
Management
Keeping production configs in your app git
repo is not always a good idea
Getting config files over control is VERY
important to avoid a messy environment
Standardization is king in server configuration

42. Configuration
Management
At this time, three choices:
Puppet
Chef
cfengine

43. Configuration
Management
Puppet
https://github.com/puppetlabs/puppet
Written in Ruby
Large community adoption
Great number of recipes
Works well

44. Configuration
Management
Chef
https://github.com/opscode/chef
Written in Ruby
Created as a Puppet alternative in order to solve
some problems with it
Large community adoption
Works good

45. Configuration
Management
cfengine
http://cfengine.com
Written in C
Largely adopted by sysadms
HARD to learn
F****** ROBUST (a single master node can
manage 4K servers)

46. Load Balancing
WTF is Load Balancing?
A network methodology to distribute workload across
multiple computers in a transparent way for users
Can be proved by /hard|soft/ware
Alternatives:
LVS with keepalived
HAProxy with heartbeat

47. Load Balancing
LVS (Linux Virtual Server)
http://www.linuxvirtualserver.org
Robust, high scalable solution
Need to share HTTP sessions? It’s the choice
Built-in Linux Kernel module since 2.4.28*, runs in layer-4 (transport
layer)
keepalived
http://www.keepalived.org
Largely used with LVS by sysadms
Uses Virtual Router Redundancy Protocol (vrrp) - RFC

48. Load Balancing
HAProxy
http://haproxy.1wt.eu
Generic solution
Works if session sharing between servers is not required
Runs in layer-7 (application layer), computationally more expensive
heartbeat
http://www.linux-ha.org/wiki/Heartbeat
Based on message exchange between clients in a cluster to detect
failures

50. Conclusion
Porting Rails apps to a HA environment is a not trivial challenge, but
nothing insanely hard
Forces you to think in application architecture, design, distribution,
modularity
brotip #915435: do it before coding
Restricts you in order to make server correctly managed
Implies you to be prudent on a deploy
“it sucks at all” = bad excuse, my friend
App tunning later, without fear :)
In fact, in HA deploys are more faster

51. Questions?
No fear, answers are free as beer :)

52. Thank you!