O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
ResqueBackground processingSystem administration on a multi-        instances setup      Experience feedback            Pl...
My project : bookandgolf.com
bookandgolf.comAt launch :  • more than 60 golfs to synchronise  • 4 different APIs  • More than 350 000 slots (starts) to...
nginx          resque-web:web             redis-server:db       worker 1                        Current       worker 2:app...
Resque job classclass MyKikoololSleepJob  @queue = :medium  def self.perform(sleep_duration)    sleep sleep_duration  ende...
Resque workersrake environment resque:work QUEUE=mediumrails@www:~$ ps aux | grep resquerails  8950  sh -c cd /home/rails/...
Resque workers Unix signals• QUIT - Wait for child to finish processing then exit• TERM / INT - Immediately kill child then...
Resque-web (Sinatra webapp)
God setup - my way!Versioned god config files by instance name, ie :•   god / app.god•   god / web.god•   god / db.god•   ...
God config# ResqueGod.watch do |w|  w.env = { RAILS_ROOT => rails_root,              RAILS_ENV  => rails_env }  w.name    ...
Workers graceful restartRake task to restart workers, the "graceful"waynamespace :resque do  task :restart_workers => :env...
Workers graceful restartnamespace :resque do  task :restart_workers, :roles => [:app, :db] do    rake "resque:restart_work...
Resque-web - Final setup• Launched externally with nginx reverse proxy (easy to add  http basic auth)• External Ruby confi...
Resque-web God monitoring%w{5678}.each do |port|  God.watch do |w|    w.env = { RAILS_ROOT => rails_root,              RAI...
ActiveRecord - timeout / stale :(Fast workaround...   ...   ActiveRecord::Base.connection.reconnect!   ...
Resque plugins• defunkt / resque-lockPrevent two workers from working on the sameJob class with same arguments
Resque plugins• jayniz / resque-lonerTIP :Do not forget to :  require "resque-loner"in your Resque-web config.rb file! or....
Other plugins...Look @ github.com/defunkt/resqueWiki!And remember that with a goodsystem architecture, Resque is just...
Background processing with Resque
Background processing with Resque
Background processing with Resque
Background processing with Resque
Próximos SlideShares
Carregando em…5
×

Background processing with Resque

15.203 visualizações

Publicada em

System administration on a multi-instances architecture, experience feedback, plugins I like

Publicada em: Tecnologia
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Seja a primeira pessoa a gostar disto

Background processing with Resque

  1. 1. ResqueBackground processingSystem administration on a multi- instances setup Experience feedback Plugins Nicolas Blanco Novapost
  2. 2. My project : bookandgolf.com
  3. 3. bookandgolf.comAt launch : • more than 60 golfs to synchronise • 4 different APIs • More than 350 000 slots (starts) to synchronize every 2 hoursAt first we tried to do the synchronization with cron tasks... but...
  4. 4. nginx resque-web:web redis-server:db worker 1 Current worker 2:app worker 3 architecture worker 4
  5. 5. Resque job classclass MyKikoololSleepJob  @queue = :medium  def self.perform(sleep_duration)    sleep sleep_duration  endendResque.enqueue(MyKikoololSleepJob, 10)
  6. 6. Resque workersrake environment resque:work QUEUE=mediumrails@www:~$ ps aux | grep resquerails  8950  sh -c cd /home/rails/www/socianalytics/current &&rake environment resque:work QUEUE=high,medium,low...rails  8951  resque-1.9.10: Forked 9105 at 1302095560               rails  9105  resque-1.9.10: Processing medium since 1302095560      
  7. 7. Resque workers Unix signals• QUIT - Wait for child to finish processing then exit• TERM / INT - Immediately kill child then exit• USR1 - Immediately kill child but dont exit• USR2 - Dont start to process any new jobs• CONT - Start to process new jobs again after a USR2
  8. 8. Resque-web (Sinatra webapp)
  9. 9. God setup - my way!Versioned god config files by instance name, ie :• god / app.god• god / web.god• god / db.god• ...God init.d file found on gist.github.com :)god.conf :GOD_CONFIG=/home/rails/www/bookgolf/current/god/db.godGOD_COMMAND="sudo -u rails /usr/local/bin/god"GOD_LOG=/home/rails/log/god.log
  10. 10. God config# ResqueGod.watch do |w|  w.env = { RAILS_ROOT => rails_root,              RAILS_ENV  => rails_env }  w.name          = "resque-worker"  w.interval      = 30.seconds  w.start         = "cd #{rails_root} && rakeenvironment resque:work QUEUE=high,medium,low"  w.start_grace   = 10.seconds    ...
  11. 11. Workers graceful restartRake task to restart workers, the "graceful"waynamespace :resque do  task :restart_workers => :environment do    pids = Array.new        Resque.workers.each do |worker|      pids << worker.to_s.split(/:/).second ifworker.to_s.include?(Settings.resque.localhost_name)    end        if pids.size > 0      system("kill -QUIT #{pids.join( )}")    end        system("rm /home/rails/.god/pids/resque-worker*.pid")  endend
  12. 12. Workers graceful restartnamespace :resque do  task :restart_workers, :roles => [:app, :db] do    rake "resque:restart_workers"  endend
  13. 13. Resque-web - Final setup• Launched externally with nginx reverse proxy (easy to add http basic auth)• External Ruby config file to change Redis setup, load plugins, etc.     resque-web config/resque-web.rb
  14. 14. Resque-web God monitoring%w{5678}.each do |port|  God.watch do |w|    w.env = { RAILS_ROOT => rails_root,              RAILS_ENV  => rails_env }    w.uid       = "rails"    w.name      = "resque-web"    w.interval  = 30.seconds    w.start     = "cd #{rails_root} && resque-web config/resque-web.rb"    w.start_grace   = 15.seconds    w.start_if do |start|      start.condition(:process_running) do |c|          c.interval = 5.seconds          c.running = false      end    end  endend
  15. 15. ActiveRecord - timeout / stale :(Fast workaround...   ...   ActiveRecord::Base.connection.reconnect!   ...
  16. 16. Resque plugins• defunkt / resque-lockPrevent two workers from working on the sameJob class with same arguments
  17. 17. Resque plugins• jayniz / resque-lonerTIP :Do not forget to :  require "resque-loner"in your Resque-web config.rb file! or... FAIL (cleared jobs wontbe relaunched again!)BAD :Need inheritance in job classes :(... But a fork exists thatdoesnt need it.
  18. 18. Other plugins...Look @ github.com/defunkt/resqueWiki!And remember that with a goodsystem architecture, Resque is just...

×