O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

How to switch stack without downtime

2.983 visualizações

Publicada em

Slides from a talk about how Stuart managed to successfully move away from a stack to another within a few months, without stopping our business.

It sounds easy, but as many of you might know it's very hard. Many of us tried and failed. We succeeded, this is how we did it and you may use our experience to make improve your chance to succeed with yours.

We'll explain how we remodeled the Engineer team, and added workflow processes, slowly moving away from the old stack to the new one.

Publicada em: Software
  • Seja o primeiro a comentar

How to switch stack without downtime

  1. 1. SUCCESS STORY: HOW TO SWITCH STACK WITHOUT DOWNTIME
  2. 2. THE LEADER OF URBAN & ON-DEMAND LOGISTICS
  3. 3. Our mission is to connect merchants, customers and local couriers to speed-up the way goods are transported in cities
  4. 4. 04 MERCHANTS TRUST US GROWING LIST OF TOP-TIER REFERENCES ACROSS VERTICALS
  5. 5. 05 TEAM GROWING AND TALENTED ORGANISATION CLÉMENT BENOIT Co-Founder & CEO Resto-in founder BENJAMIN CHEMLA Co-Founder & VP Sales Citycake founder DAMIEN BON COO BCG INSEAD Lehman Brothers FABIEN PENSO CTO Process-One CloudScreener Causes We manage a team of 80 people coming from the best companies
  6. 6. 06 ABOUT ME 1995: Started using Linux 1998: LinuxFr - a French Linux community website 1999: DBee - First company Since 2010: Causes, BBC, Cloudscreener, Stuart PAST EXPERIENCES @fabienpenso http://penso.info
  7. 7. Short Video
  8. 8. 08 INFRASTRUCTURE 2015
  9. 9. 09 SOFTWARE STACK 2015 Standard LAMP + RabbitMQ + Crontab 0% test coverage GPA: 3.40 90k LOC 150 SQL tables
  10. 10. 10 API REQUESTS PROCESSED IN 2015 SLOW PHP REQUEST PROCESSING (INCLUDES PSEUDO BACKGROUND TASKS) REACH MAX AMOUNT OF PHP PROCESSES QUICKLY MINIMUM INTERVAL FOR TASKS IS 1 MINUTE INEFFICIENT STORAGE FOR BACKGROUND TASKS
  11. 11. 11 LOOKING FOR A NEW STACK OPTION
  12. 12. 12 HOW TO SWITCH: TWO PATHWAYS Option A Write all from the ground up, new Rails project, then write a script to move all existing data. + Clean code, no legacy once moved. - Can't rollback, long time before seeing improvements - High chance of failure Option B Write small pieces of code to work with the existing context/ environment. + Business doesn't stop, can still add features, and fix existing bugs - Manage 2 platforms for months, new code connects to old storage schema, must replicate bad code structure
  13. 13. 13 QUICK CODE GENERATION Automatic code generation - ActiveRecord Ruby code from SQL tables Backoffice - Install ActiveAdmin to view all AR classes and records Continuous Integration - Setup deployment Gem - php_serialize, annotate, devise # == Schema Information # # Table name: vehicle # # id :integer not null, primary key # driver_id :integer # brand :string(100) # model :string(150) # number_plate :string(20) # color :string(45) # insurance :string(255) # vehicle_registration :string(255) class Vehicle < ActiveRecord::Base end
  14. 14. 14 RUNNING BOTH PLATFORM
  15. 15. Refactor Feature Flag Ruby Feedback Encapsulate the PHP feature within a single function call Add a feature flag in the PHP code to enable/ disable this call QA team enable the feature on staging, and verify it works as expected Implement the next "feature" Clone code in Ruby, add an opposite feature flag 15 IMPLEMENTING NEW CODE Production
  16. 16. 16 HAVE A PLAN Split the legacy codebase into virtual modules Rewrite an easy one at first Rewrite them in order of importance Finish with low priority non critical code
  17. 17. 17 IMPROVE LEGACY CODE Add AR model validation class Job < ActiveRecord::Base validates :expires_at, presence: true validate :payment_ability, on: :create def payment_ability if !has_funds? && !credit_card errors.add(:credit_card, "meaningful message") end end end Ruby Run validation for every SQL changes Warn about invalid models on Slack
  18. 18. <?php // … code class EntityListener { protected $sidekiqJobPusher; public function __construct(TransactionQueueService $redisClient) { $this->sidekiqJobPusher = $redisClient; } public function onUpdate($calledEntity, UnitOfWork $uow) { $this->sidekiqJobPusher->performActiveJob("PhpJob", [$calledEntityName, $id, 'php_after_updated', $changedValues], true, "php"); } public function onInsert($calledEntity) { $this->sidekiqJobPusher->performActiveJob("PhpJob", [$calledEntityName, $id, 'php_after_created'], true, "php"); } } ?> PHP Method injections 18 IMPROVE LEGACY CODE
  19. 19. # app/models/concerns/php_callbacks.rb module PhpCallbacks def php_after_created(_options = {}) return if valid? message = "#{self.class}: #{id} errors: #{errors.full_messages.to_sentence}" Service::Slack.ping message, channel: "#exceptions-#{Rails.env}" end alias php_after_updated php_after_created End # app/jobs/php_job.rb class PhpJob < ActiveJob::Base queue_as :php include JobLogger ACCEPTED_METHODS = %w(php_after_updated php_after_created).freeze def perform(klass_type, id, method, *args) unless ACCEPTED_METHODS.include?(method) raise ArgumentError.new("`method` can only be #{ACCEPTED_METHODS.join(' or ')}!") end klass = klass_type.constantize rescue nil if klass && klass.method_defined?(method.to_sym) object = klass.find(id) object.send(method, *args) end end end Ruby validation execution 19 IMPROVE LEGACY CODE
  20. 20. 20 FEATURE FLAG FEATURE FLAG: ABILITY TO TURN FEATURE ON/OFF PHP use OpensoftRolloutRollout;
 use OpensoftRolloutStorageArrayStorage; 
 $rollout = new Rollout(new ArrayStorage()); $rollout->isActive('outgoing_email'); Ruby require "redis"
 require "rollout"
 $redis = Redis.new
 $rollout = Rollout.new($redis) $rollout.active?(:outgoing_email) https://github.com/opensoft/rollout (PHP) https://github.com/fetlife/rollout (Ruby)
  21. 21. 21 REMOVE OVERENGINEERED CODE TOO MANY SQL TABLES Before: 150 tables After: 90 tables class DriverStatusType include TypableModel read_only_attr :id, :code, :name defaults [ [1, "off_duty", "Off Duty"], [2, "on_duty", "On Duty"], [3, "busy", "Busy"], [4, "pending", "Pending"], [5, "rejected", "Rejected"] ] end DriverStatusType.find_by(id: 1) DriverStatusType.find_by(code: :on_duty)
  22. 22. 22 MOVE API ENDPOINTS List ALL API endpoints currently used looking at the PHP logs 68 endpoints to move Make a plan
  23. 23. 23 MOVE API ENDPOINTS Move one non critical API endpoint Move by importance, most critical first Good potential: https://getkong.org Be able to turn each api endpoint on/off
  24. 24. Grape Routing Feedback Implement the API endpoint in Ruby Move this API endpoint to Ruby in the nginx routing file on staging Implement the next api endpoint QA team verify it works as expected 24 MOVE API ENDPOINTS Production
  25. 25. 25 MOVE API ENDPOINTS TOOLS: LIST OUR QA USED FOR TESTING Runscope.com Selenium Android Espresso iOS xcode test Postman and curl Charles HTTP Proxy
  26. 26. 26 INFRASTRUCTURE 2016
  27. 27. 27 CLOSING: SOFTWARE STACK 2015 Standard LAMP + RabbitMQ + Crontab 0% test coverage GPA: 3.40 90k LOC 150 SQL tables
  28. 28. 28 CLOSING: SOFTWARE STACK 2016 Grape + Sidekiq + Rails 70% test coverage GPA: 3.95 15k LOC 90 SQL tables
  29. 29. How long to move all the code ?
  30. 30. 30 CLOSING ROOM FOR IMPROVEMENT: WHAT WOULD WE DO DIFFERENT? Ability to have ephemeral servers Give more power to the QA Engineers
  31. 31. @fabienpenso

×