1. shopify
How Shopify Scales Rails
John Duff

2. • The Shopify stack
• Knowing what to scale
• How we cache
• Scaling beyond caching
• Splitting things up
Overview

3. What is Shopify?

7. The Stack

8. • Ruby 1.9.3-p385
• Rails 3.2
• Percona MySQL 5.5
• Unicorn 4.5
• Memcached 1.4.14
• Redis 2.6
The Stack

9. The Stack
• 53 App Servers
• 1590 Unicorn Workers
• 5 Job Servers
• 370 Job Workers
Nginx
Unicorn
Rails 3.2
Ruby 1.9.3-p385

10. The Stack
Firewall
Load Balancer
App Servers
Redis
Job Servers
Database
Memcached Search

11. • 55,873 Lines of application code
• 15,968 Lines of CoffeeScript application code
• 81,892 Lines of test code
• 211 Controllers
• 468 Models
The Stack

12. Current Scale

13. 9.9 M Orders
An order every 3.2 seconds

14. 2,008 Sales per Minute
Cyber Monday

15. 50,000 RPM
45 ms response time

16. 13.3 billion requests

17. Looking Back, to Look Ahead

18. • First line of code written in 2004
• Shopify released June, 2006
• Same codebase
• Over 9 years of Rails upgrades, improvements and changes
Looking Back, to Look Ahead

19. Looking Back, to Look Ahead
• 6,702 Lines of application code (55,873)
• 4,386 Lines of test code (81,892)
• 38 Controllers (211)
• 77 Models (468)

20. Looking Back, to Look Ahead
• Ruby 1.8.2
• Rails 0.13.1
• MySQL 4.1
• Lighttpd
• Memcached

21. Know The System

22. One Request, One Process

23. RPM = W * 1/R

24. RPM = 1590 * 60 / 0.072

25. 1,325,000 = 1172 * 60 / 0.072

26. ↑ Workers
↓ Response Time

27. Know The System
• Avoid network calls during requests
• Speed up unavoidable network calls
• The Storefront and Checkout
• The Chive

28. Chive Flash Sale

29. Measure ALL THE THINGS

30. Measure ALL THE THINGS
• New Relic
• Splunk
• StatsD
• Cacti
• Conan

31. New Relic

32. Splunk

35. Caching

36. cacheable

37. cacheable
• https://github.com/Shopify/cacheable
• serve gzip’d content
• ETag and 304 Not Modified
• generational caching
• no explicit expiry

38. cacheable
class PostsController < ApplicationController
def show
response_cache do
@post = @shop.posts.find(params[:id])
respond_with(@post)
end
end
def cache_key_data
{
:action => action_name,
:format => request.format,
:params => params.slice(:id),
:shop_version => @shop.version
}
end
end

39. requests
Caching Dynamic 404s

40. Identity Cache

41. Identity Cache
• https://github.com/Shopify/identity_cache
• cache full model objects in memcached
• can include associated objects in cache
• must opt in to the cache
• explicit, but automatic expiry

42. Identity Cache
class Product < ActiveRecord::Base
include IdentityCache
has_many :images
cache_index [:shop_id, :id]
cache_has_many :images, :embed => true
end
@product = Product.fetch_by_shop_id_and_id(shop_id, id)
@images = @product.fetch_images

43. Identity Cache

44. Get Out of My Process

45. Delayed Job
• Jobs stored in the db
• Workers run in their own process
• Workers poll for jobs periodically
• https://github.com/collectiveidea/delayed_job

46. Resque
• Redis backed
• O(1) operation to pop jobs
• Faster (300 jobs/sec vs 120 jobs/sec)
• Extensible
• https://github.com/defunkt/resque

47. Resque
• Sending Email
• Processing Payments
• Geolocation
• Import / Export
• Indexing for Search
• 86 Other things...

48. Background Payment Processing
ms

49. Resque
class AddressGeolocationJob
max_retries 3
def self.perform(params)
object = params[:model].constantize.find(params[:id])
object.latitude, object.longitude = Geocoder.geocode(object)
object.save!
end
end
Resque.enqueue(AddressGeolocationJob, :id => 1, :model => 'Address')

50. Redis
• Inventory reservation system
• Sessions
• Theme uploads
• Throttling
• Carts

51. All Roads Lead To MySQL

52. MySQL Hardware
• 4 x 8 Core Processor
• SSD
• 256 GB Ram
• Full working set in memory

53. MySQL Query Optimization
• pt-query-digest
• Avoid queries that generate temp tables
• Adding the right indexes
• Forcing / Ignoring Indexes

54. MySQL Tuning
• disable innodb_stats_on_metadata
• increase table_open_cache
• replace glibc memory allocator with tcmalloc
• innodb_autoinc_lock_mode=‘interleaved’

55. after_commit
db transactions best friend

56. after_commit
• After transaction has been committed
• Webhooks
• Cache expiry
• Update associated objects

57. after_commit
class OrderObserver < ActiveRecord::Observer
observe :order
def after_save(order)
if order.changes.keys.include?(:financial_status)
order.flag_for_after_commit(:update_customer)
end
end
def after_commit(order)
if order.flagged_for_after_commit?(:update_customer)
Resque.enqueue(UpdateCustomerJob, :id => order.id)
end
end
end

58. Services

59. Services
• Split out standalone services as needed
• Independently scaled
• Segmented metrics
• Overall system is more complex
• Limit to what is necessary

60. Imagery

61. Adapt and Evolve as Needed
Using data and knowledge of the system to drive decisions

62. Summary
• Know your application and infrastructure.
• Keep slow IO or CPU tasks out of the main process.
• Measure your optimizations. You can make it worse.

63. Thanks.
@johnduff | john.duff@shopify.com