Scaling PHP to 40 Million Uniques

Scaling PHP to
40 Million Uniques
Northeast PHP 2013
Jonathan Klein
@jonathanklein
Saturday, August 17, 13

Scaling PHP to
60 Million Uniques
Northeast PHP 2013
Jonathan Klein
@jonathanklein

Who Am I?

Who Am I?
• Performance Engineer at Etsy

Who Am I?
• Boston Web Perf Meetup Organizer

Who Am I?
• Previously at Wayfair

Who Am I?
• Previously at Wayfair
• Led team that converted to PHP

Slides, Links:
jkle.in/nephp

codeascraft.com

Some Etsy Stats

Some Etsy Stats
• 1.5 billion page views/month

Some Etsy Stats
• Almost $1B in sales last year

Some Etsy Stats
• Over 1M lines of PHP

Some Etsy Stats
• Over 1M lines of PHP
• 60M+ unique visitors/month

2012 Trafﬁc

August
2012 Trafﬁc

August
2012 Trafﬁc
50% Higher

Holidays:
~90M Uniques

A few others...

A few others...
• Search (Solr)

A few others...
• Search (Solr)
• Gearman

A few others...
• Search (Solr)
• Gearman
• Redis

A few others...
• Search (Solr)
• Gearman
• Redis
• Postgres (legacy)

Hardware (Supermicro)

2x 8-core Intel E5-2960 CPUs

24GB of RAM

24GB of RAM
160GB SSD

Apache

Apache
• Apache 2.2

Apache
• Apache 2.2
• Prefork MPM

Apache
• Apache 2.2
• Prefork MPM
• mod_php5

Apache
• Apache 2.2
• Prefork MPM
• mod_php5
StartServers 30
MinSpareServers 30
MaxSpareServers 60
ServerLimit 60
MaxClients 60
MaxRequestsPerChild 0

PHP
• PHP 5.4

PHP
• PHP 5.4
• Zend OPCache (3GB Memory Segment)

PHP
• PHP 5.4
• memory_limit: 128M

PHP
• PHP 5.4
• memory_limit: 128M
• max_execution_time: 30

Optimizing PHP
http://www.slideshare.net/jnklein/northeast-php-high-performance-php

Optimizing PHP
• See last year’s talk

Optimizing PHP
• Use an Opcode Cache

Optimizing PHP
• xhprof

Optimizing PHP
• xhprof
• StatsD/Graphite

Optimizing PHP
• xhprof
• StatsD/Graphite
• Find hotspots

Optimizing PHP
• xhprof
• StatsD/Graphite
• Find hotspots
• Upgrade it

Static Arrays

Translations
class language_de {
static $translations = array(
"<hash>" => array("content" => 'Accessoires', "file" => "TAXONOMY"),
"<hash>" => array("content" => 'Keramik', "file" => "CATEGORY"),
"<hash>" => array("content" => 'Dekorieren', "file" => "CATEGORY"),
"<hash>" => array("content" => 'Getaggt %s', "file" => "Foo.php"),
"<hash>" => array("content" => 'Badezusatz', "file" => "CATEGORY"),
"<hash>" => array("content" => 'Datum:{% $date %}', "file" => "bar.tpl")
... snip ...
);
}

Atomic Deploys

Deploying PHP
http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/

Deploying PHP
• mod_realdoc - Apache module

Deploying PHP
• incpath - PHP Extension

Deploying PHP
• A/B symlink swap

Deploying PHP
• A/B symlink swap
• Avoid recompilation to opcodes

Upgrading PHP

PHP 5.4 vs. 5.3 - CPU
PHP 5.3
PHP 5.4

PHP 5.4 vs. 5.3 - Memory
PHP 5.3
PHP 5.4

Understand
Framework
Overhead

http://systemsarchitect.net/performance-benchmark-of-popular-php-frameworks/

Frameworks are
Harder to Scale

Memcached

Memcached
• Roughly same hardware as webs

Memcached
• More memory (48GB)

Memcached
• Shard keys across servers

Memcached
• Shard keys across servers
• mctop - https://github.com/etsy/mctop

MySQL
• MySQL 5.5 (http://jkle.in/nephp)

MySQL
• ~20 master/master pairs

MySQL
• innodb

MySQL
• innodb
• thread_cache=800

MySQL
• innodb
• max_connections=2500

MySQL
• innodb
• max_connections=2500
• [Almost] no joins

Theoretically we can
serve ~9000 req/sec

Static Content

~500 Million Images

Offloading Requests

Offloading Requests
• Multiple CDNs

Offloading Requests
• Multiple CDNs
• Img hit rate > 90%

Offloading Requests
• Multiple CDNs
• CSS/JS hit rate ~100%

Offloading Requests
• Multiple CDNs
• CSS/JS hit rate ~100%
• Edge serves ~7000 reqs/sec

What about misses?

Change Management

Feature Flags

// Owner: Username
$server_config['foo']['bar'] = array(
'users' => array('jsmith', 'jdoe'),
);
// Owner: Username
$server_config['baz'] = [
'enabled' => [
'variant1' => 0,
'variant2' => 0,
'variant3' => 50
],
];
// Owner: Username
$server_config['qux'] = array(
'enabled' => 100,
'data' => array(
'somedata'
)
);

if (Feature::isEnabled('foo.bar')) {
// Code
}

Ramp-up:
1%, 5%, 25%, 100%

Deployinator

Schemanator

Recent Improvements

Sandy Bridge

Baseline Performance (median)

Small code change

Small code change
PHP 5.4

Small code change
PHP 5.4
Disabled
Hyperthreading

Challenges

Options

Options
• Scale Horizontally (more shards)

Options
• Scale Vertically (SSD, bigger drives)

Options
• Scale Vertically (SSD, bigger drives)
• Change architecture

Architecture Changes

• Ongoing conversation

• Separate logical/physical shards

• Complex

• Complex
• Server failure

• Complex
• Server failure
• Migrating data

• Complex
• Server failure
• Migrating data
• Consistency

Monitoring

Graphite

Stacked Search Timers

Ganglia

Nagios

Takeaways

K.I.S.S.

Use Proven
Technologies

Understand Your Stack

Measure Everything

Work at Etsy ;-)

Connect
http://www.meetup.com/Web-Performance-Boston/
www.etsy.com/careers
jonathan@etsy.com
@jonathanklein

Scaling PHP to 40 Million Uniques

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a Scaling PHP to 40 Million Uniques

Semelhante a Scaling PHP to 40 Million Uniques (20)

Último

Último (20)

Scaling PHP to 40 Million Uniques