Scale like an ant, distribute the workload - DPC, Amsterdam, 2011

Distribute the
workload
Helgi Þormar Þorbjörnsson
Dutch PHP Conference, Amsterdam, 19th May 2011
Thursday, 19 May 2011

Who am I?

VP of Engineering at Orchestra.io
Helgi

Developer at PEAR
Helgi

Developer at PEAR
From Iceland
Helgi

Developer at PEAR
From Iceland
@h on Twitter
Helgi

Why Distribute?

Why Distribute?
Budget

Why Distribute?
Budget
Eﬃciency

Why Distribute?
Budget
Eﬃciency
Perception

Budget
Spend wisely

Budget
Spend wisely
Commodity servers

Budget
Spend wisely
Commodity servers
Low overhead, high yield

Budget
Spend wisely
Commodity servers
Low overhead, high yield
Cloud Computing (EC2)

Efficiency
10 small servers > 1 big

Venue Security

1000 people can exit quicker
through 10 small doors than 1 big

Perception

Perception
Defer intensive processes

Perception
Give instant feedback

Perception
Give instant feedback
Users keep on browsing

“It all depends on how we
look at things, and not how
they are in themselves.”
- Carl G. Jung

Chapter from Nature

Ant Colonies

Algorithms

Algorithms
Scheduling

Algorithms
Scheduling
Vehicle Routing

Algorithms
Scheduling
Vehicle Routing
Assignment

Algorithms
Scheduling
Vehicle Routing
Assignment
Sets

Algorithms
Scheduling
Vehicle Routing
Assignment
Sets
Other

How do ants fit?

How do ants fit?
Strength in numbers

How do ants fit?
Strength in numbers
Work together

How do ants fit?
Strength in numbers
Work together
Size beneﬁts them

Teamwork
When faced with a problem they will
solve the problem as one.

What if they were
bigger?

Types of Ants

Types of Ants
Military

Types of Ants
Military
Maids

Types of Ants
Military
Maids
Tunnel diggers

Types of Ants
Military
Maids
Tunnel diggers
Food gatherers

How does this map
to my application?

Colony = Application

Ants = Components

Ants = Components
Ants do many diﬀerent types of work to keep
their colony running

Architect for
Distribution

Characteristics

Characteristics
Decoupling

Characteristics
Decoupling
Elasticity

Characteristics
Decoupling
Elasticity
High Availability

Characteristics
Decoupling
Elasticity
High Availability
Concurrency

Decoupling

Application
DB API
Cache FE

ApplicationDB API
Cache FE

ApplicationDB API
Cache FE
Cache

ApplicationDB API
Cache FE
Cache
API

ApplicationDB API
Cache FE
Cache
API
API

Elasticity

Cloud Computing

Load Balancing

HA Proxy
Nginx
My Favourite

Monitoring

When do I need
more servers?

Needs to be around
from the start!

Keep records

Spot trends

Different types

Different types
Hardware Performance

Different types
Software Performance

Different types
Availability

Different types
Availability
Resourcing

Applications

Applications
New Relic

Applications
New Relic
CloudKick

Applications
New Relic
CloudKick
ScoutApp

Applications
New Relic
CloudKick
ScoutApp
Nagios

Applications
New Relic
CloudKick
ScoutApp
Nagios
Cacti

Applications
New Relic
CloudKick
ScoutApp
Nagios
Cacti
Circonus

Automation

Want to sleep easy at night?

Want to sleep easy at night?
Go out partying without worrying
about getting a phone call?

Plug into your
monitoring

Bringing together Monitoring
and Elastic behaviour into one
beautiful whole!

Add some intelligence to add /
remove servers as needed based
on current information.

This is why good monitoring is
essential or this wouldn’t be
possible

Just make sure it
doesn’t turn into...

Skynet!!

High Availability

Get a highly available and
resilient setup by following a few
of those recommendations

Remember, even Google has outages

Benefits

Benefits
Easy management

Benefits
Easy management
Ability to stop / start servers quickly

Benefits
Easy management
Responsibilities are separate

Benefits
Easy management
Quickly move to a new cluster

Benefits
Easy management
Quickly move to a new cluster
Reduced risk

What to avoid

Local Sessions

Store sessions in DB / Memcache
Solution

Local Memory

Networked Memcache
Solution

Local Files

Local Uploads

Writing to /tmp

Store on S3 or a networked FS
Solution

Serve up static ﬁles from CDNs
Solution

Servers can vanish at any
given time

Internal APIs

Application
S3GFS FS
Internal Storage API

Application
MySQLMongo Cache
Internal DB API

Service Oriented
Architecture

Sort of :-)

Eventually
Consistent

CAP Therom

Consistency
Availability
Partition Tolerance

Consistency
All nodes see the same data at
the same time

Availability
Node failures do not prevent
survivors from continuing to
operate

Partition Tolerance
The system continues to operate
despite arbitrary message loss

Queue Systems

Good for

Good for
Image Processing

Good for
Image Processing
Distributed Logs

Good for
Image Processing
Distributed Logs
Data Mining

Good for
Image Processing
Distributed Logs
Data Mining
Mass Emails

Good for
Image Processing
Distributed Logs
Data Mining
Mass Emails
Intensive transformation

Good for
Image Processing
Distributed Logs
Data Mining
Mass Emails
Intensive transformation
Search

Common Tools

Common Tools
Gearman

Common Tools
Gearman
Hadoop

Common Tools
Gearman
Hadoop
Zero MQ (0MQ)

Common Tools
Gearman
Hadoop
Zero MQ (0MQ)
RabbitMQ

Common Tools
Gearman
Hadoop
Zero MQ (0MQ)
RabbitMQ
And many others!

Your Client Code
Gearman Client API
(C, PHP, Perl, MySQL UDF, ...)
Gearman Job Server
gearmand
Gearman Worker API
(C, PHP, Perl, Python, ...)
Your Worker Code
Your App Gearman

A Story!

Financial Software

3000+ Clients

Each one has 5 external
data sources

Each data source is a
web service

Ran every 6 hours every day

Cron
1
2
3
4
5
Job 1
Gearman
1
4
3
2
5
Web Services Processing

But! That wasn’t enough

Job kicked off on login

Another Story!

CloudSplit

Near Real Time Cloud
Analytics

Clients install logging
agent locally

Public API

Multiple Persistent
Gearman Servers

Internal DB API

Agent syslogd

Agent syslogd
API

Agent syslogd
API
Load
Balanced

Agent syslogd
API
Gearman
Gearman
Load
Balanced

Agent syslogd
API
Gearman
Gearman
Load
Balanced
Persistent

Agent syslogd
API
Gearman
Gearman
Worker
Worker
Worker
Load
Balanced
Persistent

Agent syslogd
API
Gearman
Gearman
Worker
Worker
Worker
Internal API
Load
Balanced
Persistent

Agent syslogd
API
Gearman
Gearman
Worker
Worker
Worker
Internal API
Load
Balanced
Load
Balanced
Persistent

Agent syslogd
API
Gearman
Gearman
CouchDB
Worker
Worker
Worker
Internal API
Load
Balanced
Load
Balanced
Persistent

CouchDB Setup

Write vs Read

Writes
Multi Master setup

Writes
Multi Master setup
Replicated

Writes
Multi Master setup
Replicated
Deals with writes only

Reads
Multi Master setup

Reads
Multi Master setup
Replicated from write cluster

Reads
Multi Master setup
Replicated from write cluster
Slaves handle website requests

Heavy Map/Reduce
usage for data

Supervisord

phpadvent.org/2009/daemonize-
your-php-by-sean-coates

Map/Reduce

Map
Master gets a problem to solve

Map
Breaks into multiple sub-problems

Map
Distributed to multiple workers

Map
A worker can take the same steps

Map
A worker can take the same steps
Answer passed back to Master

Reduce
Takes in answers from the map workers

Reduce
Combines together to get an answer

Reduce
Combines together to get an answer
There can be multiple reducers

process petabytes of data in few
hours on commodity server farm

CouchDB
Highly Concurrent

CouchDB
Highly Concurrent
Schema free, document based

CouchDB
Highly Concurrent
RESTful API

CouchDB
Highly Concurrent
RESTful API
Map/Reduce Views

CouchDB
Highly Concurrent
RESTful API
Map/Reduce Views
Easy Replication

Hadoop is a framework for running applications
on large clusters of commodity hardware.

The Hadoop framework transparently provides
applications both reliability and data motion

Uses Map/Reduce concept to
farm out work

Distributed FS to handled
node failure automagically

Join 2 datasets together of a
signiﬁcant size

500 GB worth of log ﬁles with
a large location dataset

ØMQ
Async Message System

ØMQ
Thin and lightweight

ØMQ
High Performance

ØMQ
High Performance
Simple

ØMQ
High Performance
Simple
Scalable

One socket can load balance to
multiple end points

Multiple end points can be
funnelled into a single socket

Handle deployments to
multiple servers

Scale is an example of that

Mongrel2 is a web server that
uses it in a similar way as fastcgi

Move around text (JSON) and Binary
data for real time communication

Could have replaced syslogd and the
external API in my previous example

Code time? :-)

Questions?
helgi@orchestra.io
Twitter: @h
Joind.in: http://joind.in/3212

Scale like an ant, distribute the workload - DPC, Amsterdam, 2011

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Semelhante a Scale like an ant, distribute the workload - DPC, Amsterdam, 2011

Semelhante a Scale like an ant, distribute the workload - DPC, Amsterdam, 2011 (20)

Mais de Helgi Þormar Þorbjörnsson

Mais de Helgi Þormar Þorbjörnsson (15)

Último

Último (20)

Scale like an ant, distribute the workload - DPC, Amsterdam, 2011