Massively Scalable, Massively Fault Tolerant

MASSIVELY SCALEABLE
MASSIVELY FAULT-TOLERANT
Trotter Cashion
Philly Lambda, 2011

Erlang
BUT FIRST...
AND OTP
IN ACTION
Buy this Book!
http://bit.ly/otp-book Martin Logan
Eric Merritt
Richard Carlsson
FOREWORD BY ULF WIGER

MANNING

Erlang
NO SERIOUSLY... IN ACTION
AND OTP
http://bit.ly/otp-book
Martin Logan
Eric Merritt
Richard Carlsson

MANNING

Erlang
AVAILABLE IN
PDF...
AND OTP
IN ACTION
http://bit.ly/otp-book
Martin Logan
Eric Merritt
Richard Carlsson

MANNING

@CASHION

http://www.charlesheﬂin.com/wp-content/uploads/2010/06/fail-whale.jpg

CHLOE

• Realtime Web Server

• Proxies WebSocket (and fallback) connections to your app

• http://github.com/mashion/chloe

• Written in Erlang/OTP (plus a lot of JavaScript)

BAI PHILLY!
Leaving in July...

http://www.ﬂickr.com/photos/dennis/156048151/

HAI CALIFORNIA!
See you in August...

http://www.ﬂickr.com/photos/salim/402618628/

AN ERLANG PRIMER

Yoinked from the Erlang Movie. Go Watch It!!!!

THE BASICS

• Functional

• Single Assignment Variables

• Pattern Matching is Big Here

• Everything’s a Process

• One Module Per File

FUNCTIONAL
say_hello() ->
io:fwrite("Ohai!n").

ANONYMOUS FUNCTIONS
make_hello() ->
fun () -> io:fwrite("Ohai!n")
end.

Hello = make_hello().
Hello().

SINGLE ASSIGNMENT VARS
1> Hello = 3.
3
2> Hello = 2.
** exception error: no match of right hand side value 2
3> f().
ok
4> Hello = 2.
2

OH NOES! WHAT WENT WRONG?
destructuring_bind() ->
  [{Name, Title}, _] = [{"Trotter", "Speaker"},
                        {"Aaron", "Organizer"}],
  io:fwrite("~pn", [Name]), %% => Trotter
  io:fwrite("~pn", [Title]). %% => Speaker

PATTERN MATCHING
Erlang Ruby
match_me(1) -> def match_me(val)
  loneliest;   case val
match_me(2) ->   when 1
  next_to_loneliest;     :loneliest
match_me(friend) ->   when 2
  all_ok;     :next_to_loneliest
match_me(_) ->   when :friend
  what_is_this.     :all_ok
  else
    :what_is_this
  end
end

MAKING A PROCESS
process.erl

concurrent_stuffs() ->
  spawn(fun () -> timer:sleep(1000),
                  io:fwrite("So slown") end),
  spawn(fun () -> io:fwrite("Super fastn") end).

erlang shell
5> process:concurrent_stuffs().
Super fast
<0.39.0>
So slow

RECEIVE LOOPS
food_loop() ->
  receive
    pizza -> io:fwrite("let's eatn"),
             food_loop();
    death -> io:fwrite("all done heren");
    _ -> io:fwrite("Nothing to don"),
             food_loop()
  end.

SENDING MESSAGES
make_process() ->
  spawn(?MODULE, food_loop, []).

pizza_death() ->
  Proc = make_process(),
  Proc ! pizza,
  Proc ! hi,
  Proc ! death,
  %% Goes into the void
  Proc ! huh.

HOW THAT WORKS
process 1

pizza_death

HOW THAT WORKS
process 1
spawn/3
pizza_death

HOW THAT WORKS
process 1 process 2
spawn/3
pizza_death food_loop

HOW THAT WORKS
process 1 process 2
spawn/3

food_loop’s
Mailbox

HOW THAT WORKS
process 1 process 2
spawn/3

food_loop’s
Mailbox

pizza

HOW THAT WORKS
process 1 process 2
spawn/3

food_loop’s
Mailbox

pizza

hi

HOW THAT WORKS
process 1 process 2
spawn/3

food_loop’s
Mailbox

pizza

hi

death

HOW THAT WORKS
process 1 process 2
spawn/3

food_loop’s
Mailbox

pizza

hi

death

huh

HOW THAT WORKS
process 1 process 2
spawn/3

food_loop’s
Mailbox

pizza “let’s eatn”

hi

death

huh

HOW THAT WORKS
process 1 process 2
spawn/3

food_loop’s
Mailbox

pizza

hi “Nothing to don”

death

huh

HOW THAT WORKS
process 1 process 2
spawn/3

food_loop’s
Mailbox

pizza

hi

death “all done heren”

huh

HOW THAT WORKS
process 1
spawn/3
pizza_death

food_loop’s
Mailbox

pizza

hi

death

huh

HOW THAT WORKS
process 1
spawn/3
pizza_death

pizza

hi

death

huh

HOW THAT WORKS
process 1
spawn/3
pizza_death

pizza

hi

death

BUT DOES IT SCALE?
Hells yea!

http://www.ﬂickr.com/photos/24736216@N07/4808606178/

TWO WAYS TO SCALE

Name Implementation

Horizontally Buy a lot of machines

Vertically Buy a really big machine

ERLANG SCALES WELL ON BOTH
WITH NEARLY ZERO CODE CHANGE

VERTICAL SCALING

• Your processes will take advantage of all cores available

• Yay! No code changes required

HORIZONTAL SCALING

• New nodes added to the cluster can be auto-detected

• Code must know that a service could live on another node

• You’ll need resource discovery

CONNECTING NODES
$ erl -name b
1> AHost = 'a@awesome-town.local'.
'a@awesome-town.local'

2> io:fwrite("Current nodes: ~p~n", [nodes()]).
Current nodes: []

3> io:fwrite("Ping? ~p~n", [net_adm:ping(AHost)]).
Ping? pong

4> io:fwrite("Current nodes: ~p~n", [nodes()]).
Current nodes: ['a@awesome-town.local'

5> q().

SERVER REFERENCES

Type Deﬁnition

<Pid> globally unique process identiﬁer

RegisteredName locally registered process

process locally registered on
{RegisteredName, Node}
another node

{global, GlobalName} globally registered process

LISTEN FOR INCOMING MESSAGE
listen() ->
  register(listener, self()),
  listen_loop().

listen_loop() ->
  receive
    {From, pizza} -> From ! "Nom noms", listen_loop();
    {From, _} -> From ! "Boo!", listen_loop();
    _ -> io:fwrite("Listen is dying!")
  end.

SENDING MESSAGE TO NODES
$ erl -name b
1> AHost = 'a@awesome-town.local'.
'a@awesome-town.local'

2> {listener, AHost} ! {self(), pizza}.
ok

3> receive PizzaResp -> PizzaResp end.

RESOURCE DISCOVERY

Method Purpose

add_target_resource(Type) specify a resource type you want to use

add_local_resource(Type, Pid) specify a resource type you have

fetch_resources(Type) fetch all resources of speciﬁed type

trade_resources() trade resource information

See Chapter 8 of Erlang And OTP In Action

HORIZONTAL CAVEATS

• Passing large amounts of data between nodes is inefﬁcient

• Too many nodes will impact performance (Keep it < 100)

• There are ways around both these problems

OTP BASICS
Cause managing all the processes by hand is dumb

Thanks, Homer

OTPWTFBBQ?

• Open Telecom Platform

• Abstracts common systems behavior (servers, supervisors, ...)

• Easier than pure Erlang!

• Gives Erlang its reputation for fault tolerance

BASIC CONSTRUCTS

Construct Purpose

Encapsulates a large set of code
Application
meant to run as one unit.

Supervisor Starts and restarts workers.

Worker Actually does stuff

HTTP://GITHUB.COM/MASHION/CHLOE

Chloe
(Application)

chloe_sup
(Supervisor)

websocket_sup session_sup session_manager
Other Stuffs
(Supervisor) (Supervisor) (Worker)

WORKERS

• Use the `gen_server` behaviour (for the most part)

• Contain your application logic

• You deﬁne a module with callbacks

• OTP runs a server calls back to module when messages arrive

HOW WORKERS WORK
client process (your code)

gen_server functions

gen_server code (part of OTP)

gen_server callbacks

worker code (your code)
worker process

SENDING MESSAGES TO WORKERS
say_hello_sync(Name) ->
gen_server:call(?SERVER, {say_hello, Name}).

say_hello_async() ->
gen_server:cast(?SERVER, say_hello).

gen_server_examples.erl

EXAMPLE CALLBACKS
handle_call({say_hello, Name}, _From, State) ->
  Response = string:concat("Hello, ", Name),
  {reply, {ok, Response}, State}.

handle_cast(say_hello, State) ->
  io:fwrite("Hello world!n"),
  {noreply, State}.

gen_server_examples.erl

WORKER CALLBACKS

Callback Purpose
init(Args) Initialize the server process
handle_call(Request, From, State) Handle a synchronous call
handle_cast(Request, State) Handle an asynchronous cast
handle_info(Info, State) For timeouts and random messages
terminate(Reason, State) Clean up the process when it dies
code_change(OldVsn, State, Extra) Called when upgrading code

WORKER FUNCTIONS

Callback Purpose

start_link(Module, Args, Options) Start a supervised worker

start(Module, Args, Options) Start an unsupervised worker

call(ServerRef, Request, Timeout) Synchronously call a worker

cast(ServerRef, Request) Asynchronously cast to a worker

man gen_server

SUPERVISORS

• Use the `supervisor` behaviour

• Contain little to no application logic

• Know how to start and restart failed workers

• Keep your application from ever fully dying

CHILD SPECS
{Id, StartFunc, Restart, Shutdown, Type, Modules}

Unique name used to identify
child process.

CHILD SPECS
{session_manager, StartFunc, Restart, Shutdown, Type,
Modules}

Unique name used to identify
child process.

CHILD SPECS
{session_manager, StartFunc, Restart, Shutdown, Type,
Modules}

Tuple {M, F, A} used to start
the child.

CHILD SPECS
{session_manager, {session_manager, start_link, []},
Restart, Shutdown, Type, Modules}

Tuple {M, F, A} used to start
the child.

CHILD SPECS
Restart, Shutdown, Type, Modules}

Determines when a
terminated process should be
restarted.

CHILD SPECS
permanent, Shutdown, Type, Modules}

Determines when a
terminated process should be
restarted.

CHILD SPECS
permanent, Shutdown, Type, Modules}

Determines how to kill child
processes.

CHILD SPECS
permanent, 5000, Type, Modules}

Determines how to kill child
processes.

CHILD SPECS
permanent, 5000, Type, Modules}

Is this child a worker or a
supervisor?

CHILD SPECS
permanent, 5000, worker, Modules}

Is this child a worker or a
supervisor?

CHILD SPECS
permanent, 5000, worker, Modules}

Used by code reloading. Make
it the same as the other
module name.

CHILD SPECS
permanent, 5000, worker, [session_manager]}

Used by code reloading. Make
it the same as the other
module name.

RESTART STRATEGIES

Strategy When a child dies....

one_for_one restart it

one_for_all restart all children

restart it and all children after it
rest_for_one
in child list
restart it (also allows you to
simple_one_for_one
dynamically add children)

ONE_FOR_ONE

Supervisor

Child 1 Child 2 Child 3

ONE_FOR_ONE

Supervisor

Child 2 Child 3

ONE_FOR_ALL

Supervisor


ONE_FOR_ALL

Supervisor

Child 2 Child 3

ONE_FOR_ALL

Supervisor

Child 3

ONE_FOR_ALL

Supervisor

Child 1

ONE_FOR_ALL

Supervisor

Child 1 Child 2

REST_FOR_ONE

Supervisor


REST_FOR_ONE

Supervisor

Child 1 Child 3

REST_FOR_ONE

Supervisor

Child 1

REST_FOR_ONE

Supervisor

Child 1 Child 2

SIMPLE_ONE_FOR_ONE

Supervisor

SIMPLE_ONE_FOR_ONE

Supervisor

Child 1

SIMPLE_ONE_FOR_ONE

Supervisor

Child 1 Child 2

SIMPLE_ONE_FOR_ONE

Supervisor


SIMPLE_ONE_FOR_ONE

Supervisor

Child 2 Child 3

STARTING A SUPERVISOR
init([]) ->
  Worker = {gen_server_examples,
{gen_server_examples, start_link, []},
            permanent, 5000, worker,
[gen_server_examples]},
  RestartStrategy = {one_for_one, 5, 30},
  {ok, {RestartStrategy, [Worker]}}.

SUPERVISOR CALLBACK

callback purpose
return data structure telling what
init(Args)
to supervise

SUPERVISOR FUNCTIONS

Callback Purpose

start_link(Module, Args) Start a supervised supervisor

start_child(SupRef, ChildSpec) Start a child under the supervisor

man supervisor

FAULT TOLERANCE

• When you hit an error, let the worker die

• Supervisor will restart it

• Either try what killed it again or move on

• Life is grand

LINKS

• http://bit.ly/erlang-otp-gist

• http://bit.ly/otp-book

THANK YOU!

• Questions?

• http://bit.ly/erlang-otp-gist

• http://bit.ly/otp-book

• http://mashion.net

Massively Scalable, Massively Fault Tolerant

Recomendados

Recomendados

Mais conteúdo relacionado

Último

Último (20)

Destaque

Destaque (20)

Massively Scalable, Massively Fault Tolerant

Notas do Editor