Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. In other words, it is the nervous system for how distributed processing communicates.

1. Gearman
Distributed Process Solution
Created by Google Drive Presentation.

2. Introduction
$ whoami
########################################################
####
Name
: Nguyen Duy Khanh
####
####
Job
: Student of HCMUS, Part-timer at PNC ####
####
My skills
: PHP, MySQL, JS
####
####
Love
: New technology
####
########################################################
$|

3. Problems
HEAVY WORKLOAD
function main() {
...
doSomeThing();
doSomeThingBig();
doSomeThingBigBig();
doSomeThingVeryBig();
…
return result;
}

4. Problems
console.log(result);

6. Solution
Give me a job, my
workers will do it
for you.
Worker 1
Worker 2
Worker...
Worker n

7. Gearman
❏ GEARMAN → MANAGER
❏ Open Source.
❏ Multi-language API.
❏ Multi-threaded ( 50k jobs / second ).
❏ Ultra fast ( written in C/C++ ).
❏ No limits on message size.
❏ Failover

8. Installation
1.
2.
Download source code [ https://launchpad.net/gearmand ].
Compile and install from tarball.
tar xzf gearmand-X.Y.tar.gz
cd gearmand-X.Y
./configure
make
make install
3.
Starting gearman.
gearman -d

9. Gearman Architecture
Customer
Job Manager
Worker

10. How does it work ?
Client
Hey, I can send email.
Let me do it !!!
Worker Python
Worker Node.JS
Job Manager
Worker PHP

11. How does it work ?
Client
Worker Python
(send_email)
Worker Node.JS
Job Manager
Worker PHP

12. How does it work ?
Client
And I can push
notification. Let me try !!!
Worker Python
Worker Node.JS
Job Manager
Worker PHP

13. How does it work ?
Client
Worker Python
(send_email)
Worker Node.JS
(push_notify)
Job Manager
Worker PHP

14. How does it work ?
Client
Haha, I can do them all !!!
Worker Python
Worker Node.JS
Job Manager
Worker PHP

15. How does it work ?
Client
Worker Python
(send_email)
Worker Node.JS
(push_notify)
Job Manager
Worker PHP
(send_email, push_notify)

16. How does it work ?
Client
I need send an email to
abc@xyz.com , please
help me !
Worker Python
(send_email)
Worker Node.JS
(push_notify)
Job Manager
Worker PHP
(send_email, push_notify)

17. How does it work ?
Client
OK, Let me see. Aaah …
We have 2 workers can do
it for you. But, Worker
Python is busy, let Worker
PHP do it.
Worker Python
(send_email)
Worker Node.JS
(push_notify)
Job Manager
Worker PHP
(send_email, push_notify)

18. How does it work ?
Client
Hey Worker PHP, please
send an email to
abc@xyz.com
Worker Python
(send_email)
Worker Node.JS
(push_notify)
Job Manager
Worker PHP
(send_email, push_notify)

19. How does it work ?
Client
OK Boss, email has sent
to abc@xyz.com
Worker Python
(send_email)
Worker Node.JS
(push_notify)
Job Manager
Worker PHP
(send_email, push_notify)

20. How does it work ?
Client
Hi client, your email has
sent to abc@xyz.com.
Worker Python
(send_email)
Worker Node.JS
(push_notify)
Job Manager
Worker PHP
(send_email, push_notify)

21. Hello world !!!
Install Node.js API
$ npm install node-gearman
Install PHP API
$ pecl install gearman
Install Python API
$ pip install gearman
or
$ easy_install gearman

22. Hello world !!!
worker1.js

23. Hello world !!!
worker2.py

24. Hello world !!!
client.php

25. Hello world !!!
●
Python and C/C++ and a good choices for workers.
●
PHP, Node.js should be client than worker. [ PHP Worker often crash after
1-2 hours because memory leak ]

26. Use-cases
❏ Scatter/Gather
❏ Map - Reduce
❏ Asynchronous Queues
❏ Pipeline Processing

27. Scatter / Gather
● Perform a number of task concurrently.
○ Ex : Db Query, Image Processing, Location Search,
Full-text Search…
● Take advantage of many tier.
● Speed up your web applications.
● Tasks don’t need to be related.

28. Scatter / Gather
Db Query
Client
Resize Image
Search

29. Map / Reduce

30. Asynchronous Queues
● Most popular use-case.
● Do something needn’t immediate processing.
○ Ex : send emails, push notification, index data,
crawling, ...
● Allows for batch operations.

31. Asynchronous Queues
function tweet(msg) {
// Insert to database
db.insert(msg);
// Background tasks
gearman.push_notify();
// Return
return true;
}

32. Pipeline Processing
● Do related thing.
○ Ex : search engine, data analytic, ...
● Chain workers to send data to next step.

33. Pipeline Processing
Main
Client
Worker
Step 1
Client
Worker
Step 2
Client
Worker
Last Step
Output

34. Examples
GEARMAN Auto News Crawling System
Reload
Request
MongoDB
●
Manager : GEARMAN
●
Workers
“parse_rss”
Gearman Client
Webserver : Node.JS
No
Workers : Python
●
RSS
Sources
●
Web Framework : Express
●
Database : MongoDB
●
Client Script : jQuery
Have new items ?
Crawl new items
Workers
“crawling”
End

35. Persistent Queues
❏
By default, jobs are only stored in memory.
❏
Various contributions from community :
❏
MySQL / Drizzle
❏
PostgreSQL
❏
SQLite
❏
Tokyo Cabinet
❏
memcached
❏
Hope MongoDB (or an NOSQL DB) coming soon.

36. Gearman Admin GUI
●
Command Line Tool : http://gearman.info/bin/gearadmin.html
●
Gearman-Monitor [ PHP ] : https://github.com/yugene/Gearman-Monitor
●
GearmanUI [ PHP ] : http://rripado.info/gearmanui/

38. THE
N
D
THANK YOU !!!

39. About Us
Author : Khanh Nguyen Duy
Find me at : admin@ndksolution.com
Presentation made for “Javascript HoChiMinh City Meetup”
You can find us at :
● http://meetup.com/JavaScript-Ho-Chi-Minh-City/
● https://www.facebook.com/JavaScriptHCMC
● https://plus.google.com/u/0/communities/116105314977285194967