Building and scaling distributed infrastructure. Presented at OdessaPy on Dec 7, 2013.

1. Scaling and Distributing
Dima Nedbaylo
Dima Malenko

2. So, you want to tell
us about distributed
systems?

3. OpenOffice.org on iPad

5. Systems are distributed for a reason
Our reasons were following:
• low latency for end-user connections
• horizontal scalability
• availability

6. Systems are distributed for a reason
Our reasons were following:
• low latency for end-user
connections
• horizontal scalability
• availability

7. Historia de número uno
Práctico

8. We have to be close to the user
Every 5ms of ping between the user and the server counts

9. Computing power provider with
multiple locations and great
managements capabilities?

10. Computing power provider with
multiple locations and great
managements capabilities?

11. [Only] 8 locations
[Just] $0.6/hour
c3.2xlarge
14 ECU
15 GB RAM
2*80 GB SSD
$0.6 * 720 = $432/mo

12. So, you’ve got new server…
• Minimal setup of Ubuntu 12.04
• Magic fabric script to turn minimal Ubuntu
into rollApp app server
• Works great and allows to get server up and
running within couple of hours

13. Now you’ve got 10 servers…
• And need to update one of your
components
• Or run a maintenance procedure
• Or install a couple of new packages
• Or correct config

14. Ansible
http://www.ansibleworks.com
• Learned in just 1 hour
• In 2 hours we had a script to setup new app
server
• Way better and more reliable than fabric for
the same purpose
• Way easier than chef

15. Ansible vs. Fabric
• Requires hosts inventory database
• If you need something very custom – have
to code in Python
00
00
00
00
00
00
00

16. Ansibe Ad-hoc
Ansible has a lot of modules for ad-hoc mode:
• downloading/uploading files
• managing packages, users, services
• Launching EC2 instances and other AWS
stuff
• databases and db users operations

17. Inventory Exmple
[appservers_eu]
appsrv-007.rollapp.com
ansible_ssh_user=guess_who
ansible_ssh_private_key_file=...
[appservers:children]
appservers_eu
[appservers:vars]
root_password='password'

18. Playbooks
• Plain yaml file
• Contains server configuration
• Per se it is just set of tasks that invoke
ansible modules

19. Simple Playbook
--- include: playbooks/timezone.yml
- include: playbooks/ntp.yml
- hosts: appservers
sudo: yes
vars:
prefix_dir: /opt
tasks:
# user: dnedbaylo
- name: create dnedbaylo user
user: name=dnedbaylo groups=admin shell=/bin/zsh
- name: authorized keys for dnedbaylo
authorized_key: user=dnedbaylo
key=”ssh-rsa ….”

20. Ansible vs. Fabric: playbook mode
• Plain language (YAML) for playbooks
• A lot of modules ready to use (like creating
EC2 instances, users management, apt
repositories management, etc.)
• No need to worry about details (does user
already exist?)
• Playbooks are “idempotent”

21. Ansible vs. Chef

22. Ansible vs. Chef
• No need to learn chef
• No need to learn ruby
• No weird ruby requirements (not so easy to
install chef on Linux Mint)
• No need to use additional tools to make life
with chef solo easier (hello knife-solo)

23. Historia de número dos
Instructivo

24. The difficulty with
distributed systems is that
they are … distributed

25. My First Law of Distributed
Objects Design:
Do not distribute your objects
http://martinfowler.com/bliki/FirstLaw.html

26. At all times
protect integrity of the
system…
at all cost

27. Put on your oxygen mask first before helping
others

28. App Server 1
App Server 2
Web
App Server 3
App Server N

29. Web keeps track of
consequent errors
for each app server
App Server 1
App Server 2
Web
App Server 3
App Server N

30. Web keeps track of
consequent errors
for each app server
App Server 1
App Server 2
Web
App Server 3
App Server monitors
its internal state and
deactivates itself if
bad things happen
App Server N

32. App Server 1
App Server 2
Web
App Server 3
App Server N

33. App Server 1
App Server 2
Web
App Server 3
App Server N

34. App Server 1
App Server 2
Web
App Server 3
App Server N

35. App Server 1
App Server 2
Web
App Server 3
App Server N

36. What Happened?

37. App Server 1
App Server 2
Web
App Server 3
App Server N

38. OOM killer engaged
• pings work
• simple status checks
work(!)
App Server 1
App Server 2
Web
App Server 3
App Server N

39. Requests still come
in, but never get
actually processed
App Server 1
App Server 2
Web
App Server 3
App Server N

40. DB connections pool
got saturated. Old
requests hung, new
requests fail
App Server 1
App Server 2
Web
App Server 3
App Server N

41. Irregular errors and
failures because of
resource starvation
App Server 1
App Server 2
Web
App Server 3
App Server N

42. Верить нельзя никому, порой
даже себе… Мне – можно!

43. Lessons Learned
• Timeouts on all connections to other
components
• Monitoring beyond just vitality signs
• Keep track of “in progress” requests to
prevent cascading errors

44. Things to Remember
• [Almost] all modern applications are
distributed
• Never trust any external interface
• Be ready to sacrifice part to keep the entire
system afloat
• Monitor each interface from both sides on
different layers (not just pinging)

45. Never Trusting Is Not Easy
• No (!) out of the box solutions for controlling
response timeouts
– requests only has connection timeouts
– if you are on gevent or the like – you are good to go
with greenlet timeouts
• [Almost] always opt for aggressive health
control parameters
– request processing time
– max address space
– max number of queued requests

46. Historia de número tres
Inesperado

47. Get application startup
time optimization

50. Here ought to be a screenshot
but it is not here

51. Here ought to be a screenshot
but it is not here
Always (I mean ALWAYS) make screenshots
when you come across something interesting

52. Application Startup
User
Browser
preparing to launch
connect to application
App
Server
Web
launch application
52

53. Close to one another
Application Startup
User
Browser
Web
Faster
App
Server
Slower
53

54. Application Startup
User
Browser
Web
App
Server
54

55. Do not take anything for granted

57. Any questions?
Now:
and later:
dnedbaylo@rollapp.com
@dmalenko
dmalenko@rollapp.com
www.dmalenko.org