O slideshow foi denunciado.
High availability
architecture
for legacy stuff A 10.000 feet overview
$whoami
Marco Amado
Lead Developer @ Moloni
/mjamado
www.dreamsincode.com
$whoiaint
Not a sysadmin (not worthy of the title, at least)
Not a DevOps guru
Not a high availability ninja
Not a scalabi...
Notes
●
This is code
●
Sometimes, there’s code you should change
●
“Talk to your hoster” symbol
Motivation
Or how a watched kettle
never boils, until your
kitchen’s on fre
Hypothetical Product
Find-a-Rhyme
Given a word, the application returns a set of words
that rhyme.
You can flter by word c...
Where we’re standing
Ye olde LAMP stack
●
Commonly found on shared hosting
●
Network latency between PHP and DB is
amazing...
Dictatorship!
First order: all written communications should
be in verse. And it has to rhyme.
People fock to Find-a-rhyme...
Problems
Overview
What will we encounter if we
want to avoid touching the
code (mostly)
Overview
●
Load balancing
●
DB clustering
●
Sessions
●
User assets
●
Single point of failure
●
Monitoring
●
Security
Load
Balancing
Because we’ve got to start
somewhere
Hardware
Pros
●
Faster than software (in
general)
●
Most have integrated
intrusion detection
and/or prevention
Cons
●
Pric...
Software solutions
frontend web
bind find-a-rhyme.com:80
default_backend web
backend web
mode http
balance leastconn
server s1 ip.app1:80
ser...
SSL Termination
Do it on the load balancers!
global
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-ciph...
Database
servers
All your data are
belong to us!
MySQL/MariaDB
Replication Group
Pros:
●
Battle tested
●
Big company backed
(Oracle)
Cons:
●
Confguration is a PITA
XtraDB ...
Galera Cluster
●
Included with MariaDB 10.1
●
Make sure to also install percona-xtrabackup
●
A dozen lines of confguration...
HAProxy configuration for DB
backend cluster
mode tcp
option tcpka
option mysql-check user healthUser
balance static-rr
se...
Application
servers
We’re not touching
that codebase!
Session Handling
Sticky sessions
Pros:
●
Easy confguration on
load balancer
Cons:
●
Bad UX on server fail
●
Not exactly lo...
Sessions with memcached
Easy confguration on php.ini (or included fles):
session.save_handler = memcache
session.save_path...
User assets
CDN
●
Heavy changes
to codebase
●
Lack of control
●
More expenses
Samba, NFS
●
Single point of
failure
●
Slow ...
GlusterFS
●
Distributed fle system
●
Replicated mode
●
Transparent operation
●
Easy CLI confguration:
●
fstab confguration...
Where we’re standing
LB
App1 App2
DB1 DB2 DB3
SPOF
Eliminating
the SPOF
Load balancing the
load balancers
Keepalived
Implementation of Virtual Router Redundancy Protocol
(VRRP) – in a nutshell, automatic assignment of IP
address...
keepalived.conf (extract)
vrpp_instance VI1 {
virtual_router_id 50 # mostly arbitrary – make sure it’s unique
interface NI...
Virtual IP for DB access
vrpp_instance VI2 {
virtual_router_id 60 # mostly arbitrary – make sure it’s unique
interface NIC...
Don’t forget SSL termination
Two load balancers with failover, two servers where to
make SSL termination:
Duplicate your c...
Much better...
LB1
App1 App2
DB1 DB2 DB3
LB2
Monitoring
When things go sideways,
be the frst to know
Monit
●
Monitoring and managment
●
Can do automatic maintenance and repair
●
Can execute arbitrary actions on errors
●
Can...
Monit sample config
check process php with pidfile /var/run/php/php7-fpm.pid
start program = ”/usr/bin/service php7-fpm st...
User interface
M/Monit
●
Aggregate all your Monit instances
●
Awesome UI – it’s even responsive
●
Start and stop services from the UI
●
A...
M/Monit UI
M/Monit UI
M/Monit UI
Going further Why stop now?
Keeping it secure(-ish)
●
As few public IP addresses as possible
●
Fail2ban
●
SELinux / AppArmor
●
No passwordless sudo – ...
There’s an app a tool for that
●
Centralize logs with Elastic Stack (Logstash,
Elasticsearch and Kibana)
●
Manage the cron...
One more thing Two, actually…
Geographic distribuition
●
Avoid datacenter SPOF
●
Watch your latency!
●
Should I say it aga...
Q&A
“Ask, and it shall
be given to you”
Mathew, 7:7
Thank you
Marco Amado
Lead Developer @ Moloni
/mjamado
www.dreamsincode.com
Próximos SlideShares
Carregando em…5
×

High Availability Architecture for Legacy Stuff - a 10.000 feet overview

136 visualizações

Publicada em

An overview of the tools and tricks you could use to turn a monolithic big pile of... Apache, PHP, and MariaDB into an awesome high-availability, load balanced, shiny new pile of... Apache, PHP, and MariaDB. Zero, or almost zero changes to the codebase.

Publicada em: Internet
  • Seja o primeiro a comentar

High Availability Architecture for Legacy Stuff - a 10.000 feet overview

  1. 1. High availability architecture for legacy stuff A 10.000 feet overview
  2. 2. $whoami Marco Amado Lead Developer @ Moloni /mjamado www.dreamsincode.com
  3. 3. $whoiaint Not a sysadmin (not worthy of the title, at least) Not a DevOps guru Not a high availability ninja Not a scalabilty jedi Take that into account
  4. 4. Notes ● This is code ● Sometimes, there’s code you should change ● “Talk to your hoster” symbol
  5. 5. Motivation Or how a watched kettle never boils, until your kitchen’s on fre
  6. 6. Hypothetical Product Find-a-Rhyme Given a word, the application returns a set of words that rhyme. You can flter by word class, type of rhyme, word length...
  7. 7. Where we’re standing Ye olde LAMP stack ● Commonly found on shared hosting ● Network latency between PHP and DB is amazing – as in zero amazing ● Everything is a single point of failure ● Find-a-rhyme is probably safe, right? Right? Linux Apache MySQL/MariaDB PHP Suddenly...
  8. 8. Dictatorship! First order: all written communications should be in verse. And it has to rhyme. People fock to Find-a-rhyme. Modern Infantry by Litev CC BY-SA 3.0 https://commons.wikimedia.org/wiki/File:Modern_infantry.png
  9. 9. Problems Overview What will we encounter if we want to avoid touching the code (mostly)
  10. 10. Overview ● Load balancing ● DB clustering ● Sessions ● User assets ● Single point of failure ● Monitoring ● Security
  11. 11. Load Balancing Because we’ve got to start somewhere
  12. 12. Hardware Pros ● Faster than software (in general) ● Most have integrated intrusion detection and/or prevention Cons ● Pricey as hell ● Confguration not easily portable Pros ● FOSS (mostly) ● Confguration is easy to reason about Cons ● Can be slow (depending on machine) ● If FOSS, you’re on your own Software
  13. 13. Software solutions
  14. 14. frontend web bind find-a-rhyme.com:80 default_backend web backend web mode http balance leastconn server s1 ip.app1:80 server s2 ip.app2:80 server { listen 80; location / { proxy_pass http://web; } } upstream web { least_conn; server ip.app1; server ip.app2; } ¯_( ツ )_/¯
  15. 15. SSL Termination Do it on the load balancers! global ca-base /etc/ssl/certs crt-base /etc/ssl/private ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128: DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 tune.ssl.default-dh-param 2048 frontend web bind find-a-rhyme.com:80 bind find-a-rhyme.com:443 crt path/to/certificate.pem
  16. 16. Database servers All your data are belong to us!
  17. 17. MySQL/MariaDB Replication Group Pros: ● Battle tested ● Big company backed (Oracle) Cons: ● Confguration is a PITA XtraDB Cluster & Galera Cluster Pretty much the same product Pros: ● Multi master from the start ● Partners with MariaDB ● Confguration is a breeze Cons ● Consensus can be a problem
  18. 18. Galera Cluster ● Included with MariaDB 10.1 ● Make sure to also install percona-xtrabackup ● A dozen lines of confguration: [mysqld] binlog_format=ROW default-storage-engine=innodb innodb_autoinc_lock_mode=2 bind-address=0.0.0.0 wsrep_on=ON wsrep_provider=/usr/lib/galera/libgalera_smm.so wsrep_cluster_name="my_cluster" wsrep_cluster_address="gcomm://ip.db1,ip.db2,ip.db3" wsrep_sst_method=xtrabackup-v2 wsrep_sst_auth="sst:somepassword" wsrep_node_address="each.machine.ip" wsrep_node_name="eachMachineName"
  19. 19. HAProxy configuration for DB backend cluster mode tcp option tcpka option mysql-check user healthUser balance static-rr server db1 ip.db1:3306 check server db2 ip.db2:3306 check server db3 ip.db3:3306 check frontend cluster bind loadbalancer.ip:3306 default_backend cluster Change the connection URL in your codebase to this. This confguration means the application servers must connect to the cluster via load balancers, which in turn connects to the DB servers. Network latency will be an issue.
  20. 20. Application servers We’re not touching that codebase!
  21. 21. Session Handling Sticky sessions Pros: ● Easy confguration on load balancer Cons: ● Bad UX on server fail ● Not exactly load balanced Memcached Pros: ● Easy confguration on php.ini Cons: ● Install memcached, I guess?...
  22. 22. Sessions with memcached Easy confguration on php.ini (or included fles): session.save_handler = memcache session.save_path = “tcp://ip.app1,tcp://ip.app2” memcache.allow_failover = 1 memcache.session_redundancy = 3 Number of memcached servers + 1. It’s an off-by-one bug in PHP, since 2009 (never fxed): https://bugs.php.net/bug.php?id=58585
  23. 23. User assets CDN ● Heavy changes to codebase ● Lack of control ● More expenses Samba, NFS ● Single point of failure ● Slow as hell IPFS
  24. 24. GlusterFS ● Distributed fle system ● Replicated mode ● Transparent operation ● Easy CLI confguration: ● fstab confguration: $ sudo gluster peer probe ip.other.app.server $ sudo gluster volume create volName replica 2 transport tcp ip.app1:/path ip.app2:/path force $ sudo gluster volume start volName $ sudo gluster volume set volName auth.allow ip.app1,ip.app2,127.0.0.1 localhost:/volName /path glusterfs noauto,x-systemd.automount 0 0
  25. 25. Where we’re standing LB App1 App2 DB1 DB2 DB3 SPOF
  26. 26. Eliminating the SPOF Load balancing the load balancers
  27. 27. Keepalived Implementation of Virtual Router Redundancy Protocol (VRRP) – in a nutshell, automatic assignment of IP addresses. ● First and foremost, confgure IP forwarding and non-local bind on sysctl.conf: net/ipv4/ip_forward = 1 net/ipv4/ip_nonlocal_bind = 1 “Jumping” IP addresses can be frowned upon by datacenters. Be sure to really talk to your hoster about this.
  28. 28. keepalived.conf (extract) vrpp_instance VI1 { virtual_router_id 50 # mostly arbitrary – make sure it’s unique interface NIC advert_int 1 state MASTER # BACKUP on the other loadbalancer priority 200 # 100 on the other load balancer unicast_src_ip this.loadbalancer.ip unicast_peer { other.loadbalancer.ip } virtual_ipaddress { your.public.ip dev NIC } }
  29. 29. Virtual IP for DB access vrpp_instance VI2 { virtual_router_id 60 # mostly arbitrary – make sure it’s unique interface NIC advert_int 1 state MASTER # BACKUP on the other loadbalancer priority 200 # 100 on the other load balancer unicast_src_ip this.loadbalancer.ip unicast_peer { other.loadbalancer.ip } virtual_ipaddress { a.free.private.ip dev NIC } } Change the connection URL in your codebase to this.
  30. 30. Don’t forget SSL termination Two load balancers with failover, two servers where to make SSL termination: Duplicate your certifcates!
  31. 31. Much better... LB1 App1 App2 DB1 DB2 DB3 LB2
  32. 32. Monitoring When things go sideways, be the frst to know
  33. 33. Monit ● Monitoring and managment ● Can do automatic maintenance and repair ● Can execute arbitrary actions on errors ● Can monitor system, processes, flesystem, scripts...
  34. 34. Monit sample config check process php with pidfile /var/run/php/php7-fpm.pid start program = ”/usr/bin/service php7-fpm start” stop program = ”/usr/bin/service php7-fpm stop” if failed unixsocket /var/run/php/php7-fpm.sock then restart if 2 restarts within 4 cycles then alert check filesystem disk with path / if space free < 20% then alert check network private interface eno1 start program = ”/sbin/ifup eno1” stop program = ”/sbin/ifdown eno1” if failed link for 3 cycles then restart if saturation > 90% for 20 cycles then alert
  35. 35. User interface
  36. 36. M/Monit ● Aggregate all your Monit instances ● Awesome UI – it’s even responsive ● Start and stop services from the UI ● Analytics, historical data, trend predictions, real-time charts ● Commercial product, but payment is one-time and the license is perpetual – and it’s cheap, on top* I’m in no way affliated with M/Monit. Just love the product! *In September 2017, it costs 65€ for 5 monitored hosts, up to 699€ for 1000 hosts.
  37. 37. M/Monit UI
  38. 38. M/Monit UI
  39. 39. M/Monit UI
  40. 40. Going further Why stop now?
  41. 41. Keeping it secure(-ish) ● As few public IP addresses as possible ● Fail2ban ● SELinux / AppArmor ● No passwordless sudo – ever ● Public key SSH ● External access through the load balancers: $ ssh -t you@public.ip ssh you@some.private.ip
  42. 42. There’s an app a tool for that ● Centralize logs with Elastic Stack (Logstash, Elasticsearch and Kibana) ● Manage the crontab with Crontab UI ● DB status and analytics with Cluster Control ● Continuous Integration/Deployment – GitLab is FOSS and self-hosted for greater control
  43. 43. One more thing Two, actually… Geographic distribuition ● Avoid datacenter SPOF ● Watch your latency! ● Should I say it again?… Containers ● Can be deployed pretty much on demand ● Easily switch hosting (ahem… talk to your hoster?)
  44. 44. Q&A “Ask, and it shall be given to you” Mathew, 7:7
  45. 45. Thank you Marco Amado Lead Developer @ Moloni /mjamado www.dreamsincode.com

×