IAC 2024 - IA Fast Track to Search Focused AI Solutions
Drupal Cloud to the rescue? Servers, Files, CDNs and Fun!
1. Cloud to the rescue?
Servers, Files, CDNs and Fun
Questions!
Alex McFadyen, CTO
drupal.org : a_c_m || drupal.org/user/195063
email : alex@demotix.com
twitter : @a_c_m
2. Introduction : Demotix.com
Demotix.com
A two minute (or less) introduction
http://prezi.com/_ugoyxm4qprv/demotixcom-2-minutes/
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
3. Introduction : Demotix.com
Some Demotix numbers
●
1.5million+ page views a month
●
Very very long tale of content
●
Spikes where we get 100k's views in a single day
– Oh how I love Varnish
●
14GB+ database
●
750,000+ nodes, even more terms
●
3-7k new nodes per day
All running from 3 “real” servers. 1 Web, 1 DB, 1 Util.
(Having moved “out” of the cloud 3 years ago)
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
4. Introduction : Why care?
Why not just pay for it to be someone else’s
problem?
●
Control (you NEED it)
●
Complexity (doesn't fit generic solutions)
●
Policy
●
If you can. Do.
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
5. Introduction : “The Cloud”
To the world
“Cloud computing refers to the delivery of computing and storage
capacity as a service to a heterogeneous community of end-
recipients”
- http://en.wikipedia.org/wiki/Cloud_computing
To me (a Drupal dev)
“The cloud is just a buzz word meaning quick to set up VPSs,
perhaps 'unlimited' storage space and even a CDN if your lucky”
- Me, just now.
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
6. Introduction : “The Cloud”
3 parts we were interested in
Power = Virtual Private Servers
Storage = Disks. Big ones.
Delivery = CDNs and other tricks
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
7. Power : Why VPSs rock
@ New servers anytime
@ Scale up and down anytime
@ Short/no contracts
@ Total control (I Y Drush, root access, etc)
@ Cheap !
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
8. Power : Why VPSs don't
@ Total control (I Y Drush, root access, etc)
“With great power comes great responsibility”
- Uncle Ben
●
You always secretly wanted be a sysadmin right?
●
You want to be woken up at 3am if it goes wrong?
●
root@server:~# rm -rf /
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
9. Power : Why VPSs don't
@ Location (Where did I just put that data?)
●
EU has laws about where you can send and store users
data.
●
Your business may not allow for you to store data in the
cloud.
●
Cloud may not be where your users are.
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
10. Power : Why VPSs don't
@ Slices (How big is your share?)
●
Most (not all) cloud providers slice servers up into equal
cpu/ram/disk space chunks and sell them off.
●
Some let you add on resources you need, others don't.
@ Poor disk IO
●
Are your disks local or over the network?
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
11. Power : Why VPSs don't
@ More servers, more problems
●
Once your past dedicated use servers (web, db etc) you
have to do work on the app to support more servers.
See last session “How to sleep without the server-crash-fear
(High Availability and High Performance Hosting for Drupal)”
@ Support
●
Do you have any?
●
How about at 3am?
●
On Christmas day
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
12. Power : Why VPSs don't
@ Bad neighbours & overselling
●
Starts out great, then reddit moves in next door.
@ Cloud A != Cloud B
●
Disks, CPU/Ram/Disk ratio, network speed, location,
management systems etc
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
13. Power : What to do?
@ Benchmarks
●
Drupal benchmark? Omega8.cc Room today at 4pm.
●
http://cloudharmony.com cpu/ram/disk/web/db
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
14. Power : What to do?
@ Try before you buy
●
Some companies will give you a free week.
●
Put as close as you can 1:1 copy of your current site on
it and...
@ Test, test and test some more
●
AB / Siege test your caching system.
●
Instead replay your log files with HTTPERF or Jmeter.
●
Monitor the results with Munin/Cacti/Newrelic.
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
15. Power : Review
Why VPS rock Why they don't
@ New servers anytime
@ Total control
@ Scale up and down anytime @ Location
@ Short/no contracts @ Slices
@ Poor disk IO
@ Total control @ More servers, more problems
@ Cheap ! @ Support
@ Overselling
@ Cloud A != Cloud B
What to do
@ Benchmark @ Try before you buy
@ Test, test and test some more
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
16. Storage : The problem
Storage = /sites/*/files
@ Multi web head environment
@ Web heads need more space than
their slice allows
@* LOTS of files in a single dir (10k+)
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
17. Storage : The solutions
https://www.getpantheon.com/news/inside-pantheon-
valhalla-filesystem
@ Server with
big disk(s) / SAN / EBS + NFS
●
Tried, tested, works
●
Not very exciting
●
Single point of failure
●
Depending on your cloud could need a 64GB
ram server !!
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
18. Storage : The solutions
@ API storage + Fuse (Cloudfuse*, s3fs)
●
Theoretically unlimited storage
●
CDN with little or no extra effort
●
A bit bleeding edge / untested
@ Gluster FS / Similar
●
Impressive solution
●
Makes trade-offs drupal doesn't need
●
Complicated
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
19. Delivery : Do CDNs matter?
Dev server in Paris
YES
http://www.dotcom-monitor.com/WebTools/technical-tools-browser-test.aspx
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
20. Delivery : Types of CDN
http://wimleers.com/article/key-properties-of-a-cdn
@ Push (CloudFiles, S3 etc)
●
No redundant traffic
●
Complex set up
@ Pull (lots)
●
Very easy to set up
●
Redundant traffic
@ DNS (Cloudflare/incapsula)
●
Takes over your DNS
●
Does more than “just” CDN
●
We have not tested
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
21. Conclusions
@ The Cloud isn't a magic bullet
@ The best benchmark is your own app,
with replayed traffic.
@ Cloud providers differ a LOT
@ Storage can be complicated
@ CDNs can give you an easy big win
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
22. What did we do?
@ Power : Cloud, Yes … but still in testing
●
Using Dediserve.com
https://billing.dediserve.com/?affid=148
●
Still re-tuning web heads PHP-FPM vs mod_php.
●
Hopefully live next week!
@ Storage : NFS
●
Simple, tried and tested.
●
Using a NFS mount used by all web heads.
@ CDN : Yes! CDN pull.
●
Easy win ! Very easy to set up.
●
Dediserve's CDN.
●
Also using CDN module to spread assets over several
domains (improves front end performance)
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
23. * Lots of files / * Cloud fuse
@ Lots of files ...
Close to 1M files in a single directory
●
Wrote custom drupal 6 module to fix
●
As part of the file save hooks, it moved the file to a folder
structure based on the filename.
●
/sites/default/files/<new_folder>/images/123.jpg
●
On drupal.org, but alpha
@ Cloudfuse + CDN + Lots of files
●
Working proof of concept
●
Uses Rackspace cloud files for both storage and CDN
●
Requires pre-generation of imagecache
●
Not published … yet.
Drupal Developer Days – Barcelona June 16th 2012
ALEX McFADYEN | alex@demotix.com | @a_c_m
24. Thank you
Questions?
Alex McFadyen, CTO
drupal.org : a_c_m || drupal.org/user/195063
email : alex@demotix.com
twitter : @a_c_m