1) The document presents benchmark results that compare the performance of different cloud providers and instance types across metrics like CPU, concurrency, memory, and disk performance.
2) There is significant variation in performance both between different cloud providers and instance types, and also within the same instance type over time.
3) While the cloud provides benefits like easy resizing and deployment, raw performance is generally better with dedicated hosting and benchmarks show the cloud is not always lower cost. Careful performance testing is needed to select the best instance type for each application.
Designing IA for AI - Information Architecture Conference 2024
The Secret Guide to Cloud Performance - Cloudlook
1. The Secret Guide to
Cloud Performance
Jan 30 2014 @ DevCon TLV
Gideon Greenspan
Co-Founder, Cloudlook
2. Site/DB
Site/DB
Site/DB
OS etc. Site/DB
OS + web
server + DBMS
OS etc. Site/DB
Website +
Database
OS etc. Site/DB
Types of hosting
OS + web
server + DBMS
Hardware
Hypervisor
Host OS
Hardware
Hardware
Dedicated
Cloud / VPS
Shared
Secret Guide to Cloud Performance
Slide 2
Cloudlook
3. “The Cloud” vs Old-School VPS
• Deploy/decommission in minutes
• Scale up/down in minutes
• Management APIs
• Automatic scaling
• Billing by the hour
• Migration on hardware failure
Secret Guide to Cloud Performance
Slide 3
Cloudlook
4. Some public cloud providers
Secret Guide to Cloud Performance
Slide 4
Cloudlook
7. The trouble with the cloud
• Vague system specifications
• Variation between instances
– Noisy neighbors?
– Diverse underlying hardware
• Variation over time within an instance
• One-time benchmarks insufficient
Secret Guide to Cloud Performance
Slide 7
Cloudlook
8. Cloudlook: continuous benchmarking
• 6 cloud providers, 35 instance types
• 15,000 instances per month
• 50 samples per instance over 1 hour
• All 64-bit CentOS (even on Azure!)
• $2,000 per month
• Available free for your server too!
Secret Guide to Cloud Performance
Slide 8
Cloudlook
9. What does Cloudlook measure?
• Integer CPU performance
• Concurrency
• Memory bandwidth
• Sequential disk access
All done with…
• Random disk access
• Host disk cache effect
Secret Guide to Cloud Performance
Slide 9
Cloudlook
10. The Secret Guide
• Introduction
• Benchmarks
• Conclusions
Secret Guide to Cloud Performance
Slide 10
Cloudlook
11. Measuring integer CPU
• PHP scripts run as interpreted bytecode
• Built-in PHP functions are compiled C
• We benchmarked 14 heavy functions
across 66 versions of PHP ≥ 5.0.4
• Two performed consistently (±0.2%)
– sha1() — hash function
– levenshtein() — deep string comparison
Secret Guide to Cloud Performance
Slide 11
Cloudlook
15. Concurrency results
Measured concurrency
5
4
Windows Azure
Digital Ocean + HP Cloud
Rackspace Performance 1
GoGrid
Amazon EC2
Rackspace Standard
3
2
1
1
2
3
4
5
6
7
8
Specified vCPUs / cores
Secret Guide to Cloud Performance
Slide 15
Cloudlook
16. Measuring memory bandwidth
• Create a 64 MB string in memory
– Largest L1+L2+L3 cache currently ~41 GB
• substr() to read successive windows
– memcpy() in C does the actual work
• Window length = 128 kb
– Too short ⇒ time in bytecode interpreter
– Too long ⇒ copy too big for L1+L2 cache
Secret Guide to Cloud Performance
Slide 16
Cloudlook
18. Memory performance variation
HP Cloud Standard Medium
Rackspace Cloud Standard 4GB
Digital Ocean 4GB
Amazon m1.large
Secret Guide to Cloud Performance
Slide 18
Cloudlook
19. Measuring disk performance
• Write benchmark files in /tmp
• Avoid operating system cache
– Total size > instance memory
– Explicitly clear cache (our instances)
– Read each block once (our instances)
• “Random” is pseudo-random
– Fibonacci hashing scheme
Secret Guide to Cloud Performance
Slide 19
Cloudlook
20. OS etc. Site/DB
OS etc. Site/DB
OS etc. Site/DB
Host cache interference
Hypervisor
Host OS
Hardware
Secret Guide to Cloud Performance
• Results can be skewed
• Big instances: write lots of
other data inbetween
• Small instances: wait a
week and pray…
• But still worth measuring
because it’s a good thing!
Slide 20
Cloudlook
21. Disk results (no host cache)
10,000
EC2 m1.medium Primary
Disk Seeks — ops/s
EC2 m1.medium Secondary
Rackspace Standard >= 2GB
Rackspace Performance 1 >= 2GB Both
1,000
Azure >= 1.5 GB Primary
Azure >= 1.5 GB Secondary
HP Cloud >= 2GB Both
Digital Ocean >= 1GB
100
GoGrid >= 2 GB
10
10
100
1000
Disk Throughput — MB/s
Secret Guide to Cloud Performance
Slide 21
Cloudlook
22. Who has a host cache?
Primary disk rate
(no host cache)
Primary disk rate
(host cache)
Host cache
speedup
HP Cloud Small
109 MB/s
528 MB/s
4.8 ⤬
HP Cloud Medium
124 MB/s
320 MB/s
2.6 ⤬
HP Cloud Large
110 MB/s
117 MB/s
1.1 ⤬
Digital Ocean 1 GB
244 MB/s
709 MB/s
2.9 ⤬
Digital Ocean 2 GB
267 MB/s
703 MB/s
2.6 ⤬
Digital Ocean 4 GB
260 MB/s
709 MB/s
2.7 ⤬
Digital Ocean 8 GB
318 MB/s
474 MB/s
1.5 ⤬
Secret Guide to Cloud Performance
Slide 22
Cloudlook
23. The Secret Guide
• Introduction
• Benchmarks
• Conclusions
Secret Guide to Cloud Performance
Slide 23
Cloudlook
26. Bad reasons to host in the cloud
• Hardware abstraction (peace of mind)
– Last week Rackspace Cloud lost my data
• Easy resizing
– Dedicated is at top of scale anyway
– Can’t resize Rackspace Performance servers
• Lower costs
– No. That’s why hosting companies like it…
Secret Guide to Cloud Performance
Slide 26
Cloudlook
27. Some good reasons
• Small instances
• Fast deploy/decommission
• Tools and APIs
• Other cloud services
– Storage and CDN
– Databases and messaging
Secret Guide to Cloud Performance
Slide 27
Cloudlook