ABSTRACT: Fatture in Cloud was born in late 2013 on a single-server machine and scaled from zero to 35k customers at the end of 2018. Then, we faced the mandatory electronic invoicing which came into effect in Italy on 1st January 2019, and we experienced a huge growth to 350k customers in few months. In these 5 years, I've learned a lot about cloud architecture, scalability, optimization, DevOps, and we eventually achieved a 99,99% uptime even in the huge growth period. BIO: Daniele Ratti is the Founder and CEO of Fatture in Cloud, which is currently the leader invoicing platform in Italy, counting more than 350k customers.

2. Daniele Ra)
CEO & Founder

4. Before e-invoicing hype

5. E-invoicing hype

6. Some tech numbers
• 350K paying customers
• 80-100 running instances simultaneously
• 10 MySQL databases
• 1 redis
• Handling 110M requests/day
• 88M requests/day on CDN
• 22M requests/day on EC2s
• Rendering 1.5M+ invoices/day

7. It all started with…
Landing page
Single dedicated server
Platform
Storage
Database
~100s of users
Server
Code
SFTP

8. Reliability was like…

9. Reliability was like…

10. Moving the storage outside!
Landing page
Single dedicated server
Platform
Storage on S3
Database

11. Moving the database outside!
Landing page
Single dedicated server
Platform
Storage on S3
Database on
separated instance

12. Moving everything in the cloud and decoupling
instances 🚀
Landing page
Platform
Storage on S3
Database on RDS
MySQL (then
Aurora)
Background Jobs

13. How could we be more fault tolerant?

14. Load balancing the requests
AutoScaling Group
Instance
Instance
Instance
Load Balancer

15. Autoscaling

16. CDN to cache staIc resources
AutoScaling Group
Instance
Instance
Instance
Load Balancer
CDN

17. What about versioning?
<CDN_url>/main.js?version=<commit id / checksum>
CDN miss (new version!)
<ALB_url>/main.js?version=<commit id / checksum>

18. ⚠ Not that easy when you handle million of
requests…

19. A step back to the (old) deployment system…
Server
Code
SFTP

20. The new deployment system…
GIT Codebase
Push / merge branch to master
Build on Jenkins
Webhook
Deploy via AWX
(Ansible Tower)
EC2 EC2 EC2
Package (on S3)
newborn
EC2

21. What could possibly go wrong with the CDN?

22. Look at the deploy Imes…
Build
Deploy on instance 1
Deploy on instance 2
Deploy on instance 3
Deploy on instance 4
Deploy on instance 5

23. <ALB_url>/page
LoadBalancer
EC2 (updated) EC2 (updated) EC2 (outdated)

24. <ALB_url>/page
LoadBalancer
EC2 (updated) EC2 (updated) EC2 (outdated)
<CDN_url>/main.js?version=new
LoadBalancer
EC2 (updated) EC2 (updated) EC2 (outdated)
CDN
miss (new version!)
(Thinking is the latest version!)

25. <ALB_url>/page
LoadBalancer
EC2 (updated) EC2 (updated) EC2 (outdated)
<CDN_url>/main.js?version=new
LoadBalancer
EC2 (updated) EC2 (updated) EC2 (outdated)
CDN
miss (new version!)
(Thinking is the latest version!)

26. Our soluIon (full blue/green was too slow)
Build
Deploy on instance 1
Deploy on instance 2
Deploy on instance 3
Deploy on instance 4
Deploy on instance 5
Padding time to update the package
Padding time to update the package
Padding time to update the package
Padding time to update the package
Padding time to update the package
The same moment for every instance
(The clocks are synchronized)

27. Users
Platform
Amazon ELB
Load balancer
RDS - Multi A-Z
MySQL
Storage
(expenses…)
Amazon S3
“storage” bucket
secure.fattureincloud.it
storage.fattureincloud.it
SSL
Landing page
www.fattureincloud.it
API
api.fattureincloud.it
Compute
compute.fattureincloud.it
Amazon ELB
Load balancer
cron runnings
Amazon SES
SMTP Server
Redis (caching)
Background
workers
* private IP *
EC2
SSL SSL SSLSSL
EC2 EC2
CDN
Amazon
CloudFront
SSL
cdn.fattureincloud.it
EC2
Autoscaling
Group
Autoscaling
Group
EC2 EC2
Amazon ELB
Load balancer
Autoscaling
Group
EC2 EC2
Amazon ELB
Load balancer
Autoscaling
Group
EC2 EC2

29. BoPleneck

31. Database issues & goals
• Reliability (single point of failure)
• Scaling
• Alter table on big tables
• Query analysis / opPmizaPon

32. What we found useful

33. Master (write) and slaves (read)
• Decouple read/write operaPons
• Uses 2+ different instances

34. Auto scaling & MulI-AZ
• Reduce the probability of failure
• Scaling of read replicas based on usage
• The master (write) instance become the new boWleneck as the write
instances can scale up to 15

35. Auto scaling

36. Specific-user per cluster and Performance Insights
• IdenPfy exactly where the slow queries come from
• IdenPfy which clusters are using more resources (both in read and write)

37. Specific-user per cluster and Performance Insights

38. Further performance analysis with NewRelic

39. Redis for query caching (and more…)
SELECT * FROM table WHERE …
Search valid md5(query) on redis
Response
found
Query the DB
not found
Save on Redis with a TTL (based
on the single query)

40. Altering big tables
Percona Toolkit is the best soluPon we found
Here’s what happen:
• Creates a new table with the altered structure
• Adds 3 triggers to the original table (INSERT, UPDATE, DELETE)
• Copies chunk by chunk the rows from the original table to the desPnaPon
• Atomically removes the triggers and switches the names of the tables
• [opPonal] delete the old table

41. Benchmarking ALTER TABLE vs Percona Toolkit
Technology: Aurora 1.17.8 - MySQL 5.6.10-log
RDS Aurora Instance: db.r4.8xlarge (244GB RAM - 32 vCPU)

42. Benchmarking ALTER TABLE vs Percona Toolkit

43. Benchmarking ALTER TABLE vs Percona Toolkit

44. Benchmarking ALTER TABLE vs Percona Toolkit
Conclusion:
Percona toolkit is 6,5X slower than ALTER TABLE with INPLACE algorithm and 4X
slower than ALTER TABLE with COPY algorithm but it’s predictable and safe.
It can be stopped without any consequences and it affect the DB CPU (on the
tesPng machine) by ~6%

45. CuriosiIes

46. We have a Ime machine to rollback in seconds 😎

47. We know the average lunch Ime thanks to the AutoScaling
* Minimum peak for lunch Pme @ 1:30PM

48. hWps://bit.ly/2WZzts5

49. Never stop learning
Daniele Ra)
CEO & Founder
FaWure in Cloud