Our experience with server monitoring while scaling our mobile app startup.

1. By Ajibola Aiyedogbon
Server Monitoring
(scaling while bootstrapped)

2. About me
Co-founder Amebo App
Mobile Developer (Jobberman, GTBank WP, etc)
DevOps enthusiast

3. Before before...
1 server for everything
-1 users
J2ME only
What throughput!
Cloudinary as CDN
Deployment fails
High costs, ignorance is very expensive

4. Now
5+ servers
Hundreds of thousands of users
Multi Platform apps
18000 req/min throughput
Cloudflare as CDN
Deployments with zero downtime
Managed costs

5. Scaling rhymes with Failing!

6. Server Stack

7. Server Stack
1 load balancer (layer 4, high availability, failover*)
3 web servers (vertically & horizontally scalable)
1 database server (replication*, redundancy*)
1 staging server
$65 monthly serving over 100 million requests
Cloudflare secret weapon, caches static requests (70%).

8. Technology Stack

9. Technology Stack
Haproxy (load balancer)
Nginx, Php-fpm (web server, php interpreter)
Phalcon, Php-Resque (framework, scheduler)
Redis, MongoDB, MariaDB (in-memory cache, datastores)
Git (BitBucket), Packer, Ansible (server provisioning, code provisioning)
SetCronJob, CloudFlare, Fastly (3rd party)

10. Why Iaas not Paas?
All about the pricing page!
Bandwidth costs too high
Code optimizations are hidden
behind computing power
Mission critical? Offload to PaaS
selectively, e.g. Parse EOL, death by
acquisition...

11. Why Monitor?

12. Don’t end up like these guys...

13. Why monitor?
Get Visibility
Improve usability & stability
Complicated technology stacks with
hard to trace errors
Mission critical
More sleep!

14. What to monitor apart from everything?

15. Server Metrics (infrastructure)
Ram usage, spikes
Bandwidth usage, highs vs lows
CPU usage over time, peak usage
Disk I/O
Open source vs Saas
Free mostly

18. Server Metrics (services)
Haproxy stats
Nginx Stats
Mysql performance etc
Service *something* status

19. Application Errors
Catch all exception php
User defined errors
3rd party Library errors

20. Tech Stack (Application Performance Monitoring)
Request throughput
Resource usage
Service Health
Database monitoring
Infrastructure bottlenecks
Failure Alerts
Code Errors
High level overview with deep dive

21. Log Tracking
Better way to tail -f
Http stack errors & anomalies
Multiple log files from diff services
Manual tailing is difficult
Get pre configured graphs based on logs
All server traffic is logged, access_log

24. Client Errors (Mobile)
Client side stack traces post deployment
Valuable version & device insight
Very handy at debug time & post
Catch all errors …. mostly
Memory leaks & stack traces
3rd party library errors or platform errors

26. Open Source vs Proprietary
Vendor lockin
Community support
DIY vs training
Industry standards & experience
Fault tolerance
Enterprise customer experience

27. 3rd Party vs Native monitoring tools
Core business?
Pricing again!
Support lifecycle and responsiveness
Product version, beta or 5.0?
Dashboard simplicity
Security implications? firewalled?
https? localhost only? Install certs?

28. Too many alerts….!

29. What now?
Congratulations, you reward is more work!
Customize alerts
Fix errors
Webhooks
Send to slack
Ignore at own risk

30. Be like this guy….or not!

31. Graphs on graphs on graphs on graphs
Information overload is real
Customize dashboard
Overviews only
Deep dive early to be familiar with dashboard

32. What Next?
Setup BugSnag

36. Conclusion
Why Monitor
What to Monitor
How to monitor
Pricing
Dashboards
Discuss your stack with peers

37. Thank You
@Ajibz