In this talk I will walk you through the performance tuning steps that I took to serve 1.2M JSON requests per second from a 4 vCPU c5 instance, using a simple API server written in C. At the start of the journey the server is capable of a very respectable 224k req/s with the default configuration. Along the way I made extensive use of tools like FlameGraph and bpftrace to measure, analyze, and optimize the entire stack, from the application framework, to the network driver, all the way down to the kernel. I began this wild adventure without any prior low-level performance optimization experience; but once I started going down the performance tuning rabbit-hole, there was no turning back. Fueled by my curiosity, willingness to learn, and relentless persistence, I was able to boost performance by over 400% and reduce p99 latency by almost 80%.

1. Brought to you by
Extreme HTTP Performance Tuning:
1.2M API req/s on a 4 vCPU EC2 Instance
Marc Richards
Chief Problem Solver at

2. Marc Richards
Chief Problem Solver at Talawah Solutions
Talawah Solutions
■ Based in Kingston Jamaica
■ Cloud Computing Consultant for almost a decade
■ Solutions Architect / DevOps Engineer / Performance Engineer
■ No low-level systems performance tuning experience before
this project!

3. Demystifying Systems Performance Tuning
■ You don't need to be a kernel developer or a wizard sysadmin.
■ FlameGraph and bpftrace have changed the game.
■ New ebpf based tools coming out will only make things easier!

4. Overview
■ I accidentally fell down this optimization rabbit hole.
■ Started with a simple, high-performance API server written in C.
■ Used FlameGraph and bpftrace to analyze and optimize the entire stack.

5. Overview
■ Cloud: AWS
■ Hardware: 4 vCPU c5n.xlarge** (server) / 16 vCPU c5n.4xlarge (client)
■ Benchmark: Techempower JSON Serialization test
■ Server: Techempower libreactor implementation
** In order to minimize inconsistencies at the platform level I did the final benchmark run on a c5n.9xlarge that was
restricted to 4 vCPUS using the EC2 CPU Options feature.

6. Blog post with even more details
https://talawah.io/blog/extreme-http-performance-tuning-one
-point-two-million/

7. Optimizations
Optimization Gain Req/s
Ground Zero - 224k
Application Optimizations 55% 347k
Disabling Speculative Execution Mitigations 28% 446k
Disabling Syscall Auditing / Blocking 11% 495k
Disabling iptables / netfilter 22% 603k
Perfect Locality 38% 834k
Interrupt Optimizations 28% 1.06M
The Case of the Nosy Neighbor 6% 1.12M
The Battle Against the Spin Lock 2% 1.15M
This Goes to Twelve 4% 1.20M

8. Optimizations
Optimization Gain Req/s
Ground Zero - 224k
Application Optimizations 55% 347k
Disabling Speculative Execution Mitigations 28% 446k
Disabling Syscall Auditing / Blocking 11% 495k
Disabling iptables / netfilter 22% 603k
Perfect Locality 38% 834k
Interrupt Optimizations 28% 1.06M
The Case of the Nosy Neighbor 6% 1.12M
The Battle Against the Spin Lock 2% 1.15M
This Goes to Twelve 4% 1.20M

9. Ground Zero
Running 10s test @ http://server.tfb:8080/json
16 threads and 256 connections
Latency Distribution
50.00% 1.14ms
90.00% 1.21ms
99.00% 1.26ms
99.99% 1.32ms
2243551 requests in 10.00s, 331.64MB read
Requests/sec: 224,353.73

10. * I modified nginx.conf to send back a hardcoded JSON response. This is not a part of the Techempower implementation.

11. Ground Zero

12. Application Optimizations

13. Application Optimizations

14. Application Optimizations
■ Run on all logical cores/vCPUs: ~25%
■ gcc -O3 and march=native: ~15%
■ send/recv instead of write/read: ~5%
■ Remove pthread overhead: ~3%

15. Application Optimizations
Running 10s test @ http://server.tfb:8080/json
16 threads and 256 connections
Latency Distribution
50.00% 723.00us
90.00% 0.88ms
99.00% 0.94ms
99.99% 1.08ms
3470892 requests in 10.00s, 483.27MB read
Requests/sec: 347,087.15

16. Application Optimizations
Before After

17. Disabling...
Speculative Execution Mitigations
Syscall Auditing / Blocking
iptables/netfilter

18. Disabling...
■ Speculative Execution Mitigations: 28%
● nospectre_v1 nospectre_v2 pti=off mds=off tsx_async_abort=off
■ Syscall Auditing/Blocking: 11%
● auditctl -a never,task
● docker run -d --security-opt seccomp=unconfined libreactor
■ iptables/netfilter: 22%
● modprobe -rv ip_tables
● ExecStart=/usr/bin/dockerd ---bridge=none --iptables=false --ip-forward=false

19. Disabling...
Running 10s test @ http://server.tfb:8080/json
16 threads and 256 connections
Latency Distribution
50.00% 419.00us
90.00% 479.00us
99.00% 517.00us
99.99% 575.00us
6031161 requests in 10.00s, 839.76MB read
Requests/sec: 603,112.18

20. Disabling...
Before After

21. Perfect Locality
+
Interrupt Optimizations

22. Perfect Locality + Interrupt Optimizations
■ Perfect Locality
● Pin processes to CPUs
● Pin network queues to CPUs (RSS + XPS)
● SO_REUSEPORT + SO_ATTACH_REUSEPORT_CBPF
■ Interrupt Moderation
● ethtool -C eth0 adaptive-rx on
■ Busy polling
● net.core.busy_poll=1
■ Perfect Locality + Interrupt Moderation + Busy Polling = 💯

23. Perfect Locality + Interrupt Optimizations
Running 10s test @ http://server.tfb:8080/json
16 threads and 256 connections
Latency Distribution
50.00% 233.00us
90.00% 263.00us
99.00% 292.00us
99.99% 348.00us
10660410 requests in 10.00s, 1.45GB read
Requests/sec: 1,066,034.60

24. Perfect Locality + Interrupt Optimizations
Before After

25. The Case of the Nosy Neighbor
+
The Battle Against the Spin Lock

26. The Case of the Nosy Neighbor
Someone, somewhere was spying on all my packets (kinda)
■ dev_queue_xmit_nit() -> packet_rcv()
■ packet_rcv() implicates AF_PACKET
■ sudo ss --packet --processes -> (("dhclient",pid=3191,fd=5))
■ My (extreme) solution was to disable dhclient after boot

27. The Case of the Nosy Neighbor
Running 10s test @ http://server.tfb:8080/json
16 threads and 256 connections
Latency Distribution
50.00% 218.00us
90.00% 254.00us
99.00% 285.00us
99.99% 341.00us
11279049 requests in 10.00s, 1.53GB read
Requests/sec: 1,127,894.86

28. The Case of the Nosy Neighbor
Before After

29. The Battle Against the Spin Lock
Running 10s test @ http://server.tfb:8080/json
16 threads and 256 connections
Latency Distribution
50.00% 212.00us
90.00% 246.00us
99.00% 276.00us
99.99% 338.00us
11551707 requests in 10.00s, 1.57GB read
Requests/sec: 1,155,162.15

30. The Battle Against the Spin Lock
Before After

31. This Goes to Twelve

32. This Goes to Twelve
■ Disabling Generic Receive Offload (GRO)
■ TCP Congestion Control: cubic -> reno
■ Static Interrupt Moderation

33. This Goes to Twelve
Running 10s test @ http://server.tfb:8080/json
16 threads and 256 connections
Latency Distribution
50.00% 203.00us
90.00% 236.00us
99.00% 265.00us
99.99% 317.00us
12031718 requests in 10.00s, 1.64GB read
Requests/sec: 1,203,164.22

34. Conclusion
436% increase in requests per second. 79% reduction in p99 latency.
■ Throughput: 224k req/s -> 1.2M req/s
■ p99 latency: 1.26ms -> 265.00us
■ p99.99 latency: 1.32ms -> 317.00us

35. All 11 implementations on a c5n.xlarge using the stock Amazon Linux 2 AMI without any OS/Networking optimizations

36. All 11 implementations on a c5n.xlarge with all OS/Networking optimizations applied

37. Next Steps
■ Next gen kernel: 5.10 LTS
■ Next gen technologies: io_uring
■ Next gen instances: ARM vs Intel vs AMD
■ Driving performance from the bottom-up using Rust, Java, etc

38. Brought to you by
Marc Richards
https://talawah.io/contact
@talawahtech