Talk given at CMG meetup in Pune on 22nd September, 2016. https://www.meetup.com/CMG-India-Pune-Meetup/events/234263002/

1. Rate limiters in modern
software systems
Sandeep Joshi
1
CMG Pune 22nd September 2016

2. Permit Raj
Control systems background. Traffic shaping in networking.
Two strategies:
1. Leaky bucket : A queue with constant service time. Drops packets on overflow.
2. Token bucket : Bucket holds tokens permits are acquired before transmission. Allows
burstiness in traffic.
PID controller
Backpressure
2

3. Congestion control
http://ecomputernotes.com/
3

4. Rate Limiter features
1. Enforce long term steady state rate
2. Allow short bursts to exceed limit
3. Warm-up : Gradually increase the rate after idle period
4. Allow changing the rate at run-time
5. Handle requests based on priority (fairness).
4

5. Rate-limiter implementations in software
1. RocksDB
2. Facebook WDT
3. Apache Kafka
4. Apache Spark (Google Guava toolkit)
5. Akka toolkit(not covering)
6. Node.js(not covering)
7. Conclusion
5

6. RocksDB
Key value store which uses Log-Structured Merge (LSM) trees.
It has a rate limiter to throttle disk writes done during two different workflows.
1. Flush threads which write in-memory tree to disk.
2. Compaction thread which merges trees on disk.
Throttler requires 3 parameters
1. Refill period
2. Refill bytes per period
3. Fairness which decides which of two queues to serve first
6

7. RocksDB
State kept by Throttler
1. Available bytes [i.e. tokens]
2. Next refill time
3. Queue [low] and Queue[high] into which new requests inserted
Workflow
1. Inserts requests into a queue
2. Leader of the queue awakes at “next refill time” and increments available bytes.
3. Sets the next refill time = now + refill period
4. Services all requests inside the queue until available bytes is exhausted.
7

8. RocksDB hi priority only
8
Peaks do not
rise above
average

9. RocksDB with 2 priorities
9
Low priority get
delayed

10. RocksDB
1. Enforce long term steady state rate
2. Allow short bursts to exceed rate
3. Warm-up : Gradually increase the rate after idle period
4. Allow changing the rate at run-time
5. Service requests based on priority (fairness).
10

11. Facebook WDT (Warp-speed Data transfer)
http://www.github.com/facebook/wdt
Open-source library which is used for file transfer between data centers. (e.g.
transmit MySQL backups to another data center)
Both sender and receiver spawn multiple threads
All threads on sender or receiver share the same “Throttler”
Throttler limits average rate as well as peak bursts.
11

12. Facebook WDT
12
Allows peak bursts above
average limit specified

13. Facebook WDT
1. Enforce long term steady state rate
2. Allow short bursts to exceed rate
3. Warm-up : Gradually increase the rate after idle period
4. Allow changing the rate at run-time
5. Service requests based on priority (fairness).
13

14. Apache Kafka
http://kafka.apache.org
14

15. Apache Kafka
Client-based quotas to limit publisher and consumer processing.
Enforces Fixed-rate in every window
Every request is inserted into into a DelayQueue from which elements can be
retrieved only after expiry (DelayQueue is part of java concurrent library)
Delay_time = (window_size) * (observed_rate - desired_rate) / observed_rate
Allows changing quota at run-time.
15

16. Apache Spark
Dynamic rate limiter
Two components : Driver (master) and Receiver (accepts ingest)
1. Driver (master) uses PID-based Rate estimator to recompute the desired
rate at end of every batch
2. Driver sends new rate to Receiver
3. Receiver uses Google Guava Rate limiter to throttle block generation
16

17. Apache Spark
https://issues.apache.org/jira/browse/SPARK-8975
17

18. Apache Spark
PID Controller used in the Spark Driver to estimate best rate
1. Proportional (current): correction based on current error
2. Integral gain (past): correction based on steady-state error.
3. Derivative gain (future) : prediction based on rate of change of error
18

19. Apache Spark before and after
19Source : Dean Wampler, Adding Backpressure to Spark Streaming

20. Google Guava rate limiter
https://github.com/google/guava
Used by Receiver in Apache Spark to limit the rate
Stores expected time of next request, instead of time of previous request.
Under-utilization : It stores unused permits up to a max threshold.
Warm-up period : Stored permits are given out gradually by increasing the
sleep time.
20

21. Google Guava rate limiter
21
Allows peak bursts while
maintaining average limit.
Allows warmup period

22. Rate-limiting techniques
All transmitters call some “Throttle()” function before transmission.
Some implementations push requests into a queue, others just calculate sleep time and
decrement permits.
Retain time of next estimated wakeup instead of time of previous call.
Permits are added if new epoch is detected.
Save unused permits upto some maximum limit - this handles underutilization.
While using unused permits, increase the sleep time. This increases the warmup period.
22

23. Comparison of implementations
rocksdb wdt guava/spark kafka
Type Leaky bucket Token Token bucket Leaky
Enforce Average rate Y Y Y Y
Allow short bursts
exceeding average
Y Y
Warm-up after idle
period
Y
Alter rate at runtime Y Y Y Y
Priority Y 23

24. References
1. RocksDB : util/rate_limiter.cc
2. WDT : Throttler.cpp
3. Apache Kafka : ClientQuotaManager.scala
4. Apache Spark : PIDRateEstimator.scala, RateLimiter.scala
5. Adding Back-pressure to Spark Streaming by Dean Wampler, Typesafe
(http://files.meetup.com/1634302/Backpressure%2020160112.pdf)
6. Google Guava : RateLimiter.java and SmoothRateLimiter.java
24

25. Facebook WDT
If (long term rate > average)
Sleep for (ideal - elapsed) time
Else If (short term rate > peak)
Add tokens based on time difference since last call
Sleep until tokens available are positive
25

26. Node.js
Rate limiter packages available
https://github.com/jhurliman/node-rate-limiter
express-rate
26

27. Akka toolkit
Look at TimerBasedThrottler.scala
27