Caching can bring speed to the slow systems in your enterprise. In this session, we'll explore how to avoid expensive computations and reduce database (or other systems) load with caching. Considerations include when to cache (cache policy), how to cache, and when/how to evict. This session will also explore and discuss problems and gotchas like cache sizing, eviction mistakes, and where your cache should live. Finally, we'll look at how caching is implemented in the "memory-first" Couchbase architecture.
Presented at DevIntersection 2022 in Las Vegas
Cache Rules Everything Around Me - DevIntersection - December 2022
1. C.R.E.A.M. get the memory (dollar dollar bill ya'll)
August 2022
Matthew D. Groves | Product Marketing Manager
aka âPassionate Bravoâ
Cache Rules Everything
Around Me
2. 2
Latency Numbers Every Programmer Should Know
https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
3. 3
The Need for Speed
https://www.slideshare.net/devonauerswald/walmart-pagespeedslide
4. 4
Cache Rules Everything Around Me
Dollar dollar bill ya'll
Android apps iPhone apps
CPU cache
CDNs
Web browser
DNS
6. 6
ho am I?
âą Matthew D. Groves
âą Microsoft MVP, author at Pluralsight, Manning, Apress
âą Product Marketing Manager at Couchbase
âą https://twitter.com/mgroves
âą https://www.linkedin.com/in/mgroves/
âą C# Advent: https://csadvent.christmas/
8. 8
Caching
Collection of duplicated data to serve future requests faster.
âą First formally described by Maurice Wilkes in 1965
âą "Slave Memories and Dynamic Storage Allocation" https://bit.ly/WilkesPaper
What is Caching?
Key Data
1 Proteck Ya Neck
2 C.R.E.A.M.
3 Brooklyn Zoo
4 Criminology
⊠âŠ
Key Data
2 C.R.E.A.M.
4 Criminology
Records Cache
9. 9
Powerpoint as a Cache
What is Caching?
Key Value
What year was Wu-Tang
Clan Formed?
1992
12. 12
Caching Use Case 1: Faster Database Access
âą Databases can rely heavily on disk
âą Reading the same data over again
could mean accessing the disk over
again
âą Store frequently accessed data in a
cache, to reduce pressure on
database/disk
13. 14
Caching Use Case 1: Faster Database Access
DATABASE
APP
LAN
CACHE
CACHE
CACHE
14. 15
Caching Use Case 2: Faster Web Browsing
âą Any given URL could result in
hundreds of files that are loaded
over HTTP
âą A single JS file may be loaded by
every page on a site.
âą Cache static assets (JS, images,
sound, icons, etc) on a user's laptop
to avoid loading them over the
network every time
16. 17
Caching Use Case 3: Efficient API Use
âą API usage may involve an internet
call (with latency)
âą API calls may cost per use, or may
be limited
âą Storing results in cache may help
reduce latency AND reduce costs
22. 24
Using a Cache 101
Your Code
Key Value
A Protect Ya Neck
B C.R.E.A.M.
C Brooklyn Zoo
Cache
Key Value
A Protect Ya Neck
B C.R.E.A.M.
C Brooklyn Zoo
D Criminology
E Triumph
F Uzi
G Method Man
H Wu-Tang Clan Ain't . . .
. . . . . .
Database
1. Lookup "A"
2. Lookup "B"
3. Lookup "C"
HIT!
HIT!
MISS!
4. Lookup "C" again?
WALK OF SHAME
24. 26
What about writes?
âą Data is duplicated, stored in cache and record
âą The system is responsible for keeping the cache updated
âą Commons methods:
âą write-through
âą write-around
âą write-back
Stuff needs to get in the cache somehow
25. 27
write-through
1. System gets a new/updated piece of data
2. Write to cache and record "at the same time"
1. Write to record
2. If that didn't succeed, the write failed.
3. Write to cache
4. If that didn't succeed, the write failed.
5. If the system reaches this point, the write succeeded!
Why or why not use this?
26. 28
write-around
1. System gets a new/updated piece of data
2. Write the data to the record ONLY
3. Invalidate the record in the cache
Why or why not use this?
27. 29
write-back
1. System gets a new/updated piece of data
2. Write the data to the cache.
3. Update the record asynchronously (don't wait on it to finish)
4. If the cache write succeeded, then consider the write succeeded.
Why or why not use this?
28. 30
Cache Writing Methods
Method Write performance Write integrity Best for�
write-through
â â High % reads
write-around
â â High % read later
write-back
â â Mixed reads / writes
29. 31
Couchbase
A database with a built-in managed cache
https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/memory-and-storage.html
30. 32
Couchbase and Writing to the Cache
await collection.InsertAsync("A", new { title = "Protect Ya Neck" });
await collection.InsertAsync("A", new { title = "Protect Ya Neck" }, options =>
{
options.Durability(DurabilityLevel.None);
// OR: options.Durability(DurabilityLevel.Majority);
// OR: options.Durability(DurabilityLevel.MajorityAndPersistToActive);
// OR: options.Durability(DurabilityLevel.PersistToMajority);
});
35. 37
Latency Numbers Every Programmer Should Know
https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
37. 39
Powerpoint as a Cache
Key Value
What year was Wu-Tang Clan
Formed?
1992
What was Wu-Tang's first album
called?
Enter the Wu-Tang
What year was Maurice Wilkes
born?
1913
What year did Maurice Wilkes
die?
2010
What is the most recent Wu-
Tang studio album?
Once Upon a Time in Shaolin
38. 40
Cache Invalidation is Hard
There are only two hard things in Computer Science: cache
invalidation and naming things.
-- Phil Karlton
How to decide what to invalidate (what data to "evict"):
cache policy https://www.nndb.com/people/400/000031307/
39. 41
Cache Policy 1: LRU (Least Recently Used)
Key How new?
A 0
B 1 3
C 2
D 4
Cache max size: 3
Order of reads: A, B, C, B, D
X
---------------------------------------------------------------------------------
40. 42
Cache Policy 2: NRU (Not Recently Used)
Lower scores evicted (at random)
Key Referenced? Modified? Score
A 0 0 0
B 0 0 0
C 0 1 1
D 1 0 2
E 1 1 3
42. 44
More Cache Policies
âą RR â random replacement
âą FIFO and LIFO â treat a cache like a queue
âą MRU â MOST recently used
âą LFU â least frequently used
âą Belady's algorithm â not possible to implement
âą Machine learning
âą https://en.wikipedia.org/wiki/Cache_replacement_policies
44. 46
Gotcha 1: Cache Size
âą Problem: too small or too big cache
âą Symptom: High turnover and churn (many evictions) or long lifetimes (few
evictions)
âą Solutions:
âą Monitoring
âą Size the cache
âą "Value" vs "Full" eviction
45. 47
Gotcha 2: "Close" Caching
âą Problem: running the cache on the same machine as the application
âą Symptom: redundancy, hot spots, inconsistency, availability
âą Solutions:
âą Use a distributed cache
âą Running on its own machine(s) / cluster
âą Couchbase, of course!
46. 48
Gotcha 3: Bad Eviction Policy
âą Problem: using the wrong eviction policy
âą Symptom: lots of overhead, cache checking
âą Solutions:
âą Don't Write an Eviction Policy (or better yet don't write a Cache)
âą Benchmarking: "hit ratio" and "latency"
âą "Hit ratio" â how often a hit vs miss (e.g. "45% hit ratio")
âą "Latency" â how long it takes from request to response (e.g. 10ÎŒs latency)