Enviar pesquisa
Carregar
Redis to the Rescue?
•
40 gostaram
•
3,992 visualizações
Tim Lossen
Seguir
O'Reilly MySQL Conference 2011, Santa Clara
Leia menos
Leia mais
Tecnologia
Denunciar
Compartilhar
Denunciar
Compartilhar
1 de 76
Baixar agora
Baixar para ler offline
Recomendados
Redis to the Rescue?
Redis to the Rescue?
Tim Lossen
Facilitando a Programação concorrente com o Fork/Join
Facilitando a Programação concorrente com o Fork/Join
Mario Amaral
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
Daniel Cousineau
MongoDB Performance Tuning
MongoDB Performance Tuning
MongoDB
Mythbusting: Understanding How We Measure the Performance of MongoDB
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
Json tutorial, a beguiner guide
Json tutorial, a beguiner guide
Rafael Montesinos Muñoz
Talk about ReactiveMongo at MSUG May
Talk about ReactiveMongo at MSUG May
Andrey Neverov
MySQL Without The SQL -- Oh My! PHP Detroit July 2018
MySQL Without The SQL -- Oh My! PHP Detroit July 2018
Dave Stokes
Recomendados
Redis to the Rescue?
Redis to the Rescue?
Tim Lossen
Facilitando a Programação concorrente com o Fork/Join
Facilitando a Programação concorrente com o Fork/Join
Mario Amaral
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
Daniel Cousineau
MongoDB Performance Tuning
MongoDB Performance Tuning
MongoDB
Mythbusting: Understanding How We Measure the Performance of MongoDB
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
Json tutorial, a beguiner guide
Json tutorial, a beguiner guide
Rafael Montesinos Muñoz
Talk about ReactiveMongo at MSUG May
Talk about ReactiveMongo at MSUG May
Andrey Neverov
MySQL Without The SQL -- Oh My! PHP Detroit July 2018
MySQL Without The SQL -- Oh My! PHP Detroit July 2018
Dave Stokes
MongoDB at GUL
MongoDB at GUL
Israel Gutiérrez
Building Apps with MongoDB
Building Apps with MongoDB
Nate Abele
Building a Scalable Inbox System with MongoDB and Java
Building a Scalable Inbox System with MongoDB and Java
antoinegirbal
Pinterest的数据库分片架构
Pinterest的数据库分片架构
Tommy Chiu
Operational Intelligence with MongoDB Webinar
Operational Intelligence with MongoDB Webinar
MongoDB
MongoDB @ Frankfurt NoSql User Group
MongoDB @ Frankfurt NoSql User Group
Chris Harris
Become a Java GC Hero - All Day Devops
Become a Java GC Hero - All Day Devops
Tier1app
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB
Nested and Parent/Child Docs in ElasticSearch
Nested and Parent/Child Docs in ElasticSearch
BeyondTrees
Introduction to Scala for Java Programmers
Introduction to Scala for Java Programmers
sbjug
Scala for the doubters
Scala for the doubters
Max Klyga
Introduction to MongoDB
Introduction to MongoDB
antoinegirbal
Apocalypse Later
Apocalypse Later
Tim Lossen
Embracing Failure
Embracing Failure
Tim Lossen
Embracing Failure
Embracing Failure
Tim Lossen
Enemy of the State?
Enemy of the State?
Tim Lossen
Beyond Devops
Beyond Devops
Tim Lossen
All Your Core Are Belong To Us
All Your Core Are Belong To Us
Tim Lossen
Podularity FTW!
Podularity FTW!
Tim Lossen
Cubic Foot Gardening
Cubic Foot Gardening
Tim Lossen
Dashboard Mania
Dashboard Mania
Tim Lossen
Event-Stream Processing with Kafka
Event-Stream Processing with Kafka
Tim Lossen
Mais conteúdo relacionado
Mais procurados
MongoDB at GUL
MongoDB at GUL
Israel Gutiérrez
Building Apps with MongoDB
Building Apps with MongoDB
Nate Abele
Building a Scalable Inbox System with MongoDB and Java
Building a Scalable Inbox System with MongoDB and Java
antoinegirbal
Pinterest的数据库分片架构
Pinterest的数据库分片架构
Tommy Chiu
Operational Intelligence with MongoDB Webinar
Operational Intelligence with MongoDB Webinar
MongoDB
MongoDB @ Frankfurt NoSql User Group
MongoDB @ Frankfurt NoSql User Group
Chris Harris
Become a Java GC Hero - All Day Devops
Become a Java GC Hero - All Day Devops
Tier1app
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB
Nested and Parent/Child Docs in ElasticSearch
Nested and Parent/Child Docs in ElasticSearch
BeyondTrees
Introduction to Scala for Java Programmers
Introduction to Scala for Java Programmers
sbjug
Scala for the doubters
Scala for the doubters
Max Klyga
Introduction to MongoDB
Introduction to MongoDB
antoinegirbal
Mais procurados
(12)
MongoDB at GUL
MongoDB at GUL
Building Apps with MongoDB
Building Apps with MongoDB
Building a Scalable Inbox System with MongoDB and Java
Building a Scalable Inbox System with MongoDB and Java
Pinterest的数据库分片架构
Pinterest的数据库分片架构
Operational Intelligence with MongoDB Webinar
Operational Intelligence with MongoDB Webinar
MongoDB @ Frankfurt NoSql User Group
MongoDB @ Frankfurt NoSql User Group
Become a Java GC Hero - All Day Devops
Become a Java GC Hero - All Day Devops
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
Nested and Parent/Child Docs in ElasticSearch
Nested and Parent/Child Docs in ElasticSearch
Introduction to Scala for Java Programmers
Introduction to Scala for Java Programmers
Scala for the doubters
Scala for the doubters
Introduction to MongoDB
Introduction to MongoDB
Mais de Tim Lossen
Apocalypse Later
Apocalypse Later
Tim Lossen
Embracing Failure
Embracing Failure
Tim Lossen
Embracing Failure
Embracing Failure
Tim Lossen
Enemy of the State?
Enemy of the State?
Tim Lossen
Beyond Devops
Beyond Devops
Tim Lossen
All Your Core Are Belong To Us
All Your Core Are Belong To Us
Tim Lossen
Podularity FTW!
Podularity FTW!
Tim Lossen
Cubic Foot Gardening
Cubic Foot Gardening
Tim Lossen
Dashboard Mania
Dashboard Mania
Tim Lossen
Event-Stream Processing with Kafka
Event-Stream Processing with Kafka
Tim Lossen
Into the Void
Into the Void
Tim Lossen
Treasure Island -- Concurrency in JRuby
Treasure Island -- Concurrency in JRuby
Tim Lossen
Dashboard Mania
Dashboard Mania
Tim Lossen
JRuby vs. Rubinius
JRuby vs. Rubinius
Tim Lossen
The Smallest Cluster in the World
The Smallest Cluster in the World
Tim Lossen
Announcing Euruko 2011
Announcing Euruko 2011
Tim Lossen
Memory: The New Disk
Memory: The New Disk
Tim Lossen
Key-Value-Stores -- The Key to Scaling?
Key-Value-Stores -- The Key to Scaling?
Tim Lossen
Cassandra vs. Redis
Cassandra vs. Redis
Tim Lossen
An Introduction to Membase
An Introduction to Membase
Tim Lossen
Mais de Tim Lossen
(20)
Apocalypse Later
Apocalypse Later
Embracing Failure
Embracing Failure
Embracing Failure
Embracing Failure
Enemy of the State?
Enemy of the State?
Beyond Devops
Beyond Devops
All Your Core Are Belong To Us
All Your Core Are Belong To Us
Podularity FTW!
Podularity FTW!
Cubic Foot Gardening
Cubic Foot Gardening
Dashboard Mania
Dashboard Mania
Event-Stream Processing with Kafka
Event-Stream Processing with Kafka
Into the Void
Into the Void
Treasure Island -- Concurrency in JRuby
Treasure Island -- Concurrency in JRuby
Dashboard Mania
Dashboard Mania
JRuby vs. Rubinius
JRuby vs. Rubinius
The Smallest Cluster in the World
The Smallest Cluster in the World
Announcing Euruko 2011
Announcing Euruko 2011
Memory: The New Disk
Memory: The New Disk
Key-Value-Stores -- The Key to Scaling?
Key-Value-Stores -- The Key to Scaling?
Cassandra vs. Redis
Cassandra vs. Redis
An Introduction to Membase
An Introduction to Membase
Último
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Igalia
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
hans926745
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
debabhi2
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
Pixlogix Infotech
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Enterprise Knowledge
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
Maria Levchenko
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
Malak Abu Hammad
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Martijn de Jong
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Safe Software
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
HampshireHUG
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
Michael W. Hawkins
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
Results
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Miguel Araújo
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
wesley chun
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
Delhi Call girls
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
RTylerCroy
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Neo4j
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
The Digital Insurer
Último
(20)
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Redis to the Rescue?
1.
Redis to the
Rescue? O’Reilly MySQL Conference 2011-04-13
2.
Who? • Tim Lossen
/ @tlossen • Berlin, Germany • backend developer at wooga
3.
4.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
5.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
6.
What? • key-value-store • in-memory
database • “data structure server”
7.
8.
Data Types • strings
(integers)
9.
Data Types • strings
(integers) • lists • hashes
10.
Data Types • strings
(integers) • lists • hashes • sets • sorted sets
11.
flickr.com/photos/atzu/2645776918
12.
Performance • same for
reads / writes
13.
Performance • same for
reads / writes • 50 K ops/second - regular notebook, EC2 instance
14.
Performance • same for
reads / writes • 50 K ops/second - regular notebook, EC2 instance • 200 K ops/second - intel core i7 X980 (3.33 GHz)
15.
Durability • snapshots • append-only
log
16.
Other Features • master-slave
replication • virtual memory • ...
17.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
18.
19.
Daily Active Users April
May June July Aug Sept Oct 2010
20.
Daily Active Users April
May June July Aug Sept Oct 2010
21.
Challenge • traffic growing
rapidly
22.
Challenge • traffic growing
rapidly • bottleneck: write throughput - EBS volumes pretty slow
23.
Challenge • traffic growing
rapidly • bottleneck: write throughput - EBS volumes pretty slow • MySQL already sharded - 4 x 2 = 8 shards
24.
Idea • move write-itensive
data to Redis
25.
Idea • move write-itensive
data to Redis • first candidate: inventory - integer fields - frequently changing
26.
27.
Solution • inventory =
Redis hash - atomic increment / decrement !
28.
Solution • inventory =
Redis hash - atomic increment / decrement ! • on-demand migration of users - with batch roll-up
29.
Results • quick win
- implemented in 2 weeks - 10% less load on MySQL servers
30.
Results • quick win
- implemented in 2 weeks - 10% less load on MySQL servers • decision: move over more data
31.
But ... • “honeymoon
soon over”
32.
But ... • “honeymoon
soon over” • growing memory usage (fragmentation) - servers need periodic “refresh” - replication dance
33.
Current Status • hybrid
setup - 4 MySQL master-slave pairs - 4 Redis master-slave pairs
34.
Current Status • hybrid
setup - 4 MySQL master-slave pairs - 4 Redis master-slave pairs • evaluating other alternatives - Riak, Membase
35.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
36.
37.
Challenge • expected peak
load: - 16000 concurrent users - 4000 requests/second - mostly writes
38.
“Memory is the
new Disk, Disk is the new Tape.” ⎯ Jim Gray
39.
Idea • use Redis
as main database - excellent (write) performance - virtual memory for capacity
40.
Idea • use Redis
as main database - excellent (write) performance - virtual memory for capacity • no sharding = simple operations
41.
Data Model • user
= single Redis hash - each entity stored in hash field (serialized to JSON)
42.
Data Model • user
= single Redis hash - each entity stored in hash field (serialized to JSON) • custom Ruby mapping layer (“Remodel”)
43.
44.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
45.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
46.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
47.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
48.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
49.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
50.
Virtual Memory • server:
24 GB RAM, 500 GB disk - can only keep “hot data” in RAM
51.
Virtual Memory • server:
24 GB RAM, 500 GB disk - can only keep “hot data” in RAM • 380 GB swap file - 50 mio. pages, 8 KB each
52.
Dec 2010: Crisis •
memory usage growing fast
53.
Dec 2010: Crisis •
memory usage growing fast • cannot take snapshots any more - cannot start new slaves
54.
Dec 2010: Crisis •
memory usage growing fast • cannot take snapshots any more - cannot start new slaves • random crashes
55.
Analysis • Redis virtual
memory not compatible with: - persistence - replication
56.
Analysis • Redis virtual
memory not compatible with: - persistence - replication • need to implement our own!
57.
Workaround • “dumper” process
- tracks active users - every 10 minutes, writes them into YAML file
58.
ruby
redis disk
59.
ruby
redis disk
60.
ruby
redis disk
61.
ruby
redis disk
62.
ruby
redis disk
63.
Workaround • in case
of Redis crash - start with empty database - restore users on demand from YAML files
64.
Real Solution • Redis
“diskstore” - keeps all data on disk - swaps data into memory as needed
65.
Real Solution • Redis
“diskstore” - keeps all data on disk - swaps data into memory as needed • under development (expected Q2)
66.
67.
Results • average response
time: 10 ms
68.
Results • average response
time: 10 ms • peak traffic: - 1500 requests/second - 15000 Redis ops/second
69.
Current Status • very
happy with setup - simple, robust, fast - easy to operate • still lots of spare capacity
70.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
71.
Advantages • order-of-magnitude performance
improvement - removes main bottleneck - enables simple architecture
72.
Disadvantages • main challenge:
durability - diskstore very promising
73.
Disadvantages • main challenge:
durability - diskstore very promising • no ad-hoc queries - think hard about data model - hybrid approach?
74.
Conclusion • Ruby +
Redis = killer combo
75.
Q&A
76.
redis.io github.com/tlossen/remodel wooga.com/jobs
Baixar agora