SlideShare uma empresa Scribd logo
1 de 37
Baixar para ler offline
深入了解Redis
               宋传胜
   Chuansheng.song@langtaojin.com
简单介绍
•   文本协议 memcached类似
•   KEY 可打印字符
•   VALUE支持的类型
    –   STRINGS
    –   LIST
    –   SET
    –   SORTED SET
    –   HASH
•   高性能 (100k+ SET/80k+ GET)/s
•   序列化支持
•   主从同步支持
•   客户端自己实现sharding
基本数据结构
• RedisObject (Redis.h)
  – key/value对象的基础
KEY
• 基本命令
 –   get/set/exists O(1)
 –   setnx/randomkey/rename/renamenx/dbsize/type
 –   keys pattern
 –   del 一次删除多个key
 –   expire key seconds 指定过期时间
 –   ttl key 剩余过期时间
 –   select db
 –   move key newdb
 –   flush db/flushall
EXPIRE AND LRU CACHE
• Volatile Key (Db.c)
   – When the key is set to a new value using the SET command, or when a
     key is destroyed via DEL, the timeout is removed from the key.
• Enhanced Lazy Expiration algorithm
   – Redis does not constantly monitor keys that are going to be expired.
     Keys are expired simply when some client tries to access a key, and the
     key is found to be timed out.
• Work as LRU cache (memcached)
   – Maxmemory/maxmemory-policy
   – When memory limit was already reached, server will remove some old
     data deleting a volatile key, even if the key is still far from expiring
     automatically.
   – Random get keys, delete by lru rules.
基本数据结构
• Hashtable(Dict.c)
基本Value数据结构
• STRING(sds.c)



  – set/setnx/get/getset/mget/mset/msetnx
    •nx – not exists
    •m -- multiple
  – incr/decr/incrby/decrby
  – getrange/append
基本Value数据结构
• LIST (T_list.c)
  – REDIS_LIST 类型, 如果其 entry 小于配置值: list-
    max-ziplist-entries 或 value字符串的长度小于 list-
    max-ziplist-value,使用ziplist数据结构,否则使用
    标准的Doubly linked list
  – (l/r)push/(l/r)pop/llen O(1)
  – b(l/r)pop支持阻塞等待,避免了轮循
  – lrange/ltrim/lrem/lset/rpoplpush
LIST
• Ziplist (Ziplist.c)
   – O(mem_size) add/delete
   – list-max-ziplist-entries (default: 1024)
   – list-max-ziplist-value (default: 32)
LIST
• Ziplist (continue)
  – zlentry (unencode structure)
LIST
• Doubly linked list (Adlist.c)
基本Value数据结构
• SET (T_set.c) – hashtable + Intset
  – String的无序集合
  – sadd/srem/sismember/scard O(1)
  – spop/srandmember
  – smove
  – smembers
  – sinter(store) O(C)
  – sunion(store)/sdiff(store) O(N)
INTSET
• INTSET (Intset.c)
  – 都是整型时数据结构退化成排序的intset
  – Good fit for size up to 20-50K
  – set-max-intset-entries (default: 4096)
  – O(n) search
  – O(log(n) + mem_size) add/del
基本Value数据结构
• SORTED SET(T_zset.c) hashtable + skip list
  – 按照key的score排序
  – zadd/zrem/zrank/zrevrank O(log(n))
  – zcard O(1)
  – zincrby 修改score
  – zrange/zrevrange/zrangebyscore/zscore
    O(log(N)+M)
  – zremrangebyrank/zremrangebyscore O(log(N)+M)
SKIP LIST
• Skip List (T_zset.c)
  – skip list  redblack tree  AVL tree (more
    balanced)
  – lockfree/concurrency
基本Value数据结构
• HASH (T_hash.c) zipmap + hashtable
  – 如果其 entry 小于配置值: hash-max-zipmap-
    entries 或 value字符串的长度小于 hash-max-
    zipmap-value ,使用zipmap数据结构
  – Value只能是string类型
  – hset/hget/hexists O(1)
  – hmset/hexists/hincrby/hdel/hlen
  – hmget O(N)
  – hkeys/hvals/hgetall
ZIPMAP
• zipmap
  – O(n) search O(mem_size) add/delete
内存管理
• zmalloc.c
  – 简单内存管理
  – 支持tcmalloc USE_TCMALLOC
  – zmalloc(size_t size)  malloc(size+PREFIX_SIZE)
  – fragmentation_ratio =
    zmalloc_get_rss/zmalloc_used_memory
PUBSUB
• SUBSCRIBE/UNSUBSCRIBE/PSUBSCRIBE/PU
  NSUBSCRIBE
• PUBLISH
• 连接断开后信息丢失
TRANSACTION
• MULTI (不支持CAS)
  –   MULTI
  –   SET foo 0
  –   INCR foo
  –   EXEC
• WATCH (Check and Set )
  –   WATCH theKey
  –   v = GET theKey
  –   MULTI
  –   SET theKey v+1
  –   EXEC
• UNWATCH/DISCARD
提升网络效率、减少内存占用
• MGET/MSET
• PIPELINING
  – $ (echo -en "PINGrnPINGrnPINGrn"; sleep 1) | nc localhost 6379
    +PONG
    +PONG
    +PONG
• 压缩数据,触发ziplist/zipmap/intset
• 使用 GETBIT/SETBIT/GETRANGE/SETRANGE 压缩多
  个属性
• 使用HASH
• 使用32位redis + sharding
Client Side Sharding
• Consistent Hashing
  – Redis Sharding at Craigslist
网络层
• Anet.c
• 事件驱动 ae.c 
  ae_epoll.c/ae_kqueue.c/ae_select.c
• 非阻塞单线程
• 支持timer
• 在VM场景下有可能多线程
网络层对比
• memcached/varnish/scribed的网络IO模型
 – 多线程,非阻塞IO复用
网络层对比
• lighttpd/nginx的网络IO模型
  – 多进程,单线程非阻塞IO
• apache的MPM模型
  – 多进程prefork,多线程
PERSISTENCE
• SNAPSHOT (Rdb.c)
• 可以关闭
• BGSAVE or save xseconds ychanges
• fork
   – parent - disable rehash, periodic check child on serverCron
     (100ms)
   – child - copy-on-write, save whole db, exit
• Need to turn on the overcommit_memory setting if you want to
  deal with a dataset more than 1/2 of RAM, which contradicts
  our habit to battle against OOM killer as a database
  administrator.
Append Only File
• Append to log file on every change (Aof.c)
• fsync() policy
   – always/os decide/every second
   – Always with battery backed raid controllers
   – Every second by default (innodb_flush_log_at_trx_commit=2 in
     mysql)
• Compact aof file
   – BGREWRITEAOF/REWRITEAOF
• Fork
   – child – write new aof in temporary file
   – parent – write change in both old aof file and memory buffer,
      append memory buffer to temporary file when child done, rename
      temporary to aof file.
• redis-check-aof --fix <filename>
Replication
• 支持多级同步 (Replication.c)
Replication
• 支持AUTH
• 起动过程
 – slave
   •SYNC
   •Wait
 – master
   •Issue a BGSAVE, or wait if BGSAVE in progress
Virtual Memory
• Redis VM is now deprecated. Redis 2.4 will be the
  latest Redis version featuring Virtual Memory!
• Why not OS VM
  – Varnish What’s wrong with 1975 programming ?
  – Redis What’s wrong with 2006 programming?
     • Blocking when page fault
     • 4k pages granularity
     • Optimal disk persistent format, 10:1
• Keys can’t be swapped out
• vm is read only when BGSAVE/BGREWRITEAOF in
  progress
• vm-max-memory/vm-pages/vm-page-size
Virtual Memory
• Blocking vm
• Threaded vm
  – Before command is executed, check the value
  – Block client and load, then signal by unix pipe
  – Continue
• Recommended filesystem – ext3 or any other file
  system with good support for sparse files
• redis-stat vmstat
• InnoDB innodb_buffer_pool_size – swap and
  durability
Future
• Diskstore (diskstore.c dscache.c)
  – All data on disk, B+tree in future
  – Swaps data into memory when needed
• Leveldb/InnoDB Change Buffer
  – LSM tree
其他相关的代码
• quicksort (pqsort.c)
• lzf compression (lzf_c.c lzf_d.c)
• Sha1 (Sha1.c)
• Syncio.c
• SORT (Sort.c)
线上Redis集群情况
• 16台服务器,兼做后端服务器
• 45个Redis Instance进程,三组服务
• 总内存 77G VIRT,71G RSS
• 使用了主从复制、snapshot持久化,VM支持
  ,客户端sharding、pipelining、hash优化
Redis应用
•   Resque – message queue of github (celery)
•   Nginx HttpRedis module
•   Redweb - Web administrative and query UI
•   A fast, fuzzy, full-text index using Redis
•   Redis-backed BloomFilter(s) in Ruby
•   Soulmate : Auto Complete with Redis
•   Rate limiting with Redis
•   Redis Sharding
•   Locking with setnx
•   A Collection Of Redis Use Cases
•   Radishapp Redis monitoring
•   openredis.com hosting service
Contribute to Redis
• Fork redis on github
  – https://github.com/antirez/redis
• Choose branch and commit your changes
• Create a pull request
• Wait for approval
  – https://github.com/antirez/redis/pulls
• My contribute for read only slave
  – https://github.com/antirez/redis/pull/47
Resources
• DOCUMENATION
 – http://redis.io/documentation
 – http://antirez.com/
 – http://groups.google.com/group/redis-db
• 本文参考的Redis资料集合
 – http://www.delicious.com/fakechris/redis
 – http://www.delicious.com/tag/redis

Mais conteúdo relacionado

Mais procurados

微博cache设计谈
微博cache设计谈微博cache设计谈
微博cache设计谈
Tim Y
 
Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "
Tanya Denisyuk
 

Mais procurados (20)

A simple introduction to redis
A simple introduction to redisA simple introduction to redis
A simple introduction to redis
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
redis basics
redis basicsredis basics
redis basics
 
Redis acc 2015_eng
Redis acc 2015_engRedis acc 2015_eng
Redis acc 2015_eng
 
Redis 101
Redis 101Redis 101
Redis 101
 
Redis ndc2013
Redis ndc2013Redis ndc2013
Redis ndc2013
 
Your 1st Ceph cluster
Your 1st Ceph clusterYour 1st Ceph cluster
Your 1st Ceph cluster
 
High Performance Weibo QCon Beijing 2011
High Performance Weibo QCon Beijing 2011High Performance Weibo QCon Beijing 2011
High Performance Weibo QCon Beijing 2011
 
Redis overview for Software Architecture Forum
Redis overview for Software Architecture ForumRedis overview for Software Architecture Forum
Redis overview for Software Architecture Forum
 
MyRocks Deep Dive
MyRocks Deep DiveMyRocks Deep Dive
MyRocks Deep Dive
 
Redis database
Redis databaseRedis database
Redis database
 
Ceph Performance and Sizing Guide
Ceph Performance and Sizing GuideCeph Performance and Sizing Guide
Ceph Performance and Sizing Guide
 
Development to Production with Sharded MongoDB Clusters
Development to Production with Sharded MongoDB ClustersDevelopment to Production with Sharded MongoDB Clusters
Development to Production with Sharded MongoDB Clusters
 
Caching solutions with Redis
Caching solutions   with RedisCaching solutions   with Redis
Caching solutions with Redis
 
微博cache设计谈
微博cache设计谈微博cache设计谈
微博cache设计谈
 
Redis introduction
Redis introductionRedis introduction
Redis introduction
 
Red Hat Storage Server Administration Deep Dive
Red Hat Storage Server Administration Deep DiveRed Hat Storage Server Administration Deep Dive
Red Hat Storage Server Administration Deep Dive
 
Bluestore
BluestoreBluestore
Bluestore
 
Redis trouble shooting_eng
Redis trouble shooting_engRedis trouble shooting_eng
Redis trouble shooting_eng
 
Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "
 

Destaque

高性能No sql数据库redis
高性能No sql数据库redis高性能No sql数据库redis
高性能No sql数据库redis
paitoubing
 

Destaque (20)

Redis介绍
Redis介绍Redis介绍
Redis介绍
 
新浪微博开放平台Redis实战
新浪微博开放平台Redis实战新浪微博开放平台Redis实战
新浪微博开放平台Redis实战
 
高性能No sql数据库redis
高性能No sql数据库redis高性能No sql数据库redis
高性能No sql数据库redis
 
redis 适用场景与实现
redis 适用场景与实现redis 适用场景与实现
redis 适用场景与实现
 
美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路
美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路
美团点评沙龙 飞行中换引擎--美团配送业务系统的架构演进之路
 
Redis 常见使用模式分析
Redis 常见使用模式分析Redis 常见使用模式分析
Redis 常见使用模式分析
 
美团点评技术沙龙010-Redis Cluster运维实践
美团点评技术沙龙010-Redis Cluster运维实践美团点评技术沙龙010-Redis Cluster运维实践
美团点评技术沙龙010-Redis Cluster运维实践
 
Redis cluster
Redis clusterRedis cluster
Redis cluster
 
Everything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to askEverything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to ask
 
Redis and Groovy and Grails - gr8conf 2011
Redis and Groovy and Grails - gr8conf 2011Redis and Groovy and Grails - gr8conf 2011
Redis and Groovy and Grails - gr8conf 2011
 
美团点评沙龙012-初创电商的物流摸索
美团点评沙龙012-初创电商的物流摸索美团点评沙龙012-初创电商的物流摸索
美团点评沙龙012-初创电商的物流摸索
 
美团点评技术沙龙010-点评RDS系统介绍
美团点评技术沙龙010-点评RDS系统介绍美团点评技术沙龙010-点评RDS系统介绍
美团点评技术沙龙010-点评RDS系统介绍
 
美团点评技术沙龙09 - 美团配送智能调度实践
美团点评技术沙龙09 - 美团配送智能调度实践美团点评技术沙龙09 - 美团配送智能调度实践
美团点评技术沙龙09 - 美团配送智能调度实践
 
美团点评技术沙龙010-美团Atlas实践
美团点评技术沙龙010-美团Atlas实践美团点评技术沙龙010-美团Atlas实践
美团点评技术沙龙010-美团Atlas实践
 
美团点评技术沙龙011 - 移动app兼容性测试工具Spider
美团点评技术沙龙011 - 移动app兼容性测试工具Spider 美团点评技术沙龙011 - 移动app兼容性测试工具Spider
美团点评技术沙龙011 - 移动app兼容性测试工具Spider
 
Redis Replication
Redis ReplicationRedis Replication
Redis Replication
 
美团点评沙龙012-从零到千万量级的实时物流平台架构实践
美团点评沙龙012-从零到千万量级的实时物流平台架构实践美团点评沙龙012-从零到千万量级的实时物流平台架构实践
美团点评沙龙012-从零到千万量级的实时物流平台架构实践
 
美团点评技术沙龙011 - 团购系统流量和容量评估实践
美团点评技术沙龙011 - 团购系统流量和容量评估实践美团点评技术沙龙011 - 团购系统流量和容量评估实践
美团点评技术沙龙011 - 团购系统流量和容量评估实践
 
美团点评技术沙龙011 - 客户端用户体验数据量化
美团点评技术沙龙011 - 客户端用户体验数据量化美团点评技术沙龙011 - 客户端用户体验数据量化
美团点评技术沙龙011 - 客户端用户体验数据量化
 
Taste Rabbitmq
Taste RabbitmqTaste Rabbitmq
Taste Rabbitmq
 

Semelhante a 深入了解Redis

Bottom to Top Stack Optimization - CICON2011
Bottom to Top Stack Optimization - CICON2011Bottom to Top Stack Optimization - CICON2011
Bottom to Top Stack Optimization - CICON2011
CodeIgniter Conference
 
Cassandra
CassandraCassandra
Cassandra
exsuns
 
ZFS for Databases
ZFS for DatabasesZFS for Databases
ZFS for Databases
ahl0003
 
Scaling web applications with cassandra presentation
Scaling web applications with cassandra presentationScaling web applications with cassandra presentation
Scaling web applications with cassandra presentation
Murat Çakal
 
Talk about apache cassandra, TWJUG 2011
Talk about apache cassandra, TWJUG 2011Talk about apache cassandra, TWJUG 2011
Talk about apache cassandra, TWJUG 2011
Boris Yen
 

Semelhante a 深入了解Redis (20)

Build an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on CephBuild an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on Ceph
 
Bottom to Top Stack Optimization - CICON2011
Bottom to Top Stack Optimization - CICON2011Bottom to Top Stack Optimization - CICON2011
Bottom to Top Stack Optimization - CICON2011
 
Bottom to Top Stack Optimization with LAMP
Bottom to Top Stack Optimization with LAMPBottom to Top Stack Optimization with LAMP
Bottom to Top Stack Optimization with LAMP
 
Strata - 03/31/2012
Strata - 03/31/2012Strata - 03/31/2012
Strata - 03/31/2012
 
Cassandra
CassandraCassandra
Cassandra
 
ZFS for Databases
ZFS for DatabasesZFS for Databases
ZFS for Databases
 
Comparison of foss distributed storage
Comparison of foss distributed storageComparison of foss distributed storage
Comparison of foss distributed storage
 
Scale 10x 01:22:12
Scale 10x 01:22:12Scale 10x 01:22:12
Scale 10x 01:22:12
 
美团技术沙龙04 - 高性能服务器架构设计和调优
美团技术沙龙04 - 高性能服务器架构设计和调优美团技术沙龙04 - 高性能服务器架构设计和调优
美团技术沙龙04 - 高性能服务器架构设计和调优
 
Comparison of-foss-distributed-storage
Comparison of-foss-distributed-storageComparison of-foss-distributed-storage
Comparison of-foss-distributed-storage
 
DEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them all
DEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them allDEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them all
DEF CON 27- ITZIK KOTLER and AMIT KLEIN - gotta catch them all
 
(DAT407) Amazon ElastiCache: Deep Dive
(DAT407) Amazon ElastiCache: Deep Dive(DAT407) Amazon ElastiCache: Deep Dive
(DAT407) Amazon ElastiCache: Deep Dive
 
What's new in Jewel and Beyond
What's new in Jewel and BeyondWhat's new in Jewel and Beyond
What's new in Jewel and Beyond
 
Automatic tool for static analysis
Automatic tool for static analysisAutomatic tool for static analysis
Automatic tool for static analysis
 
Scaling web applications with cassandra presentation
Scaling web applications with cassandra presentationScaling web applications with cassandra presentation
Scaling web applications with cassandra presentation
 
Community Update at OpenStack Summit Boston
Community Update at OpenStack Summit BostonCommunity Update at OpenStack Summit Boston
Community Update at OpenStack Summit Boston
 
Talk about apache cassandra, TWJUG 2011
Talk about apache cassandra, TWJUG 2011Talk about apache cassandra, TWJUG 2011
Talk about apache cassandra, TWJUG 2011
 
Talk About Apache Cassandra
Talk About Apache CassandraTalk About Apache Cassandra
Talk About Apache Cassandra
 
Hbase: an introduction
Hbase: an introductionHbase: an introduction
Hbase: an introduction
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouse
 

Mais de iammutex

Realtime hadoopsigmod2011
Realtime hadoopsigmod2011Realtime hadoopsigmod2011
Realtime hadoopsigmod2011
iammutex
 

Mais de iammutex (20)

Scaling Instagram
Scaling InstagramScaling Instagram
Scaling Instagram
 
NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析NoSQL误用和常见陷阱分析
NoSQL误用和常见陷阱分析
 
MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用MongoDB 在盛大大数据量下的应用
MongoDB 在盛大大数据量下的应用
 
8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slide8 minute MongoDB tutorial slide
8 minute MongoDB tutorial slide
 
skip list
skip listskip list
skip list
 
Thoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency ModelsThoughts on Transaction and Consistency Models
Thoughts on Transaction and Consistency Models
 
Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告Rethink db&tokudb调研测试报告
Rethink db&tokudb调研测试报告
 
Introduction to couchdb
Introduction to couchdbIntroduction to couchdb
Introduction to couchdb
 
What every data programmer needs to know about disks
What every data programmer needs to know about disksWhat every data programmer needs to know about disks
What every data programmer needs to know about disks
 
Ooredis
OoredisOoredis
Ooredis
 
Ooredis
OoredisOoredis
Ooredis
 
redis运维之道
redis运维之道redis运维之道
redis运维之道
 
Realtime hadoopsigmod2011
Realtime hadoopsigmod2011Realtime hadoopsigmod2011
Realtime hadoopsigmod2011
 
[译]No sql生态系统
[译]No sql生态系统[译]No sql生态系统
[译]No sql生态系统
 
Couchdb + Membase = Couchbase
Couchdb + Membase = CouchbaseCouchdb + Membase = Couchbase
Couchdb + Membase = Couchbase
 
Redis cluster
Redis clusterRedis cluster
Redis cluster
 
Hadoop introduction berlin buzzwords 2011
Hadoop introduction   berlin buzzwords 2011Hadoop introduction   berlin buzzwords 2011
Hadoop introduction berlin buzzwords 2011
 
Couchdb and me
Couchdb and meCouchdb and me
Couchdb and me
 
10 Key MongoDB Performance Indicators
10 Key MongoDB Performance Indicators  10 Key MongoDB Performance Indicators
10 Key MongoDB Performance Indicators
 
MongoDB开发应用实践
MongoDB开发应用实践MongoDB开发应用实践
MongoDB开发应用实践
 

Último

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Último (20)

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelMcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by Anitaraj
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKSpring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 

深入了解Redis

  • 1. 深入了解Redis 宋传胜 Chuansheng.song@langtaojin.com
  • 2. 简单介绍 • 文本协议 memcached类似 • KEY 可打印字符 • VALUE支持的类型 – STRINGS – LIST – SET – SORTED SET – HASH • 高性能 (100k+ SET/80k+ GET)/s • 序列化支持 • 主从同步支持 • 客户端自己实现sharding
  • 3. 基本数据结构 • RedisObject (Redis.h) – key/value对象的基础
  • 4. KEY • 基本命令 – get/set/exists O(1) – setnx/randomkey/rename/renamenx/dbsize/type – keys pattern – del 一次删除多个key – expire key seconds 指定过期时间 – ttl key 剩余过期时间 – select db – move key newdb – flush db/flushall
  • 5. EXPIRE AND LRU CACHE • Volatile Key (Db.c) – When the key is set to a new value using the SET command, or when a key is destroyed via DEL, the timeout is removed from the key. • Enhanced Lazy Expiration algorithm – Redis does not constantly monitor keys that are going to be expired. Keys are expired simply when some client tries to access a key, and the key is found to be timed out. • Work as LRU cache (memcached) – Maxmemory/maxmemory-policy – When memory limit was already reached, server will remove some old data deleting a volatile key, even if the key is still far from expiring automatically. – Random get keys, delete by lru rules.
  • 7. 基本Value数据结构 • STRING(sds.c) – set/setnx/get/getset/mget/mset/msetnx •nx – not exists •m -- multiple – incr/decr/incrby/decrby – getrange/append
  • 8. 基本Value数据结构 • LIST (T_list.c) – REDIS_LIST 类型, 如果其 entry 小于配置值: list- max-ziplist-entries 或 value字符串的长度小于 list- max-ziplist-value,使用ziplist数据结构,否则使用 标准的Doubly linked list – (l/r)push/(l/r)pop/llen O(1) – b(l/r)pop支持阻塞等待,避免了轮循 – lrange/ltrim/lrem/lset/rpoplpush
  • 9. LIST • Ziplist (Ziplist.c) – O(mem_size) add/delete – list-max-ziplist-entries (default: 1024) – list-max-ziplist-value (default: 32)
  • 10. LIST • Ziplist (continue) – zlentry (unencode structure)
  • 11. LIST • Doubly linked list (Adlist.c)
  • 12. 基本Value数据结构 • SET (T_set.c) – hashtable + Intset – String的无序集合 – sadd/srem/sismember/scard O(1) – spop/srandmember – smove – smembers – sinter(store) O(C) – sunion(store)/sdiff(store) O(N)
  • 13. INTSET • INTSET (Intset.c) – 都是整型时数据结构退化成排序的intset – Good fit for size up to 20-50K – set-max-intset-entries (default: 4096) – O(n) search – O(log(n) + mem_size) add/del
  • 14. 基本Value数据结构 • SORTED SET(T_zset.c) hashtable + skip list – 按照key的score排序 – zadd/zrem/zrank/zrevrank O(log(n)) – zcard O(1) – zincrby 修改score – zrange/zrevrange/zrangebyscore/zscore O(log(N)+M) – zremrangebyrank/zremrangebyscore O(log(N)+M)
  • 15. SKIP LIST • Skip List (T_zset.c) – skip list  redblack tree  AVL tree (more balanced) – lockfree/concurrency
  • 16. 基本Value数据结构 • HASH (T_hash.c) zipmap + hashtable – 如果其 entry 小于配置值: hash-max-zipmap- entries 或 value字符串的长度小于 hash-max- zipmap-value ,使用zipmap数据结构 – Value只能是string类型 – hset/hget/hexists O(1) – hmset/hexists/hincrby/hdel/hlen – hmget O(N) – hkeys/hvals/hgetall
  • 17. ZIPMAP • zipmap – O(n) search O(mem_size) add/delete
  • 18. 内存管理 • zmalloc.c – 简单内存管理 – 支持tcmalloc USE_TCMALLOC – zmalloc(size_t size)  malloc(size+PREFIX_SIZE) – fragmentation_ratio = zmalloc_get_rss/zmalloc_used_memory
  • 19. PUBSUB • SUBSCRIBE/UNSUBSCRIBE/PSUBSCRIBE/PU NSUBSCRIBE • PUBLISH • 连接断开后信息丢失
  • 20. TRANSACTION • MULTI (不支持CAS) – MULTI – SET foo 0 – INCR foo – EXEC • WATCH (Check and Set ) – WATCH theKey – v = GET theKey – MULTI – SET theKey v+1 – EXEC • UNWATCH/DISCARD
  • 21. 提升网络效率、减少内存占用 • MGET/MSET • PIPELINING – $ (echo -en "PINGrnPINGrnPINGrn"; sleep 1) | nc localhost 6379 +PONG +PONG +PONG • 压缩数据,触发ziplist/zipmap/intset • 使用 GETBIT/SETBIT/GETRANGE/SETRANGE 压缩多 个属性 • 使用HASH • 使用32位redis + sharding
  • 22. Client Side Sharding • Consistent Hashing – Redis Sharding at Craigslist
  • 23. 网络层 • Anet.c • 事件驱动 ae.c  ae_epoll.c/ae_kqueue.c/ae_select.c • 非阻塞单线程 • 支持timer • 在VM场景下有可能多线程
  • 25. 网络层对比 • lighttpd/nginx的网络IO模型 – 多进程,单线程非阻塞IO • apache的MPM模型 – 多进程prefork,多线程
  • 26. PERSISTENCE • SNAPSHOT (Rdb.c) • 可以关闭 • BGSAVE or save xseconds ychanges • fork – parent - disable rehash, periodic check child on serverCron (100ms) – child - copy-on-write, save whole db, exit • Need to turn on the overcommit_memory setting if you want to deal with a dataset more than 1/2 of RAM, which contradicts our habit to battle against OOM killer as a database administrator.
  • 27. Append Only File • Append to log file on every change (Aof.c) • fsync() policy – always/os decide/every second – Always with battery backed raid controllers – Every second by default (innodb_flush_log_at_trx_commit=2 in mysql) • Compact aof file – BGREWRITEAOF/REWRITEAOF • Fork – child – write new aof in temporary file – parent – write change in both old aof file and memory buffer, append memory buffer to temporary file when child done, rename temporary to aof file. • redis-check-aof --fix <filename>
  • 29. Replication • 支持AUTH • 起动过程 – slave •SYNC •Wait – master •Issue a BGSAVE, or wait if BGSAVE in progress
  • 30. Virtual Memory • Redis VM is now deprecated. Redis 2.4 will be the latest Redis version featuring Virtual Memory! • Why not OS VM – Varnish What’s wrong with 1975 programming ? – Redis What’s wrong with 2006 programming? • Blocking when page fault • 4k pages granularity • Optimal disk persistent format, 10:1 • Keys can’t be swapped out • vm is read only when BGSAVE/BGREWRITEAOF in progress • vm-max-memory/vm-pages/vm-page-size
  • 31. Virtual Memory • Blocking vm • Threaded vm – Before command is executed, check the value – Block client and load, then signal by unix pipe – Continue • Recommended filesystem – ext3 or any other file system with good support for sparse files • redis-stat vmstat • InnoDB innodb_buffer_pool_size – swap and durability
  • 32. Future • Diskstore (diskstore.c dscache.c) – All data on disk, B+tree in future – Swaps data into memory when needed • Leveldb/InnoDB Change Buffer – LSM tree
  • 33. 其他相关的代码 • quicksort (pqsort.c) • lzf compression (lzf_c.c lzf_d.c) • Sha1 (Sha1.c) • Syncio.c • SORT (Sort.c)
  • 34. 线上Redis集群情况 • 16台服务器,兼做后端服务器 • 45个Redis Instance进程,三组服务 • 总内存 77G VIRT,71G RSS • 使用了主从复制、snapshot持久化,VM支持 ,客户端sharding、pipelining、hash优化
  • 35. Redis应用 • Resque – message queue of github (celery) • Nginx HttpRedis module • Redweb - Web administrative and query UI • A fast, fuzzy, full-text index using Redis • Redis-backed BloomFilter(s) in Ruby • Soulmate : Auto Complete with Redis • Rate limiting with Redis • Redis Sharding • Locking with setnx • A Collection Of Redis Use Cases • Radishapp Redis monitoring • openredis.com hosting service
  • 36. Contribute to Redis • Fork redis on github – https://github.com/antirez/redis • Choose branch and commit your changes • Create a pull request • Wait for approval – https://github.com/antirez/redis/pulls • My contribute for read only slave – https://github.com/antirez/redis/pull/47
  • 37. Resources • DOCUMENATION – http://redis.io/documentation – http://antirez.com/ – http://groups.google.com/group/redis-db • 本文参考的Redis资料集合 – http://www.delicious.com/fakechris/redis – http://www.delicious.com/tag/redis