O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Fast Data at Scale with Amazon ElastiCache for Redis

3.395 visualizações

Publicada em

Database performance is a key factor for keeping your app running fast. As the amount of data and request throughput grow, it becomes harder to keep it nimble and performant. In this webinar, we will discuss using Redis, a popular NoSQL key-value store, to run your data fast at scale. Redis can be used as a cache in front of a database or as a fast in-memory data store. We will also cover how you can use ElastiCache for Redis to easily set up a large multi-terabyte Redis environment and operate it with zero management.

Learning Objectives:
• Understand fast data
• Understand the capabilities of Amazon Elasticache for Redis to cache Relational and NoSQL databases
• Use Amazon Elasticache for Redis as a primary data store

Who Should Attend:
• Developers, Data Architects

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Fast Data at Scale with Amazon ElastiCache for Redis

  1. 1. Fast Data Acting on Information µs are the new ms
  2. 2. What is Fast Data? • Generated fast • Actionable fast
  3. 3. Source: Financial Trading Bids, asks, trades, equities, bonds, derivatives …
  4. 4. Source: Sensors Any Thing in the Internet of Things • Temperature • Location • Velocity • Acceleration • Radiation • Electricity
  5. 5. Source: Clickstream You click, someone tracks…
  6. 6. Source: Log Aggregation • Servers • Storage • Applications • Network • Security
  7. 7. Capturing the Data AWS IoT IoT Device Amazon ElasticsearchAWS Lambda Hot Data Amazon ElastiCache Amazon DynamoDB Longer Retention Data Lake Amazon S3 Amazon Glacier Cold Data Amazon KinesisFirehose 7 Data Sources 1 Amazon Kinesis Streams
  8. 8. 8 In-Memory Key-Value Store High-performance Redis and Memcached Fully managed; Zero admin Highly Available and Reliable Hardened by Amazon
  9. 9. Amazon RDS Amazon S3 Request Rate High Low Latency Low High Data Volume Low High Amazon Glacier Amazon CloudSearch & Elasticsearch Structure Low High Amazon DynamoDB Amazon ElastiCache HDFS
  10. 10. Redis – the fast in-memory database Powerful ~200 commands + Lua scripting In-memory database Utility data structures strings, lists, hashes, sets, sorted sets, bitmaps & HyperLogLogs Simple Atomic operations supports transactions has ACID properties Ridiculously fast! <500microsecond latency for most commands Highly Available replication Persistent snapshots or append-only log Open Source 10
  11. 11. One Simple Example …. + on-demand c3.8xlarge 3,000,000 objects + 100 bytes each memtier benchmark 50 read : 50 write 1,228,432 TPS Source: http://highscalability.com/blog/2014/8/27/the-12m-opssec- redis-cloud-cluster-single-server-unbenchmar.html
  12. 12. 12 Fully managed service = Automated Operations Redis datastore hosted on Amazon EC2 Amazon ElastiCache for Redis
  13. 13. ElastiCache for Redis Key Attributes Extreme Performance Sub-millisecond access latencies 4.5 million writes / sec 20 million reads / sec Open Source Compatible Compatible with Open Source Redis Works with any Redis Client Fully Managed Automates node replacement, software patching, upgrades and backups CloudWatch automates monitoring of cache performance metrics Secure and Hardened Supports Amazon VPC and IAM for secure and fine grained access. Monitors nodes and applies security patches when needed Highly Available and Scalable Clusters of up to 15 shards, each with a primary node and up to 5 replica nodes. Multi-AZ with rapid automatic failover, with no human intervention or code changes needed. Cost Effective Pay as low as US$0.017 per hour. Get started with AWS Free Tier. Zero data transfer costs for cross-AZ replication
  14. 14. Amazon ElastiCache Enhanced Redis Engine on ElastiCache Hardened by Amazon RDB Without Fork •Mitigate the risk of increased swap usage during syncs and snapshots. Dynamic write throttling •To improve output buffer management when the node’s memory is close to being exhausted Smoother failovers •Clusters recover faster as replicas will avoid flushing their data to do a full re-sync with the primary.
  15. 15. GEO Commands Store and retrieve longitude, latitude and radius as a Sorted Set • GEOADD • GEODIST • GEOHASH • GEOPOS • GEORADIUS • GEORADIUSBYMEMBER 15 Amazon ElastiCache for Redis
  16. 16. BITFIELD • Memory-efficient approach to storing many small integers as a large bitmap • Treats a Redis string as an array of bits • GET, SET on INCRBY any group of bits within a string • Control increment / decrement behavior with OVERFLOW 16 Amazon ElastiCache for Redis
  17. 17. Clusters! • Enables Horizontal Scaling • In ElastiCache for Redis 3.2: • a Cluster is one or more Shards • a Shard is a primary node plus up to 5 replica nodes • Up to 15 Shards per Cluster • 15 r3.8xlarge = 3.5TiB! • Cluster-level backup and restore 17 S5 S1 S2 S4 S3 Client Amazon ElastiCache for Redis Clusters!
  18. 18. Autosharding • 16384 hash slots per Cluster • Slot for a key is CRC16 modulo {key} • Slots are distributed across the Cluster into Shards • Developers must use a Redis cluster client! • Clients are redirected to the correct shard • Smart clients store a map 18 S5 S1 S3 S4 S3 Client Shard S1 = slots 0 – 3276 Shard S2 = slots 3277 – 6553 Shard S3 = slots 6554 – 9829 Shard S4 = slots 9830 – 13106 Shard S5 = slots 13107 - 16383 Amazon ElastiCache for Redis
  19. 19. Faster Recovery • A Cluster consists of 1 – 15 Shards • Each Shard has a primary node and up to 5 replica nodes • In the event of a primary node outage, a replica node in that Shard will be promoted Primary S1a S1b S1c S2a S2b S2c S3a S3b S3c Replica Shard Shard Shard X S2c S2a Amazon ElastiCache for Redis
  20. 20. 20 Open Source Distributed Index Managed Service using Elasticsearch and Kibana Fully managed; Zero admin Highly Available and Reliable RESTful API for easy integrationAmazon Elasticsearch Service
  21. 21. Logstash REST CWL Agent EC2 Instances Amazon Kinesis Amazon RDS Amazon DynamoDB Amazon SQS Queue Logstash Cluster Amazon Elasticsearch Service Amazon CloudWatch AWS Lambda AWS CloudTrail Access Logs Amazon VPC Flow Logs Amazon S3 bucket AWS IoT Amazon Kinesis Firehose Integration with the AWS ecosystem Amazon ECS
  22. 22. Using Fast Data: Finance Fraud Detection Regulatory Compliance
  23. 23. Using Fast Data: Retail Recommendations
  24. 24. Using Fast Data: Utilities Incident Response
  25. 25. Using Fast Data: Health Care Drug Interactions
  26. 26. What’s Next? How will Fast Data change your industry? Featured Tools: Amazon ElastiCache for Redis Amazon Elasticsearch Service Darin Briskman ~ briskman@amazon.com