CROSS | AWS - Meetup Banco de dados na nuvem

459 visualizações

Publicada em

Apresentação realizada no Meetup de 28 de outubro na TOTVS.
Palestra sobre NoSQL e DynamoDB por Julio M. Faerman.

TOTVS. Juntos, fazemos melhor.

Aproveite também para conferir as novidades no site da TOTVS. Acesse: http://www.totvs.com. Ficou interessado? Agende uma visita com um de nossos consultores: https://www.totvs.com/contatos.

Siga a TOTVS nas redes sociais:
Facebook: https://www.facebook.com/totvs
Twitter: https://twitter.com/TOTVS
LinkedIn: https://www.linkedin.com/company/totvs

Publicada em: Tecnologia
0 comentários
1 gostou
Estatísticas
Notas
  • Seja o primeiro a comentar

Sem downloads
Visualizações
Visualizações totais
459
No SlideShare
0
A partir de incorporações
0
Número de incorporações
3
Ações
Compartilhamentos
0
Downloads
3
Comentários
0
Gostaram
1
Incorporações 0
Nenhuma incorporação

Nenhuma nota no slide

CROSS | AWS - Meetup Banco de dados na nuvem

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Julio M. Faerman AWS Developer Relations LatAm @jmfaerman TOTVS Arena Meetup Bancos de Dados em Nuvem 2015-10-28 DAT204 NoSQL? No Worries Building Scalable Applications on AWS NoSQL Services
  2. 2. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Julio M. Faerman AWS Developer Relations LatAm @jmfaerman TOTVS Arena Meetup Bancos de Dados em Nuvem 2015-10-28 Deep Dive: Amazon DynamoDB DAT401
  3. 3. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Julio M. Faerman AWS Developer Relations LatAm @jmfaerman TOTVS Arena Meetup Bancos de Dados em Nuvem 2015-10-28 Amazon ElastiCache Deep Dive Scaling Your Data in a Real-Time World DAT407
  4. 4. NoSQL
  5. 5. NoSQL available on AWS Managed Amazon DynamoDB Amazon ElastiCache • Memcached • Redis Unmanaged Apache Cassandra MongoDB CouchDB Riak ….
  6. 6. Why managed database services?
  7. 7. If you host your databases on premises Power, HVAC, net Rack & stack Server maintenance OS patches DB s/w patches Database backups Scaling High availability DB s/w installs OS installation you App optimization
  8. 8. If you host your databases on premises Power, HVAC, net Rack & stack Server maintenance OS patches DB s/w patches Database backups Scaling High availability DB s/w installs OS installation you App optimization
  9. 9. If you host your databases on Amazon EC2 Power, HVAC, net Rack & stack Server maintenance OS patches DB s/w patches Database backups Scaling High availability DB s/w installs OS installation you App optimization
  10. 10. OS patches DB s/w patches Database backups Scaling High availability DB s/w installs you App optimization Power, HVAC, net Rack & stack Server maintenance OS installation If you host your databases on Amazon EC2
  11. 11. If you choose a managed database service Power, HVAC, net Rack & stack Server maintenance OS patches DB s/w patches Database backups App optimization High availability DB s/w installs OS installation you Scaling
  12. 12. Amazon DynamoDB
  13. 13. Amazon DynamoDB • Managed NoSQL database service • Highly scalable • Consistent, single-digit millisecond latency at any scale • Highly durable and available—3x replication • Accessible via simple and powerful APIs • Supports both document and key-value data models • No table size or throughout limits
  14. 14. Provisioned throughput model and scaling Throughput provisioned at the table level WCU and RCU are independent Consumed capacity is measured per operation Scaling is achieved through automatic partitioning WCURCU
  15. 15. Table Table Items Attributes Hash Key Range Key Mandatory Key-value access pattern Determines data distribution Optional Model 1:N relationships Enables rich query capabilities All items for a hash key ==, <, >, >=, <= “begins with” “between” sorted results counts top/bottom N values paged responses
  16. 16. Table and item API DynamoDB ListStreams DescribeStream GetShardIterator GetRecords Streams API Admin CRUD Create Table Put/Get Item Update Table Batch Put/Get Item Delete Table Update Item Describe Table Delete Item Query Scan
  17. 17. 00 FF00 55 A954 AA FF Hash table Hash key uniquely identifies an item Hash key is used for building an unordered hash index Table can be partitioned for scale Id = 1 Name = Jim Hash (1) = 7B Id = 2 Name = Andy Dept = Eng Hash (2) = 48 Id = 3 Name = Kim Dept = Ops Hash (3) = CD Key Space
  18. 18. Hash-range table Hash key and range key together uniquely identify an Item Within unordered hash index, data is sorted by the range key No limit on the number of items (∞) per hash key • Except if you have local secondary indexes 00:0 FF:∞ Hash (2) = 48 Customer# = 2 Order# = 10 Item = Pen Customer# = 2 Order# = 11 Item = Shoes Customer# = 1 Order# = 10 Item = Toy Customer# = 1 Order# = 11 Item = Boots Hash (1) = 7B Customer# = 3 Order# = 10 Item = Book Customer# = 3 Order# = 11 Item = Paper Hash (3) = CD 55 A9:∞54:∞ AA Partition 1 Partition 2 Partition 3
  19. 19. Partitions are three-way replicated Id = 2 Name = Andy Dept = Engg Id = 3 Name = Kim Dept = Ops Id = 1 Name = Jim Id = 2 Name = Andy Dept = Engg Id = 3 Name = Kim Dept = Ops Id = 1 Name = Jim Id = 2 Name = Andy Dept = Engg Id = 3 Name = Kim Dept = Ops Id = 1 Name = Jim Replica 1 Replica 2 Replica 3 Partition 1 Partition 2 Partition N
  20. 20. Amazon ElastiCache
  21. 21. Why in-memory? • Everything is connected - phones, tablets, cars, air conditioners, toasters • Demand for real-time performance – online games, ad tech, eCommerce, social apps, etc. • Load is spikey and unpredictable • Database performance often the bottleneck
  22. 22. ELB App ExternalAPIs Modern Web / Mobile App
  23. 23. Memcached vs. Redis • In-memory • Multithreaded / Multicore • No persistence • String values • Easy horizontal clustering • In-memory • Single-threaded • Read replicas • Persistence • Atomic operations • Advanced data types - http://redis.io/topics/data-types • Pub/sub messaging
  24. 24. ElastiCache with Memcached – Scale Out Region Availability Zone A Availability Zone B Auto Scaling group ElastiCache cluster
  25. 25. Redis HA on ElastiCache Availability Zone #1 Availability Zone #2 writes use “Primary Endpoint” from Node Group reads use ‘replica’endpoints from Node Group *can use ‘primary’also Auto-Failover § Goes to replica with lowestreplication lag § No changes in DNS
  26. 26. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kuldeep Chowhan, Expedia Inc. October 7th 2015 How Expedia is Using DynamoDB and ElastiCache to Build a Real-Time Analytics System
  27. 27. What is it? (continued) Continuous flow of data ElastiCache – Redis DynamoDB Apache Storm infrastructure running on distributed EC2 instances Hit / Miss Request Response
  28. 28. Lessons learned • Take a closer look at the application access patterns • Most of the read/write requests are directed towards ‘most recent’ data • A lot of ‘repeat’ requests – wasting throughput • Read the documentation carefully • DynamoDB creates partitions for every 10 GB and in our case we have 460 partitions which caused increased response time in the application
  29. 29. Current design – performance improvements Added a caching layer • Repeat requests don’t even make it to DynamoDB • Current cache hit ratio – 3x (3000 hits per 1000 misses)
  30. 30. Results • Provisioned write capacity has been reduced from 35000 to 3500, reducing the cost significantly by 6x • Highly performant and high throughput application backed by DynamoDB and ElastiCache
  31. 31. Exciting stuff! How do I learn more? • DynamoDB: – DAT401 - Amazon DynamoDB Deep Dive: Schema Design, Indexing, JSON, Search, and More – will be available on YouTube – WRK302 - Event-Driven Programming – Wednesday, 10/7 3:15pm-5:15pm, Galileo 1006 – BDT313 - Amazon DynamoDB for Big Data – Thursday, 10/8 11am-12pm, Lando 4306 • ElastiCache – DAT407 - Amazon ElastiCache: Deep Dive, Thursday, 10/8 11am-12pm, Lando 4301B – “Performance at scale with Amazon ElastiCache” whitepaper.
  32. 32. Thank you!Obrigado! Julio M. Faerman AWS Developer Relations LatAm @jmfaerman

×