O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Polyglot Persistence Powering Microservices

297 visualizações

Publicada em

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2mFzDSg.

Roopa Tangirala takes a look at Netflix’s common platform used to manage, maintain, and scale persistence infrastructures (including building reliable systems on unreliable infrastructure). She shares the benefits, pitfalls, and lessons learned from their polyglot persistence architecture to shape the success of various services. Filmed at qconsf.com.

Roopa Tangirala leads the Cloud Database Engineering team at Netflix, responsible for cloud persistent runtime stores for Netflix, ensuring data availability, durability, and scalability to meet the growing business needs. The team specializes in providing polyglot persistence as a service which includes Cassandra, ElasticSearch, Dynomite, Mysql etc.

Publicada em: Tecnologia
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Polyglot Persistence Powering Microservices

  1. 1. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ microservices-polyglot-persistence
  2. 2. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  3. 3. Microservice Layout in Netflix
  4. 4. Polyglot Persistence Powering Microservices Roopa Tangirala Engineering Manager Netflix
  5. 5. Agenda • 5 Use Cases • Challenges • Current Approach • Takeaway
  6. 6. AWS S3 CDE
  7. 7. Search, Analyze and visualize in near real time Distributed in-memory caching solution based on memcached Distributed NOSQL database to handle large datasets providing high availability. Distributed dynamo layer for different storage engines and protocols supporting Redis, memcached, RocksDB TitanDB is scalable graph database optimized for storing and querying graph datasets.
  8. 8. Us-west-2 Oregon Us-east-1 North Virginia Eu-west-1 Ireland
  9. 9. CDN URL
  10. 10. Requirements - CDN URL • High availability • Very low latency reads/writes (less than 1ms) • High Throughput per node
  11. 11. GUESS?
  12. 12. Distributed In Memory Very low Latency responses
  13. 13. Playback Error
  14. 14. PLAYBACK CONTEXT (Tracks + Track Urls) NETFLIX OPENCONNECT CDN URLS
  15. 15. PLAYBACK CONTEXT COUNTRY USER PREFERENCES TITLE METADATA DEVICE NETWORK
  16. 16. Requirements - Playback Error • Quick Incident Resolution • Interactive Dashboards • Near realtime Search • Ad Hoc Queries
  17. 17. GUESS?
  18. 18. Powerful Search & Analytics Interactive Dashboards
  19. 19. Interactive Exploration
  20. 20. Top N queries
  21. 21. Incident To Resolution Time 2+ Hours Under 10 Minutes
  22. 22. Viewing History
  23. 23. Requirements - Viewing History • Time series dataset • Support high writes • Cross region replication • Large dataset
  24. 24. Growth of Viewing History 33
  25. 25. GUESS?
  26. 26. Multi-datacenter, multi-directional replication Highly availability and scalability
  27. 27. Data Model 36
  28. 28. New Data Model 37
  29. 29. Artwork Display Set B Display Set A Character Movie Video Track PNGWebp Audio Track JPG Video Seg Es Sub Fr Dub Montage TEXT Track Trailer Fr Sub Person Digital Asset Management
  30. 30. Requirements - DAM • One backend plane for all asset metadata • Storage of relationships/connected data • Searchable
  31. 31. GUESS?
  32. 32. Distributed GraphDB Support for various storage backends
  33. 33. Distributed Delayed Queues
  34. 34. Requirements - Delayed Queues • Distributed • Highly concurrent • At-least-once delivery semantics • Delayed queue • Priorities within the shard
  35. 35. GUESS?
  36. 36. Pluggable datastore supporting Redis Multi-datacenter replication
  37. 37. Data Model For each queue three set of Redis data structures are maintained: 1. A Sorted Set containing queued elements by score. 2. A Hash set that contains message payload, with key as message ID. 3. A Sorted Set containing messages consumed by client but yet to be acknowledged. Un-ack set.
  38. 38. Prediction @ CDE
  39. 39. Current Approach
  40. 40. Subject Matter Expert
  41. 41. CDE Service Empowering CDE to provide datastores as a service
  42. 42. CDE Service •Thresholds/SLAs •Cluster metadata •Self Service •Contact information •Maintenance windows
  43. 43. Architecture PAGE CDE Alert if needed S C H E D U L E R R E M E D I A T I O N CDE SERVICE Alert Atlas Mantis Datastore Cluster Metadata Cluster Metadata/ Advisor Maintenance Remediation MONITORING
  44. 44. SLA cass_xyz abc@netflix.com cde abc@netflix.com
  45. 45. 10 6 4 cas_xyz cass_abc cass_test abc@netf lix.com abc@netf lix.com xyz@netfl ix.com xyz@netfl ix.com 343 34 4 34 2 3 34 5 6 65 5 34 65 56 55 56 2
  46. 46. Machine learning Automic (UC4)
  47. 47. Pattern in Disk usage
  48. 48. Cde Channel
  49. 49. Common Approach CRON System Job RunnerJob RunnerJob RunnerJob Runner
  50. 50. Streaming micro-services Source Stage Sink Source Stage Sink Source Stage Sink Source - input, handles backpressure Stage - business logic Sink - output, handles backpressure Source Stage Sink Mantis Job
  51. 51. Real Time Dash (Macro View) cluster_3 cluster_6
  52. 52. Real Time Dash (Cluster View) 1
  53. 53. Takeaway
  54. 54. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ microservices-polyglot-persistence

×