SlideShare uma empresa Scribd logo
1 de 25
Baixar para ler offline
Project Voldemort
A distributed database.
Presented To:
Sir Tariq Mehmood Presented By:
Fasiha Ikram
Aniqa Naeem
Voldemort
Voldemort is a distributed data store that is
designed as a key-value store used by
LinkedIn for high-scalability storage.
It is named after the fictional Harry Potter
villain Lord Voldemort.
there are job titles, job openings for people, Groups
and companies that offerings jobs.
Big
Data
Variety
velocity
volum
e
Need frequently read write
LinkedIn Big Data Problem
Voldemort Scale Both
• The amount of data we can store (write)
• The number of requests for that data (read)
Why Not Using Hadopp
Naturally the only way to do this is to spread both
the load and the data across many servers.
1. Need to find a way to split the data so that all
servers have different data
2. Need to find a way to handle server failures
without interrupting service
3. HBase still be write-heavy (due to horizontal
partitioning and use of SSTables, which are write
optimized)
Why voldemort
• Data is automatically replicated over multiple
servers.
• Data is automatically partitioned so each server
contains only a subset of the total data
• Provides tunable consistency (strict quorum or
eventual consistency)
• Server failure is handled transparently
• Pluggable Storage Engines -- BDB-JE, MySQL,
Read-Only
Why voldemort
• Pluggable serialization -- Protocol Buffers, Thrift,
Avro and Java Serialization
• Data items are versioned to maximize data
integrity in failure scenarios without
compromising availability of the system
• Each node is independent of other nodes with no
central point of failure or coordination
Why voldemort
• Good single node performance: you can expect 10-
20k operations per second depending on the
machines, the network, the disk system, and the
data replication factor
• Support for pluggable data placement strategies to
support things like distribution across data
centers that are geographically far apart.
Voldemort Storage Engines
 Trivial to integrate new persistence mechanisms
with Voldemort
 2 Classes:
Config(data) & Storage Engine(servers info)
 3 Operations:
put(k, v), get(k), delete(k)
 Complication:
k is Versioned<Key>
Architecture of Queue
Voldemort
Voldemort
Key-Value Storage
• To enable high performance and availability it
allow only very simple key-value data access.
• Both keys and values can be complex compound
objects including lists or maps, but none-the-less
the only supported queries are effectively the
following:
value = store.get(key)
store.put(key, value)
store.delete(key)
Query execution
• Voldemort supports hashtable semantics, so a
single value can be modified at a time and
retrieval is by primary key.
• This makes distribution across machines
particularly easy since everything can be split by
the primary key.
Consistent Hashing Mechanism
• In order to effectively Scaling , the data in
Voldemort is split-up in such a way that each item
is stored on multiple Servers.
• For retrieving data first figure out which is the
correct server to use. This partitioning is done via
a consistent hashing mechanism that let’s any
server calculate the location of data without doing
any expensive look ups
Detecting Failure
• Voldemort set an SLA (service level agreement)
for the requests and ban servers who cannot meet
their SLA (this could be because they are down,
because requests are timing out).
• Servers that violate this SLA get banned for a short
period of time, after which they attempt to restore
them.
Dealing With Failure
 Since each value is stored in multiple places it is
possible that one of these servers will not get
updated (say because it is crashed when the
update occurs).
 To solve this problem Voldemort uses a data
versioning mechanism called Vector Clocks.
 This data versioning allows the servers to detect
stale data when it is read and repair it.
Comparison to Hbase databases
Query
language
Architecture
Database
Model
Replication Issues
Voldemort API calls
Big Unordered
Map
Key-value
NoSQL
Distributed
data
structure
Topology
Aware
Routing
Strategies
Not
Satisfyin
g ACID
Properti
es.
Hbase
API calls
REST
XML
Thrift
Big Multi-
dimensional
Sorted Map
HDFS
Master-
slave/Master
-master
replication
Master
Slave
Which Is
Not
Highly
Availabl
e
Acid Properties
Voldemort Hbase
Atomicity
Yes Yes
Consistency
Yes Yes
Isolation
No Yes
Durability Yes Yes
Locking model Optimistic Locking MVCC
Free for commercial use Yes Yes
Industry Implementation
 Linkdin founder of voldemort
 shopping cart used by Gilt Groupe, which is
powered by Voldemort.
Use Case
 High-Performance Key-Value Store (Amazon
Dynamo clone)
 treats the key‐value store as an API and adds an
in‐memory caching layer, which means that you
can plug into the back end that makes the most
sense for your particular needs.
Pros
• only efficient queries are possible, very predictable
performance.
• easy to distribute across a cluster.
• clean separation of storage and logic.
• The storage layer is completely mockable so
development and unit testing can be done against a
throw-away in-memory storage system without
needing a real cluster.
Cons
• no complex query filters
• no foreign key constraints
• no triggers
• No built-in support for “multiple data center”-
aware routing (there must be 1 copy of each key in
at least one data center)
Conclusion
• It is basically just a big, distributed, persistent,
fault-tolerant hash table.
• The redundancy of storage makes the system
more resilient to server failure. Since each value is
stored N times, you can tolerate as many as N –
1 machine failures without data loss.
Refrences
1. http://blog.linkedin.com/2009/04/01/project-voldemort-part-ii-how-it-works/
2. http://blog.linkedin.com/2009/03/20/project-voldemort-scaling-simple-storage-at-
linkedin/
3. http://en.wikipedia.org/wiki/Voldemort_(distributed_data_store)
4. http://highscalability.com/product-project-voldemort-distributed-database
5. http://www.dummies.com/how-to/content/using-pluggable-storage-with-nosql.html
6. http://vschart.com/compare/project-voldemort/vs/hbase
7. http://www.project-voldemort.com/voldemort/design.html

Mais conteúdo relacionado

Mais procurados

Introduction to Distributed System
Introduction to Distributed SystemIntroduction to Distributed System
Introduction to Distributed SystemSunita Sahu
 
Cloud Computing: Hadoop
Cloud Computing: HadoopCloud Computing: Hadoop
Cloud Computing: Hadoopdarugar
 
(DAT401) Amazon DynamoDB Deep Dive
(DAT401) Amazon DynamoDB Deep Dive(DAT401) Amazon DynamoDB Deep Dive
(DAT401) Amazon DynamoDB Deep DiveAmazon Web Services
 
Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...
Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...
Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...Simplilearn
 
Data warehouse architecture
Data warehouse architectureData warehouse architecture
Data warehouse architecturepcherukumalla
 
Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners ...
Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners ...Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners ...
Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners ...Simplilearn
 
Introduction To HBase
Introduction To HBaseIntroduction To HBase
Introduction To HBaseAnil Gupta
 
Schemaless Databases
Schemaless DatabasesSchemaless Databases
Schemaless DatabasesDan Gunter
 
OLAP & DATA WAREHOUSE
OLAP & DATA WAREHOUSEOLAP & DATA WAREHOUSE
OLAP & DATA WAREHOUSEZalpa Rathod
 
Big Data Architecture
Big Data ArchitectureBig Data Architecture
Big Data ArchitectureGuido Schmutz
 

Mais procurados (20)

Apache HBase™
Apache HBase™Apache HBase™
Apache HBase™
 
Introduction to Distributed System
Introduction to Distributed SystemIntroduction to Distributed System
Introduction to Distributed System
 
Cloud Computing: Hadoop
Cloud Computing: HadoopCloud Computing: Hadoop
Cloud Computing: Hadoop
 
Nosql databases
Nosql databasesNosql databases
Nosql databases
 
(DAT401) Amazon DynamoDB Deep Dive
(DAT401) Amazon DynamoDB Deep Dive(DAT401) Amazon DynamoDB Deep Dive
(DAT401) Amazon DynamoDB Deep Dive
 
Introduction to HBase
Introduction to HBaseIntroduction to HBase
Introduction to HBase
 
Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...
Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...
Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...
 
Hive Hadoop
Hive HadoopHive Hadoop
Hive Hadoop
 
Cloud Deployments Models
Cloud Deployments ModelsCloud Deployments Models
Cloud Deployments Models
 
Oltp vs olap
Oltp vs olapOltp vs olap
Oltp vs olap
 
Real-Time Event Processing
Real-Time Event ProcessingReal-Time Event Processing
Real-Time Event Processing
 
Data warehouse architecture
Data warehouse architectureData warehouse architecture
Data warehouse architecture
 
Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners ...
Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners ...Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners ...
Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners ...
 
AWS PPT.pptx
AWS PPT.pptxAWS PPT.pptx
AWS PPT.pptx
 
Introduction to Amazon Redshift
Introduction to Amazon RedshiftIntroduction to Amazon Redshift
Introduction to Amazon Redshift
 
Introduction To HBase
Introduction To HBaseIntroduction To HBase
Introduction To HBase
 
Schemaless Databases
Schemaless DatabasesSchemaless Databases
Schemaless Databases
 
OLAP & DATA WAREHOUSE
OLAP & DATA WAREHOUSEOLAP & DATA WAREHOUSE
OLAP & DATA WAREHOUSE
 
03 hive query language (hql)
03 hive query language (hql)03 hive query language (hql)
03 hive query language (hql)
 
Big Data Architecture
Big Data ArchitectureBig Data Architecture
Big Data Architecture
 

Destaque

Voldemort : Prototype to Production
Voldemort : Prototype to ProductionVoldemort : Prototype to Production
Voldemort : Prototype to ProductionVinoth Chandar
 
Project Voldemort: Big data loading
Project Voldemort: Big data loadingProject Voldemort: Big data loading
Project Voldemort: Big data loadingDan Harvey
 
Introducción a Voldemort - Innova4j
Introducción a Voldemort - Innova4jIntroducción a Voldemort - Innova4j
Introducción a Voldemort - Innova4jInnova4j
 
Project voldemort - When relation database is not enough (too much?)
Project voldemort - When relation database is not enough (too much?)Project voldemort - When relation database is not enough (too much?)
Project voldemort - When relation database is not enough (too much?)nurkiewicz
 
Voldemort on Solid State Drives
Voldemort on Solid State DrivesVoldemort on Solid State Drives
Voldemort on Solid State DrivesVinoth Chandar
 
Bancos de dados open source
Bancos de dados open sourceBancos de dados open source
Bancos de dados open sourceRodrigo Aurélio
 
Composing and Executing Parallel Data Flow Graphs wth Shell Pipes
Composing and Executing Parallel Data Flow Graphs wth Shell PipesComposing and Executing Parallel Data Flow Graphs wth Shell Pipes
Composing and Executing Parallel Data Flow Graphs wth Shell PipesVinoth Chandar
 
Sha 2 기반 인증서 업그레이드 이해
Sha 2 기반 인증서 업그레이드 이해Sha 2 기반 인증서 업그레이드 이해
Sha 2 기반 인증서 업그레이드 이해InGuen Hwang
 
Cassandra - A Decentralized Structured Storage System
Cassandra - A Decentralized Structured Storage SystemCassandra - A Decentralized Structured Storage System
Cassandra - A Decentralized Structured Storage SystemVarad Meru
 
February 2016 Webinar Series - Introduction to DynamoDB
February 2016 Webinar Series - Introduction to DynamoDBFebruary 2016 Webinar Series - Introduction to DynamoDB
February 2016 Webinar Series - Introduction to DynamoDBAmazon Web Services
 
Distributed Hash Table and Consistent Hashing
Distributed Hash Table and Consistent HashingDistributed Hash Table and Consistent Hashing
Distributed Hash Table and Consistent HashingCloudFundoo
 
Dynamo db pros and cons
Dynamo db  pros and consDynamo db  pros and cons
Dynamo db pros and consSaniya Khalsa
 
Memcached의 확장성 개선
Memcached의 확장성 개선Memcached의 확장성 개선
Memcached의 확장성 개선NAVER D2
 
Runaway complexity in Big Data... and a plan to stop it
Runaway complexity in Big Data... and a plan to stop itRunaway complexity in Big Data... and a plan to stop it
Runaway complexity in Big Data... and a plan to stop itnathanmarz
 

Destaque (20)

Voldemort : Prototype to Production
Voldemort : Prototype to ProductionVoldemort : Prototype to Production
Voldemort : Prototype to Production
 
Project Voldemort
Project VoldemortProject Voldemort
Project Voldemort
 
Project Voldemort
Project VoldemortProject Voldemort
Project Voldemort
 
Project Voldemort: Big data loading
Project Voldemort: Big data loadingProject Voldemort: Big data loading
Project Voldemort: Big data loading
 
Introducción a Voldemort - Innova4j
Introducción a Voldemort - Innova4jIntroducción a Voldemort - Innova4j
Introducción a Voldemort - Innova4j
 
Project voldemort - When relation database is not enough (too much?)
Project voldemort - When relation database is not enough (too much?)Project voldemort - When relation database is not enough (too much?)
Project voldemort - When relation database is not enough (too much?)
 
Bluetube
BluetubeBluetube
Bluetube
 
Voldemort on Solid State Drives
Voldemort on Solid State DrivesVoldemort on Solid State Drives
Voldemort on Solid State Drives
 
Bancos de dados open source
Bancos de dados open sourceBancos de dados open source
Bancos de dados open source
 
Composing and Executing Parallel Data Flow Graphs wth Shell Pipes
Composing and Executing Parallel Data Flow Graphs wth Shell PipesComposing and Executing Parallel Data Flow Graphs wth Shell Pipes
Composing and Executing Parallel Data Flow Graphs wth Shell Pipes
 
Sha 2 기반 인증서 업그레이드 이해
Sha 2 기반 인증서 업그레이드 이해Sha 2 기반 인증서 업그레이드 이해
Sha 2 기반 인증서 업그레이드 이해
 
Cassandra - A Decentralized Structured Storage System
Cassandra - A Decentralized Structured Storage SystemCassandra - A Decentralized Structured Storage System
Cassandra - A Decentralized Structured Storage System
 
Cassandra
CassandraCassandra
Cassandra
 
February 2016 Webinar Series - Introduction to DynamoDB
February 2016 Webinar Series - Introduction to DynamoDBFebruary 2016 Webinar Series - Introduction to DynamoDB
February 2016 Webinar Series - Introduction to DynamoDB
 
Distributed Hash Table and Consistent Hashing
Distributed Hash Table and Consistent HashingDistributed Hash Table and Consistent Hashing
Distributed Hash Table and Consistent Hashing
 
Dynamo db pros and cons
Dynamo db  pros and consDynamo db  pros and cons
Dynamo db pros and cons
 
Memcached의 확장성 개선
Memcached의 확장성 개선Memcached의 확장성 개선
Memcached의 확장성 개선
 
Runaway complexity in Big Data... and a plan to stop it
Runaway complexity in Big Data... and a plan to stop itRunaway complexity in Big Data... and a plan to stop it
Runaway complexity in Big Data... and a plan to stop it
 
Bases de Datos No Relacionales (NoSQL): Cassandra, CouchDB, MongoDB y Neo4j
Bases de Datos No Relacionales (NoSQL): Cassandra, CouchDB, MongoDB y Neo4jBases de Datos No Relacionales (NoSQL): Cassandra, CouchDB, MongoDB y Neo4j
Bases de Datos No Relacionales (NoSQL): Cassandra, CouchDB, MongoDB y Neo4j
 
Base de Datos Orientada a Objetos
Base de Datos Orientada a ObjetosBase de Datos Orientada a Objetos
Base de Datos Orientada a Objetos
 

Semelhante a Voldemort

Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Emprovise
 
Big Data in the Cloud with Azure Marketplace Images
Big Data in the Cloud with Azure Marketplace ImagesBig Data in the Cloud with Azure Marketplace Images
Big Data in the Cloud with Azure Marketplace ImagesMark Kromer
 
What's New in Apache Hive 3.0?
What's New in Apache Hive 3.0?What's New in Apache Hive 3.0?
What's New in Apache Hive 3.0?DataWorks Summit
 
What's New in Apache Hive 3.0 - Tokyo
What's New in Apache Hive 3.0 - TokyoWhat's New in Apache Hive 3.0 - Tokyo
What's New in Apache Hive 3.0 - TokyoDataWorks Summit
 
Mysql wp memcached
Mysql wp memcachedMysql wp memcached
Mysql wp memcachedkbour23
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra
 
An overview of modern scalable web development
An overview of modern scalable web developmentAn overview of modern scalable web development
An overview of modern scalable web developmentTung Nguyen
 
Designing modern dw and data lake
Designing modern dw and data lakeDesigning modern dw and data lake
Designing modern dw and data lakepunedevscom
 
CouchBase The Complete NoSql Solution for Big Data
CouchBase The Complete NoSql Solution for Big DataCouchBase The Complete NoSql Solution for Big Data
CouchBase The Complete NoSql Solution for Big DataDebajani Mohanty
 
Which Database is Right for My Workload: Database Week SF
Which Database is Right for My Workload: Database Week SFWhich Database is Right for My Workload: Database Week SF
Which Database is Right for My Workload: Database Week SFAmazon Web Services
 
Which Database is Right for My Workload?: Database Week San Francisco
Which Database is Right for My Workload?: Database Week San FranciscoWhich Database is Right for My Workload?: Database Week San Francisco
Which Database is Right for My Workload?: Database Week San FranciscoAmazon Web Services
 
Which Database is Right for My Workload?
Which Database is Right for My Workload?Which Database is Right for My Workload?
Which Database is Right for My Workload?Amazon Web Services
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople
 
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...Amazon Web Services
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)James Serra
 
Maginatics @ SDC 2013: Architecting An Enterprise Storage Platform Using Obje...
Maginatics @ SDC 2013: Architecting An Enterprise Storage Platform Using Obje...Maginatics @ SDC 2013: Architecting An Enterprise Storage Platform Using Obje...
Maginatics @ SDC 2013: Architecting An Enterprise Storage Platform Using Obje...Maginatics
 
Unleash the Power of Redis with Amazon ElastiCache
Unleash the Power of Redis with Amazon ElastiCache Unleash the Power of Redis with Amazon ElastiCache
Unleash the Power of Redis with Amazon ElastiCache Amazon Web Services
 

Semelhante a Voldemort (20)

Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
 
Big Data in the Cloud with Azure Marketplace Images
Big Data in the Cloud with Azure Marketplace ImagesBig Data in the Cloud with Azure Marketplace Images
Big Data in the Cloud with Azure Marketplace Images
 
Mysql wp memcached
Mysql wp memcachedMysql wp memcached
Mysql wp memcached
 
What's New in Apache Hive 3.0?
What's New in Apache Hive 3.0?What's New in Apache Hive 3.0?
What's New in Apache Hive 3.0?
 
What's New in Apache Hive 3.0 - Tokyo
What's New in Apache Hive 3.0 - TokyoWhat's New in Apache Hive 3.0 - Tokyo
What's New in Apache Hive 3.0 - Tokyo
 
Data engineering
Data engineeringData engineering
Data engineering
 
Mysql wp memcached
Mysql wp memcachedMysql wp memcached
Mysql wp memcached
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
 
Nosql databases
Nosql databasesNosql databases
Nosql databases
 
An overview of modern scalable web development
An overview of modern scalable web developmentAn overview of modern scalable web development
An overview of modern scalable web development
 
Designing modern dw and data lake
Designing modern dw and data lakeDesigning modern dw and data lake
Designing modern dw and data lake
 
CouchBase The Complete NoSql Solution for Big Data
CouchBase The Complete NoSql Solution for Big DataCouchBase The Complete NoSql Solution for Big Data
CouchBase The Complete NoSql Solution for Big Data
 
Which Database is Right for My Workload: Database Week SF
Which Database is Right for My Workload: Database Week SFWhich Database is Right for My Workload: Database Week SF
Which Database is Right for My Workload: Database Week SF
 
Which Database is Right for My Workload?: Database Week San Francisco
Which Database is Right for My Workload?: Database Week San FranciscoWhich Database is Right for My Workload?: Database Week San Francisco
Which Database is Right for My Workload?: Database Week San Francisco
 
Which Database is Right for My Workload?
Which Database is Right for My Workload?Which Database is Right for My Workload?
Which Database is Right for My Workload?
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud Computing
 
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
Migrate from Oracle to Aurora PostgreSQL: Best Practices, Design Patterns, & ...
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
 
Maginatics @ SDC 2013: Architecting An Enterprise Storage Platform Using Obje...
Maginatics @ SDC 2013: Architecting An Enterprise Storage Platform Using Obje...Maginatics @ SDC 2013: Architecting An Enterprise Storage Platform Using Obje...
Maginatics @ SDC 2013: Architecting An Enterprise Storage Platform Using Obje...
 
Unleash the Power of Redis with Amazon ElastiCache
Unleash the Power of Redis with Amazon ElastiCache Unleash the Power of Redis with Amazon ElastiCache
Unleash the Power of Redis with Amazon ElastiCache
 

Último

ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 

Último (20)

ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 

Voldemort

  • 1. Project Voldemort A distributed database. Presented To: Sir Tariq Mehmood Presented By: Fasiha Ikram Aniqa Naeem
  • 2. Voldemort Voldemort is a distributed data store that is designed as a key-value store used by LinkedIn for high-scalability storage. It is named after the fictional Harry Potter villain Lord Voldemort.
  • 3. there are job titles, job openings for people, Groups and companies that offerings jobs. Big Data Variety velocity volum e Need frequently read write LinkedIn Big Data Problem
  • 4. Voldemort Scale Both • The amount of data we can store (write) • The number of requests for that data (read)
  • 5. Why Not Using Hadopp Naturally the only way to do this is to spread both the load and the data across many servers. 1. Need to find a way to split the data so that all servers have different data 2. Need to find a way to handle server failures without interrupting service 3. HBase still be write-heavy (due to horizontal partitioning and use of SSTables, which are write optimized)
  • 6. Why voldemort • Data is automatically replicated over multiple servers. • Data is automatically partitioned so each server contains only a subset of the total data • Provides tunable consistency (strict quorum or eventual consistency) • Server failure is handled transparently • Pluggable Storage Engines -- BDB-JE, MySQL, Read-Only
  • 7. Why voldemort • Pluggable serialization -- Protocol Buffers, Thrift, Avro and Java Serialization • Data items are versioned to maximize data integrity in failure scenarios without compromising availability of the system • Each node is independent of other nodes with no central point of failure or coordination
  • 8. Why voldemort • Good single node performance: you can expect 10- 20k operations per second depending on the machines, the network, the disk system, and the data replication factor • Support for pluggable data placement strategies to support things like distribution across data centers that are geographically far apart.
  • 9. Voldemort Storage Engines  Trivial to integrate new persistence mechanisms with Voldemort  2 Classes: Config(data) & Storage Engine(servers info)  3 Operations: put(k, v), get(k), delete(k)  Complication: k is Versioned<Key>
  • 13. Key-Value Storage • To enable high performance and availability it allow only very simple key-value data access. • Both keys and values can be complex compound objects including lists or maps, but none-the-less the only supported queries are effectively the following: value = store.get(key) store.put(key, value) store.delete(key)
  • 14. Query execution • Voldemort supports hashtable semantics, so a single value can be modified at a time and retrieval is by primary key. • This makes distribution across machines particularly easy since everything can be split by the primary key.
  • 15. Consistent Hashing Mechanism • In order to effectively Scaling , the data in Voldemort is split-up in such a way that each item is stored on multiple Servers. • For retrieving data first figure out which is the correct server to use. This partitioning is done via a consistent hashing mechanism that let’s any server calculate the location of data without doing any expensive look ups
  • 16. Detecting Failure • Voldemort set an SLA (service level agreement) for the requests and ban servers who cannot meet their SLA (this could be because they are down, because requests are timing out). • Servers that violate this SLA get banned for a short period of time, after which they attempt to restore them.
  • 17. Dealing With Failure  Since each value is stored in multiple places it is possible that one of these servers will not get updated (say because it is crashed when the update occurs).  To solve this problem Voldemort uses a data versioning mechanism called Vector Clocks.  This data versioning allows the servers to detect stale data when it is read and repair it.
  • 18. Comparison to Hbase databases Query language Architecture Database Model Replication Issues Voldemort API calls Big Unordered Map Key-value NoSQL Distributed data structure Topology Aware Routing Strategies Not Satisfyin g ACID Properti es. Hbase API calls REST XML Thrift Big Multi- dimensional Sorted Map HDFS Master- slave/Master -master replication Master Slave Which Is Not Highly Availabl e
  • 19. Acid Properties Voldemort Hbase Atomicity Yes Yes Consistency Yes Yes Isolation No Yes Durability Yes Yes Locking model Optimistic Locking MVCC Free for commercial use Yes Yes
  • 20. Industry Implementation  Linkdin founder of voldemort  shopping cart used by Gilt Groupe, which is powered by Voldemort.
  • 21. Use Case  High-Performance Key-Value Store (Amazon Dynamo clone)  treats the key‐value store as an API and adds an in‐memory caching layer, which means that you can plug into the back end that makes the most sense for your particular needs.
  • 22. Pros • only efficient queries are possible, very predictable performance. • easy to distribute across a cluster. • clean separation of storage and logic. • The storage layer is completely mockable so development and unit testing can be done against a throw-away in-memory storage system without needing a real cluster.
  • 23. Cons • no complex query filters • no foreign key constraints • no triggers • No built-in support for “multiple data center”- aware routing (there must be 1 copy of each key in at least one data center)
  • 24. Conclusion • It is basically just a big, distributed, persistent, fault-tolerant hash table. • The redundancy of storage makes the system more resilient to server failure. Since each value is stored N times, you can tolerate as many as N – 1 machine failures without data loss.
  • 25. Refrences 1. http://blog.linkedin.com/2009/04/01/project-voldemort-part-ii-how-it-works/ 2. http://blog.linkedin.com/2009/03/20/project-voldemort-scaling-simple-storage-at- linkedin/ 3. http://en.wikipedia.org/wiki/Voldemort_(distributed_data_store) 4. http://highscalability.com/product-project-voldemort-distributed-database 5. http://www.dummies.com/how-to/content/using-pluggable-storage-with-nosql.html 6. http://vschart.com/compare/project-voldemort/vs/hbase 7. http://www.project-voldemort.com/voldemort/design.html