SlideShare a Scribd company logo

GFS

•Download as PPT, PDF•
•
Suman Karumuri
Suman Karumuri
Technology
1 of 70
Google File System ,[object Object],[object Object],[object Object]
GFS ,[object Object],[object Object],[object Object],[object Object]
Design Assumptions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
API ,[object Object],[object Object],[object Object],[object Object],[object Object]
Architecture
Architecture GFS Master GFS Client Application GFS Chunk Server GFS Chunk Server …
Files and Chunks ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GFS Chunk Servers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GFS Master ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Create Operation
Create GFS Master Create /home/user/filename GFS Client Application GFS Chunk Server GFS Chunk Server …
Create ,[object Object],[object Object],[object Object],rack 2 rack 1 Create /home/user/filename GFS Client Application GFS Chunk Server GFS Chunk Server …
Create ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],rack 2 rack 1 Create /home/user/filename GFS Client Application GFS Chunk Server GFS Chunk Server …
Create ,[object Object],[object Object],[object Object],[object Object],rack 2 rack 1 Returns chunk handle, Chunk locations GFS Client Application GFS Chunk Server GFS Chunk Server …
Namespaces ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Locking example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Locking ,[object Object],[object Object],[object Object],[object Object]
Read Operation
Read Operation GFS Master filename and chunk index GFS Client Application GFS Chunk Server GFS Chunk Server …
Read Operation GFS Master chunk handle, server locations GFS Client Application GFS Chunk Server GFS Chunk Server …
Read Operation GFS Master Chunk handle, bit range GFS Client Application GFS Chunk Server GFS Chunk Server …
Read Operation GFS Master Data GFS Client Application GFS Chunk Server GFS Chunk Server …
Write ,[object Object],[object Object],[object Object]
Write Operation
Write GFS Master Chunk id, chunk offset GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
Write GFS Master Chunkserver locations (caches this) GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
Write GFS Master GFS Chunk Server … data Pass along data to nearest replica GFS Client Application GFS Chunk Server GFS Chunk Server
Write GFS Master Serializes all concurrent writes GFS Chunk Server operation GFS Client Application GFS Chunk Server GFS Chunk Server …
Write GFS Master GFS Chunk Server serialized order of writes GFS Client Application GFS Chunk Server GFS Chunk Server …
Write GFS Master GFS Chunk Server ack ack ack GFS Client Application GFS Chunk Server GFS Chunk Server …
Write GFS Master GFS Chunk Server ack, chunk index GFS Client Application GFS Chunk Server GFS Chunk Server …
Write under failure GFS Master GFS Chunk Server ack ack GFS Client Application GFS Chunk Server GFS Chunk Server …
Write under failure GFS Master GFS Chunk Server retry GFS Client Application GFS Chunk Server GFS Chunk Server …
Leases ,[object Object],[object Object]
Write with primary GFS Master Chunk id chunk offset GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
Write with primary GFS Master Chunkserver locations (caches this) GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
Write with primary GFS Master GFS Chunk Server … data Pass along data to nearest replica GFS Client Application GFS Chunk Server GFS Chunk Server
Write with primary GFS Master Serializes all concurrent writes GFS Chunk Server operation GFS Client Application GFS Chunk Server GFS Chunk Server …
Write with primary GFS Master GFS Chunk Server serialized operations GFS Client Application GFS Chunk Server GFS Chunk Server …
Write with primary GFS Master GFS Chunk Server ack ack GFS Client Application GFS Chunk Server GFS Chunk Server …
Write with primary GFS Master GFS Chunk Server Ack, chunk index GFS Client Application GFS Chunk Server GFS Chunk Server …
Failures during writes ,[object Object],[object Object],[object Object]
Write with primary ,[object Object]
Record Append
Record append GFS Master Chunk id GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
Record append GFS Master GFS Chunk Server Ack, chunk index from end of file GFS Client Application GFS Chunk Server GFS Chunk Server …
Record Append Operation
Record append ,[object Object],[object Object],[object Object],[object Object]
Consistency Model
Write -Single process Chunk 1 9:Hello Chunk 1’ 9:Hello
Write – Single process Chunk 1 9:Hello 10: World Chunk 1’ 9:Hello Write(“World”, 10 ) Inconsistent State
Same with any failed mutation Chunk 1 9:Hello 10: World Chunk 1’ 9:Hello Write(“World”, 10 ) Inconsistent State
Multiple Writers Chunk 1 9:Hello 10:Wor12345 Chunk 1’ 9:Hello 10:Wor12345 Write(“World”,10:0) Write(“12345”,10:3) Consistent and Undefined
Append Chunk 1 9:Hello 10:World Chunk 1’ 9:Hello Append(“World”) Inconsistent and Undefined retry
Append Chunk 1 9:Hello 10:World 11:World Chunk 1’ 9:Hello 11:World Append(“World”) Defined interspersed with inconsistent 11 11
Same for append with multiple writers Chunk 1 9:Hello 10:World 11:World Chunk 1’ 9:Hello 11:World Append(“World”) Defined interspersed with inconsistent 11 11
Consistency model ,[object Object],[object Object],[object Object],[object Object],[object Object]
Snapshot ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Delete Operation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Delete API ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Fault Tolerance for chunks ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Fault tolerance for master ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Fault tolerance for Chunk Server ,[object Object],[object Object],[object Object]
High Availability ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Performance metrics.
Conclusions
Conclusions ,[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusions … ,[object Object],[object Object],[object Object],[object Object],[object Object]
Thank you?
Design goals ,[object Object],[object Object],[object Object],[object Object]

Recommended

The Google File System (GFS)
The Google File System (GFS)The Google File System (GFS)
The Google File System (GFS)Romain Jacotin
 
Distributed Shared Memory Systems
Distributed Shared Memory SystemsDistributed Shared Memory Systems
Distributed Shared Memory SystemsAnkit Gupta
 
GOOGLE FILE SYSTEM
GOOGLE FILE SYSTEMGOOGLE FILE SYSTEM
GOOGLE FILE SYSTEMJYoTHiSH o.s
 
CockroachDB
CockroachDBCockroachDB
CockroachDBandrei moga
 
Google File Systems
Google File SystemsGoogle File Systems
Google File SystemsAzeem Mumtaz
 
Introduction to distributed file systems
Introduction to distributed file systemsIntroduction to distributed file systems
Introduction to distributed file systemsViet-Trung TRAN
 
Consistency protocols
Consistency protocolsConsistency protocols
Consistency protocolsZongYing Lyu
 
Google File System
Google File SystemGoogle File System
Google File SystemAmir Payberah
 

Recommended

The Google File System (GFS)
The Google File System (GFS)The Google File System (GFS)
The Google File System (GFS)Romain Jacotin
 
Distributed Shared Memory Systems
Distributed Shared Memory SystemsDistributed Shared Memory Systems
Distributed Shared Memory SystemsAnkit Gupta
 
GOOGLE FILE SYSTEM
GOOGLE FILE SYSTEMGOOGLE FILE SYSTEM
GOOGLE FILE SYSTEMJYoTHiSH o.s
 
CockroachDB
CockroachDBCockroachDB
CockroachDBandrei moga
 
Google File Systems
Google File SystemsGoogle File Systems
Google File SystemsAzeem Mumtaz
 
Introduction to distributed file systems
Introduction to distributed file systemsIntroduction to distributed file systems
Introduction to distributed file systemsViet-Trung TRAN
 
Consistency protocols
Consistency protocolsConsistency protocols
Consistency protocolsZongYing Lyu
 
Google File System
Google File SystemGoogle File System
Google File SystemAmir Payberah
 
Google File System
Google File SystemGoogle File System
Google File Systemguest2cb4689
 
MapReduce: Simplified Data Processing on Large Clusters
MapReduce: Simplified Data Processing on Large ClustersMapReduce: Simplified Data Processing on Large Clusters
MapReduce: Simplified Data Processing on Large ClustersAshraf Uddin
 
Course 102: Lecture 27: FileSystems in Linux (Part 2)
Course 102: Lecture 27: FileSystems in Linux (Part 2)Course 102: Lecture 27: FileSystems in Linux (Part 2)
Course 102: Lecture 27: FileSystems in Linux (Part 2)Ahmed El-Arabawy
 
Google File System
Google File SystemGoogle File System
Google File SystemJunyoung Jung
 
5 Steps to PostgreSQL Performance
5 Steps to PostgreSQL Performance5 Steps to PostgreSQL Performance
5 Steps to PostgreSQL PerformanceCommand Prompt., Inc
 
GOOGLE BIGTABLE
GOOGLE BIGTABLEGOOGLE BIGTABLE
GOOGLE BIGTABLETomcy Thankachan
 
Google File System - GFS Presentation Slides PPT
Google File System - GFS Presentation Slides PPTGoogle File System - GFS Presentation Slides PPT
Google File System - GFS Presentation Slides PPTDiwasPandey3
 
Presentation on Shared Memory Parallel Programming
Presentation on Shared Memory Parallel ProgrammingPresentation on Shared Memory Parallel Programming
Presentation on Shared Memory Parallel ProgrammingVengada Karthik Rangaraju
 
Distributed Transactions(flat and nested) and Atomic Commit Protocols
Distributed Transactions(flat and nested) and Atomic Commit ProtocolsDistributed Transactions(flat and nested) and Atomic Commit Protocols
Distributed Transactions(flat and nested) and Atomic Commit ProtocolsSachin Chauhan
 
DISTRIBUTED DATABASE WITH RECOVERY TECHNIQUES
DISTRIBUTED DATABASE WITH RECOVERY TECHNIQUESDISTRIBUTED DATABASE WITH RECOVERY TECHNIQUES
DISTRIBUTED DATABASE WITH RECOVERY TECHNIQUESAAKANKSHA JAIN
 
Replication in Distributed Systems
Replication in Distributed SystemsReplication in Distributed Systems
Replication in Distributed SystemsKavya Barnadhya Hazarika
 
Google file system GFS
Google file system GFSGoogle file system GFS
Google file system GFSzihad164
 
ClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, CloudflareClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, CloudflareAltinity Ltd
 
management of distributed transactions
management of distributed transactionsmanagement of distributed transactions
management of distributed transactionsNilu Desai
 
File replication
File replicationFile replication
File replicationKlawal13
 
Distributed Shared Memory Systems
Distributed Shared Memory SystemsDistributed Shared Memory Systems
Distributed Shared Memory SystemsArush Nagpal
 
Distributed Shared Memory
Distributed Shared MemoryDistributed Shared Memory
Distributed Shared MemoryPrakhar Rastogi
 
Seminar Report on Google File System
Seminar Report on Google File SystemSeminar Report on Google File System
Seminar Report on Google File SystemVishal Polley
 
RDD
RDDRDD
RDDTien-Yang (Aiden) Wu
 
Multiprocessor
MultiprocessorMultiprocessor
MultiprocessorA B Shinde
 
Google file system
Google file systemGoogle file system
Google file systemAnurag Gautam
 
Gfs
GfsGfs
Gfsravi kiran
 

More Related Content

What's hot

Google File System
Google File SystemGoogle File System
Google File Systemguest2cb4689
 
MapReduce: Simplified Data Processing on Large Clusters
MapReduce: Simplified Data Processing on Large ClustersMapReduce: Simplified Data Processing on Large Clusters
MapReduce: Simplified Data Processing on Large ClustersAshraf Uddin
 
Course 102: Lecture 27: FileSystems in Linux (Part 2)
Course 102: Lecture 27: FileSystems in Linux (Part 2)Course 102: Lecture 27: FileSystems in Linux (Part 2)
Course 102: Lecture 27: FileSystems in Linux (Part 2)Ahmed El-Arabawy
 
Google File System - GFS Presentation Slides PPT
Google File System - GFS Presentation Slides PPTGoogle File System - GFS Presentation Slides PPT
Google File System - GFS Presentation Slides PPTDiwasPandey3
 
Presentation on Shared Memory Parallel Programming
Presentation on Shared Memory Parallel ProgrammingPresentation on Shared Memory Parallel Programming
Presentation on Shared Memory Parallel ProgrammingVengada Karthik Rangaraju
 
Distributed Transactions(flat and nested) and Atomic Commit Protocols
Distributed Transactions(flat and nested) and Atomic Commit ProtocolsDistributed Transactions(flat and nested) and Atomic Commit Protocols
Distributed Transactions(flat and nested) and Atomic Commit ProtocolsSachin Chauhan
 
DISTRIBUTED DATABASE WITH RECOVERY TECHNIQUES
DISTRIBUTED DATABASE WITH RECOVERY TECHNIQUESDISTRIBUTED DATABASE WITH RECOVERY TECHNIQUES
DISTRIBUTED DATABASE WITH RECOVERY TECHNIQUESAAKANKSHA JAIN
 
Google file system GFS
Google file system GFSGoogle file system GFS
Google file system GFSzihad164
 
ClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, CloudflareClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, CloudflareAltinity Ltd
 
management of distributed transactions
management of distributed transactionsmanagement of distributed transactions
management of distributed transactionsNilu Desai
 
File replication
File replicationFile replication
File replicationKlawal13
 
Distributed Shared Memory Systems
Distributed Shared Memory SystemsDistributed Shared Memory Systems
Distributed Shared Memory SystemsArush Nagpal
 
Distributed Shared Memory
Distributed Shared MemoryDistributed Shared Memory
Distributed Shared MemoryPrakhar Rastogi
 
Seminar Report on Google File System
Seminar Report on Google File SystemSeminar Report on Google File System
Seminar Report on Google File SystemVishal Polley
 
Multiprocessor
MultiprocessorMultiprocessor
MultiprocessorA B Shinde
 

What's hot (20)

Google File System
Google File SystemGoogle File System
Google File System
 
MapReduce: Simplified Data Processing on Large Clusters
MapReduce: Simplified Data Processing on Large ClustersMapReduce: Simplified Data Processing on Large Clusters
MapReduce: Simplified Data Processing on Large Clusters
 
Course 102: Lecture 27: FileSystems in Linux (Part 2)
Course 102: Lecture 27: FileSystems in Linux (Part 2)Course 102: Lecture 27: FileSystems in Linux (Part 2)
Course 102: Lecture 27: FileSystems in Linux (Part 2)
 
Google File System
Google File SystemGoogle File System
Google File System
 
5 Steps to PostgreSQL Performance
5 Steps to PostgreSQL Performance5 Steps to PostgreSQL Performance
5 Steps to PostgreSQL Performance
 
GOOGLE BIGTABLE
GOOGLE BIGTABLEGOOGLE BIGTABLE
GOOGLE BIGTABLE
 
Google File System - GFS Presentation Slides PPT
Google File System - GFS Presentation Slides PPTGoogle File System - GFS Presentation Slides PPT
Google File System - GFS Presentation Slides PPT
 
Presentation on Shared Memory Parallel Programming
Presentation on Shared Memory Parallel ProgrammingPresentation on Shared Memory Parallel Programming
Presentation on Shared Memory Parallel Programming
 
Distributed Transactions(flat and nested) and Atomic Commit Protocols
Distributed Transactions(flat and nested) and Atomic Commit ProtocolsDistributed Transactions(flat and nested) and Atomic Commit Protocols
Distributed Transactions(flat and nested) and Atomic Commit Protocols
 
DISTRIBUTED DATABASE WITH RECOVERY TECHNIQUES
DISTRIBUTED DATABASE WITH RECOVERY TECHNIQUESDISTRIBUTED DATABASE WITH RECOVERY TECHNIQUES
DISTRIBUTED DATABASE WITH RECOVERY TECHNIQUES
 
Replication in Distributed Systems
Replication in Distributed SystemsReplication in Distributed Systems
Replication in Distributed Systems
 
Google file system GFS
Google file system GFSGoogle file system GFS
Google file system GFS
 
ClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, CloudflareClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
ClickHouse Mark Cache, by Mik Kocikowski, Cloudflare
 
management of distributed transactions
management of distributed transactionsmanagement of distributed transactions
management of distributed transactions
 
File replication
File replicationFile replication
File replication
 
Distributed Shared Memory Systems
Distributed Shared Memory SystemsDistributed Shared Memory Systems
Distributed Shared Memory Systems
 
Distributed Shared Memory
Distributed Shared MemoryDistributed Shared Memory
Distributed Shared Memory
 
Seminar Report on Google File System
Seminar Report on Google File SystemSeminar Report on Google File System
Seminar Report on Google File System
 
RDD
RDDRDD
RDD
 
Multiprocessor
MultiprocessorMultiprocessor
Multiprocessor
 

Similar to GFS

GFS - Google File System
GFS - Google File SystemGFS - Google File System
GFS - Google File Systemtutchiio
 
Google File System
Google File SystemGoogle File System
Google File SystemDreamJobs1
 
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...Виталий Стародубцев
 
advanced Google file System
advanced Google file Systemadvanced Google file System
advanced Google file Systemdiptipan
 
Advance google file system
Advance google file systemAdvance google file system
Advance google file systemLalit Rastogi
 
Learning spark ch10 - Spark Streaming
Learning spark ch10 - Spark StreamingLearning spark ch10 - Spark Streaming
Learning spark ch10 - Spark Streamingphanleson
 
SRV407 Deep Dive on Amazon Aurora
SRV407 Deep Dive on Amazon AuroraSRV407 Deep Dive on Amazon Aurora
SRV407 Deep Dive on Amazon AuroraAmazon Web Services
 
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...Flink Forward
 
Netflix Keystone Pipeline at Samza Meetup 10-13-2015
Netflix Keystone Pipeline at Samza Meetup 10-13-2015Netflix Keystone Pipeline at Samza Meetup 10-13-2015
Netflix Keystone Pipeline at Samza Meetup 10-13-2015Monal Daxini
 
Dsm (Distributed computing)
Dsm (Distributed computing)Dsm (Distributed computing)
Dsm (Distributed computing)Sri Prasanna
 
Apache Hadoop India Summit 2011 talk "Hadoop Map-Reduce Programming & Best Pr...
Apache Hadoop India Summit 2011 talk "Hadoop Map-Reduce Programming & Best Pr...Apache Hadoop India Summit 2011 talk "Hadoop Map-Reduce Programming & Best Pr...
Apache Hadoop India Summit 2011 talk "Hadoop Map-Reduce Programming & Best Pr...Yahoo Developer Network
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2aspyker
 
vSphere vStorage: Troubleshooting Performance
vSphere vStorage: Troubleshooting PerformancevSphere vStorage: Troubleshooting Performance
vSphere vStorage: Troubleshooting PerformanceProfessionalVMware
 
Cassandra from tarball to production
Cassandra from tarball to productionCassandra from tarball to production
Cassandra from tarball to productionRon Kuris
 

Similar to GFS (20)

Google file system
Google file systemGoogle file system
Google file system
 
Gfs
GfsGfs
Gfs
 
Demo 0.9.4
Demo 0.9.4Demo 0.9.4
Demo 0.9.4
 
GFS - Google File System
GFS - Google File SystemGFS - Google File System
GFS - Google File System
 
Google File System
Google File SystemGoogle File System
Google File System
 
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
 
advanced Google file System
advanced Google file Systemadvanced Google file System
advanced Google file System
 
Gfs介绍
Gfs介绍Gfs介绍
Gfs介绍
 
Advance google file system
Advance google file systemAdvance google file system
Advance google file system
 
Learning spark ch10 - Spark Streaming
Learning spark ch10 - Spark StreamingLearning spark ch10 - Spark Streaming
Learning spark ch10 - Spark Streaming
 
SRV407 Deep Dive on Amazon Aurora
SRV407 Deep Dive on Amazon AuroraSRV407 Deep Dive on Amazon Aurora
SRV407 Deep Dive on Amazon Aurora
 
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...
 
Netflix Keystone Pipeline at Samza Meetup 10-13-2015
Netflix Keystone Pipeline at Samza Meetup 10-13-2015Netflix Keystone Pipeline at Samza Meetup 10-13-2015
Netflix Keystone Pipeline at Samza Meetup 10-13-2015
 
Dsm (Distributed computing)
Dsm (Distributed computing)Dsm (Distributed computing)
Dsm (Distributed computing)
 
Google file system
Google file systemGoogle file system
Google file system
 
Apache Hadoop India Summit 2011 talk "Hadoop Map-Reduce Programming & Best Pr...
Apache Hadoop India Summit 2011 talk "Hadoop Map-Reduce Programming & Best Pr...Apache Hadoop India Summit 2011 talk "Hadoop Map-Reduce Programming & Best Pr...
Apache Hadoop India Summit 2011 talk "Hadoop Map-Reduce Programming & Best Pr...
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2
 
vSphere vStorage: Troubleshooting Performance
vSphere vStorage: Troubleshooting PerformancevSphere vStorage: Troubleshooting Performance
vSphere vStorage: Troubleshooting Performance
 
Cassandra from tarball to production
Cassandra from tarball to productionCassandra from tarball to production
Cassandra from tarball to production
 
Measuring Firebird Disk I/O
Measuring Firebird Disk I/OMeasuring Firebird Disk I/O
Measuring Firebird Disk I/O
 

More from Suman Karumuri

Pintrace: Distributed tracing @Pinterest
Pintrace: Distributed tracing @PinterestPintrace: Distributed tracing @Pinterest
Pintrace: Distributed tracing @PinterestSuman Karumuri
 
Pintrace: Distributed tracing@Pinterest
Pintrace: Distributed tracing@PinterestPintrace: Distributed tracing@Pinterest
Pintrace: Distributed tracing@PinterestSuman Karumuri
 
PinTrace Advanced AWS meetup
PinTrace Advanced AWS meetup PinTrace Advanced AWS meetup
PinTrace Advanced AWS meetup Suman Karumuri
 
Practical Byzantine Fault Tolerance
Practical Byzantine Fault TolerancePractical Byzantine Fault Tolerance
Practical Byzantine Fault ToleranceSuman Karumuri
 

More from Suman Karumuri (10)

Monorepo at Pinterest
Monorepo at PinterestMonorepo at Pinterest
Monorepo at Pinterest
 
Pintrace: Distributed tracing @Pinterest
Pintrace: Distributed tracing @PinterestPintrace: Distributed tracing @Pinterest
Pintrace: Distributed tracing @Pinterest
 
Pintrace: Distributed tracing@Pinterest
Pintrace: Distributed tracing@PinterestPintrace: Distributed tracing@Pinterest
Pintrace: Distributed tracing@Pinterest
 
PinTrace Advanced AWS meetup
PinTrace Advanced AWS meetup PinTrace Advanced AWS meetup
PinTrace Advanced AWS meetup
 
Phobos
PhobosPhobos
Phobos
 
Gpu Join Presentation
Gpu Join PresentationGpu Join Presentation
Gpu Join Presentation
 
Dream Language!
Dream Language!Dream Language!
Dream Language!
 
Bittorrent
BittorrentBittorrent
Bittorrent
 
Practical Byzantine Fault Tolerance
Practical Byzantine Fault TolerancePractical Byzantine Fault Tolerance
Practical Byzantine Fault Tolerance
 
bluespec talk
bluespec talkbluespec talk
bluespec talk
 

Recently uploaded

unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxBkGupta21
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
Training state-of-the-art general text embedding
Training state-of-the-art general text embeddingTraining state-of-the-art general text embedding
Training state-of-the-art general text embeddingZilliz
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 

Recently uploaded (20)

unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptx
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
Training state-of-the-art general text embedding
Training state-of-the-art general text embeddingTraining state-of-the-art general text embedding
Training state-of-the-art general text embedding
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 

GFS

  • 1.
  • 2.
  • 3.
  • 4.
  • 6. Architecture GFS Master GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 7.
  • 8.
  • 9.
  • 11. Create GFS Master Create /home/user/filename GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 19. Read Operation GFS Master filename and chunk index GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 20. Read Operation GFS Master chunk handle, server locations GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 21. Read Operation GFS Master Chunk handle, bit range GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 22. Read Operation GFS Master Data GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 23.
  • 25. Write GFS Master Chunk id, chunk offset GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 26. Write GFS Master Chunkserver locations (caches this) GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 27. Write GFS Master GFS Chunk Server … data Pass along data to nearest replica GFS Client Application GFS Chunk Server GFS Chunk Server
  • 28. Write GFS Master Serializes all concurrent writes GFS Chunk Server operation GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 29. Write GFS Master GFS Chunk Server serialized order of writes GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 30. Write GFS Master GFS Chunk Server ack ack ack GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 31. Write GFS Master GFS Chunk Server ack, chunk index GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 32. Write under failure GFS Master GFS Chunk Server ack ack GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 33. Write under failure GFS Master GFS Chunk Server retry GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 34.
  • 35. Write with primary GFS Master Chunk id chunk offset GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 36. Write with primary GFS Master Chunkserver locations (caches this) GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 37. Write with primary GFS Master GFS Chunk Server … data Pass along data to nearest replica GFS Client Application GFS Chunk Server GFS Chunk Server
  • 38. Write with primary GFS Master Serializes all concurrent writes GFS Chunk Server operation GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 39. Write with primary GFS Master GFS Chunk Server serialized operations GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 40. Write with primary GFS Master GFS Chunk Server ack ack GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 41. Write with primary GFS Master GFS Chunk Server Ack, chunk index GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 42.
  • 43.
  • 45. Record append GFS Master Chunk id GFS Chunk Server GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 46. Record append GFS Master GFS Chunk Server Ack, chunk index from end of file GFS Client Application GFS Chunk Server GFS Chunk Server …
  • 48.
  • 50. Write -Single process Chunk 1 9:Hello Chunk 1’ 9:Hello
  • 51. Write – Single process Chunk 1 9:Hello 10: World Chunk 1’ 9:Hello Write(“World”, 10 ) Inconsistent State
  • 52. Same with any failed mutation Chunk 1 9:Hello 10: World Chunk 1’ 9:Hello Write(“World”, 10 ) Inconsistent State
  • 53. Multiple Writers Chunk 1 9:Hello 10:Wor12345 Chunk 1’ 9:Hello 10:Wor12345 Write(“World”,10:0) Write(“12345”,10:3) Consistent and Undefined
  • 54. Append Chunk 1 9:Hello 10:World Chunk 1’ 9:Hello Append(“World”) Inconsistent and Undefined retry
  • 55. Append Chunk 1 9:Hello 10:World 11:World Chunk 1’ 9:Hello 11:World Append(“World”) Defined interspersed with inconsistent 11 11
  • 56. Same for append with multiple writers Chunk 1 9:Hello 10:World 11:World Chunk 1’ 9:Hello 11:World Append(“World”) Defined interspersed with inconsistent 11 11
  • 57.
  • 58.
  • 59.
  • 60.
  • 61.
  • 62.
  • 63.
  • 64.
  • 67.
  • 68.
  • 70.