VaporStore – the design of a real-world cloud filesystem

Outline
Introduction
Implementation
Architecture
Future improvements

VaporStore the design of a real-world cloud
ﬁlesystem

Igor Bogicevic (igor.bogicevic@sbgenomics.com)

July 3, 2010

Igor Bogicevic (igor.bogicevic@sbgenomics.com) VaporStore the design of a real-world cloud ﬁlesystem

Outline
Introduction
Implementation
Architecture
Future improvements

Introduction
So what do we need?
And which options do we have?
Implementation
Challenges of implementing a cloud ﬁlesystem
Requirements revisited
Big Picture
Terminology
Architecture
Major components
Lifecycle of a ﬁle
”Lazy” updates
Future improvements
Stability and Security
Features


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements

So what do we need?

A ﬁlesystem designed to be a persistence layer for a various
Bioinformatic software that will be running in the cloud.


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements

So what do we need?

Has to support a ”Big Data” (easily 100MB-100GB per-ﬁle).


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements

So what do we need?

Has to support a partial file uploads and possibility to continue file
upload from the last uploaded fragment for large files.


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements

So what do we need?

Has to support a concurrent clients and to act as a distributed
ﬁlesystem.


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements

So what do we need?

ﬁlesystem.
Simple but complete API to mimic a ﬁlesystem interface for easy
integration with the applications.


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements

So what do we need?

filesystem.
Simple but complete API to mimic a filesystem interface for easy
integration with the applications.
Needs to have unix-like, or even finer grained access control.


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements


Plain S3 Objects - lacks support for objects bigger than 5GB and it
doesn’t have a ﬁne-grained access control.


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements


HDFS on top of S3 - probably the best match, however it doesn’t
support security requirements as well as partial ﬁle uploads.


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements


Various FUSE-to-S3 interfaces - either they don’t address the
ﬁle-size, ACL or support for multiple clients (or are just toy-FS
implementations).


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements


implementations).
Something else - we’d love to know about any project that meets
these requirements!


Outline
Introduction
So what do we need?
Implementation
Architecture
Future improvements


implementations).
Something else - we’d love to know about any project that meets
these requirements!
DIY.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements


”Cloud filesystem” means nothing more than that content and
metadata are not local and we don’t have a strict control over
location of the data. Also, content of the files is not local to the
filesystem metadata - i.e. filesystem structure, file ownership, etc.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements


Latencies for remote read/write operations are few orders of a
magnitude bigger than the latencies for operations on a local
filesystem, yet we need to ensure decent performance especially in
respect to metadata - filesystem exploration, modifications etc. need
to happen in close-to-local speed.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements


Latencies for remote read/write operations are few orders of a
magnitude bigger than the latencies for operations on a local
filesystem, yet we need to ensure decent performance especially in
respect to metadata - filesystem exploration, modifications etc. need
to happen in close-to-local speed.
Working with the large files implies that we’ll need a mechanism to
support a partial content uploads to handle the failures (connection
drops, link outages, master server downtimes etc.).


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements


Must support up to a few million ﬁles.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements


Does not have to be fast, however it needs to be reliable.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements


Must implement extensible ACL interface.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements


Must support a partial ﬁle uploads and downloads.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements


Must support a maintenance downtimes.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements


Must support a maintenance downtimes.
Has to implement ”lazy” metadata persistence.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements

Big Picture
Vaporstore
Server

WEB FS
Client
TCP
HTTP REST Client
Command
Interface Uploader/Sync
Interface
VS Shell

Session Manager Credentials
Storage

File System
Manager

Block/Object
Storage (S3)

.

Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements

Terminology

Filesystem - tree-based hierarchical structure containing ﬁles in
many aspects similar to a common UNIX ﬁlesystem.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements

Terminology

File/Node - individual element of filesystem that represent the
atomic structure exposed to the end users, which means that users
of VS can add, delete, manipulate the files. File itself contains the
metadata describing various properties of the file (i.e. ownership,
state, type, etc.) and block information and status.


Outline
Introduction
Implementation
Big Picture
Architecture
Terminology
Future improvements

Terminology

File/Node - individual element of filesystem that represent the
atomic structure exposed to the end users, which means that users
of VS can add, delete, manipulate the files. File itself contains the
metadata describing various properties of the file (i.e. ownership,
state, type, etc.) and block information and status.
Block - represents the content of a file itself. Each file can contain
many blocks scattered across the storage (i.e. S3). This construct is
in a many aspects similar to the concept of a block in HDFS.


Outline
Introduction Major components
Implementation Lifecycle of a file
Architecture ”Lazy” updates
Future improvements

Major components
HTTP RESTful Interface - all operations on a filesystem structure
are being performed through the RESTful interface and responses
are wrapped in AVRO/JSON serialized messages. You can put, get,
remove, update the files, or request information about it’s metadata.


Outline
Future improvements

Major components
TCP Command Interface - persistent connection used for a
notiﬁcations and requests between server/client represented by
simple AVRO/JSON message passing protocol.


Outline
Future improvements

Major components
Session Manager - component used for authenticating the client
sessions. Currently, one client can open only one session which
might change in a future.


Outline
Future improvements

Major components
Session Manager - component used for authenticating the client
sessions. Currently, one client can open only one session which
might change in a future.
FileSystem Manager - component which performs ﬁlesystem updates
on a persistent ﬁlesystem storage. Acts as a queue with a thread
pool of a consumers which perform operations on a remote storage
(i.e. S3).


Outline
Future improvements


When we create the ﬁle through the HTTP RESTful interface, we’re
creating a new node in ﬁlesystem structure containing it’s path,
name, metadata and the list of the blocks.


Outline
Future improvements


If operation is successful, update is queued through the Filesystem
manager and client starts uploading the block to the (S3) storage.
Once the block is uploaded it sends the updates to the filesystem
that a file’s block had been uploaded. After the file/node receives
the update it schedules the update of it’s state to the persistent
storage through the Filesystem Manager.


Outline
Future improvements


If operation is successful, update is queued through the Filesystem
manager and client starts uploading the block to the (S3) storage.
Once the block is uploaded it sends the updates to the filesystem
that a file’s block had been uploaded. After the file/node receives
the update it schedules the update of it’s state to the persistent
storage through the Filesystem Manager.
After all blocks have been uploaded, we consider that this is now a
valid file that can be accessed, and we schedule another update
through the Filesystem manager to set it’s final state.


Outline
Future improvements

”Lazy” updates
All updates to a persistent storage are being done ”lazy” - this
means that each update is ﬁrst being done locally in memory and
then scheduled for a transaction on a persistent store.


Outline
Future improvements

”Lazy” updates
This is both a performance optimization and a failure handling
mechanism.


Outline
Future improvements

”Lazy” updates
mechanism.
Cost of this mechanism is that we will have to re-initialize the
certain parts of the file upload that we’ve already done or that we
might have a ”dirty” blocks that we need to clean up if the
filesystem fails - i.e. if we don’t update file’s information that certain
blocks have been uploaded, we will need to upload them again to a
storage, or if we delete a file but we don’t remove the blocks from a
storage, we will have a ”dirty” blocks.


Outline
Future improvements

”Lazy” updates
mechanism.
Cost of this mechanism is that we will have to re-initialize the
certain parts of the file upload that we’ve already done or that we
might have a ”dirty” blocks that we need to clean up if the
filesystem fails - i.e. if we don’t update file’s information that certain
blocks have been uploaded, we will need to upload them again to a
storage, or if we delete a file but we don’t remove the blocks from a
storage, we will have a ”dirty” blocks.
Yes, we can do this smarter in order to avoid this problems - in a
future releases.


Outline
Introduction
Implementation
Features
Architecture
Future improvements


Stability - it is fully featured except the ACLs, however it’s not well
tested and not production-ready.


Outline
Introduction
Implementation
Features
Architecture
Future improvements


Security - currently a client needs to get S3 credential from a server
after it’s authenticated, however if we put the ”content proxy” in
between client and S3 we can outsorce direct S3 access to a proxy.


Outline
Introduction
Implementation
Features
Architecture
Future improvements


Partitioning and HA - support for multiple servers handling diﬀerent
branches of a ﬁlesystem tree HA for each partition


Outline
Introduction
Implementation
Features
Architecture
Future improvements


Partitioning and HA - support for multiple servers handling diﬀerent
branches of a ﬁlesystem tree HA for each partition
Bursting support - dropping the requests if we can’t handle the load.


Outline
Introduction
Implementation
Features
Architecture
Future improvements

Features

Various storage backends - it’s currently using S3, however it could
use pretty much anything... anything that makes sense.


Outline
Introduction
Implementation
Features
Architecture
Future improvements

Features

Smarter ”lazy” updates .


Outline
Introduction
Implementation
Features
Architecture
Future improvements

Features

Torrent support for a ”content proxy” - just an idea, but might
make a sense.


Outline
Introduction
Implementation
Features
Architecture
Future improvements

Features

make a sense.
Support for AWS import/export.


Outline
Introduction
Implementation
Features
Architecture
Future improvements

Features

make a sense.
Richer client support - besides API and shell, we’ll add a client
service that will perform dropbox-like synchronization.


Outline
Introduction
Implementation
Features
Architecture
Future improvements

Features

make a sense.
Richer client support - besides API and shell, we’ll add a client
service that will perform dropbox-like synchronization.
External queues support - conﬁgurable support using JMS instead of
internal queue.


VaporStore – the design of a real-world cloud filesystem

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (17)

Semelhante a VaporStore – the design of a real-world cloud filesystem

Semelhante a VaporStore – the design of a real-world cloud filesystem (20)

Último

Último (20)

VaporStore – the design of a real-world cloud filesystem