The RestFS is an experimental project to develop an open-source distributed filesystem for large environments. It is designed to scale up from a single server to thousand of nodes and delivering a high availability storage system with special features for high i/o performance and network optimization for work better in WAN environment. The Restfs is pure-python, but several of the libraries that it depends upon use C extensions (sometimes for speed, sometimes to interface to pre-existing C libraries). The Project is on the beginning stage, with some technology previews released.

1. RestFS
Fabrizio Manfredi Furuholmen
Federico Mosca
Beolink.org

2. Agenda Beolink.org
 Introduction
 Storage System
 Storage Evolution
 RestFS
 Goals
 Architecture
 Internals
 Sub project
 Demo
 Conclusion
2
PyCon Ireland 2012

3. Introduction Beolink.org
“Data stored globally is expected to
grow by 40-60% compounded
annually through 2020. Many
factors account for this rapid rate of
growth, though one thing is clear –
the information technology industry
needs to rethink how data is
shared, stored and managed…”
John H. Terpstra
By the Chairman of SambaXP 2012
3
PyCon Ireland 2012

4. Introduction Beolink.org
Web
3.0
Disaster PC
Recovery Store It
All, Serve
It
Worldwide
New
Devices
(TV,tablet,
mobile)
VM
4
PyCon Ireland 2012

5. Introduction Beolink.org
Zettabyte
All of the data on Earth today 150GB of data per person
2% of the data on Earth in 2020
5
PyCon Ireland 2012

6. Introduction Beolink.org
70’s Inode
Tree view
80’s Network filesystem (NFS/OpenAFS)
RPC
90’s Object Storage (OSD)
Parallel transfer
00’s Storage Service
WEB Base
Key Value
6
PyCon Ireland 2012

7. Introduction Beolink.org
“Late last week the number of objects
stored in Amazon S3 reached one trillion
(1,000,000,000,000 or 1012). That's 142
objects for every person on Planet Earth
or 3.3 objects for every star in our Galaxy.
If you could count one object per second
it would take you 31,710 years to count
them all.
We knew this day was coming!
Lately, we've seen the object count grow
by up to 3.5 billion objects in a single day
(that's over 40,000 new objects per
second)...”
Amazon Web Services Blog, June 12th
7
PyCon Ireland 2012

8. Introduction Beolink.org
John’s words + new usage + new services + …
8
PyCon Ireland 2012

9. RestFS Beolink.org
The RestFS is an experimental open-
source project with the goal to create a
distributed FileSystem for large
environments.
It is designed to scale up from a single
server to thousand of nodes and
delivering a high availability storage
system
9
PyCon Ireland 2012

10. Goals Beolink.org
Uniform Access Security
The Perfect Solution
• Global name support •Global
authentication/authorizatio
n
Reliability Availability
• No single point of failure •Maintenance without
disrupting the user’s
routines
Scalability Standard
• Tera/Peta/… bytes of data conformance:
Standard semantics
Performance: Elastic
• High performance •Bandwidth and capacity on
demand
10
PyCon Ireland 2012

11. Principle Beolink.org
“Moving Computation
is
Cheaper than Moving Data”
11
PyCon Ireland 2012

12. Five main areas Beolink.org
• Separation btw data and • Client side • Parallel operation • Resource discovery by • Secure connection
Security
Cache
Objects
Transmission
Distribution
metadata DNS
• Callback/Notify • Http like protocol • Encryption client side,
• Each element is marked • Data spread on multi
with a revision • Persistent • Compression node cluster • Extend ACL
• Each element is marked • Transfer by difference • Decentralize • Delegation/Federation
with an hash.
• Independents cluster • Admin Delegation
• Data Replication
12
PyCon Ireland 2012

13. RestFS Key Words Beolink.org
Bucket
virtual
container, hosted by
one or more server
Domain Object
entity (file, dir, …)
collection of servers contained in a
Bucket
RestFS
13
PyCon Ireland 2012

14. Object Beolink.org
Data Metadata
Serial
Hash
Object
Block 1 Properties
ACL
Serial
Hash
Block 2
Serial
Ext Properties
Block …
Hash
Serial
Segments
Hash
Block n
Serial
Attributes
set by user
14
PyCon Ireland 2012

15. Discovery Bucket Beolink.org
Cell 1
DNS
Lookup Bucket name
Bucket name Cell RL
IP list
N server
Client
Server list +
Load info N server
Server list
priority List
Cell 2
Server Priority Type
IP 1
.. …
15
PyCon Ireland 2012

16. Discovery and retrieve Data Beolink.org
Server Priority Type
Metadata
IP 1 Meta
IP 2 Data
IP 3 RL
.. ..
Client
Client Cache
16
PyCon Ireland 2012

17. Cache client side Beolink.org
DNS
ServerList
Resource Locator
Temporary
Cache
Tokens Federated Auth
Pub/Sub Callbacks
List
Metadata RestFS Metadata
Persistent
cache
RestFS Block
Block RestFS Block
cache RestFS Block
17
PyCon Ireland 2012

18. Server Architecture Beolink.org
S3 RestFS Auth Token Resource Sub/Pu
Interface
RPC Locator b
Service
Block Service
Callback Service
Meta Service
Token Service
Auth Service
RL Service
Managers
Storage Metadata Auth Token Resource Callbacks
Manager Manager Manager Manager Manager Manager
Distributed Cache
Drivers
Plugin
Storage Metadata Auth Token Resource Callbacks
Driver Driver Driver Driver Driver Driver
Backends
18
PyCon Ireland 2012

19. Client Architecture Beolink.org
Client Fuse
RestFuse Interface
Cache Service
Cache Manager RestFS RPC Sub/Pub
Connection Manager
storage
Locator RestFS RestFS
Sub/Pub
19
PyCon Ireland 2012

20. Beolink.org
Is everything ok ?
20
PyCon Ireland 2012

21. Beolink.org
Demo
21
PyCon Ireland 2012

22. Object Beolink.org
Object
Key Value Pair zebra.c1d2197420bd41ef24fc665f228e2c76e98da247
Key for everything Segment-id
1:16db0420c9cc29a9d89ff89cd191bd2045e47378
- Metadata: BUCKET_NAME.UUID 2:9bcf720b1d5aa9b78eb1bcdbf3d14c353517986c
- Block: BUCKET_NAME.UUID 3:158aa47df63f79fd5bc227d32d52a97e1451828c
4:1ee794c0785c7991f986afc199a6eee1fa4
5:c3c662928ac93e206e025a1b08b14ad02e77b29d
…
HASH vers:1335519328.091779
Each block has an hash to identify the content
Segment-hash
1:7d565defe000db37ad09925996fb407568466ce0
Serial 2:cc6c44efcbe4c8899d9ca68b7089506b7435fc74
Each element has a version which is identified by a serial. 3:660db9e7cd5b615173c9dc7daf955647db544580
4:fb8a076b04b550ff9d1b14a2bc655a29dcb341c4
5:b2c1ace2823620e8735dd0212e5424da976f27bc
...
Property
The property element is a collection of object information, with
Property
this element you can retrieve the most important metadata. segment_size= 512
block_size = 16k
content_type =
md5=ab86d732d11beb65ed0183d6a87b9b0
Object Serialized max_read’=1000
Backends agnostic on information stored storage_class=STANDARD
compression= none
…
22
PyCon Ireland 2012

23. Cache Beolink.org
Publish–subscribe
“… is a messaging pattern where senders of messages,
Demo http://www.websocket.org/echo.html
called publishers, do not program the messages to be sent
directly to specific receivers, called subscribers. Published
messages are characterized into classes, without
knowledge of what, if any, subscribers there may be. Use case A: 1,000
Subscribers express interest in one or more classes, and Use case B: 10,000
only receive messages that are of interest, without Use case C: 100,000
knowledge of what, if any, publishers there are… “
Wikipedia
Pattern matching
Clients may subscribe to glob-style patterns in order to
receive all the messages sent to channel names matching
a given pattern.
Distributed Cache
For server side the server share information over
distributed cache to reduce the use of backend
Client Cache
Pre allocated block with circular cache
write-through cache
23
PyCon Ireland 2012

24. Protocol Beolink.org
GET /mychat HTTP/1.1
WebSocket Host: server.example.com
is a web technology for multiplexing bi-directional, full- Upgrade: websocket
duplex communications channels over a single TCP Connection: Upgrade
connection. Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Version: 13
This is made possible by providing a standardized way Origin: http://example.com
for the server to send content to the browser without
being solicited by the client, and allowing for messages HTTP/1.1 101 Switching Protocols
to be passed back and forth while keeping the Upgrade: websocket
Connection: Upgrade
connection open… Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
Sec-WebSocket-Protocol: chat
JSON-RPC
is lightweight remote procedure call protocol similar to
--> { "method": "echo", "params": ["Hello JSON-RPC"], "id": 1}
XML-RPC. It's designed to be simple
<-- { "result": "Hello JSON-RPC", "error": null, "id": 1}
BSON
short for Binary JSON,
{"hello": "world"}
is a binary-encoded serialization of JSON-like
→
documents. Like JSON, BSON supports the embedding
"x16x00x00x00x02hellox00
of documents and arrays within other documents and
x06x00x00x00worldx00x00"
arrays.
BSON can be compared to binary interchange formats
24
PyCon Ireland 2012

25. Security Beolink.org
Security Channel
Communication over S3 protocol is based on SSL
Communication over RestFS RPC is based on WSS
Identification
Internal or through identity provider like Google, Facebook..
Token
For each identity is possible to assign more devices with
different token. This operation permits to exclude a stolen device
or enable a time period based one.
Authorization
The security control is based on extended ACL (nfs4)
Encryption
The user can encrypt the data with personal password, share
information through gnuPG framework (under development)
25
PyCon Ireland 2012

26. Pluggable Beolink.org
• Connection Handler
Protocol • Data transcoding
• High level Operations across multiple functions
(like locking)
Service • Integrity operations/transaction
• Operations handler for specific area (ex.
metadata)
Manager • Split info in sub info
• Read and write operation to storage
Driver system, agnostic operation
26
PyCon Ireland 2012

27. Backend: Metadata Beolink.org
Example of benchmark result
The test was done with 50 simultaneous clients
performing 100000 requests.
Transaction
The value SET and GET is a 256 bytes string.
The Linux box is running Linux 2.6, it's Xeon
X3320 2.5 GHz.
Text executed using the loopback interface
(127.0.0.1).
Connections
Cluster
Multi-master
Auto recovery
27
PyCon Ireland 2012

28. Backend: Storage Beolink.org
Kademlia's XOR distance is easier to calculate.
Kademlia's routing tables makes routing table
management a bit easier.
Each node in the network keeps contact information
for only log n other nodes
Kademlia implements a "least recently seen"
eviction policy, removing contacts that have not
been heard from for the longest period of time.
Key/value pair is stored on the node whose 160-bit
nodeID is closest to the key
Closest node, send a copy to neighbor
28
PyCon Ireland 2012

29. Open points: Space Beolink.org
What
happens
when
you
have
finished
the
space ?
29
PyCon Ireland 2012

30. What we are using Beolink.org
Module Software
Storage Filesystem, DHT (kademlia, Pastry*)
Metadata SQL(mysql,sqlite), Nosql (Redis)
Auth Oauth(google, twitter, facebook), kerberos*,
internal
Protocol Websocket
Message JSON-RPC 2.0, Amazon S3
Format
Encoding Plain, bson
CallBack Subscribe/Publish Websocket/Redis, Async I/O
TornadoWeb, AMPQ*
HASH Sha-XXX, MD5-XXX, AES
Encryptio SSL, ciphers supported by crypto++
n
Discovery
* are planned
DNS, file base
30
PyCon Ireland 2012

31. What is it good for ? Beolink.org
User
• Home directory
• Remote/Internet disks
Application
• Object storage
• Shared space
• Virtual Machine
Distribution
• CDN (Multimedia)
• Data replication
• Disaster Recovery
31
PyCon Ireland 2012

32. Subproject: Disaster Recovery Beolink.org
Element Configuration
Samba FS
Interfac VFX
e
Auth Samba
VFS
ACL Samba Transparent
Disaster Recovery site
Cache Queue mode RestFS Lib
Space One bucket
per share
Operation
queue
Fileserver
* Under development
32
PyCon Ireland 2012

33. Advantages Beolink.org
High Distributed
reliability Decentralized
Data replication
Nearly Horizontal scalability
unlimited Multi tier scalability
scalability
Cost- Cheap HW
efficient Optimized resource usage
Flexible User Property
Extended values and info
Enhanced Extended ACL
security OAUTH / Federation
Encryption
Token for single device
Simple to Plugin
Extend
Bricks
33
PyCon Ireland 2012

34. Roadmap Beolink.org
 0.1 Released
Single server on storage (No DHT)
S3 Interface
Federated Authentication
 0.2 Release September (codename WorstFS)
DHT on storage
Storage Encryption and compression
FUSE
 0.3 Release TBD (codename WorstFS++)
Deduplication
pub/sub
ACL
…
 Next
Clone function, Versioning, Disconnected
operation, Logging, Locks, Dlocks, Mount Bucket in
Bucket, Bucket automate provisioning, Distribution algorithms, Load
balancing, samba module, more async i/o, block replication control, negative
cache, index, user defined index
34
PyCon Ireland 2012

35. Support Beolink.org
Code, ideas, testing, insults … everything
35
PyCon Ireland 2012

36. I look forward to meeting you… Beolink.org
XVII European AFS meeting 2012
University of Edinburgh
October 16th to 18th
Who should attend:
 Everyone interested in deploying a globally accessible
file system
 Everyone interested in learning more about real
world usage of Kerberos authentication in single
realm and federated single sign-on environments
 Everyone who wants to share their knowledge and
experience with other members of the AFS and
Kerberos communities
 Everyone who wants to find out the latest
developments affecting AFS and Kerberos
More Info: http://openafs2012.inf.ed.ac.uk/
36
17/10/2012

37. Thank you
http://restfs.beolink.org
manfred.furuholmen@gmail.com
fege85@gmail.com
Beolink.org