Openstack Swift - Introduction - Features - Basic - Features - Advanced

1. Swift
Highly available, distributed, eventually consistent object
storage
Youn Sung Park

2. March 16, 2015 Presentation Title
2
1. Introduction
2. Features - Basic
3. Features - Advanced
Contents

3. Introduction
1

4. Contents
• What is Swift?
Object Storage
Swift component
March 16,
2015
Swift
4
Introduction

5. Object Storage
• Object Storage – What is Object Storage?
March 16,
2015
Swift
5
Introduction
Swift
- Highly available, distributed, eventually
consistent object storage

6. Object Storage
• Object Storage – What is Object Storage?
March 16,
2015
Swift
6
Introduction

7. Object Storage
• Object Storage – What is Object Storage?
March 16,
2015
Swift
7
Introduction
Object storage
vs File storage
vs Volume storage

8. Object Storage
• Object Storage – Legacy Storage
March 16,
2015
Swift
8
Introduction
WAS WAS WAS
NAS / SAN NAS / SAN NAS / SAN NAS / SAN
…
…

9. Object Storage
• Object Storage – Legacy Storage
March 16,
2015
Swift
9
Introduction
WAS WAS WAS
NAS / SAN NAS / SAN NAS / SAN NAS / SAN
…
…
Storage manager

10. Object Storage
• Object Storage – Legacy Storage
• Fault tolerance – RAID5, 6, 10, ...
• Interface – NFS for file, iSCSI for volume
• Multi tenancy - ??
• QoS - ??
• Authentication - ??
• Data Placement - ??
• Metadata management - ??
• Capacity management - ??
• Rebalancing - ??
March 16,
2015
Swift
10
Introduction

11. Object Storage
• Object Storage
March 16,
2015
Swift
11
Introduction
WAS WAS WAS
Object Storage
…

12. Object Storage
• Object Storage – Legacy Storage
• Fault tolerance – Replication
• Interface – REST API
• Multi tenancy – Defaut. SWAuth, Keystone, Tempauth, LDAP(?)
• QoS – Rate limit, Bandwidth limit, Software defined ...
• Authentication - SWAuth, Keystone, Tempauth
• Data Placement – The Ring algorithm
• Metadata management – Ring and database
• Capacity management – The Ring algorithm
• Rebalancing – The Ring algorithm
March 16,
2015
Swift
12
Introduction

13. Components
• Swift component – Server application
March 16,
2015
Swift
13
Introduction
Proxy Server
Account Server
Container Server
Object Server
Sqlite
Container List
Object List
Objects
Sqlite

14. Components
• Swift component – Daemon
• Replicator – Replicate objects and make a system in a consistent state.
• Recover disk failure, network outages situation.
• Updater – Update metadata
• Recover failure caused by container, account metadata high load.
• Auditor – Delete problematic account, container or objects and replicate from other server
• Recover dbs or files which have bit rot problem.
March 16,
2015
Swift
14
Introduction

15. Components
• Swift component – Key algorithm: Data placement, Metadata management
March 16,
2015
Swift
15
Introduction
RingContainer name/file name Location of a file
ex) 3 copy
Server A / sdc
Server H / sdy
Server D / sdp

16. Features - Basic
2

17. Contents
• Swift features
History – Releases
Architecture
High availability
High scalability
Eventually consistent & Self healing
REST API
March 16,
2015
Swift
17
Features - Basic

18. History - Releases
March 16,
2015
Swift
18
Features - Basic

19. Architecture
• Architecture
March 16,
2015
Swift
19
Features - Basic
Proxy Server Proxy Server Proxy Server
A/C/O Server
A/C/O Server
A/C/O Server
Zone1
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone2
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone3
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone4
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone5
A/C/O Server
…
…
…
Sqlite
Account server
Container server
Object server
Sqlite
Ring Ring Ring

20. Features
• High availability, Fault tolerance
March 16,
2015
Swift
20
Features - Basic
Proxy Server Proxy Server Proxy Server
A/C/O Server
A/C/O Server
A/C/O Server
Zone1
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone2
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone3
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone4
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone5
A/C/O Server
…
…
…
Sqlite
Account server
Container server
Object server
Sqlite
Ring Ring Ring

21. Features
• High scalability
March 16,
2015
Swift
21
Features - Basic
A/C/O Server
HDDHDDHDDHDD
A/C/O Server
HDDHDDHDDHDD
HDDHDDHDDHDD
More disk
Zone1
Zone1
Zone1
Zone1
Zone5
Zone1
Zone1
Zone1
Zone1
Zone1
Zone1
Zone7
More zones
A/C/O Server
A/C/O Server
Zone
A/C/O Server
A/C/O Server
Zone
A/C/O Server
A/C/O Server
More servers

22. Features
• Eventually consistent & self healing
March 16,
2015
Swift
22
Features - Basic
Proxy Server Proxy Server Proxy Server
A/C/O Server
A/C/O Server
A/C/O Server
Zone1
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone2
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone3
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone4
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone5
A/C/O Server
…
…
…
Sqlite
Account server
Container server
Object server
Sqlite
Fail
Eventual consistency
Ring Ring Ring

23. Features
• Eventually consistent & self healing
March 16,
2015
Swift
23
Proxy Server Proxy Server
A/C/O Server
A/C/O Server
Zone1
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone2
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone3
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone4
A/C/O Server
…
A/C/O Server
A/C/O Server
A/C/O Server
Zone5
A/C/O Server
…
…
…
Sqlite
Account server
Container server
Object server
Sqlite
Server
failure
Self healing
- replication
A/C/O Server
Proxy Server
Features - Basic
Ring RingRing

24. Features -Advanced
3

25. Contents
• Swift features
History – Features
Features
Large object support
Static web hosting
S3 compatible API
Object expiration
Temp url
Object versioning
Global cluster
Storage policy
March 16,
2015
Swift
25
Features - Advanced

26. History - Features
March 16,
2015
Swift
26
Features - Advanced
Swift 2.2 Release
Storage policy support
Swift 1.10 Release
Global clusters support
Swift 1.8 Release
Different replica policy
for A/C/O
StatsD update
Swift 1.7 Release
Object versioning support
Statd integration
Swift 1.4.8 Release
Object expiration support
Temp url support
Swift 1.4.3 Release
S3 compatible API support
Swift 1.3 Release
Static website support
Swift 1.2 Release
Large object support
Swift 1.0 Release
Swift 1.12 Release
Account ACL support

27. Features
• Large object support
• Swift limitation
• Single object: 5GB
• Split object & manage large object
• Manage segmented objects by manifest file
• Ref
• http://docs.openstack.org/developer/swift/overview_large_objects.html
March 16,
2015
Swift
27
Features - Advanced

28. Features
• Static web hosting
• Upload static web file and make web site
• Upload web site file with index and error files
• Use statiscweb middleware
• Ref
• http://docs.openstack.org/developer/swift/middleware.html#staticweb
March 16,
2015
Swift
28
Features - Advanced

29. Features
• S3 compatible API
• Support S3 API
• Support limited API less than 40%
• Use swift3 middleware
• Ref
• https://github.com/stackforge/swift3
March 16,
2015
Swift
29
Features - Advanced

30. Features
• Object expiration
• Schedule deletion of objects
• Use X-Delete-At and X-Delete-After header while using an object PUT or POST
•X-Delete-At: Delete object at specified time
•X-Delete-After: Delete object after specified time
• Ref
• http://docs.openstack.org/developer/swift/overview_expiring_objects.html
March 16,
2015
Swift
30
Features - Advanced

31. Features
• Temp url
• Provide url to access in limited time
• Need temp_url_expires time in header
• Use temporary URL middleware
• Ref
• http://docs.openstack.org/developer/swift/api/temporary_url_middleware.html
March 16,
2015
Swift
31
Features - Advanced

32. Features
• Global cluster
• Make a single cluster in distant region
• Read/Write affinity
• Deferred replication
• Ref
• http://docs.openstack.org/developer/swift/admin_guide.html
• https://swiftstack.com/blog/2012/09/16/globally-distributed-openstack-swift-cluster/
March 16,
2015
Swift
32
Features - Advanced

33. Features
• Storage policy
• Support various policy in sing storage cluster
• Use multiple ring file
• Ref
• http://docs.openstack.org/developer/swift/admin_guide.html
March 16,
2015
Swift
33
Features - Advanced

34. Thanks