1. 1Cleversafe, Inc. All rights reserved.
3 Ways Object Storage Can Save You Time in 2017
Lloyd Vickery
NZ’s Most Friendly Data Storage Consultant
2. 2Cleversafe, Inc. All rights reserved.
3 Ways Object Storage Can Save You Time in 2017
9/4/2017
1. Manage 1 device, not three
2. Don’t do backups
3. Spend 5X less time managing storage
3. 3Cleversafe, Inc. All rights reserved.
What Object Storage Does
Cleversafe software is not designed for
• High IOPS workflows
• Transaction Processing
• Inherent ILM
Cleversafe software provides a
• Secure,
• Reliable,
• Scalable,
• Cost Effective
4. 4Cleversafe, Inc. All rights reserved.
Where people use object storage
Service Providers
Media and Entertainment Life Science, FinServices, Marketing Federal
6. 6Cleversafe, Inc. All rights reserved.
Efficiency
How to build a highly reliable storage system
for 1 Petabyte of usable data?
RAID 6 + Replication Cleversafe®
1 PB
3.6 PB
900
3.6x
3.6x
3 FTE
Replication/backup
Usable Storage
Raw Storage
4TB Disks
Racks Required
Floor Space
Ops Staffing
Extra Software
$
70% +
TCO Savings
Original
1.20 PB Raw
Onsite mirror
1.20 PB Raw
Remote copy
1.20 PB Raw
1 PB
1.7 PB
432
1.7x
1.7x
.5 FTE
None
567 TB Raw 567 TB Raw567 TB Raw
7. 7Cleversafe, Inc. All rights reserved.
Original Data Encrypted Data
Writing Data to Cleversafe®
Original object is encrypted1
$
8. 8Cleversafe, Inc. All rights reserved.
Original Data
Encrypted Data
Writing Data to Cleversafe®
12
11
10
9
8
7
6
5
4
3
1
2
Erasure Coded Slices
7
6
5
4
3
1
2
Erasure coding is
used to transform
the data into a
customizable
number of slices
2
$
9. 9Cleversafe, Inc. All rights reserved.
$
Original Data Encrypted Data
Erasure Coded Slices
SITE 1 SITE 2 SITE 3
StorageNodes
Each slice is written to a
separate storage node. In
this example, the storage
nodes are geographically
dispersed across 3 sites.
3
Writing Data to Cleversafe®
12
11
10
9
8
7
6
5
4
3
2
1
12
11
10
9
8
7
6
5
4
3
1
2
10. 10Cleversafe, Inc. All rights reserved.
Reads can be performed using any 7 of the 12 slices
• The number of slices created on write is called the width (12 here)
• The number of slices required to read is called the threshold (7 here)
• This example is a “7 of 12” Information Dispersal Algorithm (IDA)
Reading Data from Cleversafe®
SITE 1 SITE 2 SITE 3
StorageNodes
7
6
5
4
3
1
2
1
3 7
2
5
6
4
$$
SITE 1 SITE 2 SITE 3
12. 12Cleversafe, Inc. All rights reserved.
Eliminate backups for unstructured data
Prod Storage DR Storage
Backup
Servers
Backup Disk
Pool
Backup Tape Offsite Tapes
Apps Apps
Object Storage
NAS Gateway
13. 13Cleversafe, Inc. All rights reserved.
PerfectBits™ Data Integrity
At Rest Integrity Checking
• Slice Integrity Check – Slicestors are checking themselves for corrupt slices
• Missing Slice Check – Slicestors are checking with each other to ensure all slices have
been written
• If Slices are found to be corrupt or missing they are added to the rebuild queue and rebuilt
Cleversafe uses a distributed rebuilder model
Rebuild events are distributed across Slicestors
Rebuilding becomes more powerful as Slicestors are added to the system
Predictability – Rebuilder is “always on” (at a moderated rate) making I/O performance
much more predictable
Benefits: No single point of failure or
choke point for performance and
highly scalable reliability
14. 14Cleversafe, Inc. All rights reserved.
PerfectBits™ Data Integrity
Check slice integrity Check data integrity Bit-perfect data
Inline Integrity Check
3 421 11 121097 865
Read
Request
If, during a read, a Slice is found to be corrupt, missing or otherwise
unusable, that Slice is added to a rebuild queue and the Accesser
picks a Slice from a different Slicestor to satisfy the read.
16. 16Cleversafe, Inc. All rights reserved.
An IBM Cloud Object Storage customer saved 8 FTE’s
9/4/2017
17. 17Cleversafe, Inc. All rights reserved.
Update drivers, OS and Object Software in 3 clicks
9/4/2017
Object Software
OS
Drivers
ClevOS
18. MANAGER®
3.2.1
SLICESTOR®
ACCESSER®
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
3.2.1
Manager pushes new code to
Accessers and a rolling upgrade
is performed.
2
Manager pushes new code to
Slicestors and a rolling
upgrade is performed.
3
3.4
3.4
3.4
3.4
3.4
3.4
3.4
3.4
3.4
3.4
3.4
3.4
3.4
3.4
3.4 No disruption to reads and
writes as long as read and
write threshold in the IDA is
maintained.
4
New code is downloaded to
Manager. Manager is upgraded.1
Always On
Non disruptive code upgrades are initiated and managed by the Manager
Non disruptive operations in the dsNet® – code upgrades
19. 19Cleversafe, Inc. All rights reserved.
System Management – Single pane of glass
Monitor
• Event monitoring via SNMPv3, email alerts
or event console
Configure
• Create Storage containers
• Select different IDAs to meet requirements
Security
• AD Integration
• LDAP
• Access Key
• IP Restrictions
•
Maintenance
• ‘Rolling Upgrade’ with no downtime
• Log collection, troubleshooting, reports
Web-Based Access
• Access Key
• IP Restrictions
Administration
• System configuration options
• Preferences, TLS, CA
Monitor Configure Security Maintenance Administration
RBAC Support
Provisioning API Available
20. 20Cleversafe, Inc. All rights reserved.
Event Console shows
incidents that require
action
Grouping shows summary
of large numbers of
devices, enables filtering
and searching
21. 21Cleversafe, Inc. All rights reserved.
3 Ways Object Storage Can Save You Time in 2017
9/4/2017
1. Manage 1 device, not three
2. Don’t do backups
3. Spend 5X less time managing storage
23. 23Cleversafe, Inc. All rights reserved.
In summary…
“What can I do today, that will save me
time in the coming weeks and months?”
Notas do Editor
Impact: Have you ever felt that no matter how hard you work there is never enough time to get everything done?
Has anyone had that chat with their manager / or sat in the quarterly business update and how am I going to have the time to do this.
If so, you’re not alone. I speak to our customers and partners every day and every time I check in with you “hi, how you doing?”, the most common response is “busy, I’m really busy Lloyd”.
So what we’re going to talk about today is three ways you can use object storage to save you time.
Intro:
I’m Lloyd, part of the storage team at IBM NZ. I’ve been working with IBM technology for 7 years, and at IBM NZ we help solve some of the countries most complicated data storage challenges.
We’re privileged to work with clients like
UoA, 2.7PB single namespace file system capable of 22,000 Concurrent CIFS connections
NeSI, 12.8PB file system
healthAlliance, 3PB of software defined storage spread across 8 sites. Data migration has gone from months months to a weekend.
Purpose: I’m really passionate about work life balance, effectives and you all getting home at reasonable hour so you can spend time with your families. That’s why I want to do something a little different today, we’re going to talk about object storage and how you can use it to save you time in the datacenter.
Object storage is proven overseas and you can be some of the first people in NZ to start reaping the rewards of using object storage.
This should be positioned in conjunction with Spectrum Scale.
Cleversafe alone is an Object storage solution. Not intended for High IOPS or Transaction processing. Analytics. ILM is not inherent.
Those requirements are met by Spectrum Scale and Cleversafe is a massively scalable long term object storage solution behind it.
Object been around since 2005.
Telco’s: KDDI
Uni’s: UoChicago, UoAkld,
MSP: Shutterfly, Basecamp
Media: Comcast, BSkyB, HBO
Marketing: Hogarth, Autodesk
Government: US Department of Justice
Aerospace: Lockheed Martin
Setup RAID for two devices, setup mirroring, update firmware for two devices.
How much time would you save if you didn’t have to do backups? Restores?
Backup software management, backup windows. Tape library. Tape management. Disk pool.
Up to 15x Nines
CRC added on write
Every read request data integrity check
~24hr data integrity check
Never lost a piece of production data
Show calculator
PerfectBits™
A trademark of Cleversafe®, Inc. for data integrity through an intelligent background process that proactively scans and corrects errors; scans of data slices for integrity and rebuilds any corrupted slices; and checks for both slice integrity and file data integrity prior to delivery. The key customer benefits are guarantees of bit-perfect data storage and delivery; assurance that data cannot be modified without authorization; detection of malicious threats; proactive detection and correction of bit errors; and discovery and correction of latent soft errors that may occur during normal read/write operations.
Shutterfly 250PB, 3 part time admins
Raptor, 1EB, 4 admins
UoChicago added 1PB in 2-3 days.
bSkyb removed 40-50 Netapp controllers to put it on ICOS and a much reduced NAS platform.
15X-25X more capacity per admin than managing NAS
Eliminate downtime.
Replacing hard drive proactively.
Stop worry about system wide outages.
Updating firmware
Scale as one system.
Move DC’s easily
SecureSlice combines AONT with IDispersal.
Imaging rolling a code update to a traditional storage array?! Time consuming, added risk, significant planning, contengency plans if things go bad... etc. And now image dong this accross several PB!?
This is the operational value our solution brings... always on!
traditional sense, planning, failover, failback, risk ...etc.
Conclusion/CTA
- Check out presentations on what you can use object storage for
- Free consultation on what workloads are object ready
Service providers felt the pain first, now all verticals are coming along.
Unstructured data (data that is not in a database) grows so fast (60%+) that traditional storage solutions are no longer viable.
At Petabyte scale this becomes unaffordable, will lead to data loss, becomes operationally unmanageable, extremely vulnerable.
A new category of storage, Object Storage, addresses these issues – Cleversafe is the industry leading object storage system (Gartner March 2016).
Architecture that makes this all happen.
dsNet System
Nodes.
Types of this node
Accessor Node - ingest, slice, disperse and retrieve.
Read/Write - 9'9s
Permanent - 11'9s
6 9's
10'9s permanent read
Designed to be always on.
Add capacity will auto rebalance content.
Added must be same width
SecureSlice combines AONT with IDispersal.
add more capacity and go! No more concerns about establishing new domain names and silo's of storage!
SecureSlice combines AONT with IDispersal.
- Traditional storage - operational, time and econ challenge
SW layer:
-- BTW... our SW is propetrual... so you are not purchasing again at refresh.
HW Layer:
mention failure on replace!
Going thru FIPS certification now, will be avail in 2016.
Indiv drives and nodes don’t have enough info to be compromised.
Data in motion protected with TLS.
AD, LDAP, Keystone authorization mechanisms for Vault access. Looking at user managed key for Softlayer, will be added soon.
SecureSlice™
SecureSlice technology combines AONT with Information Dispersal to preserve data confidentiality and integrity in the event slices are compromised. SecureSlice technology, although technically not encryption, guarantees (when enabled) that without access to a threshold number of slices, no information can be obtained without brute forcing the random symmetric key used to perform the transformation. If data obfuscation is not a consideration, some increased read/write performance may be achieved by disabling this feature.
Purpose of SecureSlice
SecureSlice is the technology Cleversafe® uses to guarantee confidentiality, integrity, and availability of data stored on a dsNet™. The goal of SecureSlice is to preserve these guarantees despite multiple compromises and breaches, up to some threshold number.
How SecureSlice Works
SecureSlice combines two algorithms: an Information Dispersal Algorithm (IDA) and an All-or-Nothing Transform, additionally a sentinel value is appended to the data prior to processing to ensure the integrity of the data upon retrieval and reassembly. When a segment of data is to be stored by our software, the first thing that happens is a sentinel value is appended. The sentinel value can be any well-known constant value, so long as its length is sufficient. In our implementation, the length is at least 16-bytes (128-bits). This value will be checked after decoding, to ensure that no corruption has occurred. If any slice used in the reassembly of the data segment had been corrupted, there is a very high probability that the sentinel value will be corrupted and therefore our software will notice this and prevent the invalid data from reaching the end-user. Furthermore, we have a technique called Corrupt Slice Detection, which attempts to find a valid combination of slices should the sentinel value be corrupt. After the sentinel is appended, the result is processed by the All-or-Nothing Transform. The first step in the All-or-Nothing Transform is to generate a random encryption key and encrypt the input using this key. The next step is to use a hash function to compute the digest of the encrypted data. The digest is then used as a One-Time-Pad to encrypt the randomly generated key. Finally, this “digest-encrypted” random key is appended to the end of the encrypted data, producing the All-or-Nothing Package. When one is in possession of this complete package, it is trivial to invert and get back to the original data. One simply strips off the digest-encrypted random key, re-calculates the digest of the encrypted data, uses this digest to decrypt the digest-encrypted random key, and finally uses that key to decrypt the encrypted data. Notice that had part of the package been missing, one could not compute the digest, and thus not decrypt the random key. Without the random key the encrypted data is as hard to decipher as any other encrypted data would be. Therefore to make the package secure, it must be split. Splitting of data, or slicing, is exactly what the IDA does as part of its normal operation. In a dispersal configuration with a threshold of 5, and 8 slices, the IDA will first split the All-or-Nothing Package into a threshold number of equally sized pieces, in this instance 5. Then the IDA uses its Forward Error Correction (FEC) algorithm to compute 3 slices containing redundancy information. The original split slices combined with the 3 computed slices results in 8 total slices. Given any combination of 5 slices from this set of 8, the original input can be derived. This is what provides the availability guarantee. In this case, the original input is the entirety of the All-or-Nothing Package, which once had can be inverted to yield the original data segment. Had any slice been accidentally or maliciously corrupted, the hash function would compute an incorrect digest, and therefore the wrong key would be used for decryption. When the encrypted data is decrypted with the wrong key, the sentinel will be decrypted incorrectly, alerting our software to the problem. Should an attacker gain access to fewer than a threshold number of slices, it is not possible to derive the original package, such a possibility would conflict with laws of information theory. The attacker is then left with two alternatives: attempt to guess, by brute force, the missing slice data or to attempt to guess by brute force the random encryption key. Both of these alternatives represent infeasible tasks given the number of possibilities is so vast.
----- Meeting Notes (10/21/14 14:04) -----
SecureSlice combines AONT with IDispersal.
think multi-tenancy in concert with data encryption
Data comes in from provider #1 - goes to vault 1
Data comes in from provider #2 - goes to vault 2
completely segregated and encrypted!
New graphic – similar style, but better graphic. Should feel broadcast-like
