Mais conteúdo relacionado Semelhante a OpenStack Days Krakow (20) 2. PROZETA © 2017
PROZETA
➡ Cloud Builder & Service Provider
➡ IoT, Industry 4.0 & Security Division
➡ Data Center Hardware Reseller
➡ Group annual turnover 20M EUR (2015)
➡ Active in 20+ countries
2
5. We started offering OpenStack
~2013 with following long-term goal:
To provide high-performance
enterprise-ready OpenStack cloud
with tight SLAs.
5
6. Tier5 Cloud
➡ Full-featured Software Defined
Datacenter
➡ Our Turn-key Private Cloud Product
➡ Hosted in our DCs or on-premise
➡ Covers all aspects of provisioning,
management & 24/7 support
➡ Based on OpenStack
➡ Fully software-defined
➡ Extensive monitoring dashboards
➡ Standardized APIs
7. We did a lot of testing and few
production installations with Ceph.
But then… stuff happened.
7
8. PROZETA © 2017
ISSUE:
recovery of Ceph storage stack became so
CPU/memory-hungry that it caused repetitive crashes of
OSD daemon… and ended up with broken objects
RESULT:
very hardly recoverable storage cluster
metadata in RADOS, randomly broken objects
everywhere, no other usable toolings around...
SOLUTION:
Can RedHat help? Maybe if we have few weeks/months
for recovery…
So a lot of scripting and low-level programming became
reality… and SLAs.
8
9. PROZETA © 2017
Next to already known
issues at that time...
➡ No compression, no deduplication
(at that time)
➡ Ineffective thin provisioning
➡ Extremely CPU hungry
➡ Bad write performance / high
latency - SLOOOOOOOOOW…
High TCO due to the need for extra
hardware, unpredictable performance
9
10. PROZETA © 2017
What was good about it?
➡ Perfect integration with OpenStack
➡ VERY stable
(BUT very difficult to fix potential issues
in a timely manner)
10
12. PROZETA © 2017
SDS Requirements
➡ High Stability
➡ High Reliability
➡ High and Predictable Performance
➡ Scaling options - both Horizontal &
Vertical
➡ Modest resource use
➡ Simple to manage (plus ability to repair in
case of disaster)
➡ Possible to get info for deep monitoring
& metrics
➡ Extensible: features like Compression,
Deduplication, Advanced Security
Features...
12
We wanted SDS on the level of Enterprise storage BUT based on a cost-effective
open-source technology we understand.
13. PROZETA © 2017
What Are The
Options?
➡ IaaS = Volumes / Block Storage
➡ Servers + Disks + Software
➡ Local storage
⇀ not scalable
➡ iSCSI
⇀ software defined & redundant: hard to
find
➡ GlusterFS, LizardFS, …
⇀ too complex to operate...
13
Goal:
“Next year I’d like to try out
a different storage software
solution…
...and I can!”
14. PROZETA © 2017
Situation on the Storage market
Open-source:
➡ Often re-inventing wheel
➡ Too old solutions
➡ Too complex solutions
➡ Over-engineered
➡ Often unstable
Proprietary:
➡ Often very expensive
➡ Closed source
➡ Not future proof (Software-defined)
➡ What if something goes wrong?
14
20. PROZETA © 2017
What is DRBD?
➡ Technology for data replication over the
network
➡ 15+ years on the market
➡ Developed by a stable company with great
reputation
➡ More than 250.000 production installations
worldwide
➡ Kernel modules (DRBD core) are standard
part of Linux kernel
➡ Super-reliable and super-fast!
20
21. PROZETA © 2017
Why is it so freakin' fast?
➡ Native Linux block device - not a block device
emulation on top of an object store
➡ Highly optimized meta data layout
➡ The SDS solution leverages on proven
technology
⇀ LVM or ZFS’ zVols
➡ Very well suited for hyperconverged
compute and storage
21
22. PROZETA © 2017
DRBD - Key Features
➡ automatic resync after node or connectivity
failure
⇀ direction, amount, no full resync needed
➡ performs under a Linux kernel implementation
⇀ 160k IOPs measured (on SSDs, of course)
➡ multiple volumes per resource (replication
group)
⇀ write order fidelity within resource
comes with Pacemaker integration
➡ synchronous and async replication
(LAN and WAN)
➡ In Linux upstream since 2.6.33 (released 2010)
22
23. PROZETA © 2017
DRBD
New Features in 9.x
➡ Up to 32 nodes per resource
⇀ Fixes the drawbacks of stacking
➡ Auto promote
➡ Transport abstraction (TCP, SCTP, RDMA)
➡ DRBD Manage
23
26. PROZETA © 2017
Stability & Reliability
We have tried to totally break it a
million times...
with NO SUCCESS!
Just what you would expect in a
production environment.
26
28. PROZETA © 2017
Building best SDS
We decided to combine the best-of-breed open
source:
➡ Linux kernel
➡ ZFS
➡ DRBD
➡ & many more for supporting systems
We are improving stability, performance &
features without additional involvement, almost
every week.
Not like others who are trying & failing to
re-invent on-disk formats
28
32. PROZETA © 2017
Policy-driven storage
➡ Use simple policies to control:
⇀ Replication strategy
⇀ Data placement
⇀ Balance speed & data protection (RPO)
⇀ QoS & SLA
⇀ Backup
⇀ … and various other knobs
➡ Assign policies to / by:
⇀ Storage objects
⇀ Consumer objects
⇀ (OpenStack tenant, Cinder volume
metadata, ...)
32
33. PROZETA © 2017
Multi-tenant & Multi-cloud
Use the same storage pool for
➡ Multiple OpenStack tenants
➡ Multiple OpenStack clusters
➡ OpenStack & Docker
➡ …..
We have QoS, so no issues with multi-tenant
environment
Linked authentication
➡ Use Keystone for storage authentication
➡ Allows OpenStack users to manage resources
(if allowed)
33
34. PROZETA © 2017
Scaling
Horizontally: Scale-out
➡ Add nodes to grow cluster
➡ Hyper-converged nodes
(more nodes, lower capacity per node)
Vertically: Scale-up
➡ Low CPU usage
➡ High density storage (high capacity, high iops)
➡ 76 TB SSD & 1-2M IOPs in one node? Why not?
Single volume striped across multiple nodes w/
read balancing
➡ Extra boost in performance
34
35. PROZETA © 2017
QoS
➡ Limit IOPs per
⇀ Volume
⇀ Volume group
⇀ Consumer object (OpenStack tenant, … )
⇀ Consumer
(OpenStack cluster for multi-cluster
environment)
➡ Set priority
⇀ Set priority per policy
➡ Alpha version - many limitations but usable
➡ Bandwidth limitation not yet implemented
35
36. PROZETA © 2017
Backup & “Recycle Bin”
➡ Ability to roll-back without any actual
recovery process
⇀ Storage snapshots (volume roll-back)
⇀ Deferred delete (volume undelete)
➡ Backup to another host
⇀ Running in background
⇀ Simple policies
⇀ No performance degradation
⇀ Encrypted over the wire
➡ Recover anything
⇀ Single volume
⇀ Full storage
36
37. PROZETA © 2017
Fully Cloud Aware Storage
Everyone's talking about it, but there
is none as of now! What it means?
➡ Bi-directional interfaces (OpenStack, Docker,
etc…)
➡ BlackStor keeps all informations about your
volumes & VMs inside
➡ Simply define QoS/SLA for OpenStack tenant
or VM…
…and see if SLA is not breached in the Web UI
37
38. PROZETA © 2017
Deep insight
➡ Cloud Native - Data + all metadata in one
place
➡ CLI, Web UI
➡ Real-time status
➡ Metrics
⇀ IOPs, bandwidth
⇀ Latency distribution
⇀ Storage health
(software, hardware)
⇀ … and many more
38
40. ➞ Tough job!
➞ We’re sitting on tons
of customers’ data
➞ Huge responsibility
➞ And GDPR makes it even worse
⇀ New rules for CSPs and up to 20M EUR
fine for a breach...
40
Cloud Service Provider
42. 42
Cloud Service Provider
How to protect us?
Encrypt all data & have no access!!
This is valid for service provider but
it’s valid for internal service
providers (internal IT teams) as well
43. PROZETA © 2017
Security
➡ Trusted Computing
➡ Full hardware life-cycle security management
➡ Data encryption, Trusted storage, Over-the-wire
encryption
➡ Trusted logging - auditability, tampering
detection
➡ Missing part in any cloud software.
➡ VMware just scratched the surface.
GDPR-compliant? Yes! Because we don't necessarily
need to be able to access the data...
43
44. How does it work in the real
world?
(It works in theory… does it
work in practice?)
44
45. PROZETA © 2017
Performance
DRBD by itself
➡ HW
⇀ 2x IBM 8247-22L
⇀ Power 8 2 sockets
⇀ 128 GB RAM
⇀ Mellanox 100GBps InfiniBand
⇀ HGST Ultrastart NVMe SSDs
➡ Ubuntu Xenial on bare metal
➡ DRBD 9.0.1 & RDMA Transport 2.0
➡ fio 2.2.10
➡ Random IO
45
52. PROZETA © 2017
Line-rate Performance
< 2 ms write latency (default config, SSD)
< 1 ms write latency (tuned config, SSD)
< 0.1 ms write latency (dedicated low latency log drives, SSD)
That means consistent high performance under workload.
All that will low CPU usage.
40x improvement over CEPH
52
56. ...but we went for it anyway.
Why? Because...
there was nothing reliable out there
great performance & production ready
sweet spot between open source
Ceph & expensive Enterprise storage
56
57. PROZETA © 2017
What do you get with
this solution?
➡ Production ready
➡ Stable & Reliable
➡ Great & Predictable Performance
➡ Simple to implement & manage
➡ Includes deep metrics information
➡ Advanced security features (eg. for GDPR)
➡ With additional support options
...and the last but not least….
➡ Fantastic TCO - it doesn't cost the Earth!
57
59. 59
PRO-ZETA a.s.
Prague, Czech Republic
prozeta@prozeta.eu
www.prozeta.eu
www.tier5.cloud
PRO-ZETA Middle East
Dubai, UAE
prozeta@prozeta.ae
www.prozeta.ae
61. Data Encryption
Parts needed:
➞ Run VM only on a trusted (attested)
platform (HW, OS, hypervisor)
➞ Validate VM consistency
➞ Protect VM’s memory
➞ Monitor malicious activity
Trusted platform? What does it mean?
Technologies:
➞ TPM, Intel TXT, SGX, Xen Guest TPM
61
62. VM Encryption
We are lost anyway unless we encrypt
the VM’s memory because of the recent
CPU bugs (Meltdown, Spectre)
Hardware-based VM memory encryption:
➞ Intel SGX (Software Guard Extensions)
➞ AMD SME (Secure Memory Encryption)
Workarounds:
➞ No admin access to the hypervisor
➞ Fully automated deployment
➞ Single-tenant hypervisor only
62
63. VM Encryption
Implement using an Glance OS image
with two partitions:
1. OS boot/initrd, untrusted part
2. OS root, trusted, encrypted
Need to encrypt of the second (OS root)
partition after the VM initialization...
… or you always need to bootstrap the OS
from the installation image
Unfortunately you can’t rekey ZFS,
dm-crypt or any other filesystem at the
moment 63
64. Storage Efficiency
Encryption?
➞ Forget compression
➞ Forget deduplication
➞ Not a big deal
○ hardware cost per GB decreases Y-to-Y
○ enable compression within the VM
➞ CSPs can’t hardly overprovision
storage anyway
64
66. OpenStack?
VM Encryption
➞ Intel SGX: KVM/Qemu, Xen
➞ AMD SME: KVM/Qemu, Xen
Key management
➞ Barbarican w/ Intel SGX secure
enclave
Don’t try this at home!
66
