Philipp explains the best performing Open Source software-defined storage software available to Apache CloudStack today. It consists of two well-concerted components. LINSTOR and DRBD. Each of them also has its independent use cases, where it is deployed alone. In this presentation, the combination of these two is examined. They form the control plane and the data plane of the SDS. We will touch on: Performance, scalability, hyper-convergence (data-locality for high IO performance), resiliency through data replication (synchronous within a site, 2-way, 3-way, or more), snapshots, backup (to S3), encryption at rest, deduplication, compression, placement policies (regarding failure domains), management CLI and webGUI, monitoring interface, self-healing (restoring redundancy after device/node failure), the federation of multiple sites (async mirroring and repeatedly snapshot difference shipping), QoS control (noisy neighbors limitation) and of course: complete integration with CloudStack for KVM guests. It is Open Source software following the Unix philosophy. Each component solves one task, made for maximal re-usability. The solution leverages the Linux kernel, LVM and/or ZFS, and many Open Source software libraries. Building on these giant Open Source foundations, not only saves LINBIT from re-inventing the wheels, it also empowers your day 2 operation teams since they are already familiar with these technologies. Philipp Reisner is one of the founders and CEO of LINBIT in Vienna/Austria. He holds a Dipl.-Ing. (comparable to MSc) degree in computer science from Technical University in Vienna. His professional career has been dominated by developing DRBD, a storage replication software for Linux. While in the early years (2001) this was writing kernel code, today he leads a company of 30 employees with locations in Austria and the USA. LINBIT is an Open Source company offering enterprise-level support subscriptions for its Open Source technologies. ----------------------------------------- CloudStack Collaboration Conference 2022 took place on 14th-16th November in Sofia, Bulgaria and virtually. The day saw a hybrid get-together of the global CloudStack community hosting 370 attendees. The event hosted 43 sessions from leading CloudStack experts, users and skilful engineers from the open-source world, which included: technical talks, user stories, new features and integrations presentations and more.

1. What CloudStackers Need to Know About LINSTOR/DRBD
Philipp Reisner, CEO LINBIT
1

2. Leading Open Source OS based SDS
COMPANY OVERVIEW
REFERENCES
• Developer of DRBD and LINSTOR
• 100% founder owned
• Offices in Europe and US
• Team of highly experienced Linux
experts
• Exclusivity Japan: SIOS
• Leading Open Source Block Storage
(included in Linux Kernel (v2.6.33)
• Open Source DRBD supported by
proprietary LINBIT products / services
• OpenStack with DRBD Cinder driver
• Kubernetes Driver
• Install base of >2 million
PRODUCT OVERVIEW
SOLUTIONS
LINBIT SDS
Since 2016
Perfectly suited for SSD/NVMe high performance storage
DRBD HA, DRBD DR
Market leading solutions since 2001, over 600 customers
Ideally suited to power HA and DR in OEM appliances (Cisco, IBM, Oracle)

3. 3
LINBIT SDS
When? Why? What?

4. 4
When is LINBIT SDS a fit?
Transaction Processing
●
Oracle DB
●
PostgreSQL
●
MariaDB
●
Message queuing systems
Analytic Processing
●
DB2 Warehouse
●
And similar read intensive workloads
PersistentVolumes
...for Containers
●
Kubernetes
●
Nomad
●
Docker
Virtualization
●
CloudStack
●
Others...

5. 5
Why is LINBIT SDS so fast?
Layout at volume allocation
• All participating machines have full replicas, which
machines participate determined when creating a
volume.
• Be faster at IO submission time
• Saving on CPU/memory
In Kernel data-path
• Reduce number of context switches
• Saving on CPU/memory resources
• Minimal latency for block-IO operations
• Optional load-balancing for READs
Hyper-Converged
Very well suitable for hyper-converged deployment
• Reduced network load for reads
• Reduces latency
• LINBIT SDS’ Low resource consumption leaves most
of CPU and memory for workload. About 0.5% of a
single core are consumed by DRBD under heavier IO
load (measured with an analytics DB)
Build on existing components
• DRBD, LVM, ZFS, LUKS, VDO, ...
• Help day2 operations by leveraging on the
opera tion teams prior knowledge
• Build on the shoulders of giants

6. 6
What is LINBIT SDS doing?
Storage Allocation
• 3 to 1000s of nodes
• Multiple tiers
• Multi tenancy
• Complex policies
Chassis - rack - room
Network
• Multiple NICs per server
• Multiple networks
• RDMA
• TCP
Business continuity
• Continuous data protection
• Multiple sites
• Backups
SSD - disk - cloud
Data Replication
• Persistence & availability
• Sync / async
• 2,3 or more replicas
• Consistency groups
• Quorum

7. 7
Linux Storage Gems
LVM, RAID, SSD cache tiers, deduplication, targets & initiators

8. 8
Linux's LVM
physical
volume
physical
volume
logical
volume
logical
volume
physical
volume
snapshot
Volume Group

9. 9
Linux's LVM
• based on device mapper
• original objects
• PVs, VGs, LVs, snapshots
• LVs can scatter over PVs in multiple segments
• thinlv
• thinpools = LVs
• thin LVs live in thinpools
• multiple snapshots became efficient!

10. 10
Linux's LVM
physical
volume
physical
volume
logical
volume
thinpool
physical
volume
snapshot
thin-LV thin-LV
thin-sLV
Volume Group

11. 11
Linux's RAID
• original MD code
• mdadm command
• Raid Levels: 0,1,4,5,6,10
• now available in LVM as well
• device mapper interface for MD code
• do not call it ‘dmraid’; that is software for hardware fake-raid
• lvcreate --type raid6 --size 100G VG_name
RAID1
A4
A3
A2
A1
A4
A3
A2
A1

12. 12
Linux’s DeDupe
• Virtual Data Optimizer (VDO) since RHEL 7.5
• Red hat acquired Permabit and is GPLing VDO
• Linux upstreaming is in preparation
• in-line data deduplication
• kernel part is a device mapper module
• indexing service runs in user-space
• async or synchronous writeback
• recommended to be used below LVM

13. 13
SSD cache for HDD
• dm-cache
• device mapper module
• accessible via LVM tools
• bcache
• generic Linux block device
• slightly ahead in the performance game
• dm-write-cache
• for combinding PMEM & NVMe drives

14. 14
Linux’s targets & initiators
• Open-ISCSI initiator
• Ietd, STGT, SCST
• mostly historical
• LIO
• iSCSI, iSER, SRP, FC, FCoE
• SCSI pass through, block IO, file IO, user-specific-IO
• NVMe-OF & NVMe/TCP
• target & initiator
Initiator Target
IO-requests
data/completion

15. 15
ZFS on Linux
• Ubuntu eco-system only
• has its own
• logic volume manager (zVols)
• thin provisioning
• RAID (RAIDz)
• caching for SSDs (ZIL, SLOG)
• and a file system!

16. 16
Put in simplest form

17. 17
DRBD – think of it as ...
Target
Initiator
IO-requests
data/completion
RAID1
A4
A3
A2
A1
A4
A3
A2
A1

18. 18
DRBD – another way to look at it

19. 19
DRBD Roles: Primary & Secondary
Primary Secondary
replication

20. 20
DRBD – multiple Volumes
• consistency group
Primary Secondary
replication

21. 21
DRBD – up to 32 replicas
• each may be synchronous or async
Primary
Secondary
Secondary

22. 22
DRBD – Diskless nodes
• intentional diskless (no change tracking bitmap)
• disks can fail
Primary
Secondary
Secondary

23. 23
DRBD - more about
• a node knows the version of the data is exposes
• automatic partial resync after connection outage
• checksum-based verify & resync
• split brain detection & resolution policies
• fencing
• quorum
• multiple resouces per node possible (1000s)
• dual Primary for live migration of VMs only!

24. 24
DRBD Recent Features & ROADMAP
• Recent
• meta-data on PMEM/NVDIMMS
• improved, fine-grained locking for parallel workloads
• support discard granularity up 128MiB (GiBs w. resync)
• release DRBD-9.2
• ROADMAP
• performance optimizations
• replace „stacking”
• NATed partitions

25. 25
The combination is more than the sum of its parts

26. 26
LINSTOR - goals
• storage build from generic Linux nodes
• for SDS consumers (CloudStack, K8s, OpenStack, ...)
• building on existing Linux storage components
• multiple tenants possible
• deployment architectures
• distinct storage nodes
• hyperconverged with hypervisors / container hosts
• LVM, thin LVM or ZFS for volume management (stratis later)
• Open Source, GPL

27. 27
Example

28. 28
LINSTOR - Hyperconverged
storage
devices
hypervisor
& storage
VM
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
VM

29. 29
LINSTOR - VM migrated
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
VM
VM

30. 30
LINSTOR - add local replica
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
VM
VM

31. 31
LINSTOR - remove 3rd
copy
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
storage
devices
hypervisor
& storage
VM
VM

32. 32
Architecture, objects and functions

33. 33
LINSTOR
Controller
LINSTOR
Satellite
LINSTOR
Satellite
LINSTOR
Satellite
LINSTOR
Controller
LINSTOR
Satellite
LINSTOR
Controller
LINSTOR
Satellite
LINSTOR
Protocol
TCP/IP
SSL/TLS
API
Library
CLI
XCP-NG

34. 34
LINSTOR Objects
• Nodes
• Resources

Volumes
• Snapshots
• Storage Pools

shared
• Resource groups
• Properties

Aux properties
node A
LVM VG
node C
shared
LVM VG
DRBD res.
2 replicas
encryped
volume
node B
ZFS pool
plain
volume
Resource Groups
Resource Groups

35. 35
LINSTOR Storage Layers
LVM VG, ZFS zPool, Exos, OpenFlex, SPDK, shared LVM VG
Mid (opt & multiple)
Bottom (required)
LUKS (encryption), caches, NVMe target & initiator
Top (opt) DRBD
Below (opt)
Bottom (required)
VDO (deduplication), software RAID

36. 36
LINSTOR data placement
Example policy
3 way redundant, where two
copies are in the same rack but in
diffeent fire compartments
(synchronous) and a 3rd
replica in
a different site (asynchronous)
Example tags
rack = number
room = number
site = city
• arbitrary tags on nodes
• require placement on equal/different/named tag values
• prohibit placements with named existing volumes
• different failure domains for related volumes

37. 37
LINSTOR network path selection
• a storage pool may preferred a NIC
• express NUMA relation of NVMe devices and NICs
• DRBD’s multi pathing supported
• load balancing with the RDMA transport
• fail-over only with the TCP transport

38. 38
in the Software Ecosystem

39. 39
LINSTOR connectors
Apache CloudStack
since 4.16
update with 4.18 (already merged)
with Qemu/KVM

40. 40
LINSTOR SDS vs LINSTOR/DRBD
LINBIT SDS From source code
yum / apt repo Red hat UBI Github → make → package
Support ✓ Enterprise, incl 24/7 Community only
GUI ✓ n.a.

41. 41
“Naming is hard” – Phil Karlton
Translation Matrix
LINSTOR Resource Group Resource/Volume
Kubernetes storageClass + file system → persistentVolume
Nomad
OpenNebula Datastore Image
OpenStack Volume Type Volume
Proxmox Storage Pool Volume
XCP-ng Storage Repository (SR) Virtual disk Image (VDI)
CloudStack Primary storage Volume

42. 42
Summary
DRBD Diskless NVMe - oF
ISCSI
Block transport systems
Orchestrators
Block Storage Features
DRBD LUKS DM - Cache
VDO
Hardware
HDD
Node-level volume management
LVM ZFS
SSD NVMe PMEM

43. 43
Thank you
https://www.linbit.com

44. 44
Appendix Slides: Example Disaggregated Architecture

45. 45
LINSTOR – disaggregated stack
storage
node
storage
node
hypervisor
VM
VM
storage
node
storage
node
hypervisor
VM
VM
hypervisor

46. 46
LINSTOR / failed Hypervisor
storage
node
storage
node
storage
node
storage
node
hypervisor
VM
VM
hypervisor
VM
VM

47. 47
LINSTOR / failed storage node
storage
node
hypervisor
VM
VM
storage
node
storage
node
hypervisor
VM
VM
hypervisor

48. 48
Appendix Slides: Possible Storage Stacks

49. 49
LINSTOR Storage Stacks
• Disaggregated Storage
• Classic enterprise workloads
• Data bases
• Message queues
• Typical Orchestrators
• OpenStack, OpenNebula
• Kubernetes
• Flexibly redundancy (1-n)
• HDDs, SSDs, NVMe SSDs
DRBD
App
DRBD
DRBD

50. 50
LINSTOR Storage Stacks
• Hyperconverged
• Classic enterprise workloads
• Data bases
• Message queues
• Typical Orchestrators
• OpenStack, OpenNebula
• Kubernetes
• Flexibly redundancy (1-n)
• HDDs, SSDs, NVMe SSDs
DRBD
DRBD
App
DRBD

51. 51
LINSTOR Storage Stacks
• Disaggregated
• Classic enterprise workloads
• Data bases
• Message queues
• Typical Orchestrators
• OpenStack, OpenNebula
• Kubernetes
• NVMe SSDs, SSDs
App
NVMe-oF
Initiator
NVMe-oF
Target
NVMe-oF
Initiator
NVMe-oF
Target
Raid1

52. 52
LINSTOR Storage Stacks
• Disaggregated
• Cloud native workload
• Ephemeral storage
• Typical Orchestrator
• Kubernetes
• Application handles redundancy
• Best suited for NVMe SSDs
App
NVMe-oF
Initiator
NVMe-oF
Target

53. 53
LINSTOR Storage Stacks
• Hyperconverged
• Cloud native workload
• Ephemeral storage
• PMEM optimized data base
• Typical Orchestrator
• Kubernetes
• Application handles redundancy
• PMEM, NVDIMMs
App

54. 54
LINSTOR Slicing Storage
• LVM or ZFS
• Thick – pre allocated
• Best performance
• Less features
• Thin – allocated on demand
• Overprovisioning possible
• Many snapshots possible
• Optional
• Encryption on top
• Deduplication below

55. 55
WinDRBD

56. 56
WinDRBD
• in public beta
• https://www.linbit.com/en/drbd-community/drbd-download/
• Windows 7sp1, Windows 10, Windows Server 2016
• wire protocol compatible to Linux version
• driver tracks Linux version with one day release offset
• WinDRBD user level tools are merged into upstream