In this session, you'll learn how RBD works, including how it: Uses RADOS classes to make access easier from user space and within the Linux kernel. Implements thin provisioning. Builds on RADOS self-managed snapshots for cloning and differential backups. Increases performance with caching of various kinds. Uses watch/notify RADOS primitives to handle online management operations. Integrates with QEMU, libvirt, and OpenStack.

1. Ceph Block Devices:
A Deep Dive
Josh Durgin
RBD Lead
June 24, 2015

2. Ceph Motivating Principles
● All components must scale horizontally
● There can be no single point of failure
● The solution must be hardware agnostic
● Should use commodity hardware
● Self-manage wherever possible
● Open Source (LGPL)
● Move beyond legacy approaches
– client/cluster instead of client/server
– Ad hoc HA

3. Ceph Components
RGW
A web services gateway
for object storage,
compatible with S3 and
Swift
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby, PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors
RBD
A reliable, fully-
distributed block device
with cloud platform
integration
CEPHFS
A distributed file system
with POSIX semantics
and scale-out metadata
management
APP HOST/VM CLIENT

4. Ceph Components
RGW
A web services gateway
for object storage,
compatible with S3 and
Swift
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby, PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors
RBD
A reliable, fully-
distributed block device
with cloud platform
integration
CEPHFS
A distributed file system
with POSIX semantics
and scale-out metadata
management
APP HOST/VM CLIENT

5. Storing Virtual Disks
M M
RADOS CLUSTER
HYPERVISOR
LIBRBD
VM

6. Kernel Module
M M
RADOS CLUSTER
LINUX HOST
KRBD

7. RBD
● Stripe images across entire cluster (pool)
● Read-only snapshots
● Copy-on-write clones
● Broad integration
– QEMU, libvirt
– Linux kernel
– iSCSI (STGT, LIO)
– OpenStack, CloudStack, OpenNebula, Ganeti, Proxmox, oVirt
● Incremental backup (relative to snapshots)

9. Ceph Components
RGW
A web services gateway
for object storage,
compatible with S3 and
Swift
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby, PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors
RBD
A reliable, fully-
distributed block device
with cloud platform
integration
CEPHFS
A distributed file system
with POSIX semantics
and scale-out metadata
management
APP HOST/VM CLIENT

10. RADOS
● Flat namespace within a pool
● Rich object API
– Bytes, attributes, key/value data
– Partial overwrite of existing data
– Single-object compound atomic operations
– RADOS classes (stored procedures)
● Strong consistency (CP system)
● Infrastructure aware, dynamic topology
● Hash-based placement (CRUSH)
● Direct client to server data path

11. RADOS Components
OSDs:
10s to 1000s in a cluster
 One per disk (or one per SSD, RAID group…)
 Serve stored objects to clients
 Intelligently peer for replication & recovery
Monitors:
 Maintain cluster membership and state
 Provide consensus for distributed decision-making
 Small, odd number (e.g., 5)
 Not part of data path
M

12. Ceph Components
RGW
A web services gateway
for object storage,
compatible with S3 and
Swift
LIBRADOS
A library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby, PHP)
RADOS
A software-based, reliable, autonomous, distributed object store comprised of
self-healing, self-managing, intelligent storage nodes and lightweight monitors
RBD
A reliable, fully-
distributed block device
with cloud platform
integration
CEPHFS
A distributed file system
with POSIX semantics
and scale-out metadata
management
APP HOST/VM CLIENT

13. Metadata
● rbd_directory
– Maps image name to id, and vice versa
● rbd_children
– Lists clones in a pool, indexed by parent
● rbd_id.$image_name
– The internal id, locatable using only the user-specified image name
● rbd_header.$image_id
– Per-image metadata

14. Data
● rbd_data.* objects
– Named based on offset in image
– Non-existent to start with
– Plain data in each object
– Snapshots handled by rados
– Often sparse

15. Striping
● Objects are uniformly sized
– Default is simple 4MB divisions of device
● Randomly distributed among OSDs by CRUSH
● Parallel work is spread across many spindles
● No single set of servers responsible for image
● Small objects lower OSD usage variance

16. I/O
M M
RADOS CLUSTER
HYPERVISOR
LIBRBD
CACHE
VM

17. Snapshots
● Object granularity
● Snapshot context [list of snap ids, latest snap id]
– Stored in rbd image header (self-managed)
– Sent with every write
● Snapshot ids managed by monitors
● Deleted asynchronously
● RADOS keeps per-object overwrite stats, so diffs are easy

18. LIBRBD
Snap context: ([], 7)
write
Snapshots

19. LIBRBD
Create snap 8
Snapshots

20. LIBRBD
Set snap context to ([8], 8)
Snapshots

21. LIBRBD
Write with ([8], 8)
Snapshots

22. LIBRBD
Write with ([8], 8)
Snap 8
Snapshots

23. Watch/Notify
● establish stateful 'watch' on an object
– client interest persistently registered with object
– client keeps connection to OSD open
● send 'notify' messages to all watchers
– notify message (and payload) sent to all watchers
– notification (and reply payloads) on completion
● strictly time-bounded liveness check on watch
– no notifier falsely believes we got a message

24. Watch/Notify
HYPERVISOR
LIBRBD
/usr/bin/rbd
LIBRBD
watch
HYPERVISOR
LIBRBD
/usr/bin/rbd
LIBRBD
watch

25. Watch/Notify
HYPERVISOR
LIBRBD
/usr/bin/rbd
LIBRBD
HYPERVISOR
LIBRBD
/usr/bin/rbd
LIBRBD
Add snapshot

26. Watch/Notify
HYPERVISOR
LIBRBD
/usr/bin/rbd
LIBRBD
notify
HYPERVISOR
LIBRBD
/usr/bin/rbd
LIBRBD
notify

27. Watch/Notify
HYPERVISOR
LIBRBD
/usr/bin/rbd
LIBRBD
notify ack
HYPERVISOR
LIBRBD
/usr/bin/rbd
LIBRBD
notify complete

28. Watch/Notify
HYPERVISOR
LIBRBD
read metadata
HYPERVISOR
LIBRBD

29. Clones
● Copy-on-write (and optionally read, in Hammer)
● Object granularity
● Independent settings
– striping, feature bits, object size can differ
– can be in different pools
● Clones are based on protected snapshots
– 'protected' means they can't be deleted
● Can be flattened
– Copy all data from parent
– Remove parent relationship

30. Clones - read
LIBRBD
clone parent

31. Clones - write
LIBRBD
clone parent

32. LIBRBD
clone parent
read
Clones - write

33. LIBRBD
clone parent
copy up,
write
Clones - write

34. Ceph and OpenStack

35. Virtualized Setup
● Secret key stored by libvirt
● XML defining VM fed in, includes
– Monitor addresses
– Client name
– QEMU block device cache setting
● Writeback recommended
– Bus on which to attach block device
● virtio-blk/virtio-scsi recommended
● Ide ok for legacy systems
– Discard options (ide/scsi only), I/O throttling if desired

36. libvirtvm.xml
QEMU
LIBRBD
CACHE
qemu ­enable­kvm ...
VM
Virtualized Setup

37. Kernel RBD
● rbd map sets everything up
● /etc/ceph/rbdmap is like /etc/fstab
● udev adds handy symlinks:
– /dev/rbd/$pool/$image[@snap]
● striping v2 and later feature bits not supported yet
● Can be used to back LIO, NFS, SMB, etc.
● No specialized cache, page cache used by filesystem on top

39. What's new in Hammer
● Copy-on-read
● Librbd cache enabled in safe mode by default
● Readahead during boot
● Lttng tracepoints
● Allocation hints
● Cache hints
● Exclusive locking (off by default for now)
● Object map (off by default for now)

40. Infernalis
● Easier space tracking (rbd du)
● Faster differential backups
● Per-image metadata
– Can persist rbd options
● RBD journaling
● Enabling new features on the fly

41. Future Work
● Kernel client catch up
● RBD mirroring
● Consistency groups
● QoS for rados (policy at rbd level)
● Active/Active iSCSI
● Performance improvements
– Newstore osd backend
– Improved cache tiering
– Finer-grained caching
– Many more

42. Questions?
Josh Durgin
jdurgin@redhat.com