This document summarizes a company's experience deploying the CEPH storage system in a production environment. They migrated about 50% of their storage to CEPH running on existing hardware with 6 object storage devices and 5 monitors. CEPH eliminated their single point of failure, allowed easy creation and destruction of virtual machine images, and provided self-healing capabilities. Some current issues include balancing bandwidth and latency for replicas and deciding how to allocate placement groups for different situations. Overall, the company is pleased with CEPH for providing the virtualized storage capabilities they desired.

1. Case Study: Flying Circus
Berlin CEPH meetup
2014-01-27, Christian Theune <ct@gocept.com>

2. /me
•
Christian Theune
•
Co-Founder of gocept
•
Software Developer
(formerly Zope, Plone, grok),
Python (lots of packages)
•
ct@gocept.com
•
@theuni

4. What worked for us?
raw image on local server
lvm volume via iSCSI (ietd + open-iscsi)

5. What didn’t work (for us)
ATA over Ethernet
Linux HA solution for iSCSI
Gluster
(sheepdog)

6. CEPH
•
been watching for ages
•
started work in December 2012
•
production roll-out since December 2013
•
about 50% migrated in production

7. Our production structure
•
KVM hosts with 2x1Gbps (STO and STB)
•
Old storages with 5*600GB RAID 5 + 1 Journal
SAS 15k drives
•
5 monitors, 6 OSDs currently
•
RBD from KVM hosts and backup server, 1 cluster
per customer project (multiple VMs)
•
Acceptable performance on existing hardware

8. Good stuff
•
No single point of failure any more.!
•
Create/destroy VM images on KVM hosts!
•
Fail-over and self-healing works nicely
•
Virtualisation for storage “as it should be”™
•
High quality of concepts, implementation, and
documentation
•
Relatively simple to configure

9. ceph -s (and -w)

10. ceph osd tree

11. Current issues
•
Bandwith vs. Latency: replicas from RBD client?!?.
•
Deciding for PG allocation in various situations.
•
Deciding for new hardware.
•
Backup has become a bottle neck.
•
I can haz “ceph osd pool stats” per RBD volume?
•
Still measuring performance. RBD is definitely sucking
up some performance.

12. Summary
•
finally … FINALLY … F I N A L L Y !
•
feels sooo good
•
well, at least we did not want to throw up using it
•
works as promised
•
can’t stop praising it …