Slides from our DreamObjects/Ceph webinar. This webinar was guest hosted by Kyle Bader of DreamHost.

1. DreamObjects
Cloud Object Storage
Powered by Ceph

2. About Me
• Senior systems engineer at dreamhost
• Been with dreamhost since 2007
• Systems lead for DreamObjects
• Excited about storage, distributed systems,
networking, linux internals, automation and
monitoring
• I built my first ceph cluster in april, 2010
twitter: @mmgaggle

3. Anatomy of DreamObjects

4. Nuts and Bolts - Storage Nodes
• Intel nehalem or AMD lisbon
• 32 GB of ram
• 8-12 enterprise 3.5" sas disks
• LSI megaraid 1078/2108
• 5x Gb nics
• IPMI 2.0 BMC

5. Nuts and Bolts - Support Nodes
• Intel westmere-ep
• 8 GB ram
• 3x Gb nics
• 4x SAS drives
• IPMI 2.0 BMC
ceph monitors, radosgw,
proxies, object sync

6. Tubes
• IPv6 native
• Dual-stack application proxy / balancer
• Application, rados, and cluster networks
• One gig links, bonded
• LLDP for discovery

7. Traffic
• IPv6 and IPv4 api endpoints
• IPv6 to radosgw nodes
• Stud for ssl termination
• HAproxy for load balancing
• 10Gb backend and frontend

8. Ceph Numbers
• Two plus petabytes of raw storage
• Triplicate replication
• 900+ OSDs
• 7 nines durability
• Single datacenter

9. CRUSH Configuration
• Row, rack, node, and disk fault tolerant
• Weighted, even distribution of data
• Object striping
• Online expansion and contraction

10. OSD Configuration
• Single disk RAID0
• Two partitions per RAID0, data + journal
• Writeback enabled
• Disk cache disabled
• Using XFS for filesystem

11. Application Tuning
• No keepalive on apache
• Increase apache thread pool
• Increase radosgw thread pool
• Raise radosgw ulimits

12. Argonaut
• Lots of failure testing
• Tested on large clusters (~900 osds)
• Weight osds in/out of the cluster

13. Bobtail
• Tunables for less aggressive remapping
• CNAME support
• OSD performance ++

14. Control and Management
• chef
• pdsh
• runit/upstart
• jenkins
• sensu
• collectd
• graphite
• logstash

15. Chef
• Bootstrap cluster, new nodes
• Packages and configuration
• Creates, destroys and encrypts osds
• Search node metadata, restful api
• Roles mapped to pdsh genders
• User and ssh key management
• Loads more..

16. pdsh
Parallel distributed shell
• Ceph monitors are a natural control point
• pdsh from monitor to other nodes
• ssh-agent + ssh agent forwarding
• Works when chef api doesn't

17. runit/upstart
Keep ceph daemons running
• ceph-osd
• ceph-mon
• radosgw

18. Jenkins
• Continuous api functional tests
• Monitor chef convergence
• Generates alerts on failures

19. Sensu
• Relatively new monitoring system
• Open source
• Written in ruby
• State stored in Redis
• Pub-sub with local agent
• Handlers monitor state and generate alerts

20. collectd
System metrics
• cpu
• memory
• disks
• entropy
• interfaces

21. graphite
• bucky - collectd, statsd, metricsd api
• carbon cache
• whisper for metric storage (sequential writes)
• graphite webui

22. logstash
• lumberjack agent to ship logs to logstash
• logstash graphite output
• logstash elasticsearch output
• elasticsearch restful api
• kibana webui sits on top of elasticsearch

23. Future
• East coast region
• Pool replication to another datacenter
• Object expiration, root site support
• Content delivery network
..not necessarily in that order ;)

24. Thanks
Questions?
kyle.bader [at] dreamhost.com
@mmgaggle

25. Inktank’s Professional Services
Consulting Services:
• Technical Overview
• Infrastructure Assessment
• Proof of Concept
• Implementation Support
• Performance Tuning
Support Subscriptions:
• Pre-Production Support
• Production Support
A full description of our services can be found at the following:
Consulting Services: http://www.inktank.com/consulting-services/
Support Subscriptions: http://www.inktank.com/support-services/

26. Check out our upcoming and on
demand webinars from Inktank
Upcoming:
Advanced Features of Ceph Distributed Storage
(delivered by Sage Weil, creator of Ceph)
February 12, 2013
10:00AM PT, 12:00PM CT, 1:00PM ET
http://www.inktank.com/news-events/webinars/
On Demand:
Getting Started with Ceph
http://www.inktank.com/news-events/webinars/
Intro to Ceph with OpenStack
http://www.inktank.com/news-events/webinars/

27. Contact Inktank
Info@inktank.com and 1-855-INKTANK
Don’t forget to follow us on:
Twitter: https://twitter.com/inktank
Facebook: http://www.facebook.com/inktank
YouTube: http://www.youtube.com/inktankstorage

28. THANK YOU!