Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Ceph Intro and Architectural Overview by Ross Turk
Semelhante a Ceph Intro and Architectural Overview by Ross Turk (20)
Mais de buildacloud
Mais de buildacloud (20)
Último
Último (20)
Ceph Intro and Architectural Overview by Ross Turk
- 1. Ceph Intro & Architectural Overview Ross Turk VP Community, Inktank
- 2. ME ME ME ME ME ME. 2 I made a slide today. It’s all about me. Ross Turk VP Community, Inktank ross@inktank.com @rossturk inktank.com | ceph.com
- 3. 3 CLOUD SERVICES COMPUTE NETWORK STORAGE the future of storage™
- 7. 7 How Much Store Things All Human History?! writing paper computers distributed storage cloud computing gaaaaaaaaahhhh!!!!!! carving
- 14. Storage Appliance Michael Moll, Wikipedia / CC BY-SA 2.0 14
- 15. SUPPORT AND MAINTENANCE PROPRIETARY SOFTWARE 15 PROPRIETARY HARDWARE DISK COMPUTE R DISK COMPUTE R DISK COMPUTE R DISK COMPUTE R 34% of 2012 revenue (5.2 billion dollars) 1.1 billion in R&D spent in 2012 1.6 million square feet of manufacturing space
- 18. 18
- 19. 19 OPEN SOURCE COMMUNITY- FOCUSED SCALABL E NO SINGLE POINT OF FAILURE SOFTWARE BASED SELF- MANAGING philosophy design
- 20. 20 8 years & 20,000 commits later…
- 21. 21 CEPH OBJECT GATEWAY A powerful S3- and Swift- compatible gateway that brings the power of the Ceph Object Store to modern applications CEPH BLOCKDEVICE A distributed virtual block device that delivers high- performance, cost- effective storage for virtual machines and legacy applications CEPH FILESYSTEM A distributed, scale-out filesystem with POSIX semantics that provides storage for a legacy and modern applications OBJECTS VIRTUAL DISKS FILES & DIRECTORIES CEPH STORAGECLUSTER A reliable, easy to manage, next-generation distributed object store that provides storage of unstructured data for applications
- 22. 22 RADOS A reliable, autonomous, distributed object store comprised of self-healing, self- managing, intelligent storage nodes LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RBD A reliable and fully- distributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOSGW A bucket-based REST gateway, compatible with S3 and Swift APP APP HOST/VM CLIENT
- 23. 23 RADOS A reliable, autonomous, distributed object store comprised of self-healing, self- managing, intelligent storage nodes LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RBD A reliable and fully- distributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOSGW A bucket-based REST gateway, compatible with S3 and Swift APP APP HOST/VM CLIENT
- 24. 24 DISK FS DISK DISK OSD DISK DISK OSD OSD OSD OSD FS FS FSFS btrfs xfs ext4 MMM
- 25. 25 M M M HUMAN
- 26. 26 Monitors: • Maintain cluster membership and state • Provide consensus for distributed decision-making • Small, odd number • These do not serve stored objects to clients M OSDs: • 10s to 10000s in a cluster • One per disk • (or one per SSD, RAID group…) • Serve stored objects to clients • Intelligently peer to perform replication and recovery tasks
- 27. 27 RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RBD A reliable and fully- distributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOSGW A bucket-based REST gateway, compatible with S3 and Swift APP APP HOST/VM CLIENT
- 29. L LIBRADOS • Provides direct access to RADOS for applications • C, C++, Python, PHP, Java, Erl ang • Direct access to storage nodes • No HTTP overhead
- 30. 30 RADOS A reliable, autonomous, distributed object store comprised of self-healing, self- managing, intelligent storage nodes LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RBD A reliable and fully- distributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOSGW A bucket-based REST gateway, compatible with S3 and Swift APP APP HOST/VM CLIENT
- 32. 32 RADOS Gateway: • REST-based object storage proxy • Uses RADOS to store objects • API supports buckets, accounts • Usage accounting for billing • Compatible with S3 and Swift applications
- 33. 33 RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOSGW A bucket-based REST gateway, compatible with S3 and Swift APP APP HOST/VM CLIENT RBD A reliable and fully- distributed block device, with a Linux kernel client and a QEMU/KVM driver
- 37. 37 RADOS Block Device: • Storage of disk images in RADOS • Decouples VMs from host • Images are striped across the cluster (pool) • Snapshots • Copy-on-write clones • Support in: • Mainline Linux Kernel (2.6.39+) • Qemu/KVM, native Xen coming soon • OpenStack, CloudStack, Nebula, Proxmox
- 38. 38 RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes LIBRADOS A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP RBD A reliable and fully- distributed block device, with a Linux kernel client and a QEMU/KVM driver CEPH FS A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE RADOSGW A bucket-based REST gateway, compatible with S3 and Swift APP APP HOST/VM CLIENT
- 40. 40 Metadata Server • Manages metadata for a POSIX-compliant shared filesystem • Directory hierarchy • File metadata (owner, timestamps, mode, et c.) • Stores metadata in RADOS • Does not serve file data to clients • Only required for shared filesystem
- 41. What Makes Ceph Unique? Part one: CRUSH 41
- 43. How Long Did It Take You To Find Your Keys This Morning? azmeen, Flickr / CC BY 2.0 43
- 45. Dear Diary: Today I Put My Keys on the Kitchen Counter Barnaby, Flickr / CC BY 2.0 45
- 47. I Always Put My Keys on the Hook By the Door vitamindave, Flickr / CC BY 2.0 47
- 48. HOW DO YOU FIND YOUR KEYS WHEN YOUR HOUSE IS INFINITELY BIG AND ALWAYS CHANGING? 48
- 49. The Answer: CRUSH!!!!! pasukaru76, Flickr / CC SA 2.0 49
- 50. 50 10 10 01 01 10 10 01 11 01 10 10 10 01 01 10 10 01 11 01 10 hash(object name) % num pg CRUSH(pg, cluster state, rule set)
- 51. 51 10 10 01 01 10 10 01 11 01 10 10 10 01 01 10 10 01 11 01 10
- 52. 52 CRUSH • Pseudo-random placement algorithm • Fast calculation, no lookup • Repeatable, deterministic • Statistically uniform distribution • Stable mapping • Limited data migration on change • Rule-based configuration • Infrastructure topology aware • Adjustable replication • Weighting
- 53. 53 CLIENT ??
- 54. 54 NAME:"foo" POOL:"bar" 0101 1111 1001 0011 1010 1101 0011 1011 "bar" = 3 hash("foo") % 256 = 0x23 OBJECT PLACEMENT GROUP 24 3 12 CRUSH TARGET OSDsPLACEMENT GROUP 3.23 3.23
- 55. 55
- 56. 56
- 57. 57 CLIENT ??
- 58. What Makes Ceph Unique Part two: thin provisioning 58
- 60. HOW DO YOU SPIN UP THOUSANDS OF VMs INSTANTLY AND EFFICIENTLY? 60
- 61. 144 61 0 0 0 0 instant copy = 144
- 64. What Makes Ceph Unique? Part three: clustered metadata 64
- 65. POSIX Filesystem Metadata Barnaby, Flickr / CC BY 2.0 65
- 67. 67 M M M
- 68. 68 one tree three metadata servers ??
- 69. 69
- 70. 70
- 71. 71
- 72. 72
- 74. Getting Started With Ceph Read about the latest version of Ceph. • The latest stuff is always at http://ceph.com/get Deploy a test cluster using ceph-deploy. • Read the quick-start guide at http://ceph.com/qsg Deploy a test cluster on the AWS free-tier using Juju. • Read the guide at http://ceph.com/juju Read the rest of the docs! • Find docs for the latest release at http://ceph.com/docs 74 Have a working cluster up quickly.
- 75. Getting Involved With Ceph Most project discussion happens on the mailing list. • Join or view archives at http://ceph.com/list IRC is a great place to get help (or help others!) • Find details and historical logs at http://ceph.com/irc The tracker manages our bugs and feature requests. • Register and start looking around at http://ceph.com/tracker Doc updates and suggestions are always welcome. • Learn how to contribute docs at http://ceph.com/docwriting 75 Help build the best storage system around!
- 76. Ceph Cuttlefish (v0.61.x) 1. New ceph-deploy provisioning tool 2. New Chef cookbooks 3. Fully-tested packages for RHEL (in EPEL) 4. RGW authentication management API 5. RADOS pool quotas 6. New ceph df 7. RBD incremental snapshots 76 Best Ceph ever.
- 77. Questions? 77 Ross Turk VP Community, Inktank ross@inktank.com @rossturk inktank.com | ceph.com
Notas do Editor
- Hi, welcome to my talk. I’m really happy that you chose to join me for this, given your many other choices. Believe me, I’m going to tell you things that will literally tear your head off. Ok, not literally. That would be really messy.
- Working through a computer means that we can store more information, and we can store it more quickly. But it also means that we’re separated from the information we’ve created.
- Working through a computer means that we can store more information, and we can store it more quickly. But it also means that we’re separated from the information we’ve created.
- Ceph was designed to be self-managing. Lots of distributed storage systems require operator intervention when something goes wrong.
- RADOS is a distributed object store, and it’s the foundation for Ceph. On top of RADOS, the Ceph team has built three applications that allow you to store data and do fantastic things. But before we get into all of that, let’s start at the beginning of the story.
- But that’s a lot to digest all at once. Let’s start with RADOS.
- Remember all that meta-data we talked about in the beginning? Feels so long ago. It has to be stored somewhere! Something has to keep track of who created files, when they were created, and who has the right to access them. And something has to remember where they live within a tree. Enter MDS, the Ceph Metadata Server. Clients accessing Ceph FS data first make a request to an MDS, which provides what they need to get files from the right OSDs.
- If you aren’t running Ceph FS, you don’t need to deploy metadata servers.
- So now that you know what Ceph is, I’m going to tell you what makes it different.
- All of that metadata for Ceph FS has to be stored somewhere. It’s a giant diary, keeping track of where everything is and who owns it.
- MDSs store all of their data within RADOS itself, but there’s still a problem…
- There are multiple MDSs!
- So how do you have one tree and multiple servers?
- If there’s just one MDS (which is a terrible idea), it manages metadata for the entire tree.
- When the second one comes along, it will intelligently partition the work by taking a subtree.
- When the third MDS arrives, it will attempt to split the tree again.
- Same with the fourth.
- A MDS can actually even just take a single directory or file, if the load is high enough. This all happens dynamically based on load and the structure of the data, and it’s called “dynamic subtree partitioning”.