This document summarizes plans to rebuild the Ceph distributed storage system using Seastar, a framework for high-performance event-driven applications. Ceph is an open-source distributed storage platform that provides object, block, and file storage at scale. It uses a thread pool model that has limitations around lock contention and context switching. Seastar uses an asynchronous message passing model without locks that could improve Ceph's performance. The plan is to backfill Ceph components starting with critical I/O paths to prioritize basic functionality, then add supporting features later to fully rebuild Ceph on Seastar.

1. Rebuild the Ceph Distributed
Storage with Seastar
Kefu Chai
Software Engineer, Red Hat

2. Kefu Chai
He is a software engineer who has been focusing on storage
systems for over a decade.
Currently, he is working on Ceph, an open source, distributed
storage solution.

3. What is Ceph anyway?

4. About Ceph
▪ A platform
• Object, block and filesystem
▪ Scalable
• from 10 to 10,000 nodes (OSDs)
▪ No single point failure
▪ Free and Open Source (LGPL)

5. Ceph Components

6. CRUSH — the “partitioner” of objects

7. CRUSH (cont.)

8. good not not good enough

9. The latency analysis for write path
courtesy of Myonngwon Oh, et al., ALCA: Locality-Aware Lock Contention
Avoidance for NVMe-Based Scale-out Storage System

10. Blank Slide for full size diagram
Storage Backend
Network Messenger
Thread Pool serving requests

11. A closer look at the thread pool

12. Random R/W performance
The performance of random I/O is throttled by CPU

13. Blank Slide for full size diagram

14. Latencies in I/O path
▪ TCP/IP stack in kernel space
▪ Coarse grained locks
▪ Internal fragmentation
▪ Memory copy for direct-I/O

15. Seastar to the Rescue

16. Non-seastar
▪ Thread pool
▪ Lock
▪ Share state
▪ TCP/IP in kernel
▪ Callbacks
▪ Multiple-threaded
▪ Fiber
▪ Lockless
▪ Message passing
▪ TCP/IP in userspace (0-copy)
▪ future/promise
▪ single-threaded
Seastar

18. A message from the alien

19. A message from the alien
▪ std::future versus seastar::future
// in rocksdb
Status SequentialFile::read(size_t n,
char* buf)
{
return alien::submit_to(io_shard,
[n,buf,this]
{
return do_dma_read(n, buf);
}).wait();
}
▪ Dedicated cores for the alien territory

20. Backfill with priority
▪ Critical path first
• Basic I/O paths
▪ Supporting features later
• recovery
• Metric reporting
• …

21. Thank You
Any Questions ?
Please stay in touch
kchai@redhat.com
@tchaik0v