Learn how ScyllaDB Cloud is moving to serverless, transforming its single tenant deployment model into a multi-tenant architecture based on Kubernetes. Discover the engineering innovation required, and the user value of the new architecture, including use of encryption (both at flight and at rest), performance isolation, and the capability to scale elastically.

1. ScyllaDB Cloud Goes
Serverless
Yaniv Kaul, VP R&D, ScyllaDB

2. Yaniv Kaul
■ Joined ScyllaDB in October 2022
■ Previously, at Red Hat, managed
■ Storage (Ceph, cloud storage),
■ Virtualization (OpenStack)
■ Performance and Scale.
■ Years ago, worked with ScyllaDB founders on the KVM
hypervisor

3. ■ Introduction
■ Tour
■ Architecture
Presentation Agenda

4. Introduction

5. ScyllaDB Cloud
■ Our managed ScyllaDB clusters offering
■ Provides clusters running on AWS or GCP clouds
■ Uses whole virtual machine(s) per tenant
■ Use our infrastructure or ‘bring-your-own-account’ nodes
■ Lifecycle, monitoring, logging, alerting - overall management by our DevOps and
Support teams
■ Free trial also available!
cloud.scylladb.com

6. ScyllaDB Cloud - K8S based
The next generation ScyllaDB Cloud is moving to run ScyllaDB workloads on Kubernetes (K8S).
■ Infrastructure components (storage, network, security, monitoring, logging and more)
deployment, management are handled by Kubernetes
■ Provides seamless integration with cloud services
■ Based on the open source ScyllaDB K8S operator

7. ScyllaDB Cloud - K8S based
Cluster Creation Phase

8. ScyllaDB Cloud - K8S based
Cluster Is Up

9. ScyllaDB Cloud - K8S based
General Tab

10. ScyllaDB Cloud - K8S based
Connection Bundle

11. ScyllaDB Cloud - K8S based
cqlsh Connection

12. docker run -it --entrypoint cqlsh
-v
/file/downloaded/from/cloud/connect-bundle-ScyllaSummit2023.yaml
:/connect-bundle-ScyllaSummit2023.yaml:Z
scylladb/scylla-nightly:5.2.0-dev-0.20221215.16c50bed5eb8
--cloudconf connect-bundle-ScyllaSummit2023.yaml

14. Python Example

15. ScyllaDB Cloud - K8S based
Monitoring

16. Architecture

17. Challenges With K8S
ScyllaDB on K8S has some of the ‘usual’ challenges and some ‘unusual’ ones:
■ Stateful application (a database)
■ Strict requirements from the operating system
■ Deployment requirements

18. The control plane is mostly running on a separate system
■ ScyllaDB Cloud management system was extended to treat serverless as
yet another cloud provider
■ Monitoring and logging are kept outside the operational K8S cluster running
the workloads
Management

19. First, need to provision a K8S node (worker), that will be used to run tenants’
ScyllaDB nodes:
■ RAID(0) configuration, OS tuning, etc.
■ Disk IO benchmarking, to get the baseline performance numbers.
■ Avoid ‘lemons’
■ Distribute baseline IO performance between tenants (per tenant CPU
shares)
Data Plane / Workers

20. The ScyllaDB K8S operator deploys a complete ScyllaDB cluster:
■ No. of nodes, size (CPU, memory, disk) , distribution, replication factor
(RF=3 for the time being), and other configuration items.
■ Configure / initialize node, wait for cluster formation
■ Manager (agent) & configuration
■ Monitoring
■ Drivers proxy (‘SNI proxy’)
Tenant Deployment

21. ■ Virtual machines forces a fixed cores/memory/disk relationship
■ Scaling can only be performed by adding or removing nodes
■ With K8S, we abstract underlying hardware and provide much more
flexibility.
Breaking the Fixed Compute Storage
Relationship

22. ■ Memory - each tenant (Pod) get their own (request/limit)
■ CPU - each tenant (Pod) get their own (request/limit)
■ Disk - each tenant get their own XFS mount point, with quota assigned
■ Scylla’s IO scheduler knows how much IOPS to give to each tenant
■ KMS is used to encrypt each tenant’s data
Tenant Isolation

23. /mnt/md0
Tenant Isolation - Storage (quota)
Instance
disks
Host path XFS Quota
/mnt/md0/local-pv-1 50G
/mnt/md0/local-pv-2 100G
kind: ScyllaCluster
spec:
datacenter:
racks:
- storage:
capacity: 50G
kind: ScyllaCluster
spec:
datacenter:
racks:
- storage:
capacity: 100G
Dynamic
storage
provisioner
RAID 0
XFS

24. Tenant Isolation - Storage (performance)
i3en.*xlarge
disks:
- mountpoint: /var/lib/scylla
read_iops: 125000
read_bandwidth: 1000000000
write_iops: 80000
write_bandwidth: 500000000
disks:
- mountpoint: /var/lib/scylla
read_iops: 125000
read_bandwidth: 1000000000
write_iops: 80000
write_bandwidth: 500000000
disks:
- mountpoint: /var/lib/scylla
read_iops: 250000
read_bandwidth: 2000000000
write_iops: 160000
write_bandwidth: 1000000000

25. Performance
Example of workload testing:
■ 4 loaders
■ 7 tenants
■ 4 OLTP, 3 OLAP
workloads
■ Mixed stress workload
Workload OLTP OLTP OLTP OLTP OLAP OLAP OLAP
CPU cores per scylla node 2 / 16 2 / 16 2 / 16 2 / 16 2 / 16 2 / 16 2 / 16
Cassandra-stress stats Latency [ms] THROUGHPUT - OP/S
loader 1 - latency p99 read 2.2 2.2 2.4 2.2 5,729 6,263 6,033
loader 2 - latency p99 read 2.2 2.2 2.2 2.1 5,629 6,294 5,979
loader 3 - latency p99 read 2.2 2.2 2.3 2.2 5,690 6,291 5,986
loader 4 - latency p99 read 2.2 2.5 2.2 2.2 5,685 6,283 5,986
loader 1 - latency p99 write 3.8 4 4.1 4.1 8,735 9,076 9,149
loader 2 - latency p99 write 3.9 4 4 3.9 8,325 8,898 8,809
loader 3 - latency p99 write 3.9 4 4 4.2 8,455 8,905 8,819
loader 4 - latency p99 write 3.8 4.2 4 4.2 8,450 8,893 8,836
loader 1 - latency p99 mixed 4.1 4 4.3 4.1 4,815 5,005 5,127
loader 2 - latency p99 mixed 4.1 4 4.2 4.1 4,552 4,835 4,899
loader 3 - latency p99 mixed 4.1 4 4.2 4.1 4,589 4,835 4,906
loader 4 - latency p99 mixed 4.1 4.2 4.2 4.1 4,587 4,819 4,903

26. The different CQL drivers connect to the cluster via a proxy, known as the SNI
(Server Name Indication) proxy:
■ Cluster data is stored in a YAML file, that clients use to connect to the
cluster.
■ Using TLS, the clients use the SNI field to indicate to which ScyllaDB node
they will connect to.
■ They still connect to all shards (shard-aware)
Connection to the Cluster

27. TLS protocol extension, enables connecting to a specific hostname
What is SNI?

28. As part of the ‘S301: ScyllaDB Operations’ course you can learn more about our K8S operator
and get some hands-on experience working with it.
ScyllaDB University Content

29. Thank You
Stay in Touch
Yaniv Kaul
yaniv.kaul@scylladb.com
@YanivKaul
github.com/mykaul
www.linkedin.com/in/ykaul/