Vangelis Koukis, Technical Lead, Synnefo

1. 12/10/2013
Ceph Day London
vkoukis@grnet.gr
SYNNEFO + GANETI + CEPH.
VANGELIS KOUKIS, TECHNICAL LEAD, SYNNEFO
1
1

2. 12/10/2013
Running a public cloud: ~okeanos
Ceph Day London
vkoukis@grnet.gr
History
- Design started late 2010
- Production since July 2011
Numbers
- Users: > 3500
- VMs: > 5500 currently active
- More than 160k VMs spawned so far, more than 44k networks
2
2

3. 12/10/2013
Running a public cloud: ~okeanos
Ceph Day London
vkoukis@grnet.gr
Our choices
- Build own AWS-like service (Compute, Network, Storage)
- Persistent VMs
- Everything open source
- Production-quality IaaS
- Super-simple UI
How?
3
3

4. 12/10/2013
Running a public cloud: ~okeanos
Ceph Day London
vkoukis@grnet.gr
The tough stuff
- Stability
- Persistent VMs: VMs are not cattle, they are pets
- Commodity hardware
- Scalability
- Manageability: Gradual rollout of upgrades and new features
4
4

5. 12/10/2013
Running a public cloud: ~okeanos
Ceph Day London
vkoukis@grnet.gr
Our approach
- Synnefo
- Google Ganeti
- DRBD
- Ceph
- OpenStack APIs
5
5

6. 6/4/2013
6/3/2013
6/2/2013
6/1/2013
6/12/2012
6/11/2012
6/10/2012
6/9/2012
6/8/2012
6/7/2012
6/6/2012
6/5/2012
6/4/2012
6/3/2012
6/2/2012
6/1/2012
6/12/2011
6/11/2011
6/10/2011
6/9/2011
6/8/2011
12/10/2013
~okeanos VMs
Ceph Day London
vkoukis@grnet.gr
4500
4000
3500
3000
2500
2000
VMs
1500
1000
500
0
6
6

7. 12/10/2013
Cluster vs Cloud
Ceph Day London
vkoukis@grnet.gr
OPENSTACK
SYNNEFO
UI
OPENSTACK
OPENSTACK
API
SYNNEFO
CLOUD
OPENSTACK
CLUSTER
GANETI
LIBVIRT
KVM
NODE
KVM
HYPERVISOR
8
7

8. 12/10/2013
Google Ganeti
Ceph Day London
vkoukis@grnet.gr
Mature, production-ready VM cluster management
- used for Google’s corporate infrastructure
Multiple storage backends out of the box
- LVM, DRBD
- Files on local or shared directory
- RBD (Ceph/RADOS)
External Storage Interface for SAN/NAS support
Ganeti cluster = masterd on master noded on nodes
master,
Easy to integrate into existing infrastructure
- Remote API over HTTP, pre/post hooks for every action!
9
8

9. 12/10/2013
Architecture
Ceph Day London
vkoukis@grnet.gr
10
9

10. 12/10/2013
Ceph Day London
vkoukis@grnet.gr
Spawn
my own Ubuntu
Freeze
11
10

11. 12/10/2013
Ceph Day London
vkoukis@grnet.gr
Clone
Ubuntu + user data
Snapshot
12
11

12. 12/10/2013
Ceph Day London
vkoukis@grnet.gr
13
12

13. 12/10/2013
Ceph Day London
vkoukis@grnet.gr
14
13

14. 12/10/2013
Ceph Day London
vkoukis@grnet.gr
RADOS
block I/O
Monitor nodes
Volume
Composer
Storage
Mapper
and
Blocker
Archipelago
object I/O
Object Storage nodes
15
14

15. 12/10/2013
Volumes: Archipelago
Ceph Day London
vkoukis@grnet.gr
16
15

16. 12/10/2013
Volumes: Archipelago
Ceph Day London
vkoukis@grnet.gr
Unified storage for Files, Images  Volumes
Thin layer over the actual storage cluster
Storage backend agnostic
- NFS, RADOS, …
Efficient syncing / sharing of Images as files on the storage service
Zero-copy cloning of volumes from Images
17
16

17. 12/10/2013
Experience
Ceph Day London
vkoukis@grnet.gr
Operations
- Rolling hardware and software upgrades
- kernel, Ganeti, RADOS, Synnefo
- with no VM downtime
- Node evacuations with live VM migrations
- Cross-datacenter move, Intel  AMD, no VM downtime
- On-the-fly migration from NFS-backed storage to RADOS
- IP renumbering of all VMs
18
17

18. 12/10/2013
Experience
Ceph Day London
vkoukis@grnet.gr
Scalability
- From few physical hosts to multiple racks
- dynamic addition of Ganeti clusters
Diverse workloads
- Different network and storage backends
- Choice exposed to the user
19
18

19. 12/10/2013
Try it out!
Ceph Day London
vkoukis@grnet.gr
http://www.synnefo.org
20
19

20. 12/10/2013
Thank you!
Ceph Day London
vkoukis@grnet.gr
21
20