In this 90-minute hands-on workshop, some of the key contributors to OpenNebula will walk attendees through the configuration and integration aspects of the storage subsystem in OpenNebula. The session will also include lightning talks by community members describing aspects related to Storage with OpenNebula: Deployment scenarios Integration Tuning & debugging Best practices

1. Storage Workshop
OpenNebulaConf 2016
4th
edition

2. Agenda for the Workshop
● OpenNebula Storage Model
○ QCOW2 Drivers
● Lightning Talks
● Creating VM Images for OpenNebula
● Cangallo!

3. OpenNebula Storage Model
Images DatastoreSystem Datastore
VM 1
VM 2
CentOS Image
Devuan Image
Ubuntu Image
Database Image Pers
Virtual Router Image
Empty FS Image
VM-1 disk.0
VM-1 disk.1
VM-2 disk.0
VM-2 disk.1
Link

4. OpenNebula Storage Model
A Simplified View (!):
● Images reside in Images Datastore and are copied to
System Datastore
● VMs use images in System Datastore as disks
● Persistent Images are not copied but a link is created
instead. The image in Images Datastore is modified
● Volatile disks can be created in the System Datastore
● Disk files have the names disk.0, disk.1, etc...

5. Image Datastore Drivers
● The drivers reside in
/var/lib/one/remotes/datastore
● These drivers manage monitoring, creation, deletion,
cloning... actions of images
○ clone: create an copy of an image
○ cp: add an image to the datastore
○ export: export to a marketplace
○ mkfs: create an empty image
○ monitor: get info about the datastore
○ rm: delete an image
○ snap_delete/flatten/revert: manage persistent
snapshots
○ stat: get info about an image that is going to be
imported

6. Image Datastore Drivers: fs
Add Image
● stat: get image size
● cp: copy image into datastore and retrieve ID

7. Transfer Drivers
● They manage system datastore
● Clone Images to System DS
● Copy images back to Images DS
● Creates contextualization CDROM
● Reside in /var/lib/one/remotes/tm:
○ clone: copy image from Images DS to System DS
○ context: generate context CD
○ cpds: copy disk to Images DS
○ ln: link a persistent Image to System DS

8. Transfer Drivers: VM Lifecycle
● VMs have Prolog and Epilog stages
● Prolog and Epilog scripts found in
/var/lib/one/vms/<id>
# cat transfer.0.prolog
CLONE qcow2 ursa:/var/lib/one//datastores/1/9c1f5c33cd858e3ae2ab07fa3b41c914
ursa:/var/lib/one//datastores/0/999/disk.0 999 1
CONTEXT qcow2 /var/lib/one/vms/999/context.sh /var/lib/one/vms/999/token.txt
ursa:/var/lib/one//datastores/0/999/disk.1 999 0
# cat transfer.0.epilog
DELETE qcow2 ursa:/var/lib/one//datastores/0/999/disk.0 999 1
DELETE qcow2 ursa:/var/lib/one//datastores/0/999 999 0

9. Transfer Drivers: VM Lifecycle
Some VM actions have trigger a TM action. Check
/usr/lib/one/mads/one_vmm_exec.rb. Attach disk:
steps = [
# Perform a PROLOG on the disk
{
:driver => :tm,
:action => :tm_attach,
:parameters => tm_command.split
},
# Run the attach vmm script
{
:driver => :vmm,
:action => :attach_disk,
:parameters => [
:deploy_id,
:disk_target_path,
target,
target_index,
drv_message
],
:fail_actions => [
{
:driver => :tm,
:action => :tm_detach,
:parameters => tm_rollback.split
}
]
}
]

10. Transfer Manager: QCOW2
● QCOW2 format can create overlays (linked clones)
● This is used in QCOW2 driver
● Makes cloning faster and occupies less disk space
● It’s slower afterwards as it adds another indirection
● Enables live disk snapshotting
● Assumes a shared filesystem for both Images and System
datastores

11. Transfer Manager: QCOW2 Clone
● A new image is created in system datastore:
qemu-img create -b <original> -f qcow2 <disk>
● A new image is created in <disk> that contains only
metadata. It’s base image is <original>
● When reading it checks if the block is in <disk> image, if
it’s not there searches in <original>. This can be
chained.
● When writing it checks if the block is already in <disk>, if
not a new block is allocated, copies the block from
<original> and writes there

12. Transfer Manager: QCOW2 Snapshot
● Instead of cloning the image directly in disk.0 a directory
is created with the snapshots and a link is made to point
to the actual snapshot:
disk.0 -> /var/lib/one/datastores/0/6363/disk.0.snap/0
disk.0.snap/
disk.0.snap/0 (base image in Images Datastore)
● New snapshots with an external snapshot to snaps dir
and the link updated:
disk.0 -> /var/lib/one/datastores/0/6363/disk.0.snap/1
disk.0.snap/
disk.0.snap/0 (base image in Images Datastore)
disk.0.snap/1 (base image snapshot 0)

13. Lightning Talks
OpenNebulaConf 2016
4th
edition

14. Creating Images for OpenNebula
OpenNebulaConf 2016
4th
edition

15. Creating Images for OpenNebula
VM images need to be prepared to support OpenNebula
features:
● context packages: automatic configuration
● ruby package: onegate dependency
● growpart: enable root partition resizing
● disable serial console: OpenNebula does not configure
it and can cause problems
● open-vm-tools package: for VMware machines
● qemu guestfs package: enable disk freeze for live
snapshots

16. qcow2 Format
$ qemu-img create -f qcow2 image.qcow2 10G
Metadata
Cluster Map
Cluster Data

17. qcow2 Image With Parent
$ qemu-img create -f qcow2 -o
backing_file=base.qcow2 image.qcow2
Metadata
Cluster Map
Cluster Data
Metadata
Cluster Map
Cluster Data
Parent

18. ● There are other ways to create your images:
○ virt-install
○ packer.io
○ foreman
○ etc...
● Sometimes modifying already created images is
convenient
● Even if you use other image formats you can
convert them

19. Mount Image
● Convert to raw and use mount -o loop
○ mount -o loop,offset=32256 image.raw /mnt
● Convert to raw and use losetup
○ losetup /dev/loop0 image.raw
○ kpartx -a /dev/loop0
○ mount /dev/loop0p1 /mnt
● Use nbd
○ modprobe nbd
○ qemu-nbd -c /dev/nbd0 image.qcow2
○ mount /dev/nbd0p1 /mnt

20. libguestfs
From its webpage http://libguestfs.org:
libguestfs is a set of tools for accessing and modifying
virtual machine (VM) disk images. You can use this for
viewing and editing files inside guests, scripting
changes to VMs, monitoring disk used/free statistics,
creating guests, P2V, V2V, performing backups, cloning
VMs, building VMs, formatting disks, resizing disks, and
much more.

21. guestfish - Read or Edit Files
$ guestfish -ia image.qcow2
><fs> cat /var/log/service/error.log
><fs> vi /etc/service.conf
$ guestfish -ia image.qcow2 <<EOF
upload service.conf /etc/service.conf
tar-in ssh-configuration.tar /etc/sshd
EOF
$ guestfish --ro -i -c qemu:///system -d vm-name
><fs> cat /var/log/service/error.log

22. virt-customize
● Starts custom VM and attach disks and connects to
network
● Change passwords, create users
● Move files
● Install packages
● Execute scripts
virt-customize [--options]
[ -d domname | -a disk.img [-a disk.img ...] ] [--chmod PERMISSIONS:FILE] [--commands-from-file FILENAME]
[--copy SOURCE:DEST] [--copy-in LOCALPATH:REMOTEDIR] [--delete PATH] [--edit FILE:EXPR] [--firstboot SCRIPT]
[--firstboot-command 'CMD+ARGS'] [--firstboot-install PKG,PKG..] [--hostname HOSTNAME] [--install PKG,PKG..]
[--link TARGET:LINK[:LINK..]] [--mkdir DIR] [--move SOURCE:DEST] [--password USER:SELECTOR] [--root-password SELECTOR]
[--run SCRIPT] [--run-command 'CMD+ARGS'] [--scrub FILE] [--sm-attach SELECTOR] [--sm-register] [--sm-remove]
[--sm-unregister] [--ssh-inject USER[:SELECTOR]] [--truncate FILE] [--truncate-recursive PATH] [--timezone TIMEZONE] [--touch FILE]
[--update] [--upload FILE:DEST] [--write FILE:CONTENT] [--no-logfile] [--password-crypto md5|sha256|sha512]
[--selinux-relabel] [--sm-credentials SELECTOR]

23. OpenNebula Marketplace Images
● Download CentOS images
● Create CDROM with OpenNebula context packages
● Create script to modify the image
○ Mount CDROM
○ Install context packages
○ Remove cloud-init and NetworkManager packages
○ Install EPEL repository
○ Install growpart packages

24. OpenNebula Images - Create CDROM
# Download context packages from github
curl -s
https://api.github.com/repos/OpenNebula/addon-
context-linux/releases | jq -r
'.[0].assets[].browser_download_url' | xargs
-L1 wget -P repo
# Create ISO image with label “EXTRA”
genisoimage -o extra-packages.iso -R -J -V
EXTRA repo/

25. OpenNebula Images - Prepare Script
mkdir /tmp/mount
mount LABEL=EXTRA /tmp/mount
# Install opennebula context package
rpm -Uvh /tmp/mount/one-context*rpm
# Remove cloud-init and NetworkManager
yum remove -y NetworkManager cloud-init
# Install growpart and upgrade util-linux
yum install -y epel-release --nogpgcheck
yum install -y cloud-utils-growpart --nogpgcheck
yum upgrade -y util-linux --nogpgcheck

26. OpenNebula Images - Calling virt-customize
# Create an overlay to preserve original image
$ qemu-img create -f qcow2 -b $orig $image
# Run customization
$ virt-customize --attach $ISO_IMAGE --run
$script --format qcow2 -v -a $image
--root-password disabled

27. Optimizing Images
● qemu-img does not know anything about filesystems
● Blocks not allocated (sparse files) or that contain zeroes
are not copied
● Normal file deletion does not zero or deallocate blocks
● Swap partitions contain information if used
● This information can be stripped to make the images
smaller
● virt-sparsify to the rescue!

28. Optimizing Images - virt-sparsify
There are two ways of doing sparsification:
● Normal Sparsification:
○ Occupies the maximum space of the image
● In Place Sparsification:
○ Create an sparse qcow2 file

29. Optimizing Images - Normal Sparsification
● Create overlay of the image
● Create a file in all filesystems and fill it with zeroes until
there is not more space and delete file
● Fill swap partitions with zeroes
● Convert it to a new qcow2 file skipping zero blocks
$ TMPDIR=/var/tmp virt-sparsify original.qcow2
new.qcow2

30. Optimizing Images - In Place Sparsification
● Uses trim command, normally used for SSD disks
● Deallocates blocks from filesystem
● Does not require the total amount of disk space
● The qcow2 file contains holes and is not the best one
for distribution
● Can be converted to a non sparse qcow2 file
● Can not be used with compression
$ virt-sparsify --in-place original.qcow2
new.qcow2

31. Optimizing Images - Compression
● qcow2 images can have the blocks compressed
● Compression rate is less that xz or bzip2
● Is more convenient as it can be directly used as is
● Use of these images trades disk space for CPU
consumption
● Can be done directly in virt-sparsify with
--compress (not In Place)

32. qemu-img tips
● There are two qcow2 file formats, pre version 0.10
and newer
○ CentOS 6 does not support post 0.10 version
○ On conversion or creation it can be specified with -o
compat=0.10
● qemu-img < 2.4 does not support creation of delta
images with compression
○ This tool can be easily compiled manually
○ Download qemu 2.4 code
○ ./configure
○ make qemu-img

33. Cangallo!
OpenNebulaConf 2016
4th
edition

34. Cangallo (pronounced canga-io) is a command-line tool
written in ruby, that uses qemu-img and libguestfs to manage,
create and organize qcow2 images. Its repository holds
images and deltas of derived images in a similar way as
Docker but in a block level instead of file level.
https://canga.io
Cangallo!

35. Cangallo: Repository
In Cangallo images are registered into a repository:
$ canga list
NAME SIZE DAYS_AGO
default:centos7 370.2 Mb 134.2
default:debian8 455.8 Mb 134.2
default:ubuntu1404 247.6 Mb 134.2
default:ubuntu1604 290.1 Mb 134.2
^ default:centos7/one 47.7 Mb 134.1
OpenNebula Compatible CentOS 7
^ default:0b5e9704c8e898ee 51.0 Mb 134.1
OpenNebula Compatible Ubuntu 14.04
^ default:0eb98b0f62e528e6 78.0 Mb 134.1
OpenNebula Compatible Ubuntu 16.04
default:centos7-1606 354.2 Mb 101.4
^ default:centos7-1606/one 50.1 Mb 92.3
OpenNebula Compatible CentOS 7
^ default:centos7-1606/one/vmware 69.7 Mb 59.3
OpenNebula Compatible CentOS 7
^ default:debian7/one 69.6 Mb 4.0
OpenNebula Compatible Debian 8
^ default:ubuntu1404/one 51.8 Mb 4.0
OpenNebula Compatible Ubuntu 14.04
^ default:ubuntu1604/one 80.1 Mb 4.0
OpenNebula Compatible Ubuntu 16.04
default:centos7-1608 137.9 Mb 4.0
^ default:centos7-1608/one 57.1 Mb 4.0
OpenNebula Compatible CentOS 7

36. Cangallo: Cangafiles
Cangafiles are small yaml files that contain instructions on
how to create a derived image.
$ cat cangafiles/centos7.canga
# vim:tabstop=2:sw=2:et:syntax=yaml:
---
description: OpenNebula Compatible CentOS 7
os: CentOS 7
parent: centos7-1608
tag: centos7-1608/one
tasks:
- copy: context /
- run:
- rpm -Uvh /context/one-context*rpm
- yum remove -y NetworkManager cloud-init
- yum install -y epel-release cloud-utils-growpart ruby
--nogpgcheck
- yum install -y util-linux --nogpgcheck
- delete: /context

37. Cangallo: Build Image
$ canga build cangafiles/centos7.canga
[ 0.0] Examining the guest ...
[ 7.0] Setting a random seed
[ 7.0] Copying: context to /
[ 7.0] Running: rpm -Uvh /context/one-context*rpm
[ 7.6] Running: yum remove -y NetworkManager cloud-init
[ 11.1] Running: yum install -y epel-release cloud-utils-growpart ruby --nogpgcheck
[ 25.8] Running: yum install -y util-linux --nogpgcheck
[ 30.8] Running: rm -rf /context
[ 31.4] Finishing off
[ 3.1] Trimming /dev/sda1
[ 3.5] Sparsify in-place operation completed with no errors
Calculating image sha256 with libguestfs (it will take some time)
Image SHA256: 7da5deee65885250577807e50f18d727f6a1147957aac5b5783bc724df9bde4c
Copying file to repository
Deleting temporary image

38. Cangallo: Use Images
● For testing in your laptop you can use overlays (empty
images with the repository as parent):
$ canga overlay centos7/one centos7.qcow2
● Export to an independent image
$ canga export centos7/one centos7.qcow2 -c
$ canga export centos7/one centos7.vmdk -f vmdk

39. Cangallo: Remote Repositories
● Repositories consist of qcow2 images, an index file
(yaml) and a signature
● The repositories can be exposed to the world using an
https server
● Cangallo is able to download the index and images
from the repository
$ cat ~/.cangallo/config.yaml
default_repo: default
repos:
default:
type: local
path: /home/jfontan/projects/cangallo/repo
remote:
type: remote
path: ~/.cangallo/remote
url: http://localhost:8000

40. OpenNebulaConf 2016
4th
edition
Platinum
Gold
Silver
Community
THANKS!