Presentation by Ignacio Blanquer, Professor of the Polytechnic University of Valencia, Spain at Cloudscape Brazil 2017 & WCN 2017

1. Lightening the burden of cloud resources
administration: from VMs to Functions
Ignacio Blanquer
Universitat Politècnica de València
Spain
iblanque@dsic.upv.es

2. Some hidden costs in IaaS
• Did Cloud Removed Sys admin Cost? Not really, there are several tasks
you do not get rid of them
• Managing and Customizing Virtual Machine Images.
• Patching them regularly.
• Configuration and contextualization of multi-VM infrastructures.
• Is cloud interoperable? Despite that there are services that are quite
platform-dependent, there should be a way to ease migration
• Compatible VMI formats.
• DevOps systems.
• Portable cloud specifications.

3. DevOps (Development Operations)
• DevOps focus on improving in terms of time, cost, reliability, security
and scalability the life cycle of application development & delivery.
• In the Cloud, large effort has been invested on cloud services
deployment automation.
• Prevent from maintaining multiple Virtual Machine Images for multiple
configurations and platforms, and concentrate on basic Vanilla VMIs and
installation recipes.
• Define and automatically install
software dependencies and
configure multi-VM
infrastructures dynamically.

4. Elastic Compute Clusters in the Cloud
(EC3 – www.grycap.upv.es/ec3)
• EC3 is a tool built up on top of Infrastructure Manager (IM -
www.grycap.upv.es/im) and Cluster Energy Savings (CLUES -
www.grycap.upv.es/clues).
• With EC3 you can deploy a self-managed Mesos Cluster on a wide range of IaaS
• Nodes are deployed on demand, then
reconfigured automatically.
• Initial deployment only starts the Front-End.
• Once a Working Node has been deployed,
a VMI is automatically registered to speed-up
the deployment of new WNs.
• Exposing a Web interface, it does not require
special cloud expertise.

5. OASIS TOSCA - Topology and
Orchestration Specification for Cloud
• TOSCA uses a Domain Specific Language to define interoperable
descriptions of cloud applications & services, infrastructure,
relationships, requirements and configurations
(*) https://github.com/indigo-dc/tosca-types/blob/master/examples/eubiosteo_server.yaml
Requires two other nodes:
- Frontend
- Worker nodes
elastic_clu
ster_front
_end
slurm_fro
nt_end
slurm_ser
ver
tosca.nodes.indi
go.ElasticCluster
tosca.nodes.in
digo.LRMS.Fr
ontEnd.Slurm
tosca.nodes
.indigo.Com
pute
wn_node
tosca.nodes.indig
o.LRMS.WorkerN
ode.Slurm
slurm_wn
tosca.nodes
.indigo.Com
pute

6. INDIGO-DATACLOUD
ORCHESTRATOR
• INDIGO-DC is an H2020 project of 11.1M€
ending on Sept. 2017 with 26 EU partners.
• It aims at developing an open source Cloud
platform for computing and data for science.
• The second and final major software
release, called ElectricIndigo, is available
http://repo.indigo- datacloud.eu/index.html
• INDIGO-DC orchestrator provides an entry
point for deploying virtual infrastructures
described as TOSCA templates.
6
Deployment of Customized Virtual
Infrastructures using INDIGO-DataCloud
Orchestrator
Service
Brokering/
Policy Service
QoS/SLA
Service
Managed Services/
Applications (MSA)
Service
Monitoring
Service
Data
Management
Services
Infrastructure
Manager
TOSCA
IAM Service
GridHPC
Clusters
SAGA
POSIX /
WebDavSAGA
POSIX /
WebDav
EUDAT
CDMI
GUI-based
Portlets
Repository
TOSCA-
compliant
Templates
1. Access
2. Authenticate
3. Select /
Customize
9.a Delegate
Deploy
5. Status
6. Prioritize
7. Agree
8.a Transfer
(Optional)
8.b. Transfer
(Optional)
4. Deploy
9.b.1
Delegate
Deploy
External
Cloud
Cloud APIs CDMI
Partner IaaS
OpenNebula
IM
CDMI
OpenStack
HEAT
TOSCA
9.b.2
Provision &
Configure
...
Virtual Infrastructure for
User Application/Service
10. Deliver VMs
...

7. Now everything is container-based
• Docker containers are lightweight and do not add a separate Operating
System
• Immediate, small footprint in disk and memory, persistent, easy to create and
manage…
• Docker containers are a very effective technic for application delivery.
• However isolation is smaller with respect to VMs
• They run as privileged processes and host is
more sensitive to malfunctions.
• No multitenancy.
• Other solutions are available
• LXC/D, rkt, uDocker.

8. Container Management Platforms
• Simple solutions: Docker Swarm
• Same interface as Docker, but managing a pool of Docker resources that can be
accessed remotely.
• Cloud-style solution: Docker or LXC as prime-class hypervisors
• ONEDock implements the whole API of a OpenNebula hypervisor driver so it can
run Docker Containers as they were ONE VMs.
• Magnum OpenStack.
• Native container management systems
• Kubernetes, Apache Mesos, Rancher, OpenShift,…

9. Apache Mesos in BIGSEA
• BIGSEA WP3 services deploy a self-managed Mesos
framework with Monitoring.
• Lemonade Spark applications, and COMPSs can run
as frameworks.
• Containerized jobs are executed through Chronos
and Marathon
• Horizontal elasticity through EC3.
• Vertical elasticity through proactive policies
• CPU Cap actuation
• Framework resource allocation update.
• Docker networks, DNS indexing, high-availability.

10. But still we need to provision an
infrastructure, even in public clouds.
• The illusion of instant elasticity is improved with Container
Management systems meanwhile you have powered-on resources.
• Still you need to provision the resources where containers will run…
• Amazon Lambda, Google Cloud Functions, Azure Functions, IBM
Whiskers are providing a serverless programming model
• You just write the functions and the triggers.
• No upfront (even virtual) infrastructure provisioning.
• Stateless, and tailored for short, massive executions.
• But limited to specific workloads
• Basic configurations and sometimes programming language limitations.

11. A new approach - Serverless Container-aware
Architectures (SCAR - https://github.com/grycap/scar)
• SCAR is a framework to transparently execute containers in serverless platforms (e.g. AWS
Lambda).
• This can be used to create highly-parallel event-driven file-processing serverless applications
that execute on customized runtime environments provided by Docker containers on AWS
Lambda.
• SCAR provides a CLI to create a Lambda function to execute a container out of a Docker image
stored in Docker Hub.
• SCAR uses udocker (https://github.com/indigo-dc/udocker/) to execute
Docker containers in user space.
• Currently limited for the restrictions of Lambda in storage (512 MB and
the execution time (5 minutes).

12. ADDRESSING BIG DATA CHALLENGES ON THE
CLOUD WITH QUALITY OF SERVICE
Build massive parallel data analytic applications
over self-managed elastic processing service.
Ignacio Blanquer – iblanque@dsic.upv.es
Wagner Meira Jr. - meira@dcc.ufmg.br
programming
models
QoS
deadlines
privacy
composition
OLAP
databases
smart
cities
www.eubra-bigsea.eu | @bigsea_eubr