Introducing Cloud Development with Mantl
•Transferir como PPTX, PDF•
1 gostou•427 visualizações
Speaker Brian Hicks and Ryan Eschinger
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a Introducing Cloud Development with Mantl
Semelhante a Introducing Cloud Development with Mantl (20)
Mais de Cisco DevNet
Mais de Cisco DevNet (20)
Último
Último (20)
Introducing Cloud Development with Mantl
- 1. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID We’re ready. Are you?
- 2. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Introducing Cloud Development with Mantl Brian Hicks (Mantl Core Committer) Ryan Eschinger (Mantl Core Committer)
- 3. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID • Introduction • Mesos • Frameworks • Docker • Consul • Mantl Agenda 3
- 4. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Introduction 4 Brian Hicks Ryan Eschinger
- 5. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mesos What is it? Program against your datacenter like it’s a single pool of resources Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. 5
- 6. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mesos 2-Tier Scheduler Frameworks: • Receive resources offered by the master nodes • Schedule work on offered resources • Are notified if workloads fail via internal and external mechanisms 6 Mesos: • Determines resources available on nodes in the cluster • Offers those resources to frameworks • Keep track of node status (online, offline, remaining resources)
- 7. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mesos Architecture Slaves (Agents in 1.0): • Tell the masters which resources are available • Run tasks on claimed resources 7 Masters: • Communicate with Schedulers • Keep track of Agents • Provide a UI for Ops / DevOps
- 8. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mesos Architecture Executor: • A binary or script that runs on the slave to perform work • Runs in an isolated environment (can use Docker) 8 Framework (Scheduler): • Schedules work based on offers received • Domain specific
- 9. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Framework Demos 9
- 10. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Marathon What is it? 1 0 A cluster-wide init and control system for services in cgroups and Docker containers Marathon is an Apache Mesos framework for long- running applications. Given that you have Mesos running as the kernel for your datacenter, Marathon is the init or upstart daemon.
- 11. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Marathon Overview 1 1 • Developed by Mesosphere • Controls app resources, ports, environment variables, and scaling • Runs “long-running” tasks
- 12. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Cassandra What is it? Distributed database designed for scalability and high availability Apache Cassandra is an open source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. 1 2
- 13. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Cassandra Overview 1 3 • Designed to handle big data workloads across multiple nodes (ring) with no single point of failure • Data is distributed among all nodes in the cluster • Uses the Gossip protocol for peer-to-peer communication • Eventually consistent (C): prioritizes Availability and Partitioning tolerance (AP)
- 14. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Cassandra Mesos Framework 1 4 • Automates bootstrapping and operations of Cassandra clusters • Bootstraps initial seed nodes and expands cluster to desired number of nodes • Runs periodic operational tasks • Restart, remove, and replace nodes on failure • Self-contained: does not require specialized software installed on agent nodes
- 15. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Spark What is it? A fast and general engine for large-scale data processing Apache Spark is an open source, high performance big data processing framework. It provides high-level APIs in Java, Scala, Python, and R. Spark includes higher-level libraries, including support for SQL queries, streaming data, machine learning and graph processing. 1 5
- 16. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Spark Overview 1 6 • Originally developed at UC Berkeley in 2009 • Up to 100x faster than Hadoop for large scale data processing • Execution engine works both in-memory and on-disk • Architecture • Data storage: supports Hadoop-compatible data sources like HDFS, HBase, Cassandra, etc. • API: Scala, Java, Python, and R. • Resource Management: standalone, Mesos, Yarn
- 17. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Spark on Mesos 1 7 • Runs as a framework • Supports the Docker containerizer • 2-level scheduling: Spark can make decisions about the offers it receives from Mesos • Enables sophisticated scheduling scenarios
- 18. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Spark Run Modes 1 8 • Coarse-grained • Runs long-running Spark executors on every node • Fast startup, better for interactive sessions, but can be inefficient • Fine Grained • Launches a Spark executor per task • Slower startup but can utilize resources better
- 19. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Docker What is it? A shipping container system for applications A container virtualization platform that abstracts underlying Linux resource isolation technologies like cgroups and kernel namespaces. Allows developers to build, package, and ship applications that can run anywhere*. *anywhere that runs a 3.10+ Linux kernel 1 9
- 20. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Docker Overview 2 0 • Compared to virtual machines, containers are smaller, lighter weight, more portable, and easier to deploy • Package an application, along with all of its dependencies, in a single artifact • Use the same artifact throughout development, testing, and production • Reduces concerns about the compatibility in different environments
- 21. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul 2 1
- 22. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul Discovery (HTTP) 2 2 curl localhost:8500/v1/catalog/service/marathon [{ "Address": "10.0.113.214", "Node": "mantl-control-01", "ServiceAddress": "", "ServiceID": "marathon", "ServiceName": "marathon", "ServicePort": 18080, "ServiceTags": ["marathon"] }]
- 23. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul Discovery (DNS) 2 3 dig +short marathon.service.consul 10.0.113.214 10.0.220.54 10.0.131.25 dig +short marathon.service.consul SRV 1 1 18080 mantl-control-02.node.mantl.consul 1 1 18080 mantl-control-01.node.mantl.consul 1 1 18080 mantl-control-03.node.mantl-consul
- 24. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul Key/Value Store 2 4 curl -X PUT -d world http://localhost:8500/v1/kv/hello true
- 25. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul Key/Value Store 2 5 curl -X GET http://localhost:8500/v1/kv/hello [{ "CreateIndex": 223, "Flags": 0, "Key": "hello", "LockIndex": 0, "ModifyIndex": 223, "Value": "d29ybGQ=" }]
- 26. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul consul-template 2 6 global maxconn {{or (key "service/haproxy/maxconn") 256}} debug {{range services}} frontend {{.Name}} bind *:{{key (printf "ports/%s" .Name)}} {{range service .Name}} server {{.Node}} {{.Address}}:{{.Port}}{{end}} {{end}}
- 27. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mantl 2 7
- 28. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Thank you © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID 28
- 29. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Notas do Editor
- Cassandra is a distributed database that was designed for fault-tolerance, high availability, and scalability. It is an open source project that was initially created by Facebook to power their inbox search feature. It became an Apache project in 2009.
- Cassandra was designed to support large amounts of data across any number of servers In a Cassandra ring, data is replicated across multiple nodes so that there is no single point of failure data is distributed in such a way that each node contains a different set of data Every node in a Cassandra ring has the same role There is no concept of a Master And any node can respond to any request Assuming sufficient redundancy, failed nodes can be replaced without downtime or interruption of service New nodes can also be added as needed without interruption Adding nodes can improve both read-and-write throughput Cassandra uses the Gossip protocol for inter-node communication. There is no central broadcaster but information is eventually transferred to all nodes The CAP theorem says that out of Consistency, Availability, and Partition Tolerance, you can only choose 2. In general, Cassandra prioritizes Availability and Partition Tolerance with an eventually Consistent model It does actually support tunable consistency so there are some knobs and levers you can use to tune it
- The Cassandra Mesos framework automates bootstrapping and (some) operational tasks for Cassandra clusters A minimum of one seed node must be running and healthy Then it adds additional nodes until it reaches the desired capacity You can configure the number of nodes and the amount of resources, such as CPU and memory, that are needed The framework automates some operational tasks such as running ‘nodetool repair’ The framework has the ability to restart, remove, and replace nodes on failure. Finally, the framework is self-contained You do not have to pre-install or configure anything on your agent or slave nodes
- Spark is a data processing platform that is designed to be fast and general-purpose It is often used a replacement for the Hadoop MapReduce model but also supports other workloads like batch applications interactive SQL queries streaming data and machine learning
- Spark was originally developed at UC Berkely in 2009 it actually was built as a way to show how easy it is to build a framework on top of Mesos One of Spark’s biggest benefits is its ability to run computations in-memory as well as on-disk At a high level, Spark’s architecture is made up of 3 components: Data storage API Resource management Spark supports creating distributed datasets from any Hadoop-compatible data source like HDFS HBase S3 Cassandra Spark’s functionality is exposed in simple APIs in Python, Scala, Java, and R Spark is designed to scale up as needed and supports a few different cluster managers including a standalone Spark scheduler Yarn and, of course, Mesos
- Spark runs as a Mesos framework One of the huge benefits of running Spark on Mesos is that it allows you to share your cluster with other applications and services. You do not have to build a dedicated cluster just for your Spark tasks Multiple frameworks can coexist on the same cluster The Spark framework supports the Docker containerizer. This means you can run Spark tasks from a docker container Otherwise, you would need to either package Spark and host it in HDFS, or at an accessible http or s3 url, or you would need to pre-install Spark on all of your mesos agents 2-level scheduling in Mesos allows for sophisticated scheduling scenarios When Mesos makes a resource offer to Spark, Spark is able to analyze it and decide whether or not to accept the resources If Spark has no tasks to run, it will reject the offer If it does have a task to run and the resource offer is sufficient, it can accept the offer and launch its task
- The Spark framework supports two different run modes Coarse-grained Fine-grained In Coarse-grained mode, the framework launches long-running Spark executors on Mesos agent nodes This allows new tasks to start very quickly since there is already an executor running Coarse-grained mode is great for interactive Spark sessions Coarse-grained mode can be less efficient Resources are tied up by the long running executors even when no tasks are running In fine-grained mode, a new executor is launched per task this obviously increases the time it takes to launch a new task however, this can be a more efficient use of resources The resources will only be used while tasks are running If no tasks are running, those resources are available for other frameworks to use
- Docker is a container virtualization platform It allows you to package an application, along with all of its dependencies, into an isolated container that you can run on any relatively modern linux system It achieves isolation by taking advantage of underlying linux technologies like cgroups and namespaces
- Containers are often compared to virtual machines They are smaller, more lightweight, more portable, and easier to deploy. Docker can simplify the development lifecycle by allowing you to use the same artifact, the container, during development, in your continuous integration and testing pipeline, and on your production systems. By having your application and all of its dependencies packaged in a container, you have much less risk of running into compatibility problems as you move your application from development to testing to production While Docker is not required in a Mesos environment, it can really simplify things as your agent nodes won’t need a lot of specialized software installed. Let’s say you were going to deploy a Ruby app on your mesos cluster, If you were not using the Docker containerizer, you would have to ensure that all of your agents had the right version of ruby installed and configured for your app If you had multiple applications that required different versions of ruby... well, you probably see where this is going. it can get really painful, really quickly By using Docker containers, you wouldn’t even need to install ruby on your mesos agents as that dependency would be isolated in the container you deploy