Downtime is not an option - day 2 operations - Jörg Schad

1
Downtime is not an option
How Fast Data and Microservices change the datacenter
@joerg_schad @dcos

© 2017 Mesosphere, Inc. All Rights Reserved. 2
Jörg Schad
Distributed Systems Engineer
@joerg_schad

In the beginning
there was a big
Monolith

© 2017 Mesosphere, Inc. All Rights Reserved.
Hardware
Operating System
Application
5
COMPUTERS

noun | ˈmīkrō/ /ˈsərvəs/ :
an approach to application development in which a
large application is built as a suite of modular services.
Each module supports a specific business goal and
uses a simple, well-defined interface to communicate
with other modules.*
Microservices are designed to be flexible, resilient,
efficient, robust, and individually scalable.
*From whatis.com
OVERVIEW

Operating
System
Operating
System
Operating
System
ServiceApp ServiceServiceAppApp
7
MICROSERVICES
- Polyglot
- Single Responsibility
- Smaller Teams
- Utilization
- Machine
types/groups
- Dependency hell
Machine
Infrastructure
Machine Machine
ServiceService ServiceServiceServiceService

© Gerard Julien/AFP
Run everything in containers!

OS
9
CONTAINERS
- Rapid deployment
- Dependency
vendoring
- Container image
repositories
- Spreadsheet
scheduling
OS OS
Machine
Infrastructure
Machine Machine
Container Runtime Container Runtime Container Runtime

CONTAINER
SCHEDULING
RESOURCE
MANAGEMENT
SERVICE
MANAGEMENT
- Load Balancing
- Readiness Checking
CONTAINER ORCHESTRATION

CONTAINER
SCHEDULING
- Placement
- Replication/Scaling
- Resurrection
- Rescheduling
- Rolling Deployment
- Upgrades
- Downgrades
- Collocation
RESOURCE
MANAGEMENT
- Memory
- CPU
- GPU
- Volumes
- Ports
- IPs
- Images/Artifacts
SERVICE
MANAGEMENT
- Labels
- Groups/Namespaces
- Dependencies
- Load Balancing
- Readiness Checking
CONTAINER ORCHESTRATION

Orchestration
12
Machine Infrastructure
Web Apps & Services
Scheduling
Resource Management
Container Runtime
Machine & OS
Service Management
CONTAINER
ORCHESTRATION
Machine & OS Machine & OS
Container Runtime Container Runtime

MapReduce is
crunching Data
Meanwhile...

But then business
demanded
FAST DATA
We need to turn faster!

Fast Data
Batch Event ProcessingMicro-Batch
Days Hours Minutes Seconds Microseconds
Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics
Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations

The SMACK Stack
EVENTS
Ubiquitous data streams
from connected devices
INGEST
Apache Kafka
STORE
Apache Spark
ANALYZE
Apache Cassandra
ACT
Akka
Ingest millions of events
per second
Distributed & highly scalable
database
Real-time and batch
process data
Visualize data and build
data driven applications
Mesos/ DC/OS
Sensors
Devices
Clients

Datacenter

NAIVE APPROACH
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Industry Average
12-15% utilization
mySQL
microservice
Cassandra
Spark/Hadoop
Kafka

MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Mesos/ DC/OS
automated schedulers, workload multiplexing onto the
same machines
mySQL
microservice
Cassandra
Spark/Hadoop
Kafka

• A top-level Apache project
• A cluster resource
negotiator
• Scalable to 10,000s of
nodes
• Fault-tolerant, battle-tested
• An SDK for distributed apps
• Native Docker support
21
Apache Mesos

Datacenter Operating System (DC/OS)
Distributed Systems Kernel (Mesos)
DC/OS ENABLES MODERN DISTRIBUTED APPS
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions &
Logic
Search
Time Series
SQL / NoSQL
Databases
Modern App Components
Any Infrastructure (Physical, Virtual, Cloud)
23

24
THE
BASICS
DC/OS is …
● 100% open source (ASL2.0)
+ A big, diverse community
● An umbrella for ~30 OSS projects
+ Roadmap and designs
+ Docs and tutorials
● Not limited in any way
● Familiar, with more features
+ Networking, Security, CLI, UI,
Service Discovery, Load Balancing,
Packages, ...

Container Options
Enhancements to the Mesos Containerizer to
allow support launching specific container
formats (Docker, AppC, OCI (future), etc)
● Reduces need to maintain and update
multiple containerizers
● Support multiple container formats with
a single containerizer
Image provisioner component added to the
Mesos containerizer - responsible for pulling,
caching, and preparing container root
filesystems
Launcher Isolators
Universal containerizer
Provisioner
Process
management
Container
lifecycle hook
Container
image support

DEMO

GEO-ENABLED IoT

DATA FLOW

Keep it running!

Monitoring
- Collecting metrics
- Routing events
- Downstream processing
- Alerting
- Dashboards
- Storage (long-term retention)
Logging
- Scopes
- Local vs. Central
- Security considerations
DAY 2 OPERATIONS

Maintenance
- Cluster Upgrades
- Cluster Resizing
- Capacity Planning
- User & Package Management
- Networking Policies
- Auditing
- Backups & Disaster Recovery
Troubleshooting
- Debugging
- Services
- System
- Access?
- Tracing
- Chaos Engineering
DAY 2 OPERATIONS

Troubleshooting
● Services: typically specific to service, use logging (for
example, dcos task log) and dcos node ssh for
per-node investigations
● dcos task exec
○ Permissions?
● System:
○ Simple diagnostics via dcos node diagnostics
○ Comprehensive dump via clump

THANK YOU!
ANY QUESTIONS?
@dcos
users@dcos.io
/groups/8295652
/dcos
/dcos/examples
/dcos/demos
chat.dcos.io

Failures
Framework
Scheduler
Executor
Task
Agent
LEADER STANDBY STANDBY
ZK
ZK
ZK
Executor
Task
Agent

Distributed Systems could be so easy...
35
1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.
*)
https://en.wikipedia.org/wiki/Fallacies_of_distributed_comp
uting

Questions?
Code: https://git.io/vXUoy
http://
grnh.se/
ie76ru

Monitoring

METRICS
Measurements captured to determine health and performance of cluster
- How utilized is the cluster?
- Are resources being optimally used?
- Is the system performing better or worse over time?
- Are there bottlenecks in the system?
- What is the response time of applications?

DC/OS METRIC SOURCES
● Mesos metrics
○ Resource, frameworks, masters, agents,
tasks, system, events
● Container Metrics
○ CPU, mem, disk, network
● Application Metrics
○ QPS, latency, response time, hits, active
users, errors
OS
Mesos
Container ContainerContainer
App App App

Before upgrading
1. Make sure cluster is healthy!
2. Perform backup
a. ZK
b. Replicated logs
c. other state
3. Review release notes
4. Generate install bundle
a. Validate versions
UPGRADE PROCEDURE
Framework
Scheduler
Executor
Task
Agent
Executor
Task
Agent
ZK
ZK
ZK

MESOS MASTER METRICS
● Metrics for the master node are available at the following URL:
○ http://<mesos-master-ip>/mesos/master/metrics/snapshot
○ The response is a JSON object that contains metrics names and values as
key-value pairs.
● Metric Groups:
○ Resources
○ Master
○ System
○ Slaves
○ Frameworks
○ Tasks
○ Messages
○ Event Queue
○ Registrar

MESOS MASTER BASIC ALERTS
Metric Value Inference
master/uptime_secs is low The master has restarted
master/uptime_secs < 60 for sustained periods of time The cluster has a flapping master node
master/tasks_lost is increasing rapidly Tasks in the cluster are disappearing. Possible causes include
hardware failures, bugs in one of the frameworks or bugs in
Mesos
master/slaves_active is low Slaves are having trouble connecting to the master
master/cpus_percent > 0.9 for sustained periods of time DCOS Cluster CPU utilization is close to capacity
master/mem_percent > 0.9 for sustained periods of time DCOS Cluster Memory utilization is close to capacity
master/disk_used & master/disk_percent DCOS Disk space consumed by Reservations
master/elected is 0 for sustained periods of time No Master is currently elected

Operations
UPGRADES

1. Master rolling upgrade
a. Start with standby
b. Install new DC/OS
2. Agent rolling upgrade
3. Framework upgrades
UPGRADE PROCEDURE
Framework
Scheduler
Executor
Task
Agent
ZK
ZK
ZK
Executor
Task
Agent

a. Uninstall DC/OS
b. Install new DC/OS
UPGRADE PROCEDURE
Framework
Scheduler
Executor
Task
Agent
ZK
ZK
ZK
Executor
Task
Agent

a. Orthogonal to DC/OS
b. Ensure changes don’t
affect existing apps
UPGRADE PROCEDURE
Framework
Scheduler
Executor
Task
Agent
ZK
ZK
ZK
Executor
Task
Agent

Failure
Handling

Failure Handling
MESOS TASK
FAILURE

MESOS TASK FAILURE
ZK
MASTERMARATHON
CLIENT AGENT AGENT AGENT
EXECUTOR
TASK

Status Update
Status Update
EXECUTOR
MESOS TASK FAILURE
ZK
MASTERMARATHON
CLIENT AGENT AGENT
TASK
AGENT
SEGFAULT :(

MESOS TASK FAILURE
ZK
MASTERMARATHON
CLIENT AGENT AGENT
EXECUTOR
TASK
Launch Task
Launch Task
AGENT
Launch Task

EXECUTOR
Status Update
Status Update
MESOS TASK FAILURE
ZK
MASTERMARATHON
CLIENT AGENT AGENT
TASK
AGENT
Status Update

Failure Handling
MESOS AGENT
FAILURE

LOCAL AGENT FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK

LOCAL AGENT FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK
Re-register

Failure Handling
MESOS HOST
FAILURE

LOCAL AGENT FAILURE
ZK
MASTERMARATHON
CLIENT AGENT AGENT

LOCAL AGENT FAILURE
ZK
MASTERMARATHON
CLIENT AGENT AGENT
Status Update

MESOS TASK FAILURE
ZK
MASTERMARATHON
CLIENT AGENT AGENT
EXECUTOR
TASK
Launch Task
Launch Task
Launch Task
Resource Offer

MESOS TASK FAILURE
ZK
MASTERMARATHON
CLIENT AGENT AGENT
EXECUTOR
TASK
Status Update
Status Update
Status Update

Failure Handling
MESOS MASTER
FAILURE

MASTER FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK

MASTER FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK
Leading Master Leading Master
Leading Master

MASTER FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK
Reregister
Reregister
Reregister
Reregister

Reregistered
MASTER FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK
Reregistered
Reregistered
Reregistered

Failure Handling
SCHEDULER
FAILURE

SCHEDULER FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK

SCHEDULER FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK
Framework ID
Leading Master

SCHEDULER FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK
Reregister

SCHEDULER FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK
Reregistered

SCHEDULER FAILURE
ZK
MASTERMARATHON
EXECUTOR
TASK
Status Update
Reconcile Tasks

a. Start with standby
b. Uninstall DC/OS
c. Install new DC/OS
UPGRADE PROCEDURE
Framework
Scheduler
Executor
Task
Agent
ZK
ZK
ZK
Executor
Task
Agent

FUTURES (TBD)
Leverage maintenance primitives in Mesos to
drain host
Upgrade management through DC/OS to
perform rolling upgrades

Monitoring
- Routing events
- Alerting
- Dashboards
- Storage (long-term retention)
Logging
- Scopes
- Local vs. Central
DAY 2 OPERATIONS

Maintenance
- Cluster Upgrades
- Cluster Resizing
- Capacity Planning
- User & Package Management
- Auditing
- Backups & Disaster Recovery
Troubleshooting
- Debugging
- Services
- System
- Tracing
- Chaos Engineering
DAY 2 OPERATIONS

MONITORING

MONITORING
CONCEPT

MONITORING
TOOLING
EXAMPLES
● local scraping:
a. collectd
b. cAdvisor*
● event router:
a. fluentd
b. Flume
c. Kafka*
d. logstash*
e. Riemann
*) available via Mesosphere Universe

MONITORING
TOOLING
EXAMPLES
● storage:
a. Elasticsearch*
b. Graphite
c. InfluxDB*
d. KairosDB/Cassandra*
e. OpenTSDB/HBase
f. others such a local filesystem, Ceph FS*,
HDFS*, etc.

MONITORING
TOOLING
EXAMPLES
● dashboard:
a. D3
b. Grafana*
c. signal fx
● alerting:
a. BigPanda
b. PagerDuty
c. signal fx
d. VictorOps

MONITORING
TOOLING
EXAMPLES
(INTEGRATED)
● Amazon CloudWatch
● AppDynamics
● Azure Monitor
● Circonus
● DataDog*
● dcos/metrics
● Ganglia
● Google Stackdriver
● Hawkular
● Icinga
● Librato
● Nagios
● New Relic
● OpsGenie
● Pingdom
● Prometheus
● Ruxit Dynatrace*
● Sensu
● Sysdig*
● Zabbix

LOGGING

LOGGING
SCOPES

LOGGING
TOOLING
EXAMPLES
(PRIMITIVES) ● DC/OS logging overview
● Docker logging drivers
● systemd's journalctl

LOGGING
TOOLING
EXAMPLES
(INTEGRATED)
● Centralized app logging with fluentd
● DC/OS
a. ELK stack log shipping
b. Splunk
● Graylog
● Loggly
● Papertrail
● Sumo Logic

MAINTENANCE

Overview
● How to install a new version of X?
● When to scale what (service-level vs. nodes)
● Who gets to access/install which services in what way?
Upgrades
Sizing
User and package management
● What services can talk to each other and in which way?
● Who accessed what, when and how?
● How is the continuous operation of the cluster and the services accomplished?
What happens when cluster (or critical infra components like ZK) go down?
Networking
Auditing
Disaster Recovery

OTHER
TROUBLESHOOTING
TECHNIQUES
● Tracing
○ Idea: identify latency issues and perform
root-cause analysis in a distributed setup
○ OpenTracing
● Chaos Engineering
○ Idea: proactively break (parts of) the system to
understand how it reacts
○ Chaos Monkey
○ DRAX

ARCHITECTURE
MESOS FUNDAMENTALS
● Agents advertise resources to Master
● Master offers resources to Framework
● Framework rejects/uses resources
● Agents report task status to Master

ARCHITECTURE
MESOS FUNDAMENTALS

Questions?
Code: https://git.io/vXUoyPsssssssst …
… we are hiring!
http://
grnh.se/
ie76ru

CONTAINER SCHEDULING
106

RESOURCE MANAGEMENT
107

SERVICE MANAGEMENT
108

Service Service Service
Web App Web App Web App
Hardware
Operating
System
109
SERVICE-
ORIENTED
ARCHITECTURE
- Separation of
concerns
- Optimization of
bottlenecks
- Smaller teams
- API Contracts
- Data replication
- Complicated
provisioning
- Dependency
management
Operating
System
Operating
System
Hardware Hardware

Operating
System
Operating
System
Operating
System
110
MICROSERVICES
- Polyglot
- Single Responsibility
- Smaller Teams
- Utilization
- Machine
types/groups
- Dependency hell
Machine
Infrastructure
Machine Machine

THE BIRTH OF MESOS
TWITTER TECH TALK
The grad students working on Mesos
give a tech talk at Twitter.
March 2010
APACHE INCUBATION
Mesos enters the Apache Incubator.
Spring 2009
CS262B
Ben Hindman, Andy Konwinski and
Matei Zaharia create “Nexus” as their
CS262B class project.
MESOS PUBLISHED
Mesos: A Platform for Fine-Grained
Resource Sharing in the Data Center is
published as a technical report.
September 2010
December 2010
DC/OS
April 2016

Monitoring
- Routing events
○ Alerting
○ Dashboards
○ Storage (long-term retention)
Logging
- Scopes
- Local vs. Central
Maintenance
- Cluster Upgrades
- Cluster Resizing
- Capacity Planning
- User & Package
Management
- Auditing
- Backups & Disaster
Recovery
Troubleshooting
- Debugging
○ Services
○ System
- Tracing
- Chaos Engineering

Downtime is not an option - day 2 operations - Jörg Schad

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a Downtime is not an option - day 2 operations - Jörg Schad

Semelhante a Downtime is not an option - day 2 operations - Jörg Schad (20)

Mais de Codemotion

Mais de Codemotion (20)

Último

Último (20)

Downtime is not an option - day 2 operations - Jörg Schad