Puppet at the centre of everything

•

0 gostou•1,942 visualizações

The document discusses Server Density's architecture which includes 100 Ubuntu servers with 50% being virtual, using Nginx, Python, and MongoDB. It handles 25TB of data per month. Puppet is used for configuration, failover, code deploys, and system updates. The document also considers colocating servers versus using a dedicated provider and factors like hardware specs, costs, skills required, and fun.

Tecnologia Diversão e humor

Server Density Architecture

•100 servers - Ubuntu 12.04

Server Density Architecture

•100 servers - Ubuntu 12.04

•50% virtual

Server Density Architecture

•100 servers - Ubuntu 12.04

•50% virtual

•Nginx, Python, MongoDB

Server Density Architecture

•100 servers - Ubuntu 12.04

•50% virtual

•Nginx, Python, MongoDB

•25TB data per month

• Conﬁg

• Failover

• Code deploys

Use cases

• Conﬁg

• Failover

• Code deploys

• System updates

Use cases

• base
• softlayer
• softlayer-dedicated
• rackspace
• ...

Failover

• /etc/hosts
<%= metrics2IP %> alerts.honshuu.prod
audit.honshuu.prod auth.honshuu.prod
cloud.honshuu.prod
datastore.honshuu.prod
inventory.honshuu.prod ...

Failover

• /etc/hosts
108.168.254.33 puppet

Dealing with outages

Really test your vendors

• Commit

• Build

• Deploy

Code deploys

/opt/puppet/bin/mco puppetd runonce -I <host>

• Commit

• Build

• Deploy

• Puppet run

Code deploys

Code deploys
Advantages
APIs
Already there

Code deploys
Advantages
APIs
Already there
System facts

Code deploys
Disadvantages
Slow
Puppet run already

Code deploys
Disadvantages
Slow
Puppet run already
Eventually consistent

System updates

• Bash script

• Remote commands

System updates

• Bash script

• Remote commands

• Notiﬁcations

System updates

• Bash script

• Remote commands

• Notiﬁcations

• Canaries

Colo experiment
• Build master (buildbot): VM x2 CPU 2.0Ghz, 2GB RAM
– $89/m
• Build slave (buildbot): VM x1 CPU 2.0Ghz, 1GB RAM
– $40/m
• Staging load balancer: VM x1 CPU 2.0Ghz, 1GB RAM
– $40/m
• Staging server 1: VM x2 CPU 2.0Ghz, 8GB RAM
– $165/m
• Staging server 2: VM x1 CPU 2.0Ghz, 2GB RAM
– $50/m
• Puppet master: VM x2 CPU 2.0Ghz, 2GB RAM
– $89/m
Total: $473/m

Colo experiment

•Dell 1U R415
•x2 8C AMD 2.8Ghz

•32GB RAM

Colo experiment

•Dell 1U R415
•x2 8C AMD 2.8Ghz

•32GB RAM

•Dual PSU, NIC

Colo experiment

•Dell 1U R415
•x2 8C AMD 2.8Ghz

•32GB RAM

•Dual PSU, NIC

•x4 1TB SATA hot swappable

Dedicated?

•Hardware replacement

•Managed/support

Dedicated?

•Hardware replacement

•Managed/support

•Networking

•Hardware spec/value

•Total cost

Colo?

•Hardware spec/value

•Total cost

•Internal skills?

Colo?

•Hardware spec/value

•Total cost

•Internal skills?

•More fun?!

Colo?

•£300-350/kWh/m

•4.5A = £520/m

•9A = £900/m

Colo: Power

David Mytton
@davidmytton
david@serverdensity.com
blog.serverdensity.com

Woop Japan!

Mais conteúdo relacionado

Mais procurados

SaltConf14 - Matthew Williams, Flowroute - Salt Virt for Linux contatiners an...SaltStack

Monitoring of OpenNebula installationsNETWAYS

Let's talk about Failures with Kubernetes - Hamburg MeetupHenning Jacobs

Jacopo Nardiello - Monitoring Cloud-Native applications with Prometheus - Cod...Codemotion

Deploying Kubernetes without scaring off your security team - KubeCon 2017Major Hayden

Workshop: Deploying and Scaling Node.js with KubernetesRoss Kukulinski

KubeCon EU 2016: Secure, Cloud-Native Networking with Project CalicoKubeAcademy

OpenNebulaConf 2016 - Icinga2 - APIFY them all by Achim Ledermüller, Netways ...OpenNebula Project

OpenNebulaConf2018 - We use OpenNebula everywhere now - Florian Heigl and Tho...OpenNebula Project

How Can OpenNebula Fit Your Needs: A European Project FeedbackNETWAYS

Sfo17 109 containerized vn fs with data plane acceleration on arm platformLinaro

Deploy High Availability Kubernetes with KubesprayChe-Chia Chang

Deploying Kubernetes on GCP with KubesprayAltoros

Как сделать высоконагруженный сервис, не зная количество нагрузки / Олег Обле...Ontico

Spinnaker - Bay Area AWS Meetup - 20160726Adam Jordens

Xen Project CI for OpenStack OverviewThe Linux Foundation

OpenNebula in a Multiuser EnvironmentNETWAYS

Is there still room for innovation in container orchestration and scheduling LinuxCon ContainerCon CloudOpen China

Highly Available Docker Networking With BGPOpenDNS

Mais procurados (19)

SaltConf14 - Matthew Williams, Flowroute - Salt Virt for Linux contatiners an...

Monitoring of OpenNebula installations

Let's talk about Failures with Kubernetes - Hamburg Meetup

Jacopo Nardiello - Monitoring Cloud-Native applications with Prometheus - Cod...

Deploying Kubernetes without scaring off your security team - KubeCon 2017

Workshop: Deploying and Scaling Node.js with Kubernetes

KubeCon EU 2016: Secure, Cloud-Native Networking with Project Calico

OpenNebulaConf 2016 - Icinga2 - APIFY them all by Achim Ledermüller, Netways ...

OpenNebulaConf2018 - We use OpenNebula everywhere now - Florian Heigl and Tho...

How Can OpenNebula Fit Your Needs: A European Project Feedback

Sfo17 109 containerized vn fs with data plane acceleration on arm platform

Deploy High Availability Kubernetes with Kubespray

Deploying Kubernetes on GCP with Kubespray

Как сделать высоконагруженный сервис, не зная количество нагрузки / Олег Обле...

Spinnaker - Bay Area AWS Meetup - 20160726

Xen Project CI for OpenStack Overview

OpenNebula in a Multiuser Environment

Is there still room for innovation in container orchestration and scheduling

Highly Available Docker Networking With BGP

Semelhante a Puppet at the centre of everything

Puppet Camp Munich 2013: Puppet at the centre of everything Puppet

HadoopCon- Trend Micro SPN Hadoop OverviewYafang Chang

Bare-metal, Docker Containers, and Virtualization: The Growing Choices for Cl...Odinot Stanislas

Production Ready Containers from IBM and DockerDocker, Inc.

Superb Supervision of Short-lived Servers with SensuPaul O'Connor

Data as a Service Kyle Hailey

How bigtop leveraged docker for build automation and one click hadoop provis...Evans Ye

Ninja, Choose Your Weapon!Anton Weiss

HD CCTV -Arecont Exacq Pivot3.pptPawachMetharattanara

Running Cloud Foundry for 12 months - An experience report | anyninesanynines GmbH

High Performance WordPress IIBarry Abrahamson

Making IT Easier to Manage Your Virtualized Environment - David Babbitt, Spic...Spiceworks

Running CentOS on the Facebook fleetDavide Cavalca

Running MySQL on LinuxGreat Wide Open

Taking devops to the Next Level - Max MartinDevopsdays

Server 2016 sneak peekMichael Rüefli

Facing enterprise specific challenges – utility programming in hadoopfann wu

Apache Directory and the OSGi Service Platform - Enrique Rodriguez, PMC Membe...mfrancis

Pandora FMS: VMware Enterprise PluginPandora FMS

Planning to Fail #phpne13Dave Gardner

Semelhante a Puppet at the centre of everything (20)

Puppet Camp Munich 2013: Puppet at the centre of everything

HadoopCon- Trend Micro SPN Hadoop Overview

Bare-metal, Docker Containers, and Virtualization: The Growing Choices for Cl...

Production Ready Containers from IBM and Docker

Superb Supervision of Short-lived Servers with Sensu

Data as a Service

How bigtop leveraged docker for build automation and one click hadoop provis...

Ninja, Choose Your Weapon!

HD CCTV -Arecont Exacq Pivot3.ppt

Running Cloud Foundry for 12 months - An experience report | anynines

High Performance WordPress II

Making IT Easier to Manage Your Virtualized Environment - David Babbitt, Spic...

Running CentOS on the Facebook fleet

Running MySQL on Linux

Taking devops to the Next Level - Max Martin

Server 2016 sneak peek

Facing enterprise specific challenges – utility programming in hadoop

Apache Directory and the OSGi Service Platform - Enrique Rodriguez, PMC Membe...

Pandora FMS: VMware Enterprise Plugin

Planning to Fail #phpne13

Mais de Server Density

Content marketing @ Server DensityServer Density

Flight training for DevOps & HumanOps - IncontroDevOps 2016Server Density

Flight training for DevOpsServer Density

How to Monitor MySQLServer Density

Handling incidentsServer Density

Scaling humans - Ops teams and incident managementServer Density

Briefing: ContainersServer Density

Why puppet? Why now?Server Density

Infrastructure choices - cloud vs colo vs bare metalServer Density

Navigating the customer lifecycleServer Density

Experiences from DevOps production: Deployment, performance, failure.Server Density

DevOps Incident Handling - Making friends not enemies.Server Density

How to monitor NGINXServer Density

How to monitor MongoDBServer Density

High performance Infrastructure Oct 2013Server Density

NoSQL Infrastructure - Late 2013Server Density

Remote startup - building a company from everywhere in the worldServer Density

NoSQL InfrastructureServer Density

StartOps: Growing an ops team from 1 founderServer Density

MongoDB: Optimising for Performance, Scale & AnalyticsServer Density

Mais de Server Density (20)

Content marketing @ Server Density

Flight training for DevOps & HumanOps - IncontroDevOps 2016

Flight training for DevOps

How to Monitor MySQL

Handling incidents

Scaling humans - Ops teams and incident management

Briefing: Containers

Why puppet? Why now?

Infrastructure choices - cloud vs colo vs bare metal

Navigating the customer lifecycle

Experiences from DevOps production: Deployment, performance, failure.

DevOps Incident Handling - Making friends not enemies.

How to monitor NGINX

How to monitor MongoDB

High performance Infrastructure Oct 2013

NoSQL Infrastructure - Late 2013

Remote startup - building a company from everywhere in the world

NoSQL Infrastructure

StartOps: Growing an ops team from 1 founder

MongoDB: Optimising for Performance, Scale & Analytics

Último

My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar

Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB

Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi

"ML in Production",Oleksandr BaganFwdays

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski

DMCC Future of Trade Web3 - Special EditionDubai Multi Commodity Centre

Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106

Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University

Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson

DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy

WordPress Websites for Engineers: Elevate Your Brandgvaughan

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada

Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar

The Future of Software Development - Devin AI Innovative Approach.pdfSeasiaInfotech2

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays

Vector Databases 101 - An introduction to the world of Vector DatabasesZilliz

Training state-of-the-art general text embeddingZilliz

Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren

Puppet at the centre of everything

1. Puppet at the centre of everything

2. David Mytton Woop Japan!

4. Server Density Architecture •100 servers - Ubuntu 12.04

5. Server Density Architecture •100 servers - Ubuntu 12.04 •50% virtual

6. Server Density Architecture •100 servers - Ubuntu 12.04 •50% virtual •Nginx, Python, MongoDB

7. Server Density Architecture •100 servers - Ubuntu 12.04 •50% virtual •Nginx, Python, MongoDB •25TB data per month

8. • Conﬁg Use cases

9. • Conﬁg • Failover Use cases

10. • Conﬁg • Failover • Code deploys Use cases

11. • Conﬁg • Failover • Code deploys • System updates Use cases

12. • Base manifest Conﬁg

13.

14.

15. • Base manifest • Sub manifests Conﬁg

16.

17. • base • softlayer • softlayer-dedicated • rackspace • ...

18. Failover • /etc/hosts <%= metrics2IP %> alerts.honshuu.prod audit.honshuu.prod auth.honshuu.prod cloud.honshuu.prod datastore.honshuu.prod inventory.honshuu.prod ...

19.

20.

21.

22.

23. Failover • /etc/hosts 108.168.254.33 puppet

24.

25. Dealing with outages Expected

26. Dealing with outages Really test your vendors

27. Dealing with outages Simulations

28. • Commit Code deploys

29.

30. • Commit • Build Code deploys

31.

32. • Commit • Build • Deploy Code deploys

33.

34.

35.

36.

37. /opt/puppet/bin/mco puppetd runonce -I <host>

38. • Commit • Build • Deploy • Puppet run Code deploys

39.

40. Code deploys Advantages APIs

41. Code deploys Advantages APIs Already there

42. Code deploys Advantages APIs Already there System facts

43. Code deploys Disadvantages Slow

44. Code deploys Disadvantages Slow Puppet run already

45. Code deploys Disadvantages Slow Puppet run already Eventually consistent

46. • Bash script System updates

47. System updates • Bash script • Remote commands

48.

49. System updates • Bash script • Remote commands • Notiﬁcations

50. System updates • Bash script • Remote commands • Notiﬁcations • Canaries

51.

52. Colo experiment • Build master (buildbot): VM x2 CPU 2.0Ghz, 2GB RAM – $89/m • Build slave (buildbot): VM x1 CPU 2.0Ghz, 1GB RAM – $40/m • Staging load balancer: VM x1 CPU 2.0Ghz, 1GB RAM – $40/m • Staging server 1: VM x2 CPU 2.0Ghz, 8GB RAM – $165/m • Staging server 2: VM x1 CPU 2.0Ghz, 2GB RAM – $50/m • Puppet master: VM x2 CPU 2.0Ghz, 2GB RAM – $89/m Total: $473/m

53.

54. Colo experiment •Dell 1U R415 •x2 8C AMD 2.8Ghz •32GB RAM

55. Colo experiment •Dell 1U R415 •x2 8C AMD 2.8Ghz •32GB RAM •Dual PSU, NIC

56. Colo experiment •Dell 1U R415 •x2 8C AMD 2.8Ghz •32GB RAM •Dual PSU, NIC •x4 1TB SATA hot swappable

57. Dedicated?

58. Dedicated? •Hardware replacement

59. Dedicated? •Hardware replacement •Managed/support

60. Dedicated? •Hardware replacement •Managed/support •Networking

61. Colo?

62. •Hardware spec/value Colo?

63. •Hardware spec/value •Total cost Colo?

64. •Hardware spec/value •Total cost •Internal skills? Colo?