SlideShare uma empresa Scribd logo
1 de 21
Baixar para ler offline
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Salt at
Web Scale
Craig Sebenik
SRE
29 January 2014
SaltConf
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Who Am I?
•Programming for 30-ish years
•Scientific computing
•Java and Perl Developer (web apps)
•HATE doing the same thing more than once
•Been at LinkedIn overy 3 years
•From the very beginning of us using salt
•Manage/architect the entire salt infrastructure at
LinkedIn
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
What is LinkedIn?
•Social media company connecting the world’s
professionals
•5000+ employees
•Offices throughout the world
•Based in Mountain View, CA
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
How Big Is lnkedin.com?
•Several data centers
•Customer facing apps (aka “production”)
•Staging for production apps
•Internal only apps
•Several Hundred Apps
•30+K Hosts
•90+% Linux
•Solaris
•Mac and Linux Desktops
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
LinkedIn Operations
•Several operations groups
•Systems (eg. OS install/config, “rack and stack”)
•Database Admins
•Network
•Application (i.e. SRE)
•Different groups have different needs for automation
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
What Is An SRE?
•Assist application developers deploy their apps
•Advise on rollout plans
•Coordinate rollouts
•Generally, the group in-between all of operations and
all of the developers
•Lots of troubleshooting
•SREs write code (automation)
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
SREs Use Salt
•Using salt since 0.8.9
!
•Installation of new apps
!
•Config management
!
•Some troubleshooting
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Salt Architecture
•Each physical data center
•multiple “fabrics” (logical grouping of hosts)
•single salt master (largest set of minions = 8+k)
•warm backup (same private key)
•minions configured with CNAME to master
•Files stored in subversion
•states, grains, modules
•runners
•reactor
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Building Salt
•Internal fork from github
•Add another number. E.g. 2014.01.0.0
•Allows for internal only patches
•Create specific package for testing
•same git repo, with same tags
•LNKD-salt-dev-2014.01.0.0-12345.noarch.rpm
•Allows for emergency changes elsewhere
•salt-dev is deployed on a set of virtual machines
•custom test suite is run
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Installing Salt
•OS is managed by cfengine
•cfengine will push new salt releases and restart
minions
•cfengine also manages minion configs
•master is a set of RPMs
•includes config
•Solaris install is handled by systems team
•Roll out to one data center at a time
•Entire process can take over a week
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Salt Master
•salt master is wrapped in a “runit” script
•runit is a process supervisor
•restarts the master if is dies/stops
•salt API
•use the reactor system to send metrics
•metrics gathering is all home grown
•trying to open source it
•file updates (every 5 mins)
•modules, states, grains
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Master Access
•Logins to the host are managed via cfengine
•Have to be in a whitelisted group to log on
•Access to salt command controlled via sudo
•sudo logs provide audit trail
•Disable cmd.* from salt cli
•If you want to automate; write a state and/or module
•salt API access via a whitelist of IPs
•Auth using LDAP
•Only a handful of commands
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Minions
•basic salt RPM
•includes “salt” command (unfortunately)
•module sync
•every hr
•small python script using client API
•minion metrics
•“age” of modules (via a tracker file)
•uptime of minion
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Deployment With Salt
•LinkedIn.com apps are deployed via a custom app
•App is showing its age and needs to be replaced
•Team outside of operations is writing new
deployment app
•Uses salt api
•Has a lot of custom code
•Not in salt
•Needs to deploy locally (for testing)
•This includes Mac desktop/laptops
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Custom Modules and
States
•couchbase management (via runner)
•runit
•Apache Traffic Server
•metrics system
•alerts
•data collection
•data display
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Module Promotion
•Small oversight last year caused massizve issues
•Developed process to “promote”modules
•Salt environments:
•dev -> vm -> test -> stage -> prod
•different dirs in svn
•sparse directories
•minions are configured to look at certain
environments
•Changes are managed with “review board”
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Problems
•Education!
•Most salt customizations in 2 groups (out of 10)
•Few power users
•Corrupted keys
•Syncing only every hour
•No syncing on solaris
•No highstate enforcement
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
More Problems
•Lots of CPU issues on master
•Key management
•Reinstall of OS with same host name
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
Future
•Multi master
•shared job cache via file system isn’t what we want
•investigating using a returner to share job info
•More training
•Whitelist of states
•Non-ops users
•Eg. devs that want to deploy just their code
•Increase amount of data in grains
©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved.
More Future
•Pillar data
•Metrics
•Better visibility when things go wrong
•Tools to see job cache
•Logs on master are too chatty
•Ability to watch all traffic from a specific minion(s)
•Key management
•reactor system, possibly
Questions?
http://www.linkedin.com/in/craigsebenik

Mais conteúdo relacionado

Mais procurados

De-centralise and Conquer: Masterless Puppet in a Dynamic Environment
De-centralise and Conquer: Masterless Puppet in a Dynamic EnvironmentDe-centralise and Conquer: Masterless Puppet in a Dynamic Environment
De-centralise and Conquer: Masterless Puppet in a Dynamic EnvironmentPuppet
 
Inside the Chef Push Jobs Service - ChefConf 2015
Inside the Chef Push Jobs Service - ChefConf 2015 Inside the Chef Push Jobs Service - ChefConf 2015
Inside the Chef Push Jobs Service - ChefConf 2015 Chef
 
Patch Management on Windows with Puppet
Patch Management on Windows with PuppetPatch Management on Windows with Puppet
Patch Management on Windows with PuppetPuppet
 
Serverspec and Sensu - Testing and Monitoring collide
Serverspec and Sensu - Testing and Monitoring collideServerspec and Sensu - Testing and Monitoring collide
Serverspec and Sensu - Testing and Monitoring collidem_richardson
 
PuppetCamp Sydney 2012 - Building a Multimaster Environment
PuppetCamp Sydney 2012 - Building a Multimaster EnvironmentPuppetCamp Sydney 2012 - Building a Multimaster Environment
PuppetCamp Sydney 2012 - Building a Multimaster EnvironmentGreg Cockburn
 
Continuous Delivery and Infrastructure as Code
Continuous Delivery and Infrastructure as CodeContinuous Delivery and Infrastructure as Code
Continuous Delivery and Infrastructure as CodeSascha Möllering
 
From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015
From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015
From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015Corley S.r.l.
 
Introduction to SaltStack
Introduction to SaltStackIntroduction to SaltStack
Introduction to SaltStackAymen EL Amri
 
Push jobs: an orchestration building block for private Chef
Push jobs: an orchestration building block for private ChefPush jobs: an orchestration building block for private Chef
Push jobs: an orchestration building block for private ChefChef Software, Inc.
 
Dependencies and Licenses
Dependencies and LicensesDependencies and Licenses
Dependencies and LicensesRobert Reiz
 
Vagrant, Chef and TYPO3 - A Love Affair
Vagrant, Chef and TYPO3 - A Love AffairVagrant, Chef and TYPO3 - A Love Affair
Vagrant, Chef and TYPO3 - A Love AffairMichael Lihs
 
Moscow DevOps Meetup P2P oct 2015
Moscow DevOps Meetup P2P oct 2015Moscow DevOps Meetup P2P oct 2015
Moscow DevOps Meetup P2P oct 2015Valiev Sergei
 
MoldCamp - multidimentional testing workflow. CIBox.
MoldCamp  - multidimentional testing workflow. CIBox.MoldCamp  - multidimentional testing workflow. CIBox.
MoldCamp - multidimentional testing workflow. CIBox.Andrii Podanenko
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflowTomas Doran
 
Chef Provisioning a Chef Server Cluster - ChefConf 2015
Chef Provisioning a Chef Server Cluster - ChefConf 2015Chef Provisioning a Chef Server Cluster - ChefConf 2015
Chef Provisioning a Chef Server Cluster - ChefConf 2015Chef
 
faastRuby - Building a FaaS platform with Redis (RedisConf19)
faastRuby - Building a FaaS platform with Redis (RedisConf19)faastRuby - Building a FaaS platform with Redis (RedisConf19)
faastRuby - Building a FaaS platform with Redis (RedisConf19)Paulo Arruda
 
Chef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of ChefChef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of ChefChef Software, Inc.
 

Mais procurados (20)

De-centralise and Conquer: Masterless Puppet in a Dynamic Environment
De-centralise and Conquer: Masterless Puppet in a Dynamic EnvironmentDe-centralise and Conquer: Masterless Puppet in a Dynamic Environment
De-centralise and Conquer: Masterless Puppet in a Dynamic Environment
 
Hacking on WildFly 9
Hacking on WildFly 9Hacking on WildFly 9
Hacking on WildFly 9
 
What's new in chef 12
What's new in chef 12 What's new in chef 12
What's new in chef 12
 
Inside the Chef Push Jobs Service - ChefConf 2015
Inside the Chef Push Jobs Service - ChefConf 2015 Inside the Chef Push Jobs Service - ChefConf 2015
Inside the Chef Push Jobs Service - ChefConf 2015
 
Patch Management on Windows with Puppet
Patch Management on Windows with PuppetPatch Management on Windows with Puppet
Patch Management on Windows with Puppet
 
Serverspec and Sensu - Testing and Monitoring collide
Serverspec and Sensu - Testing and Monitoring collideServerspec and Sensu - Testing and Monitoring collide
Serverspec and Sensu - Testing and Monitoring collide
 
PuppetCamp Sydney 2012 - Building a Multimaster Environment
PuppetCamp Sydney 2012 - Building a Multimaster EnvironmentPuppetCamp Sydney 2012 - Building a Multimaster Environment
PuppetCamp Sydney 2012 - Building a Multimaster Environment
 
Continuous Delivery and Infrastructure as Code
Continuous Delivery and Infrastructure as CodeContinuous Delivery and Infrastructure as Code
Continuous Delivery and Infrastructure as Code
 
From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015
From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015
From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015
 
Introduction to SaltStack
Introduction to SaltStackIntroduction to SaltStack
Introduction to SaltStack
 
Push jobs: an orchestration building block for private Chef
Push jobs: an orchestration building block for private ChefPush jobs: an orchestration building block for private Chef
Push jobs: an orchestration building block for private Chef
 
Salt stack introduction
Salt stack introductionSalt stack introduction
Salt stack introduction
 
Dependencies and Licenses
Dependencies and LicensesDependencies and Licenses
Dependencies and Licenses
 
Vagrant, Chef and TYPO3 - A Love Affair
Vagrant, Chef and TYPO3 - A Love AffairVagrant, Chef and TYPO3 - A Love Affair
Vagrant, Chef and TYPO3 - A Love Affair
 
Moscow DevOps Meetup P2P oct 2015
Moscow DevOps Meetup P2P oct 2015Moscow DevOps Meetup P2P oct 2015
Moscow DevOps Meetup P2P oct 2015
 
MoldCamp - multidimentional testing workflow. CIBox.
MoldCamp  - multidimentional testing workflow. CIBox.MoldCamp  - multidimentional testing workflow. CIBox.
MoldCamp - multidimentional testing workflow. CIBox.
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflow
 
Chef Provisioning a Chef Server Cluster - ChefConf 2015
Chef Provisioning a Chef Server Cluster - ChefConf 2015Chef Provisioning a Chef Server Cluster - ChefConf 2015
Chef Provisioning a Chef Server Cluster - ChefConf 2015
 
faastRuby - Building a FaaS platform with Redis (RedisConf19)
faastRuby - Building a FaaS platform with Redis (RedisConf19)faastRuby - Building a FaaS platform with Redis (RedisConf19)
faastRuby - Building a FaaS platform with Redis (RedisConf19)
 
Chef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of ChefChef Fundamentals Training Series Module 1: Overview of Chef
Chef Fundamentals Training Series Module 1: Overview of Chef
 

Destaque

Spot Trading - A case study in continuous delivery for mission critical finan...
Spot Trading - A case study in continuous delivery for mission critical finan...Spot Trading - A case study in continuous delivery for mission critical finan...
Spot Trading - A case study in continuous delivery for mission critical finan...SaltStack
 
Integration testing for salt states using aws ec2 container service
Integration testing for salt states using aws ec2 container serviceIntegration testing for salt states using aws ec2 container service
Integration testing for salt states using aws ec2 container serviceSaltStack
 
Django: utilizzo avanzato e nuove funzionalità
Django: utilizzo avanzato e nuove funzionalitàDjango: utilizzo avanzato e nuove funzionalità
Django: utilizzo avanzato e nuove funzionalitàskam
 
Deployment ripetibili e automatizzati con Salt
Deployment ripetibili e automatizzati con SaltDeployment ripetibili e automatizzati con Salt
Deployment ripetibili e automatizzati con Saltitbabu
 
SaltConf14 - Yazz Atlas, HP Cloud - Installing OpenStack using SaltStack
SaltConf14 - Yazz Atlas, HP Cloud - Installing OpenStack using SaltStackSaltConf14 - Yazz Atlas, HP Cloud - Installing OpenStack using SaltStack
SaltConf14 - Yazz Atlas, HP Cloud - Installing OpenStack using SaltStackSaltStack
 
Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...
Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...
Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...SaltStack
 
A user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management toolsA user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management toolsSaltStack
 

Destaque (7)

Spot Trading - A case study in continuous delivery for mission critical finan...
Spot Trading - A case study in continuous delivery for mission critical finan...Spot Trading - A case study in continuous delivery for mission critical finan...
Spot Trading - A case study in continuous delivery for mission critical finan...
 
Integration testing for salt states using aws ec2 container service
Integration testing for salt states using aws ec2 container serviceIntegration testing for salt states using aws ec2 container service
Integration testing for salt states using aws ec2 container service
 
Django: utilizzo avanzato e nuove funzionalità
Django: utilizzo avanzato e nuove funzionalitàDjango: utilizzo avanzato e nuove funzionalità
Django: utilizzo avanzato e nuove funzionalità
 
Deployment ripetibili e automatizzati con Salt
Deployment ripetibili e automatizzati con SaltDeployment ripetibili e automatizzati con Salt
Deployment ripetibili e automatizzati con Salt
 
SaltConf14 - Yazz Atlas, HP Cloud - Installing OpenStack using SaltStack
SaltConf14 - Yazz Atlas, HP Cloud - Installing OpenStack using SaltStackSaltConf14 - Yazz Atlas, HP Cloud - Installing OpenStack using SaltStack
SaltConf14 - Yazz Atlas, HP Cloud - Installing OpenStack using SaltStack
 
Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...
Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...
Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...
 
A user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management toolsA user's perspective on SaltStack and other configuration management tools
A user's perspective on SaltStack and other configuration management tools
 

Semelhante a SaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web Scale

August Webinar - Water Cooler Talks: A Look into a Developer's Workbench
August Webinar - Water Cooler Talks: A Look into a Developer's WorkbenchAugust Webinar - Water Cooler Talks: A Look into a Developer's Workbench
August Webinar - Water Cooler Talks: A Look into a Developer's WorkbenchHoward Greenberg
 
Connect the Dots: Logging and Custom Connectors
Connect the Dots: Logging and Custom ConnectorsConnect the Dots: Logging and Custom Connectors
Connect the Dots: Logging and Custom ConnectorsAaronLieberman5
 
A Reference Architecture to Enable Visibility and Traceability across the Ent...
A Reference Architecture to Enable Visibility and Traceability across the Ent...A Reference Architecture to Enable Visibility and Traceability across the Ent...
A Reference Architecture to Enable Visibility and Traceability across the Ent...CollabNet
 
Gartner Infrastructure and Operations Summit Berlin 2015 - DevOps Journey
Gartner Infrastructure and Operations Summit Berlin 2015 - DevOps JourneyGartner Infrastructure and Operations Summit Berlin 2015 - DevOps Journey
Gartner Infrastructure and Operations Summit Berlin 2015 - DevOps JourneyKelly Looney
 
OpenShift Origin: Build a PaaS Just Like Red Hats
OpenShift Origin: Build a PaaS Just Like Red HatsOpenShift Origin: Build a PaaS Just Like Red Hats
OpenShift Origin: Build a PaaS Just Like Red HatsMark Atwood
 
WSO2Con US 2013 - Connected Business - making it happen
WSO2Con US 2013 - Connected Business - making it happenWSO2Con US 2013 - Connected Business - making it happen
WSO2Con US 2013 - Connected Business - making it happenWSO2
 
Selenium at Mozilla: An Essential Element to our Success
Selenium at Mozilla: An Essential Element to our SuccessSelenium at Mozilla: An Essential Element to our Success
Selenium at Mozilla: An Essential Element to our SuccessStephen Donner
 
Openfest15 MySQL Plugin Development
Openfest15 MySQL Plugin DevelopmentOpenfest15 MySQL Plugin Development
Openfest15 MySQL Plugin DevelopmentGeorgi Kodinov
 
Software devops engineer in test (SDET)
Software devops engineer in test (SDET)Software devops engineer in test (SDET)
Software devops engineer in test (SDET)Sriram Angajala
 
Picnic Software - Developing a flexible and scalable application
Picnic Software - Developing a flexible and scalable applicationPicnic Software - Developing a flexible and scalable application
Picnic Software - Developing a flexible and scalable applicationNick Josevski
 
Middleware in Golang: InVision's Rye
Middleware in Golang: InVision's RyeMiddleware in Golang: InVision's Rye
Middleware in Golang: InVision's RyeCale Hoopes
 
How bigtop leveraged docker for build automation and one click hadoop provis...
How bigtop leveraged docker for build automation and  one click hadoop provis...How bigtop leveraged docker for build automation and  one click hadoop provis...
How bigtop leveraged docker for build automation and one click hadoop provis...Evans Ye
 
Make It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version ControlMake It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version Controlindiver
 
Using SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production SystemsUsing SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production SystemsMichael Kehoe
 
Les nouveautés ASP.NET 5 avec Visual Studio 2015
Les nouveautés ASP.NET 5 avec Visual Studio 2015Les nouveautés ASP.NET 5 avec Visual Studio 2015
Les nouveautés ASP.NET 5 avec Visual Studio 2015MSDEVMTL
 
be the captain of your connections deployment
be the captain of your connections deploymentbe the captain of your connections deployment
be the captain of your connections deploymentSharon James
 
Aiming for automatic updates - Drupal Dev Days Lisbon 2018
Aiming for automatic updates - Drupal Dev Days Lisbon 2018Aiming for automatic updates - Drupal Dev Days Lisbon 2018
Aiming for automatic updates - Drupal Dev Days Lisbon 2018hernanibf
 
Cloud Platforms for Java
Cloud Platforms for JavaCloud Platforms for Java
Cloud Platforms for Java3Pillar Global
 
Free Mongo on OpenShift
Free Mongo on OpenShiftFree Mongo on OpenShift
Free Mongo on OpenShiftSteven Pousty
 

Semelhante a SaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web Scale (20)

August Webinar - Water Cooler Talks: A Look into a Developer's Workbench
August Webinar - Water Cooler Talks: A Look into a Developer's WorkbenchAugust Webinar - Water Cooler Talks: A Look into a Developer's Workbench
August Webinar - Water Cooler Talks: A Look into a Developer's Workbench
 
Connect the Dots: Logging and Custom Connectors
Connect the Dots: Logging and Custom ConnectorsConnect the Dots: Logging and Custom Connectors
Connect the Dots: Logging and Custom Connectors
 
A Reference Architecture to Enable Visibility and Traceability across the Ent...
A Reference Architecture to Enable Visibility and Traceability across the Ent...A Reference Architecture to Enable Visibility and Traceability across the Ent...
A Reference Architecture to Enable Visibility and Traceability across the Ent...
 
Gartner Infrastructure and Operations Summit Berlin 2015 - DevOps Journey
Gartner Infrastructure and Operations Summit Berlin 2015 - DevOps JourneyGartner Infrastructure and Operations Summit Berlin 2015 - DevOps Journey
Gartner Infrastructure and Operations Summit Berlin 2015 - DevOps Journey
 
DevOps Days Ohio
DevOps Days OhioDevOps Days Ohio
DevOps Days Ohio
 
OpenShift Origin: Build a PaaS Just Like Red Hats
OpenShift Origin: Build a PaaS Just Like Red HatsOpenShift Origin: Build a PaaS Just Like Red Hats
OpenShift Origin: Build a PaaS Just Like Red Hats
 
WSO2Con US 2013 - Connected Business - making it happen
WSO2Con US 2013 - Connected Business - making it happenWSO2Con US 2013 - Connected Business - making it happen
WSO2Con US 2013 - Connected Business - making it happen
 
Selenium at Mozilla: An Essential Element to our Success
Selenium at Mozilla: An Essential Element to our SuccessSelenium at Mozilla: An Essential Element to our Success
Selenium at Mozilla: An Essential Element to our Success
 
Openfest15 MySQL Plugin Development
Openfest15 MySQL Plugin DevelopmentOpenfest15 MySQL Plugin Development
Openfest15 MySQL Plugin Development
 
Software devops engineer in test (SDET)
Software devops engineer in test (SDET)Software devops engineer in test (SDET)
Software devops engineer in test (SDET)
 
Picnic Software - Developing a flexible and scalable application
Picnic Software - Developing a flexible and scalable applicationPicnic Software - Developing a flexible and scalable application
Picnic Software - Developing a flexible and scalable application
 
Middleware in Golang: InVision's Rye
Middleware in Golang: InVision's RyeMiddleware in Golang: InVision's Rye
Middleware in Golang: InVision's Rye
 
How bigtop leveraged docker for build automation and one click hadoop provis...
How bigtop leveraged docker for build automation and  one click hadoop provis...How bigtop leveraged docker for build automation and  one click hadoop provis...
How bigtop leveraged docker for build automation and one click hadoop provis...
 
Make It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version ControlMake It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version Control
 
Using SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production SystemsUsing SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production Systems
 
Les nouveautés ASP.NET 5 avec Visual Studio 2015
Les nouveautés ASP.NET 5 avec Visual Studio 2015Les nouveautés ASP.NET 5 avec Visual Studio 2015
Les nouveautés ASP.NET 5 avec Visual Studio 2015
 
be the captain of your connections deployment
be the captain of your connections deploymentbe the captain of your connections deployment
be the captain of your connections deployment
 
Aiming for automatic updates - Drupal Dev Days Lisbon 2018
Aiming for automatic updates - Drupal Dev Days Lisbon 2018Aiming for automatic updates - Drupal Dev Days Lisbon 2018
Aiming for automatic updates - Drupal Dev Days Lisbon 2018
 
Cloud Platforms for Java
Cloud Platforms for JavaCloud Platforms for Java
Cloud Platforms for Java
 
Free Mongo on OpenShift
Free Mongo on OpenShiftFree Mongo on OpenShift
Free Mongo on OpenShift
 

Mais de SaltStack

SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltConf14 - Ben Cane - Using SaltStack in High Availability EnvironmentsSaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltConf14 - Ben Cane - Using SaltStack in High Availability EnvironmentsSaltStack
 
SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack
SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStackSaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack
SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStackSaltStack
 
SaltConf14 - Anita Kuno, HP & OpenStack - Using SaltStack for event-driven or...
SaltConf14 - Anita Kuno, HP & OpenStack - Using SaltStack for event-driven or...SaltConf14 - Anita Kuno, HP & OpenStack - Using SaltStack for event-driven or...
SaltConf14 - Anita Kuno, HP & OpenStack - Using SaltStack for event-driven or...SaltStack
 
SaltConf14 - Ryan Lane, Wikimedia - Immediate consistency with Trebuchet Depl...
SaltConf14 - Ryan Lane, Wikimedia - Immediate consistency with Trebuchet Depl...SaltConf14 - Ryan Lane, Wikimedia - Immediate consistency with Trebuchet Depl...
SaltConf14 - Ryan Lane, Wikimedia - Immediate consistency with Trebuchet Depl...SaltStack
 
SaltConf14 - Forrest Alvarez, Choice Hotels - Salt Formulas and States
SaltConf14 - Forrest Alvarez, Choice Hotels - Salt Formulas and StatesSaltConf14 - Forrest Alvarez, Choice Hotels - Salt Formulas and States
SaltConf14 - Forrest Alvarez, Choice Hotels - Salt Formulas and StatesSaltStack
 
SaltConf14 - Thomas Jackson, LinkedIn - Safety with Power Tools
SaltConf14 - Thomas Jackson, LinkedIn - Safety with Power ToolsSaltConf14 - Thomas Jackson, LinkedIn - Safety with Power Tools
SaltConf14 - Thomas Jackson, LinkedIn - Safety with Power ToolsSaltStack
 
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...SaltStack
 
SaltStack - An open source software story
SaltStack - An open source software storySaltStack - An open source software story
SaltStack - An open source software storySaltStack
 
Real-time Cloud Management with SaltStack
Real-time Cloud Management with SaltStackReal-time Cloud Management with SaltStack
Real-time Cloud Management with SaltStackSaltStack
 
Adding to your Python Armory - OpenWest 2013
Adding to your Python Armory - OpenWest 2013Adding to your Python Armory - OpenWest 2013
Adding to your Python Armory - OpenWest 2013SaltStack
 
Real-time Infrastructure Management with SaltStack - OpenWest 2013
Real-time Infrastructure Management with SaltStack - OpenWest 2013Real-time Infrastructure Management with SaltStack - OpenWest 2013
Real-time Infrastructure Management with SaltStack - OpenWest 2013SaltStack
 
Writing SaltStack Modules - OpenWest 2013
Writing SaltStack Modules - OpenWest 2013Writing SaltStack Modules - OpenWest 2013
Writing SaltStack Modules - OpenWest 2013SaltStack
 

Mais de SaltStack (12)

SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltConf14 - Ben Cane - Using SaltStack in High Availability EnvironmentsSaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
 
SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack
SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStackSaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack
SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack
 
SaltConf14 - Anita Kuno, HP & OpenStack - Using SaltStack for event-driven or...
SaltConf14 - Anita Kuno, HP & OpenStack - Using SaltStack for event-driven or...SaltConf14 - Anita Kuno, HP & OpenStack - Using SaltStack for event-driven or...
SaltConf14 - Anita Kuno, HP & OpenStack - Using SaltStack for event-driven or...
 
SaltConf14 - Ryan Lane, Wikimedia - Immediate consistency with Trebuchet Depl...
SaltConf14 - Ryan Lane, Wikimedia - Immediate consistency with Trebuchet Depl...SaltConf14 - Ryan Lane, Wikimedia - Immediate consistency with Trebuchet Depl...
SaltConf14 - Ryan Lane, Wikimedia - Immediate consistency with Trebuchet Depl...
 
SaltConf14 - Forrest Alvarez, Choice Hotels - Salt Formulas and States
SaltConf14 - Forrest Alvarez, Choice Hotels - Salt Formulas and StatesSaltConf14 - Forrest Alvarez, Choice Hotels - Salt Formulas and States
SaltConf14 - Forrest Alvarez, Choice Hotels - Salt Formulas and States
 
SaltConf14 - Thomas Jackson, LinkedIn - Safety with Power Tools
SaltConf14 - Thomas Jackson, LinkedIn - Safety with Power ToolsSaltConf14 - Thomas Jackson, LinkedIn - Safety with Power Tools
SaltConf14 - Thomas Jackson, LinkedIn - Safety with Power Tools
 
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
 
SaltStack - An open source software story
SaltStack - An open source software storySaltStack - An open source software story
SaltStack - An open source software story
 
Real-time Cloud Management with SaltStack
Real-time Cloud Management with SaltStackReal-time Cloud Management with SaltStack
Real-time Cloud Management with SaltStack
 
Adding to your Python Armory - OpenWest 2013
Adding to your Python Armory - OpenWest 2013Adding to your Python Armory - OpenWest 2013
Adding to your Python Armory - OpenWest 2013
 
Real-time Infrastructure Management with SaltStack - OpenWest 2013
Real-time Infrastructure Management with SaltStack - OpenWest 2013Real-time Infrastructure Management with SaltStack - OpenWest 2013
Real-time Infrastructure Management with SaltStack - OpenWest 2013
 
Writing SaltStack Modules - OpenWest 2013
Writing SaltStack Modules - OpenWest 2013Writing SaltStack Modules - OpenWest 2013
Writing SaltStack Modules - OpenWest 2013
 

Último

Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 

Último (20)

20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 

SaltConf14 - Craig Sebenik, LinkedIn - SaltStack at Web Scale

  • 1. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Salt at Web Scale Craig Sebenik SRE 29 January 2014 SaltConf
  • 2. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Who Am I? •Programming for 30-ish years •Scientific computing •Java and Perl Developer (web apps) •HATE doing the same thing more than once •Been at LinkedIn overy 3 years •From the very beginning of us using salt •Manage/architect the entire salt infrastructure at LinkedIn
  • 3. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. What is LinkedIn? •Social media company connecting the world’s professionals •5000+ employees •Offices throughout the world •Based in Mountain View, CA
  • 4. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. How Big Is lnkedin.com? •Several data centers •Customer facing apps (aka “production”) •Staging for production apps •Internal only apps •Several Hundred Apps •30+K Hosts •90+% Linux •Solaris •Mac and Linux Desktops
  • 5. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. LinkedIn Operations •Several operations groups •Systems (eg. OS install/config, “rack and stack”) •Database Admins •Network •Application (i.e. SRE) •Different groups have different needs for automation
  • 6. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. What Is An SRE? •Assist application developers deploy their apps •Advise on rollout plans •Coordinate rollouts •Generally, the group in-between all of operations and all of the developers •Lots of troubleshooting •SREs write code (automation)
  • 7. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. SREs Use Salt •Using salt since 0.8.9 ! •Installation of new apps ! •Config management ! •Some troubleshooting
  • 8. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Salt Architecture •Each physical data center •multiple “fabrics” (logical grouping of hosts) •single salt master (largest set of minions = 8+k) •warm backup (same private key) •minions configured with CNAME to master •Files stored in subversion •states, grains, modules •runners •reactor
  • 9. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Building Salt •Internal fork from github •Add another number. E.g. 2014.01.0.0 •Allows for internal only patches •Create specific package for testing •same git repo, with same tags •LNKD-salt-dev-2014.01.0.0-12345.noarch.rpm •Allows for emergency changes elsewhere •salt-dev is deployed on a set of virtual machines •custom test suite is run
  • 10. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Installing Salt •OS is managed by cfengine •cfengine will push new salt releases and restart minions •cfengine also manages minion configs •master is a set of RPMs •includes config •Solaris install is handled by systems team •Roll out to one data center at a time •Entire process can take over a week
  • 11. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Salt Master •salt master is wrapped in a “runit” script •runit is a process supervisor •restarts the master if is dies/stops •salt API •use the reactor system to send metrics •metrics gathering is all home grown •trying to open source it •file updates (every 5 mins) •modules, states, grains
  • 12. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Master Access •Logins to the host are managed via cfengine •Have to be in a whitelisted group to log on •Access to salt command controlled via sudo •sudo logs provide audit trail •Disable cmd.* from salt cli •If you want to automate; write a state and/or module •salt API access via a whitelist of IPs •Auth using LDAP •Only a handful of commands
  • 13. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Minions •basic salt RPM •includes “salt” command (unfortunately) •module sync •every hr •small python script using client API •minion metrics •“age” of modules (via a tracker file) •uptime of minion
  • 14. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Deployment With Salt •LinkedIn.com apps are deployed via a custom app •App is showing its age and needs to be replaced •Team outside of operations is writing new deployment app •Uses salt api •Has a lot of custom code •Not in salt •Needs to deploy locally (for testing) •This includes Mac desktop/laptops
  • 15. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Custom Modules and States •couchbase management (via runner) •runit •Apache Traffic Server •metrics system •alerts •data collection •data display
  • 16. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Module Promotion •Small oversight last year caused massizve issues •Developed process to “promote”modules •Salt environments: •dev -> vm -> test -> stage -> prod •different dirs in svn •sparse directories •minions are configured to look at certain environments •Changes are managed with “review board”
  • 17. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Problems •Education! •Most salt customizations in 2 groups (out of 10) •Few power users •Corrupted keys •Syncing only every hour •No syncing on solaris •No highstate enforcement
  • 18. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. More Problems •Lots of CPU issues on master •Key management •Reinstall of OS with same host name
  • 19. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. Future •Multi master •shared job cache via file system isn’t what we want •investigating using a returner to share job info •More training •Whitelist of states •Non-ops users •Eg. devs that want to deploy just their code •Increase amount of data in grains
  • 20. ©2013 LinkedIn Corporation. All Rights Reserved. ORGANIZATION NAME©2014 LinkedIn Corporation. All Rights Reserved. More Future •Pillar data •Metrics •Better visibility when things go wrong •Tools to see job cache •Logs on master are too chatty •Ability to watch all traffic from a specific minion(s) •Key management •reactor system, possibly