salt stack is distributed remote execution and configuration management tool build on python.

1. ©2014 LinkedIn Corporation. All Rights Reserved.
Who am I?
Muzammil Rehman
SRE for Linkedin
Co-founder Tom Hatch.

2. ©2014 LinkedIn Corporation. All Rights Reserved.
What is LinkedIn?
 Social media company connecting the world’s professionals
 5000+ employees
 Offices throughout the world
 HQ Mountain View, CA

3. ©2014 LinkedIn Corporation. All Rights Reserved.
How Big Is lnkedin.com?
 Several data centers
– Customer facing apps (aka “production”)
– Staging for production apps
– Internal only apps
 Several Hundred Apps
 30+K Hosts
– 90+% Linux
– Mac and Linux Desktops

4. ©2014 LinkedIn Corporation. All Rights Reserved.
What is salt?
 Configuration management
 Remote command execution framework

5. ©2014 LinkedIn Corporation. All Rights Reserved.
Why salt?
 Simplicity
 Parallel execution
 Building on proven technology
 Python client interface
 Fast, flexible, scalable
 Open

6. ©2014 LinkedIn Corporation. All Rights Reserved.
Salt Used
 Using salt since 0.8.9, now 2014.14
 Installation of new apps
 Config management
 Automation

7. ©2014 LinkedIn Corporation. All Rights Reserved.
Salt Architecture
 master The Salt master is the central server that all minions
connect to.
 minion Salt minions are the potentially hundreds or thousands of
servers that may be queried and controlled from the master.

8. ©2014 LinkedIn Corporation. All Rights Reserved.
Master
minion minion
Master
minion minion
db
Masterdb
git

9. ©2014 LinkedIn Corporation. All Rights Reserved.
Installing Salt
 cfengine will push new salt releases and restart minions
– cfengine also manages minion install and configs
 salt master is wrapped in a “runit” script
– salt API
– use the reactor system to send metrics

10. ©2014 LinkedIn Corporation. All Rights Reserved.
Where can salt help
 Running multiple commands
 Repeating the same on n # servers
 Automated installation
 Deployment system
 etc

11. ©2014 LinkedIn Corporation. All Rights Reserved.
Salt modules
 Check for available modules
– Group: https://groups.google.com/forum/#!forum/salt-users
– #salt freenode
– https://github.com/saltstack/salt
– http://www.saltstack.com/blog/

12. ©2014 LinkedIn Corporation. All Rights Reserved.
Salt modules cont..
 Small oversight last year caused massive issues
 Developed process to “promote” modules
 Salt environments:
– dev -> vm -> test -> stage -> prod
 minions are configured to look at certain environments

13. ©2014 LinkedIn Corporation. All Rights Reserved.
Salt module example
def mkdir(dir, user='root', group='root', mode=0755):
'''
Make a directory
salt '*' li.mkdir /path/to/dir [user] [group] [mode]
Returns True or error string
''’
if os.path.isdir(dir):
return True
elif os.path.exists(dir):
return False
else:
try:
os.makedirs(dir, mode)
uid = pwd.getpwnam(user)[2]
gid = grp.getgrnam(group)[2]
os.chown(dir, uid, gid)
except OSError, e:
return e
except KeyError, e:
return e
return True

14. ©2014 LinkedIn Corporation. All Rights Reserved.
Salt stale (sls) aka config managment
 require, watch and prereq.
httpd:
pkg:
- installed
file.managed:
- name: /etc/httpd/conf/httpd.conf
- source: salt://httpd/httpd.conf
- require:
- pkg: httpd
redis:
pkg:
- latest
file.managed:
- source: salt://redis/redis.conf
- name: /etc/redis.conf
- require:
- pkg: redis
service.running:
- enable: True
- watch:
- file: /etc/redis.conf
- pkg: redis

15. ©2014 LinkedIn Corporation. All Rights Reserved.
 Order
 Prereq
 Many more
vim:
pkg.installed:
- order: 1
graceful-down:
cmd.run:
- name: service apache graceful
- prereq:
- file: site-code
site-code:
file.recurse:
- name: /opt/site_code
- source: salt://site/code

16. ©2014 LinkedIn Corporation. All Rights Reserved.
Access control
 Grant access to non-administrative users
 Only sudo and block modules
client_acl:
fred:
- web*:
- pkg.list_pkgs
- test.*
- apache.*
client_acl_blacklist:
users:
- root
- '^(?!sudo_).*$' # all non sudo users
modules:
- cmd

17. ©2014 LinkedIn Corporation. All Rights Reserved.
Troubleshooting
 Run master in foreground
– salt-master -l debug
– salt-minion -l debug
 Port 4505, 4506
 Open file, salt needs at least minions x 2

18. ©2014 LinkedIn Corporation. All Rights Reserved.
Problem with Salt
 Education
– Most salt customizations by small group a users
– Few power users
 Corrupted keys
 Module sync
 No syncing on Solaris
 No high state enforcement

19. ©2014 LinkedIn Corporation. All Rights Reserved.
Demo…

20. ©2014 LinkedIn Corporation. All Rights Reserved.
Q n A