Herd your chickens: Ansible for DB2 configuration management
1. #IDUG#IDUG
Herd your chickens: Ansible
for DB2 configuration management.
Frederik Engelen
RealDolmen
Session Code: E06
Tuesday, 11 November 2014 | Platform: DB2 for Linux and Unix
2. #IDUG
“If you want to plow a field with
64 chickens instead of one mule, then yes
the unit cost per chicken is less
but herding chickens becomes the new
frontier.”
-- DB2-L discussion
4. #IDUG
Introduction to Configuration Management
• Who recognises this?
• Or this?
#!/bin/bash
# loop over all databases
for DB in `db2 list db directory | awk '/Database alias/{db = $4}
/Directory entry type/&&/Indirect/{print db}'`; do
db2 +o connect to $DB
db2 <do something>
db2 +o connect reset
done
#!/bin/bash
# stop all instances
for SERVER in dbserver1, dbserver2, db2server3
ssh db2inst1@$SERVER "db2stop force"
done
7. #IDUG
Introduction to Configuration Management
• We improved by creating scripts
• And improved our scripts
#!/bin/bash
# TODO let's hope this doesn't exist yet
echo "net.ipv4.tcp_fin_timeout = 20" >> /etc/sysctl.conf
sysctl -p
#!/bin/bash
# Change line in sysctl.conf, it’s always there on Linux
grep "net.ipv4.tcp_fin_timeout = 20" /etc/sysctl.conf > /dev/null
if [ ! $? ]; then
sed -i 's/net.ipv4.tcp_fin_timeout = .*/net.ipv4.tcp_fin_timeout =
20/g' /etc/sysctl.conf
sysctl -p
fi
9. #IDUG
Almost there…
• Everyone writes their own
• You end up with a ton of scripts
• Not always easily readable
• Different versions scattered everywhere
• What about different values for
net.ipv4.tcp_fin_timeout? Different database names
and instance users?
• Are you sure the right servers have got the right config?
• SSH loops are so much fun…
10. #IDUG
Sneak peak into Ansible
---
- name: Perform the standard Linux configuration for webservers
hosts:
- webservers
tasks:
- name: Lower TCP timeout
sysctl: name=net.ipv4.tcp_fin_timeout value=20 state=present
11. #IDUG
Introduction to Configuration Management
• Describe, create and manage your environment from a central
location
• Code-oriented approach to configuration
• Goals:
• Consistency
• Automation
• Idempotency
• Scalability
• Tools on the market: Puppet, Chef, Salt, CFEngine, Fabric,
Capistrano, Ansible, …
12. #IDUG
Why Ansible?
• It’s one of the younger, leaner members of the family
• Very low learning curve, start in minutes
• Some companies have started migrating from Puppet to
Ansible which I consider a good sign
14. #IDUG
Ansible overview
• Agent-less
• Clients require only minimal software (mainly SSH and
Python)
• Inventory and playbooks are simple text files
• Modules are written in Python
16. #IDUG
Ansible overview - Inventory
• Hosts.cfg is used for selection of target hosts
• Patterns
• Regular expressions
• Set operators (union, except, intersect)
• Support for variables
• Ad-hoc execution examples
# ping all hosts
ansible all –i hosts.cfg –m ping
# get instance status of DB2 hosts
ansible db2 –i hosts.cfg –m command –a 'db2pd –'
# stop DB2 instances on DR site
ansible 'db2:&dr-site' –i hosts.cfg –m command –a 'db2stop force'
# Use regular expression to make powerful selections
ansible '~db2-.*[tst|dev].*' –i hosts.cfg –m ping
17. #IDUG
Ansible overview - Variables
• Describe components in your environment
• Declare at host, group, playbook, command-line, …
• precedence rules apply
• In hosts.cfg file:
[db2]
db2-primary.acme.org db2instance=db2instp
db2-standby.acme.org db2instance=db2insts
[main-site:vars]
default_gateway=10.1.1.1
18. #IDUG
Ansible overview - Variables
• Keep hosts file clean, declare variables in special directories:
• host_vars
• group_vars
• Format is YAML (Yet Another Markup Language)
21. #IDUG
Ansible overview - Special variables
• Facts
• Collected for each host at the start of a playbook
• Referenced like any other variable
• Possible to extend with your own
"ansible_distribution": "Ubuntu",
"ansible_distribution_release": "precise",
"ansible_distribution_version": "12.04",
"ansible_fqdn": "db2-primary.acme.org ",
"ansible_hostname": "db2-primary",
"ansible_os_family": "Debian",
"ansible_system": "Linux",
"ansible_env": {
"COLORTERM": "gnome-terminal",
"DISPLAY": ":0",
…
}
22. #IDUG
Ansible overview - Special variables
• Magic variables
• group_names: list of hosts in specific group
• groups: groups this host is part of
• hostvars: variables for another host
{{hostvars[remote_host].db2_instance}}
• Behavioral variables
• ansible_ssh_host
• ansible_ssh_port
• ansible_ssh_user
• ansible_ssh_pass
• ansible_ssh_private_key_file
• …
23. #IDUG
Ansible overview - Modules
• Encapsulate domain-specific functionality
• Executed on remote hosts or locally
• Idempotent (try to avoid changes)
• Library of modules included
• Usually Python, but any scripting language is possible when
rolling your own
• Ad-hoc
$ ansible * –m command –a db2stop
• In playbook
- name: Stop DB2 instance
command: db2stop
25. #IDUG
Ansible overview - Interesting modules
• Mount
- name: Mount software repository
mount: name=/mnt/repo fstype=nfs src={{repository_server}}:/repo
state=mounted
• File modules (file, copy, template, …)
- name: Push DB2 instance response file
template:
src=/.../{{db2_version}}_{{instance_type}}_aese.rsp.j2
dest=/root/db2_{{db2environment}}_aese.rsp
when: instance_type is defined and db2_major is defined
#v10.5_default_aese.rsp.j2
FILE = /opt/IBM/db2/V{{db2_major}}.{{db2_minor}}_FP{{db2_fixpack}}
LIC_AGREEMENT = ACCEPT
INSTANCE = inst1
inst1.TYPE = ese
inst1.NAME = {{db2instance}}
inst1.GROUP_NAME = {{db2instance_group}}
inst1.HOME_DIRECTORY = /data/db2/{{db2instance}}
inst1.PORT_NUMBER = {{db2_instance_port | default(50004)}}
26. #IDUG
Ansible overview - Interesting modules
• Database modules
- name: Drop MySQL database once migrated to DB2
- mysql_db: name=slowdb state=absent
• Cron
- name: Configure runstats script to run at 23:00
cron: name="runstats" hour="23" job="/bin/my_runstats.sh {{item}}"
with_items: dbnames
• Other
• user/group
• lvg/lvol/
• service
• subversion/git
• mail
• uri
• …
27. #IDUG
Ansible overview - Playbooks
• Configuration, deployment, and orchestration language
• Describe configuration policy or operational steps
- hosts: webservers
vars:
http_port: 80
max_clients: 200
remote_user: root
serial: 1
tasks:
- name: ensure apache is at the latest version
yum: pkg=httpd state=latest
- name: write the apache config file
template: src=/srv/httpd.j2 dest=/etc/httpd.conf
notify:
- restart_apache
- name: ensure apache is running and started on boot
service: name=httpd enabled=yes state=started
handlers:
- name: restart_apache
service: name=httpd state=restarted
28. #IDUG
Ansible overview - Playbook execution
• Scenario:
• Only one webserver in the farm has been upgraded and reconfigured.
• The senior administrator uses Ansible to rectify this oversight of his
trainee.
39. #IDUG
Applying for DB2 – Custom module
• Module db2sql : allows execution of statements to a database,
optionally reading from file (full script in speaker's notes)
argument_spec = dict(
dbname=dict(required=True),
schema=dict(),
sql=dict(),
--%<--
)
dbname = module.params['dbname']
schema = module.params['schema']
sql = module.params['sql']
file = os.path.expanduser(
os.path.expandvars(xstr(module.params['file'])))
40. #IDUG
Applying for DB2 – Custom module
# Connect to database, optionally specifying user
if user:
(rc, out, err) = module.run_command("db2 connect to %s user %s using %s" %
(dbname, user, password))
else:
(rc, out, err) = module.run_command("db2 connect to %s" % (dbname))
# Evaluate any non-0 return codes from connect command
# For HADR Standby databases, send Skipped
if rc <> 0:
words = out.split()
# standby database
if words[0] == "SQL1776N":
module.exit_json(skipped=True, msg="Ignored standby database")
else:
module.fail_json(rc=rc, msg=out)
41. #IDUG
Applying for DB2 – Custom module
# Execute either SQL text or file (should be present)
if sql:
(rc, out, err) = module.run_command('db2 -xp "%s"' % (sql))
elif file:
(rc, out, err) = module.run_command('db2 -x -tvf %s' % (file))
# Evaluate output
# Return separate output lines
if rc <> 0:
module.fail_json(rc=rc, msg=out.split("n"))
else:
module.exit_json(rc=rc, changed=True, msg=[ line.strip() for line in
out.split("n") if line.strip()<>""])
42. #IDUG
Applying for DB2 – Custom module
• A database is not a host, ad hoc execution is impossible
• we need to execute this via a playbook.
- hosts: db-all
gather_facts: no
tasks:
- name: Execute sql on primary host
db2sql: dbname={{ dbname }} sql="{{ sql }}"
delegate_to: ${instance_host}
register: sqloutput
failed_when: sqloutput.rc > 2
- name: Execute sql on secondary host
db2sql: dbname={{ dbname }} sql="{{ sql }}"
delegate_to: ${instance_host_alternate}
when: sqloutput|skipped # only execute when first server is standby
register: sqloutput
failed_when: sqloutput.rc > 2
44. #IDUG
Applying for DB2 – Custom module - Bash
• Example of heading for BASH script (ex. db2state)
#!/bin/bash
# The name of a file containing the arguments to the module is
# given as first argument. Source the file to load the variables:
source ${1}
if [ -z "$dbname" ]; then
echo 'failed=True msg="Module needs dbname= argument"'
fi
if [ -z "$state" ]; then
echo 'failed=True msg="Module needs state= argument"'
fi
if [[ $state == "hadr_primary" ]]; then
…
45. #IDUG
Applying for DB2 – HADR use-case
• Three additional modules, all trivial to implement
• db2dbcfg: change DB CFG parameters
• db2backup: perform backup/restore operations
• db2state: controls database state (deactive, active, hadr_primary, ...)
• Add one parameter to your DB definition
[db-sap]
db-sapprd-prd dbname=SAPPRD instance_host=sapprd-primary_db2isap
instance_host_alternate=sapprd-standby_db2isap hadr_service=50040
• Use the information available in inventory
- name: Set Primary database cfg
db2dbcfg: dbname={{dbname}} param={{item.param}} value={{item.value}} connect=False
delegate_to: ${instance_host}
with_items:
-{ param:"HADR_LOCAL_HOST", value:{{hostvars[instance_host].ansible_ssh_host}}"}
-{ param:"HADR_REMOTE_HOST", value:{{hostvars[instance_host_alternate].ansible_ssh_host}}"}
-{ param:"HADR_REMOTE_INST", value:"{{db2instance}}"}
-{ param:"HADR_LOCAL_SVC", value:"{{hadr_service}}"}
-{ param:"HADR_TIMEOUT", value:"{{hadr_timeout|default(120)}}"}
- ...
46. #IDUG
Wrapping up
• Strong points
• Scales easily to any number of servers
• Short payback period
• Quality of environments has increased
• I can't imagine going back to my old way of working
• Weak points
• Playbook language could be more powerful (loops, calculations, …)
• Variable substitution when using delegate_to isn't working well
• Multi-line command output can be difficult to read
• Serial doesn't define a rolling window but a batch size
• All actions are done per step – a slow server holds back a rollout
• Need to define database-instance relation twice