it inventry management software. using python , mongodb, fluend

1. nouka inventory manager
Toshiaki Baba
toshiaki@netmark.jp / @netmarkjp

2. Inventory management software
http://en.wikipedia.org/wiki/Inventory_management_software

3. Microsoft Excel

4. (´Д｀)

7. inventory management
¤ Microsoft Excel is VERY
GOOD USEFUL PRODUCT
¤ but we need AUTOMATIC
PERIODIC update
¤ and I love Linux and OSS

8. inventory management
¤ AUTOMATIC update
¤ PERIODIC update
¤ focus to Linux machines
¤ OSS

9. nouka inventry manager

10. nouka family
• nouka data collector
• naya data store
naya
• yaoya data convertor
push command result crawl
output
yaoya
nouka

11. nouka data collector
¤ pure python 2.4
¤ run with python
standard library
¤ POST result to naya
with HTTP
¤ if use cron,
AUTOMATIC
PERIODIC update
comes true!

12. nouka default commands
¤ command_ifconfig = ifconfig -a ¤ command_dmidecode = /usr/sbin/dmidecode
¤ command_hostname = hostname ¤ command_resolv = cat /etc/resolv.conf
¤ command_uname = uname -a ¤ command_passwd = cat /etc/passwd
¤ command_df = df -akl ¤ command_group = cat /etc/group
¤ command_iptables = /sbin/iptables-save ¤ command_shadow = cat /etc/shadow
¤ command_rpm = rpm -qa ¤ command_proc_cpuinfo = cat /proc/cpuinfo
¤ command_chkconfig = /sbin/chkconfig --list ¤ command_proc_meminfo = cat /proc/meminfo
¤ command_ip_addr = /sbin/ip addr show ¤ command_proc_mounts = cat /proc/mounts
¤ command_ip_route = /sbin/ip route show
To add command, just only
define in nouka.conf!

13. naya data store
¤ fluentd / td-agent
¤ fluent-plugin-http-enhanced
¤ https://github.com/parolkar/fluent-plugin-http-enhanced
¤ fluent-plugin-mongo
¤ mongodb

14. yaoya data convertor
¤ python 2.6 + pymongo 2.2
¤ CLI tool (GUI tool: yaoyacontrib-webif)
¤ output format
¤ JSON
¤ CSV
¤ chkconfig matrix(CSV)
¤ rpm matrix(CSV)

15. yaoya output example(1)
¤ $ bin/yaoya --group system_A --name command_uname
{u'return_code': u'0', u'command_line': u'uname -a', u'execute_at':
u'1339942622', u'group_name': u'system_A', u'visible': u'True', u'host_name':
u'bb-mba.local', u'time': datetime.datetime(2012, 6, 17, 14, 17, 3), u'output':
u'Darwin bb-mba.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7
16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386n', u'_id':
ObjectId('4fdde6e0c286630ca0000009'), u'command_name':
u'command_uname'}
{u'return_code': u'0', u'command_line': u'uname -a', u'execute_at':
u'1339943547', u'group_name': u'system_A', u'visible': u'True', u'host_name':
u'bbtest-c6', u'time': datetime.datetime(2012, 6, 17, 14, 32, 28), u'output':
u'Linux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST
2012 x86_64 x86_64 x86_64 GNU/Linuxn', u'_id': ObjectId
('4fddea7dc286630ca000003c'), u'command_name': u'command_uname'}

16. yaoya output example(2)
¤ $ bin/csv --group system_A --name command_uname
"bb-mba.local","Darwin bb-mba.local 10.8.0 Darwin Kernel
Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011;
root:xnu-1504.15.3~1/RELEASE_I386 i386"
"bbtest-c6","Linux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1
SMP Tue Apr 17 23:56:34 BST 2012 x86_64 x86_64 x86_64 GNU/
Linux"

17. getting started

18. project repository
¤ nouka
¤ https://bitbucket.org/netmarkjp/nouka
¤ yaoya
¤ https://bitbucket.org/netmarkjp/yaoya
¤ yaoyacontrib-webif
¤ https://bitbucket.org/netmarkjp/yaoyacontrib-webif

19. steps
1. setup naya
2. setup nouka
3. run nouka

20. setup naya @center server(1/2)
¤ td-agent
¤ cat <<EOT >/etc/yum.repos.d/td.repo
[treasuredata]
name=TreasureData
baseurl=http://packages.treasure-data.com/redhat/
$basearch
gpgcheck=0
EOT
¤ yum -y install td-agent
¤ /usr/lib64/fluent/ruby/bin/gem install bson_ext
¤ /usr/lib64/fluent/ruby/bin/gem install fluent-plugin-http-
enhanced

21. setup naya @center server(2/2)
¤ mongod
¤ cat <<EOT >/etc/yum.repos.d/10gen.repo
[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/
x86_64
gpgcheck=0
EOT
¤ yum -y install mongo-10gen.x86_64 mongo-10gen-server.x86_64
¤ mercurial
¤ yum -y install python-devel python-setuptools gcc
¤ pip install mercurial

22. setup nouka @client
¤ curl https://bitbucket.org/netmarkjp/nouka/get/
tip.tar.gz|tar xf –
¤ vi netmarkjp-nouka-*/conf/nouka.conf
¤ change server_url, group_name
¤ netmarkjp-nouka-*/bin/nouka
¤ (set cron if you need)

23. inside nouka

24. data format
{ *** are same value in same execution
‘group_name’ : ‘<group_name’s value in config file>‘, ***
'host_name' : '<env HOSTNAME>', ***
'command_name' : '<command name in config file. named command_*>',
'command_line' : '<command line(value) in config file>',
'output' : '<output of command line>',
'return_code' : '<return code of command line>',
‘visible’ : ‘True’, ***
'execute_at' : '<execute date as unixtime>’, ***
}

25. enjoy development!