SlideShare uma empresa Scribd logo
1 de 36
Baixar para ler offline
SysDB – System DataBase
a system management and inventory collection service
Sebastian ‘tokkee’ Harl
<sh@tokkee.org>
SysDB developer
Open Source Data Center Conference 2014
April, 10 2014
Berlin
Disclaimer
WARNING:
SysDB is still under heavy development.
Flaming, bashing or other forms of constructive
feeback are very appreciated :-)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 2
Background / Motivation
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 3
Motivation
Why SysDB?
• Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
Motivation
Why SysDB?
• Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
• Who also uses Puppet/Chef/etc.?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
Motivation
Why SysDB?
• Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
• Who also uses Puppet/Chef/etc.?
• Who also uses collectd/Munin/etc.?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
Motivation
Why SysDB?
• Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
• Who also uses Puppet/Chef/etc.?
• Who also uses collectd/Munin/etc.?
• Who also uses some inventory service?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
Motivation
Why SysDB?
• Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
• Who also uses Puppet/Chef/etc.?
• Who also uses collectd/Munin/etc.?
• Who also uses some inventory service?
• Who also maintains a CMDB?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
Motivation
Why SysDB?
• Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
• Who also uses Puppet/Chef/etc.?
• Who also uses collectd/Munin/etc.?
• Who also uses some inventory service?
• Who also maintains a CMDB?
• Who likes their setup? ;-)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
History
• Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
History
• Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
• First idea: group graphs using Puppet facts.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
History
• Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
• First idea: group graphs using Puppet facts.
• Second thought: oh, then we can also add monitoring
information.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
History
• Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
• First idea: group graphs using Puppet facts.
• Second thought: oh, then we can also add monitoring
information.
• Then: this should be generic and extensible.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
History
• Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
• First idea: group graphs using Puppet facts.
• Second thought: oh, then we can also add monitoring
information.
• Then: this should be generic and extensible.
• Then I realized that I don’t like web development.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
History
• Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
• First idea: group graphs using Puppet facts.
• Second thought: oh, then we can also add monitoring
information.
• Then: this should be generic and extensible.
• Then I realized that I don’t like web development.
• Also, a system like that would need a good back-end.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
History
• Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
• First idea: group graphs using Puppet facts.
• Second thought: oh, then we can also add monitoring
information.
• Then: this should be generic and extensible.
• Then I realized that I don’t like web development.
• Also, a system like that would need a good back-end.
• → the idea for SysDB was born
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
The System DataBase
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 6
SysDB – Overview
• SysDB collects information about arbitrary hardware and software
systems.
• The central idea is to get a central view of your infrastructure.
• Simple examples:
Hosts and their attributes (“facts”)
Services and their attributes
Monitoring information (e.g. current state)
• SysDB collects these information and correlates objects from
various backends.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 7
SysDB – core features
• https://github.com/tokkee/sysdb
CI: https://travis-ci.org/tokkee/sysdb
> 60% code (function) unit-test coverage in the core
• BSD license
• Written in C
• Easy to extend (simple plugin API)
• Simple network protocol
• Most of the code implemented as a library (reusable)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 8
SysDB – Plugins
Currently available:
• collectd::unixsock – query collectd’s UNIX socket interface
• mk-livestatus – query Monitoring systems (Nagios, Naemon,
Icing, Shinken) using Check MK Livestatus
• puppet::store-configs – query Puppet
• cname::dns – canonicalize host-names using DNS
• syslog – syslog logging
Planned: Passive data collection (e.g. using Gearman), Foreman,
PuppetDB, $your favorite system (send patches!) :-)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 9
SysDB – Architecture
infra
backends
store FE
sysdbd
network
sysdb
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 10
The SysDB Store
• The actual database implementation (in memory)
• Stores generic objects:
Host – any kind of physical resource
Service – any kind of service
Attribute – attributes of hosts and services
• Canonicalization of hosts
• Each object stores the timestamp of the last update and the
(automatically calculated) update interval
• Interface to query data
• JSON is the external data representation
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 11
The SysDB Store – Examples
kvm0
kvm1
kvm2
ovirt0
vhost0
vhost0
vhost0
vhost0
vhost0
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 12
Internal Data Structures
message Host {
required string name;
required DateTime last_update;
required Interval update_interval;
repeated Attribute attributes;
repeated Object children; // Host or Service
optional Object parent;
}
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 13
Internal Data Structures
message Service {
required string name;
required DateTime last_update;
required Interval update_interval;
repeated Attribute attributes;
repeated Object children; // Host or Service
repeated Object parent; // multiple!
}
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 14
Internal Data Structures
message Attribute {
required string name;
required Type value;
required DateTime last_update;
required Interval update_interval;
optional Object parent;
}
Type is either an integer, floating point number, DateTime or binary
data.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 15
The SysDB Client
• Interactive client program for SysDB
• Connects to a SysDB daemon
• Interactive command shell
• Receives and displays asynchronous log messages
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 16
The SysDB Query Language
• Remotely similar to SQL
• Meant to be easily usable in RPCs
• Still under heavy development ;-)
• Currently supported commands:
LIST – returns all hosts
FETCH <hostname> – returns details for a host
LOOKUP hosts WHERE <expression> – query details for
multiple hosts based on their attributes
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 17
The SysDB Query Language – Example(I)
sysdb=> LIST;
{"hosts":[{
"name": "nagios.lxc.tokkee.net",
"last_update": "2014-04-03 10:26:41 +0200",
"update_interval": "5m4s"
},{
"name": "puppet.lxc.tokkee.net",
"last_update": "2014-04-05 11:04:08 +0200",
"update_interval": "5m2s"
},{
"name": "whisky.mobile.tokkee.net",
"last_update": "2014-04-05 11:09:15 +0200",
"update_interval": "10s"
}]}
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 18
The SysDB Query Language – Example(II)
sysdb=> LOOKUP hosts WHERE attribute.architecture = ’amd64’
AND service.name =˜ ’postgres’;
{ "name": "whisky.mobile.tokkee.net",
"last_update": "2014-04-05 11:09:15 +0200",
"update_interval": "10s",
"attributes": [{
"name": "architecture", "value": "amd64",
"last_update": "2014-04-03 10:26:41 +0200",
"update_interval": "5m3s" },{ ... }],
"services": [{
"name": "cpu-0/cpu-idle",
"last_update": "2014-03-31 23:34:06 +0200",
"update_interval": "9.716754301s" },{ ... }]
}
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 19
Use Cases
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 20
Use Cases
• Extend (replace?) your CMDB
→ put the CMDB behind and query it through SysDB
• Compare the back-ends (monitoring)
→ Which hosts / services are missing in a back-end?
→ What is the global status of all Windows systems in some
data-center?
• More flexible web-frontend combining multiple back-ends
→ central dashboard
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 21
Future Directions
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 22
Future Directions
• More documentation!
• RDBMS store back-end (?)
• Interface to query live data (monitoring status, metrics) from
backends
• Distributed architecture (HA and load-balancing)
• Web-Interface
• Extend the type system and filters
• Add support for related informationen (?):
persons / groups (ACLs, . . . )
Events / calendar (Change-Requests, . . . )
• . . .
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 23
SysDB – System DataBase
Thank you for your attention!
Questions, comments, rants?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 24
SysDB – System DataBase
Contakt:
Sebastian “tokkee” Harl
<sh@tokkee.org>
https://github.com/tokkee/sysdb
Please send patches! :-)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 25

Mais conteúdo relacionado

Mais procurados

Lightning Talk: Why and How to Integrate MongoDB and NoSQL into Hadoop Big Da...
Lightning Talk: Why and How to Integrate MongoDB and NoSQL into Hadoop Big Da...Lightning Talk: Why and How to Integrate MongoDB and NoSQL into Hadoop Big Da...
Lightning Talk: Why and How to Integrate MongoDB and NoSQL into Hadoop Big Da...MongoDB
 
From oracle to hadoop with Sqoop and other tools
From oracle to hadoop with Sqoop and other toolsFrom oracle to hadoop with Sqoop and other tools
From oracle to hadoop with Sqoop and other toolsGuy Harrison
 
Kite SDK: Working with Datasets
Kite SDK: Working with DatasetsKite SDK: Working with Datasets
Kite SDK: Working with DatasetsCloudera, Inc.
 
Apache zeppelin the missing component for the big data ecosystem
Apache zeppelin the missing component for the big data ecosystemApache zeppelin the missing component for the big data ecosystem
Apache zeppelin the missing component for the big data ecosystemDuyhai Doan
 
OpenStack Trove Day (19 Aug 2014, Cambridge MA) - Sahara
OpenStack Trove Day (19 Aug 2014, Cambridge MA)  - SaharaOpenStack Trove Day (19 Aug 2014, Cambridge MA)  - Sahara
OpenStack Trove Day (19 Aug 2014, Cambridge MA) - Saharaspinningmatt
 
Apache Sqoop: Unlocking Hadoop for Your Relational Database
Apache Sqoop: Unlocking Hadoop for Your Relational Database Apache Sqoop: Unlocking Hadoop for Your Relational Database
Apache Sqoop: Unlocking Hadoop for Your Relational Database huguk
 
Riak with node.js
Riak with node.jsRiak with node.js
Riak with node.jsSean Cribbs
 
Kite SDK introduction for Portland Big Data
Kite SDK introduction for Portland Big DataKite SDK introduction for Portland Big Data
Kite SDK introduction for Portland Big Data_blue
 
Why your Spark Job is Failing
Why your Spark Job is FailingWhy your Spark Job is Failing
Why your Spark Job is FailingDataWorks Summit
 
Spark Summit East 2017: Apache spark and object stores
Spark Summit East 2017: Apache spark and object storesSpark Summit East 2017: Apache spark and object stores
Spark Summit East 2017: Apache spark and object storesSteve Loughran
 
Introduction to Apache Hive | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache Hive | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to Apache Hive | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache Hive | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab
 
OpenStack Trove in Production at HP - TroveDay 2014
OpenStack Trove in Production at HP  - TroveDay 2014 OpenStack Trove in Production at HP  - TroveDay 2014
OpenStack Trove in Production at HP - TroveDay 2014 Tesora
 
Introduction to Riak and Ripple (KC.rb)
Introduction to Riak and Ripple (KC.rb)Introduction to Riak and Ripple (KC.rb)
Introduction to Riak and Ripple (KC.rb)Sean Cribbs
 
From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...
From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...
From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...Databricks
 
Spark Cassandra Connector: Past, Present, and Future
Spark Cassandra Connector: Past, Present, and FutureSpark Cassandra Connector: Past, Present, and Future
Spark Cassandra Connector: Past, Present, and FutureRussell Spitzer
 
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014NoSQLmatters
 
Dancing Elephants: Working with Object Storage in Apache Spark and Hive
Dancing Elephants: Working with Object Storage in Apache Spark and HiveDancing Elephants: Working with Object Storage in Apache Spark and Hive
Dancing Elephants: Working with Object Storage in Apache Spark and HiveSteve Loughran
 
A Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen FanA Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen FanDatabricks
 
Hive Anatomy
Hive AnatomyHive Anatomy
Hive Anatomynzhang
 

Mais procurados (20)

Lightning Talk: Why and How to Integrate MongoDB and NoSQL into Hadoop Big Da...
Lightning Talk: Why and How to Integrate MongoDB and NoSQL into Hadoop Big Da...Lightning Talk: Why and How to Integrate MongoDB and NoSQL into Hadoop Big Da...
Lightning Talk: Why and How to Integrate MongoDB and NoSQL into Hadoop Big Da...
 
From oracle to hadoop with Sqoop and other tools
From oracle to hadoop with Sqoop and other toolsFrom oracle to hadoop with Sqoop and other tools
From oracle to hadoop with Sqoop and other tools
 
Kite SDK: Working with Datasets
Kite SDK: Working with DatasetsKite SDK: Working with Datasets
Kite SDK: Working with Datasets
 
Apache Kite
Apache KiteApache Kite
Apache Kite
 
Apache zeppelin the missing component for the big data ecosystem
Apache zeppelin the missing component for the big data ecosystemApache zeppelin the missing component for the big data ecosystem
Apache zeppelin the missing component for the big data ecosystem
 
OpenStack Trove Day (19 Aug 2014, Cambridge MA) - Sahara
OpenStack Trove Day (19 Aug 2014, Cambridge MA)  - SaharaOpenStack Trove Day (19 Aug 2014, Cambridge MA)  - Sahara
OpenStack Trove Day (19 Aug 2014, Cambridge MA) - Sahara
 
Apache Sqoop: Unlocking Hadoop for Your Relational Database
Apache Sqoop: Unlocking Hadoop for Your Relational Database Apache Sqoop: Unlocking Hadoop for Your Relational Database
Apache Sqoop: Unlocking Hadoop for Your Relational Database
 
Riak with node.js
Riak with node.jsRiak with node.js
Riak with node.js
 
Kite SDK introduction for Portland Big Data
Kite SDK introduction for Portland Big DataKite SDK introduction for Portland Big Data
Kite SDK introduction for Portland Big Data
 
Why your Spark Job is Failing
Why your Spark Job is FailingWhy your Spark Job is Failing
Why your Spark Job is Failing
 
Spark Summit East 2017: Apache spark and object stores
Spark Summit East 2017: Apache spark and object storesSpark Summit East 2017: Apache spark and object stores
Spark Summit East 2017: Apache spark and object stores
 
Introduction to Apache Hive | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache Hive | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to Apache Hive | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to Apache Hive | Big Data Hadoop Spark Tutorial | CloudxLab
 
OpenStack Trove in Production at HP - TroveDay 2014
OpenStack Trove in Production at HP  - TroveDay 2014 OpenStack Trove in Production at HP  - TroveDay 2014
OpenStack Trove in Production at HP - TroveDay 2014
 
Introduction to Riak and Ripple (KC.rb)
Introduction to Riak and Ripple (KC.rb)Introduction to Riak and Ripple (KC.rb)
Introduction to Riak and Ripple (KC.rb)
 
From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...
From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...
From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...
 
Spark Cassandra Connector: Past, Present, and Future
Spark Cassandra Connector: Past, Present, and FutureSpark Cassandra Connector: Past, Present, and Future
Spark Cassandra Connector: Past, Present, and Future
 
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014
 
Dancing Elephants: Working with Object Storage in Apache Spark and Hive
Dancing Elephants: Working with Object Storage in Apache Spark and HiveDancing Elephants: Working with Object Storage in Apache Spark and Hive
Dancing Elephants: Working with Object Storage in Apache Spark and Hive
 
A Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen FanA Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen Fan
 
Hive Anatomy
Hive AnatomyHive Anatomy
Hive Anatomy
 

Semelhante a OSDC 2014: Sebastian Harl - SysDB the system management and inventory collection service

SysDB – System DataBase — a system management and inventory collection service
SysDB – System DataBase — a system management and inventory collection serviceSysDB – System DataBase — a system management and inventory collection service
SysDB – System DataBase — a system management and inventory collection serviceSysDB Project
 
SysDB — The system management and inventory collection service
SysDB — The system management and inventory collection serviceSysDB — The system management and inventory collection service
SysDB — The system management and inventory collection serviceSysDB Project
 
Managing Your Content with Elasticsearch
Managing Your Content with ElasticsearchManaging Your Content with Elasticsearch
Managing Your Content with ElasticsearchSamantha Quiñones
 
Puppet Camp Tokyo 2014: Keynote
Puppet Camp Tokyo 2014: KeynotePuppet Camp Tokyo 2014: Keynote
Puppet Camp Tokyo 2014: KeynotePuppet
 
Puppet Camp DC 2014: Keynote
Puppet Camp DC 2014: KeynotePuppet Camp DC 2014: Keynote
Puppet Camp DC 2014: KeynotePuppet
 
OSMC 2009 | Icinga by Icinga Team
OSMC 2009 | Icinga by Icinga TeamOSMC 2009 | Icinga by Icinga Team
OSMC 2009 | Icinga by Icinga TeamNETWAYS
 
Icinga 2009 at OSMC
Icinga 2009 at OSMCIcinga 2009 at OSMC
Icinga 2009 at OSMCIcinga
 
Puppet Camp London 2014: Keynote
Puppet Camp London 2014: KeynotePuppet Camp London 2014: Keynote
Puppet Camp London 2014: KeynotePuppet
 
Kuby, ActiveDeployment for Rails Apps
Kuby, ActiveDeployment for Rails AppsKuby, ActiveDeployment for Rails Apps
Kuby, ActiveDeployment for Rails AppsCameron Dutro
 
AusNOG 2017: APNIC Update
AusNOG 2017: APNIC UpdateAusNOG 2017: APNIC Update
AusNOG 2017: APNIC UpdateAPNIC
 
IoT Discovery tutorial
IoT Discovery tutorialIoT Discovery tutorial
IoT Discovery tutorialTarek Elsaleh
 
PHP Continuous Data Processing
PHP Continuous Data ProcessingPHP Continuous Data Processing
PHP Continuous Data ProcessingMichael Peacock
 
Running secured Spark job in Kubernetes compute cluster and integrating with ...
Running secured Spark job in Kubernetes compute cluster and integrating with ...Running secured Spark job in Kubernetes compute cluster and integrating with ...
Running secured Spark job in Kubernetes compute cluster and integrating with ...DataWorks Summit
 
Get started with Microsoft SQL Polybase
Get started with Microsoft SQL PolybaseGet started with Microsoft SQL Polybase
Get started with Microsoft SQL PolybaseHenk van der Valk
 
SharePoint 2014: Where to save my data, for devs!
SharePoint 2014: Where to save my data, for devs!SharePoint 2014: Where to save my data, for devs!
SharePoint 2014: Where to save my data, for devs!Ben Steinhauser
 
Prestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for PrestoPrestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for PrestoSadayuki Furuhashi
 
Machine Learning Platform in LINE Fukuoka
Machine Learning Platform in LINE FukuokaMachine Learning Platform in LINE Fukuoka
Machine Learning Platform in LINE FukuokaLINE Corporation
 

Semelhante a OSDC 2014: Sebastian Harl - SysDB the system management and inventory collection service (20)

SysDB – System DataBase — a system management and inventory collection service
SysDB – System DataBase — a system management and inventory collection serviceSysDB – System DataBase — a system management and inventory collection service
SysDB – System DataBase — a system management and inventory collection service
 
SysDB — The system management and inventory collection service
SysDB — The system management and inventory collection serviceSysDB — The system management and inventory collection service
SysDB — The system management and inventory collection service
 
Managing Your Content with Elasticsearch
Managing Your Content with ElasticsearchManaging Your Content with Elasticsearch
Managing Your Content with Elasticsearch
 
Puppet Camp Tokyo 2014: Keynote
Puppet Camp Tokyo 2014: KeynotePuppet Camp Tokyo 2014: Keynote
Puppet Camp Tokyo 2014: Keynote
 
Puppet Camp DC 2014: Keynote
Puppet Camp DC 2014: KeynotePuppet Camp DC 2014: Keynote
Puppet Camp DC 2014: Keynote
 
OSMC 2009 | Icinga by Icinga Team
OSMC 2009 | Icinga by Icinga TeamOSMC 2009 | Icinga by Icinga Team
OSMC 2009 | Icinga by Icinga Team
 
Icinga 2009 at OSMC
Icinga 2009 at OSMCIcinga 2009 at OSMC
Icinga 2009 at OSMC
 
Puppet Camp London 2014: Keynote
Puppet Camp London 2014: KeynotePuppet Camp London 2014: Keynote
Puppet Camp London 2014: Keynote
 
Kuby, ActiveDeployment for Rails Apps
Kuby, ActiveDeployment for Rails AppsKuby, ActiveDeployment for Rails Apps
Kuby, ActiveDeployment for Rails Apps
 
AusNOG 2017: APNIC Update
AusNOG 2017: APNIC UpdateAusNOG 2017: APNIC Update
AusNOG 2017: APNIC Update
 
Data Science
Data ScienceData Science
Data Science
 
IoT Discovery tutorial
IoT Discovery tutorialIoT Discovery tutorial
IoT Discovery tutorial
 
presentation.docx
presentation.docxpresentation.docx
presentation.docx
 
ACM BPM and elasticsearch AMIS25
ACM BPM and elasticsearch AMIS25ACM BPM and elasticsearch AMIS25
ACM BPM and elasticsearch AMIS25
 
PHP Continuous Data Processing
PHP Continuous Data ProcessingPHP Continuous Data Processing
PHP Continuous Data Processing
 
Running secured Spark job in Kubernetes compute cluster and integrating with ...
Running secured Spark job in Kubernetes compute cluster and integrating with ...Running secured Spark job in Kubernetes compute cluster and integrating with ...
Running secured Spark job in Kubernetes compute cluster and integrating with ...
 
Get started with Microsoft SQL Polybase
Get started with Microsoft SQL PolybaseGet started with Microsoft SQL Polybase
Get started with Microsoft SQL Polybase
 
SharePoint 2014: Where to save my data, for devs!
SharePoint 2014: Where to save my data, for devs!SharePoint 2014: Where to save my data, for devs!
SharePoint 2014: Where to save my data, for devs!
 
Prestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for PrestoPrestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for Presto
 
Machine Learning Platform in LINE Fukuoka
Machine Learning Platform in LINE FukuokaMachine Learning Platform in LINE Fukuoka
Machine Learning Platform in LINE Fukuoka
 

Último

Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevpmgdscunsri
 
Enterprise Content Managements Solutions
Enterprise Content Managements SolutionsEnterprise Content Managements Solutions
Enterprise Content Managements SolutionsIQBG inc
 
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsUniversity of Antwerp
 
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxCYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxBarakaMuyengi
 
Boost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made EasyBoost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made Easymichealwillson701
 
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfFlutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfMind IT Systems
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleShane Coughlan
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.Ritesh Kanjee
 
Unlocking AI: Navigating Open Source vs. Commercial Frontiers
Unlocking AI:Navigating Open Source vs. Commercial FrontiersUnlocking AI:Navigating Open Source vs. Commercial Frontiers
Unlocking AI: Navigating Open Source vs. Commercial FrontiersRaphaël Semeteys
 
Mobile App Development process | Expert Tips
Mobile App Development process | Expert TipsMobile App Development process | Expert Tips
Mobile App Development process | Expert Tipsmichealwillson701
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityRandy Shoup
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...MyFAA
 
Mobile App Development company Houston
Mobile  App  Development  company HoustonMobile  App  Development  company Houston
Mobile App Development company Houstonjennysmithusa549
 
8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdfOffsiteNOC
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...jackiepotts6
 
renewable energy renewable energy renewable energy renewable energy
renewable energy renewable energy renewable energy  renewable energyrenewable energy renewable energy renewable energy  renewable energy
renewable energy renewable energy renewable energy renewable energyjeyasrig
 
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...telebusocialmarketin
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJpolinaucc
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startMaxim Salnikov
 

Último (20)

Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
 
Enterprise Content Managements Solutions
Enterprise Content Managements SolutionsEnterprise Content Managements Solutions
Enterprise Content Managements Solutions
 
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
 
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxCYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
 
Boost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made EasyBoost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made Easy
 
20140812 - OBD2 Solution
20140812 - OBD2 Solution20140812 - OBD2 Solution
20140812 - OBD2 Solution
 
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfFlutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scale
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.
 
Unlocking AI: Navigating Open Source vs. Commercial Frontiers
Unlocking AI:Navigating Open Source vs. Commercial FrontiersUnlocking AI:Navigating Open Source vs. Commercial Frontiers
Unlocking AI: Navigating Open Source vs. Commercial Frontiers
 
Mobile App Development process | Expert Tips
Mobile App Development process | Expert TipsMobile App Development process | Expert Tips
Mobile App Development process | Expert Tips
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
 
Mobile App Development company Houston
Mobile  App  Development  company HoustonMobile  App  Development  company Houston
Mobile App Development company Houston
 
8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
 
renewable energy renewable energy renewable energy renewable energy
renewable energy renewable energy renewable energy  renewable energyrenewable energy renewable energy renewable energy  renewable energy
renewable energy renewable energy renewable energy renewable energy
 
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJ
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to start
 

OSDC 2014: Sebastian Harl - SysDB the system management and inventory collection service

  • 1. SysDB – System DataBase a system management and inventory collection service Sebastian ‘tokkee’ Harl <sh@tokkee.org> SysDB developer Open Source Data Center Conference 2014 April, 10 2014 Berlin
  • 2. Disclaimer WARNING: SysDB is still under heavy development. Flaming, bashing or other forms of constructive feeback are very appreciated :-) c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 2
  • 3. Background / Motivation c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 3
  • 4. Motivation Why SysDB? • Who uses Nagios/Icinga/Naemon/OpenNMS/etc.? c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
  • 5. Motivation Why SysDB? • Who uses Nagios/Icinga/Naemon/OpenNMS/etc.? • Who also uses Puppet/Chef/etc.? c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
  • 6. Motivation Why SysDB? • Who uses Nagios/Icinga/Naemon/OpenNMS/etc.? • Who also uses Puppet/Chef/etc.? • Who also uses collectd/Munin/etc.? c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
  • 7. Motivation Why SysDB? • Who uses Nagios/Icinga/Naemon/OpenNMS/etc.? • Who also uses Puppet/Chef/etc.? • Who also uses collectd/Munin/etc.? • Who also uses some inventory service? c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
  • 8. Motivation Why SysDB? • Who uses Nagios/Icinga/Naemon/OpenNMS/etc.? • Who also uses Puppet/Chef/etc.? • Who also uses collectd/Munin/etc.? • Who also uses some inventory service? • Who also maintains a CMDB? c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
  • 9. Motivation Why SysDB? • Who uses Nagios/Icinga/Naemon/OpenNMS/etc.? • Who also uses Puppet/Chef/etc.? • Who also uses collectd/Munin/etc.? • Who also uses some inventory service? • Who also maintains a CMDB? • Who likes their setup? ;-) c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4
  • 10. History • Initially, I wanted a new front-end for collectd suitable for large, heterogeneous environments. c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
  • 11. History • Initially, I wanted a new front-end for collectd suitable for large, heterogeneous environments. • First idea: group graphs using Puppet facts. c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
  • 12. History • Initially, I wanted a new front-end for collectd suitable for large, heterogeneous environments. • First idea: group graphs using Puppet facts. • Second thought: oh, then we can also add monitoring information. c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
  • 13. History • Initially, I wanted a new front-end for collectd suitable for large, heterogeneous environments. • First idea: group graphs using Puppet facts. • Second thought: oh, then we can also add monitoring information. • Then: this should be generic and extensible. c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
  • 14. History • Initially, I wanted a new front-end for collectd suitable for large, heterogeneous environments. • First idea: group graphs using Puppet facts. • Second thought: oh, then we can also add monitoring information. • Then: this should be generic and extensible. • Then I realized that I don’t like web development. c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
  • 15. History • Initially, I wanted a new front-end for collectd suitable for large, heterogeneous environments. • First idea: group graphs using Puppet facts. • Second thought: oh, then we can also add monitoring information. • Then: this should be generic and extensible. • Then I realized that I don’t like web development. • Also, a system like that would need a good back-end. c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
  • 16. History • Initially, I wanted a new front-end for collectd suitable for large, heterogeneous environments. • First idea: group graphs using Puppet facts. • Second thought: oh, then we can also add monitoring information. • Then: this should be generic and extensible. • Then I realized that I don’t like web development. • Also, a system like that would need a good back-end. • → the idea for SysDB was born c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5
  • 17. The System DataBase c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 6
  • 18. SysDB – Overview • SysDB collects information about arbitrary hardware and software systems. • The central idea is to get a central view of your infrastructure. • Simple examples: Hosts and their attributes (“facts”) Services and their attributes Monitoring information (e.g. current state) • SysDB collects these information and correlates objects from various backends. c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 7
  • 19. SysDB – core features • https://github.com/tokkee/sysdb CI: https://travis-ci.org/tokkee/sysdb > 60% code (function) unit-test coverage in the core • BSD license • Written in C • Easy to extend (simple plugin API) • Simple network protocol • Most of the code implemented as a library (reusable) c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 8
  • 20. SysDB – Plugins Currently available: • collectd::unixsock – query collectd’s UNIX socket interface • mk-livestatus – query Monitoring systems (Nagios, Naemon, Icing, Shinken) using Check MK Livestatus • puppet::store-configs – query Puppet • cname::dns – canonicalize host-names using DNS • syslog – syslog logging Planned: Passive data collection (e.g. using Gearman), Foreman, PuppetDB, $your favorite system (send patches!) :-) c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 9
  • 21. SysDB – Architecture infra backends store FE sysdbd network sysdb c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 10
  • 22. The SysDB Store • The actual database implementation (in memory) • Stores generic objects: Host – any kind of physical resource Service – any kind of service Attribute – attributes of hosts and services • Canonicalization of hosts • Each object stores the timestamp of the last update and the (automatically calculated) update interval • Interface to query data • JSON is the external data representation c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 11
  • 23. The SysDB Store – Examples kvm0 kvm1 kvm2 ovirt0 vhost0 vhost0 vhost0 vhost0 vhost0 c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 12
  • 24. Internal Data Structures message Host { required string name; required DateTime last_update; required Interval update_interval; repeated Attribute attributes; repeated Object children; // Host or Service optional Object parent; } c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 13
  • 25. Internal Data Structures message Service { required string name; required DateTime last_update; required Interval update_interval; repeated Attribute attributes; repeated Object children; // Host or Service repeated Object parent; // multiple! } c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 14
  • 26. Internal Data Structures message Attribute { required string name; required Type value; required DateTime last_update; required Interval update_interval; optional Object parent; } Type is either an integer, floating point number, DateTime or binary data. c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 15
  • 27. The SysDB Client • Interactive client program for SysDB • Connects to a SysDB daemon • Interactive command shell • Receives and displays asynchronous log messages c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 16
  • 28. The SysDB Query Language • Remotely similar to SQL • Meant to be easily usable in RPCs • Still under heavy development ;-) • Currently supported commands: LIST – returns all hosts FETCH <hostname> – returns details for a host LOOKUP hosts WHERE <expression> – query details for multiple hosts based on their attributes c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 17
  • 29. The SysDB Query Language – Example(I) sysdb=> LIST; {"hosts":[{ "name": "nagios.lxc.tokkee.net", "last_update": "2014-04-03 10:26:41 +0200", "update_interval": "5m4s" },{ "name": "puppet.lxc.tokkee.net", "last_update": "2014-04-05 11:04:08 +0200", "update_interval": "5m2s" },{ "name": "whisky.mobile.tokkee.net", "last_update": "2014-04-05 11:09:15 +0200", "update_interval": "10s" }]} c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 18
  • 30. The SysDB Query Language – Example(II) sysdb=> LOOKUP hosts WHERE attribute.architecture = ’amd64’ AND service.name =˜ ’postgres’; { "name": "whisky.mobile.tokkee.net", "last_update": "2014-04-05 11:09:15 +0200", "update_interval": "10s", "attributes": [{ "name": "architecture", "value": "amd64", "last_update": "2014-04-03 10:26:41 +0200", "update_interval": "5m3s" },{ ... }], "services": [{ "name": "cpu-0/cpu-idle", "last_update": "2014-03-31 23:34:06 +0200", "update_interval": "9.716754301s" },{ ... }] } c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 19
  • 31. Use Cases c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 20
  • 32. Use Cases • Extend (replace?) your CMDB → put the CMDB behind and query it through SysDB • Compare the back-ends (monitoring) → Which hosts / services are missing in a back-end? → What is the global status of all Windows systems in some data-center? • More flexible web-frontend combining multiple back-ends → central dashboard c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 21
  • 33. Future Directions c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 22
  • 34. Future Directions • More documentation! • RDBMS store back-end (?) • Interface to query live data (monitoring status, metrics) from backends • Distributed architecture (HA and load-balancing) • Web-Interface • Extend the type system and filters • Add support for related informationen (?): persons / groups (ACLs, . . . ) Events / calendar (Change-Requests, . . . ) • . . . c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 23
  • 35. SysDB – System DataBase Thank you for your attention! Questions, comments, rants? c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 24
  • 36. SysDB – System DataBase Contakt: Sebastian “tokkee” Harl <sh@tokkee.org> https://github.com/tokkee/sysdb Please send patches! :-) c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 25