SlideShare uma empresa Scribd logo
1 de 17
Baixar para ler offline
Gnocchi v4 (preview)
gordon chung, gord[at]live.ca
TL;DR (Executive Summary)
Gnocchi v4 accepts incoming measures faster and is more flexible. It also aggregates
measures quicker with less system requirements.
Not bad.
what’s new
Sack distribution aka. sharding aka.
buckets
- ability to distribute incoming
measures
- enable better distribution on incoming
storage (if applicable)
- better workload scheduling across
metricd workers
Redis driver
- ‘super fast’ storage
- best used for incoming storage
- improved i/o at possible expense of
durability
Ceph driver
- omap-based incoming storage
- leveldb or rocksdb
- reduced tiny object i/o
- Ceph doesn’t like tiny objects
Features
test methodology
Procedure
1. POST 1000 generic resources spread
across 20 workers, 20 metrics each
a. 20 metrics per POST
b. 1 point/metric per POST
c. 1.2 million points per run, 20K
metrics, 2 granularities x 6
aggregates
2. Wait for backlog to clear
3. Repeat
v3.x specifications
- Host1 (24Core, 256GB, CentOS)
- OpenStack Controller Node
- Ceph Monitoring service
- Host2 (24Core, 256GB, CentOS)
- OpenStack Compute Node (idle)
- Ceph OSD node (10 OSDs +
SSD Journal)
- 18 metricd
- Host3 (24Core, 256GB, CentOS)
- Ceph OSD node (10 OSDs +
SSD Journal)
- Gnocchi API (32 workers)
- 18 metricd
- Host4 (24Core, 256GB, CentOS)
- OpenStack Compute Node
- Ceph OSD node (10 OSDs +
SSD Journal)
- PostgreSQL
- 18 metricd
v3.x System Overview
v4 specifications
- Host1 (24Core, 256GB, CentOS)
- OpenStack Controller Node
- Ceph Monitoring service
- MySQL, Redis
- Host2 (24Core, 256GB, CentOS)
- OpenStack Compute Node (~idle)
- Ceph OSD node (10 OSDs + SSD
Journal)
- Host3 (24Core, 256GB, CentOS)
- OpenStack Compute Node (~idle)
- Ceph OSD node (10 OSDs + SSD
Journal)
- Gnocchi API (32 workers)
- 18 metricd workers
System differences between v3.x and
v4 tests
- 1 less host
- 36 less metricd workers
- 10 less OSDs
- 512 PGs vs 2056 PGs originally
- MySQL instead of PostgreSQL
- for simplicity only
- Gnocchi v4 configured to use 512
sacks, v3.x has essentially 1 sack
v4 System Overview
results
Notes
● ~8.5x improve POST time vs
v3.x
● v4 Redis incoming ~15%
quicker than v4 Ceph incoming
● Significantly more load on
original v3.x test because of
more metricd workers
● v3.x vs v4 POST under similar
load showed closer performance
but v3.x degraded heavily over
time
POST time
Notes
● ~12x faster at processing
measures
● v4 tests uses ⅓ the number of
metricd processing agents vs
v3.x test
● v4 redis+ceph deployment
~20% improvement compared
to v4 ceph deployment
● v3.x testing with 18 metricd
and 20OSD failed
Aggregation time
Resources
- Homepage - http://gnocchi.xyz
- Github - https://github.com/gnocchixyz

Mais conteúdo relacionado

Mais procurados

Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
DataStax
 
Am I reading GC logs Correctly?
Am I reading GC logs Correctly?Am I reading GC logs Correctly?
Am I reading GC logs Correctly?
Tier1 App
 
1404 app dev series - session 8 - monitoring & performance tuning
1404   app dev series - session 8 - monitoring & performance tuning1404   app dev series - session 8 - monitoring & performance tuning
1404 app dev series - session 8 - monitoring & performance tuning
MongoDB
 

Mais procurados (20)

Odoo Performance Limits
Odoo Performance LimitsOdoo Performance Limits
Odoo Performance Limits
 
OpenTSDB 2.0
OpenTSDB 2.0OpenTSDB 2.0
OpenTSDB 2.0
 
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
 
OpenTSDB for monitoring @ Criteo
OpenTSDB for monitoring @ CriteoOpenTSDB for monitoring @ Criteo
OpenTSDB for monitoring @ Criteo
 
Building a Fast, Resilient Time Series Store with Cassandra (Alex Petrov, Dat...
Building a Fast, Resilient Time Series Store with Cassandra (Alex Petrov, Dat...Building a Fast, Resilient Time Series Store with Cassandra (Alex Petrov, Dat...
Building a Fast, Resilient Time Series Store with Cassandra (Alex Petrov, Dat...
 
Pick diamonds from garbage
Pick diamonds from garbagePick diamonds from garbage
Pick diamonds from garbage
 
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
 
Developing High Performance Application with Aerospike & Go
Developing High Performance Application with Aerospike & GoDeveloping High Performance Application with Aerospike & Go
Developing High Performance Application with Aerospike & Go
 
Update on OpenTSDB and AsyncHBase
Update on OpenTSDB and AsyncHBase Update on OpenTSDB and AsyncHBase
Update on OpenTSDB and AsyncHBase
 
The power of streams in node js
The power of streams in node jsThe power of streams in node js
The power of streams in node js
 
Am I reading GC logs Correctly?
Am I reading GC logs Correctly?Am I reading GC logs Correctly?
Am I reading GC logs Correctly?
 
Thanos - Prometheus on Scale
Thanos - Prometheus on ScaleThanos - Prometheus on Scale
Thanos - Prometheus on Scale
 
Мониторинг. Опять, rootconf 2016
Мониторинг. Опять, rootconf 2016Мониторинг. Опять, rootconf 2016
Мониторинг. Опять, rootconf 2016
 
Metrics: where and how
Metrics: where and howMetrics: where and how
Metrics: where and how
 
Object Storage with Gluster
Object Storage with GlusterObject Storage with Gluster
Object Storage with Gluster
 
1404 app dev series - session 8 - monitoring & performance tuning
1404   app dev series - session 8 - monitoring & performance tuning1404   app dev series - session 8 - monitoring & performance tuning
1404 app dev series - session 8 - monitoring & performance tuning
 
Build a Complex, Realtime Data Management App with Postgres 14!
Build a Complex, Realtime Data Management App with Postgres 14!Build a Complex, Realtime Data Management App with Postgres 14!
Build a Complex, Realtime Data Management App with Postgres 14!
 
Путь мониторинга 2.0 всё стало другим / Всеволод Поляков (Grammarly)
Путь мониторинга 2.0 всё стало другим / Всеволод Поляков (Grammarly)Путь мониторинга 2.0 всё стало другим / Всеволод Поляков (Grammarly)
Путь мониторинга 2.0 всё стало другим / Всеволод Поляков (Grammarly)
 
Galaxy CloudMan performance on AWS
Galaxy CloudMan performance on AWSGalaxy CloudMan performance on AWS
Galaxy CloudMan performance on AWS
 
Linux Cluster and Distributed Resource Manager
Linux Cluster and Distributed Resource ManagerLinux Cluster and Distributed Resource Manager
Linux Cluster and Distributed Resource Manager
 

Semelhante a Gnocchi v4 (preview)

Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Day Beijing: Big Data Analytics on Ceph Object Store Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Community
 
Chotot k8s experiences.pptx
Chotot k8s experiences.pptxChotot k8s experiences.pptx
Chotot k8s experiences.pptx
arptit
 
Using OpenStack In a Traditional Hosting Environment
Using OpenStack In a Traditional Hosting EnvironmentUsing OpenStack In a Traditional Hosting Environment
Using OpenStack In a Traditional Hosting Environment
OpenStack Foundation
 

Semelhante a Gnocchi v4 (preview) (20)

Kubernetes Walk Through from Technical View
Kubernetes Walk Through from Technical ViewKubernetes Walk Through from Technical View
Kubernetes Walk Through from Technical View
 
Ceph for Big Science - Dan van der Ster
Ceph for Big Science - Dan van der SterCeph for Big Science - Dan van der Ster
Ceph for Big Science - Dan van der Ster
 
QCT Ceph Solution - Design Consideration and Reference Architecture
QCT Ceph Solution - Design Consideration and Reference ArchitectureQCT Ceph Solution - Design Consideration and Reference Architecture
QCT Ceph Solution - Design Consideration and Reference Architecture
 
QCT Ceph Solution - Design Consideration and Reference Architecture
QCT Ceph Solution - Design Consideration and Reference ArchitectureQCT Ceph Solution - Design Consideration and Reference Architecture
QCT Ceph Solution - Design Consideration and Reference Architecture
 
Testing kubernetes and_open_shift_at_scale_20170209
Testing kubernetes and_open_shift_at_scale_20170209Testing kubernetes and_open_shift_at_scale_20170209
Testing kubernetes and_open_shift_at_scale_20170209
 
Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Day Beijing: Big Data Analytics on Ceph Object Store Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Day Beijing: Big Data Analytics on Ceph Object Store
 
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...
 
Chotot k8s experiences.pptx
Chotot k8s experiences.pptxChotot k8s experiences.pptx
Chotot k8s experiences.pptx
 
Build an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on CephBuild an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on Ceph
 
Scaling an ELK stack at bol.com
Scaling an ELK stack at bol.comScaling an ELK stack at bol.com
Scaling an ELK stack at bol.com
 
Deep Dive Into the CERN Cloud Infrastructure - November, 2013
Deep Dive Into the CERN Cloud Infrastructure - November, 2013Deep Dive Into the CERN Cloud Infrastructure - November, 2013
Deep Dive Into the CERN Cloud Infrastructure - November, 2013
 
Ceph Deployment at Target: Customer Spotlight
Ceph Deployment at Target: Customer SpotlightCeph Deployment at Target: Customer Spotlight
Ceph Deployment at Target: Customer Spotlight
 
Ceph Deployment at Target: Customer Spotlight
Ceph Deployment at Target: Customer SpotlightCeph Deployment at Target: Customer Spotlight
Ceph Deployment at Target: Customer Spotlight
 
Sanger OpenStack presentation March 2017
Sanger OpenStack presentation March 2017Sanger OpenStack presentation March 2017
Sanger OpenStack presentation March 2017
 
Metal-k8s presentation by Julien Girardin @ Paris Kubernetes Meetup
Metal-k8s presentation by Julien Girardin @ Paris Kubernetes MeetupMetal-k8s presentation by Julien Girardin @ Paris Kubernetes Meetup
Metal-k8s presentation by Julien Girardin @ Paris Kubernetes Meetup
 
PGConf.ASIA 2019 Bali - AppOS: PostgreSQL Extension for Scalable File I/O - K...
PGConf.ASIA 2019 Bali - AppOS: PostgreSQL Extension for Scalable File I/O - K...PGConf.ASIA 2019 Bali - AppOS: PostgreSQL Extension for Scalable File I/O - K...
PGConf.ASIA 2019 Bali - AppOS: PostgreSQL Extension for Scalable File I/O - K...
 
Ceph
CephCeph
Ceph
 
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
 
Hadoop over rgw
Hadoop over rgwHadoop over rgw
Hadoop over rgw
 
Using OpenStack In a Traditional Hosting Environment
Using OpenStack In a Traditional Hosting EnvironmentUsing OpenStack In a Traditional Hosting Environment
Using OpenStack In a Traditional Hosting Environment
 

Último

Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 

Último (20)

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 

Gnocchi v4 (preview)

  • 1. Gnocchi v4 (preview) gordon chung, gord[at]live.ca
  • 2. TL;DR (Executive Summary) Gnocchi v4 accepts incoming measures faster and is more flexible. It also aggregates measures quicker with less system requirements. Not bad.
  • 4. Sack distribution aka. sharding aka. buckets - ability to distribute incoming measures - enable better distribution on incoming storage (if applicable) - better workload scheduling across metricd workers Redis driver - ‘super fast’ storage - best used for incoming storage - improved i/o at possible expense of durability Ceph driver - omap-based incoming storage - leveldb or rocksdb - reduced tiny object i/o - Ceph doesn’t like tiny objects Features
  • 6. Procedure 1. POST 1000 generic resources spread across 20 workers, 20 metrics each a. 20 metrics per POST b. 1 point/metric per POST c. 1.2 million points per run, 20K metrics, 2 granularities x 6 aggregates 2. Wait for backlog to clear 3. Repeat
  • 8. - Host1 (24Core, 256GB, CentOS) - OpenStack Controller Node - Ceph Monitoring service - Host2 (24Core, 256GB, CentOS) - OpenStack Compute Node (idle) - Ceph OSD node (10 OSDs + SSD Journal) - 18 metricd - Host3 (24Core, 256GB, CentOS) - Ceph OSD node (10 OSDs + SSD Journal) - Gnocchi API (32 workers) - 18 metricd - Host4 (24Core, 256GB, CentOS) - OpenStack Compute Node - Ceph OSD node (10 OSDs + SSD Journal) - PostgreSQL - 18 metricd v3.x System Overview
  • 10. - Host1 (24Core, 256GB, CentOS) - OpenStack Controller Node - Ceph Monitoring service - MySQL, Redis - Host2 (24Core, 256GB, CentOS) - OpenStack Compute Node (~idle) - Ceph OSD node (10 OSDs + SSD Journal) - Host3 (24Core, 256GB, CentOS) - OpenStack Compute Node (~idle) - Ceph OSD node (10 OSDs + SSD Journal) - Gnocchi API (32 workers) - 18 metricd workers System differences between v3.x and v4 tests - 1 less host - 36 less metricd workers - 10 less OSDs - 512 PGs vs 2056 PGs originally - MySQL instead of PostgreSQL - for simplicity only - Gnocchi v4 configured to use 512 sacks, v3.x has essentially 1 sack v4 System Overview
  • 12.
  • 13. Notes ● ~8.5x improve POST time vs v3.x ● v4 Redis incoming ~15% quicker than v4 Ceph incoming ● Significantly more load on original v3.x test because of more metricd workers ● v3.x vs v4 POST under similar load showed closer performance but v3.x degraded heavily over time POST time
  • 14.
  • 15.
  • 16. Notes ● ~12x faster at processing measures ● v4 tests uses ⅓ the number of metricd processing agents vs v3.x test ● v4 redis+ceph deployment ~20% improvement compared to v4 ceph deployment ● v3.x testing with 18 metricd and 20OSD failed Aggregation time
  • 17. Resources - Homepage - http://gnocchi.xyz - Github - https://github.com/gnocchixyz