ConcourseCI love Minio

•

2 likes•1,843 views

Due to some design decision with ConcourseCI there is no sharing between JOB, so Object storage is a good way of achivig this. Minio is an OSS Object storage with a great coverage of S3 api.

Technology

30/11/2016
Etourneau Gwenn
@The_shinji62
ConcourseCI
Minio

About me
Etourneau Gwenn
Sr Solution Architect
Bef.: Platform Architect
https://github.com/shinji62
https://twitter.com/the_shinji62

Pivotal
• Spring Framework
• RabbitMQ
• Concourse
• Cloud Foundry
• Pivotal Tracker
• Gemﬁre

Agenda
• Concourse
• Minio
• Minio for Concourse
• Q&A

ConcourseCI
• Do not allow snowﬂake
• Isolation between job
• Non share volume / storage / cache

Jobs 1 Jobs 2
ﬁle_out ﬁle_out
X
Concourse

Concourse
• Concourse support S3
• S3 on premise is not easy
• RiackCS, leoFS, Swift are hard
• S3 compatibility so so…

Minio
• Open Source
• Object Storage written in Go
• Distributed ? Not really more like replication
• S3 compatible
• Data protection using Bit rots , Erasure Code
• Single go binary easy to start

Minio limitation
• Do not scale-out or scale back
• Not multi-tenant, user are the same for the same pool
• number of nodes =>4 and <= 16
• Still early phase (>5 commit /day on master)

Minio and Concourse
• Support S3 native resource from concourse
Jobs 1
Jobs 2
ﬁle_out
ﬁle_out
ﬁle_out

Minio bosh release
• Deploy a pool of distributed minio in minutes
• On any IaaS (supported by bosh)

Minio bosh release
• Use bosh v2 links
• Easy to spin-off
• well, you need to learn bosh
a little….

Demo
• Deploy Minio Cluster
• Set Concourse Pipeline

Resources
https://github.com/shinji62/minio-dist-boshrelease
https://github.com/cloudfoundry-community/minio-boshrelease
• Minio bosh release
• Minio
• https://github.com/minio/minio
• https://minio.io/

Resources
• ConcourseCI
• https://concourse.ci

Slack
https://concourseci.slack.com/
#general

What's hot

Confluent Operator as Cloud-Native Kafka Operator for KubernetesKai Wähner

BigQuery implementationSimon Su

Meetup OpenTelemetry IntroDimitrisFinas1

The Patterns of Distributed Logging and ContainersSATOSHI TAGOMORI

Near real-time anomaly detection at Lyftmarkgrover

Bigquery 101Cesar Orozco Manotas

Designing a complete ci cd pipeline using argo events, workflow and cd productsJulian Mazzitelli

Power BI Create lightning fast dashboard with power bi & Its Components Vishal Pawar

CI/CD with GitHub ActionsSwaminathan Vetri

Grokking Techtalk #40: Consistency and Availability tradeoff in database clusterGrokking VN

Getting Started Monitoring with Prometheus and GrafanaSyah Dwi Prihatmoko

Shift left ObservabilityEric D. Schabell

Google Kubernetes Engine (GKE) deep diveAkash Agrawal

Snowflake Automated Deployments / CI/CD PipelinesDrew Hansen

CI/CD Tools Universe: The Ultimate ListPlutora

Container based CI/CD on GitHub ActionsCasey Lee

CI/CD with Azure DevOps and Azure DatabricksGoDataDriven

Data Lake - Multitenancy Best PracticesCitiusTech

Introduction to Kubernetes and Google Container Engine (GKE)Opsta

NiFi Best Practices for the EnterpriseGregory Keys

What's hot (20)

Confluent Operator as Cloud-Native Kafka Operator for Kubernetes

BigQuery implementation

Meetup OpenTelemetry Intro

The Patterns of Distributed Logging and Containers

Near real-time anomaly detection at Lyft

Bigquery 101

Designing a complete ci cd pipeline using argo events, workflow and cd products

Power BI Create lightning fast dashboard with power bi & Its Components

CI/CD with GitHub Actions

Grokking Techtalk #40: Consistency and Availability tradeoff in database cluster

Getting Started Monitoring with Prometheus and Grafana

Shift left Observability

Google Kubernetes Engine (GKE) deep dive

Snowflake Automated Deployments / CI/CD Pipelines

CI/CD Tools Universe: The Ultimate List

Container based CI/CD on GitHub Actions

CI/CD with Azure DevOps and Azure Databricks

Data Lake - Multitenancy Best Practices

Introduction to Kubernetes and Google Container Engine (GKE)

NiFi Best Practices for the Enterprise

Viewers also liked

Minio hearts GoGarima Kapoor

Monitor Cloud Foundry and Bosh with PrometheusGwenn Etourneau

Cloud Foundry Meetup Tokyo #1 Route serviceGwenn Etourneau

Route service-pcf-techmeetupGwenn Etourneau

Bosh 2-0-reloadedGwenn Etourneau

Concourse updatesGwenn Etourneau

Team Support in Concourse CI 2.0 #concourse_tokyoToshiaki Maki

Concourse and DatabaseGwenn Etourneau

Ceph and Storage Management with openATTIC, Ceph Tech Talks 2016-06-23Lenz Grimmer

Lattice yapc-slideshareGwenn Etourneau

Demo Pivotal Circle Of CodeGwenn Etourneau

ConcourseCi DockerimageGwenn Etourneau

[RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB ProjectRakuten Group, Inc.

从玩具到生产 - Cloud Foundry 上海峰会2015年Duncan Johnston-Watt

ECSとSpotFleetで新規ビジネスのトライアルYu Sudo

Concourse CI Meetup DemoToshiaki Maki

Pivotal Cloud Foundry: Building a diverse geo-architecture for Cloud Native A...DataStax Academy

Database Security for PCI DSSOhyama Masanori

Cloud Foundry Technical Overviewcornelia davis

Build an iOS app with ConcourseCI健仁天沼

Viewers also liked (20)

Minio hearts Go

Monitor Cloud Foundry and Bosh with Prometheus

Cloud Foundry Meetup Tokyo #1 Route service

Route service-pcf-techmeetup

Bosh 2-0-reloaded

Concourse updates

Team Support in Concourse CI 2.0 #concourse_tokyo

Concourse and Database

Ceph and Storage Management with openATTIC, Ceph Tech Talks 2016-06-23

Lattice yapc-slideshare

Demo Pivotal Circle Of Code

ConcourseCi Dockerimage

[RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project

从玩具到生产 - Cloud Foundry 上海峰会2015年

ECSとSpotFleetで新規ビジネスのトライアル

Concourse CI Meetup Demo

Pivotal Cloud Foundry: Building a diverse geo-architecture for Cloud Native A...

Database Security for PCI DSS

Cloud Foundry Technical Overview

Build an iOS app with ConcourseCI

Similar to ConcourseCI love Minio

Prototyping like it is 2022 Michael Yagudaev

Docker Meetup Tokyo #23 - Zenko Open Source Multi-Cloud Data Controller - Lau...Laure Vergeron

Engage 2020: Hello are you listening, There is stream for everythingFrank van der Linden

Managing your user data with Sitecore xDBRuud van Falier

Cloud anti-patternsMallika Iyer

How to start WebGL easily?誠人堀口

GitYervand Aghababyan

APIs in production - we built it, can we fix it?Martin Gutenbrunner

Lessons in Open Source from the MongooseJS ODMValeri Karpov

February 13th, 2014 - Unicon IAM Webinar UpdateMisagh Moayyed

Kotti 紹介Yahoo!デベロッパーネットワーク

2014 Q4 IAM Open Source Support Program UpdateJohn Gasper

Stackato v6Jonas Brømsø

Qt Creator Bootcampaccount inactive

S2GX 2012 - Spring Projects InfrastructureGunnar Hillert

Spring Projects InfrastructureRoy Clarkson

Ruby and Distributed Storage SystemsSATOSHI TAGOMORI

Tooling for the JavaScript Eramartinlippert

Ready? Bootstrap! Go! (CFUG Belgium 24 04-2012)Guust Nieuwenhuis

Similar to ConcourseCI love Minio (20)

Prototyping like it is 2022

Docker Meetup Tokyo #23 - Zenko Open Source Multi-Cloud Data Controller - Lau...

Engage 2020: Hello are you listening, There is stream for everything

Managing your user data with Sitecore xDB

Cloud anti-patterns

How to start WebGL easily?

Git

APIs in production - we built it, can we fix it?

Lessons in Open Source from the MongooseJS ODM

February 13th, 2014 - Unicon IAM Webinar Update

Kotti 紹介

2014 Q4 IAM Open Source Support Program Update

Stackato v6

Qt Creator Bootcamp

S2GX 2012 - Spring Projects Infrastructure

Spring Projects Infrastructure

Ruby and Distributed Storage Systems

Tooling for the JavaScript Era

Ready? Bootstrap! Go! (CFUG Belgium 24 04-2012)

Recently uploaded

A Call to Action for Generative AI in 2024Results

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los

Finology Group – Insurtech Innovation Award 2024The Digital Insurer

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Automating Google Workspace (GWS) & more with Apps Scriptwesley chun

Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

Developing An App To Navigate The Roads of BrazilV3cube

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

Partners Life - Insurer Innovation Award 2024The Digital Insurer

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

Histor y of HAM Radio presentation slidevu2urc

Recently uploaded (20)

A Call to Action for Generative AI in 2024

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Finology Group – Insurtech Innovation Award 2024

Presentation on how to chat with PDF using ChatGPT code interpreter

How to Troubleshoot Apps for the Modern Connected Worker

Automating Google Workspace (GWS) & more with Apps Script

Driving Behavioral Change for Information Management through Data-Driven Gree...

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

08448380779 Call Girls In Friends Colony Women Seeking Men

Developing An App To Navigate The Roads of Brazil

Injustice - Developers Among Us (SciFiDevCon 2024)

Partners Life - Insurer Innovation Award 2024

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Histor y of HAM Radio presentation slide

ConcourseCI love Minio

1. 30/11/2016 Etourneau Gwenn @The_shinji62 ConcourseCI Minio

2. About me Etourneau Gwenn Sr Solution Architect Bef.: Platform Architect https://github.com/shinji62 https://twitter.com/the_shinji62

3. Pivotal • Spring Framework • RabbitMQ • Concourse • Cloud Foundry • Pivotal Tracker • Gemﬁre

4. Meetup

5. Agenda • Concourse • Minio • Minio for Concourse • Q&A

6. ConcourseCI

7. ConcourseCI • Do not allow snowﬂake • Isolation between job • Non share volume / storage / cache

8. Jobs 1 Jobs 2 ﬁle_out ﬁle_out X Concourse

9. Concourse • Concourse support S3 • S3 on premise is not easy • RiackCS, leoFS, Swift are hard • S3 compatibility so so…

10. Minio

11. Minio • Open Source • Object Storage written in Go • Distributed ? Not really more like replication • S3 compatible • Data protection using Bit rots , Erasure Code • Single go binary easy to start

12. Minio limitation • Do not scale-out or scale back • Not multi-tenant, user are the same for the same pool • number of nodes =>4 and <= 16 • Still early phase (>5 commit /day on master)

13. Minio 1 disk by node

14. Minio Many disk by nodes

15. Minio and Concourse • Support S3 native resource from concourse Jobs 1 Jobs 2 file_out file_out file_out

16. Minio bosh release

17. Minio bosh release • Deploy a pool of distributed minio in minutes • On any IaaS (supported by bosh)

18. Minio bosh release • Use bosh v2 links • Easy to spin-off • well, you need to learn bosh a little….

19. Minio 1 disk by node

20. Demo

21. Demo • Deploy Minio Cluster • Set Concourse Pipeline

22.

23. Resources https://github.com/shinji62/minio-dist-boshrelease https://github.com/cloudfoundry-community/minio-boshrelease • Minio bosh release • Minio • https://github.com/minio/minio • https://minio.io/

24. Resources • ConcourseCI • https://concourse.ci