RedisConf18 - Easy and Repeatable Kubernetes Development with Skaffold and Redis

•

1 gostou•517 visualizações

Redis Labs

Breakout Session

Tecnologia

Easy and Repeatable Kubernetes Development
with Skaffold and Redis
Matt Rickard
Software Engineer, Google
mattrickard@

Agenda
● Developer workflow from scratch
○ Setting up a cluster
○ Building and deploying
○ Iterative development
○ The path to CI/CD and production

Who am I?
● Matt Rickard
● Software Engineer at Google SF
● Open source software
○ Containers
○ Kubernetes developer tools
● We’re hiring here in San Francisco!
○ mrick@google.com

Developing Applications on top of Kubernetes
Developing Applications on top of Kubernetes
● The Good
○ Declarative configuration
○ Cluster resources (GPUs, TPUs, RAM…)
○ No developer dependencies
○ Reproducible
● The Bad
○ Remote clusters
○ Multiple replicas
● The Ugly
○ Containers
○ Registries
○ Debugging microservices

Setting up a Cluster
● Kubernetes the Easy Way:
○ minikube
○ cloud providers (GKE, AKS, EKS)
■ GCP free trial
● Kubernetes the Harder Way
○ kubeadm
● Kubernetes the Hardest Way
○ manual

Setting up a local cluster
● Why Minikube?
○ Works on Linux, macOS, and Windows
○ Completely OSS, PRs welcome :)
○ Can run offline
○ Quick to provision, quick to destroy
● How to get started?
○ Download minikube
○ Use with virtualbox, kvm, hyperkit, or hyper-v
○ $ minikube start

Interacting with your cluster
● kubectl
○ official client for kubernetes
○ does everything
● kubernetes dashboard
○ OSS version
○ GKE version
● skaffold

Developing Applications on top of Kubernetes
● Developer workflows in a cloud-native world
● Why is it more complicated?
○ Remote clusters
○ Containers
○ Replicas
○ Microservices

Developer tools
● Skaffold by Google
○ Open source cloud-native tool
○ Rapid inner development loop
○ Pluggable build and deploy implementations
○ Multi-application development
● Telepresence
● Draft
● kubectl

Build (Local) Build (Remote) Tag Deploy
Docker
Docker for Desktop
Minikube
Bazel
Buildah
Google Cloud Builder
Kaniko
orcabuild
git
checksum
environment
timedate
Kubectl
Helm
kinflate

Mais conteúdo relacionado

Mais procurados

At Netflix, we manage petabytes of data in Apache Cassandra which must be reliably accessible to users in mere milliseconds. To achieve this, we have built sophisticated control planes that turn our persistence layer based on Apache Cassandra into a truly self-driving system. We will start with the user interface that Netflix developers use to interact with their Cassandra databases and dive deep into the automation that powers it all. From cluster creation, through scaling up, to cluster death, complex automation drives large fleets of virtual machines hosted on the AWS cloud. First, we will cover the basics of how Netflix deploys Apache Cassandra. In particular, this begins with how we mold Apache Cassandra to the Netflix philosophy of immutable infrastructure, including managing software and hardware upgrades in the face of ever-failing hardware. Then we will explore the concrete techniques needed for such a massive deployment, specifically pull-based control planes and auto-healing strategies. Next, we will cover how Netflix has automated complex but critical Apache Cassandra maintenance tasks such as continuous snapshot backups and always-on anti-entropy repair for keeping our datasets safe and consistent. Both of these systems have gone through multiple architectural evolutions, and we have learned many lessons along the way. Lastly, we will share some of the ways this has gone wrong, and what you can do to avoid them. We will cover a few case studies of major Cassandra outages at Netflix, their root cause, and what we learned from those incidents. At the end of this talk, we hope that participants leave with concrete understanding of the challenges in running massive scale Apache Cassandra as well as solid advice and techniques for building their own self-driving data persistence layer.

How netflix manages petabyte scale apache cassandra in the cloud

Vinay Kumar Chella

Dynomite @ Redis Conference 2016

Ioannis Papapanagiotou

iland has built a global data warehouse across multiple data centers, collecting and aggregating data from core cloud services including compute, storage and network as well as chargeback and compliance. iland's warehouse brings actionable intelligence that customers can use to manipulate resources, analyze trends, define alerts and share information. In this session, we would like to present the lessons learned around Cassandra, both at the development and operations level, but also the technology and architecture we put in action on top of Cassandra such as Redis, syslog-ng, RabbitMQ, Java EE, etc. Finally, we would like to share insights on how we are currently extending our platform with Spark and Kafka and what our motivations are.

iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...

DataStax Academy

After reading the topic you are probably asking yourself: “Why I’ve never heard about Cassandra Streams?”. The reason is because Cassandra didn’t have any streams support. Until now. The project started as a preparation to multi regional deployment, so we needed to test Cassandra and answer several simple questions: · what will be the replication lag, or how long will it take for each mutation to propagate? · will we be losing any mutation? · will we see any additional load and/or other problems? · and zillions of other questions To answer them, we decided to integrate Cassandra with Amazon Kinesis, so we can track any individual mutation and analyze replication stats. That is how our Cassandra Streams integration was born. Currently it supports several stream platforms, like Kinesis and Kafka. During this talk you will learn how we did it, how we used it to test Cassandra in Multi-regional setup and what are other possible applications of this concept. About the Speaker Dustin Pham, Sony Dustin is part of the small team who built the core infrastructure which delivered the PlayStation 3 store and then subsequently core services for the PlayStation 4. He has been with Sony for over 4 years and continues to focus on providing entertainment experiences to Sony customers. Dustin is an avid gamer and finds enjoyment in solving large scale problems.

PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...

DataStax

Capturing live traffic in a typical micro-services setup is one of the industry standard strategies for various needs including testing, canary, and dark launches, but when it comes to databases, capturing and replaying live traffic is quite challenging but equally critical. Cassandra is one of the few databases which supports a live traffic capture and replay feature out of the box starting with 4.0. As part of this talk, Vinay Chella will dive deeper into the query logging framework that is introduced in Cassandra 4.0, and the features built on top of this framework, including full query logging, audit logging, and traffic replay. At end of this talk, you will learn how to use audit logging, live traffic capture and replay features in the upcoming release of Cassandra.

Live traffic capture and replay in cassandra 4.0

Vinay Kumar Chella

Safer restarts, faster streaming, and better repair, just a glimpse of cassan...

Vinay Kumar Chella

Seastar Summit 2019 vectorized.io

ScyllaDB

Producer Performance Tuning for Apache Kafka

Jiangjie Qin

This talk is from Distributed Data Summit SF 2018 - http://distributeddatasummit.com/2018-sf/sessions#netflix2 Operating C* can involve a lot of required manpower, complex automation, or both. Some of this complexity comes from operational/configuration activity of the underlying kernel and hardware but much of it is operation complexity stemming from C* itself. Some examples of this complexity are restarting the database in a safe way, reliability backing up and restoring snapshots, monitoring the health of the datastore, and even ensuring eventual consistency through repair. As a result of these complexities, C* operators end up with complicated operational setups, which are expensive to build, manage and monitor. As part of this talk, we will share lessons learned in managing such complexity via our Priam sidecar including recent innovations in how our sidecar ensures the highest possible uptime and correctness of Cassandra. We then use this to motivate building in the management sidecar directly as part of C* itself (CASSANDRA-14395).

Looking towards an official cassandra sidecar netflix

Vinay Kumar Chella

Counters At Scale - A Cautionary Tale

Eric Lubow

These slides are from the recent meetup @ Uber - Apache Cassandra at Uber and Netflix on new features in 4.0. Abstract: A glimpse of Cassandra 4.0 features: There are a lot of exciting features coming in 4.0, but this talk covers some of the features that we at Netflix are particularly excited about and looking forward to. In this talk, we present an overview of just some of the many improvements shipping soon in 4.0.

A glimpse of cassandra 4.0 features netflix

Vinay Kumar Chella

RedisConf18 - Serving Automated Home Valuation with Redis & Kafka

Redis Labs

RedisConf18 - Implementing a New Data Structure for Redis

Redis Labs

RedisConf18 - Redis at LINE - 25 Billion Messages Per Day

Redis Labs

Apache Pulsar is used for various streaming use cases. There is a strong requirement for storing checkpoints while processing the stream in Pulsar Functions, so that in case of any interruption stream processing engine could go back to the last checkpoint. Pulsar uses Zookeeper not only for leader elections or service discovery like critical use cases but also for storing various metadata which puts unnecessary load on zookeeper which hampers mission critical use of Zookeeper. A durable key value store based off of the Apache Pulsar ecosystem addresses the above mentioned use cases nicely. This talk focuses on taking existing Apache Bookkeeper Table Service/State store implementation and taking it to production. Furthermore this talk also touches upon contributing all the features, bug fixes, tools and other improvements back to open source.

Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021

StreamNative

This session will re-evaluate Cassandra’s relationship with runtime and build systems, pointing out ways that the existing systems fall down, and identifying avenues for improvement. Over the past few years, a number of platforms have emerged for running user code. Container runtimes like Docker, container orchestrators such as Kubernetes, and metrics collections agents like Prometheus and Spectator have all gained popularity and mind-share. Cassandra functionality such as metrics, bootstrapping, and monitoring integrates with the newer paradigms, but in an ad-hoc and improvised fashion. By taking a purposeful approach to integrating with these new methods of deployment, the Cassandra community can more fully benefit from their advertised strengths. The Cassandra build system based on Ant+Ivy dates to the early 2000’s, and reflects legacy complexity that could be avoided with modern build systems. Cassandra’s system package builds are not much better and often fail to integrate with industry standards such as systemd. Iterating on the existing systems is difficult, but this technical debt slows innovation in our build systems. In this talk, we propose solutions to make building, deploying and monitoring Cassandra easy and low overhead, while taking advantage of cloud advancements wherever possible.

Building and running cloud native cassandra

Vinay Kumar Chella

RedisConf18 - Microservicesand Redis: A Match made in Heaven

Redis Labs

ScyllaDB is a distributed database designed to scale horizontally and vertically — in theory. What about in practice? ScyllaDB’s Benny Halevy, Director, Software Engineering, will take you through the process and results of benchmarking our NoSQL database at the petabyte level, showing how you can use advanced features like workload prioritization to control priorities of transactional (read-write) and analytic (read-only) queries on the same cluster with smooth and predictable performance. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Scylla Summit 2022: Operating at Monstrous Scales: Benchmarking Petabyte Work...

ScyllaDB

As data is ingested into a database, it must be constantly rewritten for easy querying. Scylla writes incoming data to immutable files that must later be compacted into fewer files in order to maintain good read performance. The question becomes how fast should you compact? The traditional approach is to expose throughput tunables so the user can control the compaction speed. That means finding a good value involves a lot of trial and error. And what if the workload changes? We take a different approach at ScyllaDB. We use the mathematical foundation of control theory to make automatic decisions about compactions, putting an end to compaction tuning altogether. Watch this webinar to learn: - How we created mathematical models of compaction backlog - How to use that model to feed a control theory framework that can automatically tune compactions. - Other exciting developments that are coming in this area

Webinar: Using Control Theory to Keep Compactions Under Control

ScyllaDB

OPTIMIZING THE TICK STACK

InfluxData

Mais procurados (20)

How netflix manages petabyte scale apache cassandra in the cloud

Dynomite @ Redis Conference 2016

iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...

PlayStation and Cassandra Streams (Alexander Filipchik & Dustin Pham, Sony) |...

Live traffic capture and replay in cassandra 4.0

Safer restarts, faster streaming, and better repair, just a glimpse of cassan...

Seastar Summit 2019 vectorized.io

Producer Performance Tuning for Apache Kafka

Looking towards an official cassandra sidecar netflix

Counters At Scale - A Cautionary Tale

A glimpse of cassandra 4.0 features netflix

RedisConf18 - Serving Automated Home Valuation with Redis & Kafka

RedisConf18 - Implementing a New Data Structure for Redis

RedisConf18 - Redis at LINE - 25 Billion Messages Per Day

Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021

Building and running cloud native cassandra

RedisConf18 - Microservicesand Redis: A Match made in Heaven

Scylla Summit 2022: Operating at Monstrous Scales: Benchmarking Petabyte Work...

Webinar: Using Control Theory to Keep Compactions Under Control

OPTIMIZING THE TICK STACK

Semelhante a RedisConf18 - Easy and Repeatable Kubernetes Development with Skaffold and Redis

Kubernetes & Google Kubernetes Engine (GKE)

Akash Agrawal

From zero lines of node to Kubernetes - NodeJS Global summit 2020

Marcel Dempers

In a time where container image building tools outnumber application frameworks, and deployment descriptors are lengthier than a small app, "deployment" is the stage where developer fun goes to die. In a less dramatic tone: The options and complexity of containerizing and deploying an application is, to say the least, a distraction for most developers. But it doesn't have to be. Quarkus provides extensions that help developers eliminate those distractions by making smart choices for them and by integrating with the rest of the Quarkus ecosystem. This demonstration will show that in Quarkusland, Kubernetes is not a killjoy but part of the fun, by providing a concise experience as you mix and match support for various platforms (vanilla Kubernetes & OpenShift) with image building solutions (Docker, Jib & S2i).

Quarkus: From developer joy to Kubernetes nirvana! | DevNation Tech Talk

Red Hat Developers

從打鐵到雲端 - 架構適應與變遷歷經了 2 年創業時間、3 次主機遷移、4 代架構改造，技術變革與容器編排工具的快速演進，影響了每一次架構改進的切入角度。從 Bare metal 到 Public Cloud；從單機 Docker、Docker Swarm 到 Kubernetes。每一代複雜的業務需求考量、解決方案的選用與抉擇、資源不全的過渡時期，都是滿滿的辛酸血淚。而當漸漸面對全球流量時，隻身一人的後台，又該如何以小搏大來面對？讓我們看下去。

DevOpsDays Taipei 2017 從打鐵到雲端

Hung-Yen Chen

Containerizing MongoDB with kubernetes

Brian McNamara

Kubernetes at (Organizational) Scale

Jeff Zellner

Follow along in this free workshop and experience GitOps! AGENDA:  Welcome - Tamao Nakahara, Head of DX (Weaveworks)  Introduction to Kubernetes & GitOps - Mark Emeis, Principal Engineer (Weaveworks)  Weave Gitops Overview - Tamao Nakahara  Free Gitops Workshop - David Harris, Product Manager (Weaveworks) If you're new to Kubernetes and GitOps, we'll give you a brief introduction to both and how GitOps is the natural evolution of Kubernetes. Weave GitOps Core is a continuous delivery product to run apps in any Kubernetes. It is free and open source, and you can get started today! https://www.weave.works/product/gitops-core If you’re stuck, also come talk to us at our Slack channel! #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)

Free GitOps Workshop + Intro to Kubernetes & GitOps

Weaveworks

MicroService architecture_&_Kubernetes

Akash Agrawal

Microservices at Mercari

Google Cloud Platform - Japan

Kubernetes & Google Container Engine @ mabl

Joseph Lust

Let’s see what it takes to build an automated testing/QA setup with Kubernetes and GitLab from scratch, using off-the-shelf components. The goal is to automatically deploy any feature-x or hotfix-x branch to individual environments for automated tests and to make it easier to share intermediate results within the team. This talk will touch on abstract topics like continuous delivery, the value of developer tooling and well-defined workflows, before diving deep into implementation details, learnings, gotchas and stuff-which-went-wrong.

Automated Testing Environments With Kubernetes & GitLab

Vladislav Supalov

Head in the clouds @ bol.com

Maarten Dirkse

Cloud Native Applications on Kubernetes: a DevOps Approach

Nicola Ferraro

GeoServer Feature FRENZY

GeoSolutions

For this info-packed and hands-on workshop we cover: 📍 Introduction to Kubernetes & GitOps talk: We cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you. 📍 Get Started with GitOps: You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for: - an opinionated repo structure to minimize decision fatigue - disaster recovery using GitOps - Helm charts example - Multi-cluster example - all with free and open source tools mostly in the CNCF (eg. Flux and Helm). If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)

Intro to Kubernetes & GitOps Workshop

Weaveworks

At Opendoor, we do a lot of big data processing, and use Spark and Dask clusters for the computations. Our machine learning platform is written in Dask and we are actively moving data ingestion pipelines and geo computations to PySpark. The biggest challenge is that jobs vary in memory, cpu needs, and the load in not evenly distributed over time, which causes our workers and clusters to be over-provisioned. In addition to this, we need to enable data scientists and engineers run their code without having to upgrade the cluster for every request and deal with the dependency hell. To solve all of these problems, we introduce a lightweight integration across some popular tools like Kubernetes, Docker, Airflow and Spark. Using a combination of these tools, we are able to spin up on-demand Spark and Dask clusters for our computing jobs, bring down the cost using autoscaling and spot pricing, unify DAGs across many teams with different stacks on the single Airflow instance, and all of it at minimal cost.

Scalable Clusters On Demand

Bogdan Kyryliuk

Join members from both Four Kitchens and Meredith Agrimedia as they discuss the experience of migration and relaunch of the digital presence of two magazines: Successful Farming at Agriculture.com and WOOD Magazine at woodmagazine.com. We'll start by discussing the scope of the projects, delve into the commonalities and differences, explore their common advertising and analytics implementation, and analyze the unified distribution that supports both brands. By developing the infrastructure simultaneously, brand-agnostic functionality became a priority which in turn created a more modular and flexible system that facilitated open-sourcing and cross-organizational sharing. Thanks to the codebase approach and experience, the first site took about 6 months and the second took less than 6 weeks.

Multiplier Effect: Case Studies in Distributions for Publishers

Jon Peck

Continuous Deployment of Front-end JavaScript with StriderCD, Github and Sauc...

niallo

Continuous Delivery: 5 years later (Incontro DevOps 2018)

Giovanni Toraldo

Google Cloud - Stand Out Features

GDG Cloud Bengaluru

Semelhante a RedisConf18 - Easy and Repeatable Kubernetes Development with Skaffold and Redis (20)

Kubernetes & Google Kubernetes Engine (GKE)

From zero lines of node to Kubernetes - NodeJS Global summit 2020

Quarkus: From developer joy to Kubernetes nirvana! | DevNation Tech Talk

DevOpsDays Taipei 2017 從打鐵到雲端

Containerizing MongoDB with kubernetes

Kubernetes at (Organizational) Scale

Free GitOps Workshop + Intro to Kubernetes & GitOps

MicroService architecture_&_Kubernetes

Microservices at Mercari

Kubernetes & Google Container Engine @ mabl

Automated Testing Environments With Kubernetes & GitLab

Head in the clouds @ bol.com

Cloud Native Applications on Kubernetes: a DevOps Approach

GeoServer Feature FRENZY

Intro to Kubernetes & GitOps Workshop

Scalable Clusters On Demand

Multiplier Effect: Case Studies in Distributions for Publishers

Continuous Deployment of Front-end JavaScript with StriderCD, Github and Sauc...

Continuous Delivery: 5 years later (Incontro DevOps 2018)

Google Cloud - Stand Out Features

Mais de Redis Labs

Redis Day Bangalore 2020 - Session state caching with redis

Redis Labs

Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020

Redis Labs

The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...

Redis Labs

SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020

Redis Labs

Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...

Redis Labs

Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle

Redis Labs

Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020

Redis Labs

Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020

Redis Labs

Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...

Redis Labs

JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...

Redis Labs

Highly Available Persistent Session Management Service by Mohamed Elmergawi o...

Redis Labs

Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...

Redis Labs

Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...

Redis Labs

RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020

Redis Labs

RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020

Redis Labs

RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020

Redis Labs

RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020

Redis Labs

Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...

Redis Labs

Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...

Redis Labs

Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...

Redis Labs

Mais de Redis Labs (20)