Megabase: How We Containerized Databases at New Relic

©2008–17 New Relic, Inc. All rights reserved.
Joshua Galbraith, Marty Matheny / March 28, 2018
Megabase: How We Containerized
Databases at New Relic

Confidential ©2008–17 New Relic, Inc. All rights reserved
Safe Harbor
2
This presentation and the information herein (including any information that may be incorporated by reference) is provided for informational
purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell
securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and
may not be replicated or disclosed without New Relic’s express written permission.
Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact
or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking
statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as
“believes,” “anticipates,” “expects” or words of similar import.
Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are
subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with
New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by
such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks,
uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially
from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking
statements is included in the filings New Relic makes with the SEC from time to time. Copies of these documents may be obtained by visiting
New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov.
New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes
no warranties, expressed or implied, in this presentation or otherwise, with respect to the information provided.

Building Megabase
3
Context - Why we built it
Marketing - Building support
Options - Available technologies
Requirements - Choosing scope
Results - How we built it, how it works

Building Megabase
4
Options - Available Technologies
Requirements - Choosing Scope

Building Megabase
5

Building Megabase
6

Building Megabase
7
Options - Available Technologies
Requirements - Choosing Scope

Confidential ©2008–17 New Relic, Inc. All rights reserved 8
01 Context - Why we built it

Fall 2016
🍁🍂
9

Need a database? 4 months!!
• Quarterly hardware order
• Dell ships hardware
• Rack & stack hardware
• Provision database
Problem #1: Supply Chain
10
(3 months)
(6 weeks)
(1 week)
(2 days)

• Reorg → autonomous teams
• Rise of microservices
• Agility = unpredictable demand
Problem #2: Increasing Demand
11

• 💰 $50 database, $15,000 server
• 🚫 VMware not supported
• 💵 RDS cost + latency
Problem #3: Cost Efficiency
12

What do you think about
running database on Docker?
Because everything needs to
in our new data centers.
-- Principal Engineer
13
“
”
Nerd Sniped

Is running databases on Docker crazy?
14

02 Marketing - Building support

Because branding works.
Why marketing??

Branding Matters!
18

Good idea != Project approval
● Projects require investment
● Teams are busy
● Competing initiatives

• 📅 2 months for MVP
• 👤 +1 senior engineer
• ✅ Permission (leadership approval)
What We Needed
21

Document and Socialize Plan
22

Build Brand Awareness
23

Sign Up Internal Customers
24

• Brand awareness via repetition
• Internal “press release”
• Publish adoption stats
• Broadcast customer wins
Internal Project Marketing Advice
25

03 Options - Available technologies

Open-Source Orchestration Options
27

• Dynamic
Provisioning
• Reservation
Labels
• Local
Persistent
Volumes
• External
Volumes
Stateful Service Support - Mesos and Marathon
28

• Stateful Sets
• Pods
• Headless
Service
• Persistent
Volumes
• Operators
Stateful Service Support - Kubernetes
29

• Jobs
• Task Groups
• Allocations
• Sticky
Volumes
• Volume
Plugins
• FS Drivers
Stateful Service Support - Nomad
30

Joyent’s Autopilot Pattern for Databases
31

• Application-aware
orchestration
• Autopilot pattern for
lifecycle management
• Storage fabrics and
networked storage 😞
Containerizing Databases - Emergent Patterns
32

• 💾 Database Software
– system stability and state distribution
• 🙇 Database Operators
– configuration and deployment
• 🙋 Database Developers
– availability and consistency
Containerizing Databases - Challenging Assumptions
33

Stateful Service Support - Summary
34

04 Requirements - Choosing scope

📦 Build a consistent,
deployable unit for databases
🚚 Reduce database delivery
time from months to minutes
💸 Reduce wasted resources
($50 database on $15K server)
Revisiting Goals
36

• Our team is busy with
existing databases
• Need to ship an MVP
with limited dev time
• Devs blocked on
database delivery
• Adoption → Survival
A Compressed Timeline
37

❏ Scheduler vs. fixed
container placement
❏ App-specific orchestration
❏ Distributed consensus
❏ Local object storage
❏ Container Pilot process
❏ Supervisor process
Making Trade-Offs
38

• Build on upstream images
• Run containers on our
own separate hardware
• Write an API server to
wrap Docker and LVM
• Write a management CLI
MVP Specification
39

05 Results - How we built it, how it works

1. Use upstream base image
1. Add custom labels
1. Install dependencies
1. Add custom binaries for:
○ backup sync
○ configuration sync
○ replication bootstrap
1. Entrypoint script
Docker Image Building
41

1. Validate data mount
1. SSH client for git pull
1. Sync configs from git
1. Sync state from primary
1. Initialize database and
start database server
Docker Entrypoint
42

• Environment variables
– via deploy config
• Configuration file
– via version control
• Docker Image
– via image registry
Injecting Configuration
43

Megabase API Client - Deployment Manifest
44
● Container name
● Image path and tag
● Resource requirements
● VIP for LB and DNS
● Owning team
● Max severity of outage
● Environment variables

Megabase Architecture
45

Megabase API Server - Endpoints
46

Megabase CLI - Deployment Process
47
● Target an environment
and a host cluster
● Specify a config file
● Generate passwords
● Send API requests
● Check responses
● Validate deployment

• 📦 Database deployments
are consistent, repeatable
• 🚚 Database requests can
be fulfilled within minutes
• 💰 We collocate dozens of
databases per host group
Outcomes
48

Megabase Adoption - First Year
49

• https://mesosphere.github.io/marathon/docs/persistent-volumes.html
• https://docs.mesosphere.com/1.11/tutorials/stateful-services/
• https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
• https://youtu.be/J-Ke0TxGUSg (Kubernetes StatefulSet)
• https://coreos.com/blog/introducing-operators.html
• https://youtu.be/faUQcd5_MUc (Towards Running Stateful Applications on Nomad)
• https://github.com/hashicorp/nomad/issues/150
• https://twitter.com/kelseyhightower/status/963415653930553345
• https://twitter.com/kelseyhightower/status/963418681148502016
• https://www.joyent.com/blog/dbaas-simplicity-no-lock-in
• https://www.joyent.com/blog/persistent-storage-patterns
• https://thenewstack.io/methods-dealing-container-storage/
• https://techcrunch.com/2015/11/21/i-want-to-run-stateful-containers-too/
References
50

Megabase: How We Containerized Databases at New Relic

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (10)

Semelhante a Megabase: How We Containerized Databases at New Relic

Semelhante a Megabase: How We Containerized Databases at New Relic (20)

Último

Último (20)

Megabase: How We Containerized Databases at New Relic