SlideShare a Scribd company logo
1 of 39
Download to read offline
Using NGiNX for release
automation at The Atlantic
About the speakers
Mike Howsden is the DevOps Lead at
The Atlantic
Frankie Dintino is a Senior Full-Stack
Developer at The Atlantic
One Virtual Machine
Nearly Unlimited QA environments
Our development workflow
• Two week sprints
• New code is committed to a branch in git
• When coding is complete, a Pull Request is opened in GitHub for review
• Branch can be staged onto a beta server for QA
• PR is merged into a branch named develop when code review and QA is
complete
• The develop branch is merged into the master branch when it's time for
a deployment
Discontinuous Integration (the old way)
ssh betaserver

cd /www/beta5

source bin/activate

git fetch origin && git reset --hard && git checkout -t origin/my-branch

pip install --upgrade -r requirements.txt

(cd frontend && npm install && gulp build)

importdb atl_db_beta5

django-admin collectstatic -l --noinput

touch wsgi.py

ohpleasegod --help
• Tedious!
Discontinuous Integration
Why was this bad?
• Tedious!
• Error prone!
Discontinuous Integration
Why was this bad?
Why was this bad?
Discontinuous Integration
• Tedious!
• Error prone!
• Stateful!
Why was this bad?
Discontinuous Integration
• Tedious!
• Error prone!
• Stateful!
• Enough instances to waste resources, not enough to ensure
one is always available
Why was this bad?
Discontinuous Integration
• Tedious!
• Error prone!
• Stateful!
• Enough instances to waste resources, not enough to ensure
one is always available
• Worst of all, NGINX played only a minor role
We can do better!
Business Need
• Developers need to easily stage their work for peer and
stakeholder review.
•
Beta architecture overview
Infinite* beta environments—components
• Jenkins
• Reflinks
• supervisord managing uWSGI emperor mode
• nginx-mod-passenger for NodeJS apps
• NGiNX and uWSGI socket files
• Database snapshots
• External proxy
* ish—as many as we need
New process
Two common use-cases, with different requirements:
1. Simple bug fixes and minor features
2. Long-running feature branches
New process—typical setup
Open a pull request
Jenkins build (roughly 10 minutes)
~10:00
1
2
3
Using NGiNX for release automation at The Atlantic
Using NGiNX for release automation at The Atlantic
New process—“named betas”
• Uses a distinct, persistent copy of the database (to allow for
schema changes and feature-specific data)
• Accessible outside the internal network with authentication
New process—“named betas”
Push out a branch: beta/my-feature
Jenkins build (roughly 10 minutes)
~10:00
1
2
3
Business Value
• Efficiency: Cuts down on developer time spent on tasks
unrelated to business goals.
• Morale: Automates repetitive/monotonous work.
• Improves Coverage/Consistency: No barriers to manual
testing, even for trivial changes. Each build environment is
created with the exact same process.
Infinite* beta environments—components
• Jenkins
• Reflinks
• supervisord managing uWSGI emperor mode
• nginx-mod-passenger for NodeJS apps
• NGiNX and uWSGI socket files
• Database snapshots
• External proxy
* ish—as many as we need
Jenkins—stage beta
Jenkinsfile
• A declarative build and deployment script, committed to
source control in the root of the repo
• Hooked into github notifications for pull request opens and
branch pushes
• Reasonably powerful control over conditional logic,
parallelization, pass/fail conditions, etc.
Jenkinsfile
pipeline {

// ...

stage('stage beta') {

when { anyOf {

expression { BRANCH_NAME ==~ /PR-[0-9]+/ } // a pull request

branch 'develop' // development branch

expression { BRANCH_NAME.startsWith('beta/') } // 'beta/feature'

} }

steps {

// Initialize database

sh """ ssh devdbserver "create_beta_db $BRANCH_NAME" """

// If the workspace doesn't exist, create a reflink copy

sh """ssh betaserver 

"[ -d ${WORKSPACE} ] || cp -ar --reflink=auto 

/www/builds/theatlantic/develop.base ${WORKSPACE}" """

// ...continued
Jenkinsfile (continued)
// ...

stage('stage beta') {

// ...

steps {

// Copy current build to remote workspace dir

sh """rsync -acvzhO --delete 

--exclude "*.pyc" --exclude .git --exclude ... 

$WORKSPACE/ betaserver:$WORKSPACE/ """



// Create wsgi files, which will initialize uWSGI emperor processes

sh """ssh betaserver "$WORKSPACE/support/beta/create_wsgi.py" """

}

}
Infinite* beta environments—components
• Jenkins
• Reflinks
• supervisord managing uWSGI emperor mode
• nginx-mod-passenger for NodeJS apps
• NGiNX and uWSGI socket files
• Database snapshots
• External proxy
* ish—as many as we need
Jenkins—stage beta
• Initializes with a reflink copy of a previous build to save on
disk space (1.2G => 3M)
• Uses rsync to copy the newly created environment to the
betas server over top of this copy
Reflinks (copy-on-write)
• BTRFS on debian or ubuntu
• XFS on RHEL, CentOS, or Fedora
• mkfs.xfs must be called with reflink=1
• cp -ar --reflink=auto
• only uses the space required for differences between builds
• considered “experimental”—requires custom-built kernel
(may be enabled by default in Fedora 29)
Infinite* beta environments—components
• Jenkins
• Reflinks
• supervisord managing uWSGI emperor mode
• nginx-mod-passenger for NodeJS apps
• NGiNX and uWSGI socket files
• Database snapshots
• External proxy
* ish—as many as we need
uWSGI emperor + supervisord
[program:zerg_server]

command = /sbin/uwsgi

--master --thunder-lock

--emperor "/www/builds/theatlantic/*/uwsgi/master.*.ini"

autorestart = true



[program:workers]

command = /sbin/uwsgi

--master --thunder-lock

--emperor "/www/builds/theatlantic/*/uwsgi/workers.*.ini"

autorestart = true
Infinite* beta environments—components
uWSGI zerg master uWSGI zerg workers
uWSGI emperor + supervisord
; /www/builds/theatlantic/pr-1234/uwsgi/master.theatlantic.ini

[uwsgi]

; %n: current config filename, without the .ini extension

; Strip off the leading 'master.' part of the filename to determine the app.

app = @(exec:///usr/bin/perl -e '$_ = "%n"; s/^master.//g; print')



subdomain = %3 ; the 4th path component (0-indexed); eg pr-1234



thunder-lock = true

master = true

processes = 0



http = /var/run/uwsgi/%(app)-%(subdomain).sock

zerg-server = /var/run/uwsgi/%(app)-%(subdomain)-workers.sock
uWSGI zerg server
; /www/builds/theatlantic/pr-1234/uwsgi/master.theatlantic.ini

[uwsgi]

; %n: current config filename, without the .ini extension

; Strip off the leading 'master.' part of the filename to determine the app.

app = @(exec:///usr/bin/perl -e '$_ = "%n"; s/^master.//g; print')



subdomain = %3 ; the 4th path component (0-indexed); eg pr-1234



thunder-lock = true

master = true

processes = 0



http = /var/run/uwsgi/%(app)-%(subdomain).sock

zerg-server = /var/run/uwsgi/%(app)-%(subdomain)-workers.sockzerg-server = /var/run/uwsgi/%(app)-%(subdomain)-workers.sock
uWSGI zerg workers
; /www/builds/theatlantic/pr-1234/uwsgi/worker.theatlantic.ini

[uwsgi]

; attach to zerg pool

zerg = /var/run/uwsgi/%(app)-%(subdomain)-workers.sock



cheaper-algo = busyness

cheaper = 1 ; minimum number of workers to keep at all times

cheaper-initial = 2 ; starts with minimal workers

cheaper-step = 2 ; spawn at most 2 workers at a time

cheaper-overload = 30 ; seconds between busyness checks

cheaper-busyness-multiplier = 50

cheaper-busyness-penalty = 2



; how many requests are in backlog before quick response triggered

cheaper-busyness-backlog-alert = 33

cheaper-busyness-backlog-step = 1

idle = 86400 ; workers shut down if the beta is idle for a day



wsgi-file = %(env_dir)/apps/wsgi.py

env = DJANGO_SETTINGS_MODULE=settings.%(app).live
NGiNX, proxying to uWSGI socket files
server {
server_name "~^(?<subdomain>.+).beta.theatlantic.com$";
root /www/builds/theatlantic/$subdomain;



location / {

try_files /assets/$uri @django;

}



location @django { 

internal;

include includes/proxypass.conf;

proxy_pass http://unix:/var/run/uwsgi/theatlantic-$subdomain.sock;

}

}
nginx-mod-passenger for nodejs
server {

server_name "~^(?<subdomain>.+).ampbeta.theatlantic.com$";



root /www/builds/amp/${subdomain}/public;



passenger_enabled on;

passenger_app_type node;

passenger_app_root /www/builds/amp/${subdomain};

passenger_restart_dir /www/builds/amp/${subdomain};

passenger_startup_file dist/host/bin/www.js;

}
Questions?
Thank you!
github.com/theatlantic/nginxconf-2018
frankie@theatlantic.com
mhowsden@theatlantic.com


More Related Content

Recently uploaded

Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 

Recently uploaded (20)

Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 

Featured

Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at WorkGetSmarter
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...DevGAMM Conference
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationErica Santiago
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellSaba Software
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming LanguageSimplilearn
 
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...Palo Alto Software
 
9 Tips for a Work-free Vacation
9 Tips for a Work-free Vacation9 Tips for a Work-free Vacation
9 Tips for a Work-free VacationWeekdone.com
 

Featured (20)

Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming Language
 
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
 
9 Tips for a Work-free Vacation
9 Tips for a Work-free Vacation9 Tips for a Work-free Vacation
9 Tips for a Work-free Vacation
 

Using NGiNX for release automation at The Atlantic

  • 1. Using NGiNX for release automation at The Atlantic
  • 2. About the speakers Mike Howsden is the DevOps Lead at The Atlantic Frankie Dintino is a Senior Full-Stack Developer at The Atlantic
  • 3. One Virtual Machine Nearly Unlimited QA environments
  • 4. Our development workflow • Two week sprints • New code is committed to a branch in git • When coding is complete, a Pull Request is opened in GitHub for review • Branch can be staged onto a beta server for QA • PR is merged into a branch named develop when code review and QA is complete • The develop branch is merged into the master branch when it's time for a deployment
  • 5. Discontinuous Integration (the old way) ssh betaserver
 cd /www/beta5
 source bin/activate
 git fetch origin && git reset --hard && git checkout -t origin/my-branch
 pip install --upgrade -r requirements.txt
 (cd frontend && npm install && gulp build)
 importdb atl_db_beta5
 django-admin collectstatic -l --noinput
 touch wsgi.py
 ohpleasegod --help
  • 7. • Tedious! • Error prone! Discontinuous Integration Why was this bad?
  • 8. Why was this bad? Discontinuous Integration • Tedious! • Error prone! • Stateful!
  • 9. Why was this bad? Discontinuous Integration • Tedious! • Error prone! • Stateful! • Enough instances to waste resources, not enough to ensure one is always available
  • 10. Why was this bad? Discontinuous Integration • Tedious! • Error prone! • Stateful! • Enough instances to waste resources, not enough to ensure one is always available • Worst of all, NGINX played only a minor role
  • 11. We can do better!
  • 12. Business Need • Developers need to easily stage their work for peer and stakeholder review. •
  • 14. Infinite* beta environments—components • Jenkins • Reflinks • supervisord managing uWSGI emperor mode • nginx-mod-passenger for NodeJS apps • NGiNX and uWSGI socket files • Database snapshots • External proxy * ish—as many as we need
  • 15. New process Two common use-cases, with different requirements: 1. Simple bug fixes and minor features 2. Long-running feature branches
  • 16. New process—typical setup Open a pull request Jenkins build (roughly 10 minutes) ~10:00 1 2 3
  • 19. New process—“named betas” • Uses a distinct, persistent copy of the database (to allow for schema changes and feature-specific data) • Accessible outside the internal network with authentication
  • 20. New process—“named betas” Push out a branch: beta/my-feature Jenkins build (roughly 10 minutes) ~10:00 1 2 3
  • 21. Business Value • Efficiency: Cuts down on developer time spent on tasks unrelated to business goals. • Morale: Automates repetitive/monotonous work. • Improves Coverage/Consistency: No barriers to manual testing, even for trivial changes. Each build environment is created with the exact same process.
  • 22. Infinite* beta environments—components • Jenkins • Reflinks • supervisord managing uWSGI emperor mode • nginx-mod-passenger for NodeJS apps • NGiNX and uWSGI socket files • Database snapshots • External proxy * ish—as many as we need
  • 24. Jenkinsfile • A declarative build and deployment script, committed to source control in the root of the repo • Hooked into github notifications for pull request opens and branch pushes • Reasonably powerful control over conditional logic, parallelization, pass/fail conditions, etc.
  • 25. Jenkinsfile pipeline {
 // ...
 stage('stage beta') {
 when { anyOf {
 expression { BRANCH_NAME ==~ /PR-[0-9]+/ } // a pull request
 branch 'develop' // development branch
 expression { BRANCH_NAME.startsWith('beta/') } // 'beta/feature'
 } }
 steps {
 // Initialize database
 sh """ ssh devdbserver "create_beta_db $BRANCH_NAME" """
 // If the workspace doesn't exist, create a reflink copy
 sh """ssh betaserver 
 "[ -d ${WORKSPACE} ] || cp -ar --reflink=auto 
 /www/builds/theatlantic/develop.base ${WORKSPACE}" """
 // ...continued
  • 26. Jenkinsfile (continued) // ...
 stage('stage beta') {
 // ...
 steps {
 // Copy current build to remote workspace dir
 sh """rsync -acvzhO --delete 
 --exclude "*.pyc" --exclude .git --exclude ... 
 $WORKSPACE/ betaserver:$WORKSPACE/ """
 
 // Create wsgi files, which will initialize uWSGI emperor processes
 sh """ssh betaserver "$WORKSPACE/support/beta/create_wsgi.py" """
 }
 }
  • 27. Infinite* beta environments—components • Jenkins • Reflinks • supervisord managing uWSGI emperor mode • nginx-mod-passenger for NodeJS apps • NGiNX and uWSGI socket files • Database snapshots • External proxy * ish—as many as we need
  • 28. Jenkins—stage beta • Initializes with a reflink copy of a previous build to save on disk space (1.2G => 3M) • Uses rsync to copy the newly created environment to the betas server over top of this copy
  • 29. Reflinks (copy-on-write) • BTRFS on debian or ubuntu • XFS on RHEL, CentOS, or Fedora • mkfs.xfs must be called with reflink=1 • cp -ar --reflink=auto • only uses the space required for differences between builds • considered “experimental”—requires custom-built kernel (may be enabled by default in Fedora 29)
  • 30. Infinite* beta environments—components • Jenkins • Reflinks • supervisord managing uWSGI emperor mode • nginx-mod-passenger for NodeJS apps • NGiNX and uWSGI socket files • Database snapshots • External proxy * ish—as many as we need
  • 31. uWSGI emperor + supervisord [program:zerg_server]
 command = /sbin/uwsgi
 --master --thunder-lock
 --emperor "/www/builds/theatlantic/*/uwsgi/master.*.ini"
 autorestart = true
 
 [program:workers]
 command = /sbin/uwsgi
 --master --thunder-lock
 --emperor "/www/builds/theatlantic/*/uwsgi/workers.*.ini"
 autorestart = true
  • 32. Infinite* beta environments—components uWSGI zerg master uWSGI zerg workers
  • 33. uWSGI emperor + supervisord ; /www/builds/theatlantic/pr-1234/uwsgi/master.theatlantic.ini
 [uwsgi]
 ; %n: current config filename, without the .ini extension
 ; Strip off the leading 'master.' part of the filename to determine the app.
 app = @(exec:///usr/bin/perl -e '$_ = "%n"; s/^master.//g; print')
 
 subdomain = %3 ; the 4th path component (0-indexed); eg pr-1234
 
 thunder-lock = true
 master = true
 processes = 0
 
 http = /var/run/uwsgi/%(app)-%(subdomain).sock
 zerg-server = /var/run/uwsgi/%(app)-%(subdomain)-workers.sock
  • 34. uWSGI zerg server ; /www/builds/theatlantic/pr-1234/uwsgi/master.theatlantic.ini
 [uwsgi]
 ; %n: current config filename, without the .ini extension
 ; Strip off the leading 'master.' part of the filename to determine the app.
 app = @(exec:///usr/bin/perl -e '$_ = "%n"; s/^master.//g; print')
 
 subdomain = %3 ; the 4th path component (0-indexed); eg pr-1234
 
 thunder-lock = true
 master = true
 processes = 0
 
 http = /var/run/uwsgi/%(app)-%(subdomain).sock
 zerg-server = /var/run/uwsgi/%(app)-%(subdomain)-workers.sockzerg-server = /var/run/uwsgi/%(app)-%(subdomain)-workers.sock
  • 35. uWSGI zerg workers ; /www/builds/theatlantic/pr-1234/uwsgi/worker.theatlantic.ini
 [uwsgi]
 ; attach to zerg pool
 zerg = /var/run/uwsgi/%(app)-%(subdomain)-workers.sock
 
 cheaper-algo = busyness
 cheaper = 1 ; minimum number of workers to keep at all times
 cheaper-initial = 2 ; starts with minimal workers
 cheaper-step = 2 ; spawn at most 2 workers at a time
 cheaper-overload = 30 ; seconds between busyness checks
 cheaper-busyness-multiplier = 50
 cheaper-busyness-penalty = 2
 
 ; how many requests are in backlog before quick response triggered
 cheaper-busyness-backlog-alert = 33
 cheaper-busyness-backlog-step = 1
 idle = 86400 ; workers shut down if the beta is idle for a day
 
 wsgi-file = %(env_dir)/apps/wsgi.py
 env = DJANGO_SETTINGS_MODULE=settings.%(app).live
  • 36. NGiNX, proxying to uWSGI socket files server { server_name "~^(?<subdomain>.+).beta.theatlantic.com$"; root /www/builds/theatlantic/$subdomain;
 
 location / {
 try_files /assets/$uri @django;
 }
 
 location @django { 
 internal;
 include includes/proxypass.conf;
 proxy_pass http://unix:/var/run/uwsgi/theatlantic-$subdomain.sock;
 }
 }
  • 37. nginx-mod-passenger for nodejs server {
 server_name "~^(?<subdomain>.+).ampbeta.theatlantic.com$";
 
 root /www/builds/amp/${subdomain}/public;
 
 passenger_enabled on;
 passenger_app_type node;
 passenger_app_root /www/builds/amp/${subdomain};
 passenger_restart_dir /www/builds/amp/${subdomain};
 passenger_startup_file dist/host/bin/www.js;
 }