SlideShare uma empresa Scribd logo
1 de 173
Daniel Liem & Chirayu Krishnappa
Ship Fast & Stable @
Uber Scale
11.1.2016
99.99%
● Have a dedicated Release team
● Aggressive weekly release cadence
● Build cuts (CI) from Master
● Nightly (alpha) vs. Beta / Production builds
● Internal beta dogfooding vs. External Beta testing
● FF (Feature Flagging) wherever possible
● Avoid alphafixes, betafixes, hotfixes & rollbacks
● Soft Upgrades vs. Force Upgrades
Our Process
Staged Rollouts
iOS
Android
Staged Rollouts
iOS
Sanity Tests /
Patches
Android
Sanity Tests /
Patches
Staged Rollouts
iOS
Sanity Tests /
Patches
Upload to iTunes,
enable Testflight
(Weekend dogfooding)
Android
Sanity Tests /
Patches
β channel
(Weekend
dogfooding)
Staged Rollouts
iOS
Sanity Tests /
Patches
Upload to iTunes,
enable Testflight
(Weekend dogfooding)
Wait for
Approval
Android
Sanity Tests /
Patches
β channel
(Weekend
dogfooding)
Staged Rollouts
(1% → 10% →
50% → 100%)
Staged Rollouts
iOS
Sanity Tests /
Patches
Upload to iTunes,
enable Testflight
(Weekend dogfooding)
Launch 100%
(All users at once!)
Wait for
Approval
Android
Sanity Tests /
Patches
β channel
(Weekend
dogfooding)
Staged Rollouts
(1% → 10% →
50% → 100%)
(Partner app only)
Soft Upgrade +
Force Upgrade
How many apps?
How many teams?
Developers!
How long does it take to “deploy
to production”?
Build, sign, and more.
Mostly deterministic.
Submit to App Store for approval
WAIT … !
Approved! Now what?
WAIT … !
Your users will
upgrade…eventually.
http://www.publicdomainpictures.net/view-image.php?image=139317&picture=snail-man
SPEED
Not the speed of your mobile app
SPEED
Speed of deployment
SPEED
Speed of reactions when you
discover issues
SPEED
Speed of rollbacks.
What are rollbacks?
How many versions of your app
are out there?
Adoption across versions
Need for a trusted system
Goal: Develop at full speed!
High quality bar!
Goal: Develop at full speed
High quality bar!
Goal: Develop at full speed
Because slow to deploy/rollback
Signals
Signals
Soon to be 100s of signals
Signals
How can you stay on top?
How do you react to these
signals?
Ticketing system at the core
If there’s a failure that should
block the train, there’s a ticket
for it.
Block specific versions
Verify patched versions
Track mitigation tasks
Weekly reports
Intelligent Subsystems
crash detection at
alpha stages vs.
production
Every alpha crash
gets a ticket
Feature Flagging
Automate to track features per
version and turn off based on
classified crashes
Reminders
E.g. When a new build is ready
(also in-app “upgrade” notifications)
Notifications/Alerts
We’re pushing at 4pm
It’s 2pm and we have new /
unresolved tickets.
Production alerts are separate
e.g. Spike in crashes, E2E,
App Store ratings dropped.
https://pixabay.com/en/go-button-3d-icon-sign-symbol-1067074/
Stage Everything
Rollout to employees
Test flight, alpha, beta channels
Stage Everything
Alpha channel rollout?
Stage it. 1%, 10%, 50% 100%
Stage Everything
In app upgrade prompt?
Stage that too! 1%, 10%, 100%.
In a nutshell…
Deploys are slow
Collect all signals
Ticketing at core
Automated reminders
Ad hoc notifications / alerts
Thank you
Proprietary and confidential © 2016 Uber Technologies, Inc. All rights reserved. No part of this document may be
reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any
information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use
of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise
exempt from disclosure under applicable law. All recipients of this document are notified that the information contained
herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any
way disclose this document or any of the enclosed information to any person other than employees of addressee to the
extent necessary for consultations with authorized personnel of Uber.
Ship Fast & Stable @ Uber Scale
Daniel Liem & Chirayu Krishnappa
Building At Uber Scale
Like a ‘BAUS’
Robbert van Ginkel & Gautam Korlam
11.1.2016
Overview
Challenges with mobile development at scale
Team Size
Build Time
Infrastructure
Improving Developer Experience while tackling scale
Architecture
Workflow
Mobile Scale @ Uber
Several Hundred Mobile Developers
Hundreds of commits daily
50% of code changes every month
Shared modular codebase with hundreds of modules
Team Size
Commits
Architecting for Scale
Code architecture
Features are built as Plugins and shared between apps
Code infrastructure and tooling
Monorepo helps with modularization and sharing
Regressions block the whole team
Always keep master green
Guard as much as possible at compile time
Fail fast
Workflow at Scale
Asynchronous change merging
Submit queue
Stacked Diffs
Run expensive code quality checks pre merge
UI Tests
Deep static analysis - Infer etc.
Performance regressions on real devices - cold start, battery, network etc.
Build Time
Waiting for builds...
With more modules,
come more problems
Build Tools Scaling Issues
Cocoapods
Does not scale well with more targets (15 min pod install time)
Xcode
Incorrect incremental builds (non deterministic and hard to debug)
Xcode project file merge conflicts
Gradle
Does not scale well with large android projects (15 min for a single line change)
Android Studio performance degrades
Building at Scale
Both iOS and Android use Buck to build at Uber
Incremental everywhere
Scale non exponentially as more code is added
Cache immutable state - avoid rebuilding
Transparent Dependency Management
Works well for monorepo
iOS - Clean ~4x faster, Incremental ~20x faster
Android - Clean ~6x faster, Incremental ~30x faster
Remote Build Cache
Infrastructure
Uber’s CI Infrastructure
CI capacity needs increased exponentially
400+ Busy Executors on CI hourly
50k+ CI Jobs run per day
600
400
200
100
CI Executors with Time
Optimizing the CI Pipeline
Perform relevant checks at the right stage
Code Formatting - pre diff
Build, Unit Tests - diff
UI, Static Analysis - pre merge
Use CI resources effectively
Remote build artifact caching
Build in elasticity to meet peak demand
Open Source
Projects to Watch
OkBuck - Gradle plugin that lets you use gradle projects with buck
Buck Http Cache - A distributed build artifact cache service
Coming soon: Swift Support in Buck
https://github.com/uber/buck-http-cache
1https://github.com/uber/okbuck , Slide Deck
Takeaway
Invest in the right build tools early on
Scaling hardware only works till a certain point
Having shared workflow/tools across platforms helps a lot in the long run
Fail earlier and keep master always green
Scaling the Build Process at Uber
Robbert van Ginkel & Gautam Korlam
Thank You
In-product features for release engineers
Christian Legnitto
christian@pocketship.com
App
↑
↑
↑
http://www.pocketship.com
In-product features for release engineers
In-product
Release
Engineers
Product
Engineers
App
Empathy
Agency
Agency
Fix Release Engineering pain
Gain more code context and confidence
Improve Release Engineering processes
Influence company direction
Empathy
Build times and CI turnaround
Tests, test infrastructure, and tooling
Release process overhead
Shipping to the world
Empathy
“One of us”
Specific features
Specific features
Company Product Release
Crash, OOM, hang reporting
Telemetry / analytics
Feature flags (A/B testing)
if (whatever) {
doA();
} else {
doB();
}
Feature flags (A/B testing)
if (whatever) {
doA();
} else {
doB();
}
Feature flags (A/B testing)
Bug reporter
Bug reporter
Bug reporter
Bug reporter
Bug reporter
Bug reporter
Promotion framework
Promotion framework
Promotion framework
Promotion framework
Promotion framework
Test the next version of
Facebook
Get access to new features and bug
fixes before everyone else.
Promotion framework
Promotion framework
Promotion framework
Automatic employee updater
React Native
React Native
JS Bundle
React Native
JS Bundle
JS Bundle
↑
React Native
WebViews
WebViews
Version number scheme
Version number scheme
Version number scheme
v20
Version number scheme
v20
#500
1#500
2#500
3#500
4#500
5#500
6
…
14 APKs
Version number scheme
v20
#500
1#500
2#500
3#500
4#500
5#500
6
…
v21
#504
6#504
7#504
8#504
9#505
0#505
1
…
Version number scheme
Ship production weekly
Slow rollout
Alpha and beta
Internal employee dogfooding
Version number scheme
Version number scheme
100.0.0.20.70
Version number scheme
100.0.0.20.70
Major version
Version number scheme
100.0.0.20.70Major version
Version number scheme
100.0.0.20.70Major version
Hotfix
Version number scheme
100.0.0.20.70Major version Hotfix
Version number scheme
100.0.0.20.70Major version Hotfix
Beta
Version number scheme
100.0.0.20.70Major version Hotfix Beta
Version number scheme
100.0.0.20.70Major version Hotfix
Alpha
Beta
Version number scheme
100.0.0.20.70Major version Hotfix Beta Alpha
Release
Engineers
Product
Engineers
App
Release
Engineers
Product
Engineers
App
Christian Legnitto
christian@pocketship.com
THE DARK SIDE OF
ENTERPRISE SWIFT
Jacek Suliga 🍺 Mobilize @ LinkedIn
Builds Perf
& App Size
Launch
Performance
Maintenance
Cost
Builds Perf
& App Size
Launch
Performance
Maintenance
Cost
8-23 MB
10-30%
20%
slower
Builds Perf
& App Size
Launch
Performance
Maintenance
Cost
WWDC 2016, 406, “Optimizing app startup
time”
Builds Perf
& App Size
Launch
Performance
Maintenance
Cost
0.
5
1.
0
1.
1
1.
2
2.
0
2.
1
Open
Sourc
e
2.
2
2.
3
3.
0
6/14 9/14 10/14 4/15 6/15 9/15 12/15 3/16 6/16 9/16
LinkedIn
Swift 3
migration
party
3.
0
0.
3
Any
Questions?!
Automating Mobile Releases
Rachel Brindle
TERMS
• User: Someone who uses your app.
• Environment: Place where a user can download
your app
• Staging: HockeyApp, Testflight, etc.
• Production: Enterprise MDM, App Store, Play
Store
• Automated Deployment: Using CI to push to
environment
z
Processz
WHY
• Frees up the deployer
to do other things
• Consistent deploys
• Shorter release cycle
• Documents
$ git push
pushed g4edeff4
$ # wait
$ rake check_if_deployed
Latest is g4edeff4
z
• Make sure tests pass
• Build for release
• Gather metadata
• Screenshots
• Release Notes
• Other?
• Upload to environment
• Make sure tests pass
• Build for release
• Gather metadata
• Screenshots
• Release Notes
• Other?
• Upload to environment

Mais conteúdo relacionado

Mais procurados

Agile Testing Strategy
Agile Testing StrategyAgile Testing Strategy
Agile Testing Strategy
tharindakasun
 
Automation testing strategy, approach & planning
Automation testing  strategy, approach & planningAutomation testing  strategy, approach & planning
Automation testing strategy, approach & planning
SivaprasanthRentala1975
 
Reinventing our QA roles for Quality Engineering
Reinventing our QA roles for Quality EngineeringReinventing our QA roles for Quality Engineering
Reinventing our QA roles for Quality Engineering
Antoine Craske
 
Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8
a34sharm
 

Mais procurados (20)

Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle
 
Automated testing with Cypress
Automated testing with CypressAutomated testing with Cypress
Automated testing with Cypress
 
Visual regression with applitools eyes
Visual regression with applitools eyesVisual regression with applitools eyes
Visual regression with applitools eyes
 
"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl
 
Building a Test Automation Strategy for Success
Building a Test Automation Strategy for SuccessBuilding a Test Automation Strategy for Success
Building a Test Automation Strategy for Success
 
Developing a test automation strategy by Brian Bayer
Developing a test automation strategy by Brian BayerDeveloping a test automation strategy by Brian Bayer
Developing a test automation strategy by Brian Bayer
 
Agile Testing Strategy
Agile Testing StrategyAgile Testing Strategy
Agile Testing Strategy
 
Sonar Review
Sonar ReviewSonar Review
Sonar Review
 
Automation testing strategy, approach & planning
Automation testing  strategy, approach & planningAutomation testing  strategy, approach & planning
Automation testing strategy, approach & planning
 
What Is Accessibility Testing?
What Is Accessibility Testing?What Is Accessibility Testing?
What Is Accessibility Testing?
 
Hiring or Growing Right Agile Coach by Lyssa Adkins and Michael Spayd
Hiring or Growing Right Agile Coach by Lyssa Adkins and Michael SpaydHiring or Growing Right Agile Coach by Lyssa Adkins and Michael Spayd
Hiring or Growing Right Agile Coach by Lyssa Adkins and Michael Spayd
 
Cypress e2e automation testing - day1 intor by: Hassan Hameed
Cypress e2e automation testing -  day1 intor by: Hassan HameedCypress e2e automation testing -  day1 intor by: Hassan Hameed
Cypress e2e automation testing - day1 intor by: Hassan Hameed
 
Agile Testing Framework - The Art of Automated Testing
Agile Testing Framework - The Art of Automated TestingAgile Testing Framework - The Art of Automated Testing
Agile Testing Framework - The Art of Automated Testing
 
QA Best Practices in Agile World_new
QA Best Practices in Agile World_newQA Best Practices in Agile World_new
QA Best Practices in Agile World_new
 
Agile QA and Testing process
Agile QA and Testing processAgile QA and Testing process
Agile QA and Testing process
 
Cypress
CypressCypress
Cypress
 
Reinventing our QA roles for Quality Engineering
Reinventing our QA roles for Quality EngineeringReinventing our QA roles for Quality Engineering
Reinventing our QA roles for Quality Engineering
 
Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8
 
Automation testing
Automation testingAutomation testing
Automation testing
 
Tips for Writing Better Charters for Exploratory Testing Sessions by Michael...
 Tips for Writing Better Charters for Exploratory Testing Sessions by Michael... Tips for Writing Better Charters for Exploratory Testing Sessions by Michael...
Tips for Writing Better Charters for Exploratory Testing Sessions by Michael...
 

Semelhante a Uber mobility - Build & Release

Our DevOps Journey: 6 Month Waterfalls to 1 Hour Code Deploys
Our DevOps Journey: 6 Month Waterfalls to 1 Hour Code DeploysOur DevOps Journey: 6 Month Waterfalls to 1 Hour Code Deploys
Our DevOps Journey: 6 Month Waterfalls to 1 Hour Code Deploys
Dynatrace
 

Semelhante a Uber mobility - Build & Release (20)

Continuous testing at scale
Continuous testing at scaleContinuous testing at scale
Continuous testing at scale
 
Test Automation at the Speed of Agile: Making It Work Every Build
Test Automation at the Speed of Agile: Making It Work Every BuildTest Automation at the Speed of Agile: Making It Work Every Build
Test Automation at the Speed of Agile: Making It Work Every Build
 
Agile Tools for Mobile
Agile Tools for MobileAgile Tools for Mobile
Agile Tools for Mobile
 
Our DevOps Journey: 6 Month Waterfalls to 1 Hour Code Deploys
Our DevOps Journey: 6 Month Waterfalls to 1 Hour Code DeploysOur DevOps Journey: 6 Month Waterfalls to 1 Hour Code Deploys
Our DevOps Journey: 6 Month Waterfalls to 1 Hour Code Deploys
 
Enhancing your Test automation Scenario Coverage Using Selenium by Eran Kinsb...
Enhancing your Test automation Scenario Coverage Using Selenium by Eran Kinsb...Enhancing your Test automation Scenario Coverage Using Selenium by Eran Kinsb...
Enhancing your Test automation Scenario Coverage Using Selenium by Eran Kinsb...
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at Etsy
 
Continuous, continuous, continuous
Continuous, continuous, continuousContinuous, continuous, continuous
Continuous, continuous, continuous
 
Selenium conference, 2016
Selenium conference, 2016Selenium conference, 2016
Selenium conference, 2016
 
Webinar: Automate Your Environment Provisioning for Mobile App Development
Webinar: Automate Your Environment Provisioning for Mobile App Development Webinar: Automate Your Environment Provisioning for Mobile App Development
Webinar: Automate Your Environment Provisioning for Mobile App Development
 
Device lab trials and tribulations
Device lab trials and tribulationsDevice lab trials and tribulations
Device lab trials and tribulations
 
Appium vs Espresso and XCUI Test
Appium vs Espresso and XCUI TestAppium vs Espresso and XCUI Test
Appium vs Espresso and XCUI Test
 
[Srijan Wednesday Webinar] Mastering Mobile Test Automation with Appium
[Srijan Wednesday Webinar] Mastering Mobile Test Automation with Appium[Srijan Wednesday Webinar] Mastering Mobile Test Automation with Appium
[Srijan Wednesday Webinar] Mastering Mobile Test Automation with Appium
 
Integrate Your Test Automation Tools for More Power
Integrate Your Test Automation Tools for More PowerIntegrate Your Test Automation Tools for More Power
Integrate Your Test Automation Tools for More Power
 
Mobile App Quality Roadmap for DevTest Teams
Mobile App Quality Roadmap for DevTest TeamsMobile App Quality Roadmap for DevTest Teams
Mobile App Quality Roadmap for DevTest Teams
 
Test Automation for Mobile Applications
Test Automation for Mobile ApplicationsTest Automation for Mobile Applications
Test Automation for Mobile Applications
 
Mobitop
MobitopMobitop
Mobitop
 
Mobitop
MobitopMobitop
Mobitop
 
Mobitop
MobitopMobitop
Mobitop
 
Mobitop
MobitopMobitop
Mobitop
 
Discover the power of QA automation testing
Discover the power of QA automation testingDiscover the power of QA automation testing
Discover the power of QA automation testing
 

Último

Obat Penggugur Kandungan Di Apotik Kimia Farma (087776558899)
Obat Penggugur Kandungan Di Apotik Kimia Farma (087776558899)Obat Penggugur Kandungan Di Apotik Kimia Farma (087776558899)
Obat Penggugur Kandungan Di Apotik Kimia Farma (087776558899)
Cara Menggugurkan Kandungan 087776558899
 

Último (8)

Satara Call girl escort *74796//13122* Call me punam call girls 24*7hour avai...
Satara Call girl escort *74796//13122* Call me punam call girls 24*7hour avai...Satara Call girl escort *74796//13122* Call me punam call girls 24*7hour avai...
Satara Call girl escort *74796//13122* Call me punam call girls 24*7hour avai...
 
Android Application Components with Implementation & Examples
Android Application Components with Implementation & ExamplesAndroid Application Components with Implementation & Examples
Android Application Components with Implementation & Examples
 
Mobile App Penetration Testing Bsides312
Mobile App Penetration Testing Bsides312Mobile App Penetration Testing Bsides312
Mobile App Penetration Testing Bsides312
 
Mobile Application Development-Components and Layouts
Mobile Application Development-Components and LayoutsMobile Application Development-Components and Layouts
Mobile Application Development-Components and Layouts
 
Mobile Application Development-Android and It’s Tools
Mobile Application Development-Android and It’s ToolsMobile Application Development-Android and It’s Tools
Mobile Application Development-Android and It’s Tools
 
Mobile Application Development- Configuration and Android Installation
Mobile Application Development- Configuration and Android InstallationMobile Application Development- Configuration and Android Installation
Mobile Application Development- Configuration and Android Installation
 
Bromazolam CAS 71368-80-4 high quality opiates, Safe transportation, 99% pure
Bromazolam CAS 71368-80-4 high quality opiates, Safe transportation, 99% pureBromazolam CAS 71368-80-4 high quality opiates, Safe transportation, 99% pure
Bromazolam CAS 71368-80-4 high quality opiates, Safe transportation, 99% pure
 
Obat Penggugur Kandungan Di Apotik Kimia Farma (087776558899)
Obat Penggugur Kandungan Di Apotik Kimia Farma (087776558899)Obat Penggugur Kandungan Di Apotik Kimia Farma (087776558899)
Obat Penggugur Kandungan Di Apotik Kimia Farma (087776558899)
 

Uber mobility - Build & Release