1.
Seattle | September 16-17, 2019
Automated production deployments
wit HIPAA/HITRUST compliance
MATT LAVIN

2.
Seattle | September 16-17, 2019
I’m an impatient developer
@mdlavin at @LifeOmic

3.
Seattle | September 16-17, 2019
I’m envious of Netflix and Instagram

4.
Seattle | September 16-17, 2019
I work at LifeOmic
where all data is sensitive data

5.
Seattle | September 16-17, 2019
The Sucker’s Choice
• Don’t pick between security or speed
• Get more security and more speedSecurity
Speed
Where you think the boundary is
The actual boundary
You are here

6.
Seattle | September 16-17, 2019
Tools we use
• AWS
• Git (Bitbucket)
• Jenkins
• Jira
• Snyk

7.
Seattle | September 16-17, 2019
From idea to production
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development

8.
Seattle | September 16-17, 2019
Implementation
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
Implementation of change and test automation
on the developers machine
Push change to a Git branch, test automation
runs in Jenkins

9.
Seattle | September 16-17, 2019
Review and deploy
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
After approval, changes are merged into
Git master branch and Jenkins deploys to
AWS dev account
Team members review code and click ‘Approve’

10.
Seattle | September 16-17, 2019
Request production deployment
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
Humans review the new changes being
deployed and approve if the appropriate
processes were followed
Developer creates a Jira issue to request
deployment of specific project and build to
production

11.
Seattle | September 16-17, 2019
Production deployment
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
Jenkins watches for the Jira issue and deploys
the new project version when approved

12.
Seattle | September 16-17, 2019
Process requirements
• Nobody can act alone to modify production
• Changes to production are auditable
• Security scans are run

13.
Seattle | September 16-17, 2019
Automation > Humans
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
Automate review of process compliance
Automate detailed change description
collection for Jira issue

14.
Seattle | September 16-17, 2019
Old Change Management Ticket Flow
Developer runs job to promote
Jira emails sent to approvers
Humans review and approve
Jira ticket moves to implementing
Jira CM issue is created
Deploy job waits for approval
Deploy job resumes

15.
Seattle | September 16-17, 2019
Automation possibilities
Creating the ticket
• What has changed
• Has anything been removed
Reviewing changes
• Bitbucket PRs for all changes
• All PRs were reviewed by peers
• Security scans have been run

16.
Seattle | September 16-17, 2019
New Change Management Ticket Flow
Developer runs job to promote
Jira emails sent to approvers
CM Bot Updates
Jira ticket moves to implementing
Jira CM issue is created
Deploy job waits for approval
Deploy job resumes
Humans Review
Fewer details

17.
Seattle | September 16-17, 2019
CM Bot execution
Jira Issue Created Webhook
CM Description updated
Approval (or not)
Bitbucket / Jenkins / Jira Analysis
CM Verdict in comment

18.
Seattle | September 16-17, 2019
Git Commits
Commit 1 Commit 2 Commit 3 Commit 4 Commit 5
Merge 1 Merge 2 Merge 3
Master Branch
• Every commit must be from a LifeOmic Bitbucket user
• All commits into master are through merge commits
• No Fast Forwards or squashes

19.
Seattle | September 16-17, 2019
Pull requests
PR 2 PR 3PR 1
• Every merge into master must be associated with a Bitbucket pull request
• Every pull request must have approvers who are not the committer

20.
Seattle | September 16-17, 2019
Jenkins Builds
Build 1 Build 2 Build 3
• Every merge into Master starts a Jenkins build for test and deploy

21.
Seattle | September 16-17, 2019
Change Management Jira Issues
CM 1 CM 2 CM 3
• Production deployment requests refer to a Jenkins build

22.
Seattle | September 16-17, 2019
Verifying a promotion request
Commit 4 Commit 5
Merge 3
PR 3
Build 3
CM 1 CM 2 CM 3
Master Branch
• Find previously approved CM issue
• Find all new artifacts associated with the
request
• Verify the processes were followed
• Leave a comment in the CM issue with
with changes detected

23.
Seattle | September 16-17, 2019
Automation
• Good at finding and listing every
single change in the new build
• Good at filling CM tickets with
details
• Very fast
Developers
• Good at remembering what they
did recently
• Good at listing just enough details
to get the CM ticket approved
• Comparatively slow

24.
Seattle | September 16-17, 2019
Automation audit log

25.
Seattle | September 16-17, 2019
Rejected Changes

26.
Seattle | September 16-17, 2019
Automation
• Good at checking every single change
for peer reviews and security scans
• Good at catching edge cases which
can be hard to detect manually
• Very fast
• Reviewing at odd hours
Human Approvers
• Vulnerable to being rushed and
sometimes spot checking changes
• Good at thinking big picture for
security and compliance
• Comparatively slow
• Sleeping and spending time with family

27.
Seattle | September 16-17, 2019
Git merge edge cases
• Detecting any changes in the merge commits
• Handling cases for PRs with multiple committers
• Detecting removed commits
• Squash merges

28.
Seattle | September 16-17, 2019
Changing culture
• Worst case is fallback to human review
• Teams eagerly follow review process to avoid waiting on human approvals
• Security team doesn’t have to play the bad guy, the tool is the bad guy
• Motivation to automate production changes
• Shame for squash merges is a right of passage for new hires

29.
Seattle | September 16-17, 2019
The future
• Detecting risky Terraform changes
• Allowing automatic rollbacks
• More types of security scans

30.
Seattle | September 16-17, 2019
Continuous delivery and HIPAA compliance is possible
• We’ve open-sourced a library to help
• Utilizes JupiterOne for data collection
• https://github.com/JupiterOne/jupiter-change-management-client

31.
Seattle | September 16-17, 2019
Thank you!
Ask me questions
@mdlavin