CI/CD Pipeline Security: Advanced Continuous Delivery Best Practices: Security Week at the San Francisco Loft Continuous delivery (CD) enables teams to be more agile and quickens the pace of innovation. Too often, however, teams adopt CD without putting the right safety mechanisms in place. In this talk, we discuss opportunities for you to transform your software release process into a safer one. We explore various DevOps best practices, showcasing sample applications and code with AWS CodePipeline and AWS CodeDeploy. We discuss how to set up delivery pipelines with nonproduction testing stages, failure cases, rollbacks, redundancy, canary testing and blue/green deployments, and monitoring. We'll discuss continuous delivery practices for deploying to Amazon EC2, AWS Lambda, and Containers (such as Amazon ECS or AWS Fargate). Level: 200 Speaker: Leo Zhadanovsky - Principal Solutions Architect, Cloudstart, AWS

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Advanced Continuous Delivery
Best Practices
Leo Zhadanovsky
Principal Solutions Architect
AWS

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What to you’ll learn from the Session
• Some of Amazon’s continuous deployment best practices
• How and why you should employ these practices yourself
• Where and how can you integrate security testing into your CI/CD
pipelines
• Leaving positive feedback in the post-session survey feels good.

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is Continuous Delivery?
Source Build
Pre-
Test
Deploy
Post-
Test

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Basic Continuous Delivery Best Practices
• Versioned Source
• Automated build
• Automated deployments
• Deploy to > 1 instance
• Unit tests
• Integration tests
• Continuous Delivery
• Operations dashboard
Source
Build
Deploy to
Integration Stack
Integration Tests
Deploy to
Production

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Tools used in this talk
Monitoring
Amazon CloudWatch
Software Development
Amazon SNS
AWS Lambda
Security
Amazon Inspector
Continuous Delivery
AWS CodeDeploy
Continuous Deployment
AWS CodePipeline
Testing
AWS CodeBuild

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Code Services
Source Build Test Deploy Monitor
AWS CodeBuild +
Third Party
Software Release Steps:
AWS CodeCommit AWS CodeBuild AWS CodeDeploy
AWS CodePipeline
AWS
CodeStar
AWS X-Ray
Amazon
CloudWatch

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring
& Logging
AWS DevOps Portfolio
Software Development and
Continuous Delivery Toolchain
Infrastructure
as Code

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Security Portfolio

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MyApp
CodeCommit
Source
Build
CodeCommit
Build
DeployToInteg
CodeDeploy
Integration
IntegTest
End2EndTester
DeployToProd
CodeDeploy
Production
Source
Build
Deploy to
Integration Stack
Integration Tests
Deploy to
Production
Model the release process in CodePipeline
Pipeline Run
Action
Stage
Pipeline
Source change
• starts a run; and
• creates an artifact to be used by
other actions.
Change 1

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy process: Starting point
MyApp
CodeCommit
Source
Build
Build
Build
DeployToInteg
CodeDeploy
Integration
IntegTest
End2EndTester
DeployToProd
CodeDeploy
Production
Want to dive deep into to do this with Containers and Serverless?
Check the session catalog or YouTube for:
DEV309-R - CI/CD for Serverless and Containerized Applications

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
An automated pipeline should be…
• Defined as code
• Checked into a version control repository, such as AWS CodeCommit
• Able to allow for extensibility through other AWS services or 3rd party
tools
• Able to provide FAST feedback on the success and failure of pipeline
executions
• Remove undifferentiated heavy lifting associated with releasing code
to production, while also adding in guard rails for developers

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Opportunities for automation
• Continuous Integration Processes: Builds, Integration Tests, UI Testing,
and etc.
• Health Checks
• Security Tests
• Static Analysis
• Vulnerability Scanning
• Pipeline Compliance Testing
• Application Tests
• Synthetic User Tests and Application Performance Monitoring
• Notifications and Alerts
• AWS CloudWatch Alarms and 3rd party tools, such as Splunk, Datadog
• SNS, Slack, Pagerduty, Etc.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Build and unit tests
1. Trigger pipeline
on source change
2. Build and unit
tests
3. Deploy to
integration
environment
4. Execute UI tests
5. Execute
integration tests
Tests
Source
MyAppSource
CodeCommit
Build
BuildAndUnitTests
CodeBuild
IntegrationDeploy
CodeDeploy
TestOnChrome
CodeBuild
TestOnFireFox
CodeBuild
IntegTest
End2EndTester

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 2: Notify on failed build and test
Change 1
CloudWatch
Events
(Failed Action)
Tests
Source
MyAppSource
CodeCommit
Build
BuildAndUnitTests
CodeBuild
IntegrationDeploy
CodeDeploy
TestOnChrome
CodeBuild
TestOnFireFox
CodeBuild
IntegTest
End2EndTester
Change 2
Lambda Function
NotifySlackOnPipelineActionFailure()

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automate your security testing…
• Integrate it directly into your Pipelines
• If your pipeline produces AMIs, Docker containers, etc… scan them
with tools like Inspector, Clair, and Twistlock
• If the tool has an API, you can use a custom Lambda action in
CodePipeline to trigger it
• If a security test fails, Pipeline stops, code doesn’t make it to
production

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Security Tests - Docker
1. Trigger pipeline
on source change
2. Trigger security
test against
container
3. Deploy to
integration
environment
4. Execute UI tests
5. Execute
integration tests
Tests
Source
MyAppSource
ECR
SecurityTests
TriggerClair
Lambda
IntegrationDeploy
CodeDeploy
TestOnChrome
CodeBuild
TestOnFireFox
CodeBuild
IntegTest
End2EndTester

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 2: Notify on failed security test
Change 1
CloudWatch
Events
(Failed Action)
Tests
Source
MyAppSource
ECR
SecurityTests
TriggerClair
Lambda
IntegrationDeploy
CodeDeploy
TestOnChrome
CodeBuild
TestOnFireFox
CodeBuild
IntegTest
End2EndTester
Lambda Function
NotifySlackOnPipelineActionFailure()

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security Tests - AMIs
1. Trigger pipeline
on source change
2. Build AMI
3. Launch Test
Instance With AMI
4. Trigger Inspector
5. Terminate
Instance
SecurityTests
Source
MyAppSource
CodeCommit
BuildAMI
AMIBuilder
CodeBuild
LaunchTestInstance
Lambda
TestAMI
Lambda
KillTestInstance
Lambda
https://aws.amazon.com/blogs/devops/
how-to-create-an-ami-builder-with-aws-
codebuild-and-hashicorp-packer/

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security Tests - CodeBuild
1. Trigger pipeline
on source change
2. Run Tests In
CodeBuild
3. Deploy
DeployToStaging
Source
MyAppSource
CodeCommit
SecurityTests
ComplianceTest
CodeBuild
Deploy
CodeDeploy
Sample security test script

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security Tool Options
• Containers
• Clair – https://github.com/coreos/clair
• Twistlock – https://www.twistlock.com
• Instances
• Amazon Inspector – https://aws.amazon.com/inspector
• Custom Testing
• AWS CodeBuild – https://aws.amazon.com/codebuild
• AWS Lambda – https://aws.amazon.com/lambda

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Block non-compliant pipelines
• Introducing changes or even new pipelines can cause serious problems
• Downtime risk, reputation and financial loss.
• Leverage AWS Config to ensure pipeline compliance before allowing
them to deploy to production environments

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Adding safety with AWS Config Rules
Build Config Rules
• These alert when pipelines are not configured up to company best
practices
Building a Pipeline which blocks production pushes on non-compliant
pipelines
• Use Approvals to pause production deployments
• Lambda to automatically approve when pipeline is compliant

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approval Action-based Pipeline safety check
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
safetyCheck
Approval
ProductionDeploy
CodeDeploy
Lambda Function
safetyCheck
Change 1
SNS Topic

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Config Rules-based Pipeline safety check

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Config Rules-based Pipeline safety check
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
DeployToProd
MyApp
CodeDeploy
CloudWatch
Event (event-
based)
Lambda Function
disablePushtoProduction

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary: Safety and Compliance Checks in Your
Pipelines
• Use Config Rules and CloudWatch Events to automatically remediate
non-compliant Pipelines
• Add common action to all pipelines
• Provide Developers a Best Practices Pipeline to start from with
CloudFormation Templates or Service Catalog
• Implement with Approval actions in CodePipeline

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
safetyCheck
Approval
Release and deploy: Gates
Synthetic Traffic
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Builds on top of our foundation of automation
• Purpose built to verify that a service is working
after a new deployment.
• Helps to avoid needing to do this manually.
Manage Deployment Health

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2
Rolling deployments – success

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2
Rolling deployments – fail

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Add safety to rolling deployments
1. Validate each host’s health
2. Ensure a minimum percentage of the fleet is healthy
3. Rollback if the deployment failed

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Deployment Validation – AppSpec.yml

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2
Step 1: Working tests raises more issues
Failed Deployment

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
4 failures – 60% healthy
MHH 70%, 10 hosts:
V1V2 V1V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2V2 V2 V2 V2 V2
Step 2: Use minimum healthy hosts
1 failure – 90% healthy

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 2: Use minimum health hosts - CodeDeploy

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 3: Rollback when a deployment fails
• CodeDeploy: configured in deployment group

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy: Deployment health
DeployToProd
CodeDeploy
Production
Synthetic Traffic

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Blue/Green Deploys
Failed Deployment
Blue Group Green Group
V1 V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Blue/Green Deploys
Successful Deployment
Blue Group Green Group
V1 V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2
Auto Scaling Auto Scaling

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lower deployment risk by segmenting
• Minimize the impact of deployment failures
• Potentially catch issues before real users do
• Enables you to roll back more quickly, with less impact

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Segmentation Overview
1. Break production into multiple segments
2. Deploy to a segment
3. Test a segment after a deployment
4. Repeat 2 & 3 until done

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Break production into multiple segments
Typical segment types:
• Region
• Availability Zone
• Sub-Zonal
• Single Host (Canary)
US-EAST-1
US-EAST-1A US-EAST-1B

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V2 V2 V2V2V1 V1V1
Step 1: Typical deployment segmentation
Availability Zone based
Deployment
Availability Zone based
Deployment
Availability Zone based
Deployment
V2 V2V2V1 V1V1 V2 V2V2V1 V1V1
Production Fleet
Post-deployment test
Canary
Deployment
V1
Region based Deployment

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Use deployment groups as segments
Create deployment groups per segment using:
• Tags
• Auto Scaling groups

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
DeployToInteg
CodeDeploy
Integration
IntegTest
End2EndTester
Step 2: Deploy to each segment
1. Deploy to smallest segment
2. Post-deployment tests
3. Deploy to one availability zone
4. Post-deployment tests
5. Deploy to remaining availability zones
6. Repeat for other regions
Production – Region 2
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 3: Test each segment
A deployment is valid if:
• The test has gathered enough data to gain confidence
• CloudWatch metrics
• No service alarms have fired
• CloudWatch alarms
• The test has not timed out
• Code

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Add segment tests to your pipeline
Extend CodePipeline with:
• Test Actions
• Lambda Invoke Actions
• Custom Actions
• Approval Actions
1 hour timeout
7 day timeout

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use CodePipeline approvals to trigger tests
Source
MyAppSource
CodeCommit
Deploy
DeployToSegment
CodeDeploy
ValidateSegment
Approval
putApprovalResult
Approval
message
DeployToSegment
CodeDeploy
SNS Topic

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use SNS to start an automated approval check

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Creating a post-deployment test
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda Function
registerDeployTest()
Lambda Function
evaluateDeploy()
CloudWatch
Events (1m)
Change 1
Prod-us-east-1a
CodeDeploy alarmtimeusage
SNS Topic

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Post-deployment test – registerDeployTest
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda Function
registerDeployTest()
Lambda Function
evaluateDeploy()
CloudWatch
Events (1m)
Change 1
Prod-us-east-1a
CodeDeploy alarmtimeusage
SNS Topic

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
registerDeployTest function – (Node.js 4.3)

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Post-deployment test – evaluateDeployTest
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda Function
registerDeployTest()
Lambda Function
evaluateDeploy()
CloudWatch
Events (1m)
Change 1
Prod-us-east-1a
CodeDeploy alarmtimeusage
SNS Topic

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
approveValidation function (Node.js 4.3)

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Canary Deployments – they’re different
All production hosts:
• Participates in serving production traffic
• Configured as a production instance
• Participates in production metrics stream
Canary hosts:
• Has its own metrics stream
• Canary validations use the canary metric stream

62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary: Segment production
• Segment production to reduce impact of a bad change
• Minimum segmentation:
• Region
• Canary deployment per region
• Larger service segmentation
• Zonal
• Sub-zonal
• Test each segment before moving on

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-Region Deployments
• New as of November 2018
• Allows you to deploy to multiple regions from a single pipeline
• Enables you to achieve lower latency and greater availability

65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-Region Deployment Actions
Source
MyAppSource
CodeCommit
Build
MyAppBuild
Build
Deploy
CanaryDeployR1
CodeDeploy
ValidateCanaryR1
Approval
Prod-us-east-1a
CodeDeploy
CanaryDeployR2
CodeDeploy
ValidateCanaryR2
Approval
Prod-us-west-2a
CodeDeploy
CanaryDeployR3
CodeDeploy
ValidateCanaryR3
Approval
Prod-ap-southeast-2a
CodeDeploy

66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What we’ve learned
Goal: Make your pipeline safer…
1. Identify production issues quickly
Continuous Production Testing
Automate notifications
2. Add Security Testing to Your Pipelines
Static Analysis
Vulnerability Scanning
Pipeline Compliance
3. Safely deploy changes
Manage deployment health
Segment production
4. Improve Latency and Availability
Deploy across regions
5. Automatically decide when to release changes
Block non-compliant pipelines

67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related Sessions
1:00-4:00PM: Adding the Sec to Your DevOps Pipelines
Re:Invent Sessions to Watch on YouTube
• DEV303 – Deploying and Managing .NET Pipelines and Microsoft
Workloads
• DEV310 – DevOps on AWS: Choosing the Right Software
Deployment Technique
• DEV313 – Infrastructure Continuous Deployment Using AWS
CloudFormation
• SVR307 – Application Lifecycle Management in a Serverless World

68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

69. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Leo Zhadanovsky