CI/CD Pipeline Security: Advanced Continuous Delivery Recommendations

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dave Walker, Specialist Solutions Architect, Security and Compliance
06/02/19
CI/CD Pipeline Security:
Advanced Continuous Delivery
Recommendations

Agenda
• Introduction
• Development and CI / CD Environments
• Protecting the Workload
• Reference and Reading

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introduction

developers customers
releasetestbuild
plan monitor
delivery pipeline
feedback loop
Software development lifecycle
DevOps = Efficiencies that speed up this lifecycle
DevSecOps = Validate building blocks without slowing lifecycle
“Dev + Sec + Ops”

Development and CI / CD
Environments

© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Development and CI / CD Environments
• Spanning AWS Accounts
• Git hooks
• API integration
• Presenting APIs with PrivateLink and API Gateway
• Jenkins and CI integration
• CD and penetration testing

A Typical CI/CD Pipeline
Version
Control
CI Server
Package
Builder
Deploy
Server
Commit to
Git/masterDev
Get /
Pull
Code
AMIs
Send Build Report to Dev
Stop everything if build failed
Distributed Builds
Run Tests in parallel
Staging Env
Test Env
Code
Config
Tests
Prod Env
Push
Config
Install
Create
Repo
CloudFormation
Templates for Environment
Generate

…and the Services to build it, in AWS
Local
Testing Source Build Test Deploy Monitoring
AWS SAM CLI AWS CodeCommit AWS CloudFormation Amazon CloudWatch
AWS CodePipeline
AWS CodeBuild

Multi-account Pipelines are a Good Idea for scoping:
https://aws.amazon.com/blogs/devops/aws-building-a-secure-cross-account-continuous-delivery-pipeline/

Version
Control
CI Server
Package
Builder
Promote
Process
Validate
Dev
Get /
Pull
Code
AMIs
Log for audit
Staging Env
Test Env
Code
Config
Tests
Prod Env
Audit/Validate
Config
Checksum
Continuous
Scan
…with the Opportunity for Security Integrations
Send Build Report to Security
Stop everything if audit/validation failed
CloudFormation

Git Hooks
• Example: Git-Secrets
• Preventing AWS API keys hardwired into code being
accidentally posted to public Git repos
• (aside: use IAM Roles for Lambda functions and EC2
instances, rather than putting API keys in code, please!)
• https://github.com/awslabs/git-secrets
• From GitHub WebHooks to SNS to Lambda:
• https://aws.amazon.com/blogs/compute/dynamic-github-
actions-with-aws-lambda/
• Includes creating GitHub API tokens and applying them to
your Lambda function to make it a GitHub bot

Version
Control
CI Server
Package
Builder
Promote
Process
Validate
Dev
Get /
Pull
Code
AMIs
Log for audit
Staging Env
Test Env
Code
Config
Tests
Prod Env
Audit/Validate
Config
Checksum
Continuous
Scan
Send Build Report to Security
Stop everything if audit/validation failed
CloudFormation

API Integration
• PrivateLink (and AWS Marketplace)
• Share an ENI cross-account and cross-VPC from an NLB
• Back the NLB with RESTful service
• Use RESTful APIs from 3rd parties for (eg) static source
code analysis:
• Ensure your Lambda functions have a routable signal path
to the API presentation point!

API Integration
• Examples for static source code analysis (commercial):
• Checkmarx:
• https://checkmarx.atlassian.net/wiki/spaces/KC/pages/131039271/CxR
EST+API
• Veracode:
• https://help.veracode.com/reader/LMv_dtSHyb7iIxAQznC~9w/qyPhDbX
ssVXuQl7aLWqH6g
• XML rather than JSON, but curl it :-)
• eg curl --compressed -u [api user]:[api user password]
"https://analysiscenter.veracode.com/api/5.0/beginscan.do" -F
"app_id=10886" -F "modules=284642,284653,284654”
• Whitesource:
• Direct GitHub integration -
https://whitesource.atlassian.net/wiki/spaces/WD/pages/33816749/Git
Hub+Integration

API Integration
• Examples for static source code analysis (FOSS):
• https://brakemanscanner.org/ (Ruby on Rails)
• …
• …and web framework analysis:
• https://vaddy.net/
• …and Container configuration analysis:
• https://github.com/coreos/clair

API Integration
• Container build-time anti-malware analysis
• https://github.com/deep-security/deep-security-py
• https://www.twistlock.com/2017/08/02/jenkinsdockertwistlock-delivers-
promise-continuous-delivery/
• (Container run-time analysis from Twistlock, Aqua, Neuvector –
behavioural run-time analysis from Alcide)

…and if you're up for Formal Proof…

Jenkins Integrations
• Static source analysis, again:
• Checkmarx: https://github.com/jenkinsci/checkmarx-plugin
• Whitesource: https://github.com/whitesource/jenkins-
whitesource-plugin
• CodeBuild:
https://wiki.jenkins.io/display/JENKINS/AWS+CodeBuild+Pl
ugin
• Plugin management:
https://www.praqma.com/stories/jenkins-configuration-
as-code/

Jenkins Integrations
• Container analysis:
• Build time: Trend Micro Deep Security Smart Check -
https://www.trendmicro.com/en_us/business/products/hybri
d-cloud/smart-check-image-scanning.html
• Integration via Lambda call-out (using Python SDK):
https://github.com/deep-security/deep-security-py:
• script: "python smart-check/scans.py --
smart_check_url='$SMART_CHECK_SERVER' --
smart_check_userid='$SMART_CHECK_CREDS_USR' --
smart_check_password='$SMART_CHECK_CREDS_PSW' --
scan_registry='$SCAN_REGISTRY' --
scan_repository='$JOB_BASE_NAME' --scan_tag='$BUILD_ID' --
aws_region='$AWS_REGION' --aws_access_key='$AWS_USR' --
aws_secret='$AWS_PSW'"

Dependency Checking
• Your own code may be sound – but what about the
libraries you’re linking against?
• Snyk: https://snyk.io
• Whitesource: https://www.whitesourcesoftware.com
• Both look at version info and CVEs, at import / link time
• Dependency and further static analysis:
• BlackDuck (Now part of Synopsys):
https://www.blackducksoftware.com

Checksumming
• Do you really need to?
• AMIs are stored in S3
• This makes an AMI immutable
• No bitwise editing
• …all you can do is rip and replace it with another AMI
• …so why checksum?
• If your deployment target isn’t an AMI, that’s another
matter…
• AMIs may not need checksumming, Containers do
• …and sign them, and verify signatures

Traditional Structured Deployment
Create
Skeleton
Define
Resources
Execute

Split Ownership Configurations
Who knows your solution best?
• Dev, Infra, Sec…?
• Delegate ownership
Use Yaml and split file into chunks or functions
• Separate file sources with access control – Use IAM/VPC-E/etc.
• Push files -> Validate -> Merge files -> Validate -> Deploy -> Validate
Jenkins for deployment
• Promotion flows
• Move from manual to Automation based on validation quality
• Excellent for merging jobs of split configurations

Merging
From single file or multiple files
• Maintain access control using
policies
• Use different source stores if
needed
Based on function/state
Reusable patterns
Maintain order, especially of validation
• Security validation last to execute
• Security should always win

Validation
Keep track of what section you are
validating
• Stage vs Prod
• Merged vs separated
Validate often and log/alert
• Validate part and end result
• Run-time validation
Use external agents
• AWS Simple Work Flow
• AWS Lambda
• Etc.

Structured deployment using Split ownership
Create
Skeleton
-
Infra
team
Define
Resources
-
DevOps
Execute
-
Security

Deployment Mechanisms for Software Artifacts
Amazon
Machine Images
(AMIs)
Docker Image
OS Packages
Amazon EC2
Container
Service
AWS
CloudFormation
AWS
CodeDeploy

Amazon
Machine
Images
(AMIs)
Docker
Images
OS Packages
Amazon
EC2
Container
Service
AWS
CloudFormation
AWS
CodeDeploy
Software Artifacts Deployment Services
Deployment Mechanisms for Software Artifacts

Penetration Testing
• If you’re doing Continuous Deployment, you need to be doing
Continuous Pentesting!
• Unless you’re deploying to containers or serverless (more on which,
later…) deploy to an Inspector-instrumented EC2 instance for extra
information “from the inside”
• See https://aws.amazon.com/security/penetration-testing/ and
consider:
• Synopsys tools
• Cymulate
• Pcysys
• OWASP ZAP, etc
• …or if you’re a pentest service provider, consider:

Customer
Environment
Provider
Environment
virtual private cloud
"Pentest as a
Service"
Elastic IP
instance
Elastic Load
Balancing
AWS
CodeCommit
virtual private cloud
Amazon API
Gateway
Amazon
SQS
Amazon
DynamoDB
AWS
Lambda
bucket
AWS
CodePipeline
AWS
CodeDeploy
AWS
Lambda
instance
Internet
gateway
Internet
gateway
AMI
AWS
Lambda
Amazon
Inspector
(1) (2)
(3)
(4)
instances
(5)
(6) (7) (8)
(9)
(10)
(11)
(12)
(13)
(14)(15)

Getting Meta…
• Don’t forget to consider the security of your security
tooling!
• Ensure all API calls to external services are https
• Do diligence on your providers of 3rd-party services
• https://www.slideshare.net/guypod/serverless-conf-
serverlesswhatslefttosecure
• Good practices for Lambda security…

Good Practices for Lambda Security
• There’s no option for skipping good design and planning:
• 1 function, 1 purpose, 1 IAM Role
• Least privilege in IAM Roles for each function
• See https://github.com/puresec/serverless-puresec-cli
• To VPC, or not to VPC?
• Consider your threat model (everything starts with one)
• Lambda-in-VPC gives you egress filtering; Security Groups,
potential for http(s) proxies (eg Squid)
• More in https://www.slideshare.net/guypod/serverless-conf-
serverlesswhatslefttosecure

• Also see what’s new at:
• https://www.puresec.io/ssre
• https://www.protego.io/resources/
• https://www.twistlock.com/solutions/serverless-security-
aws-lambda-azure-google-cloud/

• Secrets / credentials management: Lambda functions can
use Systems Manager Parameter Store and Secrets Mgr
• Give your functions IAM Roles with Allowed Actions:
• ssm:GetParameter
• ssm:SendCommand
• Lambda:ListTags
• …and ensure your parameter encryption key has a policy:
{ "Sid": "Allow decryption with the key",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::111122223333:role/MyLambdaFn"}, ”
Action": [ "kms:DescribeKey", "kms:Decrypt" ],
"Resource": "*”
}

Parameter Store Substitution
$ aws ssm put-parameter
--name myprivatekey
--type SecureString
--value “-----BEGIN RSA PRIVATE KEY-----
WtcUTC+57cf…”
--key-id <KMS keyID>
$ aws ssm send-command
--name Insert-Websvr-Private-Key
--parameters commands=[“cat {{ssm:myprivatekey}} >
/etc/apache2/keys/private.key ; chmod 400 /etc/apache2/keys/private.key ;
chown webserver:webserver /etc/apache2/keys/private.key”]
--target Key=tag:Name,Values=WebServer

Protecting the Workload

The Pipeline is a Workload…
• It needs all the properties of other secure workloads:
• Not deploying assets to AWS Regions you don't want it to, or using
assets in such Regions
• Only being changeable by appropriate parties
• Isolation from other workloads

When you create a new account in Organizations…
Created accounts have root and
OrganizationAccountAccessRole at creation time
• OrganizationAccountAccessRole is effectively “admin”
• Create cross-account permissions for it
• The ARN is always arn:aws:iam::<new account ID>:role/OrganizationAccountAccessRole
• Run your account baselining tools with it
• …including setting IAM Federation up, where appropriate
• You can also incorporate CIS Foundation Benchmark scripts: https://github.com/awslabs/aws-
security-benchmark
• Ensure only appropriate Organization Master roles can assume it, when done
• Landing Zone creates cross-account AWSCloudFormationStackSetExecutionRole
• …and also applies an SCP (see later)
• AWSCloudFormationStackSetExecutionRole created in aws-landing-zone-
initiation template, deployed via Account Vending Machine

Region Locking

Locking services to a Region
Why?
• Data sovereignty
• Compliance with local legislation/regulation
What services can we do this for, currently?
• All of them!
Can we centrally monitor what Regions are in use, and alert
on unexpected Regional activity?
• Yes!

aws:RequestedRegion
{
"Sid": "RestrictEC2ToUSOnly",
"Effect": "Deny",
"Action": [
"ec2:*"
],
"Resource": "*",
"Condition": {
"StringNotEquals": {
"aws:RequestedRegion": [
"us-west-1",
"us-west-2",
"us-east-1",
"us-east-2"
]
}
} }

Region usage
monitoring
and alerting
and Amazon GuardDuty

Organizations, Service Control
Policies and Mandatory Access
Control

Organizations and IAM
Create groups of AWS accounts with AWS Organizations (“ou”)
Use Organizations to attach SCPs to those groups to centrally
control AWS service use
Entities in the AWS accounts can only use the AWS services
allowed by both the SCP and the AWS IAM policy for the account

Service Control Policies ("SCPs")
Enables you to control which AWS service APIs are accessible
• Define the list of APIs that are allowed—whitelisting
• Define the list of APIs that must be blocked—blacklisting
Resultant permission on IAM user/role is the intersection
between the SCP and assigned IAM permissions
IAM policy simulator is SCP aware
SCP application integrated into Landing Zone
• (Just at account creation time, at this point…)
• See aws-landing-zone-configuration/manifest.yaml

Mandatory access control with SCPs
SCPs are:
• Immutable to all users in the child account, including root
• Invisible to all users in the child account, including root
• Applied to all users in the child account, including root
From the perspective of the account the SCP is applied to, an SCP
looks just like System Policy in a Mandatory Access Control
environment
• (…and if you want to know more about Mandatory Access Control environments on AWS,
look at SELinux, as available for all major Linux distributions running on AWS, and also MLS
configuration for FreeBSD, available at
https://aws.amazon.com/marketplace/pp/B01LWSWRED/ )
(Though you don’t have packet labelling or Bell-LaPadula / Biba
dominance relationships in Organizations…)

SCPs and immutability
But:
• You don't have to apply an SCP before you populate your account with assets;
you can do so afterwards...
• This lends the idea of "immutable infrastructure" to other services, from the
point of view of the child accounts
• (Including Serverless)
• E.g.,
• Amazon S3 websites which can't have their contents changed or be deleted
• Lambda functions which are invoke-only "black boxes"
• AWS Certificate Manager cert / key pairs which can't be deleted
• Preventing AWS CloudTrail, AWS Config, VPC Flow Logs ever being turned off
• ...and for our case here, preventing Pipelines and Lambda functions being
changed

Disable services you won't be using…
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": ”<Insert unwanted service prefix here>:*",
"Resource": "*"
}
]
}

Stop CloudTrail being disabled…
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": ”cloudtrail:StopLogging",
"Resource": "*"
}
]
}

Stop CloudWatch being disabled…
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": [
”cloudwatch:DeleteAlarms",
”cloudwatch:DeleteDashboards",
”cloudwatch:DisableAlarmActions",
”cloudwatch:PutDashboard",
”cloudwatch:PutMetricAlarm",
”cloudwatch:SetAlarmState"
]
"Resource": "*"
}
]
}

Stop VPC Flow Logs being disabled…
"Statement": [
{
"Effect": "Deny",
"Action":"ec2:DeleteFlowLogs”,
"Resource": "*"
}
]

Stop Config being disabled…
"Statement": [
{
"Effect": "Deny",
"Action": [
"config:DeleteConfigRule",
"config:DeleteConfigurationRecorder",
"config:DeleteDeliveryChannel",
"config:StopConfigurationRecorder"
],
"Resource": "*"
}
]

Make CodePipeline immutable…
"Statement": [
{
"Effect": "Deny",
"Action": [
"codepipeline:CreateCustomActionType",
"codepipeline:CreatePipeline",
"codepipeline:DeleteCustomActionType",
"codepipeline:DeletePipeline",
"codepipeline:DeleteWebhook",
"codepipeline:DeregisterWebhookWithThirdParty",
"codepipeline:DisableStageTransition",
"codepipeline:EnableStageTransition",
"codepipeline:PutWebhook",
"codepipeline:RegisterWebhookWithThirdParty",
"codepipeline:UpdatePipeline"
],
"Resource": "*"
}
]

Make S3 buckets and bucket policies immutable…
"Statement": [{
"Effect": "Deny",
"Action": [
”s3:DeleteBucket”,
“s3:DeleteBucketPolicy”,
“s3:DeleteBucketWebsite”,
“s3:PutBucketAcl”,
“s3:PutBucketCORS”,
“s3:PutBucketLogging”,
“s3:PutBucketNotification”,
“s3:PutBucketPolicy”,
“s3:PutBucketRequestPayment”,
“s3:PutBucketTagging”,
“s3:PutBucketVersioning”,
”s3:PutBucketWebsite”,
“s3:PutLifecycleConfiguration”,
“s3:PutReplicationConfiguration”,
“s3:PutObjectAcl”,
“s3:PutObjectTagging”,
“s3:PutObjectVersionAcl”,
],
"Resource": "*”
}]

Make Lambda functions immutable…
"Statement": [{
"Effect": "Deny",
"Action": [
"lambda:AddPermission”,
“lambda:CreateAlias”,
“lambda:CreateEventSourceMapping”,
“lambda:CreateFunction”,
“lambda:DeleteAlias”,
“lambda:DeleteEventSourceMapping”,
“lambda:DeleteFunction”,
“lambda:PublishVersion”,
“lambda:RemovePermission”,
“lambda:TagResource”,
“lambda:UntagResource”,
”lambda:UpdateAlias”,
“lambda:UpdateEventSourceMapping”,
“lambda:UpdateFunctionCode”,
“lambda:UpdateFunctionConfiguration”
],
"Resource": "*”
}]

(which still enables you to do this…)
"Statement": [
{
"Effect": ”Allow",
"Action": [
“lambda:GetAccountSettings”,
“lambda:GetAlias”,
“lambda:GetEventSourceMapping”,
“lambda:GetFunction”,
“lambda:GetFunctionConfiguration”,
“lambda:GetPolicy”,
“lambda:Invoke”,
“lambda:InvokeAsync”,
“lambda:ListAliases”,
“lambda:ListEventSourceMappings”,
“lambda:ListFunctions”,
“lambda:ListTags”,
“lambda:ListVersionsByFunction”
]
"Resource": "*”
}
]

…and if you don't want your VPCs to have
Internet access…
"Statement": [
{
"Effect": " ",
"Action": [
"ec2:AttachInternetGateway”,
“ec2:CreateInternetGateway”,
“ec2:AttachEgressOnlyInternetGateway”,
“ec2:CreateVpcPeeringConnection”,
“ec2:AcceptVpcPeeringConnection"
],
"Resource": "*"
}
]

What to put in an SCP and what to put in
IAM
• Statements which benefit from
being invisible, immutable and
applicable to all users in an
account (including root) or
group of accounts
• Statements which require IAM Conditions
• Statements with Resources which need
to be something other than “*”
• Statements which should be controlled
by users inside the account

…and if you still
need IAM
Resources and
Conditions with
immutability,
make IAM Policy
immutable!

Applying the theory
• Create your new Account
• Apply an SCP to turn unwanted services off
• Baseline it—including IAM policy
• Put your assets in it
• Extend your SCP to add immutability as needed
• For account maintenance, an SCP can be removed or replaced for an
appropriate maintenance window
• This requires collaboration between the SCP’s controller(s) in the
Organizations Master account and the account/service maintenance staff
• We have a 2-person rule mechanism here! J

Reference and Reading

AWS and Third-Party Papers and News
• AWS “Serverless Applications Lens” whitepaper
• https://d1.awsstatic.com/whitepapers/architecture/AWS-
Serverless-Applications-Lens.pdf
• CI/CD for Serverless and Containerized Applications:
• https://www.youtube.com/watch?v=01ewawuL-IY
• Pure-Sec ”Serverless Security Top 10 Most Common Weaknesses”
whitepaper
• https://www.puresec.io
• Protego papers and videos
• https://www.protego.io/resources/
• DevOps Weekly
• http://devopsweekly.com

Further Recommended Reading:
• The DevOps Handbook:
• https://www.amazon.com/DevOps-Handbook-World-Class-
Reliability-Organizations/dp/1942788002

Thank you!

CI/CD Pipeline Security: Advanced Continuous Delivery Recommendations

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a CI/CD Pipeline Security: Advanced Continuous Delivery Recommendations

Semelhante a CI/CD Pipeline Security: Advanced Continuous Delivery Recommendations (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

CI/CD Pipeline Security: Advanced Continuous Delivery Recommendations