From Heaven to Hell: Security Concerns in CI/CD Pipelines

!
From The Heaven to Hell CI/CD
Security concerns in the dark world
RootedCON 2020

FOLLOW AND
COMMENT THE TALK
IN REAL TIME
http://bit.do/rootedcon2020

WHO WE ARE
Security research. Hacking
tools developer, DevSecOps.
Python developer.
Daniel García (cr0hn)
Can’t define myself.
I go where my curiosity drives to.
Most of the time goes bad.
I process TeraBytes for breakfast.
César Gallego
@ggdaniel
https://bit.do/cr0hn
@CesarGallegoR
https://bit.do/cesar-gallego

Disclaimer!
Any opinions expressed are personal
opinions and don’t represent our
employer’s view in any way

Shared vocabulary
Core Concepts

In software engineering, continuous integration (CI) is the practice
of merging all developers' working copies to a shared mainline
several times a day. Grady Booch first proposed the term CI in his
1991 method, although he did not advocate integrating several
times a day. Extreme programming (XP) adopted the concept of CI
and did advocate integrating more than once per day – perhaps as
many as tens of times per day
- Wikipedia
What CONTINUOUS
INTEGRATION is ?
“
”

Continuous delivery is a software engineering approach in which
teams produce software in short cycles, ensuring that the
software can be reliably released at any time and, when releasing
the software, doing so manually.
- Wikipedia
What CONTINUOUS
DEPLOYMENT is ?
“
”

CONTINUOUS DELIVERY
VS
CONTINUOUS DEPLOYMENT

WHY CI/CD ?
CI/CD It’s all about reaching
customers faster and
more frequently.

HOW CI/CD ?
CI/CD has become so popular
that now is a complete
software landscape.

WHAT
PIPELINE IS ?
A pipeline is a chain of
processing elements.

WHAT DevOps
IS ?
DevOps is a set of practices that combines
software development (Dev) and information-
technology operations (Ops) which aims to
shorten the systems development life cycle and
provide continuous delivery with high software
quality

WHAT
DevSecOps IS ?
It’s SecDevOps when security is
a value and not an obstacle.

STEPS IN BUILDING SOFTWARE CONSTRUCTION

STEPS IN BUILDING SOFTWARE CONSTRUCTION
User Code Building step Deployment step Production

Follow us down the rabbit hole
Starting the journey

IN THE SOURCE CODE

● IDEs are great, but their configuration files
shouldn't be in the source code repository.
● Please, use gitignore file.
The IDE Leaks!

● There is not way to set fine grained permissions
at common source control versions system.
● Can’t set up permissions for specific files os
directories.
The BAD ROLE
Granularity!
Even could be worse... if the repository code is the
developer computer

Keep the code safe at Source Code servers. Do backups.
Keep the workplace safe.
The DEV Machine as only
source code server!
● There is a cowboy coder in your team?
● You have very tiny software in production only
managed by one developer?
● You a have a guru developer?
● You have the correct security at the workplace?
● Are your developers BYOD?

No all
StackOverflow
people are good
persons (or even
humans)
In STACK OVERFLOW
Works Great!
https://trojan-killer.net/the-most-copied-piece-of-java-code-on-stackoverflow-contains-an-error/

● Are your developers using safe libraries?
● Are you check the libraries they use?
● Even more… they ask you for advice when
choice a new library?
All Libraries
Allowed!
https://securityintelligence.com/news/popular-javascript-library-for-node-js-infected-with-malware-to-empty-bitcoin-
wallets/
You trust all libraries? so you know that all
libraries are malware / vulnerabilidades free?

● Passwords
● API keys
● Private keys
● ….
SECRETS & LEAKS

IN THE BUILDING STEP

● What if C.I. clone your code in read/write
mode?
● What if the build script from user can
modify the project source code in building
step?
Be sure you clone/download your source
code as read/only in building step
Non read-only source
code in building time

Control Artifacts
Repository
● What if a developer can publish an arbitrary code into
artifacts / libraries repository?
● What if a developer publish a trojanized version of a
library?
● What if a developer publish, intentionally, an artifact
with known vulnerabilities?
You must control who and what publish in the
artifact repository

● Usually C.I. / C.D. system stores sensitive
information.
● They need them to access to productive
environments, publish artifacts, perform some
checks, etc
● They, usually, store as environment vars.
The ENV Leak!
https://docs.gitlab.com/ee/ci/variables/
Ensure you mark as Protected environment
variables in your C.I. system

● What if an user can execute anything in a
Pipeline?
● What if the C.I. has not limited the output
traffic?
A reverse Shell
in the Pipeline
https://alionder.net/jenkins-script-console-code-exec-reverse-shell-java-deserialization/
Limit user permissions and output
destinations

https://www.youtube.com/watch?v=QDGGPoK4gbk

Keep in mind that the bot user do what developer
demands.
Do not grant more access permissions that you’ll grant
to developers.
The mighty CI
BOT
● Is your CI/CD executed as root?
● There is only one all mighty bot user?
● Your bot user has admin permissions over production?

● Do you control what can download a developer when
they runs in a pipeline?
● Do you control which command can launch a developer
in a C.I. / C.D. configuration file? (Jenkinsfile,
gitlab.yaml…)
● Is your C.I / C.D. in different network? Are you sure?
The EVIL AGENT (1 / 3)

The EVIL AGENT (3 / 3)
➔ Limit internet access in the pipeline.
➔ Perform a correct hardening of the
infrastructure
➔ Fix the execution permissions

● Is your company using free tier services?
● Has your company GitHub Business account?
The Greedy
Service
consumer!
Keep in mind that free tier has limits by IP. Like GitHub, Google
Maps… If your deploy rely on this services may be stuck if
someone exceed the IP quota.

A git Bomb cannot be cloned. Only a problem with
old git versions. Be aware in your older systems.
The Git BOMB!
● Are your commits PGP signed?
● You know who can access wrigths?
● Are you using third party repositories?

Allow only some agents to publish images. Check
docker layers contents. Check Dockerfile.
The DOCKER HUB
Leak!
● You have your own container registry?
● Do you check your Dockerfiles?
● Your pipelines has permissions and access to
publish in docker hub?

● Do you use the API?
● Do you control the CI/CD network access?
Keep API Safe!
Disable access from unnecessary places
with a firewall. Do not install vulnerable
Plugins.

● What frequency has your deployment?
● Do you check your dependencies?
● If some deploy software become too old, do you
perform again the security analysis?
The FOREVER
analysis!
Monitor your artifacts and check for new
vulnerabilities even if your pipeline do not trigger.

● Do you keep your source code repository safe?
● Do you keep your developers machines safe?
● Do you backup your code?
The SOURCE CODE
ransomware!
If a ransomware exists is
because it works. Check
permissions and do backups.

Break HISTORY!
● Do you monitor who modify your code?
● Do you have outsourcing in your team?
● You receive third party commits?
Are you sure of your source code history. An
attacker can break the source code history to
cover their trail.

An attacker can convert any program into their own
vector. Keep an eye on the software supply chain of
your build environments.
The evil
COMPILER!
● Are your agents ephemeral?
● Are your containers trusted?
● Do you have specialized machine for some exotic
language?

A very fat container can spend all free space and
avoid new docker builds. A fat container make
deploy a slow and error prone process.
The Fat DOCKER!
● Do you inspect your Dockerfiles?
● Do you have Docker builds correctly configured?
● Do you control where layers are built?

Without the hash in FROM clause anyone can build a container
using the name of popular one. A docker registry can add extra
security if is well configured avoiding local cache.
The evil DOCKER
twin!
● Do you think that a containers registry is a
complex overhead?
● You build a small number of containers?
● Your team don’t know how to write good
Dockerfiles?

IN THE DEPLOYMENT STEP

Use firewall and security proxies to secure the
internet access.
Run FREE
Internet!
● Some people think that CI/CD is “developer stuff”
● Good networking is critical in the place where a
lot of security secrets live
● Agents must access internet only were they need
to

Outsourcing happens, and an evil user can delay
the CI CD action to prevent production fixes. At
the same time an outsider attacker can exploit a
vulnerable system.
The EVIL
insider!
● Your company use Outsourcing?
● You think that slow pipelines are harmless?
● You suffer the lack of dependency check?

Modify a Jar to add a trojan is very easy. Hard to
detect and can be unbelievable persistent.
The Trojan Jar!
● Do you suffer the Jar Hell?
● Your servers has outdated Java stuff?
● Your company has no java artifact repo?

Critical production infrastructure as code must be
managed by security aware experts.
Secure Infra as
Code!
● Do you let your developers decide how to create
production machines?
● You rely security on cloud promises?

● ZIP Bomb is an old attack.
● The attack is very simple but very useful
● Some of system has basic routines to detect
these kinds of attacks.
The ZIP BOMB (1 / 4)

● Major of packaged software is packed as a ZIP
file: .jar, .war, .docx, .xlsx….
● Some Application Servers auto deploy them when
put files in specific path
● What if we put a ZIP bomb renamed as a valid
packed Application for a Tomcat?
The ZIP BOMB (2 / 4)

Perform a correct hardening of host and set
conservative limits of files, CPU and memory
that a processes can get
The ZIP BOMB (4/ 4)

● Memory bomb is type of attack that aims to fill all
system memory.
● Not only RAM also SWAP is affected.
● If you don’t have limits in your host it can
consume all of your HD space as a SWAP space.
Memory BOMB (1 / 5)

● What if you can run a memory bomb in a C.I. / C.D.
system?
● What if the C.I. is deployed as multi-agent?
Memory BOMB (2 / 5)

Jenkins agent 1 Jenkins agent 1 Jenkins agent 1
Jenkins behavior:
1 - You put a memory bomb in your Jenkinsfile
Memory BOMB (3 / 5)
2 - The Jenkins Master send to the job to an Jenkins
agent and it runs the pipeline and the memory bomb. So
the Jenkins agent host break down
Jenkins master
3 - Jenkins Master detect that the jobs was not finished.
So the send the same job to another Jenkins Agent
4 - Jenkins agent runs memory bomb and… break down
5 - Go to step 2

➔ Less Known but more effective in Docker.
➔ Today powerful computers can die very fast
with no clue who pipeline is responsible.
➔ You can lost all your agents berore you find
where the problem is.
Memory BOMB (5 / 5)

● Fork bomb is type of attack that aims exhaust a
system by creating new processes recursively
● It very difficult to detect if you don’t have a very
good log system configured
● Run in a Pipeline is so easy
● In multi-agent system the results are the same
that with Memory Bomb
Fork BOMB! (1 / 2)

Monitor your CI/CD as the critical system that it
is.
Must Monitor
The system!
● The health of CI/CD is very important
● Even if your pipelines are not executed for a long
time you must perform periodic security checks
● Logs are useful information
● Metrics can warn you of misused infrastructure

The API contract must be fulfilled. No less, No
more. The more is more problematic.
Is your API
Honest!?
● Do you use thread model on you APIs?
● How do you know all the endpoints that you have
deployed?
● Are debug url opened in production?

Human interactions in production environments can be
dangerous. Keep your production inmutable. Use infrastructure
as code.
No HUMANS Beyond
this point!
● Do you know how the current state of your
production?
● Do you allow manual “fixes” on production?
● Do you know how many people can access?
● Do you monitor when and how they access?

Security authentication proxies are good tools for this
job.
Trust the
PASSWORD!?
● The passwords are supplied to third party software?
● No human can access but code can?
● Humans can write code to leak passwords

You can mount trojanized libraries like libc as a volume.
Keep in mind you deploy volumes.
The Docker MOUNTED
Trojan libraries!

Containers are just a bunch o deltas on a file storage and a lot
of genius around. Don’t forget that layers can be accessed.
keep SECRETS
safe!
● Do you store secrets in your containers?
● Do you store security configurations on your
containers?
● Do you store intellectual property on your
containers?
● Where are your containers published?

● Old hack attack but useful
● Alias commands could be the best trojan in
a system.
● There are very complicated to detect
The Evil Alias!
Perform a well hardening of your host systems & be
careful with the bot users

● Do you deploy the C.I. software in your infrastructure?
● Do you have a network isolation from the building software
to the production machine?
● Do you remember the scan by using Jenkins? Can you
imagine use that with Metasploit to Production machines?
The Shared infra!
➔ PLEASE use isolated networks (VPC, VLAN o something
applicable to your infrastructure)
➔ If your C.I. system need to access to the production
machines use LIMITED access API keys.

● Agents need a firewall to avoid lateral movements
● Don’t share the production network with any
CI/CD software
● Don’t share the internal network with any CI/CD
software
keep NETWORK
safe!
If you can keep the CI/CD in their own network. Access
control is mandatory.

● When your agents go down are less problematic if agents are
deploy thru resilient infrastructure (like kubernetes).
● Containers from scratch are blessed simplicity, use often as
you can.
● Deploy Critical software thru configurations (and trusted
software) over complex code.
● You know about security just apply your knowledge in your
CI/CD
● Don’t let that your automation tools become automated
INsecurity
The TIP Of the
iceberg

➔ Who will watch the watchers? Manage your CI/CD as
a critical software (because it is).
➔ Assume that you have a lot of potential insiders
attackers.
➔ Protect your C.I. as your production systems.
➔ Monitoring. Always monitoring. Not only in the
building step.
QUIS CUSTODIET IPSOS
CUSTODES?

https://www.99cs.io
http://book.99cs.io/

From Heaven to Hell: Security Concerns in CI/CD Pipelines

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a From Heaven to Hell: Security Concerns in CI/CD Pipelines

Semelhante a From Heaven to Hell: Security Concerns in CI/CD Pipelines (20)

Mais de Daniel Garcia (a.k.a cr0hn)

Mais de Daniel Garcia (a.k.a cr0hn) (20)

Último

Último (20)

From Heaven to Hell: Security Concerns in CI/CD Pipelines

Notas do Editor