DevOps: Security 干我何事？
1

HELLO!
I am smalltown
MaiCoin Site Reliability Engineer
Taipei HashiCorp User Group Organizer
AWS User Group Taiwan Staff
2

“
Do You Think Security is
Important in Software
Development?
3

“
Everything Goes Well With
Waterfall Model
4

“
Agile Model Made Software
Delivery Faster and Faster
5

“
It’s Impossible to Test After
Done of Development
6

DevOps Power By
Testing
Ref
7

“
Security Also Encounter
The Same Problem
8

DevOps Power By
Security
Security
Thank for
Coming Today
9

Three Elements of
DevSecOps
OperationDeveloper
Information
Security
Security factor into
release pipeline
Unite developers,
security and operation
Increase Visibility
10

Outline
⊡ Security Gene in Development
⊡ Secret Management
⊡ Infrastructure As Must Be Code
⊡ System Management
⊡ C.A.S. C.A.M.S.
11

Security Gene in
Development
1
12

“
Penetration Testing Need
to be Fast and Seamless
with Development Process
13

Penetration Test in
Traditional Way
⊡ When new service online, performing
penetration testing By...
□ Information Security Department
□ Third Party Penetration Service
⊡ But … Agile development teams focus on
producing code Not Enhance Security
14

Penetration Testing
in Development
⊡ Static application security testing (SAST)
⊡ Runtime application self-protection (RASP)
Ref
15

Penetration Testing
in Development
⊡ Dynamic application security testing (DAST)
⊡ Interactive application security testing (IAST)
Ref
16

“
A Chain is Only as Strong
as Its Weakest Link
17

“
Penetration Test Need
Include Both Human and
System
18

Penetration Test for
Human
⊡ Although attend tests/inform orally is not
avoid
⊡ But the exercise make things well imprinted
on human’s brain.
⊡ After all, practice makes perfect
19

Penetration Test for
Human
⊡ Don’t Need Strong Knowledge/Skill
⊡ All you need to do is …
□ Leverage Human’s Greedy/Fear
□ Get the organization member contact info
□ Automatic send mail/sms mechanism
(Phish Insight)
□ Host Fake website to collect feedback (e.g.
CredSniper, SET)
20

“
Trust Me, The Result Will
Make Your Jaw Drop
21

Secret Management
2
22

“
Have You Rotated Production
Database Account/Password
Recently?
23

Rotate Credentials
⊡ The database credentials will be rotated
when…
□ Database migration
□ External auditing
⊡ Only if the rotate mechanism is implemented
at first, then it truly happens
24

HashiCorp Vault
⊡ Secures, stores, and tightly controls access to
tokens, passwords, certificates, API keys
⊡ Handles leasing, key revocation, key rolling,
and auditing
25

Ideal Credential
Lifecycle
Service is Accessed
Application
1. Request Access Credential (Running)
2. Use the Credential to Access Service
3. Revoke the Credential
Credentials Only
Exist in Memory
26

Authentication
⊡ Vault provide various auth method
□ Tokens, AppRole
□ AWS, Azure, Google Cloud
□ LDAP, GitHub
□ ...etc
27

Authorization
⊡ Vault store credentials like key/value
database, e.g.
□ /secret/stag/database/admin
□ /secret/prod/database/admin
⊡ Hence, predefined policy grant appropriate
permission, e.g.
path "secret/stag/database/admin" {
capabilities = ["read"]
}
28

Dynamic Credentials
⊡ Vault support many secret backend
□ AWS, Azure, GCP, Database...etc
⊡ Take database for example, you could
generate dynamic database credentials
$ vault read database/creds/my-role
Key Value
--- -----
lease_duration 1h
password 8cab931c-d62e-a73d-60d3-5ee85139cd66
username v-root-e2978cd0-
29

Infrastructure As Must Be
Code
3
30

“Network Misconfigurations
Are Major Source Of
Reliability and Security
Issues
In a report summarizing the findings of 124 penetration tests,
security firm Rapid7 found that more than two-thirds of sites
were vulnerable because of a misconfiguration
Ref 31

“
Why So Many Important
Internal Service Can Be
Accessed Publicly?!
32

Keep Server Private
Private Network
- Kubernetes
- Database
- NAT Gateway
- ...
Public Network
- Load Balancer
- Linux Gateway
- ...
- Non-Employee: Only access product service through
load balancer
- Employee: Access server through VPN/Bastion
- Don’t forgot to enable WAF
33

“
In Fact, Not Only Network Related
Configuration, Every Program
Misconfiguration Makes Issues
34

Infrastructure As
Code
⊡ There are so many benefits when adopting
IaC
□ Save time & Avoid human error
□ Code review & Knowledge transfer easily
□ Testing (kitchen, terratest...etc)
35

Infrastructure As
Code
⊡ There are many tools which can achieve IaC
⊡ Servers:
□ Ansible, Chef, Puppet, SaltStack...etc
⊡ Cloud Providers:
□ Terraform, AWS CloudFormation, Azure
Resource Manager, GCP Deployment
Manager...etc
36

System Management
4
37

9 Key Point
⊡ Authentication (Later)
⊡ Authorization (Later)
⊡ Secret Management (HashiCorp Vault)
⊡ Don’t Share Account
⊡ Least Privilege Policy
⊡ Log Everything
⊡ Manage and Record Privileged Activity (Later)
⊡ Alert and Notify of Suspicious Activity (Later)
⊡ Identity Centralize and Unify
38

Authentication for
Human
⊡ Adopt Password Manager to avoid credential
stuffing attacks
⊡ Must Enable 2FA: What-you-know,
What-you-have and What-you-are (2 out of
the 3 types)
39

“
A: I Need Sudo Permission for
Production Deployment Tomorrow
40

Manage and Record
Privileged Activity
⊡ The traditional way maybe…
□ File a ticket
□ Wait the ticket assign
□ Information collection
□ Approved by someone
□ Wait for operator change permission
□ Confirm you really get the permission
□ Start the task
□ Remove the permission by operator
41

42

“
How About Achieving This
Efficiently Through ChatBot!
43

Change Management
W/ Chatbot
1. Submit RFC
2. File Ticket to Record3. Review By Stakeholder
5. Start to Perform the Change
4. Grant Permission
44

Using ChatBot
45

“Get All Security Information and
Auditing Function Ready, No Matter
From Cloud Provider or Third-Party
Solution
46

Alert and Notify of
Suspicious Activity
⊡ Only having enough data, then the security
checks and monitoring can happen
⊡ Setup basic rule set to monitor abnormal
behavior
⊡ Using AI to train the rule set
47

Suspicious Activity
⊡ Strange Login Behavior
⊡ Abnormal Inbound/Outbound Connection
⊡ Attack Event
⊡ Access Malicious Website
⊡ ...etc
48

“
How do You Patch Vulnerability?
49

Patch Vulnerability
Actively
50

“
More Secure CI/CD Flow - GitOps
51

GitOps
WebHook GitOps
Push
Push
Push
Push
Like Agent
Pulling
More Network Settings, Credentials
52

C.A.S. C.A.M.S.
5
53

DevOps Core Value
54

Culture
⊡ Except information security department,
everyone should have security knowledge
55

Automate
⊡ Penetration test integrate with release
pipeline
⊡ IaC avoid misconfiguration
⊡ Secret management rotate credentials
⊡ Chatbot
⊡ ...
56

Measure
⊡ Continuous security testing measure how
your service health
⊡ Rich system log measure how your system
safe
57

Share
⊡ Internal sharing makes vulnerability can be
excluded
⊡ External sharing make new patch
implemented quickly, incident decrease
58

THANKS!
Any questions?
You can find me at
facebook.com/smalltown0110
59