In this talk we’ll see how Authentication and Secrets delivery work in distributed containerized applications from the inside. We’ll start from the theory of security and will go through the topics like Container Auth Role, Static & Dynamic secrets, Env vars/volumes for secret delivery, Vault & K8S secrets. After this talk you’ll get an understanding how to securely deploy your containerized workloads.

1. First secret delivery for modern
cloud-native applications
Vladlen Fedosov, Director of R&D @Namecheap, Inc

2. Continue listening or check Twitter? 🤔
● Introduction to theory, security loves theory 🤓
● How to harden your security in 5 simple steps and be promoted 😎

3. Vlad Fedosov
Director of R&D @Namecheap
TL;DR:
● 10 years in the industry
● Went path from Junior to Architect
● Amateur DevOps evangelist
● AWS ninja
● Believe in self-organized, cross-functional teams

4. “Opening the door for everyone to
a free and open Internet”

5. Theory 🧐
Please don’t fall asleep here ;)

6. What is “Security”?
● Security is the practice of risk management
○ Accepting some risks, guarding against violations of norms
○ Ex: collection of identified risks with mitigation paths
● Risk increases with system complexity
○ More moving parts, more confusion, bigger egress and ingress surface
○ Ex: SSH port opened for 0.0.0.0/0
● Anything that can elevate Risk is a Threat
○ Modeling threats helps to design security policies
○ Ex: Outdated SSH server sith CVE

7. What is “Secret”?
A Secret is something that will elevate your Risk if exposed to unauthorized
entities. Consequences:
● Regulatory fines
● Reputation loss
● Customer lawsuits
● Private data exposure
An exposed secret is a Threat

8. Chain of Trust
It is a set of links (e.g. network hops/systems) that any particular secret travels
through from Source to Destination.
Any link is an interception/access point. So therefore we have Chain of Trust.

9. Chain of Trust - more links, more risk
Every additional link increases complexity and therefore increases Risk.
● Surface area for attacker
● Accidental logging (ex: CD logs)
● Compromised employee (reality for big companies)

10. Goal
We need to securely move
secrets from originator to the
new containers RAM.

11. Omg, we have people in the room! 😧
Usually not only our systems/applications need access to secrets, but also
people. Example:
● DBA need an access to production DB
● QAs need an access to Sandbox PayPal
● etc..
Sometimes access is required to the same Secrets, sometimes
to related ones.

12. Goal
We need to securely move secrets from
originator to the new containers RAM. As well
as provide secure access to them for
authorized employees.

13. Summing up
Secrets manager
Directory service
Scheduler
Scheduler
Agent
OS /
Filesystem

14. Damn, sounds complex 😤

15. Types of the Secrets and how to
protect them

16. Types of the Secrets (simplified, our scope)
● DBs/Queues access
○ Provisioning can be automated, can be revoked, usually we have flexibility here
● Plain text
○ Ex: Accesses to legacy systems, 3rd party providers; other random things...
○ Hard to automate, hard to rotate

17. Types of the Secrets (simplified, advanced)
Out of the scope of this talk, find me after if you’re interested in...
● Service access
○ Ex: Accesses to AWS, PayPal, NewRelic, PagerDuty, CloudFlare, etc..
○ Usually support AD/OpenID
● Encryption as a Service
○ The idea is to outsource encryption/decryption process as well as keys management to
dedicated secure system rather then rely on built in programming libraries.
● SSH
○ How to automate/eliminate SSH accounts management for servers fleet

18. How to protect Secrets
First thing to do: establish success criteria based on acceptable risk
What should/may be done:
● Don’t let them live forever (rotate/expire).
● Detect unauthorized access (e.g., use one time access tokens)
● Have a break-glass procedure. What will you do in case of leakage/intrusion?
● Deliver them securely. Protect your chain of trust.

20. Real things start here
In steps. Based on our real experience at Namecheap

21. Step #1. Choose secret management tool

22. Step #1. Choose secret management tool
We use: Hashicorp Vault
Alternatives:
● AWS Secrets Manager / AWS SSM Parameter Store
● K8s Secrets
● Square Keywhiz
● Lyft Confidant
● Torus
● SecretHub
● etc..

23. Step #1. Choose secret management tool
Criterias I recommend to check:
● Dynamic secrets support (on-demand creds generation, TTLs, renew)
● Versioning support for static secrets
● People & machines can work with it
● Does it supports services you use?

24. Step #1. Choose secret management tool
Criterias I recommend to check:
● Built in audit capabilities. Every action should be logged in.
● Can you check it’s source code?
● Community support / Vendor support
● Flexible access policies engine
● Extensibility capabilities. Can you write plugins?

25. Step #1. Choose secret management tool
And please, don’t try to write your own 😉

26. Step #1. Choose secret management tool
For further examples I assume that tool we choose is Hashicorp Vault. But most
of the concepts explained in the next slides are still relevant to alternative setups.

27. Step #2. Configure secret backends
Story about dynamic secrets concept

28. Dynamic secrets concept
So instead of credentials created by DBA that look like:
“my_dear_app:Str0nGPasS”
Your app or user will get something like:
“Rw_role_f34trfds:3W5mcu5...”
And it will be unique to every instance of your app.

29. Dynamic secrets concept
4.1 Issues credentials

30. Dynamic secrets concept - Why important?
It protects us against:
● Accidental logging (old logs not a treat for us anymore)
● Compromised employee (easy to track who got creds used in attack)
It gives us ability:
● To easily rotate secrets in case of intrusion/leakage
● Be more efficient and forget about regular manual secrets rotation
● Do nothing if employee leaves company

31. Dynamic secrets concept - Practice
$ vault secrets enable database
$ vault write database/config/mydb1 plugin_name=mysql-database-plugin
connection_url="{{username}}:{{password}}@tcp(127.0.0.1:3306)/"
username="root" password="rootpassword" allowed_roles="*"
$ vault write -force database/rotate-root/mydb1
$ vault write database/roles/mydb1_ro db_name=mydb1
creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY
'{{password}}';GRANT SELECT ON *.* TO '{{name}}'@'%';"
$ vault read database/creds/mydb1_ro
Key Value
--- -----
lease_id database/creds/mydb1_ro/bd404e98-0f35-b378-269a-b7770ef01897
lease_duration 3600
password 132ae3ef-5a64-7499-351e-bfe59f3a2a21
username readonly-aefa635a-18
bit.ly/335Lxb2
bit.ly/2N1uEZx

32. Hey, what about static secrets?
$ vault kv put secret/my-secret my-value=s3cr3t
$ vault kv get secret/my-secret
====== Data ======
Key Value
--- -----
my-value s3cr3t

33. Access Control, in short
1. Read the docs first! Carefully!
2. Configure your access policies
properly
3. In Vault we prefer resource
based policies. So we create
small policies for every secret
group and assign them to
users/machines afterwards
path "database/creds/mydb1_ro" {
policy = "read"
}
path "database/roles" {
policy = "read"
}
path "database/roles/*" {
policy = "read"
}
path "database/config" {
capabilities = ["list"]
}
path "database/config/*" {
capabilities = ["read"]
}
$ vault policy write database/mydb1_ro
policy-file.hcl

34. Step #3. Configure secure introduction for
machines
How to authenticate your apps w/o any static credentials

35. 2 major approaches
1. Built-in platform capabilities
a. Ex: Kubernetes secrets, AWS Secrets Manager + ECS, Nomad + Vault, AWS CLI + IAM Roles
b. Simpler setup, usually less flexible, support fewer security related features.
c. Great for small/medium companies. Use them if you meet accepted risk level
2. 3rd party secret managers
a. Ex: Hashicorp Vault, Square Keywhiz, Lyft Confidant. Etc…
b. Complex setup, harder to start, more flexible, can be interchanged, more security features
c. Great for big companies or security sensitive projects.

36. Built-in platform capabilities
Cluster agent
(kubelet,
ecs-agent, etc..)
#1
Control plane
Instance
App container
Magic happens here
Secrets manager
#2
#3

37. 3rd party secret managers (Vault)

39. 3rd party secret managers (Vault) – Practice
$ vault auth enable kubernetes
$ vault write auth/kubernetes/config
token_reviewer_jwt="reviewer_service_account_jwt"
kubernetes_host=https://192.168.99.100:8443
kubernetes_ca_cert=@ca.crt
$ vault write auth/kubernetes/role/demo
bound_service_account_names=demo-kube-service-role
bound_service_account_namespaces=default
policies=database/mydb1_ro ttl=1h
$ vault write -format=json auth/kubernetes/login role=demo jwt=...
{
"auth": {
"client_token": "38fe9691-e623-7238-f618-c94d4e7bc674",
"accessor": "78e87a38-84ed-2692-538f-ca8b9f400ab3",
"policies": [
"default", "database/mydb1_ro"
],
"metadata": { ... },
...
}
}

40. How can I add this to existing apps?
If you use built-in platform capabilities for secrets management:
● They usually deliver secrets to container’s env variables or mounted volume

41. How can I add this to existing apps?
If you use 3rd party secret managers (Vault in our case):
● Interact with Vault directly from application’s code
● Use in-container tool like “envconsul”, “consul-template” (with Vault Agent) or
“nc-vault-env” that starts first, authenticates in SM, populates secrets, runs
your app as sub-process, renews all tokens
● Use sidecar container pattern (for ECS) or
sidecar + init container for K8s

42. Step #4. Configure human access

43. Human access - Possible ways (Vault)
Optimal way: LDAP (we use now)
Possible ways:
● OpenID Connect, GitHub, Okta, RADIUS, Userpass
You just need to get through this pain with holy help of documentation.

44. Step #5. Hardening security

45. How to prevent/detect secrets hijacking
Sidecar container /
init process /
orchestrator
App
1. Authorize & fetch secrets
2. Wrap secrets
vault write sys/wrapping/wrap data=test111
3. Create token:
vault token create -policy=default -ttl=5m
-use-limit=1
4. Writes auth & wrap token
File/Volume/Env
variable
5. Reads tokens
6. Unwraps data using passed tokens:
VAULT_TOKEN=%auth-token%
vault unwrap %wrap-token%
Raises alarm if gets auth/unwrap error

46. TTL configuration best practices (Vault)
Reality:
● We have TTL for auth tokens & dynamic secrets.
● We have TTL & Max TTL for each of them
Best practices we use:
● Keep TTL for auth token very small (~3min. for machines, ~1h. for humans)
● Set auth token Max TTL where possible & but no more than 24h. for humans
● Keep TTLs for Dynamic Secrets very big. Their revocation will be triggered by auth token
expiration.

47. Summing up

48. Vlad Fedosov
Director of R&D
@Namecheap, Inc
vlad.fedosov@gmail.com
Slides:
Or just scan it:
bit.ly/2BRrn8V
Join Q&A at sli.do #XPDAYS