1.
HashiCorp Vault:
幫 Credentials 找個窩

2.
DevOpsDays Taipei 2019
2
https://devopsdays.tw/cfs/

3.
Taipei HUG
3
http://bit.ly/taipei-hug
https://t.me/TaiwanHashiCorpUserGroup

4.
We’re Hiring !!!
4
Software Engineer in
Test
Site Reliability Engineer

5.
Outline
◉ CH00 Environment Setup
◉ CH01 Vault Basics
◉ CH02 Dynamic Credentials
◉ CH03 Daily Operation
◉ CH04 Go Production
5

6.
Environment Setup0
6
Setup Workshop Environment

7.
Setup Cloud9
Following pictures of installation guide comes from:
https://github.com/pahud/amazon-eks-workshop
7

8.
Create environment
8

9.
Name environment
9

10.
10

11.
11

12.
Name environment
12

13.
Name environment
13

14.
7. execute ‘aws configure‘ to configure the credentials for your IAM user. Make sure this IAM User has
AdministratorAccess and run ‘aws sts get-caller-identity’ - you should be able to see the returned
JSON output like this.
14

15.
Create IAM Key if You Have No One (1/4)
15

16.
Create IAM Key if You Have No One (2/4)
16

17.
Create IAM Key if You Have No One (3/4)
17

18.
Create IAM Key if You Have No One (4/4)
18

19.
Run Command in Cloud9
$ git clone https://github.com/Taipei-HUG/workshop.git
$ cd vault/CH00
$ ./step1.sh # get all binary
$ ./step2.sh # setup eks cluster
19

20.
Vault Basics
Introduction
1
20

21.
Questions
● Where do you store credentials?
● How do you rotate credentials?
21

22.
Where do you store credentials?
22

23.
How do you rotate credentials?
● Create a new credential
● Rotate credentials one by one
● Delete old credential
● What if you have 10 components
connect to db?
23

24.
Secret Management
24

25.
Where do you store credentials?
25

26.
Credential 1
Credential 2
1. Get Credential by access
token or other auth method
2. Access db via credential
Basic secret management
0. Admin create
credential
26

27.
Vault Feature
● Web UI/CLI/API/SDK
● Centralized secret management
● Secure Secret Storage
● Dynamic Secrets
● Leasing and Renewal
● Revocation
27

28.
Credential 1
User create credential Encrypt and save to
Storage
28

29.
Vault Storage
| S3
DynamoDB
29

30.
Vault Feature
● Web UI/CLI/API/SDK
● Centralized secret management
● Secure Secret Storage
● Dynamic Secrets
● Leasing and Renewal
● Revocation
30

31.
CMS cluster
Dynamic secret endpoint 1
Dynamic secret endpoint 2User: vault-cms-1
Password: dynamic-pw-1
1
2
User: vault-cms-2
Password: dynamic-pw-2
Access via dynamic secret
31

32.
Vault Feature
● Web UI/CLI/API/SDK
● Centralized secret management
● Secure Secret Storage
● Dynamic Secrets
● Leasing and Renewal
● Revocation
32

33.
Credential 1
Credential 2
1. Get Credential by access
token or other auth method
2. Access db via credential
Basic secret management
0. User create
credential
33

34.
Setup Vault on docker-compose
# Open cloud9 ide & see CH01/commands.txt
$ docker-compose up -d
$ export VAULT_ADDR=http://127.0.0.1:8080
$ export VAULT_TOKEN=my-root-token
$ vault status
34

35.
Manipulate vault kv
$ vault kv list secret
$ vault kv put secret/first-secret foo=bar
$ vault kv list secret
$ vault kv get secret/first-secret
$ vault kv put secret/first-secret foo=bar test=true
$ vault kv metadata get secret/first-secret
$ vault kv delete secret/first-secret
$ vault kv metadata delete secret/first-secret
35

36.
Vault UI
● Cloud9 → Preview → Preview Running
Application
36

37.
37

38.
Dynamic Secret

39.
CMS cluster
Dynamic secret endpoint 1
Dynamic secret endpoint 2User: vault-cms-1
Password: dynamic-pw-1
1
2
User: vault-cms-2
Password: dynamic-pw-2
Access via dynamic secret
39

40.
Vault Dynamic Secret
https://learn.hashicorp.com/vault/secrets-management/sm-dynamic-secrets 40

41.
Request a dynamic credential Create a user with certain scope
Returns a credentialReturns a credential
Access database via the credential
Revoke the credential
Graceful
shutdown
Delete the user
User deletedCredential deleted
41

42.
Setup Dynamic Secret
● See init.sh
● Setup Vault Dynamic Secret
● Integrate into our service
42

43.
$ vault secrets enable database
43

44.
Supported engines
● Database
○ MySQL, PostgreSQL, MongoDB, etc.
● SSH
● AWS IAM
● Vault docs - secret engine
44

45.
$ vault write database/config/my-database
plugin_name=mysql-database-plugin
connection_url="{{username}}:{{password}}@tcp(mysql_url:3306)/"
allowed_roles=my-role
username=${MYSQL_ROOT_USERNAME}
password=${MYSQL_ROOT_PASSWORD}
45

46.
$ vault write database/roles/my-role
db_name=my-database
creation_statements=
"CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';
GRANT SELECT ON *.* TO '{{name}}'@'%';"
default_ttl="1h"
max_ttl="2h"
46

47.
1. Request credential
---
$ vault read database/creds/my-role
2. Issue a dynamic
credential with TTL
by root credential
3. Get credential
4. Access db by dynamic credential
CREATE USER '{{name}}'@'%'
IDENTIFIED BY '{{password}}';
GRANT SELECT ON *.*
TO '{{name}}'@'%';
Execute by
username=${MYSQL_ROOT_USERNAME}
password=${MYSQL_ROOT_PASSWORD}
47

48.
$ npm install
$ node index.js
48

49.
const credential = await vault.read("database/creds/my-role");
const { username: user, password } = credential.data;
const conn = await mysql.createConnection({ host, port, user, password });
const result = await conn.query("SELECT USER() as user");
await vault.revoke({ lease_id: credential.lease_id });
49

50.
Revocation
● Manually revoke by cli/api
● Automatically revoke by TTL
$ vault read database/creds/my-role
Key Value
--- -----
lease_id
database/creds/my-role/IPUkANwU080vaJwARYm4S8NT
lease_duration 1h
lease_renewable true
password A1a-5pV9iwoVWLmvh3Fu
username v-token-my-role-wGzYgA6g8DozFW0k
50

51.
Credential 1
Credential 2
1. Get Credential by access
token or other auth method
2. Access db via credential
Basic secret management
0. User create
credential
51

52.
Auth Methods
● Vault token
● Kubernetes
● AWS IAM
● LDAP
● … a lot
52

53.
Kubernetes cluster
Deployment A
Deployment B
Credential 1
Credential 2
Role A - Policy A
Role B - Policy B
Credential 1
Credential 2
See more: Vault 與 Kubernetes 的深度整合
Vault
53

54.
54

55.
Put all together
● Dynamic Secret
● Kubernetes service authentication
● Limit permission scope
55

56.
Ideal Credential Lifecycle
56
Service is Accessed
Application
1. Request Access Credential (Running)
2. Use the Credential to Access Service
3. Revoke the Credential
Credentials Only
Exist in Memory

57.
57

58.
Daily Operation3
58
Overview of How to Maintain Vault

59.
◉ When a Vault server is first
initialized, Vault generates a
master key
◉ Immediately splits this master
key into a series of key shares
following Shamir's Secret
Sharing Algorithm
Master Key
59

60.
◉ The master key is used to decrypt the
underlying encryption key
◉ Vault uses the encryption key to encrypt data
at rest in a storage backend like the filesystem
or Consul
Encryption Key
60

61.
◉ Vault never stores the master key, therefore,
the only way to retrieve the master key is to
have a quorum of unseal keys re-generate it.
Seal/Unseal
61

62.
Practice (1/3)
# switch to CH03 folder
~$ cd vault/CH03
# boot vault server and login it
~$ ./start_local_vault.sh
# check vault status
~$ vault status
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version n/a
HA Enabled false
62

63.
Practice (2/3)
# initialize vault
~$ vault operator init
# keep the unseal key and root token
Unseal Key 1: QosKlf+rXJkDLZJX7tgpiKj8zDDNzGHrv4HrV2C9xlxH
Unseal Key 2: 8dUP2J+1vPQcRM09QdK5Lo83YnHHjDe1nLdfAjESSKfa
Unseal Key 3: t89QgyeOIC4W/7ZRCcMzrgPBYLhGeVOMtuHVNU2IQ5k2
Unseal Key 4: KPnn8uvGWu5DOpcuQA1/1DnjGCiakPWdP1ExtAcQm0L7
Unseal Key 5: LPCOtvaU944O2tbWXoeJGo7SL4d6pc0iFA8vrUv1gHyW
Initial Root Token: s.96aLKTnD3WRiT0STWeLbLDDI
63

64.
Practice (3/3)
# unseal vault
# repeat 3 times
~$ vault operator unseal
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce
a0dfd3da-0fcb-0268-baba-ef4cbe
5550bc
Version 1.1.2
HA Enabled false
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.1.2
Cluster Name
vault-cluster-59fe6b22
Cluster ID
81a9858f-a363-74c7-931b-ec2b0f42
6e08
HA Enabled false 64

65.
◉ AliCloud KMS, Amazon KMS, Azure Key Vault,
and Google Cloud KMS
Auto-Unseal
65

66.
Practice (1/6)
# exit vault server container by Ctrl+D
# create kms key
~$ aws kms create-key
{
"KeyMetadata": {
"AWSAccountId": "123456789012",
"KeyId": "xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyyyyy",
"Arn":
"arn:aws:kms:us-west-2:123456789012:key/xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyy
yyy",
"CreationDate": 1559233248.825,
...
66

67.
Practice (2/6)
◉ Append seal config section into config/default.hcl
…
seal "awskms" {
region = "us-west-2"
kms_key_id = "xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyyyyy"
}
67

68.
Practice (3/6)
◉ Add AWS AK/SK into .env file
VAULT_ADDR=http://127.0.0.1:8200
AWS_ACCESS_KEY_ID=DFJLSFKJLD8358KJLJK8
AWS_SECRET_ACCESS_KEY=JioeuJek7+jgJLIUJWTYSfv3rr49JRoqt
68

69.
Practice (4/6)
# restart vault server
~$ ./restart_local_vault.sh
# check vault status
~$ vault status
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed true
Total Recovery Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Seal Migration in Progress true
Version 1.1.2
HA Enabled false
69

70.
Practice (5/6)
# seal migration
# repeat 3 times
~$ vault operator unseal -migrate
Unseal Key (will be hidden):
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.1.2
Cluster Name
vault-cluster-59fe6b22
Cluster ID
81a9858f-a363-74c7-931b-ec2b0f42
6e08
HA Enabled false
70

71.
Practice (6/6)
# exit vault server container by Ctrl+D
# restart vault server
~$ ./restart_local_vault.sh
# check vault status
~$ vault status
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.1.2
Cluster Name
vault-cluster-59fe6b22
Cluster ID
81a9858f-a363-74c7-931b-ec2b0f42
6e08
HA Enabled false
71

72.
Authentication
72

73.
Practice (1/2)
# enable Userpass auth method
~$ export VAULT_TOKEN="s.96aLKTnD3WRiT0STWeLbLDDI"
~$ vault auth enable userpass
Success! Enabled userpass auth method at: userpass/
~$ vault write auth/userpass/users/smalltown
password=12345678 policies=default
Success! Data written to: auth/userpass/users/smalltown
73

74.
Practice (2/2)
# try to login vault by account &
password
~$ vault login -method=userpass
username=smalltown
password=12345678
Key Value
--- -----
token
s.MXYyp2Q9OB1iVQJlhso3v3an
token_accessor
hPntyUJor6sMb1Iw3XwbW9qi
token_duration 768h
token_renewable true
token_policies ["default"]
identity_policies []
policies ["default"]
token_meta_username smalltown
74

75.
Authorization
75

76.
Policies
◉ Vault store credentials like key/value database, e.g.
○ secret/stag/database/admin
○ secret/prod/database/admin
◉ Hence, predefined policy grant appropriate
permission, e.g.
path "secret/stag/database/admin" {
capabilities = ["read"]
}
76

77.
Practice (1/3)
# enable kv
~$ vault secrets enable -version=2 kv
# put database credentials into vault
~$ vault kv put kv/stag/database/admin account=stag
password=12345678
~$ vault kv put kv/prod/database/admin account=prod
password=87654321
77

78.
Practice (2/3)
# create policy
~$ vault policy write stag /vault/policy/stag.hcl
~$ vault policy write prod /vault/policy/prod.hcl
# assign policy
~$ vault write auth/userpass/users/smalltown
password=12345678 policies=stag
78

79.
Practice (3/3)
# login user and use the token
~$ unset VAULT_TOKEN
~$ vault login -method=userpass username=smalltown
password=12345678
# try to get the database credentials
~$ vault kv get kv/stag/database/admin
~$ vault kv get kv/prod/database/admin
79

80.
Storage Backend
80

81.
Practice (1/4)
# exit vault server container by Ctrl+D
# create dynamodb
~$ aws dynamodb create-table --table-name vault-workshop
--attribute-definitions AttributeName=Path,AttributeType=S
AttributeName=Key,AttributeType=S --key-schema
AttributeName=Path,KeyType=HASH
AttributeName=Key,KeyType=RANGE --provisioned-throughput
ReadCapacityUnits=10,WriteCapacityUnits=10
81

82.
Practice (2/4)
# migrate storage
~$ ./login_local_vault.sh
~$ export VAULT_TOKEN="s.96aLKTnD3WRiT0STWeLbLDDI"
~$ vault operator migrate -config /vault/config/migration.hcl
…
2019-05-31T05:37:56.188Z [INFO] copied key:
path=sys/token/id/hc99e5ba3d69e7cbaecf0489031eadbf11b55984d698d0f9784e5e1a28
25f93fe
2019-05-31T05:37:56.485Z [INFO] copied key: path=sys/token/salt
Success! All of the keys have been migrated.
82

83.
Practice (3/4)
# modify the config file config/default.hcl
storage "file" {
path = "/vault/file"
}
api_addr = "https://vault-workshop.hub.internal"
storage "dynamodb" {
ha_enabled = "true"
region = "us-west-2"
table = "vault-workshop"
} 83

84.
Practice (4/4)
# exit vault server container by Ctrl+D
# restart vault server
~$ ./restart_local_vault.sh
# check vault status
~$ vault status
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.1.2
Cluster Name
vault-cluster-59fe6b22
Cluster ID
81a9858f-a363-74c7-931b-ec2b0f42
6e08
HA Enabled true
HA Cluster
https://vault-workshop.hub.internal:
444
HA Mode active 84

85.
Audit Devices
85

86.
Practice
# enable auditing device
~$ export VAULT_TOKEN="s.96aLKTnD3WRiT0STWeLbLDDI"
~$ vault audit enable file file_path=/vault/logs/audit.log
Success! Enabled the file audit device at: file/
~$ cat /vault/logs/audit.log
{"time":"2019-05-31T06:13:54.1656812Z","type":"response","auth":{"client_token":"hmac-sha256:a5a7726a6e12c568dad5caf12
bec4841ca775d28bbd0f7683c09ebb260ae604c","accessor":"hmac-sha256:38c0629c4ecca5961e9954cbafc80cfd009d6c0
31516fc2f13da54b2eea9e1e0","display_name":"root","policies":["root"],"token_policies":["root"],"metadata":null,"entity_id":"","t
oken_type":"service"},"request":{"id":"219eb2f0-a0a7-59a0-c2b8-e7eeac31f31c","operation":"update","client_token":"hmac-s
ha256:a5a7726a6e12c568dad5caf12bec4841ca775d28bbd0f7683c09ebb260ae604c","client_token_accessor":"hmac-sha25
6:38c0629c4ecca5961e9954cbafc80cfd009d6c031516fc2f13da54b2eea9e1e0","namespace":{"id":"root","path":""},"path":"sys
/audit/file","data":{"description":"hmac-sha256:3eaeeda1a043e3504c691abdb19982eac0dc42c6d9d3cffda9e7cceaa7e3a88
7","local":false,"options":{"file_path":"hmac-sha256:a0a483f16d313b0b882262eea4bb58358ef1323be63b68eb51da01ebf26b
dd1a"},"type":"hmac-sha256:b26ec68f8abd74be4fc88990ebba948314f39b1b428091636b1e7fde2c0e9939"},"policy_override"
:false,"remote_address":"127.0.0.1","wrap_ttl":0,"headers":{}},"response":{"headers":null},"error":""} 86

87.
Clean
# exit vault server container by Ctrl+D
# shutdown vault container
~$ ./stop_local_vault.sh
# delete kms key
~$ aws kms schedule-key-deletion --key-id
xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyyyyy
# delete dynamodb
~$ aws dynamodb delete-table --table-name vault-workshop
87

88.
Go Production4
88
There are Things Need to Do Before Vault Online

89.
High Availability
89

90.
Banzai Cloud - bank-vaults
◉ Bank-Vaults provides
various tools for
Hashicorp Vault to make
its use easier
90

91.
Practice (1/2)
# install vault cluster
~$ cd vault/CH04
~$ ./setup_vault_cluster.sh
91

92.
Practice (2/2)
# check vault cluster
~$ kubectl get pod
NAME READY STATUS RESTARTS AGE
etcd-cluster-vj8r8mwslr 1/1 Running 0 3m8s
etcd-operator-588ccb98fd-twrs4 1/1 Running 0 8m46s
vault-0 3/3 Running 6 3m42s
vault-1 3/3 Running 8 3m42s
vault-configurer-759d4679dd-8lsjx 1/1 Running 0 3m42s
vault-operator-b8c5c566c-w59b5 1/1 Running 0 5m52s
92

93.
What bank-vaults Do?
bank-vaults operator
bank-vaults cli Vault storage
backend created
by ectd operator
Create everything needed by vault: bank-vaults cli
container, vault, prometheus-exporter
bank-vaults cli:
1. Initial and Unseal
automatically
2. Configure policy,
secret engine...etc
93

94.
How to Monitor Vault
94

95.
Telemetry
◉ The Vault server process collects various
runtime metrics about the performance of
different libraries and subsystems.
◉ To view the telemetry information, you must
send a signal to the Vault process
◉ Also can be streamed directly from Vault to a
range of metrics aggregation solutions
95

96.
Monitoring Mechanism
StatsD Exporter
- Collect telemetry information
- Expose to Prometheus
- Prometheus scrape metrics
- Grafana generate Dashboard
96

97.
Practice (1/5)
# install prometheus
~$ ./setup_prometheus.sh
~$ kubectl create -f vault/servicemonitor.yaml
~$ kubectl get pod -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 8m42s
alertmanager-main-1 2/2 Running 0 8m30s
alertmanager-main-2 2/2 Running 0 8m19s
grafana-74b6b56ddf-5k8mz 1/1 Running 0 8m37s
kube-state-metrics-7c7979b6bc-zwrwz 4/4 Running 0 5m56s
... 97

98.
Practice (2/5)
# check prometheus scrape vault metric
~$ ./get_prometheus.sh
Grafana
http://a951eb90e83b511e99a350a7fef3486e-1999986850.us-west-2.elb.amazonaws.
com:3000
Prometheus
http://aa1ffcdb183b511e99a350a7fef3486e-1607220011.us-west-2.elb.amazonaws.co
m:9090
...
98

99.
Practice (3/5)
◉ Visit Prometheus, then the vault telemetry can
be acquired
99

100.
◉ Visit Grafana -> Click Import
◉ Paste JSON from grafana-vault.json
◉ Click Import
Practice (4/5)
100

101.
Practice (5/5)
101

102.
Clean
# Don’t forget to remove the resource create in CH04
~$ ./uninstall.sh
102

103.
All Thing Down !!!

104.
Of Course Not!!!

105.
Production Hardening (1/3)
◉ End-to-End TLS (O)
◉ Single Tenancy (X)
◉ Firewall traffic (X)
◉ Disable SSH / Remote Desktop (X)
◉ Disable Swap (X)
◉ Don't Run as Root (O)
105

106.
Production Hardening (2/3)
◉ Turn Off Core Dumps (X)
◉ Immutable Upgrades (O)
◉ Avoid Root Tokens (X)
◉ Enable Auditing (O)
◉ Upgrade Frequently (X)
◉ Configure SELinux / AppArmor (X)
106

107.
Production Hardening (3/3)
◉ Restrict Storage Access (X)
◉ Disable Shell Command History (X)
◉ Tweak ulimits (X)
◉ Docker Containers (X)
Ref
107

108.
Any questions ?
Thanks!

109.
We’re Hiring !!!
109
Software Engineer in
Test
Site Reliability Engineer