O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

HashiCorp Vault Workshop:幫 Credentials 找個窩

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Carregando em…3
×

Confira estes a seguir

1 de 109 Anúncio

HashiCorp Vault Workshop:幫 Credentials 找個窩

Baixar para ler offline

Credentials 究竟該儲存在哪邊才算安全一直是個很尷尬的問題,外洩的消息更是時有耳聞;而一般的應用程式設計時,很少考量到如何定期去更新 Credentials,這時 DevOps 的救星 HashiCorp 發現了大家的需求,因而推出 Vault 來作為 Credentials 的歸宿,其主要最大的兩大功能就是讓大家可以將 Credentials 安心地存放在其中,並且針對一些特定的應用服務提供動態 Credentials 的功能,此 Workshop 主要分成三個主軸:
- Vault 基本使用
- Dynamic Credentials 使用方式
- Vault 在生產環境使用的最佳準則

Credentials 究竟該儲存在哪邊才算安全一直是個很尷尬的問題,外洩的消息更是時有耳聞;而一般的應用程式設計時,很少考量到如何定期去更新 Credentials,這時 DevOps 的救星 HashiCorp 發現了大家的需求,因而推出 Vault 來作為 Credentials 的歸宿,其主要最大的兩大功能就是讓大家可以將 Credentials 安心地存放在其中,並且針對一些特定的應用服務提供動態 Credentials 的功能,此 Workshop 主要分成三個主軸:
- Vault 基本使用
- Dynamic Credentials 使用方式
- Vault 在生產環境使用的最佳準則

Anúncio
Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a HashiCorp Vault Workshop:幫 Credentials 找個窩 (20)

Anúncio

Mais de smalltown (20)

Mais recentes (20)

Anúncio

HashiCorp Vault Workshop:幫 Credentials 找個窩

  1. 1. HashiCorp Vault: 幫 Credentials 找個窩
  2. 2. DevOpsDays Taipei 2019 2 https://devopsdays.tw/cfs/
  3. 3. Taipei HUG 3 http://bit.ly/taipei-hug https://t.me/TaiwanHashiCorpUserGroup
  4. 4. We’re Hiring !!! 4 Software Engineer in Test Site Reliability Engineer
  5. 5. Outline ◉ CH00 Environment Setup ◉ CH01 Vault Basics ◉ CH02 Dynamic Credentials ◉ CH03 Daily Operation ◉ CH04 Go Production 5
  6. 6. Environment Setup0 6 Setup Workshop Environment
  7. 7. Setup Cloud9 Following pictures of installation guide comes from: https://github.com/pahud/amazon-eks-workshop 7
  8. 8. Create environment 8
  9. 9. Name environment 9
  10. 10. 10
  11. 11. 11
  12. 12. Name environment 12
  13. 13. Name environment 13
  14. 14. 7. execute ‘aws configure‘ to configure the credentials for your IAM user. Make sure this IAM User has AdministratorAccess and run ‘aws sts get-caller-identity’ - you should be able to see the returned JSON output like this. 14
  15. 15. Create IAM Key if You Have No One (1/4) 15
  16. 16. Create IAM Key if You Have No One (2/4) 16
  17. 17. Create IAM Key if You Have No One (3/4) 17
  18. 18. Create IAM Key if You Have No One (4/4) 18
  19. 19. Run Command in Cloud9 $ git clone https://github.com/Taipei-HUG/workshop.git $ cd vault/CH00 $ ./step1.sh # get all binary $ ./step2.sh # setup eks cluster 19
  20. 20. Vault Basics Introduction 1 20
  21. 21. Questions ● Where do you store credentials? ● How do you rotate credentials? 21
  22. 22. Where do you store credentials? 22
  23. 23. How do you rotate credentials? ● Create a new credential ● Rotate credentials one by one ● Delete old credential ● What if you have 10 components connect to db? 23
  24. 24. Secret Management 24
  25. 25. Where do you store credentials? 25
  26. 26. Credential 1 Credential 2 1. Get Credential by access token or other auth method 2. Access db via credential Basic secret management 0. Admin create credential 26
  27. 27. Vault Feature ● Web UI/CLI/API/SDK ● Centralized secret management ● Secure Secret Storage ● Dynamic Secrets ● Leasing and Renewal ● Revocation 27
  28. 28. Credential 1 User create credential Encrypt and save to Storage 28
  29. 29. Vault Storage | S3 DynamoDB 29
  30. 30. Vault Feature ● Web UI/CLI/API/SDK ● Centralized secret management ● Secure Secret Storage ● Dynamic Secrets ● Leasing and Renewal ● Revocation 30
  31. 31. CMS cluster Dynamic secret endpoint 1 Dynamic secret endpoint 2User: vault-cms-1 Password: dynamic-pw-1 1 2 User: vault-cms-2 Password: dynamic-pw-2 Access via dynamic secret 31
  32. 32. Vault Feature ● Web UI/CLI/API/SDK ● Centralized secret management ● Secure Secret Storage ● Dynamic Secrets ● Leasing and Renewal ● Revocation 32
  33. 33. Credential 1 Credential 2 1. Get Credential by access token or other auth method 2. Access db via credential Basic secret management 0. User create credential 33
  34. 34. Setup Vault on docker-compose # Open cloud9 ide & see CH01/commands.txt $ docker-compose up -d $ export VAULT_ADDR=http://127.0.0.1:8080 $ export VAULT_TOKEN=my-root-token $ vault status 34
  35. 35. Manipulate vault kv $ vault kv list secret $ vault kv put secret/first-secret foo=bar $ vault kv list secret $ vault kv get secret/first-secret $ vault kv put secret/first-secret foo=bar test=true $ vault kv metadata get secret/first-secret $ vault kv delete secret/first-secret $ vault kv metadata delete secret/first-secret 35
  36. 36. Vault UI ● Cloud9 → Preview → Preview Running Application 36
  37. 37. 37
  38. 38. Dynamic Secret
  39. 39. CMS cluster Dynamic secret endpoint 1 Dynamic secret endpoint 2User: vault-cms-1 Password: dynamic-pw-1 1 2 User: vault-cms-2 Password: dynamic-pw-2 Access via dynamic secret 39
  40. 40. Vault Dynamic Secret https://learn.hashicorp.com/vault/secrets-management/sm-dynamic-secrets 40
  41. 41. Request a dynamic credential Create a user with certain scope Returns a credentialReturns a credential Access database via the credential Revoke the credential Graceful shutdown Delete the user User deletedCredential deleted 41
  42. 42. Setup Dynamic Secret ● See init.sh ● Setup Vault Dynamic Secret ● Integrate into our service 42
  43. 43. $ vault secrets enable database 43
  44. 44. Supported engines ● Database ○ MySQL, PostgreSQL, MongoDB, etc. ● SSH ● AWS IAM ● Vault docs - secret engine 44
  45. 45. $ vault write database/config/my-database plugin_name=mysql-database-plugin connection_url="{{username}}:{{password}}@tcp(mysql_url:3306)/" allowed_roles=my-role username=${MYSQL_ROOT_USERNAME} password=${MYSQL_ROOT_PASSWORD} 45
  46. 46. $ vault write database/roles/my-role db_name=my-database creation_statements= "CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}'; GRANT SELECT ON *.* TO '{{name}}'@'%';" default_ttl="1h" max_ttl="2h" 46
  47. 47. 1. Request credential --- $ vault read database/creds/my-role 2. Issue a dynamic credential with TTL by root credential 3. Get credential 4. Access db by dynamic credential CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}'; GRANT SELECT ON *.* TO '{{name}}'@'%'; Execute by username=${MYSQL_ROOT_USERNAME} password=${MYSQL_ROOT_PASSWORD} 47
  48. 48. $ npm install $ node index.js 48
  49. 49. const credential = await vault.read("database/creds/my-role"); const { username: user, password } = credential.data; const conn = await mysql.createConnection({ host, port, user, password }); const result = await conn.query("SELECT USER() as user"); await vault.revoke({ lease_id: credential.lease_id }); 49
  50. 50. Revocation ● Manually revoke by cli/api ● Automatically revoke by TTL $ vault read database/creds/my-role Key Value --- ----- lease_id database/creds/my-role/IPUkANwU080vaJwARYm4S8NT lease_duration 1h lease_renewable true password A1a-5pV9iwoVWLmvh3Fu username v-token-my-role-wGzYgA6g8DozFW0k 50
  51. 51. Credential 1 Credential 2 1. Get Credential by access token or other auth method 2. Access db via credential Basic secret management 0. User create credential 51
  52. 52. Auth Methods ● Vault token ● Kubernetes ● AWS IAM ● LDAP ● … a lot 52
  53. 53. Kubernetes cluster Deployment A Deployment B Credential 1 Credential 2 Role A - Policy A Role B - Policy B Credential 1 Credential 2 See more: Vault 與 Kubernetes 的深度整合 Vault 53
  54. 54. 54
  55. 55. Put all together ● Dynamic Secret ● Kubernetes service authentication ● Limit permission scope 55
  56. 56. Ideal Credential Lifecycle 56 Service is Accessed Application 1. Request Access Credential (Running) 2. Use the Credential to Access Service 3. Revoke the Credential Credentials Only Exist in Memory
  57. 57. 57
  58. 58. Daily Operation3 58 Overview of How to Maintain Vault
  59. 59. ◉ When a Vault server is first initialized, Vault generates a master key ◉ Immediately splits this master key into a series of key shares following Shamir's Secret Sharing Algorithm Master Key 59
  60. 60. ◉ The master key is used to decrypt the underlying encryption key ◉ Vault uses the encryption key to encrypt data at rest in a storage backend like the filesystem or Consul Encryption Key 60
  61. 61. ◉ Vault never stores the master key, therefore, the only way to retrieve the master key is to have a quorum of unseal keys re-generate it. Seal/Unseal 61
  62. 62. Practice (1/3) # switch to CH03 folder ~$ cd vault/CH03 # boot vault server and login it ~$ ./start_local_vault.sh # check vault status ~$ vault status Key Value --- ----- Seal Type shamir Initialized false Sealed true Total Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version n/a HA Enabled false 62
  63. 63. Practice (2/3) # initialize vault ~$ vault operator init # keep the unseal key and root token Unseal Key 1: QosKlf+rXJkDLZJX7tgpiKj8zDDNzGHrv4HrV2C9xlxH Unseal Key 2: 8dUP2J+1vPQcRM09QdK5Lo83YnHHjDe1nLdfAjESSKfa Unseal Key 3: t89QgyeOIC4W/7ZRCcMzrgPBYLhGeVOMtuHVNU2IQ5k2 Unseal Key 4: KPnn8uvGWu5DOpcuQA1/1DnjGCiakPWdP1ExtAcQm0L7 Unseal Key 5: LPCOtvaU944O2tbWXoeJGo7SL4d6pc0iFA8vrUv1gHyW Initial Root Token: s.96aLKTnD3WRiT0STWeLbLDDI 63
  64. 64. Practice (3/3) # unseal vault # repeat 3 times ~$ vault operator unseal Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 1/3 Unseal Nonce a0dfd3da-0fcb-0268-baba-ef4cbe 5550bc Version 1.1.2 HA Enabled false Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.1.2 Cluster Name vault-cluster-59fe6b22 Cluster ID 81a9858f-a363-74c7-931b-ec2b0f42 6e08 HA Enabled false 64
  65. 65. ◉ AliCloud KMS, Amazon KMS, Azure Key Vault, and Google Cloud KMS Auto-Unseal 65
  66. 66. Practice (1/6) # exit vault server container by Ctrl+D # create kms key ~$ aws kms create-key { "KeyMetadata": { "AWSAccountId": "123456789012", "KeyId": "xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyyyyy", "Arn": "arn:aws:kms:us-west-2:123456789012:key/xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyy yyy", "CreationDate": 1559233248.825, ... 66
  67. 67. Practice (2/6) ◉ Append seal config section into config/default.hcl … seal "awskms" { region = "us-west-2" kms_key_id = "xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyyyyy" } 67
  68. 68. Practice (3/6) ◉ Add AWS AK/SK into .env file VAULT_ADDR=http://127.0.0.1:8200 AWS_ACCESS_KEY_ID=DFJLSFKJLD8358KJLJK8 AWS_SECRET_ACCESS_KEY=JioeuJek7+jgJLIUJWTYSfv3rr49JRoqt 68
  69. 69. Practice (4/6) # restart vault server ~$ ./restart_local_vault.sh # check vault status ~$ vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed true Total Recovery Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Seal Migration in Progress true Version 1.1.2 HA Enabled false 69
  70. 70. Practice (5/6) # seal migration # repeat 3 times ~$ vault operator unseal -migrate Unseal Key (will be hidden): Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 Version 1.1.2 Cluster Name vault-cluster-59fe6b22 Cluster ID 81a9858f-a363-74c7-931b-ec2b0f42 6e08 HA Enabled false 70
  71. 71. Practice (6/6) # exit vault server container by Ctrl+D # restart vault server ~$ ./restart_local_vault.sh # check vault status ~$ vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 Version 1.1.2 Cluster Name vault-cluster-59fe6b22 Cluster ID 81a9858f-a363-74c7-931b-ec2b0f42 6e08 HA Enabled false 71
  72. 72. Authentication 72
  73. 73. Practice (1/2) # enable Userpass auth method ~$ export VAULT_TOKEN="s.96aLKTnD3WRiT0STWeLbLDDI" ~$ vault auth enable userpass Success! Enabled userpass auth method at: userpass/ ~$ vault write auth/userpass/users/smalltown password=12345678 policies=default Success! Data written to: auth/userpass/users/smalltown 73
  74. 74. Practice (2/2) # try to login vault by account & password ~$ vault login -method=userpass username=smalltown password=12345678 Key Value --- ----- token s.MXYyp2Q9OB1iVQJlhso3v3an token_accessor hPntyUJor6sMb1Iw3XwbW9qi token_duration 768h token_renewable true token_policies ["default"] identity_policies [] policies ["default"] token_meta_username smalltown 74
  75. 75. Authorization 75
  76. 76. Policies ◉ Vault store credentials like key/value database, e.g. ○ secret/stag/database/admin ○ secret/prod/database/admin ◉ Hence, predefined policy grant appropriate permission, e.g. path "secret/stag/database/admin" { capabilities = ["read"] } 76
  77. 77. Practice (1/3) # enable kv ~$ vault secrets enable -version=2 kv # put database credentials into vault ~$ vault kv put kv/stag/database/admin account=stag password=12345678 ~$ vault kv put kv/prod/database/admin account=prod password=87654321 77
  78. 78. Practice (2/3) # create policy ~$ vault policy write stag /vault/policy/stag.hcl ~$ vault policy write prod /vault/policy/prod.hcl # assign policy ~$ vault write auth/userpass/users/smalltown password=12345678 policies=stag 78
  79. 79. Practice (3/3) # login user and use the token ~$ unset VAULT_TOKEN ~$ vault login -method=userpass username=smalltown password=12345678 # try to get the database credentials ~$ vault kv get kv/stag/database/admin ~$ vault kv get kv/prod/database/admin 79
  80. 80. Storage Backend 80
  81. 81. Practice (1/4) # exit vault server container by Ctrl+D # create dynamodb ~$ aws dynamodb create-table --table-name vault-workshop --attribute-definitions AttributeName=Path,AttributeType=S AttributeName=Key,AttributeType=S --key-schema AttributeName=Path,KeyType=HASH AttributeName=Key,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=10,WriteCapacityUnits=10 81
  82. 82. Practice (2/4) # migrate storage ~$ ./login_local_vault.sh ~$ export VAULT_TOKEN="s.96aLKTnD3WRiT0STWeLbLDDI" ~$ vault operator migrate -config /vault/config/migration.hcl … 2019-05-31T05:37:56.188Z [INFO] copied key: path=sys/token/id/hc99e5ba3d69e7cbaecf0489031eadbf11b55984d698d0f9784e5e1a28 25f93fe 2019-05-31T05:37:56.485Z [INFO] copied key: path=sys/token/salt Success! All of the keys have been migrated. 82
  83. 83. Practice (3/4) # modify the config file config/default.hcl storage "file" { path = "/vault/file" } api_addr = "https://vault-workshop.hub.internal" storage "dynamodb" { ha_enabled = "true" region = "us-west-2" table = "vault-workshop" } 83
  84. 84. Practice (4/4) # exit vault server container by Ctrl+D # restart vault server ~$ ./restart_local_vault.sh # check vault status ~$ vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 5 Threshold 3 Version 1.1.2 Cluster Name vault-cluster-59fe6b22 Cluster ID 81a9858f-a363-74c7-931b-ec2b0f42 6e08 HA Enabled true HA Cluster https://vault-workshop.hub.internal: 444 HA Mode active 84
  85. 85. Audit Devices 85
  86. 86. Practice # enable auditing device ~$ export VAULT_TOKEN="s.96aLKTnD3WRiT0STWeLbLDDI" ~$ vault audit enable file file_path=/vault/logs/audit.log Success! Enabled the file audit device at: file/ ~$ cat /vault/logs/audit.log {"time":"2019-05-31T06:13:54.1656812Z","type":"response","auth":{"client_token":"hmac-sha256:a5a7726a6e12c568dad5caf12 bec4841ca775d28bbd0f7683c09ebb260ae604c","accessor":"hmac-sha256:38c0629c4ecca5961e9954cbafc80cfd009d6c0 31516fc2f13da54b2eea9e1e0","display_name":"root","policies":["root"],"token_policies":["root"],"metadata":null,"entity_id":"","t oken_type":"service"},"request":{"id":"219eb2f0-a0a7-59a0-c2b8-e7eeac31f31c","operation":"update","client_token":"hmac-s ha256:a5a7726a6e12c568dad5caf12bec4841ca775d28bbd0f7683c09ebb260ae604c","client_token_accessor":"hmac-sha25 6:38c0629c4ecca5961e9954cbafc80cfd009d6c031516fc2f13da54b2eea9e1e0","namespace":{"id":"root","path":""},"path":"sys /audit/file","data":{"description":"hmac-sha256:3eaeeda1a043e3504c691abdb19982eac0dc42c6d9d3cffda9e7cceaa7e3a88 7","local":false,"options":{"file_path":"hmac-sha256:a0a483f16d313b0b882262eea4bb58358ef1323be63b68eb51da01ebf26b dd1a"},"type":"hmac-sha256:b26ec68f8abd74be4fc88990ebba948314f39b1b428091636b1e7fde2c0e9939"},"policy_override" :false,"remote_address":"127.0.0.1","wrap_ttl":0,"headers":{}},"response":{"headers":null},"error":""} 86
  87. 87. Clean # exit vault server container by Ctrl+D # shutdown vault container ~$ ./stop_local_vault.sh # delete kms key ~$ aws kms schedule-key-deletion --key-id xxxxxxxx-wwww-xxxx-zzzz-yyyyyyyyyyyy # delete dynamodb ~$ aws dynamodb delete-table --table-name vault-workshop 87
  88. 88. Go Production4 88 There are Things Need to Do Before Vault Online
  89. 89. High Availability 89
  90. 90. Banzai Cloud - bank-vaults ◉ Bank-Vaults provides various tools for Hashicorp Vault to make its use easier 90
  91. 91. Practice (1/2) # install vault cluster ~$ cd vault/CH04 ~$ ./setup_vault_cluster.sh 91
  92. 92. Practice (2/2) # check vault cluster ~$ kubectl get pod NAME READY STATUS RESTARTS AGE etcd-cluster-vj8r8mwslr 1/1 Running 0 3m8s etcd-operator-588ccb98fd-twrs4 1/1 Running 0 8m46s vault-0 3/3 Running 6 3m42s vault-1 3/3 Running 8 3m42s vault-configurer-759d4679dd-8lsjx 1/1 Running 0 3m42s vault-operator-b8c5c566c-w59b5 1/1 Running 0 5m52s 92
  93. 93. What bank-vaults Do? bank-vaults operator bank-vaults cli Vault storage backend created by ectd operator Create everything needed by vault: bank-vaults cli container, vault, prometheus-exporter bank-vaults cli: 1. Initial and Unseal automatically 2. Configure policy, secret engine...etc 93
  94. 94. How to Monitor Vault 94
  95. 95. Telemetry ◉ The Vault server process collects various runtime metrics about the performance of different libraries and subsystems. ◉ To view the telemetry information, you must send a signal to the Vault process ◉ Also can be streamed directly from Vault to a range of metrics aggregation solutions 95
  96. 96. Monitoring Mechanism StatsD Exporter - Collect telemetry information - Expose to Prometheus - Prometheus scrape metrics - Grafana generate Dashboard 96
  97. 97. Practice (1/5) # install prometheus ~$ ./setup_prometheus.sh ~$ kubectl create -f vault/servicemonitor.yaml ~$ kubectl get pod -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 0 8m42s alertmanager-main-1 2/2 Running 0 8m30s alertmanager-main-2 2/2 Running 0 8m19s grafana-74b6b56ddf-5k8mz 1/1 Running 0 8m37s kube-state-metrics-7c7979b6bc-zwrwz 4/4 Running 0 5m56s ... 97
  98. 98. Practice (2/5) # check prometheus scrape vault metric ~$ ./get_prometheus.sh Grafana http://a951eb90e83b511e99a350a7fef3486e-1999986850.us-west-2.elb.amazonaws. com:3000 Prometheus http://aa1ffcdb183b511e99a350a7fef3486e-1607220011.us-west-2.elb.amazonaws.co m:9090 ... 98
  99. 99. Practice (3/5) ◉ Visit Prometheus, then the vault telemetry can be acquired 99
  100. 100. ◉ Visit Grafana -> Click Import ◉ Paste JSON from grafana-vault.json ◉ Click Import Practice (4/5) 100
  101. 101. Practice (5/5) 101
  102. 102. Clean # Don’t forget to remove the resource create in CH04 ~$ ./uninstall.sh 102
  103. 103. All Thing Down !!!
  104. 104. Of Course Not!!!
  105. 105. Production Hardening (1/3) ◉ End-to-End TLS (O) ◉ Single Tenancy (X) ◉ Firewall traffic (X) ◉ Disable SSH / Remote Desktop (X) ◉ Disable Swap (X) ◉ Don't Run as Root (O) 105
  106. 106. Production Hardening (2/3) ◉ Turn Off Core Dumps (X) ◉ Immutable Upgrades (O) ◉ Avoid Root Tokens (X) ◉ Enable Auditing (O) ◉ Upgrade Frequently (X) ◉ Configure SELinux / AppArmor (X) 106
  107. 107. Production Hardening (3/3) ◉ Restrict Storage Access (X) ◉ Disable Shell Command History (X) ◉ Tweak ulimits (X) ◉ Docker Containers (X) Ref 107
  108. 108. Any questions ? Thanks!
  109. 109. We’re Hiring !!! 109 Software Engineer in Test Site Reliability Engineer

×