20190727 HashiCorp Consul Workshop: 管管你們家 config 啦

HashiCorp Consul Workshop: 管管你們家 config 啦
Levi Chen 20190727

Levi Chen
Software Engineer in Common Service Team @ 91APP
Contact Me @
● FB: https://www.facebook.com/ChenJiunYi
● Blog: http://blog.levichen.tw/
About Me

How many services do you have?
How do you manage your configuration?

Outline
CH00 Environment SetUp
CH01 Why Conﬁguration Management?
CH02 Play with Consul
CH03 Go Live

The Following Installation guild fom:
https://github.com/pahud/amazon-eks-workshop

Step 1. Choose region: us-west-2

Step 2. Spin up your Cloud9 IDE from AWS console

Step 3. Create and name your environment

Step 4. Choose Ubuntu Server 18.04 LTS

Step 5. Click create environment
(It would typically take 30-60 seconds to create your Cloud9 IDE)

Step6. Create a IAM Role which has AdministratorAccess permission
(If you already have it, please skip it to Step 11)

Step9. Attach AdministratorAccess Policy

Step10. Click Next, Next, Next to create to an Admin IAM Role

Step 11. Turn off the Cloud9 temporarily provided IAM credentials

Step 12. After turn off the temporary credentials, you should get this error message

Step 13. Find Cloud9 EC2 on AWS Console

Step 14. Right click the EC2 then attach Admin IAM Role to EC2

Step 15. Run aws sts get-caller-identity - you should be able to see the returned JSON
output like this.

$ git clone https://github.com/levichen/consul-workshop
$ cd consul-workshop/lab00
$ sudo ./00_install.sh
$ ./01_test.sh
Step 16. Download materials, install and confirm it ...

// comment
$ ----> execute in Cloud9
# ----> execute in Container
Command types

CH01 Why Configuration Management?
● Morden app are smaller, compostable & portable
● More ﬁxable service management
● Single code base multiple deployment
● External services are unstable

DevOpsDays Taipei 2018 - https://s.itho.me/devopsdays/2018/0911tracka-3.pdf

https://www.nginx.com/blog/nginmesh-nginx-as-a-proxy-in-an-istio-service-mesh/

E = Number of Environments
N = Number of Services
M = Number of Instances
The Deployment Complexity = E * N * M

The Server-Side Service Disvoery

How many Domain Name you need to configure,
if you want to build a environment?

Environment 1
A
B
C
D
E
a.environment1.com
b.environment1.com
c.environment1.com
d.environment1.com
e.environment1.com
Environment 2
A
B
C
D
E
a.environment2.com
b.environment2.com
c.environment2.com
d.environment2.com
e.environment2.com

The Client-Side Service Disvoery

Server Side Service Discovery: Pull
Client Side Service Discovery: Push

cm.environment2.com
Environment 1
A
B
C D
E
cm.environment1.com
CM
Environment 2
A
B
C D
ECM

Multiple talents infrastructure in client side service discovery
VIP
VIP
Normal

ClientA
ClientB
Shared
Users
a.client.service.com
b.client.service.com
shared.client.service.com
Multiple talents infrastructure in server side service discovery

Environment 1
Service A
CD Server
1. Build
2. Deploy

Environment 1
Service A
CD Server
1. Build
2. Deploy
Environment 2
Service A
CD Server
1. Build
2. Deploy

Too many CD Servers, and hard to scale

Environment 1
AM
CI Server
1. Build Code
CMService B 3. Get Configuration
2. Get Artifact
Environment 2
2. Get Artifact
Configuration
(Git)
1. Build Configuration

A
B
C
SMS #1
SMS #2
You need to deploy all of your services

A
B
C
SMS #1
SMS #2
What can CM do?
CM

A
B
C
SMS #1
SMS #2
health check for SMS #1, SMS #1
CM
Health Host : #1, #2

A
B
C
SMS #1
SMS #2
Service A, B, C get healthy hosts list
CMGet Conﬁguration

A
B
C
SMS #1
SMS #2
Send to SMS #1
CM

A
B
C
SMS #1
SMS #2
When SMS #1 crash
CM

A
B
C
SMS #1
SMS #2
CM will know it in short time (depends on your health check interval)
CM
Health Host : #2

A
B
C
SMS #1
SMS #2
Only get SMS #2
CMGet Conﬁguration
Health Host : #2

A
B
C
SMS #1
SMS #2
Send to SMS #2 without any deployment
CM
Health Host : #2

We need
● Conﬁguration store
● Service discovery
● Health check

Consul is a service networking solution to connect and secure services across any
runtime platform and public or private cloud

CH02 Play with Consul
● KV
● Registering service
● Health check
● Building Consul cluster

EC2
Consul
Process
eth0
HTTP 8080
8080
For lab01, lab02
Consul
Binrary

Please launch two terminals on Cloud9
Terminal 1 For starting Consul
Terminal 2 For executing commands

// terminal 1
$ consul agent -dev -ui -http-port 8080
// terminal 2
$ export CONSUL_HTTP_ADDR=http://localhost:8080
$ consul members
Launch Consul in Develop mode and check cluster members

CH02 Play with Consul: KV
● Key value store
● Used to hold dynamic conﬁguration

// In terminal 2
// Get key value
$ consul kv get redis/config/minconns
// Insert a key value paris
$ consul kv put redis/config/minconns 1
$ consul kv put redis/config/maxconns 25
// Get single key value
// Get key value recursively
$ consul kv get -recurse

// update
$ consul kv put redis/conﬁg/minconns 9
// delete
$ consul kv delete redis/conﬁg/minconns
$ consul kv delete -recurse redis
Delete commands is dangerous check your ACL configruation before go live

● Service Definition
○ Using statis service difinition files
● HTTP API
○ Using Consul command or HTTP API
CH02 Play with Consul: Registering Service

// terminal 1
// exit the previous consul process
$ cd ../lab02
$ consul agent -dev -ui -http-port 8080 -conﬁg-dir=conf.d
Launch Consul Again

Check Service Status on Consul UI

Querying Service via DNS
// terminal 2
$ dig @127.0.0.1 -p 8600 web.service.consul

Querying Service via HTTP API
// terminal 2
$ curl $CONSUL_HTTP_ADDR/v1/catalog/service/web

// terminal 2
$ cd consul-workshop/lab02/
$ curl --request PUT --data @webapi.json
${CONSUL_HTTP_ADDR}/v1/agent/service/register
Launch Consul Again

Querying Service
// terminal 2
$ dig @127.0.0.1 -p 8600 webapi.service.consul SRV
$ curl $CONSUL_HTTP_ADDR/v1/catalog/service/webapi

Service health check
Critical component of service discovery that prevent using services that are
unhealthy.
Two approach to register check:
● Check diﬁnation ﬁles
● HTTP API
Unhealth
● exit code > 0

Launch Consul service and try get service via Consul HTTP API
// terminal 1
$ cd ../lab03
$ consul agent -dev -ui -http-port 8080 -enable-script-checks -conﬁg-dir=./
// terminal 2
$ cd ../lab03
$ curl $CONSUL_HTTP_ADDR/v1/health/state/critical
$ dig @127.0.0.1 -p 8600 web.service.consul SRV

Check Google (external) status every 30 seconds via ping

There are to check checks in this nodes

Check local service status every 10 seconds via curl

Push mode: Service needs to send heartbeat every 20 seconds

// terminal 2
// send a heartbeat
$ curl -X PUT $CONSUL_HTTP_ADDR/v1/agent/check/pass/service:Batch

● Client passive (Pull)
○ Simpler
○ Bottleneck in the server
● Client active (Push)
○ Faster
Service health check: Push v.s Pull

● KV
○ Key value store
○ Used to hold dynamic conﬁguration
○ Static
○ Dynamic
● Health check
○ Consul helps you to check internal/external services
○ Push / Pull mode
Summary

Consul Cluster
Server
(Follower)
Server
(Leader)
Server
(Follower)
GOSSIP
Replication
Leader
Forwarding
Replication
Client Client Client
RPC
Leader
Forwarding
GOSSIP
RPCGOSSIP

EC2
Docker Network
Consul
Server
Consul
Client
eth0
8080
8080

Server
- bootstrap-expect: the number of expected servers in the datacenter
- ui: Enables the built-in web UI server and the required HTTP routes.
- client: The address to which Consul will bind client interfaces, including the HTTP and DNS
servers.
- node: The name of this node in the cluster.
Client
- join: Address of another agent to join upon starting up.

// terminal 1
$ cd ../lab04
$ docker-compose up -d
Using docker-compose to launch 1 Consul Server + 1 Consul Client

// terminal 2
// go into the Docker instance
$ docker exec -it consul-client sh
// get Consul Cluster information via local Consul agent
# consul members

// terminal 1
$ docker-compose down
Exit Docker Compose

Querying Service in Cluster
● 1 Consul Server
● 3 Consul Clients
● 1 Service with 2 nodes

EC2
Docker Network
Consul
Server
Consul
Client1
eth0
8080
8080
Web1
Consul
Client2
Web2
Consul
Client3

// terminal 1
$ cd ../lab05
Using docker-compose to launch 1 Consul Server + 3 Consul Clients + 2 webs

// terminal 2
// go into the Docker instance
$ docker exec -it consul-client3 sh
// get Consul Cluster via local Consul agent
# consul members
// get service information via HTTP API
# curl -G localhost:8500/v1/catalog/service/web | jq

// Get VIP instances
# curl -G localhost:8500/v1/catalog/service/web
--data-urlencode 'ﬁlter="VIP" in ServiceTags' | jq
// Get Normal instances
# curl -G localhost:8500/v1/catalog/service/web
--data-urlencode 'ﬁlter="Normal" in ServiceTags' | jq

// Get Passing checks
# curl -G localhost:8500/v1/health/checks/web
--data-urlencode 'ﬁlter=Status == "passing"' | jq
// Get Critical checks
--data-urlencode 'ﬁlter=Status == "critical"' | jq

// Launch 3rd terminal to stop web2
$ docker stop web2
// terminal 2
// Check web status in consul-client3

// terminal 3
$ docker start web2
// terminal 2
// check web status in consul-client3

// terminal 3
$ docker stop consul-client2
// terminal 2
// check web status in consul-client3

// terminal 3
$ docker logs consul-server
Consul Graceful shutdown

// terminal 3
$ docker start consul-client2

// terminal 1
$ docker-compose down

VM1
Consul
Client1
8080
Web1
Server
(Follower)
Server
(Leader)
Server
(Follower)
VM2
Consul
Client2
Web2
VM3
Consul
Client3
Service
K
Sidecar Pattern in Consul

● Is Consul Stable?
● Monitoring Consul Cluster
● How to discover the service discovery system?
● How to push in the company?
● Production checklist
CH03 Go Live

● How to discover the service discovery service?
● How legacy service use it?
CH03 Go Live

Server
(Follower)
Server
(Leader)
Server
(Follower)
GOSSIP
Replication
Leader
Forwarding
Replication
RPC
Leader
Forwarding
GOSSIP
RPCGOSSIP
Consul Cluster Availability

Server
(Leader)
Server
(Follower)
GOSSIP
Replication
RPC
Leader
Forwarding
GOSSIP
RPCGOSSIP
Consul Cluster Availability, leader election

Consul Cluster Availability: Multiple AZ deployment

Performance tuning depending on your requirements
Ref: https://www.consul.io/docs/install/performance.html

VM1
Consul
Client1
8080
Web1
Server
(Follower)
Server
(Leader)
Server
(Follower)
VM2
Consul
Client2
Web2
VM3
Consul
Client3
Service
K
Developing a SDK for caching configuartion, prevent from calling Consul every time
SDK SDK SDK
Keep long pulling connection Keep long pulling connection Keep long pulling connection

HashiCorp Vault Workshop：幫 Credentials 找個窩
Ref: https://www.slideshare.net/smalltown20110306/hashicorp-vault-workshop-credentials

● CloudWatch Agent + StatsD
● Do not need to maintain log service
Monitoring Consul Cluster

VM1 VM2 VM3
Shipping Consul metrics to CloudWatch
8080
Server
(Follower)
Server
(Leader)
Server
(Follower)
AWS
CloudWatch
Agent
Statd port: 9125
AWS
CloudWatch
Agent
Statd port: 9125
AWS
CloudWatch
Agent
Statd port: 9125

// terminal 1
$ cd ../lab06
$ sudo dpkg -i -E amazon-cloudwatch-agent.deb
$ sudo cp amazon-cloudwatch-agent.json
/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
$ sudo
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m
ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s
$ consul agent --config-dir ./conf.d

● Monoriting Consul Cluster
● How legacy service use it?
CH03 Go Live

Ref: https://www.slideshare.net/AmazonWebServices/leveraging-elastic-webscale-computing-with-aws

Put bootstring configuration on AWS EC2 tags

Environment 1
AM
CI Server
0. Build Code
CMService B
Environment 2
CMService B
Configuration
(Git)

Environment 1
AM
CI Server
CMService B
Environment 2
CMService B
Configuration
(Git)
1. Get Instance Metadata
Version, AM URL, CM URL

Environment 1
AM
CI Server
CMService B
Environment 2
CMService B
Configuration
(Git)
2. Get Artifact
2. Get Artifact

Environment 1
AM
CI Server
Environment 2
Configuration
(Git)

Environment 1
AM
CI Server
0. Build Code
Environment 2
Configuration
(Git)
2. Get Artifact
2. Get Artifact

● Run Consul Client (Join Consul Cluster)
● Get Service Name, Service Version, Artifacts Url, Market and Environment
● Get Artifacts
● Get Confugration
● Run Service
Service Provisioning

● How legacy service use Consul?
CH03 Go Live

EC2
Docker Network
Consul
Server
Consul
Client1
eth0
8080
8080
Web1
Consul
Client2
Web2 Proxy
8081
8081
Consul
Template

Consul Template
This project provides a convenient way to populate values from Consul into the ﬁle
system using the consul-template daemon.
https://github.com/hashicorp/consul-template

// check all terminals change dir to lab07
$ cd ../lab07
// in terminal 1
// in termianl 2
$ curl localhost:8081

// in termianl 2
// launch consul template, it will regenerator nginx proxy, and you can
access web1, web2 now
$ consul-template -template
"./nginx-conﬁg-template/upstream.tpl:./nginx-conﬁg/upstream.conf:docker restart proxy"
// in termianl 3
$ docker stop web2

● Monoriting Consul Cluster
CH03 Go Live

CH03 Go Live: Production checklist
● Networking
○ Port. Like: DNS Server, HTTP API, Serf, Gossip
○ DNS Configuration
■ https://learn.hashicorp.com/consul/security-networking/forwarding
■ https://learn.hashicorp.com/consul/security-networking/dns-caching
● Consul Servers Deployment
○ Consul Binary
○ Configuration
○ Telemerty configured
● Consul Clients Deployment
○ Sidecar or not?
○ External Service Monitor has been deployed to nodes that can not run a Consul client

● Security
○ Encription of Communication
○ Enable ACLs
○ Setup a Certiﬁcate Authority
● Failure Recovery
CH03 Go Live: Production checklist

CH01 Why Conﬁguration Management?
● Morden app are smaller, compostable & portable
● More ﬁxable service management
● Single code base multiple deployment
● External services are unstable
Summary

CH02 Play With Consul
● KV
○ Key value store
○ Used to hold dynamic conﬁguration
○ Static
○ Dynamic
● Health check
○ Consul helps you to check internal/external services
○ Push / Pull mode
● Building Consul cluster
Summary

CH03 Go Live
Summary

https://www.104.com.tw/company/1a2x6bio5g

● Remember to delete your Cloud9 instance & Admin IAM Role
● CloudWatch Log will delete automatically after 2 weeks
Clean Up

● 91APP
○ Andrew Wu
○ Rick Hwang
○ Earou Huang
○ Infra & Common Service Team Members
● DevOps Taiwan & Taipei HashiCorp User Group
○ Cheng Wei Chen
○ Smalltown
○ Rico Chen
● AWS
○ Carol Chen
● eCloudvalley Technology
Thank you sooooooooooooooo much

20190727 HashiCorp Consul Workshop: 管管你們家 config 啦

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a 20190727 HashiCorp Consul Workshop: 管管你們家 config 啦

Semelhante a 20190727 HashiCorp Consul Workshop: 管管你們家 config 啦 (20)

Último

Último (20)

20190727 HashiCorp Consul Workshop: 管管你們家 config 啦