Mais conteúdo relacionado Semelhante a Configuring Highly Scalable Compile Masters, Vasco Cardoso, AWS (20) Configuring Highly Scalable Compile Masters, Vasco Cardoso, AWS1. © 2020, Amazon Web Services, Inc. or its Affiliates.
Vasco Cardoso,
Solutions Architect, AWS
05.05.20
Configuring Highly Scalable
Compile Masters with AWS
OpsWorks for Puppet Enterprise
2. © 2020, Amazon Web Services, Inc. or its Affiliates.
Puppet Scaling
Vertical Scaling
CPU
Memory
Max 4,000 nodes
Horizontal Scaling
Add Compile Masters
1.5k – 3k nodes / each
3. © 2020, Amazon Web Services, Inc. or its Affiliates.
Install Compile Masters
Infrastructure
VM to run CM
Load Balancer
Configuration
New Node needs to be classified as a Compile Master
Communication needs to be routed correctly
4. © 2020, Amazon Web Services, Inc. or its Affiliates.
Step by Step
• Create Load Balancer
• Configure load balancer to allow communication on
ports 8140 and 8142
• Create new machine to serve as compile master
• Configure load balancer to forward traffic to new
compile master
• SSH into future Compile master node and install the
agent using the dns_alt_name flag
• SSH into Master of Masters to sign the certificate
• Log into the console and pin the compile master node
to the PE Master node group
• Run puppet on the Compile Master
• Run puppet on Master of Masters
• Configure the agent install requests to be sent to the
load balancer
• Run puppet on the compile masters
• Run puppet on Master of Masters
• Configure infrastructure agents to connect the
orchestration agents to the Master of Masters
• Run puppet on all nodes under the PE Infrastructure
Group
• Configure agents to connect orchestration agents to
the load balancer
• Run puppet on the Master of Masters
• Run puppet on all the nodes, or re-install the agent
Compile Master Infrastructure Master of Masters Configuration
5. © 2020, Amazon Web Services, Inc. or its Affiliates.
Automation – Puppet Master
AWS OpsWorks for Puppet Enterprise
Fully managed Puppet Enterprise server
Automatically:
• Patched
• Updated
• Backed Up
6. © 2020, Amazon Web Services, Inc. or its Affiliates.
Automation - Infrastructure
• Load Balancer
• Elastic Load Balancing– Network Load Balancer
• Compile Master
• Amazon EC2 Instance
• Build it all
• AWS CloudFormation
7. © 2020, Amazon Web Services, Inc. or its Affiliates.
Step by Step – Now with (some) automation
• Create Load Balancer
• Configure load balancer to allow communication on
ports 8140 and 8142
• Create new machine to serve as compile master
• Configure load balancer to forward traffic to new
compile master
• SSH into future Compile master node and install the
agent using the dns_alt_name flag
• SSH into MoM to sign the certificate
• Log into the console and pin the compile master node
to the PE Master node group
• Run puppet on the compile master
• Run puppet on MoM
• Configure the agent install requests to be sent to the
load balancer
• Run puppet on the compile masters
• Run puppet on MoM
• Configure infrastructure agents to connect the
orchestration agents to the MoM
• Run puppet on all nodes under the PE Infrastructure
Group
• Configure agents to connect orchestration agents to
the load balancer
• Run puppet on the master
• Run puppet on all the nodes, or re-install the agent
Compile Master Infrastructure Master of Masters Configuration
8. © 2020, Amazon Web Services, Inc. or its Affiliates.
Automation - Configuration
Master of Masters Configuration
Node classifier service API
Requires sharing of credentials
Username/Password
SSH Keys
AWS Systems Manager
Run commands without SSH
9. © 2020, Amazon Web Services, Inc. or its Affiliates.
Automation – Step by Step
• Create Load Balancer
• Configure load balancer to allow communication on
ports 8140 and 8142
• Create new machine to serve as compile master
• Configure load balancer to forward traffic to new
compile master
• SSH into future Compile master node and install the
agent using the dns_alt_name flag
• SSH into MoM to sign the certificate
• Log into the console and pin the compile master node
to the PE Master node group
• Run puppet on the compile master
• Run puppet on MoM
• Configure the agent install requests to be sent to the
load balancer
• Run puppet on the compile masters
• Run puppet on MoM
• Configure infrastructure agents to connect the
orchestration agents to the MoM
• Run puppet on all nodes under the PE Infrastructure
Group
• Configure agents to connect orchestration agents to
the load balancer
• Run puppet on the master
• Run puppet on all the nodes, or re-install the agent
Compile Master Infrastructure Master of Masters Configuration
• Launch CloudFormation Stack
10. © 2020, Amazon Web Services, Inc. or its Affiliates.
Automation Dive Deep – ELB
• Dynamic registration of targets (Compile Masters)
• Only allows traffic ports used by the Puppet and the Orchestration Agents
• Health Checks
• Managed service
11. © 2020, Amazon Web Services, Inc. or its Affiliates.
Automation Deep Dive – Compile Master
• Bootstrap and update configuration at launch
• Puppet Agent installation includes dns_alt_names flag
• Certificate signed automatically
• Compile Master automatically pinned to the PE Master group
12. © 2020, Amazon Web Services, Inc. or its Affiliates.
Automation Deep Dive – Autoscaling Group
• Scales the number of Compile Masters in and out
• Based on metrics
• Via a dedicated trigger
• Replaces unhealthy Compile Masters automatically
• Automatic registration with Load Balancer
13. © 2020, Amazon Web Services, Inc. or its Affiliates.
Automation Deep Dive – Master of Masters
• Configure compile master pool address (load balancer address)
• Validate change
• Run Puppet
• Configure PCP Broker for Infrastructure Agents
• Validate change
• Run Puppet
• Configure PCP Broker for Agents
• Validate change
• Run Puppet
14. © 2020, Amazon Web Services, Inc. or its Affiliates.
Benefits
• Move away from manual processes
• Built-in error detection and handling
• Unattended scaling and replacement of Compile Masters
• Replicate across environments