With the support for Windows containers in Docker Swarm and Kubernetes recently hitting beta, we’re entering a world where Hybrid container environments are no longer a strange sight. Microsoft is now an important player in the container world and for this talk I’d like to spend some time explaining what that means for us in the monitoring space. I’ll show you how to install and configure a Hybrid Kubernetes and Docker Swarm environment. We will setup the opensource Prometheus tooling and the commercial CoScale platform to monitor this unique environment. We’ll take a look at infrastructure monitoring and application monitoring, and the differences between Linux and Windows nodes.
2. Samuel
Vandamme
Product specialist at CoScale
● Company focussed on container
& orchestration monitoring
● Spend 80% time helping
customers monitor Kubernetes
and Docker
@kidk
www.sava.be
2
3. What’s a hybrid container environment
Docker cluster environment with multiple
operatings systems
● Linux
● Windows
● ARM (Linux)
● Mac OS X
● Android
● IOS
3
6. Use-cases
Mix and match
● Application on Linux / Enterprise
features on Windows
● IIS with Linux SQL Server
● Modern Dotnet core linked to old .NET
API running on Windows
6
Source https://w3techs.com/
Source securityspace.com
7. Use-cases
Testing
● Building and testing your
application on Linux, Windows,
Mac, Android, ..?
● Browser testing: IE, Edge, Safari,
Firefox, Chrome, ..
7
8. Advantages
● Single orchestration environment
○ Single monitoring view
○ Single cluster maintenance
● Increased security
○ Hyper-V containers
■ VM isolation of containers
● Flexibility
○ Moving to containers allows you to deploy theoretically anywhere
8
9. ● Setting up a Hybrid cluster
● Monitoring a Hybrid cluster
9
11. Docker Swarm / EE support
● One of the first to support Windows containers
○ Built together with Microsoft
● Supported out of the box, no special configuration required
● Not container type aware, so node labels are important
○ "unsupported platform on 1 node"
11
17. Windows server 2016
Initial release of Windows container story (2016)
● Kubernetes 1.5 alpha support
● Windows Docker images
○ Core image 5 GB
○ Nano image 420 MB
17
18. Kubernetes Setup for Windows server 2016
Available in alpha from Kubernetes 1.5
● [alpha] Added support for Windows Server 2016 nodes and scheduling Windows
Server Containers
Don’t try it, unless you have time to waste
18
19. Windows server 1709
Next release of Windows containers (Dec 2017)
● Shared pod compartments
● Endpoint optimization
● Data-path optimization
● Windows Docker images
● Core image 3 GB (-60%)
● Nano image 134 MB (-80%)
19
20. Kubernetes Setup for Windows 1709
Available in beta from Kubernetes 1.9
● Works with Windows Server version 1709
● Networking
○ Host-gateway (static next-hop routes between nodes)
○ Smart ToR switch
○ Third party overlay
■ Flannel
Tutorial:
https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows
20
21. Windows server 1803
Current release of Windows Containers (May 2018)
● Overall improvements to Windows container story
● Improved Kubernetes support
○ Storage plugins
○ New third party network overlay Calico
○ Support for localhost and http proxy
○ Multiple containers per pod
● Boot and run performance improvements
● Windows Docker images
○ Core image 2 GB (-30%)
○ Nano image 137 MB
21
22. Kubernetes Setup for Windows 1803
● Azure Kubernetes Service / Azure Container Service
○ Ask Alessandro
● Docker EE
● Ansible
○ kubespray: No, but soon https://github.com/kubernetes-incubator/kubespray/pull/2978
○ custom: https://github.com/ptylenda/kubernetes-for-windows
● Others
○ kubeadm: https://github.com/kubernetes/kubernetes/pull/53553
22
23. Windows Server 2019
● Further improvements to container story
○ Improving Server core container
○ Container performance improvements
○ Improved support for Flannel and Calico
○ Hybrid cloud
○ …
● Available second half of 2018
○ LTS release
23
24. Kubernetes
● v1.10 - This release continues to enable more existing features on Windows, including
container CPU resources, image filesystem stats, and flexvolumes. It also adds
Windows service control manager support and experimental support for Hyper-V
isolation of single-container pods.
● v1.11 - Supports more of Kubernetes API for pods and containers on Windows, including:
Metrics for Pod, Container, Log filesystem, The run_as_user security contexts, Local
persistent volumes and fstype for Azure disk
● v1.12 - alpha - ?
24
26. Azure
26
az group create --name samuel-hybrid --location westeurope
az acs create --orchestrator-type=kubernetes
--resource-group samuel-hybrid
--name=HybridKub
--agent-count=2
--generate-ssh-keys
--windows --admin-username samuel
--admin-password COscale1234
27. Node constraints
In a Hybrid node environment you will need to put a nodeSelector.
Not needed if you build multi-arch versions of your image
"nodeSelector": {
"beta.kubernetes.io/os": "windows"
}
27
30. What do you need to monitor
30
● Infrastructure / Nodes - Disk, Memory, CPU, Network, ..
● Orchestrator - Health, Actions, ..
○ Network layer
● Containers - Memory, CPU, Network, ..
● Applications within the container
○ Webservices
○ Databases
○ …
Automatic detection, scalable by design, self managing, ..
31. Challenge
● Different OS’s means different ways of monitoring them
○ Different metrics are important
○ Different alerts
○ Different dashboards
● Orchestration view
○ Is it OS aware?
○ How do you handle applications?
● Number of metrics
○ (x containers (started/stopped) * 70 metrics) + (y nodes * 150 metrcs) + orchestrator (100 metrics)
○ largest CoScale customer:
■ 1.2 M containers per month
■ 100 nodes
■ 12M metric values per day
31
32. Windows
● No privileged containers
○ Deploying monitoring in container is not possible
● Resource retrieval is different
● Network layer is different
○ Harder to secure endpoints
32
33. Monitoring tools
Prometheus
● Open source
● Time series database
● Popular in container environments
● Extra’s needed
○ Grafana
○ AlertManager
○ Endpoints
33
CoScale (or other commercial vendors)
● Closed source
● Monitoring platform
● Focus on container environments
34. Prometheus
34
● Pull based
● Lots of components
○ Plug / play
● Prometheus
endpoint standard
becoming default
● Scaling customer
problem
35. Prometheus - endpoints
● Node exporter
○ Linux - https://github.com/prometheus/node_exporter
○ Windows - https://github.com/martinlindhe/wmi_exporter
● CAdvisor
○ Linux - https://github.com/google/cadvisor
○ Windows
■ Work being done to integrate it with Kubernetes
● List of > 100 application monitoring endpoints
○ More and more are being built in
35