Docker makes everything easier. But even with the easiest platforms, sometimes you run into problems. In this session, you'll learn first-hand from someone whose job is helping customers fix these problems. In this session, you'll learn how to: Ask better questions & identify the real problems you want to solve Use command line tools to inspect internal Swarm and Kubernetes components Debug and troubleshoot interactions between Enterprise Edition components Troubleshoot Docker Enterprise Edition Windows nodes Apply these skills to troubleshooting Docker Enterprise Edition

1. Ada Mancini
Support Engineer, Docker
Troubleshooting Tips
Eiichi Kitagawa
Support Engineer, Docker
from Docker Support Engineers

3. Docker EE Platform
Universal Control Plane
App Scheduler
Swarm KubernetesOR
Docker EE Cluster
Node Node Node

4. v
for docker enterprise edition
problem-solving methodology

5. how to generalize a problem solving strategy?
● evidence-based
● algorithmic (not technically)
problem-solving methodology

6. Test & AnalyzeCharacterize Communicate

7. what is the problem?

8. what is the problem?
somewhere, somehow,
something changed

9. Test & AnalyzeCharacterize
➔Ask measurable questions
Communicate

10. Test & AnalyzeCharacterize
➔Ask measurable questions
➔Know the normal
Communicate

11. Test & AnalyzeCharacterize
➔Ask measurable questions
➔Know the normal
➔Gather real data
Communicate

12. Test & AnalyzeCharacterize
➔Ask measurable questions
➔Know the normal
➔Gather real data
➔Formulate a theory to explain
observations
Communicate

13. Test & AnalyzeCharacterize
➔Ask measurable questions
➔Know the normal
➔Gather real data
➔Formulate a theory to explain
observations
➔Trust, but verify
Communicate

14. everybody lies

15. Test & Analyze
➔Make measurable changes
Characterize
➔Ask measurable questions
➔Know the normal
➔Gather real data
➔Formulate a theory to explain
observations
➔Trust, but verify
Communicate

16. Test & Analyze
➔Make measurable changes
➔Repeatable
Characterize
➔Ask measurable questions
➔Know the normal
➔Gather real data
➔Formulate a theory to explain
observations
➔Trust, but verify
Communicate

17. Test & Analyze
➔Make measurable changes
➔Repeatable
➔Use existing models
Characterize
➔Ask measurable questions
➔Know the normal
➔Gather real data
➔Formulate a theory to explain
observations
➔Trust, but verify
Communicate

18. viewing software as a stack

19. as an onion

21. combine
effective
models

22. Test & Analyze
➔Make measurable changes
➔Repeatable
➔Use existing models
Characterize
➔Ask measurable questions
➔Know the normal
➔Gather real data
➔Formulate a theory to explain
observations
➔Trust, but verify
Communicate
➔Start at the start

24. 15:21:06 raiden ╡ I want to access the ssh server on my container
15:21:57 @ada ╡ ensure there's a port published for ssh and the ssh server is
╡ running. but I usually don't recommend running ssh as part of
╡ a container
15:25:28 raiden ╡ but will it allow outbound connection to my host machine?
15:26:11 @ada ╡ what's your end goal?
15:26:44 raiden ╡ x forwarding through ssh
15:29:39 @ada ╡ what are you trying to do? run a GUI program in a container?
15:30:55 ╡ if you are trying to run a GUI program in a container, it's
╡ easier to give the container access to X's unix socket

25. Test & Analyze
➔Make measurable changes
➔Repeatable
➔Use existing models
Characterize
➔Ask measurable questions
➔Know the normal
➔Gather real data
➔Formulate a theory to explain
observations
➔Trust, but verify
Communicate
➔Start at the start
➔Avoid X/Y problems

26. Test & Analyze
➔Make measurable changes
➔Repeatable
➔Use existing models
Characterize
➔Ask measurable questions
➔Know the normal
➔Gather real data
➔Formulate a theory to explain
observations
➔Trust, but verify
Communicate
➔Start at the start
➔Avoid X/Y problems
➔Low barrier to entry

27. v
how we do it in support
practical application

28. Compute
Resources
CommunicationConfiguration/L
ogical

29. kubernetesswarmkit
service alpha
task.1
alpha.1.c6ny...
alpha.exe
task.2
alpha.2.76fby...
alpha.exe
service beta
task.1
beta.1.vh756...
beta.exe
task.2
beta.2.dj83f...
beta.exe
volumevolume
application-overlay
deployment-foo-xvgy2
pod-foo-xy765
foo.exe
otherfoo.exe
pod-foo-vy6cd
foo.exe
otherfoo.exe
deployment-foo-cftv5
pod-foo-cjg87
foo.exe
otherfoo.exe
pod-foo-a185hg
foo.exe
otherfoo.exe
volumevolume
calico-cni
volume
persistentVolumevolume
persistentVolume

30. replicaset-foo-6547 --> deployment-foo-xvgy2
pod-foo-xy765
service Foo
task.1
foo.1.c6ny...
foo.exe
task.2
foo.2.76fby...
foo.exe
foo.exe
otherfoo.exe
pod-foo-vy6cd
foo.exe
otherfoo.exe
units of orchestration

31. service Foo
task.1
foo.1.c6ny...
foo.exe
task.2
foo.2.76fby...
foo.exe
$ docker service ls
ID NAME MODE
REPLICAS
kbnw0igaoiyi Foo replicated 2/2
$ docker service ps Foo
ID NAME NODE CURRENT STATE
wlyg0zgws2in Foo.1 ucp-1 Running 38 minutes ago
umkj9b9f2cud Foo.2 worker-1 Running 21 hours ago
$ docker inspect wlyg0zgws2in
--format "{{.Status.ContainerStatus.ContainerID}}"
1a160f96b9b857987e12909119fc1e2980bdafa0c2aa65b1f53d518336c17bce
$ docker inspect 1a160f | jq ".[].State"
{
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 21480,
"ExitCode": 0,
"Error": "",
"StartedAt": "2018-06-07T19:18:35.968597799Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}

32. deployment/nginx-ingress-controller
default-http-back...
nginx
syslog-ng
default-http-back...
nginx
syslog-ng
$ kubectl get deployment --all-namespaces
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE
AGE
ingress-nginx default-http-backend 1 1 1
1 8d
ingress-nginx nginx-ingress-controller 1 1 1
0 8d
$ kubectl get pods --all-namespaces
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS
RESTARTS AGE
ingress-nginx default-http-backend-84f564ddd4-k4b57 1/1
Running 3 8d
$ kubectl get pods -n ingress-nginx -o json
nginx-ingress-controller-f5d78d679-5vjb6 | jq '.status'
"conditions": [
{
"lastProbeTime": null,
"lastTransitionTime": "2018-06-02T20:32:33Z",
"status": "True",
"type": "Initialized"
},
...

33. advanced command line usage
give me just ID's filter based on container name results in...
filter based on name of image format output with Go template
results in...

34. bash constructs enable advanced list/loop structures
$( ) -- command substitution
<( ) -- IO redirection
xargs -- process lists of arguments
jq -- "speak" JSON
curl -- "speak" HTTP
advanced command line usage

35. advanced command line usage
docker service ls
service Foo
Foo.1
Foo.1.c6ny...
alpha.ex
e
Foo.2
Foo.2.76fby...
alpha.ex
e
service beta
task.1
beta.1.vh756.
..
beta.exe
task.2
beta.2.dj83f...
beta.exe
Drill from service ls → container inspect
docker service ps -q -f desired-state=running Foo
| xargs docker inspect --type=task
--format {{.Status.ContainerStatus.ContainerID}}
| xargs docker container inspect
| jq -c

36. create a global service that will only run once
on each node before exiting
mount the docker socket so our service can
speak to the docker API
pass a command into the shell that
consumes the docker API
iterate over a list of tasks that are exited, and
fork docker cp to exfil the data we're
interested in

37. Network Interface
L1/2
Internet
L3
Transport
L4
Application
L5/6/7
HTTP/S
curl
ARP
Is it plugged in? Link light?
ifconfig, netctl
TCP, UDP
netcat, telnet
IP
nslookup, ping, netstat,
ip route, ip addr
tcpdump
wireshark

39. docker run --network container:C0 netshoot
tcpdump -p 80
docker run --network ov-net netshoot tcpdump -p 80

40. v
Windows Troubleshooting
Containers = Windows ∪ Linux ∪ IBM Z

41. 1. Exercise

42. Windows developer asks "My web server on IIS is down! Help!"
1. Exercise

43. Windows developer asks "My web server on IIS is down! Help!"
1. Exercise
If this were NGINX web server on Linux, how would I approach?

44. Windows developer asks "My web server on IIS is down! Help!"
1. Exercise
If this were NGINX web server on Linux, how would I approach?
(1) Is the service running?

45. Windows developer asks "My web server on IIS is down! Help!"
1. Exercise
If this were NGINX web server on Linux, how would I approach?
(1) Is the service running?
(2) Can I connect?

46. Windows developer asks "My web server on IIS is down! Help!"
1. Exercise
If this were NGINX web server on Linux, how would I approach?
(1) Is the service running?
(3) What does the log say?
(2) Can I connect?

47. Windows developer asks "My web server on IIS is down! Help!"
1. Exercise
If this were NGINX web server on Linux, how would I approach?
(1) Is the service running?
(3) What does the log say?
(2) Can I connect?
(4) Can I run a vanilla image for comparison?

48. (1) Is the service running?
Nginx on Linux:
[docker@linuxmanager01 ~] $ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
55q5rno6f9xi svc-nginx replicated 10/10 petshop/prod-nginx:1.3.3 443
...
[docker@linuxmanager01 ~] $ docker service ps svc-nginx
ID NAME IMAGE NODE DESIRED STATE
ry3jwdfxqox2 svc-nginx.0tj2kojk1nyfuja7ITM3333 petshop/prodnginx:1.3. linuxworker01 Running
...
[docker@linuxmanager01 ~] $ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0t33donatetonoagenda33e6 linuxmanager01 Ready Active Leader
vb5currydvorake5w8vj0uk8o linuxworker01 Ready Active
...

49. (1) Is the service running?
IIS on Windows:
[docker@linuxmanager01 ~] $ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
55q5rno6f9xi svc-iis replicated 5/5 petshop/prod-iis:1.3.3 443
...
[docker@linuxmanager01 ~] $ docker service ps svc-iis
ID NAME IMAGE NODE DESIRED STATE
3ddig76s6sdh svc-iis.vbycaqITM33l4g4vtfih0ukyo petshop/prod-iis:1.3.3 winworker01 Running
...
[docker@linuxmanager01 ~] $ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0t33donatetonoagenda33e6 linuxmanager01 Ready Active Leader
vb5currydvorake5w8vj0uk8o winworker01 Ready Active
...

50. (2) Can I connect?
[docker@linuxmanager01 ~] $ curl -I https://my-nginx.org
curl: (7) Failed connect to my-nginx.org:443; Connection timed out
...
Linux Services:

51. (2) Can I connect?
[docker@linuxmanager01 ~] $ curl -I https://my-nginx.org
curl: (7) Failed connect to my-nginx.org:443; Connection timed out
...
[docker@linuxmanager01 ~] $ ssh -i "mycert.pem" docker@linuxworker01
[docker@linuxworker01 ~] $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d0c49a4a3872 nginx "nginx -g 'daemon ..." 14 seconds ago Up 13 seconds 80/tcp svc-iis.vbycaqITM33l4g4vtfih0ukyo
...
[docker@linuxworker01 ~] $ docker exec -it d0c49a4a3872 sh
#
...
Linux Services:

52. (2) Can I connect?
Windows Services:
[docker@linuxmanager01 ~] $ curl -I https://my-iis.org
curl: (7) Failed connect to my-iis.org:443; Connection timed out
...

53. (2) Can I connect?
Windows Services:
PS C:> Enter-PSSession -Computer winworker01
PS C:> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
64b54598ff06 petshop/prod-iis:1.3.3 "C:ServiceMonitor..." 3 seconds ago Up 1 second 80/tcp vc-iis.vbycaqITM33...
PS C:Userswinworker01 > docker exec -it 64b54598ff06 powershell
Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
PS C:> Invoke-WebRequest -Uri http://127.0.0.1
StatusCode : 200
StatusDescription : OK
Content : <!DOCTYPE html
...
[docker@linuxmanager01 ~] $ curl -I https://my-iis.org
curl: (7) Failed connect to my-iis.org:443; Connection timed out
...

54. (3) What does the log say?
Windows Services:
[docker@worker33 ~] $ docker container run --rm
> --name ucp -v /var/run/docker.sock:/var/run/docker.sock
> --log-driver none
> docker/ucp:3.0.1 support > docker-support.tgz
PS C:Userswinworker01 > docker container run --name windowssupport -v
'C:ProgramDatadockerdaemoncerts:C:ProgramDatadockerdaemoncerts' -v
'C:Windowssystem32winevtlogs:C:eventlogs:ro' docker/ucp-dsinfo-win:3.0.1; docker cp
windowssupport:'C:dsinfo' .; docker rm -f windowssupport
Linux Services:

55. (3) What does the log say?
Windows Services:
[docker@worker33 ~] $ docker container run --rm
> --name ucp -v /var/run/docker.sock:/var/run/docker.sock
> --log-driver none
> docker/ucp:3.0.1 support > docker-support.tgz
PS C:Userswinworker01 > docker container run --name windowssupport -v
'C:ProgramDatadockerdaemoncerts:C:ProgramDatadockerdaemoncerts' -v
'C:Windowssystem32winevtlogs:C:eventlogs:ro' docker/ucp-dsinfo-win:3.0.1; docker cp
windowssupport:'C:dsinfo' .; docker rm -f windowssupport
Linux Services:
Or... just remember:
https://docs.docker.com/ee/get-support/

56. (4) Can I run a vanila image for comparison?
[docker@linuxmanager01 ~] $ docker service create --name nginx-plain nginx
Linux Services:

57. (4) Can I run a vanila image for comparison?
Windows Services:
[docker@linuxmanager01 ~] $ docker service create --name iis-plain microsoft/iis:nanoserver
[docker@linuxmanager01 ~] $ docker service create --name nginx-plain nginx
Linux Services:

58. Windows Cheatsheet for Linux Engineers
https://success.docker.com/article/windows-cheatsheet
2. We have a cheat Sheet

59. Windows Cheatsheet for Linux Engineers
https://success.docker.com/article/windows-cheatsheet
2. We have a cheat Sheet

60. Windows Cheatsheet for Linux Engineers
https://success.docker.com/article/windows-cheatsheet
2. We have a cheat Sheet

61. For few more months, be aware of:
- no IP layer routing mesh yet
- no Kubernettes support yet
- containers hidden behind NAT
Or, try out Windows Server Semi-Annual Channel release.
3. Known parities for Windows
Others are listed here:
https://success.docker.com/article/what-are-known-parities-windows-linux-ibmz

62. 4. Microsoft and Docker partnership
Source: https://blogs.technet.microsoft.com/virtualization/2016/05/05/windows-container-networking/
- Docker is the only container runtime on Windows
- Docker EE engine was developed through collaboration

63. 4. Microsoft and Docker partnership
Source: https://blogs.technet.microsoft.com/virtualization/2016/05/05/windows-container-networking/
- Docker is the only container runtime on Windows
- Docker EE engine was developed through collaboration
- Microsoft support provides container support
- Docker and Microsoft Support work together

64. 5. Who can support you?

65. 5. Who can support you?
Start with Microsoft Support for issues:
- reproducible with `docker run`
- reproducible on Windows nodes only cluster
- related to Windows hosts

66. 5. Who can support you?
Start with Docker Support for issues:
- related to UCP or DTR
Start with Microsoft Support for issues:
- reproducible with `docker run`
- reproducible on Windows nodes only cluster
- related to Windows hosts

67. 5. Who can support you?
Start with Docker Support for issues:
- related to UCP or DTR
Start with Microsoft Support for issues:
- reproducible with `docker run`
- reproducible on Windows nodes only cluster
- related to Windows hosts
How to get help with your Windows container issues
https://success.docker.com/article/where-to-get-help-with-windows

68. 6. Don't be scared!

69. 6. Don't be scared!
"... cause I know nothing about Windows." - most Linux experts.
"... cause I'm no Linux expert." - Windows experts and me.

70. 6. Don't be scared!
"... cause I know nothing about Windows." - most Linux experts.
"... cause I'm no Linux expert." - Windows experts and me.
"Fear is the mind-killer. Fear is the little-death that brings
total obliteration."
- Paul Atreides

71. Any Question?

72. v
@eiichikitagawa
thank you!
@ada_qwerty
#docker/irc.freenode.net