Kubernetes - Shifting the mindset from servers to containers - microxchg 2018 - Schlomo Schapiro

1. Kubernetes: Shifting the mindset from servers to containers DB Systel GmbH | Schlomo Schapiro | Chief Architect Cloud, Chief Technology Office | @schlomoschapiro | 23.03.2018 This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License DB13243 © Deutsche Bahn AG / Volker Emersleben

2. Did you ever use this in a Docker image? ssh cron supervisord daemontools upstart systemd runit runas su run.sh DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20182

3. A “typical“ server ... 3 DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.2018 SSH CRON logrotate Backup Postfix Rsyslogatd dbus-daemon Apache PHP App PHP App MySQL DB DB man-db dpkg

4. A “typical“ server is 50% cruft ... which should be centralized 4 DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.2018 SSH CRON logrotate Backup Postfix Rsyslogatd dbus-daemon Apache PHP App PHP App MySQL DB DB man-db dpkg cruft This is the „real“ server

5. 5 DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.2018 SSH CRON logrotate Backup Postfix Rsyslogatd dbus-daemon Apache PHP App PHP App MySQL DB DB man-db dpkg

6. DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20186 SSH CRON logrotate Backup Postfix Rsyslogdbus-daemon atd Apache PHP App PHP App MySQL DB DB man-db dpkg Cluster-wide orchestration, scaling, monitoring and deployment of processes Great declarative description for all IaaS needs is an abstraction layer

7. DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20187 SSH CRON logrotate Backup Postfix Rsyslogatd Apache PHP App PHP App MySQL DB DB = containers

8. Platform features DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20188 SSH CRON logrotate Backup Postfix Rsyslogatd Apache PHP App PHP App MySQL DB DB containers isolation – packaging – deployment – immutable systems Application containers

9. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.20189 Build Deploy & Configure Initialize & Run Maintain as Linux Processes

10. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201810 Build on Kubernetes Docker Build

11. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201811 Deploy & Configure on Kubernetes Config Maps Secrets Pod Spec

12. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201812 on a traditional server /etc/init.d/app /usr/sbin/app /etc/init.d/cron /usr/sbin/cron /etc/cron.daily/app Initialize & Run Maintain Exclusive access Prepare data files Restore data Apply schema upgrade Backup data Cleanup stale data Run application

13. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201813 on Initialize & Run Maintain Backup data Cleanup stale data Init Container Main Container Maintenance Container Run application Exclusive access Prepare data files Restore data Apply schema upgrade

14. Application Life Cycle DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201814 on Initialize & Run Maintain Init Container Main Container Maintenance ContainerPod

15. Running a Pod with multiple Containers DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201815 init main maintenance t DATA S3 BACKUP Backup, clean up stale data ... Restore if needed

16. Running a Pod with multiple Containers DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201816 init main maintenance t DATA S3 BACKUP Backup, clean up stale data ... Restore if needed Exclusive access Prepare data files Restore data Apply schema upgrade Run application Backup data Cleanup stale data Run applicationCoordination!

17. What happens with the „cruft“? 17 DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.2018 SSH CRON logrotate Postfix Rsyslogatd dbus-daemon man-db dpkg

18. CRON • Sends out emails • Forks multiple processes , breaking the one task per container paradigm • Not optimized for running single task • Doesn‘t correctly handle INT/KILL signals • Doesn‘t log to STDOUT / STDERR • Cannot configure schedule and cron jobs via environment variables ... made for servers and not containers DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201818 CRON

19. CRON for a single job DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201819 #!/bin/bash RUNAT=${RUNAT:-1 minute} function wait_for_maintenance_time { sleep_time=$(( $(date -d "$RUNAT" +%s ; echo - ; date +%s) )) if (( sleep_time < 0 )) ; then sleep_time=$(( 24*60*60 + sleep_time )) # wait till next day same time fi if (( sleep_time > 0 )) ; then echo "Waiting $sleep_time seconds till $RUNAT before starting maintenance" sleep $sleep_time else echo "Not waiting $sleep_time seconds" fi } while true ; do wait_for_maintenance_time # do some maintenance, e.g. backup data or purge old stuff done https://goo.gl/EqSBJU

20. Email Old server interfaces • /usr/lib/sendmail • /usr/bin/mail • SMTP to 127.0.0.1:25 • trust based on „same host“ • implicit configuration by convention DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201820 Kubernetes alternatives • cluster service for SMTP: smtp.mynamespace.svc.cluster.local. • trust based on „same cluster“ or dedicated authentication • configure via environment variables, e.g. MAILHOST

21. Secure Shell One tool – many purposes • SSH for admin access • SSH for automation between servers • SSH for pull backup SSH on Servers • Admins are (local) users • Technical users for automation • Authentication with passwords or static SSH keys • ~/.ssh/authorized_keys as command filter for some SSH keys DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201821 Kubernetes alternatives • Kubernetes provides Admin access: kubectl exec • authentication with Kubernetes temporary credentials Anti-patterns • SSH between pods: Application cluster probably not aware of pods coming and going • User authentication in pods: Pointless as pods run non-privileged and SSH deamon cannot switch user

22. Logs Typical logging interfaces • Syslog: /dev/log • Syslog: UDP 127.0.0.1:514 • Write to log file: • /var/log/messages • /var/log/auth.log • /data/myapp/some.log • ... DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201822 Kubernetes alternatives • /dev/stdout is primary logging interface for applications and containers • Kubernetes handles logging Anti-patterns • Custom log file: You‘ll need an extra sidecar container to read this log • Syslog server: Set up sidecar container to listen on UDP:514 and write to STDOUT

23. Live Demo https://commons.wikimedia.org/wiki/File:MacBook_Pro,_Late-2008.jpg

24. Demo: WebDAV server with user-provided data and backup to GitHub DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201824 WebDAV Server „main“ Container /mediaRead / Write Data Restore Backup Configure git Repo „init“ Container Create Backup Upload to GitHub „backup“ Container Demo only, git is no backup tool: http://blog.codekills.net/2009/12/08/using-git-for-backup-is-asking-for-pain/

25. Containers DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201825 kind: Deployment spec: template: spec: initContainers: - name: init ... more container spec containers: - name: main ... more container spec - name: backup ... more container spec volumes: - name: media emptyDir: {}

26. initContainers: - name: init image: schlomo/ssh-url-with-ssh-key volumeMounts: - mountPath: /media name: media command: - /bin/bash - -exc - | test -d /media/.git && exit 0 ssh-keyscan github.com >/etc/ssh/ssh_known_hosts 2>/dev/null git clone --depth 1 git~LS0t...tLQo=@github.com:schlomo/demo-data.git /media chmod 700 /media/.git chown -R 33:33 /media chown -R 0:0 /media/.git cd /media git config user.email "demo$RANDOM$RANDOM@nowhere$RANDOM$RANDOM.com" git config user.name "Demo $RANDOM" init DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201826 Hack: https://goo.gl/gqjfuy

27. containers: - name: main image: sashgorokhov/webdav ports: - containerPort: 80 protocol: TCP volumeMounts: - mountPath: /media name: media main DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201827

28. containers: - name: backup image: schlomo/ssh-url-with-ssh-key volumeMounts: - mountPath: /media name: media command: - /bin/bash - -exc - | cd /media ssh-keyscan github.com >/etc/ssh/ssh_known_hosts 2>/dev/null while true ; do sleep 15 git add -A && git commit -a -m "$(date)" && git push done backup DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 23.03.201828

29. https://commons.wikimedia.org/wiki/File:MacBook_Pro,_Late-2008.jpg Live Demo

30. Pod with multiple containers Recap: Multiple Containers in One Pod App Service SSH Service Cron Service /usr/sbin/appd /etc/init.d/app /usr/sbin/crond /etc/init.d/crond /usr/sbin/sshd /etc/init.d/ssh Prepare data files Restore data Apply schema upgrade Backup data Clean up stale data Computer Linux Processes init main maintenance

31. @schlomoschapiro DB Systel GmbH | Schlomo Schapiro | @schlomoschapiro | 22.03.201831 Throw away the old ideas, use the Kubernetes way! Blog, Slides & Code: goo.gl/EqSBJU Feedback: go.schapiro.org/feedback