Video Here: https://www.youtube.com/watch?v=a-Nk56YB9RQ&feature=youtu.be&t=3m43s

1. Realistic Container Platform Simulations
DevConf 2017
aos-scalability team + adjunct professors

2. DevConf 2017: Realistic Container Platform Simulations
Agenda
● Introduction
● Workload Classification
● Test Utilities
● Demo
● Gold Mining
2

3. DevConf 2017: Realistic Container Platform Simulations
OpenShift Performance and Scalability Team
Who are we?
Performance EngineeringRequestor
Upstream Development
QE Handoff
Test Design/Exec @ Scale
Engineering
Product Management
Marketing
Field/Customer
http://j.mp/ose-perf-scale
3

4. DevConf 2017: Realistic Container Platform Simulations
It’s all about the
workloads.

5. DevConf 2017: Realistic Container Platform Simulations
Some don’t care where they run
● Batch workloads
● Web servers
It’s all about the workloads...
Some care greatly
● Security, Isolation
● Uptime
● Performance
● Proximity/Locality to data

6. DevConf 2017: Realistic Container Platform Simulations6
Listen to your apps

7. DevConf 2017: Realistic Container Platform Simulations7
What is a workload? Subsystems

8. DevConf 2017: Realistic Container Platform Simulations8
What is a workload? Business Requirements

9. DevConf 2017: Realistic Container Platform Simulations9
Attribute Build Farm
CPU Intensive High
Memory Intensive High
Disk I/O Latency Low
Disk I/O Throughput High
Network Latency Low
Network Throughput High
Security Low
Uptime (Live
Migration)
N/A
Deployment Speed High
Alternative OS N/A
Workload →
Infrastructure Mapping:
Build Farm
Icon Meaning
Mature and/or
No Perf
Concerns
Immature and/or
Limited Perf
Concerns
Mixed Concerns
Not Applicable

10. DevConf 2017: Realistic Container Platform Simulations
10
Atomic Host - a
container-optimized, minimal
footprint OS powered by Red Hat
Enterprise Linux
Telemetry - logging and metrics for
pods/containers, services and
underlying infrastructure to make
informed decisions
Runtime and Packaging Format
- standardized container
packaging format and runtime,
powered by Docker (and OCI)
Automation and host configuration
management via Cockpit to
dynamically provision
and configure container host clusters
Orchestration - for complex
multi-container services, powered
by Kubernetes
Networking - scalable, multi-host
container networking, powered by
Open vSwitch, that runs anywhere
Red Hat Enterprise Linux runs
Cluster Services - Scheduling for
services across a container host
cluster, powered by Kubernetes
Storage, with persistent storage
plugins
to enable running of stateful services
in containers
Atomic Registry - integrated
storage and management for
sharing container images
Security to prevent tenants from
compromising other occupants
Container Infrastructure Components

11. DevConf 2017: Realistic Container Platform Simulations
11
10,000 foot overview

12. DevConf 2017: Realistic Container Platform Simulations
Realistic Container Platform Simulations:
Test Harnesses

13. DevConf 2017: Realistic Container Platform Simulations13
System Verification Test Suite (SVT)
● Red Hat OpenShift Performance and Scalability team’s test suites
● https://github.com/openshift/svt
● cluster-loader
● Networking/synthetic
● Workload Generator
● Reliability/Longevity
● https://github.com/kubernetes/perf-tests

14. DevConf 2017: Realistic Container Platform Simulations14
Image Provisioner
Runs on any RHEL instance (currently supported EC2 and qcow2/kvm)
● filesystem juggling to allow for thinpool in base RHEL cloud image
● RHEL OS setup (install latest packages, ssh keys)
● aos-ansible (pull down supporting container images)
● openshift-rpm-install (install openshift RPMs, but do not configure)
● clone necessary git repos
● collectd-install (install and configure collectd)
● docker-config (setup storage)
● repo-install (setup custom yum repos)
● pbench-install (install and configure pbench)
Ansible Automation to build a pre-configured RHEL image

15. DevConf 2017: Realistic Container Platform Simulations
What is cluster-loader?
I’d like an environment with thousands of deployment configs
(which include services and replication controllers), thousands more
routes, pods (each with a persistent storage volume automatically
attached), secrets, image streams, buildconfigs, etc.

16. DevConf 2017: Realistic Container Platform Simulations
16
Start
Parse args &
config
New Config
Obj
End
False
Create
Namespace
True
X
Exists
?
Items < N
False
Create X
Iterate Item
Count
True True
False
X can be:
Quota
Template
Service
User
Pod
RC

17. DevConf 2017: Realistic Container Platform Simulations
Cluster Loader
Demo

18. DevConf 2017: Realistic Container Platform Simulations
Contributing
- 100% golang
- Github: https://github.com/kubernetes/perf-tests
- How to reach us:
- sig-scale mailing list:
https://groups.google.com/forum/#!forum/kubernetes-sig-scale
- sig-scale slack channel: https://kubernetes.slack.com/messages/sig-scale/
- Freenode: #openshift-dev
18

19. Core/Most Urgent Issue Summary

20. DevConf 2017: Realistic Container Platform Simulations
Gold Mining:
Successes

21. DevConf 2017: Realistic Container Platform Simulations
iptables
● iptables CPU consumption BZ1387148
21

22. DevConf 2017: Realistic Container Platform Simulations
HAProxy
● Raise default HAProxy maxconn to 20000: BZ1406327
● HAProxy is forcefully restarted due to not responding to
/healthz probe when under high load: BZ1405440
● ARP tuning documentation
● nbproc
● tcp_fastopen
● busy_poll
22

23. DevConf 2017: Realistic Container Platform Simulations
Logging
● Daemonsets (used by logging) needs rate limiting:
BZ1404860
● Logging DoS's apiserver BZ1384626
23

24. DevConf 2017: Realistic Container Platform Simulations
Metrics
● Metrics stops capturing data @ 12k pods
● BZ1388815
● OpenShift Docs Update PR
24

25. DevConf 2017: Realistic Container Platform Simulations
OCP-on-OSP: etcd
● Team reminded of etcd tight perf req’s
● OCP has poor day-2 operations capabilities for etcd
○ Difficult to replace failed nodes (SSL certs, data migration)
25

26. DevConf 2017: Realistic Container Platform Simulations
Lessons Learned/Next Steps
● Ensure fast disk for etcd (SSD is a must).
● Automate real hostnames and DNS (required for metrics and route tests)
● If using boot-from-volume, ensure you use snapshots when nova booting
● Ceph pools should have correct # of pg/pgp. Use pgcalc.
● Use dedicated builder nodes in OpenShift.
● Incorporate CFME into the environment, deployed as a pod.
26

27. DevConf 2017: Realistic Container Platform Simulations
It’s all about the
workloads.

28. DevConf 2017: Realistic Container Platform Simulations
Thanks!