Performance: Observe and Tune

RED HAT CONFIDENTIAL | pvn@redhat.com1
Paul V. Novarese
Sr. Technical Account
Manager
11 September 2014
Performance: Observe and Tune

What can we do out of the box?

What is “tuned”?
Tuning profile delivery mechanism
Red Hat ships tuned profiles that improve performance
for many workloads...hopefully yours!

Tuned: Storage Performance Boost

tuned Profile Summary: RHEL6

Tuned: Updates for RHEL7

Installed by default!

Profiles automatically set based on install type:

Desktop/Workstation: balanced

Server/HPC: throughput-performance

Single tuned.conf file

Optional hook/callout capability

Inheritance (cf. httpd.conf)

Profiles updated for RHEL 7 features (obv)

tuned Throughput Profiles: RHEL 7

tuned Latency Profiles: RHEL 7

tuned Virt Profiles: RHEL 7

Let's get our hands dirty...
Tuning Strategies:
Bang for your Buck

Problem Statements

Bad
– It's slow
– Make it go faster

Better
– We expect 37 gigaflups/year but we only see 24
– We have a bottleneck to a particular LUN in the SAN

Turn bad statements into good statements

Determine victory conditions

Get data

Look at data

Tweak

GOTO 10

Questions to ask

What's actually slow?

How do we know it's slow?

What is the expectation and what is that based on?

What is actually needed to “win”?

What changed?

How long has it been slow?

Gradual or sudden change?

Are there patterns? (same time every day?)

Can you do something to (temporarily) recover?

What evidence do you have? (sar, iostat, etc?)

Identify bottlenecks

CPU

Memory

IO

Network

Application

Firmware

Basic IO Tuning Strategy
Multiple HBAs
– Install (eg) device-mapper-multipath
– Default settings in /usr/share/doc/device-mapper-multipath-
0.4.7/multipath.conf.defaults
Understand storage features / limitations
– Maximum random and sequential read and writes per port
– Maximum random and sequential read and writes for the controller
Low level I/O numbers
– Tools to use dd , aiod , aio-stress, IOzone
– Run I/O representative of the database implementation
I/O Schedulers
– CFQ, Deadline, AS, Noop

IO Schedulers
4 tunable I/O Schedulers
– CFQ – elevator=cfq. Completely Fair Queuing default, balanced,
fair for multiple luns, adaptors, smp servers
– NOOP – elevator=noop. No-operation (uses FIFO) in kernel, simple,
low cpu overhead, leave opt to ramdisk, raid cntrl etc.
– Deadline – elevator=deadline. Optimize for run-time-like behavior,
low latency per IO, balance issues with large IO luns/controllers.
Batches IO ops to produce predictable latencies.
– Anticipatory – elevator=as. Inserts delays to help stack aggregate
IO, best on system w/ limited physical IO – SATA
Changing I/O Schedulers
– echo deadline > /sys/block/<sdx>/queue/scheduler
– Append 'elevator=' to end of kernel line

Basic CPU Tuning Strategy
Limit CPU access
– One or more processes can consume all cpu cycles
– Completely Fair Scheduler (CFS) in RHEL6 uses scheduler groups to assign
different weights to each group
– Configure cgroups and set cpu.shares for each group
Manually balance interrupts
– cat /proc/interrupts to see how interrupts are distributed to each cpu
– Edit /etc/sysconfig/irqbalance and set IRQBALANCE_BANNED_CPUS=
– As an alternative, echo 1 > /proc/irq/142/smp_affinity
Pin processes to a specific CPU
– Taskset (non-NUMA)
– Numactl
– Cgroups
Utilize real-time scheduling (nice, MRG)

Basic VM Tuning Strategy
Huge Pages
– 2MB huge page size
– Set value in /etc/sysctl.conf (vm.nr_hugepages)
– Benefits - https://access.redhat.com/knowledge/solutions/2592
– Enabling - https://access.redhat.com/knowledge/solutions/46326
– Transparent Huge Pages
https://access.redhat.com/knowledge/solutions/46111
NUMA
– Localized memory access for certain workloads improves performance
Swap
– Set value of vm.swappiness (Default 60) – lower number is better for
interactive applications and avoids swapping as much as possible

VM Tuning Frequent Fliers
●
/proc/sys/vm/swappiness
●
Should I swap or drop cache?
●
/proc/sys/vm/min_free_kbytes
●
Be careful adjusting this! Extremes are bad.
●
/proc/sys/vm/dirty_ratio
●
/proc/sys/vm/dirty_background_ratio
●
/proc/sys/vm/vfs_cache_pressure

80/20 Rule

More like 95/5

At some point our time and effort is best spent
elsewhere

What tools can we use?
sar
iostat
perf - Userspace tool to read CPU counters and kernel
tracepoints
Performance Co-Pilot (pcp) – new in RHEL 7

Divider SlideTools

Tradition: start with sar
●
Built-in
●
Collects stats for all four major system components (cpu,
memory, IO, network)
●
Data can be easily graphed
●
Data collection frequency can be easily changed
●
RHEL 6 sar metadata is different than RHEL 5 - you cannot use
RHEL 6 sar to read RHEL 5 sar files.

Collectl
●
More complex, but more powerful
●
Can handle NFS, Slab data, and sub-second intervals
(i.e. -i .25)
●
Very low overhead (<.1% cpu usage) when run in
daemon mode (-D)
●
Send data to another server using socket mode (-A)
●
Infiniband support

/proc
/proc/
– meminfo
– slabinfo
– cpuinfo
– pid<#>/maps
– vmstat(RHEL4 & RHEL5)
– zoneinfo(RHEL5)
– sysrq-trigger

Performance Co-Pilot (pcp)
●
New in RHEL 7, 6.6(b)
●
Available in EPEL for RHEL 5, 6
●
Replaces legacy performance tools
●
Designed for enterprises
●
Distributed
●
Automation
●
Extensible plug-in architecture

pcp Concept

Other Helpful Utilities

Divider Slide
Questions?
pvn@redhat.com

Performance: Observe and Tune

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (12)

Semelhante a Performance: Observe and Tune

Semelhante a Performance: Observe and Tune (20)

Último

Último (20)

Performance: Observe and Tune