Presentation best practices for optimal configuration of oracle databases on sun hardware

1
Best Practices for
Optimal Configuration
of Oracle Databases on
Sun Hardware
Allan Packer Bob Sneed
Principal Engineer Senior Staff Engineer
Sun Microsystems Sun Microsystems
1
Oracle Openworld 2009, San Francisco, CA
Version 1.1, 10/13/2009
Copyright © 2009, by Sun Microsystems, Inc
All Rights Reserved

Oracle Open World 2009 2
Agenda
• Introduction to running Oracle Databases on Sun
SPARC systems – Allan Packer
• Best Practices for running Oracle Databases on Sun
SPARC systems – Bob Sneed

Running Oracle DB on Sun SPARC
• Database Performance – the Data Stall Problem
• Background to Sun's SPARC Systems
• Oracle on CMT and M-Series Servers
• Optimizing Oracle Performance on SPARC

The Dark Underbelly of DB Workloads:
Data Stalls
• Data stalls dominate execution time
> 46%-64% of execution time wasted on data stalls
> True for both OLTP and DSS database workloads
> Even more execution time wasted on instruction stalls
– Especially for OLTP, where you waste a further 10%-15%
• This problem applies to typical multi-core chips
> Designed with sophisticated out-of-order cores and
aggressive speculation
– E.g. IBM Power, Intel Core Duo
Source: “Database Servers on Chip Multiprocessors: Limitations and Opportunities”
September 2006, Carnegie Mellon University

Everything Changes with Sun's
CoolThreads CMT Systems
• Data stalls drop to 13%
> 76%-80% of execution time spent on useful computation
> True for both OLTP and DSS database workloads
> Instruction stalls are negligible
> END RESULT: 70% higher throughput than traditional
systems
• BUT: not good for unsaturated workloads
> Single-threaded workloads perform 41% worse
• So CMT is a win, provided you keep the CPUs busy
Source: “Database Servers on Chip Multiprocessors: Limitations and Opportunities”
September 2006, Carnegie Mellon University

A Simple Analogy: Cars are Good!
Designed for
speed and for
the experience
Plenty of
horsepower
Gets you there
fastest
Ideal for 1 or 2
passengers
But – not so
great with lots of
passengers

Buses are Good, Too!
Designed for
efficiency
Not flashy, but
gets you there
promptly
Excellent
fuel economy per
passenger
Carries many
more
passengers per
trip than a car
But – not so
great when
lightly loaded

How Many Application Threads
Does it Take to Consume a Server?
• Making effective use of CPU strands/threads requires
an adequate supply of application processes/threads
• The specifics will depend on the CPU implementation
1 2 4 8 16 32 64 128 256
1 100% 50% 25% 13% 6% 3% 2% 1% 0%
2 100% 100% 50% 25% 13% 6% 3% 2% 1%
4 100% 100% 100% 50% 25% 13% 6% 3% 2%
8 100% 100% 100% 100% 50% 25% 13% 6% 3%
16 100% 100% 100% 100% 100% 50% 25% 13% 6%
32 100% 100% 100% 100% 100% 100% 50% 25% 13%
64 100% 100% 100% 100% 100% 100% 100% 50% 25%
128 100% 100% 100% 100% 100% 100% 100% 100% 50%
256 100% 100% 100% 100% 100% 100% 100% 100% 100%
CPU Count (Hardware Strands/Threads)
Active Application
Processes and/or
Software Threads
Maximum Potential
CPU Utilization

CMT Product Evolution

CMT Design – True System on a Chip
• UltraSPARC-T2 – all server functions on one chip
> 8 cores, each with
– 2 Integer pipelines
– 8 hardware strands/threads (4 per pipeline) – total 64
– Solaris and Oracle see 64 CPUs
– Floating Point Unit (FPU)
– Crypto unit, delivering embedded wire-speed cryptography
> Integrated Memory Management Unit (MMU)
> Integrated L1, L2 caches
> Integrated networking
> Integrated PCI-Express

CMT – Multiple Strands Per Core
• 8 hardware strands per core
> Implemented with unique registers per strand
> 2 integer pipelines
> A different strand is scheduled per clock cycle
– Strand skipped if stalled or idle
• Simplified pipeline
> Single strand performance on a dedicated
core is lower than other SPARC cores
> Designed for throughput
• Very efficient, perform well
> Stalled strands efficiently skipped for executable strands
S2
S0
S3
S1
IdleIdleIdleIdle
Idle Idle
Stall
Stall Stall
Stall
Stall
StallRun
Run
Run
Run
Clock Cycles
T1 T2 T3 T4
Strands

CMT Design Points
• Benefits
> High throughput with lots of database connections
> Low power utilization
> Small footprint
• Trade-offs
> Single-user performance less than traditional CPU
architectures
> Single-threaded jobs usually run slower

Oracle Performance Expectations
• Most OLTP environments run very well with CMT
• Understand response time components
> DB CPU usually around 20% of response time
> IO, Network, and application tier makes up the rest
• Bad code is a bottleneck on any server
> CPU-bound bad code bites you sooner on CMT servers
• Serial Batch jobs run slower with CMT
> Check the usual suspects
– Execution Plans, Indexes, Business Logic
> Divide and Conquer
– Increase Concurrency and Parallelism

Running Oracle on CMT
• Single-threaded operations should be tuned
• Parallelism
> The key to achieving optimal throughput
– For DSS and Batch workloads
– For Create Index, RMAN backups, Analyze
> Requires Enterprise Edition
• Concurrency
> The key to achieving optimal throughput
– For OLTP workloads
– For Data Load, Import/Export
• Can often use a combination of both

SPARC Enterprise M-Series Servers
• Mainframe-class reliability
• Highly scalable performance
• Advanced virtualization
• Seamless scalability
• Easy upgradability
• Balanced SMP design
• Price/performance targeted
• Competitive RAS and S/W
• Dense rack mount package
• Highest availability
• Highest absolute performance
• Sophisticated resource control
• Highest scalability
• Entry level enterprise system
• Mainframe-class RAS
• Dense rack mount package
M4000 and M5000
M8000 and M9000
M3000

M-Series Compared to CMT
• What's the Same
> Large CPU counts (potentially)
– Need to increase concurrency and parallelism, as for CMT
– Binary compatibility
• What's Different
> Much better Single-Threaded Performance
> More Scalable
– Much greater capacity for CPU, memory, I/O (storage, network)
> Better RAS
– Reliability, Availability, Serviceability

Oracle on Sun M-Series
• Dynamic Reconfiguration
> Add or remove system boards without Oracle instance
downtime
– Leverages Oracle Dynamic SGA and Solaris DISM
> 5th
generation feature on SPARC Enterprise Servers
• Mixed processor support
> Sun M-series supports mixed speed and mixed
generation CPUs
– Allows customers to implement rolling CPU upgrades without
impacting Oracle instance uptime
– Delivers greater ROI because customers not required to
upgrade all CPUs

Oracle on Sun M-Series
• True memory mirroring
> Supported on SPARC Enterprise M4000-M9000
> Domain services can survive up to 50% loss of memory
without interruption of Oracle services
> Dynamic Reconfiguration can be used to repair memory
boards, again without interruption of Oracle services
• Mixed memory densities with no memory speed
reduction
> Industry practice is to require memory speed to be
reduced to lowest speed when mixing memory capacities
• Current PCIe Technology
> Up to 4GB/sec per card

Oracle on SPARC Servers – Summary
• Sun's CMT Server Advantages
> Low power consumption
> Maximum rack computational density
> Logical Domains (LDoms) powerful and bundled
• Sun's M-Series Server Advantages
> Outstanding scalability
> Enterprise-class RAS features
> Flexible domaining
• Solaris Containers run on both CMT and M-Series
> Secure virtual OS instances, negligible perf overhead
> Recognized by Oracle for licensing purposes

Oracle on Sun Storage
• Sun Storage F5100 Flash Array
> Unprecedented performance and scalability
> Up to 1.92TB capacity
> 1.6m read IOPS, 1.2m write IOPS, 12GB/s throughput
• Sun Flash Accelerator F20 PCIe Card
> Low profile PCIe form factor
> 96GB capacity
> 100k read IOPS, 87k write IOPS, 1.1GB/s throughput
> Windows, Solaris, OpenSolaris, Linux

Oracle Database 11gR2 Flash Cache
• New SGA feature in 11gR2
on Solaris
• Expands DB buffer cache
beyond main memory to flash
> ASM, file system file, raw
device,...
> Best with read-mostly and
read-only workloads
• Ideal with Sun Storage F5100
Flash Array, Sun Flash
Accelerator F20 PCIe Card 0x
1x
2x
3x
4x
5x
Oracle Database 11gR2 Flash Cache
Sun SPARC Enterprise M5000, Sun Storage F5100 Flash Array
No Flash Cache
Flash Cache
Response Time
DB Flash Cache Size
TransactionResponseTimeImprovement
Throughput

• Sun 7000 Unified Storage Arrays
> Hybrid Storage Pool – unique innovation with Flash
acceleration
> Analytics – accessible and comprehensive insight into
Oracle data access patterns and behavior
> Support for Oracle DB, NFS, iSCSI, Oracle Direct NFS,...
> Leverages the end-to-end checksum integrity of ZFS
– For data and metadata

• Sun's Range of Storage Arrays
> Supports an extremely scalable stack
– Running on 1u x86 through high-end SPARC servers
> Rich client protocol support
– Fibre Channel
– Infiniband files or blocks
– 10Gb Ethernet
– NFSv3 and NFSv4
– ZFS with local disks
– Note: tuning required for optimal Oracle performance on ZFS

Oracle on Solaris
• Solaris 10 is the #1 UNIX platform for Oracle
deployments
• Solaris benefits from many years of performance
enhancements designed to benefit Oracle
• Solaris offers outstanding
> Scalability, to hundreds of CPUs
> Observability (e.g. Dtrace)
> Virtualization (Containers/zones, no additional cost)
> File System technology (ZFS – no limits)
> Robustness, Security, Application support,...

Configuring Oracle for SPARC
Servers
• Take maximum possible advantage of
> Parallelism
> Concurrency

Oracle Data Load on Solaris
• Parallel Data Loading option of SQL*Loader
> Allows multiple processes to load data into the same
table
> Create multiple input files
> Create a control file for each input file
> Example of 4 parallel loads
sqlload scott/tiger CONTROL=control1.ctl
DIRECT=TRUE PARALLEL=TRUE

Creating Oracle Indexes on Solaris
• Use the parallel option when creating indexes
• The “unrecoverable” option eliminates redo
• “compute statistics” can be done at the same time
> Default in Oracle11g
• Example
create index CUSTOMER_INDEX1 on CUSTOMER
(CUST_ID, REGION)
parallel(degree 32)
unrecoverable
compute statistics;

Oracle RMAN Backups on Solaris
• Use multiple channels and parallelism to maximize
I/O throughput
> Create multiple backup channels for RMAN
> Tell RMAN how to connect to the channels
> Configure the degree of parallelism
– RMAN> CONFIGURE DEVICE TYPE DISK BACKUP TYPE
TO BACKUPSET PARALLELISM 20;
> Issue the backup command
> Throughput improvements can exceed 15x on CMT
• For an example, refer to
http://blogs.sun.com/glennf/entry/parallelizing_oracle_backup_with_rman

Running OLTP Workloads on Solaris
• Active shadow processes can be scheduled onto
available CPUs by Solaris
• The key: an adequate supply of active users
> As user load increases, response times degrade much
more gracefully on CMT than for traditional systems
> Lightly loaded systems are not optimal for CMT
– Traditional systems and Sun's M-Series deliver faster response
times on lightly loaded systems

Running Batch Workloads on Solaris
• Important candidates for parallelism
> Single-threaded batch jobs perform relatively poorly on
CMT
• Oracle Applications
> Increase the number of Concurrent Managers
> adadmin
– Specify number of workers
– Specify parallelism for maintenance functions

Running DSS Workloads on Solaris
• Parallel Query Option (PQO)
• Example: table scan
> Pre-defining a parallel hint for a table
alter table CUSTOMER parallel (degree 32);
select NAME, ADDRESS, BALANCE_OWING
from CUSTOMER;
> Using a hint to override any pre-defined table parallelism
select /*+ PARALLEL(customer, 32) */
NAME, ADDRESS, BALANCE_OWING
from CUSTOMER;

Running DSS Workloads on Solaris
• Parallel DML
• Examples
> Create as select
alter session enable parallel dml;
create table customer
parallel (degree 32)
as select /*+ parallel (temp_table, 32) */
from temp_table;
alter session disable parallel dml;
> Insert as select
alter session enable parallel dml;
insert /* parallel(customer, 32) */ into customer
select * from temp_table;
alter session disable parallel dml;

For More Information
• Check out the Sun Performance Wiki
> For Sun Performance information generally
– Servers, Storage, and Software
> For Oracle on Sun Performance information in particular
http://wikis.sun.com/display/performance

Agenda
• Introduction to running Oracle Databases on Sun
SPARC systems – Allan Packer
• Best Practices for running Oracle Databases on Sun
SPARC systems – Bob Sneed

Best Practices Categories
• Avoiding known performance bugs
> Some are surprisingly high-impact
> Architecturally-dependent factors are not widely-known
• Copying proven patterns, avoiding known antipatterns
> Filesystem and network choices
> “Big knob” tunables
> Platform- or application-specific settings (some not widely-known)
• In general, being disciplined about performance
> Holistically, including the application code, solution architectures,
operational plans, backup and recovery, business continuance
> Methods, tools, interdisciplinary and interdepartmental alignment,
frameworks, metrics, processes
NOTE: While there is no clear objective line between “Best Practices” and “tuning” - the
distinction is much like how “just do it” relates to “optimization”. Both may be needed to
meet system performance objectives.

Best Practice Topics
• Topics discussed here are the most-frequently-
recurring issues in the global support of Oracle
performance on the Sun Solaris OS
> These are the top 'escalation generators'
> Remedies are often difficult to deploy; configuration
management, logistics, and 'proof points' often get in
the way ...

Oracle Performance Bugs & Patches
• 10.2.0.4 is a Good Place to be, with these fixes
(among others) ...
> Bug #6814520 - Compute efficiency of block checksum code on
SPARC was very poor
> Bug #5752399 - For enabling direct I/O on certain filesytems,
filesystemio_options=setall was broken
• 10.2.0.4 has point patches available for these ...
> Excessive client attach latency & high cost of session initiation
– Bug #8737425 - Excessive close() calls on new session creation
> Scalability/consolidation issue: kstat overhead excessive with
many instances running
– Patch #8531434 – "KSTAT CALLS BY MMNL/CJQ0 INCUR HIGH SYS ...”
> RE: Oracle NUMA features, see Metalink #759565.1
– Patch #8199533 - Disables NUMA features as new default

Filesystem/Feature Selection
• Performance Best Practice: use an unbuffered,
concurrent I/O stack and consistent contiguous on-
disk data placement
> Solutions with these characteristics perform similarly
– RAW
– UFS direct I/O
– QFS with samaio or qwrites
– ASM
– VxFS with ODM, QIO, or direct with 'cio' (VxFS has new 'cio'
option; ODM no longer an extra fee)
– NFS with direct I/O (Of course, performance depends greatly
with NFS server factors and network factors)
> Each solution has its own footnotes
– See: http://blogs.sun.com/bobs

Buffered Filesystem I/O
Where buffered I/O is used, recent performance fixes
are often crucial ...
> With Solaris x64, new VPM cache management is vastly
more-efficient than old segmap mechanism
– CR 6256083 "Need a lightweight file page mapping mechanism
to substitute segmap”; Solaris S10U6 10/08
– ERRATA: Technical Instruction Solution 229118 "Segmap
tuning for file system performance on Solaris[TM], Solaris[TM]
x64"-- QFS 5.0 now does leverage VPM
> With VxFS, new VMODSORT improvements requires
upgrade/patch plus explicit enabling in /etc/system
– See also: SunAlert 201248 - VMODSORT article
– set vxfs:vx_vmodsort=1 * enabler in /etc/system

Control LGWR Priority on Purpose
• Run LGWR FX 60, even when FSS is in use
> Otherwise, TS scheduling will reduce its priority when it
gets busy; exactly opposite what one would want!
• Other background processes may also benefit from
special scheduling, but LGWR is uniquely critical-
path to all transactions
• This is especially important on systems with high
aggregate CPU utilization or systems with slower
single-thread speeds

PGA/UGA Page-Size Controls
• With Automatic Memory Management (AMM; when
pga_aggregate_target is used), 'realfree heap' is
engaged by default, but with default OS page size
> OS overhead from frequent re-mapping of small pages can
become high
• One of these often yields favorable performance
impact
> _realfree_heap_pagesize_hint=4194304 for SPARC (2097152 on
x64) – causes mmap'd realfree heap to use large pages
> _use_realfree_heap=false - causes local heap and malloc() to be
used instead of realfree heap

PGA/UGA Page-Size Controls
• Observability
> Use pmap -sax to observe pages sizes used
> Measuring TLB maintenance overhead depends ...
– On M-Series and older SPARC CPUs, trapstat -T
– On SPARC CMT CPUs and recent x64 CPUs, 'hardware table
walk' decreases cost but obscures observability; counters may
be visible via cpustat
• There's a third rarely-used option to consider ...
> With Oracle Shared Server (MTS), PGA allocated from
SGA will leverage the ISM or DISM performance
advantages of the SGA

RAC: Use Jumbo Frames!
• Jumbo frames delivers multiple benefits
> Reduces GC latency
> Reduces CPU utilization, %sys, and hardware interrupts
> Avoids scaling challenges of IP packet reassembly
> Radically improves scaling of RAC on CMT CPUs
• Implementation notes
> Oracle specifies dedicated interconnects for RAC; jumbo
frames should not be an 'infrastructure project'
> Symantec currently recommend MTU of 1500 with their
LLT product; set with 'lltconfig' or in /etc/llttab
• See also
> http://blogs.sun.com/glennf/entry/jumbo_frames_with_oracle_rac

RAC: Reduce LMS Daemon Count
• Default values are generally too large on SMP
systems, especially CMT and NUMA systems
> Too many LMSD causes interconnect scaling issues,
excessive CPU usage, and GC congestion symptoms
> LMSD work increases as their count increases, so more
is usually not better
> Some of the busiest RAC clusters in the world only use
2-4 LMSD per node
> With consolidation projects, the systemwide total must
not be excessive

RAC: More on LMS Daemons
• From “Oracle RAC Performance Experts Reveal
All” - by Barb Lundhild and Michael Zoll at Oracle:
“Fewer and busier LMS processes may be more
efficient. i.e. monitor their CPU utilization”
> Use prstat -mL to observe
• Best-possible LMSD CPU QoS can be assured by
running them in nointr psrsets
> This is a proven method of optimization, but somewhat
esoteric and cumbersome to administer
> This is often called 'interrupt fencing'

Oracle CPU_COUNT on Solaris
• Oracle automatically detects the number of CPUs in
the system, or current processor set (if one is in
use), and sets CPU_COUNT accordingly
• CPU_COUNT determines
> Minimum SGA size
– Larger counts require bigger SGAs
> The size of various internal Oracle structures
– Can have an impact if you dynamically increase/decrease the
number of CPUs in a domain or processor set
• CPU_COUNT pre-dates modern architectures
> Virtual CPUs and highly-thread CPUs not anticipated
> Defaults often inappropriate on highly-threaded systems
and consolidation platforms

Oracle CPU_COUNT on Solaris
• PLEASE NOTE: Most CPU concurrency with Oracle
derives from the population of client processes, not
the setting of CPU_COUNT
> CPU_COUNT does not constrain CPUs from being used
• CPU_COUNT should be set thoughtfully, depending
on the context ...
> Are CMT or VMT CPUs being used?
> Can PQ jobs take over the system?
> Are multiple instances sharing the same system?
> There is not single right answer, but the defaults are
generally too high!

Concurrency and Scaling
Keep in mind, Oracle's compute concurrency comes
mostly from the client process population ...
> Ideally, large populations are pooled to persistent
connections at the application-server tier
> Alternately, Oracle Shared Server (MTS) can be used to
reduce cost of high-rate client attach/detach
> Parallel Query Option (PQO) - useful in many cases for long-
running queries
– Picking the ideal Degree of Parallelism (DOP) depends on system
architecture, chip architecture, and how much one is sharing
resources with competing workload elements
– With virtualization, analysis of competition can be challenging
> The number of clients that can happily share a CPU depends
on the CPU power and the relative compute content of the
client's transactions

M-Series Considerations
• Consider the Architecture
> PERFORMANCE CONSIDERATIONS FOR DEVELOPERS
UTILIZING SUN SPARC ENTERPRISE® M-SERIES
SERVERS [PDF]
• Multithreading details vary between CPU chips
> http://developers.sun.com/solaris/articles/chip_multi_thread.html
> http://developers.sun.com/solaris/articles/hardware_counters.html
> http://blogs.sun.com/morganic/resource/paralleltermsarticle.html
• Cache-to-cache copies on some CPUs can be
expensive; process migration and data placement
can be important

T-Series (CMT) Considerations
• Mainly, a bottleneck thread will bottleneck sooner
due to the single-thread performance
> The value of controlling priorities and eliminating
inefficiencies is amplified
> Also consider application-critical non-Oracle threads (eg:
"dispatcher", "logger", "reader" ...)
• Out-of-box defaults did not foresee CMT
> Limit cpu_count (Oracle may not startup on high thread-
count systems otherwise)
> Rules of thumb ...
– set cpu_count <= 32 absent a specific requirement
– set cpu_count <= number_of_cores

More T-Series (CMT) Considerations
• Due to the reduced single-thread speed of a CMT
hardware thread ...
• Best Practices are more important on CMT systems!
> Getting it all right pays compound dividends!
All of the above!

Resources
• Cool Tools
> http://cooltools.sunsource.net/
• Blueprints
> http://wikis.sun.com/display/BluePrints/Main
• CoolThreads Experts Q&A Portal
> http://www.sun.com/secure/servers/coolthreads/tnb/contact.jsp
• Performance Wiki
> http://wikis.sun.com/display/performance
• Blogroll
> Glenn Fawcett - http://blogs.sun.com/glennf
> Denis Sheahan - http://blogs.sun.com/deniss
> Tanel Poder - http://blog.tanelpoder.com
> Greg Rahn - http://structureddata.org
> Bob Sneed - http://blogs.sun.com/bobs (paused)
> Allan Packer - http://blogs.sun.com/allanp

Call To Action
• Visit the Sun booth in Moscone South #1101
> See the Sun Storage and Server portfolio in person
> View Sun Oracle solutions that bring Extreme
Innovation to the Enterprise
> Talk to Sun experts and leave with answers
> Get briefed on next-gen Sun Storage and Servers
under NDA
• After the show...
> Visit Sun.COM for Sun BluePrints, White Papers,
Videos, and more!

54
Allan.Packer@Sun.COM
Bob.Sneed@Sun.COM
Best Practices for
Optimal Configuration
of Oracle Databases on
Sun Hardware
53

Presentation best practices for optimal configuration of oracle databases on sun hardware

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Presentation best practices for optimal configuration of oracle databases on sun hardware

Similar to Presentation best practices for optimal configuration of oracle databases on sun hardware (20)

More from xKinAnx

More from xKinAnx (20)

Recently uploaded

Recently uploaded (20)

Presentation best practices for optimal configuration of oracle databases on sun hardware