Trace File Analyzer - Usage and Features

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Oracle Trace File Analyzer
(TFA)
Diagnostics for the Cloud
18.3.0

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2

Introducing…
Oracle 12c
Autonomous
Health
Framework
3Confidential – Oracle Internal

TFA makes it quicker & easier
to detect & diagnose Database problems
5
Real-time health
monitoring, fault detection
& diagnosis via a single
interface
Secure log redaction &
collection
Continuous log lifecycle
management
Top problem detection &
diagnostics
Why TFA

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 6
Autonomous Usage
Oracle Grid Infrastructure
& Databases
Oracle Support
TFA
1
TFA detects a fault
2Diagnostics
are collected
3
Distributed diagnostics
are consolidated and
packaged
4
Notification of fault is sent
5 Diagnostic collection is
uploaded to Oracle
Support for root cause
analysis & resolution

On-Demand Usage
& Databases
Oracle Support
TFA
5
Upload diagnostic
collection to Oracle
Support
2
Real-time
status
summary
3
Diagnose
with DB
tools
4
Perform
diagnostic
collection
1 Request desired
action on-demand

• Specify all command options at
the command line
1. Set and change context
2. Run commands from within
the shell
8
1. Select menu navigation
options then choose the
command you want to run
Command Interfaces
Command line Shell Menu
tfactl <command>
tfactl
tfaclt > database MyDB
MyDB tfactl > oratop
tfactl menu

mask_strings.xml
• If you want to mask sensitive data
– Hostname
– IP address
– Etc
• Create mask_strings.xml
– In the directory tfa_home/resources
– Define all data replacements you want
9
Data Redaction
<mask_strings>
<mask_string>
<original>WidgetNode1</original>
<replacement>Node1</replacement>
</mask_string>
<mask_string>
<original>192.168.5.1</original>
<replacement>Node1-IP</replacement>
</mask_string>
<mask_string>
<original>WidgetNode2</original>
<replacement>Node2</replacement>
</mask_string>
<mask_string>
<original>192.168.5.2</original>
<replacement>Node2-IP</replacement>
</mask_string>
</mask_strings>

• Enable or disable user access
Configure User Access
Users & groups authorized to access tfactl by default
tfactl access enable
tfactl access disable
tfactl access add –user <user> [-local]
tfactl access remove –user <user> [-local]
tfactl access removeall [-local]
• Add users
• Remove users or everyone
All commands apply
cluster wide unless
-local used to restrict to
local node
• Reset to default access
tfactl access reset
• List user access
tfactl access lsusers
10
TFA
Grid Home Owner DB Home Owners Everyone Else

Set Email Notification Addresses
• TFA can send email notification when faults are
detected
– Configure SMTP with
– Check the current SMTP configuration
– Verify configuration by sending a test email with
• To set notification email for any problem detected:
• To set notification email for specific
ORACLE_HOMEs include the OS owner:
11
tfactl set notificationAddress=john.doe@oracle.com
tfactl set notificationAddress=oracle:another.person@oracle.com
tfactl set smtp
tfactl print smtp
tfactl sendmail {email_address}

On-Demand Usage
& Databases
Oracle Support
TFA
5
Upload diagnostic
Support
2
Real-time
status
summary
3
Diagnose
with DB
tools
4
Perform
diagnostic
collection
1 Request desired
action on-demand

Real-time Status Summary
tfactl summary
Choose an
option to drill
down
High-level summary of all
Database components

Real-time Status Summary – Drill Down
Drill downs show real-time
analytics & details of any
problems found

On-Demand Usage
& Databases
Oracle Support
TFA
5
Upload diagnostic
Support
2
Real-time
status
summary
3
Diagnose
with DB
tools
4
Perform
diagnostic
collection
1 Request desired
action on-demand

• Analyze all important recent log entries: • Search recent log entries:
16
Investigate Logs & Look for Errors
tfactl analyze –last 1d tfactl analyze -search “ora-006" -last 8h
Searching for
“ora-00600”

Perform Analysis Using the Included Tools
Not all tools are included in Grid or Database install.
Download from 1513912.1 to get full collection of tools
Tool Description
orachk or
exachk
Provides health checks for the Oracle stack.
Oracle Trace File Analyzer will install either
• Oracle EXAchk for Engineered Systems, see document 1070954.1 for
more details
or
• Oracle ORAchk for all non-Engineered Systems, see document
1268927.2 for more details
oswatcher Collects and archives OS metrics. These are useful for instance or node
evictions & performance Issues. See document 301137.1 for more details
procwatcher Automates & captures database performance diagnostics and session level
hang information. See document 459694.1 for more details
oratop Provides near real-time database monitoring. See document 1500864.1
for more details.
alertsummary Provides summary of events for one or more database or ASM alert files
from all nodes
ls Lists all files TFA knows about for a given file name pattern across all nodes
pstack Generate process stack for specified processes across all nodes
Tool Description
grep Search alert or trace files with a given database and file name pattern, for
a search string.
summary Provides high level summary of the configuration
vi Opens alert or trace files for viewing a given database and file name
pattern in the vi editor
tail Runs a tail on an alert or trace files for a given database and file name
pattern
param Shows all database and OS parameters that match a specified pattern
dbglevel Sets and unsets multiple CRS trace levels with one command
history Shows the shell history for the tfactl shell
changes Reports changes in the system setup over a given time period. This
includes database parameters, OS parameters and patches applied
calog Reports major events from the Cluster Event log
events Reports warnings and errors seen in the logs
managelogs Shows disk space usage and purges ADR log and trace files
ps Finds processes
triage Summarize oswatcher/exawatcher data
17
Verify which tools you have installed: tfactl toolstatus

Oracle Stack Health Checks
• Oracle EXAchk for Engineered Systems
• Oracle ORAchk for everything else
• Proactively scans for the most impactful problems across the layers of your stack
• Simplifies and streamlines how to investigate and analyze which known issues
present a risk to you
• Lightweight tool that runs within your environment without requiring config data to
be sent to Oracle
• High level reports show your system health risks with the ability to drill down into
specific problems and understand their resolutions
• Can be configured to send email notifications when it detects problems
• Collection Manager, a companion Application Express web app, provides a single
dashboard view of collections across your entire enterprise
ORAchk & EXAchk (Support Tools Bundle)

Automatic Start From Install
• ORAchk will be automatically setup as part of TFA install
– Only on Linux or Solaris
– Only for root install
– Only on non-engineered systems
• Autostart will configure the daemon to restart at 1am every morning to rediscover any environment
changes
• Full Local client run will be triggered at 2am every morning
• Most impactful checks will be run every 6 hours via the oratier1 profile
• Any collections older than 2 weeks will be automatically purged
• Once enabled daemon settings can be changed as per normal
• Remove with: ororachk -autostop tfactl run orachk -autostop

OS Watcher (Support Tools Bundle)
Collect & Archive OS Metrics
• Executes standard UNIX utilities (e.g. vmstat, iostat, ps,
etc) on regular intervals
• Built in Analyzer functionality to summarize, graph and
report upon collected metrics
• Output is Required for node reboot and performance
issues
• Simple to install, extremely lightweight
• Runs on ALL platforms (Except Windows)
• MOS Note: 301137.1 – OS Watcher Users Guide

Procwatcher (Support Tools Bundle)
Monitor & Examine Database Processes
• Single instance & RAC
• Generates session wait, lock and latch reports as well as call stacks
from any problem process(s)
• Ability to collect stack traces of specific processes using Oracle Tools
and OS Debuggers
• Typically reduces SR resolution for performance related issues
• Runs on ALL major UNIX Platforms
• MOS Note: 459694.1 – Procwatcher Install Guide

oratop (Support Tools Bundle)
Near Real-Time Database Monitoring
• Single instance & RAC
• Monitoring current database activities
• Database performance
• Identifying contentions and bottleneck

Analyze
• Each tool can be run using tfactl in shell mode
• Start tfactl shell with
• Run a tool with the tool name
1. Where necessary set context with database <dbname>
2. Then run tool
3. Clear context with database
23
tfactl
tfactl > database MyDB
MyDB tfactl > database
tfactl > orachk
MyDB tfactl > oratop

Automatic Database Log Purge
• TFA can automatically purge database logs
– OFF by default
– Except on a Domain Service Cluster (DSC),
which it is ON by default
• Turn auto purging on or off:
• Will remove logs older than 30 days
– configurable with:
• Purging runs every 60 minutes
– configurable with:
24
tfactl set manageLogsAutoPurge=<ON|OFF>
tfactl set manageLogsAutoPurgePolicyAge=<n><d|h>
tfactl set manageLogsAutoPurgeInterval=<minutes>

Manual Database Log Purge
• TFA can manage ADR log and trace files
– Show disk space usage of individual diagnostic destinations
– Purge these file types based on diagnostic location and or age:
• "ALERT“, "INCIDENT“, "TRACE“, "CDUMP“, "HM“, "UTSCDMP“, "LOG“
tfactl managelogs <options>
Runs as the ADR home
owner. So will only be able
to purge files this owner
has permission to delete
Option Description
–show usage Shows disk space usage per diagnostic directory for both GI and database logs
-show variation –older <n><m|h|d> Use to determine per directory disk space growth.
Shows the disk usage variation for the specified period per directory.
-purge –older <n><m|h|d> Remove all ADR files under the GI_BASE directory, which are older than the time specified
–gi Restrict command to only diagnostic files under the GI_BASE
–database [all | dbname] Restrict command to only diagnostic files under the database directory. Defaults to all,
alternatively specify a database name
-dryrun Use with –purge to estimate how many files will be affected and how much disk space will be
freed by a potential purge command.
May take a while for a
large number of files
25

tfactl managelogs –show usage tfactl managelogs –show variation –older <n><m|h|d>
Use -gi to only
show grid
infrastructure
Use –database to only
show database

tfactl managelogs –purge –older n<m|h|d> -dryrun tfactl managelogs –purge –older n<m|h|d>
Use –dryrun
for a “what if”

Change Trace Levels
• Sometimes Oracle Support may require certain specific trace levels to be
enabled when reproducing a problem
• TFA makes it easy to enable and disable the correct trace levels with a
single command
• All the required trace level settings are organised into problem specific
profiles
• The relevant profile trace logging can be enabled with:
29
tfactl dbglevel –set <profile>

On-Demand Usage
& Databases
Oracle Support
TFA
5
Upload diagnostic
Support
2
Real-time
status
summary
3
Diagnose
with DB
tools
4
Perform
diagnostic
collection
1 Request desired
action on-demand

• For certain types of problems
Oracle Support will ask you to
run a Service Request Data
Collection (SRDC)
• Previously this would have
involved:
• Reading many different
support documents
• Collecting output from
many different tasks
• Gathering lots of different
diagnostics
• Packaging & uploading
• Now just run:
One Command SRDCs
tfactl diagcollect -srdc <srdc_type> -sr <SR#>
32

Collection Uploaded to My Oracle Support
At the end of the collection the file is automatically
uploaded to the SR

With Wallet: Without wallet:
34
Upload any File
tfactl upload -wallet -sr <sr_number> <file1 file2> tfactl upload -user <user_id> -sr <sr_number> <file1 file2>

Full List of SRDCs
Type of Problem SRDC Types
ORA Errors
• ORA-00020
• ORA-00060
• ORA-00600
• ORA-00700
• ORA-01031
• ORA-01555
• ORA-01578
• ORA-01628
• ORA-04030
• ORA-04031
• ORA-07445
• ORA-08102
• ORA-08103
• ORA-27300
• ORA-27301
• ORA-27302
• ORA-29548
• ORA-30036
Database
performance
• dbperf • dbsqlperf
Database resource • dbunixresources
Other internal
database errors
• internalerror
Database patching
• dbpatchinstall
• dbpatchconflict
Transparent Data
Encryption (TDE)
problems
• dbtde
Database Export
• dbexp
• dbexpdp
• dbexpdpapi
• dbexpdpperf
• dbexpdptts
Database Import
• dbimp
• dbimpdp
• dbimpdpperf
RMAN
• dbrman
• dbrman600
• dbrmanperf
System change number • dbscn
GoldenGate
• dbggclassicmode
• dbggintegratedmode
Database install /
upgrade
• dbinstall
• dbupgrade
• dbpreupgrade
Database storage • dbasm
Corrupt block relative
dba
• dbblockcorruption
ASM/DBFS/DNFS/ACFS • dnfs
Partition problems • dbpartition
Slow partitioned
table/index
commands
• dbpartitionperf
SQL performance • dbsqlperf
UNDO corruption • dbundocorruption
Exalogic • esexalogic
Listener errors • listener_services
Naming service errors • naming_services
Database Auditing • dbaudit
Excessive SYSAUX Space • dbawrspace
Database resources • dbunixresources
Database startup /
shutdown
• dbshutdown
• dbstartup
XDB • dbxdb
Data Guard • dbdataguard
Enterprise Manager
tablespace usage
metric
• emtbsmetrics
EM general metrics • emmetricalert
EM debug log
collection
• emdebugon • emdebugoff
EM target discovery
• emcliadd
• emclusdisc
• emdbsys
• emgendisc
• emprocdisc
EM OMS restart • emrestartoms
EM Agent
performance
• emagentperf
EM crash • emomscrash
EM java heap usage
or performance
• emomsheap
EM OMS crash,
restart or
performance
• emomshungcpu
tfactl diagcollect –srdc <srdc_type> -sr <SR#>

Manual Method
1. Generate ADDM reviewing Document 1680075.1
2. Identify “good” and “problem” periods and gather AWR reviewing
Document 1903158.1
3. Generate AWR compare report (awrddrpt.sql) using “good” and
“problem” periods
4. Generate ASH report for “good” and “problem” periods reviewing
Document 1903145.1
5. Collect OSWatcher data reviewing Document 301137.1
6. Collect Hang Analyze output at Level 4
7. Generate SQL Healthcheck for problem SQL id using Document
1366133.1 -
8. Run support provided sql scripts – Log File sync diagnostic output using
Document 1064487.1
9. Check alert.log if there are any errors during the “problem” period
10. Find any trace files generated during the “problem” period
11. Collate and upload all the above files/outputs to SR
Automated One Command TFA SRDC
1. Run
36
Manual vs. “One Command SRDC” Collection - DBPerf
tfactl diagcollect –srdc dbperf [-sr <sr_number>]

Run DBPerf TFA SRDC Collection
37
1. Prompts basic information
2. All required files are identified
3. Trimmed where applicable
4. Package in a zip ready to provide to support

Manual Method
1. Opatch lsinventory
2. IPS – refer to Doc ID 443529.1
3. db information for troubleshooting register problems
4. db information for registered archived logs
5. Extract/Replicat parameter,report and discard files
6. The ggserr.log file
7. information about the table extracted/replicated by
goldengate – download and run scripts
8. information about the schema extracted/replicated by
goldengate
9. generic information used by goldengate
10. Extract/Replicat info
11. Extract/Replicat process environment
12. Goldengate Extract process environment
Automated One Command TFA SRDC
1. Run
38
Manual vs. “One Command SRDC” Collection - GoldenGate
tfactl diagcollect –srdc dbggclassicmode [-sr <sr_number>]

Run Golden Gate TFA SRDC Collection
39
1. Prompts basic information
2. All required files are identified
3. Trimmed where applicable
4. Package in a zip ready to provide to support

1. Enter default for event date/time and database name
2. Scans system to identify recent 10 events in the system (ORA600
example shown)
3. Once the relevant event is chosen, proceeds with diagnostic
collection
40
One Command SRDC
tfactl diagcollect –srdc <srdc_type>
4. All required files are
identified
5. Trimmed where
applicable
6. Package in a zip ready
to provide to support
Interactive Mode

• Run a default diagnostic
collection if there is not
yet an SRDC about your
problem:
• Will trim & collect all
important log files
updated in the past 12
hours or 5 hours around
incident time:
• Collections stored in the
repository directory
• Change diagcollect
timeframe with
–last <n>h|d
41
Default Collection
tfactl diagcollect
$ tfactl diagcollect
By default TFA will collect diagnostics for the last 12 hours. This can result in large collections
For more targeted collections enter the time of the incident, otherwise hit <RETURN> to collect for the last 12 hours
[YYYY-MM-DD HH24:MI:SS,<RETURN>=Collect for last 12 hours] : 2018-06-03 03:32:03
Collecting data for all nodes
Scanning files from jun/02/2018 23:32:03 to jun/03/2018 04:32:03
Collection Id : 20180604135715myhost1253
Detailed Logging at :
/u01/app/oracle/tfa/repository/collection_Mon_Jun_04_13_57_15_EDT_2018_node_all/diagcollect_20180604135715_myhost1253.log
2018/06/04 13:57:20 EDT : NOTE : Any file or directory name containing the string .com will be renamed to replace .com
with dotcom
2018/06/04 13:57:20 EDT : Collection Name : tfa_Mon_Jun_04_13_57_15_EDT_2018.zip
2018/06/04 13:57:20 EDT : Collecting diagnostics from hosts : [myhost1254, myhost1253]
2018/06/04 13:57:20 EDT : Scanning of files for Collection in progress...
2018/06/04 13:57:20 EDT : Collecting additional diagnostic information...
2018/06/04 13:57:30 EDT : Getting list of files satisfying time range [06/02/2018 23:32:03 EDT, 06/03/2018 04:32:03 EDT]
2018/06/04 13:57:40 EDT : Collecting ADR incident files...
2018/06/04 14:00:26 EDT : Completed collection of additional diagnostic information...
2018/06/04 14:00:30 EDT : Completed Local Collection
2018/06/04 14:00:30 EDT : Remote Collection in Progress...
.------------------------------------------.
| Collection Summary |
+---------------+-----------+-------+------+
| Host | Status | Size | Time |
+---------------+-----------+-------+------+
| myhost1254 | Completed | 3.6MB | 197s |
| myhost1253 | Completed | 2.6MB | 190s |
'---------------+-----------+-------+------'
Logs are being collected to: /u01/app/oracle/tfa/repository/collection_Mon_Jun_04_13_57_15_EDT_2018_node_all
/u01/app/oracle/tfa/repository/collection_Mon_Jun_04_13_57_15_EDT_2018_node_all/myhost1254.tfa_Mon_Jun_04_13_57_15_EDT_20
18.zip
/u01/app/oracle/tfa/repository/collection_Mon_Jun_04_13_57_15_EDT_2018_node_all/myhost1253.tfa_Mon_Jun_04_13_57_15_EDT_20
18.zip
$

Time
Custom Collection Options
Option Description
-last <n><h|d> Collect since the previous n hours or
days
-from “<yyyy-mm-dd>” Collect from the date and optionally
time specified.
Valid date / time formats:
"Mon/dd/yyyy hh:mm:ss"
or "yyyy-mm-dd hh:mm:ss"
or "yyyy-mm-ddThh:mm:ss"
or "yyyy-mm-dd "
Option Description
–from "yyyy-mm-dd"
-to "yyyy-mm-dd"
Collect to the date and optionally time
specified.
Valid date / time formats:
"Mon/dd/yyyy hh:mm:ss"
or "yyyy-mm-dd hh:mm:ss"
or "yyyy-mm-ddThh:mm:ss"
or "yyyy-mm-dd"
-for “yyyy-mm-dd” Collect for the specified date.
Valid date formats:
"Mon/dd/yyyy"
or "yyyy-mm-dd"
tfactl diagcollect <option>

Nodes, Databases & Components
Option Description
-node Limit collection to comma separated list of nodes
-database Limit collection to comma separated list of databases
-asm Collect ASM logs
-crsclient Collect Client Logs that are under GIBASE/diag/clients
-dbclient Collect Client Logs that are under DB ORABASE/diag/clients
-dbwlm Collect DBWLM logs
-tns Collect TNS logs
-rhp Collect RHP logs
-procinfo Collect Gathers stack and fd from /proc for all processes
-afd Collect AFD logs
-crs Collect CRS logs
-wls Collect WLS logs
Option Description
-emagent Collect EMAGENT logs
-oms Collect OMS logs
-ocm Collect OCM logs
-emplugins Collect EMPLUGINS logs
-em Collect EM logs
-acfs Collect ACFS logs and Data
-install Collect Oracle Installation related files
-cfgtools Collect CFGTOOLS logs
-os Collect OS files such as /var/log/messages
-ashhtml Collect Generate ASH HTML Report
-ashtext Collect Generate ASH TEXT Report
-awrhtml Collect AWRHTML logs (requires –database option as well)

Misc
Option Description
-collectdir Collects other files in specified comma separated directory list. Only files updated in the relevant time
range (12 hours by default) get collected.
-tag Store the collection in a use a custom directory name within the repository, using the tag value specified
-z Append the value specified to the collection zip name
-nocopy Prevent the copying of zip files back to the initiating node
-notrim Prevent the trimming of files
-silent Start collection as a background process so output is not echoed to the console
-nocores Do not collect core files

REST service
• REST support allows invocation & query over HTTPS
• Oracle REST Data Services (ORDS) is included within the install
• Once ORDS is running REST invocations can be made via:
• Post to Result
• Can be downloaded with:
50
tfactl rest [-status|-start|-stop|-uninstall] [-dir ] [-port ] [-user ] [-debug [-level ]]
https://host:port/ords/api
https://myhost:9090/ords/tfactl/diagcollect {
"collectionId" : "20180111011121slc12ekf",
"zipName" : "TFA_DEF_ZIP_20180111011121",
"tagName" : "TFA_DEF_TAG_20180111011121"
}
https://myhost:9090/ords/tfactl/download/20180111011121slc12ekf
The tfactl rest command can only be run by root user

REST service
51
Option Description
-status Prints the current status
-start Starts Oracle Trace File Analyzer REST Services if not already running
-stop Stops Oracle Trace File Analayzer REST services if running
-uninstall Removes the Oracle Trace File Analyzer REST configuration
-dir
The directory to use to store the Oracle Trace File Analyzer REST configuration details.
Defaults to the users home directory
-port
The port to run ORDS on
Defaults to 9090
-user
The user to start ORDS as
Defaults to the GRID owner
-debug Enables debug
-level
The level of debug to use, where available levels are:
•1 - FATAL
•2 - ERROR
•3 - WARNING
•4 - INFO (Default)
•5 - DEBUG
•6 - TRACE

REST Service Via Tomcat
• TFA includes a WAR file to enable the REST Service via Apache Tomcat
1. Deploy the WAR file located at to your Tomcat server
2. Change the tfaadmin user password
3. Change the tfarest user password
4. Add the user Tomcat runs as to the TFA access list
Confidential – Oracle Internal/Restricted/Highly Restricted 52
TFA_HOME/jlib/tfa.war
curl -k --user tfaadmin:tfaadmin https://host/tfa/tfactl/user/update
{"password" : "some_new_password" }
curl -k --user tfarest:tfarest https://host/tfa/tfactl/user/update
{"password" : "some_new_password" }
tfactl access add -user <tomcat_user>

Maintenance
• Option 1
– Applying standard RUs will automatically
update TFA
– RUs do not contain Support Tools Bundle
updates
• Option 2
– To update with latest TFA & Support Tools
Bundle 1513912.1
53
Does not require downtime for database or grid

Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
54

Trace File Analyzer - Usage and Features

Trace File Analyzer - Usage and Features

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Trace File Analyzer - Usage and Features

Similar to Trace File Analyzer - Usage and Features (20)

More from Sandesh Rao

More from Sandesh Rao (20)

Recently uploaded

Recently uploaded (20)

Trace File Analyzer - Usage and Features