Collaborate 2019 session. EBS 12.2 ADOP concepts

1. Session ID:
Prepared by:
Remember to complete your evaluation for this session within the app!
10755
Simplifying EBS 12.2
ADOP
1/12/2019
Alfredo Krieg
Senior Cloud Performance Architect
Viscosity North America
@alfredokrieg

3. @ViscosityNAviscosityna.com
Leverage what Makes Viscosity Different
Our Mission:
•We Enable Business Transformation at a Time
when Companies must Change to Survive

4. @ViscosityNAviscosityna.com
Viscosity is a team of experts specializing in
Oracle Cloud and Engineered Systems.
We are lot more than just Oracle
SharePlex Platinum Partner
MongoDB Consulting Partner
Neo4J Consulting Partner
Kyriba - Treasury Management

5. We’ve written 22 books on Data, Cloud, and Oracle…
Industry Experts
o Viscosity founders hold 27+ years each
in the Oracle space
o Authors of 22 books in the Oracle
space
o 5 ACE Directors, only 36 in the United
States. 1 Oracle ACE
o Direct connections to Oracle support
and Product Managers support
o Expertise in Engineered Systems, ZFS,
ODA, Oracle 11g, 12c, 18c, RAC/ASM,
Data Guard, Performance Tuning &
Oracle Licensing
o Hire only best of the best

6. @ViscosityNAviscosityna.com 6
Viscosity Pillars
And Delivery Models
Database
InfrastructureAppsData
Data Integration
IAAS / PAAS
Data Warehousing
Analytics
GoldenGate
SAAS / PAAS
CX ERP SCM HCM
PAAS
Web Applications
Mobility
EBS / JDE / PS
IAAS
Public Cloud
Bare Metal Cloud
AWS Azure Bluemix
Engineered Systems
Exa- ZFS @Customer
X86 / Sparc
Workshops Assessments
Proof of
Concepts
Training Turnkey Projects
Managed
Services

7. @ViscosityNAviscosityna.com
Performance
Health Checks
How’s it running?
Apps
SaaS/PaaS,
Mobility, Application
Development
DBA
Services
Remote and
On-site
On-Call Support
Managed
Services
ZERO DOWNTIME
Migrations
Professional
Services
Where you need it most
Staff Aug
Workforce Capacity on
Demand
Oracle License Management
Get the most out of your Oracle
investment
7

8. @ViscosityNAviscosityna.com 8

9. @ViscosityNAviscosityna.com 9

10. @ViscosityNAviscosityna.com 1

11. @ViscosityNAviscosityna.com 1

12. Alfredo Krieg
• Oracle ACE
• Senior Cloud Performance Architect at Viscosity North America
• Database Performance Tuning, Exadata, OEM and EBS
• Oracle Technologies since 2004
• Blog www.alfredokriegdba.com
• alfredo.krieg@viscosityna.com
@alfredokrieg

13. Disclaimer
• The opinions expressed in this presentation are solely mine, and not that of my
employer.

14. Agenda
• ADOP concepts
– ADOP phases
• Best Practices
• ETCC
• ADOP and DB Patches
• Issues and troubleshooting
• Final Recommendations

15. ADOP – How did I get here?
• Upgrade to 12.2 from 11i, 12.0 or 12.1
• Fresh install of 12.2
Photo by Caleb Jones on Unsplash

16. ADOP Concepts
• ADOP stands for AD Online Patching
• Used to apply patches to EBS 12.2 (adpatch prior releases)
• Capable of storing multiple application editions in the DB (EBR)
• Dual application tier file system (fs1 & fs2 “or” run & patch)
• 4 new environment variables
– $FILE_EDITION
– $RUN_BASE
– $PATCH_BASE
– $NE_BASE

17. ADOP Concepts
• ADOP stages
– Prepare
– Apply
– Finalize
– Cutover
– Cleanup
• Special
– Abort
– Fs_clone

18. ADOP Concepts - Prepare
• Optional Checks
– Before running prepare
• $ adop -validate
– Run from primary (master) node, adop will perform the validation on all the available nodes
– Run on a secondary (slave) node, adop will only run there
– Checks for free space in the system
» SYSTEM – min 25 GB free
» APPS_TS_SEED – min 5 GB free
– Run while a patching cycle is in progress
• Validation will take place for the cutover phase
• Prepare synchronizes patch filesystem from run filesystem

19. ADOP – 100% Online?
• Is it ADOP 100% online?
– No
• How many tasks can be done online?
– It depends
• How much downtime to incur while patching?
– It depends
Photo by bruce mars on Unsplash

20. ADOP Concepts - Prepare
• ADOP actions during prepare
– Checks weather to perform a cleanup
– Validates system configuration
– Checks if DB is ready
• APPS is edition-enabled (exits immediately)
• Patch service (fix attempt)
• Logon trigger (fix attempt)
• DB data dictionary integrity (exist immediately)
• ETCC
– Checks all application nodes
– "Online Patching In Progress" (ADZDPATCH) concurrent program
– Invokes the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
– Checks the database for the existence of a patch edition
– Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl
Photo by Shane Aldendorff on Unsplash

21. ADOP Concepts - Prepare
• Synchronization options
– Just execute adop phase=prepare (DEFAULT)
• Identify patches already applied to run APPL_TOP
• Apply patches silently to patch APPL_TOP
• skipsyncerror parameter
– Create a new patch filesystem adop phase=fs_clone
• Executed before prepare
• Option force=yes
– Delta?
• adop phase=prepare sync_mode=delta
• Delta synchronization of filesystems
• Implemented using rsync on Unix and RoboCopy on Windows
• Customizable

22. ADOP Concepts - Prepare
• Synchronization Customizations
– Patch synchronization - apply missing patches
– Delta (file) synchronization - copy file changes
– fs_clone synchronization - clone entire file system
• Examples of patching actions that are not synchronized by default include:
– Compiling user-defined JSPs
– Copying some third-party libraries
– Copying and compiling user-defined concurrent programs
– Copying and generating user-defined forms

23. ADOP Concepts - Prepare
• $APPL_TOP_NE/ad/custom/adop_sync.drv
#Begin Customization
...
#End Customization
– The adop_sync.drv file is not currently reset to its template file at any point
– You must check that the contents still meet your requirements
• Multi-node
– Execute on primary (Admin Server) node
– ssh between primary and secondary nodes
– If node is inaccessible it will be mark as abandonded

24. ADOP Concepts - Prepare
• Configuration changes
– If any configuration changes detected
• Clone configuration from run to patch
• Execute autoconfig on patch fs
09:46:46 00:00:00 ad.bin.adop EVENT Detecting configuration
changes.
09:46:46 00:00:00 ad.bin.adop PROCEDURE [START] Looking for file
system configuration changes
09:46:46 00:00:00 ad.bin.adop EVENT. File system configuration
changes detected.

25. ADOP Concepts - Apply
• ADOP applies specified patches to patch edition of DB and filesystem
– adop phase=apply patchtop=<$PATCH_TOP>
– merge=yes, listed patches will automatically be merged
– Multiple patches can be applied with one command
• adop phase=apply patches=2455636,2348464
– Apply patch and NLS
• adop phase=apply patches=2454666,2454666_ESA:u245466.drv
– Restart a failed patch
• adop phase=apply patches=2455636,2348464 restart=yes
• What about the failed worker?
– Analytics. Generates in-depth reports
• adop phase=apply analytics=yes

26. ADOP Concepts - Finalize
• Used to perform any remaining processing needed to ensure the fastest possible
cutover
– Pre-compute DDL that needs to be run at cutover.
– Compile all invalid objects.
– Validate that the system is ready for cutover.
– If finalize_mode=full, compute statistics for key data dictionary tables for improved
performance.
$ adop phase=finalize

27. ADOP Concepts - Cutover
• Critical phase – application is down
• Key actions
– Shutdown concurrent managers. ADOP signals ICM to shutdown but wait for running
requests
• cm_wait=<maximum_minutes_to_wait>
• Recommendations
– Production - do not specify cm_wait
– Non-Prod – specify cm_wait to limit the waiting time
– Shutdown application tier services
– Cutover database
– Cutover file system
– Terminate old database sessions
– Start application tier services

28. ADOP Concepts - Cutover
• $ adop phase=cutover
• Deferring application tier restart
– adop phase=cutover mtrestart=no
• If failed and restarted cutover will continue at the failure point
• JAR files
– JAR files are stored in the $APPL_TOP/admin/<SID>/out directory and uploaded to the DB
during cutover.
• Autoconfig is executed

29. ADOP Concepts - Cleanup
• If you fail to run cleanup phase explicitly it will be run during the next prepare
• Drops (removes/cleans) obsolete
– Crossedition triggers
– Seed data
– Editioned code objects (covered objects)
– Indexes
– Columns
– Editions
• 3 modes
– cleanup_mode=quick
– cleanup_mode=standard (default)
– cleanup_mode=full

30. ADOP Concepts - Abort
• Cancels/Terminates/Abandons the current patching cycle
• It won’t apply any patches to the run FS (live)
• It can only be used before the cutover phase
• It drops the patch edition in the DB
• Need to execute cleanup and fs_clone after abort
$ adop phase=abort
$ adop phase=abort,cleanup cleanup_mode=full
Photo by Clem Onojeghuo on Unsplash

31. ADOP Concepts - FS_CLONE
• fs_clone phase clones the run filesystem to the patch filesystem
• Executed:
– After ADOP Abort
– At the end of the patch cycle
– After implementing customizations on run filesystem
$ adop phase=abort
$ adop phase=fs_clone
$ adop phase=prepare

32. ADOP Concepts - ACTUALIZE_ALL
• Drop old database editions
• Editions are accumulated after patch cycles
• Recommended when 25 or more editions
• This procedure will take large amount of time
WARNING There are 25 existing editions. For optimum performance, there should be no more than
20.
WARNING To remove unneeded editions, run the actualize_all phase and then perform a full
cleanup.
EVENT Validating system is ready to prepare.
$ adop phase=prepare
$ adop phase=actualize_all
$ adop phase=finalize finalize_mode=full
$ adop phase=cutover
$ adop phase=cleanup cleanup_mode=full

33. Best Practices
• Apply all recommended database and AD-TXK patches.
How to Find E-Business Suite Recommended Patches (Doc ID 1400757.1)
• Run ETCC and ensure the system is up to date
• Disconnect any 3rd party applications prior the cutover phase
• Use prepare sync_mode=delta
$ adop phase=prepare sync_mode=delta
• Be sure there’s no active patching cycle before running adpreclone
• Turn the DB recycle bin off
######### # # recyclebin parameter #
# The database recyclebin must be turned off to allow
# the cleanup phase of the online patching cycle to be
# performed without having to connect as SYS.
# # This feature may still be used at other times.
# #########

34. Best Practices
• EBS Installation Configuration Management (ICM) Application DBA Online
Patching (ADOP) Analyzer (Doc ID 2379656.1)

35. ETCC
• E-Business Suite Technology Codelevel Checker (ETCC)
• Helps identify overlay patches needed to be applied
• Obtain ETCC Patch 17537119

36. Configuration Changes and Patching
• Online configuration changes (recommended)
– Execute prepare
– Perform configuration changes to the patch filesystem
• Weblogic configuration
• HTTP (Apache) configuration
• Filesystem changes
– Execute cutover to promote those changes
– Execute fs_clone
• Offline configuration changes
– Verify that no patching cycle is currently active
– Shutdown application services
– Perform configuration changes to run filesystem

37. Hot Patching?
• NO!
• 1987947.1, Granting Privileges On An Object May Cause Invalidations
Photo by Gemma Evans on Unsplash

38. DB Patches
• Most of the DB patches require the DB services to be down in the OH
• Rolling patches if running a RAC DB
• Recommend to install the DB patches after cutover
$ adop phase=prepare sync_mode=delta
$ adop phase=apply patches=4558512
$ adop phase=finalize
$ adop cutover mtrestart=no
SQL> shutdown immediate
$ opatch apply
SQL> startup
$ adstrtal.sh

39. DR Patching
• Most DB patches are DG Standby First Installable (README)
• Install patch to your DR Standby DB first
• Start ADOP Cycle
• Install DB patch to primary after cutover
• Install EBS patches to DR application
• Run ADOP Cleanup

40. Issues
• ADOP cycle slow
– Verify if AD_ZD_DDL_HANDLER table is fragmented
• Execute ADZDTABRDF.sql <owner> <table>
• Gather table stats
– SQL>exec dbms_stats.gather_table_stats('APPLSYS','AD_ZD_DDL_HANDLER’);
– Old Data in ADOP repository tables
• SQL>exec ad_zd_fixer.clear_adop_repo_tables;
• Cutover Fails
– Cutover_status=3
– 'Invalid Credentials' error
– database patch edition has already been promoted to be the new run edition
$ adop phase=cutover action=nodb

41. Blocking locks and Prepare
Error on session xx:
Creating database patch edition.
Log: @ADZDSHOWLOG.sql
Failed to execute SQL statement:
declare
result varchar2(10);
begin
ad_zd.PREPARE('Y’);
…
Error Message: ORA-20001: Error while calling ad_zd.prepare.ORA-04021:
timeout occurred while waiting to lock object
ORA-06512: at line 9 (DBD ERROR: OCIStmtExecute) [UNEXPECTED]Error creating
patch edition.
[UNEXPECTED]Error calling runCheckAndCreatePatchEdition subroutine.
[ERROR] Prepare phase has failed.

42. Blocking locks and Apply
• adop phase=apply error
– Adop worker timeout
– By looking at OEM huge red graph
– This was a runaway Endeca session
• Terminate the Endeca session
• Restart the worker using adctrl (option 2)
• Restart adop phase=apply
• adop phase=apply completed correctly

43. Cutover Issues
• Cutover hung
– Verify if cutover is waiting for long requests to complete
– See if those requests could be terminated
• Cutover failing due to bug 27883586
– Issue with Mviews
http://www.alfredokriegdba.com/2018/11/ebs-adop-phase-failing-due-to-ora-600.html
• Cutover fails – Timeout while starting services on secondary node
– $ adop phase=cutover mtrestart=no
ADOP Cutover Fails, Timeout While Starting Services On Secondary Node (Doc
ID 2411800.1)

44. Final Recommendations
• Always, always! Keep your system up-to date with ETCC
• Execute actualize_all regularly to cleanup old editions
• Make use of adop phase=prepare sync_mode=delta
• Be sure 3rd party applications are down before starting the prepare and apply if possible
• Decided to shutdown concurrent managers before running cutover
• Decided to execute cutover with mtrestart=no
• Plan ahead and be prepared

45. Session ID:
Remember to complete your evaluation for this session within the app!
10755
alfredo.krieg@viscosityna.com