Mike_Salt.pdf

3. 2 Safe Harbour Statement “The views expressed in this presentation are those of the author and do not necessarily reflect the views of Fujitsu U.K. nor Oracle Corporation” UKOUG Applications Unlimited 2019

4. 3 About Me…  24 Years Oracle Experience (v7+) • Developer • DBA / E-Biz DBA • Oracle Solution Architect • E-Business Suite Consultant  21 Years E-Business Suite Experience (10.5+)  21 Years Consultancy Experience: • Sun Microsystems • Fujitsu  DBA 1.5  SIG Attendance: • RDBMS • RAC • Application Server • E-Biz  Regular Presenter At UKOUG Annual Conference  Fujitsu Distinguished Engineer (2018) UKOUG Applications Unlimited 2019

5. 4 About FUJITSU . . . Where we work Approximately 140,000 Fujitsu colleagues working with customers in over 100 countries (March 2018) *EMEIA : Europe,Middle East,India and Africa. Oceania 3,000 EMEIA* 27,000 Asia 12,000 Japan 92,000 Americas 6,000 UKOUG Applications Unlimited 2019

6. 5 About FUJITSU . . . Working with world class partners UKOUG Applications Unlimited 2019

7. 6 About FUJITSU . . . Working with world class customers Public Sector Manufacturing Communications Health Retail Financial Services Transport & Logistics Utilities UKOUG Applications Unlimited 2019

8. © Copyright FUJITSU 2019 7 Oracle Infrastructure and Applications Simplify Adopt Innovate Fujitsu Multi Cloud Services Oracle IAAS and PAAS Partner of the Year 2019 Oracle ACS and Premier Support Partner Of The Year 2019 Oracle System Partner of the Year 2019

9. 8 UKOUG Applications Unlimited 2019 Introduction & Background 12.2

10. 9 UKOUG Applications Unlimited 2019 Possible Reasons for Upgrading to 12.2.X ? Regulatory Reasons New Product Features New Technical Features Better Look and Feel Longer Term Support Integration with Fusion Apps / SaaS Fun of the Challenge Applicable to our client Not Applicable to our client AKA Online Patching!! Enterprise Command Centres HTML / Customisable

11. 10 UKOUG Applications Unlimited 2019 Roadmap from Oracle (July 2017) https://www.oracle.com/assets/ebs-roadmap-4739519.pdf

12. 11 UKOUG Applications Unlimited 2019 Roadmap from Oracle (October 2018) http://www.oracle.com/us/products/applications/ebs-suite-12-2-announcement-5172019.pdf

13. 12 UKOUG Applications Unlimited 2019 Implications of New Roadmap http://www.oracle.com/us/products/applications/ebs-suite-12-2-announcement-5172019.pdf Oracle is today announcing a “Continuous Innovation” release model for Oracle E-Business Suite 12.2, delivering ongoing applications and underlying technology stack updates without a major upgrade. At the same time, we are announcing Premier Support for Oracle E-Business Suite 12.2 through at least 2030. This means that Oracle E-Business Suite customers already running on 12.2 can continue to access new applications functionality and separately uptake later versions of underlying technology for years to come, without the time and expense of a major release upgrade.

14. 13 UKOUG Applications Unlimited 2019 How it Works . . .

15. 14 UKOUG Applications Unlimited 2019 Part 1: The Upgrade Process (Non-Technical) 12.2

16. 15 UKOUG Applications Unlimited 2019 Client Architecture (Production, Simplified) • FUJITSU M10-4S • 74 x 3GHz SPARC64™ X cores • 2TB RAM • Big SAN (not shown) • FUJITSU M10-1 • 5 x 2.8GHz SPARC64™ X cores • 150GB RAM • ORACLE RDBMS 11.2.0.5 • EBS v12.1.3 Conc Manager Tier • EBS v12.1.3 “Middle Tier” • Large User Base

17. 16 UKOUG Applications Unlimited 2019 Client Upgrade Path Developer 10.1.2 COMMON_TOP APPL_TOP INST_TOP AS 10.1.3 Applications Tier v12.1.3 Developer 10.1.2 COMMON_TOP APPL_TOP INST_TOP Applications Tier V12.2.X Oracle HTTP Server (OHS) WebLogic Server (WLS) Fusion Middleware 11g DB UPGRADE?

18. 17 UKOUG Applications Unlimited 2019 Upgrade Approach: Phase A (Proof of Concept) Iteration A1  Take Recent Clone of Production  Follow the Oracle Upgrade Documentation  Document as you go!  Fix / Resolve Issues as you go! Iteration A2 . . An  Take a new Clone of Production  Follow your own documentation  Fix / Resolve Issues as you go!  Refine Documentation Hand-over for Functional Testing  Formulate List of Patches  Update Master Documentation Our Appendix!!

19. 18 UKOUG Applications Unlimited 2019 Upgrade Approach: Phase B (Refinement & Timing) Review Oracle Updates  New Patches & Documents Iteration B1  Take new Clone of Production  Use Production-Like Hardware  Use Different Resources  Follow & Refine own documentation  Conduct Performance Testing Iteration B2 . . Bn  Repeat of above until clean runs achieved  Add all timing information to documentation  Tune the migration process!! Hand-over for Functional Testing

20. 19 UKOUG Applications Unlimited 2019 Upgrade Approach: Phase C (Dress Rehearsals) Plan for Production-Like Upgrade Schedule Iteration C1  Take new Clone of Production  Use Production-Like Hardware  Use Multiple Resources  Follow your own Documentation  Verify Timings in Documentation  Refine documentation Iteration C2 . . Cn (optional)  Repeat of above until clean runs achieved  Establish Final Baseline (no more patches)  Last iteration should be done 24x7 Hand-over for Final Functional Testing

21. 20 UKOUG Applications Unlimited 2019 Upgrade Approach: Phase D (The Real Thing) Perform Pre-Upgrade Steps in Production Shut down Production Back up Production Perform Upgrade Steps  Use Multiple Resources (24x7 Tag Teams)  Follow your own documentation Internal Testing & Backup  Technical & Functional  Take Backup / Snapshot External User Testing  Small set of Functional Product Champions Go / No-Go!

22. 21 UKOUG Applications Unlimited 2019 Upgrade Gotchas Part 1 Process Can Take Many Months Oracle Things Change Frequently  AD/TXK DELTA Patches  Release Update Packs (RUPs)  Upgrade Patches  Database Patches  Application Patches  Etc etc Project Scope Creep  New Products Added  Database Upgrade Project Resources OBSOLETE! OBSOLETE! OBSOLETE! OBSOLETE! OBSOLETE! CLIENT DECISION

23. 22 UKOUG Applications Unlimited 2019 Upgrade Gotchas Part 2 Staging Software Area – Invalid Symbolic Link  References to “/d1/R12/ab/apps/V122X39”  Fixed manually Too Much Memory (yes, I said too much!!)  If >2TB Free Space under “/tmp” it causes problems!  Problem with old 10.1 Oracle Universal Installer, used by: • Rapid Wizard Installer (fails after several hours) • Adop “fs_clone” Operations (fails after an hour) applmgr> $ORACLE_HOME/oui/bin/runInstaller -help Preparing to launch Oracle Universal Installer from /tmp/OraInstall . . . Please wait ... applmgr> echo $? 255 applmgr> export TEMP=/var/tmp ## Required to fix 10.1 OUI ### Now re-check OUI applmgr> $ORACLE_HOME/oui/bin/runInstaller -help Preparing to launch Oracle Universal Installer from /tmp/OraInstall . . . Please wait ... Oracle Universal Installer, Version 10.1.0.6.0 Production . . . applmgr> echo $? 0 Problem! OK

24. 23 UKOUG Applications Unlimited 2019 Moving Goalposts – Some Things to watch for . . . Changes To Watch For . . . Migration Phase A B C D Upgrade Documentation (1320300.1 / 133070x.1 / 1349240.1 / E48839-nn / etc)   ?  Latest JDK Updates (1530033.1 / 1439822.1)     Consolidated Update Patch (CUPn – Section 3 of 1320300.1)   ?  Pre-Install Patches Report [Quarterly] (1448102.2)   ?  Latest Release Update Pack (RUP 12.2.n - Section 2 of 1320300.1)  ? ?  Latest AD/TXK Delta Rollup Patches (1583092.1 / 1617461.1)     Database PSUs & Overlay Patches (1147107.1 / ETCC Patch 17537119)     Latest Patches and Technology Bug Fixes (1594274.1 / ETCC Patch 17537119)     VERY COOL UTILITY!

25. 24 UKOUG Applications Unlimited 2019 Upgrade Process Tuning Determine Target  How much downtime is acceptable to client?  Target Timeline (Bank Holiday Weekends)  How much testing time to be built into plan? First Iterations (Early Learning)  Fix-on-Fail vs Raise Calls with Oracle  Tasks Run Serially  Focus on Long-Running tasks (30mins+)  Look out for Performance Patches Later Iterations (Get Smarter)  Determine Steps which can be done in Advance  Determine Steps which can be run in Parallel  Test, Time, Tune, Repeat . . .

26. 25 UKOUG Applications Unlimited 2019 Upgrade Process Tuning Example 1 (bsu.sh) WebLogic Patching Utility During Upgrade, needs to be run many times *  Remove Operations: 8 + Install Operations: 7  Before Tuning: 5 mins per operation (approx)  Time Taken: 75 mins!! Done in Preparation Phase, but. . . future? Two Steps For Tuning  Edit “bsu.sh” & Increase Heap Size: 512MB  2GB+  Apply Patch 27238412 (replaced 26664545) [Doc ID 2271366.1] After Tuning:  Operations now take approx 1min  Total Time Reduced to 15 mins (Saving 1 hour!!) * Numbers may vary: This was from our upgrade!

27. 26 UKOUG Applications Unlimited 2019 Upgrade Process Tuning Example 2 (Scale-up) We will see later that we need multiple “OACORE” JVMs Method for adding these is cumbersome (see below) Developed Scripted Approach Also re-used the same script for Cloning (Post Go-Live) Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle EBS Release 12.2 (Doc ID 1905593.1) 1). Logon to first target node 2). Run: $ perl <AD_TOP>/patch/115/bin/adProvisionEBS.pl ebs-create-managedserver -contextfile=<CONTEXT_FILE> -managedsrvname=oacore_server2 -servicetype=oacore -managedsrvport=7203 - logfile=<APPLRGF>/TXK/addMS_oacoreserver2.log 3). Wait a few mins for start-up / shutdown of managed server 4). Run: $ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl - contextfile=<CONTEXT_FILE> -configoption=addMS -oacore=testserver.example.com:7203 5). Repeat for each managed server (x42 for us) 1). From Primary Application Tier Node, Run: applmgr> cd /stage/R12.2/XTRA_SCRIPTS applmgr> add_new_managed_servers.sh 7 <node1> <node2> <node3> <node4> <node5> <node6> . . . ManagedServer oacore_server8 created successfully. ManagedServer oacore_server9 created successfully. ManagedServer oacore_server10 created successfully. . . . ManagedServer oacore_server49 created successfully. Complete, check logfiles to ensure success.

28. 27 UKOUG Applications Unlimited 2019 Final Upgrade Process: Prepare Phase (High Level) Download and Prepare All Software into Staging Area DB Upgrade: Preparation Phase  Install 12c RDBMS Software  Apply Latest PSU and Associated Patches  Prepare 12c Init.ora Files (Upgrade & Live Running) Apps Upgrade: Preparation Phase  Prepare Apps Staging Area  Run Pre-Upgrade Tasks  Create Upgrade Filesystems  Patch the Apps Tech Stacks  Merge AD12.2 Upgrade Driver  Merge Consolidated Upgrade Patches  Prepare Online Patching Enablement Patch and Report

29. 28 UKOUG Applications Unlimited 2019 Final Upgrade Process: Execute Phase (High Level) System Close & Backup/Snap DB Upgrade: Execute Phase  Run Pre-Upgrade Tool  Run “dbua” Upgrade Assistant  Run Manual Upgrade Post-Steps Backup/Snap Apps Upgrade: Execute Phase  Run Pre-Upgrade Tasks (Tech and Functional)  Close Down App  Apply Manual Work-Arounds  Run 12.2 Upgrade Steps (inc AD/TXK + R.U.P. etc)  Run Customisation Tasks  Run Post-Upgrade Tasks (inc Scale-out and Scale-up)

30. 29 UKOUG Applications Unlimited 2019 Final Upgrade Process: Hand-Over Phase (High Level) System Close Full System Backup / Snapshot Open System for Internal Testing Open System for Limited End-User Testing Open System Fully Take cover . . . !

31. 30 UKOUG Applications Unlimited 2019 Upgrade Summary What Worked Well What Didn’t Work Well Fully Documented Process Relatively Early Adopters (Bugs & Changes) Iterative Structured Approach Project Scope Creep (DB Upgrade) Performance Testing* & Tuning Release Update Pack 12.2.6 vs 12.2.7 Functional Testing Managing Resources Upgrade Completed On Time On Cut-Over Go Live Issues! (More on this later!!) Automated / Scripted Many Steps No Party!!

32. 31 UKOUG Applications Unlimited 2019 Part 2: EBS 12.2 Tuning (Technical) 12.2

33. 32 UKOUG Applications Unlimited 2019 E-Business Suite R12.2: Application Tuning Test Harness  Fully Sized Database  Production-Like Hardware  Load Simulator (Silk Performer) Key Tuneable Parameters  Database Parameters  OHS Parameters  Number of “OACORE” JVMs  Connection Pool Settings  WebLogic Logging  JVM Parameters Other Items For Tuning  Inefficient “Profiles” (later slide) 12.2

34. 33 UKOUG Applications Unlimited 2019 EBS Tuning: Database Parameters Based on the Previous “init.ora” Changes made for 11g  12c DB Upgrade Changes made for 12.1.3  12.2.X App Upgrade Refer to notes: [396009.1] & [174605.1]  Use “bde_chk_cbo.sql” provided in [174605.1] Some changes needed just for Upgrade Period (see below) WebLogic Connection Pool reduced backend connections  Enables Reduction to “Processes” & “Sessions” vs 12.1.3  See later slide on this! 12.2 ora[12c]$ cd $ORACLE_HOME/dbs ora[12c]$ vi init$ORACLE_SID.ora ############################################################################# # Added For 12.2 Upgrade - Remove Afterwards ############################################################################# log_buffer=100M job_queue_processes=30 parallel_max_servers = 60 max_dump_file_size = UNLIMITED # Prevents delays in adop enab (Doc ID 1946877.1)

35. 34 UKOUG Applications Unlimited 2019 EBS Tuning: OHS Parameters 12.2 vi $FMW_HOME/webtier/instances/*OHS[2-7]/config/OHS/*/httpd.conf under worker MPM. . . set: PARAMETER NEW VALUE DEFAULT VALUE COMMENTS ServerLimit 8 25 ThreadLimit 128 unset <--- This is needed to support ThreadsPerChild StartServers 8 5 MaxClients 1024 625 MinSpareThreads 100 25 <--- Gives warning when close to max-clients (<100 free) MaxSpareThreads 1024 75 <--- Will prevent any server shutdowns ThreadsPerChild 128 25 MaxRequestsPerChild 0 0 <--- No sign of memory leak on httpd.worker processes, so OK! KeepAliveTimeout 15 15 <--- Default (unchanged) MaxKeepAliveRequests 0 0 <--- Default (unchanged) Note: Also need to change "AcceptMutex fcntl" to AcceptMutex pthread And comment out the LockFile entries!

36. 35 UKOUG Applications Unlimited 2019 EBS Tuning: “OACORES” & DB Connection Pool Trials Revealed ~300 Concurrent Users* per JVM was OK Found 300 JDBC Connections per JVM worked well Connection Pool Harvesting (tweaked) Do the maths . . 12.2 *What’s a concurrent user? Your mileage will vary!

37. 36 UKOUG Applications Unlimited 2019 EBS Tuning: Load Balancer “Fencing” By Default, Oracle will use software load balancing  Incoming connection to OHS redirects to any managed server!  Can result in a lot of cross-tier network traffic  Not necessary if using Hardware Load Balancer  Less control in some maintenance scenarios Our solution was to “fence” . . .  After scale-out / scale-up . . . 12.2 applmgr> vi $FMW_HOME/webtier/instances/*OHS[2-7]/config/OHS/*/mod_wl_ohs.conf [Ensure OHS2 File has only WebLogicCluster commands referring to <Apps Node 1>] [Ensure OHS3 File has only WebLogicCluster commands referring to <Apps Node 2>] [Ensure OHS4 File has only WebLogicCluster commands referring to <Apps Node 3>] etc applmgr> vi $FMW_HOME/webtier/instances/*OHS[2-7]/config/OHS/*/apps.conf [Ensure OHS2 File has only BalancerMember commands referring to <Apps Node 1>] [Ensure OHS3 File has only BalancerMember commands referring to <Apps Node 2>] [Ensure OHS4 File has only BalancerMember commands referring to <Apps Node 3>] etc

38. 37 UKOUG Applications Unlimited 2019 EBS Tuning: WebLogic Logging By Default, WebLogic Logging is Verbose and Duplicated: 12.2 [OACORE Server Logs – Standard Out] applmgr> wc -l oacore_serverX.out 1831 oacore_serverX.out applmgr> egrep -c 'at |Truncated.|Caused By:|java:' oacore_serverX.out 1282 [OACORE Server Logs – Normal Log File] applmgr> wc -l oacore_serverX.log 17895 oacore_serverX.log applmgr> egrep -c ' at |java:|Caused By:' oacore_serverX.log 3639 applmgr> grep -c "<Info>" oacore_serverX.log 13741 [AdminServer Logs – Domain Log File] applmgr> wc -l EBS_domain_XXXXX.log 59713 EBS_domain_XXXXX.log applmgr> grep -c "<Error>" EBS_domain_XXXXX.log 872 applmgr> egrep -c ' at |ServletException|java:|Caused By:' EBS_domain_XXXXX.log 55491 Majority (70%) of Standard Out Messages are Truncated Stack Dumps Majority (75%) of Standard Log Messages are “INFO” Level Majority (94%) of Domain Log Messages are “ERROR” and Stack Dumps

39. 38 UKOUG Applications Unlimited 2019 EBS Tuning: WebLogic Logging (2) Here’s how to reduce this overhead: 12.2 Change From “Info” to “Notice” Select Select Change From “Trace” to “Notice”

40. 39 UKOUG Applications Unlimited 2019 EBS Tuning: WebLogic Logging (3) Here’s how to reduce this overhead (cont): 12.2 Change From “Notice” to “Warning” Change From “Notice” to “Critical” Reduce From 10 to 1 UNTICK!

41. 40 UKOUG Applications Unlimited 2019 EBS Tuning: JVM Parameters (OACORE) During Trials, experimented with many JVM parameters:  Increased Heap Size 1GB  3GB  Used Garbage First (G1) Collector  Set “DevPollDisabled” (doesn’t scale without this on some platforms)  Used “Tiered Compilation” (more on this later) 12.2

42. 41 UKOUG Applications Unlimited 2019 EBS Tuning: Changing JVM Parameters - Gotcha In 12.1.3:  Change JVM Parameters via “oacore_jvm_start_options”  Run Auto-Config In 12.2:  We have same parameter – yay!  However, only used when creating a new managed server!  Therefore need to do this before scale-out & scale-up! To Change in 12.2 after Managed Server Created:  Edit in WebLogic Console under Config  Server Start  Arguments  Or, edit the “$EBS_DOMAIN_HOME/config/config.xml” file • Be Careful! – Take backup first!  Or, use the “adProvisionEBS.pl ebs-set-managedsrvproperty” utility • Quite Messy, all arguments listed in full, refer to Doc ID 1905593.1 12.2

43. 42 UKOUG Applications Unlimited 2019 First Encounter with Problematic Profiles Discovered During Load Testing  Good Performance for first few hours of test  Performance Rapidly Declined  Observed the following:  Increasing size of result cache, 200MB  500MB did not fix issue! 12.2 Latch Name Tot Wait (s) Get Requests Miss % SLEEPS ------------------------------------ ------------ ------------ ------ --------- Result Cache: RC Latch 4000631.1 1268891679 125.4 59713271 row cache objects 76299.2 7095624871 13.9 50778810 cache buffers chains 1570.9 1.3298E+11 0.2 4289651 In memory undo latch 752.9 643026407 0.8 1069567 parameter table management 479.2 95434614 5.9 479927 NAMESPACE STATUS CACHE_ID ---------- --------- ------------------------------ NAME -------------------------------------------------------------------------------- NUM_RESULTS AVG_SCAN_PER_ENTRY MAX_SCAN_PER_ENTRY TOT_SCAN_CNT TOT_BLK_CNT ----------- ------------------ ------------------ ------------ ----------- PLSQL Published a77mb1r8mfppd0ktc3712ykmt2 "APPS"."FND_PROFILE"::11."GET_USER_VALUE"#4b60a496cb9dc4ee #505 206195 22 638031 4486880 206195 ICX: Language ICX:Session Timeout FND: Override Directory PA: Tasks to Display for Expenditure Entry etc etc

44. 43 UKOUG Applications Unlimited 2019 Problematic Profiles – Root Cause (1) Oracle’s Code Doesn’t Scale! Here’s how “FND_PROFILE.VALUE” function works:  As of: $Header: AFPFPROB.pls 120.43.12020000.7 2016/06/23  Moves from User  Responsibility  Application  . . .  Site  Each Sub-Function Uses RESULT_CACHE • get_user_value • get_resp_value • get_appl_value • etc 12.2 ** The algorithm is such: ** 1. Using the profile option information acquired at declaration, ** determine if the profile exists in FND_PROFILE_OPTIONS. If it does ** not, return NULL. ** 2. If it exists and the user_enabled_flag is Y, get user level value. ** 3. If there was no user-level value, proceed to check the applicable ** levels based on the profile option hierarchy. ** 4. If no value was obtained in all the other levels, check the site ** level if it is enabled for the profile option. *Refer to Bug 20198791 & Doc ID 2099303.1 for some background information

45. 44 UKOUG Applications Unlimited 2019 Problematic Profiles – Root Cause (2) Problem for Systems with many Active Users / Resps In R12.2 We Have Around 10,000 Profile Options  Even if only 50 of these were actually enabled at user / resp level  By ½ of all users and ½ of all responsibilities  That would be over 2 Million Result Cache entries (blocks)  Each Result Cache Block is 1kb in size, so that’s . . 12.2

46. 45 UKOUG Applications Unlimited 2019 Problematic Profiles – Root Cause (3) Problem is not:  Number of Profile Options that are Set  Paradoxically, better if set at User / Resp levels! Problem is:  Number of profile options that have ability to be set  . . . as NULL results still stored in result cache  . . . so what to do? Initial Solution was simple:  Disable Result Cache for “User Level” Lookups  Not documented* 12.2 ora[12c]$ vi $ORACLE_HOME/dbs/init$ORACLE_SID.ora ############################################################################# # Added For 12.2 - Enable Only After DB Upgrade Complete Inc Patch 17305006 ############################################################################# _result_cache_black_list=(a77mb1r8mfppd0ktc3712ykmt2) *The black_list feature has some refs, see bug 17305006, PSU merge patches needed before DB 12.2

47. 46 UKOUG Applications Unlimited 2019 Part 3: Post Go-Live (Technical) 12.2

48. 47 UKOUG Applications Unlimited 2019 Day 1 – Slow System & Complete Panic

49. 48 UKOUG Applications Unlimited 2019 Day 1 – Main Issues Integration Issues  Some custom static files needed amending  GOTCHA: Files changed under “OA_HTML” Need WebLogic Bounce! Lots of Performance Bugs  Especially • Email Centre – Stuck Threads on Apps Servers • Timecards – High “Log File Sync” due to excessive logging  None of these appeared during testing!  Load Test Suite focused on “Majority” Transactions  Some “esoteric features” did not scale! Return of the “Profiles” Issue  This was “Major Issue” for us!  Engaged with Senior Levels in Oracle 12.2

50. 49 UKOUG Applications Unlimited 2019 Week 1 – Example Issue: Timecards* Lots of messages appearing in “FND_LOG_MESSAGES” Piece of very bad code located (after manual search) Even though mobile timecards not in use! Common Code (clearly shows debugging): No Debug Check! So executed for every timesheet AND, “write_log” function had Explicit Commit! 12.2 $Header: hxclockapi.pkb 120.6.12020000.4 2014/01/30 09:54:17 asrajago ship -- Bug 18113601 hxc_mob_utils.write_log('p_resource_id :'||p_resource_id); hxc_mob_utils.write_log('p_process_locker_type :'||p_process_locker_type); hxc_mob_utils.write_log('p_start_time : '||p_start_time); hxc_mob_utils.write_log('p_stop_time : '||p_stop_time); hxc_mob_utils.write_log('p_time_building_block_id : '||p_time_building_block_id); hxc_mob_utils.write_log('p_time_building_block_ovn : '||p_time_building_block_ovn); hxc_mob_utils.write_log('p_expiration_time : '||p_expiration_time); hxc_mob_utils.write_log('p_row_lock_id : '||p_row_lock_id); -- Bug : Timecard Mobile -- Check if this timecard is locked in In Transit Tables. hxc_mob_utils.write_log(dbms_utility.format_call_Stack); hxc_mob_utils.write_log(p_process_locker_type); hxc_mob_utils.write_log('Checking mobile lock '); hxc_mob_utils.write_log('Checking mobile lock '||p_start_time||'-'||TO_CHAR(p_stop_time,'dd-MON-YYYY HH24:MI')); *This issue was present as of 12.2.7 Codeline!

51. 50 UKOUG Applications Unlimited 2019 Week 1 – Example Issue: Timecards* (2) We Told Oracle of the Fault in their Code! Fujitsu Proposed Interim Solution: 2 Lines of Code  Big Difference! 12.2 IF g_debug THEN -- Bug 18113601 hxc_mob_utils.write_log('p_resource_id :'||p_resource_id); hxc_mob_utils.write_log('p_process_locker_type :'||p_process_locker_type); hxc_mob_utils.write_log('p_start_time : '||p_start_time); hxc_mob_utils.write_log('p_stop_time : '||p_stop_time); hxc_mob_utils.write_log('p_time_building_block_id : '||p_time_building_block_id); hxc_mob_utils.write_log('p_time_building_block_ovn : '||p_time_building_block_ovn); hxc_mob_utils.write_log('p_expiration_time : '||p_expiration_time); hxc_mob_utils.write_log('p_row_lock_id : '||p_row_lock_id); -- Bug : Timecard Mobile -- Check if this timecard is locked in In Transit Tables. hxc_mob_utils.write_log(dbms_utility.format_call_Stack); hxc_mob_utils.write_log(p_process_locker_type); hxc_mob_utils.write_log('Checking mobile lock '); hxc_mob_utils.write_log('Checking mobile lock '||p_start_time||'-'||TO_CHAR(p_stop_time,'dd-MON-YYYY HH24:MI')); END IF; *This issue was present as of 12.2.7 Codeline!

52. 51 UKOUG Applications Unlimited 2019 Week 1 – Example Issue: Timecards* (3) Oracle’s Final Solution: Moved away from problematic “write_log” Included the “debug” check first! 12.2 $Header: hxclockapi.pkb 120.6.12020000.7 2018/01/04 16:59:42 asrajago ship $ IF g_debug THEN hr_utility.trace('p_resource_id :'||p_resource_id); hr_utility.trace('p_process_locker_type :'||p_process_locker_type); hr_utility.trace('p_start_time : '||p_start_time); hr_utility.trace('p_stop_time : '||p_stop_time); hr_utility.trace('p_time_building_block_id : '||p_time_building_block_id); hr_utility.trace('p_time_building_block_ovn : '||p_time_building_block_ovn); hr_utility.trace('p_expiration_time : '||p_expiration_time); hr_utility.trace('p_row_lock_id : '||p_row_lock_id); -- Bug : Timecard Mobile -- Check if this timecard is locked in In Transit Tables. hr_utility.trace(dbms_utility.format_call_Stack); hr_utility.trace(p_process_locker_type); hr_utility.trace('Checking mobile lock '); hr_utility.trace('Checking mobile lock '||p_start_time||'-'||TO_CHAR(p_stop_time,'dd-MON-YYYY HH24:MI')); END IF; *This issue was present as of 12.2.7 Codeline!

53. 52 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles Still issues with Result Cache (esp for “RESP”) Added “RESP” and “SITE” into Black List Result Cache Performance Improved System Just About Coped, But . . . 12.2 Elapsed Elapsed Time Time (s) Executions per Exec (s) %Total %CPU %IO SQL Id ---------------- -------------- ------------- ------ ------ ------ ------------- 250,607.2 113 2,217.76 21.5 25.9 .0 6sc8jn1bssj0c select count(*) from per_workforce_x hr,per_phones pho,hr_lookups hrl,hr_locati ons loc,hr_territories_v tr WHERE hr.person_id = pho.parent_id(+) AND pho.phon e_type(+) = 'W1' AND pho.parent_table(+) = 'PER_ALL_PEOPLE_F' AND sysdate BETWEE N pho.date_from(+) AND nvl(pho.date_to(+), sysdate) AND pho.phone_type = hrl 149,828.2 129,441,219 0.00 12.9 8.0 .0 7qwsx7rw0s3a5 SELECT PROFILE_OPTION_VALUE FROM FND_PROFILE_VALUES_USER_V WHERE PROFILE_OPTION_ ID = :B3 AND APPLICATION_ID = :B2 AND LEVEL_VALUE = :B1 AND PROFILE_OPTION_VALUE IS NOT NULL 54,806.2 156,747,772 0.00 4.7 25.3 .0 gj84g2yx56u2q SELECT PROFILE_OPTION_VALUE FROM FND_PROFILE_VALUES_SITE_V WHERE PROFILE_OPTION_ ID = :B2 AND APPLICATION_ID = :B1 AND PROFILE_OPTION_VALUE IS NOT NULL 49,114.6 108,659,530 0.00 4.2 25.1 .0 9k7gd9pn661pj SELECT PROFILE_OPTION_VALUE FROM FND_PROFILE_VALUES_RESP_V WHERE PROFILE_OPTION_ ID = :B4 AND APPLICATION_ID = :B3 AND LEVEL_VALUE = :B2 AND LEVEL_VALUE_APPLICAT ION_ID = :B1 AND PROFILE_OPTION_VALUE IS NOT NULL

54. 53 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (2) During peak times, system went into meltdown!  Huge Mutex Issues – Massive ramp up in CPU  Needed Complete Bounce to resolve! Oracle Advised “Beef Up” RC & remove black lists!  We were reluctant, but client insisted!  So we did it in the evening . . . but . . . 12.2 Total Wait Wait % DB Wait Event Waits Time (sec) Avg(ms) time Class ------------------------------ ----------- ---------- ---------- ------ -------- latch free 54,089,624 181.9K 3.36 77.3 Other DB CPU 39.2K 16.7 db file sequential read 2,273,797 9988.5 4.39 4.2 User I/O log file sync 79,076 2933.1 37.09 1.2 Commit Pct Avg Wait Pct Get Get Slps Time NoWait NoWait Latch Name Requests Miss /Miss (s) Requests Miss ------------------------ -------------- ------ ------ ------ ------------ ------ Result Cache: RC Latch 550,997,581 26.2 0.4 2.E+05 1 0.0 PL/SQL Entry Subprogram % Activity ----------------------------------------------------------------- ---------- APPS.PAY_ARCHIVE.PROCESS_EMPLOYEE 55.13 APPS.FND_PROFILE.VALUE 53.07

55. 54 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (3)

56. 55 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (4) Oracle keen to get to bottom of Result Cache Issue! 1st Problem: Invalidations  Single change to any profile value invalidates ALL profile caches!  But, we have many changes every day . . . 12.2 SQL> select APPLICATION_ID,PROFILE_OPTION_ID,LEVEL_ID,LEVEL_VALUE, 2 to_char(LAST_UPDATE_DATE,'HH24:MI:SS') 3 from apps.fnd_profile_option_values 4 where LAST_UPDATE_DATE>sysdate-(8/24) 5* order by LAST_UPDATE_DATE; APPLICATION_ID PROFILE_OPTION_ID LEVEL_ID LEVEL_VALUE TO_CHAR( -------------- ----------------- ---------- ----------- -------- 0 13791 10004 315365 08:43:12 690 4306 10004 359689 09:24:39 178 7646 10004 359689 09:24:39 690 4306 10004 283772 09:31:48 . . . 690 4306 10004 365973 11:00:46 690 4292 10004 365973 11:00:46 . . . 178 3798 10004 481853 14:07:23 0 3773 10004 481853 14:07:35 178 4108 10004 481853 14:08:55 800 1204 10004 241598 15:22:18 49 rows selected. Show all profile changes in last 8 hours Almost all changes are “USER” Level (10004) Sometimes changes are bunched together! Lots of changes!

57. 56 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (5) How to cut-down on Invalidations?  Can’t stop users from making changes!  However, found a “Password Sync” job: • Synchronising between SSO and E-biz • Developed by Oracle Consulting • Ran Every 15 minutes and made profile changes!! • Changed to run once in the evening • Improved situation, but ad hoc user changes still prevalent 12.2

58. 57 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (6) I Suggested Design Change to Oracle:  Replace “FND_PROFILE_OPTION_VALUES” Table with: • FND_PROFILE_OPTION_VALUES_USER • FND_PROFILE_OPTION_VALUES_RESP • FND_PROFILE_OPTION_VALUES_SITE • etc  Add “FND_PROFILE_OPTION_VALUES” View • Based on “UNION ALL” of all of the above • For backwards compatibility  Change “FND_PROFILE” RC functions: • Each would select from their own Specific Table  Would give Result Cache independence  Even better* option would be Partitioned Table!  Oracle said . . . NO! Too much change for them! 12.2 *Need to check result cache invalidation based on partition?

59. 58 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (5) 2nd Problem: Too many Result Cache Entries  Remember: NULL Entries are held in cache  Remember: User  Responsibility  Application  . . .  Site  Problem gets worse as you fix first problem! Major Performance Problems if Result Cache gets full  Oracle’s solution was to increase RC size Poor Performance when lots of entries in Result Cache  Especially when invalidations occurring!  Due to long hash chain length. 12.2

60. 59 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (6) How to reduce number of Result Cache entries?  Check which Profiles hitting cache hardest  Check Profiles not in use at various levels  Where not in use  DISABLE 12.2 ICX: Language disable RESP/USER ICX:Session Timeout disable USER only FND: Override Directory disable RESP/USER PA: Tasks to Display for Expenditure Entry disable RESP/USER RCV: Fail All ASN Lines if One Line Fails disable RESP/USER Achieved 10% Reduction in RC Entries with just these 5!

61. 60 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (6) How to manage Result Cache when getting full?  Manual Methods:  Fujitsu Developed Automatic Solution: • Monitor Result Cache on periodic basis (e.g. 5mins) • Invalidate Specific Cache Entries when >50,000 (approx 50MB) 12.2 SQL> exec dbms_result_cache.flush();  Whole Cache SQL> exec dbms_result_cache.INVALIDATE_OBJECT(<cache_id>);  Single Entry Wed Apr 18 16:23:13 2018 FUJITSU RC MONITOR Invalidating large (63796) cache id: a77mb1r8mfppd0ktc3712ykmt2 Wed Apr 18 16:34:16 2018 Thread 1 cannot allocate new log, sequence 651 Private strand flush not complete Beginning log switch checkpoint up to RBA [0x28b.2.10], SCN: 287205410218 Wed Apr 18 16:34:19 2018 Thread 1 advanced to log sequence 651 (LGWR switch) Wed Apr 18 16:38:14 2018 FUJITSU RC MONITOR Invalidating large (56720) cache id: a77mb1r8mfppd0ktc3712ykmt2 Wed Apr 18 16:39:22 2018 Completed checkpoint up to RBA [0x28b.2.10], SCN: 287205410218 Wed Apr 18 16:53:15 2018 FUJITSU RC MONITOR Invalidating large (57459) cache id: a77mb1r8mfppd0ktc3712ykmt2

62. 61 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (7) So what happened next? Fujitsu Spent Time Looking at “Mutex Meltdown” Issue suspected as “hot objects” [Ref: 2194448.1] Three things done for “USER” and “RESP” Queries:  1). Make SQLs as fast as possible with Indexing & Baselines  2). Set “Hot Object Copies” parameter to 16  3). Add SQLs as “HOT” during database start-up: 12.2 ### Run MarkHot Procedure To Ensure Multiple Copies Of USER/RESP Busy SQL sqlplus -s / as sysdba <<EOF set heading off select APPS.FND_PROFILE.VALUE_SPECIFIC('BOM:STANDARD_ITEM_ACCESS',1234,1234) from dual; begin sys.dbms_shared_pool.markhot (hash=>'cb73920f7392208c7b731d3df80c0d45', namespace=>0, global=>true); sys.dbms_shared_pool.markhot (hash=>'39966b9bcb73f953991ded4d686306b1', namespace=>0, global=>true); end; / exit; EOF

63. 62 UKOUG Applications Unlimited 2019 Month 1 – Main Issue: Profiles (8) New Solution Implemented  Black Lists added again for “USER” and “RESP” Queries  Good, Consistent and Stable Performance Hidden Snares  Non-Standard Solution (we are using 2 sticking plasters)  Relies on “black_list” Procedure • Not standard until DB12.2 • Have to request new merge patch with each PSU • Changes in Oracle code could generate new signatures!  Relies on “markhot” Procedure • Beware manual DB bounces without using script • Changes in Oracle code could generate new signatures!  Code is still Inefficient • Exasperated with use of “HR_SECURITY” 12.2

64. 63 UKOUG Applications Unlimited 2019 Future Plans For Profiles Issue Move back towards Result Cache Trim down as many unnecessary profiles as possible  60% of hits already removed! Keep pushing Oracle for Enhanced Profiles Solution Use new Fujitsu RC Monitor to maintain stability Keep tuning “HR Security” problematic queries 12.2

65. 64 UKOUG Applications Unlimited 2019 Online Patching 12.2 HEY – Wait a minute! This is a 12.2 Presentation and we have NOT spoken about online patching!!!

66. 65 UKOUG Applications Unlimited 2019 Much Later – Online Patching Issue 12.2 NOTE: Logarithmic Scale! *Bug 22568797 - latch: row cache objects contention with editions. *Bug 20161956 - Adop:prepare Should Throw Warning If No. Of Editions Exceed Threshold Limit

67. 66 UKOUG Applications Unlimited 2019 Much Later – Online Patching Issue (2) Oracle’s internal guides say . . .  We need to actualize when we have >20 Editions However, for busy systems need lower threshold!  Patch 22568797 Should Help, but . . .  We now “actualize_all” after every patching session! 12.2 SQL> select count(*) from dba_editions; COUNT(*) ---------- 5 SQL> select count(*) from dba_objects; COUNT(*) ---------- 459168 SQL> select count(*) from obj$; COUNT(*) ---------- 1203883 SQL> select count(*) from dba_editions; COUNT(*) ---------- 1 SQL> select count(*) from dba_objects; COUNT(*) ---------- 459164 SQL> select count(*) from obj$; COUNT(*) ---------- 460160

68. 67 UKOUG Applications Unlimited 2019 Much Later – Online Patching Issue (3) How we patch with “actualize_all” 12.2 $ adop phase=prepare $ adop phase=apply patches=xxx,xxx... $ adop phase=finalize [bring down system manually] $ adop phase=cutover mtrestart=no $ . /home/$LOGNAME/.profile $ adop phase=cleanup $ adop phase=prepare $ adop phase=actualize_all $ adop phase=finalize finalize_mode=full $ adop phase=cutover mtrestart=no $ . /home/$LOGNAME/.profile $ adop phase=cleanup cleanup_mode=full SQL> ?/rdbms/admin/utlrp [bring up system manually] SQL> exec dbms_stats.gather_fixed_objects_stats(); SQL> exec dbms_stats.gather_dictionary_stats(); Can take over 2 hours (fs_clone) 10mins  25mins ~10mins (much faster than “mtrestart”) 5mins  10mins 10mins++ (depending on number of patches) 30mins+ 35mins+ 10mins+ 25mins+ Can take over 3 hours! 10mins  30mins ~15mins ~8mins ~30mins ~10mins

69. 68 UKOUG Applications Unlimited 2019 Much Later – Online Patching Issue (4) How we patch with “actualize_all” 12.2 $ adop phase=prepare $ adop phase=apply patches=xxx,xxx... $ adop phase=finalize [bring down system manually] $ adop phase=cutover mtrestart=no $ . /home/$LOGNAME/.profile $ adop phase=cleanup $ adop phase=prepare $ adop phase=actualize_all $ adop phase=finalize finalize_mode=full $ adop phase=cutover mtrestart=no $ . /home/$LOGNAME/.profile $ adop phase=cleanup cleanup_mode=full SQL> ?/rdbms/admin/utlrp [bring up system manually] SQL> exec dbms_stats.gather_fixed_objects_stats(); SQL> exec dbms_stats.gather_dictionary_stats(); ~6 Hours for Cut-Over Rather than ~1 Hour without Actualize!

70. 69 UKOUG Applications Unlimited 2019 Much Later – Online Patching Issue (5) Patches Required before using “ACTUALIZE_ALL”:  25099339: • Trigger On Editioning View Not Firing When Updated Via Inline View  27997875 • Regression Of Base Bug 25919622 (Invalid Objects Observed In Ebs Environment After Drop Edition ORA$BASE) Beware, you may see: But*, if you check the logs: 12.2 The cleanup phase completed successfully. [ERROR] ORA-04045: errors during recompilation/revalidation of APPS.xxxobjectxxx ORA-04052: error occurred when looking up remote object MYOBJECT@XXXXXXXXX ORA-00604: error occurred at recursive SQL level 3 ORA-12541: TNS:no listener, SQL[V_20180920_1105]: drop PACKAGE "APPS"."xxxobject" [ERROR] Could not drop covered object [V_20180920_1105] APPS.xxxobjectxxx (PACKAGE): ORA-04045: errors during recompilation/revalidation of APPS.xxxobjectxxx *Many possible reasons why clean-up may silently fail, example above is reference to remote object

71. 70 UKOUG Applications Unlimited 2019 Much Later – Online Patching Issue (6) Belt and Braces Check: 12.2 SQL> select count(*) from dba_editions; COUNT(*) ---------- 5 SQL> select count(*) from dba_objects; COUNT(*) ---------- 459168 SQL> select count(*) from obj$; COUNT(*) ---------- 1203883 SQL> select * from dba_editions; EDITION_NAME PARENT_EDITION_NAME usable -------------------------------------------------------------------------------- V_20180920_1105 V_20180630_1205 YES V_20180922_1132 V_20180920_1105 YES V_20180630_1205 YES SQL> select count(*) from dba_editions; COUNT(*) ---------- 1 SQL> select count(*) from dba_objects; COUNT(*) ---------- 459164 SQL> select count(*) from obj$; COUNT(*) ---------- 460160

72. 71 UKOUG Applications Unlimited 2019 Much Later – JVM Tuning Remember this from slide 38? We hadn’t noticed . . 12.2 Warnings reported in oacore_serverX.out Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=

73. 72 UKOUG Applications Unlimited 2019 Much Later – JVM Tuning (2) Fair Re-Trial, Compare Apples with Apples Non-Tiered (Default) Compilation is Best!  But, you may need to set -XX:ReservedCodeCacheSize=nnn  We also needed to increase our heap size 3GB  4GB 12.2

74. 73 UKOUG Applications Unlimited 2019 Much Later – Quarterly PSUs Great idea to maintain latest PSUs (eg. DB and WLS) But, be careful not to go “too recent”* Example Ref: 2443497.1 Download Latest ETCC Patch 17537119 Simple check of mappings file . . .  E.g. ETCC Patch: 29th Nov 2018: 12.2 applmgr> cd $INSTALL_BASE/fs_ne/etcc/mw applmgr> grep WLS_HOME txk_R1220_MT_mappings.xml | tail -1 <Technology name="WLS_HOME" version="10.3.6.0.181016"> applmgr> cd $INSTALL_BASE/fs_ne/etcc/db/onprem applmgr> grep "DB_HOME" txk_R1220_DB_mappings.xml | grep 12.1 | tail -1 <Technology name="DB_HOME" version="12.1.0.2.181016ProactiveBP"> applmgr> cd $INSTALL_BASE/fs_ne/etcc/db/onprem applmgr> grep "DB_HOME" txk_R1220_DB_mappings.xml | grep 12.1 | grep -v Pro | tail -1 <Technology name="DB_HOME" version="12.1.0.2.181016"> *Wait ~1 month after release! Example Ref: 2443497.1 – Download latest EBS Technology Codelevel Checker

75. 74 UKOUG Applications Unlimited 2019 Part 4: Wrap Up! 12.2

76. 75 UKOUG Applications Unlimited 2019 Be Aware (bigger software stack / more ports)

77. 76 UKOUG Applications Unlimited 2019 Cool Features of R12.2 WebLogic Server  Good visibility of System Performance  Even better when integrated with OEM!  Faster Start-up / Shutdown!  Better connection pooling!

78. 77 UKOUG Applications Unlimited 2019 WebLogic Admin Console (Summary of Servers) QUICK HEALTH VIEW USEFUL!

79. 78 UKOUG Applications Unlimited 2019 Dealing with Stuck Threads 12.2

80. 79 UKOUG Applications Unlimited 2019 Thanks For Your Time! Any Questions? Contact: michael.salt@uk.fujitsu.com

81. 80 UKOUG Applications Unlimited 2019

Mike_Salt.pdf

Mike_Salt.pdf

Recomendados

Mais conteúdo relacionado

Similar a Mike_Salt.pdf

Similar a Mike_Salt.pdf(20)

Último

Último(20)

Mike_Salt.pdf