Concurrent processing is a critical functionality in any e-Business Suite system. DBAs will recognize problems like jammed concurrent manager queues, failover problems, load balancing issues causing one node to be more loaded that another, performance overhead or even an unwanted bounce of the managers when incompatibility rules change. It is important to understand the configuration of a reliable concurrent processing environment, therefore, topics like PCP, node affinity, load balancing, optimal cache size and sleep time settings, separation of manager duties, request groups and others will be discussed.

1. REMINDER
Check in on the
COLLABORATE mobile app
Ins and Outs of Concurrent Processing
Configuration in Oracle e-Business Suite
Maris Elsins
Lead Database Consultant
Pythian
Session ID#: 10275
@MarisElsins

2. Maris Elsins

3. Maris Elsins
Lead Database Consultant at Pythian
Located in Riga, Latvia
Oracle [Apps] DBA since 2005
Speaker at conferences since 2007
@MarisElsins elsins@pythian.com
http://bit.ly/getMOSPatch

4. About Pythian
Founded in 1997, Pythian is a global
leader in data consulting and managed
services specializing in planning,
optimizing, and managing mission-
critical data systems
Top 5% talent worldwide
10 Oracle ACEs
2 Oracle ACE Directors
18 years in business
400+ employees
250+ customers worldwide

5. Agenda
Photo by Giulia van Pelt / CC BY-NC-ND 2.0

6. Agenda
■ Parallel Concurrent Processing
■ Node Affinity and Load Balancing on the DB tier
■ Performance configuration
■ Specialization rules
■ Separation of Manager Duties

7. Parallel Concurrent Processing

8. Parallel Concurrent Processing
■ Increases throughput
■ Improves Availability

9. PCP Diagram

10. Apps Listener
■ TNS listener APPS_{s_appsEnvName}
■ adalnctl.sh start
■ Allows ICM to communicate to FNDSM

11. Service Manager
■ FNDSM
■ Starts and stops concurrent managers

12. Internal Monitor
■ Monitors the ICM and restarts it if it’s not running
▪ Checks ICM every 60 seconds
▪ Trying to acquire the lock “FNDCPLK_ICM”
— The exclusive lock is kept by the ICM
— Successfully acquired -> Start the manager
— Failed -> the ICM is already
■ Why “user-defined” lock is used?
▪ Extremely lightweight
▪ Only one process can acquire it
▪ Automatic cleanup
■ Not activated by default
■ No work-shift by default

13. Internal Concurrent Manager
■ ICM monitors other managers
■ Can run on any concurrent node
■ Monitors other managers during PMON cycles
▪ The cycle = 2 minutes by default
▪ Starts / Stops managers as requested
▪ Checks the statuses of other managers
— Uses LOCK mode similarly to IM
— Lock name: FNDCPLK_<app_id>_<manager_id>_<process_id>
— Lock acquired = Manager down, start it!

14. The Reviver Process
■ If the ICM looses the connectivity to the DB
▪ crash of a RAC node
▪ network issues
■ The ICM Starts the reviver.sh and exits
■ The reviver process
▪ Checks the connectivity to the DB
▪ Starts the ICM when the connectivity recovers, and exits
■ Not enabled by default, enable it:
▪ s_cp_reviver=true
▪ s_fndreviverpiddir={path}

15. Failover of Concurrent Managers
■ Configuration options
▪ Primary node
▪ Secondary node
■ How does it work?
▪ Primary=node1, Secondary=node2
▪ Primary=node1, Secondary=NULL
▪ Primary=NULL, Secondary=NULL

16. Load Balancing
on the Apps Tier
■ All processes of the same concurrent manager runs on the
same node
■ Load Balancing options?
▪ NONE!
■ But Load Balancing is still possible!
▪ Create new managers
▪ Copy the specialization rules
▪ Set the work-shifts
▪ Set the primary/secondary nodes
■ Shouldn’t it been more simple?

17. Concurrent:PCP Instance Check
■ When set to TRUE …
▪ When an instance is down, all managers connecting to it switch
to a secondary middle-tier node
■ s_cp_twotask
▪ {INSTANCE_NAME}
▪ {DB_NAME}_BALANCE
■ Why do you want to run a CM without connection to the DB?
▪ Set “Concurrent:PCP Instance Check“=TRUE
▪ The managers try to re-connect to the same node before failing
over.

18. Node Affinity and
Load Balancing on the DB tier

19. Why node affinity is important?
■ Functionally different modules may perform better if they are
executed on different RAC instances
■ Performance
▪ Use caching more efficiently
— reduce double-caching
— Maximize the cached data per functionality
▪ Reduce the interconnect traffic
■ Availability
▪ Load spike in one functionality does not affect others

20. Services on RAC?
■ logical abstractions for managing workloads in Oracle
Database*
▪ A single entity for managing applications that compete for the
same resources
▪ Allow each workload to be managed as a unit
▪ Hide the complexity of the cluster from the client
▪ Failover to available nodes
▪ Implements load balancing
■ Integrates with other features
▪ Database Resource Manager
▪ Diagnostic features
* https://docs.oracle.com/cd/E18283_01/server.112/e17120/create007.htm
Services are
goooooooooood!

21. What’s possible out of the box?
■ How EBS Concurrent Processing Should Run on Oracle RAC
▪ http://www.pythian.com/blog/pcp-and-rac/
▪ by Yury Velikanov
■ It would be nice if we could define TNS alias for each CM
■ Limitations
▪ Node-wide setting s_cp_twotask defines the connectivity
— {DB_NAME}_BALANCE
— {INSTANCE_NAME}
▪ Affects all CMs on the node
■ Workarounds
▪ Target instance – setting for Concurrent Program
▪ Add custom services and TNS aliases to connect to a subset of
nodes, but all managers will still use the same configuration

22. The most important patch for eBS on RAC
■ 18803853 “1OFF:4159920:APPSRAP:PCP/Concurrent
Manager – RAC node affinity & listener load”
▪ The ER was opened in 2005 by Dell
▪ Patch created in 2014 for R12.1
■ Gives us the “Environment” settings for each concurrent
manager
■ No patch needed in R12.2

23. Using the “Environment” feature
■ How to use it?
1. Create the DB services (i.e. PAY_STD1)
2. Create the TNS aliases in tnsnames.ora IFILEs on all
concurrent nodes.
3. Add the appropriate variable=TWO_TASK and value={service
name} to the “Environment” settings
4. Bounce the CM (only the ones affected).
5. The CM processes will connect to the service by using the new
customized tnsnames.ora entry.
■ Each CM can use different service if needed
■ Allows using all features the DB services provide
■ Especially useful for implementation of MRP/ACP Planning
failover (Doc ID 279156.1)

24. Performance configuration

25. What can be tuned for Concurrent
Managers
■ Sleep Seconds
■ Cache Size
■ Number of Processes

26. Minimize the pending time

27. «Concurrent Manager» process workflow

28. Sleep Seconds
■ Sleep seconds = Time to wait between 2 attempts to fetch the
requests queue
■ If you have multiple manager processes, sleep time is the max time
to wait until the new request is picked up from the queue.
▪ e.g. 5 Standard managers, sleep time 60 seconds = average time to
pick up the new request from the queue if all managers are idle: 60/5 =
~12 seconds.
■ Nothing is guaranteed = works only if at least 1 CM process is idle
■ Guideline for tuning:
▪ Set «sleep seconds» = avg requirement * processes * (1 - utilization)
▪ Example 1: Req = 20s, util =20%, proc = 3
— 20 * 3 * (1 - 0.2) = 20 * 3 * .8 = 48 seconds
▪ Example 2: Req = 20s, util =90%, proc = 3
— 20 * 3 * (1 - 0.9) = 20 * 3 * .1 = 6 seconds , less then requirement? Need
more processes!

29. Cache size
■ Cache size = The number of pending requests being fetched
for execution from the requests queue
■ What cache size to choose?
▪ Small cache sizes are better for managers executing long
running requests
▪ Small cache sizes are better for managers with lots of processes
▪ Larger cache size is good for managers with few processes
executing quick (or both quick and slow) requests
■ If at least one request was fetched, the queue will be checked
again right after all the cached requests have been processed
▪ This reduces the significance of the cache size
■ Guideline for tuning:
▪ None, the effects are insignificant

30. Number of processes / Workshifts /
Dedicated concurrent managers
■ Guideline for tuning:
▪ Have at least 1 idle manager process at any moment of time
▪ Having too many processes can exhaust CPU
■ Separate different types of requests by creating dedicated
managers
▪ Separate processing of slow / fast / critical requests
▪ Adjust sleep seconds / cache sizes / number of processes
based on requirements
■ Consider implementing workshifts if:
▪ There are significant differences in utilization levels of managers
at different periods of time
▪ There’s a need to change priorities of the workloads (batch /
oltp) at different periods of time

31. Specialization rules

32. Specialization rules
■ Rules that segregate the duties between different managers
▪ Read only at the startup of the manager
▪ Changes to the Specialization Rules cause the CM to restart
■ Exclude
▪ NOT IN predicate is
added to the SQL
■ Include
▪ IN predicate is added to
the in the SQL
▪ No Include rule “Includes”
all requests
■ Request types are not
used by default

33. Building SQL for querying the requests queue
…
And (P.Execution_Method_Code != 'S' OR
(R.PROGRAM_APPLICATION_ID,
R.CONCURRENT_PROGRAM_ID) IN
((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757)))
AND ((R.PROGRAM_APPLICATION_ID,
R.CONCURRENT_PROGRAM_ID) NOT IN
((510, 40032),
(510, 40033),
(510, 42156),
(510, 42157),
(530, 43793),
(530, 43794),
(535, 42626),
(535, 42627),
(535, 42628)) AND
((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND
R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR
(R.REQUEST_CLASS_APPLICATION_ID,
R.CONCURRENT_REQUEST_CLASS_ID) NOT IN ((0, 2)))))
ORDER BY NVL(R.priority, 999999999), R.Priority_Request_ID,
R.Request_ID
Select R.Rowid
From Fnd_Concurrent_Requests R
Where R.Hold_Flag = 'N'
And R.Status_Code = 'I'
And R.Requested_Start_Date <= Sysdate
And (R.Node_Name1 is null or
(R.Node_Name1 is not null and
FND_DCP.target_node_mgr_chk(R.request_id) = 1))
AND EXISTS
(Select Null
From Fnd_Concurrent_Programs P
Where P.Enabled_Flag = 'Y'
And R.Program_Application_Id = P.Application_Id
And R.Concurrent_Program_Id = P.Concurrent_Program_Id
AND EXISTS
(Select Null
From Fnd_Oracle_Userid O
Where R.Oracle_Id = O.Oracle_Id
AND EXISTS (Select Null
From Fnd_Conflicts_Domain C
Where P.Run_Alone_Flag = C.RunAlone_Flag
And R.CD_Id = C.CD_Id))
…
2010-11-29
Query only «Pending/Normal» requestsDistributed Concurrent Processing implementation«Run Alone» flag implementation with Conflict domainsImplementation of «Immediate» type executables (Subroutines)Exclusion specialization rules for concurrent programsExclusion Specialization rule for Request typeExecution order for concurrent requests

34. Avoiding the restarts
Using the Request Types

35. Avoiding the restarts
Using the Request Types

36. Avoiding the restarts
Using the Request Types
Photo by Steve Jurvetson / CC BY 2.0

37. Separation of Manager Duties

38. Default Configuration
■ Standard Manager is used for most requests
▪ Unpredictability of requests’ completion times
— Long tasks mixed with short tasks
— Processes may be used by long tasks, delaying quick ones
▪ Deadlocks
— Parent/Child programs
— All managers blocked by Parent requests

39. Segregation of Manager Duties
■ Implementation flow
▪ Define sets of concurrent programs that should be separated
— Elapsed time
— Pending Time
— Parent / Child requests
▪ Add new managers, and create a request type for each
manager
▪ Define specialization rules to use request types
▪ Add the new request types to Exclude rules to the standard
manager (downtime)
▪ Add programs to the request types. (no downtime)

40. LONG MEDIUM QUICK1 QUICK2 STANDARD
Long reqs
Medium reqs Medium reqs
Short reqs 1 Short reqs 1 Short reqs 1
Short reqs 2 Short reqs 2 Short reqs 2
Parent req 1 Parent req 2
Child req 1 Ckild req 2
All others
Segregation of Manager Duties

41. Conclusions
Photo by Anja Jonsson / CC BY-NC-SA 2.0

42. Please complete the session
evaluation
We appreciate your feedback and insight
You may complete the session evaluation either
on paper or online via the mobile app