Single Client Access Name is a concept that makes database deployment easy on Oracle Database 11g R2 grids and complete the level of abstraction from the application perspective. Starting with the understanding of why this component is vital in the 11g grid infrastructure, this presentation walks through the main concepts of SCAN. You will learn how to plan and implement SCAN, what areas to be careful with, and how to monitor SCAN infrastructure and make sure it works as expected.
Oracle 11G SCAN: Concepts and Implementation Experience Sharing
1. Oracle Database 11g SCAN: Sharing a Successful Implementation Experience
Yury Velikanov, Sr. Oracle DBA
The Pythian Group,
Todd Carlson, Sr. Manager, DBA Team
World Wide Technology
Introduction
The Single Client Access Name (SCAN) is a concept that makes a database deployment easy on Oracle Database 11g R2
grids and completes the level of abstraction from the application perspective. While it makes an Oracle network
configuration simpler on a client side, however, an additional layer of complexity is added on the server side to support
that simplicity. This paper explains the SCAN concepts, walks you through the main steps of the process of establishing
a connection, and gives troubleshooting hints you may use to investigate SCAN-related issues.
One of the main challenges for an Oracle DBA who starts to manage a SCAN infrastructure is the fact that it consists of
traditional Oracle Net components put together in a new way. Some usual ways of managing these components can't be
used in a SCAN configuration. Oracle DBAs should adjust their "habits" to ensure successful SCAN infrastructure
operations. This paper helps the Oracle DBA to start working with the SCAN configuration and to use basic
troubleshooting techniques.
Oracle Client Connection Process Using SCAN Infrastructure
This section introduces an Oracle client's connection flow in the SCAN configuration. The section's goal is to give you a
good overview of the process of establishing a connection. It should prepare you for your first SCAN infrastructure
implementation and troubleshooting efforts.
RAC host 1
eth0
SSH Public IP (OS/Fixed)
Grid Domain Service GNS virtual IP (Cluster/Fixed) `
gns.clustgrid-prod.yourdomain.com
10.10.1.201 (fixed)
7. 3.
DB Listener Virtual IP (Cluster/DHCP)
DNS
5. clustgrid-prod.yourdomain.com NS gns.clustgrid-
prod.yourdomain.com
SCAN1 Listener SCAN1 IP (Cluster/DHCP) gns.clustgrid-prod.yourdomain.com 172.30.193.201
2.
DHCP
at least 5 free IPs
3 for SCAN IPs, 2 for VIPs, optional 2 Private IPs
nameserver
10.10.192.12 (prim) eth1
10.10.192.25 (sec) 4.
Private IP (OS/ Fixed or DHCP) 1.
Storage Single Client Access Name (SCAN)
Interconnect Traffic 6.
DB ASM scan.clustgrid-prod.yourdomain.com
Private IP (OS/ Fixed or DHCP)
nameserver
10.10.192.12 (prim) eth1
10.10.192.25 (sec)
eth0
SCAN2 Listener SCAN2 IP (Cluster/DHCP)
SCAN3 Listener SCAN3 IP (Cluster/DHCP)
DB Listener Virtual IP (Cluster/DHCP)
SSH Public IP (OS/Fixed)
RAC host 2
COLLABORATE 12 1
2. Step 1: To connect to a service (Oracle Instance or Database), the Oracle Client should know the cluster's SCAN
domain name. We use scan.clustgrid-prod.yourdomain.com as the SCAN name as an example in this paper. The SCAN name is
configured at the cluster initial setup time.
Step 2: The Oracle Client sends a request to a DNS service providing the SCAN name and receives 3 IP addresses
associated to the SCAN name. On the DNS level, the IPs could be configured statically or dynamically (GNS).
Step 3: If there is a configured GNS (Oracle Grid Naming Service) then the DNS delegates the request processing to
the GNS service. I would like to mention a few points here. First of all, if you are wondering whether to use the GNS
configuration, then the most probable answer is, "No! You don't need it." The GNS service ensures the ability to add
nodes to your cluster without specifying IP addresses for such components like VIPs and Cluster Interconnect IPs.
Those are obtained dynamically from the DHCP service during the node startup process. The GNS is used to inform the
DNS about new components and associated IP addresses. The second point is that DNS' and GNS' integration and
communications are a bit more complex than just described. I will not go into the details, to keep the description simple.
Step 4: DNS returns the 3 IP addresses associated with the SCAN domain name in a round robin fashion. This way,
each subsequent client request gets the same 3 IP addresses but in a different order.
Step 5: The Oracle Client sends a TNS request (IP/PORT/Service) to one of the SCAN listeners. It starts with the first
IP address in the list returned by DNS service. Note that clients older than Oracle 11G don't know how to work with 3
IPs returned by DNS, and use only the first IP returned. If this IP for one reason or another isn't available, the
connection attempt fails. Starting from version 11G, the Oracle Client uses the second and the third IP addresses if the
first connection attempt fails.
Step 6: The SCAN listener knows about all Oracle services served by instances running on each cluster's nodes. If there
are multiple instances that could serve the incoming connection, the SCAN listeners could be configured to balance
incoming connections' load. The SCAN listener forwards the Oracle Client's connection (providing IP and PORT) to a
local DB listener running on the less loaded node.
Step 7: The Oracle Client connects to the local DB Listener and requests a connection to the Oracle Instance. The local
DB listener uses a traditional process to fork a database foreground process, and connects the Oracle Client with that
process.
Oracle Instance Registration Process in the SCAN Infrastructure
Now that we have introduced the client's connectivity flow, let's have a look at how an Oracle Instance is registered in
the SCAN infrastructure from the perspective of the backend. Let's discuss the main components and configuration
items involved in the process. An Oracle database instance should get registered within the SCAN infrastructure before
it can start accepting incoming clients' connections.
COLLABORATE 12 2
3. DNS resolves
scan-cluster_a.mycomany.com
to 3 SCAN IP addresses
SCAN (remote) listeners register
and distribute information about
local listeners serving a SERVICE
LISTENER LISTENER LISTENER
2.
Oracle Instance:
1. registers SERVICES it running in
LOCAL listeners
LISTENER
2. registers LOCAL listeners serving remote_listener (default)
a service in SCAN listeners
1. LISTENER_ERP
(custom)
local_listener Node 2
service_name
Node 1
First of all, it is very important to understand the difference between the remote_listener and local_listener init.ora
parameters. These parameters play important and slightly different roles in the instance SCAN registration process.
- The Oracle Instance (in particular PMON - process monitor) uses the list of listeners' addresses specified by
the local_listener parameter to contact each listener from the list and "says": "Hey Listener! I am instance X. I run
the following services. I am the local instance and run on the same host that you". From now on, if the local
listener receives a new session request to connect to one of the services the instance provides it spawns an
Oracle foreground process, connects it to the instance X and passes a TCP/IP socket to the client session.
Please note that the listener doesn't check if the instance is local or not by the time the PMON process
provides the information. It just blindly believes what was told. Therefore the local_listener parameter must point
only to the local listener. Otherwise, some of the incoming connections may fail.
- During the next step, the PMON process reads the remote_listener parameter and connects to each of the
listeners specified by the parameter. It informs the remote listeners about any services and the corresponding
local listeners the instance runs. It is important to mention that the SCAN listeners expect an Oracle Instance
only to report local listeners. SCAN listeners don’t check the information sent by the PMON at the time of
registration. SCAN listeners rely on the information provided and start forwarding incoming connections to the
local listeners immediately. If the listener isn't the local listener, connection attempts fails.
- To reiterate what was said: the SCAN and local listeners don’t check the information provided by the Oracle
Instance (PMON). The listeners rely on the information provided and "blindly" send sessions to other (local)
listeners, or try to spawn a new Oracle process to serve the incoming connection. Therefore it is important to
make sure the right values are set for both remote_listener and local_listener parameters.
The Oracle Instance (PMON) updates remote and local listeners on services it is running during the following events:
- at start up and shutdown at regular intervals (up to 60 seconds)
COLLABORATE 12 3
4. - at the time the "ALTER SYSTEM REGISTER" command is issued
The following is Oracle Instance's short registration flow:
Step 1: Straight after a startup and on a regular basis thereafter, the Oracle Instance reads the service_name init.ora
parameter to determine what services it should provide to the outside world. We should mention that in the 11G
infrastructure, information about Oracle services is managed by Oracle Clusterware processes. Therefore, don’t try to
change this information using the "ALTER SYSTEM..." command. Use the srvctl utility instead.
Step 2: The Oracle Instance reads the local_listener parameter and registers the list of services obtained during the first
step within the local listener.
Step3. The Oracle Instance reads the remote_listener parameter and registers the local listener along with services it is
serving within remote (SCAN) listeners.
Oracle Net: Easy Connect
Before you start implementing or troubleshooting the SCAN infrastructure, we strongly suggest that you get comfortable
with the Oracle Net Easy Connect naming method. This is one of the many Oracle Net naming methods. A naming
method is nothing more than a way to describe a network configuration that an Oracle client should know to get
connected to an Oracle Instance. A more commonly used method is the local naming method, where you edit the
tnsnames.ora configuration file to specify the listener's address and services to connect to. Easy Connect is a less popular
method. In the case of Easy Connect, the network configuration isn't stored in a separate configuration file, but provided
in one relatively short connect string. This method is used extensively in the SCAN configuration; therefore it is
important to understand it well.
We suggest that you read through the "Using the Easy Connect Naming Method" section of the "8 Configuring Naming
Methods" chapter of the "Oracle® Database Net Services Administrator's Guide" book from the standard Oracle
11GR2 Documentation Set.
As an example, you may get confused by looking at the initial value of the remote_listener init.ora parameter that is set to
"scan.clustgrid-prod.yourdomain.com" (or your cluster SCAN name). This is nothing but the following
tnsnames.ora configuration:
LISTNER=(ADDRESS = (PROTOCOL = TCP)(HOST = scan.clustgrid-prod.yourdomain.com)(PORT = 1521))
SCAN Troubleshooting Hints
If you have read the previous sections carefully, you may find this section to be redundant to some extent, as most of the
information provided below is related to the components discussed already.
service names - check the service_name init.ora parameter to start with and make sure that it corresponds to the configuration
you expect to see. You can use the following traditional command to retrieve the instance'-side information for the
troubleshooting purposes. Compare it with the next command output and your oracle client-side configuration. Please
notice that you should check each node in your cluster.
SQL> show parameter service_name
NAME TYPE VALUE
-------------------- ----------- --------------------------------------------------
service_names string DEVERP_CDC.GGT.COM, SYS$APPLSYS.WF_CONTROL.DEVERP.
WORLD, SYS$STREAMS_ADMIN.CDC$Q_ERP.DEVERP.WORLD, D
EVERP_WEBM.GGT.COM, DEVERP_W2T_B2B.GGT.COM, DEVERP
_RFUI.GGT.COM, DEVERP_IBI.GGT.COM, DEVERP_GENERAL.
W2T.COM, DEVERP_BI.GGT.COM, DEVERP_APEX.GGT.COM, D
EVERP_10g, DEVERP1, DEVERP
SQL>
COLLABORATE 12 4
5. Do not modify the service_name parameter using the "ALTER SYSTEM SET ..." command. Use the srvctl utility to
retrieve and modify service-related configuration instead. The cluster sets the service_names parameter dynamically at the
instance's start up time and each time it is modified. The following example demonstrates how you can retrieve the
services' configuration using the srvctl utility.
srvctl config service -d <DB Name>
…
Service name: DEVERP_APEX.GGT.COM
Service is enabled
Failover type: NONE
Preferred instances: DEVERP1
Available instances: DEVERP1,DEVERP2,DEVERP3,DEVERP4,DEVERP5,DEVERP6
…
local_listener and remote_listener parameters - We can't stress enough: the local_listener parameter must only point to the local
(running on the same nodes as the instance it is related to) listener address, and the remote_listener parameter should only
point to the SCAN listeners. The other important point to mention is that the instance won't start if there is a syntax or
misconfiguration issue in the parameters' settings. For example, typically you will find that the remote_listener parameter
points to the SCAN listeners using the Oracle Net: Easy Connect syntax (remote_listener=scan.clustgrid-prod.yourdomain.com).
If you forget to include an (or at some point in time remove) NAMES.DIRECTORY_PATH=(..., EZCONNECT,...)
line in the sqlnet.ora file, the instance will fail to start next time you try to restart it.
Oracle Listeners - Make sure that listeners in your SCAN infrastructure are running under the grid OS user. One of the
typical mistakes a traditional Oracle DBA makes when connecting to the 11GR2 cluster for the first time is starting the
local listener under the oracle OS user. The listener starts listening on all IP addresses available on the server, including
the entire SCAN and VIP IP addresses, on the default 1521 TCP port. Unfortunately, there is nothing preventing the
Oracle DBA from doing this. However, it introduces a lot of different issues in the SCAN configuration. As with the
services, listeners should be managed using the srvctl cluster management utility. Starting from the 11GR2 version, the
listener.ora file is managed by the cluster itself. Therefore, try to avoid editing the listener.ora file manually unless it is
absolutely necessary. For troubleshooting purposes, you still can use the lsnrctl status <name e.g. LISTENER_SCAN1 or
LISTENER > or lsnrctl status <name e.g. LISTENER_SCAN1 or LISTENER > commands to make sure that the
expected services are registered under each listener.
DNS - as Oracle clients become dependent on the SCAN domain name resolution, the DNS service becomes a critical
component of the connectivity process. You should make sure that the DNS returns the expected list of the SCAN IP
addresses each time a SCAN resolution request is coming through. Some old Windows versions may cache in local
memory domain names that have been resolved before. You should keep this in mind while troubleshooting the SCAN
connectivity issues. The other important verification you should run through is to make sure that the secondary DNS
service is configured in the same way in the SCAN context. Sometimes you may find that some of a cluster's users can't
connect to its services. This is a good indicator that there may be a configuration issue with the secondary DNS.
Conclusion
The information provided in this paper should be sufficient to get comfortable with a typical SCAN configuration and to
start troubleshooting related issues successfully. While it isn't a comprehensive technology description, it is based on
several real life implementation projects and covers typical problems Oracle DBAs face starting to work with the SCAN
configuration for the first time.
COLLABORATE 12 5