Sun Cluster 3.1 Data Service for Oracle Parallel Server/Real Application Clusters

Installing and Configuring Sun Cluster Support for Oracle Parallel Server/Real Application Clusters

This chapter describes the steps to install and configure Sun Cluster Support for Oracle Parallel Server/Real Application Clusters on your Sun Cluster nodes. This chapter contains the following procedures.

Installing and Configuring Sun Cluster Support for Oracle Parallel Server/Real Application Clusters

The following table lists the sections that describe the installation and configuration tasks.

Table 1–1 Task Map: Installing and Configuring Sun Cluster Support for Oracle Parallel Server/Real Application Clusters


Task	For Instructions, Go To …
Understand pre-installation considerations and special requirements	Overview Special Requirements
(Optional) Install volume management software	Installing Volume Management Software With Sun Cluster Support for Oracle Parallel Server/Real Application Clusters
Install data service packages	Installing Sun Cluster Support for Oracle Parallel Server/Real Application Clusters Packages
Install the UNIX Distributed Lock Manager and Oracle software	Installing the Oracle Software

Overview

Before you install the data service, consider the points listed in the following sections.

An Atypical Data Service

Sun Cluster Support for Oracle Parallel Server/Real Application Clusters is an atypical Sun Cluster high availability data service. This data service does not provide automatic failover or fault monitoring because the Oracle Parallel Server/Real Application Clusters software already provides this functionality. This data service is a set of packages that, when installed, enables Oracle Parallel Server/Real Application Clusters to run on Sun Cluster nodes.

The Oracle Parallel Server/Real Application Clusters software is not registered with or managed by the Sun Cluster Resource Group Manager (RGM). However, Sun Cluster Support for Oracle Parallel Server/Real Application Clusters is similar to other data services in that it depends on the RGM to query cluster information.

You can configure Oracle Parallel Server/Real Application Clusters to use the shared-disk architecture of the Sun Cluster software. In this configuration, a single database is shared among multiple instances of the Oracle Parallel Server/Real Application Clusters software that access the database concurrently. The UNIX Distributed Lock Manager (Oracle UDLM) controls access to shared resources between cluster nodes. Typically, these shared resources contain process and database instance membership information. The internal DLM that resides in each Oracle database instance controls access to shared resources between cluster nodes. The shared resources are typically disk blocks and other shared resources such as transaction locks. See the Oracle documentation for details about the globally shared resources that the internal DLM manages.

Pre-Installation Considerations

Before you begin the installation, note the following pre-installation considerations.

Sun Cluster Support for Oracle Parallel Server/Real Application Clusters requires a functioning cluster with the initial cluster framework already installed. See the Sun Cluster 3.1 Software Installation Guide for details about initial installation of cluster software.
Decide which volume manager you will use—either VERITAS Volume Manager (VxVM) or RAID Manager.
Verify that you have obtained the appropriate licenses for your software. If, for example, you use VxVM, run the vxlicense -p check command to ensure that you have installed a valid license for the Volume Manager cluster feature. If you install your licenses incorrectly or incompletely, the nodes might abort.
Check with a Sun Enterprise Services representative for the current supported topologies for Sun Cluster Support for Oracle Parallel Server/Real Application Clusters, cluster interconnect, volume manager, and hardware configurations.
Ensure that you have installed all of the applicable software patches for Solaris, Sun Cluster, Oracle, and your volume manager. The Oracle UDLM consists of two packages—ORCLudlm, which Oracle supplies, and SUNWudlm, which Sun supplies. You must install both of these packages. If you need to install any Sun Cluster Support for Oracle Parallel Server/Real Application Clusters patches, you must apply these patches after you install the data service.
You should install the Oracle binaries locally on each node in the cluster, rather than globally on the cluster file system, to avoid overwrite issues with configuration files and logs. However, if you plan to install the Oracle binaries on the cluster file system, contact Oracle to validate the support of this configuration. Additionally, see the Oracle documentation for configuration specifics.

Special Requirements

This section lists special requirements for Sun Cluster Support for Oracle Parallel Server/Real Application Clusters.

32-Bit or 64-Bit Mode

Before you decide on which architecture to use for the Oracle components (Oracle UDLM and RDBMS), note the following points.

The architecture of both Oracle components must match. For example, if you have 64-bit architecture for your Oracle UDLM, you must have 64-bit architecture for your RDBMS.
If you have 32-bit architecture for your Oracle components, you can boot the node on which the components reside in either 32-bit or 64-bit mode. However, if you have 64-bit architecture for your Oracle components, you must boot the node on which the components reside in 64-bit mode.
You must use the same architecture when you boot all of the nodes. For example, if you boot one node to use 32-bit architecture, you must boot all of the nodes to use 32-bit architecture.

Log File Locations

The following list shows the locations of the data service log files.

Current log – /var/cluster/ucmm/ucmm_reconf.log
Previous logs – /var/cluster/ucmm/ucmm_reconf.log.0 (0,1,...) If you cannot find the Oracle log files at this location, contact Oracle support. This location is dependent on the Oracle UDLM package.
Oracle UDLM logs – /var/cluster/ucmm/dlm_nodename/logs
Oracle UDLM core files – /var/cluster/ucmm/dlm_nodename/cores

Node Failures and Recovery Procedures

If a node fails in an Oracle Parallel Server/Real Application Clusters environment, you can configure Oracle clients to reconnect to the surviving server without the use of the IP failover that Sun Cluster failover data services use. The Sun Cluster 3.1 Concepts Guide document describes this failover process. In an Oracle Parallel Server/Real Application Clusters environment, multiple Oracle instances cooperate to provide access to the same shared database. The Oracle clients can use any of the instances to access the database. Thus, if one or more instances have failed, clients can connect to a surviving instance and continue to access the database.

Note –

If a node fails, boot the node into maintenance mode to correct the problem. See the Sun Cluster 3.1 System Administration Guide for more information.

Note –

When you install this data service, ensure that you complete all steps of all procedures that precede How to Install the Oracle RDBMS Software and Create Your Oracle Database before you reboot the nodes. Otherwise, the nodes will panic. If the nodes panic, you must boot into maintenance mode to correct the problem.

Using the Oracle Parallel Fail Safe/Real Application Clusters Guard Option With Sun Cluster 3.1

Note the following points if you plan to use the Oracle Parallel Fail Safe/Real Application Clusters Guard option with Sun Cluster 3.1.

If you use this option, before you install Sun Cluster 3.1, you must consider the following special requirement. Hostnames that you use in your cluster cannot contain special characters. You cannot change the hostname after you install Sun Cluster 3.1. See the Oracle documentation for more information about this special requirement and any others before you install Sun Cluster 3.1.
Refer to the Oracle documentation for installation, administration and operation of this product option.
Do not use Sun Cluster commands to manipulate the state of resources that Oracle Parallel Fail Safe/Real Application Clusters Guard installs. To do so might result in failures. Do not rely on the Sun Cluster commands to query the state of the resources that Oracle Parallel Fail Safe/Real Application Clusters Guard installs. This state may not reflect the actual state. To check the state of the Oracle Parallel Fail Safe/Real Application Clusters Guard, use the commands that Oracle supplies.

Installing Volume Management Software With Sun Cluster Support for Oracle Parallel Server/Real Application Clusters

For Sun Cluster Support for Oracle Parallel Server/Real Application Clusters disks, use the following configurations.

VxVM with the cluster feature enabled
Hardware RAID support

How to Use VxVM

To use the VxVM software with Sun Cluster Support for Oracle Parallel Server/Real Application Clusters, perform the following tasks.

Obtain a license for the Volume Manager cluster feature in addition to the basic VxVM license.

See your VxVM documentation for more information about VxVM licensing requirements.

Caution –
Failure to correctly install the license for the Volume Manager cluster feature might result in a panic when you install Oracle Parallel Server/Real Application Clusters support. Before you install the Oracle Parallel Server/Real Application Clusters packages, run the vxlicense -p check command to ensure that you have installed a valid license for the Volume Manager cluster feature.

Install and configure the VxVM software on the cluster nodes.

See the VxVM appendix in the Sun Cluster 3.1 Software Installation Guide and the VxVM documentation for more information.

Use VERITAS commands to create a separate shared disk group for the Oracle Parallel Server/Real Application Clusters database to use (see your VxVM documentation for details on shared disk groups).

Before you create the shared disk group, note the following points.
- Do not register the shared disk group within the cluster.
- Do not create any file systems in the shared disk group because only the raw data file will use this disk group.
- Create volumes as the gen use type.
- Disks that you add to the shared disk group must be directly attached to all of the cluster nodes.
- Ensure that your VxVM license is current. If your license expires, the node will panic.

How to Use Hardware RAID Support

You can use Sun Cluster Support for Oracle Parallel Server/Real Application Clusters with hardware RAID support.

For example, you can use Sun StorEdge^TM A3500/A3500FC disk arrays with hardware RAID support and without VxVM software. To do so, configure raw device IDs (/dev/did/rdsk*) on top of the disk arrays' logical unit numbers (LUNs). To set up the raw devices for Oracle Parallel Server/Real Application Clusters on a cluster that uses StorEdge A3500/A3500FC disk arrays with hardware RAID, perform the following steps.

Create LUNs on the disk arrays.

See the Sun Cluster 3.1 Hardware Guide for information on how to create LUNs.

After you create the LUNs, run the format(1M) command to partition the disk arrays' LUNs into as many slices as you need.

The following example lists output from the format command.

# format

0. c0t2d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
   /sbus@3,0/SUNW,fas@3,8800000/sd@2,0
1. c0t3d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
   /sbus@3,0/SUNW,fas@3,8800000/sd@3,0
2. c1t5d0 <Symbios-StorEDGEA3000-0301 cyl 21541 alt 2 hd 64 sec 64>
   /pseudo/rdnexus@1/rdriver@5,0
3. c1t5d1 <Symbios-StorEDGEA3000-0301 cyl 21541 alt 2 hd 64 sec 64>
   /pseudo/rdnexus@1/rdriver@5,1
4. c2t5d0 <Symbios-StorEDGEA3000-0301 cyl 21541 alt 2 hd 64 sec 64>
   /pseudo/rdnexus@2/rdriver@5,0
5. c2t5d1 <Symbios-StorEDGEA3000-0301 cyl 21541 alt 2 hd 64 sec 64>
   /pseudo/rdnexus@2/rdriver@5,1
6. c3t4d2 <Symbios-StorEDGEA3000-0301 cyl 21541 alt 2 hd 64 sec 64>
   /pseudo/rdnexus@3/rdriver@4,2

Note –

If you use slice 0, do not start the partition at cylinder 0.

Run the scdidadm(1M) command to find the raw device ID (DID) that corresponds to the LUNs that you created in Step 1.

The following example lists output from the scdidadm -L command.

# scdidadm -L

1        phys-schost-1:/dev/rdsk/c0t2d0   /dev/did/rdsk/d1
1        phys-schost-2:/dev/rdsk/c0t2d0   /dev/did/rdsk/d1
2        phys-schost-1:/dev/rdsk/c0t3d0   /dev/did/rdsk/d2
2        phys-schost-2:/dev/rdsk/c0t3d0   /dev/did/rdsk/d2
3        phys-schost-2:/dev/rdsk/c4t4d0   /dev/did/rdsk/d3
3        phys-schost-1:/dev/rdsk/c1t5d0   /dev/did/rdsk/d3
4        phys-schost-2:/dev/rdsk/c3t5d0   /dev/did/rdsk/d4
4        phys-schost-1:/dev/rdsk/c2t5d0   /dev/did/rdsk/d4
5        phys-schost-2:/dev/rdsk/c4t4d1   /dev/did/rdsk/d5
5        phys-schost-1:/dev/rdsk/c1t5d1   /dev/did/rdsk/d5
6        phys-schost-2:/dev/rdsk/c3t5d1   /dev/did/rdsk/d6
6        phys-schost-1:/dev/rdsk/c2t5d1   /dev/did/rdsk/d6

Use the DID that the scdidadm output identifies to set up the raw devices.

For example, the scdidadm output might identify that the raw DID that corresponds to the disk arrays' LUNs is d4. In this instance, use the /dev/did/rdsk/d4sN raw device, where N is the slice number.

Installing Sun Cluster Support for Oracle Parallel Server/Real Application Clusters Packages

Use one of the following procedures to install the packages that you need to run Sun Cluster Support for Oracle Parallel Server/Real Application Clusters.

If you use VxVM as your volume manager, perform the procedure How to Install Sun Cluster Support for Oracle Parallel Server/Real Application Clusters Packages With VxVM.
If you use hardware RAID support, perform the procedure How to Install Sun Cluster Support for Oracle Parallel Server/Real Application Clusters Packages With Hardware RAID.

How to Install Sun Cluster Support for Oracle Parallel Server/Real Application Clusters Packages With VxVM

To complete this procedure, you need the Sun Cluster 3.1 CD-ROM. Perform this procedure on all of the cluster nodes that can run Sun Cluster Support for Oracle Parallel Server/Real Application Clusters.

Note –

Due to the preparation that is required prior to installation, the scinstall(1M) utility does not support automatic installation of the data service packages.

Load the Sun Cluster 3.1 CD-ROM into the CD-ROM drive.

Become superuser.

Change the current working directory to the directory that contains the packages for the version of the Solaris operating environment that you are using.
- If you are using Solaris 8, run the following command:
  # cd /cdrom/suncluster_3_1/SunCluster_3.1/Sol_8/Packages
- If you are using Solaris 9, run the following command:
  # cd /cdrom/suncluster_3_1/SunCluster_3.1/Sol_9/Packages

On all of the nodes, run the following command to install the data service packages.
# pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWcvmr SUNWcvm

Caution –

Before you reboot the nodes, you must ensure that you have correctly installed and configured the Oracle UDLM software (How to Install the Oracle UDLM Software). Also verify that you have correctly installed your volume manager packages. If you use VxVM, check that you have installed the software and that the license for the VxVM cluster feature is valid. Otherwise, a panic will occur.

Where to Go From Here

Go to Installing the Oracle Software to install the Oracle UDLM and Oracle software.

How to Install Sun Cluster Support for Oracle Parallel Server/Real Application Clusters Packages With Hardware RAID

Note –

Due to the preparation that is required prior to installation, the scinstall(1M) utility does not support automatic installation of the data service packages.

Load the Sun Cluster 3.1 CD-ROM into the CD-ROM drive.

Become superuser.

Change the current working directory to the directory that contains the packages for the version of the Solaris operating environment that you are using.
- If you are using Solaris 8, run the following command:
  # cd /cdrom/suncluster_3_1/SunCluster_3.1/Sol_8/Packages
- If you are using Solaris 9, run the following command:
  # cd /cdrom/suncluster_3_1/SunCluster_3.1/Sol_9/Packages

On all of the nodes, run the following command to install the data service packages.
# pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWschwr

Caution –

Where to Go From Here

Go to Installing the Oracle Software to install the Oracle UDLM and Oracle software.

Installing the Oracle Software

Use the procedures in this section to perform the following tasks.

Prepare the nodes.
Install the Oracle Oracle UDLM software.
Install the Oracle RDBMS software.

How to Prepare the Sun Cluster Nodes

For the Oracle UDLM software to run correctly, sufficient shared memory must be available on all of the cluster nodes. See the Oracle Parallel Server/Real Application Clusters CD-ROM for all of the installation instructions. To prepare the Sun Cluster nodes, check that you have completed the following tasks.

You have correctly set up the Oracle user account and database administration group.
You have configured the system to support the shared memory requirements of the Oracle UDLM.

Note –

Perform the following steps as superuser on each cluster node.

On each node, create an entry for the database administrator group in the /etc/group file, and add potential users to the group.

This group normally is named dba. Verify that root and oracle are members of the dba group, and add entries as necessary for other DBA users. Verify that the group IDs are the same on all of the nodes that run Sun Cluster Support for Oracle Parallel Server/Real Application Clusters. For example, add the following entry to the /etc/group file.
dba:*:520:root,oracle
You can make the name service entries in a network name service (for example, NIS or NIS+) so that the information is available to the data service clients. You can also make entries in the local /etc files to eliminate dependency on the network name service.

On each node, create an entry for the Oracle user ID (the group and password) in the /etc/passwd file, and run the pwconv(1M) command to create an entry in the /etc/shadow file.

This Oracle user ID is normally oracle. For example, add the following entry to the /etc/passwd file.
# useradd -u 120 -g dba -d /Oracle-home oracle
Ensure that the user IDs are the same on all of the nodes that run Sun Cluster Support for Oracle Parallel Server/Real Application Clusters.

Where to Go From Here

After you set up the cluster environment for Oracle Parallel Server/Real Application Clusters, go to How to Install the Oracle UDLM Software to install the Oracle UDLM software on each cluster node.

How to Install the Oracle UDLM Software

Note –

You must install the Oracle UDLM software on the local disk of each node.

Caution –

Before you install the Oracle UDLM software, ensure that you have created entries for the database administrator group and the Oracle user ID. See How to Prepare the Sun Cluster Nodes for details.

Become superuser on a cluster node.

Install the Oracle UDLM software.

See the appropriate Oracle Parallel Server/Real Application Clusters installation documentation for instructions.

Note –
Ensure that you did not receive any error messages when you installed the Oracle UDLM packages. If an error occurred during package installation, correct the problem before you install the Oracle UDLM software.

Update the /etc/system file with the shared memory configuration information.

You must configure these parameters based on the resources that are available in the cluster. Decide on the appropriate values, but ensure that the Oracle UDLM can create a shared memory segment according to its configuration requirements.

The following example shows entries to configure in the /etc/system file.

*SHARED MEMORY/ORACLE
set shmsys:shminfo_shmmax=268435456
set semsys:seminfo_semmap=1024
set semsys:seminfo_semmni=2048
set semsys:seminfo_semmns=2048
set semsys:seminfo_semmsl=2048
set semsys:seminfo_semmnu=2048
set semsys:seminfo_semume=200
set shmsys:shminfo_shmmin=200
set shmsys:shminfo_shmmni=200
set shmsys:shminfo_shmseg=200
forceload: sys/shmsys
forceload: sys/semsys
forceload: sys/msgsys

Shut down and reboot all of the nodes.

Caution –
Before you reboot, you must ensure that you have correctly installed and configured the Oracle UDLM software. Also verify that you have correctly installed your volume manager packages. If you use VxVM, check that you have installed the software and that the license for the VxVM cluster feature is valid. Otherwise, a panic will occur.
1. From one node only—such as phys-schost-1—run the following command to shut down the cluster.
  phys-schost-1# scshutdown -g0 -y
  See the scshutdown(1M) man page for details.
2. Reboot each node into cluster mode.
  ok boot

Where to Go From Here

After you have installed the Oracle UDLM software on each cluster node, go to How to Install the Oracle RDBMS Software and Create Your Oracle Database to install the Oracle RDBMS software.

How to Install the Oracle RDBMS Software and Create Your Oracle Database

See your Oracle Parallel Server/Real Application Clusters installation documentation for instructions on how to install the RDBMS software and create your Oracle database.