Sun Cluster Data Service for Oracle RAC Guide for Solaris OS

Chapter 6 Modifying an Existing Configuration of Sun Cluster Support for Oracle RAC

This chapter explains how to modify an existing configuration of Sun Cluster Support for Oracle RAC.

Overview of Tasks for Modifying an Existing Configuration of Sun Cluster Support for Oracle RAC

Table 6–1 summarizes the administration tasks for Sun Cluster Support for Oracle RAC.

Perform these tasks whenever they are required.

Table 6–1 Tasks for Modifying an Existing Configuration of Sun Cluster Support for Oracle RAC


Task	Instructions
Modifying online the resource for a scalable device group	Modifying Online the Resource for a Scalable Device Group
Upgrade Sun Cluster Support for Oracle RAC to upgrade your cluster to use a new version of Sun Cluster	Upgrading Sun Cluster Support for Oracle RAC
Extend an existing configuration of Sun Cluster Support for Oracle RAC	Extending an Existing Configuration of Sun Cluster Support for Oracle RAC
Remove Sun Cluster Support for Oracle RAC	Removing Sun Cluster Support for Oracle RAC

Modifying Online the Resource for a Scalable Device Group

Modifying online the resource for a scalable device group involves changing the list of logical volumes that are to be monitored. The LogicalDeviceList extension property of the SUNW.ScalDeviceGroup resource type specifies the list of logical volumes in a global device group that are to be monitored.

How to Modify Online the Resource for a Scalable Device Group

Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

Modify the LogicalDeviceList extension property of the ScalDeviceGroup resource.
- To add device groups to a ScalDeviceGroup resource, type the following command:
  # clresource set -p LogicalDeviceList+=logical-device-listscal-mp-rs
  The addition of the logical volume is effective immediately.
- To remove device groups from a ScalDeviceGroup resource, type the following command:
  # clresource set -p LogicalDeviceList-=logical-device-listscal-mp-rs
  The removal of the logical volume is effective immediately.

Upgrading Sun Cluster Support for Oracle RAC

If you upgrade Sun Cluster foundation software to release 3.2, you must also upgrade the Sun Cluster Support for Oracle RAC software. For more information, see Upgrading Sun Cluster Software, in Sun Cluster Software Installation Guide for Solaris OS.

Your existing configuration of Sun Cluster Support for Oracle RAC might not include the RAC framework resource group. In this situation, you must register and configure the RAC framework resource group after upgrading to Sun Cluster 3.2. Otherwise, Oracle RAC cannot run with Sun Cluster. For more information, see Registering and Configuring the RAC Framework Resource Group.

Sun Cluster 3.2 introduces several new features for Sun Cluster Support for Oracle RAC. Information about how to upgrade an existing configuration to use these new features is provided in the subsections that follow.

Note –

The SUNW.oracle_rac_server resource type and the SUNW.oracle_listener resource type are unchanged in Sun Cluster 3.2. If you choose to continue to use the SUNW.oracle_rac_server resource type and the SUNW.oracle_listener resource type for Oracle 9i RAC database instances, no upgrade of these resource types is required.

Upgrading Resources in the RAC Framework Resource Group

Changes to resource types of resources in the RAC framework for Sun Cluster 3.2 are summarized in the following table.

Table 6–2 Changes to RAC Framework Resource Types


Resource Type	Changes
`SUNW.rac_cvm`	Extension property `reservation_timeout` removed.
`SUNW.rac_framework`	Extension property `reservation_timeout` added.
`SUNW.rac_hwraid`	Removed from Sun Cluster 3.2.
`SUNW.rac_svm`	Extension property `reservation_timeout` removed.
`SUNW.rac_udlm`	Tunability of the `oracle_config_file` extension property changed from any time to when disabled. Minimum allowed value of `port` extension property reduced from 1024 to 1.

Upgrade these resource types if you are upgrading to version 3.2 from an earlier version of Sun Cluster Support for Oracle RAC.

For general instructions that explain how to upgrade a resource type, see Upgrading a Resource Type in Sun Cluster Data Services Planning and Administration Guide for Solaris OS.

The information that you require to complete the upgrade of the RAC framework resource types is provided in the subsections that follow.

Information for Registering the New Versions of RAC Framework Resource Types

The resource type versions of changed RAC framework resource types are listed in the following table for version 3.1 and version 3.2 of Sun Cluster. This table also lists the name of the Resource Type Registration (RTR) file of each changed resource type.

Table 6–3 Information for Registering the New Version of RAC Framework Resource Types


Resource Type	Resource Type Version in Sun Cluster 3.1	Resource Type Version in Sun Cluster 3.2	RTR File
`SUNW.rac_cvm`	1.0	2	`/usr/cluster/lib/ucmm/rt/rac_cvm/etc/SUNW.rac_cvm`
`SUNW.rac_framework`	1.0	2	`/usr/cluster/lib/ucmm/rt/rac_framework/etc/SUNW.rac_framework`
`SUNW.rac_svm`	1.0	2	`/usr/cluster/lib/ucmm/rt/rac_svm/etc/SUNW.rac_svm`
SPARC: `SUNW.rac_udlm`	1.0	2	`/usr/cluster/lib/ucmm/rt/rac_udlm/SUNW.rac_udlm`

To determine the version of the resource type that is registered, use the following command:

# clresourcetype show resource-type

resource-type: Specifies the resource type whose version you are determining.

Information for Migrating Existing Instances of RAC Framework Resource Types

The information that you require to edit each instance of RAC framework resource types is listed in the following table.

Table 6–4 Information for Migrating Existing Instances of RAC Framework Resource Types


Resource Type	Value of `type_version` Property	Possible Migration Times	Changes to Extension Properties
`SUNW.rac_cvm`	2	Any time	Extension property `reservation_timeout` removed.
`SUNW.rac_framework`	2	Any time	Extension property `reservation_timeout` added.
`SUNW.rac_svm`	2	Any time	Extension property `reservation_timeout` removed.
SPARC: `SUNW.rac_udlm`	2	Any time	Tunability of the `oracle_config_file` extension property changed from any time to when disabled. Minimum allowed value of `port` extension property reduced from 1024 to 1.

The following example shows a command for editing an instance of the SUNW.rac_udlm resource type.

Note –

The example shows the setting of an extension property that can be set only when the resource is disabled. The step for disabling the resource is not shown in the example.

Example 6–1 Editing an Instance of the `SUNW.rac_udlm` Resource Type

# clresource set -p type_version=2 \
-p port=100 rac_udlm-rs

This command edits a SUNW.rac_framework resource as follows:

The SUNW.rac_udlm resource is named rac_udlm.
The type_version property of this resource is set to 2.
The communications port number that the Oracle UDLM uses is set to 100.

How to Preserve a Nondefault Value of the `reservation_timeout` Extension Property

The reservation_timeout extension property specifies the timeout value in seconds for the reservation step of a reconfiguration of Sun Cluster Support for Oracle RAC.

In Sun Cluster release 3.1, reservation_timeout is an extension property of the following resource types:

SUNW.rac_cvm
SUNW.rac_hwraid
SUNW.rac_svm

In Sun Cluster release 3.2, reservation_timeout is an extension property only of the SUNW.rac_framework resource type. This property is removed from the SUNW.rac_cvm resource type and the SUNW.rac_svm resource type. The SUNW.rac_hwraid resource type is removed from Sun Cluster release 3.2.

A nondefault value of the reservation_timeout extension property might be set for an existing resource of type SUNW.rac_hwraid, SUNW.rac_cvm, or SUNW.rac_svm. If you require this value after you upgrade to Sun Cluster release 3.2, perform this task to preserve the value.

Before you migrate or remove the resource for which the reservation_timeout extension property is set, obtain the value of this property.
# clresource show -p reservation_timeout -t resource-type
resource-type

Specifies the resource type of the resource for which the reservation_timeout extension property is set, namely, SUNW.rac_cvm, SUNW.rac_hwraid, or SUNW.rac_svm.

When you migrate the existing instance of the SUNW.rac_framework resource type, set the reservation_timeout extension property to the value that you obtained in Step 1.
# clresource set -p type_version=version \ -p reservation_timeout=timeout rac-framework-rs
version

Specifies the value of the type_version property for the version of SUNW.rac_framework to which you are migrating the instance. In Sun Cluster release 3.2, version is 2.

timeout

Specifies the value to which you are setting the reservation_timeout extension property. This value is the value that you obtained in Step 1.

rac-framework-rs

Specifies the name of the resource of type SUNW.rac_framework on your cluster. If the scsetup utility of Sun Cluster release 3.1 was used to create the RAC framework resource group, this resource is named rac_framework.

Example 6–2 Preserving a Nondefault Value of the `reservation_timeout` Extension Property

This example shows the commands for preserving a nondefault value of the reservation_timeout extension property that was set for a resource of type SUNW.rac_svm.

Before the migration of the resource of type SUNW.rac_svm, the following command is run to obtain the value of the reservation_timeout extension property:

# clresource show -p reservation_timeout -t SUNW.rac_svm

=== Resources ===                              

Resource:                                       rac-svm-rs

  --- Standard and extension properties ---    

  Reservation_timeout:                          350
    Class:                                         extension
    Description:                                   Timeout (in seconds) for the 
reservation step of Sun Cluster framework for RAC
    Per-node:                                      False
    Type:                                          int

During the migration of the resource of type SUNW.rac_framework, the following command is run to set properties as follows:

The type_version property is set to 2.
The reservation_timeout extension property is set to 350 seconds.

# clresource set -p type_version=2 \
-p reservation_timeout=350 rac_framework

How to Remove the Resource of Type `SUNW.rac_hwraid`

In Sun Cluster release 3.2, the SUNW.rac_hwraid resource type is obsolete and is not supplied with this release. Any resource of type SUNW.rac_hwraid is not required after an upgrade to Sun Cluster release 3.2. If a resource of this obsolete type is configured on your cluster, remove the resource.

Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization.

Delete the instance of SUNW.rac_hwraid, if any.
# clresource delete -F -t SUNW.rac_hwraid +
Note –
If no resource of type SUNW.rac_hwraid is configured on your cluster, a message to this effect is displayed. Ignore this message.

Adding Storage Resources for Oracle Files

Sun Cluster 3.2 introduces resource types that provide fault monitoring and automatic fault recovery for global device groups and file systems.

If you are using global device groups or shared file systems for Oracle files, add storage resources to manage the availability of the storage on which the Oracle software depends.

Before adding storage resources for Oracle files, ensure that you have upgraded resources in the RAC framework resource group. For more information, see Upgrading Resources in the RAC Framework Resource Group.

For detailed instructions for adding storage resources for Oracle files, see Registering and Configuring Storage Resources for Oracle Files.

Follow these instructions only if you are adding storage resources to an existing configuration of storage management schemes. If you are extending an existing configuration of Sun Cluster Support for Oracle RAC by supporting additional storage management schemes, see the following sections:

Adding Resources for Interoperation With Oracle 10g R2 CRS

Sun Cluster 3.2 introduces resource types that enable Sun Cluster and Oracle 10g R2 CRS to interoperate. These resource types also enable Oracle RAC database instances to be administered from Sun Cluster.

Note –

Oracle 10g R1 CRS and Sun Cluster cannot interoperate. Instead, Oracle 10g R1 CRS start and shut down Oracle RAC database instances.

Before adding resources for interoperation with Oracle 10g R2 CRS, ensure that you have performed the tasks in the following sections:

For detailed instructions for adding resources for interoperation with Oracle 10g R2 CRS, see Configuring Resources for Oracle RAC Database Instances.

Upgrading to Sun Cluster 3.2 Resource Types for Oracle 9i RAC Database Instances

Sun Cluster 3.2 replaces the failover resource types that automate the startup and shutdown of Oracle 9i RAC database instances with multi-master resource types. The multi-master resources simplify the administration and enhance the scalability of Sun Cluster Support for Oracle RAC.

Upgrading to Sun Cluster 3.2 resource types for Oracle 9i RAC database instances involves the tasks that are described in the following sections:

Perform these tasks in the order in which they are listed.

How to Take Offline Resource Groups for Oracle RAC Database Components and Restart the Components

This task involves taking offline resource groups for Oracle RAC database components and restarting the components on each cluster node individually.

This sequence of operations avoids downtime by ensuring that the Oracle RAC database components remain online on other cluster nodes.

Perform this task on each cluster node individually.

Before You Begin

Ensure that you have upgraded resources in the RAC framework resource group. For more information, see Upgrading Resources in the RAC Framework Resource Group.

Become superuser on the cluster node.

If you configured Oracle listener resources in separate resource groups, take offline the resource groups from the node.

If you configured Oracle listener resources in the same resource group as the Oracle RAC server resource, omit this step.

For each resource group that you are taking offline, type the following command:
# clresourcegroup offline -n node rac-listener-rg
-n node

Specifies the node from which you are taking the resource group offline. This node is the node where you are performing this task.

rac-listener-rg

Specifies the resource group for Oracle listener resources that you are taking offline.

When the resource group is taken offline, the Oracle listener is stopped on the node. The Oracle listeners on other nodes are unaffected.

Take offline the resource group for the RAC server from the node.
# clresourcegroup offline -n node rac-server-rg
-n node

Specifies the node from which you are taking the resource group offline. This node is the node where you are performing this task.

rac-server-rg

Specifies the resource group for the Oracle RAC server resource that you are taking offline.

When the resource group is taken offline, the Oracle RAC server is stopped on the node. The Oracle RAC servers on other nodes are unaffected.

If you configured Oracle listener resources in the same resource group as the Oracle RAC server resource, the listeners are also stopped on the node. The Oracle listeners on other nodes are unaffected.

Use Oracle utilities to restart the Oracle RAC server and Oracle listener on the node.

Next Steps

Go to How to Modify the Configuration of Sun Cluster Resources for Oracle 9i RAC Database Instances.

How to Modify the Configuration of Sun Cluster Resources for Oracle 9i RAC Database Instances

Perform this task from one cluster node.

Before You Begin

Ensure that the task that is explained in How to Take Offline Resource Groups for Oracle RAC Database Components and Restart the Components has been performed.

Become superuser or assume a role that provides solaris.cluster.admin and solaris.cluster.modify RBAC authorizations.

If you configured Oracle listener resources in separate resource groups, remove the resource groups.

If you configured Oracle listener resources in the same resource group as Oracle RAC server resources, omit this step.

For each resource group that you are removing, type the following command:
# clresourcegroup delete -F rac-listener-rg
rac-listener-rg

Specifies the resource group for Oracle listener resources that you are removing.

Remove all resource groups that contain an Oracle RAC server resource.

For each resource group that you are removing, type the following command:
# clresourcegroup delete -F rac-server-rg
rac-server-rg

Specifies the resource group that contains an Oracle RAC server resource that you are removing.

If you are using Sun StorageTek QFS shared file system, remove all resource groups that contain resources for the Sun StorageTek QFS metadata server.

If you are not using the Sun StorageTek QFS shared file system, omit this step.

For each resource group that you are removing, type the following command:
# clresourcegroup delete -F qfs-mds-rg
qfs-mds-rg

Specifies the resource group that contains resources for the Sun StorageTek QFS metadata server that you are removing.

If you are using global device groups or shared file systems for Oracle files, add storage resources to manage the availability of the storage on which the Oracle software depends.

If you are not using global device groups or shared file systems for Oracle files, omit this step.

For instructions for performing this task, see Adding Storage Resources for Oracle Files.

Configure new resources for Oracle 9i RAC database instances.

For instructions for performing this task, see Configuring Resources for Oracle RAC Database Instances.

Extending an Existing Configuration of Sun Cluster Support for Oracle RAC

Extend an existing configuration of Sun Cluster Support for Oracle RAC in any of the following situations:

You are adding nodes to a cluster and you require Sun Cluster Support for Oracle RAC to run on the nodes. For more information, see How to Add Sun Cluster Support for Oracle RAC to Selected Nodes .
You are adding a volume manager. For more information, see How to Add a Volume Manager Resource to the RAC Framework Resource Group.

How to Add Sun Cluster Support for Oracle RAC to Selected Nodes

Perform this procedure if you are adding nodes to a cluster and you require Sun Cluster Support for Oracle RAC to run on the nodes. Perform this procedure from only one node.

This procedure provides the long forms of the Sun Cluster maintenance commands. Most commands also have short forms. Except for the forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix A, Sun Cluster Object-Oriented Commands, in Sun Cluster Data Services Planning and Administration Guide for Solaris OS.

This task involves adding the selected nodes from the following resource groups in the following order:

Resource groups for scalable file-system mount-point resources
The RAC framework resource group
Resource groups for scalable device group resources
Resource groups that contain resources for the Sun StorageTek^TM QFS metadata server
Resource groups for logical hostname resources
The resource group for the RAC database

Before You Begin

Ensure that the required Sun Cluster Support for Oracle RAC software packages are installed on each node to which you are adding Sun Cluster Support for Oracle RAC. For more information, see Installing the Sun Cluster Support for Oracle RAC Packages.

Become superuser on any cluster node.

Add the nodes to any resource groups that contain scalable file-system mount-point resources.

If no resource groups that contain scalable file-system mount-point resources are configured, omit this step.

For each resource group to which you are adding nodes, run the following command:
# clresourcegroup add-node -S -n nodelist scal-mp-rg
-n nodelist

Specifies a comma-separated list of cluster nodes to which you are adding Sun Cluster Support for Oracle RAC.

scal-mp-rg

Specifies the name of the resource group to which you are adding nodes.

Add the nodes to the RAC framework resource group.
# clresourcegroup add-node -S -n nodelist rac-fmwk-rg
-n nodelist

Specifies a comma-separated list of cluster nodes to which you are adding Sun Cluster Support for Oracle RAC.

rac-fmwk-rg

Specifies the name of the resource group to which you are adding nodes.

Add the nodes to any scalable device groups that you are using for Oracle files.

If you are not using any scalable device groups for Oracle files, omit this step.

How to perform this step depends on the type of the scalable device group.
- For each Solaris Volume Manager for Sun Cluster multi-owner disk set, type the following command:
  # metaset -s set-name -M -a -h nodelist
  -s set-name
  
  Specifies the Solaris Volume Manager for Sun Cluster multi-owner disk set to which you are adding nodes.
  
  -h nodelist
  
  Specifies a space-separated list of cluster nodes that you are adding to the multi-owner disk set.
- For each VxVM shared-disk group, use Veritas commands to add the nodes to the VxVM shared-disk group.
  
  For more information, see your VxVM documentation.

Add the nodes to any resource groups that contain scalable device group resources.

If no resource groups that contain scalable device group resources are configured, omit this step.

For each resource group to which you are adding nodes, run the following command:
# clresourcegroup add-node -S -n nodelist scal-dg-rg
-n nodelist

Specifies a comma-separated list of cluster nodes to which you are adding Sun Cluster Support for Oracle RAC.

scal-dg-rg

Specifies the name of the resource group to which you are adding nodes.

Mount each shared file system that is to be accessed from the nodes that you are adding.

If no shared file systems are to be accessed from the nodes that you are adding, omit this step.

For each file system that you are mounting, type the following command:
# mount mount-point
mount-point

Specifies the mountpoint of the file system that you are mounting.

Add the nodes to any resource groups that contain resources for the Sun StorageTek QFS metadata server.

If no resource groups that contain resources for the Sun StorageTek QFS metadata server are configured, omit this step.

For each resource group to which you are adding nodes, run the following command:
# clresourcegroup add-node -n nodelist qfs-mds-rg
-n nodelist

Specifies a comma-separated list of cluster nodes to which you are adding Sun Cluster Support for Oracle RAC.

qfs-mds-rg

Specifies the name of the resource group to which you are adding nodes.

Bring online all resource groups to which you added nodes in Step 5.

These resource groups contain scalable device group resources.

If no resource groups that contain scalable device group resources are configured, omit this step.

For each resource group that you are bringing online, type the following command:
# clresourcegroup online scal-dg-rg
scal-dg-rg

Specifies the name of the resource group that you are bringing online.

(Oracle 10g R2 only) Start the Oracle CRS.

If you are using Oracle 9i or Oracle 10g R1, omit this step.
# /etc/init.d/init.crs start Startup will be queued to init within 30 seconds.

(Oracle 9i only) Add the nodes to all resource groups that contain logical hostname resources for each RAC database that is to run on the nodes.

If you are using Oracle 10g R1 or Oracle 10g R2 , omit this step. For Oracle 10g R1 and Oracle 10g R2, no resource groups for logical hostname resources are configured.

For each resource group to which you are adding nodes, run the following command:
# clresourcegroup add-node -n nodelist lh-rg
-n nodelist

Specifies a comma-separated list of cluster nodes to which you are adding Sun Cluster Support for Oracle RAC.

lh-rg

Specifies the name of the resource group to which you are adding nodes.

(Oracle 9i and Oracle 10g R2 only) Add the nodes to the resource group for each RAC database that is to run on the nodes.

If you are using Oracle 10g R1, omit this step. For Oracle 10g R1, no resource groups for RAC databases are configured.

For each resource group to which you are adding nodes, run the following command:
# clresourcegroup add-node -S -n nodelist rac-db-rg
-n nodelist

Specifies a comma-separated list of cluster nodes to which you are adding Sun Cluster Support for Oracle RAC.

rac-db-rg

Specifies the name of the resource group to which you are adding nodes.

(Oracle 10g R2 only) For each node that you are adding, create the Oracle CRS resources that are required to represent Sun Cluster resources.

Create an Oracle CRS resource for each Sun Cluster resource for scalable device groups and scalable file-system mountpoints on which Oracle components depend. For more information, see How to Create an Oracle CRS Resource for Interoperation With Sun Cluster.

(Oracle 9i and Oracle 10g R2 only) Modify each resource for RAC databases to set a value of each per-node property for each node that you are adding.

If you are using Oracle 10g R1, omit this step. For Oracle 10g R1, no resource groups for RAC databases are configured.

For each resource that you are modifying, perform these steps:

Disable the resource.
# clresource disable rac-db-rs
rac-db-rs

Specifies the name of the RAC database resource that you are disabling.

Set a value of each per-node property for each node that you are adding.

The per-node properties of each resource type for RAC databases are shown in the following table.

Resource Type	Properties
`SUNW.scalable_rac_server_proxy`	`oracle_sid`
`SUNW.scalable_rac_listener`	`listener_name`
`SUNW.scalable_rac_server`	`alert_log_file` `oracle_sid`

Resource Type

Properties

SUNW.scalable_rac_server_proxy

oracle_sid

SUNW.scalable_rac_listener

listener_name

SUNW.scalable_rac_server

alert_log_file

oracle_sid

For information about extension properties of resource types for RAC databases, see the following sections:

# clresource set \
-p property{node}=value[…]\
[-p property{node}=value[…]][…]\
rac-db-rs

property: Specifies the name of a per-node property that you are setting.
node: Specifies the node for which you are setting a value for property.
value: Specifies the value to which you are setting property for node.
rac-db-rs: Specifies the name of the RAC database resource whose per-node properties you are setting.

Enable the resource.
# clresource enable rac-db-rs
rac-db-rs

Specifies the name of the RAC database resource that you are enabling.

(Oracle 9i and Oracle 10g R2 only) Bring online each resource group for RAC databases.

If you are using Oracle 10g R1, omit this step. For Oracle 10g R1, no resource groups for RAC databases are configured.

For each resource group that you are bringing online, run the following command:
# clresourcegroup online rac-db-rg
rac-db-rg

Specifies the name of the resource group that you are bringing online.

Example 6–3 Adding Sun Cluster Support for Oracle RAC to Selected Nodes

This example shows the sequence of operations that is required to add Sun Cluster Support for Oracle RAC to nodes pclus3 and pclus4 of a four-node cluster.

The configuration of Sun Cluster Support for Oracle RAC in this example is as follows:

The version of Oracle RAC is 10g R2.
Sun StorageTek QFS shared file system on Solaris Volume Manager for Sun Cluster is used to store Oracle files.
The mountpoints of file systems that are used for Oracle files are as follows:
- Oracle database files: /db_qfs/OraData
- Oracle binary files and related files: /db_qfs/OraHome
The name of the Oracle RAC database is swb.
The Sun StorageTek QFS shared file systems use a Solaris Volume Manager for Sun Cluster multi-owner disk set that is named oradg. The creation of this disk set is shown in Example 2–1.

The configuration of resource groups in this example is shown in the following table.

Resource Group	Purpose
`rac-framework-rg`	RAC framework resource group.
`scaldg-rg`	Resource group for scalable device-group resources.
`qfsmds-rg`	Resource group for Sun StorageTek QFS metadata server resources.
`scalmnt-rg`	Resource group for scalable file-system mount-point resources.
`rac_server_proxy-rg`	RAC database resource group.

The resource groups that are required for this configuration are shown in Figure A–2.

To add the nodes to the resource group that contains scalable file-system mount-point resources, the following command is run:
# clresourcegroup add-node -S -n pclus3,pclus4 scalmnt-rg
To add the nodes to the RAC framework resource group, the following command is run:
# clresourcegroup add-node -S -n pclus3,plcus4 rac-framework-rg
To add the nodes to the Solaris Volume Manager for Sun Cluster multi-owner disk set oradg, the following command is run:
# metaset -s oradg -M -a -h pclus3 pclus4
To add the nodes to the resource group that contains scalable device group resources, the following command is run:
# clresourcegroup add-node -S -n pclus3,pclus4 scaldg-rg
To mount the shared file systems that are to be accessed from the nodes that are being added, the following commands are run:
# mount /db_qfs/OraData # mount /db_qfs/OraHome
To add the nodes to the resource group that contains resources for the Sun StorageTek QFS metadata server, the following command is run:
# clresourcegroup add-node -n pclus3,pclus4 qfsmds-rg
To bring online the resource group that contains scalable device group resources, the following command is run:
# clresourcegroup online scaldg-rg

To start the Oracle CRS and to verify the correct startup of the Oracle CRS, the following commands are run:

# /etc/init.d/init.crs start
Startup will be queued to init within 30 seconds.
# /db_qfs/OraHome/crs/bin/crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

To add the nodes to the resource group for the RAC database, the following command is run:
# clresourcegroup add-node -S -n pclus3,pclus4 rac_server_proxy-rg
After the addition of the nodes to the resource group for the RAC database, the required Oracle CRS resources are created. The creation of these Oracle CRS resources is beyond the scope of this example.
To set required per-node properties for the RAC database resource, the following commands are run:
# clresource disable rac_server_proxy-rs # clresource set -p oracle_sid\{3\}=swb3 -p \ oracle_sid\{4\}=swb4 rac_server_proxy-rs # clresource enable rac_server_proxy-rs
The per-node property oracle_sid is set to swb3 on node pclus3 and to swb4 on node pclus4.
To bring online the resource group for the RAC database, the following command is run:
# clresourcegroup online rac_server_proxy-rg

How to Add a Volume Manager Resource to the RAC Framework Resource Group

Perform this task if you are adding a volume manager to an existing configuration of Sun Cluster Support for Oracle RAC. The RAC framework resource group must contain a resource that represents the volume manager that you are adding. You can add a volume manager resource only if the RAC framework resource is disabled and if the UCMM daemon is stopped on all cluster nodes.

Caution –

This task requires downtime because you must disable the RAC framework resource and boot the nodes where Oracle RAC is running.

Before You Begin

Ensure that the volume manager for which you are adding a resource is installed and configured on all nodes where Oracle RAC is to run.

Become superuser on any cluster node.

Disable the resource of type SUNW.rac_framework in the RAC framework resource group and any other resources that depend on this resource.
# clresource disable -r rac-fmwk-rs
rac-fmwk-rs

Specifies the name of the resource of type SUNW.rac_framework that you are disabling.

Reboot all the nodes that are in the node list of the RAC framework resource group.

Register and add an instance of the resource type that represents the volume manager that you are adding.
- If you are adding Solaris Volume Manager for Sun Cluster, register and add the instance as follows:
  1. Register the SUNW.rac_svm resource type.
    # clresourcetype register SUNW.rac_svm
  2. Add an instance of the SUNW.rac_svm resource type to the RAC framework resource group.
    
    Ensure that this instance depends on the resource of type SUNW.rac_framework that you disabled in Step 2.
    # clresource create -g rac-fmwk-rg \ -t SUNW.rac_svm \ -p resource_dependencies=rac-fmwk-rs rac-svm-rs
    -g rac-fmwk-rg
    
    Specifies the name RAC framework resource group. This resource group contains the resource of type SUNW.rac_framework that you disabled in Step 2.
    
    -p resource_dependencies=rac-fmwk-rs
    
    Specifies that this instance depends on the resource of type SUNW.rac_framework that you disabled in Step 2.
    
    rac-svm-rs
    
    Specifies the name that you are assigning to the resource of type SUNW.rac_svm.
- If you are adding VxVM with the cluster feature, register and add the instance as follows.
  1. Register the SUNW.rac_cvm resource type.
    # clresourcetype register SUNW.rac_cvm
  2. Add an instance of the SUNW.rac_cvm resource type to the resource group that you disabled in Step 2.
    
    Ensure that this instance depends on the resource of type SUNW.rac_framework that you disabled in Step 2.
    # clresource create -g rac-fmwk-rg \ -t SUNW.rac_cvm \ -p resource_dependencies=rac-fmwk-rs rac-cvm-rs
    -g rac-fmwk-rg
    
    Specifies the name RAC framework resource group. This resource group contains the resource of type SUNW.rac_framework that you disabled in Step 2.
    
    -p resource_dependencies=rac-fmwk-rs
    
    Specifies that this instance depends on the resource of type SUNW.rac_framework that you disabled in Step 2.
    
    rac-cvm-rs
    
    Specifies the name that you are assigning to the resource of type SUNW.rac_cvm.

Bring online and in a managed state the RAC framework resource group and its resources.
# clresourcegroup online -emM rac-fmwk-rg
rac-fmwk-rg

Specifies that the RAC framework resource group is to be moved to the MANAGED state and brought online. This resource group contains the resource of type SUNW.rac_framework that you disabled in Step 2.

Next Steps

The next step depends on the volume manager that you are adding, as shown in the following table.

Volume Manager	Next Step
Solaris Volume Manager for Sun Cluster	How to Create a Multi-Owner Disk Set in Solaris Volume Manager for Sun Cluster for the Oracle RAC Database
VxVM with the cluster feature	How to Create a VxVM Shared-Disk Group for the Oracle RAC Database

Removing Sun Cluster Support for Oracle RAC

You can remove Sun Cluster Support for Oracle RAC from the following entities:

A cluster. For more information, see How to Remove Sun Cluster Support for Oracle RAC From a Cluster.
Selected nodes in a cluster. For more information, see How to Remove Sun Cluster Support for Oracle RAC From Selected Nodes.

How to Remove Sun Cluster Support for Oracle RAC From a Cluster

Perform this task to remove Sun Cluster Support for Oracle RAC from all nodes in a cluster.

On a cluster where multiple Oracle RAC databases are running, perform this task to remove an Oracle RAC database from the cluster. The remaining Oracle RAC databases continue to run in the cluster.

This task involves removing the following resource groups from the cluster in the following order:

The resource group for the RAC database
Resource groups for logical hostname resources
Resource groups for scalable file-system mount-point resources
Resource groups that contain resources for the Sun StorageTek QFS metadata server
Resource groups for scalable device group resources
The RAC framework resource group

Caution –

You might perform this task to remove an Oracle RAC database from a cluster where multiple Oracle RAC databases are running. In this situation, do not remove any resource group on whose resources the remaining Oracle RAC databases depend. For example, you might have configured multiple database file systems to depend on a single device group. In this situation, do not remove the resource group that contains the resource for the scalable device group. Similarly, if multiple databases depend on the RAC framework resource group, do not remove this resource group.

Before You Begin

Ensure that the cluster node from which you perform this task is booted in cluster mode.

On one node of the cluster, become superuser.

(Oracle 9i and Oracle 10g R2 only) Remove the resource group for each RAC database that you are removing.

If you are using Oracle 10g R1, omit this step. For Oracle 10g R1, no resource groups for RAC databases are configured.

For each RAC database that you are removing, type the following command:
# clresourcegroup delete -F rac-db-rg
rac-db-rg

Specifies the resource group that you are removing.

(Oracle 9i only) Remove all resource groups for logical hostname resources that are used by each RAC database that you are removing.

If you are using Oracle 10g R1 or Oracle 10g R2 , omit this step. For Oracle 10g R1 and Oracle 10g R2, no resource groups for logical hostname resources are configured.

For each resource group that you are removing, type the following command:
# clresourcegroup delete -F lh-rg
lh-rg

Specifies the resource group that you are removing.

Use Oracle utilities to remove from the cluster each RAC database that you no longer require.

If you are removing Sun Cluster Support for Oracle RAC entirely, use Oracle utilities to remove the following items from all nodes in a cluster:
- The Oracle RAC software
- The Oracle CRS software

(Oracle 10g R2 only) Disable the CRS framework resource.

If you are using Oracle 9i or Oracle 10g R1, omit this step. For Oracle 9i and Oracle 10g R1, no CRS framework resource is configured.
# clresource disable crs-framework-rs
crs-framework-rs

Specifies the name of the resource that you are disabling. This resource is the instance of the SUNW.crs_framework resource type that is configured in the cluster.

Remove any resource groups that contain scalable file-system mount-point resources.

If no resource groups that contain scalable file-system mount-point resources are configured, omit this step.

For each resource group that you are removing, type the following command:
# clresourcegroup delete -F scal-mp-rg
scal-mp-rg

Specifies the resource group that you are removing.

Remove any resource groups that contain resources for the Sun StorageTek QFS metadata server.

If no resource groups that contain resources for the Sun StorageTek QFS metadata server are configured, omit this step.

For each resource group that you are removing, type the following command:
# clresourcegroup delete -F qfs-mds-rg
qfs-mds-rg

Specifies the resource group that you are removing.

Remove the Sun StorageTek QFS shared file systems that were represented by resources in the resource group that you deleted in Step 8.

For instructions for performing this task, see Sun StorageTek QFS File System Configuration and Administration Guide, Version 4, Update 6.

Remove any resource groups that contain scalable device group resources.

If no resource groups that contain scalable device group resources are configured, omit this step.

For each resource group that you are removing, type the following command:
# clresourcegroup delete -F scal-dg-rg
scal-dg-rg

Specifies the resource group that you are removing.

Destroy any scalable device groups that were affected by the removal of resource groups in Step 10.

How to perform this step depends on the type of the scalable device group.
- For each Solaris Volume Manager for Sun Cluster multi-owner disk set, destroy the disk set as follows:
  1. Remove all metadevices such as volumes, soft partitions, and mirrors from the disk set.
    
    Use the metaclear(1M) command for this purpose.
    # metaclear -s scal-dg-ms -a
    -s scal-dg-ms
    
    Specifies the name of the disk set from which you are removing metadevices.
  2. Remove all global devices from the disk set.
    # metaset -s scal-dg-ms -d -f alldevices
    -s scal-dg-ms
    
    Specifies the name of the disk set from which you are removing global devices.
    
    alldevices
    
    Specifies a space-separated list that contains all global devices that were added to the disk set when the disk set was created. The format of each device ID path name is /dev/did/dsk/dN, where N is the device number.
  3. Remove all nodes from the disk set that you are destroying.
    
    The removal of all nodes from a disk set destroys the disk set.
    # metaset -s scal-dg-ms -d -h allnodes
    -s scal-dg-ms
    
    Specifies the name of the disk set that you are destroying.
    
    -h allnodes
    
    Specifies a space-separated list that contains all nodes that were added to the disk set when the disk set was created.
- For each VxVM shared-disk group, use Veritas commands to destroy the VxVM shared-disk group.
  
  For more information, see your VxVM documentation.
Note –
If you are removing an Oracle RAC database from a cluster where multiple Oracle RAC databases are running, omit the remaining steps in this procedure.

Remove the RAC framework resource group.
# clresourcegroup delete -F rac-fmwk-rg
rac-fmwk-rg

Specifies the resource group that you are removing.

Unregister the resource type of each resource that you removed in this procedure.
# clresourcetype unregister resource-type-list
resource-type-list

Specifies a comma-separated list of the names of the resource types that you are unregistering. For a list of the resource types that are associated with Sun Cluster Support for Oracle RAC, see Automatically Generated Names for Sun Cluster Objects.

(Optional) From each node in the cluster, uninstall the Sun Cluster Support for Oracle RAC software packages.

Use the Sun Java^TM Enterprise System uninstaller for this purpose. For more information, see Chapter 8, Uninstalling, in Sun Java Enterprise System 5 Installation Guide for UNIX.

Reboot each node in the cluster.

Example 6–4 Removing Sun Cluster Support for Oracle RAC From a Cluster

This example shows the sequence of operations that is required to remove Sun Cluster Support for Oracle RAC from all nodes of a four-node cluster. The nodes in this cluster are named pclus1, pclus2, pclus3, and pclus4. Only one RAC database is configured on the cluster.

The configuration of Sun Cluster Support for Oracle RAC in this example is as follows:

The version of Oracle RAC is 10g R2.
Sun StorageTek QFS shared file system on Solaris Volume Manager for Sun Cluster is used to store Oracle files.
The mountpoints of file systems that are used for Oracle files are as follows:
- Oracle database files: /db_qfs/OraData
- Oracle binary files and related files: /db_qfs/OraHome
The name of the Oracle RAC database is swb.
The Sun StorageTek QFS shared file systems use a Solaris Volume Manager for Sun Cluster multi-owner disk set that is named oradg. The creation of this disk set is shown in Example 2–1.

The configuration of resource groups in this example is shown in the following table.

Resource Group	Purpose
`rac-framework-rg`	RAC framework resource group.
`scaldg-rg`	Resource group for scalable device-group resources.
`qfsmds-rg`	Resource group for Sun StorageTek QFS metadata server resources.
`scalmnt-rg`	Resource group for scalable file-system mount-point resources.
`rac_server_proxy-rg`	RAC database resource group.

The resource groups that are required for this configuration are shown in Figure A–2.

To remove the resource group for RAC database, the following command is run:
# clresourcegroup delete -F rac_server_proxy-rg
After the removal of this resource group, Oracle utilities are used to remove the following items:
- The Oracle RAC database
- The Oracle RAC software
- The Oracle CRS software
The removal of these items is beyond the scope of this example.
To disable the CRS framework resource, the following command is run:
# clresource disable crs_framework-rs
To remove the resource group that contains scalable file-system mount-point resources, the following command is run:
# clresourcegroup delete -F scalmnt-rg
To remove the resource group that contains resources for the Sun StorageTek QFS metadata server, the following command is run:
# clresourcegroup delete -F qfsmds-rg
After the removal of this resource group, Sun StorageTek QFS utilities are used to remove the Sun StorageTek QFS shared file systems that are used for Oracle files. The removal of these file systems is beyond the scope of this example.
To remove the resource group that contains scalable device group resources, the following command is run:
# clresourcegroup delete -F scaldg-rg
To destroy the Solaris Volume Manager for Sun Cluster multi-owner disk set oradg, the following commands are run:
# metaclear -s oradg -a # metaset -s oradg -d \ -f /dev/did/dsk/d8 /dev/did/dsk/d9 /dev/did/dsk/d15 /dev/did/dsk/d16 # metaset -s oradg -d -h pclus1 pclus2 pclus3 pclus4
The following global devices are removed from the disk set:
- /dev/did/dsk/d8
- /dev/did/dsk/d9
- /dev/did/dsk/d15
- /dev/did/dsk/d16
To remove the RAC framework resource group, the following command is run:
# clresourcegroup delete -F rac-framework-rg
To unnregister the resource type of each resource that was removed, the following command is run:
# clresourcetype unregister \ SUNW.scalable_rac_server_proxy,\ SUNW.ScalMountPoint,\ SUNW.qfs,\ SUNW.ScalDeviceGroup,\ SUNW.rac_svm,\ SUNW.crs_framework,\ SUNW.rac_udlm,\ SUNW.rac_framework
This configuration is running on the SPARC® platform. Therefore, SUNW.rac_udlm is in the list of resource types that are unregistered.

After the unregistration of these resource types, the following operations are performed:
- The removal of the Sun Cluster Support for Oracle RAC software packages
- The reboot of each node in the cluster
These operations are beyond the scope of this example.

How to Remove Sun Cluster Support for Oracle RAC From Selected Nodes

Perform this task to remove Sun Cluster Support for Oracle RAC from selected nodes.

On a cluster where multiple Oracle RAC databases are running, perform this task to remove an Oracle RAC database from selected nodes. The Oracle RAC database that you remove continues to run on the other cluster nodes. The remaining Oracle RAC databases continue to run on the selected nodes.

This task involves removing the selected nodes from the following resource groups in the following order:

The resource group for the RAC database
Resource groups for logical hostname resources
Resource groups for scalable file-system mount-point resources
Resource groups that contain resources for the Sun StorageTek QFS metadata server
Resource groups for scalable device group resources
The RAC framework resource group

Caution –

You might perform this task to remove an Oracle RAC database from selected nodes of a cluster where multiple Oracle RAC databases are running. In this situation, do not remove the nodes from any resource group on whose resources the remaining Oracle RAC databases depend. For example, you might have configured multiple database file systems to depend on a single device group. In this situation, do not remove the nodes from the resource group that contains the resource for the scalable device group. Similarly, if multiple databases depend on the RAC framework resource group, do not remove the nodes from this resource group.

Become superuser.

(Oracle 9i and Oracle 10g R2 only) Remove the nodes from the resource group for each RAC database that you are removing.

If you are using Oracle 10g R1, omit this step. For Oracle 10g R1, no resource groups for RAC databases are configured.

For each RAC database that you are removing, perform the following steps:
1. Take offline the resource group for the RAC database on the nodes from which you are removing Sun Cluster Support for Oracle RAC.
  # clresourcegroup offline -n nodelist rac-db-rg
  -n nodelist
  
  Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.
  
  rac-db-rg
  
  Specifies the name of the resource group that you are taking offline.
2. Remove the nodes from the node list of the resource group for the RAC database.
  # clresourcegroup remove-node -n nodelist rac-db-rg
  -n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  rac-db-rg
  
  Specifies the name of the resource group from which you are removing nodes.

(Oracle 9i only) Remove the nodes from all resource groups for logical hostname resources that are used by each RAC database that you are removing.

If you are using Oracle 10g R1 orOracle 10g R2 , omit this step. For Oracle 10g R1 and Oracle 10g R2, no resource groups for logical hostname resources are configured.

For each resource group from which you are removing nodes, perform the following steps:
1. Switch the resource group to a node from which you are not removing Sun Cluster Support for Oracle RAC.
  # clresourcegroup switch -n node-to-stay lh-rg
  node-to-stay
  
  Specifies the node to which you are switching the resource group. This node must be a node from which you are not removing Sun Cluster Support for Oracle RAC.
  
  lh-rg
  
  Specifies the name of the resource group that you are switching to another node.
2. Remove the nodes from the node list of the resource group.
  # clresourcegroup remove-node -n nodelist lh-rg
  -n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  lh-rg
  
  Specifies the name of the resource group from which you are removing nodes.

(Oracle 9i only) Remove each resource group for logical hostname resources whose primary node you removed from resource groups in Step 3.

These resource groups are no longer required because the RAC database instances that the groups serve are being removed.

Do not remove any resource groups from which you removed only secondary nodes in Step 3.

For each resource group that you are removing, type the following command:
# clresourcegroup remove -F lh-rg-rm-prim
lh-rg-rm-prim

Specifies the name of the resource group that you are removing.

(Oracle 10g R2 only) Remove each node that you are removing from the list of nodes where the Oracle CRS resource for the Oracle database runs.

If you are using Oracle 9i or Oracle 10g R1 , omit this step. For Oracle 9i and Oracle 10g R1, no Oracle CRS resources that represent Sun Cluster resources are configured.

Note –
In this step, the syntax of Oracle commands for Oracle release 10g R2 is provided. If you are you are using a version Oracle other than 10g R2, see your Oracle documentation for the correct command syntax.
# crs-home/bin/crs_register ora.dbname.sid.inst \ -update -r "ora.node-name.vip"
crs-home

Specifies the Oracle CRS home directory. This directory contains the Oracle CRS binary files and Oracle CRS configuration files.

dbname

Specifies the database name of the database instance that the Oracle CRS resource represents.

sid

Specifies the Oracle SID of the database instance that the Oracle CRS resource represents.

node-name

Specifies the host name of the node where the Oracle CRS resource runs.

(Oracle 10g R2 only) From each node that you are removing, remove each Oracle CRS resource that represents a Sun Cluster resource from whose resource group you are removing nodes.

An Oracle CRS resource is configured for each Sun Cluster resource for scalable device groups and scalable file-system mountpoints on which Oracle components depend.

If you are using Oracle 9i or Oracle 10g R1 , omit this step. For Oracle 9i and Oracle 10g R1, no Oracle CRS resources that represent Sun Cluster resources are configured.

Note –
In this step, the syntax of Oracle commands for Oracle release 10g R2 is provided. If you are you are using a version Oracle other than 10g R2, see your Oracle documentation for the correct command syntax.

For each Oracle CRS resource that you are removing, perform the following steps on each node from which you are removing the resource:
1. Stop the Oracle CRS resource that you are removing.
  # crs-home/bin/crs_stop sun.node-name.sc-rs
  crs-home
  
  Specifies the Oracle CRS home directory. This directory contains the Oracle CRS binary files and Oracle CRS configuration files.
  
  node-name
  
  Specifies the host name of the node where the Oracle CRS resource runs.
  
  sc-rs
  
  Specifies the name of the Sun Cluster resource that the Oracle CRS resource represents.
2. Unregister the Oracle CRS resource that you are removing.
  # crs-home/bin/crs_unregister sun.node-name.sc-rs
  crs-home
  
  Specifies the Oracle CRS home directory. This directory contains the Oracle CRS binary files and Oracle CRS configuration files.
  
  node-name
  
  Specifies the host name of the node where the Oracle CRS resource runs.
  
  sc-rs
  
  Specifies the name of the Sun Cluster resource that the Oracle CRS resource represents.
3. Delete the profile for the Oracle CRS resource that you are removing.
  # crs-home/bin/crs_profile -delete sun.node-name.sc-rs \ -dir /var/cluster/ucmm/profile
  crs-home
  
  Specifies the Oracle CRS home directory. This directory contains the Oracle CRS binary files and Oracle CRS configuration files.
  
  node-name
  
  Specifies the host name of the node where the Oracle CRS resource runs.
  
  sc-rs
  
  Specifies the name of the Sun Cluster resource that the Oracle CRS resource represents.

Use Oracle utilities to remove the following items from each node from which you are removing Sun Cluster Support for Oracle RAC:
- The RAC database
- Oracle CRS

Switch any resource groups that contain resources for the Sun StorageTek QFS metadata server to a node from which you are not removing Sun Cluster Support for Oracle RAC.

If no resource groups that contain resources for the Sun StorageTek QFS metadata server are configured, omit this step.
# clresourcegroup switch -n node-to-stay qfs-mds-rg
node-to-stay

Specifies the node to which you are switching the resource group. This node must be a node from which you are not removing Sun Cluster Support for Oracle RAC.

qfs-mds-rg

Specifies the name of the resource group that you are switching to another node.

Remove the nodes from any resource groups that contain scalable file-system mount-point resources.

If no resource groups that contain scalable file-system mount-point resources are configured, omit this step.

For each resource group from which you are removing nodes, perform the following steps:
1. Take offline the resource group on the nodes from which you are removing Sun Cluster Support for Oracle RAC.
  # clresourcegroup offline -n nodelist scal-mp-rg
  -n nodelist
  
  Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.
  
  scal-mp-rg
  
  Specifies the name of the resource group that you are taking offline.
2. Remove the nodes from the node list of the resource group.
  # clresourcegroup remove-node -n nodelist scal-mp-rg
  -n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  scal-mp-rg
  
  Specifies the name of the resource group from which you are removing nodes.

Remove the nodes from the node list of any resource groups that contain resources for the Sun StorageTek QFS metadata server.

If no resource groups that contain resources for the Sun StorageTek QFS metadata server are configured, omit this step.

The resource groups to modify are the resource groups that you switched to another node in Step 8.
# clresourcegroup remove-node -n nodelist qfs-mds-rg
-n nodelist

Specifies a comma-separated list of cluster nodes that you are removing from the resource group.

qfs-mds-rg

Specifies the name of the resource group from which you are removing nodes.

Remove the configuration of the Sun StorageTek QFS shared file systems from the nodes.

For instructions for performing this task, see Sun StorageTek QFS File System Configuration and Administration Guide, Version 4, Update 6.

Remove the nodes from any resource groups that contain scalable device group resources.

If no resource groups that contain scalable device group resources are configured, omit this step.

For each resource group from which you are removing nodes, perform the following steps:
1. Take offline the resource group on the nodes from which you are removing Sun Cluster Support for Oracle RAC.
  # clresourcegroup offline -n nodelist scal-dg-rg
  -n nodelist
  
  Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.
  
  scal-dg-rg
  
  Specifies the name of the resource group that you are taking offline.
2. Remove the nodes from the node list of the resource group.
  # clresourcegroup remove-node -n nodelist scal-dg-rg
  -n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  scal-dg-rg
  
  Specifies the name of the resource group from which you are removing nodes.

Remove the nodes from any scalable device groups that were affected by the removal of nodes from resource groups in Step 12.

How to perform this step depends on the type of the scalable device group.
- For each Solaris Volume Manager for Sun Cluster multi-owner disk set, type the following command:
  # metaset -s scal-dg-ms -d -h nodelist
  -s scal-dg-ms
  
  Specifies the name of the disk set from which you are removing nodes.
  
  -h nodelist
  
  Specifies a space-separated list of the nodes that you are removing from the disk set.
- For each VxVM shared-disk group, use Veritas commands to remove the nodes from the VxVM shared-disk group.
  
  For more information, see your VxVM documentation.
Note –
If you are removing an Oracle RAC database from selected nodes of a cluster where multiple Oracle RAC databases are running, omit the remaining steps in this procedure.

Remove the nodes from the RAC framework resource group.
1. Take offline the resource group on the nodes from which you are removing Sun Cluster Support for Oracle RAC.
  # clresourcegroup offline -n nodelist rac-fmwk-rg
  -n nodelist
  
  Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.
  
  rac-fmwk-rg
  
  Specifies the name of the resource group that you are taking offline.
2. Remove the nodes from the node list of the resource group.
  # clresourcegroup remove-node -n nodelist rac-fmwk-rg
  -n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  rac-fmwk-rg
  
  Specifies the name of the resource group from which you are removing nodes.

(Optional) From each node that you removed, uninstall the Sun Cluster Support for Oracle RAC software packages.

Use the Sun Java Enterprise System uninstaller for this purpose. For more information, see Chapter 8, Uninstalling, in Sun Java Enterprise System 5 Installation Guide for UNIX.

Reboot each node from which you from which you removed Sun Cluster Support for Oracle RAC.

Example 6–5 Removing Sun Cluster Support for Oracle RAC From Selected Nodes

This example shows the sequence of operations that is required to remove Sun Cluster Support for Oracle RAC from nodes pclus3 and pclus4 of a four-node cluster.

The configuration of Sun Cluster Support for Oracle RAC in this example is as follows:

The version of Oracle RAC is 10g R2.
Sun StorageTek QFS shared file system on Solaris Volume Manager for Sun Cluster is used to store Oracle files.
The mountpoints of file systems that are used for Oracle files are as follows:
- Oracle database files: /db_qfs/OraData
- Oracle binary files and related files: /db_qfs/OraHome
The name of the Oracle RAC database is swb.
The Sun StorageTek QFS shared file systems use a Solaris Volume Manager for Sun Cluster multi-owner disk set that is named oradg. The creation of this disk set is shown in Example 2–1.

The configuration of resource groups in this example is shown in the following table.

Resource Group	Purpose
`rac-framework-rg`	RAC framework resource group.
`scaldg-rg`	Resource group for scalable device-group resources.
`qfsmds-rg`	Resource group for Sun StorageTek QFS metadata server resources.
`scalmnt-rg`	Resource group for scalable file-system mount-point resources.
`rac_server_proxy-rg`	RAC database resource group.

The resource groups that are required for this configuration are shown in Figure A–2.

To remove nodes pclus3 and pclus4 from the resource group for the RAC database, the following commands are run:

# clresourcegroup offline -n pclus3,pclus4 rac_server_proxy-rg
# clresourcegroup remove-node -n pclus3,pclus4 rac_server_proxy-rg

To remove nodes pclus3 and pclus4 from the node list of the Oracle CRS resource for the Oracle RAC database, the following commands are run:

# /db_qfs/OraHome/crs/bin/crs_register ora.swb.swb3.inst \
-update -r "ora.pclus3.vip"
# /db_qfs/OraHome/crs/bin/crs_register ora.swb.swb4.inst \
-update -r "ora.pclus4.vip"

To remove from nodes pclus3 and pclus4 the Oracle CRS resources that represent Sun Cluster resources, the following commands are run:

Removal of resource for Oracle database files from node plcus3
# /db_qfs/OraHome/crs/bin/crs_stop sun.pclus3.scaloramnt-OraData-rs
# /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus3.scaloramnt-OraData-rs
# /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus3.scaloramnt-OraData-rs \
-dir /var/cluster/ucmm/profile
Removal of resource for Oracle binary files from node plcus3
# /db_qfs/OraHome/crs/bin/crs_stop sun.pclus3.scaloramnt-OraHome-rs
# /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus3.scaloramnt-OraHome-rs
# /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus3.scaloramnt-OraHome-rs \
-dir /var/cluster/ucmm/profile
Removal of resource for Oracle database files from node plcus4
# /db_qfs/OraHome/crs/bin/crs_stop sun.pclus4.scaloramnt-OraData-rs
# /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus4.scaloramnt-OraData-rs
# /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus4.scaloramnt-OraData-rs \
-dir /var/cluster/ucmm/profile
Removal of resource for Oracle binary files from node plcus4
# /db_qfs/OraHome/crs/bin/crs_stop sun.pclus4.scaloramnt-OraHome-rs
# /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus4.scaloramnt-OraHome-rs
# /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus4.scaloramnt-OraHome-rs \
-dir /var/cluster/ucmm/profile

The commands remove Oracle CRS resources that represent the following Sun Cluster resources:

scaloramnt-OraData-rs – A resource of type SUNW.ScalMountPoint that represents the mountpoint of the file system for database files
scaloramnt-OraHome-rs – A resource of type SUNW.ScalMountPoint that represents the mountpoint of the file system for binary files and associated files

After the resource is removed from nodes pclus3 and pclus4, Oracle utilities are used to remove the following items from these nodes:

The Oracle RAC database
The Oracle RAC software
The Oracle CRS software

The removal of these items is beyond the scope of this example.

To switch the resource group that contains resources for the Sun StorageTek QFS metadata server to node pclus1, the following command is run:
# clresourcegroup switch -n pclus1 qfsmds-rg
To remove nodes pclus3 and pclus4 from the resource group that contains scalable file-system mount-point resources, the following commands are run:
# clresourcegroup offline -n pclus3,pclus4 scalmnt-rg # clresourcegroup remove-node -n pclus3,pclus4 scalmnt-rg
To remove nodes pclus3 and pclus4 from the node list of the resource group that contains resources for the Sun StorageTek QFS metadata server, the following command is run:
# clresourcegroup remove-node -n pclus3,pclus4 qfsmds-rg
After nodes pclus3 and pclus4 are removed from the node list, the configuration of the Sun StorageTek QFS shared file systems is removed from theses nodes. This operation is beyond the scope of this example.
To remove nodes pclus3 and pclus4 from the resource group that contains scalable device group resources, the following commands are run:
# clresourcegroup offline -n pclus3,pclus4 scaldg-rg # clresourcegroup remove-node -n pclus3,pclus4 scaldg-rg
To remove nodes pclus3 and pclus4 from the Solaris Volume Manager for Sun Cluster multi-owner disk set oradg, the following command is run:
# metaset -s oradg -d -h pclus3 pclus4
To remove nodes pclus3 and pclus4 from the RAC framework resource group, the following commands are run:
# clresourcegroup offline -n pclus3,pclus4 rac-framework-rg # clresourcegroup remove-node -n pclus3,pclus4 rac-framework-rg
After the removal of nodes pclus3 and pclus4 from the RAC framework resource group, the following operations are performed:
- The removal of the Sun Cluster Support for Oracle RAC software packages from nodes pclus3 and pclus4
- The reboot of nodes pclus3 and pclus4
These operations are beyond the scope of this example.

After the removal is complete, the status of resource groups and resources is as follows:

# clresourcegroup status

=== Cluster Resource Groups ===

Group Name             Node Name    Suspended   Status
----------             ---------    ---------   ------
rac-framework-rg       pclus1       No          Online
                       pclus2       No          Online

scaldg-rg              pclus1       No          Online
                       pclus2       No          Online

qfsmds-rg              pclus1       No          Online
                       pclus2       No          Offline

scalmnt-rg             pclus1       No          Online
                       pclus2       No          Online

rac_server_proxy-rg    pclus1       No          Online
                       pclus2       No          Online

# clresource status

=== Cluster Resources ===

Resource Name            Node Name    State     Status Message
-------------            ---------    -----     --------------
rac-framework-rs         pclus1       Online    Online
                         pclus2       Online    Online

rac-udlm-rs              pclus1       Online    Online
                         pclus2       Online    Online

rac-svm-rs               pclus1       Online    Online
                         pclus2       Online    Online

crs_framework-rs         pclus1       Online    Online
                         pclus2       Online    Online

scaloradg-rs             pclus1       Online    Online - Diskgroup online
                         pclus2       Online    Online - Diskgroup online

qfs-mds-rs               pclus1       Online    Online - Service is online.
                         pclus2       Offline   Offline

scaloramnt-OraData-rs    pclus1       Online    Online
                         pclus2       Online    Online

scaloramnt-OraHome-rs    pclus1       Online    Online
                         pclus2       Online    Online

rac_server_proxy-rs      pclus1       Online    Online - Oracle instance UP
                         pclus2       Online    Online - Oracle instance UP

Chapter 6 Modifying an Existing Configuration of Sun Cluster Support for Oracle RAC

Overview of Tasks for Modifying an Existing Configuration of Sun Cluster Support for Oracle RAC

Modifying Online the Resource for a Scalable Device Group

How to Modify Online the Resource for a Scalable Device Group

Upgrading Sun Cluster Support for Oracle RAC

Upgrading Resources in the RAC Framework Resource Group

Information for Registering the New Versions of RAC Framework Resource Types

Information for Migrating Existing Instances of RAC Framework Resource Types

Example 6–1 Editing an Instance of the SUNW.rac_udlm Resource Type

How to Preserve a Nondefault Value of the reservation_timeout Extension Property

Example 6–2 Preserving a Nondefault Value of the reservation_timeout Extension Property

See Also

How to Remove the Resource of Type SUNW.rac_hwraid

Adding Storage Resources for Oracle Files

Adding Resources for Interoperation With Oracle 10g R2 CRS

Upgrading to Sun Cluster 3.2 Resource Types for Oracle 9i RAC Database Instances

How to Take Offline Resource Groups for Oracle RAC Database Components and Restart the Components

Before You Begin

Next Steps

How to Modify the Configuration of Sun Cluster Resources for Oracle 9i RAC Database Instances

Before You Begin

Extending an Existing Configuration of Sun Cluster Support for Oracle RAC

How to Add Sun Cluster Support for Oracle RAC to Selected Nodes

Before You Begin

Example 6–3 Adding Sun Cluster Support for Oracle RAC to Selected Nodes

How to Add a Volume Manager Resource to the RAC Framework Resource Group

Before You Begin

Next Steps

Removing Sun Cluster Support for Oracle RAC

How to Remove Sun Cluster Support for Oracle RAC From a Cluster

Before You Begin

Example 6–4 Removing Sun Cluster Support for Oracle RAC From a Cluster

How to Remove Sun Cluster Support for Oracle RAC From Selected Nodes

Example 6–5 Removing Sun Cluster Support for Oracle RAC From Selected Nodes

Example 6–1 Editing an Instance of the `SUNW.rac_udlm` Resource Type

How to Preserve a Nondefault Value of the `reservation_timeout` Extension Property

Example 6–2 Preserving a Nondefault Value of the `reservation_timeout` Extension Property

How to Remove the Resource of Type `SUNW.rac_hwraid`