How to Remove Support for Oracle RAC From Selected Nodes

Language:

Perform this task to remove Support for Oracle RAC from selected nodes.

On a cluster where multiple Support for Oracle RAC databases are running, perform this task to remove a Support for Oracle RAC database from selected nodes. The Support for Oracle RAC database that you remove continues to run on the other cluster nodes. The remaining Support for Oracle RAC databases continue to run on the selected nodes.

This task involves removing the selected nodes from the following resource groups in the following order:

The resource group for the Support for Oracle RAC database
Resource groups for logical hostname resources
Resource groups for scalable file-system mount-point resources
Resource groups that contain resources for the StorageTek QFS metadata server
Resource groups for scalable device group resources
The Support for Oracle RAC framework resource group

Caution - You might perform this task to remove a Support for Oracle RAC database from selected nodes of a cluster where multiple Support for Oracle RAC databases are running. In this situation, do not remove the nodes from any resource group on whose resources the remaining Support for Oracle RAC databases depend. For example, you might have configured multiple database file systems to depend on a single device group. In this situation, do not remove the nodes from the resource group that contains the resource for the scalable device group. Similarly, if multiple databases depend on the Support for Oracle RAC framework resource group, do not remove the nodes from this resource group.

Assume the root role.
Remove the nodes from the resource group for each Support for Oracle RAC database that you are removing.
For each Support for Oracle RAC database that you are removing, perform the following steps:
1. Take offline the resource group for the Support for Oracle RAC database on the nodes from which you are removing Support for Oracle RAC.
```
# clresourcegroup offline -n nodelist rac-db-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.
  
  rac-db-rg
  
  Specifies the name of the resource group that you are taking offline.
2. Remove the nodes from the node list of the resource group for the Support for Oracle RAC database.
```
# clresourcegroup remove-node -n nodelist rac-db-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  rac-db-rg
  
  Specifies the name of the resource group from which you are removing nodes.
Remove each node that you are removing from the list of nodes where the Oracle Grid Infrastructure resource for the Oracle database runs.
```
# Grid_home/bin/crsctl delete -n node-name
```
Grid_home

Specifies the Oracle Grid Infrastructure home directory. This directory contains the Oracle Grid Infrastructure binary files and Oracle Grid Infrastructure configuration files.

node-name

Specifies the host name of the node where the Oracle Grid Infrastructure resource runs.
From each node that you are removing, remove each Oracle Grid Infrastructure resource that represents an Oracle Solaris Cluster resource from whose resource group you are removing nodes.
An Oracle Grid Infrastructure resource is configured for each Oracle Solaris Cluster resource for scalable device groups and scalable file-system mount points on which Oracle Database components depend.

For each Oracle Grid Infrastructure resource that you are removing, perform the following steps on each node from which you are removing the resource:
1. Stop the Oracle Grid Infrastructure resource that you are removing.
```
# Grid_home/bin/crsctl stop
```
2. Delete the Oracle Grid Infrastructure resource that you want to remove.
```
# Grid_home/bin/crsctl delete res
```
Use Oracle utilities to remove the following items from each node from which you are removing Support for Oracle RAC:
- The Support for Oracle RAC database
- Oracle Grid Infrastructure

Switch any resource groups that contain resources for the StorageTek QFS metadata server to a node from which you are not removing Support for Oracle RAC.
If no configured resource groups contain resources for the StorageTek QFS metadata server, omit this step.
```
# clresourcegroup switch -n node-to-stay qfs-mds-rg
```
node-to-stay

Specifies the node to which you are switching the resource group. This node must be a node from which you are not removing Support for Oracle RAC.

qfs-mds-rg

Specifies the name of the resource group that you are switching to another node.
Remove the nodes from any resource groups that contain scalable file-system mount-point resources.
If no resource groups that contain scalable file-system mount-point resources are configured, omit this step.

For each resource group from which you are removing nodes, perform the following steps:
1. Take offline the resource group on the nodes from which you are removing Support for Oracle RAC.
```
# clresourcegroup offline -n nodelist scal-mp-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.
  
  scal-mp-rg
  
  Specifies the name of the resource group that you are taking offline.
2. Remove the nodes from the node list of the resource group.
```
# clresourcegroup remove-node -n nodelist scal-mp-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  scal-mp-rg
  
  Specifies the name of the resource group from which you are removing nodes.
Remove the nodes from the node list of any resource groups that contain resources for the StorageTek QFS metadata server.
If no resource groups that contain resources for the StorageTek QFS metadata server are configured, omit this step.

The resource groups to modify are the resource groups that you switched to another node in Step 6.
```
# clresourcegroup remove-node -n nodelist qfs-mds-rg
```
–n nodelist

Specifies a comma-separated list of cluster nodes that you are removing from the resource group.

qfs-mds-rg

Specifies the name of the resource group from which you are removing nodes.
Remove the configuration of the StorageTek QFS shared file systems from the nodes.
For instructions for performing this task, see Using Sun QFS and Sun Storage Archive Manager with Oracle Solaris Cluster .

Remove the nodes from any resource groups that contain scalable device group resources.
If no resource groups that contain scalable device group resources are configured, omit this step.

For each resource group from which you are removing nodes, perform the following steps:
1. Take offline the resource group on the nodes from which you are removing Support for Oracle RAC.
```
# clresourcegroup offline -n nodelist scal-dg-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.
  
  scal-dg-rg
  
  Specifies the name of the resource group that you are taking offline.
2. Remove the nodes from the node list of the resource group.
```
# clresourcegroup remove-node -n nodelist scal-dg-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  scal-dg-rg
  
  Specifies the name of the resource group from which you are removing nodes.
Remove the nodes from any scalable device groups that were affected by the removal of nodes from resource groups in Step 10.
How to perform this step depends on the type of the scalable device group.
- For each Solaris Volume Manager for Sun Cluster multi-owner disk set, type the following command:
```
# metaset -s scal-dg-ms -d -h nodelist
```
  -s scal-dg-ms
  
  Specifies the name of the disk set from which you are removing nodes.
  
  -h nodelist
  
  Specifies a space-separated list of the nodes that you are removing from the disk set.
Note - If you are removing a Support for Oracle RAC database from selected nodes of a cluster where multiple Support for Oracle RAC databases are running, omit the remaining steps in this procedure.

Remove the nodes from the Support for Oracle RAC framework resource group.
1. Take offline the resource group on the nodes from which you are removing Support for Oracle RAC.
```
# clresourcegroup offline -n nodelist rac-fmwk-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.
  
  rac-fmwk-rg
  
  Specifies the name of the resource group that you are taking offline.
2. Remove the nodes from the node list of the resource group.
```
# clresourcegroup remove-node -n nodelist rac-fmwk-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  rac-fmwk-rg
  
  Specifies the name of the resource group from which you are removing nodes.

Remove the nodes from the multiple-owner volume-manager framework resource group, if used.
1. Take offline the resource group on the nodes from which you are removing Support for Oracle RAC.
```
# clresourcegroup offline -n nodelist vucmm-fmwk-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.
  
  vucmm-fmwk-rg
  
  Specifies the name of the resource group that you are taking offline.
2. Remove the nodes from the node list of the resource group.
```
# clresourcegroup remove-node -n nodelist vucmm-fmwk-rg
```
  –n nodelist
  
  Specifies a comma-separated list of cluster nodes that you are removing from the resource group.
  
  vucmm-fmwk-rg
  
  Specifies the name of the resource group from which you are removing nodes.

(Optional) From each node that you removed, uninstall the Support for Oracle RAC software packages.
```
# pkg uninstall ha-cluster/data-service/oracle-database ha-cluster/library/ucmm
```
Reboot each node from which you removed Support for Oracle RAC.

Example 11 Removing Support for Oracle RAC From Selected Nodes

This example shows the sequence of operations that is required to remove Support for Oracle RAC from nodes pclus3 and pclus4 of a four-node cluster.

The configuration of Support for Oracle RAC in this example is as follows:

StorageTek QFS shared file system on Solaris Volume Manager for Sun Cluster is used to store Oracle Database files.
The mount points of file systems that are used for Oracle Database files are as follows:
- Oracle Database files: /db_qfs/OraData
- Oracle Database binary files and related files: /db_qfs/OraHome
The oradg disk set is used only by the Support for Oracle RAC database.
The name of the Support for Oracle RAC database is swb.
The StorageTek QFS shared file systems use a Solaris Volume Manager for Sun Cluster multi-owner disk set that is named oradg. The creation of this disk set is shown in Example 3, Creating a Multi-Owner Disk Set in Solaris Volume Manager for Sun Cluster.
The configuration uses a multiple-owner volume-manager framework resource group.

The resource groups that are required for this configuration are shown in Configuration of Support for Oracle RAC With Solaris Volume Manager for Sun Cluster and StorageTek QFS Shared File System.

Resource Group	Purpose
`vucmm-framework-rg`	Multiple-owner volume-manager resource group.
`rac-framework-rg`	Support for Oracle RAC framework resource group.
`scaldg-rg`	Resource group for scalable device-group resources.
`qfsmds-rg`	Resource group for StorageTek QFS metadata server resources.
`scalmnt-rg`	Resource group for scalable file-system mount-point resources.
`rac_server_proxy-rg`	Support for Oracle RAC database resource group.

The resource groups that are required for this configuration are shown in Configuration of Support for Oracle RAC With Solaris Volume Manager for Sun Cluster and StorageTek QFS Shared File System.

To remove nodes pclus3 and pclus4 from the resource group for the Support for Oracle RAC database, the following commands are run:

# clresourcegroup offline -n pclus3,pclus4 rac_server_proxy-rg
# clresourcegroup remove-node -n pclus3,pclus4 rac_server_proxy-rg

To remove nodes pclus3 and pclus4 from the node list of the Oracle Clusterware resource for the Support for Oracle RAC database, the following commands are run:

# /db_qfs/OraHome/crs/bin/crs_register ora.swb.swb3.inst \
-update -r "ora.pclus3.vip"
# /db_qfs/OraHome/crs/bin/crs_register ora.swb.swb4.inst \
-update -r "ora.pclus4.vip"

To remove from nodes pclus3 and pclus4 the Oracle Clusterware resources that represent Oracle Solaris Cluster resources, the following commands are run:

Removal of resource for Oracle database files from node plcus3
# /db_qfs/OraHome/crs/bin/crs_stop sun.pclus3.scaloramnt-OraData-rs
# /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus3.scaloramnt-OraData-rs
# /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus3.scaloramnt-OraData-rs \
-dir /var/cluster/ucmm/profile

Removal of resource for Oracle binary files from node plcus3
# /db_qfs/OraHome/crs/bin/crs_stop sun.pclus3.scaloramnt-OraHome-rs
# /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus3.scaloramnt-OraHome-rs
# /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus3.scaloramnt-OraHome-rs \
-dir /var/cluster/ucmm/profile

Removal of resource for Oracle database files from node plcus4
# /db_qfs/OraHome/crs/bin/crs_stop sun.pclus4.scaloramnt-OraData-rs
# /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus4.scaloramnt-OraData-rs
# /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus4.scaloramnt-OraData-rs \
-dir /var/cluster/ucmm/profile

Removal of resource for Oracle binary files from node plcus4
# /db_qfs/OraHome/crs/bin/crs_stop sun.pclus4.scaloramnt-OraHome-rs
# /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus4.scaloramnt-OraHome-rs
# /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus4.scaloramnt-OraHome-rs \
-dir /var/cluster/ucmm/profile

The commands remove Oracle Clusterware resources that represent the following Oracle Solaris Cluster resources:

scaloramnt-OraData-rs – A resource of type SUNW.ScalMountPoint that represents the mount point of the file system for database files
scaloramnt-OraHome-rs – A resource of type SUNW.ScalMountPoint that represents the mount point of the file system for binary files and associated files

After the resource is removed from nodes pclus3 and pclus4, Oracle Database utilities are used to remove the following items from these nodes:

The Support for Oracle RAC database
The Support for Oracle RAC software
The Oracle Clusterware software

The removal of these items is beyond the scope of this example.

To switch the resource group that contains resources for the StorageTek QFS metadata server to node pclus1, the following command is run:
```
# clresourcegroup switch -n pclus1 qfsmds-rg
```
To remove nodes pclus3 and pclus4 from the resource group that contains scalable file-system mount-point resources, the following commands are run:
```
# clresourcegroup offline -n pclus3,pclus4 scalmnt-rg
# clresourcegroup remove-node -n pclus3,pclus4 scalmnt-rg
```
To remove nodes pclus3 and pclus4 from the node list of the resource group that contains resources for the StorageTek QFS metadata server, the following command is run:
```
# clresourcegroup remove-node -n pclus3,pclus4 qfsmds-rg
```
After nodes pclus3 and pclus4 are removed from the node list, the configuration of the StorageTek QFS shared file systems is removed from theses nodes. This operation is beyond the scope of this example.
To remove nodes pclus3 and pclus4 from the resource group that contains scalable device group resources, the following commands are run:
```
# clresourcegroup offline -n pclus3,pclus4 scaldg-rg
# clresourcegroup remove-node -n pclus3,pclus4 scaldg-rg
```
To remove nodes pclus3 and pclus4 from the Solaris Volume Manager for Oracle Solaris Cluster multi-owner disk set oradg, the following command is run:
```
# metaset -s oradg -d -h pclus3 pclus4
```

To remove nodes pclus3 and pclus4 from the Support for Oracle RAC framework resource group, the following commands are run:

# clresourcegroup offline -n pclus3,pclus4 rac-framework-rg
# clresourcegroup remove-node -n pclus3,pclus4 rac-framework-rg

To remove nodes pclus3 and pclus4 from the multiple-owner volume-manager framework resource group, the following commands are run:

# clresourcegroup offline -n pclus3,pclus4 vucmm-framework-rg
# clresourcegroup remove-node -n pclus3,pclus4 vucmm-framework-rg

After the removal of nodes pclus3 and pclus4 from the framework resource groups, the Support for Oracle RAC software packages can optionally be removed from the nodes.

After the removal is complete, the status of resource groups and resources is as follows:

# clresourcegroup status

=== Cluster Resource Groups ===

Group Name             Node Name    Suspended   Status
----------             ---------    ---------   ------
rac-framework-rg       pclus1       No          Online
                       pclus2       No          Online

vucmm-framework-rg     pclus1       No          Online
                       pclus2       No          Online

scaldg-rg              pclus1       No          Online
                       pclus2       No          Online

qfsmds-rg              pclus1       No          Online
                       pclus2       No          Offline

scalmnt-rg             pclus1       No          Online
                       pclus2       No          Online

rac_server_proxy-rg    pclus1       No          Online
                       pclus2       No          Online

# clresource status

=== Cluster Resources ===

Resource Name            Node Name    State     Status Message
-------------            ---------    -----     --------------
rac-framework-rs         pclus1       Online    Online
                         pclus2       Online    Online

crs_framework-rs         pclus1       Online    Online
                         pclus2       Online    Online

vucmm-svm-rs             pclus1       Online    Online
                         pclus2       Online    Online

scaloradg-rs             pclus1       Online    Online - Diskgroup online
                         pclus2       Online    Online - Diskgroup online

qfs-mds-rs               pclus1       Online    Online - Service is online.
                         pclus2       Offline   Offline

scaloramnt-OraData-rs    pclus1       Online    Online
                         pclus2       Online    Online

scaloramnt-OraHome-rs    pclus1       Online    Online
                         pclus2       Online    Online

rac_server_proxy-rs      pclus1       Online    Online - Oracle instance UP
                         pclus2       Online    Online - Oracle instance UP