Sun Cluster Data Service for Oracle RAC Guide for Solaris OS

ProcedureHow to Remove Sun Cluster Support for Oracle RAC From Selected Nodes

Perform this task to remove Sun Cluster Support for Oracle RAC from selected nodes.

On a cluster where multiple Oracle RAC databases are running, perform this task to remove an Oracle RAC database from selected nodes. The Oracle RAC database that you remove continues to run on the other cluster nodes. The remaining Oracle RAC databases continue to run on the selected nodes.

This procedure provides the long forms of the Sun Cluster maintenance commands. Most commands also have short forms. Except for the forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix A, Sun Cluster Object-Oriented Commands, in Sun Cluster Data Services Planning and Administration Guide for Solaris OS.

This task involves removing the selected nodes from the following resource groups in the following order:


Caution – Caution –

You might perform this task to remove an Oracle RAC database from selected nodes of a cluster where multiple Oracle RAC databases are running. In this situation, do not remove the nodes from any resource group on whose resources the remaining Oracle RAC databases depend. For example, you might have configured multiple database file systems to depend on a single device group. In this situation, do not remove the nodes from the resource group that contains the resource for the scalable device group. Similarly, if multiple databases depend on the RAC framework resource group, do not remove the nodes from this resource group.


  1. Become superuser.

  2. (Oracle 9i and Oracle 10g R2 only) Remove the nodes from the resource group for each RAC database that you are removing.

    If you are using Oracle 10g R1, omit this step. For Oracle 10g R1, no resource groups for RAC databases are configured.

    For each RAC database that you are removing, perform the following steps:

    1. Take offline the resource group for the RAC database on the nodes from which you are removing Sun Cluster Support for Oracle RAC.


      # clresourcegroup offline -n nodelist rac-db-rg
      
      -n nodelist

      Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.

      rac-db-rg

      Specifies the name of the resource group that you are taking offline.

    2. Remove the nodes from the node list of the resource group for the RAC database.


      # clresourcegroup remove-node -n nodelist rac-db-rg
      
      -n nodelist

      Specifies a comma-separated list of cluster nodes that you are removing from the resource group.

      rac-db-rg

      Specifies the name of the resource group from which you are removing nodes.

  3. (Oracle 9i only) Remove the nodes from all resource groups for logical hostname resources that are used by each RAC database that you are removing.

    If you are using Oracle 10g R1 orOracle 10g R2 , omit this step. For Oracle 10g R1 and Oracle 10g R2, no resource groups for logical hostname resources are configured.

    For each resource group from which you are removing nodes, perform the following steps:

    1. Switch the resource group to a node from which you are not removing Sun Cluster Support for Oracle RAC.


      # clresourcegroup switch -n node-to-stay lh-rg
      
      node-to-stay

      Specifies the node to which you are switching the resource group. This node must be a node from which you are not removing Sun Cluster Support for Oracle RAC.

      lh-rg

      Specifies the name of the resource group that you are switching to another node.

    2. Remove the nodes from the node list of the resource group.


      # clresourcegroup remove-node -n nodelist lh-rg
      
      -n nodelist

      Specifies a comma-separated list of cluster nodes that you are removing from the resource group.

      lh-rg

      Specifies the name of the resource group from which you are removing nodes.

  4. (Oracle 9i only) Remove each resource group for logical hostname resources whose primary node you removed from resource groups in Step 3.

    These resource groups are no longer required because the RAC database instances that the groups serve are being removed.

    Do not remove any resource groups from which you removed only secondary nodes in Step 3.

    For each resource group that you are removing, type the following command:


    # clresourcegroup remove -F lh-rg-rm-prim
    
    lh-rg-rm-prim

    Specifies the name of the resource group that you are removing.

  5. (Oracle 10g R2 only) Remove each node that you are removing from the list of nodes where the Oracle CRS resource for the Oracle database runs.

    If you are using Oracle 9i or Oracle 10g R1 , omit this step. For Oracle 9i and Oracle 10g R1, no Oracle CRS resources that represent Sun Cluster resources are configured.


    Note –

    In this step, the syntax of Oracle commands for Oracle release 10g R2 is provided. If you are you are using a version Oracle other than 10g R2, see your Oracle documentation for the correct command syntax.



    # crs-home/bin/crs_register ora.dbname.sid.inst \
    -update -r "ora.node-name.vip"
    
    crs-home

    Specifies the Oracle CRS home directory. This directory contains the Oracle CRS binary files and Oracle CRS configuration files.

    dbname

    Specifies the database name of the database instance that the Oracle CRS resource represents.

    sid

    Specifies the Oracle SID of the database instance that the Oracle CRS resource represents.

    node-name

    Specifies the host name of the node where the Oracle CRS resource runs.

  6. (Oracle 10g R2 only) From each node that you are removing, remove each Oracle CRS resource that represents a Sun Cluster resource from whose resource group you are removing nodes.

    An Oracle CRS resource is configured for each Sun Cluster resource for scalable device groups and scalable file-system mountpoints on which Oracle components depend.

    If you are using Oracle 9i or Oracle 10g R1 , omit this step. For Oracle 9i and Oracle 10g R1, no Oracle CRS resources that represent Sun Cluster resources are configured.


    Note –

    In this step, the syntax of Oracle commands for Oracle release 10g R2 is provided. If you are you are using a version Oracle other than 10g R2, see your Oracle documentation for the correct command syntax.


    For each Oracle CRS resource that you are removing, perform the following steps on each node from which you are removing the resource:

    1. Stop the Oracle CRS resource that you are removing.


      # crs-home/bin/crs_stop sun.node-name.sc-rs
      
      crs-home

      Specifies the Oracle CRS home directory. This directory contains the Oracle CRS binary files and Oracle CRS configuration files.

      node-name

      Specifies the host name of the node where the Oracle CRS resource runs.

      sc-rs

      Specifies the name of the Sun Cluster resource that the Oracle CRS resource represents.

    2. Unregister the Oracle CRS resource that you are removing.


      # crs-home/bin/crs_unregister sun.node-name.sc-rs
      
      crs-home

      Specifies the Oracle CRS home directory. This directory contains the Oracle CRS binary files and Oracle CRS configuration files.

      node-name

      Specifies the host name of the node where the Oracle CRS resource runs.

      sc-rs

      Specifies the name of the Sun Cluster resource that the Oracle CRS resource represents.

    3. Delete the profile for the Oracle CRS resource that you are removing.


      # crs-home/bin/crs_profile -delete sun.node-name.sc-rs \
      -dir /var/cluster/ucmm/profile
      
      crs-home

      Specifies the Oracle CRS home directory. This directory contains the Oracle CRS binary files and Oracle CRS configuration files.

      node-name

      Specifies the host name of the node where the Oracle CRS resource runs.

      sc-rs

      Specifies the name of the Sun Cluster resource that the Oracle CRS resource represents.

  7. Use Oracle utilities to remove the following items from each node from which you are removing Sun Cluster Support for Oracle RAC:

    • The RAC database

    • Oracle CRS

  8. Switch any resource groups that contain resources for the Sun StorageTek QFS metadata server to a node from which you are not removing Sun Cluster Support for Oracle RAC.

    If no resource groups that contain resources for the Sun StorageTek QFS metadata server are configured, omit this step.


    # clresourcegroup switch -n node-to-stay qfs-mds-rg
    
    node-to-stay

    Specifies the node to which you are switching the resource group. This node must be a node from which you are not removing Sun Cluster Support for Oracle RAC.

    qfs-mds-rg

    Specifies the name of the resource group that you are switching to another node.

  9. Remove the nodes from any resource groups that contain scalable file-system mount-point resources.

    If no resource groups that contain scalable file-system mount-point resources are configured, omit this step.

    For each resource group from which you are removing nodes, perform the following steps:

    1. Take offline the resource group on the nodes from which you are removing Sun Cluster Support for Oracle RAC.


      # clresourcegroup offline -n nodelist scal-mp-rg
      
      -n nodelist

      Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.

      scal-mp-rg

      Specifies the name of the resource group that you are taking offline.

    2. Remove the nodes from the node list of the resource group.


      # clresourcegroup remove-node -n nodelist scal-mp-rg
      
      -n nodelist

      Specifies a comma-separated list of cluster nodes that you are removing from the resource group.

      scal-mp-rg

      Specifies the name of the resource group from which you are removing nodes.

  10. Remove the nodes from the node list of any resource groups that contain resources for the Sun StorageTek QFS metadata server.

    If no resource groups that contain resources for the Sun StorageTek QFS metadata server are configured, omit this step.

    The resource groups to modify are the resource groups that you switched to another node in Step 8.


    # clresourcegroup remove-node -n nodelist qfs-mds-rg
    
    -n nodelist

    Specifies a comma-separated list of cluster nodes that you are removing from the resource group.

    qfs-mds-rg

    Specifies the name of the resource group from which you are removing nodes.

  11. Remove the configuration of the Sun StorageTek QFS shared file systems from the nodes.

    For instructions for performing this task, see Sun StorageTek QFS File System Configuration and Administration Guide, Version 4, Update 6.

  12. Remove the nodes from any resource groups that contain scalable device group resources.

    If no resource groups that contain scalable device group resources are configured, omit this step.

    For each resource group from which you are removing nodes, perform the following steps:

    1. Take offline the resource group on the nodes from which you are removing Sun Cluster Support for Oracle RAC.


      # clresourcegroup offline -n nodelist scal-dg-rg
      
      -n nodelist

      Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.

      scal-dg-rg

      Specifies the name of the resource group that you are taking offline.

    2. Remove the nodes from the node list of the resource group.


      # clresourcegroup remove-node -n nodelist scal-dg-rg
      
      -n nodelist

      Specifies a comma-separated list of cluster nodes that you are removing from the resource group.

      scal-dg-rg

      Specifies the name of the resource group from which you are removing nodes.

  13. Remove the nodes from any scalable device groups that were affected by the removal of nodes from resource groups in Step 12.

    How to perform this step depends on the type of the scalable device group.

    • For each Solaris Volume Manager for Sun Cluster multi-owner disk set, type the following command:


      # metaset -s scal-dg-ms -d -h nodelist
      
      -s scal-dg-ms

      Specifies the name of the disk set from which you are removing nodes.

      -h nodelist

      Specifies a space-separated list of the nodes that you are removing from the disk set.

    • For each VxVM shared-disk group, use Veritas commands to remove the nodes from the VxVM shared-disk group.

      For more information, see your VxVM documentation.


    Note –

    If you are removing an Oracle RAC database from selected nodes of a cluster where multiple Oracle RAC databases are running, omit the remaining steps in this procedure.


  14. Remove the nodes from the RAC framework resource group.

    1. Take offline the resource group on the nodes from which you are removing Sun Cluster Support for Oracle RAC.


      # clresourcegroup offline -n nodelist rac-fmwk-rg
      
      -n nodelist

      Specifies a comma-separated list of cluster nodes from which you are taking offline the resource group.

      rac-fmwk-rg

      Specifies the name of the resource group that you are taking offline.

    2. Remove the nodes from the node list of the resource group.


      # clresourcegroup remove-node -n nodelist rac-fmwk-rg
      
      -n nodelist

      Specifies a comma-separated list of cluster nodes that you are removing from the resource group.

      rac-fmwk-rg

      Specifies the name of the resource group from which you are removing nodes.

  15. (Optional) From each node that you removed, uninstall the Sun Cluster Support for Oracle RAC software packages.

    Use the Sun Java Enterprise System uninstaller for this purpose. For more information, see Chapter 8, Uninstalling, in Sun Java Enterprise System 5 Installation Guide for UNIX.

  16. Reboot each node from which you from which you removed Sun Cluster Support for Oracle RAC.


Example 6–5 Removing Sun Cluster Support for Oracle RAC From Selected Nodes

This example shows the sequence of operations that is required to remove Sun Cluster Support for Oracle RAC from nodes pclus3 and pclus4 of a four-node cluster.

The configuration of Sun Cluster Support for Oracle RAC in this example is as follows:

The configuration of resource groups in this example is shown in the following table.

Resource Group 

Purpose 

rac-framework-rg

RAC framework resource group. 

scaldg-rg

Resource group for scalable device-group resources. 

qfsmds-rg

Resource group for Sun StorageTek QFS metadata server resources. 

scalmnt-rg

Resource group for scalable file-system mount-point resources. 

rac_server_proxy-rg

RAC database resource group. 

The resource groups that are required for this configuration are shown in Figure A–2.

  1. To remove nodes pclus3 and pclus4 from the resource group for the RAC database, the following commands are run:


    # clresourcegroup offline -n pclus3,pclus4 rac_server_proxy-rg
    # clresourcegroup remove-node -n pclus3,pclus4 rac_server_proxy-rg
    
  2. To remove nodes pclus3 and pclus4 from the node list of the Oracle CRS resource for the Oracle RAC database, the following commands are run:


    # /db_qfs/OraHome/crs/bin/crs_register ora.swb.swb3.inst \
    -update -r "ora.pclus3.vip"
    # /db_qfs/OraHome/crs/bin/crs_register ora.swb.swb4.inst \
    -update -r "ora.pclus4.vip"
    
  3. To remove from nodes pclus3 and pclus4 the Oracle CRS resources that represent Sun Cluster resources, the following commands are run:


    Removal of resource for Oracle database files from node plcus3
    # /db_qfs/OraHome/crs/bin/crs_stop sun.pclus3.scaloramnt-OraData-rs
    # /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus3.scaloramnt-OraData-rs
    # /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus3.scaloramnt-OraData-rs \
    -dir /var/cluster/ucmm/profile
    Removal of resource for Oracle binary files from node plcus3
    # /db_qfs/OraHome/crs/bin/crs_stop sun.pclus3.scaloramnt-OraHome-rs
    # /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus3.scaloramnt-OraHome-rs
    # /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus3.scaloramnt-OraHome-rs \
    -dir /var/cluster/ucmm/profile
    Removal of resource for Oracle database files from node plcus4
    # /db_qfs/OraHome/crs/bin/crs_stop sun.pclus4.scaloramnt-OraData-rs
    # /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus4.scaloramnt-OraData-rs
    # /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus4.scaloramnt-OraData-rs \
    -dir /var/cluster/ucmm/profile
    Removal of resource for Oracle binary files from node plcus4
    # /db_qfs/OraHome/crs/bin/crs_stop sun.pclus4.scaloramnt-OraHome-rs
    # /db_qfs/OraHome/crs/bin/crs_unregister sun.pclus4.scaloramnt-OraHome-rs
    # /db_qfs/OraHome/crs/bin/crs_profile -delete sun.pclus4.scaloramnt-OraHome-rs \
    -dir /var/cluster/ucmm/profile
    

    The commands remove Oracle CRS resources that represent the following Sun Cluster resources:

    • scaloramnt-OraData-rs – A resource of type SUNW.ScalMountPoint that represents the mountpoint of the file system for database files

    • scaloramnt-OraHome-rs – A resource of type SUNW.ScalMountPoint that represents the mountpoint of the file system for binary files and associated files

    After the resource is removed from nodes pclus3 and pclus4, Oracle utilities are used to remove the following items from these nodes:

    • The Oracle RAC database

    • The Oracle RAC software

    • The Oracle CRS software

    The removal of these items is beyond the scope of this example.

  4. To switch the resource group that contains resources for the Sun StorageTek QFS metadata server to node pclus1, the following command is run:


    # clresourcegroup switch -n pclus1 qfsmds-rg
    
  5. To remove nodes pclus3 and pclus4 from the resource group that contains scalable file-system mount-point resources, the following commands are run:


    # clresourcegroup offline -n pclus3,pclus4 scalmnt-rg
    # clresourcegroup remove-node -n pclus3,pclus4 scalmnt-rg
    
  6. To remove nodes pclus3 and pclus4 from the node list of the resource group that contains resources for the Sun StorageTek QFS metadata server, the following command is run:


    # clresourcegroup remove-node -n pclus3,pclus4 qfsmds-rg
    

    After nodes pclus3 and pclus4 are removed from the node list, the configuration of the Sun StorageTek QFS shared file systems is removed from theses nodes. This operation is beyond the scope of this example.

  7. To remove nodes pclus3 and pclus4 from the resource group that contains scalable device group resources, the following commands are run:


    # clresourcegroup offline -n pclus3,pclus4 scaldg-rg
    # clresourcegroup remove-node -n pclus3,pclus4 scaldg-rg
    
  8. To remove nodes pclus3 and pclus4 from the Solaris Volume Manager for Sun Cluster multi-owner disk set oradg, the following command is run:


    # metaset -s oradg -d -h pclus3 pclus4
    
  9. To remove nodes pclus3 and pclus4 from the RAC framework resource group, the following commands are run:


    # clresourcegroup offline -n pclus3,pclus4 rac-framework-rg
    # clresourcegroup remove-node -n pclus3,pclus4 rac-framework-rg
    

    After the removal of nodes pclus3 and pclus4 from the RAC framework resource group, the following operations are performed:

    • The removal of the Sun Cluster Support for Oracle RAC software packages from nodes pclus3 and pclus4

    • The reboot of nodes pclus3 and pclus4

    These operations are beyond the scope of this example.

After the removal is complete, the status of resource groups and resources is as follows:


# clresourcegroup status

=== Cluster Resource Groups ===

Group Name             Node Name    Suspended   Status
----------             ---------    ---------   ------
rac-framework-rg       pclus1       No          Online
                       pclus2       No          Online

scaldg-rg              pclus1       No          Online
                       pclus2       No          Online

qfsmds-rg              pclus1       No          Online
                       pclus2       No          Offline

scalmnt-rg             pclus1       No          Online
                       pclus2       No          Online

rac_server_proxy-rg    pclus1       No          Online
                       pclus2       No          Online

# clresource status

=== Cluster Resources ===

Resource Name            Node Name    State     Status Message
-------------            ---------    -----     --------------
rac-framework-rs         pclus1       Online    Online
                         pclus2       Online    Online

rac-udlm-rs              pclus1       Online    Online
                         pclus2       Online    Online

rac-svm-rs               pclus1       Online    Online
                         pclus2       Online    Online

crs_framework-rs         pclus1       Online    Online
                         pclus2       Online    Online

scaloradg-rs             pclus1       Online    Online - Diskgroup online
                         pclus2       Online    Online - Diskgroup online

qfs-mds-rs               pclus1       Online    Online - Service is online.
                         pclus2       Offline   Offline

scaloramnt-OraData-rs    pclus1       Online    Online
                         pclus2       Online    Online

scaloramnt-OraHome-rs    pclus1       Online    Online
                         pclus2       Online    Online

rac_server_proxy-rs      pclus1       Online    Online - Oracle instance UP
                         pclus2       Online    Online - Oracle instance UP