Skip Headers
Oracle® Real Application Clusters Administration and Deployment Guide
11g Release 1 (11.1)

B28254-07
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Master Index
Master Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

10 Adding and Deleting Nodes and Instances on Windows Systems

This chapter describes how to add and delete nodes and instances in Oracle Real Application Clusters (Oracle RAC) databases on Windows systems. The preferred method to add nodes and instances to Oracle RAC databases is to use the Oracle cloning procedures. Chapter 5, "Using Cloning to Add ASM and Oracle RAC to Nodes in a New Cluster" describes the cloning procedures in detail. You can also use cloning to copy images of Oracle Clusterware, ASM, and Oracle RAC software onto the other nodes that have identical hardware and software. If you do not want to use the cloning process, then you can use the manual procedures that are described in this chapter to add nodes and instances.

The topics in this chapter are:

Note:

For all of the add node and delete node procedures, temporary directories such as %TEMP% or C:\Temp should not be shared directories. If your temporary directories are shared, then set your temporary environment variable, such as %TEMP%, to a non-shared location on a local node. In addition, use a directory that exists on all of the nodes.

Note:

In this chapter, the entries for CRS_home refer to the full path name for the Oracle Clusterware home, and the entries for Oracle_home or $ORACLE_HOME refer to substitutes for environment variables for the Oracle home with Oracle RAC.

See Also:

Oracle Database 2 Day + Real Application Clusters Guide for additional information about configuring a new Oracle RAC cluster or scaling up an existing Oracle RAC cluster.

Cloning Oracle Clusterware and Oracle RAC Software in Grid Environments

The preferred method for extending your Oracle RAC environment is to use the Oracle cloning procedures in Chapter 5. Only use the procedures in this chapter if you choose not to use Oracle cloning.

Quick-Start Node and Database Instance Addition and Deletion Procedures

This section explains node and instance addition and deletion using procedures that are presented in a quick-start format. For node addition, these procedures require that you install the required operating system patches and updates on the new nodes. Then configure the new nodes to be part of your cluster at the network level. Extend the Oracle Clusterware home from an existing Oracle Clusterware home to the new nodes and then extend the Oracle database software with Oracle RAC components to the new nodes. Finally, make the new nodes members of the existing Oracle RAC database. This section includes the following topics:

Note:

For all of the procedures in this chapter, it is very important that you perform each step in the order shown.

Note:

In some cases, your current configuration may not be compatible with an ASM activity that you are trying to perform, either explicitly or with an automatic ASM extension to other nodes. If you are using DBCA to build a database using a new Oracle home, and if the ASM version is from an earlier release of the Oracle software but does not exist on all of the nodes you selected for the database, then ASM cannot be extended. Instead, the DBCA session displays an error, prompting you either to run the add node script or to upgrade ASM using the DBUA.

See Also:

Oracle Real Application Clusters Installation Guide for Microsoft Windows for procedures about using the Database Configuration Assistant (DBCA) to create and delete Oracle RAC databases

Adding an Oracle Clusterware Home to a New Node

Use one of the procedures described in this section to use Oracle Universal Installer to add an Oracle Clusterware home to a node that is going to be part of your Oracle RAC cluster. These procedures assume that you are adding node2 and that you have already successfully installed Oracle Clusterware on node1 in a nonshared home, where CRS_home represents the successfully installed home.

Note:

Oracle recommends that you back up your voting disk and Oracle Cluster Registry files after you complete any node addition or deletion procedures.

Adding an Oracle Clusterware Home to a New Node Using Oracle Universal Installer in Interactive Mode

Use the following procedure to add an Oracle Clusterware home to a new node, node2, to your cluster with an interactive Oracle Universal Installer session.

  1. Go to CRS_home\oui\bin on node1 and run the addnode.bat script.

  2. The Oracle Universal Installer displays the Node Selection page on which you must specify the node, private node name, and the VIP address for the new node that you are adding and click Next.

  3. Verify the contents that Oracle Universal Installer displays on the summary page and click Next. This installs the Oracle Clusterware home with Oracle RAC on the new node.

  4. From CRS_home\install on node1, run the crssetup.add.bat script.You

Adding an Oracle Clusterware Home to a New Node Using Oracle Universal Installer in Silent Mode

Use the following procedure to add an Oracle Clusterware home to a new node, node2, to your cluster with a silent Oracle Universal Installer session.

  1. Go to CRS_home\oui\bin on node1 and run the addnode.bat script as follows:

    addnode.bat –silent "CLUSTER_NEW_NODES={node2}" 
    "CLUSTER_NEW_PRIVATE_NODE_NAMES={node2-priv}" 
    "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node2-vip}" 
    
  2. Perform step 4 from the previous section about using Oracle Universal Installer interactively under the heading "Adding an Oracle Clusterware Home to a New Node Using Oracle Universal Installer in Interactive Mode" to add node2 to the cluster.

Adding an Oracle Home with Oracle Real Application Clusters to a New Node

Use one of the following procedures to add an Oracle home to a new node using Oracle Universal Installer. These procedures assume that you want to extend an existing Oracle home with Oracle RAC on node1 to node2. Node2 should already be a member node of the cluster to which node1 belongs. The contents of these procedures assume that you have successfully installed Oracle RAC on node1 in a non-shared home and that Oracle_home represents the successfully installed Oracle home. This section describes the following procedures:

Note:

Oracle recommends that you back up your voting disk and Oracle Cluster Registry files after you complete any node addition or deletion procedures.

Adding an Oracle Home with Oracle RAC to a New Node Using Oracle Universal Installer in Interactive Mode

Use the following procedure to add an Oracle home to a new node, node2, using Oracle Universal Installer in interactive mode.

  1. Go to Oracle_home\oui\bin and run the addnode.bat script.

  2. The Oracle Universal Installer displays the Node Selection page on which you should select the nodes that you want to add and then click Next.

  3. Verify the contents that Oracle Universal Installer displays on the Summary page and click Next. This installs the Oracle home with Oracle RAC on the new node.

  4. On node2, run NETCA to add a listener.

    See Also:

    Oracle Database Net Services Administrator's Guide for more information about NETCA
  5. To add a database instance to the new node, use Oracle Enterprise Manager or DBCA as described under the heading "Step 5: Adding Database Instances to New Nodes"

Adding an Oracle Home with Oracle RAC to a New Node Using Oracle Universal Installer in Silent Mode

Use the following procedure to add an Oracle home to a new node, node2, using Oracle Universal Installer in interactive mode.

  1. Go to Oracle_home\oui\bin and run the addnode.bat script as follows:

    addnode.bat –silent "CLUSTER_NEW_NODES={node2}" 
    
  2. Perform steps 4 and 5 from the previous section for using Oracle Universal Installer in interactive mode under the heading "Adding an Oracle Home with Oracle RAC to a New Node Using Oracle Universal Installer in Interactive Mode" to add the listener and to create the database instance for the new node.

Note:

The same steps are applicable for node additions using a shared home.

Deleting an Oracle Home with Oracle Real Application Clusters from an Existing Node

Use the following procedures to delete nodes at the database Oracle home layer on Windows systems. These procedures assume that an Oracle RAC database home is installed on both node1 and node2 and that you want to delete the database home on node2. Also assume that Oracle_home represents the location of this home on the nodes. The procedures in this section are:

Note:

Oracle recommends that you back up your voting disk and Oracle Cluster Registry files after you complete any node addition or deletion procedures.

Deleting an Oracle Home with Oracle RAC Using Oracle Universal Installer in Interactive Mode

Use the following procedures to use Oracle Universal Installer to delete an Oracle home:

  1. Remove any instances that are on node2 with Oracle Enterprise Manager or DBCA as described under the heading "Step 1: Deleting Instances from Oracle Real Application Clusters Databases". Only delete the database instance on node2 that is associated with the Oracle home that you are deleting. Other instances may exist on node2 that you do not want to delete.

  2. If you use ASM and this is the home from which the ASM instance runs, then perform the ASM instance clean-up procedures under the heading "Step 3: ASM Instance Clean-Up Procedures for Node Deletion".

  3. If this is the home from which the node-specific listener runs, then use NETCA to remove the listener and its Oracle Clusterware resources. If necessary, re-create this listener in other homes.

    See Also:

    Oracle Database Net Services Administrator's Guide for more information about NETCA
  4. On the node to be deleted, which in this example is node2, run the following command from Oracle_home\oui\bin:

    setup.exe –updateNodeList ORACLE_HOME=Oracle_home 
    "CLUSTER_NODES={node2}" –local 
    
  5. On the node to be deleted, node2 in this case:

    • If the home is a nonshared home, then de-install the database by running the setup.exe script from Oracle_home\oui\bin.

    • If the home is a shared home then do not perform a de-install. Instead, perform the following steps on the node to be deleted:

      • Perform a detach home on node2 by running the following command from Oracle_home\oui\bin on node2:

        setup.exe –detachHome –silent ORACLE_HOME=Oracle_home
        
      • On node 2, stop and delete any services that are associated with this Oracle home. In addition, delete any Registry entries and path entries that are associated with this Oracle home. Also, delete all of the start menu items associated with this Oracle home.

  6. On node1, or in the case of an installation with more than two nodes on any node other than the node to be deleted, run the following command from Oracle_home\oui\bin where node_list is a comma-delimited list of nodes that are to remain part of the Oracle RAC installation:

    setup.exe –updateNodeList ORACLE_HOME=Oracle_home 
    "CLUSTER_NODES={node_list}" 
    

Deleting an Oracle Home with Oracle RAC Using Oracle Universal Installer in Silent Mode

Use the following procedures to delete an Oracle home by using Oracle Universal Installer in silent mode:

  1. Perform steps 1 through 4 from the previous Oracle Universal Installer procedure for node deletion.

  2. On node2, run Oracle Universal Installer by running the setup.exe script from Oracle_home\oui\bin:

    • If you have a non-shared home, then de-install the Oracle home as follows:

      setup.exe –silent  -deinstall "REMOVE_HOMES={Oracle_home}"
      
    • If you have a shared home, then do not perform a de-install. Instead, perform the following steps on the node that you want to delete:

      • Run the following command from Oracle_home\oui\bin to detach the Oracle home on node2:

        setup.exe –detachHome –silent ORACLE_HOME=Oracle_home
        
      • On node 2, stop and delete any services that are associated with this Oracle home. In addition, delete any Registry entries and path entries that are associated with this Oracle home. Also, delete all of the start menu items associated with this Oracle home.

  3. Perform step 6 from the previous Oracle Universal Installer procedure to update the node list.

Deleting an Oracle Clusterware Home from an Existing Node

Use the following procedures to delete nodes at the Oracle Clusterware layer on Windows systems. These procedures assume that an Oracle Clusterware home is installed on node1 and node2, and that you want to delete node2 from the cluster. The topics in this section are:

Note:

Oracle recommends that you back up your voting disk and Oracle Cluster Registry files after you complete any node addition or deletion procedures.

Deleting an Oracle Clusterware Home Using Oracle Universal Installer in Interactive Mode

Perform the following procedure to use Oracle Universal Installer to delete a node at the clusterware layer:

  1. Perform the delete node operation for database homes as described in the section titled "Deleting an Oracle Home with Oracle RAC Using Oracle Universal Installer in Interactive Mode".

  2. If you did not run the oifcfg command with the -global option, run the following command from node1:

    oifcfg delif –node node2
    
  3. From node1 in the CRS_home\bin directory run the following command where remote_port is the ONS remote port number:

    racgons remove_config node2:remote_port
    

    You can determine the remote port by viewing the contents of the file, CRS_home\opmn\conf\ons.config.

  4. Run srvctl to stop and remove the nodeapps from node 2. From CRS_home/bin run the following commands:

    srvctl stop nodeapps –n node2
    srvctl remove nodeapps –n node2
    
  5. On node1, or on any node that is not being deleted, run the following command from CRS_home\bin where node_name is the node to be deleted and node_number is the node's number as obtained from the output from the olsnodes -n command:

    crssetup del –nn node_name,node_number
    
  6. On the node(s) to be deleted (node2 in this case), run the following command from CRS_home\oui\bin:

    setup.exe –updateNodeList ORACLE_HOME=CRS_home 
    "CLUSTER_NODES={node2}" CRS=TRUE –local
    
  7. On node2, using Oracle Universal Installer setup.exe from CRS_home\oui\bin:

    • If you do not have a shared home, then deinstall the Oracle Clusterware installation by running the setup.exe script from CRS_home\oui\bin.

    • If you have a shared home, then do not perform a deinstallation. Instead, perform the following steps on the node that you want to delete:

      • Run the following command from CRS_home\oui\bin to detach the Oracle Clusterware home on node2:

        setup.exe –detachHome –silent ORACLE_HOME=CRS_home
        
      • On node2, stop and delete any services that are associated with this Oracle Clusterware home. In addition, delete any Registry entries and path entries that are associated with this Oracle Clusterware home. Also, delete all of the start menu items associated with this Oracle Clusterware home. Delete the central inventory and the files (not Oracle_home files) under C:\WINDOWS\system32\drivers.

  8. On node1, or in the case of a multiple node installation, on any node other than the one to be deleted, run the following command from CRS_home\oui\bin where node_list is a comma-delimited list of nodes that are to remain part of the Oracle Clusterware:

    setup.exe –updateNodeList ORACLE_HOME=CRS_home
    "CLUSTER_NODES={node_list}" CRS=TRUE
    

Deleting an Oracle Clusterware Home Using Oracle Universal Installer in Silent Mode

Use the following procedure to delete an Oracle Clusterware home by using Oracle Universal Installer in silent mode:

  1. Perform steps 1 through 6 from the previous section for using Oracle Universal Installer interactively under the heading "Deleting an Oracle Clusterware Home Using Oracle Universal Installer in Interactive Mode" to delete nodes at the Oracle Clusterware layer.

  2. On node2, using Oracle Universal Installer setup.exe from CRS_home\oui\bin:

    • If you have a non-shared home, then de-install the Oracle Clusterware home as follows:

      setup.exe –silent  -deinstall "REMOVE_HOMES={CRS_home}"
      
    • If you have a shared home, then do not perform a de-installation. Instead, perform the following steps on the node that you want to delete:

      • Run the following command from CRS_home\oui\bin to detach the Oracle Clusterware home on node2:

        setup.exe –detachHome –silent ORACLE_HOME=CRS_home
        
      • On node 2, stop and delete any services that are associated with this Oracle Clusterware home. In addition, delete any Registry entries and path entries that are associated with this Oracle Clusterware home. Also, delete all of the start menu items associated with this Oracle Clusterware home. Delete the central inventory and the files (not including CRS_home) under C:\WINDOWS\system32\drivers.

  3. Perform step 8 from the previous Oracle Universal Installer procedure to update the node list.

Detailed Node and Database Instance Addition and Deletion Procedures

This section provides detailed procedures for adding and deleting nodes to clusters. The details of these steps appear in the following sections:

See Also:

Oracle Real Application Clusters Installation Guide for Microsoft Windows for procedures about using the DBCA to create and delete Oracle RAC databases

Also, see "Extending ASM to Nodes Running Single-Instance or Oracle RAC Databases" for complete information about adding ASM instances to new nodes

Overview of Node Addition Procedures

This section explains how to add nodes to clusters using detailed manual procedures. Do this by setting up the new nodes to be part of your cluster at the network level. Then extend the Oracle Clusterware home from an existing Oracle Clusterware home to the new nodes and then extend the Oracle database software with Oracle RAC components to the new nodes. Finally, make the new nodes members of the existing Oracle RAC database.

If the nodes that you are adding to your cluster do not have Oracle Clusterware or Oracle software installed on them, then complete the following five steps. The procedures in these steps assume that you already have an operative Windows Oracle RAC environment. The details of these steps appear in the following sections.

To add a node to your cluster when the node is already configured with Oracle Clusterware and Oracle software, follow the procedure described in "Adding Nodes That Already Have Oracle Clusterware and Oracle Database Software".

See Also:

Oracle Real Application Clusters Installation Guide for Microsoft Windows for procedures about using DBCA to create and delete Oracle RAC databases

Also, see "Extending ASM to Nodes Running Single-Instance or Oracle RAC Databases" for complete information about adding ASM instances to new nodes

Step 1: Connecting New Nodes to the Cluster

Complete the following procedures to connect the new nodes to the cluster and to prepare them to support your Oracle RAC database:

Making Physical Connections

Connect the new nodes' hardware to the network infrastructure of your cluster. This includes establishing electrical connections, configuring network interconnects, configuring shared disk subsystem connections, and so on. See your hardware vendor documentation for details about this step.

Installing the Operating System

As the Windows Administrator, install a cloned image of the operating system that matches the operating system on the other nodes in your cluster. This includes installing any required service patches and drivers. See your hardware vendor documentation for details about this process.

Verifying the Installation with the Cluster Verification Utility

Verify your installation using the Cluster Verification Utility (CVU) as in the following steps:

  1. From the directory CRS_home\bin on the existing nodes, run the CVU command to verify your installation at the post hardware installation stage where node_list is a list of the nodes to include in the CVU operation as shown in the following example:

    cluvfy stage -post hwos -n node_list|all [-verbose]
    

    After you have configured the hardware and operating systems on the new nodes, you can use this command to verify node reachability, for example, to all of the nodes from the local node. You can also use this command to verify user equivalence to all given nodes from the local node, node connectivity among all of the given nodes, accessibility to shared storage from all of the given nodes, and so on.

    Note:

    You can only use the all option with the -n argument if you have set the CV_NODE_ALL variable to represent the list of nodes on which you want to perform the CVU operation.

    See Also:

    Appendix B, "Troubleshooting Oracle Real Application Clusters" for more information about enabling and using CVU
  2. From the directory CRS_home\bin on the existing nodes, run the CVU command to obtain a detailed comparison of the properties of the reference node with all of the other nodes that are part of your current cluster environment where node is the node, node_list is a list of the nodes to include, orainventory_group is the name of the Oracle inventory group, and osdba_group is the DBA group name:

    cluvfy comp peer [ -refnode node ] -n node_list [-r { 10gR1 | 10gR2 } ] [ -orainv orainventory_group ] [ -osdba osdba_group ] [-verbose]
    

    Note:

    For the reference node, select a node from your existing cluster nodes against which you want CVU to compare, for example, the newly added nodes that you specify with the -n option.

Checking the Installation

To verify that your installation is configured correctly, perform the following steps:

  1. Ensure that the new nodes can access the private interconnect. This interconnect must be properly configured before you can complete the procedures in "Step 2: Extending Oracle Software to New Nodes at the Oracle Clusterware Layer".

  2. If you are not using a cluster file system, then determine the location on which your cluster software was installed on the existing nodes. Make sure that you have at least 250MB of free space on the same location on each of the new nodes to install Oracle Clusterware. In addition, ensure you have enough free space on each new node to install the Oracle binaries.

  3. Ensure that the Oracle Cluster Registry (OCR) and the voting disk are accessible by the new nodes using the same path as the other nodes use. In addition, the OCR and voting disk devices must have the same permissions as on the existing nodes.

  4. Ensure that user equivalence is established on the new nodes by performing the following:

    • Make sure that you can run the following command from all of the existing nodes of your cluster where the hostname is the public network name of the new node:

      NET USE \\hostname\C$
      

      You have the required administrative privileges on each node if the operating system responds with:

      Command completed successfully.
      

After completing the procedures in this section, your new nodes are connected to the cluster and configured with the required software to make them visible to the clusterware. Configure the new nodes as members of the cluster by extending the cluster software to the new nodes as described in "Step 2: Extending Oracle Software to New Nodes at the Oracle Clusterware Layer".

Note:

Do not change a hostname after the Oracle Clusterware installation. This includes adding or deleting a domain qualification.

Step 2: Extending Oracle Software to New Nodes at the Oracle Clusterware Layer

The following section describes how to add new nodes to Oracle Clusterware and to the Oracle database software layers using Oracle Universal Installer. Oracle Universal Installer requires access to the private interconnect that you checked as part of the installation validation in Step 1.

  1. On one of the existing nodes go to the CRS_home\oui\bin directory and run the addnode.bat script to start Oracle Universal Installer.

  2. Oracle Universal Installer runs in the add node mode and the Oracle Universal Installer Welcome page appears. Click Next and the Specify Cluster Nodes for Node Addition page appears.

  3. The upper table on the Specify Cluster Nodes for Node Addition page shows the existing nodes, the private node names and the virtual IP (VIP) addresses that are associated with Oracle Clusterware. Use the lower table to enter the public, private node names and the virtual hostnames of the new nodes.

  4. Click Next and Oracle Universal Installer verifies connectivity on the existing nodes and on the new nodes. The verifications that Oracle Universal Installer performs include determining whether:

    • The nodes are up

    • The nodes are accessible by way of the network

      Note:

      If any of the existing nodes are down, then you can proceed with the procedure. However, once the nodes are up, you must run the following command on each of those nodes:
      setup.exe -updateNodeList -local "CLUSTER_NODES={available_node_list}" ORACLE_HOME=CRS_home
      

      This operation should be run from the CRS_home\oui\bin directory and the available_node_list value is a comma-delimited list of all of nodes currently in the cluster and CRS_home defines the Oracle Clusterware home directory.

    • The virtual hostnames are not already in use on the network

    • The user has write permission to create the Oracle Clusterware home on the new nodes

    • The user has write permission to the Oracle Universal Installer inventory in the C:\Program Files\Oracle\Inventory directory

  5. If Oracle Universal Installer detects that the new nodes do not have an inventory location, then:

    • Oracle Universal Installer automatically updates the inventory location in the Registry key

    If any verifications fail, then Oracle Universal Installer re-displays the Specify Cluster Nodes for Node Addition page with a Status column in both tables indicating errors. Correct the errors or deselect the nodes that have errors and proceed. However, you cannot deselect existing nodes; you must correct problems on nodes that are already part of your cluster before you can proceed with node addition. If all of the checks succeed, then Oracle Universal Installer displays the Node Addition Summary page.

  6. The Node Addition Summary page displays the following information showing the products that are installed in the Oracle Clusterware home that you are extending to the new nodes:

    • The source for the add node process, which in this case is the Oracle Clusterware home

    • The private node names that you entered for the new nodes

    • The new nodes that you entered

    • The required and available space on the new nodes

    • The installed products listing the products that are already installed on the existing the Oracle Clusterware home

    Click Next and Oracle Universal Installer displays the Cluster Node Addition Progress page.

  7. The Cluster Node Addition Progress page shows the status of the cluster node addition process. The table on this page has two columns showing the phase of the node addition process and the phase's status according to the following:

    This page shows the following three Oracle Universal Installer phases:

    • Copy the Oracle Clusterware Home to New Nodes—Copies the Oracle Clusterware home to the new nodes unless the Oracle Clusterware home is on the Oracle Cluster File System.

    • Perform Oracle Home Setup—Updates the Registry entries for the new nodes, creates the services, and creates folder entries.

    • Save Cluster Inventory—Updates the node list associated with the Oracle Clusterware home and its inventory.

    The Cluster Node Addition Progress page's Status column displays In Progress while the phase is in progress, Suspended when the phase is pending execution, and Succeeded after the phase completes. On completion, click Exit to end the Oracle Universal Installer session. After Oracle Universal Installer displays the End of Node Addition page, click Exit to end the Oracle Universal Installer session.

  8. From CRS_home\install on node1, run the crssetup.add.bat script.

  9. Use the following command to perform an integrated validation of the Oracle Clusterware setup on all of the configured nodes, both the pre-existing nodes and the nodes that you have added

    cluvfy stage -post crsinst -n all [-verbose]
    

The CVU -post crsinst stage check verifies the integrity of the Oracle Clusterware components. After you have completed the procedures in this section for adding nodes at the Oracle Clusterware layer, you have successfully extended the Oracle Clusterware home from your existing the Oracle Clusterware home to the new nodes. Proceed to Step 3 to prepare the storage for Oracle RAC on the new nodes.

You can optionally run addnode.bat in silent mode, replacing steps 1 through 6 as follows, where nodeI, nodeI+1, and so on are the new nodes that you are adding:

addnode.bat -silent "CLUSTER_NEW_NODES={nodeI,nodeI+1,…nodeI+n}"
"CLUSTER_NEW_PRIVATE_NODE_NAMES={node-privI,node-privI+1,…node-privI+n}" 
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={node-vipI,node-vipI+1,…,node-vipI+n}"

You can alternatively specify the variable=value entries in a response file and run addnode as follows:

addnode.bat -silent -responseFile filename

Command-line values always override response file values.

See Also:

Oracle Universal Installer and OPatch User's Guide for details about how to configure command-line response files

Run crssetup.add.bat on the local node, or on the node on which you were performing this procedure.

After you have completed the procedures in this section for adding nodes at the Oracle Clusterware layer, you have successfully extended the Oracle Clusterware home from your existing the Oracle Clusterware home to the new nodes. Proceed to "Step 3: Preparing Storage on New Nodes" to prepare storage for Oracle RAC on the new nodes.

Step 3: Preparing Storage on New Nodes

To extend an existing Oracle RAC database to your new nodes, configure the shared storage for the new instances to be added on new nodes so that the storage type is the same as the storage that is already used by the existing nodes' instances. Prepare the same type of storage on the new nodes as you are using on the other nodes in the Oracle RAC environment that you want to extend as follows:

  • Automatic Storage Management (ASM)

    You do not need to install ASM on the added node because the ASM instance is created implicitly upon node addition. If you are using ASM, then make sure that the new nodes can access the ASM disks with the same permissions as the existing nodes.

  • Oracle Cluster File System (OCFS)

    If you are using Oracle Cluster File Systems, then make sure that the new nodes can access the cluster file systems in the same way that the other nodes access them.

  • Raw Device Storage

    If your cluster database uses raw devices, then prepare the new raw devices by following the procedures described in"Raw Device Storage Preparation for New Nodes".

See Also:

Oracle Real Application Clusters Installation Guide for Microsoft Windows for more information about the Oracle Cluster File System

If you are using Oracle Cluster File System, then run the following command to verify your cluster file system and obtain detailed output where node_list includes both the pre-existing nodes and the newly added nodes:

cluvfy comp ssa -n node_list -s storageID_list [-verbose]

See Also:

Appendix B, "Troubleshooting Oracle Real Application Clusters" for more information about enabling and using the Cluster Verification Utility (CVU)

Raw Device Storage Preparation for New Nodes

To prepare raw device storage on the new nodes, you need at least two new disk partitions to accommodate the redo logs for each new instance. Make these disk partitions the same size as the redo log partitions that you configured for the existing nodes' instances. Also create an additional logical partition for the undo tablespace for automatic undo management. You can create a raw device mapping file and set the DBCA_RAW_CONFIG environment variable so that it points to the raw device mapping file.

See Also:

Oracle Real Application Clusters Installation Guide for Microsoft Windows for more information about configuring raw partitions and using raw device mapping files
Configure Raw Partitions

Perform the following steps from one of the existing nodes of the cluster:

  1. Create or identify an extended partition.

  2. Right click inside an unallocated part of the extended partition.

  3. Choose Create from the Partition menu. A dialog box appears in which you should enter the size of the partition. Ensure you use the same sizes as those you used on your existing nodes.

  4. Click the newly created partition and select Assign Drive Letter from the Tool menu.

  5. Select Don't Assign Drive Letter, and click OK.

  6. Repeat steps 2 through 5 for the second and any additional partitions.

  7. Select Commit Changes Now from the Partition menu to save the new partition information.

  8. Enable the new and the existing nodes to recognize all of the partitions on the cluster. To do this, create logical names so that the existing nodes and the new nodes recognize the partitions by following these steps:

    1. Start the Object Link Manager (OLM) by entering the following command from the CRS_home\bin directory on one of the existing nodes:

      GUIOracleOBJManager
      
    2. The OLM starts and automatically detects the logical names of the partitions and displays them in OLM's graphical user interface.

    3. Recall the disk and partition numbers for the partitions that you created in the previous section. Look for the disk and partition numbers in the OLM page and perform the following tasks:

      • Right-click next to the box under the New Link column and enter the logical name for the first partition.

      • Repeat the previous step for the second and any additional partitions.

        For example, if your Oracle RAC database name is db and your database consists of two instances running on two nodes and you are adding a third instance on the third node, then your logical names for your redo logs should be db_redo3_1, db_redo3_2, and so on.

      • To enable automatic undo management for a new node's instance, enter the logical name for the partition for the undo tablespace that you created in the previous section. For example, if your Oracle RAC database name is db and if it has two instances running on two nodes and you are adding a third instance on a third node, then your logical name for the undo tablespace should be db_undotbs3.

      • Select Commit from the Options menu. This creates the new logical names on the current node.

      • Select Sync Nodes from the Options menu. This makes the new logical names visible to all of the nodes in the cluster, including the new nodes.

      • Select Exit from the Options menu to exit the Object Link Manager.

After completing the procedures in this section, you have configured your cluster storage so that the new nodes can access the Oracle software. Additionally, the existing nodes can access the new nodes and instances.

Run the following command to verify that the prepared raw device storage is accessible from all of the configured cluster nodes where nodelist includes both the pre-existing nodes and the newly added nodes:

cluvfy comp ssa -n node_list -s storageID_list -verbose

See Also:

Appendix B, "Troubleshooting Oracle Real Application Clusters" for more information about enabling and using the Cluster Verification Utility (CVU)

Proceed to the next section, "Step 4: Adding Nodes at the Oracle Real Application Clusters Database Layer".

Step 4: Adding Nodes at the Oracle Real Application Clusters Database Layer

To add nodes at the Oracle RAC database later, run Oracle Universal Installer in add node mode to configure your new nodes. If you have multiple Oracle homes, then perform the following steps for each Oracle home that you want to include on the new nodes:

  1. On an existing node from the Oracle_home\oui\bin run the addnode.bat script. This starts Oracle Universal Installer in the add node mode and displays the Oracle Universal Installer Welcome page. Click Next on the Welcome page and Oracle Universal Installer displays the Specify Cluster Nodes for Node Addition page.

  2. The Specify Cluster Nodes for Node Addition page has a table showing the existing nodes associated with the Oracle home from which you launched Oracle Universal Installer. A node selection table appears on the bottom of this page showing the nodes that are available for addition. Select the nodes that you want to add and click Next.

  3. Oracle Universal Installer verifies connectivity and performs availability checks on both the existing nodes and on the nodes that you want to add. Some of checks performed determine whether:

    • The nodes are up

    • The nodes are accessible by way of the network

    • The user has write permission to create the Oracle home on the new nodes

    • The user has write permission to the Oracle Universal Installer inventory in the C:\Program Files\Oracle\Inventory directory on the existing nodes and on the new nodes

  4. If the new nodes do not have an inventory set up, then Oracle Universal Installer automatically updates the Registry entries for the inventory location. If any of the other checks fail, then fix the problem and proceed or deselect the node that has the error and proceed. You cannot deselect existing nodes; you must correct any problems on the existing nodes before proceeding with node addition. If all of the checks succeed, then Oracle Universal Installer displays the Node Addition Summary page.

  5. The Node Addition Summary page has the following information about the products that are installed in the Oracle home that you are going to extend to the new nodes:

    • The source for the add node process, which in this case is the Oracle home

    • The existing nodes and new nodes

    • The new nodes that you selected

    • The required and available space on the new nodes

    • The installed products listing all of the products that are already installed in the existing Oracle home

    Click Finish and Oracle Universal Installer displays the Cluster Node Addition Progress page.

  6. The Cluster Node Addition Progress page shows the status of the cluster node addition process. The table on this page has two columns showing the phase of the node addition process and the phase's status. The Cluster Node Addition Progress shows the following three Oracle Universal Installer phases:

    • Copy the Oracle Home To New Nodes—Copies the entire Oracle home to the new nodes unless the Oracle home is on a cluster file system

    • Performs Oracle Home Setup—Updates the Registry entries for the new nodes, creates the services, and creates folder entries

    • Save Cluster Inventory—Updates the node list associated with the Oracle home and its inventory

    The Cluster Node Addition Progress page's Status column displays Succeeded if the phase completes, In Progress if the phase is in progress, and Suspended when the phase is pending execution. After Oracle Universal Installer displays the End of Node Addition page, click Exit to end the Oracle Universal Installer session.

  7. Add a listener to the new node by running the Net Configuration Assistant (NetCA) from the new node and selecting only the new node on the Node Selection page.

After completing the procedures in this section, you have defined the new nodes at the cluster database layer. You can now add database instances to the new nodes as described in Step 5.

You can optionally run addnode.bat in silent mode, replacing steps 1 through 6, as follows where nodeI, nodeI+1, and so on are the new nodes that you are adding:

addnode.bat -silent "CLUSTER_NEW_NODES={nodeI,nodeI+1,…nodeI+n}" 

The variable=value entries can also be specified in a response file, filename, and addnode can be run as:

addnode.bat -silent -responseFile filename

See Also:

Oracle Universal Installer and OPatch User's Guide for details about how to configure command-line response files

Command-line values always override response file values. Proceed to the next section, "Step 5: Adding Database Instances to New Nodes".

See Also:

Oracle Database Net Services Administrator's Guide for more information about NETCA

Step 5: Adding Database Instances to New Nodes

You can use Oracle Enterprise Manager, or the DBCA in either interactive mode or in silent mode, to add database instances to new nodes. To add a database instance to a target node with Oracle Enterprise Manager, see the Oracle Database 2 Day + Real Application Clusters Guide for complete information.

The following sections describe adding instances using DBCA:

Using DBCA in Interactive Mode to Add Database Instances to New Nodes

To add a database instance to a new node with DBCA in interactive mode, perform the following procedure:

  1. Start the DBCA by entering dbca at the system prompt from the bin directory in the Oracle_home directory.

    The DBCA displays the Welcome page for Oracle RAC. Click Help on any DBCA page for additional information.

  2. Select Oracle Real Application Clusters database, click Next, and DBCA displays the Operations page.

  3. Select Instance Management, click Next, and DBCA displays the Instance Management page.

  4. Select Add Instance and click Next. The DBCA displays the List of Cluster Databases page that shows the databases and their current status, such as ACTIVE, or INACTIVE.

  5. From the List of Cluster Databases page, select the active Oracle RAC database to which you want to add an instance. Enter user name and password for the database user that has SYSDBA privileges. Click Next and DBCA displays the List of Cluster Database Instances page showing the names of the existing instances for the Oracle RAC database that you selected.

  6. Click Next to add a new instance and DBCA displays the Adding an Instance page.

  7. On the Adding an Instance page, enter the instance name in the field at the top of this page if the instance name that DBCA provides does not match your existing instance naming scheme. Then select the new node name from the list, click Next, and DBCA displays the Services Page.

  8. Enter the services information for the new node's instance, click Next, and DBCA displays the Instance Storage page.

  9. If you are using raw devices or raw partitions, then on the Instance Storage page select the Tablespaces folder and expand it. Select the undo tablespace storage object and a dialog appears on the right-hand side. Change the default datafile name to the raw device name for the tablespace.

  10. If you are using raw devices or raw partitions or if you want to change the default redo log group file name, then on the Instance Storage page select and expand the Redo Log Groups folder. For each redo log group number that you select, DBCA displays another dialog box. Enter the raw device name that you created in the section "Raw Device Storage Preparation for New Nodes" in the File Name field.

  11. If you are using a cluster file system, then click Finish on the Instance Storage page. If you are using raw devices, then repeat step 10 for all of the other redo log groups, click Finish, and DBCA displays a Summary dialog.

  12. Review the information on the Summary dialog and click OK or click Cancel to end the instance addition operation. The DBCA displays a progress dialog showing DBCA performing the instance addition operation. When DBCA completes the instance addition operation, DBCA displays a dialog asking whether you want to perform another operation.

  13. After you terminate your DBCA session, run the following command to verify the administrative privileges on the new node and obtain detailed information about these privileges where nodelist consists of the newly added nodes:

    cluvfy comp admprv -o db_config -d oracle_home -n nodelist [-verbose]
    

After adding the instances to the new nodes using the steps described in this section, perform any needed service configuration procedures as described in Chapter 4, "Introduction to Automatic Workload Management".

Using DBCA in Silent Mode to Add Database Instances to New Nodes

You can use the DBCA in silent mode to add instances to nodes onto which you have extended an Oracle Clusterware home and an Oracle home. Use the following syntax to perform this operation where node is the node onto which you want to add the instance, gdbname is the global database name, instname is the name of the new instance, sysdba is the name of an Oracle user with SYSDBA privileges, and password is the password for the user name in sysdba:

dbca -silent -addInstance -nodeList node -gdbName gdbname [-instanceName instname] -sysDBAUserName sysdba -sysDBAPassword password

Note that you need to provide an instance name only if you want to override the Oracle naming convention for Oracle RAC instance names.

After you have completed either of DBCA procedures in this section, DBCA has successfully added the new instance to the new node and completed the following steps:

  • Created and started an ASM instance on each new node if the existing instances were using ASM

  • Created a new database instance on each new node

  • Created and configured high availability components

  • Created the Oracle Net configuration

  • Started the new instance

  • Created and started services if you entered services information on the Services Configuration page

After adding the instances to the new nodes using the steps described in this section, perform any needed service configuration procedures as described in Chapter 4, "Introduction to Automatic Workload Management".

Adding Nodes That Already Have Oracle Clusterware and Oracle Database Software

To add nodes to a cluster when the new nodes already have Oracle Clusterware and Oracle Database software installed on them, you must configure the new nodes with the Oracle software that is on the existing nodes of the cluster.

To add nodes, you must run two versions of an Oracle Universal Installer process: one for the clusterware and one for the database layer as described in the following procedures:

  1. Add new nodes at the Oracle Clusterware layer by running Oracle Universal Installer using the following command:

    CRS_home\oui\bin\addnode.bat -noCopy 
    
  2. Add new nodes at the Oracle software layer by running Oracle Universal Installer from the Oracle home as follows:

    Oracle_home\oui\bin\addnode.bat -noCopy 
    

In the -noCopy mode, Oracle Universal Installer performs all add node operations except for the copying of software to the new nodes.

Note:

Oracle recommends that you back up your voting disk and Oracle Cluster Registry files after you complete the node addition process.

Overview of Node Deletion Procedures

This section explains how to delete nodes from clusters using detailed manual procedures. The details of these steps appear in the following sections:

Step 1: Deleting Instances from Oracle Real Application Clusters Databases

The procedures in this section explain how to use DBCA in interactive or silent mode, to delete an instance from an Oracle RAC database.

To delete a database instance from a target node with Oracle Enterprise Manager, see the Oracle Database 2 Day + Real Application Clusters Guide for complete information.

Using DBCA in Interactive Mode to Delete Database Instances from Existing Nodes

To delete an instance using DBCA in interactive mode, perform the following steps:

  1. Start DBCA on a node other than the node that hosts the instance that you want to delete. On the DBCA Welcome page select Oracle Real Application Clusters Database, click Next, and DBCA displays the Operations page.

  2. On the DBCA Operations page, select Instance Management, click Next, and DBCA displays the Instance Management page.

  3. On the Instance Management page, select Delete Instance, click Next, and DBCA displays the List of Cluster Databases page.

  4. Select an Oracle RAC database from which to delete an instance. Enter a user name and password for the database user that has SYSDBA privileges. Click Next and DBCA displays the List of Cluster Database Instances page. The List of Cluster Database Instances page shows the instances that are associated with the Oracle RAC database that you selected and the status of each instance.

  5. Select an instance to delete and click Finish.

  6. Review the information about the instance deletion operation on the Summary page and click OK. Otherwise, click Cancel to cancel the instance deletion operation. If you click OK, then DBCA displays a Confirmation dialog.

  7. Click OK on the Confirmation dialog to proceed with the instance deletion operation and DBCA displays a progress dialog showing that DBCA is performing the instance deletion operation. During this operation, DBCA removes the instance and the instance's Oracle Net configuration. When DBCA completes this operation, DBCA displays a dialog asking whether you want to perform another operation.

  8. Click No and exit DBCA or click Yes to perform another operation. If you click Yes, then DBCA displays the Operations page.

Using DBCA in Silent Mode to Delete Instance from Existing Nodes

Use DBCA to delete a database instance from a node as follows, where the variables are the same as those in the preceding add instance command:

dbca -silent -deleteInstance [-nodeList node] -gdbName gdbname -instanceName instname -sysDBAUserName sysdba -sysDBAPassword password

You only need to provide a node name if you are deleting an instance from a node other than the one on which you are running DBCA.

At this point, you have accomplished the following:

  • De-registered the selected instance from its associated Oracle Net Services listeners

  • Deleted the selected database instance from the instance's configured node

  • Removed the Oracle Net configuration

  • Deleted the Oracle Flexible Architecture directory structure from the instance's configured node.

Step 2: Deleting Nodes from Oracle Real Application Clusters Databases

Execute the following procedures to delete nodes from Oracle clusters on Windows systems.

Perform the following steps on a node other than the node that you want to delete:

  1. Use the DBCA to delete the instance.

  2. Use the Net Configuration Assistant (NETCA) to delete the listener.

Perform the following steps on the node that you want to delete:

See Also:

Oracle Database Net Services Administrator's Guide for more information about NETCA

Note:

You can perform some of the steps in the following procedure in silent mode as described at the end of this section.
  1. If the node that you are deleting has an ASM instance, then delete the ASM instance using the procedures in the following section, "Step 3: ASM Instance Clean-Up Procedures for Node Deletion".

  2. On the node that you are deleting, for example, node2, run the following command where location is the full path location of the Oracle home:

    setup.exe -updateNodeList ORACLE_HOME=location
    "CLUSTER_ NODES={node2}" -local
    
  3. Depending on whether you have a shared or non-shared Oracle home, complete one of the following two procedures:

    • For a non-shared home, run Oracle Universal Installer from the Oracle home and de-install this home. You can run the setup.exe command from Oracle_home\oui\bin to start Oracle Universal Installer so that you can de-install the Oracle home.

    • For a shared home:

      • Run the following command from the Oracle_home\oui\bin directory:

        setup.exe –detachHome ORACLE_HOME=Oracle_home
        
      • Delete the services associated with this Oracle home.

      • Manually update the PATH environment variable by removing the entries associated with the Oracle home that you are removing.

      • Manually delete any Start menu folder items associated with the deleted Oracle home.

  4. On node1, or in the case of multiple node installations, on any node other than the node that you are deleting, run the following command from Oracle_home\oui\bin where node_list is a comma-delimited list of nodes that are to remain part of the Oracle RAC installation:

    setup.exe –updateNodeList ORACLE_HOME=Oracle_home
    "CLUSTER_NODES={node_list}" -local
    
  5. From CRS_home\bin on any existing node, run the following command where remote_port is the number of the remote port:

    racgons remove_config node2:remote_port
    

    You can determine the remote port by viewing the contents of the file, CRS_ home\opmn\conf\ons.config.

  6. Run SRVCTL to stop and remove the node applications from node2. From CRS_ home\bin run the following commands:

    srvctl stop nodeapps –n node2
    srvctl remove nodeapps –n node2
    
  7. On node1, or on any node that you are not deleting, run the following command from CRS_home\bin where node_name is the node to be deleted and node_number is the node's number as obtained from the output of the olsnodes -n command:

    crssetup del –nn node_name,node_number
    
  8. On the node or nodes that you are deleting, node2 in this example, run the following command from CRS_home\oui\bin:

    setup.exe –updateNodeList ORACLE_HOME=CRS_home 
    "CLUSTER_NODES={node2}" CRS=TRUE –local
    
  9. On node2, using Oracle Universal Installer setup.exe from CRS_home\oui\bin:

    • If you have a non-shared home, then de-install the Oracle Clusterware installation by running the setup.exe script from CRS_home\oui\bin.

    • If you have a shared home, then do not perform a de-installation. Instead, perform the following steps on the node that you want to delete:

      • Run the following command from CRS_home\oui\bin to detach the Oracle Clusterware home on node2:

        setup.exe –detachHome ORACLE_HOME=CRS_home
        
      • On node2, stop any services that are associated with this Oracle Clusterware home. The associated service s are namely: OracleCRService, OracleEVMService, OracleCSService, and OracleClusterVolumeService.

      • Remove the Oracle services.

      • Manually delete all of the Start menu items associated with this Oracle Clusterware home.

      • Manually update the PATH environment variable to remove entries associated with this Oracle Clusterware home.

      • Then remove the following files under %SystemRoot%\System32\drivers: ocfs.sys, orafencedrv.sys, and orafenceservice.sys.

  10. On node1, or in the case of a multiple node installation, on any node other than the one to be deleted, run the following command from CRS_home\oui\bin where node_list is a comma-delimited list of nodes that are to remain part of the Oracle RAC database:

    setup.exe –updateNodeList ORACLE_HOME=CRS_home
    "CLUSTER_NODES={node_list}" CRS=TRUE
    

As mentioned at the beginning of this procedure, you can optionally delete nodes from Oracle Real Application Clusters databases in silent mode by completing the following steps:

  1. Perform steps 1 and 2 from the previous procedure.

  2. On the node to be deleted, node2 in this case:

    • If you have a non-shared home, then de-install the Oracle home as follows:

      setup.exe –silent  -deinstall "REMOVE_HOMES={Oracle_home}"
      
    • If the home is a shared home, then do not perform a de-install. Instead, perform the following steps on the node to be deleted:

      • Perform a detach home on node2 by running the following command from Oracle_home\oui\bin on node2:

        setup.exe –detachHome –silent ORACLE_HOME=Oracle_home
        
      • On node 2, stop and delete any services that are associated with this Oracle home. In addition, delete any Registry entries and path entries that are associated with this Oracle home. Also, delete all of the start menu items associated with this Oracle home.

  3. Perform steps 4 through 8 from the previous procedure.

  4. On node2, using Oracle Universal Installer setup.exe from CRS_home\oui\bin:

    • If you have a non-shared home, then de-install the Oracle Clusterware home as follows:

      setup.exe –silent  -deinstall "REMOVE_HOMES={CRS_home}"
      
    • If you have a shared home, then do not perform a de-installation. Instead, perform the following steps on the node that you want to delete:

      • Run the following command from CRS_home\oui\bin to detach the Oracle Clusterware home on node2:

        setup.exe –detachHome –silent ORACLE_HOME=CRS_home
        
      • On node2, stop any services that are associated with this Oracle Clusterware home. The associated service s are namely: OracleCRService, OracleEVMService, OracleCSService, and OracleClusterVolumeService.

      • Remove the Oracle services.

      • Manually delete all of the Start menu items associated with this Oracle Clusterware home.

      • Manually update the PATH environment variable to remove entries associated with this Oracle Clusterware home.

      • Then remove the following files under %SystemRoot%\System32\drivers: ocfs.sys, orafencedrv.sys, and orafenceservice.sys.

  5. Perform step 10 from the previous procedure.

Note:

Oracle recommends that you back up your voting disk and Oracle Cluster Registry files after you complete the node deletion process.

Step 3: ASM Instance Clean-Up Procedures for Node Deletion

If you are using ASM, then the delete node procedure requires the following additional steps to remove the ASM instances:

  1. If this is the Oracle home from which the node-specific listener named LISTENER_nodename runs, then use NETCA to remove this listener. If necessary, re-create this listener in another home.

    See Also:

    Oracle Database Net Services Administrator's Guide for more information about NETCA
  2. If this is the Oracle home from which the ASM instance runs, then remove the ASM configuration by running the following command for all nodes on which this Oracle home exists:

    srvctl stop asm -n node
    

    Then run the following command for the nodes that you are removing:

    srvctl remove asm -n node
    
  3. Run the following command on each node that you are going to delete that has an ASM instance:

    oradim -delete -asmsid +ASMnode_number
    
  4. If you are not using a cluster file system for your ASM Oracle home, then run the following commands on the deleted node:

  5. rd /s /q ORACLE_BASE\admin\+ASM
    del Oracle_home\database\*ASM*