Skip Headers
Oracle® Clusterware Administration and Deployment Guide
11g Release 1 (11.1)

B28255-07
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Master Index
Master Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

2 Administering Oracle Clusterware

This chapter describes how to administer the voting disks and the Oracle Cluster Registry (OCR) under the following topics:

Administering Voting Disks

Oracle Clusterware includes two important components: the voting disk and OCR.

Oracle recommends that you select the option to configure multiple voting disks during Oracle Clusterware installation to improve availability. If necessary, you can dynamically add voting disks after you complete the Oracle Clusterware installation process.

After installation, use the following procedures to regularly backup your voting disks and to recover them as needed:

Backing Up Voting Disks

Oracle recommends that you back up your voting disk after the initial cluster creation and after you complete any node addition or deletion procedures.

Determine the current voting disk by issuing the following command:

crsctl query css votedisk

Issue the dd or ocopy command to back up a voting disk, as appropriate. In the following examples, voting_disk_name is the name of the active voting disk and backup_file_name is the name of the file to which you want to back up the voting disk contents:

  • On Linux or UNIX systems:

    dd if=voting_disk_name of=backup_file_name
    

    Oracle recommends you use the dd command to backup the voting disk with a minimum block size of 4KB.

  • On Windows systems, use the ocopy command:

    ocopy voting_disk_name backup_file_name
    

    To display online documentation for OCOPY, enter OCOPY by itself at the Windows prompt.

Recovering Voting Disks

To restore the backup of your voting disk, issue the dd or ocopy command for Linux and UNIX systems or for Windows systems respectively. In the following examples, backup_file_name is the name of the voting disk backup file and voting_disk_name is the name of the active voting disk:

  • On Linux or UNIX systems:

    dd if=backup_file_name of=voting_disk_name
    

    Oracle recommends that you use the dd command to backup the voting disk with a minimum block size of 4KB.

  • On Windows systems, use the ocopy command:

    ocopy backup_file_name voting_disk_name
    

If you have multiple voting disks, then you can remove the voting disks and add them back into your environment using the following commands, where path is the complete path of the location where the voting disk resides:

crsctl delete css votedisk path
crsctl add css votedisk path

Adding, Moving, or Deleting Voting Disks

You can add, move, and remove voting disks after Oracle Clusterware has been installed using the following commands:

  • To retrieve the current voting disk, issue the following command:

    crsctl query css votedisk
    

    This command lists the voting disks used by Cluster Synchronization Services (CSS).

  • To add a voting disk, issue the following command as the root user, replacing the path variable with the fully qualified path name for the voting disk you want to add:

    crsctl add css votedisk path -force
    
  • To move a voting disk, issue the following commands as the root user, replacing the path variable with the fully qualified path name for the voting disk you want to move:

    crsctl delete css votedisk path -force
    crsctl add css votedisk path -force
    
  • To remove a voting disk, issue the following command as the root user, replacing the path variable with the fully qualified path name for the voting disk you want to remove:

    crsctl delete css votedisk path -force
    

    Caution:

    If your cluster is down, then you can include the -force option to modify the voting disk configuration, without interacting with active Oracle Clusterware daemons. However, using the -force option while any cluster node is active may corrupt your configuration.

After modifying the voting disk, restart Oracle Clusterware using the crsctl start crs command on all nodes, and verify the voting disk location using the following command:

crsctl query css votedisk

Administering the Oracle Cluster Registry

This section describes how to administer the OCR using the OCR tools: OCRCONFIG, OCRDUMP, and OCRCHECK.

The OCR contains information about the cluster node list, instance-to-node mapping information, and information about Oracle Clusterware resource profiles for applications that you have customized as described in Chapter 5, "Making Applications Highly Available Using Oracle Clusterware".

This section describes how to administer the OCR in the following topics:

See Also:

Appendix E, "Managing the Oracle Cluster Registry" for information about the OCRCONFIG tool, and Appendix F, "Troubleshooting Oracle Clusterware" for information about the OCRDUMP and OCRCHECK utilities

Adding, Replacing, Repairing, and Removing the Oracle Cluster Registry

The Oracle installation process for Oracle Clusterware gives you the option of automatically mirroring the OCR. You can put the mirrored OCR on an Oracle cluster file system disk, on a shared raw device, or on a shared raw logical volume. In addition, Oracle supports block devices for the OCR on Linux.

You can manually mirror the OCR, as described in the "Adding an Oracle Cluster Registry" section, if you:

  • Upgraded to release 11.1 but did not choose to mirror the OCR during the upgrade

  • Created only one OCR during the Oracle Clusterware installation

Note:

Oracle strongly recommends that you use mirrored OCRs if the underlying storage is not RAID. Mirroring can help to prevent the OCR from becoming a single point of failure.

In addition to mirroring the OCR, you can also:

Note:

The operations in this section affect the OCR clusterwide: they change the OCR configuration information in the ocr.loc file on Linux and UNIX systems and the Registry keys on Windows systems. However, the ocrconfig command cannot modify OCR configuration information for nodes that are shut down or for nodes on which Oracle Clusterware is not running.

Adding an Oracle Cluster Registry

You can add an OCR location after an upgrade or after completing the Oracle Clusterware installation. If you already mirror the OCR, then you do not need to add an OCR location; Oracle Database automatically manages two OCRs when it mirrors the OCR. Oracle Clusterware environments do not support more than two OCRs: a primary OCR and a secondary OCR.

Note:

If the OCR resides on a cluster file system file or if the OCR is on a network file system, then create a dummy file before performing the procedures in this section.

As the root user, issue the following command to add an OCR location using either destination_file or disk to designate the target location of the additional OCR:

ocrconfig -replace ocr destination_file or disk

Run the following command to add an OCR mirror location using either destination_file or disk to designate the target location of the additional OCR:

ocrconfig -replace ocrmirror destination_file or disk

Note:

On Linux and UNIX systems, you must be root user to run ocrconfig commands. On Windows systems, the user must be a member of the Administrator's group.

Replacing an Oracle Cluster Registry

If you need to change the location of an existing OCR, or change the location of a failed OCR to the location of a working one, you can use the following procedure as long as one OCR file remains online.

To change the location of an OCR:

  1. Use the OCRCHECK utility to verify that a copy of the OCR other than the one you are going to replace is online, using the following command:

    ocrcheck 
    

    OCRCHECK displays all OCR files that are registered and whether or not they are available (online). If an OCR file suddenly becomes unavailable, it might take a short period of time for Oracle Clusterware to show the change in status.

    Note:

    The OCR that you are replacing can be either online or offline.
  2. Use the following command to verify that Oracle Clusterware is running on the node on which the you are going to perform the replace operation:

    crsctl check crs
    
  3. Run the following command as root user to replace the primary OCR using either destination_file or disk to indicate the target OCR location:

    ocrconfig -replace ocr destination_file
    ocrconfig -replace ocr disk
    
  4. Run the following command as root user to replace a secondary OCR using either destination_file or disk to indicate the target OCR location:

    ocrconfig -replace ocrmirror destination_file
    ocrconfig -replace ocrmirror disk
    
  5. If any node that is part of your current Oracle RAC cluster is shut down, then run the following command on the stopped node to let that node rejoin the cluster after the node is restarted:

    ocrconfig -repair
    

Repairing an Oracle Cluster Registry Configuration on a Local Node

You may need to repair an OCR configuration on a particular node if your OCR configuration changes while that node is stopped. For example, you may need to repair the OCR on a node that was not up while you were adding, replacing, or removing an OCR. To repair an OCR configuration, run the following command on the node on which you have stopped the Oracle Clusterware daemon:

ocrconfig –repair ocrmirror device_name 

This operation only changes the OCR configuration on the node from which you run this command. For example, if the OCR mirror device name is /dev/raw1, then use the command syntax ocrconfig -repair ocrmirror /dev/raw1 on this node to repair its OCR configuration.

Note:

You cannot perform this operation on a node on which the Oracle Clusterware daemon is running.

Removing an Oracle Cluster Registry

To remove an OCR location, at least one other OCR must be online. You can remove an OCR location to reduce OCR-related overhead or to stop mirroring your OCR because you moved the OCR to redundant storage such as RAID.

Perform the following procedure as the root user to remove an OCR location from your Oracle Clusterware environment:

  1. Ensure that at least one OCR other than the OCR that you are removing is online.

    Caution:

    Do not perform this OCR removal procedure unless there is at least one other active OCR online.
  2. Run the following command on any node in the cluster to remove the OCR device:

    ocrconfig -replace ocr
    

    Run the following command on any node in the cluster to remove the mirrored OCR device:

    ocrconfig -replace ocrmirror
    

    These commands update the OCR configuration on all of the nodes on which Oracle Clusterware is running.

Note:

When removing an OCR location, the remaining OCR must be online. If you remove a primary OCR, then the mirrored OCR becomes the primary OCR.

Overriding the Oracle Cluster Registry Data Loss Protection Mechanism

The OCR has a mechanism that prevents data loss due to accidental overwrites. If you configure a mirrored OCR and if Oracle Clusterware cannot access the two mirrored OCR locations and also cannot verify that the available OCR location contains the most recent configuration, then Oracle Clusterware prevents further modification to the available OCR location. In addition, the process prevents overwriting by prohibiting Oracle Clusterware from starting on the node on which only one OCR is available. In such cases, Oracle Database displays an alert message in either Oracle Enterprise Manager, the Oracle Clusterware alert log files, or both. If this problem is local to only one node, you can use other nodes to start your cluster database.

However, if you are unable to start any cluster node in your environment and if you can neither repair the OCR nor restore access to all OCR locations, then you can override the protection mechanism. The procedure described in the following list enables you to start the cluster using the available OCR location. However, overriding the protection mechanism can result in the loss of data that was not available at the time that the previous known good state was created.

Note:

Overriding the OCR using the following procedure can result in the loss of OCR updates that were made between the time of the last known good OCR update made to the currently accessible OCR and the time at which you performed the overwrite. In other words, running the ocrconfig -overwrite command can result in data loss if the OCR location that you are using to perform the overwrite does not contain the latest configuration updates for your cluster environment.

Perform the following procedure to overwrite the OCR if a node cannot start up and if the alert log contains CLSD-1009 and CLSD-1011 messages.

  1. Attempt to resolve the cause of the CLSD-1009 and CLSD-1011 messages.

    Compare the node's OCR configuration (ocr.loc on Linux and UNIX systems and the Registry on Windows systems) with other nodes on which Oracle Clusterware is running.

    • If the configurations do not match, then run ocrconfig -repair.

    • If the configurations match, then ensure that the node can access all of the configured OCRs by running an ls command on Linux and UNIX systems. On windows, use a dir command if the OCR location is a file and run GuiOracleObjectManager.exe to verify that the partition with the name exists.

  2. Ensure that the most recent OCR contains the latest OCR updates.

    Look at output from the ocrdump command and determine whether it has your latest updates.

  3. If you cannot resolve the problem that caused the CLSD message, then run the command ocrconfig -overwrite to bring up the node.

Managing Backups and Recovering the Oracle Cluster Registry

This section describes how to backup OCR content and use it for recovery. The first method uses automatically generated OCR file copies and the second method enables you to issue a backup command on demand:

  • Automatic backups—Oracle Clusterware automatically creates OCR backups every four hours. At any one time, Oracle Database always retains the last three backup copies of the OCR. The CRSD process that creates the backups also creates and retains an OCR backup for each full day and at the end of each week. You cannot customize the backup frequencies or the number of files that Oracle Database retains.

  • Manual backups—Use the ocrconfig -manualbackup command to force Oracle Clusterware to perform a backup of the OCR at any time, rather than wait for the automatic backup that occurs at 4-hour intervals. The -manualbackup option is especially useful when you to need to obtain a binary backup on demand, such as before you make changes to the OCR.

You can use any backup software to copy the automatically generated backup files at least once daily to a different device from where the primary OCR resides.

The default location for generating backups on Linux or UNIX systems is CRS_home/cdata/cluster_name where cluster_name is the name of your cluster. The Windows default location for generating backups uses the same path structure. Because the default backup is on a local file system, Oracle recommends that you include the backup file created with the ocrconfig command as part of your operating system backup using standard operating system or third-party tools.

See Also:

You can use the ocrconfig -backuploc option to move the location where OCR creates backups. Appendix E, "Managing the Oracle Cluster Registry" describes the OCR configuration tool options.

You can issue the ocrconfig -showbackup command to display the backup location, timestamp, and the originating node name of the backup files that Oracle Database created. By default, the -showbackup option displays information for both automatic and manual backups but you can include the auto or manual flag to display only the automatic backup information or only the manual backup information, respectively.

Note:

On Linux and UNIX systems, you must be root user to run ocrconfig commands. On Windows systems, the user must be a member of the Administrator's group.

See Also:

"Administering the Oracle Cluster Registry with OCR Export and Import Commands" to use manually created OCR export files to copy OCR content and use it for recovery

Overview of Restoring the Oracle Cluster Registry from OCR Backups

If an application fails, then before attempting to restore the OCR, restart the application. As a definitive verification that the OCR failed, run an ocrcheck and if the command returns a failure message, then both the primary OCR and the OCR mirror have failed. Attempt to correct the problem using one of the following platform-specific OCR restoration procedures.

Note:

You cannot restore your configuration from an OCR backup file using the -import option, which is explained in "Administering the Oracle Cluster Registry with OCR Export and Import Commands". You must instead use the -restore option, as described in the following sections.

Restoring the Oracle Cluster Registry on Linux or UNIX Systems

Use the following procedure to restore the OCR on Linux or UNIX systems:

  1. Identify the OCR backups using the ocrconfig -showbackup command. Review the contents of the backup using ocrdump -backupfile file_name where file_name is the name of the backup file.

  2. Stop Oracle Clusterware on all of the nodes by executing the crsctl stop crs command on all of the nodes.

  3. Perform the restore by applying an OCR backup file that you identified in Step 1 using the following command where file_name is the name of the OCR that you want to restore. Make sure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid before running this command.

    ocrconfig -restore file_name
    
  4. Restart Oracle Clusterware on all of the nodes in your cluster by restarting each node or by running the crsctl start crs command.

  5. Run the following command to verify the OCR integrity where the -n node_list all argument retrieves a listing of all of the cluster nodes that are configured as part of your cluster:

    cluvfy comp ocr [-n node_list] [-verbose]
    

See Also:

Appendix F, "Troubleshooting Oracle Clusterware" for more information about enabling and using CVU

Restoring the Oracle Cluster Registry on Windows Systems

Use the following procedure to restore the OCR on Windows systems:

  1. Identify the OCR backups using the ocrconfig -showbackup command. Review the contents of the backup using ocrdump -backupfile file_name where file_name is the name of the backup file.

  2. On all of the remaining nodes, disable the following OCR clients and stop them using the Service Control Panel: OracleClusterVolumeService, OracleCSService, OracleCRService, and the OracleEVMService.

  3. Execute the restore by applying an OCR backup file that you identified in Step 1 with the ocrconfig -restore file name command. Make sure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

  4. Start all of the services that were stopped in step 2. Restart all of the nodes and resume operations in cluster mode.

  5. Run the following command to verify the OCR integrity where the -n node_list | all argument retrieves a listing of all of the cluster nodes that are configured as part of your cluster:

    cluvfy comp ocr [-n node_list] | all [-verbose]
    

    See Also:

    Appendix A for more information about enabling and using CVU

Diagnosing Oracle Cluster Registry Problems

You can use the OCRDUMP and OCRCHECK utilities to diagnose OCR problems as described under the following topics:

Using the OCRDUMP Utility

Use the OCRDUMP utility to write the OCR contents to a file so that you can examine the OCR content.

See Also:

"OCRDUMP Utility Syntax and Options" for more information about the OCRDUMP utility

Using the OCRCHECK Utility

Use the OCRCHECK utility to verify the OCR integrity.

See Also:

"Using the OCRCHECK Utility" for more information about the OCRCHECK utility

Administering the Oracle Cluster Registry with OCR Export and Import Commands

In addition to using the automatically created OCR backup files, you should also export the OCR contents before and after making significant configuration changes, such as adding or deleting nodes from your environment, modifying Oracle Clusterware resources, or creating a database. Do this by using the ocrconfig -export command. This exports the OCR content to a file format.

Using the ocrconfig -export command enables you to restore the OCR using the -import option if your configuration changes cause errors. For example, if you have unresolvable configuration problems, or if you are unable to restart your clusterware after such changes, then restore your configuration using one of the following platform-specific procedures:

Note:

Most configuration changes that you make not only change the OCR contents, configuration changes also cause file and database object creation. Some of these changes are often not restored when you restore the OCR. Do not perform an OCR restore as a correction to revert to previous configurations if some of these configuration changes should fail. This may result in an OCR that has contents that do not match the state of the rest of your system.

Importing Oracle Cluster Registry Content on Linux or UNIX Systems

Use the following procedure to import the OCR on Linux or UNIX systems:

  1. Stop Oracle Clusterware by issuing the following command (as root user) on all of the nodes:

    crsctl stop crs
    
  2. Import the OCR by applying an OCR export file by issuing the following command (as root user), where file_name is the name of the OCR file from which you want to import OCR information:

    ocrconfig -import file_name
    
  3. Restart Oracle Clusterware on all of the nodes in your cluster:

    crsctl start crs
    
  4. Verify the OCR integrity by issuing the following Cluster Verification Utility (CVU) command, where the -n node_list all argument retrieves a listing of all of the cluster nodes that are configured as part of your cluster:

    cluvfy comp ocr [-n node_list] [-verbose]
    

Note:

You cannot import an exported OCR backup file. (Managing backups is described in "Managing Backups and Recovering the Oracle Cluster Registry".) You must instead use the -import option as described in the following sections.

See Also:

Appendix F, "Troubleshooting Oracle Clusterware" for more information about enabling and using CVU

Importing Oracle Cluster Registry Content on Windows Systems

Use the following procedure to import the OCR on Windows systems:

  1. Identify the OCR export file that you want to import by running the ocrconfig -showbackup command.

  2. Stop the following OCR clients on each node in your Oracle Clusterware environment using the Service Control Panel: OracleClusterVolumeService, OracleCMService, OracleEVMService, OracleCSService, and OracleCRService.

  3. Import an OCR export file using the ocrconfig -import command from one node.

  4. Restart all of the affected services on all of the nodes.

  5. Run the following Cluster Verification Utility (CVU) command to verify the OCR integrity where node_list is a list of all of the nodes in your cluster database:

    cluvfy comp ocr [-n node_list] [-verbose] 
    

    See Also:

    Appendix F, "Troubleshooting Oracle Clusterware" for more information about enabling and using CVU

Implementing the Oracle HARD Initiative for the Oracle Cluster Registry

The Oracle Hardware Assisted Resilient Data (HARD) initiative prevents data corruptions from being written to permanent storage. If you enable HARD, then the OCR writes HARD-compatible blocks. To determine whether the device used by the OCR supports HARD and then enable it, review the Oracle HARD white paper at:

http://www.oracle.com/technology/deploy/availability/htdocs/HARD.html

Upgrading and Downgrading the Oracle Cluster Registry Configuration

When you install Oracle Clusterware, Oracle Database automatically runs the ocrconfig -upgrade command. To downgrade, follow the downgrade instructions for each component and also downgrade the OCR using the ocrconfig -downgrade command. If you are upgrading the OCR, then you can use the ocrcheck command to verify the integrity of the OCR.

Changing Network Addresses

In an Oracle Clusterware configuration, there must be a minimum of two network connections. One connection is for the public network interface on which users and application servers connect to access data on the database server, and the other connection is for the private network interface between the nodes in the cluster.

This section contains the following topics:

Changing Public Network Addresses

Clients use the Virtual Internet Protocol (VIP) address to connect to the Oracle RAC database instances. The VIP address resolves all timeouts entering the TCP/IP stack, resulting in higher application availability. The VIP address is a static IP address that defines and resolves to a hostname through the Domain Name System (DNS).

During the installation of Oracle Clusterware, you are prompted to enter a VIP address and a virtual hostname for each node in the cluster. These items are stored in the OCR and different components in the high availability framework depend on the VIP addresses.

Changing the VIP address requires modification of the node applications, including the VIP address, Global Services Daemon (GSD), the listener, and Oracle Notification Services (ONS). You can modify the VIP address while the node applications are running. However, the changes do not take effect until you restart the VIP address (hence, restarting the node applications). Other resources on a node, such as database instances and ASM instances, are dependent on the VIP address. Therefore, stopping the node applications takes the VIP address offline only if you stop the other resources.

Note:

The following instructions describe how to change only a VIP address and assume that the hostname associated with the VIP address will not change.
  • If you are changing only the VIP address, then it is not necessary to make changes to the listener.ora and tnsnames.ora files, provided they are using the virtual hostnames for the HOST= entries.

  • If you are changing only the VIP address, then update the Domain Name System (DNS) or the hosts of the clients. Also, update the hosts entries of the server, too, if those are used for VIPs.

  • If you are changing both the virtual hostname and the VIP address for a node, then you must modify the listener.ora file and change the HOST= entries to the new virtual hostname.

    You can use an editing tool to modify the listener.ora file manually or use the Net Configuration Assistant (NETCA) to remove the old listener and create a new listener. In addition, you do not need to make changes to the tnsnames.ora file of any clients connecting to the old hostname.

Perform the following steps to change a VIP address:

  1. Stop all database and ASM instances. Stop all resources that are dependent on the VIP address on a given node. This includes all Oracle RAC database instances on that node, and the ASM instance on that node, if it exists:

    1. Stop database instances:

      srvctl stop instance -d grid -i grid1
      

      This example specifies the database name (grid) using the -d option and specifies the instance (grid1) on the appropriate node using the -i option.

      Alternatively, to stop the entire database, on all nodes, you can issue the stop database command, as follows:

      srvctl stop database -d grid
      
    2. To stop the ASM instance on the node, issue the following command:

      srvctl stop asm -n mynode
      

      Note:

      Alternatively, you can stop these resources using SQL*Plus statements, or on Windows systems you can stop the associated services.
  2. Confirm the current IP address for the VIP by issuing the ifconfig -a command. This command displays the current VIP bound to one of the network interfaces. The following example displays the current VIP address that is bound to one of the network interfaces (eth0:1):

    eth0:1    Link encap:Ethernet  HWaddr 00:01:03:2C:69:BB  
              inet addr:192.168.1.125  Bcast:192.168.1.255  Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1   
              ...
              ...
    

    Note:

    On Windows systems, issue the ipconfig /all command, to list all IP addresses bound to a given adapter, including the VIP address.
  3. Stop node applications using the srvctl stop nodeapps command. For example:

    srvctl stop nodeapps -n mynode
    

    This command stops the VIP address, the GSD, the listener, and the ONS daemon currently running on the specified node.

  4. Verify that the VIP address is no longer running by issuing the ifconfig -a command on Linux and UNIX systems (or issue the ipconfig /all command on Windows systems), and confirm that the interface (in the example it was eth0:1) is no longer listed in the output.

    If the interface is still online, it indicates that a resource, which is dependent on the VIP address, is still running. Use the crs_stat command to show resources that are still online.

  5. Make any changes necessary to the /etc/hosts files on all nodes on Linux and Unix systems, or the %windir%\system32\drivers\etc\hosts file on Windows systems, and make any necessary DNS changes to associate the new IP address with the old host name.

  6. Modify the node applications and provide the new VIP address using the following srvctl modify nodeapps command syntax:

    srvctl modify nodeapps -n node_name; [-o oracle_home] [-A new_vip_address]
    

    In the syntax statement include the following options:

    • -n node_name is the node name

    • -o Oracle_home is the Oracle home for the cluster database.

    • -A new_vip_address is the node-level VIP address: name|ip/netmask[/if1[|if2|...]])

      For example, issue the following command as the root user:

      srvctl modify nodeapps -n mynode -A 192.168.2.125/255.255.255.0/eth0
      

      Attempting to issue this command as the oracle user results in the PRKO-2117: This command should be executed as the system privilege user error

  7. Start node-level applications by issuing the srvctl start nodeapps command:

    srvctl start nodeapps -n node_name
    

    The following command example starts applications on the node named mynode:

    srvctl start nodeapps -n mynode
    
  8. Repeat the steps for each node in the cluster.

    Because the SRVCTL utility is a cluster-wide management tool, you can accomplish these tasks for any specific node from any node in the cluster, without the need to log in to each of the cluster nodes.

  9. Run the following command to verify node connectivity between all of the nodes for which your cluster is configured. This command discovers all of the network interfaces available on the cluster nodes and verifies the connectivity between all of the nodes by way of the discovered interfaces. This command also lists all of the interfaces available on the nodes which are suitable for use as VIP addresses.

    cluvfy comp nodecon -n node_list all [-verbose]
    

Changing Private Network Addresses

Oracle Clusterware requires that each node is connected through a private network (in addition to the public network). The private network connection is referred to as the cluster interconnect. During the installation of Oracle Clusterware, you are prompted to enter a private network IP address for each node in the cluster. This address—unlike the VIP addresses—must be assigned already to a certain network interface on each node.

In addition, you must provide a private node name, which resolves to the IP address given, and is usually stored in the hosts file of the operating system. This private node name is stored automatically in the OCR. Oracle does not recommend using Domain Name Services (DNS) to resolve private network IP address names.

Oracle currently only supports clusters in which all nodes use the same network interface connected to the same subnet (defined as a global interface with the oifcfg command). You cannot use different network interfaces for each node (node-specific interfaces). See Appendix C, " Oracle Interface Configuration (OIFCFG) Command Reference" for more information about global and node-specific interfaces.

Table 2-1 describes how the Network Interface Card (NIC), the private IP address, and the private node name are stored.

Table 2-1 Storage for the NIC, Private IP Address, and Private Host Name

Entity Stored in ... Comments

Network Interface Card (NIC) name

Operating system

For example: eth1

Must be the same on all nodes. It can be changed globally.

Private network IP address

Operating system; assigned to the network interface. For example: eth1 10.10.1.1

Configure this as root using the ifconfig command. You can change the IP address locally if it is in the same subnet. Otherwise, change this globally.

Private node name

The /etc/hosts file, which resolves the private IP address by name

You cannot use the steps described in this section to change the private node name, which is also stored in the OCR, after Oracle Clusterware is installed.


The following topics describe how to change a private network address, the network interface, or both.

Note:

You cannot use the steps in this section to change the private node name after you have installed Oracle Clusterware.

To change a private network IP address:

If you want to change a private network IP address on one node and the new IP address is on the same subnet as the former IP address, perform the following steps:

  1. Stop Oracle Clusterware by issuing the following command as root user on the node on which you want to change the IP address:

    crsctl stop crs
    
  2. Assign a new network address to the Network Interface Card (NIC).

    As root user, assign a new network address to the NIC using the ifconfig operating system command. Ensure the new private IP address is in the same subnet as the private IP addresses of the remaining nodes in the cluster.

  3. Changes made by ifconfig are not persistent. In order to make ifconfig changes persistent, update the network configuration settings on the operating system, accordingly.

    Note:

    Oracle does not recommend that you use the Domain Name System (DNS) for the private node name resolution. However, if you use DNS, you must change the DNS entries accordingly.

    See Also:

    Your operating system documentation for more information about how to make ifconfig commands persistent
  4. Restart Oracle Clusterware by issuing the following command as root user

    crsctl start crs
    

    The changes take effect when Oracle Clusterware restarts. To change more than one private IP address on more than one node in the cluster, repeat steps 1 through 3 in a rolling manner for each of those nodes.

To change a private network IP address so that a new subnet is used:

If you want to change a private network IP address so that a new subnet is used, you must change all IP addresses on all cluster nodes to reflect the change of the IP subnet. Perform the following steps to change the subnet:

  1. On all nodes, stop Oracle Clusterware by issuing the following command as root user:

    crsctl stop crs
    
  2. Assign a new network address from the new subnet to the NIC.

    As root user, assign a new network address to the NIC using the ifconfig operating system command. Ensure all private IP addresses are taken from the same subnet.

  3. Changes made by ifconfig are not persistent. In order to make ifconfig changes persistent, update the network configuration settings on the operating system, accordingly.

    Note:

    Oracle does not recommend that you use the Domain Name System (DNS) for the private node name resolution. However, if you use DNS, you must change the DNS entries accordingly.

    See Also:

    Your operating system documentation for more information about how to make ifconfig commands persistent
  4. Restart Oracle Clusterware by issuing the following command as root user

    crsctl start crs
    

    The changes take effect when Oracle Clusterware restarts.

  5. Issue the oifcfg command to change and store the new private IP address subnet in the OCR. See Appendix C for more information about using the OIFCFG command. The IP address changes take effect when Oracle Clusterware restarts.

Note:

To minimize downtime, you can perform these steps, as follows:
  1. Perform step 1 (to stop Oracle Clusterware) on all nodes in the cluster except one.

  2. Perform steps 2 and 3 to change the IP address on each stopped node, but complete the steps on one node before performing the steps on the next node.

  3. Perform step 1 to stop Oracle Clusterware on the running (unchanged) node.

  4. Restart the nodes on which you have already changed the IP address.

  5. Perform steps 2 and 3 to change the IP address on the last unchanged node.

  6. Restart the last node.

If you use the CLUSTER_INTERCONNECTS initialization parameter, you must update it to reflect the changes. If you do not use the CLUSTER_INTERCONNECT parameter, then skip this step.

Note:

Oracle does not recommend setting the CLUSTER_INTERCONNECTS parameter.

The private network address is a static IP address. Changes made by ifconfig are not persistent. In order to make ifconfig changes persistent, update the network configuration settings on the operating system, accordingly.

Note:

Oracle does not recommend that you use the Domain Name System (DNS) for the private node name resolution. However, if you use DNS, you must change the DNS entries accordingly.

To change the Network Interface Card (NIC) for the interconnect:

If you want to change the network interface for the private interconnect (for example, eth1), you must perform the change on all nodes (globally). This is because Oracle currently does not support the use of different network interface cards in the same subnet for the cluster interconnect.

Perform the following steps:

  1. On all nodes, stop Oracle Clusterware by issuing the following command as root user:

    crsctl stop crs
    
  2. Assign the current network address to the new NIC using the ifconfig command.

    As root user, issue the ifconfig operating system command to assign the currently used private network address to the NIC intended to be used for the interconnect. This usually requires some downtime for the current interface and the new interface. See your platform-specific operating system documentation for more information about issuing the ifconfig command.

  3. Changes made by ifconfig are not persistent. In order to make ifconfig changes persistent, update the network configuration settings on the operating system, accordingly.

    Note:

    Oracle does not recommend that you use the Domain Name System (DNS) for the private node name resolution. However, if you use DNS, you must change the DNS entries accordingly.

    See Also:

    Your operating system documentation for more information about how to make ifconfig commands persistent
  4. Restart Oracle Clusterware by issuing the following command as root user on all nodes:

    crsctl start crs
    
  5. Issue the oifcfg command to change and store the new private network/private IP address subnet assignment in the OCR. See Appendix C for more information about using the OIFCFG command. The changes take effect when Oracle Clusterware restarts.

Note:

To minimize downtime, you can perform these steps as follows:
  1. Perform step 1 (to stop Oracle Clusterware) on all nodes in the cluster except one.

  2. Perform steps 2 and 3 to change the IP address on each stopped node, but complete the steps on one node before performing the steps on the next node.

  3. Perform step 1 to stop Oracle Clusterware on the running (unchanged) node.

  4. Restart the nodes on which you have already changed the IP address.

  5. Perform steps 2 and 3 to change the IP address on the last unchanged node.

  6. Restart the last node.

If you want to change both the private IP address and the network interface card in one step, assign the new IP address to the new NIC and then perform the remaining steps accordingly.