Skip Headers
Oracle® Clusterware Administration and Deployment Guide
11g Release 2 (11.2)

Part Number E16794-06
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Master Index
Master Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
View PDF

2 Administering Oracle Clusterware

This chapter describes how to administer Oracle Clusterware and includes the following topics:

Policy-Based Cluster and Capacity Management

This section contains the following topics:

Overview of Server Pools and Policy-Based Management

With Oracle Clusterware 11g release 2 (11.2) and later, resources managed by Oracle Clusterware are contained in logical groups of servers called server pools. Resources are hosted on a shared infrastructure and are contained within server pools. The resources are restricted with respect to their hardware resource (such as CPU and memory) consumption by policies, behaving as if they were deployed in a single-system environment.

You can choose to manage resources dynamically using server pools to provide policy-based management of resources in the cluster, or you can choose to manage resources using the traditional method of physically assigning resources to run on particular nodes.

Policy-based management:

  • Enables dynamic capacity assignment when needed to provide server capacity in accordance with the priorities you set with policies

  • Enables allocation of resources by importance, so that applications obtain the required minimum resources, whenever possible, and so that lower priority applications do not take resources from more important applications

  • Ensures isolation where necessary, so that you can provide dedicated servers in a cluster for applications and databases

Applications and databases running in server pools do not share resources. Because of this, server pools isolate resources where necessary, but enable dynamic capacity assignments as required. Together with role-separated management, this capability addresses the needs of organizations that have a standardized cluster environments, but allow multiple administrator groups to share the common cluster infrastructure.

See Also:

Appendix B, "Oracle Clusterware Resource Reference" for more information about resource attributes

Oracle Clusterware efficiently allocates different resources in the cluster. You need only to provide the minimum and maximum number of nodes on which a resource can run, combined with a level of importance for each resource that is running on these nodes.

Server Attributes Assigned by Oracle Clusterware

Oracle Clusterware assigns each server a set of attributes as soon as you add a server to a cluster. If you remove the server from the cluster, then Oracle Clusterware revokes those settings. Table 2-1 lists and describes server attributes.

Table 2-1 Server Attributes

Attribute Description
NAME

The node name of the server. A server name can contain any platform-supported characters except the exclamation point (!) and the tilde (~). A server name cannot begin with a period, or with ora. This attribute is required.

ACTIVE_POOLS

A space-delimited list of the names of the server pools to which a server belongs. Oracle Clusterware manages this list, automatically.

STATE

A server can be in one of the following states:

ONLINE

The server is a member of the cluster and is available for resource placement.

OFFLINE

The server is not currently a member of the cluster. Subsequently, it is not available for resource placement.

JOINING

When a server joins a cluster, Oracle Clusterware processes it to ensure that it is valid for resource placement. Oracle Clusterware also checks the state of resources configured to run on the server. Once the validity of the server and the state of the resources are determined, the server transitions out of the state.

LEAVING

When a planned shutdown for a server begins, the state of the server transitions to Leaving, making it unavailable for resource placement.

VISIBLE

Servers that have Oracle Clusterware running, but not the Cluster Ready Services daemon (crsd), are put into the Visible state. This usually indicates an intermittent issue or failure and Oracle Clusterware trying to recover (restart) the daemon. Oracle Clusterware cannot manage resources on servers while the servers are in this state.

RECONFIGURING

When servers move between server pools due to server pool reconfiguration, a server is placed into this state if resources that ran on it in the current server pool must be stopped and relocated. This happens because resources running on the server may not be configured to run in the server pool to which the server is moving. As soon as the resources are successfully relocated, the server is put back into the Online state.

Use the crsctl status server command to obtain server information.

STATE_DETAILS

This is a read-only attribute that Oracle Clusterware manages. The attribute provides additional details about the state of a server. Possible additional details about a server state are:

Server state: ONLINE:

  • AUTOSTARTING RESOURCES

    Indicates that the resource autostart procedure (performed when a server reboots or the Oracle Clusterware stack is restarted) is in progress for the server.

  • AUTOSTART QUEUED

    The server is waiting for the resource autostart to commence. Once that happens, the attribute value changes to AUTOSTARTING RESOURCES.

Server state: RECONFIGURING:

  • STOPPING RESOURCES

    Resources that are restricted from running in a new server pool are stopping.

  • STARTING RESOURCES

    Resources that can run in a new sever pool are starting.

  • RECONFIG FAILED

    One or more resources did not stop and thus the server cannot transition into the ONLINE state. At this point, manual intervention is required. You must stop or unregister resources that did not stop. After that, the server automatically transitions into the ONLINE state.

Server state: JOINING:

  • CHECKING RESOURCES

    Whenever a server reboots, the Oracle Clusterware stack restarts, or crsd on a server restarts, the policy engine must determine the current state of the resources on the server. While that procedure is in progress, this value is returned.


Understanding Server Pools

This section contains the following topics:

How Server Pools Work

Server pools divide the cluster into groups of servers hosting the same or similar resources. They distribute a uniform workload (a set of Oracle Clusterware resources) over several servers in the cluster. For example, you can restrict Oracle databases to run only in a particular server pool. When you enable role-separated management, you can explicitly grant permission to operating system users to change attributes of certain server pools.

Top-level server pools:

  • Logically divide the cluster

  • Are always exclusive, meaning that one server can only reside in one particular server pool at a certain point in time

Server pools each have three attributes that they are assigned when they are created:

  • MIN_SIZE: The minimum number of servers the server pool should contain. If the number of servers in a server pool is below the value of this attribute, then Oracle Clusterware automatically moves servers from elsewhere into the server pool until the number of servers reaches the attribute value.

  • MAX_SIZE: The maximum number of servers the server pool should contain.

  • IMPORTANCE: A number from 0 to 1000 (0 being least important) that ranks a server pool among all other server pools in a cluster.

When Oracle Clusterware is installed, two server pools are created automatically: Generic and Free. All servers in a new installation are assigned to the Free server pool, initially. Servers move from Free to newly defined server pools automatically. When you upgrade Oracle Clusterware, all nodes are assigned to the Generic server pool, to ensure compatibility with database releases before Oracle Database 11g release 2 (11.2).

The Free Server Pool

The Free server pool contains servers that are not assigned to any other server pools. The attributes of the Free server pool are restricted, as follows:

  • SERVER_NAMES, MIN_SIZE, and MAX_SIZE cannot be edited by the user

  • IMPORTANCE and ACL can be edited by the user

The Generic Server Pool

The Generic server pool stores pre-11g release 2 (11.2) databases and administrator-managed databases that have fixed configurations. Additionally, the Generic server pool contains servers that match either of the following:

  • Servers that you specified in the HOSTING_MEMBERS resource attribute of all resources of the application resource type

    See Also:

    "HOSTING_MEMBERS" for more information about this attribute
  • Servers with names you specified in the SERVER_NAMES attribute of the server pools that list the Generic server pool as a parent server pool

The Generic server pool's attributes are restricted, as follows:

  • No one can modify configuration attributes of the Generic server pool (all attributes are read-only)

  • When you specify a server name in the HOSTING_MEMBERS resource attribute, Oracle Clusterware only allows it if the server is:

    • Online and exists in the Generic server pool

    • Online and exists in the Free server pool, in which case Oracle Clusterware moves the server into the Generic server pool

    • Online and exists in any other server pool and the client is either a cluster administrator or is allowed to use the server pool's servers, in which case, the server is moved into the Generic server pool

    • Offline and the client is a cluster administrator

  • When you register a child server pool with the Generic server pool, Oracle Clusterware only allows it if the server names pass the same requirements as previously specified for the resources.

    Servers are initially considered for assignment into the Generic server pool at cluster startup time or when a server is added to the cluster, and only after that to other server pools.

How Oracle Clusterware Assigns Servers to Server Pools

When a server joins the cluster, Oracle Clusterware assigns the server to a server pool in the following order:

  1. Until all server pools are filled in order of importance to their minimum (MIN_SIZE).

  2. Until all server pools are filled in order of importance to their maximum (MAX_SIZE).

  3. By default, any servers not placed in a server pool go into the Free server pool.

    You can modify the IMPORTANCE attribute for the Free server pool.

Servers Moving from Server Pool to Server Pool

If the number of servers in a server pool falls below the value of the MIN_SIZE attribute for the server pool (such as when a server fails), based on values you set for the MIN_SIZE and IMPORTANCE attributes for all server pools, Oracle Clusterware can move servers from other server pools into the server pool whose number of servers has fallen below the value for MIN_SIZE. Oracle Clusterware selects servers from other server pools to move into the deficient server pool that meet the following criteria:

  • For server pools that have a lower IMPORTANCE value than the deficient server pool, Oracle Clusterware can take servers from those server pools even if it means that the number of servers falls below the value for the MIN_SIZE attribute.

  • For server pools with equal or greater IMPORTANCE, Oracle Clusterware only takes servers from those server pools if the number of servers in a server pool is greater than the value of its MIN_SIZE attribute.

Table 2-2 lists and describes all server pool attributes.

Table 2-2 Server Pool Attributes

Attribute Values and Format Description
ACL

String in the following format:

owner:user:rwx,pgrp:group:rwx,other::r—

Defines the owner of the server pool and which privileges are granted to various operating system users and groups. The server pool owner defines the operating system user of the owner, and which privileges that user is granted.

The value of this optional attribute is populated at the time a server pool is created based on the identity of the process creating the server pool, unless explicitly overridden. The value can subsequently be changed, if such a change is allowed based on the existing privileges of the server pool.

In the string:

  • owner: The operating system user of the server pool owner, followed by the privileges of the owner

  • pgrp: The operating system group that is the primary group of the owner of the server pool, followed by the privileges of members of the primary group

  • other: Followed by privileges of others

  • r: Read only

  • w: Modify attributes of the pool or delete it

  • x: Assign resources to this pool

By default, the identity of the client that creates the server pool is the owner. Also by default, root, and the user specified in owner have full privileges. You can grant required operating system users and operating system groups their privileges by adding the following lines to the ACL attribute:

user:username:rwx
group:group_name:rwx
ACTIVE_SERVERS

A string of server names in the following format:

server_name1 server_name2 ...

Oracle Clusterware automatically manages this attribute, which contains the space-delimited list of servers that are currently assigned to a server pool.

EXCLUSIVE_POOLS

String

This optional attribute indicates if servers assigned to this server pool are shared with other server pools. A server pool can explicitly state that it is exclusive of any other server pool that has the same value for this attribute. Two or more server pools are mutually exclusive when the sets of servers assigned to them do not have a single server in common. For example, server pools A and B must be exclusive if they both set the value of this attribute to foo_A_B.

Top-level server pools are mutually exclusive, by default.

IMPORTANCE

Any integer from 0 to 1000

Relative importance of the server pool, with 0 denoting the lowest level of importance and 1000, the highest. This optional attribute is used to determine how to reconfigure the server pools when a node joins or leaves the cluster. The default value is 0.

MAX_SIZE

Any nonnegative integer or -1 (no limit)

The maximum number of servers a server pool can contain. This attribute is optional and is set to -1 (no limit), by default.

Note: A value of -1 for this attribute spans the entire cluster.

MIN_SIZE

Any nonnegative integer

The minimum size of a server pool. If the number of servers contained in a server pool is below the number you specify in this attribute, then Oracle Clusterware automatically moves servers from other pools into this one until the that number is met.

Note: The value of this optional attribute does not set a hard limit. It governs the priority for server assignment whenever the cluster is reconfigured. The default value is 0.

NAME

String

The name of the server pool, which you must specify when you create the server pool. Server pool names must be unique within the domain of names of user-created entities, such as resources, types, and servers. A server pool name can contain any platform-supported characters except the exclamation point (!) and the tilde (~). A server pool name cannot begin with a period nor with ora. This attribute is required.

PARENT_POOLS

A string of space-delimited server pool names in the following format:

sp1 sp2 ...

Use of this attribute makes it possible to create nested server pools. Server pools listed in this attribute are referred to as parent server pools. A server pool included in a parent server pool is referred to as a child server pool.

SERVER_NAMES

A string of space-delimited server names in the following format:

server1 server2 ...

A list of candidate node names that may be associated with a server pool. If this optional attribute is empty, Oracle Clusterware assumes that any server may be assigned to any server pool, to the extent allowed by values of other attributes, such as PARENT_POOLS.

The server names identified as candidate node names are not validated to confirm that they are currently active cluster members. Cluster administrators can use this attribute to define servers as candidates that have not yet been added to the cluster.


You manage server pools that are managing Oracle RAC databases with the Server Control (SRVCTL) utility. Use the Oracle Clusterware Control (CRSCTL) utility to manage all other server pools. Only cluster administrators have permission to create top-level server pools.

How Oracle Clusterware Assigns New Servers

Oracle Clusterware assigns new servers to server pools in the following order:

  1. Generic server pool

  2. User-created server pool

  3. Free server pool

When a server joins a cluster, several things occur.

Consider the server pools configured in Table 2-3:

Table 2-3 Sample Server Pool Attributes Configuration

NAME IMPORTANCE MIN_SIZE MAX_SIZE PARENT_POOLS EXCLUSIVE_POOLS
sp1
1
1
10

 

 

sp2
3
1
6

 

 

sp3
2
1
2

 

 

sp2_1
2
1
5
sp2
S123
sp2_2
1
1
5
sp2
S123

For example, assume that there are no servers in a cluster; all server pools are empty.

When a server, named server1, joins the cluster:

  1. Server-to-pool assignment commences.

  2. Oracle Clusterware only processes top-level server pools (those that have no parent server pools), first. In this example, the top-level server pools are sp1, sp2, and sp3.

  3. Oracle Clusterware lists the server pools in order of IMPORTANCE, as follows: sp2, sp3, sp1.

  4. Oracle Clusterware assigns server1 to sp2 because sp2 has the highest IMPORTANCE value and its MIN_SIZE value has not yet been met.

  5. Oracle Clusterware processes the remaining two server pools, sp2_1 and sp2_2. The sizes of both server pools are below the value of the MIN_SIZE attribute (both server pools are empty and have MIN_SIZE values of 1).

  6. Oracle Clusterware lists the two remaining pools in order of IMPORTANCE, as follows: sp2_1, sp2_2.

  7. Oracle Clusterware assigns server1 to sp2_1 but cannot assign server1 to sp2_2 because sp2_1 is configured to be exclusive with sp2_2.

After processing, the cluster configuration appears, as follows

Table 2-4 Post Processing Server Pool Configuration

Server Pool Name Assigned Servers
sp1

 

sp2
server1
sp3

 

sp2_1
server1
sp2_2

 


Role-Separated Management

This section contains the following topics

About Role-Separated Management

Role-separated management is a feature you can implement that enables multiple resources to share the same cluster and hardware resources by setting permissions on server pools or resources, and then using access control lists (ACLs) to provide access. By default, this feature is not implemented during installation.

Role-separated management can be implemented in two ways:

  • Vertical implementation: Access permissions to server pools or resources are granted by assigning ownership of them to different users for each layer in the stack, and using ACLs assigned to those users. Oracle ASM provides an even more granular approach using groups. Careful planning is required to enable overlapping tasks.

  • Horizontal implementation: Access permissions for resources are granted using ACLS assigned to server pools and policy-managed databases or applications.

Role-separated management in Oracle Clusterware depends on a cluster administrator. The set of users that are cluster administrators is managed within Oracle Clusterware, as opposed to being an operating system group. By default, after an Oracle Grid Infrastructure for a cluster installation or after an upgrade, all users are cluster administrators (denoted by the asterisk (*) in the list of cluster administrators). However, the user that installed Oracle Clusterware in the Grid Infrastructure home (Grid home) and root are permanent cluster administrators, and only these two users can add or remove users from the group. Additionally, only a permanent cluster administrator can remove the asterisk (*) value from the cluster administrator group, changing the group to enable role-separation management. When you enable this feature, you can limit the cluster administrator group to only those users added by the permanent cluster administrators.

If the cluster is shared by various users, then the cluster administrator can restrict access to certain server pools and, consequently, to certain hardware resources to specific users in the cluster. The permissions are stored for each server pool in the ACL attribute, described in Table 2-2.

Managing Cluster Administrators in the Cluster

Use the following commands to manage cluster administrators in the cluster:

  • To query the list of users that are cluster administrators:

    $ crsctl query crs administrator
    
  • To enable role-separated management, you must remove the * value from the list of cluster administrators, as either the user who installed Oracle Clusterware or root, as follows:

    # crsctl delete crs administrator -u "*"
    

    The asterisk (*) must be enclosed in double quotation marks ("").

  • To add specific users to the group of cluster administrators:

    # crsctl add crs administrator -u user_name
    

    To make all users cluster administrators, enter -u "*".

  • To remove specific users from the group of cluster administrators:

    # crsctl delete crs administrator -u user_name
    

Configuring Horizontal Role Separation

Use the crsctl command crsctl setperm to configure horizontal role separation using ACLs that are assigned to server pools, resources, or both. The crsctl command is located in the path Grid_home/bin, where Grid_home is the Oracle Grid Infrastructure for a cluster home.

The command uses the following syntax, where the access control string (ACL) is indicated by italics:

crsctl setperm {resource|type|serverpool} name {-u aclstring|-x aclstring|-o user_name|-g group_name}

The flag options are:

  • -u

    Update the entity ACL

  • -x

    Delete the entity ACL

  • -o

    Change the entity owner

  • -g

    Change the entity primary group

The ACL strings are:

{ user:user_name[:readPermwritePermexecPerm]   |
     group:group_name[:readPermwritePermexecPerm] |
     other[::readPermwritePermexecPerm] }

where:

  • user

    designates the user ACL (access permissions granted to the designated user)

  • group

    designates the group ACL (permissions granted to the designated group members)

  • other

    designates the other ACL (access granted to users or groups not granted particular access permissions)

  • readperm

    Location of the read permission; r grants, and - forbids)

  • writeperm

    Location of the write permission; w grants, and - forbids)

  • execperm

    Location of the execute permission; x grants, and - forbids)

For example, to set permissions on a server pool called psft for the group personnel, where the administrative user has read/write/execute privileges, the psft group has read/write privileges, and users outside of the group are granted no access, enter the following command as the root user:

# crsctl setperm serverpool psft -o personadmin -g personnel:rwxr-w---

Voting Disk, Oracle Cluster Registry, and Oracle Local Registry

This section includes the following topics:

About Voting Disks, Oracle Cluster Registry, and Oracle Local Registry

Oracle Clusterware includes three components: voting disks, Oracle Cluster Registry (OCR), and Oracle Local Registry (OLR).

  • Voting disks manage information about node membership. Each voting disk must be accessible by all nodes in the cluster for nodes to be members of the cluster

  • OCR manages Oracle Clusterware and Oracle RAC database configuration information

  • OLR resides on every node in the cluster and manages Oracle Clusterware configuration information for each particular node

You can store voting disks and the OCR on Oracle Automatic Storage Management (Oracle ASM), or a certified cluster file system.

Oracle Universal Installer for Oracle Clusterware 11g release 2 (11.2), does not support the use of raw or block devices. However, if you upgrade from a previous Oracle Clusterware release, then you can continue to use raw or block devices. Oracle recommends that you use Oracle ASM to store voting disks and OCR.

Oracle recommends that you configure multiple voting disks during Oracle Clusterware installation to improve availability. If you choose to put the voting disks into an Oracle ASM disk group, then Oracle ASM ensures the configuration of multiple voting disks if you use a normal or high redundancy disk group. If you choose to store the voting disks on a cluster file system, then select the option to configure multiple voting disks, in which case you will have to specify three different file systems based on different disks.

If necessary, you can dynamically add or replace voting disks after you complete the Oracle Clusterware installation process without stopping the cluster.

Note:

If you use CRSCTL to add a new voting disk to a raw device after installation, then the file permissions of the new voting file on remote nodes may be incorrect. On each remote node, check to ensure that the file permissions for the voting file are correct (owned by the Grid Infrastructure installation owner and by members of the OINSTALL group). If the permissions are incorrect, then change them manually.

For example:

grid@myserver>$ ls -l /dev/rhdisk18
crwxrwxrwx  1 root  oinstall  36, 02 Feb 10 20:28 /dev/rhdisk18
$ su root
root@myserver> # chmod grid:oinstall /dev/rhdisk18
# exit
$ ls -l /dev/rhdisk18
crwxrwxrwx  1001 grid  oinstall  36, 19 Sep 09 20:29 /dev/rhdisk18

Managing Voting Disks

This section includes the following topics for managing voting disks in your cluster:

Notes:

  • Voting disk management requires a valid and working OCR. Before you add, delete, replace, or restore voting disks, run the ocrcheck command as root. If OCR is not available or it is corrupt, then you must restore OCR as described in "Restoring Oracle Cluster Registry".

  • If you upgrade from a previous version of Oracle Clusterware to 11g release 2 (11.2) and you want to store voting disks in an Oracle ASM disk group, then you must set the ASM Compatibility (COMPATIBLE.ASM) compatibility attribute to 11.2.0.0.

See Also:

Storing Voting Disks on Oracle ASM

Oracle ASM manages voting disks differently from other files that it stores. If you choose to store your voting disks in Oracle ASM, then Oracle ASM stores all the voting disks for the cluster in the disk group you choose. You cannot combine voting disks stored in Oracle ASM and voting disks not stored in Oracle ASM in the same cluster.

Once you configure voting disks on Oracle ASM, you can only make changes to the voting disks' configuration using the crsctl replace votedisk command. This is true even in cases where there are no working voting disks. Despite the fact that crsctl query css votedisk reports zero vote disks in use, Oracle Clusterware remembers the fact that Oracle ASM was in use and the replace verb is required. Only after you use the replace verb to move voting disks back to non-Oracle ASM storage are the verbs add css votedisk and delete css votedisk again usable.

The number of voting files you can store in a particular Oracle ASM disk group depends upon the redundancy of the disk group.

  • External redundancy: A disk group with external redundancy can store only one voting disk

  • Normal redundancy: A disk group with normal redundancy stores three voting disks

  • High redundancy: A disk group with high redundancy stores five voting disks

By default, Oracle ASM puts each voting disk in its own failure group within the disk group. A failure group is a subset of the disks in a disk group. Failure groups define disks that share components, such that if one fails then other disks sharing the component might also fail. An example of what you might define as a failure group would be a set of SCSI disks sharing the same SCSI controller. Failure groups are used to determine which Oracle ASM disks to use for storing redundant data. For example, if two-way mirroring is specified for a file, then redundant copies of file extents must be stored in separate failure groups.

If voting disks stored on Oracle ASM with normal or high redundancy, and the storage hardware in one failure group suffers a failure, then if there is another disk available in a disk group in an unaffected failure group, Oracle ASM recovers the voting disk in the unaffected failure group.

A normal redundancy disk group must contain at least two failure groups but if you are storing your voting disks on Oracle ASM, then a normal redundancy disk group must contain at least three failure groups. A high redundancy disk group must contain at least three failure groups. However, Oracle recommends using several failure groups. A small number of failure groups, or failure groups of uneven capacity, can create allocation problems that prevent full use of all of the available storage.

You must specify enough failure groups in each disk group to support the redundancy type for that disk group.

Using the crsctl replace votedisk command, you can place a given set of voting disks from one Oracle ASM disk group into another, or onto a certified file system. Doing so, you can change the number of voting disks by placing them in a disk group of a different redundancy level as the former disk group.

Notes:

  • You cannot directly influence the number of voting disks in one disk group.

  • You cannot use the crsctl add | delete votedisk commands on voting disks stored in Oracle ASM disk groups because Oracle ASM manages the number of voting disks according to the redundancy level of the disk group.

  • Neither should you add a voting disk to a cluster file system in addition to the voting disks stored in an Oracle ASM disk group. Oracle does not support having voting disks in Oracle ASM and directly on a cluster file system for the same cluster at the same time.

See Also:

Backing Up Voting Disks

In Oracle Clusterware 11g release 2 (11.2), you no longer have to back up the voting disk. The voting disk data is automatically backed up in OCR as part of any configuration change and is automatically restored to any voting disk added. If all voting disks are corrupted, however, you can restore them as described in "Restoring Voting Disks".

Restoring Voting Disks

If all of the voting disks are corrupted, then you can restore them, as follows:

  1. Restore OCR as described in "Restoring Oracle Cluster Registry", if necessary.

    This step is necessary only if OCR is also corrupted or otherwise unavailable, such as if OCR is on Oracle ASM and the disk group is no longer available.

    See Also:

    Oracle Automatic Storage Management Administrator's Guide for more information about managing Oracle ASM disk groups
  2. Run the following command as root from only one node to start the Oracle Clusterware stack in exclusive mode, which does not require voting files to be present or usable:

    # crsctl start crs -excl
    
  3. Run the following command to retrieve the list of voting files currently defined:

    $ crsctl query css votedisk
    

    This list may be empty if all voting disks were corrupted, or may have entries that are marked as status 3 or OFF.

  4. Depending on where you store your voting files, do one of the following:

    • If the voting disks are stored in Oracle ASM, then run the following command to migrate the voting disks to the Oracle ASM disk group you specify:

      crsctl replace votedisk +asm_disk_group
      

      The Oracle ASM disk group to which you migrate the voting files must exist in Oracle ASM. You can use this command whether the voting disks were stored in Oracle ASM or some other storage device.

    • If you did not store voting disks in Oracle ASM, then run the following command using the File Universal Identifier (FUID) obtained in the previous step:

      $ crsctl delete css votedisk FUID
      

      Add a voting disk, as follows:

      $ crsctl add css votedisk path_to_voting_disk
      
  5. Stop the Oracle Clusterware stack as root:

    # crsctl stop crs
    
  6. Restart the Oracle Clusterware stack in normal mode as root:

    # crsctl start crs
    

Adding, Deleting, or Migrating Voting Disks

You can add, remove, and migrate voting disks after you install Oracle Clusterware. Note that the commands you use to do this are different, depending on whether your voting disks are located in Oracle ASM, or are located in another storage option.

Use the following commands to modify voting disks that are stored in Oracle ASM:

  • To display the voting disk FUID and file path of each current voting disk, run the crsctl query css votedisk command to display output similar to the following:

    $ crsctl query css votedisk
    --  -----    -----------------                --------- ---------
    ##  STATE    File Universal Id                File Name Disk group
     1. ONLINE   7c54856e98474f61bf349401e7c9fb95 (/dev/sdb1) [DATA]
    

    This command returns a disk sequence number, the status of the disk, the FUID, the path of the disk, and the name of the Oracle ASM disk group on which the disk is stored.

  • To migrate voting disks from Oracle ASM to an alternative storage device, specify the path to the non-Oracle ASM storage device with which you want to replace the Oracle ASM disk group using the following command:

    $ crsctl replace votedisk path_to_voting_disk
    

    You can run this command on any node in the cluster.

  • To replace all voting disks not stored in Oracle ASM with voting disks managed by Oracle ASM in an Oracle ASM disk group, run the following command:

    $ crsctl replace votedisk +asm_disk_group
    

Use the following commands to modify voting disks that are not stored on Oracle ASM:

  • To display the voting disk FUID and file path of each current voting disk, run the following command:

    $ crsctl query css votedisk
    ##  STATE    File Universal Id                File Name Disk group
    --  -----    -----------------                --------- ---------
     1. ONLINE   7c54856e98474f61bf349401e7c9fb95 (/cfs/host09_vd3) []
    

    This command returns a disk sequence number, the status of the disk, the FUID, and the path of the disk and no name of an Oracle ASM disk group.

  • To add one or more voting disks, run the following command, replacing the path_to_voting_disk variable with one or more space-delimited, complete paths to the voting disks you want to add:

    $ crsctl add css votedisk path_to_voting_disk [...]
    
  • To replace voting disk A with voting disk B, you must add voting disk B, and then delete voting disk A. To add a new disk and remove the existing disk, run the following command, replacing the path_to_voting_diskB variable with the fully qualified path name of voting disk B:

    $ crsctl add css votedisk path_to_voting_diskB -purge
    

    The -purge option deletes existing voting disks.

    Use the crsctl replace votedisk command to replace a voting disk with an Oracle ASM disk group.

  • To remove a voting disk, run the following command, specifying one or more space-delimited, voting disk FUIDs or comma-delimited directory paths to the voting disks you want to remove:

    $ crsctl delete css votedisk {FUID | path_to_voting_disk[...]}
    
  • To migrate voting disks to Oracle ASM, specify the Oracle ASM disk group name in the following command:

    $ crsctl replace votedisk +asm_disk_group
    

    You can run this command on any node in the cluster.

Note:

If the cluster is down and cannot restart due to lost voting disks, then you must start CSS in exclusive mode to replace the voting disks by entering the following command:
$ crsctl start crs -excl

After modifying the voting disk, verify the voting disk location, as follows:

$ crsctl query css votedisk

See Also:

Appendix E, "CRSCTL Utility Reference" for more information about CRSCTL commands

Managing the Oracle Cluster Registry and Oracle Local Registries

This section describes how to manage the OCR and the Oracle Local Registry (OLR) with the following utilities: OCRCONFIG, OCRDUMP, and OCRCHECK.

The OCR contains information about all Oracle resources in the cluster.

The OLR is a registry similar to the OCR located on each node in a cluster, but contains information specific to each node. It contains manageability information about Oracle Clusterware, including dependencies between various services. The Oracle High Availability Services uses this information. The OLR is located on local storage on each node in a cluster. Its default location is in the path Grid_home/cdata/host_name.olr, where Grid_home is the Oracle Grid Infrastructure home, and host_name is the host name of the node.

This section describes how to administer the OCR in the following topics:

See Also:

"About OCRCONFIG" for information about the OCRCONFIG utility, and "Troubleshooting Oracle Cluster Registry" for information about the OCRDUMP and OCRCHECK utilities

Migrating Oracle Cluster Registry to Oracle Automatic Storage Management

To improve Oracle Clusterware storage manageability, OCR is configured, by default, to use Oracle ASM in Oracle Database 11g release 2 (11.2). With the Oracle Clusterware storage residing in an Oracle ASM disk group, you can manage both database and clusterware storage using Oracle Enterprise Manager.

However, if you upgrade from a previous version of Oracle Clusterware, you can migrate OCR to reside on Oracle ASM, and take advantage of the improvements in managing Oracle Clusterware storage.

Note:

If you upgrade from a previous version of Oracle Clusterware to 11g release 2 (11.2) and you want to store OCR in an Oracle ASM disk group, then you must set the ASM Compatibility compatibility attribute to 11.2.0.0.

See Also:

Oracle Automatic Storage Management Administrator's Guide for information about setting Oracle ASM compatibility attributes

To migrate OCR to Oracle ASM using OCRCONFIG:

  1. Ensure that Oracle Clusterware upgrade to 11g release 2 (11.2) is complete. Run the following command to verify the current running version:

    $ crsctl query crs activeversion
    
  2. Use the Oracle ASM Configuration Assistant (ASMCA) to configure and start Oracle ASM on all nodes in the cluster.

    See Also:

    Oracle Automatic Storage Management Administrator's Guide for more information about using ASMCA
  3. Use ASMCA to create an Oracle ASM disk group that is at least the same size of the existing OCR and has at least normal redundancy.

    Notes:

    • If OCR is stored in an Oracle ASM disk group with external redundancy, then Oracle recommends that you add another OCR location to another disk group to avoid the loss of OCR, if a disk fails in the disk group.

      Oracle does not support storing the OCR on different storage types simultaneously, such as storing OCR on both Oracle ASM and a shared file system, except during a migration.

    • If an Oracle ASM instance fails on any node, then OCR becomes unavailable on that particular node.

      If the crsd process running on the node affected by the Oracle ASM instance failure is the OCR writer, the majority of the OCR locations are stored in Oracle ASM, and you attempt I/O on OCR during the time the Oracle ASM instance is down on this node, then crsd stops and becomes inoperable. Cluster management is now affected on this particular node.

      Under no circumstances will the failure of one Oracle ASM instance on one node affect the whole cluster.

    • Ensure that Oracle ASM disk groups that you create are mounted on all of the nodes in the cluster.

    See Also:

    Oracle Grid Infrastructure Installation Guide for more detailed sizing information
  4. To add OCR to an Oracle ASM disk group, ensure that the Oracle Clusterware stack is running and run the following command as root:

    # ocrconfig -add +new_disk_group
    

    You can run this command more than once if you add multiple OCR locations. You can have up to five OCR locations. However, each successive run must point to a different disk group.

  5. To remove storage configurations no longer in use, run the following command as root:

    # ocrconfig -delete old_storage_location
    

    Run this command for every configured OCR.

The following example shows how to migrate two OCRs to Oracle ASM using OCRCONFIG.

# ocrconfig -add +new_disk_group
# ocrconfig -delete /dev/raw/raw2
# ocrconfig -delete /dev/raw/raw1

Note:

OCR inherits the redundancy of the disk group. If you want high redundancy for OCR, you must configure the disk group with high redundancy when you create it.
Migrating Oracle Cluster Registry from Oracle ASM to Other Types of Storage

To migrate OCR from Oracle ASM to another storage type:

  1. Ensure that Oracle Clusterware upgrade to 11g release 2 (11.2) is complete. Run the following command to verify the current running version:

    $ crsctl query crs activeversion
    
  2. Create a file with the following permissions: root, oinstall, 640.

    Note:

    Create at least two mirrorw of the primary storage location to eliminate a single point of failure for OCR. OCR supports up to five locations.
  3. Ensure there is at least 280 MB of space on the mount partition.

  4. Ensure that the file you created is visible from all nodes in the cluster.

  5. To add the file as an OCR location, ensure that the Oracle Clusterware stack is running and run the following command as root:

    # ocrconfig -add new_file_location
    

    You can run this command more than once if you add more than one OCR location. Each successive run of this command must point to a different file location.

  6. To remove storage configurations no longer in use, run the following command as root:

    # ocrconfig -delete unused_storage_location
    

    You can run this command more than once if there is more than one OCR location configured.

The following example shows how to migrate OCR from Oracle ASM to block devices using OCRCONFIG. For OCRs not stored on Oracle ASM, Oracle recommends that you mirror OCR on different devices.

# ocrconfig -add /dev/sdd1
# ocrconfig -add /dev/sde1
# ocrconfig -add /dev/sdf1
# ocrconfig -delete +unused_disk_group

Adding, Replacing, Repairing, and Removing Oracle Cluster Registry Locations

The Oracle installation process for Oracle Clusterware gives you the option of automatically mirroring OCR. You can manually put the mirrored OCRs on a shared network file system (NFS), or on any cluster file system that is certified by Oracle. Alternatively, you can place the OCR on Oracle ASM and allow it to create mirrors automatically, depending on the redundancy option you select.

This section includes the following topics:

You can manually mirror OCR, as described in the "Adding an Oracle Cluster Registry Location" section, if you:

  • Upgraded to 11g release 2 (11.2) but did not choose to mirror OCR during the upgrade

  • Created only one OCR location during the Oracle Clusterware installation

Notes:

  • Oracle recommends that you configure:

    At least three OCR locations, if the OCR is configured on non-mirrored or non-redundant storage. Oracle strongly recommends that you mirror the OCR if the underlying storage is not RAID. Mirroring can help prevent the OCR from becoming a single point of failure.

    At least two OCR locations if the OCR is configured on an Oracle ASM disk group. You should configure the OCR in two independent disk groups. Typically this is the work area and the recovery area.

    At least two OCR locations if the OCR is configured on mirrored hardware or third-party mirrored volumes.

  • If the original OCR location does not exist, then you must create an empty (0 byte) OCR location with appropriate permissions before you run the ocrconfig -add or ocrconfig -replace commands.

  • Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

  • Ensure that the Oracle ASM disk group that you specify exists and is mounted.

  • The new OCR file, device, or disk group must be accessible from all of the active nodes in the cluster.

See Also:

In addition to mirroring OCR locations, you can also:

Note:

The operations in this section affect OCR clusterwide: they change the OCR configuration information in the ocr.loc file on Linux and UNIX systems and the Registry keys on Windows systems. However, the ocrconfig command cannot modify OCR configuration information for nodes that are shut down or for nodes on which Oracle Clusterware is not running.
Adding an Oracle Cluster Registry Location

Use the procedure in this section to add an OCR location. Oracle Clusterware can manage up to five redundant OCR locations.

Note:

If OCR resides on a cluster file system file or a network file system, create an empty (0 byte) OCR location file before performing the procedures in this section.

As the root user, run the following command to add an OCR location to either Oracle ASM or other storage device:

# ocrconfig -add +asm_disk_group | file_name

Note:

On Linux and UNIX systems, you must be root to run ocrconfig commands. On Windows systems, the user must be a member of the Administrator's group.
Removing an Oracle Cluster Registry Location

To remove an OCR location or a failed OCR location, at least one other OCR must be online. You can remove an OCR location to reduce OCR-related overhead or to stop mirroring your OCR because you moved OCR to redundant storage such as RAID.

Perform the following procedure as the root user to remove an OCR location from your Oracle Clusterware environment:

  1. Ensure that at least one OCR location other than the OCR location that you are removing is online.

    Caution:

    Do not perform this OCR removal procedure unless there is at least one other active OCR location online.
  2. Run the following command on any node in the cluster to remove an OCR location from either Oracle ASM or other location:

    # ocrconfig -delete +ASM_disk_group | file_name
    

    The file_name variable can be a device name or a file name. This command updates the OCR configuration on all of the nodes on which Oracle Clusterware is running.

Replacing an Oracle Cluster Registry Location

If you must change an existing OCR location, or change a failed OCR location to a working location, then you can use the following procedure, if one OCR location remains online.

To change an Oracle Cluster Registry location:

Complete the following procedure:

  1. Use the OCRCHECK utility to verify that a copy of OCR other than the one you are going to replace is online, using the following command:

    $ ocrcheck
    

    OCRCHECK displays all OCR locations that are registered and whether they are available (online). If an OCR location suddenly becomes unavailable, then it might take a short period for Oracle Clusterware to show the change in status.

    Note:

    The OCR location that you are replacing can be either online or offline.
  2. Use the following command to verify that Oracle Clusterware is running on the node on which the you are going to perform the replace operation:

    $ crsctl check crs
    
  3. Run the following command as root to replace the current OCR location using either destination_file or +ASM_disk_group to indicate the current and target OCR locations:

    # ocrconfig -replace current_OCR_location -replacement new_OCR_location
    

    If you have only one OCR location, then use the following commands:

    # ocrconfig -add +new_storage_disk_group
    # ocrconfig -delete +current_disk_group
    

    See Also:

    Oracle Automatic Storage Management Administrator's Guide for more information about migrating storage
  4. If any node that is part of your current Oracle RAC cluster is shut down, then use the following command syntax on the stopped node to let that node rejoin the cluster after the node is restarted where you use either a destination_file or +ASM_disk_group to indicate the current and target OCR locations:

    ocrconfig -repair -replace current_OCR_location -replacement new_OCR_location
    
Repairing an Oracle Cluster Registry Configuration on a Local Node

It may be necessary to repair the OCR if your cluster configuration changes while that node is stopped and this node is the only member in the cluster. Repairing an OCR involves either adding, deleting, or replacing an OCR location. For example, you may have to repair an OCR location on a node that was stopped while you were adding, replacing, or removing the OCR. To repair an OCR, run the following command as root on the node on which you have stopped the Oracle Clusterware daemon:

# ocrconfig -repair -add file_name | -delete file_name | -replace
current_file_name -replacement new_file_name

This operation only changes the OCR on the node on which you run this command. For example, if the OCR location is /dev/sde1, then use the command syntax ocrconfig -repair -add /dev/sde1 on this node to repair the OCR on that node.

Notes:

  • If your cluster configuration changes while the node on which OCR resides is stopped, and the Oracle Clusterware stack is running on the other nodes, then OCR detects configuration changes and self-corrects the configuration by changing the contents of the ocr.loc file.

  • You cannot repair the OCR configuration on a node on which the Oracle Cluster Ready Services daemon is running.

  • When you repair the OCR on a stopped node using ocrconfig -repair, you must provide the same OCR file name (which should be case-sensitive) as the OCR file names on other nodes.

  • If you run the ocrconfig -add | -repair | -replace command, then the device, file, or Oracle ASM disk group that you are adding must be accessible. This means that a device must exist. You must create an empty (0 byte) OCR location, or the Oracle ASM disk group must exist and be mounted.

See Also:

Oracle Automatic Storage Management Administrator's Guide for more information about Oracle ASM disk group management
Overriding the Oracle Cluster Registry Data Loss Protection Mechanism

OCR has a mechanism that prevents data loss due to accidental overwrites. If you configure a mirrored OCR and if Oracle Clusterware cannot access the mirrored OCR locations and also cannot verify that the available OCR location contains the most recent configuration, then Oracle Clusterware prevents further modification to the available OCR location. In addition, the process prevents overwriting by prohibiting Oracle Clusterware from starting on the node on which only one OCR is available. In such cases, Oracle Database displays an alert message in either Oracle Enterprise Manager, the Oracle Clusterware alert log files, or both. If this problem is local to only one node, you can use other nodes to start your cluster database.

However, if you are unable to start any cluster node in your environment and if you can neither repair OCR nor restore access to all OCR locations, then you can override the protection mechanism. The procedure described in the following list enables you to start the cluster using the available OCR location. However, overriding the protection mechanism can result in the loss of data that was not available when the previous known good state was created.

Note:

Overriding OCR using the following procedure can result in the loss of OCR updates that were made between the time of the last known good OCR update made to the currently accessible OCR and the time at which you performed the overwrite. In other words, running the ocrconfig -overwrite command can result in data loss if the OCR location that you are using to perform the overwrite does not contain the latest configuration updates for your cluster environment.

Perform the following procedure to overwrite OCR if a node cannot start and if the alert log contains CLSD-1009 and CLSD-1011 messages.

  1. Attempt to resolve the cause of the CLSD-1009 and CLSD-1011 messages.

    Compare the node's OCR configuration (ocr.loc on Linux and UNIX systems and the Registry on Windows systems) with other nodes on which Oracle Clusterware is running.

    • If the configurations do not match, run ocrconfig -repair.

    • If the configurations match, ensure that the node can access all of the configured OCRs by running an ls command on Linux and UNIX systems. On Windows, use a dir command if the OCR location is a file and run GuiOracleObjectManager.exe to verify that the part of the cluster with the name exists.

  2. Ensure that the most recent OCR contains the latest OCR updates.

    Look at output from the ocrdump command and determine whether it has your latest updates.

  3. If you cannot resolve the problem that caused the CLSD message, then run the command ocrconfig -overwrite to start the node.

Backing Up Oracle Cluster Registry

This section describes how to back up OCR content and use it for recovery. The first method uses automatically generated OCR copies and the second method enables you to issue a backup command manually:

  • Automatic backups: Oracle Clusterware automatically creates OCR backups every four hours. At any one time, Oracle Database always retains the last three backup copies of OCR. The CRSD process that creates the backups also creates and retains an OCR backup for each full day and at the end of each week. You cannot customize the backup frequencies or the number of files that Oracle Database retains.

  • Manual backups: Run the ocrconfig -manualbackup command on a node where the Oracle Clusterware stack is up and running to force Oracle Clusterware to perform a backup of OCR at any time, rather than wait for the automatic backup. You must run the command as a user with administrative privileges. The -manualbackup option is especially useful when you want to obtain a binary backup on demand, such as before you make changes to the OCR. The OLR only supports manual backups.

When the clusterware stack is down on all nodes in the cluster, the backups that are listed by the command ocrconfig -showbackup may differ from node to node. After you install or upgrade Oracle Clusterware on a node, or add a node to the cluster, when the root.sh script finishes, it backs up the OLR.

Listing Backup Files

Run the following command to list the backup files:

ocrconfig -showbackup

The ocrconfig -showbackup command to displays the backup location, timestamp, and the originating node name of the backup files that Oracle Clusterware creates. By default, the -showbackup option displays information for both automatic and manual backups but you can include the auto or manual flag to display only the automatic backup information or only the manual backup information, respectively.

Run the following command to inspect the contents and verify the integrity of the backup file:

ocrdump -backupfile backup_file_name

You can use any backup software to copy the automatically generated backup files at least once daily to a different device from where the primary OCR resides.

The default location for generating backups on Linux or UNIX systems is Grid_home/cdata/cluster_name, where cluster_name is the name of your cluster. The Windows default location for generating backups uses the same path structure. Because the default backup is on a local file system, Oracle recommends that you include the backup file created with the OCRCONFIG utility as part of your operating system backup using standard operating system or third-party tools.

Tip:

You can use the ocrconfig -backuploc option to change the location where OCR creates backups. Appendix G, "Oracle Cluster Registry Configuration Utility Reference" describes the OCRCONFIG utility options.

Note:

On Linux and UNIX systems, you must be root user to run most but not all of the ocrconfig command options. On Windows systems, the user must be a member of the Administrator's group.

See Also:

"Administering Oracle Cluster Registry with Oracle Cluster Registry Export and Import Commands" to use manually created OCR export files to copy OCR content and use it for recovery

Restoring Oracle Cluster Registry

If a resource fails, then before attempting to restore OCR, restart the resource. As a definitive verification that OCR failed, run ocrcheck and if the command returns a failure message, then both the primary OCR and the OCR mirror have failed. Attempt to correct the problem using the OCR restoration procedure for your platform.

Notes:

See Also:

Oracle Automatic Storage Management Administrator's Guide for more information about managing Oracle ASM disk groups

Restoring the Oracle Cluster Registry on Linux or UNIX Systems

If you are storing the OCR on an Oracle ASM disk group, and that disk group is corrupt, then you must restore the Oracle ASM disk group using Oracle ASM utilities, and then mount the disk group again before recovering the OCR. Recover the OCR by running the command ocrconfig -restore.

See Also:

Oracle Automatic Storage Management Administrator's Guide for information about how to restore Oracle ASM disk groups

Use the following procedure to restore OCR on Linux or UNIX systems:

  1. List the nodes in your cluster by running the following command on one node:

    $ olsnodes
    
  2. Stop Oracle Clusterware by running the following command as root on all of the nodes:

    # crsctl stop crs
    

    If the preceding command returns any error due to OCR corruption, stop Oracle Clusterware by running the following command as root on all of the nodes:

    # crsctl stop crs -f
    
  3. If you are restoring OCR to a cluster file system or network file system, then run the following command as root to restore OCR with an OCR backup that you can identify in "Listing Backup Files":

    # ocrconfig -restore file_name
    

    After you complete this step, skip to step 8.

  4. Start the Oracle Clusterware stack on one node in exclusive mode by running the following command as root:

    # crsctl start crs -excl
    

    Ignore any errors that display.

    Check whether crsd is running. If it is, stop it by running the following command as root:

    # crsctl stop resource ora.crsd -init
    

    Caution:

    Do not use the -init flag with any other command.
  5. Restore OCR with an OCR backup that you can identify in "Listing Backup Files" by running the following command as root:

    # ocrconfig -restore file_name
    

    Notes:

    • If the original OCR location does not exist, then you must create an empty (0 byte) OCR location before you run the ocrconfig -restore command.

    • Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

    • If you configured OCR in an Oracle ASM disk group, then ensure that the Oracle ASM disk group exists and is mounted.

    See Also:

  6. Verify the integrity of OCR:

    # ocrcheck
    
  7. Stop Oracle Clusterware on the node where it is running in exclusive mode:

    # crsctl stop crs -f
    
  8. Begin to start Oracle Clusterware by running the following command as root on all of the nodes:

    # crsctl start crs
    
  9. Verify the OCR integrity of all of the cluster nodes that are configured as part of your cluster by running the following CVU command:

    $ cluvfy comp ocr -n all -verbose
    

See Also:

Appendix A, "Cluster Verification Utility Reference" for more information about enabling and using CVU

Restoring the Oracle Cluster Registry on Windows Systems

If you are storing the OCR on an Oracle ASM disk group, and that disk group is corrupt, then you must restore the Oracle ASM disk group using Oracle ASM utilities, and then mount the disk group again before recovering the OCR. Recover the OCR by running the command ocrconfig -restore.

See Also:

Oracle Automatic Storage Management Administrator's Guide for information about how to restore Oracle ASM disk groups

Use the following procedure to restore OCR on Windows systems:

  1. List the nodes in your cluster by running the following command on one node:

    C:\>olsnodes
    
  2. Stop Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

    C:\>crsctl stop crs
    

    If the preceding command returns any error due to OCR corruption, stop Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

    C:\>crsctl stop crs -f
    
  3. Start the Oracle Clusterware stack on one node in exclusive mode by running the following command as a member of the Administrators group:

    C:\>crsctl start crs -excl -nocrs
    

    Ignore any errors that display.

  4. Restore OCR with the OCR backup file that you identified in "Listing Backup Files" by running the following command as a member of the Administrators group:

    C:\>ocrconfig -restore file_name
    

    Make sure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

    Notes:

    • If the original OCR location does not exist, then you must create an empty (0 byte) OCR location before you run the ocrconfig -restore command.

    • Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

    • Ensure that the Oracle ASM disk group you specify exists and is mounted.

    See Also:

  5. Verify the integrity of OCR:

    C:\>ocrcheck
    
  6. Stop Oracle Clusterware on the node where it is running in exclusive mode:

    C:\>crsctl stop crs -f
    
  7. Begin to start Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

    C:\>crsctl start crs
    
  8. Run the following Cluster Verification Utility (CVU) command to verify the OCR integrity of all of the nodes in your cluster database:

    C:\>cluvfy comp ocr -n all -verbose
    

    See Also:

    Appendix A, "Cluster Verification Utility Reference" for more information about enabling and using CVU

Restoring the Oracle Cluster Registry in an Oracle Restart Environment

Notes:

  • OCR is present for backward compatibility.

  • Once an OCR location is created, it does not get updated in the Oracle Restart environment.

  • If the Oracle Restart home has been backed up, and if there is a failure, then restoring the Oracle Restart home restores OCR.

Use the following procedure to restore OCR in an Oracle Restart environment:

  1. Stop Oracle High Availability Services by running the following command as root on all of the nodes:

    # crsctl stop has [-f]
    
  2. Run the ocrcheck -config command to determine the OCR location and then create an empty (0 byte) OCR location with appropriate permissions in that location.

  3. Restore OCR by running the following command as root:

    # crsctl pin css -n host_name
    

    Notes:

    Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

    See Also:

    Oracle Grid Infrastructure Installation Guide for information about creating OCRs
  4. Verify the integrity of OCR:

    # ocrcheck
    
  5. Start Oracle High Availability Services by running the following command as root on all of the nodes:

    # crsctl start has
    

Diagnosing Oracle Cluster Registry Problems

You can use the OCRDUMP and OCRCHECK utilities to diagnose OCR problems as described under the following topics:

Using the OCRDUMP Utility

Use the OCRDUMP utility to write the OCR contents to a file so that you can examine the OCR content.

See Also:

"OCRDUMP Utility Syntax and Options" for more information about the OCRDUMP utility
Using the OCRCHECK Utility

Use the OCRCHECK utility to verify the OCR integrity.

See Also:

"Using the OCRCHECK Utility" for more information about the OCRCHECK utility

Administering Oracle Cluster Registry with Oracle Cluster Registry Export and Import Commands

In addition to using the automatically created OCR backup files, you should also export the OCR contents before and after making significant configuration changes, such as adding or deleting nodes from your environment, modifying Oracle Clusterware resources, and upgrading, downgrading or creating a database. Do this by using the ocrconfig -export command, which exports the OCR content to a file format.

Caution:

Note the following restrictions for restoring the OCR:
  • The file format generated by ocrconfig -restore is incompatible with the file format generated by ocrconfig -export. The ocrconfig -export and ocrconfig -import commands are compatible. The ocrconfig -manualbackup and ocrconfig -restore commands are compatible. The two file formats are incompatible and must not be interchangeably used.

  • When exporting the OCR, Oracle recommends including "ocr", the cluster name, and the timestamp in the name string. For example:

    ocr_mycluster1_20090521_2130_export
    

Using the ocrconfig -export command also enables you to restore OCR using the -import option if your configuration changes cause errors. For example, if you have unresolvable configuration problems, or if you are unable to restart Oracle Clusterware after such changes, then restore your configuration using the procedure for your platform.

Notes:

Oracle recommends that you use either automatic or manual backups and the ocrconfig -restore command instead of the ocrconfig -export and ocrconfig -import commands to restore OCR for the following reasons:
  • A backup is a consistent snapshot of OCR, whereas an export is not.

  • Backups are created when the system is online. You must shut down Oracle Clusterware on all nodes in the cluster to get a consistent snapshot using the ocrconfig -export command.

  • You can inspect a backup using the OCRDUMP utility. You cannot inspect the contents of an export.

  • You can list backups with the ocrconfig -showbackup command, whereas you must keep track of all generated exports.

This section includes the following topics:

Note:

Most configuration changes that you make not only change the OCR contents, the configuration changes also cause file and database object creation. Some of these changes are often not restored when you restore OCR. Do not restore OCR as a correction to revert to previous configurations, if some of these configuration changes should fail. This may result in an OCR location that has contents that do not match the state of the rest of your system.
Importing Oracle Cluster Registry Content on Linux or UNIX Systems

 

Note:

This procedure assumes default installation of Oracle Clusterware on all nodes in the cluster, where Oracle Clusterware autostart is enabled.

Use the following procedure to import OCR on Linux or UNIX systems:

  1. List the nodes in your cluster by running the following command on one node:

    $ olsnodes
    
  2. Stop Oracle Clusterware by running the following command as root on all of the nodes:

    # crsctl stop crs
    

    If the preceding command returns any error due to OCR corruption, stop Oracle Clusterware by running the following command as root on all of the nodes:

    # crsctl stop crs -f
    
  3. Start the Oracle Clusterware stack on one node in exclusive mode by running the following command as root:

    # crsctl start crs -excl
    

    Ignore any errors that display.

    Check whether crsd is running. If it is, stop it by running the following command as root:

    # crsctl stop resource ora.crsd -init
    

    Caution:

    Do not use the -init flag with any other command.
  4. Import the OCR by running the following command as root:

    # ocrconfig -import file_name
    

    If you are importing OCR to a cluster or network file system, then skip to step 7.

    Notes:

    • If the original OCR location does not exist, then you must create an empty (0 byte) OCR location before you run the ocrconfig -import command.

    • Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

    • If you configured OCR in an Oracle ASM disk group, then ensure that the Oracle ASM disk group exists and is mounted.

    See Also:

  5. Verify the integrity of OCR:

    # ocrcheck
    
  6. Stop Oracle Clusterware on the node where it is running in exclusive mode:

    # crsctl stop crs -f
    
  7. Begin to start Oracle Clusterware by running the following command as root on all of the nodes:

    # crsctl start crs
    
  8. Verify the OCR integrity of all of the cluster nodes that are configured as part of your cluster by running the following CVU command:

    $ cluvfy comp ocr -n all -verbose
    

Note:

You can only import an exported OCR. To restore OCR from a backup, you must instead use the -restore option, as described in "Backing Up Oracle Cluster Registry".

See Also:

Appendix A, "Cluster Verification Utility Reference" for more information about enabling and using CVU
Importing Oracle Cluster Registry Content on Windows Systems

Note:

This procedure assumes default installation of Oracle Clusterware on all nodes in the cluster, where Oracle Clusterware autostart is enabled for Oracle Clusterware 10g or 11g release 1 (11.1).

Use the following procedure to import OCR on Windows systems:

  1. List the nodes in your cluster by running the following command on one node:

    C:\>olsnodes
    
  2. Stop Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

    C:\>crsctl stop crs
    

    If the preceding command returns any error due to OCR corruption, stop Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

    C:\>crsctl stop crs -f
    
  3. Start the Oracle Clusterware stack on one node in exclusive mode by running the following command as a member of the Administrators group:

    C:\>crsctl start crs -excl
    

    Ignore any errors that display.

    Check whether crsd is running. If it is, stop it by running the following command as a member of the Administrators group:

    C:\>crsctl stop resource ora.crsd -init
    

    Caution:

    Do not use the -init flag in any other command.
  4. Import OCR by running the following command as a member of the Administrators group:

    C:\>ocrconfig -import file_name
    

    Make sure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

    Notes:

    • If the original OCR location does not exist, then you must create an empty (0 byte) OCR location before you run the ocrconfig -import command.

    • Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

    • Ensure that the Oracle ASM disk group you specify exists and is mounted.

    See Also:

  5. Verify the integrity of OCR:

    C:\>ocrcheck
    
  6. Stop Oracle Clusterware on the node where it is running in exclusive mode:

    C:\>crsctl stop crs -f
    
  7. Begin to start Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

    C:\>crsctl start crs
    
  8. Run the following Cluster Verification Utility (CVU) command to verify the OCR integrity of all of the nodes in your cluster database:

    C:\>cluvfy comp ocr -n all -verbose
    

    See Also:

    Appendix A, "Cluster Verification Utility Reference" for more information about enabling and using CVU
  1. Stop the Oracle Clusterware stack on all nodes by running the following command as a member of the Administrators group on one node:

    C:\>crsctl stop cluster -all
    
  2. Stop Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

    C:\>crsctl stop crs
    
  3. Identify the OCR export file that you want to import by running the ocrconfig -showbackup command.

  4. Stop the OracleClusterVolumeService and OracleOHService OCR clients on each node in your Oracle Clusterware environment using the Service Control Panel.

  5. Import an OCR location by running the following command as a member of the Administrators group, where file_name is the name of the OCR location from which you want to import OCR information:

    C:\>ocrconfig -import file_name
    
  6. Begin to start Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

    C:\>crsctl start crs
    

    On each node where Oracle Clusterware autostart is enabled, run the crsctl check crs command on the node until it shows that the Oracle Clusterware stack is online.

  7. Run the following Cluster Verification Utility (CVU) command to verify the OCR integrity of all of the nodes in your cluster database:

    C:\>cluvfy comp ocr -n all -verbose
    

    See Also:

    Appendix A, "Cluster Verification Utility Reference" for more information about enabling and using CVU

Oracle Local Registry

In Oracle Clusterware 11g release 2 (11.2), each node in a cluster has a local registry for node-specific resources, called an Oracle Local Registry (OLR), that is installed and configured when Oracle Clusterware installs OCR. Multiple processes on each node have simultaneous read and write access to the OLR particular to the node on which they reside, regardless of whether Oracle Clusterware is running or fully functional.

By default, OLR is located at Grid_home/cdata/host_name.olr on each node.

Manage OLR using the OCRCHECK, OCRDUMP, and OCRCONFIG utilities as root with the -local option.

  • You can check the status of OLR on the local node using the OCRCHECK utility, as follows:

    # ocrcheck -local
    
    Status of Oracle Cluster Registry is as follows :
            Version                  :          3
            Total space (kbytes)     :     262132
            Used space (kbytes)      :       9200
            Available space (kbytes) :     252932
            ID                       :  604793089
            Device/File Name         : /private2/crs/cdata/localhost/dglnx6.olr
                                       Device/File integrity check succeeded
    
            Local OCR integrity check succeeded
    
  • You can display the content of OLR on the local node to the text terminal that initiated the program using the OCRDUMP utility, as follows:

    # ocrdump -local -stdout
    
  • You can perform administrative tasks on OLR on the local node using the OCRCONFIG utility.

    • To export OLR to a file:

      # ocrconfig –local –export file_name
      

      Notes:

      • Oracle recommends that you use the -manualbackup and -restore commands and not the -import and -export commands.

      • When exporting OLR, Oracle recommends including "olr", the host name, and the timestamp in the name string. For example:

        olr_myhost1_20090603_0130_export
        
    • To import a specified file to OLR:

      # ocrconfig –local –import file_name
      
    • To manually back up OLR:

      # ocrconfig –local –manualbackup
      

      Note:

      The OLR is backed up at the end of an installation or an upgrade. After that time, you can only manually back up the OLR. Automatic backups are not supported for the OLR. You should create a new backup when you migrate the OCR from Oracle ASM to other storage, or you migrate the OCR from other storage to Oracle ASM.

      The default backup location for the OLR is in the path Grid_home/cdata/host_name.

    • To view the contents of the OLR backup file:

      ocrdump -local -backupfile olr_backup_file_name
      
    • To change the OLR backup location:

      ocrconfig -local -backuploc new_olr_backup_path
      
    • To restore OLR:

      # crsctl stop crs
      # ocrconfig -local -restore file_name
      # ocrcheck -local
      # crsctl start crs
      $ cluvfy comp olr
      

Upgrading and Downgrading the Oracle Cluster Registry Configuration

When you upgrade Oracle Clusterware, it automatically runs the ocrconfig -upgrade command. To downgrade, follow the downgrade instructions for each component and also downgrade OCR using the ocrconfig -downgrade command. If you are upgrading OCR, then you can use the OCRCHECK utility to verify the integrity of OCR.

Troubleshooting Oracle Cluster Registry and Diagnostic Output

This section describes various methods for troubleshooting problems with OCR, and obtaining diagnostic information from the utilities used to manage the OCR. You can use these utilities to troubleshoot OLR.

This section contains the following topics:

Troubleshooting Oracle Cluster Registry

Table 2-5 describes common OCR problems with corresponding resolution suggestions.

Table 2-5 Common Oracle Cluster Registry Problems and Solutions

Problem Solution

Not currently using OCR mirroring and would like to enable it.

Run the ocrconfig command with the -replace option.

OCR failed and you must replace it. Error messages in Oracle Enterprise Manager or OCR log file.

Run the ocrconfig command with the -replace option.

OCR has a misconfiguration.

Run the ocrconfig command with the -repair option as described.

You are experiencing a severe performance effect from OCR processing or you want to remove OCR for other reasons.

Run the ocrconfig command with the -replace option as described.

OCR has failed and before you can fix it, the node must be rebooted with only one OCR.

Run the ocrconfig with the -repair option to remove the bad OCR location. Oracle Clusterware cannot start if it cannot find all OCRs defined.


Using the OCRCHECK Utility

The OCRCHECK utility displays the version of the OCR's block format, total space available and used space, OCRID, and the OCR locations that you have configured. OCRCHECK performs a block-by-block checksum operation for all of the blocks in all of the OCRs that you have configured. It also returns an individual status for each file and a result for the overall OCR integrity check.

You can run the ocrcheck -help command to display usage information about this utility.

The following example shows a sample of the OCRCHECK utility output:

# ocrcheck

Status of Oracle Cluster Registry is as follows :
        Version                  :          3
        Total space (kbytes)     :     262120
        Used space (kbytes)      :        752
        Available space (kbytes) :     261368
        ID                       : 2098980155
        Device/File Name         : +ocrdg1
                                   Device/File integrity check succeeded
        Device/File Name         : +ocrdg2
                                   Device/File integrity check succeeded
                                   Device/File not configured
                                   Device/File not configured
                                   Device/File not configured
        Cluster registry integrity check succeeded 
        Logical corruption check succeeded

Note:

The logical corruption check is only performed if you run the ocrcheck command as root.

To display only configured OCRs:

$ ocrcheck -config

Oracle Cluster Registry configuration is :
        Device/File Name         : Grid_home/oracle/has_work/data.ocr
        Device/File Name         : Grid_home/oracle/has_work/mirror.ocr

Run the ocrcheck -local -config command to obtain OLR information.

$ ocrcheck -local -config

Oracle Local Registry configuration is :
        Device/File Name         : Grid_home/oracle/has_work/data.olr.stact23

OCRCHECK creates a log file in the Grid_home/log/host_name/client directory. To change the log level, edit the Grid_home/srvm/admin/ocrlog.ini file.

Using the OCRDUMP Utility to View Oracle Cluster Registry Content

This section explains how to use the OCRDUMP utility to view OCR and Oracle Local Registry (OLR) content for troubleshooting. The OCRDUMP utility enables you to view the OCR and OLR contents by writing the content to a file or stdout in a readable format.

You can use several options for OCRDUMP. For example, you can limit the output to a key and its descendents. You can also write the contents to an XML file that you can view using a browser. OCRDUMP writes the OCR keys as ASCII strings and values in a data type format. OCRDUMP retrieves header information based on a best effort basis.

OCRDUMP also creates a log file in Grid_home/log/host_name/client. To change the log level, edit the Grid_home/srvm/admin/ocrlog.ini file.

To change the logging component, edit the entry containing the comploglvl= entry. For example, to change the log level of the OCRAPI component to 3 and to change the log level of the OCRRAW component to 5, make the following entry in the ocrlog.ini file:

comploglvl="OCRAPI:3;OCRRAW:5"

Note:

Make sure that you have file creation privileges in the Grid_home directory before using the OCRDUMP utility.

This section includes the following topics:

OCRDUMP Utility Syntax and Options

This section describes the OCRDUMP utility command syntax and usage. Run the ocrdump command with the following syntax where file_name is the name of a target file to which you want Oracle Database to write the Oracle Cluster Registry output and where key_name is the name of a key from which you want Oracle Database to write Oracle Cluster Registry subtree content:

$ ocrdump [file_name | -stdout] [-local] [-backupfile backup_file_name
[-keyname key_name] [-xml] [-noheader]
]

Table 2-6 describes the OCRDUMP utility options and option descriptions.

Table 2-6 OCRDUMP Options and Option Descriptions

Options Description

file_name

The name of a file to which you want OCRDUMP to write output.

By default, OCRDUMP writes output to a predefined output file named OCRDUMPFILE. The file_name option redirects OCRDUMP output to a file that you specify.

-stdout

Use this option to redirect the OCRDUMP output to the text terminal that initiated the program.

If you do not redirect the output, OCRDUMP writes output to a predefined output file named OCRDUMPFILE.

-local

Use this option to dump the contents of OLR.

-backupfile

Use this option to view the contents of an OCR backup file. Use the -local option with this option to view the contents of an OLR backup file.

backup_file_name

The name of the backup file with the content you want to view. You can query the backups using the ocrconfig -showbackup command.

-keyname key_name

The name of an Oracle Cluster Registry key whose subtree is to be dumped.

-xml

Use this option to write the output in XML format.

-noheader

Does not print the time at which you ran the command and when the Oracle Cluster Registry configuration occurred.


OCRDUMP Utility Examples

The following ocrdump utility examples extract various types of OCR information and write it to various targets:

ocrdump

Writes the OCR content to a file called OCRDUMPFILE in the current directory.

ocrdump MYFILE

Writes the OCR content to a file called MYFILE in the current directory.

ocrdump -stdout -keyname SYSTEM

Displays the OCR content from the subtree of the key SYSTEM in the terminal window.

ocrdump -stdout -xml

Displays the OCR content in the terminal window in XML format.

ocrdump -stdout -backupfile $ORA_CRS_HOME/cdata/cluster_name/file_name

Writes the entire OCR and OLR content to a backup file located in the $ORA_CRS_HOME/cdata/cluster_name directory. You must run this command as root to be able to view all of the keys. Be sure to name the file appropriately so that it can be recognized by anyone as an OCR backup file, such as BACKUPOO.ocr.

Sample OCRDUMP Utility Output

The following OCRDUMP examples show the KEYNAME, VALUE TYPE, VALUE, permission set (user, group, world) and access rights for two sample runs of the ocrdump command. The following shows the output for the SYSTEM.language key that has a text value of AMERICAN_AMERICA.WE8ASCII37.

[SYSTEM.language]
ORATEXT : AMERICAN_AMERICA.WE8ASCII37
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ,
OTHER_PERMISSION : PROCR_READ, USER_NAME : user, GROUP_NAME : group}

The following shows the output for the SYSTEM.version key that has integer value of 3:

[SYSTEM.version]
UB4 (10) : 3
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ,
OTHER_PERMISSION : PROCR_READ, USER_NAME : user, GROUP_NAME : group}

Diagnostic Output for OCRCONFIG

The OCRCONFIG utility creates a log file in Grid_home/log/host_name/client.

To change the amount of logging, edit the path in the Grid_home/srvm/admin/ocrlog.ini file.

Configuring Oracle Grid Infrastructure

After performing a software-only installation of the Oracle Grid Infrastructure, you can configure the software using Configuration Wizard. This wizard assists you with editing the crsconfig_params configuration file. Similar to the Oracle Grid Infrastructure installer, the Configuration Wizard performs various validations of the Grid home and inputs before and after you run through the wizard.

Using the Configuration Wizard, you can configure a new Grid Infrastructure on one or more nodes, or configure an upgraded Grid Infrastructure. You can also run the Configuration Wizard in silent mode.

Notes:

  • Before running the Configuration Wizard, ensure that the Grid Infrastructure home is current, with all necessary patches applied.

  • To launch the Configuration Wizard in the following procedures:

    On Linux and UNIX, run the following command:

    Oracle_home/crs/config/config.sh
    

    On Windows, run the following command:

    Oracle_home\crs\config\config.bat
    

This section includes the following topics:

Configuring a Single Node

To use the Configuration Wizard to configure a single node:

  1. Start the Configuration Wizard, as follows:

    $ Oracle_home/crs/config/config.sh
    
  2. On the Select Installation Option page, select Configure Grid Infrastructure for a Cluster.

  3. On the Cluster Node Information page, select only the local node and corresponding VIP name.

  4. Continue adding your information on the remaining wizard pages.

  5. Review your inputs on the Summary page and click Finish.

  6. Run the root.sh script as instructed by the Configuration Wizard.

Configuring Multiple Nodes

To use the Configuration Wizard to configure multiple nodes:

  1. Start the Configuration Wizard, as follows:

    $ Oracle_home/crs/config/config.sh
    
  2. On the Select Installation Option page, select Configure Grid Infrastructure for a Cluster.

  3. On the Cluster Node Information page, select the nodes you want to configure and their corresponding VIP names. The Configuration Wizard validates the nodes you select to ensure that they are ready.

  4. Continue adding your information on the remaining wizard pages.

  5. Review your inputs on the Summary page and click Finish.

  6. Run the root.sh script as instructed by the Configuration Wizard.

Upgrading Grid Infrastructure

To use the Configuration Wizard to upgrade the Grid Infrastructure:

  1. Start the Configuration Wizard, as follows:

    $ Oracle_home/crs/config/config.sh
    
  2. On the Select Installation Option page, select Upgrade Grid Infrastructure.

  3. On the Grid Infrastructure Node Selection page, select the nodes you want to upgrade.

  4. Continue adding your information on the remaining wizard pages.

  5. Review your inputs on the Summary page and click Finish.

  6. Run the rootupgrade.sh script as instructed by the Configuration Wizard.

Running the Configuration Wizard in Silent Mode

To use the Configuration Wizard in silent mode to configure or upgrade nodes, start the Configuration Wizard from the command line with -silent -responsefile filename. The wizard validates the response file and proceeds with the configuration. If any of the inputs in the response file are found to be invalid, then the Configuration Wizard displays an error and exits. Run the root scripts and configToolAllCommands scripts as prompted.

Configuring IPMI for Failure Isolation

This section contains the following topics:

About Using IPMI for Failure Isolation

Failure isolation is a process by which a failed node is isolated from the rest of the cluster to prevent the failed node from corrupting data. The ideal fencing involves an external mechanism capable of restarting a problem node without cooperation either from Oracle Clusterware or from the operating system running on that node. To provide this capability, Oracle Clusterware 11g release 2 (11.2) supports the Intelligent Management Platform Interface specification (IPMI) (also known as Baseboard Management Controller (BMC)), an industry-standard management protocol.

Typically, you configure failure isolation using IPMI during Grid Infrastructure installation, when you are provided with the option of configuring IPMI from the Failure Isolation Support screen. If you do not configure IPMI during installation, then you can configure it after installation using the Oracle Clusterware Control utility (CRSCTL), as described in "Postinstallation Configuration of IPMI-based Failure Isolation Using CRSCTL".

To use IPMI for failure isolation, each cluster member node must be equipped with an IPMI device running firmware compatible with IPMI version 1.5, which supports IPMI over a local area network (LAN). During database operation, failure isolation is accomplished by communication from the evicting Cluster Synchronization Services daemon to the failed node's IPMI device over the LAN. The IPMI-over-LAN protocol is carried over an authenticated session protected by a user name and password, which are obtained from the administrator during installation.

In order to support dynamic IP address assignment for IPMI using DHCP, the Cluster Synchronization Services daemon requires direct communication with the local IPMI device during Cluster Synchronization Services startup to obtain the IP address of the IPMI device. (This is not true for HP-UX and Solaris platforms, however, which require that the IPMI device be assigned a static IP address.) This is accomplished using an IPMI probe command (OSD), which communicates with the IPMI device through an IPMI driver, which you must install on each cluster system.

If you assign a static IP address to the IPMI device, then the IPMI driver is not strictly required by the Cluster Synchronization Services daemon. The driver is required, however, to use ipmitool or ipmiutil to configure the IPMI device but you can also do this with management consoles on some platforms.

Configuring Server Hardware for IPMI

See Also:

Oracle Grid Infrastructure Installation Guide for your platform for information about configuring server hardware for IPMI

Postinstallation Configuration of IPMI-based Failure Isolation Using CRSCTL

This section contains the following topics:

IPMI Postinstallation Configuration with Oracle Clusterware

When you install IPMI during Oracle Clusterware installation, you configure failure isolation in two phases. Before you start the installation, you install and enable the IPMI driver in the server operating system, and configure the IPMI hardware on each node (IP address mode, admin credentials, and so on), as described in Oracle Grid Infrastructure Installation Guide. When you install Oracle Clusterware, the installer collects the IPMI administrator user ID and password, and stores them in an Oracle Wallet in node-local storage, in the Oracle Local Registry.

With postinstallation configuration, install and enable the IPMI driver, and configure the IPMI device, as described in Oracle Grid Infrastructure Installation Guide.

After you complete the server configuration, complete the following procedure on each cluster node to register IPMI administrators and passwords on the nodes.

Note:

If IPMI is configured to obtain its IP address using DHCP, it may be necessary to reset IPMI or restart the node to cause it to obtain an address.
  1. Start Oracle Clusterware, which allows it to obtain the current IP address from IPMI. This confirms the ability of the clusterware to communicate with IPMI, which is necessary at startup.

    If Oracle Clusterware was running before IPMI was configured, you can shut Oracle Clusterware down and restart it. Alternatively, you can use the IPMI management utility to obtain the IPMI IP address and then use CRSCTL to store the IP address in the Oracle Local Registry by running a command similar to the following:

    crsctl set css ipmiaddr 192.168.10.45
    
  2. Use CRSCTL to store the previously established user ID and password for the resident IPMI in the Oracle Local Registry by running the crsctl set css ipmiadmin command, and supplying the administrator and password at the prompt. For example:

    crsctl set css ipmiadmin administrator_name
    IPMI BMC password: password
    

    This command validates the supplied credentials and fails if another cluster node cannot access the local IPMI using them.

    After you complete hardware and operating system configuration, and register the IPMI administrator on Oracle Clusterware, IPMI-based failure isolation should be fully functional.

Modifying IPMI Configuration Using CRSCTL

To modify an existing IPMI-based failure isolation configuration (for example to change IPMI passwords, or to configure IPMI for failure isolation in an existing installation), use CRSCTL with the IPMI configuration tool appropriate to your platform. For example, to change the administrator password for IPMI, you must first modify the IMPY configuration as described in Oracle Grid Infrastructure Installation Guide, and then use CRSCTL to change the password in the Oracle Local Registry.

The configuration data needed by Oracle Clusterware for IPMI is kept in an Oracle Wallet in the Oracle Registry. Because the configuration information is kept in a secure store, it must be written by the Oracle Clusterware installation owner account (the Grid user), so you must log in as that installation user.

Use the following procedure to modify an existing IPMI configuration:

  1. Enter the crsctl set css ipmiadmin administrator_name command. For example, with the user IPMIadm:

    crsctl set css ipmiadmin IPMIadm
    

    Provide the administrator password. Oracle Clusterware stores the administrator name and password for the local IPMI in the Oracle Local Registry.

    After storing the new credentials, Oracle Clusterware can retrieve the new credentials and distribute them as required.

  2. Enter the crsctl set css ipmiaddr bmc_ip_address command. For example:

    crsctl set css ipmiaddr 192.0.2.244
    

    This command stores the new IPMI IP address of the local IPMI in the Oracle Local Registry, After storing the IP address, Oracle Clusterware can retrieve the new configuration and distribute it as required.

  3. Enter the crsctl get css ipmiaddr command. For example:

    crsctl get css ipmiaddr
    

    This command retrieves the IP address for the local IPMI from the Oracle Local Registry and displays it on the console.

  4. Remove the IPMI configuration information for the local IPMI from the Oracle Local Registry and delete the registry entry, as follows:

    crsctl unset css ipmiconfig
    

See Also:

"Oracle RAC Environment CRSCTL Commands" for descriptions of these CRSCTL commands

Removing IPMI Configuration Using CRSCTL

You can remove an IPMI configuration from a cluster using CRSCTL if you want to stop using IPMI completely or if IPMI was initially configured by someone other than the user that installed Oracle Clusterware. If the latter is true, then Oracle Clusterware cannot access the IPMI configuration data and IPMI is not usable by the Oracle Clusterware software, and you must reconfigure IPMI as the user that installed Oracle Clusterware.

To completely remove IPMI, perform the following steps. To reconfigure IPMI as the user that installed Oracle Clusterware, perform steps 3 and 4 repeat steps 2 and 3 in "Modifying IPMI Configuration Using CRSCTL".

  1. Disable the IPMI driver and eliminate the boot-time installation, as follows:

    /sbin/modprobe –r
    

    See Also:

    Oracle Grid Infrastructure Installation Guide for your platform for more information about the IPMI driver
  2. Disable IPMI-over-LAN for the local IPMI using either ipmitool or ipmiutil, to prevent access over the LAN or change the IPMI administrator user ID and password.

  3. Ensure that Oracle Clusterware is running and then use CRSCTL to remove the IPMI configuration data from the Oracle Local Registry by running the following command:

    crsctl unset css ipmiconfig
    
  4. Restart Oracle Clusterware so that it runs without the IPMI configuration by running the following commands as root:

    # crsctl stop crs
    # crsctl start crs
    

Cluster Time Management

The Cluster Time Synchronization Service (CTSS) is installed as part of Oracle Clusterware and runs in observer mode if it detects a time synchronization service or a time synchronization service configuration, valid or broken, on the system.If CTSS detects that there is no time synchronization service or time synchronization service configuration on any node in the cluster, then CTSS goes into active mode and takes over time management for the cluster.

When nodes join the cluster, if CTSS is in active mode, then it compares the time on those nodes to a reference clock located on one node in the cluster. If there is a discrepancy between the two times and the discrepancy is within a certain stepping limit, then CTSS performs step time synchronization, which is to step the time of the nodes joining the cluster to synchronize them with the reference.

When Oracle Clusterware starts, if CTSS is running in active mode and the time discrepancy is outside the stepping limit (the limit is 24 hours), then CTSS generates an alert in the alert log, exits, and Oracle Clusterware startup fails. You must manually adjust the time of the nodes joining the cluster to synchronize with the cluster, after which Oracle Clusterware can start and CTSS can manage the time for the nodes.

Clocks on the nodes in the cluster become desynchronized with the reference clock (a time CTSS uses as a basis and is on the first node started in the cluster) periodically for various reasons. When this happens, CTSS performs slew time synchronization, which is to speed up or slow down the system time on the nodes until they are synchronized with the reference system time. In this time synchronization method, CTSS does not adjust time backward, which guarantees monotonic increase of the system time.

When performing slew time synchronization, CTSS never runs time backward to synchronize with the reference clock. CTSS periodically writes alerts to the alert log containing information about how often it adjusts time on nodes to keep them synchronized with the reference clock.

To activate CTSS in your cluster, you must stop and deconfigure the vendor time synchronization service on all nodes in the cluster. CTSS detects when this happens and assumes time management for the cluster.

For example, to deconfigure NTP, you must remove or rename the ntp.conf file.

Similarly, if you want to deactivate CTSS in your cluster, then configure and start the vendor time synchronization service on all nodes in the cluster. CTSS detects this change and reverts back to observer mode. Use the crsctl check ctss command to ensure that CTSS is operating in observer mode. You can also use the cluvfy comp clocksync -all command to verify that the time synchronization service is operating.

See Also:

Oracle Grid Infrastructure Installation Guide for your platform for information about configuring NTP for Oracle Clusterware, or disabling it to use CTSS

Changing Network Addresses on Manually Configured Networks

This section contains the following topics:

Understanding When You Must Configure Network Addresses

An Oracle Clusterware configuration requires at least two interfaces:

  • A public network interface, on which users and application servers connect to access data on the database server

  • A private network interface for internode communication.

If you use Grid Naming Service and DHCP to manage your network connections, then you may not need to configure address information on the cluster. Using GNS allows public Virtual Internet Protocol (VIP) addresses to be dynamic, DHCP-provided addresses. Clients submit name resolution requests to your network's Domain Name Service (DNS), which forwards the requests to the grid naming service (GNS), managed within the cluster. GNS then resolves these requests to nodes in the cluster.

If you do not use GNS, and instead configure networks manually, then public VIP addresses must be statically configured in the DNS, VIPs must be statically configured in the DNS and hosts file, and private IP addresses require static configuration.

Understanding SCAN Addresses and Client Service Connections

Public network addresses are used to provide services to clients. If your clients are connecting to the Single Client Access Name addresses, then you may need to change public and virtual IP addresses as you add or remove nodes from the cluster, but you do not need to update clients with new cluster addresses.

SCANs function like a cluster alias. However, SCANs are resolved on any node in the cluster, so unlike a VIP address for a node, clients connecting to the SCAN no longer require updated VIP addresses as nodes are added to or removed from the cluster. Because the SCAN addresses resolve to the cluster, rather than to a node address in the cluster, nodes can be added to or removed from the cluster without affecting the SCAN address configuration.

The SCAN is a fully qualified name (host name+domain) that is configured to resolve to all the addresses allocated for the SCAN. The addresses resolve using Round Robin DNS either on the DNS server, or within the cluster in a GNS configuration. SCAN listeners can run on any node in the cluster. SCANs provide location independence for the databases, so that client configuration does not have to depend on which nodes run a particular database.

Oracle Database 11g release 2 (11.2) and later instances only register with SCAN listeners as remote listeners. Upgraded databases register with SCAN listeners as remote listeners, and also continue to register with all node listeners.

Changing the Virtual IP Addresses

Clients configured to use public VIP addresses for Oracle Database releases before Oracle Database 11g release 2 (11.2) can continue to use their existing connection addresses. Oracle recommends that you configure clients to use SCANs, but it is not required that you use SCANs. When an earlier version of Oracle Database is upgraded, it is registered with the SCAN, and clients can start using the SCAN to connect to that database, or continue to use VIP addresses for connections.

If you continue to use VIP addresses for client connections, you can modify the VIP address while Oracle Database and Oracle ASM continue to run. However, you must stop services while you modify the address. When you restart the VIP address, services are also restarted on the node.

This procedure cannot be used to change a static public subnet to use DHCP. Only the srvctl add network -S command creates a DHCP network.

Note:

The following instructions describe how to change only a VIP address, and assume that the host name associated with the VIP address does not change. Note that you do not need to update VIP addresses manually if you are using GNS, and VIPs are assigned using DHCP.

If you are changing only the VIP address, then update the DNS and the client hosts files. Also, update the server hosts files, if those are used for VIP addresses.

Perform the following steps to change a VIP address:

  1. Stop all services running on the node whose VIP address you want to change using the following command syntax, where database_name is the name of the database, service_name_list is a list of the services you want to stop, and my_node is the name of the node whose VIP address you want to change:

    srvctl stop service -d database_name  -s service_name_list -n my_node
    

    This example specifies the database name (grid) using the -d option and specifies the services (sales,oltp) on the appropriate node (mynode).

    $ srvctl stop service -d grid -s sales,oltp -n mynode
    
  2. Confirm the current IP address for the VIP address by running the srvctl config vip command. This command displays the current VIP address bound to one of the network interfaces. The following example displays the configured VIP address:

    $ srvctl config vip -n stbdp03
    VIP exists.:stbdp03
    VIP exists.: /stbdp03-vip/192.168.2.20/255.255.255.0/eth0
    
  3. Stop the VIP address using the srvctl stop vip command:

    $ srvctl stop vip -n mynode
    
  4. Verify that the VIP address is no longer running by running the ifconfig -a command on Linux and UNIX systems (or issue the ipconfig /all command on Windows systems), and confirm that the interface (in the example it was eth0:1) is no longer listed in the output.

  5. Make any changes necessary to the /etc/hosts files on all nodes on Linux and UNIX systems, or the %windir%\system32\drivers\etc\hosts file on Windows systems, and make any necessary DNS changes to associate the new IP address with the old host name.

  6. To use a different subnet or NIC for the default network before you change any VIP resource, you must use the srvctl modify network -S subnet/netmask/interface command as root to change the network resource, where subnet is the new subnet address, netmask is the new netmask, and interface is the new interface. After you change the subnet, then you must change each node's VIP to an IP address on the new subnet, as described in the next step.

  7. Modify the node applications and provide the new VIP address using the following srvctl modify nodeapps syntax:

    $ srvctl modify nodeapps -n node_name -A new_vip_address
    

    The command includes the following flags and values:

    • -n node_name is the node name

    • -A new_vip_address is the node-level VIP address: name|ip/netmask/[if1[|if2|...]]

      For example, issue the following command as the root user:

      srvctl modify nodeapps -n mynode -A 192.168.2.125/255.255.255.0/eth0
      

      Attempting to issue this command as the installation owner account may result in an error. For example, if the installation owner is oracle, then you may see the error PRCN-2018: Current user oracle is not a privileged user.To avoid the error, run the command as the root or system administrator account.

  8. Start the node VIP by running the srvctl start vip command:

    $ srvctl start vip -n node_name
    

    The following command example starts the VIP on the node named mynode:

    $ srvctl start vip -n mynode
    
  9. Repeat the steps for each node in the cluster.

    Because the SRVCTL utility is a clusterwide management tool, you can accomplish these tasks for any specific node from any node in the cluster, without logging in to each of the cluster nodes.

  10. Run the following command to verify node connectivity between all of the nodes for which your cluster is configured. This command discovers all of the network interfaces available on the cluster nodes and verifies the connectivity between all of the nodes by way of the discovered interfaces. This command also lists all of the interfaces available on the nodes which are suitable for use as VIP addresses.

    $ cluvfy comp nodecon -n all -verbose
    

Changing Oracle Clusterware Private Network Configuration

This section contains the following topics:

About Private Networks and Network Interfaces

Oracle Clusterware requires that each node is connected through a private network (in addition to the public network). The private network connection is referred to as the cluster interconnect.

Oracle only supports clusters in which all of the nodes use the same network interface connected to the same subnet (defined as a global interface with the oifcfg command). You cannot use different network interfaces for each node (node-specific interfaces). Refer to Appendix D, "Oracle Interface Configuration Tool (OIFCFG) Command Reference" for more information about global and node-specific interfaces. Table 2-7 describes how the network interface card (NIC) and the private IP address are stored.

Table 2-7 Storage for the Network Interface, Private IP Address, and Private Host Name

Entity Stored In... Comments

Network interface name

Operating system

For example: eth1

You can use wildcards when specifying network interface names.

For example: eth*

Private network Interfaces

Oracle Clusterware, in the Grid Plug and Play (GPnP) Profile

Configure an interface for use as a private interface during installation by marking the interface as Private, or use the oifcfg cluster_interconnects command to designate an interface as a private interface.


Redundant Interconnect Usage

You can define multiple interfaces for Redundant Interconnect Usage by classifying the interfaces as private either during installation or after installation using the oifcfg setif command. When you do, Oracle Clusterware creates from one to four (depending on the number of interfaces you define) highly available IP (HAIP) addresses, which Oracle Database and Oracle ASM instances use to ensure highly available and load balanced communications.

The Oracle software (including Oracle RAC, Oracle ASM, and Oracle ACFS, all 11g release 2 (11.2.0.2), or later), by default, uses these HAIP addresses for all of its traffic, allowing for load balancing across the provided set of cluster interconnect interfaces. If one of the defined cluster interconnect interfaces fails or becomes non-communicative, then Oracle Clusterware transparently moves the corresponding HAIP address to one of the remaining functional interfaces.

Note:

Oracle Clusterware uses at most four interfaces at any given point, regardless of the number of interfaces defined. If one of the interfaces fails, then the HAIP address moves to another one of the configured interfaces in the defined set.

When there is only a single HAIP address and multiple interfaces from which to select, the interface to which the HAIP address moves is no longer the original interface upon which it was configured. Oracle Clusterware selects the interface with the lowest numerical subnet to which to add the HAIP address.

See Also:

Oracle Grid Infrastructure Installation Guide for your platform for information about defining interfaces

Consequences of Changing Interface Names Using OIFCFG

The consequences of changing interface names depend on which name you are changing, and whether you are also changing the IP address. In cases where you are only changing the interface names, the consequences are minor. If you change the name for the public interface that is stored in OCR, then you also must modify the node applications for the cluster. Therefore, you must stop the node applications for this change to take effect.

See Also:

My Oracle Support (formerly OracleMetaLink) note 276434.1 for more details about changing the nodeapps to use a new public interface name, available at the following URL:
https://metalink.oracle.com

Changing or Deleting a Network Interface From a Cluster Configuration File

Use the following procedure to change or delete a network interface.

Caution:

The interface used by the Oracle RAC (RDBMS) interconnect must be the same interface that Oracle Clusterware is using with the hostname. Do not configure the private interconnect for Oracle RAC on a separate interface that is not monitored by Oracle Clusterware.
  1. Ensure that Oracle Clusterware is running on all of the cluster nodes by running the following command:

    olsnodes -s
    

    The command returns output similar to the following, showing that Oracle Clusterware is running on all of the nodes in the cluster:

    ./olsnodes -s
    myclustera Active
    myclusterc Active
    myclusterb Active
    
  2. Ensure that the replacement interface is configured and operational in the operating system on all of the nodes. To do this, use the ifconfig command for your platform. For example, on Linux, use:

    /sbin/ifconfig..
    
  3. Add the new interface to the cluster as follows, providing the name of the new interface and the subnet address, using the following commands:

    $ oifcfg setif -global
    /interface_name/subnet/:cluster_interconnect
    

    Note:

    You can use wildcards in the preceding command example.
  4. Ensure that the previous step completes. Then you can remove the former subnet as follows by providing the name and subnet address of the former interface:

    $ oifcfg delif -global /interface_name/subnet/
    
  5. Verify the current configuration using the following command:

    oifcfg getif
    

    For example:

    $ oifcfg getif
    eth2 10.220.52.0 global cluster_interconnect
    eth0 10.220.16.0 global public
    
  6. On all of the nodes, stop Oracle Clusterware by running the following command as the root user:

    # crsctl stop crs
    
  7. When the cluster stops, you can deconfigure the deleted network interface in the operating system using the following command:

    $ ifconfig down
    

    At this point, the IP address from network interfaces for the old subnet is deconfigured from Oracle Clusterware. This command does not affect the configuration of the IP address on the operating system.

  8. Restart Oracle Clusterware by running the following command on each cluster member node as the root user:

    # crsctl start crs
    

    The changes take effect when Oracle Clusterware restarts.

    If you use the CLUSTER_INTERCONNECTS initialization parameter, then you must update it to reflect the changes.

Changing the Network Interface for the Interconnect

To change the network interface for the private interconnect (for example, eth1), you must perform the change on all nodes (globally).

This procedure changes the network interface and IP address on each node in the cluster that Oracle Clusterware and Oracle Database were previously using.

To change the network interface, perform the following steps:

  1. Ensure that the Oracle Clusterware stack is up and running on all cluster nodes.

  2. Use operating system commands (ifconfig, or the command for your system) to ensure that the new or replacement interface is configured and up on all cluster member nodes.

  3. On a single node in the cluster add the new global interface specification:

    $ oifcfg setif -global interface_name/subnet:cluster_interconnect
    

    Note:

    You can use wildcards with the interface name. For example, oifcfg setif -global "eth*/192.168.0.0:cluster_interconnect is valid syntax. However, be careful to avoid ambiguity with other addresses or masks used with other cluster interfaces. If you use wildcards, then you see a warning similar to the following:
    eth* 192.168.0.0 global cluster_interconnect
    PRIF-29: Warning: wildcard in network parameters can cause mismatch
    among GPnP profile, OCR, and system
    

    Legacy network configuration does not support wildcards; thus wildcards are resolved using current node configuration at the time of the update.

    See Also:

    Appendix D, "Oracle Interface Configuration Tool (OIFCFG) Command Reference" for more information about using the OIFCFG command
  4. Verify the configuration with the oifcfg getif command.

  5. Stop Oracle Clusterware on all nodes by running the following command as root on each node:

    # crsctl stop crs
    

    Note:

    With cluster network configuration changes, the cluster must be fully stopped; do not use rolling stops and restarts.
  6. Update the network configuration settings on the operating system.

    You must update the operating system configuration changes, because changes made using ifconfig are not persistent.

    See Also:

    Your operating system documentation for more information about how to make ifconfig commands persistent
  7. Restart Oracle Clusterware by issuing the following command as root user on all nodes:

    # crsctl start crs
    

    You must restart Oracle Clusterware after running the oifcfg delif command, because Oracle Clusterware, Oracle ASM, and Oracle RAC continue to use the former subnet until they are restarted.

  8. Remove the former subnet, as follows, providing the name and subnet address of the former interface:

    oifcfg delif -global interface_name/subnet
    

    For example:

    $ oifcfg delif -global eth1/10.10.0.0
    

    Caution:

    This step should be performed only after a replacement interface is committed into the Grid Plug and Play configuration. Simple deletion of cluster interfaces without providing a valid replacement can result in invalid cluster configuration.