2 Advanced Installation Oracle Grid Infrastructure for a Cluster Preinstallation Tasks

This chapter describes the system configuration tasks that you must complete before you start Oracle Universal Installer (OUI) to install Oracle Grid Infrastructure for a cluster, and that you may need to complete if you intend to install Oracle Real Application Clusters (Oracle RAC) on the cluster.

This chapter contains the following topics:

2.1 Reviewing Upgrade Best Practices

Caution:

Always create a backup of existing databases before starting any configuration change.

If you have an existing Oracle installation, then record the version numbers, patches, and other configuration information, and review upgrade procedures for your existing installation. Review Oracle upgrade documentation before proceeding with installation, to decide how you want to proceed.

You can upgrade Oracle ASM 11g release 1 (11.1) without shutting down an Oracle RAC database by performing a rolling upgrade either of individual nodes, or of a set of nodes in the cluster. However, if you have a standalone database on a cluster that uses Oracle ASM, then you must shut down the standalone database before upgrading. If you are upgrading from Oracle ASM 10g, then you must shut down the entire Oracle ASM cluster to perform the upgrade.

If you have an existing Oracle Automatic Storage Management (Oracle ASM) installation, then review Oracle upgrade documentation. The location of the Oracle ASM home changes in this release, and you may want to consider other configuration changes to simplify or customize storage administration. If you have an existing Oracle ASM home from a previous release, then it should be owned by the same user that you plan to use to upgrade Oracle Clusterware.

During rolling upgrades of the operating system, Oracle supports using different operating system binaries when both versions of the operating system are certified with the Oracle Database release you are using.

Note:

Using mixed operating system versions is only supported for the duration of an upgrade, over the period of a few hours. Oracle Clusterware does not support nodes that have processors with different instruction set architectures (ISAs) in the same cluster. Each node must be binary compatible with the other nodes in the cluster. For example, you cannot have one node using an Intel 64 processor and another node using an IA-64 (Itanium) processor in the same cluster. You could have one node using an Intel 64 processor and another node using an AMD64 processor in the same cluster because the processors use the same x86-64 ISA and run the same binary version of Oracle software.

Your cluster can have nodes with CPUs of different speeds or sizes, but Oracle recommends that you use nodes with the same hardware configuration.

To find the most recent software updates, and to find best practices recommendations about preupgrade, postupgrade, compatibility, and interoperability, refer to "Oracle Upgrade Companion." "Oracle Upgrade Companion" is available through Note 785351.1 on My Oracle Support:

https://support.oracle.com

2.2 Installation Fixup Scripts

With Oracle Clusterware 11g release 2, Oracle Universal Installer (OUI) detects when the minimum requirements for an installation are not met, and creates shell scripts, called fixup scripts, to finish incomplete system configuration steps. If OUI detects an incomplete task, then it generates fixup scripts (runfixup.sh). You can run the fixup script after you click the Fix and Check Again Button.

You also can have CVU generate fixup scripts before installation.

See Also:

Oracle Clusterware Administration and Deployment Guide for information about using the cluvfy command

The Fixup script does the following:

  • If necessary sets kernel parameters to values required for successful installation, including:

    • Shared memory parameters.

    • Open file descriptor and UDP send/receive parameters.

  • Sets permissions on the Oracle Inventory (central inventory) directory.

  • Reconfigures primary and secondary group memberships for the installation owner, if necessary, for the Oracle Inventory directory and the operating system privileges groups.

  • Sets shell limits if necessary to required values.

If you have SSH configured between cluster member nodes for the user account that you will use for installation, then you can check your cluster configuration before installation and generate a fixup script to make operating system changes before starting the installation.

To do this, log in as the user account that will perform the installation, navigate to the staging area where the runcluvfy command is located, and use the following command syntax, where node is a comma-delimited list of nodes you want to make cluster members:

$ ./runcluvfy.sh stage -pre crsinst -n node -fixup -verbose

For example, if you intend to configure a two-node cluster with nodes node1 and node2, enter the following command:

$ ./runcluvfy.sh stage -pre crsinst -n node1,node2 -fixup -verbose

2.3 Logging In to a Remote System as root Using X Terminal

Before you install the Oracle software, you must complete several tasks as the root user on the system where you install Oracle software. To complete tasks as the root user on a remote server, you need to enable remote display as root.

Note:

If you log in as another user (for example, oracle), then you need to repeat this procedure for that user as well.

To enable remote display, complete one of the following procedures:

  • If you are installing the software from an X Window System workstation or X terminal, then:

    1. Start a local terminal session, for example, an X terminal (xterm).

    2. If you are not installing the software on the local system, then enter a command using the following syntax to enable remote hosts to display X applications on the local X server:

      $ xhost + remote_host
      

      where remote_host is the fully qualified remote hostname. For example:

      $ xhost + somehost.example.com
      somehost.example.com being added to the access control list
      
    3. If you are not installing the software on the local system, then use the ssh, command to connect to the system where you want to install the software:

      $ ssh remote_host
      

      where remote_host is the fully qualified remote hostname. For example:

      $ ssh somehost.example.com
      
    4. If you are not logged in as the root user, then enter the following command to switch the user to root:

      $ su - root
      password:
      #
      
  • If you are installing the software from a PC or other system with X server software installed, then:

    Note:

    If necessary, refer to your X server documentation for more information about completing this procedure. Depending on the X server software that you are using, you may need to complete the tasks in a different order.
    1. Start the X server software.

    2. Configure the security settings of the X server software to permit remote hosts to display X applications on the local system.

    3. Connect to the remote system where you want to install the software and start a terminal session on that system, for example, an X terminal (xterm).

    4. If you are not logged in as the root user on the remote system, then enter the following command to switch user to root:

      $ su - root
      password:
      #
      

2.4 Creating Groups, Users and Paths for Oracle Grid Infrastructure

Log in as root, and use the following instructions to locate or create groups and users required for installation.

Note:

Ensure that all group and user numbers are identical on all cluster member nodes.

Note:

During an Oracle Grid Infrastructure installation, both Oracle Clusterware and Oracle Automatic Storage Management are installed. You no longer can have separate Oracle Clusterware installation owners and Oracle Automatic Storage Management installation owners.

2.4.1 Determining If the Oracle Inventory and Oracle Inventory Group Exists

When you install Oracle software on the system for the first time, OUI creates the oraInst.loc file. This file identifies the name of the Oracle Inventory group (by default, oinstall), and the path of the Oracle Central Inventory directory. An oraInst.loc file has contents similar to the following:

inventory_loc=central_inventory_location
inst_group=group

In the preceding example, central_inventory_location is the location of the Oracle central inventory, and group is the name of the group that has permissions to write to the central inventory (the OINSTALL group privilege).

If you have an existing Oracle central inventory, then ensure that you use the same Oracle Inventory for all Oracle software installations, and ensure that all Oracle software users you intend to use for installation have permissions to write to this directory.

To determine if you have an Oracle central inventory directory (oraInventory) on your system:

Enter the following command:

# more /etc/oraInst.loc

If the oraInst.loc file exists, then the output from this command is similar to the following:

inventory_loc=/u01/app/oracle/oraInventory
inst_group=oinstall

In the previous output example:

  • The inventory_loc group shows the location of the Oracle Inventory

  • The inst_group parameter shows the name of the Oracle Inventory group (in this example, oinstall).

Use the command grep groupname /etc/group to confirm that the group specified as the Oracle Inventory group still exists on the system. For example:

$ grep oinstall /etc/group
oinstall:x:1000:grid,oracle

2.4.2 Creating the Oracle Inventory Group If an Oracle Inventory Does Not Exist

If the oraInst.loc file does not exist, then create the Oracle Inventory group by entering a command similar to the following:

# mkgroup id=1000 adms='root' oinstall

The preceding command creates the oraInventory group oinstall, with the group ID number 1000. Members of the oraInventory group are granted privileges to write to the Oracle central inventory (oraInventory).

By default, if an oraInventory group does not exist, then the installer lists the primary group of the installation owner for the Oracle Grid Infrastructure for a Cluster software as the oraInventory group. Ensure that this group is available as a primary group for all planned Oracle software installation owners.

Note:

Group and user IDs must be identical on all nodes in the cluster. Check to make sure that the group and user IDs you want to use are available on each cluster member node, and confirm that the primary group for each grid infrastructure for a cluster installation owner has the same name and group ID.

2.4.3 Creating the Oracle Grid Infrastructure User

You must create a software owner for Oracle Grid Infrastructure in the following circumstances:

  • If an Oracle software owner user does not exist; for example, if this is the first installation of Oracle software on the system

  • If an Oracle software owner user exists, but you want to use a different operating system user, with different group membership, to separate grid infrastructure administrative privileges from Oracle Database administrative privileges.

    In Oracle documentation, a user created to own only Oracle Grid Infrastructure software installations is called the grid user. A user created to own either all Oracle installations, or only Oracle database installations, is called the oracle user.

2.4.3.1 Understanding Restrictions for Oracle Software Installation Owners

If you intend to use multiple Oracle software owners for different Oracle Database homes, then Oracle recommends that you create a separate software owner for Oracle Grid Infrastructure software (Oracle Clusterware and Oracle ASM), and use that owner to run the Oracle Grid Infrastructure installation.

If you plan to install Oracle Database or Oracle RAC, then Oracle recommends that you create separate users for the Oracle Grid Infrastructure and the Oracle Database installations. If you use one installation owner, then when you want to perform administration tasks, you must change the value for $ORACLE_HOME to the instance you want to administer (ASM, in the Oracle Grid Infrastructure home, or the database in the Oracle home), using command syntax such as the following example, where /u01/app/11.2.0/grid is the Oracle Grid Infrastructure home:

$ ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME

If you try to administer an instance using sqlplus, lsnrctl, or asmcmd commands while $ORACLE_HOME is set to a different binary path, then you will encounter errors. When starting srvctl from a database home, $ORACLE_HOME should be set. or srvctl fails. But if you are using srvctl in the Oracle Grid Infrastructure home, then $ORACLE_HOME is ignored, and the oracle home path does not affect srvctl commands. You always have to change $ORACLE_HOME to the instance that you want to administer.

To create separate Oracle software owners to create separate users and separate operating system privileges groups for different Oracle software installations, note that each of these users must have the Oracle central inventory group (oraInventory group) as their primary group. Members of this group have write privileges to the Oracle central inventory (oraInventory) directory, and are also granted permissions for various Oracle Clusterware resources, OCR keys, directories in the Oracle Clusterware home to which DBAs need write access, and other necessary privileges. In Oracle documentation, this group is represented as oinstall in code examples.

Each Oracle software owner must be a member of the same central inventory group. Oracle recommends that you do not have more than one central inventory for Oracle installations. If an Oracle software owner has a different central inventory group, then you may corrupt the central inventory.

Caution:

For grid infrastructure for a cluster installations, the Grid home must not be placed under one of the Oracle base directories, or under Oracle home directories of Oracle Database installation owners, or in the home directory of an installation owner. During installation, ownership of the path to the Grid home is changed to root. This change causes permission errors for other installations.

2.4.3.2 Determining if an Oracle Software Owner User Exists

To determine whether an Oracle software owner user named oracle or grid exists, enter a command similar to the following (in this case, to determine if oracle exists):

# id oracle

If the user exists, then the output from this command is similar to the following:

uid=501(oracle) gid=501(oinstall) groups=502(dba),503(oper)

Determine whether you want to use the existing user, or create another user. The user and group ID numbers must be the same on each node you intend to make a cluster member node.

To use the existing user, ensure that the user's primary group is the Oracle Inventory group (oinstall). If this user account will be used for Oracle Database installations, then ensure that the Oracle account is also a member of the group you plan to designate as the OSDBA for ASM group (the group whose members are permitted to write to Oracle ASM storage).

2.4.3.3 Creating or Modifying an Oracle Software Owner User for Oracle Grid Infrastructure

If the Oracle software owner (oracle, grid) user does not exist, or if you require a new Oracle software owner user, then create it. If you want to use an existing user account, then modify it to ensure that the user ID and group IDs are the same on each cluster member node. The following procedures use grid as the name of the Oracle software owner, and dba as the OSASM group. To create separate system privilege groups to separate administration privileges, complete group creation before you create the user, as described in Section 2.4.5, "Creating Job Role Separation Operating System Privileges Groups and Users."

Note:

If necessary, contact your system administrator before using or modifying an existing user.

Oracle recommends that you do not use the UID and GID defaults on each node, as group and user IDs likely will be different on each node. Instead, provide common assigned group and user IDs, and confirm that they are unused on any node before you create or modify groups and users.

2.4.3.3.1 Creating a New User

use the following procedure to create a new user:

  1. Enter the following command:

    # smit security
    
  2. On the Security & Users menu, select Users.

  3. On the Users menu, select Add a User.

  4. Choose the appropriate menu items to create the Oracle Grid Infrastructure software installation owner (grid). In the Primary GROUP field, specify the Oracle Inventory group. Make a note of the information you provide in the entry fields, so that you can provide the same value on other nodes.

  5. Press Enter, then Esc+0 (Exit) to exit.

  6. Set the password of the Oracle Grid Infrastructure software installation owner (grid). For example:

    # passwd grid
    
  7. Ensure that the Oracle Grid Infrastructure software installation owner (grid) has the capabilities CAP_NUMA_ATTACH, CAP_BYPASS_RAC_VMM, and CAP_PROPAGATE.

    To check existing capabilities, enter the following command as root:

    # /usr/bin/lsuser -a capabilities grid
     
    

    To add capabilities, enter a command similar to the following:

    # /usr/bin/chuser 
    capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid 
    
  8. Repeat this procedure on all of the other nodes in the cluster.

2.4.3.3.2 Modifying an Existing User

use the following procedure to modify an existing user:

  1. Enter the following command:

    # smit security
    
  2. Choose the appropriate menu items to modify the grid installation owner user.

  3. In the Primary GROUP field, specify the Oracle Inventory group, for example oinstall.

  4. Press Enter, then Esc+0 (Exit) to exit.

  5. Repeat this procedure on all of the other nodes in the cluster.

2.4.4 Creating the Oracle Base Directory Path

The Oracle base directory for the grid installation owner is the location where diagnostic and administrative logs, and other logs associated with Oracle ASM and Oracle Clusterware are stored.

If you have created a path for the Oracle Clusterware home that is compliant with Oracle Optimal Flexible Architecture (OFA) guidelines for Oracle software paths then you do not need to create an Oracle base directory. When OUI finds an OFA-compliant path, it creates the Oracle base directory in that path.

For OUI to recognize the path as an Oracle software path, it must be in the form u[00-99]/app, and it must be writable by any member of the oraInventory (oinstall) group. The OFA path for the Oracle base is /u01/app/user, where user is the name of the software installation owner.

Oracle recommends that you create an Oracle Grid Infrastructure Grid home and Oracle base homes manually, particularly if you have separate Oracle Grid Infrastructure for a cluster and Oracle Database software owners, so that you can separate log files.

For example:

# mkdir -p  /u01/app/11.2.0/grid
# mkdir -p /u01/app/grid
# mkdir -p /u01/app/oracle
# chown grid:oinstall /u01/app/11.2.0/grid
# chown grid:oinstall /u01/app/grid
# chown oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/
# chown -R grid:oinstall /u01

Note:

Placing Oracle Grid Infrastructure for a cluster binaries on a cluster file system is not supported.

2.4.5 Creating Job Role Separation Operating System Privileges Groups and Users

A job role separation privileges configuration of Oracle ASM is a configuration with groups and users that divide administrative access privileges to the Oracle ASM installation from other administrative privileges users and groups associated with other Oracle installations. Administrative privileges access is granted by membership in separate operating system groups, and installation privileges are granted by using different installation owners for each Oracle installation.

Note:

This configuration is optional, to restrict user access to Oracle software by responsibility areas for different administrator users.

If you prefer, you can allocate operating system user privileges so that you can use one administrative user and one group for operating system authentication for all system privileges on the storage and database tiers.

For example, you can designate the oracle user to be the installation owner for all Oracle software, and designate oinstall to be the group whose members are granted all system privileges for Oracle Clusterware, Oracle Automatic Storage Management, and all Oracle Databases on the servers, and all privileges as installation owners. This group must also be the Oracle Inventory group.

Oracle recommends that you use at least two groups: A system privileges group whose members are granted administrative system privileges, and an installation owner group (the oraInventory group) to provide separate installation privileges the OINSTALL privilege. To simplify using the defaults for Oracle tools such as Cluster Verification Utility, if you do choose to use a single operating system group to grant all system privileges and the right to write to the oraInventory, then that group name should be oinstall.

Note:

To use a directory service, such as Network Information Services (NIS), refer to your operating system documentation for further information.

2.4.5.1 Overview of Creating Job Role Separation Groups and Users

This section provides an overview of how to create users and groups to use job role separation. Log in as root to create these groups and users.

2.4.5.1.1 Users for Oracle Installations with Job Role Separation

Oracle recommends that you create the following operating system groups and users for all installations where you create separate software installation owners:

One software owner to own each Oracle software product (typically, oracle, for the database software owner user, and grid for Oracle Grid Infrastructure.

You must create at least one software owner the first time you install Oracle software on the system. This user owns the Oracle binaries of the Oracle Grid Infrastructure software, and you can also make this user the owner of the Oracle Database or Oracle RAC binaries.

Oracle software owners must have the Oracle Inventory group as their primary group, so that each Oracle software installation owner can write to the central inventory (oraInventory), and so that OCR and Oracle Clusterware resource permissions are set correctly. The database software owner must also have the OSDBA group and (if you create it) the OSOPER group as secondary groups. In Oracle documentation, when Oracle software owner users are referred to, they are called oracle users.

Oracle recommends that you create separate software owner users to own each Oracle software installation. Oracle particularly recommends that you do this if you intend to install multiple databases on the system.

In Oracle documentation, a user created to own the Oracle Grid Infrastructure binaries is called the grid user. This user owns both the Oracle Clusterware and Oracle Automatic Storage Management binaries.

See Also:

Oracle Clusterware Administration and Deployment Guide and Oracle Database Administrator's Guide for more information about the OSDBA, OSASM and OSOPER groups and the SYSDBA, SYSASM and SYSOPER privileges
2.4.5.1.2 Database Groups for Job Role Separation Installations

The following operating system groups and user are required if you are installing Oracle Database:

  • The OSDBA group (typically, dba)

    You must create this group the first time you install Oracle Database software on the system. This group identifies operating system user accounts that have database administrative privileges (the SYSDBA privilege). If you do not create separate OSDBA, OSOPER and OSASM groups for the Oracle ASM instance, then operating system user accounts that have the SYSOPER and SYSASM privileges must be members of this group. The name used for this group in Oracle code examples is dba. If you do not designate a separate group as the OSASM group, then the OSDBA group you define is also by default the OSASM group.

    To specify a group name other than the default dba group, then you must choose the Advanced installation type to install the software or start Oracle Universal Installer (OUI) as a user that is not a member of this group. In this case, OUI prompts you to specify the name of this group.

    Members of the OSDBA group formerly were granted SYSASM privileges on Oracle ASM instances, including mounting and dismounting disk groups. This privileges grant is removed with Oracle Grid Infrastructure 11g release 2, if different operating system groups are designated as the OSDBA and OSASM groups. If the same group is used for both OSDBA and OSASM, then the privilege is retained.

  • The OSOPER group for Oracle Database (typically, oper)

    This is an optional group. Create this group if you want a separate group of operating system users to have a limited set of database administrative privileges (the SYSOPER privilege). By default, members of the OSDBA group also have all privileges granted by the SYSOPER privilege.

    To use the OSOPER group to create a database administrator group with fewer privileges than the default dba group, then you must choose the Advanced installation type to install the software or start OUI as a user that is not a member of the dba group. In this case, OUI prompts you to specify the name of this group. The usual name chosen for this group is oper.

2.4.5.1.3 Oracle ASM Groups for Job Role Separation Installations

SYSASM is a new system privilege that enables the separation of the Oracle ASM storage administration privilege from SYSDBA. With Oracle Automatic Storage Management 11g Release 2 (11.2), members of the database OSDBA group are not granted SYSASM privileges, unless the operating system group designated as the OSASM group is the same group designated as the OSDBA group.

Select separate operating system groups as the operating system authentication groups for privileges on Oracle ASM. Before you start OUI, create the following groups and users for Oracle ASM

  • The Oracle Automatic Storage Management Group (typically asmadmin)

    This is a required group. Create this group as a separate group if you want to have separate administration privilege groups for Oracle ASM and Oracle Database administrators. In Oracle documentation, the operating system group whose members are granted privileges is called the OSASM group, and in code examples, where there is a group specifically created to grant this privilege, it is referred to as asmadmin.

    If you have multiple databases on your system, and use multiple OSDBA groups so that you can provide separate SYSDBA privileges for each database, then you should create a separate OSASM group, and use a separate user from the database users to own the Oracle Grid Infrastructure installation (Oracle Clusterware and Oracle ASM). Oracle ASM can support multiple databases.

    Members of the OSASM group can use SQL to connect to an Oracle ASM instance as SYSASM using operating system authentication. The SYSASM privileges permit mounting and dismounting disk groups, and other storage administration tasks. SYSASM privileges provide no access privileges on an RDBMS instance.

  • The Oracle ASM Database Administrator group (OSDBA for ASM, typically asmdba)

    Members of the Oracle ASM Database Administrator group (OSDBA for ASM) are granted read and write access to files managed by Oracle ASM. The Oracle Grid Infrastructure installation owner and all Oracle Database software owners must be a member of this group, and all users with OSDBA membership on databases that have access to the files managed by Oracle ASM must be members of the OSDBA group for ASM.

  • Members of the Oracle ASM Operator Group (OSOPER for ASM, typically asmoper)

    This is an optional group. Create this group if you want a separate group of operating system users to have a limited set of Oracle ASM instance administrative privileges (the SYSOPER for ASM privilege), including starting up and stopping the Oracle ASM instance. By default, members of the OSASM group also have all privileges granted by the SYSOPER for ASM privilege.

    To use the Oracle ASM Operator group to create an ASM administrator group with fewer privileges than the default asmadmin group, then you must choose the Advanced installation type to install the software, In this case, OUI prompts you to specify the name of this group. In code examples, this group is asmoper.

2.4.5.2 Creating Database Groups and Users with Job Role Separation

The following sections describe how to create the required operating system user and groups:.

2.4.5.2.1 Creating the OSDBA Group to Prepare for Database Installations

If you intend to install Oracle Database to use with the Oracle Grid Infrastructure installation, then you must create an OSDBA group in the following circumstances:

  • An OSDBA group does not exist; for example, if this is the first installation of Oracle Database software on the system

  • An OSDBA group exists, but you want to give a different group of operating system users database administrative privileges for a new Oracle Database installation

If the OSDBA group does not exist, or if you require a new OSDBA group, then create it either by using smit or by using shell command lines. Use the group name dba unless a group with that name already exists. For example:

# mkgroup -'A' id='1031' adms='root' dba
2.4.5.2.2 Creating an OSOPER Group for Database Installations

Create an OSOPER group only if you want to identify a group of operating system users with a limited set of database administrative privileges (SYSOPER operator privileges). For most installations, it is sufficient to create only the OSDBA group. To use an OSOPER group, then you must create it in the following circumstances:

  • If an OSOPER group does not exist; for example, if this is the first installation of Oracle Database software on the system

  • If an OSOPER group exists, but you want to give a different group of operating system users database operator privileges in a new Oracle installation

If you require a new OSOPER group, then create it either by using smit or by using shell command lines. Use the group name oper unless a group with that name already exists.

# mkgroup -'A' id='1032' adms='root' oper1
2.4.5.2.3 Creating the OSASM Group

If the OSASM group does not exist or if you require a new OSASM group, then create it either by using smit or by using shell command lines. Use the group name asmadmin unless a group with that name already exists:

# mkgroup -'A' id='1020' adms='root' asmadmin
2.4.5.2.4 Creating the OSOPER for ASM Group

Create an OSOPER for ASM group if you want to identify a group of operating system users, such as database administrators, whom you want to grant a limited set of Oracle ASM storage tier administrative privileges, including the ability to start up and shut down the Oracle ASM storage. For most installations, it is sufficient to create only the OSASM group, and provide that group as the OSOPER for ASM group during the installation interview.

If you require a new OSOPER for ASM group, then create it either by using smit or by using shell command lines. Use the group name asmoper unless a group with that name already exists:

# mkgroup -'A' id='1022' adms='root' asmoper
2.4.5.2.5 Creating the OSDBA for ASM Group for Database Access to Oracle ASM

You must create an OSDBA for ASM group to provide access to the Oracle ASM instance. This is necessary if OSASM and OSDBA are different groups.

If the OSDBA for ASM group does not exist or if you require a new OSDBA for ASM group, then create it either by using smit or by using shell command lines. Use the group name asmdba unless a group with that name already exists. For example:

# mkgroup -'A' id='1021' adms='root' asmdba
2.4.5.2.6 When to Create the Oracle Software Owner User

You must create an Oracle software owner user in the following circumstances:

  • If an Oracle software owner user exists, but you want to use a different operating system user, with different group membership, to give database administrative privileges to those groups in a new Oracle Database installation

  • If you have created an Oracle software owner for Oracle Grid Infrastructure, such as grid, and you want to create a separate Oracle software owner for Oracle Database software, such as oracle.

2.4.5.2.7 Determining if an Oracle Software Owner User Exists

To determine whether an Oracle software owner user named oracle or grid exists, enter a command similar to the following (in this case, to determine if oracle exists):

# id oracle

If the user exists, then the output from this command is similar to the following:

uid=501(oracle) gid=501(oinstall) groups=502(dba),503(oper)

Determine whether you want to use the existing user, or create another user. To use the existing user, ensure that the user's primary group is the Oracle Inventory group and that it is a member of the appropriate OSDBA and OSOPER groups. Refer to one of the following sections for more information:

Note:

If necessary, contact your system administrator before using or modifying an existing user.

Oracle recommends that you do not use the UID and GID defaults on each node, as group and user IDs likely will be different on each node. Instead, provide common assigned group and user IDs, and confirm that they are unused on any node before you create or modify groups and users.

2.4.5.2.8 Creating an Oracle Software Owner User

If the Oracle software owner user does not exist, or if you require a new Oracle software owner user, then create it as follows. Use the user name oracle unless a user with that name already exists.

  1. To create an oracle user, use smit or enter a command similar to the following:

    # mkuser id='1101' pgrp='oinstall' groups='dba,asmdba' home='/home/oracle' oracle
    
  2. Set the password of the oracle user:

    # passwd oracle
    
2.4.5.2.9 Modifying an Existing Oracle Software Owner User

If the oracle user exists, but its primary group is not oinstall, or it is not a member of the appropriate OSDBA or OSDBA for ASM groups, then update it using smit:

  1. Start SMIT by entering smit at the shell prompt.

  2. From the Main Menu, make the following selections:

    • Security and Users

    • Groups

    • Change/Show Characteristics of a Group

    The utility displays a form in which you can type the name of a specific group.

  3. Either fill in the group name or use the F4 key to highlight a group name and press the Enter key.

    The utility displays a form that provides the following fields:

    • Group NAME, which is the group name for the account.

    • Group ID, which is the Group Identification Number (GID).

    • User list, which is a list of users who are members of the group. Use the F4 key to display a list of available users and the F7 key to mark the users you want to add.

  4. Modify these fields as needed and press the Enter key to exit the form.

2.4.5.2.10 Creating Identical Database Users and Groups on Other Cluster Nodes

Oracle software owner users and the Oracle Inventory, OSDBA, and OSOPER groups must exist and be identical on all cluster nodes. To create these identical users and groups, you must identify the user ID and group IDs assigned them on the node where you created them, and then create the user and groups with the same name and ID on the other cluster nodes.

Note:

You must complete the following procedures only if you are using local users and groups. If you are using users and groups defined in a directory service such as NIS, then they are already identical on each cluster node.

Identifying Existing User and Group IDs

To determine the user ID (UID) of the grid or oracle users, and the group IDs (GID) of the existing Oracle groups, follow these steps:

  1. Enter a command similar to the following (in this case, to determine a user ID for the oracle user):

    # id oracle
    

    The output from this command is similar to the following:

    uid=502(oracle) gid=501(oinstall) groups=502(dba),503(oper),506(asmdba)
    
  2. From the output, identify the user ID (uid) for the user and the group identities (gid) for the groups to which it belongs. Ensure that these ID numbers are identical on each node of the cluster. The user's primary group is listed after gid. Secondary groups are listed after groups.

Creating Users and Groups on the Other Cluster Nodes

To create users and groups on the other cluster nodes, repeat the following procedure on each node:

  1. Log in to the next cluster node as root.

  2. Create groups and users as needed, either by using smit or by entering command lines. To use command line entries, enter commands similar to the following to create the oinstall, asmadmin, and asmdba groups, and if required, the asmoper, dba, and oper groups. Use the id option to specify the correct gid for each group.

    # mkgroup -'A' id='1000' adms='root' oinstall 
    # mkgroup -'A' id='1020' adms='root' asmadmin
    # mkgroup -'A' id='1021' adms='root' asmdba
    # mkgroup -'A' id='1022' adms='root' asmoper
    # mkgroup -'A' id='1031' adms='root' dba
    # mkgroup -'A' id='1032' adms='root' oper
    

    Note:

    If the group already exists, then use smit to modify it if necessary. If you cannot use the same group ID for a particular group on this node, then view the /etc/group file on all nodes to identify a group ID that is available on every node. You must then change the group ID on all nodes to the same group ID.
  3. To create the oracle or Oracle Grid Infrastructure (grid) user, use smit or enter a command similar to the following (in this example, to create the oracle user):

    # mkuser id='1001' pgrp='oinstall' groups='dba,asmdba' home='/home/oracle' oracle
    

    In the preceding command:

    • The id option specifies the user ID, which must be the user ID that you identified in the previous subsection

    • The pgrp option specifies the primary group, which must be the Oracle Inventory group, for example oinstall

    • The groups option specifies the secondary groups, which can include the OSASM, OSDBA, OSDBA for ASM, and OSOPER or OSOPER for ASM groups. For example:

      • A grid installation owner: OSASM (asmadmin), whose members are granted the SYSASM privilege

      • An Oracle Database installation owner without SYSASM privileges access: OSDBA (dba), OSDBA for ASM (asmdba), OSOPER for ASM (asmoper)

      Note:

      If the user already exists, then use smit to modify it if necessary. If you cannot use the same user ID for the user on every node, then view the /etc/passwd file on all nodes to identify a user ID that is available on every node. You must then specify that ID for the user on all of the nodes.
  4. Set the password of the user. For example:

    # passwd oracle
    
  5. Complete user environment configuration tasks for each user as described in Section 2.14.1, "Environment Requirements for Oracle Grid Infrastructure Software Owner."

2.4.6 Example of Creating Standard Groups, Users, and Paths

The following is an example of how to use command lines to create the Oracle Inventory group (oinstall), and a single group (dba) as the OSDBA, OSASM and OSDBA for ASM groups. In addition, it shows how to create the Oracle Grid Infrastructure software owner (grid), and one Oracle Database owner (oracle) with correct group memberships. This example also shows how to configure an Oracle base path compliant with OFA structure with correct permissions:

# mkgroup -'A' id='1000' adms='root' oinstall 
# mkgroup -'A' id='1031' adms='root' dba
# mkuser id='1100' pgrp='oinstall' groups='dba' home='/home/grid' grid
# mkuser id='1101' pgrp='oinstall' groups='dba' home='/home/oracle' oracle
# mkdir -p  /u01/app/grid
# chown -R grid:oinstall /u01
# mkdir /u01/app/oracle
# chown oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/

After running these commands, you have the following groups and users:

  • An Oracle central inventory group, or oraInventory group (oinstall). Members who have the central inventory group as their primary group, are granted the OINSTALL permission to write to the oraInventory directory.

  • A single system privileges group that is used as the OSASM, OSDBA, OSDBA for ASM, and OSOPER for ASM group (dba), whose members are granted the SYSASM and SYSDBA privilege to administer Oracle Clusterware, Oracle ASM, and Oracle Database, and are granted SYSASM and OSOPER for ASM access to the Oracle ASM storage.

  • An Oracle grid installation for a cluster owner (grid), with the oraInventory group as its primary group, and with the OSASM group as the secondary group, with its Oracle base directory /u01/app/grid.

  • An Oracle Database owner (oracle) with the oraInventory group as its primary group, and the OSDBA group as its secondary group, with its Oracle base directory /u01/app/oracle.

  • /u01/app owned by grid:oinstall with 775 permissions before installation, and by root after the root.sh script is run during installation. This ownership and permissions enables OUI to create the Oracle Inventory directory, in the path /u01/app/oraInventory.

  • /u01 owned by grid:oinstall before installation, and by root after the root.sh script is run during installation.

  • /u01/app/11.2.0/grid owned by grid:oinstall with 775 permissions. These permissions are required for installation, and are changed during the installation process.

  • /u01/app/grid owned by grid:oinstall with 775 permissions before installation, and 755 permissions after installation.

  • /u01/app/oracle owned by oracle:oinstall with 775 permissions.

2.4.7 Example of Creating Role-allocated Groups, Users, and Paths

The following is an example of how to create role-allocated groups and users that is compliant with an Optimal Flexible Architecture (OFA) deployment:

# mkgroup -'A' id='1000' adms='root' oinstall
# mkgroup -'A' id='1020' adms='root' asmadmin
# mkgroup -'A' id='1021' adms='root' asmdba
# mkgroup -'A' id='1031' adms='root' dba1
# mkgroup -'A' id='1041' adms='root' dba2
# mkgroup -'A' id='1022' adms='root' asmoper
# mkuser id='1100' pgrp='oinstall' groups='asmadmin' home='/home/grid' grid
# mkuser id='1101' pgrp='oinstall' groups='dba1,asmdba' home='/home/oracle' oracle1
# mkuser id='1102' pgrp='oinstall' groups='dba2,asmdba' home='/home/oracle' oracle2
# mkdir -p  /u01/app/11.2.0/grid
# mkdir -p /u01/app/grid
# chown -R grid:oinstall /u01/app/grid
# mkdir -p /u01/app/oracle1
# chown oracle1:oinstall /u01/app/oracle1
# mkdir -p /u01/app/oracle2
# chown oracle2:oinstall /u01/app/oracle2
# chmod -R 775 /u01

After running these commands, you have the following groups and users:

  • An Oracle central inventory group, or oraInventory group (oinstall), whose members that have this group as their primary group are granted permissions to write to the oraInventory directory.

  • A separate OSASM group (asmadmin), whose members are granted the SYSASM privilege to administer Oracle Clusterware and Oracle ASM.

  • A separate OSDBA for ASM group (asmdba), whose members include grid, oracle1 and oracle2, and who are granted access to Oracle ASM.

  • A separate OSOPER for ASM group (asmoper), whose members are granted limited Oracle ASM administrator privileges, including the permissions to start and stop the Oracle ASM instance.

  • An Oracle grid installation for a cluster owner (grid), with the oraInventory group as its primary group, and with the OSASM (asmadmin), OSDBA for ASM (asmdba) group as a secondary group.

  • Two separate OSDBA groups for two different databases (dba1 and dba2) to establish separate SYSDBA privileges for each database.

  • Two Oracle Database software owners (oracle1 and oracle2), to divide ownership of the Oracle database binaries, with the OraInventory group as their primary group, and the OSDBA group for their database (dba1 or dba2) and the OSDBA for ASM group (asmdba) as their secondary groups.

  • An OFA-compliant mount point /u01 owned by grid:oinstall before installation.

  • An Oracle base for the grid installation owner /u01/app/grid owned by grid:oinstall with 775 permissions, and changed during the installation process to 755 permissions.

  • An Oracle base /u01/app/oracle1 owned by oracle1:oinstall with 775 permissions.

  • An Oracle base /u01/app/oracle 2 owned by oracle2:oinstall with 775 permissions.

  • A Grid home /u01/app/11.2.0/grid owned by grid:oinstall with 775 (drwxdrwxr-x) permissions. These permissions are required for installation, and are changed during the installation process to root:oinstall with 755 permissions (drwxr-xr-x).

  • /u01/app/oraInventory. This path remains owned by grid:oinstall, to enable other Oracle software owners to write to the central inventory.

2.5 Checking the Hardware Requirements

  • Select servers with the same instruction set architecture.

  • Ensure that the server is started with run level 2 (default or Normal Multi-User mode).

  • Ensure servers run the same operating system level, APARs and filesets. Oracle Grid Infrastructure installations and Oracle Real Application Clusters (Oracle RAC) support servers with different hardware in the same cluster.

Each system must meet the following minimum hardware requirements:

  • At least 2.5 GB of RAM for Oracle Grid Infrastructure for a Cluster installations, including installations where you plan to install Oracle RAC.

  • At least 1024 x 768 display resolution, so that Oracle Universal Installer (OUI) displays correctly

  • Swap space equivalent to the multiple of the available RAM, as indicated in the following table:

    Available RAM Swap Space Required
    Between 2.5 GB and 16 GB Equal to the size of RAM
    More than 16 GB 16 GB

  • 1 GB of disk space in the /tmp directory

  • 13 GB of space for the Oracle Grid Infrastructure for a cluster home (Grid home) This includes Oracle Clusterware and Oracle Automatic Storage Management (Oracle ASM) files and log files, Oracle ACFS log files, and the Cluster Health Monitor repository.

    Note:

    If you intend to install Oracle Databases or an Oracle RAC database on the cluster, be aware that the size of the /dev/shm mount area on each server must be greater than the system global areal (SGA) and the program global area (PGA) of the databases on the servers. Review expected SGA and PGA sizes with database administrators, to ensure that you do not have to increase /dev/shm after databases are installed on the cluster.
  • 7.5 GB of disk space for the Oracle Database files (Oracle base)

  • 2 GB of disk space for a preconfigured database that uses file system storage (optional)

    If you choose to configure automated backups, then you require additional disk space, either on a file system or in an Oracle Automatic Storage Management disk group.

  • Upto 10 GB of additional space in the Oracle base directory of the Grid Infrastructure owner for diagnostic collections generated by Trace File Analyzer and Collector.

To ensure that each system meets these requirements, follow these steps:

  1. To determine the physical RAM size, enter the following command:

    # /usr/sbin/lsattr -E -l sys0 -a realmem
    

    If the size of the physical RAM installed in the system is less than the required size, then you must install more memory before continuing.

  2. To determine the available RAM and swap space, enter the following command:

    # /usr/sbin/lsps -s
    

    Note:

    • Oracle recommends that you take multiple values for the available RAM and swap space before finalizing a value. This is because the available RAM and swap space keep changing depending on the user interactions with the computer.

    • Contact your operating system vendor for swap space allocation guidance for your server. The vendor guidelines supersede the swap space requirements listed in this guide.

  3. To determine the size of the configured swap space, enter the following command:

    # /usr/sbin/lsps -a
    

    If necessary, refer to your operating system documentation for information about how to configure additional swap space.

  4. To determine the amount of disk space available in the /tmp directory, enter the following command:

    # /usr/bin/df -k /tmp
    

    If there is less than 1 GB of disk space available in the /tmp directory, then complete one of the following steps:

    • Delete unnecessary files from the /tmp directory to make available the disk space required.

    • Set the TEMP and TMPDIR environment variables when setting the oracle user's environment.

    • Extend the file system that contains the /tmp directory. If necessary, contact your system administrator for information about extending file systems.

  5. To determine the amount of free disk space on the system, use one of the following commands, depending on where you intend to place Oracle Clusterware files:

    GPFS:

    # /usr/bin/df -k
    

    Raw Logical Volumes in Concurrent VG (HACMP): in the following example, the variable lv_name is the name of the raw logical volume whose space you want to verify:

    # lslv lv_name
    

    Raw hard disks; in the following example, the variable rhdisk# is the raw hard disk number that you want to verify, and the variable size_mb is the size in megabytes of the partition that you want to verify:

    # lsattr -El rhdisk# -a size_mb
    
  6. To determine if the system is started in 64-bit mode, enter the following command:

    # bootinfo -K
    

    The result of this command should be 64, indicating that the 64-bit kernel is enabled.

2.6 Checking the Network Requirements

Review the following sections to check that you have the networking hardware and internet protocol (IP) addresses required for an Oracle Grid Infrastructure for a cluster installation:

Note:

For the most up-to-date information about supported network protocols and hardware for Oracle RAC installations, refer to the Certification pages on the My Oracle Support Web site at the following URL:
https://support.oracle.com

2.6.1 Network Hardware Requirements

The following is a list of requirements for network configuration:

  • Each node must have at least two network adapters or network interface cards (NICs): one for the public network interface, and one for the private network interface (the interconnect).

    With Redundant Interconnect Usage, you should identify multiple interfaces to use for the cluster private network, without the need of using bonding or other technologies. This functionality is available starting with Oracle Database 11g Release 2 (11.2.0.2).

    When you define multiple interfaces, Oracle Clusterware creates from one to four highly available IP (HAIP) addresses. Oracle RAC and Oracle ASM instances use these interface addresses to ensure highly available, load-balanced interface communication between nodes. The installer enables Redundant Interconnect Usage to provide a high availability private network.

    By default, Oracle Grid Infrastructure software uses all of the HAIP addresses for private network communication, providing load-balancing across the set of interfaces you identify for the private network. If a private interconnect interface fails or become non-communicative, then Oracle Clusterware transparently moves the corresponding HAIP address to one of the remaining functional interfaces.

    Note:

    If you define more than four interfaces as private network interfaces, be aware that Oracle Clusterware activates only four of the interfaces at a time. However, if one of the four active interfaces fails, then Oracle Clusterware transitions the HAIP addresses configured to the failed interface to one of the reserve interfaces in the defined set of private interfaces.

    When you upgrade a node to Oracle Grid Infrastructure 11g release 2 (11.2.0.2) and later, the upgraded system uses your existing network classifications.

    To configure multiple public interfaces, use a third-party technology for your platform to aggregate the multiple public interfaces before you start installation, and then select the single interface name for the combined interfaces as the public interface. Oracle recommends that you do not identify multiple public interface names during Oracle Grid Infrastructure installation. Note that if you configure two network interfaces as public network interfaces in the cluster without using an aggregation technology, the failure of one public interface on a node does not result in automatic VIP failover to the other public interface.

    Oracle recommends that you use the Redundant Interconnect Usage feature to make use of multiple interfaces for the private network. However, you can also use third-party technologies to provide redundancy for the private network.

    Note:

    NIC bonding is not required to use multiple NICs. During installation, you are given the opportunity to specify the planned use for all interfaces detected by the Installer. Oracle Clusterware uses all NICs that you identify as Private for the private interconnect.

    Redundant Interconnect Usage requires a complete release 11.2.0.2 or higher stack (Oracle Grid Infrastructure and Oracle Databases). Prior release Oracle Databases cannot use this feature, and must use third-party NIC bonding technologies. If you consolidate different database releases in one cluster, and use databases prior to Oracle Database release 11.2.0.2, then you may require both technologies.

  • If you install Oracle Clusterware using OUI, then the public interface names associated with the network adapters for each network must be the same on all nodes, and the private interface names associated with the network adaptors should be the same on all nodes. This restriction does not apply if you use cloning, either to create a new cluster, or to add nodes to an existing cluster.

    For example: With a two-node cluster, you cannot configure network adapters on node1 with eth0 as the public interface, but on node2 have eth1 as the public interface. Public interface names must be the same, so you must configure eth0 as public on both nodes. You should configure the private interfaces on the same network adapters as well. If eth1 is the private interface for node1, then eth1 should be the private interface for node2.

    See Also:

    Oracle Clusterware Administration and Deployment Guide for information about how to add nodes using cloning
  • For the public network, each network adapter must support TCP/IP.

  • For the private network, the interface must support the user datagram protocol (UDP) using high-speed network adapters and switches that support TCP/IP (minimum requirement 1 Gigabit Ethernet).

    Note:

    UDP is the default interface protocol for Oracle RAC, and TCP is the interconnect protocol for Oracle Clusterware. You must use a switch for the interconnect. Oracle recommends that you use a dedicated switch.

    Oracle does not support token-rings or crossover cables for the interconnect.

  • Each node's private interface for interconnects must be on the same subnet, and those subnets must connect to every node of the cluster. For example, if the private interfaces have a subnet mask of 255.255.255.0, then your private network is in the range 192.168.0.0--192.168.0.255, and your private addresses must be in the range of 192.168.0.[0-255]. If the private interfaces have a subnet mask of 255.255.0.0, then your private addresses can be in the range of 192.168.[0-255].[0-255].

    For clusters using Redundant Interconnect Usage, each private interface should be on a different subnet. However, each cluster member node must have an interface on each private interconnect subnet, and these subnets must connect to every node of the cluster. For example, you can have private networks on subnets 192.168.0 and 10.0.0, but each cluster member node must have an interface connected to the 192.168.0 and 10.0.0 subnets.

  • For the private network, the endpoints of all designated interconnect interfaces must be completely reachable on the network. There should be no node that is not connected to every private network interface. You can test if an interconnect interface is reachable using ping.

2.6.2 IP Address Requirements

Before starting the installation, you must have at least two interfaces configured on each node: One for the private IP address and one for the public IP address.

You can manage IP addresses and name resolution in the cluster in one of the following ways:

  • Dynamic IP address assignment using Oracle Grid Naming Service (GNS). If you select this option, then network administrators assign static IP address for the physical host name and dynamically allocated IPs for the Oracle Clusterware managed VIP addresses. In this case, IP addresses for the VIPs are assigned by a DHCP and resolved using a multicast domain name server configured as part of Oracle Clusterware within the cluster. If you plan to use GNS, then you must have the following:

    • A DHCP service running on the public network for the cluster

    • Enough addresses on the DHCP to provide 1 IP address for each node's virtual IP, and 3 IP addresses for the cluster used by the Single Client Access Name (SCAN) for the cluster

  • Static IP address assignment. If you select this option, then network administrators assign a fixed IP address for each physical host name in the cluster and for IPs for the Oracle Clusterware managed VIPs. In addition, domain name server (DNS) based static name resolution is used for each node. Selecting this option requires that you request network administration updates when you modify the cluster.

Note:

Oracle recommends that you use a static host name for all server node public hostnames.

Public IP addresses and virtual IP addresses must be in the same subnet.

Oracle only supports DHCP-assigned networks for the default network, not for any subsequent networks.

2.6.2.1 IP Address Requirements with Grid Naming Service

If you enable Grid Naming Service (GNS), then name resolution requests to the cluster are delegated to the GNS, which is listening on the GNS virtual IP address. You define this address in the DNS domain before installation. The DNS must be configured to delegate resolution requests for cluster names (any names in the subdomain delegated to the cluster) to the GNS. When a request comes to the domain, GNS processes the requests and responds with the appropriate addresses for the name requested.

To use GNS, before installation the DNS administrator must establish DNS Lookup to direct DNS resolution of a subdomain to the cluster. If you enable GNS, then you must have a DHCP service on the public network that allows the cluster to dynamically allocate the virtual IP addresses as required by the cluster.

Note:

The following restrictions apply to vendor configurations on your system:
  • If you have vendor clusterware installed, then you cannot choose to use GNS, because the vendor clusterware does not support it.

  • You cannot use GNS with another multicast DNS. If you want to use GNS, then disable any third party mDNS daemons on your system.

2.6.2.2 IP Address Requirements for Manual Configuration

If you do not enable GNS, then the public and virtual IP addresses for each node must be static IP addresses, configured before installation for each node, but not currently in use. Public and virtual IP addresses must be on the same subnet.

Oracle Clusterware manages private IP addresses in the private subnet on interfaces you identify as private during the installation interview.

The cluster must have the following addresses configured:

  • A public IP address for each node, with the following characteristics:

    • Static IP address

    • Configured before installation for each node, and resolvable to that node before installation

    • On the same subnet as all other public IP addresses, VIP addresses, and SCAN addresses

  • A virtual IP address for each node, with the following characteristics:

    • Static IP address

    • Configured before installation for each node, but not currently in use

    • On the same subnet as all other public IP addresses, VIP addresses, and SCAN addresses

    • Conforms with the RFC 952 standard, which allows alphanumeric characters and hyphens ("-"), but does not allow underscores ("_").

  • A Single Client Access Name (SCAN) for the cluster, with the following characteristics:

    • Three Static IP addresses configured on the domain name server (DNS) before installation so that the three IP addresses are associated with the name provided as the SCAN, and all three addresses are returned in random order by the DNS to the requestor

    • Configured before installation in the DNS to resolve to addresses that are not currently in use

    • Given a name that does not begin with a numeral

    • On the same subnet as all other public IP addresses, VIP addresses, and SCAN addresses

    • Conforms with the RFC 952 standard, which allows alphanumeric characters and hyphens ("-"), but does not allow underscores ("_").

  • A private IP address for each node, with the following characteristics:

    • Static IP address

    • Configured before installation, but on a separate, private network, with its own subnet, that is not resolvable except by other cluster member nodes

The SCAN is a name used to provide service access for clients to the cluster. Because the SCAN is associated with the cluster as a whole, rather than to a particular node, the SCAN makes it possible to add or remove nodes from the cluster without needing to reconfigure clients. It also adds location independence for the databases, so that client configuration does not have to depend on which nodes are running a particular database. Clients can continue to access the cluster in the same way as with previous releases, but Oracle recommends that clients accessing the cluster use the SCAN.

Note:

In a Typical installation, the SCAN you provide is also the name of the cluster. In an advanced installation, The SCAN and cluster name are entered in separate fields during installation.

Both the SCAN and the cluster name must be at least one character long and no more than 15 characters in length, must be alphanumeric, cannot begin with a numeral, and may contain hyphens (-).

You can use the nslookup command to confirm that the DNS is correctly associating the SCAN with the addresses. For example:

root@node1]$ nslookup mycluster-scan
Server:         dns.example.com
Address:        192.0.2.001
 
Name:   mycluster-scan.example.com
Address: 192.0.2.201
Name:   mycluster-scan.example.com
Address: 192.0.2.202
Name:   mycluster-scan.example.com
Address: 192.0.2.203

After installation, when a client sends a request to the cluster, the Oracle Clusterware SCAN listeners redirect client requests to servers in the cluster.

Note:

Oracle strongly recommends that you do not configure SCAN VIP addresses in the hosts file. Use DNS resolution for SCAN VIPs. If you use the hosts file to resolve SCANs, then you will only be able to resolve to one IP address and you will have only one SCAN address.

Configuring SCANs in a DNS or a hosts file is the only supported configuration. Configuring SCANs in a Network Information Service (NIS) is not supported.

See Also:

Appendix C, "Understanding Network Addresses" for more information about network addresses

2.6.3 Broadcast Requirements for Networks Used by Oracle Grid Infrastructure

Broadcast communications (ARP and UDP) must work properly across all the public and private interfaces configured for use by Oracle Grid Infrastructure release 2 patchset 1 (11.2.0.2) and later releases.

The broadcast must work across any configured VLANs as used by the public or private interfaces.

2.6.4 Multicast Requirements for Networks Used by Oracle Grid Infrastructure

With Oracle Grid Infrastructure release 2 (11.2), on each cluster member node, the Oracle mDNS daemon uses multicasting on all interfaces to communicate with other nodes in the cluster.

With Oracle Grid Infrastructure release 2 patchset 1 (11.2.0.2) and later releases, multicasting is required on the private interconnect. For this reason, at a minimum, you must enable multicasting for the cluster:

  • Across the broadcast domain as defined for the private interconnect

  • On the IP address subnet ranges 224.0.0.0/24 and 230.0.1.0/24

You do not need to enable multicast communications across routers.

2.6.5 DNS Configuration for Domain Delegation to Grid Naming Service

If you plan to use GNS, then before Oracle Grid Infrastructure installation, you must configure your domain name server (DNS) to send to GNS name resolution requests for the subdomain GNS serves, which are the cluster member nodes. The following is an overview of what needs to be done for domain delegation. Your actual procedure may be different from this example.

Configure the DNS to send GNS name resolution requests using delegation:

  1. In the DNS, create an entry for the GNS virtual IP address, where the address uses the form gns-server.CLUSTERNAME.DOMAINNAME. For example, where the cluster name is mycluster, and the domain name is example.com, and the IP address is 192.0.2.1, create an entry similar to the following:

    mycluster-gns.example.com  A  192.0.2.1
    

    The address you provide must be routable.

  2. Set up forwarding of the GNS subdomain to the GNS virtual IP address, so that GNS resolves addresses to the GNS subdomain. To do this, create a BIND configuration entry similar to the following for the delegated domain, where cluster01.example.com is the subdomain you want to delegate:

     cluster01.example.com  NS  mycluster-gns.example.com
    
  3. When using GNS, you must configure resolve.conf on the nodes in the cluster (or the file on your system that provides resolution information) to contain name server entries that are resolvable to corporate DNS servers. The total timeout period configured—a combination of options attempts (retries) and options timeout (exponential backoff)—should be less than 30 seconds. For example, where xxx.xxx.xxx.42 and xxx.xxx.xxx.15 are valid name server addresses in your network, provide an entry similar to the following in /etc/resolv.conf:

    options attempts: 2
    options timeout: 1
    
    search cluster01.example.com example.com
    nameserver xxx.xxx.xxx.42
    nameserver xxx.xxx.xxx.15
    

    /etc/nsswitch.conf controls name service lookup order. In some system configurations, the Network Information System (NIS) can cause problems with Oracle SCAN address resolution. Oracle recommends that you place the nis entry at the end of the search list. For example:

    /etc/nsswitch.conf
         hosts:    files   dns   nis
    

    Note:

    Be aware that use of NIS is a frequent source of problems when doing cable pull tests, as host name and username resolution can fail.

2.6.6 Grid Naming Service Configuration Example

If you use GNS, then you need to specify a static IP address for the GNS VIP address, and delegate a subdomain to be delegated to that static GNS IP address.

As nodes are added to the cluster, your organization's DHCP server can provide addresses for these nodes dynamically. These addresses are then registered automatically in GNS, and GNS provides resolution within the subdomain to cluster node addresses registered with GNS.

Because allocation and configuration of addresses is performed automatically with GNS, no further configuration is required. Oracle Clusterware provides dynamic network configuration as nodes are added to or removed from the cluster. The following example is provided only for information.

With a two node cluster where you have defined the GNS VIP, after installation you might have a configuration similar to the following for a two-node cluster, where the cluster name is mycluster, the GNS parent domain is example.com, the subdomain is grid.example.com, 192.0.2 in the IP addresses represent the cluster public IP address network, and 192.168.0 represents the private IP address subnet:

Table 2-1 Grid Naming Service Example Network

Identity Home Node Host Node Given Name Type Address Address Assigned By Resolved By

GNS VIP

None

Selected by Oracle Clusterware

mycluster-gns.example.com

virtual

192.0.2.1

Fixed by net administrator

DNS

Node 1 Public

Node 1

node1

node1Foot 1 

Public

192.0.2.101

Fixed

GNS

Node 1 VIP

Node 1

Selected by Oracle Clusterware

node1-vip

Virtual

192.0.2.104

DHCP

GNS

Node 1 Private

Node 1

node1

node1-priv

Private

192.168.0.1

Fixed or DHCP

GNS

Node 2 Public

Node 2

node2

node2Footref 1

Public

192.0.2.102

Fixed

GNS

Node 2 VIP

Node 2

Selected by Oracle Clusterware

node2-vip

Virtual

192.0.2.105

DHCP

GNS

Node 2 Private

Node 2

node2

node2-priv

Private

192.168.0.2

Fixed or DHCP

GNS

SCAN VIP 1

none

Selected by Oracle Clusterware

mycluster-scan.cluster01.example.com

virtual

192.0.2.201

DHCP

GNS

SCAN VIP 2

none

Selected by Oracle Clusterware

mycluster-scan.cluster01.example.com

virtual

192.0.2.202

DHCP

GNS

SCAN VIP 3

none

Selected by Oracle Clusterware

mycluster-scan.cluster01.example.com

virtual

192.0.2.203

DHCP

GNS


Footnote 1 Node host names may resolve to multiple addresses, including VIP addresses currently running on that host.

2.6.7 Manual IP Address Configuration Example

If you choose not to use GNS, then before installation you must configure public, virtual, and private IP addresses. Also, check that the default gateway can be accessed by a ping command. To find the default gateway, use the route command, as described in your operating system's help utility.

For example, with a two node cluster where each node has one public and one private interface, and you have defined a SCAN domain address to resolve on your DNS to one of three IP addresses, you might have the configuration shown in the following table for your network interfaces:

Table 2-2 Manual Network Configuration Example

Identity Home Node Host Node Given Name Type Address Address Assigned By Resolved By

Node 1 Public

Node 1

node1

node1Foot 1 

Public

192.0.2.101

Fixed

DNS

Node 1 VIP

Node 1

Selected by Oracle Clusterware

node1-vip

Virtual

192.0.2.104

Fixed

DNS and hosts file

Node 1 Private

Node 1

node1

node1-priv

Private

192.168.0.1

Fixed

DNS and hosts file, or none

Node 2 Public

Node 2

node2

node2Footref 1

Public

192.0.2.102

Fixed

DNS

Node 2 VIP

Node 2

Selected by Oracle Clusterware

node2-vip

Virtual

192.0.2.105

Fixed

DNS and hosts file

Node 2 Private

Node 2

node2

node2-priv

Private

192.168.0.2

Fixed

DNS and hosts file, or none

SCAN VIP 1

none

Selected by Oracle Clusterware

mycluster-scan

virtual

192.0.2.201

Fixed

DNS

SCAN VIP 2

none

Selected by Oracle Clusterware

mycluster-scan

virtual

192.0.2.202

Fixed

DNS

SCAN VIP 3

none

Selected by Oracle Clusterware

mycluster-scan

virtual

192.0.2.203

Fixed

DNS


Footnote 1 Node hostnames may resolve to multiple addresses.

You do not need to provide a private name for the interconnect. If you want name resolution for the interconnect, then you can configure private IP names in the hosts file or the DNS. However, Oracle Clusterware assigns interconnect addresses on the interface defined during installation as the private interface (en1, for example), and to the subnet used for the private subnet.

The addresses to which the SCAN resolves are assigned by Oracle Clusterware, so they are not fixed to a particular node. To enable VIP failover, the configuration shown in the preceding table defines the SCAN addresses and the public and VIP addresses of both nodes on the same subnet, 192.0.2.

Note:

All host names must conform to the RFC 952 standard, which permits alphanumeric characters. Host names using underscores ("_") are not allowed.

2.6.8 Network Interface Configuration Options

The precise configuration you choose for your network depends on the size and use of the cluster you want to configure, and the level of availability you require.

If certified Network-attached Storage (NAS) is used for Oracle RAC and this storage is connected through Ethernet-based networks, then you must have a third network interface for NAS I/O. Failing to provide three separate interfaces in this case can cause performance and stability problems under load.

2.7 Identifying the Software Requirements

Depending on the products that you intend to install, verify that the following operating system software is installed on the system. To check these requirements refer to Chapter 2, "Checking the Software Requirements."

Requirements listed here are current as of the initial release date. To obtain the most current information about kernel requirements, refer to the online version on the Oracle Technology Network (OTN) at the following URL:

http://www.oracle.com/technetwork/indexes/documentation/index.html

OUI performs checks your system to verify that it meets the listed operating system package requirements. To ensure that these checks complete successfully, verify the requirements before you start OUI.

Note:

Oracle does not support running different operating system versions on cluster members, unless an operating system is being upgraded. You cannot run different operating system version binaries on members of the same cluster, even if each operating system is supported.

Table 2-3 AIX Operating System Kernel Requirements

Item Requirement

Operating systems

AIX 7.1 TL 0 SP1 (" 7100-00-01") 64-bit kernel

AIX 6.1 TL 02 SP1 ("6100-02-01"), 64-bit kernel

AIX 5L V5.3 TL 09 SP1 ("5300-09-01"), 64 bit kernel or later

AIX 7.1 operating system filesets

The following operating system filesets are required:

bos.adt.base
bos.adt.lib
bos.adt.libm
bos.perf.libperfstat
bos.perf.perfstat
bos.perf.proctools
xlC.rte.11.1.0.2 or later

You must have the IBM XL C/C++ runtime filesets for installation, but you do not require the C/C++ compilers. You do not require a license for the XL C/C++ runtime filesets.

AIX 6.1 operating system filesets

The following operating system filesets are required:

bos.adt.base
bos.adt.lib
bos.adt.libm
bos.perf.libperfstat 6.1.2.1 or later
bos.perf.perfstat
bos.perf.proctools
rsct.basic.rte
rsct.compat.clients.rte
xlC.aix61.rte.10.1.0.0 or later
xlC.rte:10.1.0.0 or later

You must have the IBM XL C/C++ runtime filesets for installation, but you do not require the C/C++ compilers. You do not require a license for the XL C/C++ runtime filesets.

Version: IBM XL C/C++ Enterprise Edition for AIX, V9.0 September 2008 PTF

AIX 5L operating system filesets

The following operating system filesets are required:

bos.adt.base
bos.adt.lib
bos.adt.libm
bos.perf.libperfstat
bos.perf.perfstat
bos.perf.proctools
rsct.basic.rte
rsct.compat.clients.rte
xlC.aix50.rte:10.1.0.0 (or later)

You must have the IBM XL C/C++ runtime filesets for installation, but you do not require the C/C++ compilers. You do not require a license for the XL C/C++ runtime filesets. Version: IBM XL C/C++ Enterprise Edition for AIX, V9.0 September 2008 PTF

Obtaining C/C++ Compilers

For AIX7.1, check My Oracle Support Certifications for current requirements. Certifications are available on the following URL:

https://support.oracle.com

The minimum supported compiler for Oracle Grid Infrastructure is V. 9.0. You can download this software from the following website:

http://www-1.ibm.com/support/docview.wss?uid=swg24019055

Note: Even if you do not install the IBM XL C/C++ compiler, you require the compiler for AIX Runtime Environment Component. The runtime environment file sets can be downloaded with no license requirements. The minimum recommended runtime environment for AIX 5.3 and AIX 6.1 is available at the following URL:

For AIX 5.3 and AIX 6.1:

IBM XL C/C++ for AIX, V10.1 Runtime Environment

http://www-01.ibm.com/support/docview.wss?rs=2239&uid=swg24019829

For AIX 7.1:

September 2010 Runtime for XL C/C++ for AIX, V11.1

http://www-01.ibm.com/support/docview.wss?uid=swg24028034

Oracle RAC and HACMP

High Availability Cluster Multi-Processing (HACMP) 5.4.1

Note: HACMP is required only if you want to use raw logical volumes for Oracle Clusterware or database file storage. However, it is supported for all installations. You cannot use raw devices for OCR or voting disk files.

For AIX7.1, if you want to use HACMP or GPFS, then check My Oracle Support Certification for current requirements. Certifications are available at the following URL:

https://support.oracle.com

If you do not want to use HACMP, then you must not have HACMP installed on your system. If you have previously installed HACMP, then you must remove the following:

  • HACMP filesets (cluster.es.*)

  • rsct.hacmp.rte

  • rsct.compat.basic.hacmp.rte

  • rsct.compat.clients.hacmp.rte

If you want to use HACMP, then review patch sets to ensure that you have required patches. Changes in the fileset packaging of HACMP 5.4, including 5.4.1, require updates to the Oracle rootpre.sh script. Download and install patch 6718715 before installing Oracle Clusterware.

Oracle RAC and GPFS

General Parallel File System (GPFS):

AIX 7.1:

GPFS 3.3.0.11

AIX 6.1:

gpfs.base 3.2.1.8 or later. 

AIX 5L:

gpfs.base 3.2.1.8 or later

Note: GPFS is not required. Install GPFS only if you want to use a cluster file system in addition to Oracle Clusterware.

SSH

Oracle Clusterware requires SSH. The required SSH software is the default SSH shipped with your operating system.

ADA

OC Systems PowerAda 5.5

For AIX7.1, check My Oracle Support Certification for current requirements. Certifications are available on the following URL:

https://support.oracle.com

JDK

Use one of the following Java versions:

Java 6 64-bit 6.0.0.50 IZ30726 (SR2)

Java 5 64-bit 5.0.0.250 IZ55274 (SR10)

For AIX7.1, check My Oracle Support Certification for current requirements. Certifications are available on the following URL:

https://support.oracle.com

Pro*FORTRAN

IBM XL Fortran v. 11.1 April 2008 PTF for AIX

For AIX7.1, check My Oracle Support Certification for current requirements. Certifications are available on the following URL:

https://support.oracle.com

Pro*C/C++, Oracle Call Interface, Oracle C++ Call Interface, Oracle XML Developer's Kit (XDK), GNU Compiler Collection (GCC)

Note: If you do not install the C/C++ compilers, then you require the C/C++ runtime filesets for installation as described in the "Operating system filesets" row in this table.

For AIX7.1, check My Oracle Support Certification for current requirements. Certifications are available on the following URL:

https://support.oracle.com

Pro*COBOL

  • IBM COBOL for AIX Version 4.1 (September 2010 PTF)

  • IBM COBOL for AIX version 3.1

  • Micro Focus Server Express 5.1

For AIX7.1, check My Oracle Support Certification for current requirements. Certifications are available on the following URL:

https://support.oracle.com

Oracle Messaging Gateway

IBM WebSphere MQ V6.0.2.0, client and server:

mqm.Client.Bnd
mqm.Server.Bnd

For AIX7.1, check My Oracle Support Certification for current requirements. Certifications are available on the following URL:

https://support.oracle.com

Verify that the following patches are installed on the system. The procedure following the table describes how to check these requirements

Note:

There may be more recent versions of the patches listed installed on your system. If a listed patch is not installed, then determine if you have a more recent patch installed that includes the listed patch before you install the patch version listed.

Table 2-4 AIX APAR and Other Operating System Fixes

Installation Type or Product Requirement

AIX 7.1 installations

If you are using the minimum operating system TL level for AIX 7.1 listed above, then install the following Authorized Problem Analysis Reports (APARs) for AIX 7.1 TL 0 SP1:

IZ87216
IZ87564
IZ89165
IZ97035

If you are using a later TL level than the minimum level listed for this release, then contact IBM to determine if the required APARs listed here are included in the TL level that you have on your system. If they are included, then you do not need to install them. If they are not included, then you must install the equivalent APAR for the appropriate TL level.

AIX 6.1 installations

If you are using a later TL level than the minimum level listed for this release, then contact IBM to determine if the required APARs listed here are included in the TL level that you have on your system. If they are included, then you do not need to install them. If they are not included, then you must install the equivalent APAR for the appropriate TL level.

If you are using the minimum operating system TL level for AIX 6.1 listed above, then install the following Authorized Problem Analysis Reports (APARs) for AIX 6.1 TL 02 SP1:

IZ41855
IZ51456
IZ52319
IZ97457
IZ89165

These 6.1 fixes are already present in the following TL levels:

  • AIX 6.1 TL 02 SP-04 and later

  • AIX 6.1 TL 03 SP-02 and later

  • AIX 6.1 TL 04

If you are using a later TL level than the minimum level listed for this release, apply the following additional operating system patch for defect:

BIND64 CORES WITH -BLAZY OPTION

Download the appropriate patch for your operating system TL level using the following APAR numbers:

  • AIX 6.1 TL 03 - use AIX APAR IZ89304

  • AIX 6.1 TL 04 - use AIX APAR IZ89302

  • AIX 6.1 TL 05 - use AIX APAR IZ89300

  • AIX 6.1 TL 06 SP4 - use AIX APAR IZ88711

  • AIX 6.1 TL 06 SP5 - use AIX APAR IZ89514

  • AIX 6.1 TL 07 - use AIX APAR IZ88880

AIX 5L installations

If you are using a later TL level than the minimum level listed for this release, then check with IBM to determine if the required APARs listed here are included in the TL level that you have on your system. If they are included, then you do not need to install them. If they are not included, then you must install the equivalent APAR for the appropriate TL level.

If you are using the minimum operating system TL level for AIX 5L listed above, then install the following Authorized Problem Analysis Reports (APARs) for AIX 5L V5.3 TL 09 SP1:

IZ42940
IZ49516
IZ52331

These fixes are already present in the following TL levels:

  • AIX 5.3 TL 09 SP-05 and later

  • AIX 5.3 TL 10 SP-02 and later

  • AIX 5.3 TL 11

Oracle JDBC/OCI Drivers

AIX 5L v5.3

Note: These APARs are required only if you are using the associated JDK version.

APAR required for JDK 1.4.2 (64-bit):

  • IY63533: JDK 1.4.2 64-bit SR1 caix64142-20040917

Oracle ODBC Drivers

You do not require ODBC drivers for Oracle Clusterware or Oracle Database.

To use ODBC with Oracle Database, you must also install gcc 3.4.5 or later.

Oracle ODBC driver on AIX is certified with ODBC Driver Manager 2.2.12. You can download and install the Driver Manager from the following link:

http://www.unixodbc.org


2.8 Checking the Software Requirements

To ensure that the system meets these requirements:

  1. To determine the version of AIX installed, enter the following command:

    # oslevel -r
    

    If the operating system version is lower than AIX 5.3, then upgrade your operating system to at least this maintenance level. AIX 5L version 5.3 maintenance packages are available from the following web site:

    http://www-933.ibm.com/support/fixcentral/

  2. To determine whether the required filesets are installed and committed, enter a command similar to the following:

    # lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.libperfstat \
    bos.perf.perfstat bos.perf.proctools rsct.basic.rte rsct.compat.clients.rte \
    xlC.aix61.rte
    

    If a fileset is not installed and committed, then install it. Refer to your operating system or software documentation for information about installing filesets.

To ensure that the system meets these requirements:

  1. To determine if required APARs are installed, enter a command similar to the following:

    # instfix -i -k "IZ41855 IZ51456 IZ52319"
    

    If an APAR is not installed, then download it from the following web site and install it:

    http://www-933.ibm.com/support/fixcentral/
    
  2. To determine whether a PTF is installed, enter a command similar to the following:

    # lslpp -l -B U489726 U485561 ...
    

    If a PTF is not installed, then download it from the following web site and install it:

    http://www-933.ibm.com/support/fixcentral/
    
  3. If you require a CSD for WebSphere MQ, then refer to the following web site for download and installation information:

    http://www-01.ibm.com/software/
    

2.9 Verifying UDP and TCP Kernel Parameters

Use NDD to ensure that the AIX kernel TCP/IP ephemeral port range is broad enough to provide enough ephemeral ports for the anticipated server workload. Ensure that the lower range is set to at least 9000 or higher, to avoid Well Known ports, and to avoid ports in the Registered Ports range commonly used by Oracle and other server ports. Set the port range high enough to avoid reserved ports for any applications you may intend to use. If the lower value of the range you have is greater than 9000, and the range is large enough for your anticipated workload, then you can ignore OUI warnings regarding the ephemeral port range.

For example:

# /usr/sbin/no -a | fgrep ephemeral
       tcp_ephemeral_low = 32768
       tcp_ephemeral_high = 65535
       udp_ephemeral_low = 32768
       udp_ephemeral_high = 65535

In the preceding example, the TCP and UDP ephemeral ports are set to the default range (32768-65536).

If you expect your workload to require a high number of ephemeral ports, then update the UDP and TCP ephemeral port range to a broader range. For example:

# /usr/sbin/no -p -o tcp_ephemeral_low=9000 -o tcp_ephemeral_high=65500
# /usr/sbin/no -p -o udp_ephemeral_low=9000 -o udp_ephemeral_high=65500

2.10 Checking Resource Limits for AIX

On AIX platforms, the /etc/security/login.cfg file controls and limits resources for users on the system. On login, control and limit resources should be set for users on the system so that authentication is carried out through the pam.conf file, to prevent denial of service attacks.

By default, resource limits are set to std_auth for AIX operating systems. Edit the file /etc/security/login.cfg file to change auth_type under the usw stanza from std_auth to pam_auth.

To ensure that these resource limits are honored, confirm that the line login session required /usr/lib/security/pam_aix is set in /etc/pam.conf. For example:

dtsession auth  required        /usr/lib/security/pam_aix
dtlogin session required        /usr/lib/security/pam_aix
ftp     session required        /usr/lib/security/pam_aix
imap    session required        /usr/lib/security/pam_aix
login   session required        /usr/lib/security/pam_aix
rexec   session required        /usr/lib/security/pam_aix
rlogin  session required        /usr/lib/security/pam_aix
rsh     session required        /usr/lib/security/pam_aix
snapp   session required        /usr/lib/security/pam_aix
su      session required        /usr/lib/security/pam_aix
swrole  session required        /usr/lib/security/pam_aix
telnet  session required        /usr/lib/security/pam_aix
xdm     session required        /usr/lib/security/pam_aix
OTHER   session required        /usr/lib/security/pam_prohibit
websm_rlogin    session    required     /usr/lib/security/pam_aix
websm_su        session    required     /usr/lib/security/pam_aix
wbem       session       required    /usr/lib/security/pam_aix

2.11 Tuning AIX System Environment

Perform the following system tuning and configuration all cluster nodes.

Note:

The parameter and shell limit values shown in this section are recommended values only. For production database systems, Oracle recommends that you tune these values to optimize the performance of the system. See your operating system documentation for more information about tuning kernel parameters.

2.11.1 Checking Asynchronous Input Output Processes

On AIX 5, run the rootpre.sh script to enable the Asynchronous Input Output (AIO) device drivers. On AIX 6, the AIO device drivers are enabled by default. For both AIX 5 and AIX 6, increase the number of aioserver processes from the default value. The recommended value for aio_maxreqs is 64k (65536). Confirm this value for both AIX 5 and AIX 6.

Confirm the aio_maxreqs value using the procedure for your release:

AIX 6.1 and 7.1:

# ioo -o aio_maxreqs
aio_maxreqs = 65536

AIX 5.3:

# lsattr -El aio0 -a maxreqs
maxreqs 65536 Maximum number of REQUESTS True

When performing an asynchronous I/O to a file system, note that each asynchronous I/O operation is tied to an asynchronous I/O server. Thus, the number of asynchronous I/O servers limits the number of concurrent asynchronous I/O operations in the system.

The initial number of servers that are started during a system restart is determined by the minservers parameter. As concurrent asynchronous I/O operations occur, additional asynchronous I/O servers are started, up to a maximum of the value set in the maxservers parameter.

On AIX 5.3, if you are using Oracle Database with data files on a file system, then increase the default values for minservers and maxservers, as the default values for these parameters are too small. Increase the minservers and maxservers values based on I/O kprocs for each processor.

In general, to set the number of asynchronous I/O servers, complete the following procedure:

  1. Adjust the initial value of maxservers to 10 times the number of disks divided by the number of CPUs that are to be used concurrently but no more than 80.

  2. Monitor the performance effects on the system during periods of high I/O activity. If all AIO server processes are started, then increase the maxservers value. Also, continue to monitor the system performance during peak I/O activity to determine if there was a benefit from the additional AIO servers. Too many asynchronous I/O servers increase memory and processor overload of additional processes, but this disadvantage is small. See your operating system vendor documentation for information about tuning AIO parameters.

To monitor the number of AIO server processes that have started, enter the following command:

# ps -ek|grep -v grep|grep –v posix_aioserver|grep -c aioserver

Note:

Starting with AIX 6.1, minservers and maxservers are replaced by the aio_minservers and aio_maxservers parameters respectively.

2.11.2 Tuning Virtual Memory Manager (VMM)

Oracle recommends that you use the vmo command to tune virtual memory using the following values:

Table 2-5 Recommended Values for Virtual Memory Manager

Parameter Value

minperm%

3 (AIX 5.3 default is 20)

maxperm%

90 (AIX 5.3 default is 80)

maxclient% = 90

90 (AIX 5.3 default is 80)

lru_file_repage

0 (AIX 5.3 default is 1)

strict_maxclient

1 (AIX 5.3 default is 1)

strict_maxperm

0 (AIX 5.3 default is 0)


For example:

vmo -p -o minperm%=3
vmo -p -o maxperm%=90
vmo -p -o maxclient%=90
vmo -p -o lru_file_repage=0
vmo -p -o strict_maxclient=1
vmo -p -o strict_maxperm=0

You must restart the system for these changes to take effect.

2.11.3 Increasing System Block Size Allocation

Oracle recommends that you increase the space allocated for ARG/ENV list to 128. The size is specified by number of 4K blocks.

For example:

/usr/sbin/chdev -l sys0 -a ncargs='128'

2.11.4 Configuring SSH LoginGraceTime Parameter for AIX

On AIX systems, the OpenSSH parameter LoginGraceTime by default is commented out, and the default behavior of OpenSSH on AIX can sometimes result in timeout errors. To avoid these errors, complete the following procedure:

  1. Log in as root.

  2. Using a text editor, open the OpenSSH configuration file /etc/ssh/sshd_config.

  3. Locate the comment line #LoginGraceTime 2m.

  4. Uncomment the line, and change the value to 0 (unlimited). For example:

    LoginGraceTime 0
    
  5. Save /etc/ssh/sshd_config.

  6. Restart SSH.

2.11.5 Configuring Shell Limits

Set shell limits for the Oracle Grid Infrastructure installation owner and for root. Verify that unlimited is set for both accounts either by using the smit utility or by editing the /etc/security/limits file. The root user requires these settings because the crs daemon (crsd) runs as root.

Shell Limit Item in limits.conf Hard Limit
Maximum number of open file descriptors nofile -1 (unlimited
Maximum number of processes available to a single user maxuproc 16384
Maximum size of the stack segment of the process stack -1 (unlimited)

To increase the shell limits:

  1. Add the following lines to the /etc/security/limits file:

    default:
            fsize = -1
            core = 2097151
            cpu = -1
            data = -1
            rss = -1
            stack = -1
            nofiles = -1
    
  2. Enter the following command to list the current setting for the maximum number of process allowed by the Oracle software user:

    /usr/sbin/lsattr -E -l sys0 -a maxuproc
    

    If necessary, change the maxuproc setting using the following command:

    /usr/sbin/chdev -l sys0 -a maxuproc=16384
    
  3. Repeat this procedure on all other nodes in the cluster.

    Caution:

    Use shell programs supported by your operating system vendor. If you use a shell program that is not supported by your operating system, then you can encounter errors during installation.

2.11.6 Configuring User Process Parameters

Verify that the maximum number of processes allowed for each user is set to 2048 or greater:

Note:

For production systems, this value should be at least 128 plus the sum of the PROCESSES and PARALLEL_MAX_SERVERS initialization parameters for each database running on the system.
  1. Enter the following command:

    # smit chgsys
    
  2. Verify that the value shown for Maximum number of PROCESSES allowed for each user is greater than or equal to 2048.

    If necessary, edit the existing value.

  3. When you have finished making changes, press Enter, then Esc+0 (Exit) to exit.

2.11.7 Configuring Network Tuning Parameters

Verify that the network tuning parameters shown in the following table are set to the values shown or higher values. The procedure following the table describes how to verify and set the values.

Network Tuning Parameter Recommended Value
ipqmaxlen 512
rfc1323 1
sb_max 4194304
tcp_recvspace 65536
tcp_sendspace 65536
udp_recvspace 655360

Note: The recommended value of this parameter is 10 times the value of the udp_sendspace parameter. The value must be less than the value of the sb_max parameter.

udp_sendspace 65536

Note: This value is suitable for a default database installation. For production databases, the minimum value for this parameter is 4 KB plus the value of the database DB_BLOCK_SIZE initialization parameter multiplied by the value of the DB_MULTIBLOCK_READ_COUNT initialization parameter:

(DB_BLOCK_SIZE * DB_FILE_MULTIBLOCK_READ_COUNT) + 4 KB


To view the current value specified for these parameters, and to change them if necessary:

  1. To check the current values of the network tuning parameters, enter commands similar to the following:

    # no -a | more
    
  2. If you must change the value of any parameter, then enter the following command to determine whether the system is running in compatibility mode:

    # lsattr -E -l sys0 -a pre520tune
    

    If the system is running in compatibility mode, then the output is similar to the following, showing that the value of the pre520tune attribute is enabled:

    pre520tune enable Pre-520 tuning compatibility mode True
    
  3. If the system is running in compatibility mode, then follow these steps to change the parameter values:

    1. Enter commands similar to the following to change the value of each parameter:

      # no -o parameter_name=value
      

      For example:

      # no -o udp_recvspace=655360
      
    2. Add entries similar to the following to the /etc/rc.net file for each parameter that you changed in the previous step:

      if [ -f /usr/sbin/no ] ; then
         /usr/sbin/no -o udp_sendspace=65536
         /usr/sbin/no -o udp_recvspace=655360
         /usr/sbin/no -o tcp_sendspace=65536
         /usr/sbin/no -o tcp_recvspace=65536
         /usr/sbin/no -o rfc1323=1
         /usr/sbin/no -o sb_max=4194304
         /usr/sbin/no -o ipqmaxlen=512
      fi
      

      By adding these lines to the /etc/rc.net file, the values persist when the system restarts.

  4. If the system is not running in compatibility mode, then enter commands similar to the following to change the parameter values:

    • ipqmaxlen parameter:

      /usr/sbin/no -r -o ipqmaxlen=512
      
    • Other parameter:

      /usr/sbin/no -p -o parameter=value
      

    Note:

    If you modify the ipqmaxlen parameter, then you must restart the system.

    These commands modify the /etc/tunables/nextboot file, causing the attribute values to persist when the system restarts.

  5. If you need to change parameters, and you do not restart your system, then use the ifconfig command to check each network parameter after you change the no global setting.

    For example:

    # ifconfig en0
    en0:
    flags=1e080863,2c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,6
    4BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN,MONITOR>
            inet 192.0.2.1 netmask 0xfffff800 broadcast 192.0.2.0
            inet 192.0.2.2 netmask 0xfffff800 broadcast 192.0.2.0
            inet 192.0.2.3 netmask 0xfffff800 broadcast 192.0.2.0
            inet 192.0.2.4 netmask 0xfffff800 broadcast 192.0.2.0
             tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0 
    

    For the ISNO parameter tcp_sendspace, use the following command to set it:

    # ifconfig en0 tcp_sendspace 65536
    

    See Also:

    The IBM documentation for more information about tuning AIX parameters for your system.

2.12 Network Time Protocol Setting

Oracle Clusterware requires the same time zone setting on all cluster nodes. During installation, the installation process picks up the time zone setting of the Grid installation owner on the node where OUI runs, and uses that on all nodes as the default TZ setting for all processes managed by Oracle Clusterware. This default is used for databases, Oracle ASM, and any other managed processes.

You have two options for time synchronization: an operating system configured network time protocol (NTP), or Oracle Cluster Time Synchronization Service. Oracle Cluster Time Synchronization Service is designed for organizations whose cluster servers are unable to access NTP services. If you use NTP, then the Oracle Cluster Time Synchronization daemon (ctssd) starts up in observer mode. If you do not have NTP daemons, then ctssd starts up in active mode and synchronizes time among cluster members without contacting an external time server.

Note:

Before starting the installation of Oracle Grid Infrastructure, Oracle recommends that you ensure the clocks on all nodes are set to the same time.

If you have NTP daemons on your server but you cannot configure them to synchronize time with a time server, and you want to use Cluster Time Synchronization Service to provide synchronization service in the cluster, then deactivate and deinstall the NTP.

To disable the NTP service, run the following command as the root user

# stopsrc -s xntpd

When the installer finds that the NTP protocol is not active, the Cluster Time Synchronization Service is installed in active mode and synchronizes the time across the nodes. If NTP is found configured, then the Cluster Time Synchronization Service is started in observer mode, and no active time synchronization is performed by Oracle Clusterware within the cluster.

To confirm that ctssd is active after installation, enter the following command as the Grid installation owner:

$ crsctl stat resource ora.ctssd -t -init

If you are using NTP, and you prefer to continue using it instead of Cluster Time Synchronization Service, then you need to modify the NTP initialization file to enable slewing, which prevents time from being adjusted backward. Restart the network time protocol daemon after you complete this task.

To do this on AIX, configure the XNTP daemon to start at each system restart by editing the file /etc/rc.tcpip:

  1. Open the /etc/rc.tcpip file, and locate the following line:

    start /usr/sbin/xntpd "$src_running"
    
  2. Change the line to the following:

    start /usr/sbin/xntpd "$src_running" "-x"
    
  3. Save the file.

To enable XNTP after it has been disabled, enter the following command on each cluster member node:

# startsrc -s xntpd -a "-x"

2.13 Automatic SSH Configuration During Installation

To install Oracle software, Secure Shell (SSH) connectivity should be set up between all cluster member nodes. OUI uses the ssh and scp commands during installation to run remote commands on and copy files to the other cluster nodes. You must configure SSH so that these commands do not prompt for a password.

Note:

SSH is used by Oracle configuration assistants for configuration operations from local to remote nodes. It is also used by Oracle Enterprise Manager.

You can configure SSH from the Oracle Universal Installer (OUI) interface during installation for the user account running the installation. The automatic configuration creates passwordless SSH connectivity between all cluster member nodes. Oracle recommends that you use the automatic procedure if possible.

To enable the script to run, you must remove stty commands from the profiles of any Oracle software installation owners, and remove other security measures that are triggered during a login, and that generate messages to the terminal. These messages, mail checks, and other displays prevent Oracle software installation owners from using the SSH configuration script that is built into the Oracle Universal Installer. If they are not disabled, then SSH must be configured manually before an installation can be run.

See Also:

Section 2.14.5, "Preventing Installation Errors Caused by Terminal Output Commands" for information about how to remove stty commands in user profiles

By default, OUI searches for SSH public keys in the directory /usr/local/etc/, and ssh-keygen binaries in /usr/local/bin. However, on AIX, SSH public keys typically are located in the path /etc/ssh, and ssh-keygen binaries are located in the path /usr/bin. To ensure that OUI can set up SSH, use the following command to create soft links:

# ln -s /etc/ssh /usr/local/etc
# ln -s /usr/bin /usr/local/bin

In rare cases, Oracle Clusterware installation may fail during the "AttachHome" operation when the remote node closes the SSH connection. To avoid this problem, set the following parameter in the SSH daemon configuration file /etc/ssh/sshd_config on all cluster nodes to set the timeout wait to unlimited:

LoginGraceTime 0

2.14 Configuring Grid Infrastructure Software Owner User Environments

You run the installer software with the Oracle Grid Infrastructure installation owner user account (oracle or grid). However, before you start the installer, you must configure the environment of the installation owner user account. Also, create other required Oracle software owners, if needed.

This section contains the following topics:

2.14.1 Environment Requirements for Oracle Grid Infrastructure Software Owner

You must make the following changes to configure the Oracle Grid Infrastructure software owner environment:

  • Set the installation software owner user (grid, oracle) default file mode creation mask (umask) to 022 in the shell startup file. Setting the mask to 022 ensures that the user performing the software installation creates files with 644 permissions.

  • Set ulimit settings for file descriptors and processes for the installation software owner (grid, oracle)

  • Set the software owner's environment variable DISPLAY environment variables in preparation for the Oracle Grid Infrastructure installation

2.14.2 Environment Requirements for Oracle Database and Oracle ASM Owners

If you intend to install Oracle Database or Oracle ASM, then complete the following additional tasks. If you plan to install other software using the role-based privileges method, then complete the following tasks for the Oracle Database software owner (oracle) and Oracle ASM software owner (asm).

  • Create an Oracle Base path. The Optimal Flexible Architecture path for the Oracle Base is /u01/app/user, where user is the name of the user account that you want to own the Oracle Database software. For example: /u01/app/oracle.

    Note:

    Do not create the Oracle Clusterware home under Oracle base. Creating an Oracle Clusterware installation in an Oracle base directory path will cause succeeding Oracle installations to fail.
  • Set the installation software owner user (asm, oracle) default file mode creation mask (umask) to 022 in the shell startup file. Setting the mask to 022 ensures that the user performing the software installation creates files with 644 permissions.

  • Set the software owners' environment variable DISPLAY environment variables in preparation for the Oracle ASM or Oracle Database installation

2.14.3 Procedure for Configuring Oracle Software Owner Environments

To set the Oracle software owners' environments, follow these steps, for each software owner (grid, oracle):

  1. Start a new terminal session; for example, start an X terminal (xterm).

  2. Enter the following command to ensure that X Window applications can display on this system:

    $ xhost + hostname
    

    The hostname is the name of the local host.

  3. If you are not already logged in to the system where you want to install the software, then log in to that system as the software owner user.

  4. If you are not logged in as the user, then switch to the software owner user you are configuring. For example, with the grid user:

    $ su - grid
    
  5. To determine the default shell for the user, enter the following command:

    $ echo $SHELL
    
  6. Open the user's shell startup file in any text editor:

    • Bourne shell (sh) or Korn shell (ksh):

      % vi .profile
      
    • C shell (csh or tcsh):

      % vi .login
      
  7. Enter or edit the following line, specifying a value of 022 for the default file mode creation mask:

    umask 022
    
  8. If the ORACLE_SID, ORACLE_HOME, or ORACLE_BASE environment variable is set in the file, then remove the appropriate lines from the file.

  9. Save the file, and exit from the text editor.

  10. To run the shell startup script, enter one of the following commands:

    • Bourne, Bash, or Korn shell:

      $ . ./.profile
      
    • C shell:

      % source ./.login
      
  11. If you are not installing the software on the local system, then enter a command similar to the following to direct X applications to display on the local system:

    • Bourne, Bash, or Korn shell:

      $ DISPLAY=local_host:0.0 ; export DISPLAY
      
    • C shell:

      % setenv DISPLAY local_host:0.0
      

    In this example, local_host is the host name or IP address of the system that you want to use to display OUI (your workstation or PC).

  12. If you determined that the /tmp directory has less than 1 GB MB of free disk space, then identify a file system with at least 1 GB of free space and set the TEMP and TMPDIR environment variables to specify a temporary directory on this file system:

    Note:

    You cannot use a shared file system as the location of the temporary file directory (typically /tmp) for Oracle RAC installation. If you place /tmp on a shared file system, then the installation fails.
    1. Use the df -k command to identify a suitable file system with sufficient free space.

    2. If necessary, enter commands similar to the following to create a temporary directory on the file system that you identified, and set the appropriate permissions on the directory:

      $ su - root
      # mkdir /mount_point/tmp
      # chmod 775 /mount_point/tmp
      # exit
      
    3. Enter commands similar to the following to set the TEMP and TMPDIR environment variables:

      • Bourne, Bash, or Korn shell:

        $ TEMP=/mount_point/tmp
        $ TMPDIR=/mount_point/tmp
        $ export TEMP TMPDIR
        
      • C shell:

        % setenv TEMP /mount_point/tmp
        % setenv TMPDIR /mount_point/tmp
        

2.14.4 Setting Display and X11 Forwarding Configuration

If you are on a remote terminal, and the local node has only one visual (which is typical), then use the following syntax to set the DISPLAY environment variable:

Bourne, Korn, and Bash shells

$ export DISPLAY=hostname:0

C shell:

$ setenv DISPLAY hostname:0

For example, if you are using the Bash shell, and if your host name is node1, then enter the following command:

$ export DISPLAY=node1:0

To ensure that X11 forwarding will not cause the installation to fail, create a user-level SSH client configuration file for the Oracle software owner user, as follows:

  1. Using any text editor, edit or create the software installation owner's ~/.ssh/config file.

  2. Make sure that the ForwardX11 attribute is set to no. For example:

    Host *
          ForwardX11 no
    

2.14.5 Preventing Installation Errors Caused by Terminal Output Commands

During an Oracle Grid Infrastructure installation, OUI uses SSH to run commands and copy files to the other nodes. During the installation, hidden files on the system (for example, .bashrc or .cshrc) will cause makefile and other installation errors if they contain stty commands.

To avoid this problem, you must modify these files in each Oracle installation owner user home directory to suppress all output on STDOUT or STDERR (for example, stty, xtitle, and other such commands) as in the following examples:

  • Bourne, Bash, or Korn shell:

    if [ -t 0 ]; then
       stty intr ^C
    fi
    
  • C shell:

    test -t 0
    if ($status == 0) then
       stty intr ^C
    endif
    

    Note:

    When SSH is not available, the Installer uses the rsh and rcp commands instead of ssh and scp.

    If there are hidden files that contain stty commands that are loaded by the remote shell, then OUI indicates an error and stops the installation.

2.15 Running the rootpre.sh Script

Note:

Do not run the rootpre.sh script if you have a later release of the Oracle Database software already installed on this system.

Run the rootpre.sh script:

  1. Switch user to root:

    $ su - root
    
  2. Complete one of the following steps, depending on the location of the installation

    If the installation files are on disc, enter a command similar to the following, where directory_path is the disc mount point directory or the path of the database directory on the DVD:

    # /directory_path/rootpre.sh
    

    If the installation files are on the hard disk, change directory to the Disk1 directory and enter the following command:

    # ./rootpre.sh
    
  3. Exit from the root account:

    # exit
    
  4. Repeat steps 1 through 3 on all nodes of the cluster.

    Note:

    Do not run the rootpre.sh script if you have a later release of Oracle Database software already installed on this system.

2.16 Adding the Grid Infrastructure Installation Owner to Hagsuser Group

If you have HACMP installed on your system, then the rootpre.sh command creates the hagsuser group. You must make the grid infrastructure installation owner (grid) a member of that group.

Enter commands similar to the following to add the Oracle Grid Infrastructure owner (grid) to haguser

# /usr/bin/chgrpmem -m + grid hagsuser

After you add the Oracle Grid Infrastructure installation owner to the hagsuser group, stop and restart HACMP before trying to use it with Oracle Grid Infrastructure.

2.17 Requirements for Creating an Oracle Grid Infrastructure Home Directory

During installation, you are prompted to provide a path to a home directory to store Oracle Clusterware binaries. Ensure that the directory path you provide meets the following requirements:

  • It should be created in a path outside existing Oracle homes, including Oracle Clusterware homes.

  • It should not be located in a user home directory.

  • It should be created either as a subdirectory in a path where all files can be owned by root, or in a unique path.

  • If you create the path before installation, then it should be owned by the installation owner of Oracle Grid Infrastructure (typically oracle for a single installation owner for all Oracle software, or grid for role-based Oracle installation owners), and set to 775 permissions.

Oracle recommends that you install Oracle Grid Infrastructure on local homes, rather than using a shared home on shared storage.

For installations with Oracle Grid Infrastructure only, Oracle recommends that you create a path compliant with Oracle Optimal Flexible Architecture (OFA) guidelines, so that Oracle Universal Installer (OUI) can select that directory during installation. For OUI to recognize the path as an Oracle software path, it must be in the form u0[1-9]/app.

When OUI finds an OFA-compliant path, it creates the Oracle Grid Infrastructure and Oracle Inventory (oraInventory) directories for you.

To create an Oracle Grid Infrastructure path manually, ensure that it is in a separate path, not under an existing Oracle base path. For example:

# mkdir -p  /u01/app/11.2.0/grid
# chown grid:oinstall /u01/app/11.2.0/grid
# chmod -R 775 /u01/app/11.2.0/grid

With this path, if the installation owner is named grid, then by default OUI creates the following path for the Grid home:

/u01/app/11.2.0/grid

Create an Oracle base path for database installations, owned by the Oracle Database installation owner account. The OFA path for an Oracle base is /u01/app/user, where user is the name of the Oracle software installation owner account. For example, use the following commands to create an Oracle base for the database installation owner account oracle:

# mkdir -p  /u01/app/oracle
# chown -R oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/app/oracle

Note:

If you choose to create an Oracle Grid Infrastructure home manually, then do not create the Oracle Grid Infrastructure home for a cluster under either the grid installation owner Oracle base or the Oracle Database installation owner Oracle base. Creating an Oracle Clusterware installation in an Oracle base directory will cause succeeding Oracle installations to fail.

Oracle Grid Infrastructure homes can be placed in a local home on servers, even if your existing Oracle Clusterware home from a prior release is in a shared location.

Homes for Oracle Grid Infrastructure for a standalone server (Oracle Restart) can be under Oracle base. Refer to Oracle Database Installation Guide for your platform for more information about Oracle Restart.

2.18 Cluster Name Requirements

The cluster name must be at least one character long and no more than 15 characters in length, must be alphanumeric, cannot begin with a numeral, and may contain hyphens (-).

In a Typical installation, the SCAN you provide is also the name of the cluster, so the SCAN name must meet the requirements for a cluster name. In an Advanced installation, The SCAN and cluster name are entered in separate fields during installation, so cluster name requirements do not apply to the SCAN name.