5 Automated Multi-Node Patching of Grid Infrastructure and RAC DB Environment Using OPatchAuto

OPatchAuto automates patch application to the Grid Infrastructure (GI) cluster, by applying patches to both the GI and the managed Oracle Real Application Cluster (RAC) homes. OPatchAuto also applies these patches to all nodes of the cluster in one step when invoked with the -remote option. Use OPatchAuto to automate the necessary steps for applying a patch on a single host or multi-host environment.

To ensure successful patching, there are several prerequisites you should complete to prepare your environment for running OPatchAuto, such as obtaining the latest version of OPatch, obtaining required patches from My Oracle Support, and backing up the environment.

5.1 Preparing to Use OPatchAuto

To ensure successful patching, there are several prerequisites you should complete to prepare your environment for running OPatchAuto, such as obtaining the latest version of OPatch, obtaining required patches from My Oracle Support, and backing up the environment.

For more information on preparing your environment, see the following topics:

  • Patching Your Environment Using OPatchAuto

    OPatchAuto is installed with the OPatch utility as a part of your installation. OPatchAuto provides several commands that you can use to Patching Your Environment Using OPatchAuto automate the application and roll back of a patch in a single host or multi-host environment.

  • Locating and Obtaining the Latest Version of OPatch and OPatchAuto

    Before you run OPatchAuto, find the OPatchAuto utility in the Oracle home and verify that you have the latest version. You must have the latest version of OPatch in all the homes of all the nodes before you request for patching.

  • Obtaining Patches Required For Your Installation

    You can search for and download the latest patches for your installation from My Oracle Support.

  • OPatchAuto Environment Variables

    Before you run OPatchAuto, ensure that you set the required ORACLE_HOME environment variable.

  • Creating a Wallet to Store Passwords

    To successfully run OPatchAuto, you must provide a wallet on the command line that contains the necessary password credentials for connecting to the environment.

  • Configuring Node Manager to Support Start and Stop Operations

    To ensure that OPatchAuto can properly stop and start your system during patching, you must configure the Node Manager(s) to support the start and stop operations.

  • Backup and Recovery Considerations for Patching

    It is highly recommended that you back up the Oracle home before any patch operation. You can back up the Oracle home using your preferred method.

5.1.1 OPatchAuto Environment Variable

Before you run OPatchAuto, ensure that you set the required ORACLE_HOME environment variable. The ORACLE_HOME environment variable is used to identify the Oracle home you are planning to patch.

5.1.2 Oracle Wallet for Credential Input

OPatchAuto accepts credentials, in Oracle wallet format, for accessing run-time entities, such as databases and Admin Servers. A Wallet file contains credentials for the hosts which are part of the cluster that requires patching. You input a wallet on the command line; if you do not supply one, and OPatchAuto needs one, it will prompt you for one on the command line. Successful usage depends on the user possessing both the wallet and the wallet password.

5.1.2.1 OPatchAuto Wallet Tool

You can use the command line OPatchAuto wallet tool to generate a wallet file. The wallet file contains credentials for the hosts that are part of the cluster which needs to be patched. The wallet tool works seamlessly during the OPatchAuto patch orchestration process by passing the wallet file path as parameter during patching operations.

The command line tool is as follows:

<ORACLE_HOME>/OPatch/auto/core/bin/patchingWallet.[sh|cmd]
[-log log_file] [-log_priority log_priority]
{ -create | -delete | -list } alias1 alias2 ... 

Table 5-1 patchingWallet Command Options

Option Description

-create

(Required) Create secrets for each alias given on the command line. If a given alias already exists in the wallet, its secret is overwritten without warning.

-delete

(Required) Delete given aliases from the wallet. Aliases that do not exist in the wallet are ignored.

-list

(Required) List aliases defined in the wallet. The secrets associated with the aliases are not displayed. The alias command line arguments are ignored.

-walletDir

(Optional) The path to the wallet directory. If omitted, the default location, if defined, will be used. If wallet does not exist at the specified location, it will be created when then -create option is used.

-useStdin

(Optional) When creating aliases, specifies that the passwords should be read from STDIN rather than the console device. Passwords will be read in the order specified by the alias options, with one per line. There will be no prompt.

-log

(Optional) Name of the log file.

-log_priority

(Optional) The priority setting for the log file. Use a Java Logging Level string or a log4j priority string.

Valid Java logging values are off, severe, info, warning, config, fine, finer, finest, and all.

Valid log4j priority strings are debug, info, warn, error, and fatal. The priority string values correspond to the levels defined in the Level class.

More information about log4j priority strings can be found at the following Web site: http://logging.apache.org/log4j/docs/api/org/apache/log4j/Level.html.

5.1.3 Remote Host Execution using SSH

Because OPatchAuto executes across all hosts in an associated system, SSH is required for Oracle Grid Infrastructure patching on all hosts. You must supply the SSH credentials to permit this execution. This can be done by supplying a wallet with the SSH credentials for one or more hosts (required for multi-host patching).

On Linux systems, you can check if SSH is installed on each node by running the following command:

rpm –qa | grep ssh.

Tip: For understanding the basic principles of SSH equivalence, try the Web search “generating SSH keys" or “SSH-keygen" to understand how SSH manages the keys that enable SSH authentication.

5.1.4 Backup and Recovery Considerations for Patching

It is highly recommended that you back up the Oracle home before any patch operation. You can back up the Oracle home using your preferred method.

You can use any method such as zip, cp -r, tar, and cpio to compress the Oracle home.

If the Oracle home does not appear when you execute the opatch lsinventory - detail command, the Oracle home might be missing from the Central Inventory, or the Central Inventory itself could be missing or corrupted.

If the Oracle home is listed when you execute the opatch lsinventory -detail command, but the products and components within the Oracle home are not listed, the inventory within the Oracle home (local inventory) might be missing or corrupted.

If the local inventory is corrupted or lost for some reason, you must restore the entire Oracle home if it was backed up. If a backup does not exist, you may have to reinstall the software.

To ensure successful patching, there are several prerequisites you should complete to prepare your environment for running OPatchAuto, such as obtaining the latest version of OPatch, obtaining required patches from My Oracle Support, and backing up the environment.

5.2 Using OPatchAuto to Patch a GI/RAC Environment

Use OPatchAuto to automate the necessary steps for applying a patch on a single host or multi-host environment.

The following topics describe how to use OPatchAuto to patch a GI/RAC Environment.

5.2.1 Summary of the Steps For Using OPatchAuto

Applying a patch with OPatchAuto involves a series of steps that must be performed to ensure successful patching.

The following table summarizes the typical steps required to patch your existing GI/RAC environment using OPatchAuto.

Table 5-2 Using OPatchAuto

Task Description Documentation

Acquire patches required for your installation

Log in, search for, and download the patches required for your specific installation.

You do not need to worry about whether OPatchAuto supports a particular patch type. If OPatchAuto does not support a particular patch type, you will be notified when you run the tool.

Obtaining the Patches You Need

Review the README.txt file for the patch.

Each patch archive includes a README file that contains important information and instructions that must be followed prior to applying your patch. It is important to review the README file because it provides any unique steps or other information specific to the patch.

The README.txt file that is packaged within the patch archive.

Check for patch prerequisites.

The OPatchAuto apply -analyze command will identify that the prerequisites for the patch have been met.

If you are patching a single host environment, see Verifying the Prerequisites for Applying a Patch on a Single Host.

If you are patching a multi-host environment, see Verifying the Prerequisites for Applying a Patch on Multiple Hosts.

Apply the patch.

After you determine the Oracle home to which you need to apply the patch, and you have read the README file, then you should apply the patch with the opatchauto apply command.

If you are patching a multi-host environment, see Applying a Patch on Multiple Hosts Using the Apply Command

Verify the patch was applied to the Oracle home successfully.

The OPatch lsinventory command will show what patches have been applied to the Oracle home. .

Using the OPatch lsinventory Command to Verify the Patches Applied to an Oracle Home

Verify that your software runs properly after you apply the patch.

After the patching is complete and your servers are restarted, you should check your product software to verify that the issue has been resolved.

Verifying Your Installation After Applying a Patch

Troubleshoot the application of a patch.

If there are problems applying a patch, your first troubleshooting task is to review the log file for the OPatchAuto session.

Troubleshooting a Patch by Viewing the OPatchAuto Log File

Roll back the application of a patch

If for some reason the result is not satisfactory, you can use the opatchauto rollback command to remove the patch from the Oracle home.

If additional assistance is required, go to My Oracle Support (formerly OracleMetaLink).

For a single host environment, see Rolling Back a Patch You Have Applied on a Single Host.

For a multi-host environment, see Rolling Back a Patch You Have Applied on Multiple Hosts.

5.2.2 Applying a Patch on a GI/RAC System Using OPatchAuto

After you obtain the necessary patches, use the opatchauto apply -analyze command to verify prerequisites. If needed, you can use opatchauto rollback to roll back the application of a patch.

Note:

In OPatch 11.2.0.3.13, while calling ‘opatch auto’ and in OPatch 12.1 and later, while calling ‘opatchauto’ with -oh option, ensure that the current directory is one of the homes specified in -oh.

Note:

From DB 12.2.0.2.0 onwards default mode of opatchauto patching will be out of place patching. To patch in inplace mode, -inplace has to be passed along with opatchauto command. DB versions prior to 12.2.0.2 will continue with inplace as default mode.

Note:

During apply/rollback of patches, opatchauto needs to query home information from all the nodes hence, you must install the same version of Opatch on homes in all the nodes before you start patching.

Patching a multi-host environment with OPatchAuto involves the following tasks:

Using OPatchAuto to Patch Oracle

Use OPatchAuto to automate the necessary steps for applying a patch on a single host or multi-host environment.

Verifying the Prerequisites for Applying a Patch on Multiple Hosts

To ensure successful patching, use the opatchauto apply -analyze command to check for any prerequisites.

Applying a Patch on Multiple Hosts Using the Apply Command

To apply a patch on multiple hosts, use the opatchauto apply -remote command. This is the same command as opatchauto apply -analyze, except you remove the -analyze argument when you are ready to apply the patch.

Rolling Back a Patch You Have Applied on Multiple Hosts

If you apply a patch and the results are not satisfactory, use the opatchauto rollback command to roll back the application of a patch. Use the opatchauto rollback -analyze -remote command to analyze the patches before rolling back and then call opatchauto rollback -remote to rollback patches from oracle homes in all the host.

5.2.2.1 Verifying the Prerequisites for Applying a Patch on Multiple Hosts

To ensure successful patching, use the opatchauto apply -analyze —remote command to check for any prerequisites.

To verify that a patch can be applied to a specific Oracle home (ORACLE_HOME) on multiple hosts, use the -analyze argument to the OPatchAuto apply command

opatchauto apply PATCH_HOME -remote -analyze -remote -wallet wallet_location

For example:

opatchauto apply /home/oracle/patches/15941858 -remote -analyze -wallet /tmp/samplewallet 

If you want to apply multiple patches in one session, use the –phBaseDir option.

This command displays the actions that will be taken by the patch, but does not actually apply the patch. As a result, it allows you to verify that the prerequisites for the patch have been met.

If any prerequisite checks fail, refer to the command output and log file to fix any issues before continuing. For example, a common failure is the detection of patch conflicts. If any patch conflicts occur, follow the instructions in the log file for how to obtain a merge patch from Oracle Support. You can also use -force-conflict and -skip-conflict option to resolve conflict. For more information see, apply

5.2.2.2 Applying a Patch on Multiple Hosts Using the Apply Command

To apply a patch on multiple hosts, use the opatchauto apply command. This is the same command as opatchauto apply -analyze —remote, except that you remove the -analyze argument when you are ready to apply the patch.

This topic shows an example of using the opatchauto apply command to apply a patch to an Oracle home environment on multiple hosts.

This example assumes that:

  • The patch you have downloaded has been saved to a directory that is named for the patch number in My Oracle Support. In this case, the patch number is 15941858.

  • The user runs the OPatchAuto command from the ORACLE_HOME/OPatch directory and includes the location of the patch (PATCH_HOME) as an argument to the command.

Note:

When you run the opatchauto apply command, make a note of the session id (for example, EKZR) in the command output. This will simplify the rollback process if you decide to roll back the patch later.

opatchauto apply PATCH_HOME -remote -wallet wallet_location

For example:

opatchauto apply /home/oracle/patches/15941858 -remote -wallet /tmp/samplewallet -walletPassword password

5.2.2.3 Rolling Back a Patch You Have Applied on Multiple Hosts

Use the opatchauto rollback -analyze -remote command to analyze the patches before rolling back and then call opatchauto rollback -remote to rollback patches from oracle homes in all the host.

To roll back a patch, you follow the same process for when you applied the patch. That is, you first do a test run of the opatchauto rollback command:

Note:

You can simplify the command if you provide the session id (for example, EKZR) that was used to apply the patch. Then, OPatchAuto can derive all the necessary command line parameters.

opatchauto rollback <PATCH_LOCATION> -remote -analyze –wallet wallet_location

For example:

opatchauto rollback <patch_location> -remote -analyze –wallet /tmp/samplewallet

When the test run successfully passes, perform the actual roll back of the patch:

opatchauto rollback <patch_location> -remote –wallet wallet_location

For example:

opatchauto rollback <patch_location> -remote –wallet /tmp/samplewallet

Alternatively, if you do not remember your session id when you applied the patch, you can roll back the patch by pointing OPatchAuto to a copy of the unzipped patch as follows:

opatchauto rollback <patch_location> -remote -analyze –wallet wallet_location

5.2.3 Using the OPatch lsinventory Command to Verify the Patches Applied to an Oracle Home

To verify what patches have been applied to an Oracle home, or to find out additional information about the Oracle home, use the opatch lsinventory command.

The following example shows sample output of the lsinventory command, which indicates that a specific interim patch has been applied.

Example 5-1 Running the opatch lsinventory Command to Obtain the Oracle Home Information

> opatch lsinventory
Oracle Interim Patch Installer version 13.3.0.0.0
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /opt/Oracle/products/fmw12c
Central Inventory : /opt/Oracle/oraInventory
from : /var/opt/Oracle/oraInst.loc
OPatch version : 13.3.0.0.0
OUI version : 13.3.0.0.0
Log file location : /opt/Oracle/products/fmw12c/cfgtoollogs/opatch/
opatch2013-06-10_12-32-37PM_1.log
OPatch detects the Middleware Home as "/opt/Oracle/products/fmw12c"
Lsinventory Output file location :
/opt/Oracle/products/fmw12c/cfgtoollogs/opatch/lsinv/
lsinventory2013-06-10_12-32-37PM.txt
--------------------------------------------------------------------------------
Interim patches (1) :
Patch 15941858 : applied on Mon Jun 10 12:39:07 PDT 2013
Unique Patch ID: 150220
Patch description: "TEST PATCH FOR WLS 12.2.1.0 - JAVA CLASSES PATCH"
Created on 17 May 2013, 11:54:20 hrs PST8PDT
Bugs fixed:
783169, 15941850
--------------------------------------------------------------------------------
OPatch succeeded.

5.2.4 Verifying Your Installation After Applying a Patch

To verify that your installation in an Oracle GI/RAC environment, check that both GI and your RAC databases are back up and running using the following commands:

For GI:

crsctl check status crs

For RAC:

srvctl status database –d <database>

If the software does not work as expected, follow the rollback instructions to roll back the application of the patch.