7 Recovery and Troubleshooting

The OPatch utility is a tool that allows the application and rollback of interim patches to Oracle products. This chapter provides information on using OPatch to apply patches.This chapter includes the following topics:

Logging and Tracing
Restoring Oracle Homes
Recovering from a Failed Patching Session Using OPatch
Troubleshooting OPatch

Logging and Tracing

Logging and tracing is a common aid for debugging. OPatch maintains logs for all apply, rollback, and lsInventory operations. The log files are located in the <ORACLE_HOME>/cfgtoollogs/opatch directory. Each log file is tagged with the timestamp of the operation. Log files are named as opatch_<date mm-dd-yyyy>_<time hh-mm-ss>.log. A new log file is created each time OPatch is executed.

For example, if a log file is created on May 17th, 2009 at 11.55 PM, it will be named as follows:

opatch_05-17-2009_23-55-00.log

Note:

You can set OPatch to debug mode by setting the environment variable OPATCH_DEBUG to TRUE.

OPatch also maintains an index of the commands executed with OPatch and the log files associated with it in the history.txt file located in the <ORACLE_HOME>/cfgtoollogs/opatch directory. A sample of the history.txt file is as follows:

Date & Time : Tue Apr 26 23:00:55 PDT 2008
Oracle Home : /private/oracle/product/10.2.0/db_1/
OPatch Ver. : 10.2.0.4.6
Current Dir : /scratch/oui/OPatch
Command     : lsinventory
Log File    : 
/private/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch-2008_Apr_26_23-00-55-PDT_Tue.log

OPatch follows the Oracle Diagnostic Logging (ODL) Guidelines. You can set the log level by using the available -logLevel <level> option. This option controls the amount of logging OPatch performs according to the ODL guidelines.

OPatch supports the following log levels:

SEVERE
WARNING
INFO
CONFIG
FINE
FINER
FINEST

Restoring Oracle Homes

Every time you apply a patch, you make changes to your inventory. Sometimes that change may corrupt the inventory. You can use the restore.sh or restore.bat script that comes with OPatch to remove any changes that were made to the inventory after the patch application.

When you apply a patch, OPatch creates a snapshot of your inventory and stores it in $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory. OPatch 10.2 supports maintaining versions of patches. You can have two or more different versions of the same patch (with the same patch ID). This version information is stored in the OPatch metadata.

If your inventory is corrupted, you need to perform the following steps to return the application to its last known good state:

Ensure that the environment variable ORACLE_HOME is set properly.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) as follows:
```
/bin/sh make.txt
```

Recovering from a Failed Patching Session Using OPatch

During patching, updates can occur in two phases:

System Update — In this phase, the files are replaced in the Oracle home.
Inventory Update — In this phase, the details of the patch applied is recorded in the inventory.

The following scenarios explain how can you recover from a failed patching session:

The ORACLE_HOME does not appear when you execute opatch lsinventory -detail.: Cause: The ORACLE_HOME might be missing from the Central Inventory or the Central Inventory itself could be missing or corrupted.; Action: Ensure that the ORACLE_HOME is in the Central Inventory and/or the Central Inventory is not missing or corrupted.

The ORACLE_HOME is listed when you execute opatch lsinventory -detail, but the products and components within the ORACLE_HOME are not listed.: Cause: The inventory within the ORACLE_HOME (local inventory) could be missing or corrupted.; Action: If the local inventory is corrupted or lost for any reason, just restore the ORACLE_HOME/inventory if it had been backed up. If a back-up does not exist, you may have to reinstall the software.

When you apply or rollback a patch, you receive an interim inventory update exception.

Cause: This occurs when the files on the system are patched, but the inventory update has failed. This may be due to a corrupted inventory.

Action: Perform the following steps:

Ensure that the environment variable ORACLE_HOME is set properly.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) as follows:
```
/bin/sh make.txt
```

When you apply or rollback a patch, you receive an OiiOneoffException exception.

Cause: This occurs when the files on the system are patched, but the inventory update has failed. This may be because the base component of the interim patch may not be present in the inventory.

Action: OPatch attempts to restore the Oracle home automatically and provides a message for this action. If OPatch does not provide a message stating that it has restored the Oracle home, perform the following steps:

Ensure that the environment variable ORACLE_HOME is set properly.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) as follows:
```
/bin/sh make.txt
```

When you apply a patch and execute opatch lsinventory, it does not return anything.

Cause: This may be due to losing all the patches applied before the application of the current patch, or the patches might not have been updated in the inventory.

Action: Perform the following steps:

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) as follows:
```
/bin/sh make.txt
```
If the files are properly patched, but the information is not updated in the inventory, execute the following command:
```
$ORACLE_HOME/OPatch/opatch apply -no_sysmod <Path_To_Patch>
```
Ensure that the patch has been applied and recorded properly in the inventory by executing the following command:
```
$ORACLE_HOME/OPatch/opatch lsinventory -detail
```
If the files are still not patched properly, but you are able to see the patch in the lsinventory flag, you need to reapply the patch using the no_inventory flag:
```
$ORACLE_HOME/OPatch/opatch apply -no_inventory <Path_To_Patch>
```

When you apply a patch and execute opatch lsinventory, it does not return the details of the patch applied.

Cause: This may be because OPatch may not have recorded the details of this patch in the inventory.

Action: Perform the following steps:

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) as follows:
```
/bin/sh make.txt
```
If the files are properly patched, but the information is not updated in the inventory, execute the following command:
```
$ORACLE_HOME/OPatch/opatch apply -no_sysmod <Path_To_Patch>
```
Ensure that the patch has been applied and recorded properly in the inventory by executing the following command:
```
$ORACLE_HOME/OPatch/opatch lsinventory -detail
```

When you press Ctrl + C during the application or rollback of a patch and execute opatch lsinventory, it does not return the details of the patch applied or rolled back.

Cause: This may be because OPatch might have stopped the application or rollback of the patch upon pressing Ctrl + C.

Action: Perform the following steps:

Ensure that the environment variable ORACLE_HOME is set properly.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command if it is available:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) as follows:
```
/bin/sh make.txt
```

When you apply a patch, the process quit when OPatch failed to relink and prompted to continue.

Cause: This may be due to relink failure.

Action: Perform the following steps:

Ensure that the environment variable ORACLE_HOME is set properly.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command.

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

Resolve the re-link failure issue by ensuring that you are able to invoke make manually on a UNIX shell. After this, apply the patch again.

Oracle Real Application Clusters (RAC) Setup

When I apply a patch on an Oracle RAC setup, and execute 'opatch lsinventory' on the local node, the patch is not listed.

Cause: This may occur if OPatch failed to update the inventory.

Action: Perform the following steps:

Ensure that the environment variable ORACLE_HOME is set properly in all the nodes of the cluster.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory of each node in the cluster and execute the restore command as follows:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) in each node of the cluster as follows:
```
/bin/sh make.txt
```
Apply the patch in each node in the cluster using the local flag:
```
$ORACLE_HOME/OPatch/opatch apply -local <Path_To_Patch>
```
Note:
Ensure that all the nodes use the same OPatch version.

When I apply a patch on an Oracle RAC setup and execute 'opatch lsinventory' on the local node, it does not return anything.

Cause: You might have lost all the patches applied earlier.

Action: Perform the following steps:

Ensure that the environment variable ORACLE_HOME is set properly in each node in the cluster.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command in each node in the cluster.

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) in each node as follows:
```
/bin/sh make.txt
```
Apply the patch in each node using the local flag:
```
$ORACLE_HOME/OPatch/opatch apply -local <Path_To_Patch>
```
Note:
Ensure that all the nodes use the same OPatch version.

When I rollback a patch on an Oracle RAC setup and execute 'opatch lsinventory' on the local node, it shows that the patch was not removed.

Cause: This may occur if OPatch failed to update the inventory.

Action: Perform the following steps:

Ensure that the environment variable ORACLE_HOME is set properly in each node in the cluster.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory in each node in the cluster and execute the restore command as follows:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) in each node in the cluster as follows:
```
/bin/sh make.txt
```
Roll back the patch in all the nodes in the cluster using the local flag:
```
$ORACLE_HOME/OPatch/opatch rollback -local -id <Patch_ID>
```
Note:
Ensure that all the nodes use the same OPatch version.

When I roll back a patch on an Oracle RAC setup and execute 'opatch lsinventory' on the local node, it does not return anything.

Cause: You might have lost all the patches applied earlier.

Action: Perform the following steps:

Ensure that the environment variable ORACLE_HOME is set properly in each node in the cluster.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command in each node in the cluster:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) as follows:
```
/bin/sh make.txt
```

Roll back the patch in the local node using the local flag:

$ORACLE_HOME/OPatch/opatch rollback -local -id <Patch_ID>

Roll back the patch on the other nodes also using local flag.

Note:
Ensure that all the nodes use the same OPatch version.

When I apply a patch on an Oracle RAC setup, the patching in one node is fine (both the files and the inventory are fine), but I am not sure about the other nodes.

Cause: This may be due to a failed system or inventory update.

Action: Perform the following steps:

Copy the Oracle home from the node that is fine to the other nodes.
After copying the Oracle home, make sure that the ORACLE_HOME/inventory/ContentsXML/comps.xml file has the latest time stamp.

Note:
On Unix, use touch to change the time stamp.
Update the nodes of the cluster. For more information on updating the nodes of the cluster, refer to the Oracle Universal Installer User's Guide.
Ensure that all the prerequisite checks listed in Chapter 3, "OPatch Prerequisite Checks" pass.

When I apply a patch on an Oracle RAC setup, the patching in one node is fine, but when I execute 'opatch lsinventory' on the other nodes, the patch is not listed.

Cause: This may be due to a failed system or inventory update.

Action: Perform the following steps:

Copy the ORACLE_HOME /inventory directory from the node that is fine to the other nodes.
After copying the ORACLE_HOME /inventory directory, make sure that the ORACLE_HOME/inventory/ContentsXML/comps.xml file has the latest time stamp.

Note:
On Unix, use touch to change the time stamp.
Update the nodes of the cluster. For more information on updating the nodes of the cluster, see the Oracle Universal Installer User's Guide.
Ensure that all the prerequisite checks listed in the section Chapter 3, "OPatch Prerequisite Checks" pass.

When I apply or roll back a patch on an Oracle RAC setup, I am not able to apply or roll back the patch on all nodes.

Cause: This may occur if the nodes are not properly updated.

Action: Perform any one or more of the following:

Ensure that all the nodes in the cluster are up-to-date. If they are not, update the nodes of the cluster. For more information on updating the nodes of the cluster, see the Oracle Universal Installer User's Guide.

Execute the appropriate command on all nodes of the cluster as follows:

opatch apply -local [patch_location]

opatch rollback -local [patch_location]

Execute the appropriate command on the local node of the cluster as follows:

opatch apply [-local_node (node_name)] [-remote_nodes (comma-separated node_names)]

opatch rollback [-local_node (node_name)] [-remote_nodes (comma-separated node_names)]

Troubleshooting OPatch

This section provides solutions to errors that may occur during patch application.

Not a valid patch area
OPatch cannot find system commands like fuser and make
Unable to remove a partially-installed interim patch

Not a valid patch area

Cause: The directory that the OPatch utility is using to do the patch does not match the template for what it is checking. This can also occur when the utility is run from an invalid shiphome directory.

Action: When starting the OPatch utility, the directory needs the following:

/etc directory that has the metadata files.
/files directory that has the payload files.
The /etc/config/inventory file and the actions file under the same directory.

If you did not start the OPatch utility from the patch_id directory, you can use the following command:

opatch apply /<Patch_Shiphome>

OPatch cannot find system commands like fuser and make

Cause: The OPatch utility uses fuser on UNIX systems to check for active Oracle instances. On certain hp-ux systems, only a super-user can run fuser.

Action: Perform the following steps to resolve this problem:

Set /tmp in your PATH.

For more information, refer to "Check for System Commands" on page 3E2.
Create an empty file named fuser.
Shut down the Oracle instances.
Run the OPatch utility.

Caution:
Another way to resolve this problem is to give executable permission to other users for fuser. However, this exposes a potential security risk in the system and is not recommended.

Unable to remove a partially-installed interim patch

Cause: Interruption in the patching process is the potential cause for this problem. This may occur if you press CTRL+C during the patching process. If the error is the one that OPatch detects, it automatically resolves it.

Action: Perform the following steps:

Ensure that the environment variable ORACLE_HOME is set properly.

Navigate to the $ORACLE_HOME/.patch_storage/<patch-id_timestamp> directory and execute the restore command as follows:

For UNIX:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.sh

For Windows:
$ORACLE_HOME/.patch_storage/<patch-id_timestamp>/restore.bat

On UNIX, source the $ORACLE_HOME/.patch_storage/<patch-id_timestamp>/make.txt file (if available) as follows:
```
/bin/sh make.txt
```

Frequently Asked Questions

This section answers the frequently asked questions for OPatch.

When I apply a patch I get an error that reads "Failed to load the patch object." Possible causes are: The specified path is not an interim patch shiphome. Meta-data files are missing from the patch area ". What do I do?

This simply means the directory OPatch is using to find the patch doesn't match the template it is checking for. For more information on this error, see "Not a valid patch area" in section "Troubleshooting OPatch" on page 7E9.

When I apply a patch I get an error that reads "Syntax error.....Patch location not valid". What do I do?

This simply means that the patch location that you specify is an invalid one. Give the correct patch location and apply the patch again.

When I apply a patch I get an error that reads "Exception in thread "main" java.lang.NoClassDefFoundError: <Class Name>". What do I do?

This may be due to OPatch not able to find the particular class listed in the error, which is supposed to be located inside $ORACLE_HOME/OPatch/jlib/opatch.jar file. Check if you have the particular class file there. To check this, execute the following command; the missing class file will be printed out:

cd $ORACLE_HOME/OPatch/jlib
jar tf opatch.jar <Class File Name>.class

It is recommended that you contact Oracle support when you encounter this error.

Another reason might be having done a file transfer of OPatch in a non-binary mode.

When I apply a patch, I get an error that reads "OPatch cannot find the required command 'ar' from Property file and your PATH". What do I do?

'ar' is a command used by OPatch. This message may appear if OPatch is not able to locate this command.

For more details and work-around for this problem, refer to "OPatch cannot find system commands like fuser, make" in section "Troubleshooting OPatch" on page 7E9.

When I apply a patch, I get an error that reads "OPatch cannot find the required command 'fuser' from Property file and your PATH". What do I do?

'fuser' is a command used by OPatch. This message may appear if OPatch is not able to locate this command.

For more details and work-around for this problem, see "OPatch cannot find system commands like fuser, make" in section "Troubleshooting OPatch" on page 7E9.

How do I get the information about a patch that I applied long back?

You can look at the folder $ORACLE_HOME/.patch_storage/<patch-id_timestamp>. It has detailed information about the patch. You can also use opatch lsinventory -detail to see the files that have been modified by the patch.

Where do I get the OPatch 10.2 log files?

You can look at the folder $ORACLE_HOME/cfgtoollogs/opatch for OPatch 10.2 log files.

How do I find out a list of Oracle home(s) for a host?

To find out the list of Oracle home(s) in a host, use the command lsinventory -all.

How can I minimize the downtime when applying a patch to a Real Application Clusters setup?

You can minimize the downtime when applying a patch to a Real Application Clusters setup by doing Minimum Downtime Patching. For more information refer to section "Minimum Downtime Patching" on page 5E3.

Can I stop applying a patch after applying it to a few nodes? What are the possible issues?

Yes, it is possible to stop applying a patch after applying it to a few nodes. But, Oracle recommends that you do not do this. There is a prompt that allows you to stop applying the patch. This means you cannot apply another patch until the process is restarted and all the nodes are patched or the partially applied patch is rolled back.

Can I run patching in scripted mode?

Yes, it is possible by using the command opatch <option> -silent. For more information, refer to the -silent option documented for several commands in Chapter 4, "OPatch Utility and Commands".

Before applying a patch I want to know what is the impact of the patch?

You can use the command opatch <option> -report. For more information, refer to the -report option documented for several commands in Chapter 4, "OPatch Utility and Commands".

What versions of OPatch can I use with Oracle Universal Installer 10.2?

Oracle recommends using OPatch version 10.2 from the Oracle home with Oracle Universal Installer 10.2. Also note that OPatch is compatible only with the version of Oracle Universal Installer that is installed in the Oracle home.

Is Opatch 10.2 backward compatible? Can I use OPatch 10.2 to apply 9.2 and 10.1 patches?

No, OPatch 10.2 is not backward compatible. You can use Opatch 10.2 only to apply 10.2 patches.

When I apply a patch, I get an error that reads as follows:

"OPatchSession cannot load inventory for the given Oracle Home <Home_Location>. Possible causes are:No read or write permission to ORACLE_HOME/.patch_storageCentral Inventory is locked by another OUI instanceNo read permission to Central InventoryThe lock file exists in ORACLE_HOME/.patch_storageThe Oracle Home does not exist in Central Inventory"

What do I do?

This error may occur because of any one or more of the following reasons:

The ORACLE_HOME/.patch_storage may not have read/write permissions. Ensure that you give read/write permissions to this folder and apply the patch again.
There may be another Oracle Universal Installer instance running. Stop it and try applying the patch again.
The Central Inventory may not have read permission. Ensure that you give read permission to the Central Inventory and apply the patch again.
The ORACLE_HOME/.patch_storage directory might be locked. If this directory is locked, you will find a file named patch_locked inside this directory. This may be due to a previously failed installation of a patch. To remove the lock, restore the Oracle home and remove the patch_locked file from the ORACLE_HOME/.patch_storage directory. For more information on restoring the Oracle home, refer to section "Restoring Oracle Homes" on page 7E2.
The Oracle home may not be present in the Central Inventory. This may be due to a corrupted or lost inventory or the inventory may not be registered in the Central Inventory. For more information, refer to "Diagnosing and Recovering from Central Inventory Corruption" in the Oracle Universal Installer User's Guide.