12 Managing Oracle Software and Applying Patches

To manage Oracle software and apply patches in Oracle Real Application Clusters (Oracle RAC) environments, you use Oracle Universal Installer or the OPatch utility.

See Also:

12.1 About Patches and Patch Sets

Oracle issues product fixes for its software called patches. They are associated with particular releases and versions of Oracle products.

The patching cycle involves downloading patches, applying patches, and verifying the applied patch to ensure that the bug fixes present in the patch reflect appropriately.

Patching involves migrating from one version of the software product to another, within a particular release, unlike upgrading which involves moving from one release of a product to another newer release of the software product. When you apply the patch to your Oracle software installation, it updates the executable files, libraries, and object files in the software home directory. The patch application can also update configuration files and Oracle-supplied SQL schemas.

Oracle periodically releases the following types of patches to fix the bugs encountered in the Oracle software:

Patch Type Description

Interim Patches

Released to fix a bug, or a collection of bugs. Previously called patch set exceptions (PSE), one-off patches, or hot fixes.

Interim Patches (for Security bug fixes)

Released to provide customer specific security fixes. Previously referred to as a test patch, fix verification binary, or e-fix.

Diagnostic Patches

Mainly help diagnose and verify a fix, or a collection of bug fixes.

Bundle Patch Updates

Cumulative collection of fixes for a specific product or component. Previously referred to as a maintenance pack, service pack, cumulative patch, update release, or MLR.

Patch Set Updates (PSU)

Cumulative patch bundles that contain well-tested and proven bug fixes for critical issues. PSUs have limited new content, and do not include any changes that require re-certification.

Security Patch Updates

A cumulative collection of security bug fixes. Previously known as Critical Patch Updates, or CPUs.

Interim patches are bug fixes available to customers in response to specific bugs. They require a particular base release or patchset to be installed before you can apply them. These patches are not versioned and are generally available in a future patchset and also the next product release. Interim patches are applied by using Enterprise Manager Cloud Control or OPatch, which is included with your Oracle Database installation.

Patch sets updates (PSUs) and patch bundles are mechanisms for delivering fully tested and integrated product fixes. All the fixes in a patch set have been tested and are certified to work with each other. Because a patch set includes only low impact patches, it does not require you to certify applications or tools against the updated Oracle Database software. When you apply a patch set, many different files and utilities are modified. This results in a release number change for your Oracle software, for example, from Oracle Database 11.2.0.3.0 to Oracle Database 11.2.0.3.2. You use OPatch to apply PSUs.

12.2 Alternative Methods of Patching

OPatch is a commonly used method for patching Oracle software homes, but this is not the only method of patching Oracle software.

These alternative methods are only described at a conceptual level in this guide. However, you might find these methods more appropriate for your organization than using OPatch.

12.2.1 Using Oracle Enterprise Manager Cloud Control for Patching Operations

Using Cloud Control with its Provisioning & Patching functionality, you can automate the patching of your Oracle Grid Infrastructure and Oracle RAC software.

Before you can use Cloud Control to patch your Oracle software, you must perform the following system configuration tasks:

  • Install the Enterprise Manager Agent on all cluster nodes

  • Use PDP setup to ensure the host user has the necessary privileges to complete patching tasks

  • Configure named and preferred credentials in Enterprise Manager

  • Configure a software library for storing patch files

Details on how to perform these tasks, and how to patch your Oracle Grid Infrastructure and Oracle RAC software using Cloud Control are available from the following PDF file: http://www.oracle.com/technetwork/oem/pdf/512066.pdf

The rest of this chapter describes how to install patches without using Cloud Control.

12.2.2 Rapid Home Provisioning, Patching, and Upgrading

Rapid Home Provisioning is a method of deploying software homes to any number of nodes in a data center from a single cluster, and also facilitates patching and upgrading software.

With Rapid Home Provisioning, you create, store, and manage templates of Oracle homes as images (called gold images) of Oracle software, such as databases, Oracle Grid Infrastructure, middleware, and applications. You can make a working copy of any gold image, and then you can provision that working copy to any node in the data center.

You store the gold images in a repository located on a Rapid Home Provisioning Server, which runs on one server in the Rapid Home Provisioning Server cluster that is a highly available provisioning system. This provisioning method simplifies quick patching of Oracle homes, thereby minimizing or completely eliminating downtime.

Using Rapid Home Provisioning, you can provision Oracle Database software for releases Oracle Database 11g release 2 (11.2.0.4) and later. You can also use Rapid Home Provisioning to provision Oracle Grid Infrastructure (11g release 2 (11.2.0.4), and 12c release 1 (12.1) and release 2 (12.2)) homes and can also be used to patch and upgrade Oracle Grid Infrastructure homes. In addition, there is a rollback capability that facilitates reverting from a patching procedure.

Starting with Oracle Database 12c Release 2 (12.2), RHP can manage Oracle Grid Infrastructure and Oracle Database 11.2 and 12.1 software installations. With a single command, RHP can provision new homes to such targets. RHP can also orchestrate the patch or upgrade process for these targets, also with a single command. There is no prerequisite for the targets; you do not need to install any client or agent software on the targets before the provision, patch, or upgrade takes place.

When you provision Oracle Database software, Rapid Home Provisioning offers additional features for creating various types of databases (such as Oracle Real Application Clusters (Oracle RAC), single instance, and Oracle Real Application Clusters One Node (Oracle RAC One Node) databases) on different types of storage, and other options, such as using templates and creating container databases (CDBs). Rapid Home Provisioning improves and makes more efficient patching of database software, allowing for rapid and remote patching of the software, in most cases, without any downtime for the database.

Rapid Home Provisioning simplifies maintenance in large environments because you have only to update software homes one time on the Rapid Home Provisioning Server. Rapid Home Provisioning enables you to create a gold image from any software home. You can then provision that software to any Rapid Home Provisioning Client or target as a working copy. The software may be any binary that you will run on a Rapid Home Provisioning Client or target.

12.3 Obtaining the Patch

You obtain patches and patch sets from My Oracle Support, which is the Oracle Support Services website.

The Oracle Support Services website is located at: https://support.oracle.com

To locate patches on the My Oracle Support website:

  1. Log in to your account on My Oracle Support.
  2. Select the Patches & Updates tab.
  3. If you know the patch number, then you can enter it into the Patch Name or Number field, then click Search.

    If you want to search for all available patches for your system, then select Product or Family (Advanced Search), which is located above the Patch Name or Number field. Supply the following information:

    • Choose the products you want to patch (for example, Oracle Clusterware, Oracle Database, or an individual product such as Universal Installer)

    • Specify the software release for the products you selected, for example, Oracle 12.2.0.1.0.

    • Specify the platform on which the software is installed.

    Click Search to look for available patches.

    The Patch Search Results page is displayed.
  4. On the Patch Search Results page, select the number of the patch you want to download.
    A details page for that patch appears on your screen.
  5. Click the ReadMe button to view the ReadMe file for the patch, or click Download to download the patch to your local computer.

    If you download the patch to a computer that is not a node in the cluster, then you must transfer the file using a binary protocol to a cluster node.

    You can also choose to add the patch to your current patch plan or create a new patch plan. A patch plan is a collection of patches that you want to apply as a group. To learn more about using patch plans, in the Add to Plan drop-down list select Why use a Plan?

See Also:

12.4 Preparing to Use OPatch

Before you apply the patch to your Oracle RAC database, your Oracle ASM installation, or to your Oracle Clusterware installation, you must prepare the operating system environment and stage the patch locally.

See Also:

12.4.1 Checking the ORACLE_HOME Environment Variable

OPatch verifies if the Oracle home is present.

You must ensure that the ORACLE_HOME environment variable is set to the Oracle home of the product you are trying to patch. Also, you must use a version of OPatch that has the same version number as the Oracle home being patched.

Check the respective vendor documentation for the details to set the environment variable.

To check the current setting of the ORACLE_HOME variable on Linux:

  1. In a command window, log in to the operating system as the grid or oracle user.
  2. Use the echo command to display the current setting of the ORACLE_HOME environment variable.
    echo $ORACLE_HOME
    

12.4.2 Performing a Backup Before Patching Software

Oracle strongly recommends that you back up the software directory you are patching before performing any patch operation. This applies to Oracle Database or Oracle Grid Infrastructure software installation directories.

To back up the software installation:

  1. Back up the software installed in the specified Oracle home using any of the following:
    1. An operating system utility, such as zip, cp -r, tar, or cpio, to back up the software in the Oracle home directory to disk.
    2. The Oracle Secure Backup utility to back up the software in the Oracle home directory to tape.

12.4.3 Staging the Patch on Each Node

After you download the patch from My Oracle Support, you must copy the patch to each node that runs the software being patched.

12.4.4 Updating the PATH Environment Variable

You can either specify the path to the opatch executable file when running OPatch, or you can update the PATH environment variable to include the OPatch directory.

The opatch binary file is located in the Oracle_home/OPatch directory.

To update the PATH environment variable on Oracle Linux systems:

  1. In a command window, log in to the operating system as the Oracle home owner, or a user that has write permission for the files in the Oracle home.
  2. Use a shell command similar to the following to update the value of the PATH environment variable, where /u01/app/oracle/product/12.2.0/dbhome_1 is the location of your Oracle home directory:
    $ export PATH=$PATH:/u01/app/oracle/product/12.2.0/dbhome_1/OPatch
    

    You could also modify the shell profile script for the current user to have this variable configured every time you log in.

12.4.5 Configuring SSH User Equivalency

User equivalency exists when the operating system user on each node is considered to be identical.

For information on configuring user equivalency, see "Configuring Operating System Users and Groups on Linux Systems".

12.4.5.1 Testing SSH User Equivalency

Use the operating system SSH command to verify the user equivalency is working.

  1. On the system where you want to run OPatch, log in as the software owner. If you are patching Oracle Grid Infrastructure, login as the grid user. If you are patching Oracle RAC software, then login as the oracle user.
  2. Use the following command to test user equivalency:
    [oracle@racnode1] $ ssh racnode2 date
    
    1. If the date is returned, then user equivalency between the source and destination node has been configured.
    2. If you see output similar to the following, then SSH user equivalency is not enabled:
      Enter passphrase for key '/home/oracle/.ssh/id_rsa': 
      
  3. If you determine from this test that SSH user equivalency does not exist, then enable SSH user equivalency before continuing with the patching operation.

See Also:

12.4.5.2 Enabling SSH User Equivalency

Before you patch a system, make sure the user equivalency is working.

To enable SSH user equivalency:

  1. On the system where you want to run OPatch, open a command window and log in as the software owner. Login as grid if patching Oracle Grid Infrastructure software, or login as oracle if patching Oracle Database software.
  2. Start the SSH agent and load the SSH keys into memory using the following commands:
    $ /usr/bin/ssh-agent $SHELL
    $ /usr/bin/ssh-add
    
    These commands start the ssh-agent on the local node, and load the RSA and DSA keys into the current session's memory so that you are not prompted to use pass phrases when issuing SSH commands.
  3. At the prompt, enter the pass phrase for each key that you generated when configuring Secure Shell, for example:
    [oracle@racnode1 .ssh]$ exec /usr/bin/ssh-agent $SHELL
    [oracle@racnode1 .ssh]$ /usr/bin/ssh-add
    Enter passphrase for /home/oracle/.ssh/id_rsa  passphrase
    Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)
    Identity added: /home/oracle/.ssh/id_dsa (/home/oracle/.ssh/id_dsa)
    
  4. To test if you have configured SSH correctly, run the following command.
    [oracle@racnode1] $ ssh racnode2 date
    
    If you have configured SSH correctly, then you are not prompted for a password or a pass phrase.

    Important:

    Do not close this command window until you have completed the patch installation. If you must close the command window in which you enabled SSH user equivalency before the patch installation is complete, then repeat Step 1 to Step 4 before starting the patch installation.

See Also:

12.5 Applying Patches

Patching in an Oracle RAC environment is slightly different compared to patching a single node. If OPatch detects a cluster, then it uses OUI to query the software inventory to find the local node name and node list.

Before you install a patch, you must stop all the applications running from the software directory that is being patched. In a cluster, you may have to shut down additional applications, depending upon which software is being patched. The following table lists the applications to stop when patching Oracle software.

Table 12-1 Patching Oracle Home Directories

Oracle Home Directory Applications to Stop

Oracle RAC Database

Oracle RAC database, Enterprise Manager Agent, listener, and any other applications that run from the Oracle RAC home directory

Oracle Grid Infrastructure

Oracle RAC database and all applications running from the Oracle RAC home directory, any single-instance databases that use the same Oracle ASM instance as the cluster database, the Oracle ASM instance, all node applications, Oracle Clusterware, and any other applications that run from the Grid home directory

12.5.1 Patching the Grid Home

Before you can apply a patch or make other modifications to the software files in the Grid home, you must first unlock the Grid home.

To patch the Grid home, perform the following steps:

  1. Change directory to the path Grid_home/crs/install, where Grid_home is the path to the Grid home. As the root user, perform the prepatch steps for the Grid home using commands similar to the following:
    cd /u01/app/12.2.0/grid/crs/install 
    rootcrs.sh -prepatch 
    When you run the rootcrs script with the -prepatch option, it stops the Oracle Clusterware stack and unlocks the files in the Grid home so they can be modified.
  2. Change user to the software owner for Oracle Grid Infrastructure for a cluster and apply the patch to the Grid home, using one of the patching methods.
  3. After you have finished modifying the Grid home, perform the postpatch steps as the root user using commands similar to the following:
    cd /u01/app/12.2.0/grid/crs/install 
    rootcrs.sh -postpatch
    
    The rootcrs script with the -postpatch option locks the Grid home again and restarts the Oracle Clusterware stack.

12.5.2 All Node Patching

In all node patching, all the nodes in the cluster are initially shut down and the patch is applied on all the nodes.

After all the nodes have been patched, then the Oracle Clusterware stack and all its registered resources are restarted on each node. This method is typically used for very critical patches and it leads to maximum downtime. OPatch uses this method if the patch cannot be applied in a rolling fashion, and you did not specify the minimize_downtime option.

12.5.3 Out-of-Place Rolling Patching for Oracle RAC

Out-of-place rolling patching involves installing the patch in a new home, modifying the Oracle home of the database, and then restarting instances in a rolling fashion.

Rolling patching is performed group by group, separately, until all the nodes in the cluster are patched. This is the most efficient means of applying an interim patch to an Oracle RAC or Oracle Grid Infrastructure for a cluster installation. By patching groups of nodes individually, there is zero downtime for the cluster database because at least one instance is always available on a different node.

While most patches can be applied in a rolling fashion, some patches cannot be applied in this fashion. The README file for the patch indicates whether you can apply the patch using the rolling patch method. If the patch cannot be applied using the rolling patch method, then you must use either "Minimum Downtime Patching" or "All Node Patching" to apply the patch.

When you perform out-of-place patching, you install the updated or patched software into a new Oracle home. Then you update Oracle Database to use the new Oracle home.

  1. Install the patched software in a new Oracle home directory on one node.
    During installation, the installer detects that this installation is for a cluster and copies the new Oracle home to each node in the cluster.
  2. Modify the Oracle home location for the Oracle RAC database using SRVCTL.
    srvctl modify database -db database_name -oraclehome patched_home_location 
  3. Stop all Oracle Database processes on this node.
    srvctl stop home -oraclehome patched_home_location -node nodename
  4. Start all Oracle Database processes on this node.
    srvctl start home -oraclehome patched_home_location -node nodename

    You must restart the Oracle Database processes for them to start using the patched software.

  5. Repeat Step 3 and Step 4 on each node in the cluster to restart all the Oracle RAC database instances.

12.5.4 Minimum Downtime Patching

Minimum downtime patching shortens the time that all the nodes have to be down to apply a patch.

In minimum downtime patching, one set of nodes is shut down and the patch is applied to those nodes. After the first set of nodes has been patched, the second set of nodes is shut down. The first set of nodes is then restarted and the patch is applied to the second set of nodes. After the patch has been applied to the second set of nodes, those nodes are restarted. This method leads to less downtime for Oracle RAC, compared to having all the nodes shut down at the same time.

When you use the minimum downtime patching method, the following actions occur:

  • The local node is always patched first.

  • The local node is used as a base to patch the other nodes.

  • The user is prompted for the set of nodes to patch first from the remaining nodes.

  • For each node in this first set, the user is asked to stop the instance and then the patch is propagated to that node before continuing to the next node. When the initial set of nodes has been patched, the user is asked to shut down the remaining nodes.

  • After the local node is patched, the patch is propagated to the last set of nodes and the inventory is updated. The last instances are stopped on the remote nodes. You can then start the patched nodes (the first set of nodes) before patching the remaining nodes.

12.6 Applying Patch Sets

For instructions on applying the latest patch set to your Oracle RAC database and Oracle Clusterware installations on Oracle Linux, search for "Oracle 12c Release 2 (12.2) Support Status and Alerts" documentation on the My Oracle Support website.

This document provides a summary of the patch sets available for Oracle 12c Release 2. Using this document, you can easily locate and view the Patch Set Notes for your platform. The Oracle Database Patch Set Notes document contains the following information:

  • System requirements and information about how to install or reinstall the patch set

  • A list of all bugs fixed to date that are specific to Oracle Database for specified platform

  • A list of known issues relating to Oracle Database for the specified platform

To locate the Patch Set notes on My Oracle Support:

  1. Log in to My Oracle Support.
  2. Select the Patches & Updates tab.
  3. Select Quick Links to the Latest Patchsets, Mini Packs, and Maintenance Packs.
  4. Under the heading Latest Oracle Server/Tools Patchsets, select Oracle Database.

    A list of operating systems appears.

  5. Place your cursor over the entry that matches your operating system, or use the triangular arrows to search for your operating system.

    When you place the cursor over the entry for your operating system, for example, Linux x86, a list of database versions appears.

  6. Select 12.2.0

    The Advanced Search page appears.

  7. Scroll to the bottom of this page to see the list of available patch sets.
  8. Select the number in the Patch column for the patch set you want to view or download.

    The Patchset description and download page appears.

  9. Click View Readme to see the patch set notes.

    On this page you can also click Download to download the patch to your computer.

  10. If you choose to download the patch, then follow the instructions in the ReadMe file of the patch set to apply the patch set to your software.

12.7 Troubleshooting Patch Deployment

Applying patches to your Oracle RAC environment can be complicated at times and you may be required to troubleshoot the patch deployment.

If you have problems applying a patch to your Oracle RAC database, then review these solutions to common problems. If the problem you encountered is not listed, then review the log and trace files and use My Oracle Support to obtain assistance.

12.7.1 Updating the Node List for OPatch

If OPatch does not automatically detect Oracle RAC or its nodes, then investigate the contents of the inventory and ensure they are complete.

To update the node list for OPatch:

  • If the list of nodes for your cluster is not complete, then you can update it by using Oracle Universal Installer and the -updateNodeList flag, as demonstrated in the following example:
    Oracle_home/oui/bin/runInstaller -updateNodeList 
    ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1 
    CLUSTER_NODES=racnode1,racnode2,racnode3 -noClusterEnabled
    

See Also:

Oracle Universal Installer Installation Guide for more information about updating the node list

12.7.2 About OPatch Log and Trace Files

Logging and tracing is a common aid for debugging. OPatch maintains logs for all apply, rollback, and lsinventory operations.

The log files are located in the Oracle_home/cfgtoollogs/opatch directory. Each log file is tagged with the time stamp of the operation. Log files are named as opatch_mm-dd-yyyy_hh-mm-ss.log, where mm-dd-yyyy is the current date and hh-mm-ss is the current time. Each time you run OPatch, a new log file is created.

For example, if a log file is created on June 9, 2016 at 10:57 PM, then it is named as follows:

opatch2016-06-09_22-07-57PM_1.log

OPatch also maintains an index of the commands processed by OPatch and the log files associated with it in the opatch_history.txt file located in the Oracle_home/cfgtoollogs/opatch directory. A sample of the opatch_history.txt file is as follows:

Date & Time : Thu Jun 09 22:07:45 MDT 2016
Oracle Home : /u01/app/oracle/product/12.2.0/dbhome_1
OPatch Ver. : 12.2.0.1.0
Current Dir : /u01/app/oracle/product/12.2.0/dbhome_1
Command     : lsinventory -xml /u01/app/oracle/product/12.2.0/dbhome_1 ...
 Log File    : /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatch/opatch2016-06-09_22-07-57PM_1.log

12.7.3 Resolving the "Not a valid patch area" Error

You might get this error if the directory that the OPatch utility is using to do the patch does not match the template for what it is checking, or if the OPatch utility is run from an invalid directory.

The Patch_Shiphome directory should have the following structure:

  • An etc subdirectory that has the metadata files

  • A files subdirectory that has the patch files

  • The etc/config/inventory.xml file and the actions.xml file under the same directory

To resolve the "Not a valid patch area" error:

  • Perform one of the following actions:
    1. Remove the patch shiphome directory and re-create it with the proper structure (by extracting the files again).
    2. Start the OPatch utility from the directory where the patch to be installed has been unzipped and staged on disk.
    3. Use the following command when starting OPatch:
      opatch apply /Patch_Shiphome
      

      where Patch_Shiphome is the location where the patch has been staged on disk.

12.7.4 Resolving the "Unable to remove a partially installed interim patch" Error

If the patching process is interrupted, then you might get the error "Unable to remove a partially installed interim patch" when you try to install the patch a second time.

To resolve the partially installed patch error:

  1. Ensure that the environment variable ORACLE_HOME is set to the Oracle home directory you are attempting to patch.
  2. Go to the Oracle_home/.patch_storage/patch-id_timestamp directory and run the restore.sh script (or restore.bat on Windows platforms) as follows:
    Oracle_home/.patch_storage/patch-id_timestamp/restore.sh
    
  3. On Linux and UNIX systems, use the Oracle_home/.patch_storage/patch-id_timestamp/make.txt file (if available) to modify your operating system environment, as follows:
    /bin/sh make.txt
    
  4. Attempt to apply the patch again.

12.8 Upgrading the Oracle Software

Upgrades are performed when you install a new software release or version, or apply a patch set.

An upgrade modifies most, if not all, of the installed Oracle software files, whereas a patch typically modifies only a few files. You can perform rolling upgrades, or you can shut down the Oracle Database and Oracle Clusterware software to perform the upgrade.

You perform out-of-place upgrades when upgrading the software to a new release. To perform an out-of-place upgrade, you install the Oracle Grid Infrastructure for a cluster into a new Grid home. When performing the upgrade, you specify the location of the new Grid home instead of selecting the existing software location.

When performing an out-of-place upgrade, the old and new version of the software are present on the nodes at the same time, each in a different home location, but only one version of the software is active at any given time. To upgrade your Oracle Database 11g software or Oracle Database 12c release 1 (12.1) software to Oracle Database 12c release 2 (12.2), you install Oracle Grid Infrastructure for a cluster and Oracle Database into new Oracle home directories. At the end of the upgrade process, the active version of the software is changed to use the Oracle Grid Infrastructure for a cluster or Oracle Database 12c release 2 (12.2) software.

You can use Database Upgrade Assistant (DBUA) to upgrade an existing database to the current release of Oracle Database. Database Upgrade Assistant (DBUA) guides you through the upgrade process and configures your database for the new release. DBUA automates the upgrade process and makes appropriate recommendations for configuration options such as tablespaces and online redo log files.

See Also: