16 Patching Oracle RAC

This chapter explains how you can patch Oracle RAC. In particular, this chapter covers the following:

Getting Started
Deployment Procedures
Core Components Patched
Supported Releases
Patching Oracle RAC in Online Mode
Patching Oracle RAC in Offline Mode

Getting Started

This section helps you get started with this chapter by providing an overview of the steps involved in patching Oracle RAC. Consider this section to be a documentation map to understand the sequence of actions you must perform to successfully patch Oracle RAC. Click the reference links provided against the steps to reach the relevant sections that provide more information.

Table 16-1 Getting Started with Patching Oracle RAC

Step	Description
	Understanding the Deployment Procedure Understand the Deployment Procedure that is offered by Enterprise Manager Grid Control for patching Oracle RAC. Know how the Deployment Procedure functions, what use cases it covers, what core components are patched, and so on.	To learn about the Deployment Procedure, see Deployment Procedures. To learn about the core components that are patched, see Core Components Patched.
	Knowing About The Supported Releases Know what releases of Oracle RAC can be patched by the Deployment Procedure.	To learn about the releases supported by the Deployment Procedure, see Supported Releases.
	Selecting the Patching Mode Identify whether the host that runs Enterprise Manager Grid Control has an Internet connection to connect to My Oracle Support. If it has, then choose to patch in online mode. Otherwise, choose to patch in offline mode.	To learn about patching Oracle RAC in online mode, see Patching Oracle RAC in Online Mode. To learn about patching Oracle RAC in offline mode, see Patching Oracle RAC in Offline Mode.
	Patching RAC Nodes Individually or Collectively Decide whether you want to patch the nodes one by one, using the Rolling methodology, or all at a time, in parallel, using the All Nodes methodology.	Online Mode: To learn about patching Oracle RAC nodes individually in online mode, see Patching Oracle RAC Nodes Individually. To learn about patching Oracle RAC nodes collectively in online mode, see Patching Oracle RAC Nodes Collectively. Offline Mode: To learn about patching Oracle RAC nodes individually in offline mode, see Patching Oracle RAC Nodes Individually. To learn about patching Oracle RAC nodes collectively in offline mode, see Patching Oracle RAC Nodes Collectively.
	Understanding the Deployment Phases The Deployment Procedure consists of a series of interview screens that take you through the different deployment phases and capture the required information. Understand the different phases involved and know what information is captured in each phase.	Online Mode: To learn about the deployment phases involved in patching Oracle RAC nodes individually in online mode, see Deployment Phases. To learn about the deployment phases involved in patching Oracle RAC nodes collectively in online mode, see Patching Oracle RAC Nodes Collectively. Offline Mode: To learn about the deployment phases involved in patching Oracle RAC nodes individually in offline mode, see Deployment Phases. To learn about the deployment phases involved in patching Oracle RAC nodes collectively in offline mode, see Patching Oracle RAC Nodes Collectively.
	Meeting the Prerequisites Before you run any Deployment Procedure, you must meet the prerequisites, such as setting up of the environment for online or offline patching, applying mandatory patches, setting up of Oracle Software Library, and so on.	Online Mode: To learn about the prerequisites for patching Oracle RAC nodes individually in online mode, see Prerequisites. To learn about the prerequisites for patching Oracle RAC nodes collectively in online mode, see Patching Oracle RAC Nodes Collectively. Offline Mode: To learn about the prerequisites for patching Oracle RAC nodes individually in offline mode, see Prerequisites. To learn about the prerequisites for patching Oracle RAC nodes collectively in offline mode, see Patching Oracle RAC Nodes Collectively.
	Running the Deployment Procedure Run the Deployment Procedure to successfully patch Oracle RAC.	Online Mode: To patch Oracle RAC nodes individually in online mode, see Patching Procedure. To patch Oracle RAC nodes collectively in online mode, see Patching Oracle RAC Nodes Collectively. Offline Mode: To patch Oracle RAC nodes individually in offline mode, follow the steps explained in Patching Procedure. To patch Oracle RAC nodes collectively in offline mode, follow the steps explained in Patching Oracle RAC Nodes Collectively.

Deployment Procedures

Enterprise Manager Grid Control offers the following Deployment Procedures for patching Oracle RAC:

Patch Oracle RAC Database - Rolling
Patch Oracle RAC Database - All Nodes

While the former one allows you to patch the nodes one by one, the latter one allows you to patch all the nodes at a time, in parallel. For more information about these patching methodologies and circumstances under which you must use the Rolling methodology or All Nodes methodology, see Understanding Patching Methodologies.

Core Components Patched

When you patch Oracle RAC, essentially, the Deployment Procedures patch the following core components:

Oracle RAC Database
Optionally, Automatic Storage Management (ASM)

Supported Releases

Using these Deployment Procedures, you can patch the following releases of Oracle RAC:

Oracle Clusterware 10g Release 1 (10.1.x.x)
Oracle Clusterware 10g Release 2 (10.2.x.x)
Oracle Clusterware 11g Release 1 (11.1.x.x)

IMPORTANT:

You cannot run this procedure on Oracle RAC installations that are registered on different Clusterware (CRS).
You can run this procedure on Oracle RAC installations that have shared Oracle homes, as long as they are not multi-shared.

For example, let us consider that the Oracle RAC installation is cluster1 and it has four nodes, mainly node1, node2, node3, and node4. You can run this procedure on this cluster if all the nodes are part of the same shared Oracle home, say /nfs_1/home/oracle/db10g. However, you cannot run this procedure if two of the nodes are from one shared Oracle home, say /nfs_1/home/oracle/db10g, and the other two are from another shared Oracle home, say /nfs_2/home/oracle/db10g.
This deployment procedure supports patching of shared Oracle home RAC installations only for Oracle Real Application Clusters (RAC) 10g Release 2 (10.2.0.1) or higher.

Patching Oracle RAC in Online Mode

This section describes how you can patch Oracle RAC in online mode, that is, by connecting to My Oracle Support. This option is best suited when you have an Internet connection on the host where Enterprise Manager Grid Control is running.

In online mode, you can either patch the nodes one by one, or patch all the nodes at a time, in parallel. This section provides instructions for both these methodologies.

In particular, this section provides instructions for the following:

Patching Oracle RAC Nodes Individually
Patching Oracle RAC Nodes Collectively

Patching Oracle RAC Nodes Individually

This section describes how you can patch the nodes of Oracle RAC one by one, using the Rolling methodology.

This methodology is best suited when you are applying one-off patches that support this methodology, and when you want to maintain high availability of your targets, so when one node is being patched, the other nodes are available for service. For example, if you are patching a clusterware that has five nodes, then the first node is shut down, patched, and restarted, and then the process is rolled over to the next node, and so on, until all the nodes are patched successfully.

This section covers the following:

Deployment Phases
Prerequisites
Patching Procedure

IMPORTANT:

You cannot use this methodology to patch shared Oracle homes. If you want to patch a shared Oracle home, then use Deployment Procedure that support the All Nodes methodology. For more information, see Patching Oracle RAC Nodes Collectively.
Although most one-off patches support this methodology, there are some one-off patches that can be applied only using the All Nodes methodology. You can identify the supported methodology while downloading the one-off patches from My Oracle Support.
If the Deployment Procedure abruptly stops or terminates at any step while it is being run, you may rerun the same procedure from where it stopped or from the beginning, all over again.

Deployment Phases

The following describes the different phases involved in this Deployment Procedure:

Table 16-2 Deployment Phases for Patching Oracle RAC in Online Mode

Phase	Description
	Software Updates Allows you to select the software updates (patches and patch sets) you want to apply, black out associated targets, apply SQL script, and run OPatch options.
	Target List Allows you to select the Oracle RAC targets you want to patch.
	Oracle Configuration Manager Allows you to configure Oracle Configuration Manager so that security updates can be sent.
	Credentials Allows you to specify Oracle home credentials required to patch RAC Oracle homes, and host credentials to access the hosts where the Oracle RAC instances are present.
	Schedule Allows you to schedule the Deployment Procedure to run immediately or later.
	Review Allows you to review the details you have provided for patching the selected Oracle RAC instances.

Prerequisites

Before running the Deployment Procedure, meet the following prerequisites:

Prerequisites for Designers

Ensure that you meet the prerequisites described in the following sections:
Ensure that operating system users such as oracle and crsuser are available on all the nodes of the cluster. These users must be a part of the relevant operating system groups such as dba and oinstall.
Ensure that you run the patchability report to understand whether the Oracle Database you want to patch is suitable for this patching operation. In the report, if the Oracle Database you want to patch appears as a target that cannot be patched, then resolve the conflicts to make it a target that can be patched. For more information about the patchability report, and to understand how you can diagnose the issues with an Oracle Database target and take corrective actions, see Appendix H, "About Patching Automation Reports".
Ensure that you run the prerequisite checker Deployment Procedure as described in Running Patch Prerequisite Checker for Oracle RAC.

Prerequisites for Operators

Ensure that you use an operating system user that has the privileges to run the Deployment Procedure and its commands on the target hosts. If you do not have the privileges to do so, that is, if you are using a locked account, then request your administrator (a designer) to either customize the Deployment Procedure to run it as another user or ignore the steps that require special privileges. For information about customization, see Chapter 23, "Customizing Steps to Run as Another User".

Patching Procedure

To patch an Oracle Cluterware, follow these steps:

In Grid Control, click the Deployments tab.
On the Deployments page, in the Deployment Procedure Manager section, click Patch Procedures.
On the Deployment Procedure Manager page, in the Procedures subtab, from the table, select Patch Oracle RAC Database - Rolling. Then click Schedule Deployment.

Enterprise Manager Grid Control displays the Software Updates page of the Deployment Procedure.
On the Software Updates page, do the following:
1. In the Select Stage Location section, retain the default selection, that is, Enter Stage Location that has the value %emd_root%/EMStage.
  
  This default value refers to a location on the target host where the Oracle RAC to be patched is running. The software updates are downloaded from My Oracle Support to Oracle Software Library, and then staged in this location on the target hosts.
  
  You can modify the location if you want to, and click Save. If you save multiple custom locations, then the Select Stage Location option gets enabled and allows you to select any of the saved custom locations the next time you visit the same page.
  
  Instead of %emd_root%, you can also specify %oracle_home% to indicate the current home location being patched or %perlbin% to indicate the location of perl binary used by Oracle Management Agent (Management Agent).
2. In the RAC Database Updates section, click Add to search and select software updates from My Oracle Support. Alternatively, if you have a text file that contains details about the patches, then click Upload From File to select and upload that text file.
  
  On clicking Add, Enterprise Manager Grid Control displays the Search and Select Software Updates page. On this page, select Search My Oracle Support, specify the details of the patches you want to search, and click Go.
  
  Enterprise Manager Grid Control connects to My Oracle Support, searches the patches, and displays details about them on the Search and Select Software Updates page. Select the ones you want to apply using the Deployment Procedure, and click Select.
  
  On clicking Upload From File, Enterprise Manager Grid Control displays the Upload File page. On this page, select Upload from Local Machine or Upload from Software Library depending on where the text file is available, click Browse to search and select that file, and then click Upload.
3. In the Upgrade OPatch section, retain the default selection so that the OPatch software on the target host is upgraded before the patches are applied on the database targets.
  
  Enterprise Manager Grid Control uses OPatch software to apply the one-off patches. It is assumed that this software is already available on the target hosts managed by Oracle Management Agents (Management Agent), but Oracle recommends you to retain the selection so that the existing software is upgraded to the latest release.
4. In the Black Out Associated Targets section, retain the default selection so that all targets associated with the clusterware to be patched are blacked out while the patching is in progress.
5. In the Apply SQL Script section, retain the default selection so that the default SQL script that is packaged with the patch is used for modifying the schemas.
  
  If you want to run any custom script, then select Enter the script to apply SQL and specify the full path to the location where the custom script is available. For example, %oracle_home%/rdbms/admin/catupgrd.sql .
  
  The variables you can use for the directory location are %emd_root% to indicate a location on the target host, %oracle_home% to indicate the current home location being patched, or %perlbin% to indicate the location of perl binary used by the Management Agent.
  
  You have an option of running both the scripts by selecting both these options or one of the scripts by selecting only one of these options. If you select both these options, then both the scripts are run, that is, the default script runs first and then the custom script runs.
6. In the Advanced OPatch Options section, specify any opatch-related options you want to pass while running this Deployment Procedure. For example, if you want the Deployment Procedure to ignore duplicate patches, then specify -skip_duplicate.
  
  The following are the options you can specify:
  
  -skip_duplicate, skips duplicate patches
  
  -skip_conflict, skips conflicting patches
  
  -skip_missing_component, skips patches whose components are missing
  
  -invPtrLoc, the location of oraInst.loc file
  
  Note:
  If you want to specify multiple options, then separate the list with a white space. For example, -skip_duplicate -skip_conflict
7. Click Next.
On the Target List page, in the RAC Database Targets to be Patched section, click Add to add the Oracle RAC database targets you want to patch. Then click Next.
On the Oracle Configuration Manager page, do the following:
1. If the host where the Oracle RAC database is being patched has a direct connection to the Internet, then specify an e-mail address and My Oracle Support password.
  
  An e-mail address is required so that security updates and install updates can be sent. You can specify any e-mail address, but Oracle recommends you to specify the My Oracle Support user name. For example, john.mathew@xyz.com.
  
  If the My Oracle Support password is incorrect, you will be allowed two more attempts. However, if your password is incorrect in all three attempts or if it is left blank, then you are registered anonymously, which means, the configuration information will be collected and uploaded to My Oracle Support but the uploaded information will not be associated with your My Oracle Support account. Therefore, if you log in to My Oracle Support with your credentials, you will not see this information displayed against your account. However, if you had specified an e-mail address, then you will continue to receive security updates and other notifications from Oracle to that e-mail address.
2. If the host where the Oracle RAC database is being patched has an indirect connection to the Internet through a proxy server, then specify an e-mail address and My Oracle Support password, and then in the Connection Details section, specify the proxy server details.
  
  Note:
  You can change the proxy server settings any time after the Deployment Procedure ends. To do so, run the configCCR command from the /ccr/bin/ directory within the Oracle home directory of the provisioned database.
3. If the host where the Oracle RAC database is being patched does not have a direct or indirect connection to the Internet, then specify the e-mail address and leave the other fields blank.
  
  In this case, after you complete the installation process, manually collect the configuration information and upload it to My Oracle Support. To understand how the configuration information can be manually collected and uploaded, see the steps outlined in Manually Collecting and Uploading Configuration Information to My Oracle Support.
On the Credentials page, in the Home Credentials section, specify the Oracle home credentials required to patch the Oracle homes, and in the Host Credentials section, specify the operating system credentials to log in to the hosts where the Oracle homes are present. Then click Next.

Note:
When there are Oracle home active steps, the Home Credentials section appears. When there are host-level active steps, the Host Credentials section appears. When Oracle home active steps and host-level active steps are present, both the sections appear.

For both sections, choose to use the preferred credentials so that the credentials stored in the Management Repository can be used.

Of course, you can always override the preferred credentials with a new set of credentials. If you choose to do so, you can specify either a common set of credentials to be used across Oracle homes and hosts, or a unique set of credentials for each Oracle home and host.

After specifying a new set of credentials to override the preferred credentials, click Save OH Credential if you want to store the new credentials in the Management Repository.
On the Schedule page, schedule the Deployment Procedure to run either immediately or later.

For Instance Name, specify a unique name for this Deployment Procedure instance so that it can be tracked later and reused with the same settings.
On the Review page, review the details you have provided for patching the selected Oracle RAC databases, and click Deploy.

Note:
If you have not applied the one-off patch as described in Applying One-Off Patch to Introduce Analyze Mode and Reports, then the Deploy button on the Review page appears as Submit.

Patching Oracle RAC Nodes Collectively

The prerequisites and the deployment instructions for patching all the nodes of Oracle RAC at a time, in parallel, using the All Nodes methodology, are the same as the ones described in Patching Oracle RAC Nodes Individually. However, in Step (3), on the Deployment Procedure Manager page, in the Procedures tab, from the table, select Patch Oracle RAC Database - All Nodes. Then click Schedule Deployment.

Although the instructions remain the same, when you submit the Deployment Procedure, Enterprise Manager Grid Control internally patches all the nodes of Oracle RAC collectively, that is, all the nodes are shut down and the patch is applied on all of them at the same time.

This methodology is best suited when you are applying a patch set or those one-off patches that support this methodology, or when you want to patch a shared Oracle home clusterware. For example, if you are patching a shared Oracle home clusterware that has five nodes, the patch and the scripts within it are run only once on the shared Oracle home, and the other nodes are ignored or skipped. This saves time as the patching operation is performed only once, that is, only on the shared Oracle home, not on all the nodes.

Note:

Patching of shared Oracle homes is supported only for Oracle Clusterware 10g Release 2 (10.2.0.1) or higher.
If the Deployment Procedure abruptly stops or terminates at any step while it is being run, you may rerun the same procedure from where it stopped or from the beginning, all over again.

Patching Oracle RAC in Offline Mode

This section describes how you can patch Oracle RAC in offline mode, that is, without connecting to My Oracle Support. This option is best suited when you do not have an Internet connection on the host where Enterprise Manager Grid Control is running.

In offline mode, you can either patch the nodes one by one, or patch all the nodes at a time, in parallel. This section provides instructions for both these methodologies.