3 Plan for Patching

This section outlines the following key considerations when planning a patching strategy for an Oracle Fusion Applications environment:

Patch Plan: Skills and Knowledge Required

A technical team with the skills and knowledge required to complete patching may include the following:

  • A Fusion Applications administrator

  • A Database administrator

  • A Identity Management (IDM) expert

At a minimum, the team responsible for downloading, evaluating, and applying patches must have the following technical knowledge:

  • Oracle Fusion Application installation(s) knowledge
  • WebLogic and SOA familiarity
  • Database knowledge
  • Identity Management knowledge
  • Relevant operating system knowledge (Linux, etc.)

Additionally, it is highly recommended to form a Patch Advisory Forum comprised of technical subject matter experts and management. The Patch Advisory Forum would act as a gate keeper and would be the only body with the authority to approve patches. Patch requests are presented to the Patch Advisory Forum, which assesses the risk and prioritizes the application of patches. Critical input into the forum typically includes timing estimates, outage planning, and testing requirements obtained from an impact assessment of the patch environment. The Patch Advisory Forum may be different for each environment and would typically include stakeholder representation for the specific environment.

Example 1: When planning to patch a Test environment, then the Test Lead would be a key stakeholder. The patch would be scheduled at a time that will not interfere with ongoing testing activities, and to ensure that the correct teams are notified about the patching window appropriately.

Example 2: When planning to patch a Production environment, then key business stakeholders will be involved in the Patch Advisory Forum. For example, if the patch plan would affect a payroll run, those stakeholders might request a patching delay until the payroll is completed.

Ensure The Patching Tools are Up-to-Date

Before installing Technical Patches, ensure that the environment has the latest version of the framework for installing the patches (OPatch). When downloading a Technical Patch Bundle (P4FA), the latest version of OPatch is included. To ensure that OPatch is up to date, perform the following steps:

  1. Set the ORACLE_HOME to the directory that will be patched. For example:

    export ORACLE_HOME=/u01/app/idm/products/app/idm
    
  2. Navigate to the OPatch directory and execute cd $ORACLE_HOME/OPatch.

  3. Note that the output contains the version of OPatch. For example:

    OPatch Version: 11.1.0.8.0
    OPatch succeeded
    

If a different version of OPatch is required, see the Update OPatch section.

Download the Latest Version of OPatch From My Oracle Support

MANDATORY: Check the patch Readme for prerequisite tips regarding the necessary OPatch version. Notice in the sample text below, the OPatch release number is given, as well as the bug placeholder number for downloading OPatch versions, and a link to documentation on My Oracle Support.
Ensure that the OPatch is 11g Release 11.1.0.8.3 or higher.
Review and download the latest version version available from patch# 6880880
For information about OPatch documentation, including any known issues, see My Oracle Support Document 224346.1 OPatch documentation list:
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=224346.1

Check the Current OPatch Version

Perform the following steps to validate the current OPatch version:
  1. Set the ORACLE_HOME to the directory to be patched. For example:
    export ORACLE_HOME=/u01/app/idm/products/app/idm
    
  2. Go to the OPatch directory and execute. For example:
    cd $ORACLE_HOME/OPatch
    
  3. Check the output for text like the following:
    OPatch Version: 11.1.0.8.0
    OPatch succeeded.
    
The same steps can be followed for database or applications ORACLE_HOME directories.

Update OPatch

Perform the following steps to update the OPatch tool installed on $ORACLE_HOME :
  1. Go to My Oracle Support (support.oracle.com) and select the Patches and Updates tab.
  2. Select search by “Patch Name or Number” and enter the number 6880880 in the search field, and choose the operating system as show in the following figure. The same number (6880880) works for all operating systems and OPatch versions.

    Figure 3-1 Patch Search window


    Description of Figure 3-1 follows
    Description of "Figure 3-1 Patch Search window"
  3. Click Search.

    The Patch Search result appears as shown in the following figure:

    Figure 3-2 Patch Search Results


    Description of Figure 3-2 follows
    Description of "Figure 3-2 Patch Search Results"
  4. Select the most recent version of OPatch for the relevant release to display Readme and Download options.
  5. Click the Read Me to read additional installation instructions.
  6. Click Download and follow the installation instructions in the Readme text.

Plan System Backups

MANDATORY: Before applying any patch, a cold (offline) backup of the database and the file system being patched must be performed to ensure data consistency and avoid synchronization problems.

To ensure that no changes are made in the WebLogic Server domains, it is recommended to lock the WebLogic Server configuration for all the domains in Oracle Fusion Applications environment. Best practice also suggests giving a unique name to the backup file, perhaps appending the date and time to the .tar file name.

Plan Impact and Maintenance

This section provides strategies for creating a patching approach that suits the enterprise’s needs while adhering to best practices. This section contains the following topics:

Types of Patches For Which to Plan

As mentioned previously, Oracle Fusion Applications includes both technical patches (which affect the underlying Middleware components) and functional patches (which affect the Fusion Applications product families).

Patches are release-specific; patch bundles designated for 11.1.10 cannot be applied to 11.1.9, nor to 11.1.11 versions of Oracle Fusion Applications. Furthermore, functional patches are released per product family. For example, HCM, SCM, and CRM each have their own functional patch bundles for a given release.

Patch planning is focused on functional and technical patches. IDM patching is normally handled either automatically or manually during upgrades, and is discussed in the Oracle Fusion Applications Upgrade Guide. There is a rare circumstance in which a patching strategy may be applied to Identity Management, as described in the Apply Identity Management (IDM) Patches section. Otherwise, IDM patching is not handled in this guide.

Time of Patch Bundle Releases

Since multiple patch bundles for technical and functional side are published for each release, it is recommended to establish a cadence that minimizes the amount of downtime taken to apply the various patches while maximizing the uptake of key patches for the organization.

For example, a particular HCM bundle may contain a critical fix but will not be released for another few weeks. The HCM AOO patch may be applied to a test environment to allow for completion of a critical project milestone while choosing to wait for the functional patch bundle for the production environment. Another facet of the decision could be that a particular organization may not allow for a maintenance window every month, but instead every other month.  Given that both the functional and the technical patch bundles are cumulative, patching every other month still ensures a pro-active approach while minimizing the needed maintenance windows. A six-month patching cycle is an example of too long a cadence while a weekly cycle may prove too short a cadence. Determining the right frequency is a combination of the organization’s maintenance windows and the need to consume a particular fix.

This concept is best summarized in the following table:

Table 3-1 When to Apply Oracle Fusion Applications Patches

Type One-Offs AOOs CPU Patch Bundles

Technical

Apply only to address critical issue, otherwise wait for patch bundle

Not applicable

Apply immediately upon release

Ideally apply monthly. Less frequently than every other month is not recommended

Functional

Apply only to address critical issue, otherwise wait for update bundle

Apply when project timelines or urgency of fix do not alight with release of patch bundle, otherwise wait for update bundle

Not applicable

Ideally apply monthly. Less frequently than every other month is not recommended

Functional Patching Modes and How They Affect Outage Windows

All technical patches, whether one-off, CPU, or P4FA bundles, require that the system be taken offline for the patching process. This comprises a true “outage window.”

For functional patches, Patch Manager supports three different modes in which patches might be applied: offline, online, or hot patching. Some functional patches and patch bundles support one mode, others require another. The appropriate mode can be found using the Patch Manager validate option. Each mode has a different effect on the extent and timing of an outage window.

Each of the three modes is described in detail in the Patch Modes section.

Impact Assessment Strategies

The following are some of the tools and strategies for assessing the effect that patching will have on a production environment:

  • READMEs: The Patch READMEs describe the bug fixes and system areas addressed by the patch.

  • Test Environments: Applying the patch on a test or patching environment and performing subsequent regression tests will help determine the timing and impact of a patch on a given Oracle Fusion Applications installation, and permit targeted planning for patching subsequent environments.

  • Reports: The Patch Impact Report is available for functional patches. For more information, see the Patch Impact Report section.

Patch Impact Report

The Patch Impact report compares the contents of the patch to be applied with the files that currently exist on the system. The report shows a complete picture of what file system changes will occur when the patch is applied. Plan the system downtime by viewing the servers that will be affected by the patch, along with any manual deployment actions that are required after the patch is applied. This report reads the patch metadata, local patch inventory, and the current view snapshot.

The Patch Impact report displays the impact information about a patch in the following section:

Bug Fixes

This section provides the following information about the bug fixes included in the patch:

  • Bug Number: The number of the bug fix or patch

  • Bug Description: The description of the bug fix or patch

  • Exists in Oracle home: Whether the bug fix or patch was already applied (Yes or No)

Prerequisite Bug Fixes

This section provides the following information about patches that must be applied before the current patch can be applied:

  • Bug Number: The number of the prerequisite bug fix or patch

  • Bug Description: The description of the prerequisite bug fix or patch

  • Exists in Oracle home: Whether the prerequisite bug fix or patch was already applied (Yes or No)

Prerequisite Bug Fixes Not in FA_ORACLE_HOME

This section provides the following information about patches that must be applied before the current patch can be applied. These patches are not applied to FA_ORACLE_HOME.

  • Bug Number: The number of the prerequisite bug fix or patch

  • Bug Description: The description of the prerequisite bug fix or patch

  • Exists in Oracle home: Whether the prerequisite bug fix or patch was already applied (No)

Product Families Impacted

This section provides the following information about which product families are impacted by the patch:

  • Product Family: The name of the product family (component) that is updated by the patch

  • Product: The name of the product that is updated by the patch

  • LBA: The logical business area that is updated by the patch

Servers Impacted

This section provides the following information about which servers will be impacted by the patch. Note that all artifacts in the patch are copied, but server life cycle actions occur only for those product families that have been deployed during the provisioning process.

  • Artifact Type: The type and name of the artifact included in the patch

  • Domain (Servers): The servers that are affected by the artifacts in the patch

  • Expectation/Impact: The description of what servers must be running, what actions will be taken during the patch apply phase by Patch Manager, and what manual actions must be taken

Files Included in the Patch

This section provides the following information about the files that are included in the patch:

  • File Name: The name of the file

  • File Type: The type of the file

  • File Version: The version of the file

For detailed information about the Patch Impact Report parameters, see the Patch Impact Report Parameter Details section.

Patch Impact Report Parameter Details

The following table describes the parameters used by the Patch Impact report:

Table 3-2 Parameters Used by the Patch Impact Report

Parameter Mandatory Description

patchtop

Yes

Identifies the directory where the patch is unzipped

outputfile

No

Sends the report output to the specified file after this parameter. An existing file name cannot be used. If this parameter is not used, no output file is created

logfile

No

Overrides the default log file name and sends the processing information to the specified file, under the APPLICATIONS_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR directory. If an existing file name is entered, the output is appended to the file. If this parameter is not used, the utility generates a log file under APPLICATIONS_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR using this naming convention:

FAPatchManager_report-patchimpact_timestamp.log

loglevel

No

Records messages in the log file at the specified level. See the Oracle Fusion Applications Patch Manager Logging section

reportwidth

No

Sets the column width to either 80 columns or 132 columns by specifying NORMAL or WIDE. The default value is 80 columns, or NORMAL

Run the Patch Impact Report

The Patch Impact report can be run when applying only one patch or multiple patches downloaded in a patch plan. Before running the Patch Impact Report, ensure that the snapshot is current for the environment.

Use the following syntax to run the Patch Impact report for a single patch:

(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh report -patchimpact -patchtop   
 path_to_unzipped_patch [optional parameters]

Use the following syntax to run the Patch Impact report for a single patch:

(UNIX)FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh report -patchimpact -grouptoptop_directory_for_patches -patchingplanfull_path_to_patching_plan [optional parameters]

Create a Patch Tracking Sheet

It is important to document the patching plan and outcomes, to enhance team coordination and knowledge transfer, and to be used for future reference in system maintenance. Every organization creates its own documentation standards. A Patch Tracking Sheet is one way of centralizing information about the patches applied to a Fusion Applications environment. In general, the tracking sheet can be generated in almost any word editor or spreadsheet software, or even as a hard copy on paper. The basic and most minimal information which should be tracked and retained as follows:

  • Requestor of the patch

  • Patch Number

  • Description of the patch

  • Date Applied

  • Target Environment

  • Time Taken to Apply

  • Issues Experienced

  • Special Instructions

  • Comments

  • Oracle Service Request # (if applicable)

Summary: Principles for Scheduling Maintenance

  • Stay Current: Stay as current as possible with patching. Patches are cumulative, so if they are delayed, the system downtime for installation can expand considerably. On the other hand, patching disrupts normal system activities and patch releases can be frequent, so discernment is required. The best practice is to establish a cadence that minimizes the amount of downtime taken to apply the various patches while maximizing the uptake of key patches for the organization. 

  • Patch in Order: Technical patches (P4FA) are always installed before functional patches. Patches are release-specific, cumulative, and sequential.

  • Plan Outage Windows: Technical patching always requires that users exit the system, tasks and jobs be stopped, and the servers be taken offline. This comprises a true outage window. Functional patches can sometimes be applied as hot patches, in which no servers are shut down, users may remain in the system, and there is a defined time frame to allow background processes and jobs to complete before the patch is launched. In these cases, the “outage” is minimal. Some functional patches inhabit a grey area of online patching, in which servers remain up, but no users, transactions, or tasks are allowed. Have a clear understanding of which patching mode will be in use for any given patching session.

  • Use a patching test environment: At a minimum, test patches on a non-production system to determine the necessary timing, analyze the patching effects and perform regression testing before applying the patch to a production system. Some organizations, in addition to test, dev, and prod environments, also establish a dedicated patch environment.

  • Ensure communication: Establish an advisory forum with stakeholders and schedule notifications with all affected parties.

  • Document the plan: Use tracking sheets and other systems to ensure that the patching team is coordinated and that patches are documented for future reference.

Plan Test for Applied Patches

Patch application follows a progressive environment path. Although every company is different, the usual path consists of Development, Test , QA, and Prod.

Development: The goal during the Development stage is to perform unit testing of the patch. The following is an example of a Development stage procedure:

  1. An issue is identified and a Service Request (SR) is filed.

  2. Oracle Support recommends to apply a patch.

  3. The users are able to reproduce the issue in the Development environment.

  4. The patch is applied to the Development environment.

  5. The issue cannot be reproduced in the Development environment anymore.

  6. If the issue is still present, the patch is discarded and the SR is updated accordingly.

Test: The goal during the Test stage is to perform integration testing of the patch. Below is an example of a Test stage procedure:

  1. After unit testing is successful, the patch is applied to the Test environment.

  2. The users test the functionality of all other components affected by the patch payload, not only the issue at hand. Normally, there is documentation describing all the tests needed and the expected output.

QA: The goal during the QA stage is to provide quality assurance testing also known as user acceptance testing. Quality assurance or user acceptance testing is a final test to verify whether the patch is approved to be applied to production. Below is an example of a QA stage procedure:

  1. After integration testing is successful, the patch is applied to the QA environment.

  2. The key users will execute and sign-off on a standardized set of tests to be sure the patch is not breaking the functionality of the system. This set of tests is the same or an updated version of the tests used when the system was deployed initially. This test set is also known as regression test and can be applied manually or automatically.