24 Patching Oracle Management Service and the Repository

Note:

The patching methodology discussed in this chapter can only be used with Enterprise Manager Cloud Control Release 13.2.0.0.0 and later.

OMSPatcher automates the patching process by generating custom patching instructions based your particular environment and then automatically applies the patch.

This chapter covers the following topics:

OMSPatcher Automation

With OMSPatcher, you can automatically patch a typical OMS configuration (core, plug-in homes) with minimal intervention.

OMSPatcher performs many of the pre-patch checks such as:

  • Configuration-based prerequisite checks

  • Patch-based binary prerequisite checks

OMSPatcher performs end-to-end configuration patching. Configuration patching is the process of patching a target based on its configuration. By incorporating the configuration information into the patch process, OMSPatcher is able to simplify patching tasks by automating most of the steps.

Supported OMS Configurations and OMSPatcher Patchability

  • Single OMS – OMS application that runs from a single OMS instance of the system. OMSPatcher performs patching and deployment operations

  • Multiple OMS – OMS applications that run on two or more machines. The OMSs are connected by the Oracle WebLogic domain and separate managed servers. There is a one-to-one mapping between the managed servers and the separate OMS bits residing on a single machine. OMSPatcher provides auto-generated bash scripts (one per OMS instance) for UNIX based systems. For Windows, it only provides context-sensitive steps (text and HTML). For both cases, administrator needs to follow the steps given by OMSPatcher.

  • Single Instance Database or Real Application Cluster - shared or Real Application Cluster (RAC)

Example: Multi-OMS System

The following figure illustrates a multi-OMS deployment. The following terms are used:

  • Administrator: Person installing patches to the OMS core and plug-in homes.

  • Local OMS: OMS instance on which the administrator runs OMSPatcher.

  • Remote OMS: OMS instances on other machines (within the same OMS domain as the local OMS) where the administrator has not started any patching operations.

Figure 24-1 Simple Multi-OMS System


OMS architecture

For a single OMS system (primary), OMSPatcher will execute the patching steps. For a multi-OMS UNIX system, OMSPatcher generates bash scripts for execution, one per OMS instance; follow the instructions given by OMSPatcher to find those scripts. For Windows multi-OMS systems, OMSPatcher will generate customized patching instructions/commands for the environment in text and HTML formats; administrators must execute these instructions to patch the various Oracle Management Services.

NextGen OUI Inventory Configurations

Apart from the target or instance-based configuration, OMSPatcher utilizes installation configuration relationships established in the NextGen OUI inventory as core and plug-in feature-sets. A typical OMS 13c home is organized as follows:

<Middleware Home>
    |_____<CORE_BITS>         
    |_____<PLUGINS_DIRECTORY>
        	|_____oracle.sysman.db.oms.plugin_13.3.1.0.0
        	|_____oracle.sysman.emas.oms.plugin_13.3.1.0.0
        	     .
               .
               .

Supported Patch Format

Beginning with Enterprise Manager Release 12.1.0.3, Enterprise Manager patches have been converted to a System patch format in order to support patch automation.

What is a System Patch?

A System patch contains several sub-patches whose locations are determined by a file called bundle.xml in the top level directory of the patch. The sub-patches are intended for different sub-systems of a system that correspond with the OMS core and plug-in home organization.

A typical System patch format is organized as follows:

<System patch location - directory>
|_____ Readme.txt (or) Readme.html
       bundle.xml
       automation
	      |_____ apply_automation.xml
         |_____rollback_automation.xml
       Sub-patch1
                |_____ etc
                       |_____config  
                                |_____ inventory.xml
                                |_____ actions.xml
                                |_____ artifact_apply.xml
                                |_____ artifact_rollback.xml
                |_____ files/Subpatch1 'payload'
       Sub-patch2
                |_____ etc
                       |_____config  
                                |_____ inventory.xml
                                |_____ actions.xml
                                |_____ artifact_apply.xml
                                |_____ artifact_rollback.xml
                |_____ files/Subpatch1 'payload'

For Enterprise Manager release 12.1.0.2 or below, OMSPatcher (earlier known as opatchauto) is not supported for the released one-off patches. For these older releases, you must use OPatch and follow the patch README instructions.

Supported Patching Methodologies

OMSPatcher supports rolling mode only for System patches without any automation (binary-only patching through OMSPatcher). For all other artifacts (MRS, SQL), OMSPatcher only supports complete system downtime patching operations.

Refer to the patch README for the explicit information on supported patching methodologies.

Required OMSPatcher Parameters

OMSPatcher for the Enterprise Manager OMS will prompt for the following input parameters when performing patching operations. These parameters were determined at the time of Enterprise Manager installation.

  • Oracle WebLogic Admin Sever URL & port number

  • Oracle WebLogic Administration Server username

  • Oracle WebLogic Administration Server password

Because OMSPatcher requires this input for each patching operation, OMSPatcher provides the ability to encrypt the username and password via WebLogic encryption APIs and pass this information using a property file when running OMSPatcher apply and rollback operations. The next section discusses how to create a property file.

Creating a Property File

The automated patching functionality achieved using OMSPatcher expects WebLogic Administration Server URL and credentials as an input for patching and configuration detection operations. Primarily, the WebLogic Administration server is the host that manages the Managed Server where the OMS instance is deployed. If you do not want to set the credentials every time you are prompted while patching the OMS, you can update the property file. OMSPatcher allows you to repeatedly provide the inputs using property file option.

Note:

If the OMS's are configured with virtual hostnames, you first need to set the following environment variable before executing the createkeys.sh command (Step 1).

export WLST_PROPERTIES="-Dweblogic.security.SSL.ignoreHostnameVerification=true"
  1. Run the following script to create the WebLogic encrypted configuration and key files.

    On UNIX:

    $ OMSPatcher/wlskeys/createkeys.sh -oh <full path of platform home> -location <location to put the encrypted files>

    On Windows:

    $ OMSPatcher\wlskeys\createkeys.cmd -oh <full path of platform home> -location <location to put the encrypted files>

    When prompted, enter the credentials of the Oracle WebLogic Administration Server that manages the Managed Server on which OMS instance is deployed. Two files are generated with the file names: config and key.

  2. Create the property file with the following entries:
    AdminServerURL=t3s://<host address from where admin server is running>:<port of the admin server>
    AdminConfigFile=<'config' file location>
    AdminKeyFile=<'key' file location>
    

    The values for host address and port of admin server can be located by running the following 'emctl command' on an Oracle Home.


    OMSPatcher sample output

    Following is the example of how a property file (constructed by the above mentioned guidelines) should appear:

    AdminServerURL=t3s://my_admin_server.oracle.com:7101
    AdminConfigFile=/scratch/patch/oms_install_dir/middleware/oms/config/config
    AdminKeyFile=/scratch/patch/oms_install_dir/middleware/oms/config/key
    

    Note:

    To retrieve the WebLogic Administration Server URL details, run the following commands on the OMS home that you are patching:

    On Unix:

    $ORACLE_HOME/bin/emctl status oms -details

    On Windows:

    %ORACLE_HOME%\bin\emctl.bat status oms -details

    The command output contains the WebLogic Administration Server details. Here is an example on how to construct the URL with these output details.

    Example:

    WLS Domain Information
     
    Domain Name : GCDomain
    Admin Server Host : my_wls.oracle.com
    Admin Server HTTPS Port: 7103

    To construct the Administrator Server URL, use the following syntax:

    t3s://<admin server host>:<port>

    In this example, the URL translates as follows:

    t3s://my_wls.oracle.com:7103

Prerequisites for Running OMSPatcher

Before running an OMSPatcher patching session, you must ensure the following configuration and inventory-based prerequisites are satisfied: Configuration-based conditions that have to be honored for OMS automation is given below.

  • The Enterprise Manager Software library must be configured.

  • The Oracle WebLogic Administration Server that controls the OMS instance (currently to be patched) through a managed server must be up and running.

  • Ensure that the Oracle Database, which houses the OMS Management Repository, and its listener are up and running.

  • Ensure that you have the latest version of the OMSPatcher in the OMS platform home of each host.

    If you do not have the latest OMSPatcher version, follow the instructions outlined in the My Oracle Support note 2135028.1 available at:

    https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=529643231574036&id=2135028.1&_adf.ctrl-state=c5dyo92tl_489#REF

  • Check your patch README to determine whether there are any specific prerequisites to be executed based on patch and patching methodologies.

Checking System Prerequisites

Note:

To run omspatcher commands, ensure that 'ORACLE_HOME/OMSPatcher is included in the PATH environment variable.

To make sure all prerequisite checks pass and no errors occur during the OMSPatcher patching session, Oracle recommends running the following commands on each OMS instance (in your OMS system).

omspatcher apply <PATCH_LOC> -analyze

Must be run from the System patch location (for apply operations)

Note:

OMS systems need not be shut down when running apply -analyze.

Note:

Check the Patch README and the instructions given for chosen patching methodologies.

OR

omspatcher rollback -analyze -id <comma (,) separated list of sub-patches to be rolled back for System patch>  

Note:

In order to roll back all sub-patches together, all sub-patches should be from same system patch.


patch output

Example 24-1 OMSPatcher rollback -analyze output

-------------------------------


omspatcher rollback -id 1111137 -analyze
OMSPatcher Automation Tool
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
 
 
OMSPatcher version : 13.8.0.0.0
OUI version        : 13.8.0.0.0
Running from       : /scratch/mw
Log file location  : /scratch/mw/cfgtoollogs/omspatcher/1111137_Nov_11_2015_22_54_57/rollback2015-11-11_22-54-57PM_1.log
 
OMSPatcher log file: /scratch/mw/cfgtoollogs/omspatcher/SystemPatch/omspatcher_2015-11-11_22-55-02PM_analyze.log
 
Please enter OMS weblogic admin server URL(t3s://myserver.myco.com:7101):> 
Please enter OMS weblogic admin server username(weblogic):> 
Please enter OMS weblogic admin server password:> 
 
Sub-patch(es) " 1111137 " are part of the OMS System patch.
Oracle Home: /scratch/mw, Sub-patch(es): [1111137, 1111126]
Do you want to rollback sub-patch(es) "1111137" only? [y|n]
y
User Responded with: Y
 
Configuration Validation: Success
 
Running rollback prerequisite checks for patch(es) "1111137" and Oracle Home "/scratch/mw"...
Sub-patch(es) "1111137" are successfully analyzed for Oracle Home "/scratch/mw"
 
Complete Summary
================
 
All log file names referenced below can be accessed from the directory "/scratch/mw/cfgtoollogs/omspatcher/2015-11-11_22-54-57PM_SystemPatch_1111192_1"
 
Prerequisites analysis summary:
-------------------------------
 
The following sub-patch(es) are rollbackable:
 
Featureset   Sub-patches                Log file
----------   -----------                --------
oracle.sysman.top.oms       1111137   1111137_RollbackPrereq2015-11-11_22-55-25PM_2.log
 
Log file location: /scratch/mw/cfgtoollogs/omspatcher/SystemPatch/omspatcher_2015-11-11_22-55-02PM_analyze.log
 
OMSPatcher succeeded.
bash-3.2$

Note:

Once the analysis finishes, you can refer to the OMSPatcher log to see what steps would be executed by OMSPatcher in non -analyze mode. The log file contains references to the HTML and text output file HTML containing detailed steps.

Using OMSPatcher

OMSPatcher must be run from the platform home of the OMS being patched. To run OMSPatcher commands from any directory include <ORACLE_HOME__PATH>/OMSPatcher in the PATH environment variable. The ORACLE_HOME environment variable must be set as the platform home or provided using the omspatcher "–oh" option. For example:

omspatcher apply <patch>

Minimum Required OMSPatcher Version: 13.8.0.0.0

Ensuring You Have the Latest Version of OMSPatcher

OMSPatcher is the patching utility that executes end to end patching procedure for OMS Patches. Ensure that the latest version of OMSPatcher is available on all instances of OMS platform homes.

To check the version of OMSPatcher residing on the system, run the following command:

omspatcher version

To get the latest OMSPatcher version, follow the instructions outlined in the My Oracle Support note 2135028.1 available at:

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=277259559046496&id=2135028.1&_afrWindowMode=0&_adf.ctrl-state=4eefxg576_200

Ensuring You Have the Latest Version of OPatch

OMSPatcher uses the OPatch utility to apply the patch. For this reason, you must ensure that you have the latest version of OPatch on all instance of OMS platform homes. To check the version of OPatch residing on the system, run the following command. Ensure to execute the command after including ORACLE_HOME/OPatch in the PATH environment variable.

opatch version

To download the latest version of OPatch, follow the instructions outlined in the My Oracle Support note 224346.1 available at the following location:

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=224346.1

Minimum Required OPatch Version: 13.8.0.0.0

Patching Quickstart

Using OMSPatcher typically involves the following phases:

1. Determining Whether Your System Meets OMSPatcher System Requirements

Run omspatcher apply -analyze

The apply -analyze command simulates an OMSPatcher apply session by running all prerequisite checks, when possible, without making changes to the system (either bits or configurations). This command does not apply the patch.

See "Prerequisites for Running OMSPatcher" for additional information.

2. Determining What System Patches Currently Exist on Your System

Run omspatcher lspatches

See "lspatches" for more information.

3. Obtaining Patches from My Oracle Support (MOS)

OMSPatcher requires that the required platform or plug-in System patches be obtained from My Oracle Support and downloaded to the OMS instance on which OMSPatcher is to be run.

See "My Oracle Support: Searching for Patches" for more information.

4. Applying a Patch

Run omspatcher apply <patch>

The apply command applies all patches within a specified System patch to the platform home from which omspatcher command is run.

See "Running omspatcher apply" for more information.

5. Deinstalling Individual Sub-patches of a System Patch

Run omspatcher rollback -id <list of comma separated sub-patches of System patch>

Note:

For a complete list of sub-patches of the System patch, refer to the patch README.

If, after applying the patch, the system is not stable, the most likely cause is the patch itself. Contact Oracle Support. They will recommend that you remove the patch using the omspatcher rollback command.

See "Running omspatcher rollback" for more information.

My Oracle Support: Searching for Patches

The first step in the patching process is to determine what patches you need from My Oracle Support (MOS). MOS is the single source of truth for patching. You can access MOS at the following location:

https://support.oracle.com

Once you have logged in, you have access to interactive support tools and information that simplify searching for and obtaining the requisite patches for your Oracle environment. .

My Oracle Support contains many features and capabilities that are grouped under tabs across the top of the application. Of primary interest is the Patches and Updates tab. From this tab you can search for the patches based on the OMS patch area (core, plug-in, or combination).

Running omspatcher apply

Once you have downloaded the patch, see the patch README for explicit patch details and instructions on applying the patch. You can find the README at the following location

<System patch location>/README.txt (or) README.html

As you step through the patching operations in the README, running omspatcher apply (depending on the configuration that is patched, primary or standby) will generate a custom, environment-specific version of the README for patching operations for the primary site multi-OMS or standby site OMS systems. For a primary site single OMS system, running omspatcher apply will perform patching and deployment operations.

On your local OMS instance, run the following command from the top level System patch directory:

omspatcher apply <patch>

Note:

Unlike omspatcher analyze, you should not run omspatcher apply on every OMS instance. OMSPatcher will either execute all patching and deployment operations, or will generate environment-specific steps that include complete configuration aspects of the System.

For a multi-OMS UNIX system, OMSPatcher generates bash scripts for execution, one per OMS instance; follow the instructions given by OMSpatcher to find those scripts. For Windows multi-OMS systems, OMSpatcher will generate customized patching instructions/commands for the environment in text and HTML formats; administrators must execute these instructions to patch the various OMSs.

Applying a Bundle Patch

If you want to apply a Bundle Patch that is available on top of 13c Release 4 release on an Oracle Home, perform the following steps:

Note:

Since omspatcher is located in ‘$ORACLE_HOME/OMSPatcher’, ensure that the directory is included in the path before running the commands.
  1. Execute omspatcher in bitonly mode for the Bundle Patch.

    For example,
    $ORACLE_HOME/OMSPatcher/omspatcher apply -bitonly
    
    OMSPatcher Automation Tool
    Copyright (c) 2017, Oracle Corporation.  All rights reserved.
    OMSPatcher version : 13.8.0.0.3
    OUI version        : 13.9.4.0.0
    Running from       : $ORACLE_HOME
    Log file location  : $ORACLE_HOME/cfgtoollogs/omspatcher/opatch<timestamp>_1.log
    OMSPatcher log file: $ORACLE_HOME/cfgtoollogs/omspatcher/1111141/omspatcher_<timestamp>_deploy.log
    
    WARNING: OMSPatcher has been invoked with 'bitonly' option but the System patch provided has deployment metadata. 
    Invocation in 'bitonly' mode will prevent OMSPatcher from deploying artifacts. 
    Do you want to proceed? [y|n]
    y
    User Responded with: Y
    
    Running apply prerequisite checks for sub-patch(es) "1111141" and Oracle Home "$ORACLE_HOME"...
    Sub-patch(es) "1111141" are successfully analyzed for Oracle Home "$ORACLE_HOME"
    To continue, OMSPatcher will do the following:
    [Patch and deploy artifacts]   : 
    Do you want to proceed? [y|n]
    y
    User Responded with: Y
    Applying sub-patch(es) "1111141"
    Please monitor log file: $ORACLE_HOME/cfgtoollogs/opatch/opatch<timestamp>_1.log
    
    Complete Summary
    ================
    All  log file names referenced below can be accessed from the directory "$ORACLE_HOME/cfgtoollogs/omspatcher/<timestamp>_SystemPatch_1111141_1"
    Patching summary:
    -----------------
    Binaries of the following sub-patch(es) have been applied successfully:
    Featureset   Sub-patches      Log file                        
    ----------   -----------      --------  
    oracle.sysman.top.oms_13.4.0.0.0       1111141      1111141_opatch<timestamp>_1.log
    --------------------------------------------------------------------------------
    The following warnings have occurred during OPatch execution:
    1) OMSPatcher has been invoked with 'bitonly' option but the System patch provided has deployment metadata.
    Invocation in 'bitonly' mode will prevent OMSPatcher from deploying artifacts.
    --------------------------------------------------------------------------------
    OMSPatcher Session completed with warnings.
    Log file location:$ORACLE_HOME/cfgtoollogs/omspatcher/1111141/omspatcher_<timestamp>_deploy.log
    OMSPatcher completed with warnings.
    
    
  2. Run omspatcher lspatches command to list all the sub-patches applied in Step 1.

    Syntax: omspatcher lspatches | grep "bp_id"

    For example,
    ************omspatcher Trace for reference
    $omspatcher lspatches | grep 30684860
    oracle.help.ohw.rcf/12.2.1.3.0                     Core
            N/A                 30684860             ADF BUNDLE PATCH
            12.2.1.3.0(ID:191219.0902.S)
    oracle.jrf.adfrt.javatools/12.2.1.3.0              Core
            N/A                 30684860             ADF BUNDLE PATCH
            12.2.1.3.0(ID:191219.0902.S)
    oracle.jrf.adfrt/12.2.1.3.0                        Core
            N/A                 30684860             ADF BUNDLE PATCH
            12.2.1.3.0(ID:191219.0902.S)
    oracle.help.ohw.share/12.2.1.3.0                   Core
            N/A                 30684860             ADF BUNDLE PATCH
            12.2.1.3.0(ID:191219.0902.S)
    oracle.jrf.adfrt.help/12.2.1.3.0                   Core
            N/A                 30684860             ADF BUNDLE PATCH
            12.2.1.3.0(ID:191219.0902.S)
    

    Note:

    The last column lists all the sub-patches applied with the Bundle Patch.
  3. Run omspatcher commit command with any one applied sub-patch-id.

    Note:

    The output of Step 2 lists all the applied sub-patches. You can pick any sub-patch id and run omspatcher commit command.

Syntax: omspatcher commit -id <subpatch_id>

For example,
OMSPatcher/omspatcher commit -id 30684860

OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.

OMSPatcher version : 13.8.0.0.3
OUI version        : 13.9.4.0.0
Running from       : /scratch/XXX
Log file location  : /scratch/XXX/cfgtoollogs/omspatcher/opatch<timestamp>_1.log

OMSPatcher will now mark the patch "30684860" as auto-executed.
Log file location: /scratch/XXX/cfgtoollogs/omspatcher/opatch<timestamp>_1.log
OMSPatcher succeeded.

Running omspatcher rollback

See the patch README for explicit patch details and instructions on deinstalling the patch. You can find the README at the following location

<System patch location>/README.txt (or) README.html

As you step through the patch deinstallation operations in the README, running omspatcher rollback (depending on the configuration that is patched, primary or standby) will generate a custom, environment-specific version of the README for patching operations for the primary site multi-OMS or standby site OMS systems. For a primary site single OMS system, running omspatcher rollback will perform the deinstallation operations.

On your local OMS instance, run the following command from the top level System patch directory:

omspatcher rollback -id <list of comma separated sub-patches of System patch

Note:

  • Unlike omspatcher analyze, you should not execute the omspatcher rollback command on every OMS instance. OMSPatcher will either execute all patching and deployment operations, or will generate environment-specific steps that include complete configuration aspects of the System.

  • The list of sub-patches within the System patch can be retrieved from patch README.

    The list of sub-patches listed in System patch README may differ from the patches that are actually installed. During System patch installation, some sub-patches may be skipped (not installed).

For a multi-OMS UNIX system, OMSPatcher generates bash scripts for execution, one per OMS instance; follow the instructions given by omspatcher to find those scripts. For Windows multi-OMS systems, OMSPatcher will generate customized patching instructions/commands for the environment in text and HTML formats; administrators must execute these instructions to patch the various OMSs.

Running omspatcher lspatches

After the patch is applied or rolled back, you can run the omspatcher lspatches command to generate a comprehensive Component type - patches map of the OMS homes and installed patches.

bash-3.2 omspatcher lapatches
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation. All rights reserved.

OMSPatcher version : 13.8.0.0.2
OUI version : 13.9.1.0.0
Running from : /scratch/em22495213/mw
Log file location : /scratch/em22495213/mw/cfgtoollogs/omspatcher/opatch2018-06-18_23-01-52PM_1.log

Component Name/Version                   Component Type   System Patch   (Sub)-Patches   Patch Description
------------------------                 -------------    -------------  --------------  ------------------
oracle.wls.shared.with.cam/12.1.3.0.0    Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417

oracle.webcenter.pageeditor/12.1.3.0.0   Core             N/A             20882747        One-off

oracle.xdk.jrf.jaxp/12.1.3.0.0           Core             N/A             19345252        One-off

oracle.wls.libraries/12.1.3.0.0          Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417
                                                          N/A             23527146        One-off
                                                          N/A             25832897        One-off

oracle.css.mod/12.1.3.0.0                Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417

oracle.opss.wls/12.1.3.0.0               Core             N/A             27074880        OPSS Bundle Patch 12.1.3.0.171124

oracle.xdk.jrf.xmlparserv2/12.1.3.0.0    Core             N/A             19345252        One-off
dponnapp: oracle.jrf.iau/12.1.3.0.0      Core             N/A             27074880        OPSS Bundle Patch 12.1.3.0.171124

oracle.opss.jrf/12.1.3.0.0               Core             N/A             27074880        OPSS Bundle Patch 12.1.3.0.171124

oracle.ldap.rsf/12.1.3.0.0               Core             N/A             27369653        One-off

oracle.perlint/5.14.4.0.0                Core             N/A             25412962

oracle.webservices.base/12.1.3.0.0       Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417

oracle.wls.admin.console.en/12.1.3.0.0   Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417
oracle.wls.core.app.server/12.1.3.0.0    Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417
                                                          N/A             19982906        One-off

oracle.wls.workshop.code.completion.supp Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417
ort/12.1.3.0.0

oracle.webservices.orawsdl/12.1.3.0.0    Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417

oracle.wls.wlsportable.mod/12.1.3.0.0    Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417

oracle.rcu.iau/12.1.3.0.0                Core             N/A             27074880        OPSS Bundle Patch 12.1.3.0.171124

oracle.wls.server.shared.with.core.engin Core             N/A             27419391        WLS PATCH SET UPDATE 12.1.3.0.180417
e/12.1.3.0.0

oracle.jrf.adfrt/12.1.3.0.0              Core             N/A             27839641        One-off




oracle.javavm.jrf/12.1.0.2.0             Core             N/A            28042003       One-off
                                                          N/A            20741228       JDBC 12.1.3.1 BP1
                                                          N/A            26933408       One-off

oracle.bi.xdo/12.1.3.0.0                 Core             N/A            23519804       One-off

oracle.wls.clients/12.1.3.0.0            Core             N/A            27419391       WLS PATCH SET UPDATE 12.1.3.0.180417
                                                          N/A            25832897       One-off

oracle.fmwconfig.common.wls.shared/12.1. Core             N/A            27419391       WLS PATCH SET UPDATE 12.1.3.0.180417
3.0.0
oracle.bali.cabo/12.1.3.0.0              Core             N/A            18814458       One-off

oracle.fmwconfig.common.shared/12.1.3.0. Core             N/A            27419391       WLS PATCH SET UPDATE 12.1.3.0.180417
0

oracle.webservices.wls/12.1.3.0.0        Core             N/A            27419391       WLS PATCH SET UPDATE 12.1.3.0.180417

oracle.ohs2/12.1.3.0.0                   Core             N/A            27244723       One-off

oracle.wls.libraries.mod/12.1.3.0.0      Core             N/A            27419391       WLS PATCH SET UPDATE 12.1.3.0.180417

oracle.wls.common.nodemanager/12.1.3.0.0 Core             N/A            27419391       WLS PATCH SET UPDATE 12.1.3.0.180417

oracle.wlsplugins/12.1.3.0.0             Core             N/A            20442348       One-off

oracle.opss.core/12.1.3.0.0              Core             N/A            27074880       OPSS Bundle Patch 12.1.3.0.171124


NOTE: N/A indicates that the subpatch mentioned in the Subpatches column was applied as a one-off patch and not as part of any system patch.

OMSPatcher has saved inventory details for the above component at below location.
"/scratch/em22495213/mw"

For more details on installed patch(s), run the following command:  "/scratch/em22495213/mw/OPatch/opatch lsinventory -details"

OMSPatcher succeeded.


Running omspatcher version

To determine the version numbers of the various OMSPatcher utilities (OPlan, OsysModel) that reside on your system, you can run omspatcher version.

bash-3.2$ omspatcher version
OMSPatcher Version: 13.8.0.0.0
OPlan Version: 12.1.0.2.2
OsysModel build: Wed Mar 21 18:20:48 PDT 2018

Patching a Standby OMS System

If you have configured a standby OMS for High Availability, refer to the chapter on "Enterprise Manager Disaster Recovery" and the appendix on Standby OMSs Using Standby WebLogic Domain" both of which can be found in the Oracle Enterprise Manager Advanced Installation and Configuration Guide.

OMSPatcher Command Syntax

This section provides a comprehensive listing and description of all OMSPatcher commands used to patch an OMS.

Note:

OMSPatcher commands must be run from the OMS Middleware home.

OMSPatcher Commands

The OMSPatcher commands are run from the OMS Middleware home out of the OMSPatcher directory. The Middleware home must be set as $ORACLE_HOME. In the following generic example, an OMSPatcher command is run from a Middleware home.

omspatcher apply <PATH_TO_PATCH_DIRECTORY>

where <PATH_TO_PATCH_DIRECTORY> is the full path to the System patch top level directory.

You can view online help for any command (except version) by specifying the -help option.

OMSPatcher Automation Tool
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


 Usage: omspatcher [ -help ] [ -analyze ] [ command ]

            command := apply
                       rollback
                       checkApplicable
                       lspatches
                       version
                       saveConfigurationSnapshot
 
 <global_arguments> := -help       Displays the help message for the command.
                       -analyze    Print the actions, steps to be performed without any execution.

 example:
   'omspatcher -help'
   'omspatcher apply -help'
   'omspatcher rollback -help'
   'omspatcher checkApplicable -help'
   'omspatcher lspatches -help'
   'omspatcher saveConfigurationSnapshot -help'


OMSPatcher succeeded.

omspatcher consists of six primary commands.

  • apply

  • rollback

  • checkapplicable

  • saveConfigurationSnapshot

  • lspatches

  • version

Apply

Apply a System patch to OMS instance homes. You must specify the patch location or the current directory will be used as the patch location.

Note:

OMSPatcher must be run from the platform home. ORACLE_HOME environment variable must be set as the platform home or provided using the –oh option.

You must run the Apply command directly from the System patch location.

When running omspatcher apply, you will be prompted the following:

  • WebLogic Admin Server URL of the primary OMS (or standby OMS)

  • Username and Password

Silent interaction is supported by using the silent and property_file options.The standby option should be used if a stand by OMS system is patched. OMSPatcher can pass 'x=y' properties through the command line. See Table 24-2.

Syntax

omspatcher apply <System patch location>
							[-custCertPath <Path to customer optional certificate>]
							[-jre <Path to JRE>] [-nonrolling]
							[-invPtrLoc <Path to oraInst.loc>]
							[-property_file <Path to property file>]
							[-analyze] [-silent] [-oh <Platform home path>]
							[-standby]

Parameters

<System patch location>

Path to the location of the patch. If the patch location is not specified, then the current directory is taken as the patch location. The patch can only be a System patch.

Apply Command Options

Table 24-1 Apply

Option Description

jre

This option tells OMSPatcher to use JRE (java) from the specified location instead of the default location under Oracle Home.

invPtrLoc

Used to locate the oraInst.loc file. Needed when the installation used the -invPtrLoc flag. This should be the path to the oraInst.loc file.

property_file

The user-defined property file for OMSPatcher to use. The path to the property file should be absolute.

The keys for 'omspatcher' are:

'AdminConfigFile' - Encrypted file for Admin Server user of OMS instance domain.

'AdminKeyFile' - Encrypted file for Admin Server password of OMS instance domain.

'AdminServerURL' - Admin Server URL of OMS instance domain.

(Example: t3s://<host address>:<port number>)

The Key, value pair is of the format 'x=y' where 'x' is omspatcher understood key and each pair is separated by new line in the property file. This option is typically used for silent operations.

This option is very useful for silent mode of 'omspatcher' invocation. In order to create encrypted files for WebLogic admin server username & password. Please use

$ORACLE_HOME/OMSPatcher/wlskeys/createkeys.sh (.cmd for windows) to get the files and load it through a custom file by 'property_file' option.

NOTE: For Windows, please make sure that directories and files in the path are separated by "\\" in the property file.

analyze

Just prints out the actions without any configuration/binary change through omspatcher.

silent

This suppresses any user-interaction.

oh

The location of EM platform home. This overrides the ORACLE_HOME environment variable.

custCertPath

This option tells OMSPatcher to use the certificate from the specified location.

nonrolling

Apply and deploy the patch in non-rolling fashion, provided it is supported by the patch.

standby

This option should be used for standby OMS patching operations.

Apply Command Properties

Table 24-2 Apply Properties

Option Description

OMSPatcher.OMS_DISABLE_HOST_CHECK=true

Used to disable host verification check for WebLogic admin server. Please set this property to true if your OMS configuration is based on virtual host.

OMSPatcher.OMS_USER=<installed OMS user>

Use this property if OMSPatcher is not able to get the installed OMS administrator name by itself. This switch is applicable only for Windows.

OMSPatcher.OMS_SCRIPTS_DIR=<existing directory>

This switch is applicable only for UNIX systems. By providing an existing directory, the bash scripts produced by OMSPatcher for multi-OMS are copied to a newly created time stamped sub-directory under the directory specified by the administrator. This would help OMS administrator to execute the scripts from pre-determined shared location, if any, rather than manual scripts copied to each OMS box.

Rollback

Roll back sub-patches of a System patch from OMS instance home. Administrator specifies the sub-patch IDs of the System patch. You can obtain the sub-patch IDs by running the omspatcher lspatches command. See "Running omspatcher lspatches".

Important: OMSPatcher must be run from the Middleware home. ORACLE_HOME environment variable must be set as platform home or provided via the –oh option.

When running omspatcher rollback, you will be prompted the following:

  • WebLogic Admin Server URL of the primary OMS (or standby OMS)

  • Username and Password

Silent interaction is supported by using the silent and property_file options.The standby option should be used if a stand by OMS system is patched. OMSPatcher can pass 'x=y' properties through the command line. See Table 24-2.

Syntax

omspatcher rollback -id <sub patches ID of System patch>
									[-custCertPath <Path to customer optional certificate>]
									[-idFile <file contains list of sub-patch IDs of System patch>]
									[-invPtrLoc <Path to oraInst.loc>
									[-jre <LOC>] [-silent] [-nonrolling]
									[-property_file <path to property file>]
									[-analyze] [-oh <Platform home path>]
									[-standby]

Parameters

Sub patch IDs for the System patch to be rolled back. If you want to rollback a whole System patch, the ids of all sub-patches of that System patch must be specified.

Rollback Options

Table 24-3 Rollback

Option Description

id

Use omspatcher lspatches option to display all patch ids for both core home and plug in homes with relation to the System patch bundles. The patch ids can be only from one bundle in a session. The list is separated by commas.

idFile

File that contains the list of sub-patch IDs of a System patch.

invPtrLoc

Used to locate the oraInst.loc file. Needed when the installation used the invPtrLoc flag. This should be the path to the oraInst.loc file.

jre

This option tells omspatcher to use JRE (java) from the specified location instead of the default location under Oracle Home.

silent

This option suppresses any user-interaction.

nonrolling

Roll back and deploy the patch in non-rolling fashion, provided it is supported by the patch.

property_file

The administrator defined property file for omspatcher to use. The path to the property file should be absolute.

The keys for 'omspatcher' are:

'AdminConfigFile' - Encrypted file for Admin Server user of OMS instance domain.

'AdminKeyFile' - Encrypted file for Admin Server password of OMS instance domain.

'AdminServerURL' - Admin Server URL of OMS instance domain.

(Example: t3s://<host address>:<port number>)

The key value pair is of the format 'x=y' where 'x' is omspatcher understood key and each pair is separated by new line in the property file. This option is typically used for silent operations.

This option is very useful for silent mode of 'omspatcher' invocation. In order to create encrypted files for WebLogic admin server username & password, Please use $ORACLE_HOME/OMSPatcher/wlskeys/createkeys.sh (command for windows) to get the files and load it through a custom file by ‘property_file' option.

NOTE: For Windows, ensure that directories and files in the path are separated by "\\" in the property file.

analyze

Displays out the actions without any configuration/binary change through 'omspatcher'.

custCertPath

This option tells OMSPatcher to use the certificate from the specified location.

oh

The location of EM platform home. This overrides the ORACLE_HOME environment variable.

standby

This option should be used for standby OMS patching operations.

Rollback Command Properties

Table 24-4 Rollback Properties

Option Description

OMSPatcher.OMS_DISABLE_HOST_CHECK=true

Used to disable host verification check for WebLogic admin server. Please set this property to true if your OMS configuration is based on virtual host.

OMSPatcher.OMS_USER=<installed OMS user>

Use this property if OMSPatcher is not able to get the installed OMS administrator name by itself. This switch is applicable only for Windows.

OMSPatcher.OMS_SCRIPTS_DIR=<existing directory>

This switch is applicable only for UNIX systems. By providing an existing directory, the bash scripts produced by OMSPatcher for multi-OMS are copied to a newly created and time stamped sub-directory under the directory specified by the administrator. This would help OMS administrator to execute the scripts from pre-determined shared location, if any, rather than a manual script used to copy to each OMS box.

lspatches

Displays the list of patches applied to the OMS home. It will show the component Name/Version, Component Type, System patch, Sub-patch and patch description where patch has been applied. Please note that OMSPatcher will be used to apply only system patches. However the OMS can have one-off patches which would have already been applied at the time of the Enterprise Manager installation. OMSPatcher provides information about whether the patch is a system patch or one-off patch and, if it is the system patch, then it will also show all other patches that are part of that system patch.

Syntax

omspatcher lspatches   [ -invPtrLoc <Path to oraInst.loc> ]
                       [-jre <LOC> ]
			                       [-oh]
                       

Options

Table 24-5 lspatches

Option Description

jre

This jre option instructs OMSPatcher to use the JRE (java) from the specified location instead of the default location under Oracle Home.

invPtrLoc

The invPtrLoc option is used to locate the Central Inventory Pointer File (oraInst.loc. Input for this option is the path to the oraInst.loc file.

oh

The location of Middleware home. This overrides the ORACLE_HOME environment variable.

version

The version command shows the current version number of the OPatch utility, dependent OPlan version, and the osysmodel version.

Important: OMSPatcher must be run from the Middleware home.

Syntax

omspatcher version [-invPtrLoc <Path to oraInst.loc>]
               [-jre <LOC>]
               [-oh <ORACLE_HOME>]
               [-help] [-h]

Options

The following table describes the options available for the version command.

Table 24-6 version Command Options

Option Description

-invPtrLoc

The invPtrLoc option is used to locate the Central Inventory Pointer File (oraInst.loc). Input for this option is the path to the oraInst.loc file.

-jre

This jre option instructs OMSPatcher to use the JRE (java) from the specified location instead of the default location under Oracle Home.

-oh

The oh option specifies the Oracle Home to work on. This takes precedence over the environment variable ORACLE_HOME.

checkApplicable

The checkApplicable command performs prerequisite binary checks on the OMS platform home and plug-in homes to determine the applicability of a System patch and/or the whether sub-patches of the System patch can be rolled back.

Syntax

omspatcher checkApplicable
										[-id <singleton or System Patch ID to be rolled back>]
										[-custCertPath <Path to customer optional certificate>]
										[-invPtrLoc <Path to oraInst.loc>]
										[-jre <LOC>]
										[-ph <System patch that is to be installed>] [-silent]

Options

The following table describes the options available for the checkApplicable command.

Table 24-7 checkApplicable Command Options

Option Description

id

This option can be used to specify the sub-patch IDs that are to be rolled back from the OMS platform home or plug in homes.

invPtrLoc

Used to locate the oraInst.loc file. Needed when the installation used the -invPtrLoc flag. This should be the path to the oraInst.loc file.

jre

This option tells OMSPatcher to use JRE (java) from the specified location instead of the default location under Oracle Home.

ph

This option can be used to specify the path to the patch location. The input must be a System patch location.

silent

This suppresses any user-interaction.

custCertPath

This option tells OMSPatcher to use the certificate from the specified location.

saveConfigurationSnapshot

The saveConfigurationSnapshot command generates configuration a snapshot for the primary OMS (along with OMS repository) and saves it to an XML file that can be read by OMSPatcher.

If file is not specified, it will be saved to a default file (configData.xml) at the following location

ORACLE_HOME/cfgtoollogs/opatch/sysconfig/configData.xml

When running the saveConfigurationSnapshot command, you will be prompted for the following:

  • WebLogic Admin Server URL of the primary OMS

  • Username and password

You can run the command in silent mode (suppress user interaction) via the silent and property_file options.

This command must be run from an OMS instance belonging to the primary OMS system. If the OMS configuration is running on a virtual host, you must set the OMSPatcher.OMS_DISABLE_HOST_CHECK=true option from the command line.

Syntax

omspatcher saveConfigurationSnapshot 
     [-configFile <File to save configuration snapshot> ] 
     [-oh <ORACLE_HOME> ]
     [-invPtrLoc <Path to oraInst.loc> ]
     [-jre <LOC> ]
     [-silent ]
     [-property_file <path to file> ] 
 

Options

The following table describes the options available for the version command.

Table 24-8 saveConfigurationSnapshot Command Options

Option Description

configFile

Enables OPatch to write the configuration for the specified product to an XML file. The XML file can only be recognized by Oracle System Model APIs and accessed through via the Enterprise Manager SDK.

oh

Specifies the Oracle home to be worked on. The Oracle Home specified takes precedence over the environment variable ORACLE_HOME.

invPtrLoc

Used to locate the oraInst.loc file. Needed when the installation used the -invPtrLoc flag. This should be the path to the oraInst.loc file.

jre

Instructs OMSPatcher to use JRE (java) from the specified location instead of the default location under Oracle Home.

silent

Suppresses any user-interaction.

property_file

The user-defined property file for OMSPatcher to use. The path to the property file must be absolute.

The keys for 'OMSPatcher' are:

  • AdminConfigFile - Encrypted file for Admin Server user of the GC Domain.

  • AdminServerURL' - Admin Server URL of GC Domain (Example: t3s://<host address>:<port number>)

  • AdminKeyFile - Encrypted file for Admin Server password of the GC Domain.

The Key, value pair is of the format 'x=y' where 'x' is an OMSPatcher understood key and each pair is separated by newline in the property file.

The property_file option is typically used when running OMSPatcher in silent mode operation (suppress user interaction)

In order to create encrypted files for a WebLogic Admin Server username & password, run the following script:

$MW_HOME/OMSPatcher/wlskeys/createKeys.sh

(createKeys.cmd for Windows) to obtain the files and load them through a custom file using the property_file option.

NOTE: For Windows, maker sure that directories, files in the path are separated by "\\" in the property file.

Troubleshooting

This chapter describes common OMSPatcher problems that may occur during patching operations or the analyze phase.

This chapter covers the following:

OMSPatcher Troubleshooting Architecture

In order for OMSPatcher to fully automate the patching process, it accesses various tools/utilities to carry out different patching tasks in their respective phases. The primary tools/utilities outside of OMSPatcher are:

  • emctl stop oms - Life cycle

  • emctl start oms - Life cycle

  • emctl applypatch, emctl rollbackpatch – Apply, rollback SQL changes in the OMS repository SYSMAN schema respectively

  • emctl register, emctl deregister – Register, de-register metadata services with the right XMLs for MRS artifacts as per patch metadata instructions respectively

These tools/utilities are accessed during the patching process. Note that failure during invocation of these utilities can also happen and the errors & remedies for those commands are not handled in this document. They need to be followed up with Oracle Support for details. However, OMSPatcher will trap errors from these commands output, push it to appropriate logs and announce it to the administrator and finally to support.

Apart from the above external tools/utilities, OMSPatcher uses the following internal utilities to do binary patching operations. They have separated log files generated by OMSPatcher. The internal utilities are patch binary prerequisite checks and patch binary apply, rollback operations.

OMSPatcher Log Management Architecture

This section refers to the information through logs published by OMSPatcher as part of its patching operations. This knowledge is needed for the administrator to obtain the appropriate logs from right area to troubleshoot and inform Oracle Support for further analysis. The following annotated example shows OMSPatcher apply output that displays the various log files that are created when running OMSPatcher.

Sample OMSPatcher apply Output

$ OMSPatcher/omspatcher apply  /scratch/dev_patches/13_3/1111191
OMSPatcher Automation Tool
Copyright (c) 2018, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.8.0.0.0
OUI version        : 13.8.0.0.0
Running from       : /scratch/oms/mw
Log file location  : /scratch/oms/mw/cfgtoollogs/omspatcher/opatch2018-05-30_03-21-43AM_1.log

OMSPatcher log file: /scratch/oms/mw/cfgtoollogs/omspatcher/1111191/omspatcher_2018-05-30_03-21-48AM_deploy.log

Please enter OMS weblogic admin server URL(t3s://myhost.myco.com:7101):> 
Please enter OMS weblogic admin server username(weblogic):> 
Please enter OMS weblogic admin server password:> 



Configuration Validation: Success


Running apply prerequisite checks for sub-patch(es) "1111126,1111155,1111137" and Oracle Home "/scratch/oms/mw"...

To continue, OMSPatcher will do the following:
[Patch and deploy artifacts]   : Apply sub-patch(es) [ 1111126 1111137 1111155 ]
                                 Register MRS artifact "eventsaux";
                                 Register MRS artifact "VCPUUtilization";
                                 Register MRS artifact "mpcui"


Do you want to proceed? [y|n]
y
User Responded with: Y

Applying sub-patch(es) "1111126,1111137,1111155"
Please monitor log file: /scratch/oms/mw/cfgtoollogs/opatch/opatch2018-05-30_03-23-31AM_3.log


Registering service "eventsaux" with register file "/scratch/oms/mw/sysman/metadata/events/auxiliary/metric_alert_aux.xml"...
Please monitor log file: /scratch/oms/mw/cfgtoollogs/omspatcher/2018-05-30_03-21-43AM_SystemPatch_1111191_1/emctl_register_eventsaux_2018-05-30_03-24-20AM.log


Registering service "VCPUUtilization" with register file "/scratch/oms/mw/plugins/metadata/vcpu/vcpu-exalogic-registration.xml"...
Please monitor log file: /scratch/oms/mw/cfgtoollogs/omspatcher/2018-05-30_03-21-43AM_SystemPatch_1111191_1/emctl_register_VCPUUtilization_2018-05-30_03-24-28AM.log


Registering service "mpcui" with register file "/scratch/oms/mw/plugins/test_mpcui_1.xml"...
Please monitor log file: /scratch/oms/mw/cfgtoollogs/omspatcher/2018-05-30_03-21-43AM_SystemPatch_1111191_1/emctl_register_mpcui_2018-05-30_03-24-34AM.log


Registering service "mpcui" with register file "/scratch/oms/mw/plugins/test_mpcui_2.xml"...
Please monitor log file: /scratch/oms/mw/cfgtoollogs/omspatcher/2018-05-30_03-21-43AM_SystemPatch_1111191_1/emctl_register_mpcui_2018-05-30_03-24-40AM.log


Complete Summary
================


All log file names referenced below can be accessed from the directory "/scratch/oms/mw/cfgtoollogs/omspatcher/2018-05-30_03-21-43AM_SystemPatch_1111191_1"

Patching summary:
-----------------

Binaries of the following sub-patch(es) have been applied successfully:

                        Featureset               Sub-patches                                                    Log file
                        ----------               -----------                                                    --------
  oracle.sysman.top.oms_13.3.0.0.0   1111126,1111137,1111155   1111126,1111137,1111155_opatch2018-05-30_03-23-31AM_3.log

Deployment summary:
-------------------

The following artifact(s) have been successfully deployed:

            Artifacts                                                   Log file
            ---------                                                   --------
        MRS-eventsaux         emctl_register_eventsaux_2018-05-30_03-24-20AM.log
  MRS-VCPUUtilization   emctl_register_VCPUUtilization_2018-05-30_03-24-28AM.log
            MRS-mpcui             emctl_register_mpcui_2018-05-30_03-24-34AM.log
            MRS-mpcui             emctl_register_mpcui_2018-05-30_03-24-40AM.log


Log file location: /scratch/oms/mw/cfgtoollogs/omspatcher/1111191/omspatcher_2018-05-30_03-21-48AM_deploy.log

OMSPatcher succeeded.
bash-3.2$ 

Log output to a consolidated directory

As shown in the example above, there is a reference to pushing all logs to a consolidated log directory. The following line in the trace example shows this consolidation log directory.

...

All log file names referenced below can be accessed from the directory "/scratch/oms/mw/cfgtoollogs/omspatcher/2018-05-30_03-21-43AM_SystemPatch_1111191_1"

...

This consolidated log directory contains the following files (here with reference to the example for rollback).

$ ls -l /scratch/oms/mw/cfgtoollogs/omspatcher/2018-05-30_03-21-43AM_SystemPatch_1111191_1 

-rw-r--r-- 1 skkurapa g900 39975 May 30 03:24 1111126,1111137,1111155_opatch2018-05-30_03-23-31AM_3.log
-rw-r--r-- 1 skkurapa g900 13219 May 30 03:24 1111126,1111155,1111137_opatch2018-05-30_03-22-01AM_2.log
-rw-r--r-- 1 skkurapa g900   120 May 30 03:22 AdminServerStatusPrerequisites_2018-05-30_03-22-01AM.log
-rw-r--r-- 1 skkurapa g900    66 May 30 03:22 RepositoryStatusPrerequisites_2018-05-30_03-22-01AM.log
-rw-r--r-- 1 skkurapa g900    71 May 30 03:22 Swlib_Prerequisite_2018-05-30_03-22-01AM.log
-rw-r--r-- 1 skkurapa g900   456 May 30 03:24 emctl_register_VCPUUtilization_2018-05-30_03-24-28AM.log
-rw-r--r-- 1 skkurapa g900   451 May 30 03:24 emctl_register_eventsaux_2018-05-30_03-24-20AM.log
-rw-r--r-- 1 skkurapa g900   418 May 30 03:24 emctl_register_mpcui_2018-05-30_03-24-34AM.log
-rw-r--r-- 1 skkurapa g900   418 May 30 03:24 emctl_register_mpcui_2018-05-30_03-24-40AM.log
-rw-r--r-- 1 skkurapa g900 12574 May 30 03:24 opatch2018-05-30_03-21-43AM_1.log
-rw-r--r-- 1 skkurapa g900  3938 May 30 03:24 temp_apply_automation.xml
-rw-r--r-- 1 skkurapa g900  3149 May 30 03:24 temp_rollback_automation.xml

All individual log files of all invocation commands are finally copied to a consolidated place as highlighted above. Each command naming convention is self-explanatory and it indicates the actual operations being performed in automation. The omspatcher log file will refer the individual log files so that administrator can easily connect to individual files to refer to any failure.

Logs for Oracle Support

If the administrator wants to contact Oracle Support, the administrator must provide the following references to Support.

  • Administrator interface trace(s).

  • Consolidated log directory as zip

  • OPatch log file

  • OMSPatcher log file

  • Output of omspatcher lspatches command on all OMS instance homes.

OMSPatcher: Cases Analysis, Error Codes, and Remedies/Suggestions

Refer to the following table for common OMSPatcher error codes.

Table 24-9 OMSPatcher Error Codes

Error Code Description Remedy/Suggestion

231

Wrong Oracle WebLogic Administration Server URL and/or invalid credentials

Correct the interview inputs and run OMSPatcher again.

234

Malformed Oracle WebLogic Administration Server URL

If the Oracle WebLogic Administration Server URL is already defaulted (value given), type <enter>. If it is not given, construct the Oracle WebLogic Administration Server URL as t3s://<WebLogic Administration Server host address>:<WebLogic Administration Server port> .of the domain that controls the managed server on which the OMS is deployed.

235

Unable to connect to OMS repository

Check the OMS repository connectivity for SYSMAN administrator and run OMSPatcher again.

236

OUI central inventory read issue

Check if the OUI inventory is locked by some other processes. Check if OUI inventory is readable.

238

Patch binary prerequisite checks failure

Check OMSPatcher, OPatch, patch binary prerequisite log files for more details on the errors. If not resolved, contact Oracle Support.

240 - 251

Binary updates (or) deployment failure

  • This is a case for single OMS system. Patching steps are decided by OMSPatcher but it failed to execute steps. OMSPatcher will print the failed executed step and the remaining steps to be executed for completion of patching operations. Administrator needs to contact Oracle support with logs, resolve why it failed and then must execute manually the failed step and steps referred by OMSPatcher (in OMSPatcher log file) to complete operations.

  • In case of multi OMS (or) stand by OMS patching operations, failure of individual commands that got executed through text/html output must be brought to support notice for further diagnosis. After the failure condition is resolved, administrator needs to execute the failed steps and further steps mentioned in HTML (or) text output to complete the patching operations.

233

Software library not configured

OMS repository connectivity not achieved. (post successful check of the same during credential inputs

Oracle WebLogic Administration Server not reachable (post successful check of the same during credential inputs)

Check the OMSPatcher log file for the failure.

OMSPatcher: External Utilities Error Codes

The following table lists exit codes for external utilities that OMSPatcher uses for life cycle and deployment. If the deployment (or) life cycle fails through OMSPatcher, the administrator can search individual log files for the error messages shown in the Error Message/Recommendation column.

Table 24-10 OMSPatcher External Utilities Error Codes

Exit Code Error Message / Recommendation

34

Displays the usage of the command.

35

Unable to read password! Exiting...

36

Unable to get a connection to the repository! Exiting...

37

The Plug-in is not deployed on this Management Server. The plug-in has to be deployed first to register metadata for that plug-in.

38

Input file does not exist

39

This operation is not supported by service.

40

Metadata operation is skipped.

41

Error occurred during Metadata registration.

42

Error occurred during Metadata de-registration.

Special Error Cases for OMSPatcher OMS Automation

This section provides issue resolution information for special cases when using OMSPatcher. This information will allow the administrator to handle these issues easily with less need for support team intervention.

Windows patching failure due to lock of files by Oracle WebLogic Administration Server

In Windows operating systems, it has been noticed that some of the Enterprise Manager related files (used for patching) are locked by running of Oracle WebLogic Administration Server. As OMSPatcher required Oracle WebLogic Administration Server to be RUNNING for the configuration detection, we need to perform the following steps to make sure that this conflict with respect to environment and patching is removed.

  1. Go to ORACLE_HOME

  2. Run OMSPatcher in non-analyze mode. For further instructions, refer to the patch README and Administrator guide.

    Once the OMSPatcher is run in non-analyze mode, it will check if active files are locked by Oracle WebLogic administration server and will provide a prompt as shown below (in silent mode it will be auto-yes):

    Running prerequisite checks to verify if any files or services are locked by admin server process...
    Please monitor OPatch log file: c:\MW_130518\oms\cfgtoollogs\opatch\1111112_Jun_
    26_2014_08_16_19\ApplyPrereq2014-06-26_08-16-57AM_8.log
     
    The details are:
     
    Following files are active :
    c:\MW_130518\oms\sysman\jlib\emCoreConsole.jar
     
    Due to active files to be patched, OMSPatcher will stop all OMS processes so tha
    t lock on active files may be released...
    Do you want to proceed? [y|n]
    y
    User Responded with: Y
    OMSPatcher has stopped all OMS processes successfully.
    

    If there is a failure while stopping OMS processes, OMSPatcher will accordingly error out. Refer to the OMSPatcher log file for details.

  3. OMSPatcher will stop the stack and then ask for a confirmation from the administrator on whether to proceed with prerequisite checks of patch binaries (in silent mode it will be auto-yes):

    OMSPatcher has stopped all OMS processes successfully. Please make sure the above listed active files are unlocked by all windows processes.
    Do you want to proceed? [y|n] y
     
    User Responded with: Y
     

    Note:

    Administrators are requested to use some open source utilities like process explorer and search for file strings given as output in (2) to check if any files are still active. If so, kill the process tree of those files so that OPatch will run the checks, patch, and deploy the automation elements.

  4. OMSPatcher will not attempt to re-start the stack. The administrator must restart the stack as needed.

    A complete sample trace of this case is shown below:

    C:\MW_130518\oms\OPatch_June26>omspatcher apply ..\patches\cmdRcu\1111112
    OMSPatcher Automation Tool
    Copyright (c) 2016, Oracle Corporation.  All rights reserved.
    
    
    OMSPatcher version : 13.8.0.0.0
    OUI version        : 13.8.0.0.0 
    Running from       : c:\MW_130518\oms
    Log file location: 
    c:\MW_130518\oms\cfgtoollogs\omspatcher\omspatcher2014-06-26_08-16-19AM_1.log
     
    omspatcher log file: c:\MW_130518\oms\cfgtoollogs\omspatcher\1111112\opatch_oms_2014-06-26_08-16-23AM_deploy.log
     
    Please enter the WebLogic Admin Server URL for primary OMS:> t3s://example.o
    racle.com:7101
    Please enter the WebLogic Admin Server username for primary OMS:> weblogic
    Please enter the WebLogic Admin Server password for primary OMS:>
     
    Configuration Validation: Success
     
     
    Running prerequisite checks to verify if any files or services are locked by admin server process...
    Please monitor OPatch log file: c:\MW_130518\oms\cfgtoollogs\omspatcher\1111112_Jun_26_2014_08_16_19\ApplyPrereq2014-06-26_08-16-57AM_8.log
     
    The details are:
     
    Following files are active:
    c:\MW_130518\oms\sysman\jlib\emCoreConsole.jar
     
    Due to active files to be patched, omspatcher will stop all OMS processes so that lock on active files may be released...
    Do you want to proceed? [y|n]
    y
    User Responded with: Y
    omspatcher has stopped all OMS processes successfully.
     
    omspatcher has stopped all OMS processes successfully. Please make sure the above listed active files are unlocked by all windows processes.
    Do you want to proceed? [y|n]
    y
    User Responded with: Y
     
    Running apply prerequisite checks for patch(es) "1111112" and Oracle Home "c:\MW
    _130518\oms"...
    Please monitor omspatcher log file: c:\MW_130518\oms\cfgtoollogs\omspatcher\1111112_Jun_26_2014_09_01_33\ApplyPrereq2014-06-26_09-03-41AM_10.log
    Patches "1111112" are successfully analyzed for Oracle Home "c:\MW_130518\oms"
     
    To continue, OMSPatcher will do the following:
    [Patch and deploy patch(es) binaries]   : Apply patch(es) [ 1111112 ] to Oracle
    Home "c:\MW_130518\oms";
    Apply RCU artifact with patch "c:\MW_130518\oms\.omspatcher_storage \1111112_Feb_21_2014_06_30_38\original_patch"
     
     
    Do you want to proceed? [y|n]
    y
    User Responded with: Y
     
    Applying patch "1111112" to Oracle Home "c:\MW_130518\oms"...
    Please monitor OMSPatcher log file: c:\MW_130518\oms\cfgtoollogs\omspatcher\1111112_Jun_26_2014_09_01_33\apply2014-06-26_09-04-17AM_12.log
     
    Updating repository with RCU reference file "c:\MW_130518\oms\.omspatcher_storage\1111112_Feb_21_2014_06_30_38\original_patch"
     
    Copying all logs to: c:\MW_130518\oms\cfgtoollogs\omspatcher\2014-06-26_09-01-32AM_SystemPatch_1111112_1
     
    Patching summary:
    Following patch(es) are successfully applied (Oracle home:patch list):
    c:\MW_130518\oms:1111112
     
     
    Log file location: c:\MW_130518\oms\cfgtoollogs\omspatcher\1111112\omspatcher_oms_2013-06-26_09-01-36AM_deploy.log
     
    OMSPatcher succeeded.

Multi-OMS Execution for UNIX based Systems

This section deals with possible issues you may encounter when running bash scripts generated by OMSPatcher in multi-OMS (UNIX-based systems) environment. The following OMSPatcher-generated output illustrates various script-based issues.

Example 24-2 OMSPatcher Output: Multi-OMS, UNIX-based Environment

omspatcher apply /net/myhost/scratch/patch_2nd_nov/em13_1/bundle_patches/1111191
OMSPatcher Automation Tool
Copyright (c) 2018, Oracle Corporation.  All rights reserved.
 
 
OMSPatcher version : 13.8.0.0.0
OUI version        : 13.8.0.0.0
Running from       : /scratch/myadmin/oms_install/mw
Log file location  : /scratch/myadmin/oms_install/mw/cfgtoollogs/omspatcher/opatch2015-11-11_23-27-35PM_1.log
 
OMSPatcher log file: /scratch/myadmin/oms_install/mw/cfgtoollogs/omspatcher/1111191/omspatcher_2015-11-11_23-27-46PM_deploy.log
 
Please enter OMS weblogic admin server URL(t3s://rwsv1452.myco.com:7101):> 
Please enter OMS weblogic admin server username():> weblogic
Please enter OMS weblogic admin server password:> 
 
 
WARNING: Could not apply the patch "1111155" because the "oracle.samples.xohs.oms.plugin with version 13.1.4.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
 
Configuration Validation: Success
 
WARNING:You have a multi-OMS setup. The patch application is not complete until the following steps are executed successfully.
 
 
Please perform the following steps to complete patching operations.
-------------------------------------------------------------------     
        1. Please copy the script "/scratch/myadmin/oms_install/mw/.omspatcher_storage/oms_session/scripts_2015-11-11_23-28-33/run_script#1_on_host_rwsv1452_us_oracle_com_as_user_myadmin.sh" to "rwsv1452.myco.com" and execute the script on host "rwsv1452.myco.com".     
        2. Please execute the script "/scratch/myadmin/oms_install/mw/.omspatcher_storage/oms_session/scripts_2015-11-11_23-28-33/run_script#2_on_host_rwsv1451_us_oracle_com_as_user_myadmin.sh" on local host.
 
 
Complete Summary
================
 
 
All log file names referenced below can be accessed from the directory "/scratch/hkumars/oms_install/mw/cfgtoollogs/omspatcher/2015-11-11_23-27-35PM_SystemPatch_1111191_1"
 
Patching summary:
-----------------
 
The following sub-patches are incompatible with components installed in the OMS system:
1111155
 
--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1)  Could not apply the patch "1111155" because the "oracle.samples.xohs.oms.plugin with version 13.1.4.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
2) You have a multi-OMS setup. The patch application is not complete until the following steps are executed successfully.
 
--------------------------------------------------------------------------------
OMSPatcher Session completed with warnings.
Log file location: /scratch/myadmin/oms_install/mw/cfgtoollogs/omspatcher/1111191/omspatcher_2015-11-11_23-27-46PM_deploy.log
 
OMSPatcher completed with warnings.

Note:

WARNING: You have a multi-OMS setup. The patch application is not complete until the following steps are executed successfully

This message means that patching and deployment are not complete until the administrator performs the bash script execution instructions generated by OMSPatcher.

The administrator needs to provide the credential while running the OMSPatcher generated bash script on secondary OMS.

Example:

/net/host00/scratch/myadmin/work/omshome1967/.omspatcher_storage/oms_session/scripts_2015-11-19_06-40-51/run_script#1_on_host_myhost_com_as_user_myadmin.sh
Verifying embedded script host-address "myhost.mycompany.com" against the network interface for a match...
Trying for a match with: fe80:0:0:0:221:f6ff:fe6f:9ac1%2(fe80:0:0:0:221:f6ff:fe6f:9ac1%2)
Trying for a match with: myhost.mycompany.com(10.248.10.100)
Script-host address matched with host network interface.
Creating session file /scratch/myadmin/work/omshome1967/.omspatcher_storage/oms_session/oms_session_2015-11-18_22-40-19PM...
Copying your script to OMSPatcher defined path /scratch/myadmin/work/omshome1967/.omspatcher_storage/oms_session/scripts_2015-11-19_06-40-51/run_script#1_on_host_myhost_com_as_user_myadmin.sh...
The System Patch directory already exists in the machine (this could mean that System Patch is already downloaded). Do you want to overwrite it (y|n)?  
n
User provided n for patch transfer. Ignoring patch transfer...
The Patch backup location /scratch/myadmin/work/omshome1967/.omspatcher_storage/1111126_Sep_7_2015_02_06_54/original_patch does not exists in the machine (this could mean that the patch automation data are not present on this host and which is mandatory to rollback this patch later from this host). You need to provide host credential to copy it. 
Executing command: mkdir -p /scratch/myadmin/work/omshome1967/.omspatcher_storage/1111126_Sep_7_2015_02_06_54;scp -r host00.myco.com:/scratch/myadmin/work/omshome1967/.omspatcher_storage/1111126_Sep_7_2015_02_06_54/original_patch /scratch/myadmin/work/omshome1967/.omspatcher_storage/1111126_Sep_7_2015_02_06_54
myadmin@host00.myco.com's password:

Here, the administrator has to provide the user credential to copy the automation data on the secondary OMS which will be used while rolling back the patches.

Troubleshooting Bash Script Execution

The following section covers the most common issues you may encounter while executing OMSPatcher-generated bash scripts in multi-OMS (UNIX-based) environments.

No Windows Support

Microsoft Windows does not support bash script execution. So, this optimization (steps reduction) is not applicable for Windows OMS environments. The older context sensitive individual steps output through OMSPatcher remains in Windows.

Bash script program availability

The scripts assume that bash is located at /bin/bash. However, if this is not true, make sure the first line of the scripts are updated with the output of whereis bash.

In-between command failure in bash script

If there is a failure in between execution of commands in the bash script, the script stops running. The OMS administrator must triage the failure and comment out (inserting a hash "#" character at the beginning of a line) the already executed portions of the script and restart the bash script execution. Make sure you do not to comment out prompts and prompt-related code in the script.

Complete execution needed for all bash scripts

ALL bash script steps must be executed. No script and no step within any script can be omitted, even in the event of failures. Patching is correct and complete if and only if all steps of all bash scripts are executed correctly as per the order specification.

Patch location (if mounted)

The patch location input may exist on a mounted location. The bash scripts try to perform a secure copy (SCP) from the local OMS (where the OMSPatcher Perl script was invoked). The SCP attempt could fail if the location is mounted. The bash script will ignore the SCP failure.

OMS repository SYSMAN password and prompts

If the script has automation steps to be executed, it will prompt for sysman password, pay attention to it and provide the requested information.

The bash script prompt will appear as follows:

Please provide credential for OMS repository SYSMAN user:

User Credentials to copy automation metadata files to remote node

While running scripts for remote nodes, user will be prompted to entire user credentials to copy automation metadata files from primary node to this node, please pay attention to this and provide the requested information.

The bash script prompt will appear as:

myadmin@host00.mycompany.com's password:

Patch Transfer/Download

The script will provide an option to download patch from local OMS to remote nodes (for the scripts that involve remote nodes). If the patch is on shared location (or) already downloaded to a specified location mentioned by the script, a user can choose to input n when prompted, and ignore this transfer.