OMSPatcher supports resume upon failure capability for both single-OMS and multi-OMS configurations.
This section covers the following topics:
On a single OMS System, OMSPatcher executes end-to-end automation of patching steps. once a failure has occurred, OMSPatcher can generate a bash script containing list of all incomplete (or) failed steps. The OMS administrator must refer to the master log file created by OMSPatcher to ascertain and resolve the root cause of the failure, and then run the bash script given by OMSPatcher. The bash script runs the steps from the point of failure.
Example
OMSPatcher, while applying an auto system patch. fails due to file permission issue.
Example:
omspatcher apply /scratch/patch_2nd_nov/em13_3/bundle_patches/1111191
OMSPatcher Automation Tool
Copyright (c) 2018, Oracle Corporation. All rights reserved.
OMSPatcher version : 13.8.0.0.0
OUI version : 13.8.0.0.0
Running from : /scratch/admin1/mw
Log file location : /scratch/admin1/mw/cfgtoollogs/omspatcher/opatch2018-12-01_01-06-42AM_1.log
OMSPatcher log file: /scratch/admin1/mw/cfgtoollogs/omspatcher/1111191/omspatcher_2018-12-01_01-06-50AM_deploy.log
Please enter OMS weblogic admin server URL(t3s://slc09jcy.us.oracle.com:7101):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>
WARNING: Could not apply the patch "1111155" because the "oracle.samples.xohs.oms.plugin with
version 13.1.4.0.0" core component of the OMS or the plug-in for which the patch is intended
is either not deployed or deployed with another version in your Enterprise Manager system.
Configuration Validation: Success
Running apply prerequisite checks for sub-patch(es) "1111126 1111137" and
Oracle Home "/scratch/admin1/mw"...
Sub-patch(es) "1111126 1111137" are successfully analyzed for Oracle Home "/scratch/admin1/mw"
To continue, OMSPatcher will do the following:
[Patch and deploy artifacts] : Apply sub-patch(es) [ 1111126 ]
Apply sub-patch(es) [ 1111137 ]
Register MRS artifact "eventsaux";
Register MRS artifact "VCPUUtilization"
Do you want to proceed? [y|n]
y
User Responded with: Y
Applying sub-patch "1111126 "
Applying sub-patch "1111137 "
OMSPatcher failed to apply following patch(es) "1111137" to core/plugin Oracle home(s).
Complete Summary
================
All log file names referenced below can be accessed from the directory
"/scratch/admin1/mw/cfgtoollogs/omspatcher/2018-12-01_01-06-42AM_SystemPatch_1111191_1"
Patching summary:
-----------------
Binaries of the following sub-patch(es) have been applied successfully:
Featureset Sub-patches Log file
---------- ----------- --------
oracle.sysman.top.oms_13.3.0.0.0 1111126 1111126_opatch2018-12-01_01-07-32AM_3.log
Binaries of the following sub-patch(es) failed to get applied:
Featureset Sub-patches Log file
---------- ----------- --------
oracle.sysman.emas.oms.plugin_13.3.1.0.0 1111137 1111137_opatch2018-12-01_01-08-06AM_4.log
The following sub-patches are incompatible with components installed in the OMS system:
1111155
OMSPatcher failed to execute some of the patching steps. Please check the Patching summary,individual logs and
try to resolve the issue. Once the issue is resolved,Please execute below script to complete patching session:
"/scratch/admin1/mw/.omspatcher_storage/oms_session/scripts_2018-12-01_01-06-42AM/run_script_singleoms_resume.sh"
------------------------------------------------------------------------------------------------
OMSPatcher wont allow any other patching operations unless the script is executed successfully
------------------------------------------------------------------------------------------------
[ Error during Patch and deploy artifacts Phase]. Detail: OMSPatcher failed to apply
some of the patches to the OMS instance home(s).
OMSPatcher failed: OMSPatcher failed to execute some of the OMS operations.
Please refer log file(s) for details.
--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1) Could not apply the patch "1111155" because the "oracle.samples.xohs.oms.plugin with
version 13.1.4.0.0" core component of the OMS or the plug-in for which the patch is intended
is either not deployed or deployed with another version in your Enterprise Manager system.
--------------------------------------------------------------------------------
Log file location: /scratch/admin1/mw/cfgtoollogs/omspatcher/1111191/omspatcher_2018-12-01_01-06-50AM_deploy.log
Recommended actions: Please refer log file(s) for more details on the errors. Please contact Oracle Support.
OMS Administrator cannot start a new patching session when there are remnants of an incomplete patching session. OMSPatcher clearly errors out with the detailed information regarding the failure and what action need to be taken to fix this issue.
Example:
omspatcher apply /scratch/patch_2nd_nov/em13_1/bundle_patches/1111191 OMSPatcher Automation Tool Copyright (c) 2018, Oracle Corporation. All rights reserved. OMSPatcher version : 13.8.0.0.0 OUI version : 13.8.0.0.0 Running from : /scratch/admin1/mw Log file location : /scratch/mw/cfgtoollogs/omspatcher/opatch2018-12-01_01-15-09AM_1.log OMSPatcher failed: OMSPatcher finds that previous patching session is not yet completed. Please refer log file "/scratch/mw/cfgtoollogs/omspatcher/1111191/omspatcher_2018-12-01_01-06-50AM_deploy.log" for the previous session and execute the script "/scratch/mw/.omspatcher_storage/oms_session/scripts_2018-12-01_01-06-42AM/run_script_singleoms_resume.sh" to complete the previous session. OMSPatcher can proceed to execute new operations only if previous session is completed successfully. Log file location: /scratch/mw/cfgtoollogs/omspatcher/opatch2018-12-01_01-15-09AM_1.log OMSPatcher failed with error code 73
Now OMS Administrator can run the single-OMS Resume script to finish the failed patching session.
Example:
/scratch/mw/.omspatcher_storage/oms_session/scripts_2018-12-01_01-06-42AM/run_script_singleoms_resume.sh Verifying embedded script host-address "myserver.myco.com" against the network interface for a match... Trying for a match with: fe80:0:0:0:221:f6ff:feb6:424%2(fe80:0:0:0:221:f6ff:feb6:424%2) Trying for a match with: myserver.myco.com(10.252.41.52) Script-host address matched with host network interface. Please provide credential for OMS repository SYSMAN user: Command to execute (Step 1): echo /scratch/patch_2nd_nov/em13_1/bundle_patches/1111191/1111137 >> /scratch/mw/.phBaseFile2018-12-01_01-06-42AM.txt Command to execute (Step 1): /scratch/mw/OPatch/opatch napply -phBaseFile /scratch/mw/.phBaseFile2018-12-01_01-06-42AM.txt -invPtrLoc /scratch/mw/oraInst.loc -oh /scratch/mw -silent Command to execute (Step 1): rm /scratch/mw/.phBaseFile2018-12-01_01-06-42AM.txt Command to execute (Step 1): mkdir -p /scratch/mw/.omspatcher_storage/1111137_Aug_31_2018_01_01_58; cp -Rf /scratch/mw/.patch_storage/1111137_Aug_31_2018_01_01_58/original_patch /scratch/mw/.omspatcher_storage/1111137_Aug_31_2018_01_01_58 Oracle Interim Patch Installer version 13.8.0.0.0 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /scratch/admin1/mw Central Inventory : /scratch/admin1/oraInventory from : /scratch/admin1/mw/oraInst.loc OPatch version : 13.8.0.0.0 OUI version : 13.8.0.0.0 Log file location : /scratch/mw/cfgtoollogs/opatch/opatch2018-12-01_01-16-33AM_1.log OPatch detects the Middleware Home as "/scratch/mw" Verifying environment and performing prerequisite checks... OPatch continues with these patches: 1111137 Do you want to proceed? [y|n] y Y (auto-answered by -silent) User Responded with: Y All checks passed. Backing up files... Applying interim patch '1111137' to OH '/scratch/mw' Patching component oracle.sysman.emas.oms.plugin, 13.1.1.0.0... Patch 1111137 successfully applied. Log file location: /scratch/mw/cfgtoollogs/opatch/opatch2018-12-01_01-16-33AM_1.log OPatch succeeded. Command to execute (Step 2): /scratch/mw/bin/emctl register oms metadata -service eventsaux -file /scratch/mw/sysman/metadata/events/auxiliary/metric_alert_aux.xml -core -sysman_pwd %EM_REPOS_PASSWORD% Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Metadata registration successful Command to execute (Step 3): /scratch/mw/bin/emctl register oms metadata -service VCPUUtilization -file /scratch/mw/plugins/oracle.sysman.emas.oms.plugin_13.1.1.0.0/metadata/vcpu/vcpu-exalogic-registration.xml -pluginId oracle.sysman.emas -sysman_pwd %EM_REPOS_PASSWORD% Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Metadata registration successful Command to execute (Step 4): /scratch/mw/OMSPatcher/omspatcher commit -id 1111126 -oh /scratch/mw -invPtrLoc /scratch/mw/oraInst.loc OMSPatcher Automation Tool Copyright (c) 2018, Oracle Corporation. All rights reserved. OMSPatcher version : 13.8.0.0.0 OUI version : 13.8.0.0.0 Running from : /scratch/admin1/mw Log file location : /scratch/mw/cfgtoollogs/omspatcher/opatch2018-12-01_01-17-14AM_1.log OMSPatcher will now mark the patch "1111126,1111137" as auto-executed. Log file location: /scratch/mw/cfgtoollogs/omspatcher/opatch2018-12-01_01-17-14AM_1.log OMSPatcher succeeded.
OMSPatcher prompts for the SYSMAN password at the start of the script. OMSPatcher cannot execute patching steps on a multi-OMS configuration; it generates a bash script containing the entire patching steps specific to each host for all the nodes. The name of the script contains the hostname and username. The OMS administrator can run a specific script for each host on all nodes to complete patching session.
OMSPatcher apply executes successfully as it generates only patching instructions without executing bash scripts.
Example:
omspatcher apply /scratch/opatchdev/targetPatchingImplRegistration/1111118
OMSPatcher Automation Tool
Copyright (c) 2018, Oracle Corporation. All rights reserved.
OMSPatcher version : 13.8.0.0.0
OUI version : 13.8.0.0.0
Running from : /scratch/aime1/work/midnew9693
Log file location : /scratch/aime1/work/midnew9693/cfgtoollogs/opatch/opatch2018-05-05_22-43-08PM_1.log
OMSPatcher log file: /scratch/aime1/work/midnew9693/cfgtoollogs/omspatcher/1111118/opatch_oms_2018-05-05_22-43-14PM_deploy.log
Please enter OMS weblogic admin server URL(t3s://linux01amd.myco.com:7101):>
Please enter OMS weblogic admin server username:> weblogic
Please enter OMS weblogic admin server password:>
Configuration Validation: Success
WARNING: OMSPatcher cannot run patching steps in multi-OMS environment.
Please perform the following steps to complete patching operations.
-------------------------------------------------------------------
1. Please copy the script "/scratch/aime1/work/midnew9693/.omspatcher_storage/oms_session/scripts_2018-05-05_22-43-51/run_script#1_on_host_linux07jdx_us_oracle_com_as_user_aime1.sh" to "linux07jdx.myco.com" and execute the script.
2. Please execute the script "/scratch/aime1/work/midnew9693/.omspatcher_storage/oms_session/scripts_2018-05-05_22-43-51/run_script#2_on_host_linux01amd_us_oracle_com_as_user_aime1.sh" on local host.
--------------------------------------------------------------------------------
The following warnings have occurred during OMSPatcher execution:
1) OMSPatcher cannot run patching steps in multi-OMS environment.
--------------------------------------------------------------------------------
OMSPatcher Session completed with warnings.
Log file location: /scratch/aime1/work/midnew9693/cfgtoollogs/omspatcher/1111118/opatch_oms_2018-05-05_22-43-14PM_deploy.log
OMSPatcher completed with warnings.
Run the bash script corresponding to the local host (primary host on a Multi-OMS configuration). Script execution has failed because of issue in connecting to database repository because of incorrect sysman password.
Example:
$ /scratch/aime1/work/midnew9693/.omspatcher _storage/oms_session/scripts_2018-05-05_22-43-51/run_script#2_on_host_linux01amd_us_oracle_com_as_user_aime1.sh
Creating master log file /scratch/aime1/work/midnew9693/.omspathcer_storage/oms_session/oms_session_log_2018-05-05_22-43-08PM...
Creating session file /scratch/aime1/work/midnew9693/.omspatcher_storage/oms_session/oms_session_2018-05-05_22-43-08PM...
Please provide credential for OMS repository SYSMAN user:
Command to execute (Step 2): /scratch/aime1/work/midnew9693Patcher/omspatcher checkApplicable -ph /scratch/opatchdev/targetPatchingImplRegistration/1111118 -oh /scratch/aime1/work/midnew9693 -invPtrLoc /scratch/aime1/work/midnew9693/oraInst.loc
OMSPatcher Automation Tool
Copyright (c) 2018, Oracle Corporation. All rights reserved.
OMSPatcher version : 13.8.0.0.0
OUI version : 13.8.0.0.0
Running from : /scratch/aime1/work/midnew9693
Log file location : /scratch/aime1/work/midnew9693/cfgtoollogs/opatch/opatch2018-05-05_22-45-52PM_1.log
OMSPatcher log file: /scratch/aime1/work/midnew9693/cfgtoollogs/omspatcher/1111118/opatch_oms_2018-05-05_22-45-53PM_analyze.log
Running apply prerequisite checks for sub-patch(es) "1111118" and Oracle Home "/scratch/aime1/work/midnew9693"...
Please monitor OPatch log file: /scratch/aime1/work/midnew9693/cfgtoollogs/opatch/1111118_May_05_2018_22_45_52/ApplyPrereq2018-05-05_22-45-57PM_2.log
Sub-patch(es) "1111118" are successfully analyzed for Oracle Home "/scratch/aime1/work/midnew9693"
Complete Summary
================
All log file names referenced below can be accessed from the directory "/scratch/aime1/work/midnew9693/cfgtoollogs/opatch/2018-05-05_22-45-52PM_SystemPatch_1111118_1"
Prerequisites analysis summary:
-------------------------------
The following sub-patch(es) are applicable:
Oracle Home Name Sub-patches Log file
---------------- ----------- --------
oms13c3 1111118 1111118_ApplyPrereq2018-05-05_22-45-57PM_2.log
Log file location: /scratch/aime1/work/midnew9693/cfgtoollogs/omspatcher/1111118/opatch_oms_2018-05-05_22-45-53PM_analyze.log
OMSPatcher succeeded.
Command to execute (Step 4): echo /scratch/opatchdev/targetPatchingImplRegistration/1111118/1111118 >> /scratch/aime1/work/midnew9693/.phBaseFile2018-05-05_22-43-08PM.txt
Command to execute (Step 4): /scratch/aime1/work/midnew9693/OPatch/opatch napply -phBaseFile /scratch/aime1/work/midnew9693/.phBaseFile2018-05-05_22-43-08PM.txt -invPtrLoc /scratch/aime1/work/midnew9693/oraInst.loc -oh /scratch/aime1/work/midnew9693 -silent
Command to execute (Step 4): rm /scratch/aime1/work/midnew9693/.phBaseFile2018-05-05_22-43-08PM.txt
Oracle Interim Patch Installer version 13.6.0.0.0
Copyright (c) 2018, Oracle Corporation. All rights reserved.
Oracle Home : /scratch/aime1/work/midnew9693
Central Inventory : /ade/aime1_opatchauto_fix_lat/oracle/work/DB112/oraInventory
from : /scratch/aime1/work/midnew9693/oraInst.loc
OPatch version : 13.8.0.0.0
OUI version : 13.8.0.0.0
Log file location : /scratch/aime1/work/midnew9693/cfgtoollogs/opatch/opatch2018-05-05_22-46-00PM_1.log
OPatch detects the Middleware Home as "/scratch/aime1/work/midnew9693"
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 1111118
Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.
Backing up files...
Applying interim patch '1111118' to OH '/scratch/aime1/work/midnew9693'
Patching component oracle.sysman.oms.core, 13.3.0.0.0...
Verifying the update...
Patch 1111118 successfully applied.
Log file location: /scratch/aime1/work/midnew9693/cfgtoollogs/opatch/opatch2018-05-05_22-46-00PM_1.log
OPatch succeeded.
Command to execute (Step 6): /scratch/aime1/work/midnew9693/bin/emctl register oms metadata -service TargetPatchingImplRegistration -debug -file /scratch/aime1/work/midnew9693/sysman/metadata/targetpatchingregister/RegisterAgentTarget.xml -core -sysman_pwd %EM_REPOS_PASSWORD%
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Starting output for debug mode.
Debug logs will be written to /scratch/aime1/work/insthome9693/em/EMGC_OMS1/sysman/log/emctl.log
EM-04036: Unable to get a connection to the repository! Exiting...
The command failed with error code 36
Script execution has failed. Please refer to log file: /scratch/aime1/work/midnew9693/.omspatcher_storage/oms_session/oms_session_log_2018-05-05_22-43-08PM for more details
Please fix the failures and re-run the same script to complete the patching session.
OMS Administrator can re-run the script by fixing the issue (provide correct sysman password to connect to database repository). Script resumes execution from the failure point and executes successfully.
Example:
$ /scratch/aime1/work/midnew9693/.omspatcher _storage/oms_session/scripts_2018-05-05_22-43-51/run_script#2_on_host_linux01amd_us_oracle_com_as_user_aime1.sh Please provide credential for OMS repository SYSMAN user: Command to execute (Step 2): /scratch/aime1/work/midnew9693/OMSPatcher/omspatcher checkApplicable -ph /scratch/opatchdev/targetPatchingImplRegistration/1111118 -oh /scratch/aime1/work/midnew9693 -invPtrLoc /scratch/aime1/work/midnew9693/oraInst.loc SKIP command for step 2... Command to execute (Step 4): echo /scratch/opatchdev/targetPatchingImplRegistration/1111118/1111118 >> /scratch/aime1/work/midnew9693/.phBaseFile2018-05-05_22-43-08PM.txt Command to execute (Step 4): /scratch/aime1/work/midnew9693/OPatch/opatch napply -phBaseFile /scratch/aime1/work/midnew9693/.phBaseFile2018-05-05_22-43-08PM.txt -invPtrLoc /scratch/aime1/work/midnew9693/oraInst.loc -oh /scratch/aime1/work/midnew9693 -silent Command to execute (Step 4): rm /scratch/aime1/work/midnew9693/.phBaseFile2018-05-05_22-43-08PM.txt SKIP command for step 4... Command to execute (Step 6): /scratch/aime1/work/midnew9693/bin/emctl register oms metadata -service TargetPatchingImplRegistration -debug -file /scratch/aime1/work/midnew9693/sysman/metadata/targetpatchingregister/RegisterAgentTarget.xml -core -sysman_pwd %EM_REPOS_PASSWORD% Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Starting output for debug mode. Debug logs will be written to /scratch/aime1/work/insthome9693/em/EMGC_OMS1/sysman/log/emctl.log Metadata registration successful Command to execute (Step 7): /scratch/aime1/work/midnew9693/OMSPatcher/omspatcher commit -id 1111118 -oh /scratch/aime1/work/midnew9693 -invPtrLoc /scratch/aime1/work/midnew9693/oraInst.loc OMSPatcher Automation Tool Copyright (c) 2018, Oracle Corporation. All rights reserved. OMSPatcher version : 13.8.0.0.0 OUI version : 13.8.0.0.0 Running from : /scratch/aime1/work/midnew9693 Log file location : /scratch/aime1/work/midnew9693/cfgtoollogs/opatch/opatch2018-05-05_22-49-34PM_1.log OMSPatcher will now mark the patch "1111118" as auto-executed. Log file location: /scratch/aime1/work/midnew9693/cfgtoollogs/opatch/opatch2018-05-05_22-49-34PM_1.log OMSPatcher succeeded. All operations for this script are appended to log file: /scratch/aime1/work/midnew9693/.omspatcher_storage/oms_session/oms_session_log_2018-05-05_22-43-08PM