Oracle® Fusion Applications Patching Guide 11g Release 7 (11.1.7) Part Number E16602-23 |
|
|
PDF · Mobi · ePub |
This chapter describes how to monitor and troubleshoot Oracle Fusion Applications patching and AD Administration processing sessions.
This chapter contains the following topics:
Oracle Fusion Applications Patch Manager (Patch Manager) creates log files for the actions it performs. These logging capabilities track the progress of actions and assist you in diagnosing issues. When you use Patch Manager, you can specify the level of logging detail. The following log levels are available:
ERROR:1 (SEVERE) For an error that results in a failure.
WARNING:1 (WARNING) For an error that does not result in failure but that you should review.
NOTIFICATION:1 (INFO) For high-level information about the progress of the process, no action necessary.
NOTIFICATION:16 (CONFIG) For more detailed information about the progress of the process, no action necessary
TRACE:1 (FINE) For generating the first level of trace messages, used for diagnosing issues.
TRACE:16 (FINER) For generating the second level of trace messages, used for diagnosing issues.
TRACE:32 (FINEST) For generating the highest level of trace messages, used for diagnosing issues.
When you select a more detailed level of logging, the log files also include the lower level of details. For example, if you choose to see INFO messages in your log file, WARNING and SEVERE messages also appear in the log files. For more information, see "Standard Logging Levels" in the Oracle Fusion Applications Administrator's Guide.
You can examine the activities performed during patching sessions by reviewing the associated log files. Patch Manager consolidates log files for each patching session under the directory, APPLICATIONS_CONFIG
/lcm/logs/11.1.7.0.0/FAPMGR
. This directory contains the top-level log file, logsummary
_fapmgr_command_timestamp.html
, along with related log files for each task performed during a fapmgr
session. During a session, you can view the Log Summary HTML file from a browser, which provides links to individual log files. You can periodically refresh the Log Summary HTML file to view the progress of the current patching session. If a task fails, you can access the links to the associated log files to assist in diagnosing the failure. For more information, see Section 11.2.1, "Log Summary".
When a patching session completes, its log files are archived in FA_ORACLE_HOME
/admin/FUSION/logarchive/
Patch Number/fapmgr_command/session ID/timestamp
. The session ID is unique and the time stamp is the start time for the session. Note that whenever Patch Manager runs a command where there is no patch number, such as bootstrap, abort,
and report
, the archive logs are named FA_ORACLE_HOME
/admin/FUSION/logarchive/fapmgr_command
/timestamp.
Log files for OPatch actions are initially written to the FA_ORACLE_HOME
/cfgtools/opatch/
patch number_timestamp directory.
Table 11-1 contains a list of log files created by Patch Manager during patching activities.
Table 11-1 Log Files for Single Oracle Fusion Applications Patch Manager Patching Activities
Log file or directory name under APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/FAPMGR | Log file generated from... |
---|---|
FAPatchManager_apply_timestamp.log |
Oracle Fusion Applications Patch Manager apply session |
FAPatchManager_abort_timestamp.log |
Patch Manager abort session |
FAPatchManager_bootstrap_timestamp.log |
Patch Manager bootstrap session |
FAPatchManager_report_reportname_timestamp.log |
Patch Manager report session |
FAPatchManager_validate_timestamp.log |
Patch Manager validate session |
adpatch_apply_timestamp.log |
Oracle Fusion Applications AutoPatch (AutoPatch) apply session |
adpatch_abort_timestamp.log |
AutoPatch abort session |
adpatch_apply_timestamp_workernumber.log |
AutoPatch worker log file |
adpatch_validate_timestamp.log |
AutoPatch |
adpatch_apply_timestamp_timingreport.lst |
AutoPatch timing report |
logsummary_fapmgr_command_timestamp.html For reports: logsummary_report_reportname_timestamp.html |
The consolidation of the log files generated by Patch Manager in HTML format for viewing and accessing links to other log files from a browser |
patch number_fapmgr_command_session id_timestamp.marker |
Marker file used while moving log files to a backup directory |
ConfigContext_timestamp.log |
Patch Manager in online mode |
ExecutionContext_timestamp.log |
Patch Manager in online mode |
FAPMgrDiagnosticsSummaryfapmgr_command_timestamp.html |
The consolidation of the Patch Manager session in HTML format, known as the Diagnostics report |
FAPMgrDiagnosticsSummary.html |
The Diagnostic report when it is run on demand using this command syntax: |
diaghtmllogs_mode_timestamp |
The Diagnostics report for Patch Manager in |
diaghtmllogs_mode_timestamp/FAPatchManager_apply_20120627022503.html |
Patch Manager apply session |
diaghtmllogs_mode_timestamp/adpatch_apply_timestamp.html |
AutoPatch apply session |
diaghtmllogs_mode_timestamp/adpatch_validate_timestamp.html |
AutoPatch validate session |
diaghtmllogs_mode_timestamp/adpatch_apply_timestamp_workernumber.html |
AutoPatch worker log file |
The multi-apply feature of Patch Manager applies multiple patches after splitting them into groups within a My Oracle Support patch plan. Each group contains at least one patch and it is run as an individual session internally. Log files and diagnostic reports are generated for each individual session, in addition to a master log file and diagnostics report for the overall multi-apply session. For more information, see Section 3.1.5, "Applying Multiple Patches Using a Patch Plan".
You can examine the activities performed during multi-apply patching sessions by reviewing the associated log files. Patch Manager consolidates log files for each patching session under the directory, APPLICATIONS_CONFIG
/lcm/logs/11.1.7.0.0/FAPMGR
. This directory contains the top-level log file, logsummary
_fapmgr_command_timestamp
.html
, along with related log files for each task performed during a multi-apply session. During a session, you can view this log summary HTML file from a browser, which provides links to individual log files. You can periodically refresh the log summary HTML file to view the progress of the current patching session. If a task fails, you can access the links to the associated log files to assist in diagnosing the failure. For more information, see Section 11.2.1, "Log Summary".
Multi-apply sessions create a master diagnostics report that provides information about the multi-apply patching session. It also includes links to corresponding diagnostics reports and archived log files for individual patches that were applied as a group.
Table 11-2 contains a list of log files created by Patch Manager during multi-apply patching activities.
Table 11-2 Log Files For Multi-apply Patch Sessions
Log file name or directory under FA_ORACLE_HOME/admin/FUSION/log | Log file description |
---|---|
FAPMgr_Multiapply_apply_timestamp.log |
Master log file from a multi-apply patching session |
FAPMgr_Multiapply_DiagnosticsSummary_timestamp.html |
Master Diagnostics report from a multi-apply patching session in HTML format |
FAPMgr_Multiapply_DiagnosticsSummary_timestamp.xml |
Master Diagnostics report from a multi-apply patching session in XML format |
diaghtmllogs_mode_timestamp |
The Diagnostics report from a multi-apply patching session in |
FAPMgrDiagnosticsSummary_apply_timestamp.html |
Diagnostics report from a multi-apply patching session in HTML format |
ADPatchDiagnosticsSummary_apply_patchnumber_timestamp.html |
Diagnostics report for one patch in a multi-apply patching session in HTML format |
logsummary_apply_timestamp.html |
The consolidation of the log files generated by multi-apply patching session in HTML format for viewing and accessing links to other log files from a browser |
At the beginning of a multi-apply patching session, the log files from the previous session move to FA_ORACLE_HOME
/admin/FUSION/logarchive/MAPPLY/
timestamp
. For each patch that is applied as part of a group, a LogFilesLocationPointer.log
is created at the location where the log files will be archived, as if the patch was applied as a standalone patch. The content of the pointer files provides the absolute location of the corresponding group apply logs and provides the ability to navigate directly to the log files for a specific patch, even though it is applied using multi-apply. For example, if patches 19191630 and 20193112 are grouped into GROUP1
and then applied, log files are archived in the following directory structure:
logarchive | |-- MAPPLY | |-- 20120610235002 | | |-- FAPMgr_Multiapply_apply_20120610235002.log | | |-- FAPMgr_Multiapply_DiagnosticsSummary_20120610235002.html | | |-- FAPMgr_Multiapply_DiagnosticsSummary_20120610235002.xml | | |-- GROUP1 | | | |-- APPLY | | | | |-- 123 | | | | | |-- 20120610235302 | | | | | | |-- FAPatchManager_apply_20120610235302.log | | | |-- VALIDATE | | | | |-- 122 | | | | | |-- 20120610235102 | | | | | | |-- FAPatchManager_validate_20120610235102.log |-- 19191630 | |-- APPLY | | |-- 123 | | | |-- 20120610235302 | | | | |-- LogFilesLocationPointer.log | |-- VALIDATE | | |-- 122 | | | |-- 20120610235102 | | | | |-- LogFilesLocationPointer.log |-- 20193112 | |-- APPLY | | |-- 123 | | | |-- 20120610235302 | | | | |-- LogFilesLocationPointer.log | |-- VALIDATE | | |-- 122 | | | |-- 20120610235102 | | | | |-- LogFilesLocationPointer.log
For information about the duration of patching tasks, refer to the Diagnostics Report, in Section 11.2.2, "Diagnostics Report".
Patch Manager coordinates patching activities by assigning tasks based on information from the patch metadata file. OPatch runs the tasks for updating middleware artifacts and AutoPatch runs the database tasks. Each of these tools generates one or more log files containing informational and error messages generated during patching. For more information, see Section 11.1.1, "Log Files for Single Patch Manager Sessions". If a task fails, the exact failure information for a given task is included in the log file. You can view the progress of the patching session from a browser, including the details of a failed task, by reviewing the Log Summary.
The Log Summary is created automatically whenever you start an Patch Manager session. The Log Summary is continuously updated as the session progresses. This report exists in the FA_ORACLE_HOME
/admin/FUSION/log
directory and is named logsummary
_fapmgr_command_timestamp.
html
. It contains links to all of the log files generated during the session. To view the report, open the report from your browser and periodically refresh the page to see updated links to log files as they are created. You can open those links and monitor the progress of the session by refreshing the browser.
After each patching sessions ends, the Diagnostics report is automatically generated so that you can view the results of the session from a browser. You can also use this report during a patching session that is currently running, by generating the report from the command line. The Diagnostics report is located in the FA_ORACLE_HOME
/admin/FUSION/log
directory and is named FAPMgrDiagnosticsSummary
fapmgr_command_timestamp
.html
. For more information, see Section 3.5.5, "Diagnostics Report".
For the fapmgr apply
and validate
commands, the Diagnostics report contains links to the line number in the logs file for each task. These links take you to the specific line in the corresponding HTML log files. The HTML log files exist in the directory diaghtmllogs_
mode_timestamp
directory.
This section contains the following general troubleshooting scenarios for patching:
Starting a New Patching Session After the Previous Session Failed
Backing Out Patches After They Have Been Successfully Applied
For troubleshooting information that is specific to patching security artifacts such as the jazn-data.xml
file, data security files, and data role templates, see Section 4.18, "Patching Security Artifacts". For troubleshooting information that is specific to patching SOA composites, see Section 11.4, "Troubleshooting Patching Sessions for SOA Composites". For troubleshooting information that is specific to patching database content, see Section 11.5, "Troubleshooting Patching Sessions for Database Content".
The previous patching session failed, and when you attempt to start a new patching session, a message appears about a previous session having failed.
The previous patching session could be in various states after its failure. The following scenarios are possible:
To abandon the previous session and start a new session, see Section 11.3.2, "Abandoning a Failed Patching Session".
In some cases, the patch tables do not correctly reflect the failed state of the patching session and may still show a patch task as running. In this case, you must use the forcefail
command to fail the session. Then you can abandon the current patching session. For more information, see Section 11.3.3, "Recovering from an Interrupted Patching Session".
There can be only one patching session active at one time for Oracle Fusion Applications, Oracle Fusion Functional Setup Manager, and Oracle Fusion Middleware Extensions for Applications (Applications Core). If there is a failed Applications Core or Functional Setup Manager patching session that must be cleaned up, see Section 7.4.6, "Abandoning a Failed Patching Session".
The previous patching session failed and you want to start a new patching session. Only one patching session can be running at a time. If aborted, the patching session cannot be restarted and any pending patching actions, such as deployment actions, must be performed manually.
Always make sure that processes associated with the previous patching session do not exist. You can abandon a previously failed session by running the fapmgr
abort
command so that you can start a new patching session. The abort
command cleans up any intermediate states tracked by fapmgr
and moves the log files for the abandoned session to an archive log directory. You cannot abandon a session that is actively running.
Use the following syntax for the fapmgr
abort
command:
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh abort [-logfile log file name] [-loglevel level] (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd abort [-logfile log file name] [-loglevel level]
Table 11-3 describes the options available for the abort
command.
Table 11-3 abort Command Options
Option | Description | Mandatory |
---|---|---|
logfile |
Name of the log file. |
No, default value is FAPatchManager_abort_timestamp.log |
loglevel |
Reporting level for messages. See Section 11.1, "Oracle Fusion Applications Patch Manager Logging". |
No, default value is INFO |
help |
Displays help. |
No |
If the fapmgr
abort
command errors with a message, such as, "Another APPLY session is already running", you may need to use the fapmgr forcefail
command first. For more information, see Section 11.3.3, "Recovering from an Interrupted Patching Session". Also confirm that the table FND_INSTALL_PROCESSES does not exist.
The patching session was interrupted by a system failure when Patch Manager and AutoPatch were running. The patching-related database tables still show the patching session as running, but no patching-related processes are actually running.
Use the fapmgr
forcefail
command to update the patching tables. Confirm that no patching processes are running before you use this command.
Confirm that all processes related to this patching session are no longer active. From your operating system, check for processes that are running fapmgr, javaworker, adpatch, adadmin
, and adworker
. If any processes are running, you must stop them using the command appropriate for your operating system.
Use the fapmgr forcefail
command to update the patching tables.
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail [-logfile log file name] [-loglevel level] (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd forcefail [-logfile log file name] [-loglevel level]
From this point you can either abandon or restart the failed session.
To abandon the failed patching session, follow the steps in Section 11.3.2, "Abandoning a Failed Patching Session".
To restart the failed patching session, use the fapmgr apply
command to apply the same patch. The session starts from the failed task.
Note:
Patch Manager, Functional Setup Manager, and Applications Core all use AutoPatch for database patching. If a patching session is hung or incomplete, you may potentially need to consider the impact of an Applications Core or Functional Setup Manager patching session. Only one patching session can be active at a time.
If you initiate a patching session from a terminal server, such as a laptop, you may lose the connection during extended periods of time when no messages are sent to the terminal. The terminal server may interpret this as inactivity and then end the session. For example, no messages are sent to the client when Patch Manager is stopping and starting servers, waiting for a failed task to be fixed, or is hung on a database task. To avoid this situation, ensure that the terminal server is configured appropriately to handle long durations of inactivity.
If you recently applied a one-off patch to a middleware artifact, you may encounter the following error the next time you apply a standard patch:
The incoming patch(es) [patch_number] target components that are locked bysingleton patch(es) [patch_number]. OPatch cannot proceed. Please refer to log file for more details.
A one-off middleware patch sets a lock on the component that the one-off patch updates. After you apply a one-off patch, you must subsequently apply the related standard patch that contains the same updates as the one-off to release the lock. The error message tells you which one-off patch is locking the component. For more information, see Section 2.2.1, "Impact of a One-off Patch".
Note:
OPatch uses the term singleton patch, which is equivalent to a one-off patch, as used by Patch Manager.
If you apply a patch that contains BI Publisher artifacts, the BI Presentation servers should not be running. The following error occurs if the BI Presentation servers are running during the deployment of BI Publisher artifacts:
java.lang.RuntimeException: Webcat patch file creation failed!
To resolve this issue, shut down the BI Presentation servers to release locks on the Oracle BI Presentation Catalog. For more information, see "fastartstop Syntax" in the Oracle Fusion Applications Administrator's Guide.
If you receive the following error during patch validation,
weblogic.management.mbeanservers.edit.EditTimedOutException
you may have to manually release the edit session. This situation can occur when a domain is already in "edit" mode during patching, such as when the server crashes when Patch Manager tries to stop and restart it.
Follow these steps to manually release the edit session:
Log in to the admin console for the domain that is locked in edit mode.
In the admin console, confirm that Release Configuration is enabled in the Change Center menu.
Click Release Configuration to release the edit session.
After you apply a patch that contains flexfield changes and you decide you are not ready to implement those flexfield changes, you have the option to revert to a previous version of that flexfield definition. The Flex Modeler creates an MDS label, FlexPatchingWatermark
date+time
, before it initiates the flexfield deployment, which establishes the watermark for what was in MDS before the patch was applied. For your reference, the name of the label is included along with the Flex Deployment report in the patching log file. To use a previous version of a flexfield definition, use the WLST command promoteMetadataLabel
. For more information, see "promoteMetadataLabel" in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.To delete all previous MDS labels for a flexfield, after you confirm that you can use the changes delivered by a patch, use the WLST command deleteFlexPatchingLabels
. Note that keeping old MDS labels adversely impacts performance. For more information, see "Using the WLST Flexfield Commands to Deploy Flexfields" in the Oracle Fusion Applications Developer's Guide.
If a patch contains BI artifacts the BI OPMN control process, which is similar to a node manager, has to be up for online mode validation to succeed. Online validation reports the following error if the BI OPMN control process is not up:
The deployment of BI Publisher artifacts will not be attempted because the BI Presentation server is neither fully started nor down. One likely cause is that the BI OPMN control process is not running. Make sure that the BI OPMN control process is up and the BI Presentation server is started successfully before applying this patch. If this server is not fully started, you must stop the BI Presentation server, manually deploy the BI Publisher artifacts, and then re-start the BI Presentation server
To resolve the issue, you must ensure that the BI OPMN control process is up and running.
The opatch -lsinventory -detail
command provides a report that lists all patches, artifacts and artifact versions that were modified within each patch applied to a given Oracle home. This report lists only artifacts that were modified. If an artifact does not appear on this report, then the artifact remains at its base version. Run this report when you are working with Oracle Support Services and you need to provide an artifact version.
Use this command syntax to generate the report:
opatch/opatch lsinventory -detail -oh FA_ORACLE_HOME -invPtrLoc \
FA_ORACLE_HOME/oraInst.loc -jre JAVA_HOME
The following example depicts the section of the report that displays patches applied. To use the report, you must find the latest entry for the specific artifact and note the version reported.
Interim patches (11) : Patch 11801 : applied on Wed Feb 02 17:57:53 PST 2011 Created on 18 Jan 2011, 16:09:54 hrs PST8PDT Bugs fixed: 11801 Patch is translatable. Files Touched: AdfPjfIntMspUi.jar, version "23.0" --> ORACLE_HOME/prj/deploy/EARProjectsFinancials.ear/EARProjectsFinancials.war/WEB-INF/lib/AdfPjfIntMspUi.jar Patch Location in Inventory: /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/inventory/oneoffs/11801 Patch Location in Storage area: /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/.patch_storage/11801_Jan_18_2011_16_09_54 Patch 12801 : applied on Tue Feb 01 21:30:17 PST 2011 Created on 28 Jan 2011, 14:26:56 hrs PST8PDT Bugs fixed: 12801 Patch is translatable. Files Touched: EFFmetadata.mar, version "35.0" --> ORACLE_HOME/hcm/deploy/EarHcmCoreSetup.ear/EFFmetadata.mar system-jazn-data.xml, version "35.0" --> ORACLE_HOME/hcm/security/policies/system-jazn-data.xml Patch Location in Inventory: /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/inventory/oneoffs/12801 Patch Location in Storage area: /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/.patch_storage/12801_Jan_28_2011_14_26_56
For more information about OPatch, see the Oracle Fusion Middleware Patching Guide.
You should always test the application of a patch and the functionality related to the patch on a test system. After the patch is successfully tested, apply the patch on the production system. There is no automated method of backing out patches. Oracle strongly recommends that you work with Oracle Support Services if you must back out a patch.
The information provided in this section describes the troubleshooting process for errors that can occur when patching Service-Oriented Architecture (SOA) composites. These processes assume that you validate the patch and then apply the patch in online mode. SOA patching errors can be divided into the following categories:
Error occurs during validation
Patch Manager can detect and report validation errors before changes have occurred to the file system. If you do not resolve validation errors before applying the patch, the patch fails and you must manually deploy the SOA composites after you resolve the validation errors.
Error occurs during the patch apply phase
These errors may require contacting Oracle Support Services to restore the system back to a known working state and can be further divided into these categories:
The SOA composite failed to deploy and Patch Manager recovered from the failure.
The SOA composite was not deployed successfully and the recovery failed. Therefore, the composite may be partially deployed.
The system is in an unknown state, as the result of a timeout occurring during deployment. Patch Manager cannot determine if the SOA composite is deployed, not deployed, or partially deployed.
When SOA composite failures occur, review the error messages in the Diagnostics report and related log files and follow the applicable steps in one or more of the following categories:
SOA composite validation and deployment can fail for multiple reasons. Review the following steps for basic troubleshooting:
After you validate or apply a patch that contains SOA composites, review the Diagnostics report for errors.
Patch Manager generates the Diagnostics report after every validation and patching session. The report output is in HTML format for viewing from a browser and is located here: FA_ORACLE_HOME
/admin/FUSION/log/FAPMgrDiagnosticsSummary
date:session
.html
.
The Module Task Details section of report displays the following information to assist in your troubleshooting:
Mode: Middleware, in this case.
Phase: Validation or Patch Application, in this case.
Product Family: The short name of the product family.
Task: The following information displays for SOA composites:
Name of composite
Domain name
Path to composite JAR in FA_ORACLE_HOME
Revision of composite
Status: Possible values of Success, Failed, or Skipped.
Duration: Total time the task ran.
Start Time: Time and date the task started.
End Time: Time and date the task ended.
Warning/Error Message: The error message displays as a java.lang.RuntimeException
. The message often includes a suggestion for resolving the failure.
Log file: The path and file name of the high level log file, FAPatchManager
_fapmgr_command
_timestamp
.log, associated with the task. From the Module Execution Summary section of the Diagnostics report, you can review log files by accessing the link to the Log Summary. For more information, see Section 11.2.1, "Log Summary".
Line Numbers: The line numbers in the log file associated with the task.
SOA log files are located in this directory: FA_ORACLE_HOME
/admin/FUSION/log/fapatch/fapatch
_release_number
/soalogs
. If merge operations are performed on a SOA composite, due to runtime customizations, such as design time and run-time (DT@RT) changes or property changes, a merge log file is generated. There is one merge log file per domain and the name of merge log files follows this naming convention: fapatch
_domain_nametimestamp
.merge.log
Restart the SOA servers and for each failure, follow Steps 4 through 9.
Determine if there is a cause for the error that must be resolved, in addition to restarting the server, by referring to the Diagnostics report, Oracle Fusion Applications log files and SOA log files. Examples of other causes include database errors, coherence configuration errors, and out of memory issues. For more information, see "Troubleshooting Oracle SOA Suite" in the Oracle Fusion Applications Administrator's Troubleshooting Guide.
Determine if you need to manually restore the system back to its state before the application of the patch was attempted. The following scenarios do not require manual restoration of the system:
Errors occurred during the validation phase.
Errors occurred during the patch application phase but the recovery was successful, so the system was recovered to its original state. The Diagnostics report displays this condition in this message:
Deployment of SOA composite[artifact_name and path]failed, but the system has recovered successfully.
Suggestion: You must manually deploy the composite using the WLST command.
If you need to restore the system, follow the steps in Step 6.
If a failed deployment leaves a composite in an inconsistent state, you must restore the system to its original state. Follow these steps to use WLST commands to restore the system. If you prefer to use Oracle Fusion Applications Control to restore the system, see Step 7.
Find out what the active revision of the composite was before the application of the patch from the Diagnostics report, as indicated by this message: The last active version of the composite before patch application began was
[version
].
In the following examples, 1.0 is the previous revision and 2.0 is the patched revision.
Undeploy the patched revision of the composite if it exists in the system.
sca_undeployComposite('http://server01:8001', 'POProcessing', '2.0')
Mark the previously active revision of the composite as active and as a default revision.
To activate the old revision:
sca_activateCompositeMb('POProcessing', '1.0')
To assign the default composite:
sca_assignDefaultCompositeMb('POProcessing', '1.0')
Follow these steps to restore the system to its original state using Oracle Fusion Applications Control. For more information, see "Accessing Fusion Applications Control" in the Oracle Fusion Applications Administrator's Guide.
Find the active revision of the composite before the application of the patch from the Diagnostics report, as indicated by this message: The last active version of the composite before patch application began was
[version
].
In the following example, 1.0 is the previous revision and 2.0 is the patched revision.
Undeploy the patched revision of the composite if it exists in the system. For more information, see "Undeploying SOA Composite Applications" in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
Mark the previously active revision of the composite as active and as a default revision. For more information, see "Managing the State of All Applications at the SOA Infrastructure Level" in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
Manually deploy the SOA composites included in the patch by following the steps in Section 4.19.2, "Manually Deploying SOA Composites".
If you are unable to resolve the failure, file a service request with Oracle Support Services and provide the logs and information as described in "Table 10-2 SOA Log Information for Oracle Support Services" in the Oracle Fusion Applications Administrator's Troubleshooting Guide.
This section describes common problems and solutions for SOA composite validation failures. Errors that occur during the validation phase must be resolved before applying the patch. If you encounter these errors during patch application, you must manually deploy the SOA composites after you resolve the validation errors. Patch Manager captures the OPatch validation log files, which can be found by referencing the Diagnostics report or the Log Summary. The errors in the log files provide information about the cause of the failure and are often followed by recommended actions.
This section contains troubleshooting information about the following failures:
An error that is related to a JDeveloper customizations occurs when you customized a SOA composite and did not save the customizations. You must save these customizations before you apply a patch that includes the next revision of the composite. Follow the steps in Section 4.19.1, "Preserving SOA Composite JDeveloper Customizations Before Applying a Patch" to resolve this error.
If the SOA server is down or not available for patching, patch validation succeeds, but you receive a warning message stating that the deployment of the composite will not be performed because the SOA infrastructure is down.
Use Oracle Enterprise Manager Fusion Applications Control (Fusion Applications Control) to check the state of the SOA server. For example, if an "Initializing SOA" warning message displays on the home page, Oracle recommends that you wait until the SOA server is completely up and running, with all composites initialized.
For more information, see "SOA Server Does Not Start" in the Oracle Fusion Applications Administrator's Troubleshooting Guide and "Accessing Fusion Applications Control" in the Oracle Fusion Applications Administrator's Guide.
If the Administration Server is down or not available for patching, patch validation fails. Use Fusion Applications Control to check the state of the Administration Server. For more information, see "Accessing Fusion Applications Control" in the Oracle Fusion Applications Administrator's Guide.
If the SOA-infra server is down or not available for patching, patch validation fails.
Use Fusion Applications Control to check the state of the SOA-infra server. Confirm that all dependent services are running and that all composites deployed into the SOA-infra server are present. It may take some time after SOA-infra is up for all services to initialize. If you are using a cluster, you must perform this check for all SOA-infra servers in the cluster. For more information, see "Accessing Fusion Applications Control" in the Oracle Fusion Applications Administrator's Guide.
If you receive an error stating that a composite with a specific revision is already deployed, the SOA composite in the patch was previously deployed by a patch or manually by a user.
You can resolve this error either by not applying the current patch or by undeploying the composite before applying the patch. Note that if you undeploy the composite, you lose any customizations you may have made to that composite. Use the following command to undeploy a composite.
sca_undeployComposite(serverURL, compositeName, revision)
For more information about using the sca_undeployComposite
command, see "Undeploy SOA Composites Using WSLT Command" in the Oracle Fusion Applications Upgrade Guide.
This section describes common problems and solutions for SOA composite deployment failures during patching. Errors that occur during the deployment phase must be resolved as soon as possible because the system has been modified. Do not try to roll back or reapply patches that have errors during deployment. After you resolve the cause of the error, you must deploy the composite manually.
This section contains the following topics:
If you receive an error in making the new composite the default, you can manually assign the new composite as the default.
Use the following WLST command to assign the new composite as the default:
sca_assignDefaultComposite('host', 'soaport', 'user', 'password', 'composite_name', 'composite_revision')
If the WLST command is successful, verify if the new composite is active. If it is not, you must then manually deploy the composite that failed, by following the steps in Section 4.19.2, "Manually Deploying SOA Composites".
For more information, see "sca_assignDefaultComposite" in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
If you receive an error in retiring the previous version of the composite, the old composite was not retired and both the new and old composites may be running. The old SOA composite was supposed to be retired so that only the new SOA composite would be active.
To resolve this error, use the following Oracle WebLogic Scripting Tool (WLST) command to retire the old composite:
sca_retireComposite('host', 'soaport', 'user', 'password', 'composite_name', 'composite_revision')
If the WLST command is successful, verify if the new composite is active. If it is not, you must then manually deploy the composite that failed, by following the steps in Section 4.19.2, "Manually Deploying SOA Composites".
For more information, see "sca_retireComposite" in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
Custom metadata and key flexfield changes are not propagated across clusters after applying a patch.
Each SOA cluster maintains its own SOA MDS schema, which results in a duplicate set of metadata for each SOA cluster that must be synchronized. Patch Manager manages this synchronization, but any custom metadata or flexfield metadata must be manually exported from a source system and then migrated to the other systems. To analyze, export, and import the metadata, follow the steps in "Task: Synchronizing Customized Flexfields in the MDS Repository for SOA" in the Oracle Fusion Applications Extensibility Guide for Developers.
The following failures may require contacting Oracle Support Services. If you are unable to resolve the failure after following the steps in Section 11.4.1, "Basic Troubleshooting for SOA Composite Failures", file a service request with Oracle Support Services and provide the logs and information as described in "SOA Log Information for Oracle Support Services" in the Oracle Fusion Applications Administrator's Guide.
An earlier revision of the SOA composite, which is being patched, is not currently deployed in the system. This could mean that the composite was not previously provisioned on the system. Therefore, the patch validation reports the following error:
The base composite is not set as default composite. Suggestion: You must manually set the base composite as the default composite using the WLST command.
The SOA composite fails during export actions, extract or attach plans, or merge updates, causing patch validation to the report the following error:
Deployment of SOA composite [{0}] failed at preparation step. Reason: [{1}] Suggestion: You must manually deploy the composite using the WLST command.
The deployment of the composite reported an unknown deployment status, as shown by the following example message:
No information is available about the recovery status. The RecoverStatus object obtained is null.
The AD Controller utility, adctrl
, can monitor and control the progress of the workers called by AutoPatch to update database content and by AD Administration. With AD Controller, you can determine the status of the workers and control their actions. For more information about workers, see Section 3.1.2.1, "Worker Processes".
The following sections contain steps for troubleshooting issues that may occur during patching sessions for database content:
Follow these steps to start AD Controller:
Start AD Controller with the adctrl
command.
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/adctrl.sh (Windows) FA_ORACLE_HOME\lcm\ad\bin\adctrl.cmd
It prompts you to:
Confirm the value of the Oracle Fusion Applications Oracle home
Specify an AD Controller log file. This log file is written to the current working directory. The default is adctrl.log
.
After the main menu displays, enter a number to select an option. You can press Enter at any time to return to the AD Controller main menu.
When AutoPatch or AD Administration runs tasks in parallel, it assigns tasks to workers for completion. There may be situations that cause a worker to stop processing. You can use AD Controller to determine the status of workers and manage worker actions. You can also find the status of workers by reviewing the Log Summary. For more information, see Section 11.2.1, "Log Summary".
Follow these steps to review the status of the workers from AD Controller:
Start AD Controller. For more information, see Section 11.5.1, "Starting AD Controller".
Review worker status.
Select Show worker status from the AD Controller main menu. AD Controller displays a summary of current worker activity. The summary columns are:
Control Code: The last instruction from the manager to the worker
Worker: The worker number
Context: The general action the manager is executing
Filename: The file the worker is running, if any
Status: The status of the worker
Table 11-4 describes the status that may be assigned to a worker.
Table 11-4 Worker Status
Status | Meaning |
---|---|
Assigned |
The manager assigned a task to the worker and the worker has not yet started. |
Completed |
The worker completed the task and the manager has not yet assigned it a new task. |
Failed |
The worker encountered a problem. |
Fixed, Restart |
You fixed the problem, and the worker will retry the task that failed. |
Restarted |
The worker is retrying a task or has successfully restarted a task. Note that the status does not change to Running. |
Running |
The worker is running a task. |
Wait |
The worker is idle. |
If the worker shows a status of Failed, refer to Section 11.5.3, "Determining Why a Worker Failed" for assistance in fixing the problem so Patch Manager can complete its processing.
When a worker fails its task, you do not have to wait until the other workers and the manager stop. You can review the worker log files to determine what caused the failure. Workers do not proceed to run tasks in the subsequent phase until all tasks in the current phase complete successfully. You must take action to resolve the failure so the workers can continue to run tasks in the next phase. If the task was deferred after the worker failed, there may be no action required from you.
The first time a task fails, the manager defers the task and assigns the worker a new task. If the deferred task fails a second time, the manager defers it a second time only if the run time of the task is less than 10 minutes. If the deferred task failed a third time, or if its run time is greater than 10 minutes, the task stays at a failed status and the worker waits for manual intervention. Action by you is then required because the worker stops any further processing. An example of when this scenario can occur is during seed data upload. The seed data upload may fail due to records being locked by another process. If the lock is released before the second or third attempt of the upload, the upload succeeds.
Follow these steps to find out why a worker failed:
In the Log Summary, located in FA_ORACLE_HOME/admin/FUSION/log/logsummary
_fapmgr_command_timestamp.
html
, review the AutoPatch Apply log file to find the worker that failed. For more information, see Section 11.2.1, "Log Summary".
Open and review the log file for the failed worker to determine the cause of the worker failure.
Each worker logs the status of tasks assigned to it in adpatch_apply
_timestamp
_worker
number
.log. The worker log file contains information that describes exactly what task it was running and what error occurred that resulted in a failure. Review the worker log file for the failed worker to determine the source of the error. For more information, see Section 11.1.1, "Log Files for Single Patch Manager Sessions".
Determine how to fix the problem that caused the failure.
Resolve the error using the information provided in the log files. If needed, search for the resolution at the My Oracle Support site or file a service request with Oracle Support Services if you do not understand how to resolve the issue.
After you resolve the problem that caused the failure, restart the failed worker.
Select Tell worker to restart a failed job from the AD Controller main menu. For more information, see Section 11.5.4, "Restarting a Failed Worker".
Verify the worker status.
Select Show worker status from the AD Controller main menu. The Status column for the worker that failed should now display Restarted or Fixed, Restart.
If a worker job failed or if you terminated a hanging worker process, you must manually restart the worker. Some worker processes spawn other processes called child processes. If you terminate a child process that is hung, the worker that spawned the process shows Failed as the status. After you fix the problem, you must restart the failed worker. After the worker restarts, the associated child processes start as well.
Follow these steps to restart a failed worker:
Start AD Controller. For more information, see Section 11.5.1, "Starting AD Controller".
Select Show worker status from the AD Controller main menu.
Take the appropriate action for each worker status:
If the worker status is Failed and its job has failed, select Tell worker to restart a failed job. When prompted, enter the number of the worker that failed.
If the worker status is Running or Restarted, but the job is hung, follow the steps in Section 11.5.5, "Terminating a Hung Worker Process".
When running AD utilities, there may be situations when a worker process appears to hang, or stop processing. If this occurs, you may need to terminate the process manually. After you do that, you must also restart that process manually.
Caution:
A process that appears to be hung could be a long-running task. Be careful when terminating processes.
To terminate a process, you start AD Controller, obtain the worker's process ID from your operating system, and then stop any hanging processes. After you make the necessary changes, you can restart the worker.
Take the following steps to terminate a worker process that is hung.
Start AD Controller. For more information, see Section 11.5.1, "Starting AD Controller".
Select Show worker status from the AD Controller main menu.
Open and review the log file for the failed worker to determine the cause of the worker failure.
Each worker logs the status of tasks assigned to it in the adpatch_worker
number
.log
. The worker log file contains information that describes exactly what tasks it runs and what errors occurred that resulted in a failure. Review the worker log file for the failed worker to determine the file being processed and review the worker log file to see if it is progressing. You can also verify whether the process is consuming CPU resources from your operating system.
For more information, see Section 11.1.1, "Log Files for Single Patch Manager Sessions".
Confirm that the operating system process associated with the worker is not running. If the task is identified as "hanging", determine the worker's process ID by looking for processes being run by the worker.
(UNIX) ps -a| grep workernumber
(Windows) Start the Task Manager (Ctrl-Alt-Delete) to view processes.
Determine what processes the worker has started, if any. If there are child processes, get their process IDs.
Stop the hung process, using the command that is appropriate for your operating system.
If you terminate a SQL*Plus session spawned by a worker, you just need to kill the SQL*Plus session. The worker immediately detects the FAILED state. For other processes, follow Steps 7 through 11.
In AD Controller, select Tell manager that a worker failed its job and enter the worker number of the hung workers. This should cause the worker to fail.
Select Tell worker to quit. When prompted, enter the worker number of the hung worker.
Select Tell manager that a worker acknowledges quit. When prompted, enter the number of the hung worker.
Fix the issue that caused the worker to hang. Search the My Oracle Support site or file a service request with Oracle Support Services if you do not know how to fix the issue.
Select Restart a worker on the current machine. When prompted, enter the number of the failed worker. The worker will restart its assigned tasks and spawn the associated child processes.
Note:
Do not select Restart a worker on the current machine if the worker process is running. Doing so creates duplicate worker processes with the same worker ID and will cause incorrect results.
There may be situations when you must shut down an AD utility while it is running. For example, you may want to shut down the database while Oracle Fusion Applications is running AutoPatch or during an AD Administration session. You should perform this shutdown in an orderly fashion so that it does not affect your data. The best way to do this is to shut down the workers manually.
Start AD Controller.
Select Tell worker to quit, and enter all
for the worker number. The worker completes its current task and then quits.
Verify that no worker processes are running, using a command that is appropriate for your operating system.
When all the workers have shut down, the manager and the AD utility quits.
No workers are running tasks. They are either failed or waiting. A restarted worker resumes the failed task immediately if the worker process is running. Workers change to a Waiting status if they cannot run any tasks because of dependencies on a failed task, or because there are no tasks left in the phase. When no workers are able to run, the manager becomes idle.
Complete the following steps for each worker:
Start AD Controller. For more information, see Section 11.5.1, "Starting AD Controller".
Determine the cause of the error.
Select Show worker status. Review the worker log file for the failed worker to determine the cause of the error.
Resolve the error using the information provided in the log files.
Search for the resolution in the My Oracle Support site or file a service request with Oracle Support Services if you do not understand how to resolve the issue.
Restart the failed worker.
Select Tell worker to restart a failed job on the AD Controller main menu. The worker process restarts, causing the AD utility and the manager to become active again.
This error occurs during patching, "Unable to start universal connection pool". Connections to the database cannot be established due to timeout limits.
Consider tuning the listener parameters INBOUND_CONNECT_TIMEOUT_listenername
and SQLNET.INBOUND_CONNECT_TIMEOUT
for the server. For more information, see "SQLNET.EXPIRE_TIME Parameter" and "INBOUND_CONNECT_TIMEOUT Parameter" in the Oracle Fusion Applications Performance and Tuning Guide.
When you patch database artifacts, your system should be in an idle state. If this is not the case, the patching session may hang due to locks. Examples of locks that can cause the patching session to hang are PL/SQL packages that are accessed by Oracle Enterprise Scheduler Service Server, a locked table, or a locked table row. After a specific wait time, such as 30 minutes, Patch Manager performs a check to determine whether the patching session is blocked by another session. If a blocking session is found, a message describing the block is sent to the log file, as shown in the following example:
"[2011-03-14T02:12:18.112-07:00] [apps] [NOTIFICATION] [] [AutoPatch] Worker 4 is blocked by session 3868 ... Please fix the session to avoid indefinite waiting
The worker that is blocked remains in a Running status. To resolve the issue and release the lock, stop the blocking session using the command that is appropriate for your operating system. After the blocking session is no longer running, the hung worker proceeds to complete its task. You can use the following SQL*Plus query to identify the sessions that are blocking patching sessions:
SELECT blocking_session, sid "Blocked Session", module "Blocked Module", seconds_in_wait FROM gv$session WHERE status = 'ACTIVE' AND module like 'PATCHING_SESSION:%' AND blocking_session_status = 'VALID' AND user = '<FUSION schema>';
Note:
To minimize blocked sessions, ensure that you follow the steps in Step 7, "Prepare the System" before you apply a patch.
If you recently applied a one-off patch to a database artifact, you may encounter the following error the next time you apply a standard patch:
Error occurred during patch conflicts checking. This may be due to infrastructure failure OR patch conflicts You may check FA_ORACLE_HOME/admin/FUSION/out/patch_number_conflict_report.xml file for any patch conflicts You should check the file FA_ORACLE_HOME/admin/FUSION/log/adpatch.log for errors
A one-off database patch sets a lock on the artifact that the one-off patch updates. After you apply a one-off patch, you must subsequently apply the related standard patch that contains the same updates as the one-off to release the lock. To find out which one-off patch is locking the artifact, review the conflict_report.xml
file. In the following example of this file, the one-off patch that created the lock was 909090.
<Patch_Conflict_Report> <instance> <appl_sys_name>FUSION</appl_sys_name> <appl_top>/server01/fusionapps</appl_top> <patch_number>909090</patch_number> </instance> <patch_type>one-off</patch_type> <conflict_details> <conflicts> <prod>HCM</prod> <subdir>hcm/components/hcmPayroll/legconfig/setup/dbSchema/database/fusionDB/FUSION</subdir> <filename>PAY_INSTALLED_LEGISLATIONS.table</filename> <bug>909090</bug> </conflicts>
To resolve this issue and remove the lock, obtain and apply the standard patch that delivers the same fix as the one-off patch.
When multiple seed data files are uploaded for the same flexfield but for different flexfield contexts, the upload tasks can fail due to locking issues. The failed tasks appear in the log file as the following error:
Loading failed with a JboException: JBO-25014: Another user has changed the row with primary keyoracle.jbo.Key ...
AutoPatch defers any failed tasks to the end of the phase and reattempts the failed tasks only after attempting all tasks in the phase at least once. Usually, the flexfield seed data tasks that failed due to the locking issue succeed on subsequent attempts. You can ignore these errors if the flexfield seed data task succeeds on the retry. If the task remains in a failed state, you must use the AD Controller utility to retry the failed task.
For more information, see Section 11.5.4, "Restarting a Failed Worker".
If you need to connect to the Oracle Fusion Applications database to troubleshoot database related issues, by running SQL*Plus, for example, you need to set up the appropriate environment variables. For setting any environment variable, run the adsetenv
script to generate the APPSORA.env
file, which when sourced, sets all environment variables.
(UNIX) sh adsetenv.sh source APPSORA.env echo $TWO_TASK (Windows, TWO_TASK is known as LOCAL) adsetenv.cmd APPSORA.cmd echo %LOCAL%