| Oracle® Fusion Applications Patching Guide 11g Release 5 (11.1.5) Part Number E16602-21 |
|
|
PDF · Mobi · ePub |
| Oracle® Fusion Applications Patching Guide 11g Release 5 (11.1.5) Part Number E16602-21 |
|
|
PDF · Mobi · ePub |
This chapter describes how to monitor and troubleshoot Oracle Fusion Applications patching and AD Administration processing sessions.
This chapter contains the following topics:
Oracle Fusion Applications Patch Manager creates log files for the actions it performs. These logging capabilities track the progress of actions and assist you in diagnosing issues. When you use Oracle Fusion Applications Patch Manager, you can specify the level of logging detail. The following log levels are available:
ERROR:1 (SEVERE) For an error that results in a failure.
WARNING:1 (WARNING) For an error that does not result in failure but that you should review.
NOTIFICATION:1 (INFO) For high-level information about the progress of the process, no action necessary.
NOTIFICATION:16 (CONFIG) For more detailed information about the progress of the process, no action necessary
TRACE:1 (FINE) For generating the first level of trace messages, used for diagnosing issues.
TRACE:16 (FINER) For generating the second level of trace messages, used for diagnosing issues.
TRACE:32 (FINEST) For generating the highest level of trace messages, used for diagnosing issues.
When you select a more detailed level of logging, the log files also include the lower level of details. For example, if you choose to see INFO messages in your log file, WARNING and SEVERE messages also appear in the log files. For more information, see "Standard Logging Levels" in the Oracle Fusion Applications Administrator's Guide.
You can examine the activities performed during patching sessions by reviewing the associated log files. Oracle Fusion Applications Patch Manager consolidates log files for each patching session under the directory, FA_ORACLE_HOME/admin/FUSION/log. This directory contains the top-level log file, logsummary_fapmgr_command_timestamp.html, along with related log files for each task performed during a fapmgr session. During a session, you can view the Log Summary HTML file from a browser, which provides links to individual log files. You can periodically refresh the Log Summary HTML file to view the progress of the current patching session. If a task fails, you can access the links to the associated log files to assist in diagnosing the failure. For more information, see Section 11.2.1, "Log Summary".
When a patching session completes, its log files are archived in FA_ORACLE_HOME/admin/FUSION/logarchive/Patch Number/fapmgr_command/session ID/timestamp. The session ID is unique and the time stamp is the start time for the session. Note that whenever Oracle Fusion Applications Patch Manager runs a command where there is no patch number, such as bootstrap, abort, and report, the archive logs are named FA_ORACLE_HOME/admin/FUSION/logarchive/fapmgr_command/timestamp.
Log files for OPatch actions are initially written to the FA_ORACLE_HOME/cfgtools/opatch/patch number_timestamp directory.
Table 11-1 contains a list of log files created by Oracle Fusion Applications Patch Manager during patching activities.
Table 11-1 Log Files for Single Oracle Fusion Applications Patch Manager Patching Activities
| Log file or directory name under FA_ORACLE_HOME/admin/FUSION/log | Log file generated from... |
|---|---|
|
FAPatchManager_apply_timestamp.log |
Oracle Fusion Applications Patch Manager apply session |
|
FAPatchManager_abort_timestamp.log |
Oracle Fusion Applications Patch Manager abort session |
|
FAPatchManager_bootstrap_timestamp.log |
Oracle Fusion Applications Patch Manager bootstrap session |
|
FAPatchManager_report_reportname_timestamp.log |
Oracle Fusion Applications Patch Manager report session |
|
FAPatchManager_validate_timestamp.log |
Oracle Fusion Applications Patch Manager validate session |
|
adpatch_apply_timestamp.log |
Oracle Fusion Applications AutoPatch apply session |
|
adpatch_abort_timestamp.log |
Oracle Fusion Applications AutoPatch abort session |
|
adpatch_apply_timestamp_workernumber.log |
Oracle Fusion Applications AutoPatch worker log file |
|
adpatch_validate_timestamp.log |
Oracle Fusion Applications AutoPatch |
|
adpatch_apply_timestamp_timingreport.lst |
Oracle Fusion Applications AutoPatch timing report |
|
logsummary_fapmgr_command_timestamp.html For reports: logsummary_report_reportname_timestamp.html |
The consolidation of the log files generated by Oracle Fusion Applications Patch Manager in HTML format for viewing and accessing links to other log files from a browser |
|
patch number_fapmgr_command_session id_timestamp.marker |
Marker file used while moving log files to a backup directory |
|
ConfigContext_timestamp.log |
Oracle Fusion Applications Patch Manager in online mode |
|
ExecutionContext_timestamp.log |
Oracle Fusion Applications Patch Manager in online mode |
|
FAPMgrDiagnosticsSummaryfapmgr_command_timestamp.html |
The consolidation of the Oracle Fusion Applications Patch Manager session in HTML format, known as the Diagnostics report |
|
FAPMgrDiagnosticsSummary.html |
The Diagnostic report when it is run on demand using this command syntax: |
|
diaghtmllogs_mode_timestamp |
The Diagnostics report for Oracle Fusion Applications Patch Manager in |
|
diaghtmllogs_mode_timestamp/FAPatchManager_apply_20120627022503.html |
Oracle Fusion Applications Patch Manager apply session |
|
diaghtmllogs_mode_timestamp/adpatch_apply_timestamp.html |
Oracle Fusion Applications AutoPatch apply session |
|
diaghtmllogs_mode_timestamp/adpatch_validate_timestamp.html |
Oracle Fusion Applications AutoPatch validate session |
|
diaghtmllogs_mode_timestamp/adpatch_apply_timestamp_workernumber.html |
Oracle Fusion Applications AutoPatch worker log file |
The multi-apply feature of Oracle Fusion Applications Patch Manager applies multiple patches after splitting them into groups within a My Oracle Support patch plan. Each group contains at least one patch and it is run as an individual session internally. Log files and diagnostic reports are generated for each individual session, in addition to a master log file and diagnostics report for the overall multi-apply session. For more information, see Section 3.1.5, "Applying Multiple Patches Using a My Oracle Support Patch Plan".
You can examine the activities performed during multi-apply patching sessions by reviewing the associated log files. Oracle Fusion Applications Patch Manager consolidates log files for each patching session under the directory, FA_ORACLE_HOME/admin/FUSION/log. This directory contains the top-level log file, logsummary_fapmgr_command_timestamp.html, along with related log files for each task performed during a multi-apply session. During a session, you can view this log summary HTML file from a browser, which provides links to individual log files. You can periodically refresh the log summary HTML file to view the progress of the current patching session. If a task fails, you can access the links to the associated log files to assist in diagnosing the failure. For more information, see Section 11.2.1, "Log Summary".
Multi-apply sessions create a master diagnostics report that provides information about the multi-apply patching session. It also includes links to corresponding diagnostics reports and archived log files for individual patches that were applied as a group.
Table 11-2 contains a list of log files created by Oracle Fusion Applications Patch Manager during multi-apply patching activities.
Table 11-2 Log Files For Multi-apply Patch Sessions
| Log file name or directory under FA_ORACLE_HOME/admin/FUSION/log | Log file description |
|---|---|
|
FAPMgr_Multiapply_apply_timestamp.log |
Master log file from a multi-apply patching session |
|
FAPMgr_Multiapply_DiagnosticsSummary_timestamp.html |
Master Diagnostics report from a multi-apply patching session in HTML format |
|
FAPMgr_Multiapply_DiagnosticsSummary_timestamp.xml |
Master Diagnostics report from a multi-apply patching session in XML format |
|
diaghtmllogs_mode_timestamp |
The Diagnostics report from a multi-apply patching session in |
|
FAPMgrDiagnosticsSummary_apply_timestamp.html |
Diagnostics report from a multi-apply patching session in HTML format |
|
ADPatchDiagnosticsSummary_apply_patchnumber_timestamp.html |
Diagnostics report for one patch in a multi-apply patching session in HTML format |
|
logsummary_apply_timestamp.html |
The consolidation of the log files generated by multi-apply patching session in HTML format for viewing and accessing links to other log files from a browser |
At the beginning of a multi-apply patching session, the log files from the previous session move to FA_ORACLE_HOME/admin/FUSION/logarchive/MAPPLY/timestamp. For each patch that is applied as part of a group, a LogFilesLocationPointer.log is created at the location where the log files will be archived, as if the patch was applied as a standalone patch. The content of the pointer files provides the absolute location of the corresponding group apply logs and provides the ability to navigate directly to the log files for a specific patch, even though it is applied using multi-apply. For example, if patches 19191630 and 20193112 are grouped into GROUP1 and then applied, log files are archived in the following directory structure:
logarchive | |-- MAPPLY | |-- 20120610235002 | | |-- FAPMgr_Multiapply_apply_20120610235002.log | | |-- FAPMgr_Multiapply_DiagnosticsSummary_20120610235002.html | | |-- FAPMgr_Multiapply_DiagnosticsSummary_20120610235002.xml | | |-- GROUP1 | | | |-- APPLY | | | | |-- 123 | | | | | |-- 20120610235302 | | | | | | |-- FAPatchManager_apply_20120610235302.log | | | |-- VALIDATE | | | | |-- 122 | | | | | |-- 20120610235102 | | | | | | |-- FAPatchManager_validate_20120610235102.log |-- 19191630 | |-- APPLY | | |-- 123 | | | |-- 20120610235302 | | | | |-- LogFilesLocationPointer.log | |-- VALIDATE | | |-- 122 | | | |-- 20120610235102 | | | | |-- LogFilesLocationPointer.log |-- 20193112 | |-- APPLY | | |-- 123 | | | |-- 20120610235302 | | | | |-- LogFilesLocationPointer.log | |-- VALIDATE | | |-- 122 | | | |-- 20120610235102 | | | | |-- LogFilesLocationPointer.log
For information about the duration of patching tasks, refer to the Diagnostics Report, in Section 11.2.2, "Diagnostics Report".
Oracle Fusion Applications Patch Manager coordinates patching activities by assigning tasks based on information from the patch metadata file. OPatch runs the tasks for updating middleware artifacts and Oracle Fusion Applications AutoPatch runs the database tasks. Each of these tools generates one or more log files containing informational and error messages generated during patching. For more information, see Section 11.1.1, "Log Files for Single Oracle Fusion Applications Patch Manager Sessions". If a task fails, the exact failure information for a given task is included in the log file. You can view the progress of the patching session from a browser, including the details of a failed task, by reviewing the Log Summary.
The Log Summary is created automatically whenever you start an Oracle Fusion Applications Patch Manager session. The Log Summary is continuously updated as the session progresses. This report exists in the FA_ORACLE_HOME/admin/FUSION/log directory and is named logsummary_fapmgr_command_timestamp.html. It contains links to all of the log files generated during the session. To view the report, open the report from your browser and periodically refresh the page to see updated links to log files as they are created. You can open those links and monitor the progress of the session by refreshing the browser.
After each patching sessions ends, the Diagnostics report is automatically generated so that you can view the results of the session from a browser. You can also use this report during a patching session that is currently running, by generating the report from the command line. The Diagnostics report is located in the FA_ORACLE_HOME/admin/FUSION/log directory and is named FAPMgrDiagnosticsSummaryfapmgr_command_timestamp.html. For more information, see Section 3.5.5, "Diagnostics Report".
For the fapmgr apply and validate commands, the Diagnostics report contains links to the line number in the logs file for each task . These links take you to the specific line in the corresponding html log files. The html log files exist in the directory diaghtmllogs_mode_timestamp directory.
This section contains the following general troubleshooting scenarios for patching:
Starting a New Patching Session After the Previous Session Failed
Backing Out Patches After They Have Been Successfully Applied
For troubleshooting information that is specific to patching security artifacts such as the jazn-data.xml file, data security files, and data role templates, see Section 4.17, "Patching Security Artifacts". For troubleshooting information that is specific to patching SOA composites, see Section 11.4, "Troubleshooting Patching Sessions for SOA Composites". For troubleshooting information that is specific to patching database content, see Section 11.5, "Troubleshooting Patching Sessions for Database Content".
The previous patching session failed, and when you attempt to start a new patching session, a message appears about a previous session having failed.
The previous patching session could be in various states after its failure. The following scenarios are possible:
To abandon the previous session and start a new session, see Section 11.3.2, "Abandoning a Failed Patching Session".
In some cases, the patch tables do not correctly reflect the failed state of the patching session and may still show a patch task as running. In this case, you must use the forcefail command to fail the session. Then you can abandon the current patching session. For more information, see Section 11.3.3, "Recovering from an Interrupted Patching Session".
There can be only one patching session active at one time for Oracle Fusion Applications, Oracle Fusion Functional Setup Manager, and Oracle Fusion Middleware Extensions for Applications (Applications Core). If there is a failed Applications Core or Functional Setup Manager patching session that needs to be cleaned up, see Section 7.4.6, "Abandoning a Failed Patching Session".
The previous patching session failed and you want to start a new patching session. Only one patching session can be running at a time.
Always make sure that processes associated with the previous patching session do not exist. You can abandon a previously failed session by running the fapmgr abort command so that you can start a new patching session. The abort command cleans up any intermediate states tracked by fapmgr and moves the log files for the abandoned session to an archive log directory. You cannot abandon a session that is actively running.
Use the following syntax for the fapmgr abort command:
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh abort [-logfile log file name] [-loglevel level] (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd abort [-logfile log file name] [-loglevel level]
Table 11-3 describes the options available for the abort command.
Table 11-3 abort Command Options
| Option | Description | Mandatory |
|---|---|---|
|
logfile |
Name of the log file. |
No, default value is FAPatchManager_abort_timestamp.log |
|
loglevel |
Reporting level for messages. See Section 11.1, "Oracle Fusion Applications Patch Manager Logging". |
No, default value is INFO |
|
help |
Displays help. |
No |
If the fapmgr abort command errors with a message, such as, "Another APPLY session is already running", you may need to use the fapmgr forcefail command first. For more information, see Section 11.3.3, "Recovering from an Interrupted Patching Session". Also confirm that the table FND_INSTALL_PROCESSES does not exist.
The patching session was interrupted by a system failure when Oracle Fusion Applications Patch Manager and AutoPatch were running. The patching-related database tables still show the patching session as running, but no patching-related processes are actually running.
Use the fapmgr forcefail command to update the patching tables. Confirm that no patching processes are running before you use this command.
Confirm that all processes related to this patching session are no longer active. From your operating system, check for processes that are running fapmgr, javaworker, adpatch, adadmin, and adworker. If any processes are running, you must stop them using the command appropriate for your operating system.
Use the fapmgr forcefail command to update the patching tables.
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail [-logfile log file name] [-loglevel level] (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd forcefail [-logfile log file name] [-loglevel level]
From this point you can either abandon or restart the failed session.
To abandon the failed patching session, follow the steps in Section 11.3.2, "Abandoning a Failed Patching Session".
To restart the failed patching session, use the fapmgr apply command to apply the same patch. The session starts from the failed task.
Note:
Oracle Fusion Applications Patch Manager, Functional Setup Manager, and Applications Core all use Oracle Fusion Applications AutoPatch for database patching. If a patching session is hung or incomplete, you may potentially need to consider the impact of an Applications Core or Functional Setup Manager patching session. Only one patching session can be active at a time.
When you apply patches in online mode using the stoponerror option and post-patching tasks fail, you can use the fapmgr retry command. The retry mode of fapmgr lets you attempt all failed online post-patching operations for a specific patching session. These actions include not only the restart of Managed Servers, but also automated actions for other artifacts, such as deployment of SOA composites, Oracle BPM templates, and IPM artifacts. If you do not use the fapmgr retry command, you manually perform the post-patching operations that failed.
The following conditions must be met for retry mode to work:
The session_id must exist in the database. To verify this, this SQL*Plus query should return one row:
select count(*) from ad_patch_util_sessions where session_id=session_id;
The failed session must have at least one failed life cycle operation.
The patch top directory from the original session must exist.
You must have followed these steps when you originally applied the patch:
Run fapmgr validate -online.
Fix the infrastructure failures reported by validation.
Run fapmgr apply -online -stoponerror
If you did not resolve the infrastructure failures reported by patch validation, the retry will most likely fail.
When you run fapmgr in retry mode, the same command line options that were used in the original session are used. Use the following command syntax:
(Unix) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh retry -sessionid [-logfile log_file_name] -loglevel log_file_name] [-loglevel level] (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.sh retry -sessionid [-logfile log_file_name] -loglevel log_file_name] [-loglevel level]
For the retry session, fapmgr uses the same session_id, but the timestamp is different. The log files for the apply session and the retry session are created in different directories, and the sessions create two separate Diagnostics reports.
If you initiate a patching session from a terminal server, such as a laptop, you may lose the connection during extended periods of time when no messages are sent to the terminal. The terminal server may interpret this as inactivity and then end the session. For example, no messages are sent to the client when Oracle Fusion Applications Patch Manager is stopping and starting servers, waiting for a failed task to be fixed, or is hung on a database task. To avoid this situation, ensure that the terminal server is configured appropriately to handle long durations of inactivity.
If you recently applied a one-off patch to a middleware artifact, you may encounter the following error the next time you apply a standard patch:
The incoming patch(es) [patch_number] target components that are locked bysingleton patch(es) [patch_number]. OPatch cannot proceed. Please refer to log file for more details.
A one-off middleware patch sets a lock on the component that the one-off patch updates. After you apply a one-off patch, you must subsequently apply the related standard patch that contains the same updates as the one-off to release the lock. The error message tells you which one-off patch is locking the component. For more information, see Section 2.2.1, "Impact of a One-off Patch".
Note:
OPatch uses the term singleton patch, which is equivalent to a one-off patch, as used by Oracle Fusion Applications Patch Manager.
If you apply a patch that contains BI Publisher artifacts, the BI Presentation servers should not be running. The following error occurs if the BI Presentation servers are running during the deployment of BI Publisher artifacts:
java.lang.RuntimeException: Webcat patch file creation failed!
To resolve this issue, shut down the BI Presentation servers to release locks on the Oracle BI Presentation Catalog. For more information, see "fastartstop Syntax" in the Oracle Fusion Applications Administrator's Guide.
If you receive the following error during patch validation,
weblogic.management.mbeanservers.edit.EditTimedOutException
you may have to manually release the edit session. This situation can occur when a domain is already in "edit" mode during patching, such as when the server crashes when Oracle Fusion Applications Patch Manager tries to stop and restart it.
Follow these steps to manually release the edit session:
Log in to the admin console for the domain that is locked in edit mode.
In the admin console, confirm that Release Configuration is enabled in the Change Center menu.
Click Release Configuration to release the edit session.
After you apply a patch that contains flexfield changes and you decide you are not ready to implement those flexfield changes, you have the option to revert to a previous version of that flexfield definition. The Flex Modeler creates an MDS label, FlexPatchingWatermarkdate+time, before it initiates the flexfield deployment, which establishes the watermark for what was in MDS before the patch was applied. For your reference, the name of the label is included along with the Flex Deployment report in the patching log file. To use a previous version of a flexfield definition, use the WLST command promoteMetadataLabel. For more information, see "promoteMetadataLabel" in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.To delete all previous MDS labels for a flexfield, after you confirm that you can use the changes delivered by a patch, use the WLST command deleteFlexPatchingLabels. Note that keeping old MDS labels adversely impacts performance. For more information, see "Using the WLST Flexfield Commands" in the Oracle Fusion Applications Developer's Guide.
If a patch contains BI artifacts the BI OPMN control process, which is similar to a node manager, has to be up for online mode validation to succeed. Online validation reports the following error if the BI OPMN control process is not up:
The deployment of BI Publisher artifacts will not be attempted because the BI Presentation server is neither fully started nor down. One likely cause is that the BI OPMN control process is not running. Make sure that the BI OPMN control process is up and the BI Presentation server is started successfully before applying this patch. If this server is not fully started, you must stop the BI Presentation server, manually deploy the BI Publisher artifacts, and then re-start the BI Presentation server
To resolve the issue, you must ensure that the BI OPMN control process is up and running.
The opatch -lsinventory -detail command provides a report that lists all patches, artifacts and artifact versions that were modified within each patch applied to a given Oracle home. This report lists only artifacts that were modified. If an artifact does not appear on this report, then the artifact remains at its base version. Run this report when you are working with Oracle Support Services and you need to provide an artifact version.
Use this command syntax to generate the report:
opatch/opatch lsinventory -detail -oh FA_ORACLE_HOME -invPtrLoc \
FA_ORACLE_HOME/oraInst.loc -jre JAVA_HOME
The following example depicts the section of the report that displays patches applied. To use the report, you must find the latest entry for the specific artifact and note the version reported.
Interim patches (11) :
Patch 11801 : applied on Wed Feb 02 17:57:53 PST 2011
Created on 18 Jan 2011, 16:09:54 hrs PST8PDT
Bugs fixed:
11801
Patch is translatable.
Files Touched:
AdfPjfIntMspUi.jar, version "23.0" --> ORACLE_HOME/prj/deploy/EARProjectsFinancials.ear/EARProjectsFinancials.war/WEB-INF/lib/AdfPjfIntMspUi.jar
Patch Location in Inventory:
/u01/FUSIONAPPS_APPLTOP/fusionapps/applications/inventory/oneoffs/11801
Patch Location in Storage area:
/u01/FUSIONAPPS_APPLTOP/fusionapps/applications/.patch_storage/11801_Jan_18_2011_16_09_54
Patch 12801 : applied on Tue Feb 01 21:30:17 PST 2011
Created on 28 Jan 2011, 14:26:56 hrs PST8PDT
Bugs fixed:
12801
Patch is translatable.
Files Touched:
EFFmetadata.mar, version "35.0" --> ORACLE_HOME/hcm/deploy/EarHcmCoreSetup.ear/EFFmetadata.mar
system-jazn-data.xml, version "35.0" --> ORACLE_HOME/hcm/security/policies/system-jazn-data.xml
Patch Location in Inventory:
/u01/FUSIONAPPS_APPLTOP/fusionapps/applications/inventory/oneoffs/12801
Patch Location in Storage area:
/u01/FUSIONAPPS_APPLTOP/fusionapps/applications/.patch_storage/12801_Jan_28_2011_14_26_56
For more information about OPatch, see the Oracle Fusion Middleware Patching Guide.
You should always test the application of a patch and the functionality related to the patch on a test system. After the patch is successfully tested, apply the patch on the production system. There is no automated method of backing out patches. Oracle strongly recommends that you work with Oracle Support Services if you must back out a patch.
The information provided in this section describes the troubleshooting process for errors that can occur when patching Service-Oriented Architecture (SOA) composites. These processes assume that you validate the patch and then apply the patch in online mode. SOA patching errors can be divided into the following categories:
Error occurs during validation
Oracle Fusion Applications Patch Manager can detect and report validation errors before changes have occurred to the file system. If you do not resolve validation errors before applying the patch, the patch fails and you must manually deploy the SOA composites after you resolve the validation errors.
Error occurs during the patch apply phase
These errors may require contacting Oracle Support Services to restore the system back to a known working state and can be further divided into these categories:
The SOA composite failed to deploy and Oracle Fusion Applications Patch Manager recovered from the failure.
The SOA composite was not deployed successfully and the recovery failed. Therefore, the composite may be partially deployed.
The system is in an unknown state, as the result of a timeout occurring during deployment. Oracle Fusion Applications Patch Manager cannot determine if the SOA composite is deployed, not deployed, or partially deployed.
When SOA composite failures occur, review the error messages in the Diagnostics report and related log files and follow the applicable steps in one or more of the following categories:
SOA composite validation and deployment can fail for multiple reasons. Review the following steps for basic troubleshooting:
After you validate or apply a patch that contains SOA composites, review the Diagnostics report for errors.
Oracle Fusion Applications Patch Manager generates the Diagnostics report after every validation and patching session. The report output is in HTML format for viewing from a browser and is located here: FA_ORACLE_HOME/admin/FUSION/log/FAPMgrDiagnosticsSummarydate:session.html.
The Module Task Details section of report displays the following information to assist in your troubleshooting:
Mode: Middleware, in this case.
Phase: Validation or Patch Application, in this case.
Product Family: The short name of the product family.
Task: The following information displays for SOA composites:
Name of composite
Domain name
Path to composite JAR in FA_ORACLE_HOME
Revision of composite
Status: Possible values of Success, Failed, or Skipped.
Duration: Total time the task ran.
Start Time: Time and date the task started.
End Time: Time and date the task ended.
Warning/Error Message: The error message displays as a java.lang.RuntimeException. The message often includes a suggestion for resolving the failure.
Log file: The path and file name of the high level log file, FAPatchManager_fapmgr_command_timestamp.log, associated with the task. From the Module Execution Summary section of the Diagnostics report, you can review log files by accessing the link to the Log Summary. For more information, see Section 11.2.1, "Log Summary".
Line Numbers: The line numbers in the log file associated with the task.
SOA log files are located in this directory: FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_release_number/soalogs. If merge operations are performed on a SOA composite, due to runtime customizations, such as design time and run-time (DT@RT) changes or property changes, a merge log file is generated. There is one merge log file per domain and the name of merge log files follows this naming convention: fapatch_domain_nametimestamp.merge.log
Restart the SOA servers and for each failure, follow Steps 4 through 9.
Determine if there is a cause for the error that needs to be resolved, in addition to restarting the server, by referring to the Diagnostics report, Oracle Fusion Applications log files and SOA log files. Examples of other causes include database errors, coherence configuration errors, and out of memory issues. For more information, see "Troubleshooting Oracle SOA Suite" in the Oracle Fusion Applications Administrator's Troubleshooting Guide.
Determine if you need to manually restore the system back to its state before the application of the patch was attempted. The following scenarios do not require manual restoration of the system:
Errors occurred during the validation phase.
Errors occurred during the patch application phase but the recovery was successful, so the system was recovered to its original state. The Diagnostics report displays this condition in this message:
Deployment of SOA composite[artifact_name and path]failed, but the system has recovered successfully.
Suggestion: You must manually deploy the composite using the WLST command.
If you need to restore the system, follow the steps in Step 6.
If a failed deployment leaves a composite in an inconsistent state, you must restore the system to its original state. Follow these steps to use WLST commands to restore the system. If you prefer to use Oracle Fusion Applications Control to restore the system, see Step 7.
Find out what the active revision of the composite was before the application of the patch from the Diagnostics report, as indicated by this message: The last active version of the composite before patch application began was [version].
In the following examples, 1.0 is the previous revision and 2.0 is the patched revision.
Undeploy the patched revision of the composite if it exists in the system.
sca_undeployComposite('http://server01:8001', 'POProcessing', '2.0')
Mark the previously active revision of the composite as active and as a default revision.
To activate the old revision:
sca_activateCompositeMb('POProcessing', '1.0')
To assign the default composite:
sca_assignDefaultCompositeMb('POProcessing', '1.0')
Follow these steps to restore the system to its original state using Oracle Fusion Applications Control. For more information, see "Starting Fusion Applications Control" in the Oracle Fusion Applications Administrator's Guide.
Find the active revision of the composite before the application of the patch from the Diagnostics report, as indicated by this message: The last active version of the composite before patch application began was [version].
In the following example, 1.0 is the previous revision and 2.0 is the patched revision.
Undeploy the patched revision of the composite if it exists in the system. For more information, see "Undeploying Applications" in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
Mark the previously active revision of the composite as active and as a default revision. For more information, see "Managing the State of All Applications at the SOA Infrastructure Level" in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
Manually deploy the SOA composites included in the patch by following the steps in Section 4.18.2, "Manually Deploying SOA Composites".
If you are unable to resolve the failure, file a service request with Oracle Support Services and provide the logs and information as described in "Table 9-2 SOA Log Information for Oracle Support Services" in the Oracle Fusion Applications Administrator's Troubleshooting Guide.
This section describes common problems and solutions for SOA composite validation failures. Errors that occur during the validation phase must be resolved before applying the patch. If you encounter these errors during patch application, you must manually deploy the SOA composites after you resolve the validation errors. Oracle Fusion Applications Patch Manager captures the OPatch validation log files, which can be found by referencing the Diagnostics report or the Log Summary. The errors in the log files provide information about the cause of the failure and are often followed by recommended actions.
This section contains troubleshooting information about the following failures:
An error that is related to a JDeveloper customizations occurs when you customized a SOA composite and did not save the customizations. You must save these customizations before you apply a patch that includes the next revision of the composite. Follow the steps in Section 4.18.1, "Preserving SOA Composite JDeveloper Customizations Before Applying a Patch" to resolve this error.
If the SOA server is down or not available for patching, patch validation succeeds, but you receive a warning message stating that the deployment of the composite will not be performed because the SOA infrastructure is down.
Use Oracle Enterprise Manager Fusion Applications Control (Fusion Applications Control) to check the state of the SOA server. For example, if an "Initializing SOA" warning message displays on the home page, Oracle recommends that you wait until the SOA server is completely up and running, with all composites initialized.
For more information, see "SOA Server Does Not Start" in the Oracle Fusion Applications Administrator's Troubleshooting Guide and "Starting Fusion Applications Control" in the Oracle Fusion Applications Administrator's Guide.
If the Administration Server is down or not available for patching, patch validation fails. Use Fusion Applications Control to check the state of the Administration Server. For more information, see "Starting Fusion Applications Control" in the Oracle Fusion Applications Administrator's Guide.
If the SOA-infra server is down or not available for patching, patch validation fails.
Use Fusion Applications Control to check the state of the SOA-infra server. Confirm that all dependent services are running and that all composites deployed into the SOA-infra server are present. It may take some time after SOA-infra is up for all services to initialize. If you are using a cluster, you must perform this check for all SOA-infra servers in the cluster. For more information, see "Starting Fusion Applications Control" in the Oracle Fusion Applications Administrator's Guide.
If you receive an error stating that a composite with a specific revision is already deployed, the SOA composite in the patch was previously deployed by a patch or manually by a user.
You can resolve this error either by not applying the current patch or by undeploying the composite before applying the patch. Note that if you undeploy the composite, you lose any customizations you may have made to that composite. Use the following command to undeploy a composite.
sca_undeployComposite(serverURL, compositeName, revision)
For more information about using the sca_undeployComposite command, see Section 5.7.25.2.1, "Undeploy SOA Composites Using WLST Commands".
This section describes common problems and solutions for SOA composite deployment failures during patching. Errors that occur during the deployment phase must be resolved as soon as possible because the system has been modified. Do not try to roll back or reapply patches that have errors during deployment. After you resolve the cause of the error, you must deploy the composite manually.
This section contains the following topics:
If you receive an error in making the new composite the default, you can manually assign the new composite as the default.
Use the following WLST command to assign the new composite as the default:
sca_assignDefaultComposite('host', 'soaport', 'user', 'password', 'composite_name', 'composite_revision')
If the WLST command is successful, check to see if the new composite is active. If it is not, you must then manually deploy the composite that failed, by following the steps in Section 4.18.2, "Manually Deploying SOA Composites".
For more information, see "sca_assignDefaultComposite" in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
If you receive an error in retiring the previous version of the composite, the old composite was not retired and both the new and old composites may be running. The old SOA composite was supposed to be retired so that only the new SOA composite would be active.
To resolve this error, use the following Oracle WebLogic Scripting Tool (WLST) command to retire the old composite:
sca_retireComposite('host', 'soaport', 'user', 'password', 'composite_name', 'composite_revision')
If the WLST command is successful, check to see if the new composite is active. If it is not, you must then manually deploy the composite that failed, by following the steps in Section 4.18.2, "Manually Deploying SOA Composites".
For more information, see "sca_retireComposite" in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
Custom metadata and key flexfield changes are not propagated across clusters after applying a patch.
Each SOA cluster maintains its own SOA MDS schema, which results in a duplicate set of metadata for each SOA cluster that must be synchronized. Oracle Fusion Applications Patch Manager manages this synchronization, but any custom metadata or flexfield metadata must be manually exported from a source system and then migrated to the other systems. To analyze, export, and import the metadata, follow the steps in "Task: Synchronizing Customized Flexfields in the SOA MDS Repository" in the Oracle Fusion Applications Extensibility Guide.
The following failures may require contacting Oracle Support Services. If you are unable to resolve the failure after following the steps in Section 11.4.1, "Basic Troubleshooting for SOA Composite Failures", file a service request with Oracle Support Services and provide the logs and information as described in "Table 22-2 SOA Log Information for Oracle Support Services" in the Oracle Fusion Applications Administrator's Guide.
An earlier revision of the SOA composite, which is being patched, is not currently deployed in the system. This could mean that the composite was not previously provisioned on the system. As a result, the patch validation reports the following error:
The base composite is not set as default composite. Suggestion: You must manually set the base composite as the default composite using the WLST command.
The SOA composite fails during export actions, extract or attach plans, or merge updates, causing patch validation to the report the following error:
Deployment of SOA composite [{0}] failed at preparation step. Reason: [{1}] Suggestion: You must manually deploy the composite using the WLST command.
The deployment of the composite reported an unknown deployment status, as shown by the following example message:
No information is available about the recovery status. The RecoverStatus object obtained is null.
The AD Controller utility, adctrl, can monitor and control the progress of the workers called by AutoPatch to update database content and by AD Administration. With AD Controller, you can determine the status of the workers and control their actions. For more information about workers, see Section 3.1.2.1, "Worker Processes".
The following sections contain steps for troubleshooting issues that may occur during patching sessions for database content:
Follow these steps to start AD Controller:
Start AD Controller with the adctrl command.
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/adctrl.sh (Windows) FA_ORACLE_HOME\lcm\ad\bin\adctrl.cmd
It prompts you to:
Confirm the value of the Oracle Fusion Applications Oracle home
Specify an AD Controller log file. This log file is written to the current working directory. The default is adctrl.log.
After the main menu displays, enter a number to select an option. You can press Enter at any time to return to the AD Controller main menu.
When AutoPatch or AD Administration runs tasks in parallel, it assigns tasks to workers for completion. There may be situations that cause a worker to stop processing. You can use AD Controller to determine the status of workers and manage worker actions. You can also find the status of workers by reviewing the Log Summary. For more information, see Section 11.2.1, "Log Summary".
Follow these steps to review the status of the workers from AD Controller:
Start AD Controller. For more information, see Section 11.5.1, "Starting AD Controller".
Review worker status.
Select Show worker status from the AD Controller main menu. AD Controller displays a summary of current worker activity. The summary columns are:
Control Code: The last instruction from the manager to the worker
Worker: The worker number
Context: The general action the manager is executing
Filename: The file the worker is running, if any
Status: The status of the worker
Table 11-4 describes the status that may be assigned to a worker.
Table 11-4 Worker Status
| Status | Meaning |
|---|---|
|
Assigned |
The manager assigned a task to the worker and the worker has not yet started. |
|
Completed |
The worker completed the task and the manager has not yet assigned it a new task. |
|
Failed |
The worker encountered a problem. |
|
Fixed, Restart |
You fixed the problem, and the worker will retry the task that failed. |
|
Restarted |
The worker is retrying a task or has successfully restarted a task. Note that the status does not change to Running. |
|
Running |
The worker is running a task. |
|
Wait |
The worker is idle. |
If the worker shows a status of Failed, refer to Section 11.5.3, "Determining Why a Worker Failed" for assistance in fixing the problem so Oracle Fusion Applications Patch Manager can complete its processing.
When a worker fails its task, you do not have to wait until the other workers and the manager stop. You can review the worker log files to determine what caused the failure. Workers do not proceed to run tasks in the subsequent phase until all tasks in the current phase complete successfully. You must take action to resolve the failure so the workers can continue to run tasks in the next phase. If the task was deferred after the worker failed, there may be no action required from you.
The first time a task fails, the manager defers the task and assigns the worker a new task. If the deferred task fails a second time, the manager defers it a second time only if the run time of the task is less than 10 minutes. If the deferred task failed a third time, or if its run time is greater than 10 minutes, the task stays at a failed status and the worker waits for manual intervention. Action by you is then required because the worker stops any further processing. An example of when this scenario can occur is during seed data upload. The seed data upload may fail due to records being locked by another process. If the lock is released before the second or third attempt of the upload, the upload succeeds.
Follow these steps to find out why a worker failed:
In the Log Summary, located in FA_ORACLE_HOME/admin/FUSION/log/logsummary_fapmgr_command_timestamp.html, review the AutoPatch Apply log file to find the worker that failed. For more information, see Section 11.2.1, "Log Summary".
Open and review the log file for the failed worker to determine the cause of the worker failure.
Each worker logs the status of tasks assigned to it in adpatch_apply_timestamp_workernumber.log. The worker log file contains information that describes exactly what task it was running and what error occurred that resulted in a failure. Review the worker log file for the failed worker to determine the source of the error. For more information, see Section 11.1.1, "Log Files for Single Oracle Fusion Applications Patch Manager Sessions".
Determine how to fix the problem that caused the failure.
Resolve the error using the information provided in the log files. If needed, search for the resolution at the My Oracle Support site or file a service request with Oracle Support Services if you do not understand how to resolve the issue.
After you resolve the problem that caused the failure, restart the failed worker.
Select Tell worker to restart a failed job from the AD Controller main menu. For more information, see Section 11.5.4, "Restarting a Failed Worker".
Verify the worker status.
Select Show worker status from the AD Controller main menu. The Status column for the worker that failed should now display Restarted or Fixed, Restart.
If a worker job failed or if you terminated a hanging worker process, you must manually restart the worker. Some worker processes spawn other processes called child processes. If you terminate a child process that is hung, the worker that spawned the process shows Failed as the status. After you fix the problem, you must restart the failed worker. After the worker restarts, the associated child processes start as well.
Follow these steps to restart a failed worker:
Start AD Controller. For more information, see Section 11.5.1, "Starting AD Controller".
Select Show worker status from the AD Controller main menu.
Take the appropriate action for each worker status:
If the worker status is Failed and its job has failed, select Tell worker to restart a failed job. When prompted, enter the number of the worker that failed.
If the worker status is Running or Restarted, but the job is hung, follow the steps in Section 11.5.5, "Terminating a Hung Worker Process".
When running AD utilities, there may be situations when a worker process appears to hang, or stop processing. If this occurs, you may need to terminate the process manually. After you do that, you must also restart that process manually.
Caution:
A process that appears to be hung could be a long-running task. Be careful when terminating processes.
To terminate a process, you start AD Controller, obtain the worker's process ID from your operating system, and then stop any hanging processes. After you make the necessary changes, you can restart the worker.
Take the following steps to terminate a worker process that is hung.
Start AD Controller. For more information, see Section 11.5.1, "Starting AD Controller".
Select Show worker status from the AD Controller main menu.
Open and review the log file for the failed worker to determine the cause of the worker failure.
Each worker logs the status of tasks assigned to it in the adpatch_workernumber.log. The worker log file contains information that describes exactly what tasks it runs and what errors occurred that resulted in a failure. Review the worker log file for the failed worker to determine the file being processed and review the worker log file to see if it is progressing. You can also verify whether the process is consuming CPU resources from your operating system.
For more information, see Section 11.1.1, "Log Files for Single Oracle Fusion Applications Patch Manager Sessions".
Confirm that the operating system process associated with the worker is not running. If the task is identified as "hanging", determine the worker's process ID by looking for processes being run by the worker.
(UNIX) ps -a| grep workernumber
(Windows) Start the Task Manager (Ctrl-Alt-Delete) to view processes.
Determine what processes the worker has started, if any. If there are child processes, get their process IDs.
Stop the hung process, using the command that is appropriate for your operating system.
If you terminate a SQL*Plus session spawned by a worker, you just need to kill the SQL*Plus session. The worker immediately detects the FAILED state. For other processes, follow Steps 7 through 11.
In AD Controller, select Tell manager that a worker failed its job and enter the worker number of the hung workers. This should cause the worker to fail.
Select Tell worker to quit. When prompted, enter the worker number of the hung worker.
Select Tell manager that a worker acknowledges quit. When prompted, enter the number of the hung worker.
Fix the issue that caused the worker to hang. Search the My Oracle Support site or file a service request with Oracle Support Services if you do not know how to fix the issue.
Select Restart a worker on the current machine. When prompted, enter the number of the failed worker. The worker will restart its assigned tasks and spawn the associated child processes.
Note:
Do not select Restart a worker on the current machine if the worker process is running. Doing so creates duplicate worker processes with the same worker ID and will cause incorrect results.
There may be situations when you must shut down an AD utility while it is running. For example, you may want to shut down the database while Oracle Fusion Applications is running AutoPatch or during an AD Administration session. You should perform this shutdown in an orderly fashion so that it does not affect your data. The best way to do this is to shut down the workers manually.
Start AD Controller.
Select Tell worker to quit, and enter all for the worker number. The worker completes its current task and then quits.
Verify that no worker processes are running, using a command that is appropriate for your operating system.
When all the workers have shut down, the manager and the AD utility quits.
No workers are running tasks. They are either failed or waiting. A restarted worker resumes the failed task immediately if the worker process is running. Workers change to a Waiting status if they cannot run any tasks because of dependencies on a failed task, or because there are no tasks left in the phase. When no workers are able to run, the manager becomes idle.
Complete the following steps for each worker:
Start AD Controller. For more information, see Section 11.5.1, "Starting AD Controller".
Determine the cause of the error.
Select Show worker status. Review the worker log file for the failed worker to determine the cause of the error.
Resolve the error using the information provided in the log files.
Search for the resolution in the My Oracle Support site or file a service request with Oracle Support Services if you do not understand how to resolve the issue.
Restart the failed worker.
Select Tell worker to restart a failed job on the AD Controller main menu. The worker process restarts, causing the AD utility and the manager to become active again.
This error occurs during patching, "Unable to start universal connection pool". Connections to the database cannot be established due to timeout limits.
Consider tuning the listener parameters INBOUND_CONNECT_TIMEOUT_listenername and SQLNET.INBOUND_CONNECT_TIMEOUT for the server. For more information, see "SQLNET.EXPIRE_TIME Parameter" and "INBOUND_CONNECT_TIMEOUT Parameter" in the Oracle Fusion Applications Performance and Tuning Guide.
When you patch database artifacts, your system should be in an idle state. If this is not the case, the patching session may hang due to locks. Examples of locks that can cause the patching session to hang are PL/SQL packages that are accessed by Oracle Enterprise Scheduler Service Server, a locked table, or a locked table row. After a specific wait time, such as 30 minutes, Oracle Fusion Applications Patch Manager performs a check to determine whether the patching session is blocked by another session. If a blocking session is found, a message describing the block is sent to the log file, as shown in the following example:
"[2011-03-14T02:12:18.112-07:00] [apps] [NOTIFICATION] [] [AutoPatch] Worker 4 is blocked by session 3868 ... Please fix the session to avoid indefinite waiting
The worker that is blocked remains in a Running status. To resolve the issue and release the lock, stop the blocking session using the command that is appropriate for your operating system. After the blocking session is no longer running, the hung worker proceeds to complete its task. You can use the following SQL*Plus query to identify the sessions that are blocking patching sessions:
SELECT blocking_session,
sid "Blocked Session",
module "Blocked Module",
seconds_in_wait
FROM gv$session
WHERE status = 'ACTIVE'
AND module like 'PATCHING_SESSION:%'
AND blocking_session_status = 'VALID'
AND user = '<FUSION schema>';
Note:
To minimize blocked sessions, ensure that you follow the steps in Step 7, "Prepare the System" before you apply a patch.
If you recently applied a one-off patch to a database artifact, you may encounter the following error the next time you apply a standard patch:
Error occurred during patch conflicts checking. This may be due to infrastructure failure OR patch conflicts You may check FA_ORACLE_HOME/admin/FUSION/out/patch_number_conflict_report.xml file for any patch conflicts You should check the file FA_ORACLE_HOME/admin/FUSION/log/adpatch.log for errors
A one-off database patch sets a lock on the artifact that the one-off patch updates. After you apply a one-off patch, you must subsequently apply the related standard patch that contains the same updates as the one-off to release the lock. To find out which one-off patch is locking the artifact, review the conflict_report.xml file. In the following example of this file, the one-off patch that created the lock was 909090.
<Patch_Conflict_Report>
<instance>
<appl_sys_name>FUSION</appl_sys_name>
<appl_top>/server01/fusionapps</appl_top>
<patch_number>909090</patch_number>
</instance>
<patch_type>one-off</patch_type>
<conflict_details>
<conflicts>
<prod>HCM</prod>
<subdir>hcm/components/hcmPayroll/legconfig/setup/dbSchema/database/fusionDB/FUSION</subdir>
<filename>PAY_INSTALLED_LEGISLATIONS.table</filename>
<bug>909090</bug>
</conflicts>
To resolve this issue and remove the lock, obtain and apply the standard patch that delivers the same fix as the one-off patch.
If you need to connect to the Oracle Fusion Applications database to troubleshoot database related issues, by running SQL*Plus, for example, you need to set up the appropriate environment variables. For setting any environment variable, run the adsetenv script to generate the APPSORA.env file, which when sourced, sets all environment variables.
(UNIX) sh adsetenv.sh source APPSORA.env echo $TWO_TASK (Windows, TWO_TASK is known as LOCAL) adsetenv.cmd APPSORA.cmd echo %LOCAL%
When multiple seed data files are uploaded for the same flexfield but for different flexfield contexts, the upload tasks can fail due to locking issues. The failed tasks appear in the log file as the following error:
Loading failed with a JboException: JBO-25014: Another user has changed the row with primary keyoracle.jbo.Key ...
AutoPatch defers any failed tasks to the end of the phase and reattempts the failed tasks only after attempting all tasks in the phase at least once. Usually the flexfield seed data tasks that failed due to the locking issue succeed on subsequent attempts. You can ignore these errors if the flexfield seed data task succeeds on the retry. If the task remains in a failed state, you must use the AD Controller utility to retry the failed task.
For more information, see Section 11.5.4, "Restarting a Failed Worker".
|
 Copyright © 2011, 2013, Oracle and/or its affiliates. All rights reserved. Legal Notices |
|