The topics related to Monitor and Troubleshoot Patches are as follows:
Oracle Fusion Applications Patch Manager (Patch Manager) creates log files for the actions it performs. These logging capabilities track the progress of actions and assist in diagnosing issues. When using Patch Manager, it is possible to specify the level of logging detail.
The following logging detail levels are available:
ERROR:1 (SEVERE) For an error that results in a failure.
WARNING:1 (WARNING) For an error that does not result in failure, but should be reviewed.
NOTIFICATION:1 (INFO) For high-level information about the progress of the process, no action necessary.
NOTIFICATION:16 (CONFIG) For more detailed information about the progress of the process, no action necessary.
TRACE:1 (FINE) For generating the first level of trace messages, used for diagnosing issues.
TRACE:16 (FINER) For generating the second level of trace messages, used for diagnosing issues.
TRACE:32 (FINEST) For generating the highest level of trace messages, used for diagnosing issues.
When selecting a more detailed level of logging, the log files also include the lower level of details. For example, in case of choose to see INFO messages in log file, WARNING and SEVERE messages also appear in the log files.
To examine the activities performed during patching sessions, review the associated log files. Patch Manager consolidates log files for each patching session under the directory, APPLICATIONS_CONFIG
APPLICATIONS_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR
.. This directory contains the top-level log file, logsummary
_fapmgr_command_timestamp.html
, along with related log files for each task performed during a fapmgr
session. During a session, it is possible to view the Log Summary HTML file from a browser, which provides links to individual log files. To view the progress of the current patching session, refresh the Log Summary HTML file periodically. If a task fails, access the links to the associated log files to assist in diagnosing the failure. For more information, see the Log Summary section.
When a patching session completes, its log files are archived in APPLICATIONS_CONFIG
/lcm/logs/<Fusion Applications Release Version>/FAPMGR/logarchive
/Patch Number/fapmgr_command/session ID/timestamp
. The session ID is unique and the time stamp is the start time for the session. Note that whenever Patch Manager runs a command where there is no patch number, such as bootstrap, abort,
and report
, the archive logs are named APPLICATION_CONFIG
/lcm/logs/<Fusion Applications Release Version>/FAPMGR/logarchive/fapmgr_command
/timestamp.
Log files for OPatch actions are initially written to the FA_ORACLE_HOME
/cfgtools/opatch/
patch number_timestamp directory.
The following table contains a list of log files created by Patch Manager during patching activities:
Table 8-1 Log Files for Single Oracle Fusion Applications Patch Manager Patching Activities
Log file or directory name under APPLICATIONS_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR | Log file generated with: |
---|---|
FAPatchManager_apply_timestamp.log |
Oracle Fusion Applications Patch Manager apply session |
FAPatchManager_abort_timestamp.log |
Patch Manager abort session |
FAPatchManager_bootstrap_timestamp.log |
Patch Manager bootstrap session |
FAPatchManager_report_reportname_timestamp.log |
Patch Manager report session |
FAPatchManager_validate_timestamp.log |
Patch Manager validate session |
adpatch_apply_timestamp.log |
Oracle Fusion Applications AutoPatch (AutoPatch) apply session |
adpatch_abort_timestamp.log |
AutoPatch abort session |
adpatch_apply_timestamp_workernumber.log |
AutoPatch worker log file |
adpatch_validate_timestamp.log |
AutoPatch |
adpatch_apply_timestamp_timingreport.lst |
AutoPatch timing report |
logsummary_fapmgr_command_timestamp.html For reports: logsummary_report_reportname_timestamp.html |
The consolidation of the log files generated by Patch Manager in HTML format for viewing and accessing links to other log files from a browser |
patch number_fapmgr_command_session id_timestamp.marker |
Marker file used while moving log files to a backup directory |
ConfigContext_timestamp.log |
Patch Manager in online mode |
ExecutionContext_timestamp.log |
Patch Manager in online mode |
FAPMgrDiagnosticsSummaryfapmgr_command_timestamp.html |
The consolidation of the Patch Manager session in HTML format, known as the Diagnostics report |
FAPMgrDiagnosticsSummary.html |
This file is updated every five minutes during patch application. |
diaghtmllogs_mode_timestamp |
The Diagnostics report for Patch Manager in |
diaghtmllogs_mode_timestamp/FAPatchManager_apply_20120627022503.html |
Patch Manager apply session |
diaghtmllogs_mode_timestamp/adpatch_apply_timestamp.html |
AutoPatch apply session |
diaghtmllogs_mode_timestamp/adpatch_validate_timestamp.html |
AutoPatch validate session |
diaghtmllogs_mode_timestamp/adpatch_apply_timestamp_workernumber.html |
AutoPatch worker log file |
Utility Code_Mode_Artifact Type_PatchAction_Action_Task Id_time_stamp.log |
Task level log files which are created for each artifact type and its actions, for example:
|
e.g.,
|
OPatch log while running and after completion.
|
fastartstop_timestamp.log <Domain name>_timestamp.log fastartstop_results.xml |
FAStartStop log: while running the logs are located in the following:
APPLICATIONS_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR/fastartstop After completion the logs are located in the following: APPLICATIONS_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR/logarchive/MAPPLY/<TimeStamp>/<Group number>/<APPLY | VALIDATE>/<Patching Session Number>/<Group time stamp>/fastartstop/fastarstop_<start | stop>_<time stamp> |
The multi-apply feature of Patch Manager applies multiple patches after splitting them into groups within a My Oracle Support patch plan. Each group contains at least one patch and it is run as an individual session internally. Log files and diagnostic reports are generated for each individual session, in addition to a primary log file and diagnostics report for the overall multi-apply session.
Examine the activities performed during multi-apply patching sessions by reviewing the associated log files. Patch Manager consolidates log files for each patching session under the directory, APPLICATIONS_CONFIG
/lcm/logs/<Fusion Applications Release Version>/FAPMGR
. This directory contains the top-level log file, logsummary
_fapmgr_command_timestamp
.html
, along with related log files for each task performed during a multi-apply session. During a session view this log summary HTML file from a browser, which provides links to individual log files. Periodically refresh the log summary HTML file to view the progress of the current patching session. If a task fails, access the links to the associated log files to assist in diagnosing the failure. See the Log Summary section to get more information how the log summary is created and how the report is created.
Multi-apply sessions create a primary diagnostics report that provides information about the multi-apply patching session. It also includes links to corresponding diagnostics reports and archived log files for individual patches that were applied as a group.
The following table contains a list of log files created by Patch Manager during multi-apply patching activities:
Table 8-2 Log Files For Multi-apply Patch Sessions
Log file name or directory under FA_ORACLE_HOME/admin/FUSION/log | Log file description |
---|---|
FAPMgr_Multiapply_apply_timestamp.log |
Primary log file from a multi-apply patching session |
FAPMgr_Multiapply_DiagnosticsSummary_timestamp.html |
Primary Diagnostics report from a multi-apply patching session in HTML format |
FAPMgr_Multiapply_DiagnosticsSummary_timestamp.xml |
Primary Diagnostics report from a multi-apply patching session in XML format |
diaghtmllogs_mode_timestamp |
The Diagnostics report from a multi-apply patching session in |
FAPMgrDiagnosticsSummary_apply_timestamp.html |
Diagnostics report from a multi-apply patching session in HTML format |
ADPatchDiagnosticsSummary_apply_patchnumber_timestamp.html |
Diagnostics report for one patch in a multi-apply patching session in HTML format |
logsummary_apply_timestamp.html |
The consolidation of the log files generated by multi-apply patching session in HTML format for viewing and accessing links to other log files from a browser |
Utility Code_Mode_Artifact Type_PatchAction_Action_Task Id_time_stamp.log |
Task level log files which are created for each artifact type and its actions, for example:
|
At the beginning of a multi-apply patching session, the log files from the previous session move to FA_ORACLE_HOME
/admin/FUSION/logarchive/MAPPLY/
timestamp
. For each patch that is applied as part of a group, a LogFilesLocationPointer.log
is created at the location where the log files will be archived, as if the patch was applied as a standalone patch. The content of the pointer files provides the absolute location of the corresponding group apply logs and provides the ability to navigate directly to the log files for a specific patch, even though it is applied using multi-apply. For example, if patches 19191630 and 20193112 are grouped into GROUP1
and then applied, log files are archived in the following directory structure:
logarchive | |-- MAPPLY | |-- 20120610235002 | | |-- FAPMgr_Multiapply_apply_20120610235002.log | | |-- FAPMgr_Multiapply_DiagnosticsSummary_20120610235002.html | | |-- FAPMgr_Multiapply_DiagnosticsSummary_20120610235002.xml | | |-- GROUP1 | | | |-- APPLY | | | | |-- 123 | | | | | |-- 20120610235302 | | | | | | |-- FAPatchManager_apply_20120610235302.log | | | |-- VALIDATE | | | | |-- 122 | | | | | |-- 20120610235102 | | | | | | |-- FAPatchManager_validate_20120610235102.log |-- 19191630 | |-- APPLY | | |-- 123 | | | |-- 20120610235302 | | | | |-- LogFilesLocationPointer.log | |-- VALIDATE | | |-- 122 | | | |-- 20120610235102 | | | | |-- LogFilesLocationPointer.log |-- 20193112 | |-- APPLY | | |-- 123 | | | |-- 20120610235302 | | | | |-- LogFilesLocationPointer.log | |-- VALIDATE | | |-- 122 | | | |-- 20120610235102 | | | | |-- LogFilesLocationPointer.log
Log files are useful files that contain informational and error messages generated during patching. Log files are generated by Patch Manager, which coordinates patching activities by assigning tasks; OPatch, which runs the tasks for updating middleware artifacts; and AutoPatch, which runs the database tasks. If a task fails at any point or in any level, exact information about the failure will be included in the log file. Furthermore, progress of the patching session, including the details of a failed task, can be viewed from a browser by reviewing the Log Summary or the Diagnostic report.
For more information about how to examine the activities performed during the patching sessions, see the Log Files for Single Patch Manager Sessions, the Log Summary section to understand the log summary creation,, and the Diagnostics Report section to see the report during the patching session.
The Log Summary is created automatically when the Patch Manager is started. The Log Summary is continuously updated as the session progresses. This report exists in the FA_ORACLE_HOME
/admin/FUSION/log
directory and is named logsummary
_fapmgr_command_timestamp.
html
. It contains links to all of the log files generated during the session. To view the report, open the report from a browser and periodically refresh the page to see updated links to log files as they are created. Open those links and monitor the progress of the session by refreshing the browser.
After each patching session ends, the Diagnostics report is automatically generated and results may be viewed in a browser. Use this report during a patching session that is currently running, by generating the report from the command line. The Diagnostics report is located in the FA_ORACLE_HOME
/admin/FUSION/log
directory and is named FAPMgrDiagnosticsSummary
fapmgr_command_timestamp
.html
.
For the fapmgr apply
and validate
commands, the Diagnostics report contains links to the line number in the logs file for each task. The links contained in the Diagnostics report go to the specific line in the corresponding HTML log files. The HTML log files exist in the directory diaghtmllogs_
mode_timestamp
directory.
During the upgrade flow, while the Loading Database Components Config Assistant or the Installing Downloaded Fusion Applications Upgrade Patches Config Assistant runs, the diagnostic report is not generated by default. It can be generated manually if required.
This section contains the following general troubleshooting scenarios for patching:
Start a New Patching Session After the Previous Session Failed
Jobs Are Run After Maintenance Wait Period Using Hot Patching
Revert to a Previous Flexfield Definition After it is Updated by a Patch
For troubleshooting information that is specific to patching security artifacts such as the jazn-data.xml
file, data security files, and data role templates, see the Patch Security Artifacts section.
For troubleshooting information that is specific to patching SOA composites, see the Troubleshoot Patching Sessions for SOA Composites section.
For troubleshooting information that is specific to patching database content, see the Troubleshoot Patching Sessions for Database Content section.
After the failure of a patching session, the following scenarios are possible:
To abandon the previous session and start a new session, see the Abandon a Failed Patching Session section.
Despite failure, some tasks may display as running. For more information, see the Recovery from an Interrupted Patching Session section.
There can be only one patching session active at one time for Oracle Fusion Applications, Oracle Fusion Functional Setup Manager, and Oracle Fusion Middleware Extensions for Applications (Applications Core). If there is a failed Applications Core or Functional Setup Manager patching session that must be cleaned up, see the Abandon a Failed Patching Session section.
If the patch session failed and there is no need to restart it, abandon the session by using the abort
command. Remember that only one patching session can be running at a time and always make sure that processes associated with the previous patching session does not exist.
Mandatory: If the patching session is aborted, this session cannot be restarted and any pending patching actions, such as deployment actions, must be performed manually.
To abandon a previously failed session, run the fapmgr
abort
command . The abort
command cleans up any intermediate states tracked by fapmgr
and moves the log files for the abandoned session to an archive log directory so that a new patching session could start. Note that a session that is actively running cannot be abandoned.
Use the following syntax for the fapmgr
abort
command:
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh abort [-force] [-logfile log file name] [-loglevel level
The following table describes the options available for the abort
command:
Table 8-3 abort Command Options
Option | Description | Mandatory |
---|---|---|
logfile |
Name of the log file |
No, default value is FAPatchManager_abort_timestamp.log |
loglevel |
Reporting level for messages see the Oracle Fusion Applications Patch Manager Logging section |
No, default value is INFO |
force |
Always forces the patching session to abort, even when the patching session has not failed see the Use Abort -force section |
No |
help |
Displays help |
No |
If the fapmgr
abort
command errors with a message "Another APPLY session is already running", use the fapmgr forcefail
command first. Also confirm that the FND_INSTALL_PROCESSES
table does not exist. For more information, see the Recovery from an Interrupted Patching Session section.
abort -force
The -force
option allows marking a current patching session as ABORTED
to proceed to a new patching session after cleaning up the current patching session.
MANDATORY: This feature is intended for hot patching sessions where APPLY
tasks are not automatically cleaned up and must be used only with the assistance of Oracle Support.
The abort command impacts normal patching and hot patching in the following ways:
Normal Patching
fapmgr abort
: If the abort fails, the patching session is set to Abort_failed
, then attempt to abort the session again.
fapmgr abort
-force
: Whether the abort fails or succeeds, the patching session is set to ABORTED
and displays any failed tasks that must be performed manually.
Hot Patching
fapmgr abort
: The patching session is set to Abort_failed
if one or more APPLY
tasks attempt to run because APPLY
tasks are not reversible in hot patching.
fapmgr abort
-force
: The patching session is set to ABORTED
and displays all tasks as "Failed to Abort". Contact Oracle Support to request assistance in manually performing these tasks.
If a patching session was interrupted by a system failure when Patch Manager and AutoPatch were running and the patching-related database tables still show the patching session as running (only one patching session can be active at a time), but no patching-related processes are actually running, use the fapmgr
forcefail
command to update the patching tables by performing the following steps:
If initiating a patching session from a terminal server, such as a laptop, the connection may be lost during extended periods of time when no messages are sent to the terminal. The terminal server may interpret this as inactivity and then end the session. For example, no messages are sent to the client when Patch Manager is stopping and starting servers, waiting for a failed task to be fixed, or is hung on a database task. To avoid this situation, ensure that the terminal server is configured appropriately to handle long durations of inactivity.
While applying a patch using hot patching, the following type of error is reported:
ESS Jobs are running on the domains (HCMDomain) even after maintenance wait period.
If this error is reported, consider the following alternatives:
Wait for ESS Jobs execution finish.
forceterminateactivetasks
option, as shown in the following example:
fapmgr.sh apply -patchtop patch_top -hotpatch -online -stoponerror -maintenancewaitperiod 1 -maintenanceendtime date_time -forceterminateactivetasks
Run the following command to validate if the patch is structured to be applied as a hot patch :
sh fapmgr.sh validate -patchtop /patch_top/18060344 -hotpatch -online
Validation FAILED. You cannot apply the patch. Check the log file for unexpected infrastructure failures.It means that the patch is not structured to be applied as a hot patch. If this error occurs, apply this patch without using the
-hotpatch
option.If a patch is being applied and it contains BI Publisher artifacts, the BI Presentation servers should not be running. If BI Presentation servers are running during the deployment of BI Publisher artifacts, the following error occurs:
java.lang.RuntimeException: Webcat patch file creation failed!
To resolve this issue, shut down the BI Presentation servers to release locks on the Oracle BI Presentation Catalog.
If during patch validation, the following error is received:
weblogic.management.mbeanservers.edit.EditTimedOutException
Manually release the edit session. This situation occurs when a domain is already in "edit" mode during patching, for example, when the server crashes when Patch Manager tries to stop and restart it.
To manually release the edit session, use the following procedure:
If flexfield changes are not ready to be implemented after applying a patch containing flexfield changes, then revert to a previous version of that flexfield definition. The Flex Modeler creates an MDS label, FlexPatchingWatermark
date+time
, before it initiates the flexfield deployment, which establishes the watermark for what was in MDS before the patch was applied. The name of the label is included along with the Flex Deployment report in the patching log file as a reference.
To use a previous version of a flexfield definition, use the WLST command promoteMetadataLabel
.
To delete all previous MDS labels for a flexfield, first confirm the changes delivered by a patch can be used (keep old MDS labels impacts performance), then use the WLST command deleteFlexPatchingLabels
.
If a patch contains BI artifacts the BI OPMN control process, which is similar to a node manager, then it has to be up for online mode validation to succeed. Online validation reports the following error if the BI OPMN control process is not up:
The deployment of BI Publisher artifacts will not be attempted because the BI Presentation server is neither fully started nor down. One likely cause is that the BI OPMN control process is not running. Make sure that the BI OPMN control process is up and the BI Presentation server is started successfully before applying this patch. If this server is not fully started, you must stop the BI Presentation server, manually deploy the BI Publisher artifacts, and then re-start the BI Presentation server
To resolve the issue, ensure that the BI OPMN control process is up and running.
If while applying a patch, Patch Manager fails with the error "Failed while trying to update task status" or "Error while updating session status", solve the issue by performing the following steps:
The opatch -lsinventory -detail
command provides a report that lists all patches, artifacts and artifact versions that were modified within each patch applied to a given Oracle home. This report lists only artifacts that were modified. If an artifact does not appear on this report, then the artifact remains at its base version. Run this report when Oracle Support Services are used and an artifact version needs to be provided.
To generate the report, use the following command syntax:
opatch/opatch lsinventory -detail -oh FA_ORACLE_HOME -invPtrLoc \
FA_ORACLE_HOME/oraInst.loc -jre JAVA_HOME
The following example depicts the section of the report that displays patches applied. To use the report, find the latest entry for the specific artifact and note the version reported.
Interim patches (11) : Patch 11801 : applied on Wed Feb 02 17:57:53 PST 2011 Created on 18 Jan 2011, 16:09:54 hrs PST8PDT Bugs fixed: 11801 Patch is translatable. Files Touched: AdfPjfIntMspUi.jar, version "23.0" --> ORACLE_HOME/prj/deploy/EARProjectsFinancials.ear/EARProjectsFinancials.war/WEB-INF/lib/AdfPjfIntMspUi.jar Patch Location in Inventory: /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/inventory/oneoffs/11801 Patch Location in Storage area: /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/.patch_storage/11801_Jan_18_2011_16_09_54 Patch 12801 : applied on Tue Feb 01 21:30:17 PST 2011 Created on 28 Jan 2011, 14:26:56 hrs PST8PDT Bugs fixed: 12801 Patch is translatable. Files Touched: EFFmetadata.mar, version "35.0" --> ORACLE_HOME/hcm/deploy/EarHcmCoreSetup.ear/EFFmetadata.mar system-jazn-data.xml, version "35.0" --> ORACLE_HOME/hcm/security/policies/system-jazn-data.xml Patch Location in Inventory: /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/inventory/oneoffs/12801 Patch Location in Storage area: /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/.patch_storage/12801_Jan_28_2011_14_26_56
Always test the application of a patch and the functionality related to the patch on a test system before applying it to production system to ensure it will be successful. There is no automated method of backing out patches.
Oracle strongly recommends to work with Oracle Support Services to back out a patch.
Problem
POD runs Out of Memory and OS kernal ends FApMgr.
Solution
Take process memory map to know which process has taken more memory.
Take a snapshot of the used and free memory.
Contact the team owner of the process to get the workaround to release the memory.
When the memory has at least of 5 GB of free memory then, contact the FAPMgr support team to perform below steps:
Run Fapmgr focefail
.
Note:
DO NOT run ABORT.Clean all the child processes if any in all the hosts.
Restart the patching session.
This section describes the troubleshooting process for errors that can occur when patching Service-Oriented Architecture (SOA) composites. These processes assume that the validation and application of the patch are done in online mode. SOA patching errors can be divided into the following categories:
Error occurs during validation
Patch Manager can detect and report validation errors before changes have occurred to the file system. If the validation errors are not resolve before applying the patch, the patch fails and the SOA composites has to be manually deployed after resolve the validation errors.
Error occurs during the patch apply phase
These errors may require contacting Oracle Support Services to restore the system back to a known working state and can be further divided into these categories:
The SOA composite failed to deploy and Patch Manager recovered from the failure.
The SOA composite was not deployed successfully and the recovery failed. Therefore, the composite may be partially deployed.
The system is in an unknown state, as the result of a timeout occurring during deployment. Patch Manager cannot determine if the SOA composite is deployed, not deployed, or partially deployed.
When SOA composite failures occur, review the error messages in the Diagnostics report and related log files and follow the applicable steps in one or more of the following categories:
SOA composite validation and deployment can fail for multiple reasons. Review the following steps for basic troubleshooting:
After validate or apply a patch that contains SOA composites, review the Diagnostics report for errors. The report output is in HTML format for viewing from a browser and is located here:
FA_ORACLE_HOME/admin/FUSION/log/FAPMgrDiagnosticsSummarydate:session.html
The Module Task Details section of the report displays the following information to assist in your troubleshooting:
Mode: Middleware, in this case.
Phase: Validation or Patch Application, in this case.
Product Family: The short name of the product family.
Task.
The following information displays for SOA composites:
Name of composite
Domain name
Path to composite JAR in FA_ORACLE_HOME
Revision of composite
Status: Possible values of Success, Failed, or Skipped.
Duration: Total time the task ran.
Start Time: Time and date the task started.
End Time: Time and date the task ended.
Warning/Error Message: The error message displays as a java.lang.RuntimeException
. The message often includes a suggestion for resolving the failure.
Log file: The path and file name of the high level log file, FAPatchManager
_fapmgr_command
_timestamp
.log, associated with the task. From the Module Execution Summary section of the Diagnostics report, review log files by accessing the link to the Log Summary. For more information, see the Log Summary section.
Line Numbers: The line numbers in the log file associated with the task.
SOA log files are located in this directory: FA_ORACLE_HOME
/admin/FUSION/log/fapatch/fapatch
_release_number
/soalogs
. If merge operations are performed on a SOA composite, due to runtime customizations, such as design time and run-time (DT@RT) changes or property changes, a merge log file is generated. There is one merge log file per domain and the name of merge log files follows this naming convention: fapatch
_domain_nametimestamp
.merge.log
Restart the SOA servers and for each failure, follow Steps 4 through 9.
Determine if there is a cause for the error that must be resolved, in addition to restarting the server, by referring to the Diagnostics report, Oracle Fusion Applications log files and SOA log files. Examples of other causes include database errors, coherence configuration errors, and out of memory issues.
Determine if a manually restore the system back is need it to its state before the application of the patch was attempted. The following scenarios do not require manual restoration of the system:
Errors occurred during the validation phase.
Errors occurred during the patch application phase but the recovery was successful, so the system was recovered to its original state. The Diagnostics report displays this message in this case:
Deployment of SOA composite[artifact_name and path]failed, but the system has recovered successfully.
Suggestion: You must manually deploy the composite using the WLST command.
If the system needs to be restored, follow the steps in Step 6.
If a failed deployment leaves a composite in an inconsistent state, restore the system to its original state. In case of prefer to use Oracle Fusion Applications Control to restore the system, see Step 7. Perform the following WLST commands to restore the system:
Before the application of the patch from the Diagnostics report, please find out the last active revision of the composite, it is indicated by this message: The last active version of the composite before patch application began was
[version
].
In the following examples, 1.0 is the previous revision and 2.0 is the patched revision.
Undeploy the patched revision of the composite if it exists in the system.
sca_undeployComposite('http://server01:8001', 'POProcessing', '2.0')
Mark the previously active revision of the composite as active and as a default revision.
To activate the old revision, use the following command:
sca_activateCompositeMb('POProcessing', '1.0')
To assign the default composite, use the following command:
sca_assignDefaultCompositeMb('POProcessing', '1.0')
Restore the system to its original state using Enterprise Manager Cloud Control with Oracle, follow up the steps below:
Find the active revision of the composite before the application of the patch from the Diagnostics report, as indicated by this message: The last active version of the composite before patch application began was
[version
].
In the following steps, 1.0 is the previous revision and 2.0 is the patched revision.
Undeploy the patched revision of the composite if it exists in the system.
Mark the previously active revision of the composite as active and as a default revision.
Manually deploy the SOA composites included in the patch by following the steps :
sca_patchComposite('SOA-Infra URL', user, password, '/FA_ORACLE_HOME/crm/deploy//sca_FinAprev2.0.jar', mergeLogFile='/tmp/mergelog
If you are unable to resolve the failure, file a service request with Oracle Support Services and provide the logs and information as described in the Get Started with Troubleshooting and Logging Basics for Oracle SOA Suite section in the Oracle Fusion Applications Administrator's Guide.
This section describes common problems and solutions for SOA composite validation failures.
MANDATORY :Errors that occur during the validation phase must be resolved before applying the patch. In case of these errors occurred during patch application, manually deploy the SOA composites after resolve the validation errors.
Patch Manager captures the OPatch validation log files, which can be found by referencing the Diagnostics report or the Log Summary. The errors in the log files provide information about the cause of the failure and are often followed by recommended actions.
This section contains troubleshooting information about the following failures:
An error that is related to a JDeveloper customization occurs when a SOA composite was customized but the customization was not saved. Before applying a patch that includes the next revision of the composite, save the customization. Follow the steps mentioned in the Preserve SOA Composite JDeveloper Customizations Before Apply a Patch section to resolve this error.
If the SOA server is down or not available for patching, patch validation succeeds, but in case of a warning message stating that the deployment of the composite will not be performed because the SOA infrastructure is down.
Use Oracle Enterprise Manager Fusion Applications Control (Fusion Applications Control) to check the state of the SOA server. For example, if an "Initializing SOA" warning message displays on the home page, Oracle recommends wait until the SOA server is completely up and running, with all composites initialized.
For more information, see the SOA Server Does Not Start section in the Oracle Fusion Applications Administrator's Guide.
If the Administration Server is down or not available for patching, patch validation fails. Use Fusion Applications Control to check the state of the Administration Server. For more information, see the Access Fusion Applications Home Pages in Cloud Control section in the Administrator's Guide.
If the SOA-infra server is down or not available for patching, patch validation fails, use Fusion Applications Control to check the state of the SOA-infra server. Confirm that all dependent services are running and that all composites deployed into the SOA-infra server are present. It may take some time after SOA-infra is up for all services to initialize. In case of use a cluster, perform this check for all SOA-infra servers in the cluster.
In case of receive an error stating that a composite with a specific revision is already deployed, the SOA composite in the patch was previously deployed by a patch or manually by a user.
Resolve this error either by not applying the current patch or by undeploying the composite before applying the patch. Note if the composite is undeployed, all customizations may have made in the composite will be lost. Use the following command to undeploy a composite:
sca_undeployComposite(serverURL, compositeName, revision)
For more information about using the sca_undeployComposite
command, see "Undeploy SOA Composites Using WSLT Command" in the Oracle Fusion Applications Upgrade Guide.
This section describes common problems and solutions for SOA composite deployment failures during patching.
MANDATORY: Errors that occur during the deployment phase must be resolved as soon as possible because the system has been modified. Do not try to roll back or reapply patches that have errors during deployment. After resolve the cause of the error, you must deploy the composite manually.
This section contains the following topics:
In case of receive an error in making the new composite the default, the option available is manually assign the new composite as the default.
Use the following WLST command to assign the new composite as the default:
sca_assignDefaultComposite('host', 'soaport', 'user', 'password', 'composite_name', 'composite_revision')
If the WLST command is successful, verify if the new composite is active. .
MANDATORY: If the new composite is not active, manually deploy the composite that failed, by following the steps in the Manually Deploying SOA Composites section.
For more information, see the ”sca_assignDefaultComposite" in the WLST Command Reference for WebLogic Server section.
In case of receive an error in retiring the previous version of the composite, the old composite was not retired and both the new and old composites may be running. The old SOA composite was supposed to be retired so that only the new SOA composite would be active.
To resolve this error, use the following Oracle WebLogic Scripting Tool (WLST) command to retire the old composite:
sca_retireComposite('host', 'soaport', 'user', 'password', 'composite_name', 'composite_revision')
If the WLST command is successful, verify if the new composite is active. If it is not, manually deploy the composite that failed, by following the steps in the Manually Deploying SOA Composites section.
For more information, see ”sca_retireComposite” in the WLST Command Reference for WebLogic Server section.
If custom metadata and key flexfield changes are not propagated across clusters after applying a patch, custom and flexfield medatada has not been manually exported from a source system and migrated to the other systems.
Each SOA cluster maintains its own SOA MDS schema, which results in a duplicate set of metadata for each SOA cluster that must be synchronized. Patch Manager manages this synchronization, but any custom metadata or flexfield metadata must be manually exported from a source system and then migrated to the other systems.
In case of the failures mentioned below are unable to resolve after following the steps in the Basic Troubleshoot for SOA Composite Failures section, please file a service request with Oracle Support Services and provide the logs and information.
No Base Composite Has Been Deployed
An earlier revision of the SOA composite, which is being patched, is not currently deployed in the system. This could mean that the composite was not previously provisioned on the system. Therefore, the patch validation reports the following error:
The base composite is not set as default composite. Suggestion: You must manually set the base composite as the default composite using the WLST command.
Failure at Preparation Step
The SOA composite fails during export actions, extract or attach plans, or merge updates, causing patch validation to the report the following error:
Deployment of SOA composite [{0}] failed at preparation step. Reason: [{1}] Suggestion: You must manually deploy the composite using the WLST command.
Unknown Deployment Status
The deployment of the composite reported an unknown deployment status, as shown by the following example message:
No information is available about the recovery status. The RecoverStatus object obtained is null.
The AD Controller utility, adctrl
, can monitor and control the progress of the workers called by AutoPatch to update database content and by AD Administration. For more information about workers, see the Worker Processes section.
The following sections contain steps for troubleshooting issues that may occur during patching sessions for database content:
When AutoPatch or AD Administration runs tasks in parallel, it assigns tasks to workers for completion. There may be situations that cause a worker to stop processing. To determine the status of workers, use AD Controller, as tracked in the database, and manage worker actions. Also the status of workers can be found by reviewing the Log Summary. For more information, see the Log Summary section.
Follow these steps to review the status of the workers from AD Controller:
If the worker shows a status of Failed, refer to the Determine Why a Worker Failed section for assistance in fixing the problem so Patch Manager can complete its processing. In certain situations, a worker may have a status of "Running" or "Waiting", but the worker has terminated unexpectedly. To verify the worker status, check whether the worker process exists by using the command that is appropriate for the operating system.
A worker usually runs continuously in the background and when it fails to complete the job it was assigned, it reports a status as Failed. When a worker fails its task, the next step is review the worker log files to determine what caused the failure, without wait until the other workers and the manager stop. Workers do not proceed to run tasks in the subsequent phase until all tasks in the current phase complete successfully. An immediate action has to be taken to resolve the failure so the workers can continue to run tasks in the next phase. If the task was deferred after the worker failed, there may be no action required.
The first time a task fails, the manager defers the task and assigns the worker a new task. If the deferred task fails a second time, the manager defers it a second time only if the run time of the task is less than 10 minutes. If the deferred task failed a third time, or if its run time is greater than 10 minutes, the task stays at a failed status and the worker waits for manual intervention. Action is then required because the worker stops any further processing. An example of when this scenario can occur is during seed data upload. The seed data upload may fail due to records being locked by another process. If the lock is released before the second or third attempt of the upload, the upload succeeds.
Follow these steps to find out why a worker failed:
If a worker job failed or if some worker process are hanging, the issues that caused the worker status must be resolved, and the worker has to be manually restarted. Some worker processes spawn other processes called child processes. In case of terminate a child process that is hung, the worker that spawned the process shows Failed as the status. After fix the problem, restart the failed worker. After the worker restarts, the associated child processes start as well.
Follow these steps to restart a failed worker, after to resolve the cause of the failure:
When running AD utilities, there may be situations when a worker process appears to hang, or crash. If this occurs, check if the worker process exists by using the command that is appropriate for the current operating system. After verifying there is no worker process, use the adctrl option to change the status of the worker process to “Failed”. After changing the status of the worker process to “Failed”, restart that process manually.
If Oracle Fusion Applications Patch Manager hangs, monitor the progress of the patch on the console and wait until only the hung task is running and the remaining tasks either completed successfully or failed.
MANDATORY: A process that appears to be hung could be a long-running task. Be careful when terminating processes.
To terminate a process, start AD Controller, obtain the worker's process ID from the current operating system, and then stop any hanging processes. After making the necessary changes, restart the worker.
Take the following steps to terminate a worker process that is hung:
There may be situations when AD utility must be shut down while it is running. For example, to shut down the database while Oracle Fusion Applications is running AutoPatch or during an AD Administration session. The shutdown should be performed in an orderly fashion so that it does not affect the data. To shut down the workers manually, use the following procedure:
all
for the worker number. The worker completes its current task and then quits.Oracle recommends to follow up the steps below in order to Reactivate the Manager, in cases where no workers are running tasks. They are either failed or waiting. A restarted worker resumes the failed task immediately if the worker process is running. Workers change to a Waiting status if they cannot run any tasks because of dependencies on a failed task, or because there are no tasks left in the phase. When no workers are able to run, the manager becomes idle.
Complete the following steps for each worker:
This error occurs during patching, "Unable to start universal connection pool". Connections to the database cannot be established due to timeout limits.
Consider tuning the listener parameters INBOUND_CONNECT_TIMEOUT_listenername
and SQLNET.INBOUND_CONNECT_TIMEOUT
for the server.
For more information, see ”SQLNET.EXPIRE_TIME Parameter” and ”INBOUND_CONNECT_TIMEOUT Parameter" sections in the Oracle Fusion Applications Performance and Tuning Guide.
When you patch database artifacts, your system should be in an idle state. If this is not the case, the patching session may hang due to locks. Examples of locks that can cause the patching session to hang are PL/SQL packages that are accessed by Oracle Enterprise Scheduler Service Server, a locked table, or a locked table row. After a specific wait time, such as 30 minutes, Patch Manager performs a check to determine whether the patching session is blocked by another session. If a blocking session is found, a message describing the block is sent to the log file, as shown in the following example:
"[2011-03-14T02:12:18.112-07:00] [apps] [NOTIFICATION] [] [AutoPatch] Worker 4 is blocked by session 3868 ... Please fix the session to avoid indefinite waiting
The worker that is blocked remains in a Running status. To resolve the issue and release the lock, stop the blocking session using the command that is appropriate for your operating system. After the blocking session is no longer running, the hung worker proceeds to complete its task. To identify the sessions that are blocking patching sessions, following SQL*Plus query :
SELECT blocking_session, sid "Blocked Session", module "Blocked Module", seconds_in_wait FROM gv$session WHERE status = 'ACTIVE' AND module like 'PATCHING_SESSION:%' AND blocking_session_status = 'VALID' AND user = '<FUSION schema>';
When multiple seed data files are uploaded for the same flexfield but for different flexfield contexts, the upload tasks can fail due to locking issues. The failed tasks appear in the log file as the following error:
Loading failed with a JboException: JBO-25014: Another user has changed the row with primary keyoracle.jbo.Key ...
AutoPatch defers any failed tasks to the end of the phase and reattempts the failed tasks only after attempting all tasks in the phase at least once. Usually, the flexfield seed data tasks that failed due to the locking issue succeed on subsequent attempts. If the flexfield seed data task succeeds on the retry, ignore these errors. If the task remains in a failed state, use the AD Controller utility to retry the failed task.
For more information, see the Restart a Failed Worker section.
If in the data log file notice values, such as display_value_1
, this most likely means that Oracle Fusion Applications Patch Manager found conflicting seed data for a non-primary unique index, due to customer defined data. To avoid a failure during seed upload, a numeric value is added to the row with the duplicate value. If subsequently update the customer defined data so that it no longer conflicts with the Oracle defined data, then the next patch that delivers this data will remove the numeric value. Note that Oracle recommends not update Oracle defined data, as this results in marking the data as customized and prevents any future updates from being delivered to the customized row.
In case of the Oracle Fusion Applications database need to be connected to troubleshoot database related issues, by running SQL*Plus, for example, the appropriate environment variables needs to be set up . For setting any environment variable, run the adsetenv
script to generate the APPSORA.env
file, which when sourced, sets all environment variables, as follows:
UNIX
sh adsetenv.sh source APPSORA.env echo $TWO_TASK
This error occurs during patching, “Error while running deployment pre-reqs: Initialization of OPatchFMWTarget failed”.
In case of the Patch validation (Run level 63) fails with the following log snippet:
Error while running deployment pre-reqs: Initialization of OPatchFMWTargetfailed.
Reason: There are no end point instances defined in <OPatchFAConfigInstance> object. Please define end point instances in <OPatchFAConfigInstance> object.Oracle Fusion Applications Patch Manager cannot proceed further with validation. Review the following steps for basic troubleshooting:
Running the following SQL query should turn up at least one record:
select count(*) from ad_patch_util_sessions where status='COMPLETED_WITH_WARNINGS';
Apply apply data fix patch from net/den01fhc/scratch/sallamse/p24811973_1111110_Fusion_GENERIC.zip (instructions included in the readme.txt bundled into the patch; the patch itself will be uploaded to ARU once certified).
Usually FASPOT scripts ran successfully and fast, and the number of issues are minimal. When applying manually it is important to read every single patch README
file carefully as it might contain special pre- and post-install instructions. Also, there might be requirements to rollback previously installed patches to resolve conflicts. In general these technology patches are applied by using opatch, adpatch and bsu (Weblogic). As mentioned before FASPOT automates this process.
The following sections contain steps for troubleshooting issues that may occur during patching sessions for FASPOT:
faspot.properties
fileWhen faspot.properties
presents any typo, run to target prepare-local-env
as mentioned in the Prepare the Patch Staging Area section.
faspot.properties
FileIn file faspot.properties
there might be some parameters which are effectively obsolete like OC_DOMAIN_HOME_LOC_LIST
.
File a service request with Oracle Support Services and provide the logs and information.
Currently FASPOT script directory is copied to PATCH_WORK_DIR folder. It is NOT required to run FASPOT scripts from there.
FASPOT has been developed for Linux 64-bit based installations of Fusion Apps. It is possible to run it on alternate platforms but some restrictions apply:
faspot.sh
does not work (instead of ant
directly).setEnv.sh
have to be done manually.