8  Monitor and Troubleshoot Patches

To examine the activities performed during patching sessions, review the associated log files. Patch Manager consolidates log files for each patching session under the directory, APPLICATIONS_CONFIGAPPLICATIONS_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR.. This directory contains the top-level log file, logsummary_fapmgr_command_timestamp.html, along with related log files for each task performed during a fapmgr session. During a session, it is possible to view the Log Summary HTML file from a browser, which provides links to individual log files. To view the progress of the current patching session, refresh the Log Summary HTML file periodically. If a task fails, access the links to the associated log files to assist in diagnosing the failure. For more information, see the Log Summary section.

When a patching session completes, its log files are archived in APPLICATIONS_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR/logarchive/Patch Number/fapmgr_command/session ID/timestamp. The session ID is unique and the time stamp is the start time for the session. Note that whenever Patch Manager runs a command where there is no patch number, such as bootstrap, abort, and report, the archive logs are named APPLICATION_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR/logarchive/fapmgr_command/timestamp.

Log files for OPatch actions are initially written to the FA_ORACLE_HOME/cfgtools/opatch/patch number_timestamp directory.

The following table contains a list of log files created by Patch Manager during patching activities:

The multi-apply feature of Patch Manager applies multiple patches after splitting them into groups within a My Oracle Support patch plan. Each group contains at least one patch and it is run as an individual session internally. Log files and diagnostic reports are generated for each individual session, in addition to a primary log file and diagnostics report for the overall multi-apply session.

Examine the activities performed during multi-apply patching sessions by reviewing the associated log files. Patch Manager consolidates log files for each patching session under the directory, APPLICATIONS_CONFIG/lcm/logs/<Fusion Applications Release Version>/FAPMGR. This directory contains the top-level log file, logsummary_fapmgr_command_timestamp.html, along with related log files for each task performed during a multi-apply session. During a session view this log summary HTML file from a browser, which provides links to individual log files. Periodically refresh the log summary HTML file to view the progress of the current patching session. If a task fails, access the links to the associated log files to assist in diagnosing the failure. See the Log Summary section to get more information how the log summary is created and how the report is created.

Multi-apply sessions create a primary diagnostics report that provides information about the multi-apply patching session. It also includes links to corresponding diagnostics reports and archived log files for individual patches that were applied as a group.

The following table contains a list of log files created by Patch Manager during multi-apply patching activities:

At the beginning of a multi-apply patching session, the log files from the previous session move to FA_ORACLE_HOME/admin/FUSION/logarchive/MAPPLY/timestamp. For each patch that is applied as part of a group, a LogFilesLocationPointer.log is created at the location where the log files will be archived, as if the patch was applied as a standalone patch. The content of the pointer files provides the absolute location of the corresponding group apply logs and provides the ability to navigate directly to the log files for a specific patch, even though it is applied using multi-apply. For example, if patches 19191630 and 20193112 are grouped into GROUP1 and then applied, log files are archived in the following directory structure:

 logarchive
|
|-- MAPPLY
|        |-- 20120610235002
|        |       |-- FAPMgr_Multiapply_apply_20120610235002.log
|        |       |-- FAPMgr_Multiapply_DiagnosticsSummary_20120610235002.html
|        |       |-- FAPMgr_Multiapply_DiagnosticsSummary_20120610235002.xml
|        |       |-- GROUP1
|        |       |       |-- APPLY
|        |       |       |       |-- 123
|        |       |       |       |       |-- 20120610235302
|        |       |       |       |       |       |-- FAPatchManager_apply_20120610235302.log
|        |       |       |-- VALIDATE
|        |       |       |       |-- 122
|        |       |       |       |       |-- 20120610235102
|        |       |       |       |       |       |-- FAPatchManager_validate_20120610235102.log
|-- 19191630
|        |-- APPLY
|        |       |-- 123
|        |       |       |-- 20120610235302
|        |       |       |       |-- LogFilesLocationPointer.log
|        |-- VALIDATE
|        |       |-- 122
|        |       |       |-- 20120610235102
|        |       |       |       |-- LogFilesLocationPointer.log
|-- 20193112
|        |-- APPLY
|        |       |-- 123
|        |       |       |-- 20120610235302
|        |       |       |       |-- LogFilesLocationPointer.log
|        |-- VALIDATE
|        |       |-- 122
|        |       |       |-- 20120610235102
|        |       |       |       |-- LogFilesLocationPointer.log 

After the failure of a patching session, the following scenarios are possible:

• To abandon the previous session and start a new session, see the Abandon a Failed Patching Session section.

• Despite failure, some tasks may display as running. For more information, see the Recovery from an Interrupted Patching Session section.

• There can be only one patching session active at one time for Oracle Fusion Applications, Oracle Fusion Functional Setup Manager, and Oracle Fusion Middleware Extensions for Applications (Applications Core). If there is a failed Applications Core or Functional Setup Manager patching session that must be cleaned up, see the Abandon a Failed Patching Session section.

If the patch session failed and there is no need to restart it, abandon the session by using the abort command. Remember that only one patching session can be running at a time and always make sure that processes associated with the previous patching session does not exist.

Mandatory: If the patching session is aborted, this session cannot be restarted and any pending patching actions, such as deployment actions, must be performed manually.

To abandon a previously failed session, run the fapmgr abort command . The abort command cleans up any intermediate states tracked by fapmgr and moves the log files for the abandoned session to an archive log directory so that a new patching session could start. Note that a session that is actively running cannot be abandoned.

Use the following syntax for the fapmgr abort command:

(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh abort [-force] [-logfile log file name] [-loglevel level

The following table describes the options available for the abort command:

If the fapmgr abort command errors with a message "Another APPLY session is already running", use the fapmgr forcefail command first. Also confirm that the FND_INSTALL_PROCESSES table does not exist. For more information, see the Recovery from an Interrupted Patching Session section.

If a patching session was interrupted by a system failure when Patch Manager and AutoPatch were running and the patching-related database tables still show the patching session as running (only one patching session can be active at a time), but no patching-related processes are actually running, use the fapmgr forcefail command to update the patching tables by performing the following steps:

If initiating a patching session from a terminal server, such as a laptop, the connection may be lost during extended periods of time when no messages are sent to the terminal. The terminal server may interpret this as inactivity and then end the session. For example, no messages are sent to the client when Patch Manager is stopping and starting servers, waiting for a failed task to be fixed, or is hung on a database task. To avoid this situation, ensure that the terminal server is configured appropriately to handle long durations of inactivity.

While applying a patch using hot patching, the following type of error is reported:

ESS Jobs are running on the domains (HCMDomain) even after maintenance wait period.

If this error is reported, consider the following alternatives:

• Wait for ESS Jobs execution finish.

• Force all active tasks to terminate after the maintenance wait period, applying the patch using the forceterminateactivetasks option, as shown in the following example:

  fapmgr.sh apply -patchtop patch_top -hotpatch -online -stoponerror -maintenancewaitperiod 1 -maintenanceendtime date_time -forceterminateactivetasks
  

Run the following command to validate if the patch is structured to be applied as a hot patch :

sh fapmgr.sh  validate -patchtop /patch_top/18060344 -hotpatch -online

Validation FAILED.
You cannot apply the patch.
Check the log file for unexpected infrastructure failures.

If a patch is being applied and it contains BI Publisher artifacts, the BI Presentation servers should not be running. If BI Presentation servers are running during the deployment of BI Publisher artifacts, the following error occurs:

java.lang.RuntimeException: Webcat patch file creation failed! 

To resolve this issue, shut down the BI Presentation servers to release locks on the Oracle BI Presentation Catalog.

If during patch validation, the following error is received:

weblogic.management.mbeanservers.edit.EditTimedOutException  

Manually release the edit session. This situation occurs when a domain is already in "edit" mode during patching, for example, when the server crashes when Patch Manager tries to stop and restart it.

To manually release the edit session, use the following procedure:

If flexfield changes are not ready to be implemented after applying a patch containing flexfield changes, then revert to a previous version of that flexfield definition. The Flex Modeler creates an MDS label, FlexPatchingWatermarkdate+time, before it initiates the flexfield deployment, which establishes the watermark for what was in MDS before the patch was applied. The name of the label is included along with the Flex Deployment report in the patching log file as a reference.

To use a previous version of a flexfield definition, use the WLST command promoteMetadataLabel.

To delete all previous MDS labels for a flexfield, first confirm the changes delivered by a patch can be used (keep old MDS labels impacts performance), then use the WLST command deleteFlexPatchingLabels.

If a patch contains BI artifacts the BI OPMN control process, which is similar to a node manager, then it has to be up for online mode validation to succeed. Online validation reports the following error if the BI OPMN control process is not up:

The deployment of BI Publisher artifacts will not be attempted because the
BI Presentation server is neither fully started nor down.
One likely cause is that the BI OPMN control process is not running. Make
sure that the BI OPMN control process is up and the BI
Presentation server is started successfully before applying this patch. If
this server is not fully started, you must stop the BI Presentation
server, manually deploy the BI Publisher artifacts, and then re-start the BI
Presentation server

To resolve the issue, ensure that the BI OPMN control process is up and running.

If while applying a patch, Patch Manager fails with the error "Failed while trying to update task status" or "Error while updating session status", solve the issue by performing the following steps:

The opatch -lsinventory -detail command provides a report that lists all patches, artifacts and artifact versions that were modified within each patch applied to a given Oracle home. This report lists only artifacts that were modified. If an artifact does not appear on this report, then the artifact remains at its base version. Run this report when Oracle Support Services are used and an artifact version needs to be provided.

To generate the report, use the following command syntax:

opatch/opatch lsinventory -detail -oh FA_ORACLE_HOME -invPtrLoc \
FA_ORACLE_HOME/oraInst.loc -jre JAVA_HOME

The following example depicts the section of the report that displays patches applied. To use the report, find the latest entry for the specific artifact and note the version reported.

Interim patches (11) :
 
Patch  11801        : applied on Wed Feb 02 17:57:53 PST 2011
   Created on 18 Jan 2011, 16:09:54 hrs PST8PDT
   Bugs fixed:
     11801
   Patch is translatable.
   Files Touched:
     AdfPjfIntMspUi.jar, version "23.0" --> ORACLE_HOME/prj/deploy/EARProjectsFinancials.ear/EARProjectsFinancials.war/WEB-INF/lib/AdfPjfIntMspUi.jar
   Patch Location in Inventory:
     /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/inventory/oneoffs/11801
   Patch Location in Storage area:
     /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/.patch_storage/11801_Jan_18_2011_16_09_54

Patch  12801        : applied on Tue Feb 01 21:30:17 PST 2011
   Created on 28 Jan 2011, 14:26:56 hrs PST8PDT
   Bugs fixed:
     12801
   Patch is translatable.
   Files Touched:
     EFFmetadata.mar, version "35.0" --> ORACLE_HOME/hcm/deploy/EarHcmCoreSetup.ear/EFFmetadata.mar
     system-jazn-data.xml, version "35.0" --> ORACLE_HOME/hcm/security/policies/system-jazn-data.xml
   Patch Location in Inventory:
     /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/inventory/oneoffs/12801
   Patch Location in Storage area:
     /u01/FUSIONAPPS_APPLTOP/fusionapps/applications/.patch_storage/12801_Jan_28_2011_14_26_56

Problem

Solution

SOA composite validation and deployment can fail for multiple reasons. Review the following steps for basic troubleshooting:

1. After validate or apply a patch that contains SOA composites, review the Diagnostics report for errors. The report output is in HTML format for viewing from a browser and is located here:

   FA_ORACLE_HOME/admin/FUSION/log/FAPMgrDiagnosticsSummarydate:session.html
   

   The Module Task Details section of the report displays the following information to assist in your troubleshooting:

   • Mode: Middleware, in this case.

   • Phase: Validation or Patch Application, in this case.

   • Product Family: The short name of the product family.

   • Task.

     The following information displays for SOA composites:

     • Name of composite

     • Domain name

     • Path to composite JAR in FA_ORACLE_HOME

     • Revision of composite

   • Status: Possible values of Success, Failed, or Skipped.

   • Duration: Total time the task ran.

   • Start Time: Time and date the task started.

   • End Time: Time and date the task ended.

   • Warning/Error Message: The error message displays as a java.lang.RuntimeException. The message often includes a suggestion for resolving the failure.

   • Log file: The path and file name of the high level log file, FAPatchManager_fapmgr_command_timestamp.log, associated with the task. From the Module Execution Summary section of the Diagnostics report, review log files by accessing the link to the Log Summary. For more information, see the Log Summary section.

   • Line Numbers: The line numbers in the log file associated with the task.

2. SOA log files are located in this directory: FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_release_number/soalogs. If merge operations are performed on a SOA composite, due to runtime customizations, such as design time and run-time (DT@RT) changes or property changes, a merge log file is generated. There is one merge log file per domain and the name of merge log files follows this naming convention: fapatch_domain_nametimestamp.merge.log

3. Restart the SOA servers and for each failure, follow Steps 4 through 9.

4. Determine if there is a cause for the error that must be resolved, in addition to restarting the server, by referring to the Diagnostics report, Oracle Fusion Applications log files and SOA log files. Examples of other causes include database errors, coherence configuration errors, and out of memory issues.

5. Determine if a manually restore the system back is need it to its state before the application of the patch was attempted. The following scenarios do not require manual restoration of the system:

   • Errors occurred during the validation phase.

   • Errors occurred during the patch application phase but the recovery was successful, so the system was recovered to its original state. The Diagnostics report displays this message in this case:

     Deployment of SOA composite[artifact_name and path]failed, but the system has recovered successfully. 
     Suggestion: You must manually deploy the composite using the WLST command.
     

   If the system needs to be restored, follow the steps in Step 6.

6. If a failed deployment leaves a composite in an inconsistent state, restore the system to its original state. In case of prefer to use Oracle Fusion Applications Control to restore the system, see Step 7. Perform the following WLST commands to restore the system:

   1. Before the application of the patch from the Diagnostics report, please find out the last active revision of the composite, it is indicated by this message: The last active version of the composite before patch application began was [version].

      In the following examples, 1.0 is the previous revision and 2.0 is the patched revision.

   2. Undeploy the patched revision of the composite if it exists in the system.

      sca_undeployComposite('http://server01:8001', 'POProcessing', '2.0') 
      

   3. Mark the previously active revision of the composite as active and as a default revision.

      To activate the old revision, use the following command:

      sca_activateCompositeMb('POProcessing', '1.0')
      

      To assign the default composite, use the following command:

      sca_assignDefaultCompositeMb('POProcessing', '1.0')
      

7. Restore the system to its original state using Enterprise Manager Cloud Control with Oracle, follow up the steps below:

   1. Find the active revision of the composite before the application of the patch from the Diagnostics report, as indicated by this message: The last active version of the composite before patch application began was [version].

      In the following steps, 1.0 is the previous revision and 2.0 is the patched revision.

   2. Undeploy the patched revision of the composite if it exists in the system.

   3. Mark the previously active revision of the composite as active and as a default revision.

8. Manually deploy the SOA composites included in the patch by following the steps :

   sca_patchComposite('SOA-Infra URL', user, password,
   '/FA_ORACLE_HOME/crm/deploy//sca_FinAprev2.0.jar', mergeLogFile='/tmp/mergelog
   

9. If you are unable to resolve the failure, file a service request with Oracle Support Services and provide the logs and information as described in the Get Started with Troubleshooting and Logging Basics for Oracle SOA Suite section in the Oracle Fusion Applications Administrator's Guide.

This section describes common problems and solutions for SOA composite validation failures.

MANDATORY :Errors that occur during the validation phase must be resolved before applying the patch. In case of these errors occurred during patch application, manually deploy the SOA composites after resolve the validation errors.

Patch Manager captures the OPatch validation log files, which can be found by referencing the Diagnostics report or the Log Summary. The errors in the log files provide information about the cause of the failure and are often followed by recommended actions.

This section contains troubleshooting information about the following failures:

• Oracle JDeveloper Customization Error

• SOA Server Not Available

• Administration Server Not Available

• SOA-Infra Server Is Ready

• Composite with Identical Revision Is Already Deployed

This section describes common problems and solutions for SOA composite deployment failures during patching.

MANDATORY: Errors that occur during the deployment phase must be resolved as soon as possible because the system has been modified. Do not try to roll back or reapply patches that have errors during deployment. After resolve the cause of the error, you must deploy the composite manually.

This section contains the following topics:

• Failed to Make New Composite Revision the Default

• Failed to Retire Previous Composite Revision

• Custom Metadata and Key Flexfield Changes Are Not Propagated Across Clusters

In case of the failures mentioned below are unable to resolve after following the steps in the Basic Troubleshoot for SOA Composite Failures section, please file a service request with Oracle Support Services and provide the logs and information.

• No Base Composite Has Been Deployed

  An earlier revision of the SOA composite, which is being patched, is not currently deployed in the system. This could mean that the composite was not previously provisioned on the system. Therefore, the patch validation reports the following error:

  The base composite is not set as default composite. Suggestion: You must manually set the base composite as the default composite using the WLST command.
  

• Failure at Preparation Step

  The SOA composite fails during export actions, extract or attach plans, or merge updates, causing patch validation to the report the following error:

  Deployment of SOA composite [{0}] failed at preparation step. Reason: [{1}] Suggestion: You must manually deploy the composite using the WLST command.
  

• Unknown Deployment Status

  The deployment of the composite reported an unknown deployment status, as shown by the following example message:

  No information is available about the recovery status. The RecoverStatus object obtained is null.
  

Follow these steps to start AD Controller:

When AutoPatch or AD Administration runs tasks in parallel, it assigns tasks to workers for completion. There may be situations that cause a worker to stop processing. To determine the status of workers, use AD Controller, as tracked in the database, and manage worker actions. Also the status of workers can be found by reviewing the Log Summary. For more information, see the Log Summary section.

Follow these steps to review the status of the workers from AD Controller:

If the worker shows a status of Failed, refer to the Determine Why a Worker Failed section for assistance in fixing the problem so Patch Manager can complete its processing. In certain situations, a worker may have a status of "Running" or "Waiting", but the worker has terminated unexpectedly. To verify the worker status, check whether the worker process exists by using the command that is appropriate for the operating system.

A worker usually runs continuously in the background and when it fails to complete the job it was assigned, it reports a status as Failed. When a worker fails its task, the next step is review the worker log files to determine what caused the failure, without wait until the other workers and the manager stop. Workers do not proceed to run tasks in the subsequent phase until all tasks in the current phase complete successfully. An immediate action has to be taken to resolve the failure so the workers can continue to run tasks in the next phase. If the task was deferred after the worker failed, there may be no action required.

The first time a task fails, the manager defers the task and assigns the worker a new task. If the deferred task fails a second time, the manager defers it a second time only if the run time of the task is less than 10 minutes. If the deferred task failed a third time, or if its run time is greater than 10 minutes, the task stays at a failed status and the worker waits for manual intervention. Action is then required because the worker stops any further processing. An example of when this scenario can occur is during seed data upload. The seed data upload may fail due to records being locked by another process. If the lock is released before the second or third attempt of the upload, the upload succeeds.

Follow these steps to find out why a worker failed:

If a worker job failed or if some worker process are hanging, the issues that caused the worker status must be resolved, and the worker has to be manually restarted. Some worker processes spawn other processes called child processes. In case of terminate a child process that is hung, the worker that spawned the process shows Failed as the status. After fix the problem, restart the failed worker. After the worker restarts, the associated child processes start as well.

Follow these steps to restart a failed worker, after to resolve the cause of the failure:

When running AD utilities, there may be situations when a worker process appears to hang, or crash. If this occurs, check if the worker process exists by using the command that is appropriate for the current operating system. After verifying there is no worker process, use the adctrl option to change the status of the worker process to “Failed”. After changing the status of the worker process to “Failed”, restart that process manually.

If Oracle Fusion Applications Patch Manager hangs, monitor the progress of the patch on the console and wait until only the hung task is running and the remaining tasks either completed successfully or failed.

MANDATORY: A process that appears to be hung could be a long-running task. Be careful when terminating processes.

To terminate a process, start AD Controller, obtain the worker's process ID from the current operating system, and then stop any hanging processes. After making the necessary changes, restart the worker.

Take the following steps to terminate a worker process that is hung:

There may be situations when AD utility must be shut down while it is running. For example, to shut down the database while Oracle Fusion Applications is running AutoPatch or during an AD Administration session. The shutdown should be performed in an orderly fashion so that it does not affect the data. To shut down the workers manually, use the following procedure:

Oracle recommends to follow up the steps below in order to Reactivate the Manager, in cases where no workers are running tasks. They are either failed or waiting. A restarted worker resumes the failed task immediately if the worker process is running. Workers change to a Waiting status if they cannot run any tasks because of dependencies on a failed task, or because there are no tasks left in the phase. When no workers are able to run, the manager becomes idle.

Complete the following steps for each worker:

This error occurs during patching, "Unable to start universal connection pool". Connections to the database cannot be established due to timeout limits.

Consider tuning the listener parameters INBOUND_CONNECT_TIMEOUT_listenername and SQLNET.INBOUND_CONNECT_TIMEOUT for the server.

For more information, see ”SQLNET.EXPIRE_TIME Parameter” and ”INBOUND_CONNECT_TIMEOUT Parameter" sections in the Oracle Fusion Applications Performance and Tuning Guide.

When you patch database artifacts, your system should be in an idle state. If this is not the case, the patching session may hang due to locks. Examples of locks that can cause the patching session to hang are PL/SQL packages that are accessed by Oracle Enterprise Scheduler Service Server, a locked table, or a locked table row. After a specific wait time, such as 30 minutes, Patch Manager performs a check to determine whether the patching session is blocked by another session. If a blocking session is found, a message describing the block is sent to the log file, as shown in the following example:

"[2011-03-14T02:12:18.112-07:00] [apps] [NOTIFICATION] [] [AutoPatch] Worker 4 is
blocked by session 3868 ... Please fix the session to avoid indefinite waiting

The worker that is blocked remains in a Running status. To resolve the issue and release the lock, stop the blocking session using the command that is appropriate for your operating system. After the blocking session is no longer running, the hung worker proceeds to complete its task. To identify the sessions that are blocking patching sessions, following SQL*Plus query :

SELECT blocking_session,
       sid "Blocked Session",
       module "Blocked Module",
       seconds_in_wait
 FROM gv$session
 WHERE status = 'ACTIVE'
 AND module like 'PATCHING_SESSION:%'
 AND blocking_session_status = 'VALID'
  AND user = '<FUSION schema>';

When multiple seed data files are uploaded for the same flexfield but for different flexfield contexts, the upload tasks can fail due to locking issues. The failed tasks appear in the log file as the following error:

Loading failed with a JboException: JBO-25014: Another user has changed the
row with primary keyoracle.jbo.Key ...

AutoPatch defers any failed tasks to the end of the phase and reattempts the failed tasks only after attempting all tasks in the phase at least once. Usually, the flexfield seed data tasks that failed due to the locking issue succeed on subsequent attempts. If the flexfield seed data task succeeds on the retry, ignore these errors. If the task remains in a failed state, use the AD Controller utility to retry the failed task.

For more information, see the Restart a Failed Worker section.

In case of the Oracle Fusion Applications database need to be connected to troubleshoot database related issues, by running SQL*Plus, for example, the appropriate environment variables needs to be set up . For setting any environment variable, run the adsetenv script to generate the APPSORA.env file, which when sourced, sets all environment variables, as follows:

UNIX

sh adsetenv.sh
source APPSORA.env
echo $TWO_TASK

This error occurs during patching, “Error while running deployment pre-reqs: Initialization of OPatchFMWTarget failed”.

In case of the Patch validation (Run level 63) fails with the following log snippet:

Error while running deployment pre-reqs: Initialization of OPatchFMWTargetfailed.

Reason:  There are no end point instances defined in <OPatchFAConfigInstance> object. Please define end point instances in <OPatchFAConfigInstance> object.Oracle Fusion Applications Patch Manager cannot proceed further with validation. Review the following steps for basic troubleshooting: