Skip Headers
Oracle® Fusion Applications Upgrade Guide
11g Release 6 (11.1.6)

Part Number E35833-07
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

6  Troubleshooting the Upgrade

This chapter provides information to assist you in troubleshooting RUP Installer and Language Pack Installer sessions.

This chapter contains the following topics:

6.1 RUP Installer Log File Directories

Table 6-1 contains a list of log directories for RUP Installer activities.

Table 6-1 Log Directories for RUP Installer Activities

Log directory name Description

oracle_inventory/logs

Installation phase and Oracle Fusion Middleware patch set installation logs.

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0

Top level directory for RUP Installer logs.

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/ARCHIVE/timestamp

Top level log directory where log files are moved when you retry the installation session.

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/configlogs

Top level log directory for configuration assistants. A log file exists for each configuration assistant. For more information, see Section 6.1.1, "Log Files for Configuration Assistants".

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/PatchManager_DBPatch

Database upload configuration assistant logs after failure or completion. For more information, see Section 6.1.2, "Log Files for the Database Upload Configuration Assistant".

APPLICATIONS_BASE/instance/BIInstance/diagnostics/logs

Oracle Business Intelligence logs.

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/StartStop

StartStop utility logs.

Note that server logs are located under respective domains. For example, the AdminServer log for CommonDomain is under APPLICATIONS_CONFIG/domains/hostname/CommonDomain/servers/AdminServer/logs.

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/soalogs

SOA artifacts configuration assistant logs.

Note that SOA server logs are located under respective domains. For example, the SOA server logs for CommonDomain are under APPLICATIONS_CONFIG/domains/hostname/CommonDomain/servers/soa_server1/logs. For more information, see Section 6.23.1, "SOA Composite Log Files".

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/configlogs/PatchManager_DownloadedPatches

Applying Downloaded Patches configuration assistant logs.


6.1.1 Log Files for Configuration Assistants

During the configuration phase of the upgrade, each configuration assistant creates its own log file under the FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_releasenumber/configlogs directory. All messages that are generated during the configuration assistant processing are written to this log file. The only information related to configuration assistants that is written to the main log file, FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_releasenumber, are those messages that indicate that a configuration assistant started and the result of its processing, such as success or failure.

6.1.2 Log Files for the Database Upload Configuration Assistant

During the execution of the database upload configuration assistant, log files are created under the FA_ORACLE_HOME/admin/FUSION/log directory. Upon completion or failure of the database upload, the log files move to the FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_releasenumber/PatchManager_DBPatch directory. The current releasenumber is 11.1.6.0.0.

6.2 Troubleshooting Failures During the Installation Phase

Perform the following steps when an error occurs during the installation phase:

  1. Click Cancel to exit out of the installer.

  2. Review the log files to determine the cause of the failure. The log files reside in oracle_inventory/logs/installtimestamp.log.

  3. Resolve the cause of the failure.

  4. Start the installer using the same command syntax that you used for the previous incomplete installation. For more information, see Section 3.1, "Run RUP Installer" or Section 5.4, "Install or Upgrade a Language". After canceling the previous installation and starting again, a pop up dialog displays, asking if you want to continue the previous incomplete installation. Select Yes to continue running the previous session. If the error is not recoverable, you must restore and restart from backup.

  5. If you choose to continue with the failed installation, the installer opens at the screen where it was canceled. When canceled during the copy action, it relaunches in the Installation Summary screen. Click Next to navigate through the Installation Summary screen. When the Installation Progress screen displays, click Install to start the installation again.

Troubleshooting steps are described for the following specific failures that may occur during the installation phase:

6.2.1 CFGLOG-00056: Exception caught while getting node-manager homes

Problem

Within a few seconds of starting the installer, you receive the following message:

In the log file:

SEVERE: CFGLOG-00056 : Exception caught while getting node-manager homes

In the user interface:

CFGLOG-00052 : Error occurred while moving instance specific files

Solution

This failure is the result of having an incompatible version of OPatch in FA_ORACLE_HOME. To resolve the issue, download and apply patch 14044793, which contains the compatible version of OPatch.

6.2.2 Invalid Oracle Home

Problem

In the Installation Location page, you receive a message about entering an invalid Oracle home, even though the location displayed on the page is correct. The installer reads /etc/oraInst.loc to determine the location of the central inventory. Review the following settings:

Solution

To resolve this problem:

  • Ensure that the /etc/oraInst.loc file on the machine where you are running the installer is pointing to the correct central inventory location.

  • Ensure that the FA_ORACLE_HOME matches the values provided during provisioning. If a /net/location was provided as the Oracle home location during provisioning, the same /net/location that corresponds to FA_ORACLE_HOME should be provided during the installation. You can find this location by following these steps:

    • Open /etc/oraInst.loc and find the path to oraInventory, which is the central inventory, for example, server01/appmgr/APPTOP/oraInventory.

    • Change directory to the ContentsXML directory under the central inventory, for example, server01/appmgr/APPTOP/oraInventory/ContentsXML.

    • Open the inventory.xml file to find the correct directory path to FA_ORACLE_HOME.

6.2.3 Error in Writing to File, Text File Busy

Problem

During the installation phase of RUP Installer, you receive the following message on a Unix platform.

Error in writing to file'/server01/APPLICATIONS_BASE/fusionapps/applications/lcm/ad/bin/adctrl'
(Text file busy)

Solution

To resolve this issue, perform the following steps.

  1. Run the lsof command using the full directory path of the file that is busy.

    /usr/bin/lsof full_path_to_file
    
  2. You should receive a list of process ids that are using the file. Kill each process using the appropriate command for your operating system.

  3. After all processes are no longer running, click Continue in RUP Installer.

6.2.4 Inventory Pointer File is Empty

Problem

After running the installer, the contents of oraInst.loc were removed.

Solution

The installer always tries to copy the inventory pointer file specified by the -invPtrLoc option to the Oracle home on which the release is to be installed. If you specify an incorrect path for the -invPtrLoc file, the inventory pointer file could result in being an empty file. Review the following possible solutions for this issue:

  • For best results, if you are using the -invPtrLoc option, use it with this value: FA_ORACLE_HOME/oraInst.loc. This avoids a situation where you may inadvertently exclude part of the directory path to the file, as in the case of using a mapped drive. For example, if Oracle home is registered in inventory with a /net path, such as /net/home/oraInst.loc, and you provide /home/oraInst.loc to the invPtrLoc option, the installer interprets the two paths as different. The end result is an empty inventory pointer file.

  • If FA_ORACLE_HOME is registered in central inventory with a /net path, then you must include /net when specifying the location of the inventory pointer file with the -invPtrLoc option, for example, -invPtrLoc /net/directory_path/oraInst.loc.

  • Restore from a backup copy of your oraInst.loc file in case the original file is damaged. You can find this in /etc/oraInst.loc.

  • You can recover from this error by creating a new oraInst.loc. See the "Creating the oraInst.loc File" section in the relevant Oracle Database installation guide, for example, Oracle Database Installation Guide, 11g Release 2 (11.2) for Linux.

    Then click Retry.

6.3 Failure During Analysis of Applications Policies

Problem

A failure occurs during applications policy analysis.

Solution

Review the log file that is generated by each stripe. These log files are located under the main log directory, FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/timestamp and are named as follows:

After you resolve the JAZN analysis error, retry the analysis for the failed stripe to confirm the issue is resolved.

6.4 General Troubleshooting During the Configuration Phase in GUI Mode

This section describes the following troubleshooting scenarios:

6.4.1 Restart a Failed Installer Session

The installer can be restarted to rerun all failed configuration assistants as well as those configuration assistants that were not started from the previous session. When a configuration assistant or step fails, the Configuration Progress screen displays the location of the log file and the exception that caused the failure. You can also view the content of the log files that appear at the bottom of the screen to obtain detailed information to assist in diagnosing the cause of the failure.

If one or more failures occur during the configuration phase, after the final configuration assistant is complete, the following message displays:

Configuration is completed with errors, exit the installer by clicking the 'Cancel' button and retry the failed configurations.

Perform the following steps to rerun the installer and retry the failed configuration assistants:

  1. Click Cancel to exit the installer.

  2. Resolve the issues that caused the failure.

  3. Start the installer using the same command syntax that you used for the previous incomplete installation.

  4. A pop up dialog displays, asking if you want to continue the previous incomplete installation. Select Yes to continue running the previous session. If you select No, the installer starts from the beginning and it will fail, indicating that a release cannot be installed again in the same environment. You would then need to restore from your backup and restart the installer.

  5. The Configuration Progress screen displays only the failed and remaining configuration assistants, and then runs these configuration assistants.

  6. Assuming all configuration assistants complete successfully, click Next to go to the Installation Complete screen and then click Finish to end the session. If a configuration assistant fails again and you want to attempt to run the session again, click Cancel to save the session. If all configuration assistants were successful for the first installer, the second installer launches automatically. If all configuration assistants completed successfully, click Finish to end the session.

    Note that Language Pack runs only one installer.

6.4.2 Troubleshoot Failures While Parallel Tasks Are Running

If one or more tasks in a group fail, you can select the failed tasks in any combination, and the Abort, Retry, and Continue buttons are enabled as appropriate for the selected tasks. For example, if two tasks in a group fail, and the first task allows you to select Continue, but the other task does not, then the Continue button is not enabled if you select both tasks.

You can process one or more failed tasks at a time. For example, if three tasks fail, you can retry one of them, and while it is running, you can abort the second task. Then you can retry the third task. When the first and third tasks finish processing, the next step depends on whether the second task is mandatory. If it is a mandatory task, the installer stops, and if it is non-mandatory the installer continues with the next task after the group. You can also pick two out of three or all three tasks and select Retry, Abort, or Continue, based on which buttons are enabled.

Note that all tasks in a group must either fail or complete successfully before the Cancel button is enabled.

The following example depicts a group of four configuration tasks that are running in parallel and three of the four tasks fail.

  1. Four tasks were running in parallel. Three tasks fail and the remaining task is successful. Note that the Abort, Retry, and Continue buttons are not enabled because the check boxes for the failed tasks are not checked. In the case of failure, the check boxes are enabled for failed tasks only after all tasks in the group have either failed or completed successfully.

    Three out of four parallel tasks failed
  2. After you select the failed tasks, the Abort, and Retry buttons are enabled. The Continue button is not enabled because the failed tasks are mandatory.

    Failed tasks are selected
  3. After you resolve the cause of the failure and click Retry, the three failed tasks run in parallel again.

6.4.3 The Next Button Is Not Enabled During Configuration Assistants

Problem

On the Configuration Progress page of the installer, the Next button is enabled only when all configuration assistants are successful.

If you see that all your configurations are complete, and the Next button is not enabled, you encountered a configuration failure and continued to the next configuration assistant.

Solution

In this case, you must retry the failed configuration assistants by following these steps:

  1. On the Configuration Progress page of the installer, click Cancel.

  2. Restart the installer. All failed configuration assistants or steps rerun upon restart. For more information, see Section 6.4.1, "Restart a Failed Installer Session".

As long as a configuration assistant is not successful, the Next button remains disabled. It may be necessary to repeat the cancel and retry procedure until all configuration assistants are successful.

6.4.4 The OPSS Security Store Goes Down While the Installer is Running

Problem

The OPSS Security Store goes down while the installer is running.

Solution

Configuration tasks that are related to the OPSS Security Store will fail if the store goes down. Perform the following steps to recover:

  1. Abort the failed configuration task.

  2. Select Cancel to end the installer session.

  3. Start the OPSS Security Store. For more information, see "Starting and Stopping Oracle Internet Directory" in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Identity Management (Oracle Fusion Applications Edition).

  4. Start a new installer session. The installer resumes with the remaining tasks because you selected Cancel, which saves the session

6.4.5 Failure During Opening of Wallet Based Credential Store

Problem

The following error occurs during the configuration phase.

Reason:oracle.security.jps.service.credstore.CredStoreException: JPS-01050:
Opening of wallet based credential store failed. 
Reason java.io.IO Exception: PKI-02002: Unable to open the wallet. Check password.

Solution

After you resolve the cause of the failure, or cancel the installation and then restart the installer. If the failure still occurs, refer to "Server with NFS-Mounted Domain Directory Fails to Start" in the Oracle Fusion Middleware Application Security Guide to further diagnose the failure.

6.4.6 FAINSTALL-0006: RUP Part 1 of version 11.1.6.0.0 already installed

Problem

After the first installer completes successfully in silent mode, you switched to GUI mode for the second installer and the following error is reported:

FAINSTALL-0006: RUP Part 1 of version 11.1.6.0.0 already installed.

Solution

Start the second installer from the REPOSITORY_LOCATION/installers/fusionapps/Disk1 directory.

6.5 General Troubleshooting During the Configuration Phase in Silent Mode

The installer can be restarted to rerun all failed configuration tasks as well as those tasks that were not started from the previous session. When a mandatory configuration task or step fails in silent mode, the installer exits. After you resolve the issue that caused the failure, restart the installer using the same command you used to start it. When the installer restarts, it restarts from the first failed task.

If any non-mandatory tasks fail in silent mode, the installer continues with the next configuration task and does not exit. You must review the logs to find any non-mandatory tasks that failed and then rerun the installer until all tasks complete successfully.

If you decide to run the installer in GUI mode, you must start it from the REPOSITORY_LOCATION/installers/farup/Disk1/ directory.

6.6 Recovering From an Installer Session That Was Shut Down

Problem

An installer session was shut down during the upgrade.

Solution

Perform the following steps:

  1. Copy the checkpoint.xml file.

  2. Restore your backup of APPLICATIONS_BASE.

  3. Restore the checkpoint.xml file.

  4. Start from the beginning of the upgrade.

6.7 Troubleshooting Bootstrapping Patch Manager

Problem

An error occurred during the Bootstrapping Patch Manager configuration assistant.

Solution

An error during Bootstrapping Patch Manager normally occurs only when the database is down. Ensure that the database is up and running. You can review the related log files in this location:

FA_ORACLE_HOME/admin/FUSION/log/configlogs/FAPatchManager_bootstrap_timestamp.log

6.8 Troubleshooting Applying Middleware Patches

This section provides the following troubleshooting information related to the Applying Pre-PSA Middleware Patches or Applying Post-PSA Middleware Patches configuration assistants:

6.8.1 Log Files for Applying Middleware Patches

Problem

An error occurred during the Applying Pre-PSA Middleware Patches or Applying Post-PSA Middleware Patches configuration assistant.

Solution

Review the log file in the relevant location to find the cause of the error:

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/configlogs/ApplyPrePSAMiddlewarePatchestimestamp.log

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/configlogs/ApplyPostPSAMiddlewarePatchestimestamp.log

For specific OPatch failures, go to each of the individual Oracle home directories to find the details of the OPatch errors. For example, for a SOA failure, go to APPLICATIONS_BASE/fusionapps/soa/cfgtoollogs/opatch.

6.8.2 Applying Post-PSA Middleware Patches Hangs

Problem

The Applying Post-PSA Middleware Patches configuration assistant hangs.

Solution

This problem is most likely due to adpatch hanging as the result of the java worker not getting the database connection. You can resolve this issue by following the steps in Section 6.13, "Troubleshooting Loading Database Components". Run the commands from ATGPF_ORACLE_HOME instead of FA_ORACLE_HOME.

6.8.3 Error Applying Database Client Patches

Problem

The following error occurs:

OPatch cannot continue because it can't load library from the directory "<dbclient Oracle Home>/oui/lib/linux64" 

Solution

This error may occur if the OUI version in the database client Oracle home is 11.2 while the OUI version in Oracle Fusion Applications Oracle home (FA_ORACLE_HOME) is 11.1.

Perform the following steps to resolve this issue:

  1. Go to the database client home.

  2. Set the ORACLE_HOME environment variable to point to the database client Oracle home.

  3. Apply the database client patches using the following command:

    $ORACLE_HOME/OPatch/opatch apply patch_location
    
  4. Because the patches have now been manually applied, perform the following steps to continue with the upgrade:

    1. Go to the FA_ORACLE_HOME/fusionapps/applications/lcm/tp/config/RUP/FMW directory.

    2. Open the pre-psa-jobs.xml file for editing.

    3. Comment out the job with the name dbclient. An example of this job follows.

      <!-- <job>
          <id>10</id>
          <target>FAMW</target>
          <component>
                <name>dbclient</name>
                <version>11.1.1.5</version>
          <component>
          <utility_name>opatch</utility_name>
          <patch_number>NA</patch_number>
          <command>%opatch% napply -silent -skip_duplicate -skip_subset
      -oh %dbclient_home% -phBaseDir %dbclient_patch% -jre %jre_loc% -invPtrLoc
      %oraInstLocFile%</command>
           <patch_location>NA</patch_location
      </job>
      

6.9 Troubleshooting Failure During Propagating Domain Configuration

This section contains information about troubleshooting issues that may occur during the Propagating Domain Configuration configuration assistant. The following topics are discussed:

6.9.1 Monitor the Propagating Domain Configuration Assistant

You can monitor the progress of this configuration assistant by reviewing log files in this location:

APPLICATIONS_CONFIG/fapatch/admin/ruplitedomain/version/output/logs

6.9.2 Confirm the Configuration Assistant Was Successful

To confirm this configuration assistant was successful, verify that the config/fusionapps_start_params.properties file exists under each local or non-admin split domain. Also ensure that the bin/setDomainEnv.sh file under each local or non-admin split domain contains the following row:

POST_CLASSPATH="${COMMON_COMPONENTS_HOME}/modules/oracle.appstrace_11.1.1/appstrace.jar${CLASSPATHSEP}${POST_CLASSPATH}"
export POST_CLASSPATH

6.9.3 WARs or EARs Not Accessible From The Primordial Host

Problem

The Propagating Domain Configuration configuration assistant fails if there are WARs or EARs installed or deployed that are not accessible from the primordial host where RUP Installer is running. An example of the error caused by this condition follows:

<< read domain from
APPTOP/instance/domains/server.company.com/SCMDomain
<< write template to
APPTOP/instance/fapatch/admin/ruplitedomain/11.1.5.0.0/output/templates/SCMDomain.jar
>>  fail: Unable to locate file:
/fusionapps/localdomain/domains/server.company.com/SCMDomain/datalens/datalens.war
>>  fail: write template to
"APPTOP/instance/fapatch/admin/ruplitedomain/11.1.5.0.0/output/templates/SCMDomain.jar"

CFGFWK-60550:  Script execution aborted. The script may contain an error.
Unable to locate file:
/fusionapps/localdomain/domains/server.company.com/SCMDomain/datalens/datalens.war

Solution

To resolve this issue, you must undeploy or uninstall the WAR or EAR, which is datalens.war in this example. Then restart RUP Installer. After the upgrade has completed successfully, you can install or deploy the WAR or EAR.

6.10 Troubleshooting Upgrading Middleware Schema

Problem

An error occurred during the Upgrading Middleware Schema configuration assistant.

Solution

Review the log file in this location to find the cause of the error:

fusionapps/oracle_common/upgrade/logs/psatimestamp.log

Problem

The Upgrading Middleware Schema configuration assistant fails because JAVA_HOME cannot be found.

Solution

Set the JAVA_HOME and then manually run the upgrade for the failed schema, as shown in the following example:

export JAVA_HOME=/u01/APPLTOP/fusionapps/jdk6
/u01/APPLTOP/fusionapps/oracle_common/bin/psa -response /u01/APPLTOP/fusionapps/applications/admin/FUSION/oui_resp/psa_response_crm.txt

6.11 Troubleshooting Applying Downloaded Patches

Problem

The Applying Downloaded Patches configuration assistant failed with the following error:

Stack Description: java.lang.RuntimeException:
PatchObject constructor: Input file
"/net/server01/Downloaded_Patches/atgpf/patch/1234567/etc
/config/inventory" does not exist. 

Solution

This type of error occurs when you do not download the patches to the appropriate directory. To resolve this issue, copy the patches to the correct directory and retry the failed configuration assistant. For more information, see Section 2.1.4, "Download Mandatory Post-Release 6 Patches".

6.12 Failure During Granting Privileges

Problem

A failure occurred during either the Grant Privileges to Application Schemas or the Creating Grants/Synonyms on Application Database Objects configuration assistant.

Solution

You can find the cause of the failure by running the script manually as the sysdba user, using SQL*Plus or SQL*Developer. After you resolve the issue, click Retry in RUP Installer.

6.13 Troubleshooting Loading Database Components

This section contains information about troubleshooting issues that may occur during the Loading Database Components configuration assistant. Depending on the type of failure, you may need to review one or more of the log files in the following locations:

The following troubleshooting issues are described in this section:

6.13.1 Error While Loading Database Components

Problem

RUP Installer reports that one or more database workers failed during the Loading Database Components configuration assistant.

Solution

You must start AD Controller to manage the failed workers. After you resolve the issue that caused the workers to fail and you restart the failed worker, click OK in the dialog box and RUP Installer continues processing. For additional information, see "Troubleshooting Patching Sessions for Database Content" in the Oracle Fusion Applications Patching Guide.

6.13.2 Database Failure While Loading Database Components

Problem

Your database goes down while RUP Installer is running the Loading Database Components configuration assistant, and the options to Abort or Retry display. If you simply bring the database up and then click Retry, you may encounter the following error:

Failed to connect to the database as fusion with error: 
No more data to read from socket

Solution

Perform the following steps to recover from this error:

  1. Force the database patching session to fail.

    (Unix) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail 
    (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd forcefail
    
  2. Start AD Controller.

    (UNIX) FA_ORACLE_HOME/lcm/ad/bin/adctrl.sh
    (Windows) FA_ORACLE_HOME\lcm\ad\bin\adctrl.cmd
    

    For more information, see "Starting AD Controller" in the Oracle Fusion Applications Patching Guide.

  3. Follow this sequence of steps in AD Controller to manage the workers:

    1. Select Tell manager that a worker failed its job and enter All for all workers.

    2. Select Tell worker to quit and enter All for all workers. Note that this does not kill the workers. It sends a command to the worker to shutdown after it completes the current task.

    3. Wait for all workers to complete their tasks and shut down normally.

    4. If there are still some worker processes that do not shut down, kill those processes manually by selecting Tell manager that a worker failed its job. Then select Tell manager that a worker acknowledges quit and enter All for all workers.

    5. From your operating system, check for processes that are running fapmgr, javaworker, adpatch, adadmin, sqlplus, and adworker. If any exist, terminate them from your operating system.

    6. Select Tell worker to restart a failed job and enter All for all workers.

  4. Select Retry to restart RUP Installer.

6.13.3 Failure During AutoPatch Validation

Problem

AutoPatch validation fails with the following message:

An active adpatch or adadmin session was found. Complete or terminate the active session to allow fapmgr to proceed.

Solution

Perform the following steps to resolve this error:

  1. Run the fapmgr forcefail command to update the patching tables.

    (UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail [-logfile log file name] [-loglevel level]
    
    (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd forcefail [-logfile log file name] [-loglevel level]
    
  2. Run the fapmgr abort command from FA_ORACLE_HOME to find out if an Oracle Fusion Applications Patch Manager session must be cleaned up.

    (UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh abort [-logfile log file name] [-logLevel level]
    
    (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd abort [-logfile log file name] [-logLevel level]
    

    If this command finds no failed session, proceed to Step 3.

  3. Run the following commands from ATGPF_ORACLE_HOME to abandon any Applications Core patching sessions or AD Administration sessions that may be running:

    (Unix) ATGPF_ORACLE_HOME/lcm/ad/bin/adpatch.sh abandon=y interactive=n defaultsfile=ATGPF_ORACLE_HOME/admin/TWO_TASK/defaults.txt
    
    (Unix) ATGPF_ORACLE_HOME/lcm/ad/bin/adadmin.sh abandon=y interactive=n defaultsfile=ATGPF_ORACLE_HOME/admin/TWO_TASK/defaults.txt
    
    (Windows) ATGPF_ORACLE_HOME\lcm\ad\bin/adpatch.exe abandon=y interactive=n defaultsfile=ATGPF_ORACLE_HOME\admin\TWO_TASK\defaults.txt
    
    (Windows) ATGPF_ORACLE_HOME\lcm\ad\bin/adadmin.cmd abandon=y interactive=n defaultsfile=ATGPF_ORACLE_HOME\admin\TWO_TASK\defaults.txt
    

6.13.4 Flexfield Seed Data Upload Fails

Problem

When multiple seed data files are uploaded for the same flexfield but for different flexfield contexts, the upload tasks can fail due to locking issues. The failed tasks appear in the log file as the following error:

Loading failed with a JboException: JBO-25014: Another user has changed the
row with primary keyoracle.jbo.Key ...

Solution

AutoPatch defers any failed tasks to the end of the phase and reattempts the failed tasks only after attempting all tasks in the phase at least once. Usually, the flexfield seed data tasks that failed due to the locking issue succeed on subsequent attempts. You can ignore these errors if the flexfield seed data task succeeds on the retry. If the task remains in a failed state, you must use the AD Controller utility to retry the failed task.

For more information, see "Restarting a Failed Worker" in the Oracle Fusion Applications Patching Guide.

6.14 Troubleshooting Deployment of Applications Policies

This section contains information about troubleshooting issues that may occur during the Deploying Application Policies configuration assistant. Log files for this configuration assistant may be found in this location:

FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/fapatch_Deploying_Applications_Policies_(jazn-data.xml)_timestamp.log.

The following topics are discussed:

6.14.1 Failure During Deploying Applications Policies

Problem

A failure occurs during Deploying Application Policies.

Solution

When a failure occurs during Deploying Application Policies, you must restore only the stripe or system policy that has failed, from your backup. Use the OPSS migrateSecurityStore command with the appropriate source and destination arguments to perform the restore. Do not restore a stripe that has not failed. Review the JAZN deployment log file to determine the cause of the failure, fapatch_Deploying_Applications_Policies_(jazn-data.xml)_timestamp.log.

After you resolve the issue, restart RUP Installer by either selecting Retry in the same session or by exiting RUP Installer and restarting it.

For more information, see "Migrating with the Script migrateSecurityStore" in the Oracle Fusion Middleware Application Security Guide.

6.14.2 Warning During Deploying Applications Policies

Problem

The following warning occurs during Deploying Application Policies:

WARNING: Failed to validate the xml content. cvc-complex-type.2.4.a: Invalid
content was found starting with element 'property'. One of
'{"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":propertySetRef,
"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":extendedProperty,
"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":extendedPropertySetRef,
"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":serviceInstanceRef}'
is expected. Location: line 165 column 96.
WLS ManagedService is not up running. Fall back to use system properties for
configuration.

Solution

You can safely ignore this message as there is no functional impact of this warning and the deployment is successful.

6.14.3 Warning during Migrate Security Store

Problem

The following warning occurs during Deploying Application Policies:

FINE: Application policies already exists for application: fscm
oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException:
Cannot create application policy context "fscm".
        at
oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.unsync_createApp
licationPolicy(LdapPolicyStore.java:833)
        at
oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.createApplicatio
nPolicy(LdapPolicyStore.java:753)
        at
oracle.security.jps.internal.tools.utility.destination.apibased.JpsDstPolicy.c
lone(JpsDstPolicy.java:805) 

Solution

You can safely ignore this message as there is no functional impact of this warning and the deployment is successful.

6.14.4 IDM Server Failure During Deployment of Applications Policies

Problem

The IDM Server goes down during Deploying Application Policies and the deployment fails.

Solution

Even if the Retry button is enabled, RUP Installer does not allow a retry after this type of failure. You must instead click Cancel and restore from your Oracle Identity Manager backup. Then restart RUP Installer.

6.15 Troubleshooting Deployment of BI Publisher Artifacts

Problem

The following error occurs if the BI Presentation servers are running during the deployment of BI Publisher artifacts:

java.lang.RuntimeException: Webcat patch file creation failed! 

Solution

If you apply a release that contains BI Publisher artifacts, the BI Presentation servers must not be running. To resolve this issue, shut down the BI Presentation servers to release locks on the Oracle BI Presentation Catalog. For more information, see "fastartstop Syntax" in the Oracle Fusion Applications Administrator's Guide.

6.16 Troubleshooting Failure During Applying Offline Setting Changes

Problem

The Applying Offline Setting Changes configuration assistant fails during the "Update OID Authentication Provider Configuration" step. The log file shows that the installer fails while attempting to find OID Authenticator, as shown in the following example:

com.oracle.cie.domain.script.jython.WLSTException: com.oracle.cie.domain.script.ScriptException: No SecurityConfiguration!Realm!AuthenticationProvider object with nameProjectsDomain!myrealm!OIDAuthenticator

Solution

The workaround is to edit the checkpoint file to allow the Applying Offline Setting Changes configuration assistant to continue, and apply the required settings changes manually to the OVDAuthenticator after RUP Installer completes successfully.

Perform the following steps:

  1. Quit out of RUP Installer.

  2. Find the checkpoint.xml file located at APPLICATIONS_BASE/oraInventory/checkpoint/farup1/11.1.6.0.0/checkpoint.xml.

  3. Change the following element:

    <aggregate name="Applying Offline Setting Changes" status="fail">
    

    to:

    <aggregate name="Applying Offline Setting Changes" status="success">
    
  4. Restart RUP Installer. Select Yes when asked if you would like to continue with a previous installation.

  5. Manually apply the settings by performing the following steps for each failed domain:

    1. Edit the ovdUpdate.py script and change the _domainPath to point to your APPLICATIONS_BASE and domain path. The content of the script follows:

      _domainPath = '/APPTOP/instance/domains/fa-mycompany.com/HCMDomain'
      _ovdAuthenticatorPath = 'SecurityConfiguration/HCMDomain/Realm/myrealm/AuthenticationProvider/OVDAuthenticator'
       
      readDomain(_domainPath)
      cd(_ovdAuthenticatorPath) 
      
      set('ConnectTimeout',60)
      set('ResultsTimeLimit',300000)
      set('ParallelConnectDelay',1)
      set('IgnoreDuplicateMembership',1)
      set('UseRetrievedUserNameAsPrincipal',1)
       
      updateDomain()
      closeDomain()
       
      print 'OVDAuthProvider successfully updated'
      exit()
      
    2. Edit the script so that the "HCMDomain" in the _ovdAuthenticatorPath is changed to the domain that you are configuring. You can find the domains on which the OID configuration failed by reviewing the fapatch_Applying_Offline_Setting_Changes_timestamp.log file. Search for the error, "No SecurityConfiguration!Realm!AuthenticationProvider object with name", to find the domains.

    3. Start WLST.

    4. Run the script using the syntax, execfile(filePath/ovdUpdate.py), on those domains that failed the OID configuration.

6.17 Troubleshooting Failure During Verifying Node Manager and OPMN Status

Problem

The Verifying Node Manager and OPMN Status configuration assistant fails.

Solution

Do not cancel and exit out of RUP Installer in response to this configuration assistant failure. Perform the following steps to recover:

  1. Review the node manager log files to determine the cause of the failure:

    APPLICATIONS_CONFIG/nodemanager/host_name/

    Note that the APPLICATIONS_CONFIG value can be obtained from the APPLICATIONS_BASE/fusionapps/faInst.loc file.

  2. After you resolve the issue that caused the failure, start the Node Manager on all hosts that are part of the Oracle Fusion Applications provisioned system. For more information, see "Task 3: Start Node Manager" in the Oracle Fusion Applications Administrator's Guide.

  3. Restart the OPMN server for BI, GOP (if GOP is installed), and Web Tier. If you run the Web Tier (OHS) installed with the Oracle Fusion Applications middle tier, you can start it using the following steps. If you run the Web Tier on a separate machine, you may be able to run the steps below on the other machine. In either case, ensure that Web Tier (OHS) is up at this point.

    Example for BI: (note the usage of start instead of startall)

    cd APPLICATIONS_CONFIG/BIInstance/bin
    ./opmnctl start
    

    Example for GOP: (note the usage of start instead of startall) Note that the OPMN server for GOP should be started from the machine that hosts the Advanced Planning Managed server. Start the OPMN server for GOP only if you have GOP installed.

    cd APPLICATIONS_CONFIG/gop_1/bin
    ./opmnctl start
    

    Example for Web Tier: (note the usage of start instead of startall)

    cd APPLICATIONS_CONFIG/CommonDomain_webtier/bin
    ./opmnctl start
    

    For more information about the location of APPLICATIONS_BASE and APPLICATIONS_CONFIG, see Section 2.1.1, "Before You Begin".

    The BI and Web Tier processes managed by OPMN are started by RUP Installer in the Starting All Servers configuration assistant. The GOP processes managed by OPMN must be started using Fusion Applications Control, as described in Section 4.8, "Start GOP Processes", after RUP Installer completes.

  4. Fix any other environment issues before retrying the session. If RUP Installer exits for any reason, make sure that all node managers and OPMN processes are running. Contact Oracle Support Services to proceed out of this step if you have unresolved environment issues.

  5. After you start the services, click Retry to proceed to the Starting All Servers configuration assistant. If the starting of servers times out, see Section 6.18, "Troubleshooting Server Start and Stop Failures".

Note:

If GOP is not installed, the user interface reports "Success" for GOP OPMN status, but the log file reports: GOP is not provisioned. Skipping check for OPMN status.

6.18 Troubleshooting Server Start and Stop Failures

This section includes the following troubleshooting topics:

6.18.1 General Server Failure Due to Time Out Failures

Problem

A failure during the Starting All Servers configuration assistant typically happens when one of the servers times out and fails to start due to resource issues or application specific issues.

Solution

Various platforms and environment configurations can impact how long it will take all servers to start during the Starting All Servers configuration assistant. Although RUP Installer waits an average amount of time for this configuration assistant to complete before it is marked as Failed, different platforms may require more time. It is not unusual to receive timeout errors in the log files if the starting of all servers for your environment requires more time than RUP Installer allows. If this configuration assistant fails, follow these steps:

  1. Monitor the status of the servers by reviewing the messages in the server log files or on the console. The log file, FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/StartStop/fastartstop_timestamp.log, indicates which server started and failed to start.

    An example of messages for a server that timed out follows:

    Time out while performing Start for domain SCMDomain. Waited for 2400 seconds
    [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [UTIL] [oracle.apps.startstop.util.MbeanUtil: runSSCommandOnDomain.868] [tid:37] Start operation is completed for domain SCMDomain. Please see SCMDomain.log for details.
    
    [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [UTIL] [oracle.apps.startstop.invoke.StartStopTask: call.221] [tid:37] StartStopTask over for domain SCMDomain
    
    [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [SST] [oracle.apps.startstop.invoke.StartStopTask: call.223] [tid:37] Finished the task for the Domain SCMDomain
    
  2. Review the log files at the domain level to see a summary of the server status for that domain: FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/StartStop/domain name_timestamp.log.

  3. Review the corresponding server logs for the failed servers under the following directory: APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs.

  4. After you determine and resolve the cause of the failure, return to RUP Installer and click Retry.

  5. When all servers are up and running, including those that exceeded the timeout limit, click Abort in RUP Installer to move to the next configuration assistant.

6.18.2 Failure to Start BIServer

Problem

The following exception during the Starting all Servers configuration action indicates a failure in starting the BIServer:

Start all servers fails to start
Start operation on the component :coreapplication_obips1:, for the instance
:BIInstance: - FAILED

The coreapplication_obips1 server log file reports the following error:

ecid:]]
[2012-04-10T00:22:20.000-07:00] [OBIPS] [ERROR:16] []
[saw.security.odbcuserpopulationimpl.initialize] [ecid: ] [tid: ] Unable to
create a system user connection to BI Server during start up. Trying again.[[
File:odbcuserpoploaderimpl.cpp
Line:325
Location:
saw.security.odbcuserpopulationimpl.initialize
saw.catalog.local.loadCatalog
saw.subsystems.catalogbootstrapper.loadcatalog
saw.webextensionbase.init
saw.sawserver
ecid:]]
[2012-04-10T00:22:25.000-07:00] [OBIPS] [NOTIFICATION:1] [] [saw.sawserver]
[ecid: ] [tid: ] Oracle BI Presentation Services are shutting down.[[
File:sawserver.cpp
Line:706
Location:
saw.sawserver
ecid:

Solution

Perform the following steps to work around this issue.

  1. Select Retry, which shuts down and starts bi_server1.

  2. Monitor the fastartstop log files and the state of bi_server1(BIDomain).

  3. As soon as bi_server1 restarts, as indicated by a RUNNING status, start the component coreapplication_obiccs1 or all the components of type OracleBIClusterControllerComponent using opmnctl.

    Example syntax follows:

    */BIInstance/bin/opmnctl startproc ias-component=coreapplication_obiccs1
    

6.18.3 Startup Failed for CommonDomain:OHSComponent (Oracle VM Only)

Problem

The OHS diagnostic log contains the following error message:

No such file or directory:  Couldn't create accept lock

Solution

This issue could be the result of the hypervisors going down, resulting in bringing the Oracle VM servers down. Perform the following steps to resolve the error:

  1. Find the entry for the lock file in httpd.config, for example:

    LockFile "/u101/ohs_inst1/diagnostics/logs/OHS/ohs1/http_lock" 
    
  2. Confirm whether the directory that contains the lock file exists.

  3. Assuming this directory does not exist, create the directory.

6.19 EditTimedOutException Error During Online Preverification

Problem

The following error is reported during online preverification:

weblogic.management.mbeanservers.edit.EditTimedOutException  

Solution

You may have to manually release the edit session. For more information, see "Resolving an EditTimedOutException Error" in the Oracle Fusion Applications Patching Guide.

6.20 OAM Configuration Step Fails Due to Special Characters in Password

If the OAM administrator password contains special characters, such as '#' or '&', the OAM Configuration step will fail. To work around this issue, you can manually validate that the OAM Administration Server host/port and surname/password are correct. After you manually validate this information, you can proceed with the upgrade by clicking Continue.

Perform the following steps to validate.

  1. Get the OAM administrator user name and password from the credential store.

  2. Run APPLICATIONS_BASE/fusionapps/oracle_common/common/bin/wlst.sh.

  3. Run the following commands to connect to the Common Domain Administration Server and get the OAM administrator surname and password:

    connect()
    listCred(map='oracle.patching', key='FUSION_APPS_PATCH_OAM_ADMIN-KEY')
    
  4. Get the OAM Administration Server host and port from the following properties in APPLICATIONS_CONFIG/fapatch/FUSION_prov.properties:

    • OAM_ADMIN_SERVER_HOST

    • OAM_ADMIN_SERVER_PORT

  5. Use oamcfgtool.jar to confirm whether the OAM server can be invoked using the values found in the previous steps.

    cd APPLICATIONS_BASE/fusionapps/oracle_common/modules/oracle.oamprovider_11.1.1
    
    java -jar oamcfgtool.jar app_domain=crm web_domain=OraFusionApp
    uris_file=APPLICATIONS_BASE/fusionapps/applications/crm/security/oam.conf
    oam_aaa_mode=open_or_simple app_agent_password=password_for_map="oracle.patching" 
    key="FUSION_APPS_PATCH_OAM_RWG-KEY"_in_credential_store
    primary_oam_servers=oam_server1 oam_admin_server=http://OAM_admin_server_host:port
    oam_admin_username=username_for_FUSION_APPS_PATCH_OAM_ADMIN-KEY
    oam_admin_password=password_for_FUSION_APPS_PATCH_OAM_ADMIN-KEY
    oam_version=11 default_authn_scheme=FAAuthScheme
    
  6. If the previous command is successful, the validation is successful. Click Continue.

6.21 Merging SOA Composite JDeveloper Customizations During SOA Preverification

If you performed JDeveloper customizations to a SOA composite and you deployed the composite to the SOA runtime, RUP Installer reports an error during SOA Preverification, which instructs you to take the newer version of the composite that is in the release. You must then merge your customizations by performing the following steps.

  1. If any customizations are detected, the SOA Preverification results display the SOA composite name, its location in the FA_ORACLE_HOME/stripe/deploy directory, and the requirement for you to merge JDeveloper customizations into the sca_*.jar file in FA_ORACLE_HOME before proceeding with RUP Installer. The stripe in the directory path refers to crm, hcm, fscm, and so on.

  2. Open the custom SOA workspace and the customized version of the Fusion Applications SOA composite in JDeveloper using "Oracle Fusion Applications Developer". For more information, see "Customizing SOA Composite Applications with JDeveloper" in the Oracle Fusion Applications Web User Interface Developer's Guide for Oracle Application Development Framework.

  3. Import the composite sca_*.jar file from FA_ORACLE_HOME/stripe/deploy into the project, for example revision 11.1.6.0.0, in JDeveloper. Make note of this revision number in the deployment window because you will need it in Step 8.

  4. Restart JDeveloper in the Oracle Fusion Applications Administrator Customization role.

  5. Verify that there are no errors in JDeveloper.

  6. Verify that the changes introduced in both the customized version and the patched version are present.

  7. Right-click the composite project in the Application Navigator, select Deploy, select the composite, click Deploy to SAR, and then click Next.

  8. Manually change the value in New Revision ID to the revision from Step 3, for example, 11.1.6.0.0, and click Finish.

  9. If the deployment folder is set to a location different from that of the FA_ORACLE_HOME/stripe/deploy directory, copy and replace the JAR in the location mentioned in the error message of this SOA Composite. If your file name is different, rename it to the original name. You must copy the jar in the correct format to FA_ORACLE_HOME/stripe/deploy. For example if you have sca_ContractsDeliverablePurchaseDocAttrReadComposite_rev11.1.6.0.0.jar in JDeveloper, then you must copy it back to the FA_ORACLE_HOME/stripe/deploy directory as sca_ContractsDeliverablePurchaseDocAttrReadComposite.jar.

  10. To proceed with the installation, select Retry.

For more information about customizing SOA composites, see "Customizing and Extending SOA Components" in the Oracle Fusion Applications Extensibility Guide.

6.22 Location of GRC Policies in the OAM Applications Domain

The location of your Governance, Risk, and Compliance (GRC) policies varies, depending on your upgrade path to Release 6. GRC polices are located in the grc OAM application domain if your Oracle Fusion Applications environment was originally provisioned with either version 11.1.1.5 or 11.1.2, then upgraded to version 11.1.3, and then upgraded to version 11.1.4. If your environment was originally provisioned with version 11.1.3 and upgraded to version 11.1.4, your GRC policies are located in the fs OAM application domain.

6.23 Troubleshooting SOA Composite Deployment Failures

This section describes how to recover from failures during the Deploying SOA Composites configuration assistant. The following topics are described:

6.23.1 SOA Composite Log Files

The following log files are generated by the deployment of SOA composites:

  • Client side log files where individual domain logs reside: FA_ORACLE_HOME/admin/FUSION/log/fapatch/fapatch_11.1.6.0.0/soalogs

  • Log files for the failed domain:

    • APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/soa_server1.log

    • APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/soa_server1.out

    • APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/soa_server1-diagnostic.log

    • APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/AdminServer.log

6.23.2 Composite Revision is Already Deployed

Problem

A failure message such as, CFGEX-00062: Composite revision "default/composite name!11.1.6.0.0" is already deployed, occurs.

An example of a complete error message follows:

[2011-12-30T04:24:38.613-08:00] [apps]
[ERROR] [] [oracle.apps.CRMDomain] [tid: 58]
[ecid: 0000JIEvTHGEGR9ZvdYBV11EzMvF00000c,0]
CFGEX-00073 : SOA composite "/u01/APP_TOP/fusionapps/applications/crm/deploy/sca_ContractsTermLibTemplatesComposite.jar"
deployment failed for Domain "CRMDomain".[[
Action : See logs for details. oracle.as.install.
fapatchconfig.exception.PatchsetConfigException:
CFGEX-00073 : SOA composite "/u01/APP_TOP/fusionapps/applications/
crm/deploy/sca_ContractsTermLibTemplatesComposite.jar" deployment failed for Domain "CRMDomain".
      ….
Caused by: oracle.as.install.fapatchconfig.exception.PatchsetConfigException: CFGEX-00062 : Composite revision "default/
ContractsTermLibTemplatesComposite!11.1.6.0.0" is already deployed.

Solution

Normally, a failed SOA composite is undeployed by RUP Installer. However, if the failure of the deployment is due to SOA servers running out of memory, then RUP Installer cannot recover.To resolve this issue, you must manually undeploy the composite.

Note:

Ensure that you undeploy only the revision deployed by RUP Installer. Do not undeploy the previous version.

To undeploy, you can use WebLogic Server Tool (WLST) commands or Fusion Applications Control. For more information see Section 6.23.2.1, "Undeploy SOA Composites Using WLST Commands" and Section 6.23.2.2, "Undeploy SOA Composites Using Fusion Applications Control".

Then return to RUP Installer and select Retry.

6.23.2.1 Undeploy SOA Composites Using WLST Commands

Follow these steps to undeploy the composite using WLST commands:

  1. Start WLST:

    (Unix) APPLICATIONS_BASE/soa/common/bin/wlst.sh
    (Windows) APPLICATIONS_BASE\soa\common\bin\wlst.cmd
    
  2. Run the sca_undeployComposite command using the following syntax:

    sca_undeployComposite(serverURL, compositeName,
    revision, [user], [password], [partition])
    

    The variables have the following values:

    • serverURL contains the host and port of the SOA cluster Managed Server of the domain on which the SOA composite failed to deploy.

    • compositeName is the name of the composite to be undeployed.

    • revision, in the case of the Release 6, this should be 11.1.6.0.0 by default.

    Example:

    wls:/mydomain/ServerConfig>  sca_undeployComposite
    ("http://myhost10:7001",
    " ContractsDeliverablePurchaseAgrmntFlowComposite ", "11.1.6.0.0")
    

    You are prompted for the user name and password to execute the command.

    For more information, see "Oracle SOA Suite Custom WLST Commands" in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.

6.23.2.2 Undeploy SOA Composites Using Fusion Applications Control

Perform the following steps to undeploy the composite using Fusion Applications Control:

  1. In Fusion Applications Control, connect to the domain where the SOA composite failed to deploy. For more information, see "Starting Fusion Applications Control" in the Oracle Fusion Applications Administrator's Guide.

  2. Navigate to Farm_Domain->soa-infra->default.

  3. Locate the composite and revision, such as 11.1.6.0.0, as shown in this example:

    ContractsDeliverablePurchaseAgrmntFlowComposite [11.1.6.0.0]
    
  4. Right click on the composite and select SOA deployment > Undeploy.

6.23.3 Wsm-pm Application is not Running in Domain (Solaris Only)

Problem

The following error is reported during SOA Composite deployment on a Solaris platform:

CFGEX-00079 : Wsm-pm application is not running in domain "domain name"

You have already confirmed that the wsm-pm application is running on this domain.

Solution

Perform the following steps:

  1. Log in to the failed domain and check the health of all managed servers and deployed applications.

  2. Bounce all managed servers of the failed domains.

  3. Exit RUP installer.

  4. Restart RUP installer.

6.23.4 Manually Deploying SOA Composites

Problem

A customized SOA composite deployment fails during the upgrade, you must manually deploy this composite using WLST commands.

Solution

You must manually deploy a composite after a deployment failure.Perform the following steps to manually.

Perform the following steps. In this example, the composite, FinAp, is patched from revision 1.0 to revision 2.0 and the SAR file of revision 2.0 is in FA_ORACLE_HOME/crm/deploy/sca_FinAp_rev2.0.jar.

Note that the parameters are for illustration purposes only.

  1. Refer to the Diagnostics report to find the name and location of the sca_*.jar file that was copied to FA_ORACLE_HOME by Oracle Fusion Applications Patch Manager. For more information, see "Diagnostics Report" in the Oracle Fusion Applications Patching Guide.

  2. If the previous revision contained JDeveloper customizations, ensure that you deploy the patched revision with the merged JDeveloper customizations. Using the sca_*.jar file from Step 1, follow the JDeveloper customizations merge instructions that are described in Section 6.21, "Merging SOA Composite JDeveloper Customizations During SOA Preverification". Then use the merged sca_*.jar for Step 3.

  3. Deploy the patched composite using the single patch composite command.

    sca_patchComposite('SOA-Infra URL', user, password, 
    '/FA_ORACLE_HOME/crm/deploy//sca_FinAprev2.0.jar', mergeLogFile='/tmp/merge-log.txt')
    

6.24 Failure During IPM Import

Problem

The configuration assistant for importing IPM artifacts fails with the following error:

importIPMApplication() & importIPMInput() WLST commands have not run successfully

Solution

Follow the instructions in Steps 1 through 7 in "Prerequisites for the Deployment of IPM Artifacts" in the Oracle Fusion Applications Patching Guide. Then return to RUP Installer and select Retry.

6.25 Troubleshooting Health Checker Pre-Down Time Checks

If any plug-ins fail, Health Checker reports the failure in the log file and the HTML summary report, including the error message and suggested corrective actions. It then runs the remaining plug-ins. After all plug-ins have been attempted, Health Checker displays the location of the log file, which is APPLICATIONS_CONFIG/fapatch/logs/release_version/healthchecker/healthcheckplugin_timestamp.log. After you run Health Checker, review the log file or HTML summary. Log archives are stored in the FA_ORACLE_HOME/admin/FUSION/log/healthchecker/ARCHIVE directory.

After you resolve the issue that caused the error, start Health Checker again to run failed tasks. For more information, see Section 2.1.13, "Run Health Checker for Pre-Down Time Checks".

This section provides additional troubleshooting information for the following pre-down time plug-in failures:

6.25.1 Verify OPatch Version Fails

Problem

The check to verify the OPatch version fails because you are using a version of OPatch that is not compatible with Oracle Fusion Applications.

Solution

Download the compatible version of Opatch, which is available on My Oracle Support under patch 14044793.

6.25.2 Verify Credentials in Oracle Directory Services Manager (ODSM) Fails

Problem

Health Checker reports the following error message:

User 'cn=PolicyRWUser' is not member of cn=DirectoryAdminGroup

Solution

This error occurs if the cn=PolicyRWUser user is not part of cn=DirectoryAdminGroup. To resolve this issue, verify that the following credentials are present in ODSM by performing the following steps:

  1. Log in to Oracle Internet Directory using ODSM: http://ldap_host:port/odsm, for example, http://IDM_HOST:7005/odsm. (Note that you cannot do this using jexplorer.)

  2. Connect to a directory. Use the OID-OID connection, for example, where the User name is cn=orcladmin.

  3. Go to the Data Browser tab. Go to the cn=oracle internet directory and within the cn=oracle internet directory, go to cn=DirectoryAdminGroup.

  4. Verify that the following user entry is present in the Members section:

    cn=PolicyRWUser,cn=users,dc=us,dc=oracle,dc=com
    

    Note the value of cn is not case sensitive.

  5. If the entry is not present, click the add [+] button in the Members section and add the entry. Then apply the changes.

Verify ODSM screen

6.25.3 Verify Free and Total Memory Fails

Problem

The verification of the free and total memory fails.

Solution

Configure the memory to meet the requirements for the upgrade.

6.25.4 Verify Open File Limit Fails

Problem

The verification of the open file limit fails.

Solution

Follow the steps under "Increase the Open Files Limit" in the Oracle Fusion Applications Installation Guide.

6.25.5 Verify Host Names Fails

Problem

The verification of host names fails.

Solution

For Unix platforms, log in with root access, and ensure that all of the following requirements are met:

  1. The file, /etc/hosts, contains an entry for IP address 127.0.0.1, with the name localhost following it.

  2. The format of each host entry in /etc/hosts is:

    IP_address canonical_hostname [aliases}
    

    Note that the usage of aliases is optional.

  3. If the machine name is a logical host name and is different from the physical host name specified in the /etc/sysconfig/network file, then the entry for the logical host name must be listed before the physical host name in /etc/hosts.

    For more information, see "Edit Host Names (Unix)" in the Oracle Fusion Applications Installation Guide.

For Windows, perform the following steps:

  1. Right click on the computer name, which is on the desktop.

  2. Click on Properties, then Advanced System Settings, and then Computer Name.

  3. Ensure the entries in C:\Windows\System32\drivers\etc\hosts are correct.

6.25.6 Verify the Local Port Range Value Fails

Problem

The verification of the local port range fails.

Solution

To set the correct local port range on a Unix environment, log in as the root user and run the following command:

echo "32768 61000" > /proc/sys/net/ipv4/ip_local_port_range

Run the following commands for Solaris:

To view:

/usr/sbin/ndd /dev/tcp tcp_smallest_anon_port tcp_largest_anon_port

To modify:

/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 16202
/usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65535

6.25.7 Verify Oracle Homes Registration in Central Inventory Fails

Problem

The verification of Oracle homes registration fails.

Solution

Oracle Provisioning records installation information about the following Oracle homes separately from information about other products: Oracle Business Intelligence (Oracle BI), Oracle Global Order Processing (GOP), Web Tier, and Web Tier Common Oracle home. RUP Installer expects information about all products to be recorded in the same place. For more information about home directories, see "Provisioned Oracle Fusion Applications Home Directories" in the Oracle Fusion Applications Administrator's Guide.

The following steps describe how to manually register the all missing Oracle homes in central inventory. Use the appropriate step(s) to resolve the error reported in the Health Checker log file.

  1. Verify that the default Inventory Pointer file points to the central inventory on the primordial host on which RUP Installer runs. The default Inventory Pointer is in the following locations:

    • Unix: /etc/oraInst.loc

    • Solaris: /var/opt/oracle/oraInst.loc

    • Windows: located in the registry key, \\HKEY_LOCAL_MACHINE\\Software\Oracle\inst_loc

    Note:

    If the attachHome command hangs, see Section 6.25.7.1, "AttachHome Script Hangs".

  2. Run attachHome from the BI Oracle home, for example, APPLICATIONS_BASE/fusionapps/bi.

    (Unix) BI_HOME/oui/bin/attachHome.sh -jreLoc JAVA_HOME_LOCATION
    (Windows) BI_HOME\oui\bin\attachHome.bat -jreLoc JAVA_HOME_LOCATION
    
  3. Run attachHome from the GOP Oracle home, for example, APPLICATIONS_BASE/fusionapps/gop.

    (Unix) GOP_HOME/oui/bin/attachHome.sh -jreLoc JAVA_HOME_LOCATION
    (Windows) GOP_HOME\oui\bin\attachHome.bat -jreLoc JAVA_HOME_LOCATION
    
  4. Run attachHome from the Web Tier Oracle home, for example, APPLICATIONS_BASE/webtier_mwhome/webtier.

    (Unix) WEBTIER_HOME/oui/bin/attachHome.sh -jreLoc JAVA_HOME_LOCATION
    (Windows) WEBTIER_HOME\oui\bin\attachHome.bat -jreLoc JAVA_HOME_LOCATION
    
  5. Run attachHome from the Web Tier Common Oracle home, for example, APPLICATIONS_BASE/webtier_mwhome/oracle_common.

    (Unix) WEBTIER_COMMON_HOME/oui/bin/attachHome.sh -jreLoc JAVA_HOME_LOCATION
    (Windows) WEBTIER_COMMON_HOME\oui\bin\attachHome.bat -jreLoc JAVA_HOME_LOCATION
    
  6. Run attachHome from the Web Tier Webgate Oracle home, for example, APPLICATIONS_BASE/webtier_mwhome/webgate.

    (Unix) WEBTIER_WEBGATE_HOME/oui/bin/attachHome.sh -jreLoc JAVA_HOME_LOCATION
    (Windows) WEBTIER_WEBGATE_HOME\oui\bin\attachHome.bat -jreLoc JAVA_HOME_LOCATION
    
  7. Run attachHome from the Oracle Common Oracle home, for example, APPLICATIONS_BASE/fusionapps/oracle_common.

    (Unix) COMMON_HOME/oui/bin/attachHome.sh -jreLoc JAVA_HOME_LOCATION
    (Windows) COMMON_HOME\oui\bin\attachHome.bat -jreLoc JAVA_HOME_LOCATION
    
  8. Register the dependency between the BI Oracle home and Oracle Common Oracle home. If the runinstaller -updateHomeDeps command hangs, see Section 6.25.7.2, "The setup.exe -updateHomeDeps Command Hangs".

    Run Oracle Universal Installer with the -updateHomeDeps option and pass a dependency list. The syntax for the dependency list is:

    HOME_DEPENDENCY_LIST={ORACLE_HOME:DEPENDENT_ORACLE_HOME}
    

    Example for Business Intelligence:

    (Unix) BI_HOME/oui/bin/runInstaller -updateHomeDeps "HOME_DEPENDENCY_LIST=
    {APPLICATIONS_BASE/fusionapps/bi:APPLICATIONS_BASE/fusionapps/oracle_common}"
    -jreLoc JAVA_HOME_LOCATION
    
    (Windows) BI_HOME\oui\bin\setup.exe -updateHomeDeps "HOME_DEPENDENCY_LIST=
    {APPLICATIONS_BASE\fusionapps\bi:APPLICATIONS_BASE\fusionapps\oracle_common}"
    -jreLoc JAVA_HOME_LOCATION
    
  9. Register the dependency between Web Tier Oracle home and Web Tier Common Oracle home.

    (Unix) WEBTIER_HOME/oui/bin/runInstaller -updateHomeDeps "HOME_DEPENDENCY_LIST=
    {APPLICATIONS_BASE/webtier_mwhome/webtier:APPLICATIONS_BASE/webtier_mwhome/oracle_common}"
    -jreLoc JAVA_HOME_LOCATION
    
    (Windows) WEBTIER_HOME\oui\bin\setup.exe -updateHomeDeps "HOME_DEPENDENCY_LIST=
    {APPLICATIONS_BASE\webtier_mwhome\webtier:APPLICATIONS_BASE\webtier_mwhome\oracle_common}"
    -jreLoc JAVA_HOME_LOCATION
    
  10. Verify that the central inventory now contains the correct GOP, BI, and Web Tier information. Open the inventory.xml file from the ContentsXML subdirectory in your central inventory directory using a text editor. You can find your central inventory directory by looking in the default Oracle Inventory pointer file mentioned in Step 1. Verify that there are entries for GOP and for BI, and that the BI entry lists the Oracle Common dependency you specified in Step 6. Do the same for Web Tier information. Ensure that you do not modify inventory.xml in any way, as this may corrupt your system.

    Example entries in inventory.xml:

    <HOME NAME="OH1109401105" LOC="APPLICATIONS_BASE/fusionapps/gop" TYPE="O" IDX="11">
    <HOME NAME="OH198367808" LOC="APPLICATIONS_BASE/fusionapps/bi" TYPE="O" IDX="12">
       <DEPHOMELIST>
          <DEPHOME LOC="APPLICATIONS_BASE/fusionapps/oracle_common"/>
       </DEPHOMELIST>
    </HOME>
    <HOME NAME="OH987588708" LOC="APPLICATIONS_BASE/webtier_mwhome/webtier" TYPE="O" IDX="13">
       <DEPHOMELIST>
          <DEPHOME LOC="APPLICATIONS_BASE/webtier_mwhome/oracle_common"/>
       </DEPHOMELIST>
    </HOME>
    <HOME NAME="OH1271096710" LOC="APPLICATIONS_BASE/webtier_mwhome/oracle_common" TYPE="O" IDX="14">
       <REFHOMELIST>
          <REFHOME LOC="APPLICATIONS_BASE/webtier_mwhome/webtier"/>
       </REFHOMELIST>
    </HOME>
    

    Note:

    Rerunning the ATTACH_HOME command does not cause any issues.

6.25.7.1 AttachHome Script Hangs

If the attachHome script hangs, run attachHome with the following additional arguments: -waitforcompletion -nowait.

6.25.7.2 The setup.exe -updateHomeDeps Command Hangs

If the runInstaller or setup.exe -updateHomeDeps command hangs, run this command with the following additional arguments: -waitforcompletion -nowait.

6.25.8 Verify DBMS Stats Reports Schemas Fails

Problem

The verification for DBMS stats fails.

Solution

You must run DBMS_STATS on any schemas that are returned by Health Checker. Run the following statement in SQL*Plus as a privileged database user, such as, SYS.

execute dbms_stats.gather_schema_stats('<Schema Name>', cascade => true)

For more information see "Configuring Oracle Metadata Services" in the "Common" chapter of the Oracle Fusion Applications Post-Installation Guide.

For more information, see "Collecting Optimizer Statistics" in the Oracle Fusion Applications Administrator's Guide. This step optimizes the performance of starting servers.

6.25.9 Verify Flexfield Metadata Fails

Problem

The verification for flexfield metadata fails.

Solution

This plug-in fails if a flexfields metadata violation is reported. The reported violation indicates that the Extensible Flexfield has a UI Page defined that references a flexfield context which has not been associated with the corresponding category or any of its parent categories.

Run the following SQL script to delete the offending references:

DELETE 
FROM   fusion.fnd_ef_ui_page_task_flows PE 
WHERE  NOT EXISTS (SELECT 1 
 FROM fusion.fnd_ef_category_contexts 
 WHERE  context_code = PE.CONTEXT_CODE 
 AND category_code IN 
 (SELECT category_code 
 FROM fusion.fnd_ef_categories_b 
 WHERE ( hierarchy_label = 
 (SELECT category_hierarchy_label 
 FROM fusion.fnd_df_flexfields_b 
 WHERE descriptive_flexfield_code = 
PE.descriptive_flexfield_code)) 
START WITH category_code = PE.category_code CONNECT BY PRIOR 
parent_category_code = category_code)); 
 
COMMIT;

6.25.10 Check for Unusable Indexes Fails

Problem

The check for unusable indexes fails.

Solution

Rebuild the unusable index using the following command:

ALTER INDEX index_name REBUILD

6.25.11 Check for Library Cache Load Lock Fails

Problem

The check for library cache load lock fails.

Solution

Kill the session this is holding the lock.

6.25.12 Check for Repository Integrity Fails

Problem

The check for repository integrity reports files missing from the repository.

Solution

Contact Oracle Support to resolve this issue.

6.26 Troubleshooting Health Checker Down Time Checks

If any plug-ins fail, Health Checker reports the failure in the log file and the HTML summary report, including the error message and suggested corrective actions. It then runs the remaining plug-ins. Health Checker displays the location of the log file, which is APPLICATIONS_CONFIG/fapatch/logs/release_version/healthchecker/healthcheckplugin_timestamp.log, after all plug-ins have been attempted. Review the log file or HTML summary after you run Health Checker. Log archives are stored in the FA_ORACLE_HOME/admin/FUSION/log/healthchecker/ARCHIVE directory.

After you resolve the issue that caused the error, start Health Checker again to run failed tasks. For more information, see Section 2.2.6, "Run Health Checker for Down Time Checks".

This section provides additional troubleshooting information for the following errors reported by the pre-upgrade plug-in:

6.26.1 Found Active fapmgr Sessions

Problem

A Patch Manager (fapmgr) session is running or was previously interrupted.

Solution

Perform the following steps to forcefail and abandon the session:

  1. Use the fapmgr forcefail command to update the patching tables.

    (UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail [-logfile log file name] [-loglevel level]
    
    (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd forcefail [-logfile log file name] [-loglevel level]
    

    If the forcefail command returns "There are no active Oracle Fusion Applications Patch Manager sessions which can be forcibly failed", then skip the next step.

  2. Use the fapmgr abort command to abandon the session, only if a session is active.

    (UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh abort [-logfile log file name] [-loglevel level]
    
    (Windows) FA_ORACLE_HOME/lcm\ad\bin\fapmgr.cmd abort [-logfile log file name] [-loglevel level]
    

6.26.2 Found Traces of AutoPatch in FA_ORACLE_HOME

Problem

Health Checker found a file named adpinit.rf9 in FA_ORACLE_HOME, which indicates that AutoPatch is running or was previously interrupted.

Solution

If Health Checker also found an active fapmgr session, then the corrective action you take in Section 6.26.1, "Found Active fapmgr Sessions" should also resolve the traces of AutoPatch. If adpinit.rf9 is found and there are no active FAPMgr sessions, then you must manually remove adpinit.rf9, which is located in FA_ORACLE_HOME/admin/restart.

6.26.3 Found Traces of AD Administration in FA_ORACLE_HOME

Problem

Health Checker found a file named adainit.rf9 in FA_ORACLE_HOME, which indicates that AD Administration is running or was previously interrupted.

Solution

Follow the steps in Section 6.26.1, "Found Active fapmgr Sessions" to forcefail and abandon any FAPMgr sessions.

The perform the following steps from FA_ORACLE_HOME to abandon the AD Administration session:

(Unix) lcm/ad/bin/adadmin.sh abandon=y interactive=n defaultsfile=FA_ORACLE_HOME/admin/TWO_TASK/defaults.txt logfile=log_file_name

(Windows) lcm\ad\bin\adadmin.cmd abandon=y interactive=n defaultsfile=FA_ORACLE_HOME\admin\LOCAL\defaults.txt logfile=log_file_name

The TWO_TASK and LOCAL values can be obtained from the FUSION_env.properties file.

6.26.4 Found Traces of AutoPatch in ATGPF_ORACLE_HOME

Problem

Health Checker found a file named adpinit.rf9 in ATGPF_ORACLE_HOME, which indicates that AutoPatch is running or was previously interrupted.

Solution

Run the following command from ATGPF_ORACLE_HOME: (This is the directory under MW_HOME that contains the Applications Core code. For more information, see "Running Oracle Fusion Applications AutoPatch" in the Oracle Fusion Applications Patching Guide.)

(Unix) lcm/ad/bin/adpatch.sh abandon=y interactive=n defaultsfile=ATGPF_ORACLE_HOME/admin/TWO_TASK/defaults.txt logfile=log_file_name

(Windows) lcm\ad\bin\adpatch.cmd abandon=y interactive=n defaultsfile=ATGPF_ORACLE_HOME\admin\LOCAL\defaults.txt logfile=log_file_name

The TWO_TASK and LOCAL values can be obtained from the ATGPF_env.properties file.

If adpinit.rf9 is still found after performing the steps in this solution, then you must manually remove adpinit.rf9, which is located in ATGPF_ORACLE_HOME/admin/restart.

6.26.5 Found Traces of AD Administration in ATGPF_ORACLE_HOME

Problem

Health Checker found a file named adainit.rf9 in ATGPF_ORACLE_HOME, which indicates that AD Administration is running or was previously interrupted.

Solution

Run the following command from ATGPF_ORACLE_HOME: (This is the directory under MW_HOME that contains the Applications Core code. For more information, see "Running Oracle Fusion Applications AutoPatch" in the Oracle Fusion Applications Patching Guide.)

(Unix) lcm/ad/bin/adadmin.sh abandon=y interactive=n defaultsfile=ATGPF_ORACLE_HOME/admin/TWO_TASK/defaults.txt logfile=log_file_name

(Windows) lcm\ad\bin\adadmin.cmd abandon=y interactive=n defaultsfile=ATGPF_ORACLE_HOME\admin\LOCAL\defaults.txt logfile=log_file_name

The TWO_TASK and LOCAL values can be obtained from the ATGPF_env.properties file.

If adainit.rf9 is still found after performing the steps in this solution, then you must manually remove adainit.rf9, which is located in ATGPF_ORACLE_HOME/admin/restart.

6.26.6 Check for Processes That Are Not Complete

Problem

One of the following processes exists: adpatch, adadmin, adworker, oracle.apps.ad.worker.AdJavaWorker, oracle.apps.ad.fapmgr.FAPManager.

Solution

If the session for the process was already cleaned up, manually terminate the process from the operating system.

6.27 Troubleshooting Health Checker Post-Upgrade Checks

If any plug-ins fail, Health Checker reports the failure in the log file and the HTML summary report, including the error message and suggested corrective actions. It then runs the remaining plug-ins. Health Checker displays the location of the log file, which is APPLICATIONS_CONFIG/fapatch/healthchecker/healthcheckplugin_timestamp.log, after all plug-ins have been attempted. After you run Health Checker, review the log file or HTML summary. Log archives are stored in the APPLICATIONS_CONFIG/fapatch/healthchecker/ARCHIVE directory.

After you resolve the issue that caused the error, start Health Checker again to run failed tasks. For more information, see Section 4.17, "Run Health Checker for Post-Upgrade Checks".

6.28 Ignorable Errors Reported by catbundle.sql

The following ignorable errors may be encountered while running the catbundle.sql script or its rollback script:

ORA-29809: cannot drop an operator with dependent objects

ORA-29931: specified association does not exist

ORA-29830: operator does not exist

ORA-00942: table or view does not exist

ORA-00955: name is already used by an existing object

ORA-01430: column being added already exists in table

ORA-01432: public synonym to be dropped does not exist

ORA-01434: private synonym to be dropped does not exist

ORA-01435: user does not exist

ORA-01917: user or role 'XDB' does not exist

ORA-01920: user name '<user-name>' conflicts with another user or role name

ORA-01921: role name '<role name>' conflicts with another user or role name

ORA-01952: system privileges not granted to 'WKSYS'

ORA-02303: cannot drop or replace a type with type or table dependents

ORA-02443: Cannot drop constraint - nonexistent constraint

ORA-04043: object <object-name> does not exist

ORA-29832: cannot drop or replace an indextype with dependent indexes

ORA-29844: duplicate operator name specified

ORA-14452: attempt to create, alter or drop an index on temporary table already in use

ORA-06512: at line <line number>. If this error follow any of above errors, then can be safely ignored.

ORA-01927: cannot REVOKE privileges you did not grant

6.29 Performing Installation Verification Steps

Perform the steps in "Verifying Installation" in the Oracle Fusion Applications Post-Installation Guide.