| Oracle® Fusion Applications Upgrade Guide 11g Release 7 (11.1.7) Part Number E35833-18 |
|
|
PDF · Mobi · ePub |
| Oracle® Fusion Applications Upgrade Guide 11g Release 7 (11.1.7) Part Number E35833-18 |
|
|
PDF · Mobi · ePub |
This chapter provides information to assist you in troubleshooting upgrade issues.
This chapter contains the following topics:
When Upgrade Orchestrator exits with a failure on any upgrade task, it sends an email to the recipients specified in the EMAIL_TO_RECIPIENT and EMAIL_CC_RECIPIENT properties in the pod.properties file. This email contains the Oracle Fusion Applications Orchestrator Upgrade Report as an attachment. The report name is FAOrchestrationUpgradeReport_release_hosttype_hostname_timestamp.html. This report specifies the location of the Fusion Applications Orchestration Action Summary report, which provides information about the failure, corrective action, and relevant log files. The orchestration log file is a good point to start for any troubleshooting, as it captures logs from different upgrade tasks as well as console messages. The orchestration log file is located in APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/orchestration/host_name-rel5-7_hosttype_timestamp.log.
Figure 6-1 depicts the high level flow for troubleshooting Upgrade Orchestrator failures.
Figure 6-2 depicts the reports and log files generated by Upgrade Orchestrator.
Previous reports are archived whenever a new report is generated, as described in Section 6.6.3, "Unable to Find the Orchestrator Upgrade Report After Failure". For more information about the report, see Section 1.3.4, "Oracle Fusion Applications Orchestrator Upgrade Report".
The following types of log files are described in this section:
The following table contains a list of log directories for Upgrade Orchestrator activities. Note that for the chained Release 5 to 6 to 7 upgrade, the log directories for Upgrade Orchestrator activities are logged under 11.1.7.0.0 even when you are running the Release 5 to Release 6 upgrade. If you want to use a different location for log directories, provide the location in the LOG_LOCATION property, which exists in the IDM.properties and OHS.properties files. For more information, see Appendix B, "Upgrade Orchestrator Properties Files".
Table 6-1 Log Directories for Upgrade Orchestrator Activities
| Log directory name | Description |
|---|---|
|
|
HTML files on the primordial and midtier hosts. |
|
|
Log files on the primordial and midtier hosts. |
|
|
HTML files on the OHS, IDM, and database hosts. |
|
|
Directory for archived log files |
|
Under the
Under the
|
Individual log files for the Health Checker plug-ins that run in parallel, which are stored in separate directories. |
The following table contains a list of log directories for RUP Installer activities.
Table 6-2 Log Directories for RUP Installer Activities
| Log directory name | Description |
|---|---|
|
|
Installation phase and Oracle Fusion Middleware patch set installation logs. |
|
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP |
Top level directory for RUP Installer logs. |
|
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/ARCHIVE/ |
Top level log directory where log files are moved when you retry the installation session. |
|
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/configlogs |
Top level log directory for configuration assistants. A log file exists for each configuration assistant. For more information, see Section 6.2.2.1, "Log Files for Configuration Assistants". |
|
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/PatchManager_DBPatch |
Database upload configuration assistant logs after failure or completion. For more information, see Section 6.2.2.2, "Log Files for the Database Upload Configuration Assistant". |
|
|
Oracle Business Intelligence logs. |
|
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/StartStop |
StartStop utility logs. Note that server logs are located under respective domains. For example, the AdminServer log for CommonDomain is under |
|
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/soalogs |
SOA artifacts configuration assistant logs. Note that SOA server logs are located under respective domains. For example, the SOA server logs for CommonDomain are under |
|
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/PatchManager_DownloadedPatches |
Applying Downloaded Patches configuration assistant logs. |
During the configuration phase of the upgrade, each configuration assistant creates its own log file under the APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/configlogs directory. All messages that are generated during the configuration assistant processing are written to this log file. The only information related to configuration assistants that is written to the main log file, APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP, are those messages that indicate that a configuration assistant started and the result of its processing, such as success or error.
During the execution of the database upload configuration assistant, log files are created under the /lcm/logs directory. Upon completion or failure of the database upload, the log files move to the APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/PatchManager_DBPatch directory.
Release 6 log files are located in this directory:
APPLICATIONS_BASE/fusionapps/applications/admin/FUSION/log/fapatch/fapatch_11.1.6.0
If you are performing a chained upgrade from Release 5 to Release 6 to Release 7, before starting the upgrade to Release 7, the Release 6 logs will move to this archive location:
APPLICATIONS_CONFIG/lcm/archive/log/fapatch/fapatch_11.1.6.0.0
The log file for running RUP Lite for OVM in pre-root mode is in the following location:
/u01/lcm/rupliteovm/output/logs/11.1.7.0.0/host_name/ruplitepre-root.log
Log files for running RUP Lite for OVM in post-root mode are in the following locations:
For the Release 5 to Release 6 hop:
/u01/lcm/rupliteovm/output/logs/11.1.6.0.0/host_name/ruplitepost-root.log
For the Release 6 to Release 7 hop:
/u01/lcm/rupliteovm/output/logs/11.1.7.0.0/host_name/ruplitepost-root.log
You can monitor the progress of the upgrade by monitoring the console output or by running the getStatus command. You can run this command on any host, to get the upgrade status for that host or for other hosts. The command follows:
(Unix) cd ORCH_LOCATION/bin ./orchestration.sh getStatus -pod POD_NAME -hosttype host_type -hostname host_name -release release_version [-phase phase_name] [-taskid task_id] [-taskstatus task_status] (Windows) cd ORCH_LOCATION\bin orchestration.cmd getStatus -pod POD_NAME -hosttype host_type -hostname host_name -release release_version [-phase phase_name] [-taskid task_id] [-taskstatus task_status]
Example 6-1 Retrieve the overall status of the upgrade
./orchestration.sh getStatus -pod fcsm -hosttype PRIMORDIAL -hostname host_name -release REL7
Example 6-2 Retrieve all tasks in a phase
./orchestration.sh getStatus -pod fcsm -hosttype PRIMORDIAL -hostname host_name -release REL7 -phase predowntime
Example 6-3 Retrieve all tasks with a specific status
./orchestration.sh getStatus -pod fscm -hosttype PRIMORDIAL -hostname host_name -release REL7 -taskstatus success
Example 6-4 Retrieve the status of a specific task
./orchestration.sh getStatus -pod fscm -hosttype PRIMORDIAL -hostname host_name -release REL7 -taskid HostTypeValidatePlugin
Table A-3 displays a complete list of options for the orchestration.sh getStatus command.
If any upgrade tasks fail, the Orchestrator Upgrade Report is generated and mailed as an attachment to the user specified in the EMAIL_TO_RECIPIENT property in the pod.properties file. For more information, see Section 1.3.4, "Oracle Fusion Applications Orchestrator Upgrade Report". For information about troubleshooting failures, refer to the appropriate section in Chapter 6, "Monitoring and Troubleshooting the Upgrade" to resolve the issue. After a failure, restart Orchestrator on the host where it failed, using the same commands you used in Section 4.1.2, "Run Upgrade Orchestrator During Down Time".
If any configuration assistants fail while RUP Installer is running, Upgrade Orchestrator does not display a message, fail, or send an email until RUP Installer exits with a failure.
If the Loading Database Components step in RUP Installer fails, you receive an email notification only when all workers are in a FAILED or IDLE (no tasks assigned to it) state. To resolve this type of issue, follow the steps in Section 6.13.1, "Error While Loading Database Components Using Orchestration".
Orchestration can be terminated on all hosts under scenarios that require you to issue an exit command across the entire environment. This section describes the commands to use to terminate orchestration on all hosts.
When you need to terminate an orchestration session on a pod for reasons such as, not being able to complete the upgrade within a certain time, or unexpected issues that may require significant time to resolve, run the following command:
cd /ORCH_LOCATION/bin ./orchestration.sh exitOrchestration –pod POD_NAME –hosttype host_type
The exitOrchestration command terminates the orchestration session on all hosts across all host types in the specified pod. This command can be run from any individual host for the entire environment and/or pods. The hosttype argument must match the host from which you issue this command. Upgrade Orchestration sends a notification after all hosts exit from orchestration. After you receive this notification, you must run the following command to clear the exit status on all hosts:
cd /ORCH_LOCATION/bin ./orchestration.sh clearExitOrchestration -pod POD_NAME –hosttype host_type
After the clearExitOrchestration command runs, you can continue with the upgrade or take other appropriate actions on the pod.
Note:
From the time exitOrchestration is issued, until clearExitOrchestration is issued, no other command can be issued on the pod. Also, exitOrchestration can be issued from any host but is applicable for all the hosts under a pod.
To cancel the upgrade and restore the system, first terminate orchestration by following the steps in Section 6.4, "Terminate Upgrade Orchestration". After orchestration terminates successfully, restore the system from the backup that was taken before starting the upgrade. In addition to restoring the environment from the backups, perform the following steps to restore and clean up the orchestration files.
Directories configured for the following properties in pod.properties are used by Upgrade Orchestrator to store checkpoint files and to archive older versions of checkpoint files.
ORCHESTRATION_CHECKPOINT_LOCATION
ORCHESTRATION_CHECKPOINT_ARCHIVE_LOCATION
Note that if these configured directories are shared among multiple instances then, orchestration would have created POD_NAME sub directories.
Run the following commands to remove any checkpoint location and its contents:
rm -rRf ORCHESTRATION_CHECKPOINT_LOCATION/POD_NAME/*
rm -rRf ORCHESTRATION_CHECKPOINT_ARCHIVE_LOCATION/ARCHIVE/POD_NAME/*
Ensure that /u02/instance/CommonDomain_webtier_local remains a mount point. If you are restoring the environment from a backup, also restore the /u02/instance directory by running the following commands:
rm -rf /u02/instance/CommonDomain_webtier_local mkdir /u02/instance/CommonDomain_webtier_local su mount | grep u01
The mount command finds the device that contains the /u01 directory. An example of the output from this command follows:
host_name:/export/fusion_imf/appohs_imf/u01 on /u01 type nfs (rw,addr=11.222.333.444)
Use the device name returned by the mount command, and run the following command:
mount -t nfs device_name
An example of this command follows:
mount -t nfs host_name:/export/fusion_imf/appohs_imf/u02/instance/CommonDomain_webtier_local /u02/instance/CommonDomain_webtier_local
exit
The following command provides an example for copying files from the backup instance to the active instance directory:
cp -rf /u02/instance_backup_location/CommonDomain_webtier_local /u02/instance/CommonDomain_webtier_local
This section describes the following specific Upgrade Orchestrator troubleshooting scenarios:
Unable to Find the Orchestrator Upgrade Report After Failure
Orchestration Fails to Generate Report With an Out Of Memory Error
Invalid property: must specify ORCHESTRATION_CHECKPOINT_LOCATION
Upgrade Orchestrator Does Not Use a Value in the Properties File
Informatica Identity Resolution (IIR) Does Not Come Up After the Upgrade
Problem
When orchestration is relaunched due to any reason, there could be an error loading checkpoint files to the appropriate location. In this case, Upgrade Orchestrator exits with the following errors.
Unable to upload orchestration checkpoints under /fsnadmin/upgrade/fusionChangeOps/11.1.7.0.0/orchestration/bin/../checkpoint. Corrective Action: Remove any existing files from older Orchestration run in /fsnadmin/upgrade/fusionChangeOps/11.1.7.0.0/orchestration/bin/../checkpoint before you proceed.
Solution
Perform the required corrective action suggested in the error message and then resume Upgrade Orchestrator to proceed with the upgrade.
Problem
Orchestration hangs during the preDowntime or downtimeFA phase, or you need to exit Upgrade Orchestrator in the middle of an upgrade for any valid reason.
Solution
Run the exitOrchestration command from another console, on any host in the pod, to gracefully exit orchestration. Then run clearExitOrchestration. For more information, see Section 6.4, "Terminate Upgrade Orchestration".
The exitOrchestration command terminates the upgrade on all hosts. Therefore, after you resolve the issue, rerun orchestration on all hosts where orchestration was previously running.
Problem
After Upgrade Orchestrator fails, the console reports the following example information:
Fusion Applications Orchestrator Upgrade Report: /u01/orchestration/orchreports/FAOrchestrationUpgradeReport_release_hosttype_hostname_timestamp.html
This html file does not exist in the /u01/orchestration/orchreports directory.
Solution
As the upgrade progresses, the Orchestrator Upgrade report is archived after the failure or completion of each task. You can find the output in the following directory, based on the example.
/u01/orchestration/orchreports/ARCHIVE/11.1.6.0.0/2013-05-13_11_38-48AM/FAOrchestrationUpgradeReport_release_hosttype_hostname_timestamp.html
Problem
Upgrade Orchestrator fails while generating the Upgrade Orchestrator report with the following error:
"Java.lang.OutOfMemoryError: PermGen space
Solution
Increase the ORCH_JVM_OPTION value in pod.properties to allocate more memory for both the startup of JVM and the total size of permgen, as shown in the following example:
ORCH_JVM_OPTION=-Xmx2048m -XX:PermSize=256M -XX:MaxPermSize=512M
Problem
Property validation fails during the PreDowntime phase with the following error:
Invalid property: must specify ORCHESTRATION_CHECKPOINT_LOCATION in orchestration properties file ./../config/pod/pod_properties.
No log file or HTML file is generated.
Solution
Populate the ORCHESTRATION_CHECKPOINT_LOCATION mandatory property in the pod.properties file. Note that no logs are generated for this type of failure by design.
Problem
You ran the updateStatus command to manually set the status of a failed task_id on the primordial host to "success" for the PreDowntime phase. After you resume orchestration on the IDM host, it fails with the following error:
Wait for peer phase: PRIMORDIAL:PreDowntime on host.mycompany.com Found peer phase: PRIMORDIAL:PreDowntime on host.mycompany.com Error.
The results of getStatus on the pod shows that all tasks were successful but the PreDownTime phase was in error status.
Solution
Setting a task status to "success" does not resolve a "Wait for peer phase" error, because a phase level status cannot be updated by the updateStatus command. The only way to resolve a "Wait for peer phase" issue is to resume orchestration so that it can verify that all tasks in the phase were successful.
Problem
An orchestration task is no longer running and the following error is reported:
Orchestration step: DowntimePreFA DeploySoaShared Running Unable to update task status from Running to Success Oracle Fusion Applications Release Upgrade Orchestration failed.
Solution
Before performing the step in this solution, confirm that there are no orchestration processes running. Then run the updateStatus command to change the status of the task specified in the error message to error and then resume Upgrade Orchestrator.
Upgrade Orchestrator supports only the following status transitions:
Error to Success
Running to Error
ManualStep to Success
Success to Error
Problem
The emails that Upgrade Orchestrator sends upon failure are not being received.
Solution
Perform the following steps to check if your mail server is configured properly:
You can check if your mail server is configured properly by running the following command:
"echo hello | /usr/sbin/sendmail <email_addr>"
If emails are not being sent, restart the mail server and test again.
/etc/init.d/sendmail restart
Ensure that all properties related to email are populated with the correct values in the pod.properties file. For more information, see Table B-1, "pod.properties".
Problem
Upgrade Orchestrator is not using a value that was recently added to one of the properties files.
Solution
If you updated the properties file after launching Upgrade Orchestrator, follow the steps to safely exit orchestration in Section 6.6.2, "Safely Exit Upgrade Orchestrator" and then relaunch orchestration.
Problem
While running various commands for Upgrade Orchestrator, the following error is reported:
Stale NFS file handle
Solution
If the Stale NFS file handle error is reported while running any of the plug-ins in orchestration or the getStatus or updateStatus commands, verify that all mount points provided in the various property files are actually accessible. For more information, see Appendix B, "Upgrade Orchestrator Properties Files".
Problem
The following error causes Upgrade Orchestrator to fail:
rm: cannot remove `/u01/ORCH/orchestration/INIT/mycompany.com/IDM/INIT/snapshot/.nfs00000000015595b30000004b': Device or resource busy Oracle Fusion Applications Release Upgrade Orchestration failed.
Solution
Remove the file that is causing the error and restart Upgrade Orchestrator.
Problem
IIR does not come up after following the steps to start IIR as part of the Start External Servers Pause Point Post Upgrade step.
Solution
Follow the steps in "Troubleshooting Informatica Identity Resolution and Data Quality Setup" in the "Define Data Quality" chapter of the Oracle Fusion Applications Customer Data Management Implementation Guide to manually check for files that need to be cleaned up and to retry the steps to start the server.
Problem
The following error is reported:
[apps] [ERROR] [] [oracle.apps.ad.rupconfig.Creating_Middleware_Schemas] from oracle.security.audit.config.dynamic.persistence.internal.ldap.AuditStoreDataManager searchFilterPresets
Solution
This error can be ignored.
This section provides information about the following RUP Installer failures:
General Troubleshooting During the Configuration Phase in GUI Mode
General Troubleshooting During the Configuration Phase in Silent Mode
RUP Installer or InstallFaSaaSLcmTools Fails Due To Thread Calls
RUP Installer or InstallFaSaaSLcmTools Fails With Internal Error
Failure During Importing Oracle Data Integrator Repositories
Deploying New Application Configuration Fails with a "NumberFormatException"
Configuration Assistant Fails With "Could not create credential store instance"
Failure During Starting All Servers or Restarting SOA Servers
This section describes the following troubleshooting scenarios related to the configuration phase in GUI mode:
The Next Button Is Not Enabled During Configuration Assistants
The OPSS Security Store Goes Down While the Installer is Running
The installer can be restarted to rerun all failed configuration assistants as well as those configuration assistants that were not started from the previous session. When a configuration assistant or step fails, the Configuration Progress screen displays the location of the log file and the exception that caused the failure. You can also view the content of the log files that appear at the bottom of the screen to obtain detailed information to assist in diagnosing the cause of the failure.
If one or more failures occur during the configuration phase, after the final configuration assistant is complete, the following message displays:
Configuration is completed with errors, exit the installer by clicking the 'Cancel' button and retry the failed configurations.
Perform the following steps to rerun the installer and retry the failed configuration assistants:
Click Cancel to exit the installer.
Resolve the issues that caused the failure.
Start the installer using the same command syntax that you used for the previous incomplete installation.
A pop up dialog displays, asking if you want to continue the previous incomplete installation. Select Yes to continue running the previous session. If you select No, the installer starts from the beginning and it will fail, indicating that a release cannot be installed again in the same environment. You would then need to restore from your backup and restart the installer.
The Configuration Progress screen displays only the failed and remaining configuration assistants, and then runs these configuration assistants.
Assuming all configuration assistants complete successfully, click Next to go to the Installation Complete screen and then click Finish to end the session. If a configuration assistant fails again and you want to attempt to run the session again, click Cancel to save the session. If all configuration assistants were successful for the first installer, the second installer launches automatically. If all configuration assistants completed successfully, click Finish to end the session.
Note that Language Pack runs only one installer.
If one or more tasks in a group fail, you can select the failed tasks in any combination, and the Abort, Retry, and Continue buttons are enabled as appropriate for the selected tasks. For example, if two tasks in a group fail, and the first task allows you to select Continue, but the other task does not, then the Continue button is not enabled if you select both tasks.
You can process one or more failed tasks at a time. For example, if three tasks fail, you can retry one of them, and while it is running, you can abort the second task. Then you can retry the third task. When the first and third tasks finish processing, the next step depends on whether the second task is mandatory. If it is a mandatory task, the installer stops, and if it is non-mandatory the installer continues with the next task after the group. You can also pick two out of three or all three tasks and select Retry, Abort, or Continue, based on which buttons are enabled.
Note that all tasks in a group must either fail or complete successfully before the Cancel button is enabled.
The following example depicts a group of four configuration tasks that are running in parallel and three of the four tasks fail.
Four tasks were running in parallel. Three tasks fail and the remaining task is successful. Note that the Abort, Retry, and Continue buttons are not enabled because the check boxes for the failed tasks are not checked. In the case of failure, the check boxes are enabled for failed tasks only after all tasks in the group have either failed or completed successfully.
After you select the failed tasks, the Abort, and Retry buttons are enabled. The Continue button is not enabled because the failed tasks are mandatory.
After you resolve the cause of the failure and click Retry, the three failed tasks run in parallel again.
Problem
On the Configuration Progress page of the installer, the Next button is enabled only when all configuration assistants are successful.
If you see that all your configurations are complete, and the Next button is not enabled, you encountered a configuration failure and continued to the next configuration assistant.
Solution
In this case, you must retry the failed configuration assistants by following these steps:
On the Configuration Progress page of the installer, click Cancel.
Restart the installer. All failed configuration assistants or steps rerun upon restart. For more information, see Section 6.7.1.1, "Restart a Failed Installer Session".
As long as a configuration assistant is not successful, the Next button remains disabled. It may be necessary to repeat the cancel and retry procedure until all configuration assistants are successful.
Problem
The OPSS Security Store goes down while the installer is running.
Solution
Configuration tasks that are related to the OPSS Security Store will fail if the store goes down. Perform the following steps to recover:
Abort the failed configuration task.
Select Cancel to end the installer session.
Start the OPSS Security Store. For more information, see Section E.7, "Starting and Stopping Oracle Internet Directory".
Start a new installer session. The installer resumes with the remaining tasks because you selected Cancel, which saves the session
Problem
The following error occurs during the configuration phase.
Reason:oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IO Exception: PKI-02002: Unable to open the wallet. Check password.
Solution
After you resolve the cause of the failure, or cancel the installation and then restart the installer. If the failure still occurs, refer to "Server with NFS-Mounted Domain Directory Fails to Start" in the Oracle Fusion Middleware Application Security Guide to further diagnose the failure.
The installer can be restarted to rerun all failed configuration tasks as well as those tasks that were not started from the previous session. When a mandatory configuration task or step fails in silent mode, the installer exits. After you resolve the issue that caused the failure, restart the installer using the same command you used to start it. When the installer restarts, it restarts from the first failed task.
If any non-mandatory tasks fail in silent mode, the installer continues with the next configuration task and does not exit. You must review the logs to find any non-mandatory tasks that failed and then rerun the installer until all tasks complete successfully.
If you decide to run the installer in GUI mode, you must start it from the REPOSITORY_LOCATION/installers/farup/Disk1/ directory.
RUP Installer is one of the tasks performed by Upgrade Orchestrator. In the case of a failure, information in Section 6.1, "General Troubleshooting for an Upgrade Orchestrator Failure" applies. In addition to the Upgrade Report and log location, the RUP Installer Report location is also included as part of the notification that is sent. For more information, see Section 5.2, "Review the Post RUP Installer Report".
Problem
RUP Installer fails due to thread calls and reports errors similar to the following example:
"Thread-11" id=29 idx=0x98 tid=25751 prio=5 alive, native_blocked
at java/io/UnixFileSystem.getBooleanAttributes0(Ljava/io/File;)I(Native
Method)
at java/io/UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
at java/io/File.exists(File.java:733)
Solution
Restart Orchestrator to proceed with upgrade. If it fails again, contact Oracle Support.
Problem
RUP Installer or InstallFaSaaSLcmTools fails with the following message:
Internal Error: File Copy failed. Aborting Install
Solution
Restart Orchestrator to proceed with upgrade. If it fails again, contact Oracle Support.
Problem
Because of intermittent issues, RUP Installer Part 1 fails on the primordial host due to the following error:
/exception in /u01/instance/lcm/log/11.1.7.0.0/RUP/fapatch_Applying_Middleware_Patchsets_timestamp.log.
{{Internal Error: File Copy failed. Aborting Install
ERROR: Failed Job ID 12
ERROR: jobs failed during Applying Middleware Patchsets}}
{{ERROR: CFGEX-00090 : Applying Middleware Patchsets failed.Action : See log file for details. java.lang.Exception: CFGEX-00090 : Applying Middleware _Patchsets failed. at_ _oracle.as.install.fatechpatchconfig.MWPatchset.doExecute
(MWPatchset.java:53) a{}t_ oracle.as.install.engine.modules.configuration.client.
ConfigAction.execute(ConfigAction.java:375)}}
In addition to the preceding error, the following message is in Installer logs using the timestamp above under /u01/inventory/admin-apps.oracleoutsourcing.com/oraInventory/logs:
OUI exception. oracle.sysman.oii.oiic.OiicInstallAPIException: OUI-10022:The target area_ /u01/oim/oraInventory cannot be used because it is in an invalid state.
Solution
Resume Upgrade Orchestrator on the primordial nodes to proceed with the upgrade.
Problem
The installer fails with the following type of errors:
Starting Oracle Universal Installer... Checking if CPU speed is above 300 MHz. Checking Temp space: must be greater than 4096 MB. Actual 9177 MB Passed Checking swap space: 3915 MB available, 4000 MB required. Failed <<<< Some requirement checks failed. You must fulfill these requirements before continuing with the installation,
Solution
Manually increase the requirement that failed, in this example, the swap space. Then resume orchestration.
Problem
An installer session was shut down during the upgrade.
Solution
If orchestration or tasks spawned by orchestration, such as RUP Installer, are killed in the middle of any process, the system may not be in a recoverable state and the state should be carefully reviewed by contacting Oracle Support before proceeding.
To recover from this situation, restore your backup of APPLICATIONS_BASE and start from the beginning of the upgrade.
Problem
RUP Installer Part 2 fails when the Importing Oracle Data Integrator Repositories configuration assistant is run in checkpointing mode and when the Offline Preverification step was already run in a previous session of the installer. An example of the exception message in the configuration assistant log follows:
odi.core.security.internal.ODIJpsHelper.createSubject Get exception. User:FUSION_APPS_PROV_PATCH_APPID. Execption msg is:java.lang.NoClassDefFoundError: javax/security/jacc/PolicyContext
Solution
Run the Offline Preverification step and resume Upgrade Orchestrator.
Back up the existing checkpoint file at /u01/inventory/host_name/oraInventory/checkpoint/farup/11.1.7.0.0/checkpoint.xml to a different location.
In the checkpoint.xml file, look for <aggregate name="Offline Preverification" status="success"/>, update the line to set the status to "fail" and save the file.
Resume Upgrade Orchestrator.
Problem
A NumberFormatException is reported when retrying "Deploying New application config" configuration assistant due to an incorrect value for numCompletedDeployments variable in the checkpoint.xml file.
Solution
To resolve this issue, convert the float value to an integer value for the "NumberOfSuccessfulArtifacts" attribute in the checkpoint file located at central_inventory_location/checkpoint/11.1.7.0.0/farup/checkpoint.xml.
The following example shows the value to be updated in bold:
<aggregate name="Deploying New Applications" status="fail">
<property name="NumberOfSuccessfulArtifacts" value="2.0"/>
...
</aggregate>
The following example shows the updated value in bold.
<aggregate name="Deploying New Applications" status="fail">
<property name="NumberOfSuccessfulArtifacts" value="2"/>
...
</aggregate>
Problem
RUP Installer fails during the Extending Certificate Validity phase when upgrading from Release 5 to Release 7 in a chained upgrade on an FSCMH environment. The Release 6 upgrade first installer fails at Extending Certificate Validity because DSS has an untrusted .jks file. RUP Installer failed to execute the command:
/u01/APPLTOP/fusionapps/jdk6/bin/keytool -delete -alias *ca_fusion -keystore fusion_trust.jks with keytool error: java.lang.Exception: Alias *ca_fusion does not exist.
Solution
Rename the file to a format that is different from _fusion_identity.jks and resume Upgrade Orchestrator.
Problem
The import of Group Space Templates fails with the following error:
Another application named "webcenter" exists. Specify the Server on which your application is deployed. Use: server= "YourServerName ".
Solution
There are multiple applications with the same name in the domain in which you are trying to register your application. This usually happens in a cluster environment, where the same application is deployed to multiple managed servers. If this is the case, specify the name of the server in which you are trying to register this application. For example, run the registerWSRPProducer WLST command with the server argument:
registerWSRPProducer(appName='myApp', name='MyWSRPSamples',url='http://host:port/application_name/portlets/wsrp2?WSDL', server=server_name)
Related Links
The following document provides additional information related to the subject discussed in this section:
For command syntax and examples, see "registerWSRPProducer" in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
Problem
A configuration assistant fails with the following error:
JPS-01055: Could not create credential store instance. Reason: oracle.security.jps.JpsException: JPS-00071: Ldap bootstrap credential retrieval failed. Reason:oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IO Exception:PKI-02002: Unable to open the wallet. Check password.
Solution
Restart Upgrade Orchestrator.
Problem
During the upgrade, RUP Installer reports the following errors while running the StartAllServers task, or when starting SOA servers:
[oracle.apps.startstop.execution.executor.StartStopTaskHelper: createMBeanSO.2053] [tid:621] SSUTIL011 java.lang.IllegalStateException: Cannot register two instances of WorkContextAccessController at weblogic.workarea.spi.WorkContextAccessController.<init>(WorkContextAccessController.java:24) at weblogic.workarea.spi.WorkContextAccessController$1.<init>(WorkContextAccessController.java:65) at weblogic.workarea.spi.WorkContextAccessController.getAccessController(WorkContextAccessController.java:65)
Solution
Resume Upgrade Orchestrator to retry the failed step.
Perform the following steps when an error occurs during the RUP Installer or Language Pack Installer installation phase:
Click Cancel to exit out of the installer.
Review the log files to determine the cause of the failure. The log files reside in oracle_inventory/logs/installtimestamp.log.
Resolve the cause of the failure.
Start the installer using the same command syntax that you used for the previous incomplete installation. After canceling the previous installation and starting again, you must choose to continue with the previously failed installation by clicking Yes on the Checkpoint Dialog. If the error is not recoverable, you can restore and restart from backup.
If you choose to continue with the failed installation, the installer opens at the screen where it was canceled. When canceled during the copy action, it relaunches in the Installation Summary screen. Click Next to navigate through the Installation Summary screen. When the Installation Progress screen displays, click Install to start the installation again.
Troubleshooting steps are described for the following specific failures that may occur during the installation phase:
Problem
Within a few seconds of starting the installer, you receive the following messages:
In the log file:
SEVERE: CFGLOG-00056 : Exception caught while getting node-manager homes
In the user interface:
CFGLOG-00052 : Error occurred while moving instance specific files
Solution
This failure is the result of having an incompatible version of OPatch in FA_ORACLE_HOME. To resolve the issue, download and apply patch 14044793, which contains the compatible version of OPatch.
Problem
In the Installation Location page, you receive a message about entering an invalid Oracle home, even though the location displayed on the page is correct. The installer reads /etc/oraInst.loc to determine the location of the central inventory.
Solution
To resolve this problem:
Ensure that the /etc/oraInst.loc file on the machine where you are running the installer is pointing to the correct central inventory location.
Ensure that the FA_ORACLE_HOME matches the values provided during provisioning. If a /net/location was provided as the Oracle home location during provisioning, the same /net/location that corresponds to FA_ORACLE_HOME should be provided during the installation. You can find this location by following these steps:
Open /etc/oraInst.loc and find the path to oraInventory, which is the central inventory, for example, server01/appmgr/APPTOP/oraInventory.
Change directory to the ContentsXML directory under the central inventory, for example, server01/appmgr/APPTOP/oraInventory/ContentsXML.
Open the inventory.xml file to find the correct directory path to FA_ORACLE_HOME.
Problem
During the installation phase of RUP Installer, you receive the following message on a UNIX platform.
Error in writing to file'/server01/APPLICATIONS_BASE/fusionapps/applications/lcm/ad/bin/adctrl' (Text file busy)
Solution
To resolve this issue, perform the following steps.
Run the lsof command using the full directory path of the file that is busy.
/usr/bin/lsof full_path_to_file
You should receive a list of process ids that are using the file. Kill each process using the appropriate command for your operating system.
After all processes are no longer running, resume Upgrade Orchestrator.
Problem
After running the installer, the contents of oraInst.loc were removed.
Solution
The installer always tries to copy the inventory pointer file specified by the -invPtrLoc option to the Oracle home on which the release is to be installed. If you specify an incorrect path for the -invPtrLoc file, the inventory pointer file could result in being an empty file. Review the following possible solutions for this issue:
For best results, if you are using the -invPtrLoc option, use it with this value: FA_ORACLE_HOME/oraInst.loc. This avoids a situation where you may inadvertently exclude part of the directory path to the file, as in the case of using a mapped drive. For example, if Oracle home is registered in inventory with a /net path, such as /net/home/oraInst.loc, and you provide /home/oraInst.loc to the invPtrLoc option, the installer interprets the two paths as different. The end result is an empty inventory pointer file.
If FA_ORACLE_HOME is registered in central inventory with a /net path, then you must include /net when specifying the location of the inventory pointer file with the -invPtrLoc option, for example, -invPtrLoc /net/directory_path/oraInst.loc.
Restore from a backup copy of your oraInst.loc file in case the original file is damaged. You can find this in /etc/oraInst.loc.
You can recover from this error by creating a new oraInst.loc. See the "Creating the oraInst.loc File" section in the relevant Oracle Database installation guide, for example, Oracle Database Installation Guide, 11g Release 2 (11.2) for Linux.
Then resume Upgrade Orchestrator.
This section provides information about the following failures:
Verifying Node Manager and OPMN Status Fails Due to Bad Certificate Issue
Troubleshooting Failure During Verifying Node Manager and OPMN Status
Node Manager Did Not Start Between First and Second Installer
Problem
Verifying Node Manager and OPMS Status fails with the following error:
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : [Security:090542]Certificate chain received from <hostname> - <host IP address> was not trusted causing SSL handshake failure.
Solution
The issue can occur when the host associated with a node manager is explicitly bounced in the middle of the upgrade, and if Upgrade Orchestrator expects the node manager to be in a shutdown state at that time. Node manager on the host may be configured to automatically start up as part of the system boot process and could cause various issues depending on which upgrade step was being performed when the host was restarted. To resolve this issue, stop and restart node manager on the host where the issue was reported.
Problem
Upgrade Orchestrator fails to stop OPMN processes with and error similar to either of the following errors:
Exception: OPMN could not be stopped. Script will exit. Please stop OPMN manually before re-running the script.
/APPLICATIONS_BASE/webtier_mwhome/oracle_common/jdk/jre/lib/fonts/ALBANWTJ.ttf – No such file exists.
Solution
This issue can occur due to an incompatible version of JDK being used during the process. To resolve the issue, perform the following steps.
cd /APPLICATIONS_BASE/webtier_mwhome/webtier
mv jdk_backup_existing_version jdk
cd /APPLICATIONS_BASE/webtier_mwhome/oracle_common
rm –rf jdk
cp –Rp jdk_bkp_130320_1250 jdk
Resume Upgrade Orchestrator.
Problem
The Verifying Node Manager and OPMN Status configuration assistant fails.
Solution
Do not exit out of Upgrade Orchestrator in response to this configuration assistant failure. Perform the following steps to recover:
Review the node manager log files to determine the cause of the failure:
APPLICATIONS_CONFIG/nodemanager/host_name/
Note that the APPLICATIONS_CONFIG value can be obtained from the APPLICATIONS_BASE/fusionapps/faInst.loc file.
After you resolve the issue that caused the failure, start the Node Manager on all hosts that are part of the Oracle Fusion Applications provisioned system. For more information, see "Task 3: Start Node Manager" in the Oracle Fusion Applications Administrator's Guide.
Restart the OPMN server for BI, GOP (if GOP is installed), and Web Tier. If you run the Web Tier (OHS) installed with the Oracle Fusion Applications middle tier, you can start it using the following steps. If you run the Web Tier on a separate machine, you may be able to run the steps below on the other machine. In either case, ensure that Web Tier (OHS) is up at this point.
Example for BI: (note the usage of start instead of startall)
cd APPLICATIONS_CONFIG/BIInstance/bin ./opmnctl start
Example for GOP: (note the usage of start instead of startall) Note that the OPMN server for GOP should be started from the machine that hosts the Supply Chain Management Administration Server domain. Start the OPMN server for GOP only if you have GOP installed.
cd APPLICATIONS_CONFIG/gop_1/bin ./opmnctl start
Example for Web Tier: (note the usage of start instead of startall)
cd APPLICATIONS_CONFIG/CommonDomain_webtier/bin ./opmnctl start
For more information about the location of APPLICATIONS_BASE and APPLICATIONS_CONFIG, see Section 2.1, "Before You Begin".
The BI and Web Tier processes managed by OPMN are started by RUP Installer in the Starting All Servers configuration assistant. The GOP processes managed by OPMN must be started using Fusion Applications Control, as described in Section 4.4.9, "Start External Servers", after RUP Installer completes.
Fix any other environment issues before resuming the upgrade. If RUP Installer exits for any reason, make sure that all node managers and OPMN processes are running. Contact Oracle Support Services to proceed out of this step if you have unresolved environment issues.
After you start the services, resume orchestration to proceed to the Starting All Servers configuration assistant. If the starting of servers times out, see Section 6.15, "Troubleshooting Server Start and Stop Failures".
Note:
If GOP is not installed, the user interface reports "Success" for GOP OPMN status, but the log file reports: GOP is not provisioned. Skipping check for OPMN status.
This section describes two scenarios that can prevent the node manager from starting between the first and second installer.
Problem
The node manager was manually started before or during the Extending Certification Validity configuration assistant. The node manager caches the previous keystore certificates so the updated certificates are not validated and the node manager fails to start.
Solution
Check the node manager logs to determine if it is running and when it was last started. If the time stamp is earlier than the Extending Certification Validity configuration assistant execution time stamp, you must restart the node manager so that it reads the updated keystore certificates.
To find out if the node manager is running for a specific host, connect to the host and run the following command. If any results are returned, the node manager is running.
ps -ef | grep nodemanager
If the node manager is running, find the time of the last entry of <Secure socket listener started on port nnnn> in the following directory.
APPLICATIONS_CONFIG/nodemanager/logical_host_name/nodemanager
To check the time stamp for the Extending Certification Validity configuration assistant, find the fapatch_Extending_Certificate_Validity_XXXX file in one of the following directories.
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/configlogs APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/ARCHIVE/timestamp/configlogs
The last time stamp entry is the execution time stamp.
Problem
The administration servers in one or more domains are running before the Extending Certification Validity configuration assistant runs. This prevents validation of the updated keystore certificates and fails to provide the correct status to orchestration.
Solution
Perform the following steps
Verify whether the administration server of the domain is running by launching the administration console of the domain. If the console comes up, then the administration server is running.
Verify the last time the administration server was started. Go to the APPLICATIONS_CONFIG/domains/logical_host_name/domain_name/servers/AdminServer/logs directory. Using the command, ls -lrt, find the most recent the AdminServer.log file. In this file, find the time of the last entry that contains text similar to the following example:
<Channel "Default" is now listening on machine_ip:port for protocols iiop, t3, ldap, snmp, http.>
The following RUP Lite for OHS troubleshooting issues are described:
Problem
RUP Lite for OHS fails during the ohs.plugin.UpgradeWebtier step due to missing the jdk directory.
Solution
Verify if there is a jdk_backup_existing_version directory under WT_ORACLE_HOME. If this directory exists, rename it to jdk and resume Upgrade Orchestrator.
Also, if the missing jdk directory is from WT_MW_HOME/oracle_common, check to see if there is a jdk_backup_existing_version directory under this directory. If so, rename it to jdk and resume Upgrade Orchestrator.
Problem
RUP Lite for OHS fails with the following error:
Failed execution of plugin: ohs.plugin.ReassociateCommonDomain
Solution
Update the server_name/instance/CommonDomain_webtier_local/config/OPMN/opmn/instance.properties file to set the registered property to true. Then check the Administration Server on either the Common Domain or the OSN Domain to ensure it is running. If not, bounce the server and retry RUP Lite for OHS by resuming orchestration.
Problem
RUP Lite for OHS reports a server side error occurs with an error message such as:
Server instance is not running for the security mode specified: "simple". Try again using a different security mode. The remote registration process did not succeed! Please find the specific error message below.
Solution
Perform the following steps.
Log in to the OAM administration console.
From the System Configuration tab, click Server_Instances, and double click the OAM server instance, such as, oam_server1.
Select "simple" from the Mode field in the right panel.
Click Apply to submit the changes.
Restart the OAM Server.
Restart all OHS servers in the environment.
Resume Upgrade Orchestrator.
Note:
Check the Oracle Fusion Applications OHS to ensure that SSO still works after the change. If it does not, you must upgrade Webgate manually for the Oracle Fusion Applications OHS.
Problem
During the Release 5 to Release 6 upgrade, the UpgradeOHSBinary task fails with an error message similar to the following:
File/dir required for ruplite execution is missing: /u01/webgate/host_name/webgate_installer_REL6/templates/webtier
Solution
Perform the following steps to resolve the issue.
Create the missing directory:
mkdir /u01/webgate/hostname/webgate_installer_REL6/templates/webtier
Copy all of the files from /u01/APPLTOP/fusionapps/applications/admin/OHS/patched_moduleconf/11.1.6.0.0 on the primordial host to the directory you created in the previous step.
Resume Upgrade Orchestrator.
This section provides information about the following issues:
Communication Exception on Primordial Console While Waiting for IDMOHS
WLS Exception - ESS Server Does Not Respond During Start all Servers
OAM Configuration Step Fails Due to Special Characters in Password
Problem
While PRIMORDIAL is waiting for IDMOHS:IDMUpgradeDone, there are communication exceptions on the PRIMORDIAL console.
Solution
These errors can be ignored and Upgrade Orchestrator can be resumed.
Problem
The Starting All Servers configuration assistant in RUP Installer fails to start ess_server1 and reports the following error in the ess_server1.log:
weblogic.rmi.extensions.DisconnectMonitorUnavailableException: Could not register a DisconnectListener
Solution
Perform the following steps to resolve this issue:
Open the Oracle Enterprise Manager console for the domain.
Navigate to following location:
From the console, expand the WebLogic Domain
Go to ESSCluster, then ess_server1
Right click and open System MBean browser
Go to ess_server1, ServerStart, select ess_server1, and click Arguments
Verify if -Doracle.ess.initialProcessorState=stopped is present. If it is, remove -Doracle.ess.initialProcessorState=stopped and click Apply. If it is not present, there is no action to take.
Restart ess_server1.
If the OAM administrator password contains special characters, such as '#' or '&', the OAM Configuration step will fail. To work around this issue, you can manually validate that the OAM Administration Server host/port and surname/password are correct. After you manually validate this information, you can proceed with the upgrade by resuming orchestration.
Perform the following steps to validate.
Get the OAM administrator user name and password from the credential store.
Run APPLICATIONS_BASE/fusionapps/oracle_common/common/bin/wlst.sh.
Run the following commands to connect to the Common Domain Administration Server and get the OAM administrator surname and password:
connect() listCred(map='oracle.patching', key='FUSION_APPS_PATCH_OAM_ADMIN-KEY')
Get the OAM Administration Server host and port from the following properties in APPLICATIONS_CONFIG/fapatch/FUSION_prov.properties:
OAM_ADMIN_SERVER_HOST
OAM_ADMIN_SERVER_PORT
Use oamcfgtool.jar to confirm whether the OAM server can be invoked using the values found in the previous steps.
cd APPLICATIONS_BASE/fusionapps/oracle_common/modules/oracle.oamprovider_11.1.1 java -jar oamcfgtool.jar app_domain=crm web_domain=OraFusionApp uris_file=APPLICATIONS_BASE/fusionapps/applications/crm/security/oam.conf oam_aaa_mode=open_or_simple app_agent_password=password_for_map="oracle.patching" key="FUSION_APPS_PATCH_OAM_RWG-KEY"_in_credential_store primary_oam_servers=oam_server1 oam_admin_server=http://OAM_admin_server_host:port oam_admin_username=username_for_FUSION_APPS_PATCH_OAM_ADMIN-KEY oam_admin_password=password_for_FUSION_APPS_PATCH_OAM_ADMIN-KEY oam_version=11 default_authn_scheme=FAAuthScheme
If the previous command is successful, the validation is successful and you can resume Upgrade Orchestrator.
The location of your Governance, Risk, and Compliance (GRC) policies varies, depending on your upgrade path to Release 7. GRC polices are located in the grc OAM application domain if your Oracle Fusion Applications environment was originally provisioned with either version 11.1.1.5 or 11.1.2, then upgraded to version 11.1.3 and beyond. If your environment was originally provisioned with version 11.1.3 and upgraded to version 11.1.4 and beyond, your GRC policies are located in the fs OAM application domain.
This section provides the following troubleshooting information related to the Applying Pre-PSA Middleware Patches or Applying Post-PSA Middleware Patches configuration assistants:
Problem
An error occurred during the Applying Pre-PSA Middleware Patches or Applying Post-PSA Middleware Patches configuration assistant.
Solution
Review the log file in the relevant location to find the cause of the error:
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/ApplyPrePSAMiddlewarePatchestimestamp.log
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/ApplyPostPSAMiddlewarePatchestimestamp.log
For specific OPatch failures, go to each of the individual Oracle home directories to find the details of the OPatch errors. For example, for a SOA failure, go to APPLICATIONS_BASE/fusionapps/soa/cfgtoollogs/opatch.
Problem
The Applying Post-PSA Middleware Patches configuration assistant hangs.
Solution
This problem is most likely due to adpatch hanging as the result of the java worker not getting the database connection. You can resolve this issue by following the steps in Section 6.13, "Troubleshooting Loading Database Components". Run the commands from ATGPF_ORACLE_HOME instead of FA_ORACLE_HOME.
Problem
The following error occurs:
OPatch cannot continue because it can't load library from the directory "<dbclient Oracle Home>/oui/lib/linux64"
Solution
This error may occur if the OUI version in the database client Oracle home is 11.2 while the OUI version in Oracle Fusion Applications Oracle home (FA_ORACLE_HOME) is 11.1.
Perform the following steps to resolve this issue:
Go to the DB Client home.
Set the ORACLE_HOME environment variable to point to the database client Oracle home.
Apply the database client patches using the following command:
$ORACLE_HOME/OPatch/opatch apply patch_location
Because the patches have now been manually applied, perform the following steps to continue with the upgrade:
Go to the FA_ORACLE_HOME/fusionapps/applications/lcm/tp/config/RUP/FMW directory.
Open the pre-psa-jobs.xml file for editing.
Comment out the job with the name dbclient An example of this job follows.
<!-- <job>
<id>10</id>
<target>FAMW</target>
<component>
<name>dbclient</name>
<version>11.1.1.5</version>
<component>
<utility_name>opatch</utility_name>
<patch_number>NA</patch_number>
<command>%opatch% napply -silent -skip_duplicate -skip_subset
-oh %dbclient_home% -phBaseDir %dbclient_patch% -jre %jre_loc% -invPtrLoc
%oraInstLocFile%</command>
<patch_location>NA</patch_location
</job>
Problem
The following error is reported:
ORA-01658: unable to create INITIAL extent for segment in tablespace FUSION_TS_SEED.
Solution
The standard output from the ORA-1658 error follows:
ORA-01658: unable to create INITIAL extent for segment in tablespace string
Cause: Failed to find sufficient contiguous space to allocate
INITIAL extent for segment being created.
Action: Use ALTER TABLESPACE ADD DATAFILE to add additional space to
the tablespace or retry with a smaller value for INITIAL
For more information, refer to Oracle Database documentation.
Problem
An error occurred during the Upgrading Middleware Schema configuration assistant.
Solution
Review the log file in this location to find the cause of the error:
fusionapps/oracle_common/upgrade/logs/psatimestamp.log
Problem
The Upgrading Middleware Schema configuration assistant fails because JAVA_HOME cannot be found.
Solution
Set the JAVA_HOME and then manually run the upgrade for the failed schema, as shown in the following example:
export JAVA_HOME=/u01/APPLTOP/fusionapps/jdk6 /u01/APPLTOP/fusionapps/oracle_common/bin/psa -response /u01/APPLTOP/fusionapps/applications/admin/FUSION/oui_resp/psa_response_crm.txt
Problem
The Applying Downloaded Patches configuration assistant failed with the following error:
Stack Description: java.lang.RuntimeException: PatchObject constructor: Input file "/net/server01/Downloaded_Patches/atgpf/patch/1234567/etc /config/inventory" does not exist.
Solution
This type of error occurs when you do not download the patches to the appropriate directory. To resolve this issue, copy the patches to the correct directory and resume Upgrade Orchestrator.
This section contains information about troubleshooting issues that may occur during the Loading Database Components configuration assistant. Depending on the type of failure, you may need to review one or more of the log files in the following locations:
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/PatchManager_DBPatch/
FAPatchManager_apply_timestamp.log
adpatch_apply_timestamp.log
adpatch_apply_timestamp_workernum.log
ATGPF_HOME/admin/FUSION/log
The following troubleshooting issues are described in this section:
Error While Loading Database Components Using RUP Installer in GUI Mode
Loading Database Components Configuration Assistant Fails When JDBC URL Is Null
Orchestration Fails on Primordial Host Reporting Active FAPatchMgr Sessions
Problem
You receive an email notification stating that one or more database workers failed during the Loading Database Components configuration assistant.
Solution
You receive this email notification only when the upgrade cannot progress any further and requires user intervention. In this scenario, all the workers are in a FAILED or IDLE status. The Configuration Assistant remains in a RUNNING status until all tasks in Loading Database Components have run. To resolve this issue, you must start AD Controller to manage the failed workers. For additional information, see "Troubleshooting Patching Sessions for Database Content" in the Oracle Fusion Applications Patching Guide. After you resolve the issue that caused the workers to fail, and restart the workers, Upgrade Orchestrator continues processing. No further intervention is required.
Note that the messages are displayed on the console for database component failures if you run orchestration with the -DlogLevel option set to FINEST.
There might be corner cases when you might receive an alert email indicating failed workers although the workers are still running. In such cases, you can ignore the email alert after ensuring the workers are running with no failures.
Problem
RUP Installer reports that one or more database workers failed during the Loading Database Components configuration assistant.
Solution
You must start AD Controller to manage the failed workers. After you resolve the issue that caused the workers to fail and you restart the failed worker, click OK in the dialog box and RUP Installer continues processing. For additional information, see "Troubleshooting Patching Sessions for Database Content" in the Oracle Fusion Applications Patching Guide.
Problem
Your database goes down while RUP Installer is running the Loading Database Components configuration assistant. If you simply bring the database up and then resume orchestration, you may encounter the following error:
Failed to connect to the database as fusion with error: No more data to read from socket
Solution
Perform the following steps to recover from this error:
Force the database patching session to fail.
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd forcefail
Start AD Controller.
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/adctrl.sh (Windows) FA_ORACLE_HOME\lcm\ad\bin\adctrl.cmd
For more information, see "Starting AD Controller" in the Oracle Fusion Applications Patching Guide.
Follow this sequence of steps in AD Controller to manage the workers:
Select Tell manager that a worker failed its job and enter All for all workers.
Select Tell worker to quit and enter All for all workers. Note that this does not kill the workers. It sends a command to the worker to shutdown after it completes the current task.
Wait for all workers to complete their tasks and shut down normally.
If there are still some worker processes that do not shut down, kill those processes manually by selecting Tell manager that a worker failed its job. Then select Tell manager that a worker acknowledges quit and enter All for all workers.
From your operating system, check for processes that are running fapmgr, javaworker, adpatch, adadmin, sqlplus, and adworker. If any exist, terminate them from your operating system.
Select Tell worker to restart a failed job and enter All for all workers.
Resume Upgrade Orchestrator.
Problem
AutoPatch validation fails with the following message:
An active adpatch or adadmin session was found. Complete or terminate the active session to allow fapmgr to proceed.
Solution
Perform the following steps to resolve this error:
Run the fapmgr forcefail command to update the patching tables.
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail [-logfile log file name] [-loglevel level] (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd forcefail [-logfile log file name] [-loglevel level]
Run the fapmgr abort command from FA_ORACLE_HOME to find out if an Oracle Fusion Applications Patch Manager session must be cleaned up.
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh abort [-logfile log file name] [-logLevel level] (Windows) FA_ORACLE_HOME\lcm\ad\bin\fapmgr.cmd abort [-logfile log file name] [-logLevel level]
If this command finds no failed session, proceed to Step 3.
Run the following commands from ATGPF_ORACLE_HOME to abandon any Applications Core patching sessions or AD Administration sessions that may be running:
(UNIX) ATGPF_ORACLE_HOME/lcm/ad/bin/adpatch.sh abandon=y interactive=n defaultsfile=APPLICATIONS_CONFIG/atgpf/admin/defaults.txt (UNIX) ATGPF_ORACLE_HOME/lcm/ad/bin/adadmin.sh abandon=y interactive=n defaultsfile=APPLICATIONS_CONFIG/atgpf/admin/defaults.txt (Windows) ATGPF_ORACLE_HOME\lcm\ad\bin/adpatch.exe abandon=y interactive=n defaultsfile=APPLICATIONS_CONFIG\atgpf\admin\defaults.txt (Windows) ATGPF_ORACLE_HOME\lcm\ad\bin/adadmin.cmd abandon=y interactive=n defaultsfile=APPLICATIONS_CONFIG\atgpf\admin\defaults.txt
Problem
When multiple seed data files are uploaded for the same flexfield but for different flexfield contexts, the upload tasks can fail due to locking issues. The failed tasks appear in the log file as the following error:
Loading failed with a JboException: JBO-25014: Another user has changed the row with primary keyoracle.jbo.Key ...
Solution
AutoPatch defers any failed tasks to the end of the phase and reattempts the failed tasks only after attempting all tasks in the phase at least once. Usually the flexfield seed data tasks that failed due to the locking issue succeed on subsequent attempts. You can ignore these errors if the flexfield seed data task succeeds on the retry. If the task remains in a failed state, you must use the AD Controller utility to retry the failed task.
For more information, see "Restarting a Failed Worker" in the Oracle Fusion Applications Patching Guide.
Problem
A failure occurred during either the Grant Privileges to Application Schemas or the Creating Grants/Synonyms on Application Database Objects configuration assistant.
Solution
You can find the cause of the failure by running the script manually as the SYSDBA user, using SQL*Plus or SQL*Developer. After you resolve the issue, resume orchestration.
Problem
The Loading Database Components configuration assistant fails with the following exception in the FAPManager log file.
Exception [2013-11-20T06:01:53.280+00:00] [] [ERROR] [] [] [tid: 34] [ecid:0000K9o6OHX6qI25Rrt1id1IZ4Pl00000d,0] java.lang.NullPointerException: Schema @ name/password/ jdbc url cannot be null[[ at oracle.apps.ad.common.db.ADDatabaseConnection. createConnection(ADDatabaseConnection.java:529) at oracle.apps.ad.common.db.ADDatabaseConnection. getConnectionWithCluster(ADDatabaseConnection.java:444) at oracle.apps.ad.common.db.ADDatabaseConnection. getConnectionWithCluster(ADDatabaseConnection.java:446) at oracle.apps.ad.common.db.ADDatabaseConnection. getConnectionWithCluster(ADDatabaseConnection.java:446) ....................................................... [2013-11-20T06:01:59.568+00:00] [apps] [ERROR] [] [oracle.apps.ad.rupconfig.Loading_Database_Components] [tid: 34] [ecid:0000K9o6OHX6qI25Rrt1id1IZ4Pl00000d,0] [[java.lang.StackOverflowError]]
Solution
Run the following command:
FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail
Resume Upgrade Orchestrator.
Problem
In a scenario where fapmgr applies patches using the multi-apply feature, and any patch validation fails, the status is set to 'SUCCESS'. This new status is treated as an active session by Health Checker and it fails, causing an orchestration failure with the error message as shown in the following example:
[ERROR]: Plugin 'PatchSessionsAndProcessesCheck': HC-PATCHSP-00004 : Check #1: Found active FAPMgr sessions. Review log files for details on which Sessions exist. (Pre-Upgrade Checks)
Solution
Run the following SQL*Plus command in the fusion schema:
update AD_PATCH_UTIL_SESSIONS set status='COMPLETED_WITH_WARNINGS' where status='SUCCESS';
Resume orchestration.
This section contains information about the following troubleshooting issues that may occur during the Deploying Application Policies configuration assistant:
Log files for this configuration assistant may be found in this location:
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/configLogs/fapatch_Deploying_Applications_Policies_(jazn-data.xml)_timestamp.log
Problem
A failure occurs during applications policy analysis.
Solution
Review the log file that is generated by each stripe. The log files are located under the main log directory, APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP and are named as follows:
fapatch_CRMJaznAnalysis_timestamp.log
fapatch_FSCMJaznAnalysis_timestamp.log
fapatch_HCMJaznAnalysis_timestamp.log
fapatch_OBIJaznAnalysis_timestamp.log
fapatch_SOAJaznAnalysis_timestamp.log
fapatch_UCMJaznAnalysis_timestamp.log
fapatch_BPMJaznAnalysis_timestamp.log
After you resolve the JAZN analysis error, retry the analysis for the failed stripe to confirm the issue is resolved.
Problem
A failure occurs during Deploying Application Policies.
Solution
When a failure occurs during Deploying Application Policies, you must restore only the stripe or system policy that has failed, from your backup. Use the OPSS migrateSecurityStore command with the appropriate source and destination arguments to perform the restore. Do not restore a stripe that has not failed. Review the JAZN deployment log file to determine the cause of the failure, fapatch_Deploying_Applications_Policies_(jazn-data.xml)_timestamp.log.
After you resolve the issue, resume Upgrade Orchestrator.
For more information, see "Migrating with the Script migrateSecurityStore" in the Oracle Fusion Middleware Application Security Guide.
Problem
The following warning occurs during Deploying Application Policies:
WARNING: Failed to validate the xml content. cvc-complex-type.2.4.a: Invalid
content was found starting with element 'property'. One of
'{"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":propertySetRef,
"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":extendedProperty,
"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":extendedPropertySetRef,
"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":serviceInstanceRef}'
is expected. Location: line 165 column 96.
WLS ManagedService is not up running. Fall back to use system properties for
configuration.
Solution
You can safely ignore this message as there is no functional impact of this warning and the deployment is successful.
Problem
The following warning occurs during Deploying Application Policies:
FINE: Application policies already exists for application: fscm
oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException:
Cannot create application policy context "fscm".
at
oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.unsync_createApp
licationPolicy(LdapPolicyStore.java:833)
at
oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.createApplicatio
nPolicy(LdapPolicyStore.java:753)
at
oracle.security.jps.internal.tools.utility.destination.apibased.JpsDstPolicy.c
lone(JpsDstPolicy.java:805)
Solution
You can safely ignore this message as there is no functional impact of this warning and the deployment is successful.
Problem
The IDM Server goes down during Deploying Application Policies and the deployment fails.
Solution
Upgrade Orchestrator does not allow a retry after this type of failure. You must instead exit orchestration and restore from your IDM backup. Then resume Upgrade Orchestrator.
This section includes the following troubleshooting topics:
Startup Failed for CommonDomain: OHSComponent (Oracle VM Only)
The SOA-infra Application is in a Warning State on All Domains
Problem
A failure during the Starting All Servers configuration assistant typically happens when one of the servers times out and fails to start due to resource issues or application specific issues.
Solution
Various platforms and environment configurations can impact how long it will take all servers to actually start during the Starting All Servers configuration assistant. Although RUP Installer waits an average amount of time for this configuration assistant to complete before it is marked as Failed, different platforms may require more time. It is not unusual to receive timeout errors in the log files if the starting of all servers for your environment requires more time than RUP Installer allows. If this configuration assistant fails, follow these steps:
Monitor the status of the servers by reviewing the messages in the server log files or on the console. The log file, APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/StartStop/fastartstop_timestamp.log, indicates which server started and failed to start.
An example of messages for a server that timed out follows.
Time out while performing Start for domain SCMDomain. Waited for 2400 seconds [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [UTIL] [oracle.apps.startstop.util.MbeanUtil: runSSCommandOnDomain.868] [tid:37] Start operation is completed for domain SCMDomain. Please see SCMDomain.log for details. [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [UTIL] [oracle.apps.startstop.invoke.StartStopTask: call.221] [tid:37] StartStopTask over for domain SCMDomain [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [SST] [oracle.apps.startstop.invoke.StartStopTask: call.223] [tid:37] Finished the task for the Domain SCMDomain
Review the log files at the domain level to see a summary of the server status for that domain: APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/StartStop/domain name_timestamp.log.
Review the corresponding server logs for the failed servers under the following directory: APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs.
After you determine and resolve the cause of the failure, restart Upgrade Orchestrator.
Problem
The following exception during the Starting all Servers configuration action indicates a failure in starting the BIServer:
Start all servers fails to start Start operation on the component :coreapplication_obips1:, for the instance :BIInstance: - FAILED
The coreapplication_obips1 server log file reports the following error:
ecid:]] [2012-04-10T00:22:20.000-07:00] [OBIPS] [ERROR:16] [] [saw.security.odbcuserpopulationimpl.initialize] [ecid: ] [tid: ] Unable to create a system user connection to BI Server during start up. Trying again.[[ File:odbcuserpoploaderimpl.cpp Line:325 Location: saw.security.odbcuserpopulationimpl.initialize saw.catalog.local.loadCatalog saw.subsystems.catalogbootstrapper.loadcatalog saw.webextensionbase.init saw.sawserver ecid:]] [2012-04-10T00:22:25.000-07:00] [OBIPS] [NOTIFICATION:1] [] [saw.sawserver] [ecid: ] [tid: ] Oracle BI Presentation Services are shutting down.[[ File:sawserver.cpp Line:706 Location: saw.sawserver ecid:
Solution
Perform the following steps to work around this issue.
Resume Upgrade Orchestrator, which shuts down and starts bi_server1.
Monitor the fastartstop log files and the state of bi_server1(BIDomain).
As soon as bi_server1 restarts, as indicated by a RUNNING status, start the component coreapplication_obiccs1 or all the components of type OracleBIClusterControllerComponent using opmnctl.
Example syntax follows:
*/BIInstance/bin/opmnctl startproc ias-component=coreapplication_obiccs1
Problem
The OHS diagnostic log contains the following error message:
No such file or directory: Couldn't create accept lock
Solution
This issue could be the result of the hypervisors going down, resulting in bringing the Oracle VM servers down. Perform the following steps to resolve the error:
Find the entry for the lock file in httpd.config, for example:
LockFile "/u101/ohs_inst1/diagnostics/logs/OHS/ohs1/http_lock"
Confirm whether the directory that contains the lock file exists.
Assuming this directory does not exist, create the directory.
Problem
The following error is reported during Online Preverification:
weblogic.management.mbeanservers.edit.EditTimedOutException
Solution
You may have to manually release the edit session. For more information, see "Resolving an EditTimedOutException Error" in the Oracle Fusion Applications Patching Guide.
Problem
After the upgrade, the following error displays after you log in to the WLS Console of CommonDomain. and navigate to Deployments:
soa-infra application is in WARNING state.
Solution
You can ignore this error as there is no functional impact for SOA users due to this error. To resolve the issue, see "Updating the soa-infra Application in Warning State" in the Oracle Fusion Middleware Patching Guide.
This section describes how to recover from failures during the Deploying SOA Composites configuration assistant. The following topics are described:
The following log files are generated by the deployment of SOA composites:
Client side log files where individual domain logs reside: APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/soalogs
Log files for the failed domain:
APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/soa_server1.log
APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/soa_server1.out
APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/soa_server1-diagnostic.log
APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/AdminServer.log
Problem
Normally, a failed SOA composite is undeployed by RUP Installer. However, if the failure of the deployment is due to an issue such as SOA servers running out of memory, then RUP Installer does not recover until you resume Upgrade Orchestrator.
Examples of error messages related to SOA Composite failures follow:
CFGLOG-00380: SOA composite "composite_name" patch failed for server "server_name". Recovery process also failed with an unknown reason. If the SOA composite patch exists on the server, it will be automatically undeployed during retry or a checkpoint run. Also if the base composite is not the default composite, it will be automatically set as default. CFGLOG-00327: SOA composite "composite_name" patch failed for server "server_name". Recovery process also failed, and the composite patch is not undeployed. The patch will be automatically undeployed during retry or a checkpoint run. CFGLOG-00328: SOA composite "composite_name" patch failed for server "server_name". Recovery process also failed, and the base composite is not set as the default composite. The base composite will be automatically set as default during retry or a checkpoint run.
Examples of report exceptions follow:
CFGEX-00087: SOA composite "composite_name" patch failed for server "server_name". Recovery process also failed. Recovery will be done automatically during retry or a checkpoint run. Action : No action required. CFGEX-00073: SOA composite "composite_name" patch failed for server "server_name". Action : See logs for details.
Solution
When patching existing SOA composites, RUP Installer automatically recovers any partially deployed SOA composites after failure when you restart Upgrade Orchestrator. The following actions can be performed by Upgrade Orchestrator:
Undeploy the partially deployed SOA composite revision if it is still present.
Set as default the same SOA composite revision that was default before the patching was attempted, if it's not already set as default.
If the failure was caused by an environment issue, such as running out of memory, resolve the cause of the failure before you restart RUP Installer.
Problem
The following error is reported during SOA Composite deployment on a Solaris platform:
CFGEX-00079 : Wsm-pm application is not running in domain "domain name"
You have already confirmed that the wsm-pm application is running on this domain.
Solution
Perform the following steps:
Log in to the failed domain and check the health of all managed servers and deployed applications.
Bounce all managed servers of the failed domains.
Exit Upgrade Orchestrator.
Restart Upgrade Orchestrator.
If a customized SOA composite deployment fails during the upgrade, you must manually deploy this composite using WLST commands.
To apply a SOA composite manually after a deployment failure
In the following steps, the example composite, FinAp, is patched from revision 1.0 to revision 2.0 and the SAR file of revision 2.0 is in FA_ORACLE_HOME/crm/deploy/sca_FinAp_rev2.0.jar.
Note that the parameters are for illustration purposes only.
Refer to the Diagnostics report to find the name and location of the sca_*.jar file that was copied to FA_ORACLE_HOME by Oracle Fusion Applications Patch Manager. For more information, see "Diagnostics Report" in the Oracle Fusion Applications Patching Guide.
If the previous revision contained JDeveloper customizations, ensure that you deploy the patched revision with the merged JDeveloper customizations. Using the sca_*.jar file from Step 1, follow the JDeveloper customization merge instructions that are described in Section 6.16.6, "Merging SOA Composite JDeveloper Customizations During SOA Preverification". Then use the merged sca_*.jar for Step 3.
Deploy the patched composite using the single patch composite command.
sca_patchComposite('SOA-Infra URL', user, password,
'/FA_ORACLE_HOME/crm/deploy//sca_FinAprev2.0.jar', mergeLogFile='/tmp/merge-log.txt')
You must run the UpdateSOAMDS SOA composite on every domain if you made any flexfield changes, by following the steps described in "Task: Synchronizing Customized Flexfields in the MDS Repository for SOA" in the Oracle Fusion Applications Extensibility Guide for Developers.
If you performed JDeveloper customizations to a SOA composite and you deployed the composite to the SOA runtime, RUP Installer reports an error during SOA Preverification, which instructs you to take the newer version of the composite that is in the release. You must then merge your customizations by performing the following steps.
If any customizations are detected, the SOA Preverification results display the SOA composite name, its location in the FA_ORACLE_HOME/stripe/deploy directory, and the requirement for you to merge JDeveloper customizations into the sca_*.jar file in FA_ORACLE_HOME before proceeding with RUP Installer. The stripe in the directory path refers to crm, hcm, fscm, and so on.
Open the custom SOA workshops and the customized version of the Fusion Applications SOA composite in JDeveloper using "Oracle Fusion Applications Developer". For more information, see "Customizing SOA Composite Applications with JDeveloper" in the Oracle Fusion Applications Extensibility Guide for Developers.
Import the composite sca_*.jar file from FA_ORACLE_HOME/stripe/deploy into the project, for example revision 11.1.7.0.0, in JDeveloper. Make note of this revision number in the deployment window because you will need it in Step 8.
Restart JDeveloper in the Oracle Fusion Applications Administrator Customization role.
Verify that there are no errors in JDeveloper.
Verify that the changes introduced in both the customized version and the patched version are present.
Right-click the composite project in the Application Navigator, select Deploy, select the composite, click Deploy to SAR, and click Next.
Manually change the value in New Revision ID to the revision from Step 3, for example, 11.1.7.0.0, and click Finish.
If the deployment folder is set to a location different from that of the FA_ORACLE_HOME/stripe/deploy directory, copy and replace the JAR in the location mentioned in the error message of this SOA Composite. If your file name is different, rename it to the original name. You must copy the jar in the correct format to FA_ORACLE_HOME/stripe/deploy. For example if you have sca_ContractsDeliverablePurchaseDocAttrReadComposite_rev11.1.7.0.0.jar in JDeveloper, then you must copy it back to the FA_ORACLE_HOME/stripe/deploy directory as sca_ContractsDeliverablePurchaseDocAttrReadComposite.jar.
To proceed with the installation, use the same command you used to start Upgrade Orchestrator.
For more information about customizing SOA composites, see "Customizing and Extending SOA Components" in the Oracle Fusion Applications Extensibility Guide for Developers.
Review the /u01/lcm/rupliteovm/output/logs/ruplite.log file to confirm there are no errors. You can also check rehydration framework logs under /assemblybuilder/logs or /var/log for any errors.
The following troubleshooting issues are also described in this section:
RUPLite for OVM in Offline Mode Fails on ovm.plugin.DisableWebchat
RUP Lite for OVM in Offline Mode Fails With Offline Data Files Issue
RUP Lite for OVM in Post-Root Mode Fails With Property Issues
RUP Installer Fails During Apply Pre-PSA Due to Smart Patch Conflict (Oracle VM Only)
Review the following troubleshooting information for specific plug-ins:
UpdateSESDBConnection: This plug-in is rerunnable. The rehydration command this plug-in calls requires that all database schema passwords be registered in the credentials store. The credential store must contain a entry for FUSION_DISCUSSIONS_CRAWLER. The following error message appears in the ruplite.log file if this entry is missing:
Executing Task: oracle.apps.fnd.provisioning.ovm.rehydratefw.cli.cmd.fasec.UpdateSESDBConnectionCmd ... FAILED. [0m31s] An error occurred: An error occurred during command execution: A password could not be retrieved because: The deploy property 'credential.FUSION_DISCUSSIONS_CRAWLER.password' does not have a value. A value from the credential store could not be read or does not exist. A reference property was not provided.
The Register Database Schema tool populates the credentials store with database schema passwords. For more information, see Section 2.4.2.3, "Prepare to Register Database Schema Information".
Perform the following steps to verify this plug-in was successful:
Open the Oracle Secure Enterprise Search administration page.
Go to the Sources tab.
Edit the Announcements data source.
Verify that Source Configuration - Database Connection String and Authorization - Authorization Database Connection String reflect the values for host, port, and service name from ovm-ha-deploy.properties. If the faovm.ha.fusiondb.new.is.rac property is set to false, the non-RAC port values will be used. If this property is set to true, the RAC port values will be used.
DeployECSF: This plug-in is rerunnable. If your environment was originally provisioned before Release 5, you can verify that this plug-in was successful by confirming that the help object, schedule and group being deployed are reported in the log file. You can also use Fusion Applications Control to connect to the Administration Server that hosts the search application and confirm that the Help instance artifacts are deployed.
DisableWebchatConnections: This plug-in is rerunnable. If your environment has WebChat enabled this plug-in does not disable the connection.
ValidateEnvironment: If this plug-in fails, RUP Lite for OVM stops. You must resolve any errors reported in the log file and then run RUP Lite for OVM again.
SetupCredentials: If this plug-in fails, RUP Lite for OVM stops. Typical causes of failure are an incorrect key for an existing wallet, or specifying a key for a new wallet that does not meet Oracle's minimum standards. You must resolve any errors reported in the log file and then run RUP Lite for OVM again.
Note that you are prompted for the password twice and that both responses must be identical. If you need to change the password in the wallet, set the ovm.plugin.SetupCredentials.enable_password_update property to true. If this property is enabled, when the SetupCredentials plug-in reruns, you are given the option to overwrite the existing password for a particular plug-in, in the wallet. By default this feature is disabled.
UpdateResolvConf: This plug-in is rerunnable. If a value already exists in the /etc/resolv.conf file, it will not be added again. If you remove a property value from metadata/env.properties, the value is not removed from /etc/resolve.conf by running the plug-in again.
ApplyMemorySettings: Check the fusionapps_start_params.properties files in the environment, which are located under the bin directory of each domain. Ensure that the minmaxmemory settings in the files are at least as high as the settings in the template under the ovm/metadata directory that corresponds to the environment's topology.
GenerateOptimizedQueryPlans: This plug-in is rerunnable. Verify this plug-in was successful by connecting to the database as fusion_mds and running the following command:
SELECT TO_CHAR(last_analyzed, 'yyyy/mm/dd hh:mi:ss am') as last_analyzed FROM user_tables;
The results should show that the tables were just analyzed.
DisableSearchUI: This plug-in is rerunnable. Verify this plug-in was successful by connecting to the database as fusion and running the following command:
select fnd_profile.value('FUSION_APPS_SEARCH_ENABLED') from DUAL;
The result should be "N".
UpdateFusionIIRDiag: This plug-in is rerunnable. Verify this plug-in was successful by confirming that the fusioniirdiag.sh script in the APPLICATIONS_BASE/InformaticaIR/bin directory is the updated version and the permissions and ownership of this script match those of the other scripts in the same directory. You can verify the version by noting the date and size of the file.
DisableWebchat: Verify this plug-in was successful by confirming that the following SQL*Plus queries return no rows:
SELECT username FROM all_users WHERE username LIKE '%BEE%'; SELECT tablespace_name FROM dba_tablespaces WHERE tablespace_name LIKE '%BEE%';
Also verify that the beehivereadonlyuser user does not exist in LDAP and that the WebChat Oracle VM is shut down.
If RUP Lite for OVM validation fails, the cause of the failure could be that a non-application user, such as root, was used to create the RUP Lite for OVM wallet. If this is the case, perform the following steps to resolve the issue.
Copy the Release 6 pre-upgrade rupliteovm to the share.
Edit env.properties to enable the CreateEMFAWallet property and set CUSTOM_WALLET_DIR to the absolute path of the output/wallet directory of rupliteovm.
Run RUP Lite for OVM in wallet mode to create the wallet.
Run RUP Lite for OVM in validate mode from that primordial host as the correct applications user, not as root.
Problem
RUP Lite for OVM runs for a long time during domain configuration.
Solution
Perform the following steps to resolve this issue:
Ensure that the IDM host is accessible and responding.
Ensure that the database is accessible and responding.
If either the IDM host or the database is not responding, update the status of the orchestrator task that runs RUP Lite for OVM to "Error", using the following command:
cd ORCH_LOCATION/bin ./orchestration.sh updateStatus -pod POD_NAME -hosttype host_type -hostname host_name -release release_number -phase phase_name -taskid plugin_name -taskstatus Error
Fix the issue with the IDM host or the database and resume Upgrade Orchestrator.
If none of the above steps solve the problem, contact Oracle Support with detailed log information.
Problem
During the PreDowntime phase, the RupLiteOvmValidatePlugin hangs.
Solution
The plug-in, RupLiteOvmValidatePlugin, creates a log with the lock file, ruplitevalidate.log.lck.
Perform the following steps to resolve this issue:
Verify if your Shared directory is mounted with the nolock option. If the Shared directory is mounted with the nolock option then remove the .lck file and proceed with ruplite validate. If this is not the case, then create the mount point of the Shared directory with nolock with the following command:
mount -o nolock host_name:directory_path_to_which_directory
Remove the .lck file.
rm ruplitevalidate.log.lck
Run ruplite from the mount point used in Step 1:
bin/ruplite.sh validate
Problem
RUPLite for OVM can fail on ovm.plugin.DisableWebchat if there are guaranteed restore points in the database.
Solution
Perform the following steps to drop the restore points and resume Upgrade Orchestrator
Connect as SYS user.
Run the following SQL*Plus query:
SELECT NAME, SCN, TIME, DATABASE_INCARNATION#,GUARANTEE_FLASHBACK_DATABASE,STORAGE_SIZE FROM V$RESTORE_POINT WHERE GUARANTEE_FLASHBACK_DATABASE='YES';
If the query returns any restore points, drop the restore points using the following command:
DROP RESTORE POINT restore_point_name;
Resume Upgrade Orchestrator.
Problem
During the RUP Lite for OVM Offline task, a failure with a message "RUP Lite for OVM in offline mode task failed" is reported.
Solution
Check the log file to see if an error exists as shown in the following example:
ORA-01191: file 28 is already offline - cannot do a normal offline ORA-01110: data file 28: '+DATA/prtff/datafile/bee_data.dbf
If this error is reported, perform the following steps.
Connect to the database as SYSDBA and run the following SQL*Plus query to determine if the data file is offline.
select FILE_NAME,ONLINE_STATUS from dba_data_files where ONLINE_STATUS<>'ONLINE';
An example of a data file that is offline follows:
DATA/prtff/datafile/bee_data.dbf, OFFLINE
If there are offline files, run the following command as SYSDBA:
alter database datafile 'PATH_TO_DATAFILE' online
The PATH_TO_DATAFILE is the path that was returned by the previous query.
Resume Upgrade Orchestrator.
Problem
RUP Lite for OVM in the post-root mode fails with the following error:
"Property faovm.topo.type not found in context file"
Solution
Rerun RUP Lite for OVM in the post-root phase, and if the issue persists, contact Oracle Support.
Problem
For the CRM stripe on an Oracle VM environment during the Release 6 to Release 7 upgrade, RUP Installer fails during the Apply Pre-PSA Middleware Patches configuration assistant, due to a smart patch conflict. The following exception is reported:
"Conflict(s) detected - resolve conflict condition and execute patch installation again. Conflict condition details follow: SEVERE: Conflict(s) detected - resolve conflict condition and execute patch installation again Patch HYKC is mutually exclusive and cannot coexist with patch(es): 3BBT,SZXM,7YZB,6D9T,56MM,F89C,9264,9887,S39F,7AAZ,JZED,E9FL,IH4D,YJTB SEVERE: Patch HYKC is mutually exclusive and cannot coexist with patch(es): 3BBT,SZXM,7YZB,6D9T,56MM,F89C,9264,9887,S39F,7AAZ,JZED,E9FL,IH4D,YJTB"
Solution
Manually roll back all conflicting WLS patches and rerun orchestration.
Health Checker checks the JAZN Analysis reports for potential conflicts and deletions that are not patched automatically by RUP Installer. The reports are located in this directory:
Release 7 location:
APPLICATIONS_CONFIG/lcm/admin/11.1.7.0.0/fapatch/JAZN/stripe/delta/report.txt
Release 6 location:
APPLICATIONS_BASE/fusionapps/applications/admin/JAZN/stripe/delta/report.txt
The stripe is crm, fscm, hcm, obi, soa, ucm or bpm.
Review the Modification section of the report to see the roles that RUP Installer did not update. For each conflict that displays in this report, you must evaluate and manually patch the role by using Oracle Authorization Policy Manager (APM). For more information, see "Upgrading Oracle Fusion Applications Policies" in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).
The following example shows a typical Application Role conflict that has been modified by both the patch and production, therefore it is not applied by RUP Installer.
MODIFICATION CONFLICTS Artifact type: Application Role Artifact Name: OBIA_PARTNER_CHANNEL_ADMINISTRATIVE_ANALYSIS_DUTY Description: This artifact is modified at attribute level in patch version and also in production.
Note the location of the following files for reference when using APM:
Location of baseline files, where stripe is crm, fscm, hcm, obi, soa, ucm or bpm:
FA_ORACLE_HOME/admin/JAZN/stripe/baseline
Location of patch files for fscm, crm, and hcm stripes:
FA_ORACLE_HOME/stripe/deploy/system-jazn-data.xml
Location of patch files for the obi, soa, ucm or bpm stripes:
FA_ORACLE_HOME/com/acr/security/jazn/bip_jazn-data.xml
Problem
Health Checker reports the following error when running the Verifying OS Attributes check:
[ERROR]: Plugin 'OsCheck': HC-OS-00002 : OS Support Check failed.Error(s): OS version is not supported(Check for OS name ,arch and version support)(Pre-Downtime Checks)
Solution
If you are using Red Hat Enterprise Linux Server release 5.* (Tikanga), a manual verification of OS platform is required to determine if this validation error can be ignored. To perform the manual validation, run the osarch.sh script, which is located under SHARED_LOCATION/ORCH_LOCATION/util. Verify that the PLATFORM_TYPE=Linux, the OS_ARCH=x86_64, and the PLATFORM_VERSION is any sub-version of version 5 (Tikanga) of RHEL Server. If all of the above requirements are satisfied, the error is safe to ignore. Otherwise, refer to the suggested corrective actions provided by Health Checker. Sample output from the osarch.sh script follows:
PLATFORM_TYPE=Linux OS_ARCH=x86_64 PLATFORM_VERSION=Enterprise Linux Server release 5.6 (Tikanga)
Problem
Health Checker fails during pre-down time or down time because of finding active ODI sessions.
Solution for Failure During Pre-Down Time
Ignore the failure and proceed.
Solution for Failure During Down Time
Run the following update statement and resume Upgrade Orchestrator.
UPDATE FUSION_ODI.SNP_SESSION SET SESS_STATUS='E' WHERE SESS_STATUS IN ('W', 'R', 'Q');
Problem
Health Checker reports the following error during the Default Keystore Size Check:
[ERROR]: Plugin 'DefaultKeystoreSizeCheck': HC-DKS-0002 : Following domain's default keystore file size is different from CommonDomain domain's default keystore file located at /u01/APPLTOP/instance/domains/admin-apps.oracleoutsourcing.com/CommonDomain/co nfig/fmwconfig/default-keystore.jks which has the size of 1,867 bytes. List of Domains that are different are: [Domain name: CRMDomain , File location: /u01/APPLTOP/instance/domains/admin-apps.oracleoutsourcing.com/CRMDomain/confi g/fmwconfig/default-keystore.jks, File size: 2658 Bytes] Corrective action: Contact Oracle Support to resolve the issue.(Checks the size of default-keystore.jks file in all domains)
Solution
This error can be ignored.
Problem
If a health check is taking too long or if it is hanging, it can be terminated.
Solution
To confirm a health check is hung, make sure that there has been no logging for more than 30 minutes by comparing the current timestamp and the last timestamp in the manifest_name.log.lck file. Perform the following steps to terminate a hanging health checker process.
Query the list of running processes for oracle.healthcheckplug.core.PluginManager using the following command:
ps -ef | grep oracle.healthcheckplug.core.PluginManager
Terminate the process id of the process returned by the previous command by using the following command:
kill -KILL java process id
After terminating the process, perform any corrective action and resume Upgrade Orchestrator.
This section provides information about the following troubleshooting issues:
Pre-Upgrade Task Fails Because SES Crawler Processes Cannot Be Stopped
Policy Store and Oracle Platform Security Services Versions Are Not Compatible
Troubleshooting Failures During Propagating Domain Configuration
RUP Lite for Domain Configuration Takes Too Long to Complete
Orchestration Fails With StartAllServers Task After Language Pack Upgrade on CRM
WLS SocketTimeoutException During Server Startup During Upgrade
Upgrade JDK in a Freshly Provisioned Release 6 Environment (AIX)
Problem
While downloading patches, as described in Section 2.3.5.2 or Section 2.3.6.3, you are setting environment variables to run the adCreateMosPlan.pl script. After you issue the setenv command for PERLLIB5, the following error occurs: Perl lib version (v5.8.3) does not match the executable version (v5.8.8).
Solution
Run the following commands:
export PERL_HOME=/u01/APPLTOP/dbclient/perl export PATH=/u01/APPLTOP/dbclient/perl/bin:$PATH
Then retry the setenv command.
Problem
SES crawler processes in the LAUNCHING state cannot be stopped by the pre-upgrade task, so the pre-upgrade task fails.
The following error message appears in the primordial host orchestration log. The schedule names listed in the following examples are the schedules that could not be stopped.
[2013-12-06T15:22:47.188+00:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 13] ERROR: Failed to stop the following Index Launching Schedules: [2013-12-06T15:22:47.188+00:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 13] [2013-12-06T15:22:47.188+00:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 13] Purchasing Contract Documents [2013-12-06T15:22:47.188+00:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 13] Purchasing Contracts [2013-12-06T15:22:47.188+00:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 13] References [2013-12-06T15:22:47.188+00:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 13] Sourcing Contracts [2013-12-06T15:22:47.188+00:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 13] FAILURE: SesDisableIndexSchedules failed.
Solution
Bounce the ESS server of the common domain and resume Upgrade Orchestrator.
Problem
The UpdateMDSSOAComposite task, which runs during the downtimePostFA phase, can cause a run time exception or stop responding for more than one hour without any updates to logs. This is an example of an exception if failure occurs:
java.lang.RuntimeException: javax.naming.NamingException: Unhandled exception in lookup [Root exception is org.omg.CORBA.MARSHAL:..
Solution
Note:
This task, when successful, may take a long time and you should ensure that the upgrade does not respond for more than an hour without any log activity before following these steps.
Perform the following steps.
Look for the Java processes that are running by using command, ps -ef | grep java, and terminate the Java processes by using command, kill -9 <process ID>.
Execute the command, cd ORCH_LOCATION/config.
Back up the release manifest file. This is either rel5-7_primordial.xml for a multi- hop upgrade or rel7_primordial.xml for a single hop upgrade from Release 6 to 7.
In the backup file, look for the plug-in name, such as <plugin id="UpdateMDSSOAComposite>.
Lines should be associated with the plug-in entry as shown in the following example:
%APPLICATIONS_BASE%/fusionapps/applications/oui/jlib/ xmlparserv2.jar;%APPLICATIONS_BASE%/fusionapps/ wlserver_10.3/server/lib/wlclient.jar; %APPLICATIONS_BASE%/fusionapps/wlserver_10.3/server/ lib/weblogic.jar"active="true"
Remove the wlclient.jar line only, and save the file. The new file looks as follows:
%APPLICATIONS_BASE%/fusionapps/applications/oui/jlib/xmlparserv2.jar;%APPLICATIONS_BASE%/fusionapps/wlserver_10.3/server/ lib/weblogic.jar"active="true"
Resume Upgrade Orchestrator on the primordial host.
Problem
After upgrading to Release 11.1.1.7.0, you receive the following error while connecting to ODI Studio:
oracle.security.jps.service.policystore.PolicyStoreIncompatibleVersionException JPS-06100: Policy Store version 11.1.1.7.0 and Oracle Platform Security Services Version 11.1.1.6.0 are not compatible.
Solution
Upgrade or reinstall the ODI studio component from Release 11.1.1.7.0. For more information, see "Setting Up Oracle Data Integrator Studio" in the "Human Capital Management" chapter in the Oracle Fusion Applications Post-Installation Guide and "Installing Oracle Data Integrator" in the Oracle Fusion Middleware Installation Guide for Oracle Data Integrator.
Problem
An error occurred during the Bootstrapping Patch Manager configuration assistant.
Solution
An error during Bootstrapping Patch Manager normally occurs only when the database is down. Ensure that the database is up and running. You can review the related log files in this location:
APPLICATIONS_CONFIG/lcm/logs/11.1.7.0.0/RUP/FAPatchManager_bootstrap_timestamp.log
This section contains information about troubleshooting issues that may occur during the Propagating Domain Configuration configuration assistant. The following topics are discussed:
Problem
The Propagating Domain Configuration tasks is taking too long to complete.
Solution
This configuration assistant can take some time to complete as it is highly dependent on the environment, specifically the number of non-admin domains and the responsiveness of the file system.
You can monitor the progress of this configuration assistant by reviewing log files in this location:
APPLICATIONS_CONFIG/lcm/admin/version/fapatch//ruplitedomain/output/logs
To confirm this configuration assistant was successful, verify that the config/fusionapps_start_params.properties file exists under each local or non-admin split domain. Also ensure that the bin/setDomainEnv.sh file under each local or non-admin split domain contains the following row:
POST_CLASSPATH="${COMMON_COMPONENTS_HOME}/modules/oracle.appstrace_11.1.1/appstrace.jar${CLASSPATHSEP}${POST_CLASSPATH}"
export POST_CLASSPATH
Problem
The Propagating Domain Configuration configuration assistant fails if there are WARs or EARs installed or deployed that are not accessible from the primordial host where RUP Installer is running. An example of the error caused by this condition follows:
<< read domain from APPTOP/instance/domains/server.company.com/SCMDomain << write template to APPLICATIONS_CONFIG/lcm/admin/11.1.7.0.0/fapatch/ruplitedomain/output/templates/SCMDomain.jar >> fail: Unable to locate file: /fusionapps/localdomain/domains/server.company.com/SCMDomain/datalens/datalens.war >> fail: write template to "APPLICATIONS_CONFIG/lcm/admin/11.1.7.0.0/fapatch/ruplitedomain/output/templates/SCMDomain.jar" CFGFWK-60550: Script execution aborted. The script may contain an error. Unable to locate file: /fusionapps/localdomain/domains/server.company.com/SCMDomain/datalens/datalens.war
Solution
To resolve this issue, you must undeploy or uninstall the WAR or EAR, which is datalens.war in this example. Then resume orchestration. After the upgrade has completed successfully, you can install or deploy the WAR or EAR.
Problem
RUP Lite for Domain Configuration takes too long to complete.
Solution
This utility can take some time to complete as time taken to propagate domain configuration is highly dependent on the environment, specifically the number of non-admin domains and the responsiveness of the file system. Note this issue is seen only in local domain environments where the utility is run between RUP Installer Part 1 and Part 2. This is not an issue for Oracle VM environments or other environments with shared domains.
Problem
The following error occurs if the BI Presentation servers are running during the deployment of BI Publisher artifacts:
java.lang.RuntimeException: Webcat patch file creation failed!
Solution
If you upgrade to a release that contains BI Publisher artifacts, the BI Presentation servers must not be running. To resolve this issue, shut down the BI Presentation servers to release locks on the Oracle BI Presentation Catalog. For more information, see "fastartstop Syntax" in the Oracle Fusion Applications Administrator's Guide.
Problem
On the primordial host, orchestration may fail while running the general system checks as part of predowntime checks.
Solution
Perform the following steps to comment out the check.
Open the manifest file located at APPLICATIONS_BASE/fusionapps/applications/lcm/hc/config.
Search for PatchInventoryCheckPlugin.
Comment out the plug-in by adding <!-- before the plug in and --> after it, as shown in the following example.
<!-- <plugin id="PatchInventoryCheckPlugin" description="Verify Patch Inventory in FA_ORACLE_HOME" invoke="" plugin.class="oracle.check.apps.PatchInventoryCheckPlugin" class.path="$HC_LOCATION/lib/precheckplugin.jar; $HC_LOCATION/lib/hccommon.jar" stoponerror="false"/> -->
Problem
The configuration assistant for importing IPM artifacts fails with the following error:
importIPMApplication() & importIPMInput() WLST commands have not run successfully
Solution
Follow the instructions in Steps 1 through 7 in "Prerequisites for the Deployment of IPM Artifacts" in the Oracle Fusion Applications Patching Guide. Then resume Upgrade Orchestrator.
Problem
In GUI mode, when installing a language pack, if any configuration action fails and then succeeds on retry, and subsequently all configuration actions were successful, the installer incorrectly shows a popup with the following message:
Configuration is completed with errors, exit the installer by clicking the 'Cancel' button and rerun the installer to retry the failed configurations.
Solution
Perform the following steps:
Cancel the installer.
Relaunch the installer in checkpoint mode. The configuration screen appears, showing no configuration actions because all actions were successful earlier.
Click Next to go to the Installation Complete screen.
Click Finish to exit the installer.
Problem
Orchestration tries to restart all servers after a Language Pack upgrade. On CRM PODs, there may be failures in starting the IIR server, which may be reported as the following error:
ORCH-DOWNTIME-SS-00005 : Failed to start all servers. Review log file /u01/APPLTOP/instance/lcm/logs/11.1.7.0.0/orchestration/host_name-rel5-7_midtier_timestamp.log for details on the failures to take appropriate corrective action. (Bounce All Servers).
Solution
Perform the following steps.
Review the orchestration log file at /u01/APPLTOP/instance/lcm/logs/11.1.7.0.0/orchestration/hostname-rel5-7_midtier_timestamp.log, and check for any high level failures.
Review all fa_control logs on the failed host and look for details on the server that failed.
If the IIR server is the only server that failed to start, update the status of the task to Success using the following updateStatus command, and resume Upgrade Orchestrator. You can restart the IIR server manually after the upgrade.
./orchestration.sh updateStatus -pod POD_NAME -hosttype host_type -hostname host_name -release REL7 -phase DowntimePostLP -taskid StartSeversAfterLP -taskstatus success
Problem
As an intermittent issue, there could be WLS socket exceptions during server startup, or during any other upgrade tasks. An example of the exception is:
bea_wls_management_internal2/Bootstrap, user: FUSION_APPS_PROV_PATCH_APPID java.net.SocketTimeoutException: Read timed out at jrockit.net.SocketNativeIO.readBytesPinned(Native Method) at jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
Solution
Find the managed server or the administration server that encounters the failure, and manually restart the server. Proceed with the upgrade by resuming Upgrade Orchestrator on the failed host.
Problem
An orchestration process can fail when the checkpoint system cannot be initialized with an error message as shown in the following example:
"Failed to load prevayler under APPTOP/11.1.7.0.0/orchestration/bin/../checkpoint/REL7/host_name/PRIMORDIAL/DowntimePreFA/snapshot: Chunk header corrupted."
Solution
Review he log file to ensure that there is no "out of disk space" exception. If there is no "out of disk space" exception, resume orchestration on the host where the failure occurred to continue the upgrade. If there is an "out of disk space" exception, ensure that there is enough disk space and then resume orchestration.
Perform the following steps to upgrade JDK only if you are upgrading from an Oracle Fusion Applications environment that was freshly provisioned in Release 6. If you upgraded to Release 6 from Release 5, you can skip this step.
Set the following environment variables, in addition to those set in Section 2.1.13, "Set Environment Variables".
UPGRADE_JDK_PATTERN: The version to which you are upgrading. You can get this JDK version from REPOSITORY_LOCATION/jdk6.
PREVIOUS_JDK_BACKUP_PATTERN: The current version of JD. You can get this version from APPLICATIONS_BASE/jdk6.
JAVA_HOME: The java home directory.
Run the following command from REPOSITORY_LOCATION/installers/farup/Disk1/upgrade/bin:
(UNIX) upgradeJDK.sh (Windows) upgradeJDK.bat
Review the log files for upgradeJDK which are located in the FA_ORACLE_HOME/admin/FUSION/log/upgradeJDK directory.
Perform the following steps to upgrade JDK only if you are upgrading from an Oracle Fusion Applications AIX environment that was freshly provisioned in Release 6. If you upgraded to Release 6 from Release 5, you can skip this step.
Set the following environment variables, in addition to those set in
Section 2.1.13, "Set Environment Variables".
UPGRADE_JDK_PATTERN: The version to which you are upgrading. You can get this JDK version from REPOSITORY_LOCATION/jdk6 by running the following command:
./java -fullversion 2>&1 | cut -d " " -f 9
PREVIOUS_JDK_BACKUP_PATTERN: The current version of JDK. You can get this version from APPLICATIONS_BASE/jdk6. by running the following command:
./java -fullversion 2>&1 | cut -d " " -f 9
JAVA_HOME: The java home directory.
Run the following command from REPOSITORY_LOCATION/installers/farup/Disk1/upgrade/bin:
(AIX) upgradeJDK.sh
Review the log files for upgradeJDK which are located in the FA_ORACLE_HOME/admin/FUSION/log/upgradeJDK directory.
The following ignorable errors may be encountered while running the catbundle.sql script or its rollback script:
ORA-29809: cannot drop an operator with dependent objects
ORA-29931: specified association does not exist
ORA-29830: operator does not exist
ORA-00942: table or view does not exist
ORA-00955: name is already used by an existing object
ORA-01430: column being added already exists in table
ORA-01432: public synonym to be dropped does not exist
ORA-01434: private synonym to be dropped does not exist
ORA-01435: user does not exist
ORA-01917: user or role 'XDB' does not exist
ORA-01920: user name '<user-name>' conflicts with another user or role name
ORA-01921: role name '<role name>' conflicts with another user or role name
ORA-01952: system privileges not granted to 'WKSYS'
ORA-02303: cannot drop or replace a type with type or table dependents
ORA-02443: Cannot drop constraint - nonexistent constraint
ORA-04043: object <object-name> does not exist
ORA-29832: cannot drop or replace an indextype with dependent indexes
ORA-29844: duplicate operator name specified
ORA-14452: attempt to create, alter or drop an index on temporary table already in use
ORA-06512: at line <line number>. If this error follow any of above errors, then can be safely ignored.
ORA-01927: cannot REVOKE privileges you did not grant
Perform the steps in "Verifying Installation" in the "Common" chapter in the Oracle Fusion Applications Post-Installation Guide.
This section contains troubleshooting information for platform specific issues.
This section contains troubleshooting information for the following issues on SUSE Linux.
Problem
The following is error is reported when the GeneralSystemHealthChecks runs.
Plugin 'OsCheck': HC-OS-00002 : OS Support Check failed.Error(s): OS version is not supported(Check for OS name,arch and version support)
Solution
Back up the FA_ORACLE_HOME/lcm/hc/healthchecks.xml file. Then edit this file to replace all instances of "SLES 11.*::" with "SUSE Linux Enterprise Server 11.*::".
Problem
The following error is reported when the GeneralSystemHealthChecks fails on HostsCheck:
[oracle.healthcheckplug] [tid: 10] [ecid: ] Plugin 'HostsCheck': HC-COMMON-00001 :Unable to perform the check: java.io.FileNotFoundException: /etc/sysconfig/network (Is a directory) Review log files for additional details to take an appropriate corrective action(Hosts Name)
Solution
You can ignore this error and proceed with the upgrade.
This section contains troubleshooting information for the following issues on Windows:
Problem
The DowntimePostFA phase of orchestration fails during the RemoveConflictingPatches task on Windows with the following error:
RollbackSession rolling back interim patch '16569379' from OH 'c:\AT\webtier_mwhome\webtier' Prerequisite check "CheckActiveFilesAndExecutables" failed. The details are: Following files are active : c:\AT\webtier_mwhome\webtier\bin\yod.dll
Solution
The cause of the failure is that the OPMN processes that is running from the BI and GOP homes are using this dll. When this failure occurs, shut down the OPMN and the OPMN-managed processes using the respective services. After making sure that the OPMN processes are down, restart orchestration. After orchestration succeeds, bring up the OPMN processes by using the respective services.
Problem
Upgrade JDK fails with the following error:
Upgrade JDK plugin command: C:\R\installers\farup\Disk1\upgrade\bin\upgradeJDK.bat --apptop C:\AT --repo C:\R [2013-07-02T14:24:34.566-06:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 12] [ecid: 0000JyWzVIIFW7HpIsDCif1HonA^000003,0] Tue 07/02/2013 14:24:34.56 upgradeJDK BEGIN [2013-07-02T14:24:34.582-06:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 12] [ecid: 0000JyWzVIIFW7HpIsDCif1HonA^000003,0] Tue 07/02/2013 14:24:34.57 Output logged to file C:\AT\fusionapps\applications\admin\FUSION\log\upgradeJDK\upgradeJDK_14243455. log [2013-07-02T14:24:34.610-06:00] [orchestration] [NOTIFICATION] []
Solution
Set the following environment variables:
set APPLICATIONS_BASE=APPLICATIONS_BASE LOCATION> set REPOSITORY_LOCATION=C:\SHARED\11.1.7.0.0\Repository
Then in the same command prompt, start orchestration on the primordial node.
|
 Copyright © 2011, 2014, Oracle and/or its affiliates. All rights reserved. Legal Notices |
|