When Upgrade Orchestrator exits with a failure on any upgrade task, it sends an email to the recipients specified in the EMAIL_TO_RECIPIENT
and EMAIL_CC_RECIPIENT
properties in the pod.properties
file. This email contains the Oracle Fusion Applications Orchestration Report as an attachment. The report name is FAOrchestrationReport_
release
_
hosttype
_
hostname
_
timestamp
.html
. This report specifies the location to the Fusion Applications Orchestration Action Summary report,
which provides information about the failure, corrective action, and relevant log files. The orchestration log file is a good point to start for any troubleshooting, as it captures logs from different upgrade tasks as well as console messages. The orchestration log file is located in APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/orchestration/
host_name
-rel12
_hosttype
_timestamp
.log
.
The following figure shows the high level flow for troubleshooting Upgrade Orchestrator failures:
Figure 9-1 Troubleshooting Flow
Previous reports are archived whenever a new report is generated, as described in Unable to Find the Orchestration Report After Failure. For more information about the report, see Oracle Fusion Applications Orchestration Report.
Note that if an orchestration session exits due to an error, its status is "Failed". If an orchestration session exits as a result of the exitOrchestration
command, its status is "Terminated".
The Classloader Analysis Tool (CAT) is a Web-based class analysis tool that simplifies filtering classloader configuration and helps in the analysis pf classloading issues, such as detecting conflicts, debugging application classpaths and class conflicts, and proposes solutions to help resolve them. Starting with 11g Release 10 (11.1.10), CAT is deployed to every WebLogic domain during the upgrade. For more information about the Classloader Analysis Tool, see the Oracle Fusion Middleware Developing Applications for Oracle Weblogic Server Guide.
In addition to the orchestration log file, APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/orchestration/
host_name
-rel12
_hosttype
_timestamp
.log
, the following types of log files are available:
The following table contains a list of log directories for Upgrade Orchestrator activities. Note that the directory name, 11.12.x.0.0
, is used to represent the appropriate version of Release 12 being upgraded to. For IDM and OHS log files, the location can be configured using the LOG_LOCATION
property, in the IDM.properties
and OHS.properties
files. For more information, see Update Orchestrator Properties Files.
Table 9-1 Upgrade Tasks and Related Log Files
Task Display Name and ID | Log File Location |
---|---|
Stopping Index Schedules and Deactivating Index Optimization (StopIndexSchedules) |
|
Stopping All Servers (StopAllServers) |
Orchestration log files:
Control log file:
|
Setting CrashRecoveryEnbled Property to False (DisableCrashRecoveryEnabled) |
|
Stopping OPMN Control Processes (StopOPMNProcesses) |
Orchestration log files:
OPMN log file:
|
Stopping Node Managers (StopNodeManager) |
|
Running Upgrade Readiness (During Downtime) Checks (DuringDowntimeChecks) |
|
Removing Conflicting Patches for Oracle Fusion Middleware Component Oracle Homes (RemoveConflictingPatches) |
|
Installing Oracle Fusion Applications LCM Tools for Oracle VM (InstallFaSaasLcmTools) |
|
Preparing for Oracle Fusion Applications LCM Tools for Oracle VM Upgrade (PrepareLCMToolsForOVMUpgrade) |
|
Applying Oracle Fusion Applications LCM Tools for Oracle VM Patches (ApplyLCMToolsForOVMPatches) |
|
Running RUP Lite for OVM in Offline Mode as Application User (RupLiteOvmOffline) |
|
Running RUP Lite for OVM in Pre-Root Mode |
|
Running Oracle Fusion Applications RUP Installation Part 1 of 2 (RunFirstRUPInstaller) |
Orchestration log file:
Oracle Fusion Applications Patch Manager log file:
|
Running RUP Lite for Domain Configuration (RunRUPLiteForDomainsConfig) |
Orchestration log file
RUPLite for Domain Config log file:
|
Starting Node Managers (StartNodeManager) |
Orchestration log file:
Oracle Fusion Applications Control log file:
|
Starting OPMN Control Processes (StartOPMNProcesses) |
Orchestration log files:
|
Running Oracle Fusion Applications RUP Installation Part 2 of 2 (RunSecondRUPInstaller) |
Orchestration log file:
Oracle Fusion Applications Patch Manager log file:
|
Invoking an Instance of UpdateSOAMDS SOA Composite (UpdateMDSSOAComposite) |
|
Preparing for Oracle Fusion Applications WebTier Upgrade (CopyWebtierUpgradeToCentralLoc) |
|
Stopping Oracle Fusion Applications - APPOHS (StopOPMNProcesses) |
Orchestration log files:
OPMN logs:
|
Removing Conflicting Patches for Oracle Fusion Applications WebTier Oracle Homes (RemoveConflictingPatches) |
|
Upgrading Oracle Fusion Applications OHS binaries (UpgradeOHSBinary) |
Orchestration log files:
WebGate log file:
|
Upgrading Oracle Fusion Applications OHS Configuration (UpgradeOHSConfig) |
Orchestration log files:
RUP Lite log file:
|
Running RUP Lite for BI (RunRUPLiteForBI) |
|
Running RUP Lite for OVM in Online Mode as Application User (RupLiteOvmOnline) |
|
Setting CrashRecoveryEnabled Property to True (EnableCrashRecoveryEnabled) |
|
Running Post Upgrade Health Checks (PostUpgradeChecks) |
|
Running Data Quality Checks (DataQualityChecks) |
|
Running Post Upgrade Cleanup Tasks (PostUpgradeCleanup) |
|
Installing binaries for Incremental Provisioning (InstallIpBinary ) |
|
Generating silent response files for Incremental Provisioning (GenerateSilentIpResponseFiles |
|
Running Incremental Provisioning Manually (RunIncrementalProvisioningManually) |
Orchestration log file:
Incremental Provisioning log file:
Provision Plan file:
|
Upgrading All Installed Languages (LanguagePackInstall) |
|
Stopping All Servers (StopServersAfterLP) |
|
Starting All Servers (StartSeversAfterLP) |
|
Running Post Language Pack Health Checks (PostLangPackChecks) - This task calls both general system and post LP upgrade checks |
|
The following table contains a list of log directories for RUP Installer activities:
Table 9-2 Log Directories for RUP Installer Activities
Log directory name | Description |
---|---|
|
Installation phase and Oracle Fusion Middleware patch set installation logs. |
|
Top level directory for RUP Installer logs. Only the messages that indicate that a configuration assistant started and the result of its processing, such as success or error, are written to this log file. |
|
Top level log directory where log files are moved when retrying the installation session. |
|
Top level log directory for configuration assistants. A log file exists for each configuration assistant. All messages that are generated during the configuration assistant processing are written to this log file. |
|
Database upload configuration assistant logs after failure or completion. For more information, see Troubleshoot Loading Database Components. |
|
Oracle Business Intelligence logs. |
|
StartStop utility logs. Server logs are located under respective domains. For example, the AdminServer log for CommonDomain is under |
|
SOA artifacts configuration assistant logs. SOA server logs are located under respective domains. For example, the SOA server logs for CommonDomain are under |
|
Applying Downloaded Patches configuration assistant logs. |
It is possible to track the progress of the upgrade by monitoring the console output that shows processes running on the primordial host and also open another session that tails the logs for the other servers. It is also possible to monitor the progress of the upgrade using the following methods:
To get the upgrade status for that host or for other hosts, run the getStatus
command on any host as follows:
Retrieve the status of all tasks in a phase ./orchestration.sh getStatus -pod fcsm -hosttype PRIMORDIAL -hostname host_name -release 11.12.x.0.0 -phase predowntime
Retrieve all tasks with a specific status ./orchestration.sh getStatus -pod fscm -hosttype PRIMORDIAL -hostname host_name -release 11.12.x.0.0 -taskstatus success
Retrieve all tasks with a specific status ./orchestration.sh getStatus -pod fscm -hosttype PRIMORDIAL -hostname host_name -release 11.12.x.0.0 -taskid HostTypeValidatePlugin
Table 10-3 displays a complete list of options for the orchestration.sh
getStatus
command.
The report command generates a report, in both HTML and XML formats, that provides information about the percentage of the upgrade that is complete. It can also inform of any processes that may be hanging. The command follows:
(Unix) cd ORCH_LOCATION/bin ./orchestration.sh report -pod comma_separated_list_of_POD_NAMES -release release_version
If any upgrade tasks fail, the Fusion Applications Orchestration Report is generated and mailed as an attachment to the user specified in the EMAIL_TO_RECIPIENT
and EMAIL_CC_RECIPIENT
properties in the pod.properties
file. For more information, see Oracle Fusion Applications Orchestration Report. For information about troubleshooting failures, refer to the appropriate section in Monitor and Troubleshoot the Upgrade to resolve the issue. After a failure, restart Orchestrator on the host where it failed, using the same commands used in Run Upgrade Orchestrator During Downtime.
If any configuration assistants fail while RUP Installer is running, Upgrade Orchestrator does not display a message, fail, or send an email until RUP Installer exits with a failure.
If the Loading Database Components configuration assistant in RUP Installer fails, an email notification is received only when all workers are in a FAILED or IDLE (no tasks assigned to it) state. To resolve this type of issue, follow the steps listed in Database Worker Fails While Loading Database Components.
Orchestration can be terminated on all hosts under scenarios that require the issue of an exit command across the entire environment. It may be needed to terminate an orchestration session on a pod for reasons such as, not being able to complete the upgrade within a certain time, or unexpected issues that may require significant time to resolve, for example.
This section describes the commands used to manage the termination of orchestration on all hosts in the following topics:
The following command terminates the orchestration session on all hosts across all host types in the specified pod. This command can be run from any individual host for the entire environment and/or pods:
cd /ORCH_LOCATION/bin ./orchestration.sh exitOrchestration –pod POD_NAME –hosttype host_type
The hosttype
argument must match the host from which this command is issued. Upgrade Orchestrator sends a notification after all hosts exit from orchestration. After this notification is received, run the command to clear the exit status on all hosts, as described in Clear the Exit Status on All Hosts. If this notification is not received on a timely basis, the status of the request can be found by running the command described in Get the ExitOrchestration Status.
From the time exitOrchestration
is issued, until clearExitOrchestration
is issued, only the getExitOrchestrationStatus
command can be issued on the pod
. Additionally, exitOrchestration
can be issued from any host but is applicable for all the hosts in an environment.
After the notification that confirms all hosts exited from orchestration is received, run the following command to clear the exit status on any individual host for the entire pod:
cd /ORCH_LOCATION/bin ./orchestration.sh clearExitOrchestration -pod POD_NAME –hosttype host_type
After this command runs, users can continue with the upgrade or take other appropriate actions on the pod.
To cancel the upgrade and to restore the system, first terminate orchestration by following the steps in Terminate Upgrade Orchestration. After orchestration terminates successfully, restore the system from the backup that was taken before starting the upgrade.
In addition to restoring the environment from the backups, perform the following steps to restore and clean up the orchestration files:
The following specific troubleshooting scenarios are described in this section:
Problem
When orchestration is relaunched for any reason, there could be an error uploading checkpoint files to the appropriate location. In this case, Upgrade Orchestrator exits with the following errors.
Unable to upload orchestration checkpoints under /fsnadmin/upgrade/fusionChangeOps/11.12.x.0.0/orchestration/bin/../checkpoint. Corrective Action: Remove any existing files from older Orchestration run in /fsnadmin/upgrade/fusionChangeOps/11.12.x.0.0/orchestration/bin/../checkpoint before you proceed.
Solution
Perform the required corrective action suggested in the error message and then resume orchestration to proceed with the upgrade.
Problem
Orchestration hangs during the preDowntime
or downtimeFA
phase, or there is a need to exit Upgrade Orchestrator in the middle of an upgrade for any valid reason. If Upgrade Orchestrator hangs, it sends an email with a subject of "ALERT: POD_NAME
: Orchestration on host_type host_name
could potentially be hung." The email includes the cause of the hang and the log file location.
Solution
If orchestration results in any hanging tasks on any host, do not use ctrl-C or ctrl-Z to exit. Follow the steps in Terminate Upgrade Orchestration. It is possible to run these commands from another console, on any host in the pod, to gracefully exit orchestration. The exitOrchestration
command terminates the upgrade on all hosts. Therefore, after resolving the issue that caused the hanging task, resume orchestration on all hosts where orchestration was previously running.
Problem
After Upgrade Orchestrator fails, the console reports the following example information:
Fusion Applications Orchestration Report: /u01/orchestration/orchreports/FAOrchestrationReport_release_hosttype_hostname_timestamp.html
This html file does not exist in the /u01/orchestration/orchreports
directory.
Solution
As the upgrade progresses, the Orchestration report is archived after the failure or completion of each task. Find the output in the following directory, based on the example:
/u01/orchestration/orchreports/ARCHIVE/release/timestamp/FAOrchestrationReport_release_hosttype_hostname_timestamp.html
Problem
Upgrade Orchestrator fails while generating the Orchestration report with the following error:
"Java.lang.OutOfMemoryError: PermGen space
Solution
Increase the ORCH_JVM_OPTION
value in pod.properties
to allocate more memory for both the startup of JVM and the total size of permgen
, as shown in the following example:
ORCH_JVM_OPTION=-Xmx2048m -XX:PermSize=256M -XX:MaxPermSize=512M
Problem
Property validation fails during the PreDowntime
phase with the following error:
Invalid property: must specify ORCHESTRATION_CHECKPOINT_LOCATION in orchestration properties file ./../config/POD_NAME/pod_properties.
No log file or HTML file is generated.
Solution
Populate the ORCHESTRATION_CHECKPOINT_LOCATION
mandatory property in the pod.properties
file. Note that no logs are generated for this type of failure, by design.
Problem
The updateStatus
command was run to manually set the status of a failed task_id
on the primordial host to "success" during the DowntimePostFA
phase, for example. After resuming orchestration on the IDM host, it fails with the following error:
Wait for peer phase: PRIMORDIAL:DowntimePostFA on host.mycompany.com Found peer phase: PRIMORDIAL:DowntimePostFA on host.mycompany.com Error.
The results of getStatus
on the pod shows that all tasks were successful but the DowntimePostFA
phase was in error status.
Solution
Setting a task status to "success" does not resolve a "Wait for peer phase" error, because a phase level status cannot be updated by the updateStatus
command. The only way to resolve a "Wait for peer phase" issue is to resume orchestration so that it can verify that all tasks in the phase were successful.
Problem
An orchestration task is no longer running and the following error is reported:
Orchestration step: DowntimePreFA DeploySoaShared Running Unable to update task status from Running to Success Oracle Fusion Applications Release Upgrade Orchestration failed.
Solution
Before performing the step in this solution, confirm that there are no orchestration processes running. Then run the updateStatus
command to change the status of the task specified in the error message to error. Then resume Upgrade Orchestrator.
Upgrade Orchestrator supports only the following status transitions:
Error to Success
Running to Error
ManualStep to Success
Success to Error
Problem
The emails that Upgrade Orchestrator sends upon failure are not being received.
Solution
Perform the following steps to check if the mail server is configured properly:
Problem
Upgrade Orchestrator is not using a value that was recently added to one of the properties files.
Solution
If the properties file was updated after launching Upgrade Orchestrator, follow the steps to safely exit orchestration in Upgrade Orchestrator Hangs and then relaunch orchestration. Upgrade Orchestrator reads property file values only before launching.
Problem
While running various commands for Upgrade Orchestrator, the following error is reported:
Stale NFS file handle
Solution
If the Stale NFS file handle error is reported while running any of the plug-ins in orchestration or the getStatus
or updateStatus
commands, verify that all mount points provided in the various property files are actually accessible. For more information, search for mount point in Upgrade Orchestrator Properties Files.
Problem
The following error is reported:
[apps] [ERROR] [] [oracle.apps.ad.rupconfig.Creating_Middleware_Schemas] from oracle.security.audit.config.dynamic.persistence.internal.ldap.AuditStoreDataManager searchFilterPresets
Solution
This error can be ignored.
Problem
The following error causes Upgrade Orchestrator to fail:
rm: cannot remove `/u01/ORCH/orchestration/INIT/mycompany.com/IDM/INIT/snapshot/.nfs00000000015595b30000004b': Device or resource busy Oracle Fusion Applications Release Upgrade Orchestration failed.
Solution
Remove the file that is causing the error and restart Upgrade Orchestrator.
Problem
During orchestration, a process can fail when the checkpoint system cannot be initialized, and the following error message is reported:
Failed to load prevayler under path_for_snapshot: Chunk header corrupted in the log file.
Solution
Perform the following steps to resolve this issue:
Problem
Orchestration fails when running the BackupOPSS
plug-in, which backs up the OPSS Security Store, with the following error:
ORCH-DOWNTIME-00001 : Plugin Failed with error. Please enter bind password: ldap_search: No such object
This error occurs because the OPSS Security Store assumes that jpsroot
is FAPolicies
.
Solution
If the jpsroot
is not FAPolicies
, the workaround is to back up the OPSS Security Store manually. To back up all data under the root node of the OPSS Security Store and to back up the bootstrap wallet, perform the following steps:
Ensure the backups are performed in directories from which they can be restored. Use any directory to back up the data, as long the location from where to restore the backup is known.
Using Fusion Applications Control, perform the following steps to identify the root node in the Oracle Internet Directory that hosts the OPSS Security store:
Open the Farm_CommonDomain.
Open the WebLogic Domain.
Open the CommonDomain.
Find the domain name of the root node under Root Node Details, which is under the Edit Security Provider region. In the case of an upgrade failure, restore this entire node.
Perform the following ldifwrite
and bulkload
operations on the system where the Oracle Internet Directory hosting the OPSS Security store resides. When initiating ldifwrite
and bulkload
, Oracle Internet Directory requires the Oracle Internet Directory process and the database behind Oracle Internet Directory to be up and running:
Set the following environment variables:
(Unix) setenv ORACLE_HOME OID_ORACLE_HOME setenv ORACLE_INSTANCE OID_INSTANCE_HOME
For example:
(Unix) setenv ORACLE_HOME /u01/oid/oid_home setenv ORACLE_INSTANCE /u01/oid/oid_inst
Create the backup in the SHARED_UPGRADE_LOCATION/POD_NAME/release/
directory as follows:
In the system where the Oracle Internet Directory is located, produce an LDIF file by running ldifwrite
as illustrated in the following command. The Operational Data Store (ODS) password is required.
OID_HOME/ldap/bin/ldifwrite connect="srcOidDbConnectStr" basedn="cn=FAPolicies", "c=us" ldiffile="srcOid.ldif"
For example:
/u01/oid/oid_home/ldif/bin/ldifwrite connect="oidddb" basedn="cn=FAPolicies" ldiffile="srcOid.ldif"
This command writes all entries under the node cn=FAPolicies
to the file srcOid.ldif
. After generated, move this file to the directory that was previously identified, to hold all backup data.
Perform the following steps if there is a need to restore the backup:
Ensure Oracle Internet Directory is up and running.
Perform a bulkdelete
on Oracle Internet Directory nodes.
In the Oracle Internet Directory system, verify that there are no schema errors or bad entries by running bulkload
, as illustrated in the following command:
OID_HOME/ldap/bin/bulkload connect="dstOidDbConnectStr" check=true generate=true restore=true file="fullPath2SrcOidLdif"
If duplicate DNs (common entries between the source and destination directories) are detected, review them to prevent unexpected results.
Load data into the Oracle Internet Directory by running bulkload
as shown in the following command:
OID_HOME/ldap/bin/bulkload connect="dstOidDbConnectStr" load=true file="fullPath2SrcOidLdif"
Back up the cwallet.sso file
in the DOMAIN_HOME
/config/fmwconfig/bootstrap
directory for each WLS domain in an Oracle Fusion Applications installation. Take backups of each cwallet.sso
file for each domain and when restoring, be careful to restore the correct file. For example, cwallet.sso
is backed up from the Common Domain, then restore it in the Common Domain upon failure. If cwallet.sso
is backed up from the BI domain, restore it to the BI Domain upon failure.
Resume orchestration to proceed with the upgrade.
Problem
The Database Credential Store Retrofit Utility or CSF Cache Utility steps fail. Sometimes, these steps fail because of a case mismatch in the value of the db unique name when compared between the value you get from the table (V$DATABASE
) and the one listed in the oratab
file on the db host.
Solution
SELECT DB_UNIQUE_NAME FROM V$DATABASE
If the result value is all lower case, ignore the next step.
If the value is all upper case or mixed case, replicate the case in the /etc/oratab
file exactly as seen in DB_UNIQUE_NAME FROM V$DATABASE
.
After the upgrade, revert the changes.
This section provides information about the following RUP Installer failures:
RUP Installer runs numerous configuration assistants and is the primary utility called by Upgrade Orchestrator. In the case of a failure while RUP Installer is running, refer to information in General Troubleshooting for Upgrade Orchestrator Failure applies. In addition to the Oracle Fusion Applications Orchestration Report and the log location, the RUP Installer Report location is also included as part of the notification that is sent. For more information, see Review the Post RUP Installer Report. For information about specific configuration assistants, see RUP Installer Configuration Assistants.
Most configuration assistants are configured to automatically retry after a failure. The default number of retries is three times and the default wait time between retries is two minutes. After ten minutes, no further retries are attempted. When reviewing log files after a failure, the number of retry attempts is indicated by "Autoretrying attempt "{0}" of "{1}".
Problem
The pre-copy phase of RUP installer fails, causing the installer to terminate.
Solution
To resolve this failure during the first RUP Installer, perform the following steps:
Open the checkpoint.xml
file located in the central_inventory_location/
checkpoint/farup1/
version
/ directory.
Edit the following line in the checkpoint file:
<module name="install" type="install" status="Success">
Update the value for status as follows:
<module name="install" type="install" status="Config">
To resolve this failure during the second RUP Installer, perform the following steps:
Open the checkpoint.xml
file located in the central_inventory_location/
checkpoint/fusionapps/
version
/
directory.
Edit the following line in the checkpoint file:
<module name="install" type="install" status="Success">
Update the value for status as follows:
<module name="install" type="install" status="Config">
Problem
RUP Installer fails due to thread calls and reports errors similar to the following example:
"Thread-11" id=29 idx=0x98 tid=25751 prio=5 alive, native_blocked at java/io/UnixFileSystem.getBooleanAttributes0(Ljava/io/File;)I(Native Method) at java/io/UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228) at java/io/File.exists(File.java:733)
Solution
Restart RUP Installer by resuming Upgrade Orchestrator.
Problem
An installer session was shut down during the upgrade.
Solution
If orchestration or tasks spawned by orchestration, such as RUP Installer, are killed in the middle of any process, the system may not be in a recoverable state and the state should be carefully reviewed by contacting Oracle Support before proceeding.
To recover from this situation, restore the backup of APPLICATIONS_BASE
and start from the beginning of the upgrade.
Problem
java.library.path
setting error opatch disablecas
operation for the fusionbhd
component:
[2017-07-22T15:03:59.188+00:00] [log] [NOTIFICATION] [] [ApplyPrePSAMiddlewarePatches2017-07-22_02-57-35PM.log] [tid: 1] Job validation successful, setting the req env to run the job... [2017-07-22T15:03:59.188+00:00] [log] [NOTIFICATION] [PAPUTL020] [ApplyPrePSAMiddlewarePatches2017-07-22_02-57-35PM.log] [tid: 1] Executing the command [<APPLTOP>/fusionbhd/OPatch/opatch disablecas -oh <APPTOP>/fusionbhd -jre <APPLTOP>/fusionapps/jdk] [2017-07-22T15:03:59.188+00:00] [log] [NOTIFICATION] [] [ApplyPrePSAMiddlewarePatches2017-07-22_02-57-35PM.log] [tid: 1] Oracle Home is: <APPLTOP>/fusionbhd The java.library.path system variable is missing or invalid. Please set java.library.path with a correct value and retry the operation.
Solution
Backup <APPLTOP>/fusionbhd/oui/oraparam.ini
.
-d64
to the JRE_MEMORY_OPTIONS
in <APPLTOP>/fusionbhd/oui/oraparam.ini
as shown in the following example:
JRE_MEMORY_OPTIONS=" -d64 -mx512m -XX:MaxPermSize=256m"
Rerun Orchestration.
Problem
The Applying Online BI Metadata and Configuration Updates configuration assistant reports the following JPS exception:
oracle.security.jps.JpsException: JPS-01016: A password credential is expected; instead found null for alias AuditDbPrincipalMap and key AuditDbPrincipalKey at location /u01/APPLTOP/instance/domains/bi.oracleoutsourcing.com/BIDomain/config/fmwconfig/bootstrap.
Solution
This exception has no functional impact on the upgrade and can be ignored.
Problem
The RUP configuration assistant "Generating OHS Reference Configuration File" fails initially while processing the context root metadata files from FMW. For example, the files under /APPTOP/fusionapps/applications/lcm/common/fmwohs
.
Solution
/APPTOP/oraInventory/checkpoint/farup/<RELEASE_VERSION>/checkpoint.xml
checkpoint file, find the following:
<aggregate name="Generating OHS Reference Configuration File" status="fail" progress="97"> <property name="OHS27" value="/scratch/mwport/REL8FA/APPTOP/fusionapps/applications/lcm/common/fmwohs /soa.modwl.conf.xml"/> .............. .............. <property name="OHS12" value="/scratch/mwport/REL8FA/APPTOP/fusionapps/applications/com/deploy/scmExt ernalUriAdf.xml"/> </aggregate>And then, replace it with the following:
<aggregate name="Generating OHS Reference Configuration File" status="success"/>
Rerun orchestration.
Problem
[2017-09-01T07:06:17.166+00:00] [apps] [ERROR] [] [oracle.apps.ad.rupconfig.Applying_Admin_Server_Online_Setting_and_Configuration_Changes] [tid: 32] [ecid: 0000LswBvjR7y0I_IpP5if1PeG7o000006,0] CFGEX-00024 : Config plugin "Updating JPS Config Timeout Settings" failed. No further plugins will be run.
Solution
This error message is expected when upgrading from release 8 or 9 directly to release 12. It does not cause any functional impact and you can skip this failed task and continue with upgrade.
cp checkpoint.xml checkpoint.xml.orig
/APPTOP/oraInventory/checkpoint/farup/<RELEASE_VERSION>/checkpoint.xml
checkpoint file, find the following:
<property name="Updating JPS Config Timeout Settings" value="Failed"/>And then, replace it with the following:
<property name="Updating JPS Config Timeout Settings" value=“Success"/>
Save the checkpoint.xml
file and resume the upgrade.
Problem
ERROR: CFGLOG-00104 : "Starting All Servers" configuration failed. ERROR: CFGLOG-00175 : Startup of server "BIInstance~coreapplication_obisch1~OracleBISchedulerComponent~coreapplication_obisch1" in domain "BIDomain" "failed". ERROR: CFGLOG-00175 : Startup of server "BIInstance~coreapplication_obis1~OracleBIServerComponent~coreapplication_obis1" in domain "BIDomain" "failed". ERROR: CFGLOG-00175 : Startup of server "BIInstance~coreapplication_obiccs1~OracleBIClusterControllerComponent~BIClusterController" in domain "BIDomain" "failed". ERROR: oracle.as.install.fapatchconfig.exception.PatchsetConfigException:CFGEX-00069 : Startup failed for following domain:server pairs" [BIDomain:BIInstance~coreapplication_obisch1~OracleBISchedulerComponent~coreapplication_obisch1, BIDomain:BIInstance~coreapplication_obis1~OracleBIServerComponent~coreapplication_obis1,BIDomain:BIInstance~coreapplication_obiccs1~OracleBIClusterControllerComponent~BIClusterController]".
Solution
java -jar $BIOH/biapps/tools/lib/biruplite.jar $BODOMAIN_HOME $BIOH falseFor example:
/slot/ems16838/appmgr/APPTOP/fusionapps/jdk/bin/java -jar /slot/ems16838/appmgr/APPTOP/fusionapps/bi/biapps/tools/lib/biruplite.jar /slot/ems16522/appmgr/APPTOP/instance/domains/slcah751.us.mycompany.com/BIDomain /slot/ems16838/appmgr/APPTOP/fusionapps/bi false
Add the [opss-DBDS
] section to $BIInstance/bifoundation/OracleBIApplication/coreapplication/setup/odbc.ini
. Refer to the BIShared
file at BIShared/Essbase/essbaseserver1/bin/.odbc.ini
.
BIcomponent
and the OPMN instance as follows:
$BIInstance/bin/opmnctl stopall
$BIInstance/bin/opmnctl start
Rerun RUP Orchestration from the Primordial host.
Problem
Verifying Node Manager and OPMS Status fails with the following error:
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : [Security:090542]Certificate chain received from <hostname> - <host IP address> was not trusted causing SSL handshake failure.
Solution
The issue can occur when the host associated with a node manager is explicitly bounced in the middle of the upgrade, and if Upgrade Orchestrator expects the node manager to be in a shutdown state at that time. The node manager on the host may be configured to automatically start up as part of the system boot process and could cause various issues depending on which upgrade step was being performed when the host was restarted. To resolve this issue, stop and restart node manager on the host where the issue was reported. For more information, see Start Node Manager in the Oracle Fusion Applications Administrator's Guide.
Problem
Upgrade Orchestrator fails to stop OPMN processes with an error similar to either of the following errors:
Exception: OPMN could not be stopped. Script will exit. Please stop OPMN manually before re-running the script.
/APPLICATIONS_BASE/webtier_mwhome/oracle_common/jdk/jre/lib/fonts/ALBANWTJ.ttf
– No such file exists
.
Solution
This issue can occur due to an incompatible version of JDK being used during the process. To resolve the issue, perform the following steps.
cd
/APPLICATIONS_BASE/
webtier_mwhome/webtier
mv jdk_backup_existing_version jdk
cd
/APPLICATIONS_BASE/
webtier_mwhome/oracle_common
rm –rf jdk
cp –Rp jdk_bkp_130320_1250 jdk
Resume orchestration.
Problem
The Verifying Node Manager and OPMN Status configuration assistant fails.
Solution
Do not exit out of Upgrade Orchestrator in response to this configuration assistant failure. Perform the following steps to recover:
Review the node manager log files to determine the cause of the failure:
APPLICATIONS_CONFIG
/nodemanager/host_name
/
Note that the APPLICATIONS_CONFIG
value can be obtained from the APPLICATIONS_BASE
/fusionapps/faInst.loc
file.
After resolving the issue that caused the failure, start the Node Manager on all hosts that are part of the Oracle Fusion Applications provisioned system. For more information, see Start Node Manager in the Oracle Fusion Applications Administrator's Guide.
Restart the OPMN server for BI, GOP (if GOP is installed), and Web Tier. If the Web Tier (OHS) installed with the Oracle Fusion Applications middle tier is run, it is possible to start it using the following steps. If the Web Tier is run on a separate machine, it may be possible to run the steps below on the other machine. In either case, ensure that Web Tier (OHS) is up at this point:
Example for BI: (note the usage of start
instead of startall
)
cd APPLICATIONS_CONFIG/BIInstance/bin ./opmnctl start
Example for GOP: (note the usage of start
instead of startall
) Note that the OPMN server for GOP should be started from the machine that hosts the Supply Chain Management Administration Server domain. Start the OPMN server for GOP only if GOP is installed.
cd APPLICATIONS_CONFIG/gop_1/bin ./opmnctl start
Example for Web Tier: (note the usage of start
instead of startall
)
cd APPLICATIONS_CONFIG/CommonDomain_webtier/bin ./opmnctl start
For more information about the location of APPLICATIONS_BASE
and APPLICATIONS_CONFIG
, see Directories Structure Overview.
The BI, Web Tier, and GOP (when applicable) processes managed by OPMN are started by the installer in the Starting All Servers configuration assistant.
Fix any other environment issues before resuming the upgrade. If the installer exits for any reason, make sure that all node managers and OPMN processes are running. Contact Oracle Support Services to proceed out of this step if there are unresolved environment issues.
After starting the services, resume orchestration to proceed to the Starting All Servers configuration assistant. If the starting of servers times out, see Troubleshoot Server Start and Stop Failures.
If GOP is not installed, the user interface reports "Success" for GOP OPMN status, but the log file reports: GOP is not provisioned. Skipping check for OPMN status.
This section describes two scenarios that can prevent the node manager from starting between the first and second installer.
Problem 1
The node manager was manually started before or during the Extending Certification Validity configuration assistant. The node manager caches the previous keystore certificates so the updated certificates are not validated and the node manager fails to start.
Solution
Check the node manager logs to determine if it is running and when it was last started. If the time stamp is earlier than the Extending Certification Validity configuration assistant execution time stamp, restart the node manager so that it reads the updated keystore certificates.
To find out if the node manager is running for a specific host, connect to the host and run the following command:
If any results are returned, the node manager is running.
ps -ef | grep nodemanager
If the node manager is running, find the time of the last entry of <Secure socket listener started on port nnnn>
in the following directory.
APPLICATIONS_CONFIG/nodemanager/logical_host_name/nodemanager
To check the timestamp for the Extending Certification Validity configuration assistant, find the fapatch_Extending_Certificate_Validity_XXXX
file in one of the following directories:
APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/configlogs APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/ARCHIVE/timestamp/configlogs
The last time stamp entry is the execution timestamp.
Problem 2
The administration servers in one or more domains are running before the Extending Certification Validity configuration assistant runs. This prevents validation of the updated keystore certificates and fails to provide the correct status to orchestration.
Solution
Perform the following steps:
Verify whether the administration server of the domain is running by launching the administration console of the domain. If the console comes up, then the administration server is running.
Verify the last time the administration server was started. Go to the APPLICATIONS_CONFIG/
domains/
logical_host_name/domain_name
/servers/AdminServer/logs
directory. Using the command, ls -lrt,
find the most recent the AdminServer.log
file. In this file, find the time of last entry that contains text similar to the following example:
<Channel "Default" is now listening on machine_ip:port for protocols iiop, t3, ldap, snmp, http.>
Problem
The following error occurs on the OHS node:
[Plugin failed]: StopOPMNProcesses [Phase DowntimePostFA]: failed ORCH-DOWNTIME-OPMN-00004 : Stopping OPMN Control Process failed. Review log file
Solution
This error can occur when OHS is on the same node as the primordial host and APPLICATIONS_CONFIG
is not APPLICATIONS_BASE/
instance
. For example, this error occurs if APPLICATIONS_BASE
=/APPTOP
and APPLICATIONS_CONFIG
=/instance
.
To resolve the issue, perform the following steps:
Create a link from APPLICATIONS_BASE/
instance
to APPLICATIONS_CONFIG
as follows:
ln -s APPLICATIONS_CONFIG APPLICATIONS_BASE/instance
Resume orchestration.
Delete the link after orchestration completes successfully.
This section describes the following RUP Lite for OHS failures:
Problem
RUP Lite for OHS fails during the ohs.plugin.UpgradeWebtier
step due to missing the jdk
directory.
Solution
Verify if there is a jdk_backup_existing_version
directory under WT_ORACLE_HOME
. If this directory exists, rename it to jdk
and resume Orchestration.
Also, if the missing jdk
directory is from WT_MW_HOME/
oracle_common
, check to see if there is a jdk_backup_existing_version
directory under this directory. If so, rename it to jdk
and resume Orchestration.
Problem
During the upgrade, RUP Lite for OHS fails with the following error:
Failed execution of plugin: ohs.plugin.ReassociateCommonDomain
Solution
Update the server_name/
instance/
CommonDomain_webtier_local/config/OPMN/opmn/instance.properties
file to set the registered
property to true
. Then check the Administration Server on either the Common Domain or the OSN Domain to ensure it is running. If not, bounce the server and retry RUP Lite for OHS by resuming orchestration.
Problem
RUP Lite for OHS reports a server side error occurs with the following error message:
Server instance is not running for the security mode specified: "simple". Try again using a different security mode. The remote registration process did not succeed! Please find the specific error message below.
Solution
To resolve this issue, perform the following steps:
Log in to the OAM administration console.
From the System Configuration tab, click Server_Instances, and double click the OAM server instance, such as oam_server1
.
Select simple from the Mode field in the right panel.
Click Apply to submit the changes.
Restart the OAM Server.
Restart all OHS servers in the environment.
Resume Upgrade Orchestrator.
Check the Oracle Fusion Applications OHS to ensure that SSO still works after the change. If it does not, upgrade Webgate manually for the Oracle Fusion Applications OHS.
This section describes the following troubleshooting issues:
Problem
While the primordial host is waiting for IDMOHS:IDMUpgradeDone
, there are communication exceptions on the PRIMORDIAL console.
Solution
These errors can be ignored and orchestration can be resumed.
Problem
If the OAM administrator password contains special characters, such as '#' or '&', the OAM Configuration step fails. To work around this issue, manually validate that the OAM Administration Server host/port and username/password are correct. After manually validating this information, proceed with the manual upgrade.
Solution
To validate, perform the following steps:
Get the OAM administrator user name and password from the credential store.
Run APPLICATIONS_BASE/
fusionapps/oracle_common/common/bin/wlst.sh
.
Run the following commands to connect to the Common Domain Administration Server and get the OAM administrator username and password:
connect() listCred(map='oracle.patching', key='FUSION_APPS_PATCH_OAM_ADMIN-KEY')
Get the OAM Administration Server host and port from the following properties in APPLICATIONS_CONFIG/
fapatch/FUSION_prov.properties
:
OAM_ADMIN_SERVER_HOST
OAM_ADMIN_SERVER_PORT
Use oamcfgtool.jar
to confirm whether the OAM server can be invoked using the values found in the previous steps. For example:
cd APPLICATIONS_BASE/fusionapps/oracle_common/modules/oracle.oamprovider_11.1.1 java -jar oamcfgtool.jar app_domain=crm web_domain=OraFusionApp uris_file=APPLICATIONS_BASE/fusionapps/applications/crm/security/oam.conf oam_aaa_mode=open_or_simple app_agent_password=password_for_map="oracle.patching" key="FUSION_APPS_PATCH_OAM_RWG-KEY"_in_credential_store primary_oam_servers=oam_server1 oam_admin_server=http://OAM_admin_server_host:port oam_admin_username=username_for_FUSION_APPS_PATCH_OAM_ADMIN-KEY oam_admin_password=password_for_FUSION_APPS_PATCH_OAM_ADMIN-KEY oam_version=11 default_authn_scheme=FAAuthScheme
If the previous command is successful, the validation is successful and orchestration can be resumed.
This troubleshooting step is only applicable if the OAM configuration fails while you are performing the steps to remove OVD from your environment as listed in Update OAM Configuration.
Problem
OAM configuration update fails for OID authenticator.
Solution
cd $OAM_ORACLE_HOME/common/bin
setenv DOMAIN_HOME /u01/app/idm/admin/IDMDomain/aserver/IDMDomain$DOMAIN_HOME/bin/setDomainEnv.sh
./wlst.sh
connect('weblogic_idm',’$WLS_ADMIN_PASSWORD’,'t3://myhost.example.com:7001')
editUserIdentityStore(name="OIMIDStore", ldapUrl="ldap://idstore.example.com:3060", ldapProvider="OID", userSearchBase=”cn=Users,dc=us,dc=oracle,dc=com”, groupSearchBase=”cn=Groups, dc=us,dc=oracle,dc=com)
The location of the Governance, Risk, and Compliance (GRC) policies varies, depending on the upgrade path to Release 12. GRC polices are located in the grc OAM application domain if the Oracle Fusion Applications environment was originally provisioned with either version 11.1.1.5 or 11.1.2, then upgraded to version 11.1.3, and beyond. If the environment was originally provisioned with version 11.1.3 and upgraded to version 11.1.4 and beyond, the GRC policies are located in the fs OAM application domain.
Problem
Due to a failure during the upgrade, it is necessary to restore all of the data under the root node of the OPSS Security Store.
Solution
To restore all of the data under the root node of the OPSS Security Store, perform the following steps:
Ensure Oracle Internet Directory is up and running.
Perform a bulkdelete
on Oracle Internet Directory nodes.
In the Oracle Internet Directory system, verify that there are no schema errors or bad entries by running bulkload
, as illustrated in the following command:
OID_HOME/ldap/bin/bulkload connect="dstOidDbConnectStr" check=true generate=true restore=true file="fullPath2SrcOidLdif"
If duplicate DNs (common entries between the source and destination directories) are detected, review them to prevent unexpected results.
Load data into the Oracle Internet Directory by running bulkload
, as illustrated in the following command:
OID_HOME/ldap/bin/bulkload connect="dstOidDbConnectStr" load=true file="fullPath2SrcOidLdif"
Problem
Applying Release12 One-Off patches on an Release 12 setup fail.
Workaround
/u01/IDMTOP
immediately after the upgrade. For example, if the real top is in /abc/idmtop
, perform the following steps:
mkdir /u01 ln -s /abc/idmtop /u01/IDMTOP
If Webgate is not configured on the OHS SO Node, perform the following steps:
Log in to the OHS SO Node.
LD_LIBRARY_PATH
with the Web Tier library path APPLICATIONS_BASE/webtier_mwhome/webtier/lib
as follows:
WEBHOST2> export LD_LIBRARY_PATH=APPLICATIONS_BASE/<webtier_mwhome>/<webtier_directory_name>/lib:$LD_LIBRARY_PATHFor example:
WEBHOST2> export LD_LIBRARY_PATH=/u01/app/idm/products/ohs/ohs/lib:$LD_LIBRARY_PATH
deployWebgateInstance.sh
as follows:
cd <WEBTIER_MW_HOME>/webgate/webgate/ohs/tools/deployWebGate ./deployWebGateInstance.sh -w <WEBTIER_INSTANCE_HOME>/config/OHS/ohs2/ -oh <WEBTIER_MW_HOME>/webgateFor example:
./deployWebGateInstance.sh -w /u02/local/idm/config/instances/ohs2/config/OHS/ohs2/ -oh /u01/app/idm/products/ohs/webgate
cd <WEBTIER_MW_HOME>/webgate/webgate/ohs/tools/setup/InstallTools ./EditHttpConf -w <WEBTIER_INSTANCE_HOME>/config/OHS/ohs2/ -oh <WBETIER_ME_HOME>/webgate -o webgate.confFor example:
./EditHttpConf -w /u02/local/idm/config/instances/ohs2/config/OHS/ohs2/ -oh /u01/app/idm/products/ohs/webgate -o webgate.conf
Copy the following from WEBHOST1
to WEBHOST2
:
<WEBTIER_INSTANCE_HOME>/config/OHS/ohs1/webgate/config
, copy the following to <WEBTIER_INSTANCE_HOME>/config/OHS/ohs2/webgate/config
on WEBHOST2
:
"ObAccessClient.xml","cwallet.sso","password.xml"
<WEBTIER_INSTANCE_HOME>/config/OHS/ohs1/webgate/config/simple
, copy the following to <WEBTIER_INSTANCE_HOME>/config/OHS/ohs2/webgate/config/simple
on WEBHOST2
:
"aaa_key.pem","aaa_cert.pem"
WEBHOST2> cd <WEBTIER_INSTANCE_HOME>/bin WEBHOST2> ./opmnctl stopall WEBHOST2> ./opmnctl startall
Ensure that the URLs are accessible as before, by temporarily stopping Oracle HTTP server on the Node 1 and making accessible the URLs.
Problem
51fea9603f49 ,lastModified:2010-01-10T17:46:52.000-0800 ,ioe:java.util.zip.ZipException: Could not find End Of Central Directory <ERROR> Error message was found in log file on line ::2905:: -->WARNING: corrupted jar found. file:/u01/IDMTOP/products/app/iam/oam/server/lib/jmx/oam-dummy-config.xml, md5sum:2c3ede41786705c57cf151fea9603f49 ,lastModified:2010-01-10T17:46:52.000-0800 ,ioe:java.util.zip.ZipException: Could not find End Of Central Directory <ERROR> rror message was found in log file on line ::2991:: -->WARNING: corrupted jar found. file:/u01/IDMTOP/products/app/iam/oam/server/lib/jmx/oam-dummy-config.xml, md5sum:2c3ede41786705c57cf151fea9603f49 ,lastModified:2010-01-10T17:46:52.000-0800 ,ioe:java.util.zip.ZipException: Could not find End Of Central DirectoryExit Value from individual subroutine check is 1
Solution
Ignore the error message.
Problem
Migration or upgrade fails with permission issues such as unable to access or unable to write.
Solution
To resolve this issue, check the failed file’s user and group privileges and permissions of the stagedir
and idmUpgrade
folders. The user and group permissions should be same as the user who is running the upgrade. In addition, all of the folders need to be writable.
Problem
The OIF URL is not accessible post migration.
Solution
Bring down all servers.
Clean up the tmp
folders of all the servers in both shared config (/u01/IDMTOP/config
) and local config (/u02/local/IDMTOP/config
).
Restart the servers and check the OIF URL.
Problem
Download email template from OIM fails in Run Upgrade Orchestrator During Downtime.
Solution
This is an expected failure. The Download email template step is executed again as part of the IDM upgrade in Manually Download OIM Email Template.
Problem
The SOA managed server fails to start on the scaled out machine with the following exception in the SOA logs under /u01/IDMTOP/config/scripts/logs
:
Starting server wls_soa2 ...Access to domain 'IDMDomain' for user 'admin' denied No stack trace available.
Solution
nmEnroll
WSLT command as follows:
nmEnroll('/u02/local/IDMTOP/config/domains/IDMDomain', '/u01/IDMTOP/config/nodemanager/<IDMSOHOSTNAME>');Where
<IDMSOHOSTNAME>
: The secondary/scaled out machine host name.
For more information about the execution of the nmEnroll
command, see Oracle WebLogic Server 12c: Configuring and Using Node Manager.
After the successful execution of the nmEnroll
command, rerun migration on the SOA scaled out node.
Problem
OIM binary upgrade fails with no "IDMTOP/products/app/utils/orainst.loc
" error.
One possible cause is that you may be using the upgradeOnPremise.properties
under the idmUpgrade
folder for upgrade rather than the one under stagedir
. For type II upgrades, upgradeOnPremise.properties
is auto-generated by the discovery tool during discovery execution by seeding necessary values for upgrade.
Solution
To resolve this issue, rerun upgrade using the upgradeOnPremise.properties
under stagedir
.
Problem
Migration exits with an error to stop the processes and restart.
Solution
Check the migration logs for details on those processes, and then stop them.
Restart migration.
This section provides the following troubleshooting information related to the Applying Pre-PSA Middleware Patches or Applying Post-PSA Middleware Patches configuration assistants:
Problem
An error occurred during the Applying Pre-PSA Middleware Patches or Applying Post-PSA Middleware Patches configuration assistant.
Solution
Review the log file in the relevant location to find the cause of the error:
APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/RUP/ApplyPrePSAMiddlewarePatches
timestamp
.log
APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/RUP/ApplyPostPSAMiddlewarePatches
timestamp
.log
For Language Pack failures, review the following log files:
Failures during Install mode and Standalone LP installation: APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/LanguagePack/language/FAPatchManager_ApplyMWLangPackPatchestimestamp.log
Failures during LP upgrade through orchestration (configuration mode):
APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/LanguagePack/MergedLPs/FAPatchManager_ApplyMWLangPackPatchestimestamp.log
For specific OPatch failures, go to each of the individual Oracle home directories to find the details of the OPatch errors. For example, for a SOA failure, go to APPLICATIONS_BASE
/fusionapps/soa/cfgtoollogs/opatch
.
Problem
The Applying Middleware Patchsets configuration assistant fails with an error as shown in the following example:
[as] [ERROR] [] [oracle.as.install.engine.modules.presentation] [tid: 15] [ecid: 0000JsNJml6AxGGpIww0yf1HRacu000006,0] sun/awt/X11GraphicsEnvironment[[ . java.lang.NoClassDefFoundError: sun/awt/X11GraphicsEnvironment at java.lang.Class.forName0(Native Method)
Solution
Unset the DISPLAY variable or set it to the correct value. To unset it, run "unset/unsetenv DISPLAY
" on all hosts. Then resume Upgrade Orchestrator.
Problem
The Applying Post-PSA Middleware Patches configuration assistant hangs.
Solution
This problem is most likely due to adpatch hanging as the result of the java worker not getting the database connection. Resolve this issue by following the steps in Troubleshoot Loading Database Components. Run the commands from ATGPF_ORACLE_HOME
instead of FA_ORACLE_HOME
.
Problem
The following error occurs:
OPatch cannot continue because it can't load library from the directory "<dbclient Oracle Home>/oui/lib/linux64"
Solution
This error may occur if the OUI version in the database client Oracle home is 11.2 while the OUI version in Oracle Fusion Applications Oracle home (FA_ORACLE_HOME) is 11.1.
To resolve this issue, perform the following steps:
Go to the DB Client home.
Set the ORACLE_HOME
environment variable to point to the database client Oracle home.
Apply the database client patches using the following command:
$ORACLE_HOME/OPatch/opatch apply patch_location
Because the patches have now been manually applied, perform the following steps to continue with the upgrade:
Go to the FA_ORACLE_HOME/
fusionapps/applications/lcm/tp/config/RUP/FMW
directory.
Open the pre-psa-jobs.xml
file for editing.
Comment out the job with the name dbclient
as shown in the following example:
<!-- <job> <id>10</id> <target>FAMW</target> <component> <name>dbclient</name> <version>11.1.1.5</version> <component> <utility_name>opatch</utility_name> <patch_number>NA</patch_number> <command>%opatch% napply -silent -skip_duplicate -skip_subset -oh %dbclient_home% -phBaseDir %dbclient_patch% -jre %jre_loc% -invPtrLoc %oraInstLocFile%</command> <patch_location>NA</patch_location </job>
Problem
The following error is reported:
ORA-01658: unable to create INITIAL extent for segment in tablespace FUSION_TS_SEED.
Solution
The standard output from the ORA-1658 error follows:
ORA-01658: unable to create INITIAL extent for segment in tablespace string Cause: Failed to find sufficient contiguous space to allocate INITIAL extent for segment being created. Action: Use ALTER TABLESPACE ADD DATAFILE to add additional space to the tablespace or retry with a smaller value for INITIAL
For more information, refer to Oracle Database documentation.
This section contains the following topics:
Problem
An error occurred during the Upgrading Middleware Schema configuration assistant.
Solution
Review the log file in this location to find the cause of the error:
fusionapps/oracle_common/upgrade/logs
/psa
timestamp
.log
Problem
The Upgrading Middleware Schema configuration assistant fails while upgrading SES component when TDE Data Vault is enabled. The following error is reported:
[RCU] [TRACE] [] [upgrade.RCU.jdbcEngine] [tid: 10] [ecid: 0000K8DIf5l9xWR5IZL6if1ISVu^000000,0] Driver: oracle.jdbc.driver.OracleDriver [2013-10-31T06:54:31.536+00:00] [RCU] [TRACE] [] [upgrade.RCU.jdbcEngine] [tid: 10] [ecid: 0000K8DIf5l9xWR5IZL6if1ISVu^000000,0] jdbcUrl = jdbc:*****:thin:sys as sysdba/*****@(DESCRIPTION=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=fusion db.*****outsourcing.com)(PORT=1616))(ADDRESS=(PROTOCOL=TCP)(HOST=fusiondb2.*** *outsourcing.com)(PORT=1616))(CONNECT_DATA=(SERVICE_NAME=fusiondb)))
Solution
To resolve this issue, perform the following steps:
Connect as searchsys.
DROP INDEX "SEARCHSYS"."EQ$DOC_PATH_IDX" force.
Execute eq_adm.use_instance(1)
.
Execute eq_ddl.create_index()
.
Resume orchestration.
This section contains information about troubleshooting issues that may occur during the Loading Database Components configuration assistant. Depending on the type of failure, it may be needed to review one or more of the log files in the following locations:
APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/RUP/PatchManager_DBPatch/
FAPatchManager_apply_
timestamp
.log
adpatch_apply_
timestamp
.log
adpatch_apply
_timestamp_
workernum
.log
ATGPF_HOME/
admin/FUSION/log
If the language is upgraded through orchestration, then the config mode LP logs for Loading DB Componenets are present in APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/PatchManager_NLS_DBPatch
.
The following troubleshooting issues are described in this section:
Problem
A failure occurred during either the Grant Privileges to Application Schemas or the Creating Grants/Synonyms on Application Database Objects configuration assistant.
Solution
Find the cause of the failure by running the script manually as the sysdba
user, using SQL*Plus or SQL*Developer. After resolving the issue, resume orchestration.
Problem
An email notification stating that one or more database workers failed during the Loading Database Components configuration assistant is received.
Solution
This email notification is only received when the upgrade cannot progress any further and requires user intervention. In this scenario, all the workers are in a FAILED or IDLE status. The configuration assistant remains in a RUNNING status until all tasks in Loading Database Components have run. To resolve this issue, you must start AD Controller to manage the failed workers. For additional information, see Troubleshoot Patching Sessions for Database Content in the Oracle Fusion Applications Patching Guide. After resolving the issue that caused the workers to fail, and restart the workers, Upgrade Orchestrator continues processing. No further intervention is required.
Note that the messages are displayed on the console for database component failures if orchestration is run with the -DlogLevel
option set to FINEST
.
There might be corner cases when an alert email indicating failed workers although the workers are still running might be received. In such cases, ignore the email alert after ensuring the workers are running with no failures.
Problem
The database goes down while RUP Installer is running the Loading Database Components configuration assistant. If simply bringing the database up and then resuming orchestration, the following error might be received:
Failed to connect to the database as fusion with error: No more data to read from socket
Solution
To recover from this error, perform the following steps:
Force the database patching session to fail as follows:
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail
Start AD Controller as follows:
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/adctrl.sh
For more information, see Start AD Controller in the Oracle Fusion Applications Patching Guide.
Perform the following steps in AD Controller to manage the workers:
Select Tell manager that a worker failed its job and enter All for all workers.
Select Tell worker to quit and enter All for all workers. Note that this does not kill the workers. It sends a command to the worker to shutdown after it completes the current task.
Wait for all workers to complete their tasks and shut down normally.
If there are still some worker processes that do not shut down, kill those processes manually by selecting Tell manager that a worker failed its job. Then select Tell manager that a worker acknowledges quit and enter All for all workers.
From the operating system, check for processes that are running fapmgr, javaworker, adpatch, adadmin
, sqlplus
, and adworker
. If any exist, terminate them from the operating system.
Select Tell worker to restart a failed job and enter All for all workers.
Resume orchestration.
Problem
AutoPatch validation fails with the following message:
An active adpatch or adadmin session was found. Complete or terminate the active session to allow fapmgr to proceed
.
Solution
Perform the following steps to resolve this error:
Run the fapmgr forcefail
command to update the patching tables as follows:
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail [-logfile log file name] [-loglevel level]
Run the fapmgr abort
command from FA_ORACLE_HOME
to find out if an Oracle Fusion Applications Patch Manager session must be cleaned up as shown in the following example:
(UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh abort [-logfile log file name] [-logLevel level]
If this command finds no failed session, proceed to Step 3.
Run the following commands from ATGPF_ORACLE_HOME
to abandon any Applications Core patching sessions or AD Administration sessions that may be running:
(UNIX) ATGPF_ORACLE_HOME/lcm/ad/bin/adpatch.sh abandon=y interactive=n defaultsfile=APPLICATIONS_CONFIG/atgpf/admin/defaults.txt (UNIX) ATGPF_ORACLE_HOME/lcm/ad/bin/adadmin.sh abandon=y interactive=n defaultsfile=APPLICATIONS_CONFIG/atgpf/admin/defaults.txt
Problem
When multiple seed data files are uploaded for the same flexfield but for different flexfield contexts, the upload tasks can fail due to locking issues. The failed tasks appear in the log file as the following error:
Loading failed with a JboException: JBO-25014: Another user has changed the row with primary keyoracle.jbo.Key ...
Solution
AutoPatch defers any failed tasks to the end of the phase and reattempts the failed tasks only after attempting all tasks in the phase at least once. Usually the flexfield seed data tasks that failed due to the locking issue succeed on subsequent attempts. You can ignore these errors if the flexfield seed data task succeeds on the retry. If the task remains in a failed state, you must use the AD Controller utility to retry the failed task.
For more information, see Restart a Failed Worker in the Oracle Fusion Applications Patching Guide.
Problem
The upgrade script, pje_txn_fix_issues_bug18504814.sql
, fails with the following error:
ORA-00001: unique constraint (FUSION.PJE_ISSUE_TYPES_TL_U1) violated.
Solution
Skip the AD worker that failed while running the pje_txn_fix_issues_bug18504814.sql
file.
Problem
[2016-11-22T10:23:05.464+00:00] [xdf] [ERROR] [XDF-10567] [oracle.xdf] [tid: 1] [ecid: 0000LY7^LAfBX7M5ING7yf1OClRT000001,0] Could not modify the column to NOT NULL. [2016-11-22T10:23:05.464+00:00] [xdf] [NOTIFICATION] [XDF-00005] [oracle.xdf] [tid: 1] [ecid: 0000LY7^LAfBX7M5ING7yf1OClRT000001,0] Alter DDL: ALTER TABLE "FUSION"."MOW_RULE_SET_DETAILS" MODIFY ("MAX_SCORE" NUMBER(9,0))
Solution
Ignore the error. Skip and restart the failed adworker.
This section contains the following information about troubleshooting issues that may occur during the Deploying Application Policies configuration assistant:
Log files for this configuration assistant may be found in this location:
APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/RUP/configlogs/fapatch_Deploying_Applications_Policies_(jazn-data.xml)_
timestamp
.log
APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/LanguagePack/language/configlogs/fapatch_Deploying_Applications_Policies_(jazn-data.xml)_
timestamp
.log
Problem
A failure occurs during applications policy analysis.
Solution
Review the log file that is generated by each stripe. The log files are located under the main log directory, APPLICATIONS_CONFIG/
lcm/logs/11.12.0.0.0/RUP
and are named as follows:
fapatch_CRMJaznAnalysis
_timestamp
.
log
fapatch_FSCMJaznAnalysis
_timestamp
.
log
fapatch_HCMJaznAnalysis
_timestamp
.
log
fapatch_OBIJaznAnalysis
_timestamp
.log
fapatch_SOAJaznAnalysis
_timestamp
.
log
fapatch_UCMJaznAnalysis
_timestamp
.log
fapatch_BPMJaznAnalysis
_timestamp
.log
fapatch_B2BJaznAnalysis
_timestamp
.log
After resolving the JAZN analysis error, retry the analysis for the failed stripe to confirm the issue is resolved.
Problem
A failure occurs during the Deploying Application Policies configuration assistant.
Solution
When a failure occurs during Deploying Application Policies, restore only the stripe or system policy that has failed, from the backup. Use the OPSS migrateSecurityStore
command with the appropriate source and destination arguments to perform the restore. Do not restore a stripe that has not failed. Review the JAZN deployment log file to determine the cause of the failure, fapatch_Deploying_Applications_Policies_(jazn-data.xml)_
timestamp
.log
.
After resolving the issue, resume orchestration. For more information, see the Migrating with the Script migrateSecurityStore section in the Oracle Fusion Middleware Applications Security Guide.
Problem
The Deploying Application Policies configuration assistant fails with the following warning:
oracle.security.jps.internal.policystore.ldap.PermissionManagerImplsearchPermissionEntry WARNING: Duplicate permissions
Solution
Contact Oracle Support to request assistance in resolving this issue.
Problem
The following warning occurs during the Deploying Application Policies configuration assistant:
WARNING: Failed to validate the xml content. cvc-complex-type.2.4.a: Invalid content was found starting with element 'property'. One of '{"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":propertySetRef, "http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":extendedProperty, "http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":extendedPropertySetRef, "http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":serviceInstanceRef}' is expected. Location: line 165 column 96. WLS ManagedService is not up running. Fall back to use system properties for configuration.
Solution
Ignore this message as there is no functional impact of this warning and the deployment is successful.
Problem
The following warning occurs during the Deploying Application Policies configuration assistant:
FINE: Application policies already exists for application: fscm oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException: Cannot create application policy context "fscm". at oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.unsync_createApp licationPolicy(LdapPolicyStore.java:933) at oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.createApplicatio nPolicy(LdapPolicyStore.java:753) at oracle.security.jps.internal.tools.utility.destination.apibased.JpsDstPolicy.c lone(JpsDstPolicy.java:805)
Solution
Ignore this message as there is no functional impact of this warning and the deployment is successful.
Problem
The IDM Server goes down during the Deploying Application Policies configuration assistant and the deployment fails.
Solution
Upgrade Orchestrator does not allow a retry after this type of failure. Instead, exit orchestration and restore from the IDM backup. Then, resume orchestration.
This section includes the following troubleshooting topics:
Problem
A failure during the Starting All Servers configuration assistant typically happens when one of the servers times out and fails to start due to resource issues or application specific issues.
Solution
Various platforms and environment configurations can impact how long it takes all servers to actually start during the Starting All Servers configuration assistant. Although the installer waits an average amount of time for this configuration assistant to complete before it is marked as Failed, different platforms may require more time. It is not unusual to receive timeout errors in the log files if the starting of all servers for the environment requires more time than the installer allows.
If this configuration assistant fails, perform the following steps:
Monitor the status of the servers by reviewing the messages in the server log files or on the console. The log file, APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/RUP/StartStop/fastartstop
_timestamp
.log
, indicates which server started and failed to start. For example:
Time out while performing Start for domain SCMDomain. Waited for 2400 seconds [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [UTIL] [oracle.apps.startstop.util.MbeanUtil: runSSCommandOnDomain.868] [tid:37] Start operation is completed for domain SCMDomain. Please see SCMDomain.log for details. [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [UTIL] [oracle.apps.startstop.invoke.StartStopTask: call.221] [tid:37] StartStopTask over for domain SCMDomain [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [SST] [oracle.apps.startstop.invoke.StartStopTask: call.223] [tid:37] Finished the task for the Domain SCMDomain
Review the log files at the domain level to see a summary of the server status for that domain: APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/RUP/StartStop
/domain name_timestamp
.log
.
APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs
After determining and resolving the cause of the failure, restart Upgrade Orchestrator.
Problem
The following exception during the Starting all Servers configuration action indicates a failure in starting the BIServer
:
Start all servers fails to start Start operation on the component :coreapplication_obips1:, for the instance :BIInstance: - FAILED
The coreapplication_obips1
server log file reports the following error:
ecid:]] [2012-04-10T00:22:20.000-07:00] [OBIPS] [ERROR:16] [] [saw.security.odbcuserpopulationimpl.initialize] [ecid: ] [tid: ] Unable to create a system user connection to BI Server during start up. Trying again.[[ File:odbcuserpoploaderimpl.cpp Line:325 Location: saw.security.odbcuserpopulationimpl.initialize saw.catalog.local.loadCatalog saw.subsystems.catalogbootstrapper.loadcatalog saw.webextensionbase.init saw.sawserver ecid:]] [2012-04-10T00:22:25.000-07:00] [OBIPS] [NOTIFICATION:1] [] [saw.sawserver] [ecid: ] [tid: ] Oracle BI Presentation Services are shutting down.[[ File:sawserver.cpp Line:706 Location: saw.sawserver ecid:
Solution
To work around this issue, perform the following steps:
Resume orchestration, which shuts down and starts bi_server1
.
Monitor the fastartstop
log files and the state of bi_server1(BIDomain).
When bi_server1
restarts, as indicated by a RUNNING status, start the component coreapplication_obiccs1
or all the components of type OracleBIClusterControllerComponent
using opmnctl
as shown in the following example:
*/BIInstance/bin/opmnctl startproc ias-component=coreapplication_obiccs1
Problem
The OHS diagnostic log contains the following error message:
No such file or directory: Couldn't create accept lock
Solution
This issue could be the result of the hypervisors going down, resulting in bringing the Oracle VM servers down. Perform the following steps to resolve the error:
Find the entry for the lock file in httpd.conf, for example:
For Apps OHS: "/dev/shm/ohs_ohs1_http_lock"
For OSN OHS: "/dev/shm/osn_ohs1_http_lock"
Confirm whether the directory that contains the lock file exists.
Assuming this directory does not exist, create the directory.
Problem
The following error is reported during Online Preverification:
weblogic.management.mbeanservers.edit.EditTimedOutException
Solution
It may be necessary to manually release the edit session. For more information, see Resolve an EditTimedOutException Error in the Oracle Fusion Applications Patching Guide.
Problem
The Starting All Servers configuration assistant in RUP Installer fails to start ess_server1
and reports the following error in the ess_server1.log
:
weblogic.rmi.extensions.DisconnectMonitorUnavailableException: Could not register a DisconnectListener
Solution
Perform the following steps to resolve this issue:
Open the Oracle Enterprise Manager console for the domain.
Navigate to the following location:
From the console, expand the WebLogic Domain
Go to ESSCluster, then ess_server1
Right click and open System MBean browser
Go to ess_server1
, ServerStart
, select ess_server1
, and click Arguments
Verify if -Doracle.ess.initialProcessorState=stopped
is present. If it is, remove -Doracle.ess.initialProcessorState=stopped
and click Apply. If it is not present, there is no action to take.
Restart ess_server1
.
Problem
As an intermittent issue, there could be WLS socket exceptions during server startup, or during any other upgrade tasks. An example of the exception is:
bea_wls_management_internal2/Bootstrap, user: FUSION_APPS_PROV_PATCH_APPID java.net.SocketTimeoutException: Read timed out at jrockit.net.SocketNativeIO.readBytesPinned(Native Method) at jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
Solution
Find the managed server or the administration server that encounters the failure, and manually restart the server. Proceed with the upgrade by resuming Upgrade Orchestrator on the failed host.
Problem
After the upgrade, the following error displays after logging in to the WLS Console of CommonDomain. and navigate to Deployments:
soa-infra application is in WARNING state.
Solution
Ignore this error as there is no functional impact for SOA users due to this error.
Problem
The soa-infra app is in a warning state in all domains and errors are reported related to "jms/bpm/CaseEventQueue".
Solution
This error can be ignored.
Problem
The custom domains are not stopped or started by FAStartStop and there errors are reported.
Solution
FAStartStop
does not recognize custom domains. Custom domains must be started and stopped manually, as required, before resuming orchestration.
This section describes how to recover from failures during the Deploying SOA Composites configuration assistant. The following topics are described:
The following documents provide additional information related to subjects discussed in this section:
For more information about working with SOA composites, see Customizing the SOA Composite Application in the Oracle Fusion Applications Extensibility Guide for Developers.
For more information about customizing SOA composites, see Customizing and Extending SOA Components in the Oracle Fusion Applications Extensibility Guide for Developers.
The following log files are generated by the deployment of SOA composites:
Client side log files where individual domain logs reside: APPLICATIONS_CONFIG/
lcm/logs/11.12.x.0.0/RUP/soalogs
Log files for the failed domain:
APPLICATIONS_CONFIG/
domains/
hostname/
domain name/
servers/
server name/
logs/soa_server1.log
APPLICATIONS_CONFIG/
domains/
hostname/
domain name/
servers/
server name/
logs/soa_server1.out
APPLICATIONS_CONFIG/
domains/
hostname/
domain name/
servers/
server name/
logs/soa_server1-diagnostic.log
APPLICATIONS_CONFIG/
domains/
hostname/
domain name/
servers/
server name/
logs/AdminServer.log
Problem
Normally, a failed SOA composite is undeployed by RUP Installer. However, if the failure of the deployment is due to an issue such as SOA servers running out of memory, then RUP Installer does not recover until orchestration is resumed.
The following are examples of error messages related to SOA Composite failures:
COMMONLOG-00049: SOA composite "composite_name" patch failed for server "server_name". Recovery process also failed with an unknown reason. If the SOA composite patch exists on the server, it will be automatically undeployed during retry or a checkpoint run. Also if the base composite is not the default composite, it will be automatically set as default. COMMONLOG-00050: SOA composite "composite_name" patch failed for server "server_name". Recovery process also failed, and the composite patch is not undeployed. The patch will be automatically undeployed during retry or a checkpoint run. COMMONLOG-00051: SOA composite "composite_name" patch failed for server "server_name". Recovery process also failed, and the base composite is not set as the default composite. The base composite will be automatically set as default during retry or a checkpoint run.
The following is an example of report exceptions:
COMMONEX-00026: SOA composite "composite_name" patch failed for server "server_name". Recovery process also failed. Recovery will be done automatically during retry or a checkpoint run. Action : No action required.
Solution
When patching existing SOA composites, RUP Installer automatically recovers any partially deployed SOA composites after failure when Upgrade Orchestrator is restarted. The following actions can be performed by Upgrade Orchestrator:
Undeploy the partially deployed SOA composite revision if it is still present.
Set as default the same SOA composite revision that was default before the patching was attempted, if it is not already set as default.
If the failure was caused by an environment issue, such as running out of memory, resolve the cause of the failure before restarting orchestration.
Problem
The following error is reported during SOA Composite deployment on a Solaris platform:
CFGEX-00079 : Wsm-pm application is not running in domain "domain name"
It has been confirmed that the wsm-pm
application is running on this domain.
Solution
To resolve this issue, perform the following steps:
Log in to the failed domain and check the health of all managed servers and deployed applications.
Bounce all managed servers of the failed domains.
Exit Upgrade Orchestrator.
Restart Upgrade Orchestrator.
If a customized SOA composite deployment fails during the upgrade, manually deploy this composite using WLST commands.
To apply a SOA composite manually after a deployment failure, perform the following steps:
The following example composite, FinAp,
is patched from revision 1.0 to revision 2.0 and the SAR file of revision 2.0 is in FA_ORACLE_HOME
/crm/deploy/sca_FinAp_rev2.0.jar
. The parameters are for illustration purposes only.
Refer to the Diagnostics report to find the name and location of the sca_*.jar
file that was copied to FA_ORACLE_HOME
by Oracle Fusion Applications Patch Manager. For more information, see Diagnostics Report in the Oracle Fusion Applications Patching Guide.
If the previous revision contained JDeveloper customizations, ensure that the patched revision is deployed with the merged JDeveloper customizations. Using the sca_*.jar
file from Step 1, follow the JDeveloper customization merge instructions that are described in Merge SOA Composite JDeveloper Customizations During SOA Preverification. Then, use the merged sca_*.jar
for Step 3.
Deploy the patched composite using the single patch composite command as shown in the following example:
sca_patchComposite('SOA-Infra URL', user, password, '/FA_ORACLE_HOME/crm/deploy//sca_FinAprev2.0.jar', mergeLogFile='/tmp/merge-log.txt')
Run the UpdateSOAMDS
SOA composite on every domain if any flexfield changes were made by following the steps described in Synchronizing Customized Flexfields in the MDS Repository for SOA in the Oracle Fusion Applications Extensibility Guide for Developers.
If JDeveloper customizations were performed to a SOA composite and the composite was deployed to the SOA runtime, RUP Installer reports an error during SOA Preverification, which asks to take the newer version of the composite that is in the release. Then, merge the customizations by performing the following steps:
If any customizations are detected, the SOA Preverification results display the SOA composite name, its location in the FA_ORACLE_HOME
/stripe
/deploy
directory, and the requirement for you to merge JDeveloper customizations into the sca_*.jar file in FA_ORACLE_HOME
before proceeding with the upgrade. The stripe in the directory path refers to crm, hcm, fscm, and so on.
Open the custom SOA workshops and the customized version of the Fusion Applications SOA composite in JDeveloper using "Oracle Fusion Applications Developer".
Import the composite sca_*.jar
file from FA_ORACLE_HOME
/stripe
/deploy
into the project, for example revision 11.12.x.0.0, in JDeveloper. Make note of this revision number in the deployment window because you will need it in Step 8.
Restart JDeveloper in the Oracle Fusion Applications Administrator Customization role.
Verify that there are no errors in JDeveloper.
Verify that the changes introduced in both the customized version and the patched version are present.
Right-click the composite project in the Application Navigator, select Deploy, select the composite, click Deploy to SAR, and click Next.
Manually change the value in New Revision ID to the revision from Step 3, for example, 11.12.x.0.0, and click Finish.
If the deployment folder is set to a location different from that of the FA_ORACLE_HOME
/stripe
/deploy
directory, copy and replace the JAR in the location mentioned in the error message of this SOA Composite. If your file name is different, rename it to the original name. You must copy the jar in the correct format to FA_ORACLE_HOME
/stripe
/deploy
. For example if you have sca_ContractsDeliverablePurchaseDocAttrReadComposite_rev11.12.x.0.0.jar
in JDeveloper, then you must copy it back to the FA_ORACLE_HOME
/stripe
/deploy
directory as sca_ContractsDeliverablePurchaseDocAttrReadComposite.jar
.
To proceed with the installation, use the same command you used to start Upgrade Orchestrator.
This section contains the following topics:
Review the APPLICATIONS_CONFIG/lcm/rupliteovm/output/logs/ruplite.log
file to confirm there are no errors. Alternatively, check rehydration framework logs under /assemblybuilder/logs
or /var/log
for any errors.
Review the following troubleshooting information for specific plug-ins:
ValidateEnvironment: This plug-in runs during every mode of RUP Lite for OVM. If this plug-in fails, RUP Lite for OVM stops. Resolve any errors reported in the log file and then run RUP Lite for OVM again.
SetupCredentials: This plug-in runs during every mode of RUP Lite for OVM. If this plug-in fails, RUP Lite for OVM stops. Typical causes of failure are an incorrect key for an existing wallet, or specifying a key for a new wallet that does not meet Oracle's minimum standards. Resolve any errors reported in the log file and then run RUP Lite for OVM again.
This plug-in prompts for all secure properties that are required by offline RUP Lite plug-ins. It stores these properties in a wallet file, which are encrypted using a key provided by the user. If a wallet already exists, the user provided key must be valid for the existing wallet. If the wallet does not exist, a new wallet is created using the provided key. If a secure property already exists in the wallet, then it is not prompted for again.
This plug-in prompts only for secure properties that are needed by other plug-ins that will execute. If a plug-in does not execute on the current node or is disabled, then its properties are not be requested. If no plug-ins that will execute require secure properties, the wallet creation or access is skipped and the user is not be prompted for the wallet key.
RequireRoot: This plug-in sets the require root flag to true so that RUP Lite for OVM checks that root user is used for the pre-root mode. If this plug-in fails, RUP Lite for OVM stops.
AutoCorrectEtcHosts: This plug-in updates the /etc/hosts
file on each non-IDM node to include entries for the VMs logical host and internal logical host (if applicable), for all VMs that were deployed as part of the current upgrade.
RemoveRunAsRoot: This plug-in removes the run_as_root
file from ovabdir/
deployfw/bin/
file path and is rerunnable. Verify this plug-in was successful by confirming that the run_as_root file no longer exists under the ovabdir/
deployfw/bin directory
. The ovabdir
is /u01/ovmext
for IDM nodes and for all non-IDM nodes it is /u01/APPLTOP/ovabext
.
ModifyOutputOwner: This plug-in modifies RUP Lite files to be owned by the application user instead of root. To verify that this plug-in was successful, check the owner of the /u01/APPLTOP/instance/lcm/ruplitovm/output
directory. The owner should be an apps user and not the root user.
GenerateOptimizedQueryPlans: This plug-in is re-runnable. Verify this plug-in was successful by connecting to the database as fusion_mds
and running the following command:
SELECT TO_CHAR(last_analyzed, 'yyyy/mm/dd hh:mi:ss am') as last_analyzed FROM user_tables;
The results should show that the tables were just analyzed.
DeployECSF: This plug-in is re-runnable. If the environment was originally provisioned before Release 5, verify that this plug-in was successful by confirming that the help object, schedule and group being deployed are reported in the log file. Alternatively, use Fusion Applications Control to connect to the Administration Server that hosts the search application and confirm that the Help instance artifacts are deployed.
/etc/hosts
on non-IDM hosts:
pstore.oracleoutsourcing.com
oim-admin.oracleoutsourcing.com
oim.oracleoutsourcing.com
oim-server.oracleoutsourcing.com
ids-db*.oracleoutsourcing.com
Problem
RUP Lite for OVM runs for a long time during domain configuration.
Solution
To resolve this issue, perform the following steps:
Ensure that the IDM host is accessible and responding.
Ensure that the database is accessible and responding.
If either the IDM host or the database is not responding, update the status of the orchestrator task that runs RUP Lite for OVM to "Error", using the following command:
cd ORCH_LOCATION/bin ./orchestration.sh updateStatus -pod POD_NAME -hosttype host_type -hostname host_name -release release_number -phase phase_name -taskid plugin_name -taskstatus Error
Fix the issue with the IDM host or the database and resume Upgrade Orchestrator.
If none of the above steps solve the problem, contact Oracle Support with detailed log information.
This section contains troubleshooting information for Incremental Provisioning (IP). The following topic is discussed:
Problem
oracle.security.jps.service.credstore.CredStoreException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreException: JPS-10702: The datasource jdbc/OPSSDBDS is not found.[[
Solution
To fix this issue, perform the following steps:
http://host:7801/console
Go to Domain Configurations, and select Data Sources under the Services section.
Select opss-DBDS from the list, and then select the Targets tab.
OrchestrationInfrastructureCluster
ConfiguratorCluster
SupplyOrchestrationCluster
PlanningCluster
ManufacturingCluster
Repeat Step 4 for the following data sources (if they exist):
opss-DBDS-rac1
opss-DBDS-rac2
Click Activate Changes for the changes to take effect.
This solution will take effect and there is no need to bounce the servers or environment.
Problem
[2017-01-24T07:20:17.739+00:00] [IdstoreConnectivityCheck] [ERROR] [] [IdstoreConnectivityCheck] [tid: 92] [ecid: 0000LbFM8W07y0I_IpP5if1OXjyg000009,0] HC-COMMON-00001 : Unable to perform the check:[[ oracle.security.jps.service.idstore.IdentityStoreException: JPS-01520: Cannot initialize identity store, cause: oracle.security.idm.ConfigurationException: Failed to connect to directory. Check configuration information.. oracle.security.idm.ConfigurationException: Failed to connect to directory. Check configuration information. Review log files for additional details to take an appropriate corrective action ]]
Solution
$ORACLE_HOME/bin/sqlplus ods/<ods_password>@<oid_db_connect_string>
oidstats.sql
available under the OID_ORACLE_HOME/ldap/admin
directory as follows:
/u01/products/dir/oid/ldap/admin/oidstats.sql
Rerun orchestration.
This section contains the following troubleshooting scenarios:
Troubleshoot Failures During Propagating Domain Configuration
Upgrade Failures on Non-Oracle VM Configuration Using OVM Templates
RUP Lite for Domain Configuration Takes Too Long to Complete
Extending Certificate Validation Fails on non-Oracle VM Environment
Ignorable Errors During Applying Online BI Metadata and Configuration Updates
Problem
adCreateMosPlan.pl
script. After issuing the setenv
command for PERLLIB5, the following error occurs:
Perl lib version (v5.8.3) does not match the executable version (v5.8.8)
Solution
Run the following commands:
export PERL_HOME=/u01/APPLTOP/dbclient/perl export PATH=/u01/APPLTOP/dbclient/perl/bin:$PATH
Then, retry the setenv
command.
Problem
Health Checker (HC) FileOwnerAndPermissionsCheck might report the following error:
[Error]: Plugin 'FileOwnerAndPermissionsCheck': HC-PERM-0003: Failed to verify the permission of files/folders in /u01/APPLTOP/fusionapps/odi/.cas/CLI/inventory using find command. Review log files for additional details to take an appropriate corrective action (verifying File Ownership and Permissions)
Solution
Ignore the error and proceed.
Note that the /u01/APPLTOP/fusionapps/odi/.cas/CLI/inventory
error is only an example. If any similar error related to ".cas
" is reported, ignore the error and proceed.
Problem
When running Health Checker, the PatchSessionsAndProcessesCheck
check fails with the following error:
HC-PATCHSP-00001: Patch sessions or processes found in your environment:HC-PATCHSP-00017 : Check #7: Running patch processes found in this environment.
For any details needed for the running processes, review the log files .
Solution
This issue can occur even when no active session is found. To resolve this issue, perform the following steps:
Open the Health Checker log file to search for the running process that was reported. The running process typically contains strings such as adpatch
, adadmin
, adworker
, adctrl
, oracle.apps.ad.worker.AdJavaWorker
, or oracle.apps.ad.fapmgr.FAPManager
.
./fapmgr.sh report -patchprogress
If ./fapmgr.sh
returns no rows from the previous step, find the origin of the session(s) identified in Step 1. Then, decide whether this session needs to be terminated or allowed to finish before rerunning Health Checker.
Terminate or allow the process(es) to finish and then rerun Health Checker.
If Health Checker detects that an active session was terminated or has completed, running Health Checker again succeeds.
Problem
oracle.jbo.client.CADatabaseConnectionProvider.loadConnectionProperties(CAData baseConnectionProvider.java:154) at oracle.jbo.client.Configuration. initializeFromConnectionName(Configuration.java:1225) at oracle.jbo.client. Configuration.getConfiguration(Configuration.java:649) at oracle.jbo.common. ampool.PoolMgr.loadConfiguration(PoolMgr.java:759) at oracle.jbo.common. ampool.PoolMgr.findPool(PoolMgr.java:589) at oracle.jbo.client.Configuration. createRootApplicationModuleHandle(Configuration.java:1589) at oracle.jbo.client. Configuration.createRootApplicationModuleHandle(Configuration.java:1559) at oracle.apps.topologyManager.model.applicationModule.TopologyManagerAMUtil. getApplicationModuleInfo(TopologyManagerAMUtil.java:122) at oracle.apps. topologyManager.model.applicationModule.TopologyManagerAMImpl.getApplicationModuleInfo(TopologyManagerAMImpl.java:73) at oracle.topologyManager.client.deployedInfo.DeployedInfoProvider.getDeployedDomainsByEnvironment(DeployedInfoProvider.java:880) at oracle.check.common.util. TMUtils.getDeployedDomains(Unknown Source)at oracle.check.common.util. MWUtils.getDomainDetailsFromTM(Unknown Source) at oracle.check.common.util. MWUtils.getDomainDetails(Unknown Source) at oracle.check.apps. OPMNManagedProcessesStatusCheck.verifyManagedProcessesAreUp(Unknown Source) at oracle.check.apps.OPMNManagedProcessesStatusCheck.execute(Unknown Source)at oracle.check.common.AbstractCheckPlugin.plugin_execute(Unknown Source)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at oracle.healthcheckplug.core.PluginProxy.invoke(Unknown Source) at com.sun.proxy.$Proxy5.plugin_execute(Unknown Source) at oracle.healthcheckplug.core.PluginCallable.call(Unknown Source) at oracle.healthcheckplug.core.PluginCallable.call(Unknown Source) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Plugin succeeded: Verifying OPMN managed processes are up
Solution
Ignore the warning and proceed.
Problem
The PostLangPackChecks
plug-in fails for all Health Checks.
Solution
If the SKIP_UPGRADE_FOR_LANGUAGE
property was enabled with one or more language codes, the Post Language Pack Health Checks are expected to fail for the skipped languages. These failures can be ignored because Post Language Patch Health Checks are run manually after the languages are upgraded manually.
Problem
The following error is reported when running RUP Lite for RDBMS:
[SEVERE] Fatal Error, Traceback (most recent call last): File "/SHARED_LOCATION/work_dir/DB_2014-05-27_10-22-31/db_server_bundle/ruplite/main.py", line 280, in _executeplugin result = _runpluginmodule(plugin_module) File "/SHARED_LOCATION/work_dir/DB_2014-05-27_10-22-31/db_server_bundle/ruplite/main.py", line 191, in _runpluginmodule errinfo = eval("plugin_module.plugin_execute()") File "<string>", line 1, in <module> File "/SHARED_LOCATION/work_dir/DB_2014-05-27_10-22-31/db_server_bundle/db/plugin/PostActions.py", line 95, in plugin_execute raise Exception('Failed to perform post DB restart actions.') Exception: Failed to perform post DB restart actions. [main] [SEVERE] Failed execution of plugin: db.plugin.PostActions [main] [SEVERE] Fatal error, exiting [main] [SEVERE] Summary of plugins: [main] [SEVERE] Succeeded: db.plugin.ValidateEnv [main] [SEVERE] Skipped: db.plugin.PreActions [main] [SEVERE] Skipped: db.plugin.ApplyDBPatches [main] [SEVERE] Failed with fatal: db.plugin.PostActions, Exception: Failed to perform post DB restart actions. [main] [SEVERE] RUPLite Installer for DB Stopped
Solution
RUP Lite for RDBMS failed while connecting to the database, which indicates an invalid value in the work_dir/
DB_
timestamp/
db_server_bundle/
metadata/env.properties
file. If there is an extra "/
" character for the ORACLE_HOME
property, in this file, remove it. This ORACLE_HOME
value must exactly match the database ORACLE_HOME
and it should not have an additional "/
" at the end.
Note that running RUP Lite for RDBMS in "validate
" or "setdbparameter
" mode runs successfully even if there as an additional "/
" in the ORACLE_HOME
property.
Problem
An error occurred during the Bootstrapping Patch Manager configuration assistant.
Solution
An error during Bootstrapping Patch Manager normally occurs only when the database is down. Ensure that the database is up and running. Review the related log files in the following location:
APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/FAPatchManager_bootstrap_timestamp.log
This section contains information about troubleshooting issues that may occur during the Propagating Domain Configuration configuration assistant. The following topics are discussed:
Problem
The Propagating Domain configuration assistant is taking too long to complete.
Solution
This configuration assistant can take some time to complete as it is highly dependent on the environment, specifically the number of non-admin domains and the responsiveness of the file system. Monitor the progress of this configuration assistant by reviewing log files in this location:
APPLICATIONS_CONFIG/lcm/admin/version/fapatch//ruplitedomain/output/logs
To confirm this configuration assistant was successful, verify that the config/fusionapps_start_params.properties
file exists under each local or non-admin split domain. Also ensure that the bin/setDomainEnv.sh
file under each local or non-admin split domain contains the following row:
POST_CLASSPATH="${COMMON_COMPONENTS_HOME}/modules/oracle.appstrace_11.1.1/appstrace.jar${CLASSPATHSEP}${POST_CLASSPATH}" export POST_CLASSPATH
Problem
The Propagating Domain Configuration configuration assistant fails if there are WARs or EARs installed or deployed that are not accessible from the primordial host where the upgrade is running. The following is an example of the error caused by this condition:
<< read domain from APPTOP/instance/domains/server.company.com/SCMDomain << write template to APPLICATIONS_CONFIG/lcm/admin/11.12.x.0.0/fapatch/ruplitedomain/output/templates/SCMDomain.jar >> fail: Unable to locate file: /fusionapps/localdomain/domains/server.company.com/SCMDomain/datalens/datalens.war >> fail: write template to "APPLICATIONS_CONFIG/lcm/admin/11.12.x.0.0/fapatch/ruplitedomain/output/templates/SCMDomain.jar" CFGFWK-60550: Script execution aborted. The script may contain an error. Unable to locate file: /fusionapps/localdomain/domains/server.company.com/SCMDomain/datalens/datalens.war
Solution
To resolve this issue, undeploy or uninstall the WAR or EAR, which is datalens.war
in this example. Then, resume orchestration. After the upgrade has completed successfully, install or deploy the WAR or EAR.
Problem
The upgrade fails when Oracle Fusion Applications is running on a non-Oracle VM configuration and is using an Oracle VM template.
Solution
This configuration is not supported. To resolve this, check if a directory named /assemblybuilder
exists in the environment. If this directory is present and this is not an Oracle VM environment, rename the directory to any other name. Then, resume orchestration.
Problem
RUP Lite for Domain Configuration takes too long to complete.
Solution
This utility can take some time to complete as time taken to propagate domain configuration is highly dependent on the environment, specifically the number of non-admin domains and the responsiveness of the file system. Note this issue is seen only in local domain environments where the utility is run between RUP Installer Part 1 and Part 2. This is not an issue for Oracle VM environments or other environments with shared domains.
Problem
The Extending Certificate Validation fails with exception reporting if there is Incentive Compensation, Enterprise Contracts, and Oracle Fusion Accounting Hub offerings on the environment:
APPLICATIONS_CONFIG/domains/CommonDomain_host/CommonDomain /config/fmwconfig/owc_discussions.jks (No such file or directory).
Solution
If the missing file cannot be found in APPLICATIONS_CONFIG/domains/
CommonDomain_host/
CommonDomain/config/fmwconfig
, perform the following steps:
Copy default_keystore.jks
to owc_discussions.jks
in APPLICATIONS_CONFIG/domains/
CommonDomain_ host/
CommonDomain/config/fmwconfig
.
Resume orchestration.
Problem
After the Release 8 upgrade step called "Deploying Data Security Grants", the fapatch_Deploying_Data_Security_Grants_timestamp.log
file contains entries as shown in the following example:
Number of records processsed : 8372 Number of records updated (grantee_key or compile_flag) : 3934 Number of records where GUIDs matched and no reconciliation done : 4366 Number of records in database missing necessary meta data : 2 Number of records in database that could not be reconciled with LDAP : 70
These messages may start with either "WARNING" or "SEVERE". The severe errors may be associated with exceptions as shown in the following examples:
SEVERE: Policy Store Exception raised in getApplicationPolicyoracle.security.jps.service.policystore. PolicyObjectNotFoundException: JPS-04028: Application with name "cn=ADRGroups,cn=Groups" does not exist. SEVERE: RuntimeException raised. Incorrect entry found in db for application role PJT_PROJECT_WORK_PLAN_MANAGEMENT_DUTY. May require reconciliation with target LDAP Processing row with grant_guid: F9C89E5D04C2322629EBE642337695FC. ROLE_NAME is PJT_PROJECT_WORK_PLAN_MANAGEMENT_DUTY ROLE_NAME_SPACE is cn=ADRGroups,cn=Groups. PJT_PROJECT_WORK_PLAN_MANAGEMENT_DUTY GUID in database is 61065B6FEA8E3824B74476B1A315FDE4 Runtime Exception is oracle.jbo.JboException: JBO-29114 ADFContext is not setup to process messages for this exception. Use the exception stack trace and error code to investigate the root cause of this exception. Root cause error code is JBO-29000. Error message parameters are {0=oracle.security.jps.service.policystore.PolicyObjectNotFoundException, 1=JPS-04028: Application with name "cn=ADRGroups,cn=Groups" does not exist.}
Solution
These warnings and errors have no impact on functionality and can be ignored.
Problem
Errors related to missing approles may be reported during the Applying Online BI Metadata and Configuration Updates configuration assistant. These errors are reported in bi_webcat_patch.log
, and can be ignored, as they have no impact on the upgrade.
Solution
If Upgrade Orchestrator stops due to this error, resume the upgrade.
The following ignorable errors may be encountered while running the catbundle.sql
script or its rollback script:
ORA-29809: cannot drop an operator with dependent objects
ORA-29931: specified association does not exist
ORA-29830: operator does not exist
ORA-00942: table or view does not exist
ORA-00955: name is already used by an existing object
ORA-01430: column being added already exists in table
ORA-01432: public synonym to be dropped does not exist
ORA-01434: private synonym to be dropped does not exist
ORA-01435: user does not exist
ORA-01917: user or role 'XDB' does not exist
ORA-01920: user name '<user-name>' conflicts with another user or role name
ORA-01921: role name '<role name>' conflicts with another user or role name
ORA-01952: system privileges not granted to 'WKSYS'
ORA-02303: cannot drop or replace a type with type or table dependents
ORA-02443: Cannot drop constraint - nonexistent constraint
ORA-04043: object <object-name> does not exist
ORA-29832: cannot drop or replace an indextype with dependent indexes
ORA-29844: duplicate operator name specified
ORA-14452: attempt to create, alter or drop an index on temporary table already in use
ORA-06512: at line <line number>
Note that if this error follow any of above errors, then can be safely ignored.
ORA-01927: cannot REVOKE privileges you did not grant
Problem
SQL> old 1: GRANT CREATE SESSION TO &&1 new new 1: GRANT CREATE SESSION TO LCM_EXP_ADMINGRANT CREATE SESSION TO LCM_EXP_ADMINERROR at line 1: RA-01917: user or role 'LCM_EXP_ADMIN' does not exist
Solution
The LCM Seed Utility needs to be re-run after correcting the issues related to the database.
Problem
upgradeidmbinaries
plug-in fails in running process check on Solaris platforms. The following is an example error:
[2016-11-25T14:54:44.631+00:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 36] Checking for server process: /u01/products/dir [2016-11-25T14:54:44.659+00:00] [orchestration] [NOTIFICATION] [] [oracle.orchestration] [tid: 36] ERROR: Some unexpected processes or servers running, stop them manually and start the upgrade again.
Solution
PERL_LOCATION
property in IDM.properties
instead of using a symlink path such as /u01
as shown in the following example:
PERL_LOCATION=/scratch/mwport/IDM_SETUP/products/dir/oid/perl
PERL_LOCATION=/u01/products/dir/oid/perl
The following is the log files location for Tagging JAZN Policies:
APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/configlogs/fapatch_Tagging_JAZN_Policies_timestamp.log
Problem
Failed to tag JAZN policy for stripe: stripename
Solution
Contact Oracle Support to request assistance in resolving the issue.