9 Monitor and Troubleshoot the Upgrade

1. Run the following command:

   "echo hello | /usr/sbin/sendmail email_address"
   

2. If emails are not being sent, restart the mail server and test again.

   /etc/init.d/sendmail restart
   

3. Ensure that all properties related to email are populated with the correct values in the pod.properties file. For more information, see Table 11-1.

Failed to load prevayler under path_for_snapshot: Chunk header corrupted in the log file.

1. Review the log file to ensure there is no "out of disk space" exception.
2. If there is no "out of disk space" exception, restart orchestration on the host where the failure occurred. If there is an "out of disk space" exception, ensure there is enough disk space and then restart orchestration.

Problem

Orchestration fails when running the BackupOPSS plug-in, which backs up the OPSS Security Store, with the following error:

ORCH-DOWNTIME-00001 : Plugin Failed with error.
Please enter bind password:
ldap_search: No such object

This error occurs because the OPSS Security Store assumes that jpsroot is FAPolicies.

Solution

If the jpsroot is not FAPolicies, the workaround is to back up the OPSS Security Store manually. To back up all data under the root node of the OPSS Security Store and to back up the bootstrap wallet, perform the following steps:

Ensure the backups are performed in directories from which they can be restored. Use any directory to back up the data, as long the location from where to restore the backup is known.

1. Using Fusion Applications Control, perform the following steps to identify the root node in the Oracle Internet Directory that hosts the OPSS Security store:

   1. Open the Farm_CommonDomain.

   2. Open the WebLogic Domain.

   3. Open the CommonDomain.

   4. Find the domain name of the root node under Root Node Details, which is under the Edit Security Provider region. In the case of an upgrade failure, restore this entire node.

2. Perform the following ldifwrite and bulkload operations on the system where the Oracle Internet Directory hosting the OPSS Security store resides. When initiating ldifwrite and bulkload, Oracle Internet Directory requires the Oracle Internet Directory process and the database behind Oracle Internet Directory to be up and running:

   1. Set the following environment variables:

      (Unix)
      setenv ORACLE_HOME  OID_ORACLE_HOME
      setenv ORACLE_INSTANCE  OID_INSTANCE_HOME
      
      

      For example:

      (Unix)
      setenv ORACLE_HOME /u01/oid/oid_home 
      setenv ORACLE_INSTANCE /u01/oid/oid_inst
       
      

   2. Create the backup in the SHARED_UPGRADE_LOCATION/POD_NAME/release/ directory as follows:

      In the system where the Oracle Internet Directory is located, produce an LDIF file by running ldifwrite as illustrated in the following command. The Operational Data Store (ODS) password is required.

      OID_HOME/ldap/bin/ldifwrite connect="srcOidDbConnectStr" basedn="cn=FAPolicies", "c=us" ldiffile="srcOid.ldif"
      

      For example:

      /u01/oid/oid_home/ldif/bin/ldifwrite connect="oidddb" basedn="cn=FAPolicies" ldiffile="srcOid.ldif"
      

      This command writes all entries under the node cn=FAPolicies to the file srcOid.ldif. After generated, move this file to the directory that was previously identified, to hold all backup data.

   3. Perform the following steps if there is a need to restore the backup:

      1. Ensure Oracle Internet Directory is up and running.

      2. Perform a bulkdelete on Oracle Internet Directory nodes.

      3. In the Oracle Internet Directory system, verify that there are no schema errors or bad entries by running bulkload, as illustrated in the following command:

         OID_HOME/ldap/bin/bulkload connect="dstOidDbConnectStr" check=true generate=true restore=true file="fullPath2SrcOidLdif"
         

         If duplicate DNs (common entries between the source and destination directories) are detected, review them to prevent unexpected results.

      4. Load data into the Oracle Internet Directory by running bulkload as shown in the following command:

         OID_HOME/ldap/bin/bulkload connect="dstOidDbConnectStr" load=true file="fullPath2SrcOidLdif"
         

3. Back up the cwallet.sso file in the DOMAIN_HOME/config/fmwconfig/bootstrap directory for each WLS domain in an Oracle Fusion Applications installation. Take backups of each cwallet.sso file for each domain and when restoring, be careful to restore the correct file. For example, cwallet.sso is backed up from the Common Domain, then restore it in the Common Domain upon failure. If cwallet.sso is backed up from the BI domain, restore it to the BI Domain upon failure.

4. Resume orchestration to proceed with the upgrade.

Problem

The Database Credential Store Retrofit Utility or CSF Cache Utility steps fail. Sometimes, these steps fail because of a case mismatch in the value of the db unique name when compared between the value you get from the table (V$DATABASE) and the one listed in the oratab file on the db host.

Solution

RUP Installer runs numerous configuration assistants and is the primary utility called by Upgrade Orchestrator. In the case of a failure while RUP Installer is running, refer to information in General Troubleshooting for Upgrade Orchestrator Failure applies. In addition to the Oracle Fusion Applications Orchestration Report and the log location, the RUP Installer Report location is also included as part of the notification that is sent. For more information, see Review the Post RUP Installer Report. For information about specific configuration assistants, see RUP Installer Configuration Assistants.

Most configuration assistants are configured to automatically retry after a failure. The default number of retries is three times and the default wait time between retries is two minutes. After ten minutes, no further retries are attempted. When reviewing log files after a failure, the number of retry attempts is indicated by "Autoretrying attempt "{0}" of "{1}".

Problem

The pre-copy phase of RUP installer fails, causing the installer to terminate.

Solution

To resolve this failure during the first RUP Installer, perform the following steps:

1. Open the checkpoint.xml file located in the central_inventory_location/checkpoint/farup1/version/ directory.

2. Edit the following line in the checkpoint file:

   <module name="install" type="install" status="Success">
   

   Update the value for status as follows:

   <module name="install" type="install" status="Config">
   

To resolve this failure during the second RUP Installer, perform the following steps:

1. Open the checkpoint.xml file located in the central_inventory_location/checkpoint/fusionapps/version/ directory.

2. Edit the following line in the checkpoint file:

   <module name="install" type="install" status="Success">
   

   Update the value for status as follows:

   <module name="install" type="install" status="Config">
   

Problem

RUP Installer fails due to thread calls and reports errors similar to the following example:

"Thread-11" id=29 idx=0x98 tid=25751 prio=5 alive, native_blocked
     at java/io/UnixFileSystem.getBooleanAttributes0(Ljava/io/File;)I(Native
 Method)
     at java/io/UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
     at java/io/File.exists(File.java:733) 

Solution

Restart RUP Installer by resuming Upgrade Orchestrator.

Problem

An installer session was shut down during the upgrade.

Solution

If orchestration or tasks spawned by orchestration, such as RUP Installer, are killed in the middle of any process, the system may not be in a recoverable state and the state should be carefully reviewed by contacting Oracle Support before proceeding.

To recover from this situation, restore the backup of APPLICATIONS_BASE and start from the beginning of the upgrade.

Problem

[2017-07-22T15:03:59.188+00:00] [log] [NOTIFICATION] []  
[ApplyPrePSAMiddlewarePatches2017-07-22_02-57-35PM.log] [tid: 1] Job  validation successful, setting the req env to run the job...  
[2017-07-22T15:03:59.188+00:00] [log] [NOTIFICATION] [PAPUTL020]  
[ApplyPrePSAMiddlewarePatches2017-07-22_02-57-35PM.log] [tid: 1] Executing  the command [<APPLTOP>/fusionbhd/OPatch/opatch disablecas  -oh <APPTOP>/fusionbhd -jre  
<APPLTOP>/fusionapps/jdk]  
[2017-07-22T15:03:59.188+00:00] [log] [NOTIFICATION] []  
[ApplyPrePSAMiddlewarePatches2017-07-22_02-57-35PM.log] [tid: 1] Oracle Home  is: <APPLTOP>/fusionbhd  
The java.library.path system variable is missing or invalid. Please set  java.library.path with a correct value and retry the operation.

Solution

Problem

The Applying Online BI Metadata and Configuration Updates configuration assistant reports the following JPS exception:

oracle.security.jps.JpsException: JPS-01016: A password credential is expected;
instead found null for alias AuditDbPrincipalMap and key AuditDbPrincipalKey at location
/u01/APPLTOP/instance/domains/bi.oracleoutsourcing.com/BIDomain/config/fmwconfig/bootstrap.
 

Solution

This exception has no functional impact on the upgrade and can be ignored.

Problem

The RUP configuration assistant "Generating OHS Reference Configuration File" fails initially while processing the context root metadata files from FMW. For example, the files under /APPTOP/fusionapps/applications/lcm/common/fmwohs.

Solution

Problem

[2017-09-01T07:06:17.166+00:00] [apps] [ERROR] [] [oracle.apps.ad.rupconfig.Applying_Admin_Server_Online_Setting_and_Configuration_Changes] [tid: 32] [ecid: 0000LswBvjR7y0I_IpP5if1PeG7o000006,0] CFGEX-00024 : Config plugin "Updating JPS Config Timeout Settings" failed. No further plugins will be run.

Solution

This error message is expected when upgrading from release 8 or 9 directly to release 12. It does not cause any functional impact and you can skip this failed task and continue with upgrade.

Problem

ERROR: CFGLOG-00104 : "Starting All Servers" configuration failed. 
ERROR: CFGLOG-00175 : Startup of server "BIInstance~coreapplication_obisch1~OracleBISchedulerComponent~coreapplication_obisch1" in domain "BIDomain" "failed".
ERROR: CFGLOG-00175 : Startup of server "BIInstance~coreapplication_obis1~OracleBIServerComponent~coreapplication_obis1" in domain "BIDomain" "failed".
ERROR: CFGLOG-00175 : Startup of server "BIInstance~coreapplication_obiccs1~OracleBIClusterControllerComponent~BIClusterController" in domain "BIDomain" "failed".
ERROR: oracle.as.install.fapatchconfig.exception.PatchsetConfigException:CFGEX-00069 : Startup failed for following domain:server pairs"
[BIDomain:BIInstance~coreapplication_obisch1~OracleBISchedulerComponent~coreapplication_obisch1,
BIDomain:BIInstance~coreapplication_obis1~OracleBIServerComponent~coreapplication_obis1,BIDomain:BIInstance~coreapplication_obiccs1~OracleBIClusterControllerComponent~BIClusterController]".

Solution

Problem

Verifying Node Manager and OPMS Status fails with the following error:

WLSTException: Error occured while performing nmConnect : 
Cannot connect to Node Manager. : 
[Security:090542]Certificate chain received from <hostname> - <host IP address> was not trusted causing SSL handshake failure.

Solution

The issue can occur when the host associated with a node manager is explicitly bounced in the middle of the upgrade, and if Upgrade Orchestrator expects the node manager to be in a shutdown state at that time. The node manager on the host may be configured to automatically start up as part of the system boot process and could cause various issues depending on which upgrade step was being performed when the host was restarted. To resolve this issue, stop and restart node manager on the host where the issue was reported. For more information, see Start Node Manager in the Oracle Fusion Applications Administrator's Guide.

Problem

Upgrade Orchestrator fails to stop OPMN processes with an error similar to either of the following errors:

• Exception: OPMN could not be stopped. Script will exit. Please stop OPMN manually before re-running the script.

• /APPLICATIONS_BASE/webtier_mwhome/oracle_common/jdk/jre/lib/fonts/ALBANWTJ.ttf – No such file exists.

Solution

This issue can occur due to an incompatible version of JDK being used during the process. To resolve the issue, perform the following steps.

1. cd /APPLICATIONS_BASE/webtier_mwhome/webtier

   mv jdk_backup_existing_version jdk

2. cd /APPLICATIONS_BASE/webtier_mwhome/oracle_common

   rm –rf jdk

   cp –Rp jdk_bkp_130320_1250 jdk

3. Resume orchestration.

Problem

The Verifying Node Manager and OPMN Status configuration assistant fails.

Solution

Do not exit out of Upgrade Orchestrator in response to this configuration assistant failure. Perform the following steps to recover:

1. Review the node manager log files to determine the cause of the failure:

   APPLICATIONS_CONFIG/nodemanager/host_name/

   Note that the APPLICATIONS_CONFIG value can be obtained from the APPLICATIONS_BASE/fusionapps/faInst.loc file.

2. After resolving the issue that caused the failure, start the Node Manager on all hosts that are part of the Oracle Fusion Applications provisioned system. For more information, see Start Node Manager in the Oracle Fusion Applications Administrator's Guide.

3. Restart the OPMN server for BI, GOP (if GOP is installed), and Web Tier. If the Web Tier (OHS) installed with the Oracle Fusion Applications middle tier is run, it is possible to start it using the following steps. If the Web Tier is run on a separate machine, it may be possible to run the steps below on the other machine. In either case, ensure that Web Tier (OHS) is up at this point:

   Example for BI: (note the usage of start instead of startall)

   cd APPLICATIONS_CONFIG/BIInstance/bin
   ./opmnctl start
   

   Example for GOP: (note the usage of start instead of startall) Note that the OPMN server for GOP should be started from the machine that hosts the Supply Chain Management Administration Server domain. Start the OPMN server for GOP only if GOP is installed.

   cd APPLICATIONS_CONFIG/gop_1/bin
   ./opmnctl start
   

   Example for Web Tier: (note the usage of start instead of startall)

   cd APPLICATIONS_CONFIG/CommonDomain_webtier/bin
   ./opmnctl start
   

   For more information about the location of APPLICATIONS_BASE and APPLICATIONS_CONFIG, see Directories Structure Overview.

   The BI, Web Tier, and GOP (when applicable) processes managed by OPMN are started by the installer in the Starting All Servers configuration assistant.

4. Fix any other environment issues before resuming the upgrade. If the installer exits for any reason, make sure that all node managers and OPMN processes are running. Contact Oracle Support Services to proceed out of this step if there are unresolved environment issues.

5. After starting the services, resume orchestration to proceed to the Starting All Servers configuration assistant. If the starting of servers times out, see Troubleshoot Server Start and Stop Failures.

If GOP is not installed, the user interface reports "Success" for GOP OPMN status, but the log file reports: GOP is not provisioned. Skipping check for OPMN status.

This section describes two scenarios that can prevent the node manager from starting between the first and second installer.

Problem 1

The node manager was manually started before or during the Extending Certification Validity configuration assistant. The node manager caches the previous keystore certificates so the updated certificates are not validated and the node manager fails to start.

Solution

Check the node manager logs to determine if it is running and when it was last started. If the time stamp is earlier than the Extending Certification Validity configuration assistant execution time stamp, restart the node manager so that it reads the updated keystore certificates.

1. To find out if the node manager is running for a specific host, connect to the host and run the following command:

   If any results are returned, the node manager is running.

   ps -ef | grep nodemanager
   

2. If the node manager is running, find the time of the last entry of <Secure socket listener started on port nnnn> in the following directory.

   APPLICATIONS_CONFIG/nodemanager/logical_host_name/nodemanager
   

3. To check the timestamp for the Extending Certification Validity configuration assistant, find the fapatch_Extending_Certificate_Validity_XXXX file in one of the following directories:

   APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/configlogs
   
   APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/ARCHIVE/timestamp/configlogs
   

   The last time stamp entry is the execution timestamp.

Problem 2

The administration servers in one or more domains are running before the Extending Certification Validity configuration assistant runs. This prevents validation of the updated keystore certificates and fails to provide the correct status to orchestration.

Solution

Perform the following steps:

1. Verify whether the administration server of the domain is running by launching the administration console of the domain. If the console comes up, then the administration server is running.

2. Verify the last time the administration server was started. Go to the APPLICATIONS_CONFIG/domains/logical_host_name/domain_name/servers/AdminServer/logs directory. Using the command, ls -lrt, find the most recent the AdminServer.log file. In this file, find the time of last entry that contains text similar to the following example:

   <Channel "Default" is now listening on machine_ip:port for protocols iiop, t3, ldap, snmp, http.>
   

Problem

The following error occurs on the OHS node:

[Plugin failed]: StopOPMNProcesses 
[Phase DowntimePostFA]: failed

ORCH-DOWNTIME-OPMN-00004 : Stopping OPMN Control Process failed. Review log file

Solution

This error can occur when OHS is on the same node as the primordial host and APPLICATIONS_CONFIG is not APPLICATIONS_BASE/instance. For example, this error occurs if APPLICATIONS_BASE =/APPTOP and APPLICATIONS_CONFIG =/instance.

To resolve the issue, perform the following steps:

1. Create a link from APPLICATIONS_BASE/instance to APPLICATIONS_CONFIG as follows:

   ln -s APPLICATIONS_CONFIG APPLICATIONS_BASE/instance
   

2. Resume orchestration.

3. Delete the link after orchestration completes successfully.

Problem

RUP Lite for OHS fails during the ohs.plugin.UpgradeWebtier step due to missing the jdk directory.

Solution

Verify if there is a jdk_backup_existing_version directory under WT_ORACLE_HOME. If this directory exists, rename it to jdk and resume Orchestration.

Also, if the missing jdk directory is from WT_MW_HOME/oracle_common, check to see if there is a jdk_backup_existing_version directory under this directory. If so, rename it to jdk and resume Orchestration.

Problem

During the upgrade, RUP Lite for OHS fails with the following error:

Failed execution of plugin: ohs.plugin.ReassociateCommonDomain 

Solution

Update the server_name/instance/CommonDomain_webtier_local/config/OPMN/opmn/instance.properties file to set the registered property to true. Then check the Administration Server on either the Common Domain or the OSN Domain to ensure it is running. If not, bounce the server and retry RUP Lite for OHS by resuming orchestration.

Problem

RUP Lite for OHS reports a server side error occurs with the following error message:

Server instance is not running for the security mode specified: "simple". Try again using a different security mode. The remote registration process did not succeed! Please find the specific error message below.

Solution

To resolve this issue, perform the following steps:

1. Log in to the OAM administration console.

2. From the System Configuration tab, click Server_Instances, and double click the OAM server instance, such as oam_server1.

3. Select simple from the Mode field in the right panel.

4. Click Apply to submit the changes.

5. Restart the OAM Server.

6. Restart all OHS servers in the environment.

7. Resume Upgrade Orchestrator.

Check the Oracle Fusion Applications OHS to ensure that SSO still works after the change. If it does not, upgrade Webgate manually for the Oracle Fusion Applications OHS.

Problem

While the primordial host is waiting for IDMOHS:IDMUpgradeDone, there are communication exceptions on the PRIMORDIAL console.

Solution

These errors can be ignored and orchestration can be resumed.

Problem

If the OAM administrator password contains special characters, such as '#' or '&', the OAM Configuration step fails. To work around this issue, manually validate that the OAM Administration Server host/port and username/password are correct. After manually validating this information, proceed with the manual upgrade.

Solution

To validate, perform the following steps:

1. Get the OAM administrator user name and password from the credential store.

2. Run APPLICATIONS_BASE/fusionapps/oracle_common/common/bin/wlst.sh.

3. Run the following commands to connect to the Common Domain Administration Server and get the OAM administrator username and password:

   connect()
   listCred(map='oracle.patching', key='FUSION_APPS_PATCH_OAM_ADMIN-KEY')
   

4. Get the OAM Administration Server host and port from the following properties in APPLICATIONS_CONFIG/fapatch/FUSION_prov.properties:

   • OAM_ADMIN_SERVER_HOST

   • OAM_ADMIN_SERVER_PORT

5. Use oamcfgtool.jar to confirm whether the OAM server can be invoked using the values found in the previous steps. For example:

   cd APPLICATIONS_BASE/fusionapps/oracle_common/modules/oracle.oamprovider_11.1.1
   
   java -jar oamcfgtool.jar app_domain=crm web_domain=OraFusionApp
   uris_file=APPLICATIONS_BASE/fusionapps/applications/crm/security/oam.conf
   oam_aaa_mode=open_or_simple app_agent_password=password_for_map="oracle.patching" 
   key="FUSION_APPS_PATCH_OAM_RWG-KEY"_in_credential_store
   primary_oam_servers=oam_server1 oam_admin_server=http://OAM_admin_server_host:port
   oam_admin_username=username_for_FUSION_APPS_PATCH_OAM_ADMIN-KEY
   oam_admin_password=password_for_FUSION_APPS_PATCH_OAM_ADMIN-KEY
   oam_version=11 default_authn_scheme=FAAuthScheme
   

6. If the previous command is successful, the validation is successful and orchestration can be resumed.

This troubleshooting step is only applicable if the OAM configuration fails while you are performing the steps to remove OVD from your environment as listed in Update OAM Configuration.

Problem

OAM configuration update fails for OID authenticator.

Solution

cd  $OAM_ORACLE_HOME/common/bin
setenv DOMAIN_HOME /u01/app/idm/admin/IDMDomain/aserver/IDMDomain$DOMAIN_HOME/bin/setDomainEnv.sh
./wlst.sh
connect('weblogic_idm',’$WLS_ADMIN_PASSWORD’,'t3://myhost.example.com:7001')
editUserIdentityStore(name="OIMIDStore", ldapUrl="ldap://idstore.example.com:3060", ldapProvider="OID", userSearchBase=”cn=Users,dc=us,dc=oracle,dc=com”, groupSearchBase=”cn=Groups, dc=us,dc=oracle,dc=com)

The location of the Governance, Risk, and Compliance (GRC) policies varies, depending on the upgrade path to Release 12. GRC polices are located in the grc OAM application domain if the Oracle Fusion Applications environment was originally provisioned with either version 11.1.1.5 or 11.1.2, then upgraded to version 11.1.3, and beyond. If the environment was originally provisioned with version 11.1.3 and upgraded to version 11.1.4 and beyond, the GRC policies are located in the fs OAM application domain.

Problem

Due to a failure during the upgrade, it is necessary to restore all of the data under the root node of the OPSS Security Store.

Solution

To restore all of the data under the root node of the OPSS Security Store, perform the following steps:

1. Ensure Oracle Internet Directory is up and running.

2. Perform a bulkdelete on Oracle Internet Directory nodes.

3. In the Oracle Internet Directory system, verify that there are no schema errors or bad entries by running bulkload, as illustrated in the following command:

   OID_HOME/ldap/bin/bulkload connect="dstOidDbConnectStr" check=true generate=true restore=true file="fullPath2SrcOidLdif"
   

   If duplicate DNs (common entries between the source and destination directories) are detected, review them to prevent unexpected results.

4. Load data into the Oracle Internet Directory by running bulkload, as illustrated in the following command:

   OID_HOME/ldap/bin/bulkload connect="dstOidDbConnectStr" load=true file="fullPath2SrcOidLdif"
   

Problem

Applying Release12 One-Off patches on an Release 12 setup fail.

Workaround

mkdir /u01
ln -s /abc/idmtop /u01/IDMTOP 

If Webgate is not configured on the OHS SO Node, perform the following steps:

1. Log in to the OHS SO Node.

2. Append environment variable LD_LIBRARY_PATH with the Web Tier library path APPLICATIONS_BASE/webtier_mwhome/webtier/lib as follows:

   WEBHOST2> export LD_LIBRARY_PATH=APPLICATIONS_BASE/<webtier_mwhome>/<webtier_directory_name>/lib:$LD_LIBRARY_PATH
   

   For example:

   WEBHOST2> export LD_LIBRARY_PATH=/u01/app/idm/products/ohs/ohs/lib:$LD_LIBRARY_PATH 
   

3. Execute deployWebgateInstance.sh as follows:

   1. Execute the following command:

      cd <WEBTIER_MW_HOME>/webgate/webgate/ohs/tools/deployWebGate 
      ./deployWebGateInstance.sh -w <WEBTIER_INSTANCE_HOME>/config/OHS/ohs2/ -oh <WEBTIER_MW_HOME>/webgate 
      

      For example:

      ./deployWebGateInstance.sh -w 
      /u02/local/idm/config/instances/ohs2/config/OHS/ohs2/ -oh 
      /u01/app/idm/products/ohs/webgate
      

   2. Execute the following command:

      cd <WEBTIER_MW_HOME>/webgate/webgate/ohs/tools/setup/InstallTools 
      ./EditHttpConf -w <WEBTIER_INSTANCE_HOME>/config/OHS/ohs2/ -oh <WBETIER_ME_HOME>/webgate -o webgate.conf 
      

      For example:

      ./EditHttpConf -w /u02/local/idm/config/instances/ohs2/config/OHS/ohs2/ -oh
      /u01/app/idm/products/ohs/webgate -o webgate.conf 
      

   3. Copy the following from WEBHOST1 to WEBHOST2:

      • From <WEBTIER_INSTANCE_HOME>/config/OHS/ohs1/webgate/config, copy the following to <WEBTIER_INSTANCE_HOME>/config/OHS/ohs2/webgate/config on WEBHOST2:

        "ObAccessClient.xml","cwallet.sso","password.xml"
        

      •  From <WEBTIER_INSTANCE_HOME>/config/OHS/ohs1/webgate/config/simple, copy the following to <WEBTIER_INSTANCE_HOME>/config/OHS/ohs2/webgate/config/simple on WEBHOST2:

        "aaa_key.pem","aaa_cert.pem"
        

4. Restart Oracle HTTP Server as follows:

   WEBHOST2> cd <WEBTIER_INSTANCE_HOME>/bin 
   WEBHOST2> ./opmnctl stopall  
   WEBHOST2> ./opmnctl startall
   

5. Ensure that the URLs are accessible as before, by temporarily stopping Oracle HTTP server on the Node 1 and making accessible the URLs.

Problem

51fea9603f49 ,lastModified:2010-01-10T17:46:52.000-0800 
,ioe:java.util.zip.ZipException: Could not find End Of Central Directory 
<ERROR> Error message was found in log file on line ::2905:: -->WARNING: 
corrupted jar found. 
file:/u01/IDMTOP/products/app/iam/oam/server/lib/jmx/oam-dummy-config.xml, 
md5sum:2c3ede41786705c57cf151fea9603f49 
,lastModified:2010-01-10T17:46:52.000-0800 ,ioe:java.util.zip.ZipException: 
Could not find End Of Central Directory 
<ERROR> rror message was found in log file on line ::2991:: -->WARNING: 
corrupted jar found.
file:/u01/IDMTOP/products/app/iam/oam/server/lib/jmx/oam-dummy-config.xml, 
md5sum:2c3ede41786705c57cf151fea9603f49 
,lastModified:2010-01-10T17:46:52.000-0800 ,ioe:java.util.zip.ZipException: 
Could not find End Of Central DirectoryExit Value from individual subroutine 
check is 1

Solution

Ignore the error message.

Problem

Migration or upgrade fails with permission issues such as unable to access or unable to write.

Solution

To resolve this issue, check the failed file’s user and group privileges and permissions of the stagedir and idmUpgrade folders. The user and group permissions should be same as the user who is running the upgrade. In addition, all of the folders need to be writable.

Problem

The OIF URL is not accessible post migration.

Solution

Problem

Download email template from OIM fails in Run Upgrade Orchestrator During Downtime.

Solution

This is an expected failure. The Download email template step is executed again as part of the IDM upgrade in Manually Download OIM Email Template.

Problem

The SOA managed server fails to start on the scaled out machine with the following exception in the SOA logs under /u01/IDMTOP/config/scripts/logs:

Starting server wls_soa2 ...Access to domain 'IDMDomain' for user 'admin' denied 
No stack trace available.

Solution

Problem

OIM binary upgrade fails with no "IDMTOP/products/app/utils/orainst.loc" error.

One possible cause is that you may be using the upgradeOnPremise.properties under the idmUpgrade folder for upgrade rather than the one under stagedir. For type II upgrades, upgradeOnPremise.properties is auto-generated by the discovery tool during discovery execution by seeding necessary values for upgrade.

Solution

To resolve this issue, rerun upgrade using the upgradeOnPremise.properties under stagedir.

Problem

Migration exits with an error to stop the processes and restart.

Solution

Problem

An error occurred during the Applying Pre-PSA Middleware Patches or Applying Post-PSA Middleware Patches configuration assistant.

Solution

Review the log file in the relevant location to find the cause of the error:

APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/ApplyPrePSAMiddlewarePatchestimestamp.log

APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/ApplyPostPSAMiddlewarePatchestimestamp.log

For Language Pack failures, review the following log files:

• Failures during Install mode and Standalone LP installation: APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/LanguagePack/language/FAPatchManager_ApplyMWLangPackPatchestimestamp.log

• Failures during LP upgrade through orchestration (configuration mode):

  APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/LanguagePack/MergedLPs/FAPatchManager_ApplyMWLangPackPatchestimestamp.log

For specific OPatch failures, go to each of the individual Oracle home directories to find the details of the OPatch errors. For example, for a SOA failure, go to APPLICATIONS_BASE/fusionapps/soa/cfgtoollogs/opatch.

Problem

The Applying Middleware Patchsets configuration assistant fails with an error as shown in the following example:

[as] [ERROR] [] [oracle.as.install.engine.modules.presentation] [tid: 15]
[ecid: 0000JsNJml6AxGGpIww0yf1HRacu000006,0] sun/awt/X11GraphicsEnvironment[[
.
java.lang.NoClassDefFoundError: sun/awt/X11GraphicsEnvironment
         at java.lang.Class.forName0(Native Method) 

Solution

Unset the DISPLAY variable or set it to the correct value. To unset it, run "unset/unsetenv DISPLAY" on all hosts. Then resume Upgrade Orchestrator.

Problem

The Applying Post-PSA Middleware Patches configuration assistant hangs.

Solution

This problem is most likely due to adpatch hanging as the result of the java worker not getting the database connection. Resolve this issue by following the steps in Troubleshoot Loading Database Components. Run the commands from ATGPF_ORACLE_HOME instead of FA_ORACLE_HOME.

Problem

The following error occurs:

OPatch cannot continue because it can't load library from the directory "<dbclient Oracle Home>/oui/lib/linux64"
 

Solution

This error may occur if the OUI version in the database client Oracle home is 11.2 while the OUI version in Oracle Fusion Applications Oracle home (FA_ORACLE_HOME) is 11.1.

To resolve this issue, perform the following steps:

1. Go to the DB Client home.

2. Set the ORACLE_HOME environment variable to point to the database client Oracle home.

3. Apply the database client patches using the following command:

   $ORACLE_HOME/OPatch/opatch apply patch_location
   

4. Because the patches have now been manually applied, perform the following steps to continue with the upgrade:

   1. Go to the FA_ORACLE_HOME/fusionapps/applications/lcm/tp/config/RUP/FMW directory.

   2. Open the pre-psa-jobs.xml file for editing.

   3. Comment out the job with the name dbclient as shown in the following example:

      <!-- <job>
          <id>10</id>
          <target>FAMW</target>
          <component>
                <name>dbclient</name>
                <version>11.1.1.5</version>
          <component>
          <utility_name>opatch</utility_name>
          <patch_number>NA</patch_number>
          <command>%opatch% napply -silent -skip_duplicate -skip_subset
      -oh %dbclient_home% -phBaseDir %dbclient_patch% -jre %jre_loc% -invPtrLoc
      %oraInstLocFile%</command>
           <patch_location>NA</patch_location
      </job>
      

Problem

The following error is reported:

ORA-01658: unable to create INITIAL extent for segment in tablespace FUSION_TS_SEED.

Solution

The standard output from the ORA-1658 error follows:

ORA-01658: unable to create INITIAL extent for segment in tablespace string
    Cause: Failed to find sufficient contiguous space to allocate
    INITIAL extent for segment being created.
    Action: Use ALTER TABLESPACE ADD DATAFILE to add additional space to
    the tablespace or retry with a smaller value for INITIAL 

For more information, refer to Oracle Database documentation.

This section contains the following topics:

• Log Files for Upgrading Middleware Schema

• Upgrading SES Component Fails When TDE Data Vault is Enabled

Problem

A failure occurred during either the Grant Privileges to Application Schemas or the Creating Grants/Synonyms on Application Database Objects configuration assistant.

Solution

Find the cause of the failure by running the script manually as the sysdba user, using SQL*Plus or SQL*Developer. After resolving the issue, resume orchestration.

Problem

An email notification stating that one or more database workers failed during the Loading Database Components configuration assistant is received.

Solution

This email notification is only received when the upgrade cannot progress any further and requires user intervention. In this scenario, all the workers are in a FAILED or IDLE status. The configuration assistant remains in a RUNNING status until all tasks in Loading Database Components have run. To resolve this issue, you must start AD Controller to manage the failed workers. For additional information, see Troubleshoot Patching Sessions for Database Content in the Oracle Fusion Applications Patching Guide. After resolving the issue that caused the workers to fail, and restart the workers, Upgrade Orchestrator continues processing. No further intervention is required.

Note that the messages are displayed on the console for database component failures if orchestration is run with the -DlogLevel option set to FINEST.

There might be corner cases when an alert email indicating failed workers although the workers are still running might be received. In such cases, ignore the email alert after ensuring the workers are running with no failures.

Problem

The database goes down while RUP Installer is running the Loading Database Components configuration assistant. If simply bringing the database up and then resuming orchestration, the following error might be received:

Failed to connect to the database as fusion with error: 
No more data to read from socket

Solution

To recover from this error, perform the following steps:

1. Force the database patching session to fail as follows:

   (UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail 
   

2. Start AD Controller as follows:

   (UNIX) FA_ORACLE_HOME/lcm/ad/bin/adctrl.sh
   

   For more information, see Start AD Controller in the Oracle Fusion Applications Patching Guide.

3. Perform the following steps in AD Controller to manage the workers:

   1. Select Tell manager that a worker failed its job and enter All for all workers.

   2. Select Tell worker to quit and enter All for all workers. Note that this does not kill the workers. It sends a command to the worker to shutdown after it completes the current task.

   3. Wait for all workers to complete their tasks and shut down normally.

   4. If there are still some worker processes that do not shut down, kill those processes manually by selecting Tell manager that a worker failed its job. Then select Tell manager that a worker acknowledges quit and enter All for all workers.

   5. From the operating system, check for processes that are running fapmgr, javaworker, adpatch, adadmin, sqlplus, and adworker. If any exist, terminate them from the operating system.

   6. Select Tell worker to restart a failed job and enter All for all workers.

4. Resume orchestration.

Problem

AutoPatch validation fails with the following message:

An active adpatch or adadmin session was found. Complete or terminate the active session to allow fapmgr to proceed.

Solution

Perform the following steps to resolve this error:

1. Run the fapmgr forcefail command to update the patching tables as follows:

   (UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh forcefail [-logfile log file name] [-loglevel level]
   

2. Run the fapmgr abort command from FA_ORACLE_HOME to find out if an Oracle Fusion Applications Patch Manager session must be cleaned up as shown in the following example:

   (UNIX) FA_ORACLE_HOME/lcm/ad/bin/fapmgr.sh abort [-logfile log file name] [-logLevel level]
   

   If this command finds no failed session, proceed to Step 3.

3. Run the following commands from ATGPF_ORACLE_HOME to abandon any Applications Core patching sessions or AD Administration sessions that may be running:

   (UNIX) ATGPF_ORACLE_HOME/lcm/ad/bin/adpatch.sh abandon=y interactive=n defaultsfile=APPLICATIONS_CONFIG/atgpf/admin/defaults.txt
   
   (UNIX) ATGPF_ORACLE_HOME/lcm/ad/bin/adadmin.sh abandon=y interactive=n defaultsfile=APPLICATIONS_CONFIG/atgpf/admin/defaults.txt
   

Problem

When multiple seed data files are uploaded for the same flexfield but for different flexfield contexts, the upload tasks can fail due to locking issues. The failed tasks appear in the log file as the following error:

Loading failed with a JboException: JBO-25014: Another user has changed the
row with primary keyoracle.jbo.Key ...

Solution

AutoPatch defers any failed tasks to the end of the phase and reattempts the failed tasks only after attempting all tasks in the phase at least once. Usually the flexfield seed data tasks that failed due to the locking issue succeed on subsequent attempts. You can ignore these errors if the flexfield seed data task succeeds on the retry. If the task remains in a failed state, you must use the AD Controller utility to retry the failed task.

For more information, see Restart a Failed Worker in the Oracle Fusion Applications Patching Guide.

Problem

The upgrade script, pje_txn_fix_issues_bug18504814.sql, fails with the following error:

ORA-00001: unique constraint (FUSION.PJE_ISSUE_TYPES_TL_U1) violated.

Solution

Skip the AD worker that failed while running the pje_txn_fix_issues_bug18504814.sql file.

Problem

[2016-11-22T10:23:05.464+00:00] [xdf] [ERROR] [XDF-10567] [oracle.xdf] 
[tid: 1] [ecid: 0000LY7^LAfBX7M5ING7yf1OClRT000001,0] Could not modify the column to NOT NULL. 
[2016-11-22T10:23:05.464+00:00] [xdf] [NOTIFICATION] [XDF-00005] [oracle.xdf] 
[tid: 1] [ecid: 0000LY7^LAfBX7M5ING7yf1OClRT000001,0] Alter DDL: ALTER TABLE "FUSION"."MOW_RULE_SET_DETAILS" MODIFY ("MAX_SCORE" NUMBER(9,0))

Solution

Ignore the error. Skip and restart the failed adworker. 

Log files for this configuration assistant may be found in this location:

APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/configlogs/fapatch_Deploying_Applications_Policies_(jazn-data.xml)_timestamp.log

APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/LanguagePack/language/configlogs/fapatch_Deploying_Applications_Policies_(jazn-data.xml)_timestamp.log

Problem

A failure occurs during applications policy analysis.

Solution

Review the log file that is generated by each stripe. The log files are located under the main log directory, APPLICATIONS_CONFIG/lcm/logs/11.12.0.0.0/RUP and are named as follows:

• fapatch_CRMJaznAnalysis_timestamp.log

• fapatch_FSCMJaznAnalysis_timestamp.log

• fapatch_HCMJaznAnalysis_timestamp.log

• fapatch_OBIJaznAnalysis_timestamp.log

• fapatch_SOAJaznAnalysis_timestamp.log

• fapatch_UCMJaznAnalysis_timestamp.log

• fapatch_BPMJaznAnalysis_timestamp.log

• fapatch_B2BJaznAnalysis_timestamp.log

After resolving the JAZN analysis error, retry the analysis for the failed stripe to confirm the issue is resolved.

Problem

A failure occurs during the Deploying Application Policies configuration assistant.

Solution

When a failure occurs during Deploying Application Policies, restore only the stripe or system policy that has failed, from the backup. Use the OPSS migrateSecurityStore command with the appropriate source and destination arguments to perform the restore. Do not restore a stripe that has not failed. Review the JAZN deployment log file to determine the cause of the failure, fapatch_Deploying_Applications_Policies_(jazn-data.xml)_timestamp.log.

After resolving the issue, resume orchestration. For more information, see the Migrating with the Script migrateSecurityStore section in the Oracle Fusion Middleware Applications Security Guide.

Problem

The Deploying Application Policies configuration assistant fails with the following warning:

oracle.security.jps.internal.policystore.ldap.PermissionManagerImplsearchPermissionEntry

WARNING: Duplicate permissions

Solution

Contact Oracle Support to request assistance in resolving this issue.

Problem

The following warning occurs during the Deploying Application Policies configuration assistant:

WARNING: Failed to validate the xml content. cvc-complex-type.2.4.a: Invalid
content was found starting with element 'property'. One of
'{"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":propertySetRef,
"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":extendedProperty,
"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":extendedPropertySetRef,
"http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd":serviceInstanceRef}'
is expected. Location: line 165 column 96.
WLS ManagedService is not up running. Fall back to use system properties for
configuration.

Solution

Ignore this message as there is no functional impact of this warning and the deployment is successful.

Problem

The following warning occurs during the Deploying Application Policies configuration assistant:

FINE: Application policies already exists for application: fscm
oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException:
Cannot create application policy context "fscm".
        at
oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.unsync_createApp
licationPolicy(LdapPolicyStore.java:933)
        at
oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.createApplicatio
nPolicy(LdapPolicyStore.java:753)
        at
oracle.security.jps.internal.tools.utility.destination.apibased.JpsDstPolicy.c
lone(JpsDstPolicy.java:805) 

Solution

Ignore this message as there is no functional impact of this warning and the deployment is successful.

Problem

The IDM Server goes down during the Deploying Application Policies configuration assistant and the deployment fails.

Solution

Upgrade Orchestrator does not allow a retry after this type of failure. Instead, exit orchestration and restore from the IDM backup. Then, resume orchestration.

Problem

A failure during the Starting All Servers configuration assistant typically happens when one of the servers times out and fails to start due to resource issues or application specific issues.

Solution

Various platforms and environment configurations can impact how long it takes all servers to actually start during the Starting All Servers configuration assistant. Although the installer waits an average amount of time for this configuration assistant to complete before it is marked as Failed, different platforms may require more time. It is not unusual to receive timeout errors in the log files if the starting of all servers for the environment requires more time than the installer allows.

If this configuration assistant fails, perform the following steps:

1. Monitor the status of the servers by reviewing the messages in the server log files or on the console. The log file, APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/StartStop/fastartstop_timestamp.log, indicates which server started and failed to start. For example:

   Time out while performing Start for domain SCMDomain. Waited for 2400 seconds
   [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [UTIL] [oracle.apps.startstop.util.MbeanUtil: runSSCommandOnDomain.868] [tid:37] Start operation is completed for domain SCMDomain. Please see SCMDomain.log for details.
   
   [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [UTIL] [oracle.apps.startstop.invoke.StartStopTask: call.221] [tid:37] StartStopTask over for domain SCMDomain
   
   [2011-10-21T03:57:52.052--8:00] [fastartstop] [NOTIFICATION:1] [SST] [oracle.apps.startstop.invoke.StartStopTask: call.223] [tid:37] Finished the task for the Domain SCMDomain
   

2. Review the log files at the domain level to see a summary of the server status for that domain: APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/StartStop/domain name_timestamp.log.

3. Review the corresponding server logs for the failed servers under the following directory:

   APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs
   

4. After determining and resolving the cause of the failure, restart Upgrade Orchestrator.

Problem

The following exception during the Starting all Servers configuration action indicates a failure in starting the BIServer:

Start all servers fails to start
Start operation on the component :coreapplication_obips1:, for the instance
:BIInstance: - FAILED

The coreapplication_obips1 server log file reports the following error:

ecid:]]
[2012-04-10T00:22:20.000-07:00] [OBIPS] [ERROR:16] []
[saw.security.odbcuserpopulationimpl.initialize] [ecid: ] [tid: ] Unable to
create a system user connection to BI Server during start up. Trying again.[[
File:odbcuserpoploaderimpl.cpp
Line:325
Location:
saw.security.odbcuserpopulationimpl.initialize
saw.catalog.local.loadCatalog
saw.subsystems.catalogbootstrapper.loadcatalog
saw.webextensionbase.init
saw.sawserver
ecid:]]
[2012-04-10T00:22:25.000-07:00] [OBIPS] [NOTIFICATION:1] [] [saw.sawserver]
[ecid: ] [tid: ] Oracle BI Presentation Services are shutting down.[[
File:sawserver.cpp
Line:706
Location:
saw.sawserver
ecid:

Solution

To work around this issue, perform the following steps:

1. Resume orchestration, which shuts down and starts bi_server1.

2. Monitor the fastartstop log files and the state of bi_server1(BIDomain).

3. When bi_server1 restarts, as indicated by a RUNNING status, start the component coreapplication_obiccs1 or all the components of type OracleBIClusterControllerComponent using opmnctl as shown in the following example:

   */BIInstance/bin/opmnctl startproc ias-component=coreapplication_obiccs1
   

Problem

The OHS diagnostic log contains the following error message:

No such file or directory:  Couldn't create accept lock

Solution

This issue could be the result of the hypervisors going down, resulting in bringing the Oracle VM servers down. Perform the following steps to resolve the error:

1. Find the entry for the lock file in httpd.conf, for example:

   • For Apps OHS: "/dev/shm/ohs_ohs1_http_lock"  

   • For OSN OHS: "/dev/shm/osn_ohs1_http_lock"

2. Confirm whether the directory that contains the lock file exists.

3. Assuming this directory does not exist, create the directory.

Problem

The following error is reported during Online Preverification:

weblogic.management.mbeanservers.edit.EditTimedOutException 
 

Solution

It may be necessary to manually release the edit session. For more information, see Resolve an EditTimedOutException Error in the Oracle Fusion Applications Patching Guide.

Problem

The Starting All Servers configuration assistant in RUP Installer fails to start ess_server1 and reports the following error in the ess_server1.log:

weblogic.rmi.extensions.DisconnectMonitorUnavailableException: Could not
register a DisconnectListener

Solution

Perform the following steps to resolve this issue:

1. Open the Oracle Enterprise Manager console for the domain.

2. Navigate to the following location:

   • From the console, expand the WebLogic Domain

   • Go to ESSCluster, then ess_server1

   • Right click and open System MBean browser

   • Go to ess_server1, ServerStart, select ess_server1, and click Arguments

3. Verify if -Doracle.ess.initialProcessorState=stopped is present. If it is, remove -Doracle.ess.initialProcessorState=stopped and click Apply. If it is not present, there is no action to take.

4. Restart ess_server1.

Problem

As an intermittent issue, there could be WLS socket exceptions during server startup, or during any other upgrade tasks. An example of the exception is:

bea_wls_management_internal2/Bootstrap, user: FUSION_APPS_PROV_PATCH_APPID 
java.net.SocketTimeoutException: Read timed out 
at jrockit.net.SocketNativeIO.readBytesPinned(Native Method) 
at jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32) 

Solution

Find the managed server or the administration server that encounters the failure, and manually restart the server. Proceed with the upgrade by resuming Upgrade Orchestrator on the failed host.

Problem

After the upgrade, the following error displays after logging in to the WLS Console of CommonDomain. and navigate to Deployments:

soa-infra application is in WARNING state.

Solution

Ignore this error as there is no functional impact for SOA users due to this error.

Problem

The soa-infra app is in a warning state in all domains and errors are reported related to "jms/bpm/CaseEventQueue".

Solution

This error can be ignored.

Problem

The custom domains are not stopped or started by FAStartStop and there errors are reported.

Solution

FAStartStop does not recognize custom domains. Custom domains must be started and stopped manually, as required, before resuming orchestration.

The following log files are generated by the deployment of SOA composites:

• Client side log files where individual domain logs reside: APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/soalogs

• Log files for the failed domain:

  • APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/soa_server1.log

  • APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/soa_server1.out

  • APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/soa_server1-diagnostic.log

  • APPLICATIONS_CONFIG/domains/hostname/domain name/servers/server name/logs/AdminServer.log

Problem

Normally, a failed SOA composite is undeployed by RUP Installer. However, if the failure of the deployment is due to an issue such as SOA servers running out of memory, then RUP Installer does not recover until orchestration is resumed.

The following are examples of error messages related to SOA Composite failures:

COMMONLOG-00049: SOA composite "composite_name" patch failed for server "server_name".
Recovery process also failed with an unknown reason. If the SOA composite 
patch exists on the server, it will be automatically undeployed during retry
or a checkpoint run. Also if the base composite is not the default composite,
it will be automatically set as default.

COMMONLOG-00050: SOA composite "composite_name" patch failed for server "server_name".
Recovery process also failed, and the composite patch is not undeployed.
The patch will be automatically undeployed during retry or a checkpoint run.

COMMONLOG-00051: SOA composite "composite_name" patch failed for server "server_name".
Recovery process also failed, and the base composite is not set as the
default composite. The base composite will be automatically set as default
during retry or a checkpoint run.

The following is an example of report exceptions:

COMMONEX-00026: SOA composite "composite_name" patch failed for server "server_name".
Recovery process also failed. Recovery will be done automatically during retry or a checkpoint run.
Action : No action required.

Solution

When patching existing SOA composites, RUP Installer automatically recovers any partially deployed SOA composites after failure when Upgrade Orchestrator is restarted. The following actions can be performed by Upgrade Orchestrator:

• Undeploy the partially deployed SOA composite revision if it is still present.

• Set as default the same SOA composite revision that was default before the patching was attempted, if it is not already set as default.

If the failure was caused by an environment issue, such as running out of memory, resolve the cause of the failure before restarting orchestration.

Problem

The following error is reported during SOA Composite deployment on a Solaris platform:

CFGEX-00079 : Wsm-pm application is not running in domain "domain name"

It has been confirmed that the wsm-pm application is running on this domain.

Solution

To resolve this issue, perform the following steps:

1. Log in to the failed domain and check the health of all managed servers and deployed applications.

2. Bounce all managed servers of the failed domains.

3. Exit Upgrade Orchestrator.

4. Restart Upgrade Orchestrator.

If a customized SOA composite deployment fails during the upgrade, manually deploy this composite using WLST commands.

To apply a SOA composite manually after a deployment failure, perform the following steps:

The following example composite, FinAp, is patched from revision 1.0 to revision 2.0 and the SAR file of revision 2.0 is in FA_ORACLE_HOME/crm/deploy/sca_FinAp_rev2.0.jar. The parameters are for illustration purposes only.

1. Refer to the Diagnostics report to find the name and location of the sca_*.jar file that was copied to FA_ORACLE_HOME by Oracle Fusion Applications Patch Manager. For more information, see Diagnostics Report in the Oracle Fusion Applications Patching Guide.

2. If the previous revision contained JDeveloper customizations, ensure that the patched revision is deployed with the merged JDeveloper customizations. Using the sca_*.jar file from Step 1, follow the JDeveloper customization merge instructions that are described in Merge SOA Composite JDeveloper Customizations During SOA Preverification. Then, use the merged sca_*.jar for Step 3.

3. Deploy the patched composite using the single patch composite command as shown in the following example:

   sca_patchComposite('SOA-Infra URL', user, password, 
   '/FA_ORACLE_HOME/crm/deploy//sca_FinAprev2.0.jar', mergeLogFile='/tmp/merge-log.txt')
   

Run the UpdateSOAMDS SOA composite on every domain if any flexfield changes were made by following the steps described in Synchronizing Customized Flexfields in the MDS Repository for SOA in the Oracle Fusion Applications Extensibility Guide for Developers.

If JDeveloper customizations were performed to a SOA composite and the composite was deployed to the SOA runtime, RUP Installer reports an error during SOA Preverification, which asks to take the newer version of the composite that is in the release. Then, merge the customizations by performing the following steps:

1. If any customizations are detected, the SOA Preverification results display the SOA composite name, its location in the FA_ORACLE_HOME/stripe/deploy directory, and the requirement for you to merge JDeveloper customizations into the sca_*.jar file in FA_ORACLE_HOME before proceeding with the upgrade. The stripe in the directory path refers to crm, hcm, fscm, and so on.

2. Open the custom SOA workshops and the customized version of the Fusion Applications SOA composite in JDeveloper using "Oracle Fusion Applications Developer".

3. Import the composite sca_*.jar file from FA_ORACLE_HOME/stripe/deploy into the project, for example revision 11.12.x.0.0, in JDeveloper. Make note of this revision number in the deployment window because you will need it in Step 8.

4. Restart JDeveloper in the Oracle Fusion Applications Administrator Customization role.

5. Verify that there are no errors in JDeveloper.

6. Verify that the changes introduced in both the customized version and the patched version are present.

7. Right-click the composite project in the Application Navigator, select Deploy, select the composite, click Deploy to SAR, and click Next.

8. Manually change the value in New Revision ID to the revision from Step 3, for example, 11.12.x.0.0, and click Finish.

9. If the deployment folder is set to a location different from that of the FA_ORACLE_HOME/stripe/deploy directory, copy and replace the JAR in the location mentioned in the error message of this SOA Composite. If your file name is different, rename it to the original name. You must copy the jar in the correct format to FA_ORACLE_HOME/stripe/deploy. For example if you have sca_ContractsDeliverablePurchaseDocAttrReadComposite_rev11.12.x.0.0.jar in JDeveloper, then you must copy it back to the FA_ORACLE_HOME/stripe/deploy directory as sca_ContractsDeliverablePurchaseDocAttrReadComposite.jar.

10. To proceed with the installation, use the same command you used to start Upgrade Orchestrator.

Review the APPLICATIONS_CONFIG/lcm/rupliteovm/output/logs/ruplite.log file to confirm there are no errors. Alternatively, check rehydration framework logs under /assemblybuilder/logsor /var/log for any errors.

Review the following troubleshooting information for specific plug-ins:

• ValidateEnvironment: This plug-in runs during every mode of RUP Lite for OVM. If this plug-in fails, RUP Lite for OVM stops. Resolve any errors reported in the log file and then run RUP Lite for OVM again.

• SetupCredentials: This plug-in runs during every mode of RUP Lite for OVM. If this plug-in fails, RUP Lite for OVM stops. Typical causes of failure are an incorrect key for an existing wallet, or specifying a key for a new wallet that does not meet Oracle's minimum standards. Resolve any errors reported in the log file and then run RUP Lite for OVM again.

  This plug-in prompts for all secure properties that are required by offline RUP Lite plug-ins. It stores these properties in a wallet file, which are encrypted using a key provided by the user. If a wallet already exists, the user provided key must be valid for the existing wallet. If the wallet does not exist, a new wallet is created using the provided key. If a secure property already exists in the wallet, then it is not prompted for again.

  This plug-in prompts only for secure properties that are needed by other plug-ins that will execute. If a plug-in does not execute on the current node or is disabled, then its properties are not be requested. If no plug-ins that will execute require secure properties, the wallet creation or access is skipped and the user is not be prompted for the wallet key.

• RequireRoot: This plug-in sets the require root flag to true so that RUP Lite for OVM checks that root user is used for the pre-root mode. If this plug-in fails, RUP Lite for OVM stops.

• AutoCorrectEtcHosts: This plug-in updates the /etc/hosts file on each non-IDM node to include entries for the VMs logical host and internal logical host (if applicable), for all VMs that were deployed as part of the current upgrade.

• RemoveRunAsRoot: This plug-in removes the run_as_root file from ovabdir/deployfw/bin/ file path and is rerunnable. Verify this plug-in was successful by confirming that the run_as_root file no longer exists under the ovabdir/deployfw/bin directory. The ovabdir is /u01/ovmext for IDM nodes and for all non-IDM nodes it is /u01/APPLTOP/ovabext.

• ModifyOutputOwner: This plug-in modifies RUP Lite files to be owned by the application user instead of root. To verify that this plug-in was successful, check the owner of the /u01/APPLTOP/instance/lcm/ruplitovm/output directory. The owner should be an apps user and not the root user.

• GenerateOptimizedQueryPlans: This plug-in is re-runnable. Verify this plug-in was successful by connecting to the database as fusion_mds and running the following command:

  SELECT TO_CHAR(last_analyzed, 'yyyy/mm/dd hh:mi:ss am') as last_analyzed FROM user_tables;
  

  The results should show that the tables were just analyzed.

• DeployECSF: This plug-in is re-runnable. If the environment was originally provisioned before Release 5, verify that this plug-in was successful by confirming that the help object, schedule and group being deployed are reported in the log file. Alternatively, use Fusion Applications Control to connect to the Administration Server that hosts the search application and confirm that the Help instance artifacts are deployed.

• IDMDecouplingCleanupEtcHosts: This plug-in only applies to upgrades where IDM decoupling is enabled. The plug-in will remove the following entries from /etc/hosts on non-IDM hosts:

  • pstore.oracleoutsourcing.com

  • oim-admin.oracleoutsourcing.com

  • oim.oracleoutsourcing.com

  • oim-server.oracleoutsourcing.com

  • ids-db*.oracleoutsourcing.com

Problem

RUP Lite for OVM runs for a long time during domain configuration.

Solution

To resolve this issue, perform the following steps:

1. Ensure that the IDM host is accessible and responding.

2. Ensure that the database is accessible and responding.

3. If either the IDM host or the database is not responding, update the status of the orchestrator task that runs RUP Lite for OVM to "Error", using the following command:

   cd ORCH_LOCATION/bin
   ./orchestration.sh updateStatus -pod POD_NAME -hosttype host_type -hostname host_name -release release_number -phase phase_name -taskid plugin_name -taskstatus Error
   

   Fix the issue with the IDM host or the database and resume Upgrade Orchestrator.

If none of the above steps solve the problem, contact Oracle Support with detailed log information.

Problem

oracle.security.jps.service.credstore.CredStoreException: JPS-01055: Could not create credential store instance. Reason 
oracle.security.jps.service.policystore.PolicyStoreException: JPS-10702: The datasource jdbc/OPSSDBDS is not found.[[

Solution

To fix this issue, perform the following steps:

1. Log in to the SCM Domain Admin WebLogic Console as an administrator:

   http://host:7801/console
   

2. Go to Domain Configurations, and select Data Sources under the Services section.

3. Select opss-DBDS from the list, and then select the Targets tab.

4. Ensure that the ”All servers in the cluster” option is checked for the following SCM clusters. If not, select Lock to edit and check both boxes:

   • OrchestrationInfrastructureCluster

   • ConfiguratorCluster

   • SupplyOrchestrationCluster

   • PlanningCluster

   • ManufacturingCluster

5. Repeat Step 4 for the following data sources (if they exist):

   • opss-DBDS-rac1

   • opss-DBDS-rac2

6. Click Activate Changes for the changes to take effect.

This solution will take effect and there is no need to bounce the servers or environment.

Problem

[2017-01-24T07:20:17.739+00:00] [IdstoreConnectivityCheck] [ERROR] [] 
[IdstoreConnectivityCheck] [tid: 92] [ecid: 
0000LbFM8W07y0I_IpP5if1OXjyg000009,0] HC-COMMON-00001 : Unable to perform the 
check:[[ 
oracle.security.jps.service.idstore.IdentityStoreException: JPS-01520: Cannot 
initialize identity store, cause: oracle.security.idm.ConfigurationException: 
Failed to connect to directory. Check configuration information.. 
oracle.security.idm.ConfigurationException: Failed to connect to directory. 
Check configuration information. 
 Review log files for additional details to take an appropriate corrective 
action ]]

Solution

Problem

Perl lib version (v5.8.3) does not match the executable version (v5.8.8)

Solution

Run the following commands:

export PERL_HOME=/u01/APPLTOP/dbclient/perl
export PATH=/u01/APPLTOP/dbclient/perl/bin:$PATH

Then, retry the setenv command.

Problem

Health Checker (HC) FileOwnerAndPermissionsCheck might report the following error:

[Error]: Plugin 'FileOwnerAndPermissionsCheck': HC-PERM-0003: Failed to verify the permission of files/folders in /u01/APPLTOP/fusionapps/odi/.cas/CLI/inventory using find command. Review log files for additional details to take an appropriate corrective action (verifying File Ownership and Permissions)

Solution

Ignore the error and proceed.

Note that the /u01/APPLTOP/fusionapps/odi/.cas/CLI/inventory error is only an example.  If any similar error related to ".cas" is reported, ignore the error and proceed.

Problem

When running Health Checker, the PatchSessionsAndProcessesCheck check fails with the following error:

HC-PATCHSP-00001: Patch sessions or processes found in your environment:HC-PATCHSP-00017 : Check #7: Running patch processes found in this environment.

For any details needed for the running processes, review the log files .

Solution

This issue can occur even when no active session is found. To resolve this issue, perform the following steps:

1. Open the Health Checker log file to search for the running process that was reported. The running process typically contains strings such as adpatch, adadmin, adworker, adctrl, oracle.apps.ad.worker.AdJavaWorker, or oracle.apps.ad.fapmgr.FAPManager.

2. To see if there are active sessions, run the following command:

   ./fapmgr.sh report -patchprogress
   

3. If ./fapmgr.sh returns no rows from the previous step, find the origin of the session(s) identified in Step 1. Then, decide whether this session needs to be terminated or allowed to finish before rerunning Health Checker.

4. Terminate or allow the process(es) to finish and then rerun Health Checker.

   If Health Checker detects that an active session was terminated or has completed, running Health Checker again succeeds.

Problem

oracle.jbo.client.CADatabaseConnectionProvider.loadConnectionProperties(CAData
baseConnectionProvider.java:154) at oracle.jbo.client.Configuration. initializeFromConnectionName(Configuration.java:1225) at oracle.jbo.client. Configuration.getConfiguration(Configuration.java:649) at oracle.jbo.common. ampool.PoolMgr.loadConfiguration(PoolMgr.java:759) at oracle.jbo.common. ampool.PoolMgr.findPool(PoolMgr.java:589) at oracle.jbo.client.Configuration. createRootApplicationModuleHandle(Configuration.java:1589) at oracle.jbo.client. Configuration.createRootApplicationModuleHandle(Configuration.java:1559)
at oracle.apps.topologyManager.model.applicationModule.TopologyManagerAMUtil.
getApplicationModuleInfo(TopologyManagerAMUtil.java:122) at oracle.apps. topologyManager.model.applicationModule.TopologyManagerAMImpl.getApplicationModuleInfo(TopologyManagerAMImpl.java:73) at
oracle.topologyManager.client.deployedInfo.DeployedInfoProvider.getDeployedDomainsByEnvironment(DeployedInfoProvider.java:880) at oracle.check.common.util. TMUtils.getDeployedDomains(Unknown Source)at oracle.check.common.util. MWUtils.getDomainDetailsFromTM(Unknown Source) at oracle.check.common.util. MWUtils.getDomainDetails(Unknown Source) at oracle.check.apps. OPMNManagedProcessesStatusCheck.verifyManagedProcessesAreUp(Unknown Source)
at oracle.check.apps.OPMNManagedProcessesStatusCheck.execute(Unknown
Source)at oracle.check.common.AbstractCheckPlugin.plugin_execute(Unknown
Source)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at oracle.healthcheckplug.core.PluginProxy.invoke(Unknown Source)
        at com.sun.proxy.$Proxy5.plugin_execute(Unknown Source)
        at oracle.healthcheckplug.core.PluginCallable.call(Unknown Source)
        at oracle.healthcheckplug.core.PluginCallable.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Plugin succeeded: Verifying OPMN managed processes are up

Solution

Ignore the warning and proceed.

Problem

The PostLangPackChecks plug-in fails for all Health Checks.

Solution

If the SKIP_UPGRADE_FOR_LANGUAGE property was enabled with one or more language codes, the Post Language Pack Health Checks are expected to fail for the skipped languages. These failures can be ignored because Post Language Patch Health Checks are run manually after the languages are upgraded manually.

Problem

The following error is reported when running RUP Lite for RDBMS:

[SEVERE] Fatal Error, Traceback (most recent call last):
 
File "/SHARED_LOCATION/work_dir/DB_2014-05-27_10-22-31/db_server_bundle/ruplite/main.py", line 280, in _executeplugin 
 result = _runpluginmodule(plugin_module) 

File "/SHARED_LOCATION/work_dir/DB_2014-05-27_10-22-31/db_server_bundle/ruplite/main.py", line 191, in _runpluginmodule 
    errinfo = eval("plugin_module.plugin_execute()") 
  File "<string>", line 1, in <module> 

File "/SHARED_LOCATION/work_dir/DB_2014-05-27_10-22-31/db_server_bundle/db/plugin/PostActions.py", line 95, in plugin_execute 
    raise Exception('Failed to perform post DB restart actions.') 
Exception: Failed to perform post DB restart actions. 
 
[main] [SEVERE] Failed execution of plugin: db.plugin.PostActions
 
 [main] [SEVERE] Fatal error, exiting 
 [main] [SEVERE] Summary of plugins: 
 [main] [SEVERE]  Succeeded: db.plugin.ValidateEnv 
 [main] [SEVERE]  Skipped: db.plugin.PreActions 
 [main] [SEVERE]  Skipped: db.plugin.ApplyDBPatches 
 [main] [SEVERE]  Failed with fatal: 
db.plugin.PostActions, Exception: Failed to perform post DB restart actions. 
[main] [SEVERE] RUPLite Installer for DB Stopped 

Solution

RUP Lite for RDBMS failed while connecting to the database, which indicates an invalid value in the work_dir/DB_timestamp/db_server_bundle/metadata/env.properties file. If there is an extra "/" character for the ORACLE_HOME property, in this file, remove it. This ORACLE_HOME value must exactly match the database ORACLE_HOME and it should not have an additional "/" at the end.

Note that running RUP Lite for RDBMS in "validate" or "setdbparameter" mode runs successfully even if there as an additional "/" in the ORACLE_HOME property.

Problem

An error occurred during the Bootstrapping Patch Manager configuration assistant.

Solution

An error during Bootstrapping Patch Manager normally occurs only when the database is down. Ensure that the database is up and running. Review the related log files in the following location:

APPLICATIONS_CONFIG/lcm/logs/11.12.x.0.0/RUP/FAPatchManager_bootstrap_timestamp.log

This section contains information about troubleshooting issues that may occur during the Propagating Domain Configuration configuration assistant. The following topics are discussed:

• Propagating Domain Configuration Assistant Takes Too Long to Complete

• Confirm the Propagating Domain Configuration Assistant Was Successful

• WARs or EARs Are Not Accessible From The Primordial Host

Problem

The upgrade fails when Oracle Fusion Applications is running on a non-Oracle VM configuration and is using an Oracle VM template.

Solution

This configuration is not supported. To resolve this, check if a directory named /assemblybuilder exists in the environment. If this directory is present and this is not an Oracle VM environment, rename the directory to any other name. Then, resume orchestration.

Problem

RUP Lite for Domain Configuration takes too long to complete.

Solution

This utility can take some time to complete as time taken to propagate domain configuration is highly dependent on the environment, specifically the number of non-admin domains and the responsiveness of the file system. Note this issue is seen only in local domain environments where the utility is run between RUP Installer Part 1 and Part 2. This is not an issue for Oracle VM environments or other environments with shared domains.

Problem

The Extending Certificate Validation fails with exception reporting if there is Incentive Compensation, Enterprise Contracts, and Oracle Fusion Accounting Hub offerings on the environment:

APPLICATIONS_CONFIG/domains/CommonDomain_host/CommonDomain /config/fmwconfig/owc_discussions.jks (No such file or directory).

Solution

If the missing file cannot be found in APPLICATIONS_CONFIG/domains/CommonDomain_host/CommonDomain/config/fmwconfig, perform the following steps:

1. Copy default_keystore.jks to owc_discussions.jks in APPLICATIONS_CONFIG/domains/CommonDomain_ host/CommonDomain/config/fmwconfig.

2. Resume orchestration.

After the Release 8 upgrade step called "Deploying Data Security Grants", the fapatch_Deploying_Data_Security_Grants_timestamp.log file contains entries as shown in the following example:

Number of records processsed : 8372
Number of records updated (grantee_key or compile_flag) : 3934
Number of records where GUIDs matched and no reconciliation done : 4366
Number of records in database missing necessary meta data : 2
Number of records in database that could not be reconciled with LDAP : 70 

These messages may start with either "WARNING" or "SEVERE". The severe errors may be associated with exceptions as shown in the following examples:

SEVERE: Policy Store Exception raised in getApplicationPolicyoracle.security.jps.service.policystore.
PolicyObjectNotFoundException: JPS-04028: Application with name "cn=ADRGroups,cn=Groups" does not exist.

SEVERE: RuntimeException raised. Incorrect entry found in db for application role PJT_PROJECT_WORK_PLAN_MANAGEMENT_DUTY.
May require reconciliation with target LDAP Processing row with grant_guid:
F9C89E5D04C2322629EBE642337695FC. ROLE_NAME is
PJT_PROJECT_WORK_PLAN_MANAGEMENT_DUTY ROLE_NAME_SPACE is
cn=ADRGroups,cn=Groups. PJT_PROJECT_WORK_PLAN_MANAGEMENT_DUTY GUID in
database is 61065B6FEA8E3824B74476B1A315FDE4 Runtime Exception is
oracle.jbo.JboException: JBO-29114 ADFContext is not setup to process
messages for this exception. Use the exception stack trace and error code to
investigate the root cause of this exception. Root cause error code is
JBO-29000. Error message parameters are
{0=oracle.security.jps.service.policystore.PolicyObjectNotFoundException,
1=JPS-04028: Application with name "cn=ADRGroups,cn=Groups" does not exist.}

These warnings and errors have no impact on functionality and can be ignored.

Problem

Errors related to missing approles may be reported during the Applying Online BI Metadata and Configuration Updates configuration assistant. These errors are reported in bi_webcat_patch.log, and can be ignored, as they have no impact on the upgrade.

Solution

If Upgrade Orchestrator stops due to this error, resume the upgrade.

The following ignorable errors may be encountered while running the catbundle.sql script or its rollback script:

ORA-29809: cannot drop an operator with dependent objects

ORA-29931: specified association does not exist

ORA-29830: operator does not exist

ORA-00942: table or view does not exist

ORA-00955: name is already used by an existing object

ORA-01430: column being added already exists in table

ORA-01432: public synonym to be dropped does not exist

ORA-01434: private synonym to be dropped does not exist

ORA-01435: user does not exist

ORA-01917: user or role 'XDB' does not exist

ORA-01920: user name '<user-name>' conflicts with another user or role name

ORA-01921: role name '<role name>' conflicts with another user or role name

ORA-01952: system privileges not granted to 'WKSYS'

ORA-02303: cannot drop or replace a type with type or table dependents

ORA-02443: Cannot drop constraint - nonexistent constraint

ORA-04043: object <object-name> does not exist

ORA-29832: cannot drop or replace an indextype with dependent indexes

ORA-29844: duplicate operator name specified

ORA-14452: attempt to create, alter or drop an index on temporary table already in use

ORA-06512: at line <line number>

Note that if this error follow any of above errors, then can be safely ignored.

ORA-01927: cannot REVOKE privileges you did not grant