A Troubleshooting the Upgrade

This appendix describes some common procedures for troubleshooting a failed upgrade, domain reconfiguration or server start issues.

• Reviewing the Release Notes
  
• Resolving Server Start Errors
  If the administration or managed servers do not start after the upgrade, you may need to re-apply any customizations added to startup scripts, files and classes.
• Recovering From a Failed Upgrade
  
• Error while Copying User Messaging Service (UMS) Configuration Files
  If the Upgrade Assistant fails to automatically copy the UMS configuration files, you must stop the upgrade and manually copy the configuration files before attempting to upgrade UMS. This process is required only if the Upgrade Assistant fails to automatically copy the configuration files or if you prefer to copy the configuration files manually.
• OWSM Data Source Connection Failure During Upgrade from 12c (12.1.3 or 12.2.1.0) to 12c (12.2.1.4.0)
  
• Troubleshooting a Failed BAM Upgrade
  
• Reapplying an EDNTopic to SOA JMS Module After Upgrade
  
• Troubleshooting Oracle Service Bus
  
• Troubleshooting Oracle Managed File Transfer (MFT) Upgrade Issues
  If you encounter an upgrade error while upgrading Oracle Managed File Transfer, refer to these troubleshooting tasks to correct the issue.
• Error Starting OWSM After Upgrading to 12c
  
• Encryption Issues During Upgrade
  
• Upgrading Unsupported Domains with the Upgrade Assistant
  
• Business Rules Audit Trail Not Showing After Instance Upgrade
  
• Resolving a Coherence Cache Exception
  
• WSDL Generated Missing Elements for Custom Exception
  
• Failure to Connect to the ServerSocket through Remote Clients
  
• Troubleshooting Invalid Objects in Schema Registry
  Schemas with a post-upgrade status of INVALID may indicate a failed upgrade, but not in all situations.
• Wires Missing After Migrating SOA Composite
  Wires that connect services and references may be missing from composite after an upgrade from 11g. You must apply a patch to correct this issue.

Reviewing the Release Notes

Make sure that you review the release notes to determine if any known issues could be impacting your upgrade. You can find the release notes in the Oracle Fusion Middleware 12c (12.2.1.4.0) library.

Resolving Server Start Errors

If the administration or managed servers do not start after the upgrade, you may need to re-apply any customizations added to startup scripts, files and classes.

If servers do not start, or they start in AdminMode, the cause is most likely that changes to startup scripts or domain variables from the previous environment were not reapplied to the newly configured 12c domain. During the upgrade process, startup scripts are replaced with the latest version. If you made any modifications to these files, then you will need to edit the new startup scripts with the same information.

To determine if this is the cause, compare the pre-upgrade startup scripts or files from your backups to the new 12c scripts and files. If there are differences, then update files as described in the following procedures.

• Reapplying Customizations to setDomainEnv.sh
  
• Reapplying Start Script Properties for JVM
  
• Reapplying Customizations to XEngine Configuration Files
  
• Copying Custom XPath Classes
  

Reapplying Customizations to setDomainEnv.sh

If servers do not start, or they start in AdminMode, the cause is most likely that the setDomainEnv.sh changes from the previous environment were not reapplied to the newly configured 12c domain. During the upgrade process, startup scripts are replaced with the latest version. If you made any modifications to these files, then you will need to edit the new startup scripts with the same information.

To determine if this is the cause, compare the setDomainEnv file from your pre-upgrade backup to the new 12c setDomainEnv file. If there are differences, then make the same changes in the new setDomainEnv file.

Reapplying Start Script Properties for JVM

If you used a start script to specify required startup properties, or to perform any other work required at start up in your 11g environment, then you will need to reapply the properties post-upgrade.

Specifically, if you have configured JRockit JVM arguments in your 11g environment, then these configurations must be reapplied post-upgrade. Oracle recommends that you use either startup-plan.xml or startscript.xml for configuring JVM startup parameters.

Caution:

Failure to update the start script arguments may prevent you from starting the SOA and OSB servers after the upgrade.

To enable the scripts:

1. In the nodemanager.properties file, set the StartScriptEnabled property to true. (The default is false.) If your start script is named startWebLogic.sh or startWebLogic.cmd, Node Manager uses one of those scripts as the default.

2. If you want to specify a custom start script, set the StartScriptName property to the name of your script in the nodemanager.properties file.

Node Manager sets the JAVA_VENDOR, JAVA_HOME, JAVA_OPTIONS, SECURITY_POLICY, CLASSPATH, and ADMIN_URL. It retrieves these values from the ServerMBean, ServerStartMBean, and SSLMBean when you use the Administration Console to start the server, or WLST connected to the Administration Server. When you use WLST connected directly to the Node Manager, you can specify the values; otherwise, they are left empty.

Node Manager combines all of the command line startup options (-D flags) that are specified in the ServerStartMBean Arguments attribute, as well as the SSLArguments into a single environmental variable called JAVA_OPTIONS. SSLArguments are retrieved from the values in the SSLMBean. The SSLMBean is inspected for ignoreHostnameVerification, HostnameVerifier, and ReverseDNSAllowed values, then those values are appended to the -D flags. All of those flags comprise the SSLArguments parameter. All of the values for SSLArguments as well as Arguments in the ServerStartMBean comprise the JAVA_OPTIONS environment variable that is defined for the start script. In addition, the script will append any of its own defined values onto this environment variable.

Reapplying Customizations to XEngine Configuration Files

Any pre-upgrade changes made to the XEngine configuration files, such as SeverityConfig.xml, will be overwritten by new, regenerated configuration files during the domain reconfiguration process. Therefore, all customized settings used in the pre-upgrade configuration files will need to be reapplied after the upgrade.

For example, if you added a section for SNIP in the pre-upgrade XEngine configuration file, SeverityConfig.xml, the same section will have to be added to the new, post-upgrade SeverityConfig.xml file.

Copying Custom XPath Classes

If you modified the default XPath classes in your pre-upgrade environment, then after the upgrade you will need to copy the customized XPath classes to the new 12c Oracle home as shown in the example below:

Copy the custom XPath classes from your pre-upgrade backups. Classes are found in the following directory:

/11g_ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1/classes

to the following 12c directory:

/12c_ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1/classes

Recovering From a Failed Upgrade

Recovering from a failed upgrade depends on when the error(s) occurred. Review the following to determine how to recover:

• If there are errors while running the Upgrade Assistant to upgrade _SOAINFRA schema, you must fix the errors in the schema and rerun batch jobs.

  Note that this recovery method only applies when you are running the Upgrade Assistant for the first time and you selected the Schema option.

• If there are errors while running the Reconfiguration Wizard, you must restore from source environment and restart the upgrade from the beginning.

• If there are errors while running the Upgrade Assistant to upgrade WebLogic Component Configurations option, then you can fix the errors and rerun the Upgrade Assistant. The second time you run the Upgrade Assistant there is no need to restore from backup and restart the upgrade process from the beginning. This process is reentrant.

• If there are errors while running the Upgrade Assistant to upgrade schemas, and the error occurs during the upgrade phase, you will have to restore from backup, correct the issues, and then restart the upgrade from the beginning. If the error occurs during the examine phase, however, you can correct the issues and restart the Upgrade Assistant. Errors that occur prior to the upgrade phase are reentrant.

For more information on troubleshooting your upgrade, see "General Troubleshooting Guidelines" in the Upgrading with the Upgrade Assistant.

Note:

If you received the CFGFWK-60950 error, rename the BAM templates as described in "Renaming the Oracle BAM Templates Before Upgrading the 11g Schemas" and launch the Reconfiguration Wizard again.

If you received this error, you will need restore your entire pre-upgrade environment, perform the necessary pre-upgrade tasks and then perform the steps in the section listed above before you can attempt the reconfiguration process again.

For more information on resolving BAM-specific issues, see Recovering from a Failed Oracle BAM Upgrade.

Error while Copying User Messaging Service (UMS) Configuration Files

If the Upgrade Assistant fails to automatically copy the UMS configuration files, you must stop the upgrade and manually copy the configuration files before attempting to upgrade UMS. This process is required only if the Upgrade Assistant fails to automatically copy the configuration files or if you prefer to copy the configuration files manually.

This section describes the location of the UMS configuration files that are copied from the remote managed server nodes to the Admin server while upgrading UMS from 11g to 12c. Note that the Upgrade Assistant can automatically copy the remote configuration files, if all necessary prerequisites are met and the required login information is provided. For more information about using Upgrade Assistant to copy configuration files, see Identifying Configurations that can be Upgraded with the Upgrade Assistant in Upgrading with the Upgrade Assistant.

However, if the Upgrade Assistant cannot locate your files, then you must copy the configuration files from the remote managed server to the same location on the Administration server running the upgrade. The configuration files that must be copied include the UMS server configuration files (appconfig.xml), driver configuration files (driverconfig.xml), and the user preferences files (businessterms.xml). These files are located in the /applications folder for each managed server, as shown in Table A-1.

After manually copying the configuration files from the managed server to the Administration server, you must start the Upgrade Assistant again.

Table A-1 Configuration File locations

Configuration file	Location
UMS Server (appconfig.xml)	DOMAIN_HOME/config/fmwconfig/servers/MANAGED_SERVER_NAME/applications/usermessagingserver/configuration/appconfig.xml
Driver Configuration (driverconfig.xml)	DOMAIN_HOME/config/fmwconfig/servers/MANAGED_SERVER_NAME/applications/usermessagingdriver-DRIVER_NAME/configuration/driverconfig.xml
User Preferences (businessterms.xml)	DOMAIN_HOME/config/fmwconfig/servers/MANAGED_SERVER_NAME/applications/usermessagingserver/configuration/businessterms.xml

Note:

If there are multiple drivers deployed in a domain, then you must ensure that configuration files for all drivers are copied. This can be achieved by replacing the DRIVER_NAME with as many drivers deployed in that domain.

OWSM Data Source Connection Failure During Upgrade from 12c (12.1.3 or 12.2.1.0) to 12c (12.2.1.4.0)

When you select All configurations used by the domain when running the Upgrade Assistant, the upgrade can fail at the examination phase with the WSMERROR-00015 error. This error can occur when the pre-upgrade domain is created with Multi-DataSource connection.

Error Message:

[2015-09-22T10:46:54.552-07:00] [WSM] [INCIDENT_ERROR] [upgrade.WSM.WSMPLUGIN]oracle.ias.update.exception.UpgradeException: WSMERROR-00015: Failed to read the Oracle WSM datasource connection details.at oracle.wsm.lifecycle.upgrade.impl.WSMUpgradePlugin.initializePluginData(WSMUpgradePlugin.java:396)

When upgrading to 12c (12.2.1.4.0), the Upgrade Assistant expects a generic datasource connection. Since this error is detected during the Examination phase, you can go back and correct the issue and continue with the upgrade without restoring from backup.

To complete the upgrade, complete the following steps:

1. Navigate back to the Datasource screen in the Upgrade Assistant.

2. Change the "mds-owsm" data source to be a generic data source.

3. Restart the Upgrade Assistant and, when prompted, select All configurations used by the domain.

4. After a successful upgrade, you can change the "mds-owsm" data source back to a multi-DS.

Troubleshooting a Failed BAM Upgrade

When upgrading a domain containing Oracle Business Activity Monitoring (BAM), note that there are additional BAM-specific troubleshooting procedures.

For more information, see Recovering from a Failed Oracle BAM Upgrade.

Reapplying an EDNTopic to SOA JMS Module After Upgrade

After upgrading to 12c (12.2.1.4.0), the upgraded SOA JMS module may be missing the EDNTopic. If the JMS module is missing the EDNTopic, you must manually add the topic or UDD for this topic using the Administration Console or WLST.

This is a known issue and can occur in both clustered and unclustered environments.

See the Administration Console online help for more information on reapplying the EDNTopic or contact Oracle Support.

Troubleshooting Oracle Service Bus

If you experience post-upgrade issues with Oracle Service Bus, review the troubleshooting procedures described in Troubleshooting Oracle Service Bus Upgrade.

Troubleshooting Oracle Managed File Transfer (MFT) Upgrade Issues

If you encounter an upgrade error while upgrading Oracle Managed File Transfer, refer to these troubleshooting tasks to correct the issue.

Some common upgrade error messages for Managed File Transfer are listed below:

SQLException: ORA-04020: deadlock detected while trying to lock object

Resolution: Make sure that you selected Managed File Transfer on the Available Components screen of the Upgrade Assistant. If you do not select Oracle Managed File Transfer, the upgrade will not include MFT schema.

Creating Backward Compatibility of SOAP Services 

If you have Managed File Transfer-specific projects created in older versions using JDev, you must correct the WSDL definition of existing SOA/SOAP projects by opening them with JDev and redeploying the composite.

This is necessary when MFT is the target for SOA composite and not when it is a source for SOA.

Error Starting OWSM After Upgrading to 12c

If there was a custom trust keystore configured in Enterprise Manager 11g prior to the upgrade, you may encounter issues with starting the OWSM.

Specifically, if after upgrading an 11g domain running OWSM to 12c, you receive the following error in the OWSM server log (after the second startup), then you must manually correct this issue:

<Error> <HTTP> <srvgdysoap01.nov.com> <wls_wsm1> <[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <26c804bb-15a7-46de-a81e-82565fcd2f28-00000004> <1418929621034> <BEA-101216> <Servlet: "PolicyManagerValidator" failed to preload on startup in Web application: "/wsm-pm".

If wsm-pm application will not start, you must perform the following steps:

1. Roll back the upgrade to 11g by restoring the backup files.

2. Complete the upgrade steps again using the Upgrade Assistant.

3. Start the OWSM server

   Note:

   It is very important to only start the OWSM server once and leave it running. If you stop and restart it then the NPE will present itself and you will have to roll back again

4. Execute the following WLST command against the running OWSM server from the <domain_home>/oracle_common/common/bin location:

   exportMetadata('wsm-pm','<wsm server>','location to write the zip')
    
   where <wsm_server> is the name of the WLS server running OWSM ('wsm_server1' for example)

5. Extract the MDS archive and go to /configuration/WLS/ and open the file there. The file name is the name of the domain.

6. Search for the property entries containing the string 'keystore.inst.0'. There are probably several of them in a row and they look like

   <orares:property ........</orares:property>
   

7. Delete these properties from the file.

8. Rebuild the archive and import it back to the running server with the command:

   importMetadata('wsm-pm','<wsm server>','location of zip')

9. Restart the servers.

Encryption Issues During Upgrade

If you received the following error message during the reconfiguration, you may need to apply additional policy files to the JDK and restart the upgrade from your backup:

JPS-06513: Failed to save keystore. Reason oracle.security.jps.service.keystore.KeyStoreServiceException: Failed to perform cryptographic operation

To prevent this error from reoccurring, apply the policy files before the subsequent upgrade as described in:

• Updating Policy Files when Using Enhanced Encryption (AES 256)
  If you plan to use enhanced encryption, such as Advanced Encryption Standard (AES 256), in your upgraded environment, Oracle recommends that you apply the latest required policy files to the JDK before you upgrade.

Updating Policy Files when Using Enhanced Encryption (AES 256)

If you plan to use enhanced encryption, such as Advanced Encryption Standard (AES 256), in your upgraded environment, Oracle recommends that you apply the latest required policy files to the JDK before you upgrade.

The Java platform defines a set of APIs spanning major security areas, including cryptography, public key infrastructure, authentication, secure communication, and access control. These APIs allow developers to easily integrate security mechanisms into their application code.

Some of the security algorithms used in Fusion Middleware 12c (12.2.1.4.0) require additional policy files for the JDK. See Java Cryptography Architecture Oracle Providers Documentation.

Note:

If you attempt to use enhanced encryption without applying these policy files to the JDK before you begin the upgrade, the upgrade can fail and you must restore the entire pre-upgrade environment and start the upgrade from the beginning.

Upgrading Unsupported Domains with the Upgrade Assistant

If you receive an error from the Upgrade Assistant stating that the specified domain cannot be upgraded, then your domain configurations are not supported for this release. For more information on supported configurations and domain restrictions, see Understanding SOA Domain Upgrade Restrictions.

Do not attempt to upgrade or schemas or domain configurations in an unsupported domain.

Business Rules Audit Trail Not Showing After Instance Upgrade

The audit trail for upgraded 11g instances of the Decision Service Component will not be available post-upgrade. The audit trail for new 12c instances will continue to display.

Resolving a Coherence Cache Exception

If you see the following WebLogic Cache Provider Coherence exception then it is likely that you are not following an enterprise deployment topology recommendation to specify a specific ListenAddress.

When you see this exception, you must set the ListenAddress for your managed server as shown below:

Exception:

weblogic.cacheprovider.coherence.CoherenceException:
  at weblogic.cacheprovider.coherence.CoherenceClusterManager.ensureWKAAddresses(CoherenceClusterManager.java:510)
  at weblogic.cacheprovider.coherence.CoherenceClusterManager.configureClusterService(CoherenceClusterManager.java:236)
  at weblogic.cacheprovider.CacheProviderServerService.bootCoherenceFromWLSCluster(CacheProviderServerService.java:225)
  at weblogic.cacheprovider.CacheProviderServerService.initCoherence(CacheProviderServerService.java:94)

Resolution:

1. Log in to the WebLogic Server Console.

2. Navigate to Servers.

3. Locate the Managed Servers (SOA or OSB, for example).

4. Modify the Listen Address from localhost to 127.0.0.1 or provide the actual machine name.

WSDL Generated Missing Elements for Custom Exception

If your EJBs contain custom exceptions, and you export the Web Service Description Language (WSDL) file from your EJB business service, the generated WSDL file will not have the custom exception properties in it. You will need to manually edit the WSDL file to include these custom exception properties after the upgrade.

The issue is limited only to the WSDL generation part of the file. During runtime, the custom exception thrown from the EJB will be mapped to the respective elements in the SOAP fault. The response payload will have the elements populated corresponding to the properties of the custom exception.

Failure to Connect to the ServerSocket through Remote Clients

There is a change in behavior in which the ServerSocket is created when you upgrade from Oracle Release 11g to Release 12g. Because of this, remote clients might not able to connect to the ServerSocket when the hostname is configured as localhost. As a workaround, the localhost should be changed to hostname.

For more information, see "Configuring Oracle Socket Adapter" Understanding Technology Adapters.

Troubleshooting Invalid Objects in Schema Registry

Schemas with a post-upgrade status of INVALID may indicate a failed upgrade, but not in all situations.

If the post-upgrade schema status appears as INVALID, it may indicate that the schema update failed. You should examine the logs files to determine the reason for the failure.

EXCEPTION: Synonym objects owned by IAU_APPEND and IAU_VIEWER will appear as INVALID in the schema version registry table, but that does not indicate a failure. Synonym objects become invalid because the target object changes after the creation of the synonym. The synonyms objects will become valid when they are accessed. You can safely ignore these INVALID objects.

Wires Missing After Migrating SOA Composite

Wires that connect services and references may be missing from composite after an upgrade from 11g. You must apply a patch to correct this issue.

After you upgrade from 11g, you may notice that the wires that connect services and references may be missing from composite. This issue is caused when the 11g JDev version of the SCA project migrator is higher than the new 12c version. To fix this, you must apply a patch to modify the 11g SCA project migrator version.

To apply the patch, go to https://support.oracle.com and search for Doc ID 2356254.1.

Note:

In most cases the 11g SCA project migrator will have a lower version number than the newly installed 12c migrator and this issue will not occur.