A Troubleshooting the Upgrade

This appendix describes some common procedures for troubleshooting a failed upgrade, domain reconfiguration or server start issues.

A.1 Reviewing the Release Notes

Make sure that you review the release notes to determine if any known issues could be impacting your upgrade. You can find the release notes in the Oracle Fusion Middleware 12c (12.2.1) library.

A.2 Resolving Server Start Errors

If servers do not start, or they start in AdminMode, the cause is most likely that the setDomainEnv.sh changes from the previous environment were not reapplied to the 12c domain. Compare the setDomainEnv file from 11g to the new 12c setDomainEnv file and then add any custom changes after the upgrade.

For more information, see "Re-apply Customizations to Startup Scripts".

A.3 Recovering From a Failed Upgrade

Recovering from a failed upgrade depends on when the error(s) occurred. Review the following to determine how to recover:

  • If there are errors while running the Upgrade Assistant to upgrade _SOAINFRA schema, you must fix the errors in the schema and rerun batch jobs.

    Note that this recovery method only applies when you are running the Upgrade Assistant for the first time and you selected the Schema option.

  • If there are errors while running the Reconfiguration Wizard, you must restore from source environment and restart the upgrade from the beginning.

  • If there are errors while running the Upgrade Assistant to upgrade WebLogic Component Configurations option, then you can fix the errors and rerun the Upgrade Assistant. The second time you run the Upgrade Assistant there is no need to restore from backup and restart the upgrade process from the beginning. This process is reentrant.

  • If there are errors while running the Upgrade Assistant to upgrade schemas, and the error occurs during the upgrade phase, you will have to restore from backup, correct the issues, and then restart the upgrade from the beginning. If the error occurs during the examine phase, however, you can correct the issues and restart the Upgrade Assistant. Errors that occur prior to the upgrade phase are reentrant.

For more information on troubleshooting your upgrade, see "General Troubleshooting Guidelines" in the Upgrading with the Upgrade Assistant.

Note:

If you received the CFGFWK-60950 error, rename the BAM templates as described in "Renaming the Oracle BAM Templates Before Upgrading the 11g Schemas" and launch the Reconfiguration Wizard again.

If you received this error, you will need restore your entire pre-upgrade environment, perform the necessary pre-upgrade tasks and then perform the steps in the section listed above before you can attempt the reconfiguration process again.

For more information on resolving BAM-specific issues, see Recovering from a Failed Oracle BAM Upgrade.

A.4 OWSM Data Source Connection Failure During Upgrade (12.1.3 to 12.2.1)

When you select All configurations used by the domain when running the Upgrade Assistant, the upgrade fails at the examination phase with the WSMERROR-00015 error.

Error Message:

[2015-09-22T10:46:54.552-07:00] [WSM] [INCIDENT_ERROR] [upgrade.WSM.WSMPLUGIN]oracle.ias.update.exception.UpgradeException: WSMERROR-00015: Failed to read the Oracle WSM datasource connection details.at oracle.wsm.lifecycle.upgrade.impl.WSMUpgradePlugin.initializePluginData(WSMUpgradePlugin.java:396)

This error occurs because the 12.1.3 domain is created with Multi-DataSource connection. When upgrading to 12.2.1, the Upgrade Assistant expects a generic datasource connection.

To perform the upgrade, complete the following steps:

  1. Change the "mds-owsm" data source to be a generic data source.

  2. Restart the Upgrade Assistant and, when prompted, select All configurations used by the domain.

  3. After a successful upgrade, you can change the "mds-owsm" data source back to a multi-DS.

A.5 Troubleshooting a Failed BAM Upgrade

When upgrading a domain containing Oracle Business Activity Monitoring (BAM), note that there are additional BAM-specific troubleshooting procedures.

For more information, see Recovering from a Failed Oracle BAM Upgrade.

A.6 Reapplying an EDNTopic to SOA JMS Module After Upgrade

EDN Topic is not found when starting SOA Server post upgrade. This is a known issue and the workaround is to re-create the EDN Topic manually.

After upgrading to SOA 12.2.1, the upgraded SOA JMS module may be missing the EDNTopic. If the JMS module is missing the EDNTopic, you must manually add the topic or UDD for this topic using the Administration Console or WLST.

This is a known issue and can occur in both clustered and unclustered environments.

See the Administration Console online help for more information on reapplying the EDNTopic or contact Oracle Support.

A.7 Troubleshooting Oracle Service Bus

If you experience post-upgrade issues with Oracle Service Bus, review the troubleshooting procedures described in Troubleshooting Oracle Service Bus Upgrade.

A.8 Troubleshooting Oracle Managed File Transfer (MFT) Upgrade Issues

If you encounter an upgrade error while upgrading Oracle Managed File Transfer, refer to these troubleshooting tasks to correct the issue.

Some common upgrade error messages for Managed File Transfer are listed below:

SQLException: ORA-04020: deadlock detected while trying to lock object

Resolution: Make sure that you selected Managed File Transfer on the Available Components screen of the Upgrade Assistant. If you do not select Oracle Managed File Transfer, the upgrade will not include MFT schema.

Description of GUID-C60649BE-E01A-4571-982C-090AA493C64B-default.png follows
Description of the illustration GUID-C60649BE-E01A-4571-982C-090AA493C64B-default.png

A.9 Error Starting OWSM After Upgrading to 12c

If there was a custom trust keystore configured in Enterprise Manager 11g prior to the upgrade, you may encounter issues with starting the OWSM.

Specifically, if after upgrading an 11g domain running OWSM to 12c, you receive the following error in the OWSM server log (after the second startup), then you must manually correct this issue:

####<Dec 18, 2014 12:07:01 PM MST> <Error> <HTTP> <srvgdysoap01.nov.com> <wls_wsm1> <[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <26c804bb-15a7-46de-a81e-82565fcd2f28-00000004> <1418929621034> <BEA-101216> <Servlet: "PolicyManagerValidator" failed to preload on startup in Web application: "/wsm-pm".
 
Stack:
java.lang.NullPointerException
at oracle.wsm.config.ConfigurationProcessor.isContextValid(ConfigurationProcessor.java:297)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)

If wsm-pm application will not start, you must perform the following steps:

  1. Roll back the upgrade to 11g

  2. Run through the upgrade steps again using the Upgrade Assistant.

  3. Start the OWSM server

    Note:

    It is very important to only start the OWSM server once and leave it running. If you stop and restart it then the NPE will present itself and you will have to roll back again

  4. Execute the following WLST command against the running OWSM server from the <domain_home>/oracle_common/common/bin location:

    exportMetadata('wsm-pm','<wsm server>','location to write the zip')
 
where <wsm_server> is the name of the WLS server running OWSM ('wsm_server1' for example)

  5. Extract the MDS archive and go to /configuration/WLS/ and open the file there. The file name is the name of the domain.

  6. Search for the property entries containing the string 'keystore.inst.0'. There are probably several of them in a row and they look like

    <orares:property ........</orares:property>

  7. Delete these properties from the file.

  8. Rebuild the archive and import it back to the running server with the command:

    importMetadata('wsm-pm','<wsm server>','location of zip')

  9. Restart the servers.

A.10 Encryption Issues During Upgrade

If you received the following error message during the reconfiguration, you may need to apply additional policy files to the JDK and restart the upgrade from your backup:

JPS-06513: Failed to save keystore. Reason oracle.security.jps.service.keystore.KeyStoreServiceException: Failed to perform cryptographic operation

To prevent this error from reoccurring, apply the policy files before the subsequent upgrade using the information in Using Enhanced Encryption (AES 256).

A.11 Upgrading Unsupported Domains with the Upgrade Assistant

If you receive an error from the Upgrade Assistant stating that the specified domain cannot be upgraded, contact Oracle Support for more information. Supported domain configurations are described in Understanding SOA Domain Upgrade Restrictions.

Do not attempt to upgrade or schemas or domain configurations in an unsupported domain.

A.12 Business Rules Audit Trail Not Showing After Instance Upgrade

The audit trail for upgraded 11g instances of the Decision Service Component will not be available post-upgrade. The audit trail for new 12c instances will continue to display.

A.13 Resolving a Coherence Cache Exception

If you see the following WebLogic Cache Provider Coherence exception then it is likely that you are not following an enterprise deployment topology recommendation to specify a specific ListenAddress.

When you see this exception, you must set the ListenAddress for your managed server as shown below:

Exception:

weblogic.cacheprovider.coherence.CoherenceException:
  at weblogic.cacheprovider.coherence.CoherenceClusterManager.ensureWKAAddresses(CoherenceClusterManager.java:510)
  at weblogic.cacheprovider.coherence.CoherenceClusterManager.configureClusterService(CoherenceClusterManager.java:236)
  at weblogic.cacheprovider.CacheProviderServerService.bootCoherenceFromWLSCluster(CacheProviderServerService.java:225)
  at weblogic.cacheprovider.CacheProviderServerService.initCoherence(CacheProviderServerService.java:94)

Resolution:

  1. Log in to the WebLogic Server Console.

  2. Navigate to Servers.

  3. Locate the Managed Servers (SOA or OSB, for example).

  4. Modify the Listen Address from localhost to 127.0.0.1 or provide the actual machine name.

A.14 WSDL Generated Missing Elements for Custom Exception

If your EJBs contain custom exceptions, and you export the Web Service Description Language (WSDL) file from your EJB business service, the generated WSDL file will not have the custom exception properties in it. You will need to manually edit the WSDL file to include these custom exception properties after the upgrade.

The issue is limited only to the WSDL generation part of the file. During runtime, the custom exception thrown from the EJB will be mapped to the respective elements in the SOAP fault. The response payload will have the elements populated corresponding to the properties of the custom exception.

A.15 Connecting to the ServerSocket through Remote Clients

There is a change in behavior in which the ServerSocket is created when you upgrade from Oracle Release 11g to Release 12g. Because of this, remote clients might not able to connect to the ServerSocket when the hostname is configured as localhost. As a workaround, the localhost should be changed to hostname.

For more information, see "Configuring Oracle Socket Adapter" Understanding Technology Adapters.