Troubleshooting an Upgrade
Understanding the errors that you may encounter while running the Upgrade Assistant will help you to effectively troubleshoot them.
Note:Your course of action depends on the phase in which the error occurred.
If errors occur while you are running the Upgrade Assistant, use the following steps to troubleshoot the problem:
- Locate and open the Upgrade Assistant log file with a text editor:
- To verify the correct version of the Upgrade Assistant
Got to the the
Enter the following command:
unzip -p ua.jar META-INF/MANIFEST.MF
Note:The output of the
unzip -p ua.jar META-INF/MANIFEST.MFcommand identifies the development label that was used to build the Upgrade Assistant, and that information identifies the date and version of the Upgrade Assistant software that was run. If you submit a service request, you need to provide this information to Oracle.
- Locate any error messages that are identified by number; for example,
- Look up the error number in Error Messages.
The error description may include the cause of the error, as well as the action you should take to resolve the error.
- Based on whether you can locate an error message and the error message description, do one of the following:
If you are able to identify a solution to the upgrade failure, implement your solution, then restart the Upgrade Assistant to perform the upgrade again.
When you rerun the Upgrade Assistant, any components that were upgraded successfully during the previous run are not affected. However, the Upgrade Assistant attempts to upgrade any components that were not upgraded successfully.
Contact Oracle Support about any errors that are not documented or that cannot be resolved by following documented actions. Note that some errors that occur require the repository to be restored from backup, the problem to be resolved, and another upgrade to be run. Note that Oracle Support requires you to provide both the
UA.logfile and the
UA.outfile, if present. Provide complete logs and not just excerpts of those files.
If you get any errors during Examination phase, and no components or schemas have been upgraded yet, run the readiness check. The types of checks performed by the readiness check are more thorough than Examination phase of upgrade.
Errors that occur during or after the upgrade, however, require you to restore your environment from your backup copies, correct the errors and then restart the upgrade process from the beginning.
Reviewing Log Files
Do not delete the log files. They help diagnose and correct the problem while you run the Upgrade Assistant.
-logLevelvalue on the command line. The default value is
-logLevel NOTIFICATION. You can alter the location of your log files using the
-logDirparameter. You can obtain more detailed logging information by running the Upgrade Assistant with
-logLevel TRACE. However, this may cause the log file to become very large.
TRACEmessages are not included in the Upgrade Assistant Log File Viewer. To view
TRACEmessages, use another tool.
Tip:To expedite the review process, search for the word
For more information about the error messages in your log files, see Resolving Common Upgrade Assistant Errors.
Log files are stored in the following default directory:
timestamp states the actual date and time when the Upgrade Assistant was run.
Some components create a second log file called
uatimestamp.out, also in the same location.
For database schema upgrades of certain components, there can also be an output (
.out) file that contains the screen output of commands that were run in a shell process or as PL/SQL scripts. You can locate these output files in the same default directory.
If there are questions or issues about an upgrade failure that cannot be resolved with the information in this guide, it is important to retain the log files. If you submit a service request, you must upload the entire Upgrade Assistant
.log file and the
.out files (if available) along with the service request.
Investigating Examination Failures
Identify and troubleshoot failures that occur during the Examination phase of the upgrade.
To determine the cause of an Examination failure:
- Note the name of the failed component in the Upgrade Assistant dialog or command-line output.
- Open the following Upgrade Assistant log file.
- In the log file, search for the message
Starting to examine
To complete the upgrade, resolve the issues and then launch the Upgrade Assistant again, or, if possible, click Back to return to a previous screen and make the necessary changes.
Note:The readiness check performs checks to a far greater level of detail than the Examination phase. If Examine fails, you can run the Upgrade Assistant with the
-readinessparameter and make sure that the report does not show any test failures.
Issues detected during the Examination phase can be resolved without restoring from backup (as is required to resolve errors encountered during the actual upgrade). However, if you attempt to resolve an Examination error in a way that changes the state of the system, you need to restore the entire system to the pre-upgrade state (before any upgrade operations were attempted).
Investigating Upgrade Failures
Identify and troubleshoot failures that occur during the upgrade.
To determine the cause of an upgrade failure:
- Note the name of the failed component in the Upgrade Assistant dialog or command-line output.
- Open the upgrade log file:
- Search for the message
Starting to upgrade
To complete the upgrade, restore the entire environment using your pre-upgrade backup to a point in time before any upgrade operations were attempted, resolve the issues, then launch the Upgrade Assistant again. You will have to start the upgrade process from the beginning to ensure a successful (complete) upgrade.
Note:You should back up all databases with
RMANand be able to do a point-in-time recovery from those backups. If the Fusion Middleware repository for your domains spans multiple Oracle Database servers, you must restore from each of those backups.
Resolving Common Upgrade Assistant Errors
Attempt to resolve common Upgrade Assistant errors before contacting Oracle Support.
The following sections provide descriptions of the most common upgrade errors. For a complete list of Fusion Middleware errors, see Error Messages.
Ensuring there is sufficient disk space
If an upgrade fails because the database server has run out of disk space, you must restore the database server environment from backups, add sufficient disk space or remove unwanted files (such as temp or trace files) from the database server, and then retry the upgrade.
Note:Once a database schema upgrade has failed due to this class of error, you cannot simply add more disk space and retry the upgrade. The schemas have been left in an inconsistent state and may have been marked
INVALID. You cannot recover from this error without restoring the original database state from backups.
The following examples show some insufficient disk space errors that you may encounter:
ORA-01658: unable to create INITIAL extent for segment in tablespace
Cause: The existing schema tablespace does not have sufficient space to complete the upgrade.
Action: Make sure that the tablespace has sufficient room (space) for a successful upgrade. Oracle recommends that you add more data files to the existing database tablespaces, otherwise the upgrade will fail.
ORA-01114: IO error writing block to file block number
Cause: The device on which the file resides is probably offline. If the file is a temporary file, then it is also possible that the device has run out of space. This could happen because disk space of temporary files is not necessarily allocated at file creation time.
Action: Restore access to the device or remove unnecessary files to free up space.
ORA-09945: Unable to initialize the audit trail file
Cause: The system is unable to write header information to the file being used as the audit trail. The audit_trail_dest or audit trail destination is full for generation of audit file.
Action: Free up space and retry the operation.
Resolving Database Connection Problems When Upgrading Schemas
If you have trouble connecting to a database when using the Upgrade Assistant to upgrade a component schema, try connecting to the database using another tool, such as SQL*Plus. This helps you troubleshoot the problem by verifying that the database is up and running and available on the network.
Attempting to Upgrade an Unsupported Domain
Do not attempt to upgrade the schemas or domain configurations in an unsupported domain.
If you receive an error stating that the specified domain cannot be upgraded, you must first upgrade the domain to a supported version. Do not attempt to upgrade schemas or domain configurations in an unsupported domain.
Restarting the Upgrade Assistant After a Failure
You must resolve errors before you restart the Upgrade Assistant.
If the Upgrade Assistant fails during the upgrade phase, or only partially upgrades your components, try to resolve the issues and then follow these steps:
- Identify and resolve the issue. Review the log files and note that you may need to apply a patch. If you continue to experience upgrade failures, consider setting
TRACEso that more information is logged. This is useful when troubleshooting a failed upgrade, but be sure to reset
NOTIFICATIONafter the issue has been resolved to avoid performance issues.
- Restore the entire 11g or 12c environment from your pre-upgrade backup.
- Restart the upgrade as described in your component-specific upgrade guide.