23 Troubleshooting RMAN Operations
Use RMAN message output and dynamic performance views to troubleshoot RMAN operations.
23.1 Interpreting RMAN Message Output
Recovery Manager provides detailed error messages that can aid in troubleshooting problems.
Also, Oracle Database and the third-party media vendors generate useful debugging output of their own. The following discussion explains how to identify and interpret the different errors that you may encounter.
23.1.1 Identifying Types of RMAN Message Output
Output that is useful for troubleshooting failed or unresponsive RMAN jobs is located in several different places.
The following table provides an overview of where to locate message output that can be used to troubleshoot RMAN backup problems.
Table 23-1 Types of Message Output
Type of Output | Produced By | Location | Description |
---|---|---|---|
RMAN messages |
RMAN |
Completed job information is in When running RMAN from the command line, you can direct output to the following places:
|
Contains actions relevant to the RMAN job and error messages generated by RMAN, the database server, and the media vendor. RMAN error messages have an You can execute the following PL/SQL to remove all entries from update node set high_rsr_recid=0
where db_key = our_target_database_db_key ; The preceding function removes all job-related entries. No rows are visible until new backup jobs are shown in |
|
Oracle Database |
The |
Contains a chronological log of errors, initialization parameter settings, and administration operations. Records values for overwritten control file records. |
Oracle trace file |
Oracle Database |
The |
Contains detailed output generated by Oracle Database processes. This file is created when an |
|
Third-party media management software |
The |
Contains vendor-specific information written by the media management software. This log does not contain Oracle Database or RMAN errors. |
Media manager log file |
Third-party media management software |
The file names for any media manager logs other than |
Contains information about the functioning of the media management device |
23.1.2 Troubleshooting Long-Running RMAN Operations
RMAN message output provides information about the progress of backup and recovery operations. Use this information to take any required actions to troubleshoot operations that are stuck or awaiting resources.
Certain operations such as backup, restore, recovery, and duplication for large databases typically take a long time to complete. However, it is not always clear if the operation is progressing or waiting on some resources. Starting with Oracle Database Release 18c, RMAN message output contains additional logging information that indicates if a job is waiting on resources. Every 10 minutes, RMAN checks if there is a change in the number of blocks processed. If there no change in the blocks processed, then RMAN displays a message with the associated wait event.
The following is an example of the RMAN output for a RESTORE
operation:
allocated channel: c1
channel c1: SID=123 device type=SBT_TAPE
channel c1: WARNING: Oracle Test Disk API
Starting restore at 18-JAN-18
channel c1: starting datafile backup set restore
channel c1: specifying datafile(s) to restore from backup set
channel c1: restoring datafile 00002 to /ade/b/2776899351/oracle/dbs/tbs_ax1.f
channel c1: reading from backup piece 01sov1t4_1_1
***** Hang Detected ***** at 2018-01-18 04:11:23 for channel c1, INSTID: 1, SID: 123, serial: 35831
No change in read blocks, thus showing wait event[Total blocks = 192000, Blocks read/recovered = 41530]
Seq_No Event Waiting Time(mirco secs)
602 Backup: MML read backup piece 38094371
***** Hang Detected ***** at 2018-01-18 04:11:33 for channel c1, INSTID: 1, SID: 123, serial: 35831
No change in read blocks, thus showing wait event[Total blocks = 192000, Blocks read/recovered = 41530]
Seq_No Event Waiting Time(mirco secs)
602 Backup: MML read backup piece 48106104
channel c1: piece handle=01sov1t4_1_1 tag=TAG20180118T040804
channel c1: restored backup piece 1
channel c1: restore complete, elapsed time: 00:02:35
Finished restore at 18-JAN-18
released channel: c1
The output indicates that the restore was stuck because of a problem with a media manager read operation. After the read operation completed, the RMAN restore was successful.
23.1.3 Recognizing RMAN Error Message Stacks
RMAN reports errors as they occur. If an error is not retrievable, that is, if RMAN cannot perform failover to another channel to complete a particular job step, then RMAN also reports a summary of the errors after all job sets complete. This feature is known as deferred error reporting.
One way to determine whether RMAN encountered an error is to examine its return code. A second way is to search the RMAN output for the string RMAN-00569
, which is the message number for the error stack banner. All RMAN errors are preceded by this error message. If you do not see an RMAN-00569
message in the output, then there are no errors.
RMAN-01109
RMAN client failures are logged as a
RMAN-01109
message in the output.RMAN-01109
indicates the file name where the client diagnostics traces are loggedRMAN-01110
RMAN failures during the command execution phase are logged as a
RMAN-01110
message in the output.RMAN-01110
indicates the file name where the server diagnostics traces are logged.- If the RMAN client fails to connect to a server, then the RMAN output includes the error code
RMAN-01111
indicating that RMAN server traces were not dumped
Example 23-1 RMAN Syntax Error
This example shows an RMAN syntax error. The RMAN-00569
message is followed by other error messages that indicate the reason for the error.
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-00558: error encountered while parsing input commands RMAN-01005: syntax error: found ")": expecting one of: "archivelog, backup, backupset, controlfilecopy, current, database, datafile, datafilecopy, (, plus, ;, tablespace" RMAN-01007: at line 1 column 18 file: standard input
Example 23-2 RMAN Error Indicating the Trace File Names
This examples shows an RMAN error message with RMAN-01109
and RMAN-01110
error codes indicating the location of the RMAN client and server process trace files.
RMAN-00571: =========================================================== RMAN-03002: failure of sql command at 06/08/2023 10:33:29 RMAN-03014: implicit resync of recovery catalog failed RMAN-03009: failure of partial resync command on default channel at 06/08/2023 10:33:29 RMAN-20110: set_stamp set_count conflict RMAN-01109: RMAN Client Diagnostic Trace file : /ade/b/368400417/oracle/log/diag/clients/user_name/host_name/trace/ora_570883_140446658743104.trc RMAN-01111: RMAN Server Diagnostic Trace file : /ade/b/368400417/oracle/log/diag/rdbms/tst/tst/trace/tst_ora_570897.trc
Note:
To avoid storage space related issues, you must ensure that the in-memory trace files are regularly purged. Use the Automatic Diagnostic Repository Command Interpreter (ADRCI) utility to set policies that control automatic purging of diagnostic trace files. See the Oracle Database Utilities to learn how to purge diagnostic data.See Also:
23.1.4 Identifying RMAN Error Codes
You can use the error codes in RMAN message stacks to troubleshoot problems with RMAN commands.
Typically, you find the following types of error codes in RMAN message stacks:
-
Errors prefixed with
RMAN-
These are RMAN errors.
-
Errors prefixed with
ORA-
Media manager errors use the
ORA-
prefix. -
Errors preceded by the line
Additional information:
See Also:
-
RMAN Error Message Numbers for the error ranges of RMAN errors
-
ORA-19511: Media Manager Errors for the error ranges of media manager errors
-
Oracle Database Error Messages Referencefor explanations of
RMAN
andORA
error codes
23.1.4.1 RMAN Error Message Numbers
RMAN error messages are prefixed with RMAN-.
The following table indicates the error ranges for common RMAN error messages, all of which are described in Oracle Database Error Messages Reference.
Table 23-2 RMAN Error Message Ranges
Error Range | Cause |
---|---|
01108-01112 |
Information about in-memory debug tracing of RMAN client and server processes |
0550-0999 |
Command-line interpreter |
1000-1999 |
Keyword analyzer |
2000-2999 |
Syntax analyzer |
3000-3999 |
Main layer |
4000-4999 |
Services layer |
5000-5499 |
Compilation of |
5500-5999 |
Compilation of |
6000-6999 |
General compilation |
7000-7999 |
General execution |
8000-8999 |
PL/SQL programs |
9000-9999 |
Low-level keyword analyzer |
10000-10999 |
Server-side execution |
11000-11999 |
Interphase errors between PL/SQL and RMAN |
12000-12999 |
Recovery catalog packages |
23.1.4.2 ORA-19511: Media Manager Errors
If a media manager error occurs, ORA-19511
is signaled, and the media manager is expected to provide RMAN a descriptive error. RMAN displays the error passed back to it by the media manager.
For example, you might see this:
ORA-19511: Error received from media manager layer, error text: sbtpvt_open_input: file .* does not exist or cannot be accessed, errno = 2
The message from the media manager should provide you with enough information to let you fix the root problem. If it does not, then refer to the documentation for your media manager or contact your media management vendor support representative for further information. ORA-19511
errors originate with the media manager, not with Oracle Database. The database just passes on the message from the media manager. The cause can be addressed only by the media management vendor.
If you are still using an SBT 1.1-compliant media management layer, you may see some additional error message text. Output from an SBT 1.1-compliant media management layer is similar to the following:
ORA-19507: failed to retrieve sequential file, handle="c-140148591-20031014-06", parms="" ORA-27007: failed to open file Additional information: 7000 Additional information: 2 ORA-19511: Error received from media manager layer, error text: SBT error = 7000, errno = 0, sbtopen: backup file not found
The "Additional information" provided uses error codes specific to SBT 1.1. The values displayed correspond to the media manager message numbers and error text listed in Table 23-3. RMAN again signals the error, as an ORA-19511 Error received from media manager layer
error, and a general error message related to the error code returned from the media manager and including the SBT 1.1 error number is then displayed.
The SBT 1.1 error messages are listed here for your reference. Table 23-3 lists media manager message numbers and their corresponding error text. In the error codes, O/S
stands for operating system. The errors marked with an asterisk (*) are internal and are not typically seen during normal operation.
Table 23-3 Media Manager Error Message Ranges
Cause | No. | Message |
---|---|---|
sbtopen |
7000 7001 7002* 7003 7004 7005 7006 7007 7008 7009 7010 7011 7012* |
Backup file not found (only returned for read) File exists (only returned for write) Bad mode specified Invalid block size specified No tape device found Device found, but busy; try again later Tape volume not found Tape volume is in-use I/O Error Can't connect with Media Manager Permission denied O/S error for example malloc, fork error Invalid argument(s) to sbtopen |
sbtclose |
7020* 7021* 7022 7023 7024* 7025 |
Invalid file handle or file not open Invalid flags to sbtclose I/O error O/S error Invalid argument(s) to sbtclose Can't connect with Media Manager |
sbtwrite |
7040* 7041 7042 7043 7044* |
Invalid file handle or file not open End of volume reached I/O error O/S error Invalid argument(s) to sbtwrite |
sbtread |
7060* 7061 7062 7063 7064 7065* |
Invalid file handle or file not open EOF encountered End of volume reached I/O error O/S error Invalid argument(s) to sbtread |
sbtremove |
7080 7081 7082 7083 7084 7085 7086* |
Backup file not found Backup file in use I/O Error Can't connect with Media Manager Permission denied O/S error Invalid argument(s) to sbtremove |
sbtinfo |
7090 7091 7092 7093 7094 7095* |
Backup file not found I/O Error Can't connect with Media Manager Permission denied O/S error Invalid argument(s) to sbtinfo |
sbtinit |
7110* 7111 7112 7113 7114 |
Invalid argument(s) to sbtinit O/S error The current RDBMS version does not support RA_FORMAT=TRUE. Auto allocated buffers required when using RA_FORMAT=TRUE. RA_REPLICATION and RA_FORMAT cannot both be set to true. |
common error codes |
7500 7501 7502 7503 7504 7505 7506 7507 7508 |
Operator intervention requested Media Management Error File not found File already exists End of file cannot proxy copy the specified file no proxy work in progress no buffer available aborting due to signal |
Starting with Oracle Database 23ai, native SBT libraries are available in Oracle home directory as part of the database installation. After installing the database, you can configure RMAN channels to use the native SBT libraries for backup and restores with Oracle Cloud, Recovery Appliance, and Amazon S3.
If any issues occur when Oracle uses the native SBT media libraries, RMAN signals ORA-19511
followed by a descriptive error message that is prefixed with KBHS. Error messages prefixed with KBHS provide a clear indication of the cause of the error.
Example 23-3 KBHS Error Message Displayed when Oracle uses Native SBT Libraries to Access Cloud Storage Locations
This example shows an error that occurs when Oracle access a native SBT library. The ORA-19511
is followed by the KBHS-01025
message that indicates the cause.
RMAN-03090: Starting backup at APR 17 2018 16:00:21 RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of backup command at 04/17/2018 16:00:23 ORA-19554: error allocating device, device type: SBT_TAPE, device name: ORA-27023: skgfqsbi: media manager protocol error ORA-19511: non RMAN, but media manager or vendor specific failure, error text: KBHS-01025: Parameter OPC_WALLET and OPC_CREDENTIAL_OBJECT cannot both be specified
KBHS error messages are listed here for your reference, and also described in the Oracle Database Error Messages Reference.
Table 23-4 KBHS Error Messages (Range 00100 - 00900)
KBHS - Error No - Message | Cause | Action |
---|---|---|
KBHS - 00100 - Parameter string value string is not a multiple of BLKSIZE string specified in SBT parms |
CHUNK_SIZE specified in configuration was not a multiple of BLKSIZE |
Change BLKSIZE or CHUNK_SIZE configuration and retry the command. |
KBHS - 00101 - file string already exists; use |
An attempt was made to create a file with a name that is already used. |
Use |
KBHS - 00103 - Version strings of string is not licensed |
The indicated version of the library was not licensed. |
Contact Oracle Support Services for assistance. |
KBHS - 00104 - License of string version string expired on string |
The indicated version of the library has expired. |
Renew your license. Contact Oracle Support Services for assistance. |
KBHS - 00200 - Product string is not licensed |
The system backup tape (SBT) library was not licensed. |
Contact Oracle Support Services for assistance. |
KBHS - 00201 - The license of string expired on string |
The system backup tape (SBT) library license has expired. |
Renew your license. Contact Oracle Support Services for assistance. |
KBHS - 00202 - cannot create log bucket string; retry after library registration |
An attempt was made to re-create the log bucket specified in the license file. This attempt failed. |
Retry the command after library registration. |
KBHS - 00203 - maximum number of session licenses (string) exceeded |
All licenses were in use. |
Increase the value of the LICENSE_MAX_SESSIONS parameter. |
KBHS - 00204 - data bucket string location conflicts with log bucket string location |
Log and Data buckets were in different geographic location. |
Delete the bucket that is in conflict and retry the command. |
KBHS - 00205 - location of bucket string is string, which conflicts with location string specified |
Log or Data bucket were in different geographic location specified. |
Re-run installer with location and reRegister option to change geographic location and retry the command. |
KBHS - 00206 - file string already exists; use |
An attempt was made to create a file with a name that is already used. |
Use ' |
KBHS - 00207 - parameter string value string conflicts with bucket/container string property string |
Bucket/Container with different property specified. |
Check the specified parameter in the configuration file. |
KBHS - 00208 - server side encryption container string is not supported |
Container specified has server side encryption enabled. This is not supported. |
Disable server side encryption or retry the command with different container name. |
KBHS - 00209 - life cycle tiering policy for container string must exclude XML objects |
The specified tiering-enabled container would archive XML objects. This is not supported. |
Exclude XML objects in the container LTP. |
KBHS - 0400* - NETTEST BACKUP: number bytes sent in number microseconds |
Displays NETTEST BACKUP result. |
This is an undocumented output for internal usage only. No action required. |
KBHS - 00401* - NETTEST RESTORE: number bytes received in number microseconds |
Displays NETTEST RESTORE result. |
This is an undocumented output for internal usage only. No action required. |
KBHS - 00402* - NETTEST successfully completed |
NETTEST command returns successfully. |
This is an undocumented output for internal usage only. No action required. |
KBHS - 00600 - internal error |
An internal error in Oracle HTTP SBT occurred. |
Contact Oracle Customer Support. |
KBHS - 00601 - fatal error in Oracle HTTP SBT |
A fatal error has occurred. |
This message should be accompanied by other error message(s) indicating the cause of the error. See |
KBHS - 00700 - HTTP response error 'string' |
An HTTP operation returned an error. |
See accompanying error messages for more information. |
KBHS - 00701 - no host name or port number or host name too long |
Host name absent or too long, port number absent or invalid. |
Ensure the host name is correct and the port number is valid. |
KBHS - 00702 - connect failed because host string is unreachable |
The address specified was unreachable or not valid, or the service at the address was unavailable. |
Verify that HOST and PROXY parameters are valid. |
KBHS - 00703 - unable to connect to HTTP server string; received ORA-string |
Connect to HTTP server failed or timed out. |
Verify HOST and PROXY parameter values. See ORA error description for more information. |
KBHS - 00704 - unable to do register connection; received ORA-string |
Registering the connection failed. |
See ORA error description for more information. |
KBHS - 00705 - connection closed or lost contact; received ORA-string |
HTTP server has disconnected. |
See ORA error description for more information. |
KBHS - 00706 - unable to wait for notification; received ORA-string |
Wait for event notification on registered connection failed. |
See ORA error description for more information. |
KBHS - 00707 - HTTP response timed out; waited string seconds |
The HTTP server did not respond within the specified timeout. |
This is usually a HTTP server issue. Contact HTTP server administrator or increase the timeout value. |
KBHS - 00708 - unable to receive data from HTTP server; error ORA-string |
Receive from HTTP server failed. |
See ORA error description for more information. |
KBHS - 00709 - unable to send data to HTTP server; error ORA-string |
HTTP send operation to server failed. |
See ORA error description for more information. |
KBHS - 00710 - unable to flush data from buffer; error ORA-string |
Flushing data from local buffer to HTTP server failed. |
See ORA error description for more information. |
KBHS - 00711 - unable to close HTTP session; error ORA-string |
HTTP close session operation failed. |
See ORA error description for more information. |
KBHS - 00712 - ORA-string received from local HTTP service |
An error occurred during HTTP processing. |
See accompanying error messages for more information. |
KBHS - 00713 - HTTP client error 'string' |
The HTTP response indicated an HTTP client error. |
Fix the HTTP client error and retry the HTTP request. |
KBHS - 00714 - HTTP server error 'string' |
The HTTP response indicated an HTTP server error. |
Fix the HTTP server error and retry the HTTP request. Contact the administrator of the HTTP server when necessary. |
KBHS - 00715 - HTTP error occurred 'string' |
The specified HTTP protocol error happened. |
See accompanying error messages for more information. |
KBHS - 00717 - file name string contains an illegal character string |
The specified file name contained a character that was not allowed within an SBT backup file name. |
Use a file name that does not contain the specified illegal character and retry the command. |
KBHS - 00718 - operation failed, retry possible |
A HTTP HEAD, DELETE, PUT or GET operation failed with an error. The operation may be retried. |
This message is used by SBT HTTP API to decide whether or not to retry the operation. |
KBHS - 00719 - Error 'string'; string |
Description of error returned by HTTP operation. |
See accompanying error messages for more information. |
KBHS - 00720 - bucket cannot be accessed using end-point for request string |
The end-point specified in the HOST parameter was not the home location for the bucket in the URL. |
Change HOST to the home location of the bucket. |
KBHS - 00721 - resource busy, retry possible |
An HTTP HEAD, DELETE, PUT or GET operation failed because the resource was busy. The operation may be retried. |
This message is used by SBT HTTP API to decide whether or not to retry the operation. |
KBHS - 00722 - missing or invalid EC2 instance metadata for IAM role string |
The metadata corresponding to the specified IAM role does not exist or the temporary security credentials are invalid. |
Verify IAM role name. |
KBHS - 00723 - missing header string in HTTP response |
The expected header is not available in the HTTP response. |
Check if the submitted HTTP request is appropriate. |
KBHS - 00724 - invalid value string for header string |
header value is invalid |
Check if the submitted HTTP request is appropriate. |
KBHS - 00900 - virtual file initialization failed |
Unable to initialize virtual file. |
Contact Oracle Customer Support. |
Table 23-5 KBHS Error Messages (Range 01000-01803)
KBHS - Error No - Message | Cause | Action |
---|---|---|
KBHS-01000 - value of parameter string must be a number; specified string |
The value given for the specified parameter was not numeric. |
Change the value of the parameter to numeric and retry the command. |
KBHS-01001 - value of parameter string must be TRUE or FALSE; specified string |
The value given for the specified parameter was not TRUE or FALSE. |
Change the parameter to TRUE or FALSE and retry the command. |
KBHS - 01002 - unable to open string parameter file string |
Encountered an error while trying to open the parameter file. |
Ensure that the specified file exists and has read permission. |
KBHS - 01003 string |
An error occurred when parsing a parameter. |
See accompanying error messages indicating the cause of error. |
KBHS-01004 - LRM-string occurred when parsing a parameter |
An error occurred when parsing a parameter. |
See accompanying error messages indicating the cause of error. |
KBHS-01005 - syntax error was encountered while parsing string parameter file string |
An error occurred when parsing parameter file. |
See accompanying error messages for more information. |
KBHS-01006 - Parameter string was not specified |
Failed to specify the indicated parameter. |
Specify the value of the parameter and retry the command. |
KBHS-01008 - syntax error was encountered while parsing string parameter string |
An error occurred when parsing parameter string. |
See accompanying error messages for more information. |
KBHS-01009 - Parameter string must not be specified as sub-domain format |
The specified parameter cannot be used in sub-domain. |
Remove the sub-domain format and retry the command. |
KBHS-01010 - user name or password was not specified in the wallet |
user name or password specified in the wallet was null. |
Ensure that username and password is not null and retry the command. |
KBHS-01012 - ORA-string occurred during wallet operation; WRL string |
An operation on the wallet failed due to indicated error. |
Refer to indicated oracle message for more information. |
KBHS-01013 - specified string alias string not found in wallet |
The specified WALLET alias did not appear in the wallet. |
Check the WALLET alias or create an alias in the wallet for the specified attribute and retry the command. |
KBHS-01014 - Parameter string must contain string attribute |
The WALLET parameter was missing the mandatory attribute. |
Add the missing attribute and retry the command. |
KBHS-01015 - value of parameter string is out of range (string - string) |
The value given for the specified parameter was not valid. |
Change the value of the parameter within the range and retry the command. |
KBHS-01016 - syntax error was encountered while parsing parameter string |
An error occurred when parsing parameter. |
See accompanying error messages for more information. |
KBHS-01017 - Parameter string value string contains too many attributes |
The WALLET parameter contains many attributes. |
Check the syntax of |
KBHS-01018 - parameter string must not be specified |
The indicated parameter should not be specified. |
Remove the indicated parameter and retry the command. |
KBHS-01019 - value of parameter string is invalid; specified string |
The value given for the specified parameter was not valid. |
Change the parameter to valid value and retry the command. |
KBHS-01020 - value of parameter string must be TRUE for authentication string |
The value given for the specified parameter was not valid. |
Change the parameter to valid value and retry the command. |
KBHS-01021 - AWS region could not be determined |
AWS region could not be determined. |
Change authentication scheme and retry the command. |
KBHS-01022 - endpoint to restore archive objects cannot be constructed |
HOST parameter is not under the v1 namespace. |
Use HOST parameter under v1 namespace. |
KBHS - 01023 - Either parameter string or parameter string must be specified |
Specifying one of the parameter is mandatory. |
Specify one of the parameter and retry the command. |
KBHS - 01024 - invalid credential object [schema = string, name = string] |
user name or password not specified in the credential object. |
Ensure that the credential object is valid and retry the command. |
KBHS - 01025 - Parameter string and string cannot both be specified |
Cannot specify both parameter. |
Specify only one of the parameter and retry the command. |
KBHS - 01026 - SSLWallet location could not be retrieved from database |
SSLWallet location was not found in database. |
Ensure that SSLWallet location is specified in database and retry the command. |
KBHS - 01100 - end-of-file for backup piece string |
There were no more chunks for this backup piece to read or read bytes specified in the metadata file. |
If the end-of-file was reached prematurely, check if all the chunks are present in the backup piece. |
KBHS - 01102 - unable to read backup piece string because of missing chunk string |
Unable to read the backup piece because the specified chunk was found to be missing. |
Retry the command after restoring the chunk file in the specified location. |
KBHS - 01103 - backup piece string missing |
Unable to read the backup piece because the specified piece was found to be missing. |
Retry the command after restoring the file in the specified location. |
KBHS - 01104 - file string is not in backup piece format |
Unable to read the backup piece header because the specified file was not in backup piece format. |
Retry the command after restoring the file in the specified location. |
KBHS - 01105 - Block Pool format restore does not support password decryption |
When a backup is performed using |
Retry the command without password. |
KBHS - 01300 - unable to read backup piece string because of missing metadata file string |
The backup piece could not be read because the metadata file was missing. |
Retry the command after restoring the metadata file in the specified location. |
KBHS - 01400 - Unable to load message file from virtual file system |
Unable to find the message file in virtual file system. |
Contact Oracle Customer Support. |
KBHS - 01401 - memory allocation failure |
Insufficient memory available to satisfy requested allocation. |
Terminate other processes to free up memory or add memory to the system. |
KBHS - 01402 - Oracle error occurred while converting a date: ORA-number: string |
An internal error occurred while converting a date. |
Contact Oracle Support Services. |
KBHS - 01403 - could not open trace file string |
An error occurred when trying to open the trace file. |
Check that the directory exists and that the file has write permissions. |
KBHS - 01405 - Automatic retry wait time limit string reached |
An attempt was made to automatically retry the operation but the configured wait time limit was reached. |
Retry the operation manually. |
KBHS - 01406 - value of parameter string must be between string and string; specified string |
The length of the parameter was not within the limits. |
Specify a value of length within the limits and retry the command. |
KBHS - 01407 - bucket name string contains an illegal character string |
One of the following restrictions were not followed in the specified bucket name:
|
Use a bucket name that follows the restriction and retry the command. |
KBHS - 01408 - log in to Recovery Appliance failed with Oracle error:string |
An Oracle error was reported while attempting to log in to the Recovery Appliance. |
Follow the actions for the specified Oracle error. |
KBHS - 01409 - Oracle error reported from Recovery Appliance while executing string |
An Oracle error was reported when executing an Oracle Call Interface (OCI) operation in Recovery Appliance. |
Follow the actions for the specified Oracle error. |
KBHS - 01410 - Oracle HTTP Server not running in Recovery Appliance |
Oracle HTTP Server was not running in Recovery Appliance. |
Start the HTTP Server in Recovery Appliance and retry the command. |
KBHS - 01411 - Recovery Appliance access was not granted |
Recovery Appliance access was not granted to this user. |
Grant database access to this user and then retry the command. |
KBHS - 01412 - failed for connection string (@string) |
This is an informational message. This precedes error 1410 or 1411. |
No action is required. |
KBHS - 01413 - connect string could not be resolved |
The specified host is not resolvable. |
Check the spelling of the host name or the IP address. Make sure that the host name or the IP address is resolvable on the Recovery Appliance. |
KBHS - 01500 - syntax error was encountered while parsing SBT command string |
An error occurred when parsing SBT command. |
See accompanying error messages for more information. |
KBHS - 01600 - backup piece string header validation failed |
The header was not recognized as a valid backup piece header. One reason could be that the backup piece was converted at the target and a converted backup piece cannot be transmitted. |
Ensure the backup piece is not converted and retry the backup operation. |
KBHS - 01601 - Data Pump dump file backup piece string is not supported |
Data Pump dump file backup was requested. This cannot be transmitted and is not supported. |
Ensure the backup piece does not include a Data Pump dump file and retry the backup operation. |
KBHS - 01602 - backup piece string is not encrypted |
Backups sent to Oracle Public Cloud Storage was not encrypted. Only encrypted backups can be stored in Oracle Public Cloud Storage. |
Configure RMAN to create encrypted backups and retry the command. |
KBHS - 01603 - Incremental backups that include control or |
Backups with |
Configure RMAN to use |
KBHS - 01605 - The current ZDLRA version does not support |
When |
Update the ZDLRA software to the appropriate version. |
KBHS - 01606 - Invalid compression algorithm when using |
When |
Set the RMAN compression algorithm to LOW. |
KBHS - 01607 - Password encryption is not supported when using |
When RA_FORMAT=TRUE is set, the only supported encryption method is Transparent Data Encryption. |
Use Transparent Data Encryption instead of password encryption. |
KBHS - 01608 - Previously compressed backups not supported with |
When |
Take the initial backup without compression. |
KBHS - 01609 - Previously encrypted backups not supported with |
When |
Take the initial backup without encryption. |
KBHS - 01610 - |
Backups with |
If you need maxpiecesize backups, configure RMAN without |
KBHS - 01700 - Error occurred in XML processing string |
An error occurred when processing the XML document. |
Check the given error message and fix the appropriate problem. |
KBHS - 01701 - XML parsing failed |
XML parser returned an error while trying to parse the document. |
Check if the document to be parsed is valid. |
KBHS - 01800 - unmatched quote in parameter string |
Missing close quote was encountered. |
Add missing quote and retry command. |
KBHS - 01801 - invalid keyword (string) |
This is an informational message indicating unrecognized keyword was encountered. |
Correct the syntax and retry the command. |
KBHS - 01802 - incomplete/malformed command |
This is an informational message indicating the command was not complete. |
Correct the syntax and retry the command. |
KBHS - 01803 - invalid parameter (string) |
This is an informational message indicating the identifier token that caused a syntax error. |
Correct the syntax and retry the command. |
23.1.5 Interpreting RMAN Error Stacks
It is important to identify the relevant messages in the RMAN error stack.
Note the following tips and suggestions while interpreting RMAN messages:
-
Read the messages from the bottom up, because this is the order in which RMAN issues the messages. The last one or two errors displayed in the stack are often the most informative.
-
When you are using an SBT 1.1 media management layer and you are presented with SBT 1.1 style error messages containing the "
Additional information:
" numeric error codes, look for theORA-19511
message that follows for the text of error messages passed back to RMAN by the media manager. These messages identify the real failure in the media management layer. -
Look for the
RMAN-03002
orRMAN-03009
message (RMAN-03009
equalsRMAN-03002
but includes the channel ID), immediately following the error banner. These messages indicate which command failed. Syntax errors generateRMAN-00558
. -
Identify the basic type of error according to the error range chart in Table 23-2 and then refer to the error messages for information about the most important messages.
See Also:
-
Interpreting RMAN Errors: Example and Interpreting Server Errors: Example for examples of RMAN error messages
-
Interpreting SBT 2.0 Media Management Errors: Example and Interpreting SBT 1.1 Media Management Errors: Example for examples of interpreting media management errors
-
Oracle Database Error Messages for information about the error messages
23.1.5.1 Interpreting RMAN Errors: Example
Errors prefixed by RMAN- indicate errors caused by RMAN commands.
You attempt a backup of tablespace users
and receive the following message:
Starting backup at 29-AUG-13 using channel ORA_DISK_1 RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of backup command at 08/29/2013 15:14:03 RMAN-20202: tablespace not found in the recovery catalog RMAN-06019: could not translate tablespace name "USESR"
The RMAN-03002
error indicates that the BACKUP
command failed. You read the last two messages in the stack first and immediately see the problem: no tablespace users
appears in the recovery catalog because you mistyped the name as usesr
.
23.1.5.2 Interpreting Server Errors: Example
Errors from the server are prefixed with ORA-.
Assume that you attempt to recover a tablespace and receive the following errors:
RMAN> RECOVER TABLESPACE users; Starting recover at 29-AUG-13 using channel ORA_DISK_1 starting media recovery media recovery failed RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 08/29/2013 15:18:43 RMAN-11003: failure during parse/execution of SQL statement: alter database recover if needed tablespace USERS ORA-00283: recovery session canceled due to errors ORA-01124: cannot recover data file 8 - file is in use or recovery ORA-01110: data file 8: '/oracle/oradata/trgt/users01.dbf'
As suggested, you start reading from the bottom up. The ORA-01110
message explains there was a problem with the recovery of data file users01.dbf
. The second error indicates that the database cannot recover the data file because it is in use or being recovered. The remaining RMAN errors indicate that the recovery session was canceled due to the server errors. Hence, you conclude that because you were not recovering this data file, the problem must be that the data file is online and you must take it offline and restore a backup.
23.1.5.3 Interpreting SBT 2.0 Media Management Errors: Example
This example shows how to interpret errors caused at the media manager level.
Assume that you use a tape drive and see the following output during a backup job:
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== ORA-19624: operation failed, retry possible ORA-19507: failed to retrieve sequential file, handle="/tmp/mydir", parms="" ORA-27029: skgfrtrv: sbtrestore returned error ORA-19511: Error received from media manager layer, error text: sbtpvt_open_input:file /tmp/mydir does not exist or cannot be accessed, errno=2
The error text displayed following the ORA-19511
error is generated by the media manager and describes the real source of the failure. See the media manager documentation to interpret this error.
23.1.5.4 Interpreting SBT 1.1 Media Management Errors: Example
This example shows the output of a backup job that has errors media management errors.
Assume that you use a tape drive and see the following output during a backup job:
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03009: failure of backup command on c1 channel at 09/04/2013 13:18:19 ORA-19506: failed to create sequential file, name="07d36ecp_1_1", parms="" ORA-27007: failed to open file SVR4 Error: 2: No such file or directory Additional information: 7005 Additional information: 1 ORA-19511: Error received from media manager layer, error text: SBT error = 7005, errno = 2, sbtopen: system error
The main information of interest returned by SBT 1.1 media managers is the error code in the "Additional information" line:
Additional information: 7005
Referring to Table 23-3, you discover that error 7005
means that the media management device is busy. So, the media management software is not able to write to the device because it is in use or there is a problem with it.
Note:
The sbtio.log
contains information written by the media management software, not Oracle Database. Thus, you must consult your media vendor documentation to interpret the error codes and messages. If no information is written to the sbtio.log
, then contact your media manager support to ask whether they are writing error messages in some other location, or whether there are steps you must take to have the media manager errors appear in sbtio.log
.
23.1.6 Identifying RMAN Return Codes
One way to determine whether RMAN encountered an error is to examine its return code or exit status. The RMAN client returns 0 to the shell from which it was invoked if no errors occurred, and a nonzero error value otherwise.
How you access this return code depends upon the environment from which you invoked the RMAN client. For example, if you run UNIX with the C shell, then, when RMAN completes, the return code is placed in a shell variable called $status
. The method of returning exit status is a detail specific to the host operating system rather than the RMAN client.
23.2 Using V$ Views for RMAN Troubleshooting
When LIST
, REPORT
, and SHOW
do not provide all the information that you need for RMAN operations, some V$ views can provide useful details.
Sometimes it is useful to identify exactly what a server session performing a backup and recovery job is doing. The views described in the following table are useful for obtaining information about RMAN jobs.
Table 23-6 Useful V$ Views for Troubleshooting
View | Description |
---|---|
|
Identifies currently active processes |
|
Identifies currently active sessions. Use this view to determine which database server sessions correspond to which RMAN allocated channels. |
|
Lists the events or resources for which sessions are waiting |
You can use the preceding views to perform the following tasks:
23.2.1 Monitoring RMAN Interaction with the Media Manager
You can use the event names in the dynamic performance event views to monitor RMAN calls to the media management API. The event names have one-to-one correspondence with SBT functions.
See the following example:
Backup: MML v1 open backup piece Backup: MML v1 read backup piece Backup: MML v1 write backup piece Backup: MML v1 query backup piece Backup: MML v1 delete backup piece Backup: MML v1 close backup piece . . .
To obtain the complete list of SBT events, you can use the following query:
SELECT NAME FROM V$EVENT_NAME WHERE NAME LIKE '%MML%';
Before making a call to any of functions in the media management API, the server adds a row in V$SESSION_WAIT
, with the STATE
column including the string WAITING
. The V$SESSION_WAIT.SECONDS_IN_WAIT
column shows the number of seconds that the server has been waiting for this call to return. After an SBT function is returned from the media manager, this row disappears.
A row in V$SESSION_WAIT
corresponding to an SBT event name does not indicate a problem, because the server updates these rows at run time. The rows appear and disappear as calls are made and returned. However, if the SECONDS_IN_WAIT
column is high, then the media manager may be suspended.
To monitor the SBT events, you can run the following SQL query:
COLUMN EVENT FORMAT a17 COLUMN SECONDS_IN_WAIT FORMAT 999 COLUMN STATE FORMAT a15 COLUMN CLIENT_INFO FORMAT a30 SELECT p.SPID, s.EVENT, s.SECONDS_IN_WAIT AS SEC_WAIT, sw.STATE, s.CLIENT_INFO FROM V$SESSION_WAIT sw, V$SESSION s, V$PROCESS p WHERE sw.EVENT LIKE '%MML%' AND s.SID=sw.SID AND s.PADDR=p.ADDR;
Examine the SQL output to determine which SBT functions are waiting. For example, the following output indicates that RMAN has been waiting for the sbtbackup
function to return for 10 minutes:
SPID EVENT SEC_WAIT STATE CLIENT_INFO ---- ----------------- ---------- --------------- ------------------------------ 8642 Backup: MML creat 600 WAITING rman channel=ORA_SBT_TAPE_1
Note:
The V$SESSION_WAIT
view shows only database events, not media manager events.
See Also:
Oracle Database
Reference for descriptions of the V$SESSION_WAIT
view.
23.2.2 Correlating Server Sessions with RMAN Channels
To identify which server sessions correspond to which RMAN channels, you can query V$SESSION
and V$PROCESS
.
The SPID
column of V$PROCESS
identifies the operating system ID number for the process or thread. For example, on UNIX the SPID
column shows the process ID, whereas on Windows the SPID
column shows the thread ID. You have two basic methods for obtaining this information, depending on whether you have multiple RMAN sessions active concurrently.
This section contains the following topics:
23.2.2.1 Matching Server Sessions with Channels When One RMAN Session Is Active
When only one RMAN session is active, the easiest method for determining the server session ID for an RMAN channel is to query the target database.
Run the following query on the target database while the RMAN job is executing:
COLUMN CLIENT_INFO FORMAT a30 COLUMN SID FORMAT 999 COLUMN SPID FORMAT 9999 SELECT s.SID, p.SPID, s.CLIENT_INFO FROM V$PROCESS p, V$SESSION s WHERE p.ADDR = s.PADDR AND CLIENT_INFO LIKE 'rman%';
The following shows sample output:
SID SPID CLIENT_INFO ---- ------------ ------------------------------ 14 8374 rman channel=ORA_SBT_TAPE_1
If you set an ID using the RMAN SET COMMAND ID
command instead of using the system-generated default ID, then search for that value in the CLIENT_INFO
column instead of 'rman%'
.
23.2.2.2 Matching Server Sessions with Channels in Multiple RMAN Sessions
If multiple RMAN sessions are active, then the V$SESSION.CLIENT_INFO
column can yield the same information for a channel in each session.
For example:
SID SPID CLIENT_INFO ---- ------------ ------------------------------ 14 8374 rman channel=ORA_SBT_TAPE_1 9 8642 rman channel=ORA_SBT_TAPE_1
In this case, you have the following methods for determining which channel corresponds to which SID
value.
23.2.2.2.1 Obtaining the Channel ID from the RMAN Output
You must first obtain the sid
values from the RMAN output and then use these values in your SQL query.
To correlate a process with a channel during a backup:
23.2.2.2.2 Correlating Server Sessions with Channels by Using SET COMMAND ID
You specify a command ID string in the RMAN backup script. You can then query V$SESSION.CLIENT_INFO
for this string.
To correlate a process with a channel during a backup:
See Also:
Oracle Database Backup and
Recovery Reference for SET COMMAND ID
syntax, and Oracle Database
Reference for more information about V$SESSION
and V$PROCESS
23.3 Testing the Media Management API
On some platforms, Oracle provides a diagnostic tool called sbttest
. This utility performs a simple test of the media management software by acting as the Oracle database server and attempting to communicate with the media manager.
This section contains the following topics:
23.3.1 Obtaining the sbttest Utility
The default location of the sbttest
utility depends on the platform.
On UNIX, the sbttest
utility is typically located in $ORACLE_HOME/bin
. If for some reason the utility is not included with your platform, then contact Oracle Support Services to obtain the C version of the program. You can compile this version of the program on all UNIX platforms.
On platforms such as Solaris, you do not have to relink when using sbttest
. On other platforms, relinking may be necessary.
23.3.2 Obtaining Online Documentation for the sbttest Utility
Use the sbttest
command, without arguments, to list the various arguments for this program.
For online documentation of sbttest
, issue the following on the command line:
% sbttest
The program displays the list of possible arguments for the program:
Error: backup file name must be specified Usage: sbttest backup_file_name # this is the only required parameter <-dbname database_name> <-trace trace_file_name> <-remove_before> <-no_remove_after> <-read_only> <-no_regular_backup_restore> <-no_proxy_backup> <-no_proxy_restore> <-file_type n> <-copy_number n> <-media_pool n> <-os_res_size n> <-pl_res_size n> <-block_size block_size> <-block_count block_count> <-proxy_file os_file_name bk_file_name [os_res_size pl_res_size block_size block_count]> <-libname sbt_library_name>
The display also indicates the meaning of each argument. For example, following is the description for two optional parameters:
Optional parameters: -dbname specifies the database name which will be used by SBT to identify the backup file. The default is "sbtdb" -trace specifies the name of a file where the Media Management software will write diagnostic messages.
23.3.3 Using the sbttest Utility
Use sbttest
to perform a quick test of the media manager.
If sbttest
returns 0, then the test ran without error, which means that the media manager is correctly installed and can accept a data stream and return the same data when requested. If sbttest
returns a nonzero value, then either the media manager is not installed or it is not configured correctly.
To use sbttest:
In some cases, sbttest
can work but an RMAN backup does not. The reasons can be the following:
-
The user who starts
sbttest
is not the owner of the Oracle Database processes. -
If the database server is not linked with the media management library or cannot load it dynamically when needed, then RMAN backups to the media manager fail, but
sbttest
may still work. -
The
sbttest
program passes all environment parameters from the shell but RMAN does not.
23.4 Terminating an RMAN Command
There are several ways to terminate an RMAN command in the middle of execution.
They include the following:
-
The preferred method is to press Ctrl+C (or the equivalent "attention" key combination for your system) in the RMAN interface. This also terminates allocated channels, unless they are suspended in the media management code, as happens when, for example, they are waiting for a tape to be mounted.
-
You can end the server session corresponding to the RMAN channel by running the SQL
ALTER SYSTEM KILL SESSION
statement as described in Terminating the Session with ALTER SYSTEM KILL SESSION. -
You can terminate the server session corresponding to the RMAN channel on the operating system as described in Terminating the Session at the Operating System Level.
23.4.1 Terminating the Session with ALTER SYSTEM KILL SESSION
To terminate an RMAN session by using the ALTER SYSTEM statement, you need the Oracle session ID for the RMAN channel and the serial number. This information is contained in the RMAN log for messages.
Search for messages with the format shown in the following example:
channel ch1: sid=15 devtype=SBT_TAPE
The sid
and devtype
are displayed for each allocated channel. The Oracle Database sid
is different from the operating system process ID. You can end the session using a SQL ALTER
SYSTEM
KILL
SESSION
statement.
ALTER SYSTEM KILL SESSION
takes two arguments, the sid
printed in the RMAN message and a serial number, both of which can be obtained by querying V$SESSION
.
For example, run the following statement, where sid_in_rman_output
is the number from the RMAN message:
SELECT SERIAL#
FROM V$SESSION
WHERE SID=sid_in_rman_output
;
Then, run the following statement, substituting the sid_in_rman_output
and serial number obtained from the query:
ALTER SYSTEM KILL SESSION 'sid_in_rman_output
,serial#
';
This statement has no effect on the session if the session stopped in media manager code.
23.4.2 Terminating the Session at the Operating System Level
Finding and terminating the processes that are associated with the server sessions is operating system-specific. On some platforms, the server sessions are not associated with any processes at all. See your operating system-specific documentation for more information.
23.4.3 Terminating an RMAN Session That Is Not Responding in the Media Manager
You may sometimes need to terminate an RMAN job that is not responding in the media manager. The best way to terminate RMAN when the channel connections are not responding in the media manager is to terminate the session in the media manager.
If this action does not solve the problem, then on some platforms, such as Linux, you may be able to terminate the Oracle Database processes of the connections. (Terminating the Oracle processes may cause problems with the media manager. See your media manager documentation for details.)
23.4.3.1 Components of an RMAN Session
The nature of an RMAN session depends on the operating system.
In UNIX, an RMAN session has the following processes associated with it:
-
The RMAN client process itself
-
The default channel, the initial connection to the target database
-
One target connection to the target database corresponding to each allocated channel
-
The catalog connection to the recovery catalog database, if you use a recovery catalog
-
An auxiliary connection to an auxiliary instance, during
DUPLICATE
or TSPITR operations -
A polling connection to the target database, used for monitoring RMAN command execution on the various allocated channels. By default, RMAN makes one polling connection. RMAN makes additional polling connections if you use different connect strings in the
ALLOCATE CHANNEL
orCONFIGURE CHANNEL
commands. One polling connection exists for each distinct connect string used in theALLOCATE CHANNEL
orCONFIGURE CHANNEL
command.
23.4.3.2 Process Behavior During a Suspended Job
RMAN usually stops responding because a channel connection is waiting in the media manager code for a tape resource. The catalog connection and the default channel appear to suspend, because they are waiting for RMAN to tell them what to do. Polling connections seem to be in an infinite loop while polling the RPC under the control of the RMAN process.
If you terminate the RMAN process itself, then you also terminate the catalog connection, the auxiliary connection, the default channel, and the polling connections. If target and auxiliary connections are suspended but not while executing media manager code, they also terminate. If either the target connection or any of the auxiliary connections are executing in the media management layer, then they do not terminate until the processes are manually terminated at the operating system level.
Not all media managers can detect the termination of the Oracle Database process. Those which cannot may keep resources busy or continue processing. Consult your media manager documentation for details.
Terminating the catalog connection does not cause the RMAN process to terminate because RMAN is not performing catalog operations while the backup or restore is in progress. Removing default channel and polling connections causes the RMAN process to detect that a channel is no longer present and then to exit. In this case, the connections to the unresponsive channels remain active as described previously.
23.4.3.3 Terminating an RMAN Session: Basic Steps
After the unresponsive channels in the media manager code are terminated, the RMAN process detects this termination and exits, removing all connections except target connections that are still operative in the media management layer.
The warning about the media manager resources still applies in this case.
To terminate an Oracle Database process that is not responding in the media manager: