This chapter describes basic troubleshooting methods and includes the following information:
This section describes possible Metadata Database failures and resolutions.
The typical DIVArchive Metadata Database backup configuration backs up the database and transfers the backup files to remote systems (as defined in the configuration) every 15 minutes. Oracle recommends having at least two remote backup systems for redundancy.
The following are examples of possible failure scenarios. Each scenario includes the method of detection, the cause of the failure, a description of the failure, and recovery procedures. Contact Oracle Support if you require additional assistance to resolve any of these issues.
You can identify a disk failure on the Main Manager because no more complex objects can be archived into the DIVArchive system. Only Delete requests are possible on existing complex objects. DIVArchive is still operational for archiving non-complex objects.
New Metadata files created for complex objects archived since the last successful backup, up until the disk failure, are not available immediately. However, they can be recovered from the AXF file.
You can identify a disk failure on one of the backup systems because the Metadata Database files created by a new Archive request since the disk failure are backed up only to one backup system, instead of all identified backup systems.
The method of detection for this failure is that a complex object request fails with the error Internal error: metadata database error. A Metadata Database Backup Failure notification is displayed on the Control GUI, and the backup failure events are logged in the Manager Event Log.
The possible causes of this failure include the following:
RAID controller failures
Power surges
External process errors
Disk volume reconstruction error if the RAID was previously rebuilt
Even though Oracle recommends storing the Metadata Database on a RAID disk, disk failure scenarios cannot be totally eradicated, and the unlikely chance of Disk Failure still exist.
Use the following procedure to attempt recovery from disk failure on the Main Manager:
Stop the Manager and Backup Service.
Replace the failed disk.
Navigate to the Manager Setting tab in the Configuration Utility and confirm that the Metadata Database Location setting is pointing to the replaced disk.
Start the Manager and Backup Service.
Copy all of the Metadata files from a backup system to the Metadata Database Location on the replaced disk.
Execute the dbbackup reconcile command to confirm no complex objects are lost.
The Metadata files of complex objects archived since the last successful backup, and before the disk failure, are not immediately available. However, they are recoverable from the AXF file. Recovery from AXF files is not supported in this DIVArchive release; contact Oracle Support for assistance.
Use the following procedure to attempt recovery from disk failure on one of the backup systems. The system can be operational if the backups made to other backup systems were successful.
Replace the failed disk.
Copy all Metadata files from the second Backup System and Main Manager System to the folder identified in the Metadata Database Location on the replaced disk.
No operations or requests are possible on complex objects whose Metadata files are corrupted, except Delete Object requests, until it is restored. A Metadata file modified by any external source (other than DIVArchive) after it is backed up will not affect its backup copies in the backup systems.
You can identify when a Metadata Database file becomes corrupted because complex object requests fail with the following error:
Internal error: metadata database error: Message: Metadata file read error.
The possible causes of this failure include the following:
External process errors
The file is modified manually by mistake
Use the following procedure to attempt recovery from a corrupt Metadata Database file. If the corruption occurred after the Metadata file is backed up, the Metadata file can be restored from one of the backups servers.
Execute the FindMetadataFile.bat utility located in the %DIVA_HOME%/programs/utilities/bin folder on the Main Manager System.
This utility prints out the location of the Metadata file with its file name inside the specified Metadata Database Location, and accepts the database connection parameters and the complex object name and category as parameters.
Locate the file with the file name and path printed from the utility in the Metadata Database backup location on one of the backup servers.
Replace the Metadata file on the Main Manager System in the configured Metadata Database Location with the copy from the backup server.
If the corruption occurred before the Metadata file was backed up, the Metadata file is not immediately available. However, it is recoverable from the AXF file. Recovery from AXF files is not supported in this DIVArchive release; contact Oracle Support for assistance.
Metadata deleted by any external source other than DIVArchive after it is successfully backed up does not affect its backup copies on the backup systems.
You cannot perform any operations or requests on complex objects whose Metadata file is corrupt, except Delete Object, until the Metadata file is restored.
You can identify when a Metadata Database file is lost or deleted because complex object requests fail with the following error message:
Internal error: metadata database error: Message: get: Error opening metadata for objectname/category, db error=Error file not found.
The possible causes of this failure include the following:
External process errors
The file was manually deleted by mistake
If the file is lost after the Metadata File is backed up, the Metadata File can be restored from one of the Backup Servers. Use the following process to attempt recovery from a lost or deleted Metadata Database file:
Execute the FindMetadataFile.bat utility located in the %DIVA_HOME%/programs/utilities/bin folder on the Main Manager system.
This utility prints out the location of the Metadata file with its file name inside the specified Metadata Database Location, and accepts the database connection parameters and the complex object name and category as parameters.
Locate the file with the file name and path printed from the utility in the Metadata Database backup location on one of the backup servers.
Replace the Metadata file on the Main Manager System in the configured Metadata Database Location with the copy from the backup server.
If the file was lost before the Metadata file was backed up, the Metadata file is not immediately available. However, it is recoverable from the AXF file. Recovery from AXF files is not supported in this DIVArchive release; contact Oracle Support for assistance.
Failure to back up the Metadata Database to all backup systems results in all complex objects archived after this failure not being backed up. You must resolve this failure as soon as possible because the DIVArchive system is at risk of data loss.
You can identify this error when a Metadata Database Backup Failure notification is displayed on the Control GUI. The backup failure events are also logged in the Manager Event Log.
The possible causes of this error are as follows:
Network errors
The backup systems are offline
The RSYNC daemon is offline
The Backup Service has failed
Use the following referenced resolutions to attempt correction of this issue:
Resolve the network error.
Start, or restart, the Backup System.
Start, or restart, the RSYNC daemon.
Restart the Backup Service and collect the logs for investigation.
After the problem is resolved, all of the Backup Systems sync automatically, and the missing Metadata files are backed up during the process. There is no data recovery required for this scenario.
In this scenario, the Metadata Database fails to back up to (only) one of the Backup Systems. However, the back ups to other Backup Systems continue successfully.
You can identify this error when a Metadata Database Backup Failure notification is displayed on the Control GUI. The backup failure events are also logged in the Manager Event Log.
The possible causes of this error are as follows:
Network errors
The Backup System where the error occurred is offline
Use the following referenced resolutions to attempt correction of this issue:
Resolve the network error.
Start, or restart, the Backup System.
After the problem is resolved, all of the Backup Systems sync automatically, and the missing Metadata files are backed up during the process. There is no data recovery required for this scenario.
When the Manager starts it checks the following parameters. The Manager will not start if any combination of these parameters is incorrect. Confirm the Enable Metadata Database parameter is configured correctly, and the Metadata Database Path is a valid path that is not empty.
The DIVArchive Backup Service is designed to terminate execution immediately after attempting to start if it is configured incorrectly. This behavior can be caused by any of the following reasons:
The configuration file is missing.
The database connection information is incorrect, or the database is not running.
The BACKUP_SERVICE_MANAGE_METADATA_BACKUPS parameter is set to Y (Yes, or enabled) in the Configuration file, but not enabled under the Manager Settings panel in the Configuration Utility.
The BACKUP_SERVICE_MANAGE_METADATA_BACKUPS parameter is set to Y (Yes, or enabled) in the Configuration file, but the Metadata Database Location is not set, or set to an invalid directory under the Manager Settings panel in the Configuration Utility.
The BACKUP_SERVICE_MANAGE_METADATA_BACKUPS parameter is set to Y (Yes, or enabled) in the Configuration file, and the Metadata Database Backup is enabled under the Manager Settings panel in the Configuration Utility, but the Metadata Database Location is not set, or set to an invalid directory.
BACKUP_SERVICE_MANAGE_DATABASE_BACKUPS and BACKUP_SERVICE_MANAGE_METADATA_BACKUPS parameters are set to N (No, or disabled) in the Configuration file.
RMANRecoverWindow.bat is not in the bin folder for the Backup Service.