11 Oracle Secure Backup Catalog Recovery

The computers we use to back up our data can themselves fail. To guard against such failures Oracle Secure Backup protects its own catalog and settings data, without which all the other backups it has performed are just so many assorted tapes. If the catalog data is lost, then Oracle Secure Backup can restore these data to the state they were in before the failure.

When Oracle Secure Backup is first installed on your administrative server, a scheduled job is set up by the installer to back up the catalog data every day at midnight.

Oracle Secure Backup catalog recovery does not impose any burden on the backup administrator unless there is an actual failure. No configuration is required, but Oracle Secure Backup catalog recovery can be extended and customized by the backup administrator.

Oracle Secure Backup catalog recovery protects only the catalog and settings on an Oracle Secure Backup administrative server. The operating system and other installed software are not automatically backed up.

This chapter contains these sections:

Catalog Recovery Concepts
Catalog Backup Jobs
Catalog Recovery Procedure

Catalog Recovery Concepts

Oracle Secure Backup catalog recovery creates four reserved objects:

Catalog Recovery Schedule Object
Catalog Recovery Media Family Object
Catalog Recovery Dataset Object
Catalog Recovery Summary Object
Modifying Catalog Recovery Objects

These reserved objects cannot be deleted, and some of their properties cannot be changed. The idea is to prevent catalog backups from being disabled or given settings which make them useless by accident.

Catalog Recovery Schedule Object

This object drives the catalog recovery backup process. It is associated with a catalog recovery dataset object, which specifies the data to be backed up, and a catalog recovery media family object, which specifies the characteristics of the resulting tape volume.

The catalog recovery schedule object is created by the Oracle Secure Backup installer to perform a full backup at midnight each day. The priority is set at 50, rather than the default 100. A suitably-privileged Oracle Secure Backup user can:

Add or remove a trigger
Modify the priority
Change tape drive restrictions
Add or remove comments

Catalog backups can be disabled by removing the trigger from this reserved schedule object. The associated dataset cannot be changed, and only non-encrypted full backups are permitted. An incremental backup of the catalog data is disallowed. It adds complexity during the restore process, which must be kept simple because it is done in the absence of catalog data.

Note:

A backup using an automatically generated encryption key would be useless without the on-disk key store, which would be lost if the administrative server were destroyed.

Catalog Recovery Media Family Object

A catalog recovery media family object describes the tape volumes which result from a catalog recovery backup. The Oracle Secure Backup installer creates a catalog recovery media family object with a write window of 7 days, and a retention period of 14 days. Oracle recommends rotating backups across two volume sets.

A suitably privileged Oracle Secure Backup user can:

Alter the write window
Alter the retention time
Modify the volume ID generation parameters
Modify volume duplication attributes
Associate a rotation policy
Add or remove comments

The catalog recovery media family object must have a time-managed expiration policy. Oracle Secure Backup does not allow the catalog recovery media family object to be content-managed, because backups of file system data cannot be content-managed.

Catalog Recovery Dataset Object

A catalog recovery dataset object specifies what data is to be backed up. It incorporates an include catalog dataset directive to specify catalog data. This directive is expanded by Oracle Secure Backup to a definition of all files and databases that must be included in a catalog recovery backup. The catalog data itself is always backed up without storage encryption, regardless of the encryption policy.

Other files and hosts can be added to the catalog recovery dataset object. To add files and paths on the administrative server to the catalog backup, enclose them within block delimiters beneath the include catalog directive in a dataset. You can add the following directives to an include catalog block:

include path
exclude path
exclude name

No other directives are allowed within the include catalog block. The following example directive would cause the files in /usr/local/bin on the administrative host to be included in every catalog backup:

include catalog {
    include path "/usr/local/bin"
}

Note:

The include catalog directive cannot be added within an include host block, because it implicitly applies only to the administrative server. The dataset parser will report an error in this case.

You can add the include catalog directive to other datasets as well. There is no restriction on what else might be backed up by a database that includes it. The expanded catalog directive and its children, however, are handled as a separate job by the scheduler.

A suitably-privileged Oracle Secure Backup user can modify the catalog recovery dataset object using the standard dataset language. But Oracle Secure Backup does not allow you to remove the include catalog directive from the catalog recovery dataset object.

Catalog Recovery Summary Object

A catalog recovery summary object causes Oracle Secure Backup to generate a summary report detailing each backup operation within the last 24 hours. This summary report is generated with a --catalog option which causes Oracle Secure Backup to include extended information about catalog recovery backups. When a summary report is generated with the --catalog option, Oracle Secure Backup also checks for catalog backup failures and generates an e-mail to the backup administrator if any are found.

Note:

The Oracle Secure Backup installer asks for the e-mail address of the admin user. On Windows, the installer also asks for an e-mail server. If no e-mail address is specified, or if no e-mail server is specified on Windows, then e-mail notifications are not sent.

A report generated with the --catalog option set includes:

The volume ID and barcode for the catalog backup
The file number for the catalog backup
Results of the verification step

Catalog backups also appear in summary reports that include information on each backup job, but they are not flagged as catalog backups, and will be mixed with the other backup jobs. The --catalog option is intended to make it easier for a backup administrator to check the status of catalog backups separately from other backup jobs.

Modifying Catalog Recovery Objects

All reserved catalog recovery objects are just instances of the usual Oracle Secure Backup objects with some added restrictions. The restrictions are meant to prevent you from accidentally disabling the catalog backup or changing the backup settings to something that does not perform correctly.

To modify catalog recovery objects, you can use obtool commands chsched, chmf, chsum, and edds. You can also use the Oracle Secure Backup Web tool or Oracle Enterprise Manager equivalents. The interface does not allow some things to be changed, but for everything else the reserved objects act just like normal objects.

Catalog Backup Jobs

Catalog recovery backup jobs always include a catalog backup, and they can include other files as well. Catalog backup jobs use the include catalog dataset extension to specify that all catalog data for the administrative server is included in the backup. Every catalog backup job is a full backup. Oracle Secure Backup is configured on installation to perform regular catalog backup jobs.

Storage encryption is disabled for all catalog backup jobs. You cannot recover encrypted backup data without the encryption wallet. But in a disaster scenario the encryption wallet would be lost, because it is part of the catalog data. So if the catalog backup data were encrypted, there would be no way to decipher it. Catalog backups can use transient passphrase encryption, because this does not require a wallet. Transient passphrase encryption is not enabled for catalog backup by default, but it can be added in the usual way.

Catalog Recovery Procedure

This section describes the basic procedure for restoring the admin directory in the event of media failure or loss of the administrative server. Catalog restore operations are needed when some grave misfortune has befallen the catalog on the administrative server.

Oracle highly recommends that you maintain a record of Oracle Secure Backup device attachments (especially for the devices you intend to use in the case of disaster recovery), because it will be invaluable when reinstalling Oracle Secure Backup in the event of a disaster. Otherwise you must add/install/configure all media servers and their tape devices in the new administrative domain, and then examine the volumes in your inventory one-by-one to locate the required Oracle Secure Backup catalog backup.

The best way to prepare for a catalog recovery emergency is to:

Retain a copy of your obtool lsd -l output.
Note the attachment information listed.
Retain a copy of your most recent e-mail of the job summary report for a catalog backup. The job summary for a catalog backup provides the information required to identify the volume and file number that holds the latest catalog backup.

To restore your Oracle Secure Backup administrative server:

Perform a fresh install of Oracle Secure Backup on the administrative server.

Note:
To complete the process of restoring the original administrative server, you must have the password that was used to create the original encryption wallet. The encryption wallet created during the fresh install of Oracle Secure Backup is used only during the restore of the original administrative server. It will be replaced with the original encryption wallet.

See Also:
Oracle Secure Backup Installation and Configuration Guide for instructions on installing Oracle Secure Backup on Windows and Linux or UNIX
Add a media server to your newly created administrative domain and re-create your tape devices.

If your administrative server does not have an attached tape device, then you must add a remote media server host from the previous Oracle Secure Backup administrative domain to this new administrative domain. This media server host should also have access to the library containing the most recent catalog backup and a tape drive. Alternatively, if you can locate the volume by its label, then the volume can be manually loaded into the most convenient tape drive. In either case, you must configure a tape device through the Oracle Secure Backup Web tool or obtool to be used for restoring the catalog data.

If you do not have a media server local to the administrative host and need to add a remote media server that was part of the Oracle Secure Backup administrative domain before the administrative server failed in order to perform disaster recovery, then Oracle Secure Backup will try to add it with a different Universal Unique Identifier (UUID). The new UUID will cause media server operations to fail, because it will not match the identity certificate residing on the media server.

The workaround is to decertify the media server by running obcm decertify on the media server (as root) before running the obtool mkhost command on the administrative server. The obcm utility is located in $OSB_HOME/bin.

You must repeat this step after the catalog is restored, and then you must do the following on the Oracle Secure Backup administrative server to recertify the remote media server and complete the process:
```
cd OSB_HOME/admin/history/host
mv media_server media_server_save
ob> rmhost --nocomm media_server
remove host media_server? (a, n, q, y, ?) [n]: y
ob> mkhost media_server 
Info: waiting for host to update certification status... 
Info: waiting for host to update certification status... 
ob> lsh -l media_server
media_server: 
    Access mode:            OB 
    TCP/IP buffer size:     not set (global policy) 
    Algorithm:              aes192 
    Encryption policy:      allowed 
    Rekey frequency:        1 month (system default) 
    Key type:               transparent 
    In service:             yes 
    Roles:                  client 
    Trusted host:           no 
    Certificate key size:   1024 
    UUID:                   50d97a76-0500-102b-a4bb-080020a0a249
ob> pinghost media_server
media_server:     Oracle Secure Backup and NDMP services are available 
```
After you have recertified the remote media server, you can move the saved catalog directory back into place:
```
cd OSB_HOME/admin/history/host
mv media_server_save media_server
```
This first example configures a tape library and tape drive attached to the administrative host jfersten-sun:
```
jfersten-sun# obtool
Oracle Secure Backup 10.2.0.2.0.
login: admin
ob> chhost --addrole mediaserver jfersten-sun
ob> mkdev -t library -a jfersten-sun:/dev/obl0 jflib
ob> mkd -t tape -a jfersten-sun:/dev/obt0 -d 1 -l jflib jftape
ob> lshost
 jfersten-sun        admin,mediaserver,client (via OB)   in service
```
This second example configures a remote media server and its devices from the administrative host jfersten-sun:
```
ob> mkhost storabck04
Info: waiting for host to update certification status...
ob> chhost --addrole mediaserver storabck04
ob> mkdev -t lib --attach storabck04:/dev/sg4 stbk4lib
ob> mkdev -t tape -l stbk4lib -d 1 --attach storabck04:/dev/sg5 stbk4tape
```
Note:
Before running mkhost on the Oracle Secure Backup administrative server, be sure to run $OSB_HOME/bin/obcm decertify on the remote media server.

Locate the volume that contains your most recent OSB_CATALOG backup and load it into the drive. It will be helpful if you have saved a record of your catalog backups so that the tape is easier to locate. The job summary for a catalog backup provides the volume ID, bar code, and file number for the catalog backup. If you do not have this information but know that the volume is in your library, then run obtool inventory and identifyvol commands on the library.

The following example shows a job summary for a catalog backup:

Job ID      Scheduled At        Completed At       Content
  Backup Size  File #   Volume ID (Bar Code)
admin/1.1   2008/03/26.11:48    2008/03/26.11:49   *catalog jfersten-sun
  455 KB         1      OSB-CATALOG-MF-000002 (e744f09c4eeb4dabf3ac02ae2d332c0)

Note:

It will be necessary to run an initial inventory on the library before using it for the first time even if you know which volume contains your OSB_CATALOG backup.

ob> inv -L jflib
ob> lsvol -L jflib
Inventory of library jflib:
    in    3:             occupied
    in    4:             unlabeled
    in    5:             unlabeled
    in    6:             unlabeled
    in    7:             unlabeled
    in    8:             unlabeled
    in    9:             unlabeled

Run the identifyvol command only if you do not know which volume contains your catalog backup. If you do know which volume contains your catalog backup, then simply load that volume into the drive and skip to the obtar command shown in step 5.

ob> identifyvol --import -D jftape 3-9

Seq       Volume              Volume    Archive     Client      Backup  
 #         ID                  Tag     File Sect     Host        Level
 1    OSB-CATALOG-MF-000002              1   1    jfersten-sun     0      
Archive Create 
 Date & Time
2008/03/23 10:39:54s
 
ob> lsvol -L jflib
Inventory of library jflib:
   in    3:             volume OSB-CATALOG-MF-000002, 6891336 kb remaining, expires 2008/04/13.10:39

If you are restoring a Linux or UNIX Oracle Secure Backup administrative server, then go to steps 5 and 6.

If you are restoring a Windows administrative server, then the procedure described in steps 5 and 6 for extracting the admin and db directories is not applicable. Use the procedure described in steps 7, 8, and 9 instead.

This step applies only to Linux and UNIX administrative servers.

Load your OSB-CATALOG-MF backup volume into the drive and re-create an Oracle Secure Backup catalog from the volume using the obtar -Gtf drive_name -F file_number command:

# obtar -Gtf jftape -F 1
Volume label:
    Volume UUID:        4cfe75d8-db47-102a-9d77-080020a0a249
    Volume ID:          OSB-CATALOG-MF-000002
    Volume sequence:    1
    Volume set owner:   root
    Volume set created: Sun Mar 23 10:39:54 2008
    Volume set closes:  Sun Mar 30 10:39:54 2008 (no writes after this time)
    Volume set expires: Sun Apr 13 11:39:54 2008
    Media family:       OSB-CATALOG-MF
    Original UUID:      4cfe75d8-db47-102a-9d77-080020a0a249
Archive label:
    File number:        1
    File section:       1
    Owner:              root
    Client host:        jfersten-sun
    Backup level:       0
    S/w compression:    no
    Archive created:    Sun Mar 23 10:39:54 2008
    Encryption:         off

Note:

There might be more than one file on the volume. Be sure you are using the correct file number in your restore. In the preceding example, there is only one and it corresponds to the File Sect listed. The file number information can also be found in your catalog summary report. Use this file number with the -F switch in the obtar -Gtf command.

This step applies only to Linux and UNIX administrative servers.

Use either obtool or the Oracle Secure Backup Web tool to browse the newly re-created catalog and restore the items to an alternate directory:
```
ob> set host jfersten-sun
ob> cd /usr/local/oracle/backup
ob> restore admin --aspath /tmp/admin.restored
ob> cd /usr/etc/
ob> restore ob --aspath /tmp/ob.restored
ob> restore -go
Info: 2 catalog restore request items submitted; job id is admin/45.
```
Note:
You absolutely must restore the catalog backup to an alternate directory. Restoring the catalog backup to its original location will leave Oracle Secure Backup configuration in an inconsistent state.
This step applies only to Windows administrative servers.

Load your OSB_CATALOG-MF backup volume into the drive. Ensure you have the correct catalog recovery volume and file number by indexing the tape with obtar, checking the output to confirm that your Oracle Secure Backup catalog data is present on the tape, and checking that you have the correct file number:
```
C:\ >obtar -tf drive_name -F 1 > output_file

C:\ >more output_file
C:/osb/backup/admin/
C:/osb/backup/admin/config/
C:/osb/backup/admin/config/class/
C:/osb/backup/admin/config/class/admin
.
.
.
C:/osb/backup/db/xcr/syssbt@98.1
C:/osb/backup/db/xcr/syssbt@99.1
```
This step applies only to Windows administrative servers.

Extract the backup directory to an alternate location:
```
C:\>obtar -xf tape_device -F 1 -s,C:/osb/backup,C:/osb/backup-restored,
```
Caution:
The path substitution syntax for obtar (-s ,P,R,) replaces prefix P with string R. It is crucial that you use a substitution path and restore your catalog recovery files to an alternate directory to avoid overwriting the existing files in your current admin directory. Overwriting the existing files would leave Oracle Secure Backup in an indeterminate state and render it unusable.

This step applies only to Windows administrative servers.

Confirm that your Catalog Recovery files have been properly restored:

C:\>cd osb 
 
C:\osb>ls 
backup           backup-restored 
 
C:\osb>cd backup-restored 
 
C:\osb\backup-restored>ls 
admin  db

Terminate the observiced process.
1. On Linux or UNIX, run the following command:
```
# /etc/init.d/observiced stop
```
  Confirm that all ob processes have stopped by running the following command:
```
# ps -auwx | grep ob
```
2. On Windows, run the following command:
```
net stop observiced
```
Move the newly restored admin and ob directories into place and re-create the obfuscated wallet before restarting observiced. Although the restore process also restores the password-protected encryption wallet to the administrative server, for security reasons the obfuscated wallet is not backed up. You must re-create it manually after a restore operation. You must re-create the wallet with the password that was used to create the original encryption wallet. To manually re-create the wallet use the mkow command.

Note:
You must know your original keystore password in order to accomplish this task.
1. On Linux or UNIX:
```
cd /usr/local/oracle/backup
mv admin admin.orig
mv /tmp/admin.restored admin
cd /usr/etc
mv ob ob.orig
mv /tmp/ob.restored ob
$OSB_HOME/bin/obcm mkow -k  -p wallet_password
Obfuscated wallet has been re-created
/etc/init.d/observiced -start
```
  Note:
  The default OSB_HOME is /usr/local/oracle/backup.
2. On Windows:
```
C:\Program Files\Oracle\Backup> ren admin admin.orig
        C:\Program Files\Oracle\Backup> move C:\tmp\admin .
        C:\Program Files\Oracle\Backup> ren db db.orig
        C:\Program Files\Oracle\Backup> move C:\tmp\db 
        C:\Program Files\Oracle\Backup\bin\obcm mkow -k -p wallet_password
        net start observiced
```
The Oracle Secure Backup Web tool is not going to work after the catalog restore operation, and the apache password must be reset. Run the following command to reset the apache webpass:
```
obtool setp daemons/webpass default
```
Stop and restart the Oracle Secure Backup daemons.

Your environment is now restored to the point of your last backup of the admin directory. You can use obtool or the Oracle Secure Backup Web tool to review your administrative domain configuration.