1. Oracle Globally Distributed Database
  2. Oracle Globally Distributed Database Administration
  3. Backing Up and Recovering Oracle Globally Distributed Database

Backing Up and Recovering Oracle Globally Distributed Database

The GDSCTL utility lets you define a backup policy for an Oracle Globally Distributed Database and restore one or more shards, or the entire Oracle Globally Distributed Database, to the same point in time. Configured backups are run automatically, and you can define a schedule to run backups during off-peak hours.

GDSCTL commands in Oracle Globally Distributed Database enable and simplify the centralized management of backup policies for a sharded database, using Oracle MAA best practices. You can create a backup schedule using an incremental scheme that leverages the Oracle Job Scheduler. Oracle Recovery Manager (RMAN) performs the actual backup and restore operations.

Using the GDSCTL centralized backup and restore operations, you can

Backup and Restoration Terminology

The following are some terms you will encounter in the Oracle Globally Distributed Database backup and restore procedures.

  • Target database - A database RMAN is to back up.

  • Global SCN - A common point in time for all target databases for which a restore of the entire sharded database is supported. A restore point is taken at this global SCN, and the restore point is the point to which the sharded database (including the shard catalog) can be restored.

    Note that you are not prohibited from restoring the shard catalog or a specific shard to an arbitrary point in time. However, doing so may put that target in an inconsistent state with the rest of the sharded database and you may need to take corrective action outside of the restore operation.

  • Incremental backup - Captures block-level changes to a database made after a previous incremental backup.

  • Level 0 incremental backup (level 0 backup) - The incremental backup strategy starting point, which backs up blocks in the database. This backup is identical in content to a full backup; however, unlike a full backup, the level 0 backup is considered a part of the incremental backup strategy.

  • Level 1 incremental backup (level 1 backup) - A level 1 incremental backup contains only blocks changed after a previous incremental backup. If no level 0 backup exists in either the current or parent database incarnation and you run a level 1 backup, then RMAN takes a level 0 backup automatically. A level 1 incremental backup can be either cumulative or differential.

Automated and On-Demand Backups

There are two type of backups in Oracle Globally Distributed Database: automated backups and on-demand backups.

  • Automated backups are started by Oracle Scheduler jobs based on the job schedules, and they run in the background on the database servers.

  • On-demand backups are started by users from GDSCTL.

    Internally, the on-demand backups are also started by Oracle Scheduler jobs on the database servers. The jobs are created on-fly when the on-demand backup commands are issued. They are temporary jobs and automatically dropped after the backups have finished.

See also: Scheduling Jobs with Oracle Scheduler

Supported Backup Destinations

The following are supported backup destinations for Oracle Globally Distributed Database.

  • Common disk/directory structure (NFS mount) which can be located anywhere, including the shard catalog database host.

  • Zero Data Loss Recovery Appliance (advantage is continuous backup, can leverage Data Guard broker to manage and monitor redo transport)

  • Oracle Object Storage

  • Amazon S3 (Amazon Simple Storage Service)

Limitations

Note the following limitations for Oracle Globally Distributed Database backup and restore using GDSCTL.

  • Microsoft Windows is not supported.

  • You must provide for backup of Clusterware Repository if Clusterware is deployed

Prerequisites to Configuring Centralized Backup and Restore

Before configuring backup for an Oracle Globally Distributed Database, make sure the following prerequisites are met.

  • Create one or more recovery catalogs in a dedicated database. If Recovery Appliance is used for the backup, the recovery catalog in the Recovery Appliance will be used.

    Before you can backup or restore a sharded database using GDSCTL, you must have access to a recovery catalog created in a dedicated database. This recovery catalog serves as a centralized RMAN repository for the shard catalog database and all of the shard databases.

    Note the following:

    • The version of the recovery catalog schema in the recovery catalog database must be compatible with the sharded database version because RMAN has compatibility requirements for the RMAN client, the target databases, and the recovery catalog schema. For more information, see Oracle Database Backup and Recovery Reference, cross-referenced below.

    • The recovery catalog must not share a host database with the shard catalog because the shard catalog database is one of the target databases in the sharded database backup configuration, and RMAN does not allow the recovery catalog to reside in a target database.

    • It is recommended that you back up the recovery catalog backup periodically, following appropriate best practices.

    • The shard catalog database and all of the shard databases can be configured to use the same recovery catalog or split between various recovery catalogs.

  • Configure backup destinations for the shard catalog database and all of the shard databases.

    The backup destination types are either DISK or system backup to tape. The supported DISK destinations are NFS and Oracle ASM file systems.

    Note that, if you are using Oracle Object Storage, Recovery Appliance, or Amazon S3 as a backup destionation, RMAN treats them as tape internally. A spcial backup module needs to be installed on the target database host.

    System backup to tape destinations require additional software modules to be installed on the database host. They must be properly configured to work with RMAN.

    Using Recovery Appliance as the sharded database backup destination is special case. Recovery Appliance comes with a built-in recovery catalog. Because a database can only be registered as a target database with a single recovery catalog, when configuring the sharded database backup with a Recovery Appliance as the backup destination, the built-in recovery catalog is used for the sharded database backup and restore.

    The shard catalog database and all of the shard databases must be configured to use the same Recovery Appliance as the backup destination.

    If the shard catalog database or the shard databases are in Data Guard configurations, you can choose to back up either the primary or standby databases.

    See also:

  • RMAN connects to the target databases as specific internal users to do database backup and restore with the exception of the shard catalog.

    For the shard catalog, a common user in the CDB hosting the shard catalog PDB must be provided at the time when sharded database backup is configured. This user must be granted the SYSDG and SYSBACKUP privileges. If the CDB is configured to use local undo for its PDBs, the SYSBACKUP privilege must also be granted commonly.

    For the shard databases, the internal CDB common user, GSMROOTUSER, is used. This user must be unlocked in the shard CDB root databases and granted the SYSBACKUP privilege in addition to other privileges that the sharded database requires for GSMROOUSER. If the CDB is configured to use local undo for its PDBs, the SYSBACKUP privilege must be granted commonly to GSMROOTUSER, meaning the CONTAINER=ALL clause must be used when granting the SYSBACKUP privilege.

  • All of the GDSCTL commands for sharded database backup and restore operations require the shard catalog database to be open. If the shard catalog database itself must be restored, you must manually restore it.
  • You are responsible for offloading backups to tape or other long-term storage media and following the appropriate data retention best practices.

Note:

See RMAN Compatibility in Oracle Database Backup and Recovery Reference

Configuring Automated Backups

Use the GDSCTL CONFIG BACKUP command to configure automated backups.

You should connect to a shard director (GSM) host to run the GDSCTL backup commands. If the commands are run from elsewhere, you must explicitly connect to the shard catalog database using the GDSCTL CONNECT command.

When you run the GDSCTL backup configuration, you can provide the following inputs.

  • A list of databases

    The databases are the shard catalog database and shard databases. Backup configuration requires that the primary databases of the specified databases be open for read and write, but the standby databases can be mounted or open.

    If a database is in a Data Guard configuration when it is configured for backup, all of the databases in the Data Guard configuration are configured for backup. For a shard in Data Guard configuration, you must provide the backup destinations and start times for the primary and all of the standby shards.

    This is different for the shard catalog database. The shard catalog database and all the shard catalog standby databases will share a backup destination and a start time.

  • Connect strings to the recovery catalog databases

    For the connect string you need a user account with privileges for RMAN, such as RECOVERY_CATALOG_OWNER role.

  • RMAN backup destination parameters

    These parameters include backup device and channel configurations. Different backup destinations can be used for different shards.

    Please note the following:

    • Backup destinations for shards in Data Guard configuration must be properly defined to ensure that the backups created from standby databases can be used to restore the primary database and conversely. See "Using RMAN to Back Up and Restore Files" in Oracle Data Guard Concepts and Administration for Data Guard RMAN support.

    • The same destination specified for the shard catalog database is used as the backup destination for the shard catalog standby databases.

    • For system backup to tape devices, the media managers for the specific system backup to tape devices are needed for RMAN to create channels to read and write data to the devices. The media manager must be installed and properly configured.

    See also:

  • Backup target type

    Backup target type defines whether the backups for the shard catalog database and shards should be done at the primary or one of the standby databases. It can be either PRIMARY or STANDBY. The default backup target type is STANDBY. For the shard catalog database or shards that are not in Data Guard configurations, the backups will be done on the shard catalog database or the shards themselves even when the backup target type is STANDBY.

  • Backup retention policy

    The backup retention policy specifies a database recovery window for the backups. It is specified as a number of days.

    Obsolete backups are not deleted automatically, but a GDSCTL command is provided for you to manually delete them.

  • Backup schedule

    Backup schedules specify the automated backup start time and repeat intervals for the level 0 and level 1 incremental backups. Different automated backup start times can be used for the shard catalog database and individual shards.

    The time is a local time in the time zone in which the shard catalog database or shard is located. The backup repeat intervals for the level 0 and level 1 incremental backups are the same for the shard catalog database and all the shards in the sharded database,

  • CDB root database connect string for the shard catalog database

    The provided user account must have common SYSBACKUP privilege in the provided CDB.

When no parameters are provided for the CONFIG BACKUP command, GDSCTL displays the current sharded database backup configuration. If the backup has not been configured yet when the command is used to show the backup configuration, it displays that the backup is not configured.

To configure a backup, run GDSCTL CONFIG BACKUP as shown in the following example. For complete syntax, command options, and usage notes, run HELP CONFIG BACKUP.

The following example configures a backup channel of type DISK for the shard catalog database, two parallel channels of type DISK for each of the shards (shard spaces dbs1 and dbs2 are used in the shard list), the backup retention window is set to 14 days, the level 0 and level 1 incremental backup repeat intervals are set to 7 and 1 day, and the backup start time is set to 12:00 AM, leaving the incremental backup type the default DIFFERENTIAL, and the backup target type the default STANDBY. 

GDSCTL> config backup -rccatalog rccatalog_connect_string 
-destination "CATALOG::configure channel device type disk format '/tmp/rman/backups/%d_%U'" 
-destination "dbs1,dbs2:configure device type disk parallelism 2:configure channel 1 device type disk format '/tmp/rman/backups/1/%U';configure channel 2 device type disk format '/tmp/rman/backups/2/%U'" 
-starttime ALL:00:00 -retention 14 -frequency 7,1 -catpwd gsmcatuser_password -cdb catcdb_connect_string;

Once GDSCTL has the input it displays output similar to the following, pertaining to the current status of the configuration operation.

Configuring backup for database "sales_catalog" ...

Updating wallet ...
The operation completed successfully
 
Configuring RMAN ...
new RMAN configuration parameters:
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/tmp/rman/backups/%d_%u';
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete
 
new RMAN configuration parameters:
CONFIGURE BACKUP OPTIMIZATION ON;
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete
...

Creating RMAN backup scripts ...
replaced global script full_backup 
replaced global script incremental_backup
...
Creating backup scheduler jobs ...
The operation completed successfully
 

Creating restore point creation job ...
The operation completed successfully

Configuring backup for database "sales_east" ...
 
Updating wallet ...
The operation completed successfully
 
Configuring RMAN ...
new RMAN configuration parameters:
CONFIGURE DEVICE TYPE DISK PARALLELISM 2 BACKUP TYPE TO BACKUPSET;
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete
 
new RMAN configuration parameters:
CONFIGURE CHANNEL 1 DEVICE TYPE DISK FORMAT '/tmp/rman/backups/1/%u';
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete
...
 
Configuring backup for database "sales_west" ... 
Updating wallet ...
The operation completed successfully
 
Configuring RMAN ...
...

Recovery Manager complete.

As shown in the CONFIG BACKUP command output above, GDSCTL does the following steps.

  1. GDSCTL updates the shard wallets.

    The updated wallets will contain:

    • Connect string and authentication credentials to the RMAN catalog database.

    • Connect string and authentication credentials to the RMAN TARGET database.

    • Automated backup target type and start time.

  2. GDSCTL sets up the RMAN backup environment for the database.

    This includes the following tasks.

    • Registering the database as a target in the recovery catalog.

    • Setting up backup channels.

    • Setting up backup retention policies.

    • Enabling control file and server parameter file auto-backup.

    • Enabling block change tracking for all the target databases.

  3. On the shard catalog, GDSCTL creates global RMAN backup scripts for level 0 and level 1 incremental backups.

  4. On the shard catalog, GDSCTL creates a global restore point creation job.

  5. On the shard catalog and each of the primary databases, GDSCTL

    • Creates DBMS Scheduler database backup jobs for level 0 and level 1 incremental backups

    • Schedules the jobs based on the backup repeat intervals you configure.

Specifying Multiple Recovery Catalogs

When configuring backups, you can specify different recovery catalogs for different shards and the shard catalog by running GDSCTL CONFIG BACKUP multiple times with different recovery catalogs and different shards or the shard catalog specified in each run of the command.

For example:

CONFIG BACKUP –shard shard_east -rccatalog rcadmin/rman@rc_east
CONFIG BACKUP –shard shard_west -rccatalog rcadmin/rman@rc_west

After running the above two commands, the shard “shard_east” will use “rc_east” as the recovery catalog, while “shard_west” will use “rc_west”, for all following manual and automatic backup jobs, and LIST/DELETE/VALIDATE/RESTORE BACKUP commands.

If the parameter -rccatalog is not provided, then the recovery catalog credentials and connect identifier specified previously will be used. For example, after running the above two commands, if you issue the following command without parameter -rccatalog: 

CONFIG BACKUP –shard catalog ...

Then the shard catalog will use “rc_west” as the recovery catalog.

Note that if running CONFIG BACKUP against a sharded database for the first time but without -rccatalog, the command will do nothing but print an error message.

Backup Set Encryption

Backup sets to be stored in the Oracle Object Store must be encrypted. Set up encryption on all shards and shard catalog in a sharded database using the following procedure.

Note that you can use encryption even if you are not using Oracle Object Store as a backup destination. This may be preferred when there is sensitive data in the database or if you are using third-party cloud storage such as Amazon S3.

Task 1: Configure TDE Encryption

To enable the encryption of backup sets, a user having ADMINISTER KEY MANAGEMENT or SYSKM privilege needs to perform the following actions on all container databases (CDBs) at the root level for the shards and the shard catalog:

  1. Configure the keystore location and type by setting the WALLET_ROOT and TDE_CONFIGURATION initialization parameters.

    1. First check if WALLET_ROOT points to an existing file location.

      If necessary, you can use the following SQL*Plus statement to change it.

      ALTER SYSTEM SET WALLET_ROOT=wallet-root-location SCOPE=BOTH

    2. Specify the keystore type.

      ALTER SYSTEM SET TDE_CONFIGURATION="KEYSTORE_CONFIGURATION=FILE" SCOPE=BOTH;

    3. Restart the database.

    Note the following:

    • The WALLET_ROOT parameter specifies the top directory for many different software keystores (such as TDE, Oracle Enterprise User Security (EUS), TLS). For TDE, the directory for automated discovery is WALLET_ROOT/tde.

    • In releases older than 19c, the SQLNET.ENCRYPTION_WALLET_LOCATION parameter was used to define the keystore directory location. This parameter has since been deprecated. Oracle recommends that you use the WALLET_ROOT static initialization parameter and TDE_CONFIGURATION dynamic initialization parameter instead.
    • If the value of parameter TDE_CONFIGURATION is KEYSTORE_CONFIGURATION=FILE, software keystore will be used. Two alternative keystore type values are KEYSTORE_CONFIGURATION=OKV|HSM for Oracle Key Vault or hardware security module keystores.

  2. Create a password-protected software keystore.

    ADMINISTER KEY MANAGEMENT CREATE KEYSTORE 'wallet-root-location/tde'
 IDENTIFIED BY keystore_password;

    Note that wallet-root-location should be the same value as the initialization parameter WALLET_ROOT.

  3. Open the software keystore.

    ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN
 IDENTIFIED BY keystore_password CONTAINER=ALL;

  4. Set the TDE encryption master key.

    ADMINISTER KEY MANAGEMENT SET KEY
 IDENTIFIED BY keystore_password WITH BACKUP CONTAINER=ALL;

See also: Configuring a Software Keystore in Oracle Database Advanced Security Guide

Task 2: Enable Encryption

To enable the encryption of backup sets, specify an encryption algorithm in the GDSCTL CONFIG BACKUP command using option -encryption. For example: 

GDSCTL> CONFIG BACKUP … -encryption shard-list:encryption-algorithm

You can get a list of supported algorithms from column ALGORITHM_NAME in V$RMAN_ENCRYPTION_ALGORITHMS. When an algorithm is given, the CONFIG BACKUP command invokes the following RMAN scripts after registering a backup target database in the recovery catalog: 

CONFIGURE ENCRYPTION FOR DATABASE ON; 
CONFIGURE ENCRYPTION ALGORITHM TO encryption_algorithm;

Note:

You must configure Transparent Data Encryption on the shards and shard catalog where encryption is needed. Otherwise, manual and automatic backup jobs, or VALIDATE/RESTORE BACKUP commands will fail.

Disabling Backup Set Encryption

To disable encryption, you can set the CONFIG BACKUP -encryption option to OFF.

When OFF is specified, the encryption on the shards and/or shard catalog in shard-list is disabled. For example: 

GDSCTL> CONFIG BACKUP … -encryption shard-list:OFF

Then the GDSCTL invokes the following RMAN script: 

CONFIGURE ENCRYPTION FOR DATABASE OFF;

Note:

When the encryption is disabled, you cannot restore the database with encrypted backup sets.

Using Oracle Object Storage as a Backup Destination

Oracle Object Storage is a convenient location on Oracle Cloud Infrastructure where you can store files. Oracle Recovery Manager (RMAN) has a media library that lets you set Object Storage as a backup destination for sharded databases through the GDSCTL CONFIG BACKUP command.

To use Oracle Object Store as a repository for sharded database backups, complete the following prerequisites, and specify the CONFIG BACKUP destination as described below.

Prerequisites for Oracle Object Store Backup Destination

  1. Configure Transparent Data Encryption (TDE) on all backup target databases, including the shard catalog and the shards where backup destinations will be the Object Storage as described in Backup Set Encryption.

    This includes:

    • Configure the keystore location and type by setting initialization parameters WALLET_ROOT and TDE_CONFIGURATION.

    • Create the software keystore used by TDE.

  2. Enable the encryption of backup sets as described in Backup Set Encryption.

  3. Get the following Oracle Cloud related accounts and IDs:

    • An Oracle Cloud account with access to Oracle Cloud Infrastructure Object Storage
    • Oracle Cloud Infrastructure API signing keys, tenant OCID, and user OCID

  4. Create an Oracle Object Storage bucket to use as the backup destination..

  5. Install Oracle Cloud Infrastructure Backup Modules on the shard catalog and all shards.

    During the installation specify:

    • The credentials for Oracle Cloud console - Note that the installer will put these credentials in a wallet.
    • Bucket name and endpoint URL
    • Oracle Cloud Infrastructure API signing keys, tenant OCID, and user OCID

    See Installing the Oracle Database Cloud Backup Module for OCI in Using Oracle Database Backup Cloud Service for more details.

Set the Backup Destination in the Sharded Database Backup Configuration

In the following example, Object Storage is set as the backup destination of shardspace dbs1.

GDSCTL> config backup
 –shard dbs1
 -rccatalog rccatalog_connect_string
 -catpwd password
 -cdb catcdb_connect_string
 -encryption dbs1:AES256 
 -destination dbs1:”CONFIGURE DEFAULT DEVICE TYPE TO SBT”:”CONFIGURE CHANNEL DEVICE TYPE SBT
 PARMS='SBT_LIBRARY=/orclhome/lib/libopc.so,SBT_PARMS=(OPC_PFILE=/orclhome/dbs/opct1.ora)’”

As shown above, Object Storage requires the following CONFIG BACKUP parameter settings in addition to the standard options:

  • Enable encryption on the shardspace with option -encryption. To support the Oracle Object Storage as a backup destination, the backup sets must be encrypted.

  • Specify the following in option -destination using RMAN device configuration and channel configuration statements.

    • SBT_TAPE as the backup default device

    • SBT library file and OPC configuration file locations in the backup channel parameters, using the following template:
      CONFIGURE CHANNEL DEVICE TYPE sbt 
   PARMS='SBT_LIBRARY=location-of-SBT-library-for-backup-module, 
   SBT_PARMS=(OPC_PFILE=location-of-configuration-file)';

Using Recovery Appliance as a Backup Destination

Zero Data Loss Recovery Appliance (Recovery Appliance), configured for real-time redo transport, directly transports redo data from the protected database and stores it on the Recovery Appliance. This reduces the window of potential data loss that exists between successive archived log backups.

To use Recovery Appliance as a repository for sharded database backups, complete the following prerequisites and specify the CONFIG BACKUP destination as described below.

Prerequisites

  1. Copy the shared library libra.so from the Recovery Appliance Backup Module to the hosts where the shard catalog and each shard are located.

    You can find this library file in the $ORACLE_HOME/lib directory of the Recovery Appliance.

  2. (Required only if configuring real-time redo transport) Create a redo transport user at the root level of the shard catalog and shards and grant CREATE SESSION and SYSOPER privileges to this user. 

    create user c##rman1 identified by rman1; 
grant create session, sysoper to rman1;

    Note that

    • This user must be a common user on the shard catalog or shard.

    • This user should have the same user name and password as the Recovery Appliance virtual private catalog user assigned to the shard catalog or shards.

    • If in a Data Guard configuration, you only create this user on the primary database, then the user is propagated to the standby databases.

This waives the requirements to install the Recovery Appliance Backup Module on the protected databases. Moreover, you do not need to enroll the protected database with DBMS_RA package. The CONFIG BACKUP command automates the setup.

Collect Required Information

  • The location of the shared library, libra.so, on the shard catalog and each shard

  • Connect string to the Recovery Appliance catalog

  • The user name and password of a catalog owner to use in RMAN command CONNECT CATALOG

  • The user name and password of a virtual private catalog (VPC) account of the Recovery Appliance

    HTTP digest access authentication must be enabled for this user.

    For example:

    create user vpc_user1 identified by vpc;
grant create session to vpc_user1 identified by vpc; 
# enable HTTP digest access authentication
alter user vpc_user1 identified by vpc digest enable;

  • The name of a protection policy created on the Recovery Appliance

  • The amount of disk space on Recovery Appliance reserved for the shard catalog and each shard

  • (Required only if configuring real-time redo transport) The location of the auto login wallets used by shard catalog or shards for real-time redo transport. The GDSCTL CONFIG BACKUP command creates the wallet in this location.

Set the Backup Destination in the Sharded Database Backup Configuration

The following example shows how to issue GDSCTL command CONFIG BACKUP to set up a backup configuration using the Recovery Appliance “ra_east” as the backup destination of a sharded database: 

GDSCTL> config backup
 –shard ALL
 -cdb catcdb_connect_string
 -zdlra_catalog username@ra_east
 -zdlra_vpc CATALOG:catalog_vpc_username
 -zdlra_vpc SH1:shard_vpc_username 
 -zdlra_policy CATALOG:catalog_protection_policy_name
 -zdlra_policy SH1:shard_protection_policy_name
 -zdlra_space CATALOG:20G
 -zdlra_space SH1:2T
 -zdlra_lib_dir CATALOG:?/lib
 -zdlra_lib_dir SH1:/u01/app/oracle/product/19.0.0/dbhome_1/lib

Note:

The -destination and -rccatalog parameters are not used for configuring Recovery Appliance as the backup destination.

The -zdlra_* parameters used in the example are explained below:

Parameter Description/Usage Notes/Examples
-zdlra_catalog username[/password]@connect-string

This parameter is mutually exclusive with -rccatalog.

If specified, it indicates that all listed shards and/or shard catalog in parameter -shard will use the specified Recovery Appliance as the backup destination.

Provide the user name and connect string of the Recovery Appliance administrator that should have RASYS privilege.

For example:

-zdlra_catalog cat_username@ra_east

Note:

  • The password will be prompted if not specified.
  • The user name specified in this parameter is the administrator of the Recovery Appliance that should have RASYS privilege. It is different from the virtual private catalog user provided in parameter -zdlra_vpc.
-zdlra_vpc(ALL|CATALOG|shard-list):username[/password]

Mandatory if -zdlra_catalog is specified.

It specifies the user of a Recovery Appliance VPC (virtual private catalog) user for a specified shard, shardgroup, shardspace, or region.

For example:

-zdlra_vpc CATALOG:catalog_vpc_username

-zdlra_vpc SH1:shard_vpc_username

Note:

  • This parameter can appear multiple times on one command line to allow different VPC users for different shards.
  • The shard list specifies a comma-separated list of shard identifiers. They can be shardspaces, shardgroups, regions, or shard names.
  • The password will be prompted if not specified.

This user is different from the Recovery Appliance administrator (RASYS user) provided in the parameter -zdlra_catalog.

-zdlra_policy (ALL|CATALOG|shard-list):protection-policy-name

Mandatory if -zdlra_catalog is specified.

It specifies the name of a protection policy defined in Recovery Appliance.

For example:

-zdlra_policy CATALOG:brzone_dev

-zdlra_policy SH1:silver_dev

Note:

  • This parameter can appear multiple times on one command line to allow different protection policies for different shards.
  • The shard list specifies a comma-separated list of shard identifiers. They can be shardspaces, shardgroups, regions, or shard names.
-zdlra_space (ALL|CATALOG|shard-list):(0-9)+(K|M|G|T|P|E|Z|Y)

Mandatory if -zdlra_catalog is specified.

Specifies the amount of disk space allocated on the Recovery Appliance for each shard and shard catalog.

For example:

-zdlra_space CATALOG:20G

-zdlra_space SH1:2T

Note:

  • This parameter can appear multiple times on one command line in case the space requirements for different shards are different.
  • The shard list specifies a comma-separated list of shard identifiers. They can be shardspaces, shardgroups, regions, or shard names.
  • The letter at the end is a unit specifier in the following list:

    K: Kilobytes

    M: Megabytes

    G: Gigabytes

    T: Terabytes

    P: Petabytes

    E: Exabytes

    Z: Zettabytes

    Y: Yottabytes
-zdlra_lib_dir (ALL|CATALOG|shard-list):library-location

Mandatory if -zdlra_catalog is specified.

Specifies the location of the shared library, libra.so, for the Recovery Appliance backup module. This location should be accessible by the specified shard or shard catalog.

For example:

-zdlra_lib_dir CATALOG:?/lib

-zdlra_lib_dir SH1:/u01/app/oracle/product/23.1.0/dbhome_1/lib

Note:

  • This parameter can appear multiple times on one command line in case the library locations on different shards are different.
  • You can use a question mark (?) to represent ORACLE_HOME environment variable. The "at" sign (@) stands for ORACLE_SID environment variable. This can help simplify the setup if the Oracle installation folders for different shards are different.

    For example, if the library file location is in the folder $ORACLE_HOME/lib/, then the user can use the following parameter to specify the library file location for all shards:

    -zdlra_lib_dir ALL:?/lib
-zdlra_redo_wallet (ALL|CATALOG|shard-list):wallet-location

Optional.

Specifies the location of the auto-login wallet used by the redo transport layer to log into Recovery Appliance.

Once specified, CONFIG BACKUP command will configure real-time redo transport for the databases specified in parameter -shard and create an auto-login wallet for each protected database.

For example:

-zdlra_redo_wallet CATALOG:?/wallet

-zdlra_redo_wallet SH1:/u01/app/oracle/product/23.1.0/dbhome_1/wallet

Note:

  • This parameter can appear multiple times on one command line in case the wallet locations on different shards are different.
  • You can use question mark (?) and "at" sign (@) to represent ORACLE_HOME and ORACLE_SID environment variables respectively. This can help simplify the setup if the Oracle installation folders for different shards are different.
  • The following entry will be added into the auto-login wallet:

    (connect-string-specified-in-parameter-zdlracatalog, VPC-username, VPC-user-password)

    where VPC user name and password are from parameter -zdlra_vpc. As explained in the prerequisites, you must create a redo transport user on the protected database having the same user name as VPC user.

See Global Data Services Control Utility (GDSCTL) Command Reference in Global Data Services Concepts and Administration Guide for details about other CONFIG BACKUP parameters.

Real-Time Redo Transport Post-Configuration Requirements

If the real-time redo transport is required, perform the following actions after running CONFIG BACKUP:

  • Add the location of wallets used by redo transport to sqlnet.ora and set SQLNET.WALLET_OVERRIDE to TRUE. 

    WALLET_LOCATION= 
 (SOURCE=(METHOD=FILE)(METHOD_DATA= 
 (DIRECTORY= wallet_location))) 
SQLNET.WALLET_OVERRIDE=TRUE

  • Reboot the primary databases of the shard catalog and shards, so the changes to sqlnet.ora can take effect.

  • In the shard catalog and shards (primary and standby databases for all), change the initialization parameter REDO_TRANSPORT_USER to the redo transport user (which uses the same user name as the virtual private catalog (VPC) user).

    For example:

    alter system set "redo_transport_user"=" c##rman1";

  • You can run a manual backup once, or wait until the first automatic backup runs, then the real-time redo transport is active.

    Once active, the following query on Recovery Appliance should return TRUE: 

    select NZDL_ACTIVE
from DBMS_RA. RA_DATABASE 
where DB_UNIQUE_NAME='DB_UNIQUE_NAME-of-the-protected-database'

Using Amazon S3 as a Backup Destination

Amazon S3 (Amazon Simple Storage Service) is a cloud-based storage service. Oracle’s Secure Backup Cloud Module can store backup sets on Amazon S3 storage using Oracle Recovery Manager (RMAN).

To use Amazon S3 as a repository for sharded database backups, complete the following prerequisites and specify the CONFIG BACKUP destination as described below.

Prerequisites for Amazon S3 Backup Destination

The Oracle Secure Backup Cloud Module must be installed on the shard catalog and the shards where backup sets will be sent to Amazon S3 cloud storage.

During the installation, you need to provide some parameters, including, but not limited to:

  • (Mandatory) AWSID, the access key ID for Amazon S3 cloud storage service
  • (Mandatory) AWSKey, the password for the above ID
  • (Optional) awsEndpoint, the host name where the backup sets will be sent
  • (Optional) awsPort, the HTTP/HTTPS connection port number
  • (Optional) location, the Amazon S3 location
  • (Optional) walletDir, a directory for Oracle wallet to store Amazon S3 credentials, and proxy information if applicable
  • (Optional) configFile, a configuration file that contains the parameters used by RMAN, including the Oracle wallet directory.
  • (Optional) libDir, a directory to store the downloaded Oracle Secure Backup Cloud Module library

During the installation, you will create a wallet to store Amazon S3 credentials and proxy information if applicable, create a configuration file, and download Oracle Secure Backup Cloud Module library.

For detailed information about the installation of backup module, See Using Oracle Secure Backup Cloud Module on Amazon S3 in Oracle Database Backup and Recovery Reference.

It is also recommended that you encrypt backup sets destined for Amazon S3 storage. See Backup Set Encryption for details.

Set the Backup Destination in the Sharded Database Backup Configuration

The following example shows how to issue GDSCTL command CONFIG BACKUP to set up a backup configuration using the Amazon S3 as the backup destination of shardspace DBS1, and enabling backup set encryption. 

GDSCTL> config backup
 –shard dbs1
 -rccatalog rccatalog_connect_string
 -catpwd password
 -cdb catcdb_connect_string
 -encryption dbs1:AES256 
 -destination dbs1: "CONFIGURE DEFAULT DEVICE TYPE TO SBT":
"CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' 
PARMS 'SBT_LIBRARY=/u01/app/oracle/product/19.0.0/dbhome_1/lib/libosbws.so 
ENV=(OSB_WS_PFILE=/u01/app/oracle/product/19.0.0/dbhome_1/dbs/osbwssales.ora)'"

To use Amazon S3 as a backup destination, you must provide the following information when writing channel config statement in option -destination.

  • The path to shared library libra.so
  • The location of the configuration file used by RMAN

    The template for the CONFIGURE CHANNEL command is: 

    CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE'
   PARMS 'SBT_LIBRARY=secure-backup-library-location 
   ENV=(OSB_WS_PFILE=configuration-file-location)'

    For more information on RMAN channel settings related to Amazon S3, See Configuring RMAN SBT Channels for Recovery Appliance in Zero Data Loss Recovery Appliance Protected Database Configuration Guide.

Enabling and Disabling Automated Backups

You can enable or disable backups on all shards, or specific shards, shardspaces, or shardgroups.

All backup jobs are initially disabled. They can be enabled by running the GDSCTL ENABLE BACKUP command. 

GDSCTL> ENABLE BACKUP

When not specified, ENABLE BACKUP enables the backup on all shards. You can optionally list specific shards, shardspaces, or shardgroups on which to enable the backup. 

GDSCTL> ENABLE BACKUP -shard dbs1

The DISABLE BACKUP command disables an enabled backup. 

GDSCTL> DISABLE BACKUP -shard dbs1

Backup Job Operation

Once configured and enabled, backup jobs run on the primary shard catalog database and the primary shards as scheduled.

After a backup job is configured, it is initially disabled. You must enable a backup job for it to run as scheduled. Use the GDSCTL commands ENABLE BACKUP and DISABLE BACKUP to enable or disable the jobs.

Backup jobs are scheduled based on the backup repeat intervals you configure for the level 0 and level 1 incremental backups, and the backup start time for the shard catalog database and the shards.

Two separate jobs are created for level 0 and level 1 incremental backups. The names of the jobs are AUTOMATED_SDB_LEVEL0_BACKUP_JOB and AUTOMATED_SDB_LEVEL1_BACKUP_JOB. Full logging is enabled for both jobs.

When running, the backup jobs find the configured backup target type (PRIMARY or STANDBY), figure out the correct target databases based on the backup target type, and then launch RMAN to back up the target databases. RMAN uses the shard wallets updated during the backup configuration for database connection authentication.

Note that sharded database chunk moves do not delay automated backups.

Monitoring Backup Status

There are a few different ways to monitor the status of automated and on-demand backup jobs.

Monitoring an Automated Backup Job

Because full logging is enabled for the automated backup jobs, DBMS Scheduler writes job processing details in the job log and views. The Scheduler job log and views are your basic resources and starting point for monitoring the automated backups. Note that although the DBMS Scheduler makes a list of job state change events available for email notification subscription. This capability is not used for sharded database automated backups.

You can use the GDSCTL command LIST BACKUP to view the backups and find out whether backups are created at the configured backup job repeat intervals.

Automated backups are not delayed by chunk movement in the sharded database, so the backup creation times should be close to the configured backup repeat intervals and the backup start time.

Monitoring an On-Demand Backup Job

Internally, on-demand backup jobs are also started by DBMS Scheduler jobs on the database servers. The names of the temporary jobs are prefixed with tag MANUAL_BACKUP_JOB_. On-demand backups always run in the same session that GDSCTL uses to communicate with the database server. Failures from the job are sent directly to the client.

Using DBMS Scheduler Jobs Views

The automated backup jobs only run on the primary shard catalog database and the primary shards. To check the backup job details for a specific target database, connect to the database, or its primary database if the database is in a Data Guard configuration, using SQL*PLUS, and query the DBMS Scheduler views *_SCHEDULER_JOB_LOG and *_SCHEDULER_JOB_RUN_DETAILS based on the job names.

The names of the two automated backup jobs are AUTOMATED_SDB_LEVEL0_BACKUP_JOB and AUTOMATED_SDB_LEVEL1_BACKUP_JOB.

You can also use the GDSCTL command STATUS BACKUP to retrieve the job state and run details from these views. See Viewing Backup Job Status for more information about running STATUS BACKUP.

The job views only contain high level information about the job. For job failure diagnosis, you can find more details about the job in the RDBMS trace files by grepping the job names.

If no errors are found in the job, but still no backups have been created, you can find the PIDs of the processes that the jobs have created to run RMAN for the backups in the trace files, and then look up useful information in the trace files associated with the PIDs.

Using Backup Command Output

This option is only available for on-demand backups.

When you start on-demand backups with GDSCTL RUN BACKUP, you can specify the -sync command option. This forces all backup tasks to run in the foreground, and the output from the internally launched RMAN on the database servers is displayed in the GDSCTL console.

The downside of running the backup tasks in the foreground is that the tasks will be run in sequence, therefore the whole backup will take more time to complete.

See the GDSCTL reference in Oracle Database Global Data Services Concepts and Administration Guide for detailed command syntax and options.

Viewing an Existing Backup Configuration

When GDSCTL CONFIG BACKUP is not provided with any parameters, it shows the current backup configuration.

Because the parameters -destination and -starttime can appear more than once in CONFIG BACKUP command line for different shards and backup configuration can be done more than once, multiple items could be listed in each of the Backup destinations and Backup start times sections. The items are listed in the same order as they are specified in the CONFIG BACKUP command line and the order the command is repeatedly run.

To view an existing backup configuration, run CONFIG BACKUP, as shown here. 

GDSCTL> CONFIG BACKUP;

If a sharded database backup has not been configured yet, the command output will indicate it. Otherwise the output looks like the following:

GDSCTL> config backup
Recovery catalog database user: rcadmin
Recovery catalog database connect descriptor: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=den02qxr)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=cdb6_pdb1.example.com)))
Catalog database root container user: gsm_admin
Catalog database root container connect descriptor: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=den02qxr)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=v1908.example.com)))
Backup retention policy in days: 14
Level 0 incremental backup repeat interval in minutes: 10080
Level 1 incremental backup repeat interval in minutes: 1440
Level 1 incremental backup type : DIFFERENTIAL
Backup target type: STANDBY
Backup destinations:
catalog::channel device type disk format '/tmp/rman/backups/%d_%u'
dbs1,dbs2:device type disk parallelism 2:channel 1 device type disk format '/tmp/rman/backups/1/%u';channel 2 device type disk format '/tmp/rman/backups/2/%u'
catalog::configure channel device type disk format '/tmp/rman/backups/%d_%u'
dbs1,dbs2:configure device type disk parallelism 2:configure channel 1 device type disk format '/tmp/rman/backups/1/%u';configure channel 2 device type disk format '/tmp/rman/backups/2/%u'
Backup start times:
all:00:00

CONFIG BACKUP Output for Multiple Recovery Catalogs

The output of CONFIG BACKUP issued without any parameters on a configuration with multiple recovery catalogs prints the recovery catalog users and connect identifiers for different shards and shard catalog. The last used recovery catalog will be printed as well. For example: 

GDSCTL> config backup
...  
Recovery catalog database user:  
last::rcadmin_west 
shard_east::rcadmin_east 
shard_west::rcadmin_west  
Recovery catalog databaseconnect identifier:   
last::<theconnect identifier of rc_west> 
shard_east::<theconnect identifier of rc_east> 
shard_west::<theconnect identifier of rc_west> 
...

Note:

  • In the actual output for the above example, three real connector identifiers will be printed.
  • The example shows only the parts related to recovery catalog settings.

Running On-Demand Backups

The GDSCTL RUN BACKUP command lets you start backups for the shard catalog database and a list of shards.

All on-demand backups are level 0 incremental backups. On-demand backups have no impact on the automated backup schedules configured for the shard catalog database and the shards.

Internally, on-demand backups are started by DBMS Scheduler jobs on the database servers. The jobs are created on-the-fly when the on-demand backup command RUN BACKUP is run.

On-demand backup jobs are temporary jobs, and they are automatically dropped after the backups have finished.

The names of the temporary jobs are prefixed with tag MANUAL_BACKUP_JOB_.

To use RUN BACKUP, you must have already set up the backup configuration with the CONFIG BACKUP command.

The RUN BACKUP command requires the shard catalog database and any primary shards to be backed up to be open. 

GDSCTL> RUN BACKUP -shard dbs1

The -shard option lets you specify a set of shards, shardspaces or shardgroups on which to run the backup. To take an on-demand backup on shardspace dbs1, you can run RUN BACKUP as shown in the example above.

See the GDSCTL reference in Oracle Database Global Data Services Concepts and Administration Guide for detailed command syntax and options.

Viewing Backup Job Status

Use GDSCTL command STATUS BACKUP to view the detailed state on the scheduled backup jobs in the specified shards. Command output includes the job state (enabled or disabled) and the job run details.

By default, the command displays the job run details of all the runs that the automated backup jobs have had from 30 days ago in the specified shards. If the job run details for different periods are needed, options -start_time and -end_time must be used.

Run STATUS BACKUP as shown in the following examples.

The following STATUS BACKUP command example lists the job state and all job run details from the SDB catalog and the primary shard “rdbmsb_cdb2_pdb1”: 

GDSCTL> status backup -catpwd -shard catalog,rdbmsb_cdb2_pdb1;
"GSMCATUSER" password:***
 
Retrieving scheduler backup job status for database "rdbms" ...
Jobs:
  Incremental Level 0 backup job is enabled
    Job schedule start time: 2020-07-27 00:00:00.000 -0400
    Job repeat interval: freq=daily;interval=1
  Incremental Level 1 backup job is enabled
    Job schedule start time: 2020-07-27 00:00:00.000 -0400
    Job repeat interval: freq=minutely;interval=60
  Global restore point create job is enabled
    Job schedule start time: 2020-07-27 23:59:55.960 -0400
    Job repeat interval: freq=hourly
 
Run Details:
  Incremental Level 1 backup job status: SUCCEEDED
    Job run actual start time: 2020-07-26 14:00:00.177 -0400
    Job run slave process ID: 9023
  Incremental Level 1 backup job status: SUCCEEDED
    Job run actual start time: 2020-07-26 22:00:01.305 -0400
Job run slave process ID: 59526
…
Global restore point create job status: SUCCEEDED
    Job run actual start time: 2020-07-27 15:28:37.603 -0400
    Job run slave process ID: 44227
  …
  Global restore point create job status: SUCCEEDED
    Job run actual start time: 2020-07-27 17:28:38.251 -0400
    Job run slave process ID: 57611
  
Retrieving scheduler backup job status for database "rdbmsb_cdb2_pdb1" ...
Jobs:
  Incremental Level 0 backup job is enabled
    Job schedule start time: 2020-07-28 00:00:00.000 -0400
    Job repeat interval: freq=daily;interval=1
  Incremental Level 1 backup job is enabled
    Job schedule start time: 2020-07-28 00:00:00.000 -0400
    Job repeat interval: freq=minutely;interval=60
 
Run Details:
  Incremental Level 1 backup job status: SUCCEEDED
    Job run actual start time: 2020-07-26 14:00:00.485 -0400
    Job run slave process ID: 9056
  …
  Incremental Level 1 backup job status: SUCCEEDED
    Job run actual start time: 2020-07-27 14:33:42.702 -0400
    Job run slave process ID: 9056
  Incremental Level 0 backup job status: SUCCEEDED
    Job run actual start time: 2020-07-27 00:00:01.469 -0400
    Job run slave process ID: 75176

The following command lists the scheduler backup job state and the details of the job runs in the time frame from 2020/07/26 12:00:00 to 07/27 00:00 from the SDB catalog and the primary shard “rdbmsb_cdb2_pdb1”:

GDSCTL> status backup -start_time "2020-07-26 12:00:00" -end_time "2020-07-27 00:00:00" -catpwd -shard catalog,rdbmsb_cdb2_pdb1;
"GSMCATUSER" password:***

Retrieving scheduler backup job status for database "rdbms" ...
Jobs:
  Incremental Level 0 backup job is enabled
    Job schedule start time: 2020-07-27 00:00:00.000 -0400
    Job repeat interval: freq=daily;interval=1
  Incremental Level 1 backup job is enabled
    Job schedule start time: 2020-07-27 00:00:00.000 -0400
    Job repeat interval: freq=minutely;interval=60
  Globa1 restore point create job is enabled
    Job schedule start time: 2020-07-27 23:59:55.960 -0400
    Job repeat interval: freq=hourly
 
Run Details:
  Incremental Level 1 backup job status: SUCCEEDED
    Job run actual start time: 2020-07-26 14:00:00.177 -0400
    Job run slave process ID: 9023
  …
  Incremental Level 1 backup job status: SUCCEEDED
    Job run actual start time: 2020-07-26 23:50:00.293 -0400
    Job run slave process ID: 74171
  Globa1 restore point create job status: SUCCEEDED
    Job run actual start time: 2020-07-26 14:28:38.263 -0400
    Job run slave process ID: 11987
  …
  Globa1 restore point create job status: SUCCEEDED
    Job run actual start time: 2020-07-26 23:28:37.577 -0400
    Job run slave process ID: 69451
 
Retrieving scheduler backup job status for database "rdbmsb_cdb2_pdb1" ...
Jobs:
  Incremental Level 0 backup job is enabled
    Job schedule start time: 2020-07-28 00:00:00.000 -0400
    Job repeat interval: freq=daily;interval=1
  Incremental Level 1 backup job is enabled
    Job schedule start time: 2020-07-28 00:00:00.000 -0400
    Job repeat interval: freq=minutely;interval=60
 
Run Details:
  Incremental Level 1 backup job status: SUCCEEDED
    Job run actual start time: 2020-07-26 14:00:00.485 -0400
    Job run slave process ID: 9056
  Incremental Level 1 backup job status: SUCCEEDED
    Job run actual start time: 2020-07-26 22:11:50.931 -0400
    Job run slave process ID: 9056

Listing Backups

Use GDSCTL LIST BACKUP to list backups usable to restore a sharded database or a list of shards to a specific global restore point.

The command requires the shard catalog database to be open, but the shards can be in any of the started states: nomount, mount, or open.

You can specify a list of shards to list backups for in the command. You can also list backups usable to restore the control files of the listed databases and list backups for standby shards.

The following example shows the use of the command to list the backups from shard cdb2_pdb1 recoverable to restore point BACKUP_BEFORE_DB_MAINTENANCE. 

GDSCTL> LIST BACKUP -shard cdb2_pdb1 -restorepoint BACKUP_BEFORE_DB_MAINTENANCE

If option -controlfile is used, LIST BACKUPS will only list the backups usable to restore the control files of the specified shards. If option -summary is used, the backup will be listed in a summary format. 

GDSCTL> list backup -shard shd1,shd2 -controlfile -summary

Validating Backups

Run the GDSCTL VALIDATE BACKUP command to validate sharded database backups against a specific global restore point for a list of shards. The validation confirms that the backups to restore the databases to the specified restore point are available and not corrupted.

The shard catalog database must be open, but the shard databases can be either mounted or open. If the backup validation is for database control files, the shards can be started nomount.

The following example validates the backups of the control files from the shard catalog databases recoverable to restore point BACKUP_BEFORE_DB_MAINTENANCE. 

GDSCTL> VALIDATE BACKUP -shard shd1,shd2 -controlfile -restorepoint BACKUP_BEFORE_DB_MAINTENANCE

Backup validation for shards are done one shard a time sequentially.

Deleting Backups

Use the GDSCTL DELETE BACKUP command to delete backups from the recovery repository.

The DELETE BACKUP command deletes the sharded database backups identified with specific tags from the recovery repository. It deletes the records in the recovery database for the backups identified with the provided tags, and, if the media where the files are located is accessible, the physical files from the backup sets from those backups. This is done for each of the target databases. You will be prompted to confirm before the actual deletion starts.

To run this command, the shard catalog database must be open, but the shard databases can be either mounted or open.

The following is an example of deleting backups with tag odb_200414205057124_0400 from shard cdb2_pdb1. 

GDSCTL> DELETE BACKUP -shard cdb2_pdb1 -tag ODB_200414205057124_0400
"GSMCATUSER" password:

This will delete identified backups, would you like to continue [No]?y

Deleting backups for database "cdb2_pdb1" ...

Creating and Listing Global Restore Points

A restore point for a sharded database that we call a global restore point, actually maps to a set of normal restore points in the individual primary databases in a sharded database.

These restore points are created at a common SCN across all of the primary databases in the sharded database. The restore points created in the primary databases are automatically replicated to the Data Guard standby databases. When the databases are restored to this common SCN, the restored sharded database is guaranteed to be in a consistent state.

The global restore point creation must be mutually exclusive with sharded database chunk movement. When the job runs, it first checks whether any chunk moves are going on and waits for them to finish. Sometimes the chunk moves might take a long time. Also, new chunk moves can start before the previous ones have finished. In that case the global restore point creation job might wait for a very long time before there is an opportunity to generate a common SCN and create a global restore point from it. Therefore, it is not guaranteed that a global restore point will be created every hour.

To create the global restore point, run the GDSCTL command CREATE RESTOREPOINT as shown here. 

GDSCTL> CREATE RESTOREPOINT

The global restore point creation job is configured on the shard catalog database. The name of the job is AUTOMATED_SDB_RESTOREPOINT_JOB. Full logging for this job is enabled.

You can optionally enter a name for the restore point by using the -name option as shown here. 

GDSCTL> CREATE RESTOREPOINT -name CUSTOM_SDB_RESTOREPOINT_JOB

The job is initially disabled, so you must use GDSCTL ENABLE BACKUP to enable the job. The job runs every hour and the schedule is not configurable.

To list all global restore points, run LIST RESTOREPOINT. 

GDSCTL> LIST RESTOREPOINT

This command lists all of the available global restore points in the sharded database that were created during the specified time period with SCNs (using the -start_scn and -end_scn options) in the specified SCN interval (using the -start_time and -end_time options).

The following command lists the available restore points in the sharded database with the SCN between 2600000 and 2700000.

GDSCTL> LIST RESTOREPOINT -start_scn 2600000 -end_scn 2700000

The command below lists the available restore points in the sharded database that were created in the time frame from 2020/07/27 00:00:00 to 2020/07/28 00:00:00.

GDSCTL> LIST RESTOREPOINT -start_time "2020-07-27 00:00:00" -end_time "2020-07-28 00:00:00"

Restoring Shards From Backup

The GDSCTL RESTORE BACKUP command lets you restore a sharded database to a specific global restore point.

This command is used to restore shard database to a specific global restore point. It can also be used to restore only the shard database control files.

The typical procedure for restoring a sharded database is:

  1. List the available restore points.
  2. Select a restore point to validate the backups.
  3. Restore the databases to the selected restore point.

You should validate the backups for a shard against the selected restore point to verify that all the needed backups are available before you start to restore the shard to the restore point.

Note that you are not prohibited from restoring the shard catalog or a specific shard to an arbitrary point in time. However, doing so may put that target in an inconsistent state with the rest of the sharded database and you may need to take corrective action outside of the restore operation.

The database to be restored must be in NOMOUNT state. This command alters the database to MOUNT state after it has restored the control file.

The RESTORE BACKUP command requires the shard catalog database to be open.

For data file restore, the shards must be in MOUNT state, but if the command is to restore the control files, the shard databases must be started in NOMOUNT state. To bring the databases to the proper states will be a manual step.

To restore the shard database control files, the database must be started in nomount mode. The control files will be restored from AUTOBACKUP. To restore the database data files, the database must be mounted. The shard catalog database must be open for this command to work.

The following example restores the control files of shard cdb2_pdb1 to restore point BACKUP_BEFORE_DB_MAINTENANCE. 

GDSCTL> RESTORE BACKUP -shard cdb2_pdb1 -restorepoint BACKUP_BEFORE_DB_MAINTENANCE –controlfile

The restore operation can be done for the shards in parallel. When the restore for the shards happens in parallel, you should not close GDSCTL until the command has finished running, because interrupting the restore operation can result in database corruption or get the sharded database into an inconsistent state.

Backup validation only logically restores the database while RESTORE BACKUP will do both the physical database restore and the database recovery. Therefore, after RESTORE BACKUP is done, usually the restored the databases need to be opened with the resetlogs option.

After the database restore is completed, you should open the database and verify that the database has been restored as intended and it is in a good state.

Restoring the Shard Catalog from Backup

To configure a shard catalog to be restored from backup:

  • The sharded database must have been configured for backup and restore.
  • The CDB must be started in NOMOUNT state for control file restore, in MOUNT or OPEN state for the shard catalog restore.

Note that no database connection to the shard catalog is needed for GDSCTL to run the RESTORE BACKUP command to restore the sharded database control files and the shard catalog PDB.

Shard catalog restore uses GDSCTL RESTORE BACKUP with some different parameters than are used for a shard restore, as shown in the syntax here. 

GDSCTL> RESTORE BACKUP -shard CATALOG
    -cdb connect-string
    -catalog_name pdb-name
    -catalog_dbid dbid
    [-scn scn]
    [-controlfile]
    [-restore_only | -recover_only]

Because the shard catalog database must not be open when it is restored, some information obtainable from the catalog that is needed for the database restore will not be available automatically, so it will need to be provided with the RESTORE BACKUP command. This includes:

  • A common user with SYSDG privilege in the root container and SYSBACKUP privilege for all containers in the shard catalog CDB and its password
  • A connect identifier to the shard catalog CDB root (-cdb connect-string)
  • The name of the shard catalog PDB (-catalog_name pdb-name)
  • The shard catalog DBID (-catalog_dbid dbid)
  • If the shard catalog needs to be restored to a specific global restore point, instead of the name of the global restore point, the associated SCN must be provided for the command (-scn scn)

Removing Backup Configuration from a Shard

When you remove a shard from the sharded database configuration some backup artifacts remain on the database. You can remove these artifacts with GDSCTL CONFIG BACKUP option -REMOVE.

When a shard is removed from a sharded database configuration, the shard will no longer be included in automated backup jobs, but the artifacts created on the database host when the shard was configured for backup (for example, the backup wallets and the backup jobs) are not deleted.

To remove these artifacts from the shard before it is removed from the sharded database, run CONFIG BACKUP with the -REMOVE option, and provide a list of shards that should be removed, as shown here. 

GDSCTL> CONFIG BACKUP -remove -shard shard_list

The command does the following tasks:

  • Deletes the shard PDB container-level backup wallet.
  • Deletes the DBMS scheduler backup jobs from the shard database.
  • If the CDB is not a shared target database by some other sharded database, which happens when multiple sharded databases are placed inside the same set of CDBs, deletes the CDB root container level backup wallet.

This command should only be used if the shard is removed and is not expected to be added back to the same sharded database.

Running RMAN Commands from GDSCTL

RMAN commands can be submitted from the GDSCTL CLI to run against multiple shards and the shard catalog, either in serial or in parallel.

Prerequisites

To run RMAN commands from the GDSCTL CLI, the sharded database must have been configured for backup and restore using the GDSCTL CONFIG BACKUP command.

Some RMAN commands also require the target database in a specific state: OPEN, MOUNT or NOMOUNT, to be able to run. Also, some RMAN commands can be run only when RMAN is connected to the CDB root as the target database. Therefore, before submitting RMAN commands to run against a shard, make sure the target shard and its CDB are in the specific state required by the commands.

Using the GDSCTL RMAN Command

GDSCTL provides the RMAN command to allow you to run RMAN commands in a GDSCTL session.

You can pass RMAN commands either by referencing a file or including the RMAN statements in the GDSCTL RMAN command.

  • Enter a semi-colon after each RMAN statement:

    GDSCTL> RMAN -shard cdb1_pdb1 rman-stmt1;rman-stmt2;

    The RMAN statement must be contained in single or double quotation marks if the provided RMAN commands contain spaces and quotation marks:

    GDSCTL> RMAN -shard cdb1_pdb1 'rman-stmt1;rman-stmt2;'

  • Specify the file path of a RMAN command file:

    GDSCTL> RMAN -shard cdb2_pdb1 -cmd_file file-path

The following options are available:

Option Description
-async

Specifies that all tasks created to run this command will run in the background.

By default, these tasks run in the foreground.
-catpwd password

Specifies the password for user GSMCATUSER, which is prompted if not specified.

It needs to be specified only once for the entire GDSCTL session.
-check_syntax

Runs the command as a validation the RMAN command syntax.
-from_cdb userid/password

Lets you specify a common user in the shard CDB root and its password.

This option is required if the provided RMAN commands must be run from the CDB root, meaning that RMAN will be connected to the CDB root as the target database to run the RMAN commands.

The provided user must have SYSBACKUP privileges.

By default, the provided RMAN commands are run from the shard PDB.
-shard shard-list

Lets you specify a comma separated list of shard identifiers.

Each identifier can be a shardspace, a shardgroup, or a shard name.

The default is “no shards”

Error Handling for Automated Backup Operations

Automated Backup Error Handling

After the RMAN BACKUP command completes, the scheduler job continues running. It checks the RMAN output for errors. If no errors are found, the file is deleted, the job continues and is expected to complete successfully. If errors are found in the RMAN output file, then:

  • The RMAN output file is retained.
  • A job run error status is recorded in the table ALL_SCHEDULER_JOB_RUN_DETAILS.
  • The RMAN output file name is stored in the ADDITIONAL_INFO column of the ALL_SCHEDULER_JOB_RUN_DETAILS table, along with a path to the original RMAN output file.

Background Task Error Handling

When a command is run in the background, the fetched RMAN output is kept in memory. After the command is completed, the task checks the RMAN output for errors. If errors are detected, the last 1024 characters of the RMAN output are displayed in the GDSCTL console.

In this case, the entire the RMAN output are logged in the GDSCTL log file as well, which is specified using command CONFIGURE -LOG_FILE.