Setup and Configuration for Failover

This section establishes VPC users for the Recovery Appliances to use later for failover.It modifies the network configuration files needed, configures the replication server, creates protection policies, registers the protected database, and adds several grants to the upstream and downstream Recovery Appliances.

Creating VPC Users

This task creates database VPC user accounts in the upstream and downstream Recovery Appliances.

When creating the accounts, keep in mind these password requirements.

  • The first VPC user (HADR_LOCAL_VPCUSER) account may be used by other protected databases and can have different passwords between the RAHADR1 and RAHADR2 Recovery Appliances.
  • The second VPC user (HADR_COMMON_VPCUSER) account must use the same password on both the RAHADR1 and RAHADR2 Recovery Appliances and can be used by other protected databases

The following conditions are applicable to this specific example.

  • Recovery Appliance RAHADR1 has previously been installed with a DB_UNIQUE_NAME of rahadr1.
  • Recovery Appliance RAHADR2 has previously been installed with a DB_UNIQUE_NAME of rahadr2.
  1. Create two VPC users for the protected database on the upstream Recovery Appliance RAHADR1.
    # racli add db_user --user_name HADR_LOCAL_VPCUSER --user_type=vpc
[HADR_LOCAL_VPCUSER] New Password: ***********
Sun Mar 25 08:27:53 2018: Start: Add vpc user HADR_LOCAL_VPCUSER.
Sun Mar 25 08:27:53 2018: Add vpc user HADR_LOCAL_VPCUSER successfully.
Sun Mar 25 08:27:53 2018: End: Add vpc user HADR_LOCAL_VPCUSER.

# racli add db_user --user_name HADR_COMMON_VPCUSER --user_type=vpc
[HADR_COMMON_VPCUSER] New Password: ***********
Sun Mar 25 08:27:53 2018: Start: Add vpc user HADR_COMMON_VPCUSER.
Sun Mar 25 08:27:53 2018: Add vpc user HADR_COMMON_VPCUSER successfully.
Sun Mar 25 08:27:53 2018: End: Add vpc user HADR_COMMON_VPCUSER.
  2. Create two VPC users for the protected database on the downstream Recovery Appliance RAHADR2.
    # racli add db_user --user_name HADR_LOCAL_VPCUSER --user_type=vpc
[HADR_LOCAL_VPCUSER] New Password: ***********
Sun Mar 25 08:27:53 2018: Start: Add vpc user HADR_LOCAL_VPCUSER.
Sun Mar 25 08:27:53 2018: Add vpc user HADR_LOCAL_VPCUSER successfully.
Sun Mar 25 08:27:53 2018: End: Add vpc user HADR_LOCAL_VPCUSER.

# racli add db_user --user_name HADR_COMMON_VPCUSER --user_type=vpc
[HADR_COMMON_VPCUSER] New Password: ***********
Sun Mar 25 08:27:53 2018: Start: Add vpc user HADR_COMMON_VPCUSER.
Sun Mar 25 08:27:53 2018: Add vpc user HADR_COMMON_VPCUSER successfully.
Sun Mar 25 08:27:53 2018: End: Add vpc user HADR_COMMON_VPCUSER.
  3. If the VPC user account used by the replication server for sending backups from the upstream (HARADR1) to the downstream (RAHADR2) Recovery Appliances hasn't been created, create the VPC user now.
    # racli add db_user --user_name REPUSER_FROM_HADR1 --user_type=vpc
[REPUSER_FROM_HADR1] New Password: ***********

Sun Mar 25 08:35:01 2018: Start: Add vpc user REPUSER_FROM_HADR1.
Sun Mar 25 08:35:01 2018: Add vpc user REPUSER_FROM_HADR1 successfully.
Sun Mar 25 08:35:01 2018: End: Add vpc user REPUSER_FROM_HADR1.

Modifying Configuration for Transport Failover

This task modifies the Oracle network configuration files that are used for transparent failover to the downstream Recovery Appliance.

If you have a RAC database, this should be performed on each host where the protected database runs.
  1. Verify that there are no ${ORACLE_HOME}/dbs/ra${ORACLE_SID}.ora files on any of the hosts.
    This file has the effect of overriding all the configuration parameters defined in this step and should be removed if present.
  2. Configure a TNS alias in the tnsnames.ora file that will be used by RMAN to connect to the correct Recovery Appliance.
    $ cd ${ORACLE_HOME}/network/admin
  3. Edit tnsnames.ora and add the following entry:
    DR_RAHADR =
(DESCRIPTION_LIST =
  (LOAD_BALANCE = off)
  (FAILOVER = on)
  (DESCRIPTION =
    (CONNECT_TIMEOUT = 5)
    (TRANSPORT_CONNECT_TIMEOUT = 3)
    (RETRY_COUNT = 3)
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ra1ingest-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = rahadr1)
    )
  )
  (DESCRIPTION =
    (CONNECT_TIMEOUT = 5)
    (TRANSPORT_CONNECT_TIMEOUT = 3)
    (RETRY_COUNT = 3)
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ra2ingest-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = rahadr2)
    )
  )
)
DR_RAHADR1 =
(DESCRIPTION_LIST =
  (DESCRIPTION =
    (CONNECT_TIMEOUT = 5)
    (TRANSPORT_CONNECT_TIMEOUT = 3)
    (RETRY_COUNT = 3)
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ra1ingest-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = rahadr1)
    )
  )
)
DR_RAHADR2 =
(DESCRIPTION_LIST =
  (DESCRIPTION =
    (CONNECT_TIMEOUT = 5)
    (TRANSPORT_CONNECT_TIMEOUT = 3)
    (RETRY_COUNT = 3)
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ra2ingest-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = rahadr2)
    )
  )
)

Configuring the Replication Server

This task configures the replication server that sends the database backups from RAHADR1 to the RAHADR2 Recovery Appliance.

The operations and naming conventions used here are equivalent to those in Enterprise Manager when there is no dedicated replication network. For more information, see Replicating Backups with Recovery Appliance.
The replication server between RAHADR1 and RAHADR2 has not already been created.
  1. If a replication wallet does not exist on RAHADR1, create a replication wallet that points to RAHADR2.
    $ mkstore -wrl file:/dbfs_repdbfs/REPLICATION -createALO
  2. Add the credentials to the wallet. On RAHADR1, add the credentials for logging into RAHADR2. 
    $ mkstore -wrl file:/dbfs_repdbfs/REPLICATION 
-createCredential <rahadr2-scan>:1521/rahadr2 REPUSER_FROM_HADR1 my_v3ry_c0mplex_pa55w0rd
  3. Create the Recovery Appliance replication server on RAHADR1.
    $ sqlplus rasys/ra

SQL> exec dbms_ra.create_replication_server( 
replication_server_name =>'RAHADR2_REP', 
sbt_so_name => 'libra.so', max_streams => 8, 
catalog_user_name=> 'RASYS', 
wallet_alias => '<rahadr2-scan>:1521/rahadr2',
wallet_path => 'file:/dbfs_repdbfs/REPLICATION');

PL/SQL procedure successfully completed.

Configuring Upstream and Downstream Recovery Appliances

This task configures the protection policies for the protected database on the downstream and upstream Recovery Appliance, and then adds the protection policy to the replication server.

If a protection policy that is used, for example, by the CBR122DR database does not exist on the respective Recovery Appliances, these steps create them. The protection policy name does not have to be unique between the downstream and upstream Recovery Appliances.

To prevent a circular references between RAHADR1 and RAHADR2, the protection policy from RAHADR2 is not added to the replication server while the protection policy from RAHADR1 is added. All databases in the protection policy are replicated.

Note: Because RAHADR2 does not normally accept redo from the CDB122DR database, set the unprotected data window parameter is set to 1.25 days to avoid false alerts from occurring if the CDB122DR database is idle.

  1. Log into SQLPLS as rasys/ra on RAHADR2, the downstream Recovery Appliance. This step and the next few are performed on RAHADR2, unless otherwise stated.
    $ sqlplus rasys/ra
  2. Create the protection policy.
    SQL> exec dbms_ra.create_protection_policy( 
protection_policy_name => 'cdb122dr_PP', 
storage_location_name => 'DELTA', 
recovery_window_goal => numtodsinterval(3,'DAY'), 
unprotected_window => numtodsinterval(1.25,'DAY'), 
allow_backup_deletion => 'NO');

PL/SQL procedure successfully completed.
  3. Add the database (for this example) and its protection policy to the list of those protected by the Recovery Appliance.
    SQL> exec dbms_ra.add_db(
db_unique_name => 'cdb122dr', 
protection_policy_name=> 'cdb122dr_PP', 
reserved_space => '1T');

PL/SQL procedure successfully completed.
  4. Grant access to the replication user to the database (for this example).
    SQL> exec dbms_ra.grant_db_access(
username => 'REPUSER_FROM_HADR1',
db_unique_name => 'cdb122dr');

PL/SQL procedure successfully completed.
  5. Log into sqlplus as rasys/ra on RAHADR1, the upstream Recovery Appliance. This step and all that follow are performed on RAHADR1.
    $ sqlplus rasys/ra
  6. Create a protection policy. The protection policy name does not have to be unique.
    SQL> exec dbms_ra.create_protection_policy( 
protection_policy_name =>'cdb122dr_PP', 
storage_location_name => 'DELTA', 
recovery_window_goal => numtodsinterval(3,'DAY'), 
unprotected_window => numtodsinterval(5,'MINUTE'), 
allow_backup_deletion => 'NO');

PL/SQL procedure successfully completed.
  7. Add the database (for this example) and its protection policy to the list of those protected by the Recovery Appliance.
    SQL> exec dbms_ra.add_db(
db_unique_name => 'cdb122dr', 
protection_policy_name=> 'cdb122dr_PP', 
reserved_space => '1T');

PL/SQL procedure successfully completed.
  8. Grant access to the replication user to the database (for this example).
    SQL> exec dbms_ra.grant_db_access(
username => 'HADR_LOCAL_VPCUSER',
db_unique_name => 'cdb122dr');

PL/SQL procedure successfully completed.
  9. Add the protection policy to the replication server. This step is performed on the upstream Recovery Appliance (RAHADR1). This step was not performed on the downstream Recovery Appliance, in order to prevent a circular reference between the two Recovery Appliances.
    SQL> exec dbms_ra.add_replication_server( 
replication_server_name =>'RAHADR2_REP', 
protection_policy_name => 'cdb122dr_PP');

PL/SQL procedure successfully completed.

Registering the Protected Database on the Upstream Recovery Appliance

This task configures the wallet, adds VPC user credentials, tests those credentials, and registers the protected database with the upstream Recovery Appliance. If it is a RAC database, the steps need to be performed on each host where the protected database runs.

The operations and naming conventions used here are equivalent to those in Enterprise Manager.
  1. Configure the sqlnet.ora file that will be used by RMAN to connect to the correctRecovery Appliance. Go to the proper directory.
    $ cd ${ORACLE_HOME}/network/admin
  2. Edit the sqlnet.ora file and ensure the following parameters are set correctly:
    SQLNET.WALLET_OVERRIDE = true

NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)

WALLET_LOCATION =
  (SOURCE =
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY = /u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/zdlra)
    )
  )

SQLNET.EXPIRE_TIME = 10
  3. Create a replication wallet that stores each of the VPC user credentials. Perform this step only if the replication wallet doesn't already exist. On each host:
    $ mkstore -wrl file:/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/zdlra -
createALO
  4. Create credential aliases for each of the three credentials that will be used by RMAN. On each host, run the mkstore command. Enter the appropriate password when prompted.
    $ mkstore -wrl file:/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/zdlra -
createCredential dr_rahadr2 hadr_local_vpcuser hadr2_L0cal_Pa55w0rd

$ mkstore -wrl file:/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/zdlra -
createCredential dr_rahadr1 hadr_local_vpcuser hadr1_L0cal_Pa55w0rd

$ mkstore -wrl file:/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/zdlra -
createCredential dr_rahadr hadr_common_vpcuser c0mm0n_Pa55w0rd
  5. Verify the credentials are working correctly by logging into each target using only the credential alias. On each host, run the following:
    $ sqlplus /@dr_rahadr1
  6. Register the protected database with the Recovery Appliance in RAHADR1. On one of the hosts, run:
    $ rman target / catalog /@dr_rahadr1

RMAN> register database;
  7. Perform a test backup of the current control file to Recovery Appliance hadr1 (RAHADR1). On one of the protected database hosts, perform a backup of the current control file.
    $ rman target / catalog /@dr_rahadr1

RMAN> CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS
"SBT_LIBRARY=/u01/app/oracle/product/12.2.0.1/dbhome_1/lib/libra.so,
ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/z
dlra credential_alias=dr_rahadr1')";

RMAN> backup device type sbt current controlfile tag 'controltest';

Starting backup at 05-JUN-18
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=2320 instance=cdb122dr1 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: RA Library (RAHADR1)
SID=6DE9FE3D49ED4598E05311F3850AC59F
allocated channel: ORA_SBT_TAPE_2
channel ORA_SBT_TAPE_2: SID=2516 instance=cdb122dr1 device type=SBT_TAPE
channel ORA_SBT_TAPE_2: RA Library (RAHADR1)
SID=6DE9FE48D84C48C8E05311F3850A89BE
channel ORA_SBT_TAPE_1: starting full datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_SBT_TAPE_1: starting piece 1 at 05-JUN-18
channel ORA_SBT_TAPE_1: finished piece 1 at 05-JUN-18
piece handle=CDB122DR_2kt4m80u_1_1 tag=CONTROLTEST comment=API Version
2.0,MMS Version 3.17.1.26
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:15
Finished backup at 05-JUN-18
Starting Control File and SPFILE Autobackup at 05-JUN-18
piece handle=c-3244939197-20180605-00 comment=API Version 2.0,MMS Version
3.17.1.26
Finished Control File and SPFILE Autobackup at 05-JUN-18
  8. List the backup set just created. Verify there are two copies of the control file, one on Recovery Appliance hadr1 (RAHADR1) and the other on Recovery Appliance hadr2 (RAHADR2).
    RMAN> list backupset tag CONTROLTEST;

List of Backup Sets
===================
BS Key Type LV Size
------- ---- -- ----------
220 Full 138.75M
Control File Included: Ckp SCN: 9076177 Ckp time: 05-JUN-18
Backup Set Copy #1 of backup set 220
Device Type Elapsed Time Completion Time Compressed Tag
----------- ------------ --------------- ---------- ---
SBT_TAPE 07:00:21 05-JUN-18 NO CONTROLTEST
List of Backup Pieces for backup set 220 Copy #1
BP Key Pc# Status Media Piece Name
------- --- ----------- ----------------------- ----------
221 1 AVAILABLE Recovery Appliance (RAHADR1)
CDB122DR_2kt4m80u_1_1
Backup Set Copy #2 of backup set 220
Device Type Elapsed Time Completion Time Compressed Tag
----------- ------------ --------------- ---------- ---
SBT_TAPE 07:00:21 05-JUN-18 NO CONTROLTEST
List of Backup Pieces for backup set 220 Copy #2
BP Key Pc# Status Media Piece Name
------- --- ----------- ----------------------- ----------
246 1 AVAILABLE Recovery Appliance (RAHADR2)
RA_SBT_CDB122DR_3244939197_230_2kt4m80u_1_2_220

Adding Remaining Grants to the Upstream and Downstream Recovery Appliance

This task grants access to VPC users on both the upstream and downstream Recovery Appliances.

  1. On RAHADR1, add the grant access to the one remaining VPC users.
    SQL> exec dbms_ra.grant_db_access(
username => 'HADR_COMMON_VPCUSER',
db_unique_name => 'cdb122dr');

PL/SQL procedure successfully completed.
  2. On RAHADR2, add the grant access to the two remaining VPC users. These users are pre-setup in the event that backups failover, due to RAHADR1 not being available.
    SQL> exec dbms_ra.grant_db_access(
username => 'HADR_LOCAL_VPCUSER',
db_unique_name => 'cdb122dr');

PL/SQL procedure successfully completed.

SQL> exec dbms_ra.grant_db_access(
username => 'HADR_COMMON_VPCUSER',
db_unique_name => 'cdb122dr');

PL/SQL procedure successfully completed.
  3. Verify the credentials are working correctly by logging into each target using only the credential alias. On each host run:
    $ sqlplus /@dr_rahadr2

$ sqlplus /@dr_rahadr

Configuring Channel Device Parameters

This task configures the channel device parameters for use with the DR_RAHADR alias.

  1. On one of the protected database hosts, run: 
    $ rman target / catalog /@dr_rahadr1

RMAN> CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS
"SBT_LIBRARY=/u01/app/oracle/product/12.2.0.1/dbhome_1/lib/libra.so,
ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/z
dlra credential_alias=dr_rahadr')";
  2. (Optional) configure the following parameters, which are best practice recommendations.
    RMAN> CONFIGURE BACKUP OPTIMIZATION on;

RMAN> CONFIGURE CONTROLFILE AUTOBACKUP on;

RMAN> CONFIGURE DEFAULT DEVICE TYPE TO sbt;

RMAN> CONFIGURE DEVICE TYPE SBT_TAPE PARALLELISM 2 BACKUP TYPE TO BACKUPSET;

RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+RECOC1/cdb122dr/snapcf.f';

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO backed up 1 times to device
type sbt;