7 Scenario: Configure Oracle Data Guard Manually on the DCS Stack

This scenario describes setting up Oracle Data Guard on Oracle Database Appliance on the DCS stack.

Environment

Understand the primary and standby database environment topologies used in the subsequent Data Guard setup example using Oracle Database Appliance.

Description of manual_dg_configuration.png follows
Description of the illustration manual_dg_configuration.png
Component Primary Oracle Database Appliance Standby Oracle Database Appliance
Appliance Name appliance#1 appliance#2
Host Names proddb1, proddb2 stbydb1, stbydb2
Database Name chicago chicago
Database Unique Name chicago boston
Instance Name chicago1, chicago2 boston1, boston2
SCAN Name and IPs proddb-scan (10.1.27.2, 10.1.27.3) stbydb-scan (10.1.27.4, 10.1.27.5)
Grid Infrastructure Software Installation /u01/app/19.14.0.0/grid /u01/app/19.14.0.0/grid
Oracle Database Software Installation /u01/app/odaorahome/oracle/product/19.0.0.0/db_home1 /u01/app/odaorahome/oracle/product/19.0.0.0/db_home1
Database storage ASM ASM
ARCHIVELOG mode Yes Yes
FORCE LOGGING mode Yes Yes

Configuring Oracle Data Guard

Understand the steps to configure Oracle Data Guard manually. it is highly recommended to configure Oracle Data Guard with Oracle Data Guard Broker.

Follow these steps:
  1. Create Standby Redo Logs.
    Standby Redo Logs (SRLs) receives redo data from the primary database in real time minimizing transport and apply lag. In advance of the primary standby setup, Oracle recommends that standby redo logs be created on the primary database as well so that it is immediately ready to receive redo data following a Data Guard role transition. Create Standby Redo Logs (SRL) on the primary database. Each thread of the standby redo log must have at least one more redo log group than the corresponding thread of the online redo log. For example:
    SQL> alter database add standby logfile thread 1 group 7 size 1G, group 8 size 1G, group 9 size 1G, group 10 size 1G; 
    SQL> alter database add standby logfile thread 2 group 11 size 1G,group 12 size 1G, group 13 size 1G, group 14 size 1G;
    To check the number of online redo logs and their sizes, use the following query:
    SQL> select thread#, group#, bytes/1024/1024/1024 SIZE_IN_GB, status from v$log; 

    Note that the size of the standby redo logs must match the size of the redo logs. On the Oracle Database Appliance platform, the standby redo logs must be created on the REDO disk group which resides on the solid state disks. On Oracle Database Appliance Small/Medium/Large and on X8-2 HA models the control file and online logs are stored in RECO diskgroup as there is no REDO disk group. To validate the size of each log file and number of log groups in the standby redo log, use the following query:

     SQL> select group#, thread#, bytes/1024/1024/1024 SIZE_IN_GB from v$standby_log;
  2. Enable archivelog mode on primary database.

    Information:

    Archiving is the process of saving and protecting redo information in the form of archive files before the redo logs of an active database are overwritten in a circular manner. Databases created on Oracle Database Appliance have archiving turned on by default.
    Verify that the primary database is running in ARCHIVELOG mode.
    SQL> archive log list
    If the primary database is not running in ARCHIVELOG mode, then enable ARCHIVELOG mode as follows:
    1. Shut down both instances on Oracle Database Appliance.
      $ srvctl stop database -d chicago
    2. Start and mount one instance in exclusive mode.
      SQL> startup mount exclusive;
    3. Turn on archiving.
      SQL> alter database archivelog;
    4. Shut down the instance.
      SQL> shutdown immediate;
    5. Restart the database.
      $ srvctl start database -d chicago
  3. Enable FORCE LOGGING mode.
    Force logging enables you to capture database operations performed with the NOLOGGING attribute. This ensures integrity of your standby database. Verify if FORCE LOGGING has already been enabled on your primary database.
    SQL> select force_logging from v$database;
    If FORCE LOGGING is not enabled, then enable it using the following commands:
    SQL> alter database force logging;
  4. Configure Flashback Database feature.
    The Oracle Flashback Database feature provides a fast alternative to performing incomplete database recovery. Although using the Flashback Database feature is optional, it can be very useful for faster reinstating of the old primary database after a failover. Thus, if you do a failover to the standby and the old primary can be repaired, you do not have to rebuild the old primary database as a standby database but simply flashback and let Oracle Data Guard resynchronize from that point onwards. Check if the primary database has Flashback Database enabled, and if required, enable it.
    SQL> select flashback_on from v$database; 
    SQL> alter database flashback on;
    Note that enabling Flashback Database requires additional space consumption in the Fast Recovery Area, that is, RECO disk group. The space used by flashback logs can be controlled by setting the parameter DB_FLASHBACK_RETENTION_TARGET to a desired value. This value is specified in minutes. For example:
    SQL> alter system set DB_FLASHBACK_RETENTION_TARGET=120 scope=both sid='*';
  5. Enable standby file management.
    When the primary database adds or drops a datafile, the corresponding action must also be automatically taken on the standby database. This operation can be enabled using automated standby file management.
    SQL> alter system set STANDBY_FILE_MANAGEMENT=AUTO scope=both sid='*';
  6. Create the database home on the standby if it does not exist. For example:
    [root@stbydb1]# odacli create-dbhome -v 19.14.0.0.220118

    The database home version on the standby must be identical to database home version on the primary.

  7. Setup TNS entries and listeners.
    Oracle Net Service Names must be configured to enable redo transportation across the databases. Update tnsnames.ora file to include the TNS alias for both primary and standby databases. Note that in the Oracle Database Appliance, the tnsnames.ora file is located in the network/admin directory of the Oracle database home.
    $ vi $ORACLE_HOME/network/admin/tnsnames.ora
    Primary
    chicago = 
    (DESCRIPTION = 
     (ADDRESS = (PROTOCOL = TCP)(HOST = proddb-scan)(PORT = 1521))
     (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = chicago.oracle.com) 
      ) 
     )
    
    boston = 
    (DESCRIPTION = 
     (ADDRESS = (PROTOCOL = TCP)(HOST = stbydb-scan)(PORT = 1521)) 
     (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = boston.oracle.com) 
      ) 
     ) 
    Standby
    chicago = 
    (DESCRIPTION = 
     (ADDRESS = (PROTOCOL = TCP)(HOST = proddb-scan)(PORT = 1521)) 
     (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = chicago.oracle.com) 
      ) 
     ) 
    boston = 
    (DESCRIPTION = 
     (ADDRESS = (PROTOCOL = TCP)(HOST = stbydb-scan)(PORT = 1521)) 
    (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = boston.oracle.com) 
     ) )
  8. Setup Redo Transport Service in deferred mode. This step is not needed if Oracle Data Guard Broker is configured.
    The Oracle Data Guard redo transport mechanism uses Oracle Net connections to send the redo between the databases. Redo transport is enabled by setting the LOG_ARCHIVE_DEST_n parameter. For example, the following setup enables log shipping and uses LGWR based transmission in asynchronous mode.
    SQL> alter system set log_archive_dest_2='SERVICE=boston LGWR ASYNC REGISTER VALID_FOR=(online_logfile,primary_role) 
    REOPEN=60 DB_UNIQUE_NAME=boston' scope=both sid='*'; 
    SQL> alter system set log_archive_dest_state_2='defer' scope=both sid='*';

    For more information about redo log transmission options, see the Oracle Data Guard Concepts and Administration Guide.

  9. Setup Fetch Archive Log Server. This step is not needed if Oracle Data Guard Broker is configured.
    When the database is in standby role and the primary is unable to send any missing log files, then the standby database can use the FAL_SERVER setting to pull those missing log files. The FAL_SERVER parameter is uses the Oracle Net service name.
    SQL> alter system set FAL_SERVER=boston scope=both sid='*';
  10. Create a pfile from the spfile on the primary database.
    [oracle@proddb1]$ export ORACLE_HOME=u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 
    [oracle@proddb1]$ export ORACLE_SID=chicago1 
    [oracle@proddb1]$ export PATH=$ORACLE_HOME/bin:$PATH 
    [oracle@proddb1]$ sqlplus / as sysdba 
    SQL> create pfile='/tmp/chicago.pfile' from spfile;
  11. Add or modify the parameters on the primary and standby.

    Primary:

    *.cluster_database=TRUE 
    chicago2.instance_number=2 
    chicago1.instance_number=1 
    chicago2.thread=2 
    chicago1.thread=1 
    chicago2.undo_tablespace='UNDOTBS2' 
    chicago1.undo_tablespace='UNDOTBS1' 
    *.db_block_checking=FULL 
    *.db_block_checksum=FULL 
    *.db_lost_write_protect=TYPICAL ………. 
    *.db_unique_name=chicago 
    *.listener_networks='((NAME=net1)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=primary node0 vip)(PORT=1521)))))','((NAME=net1)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=primary node1 vip) (PORT=1521)))))','((NAME=net1)(REMOTE_LISTENER=primary scan name:1521))' 
    *.LOG_FILE_NAME_CONVERT='+REDO/BOSTON/','+REDO/CHICAGO/' *.DB_FILE_NAME_CONVERT='+DATA/BOSTON/','+DATA/CHICAGO/' *.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) MAX_FAILURE=1 REOPEN=5 DB_UNIQUE_NAME=chicago ALTERNATE=log_archive_dest_10'
    *.log_archive_dest_10='LOCATION=+DATA/db19c/arc10 VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=chicago ALTERNATE=log_archive_dest_1' 
    
    Add the following parameters:
    *.audit_file_dest='/u01/app/oracle/admin/chicago/adump' 
    *.fal_server='boston' *.remote_login_passwordfile='exclusive'

    Standby:

    *.cluster_database=TRUE 
    boston2.instance_number=2 
    boston1.instance_number=1 
    boston2.thread=2 boston1.thread=1 
    boston2.undo_tablespace='UNDOTBS2' 
    boston1.undo_tablespace='UNDOTBS1' 
    *.db_block_checking=FULL 
    *.db_block_checksum=FULL 
    *.db_lost_write_protect=TYPICAL ………. 
    *.db_unique_name=boston 
    *.listener_networks='((NAME=net1)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=standby node0 vip) (PORT=1521)))))','((NAME=net1)(LOCAL_LISTENER=( DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=standby node1 vip)(PORT=1521) ))))','((NAME=net1)(REMOTE_LISTENER=standby scan name:1521))' 
    *.LOG_FILE_NAME_CONVERT='+REDO/CHICAGO/','+REDO/BOSTON/' *.DB_FILE_NAME_CONVERT='+DATA/CHICAGO/','+DATA/BOSTON/' *.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) MAX_FAILURE=1 REOPEN=5 DB_UNIQUE_NAME=boston ALTERNATE=log_archive_dest_10'
    *.log_archive_dest_10='LOCATION=+DATA/db19c/arc10 VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=boston ALTERNATE=log_archive_dest_1' 
    
    Add the following parameters:
    *.audit_file_dest='/u01/app/oracle/admin/boston/adump' *.fal_server='chicago' 
    *.remote_login_passwordfile='exclusive'

    Set data protection parameters. Refer to My Oracle Support Note 1302539.1 - Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration On ODA Small/Medium/Large On Oracle Database Appliance X8-2-HA models, the controlfile and online logs are stored in the RECO disk group as there is no REDO disk group. Databases use listener_networks instead of local_listener and remote_listener parameters starting from Oracle Database Appliance release 19.6 on bare metal systems.

  12. Create storage structures for the database on the standby.
    [root@stbydb1]$ # odacli create-dbstorage -n chicago -u boston
    { 
    "jobId" : "054dac68-9efe-4f0d-a027-5515d46ada8a", 
    "status" : "Created", 
    "message" : null, 
    "reports" : [ ], 
    "createTimestamp" : "October 18, 2021 14:14:11 PM CEST", 
    "resourceList" : [ ], 
    "description" : "Database storage service creation with db name: chicago", 
    "updatedTime" : "October 18, 2021 14:14:11 PM CEST" 
    } 
    [root@stbydb1]# odacli describe-job -i "054dac68-9efe-4f0d-a027-5515d46ada8a"
    Job details 
    ---------------------------------------------------------------- 
    ID: 054dac68-9efe-4f0d-a027-5515d46ada8a 
    Description: Database storage service creation with db name: chicago 
    Status: Success
  13. Copy the password file from the primary database to the first standby system.
    [oracle@proddb1]$ srvctl config database -d chicago |grep Password Password file: +DATA/CHICAGO/PASSWORD/pwdchicago.386.1086365117
    [grid@proddb1 ~]$ asmcmd 
    ASMCMD> pwcopy +DATA/CHICAGO/PASSWORD/pwdchicago.386.1086365117 /tmp/pwdboston copying +DATA/CHICAGO/PASSWORD/pwdchicago.386.1086365117 -> /tmp/pwdboston 
    [grid@proddb1]$ scp /tmp/pwdboston oracle@stbydb1:/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1/dbs/orapwboston
  14. Copy the modified pfile to the first standby host and mount the standby database. Make a note of the path where the standby control file is created.
    [oracle@proddb1]$ scp /tmp/chicago.pfile oracle@stbydb1.oracle.com:/tmp/boston.pfile 
    [oracle@stbydb1]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 
    [oracle@stbydb1]$ export ORACLE_SID=boston1 
    [oracle@stbydb1]$ export PATH=$ORACLE_HOME/bin:$PATH 
    [oracle@stbydb1]$ cp /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1/dbs/orapwboston /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1/dbs/orapwboston1 
    [oracle@stbydb1]$ rman target / 
    RMAN> startup nomount pfile='/tmp/boston.pfile'; 
    RMAN> restore standby controlfile from service chicago; 
    Starting restore at 19-OCT-21 
    using target database control file 
    instead of recovery catalog 
    allocated channel: ORA_DISK_1 
    channel ORA_DISK_1: SID=483 instance=boston1 device type=DISK 
    channel ORA_DISK_1: starting datafile backup set restore 
    channel ORA_DISK_1: using network backup set from service chicago 
    channel ORA_DISK_1: restoring control file 
    channel ORA_DISK_1: restore complete, elapsed time: 00:00:02 output file name=+FLASH/BOSTON/CONTROLFILE/current.256.1086380745 
    Finished restore at 19-OCT-21
  15. Update the Control File parameter Edit the pfile /tmp/chicago.pfile and replace the control_files parameter to show the new path from the previous output. For example:
    control_files= ‘+FLASH/DB19CSTBY/CONTROLFILE/current.256.1086380745'
    [oracle@stbydb1]$ vi /tmp/boston.pfile
  16. Start the standby instance in nomount mode using the modified pfile. Create the spfile and restart the instance with the spfile.
    [oracle@stbydb1$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 [oracle@stbydb1$ export ORACLE_SID=boston1 
    [oracle@stbydb1$ export PATH=$ORACLE_HOME/bin:$PATH 
    [oracle@stbydb1]$ sqlplus / as sysdba 
    SQL> create spfile='+DATA/BOSTON/PARAMETERFILE/spfileboston' from pfile='/tmp/boston.pfile'; 
    SQL> !echo "spfile='+DATA/BOSTON/PARAMETERFILE/spfileboston'" > /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1/dbs/initboston1.ora 
    SQL> !echo "spfile='+DATA/BOSTON/PARAMETERFILE/spfileboston'" > /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1/dbs/initboston2.ora 
    SQL> startup mount force;
  17. Enable parallelism and set SECTION SIZE=64MB. To use parallelism during the restore, determine the number of CPUs on your server by running the following:
    [oracle@stbydb1]$ grep -c ^processor /proc/cpuinfo 20
    Make the following RMAN configuration changes on the standby database. The following example uses 8 preconfigured channels for RMAN to use during the recovery process.
    [oracle@stbydb1]$ rman target / 
    RMAN> CONFIGURE DEFAULT DEVICE TYPE TO DISK; 
    RMAN> CONFIGURE DEVICE TYPE DISK PARALLELISM 8;
  18. Restore the Standby Database from the primary database service Backing up a single large file in parallel, The multi section backup and restore capability in RMAN improves backup and recovery rates. RMAN divides the work among multiple channels and each channel acts upon a file section in a file. If you specify a small section size that would produce more than 256 sections, then RMAN increases the section size to a value that results in exactly 256 sections. The Section size clause depends on various factor such as network bandwidth, number of channels, sizes of data files, and application datafile sizes.
    oracle@stbydb1]$ sqlplus system/welcome1@chicago 
    SQL> select TABLESPACE_NAME, bytes/1024/1024/1024 SIZE_IN_GB from dba_data_files;
    TABLESPACE_NAME SIZE_IN_GB 
    ------------------------------ ---------- 
    UNDOTBS1 .102539063 
    SYSTEM .947265625 
    SYSAUX .91796875 
    UNDOTBS2 .024414063 
    USERS .004882813
    For example, when you run the following command on the standby system, you specify a backup section size of 64MB.
    [oracle@stbydb1]$ rman target / 
    RMAN> restore database from service chicago section size 64M; 
    RMAM> recover database from service chicago; 
    RMAN> backup spfile;
  19. Enable log shipping on the primary. Run this step only if you have not configured Oracle Data Guard Broker.
    [oracle@proddb1]$ sqlplus / as sysdba 
    SQL> alter system set log_archive_dest_state_2='enable' scope=both;
  20. Enable Flashback Database on the standby and adjust retention as required. Run this step only if you have not configured Oracle Data Guard Broker.
    SQL> alter database flashback on; 
    SQL> alter system set DB_FLASHBACK_RETENTION_TARGET=120;
  21. Start managed recovery on the standby. Run this step only if you have not configured Oracle Data Guard Broker.
    [oracle@stbydb1]$ sqlplus / as sysdba 
    SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
  22. Register the standby database with Oracle Clusterware.
    [oracle@stbydb1]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 
    [oracle@stbydb1]$ export ORACLE_SID=boston1 
    [oracle@stbydb1]$ export PATH=$ORACLE_HOME/bin:$PATH
    
    Example with single-instance Oracle Database:
    [oracle@stbydb1]$ srvctl add database -db boston -oraclehome /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 -dbtype SINGLE -instance boston1 -node stbydb1 -dbname chicago -diskgroup 'DATA,RECO,FLASH' -role physical_standby -spfile '+DATA/BOSTON/PARAMETERFILE/spfileboston' -startoption mount -acfspath '/u01/app/odaorahome,/u01/app/odaorabase0,/u01/app/odaorabase1'
    
    Example with Oracle RAC Database:
    [oracle@stbydb1]$ srvctl add database -db boston -oraclehome /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 -dbtype RAC -dbname chicago -diskgroup 'DATA,RECO,FLASH' -role physical_standby -spfile '+DATA/BOSTON/PARAMETERFILE/spfileboston' -startoption mount -acfspath '/u01/app/odaorahome,/u01/app/odaorabase0,/u01/app/odaorabase1' 
    [oracle@stbydb1]$ srvctl add instance -db boston -instance boston1 -node stbydb1 
    [oracle@stbydb1]$ srvctl add instance -db boston -instance boston2 -node stbydb2
  23. Copy the password file to Oracle ASM and verify that the password file points to Oracle ASM.
    [oracle@stbydb1]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 
    [oracle@stbydb1]$ export ORACLE_SID=boston1 
    [oracle@stbydb1]$ export PATH=$ORACLE_HOME/bin:$PATH 
    [oracle@stbydb1 ~]$ asmcmd --privilege sysdba
    ASMCMD> pwcopy /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1/dbs/orapwboston +DATA/DB19CSTBY/PASSWORDFILE/pwdboston --dbuniquename boston 
    copying /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1/dbs/orapwboston -> +DATA/DB19CSTBY/PASSWORDFILE/pwdboston 
    [oracle@stbydb1]$ srvctl config database -db boston|grep Password
  24. Set the parameters and create the Oracle Data Guard Broker configuration.
    Note: Flashback database is required to re-instantiate a failed primary after a failover role transition. Optionally enable flashback on both primary and standby. The standby database can begin using flashback with the PostCR script as follows:
    [oracle@stbydb1]$ sqlplus / as sysdba 
    connect / as sysdba 
    alter system set dg_broker_config_file1='+DATA/BOSTON/dr1.dat' scope=both; 
    alter system set dg_broker_config_file2='+DATA/BOSTON/dr2.dat' scope=both; 
    alter system set db_flashback_retention_target=120 scope=spfile; 
    alter database flashback on; 
    alter system set dg_broker_start=true; 
    [oracle@stbydb1]$ srvctl stop database -db boston 
    [oracle@stbydb1]$ srvctl start database -db boston -startoption mount 
    [oracle@stbydb1]$ sqlplus sys/welcome1@chicago as sysdba 
    alter system set dg_broker_config_file1='+DATA/CHICAGO/dr1.dat' scope=both; 
    alter system set dg_broker_config_file2='+DATA/CHICAGO/dr2.dat' scope=both; 
    alter system set dg_broker_start=TRUE;
    Wait 1 min 
    [oracle@stbydb1]$ dgmgrl sys/welcome1@chicago 
    CREATE CONFIGURATION dgconfig AS PRIMARY DATABASE IS CHICAGO CONNECT IDENTIFIER IS CHICAGO; 
    ADD DATABASE BOSTON AS CONNECT IDENTIFIER IS BOSTON ; 
    ENABLE CONFIGURATION
    If ALTER DATABASE FLASHBACK ON failed with ORA-38788, let the standby sync up and run the following steps to enable flashback after that:
    [oracle@stbydb1]$ sqlplus / as sysdba 
    SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; 
    SQL> alter database flashback on; 
    SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;
  25. Verify using SQL*Plus and SRVCTL.
    [oracle@stbydb1]$ srvctl config database -d chicago 
    [oracle@stbydb1]$ srvctl config database -d boston 
    [oracle@stbydb1]$ sqlplus / as sysdba 
    SQL> select FORCE_LOGGING, FLASHBACK_ON, OPEN_MODE, DATABASE_ROLE, SWITCHOVER_STATUS, DATAGUARD_BROKER, PROTECTION_MODE from v$database; 
    SQL> select PROCESS,PID,DELAY_MINS from V$MANAGED_STANDBY;
  26. Verify Oracle Data Guard using DGMGRL.
    $ dgmgrl DGMGRL> connect sys/welcome1@boston 
    DGMGRL> show configuration verbose 
    DGMGRL> show database verbose chicago 
    DGMGRL> show database verbose boston 
    DGMGRL> validate database chicago 
    DGMGRL> validate database boston
  27. Setup Oracle Clusterware Role Based Services. Refer to Client Failover Best Practices for Highly Available Oracle Databases.
  28. Register the databases:
    [oracle@stbydb1]$ dgmgrl sys/welcome1@boston as sysdba 
    DGMGRL> edit database 'boston' set state='apply-off'; Succeeded. 
    DGMGRL> sql 'ALTER DATABASE OPEN READ ONLY'; Succeeded.
    [oracle@stbydb1]# odacli list-databases DCS-10032:Resource database is not found.
    [oracle@stbydb1]# odacli register-database -c OLTP -s odb2 -sn boston.oracle.com -nn Public-network -t RAC Job details 
    ---------------------------------------------------------------- 
    ID: 841f99e0-a66f-4b23-b753-b04f992a6c33 Description: Discover Components : db [oracle@stbydb1]# odacli describe-job -i 841f99e0-a66f-4b23-b753-b04f992a6c33
    Job details 
    ---------------------------------------------------------------- 
    ID: 9947df75-e9f4-4a42-bcd7-ec23561a2f3f 
    Description: Database service registration with db service name: test.com 
    Status: Success 
    Created: February 18, 2022 12:52:04 PM CET 
    Message: Task Name Start Time End Time Status 
    ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Validate Hugepages For Register DB February 18, 2022 12:52:05 PM CET February 18, 2022 12:52:05 PM CET Success Enable OMF parameters February 18, 2022 12:52:06 PM CET February 18, 2022 12:52:07 PM CET Success Setting db character set February 18, 2022 12:52:07 PM CET February 18, 2022 12:52:07 PM CET Success Move Spfile to right location February 18, 2022 12:52:07 PM CET February 18, 2022 12:52:15 PM CET Success Enable DbSizing Template February 18, 2022 12:52:15 PM CET February 18, 2022 12:53:26 PM CET Success Running DataPatch February 18, 2022 12:53:26 PM CET February 18, 2022 12:53:28 PM CET Success Reset Associated Networks for Databse February 18, 2022 12:53:29 PM CET February 18, 2022 12:53:33 PM CET Success Reset Associated Networks February 18, 2022 12:53:33 PM CET February 18, 2022 12:53:33 PM CET Success
    [oracle@stbydb1]# odacli list-databases
    ID DB Name DB Type DB Version CDB Class Shape Storage Status DbHomeID 
    ---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ---------------------------------------- 9139ea53-449d-413a-841b-b157c084f3e0 bikazug RAC 19.14.0.0.220118 false OLTP odb2 ASM CONFIGURED 2afd69ed-f2cd-4345-9860-480f9e21f3ad
    [oracle@stbydb1]# odacli describe-database -i fbc4a32e-fec4-403d-b7b8-b08a3c01ab46
    Database details 
    ---------------------------------------------------------------- 
    ID: 9139ea53-449d-413a-841b-b157c084f3e0 
    Description: chicago 
    DB Name: chicago 
    DB Version: 19.14.0.0.220118 
    DB Type: RAC 
    DB Role: STANDBY DB 
    Target Node Name: 
    DB Edition: EE 
    DBID: 1128302500 Instance Only 
    Database: false 
    CDB: false 
    PDB Name: 
    PDB Admin User Name: 
    SEHA Enabled: false 
    Class: OLTP 
    Shape: odb2 
    Storage: ASM 
    DB Redundancy: MIRROR 
    CharacterSet: AL32UTF8 
    National CharacterSet: AL16UTF16 
    Language: AMERICAN 
    Territory: AMERICA 
    Home ID: 2afd69ed-f2cd-4345-9860-480f9e21f3ad 
    Console Enabled: false 
    TDE Wallet Management: 
    TDE Enabled: false 
    Level 0 Backup Day: 
    AutoBackup Enabled: true 
    Created: February 18, 2022 12:52:02 PM CET 
    DB Domain Name: 
    Associated Networks: Public-network 
    CPU Pool Name:
  29. Enable log shipping again and restart the standby database.
    [oracle@stbydb1]$ dgmgrl sys/welcome1@boston as sysdba 
    DGMGRL> edit database 'boston' set state='apply-on';
    Succeeded.
    [oracle@stbydb1]$ srvctl stop database -db boston 
    [oracle@stbydb1]$ srvctl start database -db boston
  30. Verify switchover operation with Oracle Data Guard:
    $ dgmgrl DGMGRL> connect sys/welcome1@boston 
    DGMGRL> switchover to boston
    DGMGRL> connect sys/welcome1@chicago 
    DGMGRL> switchover to chicago;
  31. Verify failover operation with Oracle Data Guard.

    Connect to standby before failover:

    $ dgmgrl DGMGRL> connect sys/welcome1@boston 
    DGMGRL> failover to boston 
    DGMGRL> reinstate database chicago

    Connect to former primary before failover:

    DGMGRL> connect sys/welcome1@chicago 
    DGMGRL> failover to chicago; 
    DGMGRL> reinstate database boston