1. Disaster Recovery on Oracle Database Appliance Using Oracle Data Guard
  2. Scenario: Upgrading Integrated Oracle Data Guard

6 Scenario: Upgrading Integrated Oracle Data Guard

This scenario describes how to upgrade Integrated Oracle Data Guard on Oracle Database Appliance on the DCS stack.

Note:

Starting with Oracle Database Appliance release 19.17, you cannot manage Oracle Database releases earlier than release 19c using ODACLI or BUI. Use Oracle Database Upgrade Assistance (DBUA) or the Auto Upgrade tool to upgrade your databases of releases Oracle Database 12.1, 12.2, 18c manually on Oracle Database Appliance release 19.17 or later. Similarly, upgrade Oracle Data Guard configurations on earlier releases.
  • Environment
    Understand the primary and standby database environment topologies used in the subsequent Data Guard setup example using Oracle Database Appliance.
  • Upgrading Integrated Oracle Data Guard
    This scenario describes upgrading a database with ODACLI that you configured Oracle Data Guard with odacli configure-dataguard or if you registered Oracle Data Guard using odacli register-dataguard.

Environment

Understand the primary and standby database environment topologies used in the subsequent Data Guard setup example using Oracle Database Appliance.

Description of oda_with_data_guard.png follows
Description of the illustration oda_with_data_guard.png
Component Primary Oracle Database Appliance Standby Oracle Database Appliance
Host Names proddb1, proddb2 stbydb1, stbydb2
Database Name croatia croatia
Database Unique Name zadar split
Instance Name croatia1, croatia2 croatia1, croatia2
Grid Infrastructure Software Installation /u01/app/19.19.0.0/grid /u01/app/19.19.0.0/grid
Source Oracle Database Software Installation /u01/app/odaorahome/oracle/product/12.1.0.2/db_home1 /u01/app/odaorahome/oracle/product/12.1.0.2/db_home1
Target Oracle Database Software Installation /u01/app/odaorahome/oracle/product/19.0.0.0/db_home5 /u01/app/odaorahome/oracle/product/19.0.0.0/db_home5

Parent topic: Scenario: Upgrading Integrated Oracle Data Guard

Upgrading Integrated Oracle Data Guard

This scenario describes upgrading a database with ODACLI that you configured Oracle Data Guard with odacli configure-dataguard or if you registered Oracle Data Guard using odacli register-dataguard.

Prerequisites:
  • Server and optionally, storage are already patched to the latest version on the source and target Oracle Database Appliance bare metal system.
  • Server must be on Oracle Database Appliance release 19.19 or later to ensure that when you run the odacli update-registry command, the DCS metadata is intact.
  • Repository is updated with the target database clone files on the source and target appliances.
Notes for patching server and storage:
  • Patch server and storage on the standby Oracle Database Appliance, switchover the database, patch the former primary server and storage, switch back the database
  • Server and storage prepatch reports provide information about whether local patching is applicable. If local patching is available, then patch the primary and the standby server and storage in a rolling fashion: node by node using the --local flag.
Follow these steps to upgrade the database:
  1. Verify the primary and standby Oracle Database Appliance systems. 
    [root@proddb1 ~]# odacli list-dataguardstatus
Updated about 9 day(s) ago
It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
855617b1-b2db-40e9-ba4f-60976ca96c0d     zadar_split                       croatia                PRIMARY    MAX_PERFORMANCE    0 seconds       0 seconds       421.00 KByte/s  CONFIGURED


[root@stbydb1 ~]# odacli list-dataguardstatus
Updated about 9 day(s) ago
It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
855617b1-b2db-40e9-ba4f-60976ca96c0d     zadar_split                       croatia                STANDBY    MAX_PERFORMANCE    0 seconds       0 seconds       361.00 KByte/s  CONFIGURED
  2. Disable Oracle Data Guard configuration on the primary. 
    [oracle@proddb1 ~]$ dgmgrl /
DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected as SYSDG.
DGMGRL> show configuration

Configuration - zadar_split

  Protection Mode: MaxPerformance
  Members:
  zadar  - Primary database
    split - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 59 seconds ago)

DGMGRL> disable configuration
Disabled.
DGMGRL> show configuration

Configuration - zadar_split

  Protection Mode: MaxPerformance
  Members:
  zadar  - Primary database
    split - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
DISABLED
  3. Stop Oracle Data Guard Broker for the primary and standby databases and query the configuration files location of the Oracle Data Guard Broker and back them up.

    Primary:

    [oracle@proddb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
[oracle@proddb1 ~]$ export ORACLE_SID=croatia1
[oracle@proddb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@proddb1 ~]$ sqlplus / as sysdba

SQL> ALTER SYSTEM SET DG_BROKER_START=FALSE scope=both;
SQL> show parameter dg_
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cell_offloadgroup_name string 
dg_broker_config_file1 string /u02/app/oracle/oradata/zadar/dbs/dr1zadarzadar.dat
dg_broker_config_file2 string /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat
dg_broker_start boolean FALSE

[oracle@proddb1 ~]$ cp /u02/app/oracle/oradata/zadar/dbs/dr1zadar.dat /u02/app/oracle/oradata/zadar/dbs/dr1zadar.dat.bkp 
[oracle@proddb1 ~]$ cp /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat.bkp
    Standby:
    [oracle@stdbydb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
[oracle@stdbydb1 ~]$ export ORACLE_SID=croatia1
[oracle@stdbydb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@stdbydb1 ~]$ sqlplus / as sysdba

SQL> ALTER SYSTEM SET DG_BROKER_START=FALSE scope=both;
SQL> show parameter dg_
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cell_offloadgroup_name string
dg_broker_config_file1 string /u02/app/oracle/oradata/split/dbs/dr1split.dat
dg_broker_config_file2 string /u02/app/oracle/oradata/split/dbs/dr2split.dat
dg_broker_start boolean FALSE

[oracle@stbydb1 ~]$ cp /u02/app/oracle/oradata/split/dbs/dr1split.dat /u02/app/oracle/oradata/split/dbs/dr1split.dat.bkp 
[oracle@stbydb1 ~]$ cp /u02/app/oracle/oradata/split/dbs/dr2split.dat /u02/app/oracle/oradata/split/dbs/dr2split.dat.bkp
  4. Create the destination DB home on the primary and the standby system.
    Primary:
    [root@proddb1 ~]# odacli create-dbhome -v 19.18.0.0.230117
[root@proddb1 ~]# odacli list-databases

[root@proddb1 ~]# odacli list-dbhomes
ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
f90adcc1-f64a-41ce-b72d-154db155b1fa     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
562a7428-9ea7-4878-9005-62c9d732a12b     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED
    Standby:
    [root@stbydb1 ~]# odacli list-dbhomes
ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
fe72fa84-b609-4cea-b040-4fd7308008c8     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
d5d63d8d-91c7-416e-a8af-3e957420aafa     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED
  5. Log in as operating system user oracle on the primary, and disable SSHCleanerJob and configure SSH user equivalence between both nodes for Oracle Database Appliance high-availability system.

    Primary:

    [root@proddb1 ~]# odacli list-schedules|grep "Name\|SSH"
ID                                       Name                      Description                                        CronExpression                 Disabled
44ad4fe2-4893-4c7d-a61c-15845cb74aa5     SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *             false

[root@proddb1 ~]# odacli modify-schedule -i 44ad4fe2-4893-4c7d-a61c-15845cb74aa5 -d
Modify job schedule success

[root@proddb1 ~]# odacli list-schedules|grep "Name\|SSH"
ID                                       Name                      Description                                        CronExpression                 Disabled
44ad4fe2-4893-4c7d-a61c-15845cb74aa5     SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *             true

[oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/deinstall/sshUserSetup.sh -user oracle -hosts "proddb1 proddb2" -noPromptPassphrase
  6. Create the autoupgrade configuration file as operating system user oracle on the primary on proddb1.
    [oracle@proddb1 ~]$ cat autoupgrade.conf
global.autoupg_log_dir=/u01/app/odaorabase/oracle/autoupgrade
upg1.dbname=croatia
upg1.start_time=NOW
upg1.source_home=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
upg1.target_home=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
upg1.sid=croatia1
upg1.log_dir=/u01/app/odaorabase/oracle/autoupgrade/croatia
upg1.upgrade_node=localhost
upg1.target_version=19
upg1.run_utlrp=yes
upg1.timezone_upg=no
  7. Change Fast Recovery Area to ACFS on the primary if the database was created on Oracle ASM as follows.
    Note: Run this step to avoid the error: 
    AutoUpgrade tool upg> "Database check failed with a runtime exception" (conName="CDB$ROOT", stage="PRECHECKS", checkName="DISK_SPACE_FOR_RECOVERY_AREA")
    1. Verify whether the dbstorage of the database is ACFS or ASM.
      [root@proddb1 ~]# odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
222a1d47-24ea-4a00-82f0-20d7fe17f59e     croatia    RAC      12.1.0.2.220719      true    OLTP     EE       odb2     ACFS     CONFIGURED   562a7428-9ea7-4878-9005-62c9d732a12b
8f90d26d-c17a-45e3-abbc-67c981c24a3f     hun        RAC      19.18.0.0.230117     true    OLTP     EE       odb1     ASM      CONFIGURED   f90adcc1-f64a-41ce-b72d-154db155b1fa

[root@proddb1 ~]# odacli list-dbhomes
ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
f90adcc1-f64a-41ce-b72d-154db155b1fa     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
562a7428-9ea7-4878-9005-62c9d732a12b     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED

[oracle@proddb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
[oracle@proddb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@proddb1 ~]$ export ORACLE_SID=croatia1
[oracle@proddb1 ~]$ sqlplus / as sysdba

SQL> show parameter db_recovery_file_dest

ACFS

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest             string     /u03/app/oracle/fast_recovery_area/
db_recovery_file_dest_size         big integer 53862M

ASM

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest             string     +RECO(FG$FILEGROUP_TEMPLATE_MIRROR)
db_recovery_file_dest_size         big integer 200G
    2. In case dbstorage is ASM, create a vmstorage temporarily or use an existing one with a slightly larger size than db_recovery_file_dest_size.
      [root@proddb1 ~]# odacli create-vmstorage -n tempfra -r mirror -s 250G
[root@proddb1 ~]# odacli list-vmstorages
Name                  Disk group       Volume name      Volume device                   Size        Used        Used %      Available   Mount Point                          Created                   Updated
--------------------  ---------------  ---------------  ------------------------------  ----------  ----------  ----------  ----------  -----------------------------------  ------------------------  ------------------------
tempfra               DATA             TEMPFRA          /dev/asm/tempfra-18             250.00 GB   1.09 GB     0.44%       248.91 GB   /u05/app/sharedrepo/tempfra          2023-04-18 19:05:55 CEST  2023-04-18 19:05:55 CEST
    3. Create a folder on the VM storage and change ownership of the folder to oracle:dba
      [root@proddb1 ~]# mkdir /u05/app/sharedrepo/tempfra/croatia
[root@proddb1 ~]# chown oracle:dba /u05/app/sharedrepo/tempfra/croatia
    4. Change db_recovery_file_dest to ACFS.
      SQL> alter system set db_recovery_file_dest='/u05/app/sharedrepo/tempfra/croatia/' scope=both;
  8. Run prechecks and review the findings. 
    [oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/jdk/bin/java -jar /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/rdbms/admin/autoupgrade.jar -config ~/autoupgrade.conf -mode analyze
AutoUpgrade 22.4.220712 launched with default internal options
Processing config file ...
+--------------------------------+
| Starting AutoUpgrade execution |
+--------------------------------+
1 CDB(s) plus 2 PDB(s) will be analyzed
Type 'help' to list console commands
upg> Job 100 completed
------------------- Final Summary --------------------
Number of databases            [ 1 ]

Jobs finished                  [1]
Jobs failed                    [0]

Please check the summary report at:
/u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
/u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log
  9. Run the AutoUpgrade tool in fixup mode to fix issues that may prevent a successful upgrade. 
    [oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/jdk/bin/java -jar /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/rdbms/admin/autoupgrade.jar -config ~/autoupgrade.conf -mode fixups
AutoUpgrade 22.4.220712 launched with default internal options
Processing config file ...
+--------------------------------+
| Starting AutoUpgrade execution |
+--------------------------------+
1 CDB(s) plus 2 PDB(s) will be processed
Type 'help' to list console commands
upg> Job 101 completed
------------------- Final Summary --------------------
Number of databases            [ 1 ]

Jobs finished                  [1]
Jobs failed                    [0]

Please check the summary report at:
/u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
/u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log
  10. Re-run prechecks and review the findings.
    [oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/jdk/bin/java -jar /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/rdbms/admin/autoupgrade.jar -config ~/autoupgrade.conf -mode analyze
AutoUpgrade 22.4.220712 launched with default internal options
Processing config file ...
+--------------------------------+
| Starting AutoUpgrade execution |
+--------------------------------+
1 CDB(s) plus 2 PDB(s) will be analyzed
Type 'help' to list console commands
upg> Job 102 completed
------------------- Final Summary --------------------
Number of databases            [ 1 ]

Jobs finished                  [1]
Jobs failed                    [0]

Please check the summary report at:
/u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
/u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log
  11. Upgrade the database.
    [oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/jdk/bin/java -jar /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/rdbms/admin/autoupgrade.jar -config ~/autoupgrade.conf -mode deploy

AutoUpgrade 22.4.220712 launched with default internal options
Processing config file ...
+--------------------------------+
| Starting AutoUpgrade execution |
+--------------------------------+
1 CDB(s) plus 2 PDB(s) will be processed
Type 'help' to list console commands

upg> Job 103 completed
------------------- Final Summary --------------------
Number of databases            [ 1 ]

Jobs finished                  [1]
Jobs failed                    [0]
Jobs restored                  [0]
Jobs pending                   [0]

---- Drop GRP at your convenience once you consider it is no longer needed ----
Drop GRP from croatia1: drop restore point AUTOUPGRADE_9212_ZADAR121020

Please check the summary report at:
/u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
/u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log

Note: once you confirmed that database was running as expected from all perspective and downgrade surely would not be needed, drop the restore point which was created by AutoUpgrade tool as the output also instructed

---- Drop GRP at your convenience once you consider it is no longer needed ----
Drop GRP from croatia1: drop restore point AUTOUPGRADE_9212_ZADAR121020
  12. Upgrade the standby database.

    Standby:

    [oracle@stbydb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
[oracle@stbydb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@stbydb1 ~]$ srvctl stop database -d split

[oracle@stbydb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
[oracle@stbydb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@stbydb1 ~]$ srvctl upgrade database -d split -oraclehome /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5

[oracle@stbydb1 ~]$ srvctl start database -d split
  13. Enable Oracle Data Guard Broker on the primary and on the standby.

    Primary:

    [oracle@proddb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
[oracle@proddb1 ~]$ export ORACLE_SID=croatia1
[oracle@proddb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@proddb1 ~]$ sqlplus / as sysdba
SQL> alter system set dg_broker_start=true scope=both;

    Standby:

    [oracle@stbydb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
[oracle@stbydb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@stbydb1 ~]$ srvctl upgrade database -d split -oraclehome /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5

[oracle@stbydb1 ~]$ sqlplus / as sysdba
SQL> alter system set dg_broker_start=true scope=both;
  14. Stop the database on the primary and the standby.

    Primary:

    [oracle@proddb1 ~]$ srvctl stop database -d

    Standby:

    [oracle@stbydb1 ~]$ srvctl stop database -d
  15. Copy the tnsnames.ora and sqlnet.ora files from the old home to the new home on the primary and the standby.

    Primary:

    [oracle@proddb1 ~]$ cp /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1/network/admin/* /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/network/admin/

    Standby:

    [oracle@stbydb1 ~]$ cp /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1/network/admin/* /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/network/admin/
  16. Restore Oracle Data Guard Broker configuration files on both sides.

    Primary:

    [oracle@proddb1 ~]$ cp /u02/app/oracle/oradata/zadar/dbs/dr1zadar.dat.bkp /u02/app/oracle/oradata/zadar/dbs/dr1zadar.dat
[oracle@proddb1 ~]$ cp /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat.bkp /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat

    Standby:

    [oracle@stbydb1 ~]$ cp /u02/app/oracle/oradata/split/dbs/dr1split.dat.bkp /u02/app/oracle/oradata/split/dbs/dr1split.dat
[oracle@stbydb1 ~]$ cp /u02/app/oracle/oradata/split/dbs/dr2split.dat.bkp /u02/app/oracle/oradata/split/dbs/dr2split.dat
  17. Start primary and standby databases.

    Primary:

    [oracle@proddb1 ~]$ srvctl start database -d zadar

    Standby:

    [oracle@stbydb1 ~]$ srvctl start database -d split
  18. Enable Oracle Data Guard configuration on the primary.

    Primary:

    [oracle@proddb1 ~]$ dgmgrl /
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Tue Apr 18 11:48:58 2023
Version 19.18.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "zadar"
Connected as SYSDG.
DGMGRL> show configuration

Configuration - zadar_split

  Protection Mode: MaxPerformance
  Members:
  zadar - Primary database
    split - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
DISABLED

DGMGRL> enable configuration
Enabled.

DGMGRL> show configuration

Configuration - zadar_split

  Protection Mode: MaxPerformance
  Members:
  zadar - Primary database
    split - Physical standby database
      Warning: ORA-16853: apply lag has exceeded specified threshold

Fast-Start Failover:  Disabled

Configuration Status:
WARNING   (status updated 17 seconds ago)

DGMGRL> show database split;

Database - split

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 1 second ago)
  Apply Lag:          53 minutes 54 seconds (computed 1 second ago)
  Average Apply Rate: 34.18 MByte/s
  Real Time Query:    OFF
  Instance(s):
    croatia1 (apply instance)
    croatia2

  Database Warning(s):
    ORA-16853: apply lag has exceeded specified threshold

Database Status:
WARNING

Note: It might take some time to sync up the standby

DGMGRL> show database split;

Database - split

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 0 seconds ago)
  Apply Lag:          0 seconds (computed 0 seconds ago)
  Average Apply Rate: 16.42 MByte/s
  Real Time Query:    OFF
  Instance(s):
    croatia1 (apply instance)
    croatia2

Database Status:
SUCCESS
  19. Enable SSHCleanerJob and remove SSH user equivalence between both nodes for Oracle Database Appliance high-availability system on the primary.
    root@proddb1 ~]# odacli list-schedules|grep "Name\|SSH"
ID                                       Name                      Description                                        CronExpression                 Disabled
44ad4fe2-4893-4c7d-a61c-15845cb74aa5     SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *             true

[root@proddb1 ~]# odacli modify-schedule -i 44ad4fe2-4893-4c7d-a61c-15845cb74aa5 -e
Modify job schedule success

[root@proddb1 ~]# odacli list-schedules|grep "Name\|SSH"
ID                                       Name                      Description                                        CronExpression                 Disabled
44ad4fe2-4893-4c7d-a61c-15845cb74aa5     SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *             false

    Remove local and remote node from /home/oracle/.ssh/authorized_keys files on both nodes as operating system user oracle using the vi command. 

    [oracle@proddb1 ~]$ vi /home/oracle/.ssh/authorized_keys
[oracle@proddb2 ~]$ vi /home/oracle/.ssh/authorized_keys

    Remove all id* files under /home/oracle/.ssh as operating system user oracle on both nodes. 

    [oracle@proddb1 ~]$ rm /home/oracle/.ssh/id*
[oracle@proddb2 ~]$ rm /home/oracle/.ssh/id*
  20. Sync up DCS metadata on the primary.

    Primary:

    [root@proddb1 ~]# odacli list-dbhomes
ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
f90adcc1-f64a-41ce-b72d-154db155b1fa     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
562a7428-9ea7-4878-9005-62c9d732a12b     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED
[root@proddb1 ~]# odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
222a1d47-24ea-4a00-82f0-20d7fe17f59e     croatia    RAC      12.1.0.2.220719      true    OLTP     EE       odb2     ACFS     CONFIGURED   562a7428-9ea7-4878-9005-62c9d732a12b

[root@proddb1 ~]# odacli update-registry -n db -u zadar

Job details
----------------------------------------------------------------
                     ID:  fc54b821-c407-4174-8a1a-c90ba66e6cd2
            Description:  Discover Components : db
                 Status:  Created
                Created:  April 18, 2023 12:04:55 PM CEST
                Message:

Task Name                                Node Name                 Start Time                          End Time                            Status
---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------

[root@proddb1 ~]# odacli describe-job -i fc54b821-c407-4174-8a1a-c90ba66e6cd2

Job details
----------------------------------------------------------------
                     ID:  fc54b821-c407-4174-8a1a-c90ba66e6cd2
            Description:  Discover Components : db
                 Status:  Success
                Created:  April 18, 2023 12:04:55 PM CEST
                Message:

Task Name                                Node Name                 Start Time                          End Time                            Status
---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------
Discover DBHome                          proddb1                April 18, 2023 12:05:01 PM CEST     April 18, 2023 12:05:04 PM CEST     Success
Discover DBHome                          proddb1                April 18, 2023 12:05:04 PM CEST     April 18, 2023 12:05:07 PM CEST     Success
Discover DBHome                          proddb1                April 18, 2023 12:05:07 PM CEST     April 18, 2023 12:05:09 PM CEST     Success
Discover DB: zadar                       proddb1                April 18, 2023 12:05:09 PM CEST     April 18, 2023 12:05:20 PM CEST     Success

[root@proddb1 ~]# odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
222a1d47-24ea-4a00-82f0-20d7fe17f59e     croatia    RAC      19.18.0.0.230117     true    OLTP     EE       odb2     ACFS     CONFIGURED   f90adcc1-f64a-41ce-b72d-154db155b1fa
  21. Sync up DCS metadata on the standby.

    Standby:

    [root@stbydb1 ~]# odacli list-dbhomes
ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
fe72fa84-b609-4cea-b040-4fd7308008c8     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
d5d63d8d-91c7-416e-a8af-3e957420aafa     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED

[root@stbydb1 ~]# odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
a85a0120-2343-4a42-af5b-93b958353c38     croatia    RAC      12.1.0.2.220719      true    OLTP     EE       odb2     ACFS     CONFIGURED   d5d63d8d-91c7-416e-a8af-3e957420aafa

[root@stbydb1 ~]# odacli update-registry -n db -u split

Job details
----------------------------------------------------------------
                     ID:  4694f9d9-a569-433d-9d76-69b3b8b9ddcf
            Description:  Discover Components : db
                 Status:  Created
                Created:  April 18, 2023 10:08:03 AM GMT
                Message:

Task Name                                Node Name                 Start Time                          End Time                            Status
---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------

[root@stbydb1 ~]# odacli describe-job -i 4694f9d9-a569-433d-9d76-69b3b8b9ddcf

Job details
----------------------------------------------------------------
                     ID:  4694f9d9-a569-433d-9d76-69b3b8b9ddcf
            Description:  Discover Components : db
                 Status:  Success
                Created:  April 18, 2023 10:08:03 AM GMT
                Message:

Task Name                                Node Name                 Start Time                          End Time                            Status
---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------
Discover DBHome                          stbydb1                April 18, 2023 10:08:15 AM GMT      April 18, 2023 10:08:19 AM GMT      Success
Discover DBHome                          stbydb1                April 18, 2023 10:08:19 AM GMT      April 18, 2023 10:08:22 AM GMT      Success
Discover DB: split                       stbydb1                April 18, 2023 10:08:22 AM GMT      April 18, 2023 10:08:35 AM GMT      Success

[root@stbydb1 ~]# odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
a85a0120-2343-4a42-af5b-93b958353c38     croatia    RAC      19.18.0.0.230117     true    OLTP     EE       odb2     ACFS     CONFIGURED   fe72fa84-b609-4cea-b040-4fd7308008c8
  22. Check Oracle Data Guard status using the odacli list-dataguardstatus command to get the latest status.

    Primary:

    [root@proddb1 ~]# odacli list-dataguardstatus
Updated about 35 day(s) ago
It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
54b4390a-5078-4e5d-8cef-53888d6b7b16     zadar_split                      croatia              PRIMARY    MAX_PERFORMANCE    ---             ---             ---             STALE

[root@proddb1 ~]# odacli describe-dataguardstatus -i 54b4390a-5078-4e5d-8cef-53888d6b7b16 -s
Updated about 1 minute(s) ago
It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
Dataguard Status details
----------------------------------------------------------------
                     ID: 54b4390a-5078-4e5d-8cef-53888d6b7b16
                   Name: zadar_split
            Database ID: 222a1d47-24ea-4a00-82f0-20d7fe17f59e
                   Role: PRIMARY
        Protection Mode: MAX_PERFORMANCE
              Apply Lag: 0 seconds
          Transport Lag: 0 seconds
             Apply Rate: 10.83 MByte/s
                 Status: CONFIGURED
           Updated Time: April 18, 2023 12:11:44 PM CEST

[root@stbydb1 ~]# odacli list-dataguardstatus
Updated about 35 day(s) ago
It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
54b4390a-5078-4e5d-8cef-53888d6b7b16     zadar_split                      croatia              STANDBY    MAX_PERFORMANCE    0 seconds       0 seconds       8.00 KByte/s    CONFIGURED

    Standby:

    [root@stbydb1 ~]# odacli describe-dataguardstatus -i 54b4390a-5078-4e5d-8cef-53888d6b7b16 -s
Updated about 26 second(s) ago
It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
Dataguard Status details
----------------------------------------------------------------
                     ID: 54b4390a-5078-4e5d-8cef-53888d6b7b16
                   Name: zadar_split
            Database ID: a85a0120-2343-4a42-af5b-93b958353c38
                   Role: STANDBY
        Protection Mode: MAX_PERFORMANCE
              Apply Lag: 0 seconds
          Transport Lag: 0 seconds
             Apply Rate: 9.70 MByte/s
                 Status: CONFIGURED
           Updated Time: April 18, 2023 10:13:31 AM GMT

[root@stbydb1 ~]# odacli list-dataguardstatus
Updated about 35 day(s) ago
It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
54b4390a-5078-4e5d-8cef-53888d6b7b16     zadar_split                      croatia              STANDBY    MAX_PERFORMANCE    0 seconds       0 seconds       1.72 MByte/s    CONFIGURED
  23. If the db_recovery_file_dest file was changed in step 7, revert that change.
    [oracle@proddb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
[oracle@proddb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@proddb1 ~]$ export ORACLE_SID=croatia1
[oracle@proddb1 ~]$ sqlplus / as sysdba

SQL> alter system set db_recovery_file_dest='+RECO(FG$FILEGROUP_TEMPLATE_MIRROR)' scope=both;
  24. Take a full backup.
    [oracle@proddb1 ~]# odacli create-backup -n croatia -bt Regular-L0
  25. Remove the temporary VM storage.
    [oracle@proddb1 ~]# odacli delete-vmstorage -n tempfra
  26. Test switchover, failover, and reinstate operations after the upgrade using ODACLI commands.

Parent topic: Scenario: Upgrading Integrated Oracle Data Guard