Previous Next JavaScript must be enabled to correctly display this content

## 6 Scenario: Upgrading Integrated Oracle Data Guard

This scenario describes how to upgrade Integrated Oracle Data Guard on Oracle Database Appliance on the DCS stack.

Note:

Starting with Oracle Database Appliance release 19.17, you cannot manage Oracle Database releases earlier than release 19c using ODACLI or BUI. Use Oracle Database Upgrade Assistance (DBUA) or the Auto Upgrade tool to upgrade your databases of releases Oracle Database 12.1, 12.2, 18c manually on Oracle Database Appliance release 19.17 or later. Similarly, upgrade Oracle Data Guard configurations on earlier releases.

- Environment
  Understand the primary and standby database environment topologies used in the subsequent Data Guard setup example using Oracle Database Appliance.
- Upgrading Integrated Oracle Data Guard
  This scenario describes upgrading a database with ODACLI that you configured Oracle Data Guard with `odacli configure-dataguard` or if you registered Oracle Data Guard using `odacli register-dataguard`.

### Environment

Understand the primary and standby database environment topologies used in the subsequent Data Guard setup example using Oracle Database Appliance.

Description of the illustration oda_with_data_guard.png

Component | Primary Oracle Database Appliance | Standby Oracle Database Appliance
--- | --- | ---
Host Names | proddb1, proddb2 | stbydb1, stbydb2
Database Name | croatia | croatia
Database Unique Name | zadar | split
Instance Name | croatia1, croatia2 | croatia1, croatia2
Grid Infrastructure Software Installation | /u01/app/19.19.0.0/grid | /u01/app/19.19.0.0/grid
Source Oracle Database Software Installation | /u01/app/odaorahome/oracle/product/12.1.0.2/db_home1 | /u01/app/odaorahome/oracle/product/12.1.0.2/db_home1
Target Oracle Database Software Installation | /u01/app/odaorahome/oracle/product/19.0.0.0/db_home5 | /u01/app/odaorahome/oracle/product/19.0.0.0/db_home5

Parent topic: Scenario: Upgrading Integrated Oracle Data Guard

### Upgrading Integrated Oracle Data Guard

This scenario describes upgrading a database with ODACLI that you configured Oracle Data Guard with `odacli configure-dataguard` or if you registered Oracle Data Guard using `odacli register-dataguard`.

Prerequisites:

- Server and optionally, storage are already patched to the latest version on the source and target Oracle Database Appliance bare metal system.
- Server must be on Oracle Database Appliance release 19.19 or later to ensure that when you run the `odacli update-registry` command, the DCS metadata is intact.
- Repository is updated with the target database clone files on the source and target appliances.

Notes for patching server and storage:

- Patch server and storage on the standby Oracle Database Appliance, switchover the database, patch the former primary server and storage, switch back the database
- Server and storage prepatch reports provide information about whether local patching is applicable. If local patching is available, then patch the primary and the standby server and storage in a rolling fashion: node by node using the `--local` flag.

Follow these steps to upgrade the database:

1. Verify the primary and standby Oracle Database Appliance systems.

       [root@proddb1 ~]# odacli list-dataguardstatus
       Updated about 9 day(s) ago
       It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
       ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
       ---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
       855617b1-b2db-40e9-ba4f-60976ca96c0d     zadar_split                       croatia                PRIMARY    MAX_PERFORMANCE    0 seconds       0 seconds       421.00 KByte/s  CONFIGURED

       [root@stbydb1 ~]# odacli list-dataguardstatus
       Updated about 9 day(s) ago
       It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
       ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
       ---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
       855617b1-b2db-40e9-ba4f-60976ca96c0d     zadar_split                       croatia                STANDBY    MAX_PERFORMANCE    0 seconds       0 seconds       361.00 KByte/s  CONFIGURED

2. Disable Oracle Data Guard configuration on the primary.

       [oracle@proddb1 ~]$ dgmgrl /
       DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

       Copyright (c) 2000, 2013, Oracle. All rights reserved.

       Welcome to DGMGRL, type "help" for information.
       Connected as SYSDG.
       DGMGRL> show configuration

       Configuration - zadar_split

         Protection Mode: MaxPerformance
         Members:
         zadar  - Primary database
           split - Physical standby database

       Fast-Start Failover: DISABLED

       Configuration Status:
       SUCCESS   (status updated 59 seconds ago)

       DGMGRL> disable configuration
       Disabled.
       DGMGRL> show configuration

       Configuration - zadar_split

         Protection Mode: MaxPerformance
         Members:
         zadar  - Primary database
           split - Physical standby database

       Fast-Start Failover: DISABLED

       Configuration Status:
       DISABLED

3. Stop Oracle Data Guard Broker for the primary and standby databases and query the configuration files location of the Oracle Data Guard Broker and back them up.

   Primary:

       [oracle@proddb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
       [oracle@proddb1 ~]$ export ORACLE_SID=croatia1
       [oracle@proddb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
       [oracle@proddb1 ~]$ sqlplus / as sysdba

       SQL> ALTER SYSTEM SET DG_BROKER_START=FALSE scope=both;
       SQL> show parameter dg_
       NAME TYPE VALUE
       ------------------------------------ ----------- ------------------------------
       cell_offloadgroup_name string
       dg_broker_config_file1 string /u02/app/oracle/oradata/zadar/dbs/dr1zadarzadar.dat
       dg_broker_config_file2 string /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat
       dg_broker_start boolean FALSE

       [oracle@proddb1 ~]$ cp /u02/app/oracle/oradata/zadar/dbs/dr1zadar.dat /u02/app/oracle/oradata/zadar/dbs/dr1zadar.dat.bkp
       [oracle@proddb1 ~]$ cp /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat.bkp

   Standby:

       [oracle@stdbydb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
       [oracle@stdbydb1 ~]$ export ORACLE_SID=croatia1
       [oracle@stdbydb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
       [oracle@stdbydb1 ~]$ sqlplus / as sysdba

       SQL> ALTER SYSTEM SET DG_BROKER_START=FALSE scope=both;
       SQL> show parameter dg_
       NAME TYPE VALUE
       ------------------------------------ ----------- ------------------------------
       cell_offloadgroup_name string
       dg_broker_config_file1 string /u02/app/oracle/oradata/split/dbs/dr1split.dat
       dg_broker_config_file2 string /u02/app/oracle/oradata/split/dbs/dr2split.dat
       dg_broker_start boolean FALSE

       [oracle@stbydb1 ~]$ cp /u02/app/oracle/oradata/split/dbs/dr1split.dat /u02/app/oracle/oradata/split/dbs/dr1split.dat.bkp
       [oracle@stbydb1 ~]$ cp /u02/app/oracle/oradata/split/dbs/dr2split.dat /u02/app/oracle/oradata/split/dbs/dr2split.dat.bkp

4. Create the destination DB home on the primary and the standby system.
   Primary:

       [root@proddb1 ~]# odacli create-dbhome -v 19.18.0.0.230117
       [root@proddb1 ~]# odacli list-databases

       [root@proddb1 ~]# odacli list-dbhomes
       ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
       ---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
       f90adcc1-f64a-41ce-b72d-154db155b1fa     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
       562a7428-9ea7-4878-9005-62c9d732a12b     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED

   Standby:

       [root@stbydb1 ~]# odacli list-dbhomes
       ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
       ---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
       fe72fa84-b609-4cea-b040-4fd7308008c8     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
       d5d63d8d-91c7-416e-a8af-3e957420aafa     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED

5. Log in as operating system user `oracle` on the primary, and disable `SSHCleanerJob` and configure SSH user equivalence between both nodes for Oracle Database Appliance high-availability system.

   Primary:

       [root@proddb1 ~]# odacli list-schedules|grep "Name\|SSH"
       ID                                       Name                      Description                                        CronExpression                 Disabled
       44ad4fe2-4893-4c7d-a61c-15845cb74aa5     SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *             false

       [root@proddb1 ~]# odacli modify-schedule -i 44ad4fe2-4893-4c7d-a61c-15845cb74aa5 -d
       Modify job schedule success

       [root@proddb1 ~]# odacli list-schedules|grep "Name\|SSH"
       ID                                       Name                      Description                                        CronExpression                 Disabled
       44ad4fe2-4893-4c7d-a61c-15845cb74aa5     SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *             true

       [oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/deinstall/sshUserSetup.sh -user oracle -hosts "proddb1 proddb2" -noPromptPassphrase

6. Create the autoupgrade configuration file as operating system user `oracle` on the primary on `proddb1`.

       [oracle@proddb1 ~]$ cat autoupgrade.conf
       global.autoupg_log_dir=/u01/app/odaorabase/oracle/autoupgrade
       upg1.dbname=croatia
       upg1.start_time=NOW
       upg1.source_home=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
       upg1.target_home=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
       upg1.sid=croatia1
       upg1.log_dir=/u01/app/odaorabase/oracle/autoupgrade/croatia
       upg1.upgrade_node=localhost
       upg1.target_version=19
       upg1.run_utlrp=yes
       upg1.timezone_upg=no

7. Change Fast Recovery Area to ACFS on the primary if the database was created on Oracle ASM as follows.
   Note: Run this step to avoid the error:

       AutoUpgrade tool upg> "Database check failed with a runtime exception" (conName="CDB$ROOT", stage="PRECHECKS", checkName="DISK_SPACE_FOR_RECOVERY_AREA")

   1. Verify whether the dbstorage of the database is ACFS or ASM.

          [root@proddb1 ~]# odacli list-databases
          ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
          ---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
          222a1d47-24ea-4a00-82f0-20d7fe17f59e     croatia    RAC      12.1.0.2.220719      true    OLTP     EE       odb2     ACFS     CONFIGURED   562a7428-9ea7-4878-9005-62c9d732a12b
          8f90d26d-c17a-45e3-abbc-67c981c24a3f     hun        RAC      19.18.0.0.230117     true    OLTP     EE       odb1     ASM      CONFIGURED   f90adcc1-f64a-41ce-b72d-154db155b1fa

          [root@proddb1 ~]# odacli list-dbhomes
          ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
          ---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
          f90adcc1-f64a-41ce-b72d-154db155b1fa     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
          562a7428-9ea7-4878-9005-62c9d732a12b     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED

          [oracle@proddb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
          [oracle@proddb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
          [oracle@proddb1 ~]$ export ORACLE_SID=croatia1
          [oracle@proddb1 ~]$ sqlplus / as sysdba

          SQL> show parameter db_recovery_file_dest

          ACFS

          NAME                     TYPE     VALUE
          ------------------------------------ ----------- ------------------------------
          db_recovery_file_dest             string     /u03/app/oracle/fast_recovery_area/
          db_recovery_file_dest_size         big integer 53862M

          ASM

          NAME                     TYPE     VALUE
          ------------------------------------ ----------- ------------------------------
          db_recovery_file_dest             string     +RECO(FG$FILEGROUP_TEMPLATE_MIRROR)
          db_recovery_file_dest_size         big integer 200G

   2. In case dbstorage is ASM, create a vmstorage temporarily or use an existing one with a slightly larger size than `db_recovery_file_dest_size`.

          [root@proddb1 ~]# odacli create-vmstorage -n tempfra -r mirror -s 250G
          [root@proddb1 ~]# odacli list-vmstorages
          Name                  Disk group       Volume name      Volume device                   Size        Used        Used %      Available   Mount Point                          Created                   Updated
          --------------------  ---------------  ---------------  ------------------------------  ----------  ----------  ----------  ----------  -----------------------------------  ------------------------  ------------------------
          tempfra               DATA             TEMPFRA          /dev/asm/tempfra-18             250.00 GB   1.09 GB     0.44%       248.91 GB   /u05/app/sharedrepo/tempfra          2023-04-18 19:05:55 CEST  2023-04-18 19:05:55 CEST

   3. Create a folder on the VM storage and change ownership of the folder to oracle:dba

          [root@proddb1 ~]# mkdir /u05/app/sharedrepo/tempfra/croatia
          [root@proddb1 ~]# chown oracle:dba /u05/app/sharedrepo/tempfra/croatia

   4. Change `db_recovery_file_dest` to ACFS.

          SQL> alter system set db_recovery_file_dest='/u05/app/sharedrepo/tempfra/croatia/' scope=both;

8. Run prechecks and review the findings.

       [oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/jdk/bin/java -jar /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/rdbms/admin/autoupgrade.jar -config ~/autoupgrade.conf -mode analyze
       AutoUpgrade 22.4.220712 launched with default internal options
       Processing config file ...
       +--------------------------------+
       | Starting AutoUpgrade execution |
       +--------------------------------+
       1 CDB(s) plus 2 PDB(s) will be analyzed
       Type 'help' to list console commands
       upg> Job 100 completed
       ------------------- Final Summary --------------------
       Number of databases            [ 1 ]

       Jobs finished                  [1]
       Jobs failed                    [0]

       Please check the summary report at:
       /u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
       /u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log

9. Run the AutoUpgrade tool in fixup mode to fix issues that may prevent a successful upgrade.

       [oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/jdk/bin/java -jar /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/rdbms/admin/autoupgrade.jar -config ~/autoupgrade.conf -mode fixups
       AutoUpgrade 22.4.220712 launched with default internal options
       Processing config file ...
       +--------------------------------+
       | Starting AutoUpgrade execution |
       +--------------------------------+
       1 CDB(s) plus 2 PDB(s) will be processed
       Type 'help' to list console commands
       upg> Job 101 completed
       ------------------- Final Summary --------------------
       Number of databases            [ 1 ]

       Jobs finished                  [1]
       Jobs failed                    [0]

       Please check the summary report at:
       /u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
       /u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log

10. Re-run prechecks and review the findings.

        [oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/jdk/bin/java -jar /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/rdbms/admin/autoupgrade.jar -config ~/autoupgrade.conf -mode analyze
        AutoUpgrade 22.4.220712 launched with default internal options
        Processing config file ...
        +--------------------------------+
        | Starting AutoUpgrade execution |
        +--------------------------------+
        1 CDB(s) plus 2 PDB(s) will be analyzed
        Type 'help' to list console commands
        upg> Job 102 completed
        ------------------- Final Summary --------------------
        Number of databases            [ 1 ]

        Jobs finished                  [1]
        Jobs failed                    [0]

        Please check the summary report at:
        /u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
        /u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log

11. Upgrade the database.

        [oracle@proddb1 ~]$ /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/jdk/bin/java -jar /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/rdbms/admin/autoupgrade.jar -config ~/autoupgrade.conf -mode deploy

        AutoUpgrade 22.4.220712 launched with default internal options
        Processing config file ...
        +--------------------------------+
        | Starting AutoUpgrade execution |
        +--------------------------------+
        1 CDB(s) plus 2 PDB(s) will be processed
        Type 'help' to list console commands

        upg> Job 103 completed
        ------------------- Final Summary --------------------
        Number of databases            [ 1 ]

        Jobs finished                  [1]
        Jobs failed                    [0]
        Jobs restored                  [0]
        Jobs pending                   [0]

        ---- Drop GRP at your convenience once you consider it is no longer needed ----
        Drop GRP from croatia1: drop restore point AUTOUPGRADE_9212_ZADAR121020

        Please check the summary report at:
        /u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
        /u01/app/odaorabase/oracle/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log

        Note: once you confirmed that database was running as expected from all perspective and downgrade surely would not be needed, drop the restore point which was created by AutoUpgrade tool as the output also instructed

        ---- Drop GRP at your convenience once you consider it is no longer needed ----
        Drop GRP from croatia1: drop restore point AUTOUPGRADE_9212_ZADAR121020

12. Upgrade the standby database.

    Standby:

        [oracle@stbydb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1
        [oracle@stbydb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
        [oracle@stbydb1 ~]$ srvctl stop database -d split

        [oracle@stbydb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
        [oracle@stbydb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
        [oracle@stbydb1 ~]$ srvctl upgrade database -d split -oraclehome /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5

        [oracle@stbydb1 ~]$ srvctl start database -d split

13. Enable Oracle Data Guard Broker on the primary and on the standby.

    Primary:

        [oracle@proddb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
        [oracle@proddb1 ~]$ export ORACLE_SID=croatia1
        [oracle@proddb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
        [oracle@proddb1 ~]$ sqlplus / as sysdba
        SQL> alter system set dg_broker_start=true scope=both;

    Standby:

        [oracle@stbydb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
        [oracle@stbydb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
        [oracle@stbydb1 ~]$ srvctl upgrade database -d split -oraclehome /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5

        [oracle@stbydb1 ~]$ sqlplus / as sysdba
        SQL> alter system set dg_broker_start=true scope=both;

14. Stop the database on the primary and the standby.

    Primary:

        [oracle@proddb1 ~]$ srvctl stop database -d

    Standby:

        [oracle@stbydb1 ~]$ srvctl stop database -d

15. Copy the `tnsnames.ora` and `sqlnet.ora` files from the old home to the new home on the primary and the standby.

    Primary:

        [oracle@proddb1 ~]$ cp /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1/network/admin/* /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/network/admin/

    Standby:

        [oracle@stbydb1 ~]$ cp /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1/network/admin/* /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5/network/admin/

16. Restore Oracle Data Guard Broker configuration files on both sides.

    Primary:

        [oracle@proddb1 ~]$ cp /u02/app/oracle/oradata/zadar/dbs/dr1zadar.dat.bkp /u02/app/oracle/oradata/zadar/dbs/dr1zadar.dat
        [oracle@proddb1 ~]$ cp /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat.bkp /u02/app/oracle/oradata/zadar/dbs/dr2zadar.dat

    Standby:

        [oracle@stbydb1 ~]$ cp /u02/app/oracle/oradata/split/dbs/dr1split.dat.bkp /u02/app/oracle/oradata/split/dbs/dr1split.dat
        [oracle@stbydb1 ~]$ cp /u02/app/oracle/oradata/split/dbs/dr2split.dat.bkp /u02/app/oracle/oradata/split/dbs/dr2split.dat

17. Start primary and standby databases.

    Primary:

        [oracle@proddb1 ~]$ srvctl start database -d zadar

    Standby:

        [oracle@stbydb1 ~]$ srvctl start database -d split

18. Enable Oracle Data Guard configuration on the primary.

    Primary:

        [oracle@proddb1 ~]$ dgmgrl /
        DGMGRL for Linux: Release 19.0.0.0.0 - Production on Tue Apr 18 11:48:58 2023
        Version 19.18.0.0.0

        Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

        Welcome to DGMGRL, type "help" for information.
        Connected to "zadar"
        Connected as SYSDG.
        DGMGRL> show configuration

        Configuration - zadar_split

          Protection Mode: MaxPerformance
          Members:
          zadar - Primary database
            split - Physical standby database

        Fast-Start Failover:  Disabled

        Configuration Status:
        DISABLED

        DGMGRL> enable configuration
        Enabled.

        DGMGRL> show configuration

        Configuration - zadar_split

          Protection Mode: MaxPerformance
          Members:
          zadar - Primary database
            split - Physical standby database
              Warning: ORA-16853: apply lag has exceeded specified threshold

        Fast-Start Failover:  Disabled

        Configuration Status:
        WARNING   (status updated 17 seconds ago)

        DGMGRL> show database split;

        Database - split

          Role:               PHYSICAL STANDBY
          Intended State:     APPLY-ON
          Transport Lag:      0 seconds (computed 1 second ago)
          Apply Lag:          53 minutes 54 seconds (computed 1 second ago)
          Average Apply Rate: 34.18 MByte/s
          Real Time Query:    OFF
          Instance(s):
            croatia1 (apply instance)
            croatia2

          Database Warning(s):
            ORA-16853: apply lag has exceeded specified threshold

        Database Status:
        WARNING

        Note: It might take some time to sync up the standby

        DGMGRL> show database split;

        Database - split

          Role:               PHYSICAL STANDBY
          Intended State:     APPLY-ON
          Transport Lag:      0 seconds (computed 0 seconds ago)
          Apply Lag:          0 seconds (computed 0 seconds ago)
          Average Apply Rate: 16.42 MByte/s
          Real Time Query:    OFF
          Instance(s):
            croatia1 (apply instance)
            croatia2

        Database Status:
        SUCCESS

19. Enable `SSHCleanerJob` and remove SSH user equivalence between both nodes for Oracle Database Appliance high-availability system on the primary.

        root@proddb1 ~]# odacli list-schedules|grep "Name\|SSH"
        ID                                       Name                      Description                                        CronExpression                 Disabled
        44ad4fe2-4893-4c7d-a61c-15845cb74aa5     SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *             true

        [root@proddb1 ~]# odacli modify-schedule -i 44ad4fe2-4893-4c7d-a61c-15845cb74aa5 -e
        Modify job schedule success

        [root@proddb1 ~]# odacli list-schedules|grep "Name\|SSH"
        ID                                       Name                      Description                                        CronExpression                 Disabled
        44ad4fe2-4893-4c7d-a61c-15845cb74aa5     SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *             false

    Remove local and remote node from `/home/oracle/.ssh/authorized_keys` files on both nodes as operating system user `oracle` using the `vi` command.

        [oracle@proddb1 ~]$ vi /home/oracle/.ssh/authorized_keys
        [oracle@proddb2 ~]$ vi /home/oracle/.ssh/authorized_keys

    Remove all `id*` files under `/home/oracle/.ssh` as operating system user `oracle` on both nodes.

        [oracle@proddb1 ~]$ rm /home/oracle/.ssh/id*
        [oracle@proddb2 ~]$ rm /home/oracle/.ssh/id*

20. Sync up DCS metadata on the primary.

    Primary:

        [root@proddb1 ~]# odacli list-dbhomes
        ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
        ---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
        f90adcc1-f64a-41ce-b72d-154db155b1fa     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
        562a7428-9ea7-4878-9005-62c9d732a12b     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED
        [root@proddb1 ~]# odacli list-databases
        ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
        ---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
        222a1d47-24ea-4a00-82f0-20d7fe17f59e     croatia    RAC      12.1.0.2.220719      true    OLTP     EE       odb2     ACFS     CONFIGURED   562a7428-9ea7-4878-9005-62c9d732a12b

        [root@proddb1 ~]# odacli update-registry -n db -u zadar

        Job details
        ----------------------------------------------------------------
                             ID:  fc54b821-c407-4174-8a1a-c90ba66e6cd2
                    Description:  Discover Components : db
                         Status:  Created
                        Created:  April 18, 2023 12:04:55 PM CEST
                        Message:

        Task Name                                Node Name                 Start Time                          End Time                            Status
        ---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------

        [root@proddb1 ~]# odacli describe-job -i fc54b821-c407-4174-8a1a-c90ba66e6cd2

        Job details
        ----------------------------------------------------------------
                             ID:  fc54b821-c407-4174-8a1a-c90ba66e6cd2
                    Description:  Discover Components : db
                         Status:  Success
                        Created:  April 18, 2023 12:04:55 PM CEST
                        Message:

        Task Name                                Node Name                 Start Time                          End Time                            Status
        ---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------
        Discover DBHome                          proddb1                April 18, 2023 12:05:01 PM CEST     April 18, 2023 12:05:04 PM CEST     Success
        Discover DBHome                          proddb1                April 18, 2023 12:05:04 PM CEST     April 18, 2023 12:05:07 PM CEST     Success
        Discover DBHome                          proddb1                April 18, 2023 12:05:07 PM CEST     April 18, 2023 12:05:09 PM CEST     Success
        Discover DB: zadar                       proddb1                April 18, 2023 12:05:09 PM CEST     April 18, 2023 12:05:20 PM CEST     Success

        [root@proddb1 ~]# odacli list-databases
        ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
        ---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
        222a1d47-24ea-4a00-82f0-20d7fe17f59e     croatia    RAC      19.18.0.0.230117     true    OLTP     EE       odb2     ACFS     CONFIGURED   f90adcc1-f64a-41ce-b72d-154db155b1fa

21. Sync up DCS metadata on the standby.

    Standby:

        [root@stbydb1 ~]# odacli list-dbhomes
        ID                                       Name                 DB Version                     DB Edition Home Location                                           Status
        ---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
        fe72fa84-b609-4cea-b040-4fd7308008c8     OraDB19000_home5     19.18.0.0.230117               EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5    CONFIGURED
        d5d63d8d-91c7-416e-a8af-3e957420aafa     OraDB12102_home1     12.1.0.2.220719                EE         /u01/app/odaorahome/oracle/product/12.1.0.2/dbhome_1    CONFIGURED

        [root@stbydb1 ~]# odacli list-databases
        ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
        ---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
        a85a0120-2343-4a42-af5b-93b958353c38     croatia    RAC      12.1.0.2.220719      true    OLTP     EE       odb2     ACFS     CONFIGURED   d5d63d8d-91c7-416e-a8af-3e957420aafa

        [root@stbydb1 ~]# odacli update-registry -n db -u split

        Job details
        ----------------------------------------------------------------
                             ID:  4694f9d9-a569-433d-9d76-69b3b8b9ddcf
                    Description:  Discover Components : db
                         Status:  Created
                        Created:  April 18, 2023 10:08:03 AM GMT
                        Message:

        Task Name                                Node Name                 Start Time                          End Time                            Status
        ---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------

        [root@stbydb1 ~]# odacli describe-job -i 4694f9d9-a569-433d-9d76-69b3b8b9ddcf

        Job details
        ----------------------------------------------------------------
                             ID:  4694f9d9-a569-433d-9d76-69b3b8b9ddcf
                    Description:  Discover Components : db
                         Status:  Success
                        Created:  April 18, 2023 10:08:03 AM GMT
                        Message:

        Task Name                                Node Name                 Start Time                          End Time                            Status
        ---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------
        Discover DBHome                          stbydb1                April 18, 2023 10:08:15 AM GMT      April 18, 2023 10:08:19 AM GMT      Success
        Discover DBHome                          stbydb1                April 18, 2023 10:08:19 AM GMT      April 18, 2023 10:08:22 AM GMT      Success
        Discover DB: split                       stbydb1                April 18, 2023 10:08:22 AM GMT      April 18, 2023 10:08:35 AM GMT      Success

        [root@stbydb1 ~]# odacli list-databases
        ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
        ---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
        a85a0120-2343-4a42-af5b-93b958353c38     croatia    RAC      19.18.0.0.230117     true    OLTP     EE       odb2     ACFS     CONFIGURED   fe72fa84-b609-4cea-b040-4fd7308008c8

22. Check Oracle Data Guard status using the `odacli list-dataguardstatus` command to get the latest status.

    Primary:

        [root@proddb1 ~]# odacli list-dataguardstatus
        Updated about 35 day(s) ago
        It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
        ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
        ---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
        54b4390a-5078-4e5d-8cef-53888d6b7b16     zadar_split                      croatia              PRIMARY    MAX_PERFORMANCE    ---             ---             ---             STALE

        [root@proddb1 ~]# odacli describe-dataguardstatus -i 54b4390a-5078-4e5d-8cef-53888d6b7b16 -s
        Updated about 1 minute(s) ago
        It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
        Dataguard Status details
        ----------------------------------------------------------------
                             ID: 54b4390a-5078-4e5d-8cef-53888d6b7b16
                           Name: zadar_split
                    Database ID: 222a1d47-24ea-4a00-82f0-20d7fe17f59e
                           Role: PRIMARY
                Protection Mode: MAX_PERFORMANCE
                      Apply Lag: 0 seconds
                  Transport Lag: 0 seconds
                     Apply Rate: 10.83 MByte/s
                         Status: CONFIGURED
                   Updated Time: April 18, 2023 12:11:44 PM CEST

        [root@stbydb1 ~]# odacli list-dataguardstatus
        Updated about 35 day(s) ago
        It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
        ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
        ---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
        54b4390a-5078-4e5d-8cef-53888d6b7b16     zadar_split                      croatia              STANDBY    MAX_PERFORMANCE    0 seconds       0 seconds       8.00 KByte/s    CONFIGURED

    Standby:

        [root@stbydb1 ~]# odacli describe-dataguardstatus -i 54b4390a-5078-4e5d-8cef-53888d6b7b16 -s
        Updated about 26 second(s) ago
        It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
        Dataguard Status details
        ----------------------------------------------------------------
                             ID: 54b4390a-5078-4e5d-8cef-53888d6b7b16
                           Name: zadar_split
                    Database ID: a85a0120-2343-4a42-af5b-93b958353c38
                           Role: STANDBY
                Protection Mode: MAX_PERFORMANCE
                      Apply Lag: 0 seconds
                  Transport Lag: 0 seconds
                     Apply Rate: 9.70 MByte/s
                         Status: CONFIGURED
                   Updated Time: April 18, 2023 10:13:31 AM GMT

        [root@stbydb1 ~]# odacli list-dataguardstatus
        Updated about 35 day(s) ago
        It can take up to several minutes to update Data Guard status. You can re-run the command to obtain the latest status.
        ID                                       Name                             Database Name        Role       Protection Mode    Apply Lag       Transport Lag   Apply Rate      Status
        ---------------------------------------- -------------------------------- -------------------- ---------- ------------------ --------------- --------------- --------------- ----------
        54b4390a-5078-4e5d-8cef-53888d6b7b16     zadar_split                      croatia              STANDBY    MAX_PERFORMANCE    0 seconds       0 seconds       1.72 MByte/s    CONFIGURED

23. If the `db_recovery_file_dest` file was changed in step 7, revert that change.

        [oracle@proddb1 ~]$ export ORACLE_HOME=/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5
        [oracle@proddb1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
        [oracle@proddb1 ~]$ export ORACLE_SID=croatia1
        [oracle@proddb1 ~]$ sqlplus / as sysdba

        SQL> alter system set db_recovery_file_dest='+RECO(FG$FILEGROUP_TEMPLATE_MIRROR)' scope=both;

24. Take a full backup.

        [oracle@proddb1 ~]# odacli create-backup -n croatia -bt Regular-L0

25. Remove the temporary VM storage.

        [oracle@proddb1 ~]# odacli delete-vmstorage -n tempfra

26. Test switchover, failover, and reinstate operations after the upgrade using ODACLI commands.

Parent topic: Scenario: Upgrading Integrated Oracle Data Guard