Patching and Upgrading Oracle Globally Distributed Database

There are special considerations for patching and upgrading an Oracle Globally Distributed Database deployment.

Patching and Upgrading Oracle Globally Distributed Database

Applying an Oracle patch to a sharded database environment can be done on a single shard or all shards; however, the method you use depends on the replication option used for the environment and the type of patch being applied.

Oracle Globally Distributed Database uses consolidated patching to update a shard director (GSM) ORACLE_HOME, so you must apply the Oracle Database release updates to the ORACLE_HOME to get security and Global Data Services fixes.

Patching a Sharded Database

Most patches can be applied to a single shard at a time; however, some patches should be applied across all shards. Use Oracle’s best practices for applying patches to single shards just as you would a non-sharded database, keeping in mind the replication method that is being used with the sharded database. Oracle opatchauto can be used to apply patches to multiple shards at a time, and can be done in a rolling manner. Data Guard configurations are applied one after another, and in some cases (depending on the patch) you can use Standby First patching.

If a patch addresses an issue with multi-shard queries, replication, or the sharding infrastructure, it should be applied to all of the shards in the sharded database.

Note:

Because logical standbys are not supported in Oracle Sharding, rolling upgrades may run into a DDL recovery issue because a physical standby database becomes a 'transient logical standby' during a rolling upgrade. To avoid this issue, follow the steps in Performing a Rolling Upgrade.

Upgrading a Sharded Database

Upgrading the Oracle Globally Distributed Database environment is not much different from upgrading other Oracle Database and global service manager environments; however, the components must be upgraded in a particular sequence such that the shard catalog is upgraded first, followed by the shard directors, and finally the shards.

Performing a Rolling Upgrade

To avoid this issue, perform the following steps.

Shut down the shard catalog database.
Shutting down the shard catalog database prevents any shard director (GSM) from becoming the master, and the catalog will not try to apply any DDL in this state, but the shard director will continue in steady-state allowing production applications to connect and run.
Perform the rolling upgrade.
When the rolling upgrade is complete, start up the shard catalog database.

Note:

During a rolling upgrade, some operations such as automatic failover may not be available while the shard catalog is shut down.

Upgrading Oracle Globally Distributed Database Components

The order in which Oracle Globally Distributed Database components are upgraded is important for limiting downtime and avoiding errors as components are brought down and back online.

Before upgrading any Oracle Globally Distributed Database components you must

Complete any pending MOVE CHUNK operations that are in progress.
Do not start any new MOVE CHUNK operations.
Do not add any new shards during the upgrade process.

Upgrade the shards with the following points in mind.
- For system-managed sharded databases: upgrade each set of shards in a Data Guard Broker configuration in a rolling manner.
- For user-defined sharded databases: upgrade each set of shards in a shardspace in a rolling manner.
- For composite sharded databases: in a given shardspace, upgrade each set of shards in a Data Guard Broker configuration in a rolling manner.
- If you are upgrading an Oracle Database 18c sharded database configuration containing pluggable database (PDB) shards, follow the PDB-specific upgrade instructions in Compatibility and Migration from Oracle Database 18c.
Upgrade the shard catalog database.

For best results the catalog should be upgraded using a rolling database upgrade; however, global services will remain available during the upgrade if the catalog is unavailable, although service failover will not occur.
Upgrade any shard directors that are used to run GDSCTL clients, and which do not also run a global service manager server, before you update the shard directors running global service managers.
For shard directors running global service managers, do the following steps on one global service manager at a time.
To ensure zero downtime, at least one shard director server should always be running. Shard director servers at an earlier version than the catalog will continue to operate fully until catalog changes are made.
1. Stop one of the global service managers to be upgraded.
2. Install the 21c global service manager kit
3. Copy the tnsnames.ora,gsm.ora,gsm_observer_1.dat,gsmwallt directory from the previous version to the new version.
4. Connect to the new version GDSCTL and start the global service manager which was stopped in the old version
5. Stop the old global service manager and start it the new global service manager.

Post-Upgrade Steps for Oracle Globally Distributed Database 21c

If you have a fully operational Oracle Globally Distributed Database environment in a release earlier than 21c, no wallets exist, and no deployment will be done by Oracle Globally Distributed Database after an upgrade to 21c to create them. You must perform manual steps to create the wallets.

Note:

The steps must be followed in EXACTLY this order.

Modify the GSMCATUSER password on the shard catalog database using ALTER USER, which automatically creates the wallet on the primary shard catalog.
```
SQL> alter user gsmcatuser identified by gsmcatuser_password;
```
This statement creates a shard wallet file if it does not exist, but does not change the password if the previous password is re-used.

You can keep the same password for both GSMCATUSER accounts, but you must run ALTER USER to create the wallet.
If the GSMCATUSER password was changed, run GDSCTL modify catalog to update the system with the new GSMCATUSER password.

This step is not necessary if the GSMCATUSER password was not changed.
If the GSMCATUSER password was changed, also run GDSCTL modify gsm for each shard director (GSM) to inform the shard director of the new password.

Again, this step is not necessary if the GSMCATUSER password was not changed.
Modify the GSMUSER password on each primary shard using ALTER USER, which automatically creates the wallet on that shard.
```
SQL> alter user gsmuser identified by gsmuser_password;
```
This statement creates a shard wallet file if it does not exist, but does not change the password if the previous password is re-used.

You can keep the same password for both GSMCATUSER accounts, but you must run ALTER USER to create the wallet.
If the GSMUSER password was changed, run GDSCTL modify shard on each primary shard to update the system with the new GSMUSER password for that shard.

This step is not necessary if the GSMUSER password was not changed.
Run GDSCTL sync database on each shard to populate the shard wallet file with the required information.
After completing all of the above steps, locate the primary wallet file using the method described below, and copy this file to the wallet_root location on the standby database.
All standby databases need an identical copy of the wallet file corresponding to its primary.

This also applies to standby shard catalogs (if any).

Before copying the wallet, you should check that wallet_root has been set on the standby database, if it has not already been set, then set it before copying the wallet file.

Note that wallet_root is a global parameter that can only be set in the container database (CDB).

The wallets themselves are created on the shard catalog, and any shard catalog replicas, during the GDSCTL create shardcatalog command, and on the shards during the GDSCTL deploy command.

All primary databases and their replicas must have a sharding-specific wallet file present to ensure proper operation. The location of the wallet files is under the directory specified by the wallet_root database initialization parameter. If it is not set before running create shardcatalog or deploy, wallet_root is set to $ORACLE_BASE/admin/db_unique_name on either the shard catalog or shards, respectively.

For example, assume the following after logging into the shard catalog or into a shard:
```
SQL> select guid from v$pdbs where con_id = sys_context('userenv','con_id');

GUID
--------------------------------
C23E7C78D5B77D50E0537517C40ACE4A

SQL> select value from v$parameter where name='wallet_root';

VALUE
--------------------------------------------------------------------------------
your-path-to-keystore
```
Given these values, the sharding-specific wallet file name is

your-path-to-keystore/C23E7C78D5B77D50E0537517C40ACE4A/shard/cwallet.sso

Compatibility and Migration from Oracle Database 18c

When upgrading from an Oracle Database 18c installation which contains a single PDB shard for a given CDB, you must update the shard catalog metadata for any PDB.

Specifically, in 18c, the name of a PDB shard is the DB_UNIQUE_NAME of its CDB; however, in later Oracle Database releases, the shard names are db_unique_name_of_CDB_pdb_name.

To update the catalog metadata to reflect this new naming methodology, and to also support the new GSMROOTUSER account as described in About the GSMROOTUSER Account, perform the following steps during the upgrade process as described in Upgrading Oracle Globally Distributed Database Components.

After upgrading any CDB that contains a PDB shard, ensure that the GSMROOTUSER account exists, is unlocked, has been assigned a password, and has been granted SYSDG, SYSBACKUP, and gsmrootuser_role privileges.
The following SQL statements in SQL*Plus will successfully set up GSMROOTUSER while connected to the root container (CDB$ROOT) of the CDB.
```
SQL> alter session set "_oracle_script"=true;
Session altered.
     
SQL> create user gsmrootuser;
User created.

SQL> alter user gsmrootuser identified by new_GSMROOTUSER_password
  account unlock;
User altered.

SQL> grant sysdg, sysbackup, gsmrootuser_role to gsmrootuser container=current;
Grant succeeded.

SQL> alter session set "_oracle_script"=false;
Session altered.
```
After upgrading the catalog database to the desired Oracle Database version, run the following PL/SQL procedure to update the catalog metadata to reflect the new name for the PDB shards present in the configuration.
This procedure must be run for each Oracle Database 18c PDB shard.

The first parameter to pdb_fixup is the value of db_unique_name in the CDB that contains the PDB shard. In Oracle Database 18c, this is the same as the shard name as shown by gdsctl config shard.

The second parameter is the PDB name of the shard PDB as shown by show con_name in SQL*Plus when connected to the shard PDB.

The pdb_fixup procedure will update the catalog metadata to make it compatible with the new naming method for PDB shards.
```
SQL> connect sys/password as sysdba
Connected.
SQL> set serveroutput on
SQL> execute gsmadmin_internal.dbms_gsm_pooladmin.pdb_fixup('cdb1', 'pdb1');
```
After upgrading all of the shard directors to the desired version, run the following GDSCTL command once for each CDB in the configuration to inform the shard directors of the password for the GSMROOTUSER in each CDB.
```
GDSCTL> modify cdb -cdb CDB_name -pwd new_GSMROOTUSER_password
```

Downgrading an Oracle Globally Distributed Database

Oracle Globally Distributed Database does not support downgrading.

Sharded database catalogs and shards cannot be downgraded.