OnPremises and Cloud: Oracle GoldenGate Microservices Architecture Integrated with Active Data Guard

The combination and integration of Oracle GoldenGate Microservices and Oracle Data Guard enables you to achieve an MAA Platinum service-level configuration that achieves zero or near zero downtime for all planned and unplanned outages.

Overview

Use these best practices to integrate Oracle GoldenGate Microservices Architecture deployed on Oracle Real Application Clusters (Oracle RAC), on-premises Oracle Exadata, Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D, formerly known as Exadata Cloud Service - ExaCS), or Oracle Exadata Database Service on Cloud@Customer (ExaDB-C@C) with Oracle Data Guard.

The architecture uses Oracle RAC, Oracle Clusterware, Oracle Database File System (DBFS), and Oracle Grid Infrastructure Standalone Agent (XAG) to provide high availability for Oracle GoldenGate and coordinate its dependencies on the database, network, and file system. Oracle GoldenGate software is installed locally on the eligible Oracle RAC nodes, while the GoldenGate deployment files are stored in DBFS so that they are available to the appropriate system following RAC node failures, planned maintenance, and Oracle Data Guard role transitions.

When the database is protected by Oracle Data Guard, the configuration enables Oracle GoldenGate replication to continue following a Data Guard switchover or failover. Database services, DBFS, Clusterware resources, and the GoldenGate deployment are configured so that GoldenGate can start on the system hosting the new primary database and resume replication from its persisted state.

These best practices apply with Oracle Data Guard Maximum Performance, Maximum Availability, and Maximum Protection modes.

For Oracle GoldenGate Microservices Architecture deployment and configuration best practices on Oracle RAC and Exadata Database Service, see OnPremises and Cloud: Oracle GoldenGate Microservices Architecture on Oracle Exadata Configuration Best Practices.

Scope and applicability

This chapter provides configuration and operational best practices for deploying and integrating Oracle GoldenGate Microservices Architecture with Oracle Data Guard to provide high availability and support GoldenGate replication following an Oracle Data Guard role transition.

The procedures in this chapter apply to Oracle GoldenGate deployments running on Oracle RAC or Exadata Database Service systems hosting databases configured with Oracle Data Guard.

Deployment Platform Applicability
On-premises Oracle RAC / Exadata Applies to the common configuration and Data Guard integration procedures in this chapter.
Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D/ExaCS) Applies to the common procedures and the platform-specific steps identified in this chapter.
Oracle Exadata Database Service on Cloud@Customer (ExaDB-C@C) Applies to the common procedures and the platform-specific steps identified in this chapter.

The chapter uses a common configuration approach where possible. Platform-specific requirements and commands are identified where they differ.

The Oracle RAC or Exadata Database Service system that hosts Oracle GoldenGate Microservices Architecture can act as the source database, target database, or, in some cases, both source and target databases.

Summary of Recommendations

When integrating Oracle GoldenGate Microservices Architecture with Oracle Data Guard, follow these recommendations:

Configuration Prerequisites

Before starting the configuration in this chapter, ensure the following prerequisites are met:

Configure Oracle GoldenGate Microservices Architecture with Oracle Data Guard

Task 1: Configure the Standby Database for Oracle GoldenGate

Configure the standby database so that it is prepared to assume the same Oracle GoldenGate source or target role as the primary database following an Oracle Data Guard role transition.

Ensure that the applicable Oracle GoldenGate database configuration is consistent between the primary and standby databases, including:

For detailed database preparation and supplemental logging requirements, see Prepare the Database for Oracle GoldenGate in the Oracle GoldenGate Microservices Architecture Documentation.

Task 2: Configure Database Services for Oracle Data Guard Role Transitions

Configure the database services used by Oracle GoldenGate with the PRIMARY role so that they run on the database that holds the Oracle Data Guard primary role following a switchover or failover.

For multitenant databases, configure a service for each PDB used by Oracle GoldenGate. A separate CDB root service is required only when using a CDB-level Integrated Extract.

Step 2.1: Validate or Modify the Primary Database Service

On the primary database system, verify that the existing Oracle GoldenGate database service is configured with the PRIMARY role.

As the oracle OS user, run:

srvctl config service -db <db_unique_name>

Verify that the applicable service shows:

Service role: PRIMARY

If the service is not configured with the PRIMARY role, modify it:

srvctl modify service -db <db_unique_name> -service <service_name> -role PRIMARY

For multitenant databases, verify each PDB service used by Oracle GoldenGate and the CDB root service when a CDB-level Integrated Extract is used.

Step 2.2: Create the Standby Database Service

On the standby database system, create a corresponding service using the same service name and configuration as the primary database service.

Configure the service as a singleton service with a preferred Oracle RAC instance and one or more available instances, and set the service role to PRIMARY:

srvctl add service -db <db_unique_name> \
-service <service_name> \
-preferred <instance_1> \
-available <instance_2> \
-pdb <pdb_name> \
-role PRIMARY
For example:

srvctl add service -db ggdbs \
-service oggserv_pdb \
-preferred ggdbs1 \
-available ggdbs2 \
-pdb GGPDB01 \
-role PRIMARY

For Oracle Database 19c non-CDB databases, omit the -pdb parameter. Also omit -pdb when creating a service for a CDB-level Integrated Extract.

Using the PRIMARY service role ensures that the service starts when the standby database becomes the new primary following an Oracle Data Guard role transition.

Task 3: Configure DBFS on the Standby System

Oracle Database File System (DBFS) is the only recommended solution when configuring Oracle GoldenGate with Oracle Data Guard. DBFS stores the Oracle GoldenGate deployment files. Because DBFS is contained in the database protected by Data Guard, the GoldenGate deployment files remain synchronized with the database and are available when the standby database assumes the PRIMARY role.

The DBFS repository, including the DBFS user, tablespace, and file system, is created in the primary database and protected by Oracle Data Guard. On the standby system, configure the required operating system components, mount point, Oracle Net connectivity, DBFS mount scripts, and Oracle Clusterware resource.

Use the same DBFS mount point on the primary and standby systems because the GoldenGate deployment path is stored in the deployment configuration.

Step 3.1: Prepare the Standby Nodes for DBFS

Configure all standby Oracle RAC nodes where Oracle GoldenGate may run.

On-premises Oracle RAC / Exadata

Install the required FUSE components, if they are not already installed. Refer to My Oracle Support Knowledge Article KB140582 for DBFS and FUSE configuration requirements.

Create the same DBFS mount point used on the primary system:

mkdir -p <dbfs_mount_point>
chown oracle:oinstall <dbfs_mount_point>

For example:

mkdir -p /mnt/dbfs
chown oracle:oinstall /mnt/dbfs

ExaDB-D and ExaDB-C@C

On the standby Exadata Database Service system, identify the database nodes and add the grid OS user to the fuse group:

sudo -u grid $(grep ^crs_home /etc/oracle/olr.loc | cut -d= -f2)/bin/olsnodes > ~/dbs_group
dcli -g ~/dbs_group -l opc sudo usermod -a -G fuse grid

Verify that /etc/fuse.conf contains:

user_allow_other

If required, add it on all database nodes:

dcli -g ~/dbs_group -l opc "echo user_allow_other | sudo tee -a /etc/fuse.conf"

Create the same DBFS mount point used on the primary system on all standby database nodes:

dcli -g ~/dbs_group -l opc sudo mkdir -p <dbfs_mount_point>
dcli -g ~/dbs_group -l opc sudo chown oracle:oinstall <dbfs_mount_point>

For example:

dcli -g ~/dbs_group -l opc sudo mkdir -p /mnt/dbfs
dcli -g ~/dbs_group -l opc sudo chown oracle:oinstall /mnt/dbfs

Step 3.2: Configure the DBFS Oracle Net Connection

On the standby system, configure an Oracle Net connection alias for DBFS. For a multitenant database, use the service for the PDB that contains the DBFS repository.

Add the DBFS connection alias to the tnsnames.ora file in the configured TNS_ADMIN location:

dbfs =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = IPC)(KEY=LISTENER))
    (CONNECT_DATA =
      (SERVICE_NAME = <PDB_SERVICE_NAME>)
    )
  )

Use the PDB service created in Task 2 and ensure the Oracle Net configuration is available on all standby Oracle RAC nodes where DBFS may be mounted.

For Oracle Database 19c non-CDB configurations, use the corresponding database service for the DBFS repository.

On Exadata systems, dcli can be used to distribute the updated tnsnames.ora file to the remaining database nodes.

For example:

/usr/local/bin/dcli -l oracle -g ~/dbs_group -f $TNS_ADMIN/tnsnames.ora -d $TNS_ADMIN/

Step 3.3: Configure the DBFS Mount Scripts

Copy the following DBFS mount files from the primary system to the standby system:

Use the same file locations on the primary and standby systems.

Update mount-dbfs.conf for the standby database environment. Modify the following parameters as applicable:

Copy mount-dbfs.conf to /etc/oracle on all standby Oracle RAC nodes and set the appropriate ownership and permissions:

chown oracle:oinstall /etc/oracle/mount-dbfs.conf
chmod 660 /etc/oracle/mount-dbfs.conf

Copy mount-dbfs.sh to the corresponding Oracle Grid Infrastructure script directory on all standby Oracle RAC nodes and use the same ownership and permissions as on the primary system.

For example:

$GI_HOME/crs/script/mount-dbfs.sh

Ensure that both files are available on every standby Oracle RAC node where DBFS may be mounted.

On Exadata systems, dcli can be used to distribute the DBFS mount files and apply the required ownership and permissions across the database nodes.

For example:

/usr/local/bin/dcli -g ~/dbs_group -l opc -d /tmp -f /tmp/mount-dbfs.conf

/usr/local/bin/dcli -g ~/dbs_group -l opc sudo cp /tmp/mount-dbfs.conf /etc/oracle

/usr/local/bin/dcli -g ~/dbs_group -l opc sudo chown oracle:oinstall /etc/oracle/mount-dbfs.conf

/usr/local/bin/dcli -g ~/dbs_group -l opc sudo chmod 660 /etc/oracle/mount-dbfs.conf

Then for mount-dbfs.sh:

/usr/local/bin/dcli -g ~/dbs_group -l grid -d $GI_HOME/crs/script -f /tmp/mount-dbfs.sh

/usr/local/bin/dcli -g ~/dbs_group -l grid chmod 770 $GI_HOME/crs/script/mount-dbfs.sh

Step 3.4: Register the DBFS Resource with Oracle Clusterware

Register DBFS as an Oracle Clusterware resource on the standby system so that DBFS is automatically mounted when the required database service is available.

For a multitenant database, configure the DBFS resource dependency using the service for the PDB that contains the DBFS repository.

In the shell, define the following values for the standby environment before registering the DBFS resource:

For example:

DBNAME=ggdbs
RESNAME=dbfsgg
ACTION_SCRIPT=$GI_HOME/crs/script/mount-dbfs.sh
DEPNAME=ora.$DBNAME.oggserv_pdb.svc

As the root OS user, register the DBFS resource:

$GI_HOME/bin/crsctl add resource $RESNAME -type cluster_resource -attr "ACTION_SCRIPT=$ACTION_SCRIPT,CHECK_INTERVAL=30,RESTART_ATTEMPTS=10,START_DEPENDENCIES='hard($DEPNAME)pullup($DEPNAME)',STOP_DEPENDENCIES='hard($DEPNAME)',SCRIPT_TIMEOUT=300"

The dependency on the database service ensures that DBFS is mounted only when the corresponding service is available on the database holding the PRIMARY role.

Verify that the DBFS resource is registered and that the required dependencies are configured correctly.

As the root OS user, verify the DBFS Clusterware resource:

$GI_HOME/bin/crsctl status resource dbfsgg
$GI_HOME/bin/crsctl status resource dbfsgg -f

After creating the $RESNAME resource, in order to stop the $DBNAME database when the $RESNAME resource is ONLINE, you will have to specify the force flag when using srvctl.

For example:

srvctl stop database -d $ORACLE_UNQNAME -f

Task 4: Install Oracle GoldenGate Software on the Standby System

Install Oracle GoldenGate software locally on all standby Oracle RAC nodes where the GoldenGate deployment may run.

Install the same Oracle GoldenGate release and patch level used on the primary system, and use the same Oracle GoldenGate installation directory on the primary and standby systems.

For example, if the Oracle GoldenGate home on the primary system is:

/u01/oracle/goldengate/gg26ai_MS

Use the same location on all eligible standby Oracle RAC nodes.

Install Oracle GoldenGate using the software obtained in Configuration Prerequisites. For installation instructions, see Installing Oracle GoldenGate in the Oracle GoldenGate Microservices Architecture Documentation.

Task 5: Prepare the Oracle GoldenGate Environment on the Standby System

The Oracle GoldenGate Service Manager and deployment are already configured on the primary system. On the standby system, create the corresponding local directories and symbolic links required by the GoldenGate deployment.

Follow the applicable directory and symbolic-link configuration described in OnPremises and Cloud: Oracle GoldenGate Microservices Architecture on Oracle Exadata Configuration Best Practices, and use the same directory structure on all standby Oracle RAC nodes where the GoldenGate deployment may run.

Consider the following additional requirements:

Task 6: Configure the Standby NGINX Reverse Proxy

Configure NGINX on the standby system to provide the reverse proxy for the Oracle GoldenGate Microservices deployment after an Oracle Data Guard role transition.

The standby system uses its own application VIP and therefore requires an NGINX configuration and SSL certificate that correspond to the standby VIP.

Repeat these configuration steps for all standby Oracle RAC nodes where the GoldenGate deployment may run.

Step 6.1: Configure the NGINX Reverse Proxy

Install NGINX on the standby Oracle RAC nodes if it is not already installed. For NGINX installation instructions, see NGINX: Linux packages and installation instructions.

Copy the Oracle GoldenGate NGINX configuration file from the primary system to the corresponding location on the standby system.

For example:

scp <primary_node>:/etc/nginx/conf.d/<ogg_config>.conf /etc/nginx/conf.d/

Update the server_name in the copied configuration file to use the standby application VIP name.

For example:

server_name <standby_vip_fqdn>;

Ensure the same NGINX configuration is available on all standby Oracle RAC nodes where the GoldenGate deployment may run.

Step 6.2: Configure the SSL/TLS Certificates

Install the SSL/TLS certificate and private key obtained for the standby application VIP.

The certificate must match the standby VIP name used by NGINX. Use the certificate and key locations specified in the NGINX configuration:

ssl_certificate     /etc/nginx/ssl/<standby_certificate>.pem;
ssl_certificate_key /etc/nginx/ssl/<standby_certificate>.key;

For CA-signed certificates, ensure the certificate file contains the required server, intermediate, and root certificates in the appropriate certificate chain.

Protect the private key using appropriate ownership and file permissions.

Step 6.3: Validate and Restart NGINX

Because the standby application VIP is not active until the associated resources are started, enable NGINX to bind to a non-local IP address.

Set:

net.ipv4.ip_nonlocal_bind = 1

Apply the setting:

sysctl -p

Validate the NGINX configuration:

nginx -t

If validation succeeds, restart NGINX:

systemctl restart nginx

Ensure the NGINX configuration and required certificates are available in the corresponding locations on all eligible standby Oracle RAC nodes.

Step 6.4: Register NGINX with Oracle Clusterware

Register NGINX as an Oracle Clusterware resource so that it is automatically managed by Clusterware and available before the Oracle GoldenGate deployment starts.

First, determine the Oracle Clusterware network resource:

$GI_HOME/bin/crsctl stat res -w "TYPE == ora.network.type" | grep NAME
NAME=ora.net1.network

As the root OS user, register NGINX with Oracle Clusterware. For example:

$GI_HOME/bin/crsctl add resource nginx -type generic_application -attr "ACL='owner:root:rwx,pgrp:root:rwx,other::r--,group:oinstall:r-x,user:oracle:rwx',EXECUTABLE_NAMES=nginx,START_PROGRAM='/bin/systemctl start -f nginx',STOP_PROGRAM='/bin/systemctl stop -f nginx',CHECK_PROGRAMS='/bin/systemctl status nginx',START_DEPENDENCIES='hard(ora.net1.network) pullup(ora.net1.network)',STOP_DEPENDENCIES='hard(intermediate:ora.net1.network)',RESTART_ATTEMPTS=0,HOSTING_MEMBERS='<node1>,<node2>',CARDINALITY=2"

The NGINX resource in this example runs concurrently on the nodes specified by HOSTING_MEMBERS. This configuration is recommended when multiple Oracle GoldenGate Service Manager deployments are configured and can independently move between Oracle RAC nodes.

Task 7: Configure Oracle Grid Infrastructure Standalone Agent (XAG)

Configure Oracle Grid Infrastructure Standalone Agent (XAG) to manage the Oracle GoldenGate deployment and its dependent resources. XAG coordinates the database service, DBFS, application VIP, NGINX, and GoldenGate resources, and enables the GoldenGate deployment to start on the new primary system following an Oracle Data Guard role transition.

Step 7.1: Configure the Primary XAG GoldenGate Instance for Data Guard

Modify the existing XAG GoldenGate instance on the primary system to enable automatic startup following an Oracle Data Guard role transition.

As the root OS user on the primary system, run:

<XAG_HOME>/bin/agctl modify goldengate <instance_name> --dataguard_autostart yes

Step 7.2: Install XAG on the Standby System

Download the latest XAG version as recommended in the Configuration Prerequisites section. Unzip the downloaded software and use xagsetup.sh to install XAG on the standby Oracle RAC cluster.

Install XAG outside of the Oracle Grid Infrastructure home and use the same installation directory on all Oracle RAC nodes where Oracle GoldenGate is installed.

As the Oracle Grid Infrastructure OS user, install XAG on all cluster nodes:

./xagsetup.sh --install --directory /u01/oracle/xag --all_nodes

Add the XAG bin directory to the PATH environment variable so that the agctl command is available:

export PATH=/u01/oracle/xag/bin:$PATH

For more information, see Oracle Grid Infrastructure Standalone Agents for Oracle Clusterware.

Step 7.3: Register Oracle GoldenGate with XAG on the Standby System

Use the XAG configuration information from the primary system to identify the GoldenGate home, Service Manager configuration and variable directories, port, administration user, database service, DBFS resource, and other values required when registering the standby XAG resource.

As the grid OS user on the primary system, review the existing configuration:

<XAG_HOME>/bin/agctl config goldengate <instance_name>

The configuration identifies the GoldenGate home, Service Manager configuration and variable directories, database services, DBFS and NGINX resources, application VIP, and Oracle RAC nodes associated with the deployment.

Use the same GoldenGate XAG instance name on the primary and standby systems.

Before registering the standby XAG resource, note these two Data Guard-specific requirements:

<XAG_HOME>/bin/agctl add goldengate <instance_name> \
--gg_home <OGG_HOME> \
--service_manager \
--config_home <DBFS_deployment_path>/etc/conf \
--var_home <DBFS_deployment_path>/var \
--port <service_manager_port> \
--oracle_home <OGG_HOME>/lib/instantclient \
--adminuser <ogg_admin_user> \
--user oracle \
--group oinstall \
--network 1 --ip <standby_virtual_IP_address> \
--filesystems <dbfs_resource>,nginx \
--db_services <database_service_resource> \
--use_local_services \
--nodes <standby_node1>,<standby_node2> \
--filesystem_verify no \
--dataguard_autostart yes

The filesystems setting makes the GoldenGate XAG resource dependent on both the DBFS resource from Task 3 and the NGINX resource from Task 6.

The db_services setting should reference the appropriate PRIMARY role database service configured in Task 2.

Task 8: Configure Oracle Net Connections for Oracle GoldenGate

Configure the Oracle Net connection aliases used by the Oracle GoldenGate deployment on the standby system. Use the same TNS_ADMIN location and TNS alias names on the primary and standby systems so that the existing GoldenGate database connections remain valid following an Oracle Data Guard role transition.

On the primary system, determine the TNS_ADMIN location configured for the GoldenGate deployment. For example:

grep -A1 TNS_ADMIN <deployment_directory>/etc/conf/deploymentRegistry.dat

Example output:

"name": "TNS_ADMIN",
"value": "/u01/oracle/goldengate/network/admin"

Create the same TNS_ADMIN directory on all eligible standby Oracle RAC nodes.

Configure tnsnames.ora in this directory using the same aliases as the primary system. For a per-PDB Extract or target PDB connection, configure the alias using the corresponding PDB service created in Task 2.

For example:

OGGSRV_GGPDB01 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = IPC)(KEY=LISTENER))
    (CONNECT_DATA =
      (SERVICE_NAME = oggserv_pdb)
    )
  )

If the deployment uses a CDB-level Integrated Extract, also configure the corresponding CDB level service alias.

For Oracle Database 19c non-CDB configurations, configure the alias using the applicable database service.

Ensure the Oracle Net configuration files are available in the same TNS_ADMIN location on all eligible standby Oracle RAC nodes.

On Exadata systems, dcli can be used to create the directory and distribute the Oracle Net configuration files. For example:

/usr/local/bin/dcli -l oracle -g ~/dbs_group mkdir -p <TNS_ADMIN>

/usr/local/bin/dcli -l oracle -g ~/dbs_group -f <TNS_ADMIN>/tnsnames.ora -d <TNS_ADMIN>

Note: When tnsnames.ora or sqlnet.ora in the TNS_ADMIN directory used by the Oracle GoldenGate deployment is modified, restart the GoldenGate deployment for the changes to take effect.

Task 9: Configure Oracle GoldenGate Processes for Data Guard

Configure the Oracle GoldenGate processes to account for Oracle Data Guard role transitions and, where applicable, potential data loss during a failover.

Step 9.1: Configure Extract for Oracle Data Guard

For Oracle GoldenGate source databases protected by Oracle Data Guard using Maximum Performance or Maximum Availability mode, add the following parameter to each applicable Extract on the primary system:

TRANLOGOPTIONS HANDLEDLFAILOVER

HANDLEDLFAILOVER prevents Extract from processing transaction data beyond the point applied to the Data Guard standby.

This helps prevent logical data inconsistencies after a data-loss failover where the GoldenGate target could otherwise contain transactions that do not exist on the new source primary.

Use the following TRANLOGOPTIONS parameters as applicable:

Parameter Purpose Recommendation
HANDLEDLFAILOVER Prevents Extract from getting ahead of the redo applied to the Data Guard standby. Required for source databases using Data Guard Maximum Performance or Maximum Availability.
HANDLEDLFAILOVER STANDBY_WARNING <seconds> Controls how long Extract waits before reporting a warning when it cannot determine standby apply progress. Default is 60 seconds. Optional. Use when the default warning interval needs to be changed. Valid for Oracle Database 21c and later.
HANDLEDLFAILOVER STANDBY_ABEND <seconds> Controls how long Extract waits for standby apply progress before abending. Optional. Use when the default wait period needs to be changed. Valid for Oracle Database 21c and later.
DLFAILOVER_TIMEOUT <seconds> Controls how long HANDLEDLFAILOVER is temporarily suspended following a Data Guard role transition while the new standby is reinstated. MAA recommends retaining the default 300 seconds (5 minutes).
FAILOVERTARGETDESTID Identifies the LOG_ARCHIVE_DEST_n corresponding to the Data Guard standby that Extract must remain behind. Not required when FSFO is enabled.

HANDLEDLFAILOVER Operational Considerations

The following table summarizes how Extract behaves when the standby database is unavailable, falls behind, or undergoes a Data Guard role transition, and identifies the relevant HANDLEDLFAILOVER options and MAA recommendations.

Condition Extract behavior and recommendation
Standby temporarily falls behind or becomes unavailable Extract waits for the standby applied SCN before continuing.
Standby remains unavailable By default, Extract reports a warning after 60 seconds and abends after 30 minutes.
Longer or shorter standby outage tolerance is required Use STANDBY_WARNING and STANDBY_ABEND to adjust the warning and abend intervals.
Data Guard role transition occurs Extract can restart and temporarily process without enforcing HANDLEDLFAILOVER while the new standby is reinstated. The default timeout is 5 minutes.
New standby requires more time to become available DLFAILOVER_TIMEOUT controls the post-role-transition timeout. MAA recommends retaining the default 5-minute value.
Standby will be unavailable for an extended period and Extract must continue Remove HANDLEDLFAILOVER and restart Extract. Re-enable the parameter after the standby is available and caught up. Be aware that a subsequent data-loss failover can result in logical divergence between the source and GoldenGate target.

Data Guard Role Transition Behavior

Following a Data Guard role transition, GoldenGate temporarily suspends HANDLEDLFAILOVER to allow the former primary database to be reinstated as the new standby. During the default 300-second (5-minute) DLFAILOVER_TIMEOUT interval, Extract can continue processing without regard to standby apply progress, so temporary data divergence between the new standby and the GoldenGate target is possible.

If the new standby is not available when this interval expires, HANDLEDLFAILOVER is enforced again and Extract can abend until the standby becomes available. While HANDLEDLFAILOVER is suspended or disabled, a subsequent data-loss failover can result in transactions existing on the GoldenGate target that do not exist on the new source primary.

Configure FAILOVERTARGETDESTID When FSFO Is Not Enabled

When Data Guard Fast-Start Failover is not enabled, configure:

TRANLOGOPTIONS FAILOVERTARGETDESTID <n>

FAILOVERTARGETDESTID identifies the standby destination that Extract must remain behind. It is used with HANDLEDLFAILOVER. When FSFO is enabled, FAILOVERTARGETDESTID is not required.

Determine the appropriate destination from the LOG_ARCHIVE_DEST_n parameters on the GoldenGate source database:

SHOW PARAMETER log_archive_dest

For example, if LOG_ARCHIVE_DEST_2 sends redo to the applicable standby database, configure:

TRANLOGOPTIONS FAILOVERTARGETDESTID 2

Apply the Extract Parameters

Using the Oracle GoldenGate Administration Service, edit the parameter file for each applicable Extract and add the required TRANLOGOPTIONS parameters.

Stop the Extract before modifying its parameter file. After saving the changes, restart the Extract for the new parameters to take effect.

Alternatively, use Admin Client to edit the Extract parameter file:

EDIT PARAMS <extract_name>

Save the parameter file, and then restart the Extract.

For more information about editing parameter files, see Creating a Parameter File Using Admin Client and the TRANLOGOPTIONS parameter in the Oracle GoldenGate 26ai documentation.

Step 9.2: Configure Distribution Paths for Oracle Data Guard Role Transitions

A GoldenGate Distribution Path sends trail data from the source Distribution Service to the target Receiver Service using the target address configured for the path. When the target database and its GoldenGate deployment are protected by Oracle Data Guard, the Receiver Service can move from the original primary system to the standby system following a Data Guard switchover or failover.

Because the primary and standby systems use different application VIPs, the Distribution Path target address must be updated to point to the Receiver Service on the system hosting the new primary database. Without this update, the Distribution Service continues to use the previous target address and cannot resume trail delivery to the relocated Receiver Service.

This configuration is required when the target GoldenGate deployment moves following a target database Data Guard role transition.

A source database role transition does not require changing the target address of a source-initiated Distribution Path because the Receiver Service address remains unchanged.

Data Guard role transition GoldenGate component that relocates Distribution Path change
Source database role transition Distribution Service moves with the source GoldenGate deployment No target-address change required because the Receiver Service address has not changed.
Target database role transition Receiver Service moves to the new target primary system Distribution Path target address must change to the new Receiver Service VIP.

Automate this update by using a database role-change trigger on the target database to invoke a DBMS_SCHEDULER job. The job runs a shell script that uses the GoldenGate REST API to update the Distribution Path target address. The 26ai REST API continues to expose Distribution Path operations under the Distribution Service, including retrieving and modifying Distribution Path resources.

Certificate consideration: If the primary and standby Receiver Service VIPs use certificates signed by different root CAs, add the required CA certificates to the source GoldenGate deployment so that the Distribution Service can establish a secure connection to the Receiver Service on either system following a role transition.

Step 9.2.1: Create the Distribution Path Update Script

Create a shell script that updates the Distribution Path target address after a target database Data Guard role transition. The script uses the GoldenGate REST API to identify the active source Distribution Service and change the Distribution Path to the Receiver Service VIP on the new target primary system.

The example change_path_target.sh script is provided in Appendix A of this chapter. Update the variables and values in the script as appropriate for your GoldenGate environment.

Place the script in the same local directory on all Oracle RAC nodes of both the target primary and standby systems.

For example:

mkdir -p /u01/oracle/goldengate/scripts

Copy change_path_target.sh to this directory and set the required permissions:

chmod 6751 /u01/oracle/goldengate/scripts/change_path_target.sh

On Exadata systems, dcli can be used to create the directory and distribute the script to all database nodes. For example:

/usr/local/bin/dcli -l oracle -g ~/dbs_group mkdir -p /u01/oracle/goldengate/scripts

/usr/local/bin/dcli -l oracle -g ~/dbs_group -f /u01/oracle/goldengate/scripts/change_path_target.sh -d /u01/oracle/goldengate/scripts
Step 9.2.2: Create a DBMS_SCHEDULER Job

Create a DBMS_SCHEDULER executable job on the target primary database to run the Distribution Path update script after a Data Guard role transition.

As the SYSDBA user in the root container of the target primary database, create the job:

exec dbms_scheduler.create_job(job_name=>'gg_change_path_target', job_type=>'EXECUTABLE', number_of_arguments=>6, job_action=>'/u01/oracle/goldengate/scripts/change_path_target.sh', enabled=>FALSE);

The job is created disabled and is executed by the database role-change trigger in Step 9.2.4.

Configure the External Job User

For the scheduler to execute the operating system script, configure the externaljob.ora file in the Oracle Database home on all RAC nodes of both the target primary and standby systems.

# Before
run_user = nobody
run_group = nobody

# After
run_user = oracle
run_group = oinstall

Repeat this configuration on every eligible RAC node in both the target primary and standby databases.

For ExaDB-D and ExaDB-C@C use dbaascli database getDetails to identify the database home associated with the target database when locating externaljob.ora.

Step 9.2.3: Create the GoldenGate REST API Credentials File

Create a configuration file containing the GoldenGate administrator credentials used by the Distribution Path update script to make authenticated REST API calls.

Create the file on the target primary and standby systems in the same local directory used for the script in Step 9.2.1.

For example:

user = "oggadmin:<password>"

Save the file with permissions that restrict access to the Oracle software owner:

chmod 600 /u01/oracle/goldengate/scripts/<deployment_name>.cfg

Copy the credentials file to all eligible RAC nodes on both the target primary and standby systems.

On Exadata systems, dcli can be used to distribute the file. For example:

/usr/local/bin/dcli -l oracle -g ~/dbs_group -f /u01/oracle/goldengate/scripts/<deployment_name>.cfg -d /u01/oracle/goldengate/scripts

Use the credentials file with the Distribution Path update script rather than embedding the password directly in the script.

Step 9.2.4: Create the Database Role-Change Trigger

Create a database role-change trigger on the target primary database to update the Distribution Path target address when a standby database becomes the primary database following an Oracle Data Guard role transition.

The trigger fires after the database role changes and supplies the six arguments required by the gg_change_path_target scheduler job. The scheduler job then executes change_path_target.sh, which updates the Distribution Path target URI through the GoldenGate REST API.

As the SYSDBA user on the target primary system, create the trigger:

CREATE OR REPLACE TRIGGER gg_change_path
AFTER DB_ROLE_CHANGE ON DATABASE
DECLARE
  role     VARCHAR2(30);
  hostname VARCHAR2(64);
BEGIN
  SELECT database_role INTO role FROM v$database;
  SELECT host_name INTO hostname FROM v$instance;

  DBMS_SCHEDULER.SET_JOB_ARGUMENT_VALUE(
    'gg_change_path_target', 1, '<PRIMARY_SOURCE_VIP>');

  DBMS_SCHEDULER.SET_JOB_ARGUMENT_VALUE(
    'gg_change_path_target', 2, '<STANDBY_SOURCE_VIP>');

  DBMS_SCHEDULER.SET_JOB_ARGUMENT_VALUE(
    'gg_change_path_target', 4, '<DISTRIBUTION_PATH_NAME>');

  DBMS_SCHEDULER.SET_JOB_ARGUMENT_VALUE(
    'gg_change_path_target', 5, '<DEPLOYMENT_NAME>');

  DBMS_SCHEDULER.SET_JOB_ARGUMENT_VALUE(
    'gg_change_path_target', 6, '<CREDENTIALS_FILE>');

  IF role = 'PRIMARY' AND hostname LIKE '<PRIMARY_TARGET_CLUSTER>%' THEN
    DBMS_SCHEDULER.SET_JOB_ARGUMENT_VALUE(
      'gg_change_path_target', 3, '<PRIMARY_TARGET_VIP>:443');
  ELSIF role = 'PRIMARY' THEN
    DBMS_SCHEDULER.SET_JOB_ARGUMENT_VALUE(
      'gg_change_path_target', 3, '<STANDBY_TARGET_VIP>:443');
  END IF;

  DBMS_SCHEDULER.RUN_JOB(
    job_name => 'gg_change_path_target');
END;
/

Use the following argument order, which must match the change_path_target.sh script in Appendix A:

Argument Value
1 Primary source Distribution Service VIP
2 Standby source Distribution Service VIP if applicable
3 Distribution Path target VIP
4 Distribution Path name
5 GoldenGate deployment name
6 Credentials file

Note: Ensure that the six argument values in the trigger match the argument order and variable definitions in the change_path_target.sh script in Appendix A. Update the VIP names, Distribution Path name, deployment name, and credentials file to match your environment.

Step 9.3: Configure Replicat for Oracle Data Guard

The Replicat configuration stored with the Oracle GoldenGate deployment in DBFS is available on the standby system after an Oracle Data Guard role transition. Configure the standby environment as described in the preceding tasks so that Replicat can connect to the target database and resume processing when the standby becomes the new primary.

For multitenant databases, use the corresponding PDB service and Oracle Net alias configured for the target PDB.

For active-active replication, MAA recommends using Auto Conflict Detection and Resolution (Auto-CDR) with Parallel Replicat in integrated mode. Auto-CDR automatically detects and resolves supported conflicts that can occur when the same data is modified at multiple replication sites.

Manual conflict detection and resolution using COMPARECOLS and RESOLVECONFLICT can be used when Auto-CDR is not appropriate for the application. Use Non-integrated Replicat with manual conflict detection and resolution where required.

For the general Replicat configuration and high-availability best practices, see <Combined Ch 5-8 Replicat Section>

Task 10: Validate the Oracle GoldenGate and Data Guard Configuration

Before performing an Oracle Data Guard role transition, validate the Oracle GoldenGate configuration and its high-availability dependencies on the primary and standby systems.

Use the below checklist and verify:

Component Validation
Database services Verify that the database services used by Oracle GoldenGate are configured with the PRIMARY role and are running on the current primary database.
DBFS Verify that the DBFS resource is online on the current primary system and that the GoldenGate deployment files are accessible from the configured DBFS mount point.
Oracle GoldenGate software Verify that the same GoldenGate release and patch level is installed in the same location on all eligible primary and standby Oracle RAC nodes.
Oracle Net Verify that the GoldenGate Oracle Net aliases and TNS_ADMIN configuration are consistent on the primary and standby systems.
NGINX Verify that NGINX is running on the required primary system nodes and that the GoldenGate deployment can be accessed through the application VIP.
XAG Verify that the GoldenGate XAG resource is online and that the configured database service, DBFS, NGINX, and network dependencies are correct.
GoldenGate deployment Verify that the GoldenGate deployment and required services are running and accessible on the current primary system.
Extract Verify that each Extract is running and that the Data Guard failover parameters configured in Step 9.1 are present.
Distribution Paths Verify that each Distribution Path is running if applicable and is configured with the Receiver Service address for the current target primary.
Replicat Verify that each required Replicat is running and processing trail data without errors.
Role-change automation Verify that the gg_change_path_target Scheduler job and database role-change trigger are configured on the target primary database and that the REST API credentials file and change_path_target.sh are available in the required local locations.

During a role transition, the database services configured for the PRIMARY role start on the new primary database. Oracle Clusterware and XAG use the configured dependencies to make DBFS and the required network resources available and start the GoldenGate deployment on the system hosting the new primary.

Appendix A: Distribution Path Target Update Script

The below example script named change_path_target.sh updates the target URI of an Oracle GoldenGate Distribution Path after an Oracle Data Guard role transition of the target primary database.

The database role-change trigger configured in Step 9.2.4 invokes the gg_change_path_target Scheduler job, which executes this script.

The script uses the GoldenGate REST API to update the Distribution Path to use the Receiver Service application VIP on the current target primary.

change_path_target.sh

#!/bin/bash

# change_path_target.sh - changes the target host of a GG Distribution Path when the target
# moves between primary/standby systems.
# Example usage:
# ./change_path_target.sh <primary source VIP>:443 <standby source VIP>:443 <path target VIP> <path name> <deployment name> <credentials file>

SOURCE1=$1      # PRIMARY Distribution Service VIP
SOURCE2=$2      # STANDBY Distribution Service VIP
TARGET=$3       # Distribution path target VIP
DPATH=$4        # Distribution path name
DEP=$5          # Deployment name
ACCESS=$6       # Config file containing the deployment credentials. Example contents:
                # user = "oggadmin:<password>"

CONNECT=0

# echo "#${i} - `date`:"
LOGFILE=/tmp/ogg_dpatch_change.txt

result=$(curl -si -K $ACCESS https://$SOURCE1/$DEP/distsrvr/services/v2/sources/$DPATH -X GET| grep HTTP |  awk '{print $2}')

# Will return NULL of nginx not running, 502 if cannot contact server, 200 if contact to server good, and others (404) for other bad reasons:

if [[ -z $result || $result -ne 200 ]]; then  # Managed to access the Distr Server
  echo "`date` - Couldn't contact Distribution Server at $SOURCE1 Deployment $DEP ****" >> $LOGFILE
else # Try the other source host:
  echo "`date` - Got status of Distribution Server at $SOURCE1 Deployment $DEP ***"  >> $LOGFILE
  SOURCE=$SOURCE1
  CONNECT=1
fi

if [ $CONNECT -eq 1 ]; then
# For secure NGINX patch destination (wss)
  PAYLOAD='{"target":{"uri":"wss://'${TARGET}'/services/ggnorth/v2/targets?trail=bb"}}'
  curl -s -K $ACCESS https://$SOURCE/$DEP/distsrvr/services/v2/sources/$DPATH -X PATCH --data '{"status": "stopped"}'

# Set new target for path:
  curl -s -K $ACCESS https://$SOURCE/$DEP/distsrvr/services/v2/sources/$DPATH -X PATCH --data "$PAYLOAD"
  echo "`date` - Set path $DPATH on $SOURCE deployment $DEP:" >> $LOGFILE

  curl -s -K $ACCESS https://$SOURCE/$DEP/distsrvr/services/v2/sources/$DPATH -X GET | python -m json.tool | grep uri >> $LOGFILE
  curl -s -K $ACCESS https://$SOURCE/$DEP/distsrvr/services/v2/sources/$DPATH -X PATCH --data '{"status": "running"}'
  exit 0
else
  echo "`date` - ERROR: COULDN'T CHANGE DISTRIBUTION PATH ($DPATH) in Deployement $DEP at $SOURCE! ***" >> $LOGFILE
fi

# If here, means we couldn't connect to either Distribution Services
exit 1