OnPremises and Cloud: Oracle GoldenGate Microservices Architecture on Oracle Exadata Configuration Best Practices

Use these best practices for configuring Oracle GoldenGate Microservices Architecture to work with Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D) or Oracle Exadata Database Service on Cloud@Customer (ExaDB-C@C), and with Oracle Database File System (DBFS) or Oracle ASM Cluster File System (ACFS).

Overview

Use these best practices to configure Oracle GoldenGate Microservices Architecture on Oracle Real Application Clusters (Oracle RAC), on-premises Oracle Exadata, Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D), or Oracle Exadata Database Service on Cloud@Customer (ExaDB-C@C).

The architecture uses Oracle RAC, Oracle Clusterware, shared storage on Oracle Advanced Cluster File System (ACFS) or Oracle Database File System (DBFS), and Oracle Grid Infrastructure Standalone Agent (XAG) to provide high availability for Oracle GoldenGate and coordinate its dependencies on the database, network, and file system. Oracle GoldenGate software is installed locally on the eligible Oracle RAC nodes, while the GoldenGate deployment files are stored in shared storage so that they are available to the appropriate system following RAC node failures and planned maintenance.

When the database is protected by Oracle Data Guard, complete the configuration described in this chapter first, then proceed to OnPremises and Cloud: Oracle GoldenGate Microservices Architecture Integrated with Active Data Guard to perform additional required configuration.

Scope and applicability

This chapter provides configuration best practices for deploying Oracle GoldenGate Microservices Architecture on the same servers that host Oracle Database in an Oracle Real Application Clusters (Oracle RAC) or Oracle Exadata Database Service environment. The database can be a GoldenGate source, target, or both.

Deployment platform Applicability
On-premises Oracle RAC, including Oracle Exadata Use the common Tasks 1–11.
Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D) Use the common Tasks 1–11 and the ExaDB-D prerequisites in Exadata Database Service additions.
Oracle Exadata Database Service on Cloud@Customer (ExaDB-C@C) Use the common Tasks 1–11 and the ExaDB-C@C prerequisites in Exadata Database Service additions.

The common task sequence covers Oracle Database configuration, shared storage, Oracle GoldenGate installation and deployment, Oracle Clusterware and XAG integration, NGINX reverse proxy, Oracle Net aliases, replication processes, and process autostart. Platform-specific content is limited to prerequisites and decision points that differ for ExaDB-D or ExaDB-C@C.

Summary of Recommendations

When configuring Oracle GoldenGate in an Oracle RAC environment, follow these recommendations.

Shared-storage decision

Use ACFS for a RAC-only deployment when Oracle GoldenGate does not need to relocate with a Data Guard primary database. Use DBFS when the GoldenGate deployment must follow the protected database to a new primary after a Data Guard role transition. DBFS is stored in and protected with the database and can be mounted by XAG before GoldenGate starts on the new primary.

Follow Task 4 for the common ACFS or DBFS requirements. Keep the software installation local to each eligible node and place the Service Manager and deployment homes on the selected shared file system.

Exadata Database Service additions

Use this section with the common Tasks 1–11 when Oracle GoldenGate is co-located with Oracle Database on ExaDB-D or ExaDB-C@C.

Platform prerequisites

ExaDB-D networking and application VIP

ExaDB-C@C networking and application VIP

Cloud network validation

Before configuring a Distribution Path:

  1. Resolve the target application VIP FQDN from every eligible source node.
  2. Verify route and firewall reachability to the target NGINX HTTPS listener.
  3. Confirm the NGINX certificate chain is trusted and its Subject Alternative Name matches the target FQDN.
  4. Confirm the GoldenGate internal HTTP ports are unreachable from a remote host.
  5. After configuration, verify HTTPS administration and WSS Distribution Path connectivity through the application VIP.

Task 1: Configure the Oracle Database for Oracle GoldenGate

The source and target Oracle GoldenGate databases should be configured using the following recommendations.

Replication Use Case FORCE LOGGING Subset Database Replication Logging Schema/Table Supplemental Logging
Multiple or all PDBs in a CDB Enable CDB level or PDB level FORCE LOGGING. PDB-level FORCE LOGGING can be used when the scope is limited to specific PDB(s). Determine whether per-PDB supplemental logging is enabled. Configure Subset Database Replication Logging at the PDB level when per-PDB supplemental logging is enabled; otherwise configure it at CDB$ROOT. Configure supplemental logging for the schemas or tables being replicated in each source PDB.
Single PDB Enable at the CDB or PDB level as appropriate. Configure according to the per-PDB supplemental logging setting. Configure supplemental logging for the schemas or tables being replicated in the source PDB.
All or most tables in a schema Enable at the CDB or PDB level as appropriate. Configure according to the per-PDB supplemental logging setting. Prefer schema-level supplemental logging for the replicated schema.
Selected tables in a schema Enable at the CDB or PDB level as appropriate. Configure according to the per-PDB supplemental logging setting. Use table-level supplemental logging for the replicated tables.

Schema and Table Supplemental Logging Selection

Choose the schema-level or table-level supplemental logging option according to the replication scope and Replicat mode:

Subset Database Replication Logging Configuration

Determine whether per-PDB supplemental logging is enabled and supported for the deployed Oracle Database release and configuration. Connect to the source PDB and execute:

SELECT SYS_CONTEXT( 'USERENV', 'IS_PER_PDB_SUPPLEMENTAL_LOGGING' ) AS PERPDB_SUPLOG FROM DUAL; If the query indicates that per-PDB supplemental logging is enabled, configure Subset Database Replication logging in each source PDB:

ALTER PLUGGABLE DATABASE <pdbname> ADD SUPPLEMENTAL LOG DATA SUBSET DATABASE REPLICATION; Validate Subset Database Replication logging while connected to the source PDB:

SELECT * FROM DBA_SUPPLEMENTAL_LOGGING; If the query does not indicate that per-PDB supplemental logging is enabled, connect to CDB$ROOT and configure Subset Database Replication logging at the CDB level so that the user PDBs inherit the setting:

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA SUBSET DATABASE REPLICATION; Run the validation query in the same container where the setting was configured. For per-PDB logging, validate it in each source PDB. For CDB-level logging, validate it in CDB$ROOT and confirm that the setting is effective in every source PDB before starting Extract.

Logging Validation

Complete the following validation before starting Extract:

For more information about Subset Database Replication see GoldenGate Microservices Architecture Documentation section at https://docs.oracle.com/en/database/goldengate/core/26/coredoc/prepare-transaction-logs-settings-and-requirements-ogg-oracle.html#GUID-4BF9C545-1AD7-4364-8B26-1D3A1F6460E4.

Task 2: Create the Database Replication Administrator User

The source and target Oracle databases need a GoldenGate Administrator user with appropriate privileges assigned.

For information about creating the Oracle GoldenGate administrator user and granting the required privileges for both CDB and non-CDB (19c) databases, see Prepare Database Users and Privileges for Oracle.

For a multitenant source database, Oracle recommends using per-PDB Extract with a local GoldenGate administrator user in each source PDB. For a multitenant target database, a separate GoldenGate administrator user is needed for each PDB that Replicat applies data to.

If using a CDB-level integrated Extract, configure a common GoldenGate administrator user in the root container as required for that configuration.

For more details about creating a GoldenGate Administrator in an Oracle multitenant database, see Configuring a Multitenant Container Database section in the document Configure a Multitenant Container Database

Task 3: Create the Database Services

A database service is required so that the Oracle Grid Infrastructure Agent automatically starts the GoldenGate deployment when the database is opened. When DBFS is used for the shared file system, the database service is also used to mount DBFS to the correct Oracle RAC instance.

When using a source multitenant database, configure a service for each PDB that contains schemas being replicated. A separate service for the root container (CDB) is required only for a CDB-level integrated Extract. For a target multitenant database, configure a service for the target PDB.

Create the database service as a singleton service with a preferred Oracle RAC instance and one or more available instances using the following command, as the oracle user.

$ srvctl add service -db <db_name> -service <service name> -preferred <instance_1>
-available <instance_2, instance_3 etc.> -pdb <PDB name>

For example:

$ srvctl add service -db ggdb -service oggserv_pdb -preferred ggdb1 -available ggdb2
-pdb GGPDB01

For Oracle Database 19c non-CDB databases, omit the -pdb parameter. The -pdb parameter is also not used for a root CDB Extract service.

Refer to the Real Application Clusters Administration and Deployment Guide for further details on creating a database service at SVRCTL Usage Information.

Task 4: Set Up a File System on Oracle RAC

Oracle GoldenGate Microservices Architecture is designed with a simplified installation and deployment directory structure. The software installation directory should be placed on local storage on each Oracle RAC node to provide minimized downtime during software patching.

The Oracle GoldenGate deployment directory must be placed on a shared file system. The deployment directory is created when the GoldenGate deployment is created and contains the deployment-specific configuration, security artifacts, parameter files, logs, trail files, checkpoint files, and other runtime data required by the deployment.

Placing the deployment in DBFS or ACFS provides the best recovery and failover capabilities in the event of a system failure. Ensuring the availability of the checkpoint files cluster-wide is essential so that after a failure occurs the GoldenGate processes can continue running from their last known position.

Place the Oracle GoldenGate deployment directories on shared storage that is accessible through the same path on every eligible Oracle RAC node. For a RAC-only deployment, Oracle ACFS is the recommended shared file system. When the Oracle GoldenGate deployment must relocate to the new primary database following an Oracle Data Guard role transition, use DBFS and complete the additional configuration described in the next chapter. Because DBFS is stored in and protected with the database, XAG can mount it on the new primary system before starting Oracle GoldenGate.

Follow the instructions in the appropriate section below to configure the file system.

Exadata Database Service storage notes

For ExaDB-D and ExaDB-C@C, apply the same shared-path, ownership, Clusterware dependency, and single-active-deployment requirements described below. When DBFS is selected, ensure the required FUSE packages, mount point, database objects, Oracle Net alias, and Clusterware resource are configured on every eligible node. Use prompted credential entry, an Oracle wallet, or an approved external password store; do not place the DBFS password on a command line.

Oracle Advanced Cluster File System (ACFS)

Use Oracle ACFS for the shared Oracle GoldenGate file system in a RAC-only configuration. If Oracle Data Guard integration is planned and Oracle GoldenGate must relocate automatically after a role transition, select DBFS during the initial design to avoid a later migration of the deployment directories.

Review My Oracle Support Knowledge Article KB337533, Oracle ACFS Support on Oracle Exadata Database Machine (Linux only), for Oracle ACFS configuration requirements.

Create a single ACFS file system for storing the GoldenGate deployment files.

It is recommended that you allocate enough trail file disk space to permit storage of up to 12 hours of trail files. Doing this provides sufficient space for trail file generation should a problem occur with the target environment that prevents it from receiving new trail files. The amount of space needed for 12 hours can only be determined by testing trail file generation rates with real production data.

1. Create the file system using ASMCMD as the Oracle ASM administrator user.

ASMCMD [+] > volcreate -G datac1 -s 1200G ACFS_GG

Note: Modify the file system size according to the determined size requirements.

ASMCMD> volinfo -G datac1 acfs_gg

Diskgroup Name: DATAC1
         Volume Name: ACFS_GG
         Volume Device: /dev/asm/acfs_gg-151
         State: ENABLED
         Size (MB): 1228800
         Resize Unit (MB): 64
         Redundancy: MIRROR
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage:
         Mountpath:

2. Make the file system with the following mkfs command.

$ /sbin/mkfs -t acfs /dev/asm/acfs_gg-151

3. Create the CRS resource for the newly created ACFS file system, if not already created. Check to see if the file system resource was already created.

$ srvctl status filesystem -volume ACFS_GG -diskgroup DATAC1

ACFS file system /mnt/acfs_gg is mounted on nodes oggadm07

If not already created, create the ACFS mount point on all of the Oracle RAC nodes.

$ mkdir -p /mnt/acfs_gg

4. Create the file system resource as the root user.

For this Oracle GoldenGate configuration, it is not recommended to mount the ACFS file system on more than one node at a time. This prevents accidental starting of the GoldenGate deployment on multiple nodes, which can lead to file corruptions.

5. Create the ACFS resource using srvctl from the Oracle Grid Infrastructure ORACLE_HOME.

$ srvctl add filesystem -device /dev/asm/acfs_gg-151 -volume ACFS_GG -diskgroup DATAC1 -path /mnt/acfs_gg -user oracle -autostart RESTORE

6. To verify the currently configured ACFS file systems, use the following command to view the file system details.

$ srvctl config filesystem

Volume device: /dev/asm/acfs_gg-151
Diskgroup name: datac1
Volume name: ACFS_GG
Canonical volume device: /dev/asm/acfs_gg-151
Accelerator volume devices:
Mountpoint path: /mnt/acfs_gg
Mount point owner: oracle

7. Check the status of the ACFS resource and mount it.

$ srvctl status filesystem -volume ACFS_GG -diskgroup DATAC1

ACFS file system /mnt/acfs_gg is not mounted

$ srvctl start filesystem -volume ACFS_GG -diskgroup DATAC1 -node dc1north01

The CRS resource that is created is named using the format ora.diskgroup_name.volume_name.acfs. Using the above file system example, the CRS resource is called ora.datac1.acfs_gg.acfs.

8. To see all ACFS file system CRS resources that currently exist, use the following command.

$ crsctl stat res -w "((TYPE = ora.acfs.type) OR (TYPE = ora.acfs_cluster.type))"

NAME=ora.datac1.acfs_gg.acfs
TYPE=ora.acfs.type
TARGET=ONLINE              , OFFLINE
STATE=ONLINE on dc1north01, OFFLINE

9. Create a GoldenGate deployment directory on ACFS. After the file system is mounted, create the directory for storing the GoldenGate deployments.

$ cd /mnt/acfs_gg
$ mkdir deployments

Refer to the Oracle Automatic Storage Management Administrator’s Guide for more information about ACFS see the Oracle Automatic Storage Management Administrator’s Guide.

Note: Leave the ACFS file system mounted. It is required for creating the GoldenGate deployment in a later step.

Oracle Database File System (DBFS)

It is required that you create the DBFS tablespace inside the same database that the Oracle GoldenGate processes will be connected to. For example, if a GoldenGate Extract process is extracting from a database called GGDB, the DBFS tablespace would be located in the same GGDB database.

Ensure that the required FUSE packages and libraries are installed on each server hosting the Oracle GoldenGate deployment. If they are not already installed, follow the instructions in My Oracle Support Knowledge Article KB140582, Installing the Database File System (DBFS).

Follow the instructions in My Oracle Support Knowledge Article KB136383, Configuring DBFS on Oracle Exadata Database Machine, to configure the database, DBFS tablespace, database user, Oracle Net (tnsnames.ora) connection alias, and the required permissions for DBFS on the source or target Oracle GoldenGate environment.

Note: For Multitenant databases, create the DBFS tablespace in the same PDB used by the GoldenGate Extract or Replicat. It is recommended that you use the same PDB that the GoldenGate Extract or Replicat processes are connecting to, allowing DBFS to use the same database service, created above, for its database dependency.

When using a root CDB Extract, a PDB should be designated as the DBFS data store, and a PDB service must be created for the designated PDB.

When you create a file system for storing the GoldenGate deployment files, it is recommended that you allocate enough trail file disk space to permit storage of up to 12 hours of trail files. Doing this provides sufficient space for trail file generation should a problem occur with the target environment that prevents it from receiving new trail files. The amount of space needed for 12 hours can only be determined by testing trail file generation rates with real production data.

Example DBFS creation:

$ cd $ORACLE_HOME/rdbms/admin $ sqlplus /nolog SQL> CONNECT
dbfs_user@<database_tns_alias> Enter password: [prompted] SQL> start
dbfs_create_filesystem dbfs_gg_tbs goldengate

Security: Do not place the DBFS database password on the command line. Use the prompted connection shown above or an organizationally approved Oracle wallet or external password store for unattended execution.

Follow the instructions in My Oracle SupportKnowledge Article KB136383 to configure the newly created DBFS file system so that the DBFS instance and mount point resources are automatically started by Cluster Ready Services (CRS) after a node failure, with the following DBFS configuration and script file modifications.

1. Change the mount-dbfs.conf parameters to reflect your database environment.

Modify the MOUNT_OPTIONS parameter to the following:

MOUNT_OPTIONS=allow_other,direct_io,failover,nolock

The failover option forces all file writes to be committed to the DBFS database in an IMMEDIATE WAIT mode. This prevents data getting lost when it has been written into the dbfs_client cache but not yet written to the database at the time of a database or node failure.

The nolock mount option is required if you are using Oracle Database 18c or later versions, due to a change in the DBFS file locking, which can cause issues for GoldenGate processes after a RAC node failure when a file is currently locked.

2. Modify the mount-dbfs.sh script to force unmounting of DBFS when the CRS resource is stopped. Change two occurrences of:

$FUSERMOUNT -u $MOUNT_POINT

To the following:

$FUSERMOUNT -uz $MOUNT_POINT

When registering the resource with Oracle Clusterware, be sure to create it as a cluster_resource instead of a local_resource as specified in the My Oracle Support note. The reason for using cluster_resource is so the file system can only be mounted on a single node at one time, preventing mounting of DBFS from concurrent nodes, which creates the potential for concurrent file writes, causing file corruption problems.

3. Make sure to use the database service name created in a previous step for the DBFS service dependency.

For example:

DBNAME=ggdb
DEPNAME=ora.$DBNAME.oggserv.svc

crsctl add resource $RESNAME   -type cluster_resource \

-attr "ACTION_SCRIPT=$ACTION_SCRIPT,          CHECK_INTERVAL=30,RESTART_ATTEMPTS=10, \
START_DEPENDENCIES='hard($DEPNAME)pullup($DEPNAME)',          STOP_DEPENDENCIES='hard($DEPNAME)',          SCRIPT_TIMEOUT=300"

4. Once the DBFS resource has been created, the file system should be mounted and tested.

$ crsctl start res dbfs_mount
$ crsctl stat res dbfs_mount

5. After the file system is mounted, create the directory for storing the GoldenGate files.

$ cd /mnt/dbfs/goldengate
$ mkdir deployments

Note: Leave the file system mounted. It is required for creating the GoldenGate deployment in a later step.

Task 5: Install Oracle GoldenGate

Download and install a currently supported Oracle GoldenGate release and the latest recommended bundle patch certified for every source and target database release in the deployment.

Obtain the latest recommended GoldenGate software using guidelines provided at How to obtain the latest Oracle GoldenGate Microservices Architecture bundle patches and My Oracle Support Knowledge Article KB159960, Primary Note for Oracle GoldenGate Core Product Patch Sets.

Note: Use a currently supported Oracle GoldenGate release and apply the latest recommended bundle patch that is certified for all source and target database releases in the deployment. Before installation or upgrade, review current certification information, applicable security advisories, and My Oracle Support Knowledge Article KB159960. Do not rely on a fixed minimum release in this chapter because security and patch requirements change over time.

Install the Oracle GoldenGate software locally on all nodes in the Oracle RAC configuration that will be part of the GoldenGate configuration. Make sure the installation directory is identical on all nodes.

Follow the generic installation instructions detailed in Oracle GoldenGate Microservices Architecture Documentation.

Task 6: Create the Oracle GoldenGate Deployment

Once the Oracle GoldenGate software is installed, create the Service Manager and Oracle GoldenGate deployment. The Service Manager and deployment can be created using the Oracle GoldenGate Configuration Assistant (oggca.sh) or through automation such as the MAA GGHub Ansible playbooks. This section describes the deployment requirements and MAA-recommended configuration when using the Oracle GoldenGate Configuration Assistant.

Review My Oracle Support Knowledge Article KB422397, Best Practices - Oracle GoldenGate Microservices Architecture with XAG, for the current requirements and best practices when configuring Oracle GoldenGate Microservices Architecture with the Oracle Grid Infrastructure Standalone Agent (XAG):

  1. A single Service Manager can manage multiple deployments when registered with XAG, provided that all deployments use the same OGG\_HOME as the Service Manager.
  2. The Oracle GoldenGate administrator username and password configured for the Service Manager and deployments must match the credentials registered with XAG.
  3. Multiple Service Managers can run on the same server or cluster. Each Service Manager must use a unique OGG\_HOME. The OGG\_HOME can be a physical directory or a symbolic link. Rather than installing the Oracle GoldenGate software multiple times, the recommended approach is to install the software once and create a unique symbolic link to the physical OGG\_HOME for each Service Manager.

For example, assume Oracle GoldenGate is installed in the following physical OGG\_HOME:

$ echo $OGG_HOME
/u01/oracle/goldengate/gg26ai_MS

Create a unique symbolic link to the physical OGG\_HOME for the Service Manager:

$ ln -s /u01/oracle/goldengate/gg26ai_MS /u01/oracle/goldengate/gg26ai_MS_ggnorth

Set OGG_HOME to the symbolic-link path:

$ export OGG_HOME=/u01/oracle/goldengate/gg26ai_MS_ggnorth

Run the Oracle GoldenGate Configuration Assistant from this OGG_HOME:

$ $OGG_HOME/bin/oggca.sh

The Service Manager and deployment are then configured using the OGG\_HOME represented by the symbolic link.

Note: Create the OGG\_HOME symbolic link and set the OGG\_HOME environment variable on all Oracle RAC nodes before running the Oracle GoldenGate Configuration Assistant. The symbolic-link path and target must be consistent across all nodes that can host the GoldenGate deployment.

Oracle GoldenGate Configuration Assistant Recommendations

The following sections describe the Oracle GoldenGate Configuration Assistant (OGGCA) screens and highlight the MAA-recommended settings for creating the Service Manager and GoldenGate deployment.

  1. Service Manager Deployment
    • Select New Service Manager when creating a new Service Manager, or select Existing Service Manager when adding a deployment to an already configured Service Manager that meets the XAG requirements described earlier in this task.
    • Verify that Software Home specifies the OGG\_HOME configured for the GGHub instance. When multiple Service Managers are configured on the same cluster, use the unique OGG\_HOME or symbolic-link path associated with that Service Manager.
    • Specify the Service Manager Deployment Home on the shared DBFS or ACFS file system when creating a new Service Manager.
    • Specify 127.0.0.1 (localhost) for Hostname/IP Address.
      • This prevents direct remote access to the GoldenGate Microservices while allowing the deployment to run on any eligible Oracle RAC node. External access is provided securely through the application VIP and NGINX reverse proxy described in Task 8.
    • Select Integrate with XAG.
    • Specify an available Service Manager port e.g. 9300. Ensure that the Service Manager port and the additional ports required for the GoldenGate deployment services are available and do not conflict with other services on the Oracle RAC nodes.
    • TLS is not required between NGINX and the Oracle GoldenGate Microservices only when the Service Manager and every deployment microservice are bound exclusively to 127.0.0.1 and direct remote access to their internal HTTP ports is prevented. All external access must use the NGINX HTTPS listener. Validate that the Service Manager, Administration Service, Distribution Service, Receiver Service, and Performance Metrics Service cannot be reached remotely through their internal ports.
  2. Service Manager Administrator
    • Configure the GoldenGate administrator account.
      • The administrator credentials must match the credentials used when registering with XAG.
  3. User Deployment
    • Specify a unique Deployment Name.
    • Verify that Software Home specifies the OGG_HOME configured for the GGHub instance.
    • Specify the Deployment Home on the shared DBFS or ACFS file system. The GoldenGate Deployment Home must be separate from and not reside within the Service Manager Deployment Home.
    • OGGCA automatically selects and displays ports for the Administration Service, Distribution Service, Receiver Service, and Performance Metrics Service based on the specified Service Manager port. Verify that the selected ports are available on all Oracle RAC nodes.
    • Configure TNS_ADMIN to reference the directory containing the Oracle Net configuration files. The same path and required files must be available on all Oracle RAC nodes.
    • If Performance Metrics Service is enabled, configure the required metrics datastore. If a local datastore directory is used, the same directory path must exist on all Oracle RAC nodes.
  4. User Deployment Administrator
    • Use the same administrator credentials as the Service Manager administrator, as required for the XAG-managed configuration.
  5. Summary
    • Review the Service Manager, deployment directories, environment variables, administrator accounts, security settings, service ports, and replication settings before creating the deployment.
    • Optionally save the configuration response file for future OGGCA silent-mode deployments.

Continue through the Oracle GoldenGate Configuration Assistant to create the deployment.

For detailed descriptions of the Oracle GoldenGate Configuration Assistant screens and available deployment options, see Add a Deployment in the Oracle GoldenGate Microservices Architecture Documentation.

Task 7: Oracle Clusterware Configuration

The following procedure shows you how to configure Oracle Clusterware to manage Oracle GoldenGate using the Oracle Grid Infrastructure Standalone Agent (XAG). Using XAG automates the mounting of the shared file system (DBFS or ACFS) and the stopping and starting of the GoldenGate deployment when relocating between Oracle RAC nodes.

1. Install the Oracle Grid Infrastructure Standalone Agent.

It is recommended that you install the XAG software as a standalone agent outside of the Grid Infrastructure ORACLE_HOME. This allows you to use the latest XAG release available, and the software can be updated without impact to the Grid Infrastructure.

When using Oracle GoldenGate Microservices Architecture you MUST use XAG version 10.2 or later.

The latest agent software is available for download from the following location:

Oracle Grid Infrastructure Standalone Agents for Oracle Clusterware

Install the XAG standalone agent outside of the Oracle Grid Infrastructure home directory. XAG must be installed in the same directory on all RAC nodes in the cluster where Oracle GoldenGate is installed.

For example, as the Oracle Grid Infrastructure user, the default of oracle:

$ ./xagsetup.sh --install --directory /u01/oracle/xag --all_nodes

Add the location of the newly installed XAG software to the PATH variable so that the location of agctl is known when the oracle user logs on to the machine.

$ cat .bashrc
export PATH=/u01/oracle/xag/bin:$PATH

Note: It is important to make sure that the XAG bin directory is specified BEFORE the Grid Infrastructure bin directory, to ensure the correct agctl binary is found. Set this location in the oracle user environment to take effect at time of logging on, such as in the .bashrc file when the Bash shell is in use.

2. Prepare for Application Virtual IP Address (VIP) Creation. A dedicated application VIP is required to allow access to the GoldenGate Microservices using the same host name, regardless of which RAC node is hosting the services. An application VIP also ensures that the GoldenGate Distribution Server can communicate with the Distribution Receiver running on the current RAC node.

The VIP is a cluster resource that Oracle Clusterware manages. The VIP is assigned to a cluster node and is automatically migrated to another node in the event of a node failure.

There are two pieces of information needed before creating the application VIP:

The network number, which can be identified using the following command.

$ crsctl status resource -p -attr ADDRESS_TYPE,NAME,USR_ORA_SUBNET -w "TYPE = ora.network.type" |sort | uniq

ADDRESS_TYPE=IPV4
NAME=ora.net1.network

USR_ORA_SUBNET=10.133.16.0

The net1 in NAME=ora.net1.network indicates that this is network 1, and it is of type IPV4.

The IP address for the new Application VIP, provided by your system administrator. This IP address must be in the same subnet of the cluster environment as determined above. The VIP will be created in the next step, when configuring the Oracle Grid Infrastructure Agent.

3. Configure Oracle Grid Infrastructure Agent (XAG).

Oracle GoldenGate must be registered with XAG so that the deployment is started and stopped automatically when the database is started and the file system is mounted.

To register Oracle GoldenGate Microservices Architecture with XAG use the following command format.

agctl add goldengate <instance_name>
  --gg_home <GoldenGate_Home>
  --service_manager yes
  --config_home <GoldenGate_SvcMgr_Config>
  --var_home <GoldenGate_SvcMgr_Var Dir>
  --port <port number>
  --oracle_home <$OGG_HOME/lib/instantclient>
  --adminuser <OGG admin user>
  --user <GG instance user>
  --group <GG instance group>
  --network <network_number>
  --ip <ip_address>
  --vip_name <vip_name>
  --filesystems <CRS_resource_name>
  --db_services <service_name>
  --use_local_services
  --nodes <node1, node2, ... ,nodeN>

Where:

Note:

Below are some examples of registering Oracle GoldenGate with XAG.

Example 1: Oracle RAC cluster using DBFS and an existing application VIP

$ agctl add goldengate GGNORTH \ --gg_home /u01/oracle/goldengate/gg26ai_MS \ --service_manager yes \ --config_home /mnt/dbfs/goldengate/deployments/ggsm01/etc/conf \ --var_home /mnt/dbfs/goldengate/deployments/ggsm01/var \ --port 9100 \ --oracle_home /u01/oracle/goldengate/gg26ai_MS/lib/instantclient \ --adminuser oggadmin \ --user oracle \ --group oinstall \ --vip_name gg_vip_prmy \ --filesystems dbfs_mount \ --db_services ora.ds.oraserv.svc \ --use_local_services \ --nodes dc1north01,dc1north02

Where:

Example 2: Oracle RAC cluster using ACFS and an application VIP, with Oracle GoldenGate running on a subset of the cluster nodes

$ agctl add goldengate GGNORTH \ --gg_home /u01/oracle/goldengate/gg26ai_MS \ --service_manager yes \ --config_home /mnt/acfs/goldengate/deployments/ggsm01/etc/conf \ --var_home /mnt/acfs/goldengate/deployments/ggsm01/var \ --port 9100 \ --oracle_home /u01/oracle/goldengate/gg26ai_MS/lib/instantclient \ --adminuser oggadmin \ --user oracle \ --group oinstall \ --network 1 --ip 10.13.11.203 \ --filesystems ora.datac1.acfs_gg.acfs \ --db_services ora.ds.oraserv.svc \ --use_local_services \ --nodes dc1north01,dc1north02

Where:

Example AGCTL Commands

The following example agctl commands can be used to manage the GoldenGate deployment with XAG.

To check the status of Oracle GoldenGate:

$ agctl status goldengate

Goldengate instance ‘GGNORTH’ is running on dc1north01

To start the GoldenGate deployment and all GoldenGate processes configured to automatically start (instructions are provided in a later step):

$ agctl start goldengate GGNORTH --node dc1north02

To stop the GoldenGate deployment:

$ agctl stop goldengate GGNORTH

To manually relocate the GoldenGate deployment to another node:

$ agctl relocate goldengate GGNORTH --node dc1north02

Note: agctl relocate is the preferred method for planned relocation of a GoldenGate deployment. Proper relocation depends on the GoldenGate resource having the correct Clusterware dependencies (application VIP, DBFS/ACFS filesystem, etc.), which are configured when registering the GoldenGate instance with XAG.

To view the configuration parameters for the GoldenGate resource:

$ agctl config goldengate GGNORTH

Instance name: GGNORTH
Application GoldenGate location is: /u01/oracle/goldengate/gg26ai_MS
Goldengate MicroServices Architecture environment: yes
Goldengate Service Manager configuration directory: /mnt/dbfs/goldengate/deployments/ggsm01/etc/conf

Goldengate Service Manager var directory: /mnt/dbfs/goldengate/deployments/ggsm01/var

Service Manager Port: 9100
Goldengate Administration User: oggadmin
Autostart on DataGuard role transition to PRIMARY: no
Configured to run on Nodes: dc1north01 dc1north02
ORACLE_HOME location is: /u01/oracle/goldengate/gg26ai_MS/lib/instantclient
Database Services needed: ora.cdb1.oggcdb.svc [use_local_services]
File System resources needed: dbfs_mount or ora.datac1.acfs_gg.acfs
Network: 1, IP: 10.13.11.203, User:oracle, Group:oinstall

To delete the GoldenGate XAG resource:

$ agctl stop goldengate GGNORTH
# agctl remove goldengate GGNORTH

For additional Oracle GoldenGate Microservices Architecture with XAG configuration guidance, see My Oracle Support Knowledge Article KB422397, Best Practices - Oracle GoldenGate Microservices Architecture with XAG and XAG download link.

Task 8: Configure NGINX Reverse Proxy

Follow the instructions in My Oracle Support Knowledge Article KB101098 to install and configure the NGINX reverse proxy with SSL for secure external communication.

Configure a separate NGINX reverse-proxy configuration for each Service Manager and application VIP pair. Bind the Service Manager and all Oracle GoldenGate deployment microservices exclusively to 127.0.0.1, and verify that direct access to their internal HTTP ports fails from a remote host. Use a CA-signed server certificate whose Subject Alternative Name matches the external VIP host name, install the complete certificate chain required by NGINX, and define procedures for certificate renewal and rotation. After configuration or relocation, run nginx -t, verify access through the NGINX HTTPS listener, and repeat the direct-port negative test.

Note: Set the NGINX ssl_certificate parameter to a single certificate file containing the server certificate followed by the required intermediate CA certificates. Install and protect the corresponding private key according to organizational security standards, and ensure that clients trust the issuing root CA.

Oracle Clusterware needs to have control over starting the NGINX reverse proxy so that it can be started automatically before the GoldenGate deployments are started.

The NGINX resource is created with a dependency on the underlying network CRS resource, the name of which can be determined using the following command:

$ $GRID_HOME/bin/crsctl stat res -w "TYPE == ora.network.type"|grep NAME

NAME=ora.net1.network

Create the Clusterware resource to manage NGINX using root privileges, either as the root user or by using sudo with an appropriately privileged user. Use the network name identified in the previous step.

$ $GRID_HOME/bin/crsctl add resource nginx -type generic_application -attr "ACL='owner:root:rwx,pgrp:root:rwx,other::r--,group:oinstall:r-x,user:oracle:rwx',EXECUTABLE_NAMES=nginx,START_PROGRAM='/bin/systemctl start -f nginx',STOP_PROGRAM='/bin/systemctl stop -f nginx',CHECK_PROGRAMS='/bin/systemctl status nginx' ,START_DEPENDENCIES='hard(ora.net1.network) pullup(ora.net1.network)', STOP_DEPENDENCIES='hard(intermediate:ora.net1.network)', RESTART_ATTEMPTS=0, HOSTING_MEMBERS='dc1north01,dc1north02', CARDINALITY=2"

The NGINX resource created in this example runs on the cluster nodes specified by HOSTING_MEMBERS. This configuration is recommended when multiple GoldenGate Service Managers are configured and can independently relocate between cluster nodes.

Once the NGINX Clusterware resource is created, alter the GoldenGate XAG resources so that NGINX must be started before the GoldenGate deployments are started.

As the oracle user, modify the XAG resources using the following example commands.

Determine the current –filesystems parameter:

$ agctl config goldengate GGNORTH|grep "File System"

File System resources needed: ora.datac1.acfs_gg.acfs

Modify the –filesystems parameter:

$ agctl modify goldengate GGNORTH --filesystems ora.datac1.acfs_gg.acfs,nginx

Repeat the above commands for each of the XAG GoldenGate registrations relying on NGINX.

Task 9: Create Oracle Net TNS Alias for Oracle GoldenGate Database Connections

Create a TNS alias on all of the RAC nodes where Oracle GoldenGate may be started to provide local database connections for the GoldenGate processes when switching between Oracle RAC nodes. Create the TNS alias in the tnsnames.ora file in the TNS_ADMIN directory specified in the deployment creation.

For a multitenant source database, create a TNS alias for each PDB containing schemas replicated by a per-PDB Extract. If a CDB-level integrated Extract is configured, also create a TNS alias for the CDB service. For a target multitenant database, create a TNS alias for the PDB where replicated data is applied. Set the SERVICE_NAME in each TNS alias to the corresponding database service created in Task 3: Create the Database Services.

Define the following PDB alias locally on every Oracle RAC node that is eligible to run the Oracle GoldenGate deployment:

OGGSOURCE_PDB =     (DESCRIPTION =       (ADDRESS = (PROTOCOL=IPC)(KEY=LISTENER))       (CONNECT_DATA =         (SERVICE_NAME = oggserv_pdb)        )     )

The following CDB alias is required only when a CDB-level integrated Extract is configured:

OGGSOURCE_CDB =     (DESCRIPTION =       (ADDRESS = (PROTOCOL=IPC)(KEY=LISTENER))       (CONNECT_DATA =         (SERVICE_NAME = oggserv_cdb)        )     )

On every eligible Oracle RAC node, validate each required alias with tnsping and an authenticated database connection. For example, run tnsping OGGSOURCE_PDB, then use sqlplus /nolog and CONNECT <GoldenGate_administrator>@OGGSOURCE_PDB so that the password is entered at the prompt. Repeat the validation for OGGSOURCE_CDB when the CDB-level alias is required.

Note: When the tnsnames.ora or sqlnet.ora, located in the TNS_ADMIN directory for the GoldenGate deployment, are modified, the deployment needs to be restarted in order to pick up the changes.

With the GoldenGate deployment created, use the Administration Service to create the required database connections using the TNS alias names. For more information, see ⁠Add Database Connections in the Oracle GoldenGate Microservices Architecture documentation.

For a multitenant source database, create a database connection for each PDB used by a per-PDB Extract. If a CDB-level integrated Extract is configured, also create a database connection for the CDB. For a multitenant target database, create a database connection for the target PDB.

Task 10: Configure Oracle GoldenGate Processes

When creating Extract, Distribution Path, and Replicat processes in Oracle GoldenGate Microservices Architecture, all deployment-specific configuration and runtime files that must be shared across Oracle RAC nodes are stored in the Oracle GoldenGate deployment directory on the shared file system (DBFS or ACFS).

The following are selected Oracle MAA-recommended configuration settings for Extract, Distribution Path, and Replicat processes running with Oracle GoldenGate Microservices Architecture on Oracle RAC.

Extract Configuration

When configuring an Extract in an Oracle RAC environment, follow these Oracle MAA best practice recommendations:

  1. When creating an Extract using the Oracle GoldenGate Administration Service, leave the Trail Subdirectory field blank so that the trail files are created in the default trail location within the GoldenGate deployment directory on the shared file system. The default trail location is <deployment directory>/var/lib/data.
  2. If DBFS is used for shared storage and the deployment var/temp directory was moved to local storage as described in Task 6: Create the Oracle GoldenGate Deployment, use the Extract CACHEMGR CACHEDIRECTORY parameter to place the temporary cache files on shared storage.

Create a new directory under the DBFS deployment mount point. For example:

$ mkdir -p /mnt/dbfs/goldengate/deployments/ggnorth/temp_cache

Set the Extract parameter to the new directory:

CACHEMGR CACHEDIRECTORY /mnt/dbfs/goldengate/deployments/ggnorth/temp_cache

For complete instructions for creating and configuring an Extract using the Administration Service, see ⁠Add a Primary Extract in the Oracle GoldenGate Microservices Architecture documentation.

Distribution Path Configuration

When using Oracle GoldenGate distribution path with the NGINX reverse proxy, additional configuration is required to establish secure connections between the source and target deployments.

Configuration recommendations:

  1. Determine the target deployment inbound authentication mode. When the target uses the default clientOptional_server mode and the Distribution Path uses UserID Alias authentication, a client certificate is not required. When the target uses client_server mode, configure a client certificate for the source deployment, add it to the source Service Manager, and configure the matching certificate identity as an x-Cert user with the Operator role in the target deployment.
  2. Ensure that the target NGINX endpoint presents the required server and intermediate certificate chain. Import the issuing root CA certificate as a trusted certificate in the source deployment Service Manager.
  3. Create a user with the Operator role in the target deployment for the Distribution Path connection.
  4. Create a credential in the source deployment using the target deployment user created in the previous step. For example, use a credential domain of GGNORTH_to_GGSOUTH and an alias of PathReceiver.

When configuring the Distribution Path, use the following Oracle MAA best practice recommendations:

  1. For the Generated Source URI, specify localhost as the server name. This allows the Distribution Path to start on any eligible Oracle RAC node.
  2. Set the Target Authentication Method to UserID Alias and the Target Transfer Protocol to wss (secure WebSocket). Set the Target Host to the target host name or application VIP used to connect to the target NGINX reverse proxy, and set the Port Number to the NGINX listener port (typically 443). The Target Host name must match a name included in the CA-signed certificate presented by NGINX.
  3. Set the Domain and Alias to the credential domain and credential alias created for the target deployment connection. For example, use the domain GGNORTH_to_GGSOUTH and alias PathReceiver.
  4. Configure the Distribution Path to automatically restart when the Distribution Service starts so that manual intervention is not required following an Oracle RAC node relocation.
    1. Set Retries to 10 and Delay to 1 minute.

For complete information about creating and configuring a Distribution Path, see ⁠Add a Distribution Path in the Oracle GoldenGate Microservices Architecture Documentation.

Replicat Configuration

When configuring a Replicat in an Oracle RAC environment, follow these Oracle MAA best practice recommendations:

Replicat mode selection

For the supported feature differences, see Select a Replicat Type for the Deployment in the Oracle GoldenGate documentation.

  1. A checkpoint table is required for Oracle GoldenGate Replicat processes. Ensure that a checkpoint table is created in the GoldenGate administrator (e.g. GGADMIN) schema.
    1. A checkpoint table can be created from the Administration Service using the target database connection on the DB Connections page.
    2. For more information about creating a checkpoint table, see ⁠Add a Checkpoint Table in the Oracle GoldenGate Microservices Architecture Documentation.
  2. When creating a Replicat using the Administration Service, set the Trail Subdirectory to the location where the Distribution Path or local Extract creates the trail files.
    1. Select the previously created checkpoint table from the Checkpoint Table list.

Task 11: Configure Autostart of Extract and Replicat Processes

Configure the Extract and Replicat processes to automatically start when the Oracle GoldenGate Administration Service starts and to automatically restart if a process abends. In Oracle GoldenGate Microservices Architecture, automatic start and restart behavior is managed using Managed Process Profiles.

Using the Oracle GoldenGate Administration Service, create a Managed Process Profile that can be assigned to the Extract and Replicat processes.

Profile Configuration Option Recommended Setting
Default Profile Enabled
Auto Start Enabled
Startup Delay 1 minute
Auto Restart Enabled
Max Retries 5
Retry Delay 30 seconds
Retries Window 30 minutes
Restart on Failure only Enabled
Disable Task After Retries Exhausted Enabled

After the profile is created and set as the default profile, it is assigned to new GoldenGate processes. Assign the profile to any existing Extract and Replicat processes as applicable.

For detailed information about configuring Managed Options and assigning Managed Process Profiles during Extract and Replicat creation, refer to the Oracle GoldenGate Microservices Architecture documentation for Add an Extract and Add a Replicat.

Note: When using Oracle GoldenGate Microservices Architecture with XAG, do not enable Critical to deployment health for Extract or Replicat processes. A failure of a process marked as critical can affect the health of the entire GoldenGate deployment and can interfere with XAG restarting the deployment.

Troubleshooting Oracle GoldenGate on Oracle RAC and Exadata Database Service

When a deployment does not start or relocate as expected, review evidence in this order:

Symptom First evidence Required checks
XAG start or relocation fails /log//agctl_goldengate_grid.trc and _agent_goldengate.trc XAG environment, Service Manager credentials, application VIP, file-system resource, database-service dependencies
DBFS does not mount /diag/crs//crs/trace/crsd_scriptagent_oracle.trc Database service, DBFS alias and user, mount point, approved secret storage, mount options, resource dependencies
GoldenGate service does not start /var/log Listener binding, shared paths, permissions, stale processes, file locks
Extract or Replicat abends /var/lib/report and deployment event logs Trail and checkpoint access, cache path, database connection, process parameters and Managed Process Profile
Relocation leaves lock errors Service/process logs plus DBFS mount state Confirm the process is absent on all nodes, stop cleanly, unmount as required, and verify current DBFS locking requirements before restart

For DBFS mount failures, correlate timestamps across the XAG log, CRS trace, and GoldenGate deployment logs. Validate the database-service dependency and every environment-specific value used by the DBFS resource. For lock errors after relocation, verify that no GoldenGate process remains active on another node before taking corrective action.