On-Premises: Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices

Summary of Recommendations when Deploying Oracle GoldenGate on Oracle RAC

When configuring Oracle GoldenGate in an Oracle RAC environment, follow these recommendations.

Install the latest version of Oracle GoldenGate software locally on each Oracle RAC node, making sure that the software location is the same on all Oracle RAC nodes.
Use the Oracle Database File System (DBFS) or Oracle Advanced Cluster File System (ACFS) for the file system where the Oracle GoldenGate files are stored (trail, checkpoint, temporary, report, and parameter files).
Use the same DBFS or ACFS mount point on all of the Oracle RAC nodes that may run Oracle GoldenGate.
When creating the GoldenGate deployment, specify either DBFS or ACFS for the deployment location.
Install Grid Infrastructure agent (XAG) version 10 or later on all Oracle RAC nodes that will run Oracle GoldenGate.
Configure the GoldenGate processes to automatically start and restart when the deployment is started.

Task 1: Configure the Oracle Database for Oracle GoldenGate

The source and target Oracle GoldenGate databases should be configured using the following recommendations.

Enable Oracle GoldenGate replication by setting the database initialization parameter ENABLE_GOLDENGATE_REPLICATION=TRUE.
Run the Oracle GoldenGate source database in ARCHIVELOG mode.
Enable FORCE LOGGING mode in the Oracle GoldenGate source database.
Enable minimal supplemental logging in the source database. Additionally, add schema or table level logging for all replicated objects.
If the Replicat process will be used, configure the streams pool in the System Global Area (SGA) on the source database using the STREAMS_POOL_SIZE initialization parameter.

Note that the streams pool is only needed on the target database if integrated Replicat will be used.

Use the following equation to determine the value for STREAMS_POOL_SIZE:

STREAMS_POOL_SIZE = (#Extracts and #Integrated Replicats * 1GB) * 1.25

For example, in a database with 2 Extracts and 2 integrated Replicats:

STREAMS_POOL_SIZE = 4GB * 1.25 = 5GB

When adding Extract or Replicat processes, it is important to recalculate and configure the new streams pool size requirement.

For more information about preparing the database for Oracle GoldenGate, see Preparing the Database for Oracle GoldenGate.

Task 2: Create the Database Replication Administrator User

The source and target Oracle databases need a GoldenGate Administrator user with appropriate privileges assigned.

For single tenant (non-CDB architecture) databases, see Establishing Oracle GoldenGate Credentials

For a multitenant source database, GoldenGate Extract must be configured to connect to a user in the root container database, using a c## account. For a multitenant target database, a separate GoldenGate administrator user is needed for each PDB that a Replicat applies data to.

For more details about creating a GoldenGate Administrator in an Oracle Multitenant Database, see Configuring Oracle GoldenGate in a Multitenant Container Database

Task 3: Create the Database Services

A database service is required so that the Oracle Grid Infrastructure Agent automatically starts the GoldenGate deployment when the database is opened. When DBFS is used for the shared file system, the database service is also used to mount DBFS to the correct Oracle RAC instance.

When using a source multitenant database, a separate service is required for the root container database (CDB) and the pluggable database (PDB) that contains the schema being replicated. For a target multitenant database, a single service is required for the PDB.

Create the service using the following command, as the oracle user.

$ srvctl add service -db db_name -service service_name
 -preferred instance_1 -available instance_2, instance_3 etc.
 -pdb PDB_name

For example:

$ srvctl add service -db ggdb -service oggserv_pdb -preferred ggdb1
    -available ggdb2 –pdb GGPDB01

It you are not using Oracle Multitenant Database, omit the -pdb parameter.

Task 4: Set Up a File System on Oracle RAC

Oracle GoldenGate Microservices Architecture is designed with a simplified installation and deployment directory structure. The installation directory should be placed on local storage on each Oracle RAC node to provide minimized downtime during software patching.

The deployment directory, which is created during deployment creation using the Oracle GoldenGate Configuration Assistant (oggca.sh), must be placed on a shared file system. The deployment directory contains configuration, security, log, parameter, trail, and checkpoint files.

Placing the deployment in DBFS or ACFS provides the best recovery and failover capabilities in the event of a system failure. Ensuring the availability of the checkpoint files cluster-wide is essential so that after a failure occurs the GoldenGate processes can continue running from their last known position.

If Oracle GoldenGate will be configured along with Oracle Data Guard, the recommended file system is DBFS. DBFS is contained in the database protected by Data Guard, and can be fully integrated with XAG. In the event of a Data Guard role transition, the file system can be automatically mounted on the new primary server, followed by automated start-up of Oracle GoldenGate. This is currently not possible with ACFS, because it is not part of the Oracle Data Guard configuration.

Follow the instructions in the appropriate section below to configure the file system for either DBFS or ACFS.

Oracle Database File System (DBFS)

It is required that you create the DBFS tablespace inside the same database that the Oracle GoldenGate processes are connected to. For example, if a GoldenGate integrated Extract process is extracting from a database called GGDB, the DBFS tablespace would be located in the same GGDB database.

Follow instructions in My Oracle Support note 869822.1 to install the required FUSE libraries if they are not already installed.

Use the instructions in My Oracle Support note 1054431.1 to configure the database, tablespace, database user, tnsnames.ora Oracle Net connection alias, and permissions on source or target GoldenGate environments required for DBFS.

Note:

When using an Oracle Multitenant Database, the DBFS tablespace MUST be created in a Pluggable Database (PDB). It is recommended that you use the same PDB that the GoldenGate Extract or Replicat processes are connecting to, allowing DBFS to use the same database service, created above in Task 2, for its database dependency.

When you create a file system for storing the GoldenGate deployment files, it is recommended that you allocate enough trail file disk space to permit storage of up to 12 hours of trail files. Doing this provides sufficient space for trail file generation should a problem occur with the target environment that prevents it from receiving new trail files. The amount of space needed for 12 hours can only be determined by testing trail file generation rates with real production data.

Example DBFS creation:

$ cd $ORACLE_HOME/rdbms/admin
$ sqlplus dbfs_user/dbfs_password@database_tns_alias
SQL> start dbfs_create_filesystem dbfs_gg_tbs goldengate

Follow the instructions in My Oracle Support note 1054431.1 to configure the newly created DBFS file system so that the DBFS instance and mount point resources are automatically started by Cluster Ready Services (CRS) after a node failure, with the following DBFS configuration and script file modifications.

Change the mount-dbfs.conf parameters to reflect your database environment.

Modify the MOUNT_OPTIONS parameter to the following:
```
MOUNT_OPTIONS=allow_other,direct_io,failover,nolock
```
The failover option forces all file writes to be committed to the DBFS database in an IMMEDIATE WAIT mode. This prevents data getting lost when it has been written into the dbfs_client cache but not yet written to the database at the time of a database or node failure.

The nolock mount option is required if you are using Oracle Database 18c or a later release, due to a change in the DBFS file locking which can cause issues for GoldenGate processes after an Oracle RAC node failure when a file is currently locked.

If you are using a dbfs_client from Oracle Database 12c Release 2 (12.2), make sure you have applied the latest release update that includes the fix for bug 27056711. Once the fix has been applied, the MOUNT_OPTIONS should also include the nolock option.
Modify the mount-dbfs.sh script to force unmounting of DBFS when the CRS resource is stopped.

Change two occurrences of:

$FUSERMOUNT -u $MOUNT_POINT

To the following:

$FUSERMOUNT -uz $MOUNT_POINT
When registering the resource with Oracle Clusterware, be sure to create it as a cluster_resource instead of a local_resource, as specified in the My Oracle Support note.

The reason for using cluster_resource is so that the file system can only be mounted on a single node at a time, preventing mounting of DBFS from concurrent nodes, which creates the potential for concurrent file writes, causing file corruption problems.

Make sure to use the database service name created in a previous step for the DBFS service dependency.

For example:
```
DBNAME=ggdb
DEPNAME=ora.$DBNAME.oggserv.svc

crsctl add resource $RESNAME \
  -type cluster_resource \
  -attr "ACTION_SCRIPT=$ACTION_SCRIPT, \
         CHECK_INTERVAL=30,RESTART_ATTEMPTS=10, \      
START_DEPENDENCIES='hard($DEPNAME)pullup($DEPNAME)',\
         STOP_DEPENDENCIES='hard($DEPNAME)',\
         SCRIPT_TIMEOUT=300"
```
Once the DBFS resource has been created, the file system should be mounted and tested.
```
$ crsctl start res dbfs_mount
$ crsctl stat res dbfs_mount
```
After the file system is mounted, create the directory for storing the GoldenGate files.
```
$ cd /mnt/dbfs/goldengate
$ mkdir deployments
```
Note:
Leave the shared file system mounted. It is required for creating the GoldenGate deployment in a later step.

Oracle Advanced Cluster File System (ACFS)

Oracle ACFS is an alternative to DBFS for the shared GoldenGate files in an Oracle RAC configuration.

Refer to My Oracle Support note 1929629.1 for more information about ACFS configuration requirements for Oracle Exadata Database Machine.

Create a single ACFS file system for storing the Oracle deployment files.

It is recommended that you allocate enough trail file disk space to permit storage of up to 12 hours of trail files. Doing this provides sufficient space for trail file generation should a problem occur with the target environment that prevents it from receiving new trail files. The amount of space needed for 12 hours can only be determined by testing trail file generation rates with real production data.

Create the file system using ASMCMD as the Oracle ASM administrator user.

ASMCMD [+] > volcreate -G datac1 -s 1200G ACFS_GG

Note:

Modify the file system size according to the determined size requirements.

ASMCMD> volinfo -G datac1 acfs_gg

Diskgroup Name: DATAC1
         Volume Name: ACFS_GG
         Volume Device: /dev/asm/acfs_gg-151
         State: ENABLED
         Size (MB): 1228800
         Resize Unit (MB): 64
         Redundancy: MIRROR
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage:
         Mountpath:

Make the file system with the following mkfs command.

$ /sbin/mkfs -t acfs /dev/asm/acfs-gg-151

Create the CRS resource for the newly created ACFS file system, if not already created.

Check to see if the file system resource was already created.
```
$ srvctl status filesystem -volume ACFS_GG -diskgroup DATAC1

ACFS file system /mnt/acfs_gg is mounted on nodes oggadm07, oggadm08
```
If not already created, create the ACFS mount point on all of the Oracle RAC nodes.
```
# mkdir -p /mnt/acfs_gg
```
Create the file system resource as the root user. Due to the implementation of distributed file locking on ACFS, unlike DBFS, it is acceptable to mount ACFS on more than one RAC node at any one time.

Create the ACFS resource using srvctl from the Oracle Grid Infrastructure ORACLE_HOME.
```
# srvctl add filesystem -device /dev/asm/acfs_gg-151 -volume ACFS_GG
-diskgroup DATAC1 -path /mnt/acfs_gg -user oracle -autostart RESTORE
```
To verify the currently configured ACFS file systems, use the following command to view the file system details.
```
$ srvctl config filesystem

Volume device: /dev/asm/acfs_gg-151
Diskgroup name: datac1
Volume name: ACFS_GG
Canonical volume device: /dev/asm/acfs_gg-151
Accelerator volume devices:
Mountpoint path: /mnt/acfs_gg
Mount point owner: oracle

Check the status of the ACFS resource and mount it.

$ srvctl status filesystem -volume ACFS_GG -diskgroup DATAC1

ACFS file system /mnt/acfs is not mounted

$ srvctl start filesystem -volume ACFS_GG -diskgroup DATAC1 –node dc1north01
```
The CRS resource that is created is named using the format ora.diskgroup_name.volume_name.acfs. Using the above file system example, the CRS resource is called ora.datac1.acfs_gg.acfs.

To see all ACFS file system CRS resources that currently exist, use the following command.
```
$ crsctl stat res -w "((TYPE = ora.acfs.type) OR (TYPE = ora.acfs_cluster.type))"

NAME=ora.datac1.acfs_gg.acfs
TYPE=ora.acfs.type
TARGET=ONLINE              , OFFLINE
STATE=ONLINE on dc1north01, OFFLINE
```
Create a GoldenGate deployment directory on ACFS.

After the file system is mounted, create the directory for storing the GoldenGate deployments.
```
$ cd /mnt/acfs_gg
$ mkdir deployments
```

Note:

Leave the shared file system mounted. It is required for creating the GoldenGate deployment in a later Task.

Task 5: Install Oracle GoldenGate

Download and install the Oracle GoldenGate 21c Microservices software, or later release.

Download the software at https://www.oracle.com/middleware/technologies/goldengate-downloads.html.

Install the Oracle GoldenGate software locally on all nodes in the Oracle RAC configuration that will be part of the GoldenGate configuration. Make sure the installation directory is identical on all nodes.

Follow the generic installation instructions detailed in Oracle GoldenGate Microservices Documentation.

Task 6: Create the Oracle GoldenGate Deployment

Once the Oracle GoldenGate software has been installed, the next step is to create a deployment using the Oracle GoldenGate Configuration Assistant (oggca).

There are two limitations that currently exist with Oracle GoldenGate and XAG:

A Service Manager that is registered with XAG can only manage a single deployment. If multiple deployments are required, each deployment must use their own Service Manager. Oracle GoldenGate release 21c simplifies this requirement because it uses a single deployment to support Extract and Replicat processes connecting to different versions of the Oracle Database.
Each Service Manager registered with XAG must belong to separate OGG_HOME software installation directories. Instead of installing Oracle GoldenGate multiple times, the recommended approach is to install Oracle GoldenGate one time, and then create a symbolic link for each Service Manager OGG_HOME.

For example:

$ echo $OGG_HOME
/u01/oracle/goldengate/gg21c_MS

$ ln –s /u01/oracle/goldengate/gg21c_MS /u01/oracle/goldengate/gg21c_MS_ggnorth

$ export OGG_HOME=/u01/oracle/goldengate/gg21c_MS_ggnorth
$ $OGG_HOME/bin/oggca.sh

The symbolic link and OGG_HOME environment variable must be configured before running the Oracle GoldenGate Configuration Assistant on all Oracle RAC nodes.

Recommendations for creating the GoldenGate deployment in the Oracle GoldenGate Configuration Assistant are as follows.

In Service Manager Options, specify the following for the creation of a new Service Manager.
1. In the Service Manager Details pane, select Create New Service Manager.
2. Enter the Service Manager Deployment Home location on the shared DBFS or ACFS file system.
3. Select to Integrate with XAG.
4. In the Service Manager Connection Details pane, specify localhost in the Listening hostname/address field.
  
  Using localhost allows the deployment to be started on all of the Oracle RAC nodes without the need for a Virtual IP address (VIP).
5. Enter the port number in Listening port.
In Deployment Directories, specify the Deployment home directory on the shared DBFS or ACFS file system.
In Environment Variables, specify a correct TNS_ADMIN directory.

Starting with Oracle GoldenGate release 21.3, a database ORACLE_HOME is no longer required because the required database libraries are installed as part of the Oracle GoldenGate installation. It is recommended that you use TNS_ADMIN directory outside of any existing ORACLE_HOME directories.
In Security Options, do not select SSL/TLS Security.

External access to the Oracle GoldenGate Microservices server is achieved by using NGINX Reverse Proxy SSL-termination. Secure access and communication to the GoldenGate deployments will be exclusively through the SSL port 443. Internal connectivity within the same local host between NGINX and GoldenGate does not require SSL.
In Port Settings, if the Management Pack for Oracle GoldenGate has been licensed, select Enable Monitoring to use the performance metric server using either Berkeley Database (BDB) or Lightening Memory Database (LMDB).

For both BDB and LMDB Metrics Service DataStore types, set the Metrics Service DataStore home directory to a local directory that exists on all Oracle RAC nodes. For example: /u01/oracle/goldengate/datastores/deployment name
Continue through the Oracle GoldenGate Configuration Assistant until the deployment is created.

After the deployment has been created, if you are using DBFS for the shared file system and the database version is a release earlier than Oracle Database Release 21c (21.3), run the following commands to move the Oracle GoldenGate deployment temp directory from DBFS to local storage.

On the first node:

$ cd <DBFS GoldenGate deployment home directory/var
$ mkdir -p local_storage_directory/deployment_name
$ mv temp local_storage_directory/deployment_name
$ ln -s local_storage_directory/deployment_name/temp temp

On all other nodes:

$ mkdir local_storage_directory/deployment_name/temp

First node example:

$ cd /mnt/dbfs/goldengate/deployments/ggnorth/var
$ mkdir –p /u01/oracle/goldengate/deployments/ggnorth
$ mv temp /u01/oracle/goldengate/deployments/ggnorth
$ ln -s /u01/oracle/goldengate/deployments/ggnorth/temp temp

On all other nodes:

$ mkdir /u01/oracle/goldengate/deployments/ggnorth/temp

Task 7: Oracle Clusterware Configuration

The following procedure shows you how to configure Oracle Clusterware to manage Oracle GoldenGate using the Oracle Grid Infrastructure Standalone Agent (XAG).

Using XAG automates the mounting of the shared file system (DBFS or ACFS) and the stopping and starting of the GoldenGate deployment when relocating between Oracle RAC nodes.

Install the Oracle Grid Infrastructure Standalone Agent.

It is recommended that you install the XAG software as a standalone agent outside of the Grid Infrastructure ORACLE_HOME. This allows you to use the latest XAG release available, and the software can be updated without impact to the Grid Infrastructure.

When using Oracle GoldenGate Microservices Architecture you MUST use XAG version 10.2 or later.

The latest agent software is available for download from the following location:

http://www.oracle.com/technetwork/database/database-technologies/clusterware/downloads/xag-agents-downloads-3636484.html

Install the XAG standalone agent outside of the Oracle Grid Infrastructure home directory. XAG must be installed in the same directory on all RAC nodes in the cluster where Oracle GoldenGate is installed.

For example, as the Oracle Grid Infrastructure user, the default of oracle:
```
$ ./xagsetup.sh --install --directory /u01/oracle/xag --all_nodes
```
Add the location of the newly installed XAG software to the PATH variable so that the location of agctl is known when the oracle user logs on to the machine.
```
$ cat .bashrc
export PATH=/u01/oracle/xag/bin:$PATH
```
Note:
It is important to make sure that the XAG bin directory is specified BEFORE the Grid Infrastructure bin directory, to ensure the correct agctl binary is found. Set this location in the oracle user environment to take effect at time of logging on, such as in the .bashrc file when the Bash shell is in use.
Prepare for Application Virtual IP Address (VIP) Creation.

A dedicated application VIP is required to allow access to the GoldenGate Microservices using the same host name, regardless of which Oracle RAC node is hosting the services. An application VIP also ensures that the GoldenGate Distribution Server can communicate with the Distribution Receiver running the current Oracle RAC node.

The VIP is a cluster resource that Oracle Clusterware manages. The VIP is assigned to a cluster node and is automatically migrated to another node in the event of a node failure.

There are two pieces of information needed before creating the application VIP:
- The network number, which can be identified using the following command.
```
$ crsctl status resource -p -attr ADDRESS_TYPE,NAME,USR_ORA_SUBNET -w "TYPE = ora.network.type" |sort | uniq

ADDRESS_TYPE=IPV4
NAME=ora.net1.network

USR_ORA_SUBNET=10.133.16.0
```
  The net1 in NAME=ora.net1.network indicates that this is network 1, and it is of type IPV4.
- The IP address for the new Application VIP, provided by your system administrator. This IP address must be in the same subnet of the cluster environment as determined above.
  
  The VIP will be created in the next step, when configuring the Oracle Grid Infrastructure Agent.
Configure Oracle Grid Infrastructure Agent (XAG).

Oracle GoldenGate must be registered with XAG so that the deployment is started and stopped automatically when the database is started and the file system is mounted.

To register Oracle GoldenGate Microservices Architecture with XAG use the following command format.
```
agctl add goldengate instance_name
  --gg_home GoldenGate_Home
  --service_manager
  --config_home GoldenGate_SvcMgr_Config
  --var_home GoldenGate_SvcMgr_Var_Dir
  --port port_number
  --oracle_home $OGG_HOME/lib/instantclient
  --adminuser OGG_admin_user
  --user GG_instance_user
  --group GG_instance_group
  --network network_number
  --ip ip_address
  --vip_name vip_name
  --filesystems CRS_resource_name
  --db_services service_name
  --use_local_services
  --nodes node1, node2, ... ,nodeN
```
Where:

--gg_home specifies the location of the Oracle GoldenGate software. Specify the OGG_HOME symbolic link for the OGG_HOME if registering multiple Service Managers (see Task 6: Create the Oracle GoldenGate Deployment).

--service_manager indicates this is a GoldenGate Microservices instance.

--config_home specifies the GoldenGate Service Manager deployment configuration home directory.

--var_home specifies the GoldenGate Service Manager deployment variable home directory.

--port specifies the deployment Service Manager port number.

--oracle_home specifies the location of the Oracle database libraries that are included as part of Oracle GoldenGate 21c and later releases. Example: $OGG_HOME/lib/instantclient

--adminuser specifies the Oracle GoldenGate Microservices administrator account name.

--user specifies the name of the operating system user that owns the GoldenGate deployment.

--group specifies the name of the operating system group that owns the GoldenGate deployment.

--network specifies the network subnet for the VIP, determined above.

--ip specifies the IP address for the VIP, which was determined above. If you have already created a VIP, then specify it using the --vip_name parameter in place of --network and --ip.

--vip_name specifies a CRS resource name for an application VIP that has previously been created. This parameter replaces --network and -–ip (optional).

--filesystems specifies the DBFS or ACFS CRS file system resource that must be mounted before the deployment is started.

--db_services specifies the ora.database.service_name.svc service name that was created in the previous step. If using Oracle Multitenant Database, specify the PDB database service for Replicat, or the CDB database service for an Extract. If using both Replicat and Extract, specify both services names, separated by a comma.

--use_local_services specifies that the GoldenGate instance must be co-located on the same Oracle RAC node where the db_services service is running.

--nodes specifies which of the Oracle RAC nodes this GoldenGate instance can run on. If GoldenGate is configured to run on any of the Oracle RAC nodes in the cluster, this parameter should still be used to determine the preferred order of nodes to run Oracle GoldenGate.

Notes:

The GoldenGate instance registration with XAG MUST be run as the root user.
The user and group parameters are mandatory because the GoldenGate registration with XAG is run as the root user.

Below are some examples of registering Oracle GoldenGate with XAG.

Example 1: Oracle RAC cluster using DBFS, using an already created application VIP

# agctl add goldengate GGNORTH \
--gg_home /u01/oracle/goldengate/gg21c_MS \
--service_manager \
--config_home /mnt/dbfs/goldengate/deployments/ggsm01/etc/conf \
--var_home /mnt/dbfs/goldengate/deployments/ggsm01/var \
--port 9100 \
--oracle_home /u01/oracle/goldengate/gg21c_MS/lib/instantclient
--adminuser oggadmin
--user oracle \
--group oinstall \
--vip_name gg_vip_prmy \
--filesystems dbfs_mount \
--db_services ora.ds19c.oggserv.svc \
--use_local_services \
--nodes dc1north01,dc1north02

Where:

GoldenGate instance is GGNORTH
GoldenGate home directory is /u01/oracle/goldengate/gg21c_MS
This is an Oracle GoldenGate Microservices Architecture instance (--service_manager)
GoldenGate deployment configuration home directory is /mnt/dbfs/goldengate/deployments/ggsm01/etc/conf
GoldenGate deployment variable home directory is /mnt/dbfs/goldengate/deployments/ggsm01/var
Deployment Service Manager port number is 9100
Oracle GoldenGate Microservices administrator account name is oggadmin
The GoldenGate user is oracle in the group oinstall
Application VIP name, managed by CRS, is called gg_vip_prmy
The CRS resource name for the file system the deployment depends on is dbfs_mount
The GoldenGate instance will be started on the same Oracle RAC node as the CRS service called ora.ds19c.oraserv.svc will be co-located on the same node as this GoldenGate instance.

Example 2: Oracle RAC cluster, using ACFS, with an application VIP running on a subset of the nodes in the cluster.

# agctl add goldengate GGNORTH \
--gg_home /u01/oracle/goldengate/gg21c_MS \
--service_manager \
--config_home /mnt/acfs/goldengate/deployments/ggsm01/etc/conf \
--var_home /mnt/acfs/goldengate/deployments/ggsm01/var \
--port 9100 \
--oracle_home /u01/oracle/goldengate/gg21c_MS/lib/instantclient
--adminuser admin \
--user oracle \
--group oinstall \
--network 1 --ip 10.13.11.203 \
--filesystems ora.datac1.acfs_gg.acfs \
--db_services ora.ds19c.oraserv.svc \
--use_local_services \
--nodes dc1north01,dc1north02

Where:

GoldenGate instance is GGNORTH
GoldenGate home directory is /u01/oracle/goldengate/gg21c_MS
This is an Oracle GoldenGate Microservices Architecture instance (--service_manager)
GoldenGate deployment configuration home directory is /mnt/acfs/goldengate/deployments/ggsm02/etc/conf
GoldenGate deployment variable home directory is /mnt/acfs/goldengate/deployments/ggsm02/var
Deployment Service Manager port number is 9100
Oracle GoldenGate Microservices administrator account name is admin
GoldenGate user is oracle in the group oinstall
The network is the default ora.net1.network and the VIP is 10.13.11.203
The CRS resource name for the file system the deployment depends on is ora.datac1.acfs_gg.acfs
This GoldenGate instance will be started on the same Oracle RAC node as the CRS service called ora.ds19c.oraserv.svc will be co-located on the same node as this GoldenGate instance
Oracle GoldenGate will only run on Oracle RAC nodes dc1north01 and dc1north02, listed in priority order.

Example AGCTL Commands

Below are some example agctl commands that are used to manage the Oracle GoldenGate deployment with XAG.

To check the status of Oracle GoldenGate:

% agctl status goldengate
Goldengate  instance 'GGNORTH' is running on dc1north01

To start the GoldenGate deployment, and all Extract/Replicat processes that have been configured to autostart (instructions in a later step):

% agctl start goldengate GGNORTH --node dc1north02

To stop the GoldenGate deployment:

% agctl stop goldengate GGNORTH

To manually relocate the GoldenGate deployment to another node:

% agctl relocate goldengate GGNORTH --node dc1north02

To view the configuration parameters for the GoldenGate resource:

% agctl config goldengate GGNORTH

Instance name: GGNORTH
Application GoldenGate location is: /u01/oracle/goldengate/gg21c_MS
Goldengate MicroServices Architecture environment: yes
Goldengate Service Manager configuration directory: /mnt/dbfs/goldengate/deployments/ggsm01/etc/conf

Goldengate Service Manager var directory: /mnt/dbfs/goldengate/deployments/ggsm01/var

Service Manager Port: 9100
Goldengate Administration User: oggadmin
Autostart on DataGuard role transition to PRIMARY: no
Configured to run on Nodes: dc1north01 dc1north02
ORACLE_HOME location is: /u01/oracle/goldengate/gg21c_MS/lib/instantclient
Database Services needed: ora.cdb1.oggcdb.svc [use_local_services]
File System resources needed: ora.datac1.acfs_gg.acfs
Network: 1, IP: 10.13.11.203, User:oracle, Group:oinstall

To delete the GoldenGate XAG resource:

$ agctl stop goldengate GGNORTH
# agctl remove goldengate GGNORTH

For more information about the Oracle Grid Infrastructure Bundled Agent see Oracle Grid Infrastructure Standalone Agents for Oracle Clusterware 11g Rel. 2, 12c, 18c and 19c.

Task 8: Configure NGINX Reverse Proxy

Follow the instructions provided in My Oracle Support note 2826001.1 to install and configure NGINX Reverse Proxy with SSL connection, and to ensure all external communication is completely secure.

Note:

When using CA Signed Certificates with NGINX, make sure the NGINX ssl_certificate parameter points to a certificate file that contains the certificates in the correct order of CA signed certificate, intermediate certificate and root certificate.

Oracle Clusterware needs to have control over starting the NGINX reverse proxy so that it can be started automatically before the GoldenGate deployments are started.

The NGINX resource is created with a dependency on the underlying network CRS resource, the name of which can be determined using the following command:

$ $GRID_HOME/bin/crsctl stat res -w "TYPE == ora.network.type"|grep NAME

NAME=ora.net1.network

As the root user, use the following example command to create a Clusterware resource to manage NGINX.

# $GRID_HOME/bin/crsctl add resource nginx -type generic_application -attr
 "ACL='owner:root:rwx,pgrp:root:rwx,other::r--,group:oinstall:r-x,user:oracle:rwx',
EXECUTABLE_NAMES=nginx,START_PROGRAM='/bin/systemctl
 start -f nginx',STOP_PROGRAM='/bin/systemctl
 stop -f nginx',CHECK_PROGRAMS='/bin/systemctl
 status nginx' ,START_DEPENDENCIES='hard(ora.net1.network)
 pullup(ora.net1.network)', STOP_DEPENDENCIES='hard(intermediate:ora.net1.network)',
 RESTART_ATTEMPTS=0, HOSTING_MEMBERS='dc1north01,dc1north02', CARDINALITY=2"

The NGINX resource created in this example run on the named cluster nodes at the same time, specified by HOSTING_MEMBERS. This is recommended when multiple GoldenGate Service Manager deployments are configured, and they can independently move between cluster nodes.

Once the NGINX Clusterware resource is created, alter the GoldenGate XAG resources so that NGINX must be started before the GoldenGate deployments are started.

As the oracle user, modify the XAG resources using the following example commands.

Determine the current --filesystems parameter:

$ agctl config goldengate SOURCE|grep "File System"

File System resources needed: ora.datac1.acfs_gg.acfs

Modify the --filesystems parameter:

$ agctl modify goldengate SOURCE --filesystems ora.datac1.acfs_gg.acfs,nginx

Repeat the above commands for each of the XAG GoldenGate registrations relying on NGINX.

Task 9: Create Oracle Net TNS Alias for Oracle GoldenGate Database Connections

Create a TNS alias on all of the Oracle RAC nodes where Oracle GoldenGate may be started to provide local database connections for the GoldenGate processes when switching between Oracle RAC nodes. Create the TNS alias in the tnsnames.ora file in the TNS_ADMIN directory specified in the deployment creation.

If the source database is a multitenant database, two TNS alias entries are required: one for the container database (CDB) and one for the pluggable database (PDB) that is being replicated. For a target multitenant database, the TNS alias connects the PDB where replicated data is being applied to. The pluggable database SERVICE_NAME should be set to the database service created in an earlier step (refer to Task 3: Create the Database Services).

Below are some example source database TNS alias definitions using the IPC protocol, which must be defined locally on all RAC nodes.

OGGSOURCE_CDB =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL=IPC)(KEY=LISTENER))
      (CONNECT_DATA =
        (SERVICE_NAME = oggserv_cdb)
       )
    )

OGGSOURCE_PDB =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL=IPC)(KEY=LISTENER))
      (CONNECT_DATA =

        (SERVICE_NAME = oggserv_pdb)
       )
    )

Note:

When the tnsnames.ora or sqlnet.ora, located in the TNS_ADMIN directory for the GoldenGate deployment, are modified, the deployment needs to be restarted in order to pick up the changes.

With the GoldenGate deployment created, use the Administration Server home page to create the database credentials using the above TNS alias names. See Figure 6 below for an example of the database credential creation using the TNS alias appended to the database user name in the ‘User ID” field.

If the source database is a multitenant database, create database credentials for the CDB and PDB. If the target database is a multitenant database, create a single credential for the PDB.

Task 10: Configure Oracle GoldenGate Processes

When creating Extract, Distribution Paths, and Replicat processes with Oracle GoldenGate Microservices Architecture, all files that need to be shared between Oracle RAC nodes are already shared with the deployment files stored on a shared file system (DBFS or ACFS).

Listed below are important configuration details that are recommended for running Oracle GoldenGate Microservices on Oracle RAC for Extract, Distribution Paths and Replicat processes.

Extract Configuration

When creating an Extract using the Oracle GoldenGate Administration Server GUI interface, leave the Trail SubDirectory parameter blank, so that the trail files are automatically created in the deployment directories stored on the shared file system.

The default location for trail files is the /<deployment directory>/var/lib/data
If you are using DBFS for shared storage, and the deployment var/temp directory was moved to local storage as described in Task 6: Create the Oracle GoldenGate Deployment, it is recommended that you use the Extract CACHEMGR parameter to place the temporary cache files on the shared storage.

Create a new directory under the DBFS deployment mount point. For example:

$ mkdir –p /mnt/dbfs/goldengate/deployments/ggnorth/temp_cache

Set the Extract parameter to the new directory:

CACHEMGR CACHEDIRECTORY /mnt/dbfs/goldengate/deployments/ggnorth/temp_cache

Shown below is an example of how the parameters specified for an integrated Extract with the Oracle GoldenGate Administration Server GUI looks in the UI.

Figure 24-1 Extract parameters for defining the temporary cache files

Distribution Path Configuration

When using Oracle GoldenGate distribution paths with the NGINX Reverse Proxy, there are additional steps that must be performed to ensure that the path server certificates are configured.

Follow the instructions provided in the following video to correctly configure the certificates: https://apexapps.oracle.com/pls/apex/f?p=44785:112:0::::P112_CONTENT_ID:31380

Configuration highlights presented in this video:

Create a client certificate for the source deployment and add the client certificate to the source deployment Service Manager. (This is not required when using Oracle GoldenGate 21c or later releases.)
Download the target deployment server’s root certificate and add the CA certificate to the source deployment Service Manager.
Create a user in the target deployment for the distribution path to connect to.
Create a credential in the source deployment connecting to the target deployment with the user created in the previous step.

For example, a domain of GGNORTH_to_GGSOUTH and an alias of PathReceiver.

After configuring the client and server certificates, the following configuration options need to be set. Refer to the figures below to see where these options are set in the UI.

Change the Generated Source URI specifying localhost for the server name.

This allows the distribution path to be started on any of the Oracle RAC nodes.
Set the Target Authentication Method to UserID Alias and the Target transfer protocol to wss (secure web socket).

Set the Target Host to the target host name/VIP that will be used for connecting to the target system along with the Port Number that NGINX was configured with (default is 443).

The target host name/VIP should match the common name in the CA signed certificate used by NGINX.
Set the Domain to the credential domain created above in step 4 and presented in the video, for example GGNORTH_to_GGSOUTH.

The Alias is set to the credential alias, also created in step 4 in the video.
Set the distribution path to automatically restart when the Distribution Server starts.

This is required so that manual intervention is not required after an Oracle RAC node relocation of the Distribution Server. It is recommended that you set the number of Retries to 10. Set the Delay, which is the amount of time in minutes to pause between restart attempts, to 1.

Figure 24-2 Distribution Path Creation steps 1-3

Figure 24-3 Distribution Path Creation step 4

Replicat Configuration

The checkpoint table is a required component for GoldenGate Replicat processes. Make sure that a checkpoint table has been created in the database GoldenGate administrator (GGADMIN) schema.

The checkpoint table can be created using the Oracle GoldenGate Administration Server GUI, clicking on the ‘+’ button and entering the checkpoint table name in the format of schema.tablename. This is shown in the image below

Figure 24-4 Creating the checkpoint table for Replicat processes

See About Checkpoint Table for more information about creating a checkpoint table.
When creating a Replicat using the Oracle GoldenGate Administration Server GUI interface, set the Trail SubDirectory parameter to the location where the distribution path or local Extract are creating the trail files.
If a checkpoint table was created previously, select the table name from the Checkpoint Table pulldown list.

Figure 24-5 Replicat creation with Trail SubDirectory and Checkpoint Table

Task 11: Configure Autostart of Extract and Replicat Processes

Configure the Extract and Replicat processes to automatically start when the Oracle GoldenGate Administration Server is started, and then to restart if any Extract or Replicat processes abend. With GoldenGate Microservices auto start and restart is managed by Profiles.

Using the Oracle GoldenGate Administration Server GUI, create a new profile which can be assigned to each of the Oracle GoldenGate processes.

Profile Configuration Option	Recommended Setting
Default Profile	Enabled
Auto Start	Enabled
Startup Delay	1 minute
Auto Restart	Enabled
Max Retries	5
Retry Delay	30 seconds
Retries Window	30 minutes
Restart on Failure only	Enabled
Disable Task After Retries Exhausted	Enabled

After the profile has been created, and set as the default profile, all new GoldenGate processes created are assigned this profile. For all existing processes, the profile must be assigned to each process.

In the Overview pane, on the Process Information tab, select the Profile Name under Managed Options.

Note:

When using Oracle GoldenGate Microservices with XAG, it is strongly recommended not to enable the ‘Critical to deployment health’ flag for any Extract or Replicat processes. Doing so can cause an entire GoldenGate deployment outage from a single Extract or Replicat failure, and also prevents XAG from being able to restart GoldenGate. Refer to Troubleshooting Oracle GoldenGate on Oracle RAC for an example of troubleshooting an outage caused by setting a Replicat to critical.