24 On-Premises: Oracle GoldenGate Microservices Architecture with Oracle Real Application Clusters Configuration Best Practices
Use these best practices for configuring Oracle GoldenGate Microservices Architecture for on-premises systems, including Oracle Exadata, to work with Oracle Real Application Clusters (RAC), Oracle Clusterware, and Oracle Database File System (DBFS) or Oracle Advanced Cluster File System (ACFS).
The target Oracle RAC system that hosts Oracle GoldenGate Microservices Architecture can act as the source database, as the target database, or in some cases as both source and target databases, for Oracle GoldenGate operations.
See the following topics:
- Summary of Recommendations when Deploying Oracle GoldenGate on Oracle RAC
- Task 1: Configure the Oracle Database for Oracle GoldenGate
- Task 2: Create the Database Replication Administrator User
- Task 3: Create the Database Services
- Task 4: Set Up a File System on Oracle RAC
- Task 5: Install Oracle GoldenGate
- Task 6: Create the Oracle GoldenGate Deployment
- Task 7: Oracle Clusterware Configuration
- Task 8: Configure NGINX Reverse Proxy
- Task 9: Create Oracle Net TNS Alias for Oracle GoldenGate Database Connections
- Task 10: Configure Oracle GoldenGate Processes
- Task 11: Configure Autostart of Extract and Replicat Processes
Summary of Recommendations when Deploying Oracle GoldenGate on Oracle RAC
When configuring Oracle GoldenGate in an Oracle RAC environment, follow these recommendations.
- Install the latest version of Oracle GoldenGate software locally on each Oracle RAC node, making sure that the software location is the same on all Oracle RAC nodes.
- Use the Oracle Database File System (DBFS) or Oracle Advanced Cluster File System (ACFS) for the file system where the Oracle GoldenGate files are stored (trail, checkpoint, temporary, report, and parameter files).
- Use the same DBFS or ACFS mount point on all of the Oracle RAC nodes that may run Oracle GoldenGate.
- When creating the GoldenGate deployment, specify either DBFS or ACFS for the deployment location.
- Install Grid Infrastructure agent (XAG) version 10 or later on all Oracle RAC nodes that will run Oracle GoldenGate.
- Configure the GoldenGate processes to automatically start and restart when the deployment is started.
Task 1: Configure the Oracle Database for Oracle GoldenGate
The source and target Oracle GoldenGate databases should be configured using the following recommendations.
-
Enable Oracle GoldenGate replication by setting the database initialization parameter
ENABLE_GOLDENGATE_REPLICATION=TRUE
. -
Run the Oracle GoldenGate source database in
ARCHIVELOG
mode. -
Enable
FORCE LOGGING
mode in the Oracle GoldenGate source database. -
Enable minimal supplemental logging in the source database. Additionally, add schema or table level logging for all replicated objects.
-
If the Replicat process will be used, configure the streams pool in the System Global Area (SGA) on the source database using the
STREAMS_POOL_SIZE
initialization parameter.Note that the streams pool is only needed on the target database if integrated Replicat will be used.
Use the following equation to determine the value for
STREAMS_POOL_SIZE
:STREAMS_POOL_SIZE
= (#Extracts and #Integrated Replicats * 1GB) * 1.25For example, in a database with 2 Extracts and 2 integrated Replicats:
STREAMS_POOL_SIZE
= 4GB * 1.25 = 5GBWhen adding Extract or Replicat processes, it is important to recalculate and configure the new streams pool size requirement.
For more information about preparing the database for Oracle GoldenGate, see Preparing the Database for Oracle GoldenGate.
Task 2: Create the Database Replication Administrator User
The source and target Oracle databases need a GoldenGate Administrator user with appropriate privileges assigned.
For single tenant (non-CDB architecture) databases, see Establishing Oracle GoldenGate Credentials
For a multitenant source database, GoldenGate Extract must be configured to connect
to a user in the root container database, using a c##
account. For
a multitenant target database, a separate GoldenGate administrator user is needed
for each PDB that a Replicat applies data to.
For more details about creating a GoldenGate Administrator in an Oracle Multitenant Database, see Configuring Oracle GoldenGate in a Multitenant Container Database
Task 3: Create the Database Services
A database service is required so that the Oracle Grid Infrastructure Agent automatically starts the GoldenGate deployment when the database is opened. When DBFS is used for the shared file system, the database service is also used to mount DBFS to the correct Oracle RAC instance.
When using a source multitenant database, a separate service is required for the root container database (CDB) and the pluggable database (PDB) that contains the schema being replicated. For a target multitenant database, a single service is required for the PDB.
Create the service using the following command, as the oracle
user.
$ srvctl add service -db db_name -service service_name
-preferred instance_1 -available instance_2, instance_3 etc.
-pdb PDB_name
For example:
$ srvctl add service -db ggdb -service oggserv_pdb -preferred ggdb1
-available ggdb2 –pdb GGPDB01
It you are not using
Oracle Multitenant Database, omit the -pdb
parameter.
Task 4: Set Up a File System on Oracle RAC
Oracle GoldenGate Microservices Architecture is designed with a simplified installation and deployment directory structure. The installation directory should be placed on local storage on each Oracle RAC node to provide minimized downtime during software patching.
The deployment directory, which is created during deployment creation using the Oracle GoldenGate Configuration Assistant (oggca.sh), must be placed on a shared file system. The deployment directory contains configuration, security, log, parameter, trail, and checkpoint files.
Placing the deployment in DBFS or ACFS provides the best recovery and failover capabilities in the event of a system failure. Ensuring the availability of the checkpoint files cluster-wide is essential so that after a failure occurs the GoldenGate processes can continue running from their last known position.
If Oracle GoldenGate will be configured along with Oracle Data Guard, the recommended file system is DBFS. DBFS is contained in the database protected by Data Guard, and can be fully integrated with XAG. In the event of a Data Guard role transition, the file system can be automatically mounted on the new primary server, followed by automated start-up of Oracle GoldenGate. This is currently not possible with ACFS, because it is not part of the Oracle Data Guard configuration.
Follow the instructions in the appropriate section below to configure the file system for either DBFS or ACFS.
Oracle Database File System (DBFS)
It is required that you create the DBFS tablespace inside the same database that the Oracle GoldenGate processes are connected to. For example, if a GoldenGate integrated Extract process is extracting from a database called GGDB, the DBFS tablespace would be located in the same GGDB database.
Follow instructions in My Oracle Support note 869822.1 to install the required FUSE libraries if they are not already installed.
Use the instructions in My Oracle Support note 1054431.1 to configure the
database, tablespace, database user, tnsnames.ora
Oracle Net
connection alias, and permissions on source or target GoldenGate environments
required for DBFS.
Note:
When using an Oracle Multitenant Database, the DBFS tablespace MUST be created in a Pluggable Database (PDB). It is recommended that you use the same PDB that the GoldenGate Extract or Replicat processes are connecting to, allowing DBFS to use the same database service, created above in Task 2, for its database dependency.
When you create a file system for storing the GoldenGate deployment files, it is recommended that you allocate enough trail file disk space to permit storage of up to 12 hours of trail files. Doing this provides sufficient space for trail file generation should a problem occur with the target environment that prevents it from receiving new trail files. The amount of space needed for 12 hours can only be determined by testing trail file generation rates with real production data.
Example DBFS creation:
$ cd $ORACLE_HOME/rdbms/admin
$ sqlplus dbfs_user/dbfs_password@database_tns_alias
SQL> start dbfs_create_filesystem dbfs_gg_tbs goldengate
Follow the instructions in My Oracle Support note 1054431.1 to configure the newly created DBFS file system so that the DBFS instance and mount point resources are automatically started by Cluster Ready Services (CRS) after a node failure, with the following DBFS configuration and script file modifications.
-
Change the
mount-dbfs.conf
parameters to reflect your database environment.Modify the
MOUNT_OPTIONS
parameter to the following:MOUNT_OPTIONS=allow_other,direct_io,failover,nolock
The
failover
option forces all file writes to be committed to the DBFS database in anIMMEDIATE WAIT
mode. This prevents data getting lost when it has been written into thedbfs_client
cache but not yet written to the database at the time of a database or node failure.The
nolock
mount option is required if you are using Oracle Database 18c or a later release, due to a change in the DBFS file locking which can cause issues for GoldenGate processes after an Oracle RAC node failure when a file is currently locked.If you are using a
dbfs_client
from Oracle Database 12c Release 2 (12.2), make sure you have applied the latest release update that includes the fix for bug 27056711. Once the fix has been applied, theMOUNT_OPTIONS
should also include thenolock
option. -
Modify the
mount-dbfs.sh
script to force unmounting of DBFS when the CRS resource is stopped.Change two occurrences of:
$FUSERMOUNT -u $MOUNT_POINT
To the following:
$FUSERMOUNT -uz $MOUNT_POINT
-
When registering the resource with Oracle Clusterware, be sure to create it as a
cluster_resource
instead of alocal_resource
, as specified in the My Oracle Support note.The reason for using
cluster_resource
is so that the file system can only be mounted on a single node at a time, preventing mounting of DBFS from concurrent nodes, which creates the potential for concurrent file writes, causing file corruption problems.Make sure to use the database service name created in a previous step for the DBFS service dependency.
For example:
DBNAME=ggdb DEPNAME=ora.$DBNAME.oggserv.svc crsctl add resource $RESNAME \ -type cluster_resource \ -attr "ACTION_SCRIPT=$ACTION_SCRIPT, \ CHECK_INTERVAL=30,RESTART_ATTEMPTS=10, \ START_DEPENDENCIES='hard($DEPNAME)pullup($DEPNAME)',\ STOP_DEPENDENCIES='hard($DEPNAME)',\ SCRIPT_TIMEOUT=300"
Once the DBFS resource has been created, the file system should be mounted and tested.
$ crsctl start res dbfs_mount $ crsctl stat res dbfs_mount
After the file system is mounted, create the directory for storing the GoldenGate files.
$ cd /mnt/dbfs/goldengate $ mkdir deployments
Note:
Leave the shared file system mounted. It is required for creating the GoldenGate deployment in a later step.
Oracle Advanced Cluster File System (ACFS)
Oracle ACFS is an alternative to DBFS for the shared GoldenGate files in an Oracle RAC configuration.
Refer to My Oracle Support note 1929629.1 for more information about ACFS configuration requirements for Oracle Exadata Database Machine.
Create a single ACFS file system for storing the Oracle deployment files.
It is recommended that you allocate enough trail file disk space to permit storage of up to 12 hours of trail files. Doing this provides sufficient space for trail file generation should a problem occur with the target environment that prevents it from receiving new trail files. The amount of space needed for 12 hours can only be determined by testing trail file generation rates with real production data.
-
Create the file system using ASMCMD as the Oracle ASM administrator user.
ASMCMD [+] > volcreate -G datac1 -s 1200G ACFS_GG
Note:
Modify the file system size according to the determined size requirements.ASMCMD> volinfo -G datac1 acfs_gg Diskgroup Name: DATAC1 Volume Name: ACFS_GG Volume Device: /dev/asm/acfs_gg-151 State: ENABLED Size (MB): 1228800 Resize Unit (MB): 64 Redundancy: MIRROR Stripe Columns: 8 Stripe Width (K): 1024 Usage: Mountpath:
Make the file system with the following mkfs command.
$ /sbin/mkfs -t acfs /dev/asm/acfs-gg-151
-
Create the CRS resource for the newly created ACFS file system, if not already created.
Check to see if the file system resource was already created.
$ srvctl status filesystem -volume ACFS_GG -diskgroup DATAC1 ACFS file system /mnt/acfs_gg is mounted on nodes oggadm07, oggadm08
If not already created, create the ACFS mount point on all of the Oracle RAC nodes.
# mkdir -p /mnt/acfs_gg
Create the file system resource as the root user. Due to the implementation of distributed file locking on ACFS, unlike DBFS, it is acceptable to mount ACFS on more than one RAC node at any one time.
Create the ACFS resource using srvctl from the Oracle Grid Infrastructure ORACLE_HOME.
# srvctl add filesystem -device /dev/asm/acfs_gg-151 -volume ACFS_GG -diskgroup DATAC1 -path /mnt/acfs_gg -user oracle -autostart RESTORE
To verify the currently configured ACFS file systems, use the following command to view the file system details.
$ srvctl config filesystem Volume device: /dev/asm/acfs_gg-151 Diskgroup name: datac1 Volume name: ACFS_GG Canonical volume device: /dev/asm/acfs_gg-151 Accelerator volume devices: Mountpoint path: /mnt/acfs_gg Mount point owner: oracle Check the status of the ACFS resource and mount it. $ srvctl status filesystem -volume ACFS_GG -diskgroup DATAC1 ACFS file system /mnt/acfs is not mounted $ srvctl start filesystem -volume ACFS_GG -diskgroup DATAC1 –node dc1north01
The CRS resource that is created is named using the format ora.diskgroup_name.volume_name.acfs. Using the above file system example, the CRS resource is called
ora.datac1.acfs_gg.acfs
.To see all ACFS file system CRS resources that currently exist, use the following command.
$ crsctl stat res -w "((TYPE = ora.acfs.type) OR (TYPE = ora.acfs_cluster.type))" NAME=ora.datac1.acfs_gg.acfs TYPE=ora.acfs.type TARGET=ONLINE , OFFLINE STATE=ONLINE on dc1north01, OFFLINE
-
Create a GoldenGate deployment directory on ACFS.
After the file system is mounted, create the directory for storing the GoldenGate deployments.
$ cd /mnt/acfs_gg $ mkdir deployments
Note:
Leave the shared file system mounted. It is required for creating the GoldenGate deployment in a later Task.
Task 5: Install Oracle GoldenGate
Download and install the Oracle GoldenGate 21c Microservices software, or later release.
Download the software at https://www.oracle.com/middleware/technologies/goldengate-downloads.html.
Install the Oracle GoldenGate software locally on all nodes in the Oracle RAC configuration that will be part of the GoldenGate configuration. Make sure the installation directory is identical on all nodes.
Follow the generic installation instructions detailed in Oracle GoldenGate Microservices Documentation.
Task 6: Create the Oracle GoldenGate Deployment
Once the Oracle GoldenGate software has been installed, the next step is to
create a deployment using the Oracle GoldenGate Configuration Assistant
(oggca
).
There are two limitations that currently exist with Oracle GoldenGate and XAG:
-
A Service Manager that is registered with XAG can only manage a single deployment. If multiple deployments are required, each deployment must use their own Service Manager. Oracle GoldenGate release 21c simplifies this requirement because it uses a single deployment to support Extract and Replicat processes connecting to different versions of the Oracle Database.
-
Each Service Manager registered with XAG must belong to separate
OGG_HOME
software installation directories. Instead of installing Oracle GoldenGate multiple times, the recommended approach is to install Oracle GoldenGate one time, and then create a symbolic link for each Service ManagerOGG_HOME
.
For example:
$ echo $OGG_HOME
/u01/oracle/goldengate/gg21c_MS
$ ln –s /u01/oracle/goldengate/gg21c_MS /u01/oracle/goldengate/gg21c_MS_ggnorth
$ export OGG_HOME=/u01/oracle/goldengate/gg21c_MS_ggnorth
$ $OGG_HOME/bin/oggca.sh
The symbolic link and OGG_HOME
environment variable must be
configured before running the Oracle GoldenGate Configuration Assistant on
all Oracle RAC nodes.
Recommendations for creating the GoldenGate deployment in the Oracle GoldenGate Configuration Assistant are as follows.
-
In Service Manager Options, specify the following for the creation of a new Service Manager.
-
In the Service Manager Details pane, select Create New Service Manager.
-
Enter the Service Manager Deployment Home location on the shared DBFS or ACFS file system.
-
Select to Integrate with XAG.
-
In the Service Manager Connection Details pane, specify
localhost
in the Listening hostname/address field.Using
localhost
allows the deployment to be started on all of the Oracle RAC nodes without the need for a Virtual IP address (VIP). -
Enter the port number in Listening port.
-
-
In Deployment Directories, specify the Deployment home directory on the shared DBFS or ACFS file system.
-
In Environment Variables, specify a correct TNS_ADMIN directory.
Starting with Oracle GoldenGate release 21.3, a database
ORACLE_HOME
is no longer required because the required database libraries are installed as part of the Oracle GoldenGate installation. It is recommended that you useTNS_ADMIN
directory outside of any existingORACLE_HOME
directories. -
In Security Options, do not select SSL/TLS Security.
External access to the Oracle GoldenGate Microservices server is achieved by using NGINX Reverse Proxy SSL-termination. Secure access and communication to the GoldenGate deployments will be exclusively through the SSL port 443. Internal connectivity within the same local host between NGINX and GoldenGate does not require SSL.
-
In Port Settings, if the Management Pack for Oracle GoldenGate has been licensed, select Enable Monitoring to use the performance metric server using either Berkeley Database (BDB) or Lightening Memory Database (LMDB).
For both BDB and LMDB Metrics Service DataStore types, set the Metrics Service DataStore home directory to a local directory that exists on all Oracle RAC nodes. For example:
/u01/oracle/goldengate/datastores/deployment name
-
Continue through the Oracle GoldenGate Configuration Assistant until the deployment is created.
-
After the deployment has been created, if you are using DBFS for the shared file system and the database version is a release earlier than Oracle Database Release 21c (21.3), run the following commands to move the Oracle GoldenGate deployment temp directory from DBFS to local storage.
On the first node:
$ cd <DBFS GoldenGate deployment home directory/var $ mkdir -p local_storage_directory/deployment_name $ mv temp local_storage_directory/deployment_name $ ln -s local_storage_directory/deployment_name/temp temp
On all other nodes:
$ mkdir local_storage_directory/deployment_name/temp
First node example:
$ cd /mnt/dbfs/goldengate/deployments/ggnorth/var $ mkdir –p /u01/oracle/goldengate/deployments/ggnorth $ mv temp /u01/oracle/goldengate/deployments/ggnorth $ ln -s /u01/oracle/goldengate/deployments/ggnorth/temp temp
On all other nodes:
$ mkdir /u01/oracle/goldengate/deployments/ggnorth/temp
Task 7: Oracle Clusterware Configuration
The following procedure shows you how to configure Oracle Clusterware to manage Oracle GoldenGate using the Oracle Grid Infrastructure Standalone Agent (XAG).
Using XAG automates the mounting of the shared file system (DBFS or ACFS) and the stopping and starting of the GoldenGate deployment when relocating between Oracle RAC nodes.
-
Install the Oracle Grid Infrastructure Standalone Agent.
It is recommended that you install the XAG software as a standalone agent outside of the Grid Infrastructure ORACLE_HOME. This allows you to use the latest XAG release available, and the software can be updated without impact to the Grid Infrastructure.
When using Oracle GoldenGate Microservices Architecture you MUST use XAG version 10.2 or later.
The latest agent software is available for download from the following location:
Install the XAG standalone agent outside of the Oracle Grid Infrastructure home directory. XAG must be installed in the same directory on all RAC nodes in the cluster where Oracle GoldenGate is installed.
For example, as the Oracle Grid Infrastructure user, the default of oracle:
$ ./xagsetup.sh --install --directory /u01/oracle/xag --all_nodes
Add the location of the newly installed XAG software to the PATH variable so that the location of
agctl
is known when theoracle
user logs on to the machine.$ cat .bashrc export PATH=/u01/oracle/xag/bin:$PATH
Note:
It is important to make sure that the XAG bin directory is specified BEFORE the Grid Infrastructure bin directory, to ensure the correct agctl binary is found. Set this location in the oracle user environment to take effect at time of logging on, such as in the .bashrc file when the Bash shell is in use. -
Prepare for Application Virtual IP Address (VIP) Creation.
A dedicated application VIP is required to allow access to the GoldenGate Microservices using the same host name, regardless of which Oracle RAC node is hosting the services. An application VIP also ensures that the GoldenGate Distribution Server can communicate with the Distribution Receiver running the current Oracle RAC node.
The VIP is a cluster resource that Oracle Clusterware manages. The VIP is assigned to a cluster node and is automatically migrated to another node in the event of a node failure.
There are two pieces of information needed before creating the application VIP:
-
The network number, which can be identified using the following command.
$ crsctl status resource -p -attr ADDRESS_TYPE,NAME,USR_ORA_SUBNET -w "TYPE = ora.network.type" |sort | uniq ADDRESS_TYPE=IPV4 NAME=ora.net1.network USR_ORA_SUBNET=10.133.16.0
The net1 in
NAME=ora.net1.network
indicates that this is network 1, and it is of type IPV4. -
The IP address for the new Application VIP, provided by your system administrator. This IP address must be in the same subnet of the cluster environment as determined above.
The VIP will be created in the next step, when configuring the Oracle Grid Infrastructure Agent.
-
-
Configure Oracle Grid Infrastructure Agent (XAG).
Oracle GoldenGate must be registered with XAG so that the deployment is started and stopped automatically when the database is started and the file system is mounted.
To register Oracle GoldenGate Microservices Architecture with XAG use the following command format.
agctl add goldengate instance_name --gg_home GoldenGate_Home --service_manager --config_home GoldenGate_SvcMgr_Config --var_home GoldenGate_SvcMgr_Var_Dir --port port_number --oracle_home $OGG_HOME/lib/instantclient --adminuser OGG_admin_user --user GG_instance_user --group GG_instance_group --network network_number --ip ip_address --vip_name vip_name --filesystems CRS_resource_name --db_services service_name --use_local_services --nodes node1, node2, ... ,nodeN
Where:
--gg_home
specifies the location of the Oracle GoldenGate software. Specify theOGG_HOME
symbolic link for theOGG_HOME
if registering multiple Service Managers (see Task 6: Create the Oracle GoldenGate Deployment).--service_manager
indicates this is a GoldenGate Microservices instance.--config_home
specifies the GoldenGate Service Manager deployment configuration home directory.--var_home
specifies the GoldenGate Service Manager deployment variable home directory.--port
specifies the deployment Service Manager port number.--oracle_home
specifies the location of the Oracle database libraries that are included as part of Oracle GoldenGate 21c and later releases. Example:$OGG_HOME/lib/instantclient
--adminuser
specifies the Oracle GoldenGate Microservices administrator account name.--user
specifies the name of the operating system user that owns the GoldenGate deployment.--group
specifies the name of the operating system group that owns the GoldenGate deployment.--network
specifies the network subnet for the VIP, determined above.--ip
specifies the IP address for the VIP, which was determined above. If you have already created a VIP, then specify it using the--vip_name
parameter in place of--network
and--ip
.--vip_name
specifies a CRS resource name for an application VIP that has previously been created. This parameter replaces--network
and-–ip
(optional).--filesystems
specifies the DBFS or ACFS CRS file system resource that must be mounted before the deployment is started.--db_services
specifies theora.database.service_name.svc
service name that was created in the previous step. If using Oracle Multitenant Database, specify the PDB database service for Replicat, or the CDB database service for an Extract. If using both Replicat and Extract, specify both services names, separated by a comma.--use_local_services
specifies that the GoldenGate instance must be co-located on the same Oracle RAC node where thedb_services
service is running.--nodes
specifies which of the Oracle RAC nodes this GoldenGate instance can run on. If GoldenGate is configured to run on any of the Oracle RAC nodes in the cluster, this parameter should still be used to determine the preferred order of nodes to run Oracle GoldenGate.
Notes:
- The GoldenGate instance registration with XAG MUST be run as the
root
user. - The
user
andgroup
parameters are mandatory because the GoldenGate registration with XAG is run as theroot
user.
Below are some examples of registering Oracle GoldenGate with XAG.
Example 1: Oracle RAC cluster using DBFS, using an already created application VIP
# agctl add goldengate GGNORTH \
--gg_home /u01/oracle/goldengate/gg21c_MS \
--service_manager \
--config_home /mnt/dbfs/goldengate/deployments/ggsm01/etc/conf \
--var_home /mnt/dbfs/goldengate/deployments/ggsm01/var \
--port 9100 \
--oracle_home /u01/oracle/goldengate/gg21c_MS/lib/instantclient
--adminuser oggadmin
--user oracle \
--group oinstall \
--vip_name gg_vip_prmy \
--filesystems dbfs_mount \
--db_services ora.ds19c.oggserv.svc \
--use_local_services \
--nodes dc1north01,dc1north02
Where:
- GoldenGate instance is GGNORTH
- GoldenGate home directory is
/u01/oracle/goldengate/gg21c_MS
- This is an Oracle GoldenGate Microservices Architecture instance
(
--service_manager
) - GoldenGate deployment configuration home directory is
/mnt/dbfs/goldengate/deployments/ggsm01/etc/conf
- GoldenGate deployment variable home directory is
/mnt/dbfs/goldengate/deployments/ggsm01/var
- Deployment Service Manager port number is 9100
- Oracle GoldenGate Microservices administrator account name is
oggadmin
- The GoldenGate user is
oracle
in the groupoinstall
- Application VIP name, managed by CRS, is called
gg_vip_prmy
- The CRS resource name for the file system the deployment depends on is
dbfs_mount
- The GoldenGate instance will be started on the same Oracle RAC node as the CRS
service called
ora.ds19c.oraserv.svc
will be co-located on the same node as this GoldenGate instance.
Example 2: Oracle RAC cluster, using ACFS, with an application VIP running on a subset of the nodes in the cluster.
# agctl add goldengate GGNORTH \
--gg_home /u01/oracle/goldengate/gg21c_MS \
--service_manager \
--config_home /mnt/acfs/goldengate/deployments/ggsm01/etc/conf \
--var_home /mnt/acfs/goldengate/deployments/ggsm01/var \
--port 9100 \
--oracle_home /u01/oracle/goldengate/gg21c_MS/lib/instantclient
--adminuser admin \
--user oracle \
--group oinstall \
--network 1 --ip 10.13.11.203 \
--filesystems ora.datac1.acfs_gg.acfs \
--db_services ora.ds19c.oraserv.svc \
--use_local_services \
--nodes dc1north01,dc1north02
Where:
- GoldenGate instance is GGNORTH
- GoldenGate home directory is
/u01/oracle/goldengate/gg21c_MS
- This is an Oracle GoldenGate Microservices Architecture instance
(
--service_manager
) - GoldenGate deployment configuration home directory is
/mnt/acfs/goldengate/deployments/ggsm02/etc/conf
- GoldenGate deployment variable home directory is
/mnt/acfs/goldengate/deployments/ggsm02/var
- Deployment Service Manager port number is 9100
- Oracle GoldenGate Microservices administrator account name is
admin
- GoldenGate user is
oracle
in the groupoinstall
- The network is the default
ora.net1.network
and the VIP is10.13.11.203
- The CRS resource name for the file system the deployment depends on is
ora.datac1.acfs_gg.acfs
- This GoldenGate instance will be started on the same Oracle RAC node as the CRS
service called
ora.ds19c.oraserv.svc
will be co-located on the same node as this GoldenGate instance - Oracle GoldenGate will only run on Oracle RAC nodes dc1north01 and dc1north02, listed in priority order.
Example AGCTL Commands
Below are some example agctl commands that are used to manage the Oracle GoldenGate deployment with XAG.
To check the status of Oracle GoldenGate:
% agctl status goldengate
Goldengate instance 'GGNORTH' is running on dc1north01
To start the GoldenGate deployment, and all Extract/Replicat processes that have been configured to autostart (instructions in a later step):
% agctl start goldengate GGNORTH --node dc1north02
To stop the GoldenGate deployment:
% agctl stop goldengate GGNORTH
To manually relocate the GoldenGate deployment to another node:
% agctl relocate goldengate GGNORTH --node dc1north02
To view the configuration parameters for the GoldenGate resource:
% agctl config goldengate GGNORTH
Instance name: GGNORTH
Application GoldenGate location is: /u01/oracle/goldengate/gg21c_MS
Goldengate MicroServices Architecture environment: yes
Goldengate Service Manager configuration directory: /mnt/dbfs/goldengate/deployments/ggsm01/etc/conf
Goldengate Service Manager var directory: /mnt/dbfs/goldengate/deployments/ggsm01/var
Service Manager Port: 9100
Goldengate Administration User: oggadmin
Autostart on DataGuard role transition to PRIMARY: no
Configured to run on Nodes: dc1north01 dc1north02
ORACLE_HOME location is: /u01/oracle/goldengate/gg21c_MS/lib/instantclient
Database Services needed: ora.cdb1.oggcdb.svc [use_local_services]
File System resources needed: ora.datac1.acfs_gg.acfs
Network: 1, IP: 10.13.11.203, User:oracle, Group:oinstall
To delete the GoldenGate XAG resource:
$ agctl stop goldengate GGNORTH
# agctl remove goldengate GGNORTH
For more information about the Oracle Grid Infrastructure Bundled Agent see Oracle Grid Infrastructure Standalone Agents for Oracle Clusterware 11g Rel. 2, 12c, 18c and 19c.
Task 8: Configure NGINX Reverse Proxy
Follow the instructions provided in My Oracle Support note 2826001.1 to install and configure NGINX Reverse Proxy with SSL connection, and to ensure all external communication is completely secure.
Note:
When using CA Signed Certificates with NGINX, make sure the NGINX
ssl_certificate
parameter points to a certificate file that
contains the certificates in the correct order of CA signed certificate,
intermediate certificate and root certificate.
Oracle Clusterware needs to have control over starting the NGINX reverse proxy so that it can be started automatically before the GoldenGate deployments are started.
The NGINX resource is created with a dependency on the underlying network CRS resource, the name of which can be determined using the following command:
$ $GRID_HOME/bin/crsctl stat res -w "TYPE == ora.network.type"|grep NAME
NAME=ora.net1.network
As the root
user, use the following example command to create a
Clusterware resource to manage NGINX.
# $GRID_HOME/bin/crsctl add resource nginx -type generic_application -attr
"ACL='owner:root:rwx,pgrp:root:rwx,other::r--,group:oinstall:r-x,user:oracle:rwx',
EXECUTABLE_NAMES=nginx,START_PROGRAM='/bin/systemctl
start -f nginx',STOP_PROGRAM='/bin/systemctl
stop -f nginx',CHECK_PROGRAMS='/bin/systemctl
status nginx' ,START_DEPENDENCIES='hard(ora.net1.network)
pullup(ora.net1.network)', STOP_DEPENDENCIES='hard(intermediate:ora.net1.network)',
RESTART_ATTEMPTS=0, HOSTING_MEMBERS='dc1north01,dc1north02', CARDINALITY=2"
The NGINX resource created in this example run on the named cluster nodes at the same
time, specified by HOSTING_MEMBERS
. This is recommended when
multiple GoldenGate Service Manager deployments are configured, and they can
independently move between cluster nodes.
Once the NGINX Clusterware resource is created, alter the GoldenGate XAG resources so that NGINX must be started before the GoldenGate deployments are started.
As the oracle
user, modify the XAG resources using the following
example commands.
Determine the current --filesystems
parameter:
$ agctl config goldengate SOURCE|grep "File System"
File System resources needed: ora.datac1.acfs_gg.acfs
Modify the --filesystems
parameter:
$ agctl modify goldengate SOURCE --filesystems ora.datac1.acfs_gg.acfs,nginx
Repeat the above commands for each of the XAG GoldenGate registrations relying on NGINX.
Task 9: Create Oracle Net TNS Alias for Oracle GoldenGate Database Connections
Create a TNS alias on all of the Oracle RAC nodes where Oracle GoldenGate may
be started to provide local database connections for the GoldenGate processes when
switching between Oracle RAC nodes. Create the TNS alias in the
tnsnames.ora
file in the TNS_ADMIN
directory
specified in the deployment creation.
If the source database is a multitenant database, two TNS alias entries
are required: one for the container database (CDB) and one for the pluggable
database (PDB) that is being replicated. For a target multitenant database, the TNS
alias connects the PDB where replicated data is being applied to. The pluggable
database SERVICE_NAME
should be set to the database service created
in an earlier step (refer to Task 3: Create the Database Services).
Below are some example source database TNS alias definitions using the IPC protocol, which must be defined locally on all RAC nodes.
OGGSOURCE_CDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL=IPC)(KEY=LISTENER))
(CONNECT_DATA =
(SERVICE_NAME = oggserv_cdb)
)
)
OGGSOURCE_PDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL=IPC)(KEY=LISTENER))
(CONNECT_DATA =
(SERVICE_NAME = oggserv_pdb)
)
)
Note:
When the tnsnames.ora
or sqlnet.ora
, located in
the TNS_ADMIN
directory for the GoldenGate deployment, are
modified, the deployment needs to be restarted in order to pick up the
changes.
With the GoldenGate deployment created, use the Administration Server home page to create the database credentials using the above TNS alias names. See Figure 6 below for an example of the database credential creation using the TNS alias appended to the database user name in the ‘User ID” field.
If the source database is a multitenant database, create database credentials for the CDB and PDB. If the target database is a multitenant database, create a single credential for the PDB.
Task 10: Configure Oracle GoldenGate Processes
When creating Extract, Distribution Paths, and Replicat processes with Oracle GoldenGate Microservices Architecture, all files that need to be shared between Oracle RAC nodes are already shared with the deployment files stored on a shared file system (DBFS or ACFS).
Listed below are important configuration details that are recommended for running Oracle GoldenGate Microservices on Oracle RAC for Extract, Distribution Paths and Replicat processes.
Extract Configuration
-
When creating an Extract using the Oracle GoldenGate Administration Server GUI interface, leave the Trail SubDirectory parameter blank, so that the trail files are automatically created in the deployment directories stored on the shared file system.
The default location for trail files is the
/<deployment directory>/var/lib/data
-
If you are using DBFS for shared storage, and the deployment
var/temp
directory was moved to local storage as described in Task 6: Create the Oracle GoldenGate Deployment, it is recommended that you use the ExtractCACHEMGR
parameter to place the temporary cache files on the shared storage.
Create a new directory under the DBFS deployment mount point. For example:
$ mkdir –p /mnt/dbfs/goldengate/deployments/ggnorth/temp_cache
Set the Extract parameter to the new directory:
CACHEMGR CACHEDIRECTORY /mnt/dbfs/goldengate/deployments/ggnorth/temp_cache
Shown below is an example of how the parameters specified for an integrated Extract with the Oracle GoldenGate Administration Server GUI looks in the UI.
Figure 24-1 Extract parameters for defining the temporary cache files
Distribution Path Configuration
When using Oracle GoldenGate distribution paths with the NGINX Reverse Proxy, there are additional steps that must be performed to ensure that the path server certificates are configured.
Follow the instructions provided in the following video to correctly configure the certificates: https://apexapps.oracle.com/pls/apex/f?p=44785:112:0::::P112_CONTENT_ID:31380
Configuration highlights presented in this video:
-
Create a client certificate for the source deployment and add the client certificate to the source deployment Service Manager. (This is not required when using Oracle GoldenGate 21c or later releases.)
-
Download the target deployment server’s root certificate and add the CA certificate to the source deployment Service Manager.
-
Create a user in the target deployment for the distribution path to connect to.
-
Create a credential in the source deployment connecting to the target deployment with the user created in the previous step.
For example, a domain of
GGNORTH_to_GGSOUTH
and an alias ofPathReceiver
.
After configuring the client and server certificates, the following configuration options need to be set. Refer to the figures below to see where these options are set in the UI.
-
Change the Generated Source URI specifying
localhost
for the server name.This allows the distribution path to be started on any of the Oracle RAC nodes.
-
Set the Target Authentication Method to
UserID Alias
and the Target transfer protocol towss
(secure web socket).Set the Target Host to the target host name/VIP that will be used for connecting to the target system along with the Port Number that NGINX was configured with (default is 443).
The target host name/VIP should match the common name in the CA signed certificate used by NGINX.
-
Set the Domain to the credential domain created above in step 4 and presented in the video, for example
GGNORTH_to_GGSOUTH
.The Alias is set to the credential alias, also created in step 4 in the video.
-
Set the distribution path to automatically restart when the Distribution Server starts.
This is required so that manual intervention is not required after an Oracle RAC node relocation of the Distribution Server. It is recommended that you set the number of Retries to 10. Set the Delay, which is the amount of time in minutes to pause between restart attempts, to 1.
Figure 24-2 Distribution Path Creation steps 1-3
Figure 24-3 Distribution Path Creation step 4
Replicat Configuration
-
The checkpoint table is a required component for GoldenGate Replicat processes. Make sure that a checkpoint table has been created in the database GoldenGate administrator (GGADMIN) schema.
The checkpoint table can be created using the Oracle GoldenGate Administration Server GUI, clicking on the ‘+’ button and entering the checkpoint table name in the format of schema.tablename. This is shown in the image below
Figure 24-4 Creating the checkpoint table for Replicat processes
See About Checkpoint Table for more information about creating a checkpoint table.
-
When creating a Replicat using the Oracle GoldenGate Administration Server GUI interface, set the Trail SubDirectory parameter to the location where the distribution path or local Extract are creating the trail files.
-
If a checkpoint table was created previously, select the table name from the Checkpoint Table pulldown list.
Figure 24-5 Replicat creation with Trail SubDirectory and Checkpoint Table
Task 11: Configure Autostart of Extract and Replicat Processes
Configure the Extract and Replicat processes to automatically start when the Oracle GoldenGate Administration Server is started, and then to restart if any Extract or Replicat processes abend. With GoldenGate Microservices auto start and restart is managed by Profiles.
Using the Oracle GoldenGate Administration Server GUI, create a new profile which can be assigned to each of the Oracle GoldenGate processes.
Profile Configuration Option | Recommended Setting |
---|---|
Default Profile | Enabled |
Auto Start | Enabled |
Startup Delay | 1 minute |
Auto Restart | Enabled |
Max Retries | 5 |
Retry Delay | 30 seconds |
Retries Window | 30 minutes |
Restart on Failure only | Enabled |
Disable Task After Retries Exhausted | Enabled |
After the profile has been created, and set as the default profile, all new GoldenGate processes created are assigned this profile. For all existing processes, the profile must be assigned to each process.
In the Overview pane, on the Process Information tab, select the Profile Name under Managed Options.
Note:
When using Oracle GoldenGate Microservices with XAG, it is strongly recommended not to enable the ‘Critical to deployment health’ flag for any Extract or Replicat processes. Doing so can cause an entire GoldenGate deployment outage from a single Extract or Replicat failure, and also prevents XAG from being able to restart GoldenGate. Refer to Troubleshooting Oracle GoldenGate on Oracle RAC for an example of troubleshooting an outage caused by setting a Replicat to critical.