1. High Availability Overview and Best Practices
  2. MAA Platinum and Oracle GoldenGate Best Practices
  3. Troubleshooting Oracle GoldenGate
  4. Troubleshooting Oracle GoldenGate on Oracle RAC

Troubleshooting Oracle GoldenGate on Oracle RAC

There may be occasions when Oracle GoldenGate processes are not successfully started on an Oracle RAC node. Several files generated by Oracle GoldenGate, XAG, and CRS should be reviewed to determine the cause of the problem.

Below is a list of important log and trace files, their example locations, and some examples of output.

XAG log file

Location: <XAG installation directory>/log/<hostname>

Example location: /u01/app/grid/xag/log/`hostname`

File name: agctl_goldengate_grid.trc

Contains all commands executed with agctl along with the output from the commands, including those that CRS executes.

Example:

2022-04-18 11:52:21: stop resource success
2022-04-18 11:52:38: agctl start goldengate <instance_name>
2022-04-18 11:52:38: executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl status res xag.<INSTANCE_NAME>.goldengate
2022-04-18 11:52:38: executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl status res xag.<INSTANCE_NAME>.goldengate -f
2022-04-18 11:52:38: executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl start resource xag.<INSTANCE_NAME>.goldengate -f
2022-04-18 11:52:45: Command output:
>  CRS-2672: Attempting to start 'xag.<INSTANCE_NAME>.goldengate' on 'exadb-node1'
>  CRS-2676: Start of 'xag.<INSTANCE_NAME>.goldengate' on 'exadb-node1' succeeded
>End Command output
2022-04-18 11:52:45: start resource success

XAG GoldenGate instance trace file

Location: <XAG installation directory>/log/<hostname>

Example location: /u01/app/grid/xag/log/`hostname`

File name: <GoldenGate_instance_name>_agent_goldengate.trc

It contains the output from the commands executed by agctl, the environment variables used, and any debug output enabled for the underlying commands.

Example:

2022-04-18 12:14:46: Exported ORACLE_SID ggdg1
2022-04-18 12:14:46: Exported GGS_HOME /u01/oracle/goldengate/gg21c_MS
2022-04-18 12:14:46: Exported OGG_CONF_HOME /mnt/dbfs/goldengate/deployments/ggsm01/etc/conf
2022-04-18 12:14:46: Exported LD_LIBRARY_PATH
/u01/oracle/goldengate/gg21c_MS:/u01/app/19.0.0.0/grid/lib:/etc/ORCLcluster/lib
2022-04-18 12:14:46: Exported LD_LIBRARY_PATH_64 /u01/oracle/goldengate/gg21c_MS
2022-04-18 12:14:46: Exported LIBPATH /u01/oracle/goldengate/gg21c_MS
2022-04-18 12:14:46: ogg input = {"oggHome":"/u01/oracle/goldengate/gg21c_MS","serviceManager":{"oggConfHome":"/mnt/dbfs/goldengate/deployments/ggsm01/etc/conf","portNumber":9100},"username":"admin","credential":"xyz"}
2022-04-18 12:14:46: About to exec /u01/oracle/goldengate/gg21c_MS/bin/XAGTask HealthCheck
2022-04-18 12:14:47: XAGTask retcode = 0

CRS trace file

Location: /u01/app/grid/diag/crs/<hostname>/crs/trace

Example location: /u01/app/grid/diag/crs/`hostname`/crs/trace

File name: crsd_scriptagent_oracle.trc

Contains the output created by any CRS resource action scripts, like XAG or dbfs_mount. This trace file is crucial to determining why DBFS or GoldenGate did not start on a RAC node.

Example:

2022-04-18 11:52:38.634 :    AGFW:549631744: {1:30281:59063} Agent received the message: RESOURCE_START[xag.<INSTANCE_NAME>.goldengate 1 1] ID 4098:4125749
2022-04-18 11:52:38.634 :    AGFW:549631744: {1:30281:59063} Preparing START command for: xag.<INSTANCE_NAME>.goldengate 1 1
2022-04-18 11:52:38.634 :    AGFW:549631744: {1:30281:59063} xag.<INSTANCE_NAME>.goldengate 1 1 state changed from: OFFLINE to: STARTING
2022-04-18 11:52:38.634 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [start] Executing action script: /u01/oracle/XAG_MA/bin/aggoldengatescaas[start]
2022-04-18 11:52:38.786 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [start] GG agent running command 'start' on xag.<INSTANCE_NAME>.goldengate
2022-04-18 11:52:42.140 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [start] ServiceManager fork pid = 265747
2022-04-18 11:52:42.140 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [start] Waiting for /mnt/dbfs/goldengate/deployments/ggsm01/var/run/ServiceManager.pid
2022-04-18 11:52:42.140 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [start] Waiting for SM to start
2022-04-18 11:52:42.140 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [start] ServiceManager PID = 265749
2022-04-18 11:52:43.643 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [start] XAGTask retcode = 0
2022-04-18 11:52:43.643 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [start] XAG HealthCheck after start returned 0
2022-04-18 11:52:43.643 :    AGFW:558036736: {1:30281:59063} Command: start for resource: xag.<INSTANCE_NAME>.goldengate 1 1 completed with status: SUCCESS
2022-04-18 11:52:43.643 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [check] Executing action script: /u01/oracle/XAG_MA/bin/aggoldengatescaas[check]
2022-04-18 11:52:43.644 :    AGFW:549631744: {1:30281:59063} Agent sending reply for: RESOURCE_START[xag.<INSTANCE_NAME>.goldengate 1 1] ID 4098:4125749
2022-04-18 11:52:43.795 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [check] GG agent running command 'check' on xag.<INSTANCE_NAME>.goldengate
2022-04-18 11:52:45.548 :CLSDYNAM:558036736: [xag.<INSTANCE_NAME>.goldengate]{1:30281:59063} [check] XAGTask retcode = 0
2022-04-18 11:52:45.548 :    AGFW:549631744: {1:30281:59063} xag.<INSTANCE_NAME>.goldengate 1 1 state changed from: STARTING to: ONLINE

GoldenGate deployment log files

Location: <Goldengate_deployment_directory>/<instance_name>/var/log

Example location: /mnt/dbfs/goldengate/deployments/<instance_name>/var/log

File names: adminsrvr.log, recvsrvr.log, pmsrvr.log, distsrvr.log

Contains the output of start, stop, and status checks of the Oracle GoldenGate deployment processes (Administration Server, Distribution Server, Receiver Server, and Performance Metrics Server).

Example:

2022-04-18T11:52:42.645-0400 INFO | Setting deploymentName to '<instance_name>'. (main)
2022-04-18T11:52:42.665-0400 INFO | Read SharedContext from store for length 19 of file '/mnt/dbfs/goldengate/deployments/<instance_name>/var/lib/conf/adminsrvr-resources.dat'. (main)
2022-04-18T11:52:42.723-0400 INFO | XAG Integration enabled (main)
2022-04-18T11:52:42.723-0400 INFO | Configuring security. (main)
2022-04-18T11:52:42.723-0400 INFO | Configuring user authorization secure store path as '/mnt/dbfs/goldengate/deployments/<instance_name>/var/lib/credential/secureStore/'. (main)
2022-04-18T11:52:42.731-0400 INFO | Configuring user authorization as ENABLED. (main)
2022-04-18T11:52:42.749-0400 INFO | Set network configuration. (main)
2022-04-18T11:52:42.749-0400 INFO | Asynchronous operations are enabled with default synchronous wait time of 30 seconds (main)
2022-04-18T11:52:42.749-0400 INFO | HttpServer configuration complete. (main)
2022-04-18T11:52:42.805-0400 INFO | SIGHUP handler installed. (main)
2022-04-18T11:52:42.813-0400 INFO | SIGINT handler installed. (main)
2022-04-18T11:52:42.815-0400 INFO | SIGTERM handler installed. (main)
2022-04-18T11:52:42.817-0400 WARN | Security is configured as 'disabled'. (main)
2022-04-18T11:52:42.818-0400 INFO | Starting service listener... (main)
2022-04-18T11:52:42.819-0400 INFO | Mapped 'ALL' interface to address 'ANY:9101' with default IPV4/IPV6 options identified by 'exadb-node1.domain'. (main)
2022-04-18T11:52:42.821-0400 INFO | Captured 1 interface host names: 'exadb-node1.domain' (main)
2022-04-18T11:52:42.824-0400 INFO | The Network ipACL specification is empty. Accepting ANY address on ALL interfaces. (main)
2022-04-18T11:52:42.826-0400 INFO | Server started at 2022-04-18T11:52:42.827-05:00 (2022-04-18T15:52:42.827Z GMT) (main)

GoldenGate report files

Location: <Goldengate_deployment_directory>/<instance_name>/var/lib/report

Example location: /mnt/dbfs/goldengate/deployments/<instance_name>/var/lib/report

The GoldenGate report files contain important information, warning messages, and errors for all GoldenGate processes, including the Manager processes. If any of the GoldenGate processes fail to start or abend when running, the process report file will contain important information that can be used to determine the cause of the failure.

Example errors from an Extract report file:

2022-04-23 13:01:50  ERROR   OGG-00446  Unable to lock file " /mnt/acfs_gg/deployments/<instance_name>/var/lib/checkpt/EXT_1A.cpe" (error 95, Operation not supported).
2022-04-23 13:01:50  ERROR   OGG-01668  PROCESS ABENDING.

Example Configuration Problems

Below are some configuration problems that can be encountered with GoldenGate in a RAC environment and how to diagnose and resolve them.

Incorrect parameter settings in the mount-dbfs.conf file

When XAG fails to mount DBFS, the failure will be reported either on the command line (if you are running the manual agctl command) or in the XAG log file:

$ agctl start goldengate <instance_name> --node exadb-node1

CRS-2672: Attempting to start 'dbfs_mount' on 'exadb-node1'
CRS-2674: Start of 'dbfs_mount' on 'exadb-node1' failed
CRS-2679: Attempting to clean 'dbfs_mount' on 'exadb-node1'
CRS-2681: Clean of 'dbfs_mount' on 'exadb-node1' succeeded
CRS-4000: Command Start failed, or completed with errors.

The XAG log file (agctl_goldengate_grid.trc) has the advantage that it shows timestamps that can be used when looking at other log or trace files:

2022-04-19 15:32:16: executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl start resource xag.<INSTANCE_NAME>.goldengate -f -n exadb-node1
2022-04-19 15:32:19: Command output:
>  CRS-2672: Attempting to start 'dbfs_mount' on 'exadb-node1'
>  CRS-2674: Start of 'dbfs_mount' on 'exadb-node1' failed
>  CRS-2679: Attempting to clean 'dbfs_mount' on 'exadb-node1'
>  CRS-2681: Clean of 'dbfs_mount' on 'exadb-node1' succeeded
>  CRS-4000: Command Start failed, or completed with errors.
>End Command output
2022-04-19 15:32:19: start resource failed rc=1

Next, check the CRS trace file (crsd_scriptagent_oracle.trc), which shows why DBFS failed to mount. Below are some example errors caused by incorrect parameter settings in the mount-dbfs.conf file.

  • Incorrect DBNAME
    2022-04-19 15:32:16.679 :    AGFW:1190405888: {1:30281:17383} dbfs_mount
 1 1 state changed from: UNKNOWN to: STARTING
2022-04-19 15:32:16.680 :CLSDYNAM:1192507136: [dbfs_mount]{1:30281:17383} [start]
 Executing action script: /u01/oracle/scripts/mount-dbfs.sh[start]
2022-04-19 15:32:16.732 :CLSDYNAM:1192507136: [dbfs_mount]{1:30281:17383} [start]
 mount-dbfs.sh mounting DBFS at /mnt/dbfs from database ggdg
2022-04-19 15:32:17.883 :CLSDYNAM:1192507136: [dbfs_mount]{1:30281:17383} [start]
 ORACLE_SID is
2022-04-19 15:32:17.883 :CLSDYNAM:1192507136: [dbfs_mount]{1:30281:17383} [start]
 No running ORACLE_SID available on this host, exiting
2022-04-19 15:32:17.883 :    AGFW:1192507136: {1:30281:17383} Command: start for
 resource: dbfs_mount 1 1 completed with invalid status: 2
  • Incorrect MOUNT_POINT
    2022-04-19 16:45:14.534 :    AGFW:1734321920: {1:30281:17604} dbfs_mount
 1 1 state changed from: UNKNOWN to: STARTING
2022-04-19 16:45:14.535 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start]
 Executing action script: /u01/oracle/scripts/mount-dbfs.sh[start]
2022-04-19 16:45:14.586 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start]
 mount-dbfs.sh mounting DBFS at /mnt/dbfs from database ggdgs
2022-04-19 16:45:15.638 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start]
 ORACLE_SID is ggdg1
2022-04-19 16:45:15.738 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start]
 spawning dbfs_client command using SID ggdg1
2022-04-19 16:45:20.745 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start]
 fuse: bad mount point `/mnt/dbfs': No such file or directory
2022-04-19 16:45:21.747 :CLSDYNAM:1736423168: [dbfs_mount]{1:30281:17604} [start]
 Start - OFFLINE
2022-04-19 16:45:21.747 :    AGFW:1736423168: {1:30281:17604} Command: start for
 resource: dbfs_mount 1 1 completed with status: FAIL
  • Incorrect DBFS_USER or DBFS_PASSWD
    2022-04-19 16:47:47.855 :    AGFW:1384478464: {1:30281:17671} dbfs_mount
 1 1 state changed from: UNKNOWN to: STARTING
2022-04-19 16:47:47.856 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
 Executing action script: /u01/oracle/scripts/mount-dbfs.sh[start]
2022-04-19 16:47:47.908 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
 mount-dbfs.sh mounting DBFS at /mnt/dbfs from database ggdgs
2022-04-19 16:47:48.959 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
 ORACLE_SID is ggdg1
2022-04-19 16:47:49.010 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
 spawning dbfs_client command using SID ggdg1
2022-04-19 16:47:55.118 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
 Fail to connect to database server. Error: ORA-01017: invalid username/password;
 logon denied
2022-04-19 16:47:55.118 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
2022-04-19 16:47:56.219 :CLSDYNAM:1386579712: [dbfs_mount]{1:30281:17671} [start]
 Start - OFFLINE
2022-04-19 16:47:56.220 :    AGFW:1386579712: {1:30281:17671} Command: start for
 resource: dbfs_mount 1 1 completed with status: FAIL
  • Incorrect ORACLE_HOME
    2022-04-19 16:50:38.952 :    AGFW:567502592: {1:30281:17739} dbfs_mount
 1 1 state changed from: UNKNOWN to: STARTING
2022-04-19 16:50:38.953 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
 Executing action script: /u01/oracle/scripts/mount-dbfs.sh[start]
2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
 mount-dbfs.sh mounting DBFS at /mnt/dbfs from database ggdgs
2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
 /u01/oracle/scripts/mount-dbfs.sh: line 136:
 /u01/app/oracle/product/19.0.0.0/rdbms/bin/srvctl: No such file or directory
2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
 /u01/oracle/scripts/mount-dbfs.sh: line 139:
 /u01/app/oracle/product/19.0.0.0/rdbms/bin/srvctl: No such file or directory
2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
 ORACLE_SID is
2022-04-19 16:50:39.004 :CLSDYNAM:569603840: [dbfs_mount]{1:30281:17739} [start]
 No running ORACLE_SID available on this host, exiting
2022-04-19 16:50:39.004 :    AGFW:569603840: {1:30281:17739} Command: start for
 resource: dbfs_mount 1 1 completed with invalid status: 2

To resolve these configuration issues, set the correct parameter values in mount-dbfs.conf.

Problems with file locking on DBFS

If using Oracle Database 12c Release 2 (12.2) and the nolock DBFS mount option is not used, there can be problems with GoldenGate processes trying to lock checkpoint or trail files. The same problem will be encountered if using Oracle Database 11g Release 2 (11.2.0.4) or 12c Release 1 (12.1) with a patch for bug 22646150 applied. This patch changes how DBFS handles file locking to match Oracle Database 12c Release 2 (12.2). To add the nolock DBFS mount option, a patch for bug 27056711 must be applied to the database. If the patch for bug 22646150 has not been applied to the database, the patch for bug 27056711 and the nolock mount option is not required.

Below is an example of diagnosing a GoldenGate Microservices Architecture locking problem.

When starting a deployment with XAG, one or more processes may not start due to detecting a locking conflict on one or more files. This will often occur after a RAC node failover where the deployment did not get a chance to shut down cleanly.

When one of the deployment server processes fails to start (Administration Server, Performance Metrics Server, Distribution Server, Receiver Server, or Service Manager), check the log file for the particular server located in the deployment var/log directory.

For example, the log file /mnt/dbfs/goldengate/deployments/<INSTANCE_NAME>/var/log/pmsrvr.log shows the following error on startup:

2022-04-11T12:41:57.619-0700 ERROR| SecureStore failed on open after
 retrying due to extended file lock. (main)
2022-04-11T12:41:57.619-0700 ERROR| SecureStore failed to close (28771). (main)
2022-04-11T12:41:57.619-0700 INFO | Set network configuration. (main)
2022-04-11T12:41:57.619-0700 INFO | Asynchronous operations are enabled with default
 synchronous wait time of 30 seconds (main)
2022-04-11T12:41:57.619-0700 INFO | HttpServer configuration complete. (main)
2022-04-11T12:42:07.674-0700 ERROR| Unable to lock process file, Error is [1454]
 - OGG-01454 (main)
2022-04-11T12:42:07.675-0700 ERROR| Another Instance of PM Server is Already Running
 (main)

An Extract process will report start-up failures in the ER-events.log logfile located in the deployment log file directory.

For example, /mnt/dbfs/goldengate/deployments/<instance_name>/var/log/ER-events.log shows the following error:

2022-04-11T00:14:56.845-0700  ERROR   OGG-01454  Oracle GoldenGate Capture for
 Oracle, EXT1.prm:  Unable to lock file
 "/mnt/dbfs/goldengate/deployments/<instance_name>/var/run/EXT1.pce" (error 11, Resource
 temporarily unavailable).  Lock currently held by process id (PID) 237495.
2022-04-11T00:14:56.861-0700  ERROR   OGG-01668  Oracle GoldenGate Capture for Oracle,
 EXT1.prm:  PROCESS ABENDING.

Next, check to ensure the process failing to start up is not running on any of the RAC nodes.

Example:

$ ps -ef|grep EXT1|grep -v grep

Once it has been determined that the process is not running, the deployment must be shutdown cleanly, the file system unmounted, and the correct DBFS patch applied.

Example:

$ agctl stop goldengate <INSTANCE_NAME>
$ crsctl stop resource dbfs_mount

Check the DBFS mount options:

$ ps -ef|grep dbfs_client

oracle   204017      1  0 14:37 ? 
       00:00:00 /u01/app/oracle/product/19.1.0.0/dbhome_1/bin/dbfs_client dbfs@dbfs.local
 -o allow_other,failover,direct_io /mnt/dbfs

It is clear the nolock mount option was not used, which leads to the locking errors.

Use the guidelines above to determine if a DBFS patch is required. After which, add the nolock mount option to the mount-dbfs.conf file on all Oracle RAC nodes that are part of the deployment.

Example:

MOUNT_OPTIONS=allow_other,direct_io,failover,nolock

Finally, restart the deployment:

$ agctl start goldengate <INSTANCE_NAME>

Critical Extract/Replicat process preventing GoldenGate start-up

When an Extract or Replicat process has been set as critical to the health of the deployment in the Administration Server, if the process abends, the Service Manager and deployment will be stopped.

When restarting GoldenGate using XAG (agctl start goldengate) it will fail with the following error: 

$ agctl start goldengate GGNORTH --node dc1north01
CRS-2672: Attempting to start 'xag.GGNORTH.goldengate' on 'dc1north01'
CRS-2674: Start of 'xag.GGNORTH.goldengate' on 'dc1north01' failed
CRS-2679: Attempting to clean 'xag.GGNORTH.goldengate' on ' dc1north01'
CRS-2681: Clean of 'xag.GGNORTH.goldengate' on ' dc1north01' succeeded
CRS-2632: There are no more servers to try to place resource 'xag.GGNORTH.goldengate' on
that would satisfy its placement policy
CRS-4000: Command Start failed, or completed with errors.

The CRS trace file (crsd_scriptagent_oracle.trc) does not provide enough information to determine the reason behind the startup failure.

Example:

2022-06-30 20:20:28.675 :    AGFW:521639680: [     INFO] {1:29336:52291}
 Command: start for resource: xag.GGNORTH.goldengate 1 1 completed with status: SUCCESS
2022-06-30 20:20:28.675 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52291}
 [check] Executing action script: /u01/oracle/XAG/bin/aggoldengatescaas[check]
2022-06-30 20:20:28.677 :    AGFW:517437184: [     INFO] {1:29336:52291} Agent sending
 reply for: RESOURCE_START[xag.GGNORTH.goldengate 1 1] ID 4098:178187
2022-06-30 20:20:28.827 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52291}
 [check] GG agent running command 'check' on xag.GGNORTH.goldengate
2022-06-30 20:20:28.978 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52291}
 [check] execute XAGTask HealthCheck
2022-06-30 20:20:29.779 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52291}
 [check] XAGTask retcode = 3
2022-06-30 20:20:29.780 :    AGFW:517437184: [     INFO] {1:29336:52291}
 xag.GGNORTH.goldengate 1 1 state changed from: STARTING to: FAILED

Starting with Oracle GoldenGate version 21.11, when using XAG, additional GoldenGate status information is logged into the Service Manager $OGG_VAR_HOME/log/XAGTask.log file.

For earlier releases of Oracle GoldenGate, enable the CRS debug logging for XAG using the following commands.

$ crsctl set log res xag.GGNORTH.goldengate=5
$ agctl start goldengate GGNORTH --node dc1north01
$ crsctl set log res xag.GGNORTH.goldengate=1

Check the Service Manager $OGG_VAR_HOME/log/XAGTask.log or the CRS trace file (crsd_scriptagent_oracle.trc) for more information on the failure.

Example:

2022-06-30 20:24:36.389 :CLSDYNAM:521639680:
 [xag.GGNORTH.goldengate]{1:29336:52435} [check]   Deployment: GGNORTH
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
 [check]      Service: adminsrvr
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
 [check]       Status: running
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
 [check]      Started: 2022-07-01T03:24:35.911Z
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
 [check]       Health: Failed
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
 [check]    Resources: Name                              Type        Status      Health
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
 [check]               --------------------------------  ----------  ----------  ------
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
 [check]               REP1                              replicat    abended     Failed
2022-06-30 20:24:36.389 :CLSDYNAM:521639680: [xag.GGNORTH.goldengate]{1:29336:52435}
 [check]  HealthCheck failed

When Extract or Replicat processes are not set to critical, they will not appear in the CRS trace file. Because Replicat REP1 is shown as failed, this indicates that REP1 is set to critical and is preventing GoldenGate from starting.

To disable the critical setting for Replicat, the GoldenGate Service Manager and deployment must be started manually.

Example:

$ export OGG_ETC_HOME=/mnt/acfs/goldengate/deployments/ggsm01/etc
$ export OGG_VAR_HOME=/mnt/acfs/goldengate/deployments/ggsm01/var
$ export OGG_HOME=/u01/oracle/goldengate/ggMS_21c
$ $OGG_HOME/bin/ServiceManager --xagEnabled

Using the Administration Server GUI, select the Replicat or Extract details, and unset the critical flag.

Once the critical setting has been disabled, XAG can be used to start and stop Oracle GoldenGate Microservices.

Example:

$ agctl start goldengate GGNORTH --node dc1north01
$ agctl status goldengate GGNORTH