6 Troubleshooting Global Data Services

If you encounter issues with Global Data Services (GDS) tools and solutions, you can use these topics to help you to resolve them.

Obtaining the Status of GDS Components

To obtain the running status of the GDS components, you can use the status command.

The following is a set of commands available for you to use for component checks: 

GDSCTL>status gsm
 
GDSCTL>status service
 
GDSCTL>status database

Parent topic: Troubleshooting Global Data Services

Viewing GDS Configuration Information

To obtain the static configuration information of various GDS components, you can use the gdsctl config command.

The following is a set of commands available for you to use for component checks: 

GDSCTL>config

GDSCTL>config gsm

GDSCTL>config region

GDSCTL>config gdspool

GDSCTL>config database

GDSCTL>config service

GDSCTL>config invitednode

Parent topic: Troubleshooting Global Data Services

Networking Issues

Identify and correct Oracle Global Data Services networking issues.

Parent topic: Troubleshooting Global Data Services

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Connection fails with ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Review the following potential causes:

The global service may be down. Verify that the pool databases are up and the service is started.

The global service may be disabled. Ensure that the pool databases are up and the service is enabled and started.

The GDS pool database may be down. Ensure that the GDS pool databases are up and the service is enabled and started.

Parent topic: Networking Issues

ORA-12516: TNS:listener could not find available handler with matching protocol stack

Cause: The GDS pool database's local listener may be down. Ensure that the GDS pool database local listener is running.

Parent topic: Networking Issues

ORA-12541: TNS:no listener

Cause: All global service managers may be down. Verify that the global service managers are running.

Parent topic: Networking Issues

GSM-40167: VNCR entry "<hostname>" is not resolvable on GSM host

GSM restart after GSM-upgrade fails with error: GSM-40167: VNCR entry "<hostname>" is not resolvable on GSM host

This issue is most likely due to the presence of short hostnames in the VNCR list causing an issue during GSM start in Oracle AI Database. If this issue is encountered, then remove invitednode of the two short shard hostname entries and then try to start the GSM.

Parent topic: Networking Issues

Resolving Database Registration Issues when Using add brokerconfig in GDS Environments

If the GDS database not registering after add brokerconfig, then review common causes and their solutions.

When configuring a Global Data Services (GDS) environment, you might encounter database registration issues with the GDS listeners. The following are common causes and solutions for missing database registration.

A common sign of this issue is the absence of database registration on GSM listeners. You can detect this by running the GDSCTLstatus database command. For example: 

status database
Database: "test" Registered: N State: Ok ONS: Y. Role: N/A Instances: 0 Region: region1
Service: "test_failover" Globally started: Y Started: N
Scan: Y Enabled: Y Preferred: Y

The important part of this output is Registered property. If this property is set to N, then registration has not occurred.

For a database to register with a GDS listener, you must assign a region to the database. If you used the add brokerconfig command to add databases to a GDS pool that contains two or more regions, and the status database command shows no registration, then assign the region manually using the modify database command: 

modify database … -region <region_name>

If you add more than one standby database to the GDS configuration simultaneously, then you can receive the error ORA-45539: Database db_name has already been added to another pool. To resolve this issue, add and synchronize standby databases one at a time.

Parent topic: Troubleshooting Global Data Services

Invalid Objects Related Issues

Database Release Update (DBRU) patches can invalidate GSMADMIN_INTERNAL objects; verify schema empty, drop or recreate with scripts, recompile, validate, and drop SYS duplicates.

After installing a Database Release Update (DBRU) or individual patches, some objects in the GSMADMIN_INTERNAL schema can become invalid in the database as a whole or in individual containers (PDBs). For example, assume the query below is issued after a DBRU: 

select owner, object_name, object_type, status from dba_objects 
where status='INVALID' and owner='GSMADMIN_INTERNAL' order by owner, object_name, object_type;
OWNER                 OBJECT_NAME               OBJECT_TYPE      STATUS     CON_ID
-----------------     ----------------------    -------------    -------    ------
GSMADMIN_INTERNAL     DBMS_GSM_POOLADMIN        PACKAGE BODY     INVALID    3
GSMADMIN_INTERNAL     CAT_ROLLBACK_TRIGGER      TRIGGER          INVALID    3
GSMADMIN_INTERNAL     REQUEST_DELETE_TRIGGER    TRIGGER          INVALID    3
GSMADMIN_INTERNAL     DONE_TRIGGER              TRIGGER          INVALID    3
...

The datapatch script execution failed due to an Invalid Objects error on the GSMADMIN_INTERNAL schema. To resolve the invalid object issues, re-running Oracle-provided scripts is necessary. Complete these steps:

1. Ensure that the GSMADMIN_INTERNAL schema does not contain any user data.

If the Global Data Services (GDS) or Oracle Sharding features are in use, the GSMADMIN_INTERNAL schema may contain important configuration data.

Run the following query as SYS to verify that the schema is empty. For container databases (CDBs), this query MUST be run in CDB$ROOT and in all pluggable databases (PDBs).

select count(*) from gsmadmin_internal.cloud;

If this query returns 0 (zero), then move on to the next step. If it returns non-zero, contact Oracle Customer Support for further guidance.

2. Connect to the database as SYS using SQL*Plus. If the database is a CDB, connect to CDB$ROOT, not to any individual PDB.

3. Run the following commands:


SQL> spool /tmp/invalids.out
SQL> alter session set “_ORACLE_SCRIPT” = true;
SQL> drop user gsmadmin_internal cascade;
SQL> alter session set “_ORACLE_SCRIPT” = false;
SQL> @?/rdbms/admin/catgwm.sql
SQL> @?/rdbms/admin/dbmsgwm.sql
SQL> @?/rdbms/admin/catgwmcat.sql
SQL> @?/rdbms/admin/prvtgwm.sql
SQL> @?/rdbms/admin/utlrp.sql
SQL> spool off

4. Validate that there are no invalid objects owned by GSMADMIN_INTERNAL by running the following query: select owner, object_name, object_type, status from dba_objects where status='INVALID' and owner='GSMADMIN_INTERNAL' order by owner, object_name, object_type;

If no rows are returned, then the GSMADMIN_INTERNAL schema is now correct.

If there are still invalid objects, then contact Oracle Customer Support for further guidance and provide the output from the scripts as found in /tmp/invalids.out.

5. Finally, in certain instances, it is possible that some GSMADMIN_INTERNAL objects have been incorrectly created in the SYS schema.

These objects should therefore be dropped since they are duplicates of the correct objects in the GSMADMIN_INTERNAL schema.

To identify objects owned by SYS which should be dropped, run the following query as SYS: select object_name, object_type from dba_objects where owner = ‘SYS’ and (object_name, object_type) in (select object_name, object_type from dba_objects where owner = ‘GSMADMIN_INTERNAL’) order by object_name, object_type;

This will generate a list of all objects owned by SYS that are also in GSMADMIN_INTERNAL.

Since these objects were created in the wrong schema, they should all be dropped using the appropriate DROP commands from SQL*Plus.

Re-running this query should return 0 (zero) rows when all listed objects have been dropped.

Parent topic: Troubleshooting Global Data Services

User and Password Management Issues

Review this content to obtain suggestions for Global Data Services issues related to user and password management.

Parent topic: Troubleshooting Global Data Services

ORA-01045: user GSMADMIN_INTERNAL lacks CREATE SESSION privilege; logon denied

Do not unlock or grant privileges to GSMADMIN_INTERNAL. Instead, trace failed logons to misconfigured GDS credentials or scripts.

No direct modifications should be made on the Global Data Services schema objects unless directed by Oracle Technical Support.

Why does this error occur? The user GSMADMIN_INTERNAL is an internal-only user for controlled operations. It should never be unlocked or used for any database login. It is intentionally restricted (including lacking CREATE SESSION), so it cannot be used as a general login.

Seeing ORA-01045 typically means a component is attempting a direct DB login using GSMADMIN_INTERNAL, which usually indicates:

  • Misconfiguration (wrong username stored somewhere).
  • Someone attempted to connect manually with that username.
  • A script or tool mistakenly references it.
  • There are credentials, wallet or connection string issues that cause fallback to an unintended user.

Parent topic: User and Password Management Issues

Troubleshooting GDS Issues

Identify and correct Oracle Global Data Services issues.

Parent topic: Troubleshooting Global Data Services

GSM-45034: Connection to GDS catalog is not established

The GDS catalog database or its listener may be down, or is unreachable for another reason.

The error message "GSM-45034: Connection to GDS catalog is not established" means the GDS/GSM instance (Global Service Manager) cannot connect to its GDS catalog database (the repository database that stores the GDS configuration).

To adress this issue, verify that the GDS catalog is reachable, that credentials are correct, that the GSM is using the right connect descriptor or service, and that the GDS catalog database and its local listeners are running.

Parent topic: Troubleshooting GDS Issues

Connecting to GDS Configuration Databases When No Global Service Managers Are Running

Troubleshoot GSM by using three address lists: regional GSM, buddy GSM, and then the local/SCAN; fall back only if GSM is down.

When troubleshooting GSM connectivity, check the configuration and connect using your region’s GSM listeners first. This first list should be exclusively regional global service manager listeners. The second list contains global service manager listeners of the buddy region. The third list should be your local or SCAN listeners. With these three lists, connections use GSM when it’s available and only fall back to local listeners if all GSM listeners are down.

You can always connect through a global service manager while it is up, and only fail over to local listeners when all global service manager listeners are down.

Template

Use this template as a model. 

(DESCRIPTION=
  (FAILOVER=on)
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
   )
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
   )
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(local_protocol_address_information))
    (ADDRESS=(local_protocol_address_information))
   )
  (CONNECT_DATA=
   (SERVICE_NAME=global_service_name)
   (REGION=region_name)))

Note that this example uses three address lists, where gsmhost1, gsmhost2 and gsmhost3 are the regional GSMs. gsmhost4, gsmhost5 and gsmhost6 are buddy GSMs. Finally, server1 has the local listener. 

(DESCRIPTION=
     (FAILOVER=on)
     (ADDRESS_LIST=
       (LOAD_BALANCE=ON)
       (ADDRESS=(HOST=gsmhost1)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost2)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost3)(PORT=1523)(PROTOCOL=TCP))
      )
     (ADDRESS_LIST=
       (LOAD_BALANCE=ON)
       (ADDRESS=(HOST=gsmhost4)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost5)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost6)(PORT=1523)(PROTOCOL=TCP))
      )
     (ADDRESS_LIST=
       (LOAD_BALANCE=ON)
       (ADDRESS=(HOST=server1)(PORT=1521)(PROTOCOL=TCP))
      )
     (CONNECT_DATA=
      (SERVICE_NAME=sales_read_service.dbpoolora.oradbcloud)
      (REGION=WEST)))

Note:

In the case of an Oracle RAC enabled GDS database, the third address list contains the local Oracle RAC database's SCAN listeners.

Parent topic: Troubleshooting GDS Issues

Connecting to Catalog Databases When No Global Service Managers Are Running

Maintain access to the GDS catalog database for administration when all Global Service Managers are down.

In the event that all Global Service Managers (GSM) go down, and you need access for a database administration or maintenance activity on the catalog database, you can connect directly by using the catalog database’s local listener (or Oracle RAC SCAN listener). To do this, use a client connect descriptor/tnsnames entry that includes local listener addresses as the final failover option. This method bypasses GSM and allows DB maintenance even if no GSM processes are running.

Parent topic: Troubleshooting GDS Issues

Using SYS_CONTEXT Parameters in a GDS Environment

To identify GDS global-service sessions as a unified pool, query SYS_CONTEXT('USERENV', ...) for GDS-specific values.

For a session established using a connection to a global service, some parameters of namespace USERENV have values that are different from values set when connecting to a local service on the same database. The different values for a global service are set to make the database pool appear to clients as a single database with many instances.

When a client connects to a global service, GDS sets the following in the session context differently.

  • DB_UNIQUE_NAME and DB_DOMAIN are set to <gdspool_name>.<config_name>.

  • INSTANCE is set to a system-generated number <inst_num> that is unique within a GDS configuration.

  • INSTANCE_NAME is set to <gdspool_name>%<virtual_instance_num>.

  • SERVICE_NAME is set to <region_name>%<service_name>.

Parent topic: Troubleshooting GDS Issues

Troubleshooting GSM Issues

Identify and correct issues with the Global Service Manager (GSM) component of Global Data Services (GDS).

Parent topic: Troubleshooting Global Data Services

GSM-45054: GSM error or NET-40006: unable to start GSM

Learn how to identify and correct Global Service Manager (GSM) initialization issues.

The errors GSM-45054: GSM error and NET-40006: unable to start GSM commonly show up when a Global Service Manager (GSM) process is unable to initialize.

The common cause is that the GSM cannot reach the GDS (GSM) catalog. GSMs depend on the catalog at startup to read configuration (regions, pools, services, and so on). If the catalog database is down, then its listener is down, or the connect descriptor is wrong or unreachable, then GSM startup fails.

To correct this issue, verify that the GDS catalog database and its local listener are running

Parent topic: Troubleshooting GSM Issues

GDS Logs and Tracing

To effectively troubleshoot GDS issues, locate GSM logs and trace files and enable or collect diagnostic data for support.

Parent topic: Troubleshooting Global Data Services

Using Global Data Services Log and Trace Files

To diagnose GDS problems, use logs to pinpoint issues and enable tracing/logging to capture detailed runtime evidence.

If logging and tracing has been enabled for GDS components, you can find the exact location of a given global service manager's log and trace files using the status gsm command as shown in the following example. 

GDSCTL>status gsm
 
Alias                     MYGSM
Version                   19.1.0.0.1
Start Date                22-OCT-2022 14:05:11
Trace Level               support
Listener Log File         /scratch/oracle/diag/gsm/myhost/mygsm/alert/log.xml
Listener Trace File       /scratch/oracle/diag/gsm/myhost/mygsm/trace/ora_1829_
 47542149303936.trc
Endpoint summary          (ADDRESS=(HOST=myhost.com)(PORT=1571)(PROTOCOL=tcp))
GSMOCI Version            0.1.7
Mastership                N
Connected to GDS catalog  Y
Process Id                1833
Number of reconnections   0
Pending tasks.     Total  0
Tasks in  process. Total  0
Regional Mastership       TRUE
Total messages published  34261
Time Zone                 -07:00
Orphaned Buddy Regions:   None
GDS region                east
Network metrics:
   Region: euro RTT:34 Bandwidth:40

In this example myhost is the global service manager host name and mygsm is the name of the global service manager.

Although not strictly a GDS component, the LISTENER log file can be helpful resolving some issues. The text-based LISTENER log can be found in /scratch/oracle/diag/gsm/hostname/gsm_name/trace directory. The file is called alert_gsm*.log (for example, alert_gsm1.log).

If logging and tracing have not been enabled for GDS components, then follow the steps below to do so:

  1. Enable logging for GDSCTL. On the GSM host, edit the $ORACLE_HOME/network/admin/gsm.ora file and add the following: _GDSCTL=(log=ALL)

    This will log any commands run using GDSCTL in the following log file on the GSM host. $ORACLE_HOME/network/admin/GDSCTL.log

  2. Enable tracing for GSM processes.

    GDSCTL> set trace_level -gsm <gsm_name> SUPPORT

    This will generate trace files in DIAGNOSTICS_DEST for GSM. For example, $ORACLE_BASE/diag/gsm/<hostname>/<gsm-name>/trace

  3. Enable tracing on the catalog database.

    SQL> alter system set events '10798 trace name context forever, level 7';

  4. Enable tracing on the GDS pool databases.

    SQL> alter system set events '10798 trace name context forever, level 7';

To disable logging and tracing, follow the steps below:

  1. Turn off GSM tracing.

    GDSCTL> set trace_level -gsm <gsm-name> OFF

  2. Turn off GDSCTL command logging by editing the gsm.ora file and removing the following line:

    _GDSCTL=(log=ALL)

  3. Turn off tracing on the catalog database and GDS pool databases.

    SQL> alter system set events '10798 trace name context forever, level 0';

Parent topic: GDS Logs and Tracing

Advanced Global Data Services Troubleshooting

To troubleshoot GDS effectively, you must collect OPatch inventory, run scripted GDSCTL configuration and status commands, capture listener status, and gather logs/traces.

Use these steps to provide a good data pool for troubleshooting issues, and potentially to help Oracle Support assist you.

  1. Collecting patch information can be useful for code version-related issues:

    $ cd $ORACLE_HOME/OPatch $ ./opatch lsinventory -detail > lsinventory_info.txt

  2. Collect all GDS configuration data. Save the following commands in a script and run it:

    #!/bin/bash
# Stop script if any command fails
set -e
# Check if gdsctl is available in PATH
if ! command -v gdsctl &> /dev/null
then
    echo "gdsctl could not be found. Please make sure it is installed and available in PATH."
    exit 1
fi
echo "Starting gdsctl session..."
# Start the gdsctl session
gdsctl << EOF
# List GSMs, GSM status
echo "GDSCTL COMAND: config"
config
echo "GDSCTL COMAND: config gsm -gsm <gsm name>"
#config gsm -gsm <gsm name>
echo "GDSCTL COMAND: gsm status"
gsm status
# List services
echo "GDSCTL COMAND: services"
services
echo "GDSCTL COMAND: config service -service <service name>"
#config service -service <service name>
# List databases
echo "GDSCTL COMAND: databases"
databases
# Network
echo "GDSCTL COMAND: config vncr"
config vncr
# Validate config
echo "GDSCTL COMAND: validate"
validate
# End of gdsctl commands
EOF
echo "gdsctl session completed successfully."
# Exit the script
exit 0

  3. Collect GSM listener status.

    $ lsnrctl status <gsm-name>

If it is necessary to contact Oracle Support, then the data you have collected with these procedures will prove useful.

In addition, the following files would also be helpful:

From GSM Host: $ORACLE_HOME/network/admin

From GSM Host: GSM alert log file

From GSM Host: GDSCTL.log

From GSM Host: GSM trace files located in <DIAGNOSTIC_DEST>

From GSM Host: lsinventory, lsinventory_info.txt

From GDS Catalog Host: lsinventory, lsinventory_info.txt

Parent topic: GDS Logs and Tracing