4 Troubleshooting Global Data Services

The following topics provide information about tools and solutions for troubleshooting the GDS issues you might encounter.

Troubleshooting Oracle Error Codes

This section contains information for troubleshooting specific Oracle error messages you might encounter, such as:

Parent topic: Troubleshooting Global Data Services

ORA-01045: user GSMADMIN_INTERNAL lacks CREATE SESSION privilege; logon denied

The user GSMADMIN_INTERNAL is an internal only user, it should never be unlocked or used for any database login. No direct modifications should be made on the Global Data Services schema objects unless directed by Oracle Technical Support.

Parent topic: Troubleshooting Oracle Error Codes

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

The global service may be down. Verify that the pool databases are up and the service is started.

The global service may be disabled. Ensure that the pool databases are up and the service is enabled and started.

The GDS pool database may be down. Ensure that the GDS pool databases are up and the service is enabled and started.

Parent topic: Troubleshooting Oracle Error Codes

ORA-12516: TNS:listener could not find available handler with matching protocol stack

The GDS pool database's local listener may be down. Ensure that the GDS pool database local listener is running.

Parent topic: Troubleshooting Oracle Error Codes

ORA-12541: TNS:no listener

All global service managers may be down. Verify that the global service managers are running.

Parent topic: Troubleshooting Oracle Error Codes

GSM-45034: Connection to GDS catalog is not established

The GDS catalog database or its listener may be down. Verify that the GDS catalog database and its local listener are running.

Parent topic: Troubleshooting Oracle Error Codes

GSM-45054: GSM error or NET-40006: unable to start GSM

The GDS catalog database or its listener may be down. Verify that the GDS catalog database and its local listener are running.

Parent topic: Troubleshooting Oracle Error Codes

Common GDS Issues

To solve some general issues you might encounter with Global Data Services (GDS), review these issues.

Parent topic: Troubleshooting Global Data Services

Connecting to GDS Configuration Databases When No Global Service Managers Are Running

Troubleshoot GSM by using three address lists: regional GSM, buddy GSM, and then the local/SCAN; fall back only if GSM is down.

When troubleshooting GSM connectivity, check the configuration and connect using your region’s GSM listeners first. This first list should be exclusively regional global service manager listeners. The second list contains global service manager listeners of the buddy region. The third list should be your local or SCAN listeners. With these three lists, connections use GSM when it’s available and only fall back to local listeners if all GSM listeners are down.

You can always connect through a global service manager while it is up, and only fail over to local listeners when all global service manager listeners are down.

Template

Use this template as a model. 

(DESCRIPTION=
  (FAILOVER=on)
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
   )
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
    (ADDRESS=(global_protocol_address_information))
   )
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(local_protocol_address_information))
    (ADDRESS=(local_protocol_address_information))
   )
  (CONNECT_DATA=
   (SERVICE_NAME=global_service_name)
   (REGION=region_name)))

Note that this example uses three address lists, where gsmhost1, gsmhost2 and gsmhost3 are the regional GSMs. gsmhost4, gsmhost5 and gsmhost6 are buddy GSMs. Finally, server1 has the local listener. 

(DESCRIPTION=
     (FAILOVER=on)
     (ADDRESS_LIST=
       (LOAD_BALANCE=ON)
       (ADDRESS=(HOST=gsmhost1)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost2)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost3)(PORT=1523)(PROTOCOL=TCP))
      )
     (ADDRESS_LIST=
       (LOAD_BALANCE=ON)
       (ADDRESS=(HOST=gsmhost4)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost5)(PORT=1523)(PROTOCOL=TCP))
       (ADDRESS=(HOST=gsmhost6)(PORT=1523)(PROTOCOL=TCP))
      )
     (ADDRESS_LIST=
       (LOAD_BALANCE=ON)
       (ADDRESS=(HOST=server1)(PORT=1521)(PROTOCOL=TCP))
      )
     (CONNECT_DATA=
      (SERVICE_NAME=sales_read_service.dbpoolora.oradbcloud)
      (REGION=WEST)))

Note:

In the case of an Oracle RAC enabled GDS database, the third address list contains the local Oracle RAC database's SCAN listeners.

Parent topic: Common GDS Issues

Connecting to Catalog Databases When No Global Service Managers Are Running

Maintain access to the GDS catalog database for administration when all Global Service Managers are down.

In the event that all Global Service Managers (GSM) go down, and you need access for a database administration or maintenance activity on the catalog database, you can connect directly by using the catalog database’s local listener (or Oracle RAC SCAN listener). To do this, use a client connect descriptor/tnsnames entry that includes local listener addresses as the final failover option. This method bypasses GSM and allows DB maintenance even if no GSM processes are running.

Parent topic: Common GDS Issues

GSM-40148: Database task failed with ORA-12170

How to resolve GSM-40148: Database task failed with ORA-12170 TNS connect timeout occurred when adding a broker in GSM.

The error message "GSM-40148: Database task failed with ORA-12170" with ORA-12170 TNS connect timeout typically occurs due to a configuration issue with the GDS/GSM instance (Global Service Manager)).

  1. Ensure that the GSM host is available from the database host. To do this, run the following on each new database from the database host:

    SQL> show parameter "_gsm"

    For each of the GSM TNS addrssses in that list, run:

    $ tnsping tns_address

    where tns_address is the TNS connect address.

    If any address does not respond, then peform the following checks:

    • Ensure that the GSM hostname is known to the database host (ping the host)
    • Ensure that the firewall from the database to the GSM host has been opened for the GSM listener port (usually that port is 1572, but check the port number in the TNS address).

    If either of these checks turn up issues, then address that problem with your networking team.

  2. Check that all of the database hosts are available from the GSM host. To do this, select the connect string for each standby database. On the primary database, enter the following:

    SQL> select connect_identifier from v$dg_broker_config;

    Next, on each GSM host, run the following checks:

    • Run tnsping on the primary database TNS that you used to add the broker.
    • Run tnsping on each of the addresses from v$dg_broker_config.

Note:

Be aware that you may have used a TNS alias in any of these addresses. If you used a TNS alias, then you must use that same TNS alias in the TNS ping.

Be aware that all TNS aliases must be in the tnsnames.ora file on all of the GSM hosts.

Parent topic: Common GDS Issues

Obtaining the Running Status of Global Data Services Components

The status command can be used to obtain the running status of the GDS components. 

GDSCTL>status gsm
 
GDSCTL>status service
 
GDSCTL>status database

Parent topic: Troubleshooting Global Data Services

Viewing Static Configuration Information for Global Data Services Components

The gdsctl config command can be used to obtain the static configuration information of various GDS components. 

GDSCTL>config

GDSCTL>config gsm

GDSCTL>config region

GDSCTL>config gdspool

GDSCTL>config database

GDSCTL>config service

GDSCTL>config invitednode

Parent topic: Troubleshooting Global Data Services

Enabling and Disabling Tracing on a Global Service Manager

You can enable tracing using the set trace_level command. 

GDSCTL>set trace_level -gsm gsm_name SUPPORT

The SUPPORT option provides trace with troubleshooting information for Oracle Support Services. The other options are ADMIN and USER.

To disable tracing:

GDSCTL>set trace_level -gsm gsm_name OFF

Parent topic: Troubleshooting Global Data Services

Using Global Service Manager Log and Trace Files

The exact location of a given global service manager's log and trace files can be obtained using the status gsm command as shown in the following example. 

GDSCTL>status gsm
 
Alias                     MYGSM
Version                   12.1.0.0.2
Start Date                13-OCT-2012 12:20:16
Trace Level               support
Listener Log File         /scratch/oracle/diag/gsm/myhost/mygsm/alert/log.xml
Listener Trace File       /scratch/oracle/diag/gsm/myhost/mygsm/trace/ora_1829_
 47542149303936.trc
Endpoint summary          (ADDRESS=(HOST=myhost.com)(PORT=1571)(PROTOCOL=tcp))
GSMOCI Version            0.1.7
Mastership                N
Connected to GDS catalog  Y
Process Id                1833
Number of reconnections   0
Pending tasks.     Total  0
Tasks in  process. Total  0
Regional Mastership       TRUE
Total messages published  34261
Time Zone                 -07:00
Orphaned Buddy Regions:   None
GDS region                east
Network metrics:
   Region: euro RTT:34 Bandwidth:40

In this example myhost is the global service manager host name and mygsm is the name of the global service manager.

The text based listener log can be found in /scratch/oracle/diag/gsm/hostname/gsm_name/trace directory. The file is called alert_gsm*.log (for example, alert_gsm1.log)

Parent topic: Troubleshooting Global Data Services

Using SYS_CONTEXT Parameters in a GDS Environment

For a session established using a connection to a global service, some parameters of namespace USERENV have values that are different from values set when connecting to a local service on the same database. The different values for a global service are set to make the database pool appear to clients as a single database with many instances. This was done to provide backward compatibility with pre-12c clients which expect multiple instances of a service to exist only on an Oracle RAC database.

When a client connects to a global service, GDS sets the following in the session context differently.

  • DB_UNIQUE_NAME and DB_DOMAIN are set to <gdspool_name>.<config_name>

  • INSTANCE is set to a system generated number <inst_num> which is unique within a GDS configuration

  • INSTANCE_NAME is set to <gdspool_name>%<virtual_instance_num>

  • SERVICE_NAME is set to <region_name>%<service_name>

Parent topic: Troubleshooting Global Data Services