C.8 Primary-Standby Configuration Issues

Review these troubleshooting tips for commonly encountered primary-standby configuration related issues when working with Oracle Key Vault.

• Write Operations Fail in Restricted Mode
  When the primary or standby server is in read-only restricted mode, write operations like creation of key fails.
• Fast-Start Failover Suspended
  Oracle Key Vault displays a Fast-Start Failover suspended error.
• How to Verify Primary-Standby Status
  After the Primary-Standby nodes pairing, verify the status of the primary-standby nodes.

C.8.1 Write Operations Fail in Restricted Mode

When the primary or standby server is in read-only restricted mode, write operations like creation of key fails.

In Primary-Standby configuration, the status of the node on the Oracle Key Vault management console is, Read-Only Restricted.

Probable Cause

The recommended ports are not open between the primary and standby servers.

Solution

1. Verify the alerts on the Oracle Key Vault management console home page.
2. Log in to the Oracle Key Vault server through SSH as the support user. Switch to the root user and verify if the network connectivity between the primary and standby servers is successful or not.

3. Verify ports 1521, 1522, 7443, and 5696.

   curl -v telnet://ipaddress:port

4. If the command fails, fix the port issue and ensure that the ports are available and open.
5. Reboot the Oracle Key Vault server.
6. Verify if the issue is resolved.

C.8.2 Fast-Start Failover Suspended

Oracle Key Vault displays a Fast-Start Failover suspended error.

Example

On Oracle Key Vault Server, you may see the error in /var/log/messages or when you run the show_configuration command from dgmgrl .

ORA-16818: Fast-Start Failover suspended

Probable Cause

An ORA-16818: Fast-Start Failover suspended appears because of a fast start fail over operation failure.

Solution

1. Gracefully shut down the primary server in a controlled manner using the Power Off option instead of manually turning off the computer. When you power off the Primary Server, you cannot perform a fast start failover function, resulting in an ORA-16818 error.

2. In a graceful shutdown operation, the primary server's failover status goes into a suspended state with the standby waiting indefinitely for the primary server to be available. This is the expected behavior for a Fast-Start Failover (FSFO) operation.

   Note:

   Avoid a split-brain scenario, as defined by Oracle Data Guard.
3. By design, an FSFO operation error occurs only when the primary server shuts down unexpectedly.
4. If you perform a SHUTDOWN IMMEDIATE or SHUTDOWN NORMAL command in SQL*Plus, then the FSFO does not occur because the database shuts down gracefully.

C.8.3 How to Verify Primary-Standby Status

After the Primary-Standby nodes pairing, verify the status of the primary-standby nodes.

1. Check the Primary-Standby nodes status from the Oracle Key Vault management console.
   1. Log in to the Oracle Key Vault management console.
   2. On the Primary-Standby page, go to System, Settings.
   3. Check the nodes status.
2. Check the status from Primary-Standby nodes.
   1. Log in to the Primary-Standby nodes as a support user using SSH, and then switch to the root user.
   2. Run these commands to check the status.

      /usr/local/dbfw/bin/setup_ha.rb ––status
      /usr/local/dbfw/bin/setup_ha.rb ––dg_status
      sudo -u oracle /usr/local/dbfw/bin/setup_ha.rb --ha_role

      Sample Output

      $ /usr/local/dbfw/bin/setup_ha.rb --status
      DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production
      
      Copyright (c) 2000, 2013, Oracle. All rights reserved.
      
      Welcome to DGMGRL, type "help" for information.
      Connected as SYSDG.
      DGMGRL>
      Configuration - DBFWDB
      
      Protection Mode: MaxAvailability
      Members:
      DBFWDB_HA2 - Primary database
      DBFWDB_HA1 - (*) Physical standby database
      
      (*) Fast-Start Failover target
      
      Properties:
      FastStartFailoverThreshold = '120'
      OperationTimeout = '30'
      TraceLevel = 'USER'
      FastStartFailoverLagLimit = '30'
      CommunicationTimeout = '60'
      ObserverReconnect = '30'
      FastStartFailoverAutoReinstate = 'TRUE'
      FastStartFailoverPmyShutdown = 'FALSE'
      BystandersFollowRoleChange = 'ALL'
      ObserverOverride = 'FALSE'
      ExternalDestination1 = ''
      ExternalDestination2 = ''
      PrimaryLostWriteAction = 'CONTINUE'
      
      Fast-Start Failover: ENABLED
      
      Threshold: 120 seconds
      Target: DBFWDB_HA1
      Observer: *******
      Lag Limit: 30 seconds (not in use)
      Shutdown Primary: FALSE
      Auto-reinstate: TRUE
      Observer Reconnect: 30 seconds
      Observer Override: FALSE
      
      Configuration Status:
      SUCCESS
      
      DGMGRL>
      Database - DBFWDB_HA1
      
      Role: PHYSICAL STANDBY
      Intended State: APPLY-ON
      Transport Lag: 0 seconds (computed 1 second ago)
      Apply Lag: 0 seconds (computed 1 second ago)
      Average Apply Rate: 2.00 KByte/s
      Active Apply Rate: (unknown)
      Maximum Apply Rate: (unknown)
      Real Time Query: ON
      Instance(s):
      dbfwdb
      
      Properties:
      DGConnectIdentifier = 'DBFWDB_HA1'
      ObserverConnectIdentifier = ''
      LogXptMode = 'sync'
      RedoRoutes = ''
      DelayMins = '0'
      Binding = 'optional'
      MaxFailure = '0'
      MaxConnections = '1'
      ReopenSecs = '10'
      NetTimeout = '30'
      RedoCompression = 'DISABLE'
      LogShipping = 'ON'
      PreferredApplyInstance = ''
      ApplyInstanceTimeout = '0'
      ApplyLagThreshold = '0'
      TransportLagThreshold = '0'
      TransportDisconnectedThreshold = '30'
      ApplyParallel = 'AUTO'
      StandbyFileManagement = 'AUTO'
      ArchiveLagTarget = '0'
      LogArchiveMaxProcesses = '4'
      LogArchiveMinSucceedDest = '1'
      DbFileNameConvert = ''
      LogFileNameConvert = ''
      FastStartFailoverTarget = 'DBFWDB_HA2'
      InconsistentProperties = '(monitor)'
      InconsistentLogXptProps = '(monitor)'
      SendQEntries = '(monitor)'
      LogXptStatus = '(monitor)'
      RecvQEntries = '(monitor)'
      StaticConnectIdentifier = 'DBFWDB_HA1_DGMGRL'
      StandbyArchiveLocation = 'USE_DB_RECOVERY_FILE_DEST'
      AlternateLocation = ''
      LogArchiveTrace = '0'
      LogArchiveFormat = '%t_%s_%r.dbf'
      TopWaitEvents = '(monitor)'
      
      Database Status:
      SUCCESS
      
      DGMGRL>
      Database - DBFWDB_HA2
      
      Role: PRIMARY
      Intended State: TRANSPORT-ON
      Instance(s):
      dbfwdb
      
      Properties:
      DGConnectIdentifier = 'DBFWDB_HA2'
      ObserverConnectIdentifier = ''
      LogXptMode = 'sync'
      RedoRoutes = ''
      DelayMins = '0'
      Binding = 'optional'
      MaxFailure = '0'
      MaxConnections = '1'
      ReopenSecs = '10'
      NetTimeout = '30'
      RedoCompression = 'DISABLE'
      LogShipping = 'ON'
      PreferredApplyInstance = ''
      ApplyInstanceTimeout = '0'
      ApplyLagThreshold = '0'
      TransportLagThreshold = '0'
      TransportDisconnectedThreshold = '30'
      ApplyParallel = 'AUTO'
      StandbyFileManagement = 'AUTO'
      ArchiveLagTarget = '0'
      LogArchiveMaxProcesses = '4'
      LogArchiveMinSucceedDest = '1'
      DbFileNameConvert = ''
      LogFileNameConvert = ''
      FastStartFailoverTarget = 'DBFWDB_HA1'
      InconsistentProperties = '(monitor)'
      InconsistentLogXptProps = '(monitor)'
      SendQEntries = '(monitor)'
      LogXptStatus = '(monitor)'
      RecvQEntries = '(monitor)'
      StaticConnectIdentifier = 'DBFWDB_HA2_DGMGRL'
      StandbyArchiveLocation = 'USE_DB_RECOVERY_FILE_DEST'
      AlternateLocation = ''
      LogArchiveTrace = '0'
      LogArchiveFormat = '%t_%s_%r.dbf'
      TopWaitEvents = '(monitor)'
      
      Database Status:
      SUCCESS
      
      DGMGRL>

   3. Run these commands to check Oracle Data Guard status. Make sure the Observer is running.

      su oracle
      dgmgrl /
      show configuration;
      show database ‘DBFWDB_HA1’;
      show database ‘DBFWDB_HA2’;

      Sample Output

      $ dgmgrl /
      DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production
      
      Copyright (c) 2000, 2013, Oracle. All rights reserved.
      
      Welcome to DGMGRL, type "help" for information.
      Connected as SYSDG.
      DGMGRL> show configuration;
      
      Configuration - DBFWDB
      
      Protection Mode: MaxAvailability
      Members:
      DBFWDB_HA2 - Primary database
      DBFWDB_HA1 - (*) Physical standby database
      
      Fast-Start Failover: ENABLED
      
      Configuration Status:
      SUCCESS (status updated 25 seconds ago)
      
      DGMGRL> show database 'DBFWDB_HA1';
      
      Database - DBFWDB_HA1
      
      Role: PHYSICAL STANDBY
      Intended State: APPLY-ON
      Transport Lag: 0 seconds (computed 1 second ago)
      Apply Lag: 0 seconds (computed 1 second ago)
      Average Apply Rate: 2.00 KByte/s
      Real Time Query: ON
      Instance(s):
      dbfwdb
      
      Database Status:
      SUCCESS
      
      DGMGRL> show database 'DBFWDB_HA2';
      
      Database - DBFWDB_HA2
      
      Role: PRIMARY
      Intended State: TRANSPORT-ON
      Instance(s):
      dbfwdb
      
      Database Status:
      SUCCESS