9 Workarounds

9.1 Workaround to Resolve DB Site Replication Alarms

The following procedure resolves DB site replication alarms if encountered during the upgrade. This procedure restarts the inetrep process on the server that has a DB replication failure alarm. Database (DB) replication failure alarms may display during an Auto Site Upgrade or during an event that resets multiple servers in parallel. The DB on the child servers is not updated until resolved.
  1. Server CLI: Log in to the server.
    1. Use the SSH command (on UNIX systems – or putty if running on Windows) to log in to the active NOAM:

      ssh admusr@<server address>

      password: <enter password>

      Answer yes if you are asked to confirm the identity of the server.

  2. Server CLI: Check if the replication links are up.
    1. Run this command:

      irepstat

      Some of the B-C and C-C replications links may be down.

  3. Server CLI: Resolve replication issue
    1. Run this command:

      sudo pm.kill inetrep

      Note:

      Repeat this procedure on each affected server.

9.2 Workaround to Resolve Device Deployment Failed Alarm

This procedure resolves the device deployment failed alarm i.e. 10054. If this procedure fails, it is recommended to contact My Oracle Support (MOS) and ask for assistance.

9.2.1 NOAMP VIP GUI: Log In

  1. Open the web browser and enter the following URL:

    http://<Primary_NOAM_VIP_IP_Address>

  2. Log in to the NOAM GUI as the guiadmin user.

9.2.2 NOAMP VIP GUI: Identify Server(s) and Interface(s) with Alarm

Navigate to current alarm details and identify the server and interface where the 10054 - Device Deployment Failed alarm is displayed.
  1. Navigate to Alarms & Events, then View Active.
  2. Look for the 10054 alarm and make a list of the server(s) and interface(s).

9.2.3 NOAMP VIP GUI: Corrective Action for Alarm 10054

Interfaces like XMI and IMI are in locked state and do not allow editing as a corrective action. For XMI and IMI interfaces, first unlock the interface and for other interfaces skip steps 1 to 4 below.
  1. Navigate to Configuration, then Networking, and then Networks, select the respective “Network element” tab used for the server configuration.
  2. Click the Network Name row.
  3. Click Unlock. Click the checkbox to confirm it and click OK.
  4. To unlock the network for the particular device, navigate to Configuration, then Networking, and then Devices.
  5. Click the Server tab from the list in task 2 in this procedure.
  6. Select each interface row one by one for which alarm is showing and click Edit.
  7. Click OK.

    Note:

    Give some time to the system to auto correct the condition to clear the alarm. Once this step is done, lock the network back again which were unlocked above.

    For XMI and IMI interfaces, lock the interface back, for other interfaces skip 1 to 4 below.

  8. To lock the network for a specific device, navigate to Configuration, then Networking, and then Networks, select the respective Network element tab used for the server configuration.
  9. Click the Network Name row.
  10. Click Lock. Click the checkbox to confirm it and click OK.

9.3 Workaround to Resolve syscheck Error for CPU Failure

This procedure details the workaround to resolve syscheck error for CPU failure.

9.3.1 Log in to the server using CLI on which syscheck is failing

  1. Use the SSH command (on UNIX systems – or putty if running on windows) to log in to the server identified:

    ssh admusr@<SERVER_XMI>

    password: <enter password>

    Answer yes if you are asked to confirm the identity of the server.

9.3.2 Server CLI: Execute Workaround

  1. Edit the cpu config file.

    $ sudo vim /usr/TKLC/plat/lib/Syscheck/modules/system/cpu/config

  2. Comment out the all the text that reads: EXPECTED_CPUS= by putting # at the beginning of the line, for example:

    # EXPECTED_CPUS=2

  3. Save the cpu config file.
  4. Reconfig the syscheck by running these commands:

    sudo syscheck --unconfig

    sudo syscheck --reconfig

    sudo syscheck

    CPU related errors do not display.

9.4 Workaround to Resolve the Server HA Switchover Issue

The following procedure resolves the HA switchover issue. It restarts the cmha process on the server that has HA switchover issue.
  1. Server CLI: Log in to the server.
    1. Use the SSH command (on UNIX systems – or putty if running on Windows) to log in to the NOAM server which is experiencing the HA switchover issue:

      ssh admusr@<server address>

      password: <enter password>

      Answer yes if you are asked to confirm the identity of the server.

  2. Server CLI: Resolve HA switchover issue(s).
    1. Run this command:

      sudo pm.kill cmha

  3. Repeat this procedure on each affected server.

9.5 Workaround for Errors During Dual Image Upgrade

Fatal Errors

During any fatal errors the server will not be recoverable and must be rebuilt. Rebuild the server with the same DSR release of its mate server.

Failures

During typical failures the system can be recovered by running the following commands and then restart the server.
/var/TKLC/backout/diUpgrade --clearError
sudo /usr/TKLC/appworks/sbin/backout_restore
sudo init 6

Early Check Failure

In case the upgrade fails due to an early check, restart the server before retrying the upgrade.

9.6 Workaround to Resolve Failed Upgrade

Error: Upgrade failed from 9.x to 9.0.2 with error: "Server could not restart the application to complete the upgrade".

Figure 9-1 Failed Upgrade


Failed Upgrade

Following are the steps to resolve a failed upgrade:

  1. Set Max HA Role to Active for the failed server on HA screen.
  2. Restart the server on the server screen.

9.7 Workaround to Resolve False Alarms for DA MP and vSTP MP

The following critical alarms raised for DA MP (Diameter Agent Message Processor), vSTP MP in DSR or vSTP for combined deployments with multiple sites can be ignored:

  • 25500 - No DA MP Leader Detected
  • 70371 - No vSTP MP Leader Detected

If required, perform the following procedure to resolve the below alarms:

  1. On DSR SOAM, run the following commands:
    
iset -fcntl=off PmControl where "procTag='vstpoam'"
iset -fcntl=respawn PmControl where "procTag='dsroam'"
  2. On vSTP SOAM, run the following commands:
    
iset -fcntl=off PmControl where "procTag='dsroam'"
iset -fcntl=respawn PmControl where "procTag='vstpoam'"

Note:

In future, if the system is restarted, the alarms will not be restored.

9.8 Workaround to Resolve 31000 Critical Alarm Observed After Upgrade

The following critical alarm is observed after upgrade: C GN_TIMEOUT/WRN DNS lookups for finding NodeID took over 3000ms.

Perform the following procedure to resolve 31000 Critical alarm (cmsoapa#31000{S/W Fault}:

  1. Log in to the server where the alarm is raised.
  2. Run the following SSH command (on UNIX systems – or putty if running on Windows) to log in to the active NOAM:
    ssh admusr@<server address>
  3. Enter the password.
  4. If asked to confirm the server's identity, answer Yes.
  5. To check if the alarm exists, run the following command in the server CLI:
    ra.stat

    Output

    cmsoapa#31000{S/W Fault}
  *C GN_TIMEOUT/WRN DNS lookups for finding NodeID took over 3000ms
  6. Run the following command to restart the Named process on the server:
    sudo systemctl restart named

Note:

Repeat the similar procedure on each affected server.