9 Workarounds

9.1 Workaround to Resolve DB Site Replication Alarms

The following procedure resolves DB site replication alarms if encountered during the upgrade. This procedure restarts the inetrep process on the server that has a DB replication failure alarm. Database (DB) replication failure alarms may display during an Auto Site Upgrade or during an event that resets multiple servers in parallel. The DB on the child servers is not updated until resolved.
  1. Server CLI: Log in to the server.
    1. Use the SSH command (on UNIX systems – or putty if running on Windows) to log in to the active NOAM:

      ssh admusr@<server address>

      password: <enter password>

      Answer yes if you are asked to confirm the identity of the server.

  2. Server CLI: Check if the replication links are up.
    1. Run this command:

      irepstat

      Some of the B-C and C-C replications links may be down.

  3. Server CLI: Resolve replication issue
    1. Run this command:

      sudo pm.kill inetrep

      Note:

      Repeat this procedure on each affected server.

9.2 Workaround to Resolve Device Deployment Failed Alarm

This procedure resolves the device deployment failed alarm i.e. 10054. If this procedure fails, it is recommended to contact My Oracle Support (MOS) and ask for assistance.

9.2.1 NOAMP VIP GUI: Log In

  1. Open the web browser and enter the following URL:

    http://<Primary_NOAM_VIP_IP_Address>

  2. Log in to the NOAM GUI as the guiadmin user.

9.2.2 NOAMP VIP GUI: Identify Server(s) and Interface(s) with Alarm

Navigate to current alarm details and identify the server and interface where the 10054 - Device Deployment Failed alarm is displayed.
  1. Navigate to Alarms & Events, then View Active.
  2. Look for the 10054 alarm and make a list of the server(s) and interface(s).

9.2.3 NOAMP VIP GUI: Corrective Action for Alarm 10054

Interfaces like XMI and IMI are in locked state and do not allow editing as a corrective action. For XMI and IMI interfaces, first unlock the interface and for other interfaces skip steps 1 to 4 below.
  1. Navigate to Configuration, then Networking, and then Networks, select the respective “Network element” tab used for the server configuration.
  2. Click the Network Name row.
  3. Click Unlock. Click the checkbox to confirm it and click OK.
  4. To unlock the network for the particular device, navigate to Configuration, then Networking, and then Devices.
  5. Click the Server tab from the list in task 2 in this procedure.
  6. Select each interface row one by one for which alarm is showing and click Edit.
  7. Click OK.

    Note:

    Give some time to the system to auto correct the condition to clear the alarm. Once this step is done, lock the network back again which were unlocked above.

    For XMI and IMI interfaces, lock the interface back, for other interfaces skip 1 to 4 below.

  8. To lock the network for a specific device, navigate to Configuration, then Networking, and then Networks, select the respective Network element tab used for the server configuration.
  9. Click the Network Name row.
  10. Click Lock. Click the checkbox to confirm it and click OK.

9.3 Workaround to Resolve syscheck Error for CPU Failure

This procedure details the workaround to resolve syscheck error for CPU failure.

9.3.1 Log in to the server using CLI on which syscheck is failing

  1. Use the SSH command (on UNIX systems – or putty if running on windows) to log in to the server identified:

    ssh admusr@<SERVER_XMI>

    password: <enter password>

    Answer yes if you are asked to confirm the identity of the server.

9.3.2 Server CLI: Execute Workaround

  1. Edit the cpu config file.

    $ sudo vim /usr/TKLC/plat/lib/Syscheck/modules/system/cpu/config

  2. Comment out the all the text that reads: EXPECTED_CPUS= by putting # at the beginning of the line, for example:

    # EXPECTED_CPUS=2

  3. Save the cpu config file.
  4. Reconfig the syscheck by running these commands:

    sudo syscheck --unconfig

    sudo syscheck --reconfig

    sudo syscheck

    CPU related errors do not display.

9.4 Workaround to Resolve the Server HA Switchover Issue

The following procedure resolves the HA switchover issue. It restarts the cmha process on the server that has HA switchover issue.
  1. Server CLI: Log in to the server.
    1. Use the SSH command (on UNIX systems – or putty if running on Windows) to log in to the NOAM server which is experiencing the HA switchover issue:

      ssh admusr@<server address>

      password: <enter password>

      Answer yes if you are asked to confirm the identity of the server.

  2. Server CLI: Resolve HA switchover issue(s).
    1. Run this command:

      sudo pm.kill cmha

  3. Repeat this procedure on each affected server.

9.5 Workaround for Errors During Dual Image Upgrade

Fatal Errors

During any fatal errors the server will not be recoverable and must be rebuilt. Rebuild the server with the same DSR release of its mate server.

Failures

During typical failures the system can be recovered by running the following commands and then restart the server.
/var/TKLC/backout/diUpgrade --clearError
sudo /usr/TKLC/appworks/sbin/backout_restore
sudo init 6

Early Check Failure

In case the upgrade fails due to an early check, restart the server before retrying the upgrade.

9.6 Workaround to Resolve Failed Upgrade

Error: Upgrade failed from 9.x to 9.0.2 with error: "Server could not restart the application to complete the upgrade".

Following are the steps to resolve a failed upgrade:

Figure 9-1 Failed upgrade


Failed upgrade

  1. Set Max HA Role to Active for the failed server on HA screen.
  2. Restart the server on the server screen.

9.7 Workaround To Resolve Alarms while upgrading from build 8.5 to 9.0.2.0.0_99.13.0

Following are the alarms received while upgrading from the build 8.5 to 9.0.2.0.0_99.13.0:

  • 14152 MAJOR Failed to transfer file from remote host, see trace log for details.
  • 31214 MINOR Scheduled Process Fault.

Resolution

To resolve the alarm, run the following command 

sudo irem RecentAlarmEv.0 where "eventNumber=14152"
.

9.8 Workaround to resolve DSR process restart when multiple sites have both DSR and vSTP MP

Workaround for DSR process restart when the user is upgrading from 9.x to 9.1.

Resolution:

Run the following commands on DA-MP post upgrade and then restart the MP Server.

iset -fcntl=off PmControl where "procTag='traceoam'"
sudo init 6

9.9 Workaround to Resolve False Alarms for DA MP and vSTP MP

The following critical alarms raised for DA MP (Diameter Agent Message Processor), vSTP MP in DSR or vSTP for combined deployments with multiple sites can be ignored:

  • 25500 - No DA MP Leader Detected
  • 70371 - No vSTP MP Leader Detected

If required, perform the following procedure to resolve the below alarms:

  1. On DSR SOAM, run the following commands:
    
iset -fcntl=off PmControl where "procTag='vstpoam'"
iset -fcntl=respawn PmControl where "procTag='dsroam'"
  2. On vSTP SOAM, run the following commands:
    
iset -fcntl=off PmControl where "procTag='dsroam'"
iset -fcntl=respawn PmControl where "procTag='vstpoam'"

Note:

In future, if the system is restarted, the alarms will not be restored.

9.10 Workaround for Configuring GUI and MMI Using Label Format for FQDN/Realm

For the users who prefer to configure with label format instead label.label format for FQDN/Realm, perform the following procedure for GUI and MMI configuration changes:

Note:

By default code is compliant.
  1. Navigate to cd /usr/TKLC/dpi/prod/maint/scripts/ folder.
  2. Run the following command:
    sudo ./rfcRealmFqdn.sh
  3. Select an option from the list and proceed with the FQDN/Realm configuration:
    1. Non-Compliant: to allow RFC (Request for Comments) non-compliant label format.
    2. Compliant: to allow RFC compliant label format.
    3. Quit

    Note:

    To continue with the switchover, this manual step must be performed on every OAM server.