1. DSR Software Upgrade Guide
  2. Upgrading NOAM

5 Upgrading NOAM

The NOAM upgrade section is common to all topologies. This section must be completed before executing the site upgrade procedures. Procedures for the NOAM upgrade include steps for the upgrade of the Disaster Recovery NOAM (DR NOAM) servers also. If no DR NOAM is present in the customer deployment, then the DR NOAM-related steps can be safely ignored.

Global Provisioning is disabled before upgrading the NOAM servers. Provisioning activities at the NOAM and SOAM servers have certain limitations during the period where the NOAMs are upgraded and the sites are not yet upgraded.

5.1 NOAM Pre-Upgrade Checks and Backup

The procedures in this section perform health checks and backups to prepare the NOAM NE for upgrade. These procedures must be executed on the active NOAM.

Note:

  • These procedures may be executed outside of the maintenance window, but should be executed within 6 to 8 hours before NOAM Upgrade procedure.
  • If syscheck fails on any server during pre-upgrade checks or in early checks stating that cpu: FAILURE:: No record in alarm table for FAILURE!, see Workaround to Resolve syscheck Error for CPU Failure procedure.
  • Increase the Maximum Number of Open Files As the number of servers in the topology grows, so does the need for additional files to handle data merge to the NOAM. This procedure checks the number of files currently in use, and, if necessary, increases the maximum number of open files. See Increase Maximum Number of Open Files to increase the maximum number of open files.

5.1.1 NOAM Health Check for Source Release 8.0 and Later

This procedure is used to determine the health and status of the network and servers when the NOAM is on source release 8.0 or later. This procedure must be executed on the active NOAM.
5.1.1.1 Active NOAM VIP: Verify DSR ISO upgrade transfer to all servers
  1. Navigate to Status & Manage, then Files.
  2. Select the target release DSR ISO and click View ISO Deployment Report.
  3. Review the report to ensure the ISO is deployed to all servers in the topology.

    Sample report :Deployment report for DSR-8.6.0.7.0_96.34.0.iso:Deployed on 7/7 serversNO1: Deployed NO2: Deployed SO1: Deployed SO2: Deployed MP1: Deployed MP2: Deployed IPFE: Deployed

5.1.1.2 Active NOAM VIP: Export and Archive the Diameter Configuration Data
  1. Navigate to Diameter Common, then Export.
  2. Capture and archive the Diameter data by selecting the All option for the Export Application.
  3. Verify the requested data is exported by clicking Tasks at the top of the screen.
  4. Navigate to Status & Manage, then Files and download all the exported files to the client machine, or use the SCP utility to download the files from the active NOAM to the client machine.
5.1.1.3 Active NOAM VIP: Initiate NOAM Health Checks
This procedure runs the automated pre-upgrade health checks.
  1. Navigate to Administration, then Software Management, and then Upgrade.
  2. Select the active NOAM.
  3. Click Checkup.
  4. Under Health Check options, select the Pre Upgrade option.
  5. From the Upgrade ISO option, select the target release ISO.
  6. Click OK. Control returns to the Upgrade screen.
5.1.1.4 Active NOAM VIP: Monitor Health Check Progress for Completion
  1. Click the Tasks option to display the currently executing tasks. The Health Check task name appears as <NOServerGroup> PreUpgrade Health Check.
  2. Monitor the Health Check task until the Task State is completed. The Details column displays a hyperlink to the Health Check report.
  3. Click the hyperlink to download the Health Check report.
  4. Open the report and review the results.
5.1.1.5 Active NOAM VIP: Analyze Health Check Results
Analyze health check report for failures. If the Health Check report status is anything other than Pass, analyze the Health Check logs to determine if the upgrade can proceed.
  1. Navigate to Status & Manage, then Files.
  2. Select the AdvancedUpgrade_HealthCheck_<NOAM SG>_<TIMESTMP>.txt file and click View.
  3. Locate the log entries for the most recent health check.
  4. Review the log for failures.
  5. Analyze the failures and determine if it is safe to continue the upgrade. If necessary, contact My Oracle Support (MOS).

5.1.2 NOAM Pre-Upgrade Backup

This procedure takes a backup of the NOAM servers just prior to the upgrade.If this procedure fails, it is recommended to contact My Oracle Support (MOS) and ask for assistance.
5.1.2.1 Active NOAM VIP: Backup All Global Configuration Databases for NOAM
This procedure is required for disaster recovery.
  1. Navigate to Status & Manage, then Database to return to the Database Status screen.
  2. Click to highlight the active NOAM server and click Backup.
  3. Mark the Configuration checkbox.
  4. Select the desired compression type. Retain the default selection unless there is a specific reason or direction to change it.
  5. Enter Comments (Optional).
  6. Click OK.
5.1.2.2 Active NOAM VIP: Download/Save Database Files Backups for NOAM
This procedure is required for disaster recovery.
  1. Navigate to Status & Manage, then Files.
  2. Click on the active NOAM server tab.
  3. Select the configuration database backup file and click Download.
  4. If a confirmation window displays, click Save.
  5. If the Choose File screen displays, select a destination folder on the local workstation to store the backup file. Click Save.
  6. If a Download Complete confirmation displays, click Close.

5.2 Increase Maximum Number of Open Files

The following procedure increases the maximum number of files that can be opened for reading and writing. As the number of servers in the topology grows, so does the need for additional files to handle merging data to the NOAM. This procedure checks the number of files currently in use, and, if necessary, increases the maximum number of open files.

Note:

The following procedures are for one NOAM server. Repeat these procedures for other NOAM servers.
  1. Determine the Number of Files Currently Open on Active NOAM CLI
    1. Use an SSH client to connect to the active NOAM.

      ssh <NOAM XMI IP address>

      login as: admusr

      password: <enter password>

      Note:

      The static XMI IP address for each server should be available in Table 3-1.
    2. Enter the following command to retrieve the pid of idbsvc. The pid is highlighted in this sample output:

      $ ps -ef | grep -i idbsvc

      root 4369 idbsvc Up 03/01 13:03:28 1 idbsvc -M10 -ME204 -D40 -DE820 -W1 -S2

    3. The number of open files is output with the ‘lsof’ command. Use the highlighted value from sub-step 2 in place of XXXX in the lsof command.

      $ sudo lsof -p XXXX | wc -l

      1278

    4. Record the number of files currently open (the output of step 3).
    5. Enter the following command to retrieve the pid of tpdProvd. The pid is highlighted in this sample output:

      $ ps -ef | grep -i tpdProvd

      tpdProvd 347635 1 0 06:09 ? 00:00:11 /usr/TKLC/plat/bin/tpdProvd

    6. The number of open files is output with the ‘lsof’ command. Use the highlighted value from sub-step 4 in place of XXXX in the lsof command.

      $ sudo lsof -p XXXX | wc –l

      1280

    7. Record the number of files currently open (the output of step 5).
  2. Display the Maximum Number of Open Files for idbsvc
    1. Use the highlighted value from step 1, sub-step 2 in place of XXXX in the cat command.

      $ sudo cat /proc/XXXX/limits | grep -i open

      Max open files 32768 32768 files

      The output of the cat command displays the maximum number of files that can be open by the idbsvc process. Record both values here: Soft Limit (1st value) and Hard Limit (2nd value). This system has over 1024 open files, but its current ulimit for idbsvc is high enough during normal operation that the amount of open files does not pose a problem. However, when an attempt to upgrade another process (tpdProvd) updates idbsvc max # of open files to 1024, it causes the upgrade to fail. Display the maximum number of open files for tpdProvd.

    2. Use the highlighted value from step 1, sub-step 4 for tpdProvd in place of XXXX in the cat command.

      $ sudo cat /proc/XXXX/limits | grep -i open

      Max open files 1024 4096 files

      The output of the cat command displays the maximum number of files that can be open by the tpdProvd process. Record both values here: Soft Limit (1st value) and Hard Limit (2nd value).

      Caution:

      If the number of currently open files of idbsvc is less than the maximum allowed, this procedure is complete, that is, number of currently open files (used by idbsvc) is less than 1024.

      If the maximum number of open files value for tpdProvd is already set to 32768, this procedure is complete.

      Further steps are not required to be executed on this NOAM server. If maximum value is not already set, continue to the next step.

  3. Increase maximum number of open files in Acive NOAM CLI
    1. Using a text editor with sudo, edit the file Foot 1 to add these two lines just before the comment line in the file Foot 2 that reads Start the daemon:

      # increase open file limit

      limit nofile 32768 32768

    2. Save the file and close the editor.

      Note:

      Do not edit any other line in this file. You can back up the file, if required.
  4. Restart tpdProvd service using active NOAM CLI
    1. Enter this command to stop tpdProvd:
      $ sudo initctl stop tpdProvd
    2. Enter this command to restart tpdProvd:
      $ sudo initctl start tpdProvd

      Sample output:

      tpdProvd start/running, proceed 186743

  5. Recheck open file maximum limit using active NOAM CLI.
    1. Enter the following command to retrieve the pid of idbsvc. The pid is highlighted in this sample output:

      $ ps –ef | grep -i idbsvc

      root 8670 idbsvc Up 03/01 13:03:28 1 idbsvc -M10 -ME204 -D40 -DE820 -W1 -S2

    2. Use the highlighted value from sub-step 1in place of XXXX in the cat command.
      $ sudo cat /proc/XXXX/limits | grep -i open 
Max open files 32768 32768 files
    3. Verify the output of step 2 indicates that the maximum number of open files is 32768. If the value is not 32768, it is recommended to contact My Oracle Support.

5.3 Disable Global Provisioning

The following procedure disables provisioning on the NOAM and Configuration Updates on the Entire Network. This step ensures no changes are made to the database while the NOAMs are upgraded. Provisioning is re-enabled once the NOAM upgrade is complete. If this procedure fails, it is recommended to contact My Oracle Support (MOS) and ask for assistance.
  1. Log in to the active NOAM GUI using the VIP.
  2. Navigate to Status & Manage, then Database.
  3. Click Disable Provisioning.
  4. Confirm the operation by clicking OK on the screen.
  5. Verify the button text changes to Enable Provisioning; a yellow information box should also display at the top of the view screen that states: [Warning Code 002] – Global provisioning has been manually disabled. The active NOAM server has the following expected alarm: Alarm ID = 10008 (Provisioning Manually Disabled)

5.4 Verify NOAM Post Upgrade Status

This procedure determines the validity of the upgrade and the health and status of the network and servers. If this procedure fails, contact My Oracle Support (MOS) and ask for assistance.

5.4.1 Active NOAM VIP: Post-upgrade Health Checks

This procedure runs the automated post-upgrade health checks.
  1. Navigate to Administration, then Software Management, and then Upgrade.
  2. Select the active NOAM.

    Figure 5-1 Active NOAM VIP Upgrade State

    Active NOAM VIP Upgrade State
  3. Click Checkup.
  4. Under Health check options, select the Post Upgrade option.
  5. Click OK.

    Control returns to the Upgrade screen.

    Figure 5-2 NOAM Upgrade Screen

    NOAM Upgrade Screen

5.4.2 Active NOAM VIP: Monitor Health Check Progress

  1. Click the Tasks option to display the currently executing tasks. The Health Check task name appears as <NOServerGroup> PostUpgrade Health Check.
  2. Monitor the health check task until the Task State is completed. The Details column displays a hyperlink to the Health Check report.
  3. Click the hyperlink to download the Health Check report.
  4. Open the report and review the results.

    Figure 5-3 Active NOAM VIP Health Check Progress

    Active NOAM VIP Health Check Progress

5.4.3 Active NOAM VIP: Analyze Health Check Failures

If the Health Check report status is anything other than Pass, the Health Check logs can be analyzed to determine if the upgrade can proceed.
  1. Navigate to Status & Manage, then Files.
  2. Select the file named UpgradeHealthCheck.log and click View.
  3. Locate the log entries for the most recent health check.
  4. Review the log for failures.

    Analyze the failures and determine if it is safe to continue the upgrade. If necessary, contact My Oracle Support (MOS) for guidance.

5.5 Allow Provisioning

The following procedure enables Global Provisioning after the NOAM upgrade for the NOAM and DR NOAM servers.

Caution:

Any network-wide provisioning changes made at the NOAM site before the upgrade is accepted are lost if the upgrade is backed out.

5.5.1 Active NOAM VIP: Enable Global Provisioning and Configuration Updates on the Entire Network

  1. Log in to the active NOAM GUI using the VIP.
  2. Navigate to Status & Manage, then Database.
  3. Click Enable Provisioning.
  4. Confirm the operation by clicking OK on the screen.
  5. Verify the button text changes to Disable Provisioning.

    Note:

    After enabling provisioning at the NOAM, the SOAM GUI may display a banner indicating that global provisioning is disabled. This message can be ignored – global provisioning is enabled. This is a display issue only and is corrected when the SOAMs are upgraded.

5.5.2 Active NOAM VIP: Add New Network Element

Perform this procedure only if you want to add a new network element at this time.

If a new network element is to be added, start this procedure now. The addition of the new network element requires a separate maintenance window. The servers in the new network element must be installed with the same DSR release as that of the upgraded NOAMs. Follow the release specific installation procedures from DSR Cloud Installation Guide to install the software on the new servers and add the new network element under the existing NOAMs.

Skip the sections of the installation procedure related to installing and configuring the NOAMs. This adds a new DSR SOAM site under the existing NOAMs.

5.6 SNMP Configuration Update (Post NOAM Upgrade)

Apply SNMP workaround in following cases:

  • If SNMP is not configured in DSR.
  • If SNMP is already configured and SNMPv3 is selected as enabled version.

This can be checked by navigating to Administration > Remote Servers >SNMP Trapping screen using GUI session of NOAM server VIP IP address.


Footnote Legend

Footnote 1: /etc/init/tpdProvd.conf
Footnote 2: /etc/init/tpdProvd.conf