1. DSR Software Upgrade Guide
  2. Site Upgrade Execution

6 Site Upgrade Execution

This section contains the procedures for upgrading an entire site, starting with the preupgrade activities, upgrading the SOAMs and C-level servers, and finishing with verifying the upgrade. To maximize the Maintenance Window usage, the procedures in this section make full use of the parallel upgrade capabilities of the DSR, while ensuring traffic continuity and redundancy to the maximum extent possible.

The Automated Site Upgrade procedures are explained in Automated Site Upgrade. Use the procedures in this section if the Automated Site Upgrade was recommended in Site Upgrade Methodology. For more details, see Rearrange Automated Site Upgrade Cycles.

The manual site upgrade procedures are in Overview of Automated/Manual Server Group Upgrade. Use the procedures in this section if the manual upgrade was recommended in Site Upgrade Methodology.

Note:

Refer to Automated Site Upgrade for details and limitations/solutions while planning for upgrade cycles.

6.1 Site Preupgrade Activities

This section contains the procedures for site upgrade planning, preupgrade backups, health checks, and disabling site provisioning. Following are the procedures to be executed for the site upgrade, along with the estimated time to complete each step. Use the following list for determining the order in which the procedures are to be executed.
  • Site Preupgrade Backups
  • Site Preupgrade Health Check for Release 8.0 and Later
  • Site Upgrade Options Check
  • Disable Site Provisioning
  • Site Upgrade Pre-Checks
  • Automated Site Upgrade
  • Rearrangement of upgrade cycles for Automated Site Upgrade
  • Rearrangement of upgrade cycles for Automated Site Upgrade

6.1.1 Site Preupgrade Backups

This procedure is non-intrusive and is used to perform a backup of all servers associated with the SOAM Site(s) being upgraded. It is used to conduct a full backup of the Configuration database and run environment on site being upgraded so that each server has the latest data to perform a backout, if necessary. It is recommended that this procedure be executed no earlier than 36 hours before the upgrade starts.

Since this backup is to be used in the event of disaster recovery, any site configuration changes made after this backup should be recorded and re-entered after the disaster recovery. This is an alternate procedure that can be used to backup a site using the command line. It should only be used as per the directions from My Oracle Support.

6.1.1.1 Active SOAM VIP: Back Up Site Configuration Data
This procedure is required for disaster recovery.
  1. Log in to the SOAM GUI using the VIP.
  2. Navigate to Status & Manage, then Database to return to the Database Status screen.
  3. Click to highlight the Active SOAM server, and click Backup.

    Note:

    Backup is only enabled when the active server is selected.
  4. Mark the Configuration checkbox.
  5. Select the desired compression type. Retain the default selection unless there is a specific reason or direction to change it.
  6. Enter Comments (optional).
  7. Click OK.

    Note:

    The active SOAM can be determined by navigating to Status & Manage, then HA and identifying which server is currently assigned the VIP in the Active VIPs field. The server having VIP assigned is Active.
6.1.1.2 Active SOAM VIP: Download and Save Database Backup Files
This procedure is required for disaster recovery.
  1. Navigate to Status & Manage, then Files.
  2. Click the active SOAM server tab.
  3. Select the configuration database backup file and click Download.
  4. If a confirmation window appears, click Save.
  5. If the Choose File window displays, select a destination folder on the local workstation to store the backup file. Click Save.
  6. If a download complete confirmation displays, click Close.
6.1.1.3 Active NOAM VIP: Upgrade and Back Up DB Run Environment for Site
  1. Log in to the NOAM GUI using the VIP.
  2. Navigate to Administration, then Software Management, and then Upgrade.
  3. Click Backup All.
6.1.1.4 Active NOAM VIP: Set Backup Parameters
The Upgrade Backup All screen displays the various network elements and identifies which servers are ready for backup.
  1. In the Action column, mark the Backup checkbox for each network element.
  2. Verify the NOAM server group checkbox is not marked.

    Note:

    Backing up the NOAM servers at this point overwrites the preupgrade backup files needed for backing out the target release. Do not back up the NOAM servers.
  3. In the Full Backup Options section, verify the Exclude option is selected.
  4. Click OK.

    This initiates a full backup on each eligible server.

6.1.1.5 Active NOAM VIP: Monitor Tasks for Backup Completion
  1. From the Upgrade screen, click the Tasks option.
  2. Monitor the progress of the backups until the network element(s) selected in step four are complete.
6.1.1.6 Active NOAM VIP: Verify Backup Files are Present on Each Server
  1. Log in to the active NOAM or SOAM GUI.
  2. Navigate to Status & Manage, then Files.
  3. Click each server tab.
  4. For each server, verify the following 2 files have been created:
    Backup.DSR.<server_name>.FullDBParts.NETWORK_OAMP.<time_stamp>.UPG.tar.bz2Backup.DSR.<server_name>.FullRunEnv.NETWORK_OAMP.<time_stamp>.UPG.tar.bz2
  5. Repeat sub-steps one through four for each site being upgraded.

6.1.2 Site Preupgrade Health Check for Release 8.0 and Later

This section provides procedures to verify the health of the SOAM site prior to upgrade. This is the primary procedure to be executed when the active SOAM is on release 8.0 and later. Alternate release-specific procedures are also provided, to be used as directed. The procedure is non-intrusive and performs a health check of the site prior to upgrading.

Note:

If syscheck fails on any server during preupgrade checks or in early checks stating that cpu: FAILURE:: No record in alarm table for FAILURE!, see the Workaround to Resolve syscheck Error for CPU Failure.
  1. Run Site Health Checks (Phase 1)
    1. Select the SOAM on which health checks are run.
    2. Navigate to Administration, then Software Management, and then Upgrade.
    3. Select the tab of the site to be upgraded.
    4. Select the SOAM server group link.
    5. Select the active SOAM.
    6. Click Checkup.
  2. Run site health checks (Phase 2)
    1. Click Checkup.
    2. In the Health Check options section, select the Pre Upgrade option.
    3. Use the Upgrade ISO option to select the target release ISO.
    4. Click OK to initiate the health check.
      Control returns to the Upgrade Administration screen.
  3. Monitor Health Check Progress for Completion
    1. Click the Tasks option to display the currently executing tasks. The Health Check task name appears as <SOServerGroup> PreUpgrade Health Check.
    2. Monitor the Health Check task until the Task State is completed.
      The Details column displays a hyperlink to the Health Check report.
    3. Click the hyperlink to download the Health Check report.
    4. Open the report and review the results.
  4. Analyze Any Health Check Failures
    If the Health Check report status is anything other than Pass, the Health Check logs must be analyzed to determine if the upgrade can proceed. The Health Check log is located in the File Management area of the active SOAM. Select the active SOAM tab to see the Health Check log.
    1. Navigate to Status & Manage, then Files.
    2. Select the active SOAM tab.
    3. Select the UpgradeHealthCheck.log file and click View.
    4. Locate the log entries for the most recent health check.
    5. Review the log for failures.

      Analyze the failures and determine if it is safe to continue the upgrade. If necessary, contact My Oracle Support (MOS) for guidance.

      If the health check log contains the Unable to perform Health Check on <Active SOAM hostname> message, perform an alternate health check procedure. If the active SOAM release is 8.0 or 8.1, perform Automated SOAM Upgrade (Active/Standby).

  5. Export and Archive the Diameter Configuration Data on Active SOAM GUI
    1. Navigate to Diameter Common, then Export.
    2. Capture and archive the Diameter data by selecting the All option for the Export Application.
    3. Click OK.
    4. Verify the requested data is exported by clicking Tasks at the top of the screen.
    5. Click File Management to view the files available for download. Download all of the exported files to the client machine, or use the SCP utility to download the files from the active NOAM to the client machine.
      Capture data for each configured SOAM site.

6.1.3 Check Site Upgrade Options

Automated Site Upgrade provides user-configurable options that control certain upgrade behaviors. To find these options, navigate to SOAM’s Administration, then General Options screen and are described in detail in Site Upgrade Options. Before initiating a site upgrade, review these options to verify the current settings are correct, or to modify the settings to meet customer requirements/preferences. This procedure is applicable only to Auto Site Upgrade. The options have no effect on manual upgrades or Automated Server Group upgrades.
  1. Log in to the active SOAM GUI.
  2. Navigate to Administration, then General Options.
  3. Scroll down to the Site Upgrade Bulk Availability option.
  4. Review the existing value of this option and determine if changes are needed. If the option is changed, click OK to save the change.
  5. Scroll down to the Site Upgrade SOAM Method option.
  6. Review the existing value of this option and determine if changes are needed. If the option is changed, click OK to save the change.

6.1.4 Disable Site Provisioning

This procedure disables Site Provisioning in preparation for upgrading the site.

Note:

This procedure may only be performed in the maintenance window immediately before the start of the SOAM site upgrade.

  1. Log in to the SOAM GUI of the site to be upgraded.
  2. Navigate to Status & Manage, then Database.
  3. Click Disable Provisioning.
  4. Confirm the operation by clicking OK on the screen.
  5. Verify the button text changes to Enable Provisioning. A yellow information box also displays at the top of the view screen that states:

    [Warning Code 004] – Site provisioning has been manually disabled.

    The active SOAM server has the following expected alarm:

    Alarm ID = 10008 (Provisioning Manually Disabled)

    Note:

    Repeat this procedure for each configured SOAM site to be upgraded.

6.2 Site Upgrade Pre-Checks

This procedure verifies that the system is prepared for Automated Site Upgrade. It verifies the traffic status and verifies that Site Provisioning is disabled, in preparation for upgrading the site.
The following procedures must be completed before the start of automated site upgrade:
  • Site Preupgrade Backups
  • Site Preupgrade Health Check for Release 8.0 and Later
  • Check Site Upgrade Options
  • Disable Site Provisioning
  • Site Upgrade Pre-Checks

Also read Automated Site Upgrade section for details. Upgrade cycles are created when using the Automated Site Upgrade. Limitations in Limitations of Automated Server Group and Automated Site Upgrade for Automated Site Upgrade can be solved by rearranging/adding the upgrade cycles. If the user does not want to create a custom upgrade plan by rearranging/adding cycles, then manually upgrade using Overview of Automated/Manual Server Group Upgrade.

  1. View KPIs to Verify Traffic Status in active SOAM VIP
    1. Log in to the active SOAM GUI using the VIP.
    2. Navigate to Status & Manage, then KPIs.
    3. Inspect KPI reports to verify traffic is at the expected condition.
  2. Verify site provisioning is disabled
    1. Verify that Site Provisioning was properly disabled in Disable Site Provisioning section.
    2. In the GUI status bar, where it says Connected using …, check for the message Site Provisioning disabled.

      Note:

      If the message is present, continue with the next procedure; otherwise, follow Disable Site Provisioning procedure.

6.2.1 Initiate Automated Site Upgrade

This procedure upgrades an entire site using the Automated Site Upgrade option. If this procedure fails, it is recommended to contact My Oracle Support and ask for assistance.
  1. Review Site Upgrade Plan and Site Readiness
    Review the site upgrade plan created in Upgrade Maintenance Windows. This step verifies that the servers and server groups to be upgraded are in the appropriate state.
    1. Log in to the NOAM GUI using the VIP.
    2. Select Administration, then Software Management, and then Upgrade.
    3. Select the SOAM tab of the site to be upgraded.
    4. Verify the Entire Site link is selected.

      The Entire Site screen provides a summary of the server states and upgrade readiness. More detailed server status is available by selecting a specific server group link.

      The Site Upgrade option can be used to upgrade an entire site, or a subset of site elements. The servers within the site may be in various states of readiness, including Accept or Reject, Ready, Backup Needed, Failed, or Not Ready. Only the servers in the Ready or Failed state are upgrade eligible

  2. Initiate Site Upgrade
    1. Verify no server groups are selected on the upgrade administration screen. The Site Upgrade button is not available if a server group is selected.
    2. Click Site Upgrade.
    3. Review the upgrade plan as presented on the Site Initiate screen.

      Note:

      • This plan represents an approximation of how the servers are upgraded. Due to the dynamic nature of the upgrade, some servers (typically only C-level) may be upgraded in a different cycle than displayed here.
      • Review the upgrade plan again and ensure all concerns noted in Table 6 have been addressed with the upgrade plan shown on the screen.

      If you need to rearrange the upgrade cycle, see Rearrange Automated Site Upgrade Cycles. Otherwise, continue with the next step. There are some limitations with upgrading the DC server during its server group upgrade, which are upgraded in a group of servers. This is applicable for all upgrade options, for example DA-MP, vSTP MP(s). Hence, make sure the DC server is not upgraded in first upgrade cycle of the C-Level servers in its server group. Identify the DC server. If the DC server displays by default in the first upgrade cycle of its server group, then rearrange the upgrade cycles by referring to Rearrange Automated Site Upgrade Cycles section such that the DC server is not upgraded in the first upgrade cycle of its server group. vSTP MPs should be divided in cycles to avoid a network outage. In all cases, regardless of the number of cycles used to upgrade the DA-MP/vSTP server group, the DA-MP leader/vSTP MP leader should be the last server upgraded. By upgrading the MP leader last, the number of leader changes is minimized during the upgrade.

      You can acces the DA-MP leader by navigating to Diameter, then Maintenance, then DA-MPs, and then Peer DA-MP Status, where MP Leader = Yes. Also, check for the MP leader on the vSTP. This is done on the active SOAM CLI.
      1. From the MMI command using the REST Client for the vSTP configuration. The MMI user guide can accessed by navigating to Main Menu, then MMI Guide
      2. Use the /vstp/mpleader MO.

        The result is the hostname of the MP leader server.

      3. In the Upgrade Settings section of the form, use the Upgrade ISO options to select the target ISO.
      4. Click OK to start the upgrade sequence.

        Control returns to the Upgrade Administration screen.

  3. View the Upgrade Administration Form to Monitor Upgrade Progress
    After selecting the Entire Site link , a summary of the upgrade status for the selected site appears. This summary identifies the server group(s) currently upgrading, the number of servers within each server group that are upgrading, and the number of servers that are pending upgrade. Use this view to monitor the upgrade status of the overall site.

    More detailed status is available by selecting the individual server group links. The server group view shows the status of each individual server within the selected server group. During the upgrade, the servers may have a combination of the following expected alarms.

    • Alarm ID = 10008 (Provisioning Manually Disabled)
    • Alarm ID = 10073 (Server Group Max Allowed HA Role Warning)
    • Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)
    • Alarm ID = 31101 (DB Replication to Slave Failure)
    • Alarm ID = 31106 (DB Merge to Parent Failure)
    • Alarm ID = 31107 (DB Merge from Child Failure)
    • Alarm ID = 31228 (HA Highly available server failed to receive mate heartbeats) or (Lost Communication with Mate Server)
    • Alarm ID = 31233 (HA Secondary Path Down)
    • Alarm ID = 31283 (Highly available server failed to receive mate heartbeats)
    • Alarm ID = 32515 (Server HA Failover Inhibited)
    • Alarm ID = 31114 (DB Replication over SOAP has failed)
    • Alarm ID = 31225 (HA Service Start Failure)
    • Alarm ID = 31149 (DB Late Write Nonactive)
    • Do not accept any upgrades at this time.
    • If the upgrade fails, do not proceed. Refer to Recover from a Failed Upgrade for failed server recovery procedures.
  4. If the Upgrade of a Server Fails
    If the upgrade of a server fails, access the server command line (via ssh or a console), and collect the following files: 
    /var/TKLC/log/upgrade/upgrade.log
/var/TKLC/log/upgrade/ugwrap.log
/var/TKLC/log/upgrade/earlyChecks.log
/var/TKLC/log/platcfg/platcfg.log

    It is recommended to contact My Oracle Support (MOS) and refer to My Oracle Support and provide these files. Refer to Recover from a Failed Upgradefor failed server recovery procedures.

    When upgrade failure issue is identified and resolved, then Auto Site upgrade can be started again without executing any failed server recovery procedure.

  5. Post Upgrade Verification
    Proceed to Site Post-Upgrade Procedures section for post upgrade verification procedures.

6.2.2 Rearrange Automated Site Upgrade Cycles

This procedure provides details to rearrange the Automated Site Upgrade cycles if required. Automated Site Upgrade provides an option to rearrange servers in the cycles thus eliminating the risks of a potential network outage. ASU provides the flexibility to user to order the servers within the cycles without breaking the Minimum Availability and DA-MP Leader/vSTP MP leader criteria. .
  1. Rearrange the Upgrade Cycle as Needed
    Click Rearrange Cycles.
  2. Rearrange Servers in Cycles
    1. Automated Site Upgrade Cycles across the sites.

      Note:

      You can rearrange only DA-MPs and vSTPs. Re-arranging SBR and IPFE servers is restricted.
    2. When a server needs to be removed from cycle and needs to be added in an existing cycle or a new cycle, perform the following steps:
      1. Select the desired server in the list and click Remove from Cycle.

        Note:

        The server moves to the Free Pool on the right side.
      2. Add the servers in Free Pool to another existing cycle or new cycle.

        Note:

        If there is no need to add a new cycle, then the procedure to rearrange the cycle is complete.
  3. Add New Cycle (If needed)
    1. Click Add Cycle.
      After adding new cycle, servers available in free pool can be added in new cycle.
    2. Click OK.

6.3 Overview of Automated/Manual Server Group Upgrade

This section contains alternative site upgrade procedures that can be used when Automated Site Upgrade does not meet the needs or concerns of the customer. These procedures use a combination of Automated Server Group upgrade and manual server upgrades to upgrade a specific site.

The following details the site upgrade plan for a non-PCA/PDRA site, which divides the upgrade into four cycles. A cycle is defined as the complete upgrade of one or more servers, from initiate upgrade to success or failure. The first two cycles consist of information to upgrade the SOAMs. The first cycle upgrades the standby SOAM, followed by the second cycle, which upgrades the active SOAM. Cycle 3 cannot begin until cycle 2 is complete. This ensures that the OAM controllers are always upgraded before any C-level servers.

The third cycle begins the upgrade of the C-level servers. In cycle 3, one-half of the DA-MPs, vSTP MPs, and IPFEs are upgraded. This leaves the remaining half of these server functions in -service to process traffic. The fourth cycle upgrades the second half of the DA-MPs, and IPFEs to complete the site upgrade.

Table 6-1 Non-PCA/PDRA Site Upgrade Plan

Cycle 1 Cycle 2 Cycle 3 Cycle 4
Standby SOAM Active SOAM    
    ½ DA-MPs ½ DA-MPs
    ½ IPFEs ½ IPFEs
    ½ vSTP MPs ½ vSTP MPs

The following table details the site upgrade plan for a PCA/PDRA system with two-site redundancy. This upgrade plan is divided into five cycles. The first two cycles consist of upgrading the SOAMs - the first cycle upgrades the standby and spare SOAMs in parallel, followed by the second cycle, which upgrades the active SOAM. Cycle 3 cannot begin until cycle 2 is complete. This ensures that the OAM controllers are always upgraded before any C-level servers. The third cycle begins the upgrade of the C-level servers. In cycle 3, one-half of the DA-MPs, IPFEs, and vSTP servers are upgraded in parallel with all of the spare SBRs. This leaves the remaining server functions in-service to process traffic.

The fourth cycle upgrades the second half of the DA-MPs, and IPFEs in parallel with the standby SBRs. The fifth cycle is required to upgrade the active SBR(s), completing the site upgrade.

Table 6-2 Two-Site Redundancy PCA Site Upgrade Plan

Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5
Standby SOAM, Spare SOAM Active SOAM      
    ½ DA-MPs ½ DA-MPs  
    ½ IPFEs ½ IPFEs  
    Spare SBR(s) Standby SBR(s) Active SBR(s)
The following table details the site upgrade plan for a PCA/PDRA system with three-site redundancy. This upgrade plan is divided into six cycles.

Note:

It is mandatory to follow the mentioned division and execution order of the cycles. This ensures the OAM controllers are always upgraded before any C-level servers.

For C-level servers, the division of servers can be planned in different cycles depending on customer requirements, which means SBR and DA-MPs can be upgraded in different cycles. But, as mentioned, spare, standby, and active SBRs should be upgraded in different cycles.

The first two cycles consist of the information to upgrade the SOAMs – the first cycle upgrades the standby and spare SOAMs in parallel, followed by the second cycle, which upgrades the active SOAM. Cycle 3 cannot begin until cycle 2 is complete. This ensures the OAM controllers are always upgraded before any C-level servers. The third cycle begins the upgrade of the C-level servers. In cycle 3, one-half of the DA-MPs, and IPFEs are upgraded in parallel with one spare SBR. This leaves the remaining server functions in-service to process traffic. The fourth cycle upgrades the second half of the DA-MPs, and IPFEs in parallel with the second spare SBR. The fifth cycle upgrades the standby SBR(s), and the sixth cycle is required to upgrade the active SBR(s), completing the site upgrade.

Table 6-3 Three-Site Redundancy PCA Site Upgrade Plan

Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6
Standby SOAM, Spare SOAM Active SOAM        
    ½ DA-MPs ½ DA-MPs    
    ½ IPFEs ½ IPFEs    
    Spare SBR(s) Standby SBR(s) Standby SBR(s) Active SBR(s)

6.3.1 Site Upgrade Planning

The upgrade of the site servers involve multiple automated upgrades using the Automated Server Group upgrade feature, along with manual upgrades that are a comparatively less automated.

For the server groups that are upgraded using ASG, the only planning necessary is to record the name of the server group. ASG automatically selects the individual servers to upgrade. The IPFE and vSTP (if equipped) server groups must be upgraded manually since there is only one server per server group. Planning is necessary for these server groups to ensure traffic continuity. Record the host name of the servers to be upgraded in each iteration. vSTP MPs should be divided in cycles to avoid a network outage. While choosing ASG and Manual upgrades for multi-active MP servers, see the limitations in Limitations of Automated Server Group and Automated Site Upgrade for the Automated Server Group upgrade option.

While choosing ASG and Manual upgrades for multi-active MP servers, see the limitations in Limitations of Automated Server Group and Automated Site Upgrade for the Automated Server Group upgrade option. If your network aligns with any of the scenarios listed in Limitations of Automated Server Group and Automated Site Upgrade, then do NOT use the Automated Server Group. This eliminates the risks of a potential network outage.

There are some limitations with upgrading the DC server in a C-level server group, which are upgraded in a group of servers, for example, DA-MP, vSTP MP(s). So, make sure the DC server is not upgraded in the first upgrade cycle of the C-Level servers in its server group. Identify the DC server using Identify the DC Server.

In all cases, regardless of the number of cycles used to upgrade the DA-MP/vSTP server group, the DA-MP leader/vSTP MP leader should be the last server upgraded. By upgrading the MP leader last, the number of leader changes is minimized during the upgrade.

Access the DA-MP leader on the active SOAM by navigating to Diameter, then Maintenance, then DA-MPs, and then Peer DA-MP Status, where MP Leader = Yes.

Also, check for the MP leader on the vSTP. This is done on the active SOAM CLI from the MMI command using the REST Client for the vSTP configuration. Complete this procedure by performing the following steps:
  1. Access the MMI user guide by navigating to Main Menu, then MMI Guide.
  2. Use the /vstp/mpleader MO.

    The result is the hostname of the MP leader server.

Note:

In iteration 1, if a spare SOAM exists, the spare and standby SOAMs are upgraded manually. Otherwise, the SOAMs are upgraded with ASG.

In iteration 2, the active SOAM is upgraded either manually or by ASG.

In iteration 3 and 4, ASG automatically selects DA-MPs for upgrade in DA-MP Group 1 and DA-MP Group 2 respectively. ASG also automatically selects the spare SBR(s) for upgrade. However, IPFE 1 Hostname and IPFE 3 Hostname are upgraded manually.

In iteration 5, ASG automatically selects the active SBR(s) for upgrade.

6.3.2 SOAM Upgrade Overview

This section contains the steps required to perform a major or incremental upgrade of the SOAMs for a DSR site. During the site upgrade (SOAMs plus all C-level servers), site provisioning is disabled. Provisioning is re-enabled at the completion of the site upgrade. For each site in the DSR, the SOAM(s) and associated MPs and IPFEs should be upgraded within a single maintenance window. The above shows the estimated execution times for the SOAM upgrade. Manual SOAM Upgrade (Active/Standby/Spare) is recommended for upgrading the SOAMs when there is no spare SOAM. ASG automatically upgrades the standby SOAM followed by the active SOAM.

Manual SOAM Upgrade (Active/Standby/Spare) procedure is also recommended when the site has a spare SOAM.The manual upgrade procedure upgrades the standby and spare SOAMs in parallel, followed by the active SOAM.

Note:

For information on SOAM VM profile for increased MP Capacity, refer to Create a Link for ComAgent.

6.3.3 Upgrade SOAMs

This section provides the procedures to upgrade the SOAMs. The SOAMs can be upgraded manually under user control, or automatically using the Automated Server Group Upgrade option. The recommended method for SOAM upgrade depends on the existence of a spare SOAM. If the site includes a spare SOAM, the SOAMs are upgraded manually so that the spare and standby SOAMs can be upgraded concurrently. This reduces the time required to upgrade the SOAMs. Regardless of which SOAM upgrade option is used, refer to SOAM Upgrade Pre-Checks section to ensure site provisioning is disabled. If the site does not include a spare SOAM, refer to Automated SOAM Upgrade (Active/Standby) section. If the site does include a spare SOAM, refer to Manual SOAM Upgrade (Active/Standby/Spare) section.

Note:

Site Preupgrade Backups, Site Preupgrade Health Check for Release 8.0 and Later, and Disable Site Provisioning procedures must be completed before the start of SOAM upgrade:
6.3.3.1 Active SOAM VIP: View KPIs to Verify Traffic Status
This procedure verifies the traffic status by viewing the KPIs.
  1. Log in to the active SOAM GUI using the VIP.
  2. Navigate to Status & Manage, then KPIs.
  3. Inspect KPI reports to verify traffic is at the expected condition.
6.3.3.2 Active SOAM VIP: Verify Site Provisioning is Disabled
This procedure verifies that site provisioning is disabled.
  1. Verify that Site Provisioning was properly disabled. In the GUI status bar, where it says Connected using …, check for the message Site Provisioning disabled.
  2. If the message is present, continue with the next procedure. Otherwise, perform Disable Site Provisioning procedure.
6.3.3.3 Automated SOAM Upgrade (Active/Standby)
This procedure is the recommended method for upgrading the SOAMs if the site does not include a spare SOAM. If the site has a spare SOAM, refer to Manual SOAM Upgrade (Active/Standby/Spare) section and upgrade. Upon completion of this procedure, proceed to Rearrange Automated Site Upgrade Cycles section, Upgrade Iteration 3.
6.3.3.3.1 Upgrade SOAM Server Group
This procedure upgrades the SOAM(s) using the Automated Server Group Upgrade option.
  1. Upgrade the SOAM server group using the Upgrade Multiple Servers procedure with the following options:
    1. Use the Automated Server Group Upgrade option
    2. Select the Serial Upgrade mode
  2. Execute Appendix D Upgrade Multiple Servers – Upgrade Administration.
After successfully completing the procedure in Appendix D, return to this point and proceed to Upgrade Iteration 3 section.

Note:

Once the network element SOAMs are upgraded, if any C-level server is removed from a server group and re-added, the server must be restored using disaster recovery procedures. The normal replicatiochannel to the C-level server is inhibited due to the difference in release versions.

6.3.3.4 Manual SOAM Upgrade (Active/Standby/Spare)
This procedure upgrades the SOAM server group if the site includes a spare SOAM. If the SOAM server group was upgraded via Automated SOAM Upgrade (Active/Standby) section, then do not perform this procedure; proceed to Upgrade Iteration 3 section.
6.3.3.4.1 Upgrade Standby and Spare SOAMs in parallel using the Upgrade Multiple Servers Procedure

This procedure upgrades the SOAMs in a DSR manually. If this procedure fails, it is recommended to contact My Oracle Support (MOS) and ask for assistance.

  1. Execute Appendix D Upgrade Multiple Servers – Upgrade Administration. After successfully completing the procedure in Appendix D, return to this point and continue with the next step.
6.3.3.4.2 Upgrade Active SOAM using Upgrade Single Server Procedure
This procedure upgrades the SOAMs in a DSR manually. If this procedure fails, it is recommended to contact My Oracle Support (MOS) and ask for assistance.
  1. Execute Appendix C Upgrade Single Server – DSR 8.x. After successfully completing the procedure in Appendix C, return to this point and proceed to Upgrade Iteration 3.

6.3.4 Upgrade Iteration 3

Upgrade iteration 3 begins the upgrade of the site C-level servers. Iteration 3 consists of upgrading the DA-MPs, IPFEs, spare SBR(s), and vSTP MP server, if equipped. The C-level components are upgraded in parallel to maximize Maintenance Window usage. The estimated time required to upgrade the C-level servers for iteration 3.

Note:

  • The estimated time required to upgrade the C-level servers for iteration 3 is 0:40-1:00, and this procedure upgrades ½ of the DA-MPs, ½ of the IPFEs, ½ of the vSTPs, and the spare SBR(s).
  • This procedure upgrades a portion of the C-level servers for iteration 3. If this procedure fails, it is recommended to contact My Oracle Support (MOS) and ask for assistance.

Caution:

ASG does not allow the operator to specify the upgrade order of the DA-MP servers. If a manual upgrade was recommended in section Site Upgrade Methodology Selection section, do not use ASG to upgrade the DA-MPS in this iteration. Alternate upgrade procedures are provided in Appendix F.3.
  1. Select the DA-MP server group to view preupgrade status of DA-MPs
    1. Log in to the NOAM GUI using the VIP.
    2. Navigate to Administration, select Software Management, and click Upgrade.
    3. Select the SOAM tab of the site being upgraded.
    4. Select the DA-MP Server Group link.
    5. For the DA-MP servers to be upgraded in iteration 3, verify the application version value is the expected source software release version.
  2. View preupgrade status of DA-MP servers on active NOAM VIP
    1. If the servers are in Backup Needed state, select the servers and click Backup. The Upgrade State changes to Backup in Progress. When the backup is complete, the Upgrade State changes to Ready.
    2. Verify the OAM Max HA Role is in the expected condition (either standby or active). This depends on the server being upgraded.
  3. Verify if upgrade status is ready for the server to be upgraded.

    Note:

    This may take a minute if a backup is in progress. Depending on the server being upgraded, new alarms may occur.

    Servers may have a combination of the following expected alarms.

    Note:

    Not all servers have all alarms.
    • Alarm ID = 10008 (Provisioning Manually Disabled)
    • Alarm ID = 10073 (Server Group Max Allowed HA Role Warning)
    • Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)
    • Alarm ID = 32515 (Server HA Failover Inhibited)
    • Alarm ID = 31101 (DB Replication to slave DB has failed)
    • Alarm ID = 31106 (DB Merge to Parent Failure)
    • Alarm ID = 31107 (DB Merge From Child Failure)
    • Alarm ID = 31228 (HA Highly available server failed to receive mate heartbeats) or (Lost Communication with Mate Server)
    • Alarm ID = 31114 (DB Replication over SOAP has failed)
    • Alarm ID = 31225 (HA Service Start Failure)
  4. Initiate the Automated Server Group upgrade of the DA-MP Servers (Part 1)
    1. To use the Automated Server Group upgrade option, verify no servers in the server group are selected.
    2. Click Auto Upgrade.
  5. Initiate the Automated Server Group upgrade of the DA-MP Server (Part 2)
    1. The Upgrade Settings section of the Initiate screen controls the behavior of the server group upgrade. Select Bulk mode.
    2. Select 50% for the Availability setting.
    3. Select the appropriate ISO from the Upgrade ISO options.
    4. Click OK to start the upgrade.
  6. View the upgrade administration form to monitor upgrade progress.
    1. Observe the upgrade state of the DA-MP servers. Upgrade status displays under the Status Message column.
    2. While the DA-MP servers are upgrading, continue with the next step to upgrade additional C-level components in parallel.
  7. Identify the IPFE server group(s) to upgrade
    From the data captured in Site Upgrade Planning section, identify the IPFE server group(s) to upgrade in iteration 3.
  8. View preupgrade status of IPFEs
    1. Navigate to Administration, select Software Management Upgrade.
    2. Select the SOAM tab of the site being upgraded.
    3. Select the link for each IPFE server group to upgrade.
    4. For the IPFE servers to be upgraded in iteration 3, verify the application version value is the expected source software release version.
    5. If a server is in Backup Needed state, select the servers and click Backup. The Upgrade State changes to Backup in Progress. When the backup is complete, the Upgrade State changes to Ready.
    6. Verify the OAM Max HA Role is in the expected condition (either standby or active). This depends on the server being upgraded.
  9. Verify Upgrade Status is Ready for the server to be upgraded

    Note:

    This may take a minute if a backup is in progress. Depending on the server being upgraded, new alarms may occur.

    The Upgrade Administration screen appears. Navigate to the IPFE server group being upgraded.

    Servers may have a combination of the following expected alarms:

    Note:

    Not all servers have all alarms.
    • Alarm ID = 10008 (Provisioning Manually Disabled)
    • Alarm ID = 10073 (Server Group Max Allowed HA Role Warning)
    • Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)
    • Alarm ID = 32515 (Server HA Failover Inhibited)
    • Alarm ID = 31101 (DB Replication to slave DB has failed)
    • Alarm ID = 31106 (DB Merge to Parent Failure)
    • Alarm ID = 31107 (DB Merge From Child Failure)
    • Alarm ID = 31228 (HA Highly available server failed to receive mate heartbeats) or (Lost Communication with Mate Server)
    • Alarm ID = 31114 (DB Replication over SOAP has failed)
    • Alarm ID = 31225 (HA Service Start Failure)
  10. Initiate IPFE Upgrade (Part 1)
    1. Select the Upgrade Server method.
    2. From the Upgrade Administration screen, select the server to upgrade.
    3. Click Upgrade Server.
  11. Initiate IPFE Upgrade (Part 2)
    1. Select target ISO.
    2. On the Upgrade Initiate screen, select the target ISO from the Upgrade ISO options.
    3. Click OK to start the upgrade.
  12. View the upgrade administration form to monitor upgrade progress.
    Observe the upgrade state of the IPFE server. Upgrade status displays under the Status Message column.
  13. Identify the SBR Server Group(s) to Upgrade
    From the data captured in Site Upgrade Planning section, identify the SBR server group(s) to upgrade in iteration 3.

    Note:

    ASG steps (Auto Upgrade), mentioned in the next steps, do not provide any liberty to the operator to verify observations during the upgrade. If a manual upgrade was recommended for the SBR upgrade, do not use ASG to upgrade all the SBR servers from the same server group in this single iteration. Alternate upgrade procedures are provided in Manual SBR Upgrade section. Spare SBR server(s) need to be upgraded. If the Manual Upgrade is used, skip ASG steps 15. to 19.
  14. View preupgrade status of SBRs to upgrade.
    1. Navigate to Administration, then Software Management, and then Upgrade.
    2. Select the SOAM tab of the site being upgraded.
    3. Select the link for each SBR server group to upgrade.
    4. For the SBR servers to be upgraded in iteration 3, verify the application version value is the expected source software release version.
    5. If the server is in Backup Needed state, select the servers and click Backup. The Upgrade State changes to Backup in Progress. When the backup is complete, the Upgrade State changes to Ready.
    6. Verify the OAM Max HA Role is in the expected condition (either standby or active). This depends on the server being upgraded.
  15. Verify upgrade status is Ready for the server to be upgraded.

    Note:

    This procedure defines the steps to verify that the upgrade status is ready for the server to be upgraded. This may take a minute if a backup is in progress. Depending on the server being upgraded, new alarms may occur.

    The Upgrade Administration screen appears. Navigate to the SBR server group being upgraded.

    Servers may have a combination of the following expected alarms. However, not all servers have all alarms.

    • Alarm ID = 10008 (Provisioning Manually Disabled)
    • Alarm ID = 10073 (Server Group Max Allowed HA Role Warning)
    • Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)
    • Alarm ID = 32515 (Server HA Failover Inhibited)
    • Alarm ID = 31101 (DB Replication to slave DB has failed)
    • Alarm ID = 31106 (DB Merge to Parent Failure)
    • Alarm ID = 31107 (DB Merge From Child Failure)
    • Alarm ID = 31228 (HA Highly available server failed to receive mate heartbeats) or (Lost Communication with Mate Server)
    • Alarm ID = 31114 (DB Replication over SOAP has failed)
    • Alarm ID = 31225 (HA Service Start Failure)
  16. Initiate SBR Upgrade (Part 1)
    1. Select the Auto Upgrade method.
    2. To use the Automated Server Group upgrade option, select the SBR server group to upgrade.
    3. Verify no servers in the server group are selected.
    4. Click Auto Upgrade.
  17. Initiate SBR Upgrade (Part 2)
    1. The Upgrade Settings section of the Initiate screen controls the behavior of the automated upgrade. Select Serial mode.
    2. Set upgrade options and start the Automated Server Group Upgrade.
    3. Select the appropriate ISO from the Upgrade ISO options.
    4. Click OK to start the upgrade.
  18. View the upgrade administration form to monitor upgrade progress.
    Observe the Upgrade State of the SBR server group. Upgrade status displays under the Status Message column.
  19. View Pre-upgrade Status of vSTP MP Servers
    1. Navigate to Administration, then Software Management, and then Upgrade.
    2. Select the SOAM tab of the site being upgraded.
    3. Select the link for each vSTP server group to upgrade.
    4. For the vSTP servers to be upgraded in iteration 3, verify the Application Version value is the expected source software release version.
    5. If a server is in Backup Needed state, select the server and click Backup. The Upgrade State changes to Backup in Progress. When the backup is complete, the Upgrade State changes to Ready.
    6. Verify the OAM Max Ha Role is the expected condition (either standby or active). This depends on the server being upgraded.
  20. Verify upgrade status is Ready for the server to be upgraded.

    Note:

    This may take a minute if a backup is in progress. Depending on the server being upgraded, new alarms may occur.

    The Upgrade Administration screen displays. Navigate to the vSTP MP server group being upgraded.

    Note:

    Servers may have a combination of the following expected alarms. However, not all servers have all alarms:
    • Alarm ID = 10008 (Provisioning Manually Disabled)
    • Alarm ID = 10073 (Server Group Max Allowed HA Role Warning)
    • Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)
    • Alarm ID = 32515 (Server HA Failover Inhibited)
    • Alarm ID = 31101 (DB Replication to slave DB has failed)
    • Alarm ID = 31106 (DB Merge to Parent Failure)
    • Alarm ID = 31107 (DB Merge From Child Failure)
    • Alarm ID = 31228 (HA Highly available server failed to receive mate heartbeats) or (Lost Communication with Mate Server)
    • Alarm ID = 31114 (DB Replication over SOAP has failed)
    • Alarm ID = 31225 (HA Service Start Failure)
  21. Initiate vSTP MP Upgrade (Part 1) on active NOAM VIP
    1. Select the Upgrade Server method.
    2. From the Upgrade Administration screen, select the server to be upgraded.
    3. Click Upgrade Server.
  22. Initiate vSTP Upgrade (Part 2) on active NOAM VIP
    1. Select target ISO
    2. On the Upgrade Initiate screen, select the target ISO from the Upgrade ISO options.
    3. Click OK to initiate the upgrade.
  23. View the upgrade administration form to monitor upgrade progress. Observe the Upgrade State of the vSTP MP server. Upgrade status displays under the Status Message column.
  24. View the upgrade administration form to monitor upgrade progress

    Note:

    If the upgrade of a server fails section for instructions if the upgrade fails, or if execution time exceeds 60 minutes. If the upgrade processing encounters a problem, it may attempt to ROLL BACK to the original software release. In this case, the upgrade displays as FAILED.
    1. Navigate to Administration, select Software Management, and click Upgrade.
    2. Select the SOAM tab of the site being upgraded.
    3. Sequence through the server group links for the server groups being upgraded. Observe the Upgrade State of the servers of interest. Upgrade status displays under the Status Message column.
      • Alarm ID = 10008 (Provisioning Manually Disabled)
      • Alarm ID = 10073 (Server Group Max Allowed HA Role Warning)
      • Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)
      • Alarm ID = 31101 (DB Replication To Slave Failure)
      • Alarm ID = 31106 (DB Merge To Parent Failure)
      • Alarm ID = 31107 (DB Merge From Child Failure)
      • Alarm ID = 31228 (HA Highly available server failed to receive mate heartbeats) or (Lost Communication with Mate Server)
      • Alarm ID = 31233 (HA Secondary Path Down)
      • Alarm ID = 31283 (Highly available server failed to receive mate heartbeats)
      • Alarm ID = 32515 (Server HA Failover Inhibited)
      • Alarm ID = 31114 (DB Replication over SOAP has failed)
      • Alarm ID = 31225 (HA Service Start Failure)

      Note:

      Database (DB) replication failure alarms may display during an Automated Site Upgrade or during an event that resets multiple servers in parallel. The DB on the child servers is not updated until resolved. Refer to Limitations of Automated Server Group and Automated Site Upgrade to resolve this issue.
  25. If the upgrade of a server fails, access the server command line (using ssh or a console), and collect the following files: 
    /var/TKLC/log/upgrade/upgrade.log
/var/TKLC/log/upgrade/ugwrap.log
/var/TKLC/log/upgrade/earlyChecks.log 
/var/TKLC/log/platcfg/upgrade.log

    Note:

    It is recommended to contact My Oracle Support (MOS) and provide these files. Refer to Appendix I for failed server recovery procedures.

6.3.5 Upgrade Iteration 4

Upgrade iteration 4 continues the upgrade of the site C-level servers. Iteration 4 consists of details to upgrade the second half of the DA-MPs, vSTPs, and IPFEs, as well as the standby SBR(s), if equipped. The procedures in this section provide the steps to upgrade, ½ of the vSTPs servers and ½ of the IPFEs. ASG automatically upgrades the DA-MPs and SBRs.

From the data captured in Site Upgrade Planning section, identify the IPFE server group(s) to be upgraded in iteration 4.

6.3.5.1 Active NOAM VIP: View Pre-upgrade Status of IPFEs
  1. Navigate to Administration, and select Software Management and click Upgrade.
  2. Select the NOAM tab of the site being upgraded.
  3. Select the link of each IPFE server group to be upgraded.
  4. For the IPFE servers to be upgraded in iteration 4, verify the application version value is the expected source software release version.
  5. If a server is in Backup Needed state, select the servers and click Backup. The Upgrade State changes to Backup in Progress. When the backup is complete, the Upgrade State changes to Ready.
  6. Verify the OAM Max HA Role is in the expected condition (either standby or active). This depends on the server being upgraded.

6.3.6 Upgrade Iteration 5

  1. At iteration 5, the active SBR is upgraded, causing the standby to become active.
  2. View the upgrade administration form to monitor upgrade progress.

    See step 3 for instructions if the upgrade fails, or if execution time exceeds 60 minutes.

    Note:

    If the upgrade processing encounters a problem, it may attempt to ROLL BACK to the original software release. In this case, the Upgrade displays as FAILED.

    The execution time may be shorter or longer, depending on the point in the upgrade where there was a problem.

    1. Navigate to Administration, select Software Management, and click Upgrade.
    2. Select the SOAM tab of the site being upgraded.
    3. Sequence through the server group links for the server groups being upgraded. Observe the upgrade state of the servers of interest. Upgrade status displays under the Status Message column.
    During the upgrade, the servers may have a combination of the following expected alarms.
    • Alarm ID = 10008 (Provisioning Manually Disabled)
    • Alarm ID = 10073 (Server Group Max Allowed HA Role Warning)
    • Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)
    • Alarm ID = 31101 (DB Replication To Slave Failure)
    • Alarm ID = 31106 (DB Merge To Parent Failure)
    • Alarm ID = 31107 (DB Merge From Child Failure)
    • Alarm ID = 31228 (HA Highly available server failed to receive mate heartbeats) or (Lost Communication with Mate Server)
    • Alarm ID = 31233 (HA Secondary Path Down)
    • Alarm ID = 31283 (Highly available server failed to receive mate heartbeats)
    • Alarm ID = 32515 (Server HA Failover Inhibited)
    • Alarm ID = 31114 (DB Replication over SOAP has failed)
    • Alarm ID = 31225 (HA Service Start Failure)

    Database (DB) replication failure alarms may display during an Automated Site Upgrade or during an event that resets multiple servers in parallel. The DB on the child servers is not updated until resolved. Refer to Limitations of Automated Server Group and Automated Site Upgrade to resolve this issue.

    Wait for the SBR upgrades to complete. The Status Message column displays Success. This step takes approximately 20 to 50 minutes.

  3. If the upgrade of a server fails, access the server command line (through ssh or a console), and collect the following files:
    Foot 1

    If any upgrade fails, do not proceed. It is recommended to consult with on the best course of action. Refer to Recover from a Failed Upgrade for failed server recovery procedures.

6.4 Upgrade Single Server – DSR 8.x

The following procedures upgrade a single DSR server of any type (For example: NOAM, SOAM, MP) when the active NOAM is on DSR 8.x.

Note:

This procedure may be executed multiple times during the overall upgrade, depending on the number of servers in the DSR and the chosen upgrade methodology. Make multiple copies of this procedure to mark up, or keep another form of written record of the steps performed.
  1. View the preupgrade status of servers in active NOAM VIP
    1. Log in to the NOAM GUI using the VIP.
    2. Navigate to Administration, then Software Management, and then Upgrade.
    3. Select the Network Element of the server to be upgraded (NOAM or site).

      The active NOAM server may have some or all of these expected alarms:

      Alarm ID = 10008 (Provisioning Manually Disabled)

      Alarm ID = 32532 (Server Upgrade Pending Accept/Reject)

  2. Verify Status of Server to be Upgraded
    1. Identify the server to be upgraded (NOAM, SOAM, MP, and so on) and record hostname.
    2. Verify the Application Version value is the expected source software release version.
    3. If the server is in the Backup Needed state, select the server and click Backup.
    4. On the Upgrade Backup screen, click OK.

      The Upgrade State changes to Backup in Progress.

    5. Verify the OAM Max HA Role is the expected condition (either standby or active). This depends on the server being upgraded.
    6. When the backup is complete, verify the server state changes to Ready.
  3. Initiate the Server Upgrade
    1. From the Upgrade Administration screen, select the server to be upgraded.
    2. Click Upgrade Server.

      The Initiate Upgrade form appears.

  4. Select Upgrade ISO
    1. Initiate the server upgrade. From the Upgrade Settings – Upgrade ISO options, select the ISO to use in the server upgrade.

      Note:

      When the active NOAM is the server being upgraded, click OK to initiate an HA switchover and cause the GUI session to log out.

      If the selected server is the active server in an active/standby pair, the OAM Max HA Role column displays Active with a red background. This is NOT an alarm condition. This indicator is to make the user aware the Make Ready action causes an HA switchover.

    2. Click OK.

      The upgrade begins and control returns to the Upgrade Administration screen.

      Note:

      Do not omit this step.
    3. Log out of the GUI, clear the browser cache, and log back into the active NOAM via the VIP before continuing. Some GUI forms may exhibit incorrect behaviors if the browser cache is not cleared.
  5. View the Upgrade Administration Form to Monitor Upgrade Progress
    1. Observe the upgrade status of the site on the Upgrade Administration screen by selecting the Entire Site link. An upgrade status summary of each server group in the site displays in the Server Upgrade States column.

      Servers may have a combination of the following expected alarms. However, not all servers have all alarms.

      Alarm ID = 10008 (Provisioning Manually Disabled)

      Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)

      Alarm ID = 10073 (Server Group Max Allowed HA Role Warning)

      Alarm ID = 32515 (Server HA Failover Inhibited)

      Alarm ID = 31228 (HA Highly available server failed to receive mate heartbeats) or (Lost Communication with Mate Server)

      Alarm ID = 31283 (Highly available server failed to receive mate heartbeats)

      Alarm ID = 31106 (DB Merge tTo Parent Failure)

      Alarm ID = 31107 (DB Merge fFrom Child Failure)

      Alarm ID = 31233 (HA Secondary Path Down)

      Alarm ID = 31101 (DB Replication tTo Slave Failure)

      Alarm ID = 31104 (DB Replication over SOAP has failed

      Alarm ID = 31282 (The HA manager (cmha) is impaired by a s/w fault)

      Alarm ID = 31225 (HA Service Start Failure)

      Alarm ID = 31226 (HA Availability Status Degraded)

      Alarm ID = 31114 (DB Replication over SOAP has failed)

      Alarm ID = 31149 (DB Late Write Nonactive)

    2. Wait for the upgrade to complete. The Status Message column displays Success. This step takes approximately 20 to 50 minutes.

      Note:

      In the unlikely event that after the upgrade, if the Upgrade State of server is Backout Ready or Failed and the Status Message displays Server could not restart the application to complete the upgrade, then perform the stepms mentioned in Manual Completion of Server Upgrade to restore the server to full operational status and return to this step to continue the upgrade.

      Note:

      Perform Create a Link for ComAgent to create a link of Comagent. If the upgrade fails, do not proceed. It is recommended to consult with Create a Link for ComAgent on the best course of action. Refer to Recover from a Failed Upgrade for failed server recovery procedures.

      See Server CLI: (Optional) View in progress status from command line of server section for an optional method of monitoring upgrade progress. See Server CLI: If the upgrade fails section for instructions if the upgrade fails.

  6. View In Progress Status from Command Line of Server in Server CLI

    Note:

    This is an optional method to view the upgrade progress from the command line.

    To view the detailed progress of the upgrade, access the server command line (via SSH or Console), and enter: 

    $ tail -f /var/TKLC/log/upgrade/upgrade.log

    This command displays the upgrade log entries as the events occur. Once the upgrade is complete, the server reboots. It takes a couple of minutes for the DSR application processes to start up. For example, this command displays the current rev on the server:

    [admusr@NO2 ~]$ appRev

    Install Time: Thu Dec 15 00:05:46 2016

    Product Name: DSR

    Product Release: 8.6.0.7.0_96.34.0

    Base Distro Product: TPD

    Base Distro Release: 7.8.3.0.0-89.21.0

    Base Distro ISO: TPD.install-7.8.3.0.0-89.21.0-OracleLinux6.10-x86_64.iso

    ISO name: DSR-8.6.0.7.0_96.34.0.iso

    OS: OracleLinux 6.10

    Note:

    If the upgrade fails, do not proceed. It is recommended to consult with on the best course of action. Refer to Recover from a Failed Upgrade for failed server recovery procedures.
  7. If the upgrade of a server fails, access the server command line (through ssh or a console), and collect the following files:

    /var/TKLC/log/upgrade/upgrade.log

    /var/TKLC/log/upgrade/ugwrap.log

    /var/TKLC/log/upgrade/earlyChecks.log

    /var/TKLC/log/platcfg/upgrade.log

    Note:

    It is recommended to contact My Oracle Support by referring to Create a Link for ComAgent of this document and provide these files. Refer to Recover from a Failed Upgrade for failed server recovery procedures.
  8. Verify post upgrade status using active NOAM VIP.
    1. Navigate to Administration, then Software Management, and then Upgrade.
    2. Select the tab of the NOAM or site being upgraded.
    3. Verify the Application Version value for this server has been updated to the target software release version.
    4. Verify the Upgrade State of the upgraded server is Accept or Reject.
  9. Verify if the Server was Successfully Upgraded

    Navigate to Alarm & Events, then View Active.

    The active NOAM or SOAM server may have some or all the following expected alarms:
    • Alarm ID = 10008 (Provisioning Manually Disabled)
    • Alarm ID = 10010 (Stateful database not yet synchronized with mate database)
    • Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)
    • Alarm ID = 31000 (Program impaired by S/W Fault)
    • Alarm ID = 31201 (Process Not Running) for eclipseHelp process
    • Alarm ID = 31282 (The HA manager (cmha) is impaired by a s/w fault)
    The active NOAM or SOAM has these expected alarms until both NOAMs/SOAMs are upgraded:
    • Alarm ID = 31233 – HA Secondary Path Down
    • Alarm ID = 32532 (Server Upgrade Pending Accept/Reject)

    Note:

    Do not accept upgrade at this time. This alarm is OK.

6.5 Upgrade Multiple Servers – Upgrade Administration

The procedures in this section upgrade multiple servers in parallel.

Note:

  • This procedure is executed multiple times during the overall upgrade, depending on the number of servers in your DSR. Make multiple copies of Appendix D to mark up or keep another form of written record of the steps performed.
  • If the upgrade is required from 8.6.x VM to 9.0.1, refer to unresolvable-reference.html section.
  1. View Preupgrade Status of the Servers
  2. Verify status of servers to be upgraded
  3. Verify upgrade status is Ready.
    The Upgrade Administration form refreshes and the server to upgrade displays Upgrade Status = Ready. This may take a minute. Depending on the server being upgraded, new alarms may occur.
    • Alarm ID = 10008 (Provisioning Manually Disabled)
    • Alarm ID = 10073 (Server Group Max Allowed HA Role Warning)
    • Alarm ID = 10075 (The server is no longer providing services because application processes have been manually stopped)
    • Alarm ID = 32515 (Server HA Failover Inhibited)
    • Alarm ID = 31101 (DB Replication to slave DB has failed)
    • Alarm ID = 31106 (DB Merge to Parent Failure)
    • Alarm ID = 31107 (DB Merge From Child Failure)
    • Alarm ID = 31228 (HA Highly available server failed to receive mate heartbeats) or (Lost Communication with Mate Server)
    • Alarm ID = 31114 (DB Replication over SOAP has failed)
    • Alarm ID = 31225 (HA Service Start Failure)
  4. Determine Upgrade Method
    1. To upgrade multiple servers in parallel using the manual option, perform Active NOAM VIP: Initiate upgrade (part 1) and Active NOAM VIP: Initiate upgrade (part 2).
    2. To upgrade a server group using the Automated Server Group Upgrade option, proceed to Active NOAM VIP: Initiate (part 1) – Automated Server Group Upgrade option.
  5. Initiate Upgrade (Part 1)
    1. From the Upgrade Administration screen, select the servers to upgrade.
    2. Click Upgrade Server

    The Initiate Upgrade form displays on the Administration, then Software Management, and then Upgrade Initiate screen.

  6. Initiate Upgrade (Part 2) – Select ISO Form
    1. From the Upgrade Settings – Upgrade ISO options, select the ISO to use in the server upgrade.
    2. Click OK

      The upgrade begins and control returns to the Upgrade Administration screen.

    3. Proceed to Active NOAM VIP: Initiate (part 2) – Automated Server Group Upgrade procedure to complete this procedure.
  7. Initiate Part 1 – Automated Server Group Upgrade Option
    1. To utilize the Automated Server Group upgrade option, verify no servers in the server group are selected.
    2. Click Auto Upgrade.
  8. Initiate Part 2 – Automated Server Group Upgrade
    1. The Upgrade Settings section of the Initiate screen controls the behavior of the automated upgrade. Select the settings that apply to the server type being upgraded.
      • Bulk: Select this option for active/standby and multi-active server groups. For servers in an active/standby configuration, the standby server is upgraded first, followed by the active. Servers in a multi-active configuration are upgraded in parallel to the extent allowed by the Availability setting.
      • Serial: Select this option to upgrade multiple servers one at a time.
      • Grouped Bulk: Select this option for SBR server groups. Grouped bulk always upgrades the spare(s), followed by the standby, followed by the active.
      • Availability: This setting determines how many servers remain in service while servers in the server group are upgraded. For example, a setting of 50% ensures at least half of the servers in the server group remain in service.

      Note:

      The Serial upgrade mode is available as an alternative to Bulk and Grouped Bulk for a more conservative upgrade scenario. Serial mode upgrades each server in the server group one at a time, and can be used on any server group type.
    2. Select the appropriate ISO from the Upgrade ISO options.
    3. Click OK to start the upgrade.
  9. View the Upgrade Administration Form to Monitor Upgrade Progress.
    Repeat the steps mentioned in Active NOAM VIP: View the Upgrade Administration Form to Monitor Upgrade Progress

    Note:

    See Server CLI: (Optional) View in-progress status from command line procedure for an optional method of monitoring upgrade progress.

    See Server CLI: If upgrade fails procedure for instructions if the Upgrade fails, or if execution time exceeds 60 minutes.

  10. View In-Progress Status from Command Line
  11. If Upgrade Fails
    Repeat the Server CLI: If Upgrade Fails procedure.
  12. Verify Post-Upgrade Status
  13. Verify the Upgrade
    Repeat the Verify the Upgrade procedure.

6.6 Manual Completion of Server Upgrade

This procedure provides the details about manual completion of server upgrade.
In the unlikely event that after the upgrade, if the Upgrade State of server is Backout Ready and the Status Message displays Server could not restart the application to complete the upgrade, then perform this procedure to restore the server to full operational status and return to this step to continue the upgrade. Perform the steps in Appendix U to create a link of Comagent.

6.6.1 NOAMP VIP GUI: Log in to the Server (If Not Already Done)

  1. Establish a GUI session on the NOAM server using the VIP IP address of the NOAM server. Open the web browser and enter the following URL:

    http://<Primary_NOAM_VIP_IP_Address>

  2. Log in to the NOAM GUI as the guiadmin user.

6.6.2 NOAMP VIP GUI: Verify Server Status

  1. Navigate to Status and Manage, then HA.
  2. Locate the server you want to upgrade.
  3. Verify the Max Allowed HA Role is Standby.
  4. Click Edit.

6.6.3 NOAMP VIP GUI: Change the Role

  1. Change the Max Allowed HA Role to Active.
  2. Click OK.

6.6.4 NOAMP VIP GUI: Verify Change

  1. Verify the Max Allowed HA Role changes to Active.

6.6.5 NOAMP VIP GUI: Restart the Server

  1. Navigate to Status & Manage, then Server.
  2. Select the server to be upgraded.
  3. Click Restart.

    After a few minutes, the Appl State changes to Enabled.

6.6.6 NOAMP VIP GUI: Verify Status

  1. Navigate to Administration, then Software Management, and then Upgrade.
  2. Verify the Upgrade State changes to Accept or Reject and the Status Message changes to Success: Server manually completed.

6.7 Site Post-Upgrade Procedures

You need to perform the post-upgrade procedures after all the site upgrades are complete. The final health check of the system collects alarm and status information to verify that the upgrade did not degrade system operation. After an appropriate soak time, the upgrade is accepted.

Note:

Allow Site Provisioning and Site Post-Upgrade Health Check procedures must be executed at the completion of each SOAM site upgrade.

After all SOAM sites in the topology have completed upgrade, the upgrade may be accepted using the Accept Upgrade procedure.

6.7.1 Allow Site Provisioning

This procedure enables Site Provisioning for SOAM and MP servers. If this procedure fails, it is recommended to contact My Oracle Support (MOS) and ask for assistance.

Active SOAM VIP: Enable Site Provisioning

  1. Log in to the SOAM GUI of the site just upgraded using the VIP.
  2. Navigate to Status & Manage, then Database.
  3. Click Enable Site Provisioning.
  4. Confirm the operation by clicking OK on the screen.
  5. Verify the button text changes to Disable Site Provisioning.

6.7.2 Post-Upgrade Health Checks

This section provides procedures to verify the validity and health of the site upgrade. It consists of the procedures that determine the validity of the upgrade as well as the health and status of the network and servers. If the 10054 - Device Deployment Failed alarm displays after the upgrade for any server, refer to Appendix S Workaround to Resolve Device Deployment Failed Alarm corrective steps.

Note:

If syscheck fails on any server during preupgrade checks or in early checks stating that cpu: FAILURE: No record in alarm table for FAILURE!, see Workaround to Resolve syscheck Error for CPU Failure procedure.
6.7.2.1 Active NOAM VIP: Run Automated Post-upgrade Health Checks
  1. Navigate to Administration > Software Management > Upgrade.
  2. Select the SOAM tab of the site being upgraded.
  3. Select the SOAM server group link for the site being upgraded.
  4. Select the active SOAM.
  5. Click Checkup.
  6. Under Health check options, select Post Upgrade.
  7. Click OK.

    Control returns to the Upgrade screen.

6.7.2.2 Active NOAM VIP: Monitor Health Check Progress for Completion
  1. Click the Tasks option to display the currently executing tasks. The Health Check task name appears as <SOServerGroup> PostUpgrade Health Check.
  2. Monitor the Health Check task until the Task State is completed. The Details column displays a hyperlink to the Health Check report.
  3. Click the hyperlink to download the Health Check report.
  4. Open the report and review the results.
6.7.2.3 Active NOAM VIP: Analyze Health Check Results
Follow this procedure to analyze Health Check failure. If the Health Check report status is anything other than Pass, the Health Check logs can be analyzed to determine if the upgrade can proceed.
  1. Navigate to Status & Manage, then Files.
  2. Select the active SOAM tab.
  3. Select the UpgradeHealthCheck.log file and click View.
  4. Locate the log entries for the most recent health check.

    Note:

    Review the log for failures. Analyze the failures and determine if it is safe to continue the upgrade. If necessary, it is recommended to contact My Oracle Support (MOS) for guidance.

    If the health check log contains the Unable to Execute Health Check on <Active NOAM hostname> message, perform the health checks in Alternate SOAM Post-Upgrade Health Check procedure.

6.7.2.4 Active SOAM VIP: Export and Archive the Diameter Configuration Data
  1. Navigate to Diameter Common, then Export.
  2. Capture and archive the Diameter data by selecting the All option for the Export Application.
  3. Verify the requested data is exported by clicking Tasks at the top of the screen.
  4. Navigate to Status & Manage, then Files and download all the exported files to the client machine, or use the SCP utility to download the files from the active NOAM to the client machine.
  5. Navigate to Diameter, then Maintenance, and then Applications.
  6. Verify Operational Status is Available for all applications.
6.7.2.5 Active SOAM Server: Verify if the Setup has an Apache Certificate
Check if the setup has a customer supplied Apache certificate installed and is protected with a passphrase, which was renamed before starting with upgrade.
  1. If the setup has a customer-supplied Apache certificate installed and is protected with passphrase before the start of the upgrade. Refer to Verification of Configuration Data procedure and rename the certificate back to the original name.
6.7.2.6 Active SOAM Server: Compare Health Check Data to PRe-Upgrade Health Check Data
Verify that the health check status of the upgraded site is the same as the preupgrade health checks taken in Preupgrade Health Checks section. If system operation is degraded, it is recommended to contact My Oracle Support (MOS).
6.7.2.7 Alternate SOAM Post-Upgrade Health Check
This procedure determines the validity of the upgrade, as well as the health and status of the network and servers. This procedure is an alternative to the normal post upgrade health check in Post-Upgrade Procedures section.
6.7.2.7.1 Active SOAM CLI: Run/verify SOAM Post-upgrade Health Check Status
  1. Use an SSH client to connect to the active SOAM:ssh admusr@<SOAM XMI IP address>password: <enter password>

    Note:

    The static XMI IP address for each server should be available in Logins, Passwords, and Server IP Addresses section.
  2. Enter the command: $ upgradeHealthCheck postUpgradeHealthCheckOnSoam

    This command creates two files in /var/TKLC/db/filemgmt/ UpgradeHealthCheck/ with the filename format:<SOserver_name>_ServerStatusReport_<date-time>.xml<SOserver_name>_ComAgentConnStatusReport_<date-time>.xml

    If any alarms are present in the system: <SOserver_name>_AlarmStatusReport_<date-time>.xml

    If the system is PDRA, one additional file is generated: <SOserver_name>_SBRStatusReport_<date-time>.xml

    Note:

    The FIPS integrity verification test failed message may display when the upgradeHealthCheck command runs. This message can be ignored.
  3. If the Server <hostname> needs operator attention before upgrade message displays, inspect the Server Status Report to determine the reason for the message. If the Server <hostname> has no alarm with DB State as Normal and Process state as Kill message displays in the Server Status Report, the alert can be ignored.

    Note:

    If any server status is not as expected, do not proceed with the upgrade. It is recommended to contact My Oracle Support (MOS) for guidance.
  4. Keep these reports for future reference. These reports are compared to alarm and status reports after the upgrade is complete.
6.7.2.7.2 Active SOAM CLI: Capture Diameter Maintenance Status
  1. Enter the command:$ upgradeHealthCheck diameterMaintStatus

    This command displays a series of messages providing Diameter Maintenance status. Capture this output and save for later use.

    Note:

    The output is also captured in /var/TKLC/db/filemgmt/UpgradeHealthCheck.log. The FIPS integrity verification test failed message may display when the upgradeHealthCheck command runs. This message can be ignored.
6.7.2.7.3 Active SOAM CLI: View DA-MP Status
  1. Enter the command:$ upgradeHealthCheck daMpStatus

    This command outputs status to the screen for review.

    Note:

    Note: The FIPS integrity verification test failed message may display when the upgradeHealthCheck command runs. This message can be ignored.
  2. Verify all peer MPs are available.
  3. 3. Note the number of Total Connections Established.
6.7.2.7.4 Compare Data to the Pre-Upgrade Health Check

Compare data to the pre-upgrade health check to verify if the system has degraded after the second maintenance window.

  1. Compare data to the pre-upgrade health check to verify if the system has degraded after the second maintenance window Verify the health check status of the upgraded site as collected in this procedure is the same as the pre-upgrade health checks taken in Pre-Upgrade Health Checks section If system operation is degraded, it is recommended to report it to My Oracle Support (MOS).

    Note:

    If another site is to be upgraded, all procedures specified in Site Pre-Upgrade Activities section must be executed. However, the user should be aware that mated sites should not be upgraded in the same maintenance window.

6.7.3 Post-Upgrade Procedures

The procedures in this section are to be executed after the site upgrade is verified as valid and healthy. These procedures should be executed in the maintenance window.

Active SOAM VIP: Enable the Signaling Firewall for the Upgraded Site

The firewall enables the DSR to dynamically determine and customize the Linux firewall on each DA-MP server in the DSR Signaling node to allow only the essential network traffic pertaining to the active signaling configuration. There are some limitations related to enabling the signaling firewall in DSR 8.6.0.7.0 and later releases. See Review Release Notes for more details.

  1. Navigate to Diameter, then Maintenance, and then Signaling Firewall.
  2. Select the Signaling Node that was just upgraded.
  3. Click Enable.
  4. Click OK to confirm the action.
  5. Verify the Admin State changes to Enabled.

    Note:

    There may be a short delay while the firewall is enabled on the site.

Footnote Legend

Footnote 1:

/var/TKLC/log/upgrade/upgrade.log

/var/TKLC/log/upgrade/ugwrap.log

/var/TKLC/log/upgrade/earlyChecks.log

/var/TKLC/log/platcfg/upgrade.log