Upgrade Planning

3 Upgrade Planning

This section contains all the necessary information to carry out an upgrade. The materials required to perform an upgrade are described, as are pre-upgrade procedures that should be run to ensure the system is fully ready for upgrade.

The stated time durations for each step or group of steps are estimates only. Do not use the overview tables to execute any actions on the system. Only the procedures should be used when performing upgrade actions, beginning with the procedures in Check Required Materials.

Caution:

For vSTP-related deployments, it is not allowed to do any adding/updating/deleting of configuration until the upgrade is completed on all sites and accepted.

Note:

Be aware that once the upgrade starts and OAM level servers are on different releases than different sites, OAM level provisioning data is not replicated to sites that have not been upgraded. After the upgrade is completed, replication from OAM level server is restored and upgraded servers start receiving provisioning data.

Refer to Automated Site Upgrade section for details and limitations/solutions while planning upgrade cycles.

There are some limitations with upgrading the DC server in a C-level server group that are upgraded in a group of servers, for example, DA-MP, vSTP MP(s). While manually upgrading, ensure the DC server is not upgraded in the first upgrade cycle of the C-Level servers in its server group. Identify the DC server using Appendix N and Identify the DC server. In all cases, regardless of the number of cycles used to upgrade the DA-MP/vSTP server group, the DA-MP leader/vSTP MP leader should be the last server upgraded. By upgrading the MP leader last, the number of leader changes is minimized during the upgrade.

You can access the DA-MP leader by navigating to the Diameter Maintenance, then DA-MPs, and then Peer DA-MP Status active SOAM, where MP Leader = Yes. Also, check for the MP leader on the vSTP. This is done on the active SOAM CLI.

From the MMI command using the REST Client for the vSTP configuration. Navigate to Main Menu, then MMI Guide to access the user guide.
Use the /vstp/mpleader MO. The result is the host name of the MP leader server.

Note:

If the Alarm 31149 - DB Late Write Nonactive appears on the screen, ignore it. This alarm does not have any effect on functionality.

3.1 Data Required for Upgrade

The following materials and information are needed to execute an upgrade:

Target-release application ISO image file or target-release application media

The capability to log into the network OAM servers with administrator privileges

User logins, passwords, IP addresses, and other administration information. For more details, see Table 3-1

VPN access to the customer’s network is required if that is the only method to log in to the OAM servers.

Note:

All logins into the DSR NOAM servers are made using the external management VIP unless otherwise stated.

3.1.1 Application ISO Image File

Obtain a copy of the target release and ISO image file or media. This file is necessary to perform the upgrade. The DSR ISO image file name is in the following format (version changes from release to release):

DSR-8.6.0.7.0_96.34.0.iso

Note:

Before the execution of this upgrade procedure, it is assumed that the DSR ISO image file has already been delivered to the customer’s premises. The ISO image file must reside on the local workstation used to perform the upgrade, and any user performing the upgrade must have access to the ISO image file. If the user performing the upgrade is at a remote location, it is assumed the ISO file is already available before starting the upgrade procedure.

The ISO is deployed as part of the pre-upgrade activities in Table 4-1.

3.1.2 Logins, Passwords, and Server IP Addresses

The following table identifies the information that is called out in the upgrade procedures such as server IP addresses and login credentials. While all of the information mentioned in the following table is required to complete the upgrade, there may be security policies in place that prevent the actual recording of this information in hard-copy form.

Table 3-1 Logins, Passwords, and Server IP Addresses

Item	Description
Target Release	Target DSR Upgrade Release
Credentials	GUI Admin Username
	GUI Admin Password
	DSR admusr Password
	DSR Root Password
VPN Access Details	Customer VPN Information (if needed)
NOAM	XMI VIP address
	NOAM 1 XMI IP Address
	NOAM 2 XMI IP Address
SOAM	XMI VIP address
	SOAM 1 XMI IP Address ( Site 1)
	SOAM 2 XMI IP Address (Site 1)
	PCA (DSR) Spare System OAM&P server – Site 1 Spare in Site 2, XMI IP Address
	SOAM 1 XMI IP Address ( Site 2)
	SOAM 2 XMI IP Address ( Site 2)
	PCA (DSR) Spare System OAM&P server – Site 2 Spare in Site 1, XMI IP Address
Binding SBR Server Groups	Binding SBR SR1 Server Group Servers (Site 1)
	Binding SBR SR2 Server Group Servers (Site 1)
	Binding SBR SR3 Server Group Servers (Site 1)
	Binding SBR SR4 Server Group Servers (Site 1)
PCA MP Server Group	PCA MP Server Group Servers (Site 1)
PCA MP Server Group	PCA MP Server Group Servers (Site 1)
IPFE Server Groups(For PDRA)	PCA IPFE A1 Server Group Server (Site 1)
	PCA IPFE A 2 Server Group Server (Site 1)
	PCA IPFE B 1 Server Group Server (Site 1)
	PCA IPFE B 2 Server Group Server (Site 1)
Binding SBR Server Groups	Binding SBR SR1 Server Group Servers (Site 2)
	Binding SBR SR2 Server Group Servers (Site 2)
	Binding SBR SR3 Server Group Servers (Site 2)
	Binding SBR SR4 Server Group Servers (Site 2)
PCA MP Server Group	PCA MP Server Group Servers (Site 2)
IPFE Server Groups (For PCA)
	PCA IPFE A2 Server Group Server (Site 2)
	PCA IPFE B 1 Server Group Server (Site 2)
	PCA IPFE B 2 Server Group Server (Site 2)
vSTP MP Server Group	vSTP MP server(s)
Software	Target Release Number
Software	ISO Image (.iso) file name
Misc	Miscellaneous additional data

3.1.2.1 Expired Password Workaround Procedure

This section provides the procedures to handle password expiration during upgrade. This procedure is a temporary workaround to allow an expired password to be used on a non-upgrade site. This procedure is provided as a workaround when a password expires after the NOAM has been upgraded and before all sites have been upgraded. The workaround must be removed using Expired Password Workaround Removal Procedure after the site is upgraded. Failure to remove the workaround inhibits password aging on the server.

3.1.2.1.1 Inhibit Password Aging

The following procedure enacts a workaround that inhibits password aging on the SOAM. This procedure should be used only when the following conditions apply:

An upgrade is in progress.

The NOAMs have been upgraded, but one or more sites have not been upgraded.

A login password has expired on a non-upgraded site.

Once the workaround is enacted, no passwords expire at that site. Remove the workaround once the site is upgraded.

3.1.2.1.1.1 Expired Password Workaround Removal Procedure

Active SOAM CLI: SSH to Active SOAM Server. Disable Password Aging

Use the SSH command (on UNIX systems – or putty if running on windows) to log into the active SOAM of the first non-upgraded site:

ssh admusr@<SOAM_VIP>

password: <enter password>

Answer yes if you are asked to confirm the identity of the server.
Create a text file with the following content (exactly as formatted):

[production]

aw.policy.pwchange.isExpired =

aw.policy.db.checkPw =

[development : production]

[test : development]
Save the file as:

/var/TKLC/appworks/ini/pw.ini
Change the file permissions:

sudo chmod 644 pw.ini
Run the following command:
clearCache

Note:
For each server on which this workaround is enacted, the old expired password must be used for login. The new password used on the NOAM does not work on these servers.

Note:
Repeat this step for the standby SOAM and all non-upgraded sites.

3.1.2.1.2 Enable Password Aging

The following procedure removes the password expiration workaround enabled in the Inhibit Password Aging procedure.

Active SOAM CLI: SSH to Active SOAM Server. Re-enable Password Aging.

Use the SSH command (on UNIX systems – or putty if running on windows) to log into the active SOAM of the first non-upgraded site:

ssh admusr@<SOAM_VIP>

password: <enter password>

Answer yes if you are asked to confirm the identity of the server.
Delete the pw.ini file:

$ sudo rm /var/TKLC/appworks/ini/pw.ini
Run this command:
```
$ sudo clearCache 
```
Repeat sub-steps 1 to 3 for the standby SOAM.

Note:
Repeat this procedure for all non-upgraded sites.

3.1.2.1.3 Password Reset

The following procedure resets the GUI Admin (guiadmin) password on the NOAM. In a backout scenario where the password expired during the upgrade, it is possible for the customer to get locked out due to global provisioning being disabled. When this happens, this procedure can be used to reset the password to gain access to the GUI.

Active NOAM CLI: SSH to Active NOAM Server. Reset the Password

Use the SSH command (on UNIX systems – or PuTTY if running on windows) to log into the active NOAM:

ssh admusr@<NOAM_VIP>

password: <enter password>

Answer yes if you are asked to confirm the identity of the server.
Run the reset command:
$ sudo /usr/TKLC/appworks/sbin/resetPassword guiadmin
At the Enter new Password for guiadmin prompt, enter a new password.
Attempt to log in to the NOAM GUI using the new password. If the login is not successful, it is recommended to contact My Oracle Support (MOS) for guidance.

3.2 Upgrade Maintenance Windows

This section provides a high-level checklist to aid in tracking individual server upgrades. The servers are grouped by maintenance window, and all servers in a group are expected to be successfully upgrade in a single maintenance window. Use this high-level checklist to upgrade maintenance windows together with the detailed procedures that appear later in this document.

Figure 3-1 Upgrade Maintenance Windows for 3-Tier Upgrade

Note:

Mated SOAM sites must be upgraded in separate maintenance windows.

3.2.1 Calculating Maintenance Windows

You can calculate the number of maintenance windows required for DSR setup and upgrade using the Maintenance Window Analysis Tool. For more information, see Maintenance Window Analysis Tool . It takes setup details as input from the user and accordingly calculates the number of maintenance windows required for upgrade. The spreadsheet also specifies in detail which servers need to be upgraded in which maintenance window. Complete DSR upgrade maintenance window details and timings can be found in Maintenance Window Analysis Tool.

3.3 Site Upgrade Methodology

There are three primary methods for upgrading a DSR site:

Auto Site Upgrade
Auto Server Group Upgrade
Manual Upgrade

The Auto Site Upgrade is the easiest and most efficient site upgrade method; however, it is not suitable for all customers or all configurations. The Auto Server Group Upgrade incorporates many of the conveniences of Auto Site Upgrade, but provides more control of the upgrade process to the customer. of the upgrade process. The Automated Site Upgrade supports 0% availability that requires the least amount of time to upgrade the sites. This can be achieved by changing the following parameters: Site Upgrade SOAM Method setting to 0 - Changing the Site Upgrade SOAM Method setting to 0 causes the standby SOAM and the spare SOAM(s) to be upgraded serially. With this mode, the SOAM upgrade could take as many as four cycles to complete (that is, spare – spare – standby – active). If there are no spare SOAMs, then this setting has no effect on the SOAM upgrade. Site Upgrade Bulk Availability setting to 0 - Changing the Site Upgrade Bulk Availability setting to 0 equates to 0% availability that means no servers are required to stay in service during the upgrade. This setting requires the minimum number of cycles, thus the least amount of time. This setting allows all of the DA-MPs to be upgraded at once .

Figure 3-2 Select Site Upgrade Methodology

Again, Auto Server Group Upgrade is not for all customers or all configurations. The manual upgrade method gives maximum control to the customer and can be used for all configurations. The users can utilize a combination of upgrade methods to upgrade a given site to maximize efficiency with customer peace of mind. The following table is a checklist for determining which upgrade method meets the needs of the customer while ensuring compatibility with the DSR configuration. Upon completion of the checklist, a recommended upgrade method is identified.

Table 3-2 Traffic Analysis Checklist

	Criteria	Notes
1	Do any of the site’s DA-MPs have fixed diameter connections to any peer node, similar to this depiction?	Automated Site Upgrade and Automated Server Group Upgrade, by default, do not consider fixed peer connections when selecting servers to upgrade. It is possible that all DA-MPs servicing a given peer (such as DA-MPs 1 and 3) could be upgraded simultaneously, thereby isolating the peer. For this reason, analyze the generic upgrade plan generated by the Automated Site Upgrade and Auto Server Group Upgrade carefully to ensure all DA-MPs servicing a given peer are not upgraded simultaneously. If the generic plan has the DA-MPs upgrading simultaneously, you must rearrange the upgrade and/or add cycles as necessary to develop a suitable plan. If yes, proceed to Rearrange Automated Site Upgrade Cycles to rearrange or add cycles for ASU or proceed to step 7 for a manual upgrade. If no, continue with step 2.
2	If peer nodes are configured via IPFE TSAs, are there any TSAs that are not distributed across all DA-MPs, similar to this depiction?	Automated Site Upgrade and Automated Server Group Upgrade, by default, do not consider non-uniformly distributed TSAs when selecting servers to upgrade. It is possible that all DA-MPs servicing a given TSA (such as DA-MPs 1 and 2) could be upgraded simultaneously, thereby isolating the peer. For this reason, analyze the generic upgrade plan generated by the Automated Site Upgrade and Auto Server Group Upgrade carefully to ensure all DA-MPs servicing a given TSA are not upgraded simultaneously. If the generic plan has the DA-MPs upgrading simultaneously, you must rearrange the upgrade and/or add cycles as necessary to develop a suitable plan. If yes, proceed to Rearrange Automated Site Upgrade Cycles to rearrange or add cycles for ASU or proceed to step 7 for a manual upgrade. If no, continue with step 3.
3	Do any of the site’s DA-MPs have specialized distribution of DSR features, similar to this depiction?	Automated Site Upgrade and Automated Server Group Upgrade, by default, do not consider non-uniform distribution of features when selecting servers to upgrade. It is possible that all DA-MPs hosting a given feature (such as DCA) could be upgraded simultaneously, thereby eliminating service functionality. For this reason, analyze the generic upgrade plan generated by the Automated Site Upgrade and Auto Server Group Upgrade carefully to ensure all DA-MPs hosting a given feature are not upgraded simultaneously. If the generic plan has the DA-MPs upgrading simultaneously, you must rearrange the upgrade and/or add cycles as necessary to develop a suitable plan. If yes, proceed to Rearrange Automated Site Upgrade Cycles to rearrange or add cycles for ASU or proceed to step 7 for manual upgrade. If no, continue with step 4.
4	Automated Site Upgrade is a candidate for this system. Automated Site Upgrade supports 50% minimum server availability by default. A general option allows availability percentage settings of 66% or 75%. Is 50%, 66%, or 75% server availability during upgrade acceptable to the customer?	In general, a higher minimum availability setting increases the time required to upgrade a site. On the other hand, a lower minimum availability may reduce operational redundancy during the upgrade. If none of the minimum availability options are acceptable, Automated Site Upgrade should not be used to upgrade the site. If yes, continue with step 6. If no, proceed to step 7.
5	Is the customer comfortable with minimum user intervention (that is, user input) during the upgrade?	Once initiated, Automated Site Upgrade requires no additional user input to complete the upgrade. User control is limited to canceling the site upgrade task. If yes, Automated Site Upgrade is the recommended upgrade method. If no, proceed to step 7.
6	Automated Server Group Upgrade is a candidate for this system. Is the customer comfortable with the level of control afforded by the Automated Server Group upgrade?	Auto Server Group upgrade allows the user to initiate the upgrade of each server group, while the individual servers within the server group upgrade automatically. If yes, Auto Server Group upgrade is the recommended upgrade method. If no, proceed to step 7.
7	A manual upgrade affords the maximum level of control over upgrade sequencing. With this method, the upgrade of each server is individually initiated, allowing the user to control the level of parallelism and speed of the upgrade. Note: A site upgrade can include a combination of Automated Server Group upgrade and manual upgrades to improve efficiency. For example, SBRs can be upgraded with Automated Server Group upgrade, while the DA-MPs may be upgraded manually to control the order of upgrade for traffic continuity.	A manual upgrade is the recommended upgrade method.

3.3.1 DA-MP Upgrade Planning

If a manual upgrade is recommended by the Traffic Analysis Checklist, additional planning is required to ensure a successful upgrade of the DA-MP server group. A manual upgrade is typically required/recommended when the DA-MPs are configured in a way such that an upgrade could result in a traffic outage. Pre-planning the upgrade of the DA-MPs is key to avoiding an outage.

Note:

If complete site upgrade is selected with 0% availability, then DA-MP upgrade planning is not required.

Table 3-3 is an aid to laying out the sequence of the DA-MP upgrades, taking into consideration configuration and traffic continuity. This worksheet must be completed by the customer and provided to Oracle if Oracle personnel are performing the upgrade. It is highly recommended that the worksheet be completed for customer-driven upgrades as well.

Customers need to perform an analysis of the diameter application and connection configurations to assess any potential traffic loss due to the DA-MP upgrade. Complete the worksheet, specifying the order in which the DA-MPs will be upgraded, and which MPs, if any, can be upgraded in parallel.

The worksheet is divided into four upgrade Cycles. Each cycle represents an upgrade period during which one or more servers are upgraded. Distributing the DA-MPs servers over two or more cycles, takes advantage of parallels, thereby reducing the time required to upgrade the entire server group. To achieve 50% server availability, half of host names would be listed in Cycle 1 while the other half would be listed in Cycle 2, requiring two upgrade cycles. Similarly, 75% availability can be achieved by spreading the host name over all four cycles.

In all cases, regardless of the number of cycles used to upgrade the DA-MP/vSTP server group, the DA-MP leader/vSTP MP leader should be the last server upgraded. By upgrading the MP leader last, the number of leader changes is minimized during the upgrade.

You can access the DA-MP leader by navigating to Diameter, then Maintenance, then DA-MPs, and then Peer DA-MP Status on the active SOAM, where MP Leader = Yes. Also, check for the MP leader on the vSTP. This is done on the active SOAM CLI from the MMI command using the REST Client for the vSTP configuration.

Navigate to Main Menu, then MMI Guide to access the MMI user.
Use the the /vstp/mpleader MO to obtain the host name of the MP Leader server.

Note:

If needed, the users can upgrade DA-MPs serially, in which case, all host names would be listed in cycle 1. List the DA-MPs in the order in which they will be upgraded.

Table 3-3 DA-MP Upgrade Worksheet

	Host Names
Upgrade Cycle 1 or Serial Upgrade



Upgrade Cycle 2



Upgrade Cycle 3


Upgrade Cycle 4

3.3.2 Pre-upgrade Validation

The HA framework enhancements cause the inter-connectivity issue between the old-DC and non-DC MP nodes during the upgrade scenario. To overcome the inter-connectivity issue:

Note:

This procedure resolves the inter-connectivity issue between the old-DC and non-DC MP at the time of upgrade for the BUG 27428669.

Check the Designated Coordinator (DC) node in the system by using the command:ssh admusr@<MP_server> $ ha.info –d
Before starting the MP server upgrade, disable the DSR application on current DC node, using the following command:
1. On Active SOAM, go to Status & Manage, then Server.
2. Disable the DSR application by selecting the MP (DC Node) and click Stop.
Select an MP to be upgraded:
1. In cases where there is existing IPFE-based floating (Diameter) connections, choose an MP from TSA with more than 2 MPs. If there exists a TSA with just two MPs, and one having DC role, you should avoid using other MP (non-DC) in this TSA for upgrade at this step.
2. In cases where there are MP based (Diameter) connection, select any MP except the MP having with DC role.
After upgrade, one of the upgraded MP with new release takes over the new -DC role.
The DSR application remains disabled on the old-DC node, as performed in Step 2.
The old-DC is upgraded in the next upgrade cycle.
When the upgrade is completed, from Active SOAM, navigate to Status & Manage GUI , then Server and check if the DSR application is Enabled on MP node (old-DC). If not, then Enable it by clicking restart.

3.3.3 Maintenance Window 1 for NOAM Site Upgrades

In the first maintenance window, the NOAM servers are upgraded .

Maintenance Window 1

(NOAM Sites)

Date:

Note: You can view the form in DSR NOAM GUI by clicking Configuration, then Network Elements.

Record the Site NE Name of the DSR NOAM to be upgraded during Maintenance Window 1 in the space provided below. Verify if the upgrade is completed for the following servers:

DR Standby NOAM (Guest):

DR Active NOAM (Guest):

Primary Standby NOAM (Guest):

Primary Active NOAM (Guest):

3.3.4 Maintenance Window 2 for SOAM Site Upgrades and Rest of the Servers

During Maintenance Window 2, all servers associated with the first SOAM site are upgraded. All servers associated with the second SOAM site are upgraded during Maintenance Window 3. For DSRs configured with multiple mated-pair sites, or DSRs having multiple, distinct sites (for example, georedundant PCA installations), copy and use the following form for the subsequent SOAM site upgrades.

Caution:

It is recommended that mated-pair SOAM sites are not upgraded in the same Maintenance Window.

Maintenance Window	Steps
SOAM Sites Date:	Record the site NE Name of the DSR SOAM and the MP(s) to be upgraded during Maintenance Window 2 in the space provided. Verify if the upgrade is completed for the following sites: SOAM Site: Spare SOAM1 (Guest): (if equipped) Spare SOAM2 (Guest): (if equipped): Standby SOAM (Guest): Active SOAM (Guest):
	DA-MP 1: DA-MP 2: DA-MP 3: DA-MP 4: DA-MP 5: DA-MP 6: DA-MP 7: DA-MP 8: DA-MP 9: DA-MP 10: DA-MP 11: DA-MP 12: DA-MP 13: DA-MP 14: DA-MP 15: DA-MP 16:
	IPFE1: IPFE 2: IPFE 3: IPFE 4:
	Binding Server Group 1 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 2 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 3 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 4 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 5 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 6 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 7 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 8 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped)
	Session Server Group 1 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 2 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 3 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 4 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 5 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 6 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 7 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 8 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped)
	vSTP MP Server Group vSTP MP(s): (If equipped)

3.3.5 IDIH Preupgrade

If IDIH is a component of a Network Element, it should be upgraded only after DSR is upgraded. However, note that certain compatibility limitations may exist while the two components (DSR and IDIH) are not on the compatible release. The IDIH upgrade procedures are provided in IDIH Upgrade at a Site and may be performed at any time after adding the Mediation and Application OVAs as described in IDIH Upgrade Preparation.

3.3.5.1 IDIH Upgrade Preparation

Follow the hypervisor’s instructions to add the Mediation and Application OVAs to the cloud repository.