3 Upgrade Planning

This section contains all the necessary information to carry out an upgrade. The materials required to perform an upgrade are described, as are pre-upgrade procedures that should be run to ensure the system is fully ready for upgrade.

The stated time durations for each step or group of steps are estimates only. Do not use the overview tables to execute any actions on the system. Only the procedures should be used when performing upgrade actions, beginning with the procedures in Check Required Materials.

Caution:

For vSTP-related deployments, it is not allowed to do any adding/updating/deleting of configuration until the upgrade is completed on all sites and accepted.

Note:

Be aware that once the upgrade starts and OAM level servers are on different releases than different sites, OAM level provisioning data is not replicated to sites that have not been upgraded. After the upgrade is completed, replication from OAM level server is restored and upgraded servers start receiving provisioning data.

Refer to Automated Site Upgrade section for details and limitations/solutions while planning upgrade cycles.

There are some limitations with upgrading the DC server in a C-level server group that are upgraded in a group of servers, for example, DA-MP, vSTP MP(s). While manually upgrading, ensure the DC server is not upgraded in the first upgrade cycle of the C-Level servers in its server group. Identify the DC server using Appendix N and Identify the DC server. In all cases, regardless of the number of cycles used to upgrade the DA-MP/vSTP server group, the DA-MP leader/vSTP MP leader should be the last server upgraded. By upgrading the MP leader last, the number of leader changes is minimized during the upgrade.

You can access the DA-MP leader by navigating to the Diameter Maintenance, then DA-MPs, and then Peer DA-MP Status active SOAM, where MP Leader = Yes. Also, check for the MP leader on the vSTP. This is done on the active SOAM CLI.

1. From the MMI command using the REST Client for the vSTP configuration. Navigate to Main Menu, then MMI Guide to access the user guide.
2. Use the /vstp/mpleader MO. The result is the host name of the MP leader server.

Note:

• If the Alarm 31149 - DB Late Write Nonactive appears on the screen, ignore it. This alarm does not have any effect on functionality.
• If the upgrade is required from 8.x VM to 9.0.2, refer to Dual Hop Upgrade from DSR-8.x to DSR-9.0.2 Using Ansible section.

3.1 Upgrade Check

• It is recommended to upgrade the SDS topology (NOAMs, SOAMs, DPs) before the DSR topology. See SDS Software Upgrade Guide for SDS upgrade documentation.

  Caution:

  SDS Upgrade- If the customer deployment has both the FABR and PCA features enabled, then upgrade the DSR nodes first before upgrading the SDS nodes.
• If DSA is used with UDR, then it is recommended to upgrade the UDR topology (NOAMs, SOAMs, DPs) before the DSR topology. See UDR Cloud Software Upgrade Guide for UDR upgrade documentation.

  Caution:

  UDR Upgrade- If the customer deployment has both the FABR and PCA features enabled, then upgrade the DSR nodes first before upgrading the UDR nodes.

3.2 Data Required for Upgrade

The following materials and information are needed to run an incremental upgrade:

• Target-release application DIU ISO image file or target-release application media.

  Note:

  For a major upgrade, along with DIU ISO, the tar file and TPD OL7 DIU ISO is required.

• The capability to log into the network OAM servers with administrator privileges

• User logins, passwords, IP addresses, and other administration information. For more details, see Table 3-1

• VPN access to the customer’s network is required if that is the only method to log in to the OAM servers.

Note:

All logins into the DSR NOAM servers are made using the external management VIP unless otherwise stated.

3.2.1 Application ISO Image File

Obtain a copy of the target release and ISO image file or media. This file is necessary to perform the upgrade. The DSR ISO image file name is in the following format (version changes from release to release):

DSR-9.0.2.1.0-99.16.0.iso

Note:

Before the execution of this upgrade procedure, it is assumed that the DSR ISO image file has already been delivered to the customer’s premises. The ISO image file must reside on the local workstation used to perform the upgrade, and any user performing the upgrade must have access to the ISO image file. If the user performing the upgrade is at a remote location, it is assumed the ISO file is already available before starting the upgrade procedure.

The ISO is deployed as part of the pre-upgrade activities in Table 4-1.

3.2.2 Logins, Passwords, and Server IP Addresses

The following table identifies the information that is called out in the upgrade procedures such as server IP addresses and login credentials. While all of the information mentioned in the following table is required to complete the upgrade, there may be security policies in place that prevent the actual recording of this information in hard-copy form.

Table 3-1 Logins, Passwords, and Server IP Addresses

Item	Description
Target Release	Target DSR Upgrade Release
Credentials	GUI Admin Username
	GUI Admin Password
	DSR admusr Password
	DSR Root Password
VPN Access Details	Customer VPN Information (if needed)
NOAM	XMI VIP address
	NOAM 1 XMI IP Address
	NOAM 2 XMI IP Address
SOAM	XMI VIP address
	SOAM 1 XMI IP Address ( Site 1)
	SOAM 2 XMI IP Address (Site 1)
	PCA (DSR) Spare System OAM&P server – Site 1 Spare in Site 2, XMI IP Address
	SOAM 1 XMI IP Address ( Site 2)
	SOAM 2 XMI IP Address ( Site 2)
	PCA (DSR) Spare System OAM&P server – Site 2 Spare in Site 1, XMI IP Address
Binding SBR Server Groups	Binding SBR SR1 Server Group Servers (Site 1)
	Binding SBR SR2 Server Group Servers (Site 1)
	Binding SBR SR3 Server Group Servers (Site 1)
	Binding SBR SR4 Server Group Servers (Site 1)
PCA MP Server Group	PCA MP Server Group Servers (Site 1)
	PCA MP Server Group Servers (Site 1)
IPFE Server Groups(For PDRA)	PCA IPFE A1 Server Group Server (Site 1)
	PCA IPFE A 2 Server Group Server (Site 1)
	PCA IPFE B 1 Server Group Server (Site 1)
	PCA IPFE B 2 Server Group Server (Site 1)
Binding SBR Server Groups	Binding SBR SR1 Server Group Servers (Site 2)
	Binding SBR SR2 Server Group Servers (Site 2)
	Binding SBR SR3 Server Group Servers (Site 2)
	Binding SBR SR4 Server Group Servers (Site 2)
PCA MP Server Group	PCA MP Server Group Servers (Site 2)
IPFE Server Groups (For PCA)
	PCA IPFE A2 Server Group Server (Site 2)
	PCA IPFE B 1 Server Group Server (Site 2)
	PCA IPFE B 2 Server Group Server (Site 2)
vSTP MP Server Group	vSTP MP server(s)
Software	Target Release Number
	ISO Image (.iso) file name
Misc	Miscellaneous additional data

3.2.2.1 Expired Password Workaround Procedure

This section provides the procedures to handle password expiration during upgrade. This procedure is a temporary workaround to allow an expired password to be used on a non-upgrade site. This procedure is provided as a workaround when a password expires after the NOAM has been upgraded and before all sites have been upgraded. The workaround must be removed using Expired Password Workaround Removal Procedure after the site is upgraded. Failure to remove the workaround inhibits password aging on the server.

3.2.2.1.1 Inhibit Password Aging

The following procedure enacts a workaround that inhibits password aging on the SOAM. This procedure should be used only when the following conditions apply:

• An upgrade is in progress.

• The NOAMs have been upgraded, but one or more sites have not been upgraded.

• A login password has expired on a non-upgraded site.

Once the workaround is enacted, no passwords expire at that site. Remove the workaround once the site is upgraded.

3.2.2.1.1.1 Expired Password Workaround Removal Procedure

Active SOAM CLI: SSH to Active SOAM Server. Disable Password Aging

1. Use the SSH command (on UNIX systems – or putty if running on windows) to log into the active SOAM of the first non-upgraded site:

   ssh admusr@<SOAM_VIP>

   password: <enter password>

   Answer yes if you are asked to confirm the identity of the server.

2. Create a text file with the following content (exactly as formatted):

   [production]

   aw.policy.pwchange.isExpired =

   aw.policy.db.checkPw =

   [development : production]

   [test : development]

3. Save the file as:

   /var/TKLC/appworks/ini/pw.ini

4. Change the file permissions:

   sudo chmod 644 pw.ini

5. Run the following command:
   clearCache

   Note:

   For each server on which this workaround is enacted, the old expired password must be used for login. The new password used on the NOAM does not work on these servers.

   Note:

   Repeat this step for the standby SOAM and all non-upgraded sites.

3.2.2.1.2 Enable Password Aging

The following procedure removes the password expiration workaround enabled in the Inhibit Password Aging procedure.

Active SOAM CLI: SSH to Active SOAM Server. Re-enable Password Aging.

1. Use the SSH command (on UNIX systems – or putty if running on windows) to log into the active SOAM of the first non-upgraded site:

   ssh admusr@<SOAM_VIP>

   password: <enter password>

   Answer yes if you are asked to confirm the identity of the server.

2. Delete the pw.ini file:

   $ sudo rm /var/TKLC/appworks/ini/pw.ini

3. Run this command:

   $ sudo clearCache 

4. Repeat sub-steps 1 to 3 for the standby SOAM.

   Note:

   Repeat this procedure for all non-upgraded sites.

3.2.2.1.3 Password Reset

The following procedure resets the GUI Admin (guiadmin) password on the NOAM. In a backout scenario where the password expired during the upgrade, it is possible for the customer to get locked out due to global provisioning being disabled. When this happens, this procedure can be used to reset the password to gain access to the GUI.

Active NOAM CLI: SSH to Active NOAM Server. Reset the Password

1. Use the SSH command (on UNIX systems – or PuTTY if running on windows) to log into the active NOAM:

   ssh admusr@<NOAM_VIP>

   password: <enter password>

   Answer yes if you are asked to confirm the identity of the server.

2. Run the reset command:
   $ sudo /usr/TKLC/appworks/sbin/resetPassword guiadmin
3. At the Enter new Password for guiadmin prompt, enter a new password.
4. Attempt to log in to the NOAM GUI using the new password. If the login is not successful, it is recommended to contact My Oracle Support (MOS) for guidance.

3.3 MySql User Accounts Password

This section provides the procedure to check for the presence of any forbidden special characters in the mysql passwords for awadmin and root user accounts.

Mysql password can contain the following:

• Upper case alphabets (A-Z)
• Lower case alphabets (a-z)
• Digits (0-9)
• 21 allowed special characters

Allowed Special Characters

There are a total of 32 special characters on the standard qwerty keyboard. Out of these 32 special characters, 21 characters are supported in the MySql passwords.

The following table provides the list of these 21 allowed special characters.

Table 3-2 Allowed Special Characters

Allowed Special Characters	Name
#	Octothorpe or hash or pound sign
!	Exclamation point
~	Tilde
%	Percent
^	Caret or circumflex
*	Asterisk
_	Underscore
-	Hyphen or dash
+	Plus
=	Equal
?	Question Mark
{	Open Braces
}	Close Braces
(	Open Parenthesis
)	Close Parenthesis
<	Open angle bracket or less than
>	Close angle bracket or greater than
|	Pipe or Vertical bar
.	Dot
,	Comma
;	Semi Colon

Forbidden Special Characters

There are a total of 32 special characters on the standard qwerty keyboard. Out of these 32 special characters, 11 characters are currently not supported in the MySql passwords. Usage of these forbidden special characters in the password will set the incorrect password in the database of MySql Server.

The following table provides the list of these 11 forbidden special characters.

Table 3-3 Forbidden Special Characters

Forbidden Special Characters	Name
@	Ampersat
$	Dollar
&	Ampersand
`	Backtick or backquote or grave accent
\	Backslash
/	Forward slash
[	Open Square Bracket
]	Close Square Bracket
‘	Single quotation mark or apostrophe
“	Double quotation mark
:	Colon

3.3.1 Sanity Check on MySql Passwords

Perform the following procedure to sanity check MySql passwords.

1. Log in to the source server as admusr.

   Username: admusr
   Password: <current admin user password>

2. Verify the mysql passwords using the following commands.
   For awadmin user account:

   sudo /usr/TKLC/appworks/bin/aw.wallet credential get mysql default

   For root user account, use the following command:

   sudo /usr/TKLC/appworks/bin/aw.wallet credential get mysql root

3. If passwords contain forbidden special characters mentioned in the Table 3-3 table, then reset the mysql password using the allowed special character mentioned in the Table 3-2 table.

   Note:

   To reset the mysql password, see Updating the MySQL Password in DSR Security Guide.

3.4 Upgrade Maintenance Windows

This section provides a high-level checklist to aid in tracking individual server upgrades. The servers are grouped by maintenance window, and all servers in a group are expected to be successfully upgrade in a single maintenance window. Use this high-level checklist to upgrade maintenance windows together with the detailed procedures that appear later in this document.

Figure 3-1 Upgrade Maintenance Windows for 3-Tier Upgrade

Note:

Mated SOAM sites must be upgraded in separate maintenance windows.

3.4.1 Calculating Maintenance Windows

You can calculate the number of maintenance windows required for DSR setup and upgrade using the Maintenance Window Analysis Tool. For more information, see Maintenance Window Analysis Tool . It takes setup details as input from the user and accordingly calculates the number of maintenance windows required for upgrade. The spreadsheet also specifies in detail which servers need to be upgraded in which maintenance window. Complete DSR upgrade maintenance window details and timings can be found in Maintenance Window Analysis Tool.

3.5 Site Upgrade Methodology

There are three primary methods for upgrading a DSR site:

• Auto Site Upgrade
• Auto Server Group Upgrade
• Manual Upgrade

The Auto Site Upgrade is the easiest and most efficient site upgrade method; however, it is not suitable for all customers or all configurations. The Auto Server Group Upgrade incorporates many of the conveniences of Auto Site Upgrade, but provides more control of the upgrade process to the customer. of the upgrade process. The Automated Site Upgrade supports 0% availability that requires the least amount of time to upgrade the sites. This can be achieved by changing the following parameters: Site Upgrade SOAM Method setting to 0 - Changing the Site Upgrade SOAM Method setting to 0 causes the standby SOAM and the spare SOAM(s) to be upgraded serially. With this mode, the SOAM upgrade could take as many as four cycles to complete (that is, spare – spare – standby – active). If there are no spare SOAMs, then this setting has no effect on the SOAM upgrade. Site Upgrade Bulk Availability setting to 0 - Changing the Site Upgrade Bulk Availability setting to 0 equates to 0% availability that means no servers are required to stay in service during the upgrade. This setting requires the minimum number of cycles, thus the least amount of time. This setting allows all of the DA-MPs to be upgraded at once .

Figure 3-2 Select Site Upgrade Methodology

Again, Auto Server Group Upgrade is not for all customers or all configurations. The manual upgrade method gives maximum control to the customer and can be used for all configurations. The users can utilize a combination of upgrade methods to upgrade a given site to maximize efficiency with customer peace of mind. The following table is a checklist for determining which upgrade method meets the needs of the customer while ensuring compatibility with the DSR configuration. Upon completion of the checklist, a recommended upgrade method is identified.

Table 3-4 Traffic Analysis Checklist

	Criteria	Yes/No	Notes
1	Do any of the site’s DA-MPs have fixed diameter connections to any peer node, similar to this depiction?		Automated Site Upgrade and Automated Server Group Upgrade, by default, do not consider fixed peer connections when selecting servers to upgrade. It is possible that all DA-MPs servicing a given peer (such as DA-MPs 1 and 3) could be upgraded simultaneously, thereby isolating the peer. For this reason, analyze the generic upgrade plan generated by the Automated Site Upgrade and Auto Server Group Upgrade carefully to ensure all DA-MPs servicing a given peer are not upgraded simultaneously. If the generic plan has the DA-MPs upgrading simultaneously, you must rearrange the upgrade and/or add cycles as necessary to develop a suitable plan. If yes, proceed to Rearrange Automated Site Upgrade Cycles to rearrange or add cycles for ASU or proceed to step 7 for a manual upgrade. If no, continue with step 2.
2	If peer nodes are configured via IPFE TSAs, are there any TSAs that are not distributed across all DA-MPs, similar to this depiction?		Automated Site Upgrade and Automated Server Group Upgrade, by default, do not consider non-uniformly distributed TSAs when selecting servers to upgrade. It is possible that all DA-MPs servicing a given TSA (such as DA-MPs 1 and 2) could be upgraded simultaneously, thereby isolating the peer. For this reason, analyze the generic upgrade plan generated by the Automated Site Upgrade and Auto Server Group Upgrade carefully to ensure all DA-MPs servicing a given TSA are not upgraded simultaneously. If the generic plan has the DA-MPs upgrading simultaneously, you must rearrange the upgrade and/or add cycles as necessary to develop a suitable plan. If yes, proceed to Rearrange Automated Site Upgrade Cycles to rearrange or add cycles for ASU or proceed to step 7 for a manual upgrade. If no, continue with step 3.
3	Do any of the site’s DA-MPs have specialized distribution of DSR features, similar to this depiction?		Automated Site Upgrade and Automated Server Group Upgrade, by default, do not consider non-uniform distribution of features when selecting servers to upgrade. It is possible that all DA-MPs hosting a given feature (such as DCA) could be upgraded simultaneously, thereby eliminating service functionality. For this reason, analyze the generic upgrade plan generated by the Automated Site Upgrade and Auto Server Group Upgrade carefully to ensure all DA-MPs hosting a given feature are not upgraded simultaneously. If the generic plan has the DA-MPs upgrading simultaneously, you must rearrange the upgrade and/or add cycles as necessary to develop a suitable plan. If yes, proceed to Rearrange Automated Site Upgrade Cycles to rearrange or add cycles for ASU or proceed to step 7 for manual upgrade. If no, continue with step 4.
4	Automated Site Upgrade is a candidate for this system. Automated Site Upgrade supports 50% minimum server availability by default. A general option allows availability percentage settings of 66% or 75%. Is 50%, 66%, or 75% server availability during upgrade acceptable to the customer?		In general, a higher minimum availability setting increases the time required to upgrade a site. On the other hand, a lower minimum availability may reduce operational redundancy during the upgrade. If none of the minimum availability options are acceptable, Automated Site Upgrade should not be used to upgrade the site. If yes, continue with step 6. If no, proceed to step 7.
5	Is the customer comfortable with minimum user intervention (that is, user input) during the upgrade?		Once initiated, Automated Site Upgrade requires no additional user input to complete the upgrade. User control is limited to canceling the site upgrade task. If yes, Automated Site Upgrade is the recommended upgrade method. If no, proceed to step 7.
6	Automated Server Group Upgrade is a candidate for this system. Is the customer comfortable with the level of control afforded by the Automated Server Group upgrade?		Auto Server Group upgrade allows the user to initiate the upgrade of each server group, while the individual servers within the server group upgrade automatically. If yes, Auto Server Group upgrade is the recommended upgrade method. If no, proceed to step 7.
7	A manual upgrade affords the maximum level of control over upgrade sequencing. With this method, the upgrade of each server is individually initiated, allowing the user to control the level of parallelism and speed of the upgrade. Note: A site upgrade can include a combination of Automated Server Group upgrade and manual upgrades to improve efficiency. For example, SBRs can be upgraded with Automated Server Group upgrade, while the DA-MPs may be upgraded manually to control the order of upgrade for traffic continuity.		A manual upgrade is the recommended upgrade method.

3.5.1 DA-MP Upgrade Planning

If a manual upgrade is recommended by the Traffic Analysis Checklist, additional planning is required to ensure a successful upgrade of the DA-MP server group. A manual upgrade is typically required/recommended when the DA-MPs are configured in a way such that an upgrade could result in a traffic outage. Pre-planning the upgrade of the DA-MPs is key to avoiding an outage.

Note:

If complete site upgrade is selected with 0% availability, then DA-MP upgrade planning is not required.

Table 3-5 is an aid to laying out the sequence of the DA-MP upgrades, taking into consideration configuration and traffic continuity. This worksheet must be completed by the customer and provided to Oracle if Oracle personnel are performing the upgrade. It is highly recommended that the worksheet be completed for customer-driven upgrades as well.

Customers need to perform an analysis of the diameter application and connection configurations to assess any potential traffic loss due to the DA-MP upgrade. Complete the worksheet, specifying the order in which the DA-MPs will be upgraded, and which MPs, if any, can be upgraded in parallel.

The worksheet is divided into four upgrade Cycles. Each cycle represents an upgrade period during which one or more servers are upgraded. Distributing the DA-MPs servers over two or more cycles, takes advantage of parallels, thereby reducing the time required to upgrade the entire server group. To achieve 50% server availability, half of host names would be listed in Cycle 1 while the other half would be listed in Cycle 2, requiring two upgrade cycles. Similarly, 75% availability can be achieved by spreading the host name over all four cycles.

In all cases, regardless of the number of cycles used to upgrade the DA-MP/vSTP server group, the DA-MP leader/vSTP MP leader should be the last server upgraded. By upgrading the MP leader last, the number of leader changes is minimized during the upgrade.

You can access the DA-MP leader by navigating to Diameter, then Maintenance, then DA-MPs, and then Peer DA-MP Status on the active SOAM, where MP Leader = Yes. Also, check for the MP leader on the vSTP. This is done on the active SOAM CLI from the MMI command using the REST Client for the vSTP configuration.

• Navigate to Main Menu, then MMI Guide to access the MMI user.
• Use the the /vstp/mpleader MO to obtain the host name of the MP Leader server.

Note:

If needed, the users can upgrade DA-MPs serially, in which case, all host names would be listed in cycle 1. List the DA-MPs in the order in which they will be upgraded.

Table 3-5 DA-MP Upgrade Worksheet

	Host Names
Upgrade Cycle 1 or Serial Upgrade
Upgrade Cycle 2
Upgrade Cycle 3
Upgrade Cycle 4

3.5.2 Pre-upgrade Validation

The HA framework enhancements cause the inter-connectivity issue between the old-DC and non-DC MP nodes during the upgrade scenario. To overcome the inter-connectivity issue:

Note:

This procedure resolves the inter-connectivity issue between the old-DC and non-DC MP at the time of upgrade for the BUG 27428669.

1. Check the Designated Coordinator (DC) node in the system by using the command:ssh admusr@<MP_server> $ ha.info –d
2. Before starting the MP server upgrade, disable the DSR application on current DC node, using the following command:
   1. On Active SOAM, go to Status & Manage, then Server.
   2. Disable the DSR application by selecting the MP (DC Node) and click Stop.
3. Select an MP to be upgraded:
   1. In cases where there is existing IPFE-based floating (Diameter) connections, choose an MP from TSA with more than 2 MPs. If there exists a TSA with just two MPs, and one having DC role, you should avoid using other MP (non-DC) in this TSA for upgrade at this step.
   2. In cases where there are MP based (Diameter) connection, select any MP except the MP having with DC role.
4. After upgrade, one of the upgraded MP with new release takes over the new -DC role.
5. The DSR application remains disabled on the old-DC node, as performed in Step 2.
6. The old-DC is upgraded in the next upgrade cycle.
7. When the upgrade is completed, from Active SOAM, navigate to Status & Manage GUI , then Server and check if the DSR application is Enabled on MP node (old-DC). If not, then Enable it by clicking restart.

3.5.3 Maintenance Window 1 for NOAM Site Upgrades

In the first maintenance window, the NOAM servers are upgraded .

Maintenance Window 1 (NOAM Sites) Date: Note: You can view the form in DSR NOAM GUI by clicking Configuration, then Network Elements.

Record the Site NE Name of the DSR NOAM to be upgraded during Maintenance Window 1 in the space provided below. Verify if the upgrade is completed for the following servers: DR Standby NOAM (Guest): DR Active NOAM (Guest): Primary Standby NOAM (Guest): Primary Active NOAM (Guest):

3.5.4 Maintenance Window 2 for SOAM Site Upgrades and Rest of the Servers

During Maintenance Window 2, all servers associated with the first SOAM site are upgraded. All servers associated with the second SOAM site are upgraded during Maintenance Window 3. For DSRs configured with multiple mated-pair sites, or DSRs having multiple, distinct sites (for example, georedundant PCA installations), copy and use the following form for the subsequent SOAM site upgrades.

Caution:

It is recommended that mated-pair SOAM sites are not upgraded in the same Maintenance Window.

Maintenance Window	Steps
SOAM Sites Date:	Record the site NE Name of the DSR SOAM and the MP(s) to be upgraded during Maintenance Window 2 in the space provided. Verify if the upgrade is completed for the following sites: SOAM Site: Spare SOAM1 (Guest): (if equipped) Spare SOAM2 (Guest): (if equipped): Standby SOAM (Guest): Active SOAM (Guest):
	DA-MP 1: DA-MP 2: DA-MP 3: DA-MP 4: DA-MP 5: DA-MP 6: DA-MP 7: DA-MP 8: DA-MP 9: DA-MP 10: DA-MP 11: DA-MP 12: DA-MP 13: DA-MP 14: DA-MP 15: DA-MP 16:
	IPFE1: IPFE 2: IPFE 3: IPFE 4:
	Binding Server Group 1 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 2 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 3 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 4 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 5 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 6 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 7 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped) Binding Server Group 8 Standby SBR: Active SBR: Spare SBR 1 (Mate): Spare SBR2 (Mate): (If equipped)
	Session Server Group 1 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 2 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 3 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 4 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 5 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 6 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 7 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped) Session Server Group 8 Standby SBR: Active SBR: Spare SBR1 ( Mate): Spare SBR2 ( Mate): (If equipped)
	vSTP MP Server Group vSTP MP(s): (If equipped)

3.5.5 IDIH Preupgrade

If IDIH is a component of a Network Element, it should be upgraded only after DSR is upgraded. However, note that certain compatibility limitations may exist while the two components (DSR and IDIH) are not on the compatible release. The IDIH upgrade procedures provided in the IDIH Upgrade at a Site section can be performed at any time after adding the Mediation and Application OVAs as described in IDIH Upgrade Preparation.

3.5.5.1 IDIH Upgrade Preparation

Follow the hypervisor’s instructions to add the Mediation and Application OVAs to the cloud repository.

3.5.5.2 IDIH Upgrade at Site

In IDIH 7.1 and later, the mediation and application instance data is stored in the Oracle database. This allows the Application and Mediation servers to be upgraded by performing a fresh installation. After completion of the upgrade, the mediation and application guests automatically restore the configuration data from the Oracle database.

3.5.5.2.1 Upgrading IDIH Oracle Guest

Perform the following procedure to upgrade the IDIH Oracle Guest.

IDIH CLI: Perform a System Health Check on the Oracle Guest

Following are the steps to perform system health check on the Oracle Guest:

1. Log in to the Oracle Guest as a system administrator.

   ssh <IDIH IP address>

   login as: admusr

   password: <enter password>

2. Run the analyze_server.sh script.

   $ sudo /usr/TKLC/xIH/plat/bin/analyze_server.sh -i

   Sample output:

   [admusr@cat-ora ~]$ /usr/TKLC/xIH/plat/bin/analyze_server.sh -i

   13:24:52: STARTING HEALTHCHECK PROCEDURE

   13:24:52: date: 03-17-15, hostname: cat-ora

   13:24:52: TPD VERSION: 7.8.3.0.0-89.21.0

   13:24:52: -------------------------------------------

   13:24:52: Checking disk free space

   13:24:52: No disk space issues found

   :

   13:25:02: All tests passed!

   13:25:02: ENDING HEALTHCHECK PROCEDURE WITH CODE 0

   If the output indicates the following error, ignore the error and continue the upgrade. This error indicates the target release and the running release are the same.

   00:47:29: Checking runlevel

   00:47:29: >>> Error: Runlevel value "3 4" is different from "N 4"

   If the output indicates any other errors, do not proceed with the upgrade. It is recommended to contact My Oracle Support (MOS) for guidance.

3.5.5.2.1.1 IDIH CLI: Shut Down the Mediation Guest

Following are the steps to shut down the Mediation Guest:

1. Log in to the Mediation Guest as a system administrator.

   ssh <IDIH IP address>

   login as: admusr

   password: <enter password>

2. Run the following command to shut down the Mediation Guest:

   $ sudo init 0

   The active SOAM server may have some or all of the following expected alarms:

   • Alarm ID = 19800 Communication Agent Connection Down
   • Alarm ID = 11511 Unable to connect via Comagent to remote DIH server with hostname

   The active NOAM server may have the following expected alarm:

   • Alarm ID = 19800 Communication Agent Connection Down

3.5.5.2.1.2 IDIH CLI: Shut Down the Application Guest

Following are the steps to shut down the Application Guest:

1. Log in to the Application Guest as a system administrator.

   ssh <IDIH IP address>

   login as: admusr

   password: <enter password>

2. Run the following command to shut down the Application Guest:

   $ sudo init 0

   The active SOAM server may have some or all of the following expected alarms:

   Alarm ID = 19800 Communication Agent Connection Down

   Alarm ID = 11511 Unable to connect via Comagent to remote DIH server with hostname

   The active NOAM server may have the following expected alarm:

   Alarm ID = 19800 Communication Agent Connection Down

3.5.5.2.1.3 IDIH Application Guest CLI: Increase Size of /var/TKLC

Following are the steps to increase the size of /var/TKLC directory. It is seen that space available in/var/TKLC directory is less than the size of ISO. Hence there is a need to increase the space of this directory.

1. Log in to the Application Guest as a system administrator.

   ssh <IDIH IP address>

   login as: admusr

   password: <enter password>

2. Run the following command to check the space:

   df -kh /var/TKLC

   Note:

   The current space available should be more than 6 GB. In case sufficient space is already there, skip the next substeps.
3. Run the following command to increase the space:

   sudo lvresize -L +6G /dev/mapper/vgroot-plat_var_tklc

4. Run the following command to resize the space:

   sudo resize2fs /dev/mapper/vgroot-plat_var_tklc

5. Run the following command to check the space:

   df -kh /var/TKLC

3.5.5.2.1.4 Move Oracle ISO

Use a file transfer tool to copy the Oracle ISO to the Oracle Guest as system administrator.

Example:

$ scp oracle- DSR-8.6.0.1.0_96.15.0-x86_64.iso

admusr@<ora-guest-ip>:/var/TKLC/upgrade

3.5.5.2.1.5 IDIH CLI: Start Oracle Guest Upgrade

Following are the steps to upgrade the Oracle Guest using the Platform Configuration utility:

1. Launch the Platform Configuration utility by running the following command:

   $ sudo su - platcfg

2. In the Resulting menu, navigate to Maintenance, select Upgrade, and then click Initiate Upgrade.
3. At the ISO selection menu, select the target release Oracle ISO and click Enter to launch Platform Configuration utility.

   Figure 3-3 Upgrade Media Menu

   
   

   

   
   

3.5.5.2.1.6 IDIH CLI: Monitor Upgrade Progress

The Platform Configuration menu exits, and the guest restarts when the upgrade completes.

To view the detailed progress of the upgrade, access the server command line (through SSH or Console), and run the following command:

$ tail -f /var/TKLC/log/upgrade/upgrade.log

When the server has upgraded, it restarts and takes a couple of minutes for the Oracle processes to start up.

Note:

The ISO file for DB upgrade contains the default ASM Setup file and it is possible for it to overwrite “sd” definition with vd for disks. This affects the upgrade on setups that has sd disks, like sda, sdb and so on. In this scenario, just after the upgrade, when the restart is triggered, it is possible that ASM will not be able to assign sdb disk to be used for Oracle DB. This can be verified in the following file/etc/udev/rules.d/98-asm.rules. It would contain KERNEL=="dm-7" SYMLINK+="asm/ASM0" OWNER="grid" GROUP="oinstall" MODE="0660" entry and KERNEL=="sdb" SYMLINK+="asm/ASM1" OWNER="grid" GROUP="oinstall" MODE="0660" which may be removed. If sdb is missing from this file, it is recommended to edit ASM Setup file, $sudo vi /opt/xIH/oracle/instances/ASMSetup. Locate line 94, modify the expression ^vd by ^sd and save the file. Restart the VM after making the changes. This resolves the issue.

3.5.5.2.1.7 IDIH CLI: Perform a System Health Check on the Oracle Guest

Wait for few minutes to allow the Oracle Guest to stabilize after the restart, and repeat step1 to perform the postupgrade system health check.

Note:

The following behavior is expected when Mediation and Application servers are shut down:

• Mediation server is not reachable (or ping response exceeds 3 seconds).
• Application server is not reachable (or ping response exceeds 3 seconds).

3.5.5.2.2 Upgrade the Mediation and Application Guests

The procedures in this section include the steps to upgrade Mediation and Application Guest.

3.5.5.2.2.1 Cloud GUI: Remove the Existing Application Server

Use the hypervisor-specific procedure to remove the current IDIH Application and IDIH Mediation guests.

3.5.5.2.2.2 Cloud GUI: Deploy the Latest Application and Mediation Guest Images

1. Use the hypervisor-specific procedure to deploy the latest Application and Mediation guests.
2. Configure the IDIH Mediation and Application guests to reflect the guest profile in DSR Cloud Installation Guide.

3.5.5.2.2.3 IDIH CLI: Configure the IDIH VM Networks

Configure the IDIH Mediation and Application guests according to Configure IDIH Virtual Machines procedure in DSR Cloud Installation Guide.

3.5.5.2.2.4 IDIH CLI: Run Post Installation Scripts on IDIH VMs

Run Post Installation IDIH Mediation and Application specific scripts on the respective IDIH guests according to Run Post Installation scripts on IDIH VM procedure in DSR Cloud Installation Guide.

3.5.5.2.2.5 NOAM CLI: Reset SOAP Password

In case of upgrading to latest release of IDIH, reset the SOAP password to allow self-authentication of DSR with IDIH to send traces. To reset the SOAP password, see Reset the SOAP Password.