Procedure 29: Assess MPS Server Readiness

4.6.1 Procedure 29: Assess MPS Server Readiness

Table 4-10 Procedure 29: Assess MPS Server Readiness

STEP # This procedure includes the steps required to assess the readiness of a system to be DIU upgraded. Check off (✓)each step as it is completed. Boxes have been provided for this purpose under each step number. IF THIS PROCEDURE FAILS, CONTACT MY ORACLE SUPPORT AND ASK FOR DIU UPGRADE ASSISTANCE.

STEP #	This procedure includes the steps required to assess the readiness of a system to be DIU upgraded. Check off (✓)each step as it is completed. Boxes have been provided for this purpose under each step number. IF THIS PROCEDURE FAILS, CONTACT MY ORACLE SUPPORT AND ASK FOR DIU UPGRADE ASSISTANCE.
1. ☐	MPS A: Log in as the user “elapdev” user.	If not already logged-in, then log in. `<hostname> console login: elapdev` `password: <password>`
2. ☐	MPS A: Verify High Availability status.	Run the following command to display the high availability status of the ELAP pair. `$ hastatus` `ACTIVE` Note: HA status could be Active or Standby. If HA status is not Active/Standby, contact the My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.
3. ☐	MPS A: Verify DRBD status. Check the CS value as ‘Connected’. Note: If CS value is other than ‘Connected’, periodically run drbd status until both ELAPs get synced.	Run the following command to display the DRBD status. `$ sudo drbdadm status all drbd0 role:Primary` `disk:UpToDate` `natal-B role:Secondary` `peer-disk:UpToDate` `Expected` `status:` `ST:` `Primary/Secondary` `DS:` `UpToDate/UpToDate` If any status is not as expected then contact My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.
4. ☐	MPS A: For logging purposes cat the hosts file. Note: The hostname in this file will be as per the configuration.	`$ cat /etc/hosts` # `# Do not modify this file by hand. Refer to Tekelec` `Configuration` `# documentation. # # The order of the aliases in this file is significant # to the installation process.` `#` 127.0.0.1 localhost loghost devloan-01-prova-bkup mate-provb-bkup 192.168.120.100 dsmm-a 192.168.121.100 dsmb-a 192.168.120.200 dsmm-b 192.168.121.200 dsmb-b 169.254.1.200 mate sync-b hasync-1a 169.254.1.100 sync-a hasync-1b 192.168.123.100 dsmvip-a 192.168.123.200 dsmvip-b 169.254.1.201 mate-ipdptp0 server_ppp0 169.254.1.202 mate-ppp client_ppp0 169.254.1.101 devloan-01-ipdptp0 server_ppp1 169.254.1.102 devloan-01-ppp client_ppp1 169.254.1.1 switch1A 169.254.1.2 switch1B 10.250.32.10 ntpserver1 192.168.59.9 devloan-01 prova-ip 192.168.59.10 mate-prov provb-ip devloan-02 192.168.59.22 prov-vip
5. ☐	MPS A: Check the static routes.	Run the following command to display the static routes. `$netstat –r –n Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface` `192.168.122.1 192.168.121.1 255.255.255.255 UGH 0 0 0 bond0.3 10.248.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth01 169.254.1.0 0.0.0.0 255.255.255.0 U 0 0 0 bond0.1 192.168.120.0 0.0.0.0 255.255.255.0 U 0 0 0 eth02 192.168.121.0 0.0.0.0 255.255.255.0 U 0 0 0 bond0.3 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 bond0.3 0.0.0.0 10.248.10.1 0.0.0.0 UG 0 0 0 eth01`
6. ☐	MPS A: Delete unwanted ISO images.	Run the following command to display the presence of ELAP software ISO images. `$ ls –la /var/TKLC/upgrade` Below is an example of the output of the ‘ls -la’ command: `total` `877220` `drwxrwxr-x.` `2` `root` `admgrp` `4096` `May` `30` `06:09` `.` `dr-xr-xr-x.` `22` `root` `root` `4096` `May` `30` `22:24` `..` `-r--------` `1 admusr admgrp 898260992 May 30 06:09 ELAP-` `10.2.0.0.0_101.2.0-x86_64.iso` Remove any ISO images that are not the target software ISO image using the following command: `$ rm –f /var/TKLC/upgrade/<filename>`
7. ☐	MPS A: Determine when last reboot occurred. For any server up longer than 180 days would be a candidate for reboot during a maintenance window.	`$ uptime 23:41:53 up 1:16, 3 users, load average: 0.16, 0.15, 0.17`
8. ☐	MPS A: Following self test on the disk.	Run the following command: `$sudo smartctl -t short /dev/sda smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32- 573.18.1.el6prerel7.0.3.0.0_86.44.0.x86_64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net` `===` `START` `OF` `OFFLINE` `IMMEDIATE` `AND` `SELF-TEST` `SECTION` `===` `Sending command: "Run SMART Short self-test routine immediately in` `off-line` `mode".` `Drive command "Run SMART Short self-test routine immediately in` `off-line` `mode"` `successful.` `Testing` `has` `begun.` `Please` `wait` `1` `minutes` `for` `test` `to` `complete.` `Test` `will` `complete` `after` `Mon` `May` `30` `23:46:59` `2016` `Use` `smartctl -X` `to` `end` `test.` Note: Please wait for 5 minutes for the test to complete.
9. ☐	MPS A: Examine the results of self test on the disk. In case of any error/failure, contact the My Oracle Suppor tby following the instructions on the front page or the instructions in the Appendix E.	Run the following command: `$ smartctl -l selftest /dev/sda smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32- 573.18.1.el6prerel7.0.3.0.0_86.44.0.x86_64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net` `=======>` `INVALID` `ARGUMENT` `TO` `-l:` `selftest/dev/sda` `=======> VALID ARGUMENTS ARE: error, selftest, selective, directory[,g\|s], xerror[,N][,error], xselftest[,N][,selftest], background, sasphy[,reset], sataphy[,reset], scttemp[sts,hist], scttempint,N[,p], scterc[,N,M], devstat[,N], ssd, gplog,N[,RANGE], smartlog,N[,RANGE] <=======` `Use` `smartctl` `-h` `to` `get` `a` `usage` `summary`
10. ☐	MPS A: Disk Integrity step	Run the following command: `$sudo smartctl -a /dev/sda \| grep -i LBA` The output would be like: `241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 350550 242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 1695220 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS` If any output shows “Completed: read failure” or “Error: UNC xxx sectors”, contact the My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.
11. ☐	MPS A: Disk Integrity Test.	Repeat steps 8 to 10, for the ‘/dev/sdb’ disk drive on the APPD card.
12. ☐	MPS A:Inspect the banner for any messages.	Run the following command to display the banner messages. `$ manageBannerInfo –l` There are currently no BannerInfo messages for this side in the database. If unexpected output is returned, then contact My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.
13. ☐	MPS B: Repeat checks on Server B.	Repeat steps 1 to 12, on MPS B.
14. ☐ ☐ ☐ ☐	Active MPS: Verify RTDB status Login to ELAP GUI using VIP. Expand the “RTDB” Folder. Select the “View RTDB Status”. Ensure that the DB Status is Coherent.	If the RTDB status is other than Coherent, contact My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.
15. ☐	Procedure Complete.	This procedure is complete.

☐

MPS A: Log in as the user “elapdev” user.

If not already logged-in, then log in.

<hostname> console login: elapdev

password: <password>

☐

MPS A: Verify High Availability status.

Run the following command to display the high availability status of the ELAP pair.

$ hastatus

ACTIVE

Note: HA status could be Active or Standby. If HA status is not Active/Standby, contact the My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.

☐

MPS A: Verify DRBD status. Check the CS value as ‘Connected’.

Note: If CS value is other than ‘Connected’, periodically run drbd status until both ELAPs get synced.

Run the following command to display the DRBD status.

$ sudo drbdadm status all drbd0 role:Primary

disk:UpToDate

natal-B role:Secondary peer-disk:UpToDate

Expected status:

ST: Primary/Secondary

DS: UpToDate/UpToDate

If any status is not as expected then contact My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.

☐

MPS A: For logging purposes cat the hosts file.

Note: The hostname in this file will be as per the configuration.

$ cat /etc/hosts

# Do not modify this file by hand. Refer to Tekelec

Configuration

# documentation. #
# The order of the aliases in this file is significant
# to the installation process.

#

127.0.0.1    localhost loghost devloan-01-prova-bkup
mate-provb-bkup
192.168.120.100    dsmm-a
192.168.121.100    dsmb-a
192.168.120.200    dsmm-b
192.168.121.200    dsmb-b
169.254.1.200    mate sync-b hasync-1a
169.254.1.100    sync-a hasync-1b
192.168.123.100    dsmvip-a
192.168.123.200    dsmvip-b
169.254.1.201    mate-ipdptp0 server_ppp0
169.254.1.202    mate-ppp client_ppp0
169.254.1.101    devloan-01-ipdptp0 server_ppp1
169.254.1.102    devloan-01-ppp client_ppp1
169.254.1.1    switch1A
169.254.1.2    switch1B
10.250.32.10    ntpserver1
192.168.59.9    devloan-01 prova-ip
192.168.59.10    mate-prov provb-ip devloan-02
192.168.59.22    prov-vip

☐

MPS A: Check the static routes.

Run the following command to display the static routes.

$netstat –r –n
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface

192.168.122.1 192.168.121.1 255.255.255.255 UGH    0 0 0 bond0.3
10.248.10.0 0.0.0.0 255.255.255.0 U    0 0 0 eth01
169.254.1.0 0.0.0.0 255.255.255.0 U    0 0 0 bond0.1
192.168.120.0 0.0.0.0 255.255.255.0 U    0 0 0 eth02
192.168.121.0 0.0.0.0 255.255.255.0 U    0 0 0 bond0.3
169.254.0.0 0.0.0.0 255.255.0.0 U    0 0 0 bond0.3
0.0.0.0 10.248.10.1 0.0.0.0 UG    0 0 0 eth01

☐

MPS A: Delete unwanted ISO images.

Run the following command to display the presence of ELAP software ISO images.

$ ls –la /var/TKLC/upgrade

Below is an example of the output of the ‘ls -la’ command:

total 877220

drwxrwxr-x. 2 root admgrp 4096 May 30 06:09 .

dr-xr-xr-x. 22 root root 4096 May 30 22:24 ..

-r-------- 1 admusr admgrp 898260992 May 30 06:09 ELAP- 10.2.0.0.0_101.2.0-x86_64.iso

Remove any ISO images that are not the target software ISO image using the following command:

$ rm –f /var/TKLC/upgrade/<filename>

☐

MPS A: Determine when last reboot occurred.

For any server up longer than 180 days would be a candidate for reboot during a maintenance window.

$ uptime
23:41:53 up 1:16, 3 users, load average: 0.16, 0.15, 0.17

☐

MPS A: Following self test on the disk.

Run the following command:

$sudo smartctl -t short /dev/sda
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32- 573.18.1.el6prerel7.0.3.0.0_86.44.0.x86_64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===

Sending command: "Run SMART Short self-test routine immediately in off-line mode".

Drive command "Run SMART Short self-test routine immediately in off-line mode" successful.

Testing has begun.

Please wait 1 minutes for test to complete.

Test will complete after Mon May 30 23:46:59 2016 Use smartctl -X to end test.

Note: Please wait for 5 minutes for the test to complete.

☐

MPS A: Examine the results of self test on the disk.

In case of any error/failure, contact the My Oracle Suppor tby following the instructions on the front page or the instructions in the Appendix E.

Run the following command:

$ smartctl -l selftest /dev/sda
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-
573.18.1.el6prerel7.0.3.0.0_86.44.0.x86_64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=======> INVALID ARGUMENT TO -l: selftest/dev/sda

=======> VALID ARGUMENTS ARE: error, selftest, selective, directory[,g|s], xerror[,N][,error], xselftest[,N][,selftest], background, sasphy[,reset], sataphy[,reset], scttemp[sts,hist], scttempint,N[,p], scterc[,N,M], devstat[,N], ssd, gplog,N[,RANGE], smartlog,N[,RANGE] <=======

Use smartctl -h to get a usage summary

10.

☐

MPS A: Disk Integrity step

Run the following command:

$sudo smartctl -a /dev/sda | grep -i LBA

The output would be like:

241 Total_LBAs_Written    0x0032  100  100  000
Old_age  Always  -  350550
242 Total_LBAs_Read       0x0032  100  100  000
Old_age  Always  -  1695220
Num Test_Description  Status  Remaining
LifeTime(hours) LBA_of_first_error
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS

If any output shows “Completed: read failure” or “Error: UNC xxx sectors”, contact the My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.

11.

☐

MPS A: Disk Integrity Test.

Repeat steps 8 to 10, for the ‘/dev/sdb’ disk drive on the APPD card.

12.

☐

MPS A:Inspect the banner for any messages.

Run the following command to display the banner messages.

$ manageBannerInfo –l

There are currently no BannerInfo messages for this side in the database.

If unexpected output is returned, then contact My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.

13.

☐

MPS B: Repeat checks on Server B.

Repeat steps 1 to 12, on MPS B.

14.

☐

Active MPS: Verify RTDB status

Expand the “RTDB” Folder.

Select the “View RTDB Status”.

Ensure that the DB Status is Coherent.

ELAP RTDB status view showing the database status as Coherent.

If the RTDB status is other than Coherent, contact My Oracle Support by following the instructions on the front page or the instructions in the Appendix E.

15.

☐

Procedure Complete.

This procedure is complete.