| 1. |
MPS X: Login as admusr
|
login: admusr
password: <admusr_password> |
| 2. |
MPS X: Record the TPD version
|
$ getPlatRev
7.4.0.0.0_88.37.0
|
| 3. |
MPS X: Verify that the time difference between servers is 30 seconds or less.
|
$ sudo date ; sudo clock
Mon Jul 17 11:48:24 EDT 2017
Mon Jul 17 2017 11:48:25 AM EDT -0.031459 seconds
|
| 4. |
MPS X:Verify that the chrony server is in sync. Delay should be less than 30 seconds.
|
# chronyc ntpdata
Remote address : 10.75.124.247 (0A4B7CF7)
Remote port : 123
Local address : 10.75.141.70 (0A4B8D46)
Leap status : Normal
Version : 4
Mode : Server
Stratum : 3
Poll interval : 6 (64 seconds)
Precision : -24 (0.000000060 seconds)
Root delay : 0.065979 seconds
Root dispersion : 0.111664 seconds
Reference ID : 0A4B0008 ()
Reference time : Thu Nov 23 05:47:18 2023
Offset : +0.000003993 seconds
Peer delay : 0.000072944 seconds
Peer dispersion : 0.000000093 seconds
Response time : 0.000078588 seconds
Jitter asymmetry: +0.00
NTP tests : 111 111 1111
Interleaved : No
Authenticated : No
TX timestamping : Kernel
RX timestamping : Kernel
Total TX : 831
Total RX : 831
Total valid RX : 831
|
| 5. |
MPS X: Record the last reboot occurred
|
$ uptime
04:54:34 up 7 days, 4:31, 2 users, load average: 0.60, 0.53, 0.46
Note: A server reboot is recommended after every 180 days.
|
| 6. |
MPS X: Record the ELAP release number from rpm query.
|
[root@ITHACA-A ~]# rpm -qi TKLCelap
Name : TKLCelap
Version : 5.0.66
Release : 11.0.0.0.0_110.4.0
Architecture: x86_64
Install Date: Tue 12 Dec 2023 02:56:56 AM EST
Group : Development/Build
Size : 197639401
License : ▒ TEKELEC 2018
Signature : (none)
Source RPM : TKLCelap-5.0.66-11.0.0.0.0_110.4.0.src.rpm
Build Date : Thu 23 Nov 2023 07:53:49 AM EST
Build Host : coach-14.tekelec.com
Relocations : (not relocatable)
Packager : <@tekelec.com>
Vendor : Tekelec
URL : http://www.tekelec.com/
Summary : Oracle Communications ELAP Package
Description :
This is the Oracle Communications EAGLE LNP Application Processor(ELAP) package.
The package installs ELAP software. Eagle LNP Application Processor (ELAP)
provides REALLY INCREDIBLE Database (RIDB). ELAP provides the LNP feature.$ rpm -qi TKLCelap
|
| 7 |
MPS X: Verify system health check
|
$ sudo syscheck
Running modules in class disk...
OK
Running modules in class hardware...
OK
Running modules in class net...
OK
Running modules in class proc...
OK
Running modules in class services...
OK
Running modules in class system...
OK
Running modules in class upgrade...
OK
LOG LOCATION: /var/TKLC/log/syscheck/fail_log
Note: In case one or more modules FAILED, rerun the command with verbose option like: $ sudo syscheck -v Record the output and contact My Oracle Support.
|
| 8. |
MPS X: Verify all current banner header messages
|
$ manageBannerInfo -l
There are currently no BannerInfo messages for this side in the database.
Note: In case there are any alarms, record the alarms and contact My Oracle Support.
|
| 9. |
MPS X: Retrieve alarm status from alarm manager.
|
$ alarmMgr --alarmStatus
Note: No output will be displayed if there are no alarms on the system. In case any output is observed, record the output. Sample alarms are displayed below:
$ alarmMgr --alarmStatus
SEQ: 17272594 UPTIME: 14280330 BIRTH: 1356031430 TYPE: SET ALARM: TKSPLATMA1|tpdFanError|1.3.6.1.4.1.323.5.3.18.3.1.2.1
|
| 10. |
MPS X: Record the last lines of alarm log.
|
$ tail -40 /var/TKLC/elap/logs/alarm.log
2017-07-09 01:55:54 -
************************************************************
Beginning Alarm Log
************************************************************
2017-07-09 01:55:54 - |ELAP|HSOPD|S|Critical Application|1
2017-07-09 01:55:55 - |ELAP|SysCheck|S|Major Platform|10000000
2017-07-09 01:55:55 - |ELAP|SysCheck|S|Minor Platform|800000
2017-07-09 01:56:15 - |ELAP|SysCheck|S|Minor Platform|8000
2017-07-09 01:56:55 - |ELAP|SysCheck|C|Major Platform|10000000
2017-07-09 01:57:15 - |ELAP|SysCheck|C|Minor Platform|800000
2017-07-09 01:57:15 - |ELAP|SysCheck|C|Minor Platform|8000
2017-07-09 06:00:08 - |ELAP|SysCheck|S|Minor Application|400
|
| 11 |
MPS X: Record the last lines of messages log.
|
$ tail -40 /var/log/messages
Jul 16 05:05:17 Floater-3 sudo: admusr : TTY=pts/0 ; PWD=/home/admusr ; USER=root ; COMMAND=/usr/TKLC/plat/bin/syscheck
|
| 12. |
MPS X: Verify the attributes of volume groups If the output does not contain the “logical volume” sections, contact My Oracle Support so that corrective procedures can be scheduled to be performed.
|
$ sudo vgdisplay -v
Using volume group(s) on command line.
Finding all volume groups.
[root@ITHACA-A ~]# vgdisplay -v
--- Volume group ---
VG Name vgroot
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 11
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 10
Open LV 10
Max PV 0
Cur PV 1
Act PV 1
VG Size 245.73 GiB
PE Size 4.00 MiB
Total PE 62908
Alloc PE / Size 62908 / 245.73 GiB
Free PE / Size 0 / 0
VG UUID dgSFvl-mr7T-qv4v-bmSS-U2Qq-az3d-AlbBPp
--- Logical volume ---
LV Path /dev/vgroot/plat_var_tklc
LV Name plat_var_tklc
VG Name vgroot
LV UUID xOHVMU-qlCY-iwgr-wm3D-36fP-6rWS-2bYRh8
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2023-12-12 00:56:27 -0500
LV Status available
# open 1
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:3
--- Logical volume ---
LV Path /dev/vgroot/plat_tmp
LV Name plat_tmp
VG Name vgroot
LV UUID Hv1Nxs-fIZx-MIik-Sdxv-LkUn-qJCw-p8vmg1
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2023-12-12 00:56:28 -0500
LV Status available
# open 1
LV Size 1.00 GiB
Current LE 256
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:4
--- Logical volume ---
LV Path /dev/vgroot/plat_usr
LV Name plat_usr
VG Name vgroot
LV UUID Nw2rTL-x6U8-mtVP-RtN5-bFmK-Pens-BIvOQ3
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2023-12-12 00:56:29 -0500
LV Status available
# open 1
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:2
--- Logical volume ---
LV Path /dev/vgroot/plat_var
LV Name plat_var
VG Name vgroot
LV UUID oeAWS4-fDgM-eBu4-ijdK-BcyE-1bZi-ejqnJ0
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2023-12-12 00:56:30 -0500
LV Status available
# open 1
LV Size 2.00 GiB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:5
--- Logical volume ---
LV Path /dev/vgroot/plat_swap
LV Name plat_swap
VG Name vgroot
LV UUID BUSMc0-zKpC-9566-yRO0-x2cL-68uK-PN1P6i
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2023-12-12 00:56:31 -0500
LV Status available
# open 2
LV Size 1.95 GiB
Current LE 500
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:1
--- Logical volume ---
LV Path /dev/vgroot/plat_root
LV Name plat_root
VG Name vgroot
LV UUID uaWkoG-uXLg-Iwoh-kQEy-qedR-obJ5-wtBBuc
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2023-12-12 00:56:31 -0500
LV Status available
# open 1
LV Size 2.00 GiB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0
--- Logical volume ---
LV Path /dev/vgroot/elaproot
LV Name elaproot
VG Name vgroot
LV UUID lPbpuh-zJ5E-4Zgf-mAAO-51RW-EnFI-XvY7J5
LV Write Access read/write
LV Creation host, time ITHACA-A, 2023-12-12 02:46:57 -0500
LV Status available
# open 1
LV Size 4.00 GiB
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:6
--- Logical volume ---
LV Path /dev/vgroot/logs
LV Name logs
VG Name vgroot
LV UUID qctEIA-jBRX-s23u-E1oe-lO3w-yy5m-DGXaMA
LV Write Access read/write
LV Creation host, time ITHACA-A, 2023-12-12 02:46:58 -0500
LV Status available
# open 1
LV Size 40.00 GiB
Current LE 10240
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:7
--- Logical volume ---
LV Path /dev/vgroot/drbdmeta
LV Name drbdmeta
VG Name vgroot
LV UUID PsHPy3-3bIa-WuU0-Qrg0-cQic-8ZRy-9cbDAu
LV Write Access read/write
LV Creation host, time ITHACA-A, 2023-12-12 02:47:00 -0500
LV Status available
# open 1
LV Size 128.00 MiB
Current LE 32
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:8
--- Logical volume ---
LV Path /dev/vgroot/free
LV Name free
VG Name vgroot
LV UUID sfy2P9-Ebbg-qAW7-Ic6F-OnsO-Rxpp-VZd2f0
LV Write Access read/write
LV Creation host, time ITHACA-A, 2023-12-12 02:47:00 -0500
LV Status available
# open 1
LV Size <178.66 GiB
Current LE 45736
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:9
--- Physical volumes ---
PV Name /dev/md2
PV UUID aIf59X-mkP8-gY2P-OIno-vljS-DTIU-zVCTVf
PV Status allocatable
Total PE / Free PE 62908 / 0
--- Volume group ---
VG Name vgdrbd0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 84
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 32.96 GiB
PE Size 4.00 MiB
Total PE 8439
Alloc PE / Size 2560 / 10.00 GiB
Free PE / Size 5879 / 22.96 GiB
VG UUID 9JdXH5-wt8m-hJxi-XymZ-AQUr-XjbJ-G0HxP3
--- Logical volume ---
LV Path /dev/vgdrbd0/lnpdb
LV Name lnpdb
VG Name vgdrbd0
LV UUID ch9age-tsEh-OoGt-7Mr4-lZvk-0f7w-1fldam
LV Write Access read/write
LV Creation host, time ITHACA-A, 2023-12-12 02:56:50 -0500
LV Status available
# open 0
LV Size 4.00 GiB
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:10
--- Logical volume ---
LV Path /dev/vgdrbd0/mysql
LV Name mysql
VG Name vgdrbd0
LV UUID GQHdBb-1SBK-if8F-xxC5-cCcW-1dEB-0Hn6qI
LV Write Access read/write
LV Creation host, time ITHACA-A, 2023-12-12 02:56:50 -0500
LV Status available
# open 1
LV Size 6.00 GiB
Current LE 1536
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:11
--- Physical volumes ---
PV Name /dev/drbd0
PV UUID sMe0Fc-6Tln-m2IZ-ShTt-iPV7-7thL-jqzQC2
PV Status allocatable
Total PE / Free PE 8439 / 5879
[root@ITHACA-A ~]#
|
| 13. |
MPS X: Record the total amount of free and used physical and swap memory in the system.
|
$ free
total used free shared buffers cached
Mem: 8059508 4914216 3145292 17020 967576 1589512
-/+ buffers/cache: 2357128 5702380
Swap: 2064380 0 2064380
|
| 14. |
MPS X: Verify db filesystem use is less than 90%. Note any other filesystem at 80% or higher use. Output will vary for each server.
|
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-plat_root
976M 293M 633M 32% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/md1 244M 49M 183M 22% /boot
/dev/mapper/vgroot-plat_tmp
976M 1.9M 923M 1% /tmp
/dev/mapper/vgroot-plat_usr
3.9G 2.0G 1.7G 55% /usr
/dev/mapper/vgroot-plat_var
976M 248M 678M 27% /var
/dev/mapper/vgroot-plat_var_tklc
3.9G 1.3G 2.4G 35% /var/TKLC
/dev/mapper/vgroot-elaproot
3.9G 195M 3.5G 6% /usr/TKLC/elap
/dev/mapper/vgroot-free
359G 71M 341G 1% /var/TKLC/elap/free
/dev/mapper/vgroot-logs
40G 50M 38G 1% /var/TKLC/elap/logs
/dev/sdc1 1.1G 1.1G 7.4M 100% /media/sdc1
/dev/mapper/vgdrbd0-mysql
6.0G 251M 5.4G 5% /var/TKLC/elap/drbd/mysql
|
| 15. |
MPS X: Verify disk mirroring configuration and RAID status.
|
$ cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb3[1] sda3[0]
262080 blocks super 1.0 [2/2] [UU]
md2 : active raid1 sda2[0] sdb2[1]
26198016 blocks super 1.1 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md3 : active raid1 sda1[0] sdb1[1]
442224640 blocks super 1.1 [2/2] [UU]
bitmap: 3/4 pages [12KB], 65536KB chunk
unused devices: <none>
|
| 16. |
MPS X: Record the hard drive and partition size.
|
$ sudo fdisk -l /dev/sd[a-z]
Disk /dev/sda: 480.1 GB, 480103981056 bytes
255 heads, 63 sectors/track, 58369 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000e7573
Device Boot Start End Blocks Id System
/dev/sda1 1 55071 442355712 fd Linux raid autodetect
/dev/sda2 55071 58335 26214400 fd Linux raid autodetect
/dev/sda3 * 58335 58368 262144 fd Linux raid autodetect
Disk /dev/sdb: 480.1 GB, 480103981056 bytes
255 heads, 63 sectors/track, 58369 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000116f6
Device Boot Start End Blocks Id System
/dev/sdb1 1 55071 442355712 fd Linux raid autodetect
/dev/sdb2 55071 58335 26214400 fd Linux raid autodetect
/dev/sdb3 * 58335 58368 262144 fd Linux raid autodetect
Disk /dev/sdc: 2013 MB, 2013265920 bytes
256 heads, 63 sectors/track, 243 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x488965c1
Device Boot Start End Blocks Id System
/dev/sdc1 * 1 133 1072480+ b W95 FAT32
|
| 17. |
MPS X: Verify smartctl output.
|
$ sudo smartctl -A -l error /dev/sda
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-573.26.1.el6prerel7.0.3.0.0_86.46.0.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 6936
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 24
170 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 20
175 Program_Fail_Count_Chip 0x0033 100 100 010 Pre-fail Always - 180519371374
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 081 071 000 Old_age Always - 19 (Min/Max 13/32)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 20
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 30
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
225 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 554362
226 Load-in_Time 0x0032 100 100 000 Old_age Always - 1699
227 Torq-amp_Count 0x0032 100 100 000 Old_age Always - 74
228 Power-off_Retract_Count 0x0032 100 100 000 Old_age Always - 415883
232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0
233 Media_Wearout_Indicator 0x0032 099 099 000 Old_age Always - 0
234 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 554362
242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 1654705
SMART Error Log Version: 1
No Errors Logged
Note: If any error is observed, record the error and contact My Oracle Support.
|
| 18. |
MPS X: Start Disk Integrity Check
|
$ sudo smartctl -t short /dev/sda
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-642.15.1.el6prerel7.4.0.0.0_88.37.0.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 1 minutes for test to complete.
Test will complete after Thu Aug 17 03:49:25 2017
Use smartctl -X to abort test.
|
| 19. |
MPS X: Verify and record Disk Integrity Check results
|
$ sleep 60; sudo smartctl -l selftest /dev/sda
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-642.15.1.el6prerel7.4.0.0.0_88.37.0.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Vendor (0xd0) Completed without error 00% 48283 -
# 2 Vendor (0xf0) Completed without error 00% 48283 -
# 3 Offline Completed without error 00% 48283 -
# 4 Reserved (0x28) Completed without error 00% 47639 -
# 5 Vendor (0x9a) Completed without error 00% 47421 -
# 6 Reserved (0x2a) Completed without error 00% 19961 -
# 7 Reserved (0x32) Completed without error 00% 19942 -
Note: Record if any error is reported and contact My Oracle Support.
|
| 20. |
MPS X: Record any hard disk sector error.
|
$ sudo smartctl -a /dev/sda | grep -i LBA
241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 554366
242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 1654885
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
Note: No error should be observed. In case any error is observed, output will be like as mentioned below, record the output and contact My Oracle Support.
40 51 a0 11 8e 57 e0 Error: UNC 160 sectors at LBA = 0x00578e11 = 538001
40 51 a8 11 8e 57 e0 Error: UNC 168 sectors at LBA = 0x00578e11 = 538001
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
If UNC errors are found, execute following command: $ sudo smartctl -a /dev/sda |
| 21. |
MPS X: Disk integrity step on second HDD.
|
Repeat steps from 17 to 20 for “/dev/sdb” disk drive.
|
| 22. |
MPS X: Repeat the procedure for mate ELAP.
|
Repeat steps from 1 to 21 on mate ELAP server. |