A.1 Health Check Procedures

This procedure is part of software upgrade preparation and is used to determine the health and status of the SDS network and servers.

Note:

If syscheck fails on any server during pre-upgrade Checks or in early checks stating that "cpu: FAILURE:: No record in alarm table for FAILURE", see Workaround to Resolve Syscheck Error for CPU Failure.

If the 31201 - Process Not Running alarm displays, for instance, as cmsoapa, then run Workaround to Fix cmsoapa Restart.

  1. In the SDS NOAM GUI, log in use the VIP address to access the primary SDS NOAM GUI as described in Access the OAM GUI Using the VIP (NOAM/SOAM).
  2. In the Primary SDS NOAM VIP, verify status. Expand Status & Manage click Server.

    Figure A-5 Server


    Server

  3. Verify Server Status is Normal (Norm) for Alarm (Alm), Database (DB), Reporting Status, and Processes (Proc).

    Figure A-6 Server Status


    Server Status

    If any other server status displays, it appears in a colored box.

    Note:

    Other server states include Err, Warn, Man, and Unk.

    Note:

    Post-Upgrade, upgraded servers have an Alm status of Err due to the Event ID (s): 32532 Server Upgrade Pending Accept/Reject expected alarm. This alarm displays until the upgrade is accepted and may be ignored at this time.

    Note:

    During any time of upgrade in case 31149- DB Late Write Nonactive alarm is seen, please ignore it. This alarm does not have any effect on any functionality.

    If 31201 - Process Not Running alarm is getting raised for Instance as cmsoapa then run Workaround to Fix cmsoapa Restart to solve this issue.

  4. In the Primary SDS NOAM VIP, verify connection counts. Expand Communication Agent select Maintenance click Connection.

    Figure A-7 Connection


    Connection

  5. Verify all Connection Counts show equivalent counts (that is, n of n InService for Automatic or y of y InService for Configured)

    Figure A-8 Maintenance


    Maintenance

    Note:

    DPs show a Configured Connections Count of 1 of 2 InService for Active/Standby configurations. This is normal and can be ignored.
  6. In the Primary SDS NOAM VIP, view alarm status. Expand Alarms & Events click View Active.

    Figure A-9 View Active


    View Active

  7. When viewing pre-upgrade status, if any alarms are present, stop and contact My Oracle Support for assistance before attempting to continue.
  8. When viewing post-upgrade status, verify the following:
    Active NO server may have the following expected alarms:
    • Alarm ID = 10075 (Application processes have been manually stopped)

    Servers that still have replication disabled have the following expected alarm:

    • Alarm ID = 31113 (Replication Manually Disabled)

    The following alarms may also be seen:

    • Alarm ID = 10010 (Stateful database not yet synchronized with mate database)
    • Alarm ID = 32532 (Server Upgrade Pending Accept/Reject)
    • Alarm ID = 31114 (DB Replication over SOAP has failed)
    • Alarm ID = 31225 (HA Service Start Failure)

    Following alarms can be ignored during the upgrade:

    • Alarm ID = 31109 (Topology Config Error)
    • Alarm ID = 31282 (HA Management Fault)
    • Alarm ID = 31283 (Lost Communication with server)
    • Alarm ID = 31106 (DB Merge To Parent Failure)
    • Alarm ID = 31107 (DB Merge From Child Failure)
    • Alarm ID = 10009 (Config and Prov DB not yet synchronized)

    Note:

    If Alarm 10009 persists after the upgrade, reboot the server once using the sudo init 6 command on the effected server.

    These alarms may display until all the NOAM and DR-NOAM servers upgrade has been completed.

  9. In the Primary SDS NOAM VIP, create Alarms and Events report. Click Export.

    Figure A-10 Export


    Export

  10. Click OK.

    Figure A-11 OK


    OK

  11. The name of the exported alarms CSV file displays in the Tasks tab.

    Figure A-12 Tasks


    Tasks

  12. Primary SDS NOAM VIP, record the file names of alarm CSV files. The files have the format Alarms<yyyymmdd>_<hhmmss>.csv.

    Record the following files:

    • Pre-ISO Administration
    • Post-ISO Administration
    • Pre-Primary NOAM Upgrade (MW1)
    • Post-DR NOAM Upgrade (MW1)
    • Pre-SOAM Upgrade (MW2)
    • Post-SOAM Upgrade (MW2)
    • Pre-SOAM Upgrade (MW3)
    • Post-SOAM Upgrade (MW3)
    • Pre-SOAM Upgrade (MW4)
    • Post-SOAM Upgrade (MW4)
    • Pre-SOAM Upgrade (MW5)
    • Post-SOAM Upgrade (MW5)
  13. In the Primary SDS NOAM VIP, save the Alarms and Events report. Click Report

    Figure A-13 Report


    Report

    Figure A-14 View Active Alarms


    View Active Alarms

  14. Click Save on the Alarms and Events report and click Save on the File Download screen.

    Figure A-15 Save


    Save

    Figure A-16 Save


    Save

  15. Select a directory on a local disk drive to store the active Alarms and Events report and click Save.

    Figure A-17 Save As


    Save As

  16. In the Primary SDS NOAM VIP, create Network Element report. Before 8.x, expand Configuration click Network Elements.

    Figure A-18 Network Elements


    Network Elements

  17. Expand Configuration, select Networking and click Networks.

    Figure A-19 Networks


    Networks

  18. Click Report.

    Figure A-20 Report


    Report

  19. The Network Element Report is generated.

    Figure A-21 Network Element Report


    Network Element Report

  20. In the Primary SDS NOAM VIP, save the Network Element report. Click Save on the Network Element report and click Save on the File Download screen.

    Figure A-22 Save


    Save

    Figure A-23 Save File


    Save File

  21. Select a directory on a local disk drive to store the Network Element report and click Save.

    Figure A-24 Save As


    Save As

  22. In the Primary SDS NOAM VIP, create the Servers report. Expand Configuration click Servers.

    Figure A-25 Servers


    Servers

  23. Click Report.

    Figure A-26 Report


    Report

    Figure A-27 Server Report


    Server Report

  24. In the Primary SDS NOAM VIP, save the Servers report. Click Save on the Servers report and click Save on the File Download screen.

    Figure A-28 Save


    Save

    Figure A-29 Save File


    Save File

  25. Select a directory on a local disk drive to store the Servers report and click Save.

    Figure A-30 Save Server Report


    Save Server Report

  26. In the Primary SDS NOAM VIP, create Server Groups the report. Expand Configuration click Server Groups.

    Figure A-31 Server Groups


    Server Groups

  27. Click Report.

    Figure A-32 Report


    Report

    Figure A-33 Sever Groups Report


    Sever Groups Report

  28. In the Primary SDS NOAM VIP, save the Servers report. Click Save on the Server Groups report and click Save on the File Download screen.

    Figure A-34 Save


    Save

    Figure A-35 Save File


    Save File

  29. Select a directory on a local disk drive to store the Server Groups report and click Save.

    Figure A-36 Save Server Report


    Save Server Report

  30. Share the saved files with My Oracle Support. If these procedures are run as pre- or post-upgrade health check (HC1/HC2/HC3), sharing the files with My Oracle Support to obtain a proper health check analysis.

    A health check analysis includes verifying the following information collected from Access the OAM GUI Using the VIP (NOAM/SOAM) procedure.

    • Active Alarms and Events report.
    • Network Elements report.
    • Server report.
    • Server Group report.
  31. Verify OAM HA Role status, expand Status & Manage click HA

    Figure A-37 HA


    HA

  32. Verify the OAM HA Role for all servers shows either Active or Standby.

    Figure A-38 OAM HA Role


    OAM HA Role

    Note:

    An OAM HA Role shown as Observer is allowed when the server role is Query Server.
  33. Verify the OAM HA Role for all remaining servers, expand Main Menu select Status & Manage click HA. Scroll through each page until the OAM HA Role for has been verified for all servers in the topology.