1. DSR Software Upgrade Guide
  2. Backout Procedure Overview

7 Backout Procedure Overview

The procedures provided in this section return the individual servers and the overall DSR system to the source release after an upgrade is aborted. The backout procedures support two options for restoring the source release: Emergency Backout and Normal Backout.

The emergency backout overview procedures back out the target release software in the fastest possible manner, without regard to traffic impact. The normal backout overview procedures back out the target release software in a more controlled manner, sustaining traffic to the extent possible. All backout procedures are executed inside a maintenance window.

The backout procedure times provided in the following tables are only estimates as the reason to execute a backout has a direct impact on any additional backout preparation that must be done.

Note:

While not specifically covered by this procedure, it may be necessary to re-apply patches to the source release after the backout. If patches are applicable to the source release, verify all patches are on-hand before completing the backout procedures.

Table 7-1 Emergency and Normal Backout Procedure Overview

Procedure Elapsed Time (This Step) Elapsed Time (Cum.) Procedure Title Impact
Backout Health Check 0:10-0:30 0:10-0:30 Backout Health Check

The reason to execute a backout has a direct impact on any additional backout preparation that must be done. Since all possible reasons cannot be predicted ahead of time, only estimates are given here. Execution time varies.

 None
Disable Global Provisioning 0:01 0:11-0:31 Disable Global Provisioning Disables global provisioning
Emergency Site Backout See Note See Note Emergency Site Backout

Note:

Execution time of downgrading entire network is approximately equivalent to execution time taken during upgrade.

0:05 (5 minutes) can be subtracted from total time because ISO Administration is not executed during Backout procedures.
All impacts as applicable in upgrade apply in this procedure. Also, backout procedures cause traffic loss.
Backout Multiple Servers See Note See Note Backout Multiple Servers

Note:

Execution time of downgrading a single server is approximately equivalent to execution time to upgrade the server.
All impacts as applicable in upgrade apply in this procedure. Also, backout procedures cause traffic loss.
Emergency NOAM Backout See Note See Note Emergency NOAM Backout

Note:

Execution time of downgrading a single server is approximately equivalent to execution time to upgrade the server.
All impacts as applicable in upgrade apply in this procedure. Also, backout procedures cause traffic loss.
Post-Backout Health Check 0:01-0:05 Varies Post-Backout Health Check None

7.1 Recovery Procedures

It is recommended to direct upgrade procedure recovery issues to My Oracle Support (MOS) before executing any of these procedures. Execute this section only if there is a problem and it is desired to revert back to the pre-upgrade version of the software.

Caution:

Before attempting to perform these backout procedures, it is recommended to first contact My Oracle Support (MOS). Backout procedures cause traffic loss.

Note:

These recovery procedures are provided for the backout of an upgrade only (that is, from a failed 8.6.0.7.0 release to the previously installed 7.1.w release). Backout of an initial installation is not supported.

During the backout, servers may have the following expected alarms until the server is completely backed out. The servers may have some or all of the following expected alarms, but are not limited to event IDs:

Alarm ID = 31283 (Highly available server failed to receive mate heartbeats)

Alarm ID = 31109 (Topology config error)

Alarm ID = 31114 (DB Replication over SOAP has failed)

Alarm ID = 31106 (DB Merge to Parent Failure)

Alarm ID = 31134 (DB replication to slave failure)

Alarm ID = 31102 (DB replication from master failure)

Alarm ID = 31282 (HA management fault)

7.2 Backout Health Check

This section provides the procedure to verify that the DSR is ready for backout. The site post-upgrade Health Check is used to perform the backout Health Check.

7.2.1 Active NOAM VIP: Run the Automated Post-upgrade Health Checks for Backout

Use this procedure to perform a Health Check on the site prior to backing out the upgrade.
  1. Navigate to Administration, then Software Management, and then Upgrade.
  2. Select the SOAM tab of the site being backed out.
  3. Select the SOAM server group link for the site being backed out.
  4. Select the active SOAM.
  5. Click Checkup.
  6. Under Health check options, click Post Upgrade.
  7. Click OK.

    Control returns to the Upgrade screen.

7.2.2 Active NOAM VIP: Monitor Health Check Progress for Completion

This procedure details the steps to monitor the rogress of health check .
  1. Click the Tasks option to display the currently executing tasks. The Health Check task name appears as <SOServerGroup> PostUpgrade Health Check.
  2. Monitor the Health Check task until the Task State is completed. The Details column displays a hyperlink to the Health Check report.
  3. Click the hyperlink to download the Health Check report.
  4. Open the report and review the results.

7.2.3 Active NOAM VIP: Analyze Health Check Results

Follow the steps in this procedure to analyze health check report for failures. If the Health Check report status is anything other than Pass, the Health Check logs can be analyzed to determine if the upgrade can proceed.
  1. Navigate to Status & Manage, then Files.
  2. Select the active NOAM tab.
  3. Select the UpgradeHealthCheck.log file and click View.
  4. Locate the log entries for the most recent health check.
Review the log for failures. Analyze the failures and determine if it is safe to continue the upgrade. If necessary, it is recommended to contact My Oracle Support (MOS) for guidance.

7.2.4 Active NOAM VIP: Identify IP Addresses of Servers to be Backed Out

  1. Navigate to Administration, then Software Management , and then Upgrade.
  2. Select the SOAM tab of the site being backed out.
  3. Select each server group link, making note of the application version of each server.
  4. Based on the Application Version column, identify all the hostnames that need to be backed out.
  5. Navigate to Configuration, then Servers.
  6. Using the data recorded in Table 5, note the XMI/iLO/LOM IP addresses of all the hostnames to be backed out. These are required to access the server when performing the backout.

    The reason to execute a backout has a direct impact on any additional backout preparation that must be done. The backout procedures cause traffic loss. Since all possible reasons cannot be predicted ahead of time, it is recommended to contact My Oracle Support (MOS) as stated in the Warning box.

7.2.5 Active NOAM VIP: Verify Backup Archive Files

  1. Navigate to Status & Manage, then Files.
  2. For each server to be backed out, select the server tab on the Files screen. Verify the two backup archive files, created in section 3.4.4, are present on every server that is to be backed out. These archive files have the format:Backup.<application>.<server>.FullDBParts.<role>.<date_time>.UPG.tar.bz2
    Backup. <application>.<server>.FullRunEnv.<role>.<date_time>.UPG.tar.bz2

7.2.6 Active NOAM CLI: Verify Disk Usage

This procedure lists the steps to verify the disk usage.
Starting with the active NOAM, log in to each server to be backed out to verify the disk usage is within acceptable limits.
  1. Use the SSH command (on UNIX systems – or putty if running on windows) to log into the active NOAM.ssh admusr@<server IP>
    password: <enter password>

    Answer yes if you are asked to confirm the identity of the server.

  2. Enter the command:[admusr@EVO-NO-1 ~]$ df

    Sample output (abridged):

    Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vgroot-plat_root 999320 294772 652120 32% / tmpfs 12303460 0 12303460 0% /dev/shm /dev/vda1 245679 41967 190605 19% /boot /dev/mapper/vgroot-plat_tmp 999320 1548 945344 1% /tmp /dev/mapper/vgroot-plat_usr 5029504 2962552 1804824 63% /usr /dev/mapper/vgroot-plat_var 999320 558260 388632 59% /var /dev/mapper/vgroot-plat_var_tklc 3997376 2917284 870380 78% /var/TKLC

  3. Observe the line for the /var and /usr partition. If the Use% column is 70% or less, this procedure is complete. Continue with the back out per Table 22 (Emergency) or Table 23 (Normal).

    If the Use% of the /var and /usr partition is at 70% or greater, search the partition for files that can be safely deleted. Use extreme caution in selecting files to be deleted. The deletion of critical system files could severely impair the DSR functionality.

  4. Repeat this step for all servers to be backed out.

7.3 Disable Global Provisioning

The following procedure disables provisioning on the NOAM. This step ensures no changes are made to the database while the NOAMs and sites are backed out. Provisioning is re-enabled once the NOAM upgrade is complete.

7.3.1 Active NOAM VIP: Disable global provisioning and configuration updates on the entire network

This procedure lists the steps to disable global provisioning and configuration updates on the entire network.
  1. Log in to the active NOAM GUI using the VIP.
  2. Navigate to Status & Manage, then Database.
  3. Click Disable Provisioning.
  4. Confirm the operation by clicking OK on the screen.
  5. Verify the button text changes to Enable Provisioning. A yellow information box should also be displayed at the top of the view screen which states:

    [Warning Code 002] – Global provisioning has been manually disabled.

    The active NOAM server has the following expected alarm:

    Alarm ID = 10008 (Provisioning Manually Disabled)

7.4 Perform Emergency Backout

The procedures in this section perform a backout of all servers to restore the source release. An emergency backout can only be executed once all the necessary corrective setup steps have been taken to prepare for the backout. It is recommended to contact My Oracle Support (MOS) as stated in the warning box in Section 6.1, to verify that all corrective setup steps have been taken..

7.4.1 Emergency Site Backout

The procedures in this section backout all servers at a specific site without regard to traffic impact.

Note:

Executing this procedure results in a total loss of all traffic being processed by this DSR. Traffic being processed by the mate DSR is not affected.
7.4.1.1 Active NOAM VIP: Identify all the servers that require backout (within a site)
  1. Log in to the NOAM GUI using the VIP.
  2. Navigate to Administration, then Software Management, and then Upgrade.
  3. Select the NOAM tab of the site being backed out.
  4. Select each server group link, making note of the application version of the servers.
  5. Identify the servers in the respective server groups with the target release Application Version value. These servers were previously upgraded but now require backout.
  6. Make note of these servers. They have been identified for backout.
  7. Before initiating the backout procedure, remove all new blades and/or sites configured after upgrade was started.
7.4.1.2 Active SOAM VIP: Disable site provisioning for the site to be backed out
  1. Log in to the SOAM GUI using the VIP.
  2. Navigate to Status & Manage, then Database.
  3. Click Disable Provisioning.
  4. Confirm the operation by clicking OK on the screen.
  5. Verify the button text changes to Enable Provisioning. A yellow information box displays at the top of the view screen which states:

    [Warning Code 004] – Site provisioning has been manually disabled.

    The active SOAM server has the following expected alarm:

    Alarm ID = 10008 (Provisioning Manually Disabled)

7.4.1.3 Backout all C-level Servers
For all configurations, backout all C-level servers (IPFEs, SBRs, SBRs, and DA-MPs) identified in Active NOAM VIP:
  1. Identify all servers that require backout (within a site) procedure in this section and perform Backout Multiple Servers procedure.

    Note:

    This process results in a total loss of all traffic being processed by this DSR.
  2. After all the servers in a particular server group are backed out, revert back the changes for the SBR server by executing Appendix L Additional Post-Backout Steps. Perform Appendix U to create a link of Comagent.
  3. Back out the standby and spare DSR SOAM servers: If standby and spare SOAM servers are present, perform Backout Multiple Servers procedure. If only a spare SOAM server is present, perform Backout Single Server procedure.
  4. Perform Backout Single Server procedure to backout the active DSR SOAM server.
    After all the servers in a particular server group are backed out, revert back the changes for the SOAM server(s) by executing Appendix L, Additional Post-Backout Steps.
7.4.1.4 Active SOAM VIP: Enable site provisioning
  1. Log in to the SOAM GUI using the VIP.
  2. Navigate to Status & Manage, then Database.
  3. Click Enable Site Provisioning.
  4. Confirm the operation by clicking OK on the screen.
  5. Verify the button text changes to Disable Site Provisioning

    Note:

    If another site is to be backed out, follow all procedures in Emergency Backout Procedure Overview section in another maintenance window.

7.4.2 Emergency NOAM Backout

This procedure is used to perform an emergency backout of the DSR application software from the NOAM servers. This procedure backs out the application software as quickly as possible, without regard to operational impact. If this procedure fails, it is recommended to contact My Oracle Support (MOS) and ask for assistance.
  1. Perform Backout Single Server procedure to:
    • Back out the standby DR NOAM server (if equipped)
    • Back out the active DR NOAM server (now the standby) (if equipped)
    • Back out the standby DSR NOAM server (as applicable)
    • Back out the active DSR NOAM server (now the standby)
  2. After all the servers in a particular server group are backed out, revert back the changes for the NOAM server(s) by executing Appendix L Additional Post-Backout Steps.
  3. Active NOAM VIP: Enable global provisioning and configuration updates on the entire network
    1. Log into the NOAM GUI using the VIP.
    2. Navigate to Status & Manage, then Database.
    3. Click Enable Provisioning.
    4. Verify the button text changes to Disable Provisioning.
  4. Active NOAM VIP: Remove Ready state for any backed out server
    1. Navigate to Status & Manage, then Servers.
    2. If any backed-out server Application Status is Disabled, then navigate to the server row and click Restart.
    3. Navigate to Administration, then Software Management, and then Upgrade.
    4. If any backed-out server shows an Upgrade State of Ready or Success, then select that server and click Complete Upgrade. Otherwise, skip this step.
    5. Click OK.

      This removes the Forced Standby designation for the backed-out server.

      This removes the Forced Standby designation for the backed-out server. Note: Due to backout being initiated from the command line instead of through the GUI, the following SOAP error may appear in the GUI banner.

      SOAP error while clearing upgrade status of hostname=[frame10311b6] ip=[172.16.1.28]

      It is safe to ignore this error message.

  5. Verify the Application Version value for servers has been downgraded to the original release version.

7.5 Perform Normal Backout

Execute the following procedures to perform a normal backout only when all necessary corrective setup steps have been taken to prepare for the backout. It is recommended to contact My Oracle Support (MOS), as stated in the warning box in Section 6.1, to verify that all corrective setup steps have been taken.

7.5.1 Normal Site Backout

The procedures in this section back out an upgrade of the DSR application software from multiple servers in the network. Any server requiring backout can be included: SOAMs, DA-MPs, IPFEs, and SBRs.
7.5.1.1 Active NOAM VIP: Identify all Servers That Require Backout (Within a Site)
Repeat the steps mentioned in Active NOAM VIP: Identify all servers that require backout (within a site) procedure in Emergency Site Backout.
7.5.1.2 Active SOAM VIP: Disable Site Provisioning for the Site to be Backed Out
Repeat the steps mentioned in Active SOAM VIP: Disable site provisioning for the site to be backed out procedure in Emergency Site Backout.
7.5.1.3 Back out First Set of C-level Servers
Follow these steps to backout the first set of C-level servers, as applicable.
  1. The following C-level servers can be backed out in parallel, as applicable:
    • ½ of all DA-MPs for N+0 (multi-active) configuration
    • Standby SBR(s)
    • Spare SBR(s)
    • ½ of all IPFEs

    Note:

    Execute Backout Single Server procedure for each standby/spare C‑level server identified.

    In a PCA System, the spare SBR server is located at the mated site of the site being backed out.

7.5.1.4 Active NOAM VIP: Verify Standby SBR Server Status
If the server being backed out is the standby SBR, execute this step. Otherwise, continue with Backout remaining C-level servers, as applicable procedure mentioned in this section.
  1. Navigate to SBR, then Maintenance, and then SBR Status. Open the tab of the server group being upgraded.
  2. Do not proceed to step 6 until the Resource HA Role for the standby server has a status of Standby.
7.5.1.5 Active NOAM VIP: Verify Bulk Download is Complete
Perform this procedure to verify the bulk download is complete between the active SBR in the server group to the standby and spare SBRs.
  1. Navigate to Alarm & Event, then View History.
  2. Export the Event log using the following filter:

    Server Group: Choose the SBR group that is in upgrade

    Display Filter: Event ID = 31127 – DB Replication Audit Complete

    Collection Interval: X hours ending in current time, where X is the time from upgrade completion of the standby and spare servers to the current time.

  3. Wait for the following instances of Event 31127:
    • 1 for the Standby Binding SBR server
    • 1 for the Standby Session SBR server
    • 1 for the Spare Binding SBR server
    • 1 for the Spare Session SBR server
    • 1 for the 2nd Spare Binding SBR server, if equipped
    • 1 for the 2nd Spare Session SBR server, if equipped

    Note:

    There is an expected loss of traffic depending on size of the bulk download. This must be noted along with events captured.
7.5.1.6 Backout Remaining C-level Servers
Execute Backout Single Server procedure to backout the remaining servers in parallel, as applicable:
  • ½ of all DA-MPs for N+0 (multi-active) configuration
  • Active SBR(s)
  • ½ of all IPFEs
  • Backout the standby DSR SOAM server
  • Backout spare DSR SOAM server, if applicable
  • Backout active DSR SOAM server

Note:

After all the servers in a particular server group are backed out, revert back the changes for the SOAM server(s) by executing Appendix L Additional Post-Backout Steps. Perform Appendix U to create a link of Comagent.
7.5.1.7 Active SOAM VIP: Enable Site Provisioning
  1. Log in to the SOAM GUI using the VIP.
  2. Navigate to Status & Manage, then Database.
  3. Click Enable Site Provisioning.
  4. Confirm the operation by clicking OK on the screen.
  5. Verify the button text changes to Disable Site Provisioning.

    Note:

    If another site is to be backed out, follow all procedures in Emergency Table 7-1 in another maintenance window.

7.5.2 Normal NOAM Backout

This procedure is used to perform a normal backout of the DSR application software from the NOAM servers.

  1. Repeat steps 1 to 3 in Emergency NOAM BackoutEmergency NOAM Backout procedure.

7.6 Backout Single Server

This section provides the procedures to back out the application software on a single server.

Note:

This procedure is executed as a component of the Emergency Site Backoutor Normal Site Backout. This procedure should never be executed as a standalone procedure.

7.6.1 Active NOAM VIP: Prepare the Server for Backout

  1. Navigate to Administration, then Software Management, and then Upgrade.
  2. Select the NOAM tab of the site being backed out.
  3. Select the server group link containing the server to be backed out.
  4. Verify the Upgrade State is Accept or Reject.

    Note:

    Make the server Backout Ready as follows:
  5. To make the server backout ready, navigate to Status & Manage, then HA.
  6. Click Edit.
  7. Select the server to be backed out and choose a Max Allowed HA Role value of Standby (unless it is a Query server, in which case the value should remain set to Observer).

    Note:

    Note: When the active NOAM is the server being backed out, click OK to initiate an HA switchover and cause the GUI session to log out.
  8. Click OK.

    Note:

    If the server being backed out is the active NOAM and HA switchover does not happen, and the OAM HA Role of the NOAMP server to be backed out on the HA status screen is still Active, then you have encountered a known issue. Apply the workaround using Appendix Q to have the NOAMP HA switchover.

    Do not omit this step.

  9. Log out of the GUI, clear the browser cache, and log back into the active NOAM via the VIP before continuing. Some GUI forms may exhibit incorrect behaviors if the browser cache is not cleared.
  10. Verify the Max Allowed HA Role is set to the desired value for the server on the HA Status screen.
  11. Navigate to Status & Manage, then Server.
  12. Select the server to backout and click Stop.
  13. Click OK to confirm the operation and verify the Appl State changes to Disabled.
  14. Navigate to Administration, then Software Management, and then Upgrade.
  15. Select the NOAM tab of the site being backed out.
  16. Select the link of the server group containing the server to be backed out. Verify the Upgrade State is now Backout Ready.

    Note:

    It may take a couple of minutes for the status to update.

7.6.2 Server CLI: SSH to Server

  1. Use an SSH client to connect to the server (e.g., ssh, putty):
    ssh admusr@<server address>

    password: <enter password>

    Note:

    If direct access to the IMI is not available, or if TVOE is installed on a blade, then access the target server via a connection through the active NOAM. SSH to the active NOAM XMI first. From there, SSH to the target server’s IMI address.

7.6.3 Server CLI: Execute the Backout

  1. Execute this command to find the state of the server to be backed out:
    $ ha.mystate

    $ sudo /var/TKLC/backout/reject

    Note:

    The reject command creates a no-hang-up shell session, so the command continues to execute if the user session is lost.

    Note:

    If back out asks to continue, answer y.
    Many informational messages display to the terminal screen as the backout proceeds. After backout is complete, the server automatically reboots.

7.6.4 Server CLI: SSH to Server

  1. Use an SSH client to connect to the server (e.g., ssh, putty):

    ssh admusr@<server address>

    password: <enter password>

  2. Perform Appendix U to create a link of Comagent.

7.6.5 Server CLI: Restore the Full DB Run Environment

  1. Execute the backout_restore utility to restore the full database run environment:

    $ sudo /var/tmp/backout_restore

  2. If asked to proceed, answer y.

    Note:

    In some incremental upgrade scenarios, the backout_restore file is not found in the /var/tmp directory, resulting in the following error message:

    /var/tmp/backout_restore: No such file or directory

    If this message occurs, copy the file from /usr/TKLC/appworks/sbin to /var/tmp and repeat sub-step 1. The backout_restore command creates a no-hang-up shell session, so the command continues to execute if the user session is lost. If the restore was successful, the following displays:

    Success: Full restore of COMCOL run env has completed.

    Return to the backout procedure document for further instruction.

    If an error is encountered and reported by the utility, it is recommended to consult with My Oracle Support (MOS) for further instructions.

7.6.6 Server CLI: Verify the Backout

  1. Examine the output of the following command to determine if any errors were reported:
    $ sudo verifyUpgrade

    Note:

    The verifyUpgrade command detected errors that occurred in the initial upgrade and during the backout. Disregard the errors occurred in the initial upgrade.
  2. Disregard the following TKLCplat.sh error:

    [root@NO1 ~]# verifyUpgrade

    ERROR: TKLCplat.sh is required by upgrade.sh!

    ERROR: Could not load shell library!

    ERROR: LIB: /var/TKLC/log/upgrade/verifyUpgrade/upgrade.sh

    ERROR: RC: 1

    Also, disregard this error:

    ERROR: Upgrade log (/var/TKLC/log/upgrade/upgrade.log) reports errors!

    ERROR: 1513202476::zip error: Nothing to do!

    /usr/share/tomcat6/webapps/ohw.war

    This command displays the current sw rev on the server:

    $ appRev

    Install Time: Wed Apr 4 05:03:13 2018

    Product Name: DSR

    Product Release: 8.6.0.7.0_96.34.0

    Base Distro Product: TPD

    Base Distro Release: 7.8.7.0.0_89.29.0

    Base Distro ISO: TPD.install-7.8.7.0.0_89.29.0-OracleLinux6.10-x86_64.iso

    ISO name: 8.6.0.7.0_96.34.0.iso

    OS: OracleLinux 6.10

  3. Run the following command.
    $ sudo verifyBackout

    The verifyBackout command searches the upgrade log and report all errors found.

  4. If the backout is successful (no errors or failures reported), then proceed to Server CLI: Reboot the Server procedure in this section.
  5. If the backout failed with the following error, this error can be ignored and the backout may continue.

    ERROR: Upgrade log (/var/TKLC/log/upgrade/upgrade.log) reports errors!

    ERROR: 1485165801::ERROR: <rpm name>-8.6.0.7.0_96.34.0: Failure running command '/usr/TKLC/appworks/bin/eclipseHelp reconfig'

    Also, disregard following error.

    ERROR: Upgrade log (/var/TKLC/log/upgrade/upgrade.log) reports errors!

    ERROR: 1513202476::zip error: Nothing to do!

    /usr/share/tomcat6/webapps/ohw.war

    RCS_VERSION=1.12

    ERROR: Backing out changes from BACKOUT_SERVER on backwards...

    ERROR: Backout was unsuccessful!!!

    ERROR: Trouble when running backout command!

    ERROR: CMD: /var/TKLC/backout/ugwrap --backout

    ERROR: Failed to reject upgrade.

    Rebuilding RPM database. This may take a moment...

    rpmdb_load: /var/lib/rpm/Packages: unexpected file type or format

    Cleaning up chroot environment...

    Stopping remoteExec background process

    Shutting down /var/TKLC/backout/remoteExec...

    /usr/TKLC/plat/sbin/savelogs_plat logs:

    1530516317::ERROR: TKLCdpi-8.0.33-8.0.1.0.0_80.28.0: Adding the DSR helpset failed!

    1530516320::error: %post(TKLCdpi-0:8.0.33-8.0.1.0.0_80.28.0.x86_64) scriptlet failed, exit status 1

    Note:

    Refer to Appendix Y for the workaround.
    ERROR: The upgrade log does not exist!

    Note:

    The above error can be ignored.

    Examine the upgrade log at /var/TKLC/log/upgrade/upgrade.log for errors that occurred during the backout.

    Note:

    If the backout failed due to errors found in the upgrade log, it is recommended to contact My Oracle Support (MOS) for further instructions.

7.6.7 Server CLI: Reboot the Server

Enter the following command to reboot the server:
$ sudo init 6
This step can take several minutes.

Note:

If in case the following alarms are found, then delete core files from /var/TKLC/core directory and restart the server using above command:
>database health impacted.
>persistent database failure
>writing the database to disk failed.
> server core file detected.

7.6.8 Server CLI: Verify OAM services restart (NOAM/SOAM only)

If the server being backed out is a NOAM or SOAM, perform this step; otherwise proceed to step 10.
  1. Wait several (approximately 6 minutes) minutes for a reboot to complete before attempting to log back into the server.
  2. SSH to the server and log in.

    login as: admusr

    password: <enter password>

  3. Execute the following command to verify the httpd service is running.

    $ sudo service httpd status

    The expected output displays httpd is running (the process IDs are variable so the list of numbers can be ignored):

    httpd <process IDs will be listed here> is running...

    If httpd is not running, repeat sub-steps 3 for a few minutes. If httpd is still not running after 3 minutes, then services have failed to restart. It is recommended to contact My Oracle Support (MOS) for further instructions.

  4. Verify if the file id_dsa has required ownership:
    1. Check the ownership of the file:

      sudo ls –ltr /home/awadmin/.ssh/

      The file permission should be defined as shown:

    2. If the file ownership is not set for awadmin, then change the permission:

      sudo chown awadmin:awadm /home/awadmin/.ssh/id_dsa

    3. Verify file ownership is changed to awadmin awadm.

7.6.9 Active NOAM VIP: Verify Server State is Correct after Backout

  1. Navigate to Administration, then Software Management, and then Upgrade to observe the server upgrade status.
  2. Select the SOAM tab of the site being backed out.
  3. Select the link of the server group containing the server being backed out.

    If the server status is Not Ready, proceed to the next step; otherwise, proceed to Active NOAM VIP: Verify application version is correct for the backed out server procedure in this section.

7.6.10 Active NOAM VIP: Change/Correct the Upgrade State on Backed out Server to Ready

  1. Navigate to Status & Manage, then HA.
  2. Click Edit.
  3. Select the backed out server and choose a Max Allowed HA Role value of Active (unless it is a Query server, in which case the value should remain set to Observer).
  4. Click OK.
  5. Verify the Max Allowed HA Role is set to the desired value for the server on the HA Status screen.
  6. Navigate to Status & Manage, then Server.
  7. Select the server being backed out and click Restart.
  8. Click OK to confirm the operation.
  9. Verify the Appl State updates to Enabled.
  10. Navigate to Administration, then Software Management, and then Upgrade.
  11. Select the tab of the server group containing the server to be backed out.
  12. Verify the Upgrade State is now Ready.

    It may take a couple minutes for the grid to update.

7.6.11 Active NOAM VIP: Verify Application Version is Correct for the Backed Out Server

  1. Navigate to Administration, then Software Management, and then Upgrade.
  2. Select the SOAM tab of the site being backed out.
  3. Select the link of the server group containing the server that was backed out.
  4. Verify the Application Version value for this server has been downgraded to the original release version.

    Note:

    To support backout for major upgrade paths on the NOAM, SOAM, and SBR server(s), follow the steps in Additional Backout Steps.

7.7 Backout Multiple Servers

This section provides the procedures to backout the application software on multiple servers. These procedures back out the upgrade of DSR 8.6.0.7.0 application software for multiple servers. Any server requiring a backout can be included, such as DA-MPs, IPFEs, and SBRs.

Caution:

This procedure is executed as a component of the Emergency Site Backout or Normal Site Backout. This procedure should never be executed as a standalone procedure.

7.7.1 Active NOAM VIP: Prepare the Server for Backout

Follow the steps listed in this procedure to prepare the server for backout.
  1. Repeat the steps listed in Active NOAM VIP: Prepare the Server for Backout section to complete this procedure.

7.7.2 Server CLI: Log in to the Server(s)

  1. Use an SSH client to connect to the server (for example, ssh, putty):

    ssh admusr@<server address>

    password: <enter password>

    Note:

    If direct access to the IMI is not available, then access the target server via a connection through the active NOAM. SSH to the active NOAM XMI first. From there, SSH to the target server’s IMI address.

7.7.3 Server CLI: Execute the Backout

  1. Determine the state of the server to be backed out. The server role must be either Standby or Spare.
  2. Execute following command to find the server role:

    $ ha.mystate

    In this example output, the HA state is Standby.

    [admusr@SO2 ~]$ ha.mystate

    resourceId role node subResources lastUpdate

    DbReplication Stby B2435.024 0 0127:113603.435

    VIP Stby B2435.024 0 0127:113603.438

    SbrBBaseRepl OOS B2435.024 0 0127:113601.918

    SbrBindingRes OOS B2435.024 0 0127:113601.918

    SbrSBaseRepl OOS B2435.024 0 0127:113601.918

    SbrSessionRes OOS B2435.024 0 0127:113601.918

    CacdProcessRes OOS B2435.024 0 0127:113601.918

    DA_MP_Leader OOS B2435.024 0 0127:113601.917

    DSR_SLDB OOS B2435.024 0-63 0127:113601.917

    VIP_DA_MP OOS B2435.024 0-63 0127:113601.917

    EXGSTACK_Process OOS B2435.024 0-63 0127:113601.917

    DSR_Process OOS B2435.024 0-63 0127:113601.917

    CAPM_HELP_Proc Stby B2435.024 0 0127:113603.272

    DSROAM_Proc OOS B2435.024 0 0128:081123.951

    If the state of the server is Active, then return to Active NOAM VIP: Prepare the server for backout. Execute the reject command to initiate the backout:

    Note:

    Many informational messages display to the terminal screen as the backout proceeds. After backout is complete, the server automatically reboots.

7.7.4 Server CLI: Restore the Full DB Run Environment

Follow this procedure to restore the full DB run environment.
  1. Repeat the steps listed in Server CLI: Restore the Full DB Run Environment.

7.7.5 Server CLI: Verify the Backout

Follow this procedure to verify backout has been carried out.
  1. Repeat the steps listed in Server CLI: Verify the Backout.

7.7.6 Server CLI: Reboot the Server

Enter the following command to reboot the server:
$ sudo init 6
This step can take several minutes.

Note:

If in case the following alarms are found, then delete core files from /var/TKLC/core directory and restart the server using above command:
>database health impacted.
>persistent database failure
>writing the database to disk failed.
> server core file detected.

7.7.7 Server CLI: Verify OAM Services Restart (NOAM/SOAM Only)

  1. Repeat the steps listed in Server CLI: Verify OAM Services Restart (NOAM/SOAM Only)

    Note:

    To support backout for incremental upgrade paths, execute Appendix K (Additional Backout Steps).

7.7.8 Active NOAM VIP: Verify Server State is Correct after Backout

  1. Navigate to Administration, then Software Management, and then Upgrade to observe the server upgrade status.
  2. If the active NOAM is on release 8.0 or later, and the server status is Not Ready, proceed to the next step; otherwise, proceed to Active NOAM VIP: Verify application version is correct for the backed out server procedure.

7.7.9 Active NOAM VIP: Change/Correct the Upgrade State on Backed Out Server to Ready

  1. Repeat steps 1 to 12 from Active NOAM VIP: Change/Correct the Upgrade State on Backed Out Server to Ready.
  2. Proceed to Active NOAM VIP: Verify application version is correct for the backed out server procedure to complete this procedure.

7.7.10 Active NOAM VIP: Remove Upgrade Ready Status

  1. Log in to the NOAM GUI using the VIP.
  2. Navigate to Status & Manage, then Server.
  3. If the servers just backed-out show an Appl State of Enabled, then multi-select the server rows and click Stop.
  4. Click OK to confirm the operation.

7.7.11 Active NOAM VIP: Correct Upgrade Status on the Backed Out Server

Correct the upgrade status on the backed out server.
  1. Navigate to Administration, then Software Management, and then Upgrade.
  2. If the servers just backed out show an Upgrade State of Ready or Success, then select the backed-out server and click Complete. If the servers just backed out show Upgrade State of Not Ready, then proceed to the next step.
  3. Leave the Action set to the default value of Complete on the Upgrade Complete screen.
  4. Click OK. This updates the Max Allowed HA Role of the backed-out server to active, which causes the server’s Upgrade State to change to Not Ready.

    Note:

    The following SOAP error may appear in the GUI banner:

    SOAP error while clearing upgrade status of hostname=[frame10311b6] ip=[172.16.1.28]

    It is safe to ignore this error message.

7.7.12 Active NOAM VIP: Verify Application Version is Correct for the Backed Out Server

  1. Navigate to Administration, then Software Management, and then Upgrade.
  2. Select the SOAM tab of the site being backed out.
  3. Select the link of the server group containing the server that was backed out.
  4. Verify the Application Version value for this server has been downgraded to the original release version.

    Note:

    To support backout for incremental upgrade paths on the NOAM, SOAM, and SBR server(s), follow the steps in Additional Backout Steps.

7.8 Additional Backout Steps

This procedure provides the details about additional backout steps for NOAM, SOAM, and SBR server(s) to support backout for incremental upgrade release paths.

7.8.1 Server CLI: Log in to the Server

  1. Use the SSH command (on UNIX systems – or putty if running on Windows) to log in to the server under backout:

    ssh admusr@<server address>

    password: <enter password>

    Answer yes if you are asked to confirm the identity of the server.

    If the server is NOAM or SOAM server, execute tasks 2 to 5 in this procedure and if server is SBR server, execute tasks 6. to 7. Please note down the host name of the server on which these steps are executed. Once all the servers in a server group will be backed out then the additional post-backout steps will be executed to revert back the changes done in this procedure.

7.8.2 Server CLI: Set the Resource as Optional for OAM Servers Only

  1. Check for the resource:

    iqt -E HaResourceCfg where "name='<resource_name>'"

  2. Execute this command:

    iset -W -foptional='Yes' HaResourceCfg where "name='DSROAM_Proc'"

    These commands change/update the results of some records.

    Note:

    Make sure the resource being set is in system. Some of the resources shown are introduced in different releases. If the resource is not in the system, presence check will not result any output records. In this case, skip updating these fields for the resource not in the system.

7.8.3 Server CLI: Restart the HTTPD Service (For OAM Servers Only)

  1. Execute this command:

    sudo systemctl restart httpd.service

7.8.4 Active NOAM/SOAM Server CLI: Log in to the Server

  1. Use the SSH command (on UNIX systems – or putty if running on Windows) to log in to the Active NOAM/SOAM server in the same server group, in which server is under backout:

    ssh admusr@<server address>

    password: <enter password>

    Answer yes if you are asked to confirm the identity of the server.

7.8.5 Server CLI: Verify that the Replication is Working Appropriately (For OAM Servers Only)

  1. Execute this command on an active NOAM/SOAM server in the same server group being backed out:

    irepstat

  2. Verify the irepstat command displays a replication row for the server which is being backed out.

    Note the replication status is Active before proceeding. If it is Audit, then wait until replication becomes Active.

    If this step is missed, data is lost and is unrecoverable.

    Example:

    Here Ford-B-NO is Active NOAM Server and Ford-A-NO is backed out.

    Ford-B-NO A3301.157 Ford-B-NO 09:32:17 [Rw]

    Policy 0 ActStb [DbReplication] ----------------------------------------

    AA To P0 Ford-A-NO Active 0 0.00 1%R 0.12%cpu 1.88k/s

    AA To P1 Chevy-DRNO-B Active 0 0.00 1%R 0.08%cpu 1.89k/s

    AB To D0 Camaro-SO-B Active 0 0.00 1%R 0.09%cpu 1.89k/s

    AB To D0 Nova-SO-B Active 0 0.00 1%R 0.08%cpu 1.90k/s

    AB To D0 Pinto-SO-B Active 0 0.00 1%R 0.10%cpu 1.89k/s

    AB To D0 Mustang-SO-B Active 0 0.00 1%R 0.10%cpu 2.14k/s

  3. Press q if you want to exit the irepstat command output.
  4. Execute irepstat again, if required.

7.8.6 Server CLI: Set the Resource as Optional (For SBR Servers Only)

  1. If a resource is not in the system, presence check does not result in any output records. In this case, do not update the fields for the resource.

    Resource presence can be checked using:

    iqt -E HaResourceCfg where "name='<resource_name>'"

    For example:

    iqt -E HaClusterResourceCfg where "resource='uSbrRes'"

    Execute this command for Session SBR only:

    iset -W -foptional='Yes' HaResourceCfg where "name='pSbrSBaseRepl'"

    iset -W -foptional='Yes' HaClusterResourceCfg where "resource='uSbrRes'"

    iset -W -foptional='Yes' HaClusterResourceCfg where "resource='pSbrSessionRes'"

    Execute this command for Binding SBR only:

    iset -W -foptional='Yes' HaResourceCfg where "name='pSbrBBaseRepl'"

    iset -W -foptional='Yes' HaClusterResourceCfg where "resource='uSbrRes'"

    iset -W -foptional='Yes' HaResourceCfg where "name='pSbrBindingRes'"

    These commands change/update the results of some records.

    Note:

    Make sure the resource being set is in the system. Some of the resources listed below are introduced in different releases.

7.8.7 Server CLI: Verify that the Replication is Working Appropriately (For SBR Servers Only)

  1. Execute this command on an active SBR server in the same server group as the server being backed out:

    irepstat

  2. Verify the irepstat command displays a replication row for the server which is being backed out.

    Note the replication status is Active before proceeding, if it is Audit, then wait until replication becomes Active.

    If this step is missed, data is lost and is unrecoverable.

    Example:

    Here Pinto-SBR-2 is Active SBR Server and Pinto-SBR-1 is backed out.

    Also, on Binding SBR, resource will be pSbrBindingPolicy.

    And on Session SBR, resource will be pSbrSessionPolicy.

    Pinto-SBR-2 C3783.034 Pinto-SBR-2 13:39:38 [Rw]

    Policy 0 ActStb [DbReplication] ----------------------------------------

    BC From D0 Pinto-SO-B Active 0 0.10 ^0.10%cpu 67.0/s

    CC To P0 Pinto-SBR-1 Active 0 0.10 1%S 0.31%cpu 30.9/s

    CC To P1 Mustang-SBR-3 Active 0 0.10 1%S 0.28%cpu 39.6/s

    Policy 21 pSbrBindingPolicy [pSbrBBaseRepl] ----------------------------

    CC To P0 Pinto-SBR-1 Active 0 0.10 1%S 0.63%cpu 186k/s

    CC To P1 Mustang-SBR-3 Active 2 0.13 1%S 0.55%cpu 189k/s

  3. Press q if you want to exit the irepstat command output.
  4. Execute irepstat again, if required.

7.9 Additional Post-Backout Steps

This procedure provides the details about additional post-backout steps for NOAM, SOAM, and SBR server(s) to support backout for incremental upgrade release paths.

7.9.1 Server CLI: Log in to the Server (If Not Already Done)

  1. Use the SSH command (on UNIX systems – or putty if running on Windows) to log into the server under backout:

    ssh admusr@<server address>

    password: <enter password>

    Answer yes if you are asked to confirm the identity of the server.

    If the server is an NOAM or SOAM server, execute step 2.

    If the server is an SBR server, execute steps 3.

    Note:

    The host name of the server on which these steps are executed. Once all servers in a server group are backed out, additional post-backout steps are executed to revert the changes done in this procedure. Execute the following commands on servers where the services are in pending state:

    Execute the following commands on servers where the services are in pending state:

    rm –rf /etc/ld.so.cache

    echo "/usr/TKLC/dsr/lib" | sudo tee -a /etc/ld.so.conf.d/dsr.conf

    sudo cat /etc/ld.so.conf.d/dsr.conf

    sudo ldconfig

    Check for configured libraries, for example:

    sudo ldconfig -p | grep -i pdra

    Output must have the following information:

    libPdraTraps.so (libc6,x86-64) => /usr/TKLC/dsr/lib/libPdraTraps.so

    Check whether all the services are up:

    pl

7.9.2 Server CLI: Set the Resource as Optional (For OAM Servers Only)

  1. Repeat the steps listed in Server CLI: Set the Resource as Optional (For OAM Servers Only) .

7.9.3 Server CLI: Set the Resource as Optional (For SBR Servers Only)

  1. Repeat the steps listed in Server CLI: Set the Resource as Optional (For SBR Servers Only).

7.10 Post-Backout Health Check

This procedure is used to determine the health and status of the DSR network and servers following the backout of the entire system.

7.10.1 Active NOAM VIP: Verify Server Status is Normal

  1. Log in to the NOAM GUI using the VIP.
  2. Navigate to Status & Manage, then Server.
  3. Verify Server Status is Normal (Norm) for Alarm (Alm), Database (DB) and Processes (Proc).
  4. Do not proceed with the upgrade if any server status is not Norm.
  5. Do not proceed with the upgrade if there are any Major or Critical alarms.

    Refer to Critical and Major Alarms Analysis for details.

    Note:

    It is recommended to troubleshoot if any server status is not Norm. A backout should return the servers to their pre-upgrade status.

7.10.2 Active NOAM VIP: Log All Current Alarms in the System

  1. Navigate to Alarms & Events, then View Active.
  2. Click Report to generate an Alarms report.
  3. Save the report and print the report. Keep these copies for future reference.

7.11 IDIH Backout

The procedures in this section define the steps to back out the Oracle, Application, and Mediation servers to the previous release.

Oracle Server Backout

Backout of Oracle Server is not supported for release 7.1 or later.

The Oracle server is backed out using the disaster recovery procedure documented in Cloud DSR Disaster Recovery Guide.

Mediation and Application Server Backout

The Mediation and Application servers are backed out using the disaster recovery procedure documented in Cloud DSR Disaster Recovery Guide.