A.3 Back Out a Single Server

The following procedure performs backout on a single server. Backout can be performed on all upgrades except a major upgrade.

1. In the Primary SDS NOAM VIP, ensure the server to be downgraded is in the Accept or Reject state.
2. Expand Administration select Software Management click Upgrade.
3. Select the tab containing the server(s) to be backed out.
4. Verify the Upgrade State is Accept or Reject.
5. Set the Max Allowed HA Role to Standby.
6. Expand Status & Manage click HA.
7. Click Edit.
8. Select the server(s) to be backed out and select a Max Allowed HA Role value of Standby (unless it is a Query server, in which case the value should remain set to Observer).
9. Click OK.

   Note:

   If downgrading the active primary SDS NOAM server, then continue with the next step of this procedure; otherwise, skip to step 14 of this procedure.
10. If downgrading the active primary SDS NOAM server, a HA fail over occurs. The user’s GUI session ends as the active primary SDS server goes through HA fail over and becomes the Standby server.

    Note:

    If the server being backed out is the active NOAM and an HA fail over does not happen after step 2, and the OAM HA Role of the NOAMP server to be backed out on the HA status screen is still Active, then you have encountered a known issue. Apply the workaround using Appendix L to have the NOAMP HA fail over.
11. Click Logout to log out of the SDS NOAM GUI.

    Figure A-48 Log out

    
    

    

    
    
12. In the Primary SDS NOAM VIP, clear the cached data. JavaScript libraries, images, and other objects are often modified in the upgrade. Browsers can sometimes cause GUI problems by holding on to the old objects in the built-in cache. To prevent these problems, always clear the browser cache before logging into an OAM GUI that has just been upgraded:
    1. Simultaneously press and hold the Ctrl, Shift, and Delete keys (most Web browsers).
    2. Select the appropriate object types to delete from the cache (for example, Temporary Internet Files, Cache, or Cached images and files, and so on). Other browsers may label these objects differently.
    3. Clear the cached data.

    Note:

    Do NOT proceed until the browser cache has been cleared.
13. Access the primary SDS NOAM GUI, use the VIP address to access the primary SDS NOAM GUI as described in Access the OAM GUI Using the VIP (NOAM/SOAM).
14. In the Primary SDS NOAM VIP, record PDB Relay Enabled state. Expand SDS select Configuration click Options.

    Figure A-49 Options

    
    

    

    
    
15. Locate the PDB Relay Enable check box and record if it is checked or not checked.

    Figure A-50 PDB Relay Enable check box

    
    

    

    
    

    Note:

    If the PDB Relay Enabled checkbox is CHECKED, then continue with the next step of this procedure. If the PDB Relay Enabled checkbox is NOT CHECKED, then skip to step 19 of this procedure.
16. In the Primary SDS NOAM VIP (CLI), access the active primary SDS NOAM. Use the VIP address to log into the active primary SDS NOAM with the admusr account.

    sds-rlghnc-a login: admusr
    Password: <admusr_password>
    *** TRUNCATED OUTPUT ***
    RELEASE=6.4
    RUNID=00
    VPATH=/var/TKLC/rundb:/usr/TKLC/appworks:/usr/TKLC/awpcommon:/usr/TKLC/comagent-gui:/usr/TKLC/comagent-gui:/usr/TKLC/comagent:/usr/TKLC/sds
    PRODPATH=/opt/comcol/prod
    RUNID=00
    [admusr@sds-rlghnc-a ~]$
    

17. Set the pdbRelay TimeStamp to "0".

    [admusr@sds-rlghnc-b ~]$ sudo iset -fvalue=0 ProvOptions where "var='pdbRelayMsgLogTimeStamp'"

18. Exit the CLI for the active primary SDS NOAM.
    [admusr@sds-rlghnc-b ~]$ exit
    logout
19. In the Primary SDS NOAM VIP, stop the software. Expand Status & Manage click Server.
20. Select the server(s) to be backed out and click Stop.
21. Click OK to confirm.
22. Verify the Appl State updates to Disabled.
23. Verify the server(s) are back out ready. Expand Administration select Software Management click Upgrade. Select the tab for the server group containing the server(s) to be backed out.

    Note:

    It may take a couple minutes for the grid to update.
24. Verify the Upgrade State displays as Backout Ready.

    Note:

    If this is the active server in an Active-Standby pair, these steps cause an HA fail over. The HA fail over is an expected outcome. Continue with the steps on the new active NOAMP.
25. In the Server CLI, the SSH to the server(s) to be backed out. Use the SSH command (on UNIX systems — or putty if running on Windows) to log into the active NOAM.
    ssh <NOAM XMI IP address>
    login as: admusr
    password: <enter password>

    Note:

    If direct access to the XMI is not available, then access the target server using a connection through the active NO. SSH to the active NO XMI first. Once logged into the NO, SSH to the target server’s XMI address.
26. Run the back out using the reject script.

    $ sudo /var/TKLC/backout/diUpgrade --reject
            ***Executing.. /var/TKLC/backout/backout_server
            --checkVerifying that backout is possible.Checking for stale RPM DB locks...Current platform version: 
            7.0.2.0.0-86.30.0Continue backout? 
            [y/N]: y

    The server reboots and the user is automatically logged out.
27. Use the SSH command (on UNIX systems — or putty if running on Windows) to log into the active NOAM.
    ssh <NOAM XMI IP address>
    login as: admusr
    password: <enter password>
28. Verify the Back out, examine the upgrade logs in the /var/TKLC/log/upgrade directory and verify no errors are reported.
    $ grep ERROR /var/TKLC/log/upgrade/upgrade.log

    Note:

    The following errors can be ignored:

    • DEBUG: 'iqt' command failed (is IDB running?)
    • 1477080063::ERROR: TKLCsds-7.0.0-7.0.1_70.12.0: Failure running command '/usr/TKLC/appworks/bin/eclipseHelp reconfig'
    • 1477080521::ERROR: prod.dbdown: unknown option (-i)
    • 1517455316::ERROR: Cannot execute command!
    • 1517455316::ERROR: CMD: /usr/sbin/hpacucli controller all show config detail
    • 1517455316::ERROR: ERROR: No such file or directory
    • 1517455316::ERROR: Unable to get the HP disk configuration!
    • 1517455316::ERROR: Command Failed!
    • 1517455316::ERROR: Child process has exited with
    • 1517455316::SYSERROR: No such file or directory
    • 1526453748::ERROR: Cannot reduce filemgmt enough to leave room for dual image upgrade

    If the back out was not successful, because other errors were recorded in the logs, then contact My Oracle Support for further instructions. If the back out was successful (no errors or failures), then continue with the remaining steps.

29. Restore the COMCOL Full DB/Run environment, Run the backout_restore utility to restore the full database run environment.

    $ sudo /var/tmp/backout_restore
    *** TRUNCATED OUTPUT ***
    This process will totally destroy the existing DB on this server.  This should only be done to recover a server when an upgrade has been backed-out/rolled-back.
    Are you sure you want to proceed? (y|n): y
    Answer y to continue the restore.
    

    Note:

    The COMCOL restore process may take several minutes to complete. If the restore was successful, the following displays:

    Success: Full restore of COMCOL run env has completed.
    If an error is encountered and reported by the utility, then work with My Oracle Support for further instructions.

    Note:

    In some incremental upgrade scenarios, the backout_restore file is not found in the /var/tmp directory, resulting in the /var/tmp/backout_restore: No such file or directory error message. If this message occurs, copy the file using sudo from /usr/TKLC/appworks/sbin to /var/tmp and repeat the command.
30. Reboot the server. Run the command:
    $ sudo init 6

    This step can take several minutes and terminates the SSH session.

31. Use the SSH command (on UNIX systems — or putty if running on Windows) to log into the active NOAM.
    ssh <NOAM XMI IP address>
    login as: admusr
    password: <enter password>
32. Restore softlink for Comagent directory.
    [admusr@HPC-NO1 ~]$ cd /var/TKLC/appworks/library
    $ sudo ln -s /usr/TKLC/comagent-gui/gui/ Comagent

    Verify if the Comagent link has been restored:

    Figure A-51 Comagent link

    
    

    

    
    

    If the output is received as highlighted in red, the softlink for Comagent directory has been restored.

33. In the Server CLI, verify if the httpd service has restarted. If this is an NO or SO, verify httpd service is running.
    sudo systemctl status httpd.service
    httpd (pid xxxx) is running...

    Note:

    The process IDs are variable so the actual number value can be ignored.
34. If httpd is not running, wait for a few minutes and retry the command. If httpd is still not running after 3 minutes, then services have failed to restart. Contact My Oracle Support for further instructions.
35. Verify if the file id_rsa has required ownership, check the ownership of the file:
    ls –ltr /home/awadmin/.ssh/
    The file permission should be defined as shown:

    Figure A-52 Permission

    
    

    

    
    
    If the file ownership is not set for awadmin, then change the permission:
    sudo chown awadmin:awadm /home/awadmin/.ssh/id_rsa
    Verify file ownership is changed to awadmin awadm.
36. In the Primary SDS NOAM VIP, verify the server(s) application version and upgrade state. Expand Administration select Software Management click Upgrade. Select the tab containing the server(s) that were backed out. Verify the Application Version value for this server has been backed out to the source release version. Verify the Upgrade State.

    Note:

    Full audit between active NO and backed out server is conducted and it may take up to 10 minutes before the Upgrade State is changed to Ready.
37. In the Primary SDS NOAM VIP, set the Max Allowed HA Role to Active. Due to back out being initiated from the command line instead of through the GUI, modify the backed out server so its Upgrade State changes to Ready. Expand Status & Manage click HA. Click Edit.
38. Select the backed out server(s) and choose a Max Allowed HA Role value of Active (unless it is a Query server, in which case the value should remain set to Observer). Click OK.Verify the Max Allowed HA Role is set.
39. Restart the software, Expand Status & Manage click Server. If the server(s) that was backed out displays an Appl State, state of Enabled, skip to the next step. If the server(s) that was backed out displays an Appl State, state of Disabled, select the server(s) and click Restart. Click OK to confirm. Verify the Appl State changes to Enabled.
40. In the Primary SDS NOAM VIP, verify the Upgrade State. Expand Administration select Software Management click Upgrade. Select the tab of the server group containing the server(s) that was backed out. Verify the Upgrade State is now Ready(it may take several seconds for the grid to update).
41. Stop the software (if necessary). Due to backout being initiated from the command line instead of through the GUI, modify the Upgrade State of the backed out server(s) to achieve a state of Not Ready. Expand Status & Manage click Server. If the server(s) that was backed out displays an Appl State state of Enabled, then select the server(s) and click Stop.
42. In the primary SDS NOAM VIP, verify the server(s) Upgrade State. Expand Administration select Software Management click Upgrade. If the server(s) that was backed out displays an Upgrade State of Not Ready, then go back to step 37 of this procedure.
43. Complete the backout action (if necessary), If the server(s) that was backed out displays an Upgrade State of Ready or Success, then:
    • Select the server(s) that was backed out and click Complete.Leave the Action set to its default value of Complete.
    • Click OK to confirm the action.

    This changes the Max Allowed HA Role of the backed out server(s) to Active, which causes the server Upgrade State to change to Not Ready.

    The user may see the following SOAP error display on the GUI banner.

    SOAP error while clearing upgrade status of hostname=[frame10311b6]
            ip=[172.16.1.28]

    It is safe to ignore this error message.