E Maintenance and Troubleshooting Tasks for ACSLS on the Feature Card

This appendix provides maintenance and troubleshooting information for configurations with ACSLS on the SL4000 feature card.

Topics include:

Note:

All hardware-related tasks described in this appendix are to be performed by Oracle Support. These tasks are combined with ACSLS software recovery tasks, which can be performed by Oracle Support or by the customer, in conjunction with Oracle Support.

General Feature Card Maintenance and Troubleshooting Tasks

The following tasks apply to configurations with ACSLS on the feature card.

Performing a Soft Boot of the Feature Card

In certain situations, you may be required to perform a soft reboot of the feature card. If possible, shut down the applications running on the card before performing a soft boot.

To perform the soft boot, log in as user root and issue the following command:

reboot -f

Monitoring the Feature Card

To monitor your feature card hardware performance, follow the directions outlined for viewing the SL4000 GUI Hardware tab. Refer to the SL40000 Library Guide for more information.

Updating Your IP Address, Host Name, or Other Network Related Items

If you need to change your IP address, host name, or other network-related information for the feature card, complete the following steps described in "Configuring the Feature Card and Preparing for ACSLS Installation":

See:

Not Enough Disk Space in the root Partition on the Feature Card

Depending on where you store your ACSLS diagnostic bundles, you may run out of disk space in the root partition on the feature card. You may need to manage this space to maintain a reasonable size or create a soft link to point to the logs elsewhere. The following is an example of the commands you may perform as user root to enable more space, if required:

mkdir /u01/tmp
ln -s /u01/tmp/diags /tmp/diags
chmod 777 /u01/tmp /tmp/diags /u01/tmp/diags

Establishing Mount Points after Removing and Re-inserting a Feature Storage Card

When you remove and re-insert a feature storage card in the SL4000 library, a new mount point is created and the previous mount point may be retained. To eliminate confusion, use one of the following methods to remove the previous mount point if it is also displayed:

  • Automatic unmount (preferred)

    Reboot the impacted feature card following the procedure described in "Performing a Soft Boot of the Feature Card".

  • Manual unmount

    1. Manually unmount the devices associated with '/bkupa' and '/bkupb' using the following operating system command:

      umount <filesystemName>
      

      where <filesystemName> is the /bkupa or /bkupb name.

    2. Run the following script:

      /usr/local/bin/featureCard_acslsStorageManager.sh
      

Troubleshooting and Recovering from Unexpected ACSLS Storage Problems

If you encounter issues where disk layout or mount points have been lost, you may need to repair the file system and storage setup for ACSLS. Depending on the issue, you may need to complete the following steps one or more times to return the system to a normal state:

  1. Before running any of the commands detailed in the following steps, log in as user acsss and enter the following command to shut down the ACSLS application:

    acsss shutdown
    
  2. You may be required to rerun featureCard_acslsStorageManager.sh as user root. This command assumes that you have shut down the ACSLS application. To repair ACSLS storage-related issues, issue the following command:

    cd /usr/local/bin./featureCard_acslsStorageManager.sh
    
  3. A soft reboot may be required as part of the command sequence above, to ensure that all settings have been updated correctly. To do this, log in as user root and issue the following command:

    reboot
    
  4. An issue may require you to mirror RAID storage. Monitor the mirroring process and ensure that it has been completed before continuing to enable ACSLS. As user root, you can use the following command to monitor the rebuilding of the mirror.

    Open a separate terminal window and enter:

    watch cat /proc/mdstat
    

    Monitor the mdstat display to determine when rebuilding of the mirrored drive is complete. Within minutes, [UU] is displayed, indicating that the rebuilding operation has completed.After [UU] is displayed, you can proceed with enabling ACSLS.

FCAT Related Maintenance and Troubleshooting Tasks

The following tasks apply to configurations using the ACSLS Feature Card Availability Toolkit (FCAT) in a dual feature card configuration with ACSLS 8.5.1.

Breaking Trust Between Two Feature Cards

Complete the following steps:

  1. Enter the following command:

    fcatServer_clusterizeFeatureCards.sh -d
    

    The following message is displayed:

    =============================================================
     
    Trust between two feature cards plugged in the same library
    form the backbone for ACSLS availability solution on
    feature cards as it enables the exchange of data and files between
    those two feature cards.
     
    ACSLS availability solution for feature card is *not*
    possible without trust between the two feature cards.
     
    =============================================================
    Do you want to break the trust [root : Side-A => Side-B] ? [Y/N] y
    
  2. At the prompt, enter Y.

    The following message is displayed:

    root: fcatServer_clusterizeFeatureCards.sh: INFO: A simplex trust [root : Side-A => Side-B] is destroyed.
    

To ensure that all trusts are removed, run the above command on both Side-A and Side-B feature cards, first as user acsss and then as user root.

Breaking Trust Between the FCAT Client and Feature Cards

Reference the configuration tasks as described in "Configuring the FCAT Environment":

See "Step 7: Remove Existing Trusts Between the FCAT Client and Feature Cards".

Updating Passwords Associated with Feature Cards

Complete the following steps:

  1. Login to the Side-A feature card as user acsss and shut down the feature card:

    acsss shutdown
    
  2. Repeat the above step for the Side-B feature card.

  3. Remove existing trust between the FCAT Client and feature cards, as described in Appendix C, "Using the ACSLS Feature Card Availability Toolkit":

    See "Step 7: Remove Existing Trusts Between the FCAT Client and Feature Cards".

  4. Remove existing trust between the two feature cards for both user acsss and user root, as described in "Breaking Trust Between Two Feature Cards".

  5. Perform your password changes.

  6. Re-create trust between the feature cards, as described in Appendix C, "Using the ACSLS Feature Card Availability Toolkit".

    See "Step 5: Cluster the Feature Cards and Establish Trust".

  7. Re-create trust between the FCAT client and feature cards, as described in Appendix C, "Using the ACSLS Feature Card Availability Toolkit":

    See "Step 8: Configure the FCAT Client to Recognize Both Side-A and Side-B Feature Cards".

  8. Enable ACSLS as user acsss to begin operations:

    acsss enable
    

Verifying that Trusts are Established Between Feature Cards

Use the fcatClient_featureCardStatus.sh command to verify that trusts are established between the two feature cards installed in the upper two LOC card slots within the same SL4000 library.

If one of the feature cards indicates Unreachable for a given user, then attempt to establish trust again using the fcatServer_clusterizeFeatureCards.sh utility.

If you cannot view both feature cards on the ACSLS Availability Status screen, then verify that you have correctly established trusts in your /root/.ssh/known_hosts file.

Verifying that Database Snapshots are Successfully Replicated

When the FCAT tools are properly configured, the database is automatically replicated between the two feature cards and all four feature storage cards when an ACSLS change results in a database update.

To verify that the backup files and all four feature storage cards are in sync, review the directory /export/backup/ and look for files in the form 2019-05-22_18:13:31.tar. You should see the same set of files for each feature card. Copies are also located on all of the feature storage cards.

Note:

Ensure that the current time for both feature cards is in sync. See "Step 2: Sync Time Between Feature Cards and the SL4000".

Troubleshooting Feature Card Tools

All of the fcatClient scripts are certified to run on the OBI Linux release 6.8, 6.10, 7.3, 7.6, and 7.8 platforms. Other versions of Unix may require standard porting methods.