C Using the ACSLS Feature Card Availability Toolkit

This appendix describes how to use the ACSLS Feature Card Availability Toolkit (FCAT) in a dual feature card configuration with ACSLS 8.5.1.

Topics include:

• Overview

• Configuring the FCAT Environment

Overview

Beginning with ACSLS 8.5.1, you can install and run ACSLS on dual feature cards within the SL4000 library.

Depending on your available hardware, application installation, and configuration, different forms of failover are supported by ACSLS to enable continuous operation for both ACSLS and attached client applications. The ACSLS Feature Card Availability Toolkit (FCAT) is a suite of tools and capabilities that have been both expanded and newly developed to provide this application support on the feature card.

FCAT scripts are included in the bundle of scripts you extracted during initial feature card configuration. See "Step 4: Extract the ACSLS Feature Card Scripts".

Pre-Installation Requirements for FCAT

Ensure that the following equipment and system information are in place prior to installation:

Note:

All hardware installation tasks are to be completed by Oracle Support.

• An additional FCAT Client Server, running a version of Linux supported by ACSLS. This can be:

  • A standalone server

  • A server in use with a separate Linux partition

  • A server with the space and network connectivity available to run the ACSLS FCAT monitoring and switch utilities.

• FCAT client server credentials

• User and library equipment as described in "Pre-Installation Requirements for ACSLS on the Feature Card".

• A separate ACSLS license for each feature card. Work with your Oracle Support and Sales representatives as appropriate.

• Both feature cards must be inserted and active for all of the configuration tasks described in this appendix. However, the bond3 (public) network interface must be inactive on both feature cards. Do not insert both feature cards simultaneously unless you have deactivated the bond3 network. To do this, enter the following command on each feature card separately:

  ./featureCard_bond3.sh  -d
  

Configuring the FCAT Environment

Perform the following tasks to configure the FCAT environment:

Note:

In the steps below that require you to manually edit specific configuration files, please make a restorable copy of the original.

• Step 1: Install and Configure ACSLS on Both Feature Cards

• Step 2: Sync Time Between Feature Cards and the SL4000

• Step 3: Extract and Install the Feature Card Availability Toolkit

• Step 4: Shut Down ACSLS on Both Feature Cards

• Step 5: Cluster the Feature Cards and Establish Trust

• Step 6: Copy Select Feature Card Scripts to a Separate FCAT Linux Client

• Step 7: Remove Existing Trusts Between the FCAT Client and Feature Cards

• Step 8: Configure the FCAT Client to Recognize Both Side-A and Side-B Feature Cards

• Step 9: Monitor ACSLS Availability from the FCAT Client

• Step 10: Enable ACSLS on the Side-A feature Card

• Step 11: Trigger Failover from One Feature Card to the Other

• Step 12: Start ACSLS (with optional Database Restore)

Step 1: Install and Configure ACSLS on Both Feature Cards

To begin, choose one of the following configuration options, based on whether you want to configure both feature cards with the same IP Address and DNS name, or with two different IP addresses and DNS names:

• Option 1

  In this option, both feature cards use the same IP address and DNS name.

  Figure C-1 FCAT Configuration Option 1

  
  

  As shown in Figure C-1, for clients attached to ACSLS through the feature card, all client addressing uses the same IP address and DNS name, regardless of which feature card you are using at a given time. bond2 is the network interface used only for internal library communications. bond3 is the customer network interface used to communicate with the feature cards.

  This is the default option.

• Option 2

  In this option, each feature card uses a different IP address and DNS name.

  Figure C-2 FCAT Configuration Option 2

  
  

  As shown in Figure C-2, for clients attached to ACSLS through the feature card, client addressing uses distinct IP addresses and DNS names for each feature card. This option allows you to switch connected clients to run against a specific instance of ACSLS. bond2 is the network interface used only for internal library communications. bond3 is the customer network interface used to communicate with the feature cards.

Once you have chosen your option, install and configure ACSLS on each feature card as described in Chapter 4, "Installing ACSLS on the SL4000 Feature Card".

WARNING:

When installing ACSLS, ensure that you have configured both feature card platforms and installed instances of ACSLS identically. Otherwise, ACSLS failover, startup, and other operations may not function properly.

Step 2: Sync Time Between Feature Cards and the SL4000

As noted in the ACSLS Administrator's Guide, for proper operation of ACSLS with the SL4000, the ACSLS server clock must be in sync with the SL4000 internal clock. This is important on a standalone ACSLS server running against the SL4000 as well as when using ACSLS on the SL4000 feature cards. This synchronization ensures that database backups are current and correct.

Synchronize the following:

• The time between both feature cards and the SL4000.

• The time between both feature cards.

  If you do not already have access to an NTP (Network Time Protocol) server, establish one to keep the information replicated between the two feature cards in sync.

Step 3: Extract and Install the Feature Card Availability Toolkit

The Feature Card Availability Tool kit is bundled with the initial feature card scripts extracted during feature card installation, as described in "Configuring the Feature Card and Preparing for ACSLS Installation".

The following FCAT scripts are included:

• fcatServer_clusterizeFeatureCards.sh

• fcatServer_featureCardStatusPayload.sh

• fcatServer_switchActivePassiveCard.sh

• fcatClient_featureCardStatus.sh

• fcatClient_triggerSwitch.sh

Ensure that these scripts are available under /usr/local/bin on both the Side-A and Side-B feature card, and reload them if necessary.

• The Side-A feature card is installed in the Feature Card Kit 1 location.

• The Side-B feature card is installed in the Feature Card Kit 2 location.

See "Feature Card Locations" for more information.

Step 4: Shut Down ACSLS on Both Feature Cards

To shut down the ACSLS application on both the Side-A and Side-B feature card:

1. Log in to the Side-A feature card as user acsss and enter the following command to shut down the ACSLS application:

   cd $ACS_HOME/bin
   cmd_proc_shell idle
   acsss shutdown
   

2. Verify that ACSLS is completely shut down.

   acsss status
   

3. Repeat this procedure for the Side-B feature card.

Step 5: Cluster the Feature Cards and Establish Trust

You must cluster the feature cards to establish trust between them. This enables ACSLS to keep both feature cards current with their database updates and ensures that ACSLS can be quickly resumed.

To establish trust between both feature cards, you must establish trust from the Side-A to Side-B feature card, and from the Side-B feature card to Side-A feature card, as follows:

1. Establish Trust from Side-A to Side-B feature card for user acsss.

   1. On the Side-A feature card, as user acsss, access the directory /usr/local/bin:

      cd /usr/local/bin
      
      

   2. Run the following command:

      fcatServer_clusterizeFeatureCards.sh
      
      

      The following message appears:

      =============================================================Trust between two feature cards plugged in the same libraryform the backbone for ACSLS availability solution on featurecards as it enables the exchange of data and files betweenthose two feature cards.ACSLS availability solution for feature card is *not*possible without trust between the two feature cards.=============================================================Do you want to create the trust [acsss:Side-A => Side-B](yes/no)?
      
      

   3. At the prompt, enter yes to create the trust.

   4. At the password prompt, enter the password for user acsss on the Side-B feature card.

      Trust is established and a confirmation message appears:

      root: fcatServer_clusterizeFeatureCards.sh: INFO: A simplex trust
      [acsss : Side-A => Side-B] is created successfully.
           
      =============================================================A simplex trust is created between feature cards from Side-Ato Side-B for 'acsss'. In order to break this trust, executerunuser -l acsss -c 'fcatServer_clusterizeFeatureCards.sh -d'on <server_name> from command line.=============================================================
      

   5. As user acsss, exit the session:

      exit
      

2. Establish trust from the Side-A to Side-B feature card for user root.

   1. On the Side-A feature card, as user root, access the directory /usr/local/bin:

      cd /usr/local/bin
      
      

   2. Run the following command:

      fcatServer_clusterizeFeatureCards.sh
      
      

      The following message appears:

      =============================================================Trust between two feature cards plugged in the same libraryform the backbone for ACSLS availability solution on featurecards as it enables the exchange of data and files betweenthose two feature cards.ACSLS availability solution for feature card is *not*possible without trust between the two feature cards.=============================================================
      Do you want to create the trust [root:Side-A => Side-B](yes/no)?
      
      

   3. At the prompt, enter yes to create the trust.

   4. At the password prompt, enter the password for user root on the Side-B feature card.

      Trust is established and a confirmation message appears:

      root: fcatServer_clusterizeFeatureCards.sh: INFO: A simplex trust
      [root : Side-A => Side-B] is created successfully.
      
      =============================================================A simplex trust is created between feature cards from Side-Ato Side-B for 'root'. In order to break this trust, executerunuser -l root -c 'fcatServer_clusterizeFeatureCards.sh -d'on <server_name> from command line.=============================================================
      

   5. As user root, exit the session:

      exit
      

3. Repeat steps 1 and 2 from the Side-B feature card to establish trust from the Side-B to Side-A feature card for both user acsss and user root.

   Once trust is fully established, information can flow between the two feature cards and you can use the ACSLS Feature Card Availability Toolkit tools to monitor the feature cards for any issues that may be encountered.

Step 6: Copy Select Feature Card Scripts to a Separate FCAT Linux Client

To enable the ability to monitor ACSLS and trigger a switch from one feature card to another, copy the following FCAT scripts to a dedicated Linux Client server (Client host):

• fcatClient_featureCardStatus.sh

• fcatClient_triggerSwitch.sh

These scripts can be copied from the /usr/local/bin directory on either the Side-A or Side-B feature card, whichever has an enabled bond3 IP address. Although these client scripts are agnostic to the directory in which they will reside on the client host, it is recommended that you dedicate a directory for these scripts.

When you are finished, verify that the files are listed in the destination directory.

Step 7: Remove Existing Trusts Between the FCAT Client and Feature Cards

In preparation for configuring the FCAT client to recognize your feature cards, you must first ensure that any older, existing trusts are removed.

1. On the FCAT client, as user root, open the following file in the text editor of your choice:

   /root/.ssh/known_hosts

   This file includes entries for various known hosts. For example:

   <FC_HOSTNAME>,<IP_BOND3> ssh-rsa
   AAB3NzaC1yc2EAABIwAAQEA3EMv/fPWJoa9ZAVWYrdr5yfs5N2G/AsBSN/Mu/GI79KFELw
   6qfFCxagQaf7f/w0taer+Rzbovog3Tp2NGikdstdCX02/ucpcDbpp2CNcF8imnEsL5H76I
   1y8CMEQ1t3xDNZz5WXuPeCDT17Nq3KXtRt7CO0iNgPQhQB210jG02S/Nt9AJK7xiaTh8OM
   FwiaBrCowQugCGPHanZo7NP1X9ZT1VP5RGnqIyfYyZSDZzkUBS73GxGcGiEmARS0BODjFS
   kKrqOKpdhc/Z7EYsw==
   

   Where:

   • <FC_HOSTNAME> is the feature card name for Side-A or Side-B.

   • <IP_BOND3> is the feature card IP address for Side-A or Side-B.

2. Review this file to locate any entries associated with the two feature cards. Delete these outdated entries for IP addresses and associated host names.

3. Save your changes and close the file.

Step 8: Configure the FCAT Client to Recognize Both Side-A and Side-B Feature Cards

Complete the following steps to establish Side-A and Side-B feature card keys in the known_hosts file. This allows net operations between the platforms.

1. Proceed according to your chosen configuration option:

   • If you are using Option 1, using one IP address and DNS name for both feature cards, proceed with step 2.

   • If you are using Option 2, with distinct IP addresses and different DNS names for each feature card, skip to Step 5.

2. Ensure that the Side-B feature card is disabled on the bond3 public network. Use the following commands if necessary:

   cd /usr/local/bin
   featureCard_bond3.sh -d
   

3. Ensure that the Side-A feature card is enabled on the bond3 public network. Use the following commands if necessary:

   cd /usr/local/bin
   featureCard_bond3.sh -e
   

4. Perform the following steps to update the known_hosts file on your FCAT Client, adding the feature card IP address and DNS name for the Side-A feature card.

   1. On the FCAT Client, as user root, run the following command using the
      FC_HOSTNAME for the Side-A feature card:

      ssh <FC_HOSTNAME>
      

      The following message appears:

      The authenticity of host '<FC_HOSTNAME> (FC_BOND3_IP)' can't be established.
      RSA key fingerprint is 85:44:72:86:3f:e1:6d:44:42:8c:6d:31:5d:b4:97:5c.
       
      Are you sure you want to continue connecting (yes/no)? yes
       
      Warning: Permanently added '<FC_HOSTNAME>, <IP_BOND3_IP>' (RSA) to the list of known hosts.
       
      root@<FC_HOSTNAME>'s password:
      

   2. At the prompt, enter yes to continue the connection.

      A password prompt appears.

   3. At the password prompt, enter the root user password for the feature card you are attempting to connect to.

      A Login message appears, indicating that you are logged into the targeted feature card.

   4. Exit the session to return to your FCAT Client. The key for the Side-A feature card is now in the /root/.ssh/known_hosts file.

5. You have now successfully established the key for the Side-A feature card. If you are using an additional (Side-B) feature card, complete the following steps:

   1. On the FCAT client, as user root, open the following file in the text editor of your choice:

      /root/.ssh/known_hosts

      This file includes entries for various known hosts.

   2. Edit the file, placing a # sign at the beginning of the entry for the Side-A feature card to comment out this entry.

   3. Save your changes and close the file.

   4. Return to step 1 of this procedure and perform steps 1-4 to establish the key for the Side-B feature card, just as you did for Side-A.

   5. Update the known_hosts file again by removing the # sign to make the Side-A feature card active.

   6. On the Side-A feature card, as user root, enable the bond3 public network:

      cd /usr/local/bin
      featureCard_bond3.sh -e
      

   7. On the Side-B feature card, as user root, disable the bond3 public network:

      cd /usr/local/bin
      featureCard_bond3.sh -d
      

6. You have now successfully established the keys for both Side-A and Side-B feature cards, allowing the FCAT Client to recognize them.

Step 9: Monitor ACSLS Availability from the FCAT Client

To monitor the status for both feature cards:

1. On the FCAT Client, locate the directory where you copied the following files:

   • fcatClient_featureCardStatus.sh

   • fcatClient_triggerSwitch.sh

2. Run one of the following commands:

   • If you are using Option 1, with both feature cards using the same IP address and DNS name, enter one of the following commands:

     fcatClient_featureCardStatus.sh <HN-1>
     
     fcatClient_featureCardStatus.sh <IP_BOND3>
     

     where HN-1 is the host name used for both feature cards.

   • If you are using Option 2, with each feature card using a distinct IP address and DNS name, enter one of the following commands:

     fcatClient_featureCardStatus.sh <HN-1> [<HN-2>]
     
     fcatClient_featureCardStatus.sh <IP_BOND3_AorB> [<IP_BOND3_AorB>]
     

     where:

     • HN-1 and HN-2 are the distinct host names for your feature cards.

     • AorB indicates the feature card location, Side-A or Side-B, as appropriate.

3. You may be prompted to enter a password if the trust between the FCAT Client and feature cards have been established, or conditions for the previous trust have changed.

4. Once the fcatClient_featureCardStatus.sh command is processed, monitoring is enabled for ACSLS on both feature cards. The monitoring screen is refreshed every 30-35 seconds, and displays the following:

   • Active or passive status for each feature card

   • Whether trust is established and active (Peer status is Reachable).

   • Payload ID (increments with every screen refresh)

   • Additional information, including the status of the ACSLS application

The following is an example of the ACSLS Availability Status screen at initial startup:

=======================================================================
 
                ACSLS Availability Status
 
Payload ID      : 1
Request Time    : Wed May 22 13:46:20 MDT 2019
Report Time     : Wed May 22 13:46:22 MDT 2019
 
*** Feature cards need to synchronize their clocks.----------------------------------------------------------------------
 
Side                : Side-A
Node                : FC_hostname_A [ <IP_BOND3_A> ]
Availability Status : Active since 2019-05-22 19:33:14
Peer Status (root)  : Reachable [ EXTERNAL ]
Peer Status (acsss) : Reachable [ EXTERNAL ]
Hardware Status     : up 2:08, 6 users, load average: 0.00, 0.02, 0.05
ACSLS Uptime        : 0-00:00:00 <days-hrs:min:sec>
 
   rmi-registry is offline
   surrogate is offline
   acsdb is offline
   acsls is offline
 
----------------------------------------------------------------------
----------------------------------------------------------------------
 
Side                : Side-B
Node                : FC_hostname_B [ <IP_BOND3_B> ]
Availability Status : Passive 
Peer Status (root)  : Reachable [ EXTERNAL ]
Peer Status (acsss) : Reachable [ EXTERNAL ]
Hardware Status     : up 1:51, 3 users, load average: 0.08, 0.03, 0.05
ACSLS Uptime        : 0-00:00:00 <days-hrs:min:sec>
 
   rmi-registry is offline
   surrogate is offline
   acsdb is offline
   acsls is offline
 
----------------------------------------------------------------------
 
======================================================================

This display is based on Option 2, with each feature card using a distinct IP address and different DNS name.

• Status is displayed for both feature cards, as they are both available.

• [EXTERNAL] on the Peer status lines indicates Option 2. [INTERNAL] would indicate Option 1.

• ACSLS is offline as it is not yet enabled.

• The display alerts you that the clocks must be synchronized between both feature cards.

Step 10: Enable ACSLS on the Side-A feature Card

On the Side-A feature card, enter the following command to enable ACSLS:

acsss enable

Only one instance of ACSLS on the feature card can be enabled at one time.

Step 11: Trigger Failover from One Feature Card to the Other

In the event that failover is required, you must trigger a switch between the two feature cards from the FCAT Client. The active feature card becomes passive and the passive feature card becomes active.

Note:

• Before triggering a switch, quiesce client jobs and other operations to ensure a smooth transition to the newly active feature card.

• While the switchover activates the other card, it does not automatically activate ACSLS on that card. You will need to do so manually after the card switchover has occurred.

To trigger a feature card switch:

1. On the FCAT Client, locate the directory where you copied the
   fcatClient_triggerSwitch.sh file:

2. Run one of the following commands:

   fcatClient_triggerSwitch.sh <HN>
   fcatClient_triggerSwitch.sh <IP_BOND3>
   

   where HN is the host name of the feature card to perform the switchover.

3. You may be prompted to enter a password if the trust between the FCAT Client and feature cards have been established, or conditions for the previous trust have changed.

4. Once the fcatClient_triggerSwitch.sh command is entered, an updated ACSLS Availability Status screen appears. This screen is refreshed every 30-35 seconds.

   Switch processing may take several minutes. The following message:

   Recovery (PID=number) is in progress...
   

   indicates that the switch is still in process. This screen continues to refresh every 30-35 seconds until the switch is complete. PID is the process ID number.

   Once the feature card switchover is complete, the Status screen is updated.

   • The previously active feature card is now passive and is disabled (offline).

   • The previously passive feature card is now active. ACSLS can now be manually enabled on that card.

   For example:

   =======================================================================
    
                   ACSLS Availability Status
    
   Payload ID      : 55
   Request Time    : Wed May 22 14:59:46 MDT 2019
   Report Time     : Wed May 22 14:59:48 MDT 2019
    
   ----------------------------------------------------------------------
    
   Side                : Side-A
   Node                : FC_hostname_A [ <IP_bond3_A> ]
   Availability Status : Passive
   Peer Status (root)  : Reachable [ INTERNAL ]
   Peer Status (acsss) : Reachable [ INTERNAL ]
   Hardware Status     : up 3:22, 8 users, load average: 0.10, 0.26, 0.16
   ACSLS Uptime        : 04:19 <days-hrs:min:sec>
    
      rmi-registry is offline
      surrogate is offline
      acsdb is online
      acsls is offline
    
   ----------------------------------------------------------------------
   ----------------------------------------------------------------------
    
   Side                : Side-B
   Node                : FC_hostname_B [ <IP_bond3_B> ] 
   Availability Status : Active since 2019-05-22 20:41:11
   Peer Status (root)  : Reachable [ INTERNAL ]
   Peer Status (acsss) : Reachable [ INTERNAL ]
   Hardware Status     : up 3:04, 4 users, load average: 0.08, 0.03, 0.05
   ACSLS Uptime        : 0-00:00:00 <days-hrs:min:sec>
    
      Recovery (PID=12698) is in progress. Please try later...
    
   ----------------------------------------------------------------------
    
   =======================================================================
   

   Note:

   • In this example, there was no true hardware failure. Therefore, status is displayed for both feature cards. If a true hardware failure occurs and a feature card is unreachable, you may see an error or incomplete status screen.

   • If the time delta between the two feature cards is significant, a warning message is displayed and the two feature cards must be resynchronized.

Step 12: Start ACSLS (with optional Database Restore)

After completing failover from one card to the other, ACSLS must be restarted manually on the newly active card. You can choose to do a database restore before starting ACSLS, or simply start ACSLS and it will use the last automated snapshot, which is synchronized between the paired Feature Cards automatically.

To simply start ACSLS and use the latest database snapshot, on the newly active feature card, perform the following steps:

1. Ensure ACSLS and the database are in a known and synchronized state by shutting down:

   su - acsss
   acsss shutdown
   

2. Start ACSLS for continued operation:

   acsss enable
   

If you wish to restore another database (other than the one synchronized between the Feature Cards), perform the following steps:

1. As user acsss, restore the ACSLS database from the most recent backup or an accessible backup location:

   Run db_restore.sh:

   su - acsss
   cd /bin
   ./db_restore.sh latest
   or
   ./db_restore.sh/bkupb/<DB_SNAPSHOT_FILENAME>
   

   Substitute the specific file name of your backup for latest and
   <DB_SNAPSHOT_FILENAME>.

   Redundant copies of recent backup files reside under /bkupa, /bkupb or /bkupc. If no ACSLS database backups are available to you locally, you may want to pull the latest database backup from your remote backup server. If you do not want to use an existing database backup, and would rather create a new one, then run
   accss_config instead.

2. Enable ACSLS as user acsss:

   su – acsss
   acsss enable
   

   Once ACSLS is enabled, you can set your client jobs to run and resume normal operations.