7.1. Arrays

In SGD, an array is a collection of SGD servers that share configuration information.

Arrays have the following benefits:

Users see the same webtop and can resume applications, no matter which SGD server they log in to.

This section includes the following topics:

7.1.1. The Structure of an Array

An array contains the following:

  • One primary server. This server is the authoritative source for global SGD information, and maintains the definitive copy of the organizational hierarchy, called the local repository.

  • One or more secondary servers. The primary server replicates information to these servers.

A single, standalone server is considered to be the primary server in an array with no secondary servers.

SGD servers in an array might run different operating systems. However, all the array members must run the same version of SGD.

As the SGD servers in an array share information about user sessions and application sessions, it is important to synchronize the clocks on the SGD hosts. Use Network Time Protocol (NTP) software, or the rdate command, to ensure the clocks on all SGD hosts are synchronized.

7.1.2. Replicating Data Across the Array

When the primary server replicates data to the secondary servers, it replicates the following data:

  • The local repository

  • Session information

  • Configuration information, including global configuration

  • Client profiles created by SGD Administrators

  • User preferences created by SGD users from the webtop

  • The application server password cache

  • Resource files, such as application server login scripts

Apart from the resource files, any changes to the above data are replicated immediately.

The synchronization of resource files occurs once daily, and only while the servers are running. The resource files that are synchronized are the files from the /opt/tarantella/var/serverresources directory. Only add, modify, or delete the files in these directories on the primary server.

The time and effort that it takes to synchronize an array is directly proportional to the size of the array.

Resource synchronization can be scheduled to take place at a time of your choice. In the Administration Console, this is configured with the Daily Resource Synchronization Time attribute on the Performance tab for each SGD server.

7.1.3. Communication Between Array Members

In the array, each SGD server has a peer Domain Name System (DNS) name and one or more external DNS names. SGD servers always use peer DNS names to communicate with each other. You also use peer DNS names when specifying array members in the SGD configuration tools. External DNS names are only used by SGD Clients when connecting to SGD servers. See Section 1.2, “DNS Names” more details.

Connections between the SGD servers in an array are made on TCP port 5427. By default, this connection is encrypted using secure intra-array communication. See Section 7.1.4, “Secure Intra-Array Communication”.

Each server in the array has a record of the peer DNS names of all the SGD servers in an array. A server only accepts connections on TCP port 5427 if the following occurs:

  • The connection is from an array member, according to its own records.

  • A shared secret, known only to array members, is used to authenticate connections between array members. Secret Key Identification (SKID) authentication is used. SKID authentication does not encrypt the data.

Most connections are made from the primary server to a secondary server. These connections replicate data to keep the array synchronized. However, array members must be able to communicate directly with other array members.

7.1.4. Secure Intra-Array Communication

In a standard installation, the data transmitted between the SGD servers in an array is encrypted. The connections between array members are secured using SSL. Using SSL for these connections ensures the integrity of the data as follows:

  • Communication only takes place between SGD servers that have authenticated to each other

  • Data is encrypted before transmission

  • Data can be checked to ensure that it has not changed during transmission

Using SSL in this way is known as secure intra-array communication.

In a standard installation, secure intra-array communication is enabled automatically for an SGD server.

Secure intra-array communication can only be enabled on an SGD server that is not joined with other SGD servers in an array. When secure intra-array communication is enabled for an array, an SGD server can only join the array if it also has secure intra-array communication enabled. See Section 7.1.7.1, “How to Enable Secure Intra-Array Communication” for details.

Using secure intra-array communication means that each SGD server in the array has to have a valid SSL certificate that has been signed by a trusted certificate authority (CA).

As the SSL certificates used for secure intra-array communication are used only internally by SGD, the primary SGD server in the array acts as the CA. The primary SGD server has a self-signed CA certificate and a private key. All secondary SGD servers in the array have a copy of the primary SGD server's CA certificate in a trusted certificate store, the truststore.

All SGD servers in the array, including the primary, have an SSL certificate and a private key. The SSL certificate is signed with the primary SGD server's CA certificate and contains a common name (CN) which is the peer DNS name of the SGD server. As the SSL certificates are created using a self-signed CA certificate, they cannot be used to secure any other SGD-related connection. These certificates are referred to as peer SSL certificates to distinguish them from other types of SSL certificates.

When one SGD server in the array connects to another, including when using an administration tool, the SGD server being connected to presents its peer SSL certificate as part of the SSL negotiation. The connecting server evaluates the peer SSL certificate and checks the following:

  • The CN of the certificate matches the peer DNS name of the connecting server

  • The expiry date of the certificate

  • The issuer of the certificate, which must be the CA certificate of the primary

If the peer SSL certificate is valid, a secure connection is established.

When secure intra-array communication is enabled, SGD automatically generates and distributes the CA and peer SSL certificates to the members of the array. Whenever there is a change in the array structure, SGD automatically updates the CA and peer SSL certificates. The following table summarizes what happens.

Array Change

Action

Server joins the array

  1. The primary SGD server CA certificate is installed on the new secondary server.

  2. The new secondary SGD server obtains a new peer SSL certificate signed with the primary SGD server CA certificate.

Server leaves the array

  1. The detached SGD server becomes the primary SGD server in an array containing one server.

  2. The detached SGD server creates a new CA certificate for itself.

  3. The detached SGD server creates a new peer SSL certificate for itself.

New primary server appointed

  1. The new primary SGD server generates a new CA certificate.

  2. The new primary CA certificate is installed on all secondary SGD servers.

  3. All SGD servers obtain a new peer SSL certificate signed with the new primary SGD server CA certificate.

SGD Administrators can use the tarantella security peerca --show command to view certificates in the truststore. The truststore contains the primary SGD server's CA certificate.

By default, SGD uses the TLS_RSA_WITH_AES_128_CBC_SHA cipher suite for secure intra-array communication. For details on how to change the cipher suite, see Section 7.1.7.6, “How to Change the Cipher Suite for Secure Intra-Array Communication”.

7.1.5. Managing Arrays and SGD Servers

You add and remove SGD servers from an array using the Secure Global Desktop Servers tab in the Administration Console or by using the tarantella array command. It is best to perform all array operations on the primary SGD server in the array. For details on configuring arrays, see the following:

In the Administration Console, the attributes on the Global Settings tabs are the settings that apply to the array as a whole, for example how users authenticate to SGD. Appendix A, Global Settings and Caches has details of all the global settings. If you click the name of an SGD server on the Secure Global Desktop Servers tab, you display the attributes that apply only to that SGD server, for example the server's external DNS names. Appendix B, Secure Global Desktop Server Settings has details of all the server-specific settings.

On the command line, you list and edit global settings or server-specific settings using the tarantella config command.

7.1.6. Array Resilience

Array resilience is a feature where the loss of the primary server in an SGD array is handled automatically. The primary server can become unavailable either because of network problems, or because the SGD server goes down.

This section includes the following topics:

7.1.6.1. How Array Resilience Works

Array resilience consists of the following stages:

  • Failover stage. When the primary server becomes unavailable, the array is reconfigured automatically into one or more arrays, each with their own primary server. See Section 7.1.6.1.1, “Failover Stage”.

  • Recovery stage. When the original primary server becomes available, the original array formation is recreated. This can be done automatically or manually. See Section 7.1.6.1.2, “Recovery Stage”.

7.1.6.1.1. Failover Stage

If array failover is enabled for an array, the failover stage starts automatically after the primary server has been unavailable for a user-configurable time period, called the grace period. The default grace period is 10 minutes.

The grace period is calculated from the values for the Monitor Interval (--array-monitortime) and Monitor Attempts (--array-maxmonitors) attributes, as follows:

Grace period = Monitor Interval x Monitor Attempts

Using the default settings:

Grace period = 60 seconds x 10 = 600 seconds, or 10 minutes

The failover stage uses the backup primaries list to select a secondary server to promote to be the new primary server in the array. The backup primaries list is a list of secondary servers for the array, with the priority decreasing from top to bottom. If available, the highest priority secondary server in this list is contacted and promoted to be the new primary server for the array.

The new primary server must be contacted within a time period called the find new primary timeout. If the new primary cannot be contacted during this timeout period, the next server in the backup primaries list is contacted.

The find new primary timeout period is calculated from the values for the Find Primary Interval (--array-resubmitfindprimarywait) and Find Primary Attempts (--array-resubmitfindprimarymax) attributes, as follows:

Find new primary timeout = Find Primary Interval x Find Primary Attempts

Using the default settings:

Find new primary timeout = 60 seconds x 3 = 180 seconds, or 3 minutes

Only SGD servers that are in the backup primaries list can be selected for promotion to be the new primary server.

When you build an SGD array, a backup primaries list is created for you automatically. If you add a secondary server to the array, an entry is added at the end of the list. If you remove a secondary server from the array, the entry for the server is removed from the list.

The backup primaries list is stored on each SGD server in the array. Any changes made to the list are copied to each SGD server in the array.

If the backup primaries list is empty, all SGD servers in the array become standalone servers after array failover.

When the failover stage has completed, the array is said to be in a repaired state.

The tarantella status command indicates if an array is in a repaired state. You can use the --originalstate option of this command to list the members of the array before it was repaired. See Section 7.7.1.1, “Showing Status Information For an SGD Array” for more details about using tarantella status to display status information for an array.

Caution

During the failover stage, do not change the array formation, or any array resilience settings. The original array formation might not be recreated successfully by the recovery stage if you do so.

7.1.6.1.2. Recovery Stage

If the original primary server becomes available when the array is in a repaired state, the recovery stage starts automatically.

By default, the recovery stage restores the original primary server as the primary server for the array. You can use the Action When Failover Ends (--array-primaryreturnaction) attribute to determine how the array is reconfigured during the recovery stage.

In some situations after using array resilience you might have to rebuild an array manually. This is called manual recovery.

For example, if the recovery stage has failed to recreate the original array formation automatically you can rebuild the original array manually, starting from single, standalone SGD servers. You use the tarantella array clean command to do this.

Caution

After you run tarantella array clean on the primary server in an SGD array, the secondary servers will not be able to contact the primary server.

If an array splits into more than two arrays during the failover stage and the Action When Failover Ends (--array-primaryreturnaction) attribute is configured as Restore original primary (accept), the original array formation is recreated automatically.

If the Action When Failover Ends attribute is configured as Restore array with a new primary (acceptsecondary), the original array formation cannot be recreated automatically. Manual recovery must be used.

7.1.6.2. Examples of How Array Resilience Works

There are many possible array resilience scenarios, where the primary server becomes unavailable for one or more servers in the SGD array. This section includes examples of how array resilience works in the following scenarios:

In the following examples, the domain example.com has a four-node array of SGD servers:

  • Primary server – boston

  • Secondary servers – newyork, detroit, seattle

Figure 7.1, “Original Network Configuration” shows the original network configuration before using array resilience.

Figure 7.1. Original Network Configuration

Diagram Showing Original Network Configuration Before Array Resilience

7.1.6.2.1. Primary Server Goes Down

A typical sequence of events for array resilience when the primary server in an SGD array goes down is as follows:

  1. The primary server, boston, goes down and is unavailable to any of the secondary servers in the array.

  2. If boston is still unavailable after the grace period has elapsed, the failover stage begins.

  3. The first available secondary server from the array's backup primaries list is promoted to be the new primary server for the array.

  4. Each of the existing secondary servers are reconfigured automatically to work with the new primary server. The array becomes a three-node array. The array is now in a repaired state.

    Figure 7.2, “Network Configuration After the Failover Stage When the Primary Server Goes Down” shows the network configuration after the failover stage.

  5. When boston becomes available again, the recovery stage begins. By default, boston automatically rejoins the array as the primary server.

  6. The other servers in the array are reconfigured automatically to work with the new primary server, boston.

Figure 7.2. Network Configuration After the Failover Stage When the Primary Server Goes Down

Diagram Showing Network Configuration After the Failover Stage When the Primary Server Goes Down

7.1.6.2.2. Array Splits into Two Arrays

A typical sequence of events for array resilience when an SGD array splits into two arrays is as follows:

  1. Because of network problems, the primary server, boston, can only contact the newyork secondary server. The remaining secondary servers in the array, seattle and detroit, cannot be contacted.

  2. After the grace period has elapsed, if the primary server still cannot contact seattle and detroit, the failover stage begins.

  3. The original array remains as a four-node array, but with the seattle and detroit servers reported as uncontactable in the array. The same primary server, boston, is used but the original array now has only a single contactable secondary server, newyork.

  4. The secondary servers seattle and detroit can contact each other. These servers join to form a new two-node array. The first available secondary server from the backup primaries list is promoted to be the primary server for this array.

    Figure 7.3, “Network Configuration After the Failover Stage When the Array Splits into Two Arrays” shows the network configuration after the failover stage.

  5. Network problems are fixed. The recovery stage begins. By default, the two arrays join together. The original array formation is recreated automatically, with boston as the primary server.

Figure 7.3. Network Configuration After the Failover Stage When the Array Splits into Two Arrays

Diagram Showing Network Configuration After the Failover Stage When the Array Splits into Two Arrays

7.1.7. Configuring Arrays

Configuring an array involves the following steps:

  1. Add SGD servers to the array.

    Before building an array, you might want to enable secure intra-array communication. See Section 7.1.7.1, “How to Enable Secure Intra-Array Communication”. In a standard installation, secure intra-array communication is enabled for an SGD server.

    How you add servers to an array depends on whether or not you are using secure intra-array communication, see the following:

  2. Change the structure of an array.

    See the following:

  3. (Optional) Change the cipher suite used for secure intra-array communication.

    See Section 7.1.7.6, “How to Change the Cipher Suite for Secure Intra-Array Communication”.

7.1.7.1. How to Enable Secure Intra-Array Communication

In a standard installation, secure intra-array communication is enabled automatically for an SGD server.

Secure intra-array communication can only be enabled on an SGD server that is not joined with other SGD servers in an array.

Ensure that no users are logged in to the SGD server and that there are no running application sessions, including suspended application sessions.

You can only enable secure intra-array communication from the command line.

  1. Log in as superuser (root) on the SGD server.

  2. Stop the SGD server.

  3. Enable secure intra-array communication.

    Use the following command:

    # tarantella config edit \
    --tarantella-config-security-peerssl-enabled 1
  4. Start the SGD server.

7.1.7.2. How to Add a Server to an Array (Secure Intra-Array Communication Enabled)

When secure intra-array communication is enabled for an array, an SGD server can only join the array if it also has secure intra-array communication enabled. In a standard installation, secure intra-array communication is enabled automatically for an SGD server.

The clock on the server joining the array must be in synchronization with the clocks on the other servers in the array. If the time difference is more than one minute, the array join operation fails.

  1. Log in to the SGD server that you want to add to the array.

  2. Display the fingerprint of the SGD server's CA certificate.

    Use the following command:

    $ tarantella security peerca --show
  3. Make a note of the fingerprint of the SGD server's CA certificate.

  4. Log in to the primary SGD server in the array.

  5. Join the SGD server to the array as a secondary server.

    Use the following command to add the SGD server.

    $ tarantella array join --secondary serv
    

    Enter the peer DNS name for serv. You must use a fully-qualified DNS name, such as boston.example.com

    You are prompted to trust the secondary SGD server's CA certificate, and the fingerprint of the certificate is displayed.

  6. Check that the fingerprint is correct and complete the array join.

    Check that the certificate fingerprint matches the fingerprint displayed in Step 2. This is important as it verifies that the primary SGD server is communicating with the genuine secondary SGD server.

    If the fingerprints match, complete the array join by accepting the secondary SGD server's CA certificate.

  7. Check the status of the array.

    Use the tarantella status command to check the status of the array.

    Note

    After making a change to the structure of an array, wait until SGD has copied the changes to all the SGD servers in the array before making any further changes. Run the tarantella status command on the primary SGD server to check the status of the array.

7.1.7.3. How to Add a Server to an Array (Secure Intra-Array Communication Disabled)

The server joining the array must be a standalone server. In other words, the server must be in an array on its own.

Ensure that the clock on the server joining the array is in synchronization with the clocks on the other servers in the array. If the time difference is more than one minute, the add server operation fails.

  1. Log in to the Administration Console on the primary SGD server.

  2. Go to the Secure Global Desktop Servers tab.

  3. In the Secure Global Desktop Server List, click the Add button.

    The Add a Secure Global Desktop Server screen is displayed.

    Tip

    You can also use the tarantella array join command to add an SGD server to an array.

  4. Enter the peer DNS name of an SGD server in the DNS Name field.

    You must use a fully-qualified DNS name, such as boston.example.com .

  5. Enter the user name and password of an SGD Administrator in the User Name and Password fields.

  6. Click Add.

    The Secure Global Desktop Servers tab is displayed.

    The Secure Global Desktop Servers tab shows messages advising you to wait for the server change and synchronization processes to complete.

    Note

    After making a change to the structure of an array, wait until SGD has copied the changes to all the SGD servers in the array before making any further changes. Run the tarantella status command on the primary SGD server to check the status of the array.

    If the server you add has been load balancing application servers using Advanced Load Management, you must do a warm restart, tarantella restart sgd --warm, of the new server after it has joined the array. See also Section 7.2.6, “How Advanced Load Management Works”.

7.1.7.4. How to Change the Primary Server in an Array

  1. Log in to the Administration Console on the primary SGD server.

  2. Go to the Secure Global Desktop Servers tab.

  3. In the Secure Global Desktop Server List, click the Make Primary button.

    Tip

    You can also use the tarantella array make_primary command to change the primary server in an array.

  4. When prompted, click OK.

    The Secure Global Desktop Servers tab shows messages advising you to wait for the server change and synchronization processes to complete.

    The previous primary server becomes a secondary server.

    Note

    After making a change to the structure of an array, wait until SGD has copied the changes to all the SGD servers in the array before making any further changes. Run the tarantella status command on the primary SGD server to check the status of the array.

7.1.7.5. How to Remove a Server From an Array

To remove the primary server from an array, you must first make another server the primary server and then remove the old primary server.

  1. Log in to the Administration Console on the primary SGD server.

  2. Go to the Secure Global Desktop Servers tab.

  3. In the Secure Global Desktop Server List, click the Remove button.

    Tip

    You can also use the tarantella array detach command to remove an SGD server from an array.

  4. When prompted, click OK.

    The Secure Global Desktop Servers tab shows messages advising you to wait for the server change and synchronization processes to complete.

    Note

    After making a change to the structure of an array, wait until SGD has copied the changes to all the SGD servers in the array before making any further changes. Run the tarantella status command on the primary SGD server to check the status of the array.

7.1.7.6. How to Change the Cipher Suite for Secure Intra-Array Communication

Ensure that no users are logged in to the SGD servers in the array and that there are no running application sessions, including suspended application sessions.

  1. Stop all the SGD servers in the array.

  2. Log in as superuser (root) on the primary SGD server in the array.

  3. Specify the cipher suite.

    Use the following command:

    # tarantella config edit \
    --tarantella-config-security-peerssl-ciphers cipher-suite
    

    where cipher-suite is the JSSE name of a cipher suite.

    The following table lists the available cipher suites.

    JSSE Name

    Cipher Suite

    TLS_RSA_WITH_AES_256_CBC_SHA

    RSA_WITH_AES_256_CBC_SHA

    TLS_RSA_WITH_AES_128_CBC_SHA

    RSA_WITH_AES_128_CBC_SHA

    SSL_RSA_WITH_3DES_EDE_CBC_SHA

    RSA_WITH_3DES_EDE_CBC_SHA

    SSL_RSA_WITH_RC4_128_SHA

    RSA_WITH_RC4_128_SHA

    SSL_RSA_WITH_RC4_128_MD5

    RSA_WITH_RC4_128_MD5

    SSL_RSA_WITH_DES_CBC_SHA

    RSA_WITH_DES_CBC_SHA

    The default setting is TLS_RSA_WITH_AES_128_CBC_SHA.

  4. Start all the SGD servers in the array.

7.1.8. Configuring Array Resilience

Configuring array resilience involves the following steps:

  1. Enable array failover for the array.

    Array failover is disabled by default for an SGD array.

  2. (Optional) Configure the array failover grace period.

    Array failover starts automatically after the grace period has elapsed.

  3. (Optional) Configure the backup primaries list.

    The backup primaries list determines which secondary server is promoted to be the new primary server.

    For more details, see the following:

  4. (Optional) Configure the find new primary timeout period.

    If the new primary cannot be contacted during this timeout period, the next server in the backup primaries list is contacted.

  5. (Optional) Configure the recovery stage.

    By default, the recovery stage recreates the original array formation automatically when the original primary server becomes available.

    You can use manual recovery to recreate the original array formation manually.

7.1.8.1. How to Enable Array Failover for an Array

  1. In the Administration Console, go to the Global Settings → Resilience tab.

  2. Enable array failover for the SGD array.

    Select the Array Failover check box.

    Tip

    You can also use the tarantella config edit command to enable the Array Failover (--array-failoverenabled) attribute.

7.1.8.2. How to Configure the Array Failover Grace Period

  1. In the Administration Console, go to the Global Settings → Resilience tab.

  2. Configure the grace period.

    Type values for the Monitor Interval and Monitor Attempts attributes.

    For example, to change the grace period to 120 seconds, or 2 minutes, set the Monitor Interval attribute to 60 and the Monitor Attempts attribute to 2.

    The default grace period is 10 minutes.

    Tip

    You can also use the tarantella config edit command to configure the Monitor Interval (--array-monitortime) and Monitor Attempts (--array-maxmonitors) attributes.

7.1.8.3. How to Show the Backup Primaries List for an Array

  1. In the Administration Console, go to the Global Settings → Resilience tab.

  2. View the entries in the backup primaries list.

    The Backup Primaries table shows the backup primaries list for the array.

    Tip

    You can also use the tarantella array list_backup_primaries command to show the backup primaries list for an array.

7.1.8.4. How to Add an Entry to the Backup Primaries List

  1. In the Administration Console, go to the Global Settings → Resilience tab.

  2. Add an entry to the backup primaries list.

    1. Click the New button in the Backup Primaries table.

      The Available Secondaries table is displayed, showing the available secondary servers that are not in the backup primaries list.

    2. Select a server in the Available Secondaries table and click Add.

      The Backup Primaries table is updated automatically.

    Tip

    You can also use the tarantella array add_backup_primary command to add an entry to the backup primaries list.

7.1.8.5. How to Change the Position of an Entry in the Backup Primaries List

  1. In the Administration Console, go to the Global Settings → Resilience tab.

  2. Change the position of an entry in the backup primaries list.

    Select the server in the Backup Primaries table and click Move Up or Move Down.

    Tip

    You can also use the tarantella array edit_backup_primary command to change the position of an entry in the backup primaries list.

7.1.8.6. How to Delete an Entry From the Backup Primaries List

  1. In the Administration Console, go to the Global Settings → Resilience tab.

  2. Delete an entry in the backup primaries list.

    Select the server in the Backup Primaries table and click Delete.

    Tip

    You can also use the tarantella array remove_backup_primary command to delete an entry from the backup primaries list.

7.1.8.7. How to Configure the Find New Primary Timeout

  1. In the Administration Console, go to the Global Settings → Resilience tab.

  2. Configure the find new primary timeout period.

    Type values for the Find Primary Interval and Find Primary Attempts attributes.

    For example, to change the find new primary timeout to 60 seconds, or 1 minute, set the Find Primary Interval attribute to 60 and the Find Primary Attempts attribute to 1.

    The default find new primary timeout period is 3 minutes.

    Tip

    You can also use the tarantella config edit command to configure the Find Primary Interval (--array-resubmitfindprimarywait) and Find Primary Attempts (--array-resubmitfindprimarymax) attributes.

7.1.8.8. How to Configure the Action When Failover Ends

  1. In the Administration Console, go to the Global Settings → Resilience tab.

  2. Configure how the array is reconfigured when the original primary server becomes available.

    Select the required option for the Action When Failover Ends attribute.

    • To accept the original primary server back into the array as the primary server, select the Restore original primary option.

      The original primary server, and any attached secondary servers, rejoin the array. The original primary server is restored as the primary server for the array. The current primary server becomes a secondary server. This is the default setting.

    • To exclude the original primary server from the array, select the Do not restore original array option.

      The original primary server, and any attached secondary servers, do not rejoin the array. The original primary server, and any attached secondary servers, stay in the state they were in after the failover stage.

    • To accept the original primary server back into the array as a secondary server, select the Restore array with a new primary option.

      The original primary server, and any attached secondary servers, rejoin the array as secondary servers.

    Tip

    You can also use the tarantella config edit command to configure the Action When Failover Ends (--array-primaryreturnaction) attribute.

7.1.8.9. How to Rebuild an Array Manually

  1. Remove all array state information.

    Run the following command on each SGD server in the array.

    $ tarantella array clean

    By default, the tarantella array clean command deletes all array information and configures the SGD server as a single, standalone server. Use the --contactmembers option of this command if you want the SGD server to remain in an array with other SGD servers that are contactable and that report the same array membership.

  2. Rebuild the array manually.

    Use the tarantella array command. See Section 7.1.5, “Managing Arrays and SGD Servers” for details of how to do this.