Chapter 3 Tips for Writing and Testing HA Data Services

This chapter provides tips for writing and testing new highly available data services.

Overview

This chapter describes how to:

• Modify your data service to better suit your particular task

• Test your data service to ensure that it will operate correctly in the HA environment

• Coordinate dependencies between data services

Deciding Which Methods to Use

This section provides some tips about when to use the start_net, stop_net, and abort_net methods versus using the start, stop, and abort methods.

Generally, it is easier to start, stop, or abort the data service using start_net, stop_net, or abort_net, because the logical network addresses are configured to be up at the point where these methods are called.

To start, stop, or abort a data service, you often will have to invoke the data service's administrative utilities or libraries. Sometimes, the data service has administrative utilities or libraries that use a client-server networking interface to perform the administration. That is, an administrative utility makes a call to the server daemon, so the logical network address might need to be up to use the administrative utility or library.

Consider whether your client software will respond differently depending on whether the network interface or the data service comes online first after a reboot, takeover, or switchover. Use the methods that will ensure adequate retries occur before giving up. For example, if your client implementation does minimal retries when it determines that the data service port is not available, ensure that the data service starts before the network interface is configured. In this case, use the start method rather than the start_net method.

If you use the stop or abort method, the data service is still up at the point where the logical network address is configured to be down; it is only after the logical network address is configured down that the stop and abort methods are invoked.

This creates the invariant that the data service's TCP or UDP service port, or its RPC program number, always appears to be available to clients on the network--except when the logical host network address also is not responding. This invariant is important only if the client code behaves in a significantly different way when it finds that the TCP or UDP service port, or RPC program number, is not responding, but that the logical host's network address is responding. For example, a client might decide to abandon its retry path early in this scenario. This means that the client code is going down a different code path when it receives an explicit error packet back from the server host saying "ICMP port unreachable" or "Program not registered."

You need in-depth knowledge of the client and the data service's client-server networking protocol to know whether a client implementation depends on this invariant.

Using Keep-Alives

If the client-server communication uses a TCP stream, then both the client and the server should enable the TCP keep-alive mechanism. This is applicable even in the non-HA single server case.

Other connection-oriented protocols might also have keep-alive mechanisms.

On the server side, using TCP keep-alives protects the server from wasting resources for a down (or network partitioned) client. If those resources are not cleaned up (in a server that stays up long enough), eventually the wasted resources will grow without bound as clients crash and reboot.

On the client side, using TCP keep-alives enables the client to be notified when a logical host has failed over or switched over from one physical host to another. That transfer of the logical host breaks the TCP connection. However, unless the client has enabled the keep-alive, it would not necessarily learn of the connection break if the connection happens to be quiescent at the time.

For example, consider the case in which the client is waiting for a response from the server to a long-running request. In this scenario, the client's request message has already arrived at the server and has been acknowledged at the TCP layer, so the client's TCP module has no need to keep retransmitting it. The client application is now blocked, waiting for a response to the request.

Where possible, in addition to using the TCP keep-alive mechanism, the client application also should perform its own periodic keep-alive at its level, because the TCP keep-alive mechanism is not perfect in all possible boundary cases. Using an application-level keep-alive typically requires that the client-server protocol supports a null operation or at least an efficient read-only operation such as a status operation.

Testing HA Data Services

You will want to test your data service implementation thoroughly before putting it into a production environment. This section provides suggestions about how to test your implementation in the HA environment. The test cases are suggestions and are not exhaustive. For testing, you need to have access to a test-bed Sun Cluster configuration, so that your work will not impact production machines.

Test that your HA data service behaves properly in all cases where a logical host is moved between physical hosts. These include system crashes and the use of the haswitch(1M) and scadm(1M) stopnode commands. Test that client machines continue to get service after these events.

Test the idempotency of the methods. An important way to do this is to configure logical hosts with manual mode ON and repeatedly abort and rejoin one physical host, without ever doing an haswitch(1M) of a logical host to it. Let the rejoining host complete cluster reconfiguration before aborting it again. Note that when a rejoining host rejoins the cluster, cluster reconfiguration runs, but no logical host is moved between physical hosts during that reconfiguration.

Another way to test idempotency is to replace each method temporarily with a short shell script that calls the original method twice.

To test that your data service properly implements the abort and abort_net methods, make one physical host look very sick to Sun Cluster, but without crashing the host outright, so that Sun Cluster will take it out of the system on the "last wishes" path. First, do an haswitch(1M) of all logical hosts to that physical host. Then make that host appear to be sick by unplugging all the public network connections to that host. Sun Cluster network fault monitoring will notice the problem and take the physical host out of the cluster, using the aborting "last wishes" path.

Coordinating Dependencies Between Data Services

Sometimes, one client-server data service makes requests upon another client-server data service while fulfilling a request for a client. Informally, a data service A depends on a data service B if, for A to provide its service, B must be providing its service.

Sun Cluster enables having dependent data services by providing the -d switch to the hareg(1M) program. The dependencies affect the order in which Sun Cluster starts and stops data services. See the hareg(1M) man page for details.

Determine whether there are any data service dependencies and whether to supply the appropriate -d switches to hareg(1M). Sun Cluster does not check the completeness of the supplied -d switches.

Decide whether to use the -d switches or to omit them and poll for the availability of the other data service(s) in your HA data service's own code. In some cases, polling is required anyway, because the other data service's start method might be asynchronous--it might start the data service but not wait for the data service to actually be available to clients before returning from the start or start_net method. Database services typically exhibit this behavior because database recovery time is often lengthy.

Dependent Data Service Using Another Back-End Data Service

Some data services store no data directly themselves, but instead depend upon another back-end data service to store all their data. Such a data service translates all read and update requests into calls on the back-end data service. For example, consider a hypothetical client-server appointment calendar service that keeps all of its data in an SQL database such as Oracle. The appointment calendar service has its own client-server network protocol. For example, it might have defined its protocol using an RPC specification language, such as ONC^(TM) RPC.

In the Sun Cluster environment, you can use Sun Cluster HA for Oracle to make the back-end Oracle database highly available. Then, you can write simple methods for starting and stopping the appointment calendar daemon. You can register the appointment calendar data service with Sun Cluster as one that depends upon another Sun Cluster data service, Sun Cluster HA for Oracle. Specify this dependency using the -d option to hareg(1M).

The start method for Oracle might initiate only database recovery and might not wait for the recovery to complete. Therefore, our calendar data service daemon, once it has been started, must poll waiting for the Oracle database to become available.