Chapter 19 Managing Clusters (Enterprise Edition)

The use of Sun Java™ System Application Server clusters provides high availability for your J2EE applications. This chapter explains how to configure and manage a group of load balanced clusters. It also provides important information about maintaining and operating clusters to fully utilize the failover capabilities of Sun Java™ System Application Server Enterprise Edition.

This chapter mainly discusses HTTP clusters. For information on working with RMI/IIOP clusters, see Configuring RMI/IIOP Failover.

About Application Server Clusters

Administering Application Server Clusters

Online Reconfiguration of HTTP Clusters

Using Multiple Load Balancers

About Application Server Clusters

A cluster is a group of Application Server instances that work together as one logical entity. A cluster provides a runtime environment for one or more J2EE applications.

High availability by allowing for failover protection of Application Server instances in a cluster. If one Application Server instance goes down, other Application Server instances take over the requests that the unavailable server was serving.

Scalability by allowing for the addition of Application Server instances to a cluster, thus increasing the capacity of the system. The load balancer of Sun Java System Application Server distributes requests to the available Application Server instances within the cluster.



Note	You can use multiple clusters to perform online upgrades without loss of service on the HTTP path. This is explained in Using Multiple Clusters for Online Upgrades Without Loss of Service.

A few important facts about Sun Java System Application Server clusters are listed here:

Sun Java System Application Server clusters support failover of HTTP and RMI/IIOP sessions. Failover of certain J2EE object references that are stored within HTTP sessions is also supported.

The Application Server instances in a Sun Java System Application Server cluster must be homogenous. That is:

Session persistence configuration must be identical for a given application across all instances in a cluster. Do not deploy an application with different settings for different Application Server instances in a cluster.

For these applications to take part in failover, they must be distributable, that is, enabled for session persistence.

For more information, see Chapter 20, "Configuring Session Persistence (Enterprise Edition)."

All the instances should point to the same session store. For more information, see Managing the Session Store for Clusters.

The value of the cluster-id property should be the same for all Application Server instances. For more information, see Managing the Session Store for Clusters.

Each Application Server instance in a cluster can host more than one application.

The applications can serve HTTP and RMI/IIOP clients.

Although you can add the same Application Server instance to multiple clusters, it is not recommended as management of such clusters will become very complex. For example, when an instance, which is part of more than one cluster, is removed from just one of the clusters.

It is not mandatory that an Application Server instance be part of a cluster. However, an instance that is not part of a cluster cannot take advantage of high availability through transfer of session state from one instance to other instances.

Application Server instances within a cluster can be hosted on different machines or on the same machine. That is, you can group Application Server instances across different machines into a single cluster.

A particular load balancer can forward requests to Application Server instances on multiple clusters. You can use this ability of the HTTP load balancer plug-in to perform an online upgrade without loss of service.

For more information, see Using Multiple Clusters for Online Upgrades Without Loss of Service. Dynamic reconfiguration of the RMI/IIOP load balancer is not supported.

A single cluster can receive requests from multiple load balancers. If a cluster is served by more than one load balancer, you must configure the cluster in exactly the same way on each load balancer. This means that the entries corresponding to the cluster in each loadbalancer.xml (for HTTP sessions) and server.xml (for RMI/IIOP sessions) must be identical.

Each session is tied to a particular cluster. Therefore, though an application can be deployed on multiple clusters, sessions will only be failed over to Application Server instances within the same cluster.

Defining a Cluster

To define a cluster, specify a name for the cluster in the loadbalancer.xml file. You also set up other parameters for the cluster.

Use the name attribute of the cluster element to define the name of the cluster (cluster is a subelement of the element loadbalancer in the loadbalancer.xml file).

In the loadbalancer.xml file, create a new cluster sub-element for the loadbalancer element.

Give a unique name to the name attribute of cluster.



Note	You can create more than one cluster in the loadbalancer.xml file.

Specify a URL for the health checker for the cluster.

The health checker URL is suffixed to the URL of the listener for the Application Server instance to formulate the URL at which health check requests are sent to the instance. For example, if the listener URL of an instance is http://www.example.com:80 and the health checker URL is /fortune, the health check requests for that instance are sent to http://www.example.com:80/fortune

The health check requests are HTTP requests. The response from the instance should be in the range of 100 to 500.

You specify the healthcheck URL by using the url attribute of the health-checker element in the loadbalancer.xml file (the health-checker element is a subelement of the cluster element.)

Specify the time interval in seconds after which this health check is performed by using the interval-in-seconds attribute of the health-checker element. This is a numeric value. The default value is 30 seconds.

Specify the timeout value in seconds for the health check request by using the timeout-in-seconds attribute of the health-checker element. This is the time period within which the response should be returned by the Application Server instance to the health check request sent by the load balancer failing which the Application Server instance is considered unavailable. The default value is 10 seconds.

Code Example 19-1 Defining a Cluster

Requirements for Adding an Application Server Instance to a Cluster

The following conditions must be true before you add an Application Server instance to a cluster:

Before you can add any Application Server instance to a cluster, you must have created the instance through the Administration interface or the command-line interface. For information on creating an Application Server instance, see Adding an Application Server Instance.

Any new Application Server instance that you add to a cluster must have the same version and the same patch level of Sun Java System Administration Server as all the existing Application Server instances in the cluster.

You must ensure the availability of all required peripheral products and libraries (for example JDBC driver classes) in the correct locations and provide the needed access rights.

The Application Server instances in a Sun Java System Application Server cluster must be homogenous. That is:

Session persistence configuration must be identical for a given application across all instances of a cluster. Do not deploy an application with one set of settings to one segment of the instances in a cluster and another set of settings to another segment of instances in the cluster.

All the instances should point to the same session store. For more information, see Managing the Session Store for Clusters.

The value of the cluster-id property should be the same for all Application Server instances. For more information, see Managing the Session Store for Clusters.

The system time of all the servers on which Application Server instances in a cluster are deployed must be synchronized.

Adding Application Server Instances to a Cluster

For adding Application Server instances to a cluster, use the instance sub element of the cluster element in the loadbalancer.xml file.

In the loadbalancer.xml file, create a new instance sub-element for the cluster element.

Provide a name to the Application Server instance using the name attribute of instance. This name need not be the same as the name of the Application Server instance that you specified when creating the instance. This name must be unique across all the clusters referenced in a single loadbalancer.xml file. That is, no two Application Server instances within a cluster can have the same name.

Specify whether the Application Server instance is enabled or disabled by specifying true or false for the enabled attribute of instance. The default value is true, that is the Application Server instance is enabled.

Specify the quiescing period in minutes by using the disable-timeout-in-minutes attribute of instance. This is a numeric value. The default value is 31, which is one minute more than the default session idle timeout of 30 minutes.

If you need to disable an Application Server instance, this is the time interval after which the load balancer stops sending assigned requests to that instance. For more information on quiescing and quiescing period, see Disabling and Quiescing an Application Server Instance in a Cluster.

Specify the listeners for the Application Server instance by providing the URL of the HTTP listener for the Application Server instance.

You provide this information using the listeners attribute of the instance element in the loadbalancer.xml file. You can assign one or more listeners to the Application Server instance. To specify multiple listeners for an Application Server instance, separate the URLs of the listeners with a space. For example, https://www.example.com:41 https://www.example.com:42.



Note	After you add an HTTP listener to an Application Server instance, you must restart the SNMP monitoring subagent. If you do not do this, it is possible that information about the new listener will not be available through the SNMP monitoring subagent.

Here is an example where an Application Server instance named myinstance1 has been added to the cluster named mycluster1. Two listeners have been specified for the Application Server instance. The health checker for this cluster has also been configured.

Defining Multiple Clusters


Note	When you configure the session persistence settings, create a different session store for each cluster. For more information on creating a session store for a cluster, see Managing the Session Store for Clusters.

Administering Application Server Clusters

Starting a Cluster

Deploying a Web Application

Undeploying a Web Application

Redeploying an Existing Web Application on a Cluster

Stopping an Application Server Instance in a Cluster

Removing an Application Server Instance from a Cluster

Stopping a Cluster

Removing a Cluster

Starting a Cluster


Note	The procedures mentioned in this chapter contain UNIX specific examples. The same commands and examples, with appropriate modifications, are applicable for Microsoft Windows platforms.

Start all Application Server instances in the cluster.

To start all instances in the cluster, use the cladmin command as follows:

./cladmin [--instancefile instance_file_location] [--passwordfile password_file_location]start-instance

where:

instance_file_location is the location of the clinstance.conf file.

password_file_location is the location of the clpassword.conf file.

For more information on the cladmin command, see Appendix F, "Using the cladmin Command for Administration (Enterprise Edition)."

Enable all the Application Server instances in the cluster.

Set the value of the enabled attribute of the instance element to true in the loadbalancer.xml file.

Deploying a Web Application

Deploy the web application on all Application Server instances in the cluster.

To deploy the web application to all instances in the cluster, use the cladmin command as follows:

./cladmin deploy [--instancefile instance_file_location] [--passwordfile password_file_location][--secure | -s] [--virtualservers virtual_servers] [--type application|ejb|web|connector] [--contextroot contextroot] [--force=true|false] [--precompilejsp=true|false] [--name component_name] [--upload=true|false] [--retrieve local_dirpath] [--instance instance_name] filepath

where:

instance_file_location is the location of the clinstance.conf file.

password_file_location is the location of the clpassword.conf file.

For more information on the cladmin command, see Appendix F, "Using the cladmin Command for Administration (Enterprise Edition)." For more information on deploying an application to an Application Server instance, see Tools for Deployment.

In the loadbalancer.xml file, create a new web-module sub-element (within the cluster element) for the cluster to which you want to deploy the application.

Provide the context root URL for the web module that you want to deploy to the cluster by using the context-root attribute of the web-module element.



Note	One cluster cannot have more than one web-module element with the same context-root.

Specify whether the web application is enabled or disabled by specifying true or false for the enabled attribute of web-module. The default value is true, that is, the web application is enabled.

Typically, when you deploy applications to a cluster, you enable the application. Enabling an application in a cluster means that the load balancer will route requests to this application.

Specify the quiescing period in minutes by specifying a value for the disable-timeout-in-minutes attribute of web-module. This is a numeric value. The default value is 31, which is one minute more than the default session idle timeout of 30 minutes.

If you disable the web application, this is the time interval after which the load balancer stops sending assigned requests to the web application. For more information on quiescing, see Disabling and Quiescing a Web Application in a Cluster.

Here is an example in which a web application with the context root /fortune has been added to a cluster called cluster1.

Undeploying a Web Application

Disable the web application from the cluster by setting the value (in the loadbalancer.xml file) of the enabled attribute of the web-module element to false.

After the quiescing period is over, undeploy the web application from all the Application Server instances in the cluster.

To undeploy the web application from all instances in the cluster, use the cladmin command as follows:

./cladmin undeploy [--instancefile instance_file_location] [--passwordfile password_file_location][--secure | -s] [--type application|ejb|web|connector] [--instance instance_name] component_name

where:

instance_file_location is the location of the clinstance.conf file

password_file_location is the location of the clpassword.conf file

In the loadbalancer.xml file, remove the web-module sub-element (within the cluster element) that references the undeployed application.



Note	After an application is undeployed, the session state information for that application is not immediately removed from the HADB. This session state information is removed by the Sun Java System Application Server in the subsequent cycle when the timed-out sessions are removed.

Redeploying an Existing Web Application on a Cluster

There may be cases when you want to make changes to a deployed web application that require you to undeploy the web application and then again deploy it. For example, you may want to deploy an upgraded version of a web application. In such cases, follow these steps:

Disable the web application by setting the value (in the loadbalancer.xml file) of the enabled attribute of the corresponding web-module element to false.

After the quiescing period is over, redeploy the web application on the Application Server instances in the cluster.

Enable the web application in the cluster by setting the value (in the loadbalancer.xml file) of the enabled attribute of the corresponding web-module element to true.

Stopping an Application Server Instance in a Cluster

Disable the Application Server instance by setting the value (in the loadbalancer.xml file) of the enabled attribute of the instance element to false.

After the quiescing period is over, plus a sufficient time to allow the requests being served by the Application Server instance to complete, stop the Application Server instance.

Removing an Application Server Instance from a Cluster

Disable the Application Server instance by setting the value (in the loadbalancer.xml file) of the enabled attribute of the corresponding instance element to false.

After the quiescing period plus a sufficient time to allow the requests being served by the Application Server instance to complete, remove the entries corresponding to the Application Server instance in loadbalancer.xml file.

Once you remove an Application Server instance from a cluster, any assigned request for that instance is routed by the load balancer to other instances as part of the round-robin algorithm. If the Application Server instance to which the request is routed is part of the same cluster and the session has not been invalidated, the session is failed over. Otherwise, the Application Server instance to which the request is routed treats the request as an unassigned request.

Stopping a Cluster

Disable all Application Server instances from the cluster by setting the value (in the loadbalancer.xml file) of the enabled attribute of the corresponding instance element to false.

After the quiescing period is over, plus a sufficient time to allow the requests being served by the Application Server instance to complete, stop all Application Server instances that belong to the cluster.

To stop all instances in the cluster, use the cladmin command as follows:

./cladmin [--instancefile instance_file_location] [--passwordfile password_file_location] stop-instance

Removing a Cluster

Disable each web application deployed to the Application Server instances in the cluster by setting the value (in the loadbalancer.xml file) of the enabled attribute of the corresponding web-module element to false.

(Optional) After all the web applications have been quiesced, undeploy the web applications from all the individual Application Server instances in the cluster. To undeploy each application from all the instances in the cluster, use the cladmin command. For information on how to do this, see Using Multiple Load Balancers.

Remove all entries for the cluster (including entries for the Application Server instances and the web applications in the cluster) from the loadbalancer.xml file.

Sample Cluster Configuration in a loadbalancer.xml File

Here are sample entries for a cluster in the loadbalancer.xml file. Note that in a complete loadbalancer.xml file, there will be other entries, for example related to load balancer polling interval, HTTPS routing enabling, and so on. For more information about the other possible entries, see Known Issues in Load Balancing Requests.

Online Reconfiguration of HTTP Clusters

Suppose an Application Server instance in a cluster has received requests and is in the process of serving them. If you now want to stop this instance for any reason (for example to add a JDBC resource), you would want this instance to complete serving these requests before you stop it. Similarly, if you want to undeploy a web application from a cluster, you would want all instances in the cluster that are serving requests for these application to complete serving the requests before you undeploy the web application. Quiescing is a process that helps achieve this.

Disabling and Quiescing an Application Server Instance in a Cluster

Disabling and Quiescing a Web Application in a Cluster

Modifying the Quiescing Period While Quiescing is On

Disabling and Quiescing an Application Server Instance in a Cluster

Quiescing an Application Server instance is the process of shutting it down in a phased manner. First, the load balancer stops sending any unassigned requests to the instance and instead diverts these unassigned requests to other available instances in the cluster.

However, for a time interval called the quiescing period, the load balancer still sends to the Application Server instance the assigned requests that it was serving. For information about specifying the quiescing period for an Application Server instance, see Adding Application Server Instances to a Cluster.

Next, after the quiescing period is over, the load balancer stops sending to the Application Server instance the assigned requests that the instance was serving and fails over these assigned requests to other available instances in the cluster.

Even after the quiescing period is over, the load balancer allows the instance to serve all the requests that the load balancer dispatched to the instance before the quiescing period ended. For such requests, the load balancer returns the results to the client even after the quiescing period is over. Therefore, you should wait for the quiescing period plus a sufficient time to allow the requests being served by the Application Server instance to complete before you shut down the Application Server instance.

In some cases, however, these requests may be very long running, and it is possible that some such requests will not be served when you shut down the Application Server instance.


Note	These assigned requests are honored only if high availability has been enabled for the corresponding web applications. For more information on enabling high availability for web applications, see Making a Web Application Distributable.

Quiescing does not start as soon as you make changes to the loadbalancer.xml file and save the changes. The load balancer polls the loadbalancer.xml file periodically to check if that the time stamp of the file has changed from the last time polling happened. For more information, see Monitoring the Load Balancer Plug-in. If the load balancer detects that the time stamp of the file has changed from the last time, it reloads the entire configuration of the loadbalancer.xml file. Therefore, if you disable an Application Server instance, the load balancer will start quiescing the Application Server instance when it next polls the loadbalancer.xml file. Similarly, if you mark an instance as enabled in the load balancer.xml file, the instance will be enabled when the load balancer next polls the loadbalancer.xml file.

As an example, let us take an Application Server instance whose quiescing period is 31 minutes. At the beginning of the quiescing period, the load balancer stops sending all unassigned requests to the Application Server instance but continues to send the assigned requests to the Application Server instance. After 31 minutes, the loadbalancer stops sending even the assigned requests to the Application Server instance and fails over the assigned requests to other available Application Server instances in the cluster. However, if at 30 minutes and 50 seconds the load balancer had sent an assigned request to the Application Server instance, the Application Server instance will be allowed to serve this assigned request even after 31 minutes are over.

To enable quiescing, you use the enabled attribute of the instance element in the loadbalancer.xml file. In a running cluster, setting the value of enabled for an Application Server instance in that cluster to false in the corresponding loadbalancer.xml file(s) initiates quiescing for that Application Server instance (when the load balancer next polls the loadbalancer.xml file).

You specify the quiescing period by specifying an appropriate value for the disable-timeout-in-minutes subelement of the instance element in the loadbalancer.xml file. The default value of disable-timeout-in-minutes is 31 minutes, which is one minute more than the default session-idle-timeout value of 30 minutes.

You can use the load balancer logs to ascertain if the Application Server instance has been quiesced. Once an instance has been quiesced, the log will contain the following entry:

Instance: instance_name quiesced successfully over the cluster: cluster_name.

instance_name is the name specified for the instance in the loadbalancer.xml file. cluster_name is the name of the cluster, in which the instance exists, as specified in the loadbalancer.xml file.

Disabling and Quiescing a Web Application in a Cluster

If you disable a web application on a cluster, the load balancer stops sending any unassigned request for the web application to Application Server instances in the cluster. If this web application is deployed on other clusters that the load balancer is serving, these unassigned requests are sent to Application Server instances in the other clusters.

However, assigned requests for this web application that are being serviced by Application Server instances in the cluster will continue to be serviced till the quiescing period ends. After the quiescing period is over, the load balancer stops sending even these assigned requests to Application Server instances in the cluster.

You use the enabled attribute of the web-module element in the loadbalancer.xml file to disable a web application in a cluster. In a running cluster, setting the value of enabled for a web application in that cluster to false disables the web application on the cluster. As in the case of Application Server instances, the quiescing does not start immediately after you make the changes and save them. The quiescing starts after the next load balancer polling occurs.

Specify the quiescing period by specifying an appropriate value for the disable-timeout-in-minutes subelement of the web-module element in the loadbalancer.xml file. The default value of disable-timeout is 31 minutes, which is one minute more than the default session-idle-timeout value of 30 minutes.

You can use the load balancer logs to ascertain if the web application has been quiesced. Once a web application has been quiesced, the log will contain the following entry:

Application: application_name quiesced successfully over the cluster cluster_name where,

application_name is the context root of the application as specified in the loadbalancer.xml file.

cluster_name is the name of the cluster, on which the application has been deployed, as specified in the loadbalancer.xml file.

Modifying the Quiescing Period While Quiescing is On

Suppose an Application Server instance or a web application is being quiesced and that the quiescing period is more than the load balancer polling interval.

You now make some other changes to the loadbalancer.xml file before the next load balancer polling interval. When the load balancer next polls the loadbalancer.xml file, it detects a change in the time stamp from the last time it polled the file. Because the quiescing period is more than the load balancer polling period, the quiescing is not over yet. However, the load balancer reloads the entire configuration and starts the quiescing process again.

You can use this feature to alter the quiescing period of an Application Server instance or a web application that is being quiesced. As an example, consider the following scenario.

The quiescing period of an Application Server instance is 30 minutes and the load balancer polling interval is 1 second. You now disable the Application Server instance. When the load balancer polling occurs again, the load balancer starts to quiesce the instance.

Suppose after 10 minutes, you want to reduce the overall quiescing period from 30 minutes to 15 minutes, that is, you want the quiescing period to be over in the next 5 minutes. For this, you change the quiescing period of the Application Server instance to 5 minutes. When the load balancer polling occurs again, the load balancer quiesces the instance once more (because the quiescing is not over yet), with a new quiescing period of 5 minutes.

The overall quiescing period is therefore 15 minutes (10 minutes during the first cycle and 5 minutes during the second cycle).

Obviously, while using this feature, you cannot reduce the quiescing period to a value less than the load balancer polling interval.

Using Multiple Clusters for Online Upgrades Without Loss of Service

You can use the load balancer and multiple clusters to upgrade components within the Sun Java System Application Server without any loss of service. A component can, for example, be a JVM, the Sun Java System Application Server, or a web application.


Caution	Unless you want to use this feature, if the quiescing period is more than the load balancer polling interval, you must ensure that you do not make any other changes to the loadbalancer.xml file before the Application Server instance or the web application has been quiesced. Otherwise, the load balancer will quiesce the Application Server instance or the web application more than once as explained earlier in the section.

Stop one of the clusters. For more information on stopping a cluster, see Stopping a Cluster.

Upgrade the component in that cluster.

Start the cluster. For more information on starting a cluster, see Starting a Cluster.

Repeat the process with the other clusters, one by one.

Because sessions within one cluster will never failover to sessions within another cluster, there is no risk of version mismatch caused by a session failing over from an Application Server instance that is running one version of the component failing over to another Application Server instance (in a different cluster) that is running a different version of the component. A cluster in this way acts as a safe boundary for session failover for the Application Server instances within it.

Reconfiguring an Application Server Instance in a Running Cluster

There may be cases where you want to make changes to the configuration of an (enabled) Application Server instance that require restarting the Application Server instance. For example, you may want to add a JDBC resource to the Application Server instance, which will necessitate the restarting of the Application Server instance. In such cases, follow these steps:


Note	This approach is not possible in the following cases: When you change the schema of the high-availability database (HADB). For more information, see Chapter 21, "Administering the High-Availability Database on Unix(Enterprise Edition)." When you perform an application upgrade that involves a change to the application database schema.


Caution	You must upgrade all instances in a cluster together. Otherwise, there is a risk of version mismatch caused by a session failing over from one Application Server instance to another where the Application Server instances have different versions of components running.

Split the cluster into two clusters with equally distributed Application Server instances by editing the loadbalancer.xml file. There should be a minimum of two listeners configured for each cluster (for failover).

Disable an Application Server instance in one of the clusters by setting the value of the enabled attribute of the instance element to false in the loadbalancer.xml file.

After the quiescing period is over, plus a sufficient time to allow the requests being served by the Application Server instance to complete, reconfigure or upgrade the Application Server instance.

Restart the Application Server instance.

Enable the Application Server instance by setting the value (in the loadbalancer.xml file) of the enabled attribute of the instance element to true.

Follow steps 2 to 5 for all the Application Server instances in both clusters until all instances are reconfigured.

Join the two clusters back to form a single cluster to return to your original configuration.

Using Multiple Load Balancers

To improve fault tolerance and scalability you can configure a set of clusters using multiple load balancers. You can also have multiple load balancers (on multiple web servers) serving a single Application Server cluster.

All the load balancers must have identical configurations and the loadbalancer.xml files for all the load balancers must be identical. It is recommended that you have a master copy of the loadbalancer.xml file and use a script to distribute this master copy to all the load balancers in your system. This ensures that the configuration change is done simultaneously to all the configured load balancers. This is particularly useful when enabling or disabling clusters, instances, and applications.

Typically, requests are sent to these load balancers through a single source, for example a hardware load balancer or a DNS-based load distribution mechanism.

Multiple load balancers can also be used for serving two entirely different set of clusters. For example, you might want your secure (HTTPS) applications to be handled by one load balancer and the rest of the applications by another load balancer. In this case, the loadbalancer.xml files for these load balancers will be different.

Previous Contents Index Next
Sun Java System Application Server Standard and Enterprise Edition 7 2004Q2 Update 3 Administration Guide