Sun GlassFish Enterprise Server v2.1.1 High Availability Administration Guide

Chapter 10 Java Message Service Load Balancing and Failover

This chapter describes how to configure load balancing and failover of the Java Message Service (JMS) for use with the Enterprise Server. It contains the following topics:

Overview of Java Message Service

The Java Message Service (JMS) API is a messaging standard that allows Java EE applications and components to create, send, receive, and read messages. It enables distributed communication that is loosely coupled, reliable, and asynchronous. The Sun Java System Message Queue (MQ), which implements JMS, is tightly integrated with Enterprise Server, enabling you to create components such as message-driven beans (MDBs).

MQ is integrated with Enterprise Server using a connector module, also known as a resource adapter, defined by the Java EE Connector Architecture Specification 1.5. Java EE components deployed to the Enterprise Server exchange JMS messages using the JMS provider integrated via the connector module. Creating a JMS resource in Enterprise Server creates a connector resource in the background. So, each JMS operation invokes the connector runtime and uses the MQ resource adapter in the background.

You can manage the Java Message Service through the Admin Console or the asadmin command-line utility.

Further Information

For more information on configuring JMS resources, see Chapter 4, Configuring Java Message Service Resources, in Sun GlassFish Enterprise Server v2.1.1 Administration Guide . For more information on JMS, see Chapter 18, Using the Java Message Service, in Sun GlassFish Enterprise Server v2.1.1 Developer’s Guide. For more information on connectors (resource adapters), see Chapter 12, Developing Connectors, in Sun GlassFish Enterprise Server v2.1.1 Developer’s Guide.

For more information about the Sun GlassFish Message Queue, see the Message Queue documentation. For general information about the JMS API, see the JMS web page

Configuring the Java Message Service

The Java Message Service configuration is available to all inbound and outbound connections to the Sun GlassFish Enterprise Server cluster or instance. You can configure the Java Message Service with:

The Administration Console. Open the Java Message Service component under the relevant configuration. For details, see Chapter 4, Configuring Java Message Service Resources, in Sun GlassFish Enterprise Server v2.1.1 Administration Guide.

The asadmin set command. You can set the following attributes:

server.jms-service.init-timeout-in-seconds = 60
server.jms-service.type = LOCAL
server.jms-service.start-args =
server.jms-service.default-jms-host = default_JMS_host
server.jms-service.reconnect-interval-in-seconds = 60
server.jms-service.reconnect-attempts = 3
server.jms-service.reconnect-enabled = true
server.jms-service.addresslist-behavior = random
server.jms-service.addresslist-iterations = 3
server.jms-service.mq-scheme = mq
server.jms-service.mq-service = jms

You can also set these properties:

server.jms-service.property.instance-name = imqbroker
server.jms-service.property.instance-name-suffix =
server.jms-service.property.append-version = false

Use the asadmin getcommand to list all the Java Message Service attributes and properties. For more information on asadmin get, see get(1). For more information on asadmin set, see set(1).

You can override the Java Message Service configuration using JMS connection factory settings. For details, see JMS Connection Factories in Sun GlassFish Enterprise Server v2.1.1 Administration Guide.

Note –

You must restart the Enterprise Server instance after changing the configuration of the Java Message Service.

For more information about JMS administration, see Chapter 4, Configuring Java Message Service Resources, in Sun GlassFish Enterprise Server v2.1.1 Administration Guide

Java Message Service Integration

MQ can be integrated with Enterprise Server in three ways: LOCAL, REMOTE, and EMBEDDED. These modes are represented in the Admin Console by the Java Message Service Type attribute.

LOCAL Java Message Service

When the Type attribute is LOCAL (the default for cluster instances), the Enterprise Server will start and stop the MQ broker specified as the Default JMS host. The MQ process is started out-of-process, in a separate VM, from the appliciation server process. Enterprise Server supplies an additional port to the broker . This port will be used by the broker to start the RMI registry. This port number will be equal to the configured JMS port for that instance plus 100. For example, if the JMS port number is 37676, then this additional port number will be 37776.

To create a one-to-one relationship between Enterprise Server instances and Message Queue brokers, set the type to LOCAL and give each Enterprise Server instance a different default JMS host. You can do this regardless of whether clusters are defined in the Enterprise Server or MQ.

With LOCAL type, use the Start Arguments attribute to specify MQ broker startup parameters.

REMOTE Java Message Service

When the Type attribute is REMOTE, the MQ broker must be started separately. For information about starting the broker, see the Sun GlassFish Message Queue Administration Guide.

In this case, Enterprise Server will use an externally configured broker or broker cluster. Also, you must start and stop MQ brokers separately from Enterprise Server, and use MQ tools to configure and tune the broker or broker cluster. REMOTE type is most suitable for Enterprise Server clusters.

With REMOTE type, you must specify MQ broker startup parameters using MQ tools. The Start Arguments attribute is ignored.

EMBEDDED Java Message Service

When the JMS Type attribute is EMBEDDED, it means that the Enterprise Server and the JMS broker are co-located in the same VM and the JMS service is started in-process and managed by the Enterprise Server. In this mode, the JMS operations by pass the networking stack leading to performance optimization.

JMS Hosts List

A JMS host represents an MQ broker. The Java Message Service contains a JMS Hosts list (also called AddressList) that contains all the JMS hosts that Enterprise Server uses.

The JMS Hosts list is populated with the hosts and ports of the specified MQ brokers and is updated whenever a JMS host configuration changes. When you create JMS resources or deploy MDBs, they inherit the JMS Hosts list.

Note –

In the Sun GlassFish Message Queue software, the AddressList property is called imqAddressList.

Default JMS Host

One of the hosts in the JMS Hosts list is designated the default JMS host, named Default_JMS_host. The Enterprise Server instance starts the default JMS host when the Java Message Service type is configured as LOCAL.

If you have created a multi-broker cluster in the Sun GlassFish Message Queue software, delete the default JMS host, then add the Message Queue cluster’s brokers as JMS hosts. In this case, the default JMS host becomes the first one in the JMS Hosts list.

When the Enterprise Server uses a Message Queue cluster, it executes Message Queue specific commands on the default JMS host. For example, when a physical destination is created for a Message Queue cluster of three brokers, the command to create the physical destination is executed on the default JMS host, but the physical destination is used by all three brokers in the cluster.

Creating JMS Hosts

You can create additional JMS hosts in the following ways:

Use the Administration Console. Open the Java Message Service component under the relevant configuration, select the JMS Hosts component, and then click New. For more information, see the Admin Console online help.
Use the asadmin create-jms-host command. For details, see create-jms-host(1).

The JMS Hosts list is updated whenever a JMS host configuration changes.

Connection Pooling and Failover

Enterprise Server supports JMS connection pooling and failover. The Sun GlassFish Enterprise Server pools JMS connections automatically. When the Address List Behavior attribute is random (the default), Enterprise Server selects its primary broker randomly from the JMS host list. When failover occurs, MQ transparently transfers the load to another broker and maintains JMS semantics. The default value for the Address List Behavior attribute is priority, if the JMS type is of type LOCAL.

To specify whether the Enterprise Server tries to reconnect to the primary broker when the connection is lost, select the Reconnect checkbox. If enabled and the primary broker goes down, Enterprise Server tries to reconnect to another broker in the JMS Hosts list.

When Reconnect is enabled, also specify the following attributes:

Address List Behavior: whether connection attempts are in the order of addresses in the JMS Hosts List (priority) or random order (random). If set to Priority, Java Message Service tries to connect to the first MQ broker specified in the JMS Hosts list and uses another one only if the first broker is not available. If set to Random, Java Message Service selects the MQ broker randomly from the JMS Hosts list. If there are many clients attempting a connection using the same connection factory, use this setting to prevent them from all attempting to connect to the same address.
Address List Iterations: number of times the Java Message Service iterates through the JMS Hosts List in an effort to establish (or re-establish) a connection). A value of -1 indicates that the number of attempts is unlimited.
Reconnect Attempts: the number of attempts to connect (or reconnect) for each address in the JMS hosts list before the client runtime tries the next address in the list. A value of -1 indicates that the number of reconnect attempts is unlimited (the client runtime attempts to connect to the first address until it succeeds).
Reconnect Interval: number of seconds between reconnect attempts. This applies for attempts on each address in the JMS hosts list and for successive addresses in the list. If it is too short, this time interval does not give a broker time to recover. If it is too long, the reconnect might represent an unacceptable delay.

You can override these settings using JMS connection factory settings. For details, see JMS Connection Factories in Sun GlassFish Enterprise Server v2.1.1 Administration Guide.

Load-Balanced Message Inflow

You can configure ActivationSpec properties of the jmsra resource adapter in the sun-ejb-jar.xml file for a message-driven bean using activation-config-property elements. Whenever a message-driven bean (EndPointFactory) is deployed, the connector runtime engine finds these properties and configures them accordingly in the resource adapter. See activation-config-property in Sun GlassFish Enterprise Server v2.1.1 Application Deployment Guide.

The Enterprise Server transparently enables messages to be delivered randomly to message-driven beans having the same ClientID . The ClientID is required for durable subscribers.

For non-durable subscribers in which the ClientID is not configured, all instances of a specific message-driven bean that subscribe to same topic are considered equal. When a message-driven bean is deployed to multiple instances of the Enterprise Server, only one of the message-driven beans receives the message. If multiple distinct message-driven beans subscribe to same topic, one instance of each message-driven bean receives a copy of the message.

To support multiple consumers using the same queue, set the maxNumActiveConsumers property of the physical destination to a large value. If this property is set, the Sun Message Queue software allows up to that number of message-driven beans to consume messages from same queue. The message is delivered randomly to the message-driven beans. If maxNumActiveConsumers is set to -1, there is no limit to the number of consumers.

To ensure that local delivery is preferred, set addresslist-behavior to priority. This setting specifies that the first broker in the AddressList is selected first. This first broker is the local colocated Message Queue instance. If this broker is unavailable, connection attempts are made to brokers in the order in which they are listed in the AddressList. This setting is the default for Enterprise Server instances that belong to a cluster.

Note –

Clustering features are not available in the developer profile. For information about profiles, see Usage Profiles in Sun GlassFish Enterprise Server v2.1.1 Administration Guide.

JMS Service High Availability

There are two levels of availability for JMS components:

Service availability – At this level, availability of the JMS service matters, but it's not important if messages are not available for a while. As long as a connection gets failed over to a new available instance providing the service, the JMS component considers that the service is available and functions normally. This level of availability is described in Connection Failover in Sun GlassFish Enterprise Server v2.1.1 Developer’s Guide.
Data availability – At this level, both availability of the service and persistent messages are required. JMS semantics of once and only once delivery and message ordering are also handled at this level.

You can enable data availability in the Sun GlassFish Message Queue cluster that comprises the Java Message Service (JMS). Messages are persisted to the common persistence store and are available from the high-availability database (HADB) if it is installed and the enterprise profile is selected. For information about profiles, see Usage Profiles in Sun GlassFish Enterprise Server v2.1.1 Administration Guide. You must enable availability for the Enterprise Server instances before you can enable data availability for the corresponding brokers.

Note –

Individual applications and modules cannot control or override JMS availability.

To enable data availability, select the Availability Service component under the relevant configuration in the Admin Console. Check the Availability Service box. To enable availability for the JMS service, select the JMS Availability tab, then check the Availability Service box. All instances in an Enterprise Server cluster should have the same instance availability and JMS availability settings to ensure consistent behavior. For details, see the Sun GlassFish Enterprise Server v2.1.1 High Availability Administration Guide.

Note –

Clustering features are not available in the developer profile. For information about profiles, see Usage Profiles in Sun GlassFish Enterprise Server v2.1.1 Administration Guide.

Using MQ Clusters with Enterprise Server

MQ Enterprise Edition supports multiple interconnected broker instances known as a broker cluster. With broker clusters, client connections are distributed across all the brokers in the cluster. Clustering provides horizontal scalability and improves availability.

This section describes how to configure Enterprise Server to use highly available Sun Java System Message Queue clusters. It explains how to start and configure Message Queue clusters.

For more information about the topology of Enterprise Server and MQ deployment, see the Planning Message Queue Broker Deployment chapter in the Sun GlassFish Enterprise Server 2.1 Deployment Planning Guide. .

Highly Available MQ Clusters

Sun Java System Message Queue 4.1 provides a highly available messaging service through a new "Highly Available" cluster type. In an MQ cluster of this type, all the broker instances would share a peer– to–peer relationship and all its broker instances would share a common persistent data store thus offering 'Data Availability.' Instances would automatically be able to detect if an instance fails and perform a takeover of the failed broker's persistent messages, dynamically through a takeover election. Application components that are deployed in the application server could thus leverage these availability features.

With this cluster type, there would not be any loss of transacted persistent messages to a queue or a durable topic subscription. Non-persistent messages or persistent messages to non-durable subscribers are likely to be lost when the broker that the client runtime is connected to is unavailable.

Configuring a Highly Available Broker Cluster in the Local Mode

Start HADB.
Create a domain and start it. To create a domain and start it, use the asadmin commands, create-domain and start-domain respectively. For more information about these commands, see create-domain(1) and start-domain(1).
Create a node agent and start it. To create a node agent and start it, use the asadmin commands, create-node-agent and start-node-agent respectively. For more information about these commands, see create-node-agent(1) and start-node-agent(1).
Create a cluster. You can create a cluster either using the asadmincommand, create-cluster or using the Admin Console. For more information on the create-cluster command, see create-cluster(1). For information on how to create a cluster using the Admin Console, see the Admin Console Online Help.
Create instances in the cluster. While creating an instance, specify the JMS Provider port number being used by the remote broker. If you do not specify one, it will take the default JMS Provider port number. You can create instances using the Admin Console or the asadmin command, create-instance. For information on how to create instances using the Admin Console, see the Admin Console Online Help. For information on the create-instance command, see create-instance(1).
Start the cluster. You can do this from the Admin Console or using the asadmin command start-cluster. For information on how to start a cluster using the Admin Console, refer to the Admin Console Online Help. For information on the start-cluster command, see start-cluster(1).
Configure the HA cluster using the asadmin command, configure-ha-cluster. For more information about the command, see configure-ha-cluster(1).

Configuring a Highly Available Broker Cluster in the Remote Mode

Start HADB.
Create database tables.
Copy the HA driver if you are creating a highly available broker cluster.
```
cp $AS_HOME/hadb/4.4.3-6/lib/hadbjdbc4.jar $S1AS_HOME/imq/lib/ext
```
Create a domain and start it. To do this, use the commands asadmin create-domain and start-domain. For more information about these commands, see create-domain(1) and start-domain(1).
Create a node agent and start it. To do this, use the asadmin commands create-domain and start-node-agent. For more information about these commands, see create-node-agent(1) and start-node-agent(1).
Create a cluster. You can create a cluster either using the asadmin command create-cluster or using the Admin Console. For more information, see create-cluster(1). For information on how to create a cluster using the Admin Console, see the Admin Console Online Help.
Create instances in the cluster. While creating an instance, specify the JMS Provider port number being used by the remote broker. If you do not specify one, it will take the default JMS Provider port number.
Delete the default JMS Host and create JMS hosts to which the instances can connect. Be sure to add each broker as a separate JMS host. For more information on JMS Hosts, see JMS Hosts List.
Set the JMS type as Remote. You can do this using the asadmin command set or from the JMS Service page in the Admin Console.
Set JMS Availability to true if you are configuring a highly available broker. You can do this using the asadmin command set or from the JMS Availability page in the Admin Console.
Start the broker instances.
Start the cluster. For more information, see start-cluster(1).

Auto-clustering for non-HA Clusters

Till now, the administrator had to set up 'non-Highly Available' MQ clusters (MQ clusters with a master broker) separately as explained in the procedure following this section. In this release, in addition to the manual process (of type REMOTE) of setting up an MQ cluster, Enterprise Server provides 'auto-clustering,' which means that a co-located non-HA cluster (of type LOCAL) will be created automatically when a user creates an Application Server cluster. This will be the default mode of creating MQ clusters. For example, when the administrator creates an Application Server cluster with three Application Server instances, each Application Server instance will be configured to work with a co-located broker, and as a result the three MQ broker instances will be made to form an MQ cluster transparently. The first Application Server instance's MQ broker will be set to be the master broker. Auto-clustering, however, has a disadvantage too. If the administrator adds an instance to the cluster, the MQ broker instance created automatically will not be able to take part in the cluster. This behavior also applies if an instance is removed from the cluster.

To Enable MQ Clusters with Enterprise Server Clusters

Before You Begin

Perform the following steps if the cluster is of type REMOTE. If the cluster is of type LOCAL, steps 1 to 4 are not applicable .

Create a cluster, if one does not already exist.

For information on creating clusters, see To Create a Cluster.

Create an MQ broker cluster.

First, delete the default JMS host that refers to the broker started by the Domain Administration Server, and then create three external brokers (JMS hosts) that will be in the MQ broker cluster.

Create a JMS host with either the Admin Console or the asadmin command-line utility.

To use asadmin, the commands are for example:

asadmin delete-jms-host --target cluster1 default_JMS_host
asadmin create-jms-host --target cluster1
     --mqhost myhost1 --mqport 6769
     --mquser admin --mqpassword admin broker1
asadmin create-jms-host --target cluster1
     --mqhost myhost2 --mqport 6770
     --mquser admin --mqpassword admin broker2
asadmin create-jms-host --target cluster1
     --mqhost myhost3 --mqport 6771
     --mquser admin --mqpassword admin broker3

To create the hosts with Admin Console:

Navigate to the JMS Hosts node (Configurations > config-name > Java Message Service > JMS Hosts)

Delete the default broker (default_JMS_host).

Select the checkbox next to it, and then click Delete.

Click New to create each JMS host and enter its property values.

Fill in the values for host name, DNS name or IP address, port number, administrative user name and password.

Start the master MQ broker and the other MQ brokers.

In addition to the three external brokers started on JMS host machines, start one master broker on any machine. This master broker need not be part of a broker cluster. For example:
/usr/bin/imqbrokerd -tty -name brokerm -port 6772 -cluster myhost1:6769,myhost2:6770,myhost2:6772,myhost3:6771 -D"imq.cluster.masterbroker=myhost2:6772"

Start the instances in the cluster.

Create JMS resources on the cluster:

Create JMS physical destinations.

For example, using asadmin:
asadmin create-jmsdest --desttype queue --target cluster1 MyQueue asadmin create-jmsdest --desttype queue --target cluster1 MyQueue1
To use Admin Console:
1. Navigate to the JMS Hosts page (Configurations > config-name > Java Message Service > Physical Destinations).
2. Click New to create each JMS physical destination.
3. For each destination, enter its name and type (queue).

Create JMS connection factories.

For example, using asadmin:

asadmin create-jms-resource --target cluster1
--restype javax.jms.QueueConnectionFactory jms/MyQcf
asadmin create-jms-resource --target cluster1
--restype javax.jms.QueueConnectionFactory jms/MyQcf1

To use Admin Console:

Navigate to the JMS Connection Factories page (Resources > JMS Resources > Connection Factories).

To create each connection factory, click New.

The Create JMS Connection Factory page opens.

For each connection factory, enter JNDI Name (for example jms/MyQcf) and Type, javax.jms.QueueConnectionFactory

Select the cluster from the list of available targets at the bottom of the page and click Add.

Click OK to create the connection factory.

Create JMS destination resources.

For example, using asadmin:

asadmin create-jms-resource --target cluster1
     --restype javax.jms.Queue
     --property imqDestinationName=MyQueue jms/MyQueue
asadmin create-jms-resource --target cluster1
     --restype javax.jms.Queue
     --property imqDestinationName=MyQueue1 jms/MyQueue1

To use Admin Console:

Navigate to the JMS Destination Resources page (Resources > JMS Resources > Connection Factories).

To create each destination resource, click New.

The Create JMS Destination Resource page opens.

For each destination resource, enter JNDI Name (for example jms/MyQueue) and Type javax.jms.Queue.

Select the cluster from the list of available targets at the bottom of the page and click Add.

Click OK to create the destination resource.

Deploy the applications with the – retrieve option for application clients. For example:
asadmin deploy --target cluster1 --retrieve /opt/work/MQapp/mdb-simple3.ear

Access the application and test it to ensure it is behaving as expected.

If you want to return the Enterprise Server to its default JMS configuration, delete the JMS hosts you created and recreate the default. For example:

asadmin delete-jms-host --target cluster1 broker1
asadmin delete-jms-host --target cluster1 broker2
asadmin delete-jms-host --target cluster1 broker3
asadmin create-jms-host --target cluster1
 --mqhost myhost1 --mqport 7676
 --mquser admin --mqpassword admin
 default_JMS_host

You can also perform the equivalent operation with Admin Console.

Troubleshooting

If you encounter problems, consider the following:

View the server log file located at as-install-dir/nodeagents/node-agent-name/instance-name/logs/server.log. If you see in the log file that an MQ broker does not respond to a message, stop the broker and then restart it.
View the broker log available at as-install-dir/nodeagents/node-agent-name/instance-name/imq/imq-instance-name/log/log.txt.
For the Remote JMS type, always be sure to start MQ brokers first, and then the instances.
When all MQ brokers are down, it takes 30 minutes for Enterprise Server to go down or up, with the default values in Java Message Service. Tune Java Message Service values to get acceptable values for this timeout. For example:
```
asadmin set --user admin --password administrator 
cluster1.jms-service.reconnect-interval-in-seconds=5
```