Chapter 7 Java Message Service Load Balancing and Failover

This chapter describes how to configure load balancing and failover of the Java Message Service (JMS) for use with the Communications Server. It contains the following topics:

• Overview of Java Message Service

• Configuring the Java Message Service

• Connection Pooling and Failover

• Using MQ Clusters with Communications Server

Overview of Java Message Service

The Java Message Service (JMS) API is a messaging standard that allows Java EE applications and components to create, send, receive, and read messages. It enables distributed communication that is loosely coupled, reliable, and asynchronous. The Sun Java System Message Queue (MQ), which implements JMS, is tightly integrated with Communications Server, enabling you to create components such as message-driven beans (MDBs).

MQ is integrated with Communications Server using a connector module, also known as a resource adapter, defined by the Java EE Connector Architecture Specification 1.5. Java EE components deployed to the Communications Server exchange JMS messages using the JMS provider integrated via the connector module. Creating a JMS resource in Communications Server creates a connector resource in the background. So, each JMS operation invokes the connector runtime and uses the MQ resource adapter in the background.

You can manage the Java Message Service through the Admin Console or the asadmin command-line utility.

Further Information

For more information on configuring JMS resources, see Chapter 4, Configuring Java Message Service Resources, in Sun GlassFish Communications Server 2.0 Administration Guide. For more information on JMS, see Chapter 18, Using the Java Message Service, in Sun GlassFish Communications Server 2.0 Developer’s Guide. For more information on connectors (resource adapters), see Chapter 12, Developing Connectors, in Sun GlassFish Communications Server 2.0 Developer’s Guide.

For more information about the Sun GlassFish Message Queue, see the Message Queue documentation. For general information about the JMS API, see the JMS web page

Configuring the Java Message Service

The Java Message Service configuration is available to all inbound and outbound connections to the Sun GlassFish Communications Server cluster or instance. You can configure the Java Message Service with:

• The Administration Console. Open the Java Message Service component under the relevant configuration. For details, see Chapter 4, Configuring Java Message Service Resources, in Sun GlassFish Communications Server 2.0 Administration Guide.

• The asadmin set command. You can set the following attributes:

  server.jms-service.init-timeout-in-seconds = 60
  server.jms-service.type = LOCAL
  server.jms-service.start-args =
  server.jms-service.default-jms-host = default_JMS_host
  server.jms-service.reconnect-interval-in-seconds = 60
  server.jms-service.reconnect-attempts = 3
  server.jms-service.reconnect-enabled = true
  server.jms-service.addresslist-behavior = random
  server.jms-service.addresslist-iterations = 3
  server.jms-service.mq-scheme = mq
  server.jms-service.mq-service = jms

  You can also set these properties:

  server.jms-service.property.instance-name = imqbroker
  server.jms-service.property.instance-name-suffix =
  server.jms-service.property.append-version = false

  Use the asadmin getcommand to list all the Java Message Service attributes and properties. For more information on asadmin get, see get(1).  For more information on asadmin set, see set(1).

You can override the Java Message Service configuration using JMS connection factory settings. For details, see JMS Connection Factories in Sun GlassFish Communications Server 2.0 Administration Guide.

You must restart the Communications Server instance after changing the configuration of the Java Message Service.

For more information about JMS administration, see Chapter 4, Configuring Java Message Service Resources, in Sun GlassFish Communications Server 2.0 Administration Guide

Java Message Service Integration

MQ can be integrated with Communications Server in three ways: LOCAL, REMOTE, and EMBEDDED. These modes are represented in the Admin Console by the Java Message Service Type attribute.

LOCAL Java Message Service

When the Type attribute is LOCAL (the default for cluster instances), the Communications Server will start and stop the MQ broker specified as the Default JMS host. The MQ process is started out-of-process, in a separate VM, from the appliciation server process. Communications Server supplies an additional port to the broker . This port will be used by the broker to start the RMI registry. This port number will be equal to the configured JMS port for that instance plus 100. For example, if the JMS port number is 37676, then this additional port number will be 37776.

To create a one-to-one relationship between Communications Server instances and Message Queue brokers, set the type to LOCAL and give each Communications Server instance a different default JMS host. You can do this regardless of whether clusters are defined in the Communications Server or MQ.

With LOCAL type, use the Start Arguments attribute to specify MQ broker startup parameters.

REMOTE Java Message Service

When the Type attribute is REMOTE, the MQ broker must be started separately. For information about starting the broker, see the Sun GlassFish Message Queue Administration Guide.

In this case, Communications Server will use an externally configured broker or broker cluster. Also, you must start and stop MQ brokers separately from Communications Server, and use MQ tools to configure and tune the broker or broker cluster. REMOTE type is most suitable for Communications Server clusters.

With REMOTE type, you must specify MQ broker startup parameters using MQ tools. The Start Arguments attribute is ignored.

EMBEDDED Java Message Service

When the JMS Type attribute is EMBEDDED, it means that the Communications Server and the JMS broker are co-located in the same VM and the JMS service is started in-process and managed by the Communications Server. In this mode, the JMS operations by pass the networking stack leading to performance optimization.

JMS Hosts List

A JMS host represents an MQ broker. The Java Message Service contains a JMS Hosts list (also called AddressList) that contains all the JMS hosts that Communications Server uses.

The JMS Hosts list is populated with the hosts and ports of the specified MQ brokers and is updated whenever a JMS host configuration changes. When you create JMS resources or deploy MDBs, they inherit the JMS Hosts list.

In the Sun GlassFish Message Queue software, the AddressList property is called imqAddressList.

Default JMS Host

One of the hosts in the JMS Hosts list is designated the default JMS host, named Default_JMS_host. The Communications Server instance starts the default JMS host when the Java Message Service type is configured as LOCAL.

If you have created a multi-broker cluster in the Sun GlassFish Message Queue software, delete the default JMS host, then add the Message Queue cluster’s brokers as JMS hosts. In this case, the default JMS host becomes the first one in the JMS Hosts list.

When the Communications Server uses a Message Queue cluster, it executes Message Queue specific commands on the default JMS host. For example, when a physical destination is created for a Message Queue cluster of three brokers, the command to create the physical destination is executed on the default JMS host, but the physical destination is used by all three brokers in the cluster.

Creating JMS Hosts

You can create additional JMS hosts in the following ways:

• Use the Administration Console. Open the Java Message Service component under the relevant configuration, select the JMS Hosts component, and then click New. For more information, see the Admin Console online help.

• Use the asadmin create-jms-host command. For details, see create-jms-host(1).

  The JMS Hosts list is updated whenever a JMS host configuration changes.

Connection Pooling and Failover

Communications Server supports JMS connection pooling and failover. The Sun GlassFish Communications Server pools JMS connections automatically. When the Address List Behavior attribute is random (the default), Communications Server selects its primary broker randomly from the JMS host list. When failover occurs, MQ transparently transfers the load to another broker and maintains JMS semantics. The default value for the Address List Behavior attribute is priority, if the JMS type is of type LOCAL.

To specify whether the Communications Server tries to reconnect to the primary broker when the connection is lost, select the Reconnect checkbox. If enabled and the primary broker goes down, Communications Server tries to reconnect to another broker in the JMS Hosts list.

When Reconnect is enabled, also specify the following attributes:

• Address List Behavior: whether connection attempts are in the order of addresses in the JMS Hosts List (priority) or random order (random). If set to Priority, Java Message Service tries to connect to the first MQ broker specified in the JMS Hosts list and uses another one only if the first broker is not available. If set to Random, Java Message Service selects the MQ broker randomly from the JMS Hosts list. If there are many clients attempting a connection using the same connection factory, use this setting to prevent them from all attempting to connect to the same address.

• Address List Iterations: number of times the Java Message Service iterates through the JMS Hosts List in an effort to establish (or re-establish) a connection). A value of -1 indicates that the number of attempts is unlimited.

• Reconnect Attempts: the number of attempts to connect (or reconnect) for each address in the JMS hosts list before the client runtime tries the next address in the list. A value of -1 indicates that the number of reconnect attempts is unlimited (the client runtime attempts to connect to the first address until it succeeds).

• Reconnect Interval: number of seconds between reconnect attempts. This applies for attempts on each address in the JMS hosts list and for successive addresses in the list. If it is too short, this time interval does not give a broker time to recover. If it is too long, the reconnect might represent an unacceptable delay.

You can override these settings using JMS connection factory settings. For details, see JMS Connection Factories in Sun GlassFish Communications Server 2.0 Administration Guide.

Load-Balanced Message Inflow

You can configure ActivationSpec properties of the jmsra resource adapter in the sun-ejb-jar.xml file for a message-driven bean using activation-config-property elements. Whenever a message-driven bean (EndPointFactory) is deployed, the connector runtime engine finds these properties and configures them accordingly in the resource adapter. See activation-config-property in Sun GlassFish Communications Server 2.0 Application Deployment Guide.

The Communications Server transparently enables messages to be delivered randomly to message-driven beans having the same ClientID . The ClientID is required for durable subscribers.

For non-durable subscribers in which the ClientID is not configured, all instances of a specific message-driven bean that subscribe to same topic are considered equal. When a message-driven bean is deployed to multiple instances of the Communications Server, only one of the message-driven beans receives the message. If multiple distinct message-driven beans subscribe to same topic, one instance of each message-driven bean receives a copy of the message.

To support multiple consumers using the same queue, set the maxNumActiveConsumers property of the physical destination to a large value. If this property is set, the Sun Message Queue software allows up to that number of message-driven beans to consume messages from same queue. The message is delivered randomly to the message-driven beans. If maxNumActiveConsumers is set to -1, there is no limit to the number of consumers.

To ensure that local delivery is preferred, set addresslist-behavior to priority. This setting specifies that the first broker in the AddressList is selected first. This first broker is the local colocated Message Queue instance. If this broker is unavailable, connection attempts are made to brokers in the order in which they are listed in the AddressList. This setting is the default for Communications Server instances that belong to a cluster.

Clustering features are not available in the developer profile. For information about profiles, see Usage Profiles in Sun GlassFish Communications Server 2.0 Administration Guide.

Using MQ Clusters with Communications Server

MQ Enterprise Edition supports multiple interconnected broker instances known as a broker cluster. With broker clusters, client connections are distributed across all the brokers in the cluster. Clustering provides horizontal scalability and improves availability.

This section describes how to configure Communications Server to use Sun Java System Message Queue clusters. It explains how to start and configure Message Queue clusters.

For more information about the topology of Communications Server and MQ deployment, see the Planning Message Queue Broker Deployment chapter in the Sun GlassFish Enterprise Server 2.1 Deployment Planning Guide. .

Auto-clustering for non-HA Clusters

Till now, the administrator had to set up 'non-Highly Available' MQ clusters (MQ clusters with a master broker) separately as explained in the procedure following this section. In this release, in addition to the manual process (of type REMOTE) of setting up an MQ cluster, Communications Server provides 'auto-clustering,' which means that a co-located non-HA cluster (of type LOCAL) will be created automatically when a user creates an Application Server cluster. This will be the default mode of creating MQ clusters. For example, when the administrator creates an Application Server cluster with three Application Server instances, each Application Server instance will be configured to work with a co-located broker, and as a result the three MQ broker instances will be made to form an MQ cluster transparently. The first Application Server instance's MQ broker will be set to be the master broker. Auto-clustering, however, has a disadvantage too. If the administrator adds an instance to the cluster, the MQ broker instance created automatically will not be able to take part in the cluster. This behavior also applies if an instance is removed from the cluster.

To Enable MQ Clusters with Communications Server Clusters

Before You Begin

Perform the following steps if the cluster is of type REMOTE. If the cluster is of type LOCAL, steps 1 to 4 are not applicable .

1. Create a cluster, if one does not already exist.

   For information on creating clusters, see To Create a Cluster.

2. Create an MQ broker cluster.

   First, delete the default JMS host that refers to the broker started by the Domain Administration Server, and then create three external brokers (JMS hosts) that will be in the MQ broker cluster.

   Create a JMS host with either the Admin Console or the asadmin command-line utility.

   To use asadmin, the commands are for example:

   asadmin delete-jms-host --target cluster1 default_JMS_host asadmin create-jms-host --target cluster1 --mqhost myhost1 --mqport 6769 --mquser admin --mqpassword admin broker1 asadmin create-jms-host --target cluster1 --mqhost myhost2 --mqport 6770 --mquser admin --mqpassword admin broker2 asadmin create-jms-host --target cluster1 --mqhost myhost3 --mqport 6771 --mquser admin --mqpassword admin broker3

   To create the hosts with Admin Console:

   1. Navigate to the JMS Hosts node (Configurations > config-name > Java Message Service > JMS Hosts)

   2. Delete the default broker (default_JMS_host).

      Select the checkbox next to it, and then click Delete.

   3. Click New to create each JMS host and enter its property values.

      Fill in the values for host name, DNS name or IP address, port number, administrative user name and password.

3. Start the master MQ broker and the other MQ brokers.

   In addition to the three external brokers started on JMS host machines, start one master broker on any machine. This master broker need not be part of a broker cluster. For example:

   /usr/bin/imqbrokerd -tty -name brokerm -port 6772 -cluster myhost1:6769,myhost2:6770,myhost2:6772,myhost3:6771 -D"imq.cluster.masterbroker=myhost2:6772"

4. Start the instances in the cluster.

5. Create JMS resources on the cluster:

   1. Create JMS physical destinations.

      For example, using asadmin:

      asadmin create-jmsdest --desttype queue --target cluster1 MyQueue asadmin create-jmsdest --desttype queue --target cluster1 MyQueue1

      To use Admin Console:

      1. Navigate to the JMS Hosts page (Configurations > config-name > Java Message Service > Physical Destinations).

      2. Click New to create each JMS physical destination.

      3. For each destination, enter its name and type (queue).

   2. Create JMS connection factories.

      For example, using asadmin:

      asadmin create-jms-resource --target cluster1 --restype javax.jms.QueueConnectionFactory jms/MyQcf asadmin create-jms-resource --target cluster1 --restype javax.jms.QueueConnectionFactory jms/MyQcf1

      To use Admin Console:

      1. Navigate to the JMS Connection Factories page (Resources > JMS Resources > Connection Factories).

      2. To create each connection factory, click New.

         The Create JMS Connection Factory page opens.

      3. For each connection factory, enter JNDI Name (for example jms/MyQcf) and Type, javax.jms.QueueConnectionFactory

      4. Select the cluster from the list of available targets at the bottom of the page and click Add.

      5. Click OK to create the connection factory.

   3. Create JMS destination resources.

      For example, using asadmin:

      asadmin create-jms-resource --target cluster1 --restype javax.jms.Queue --property imqDestinationName=MyQueue jms/MyQueue asadmin create-jms-resource --target cluster1 --restype javax.jms.Queue --property imqDestinationName=MyQueue1 jms/MyQueue1

      To use Admin Console:

      1. Navigate to the JMS Destination Resources page (Resources > JMS Resources > Connection Factories).

      2. To create each destination resource, click New.

         The Create JMS Destination Resource page opens.

      3. For each destination resource, enter JNDI Name (for example jms/MyQueue) and Type javax.jms.Queue.

      4. Select the cluster from the list of available targets at the bottom of the page and click Add.

      5. Click OK to create the destination resource.

6. Deploy the applications with the – retrieve option for application clients. For example:

   asadmin deploy --target cluster1 --retrieve /opt/work/MQapp/mdb-simple3.ear

7. Access the application and test it to ensure it is behaving as expected.

8. If you want to return the Communications Server to its default JMS configuration, delete the JMS hosts you created and recreate the default. For example:

   asadmin delete-jms-host --target cluster1 broker1 asadmin delete-jms-host --target cluster1 broker2 asadmin delete-jms-host --target cluster1 broker3 asadmin create-jms-host --target cluster1 --mqhost myhost1 --mqport 7676 --mquser admin --mqpassword admin default_JMS_host

   You can also perform the equivalent operation with Admin Console.

Troubleshooting

If you encounter problems, consider the following:

• View the server log file located at as-install-dir/nodeagents/node-agent-name/instance-name/logs/server.log. If you see in the log file that an MQ broker does not respond to a message, stop the broker and then restart it.

• View the broker log available at as-install-dir/nodeagents/node-agent-name/instance-name/imq/imq-instance-name/log/log.txt.

• For the Remote JMS type, always be sure to start MQ brokers first, and then the instances.

• When all MQ brokers are down, it takes 30 minutes for Communications Server to go down or up, with the default values in Java Message Service. Tune Java Message Service values to get acceptable values for this timeout. For example:

  asadmin set --user admin --password administrator 
  cluster1.jms-service.reconnect-interval-in-seconds=5