9 Using Distributed Destinations

This chapter describes the concepts and functionality of distributed destinations necessary to design higher availability (HA) applications.

What is a Distributed Destination?

A distributed destination is a set of destinations (queues or topics) that are accessible as a single, logical destination to a client. A distributed destination has the following characteristics:

  • It is referenced by its own JNDI name.

  • Members of the set are usually distributed across multiple servers within a cluster, with each destination member belonging to a separate JMS server.

Why Use a Distributed Destination

Applications that use distributed destinations are more highly available than applications that use simple destinations because WebLogic JMS provides load balancing and failover for member destinations of a distributed destination within a cluster. Once properly configured, your producers and consumers are able to send and receive messages through the distributed destination. WebLogic JMS then balances the messaging load across all available members of the distributed destination. When one member becomes unavailable due a server failure, traffic is then redirected toward other available destination members in the set. For more information on how destination members are load balanced, see "Configuring Distributed Destination Resources" in Administering JMS Resources for Oracle WebLogic Server.

Creating a Distributed Destination

Distributed destinations are created by the system administrator using the Administration Console. For more information, see "Configuring Distributed Destination Resources" in Administering JMS Resources for Oracle WebLogic Server.

Types of Distributed Destinations

WebLogic Server supports two types of distributed destinations:

Uniform Distributed Destinations

In a uniform distributed destination (UDD), each of the member destinations has a consistent configuration of all distributed destination parameters, particularly in regards to weighting, security, persistence, paging, and quotas.

Oracle recommends using UDDs because you no longer need to create or designate destination members, but instead rely on WebLogic Server to uniformly create the necessary members on the JMS servers to which a UDD is targeted. This feature of UDDs provides dynamic updating of a UDD when a new member is added or a member is removed.

For example, if UDD is targeted to a cluster, there is a UDD member on every JMS server in the cluster. If a new JMS server is added, a new UDD member is dynamically added to the UDD. Likewise, if a JMS server is removed, the corresponding UDD member is removed from the UDD. This allows UDDs to provide higher availability by eliminating bottlenecks caused by configuration errors. For more information, see "Configuring Distributed Destination Resources" in Administering JMS Resources for Oracle WebLogic Server.

Weighted Distributed Destinations

Note:

Weighted distributed destinations are deprecated in Weblogic Server 10.3.4.0. Oracle recommends using Uniform Distributed Destinations.

In a weighted distributed destination, the member destinations do not have a consistent configuration of all distributed destination parameters, particularly in regards to weighting, security, persistence, paging, and quotas.

Oracle recommends converting weighted distributed destinations to UDDs because of the administrative inflexibility when creating members that are intended to carry extra message load or have extra capacity (more weight). Lack of a consistent member configuration can lead to unforeseen administrative and application problems because the weighted distributed destination can not be deployed consistently across a cluster.

For more information, see "Configuring Distributed Destination Resources" in Administering JMS Resources for Oracle WebLogic Server.

Using Distributed Destinations

A distributed destination is a set of physical JMS destination members (queues or topics) that is accessed through a single JNDI name. As such, a distributed destination can be looked up using JNDI. It implements the javax.jms.Destination interface, at http://docs.oracle.com/javaee/6/api/javax/jms/Destination.html, and can be used to create producers, consumers, and browsers.

For information on obtaining a reference to a distributed destination, see How to Lookup a Destination.

Using Distributed Queues

A distributed queue is a set of physical JMS queue members. As such, a distributed queue can be used to create a QueueSender, QueueReceiver, and a QueueBrowser. The fact that a distributed queue represents multiple physical queues is mostly transparent to your application.

The queue members can be located anywhere, but must all be served by JMS servers in a single server cluster. When a message is sent to a distributed queue, it is sent to exactly one of the physical queues in the set of members for the distributed queue. Once the message arrives at the queue member, it is available for receipt by consumers of that queue member only. '

This section provides information on using distributed queues:

Queue Forwarding

Queue members can forward messages to other queue members by configuring the Forward Delay attribute in the Administration Console, which is disabled by default. This attribute defines the amount of time, in seconds, that a distributed queue member with messages, but which has no consumers, will wait before forwarding its messages to other queue members that do have consumers. By default, WebLogic Server resets the delivery count when forwarding between distributed queue members. See Reset Delivery Count On Forward.

QueueSenders

After creating a queue sender, if the queue supplied at creation time was a distributed queue, then each time a message is produced using the sender a decision is made as to which queue member will receive the message. Each message is sent to a single physical queue member.

The message is not replicated in any way. As such, the message is only available from the queue member where it was sent. If that physical queue becomes unavailable before a given message is received, then the message is unavailable until that queue member comes back online.

It is not enough to send a message to a distributed queue and expect the message to be received by a queue receiver of that distributed queue. Since the message is sent to only one physical queue member, there must be a queue receiver receiving or listening on that queue member.

Note:

For information on the load-balancing heuristics for distributed queues with zero consumers, see "Configuring Distributed Destination Resources" in Administering JMS Resources for Oracle WebLogic Server.

QueueReceivers

When creating a queue receiver, if the supplied queue is a distributed queue, then a single physical queue member is chosen for the receiver at creation time. The created QueueReceiver is pinned to that queue member until the queue receiver loses its access to the queue member. At that point, the consumer will receive a JMSException, as follows:

  • If the queue receiver is synchronous, then the exception is returned to the user directly.

  • If the queue receiver is asynchronous, then the exception is delivered inside of a ConsumerClosedException that is delivered to the ExceptionListener defined for the consumer session, if any.

Upon receiving such an exception, an application can close its queue receiver and recreate it. If any other queue members are available within the distributed queue, then the creation will succeed and the new queue receiver will be pinned to one of those queue members. If no other queue member is available, then the application won't be able to recreate the queue receiver and will have to try again later.

Note:

For information on the load-balancing heuristics for distributed queues with zero consumers, see "Configuring Distributed Destination Resources" in Administering JMS Resources for Oracle WebLogic Server.

QueueBrowsers

When creating a queue browser, if the supplied queue is a distributed queue, then a single physical queue member is chosen for the browser at creation time. The created queue browser is pinned to that queue member until the receiver loses its access to the queue member. At that point, any calls to the queue browser will receive a JMSException. Any calls to the enumeration will return a NoSuchElementException.

Note:

The queue browser can only browse the queue member that it is pinned to. Even though a distributed queue was specified at creation time, the queue browser cannot see or browse messages for the other queue members in the distributed destination.

Using Replicated Distributed Topics

A distributed topic is a set of physical JMS topic members. As such, a distributed topic can be used to create a TopicPublisher and TopicSubscriber. The fact that a distributed topic represents multiple physical topics is mostly transparent to the application.

Note:

Durable subscribers (DurableTopicSubscriber) cannot be created for distributed topics. However, you can still create a durable subscription on distributed topic member and the other topic members will forward the messages to the topic member that has the durable subscription.

The topic members can be located anywhere but must all be served either by a single WebLogic Server or any number of servers in a cluster. When a message is sent to a distributed topic, it is sent to all of the topic members in the distributed topic set. This allows all subscribers to the distributed topic to receive messages published for the distributed topic.

A message published directly to a topic member of a distributed destination (that is, the publisher did not specify the distributed destination) is also forwarded to all the members of that distributed topic. This includes subscribers that originally subscribed to the distributed topic, and which happened to be assigned to that particular topic member. In other words, publishing a message to a specific distributed topic member automatically forwards it to all the other distributed topic members, just as publishing a message to a distributed topic automatically forwards it to all of its distributed topic members. For more information about looking up specific distributed destination members, see Accessing Distributed Destination Members.

This section provides information on using distributed topics:

TopicPublishers

When creating a topic publisher, if the supplied destination is a distributed destination, then any messages sent to that distributed destination are sent to all available topic members for that distributed topic, as follows:

  • When some of the members of a uniform distributed topic are offline, non-persistent messages published to the DT are saved for those members and made available when the members come back online.

    In releases prior to 9.0, if you did not configure a persistent store for a JMS server or if there was a persistent store defined and storedEnabled=false was set on the distributed topic (DT) member, non-persistent messages were dropped and not made available when the DT member came back online. If your application depends on dropping these messages, you can approximate similar behavior by setting the time-to-live for a server to a very low value. This will allow the messages to be disregarded before an offline DT member would come back online. New applications developed on WebLogic Server releases 10.3.4.0 and higher can use partitioned distributed topics with message-driven beans (MDBs) as message consumers to provide a similar capability. See "Developing Advanced Pub/Sub Applications" in Programming JMS for Oracle WebLogic Server.

  • If one or more of the distributed topic members is not reachable, and the message being sent is persistent, then the message is stored and forwarded to the other topic members when they become reachable. However, the message can only be persistently stored if the topic member has a JMS store configured.

    Note:

    Every effort is made to first forward the message to distributed members that utilize a persistent store. However, if none of the distributed members utilize a store, then the message is still sent to one of the members according to the selected load-balancing algorithm, as described in "Configuring Distributed Destination Resources" in Administering JMS Resources for Oracle WebLogic Server.

  • If all of the distributed topic members are unreachable (regardless of whether the message is persistent or non-persistent), then the publisher receives a JMSException when it tries to send a message.

TopicSubscribers

When creating a topic subscriber, if the supplied topic is a distributed topic, then the topic subscriber receives messages published to that distributed topic. If one or more of the topic members for the distributed topic are not reachable by a topic subscriber, then depending on whether the messages are persistent or non-persistent the following occurs:

  • Any persistent messages published to one or more unreachable distributed topic members are eventually received by topic subscribers of those topic members once they become reachable. However, the messages can only be persistently stored if the topic member has a JMS store configured.

  • Any non-persistent messages published to those unreachable distributed topic members will not be received by that topic subscriber.

    Note:

    If a JMS store is configured for a JMS server that is hosting a distributed topic member, then all the Distributed Topic System Subscribers associated with that member destination are treated as durable subscriptions, even when a topic member does not have a JMS store explicitly configured. As such, the saving of all the messages sent to these distributed topic subscribers in memory can result in unexpected memory and disk consumption. Therefore, a recommended best design practice when deploying distributed destination is to consistently configure all member destinations: either with a JMS store for durable messages, or without a JMS store for non-durable messages. For example, if you want all of your distributed topic subscribers to be non-durable, but some member destinations implicitly have a JMS store configured because their associated JMS server uses a JMS store, then you need to explicitly set the StoreEnabled attribute to False for each member destination to override the JMS server setting.

Ultimately, a topic subscriber is pinned to a physical topic member. If that topic member becomes unavailable, then the topic subscriber will receive a JMSException, as follows:

  • If the topic subscriber is synchronous, then the exception is returned to the user directly.

  • If the topic subscriber is asynchronous, then the exception is delivered inside of a ConsumerClosedException that is delivered to the ExceptionListener defined for the consumer session, if any.

Upon receiving such an exception, an application can close its topic subscriber and recreate it. If any other topic member is available within the distributed topic, then the creation should be successful and the new topic subscriber will be pinned to one of those topic members. If no other topic member is available, then the application will not be able to recreate the topic subscriber and will have to try again later.

Deploying Message-Driven Beans on a Distributed Topic

For information on how to deploy MDBx on topics, see Configuring and Deploying MDBs Using Distributed Topics in Developing Message-Driven Beans for Oracle WebLogic Server.

Using Partitioned Distributed Topics

Starting in WebLogic Server 10.3.4.0, partitioned distributed topics, combined with the ability to share subscriptions and allow multiple connections to use the same Client ID, provide the following application design patterns that provide parallel processing and HA capabilities similar to distributed queues:

  • One-copy-per-instance: Each instance of an application gets one copy of each message that is published to the Topic.

  • One-copy-per-application: Each application as a whole (that is all instances of the application together) receives one copy of each message that is published to the DT. That is each instance only receives a subset of the messages that are sent to the DT.

Note:

Oracle recommends designing applications that utilize WebLogic Server MDBs. See Configuring and Deploying MDBs Using Distributed Topics in Developing Message-Driven Beans for Oracle WebLogic Server for detailed information on how to design and implement applications that use message-driven beans to provide improved HA and scalability.

For more information on using Partitioned Distributed Topics, see Developing Advanced Pub/Sub Applications.

Accessing Distributed Destination Members

For information on how to access distributed destinations and their members, see How to Lookup a Destination.

Distributed Destination Failover

Note:

If the distributed queue member on which a queue producer is created should fail, yet the WebLogic Server instance where the producer's JMS connection resides is still running, the producer remains alive and WebLogic JMS will fail it over to another distributed queue member, irrespective of whether the Load Balancing option is enabled. For example, a WebLogic cluster contains WLSServer1, WLSServer2, and WLSServer3 and you are connected to WLServer2. If server WLSServer 2 fails, WebLogic JMS fail the producer over to one of the remaining cluster members. For more information, see "Configuring Distributed Destination Resources" in Administering JMS Resources for Oracle WebLogic Server.

A simple way to failover a client connected to a failed distributed destination is to write reconnect logic in the client code to connect to the distributed destination after catching onException.

Using Message-Driven Beans with Distributed Destinations

A message-driven bean (MDB) acts as a JMS message listener, which is similar to an event listener except that it receives messages instead of events. For more information on MDBs, see:

Common Use Cases for Distributed Destinations

The following sections provide common use case scenarios when using distributed destinations:

Maximizing Production

To maximize message production, Oracle recommends that each member of a distributed destination be associated with a producer and a consumer. The following diagram demonstrates how to efficiently provide maximum message production and high availability using a UDD without using load balancing:

Figure 9-1 Paired Producers and Consumers

Description of Figure 9-1 follows
Description of "Figure 9-1 Paired Producers and Consumers"

In this situation, UDD1 is a uniform distributed destination composed of two physical members: D1 and D2. Each physical destination has a producer/consumer pair and the effective path for a message follows the solid line from the producer through the destination member to the consumer. If you are using ordering, you should have a producer for each expected Unit-of-Order. See Using Unit-of-Order with Distributed Destinations.

Maximizing Availability

This section provides information on how to maximize message availability.

Using Queues

Ideally, its best to pair a producer with a consumer but it is not always practical. The rate that messages are consumed is the limiting factor that determines the message throughput of your application. You can increase the availability of consumers by using load balancing between member destinations. In this situation, consumers are not paired with a producer as the UDD load balances an incoming message to the next available consumer using the assigned load balancing algorithm.

Note:

Some combinations of Unit-of-Order features can result in the starvation of competing Unit-of-Order message streams, including the under utilization of resources when the number of consumers exceed the number of in-flight messages with different Unit-of-Order names. You will need to test your applications under maximum loads to optimize your system's performance and eliminate conditions that under utilize resources.

Using Topics

When using a distributed topic, every member destination will forward its messages to every other member of the distributed topic.

Figure 9-2 Using Distributed Topics

Description of Figure 9-2 follows
Description of "Figure 9-2 Using Distributed Topics"

In this situation, UDD1 is a uniform distributed destination composed of two physical members: D1 and D2. Each physical destination has a producer/consumer pair. Each consumer receives messages sent by Producer 1 and Producer 2.

Stuck Messages

In this situation, a producer is sending messages to one member of a UDD but there is no consumer available to get the message. This typically happens as a producer sends a message to one of the destinations (D1) and a consumer is listening for messages on another destination (D2).

UDD1 is a uniform distributed destination composed of two physical members: D1 and D2. D1 has a producer and D2 has a consumer. Avoid this configuration by using producer/consumer pairs or by configuring forwarding on the destination.