1. High Availability Guide
  2. Creating a High Availability Environment
  3. JMS and JTA High Availability

8 JMS and JTA High Availability

To configure Java Message Service (JMS) and Java Transaction API (JTA) services for high availability, you deploy them to migratable targets that can migrate from one server in a cluster to another server.
  • About JMS and JTA Services for High Availability
    Java Message Service (JMS) is an application program interface (API) that supports the formal communication known as messaging between computers in a network.
  • About Migratable Targets for JMS and JTA Services
    To configure JMS and JTA services for high availability, you deploy them to a migratable target, a special target that can migrate from one server in a cluster to another.
  • Configuring Migratable Targets for JMS and JTA High Availability
    To configure a migratable target, you specify servers that can host a target; only one server can host a migratable target at any one time. You also set the host you prefer for services and back up servers if the preferred host fails.
  • User-Preferred Servers and Candidate Servers
    When you deploy a JMS service to a migratable target, you can select a user-preferred server target to host the service. You can also specify constrained candidate servers (CCS) that can host a service if the user-preferred server fails.
  • Using File Persistence
    Oracle recommends storing the JMS and JTA data in the database for higher reliability, storage in cloud environments, and better consistency in disaster recovery scenarios instead of using file persistence. For example, use the JDBC Store and TLOG-in-DB features for JMS and JTA respectively. If you choose to use a file system, you must use a shared file system for high availability.
  • Using File Stores on NFS
    If you store JMS messages and transaction logs on an NFS-mounted directory, Oracle strongly recommends that you verify server restart behavior after an abrupt machine failure. Depending on the NFS implementation, different issues can arise after a failover/restart.
  • Configuring WLS JMS with a Database Persistent Store
    You can change WLS JMS configuration from a file-based persistent store (default configuration) to a database persistent store.
  • Configuring Database Stores to Persist Transaction Logs
    After you confirm that your setup has a standard Oracle Fusion Middleware installation, you can configure JDBC Transaction Logs (TLOG) Stores.
  • Using the Config Wizard for configuring Automatic Service Migration and JDBC Persistent stores for FMW components
    You can use the HA Options screen in the Configuration Wizard to automate the JDBC store persistence and configure service migration. This screen appears for the first time when you create a Fusion Middleware cluster that may use Automatic Service Migration, persistent stores, or both, and all subsequent clusters that are added to the domain by using the Configuration Wizard, automatically apply the selected HA options.

Parent topic: Creating a High Availability Environment

About JMS and JTA Services for High Availability

Java Message Service (JMS) is an application program interface (API) that supports the formal communication known as messaging between computers in a network.

Java Transaction API (JTA) specifies standard Java interfaces between a transaction manager and parties involved in a distributed transaction system: the resource manager, the application server, and the transactional applications.

In WebLogic JMS, a message is available only if its host JMS server for the destination is running. If a message is in a central persistent store, the only JMS server that can access the message is the server that originally stored the message. WebLogic has features to restart and/or migrate a JMS server automatically after failures. It also has features for clustering (distributing) a destination across multiple JMS servers within the same cluster.

You automatically restart and / or migrate (fail over) JMS Servers using either Whole Server Migration or Automatic Service Migration.

Note:

For more on working with JMS or JTA, see:

Note:

For more on Whole Server Migration, see Whole Server Migration.

Parent topic: JMS and JTA High Availability

About Migratable Targets for JMS and JTA Services

To configure JMS and JTA services for high availability, you deploy them to a migratable target, a special target that can migrate from one server in a cluster to another.

A migratable target groups migratable services that should move together. When a migratable target migrates, all services that it hosts also migrate.

A migratable target specifies a set of servers that can host a target. Only one server can host a migratable target at any one time. It can also specify:

  • A user-preferred host for services

  • An ordered list of backup servers if a preferred server fails

After you configure a service to use a migratable target, it is independent from the server member that currently hosts it. For example, if you configure a JMS server with a deployed JMS queue to use a migratable target, the queue is independent of when a specific server member is available. The queue is always available when any server in the cluster hosts the migratable target.

You can migrate pinned migratable services manually from one server to another in the cluster if 1) a server fails, or 2) as part of scheduled maintenance. If you do not configure a migratable target in the cluster, migratable services can migrate to any server in the cluster.

See Configuring Migratable Targets for JMS and JTA High Availability to configure migratable targets.

Note:

For more on administering JMS, see the following topics in Oracle Fusion Middleware Administering JMS Resources for Oracle WebLogic Server:

Parent topic: JMS and JTA High Availability

Configuring Migratable Targets for JMS and JTA High Availability

To configure a migratable target, you specify servers that can host a target; only one server can host a migratable target at any one time. You also set the host you prefer for services and back up servers if the preferred host fails.

To configure migratable targets, see these topics in Administration Console Online Help:

Parent topic: JMS and JTA High Availability

User-Preferred Servers and Candidate Servers

When you deploy a JMS service to a migratable target, you can select a user-preferred server target to host the service. You can also specify constrained candidate servers (CCS) that can host a service if the user-preferred server fails.

If a migratable target doesn't specify a CCS, you can migrate the JMS server to any available server in the cluster.

You can create separate migratable targets for JMS services so that you can always keep each service running on a different server in the cluster, if necessary. Conversely, you can configure the same selection of servers as the CCSs for both JTA and JMS, to ensure that services stay co-located on the same server in the cluster.

Parent topic: JMS and JTA High Availability

Using File Persistence

Oracle recommends storing the JMS and JTA data in the database for higher reliability, storage in cloud environments, and better consistency in disaster recovery scenarios instead of using file persistence. For example, use the JDBC Store and TLOG-in-DB features for JMS and JTA respectively. If you choose to use a file system, you must use a shared file system for high availability.

WebLogic supplies multiple types of file stores which have multiple purposes as follows:
  • Each WebLogic Server has a default file store. However, default file stores should not be used for storing critical data such as JMS messages, JTA transactions, and EJB timers. You should use JDBC stores, TLOG-in-DB, and database stored timers instead. An example of non-critical data that is stored in a default store is application life-cycle state, such as whether a particular application has been administratively paused. If there is no critical data in a default file store, it is safe to delete such stores in the event of a catastrophic corruption as this mitigates the risk of disabling file locking for the default file store.
  • WebLogic JMS paging stores are active if JMS has a large message backlog. The data in JMS paging files is not reloaded when the server is not running, and so the files can be safely deleted when a WebLogic Server is not running. The corresponding persistent messages are simultaneously stored in default file stores, custom file stores, or custom JDBC stores.
  • WebLogic diagnostic stores contain non-critical diagnostic data. They are run within a buffering mode that allows for very high performance in order to minimize the overhead of diagnostics, but this increases the risk of corruption after a failure. If such files become corrupt, then it is safe for WebLogic Servers to reboot, and it is also safe to delete the files.

See Tuning the WebLogic Persistent Store section in Tuning Performance of Oracle WebLogic Server.

Parent topic: JMS and JTA High Availability

Using File Stores on NFS

If you store JMS messages and transaction logs on an NFS-mounted directory, Oracle strongly recommends that you verify server restart behavior after an abrupt machine failure. Depending on the NFS implementation, different issues can arise after a failover/restart.

Parent topic: JMS and JTA High Availability

Prerequisites for Disabling File Locking

All file stores are locked by default. The WebLogic file locking feature is designed to help prevent severe file corruptions that can occur in concurrency scenarios. Perform the following prerequisite tasks to mitigate the risk of disabling file locks:
  • If the server using the file store is configured for server migration, always configure the database-based cluster leasing option instead of the default consensus leasing. This enforces additional locking mechanisms using database tables and prevents automated concurrent restart of more than one instance of a particular WebLogic Server.
  • Avoid disabling file locks on a file store that is using Automatic Service Migration (ASM).
    • ASM requires file store locking to work safely and is activated in the following scenarios:
      1. A custom file store target is set to a Migratable Target and the Migratable Target is configured with a Migration Policy other than 'manual' (the default).
      2. A custom file store target is set to a WebLogic cluster when the store is configured with a Migration Policy other than 'Off' (the default).
      3. A WebLogic Server is configured with a JTA Migratable Target with a Migration Policy value other than ‘manual’ (the default), as this in turn can lead to default file store migrations.
    • If both ASM and disabling file locks are required, store your critical data in database stores instead of file stores to avoid the risk of file corruptions. For example, use a custom JDBC store instead of a file store and configure JTA to use a JDBC TLOG store instead of each WebLogic Server’s default file store.
  • Additional procedural precautions must be implemented to avoid any human error and to ensure that only one instance of a server is manually started at any given point in time. Similarly, precautions must be taken to ensure that no two domains have a store with the same name that references the same directory.

Parent topic: Using File Stores on NFS

Disabling File Locking for all Stores Using a System Property

Starting from WebLogic Server 14.1.2 release, you can specify a Java system property on the weblogic.Server command line or start script of the JVM to disable locking on all of its file stores including default, paging, diagnostic, and custom file stores as shown below:

"-Dweblogic.store.file.LockEnabled=false"

Parent topic: Using File Stores on NFS

Verifying Server Restart Behavior

To verify server restart behavior, abruptly shut down the node that hosts WebLogic servers while the servers are running.

  • If you configured the server for server migration, it should start automatically in failover mode after the failover period.

  • If you did not configure the server for server migration, you can manually restart the WebLogic Server on the same host after the node completely reboots.

If WebLogic Server doesn't restart after abrupt machine failure, review server log files to verify whether or not it is due to an I/O exception similar to the following:

<MMM dd, yyyy hh:mm:ss a z> <Error> <Store> <BEA-280061> <The persistent 
store "_WLS_server_1" could not be deployed: 
weblogic.store.PersistentStoreException: java.io.IOException: 
[Store:280021]There was an error while opening the file store file 
"_WLS_SERVER_1000000.DAT" 
weblogic.store.PersistentStoreException: java.io.IOException: 
[Store:280021]There was an error while opening the file store file 
"_WLS_SERVER_1000000.DAT" 
at weblogic.store.io.file.Heap.open(Heap.java:168) 
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:88)
...
java.io.IOException: Error from fcntl() for file locking, Resource
temporarily unavailable, errno=11

This error occurs when the NFSv3 system doesn't release locks on file stores. WebLogic Server maintains locks on files that store JMS data and transaction logs to prevent data corruption that can occur if you accidentally start two instances of the same Managed Server. Because the NFSv3 storage device doesn't track lock owners, NFS holds the lock indefinitely if a lock owner fails. As a result, after abrupt machine failure followed by a restart, subsequent attempts by WebLogic Server to acquire locks may fail.

How you resolve this error depends on your NFS environment: (See Oracle Fusion Middleware Release Notes for updates on this topic.)

  • For NFSv4 environments, you can set a tuning parameter on the NAS server to release locks within the approximate time required to complete server migration; you don't need to follow procedures in this section. See your storage vendor's documentation for information on locking files stored in NFS-mounted directories on the storage device, and test the results.

  • For NFSv3 environments, the following sections describe how to disable WebLogic file locking mechanisms for: the default file store, a custom file store, a JMS paging file store, a diagnostics file store.

WARNING:

NFSv3 file locking prevents severe file corruptions that occur if more than one Managed Server writes to the same file store at any point in time.

If you disable NFSv3 file locking, you must implement administrative procedures /policies to ensure that only one Managed Server writes to a specific file store. Corruption can occur with two Managed Servers in the same cluster or different clusters, on the same node or different nodes, or on the same domain or different domains.

Your policies could include: never copy a domain, never force a unique naming scheme of WLS-configured objects (servers, stores), each domain must have its own storage directory, no two domains can have a store with the same name that references the same directory.

When you use a file store, always configure the database-based leasing option for server migration. This option enforces additional locking mechanisms using database tables and prevents automated restart of more than one instance of a particular Managed Server.

Parent topic: Using File Stores on NFS

Disabling File Locking for the Default File Store

If WebLogic Server doesn't restart after abrupt machine failure and server log files show the NFS system doesn't release locks on file stores, you can disable file locking.

To disable file locking for the default file store using the Administration Console:

  1. If necessary, click Lock & Edit in the Change Center.
  2. In the Domain Structure tree, expand the Environment node and select Servers.
  3. In the Summary of Servers list, select the server you want to modify.
  4. Select the Configuration > Services tab.
  5. Scroll down to the Default Store section and click Advanced.
  6. Scroll down and deselect the Enable File Locking check box.
  7. Click Save. If necessary, click Activate Changes in the Change Center.
  8. Restart the server you modified for the changes to take effect.

The resulting config.xml entry looks like the following: 

<server>
 <name>examplesServer</name>
 ...
 <default-file-store>
 <synchronous-write-policy>Direct-Write</synchronous-write-policy>
 <io-buffer-size>-1</io-buffer-size>
 <max-file-size>1342177280</max-file-size>
 <block-size>-1</block-size>
 <initial-size>0</initial-size>
 <file-locking-enabled>false</file-locking-enabled>
 </default-file-store>
 </server>

Parent topic: Using File Stores on NFS

Disabling File Locking for a Custom File Store

If WebLogic Server doesn't restart after abrupt machine failure and server log files show the NFS system doesn't release locks on custom file stores, you can disable file locking.

To disable file locking for a custom file store using the Administration Console:

  1. If necessary, click Lock & Edit in the Change Center to get an Edit lock for the domain.
  2. In the Domain Structure tree, expand the Services node. Select Persistent Stores.
  3. In the Summary of Persistent Stores list, select the custom file store you want to modify.
  4. On the Configuration tab for the custom file store, click Advanced.
  5. Scroll down and deselect the Enable File Locking check box.
  6. Click Save. If necessary, click Activate Changes in the Change Center.
  7. If the custom file store was in use, you must restart the server for changes to take effect.

The config.xml entry looks like the following example: 

<file-store>
 <name>CustomFileStore-0</name>
 <directory>C:\custom-file-store</directory>
 <synchronous-write-policy>Direct-Write</synchronous-write-policy>
 <io-buffer-size>-1</io-buffer-size>
 <max-file-size>1342177280</max-file-size>
 <block-size>-1</block-size>
 <initial-size>0</initial-size>
 <file-locking-enabled>false</file-locking-enabled>
 <target>examplesServer</target>
 </file-store>

Parent topic: Using File Stores on NFS

Disabling File Locking for a JMS Paging File Store

If WebLogic Server doesn't restart after abrupt machine failure and server log files show the NFS system doesn't release locks on JMS paging file stores, you can disable file locking.

To disable file locking for a JMS paging file store using the Administration Console:

  1. If necessary, click Lock & Edit in the Change Center to get an Edit lock for the domain.
  2. In the Domain Structure tree, expand the Services node, expand the Messaging node, and select JMS Servers.
  3. In the Summary of JMS Servers list, select a JMS server to modify.
  4. On the Configuration > General tab for the JMS Server, scroll down. Deselect the Paging File Locking Enabled check box.
  5. Click Save. If necessary, click Activate Changes in the Change Center.
  6. Restart the server you modified for changes to take effect.

The config.xml file entry looks like the following example: 

 <jms-server>
 <name>examplesJMSServer</name>
 <target>examplesServer</target>
 <persistent-store>exampleJDBCStore</persistent-store>
 ...
 <paging-file-locking-enabled>false</paging-file-locking-enabled>
 ...
 </jms-server>

Parent topic: Using File Stores on NFS

Disabling File Locking for Diagnostics File Stores

If WebLogic Server doesn't restart after abrupt machine failure and server log files show the NFS system doesn't release locks on diagnostics paging file stores, you can disable file locking.

To disable file locking for a Diagnostics file store using the Administration Console:

  1. If necessary, click Lock & Edit in the Change Center to get an Edit lock for the domain.
  2. In the Domain Structure tree, expand the Diagnostics node. Select Archives.
  3. In the Summary of Diagnostic Archives list, select the server name of the archive that you want to modify.
  4. On the Settings for [server_name] page, deselect the Diagnostic Store File Locking Enabled check box.
  5. Click Save. If necessary, click Activate Changes in the Change Center.
  6. Restart the server you modified for the changes to take effect.

The resulting config.xml file looks like this: 

 <server>
 <name>examplesServer</name>
 ...
 <server-diagnostic-config>
 <diagnostic-store-dir>data/store/diagnostics</diagnostic-store-dir>
 <diagnostic-store-file-locking-enabled>false</diagnostic-store-file-locking-
enabled>
 <diagnostic-data-archive-type>FileStoreArchive</diagnostic-data-archive-type>
 <data-retirement-enabled>true</data-retirement-enabled>
 <preferred-store-size-limit>100</preferred-store-size-limit>
 <store-size-check-period>1</store-size-check-period>
 </server-diagnostic-config>
 </server>

Note:

See Configure JMS Servers and Persistent Stores in Oracle Fusion Middleware Administering JMS Resources for Oracle WebLogic Server.

Parent topic: Using File Stores on NFS

Configuring WLS JMS with a Database Persistent Store

You can change WLS JMS configuration from a file-based persistent store (default configuration) to a database persistent store.

Parent topic: JMS and JTA High Availability

About the Persistent Store

The persistent store is a built-in storage solution for WebLogic Server subsystems and services that require persistence. For example, it can store persistent JMS messages.

The persistent store supports persistence to a file-based store or to a JDBC-accessible store in a database. For information on the persistent store, see The WebLogic Persistent Store in Administering the WebLogic Server Persistent Store.

For information on typical tasks to monitor, control, and configure WebLogic messaging components, see WebLogic Server Messaging in Administering Oracle WebLogic Server with Fusion Middleware Control.

Parent topic: Configuring WLS JMS with a Database Persistent Store

Prerequisites for Configuring WLS JMS with a Database Persistent Store

To configure WLS JMS with database persistent stores, verify that your setup meets specific requirements.

Your setup must meet these requirements:

  • An Oracle Fusion Middleware installation with at least one cluster and one or more JMS servers

  • JMS servers that use file persistent stores, the default configuration.

Parent topic: Configuring WLS JMS with a Database Persistent Store

Switching WLS JMS File-Based Persistent Stores to Database Persistent Store

You can swap JMS servers from file-based to database persistent stores.

You must follow steps in this procedure for each JMS server that you must configure to use database persistent stores.

  1. Create a JDBC store. See Using a JDBC Store in Oracle Fusion Middleware Administering Server Environments for Oracle WebLogic Server.

    Note:

    You must specify a prefix to uniquely name the database table for the JDBC store.

  2. Associate the JDBC store with the JMS server:

    1. In the Weblogic Server Administration Console, go to Services->Messaging->JMS Servers.

    2. Verify that there are no pending messages in this server. In the Control tab, stop production and insertion of messages for all destinations and wait for any remaining messages to drain.

    3. Select the General Configuration tab. Under Persistent Store, select the new JDBC store then click Save.

The JMS server starts using the database persistent store.

Parent topic: Configuring WLS JMS with a Database Persistent Store

Configuring Database Stores to Persist Transaction Logs

After you confirm that your setup has a standard Oracle Fusion Middleware installation, you can configure JDBC Transaction Logs (TLOG) Stores.

Parent topic: JMS and JTA High Availability

Requirements for Configuring JDBC TLOG Stores

You must have a standard Oracle Fusion Middleware installation before you configure a JDBC Transaction Logs (TLOG) Store.

Post installation, the TLOG Store is configured in the file system. In some instances, Oracle recommends that you configure TLOGs to store in the database. To configure JDBC TLOGs to stored to a database store, see Using a JDBC TLOG Store in Administering the WebLogic Server Persistent Store.

Parent topic: Configuring Database Stores to Persist Transaction Logs

Configuring JDBC TLOG Stores

There are a few guidelines to follow when you configure JDBC TLOG Stores for Managed Servers in a cluster.

When you configure JDBC TLOG Stores:

  • You must repeat the procedure for each Managed Server in the cluster.

  • Use the Managed Server name as a prefix to create a unique TLOG store name for each Managed Server.

  • Verify that the data source that you created for the persistent store targets the cluster for a high availability setup.

When you finish the configuration, TLOGs are directed to the configured database-based persistent store.

Note:

When you add a new Managed Server to a cluster by scaling up or scaling out, you must also create the corresponding JDBC TLOG Store for the new Managed Server.

Parent topic: Configuring Database Stores to Persist Transaction Logs

Using the Config Wizard for configuring Automatic Service Migration and JDBC Persistent stores for FMW components

You can use the HA Options screen in the Configuration Wizard to automate the JDBC store persistence and configure service migration. This screen appears for the first time when you create a Fusion Middleware cluster that may use Automatic Service Migration, persistent stores, or both, and all subsequent clusters that are added to the domain by using the Configuration Wizard, automatically apply the selected HA options.

If you select the Enable Automatic Service Migration option, it configures migrtable target definitions that are required for automatic service migration. You can either select database or consensus leasing option. If you select Database Leasing, the leasing datasource is also configured.

In the same screen, use the options JTA Transaction Log Persistence and JMS Server Persistence to configure JDBC stores automatically. Fusion Middleware component templates automatically define the JDBC persistent stores for JMS Servers and Transaction Logs.

For information about what to select in the High Availability options screen during the domain creation process, see Configuring High Availability Options .

Parent topic: JMS and JTA High Availability