Sun Java System Message Queue 4.3 Administration Guide

Chapter 8 Configuring Persistence Services

For a broker to recover in case of failure, it needs to re-create the state of its message delivery operations. To do this, the broker must save state information to a persistent data store. When the broker restarts, it uses the saved data to re-create destinations and durable subscriptions, recover persistent messages, roll back open transactions, and rebuild its routing table for undelivered messages. It can then resume message delivery.

A persistent data store is thus a key aspect of providing for reliable message delivery. This chapter describes the two different persistence implementations supported by the Message Queue broker and how to set each of them up:

Introduction to Persistence Services

A broker’s persistent data store holds information about physical destinations, durable subscriptions, messages, transactions, and acknowledgments.

Message Queue supports both file-based and JDBC-based persistence modules, as shown in the following figure. File-based persistence uses individual files to store persistent data; JDBC-based persistence uses the Java Database Connectivity (JDBC) interface to connect the broker to a JDBC-based data store. While file-based persistence is generally faster than JDBC-based persistence, some users prefer the redundancy and administrative control provided by a JDBC database. The broker configuration property imq.persist.store (see Table 16–4) specifies which of the two persistence modules (file or jdbc) to use.

Figure 8–1 Persistent Data Stores

Diagram showing that persistence services use either
a flat file-based or a JDBC-based data store.

Message Queue brokers are configured by default to use a file-based persistent store, but you can reconfigure them to plug in any data store accessible through a JDBC-compliant driver. The broker configuration property imq.persist.store (see Table 16–4) specifies which of the two forms of persistence to use.

File-Based Persistence

By default, Message Queue uses a file-based data store, in which individual files store persistent data (such as messages, destinations, durable subscriptions, transactions, and routing information).

The file-based data store is located in a directory identified by the name of the broker instance (instanceName) to which the data store belongs:

   …/instances/instanceName/fs370

(See Appendix A, Platform-Specific Locations of Message Queue Data for the location of the instances directory.) Each destination on the broker has its own subdirectory holding messages delivered to that destination.


Note –

Because the data store can contain messages of a sensitive or proprietary nature, you should secure the …/instances/instanceName/fs370 directory against unauthorized access; see Securing a File-Based Data Store.


File-Based Persistence Properties

Broker configuration properties related to file-based persistence are listed under File-Based Persistence Properties. These properties let you configure various aspects of how the file-based data store behaves.

All persistent data other than messages is stored in separate files: one file each for destinations, durable subscriptions, and transaction state information. Most messages are stored in a single file consisting of variable-size records. You can compact this file to alleviate fragmentation as messages are added and removed (see Managing Physical Destination Disk Utilization). In addition, messages above a certain threshold size are stored in their own individual files rather than in the variable-sized record file. You can configure this threshold size with the broker property imq.persist.file.message.max_record_size.

The broker maintains a file pool for these individual message files: instead of being deleted when it is no longer needed, a file is returned to the pool of free files in its destination directory so that it can later be reused for another message. The broker property imq.persist.file.destination.message.filepool.limit specifies the maximum number of files in the pool. When the number of individual message files for a destination exceeds this limit, files will be deleted when no longer needed instead of being returned to the pool.

When returning a file to the file pool, the broker can save time at the expense of storage space by simply tagging the file as available for reuse without deleting its previous contents. You can use the imq.persist.file.message.filepool.cleanratio broker property to specify the percentage of files in each destination’s file pool that should be maintained in a “clean” (empty) state rather than simply marked for reuse. The higher you set this value, the less space will be required for the file pool, but the more overhead will be needed to empty the contents of files when they are returned to the pool. If the broker’s imq.persist.file.message.cleanup property is true, all files in the pool will be emptied at broker shutdown, leaving them in a clean state; this conserves storage space but slows down the shutdown process.

In writing data to the data store, the operating system has some leeway in whether to write the data synchronously or “lazily” (asynchronously). Lazy storage can lead to data loss in the event of a system crash, if the broker believes the data to have been written to the data store when it has not. To ensure absolute reliability (at the expense of performance), you can require that all data be written synchronously by setting the broker property imq.persist.file.sync.enabled to true. In this case, the data is guaranteed to be available when the system comes back up after a crash, and the broker can reliably resume operation.

Configuring a File-Based Data Store

A file-based data store is automatically created when you create a broker instance. However, you can configure the data store using the properties described in File-Based Persistence Properties.

For example, by default, Message Queue performs asynchronous write operations to disk. However, to attain the highest reliability, you can set the broker property imq.persist.file.sync to write data synchronously instead. See Table 16–5.

When you start a broker instance, you can use the imqbrokerd command’s -- reset option to clear the file-based data store. For more information about this option and its suboptions, see Broker Utility.

Securing a File-Based Data Store

The persistent data store can contain, among other information, message files that are being temporarily stored. Since these messages may contain proprietary information, it is important to secure the data store against unauthorized access. This section describes how to secure data in a file-based data store.

A broker using file-based persistence writes persistent data to a flat-file data store whose location is platform-dependent (see Appendix A, Platform-Specific Locations of Message Queue Data):

   …/instances/instanceName/fs370

where instanceName is a name identifying the broker instance. This directory is created when the broker instance is started for the first time. The procedure for securing this directory depends on the operating system platform on which the broker is running:

JDBC-Based Persistence

Instead of using a file-based data store, you can set up a broker to access any data store accessible through a JDBC-compliant driver. This involves setting the appropriate JDBC-related broker configuration properties and using the Database Manager utility (imqdbmgr) to create the proper database schema. See Configuring a JDBC-Based Data Store for specifics.

JDBC-Based Persistence Properties

The full set of properties for configuring a broker to use a JDBC database are listed in Table 16–6. You can specify these properties either in the instance configuration file (config.properties) of each broker instance or by using the -D command line option to the Broker utility (imqbrokerd) or the Database Manager utility (imqdbmgr).

In practice, however, JDBC properties are preconfigured by default, depending on the database vendor being used for the data store. The property values are set in the default.properties file, and only need to be explicitly set if you are overriding the default values. In general, you only need to set the following properties:


Example 8–1 Broker Properties for MySQL Database


imq.persist.store=jdbc
imq.persist.jdbc.dbVendor=mysql
imq.persist.jdbc.mysql.user=userName
imq.persist.jdbc.mysql.password=password
imq.persist.jdbc.mysql.property.url=jdbc:mysql://hostName:port/dataBase

If you expect to have messages that are larger than 1 MB, configure MySQL's max_allowed_packet variable accordingly when starting the database. For more information see Appendix B of the MySQL 5.0 Reference Manual.



Example 8–2 Broker Properties for HADB Database


imq.persist.store=jdbc
imq.persist.jdbc.dbVendor=hadb
imq.persist.jdbc.hadb.user=userName
imq.persist.jdbc.hadb.password=password
imq.persist.jdbc.hadb.property.serverlist=hostName:port,hostName:port,...

You can obtain the server list using the hadbm get jdbcURL command.


In addition, in an enhanced broker cluster, in which a JDBC database is shared by multiple broker instances, each broker must be uniquely identified in the database (unnecessary for an embedded database, which stores data for only one broker instance). The configuration property imq.brokerid specifies a unique instance identifier to be appended to the names of database tables for each broker. See Enhanced Broker Cluster Properties.

After setting all of the broker’s needed JDBC configuration properties, you must also install your JDBC driver’s .jar file in the appropriate directory location, depending on your operating-system platform (as listed in Appendix A, Platform-Specific Locations of Message Queue Data) and then create the database schema for the JDBC-based data store (see To Set Up a JDBC-Based Data Store).

Configuring a JDBC-Based Data Store

To configure a broker to use a JDBC database, you set JDBC-related properties in the broker’s instance configuration file and create the appropriate database schema. The Message Queue Database Manager utility (imqdbmgr) uses your JDBC driver and the broker configuration properties to create the schema and manage the database. You can also use the Database Manager to delete corrupted tables from the database or if you want to use a different database as a data store. See Database Manager Utility for more information.


Note –

If you use an embedded database, it is best to create it under the following directory:

   
.../instances/instanceName/dbstore/databaseName

If an embedded database is not protected by a user name and password, it is probably protected by file system permissions. To ensure that the database is readable and writable by the broker, the user who runs the broker should be the same user who created the embedded database using the imqdbmgr command.


ProcedureTo Set Up a JDBC-Based Data Store

  1. Set JDBC-related properties in the broker’s instance configuration file.

    The relevant properties are discussed, with examples, in JDBC-Based Persistence Properties and listed in full in Table 16–6. In particular, you must specify a JDBC-based data store by setting the broker’s imq.persist.store property to jdbc.

  2. Place a copy of, or a symbolic link to, your JDBC driver’s .jar file in the Message Queue external resource files directory, depending on your platform (see Appendix A, Platform-Specific Locations of Message Queue Data):

      Solaris: /usr/share/lib/imq/ext


      Linux: /opt/sun/mq/share/lib/ext


      AIX: IMQ_VARHOME/lib/ext


      Windows: IMQ_VARHOME\lib\ext


    For example, if you are using HADB on a Solaris system, the following command copies the driver’s .jar file to the appropriate location:

       
    cp  /opt/SUNWhadb/4/lib/hadbjdbc4.jar  /usr/share/lib/imq/ext
    

    The following command creates a symbolic link instead:

       
    ln  -s  /opt/SUNWhadb/4/lib/hadbjdbc4.jar  /usr/share/lib/imq/ext
    
  3. Create the database schema needed for Message Queue persistence.

    Use the imqdbmgr create all command (for an embedded database) or the imqdbmgr create tbl command (for an external database); see Database Manager Utility.

    1. Change to the directory where the Database Manager utility resides, depending on your platform:

        Solaris: cd /usr/bin


        Linux: cd /opt/sun/mq/bin


        AIX: cd IMQ_HOME/bin


        Windows: cd IMQ_HOME\bin


    2. Enter the imqdbmgr command:

         
      imqdbmgr create all
      

ProcedureTo Display Information About a JDBC-Based Data Store

You can display information about a JDBC-based data store using the Database Manager utility (imqdbmgr) as follows:

  1. Change to the directory where the Database Manager utility resides, depending on your platform:

      Solaris: cd /usr/bin


      Linux: cd /opt/sun/mq/bin


      AIX: cd IMQ_HOME/bin


      Windows: cd IMQ_HOME\bin


  2. Enter the imqdbmgr command:

       
    imqdbmgr query
    

    The output should resemble the following

    dbmgr query
    
    [04/Oct/2005:15:30:20 PDT] Using plugged-in persistent store:
            version=400
            brokerid=Mozart1756
            database connection url=jdbc:oracle:thin:@Xhome:1521:mqdb
            database user=scott
    Running in standalone mode.
    Database tables have already been created.

Securing a JDBC-Based Data Store

The persistent data store can contain, among other information, message files that are being temporarily stored. Since these messages may contain proprietary information, it is important to secure the data store against unauthorized access. This section describes how to secure data in a JDBC-based data store.

A broker using JDBC-based persistence writes persistent data to a JDBC-compliant database. For a database managed by a database server (such as Oracle), it is recommended that you create a user name and password to access the Message Queue database tables (tables whose names start with MQ). If the database does not allow individual tables to be protected, create a dedicated database to be used only by Message Queue brokers. See the documentation provided by your database vendor for information on how to create user name/password access.

The user name and password required to open a database connection by a broker can be provided as broker configuration properties. However it is more secure to provide them as command line options when starting up the broker, using the imqbrokerd command’s -dbuserand -dbpassword options (see Broker Utility).

For an embedded database that is accessed directly by the broker by means of the database’s JDBC driver, security is usually provided by setting file permissions on the directory where the persistent data will be stored, as described above under Securing a File-Based Data Store To ensure that the database is readable and writable by both the broker and the Database Manager utility, however, both should be run by the same user.

Data Store Formats

Changes in the file formats for both file-based and JDBC-based persistent data stores were introduced in Message Queue 3.7, with further JDBC changes in version 4.0 and 4.1. As a result of these changes, the persistent data store version numbers have been updated to 370 for file-based data stores and 410 for JDBC-based stores. You can use the imqdbmgr query command to determine the version number of your existing data store.

On first startup, the Message Queue Broker utility (imqbrokerd) will check for the presence of an older persistent data store and automatically migrate it to the latest format:

The upgrade leaves the older copy of the persistent data store intact, allowing you to roll back the upgrade if necessary. To do so, you can uninstall the current version of Message Queue and reinstall the earlier version you were previously running. The older version’s message brokers will locate and use the older copy of the data store.