File-Based Persistence

By default, Message Queue uses a file-based data store, in which individual files store persistent data (such as messages, destinations, durable subscriptions, transactions, and routing information).

The file-based data store is located in a directory identified by the name of the broker instance (instanceName) to which the data store belongs:

Each destination on the broker has its own subdirectory holding messages delivered to that destination.

File-Based Persistence Properties

Broker configuration properties related to file-based persistence are listed under File-Based Persistence Properties. These properties let you configure various aspects of how the file-based data store behaves.

All persistent data other than messages is stored in separate files: one file each for destinations, durable subscriptions, and transaction state information. Most messages are stored in a single file consisting of variable-size records. You can compact this file to alleviate fragmentation as messages are added and removed (see Managing Physical Destination Disk Utilization). In addition, messages above a certain threshold size are stored in their own individual files rather than in the variable-sized record file. You can configure this threshold size with the broker property imq.persist.file.message.max_record_size.

The broker maintains a file pool for these individual message files: instead of being deleted when it is no longer needed, a file is returned to the pool of free files in its destination directory so that it can later be reused for another message. The broker property imq.persist.file.destination.message.filepool.limit specifies the maximum number of files in the pool. When the number of individual message files for a destination exceeds this limit, files will be deleted when no longer needed instead of being returned to the pool.

When returning a file to the file pool, the broker can save time at the expense of storage space by simply tagging the file as available for reuse without deleting its previous contents. You can use the imq.persist.file.message.filepool.cleanratio broker property to specify the percentage of files in each destination’s file pool that should be maintained in a “clean” (empty) state rather than simply marked for reuse. The higher you set this value, the less space will be required for the file pool, but the more overhead will be needed to empty the contents of files when they are returned to the pool. If the broker’s imq.persist.file.message.cleanup property is true, all files in the pool will be emptied at broker shutdown, leaving them in a clean state; this conserves storage space but slows down the shutdown process.

In writing data to the data store, the operating system has some leeway in whether to write the data synchronously or “lazily” (asynchronously). Lazy storage can lead to data loss in the event of a system crash, if the broker believes the data to have been written to the data store when it has not. To ensure absolute reliability (at the expense of performance), you can require that all data be written synchronously by setting the broker property imq.persist.file.sync.enabled to true. In this case, the data is guaranteed to be available when the system comes back up after a crash, and the broker can reliably resume operation.

Configuring a File-Based Data Store

A file-based data store is automatically created when you create a broker instance. However, you can configure the data store using the properties described in File-Based Persistence Properties.

For example, by default, Message Queue performs asynchronous write operations to disk. However, to attain the highest reliability, you can set the broker property imq.persist.file.sync to write data synchronously instead. See Table 17-6.

When you start a broker instance, you can use the imqbrokerd command’s -- reset option to clear the file-based data store. For more information about this option and its suboptions, see Broker Utility.

Securing a File-Based Data Store

The persistent data store can contain, among other information, message files that are being temporarily stored. Since these messages may contain proprietary information, it is important to secure the data store against unauthorized access. This section describes how to secure data in a file-based data store.

A broker using file-based persistence writes persistent data to a flat-file data store:

where instanceName is a name identifying the broker instance. This directory is created when the broker instance is started for the first time. The procedure for securing this directory depends on the operating system platform on which the broker is running:

• On Solaris and Linux, the directory’s permissions are determined by the file mode creation mask (umask) of the user who started the broker instance. Hence, permission to start a broker instance and to read its persistent files can be restricted by setting the mask appropriately. Alternatively, an administrator (superuser) can secure persistent data by setting the permissions on the instances directory to 700.

• On Windows, the directory’s permissions can be set using the mechanisms provided by the Windows operating system. This generally involves opening a Properties dialog for the directory.

Optimizing File-Based Transaction Persistence

Because many activities can occur during a transaction, persisting a transaction's state over the complete life cycle of the transaction can adversely affect overall performance, especially when the imq.persist.file.sync.enabled property is set to true to avoid data loss in case of a system crash.

Message Queue provides a transaction logging mechanism that can improve performance of transaction persistence. This transaction log offers tuning parameters that can improve performance of file-based persistence for other objects, such as message payloads.

To enable this transaction logging mechanism, set the imq.persist.file.newTxnLogenabled broker property to true.

After enabling the transaction log, essential changes to the state of a JMS transaction are written to the transaction log. When the transaction is committed, all details regarding it are gathered and written to the persistent store. Additionally, the logging mechanism periodically performs a “checkpoint” operation to ensure that the persistent store and the transaction log are synchronized and that the log size remains manageable.

As a further refinement, the operation of the logging mechanism is subject to the value of the imq.persist.file.sync.enabled broker property:

• When imq.persist.file.sync.enabled is true, write operations to the transaction log are written synchronously to disk. Non-transacted message and non-transacted message acknowledgements are also written synchronously to the transaction log before being written asynchronously to the persistent store.

• When imq.persist.file.sync.enabled is false, write operations to the transaction log are written asynchronously to disk. Non-transacted message and non-transacted message acknowledgements are not written to the transaction log.

The tuning parameters supported by the transaction logging mechanism are:

• imq.persist.file.txnLog.groupCommit

• imq.persist.file.txnLog.logNonTransactedMsgSend

• imq.persist.file.txnLog.logNonTransactedMsgAck

Information about these parameters can be found in Table 17-7.