24 JMS

This chapter describes how to work with Java Message Services (JMS) in Oracle Data Integrator.

This chapter includes the following sections:

24.1 Introduction

Oracle Data Integrator provides a simple and transparent method to integrate JMS destinations. This chapter focuses on processing JMS messages with a text payload in batch mode. For XML payload processing, refer to Chapter 25, "JMS XML".

24.1.1 Concepts

The JMS Knowledge Modules apply to most popular JMS compliant middleware, including Oracle Service Bus, Sonic MQ, IBM Websphere MQ, and so forth. Most of these Knowledge Modules include transaction handling to ensure message delivery.

24.1.1.1 JMS Message Structure

This section describes the structure of a message in a JMS destination.

A JMS Message consists of three sections:

Header

The header contains in the header fields standard metadata concerning the message, including the destination (JMSDestination), Message ID (JMSMessageID), Message Type (JMSType), and so forth.

Properties

The properties section contains additional metadata concerning the message. These metadata are properties, that can be separated in three groups:

  • JMS-Defined properties which are optional JMS Headers. Their name begins with JMSX(JMSXUserID, JMSXAppID, etc.).

  • Provider-specific properties. They are specific to the router vendor. Their names start with JMS_<vendor name>.

  • Application-specific properties. These properties depend on the application sending the messages. These are user-defined information that is not included in the message payload.

The Header and Properties sections provide a set of header fields and properties that:

  • Have a specific Java data type (Boolean, string, short, and so forth),

  • Can be accessed for reading and/or writing,

  • Can be used for filtering on the router through the JMS Selector.

Payload

The payload section contains the message content. This content can be anything (text, XML, binary, and so forth).

24.1.1.2 Using a JMS Destination

Oracle Data Integrator is able to process JMS Text and Byte messages that are delivered by a JMS destination. Each message is considered as a container for rows of data and is handled through the JMS Queue or JMS Topic technology.

With JMS Queue/JMS Topic technologies, each JMS destination is defined similarly to a flat file datastore. Each message in the destination is a record in the datastore.

In the topology, each JMS router is defined as a JMS Topic/Queue data server, with a single physical schema. A JMS router may be defined therefore twice to access its topics using one data server, and its queues using another one.

Each JMS destination (Topic of Queue) is defined as a JMS datastore which resource name matches the name of the JMS destination (name of the queue or topic as defined in the router). A model groups message structures related to different topics or queues.

The JMS datastore structure is defined similarly to a flat file (delimited or fixed width). The properties or header fields of the message can be declared with JMS-specific data types as additional pseudo-columns in this flat file structure. Each message in the destination is processed as a record of a JMS datastore.

Processing Messages

JMS destinations are handled as regular file datastores and messages as rows from these datastores. With these technologies, entire message sets are produced and consumed within each interface.

Message publishing as well consumption requires a commit action to finalize removing/posting the message from/to the JMS destination. Committing is particularly important when reading. Without a commit, the message is read but not consumed. It remains in the JMS Topic/Queue and will be re-read at a later time.

Both the message content and pseudo-columns can be used as regular columns in the integration interfaces (for mapping, filter, etc.). Certain pseudo-columns (such as the one representing the MESSAGE_ID property) are read-only, and some properties of header fields are used (or set) through the Knowledge Module options.

Using Data Integrator you can transfer information either through the message payload - the columns - , or through the properties - pseudo-columns - (application properties, for example).

Using the properties to carry information is restricted by third-party applications producing or consuming the messages.

Filtering Messages

It is possible to filter messages from a JMS destination in two ways:

  • By defining a filter using the datastore's columns and pseudo-columns. In this case Data Integrator performs the filtering operation after consuming the messages. This implies that messages rejected by this filter may also be consumed.

  • By defining a Message Selector (MESSAGE_SELECTOR KM option). This type of filter can only use the properties or header fields of the message. The filter is processed by the router, and only the messages respecting the filter are consumed, reducing the number of messages transferred.

24.1.2 Knowledge Modules

Oracle Data Integrator provides the Knowledge Modules (KM) listed in Table 24-1 for handling JMS messages.

Table 24-1 JMS Knowledge Modules

Knowledge Module Description

IKM SQL to JMS Append

Integrates data into a JMS compliant message queue or topic in text or binary format from any SQL compliant staging area.

Consider using this IKM if you plan to transform and export data to a target JMS queue or topic. If most of your source datastores are located on the same data server, we recommend using this data server as staging area to avoid extra loading phases (LKMs).

To use this IKM, the staging area must be different from the target.

LKM JMS to SQL

Loads data from a text or binary JMS compliant message queue or topic to any SQL compliant database used as a staging area. This LKM uses the Agent to read selected messages from the source queue/topic and write the result in the staging temporary table created dynamically.

To ensure message delivery, the message consumer (or subscriber) does not commit the read until the data is actually integrated into the target by the IKM.

Consider using this LKM if one of your source datastores is a text or binary JMS message.


24.2 Installation and Configuration

Make sure you have read the information in this section before you start using the JMS Knowledge Modules:

24.2.1 System Requirements and Certifications

Before performing any installation you should read the system requirements and certification documentation to ensure that your environment meets the minimum installation requirements for the products you are installing.

The list of supported platforms and versions is available on Oracle Technical Network (OTN):

http://www.oracle.com/technology/products/oracle-data-integrator/index.html.

24.2.2 Technology Specific Requirements

The JMS destinations are usually accessed via a JNDI service. The configuration and specific requirements for JNDI and JMS depends on the JMS Provider you are connecting to. Refer to the JMS Provider specific documentation for more details.

24.2.3 Connectivity Requirements

Oracle Data Integrator does not include specific drivers for JMS providers. Refer to the JMS Provider documentation for the connectivity requirement of this provider.

24.3 Setting up the Topology

Setting up the Topology consists of:

  1. Creating a JMS Data Server

  2. Creating a JMS Physical Schema

24.3.1 Creating a JMS Data Server

A JMS data server corresponds to one JMS provider/router that is accessible through your local network.

It exists two types of JMS data servers: JMS Queue and JMS Topic.

  • A JMS Queue data server is used to access several queues in the JMS router.

  • A JMS Topic data server is used to access several topics in the JMS router

24.3.1.1 Creation of the Data Server

Create a data server either for the JMS Queue technology or for the JMS Topic technology using the standard procedure, as described in "Creating a Data Server" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator. This section details only the fields required or specific for defining a JMS Queue or JMS Topic data server.

  1. In the Definition tab:

    • Name: Name of the data server as it will appear in Oracle Data Integrator.

    • User/Password: Not used here. Leave these fields empty.

  2. In the JNDI tab:

    • JNDI Authentication: Set this field to None.

    • JNDI User: Enter the username to connect to the JNDI directory (optional step).

    • Password: This user's password (optional step).

    • JNDI Protocol: From the list, select the JNDI protocol (optional step).

    • JNDI Driver: Name of the initial context factory java class to connect to the JNDI provider, for example: com.sun.jndi.ldap.LdapCtxFactory for LDAP

    • JNDI URL: <JMS_RESOURCE>, for example ldap://<host>:<port>/<dn> for LDAP

    • JNDI Resource: Logical name of the JNDI resource corresponding to your JMS Queue or Topic connection factory.

      For example, specify QueueConnectionFactory if you want to access a message queue and TopicConnectionFactory if you want to access a topic. Note that these parameters are specific to the JNDI directory and the provider.

24.3.2 Creating a JMS Physical Schema

Create a JMS physical schema using the standard procedure, as described in "Creating a Physical Schema" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator.

Note:

Only one physical schema is required per JMS data server.

Create for this physical schema a logical schema using the standard procedure, as described in "Creating a Logical Schema" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator and associate it in a given context.

24.4 Setting Up an Integration Project

Setting up a project using JMS follows the standard procedure. See "Creating an Integration Project" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator.

It is recommended to import the following knowledge modules into your project for getting started with JMS:

  • IKM SQL to JMS Append

  • LKM JMS to SQL

24.5 Creating and Defining a JMS Model

This section contains the following topics:

Note:

It is not possible to reverse-engineer a JMS model. To create a datastore you have to create a JMS model and define the JMS datastores.

24.5.1 Create a JMS Model

Create a JMS Model using the standard procedure, as described in "Creating a Model" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator.

A JMS Model is a set of datastores corresponding to the Topics or Queues of a router. Each datastore corresponds to a specific Queue or Topic. The datastore structure defines the message structure for this queue or topic. A model is always based on a Logical Schema. In a given Context, the Logical Schema corresponds to one JMS Physical Schema. The Data Schema corresponding to this Physical Schema contains the Topics or Queues.

24.5.2 Defining the JMS Datastores

In Oracle Data Integrator, each datastore is a JMS Topic or Queue. Each message in this topic or queue is a row of the datastore.

A JMS message may carry any type of information and there is no metadata retrieval method available. Therefore reverse-engineering is not possible.

To define the datastore structure, do one of the following:

  • Create the datastore as a file datastore and manually declare the message structures.

  • Use the File reverse-engineering through an Excel spreadsheet in order to automate the reverse engineering of messages. See Chapter 3, "Files" for more information about this reverse-engineering method.

  • Duplicate a datastore from another model into the JMS model.

Important:

The datastores' resource names must be identical to the name of JMS destinations (this is the logical JNDI name) that will carry the message corresponding to their data. Note that these names are frequently case-sensitive.

Declaring JMS Properties as Pseudo-Columns

The property pseudo-columns represent properties or header fields of a message. These pseudo-columns are defined in the Oracle Data Integrator model as columns in the JMS datastore, with JMS-specific datatypes. The JMS-specific datatypes are called JMS_xxx (for example: JMS String, JMS Long, JMS Int, and so forth).

To define these property pseudo-columns, simply declare additional columns named identically to the properties and specified with the appropriate JMS-specific datatypes.

If you define pseudo-columns that are named like standard, provider-specific or application-specific properties, they will be consumed or published with the message as such. If a pseudo-column is not listed in the standard or provider-specific set of JMS properties, It is considered as additional application-specific property.

For example, to use or set in interfaces the JMSPriority default property on messages consumed from or pushed to a JMS queue called CUSTOMER, you would add a column called JMSPriority (with this exact case) to the CUSTOMER datastore. This column would have the JMS Int datatype available for the JMS Queue technology.

Warning:

  • Property pseudo-columns must be defined and positioned in the JMS datastore after the columns making up the message payload. Use the Order field in the column definition to position these columns. The order of the pseudo-columns themselves is not important as long as they appear at the end of the datastore definition.

  • Pseudo-columns names are case-sensitive.

For more information about JMS Properties, see:

24.6 Designing an Interface

You can use JMS as a source or a target of an integration interface. It cannot be used as the staging area.

The KM choice for an interface or a check determines the abilities and performance of this interface or check. The recommendations in this section help in the selection of the KM for different situations concerning JMS messages.

24.6.1 Loading Data from a JMS Source

JMS can be used as a source or a target in an interface. Data from a JMS message Queue or Topic can be loaded to any SQL compliant database used as a staging area. The LKM choice in the Interface Flow tab to load data between JMS and another type of data server is essential for the performance of an interface.

Oracle Data Integrator provides the LKM JMS to SQL for loading data from a JMS source to a Staging Area. This LKM loads data from a text or binary JMS compliant message queue or topic to any SQL compliant database used as a staging area.

Table 24-2 lists the JMS specific options.

24.6.2 Integrating Data in a JMS Target

Oracle Data Integrator provides the IKM SQL to JMS Append that implements optimized data integration strategies for JMS. This IKM integrates data into a JMS compliant message queue or topic in text or binary format from any SQL compliant staging area. Table 24-2 lists the JMS specific KM options of this IKM.

The IKM choice in the Interface Flow tab determines the performances and possibilities for integrating.

JMS Knowledge Modules Options

Table 24-2 lists the JMS specific KM options of the JMS IKM and LKM.

The JMS specific options of this LKM are similar to the options of the IKM SQL to JMS Append. There are only two differences:

  • The DELETE_TEMPORARY_OBJECTS option is only provided for the LKM.

  • The PUBLISH option is only provided for the IKM.

Table 24-2 JMS Specific KM Options

Option Used to Description

PUBLISH

Write

Check this option if you want to publish new messages in the destination. This option is set to Yes by default.

JMS_COMMIT

Read/Write

Commit the publication or consumption of a message. Uncheck this option if you don't want to commit your publication/consumption on your router. This option is set to Yes by default.

JMSDELIVERYMODE

Write

JMS delivery mode (1: Non Persistent, 2: Persistent). A persistent message remains on the server and is recovered on server crash.

JMSEXPIRATION

Write

Expiration delay in milliseconds for the message on the server [0..4 000 000 000]. 0 signifies that the message never expires.

Warning! After this delay, a message is considered as expired, and is no longer available in the topic or queue. When developing interfaces it is advised to set this parameter to zero.

JMSPRIORITY

Write

Relative Priority of the message: 0 (lowest) to 9 (highest).

SENDMESSAGETYPE

Write

Type of message to send (1 -> BytesMessage, 2 ->TextMessage).

JMSTYPE

Write

Optional name of the message.

CLIENTID

Read

Subscriber identification string. This option is described only for JMS compatibility.

Not used for publication.

DURABLE

Read

D: Session is durable. Indicates that the subscriber definition remains on the router after disconnection.

MESSAGEMAXNUMBER

Read

Maximum number of messages retrieved [0 .. 4 000 000 000]. 0: All messages are retrieved.

MESSAGETIMEOUT

Read

Time to wait for the first message in milliseconds [0 .. 4 000 000 000]. if MESSAGETIMEOUT is equal to 0, then there is no timeout.

MESSAGETIMEOUT and MESSAGEMAXNUMBER cannot be both equal to zero. if MESSAGETIMEOUT= 0 and MESSAGEMAXNUMBER =0, then MESSAGETIMEOUT takes the value 1.

Warning! An interface may retrieve no message if this timeout value is too small.

NEXTMESSAGETIMEOUT

Read

Time to wait for each subsequent message in milliseconds [0 .. 4 000 000 000]. The default value is 1000.

Warning! An interface may retrieve only part of the messages available in the topic or the queue if this value is too small.

MESSAGESELECTOR

Read

Message selector in ISO SQL syntax. See Section 24.7.1, "Using JMS Properties" for more information on message selectors.


24.7 JMS Standard Properties

This section describes the JMS properties contained in the message header and how to use them.

In Oracle Data Integrator, pseudo-columns corresponding to the JMS Standard properties should be declared in accordance with the descriptions provided in Table 24-3.

The JMS type and access mode columns refer to the use of these properties in Oracle Data Integrator or in Java programs. In Oracle Data Integrator, some of these properties are used through the IKM options, and the pseudo-column values should not be set by the interfaces.

For more details on using these properties in a Java program, see http://java.sun.com/products/jms/.

Table 24-3 Standard JMS Properties of Message Headers

Property JMS Type Access (Read/Write) Description

JMSDestination

JMS String

R

Name of the destination (topic or queue) of the message.

JMSDeliveryMode

JMS Integer

R/W (set by IKM option)

Distribution mode: 1 = Not Persistent or 2 = Persistent. A persistent message is never lost, even if a router crashes.

When sending messages, this property is set by the JMSDELIVERYMODE KM option.

JMSMessageID

JMS String

R

Unique Identifier for a message. This identifier is used internally by the router.

JMSTimestamp

JMS Long

R

Date and time of the message sending operation. This time is stored in a UTC standard format (1).

JMSExpiration

JMS Long

R/W (set by IKM option)

Message expiration date and time. This time is stored in a UTC standard format (1).

To set this property the JMSEXPIRATION KM option must be used.

JMSRedelivered

JMS Boolean

R

Indicates if the message was resent. This occurs when a message consumer fails to acknowledge the message reception.

JMSPriority

JMS Int

R/W

Name of the destination (topic or queue) the message replies should be sent to.

JMSCorrelationID

JMS String

R/W

Correlation ID for the message. This may be the JMSMessageID of the message this message generating this reply. It may also be an application-specific identifier.

JMSType

JMS String

R/W (set by IKM option)

Message type label. This type is a string value describing the message in a functional manner (for example SalesEvent, SupportProblem).

To set this property the JMSTYPE KM option must be used.


Table 24-4 lists the optional JMS-defined properties in the JMS standard.

Table 24-4 Standard JMS Properties of Message Headers

Property JMS Type Access (Read/Write) Description

JMSXUserID

JMS String

R

Client User ID.

JMSXAppID

JMS String

R

Client Application ID.

JMSSXProducerTXID

JMS String

R

Transaction ID for the production session. This ID is the same for all the messages sent to a destination by a producer between two JMS commit operations.

JMSSXConsumerTXID

JMS String

R

Transaction ID for current consumption session. This ID is the same of a batch of message read from a destination by a consumer between two JMS commit read operations.

JMSXRcvTimestamp

JMS Long

R

Message reception date and time. This time is stored in a UTC standard format (1).

JMSXDeliveryCount

JMS Int

R

Number of times a message is received. Always set to 1.

JMSXState

JMS Int

R

Message state. Always set to 2 (Ready).

JMSXGroupID

JMS String

R/W

ID of the group to which the message belongs.

JMSXGroupSeq

JMS Int

R/W

Sequence number of the message in the group of messages.


(1): The UTC (Universal Time Coordinated) standard is the number of milliseconds that have elapsed since January 1st, 1970

24.7.1 Using JMS Properties

In addition to their contents, messages have a set of properties attached to them. These may be provider-specific, application-specific (user defined) or JMS Standard Properties.

JMS properties are used in Oracle Data Integrator as complementary information to the message, and are used, for example, to filter the messages.

24.7.1.1 Declaring JMS Properties

When Defining the JMS Datastores, you must append pseudo-columns corresponding to the JMS properties that you want to use in your interfaces. See Declaring JMS Properties as Pseudo-Columns for more information.

24.7.1.2 Filtering on the Router

With this type of filtering, the filter is specified when sending the JMS read query. Only messages matching the message selector filter are retrieved. The message selector is specified in Oracle Data Integrator using the MESSAGE_SELECTOR KM option

Note:

Router filtering is not a JMS mandatory feature. It may be unavailable. Please confirm that it is available by reviewing the JMS provider documentation.

The MESSAGE_SELECTOR is programmed in an SQL WHERE syntax. Comparison, boolean and mathematical operators are supported:

+, -, *, /, =, >, <, <>, >=, <=, OR, AND, BETWEEN, IN, NOT, LIKE, IS NULL.

Notes:

  • The IS NULL clause handles properties with an empty value but does not handle nonexistent application-specific properties.

    For example, if the selector COLOR IS NULL is defined, a message with the application-specific property COLOR specified with an empty value is consumed correctly. Another message in the same topic/queue without this property specified would raise an exception.

Examples

Filter all messages with priority greater than 5

JMSPriority > 5

Filter all messages with priority not less than 6 and with the type Sales_Event.

NOT JMSPriority < 6 AND JMSType = 'Sales_Event'

24.7.1.3 Filtering on the Client

Filtering is performed after receiving the messages, and is setup by creating a standard Oracle Data Integrator interface filter which must be executed on the staging area. A filter uses pseudo-columns from the source JMS datastore. The pseudo-columns defined in the Oracle Data Integrator datastore represent the JMS properties. See Declaring JMS Properties as Pseudo-Columns for more information. Note that messages filtered this way are considered as consumed from the queue or topic.

24.7.1.4 Using Property Values as Source Data

It is possible to use the values of JMS properties as source data in an interface. This is carried out by specifying the pseudo-columns of the source JMS datastore in the mapping. See Declaring JMS Properties as Pseudo-Columns for more information.

24.7.1.5 Setting Properties when Sending a Message

When sending messages it is possible to specify JMS properties by mapping values of the pseudo-columns in an interface targeting a JMS datastore. Certain properties may be set using KM options. See Section 24.7, "JMS Standard Properties" for more information.