Oracle® Fusion Middleware Connectivity and Knowledge Modules Guide for Oracle Data Integrator 11g Release 1 (11.1.1) E12644-05 |
|
![]() Previous |
![]() Next |
This chapter describes how to work with Java Message Services (JMS) in Oracle Data Integrator.
This chapter includes the following sections:
Oracle Data Integrator provides a simple and transparent method to integrate JMS destinations. This chapter focuses on processing JMS messages with a text payload in batch mode. For XML payload processing, refer to Chapter 25, "JMS XML".
The JMS Knowledge Modules apply to most popular JMS compliant middleware, including Oracle Service Bus, Sonic MQ, IBM Websphere MQ, and so forth. Most of these Knowledge Modules include transaction handling to ensure message delivery.
This section describes the structure of a message in a JMS destination.
A JMS Message consists of three sections:
Header
The header contains in the header fields standard metadata concerning the message, including the destination (JMSDestination), Message ID (JMSMessageID), Message Type (JMSType), and so forth.
Properties
The properties section contains additional metadata concerning the message. These metadata are properties, that can be separated in three groups:
JMS-Defined properties which are optional JMS Headers. Their name begins with JMSX(JMSXUserID, JMSXAppID, etc.).
Provider-specific properties. They are specific to the router vendor. Their names start with JMS_<vendor name>.
Application-specific properties. These properties depend on the application sending the messages. These are user-defined information that is not included in the message payload.
The Header and Properties sections provide a set of header fields and properties that:
Have a specific Java data type (Boolean, string, short, and so forth),
Can be accessed for reading and/or writing,
Can be used for filtering on the router through the JMS Selector.
Payload
The payload section contains the message content. This content can be anything (text, XML, binary, and so forth).
Oracle Data Integrator is able to process JMS Text and Byte messages that are delivered by a JMS destination. Each message is considered as a container for rows of data and is handled through the JMS Queue or JMS Topic technology.
With JMS Queue/JMS Topic technologies, each JMS destination is defined similarly to a flat file datastore. Each message in the destination is a record in the datastore.
In the topology, each JMS router is defined as a JMS Topic/Queue data server, with a single physical schema. A JMS router may be defined therefore twice to access its topics using one data server, and its queues using another one.
Each JMS destination (Topic of Queue) is defined as a JMS datastore which resource name matches the name of the JMS destination (name of the queue or topic as defined in the router). A model groups message structures related to different topics or queues.
The JMS datastore structure is defined similarly to a flat file (delimited or fixed width). The properties or header fields of the message can be declared with JMS-specific data types as additional pseudo-columns in this flat file structure. Each message in the destination is processed as a record of a JMS datastore.
Processing Messages
JMS destinations are handled as regular file datastores and messages as rows from these datastores. With these technologies, entire message sets are produced and consumed within each interface.
Message publishing as well consumption requires a commit action to finalize removing/posting the message from/to the JMS destination. Committing is particularly important when reading. Without a commit, the message is read but not consumed. It remains in the JMS Topic/Queue and will be re-read at a later time.
Both the message content and pseudo-columns can be used as regular columns in the integration interfaces (for mapping, filter, etc.). Certain pseudo-columns (such as the one representing the MESSAGE_ID property) are read-only, and some properties of header fields are used (or set) through the Knowledge Module options.
Using Data Integrator you can transfer information either through the message payload - the columns - , or through the properties - pseudo-columns - (application properties, for example).
Using the properties to carry information is restricted by third-party applications producing or consuming the messages.
Filtering Messages
It is possible to filter messages from a JMS destination in two ways:
By defining a filter using the datastore's columns and pseudo-columns. In this case Data Integrator performs the filtering operation after consuming the messages. This implies that messages rejected by this filter may also be consumed.
By defining a Message Selector (MESSAGE_SELECTOR KM option). This type of filter can only use the properties or header fields of the message. The filter is processed by the router, and only the messages respecting the filter are consumed, reducing the number of messages transferred.
Oracle Data Integrator provides the Knowledge Modules (KM) listed in Table 24-1 for handling JMS messages.
Table 24-1 JMS Knowledge Modules
Knowledge Module | Description |
---|---|
IKM SQL to JMS Append |
Integrates data into a JMS compliant message queue or topic in text or binary format from any SQL compliant staging area. Consider using this IKM if you plan to transform and export data to a target JMS queue or topic. If most of your source datastores are located on the same data server, we recommend using this data server as staging area to avoid extra loading phases (LKMs). To use this IKM, the staging area must be different from the target. |
LKM JMS to SQL |
Loads data from a text or binary JMS compliant message queue or topic to any SQL compliant database used as a staging area. This LKM uses the Agent to read selected messages from the source queue/topic and write the result in the staging temporary table created dynamically. To ensure message delivery, the message consumer (or subscriber) does not commit the read until the data is actually integrated into the target by the IKM. Consider using this LKM if one of your source datastores is a text or binary JMS message. |
Make sure you have read the information in this section before you start using the JMS Knowledge Modules:
Before performing any installation you should read the system requirements and certification documentation to ensure that your environment meets the minimum installation requirements for the products you are installing.
The list of supported platforms and versions is available on Oracle Technical Network (OTN):
http://www.oracle.com/technology/products/oracle-data-integrator/index.html
.
The JMS destinations are usually accessed via a JNDI service. The configuration and specific requirements for JNDI and JMS depends on the JMS Provider you are connecting to. Refer to the JMS Provider specific documentation for more details.
Oracle Data Integrator does not include specific drivers for JMS providers. Refer to the JMS Provider documentation for the connectivity requirement of this provider.
Setting up the Topology consists of:
A JMS data server corresponds to one JMS provider/router that is accessible through your local network.
It exists two types of JMS data servers: JMS Queue and JMS Topic.
A JMS Queue data server is used to access several queues in the JMS router.
A JMS Topic data server is used to access several topics in the JMS router
Create a data server either for the JMS Queue technology or for the JMS Topic technology using the standard procedure, as described in "Creating a Data Server" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator. This section details only the fields required or specific for defining a JMS Queue or JMS Topic data server.
In the Definition tab:
Name: Name of the data server as it will appear in Oracle Data Integrator.
User/Password: Not used here. Leave these fields empty.
In the JNDI tab:
JNDI Authentication: Set this field to None
.
JNDI User: Enter the username to connect to the JNDI directory (optional step).
Password: This user's password (optional step).
JNDI Protocol: From the list, select the JNDI protocol (optional step).
JNDI Driver: Name of the initial context factory java class to connect to the JNDI provider, for example: com.sun.jndi.ldap.LdapCtxFactory
for LDAP
JNDI URL: <JMS_RESOURCE>
, for example ldap://<host>:<port>/<dn>
for LDAP
JNDI Resource: Logical name of the JNDI resource corresponding to your JMS Queue or Topic connection factory.
For example, specify QueueConnectionFactory
if you want to access a message queue and TopicConnectionFactory
if you want to access a topic. Note that these parameters are specific to the JNDI directory and the provider.
Create a JMS physical schema using the standard procedure, as described in "Creating a Physical Schema" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator.
Note: Only one physical schema is required per JMS data server. |
Create for this physical schema a logical schema using the standard procedure, as described in "Creating a Logical Schema" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator and associate it in a given context.
Setting up a project using JMS follows the standard procedure. See "Creating an Integration Project" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator.
It is recommended to import the following knowledge modules into your project for getting started with JMS:
IKM SQL to JMS Append
LKM JMS to SQL
This section contains the following topics:
Note: It is not possible to reverse-engineer a JMS model. To create a datastore you have to create a JMS model and define the JMS datastores. |
Create a JMS Model using the standard procedure, as described in "Creating a Model" of the Oracle Fusion Middleware Developer's Guide for Oracle Data Integrator.
A JMS Model is a set of datastores corresponding to the Topics or Queues of a router. Each datastore corresponds to a specific Queue or Topic. The datastore structure defines the message structure for this queue or topic. A model is always based on a Logical Schema. In a given Context, the Logical Schema corresponds to one JMS Physical Schema. The Data Schema corresponding to this Physical Schema contains the Topics or Queues.
In Oracle Data Integrator, each datastore is a JMS Topic or Queue. Each message in this topic or queue is a row of the datastore.
A JMS message may carry any type of information and there is no metadata retrieval method available. Therefore reverse-engineering is not possible.
To define the datastore structure, do one of the following:
Create the datastore as a file datastore and manually declare the message structures.
Use the File reverse-engineering through an Excel spreadsheet in order to automate the reverse engineering of messages. See Chapter 3, "Files" for more information about this reverse-engineering method.
Duplicate a datastore from another model into the JMS model.
Important: The datastores' resource names must be identical to the name of JMS destinations (this is the logical JNDI name) that will carry the message corresponding to their data. Note that these names are frequently case-sensitive. |
Declaring JMS Properties as Pseudo-Columns
The property pseudo-columns represent properties or header fields of a message. These pseudo-columns are defined in the Oracle Data Integrator model as columns in the JMS datastore, with JMS-specific datatypes. The JMS-specific datatypes are called JMS_xxx (for example: JMS String, JMS Long, JMS Int, and so forth).
To define these property pseudo-columns, simply declare additional columns named identically to the properties and specified with the appropriate JMS-specific datatypes.
If you define pseudo-columns that are named like standard, provider-specific or application-specific properties, they will be consumed or published with the message as such. If a pseudo-column is not listed in the standard or provider-specific set of JMS properties, It is considered as additional application-specific property.
For example, to use or set in interfaces the JMSPriority default property on messages consumed from or pushed to a JMS queue called CUSTOMER, you would add a column called JMSPriority (with this exact case) to the CUSTOMER datastore. This column would have the JMS Int datatype available for the JMS Queue technology.
Warning:
|
For more information about JMS Properties, see:
You can use JMS as a source or a target of an integration interface. It cannot be used as the staging area.
The KM choice for an interface or a check determines the abilities and performance of this interface or check. The recommendations in this section help in the selection of the KM for different situations concerning JMS messages.
JMS can be used as a source or a target in an interface. Data from a JMS message Queue or Topic can be loaded to any SQL compliant database used as a staging area. The LKM choice in the Interface Flow tab to load data between JMS and another type of data server is essential for the performance of an interface.
Oracle Data Integrator provides the LKM JMS to SQL for loading data from a JMS source to a Staging Area. This LKM loads data from a text or binary JMS compliant message queue or topic to any SQL compliant database used as a staging area.
Table 24-2 lists the JMS specific options.
Oracle Data Integrator provides the IKM SQL to JMS Append that implements optimized data integration strategies for JMS. This IKM integrates data into a JMS compliant message queue or topic in text or binary format from any SQL compliant staging area. Table 24-2 lists the JMS specific KM options of this IKM.
The IKM choice in the Interface Flow tab determines the performances and possibilities for integrating.
JMS Knowledge Modules Options
Table 24-2 lists the JMS specific KM options of the JMS IKM and LKM.
The JMS specific options of this LKM are similar to the options of the IKM SQL to JMS Append. There are only two differences:
The DELETE_TEMPORARY_OBJECTS option is only provided for the LKM.
The PUBLISH option is only provided for the IKM.
Table 24-2 JMS Specific KM Options
Option | Used to | Description |
---|---|---|
PUBLISH |
Write |
Check this option if you want to publish new messages in the destination. This option is set to Yes by default. |
JMS_COMMIT |
Read/Write |
Commit the publication or consumption of a message. Uncheck this option if you don't want to commit your publication/consumption on your router. This option is set to Yes by default. |
JMSDELIVERYMODE |
Write |
JMS delivery mode (1: Non Persistent, 2: Persistent). A persistent message remains on the server and is recovered on server crash. |
JMSEXPIRATION |
Write |
Expiration delay in milliseconds for the message on the server [0..4 000 000 000]. 0 signifies that the message never expires. Warning! After this delay, a message is considered as expired, and is no longer available in the topic or queue. When developing interfaces it is advised to set this parameter to zero. |
JMSPRIORITY |
Write |
Relative Priority of the message: 0 (lowest) to 9 (highest). |
SENDMESSAGETYPE |
Write |
Type of message to send (1 -> BytesMessage, 2 ->TextMessage). |
JMSTYPE |
Write |
Optional name of the message. |
CLIENTID |
Read |
Subscriber identification string. This option is described only for JMS compatibility. Not used for publication. |
DURABLE |
Read |
D: Session is durable. Indicates that the subscriber definition remains on the router after disconnection. |
MESSAGEMAXNUMBER |
Read |
Maximum number of messages retrieved [0 .. 4 000 000 000]. 0: All messages are retrieved. |
MESSAGETIMEOUT |
Read |
Time to wait for the first message in milliseconds [0 .. 4 000 000 000]. if MESSAGETIMEOUT is equal to 0, then there is no timeout. MESSAGETIMEOUT and MESSAGEMAXNUMBER cannot be both equal to zero. if MESSAGETIMEOUT= 0 and MESSAGEMAXNUMBER =0, then MESSAGETIMEOUT takes the value 1. Warning! An interface may retrieve no message if this timeout value is too small. |
NEXTMESSAGETIMEOUT |
Read |
Time to wait for each subsequent message in milliseconds [0 .. 4 000 000 000]. The default value is 1000. Warning! An interface may retrieve only part of the messages available in the topic or the queue if this value is too small. |
MESSAGESELECTOR |
Read |
Message selector in ISO SQL syntax. See Section 24.7.1, "Using JMS Properties" for more information on message selectors. |
This section describes the JMS properties contained in the message header and how to use them.
In Oracle Data Integrator, pseudo-columns corresponding to the JMS Standard properties should be declared in accordance with the descriptions provided in Table 24-3.
The JMS type and access mode columns refer to the use of these properties in Oracle Data Integrator or in Java programs. In Oracle Data Integrator, some of these properties are used through the IKM options, and the pseudo-column values should not be set by the interfaces.
For more details on using these properties in a Java program, see http://java.sun.com/products/jms/
.
Table 24-3 Standard JMS Properties of Message Headers
Property | JMS Type | Access (Read/Write) | Description |
---|---|---|---|
JMSDestination |
JMS String |
R |
Name of the destination (topic or queue) of the message. |
JMSDeliveryMode |
JMS Integer |
R/W (set by IKM option) |
Distribution mode: 1 = Not Persistent or 2 = Persistent. A persistent message is never lost, even if a router crashes. When sending messages, this property is set by the JMSDELIVERYMODE KM option. |
JMSMessageID |
JMS String |
R |
Unique Identifier for a message. This identifier is used internally by the router. |
JMSTimestamp |
JMS Long |
R |
Date and time of the message sending operation. This time is stored in a UTC standard format (1). |
JMSExpiration |
JMS Long |
R/W (set by IKM option) |
Message expiration date and time. This time is stored in a UTC standard format (1). To set this property the JMSEXPIRATION KM option must be used. |
JMSRedelivered |
JMS Boolean |
R |
Indicates if the message was resent. This occurs when a message consumer fails to acknowledge the message reception. |
JMSPriority |
JMS Int |
R/W |
Name of the destination (topic or queue) the message replies should be sent to. |
JMSCorrelationID |
JMS String |
R/W |
Correlation ID for the message. This may be the JMSMessageID of the message this message generating this reply. It may also be an application-specific identifier. |
JMSType |
JMS String |
R/W (set by IKM option) |
Message type label. This type is a string value describing the message in a functional manner (for example To set this property the JMSTYPE KM option must be used. |
Table 24-4 lists the optional JMS-defined properties in the JMS standard.
Table 24-4 Standard JMS Properties of Message Headers
Property | JMS Type | Access (Read/Write) | Description |
---|---|---|---|
JMSXUserID |
JMS String |
R |
Client User ID. |
JMSXAppID |
JMS String |
R |
Client Application ID. |
JMSSXProducerTXID |
JMS String |
R |
Transaction ID for the production session. This ID is the same for all the messages sent to a destination by a producer between two JMS commit operations. |
JMSSXConsumerTXID |
JMS String |
R |
Transaction ID for current consumption session. This ID is the same of a batch of message read from a destination by a consumer between two JMS commit read operations. |
JMSXRcvTimestamp |
JMS Long |
R |
Message reception date and time. This time is stored in a UTC standard format (1). |
JMSXDeliveryCount |
JMS Int |
R |
Number of times a message is received. Always set to 1. |
JMSXState |
JMS Int |
R |
Message state. Always set to 2 (Ready). |
JMSXGroupID |
JMS String |
R/W |
ID of the group to which the message belongs. |
JMSXGroupSeq |
JMS Int |
R/W |
Sequence number of the message in the group of messages. |
(1): The UTC (Universal Time Coordinated) standard is the number of milliseconds that have elapsed since January 1st, 1970
In addition to their contents, messages have a set of properties attached to them. These may be provider-specific, application-specific (user defined) or JMS Standard Properties.
JMS properties are used in Oracle Data Integrator as complementary information to the message, and are used, for example, to filter the messages.
When Defining the JMS Datastores, you must append pseudo-columns corresponding to the JMS properties that you want to use in your interfaces. See Declaring JMS Properties as Pseudo-Columns for more information.
With this type of filtering, the filter is specified when sending the JMS read query. Only messages matching the message selector filter are retrieved. The message selector is specified in Oracle Data Integrator using the MESSAGE_SELECTOR KM option
Note: Router filtering is not a JMS mandatory feature. It may be unavailable. Please confirm that it is available by reviewing the JMS provider documentation. |
The MESSAGE_SELECTOR is programmed in an SQL WHERE syntax. Comparison, boolean and mathematical operators are supported:
+, -, *, /, =, >, <, <>, >=, <=, OR, AND, BETWEEN, IN, NOT, LIKE, IS NULL
.
Notes:
|
Examples
Filter all messages with priority greater than 5
JMSPriority > 5
Filter all messages with priority not less than 6 and with the type Sales_Event.
NOT JMSPriority < 6 AND JMSType = 'Sales_Event'
Filtering is performed after receiving the messages, and is setup by creating a standard Oracle Data Integrator interface filter which must be executed on the staging area. A filter uses pseudo-columns from the source JMS datastore. The pseudo-columns defined in the Oracle Data Integrator datastore represent the JMS properties. See Declaring JMS Properties as Pseudo-Columns for more information. Note that messages filtered this way are considered as consumed from the queue or topic.
It is possible to use the values of JMS properties as source data in an interface. This is carried out by specifying the pseudo-columns of the source JMS datastore in the mapping. See Declaring JMS Properties as Pseudo-Columns for more information.
When sending messages it is possible to specify JMS properties by mapping values of the pseudo-columns in an interface targeting a JMS datastore. Certain properties may be set using KM options. See Section 24.7, "JMS Standard Properties" for more information.