1. Integrating Enterprise Data Quality with External Systems
  2. Using the Java Messaging Service (JMS) with EDQ

12 Using the Java Messaging Service (JMS) with EDQ

This document describes how to get started using Java Messaging Service (JMS) technology with Oracle Enterprise Data Quality (EDQ). This documentation is intended for system administrators responsible for installing and maintaining EDQ applications.

This chapter includes the following sections:

12.1 Understanding the Message Queue Architecture

JMS is a flexible technology that allows EDQ to integrate with a variety of different messaging systems and technologies, including WebLogic Server Messaging, WebSphere MQ, Oracle Advanced Queueing (OAQ), and many more.

The EDQ server acts as a ‘client’ of a message queue. The server component may be installed on the same physical server as EDQ, but more commonly the server (or servers) will be remote to it. EDQ is shipped with the client jars needed to connect to either ActiveMQ or Artemis message queue providers. If EDQ is deployed on WebLogic Server, it will also be able to connect natively to WebLogic JMS. Otherwise, for example if EDQ needs to act as a client to an alternative message queue (MQ) system, such as WebSphere MQ, it will need additional client jars to be installed. In this case, the administrator of the MQ system should be consulted to determine which files are needed for a Java application such as EDQ to act as a client.

12.2 Uses of JMS with EDQ

There are two main uses of JMS with EDQ, as listed below: We will focus here on the first use of JMS with EDQ.

  • Consuming and Providing Messages: You can configure EDQ to read messages from JMS queues, and write messages to them. This can be beneficial where several EDQ servers need to read from a single stream of messages that needs to be persistent to ensure no loss of messages. In this mode, each EDQ server will consume messages from a queue and only ‘commit’ when it has finished processing each message. Note that JMS is best used for Asynchronous communication, where EDQ is not expected to return a response to a calling application for a specific message.
  • Using JMS in Triggers: EDQ may send JMS messages to other systems, or may consume JMS messages to start triggers (for example to use a JMS queue to distribute batch jobs amongst several EDQ servers;). For more details, see the Using JMS in Triggers.

12.3 Configuring EDQ to Read and Write JMS Messages

JMS interfaces to and from EDQ are configured using XML interface files that define:

  • The path to the queue of messages
  • Properties that define how to work with the specific JMS technology
  • How to decode the message payload into a format understood by an EDQ process (for Message Providers – where EDQ reads messages from a queue), or convert messages to a format expected by an external process (for Message Consumers – where EDQ writes messages to a queue).

The XML files are located in the EDQ Local Home directory (formerly known as the config directory), in the following paths:

  • buckets/realtime/providers (for interfaces ‘in’ to EDQ)
  • buckets/realtime/consumers (for interfaces ‘out’ from EDQ)

Once the XML files have been configured, Message Provider interfaces are available in Reader processors in EDQ and to map to Data Interfaces as ‘inputs’ to a process, and Message Consumer interfaces are available in Writer processors, and to map from Data Interfaces as ‘outputs’ from a process as shown in Figures 1 and 2 below:

12.4 Defining the Interface Files

An interface file in EDQ consists of three sections, as follows:

  • The <attributes> section, defining the shape of the interface as understood by EDQ
  • The <messengerconfig> section, defining how to connect to the JMS queue or topic
  • The <messagebody> section, defining how to extract contents of a message (e.g. from XML) into attribute data readable by EDQ (for inbound interfaces), or how to convert attribute data from EDQ to message data (e.g. in XML).

12.4.1 Knowing the <attributes> section

The <attributes> section defines the shape of the interface. It constitutes the attributes that are available in EDQ when configuring a Reader or Writer. For example the following excerpt from the beginning of an interface file configures three string attributes that can be used in EDQ, and their names are:

<?xml version="1.0" encoding="UTF-8"?>
<realtimedata messenger="jms">
 <attributes> 
<attribute type="string" name="messageID"/> 
<attribute type="string" name="name"/> 
<attribute type="string" name="AccountNumber"/> 
</attributes> 

[file continues]...

EDQ supports all the standard attribute types and they are:

  • string
  • number
  • date
  • stringarray
  • numberarray
  • datearray

12.4.2 Knowing the <messengerconfig> Section

The <messengerconfig> section of the interface file configures the settings needed to connect to a given JMS queue or topic on a particular type of JMS technology. The text in <messengerconfig> is parsed as a Java properties file and is merged with the set of properties from the realtime.properties file in the EDQ configuration path, with settings in the <messengerconfig> section overriding any matching settings inrealtime.properties. The realtime.properties file therefore allows the configuration of a number of global properties that do not need to be stated in every JMS interface file.

Note:

  • As with all property files stored in the EDQ Local Home directory, properties are themselves merged with a set of ‘base’ properties in the EDQ Home directory which should not be modified, with any property stated in a file in the Local Home directory used in preference to any property in the EDQ Home directory. For example therealtime.properties file in the EDQ Local Home directory will be merged with the file of the same name in the EDQ Home directory to form the final set of properties used.
  • The realtime.properties values may also be used to set properties for Web Services in EDQ. A prefix of jms. is therefore used (in this file only, not in the <messengerconfig> section of a JMS interface file) to set properties for JMS (ws. is used for Web Services).

The messenger configuration properties, specified either in the <messengerconfig> section of a JMS interface file, or inherited from realtime.properties, are used:

  • To configure a JNDI name store to look up queues
  • To specify the name of the queue/topic and connection factory
  • To supply authentication information to the MQ server, if required
JNDI Setup Properties
  • java.naming.factory.initial - It denotes the class name of the JNDI initial context factory, used to bootstrap JNDI naming
  • java.naming.provider.url - the JNDI URL used to connect to the JNDI server or local store

In some cases, authentication is required to connect to the JNDI service. In this case the following properties must be added:

  • java.naming.security.principal - JNDI user name
  • java.naming.security.credentials - JNDI password

These JNDI properties are a standard Java feature and for more details, see JNDI Documentation.

JMS Names

The other properties are:

  • connectionfactory - the JNDI name of the JMS ‘connection factory’. The default for this (if the property is not set) is ‘ConnectionFactory’ which is correct for several MQ servers.
  • destination - the JNDI name of the JMS queue or topic. There is no default for this and it is always required.
Authentication

If the MQ server requires authentication when making connections, add the below properties:

username: connection user name
password: connection password

Note:

These properties are used to authenticate against the MQ server and are used to connect to JNDI. In rare cases, both sets may be required.
Notes for specific MQ servers
  1. ActiveMQ -

    To connect to an ActiveMQ server, use a property setting as listed below:

    This can be set in the <messengerconfig> section to apply to a single interface, or can be set in realtime.properties (prefixed with jms.) to set the default for all interfaces.

    java.naming.provider.url=tcp://host:port

    ActiveMQ servers generally require connection authentication, so the username and password properties will be required.

    To use ActiveMQ on an EDQ server installed on WebLogic, the property settings above need to be entered into the realtime.properties in the EDQ Local Home directory. (The settings are present, but commented out, in the file in the Home directory, as it is assumed that WebLogic JMS will normally be used where EDQ is installed on WebLogic.)

  2. WebLogic JMS -

    When running in a full JavaEE application server, such as WebLogic, Glassfish or WebSphere, the JNDI settings for local JMS are configured automatically and the JNDI factory and url information do not need to be specified in any EDQ configuration file.

    The recommended approach in these cases is to define the JMS configuration within the application server and then just specify the JNDI connection factory and destination names in <messengerconfig>.

    WebLogic supports the concept of ‘foreign’ JMS servers. In this case, you define the connection information in the WebLogic Console and expose the destination(s) and connection factory as names in the native WebLogic JNDI store. This works well with Oracle Advanced Queueing (AQ). See below for a snippet of the configuration of a ‘foreign JMS server’ pointing at an AQ schema.

    When JMS is configured like this, a typical <messengerconfig> section might be:
    <messengerconfig>
   connectionfactory = jms/cf1
   destination       = jms/queue1
</messengerconfig>

    Here jms/cf1and jms/queue1 are JNDI names defined in WebLogic.

12.4.3 Knowing the <messagebody> section

This section uses JavaScript to parse message payloads into attributes that EDQ can use for inbound interfaces, and perform the reverse operation (convert EDQ attribute data into message payload data) for outbound interfaces. A function named ‘extract’ is used to extract data from XML into attribute data for inbound interfaces, and a function named ‘build’ is used to build XML data from attribute data.

For more details, refer to Illustrations, wherein the scripts are best illustrated using examples.

12.5 Illustrations

Example 1 – Simple Provider File

The following XML is a simple example of a provider interface file, reading messages from a queue in the path ‘dynamicQueues/InputQueue’. 

<?xml version="1.0" encoding="UTF-8"?>
<realtimedata messenger="jms">
 <attributes> 
<attribute type="string" name="messageID"/> 
<attribute type="string" name="name"/> 
<attribute type="string" name="AccountNumber"/> 
<attribute type="string" name="AccountName"/>
 		<attribute type="string" name="Country"/>
 </attributes> 
<messengerconfig> destination = dynamicQueues/InputQueue </messengerconfig>
 <incoming> 
<messagebody> 
<script> 
<![CDATA[ function extract(str) { var screeningRequest = new XML(str); var rec = new Record(); rec.messageID = screeningRequest.individual.messageID; rec.name = screeningRequest.individual.name; rec.AccountNumber = screeningRequest.individual.AccountNumber; rec.AccountName = screeningRequest.individual.AccountName; rec.Country = screeningRequest.individual.Country; return [rec]; } ]]>
 </script>
 </messagebody> 
<eof> 
<messageheader name="JMSType" value="t1eof"/> 
</eof>
 </incoming> 
</realtimedata>

Example 2 – Simple Consumer File

The following XML is a simple example of a consumer file representing similar data to the provide file listed above, but with additional attributes, which can be added by an EDQ process:

<?xml version="1.0" encoding="UTF-8"?>
<realtimedata messenger="jms">
 <attributes> 
<attribute type="string" name="messageID"/> 
<attribute type="string" name="AccountNumber"/> 
<attribute type="string" name="AccountName"/> 
<attribute type="string" name="Country"/> 
<attribute type="string" name="AccountType"/> 
</attributes>
<messengerconfig> destination = dynamicQueues/OutputQueue </messengerconfig> 
<outgoing> 
<messagebody> 
<script> 
<![CDATA[ function build(recs) { var rec = recs[0]; var xml = <Request> <individual> <messageID>{checkNull(rec.messageID)}</messageID> <AccountNumber>{checkNull(rec.AccountNumber)}</AccountNumber> <AccountName>{checkNull(rec.AccountName)}</AccountName> <Country>{checkNull(rec.Country)}</Country> <AccountType>{checkNull(rec.AccountType)}</AccountType> </individual> </Request>; return xml.toXMLString(); } function checkNull(value) { return value != null ? value : ""; } ]]>
 	</script> 
</messagebody> 
</outgoing>
</realtimedata>

Example 3 – Date Parsing and Formatting

The following example file shows how to define a DATE attribute type, and include date parsing and formatting when decoding the message payload:

<?xml version="1.0" encoding="UTF-8"?>
<realtimedata messenger="jms">
<attributes> 
<attribute type="date" name="processingDate"/> 
</attributes> 
<messengerconfig> destination = dynamicQueues/HoldQueue </messengerconfig> 
<incoming> 
<messagebody>
 <script> 
<![CDATA[ var df = Formatter.newDateFormatter("yyyy-MM-dd'T'HH:mm:ss.SSSZ"); function extract(str) { var screeningRequest = new XML(str); var rec = new Record();rec.processingDate = date(screeningRequest.processingDate; return [rec]; } function date(x) { var s = x.text().toString(); return s == "" ? null : df.parse(s); } ]]>
 </script>
 </messagebody>
 <eof> 
<messageheader name="JMSType" value="t1eof"/>
 </eof>
</incoming>
 </realtimedata>