1. Integration Guide
  2. Activating Integration Data Queues

Activating Integration Data Queues

There are several default Integration Data Queues available in the PUBLIC domain:

  • INTEGRATION IN: Used to manage all Inbound Transmissions sent in via UI (Upload Transmission), HTTP (WMServlet), and SERVICE (TransmissionService).
  • INTEGRATION OUT XML BUILD: Used to manage outbound process of generating Transmission content from database objects and staging in Transmission tables ready for transport.
  • INTEGRATION OUT TRANSPORT – HTTP: Used to manage transmissions ready to be transported via HTTP to external system.
  • INTEGRATION OUT TRANSPORT – SERVICE: Used to manage transmissions ready to be transported via web service call to external system.

The inbound data queue events reside in the Q_INTEGRATION_IN Data Queue table. The outbound data queue events reside in the Q_INTEGRATION_OUT Data Queue table.

By default, the PUBLIC data queues are inactive, except for the INTEGRATION IN data queue which is active. To activate the other queues, login as DBA.ADMIN and edit the Data Queue records via the Data Queue Manager (under Business Process Automation > Integration menu).

Alternatively, the PUBLIC records can be copied to DOMAIN specific records and activated there. In order for the application server to recognize the new queue names as the default data queues, the following properties must also be set:

glog.integration.dataqueue.inbound
glog.integration.dataqueue.xmlBuild
glog.integration.dataqueue.transport.http
glog.integration.dataqueue.transport.service

For example,

glog.integration.dataqueue.inbound=MYDOMAIN.MYINBOUNDQ

Note: An application server can only process one default data queue of each type. For example, it is not possible to configure two inbound default data queues: MYDOMAIN.INBOUNDQ1 & MYDOMAIN.INBOUNDQ2 or MYDOMAIN2.INBOUNDQ1.

Although only a single default inbound data queue can be specified, it is possible to configure as many additional inbound data queues as required. Additional data queues are useful for scenarios where different inbound transactions require different prioritization or throughput management. For example, it may be desirable to process inbound Tracking Event transactions on a data queue with more threads or that while Order Release creation and update must be regarded as top priority, other lower priority transactions must still be processed.

It is now possible to specify a data queue GID and/or event priority in the Transmission header for each individual Transmission to override the default inbound data queue. Consult the Transmission schema documentation for details on how to specify these overrides.

Using the earlier example where Order Releases are top priority, but others must still be processed, it should be clear from the following discussion that multiple data queues would best be used in this scenario.

The priority value for an event corresponds to the q_preemption_priority column in the data queue. The poller configured for the data queue can then use this column in the 'order by' clause for retrieving batches of events to process. The INTEGRATION IN poller is an example of such a poller. For the single data queue scenario where orders were higher priority than all other transactions, if there were 100 events in the queue – 60 order releases and 40 tracking events – and the data queue batch size was 50, then only the first 50 order releases would be selected for processing. If, before the poller attempts to poll a new batch another 50 order release transmissions arrive, then these would be selected for the next batch thus meaning that the processing of tracking events message is delayed longer than might be preferable. However, with different data queues each queue can process events in parallel.

The inbound Order Release Transmissions should specify the data queue used to handle orders and the other Transmissions should specify a different data queue and possibly also a priority value to further control the order of processing in this data queue. Obviously, there are many other possible configurations of batch sizes, poller frequency and threads counts that can handle similar scenarios. The addition of support for multiple data queues expands these options considerably.

Once one or more Integration Data Queues are active, the processing can be controlled on a domain-by-domain basis by setting one of the following PARAMETER SET parameters to 'TRUE' on the DOMAIN PARAMETER SET.

  • DATA QUEUES - USE INBOUND DATA QUEUE
  • DATA QUEUES – USE QUEUE FOR XML BUILD
  • DATA QUEUES – USE QUEUE FOR TRANSPORT – HTTP
  • DATA QUEUES – USE QUEUE FOR TRANSPORT – SERVICE

If either the queue is not active or the domain parameter is FALSE, the default in-memory queues will be used.

Note: It is possible to override this default to process via in-memory queues. Set property glog.integration.dataqueue.inbound.useMemoryQueue=false. This will cause the inbound Transmissions to remain in 'STAGED' status.