Data Collection Steps
A data collection step defines the way in which an integration obtains data from a source system. A specific integration may contain one or more data collection steps.
Query Step
This step makes it possible to get a real-time response from Oracle Health Insurance applications, for example, Claims, using Query API. Query API is an HTTP API framework supported by Oracle Health Insurance applications.
The format in which the data gets queried from the Oracle Health Insurance source application is JSON and is Oracle Health Insurance driven. The response can be transformed before delivery by configuring a subsequent Delivery step or Transform step within the integration.
The following table lists the fields valid for this type of step.
| Field | Description | Mandatory? | ||
|---|---|---|---|---|
Code |
The code of the Query step. |
Yes |
||
Sequence |
The execution sequence for this step. The integration steps run in ascending order. |
Yes |
||
Subtype |
QUERY |
Yes |
||
A dynamic logic condition to check if this step should be invoked. The step gets skipped if the condition returns false. |
No |
|||
A dynamic logic function to post process the results of the step. |
No |
|||
Destination |
Reference to the REST destination. For more details on REST destinations refer to the "Destinations" page in the Building Blocks chapter of the Configuration Guide.
|
Yes |
||
Output Name |
Dynamic logic bind variable name. This variable holds the queried results as raw payload and is available as a bind variable to all the subsequent steps and dynamic logic. If the output name is not specified, then the integration step code is used as the bind variable. |
No |
||
Type Config |
The configuration of the query includes supplying information about:
{ "resource": { } }
|
Yes |
| Query API response size is limited to a maximum of 50 records. For large data sets, Oracle recommends the use of Extract step to gather data. |
Sample Configuration: Query member data from an Oracle Health Insurance application for a given member code.
Configure the Query step as follows to query member data from Oracle Health Insurance application.
| Field | Configuration | Remarks |
|---|---|---|
code |
getMemberData |
|
sequence |
1 |
Assuming this is the first step |
destination |
policies |
Refers to a rest destination with code "policies" (assuming the person data gets queried from Policies application) |
outputName |
memberData |
The Query API response can be accessed in the dynamic logic of any subsequent step using "memberData" bind variable. |
type config |
resourceName: persons q: code.eq("{memberCode}") expand:addressList |
maritalStatusList |
An example of the JSON payload for the creation of a Query step using API is given below. Note that this JSON is part of the larger payload needed for the creation of the Integration.
{
"code" : "getMemberData",
"sequence" : 1,
"subtype" : "QUERY",
"destination" : { "code" : "policies"},
"outputName" : "memberData",
"typeConfig" : {
"resourceName" : "persons",
"q" : "code.eq("<memberCode>")",
"expand" : "addressList|maritalStatusList"
}
}
Configure the REST destination as follows :
| Field | Configuration | Remarks |
|---|---|---|
code |
policies |
|
credentialKey |
|
The following must be done to set up credential key "policy_user" Step 1:: Create credential using credential IP or Credential Page Example: To setup credential key "policy_user" using Credential IP the following must be done [source,text,subs="verbatim,attributes"] ---- PUT: http://[hostName]:[portNumber]/[api-context-root]/credentials/credential/policy_user with payload { "username" : "user that OIG should use to access policy application", "password": "password to access policy application" } ---- Step 2:: Set the authentication mechanism using Generic API - properties or using property configuration page Example: To set up authentication using Generic API - /properties (assuming the policies application uses basic authentication) the following must be done ---- POST : http(s)://{host:port}/{oig application context}/generic/properties with payload { "name": "ohi.service.policy_user.client.authentication", "value": "BasicAuthentication" } ---- TIP: For details on "Generic API" refer to Developer Guide, section Integration Generic API and Operations. |
addressKey |
address.key.policies.baseurl |
The address key can be setup by Generic API /properties or properties page Example : To setup address key through Generic API - /properties the following must be done POST: http(s)://{host:port}/{oig application context}/generic/properties with payload
{
"name": "address.key.policies.baseurl",
"value": "URL till {policiesComponent} context root
"e.g." http://{host"port}/{context-root}"
}
|
typeConfig |
path: {path} httpMethod: {httpMethod} |
NOTE: For query step, the application internally sets the path to 'generic/{resource}/search' and the method as 'POST'. Here, the path and http method are parameterized (recommended). This way a single rest destination can be used by multiple integrations with the policies application. For more details on parameterized configurations, refer to the section Integration Parameters and Exchange Properties. |
An example of the JSON payload for the creation of policies destination through API /restdestinations is given below.
{
"code": "policies",
"credentialKey": "policy_user",
"addressKey": "address.key.policies.baseurl",
"typeConfig": {
"path": "{path}",
"httpMethod": "{httpMethod}"
}
}
Extract Step
Extract step enables the users to collect data from the source Oracle Health Insurance application as a file extract This step triggers Extract Integration Point in the end applications and therefore offers all the possibilities supported by the extract integration point. For more details on extract integration point, please refer to the Developer Guide in the core application.
The following table lists the fields valid for this type of step.
| Field | Description | Mandatory? | ||||
|---|---|---|---|---|---|---|
Code |
Code of the Extract step |
Yes |
||||
Sequence |
The execution sequence for this step. The integration steps get executed in an ascending order of their sequence. |
Yes |
||||
Subtype |
EXTRACT |
Yes |
||||
A dynamic logic condition to check if this step should be invoked. The step gets skipped if the condition returns false |
No |
|||||
A dynamic logic function to post process the results of the step. |
No |
|||||
Destination |
Reference to the REST destination. For more details on REST destinations refer to the "Destinations" page in the Building Blocks chapter of the Configuration Guide.
|
Yes |
||||
Output Name |
Dynamic logic bind variable name. This variable holds the data files extracted (when exists) or the response payload as received from the external system and is available as a bind variable to all the subsequent dynamic logic/steps. If the output name is not specified, then the integration step code is used as the bind variable.
|
No |
||||
Type Config |
The configuration of the extract includes supplying information about:
For details on fields/expand, refer to the "Influence Resource Representation" page of the HTTP API chapter in the Developer Guide. For more details on Oracle Health Insurance extracts functionality, refer to Extracts Integration Point in the Oracle Health Insurance application’s Developer Guide. |
Yes |
| Configuring multiple Extract steps for merging is not recommended. From a performance standpoint, considering the high volume nature of the extract, transforming/merging two high volume files is not recommended. |
Join
The grammar used for the join criteria is as follows. As a convention the join criteria left-hand side expression is the root resource and right side expression is linked resource.
join := expression
expression := [[(]simple_expression | complex_expression[)]]
simple_expression := field_name.operator(linked_resource_name.field_name)
operator := eq | neq
field_name := attribute[(+)]?[.field_name]* -- for joins
complex_expression := expression.complex_operator.expression
complex_operator := and | or
- Example: Simple Expression
-
The extract must fetch the claim and linked ctrClaim. The join between ctrClaim and claim is on code so the join criteria is a simple expression.
code.eq(code)
The left hand side refers to the claims root resource and the right hand side refers to the ctrClaims linkedResource.
- Example: Complex Expression
-
The extract must fetch policies and other versions of the policies as linkedResources. The join between policy and its versions is on gid and the id of the two is not the same, so the join criteria is a complex expression.
gid.eq(gid).and.id.neq(id)
The left-hand side refers to the policies root resource which is the last version and the right-hand side refers to policies as the linked resource to fetch the previous versions.
Incremental or Full Extract?
Extract provides the ability to do full or incremental data collection. All extracts are by default incremental in nature. The integration invocation can control whether it is full or incremental, this is done by using the invocation parameter "extractIncremental" which takes a boolean value. Setting this parameter to 'false' results in a full extract. For details on setting the parameters while invoking an integration, refer to the Exchange Integration Point in the Developer Guide.
- Incremental
-
By specifying the parameter "incrementalField" during integration invocation, it is possible to specify which field to add watermark check on. By default, this field is the root resource’s lastUpdatedDate. It can be overridden during the invocation of the extract to a different value. This field can also specify an attribute on a dependent/linked resource for example: when an incremental extract of claims is desired based on the lastUpdatedDate of servicedMember, the incrementalField is required to be sent as servicedMember.lastUpdatedDate.
Every integration invocation start time is logged with a timestamp. This current invocation’s timestamp acts as the high watermark for the incremental extract and the previous invocation’s timestamp (if any) acts as the low watermark.
If the previous invocation was not successful, the current invocation still incrementally collects data, and it is envisioned that the operator recovers/restarts the previous invocation.
The q-part or query part of the configuration in which all parameters are resolved into actual values is used as part of the criteria for retrieving the previous exchange. The following example, which periodically extracts data from Policies resource "policygroupaccounts" using the following query:
{
"resource": {
"q": "groupAccount.code.eq({code})"
}
}
demonstrates how the system takes query parts of previous invocations into account.
At time 1, Policy Group Account data is extracted for group account "LARGE_ACCOUNT" using query
{
"resource": {
"q": "groupAccount.code.eq(LARGE_ACCOUNT)"
}
}
As this is the first time an extract runs for this specific query it is not done in an incremental way.
At time 2, Policy Group Account data is extracted for group account "SMALL_ACCOUNT" using query
{
"resource": {
"q": "groupAccount.code.eq(SMALL_ACCOUNT)"
}
}
Again, as this is the first time an extract runs for this specific query it is not done in an incremental way.
At time 3, Policy Group Account data is again extracted for group account "LARGE_ACCOUNT" using query
{
"resource": {
"q": "groupAccount.code.eq(LARGE_ACCOUNT)"
}
}
As the system retrieves the invocation that ran at time 1 it applies that invocation’s timestamp as the low watermark, effectively executing query
{
"resource": {
"q": "roupAccount.code.eq(LARGE_ACCOUNT).and.groupAccount.lastUpdatedDate.gte(time 1)"
}
}
Timeout
When the extract process is invoked, the target application replies back with the location header of the extract. If a timeout is configured and when the timeout expires, the Integration Gateway framework tries to fetch the status of the extract through the location header of the invoked extract process. If the status is a final state, the process at the gateway end continues in the same way as when a notification is received. If the status is not in a final state, the process sleeps again and retries again for a maximum of three times (this can be overridden by system property ohi.timeout.maxRepeatAttempts). The timeout value can be overwritten at the time of invocation by specifying the invocation parameter < Integration Step Code>-timeAllowed (for below sample, it would be: claimsExtract-timeAllowed). This overwritten value is in milliseconds. If there is no configuration present and there is no invocation time parameter, the default value of 5 minutes applies.
| If the process resumes through a timeout and at a later point in time, notification of the same process also arrives, the notification fails as the process had already resumed, and further resuming the process is not possible anymore. |
Sample Configuration: Extract incremental claims data periodically for the data warehouse
The integration can be configured as follows:
| Field | Configuration | |
|---|---|---|
Remarks |
Integration |
code |
claimsExtract |
||
schedule |
0 0 23 * * * |
Every night at 11 PM |
Integration Step |
code |
extractStep |
Extract of all claims periodically for the data warehouse |
||
by default the extract process informs the Oracle Insurance Gateway when it completes, as extracts is a long running process. |
Set the 'extract notification endpoint' in the claims application to 'Oracle Insurance Gateway notification endpoint'. This requires the system property ohi.extract.gateway.notification.endpoint to be set to: http(s)://{host:port}/{oig application context}/notification
In the claims application, the credential and authentication property must be set for the key gateway. Here, 'gateway' is the credential key/client id based on which all endpoint and authentication information must be set in Claims Application. This is similar to the setting of credential and authentication for the key policies_user as explained in the Query step example |
|
destination |
claims |
Refers to a rest destination with code "claims" (for details refer to "policies" destination explained with Query step) |
sequence |
1 |
|
outputName |
||
claimsData |
||
type config |
resourceName: claims expand: dynamicData |
subResources |
The runtime invocation of the above configuration results in an integration invocation (exchange) log with timestamp.
| Id | Integration ID | Timestamp |
|---|---|---|
1 |
1 |
11 Jan 2019 23:00:45 |
2 |
1 |
12 Jan 2019 23:00:35 |
-
First invocation of such an extract, (id=1), takes the high watermark as 2019-01-11 23:00:45, hence it adds a selection clause to the extract query as lastUpdatedDate < {invocation_timestamp}
-
Second invocation of such an extract, (id=2), takes the high watermark as 2019-01-12 23:00:35 and low watermark as the previous invocation timestamp, hence it adds a selection clause to the extract query as lastUpdatedDate < {invocation_timestamp} and lastUpdatedDate >= {previous_invocation_timestamp}
An example of the JSON payload for the creation of such an Extract step through API is given below. Note that this JSON is part of the larger payload for the creation of the Integration.
{
"code": "extractStep",
"sequence": 1,
"subtype" : "EXTRACT",
"outputName" : "claimsData",
"typeConfig": {
"resourceName" : "claims",
"expand" : "dynamicData|subResources"
}
}
Data Transfer Step
The Data Transfer step enables you to collect data from the source Oracle Health Insurance application as a file data transfer. This step triggers the Data Transfer Integration Point in the target application and therefore offers all the possibilities supported by the data transfer integration point.
-
For more information on data transfer integration point, see Data Transfer Integration Point
-
For an example, see the Transfer Group Client Data section at Collect Data from Oracle Health Insurance Applications.
The following table lists the fields valid for this type of step.
| Field | Description | Mandatory? | ||||
|---|---|---|---|---|---|---|
Code |
The code of the Data Transfer step. |
Yes |
||||
Sequence |
The execution sequence for this step. The integration steps run in ascending order of their sequence. |
Yes |
||||
Subtype |
DATA TRANSFER |
Yes |
||||
A dynamic logic condition to check if this step should be invoked. The step gets skipped if the condition returns false. |
No |
|||||
A dynamic logic function to post process the results of the step. |
No |
|||||
Destination |
Reference to the REST destination. For more details on REST destinations, see Destinations.
|
Yes |
||||
Output Name |
Dynamic logic bind variable name. This variable holds the data files transferred (when exists) or the response payload as received from the external system and is available as a bind variable to all the subsequent dynamic logic/steps. If the output name is not specified, then the integration step code is used as the bind variable.
|
No |
||||
Type Config |
The configuration of the data transfer includes supplying information about:
For more details on Oracle Health Insurance data transfer functionality, see Data Transfer Integration Point. |
Yes |
| Pagination is not supported in OIG Data Transfer Step. |
| Configuring multiple Data Transfer steps for merging is not recommended. From a performance standpoint, considering the high volume nature of the data transfer, transforming, or merging two high volume files is not recommended. |
| There are no retries associated with the time allowed. |
Timeout
When the data transfer process is invoked, the target application replies with the location header of the data transfer. Using the location header, the system polls for the status of the data transfer. If a timeout is configured for the data transfer step and when the data transfer does not complete within the time allowed, the exchange times out. There are no retries associated with the time allowed.
The timeout value can be overwritten at the time of invocation by specifying the invocation parameter <Integration Step Code>-timeAllowed. The overwrite value is in milliseconds. If there is no configuration present and there is no invocation time parameter, the default value of 5 minutes applies.
Incremental or Full Data Transfer?
Data Transfer provides the ability to perform either a full or an incremental transfer. Data transfers are incremental by default when a matching previous exchange exists.
The integration invocation can control this behavior by using the dataTransferIncremental parameter.
This parameter takes a boolean value.
-
true: The system adds both a low watermark and a high watermark to the data transfer query. This is the default behavior. -
false: The system adds only the high watermark to the data transfer query. This supports a full data transfer first, followed by incremental transfers in later invocations.
For details on setting parameters while invoking an integration, see Exchange Integration Point in the Developer Guide.
Incremental
By specifying the incrementalField parameter during integration invocation, users can specify the field to which the watermark filter applies.
By default, this field is LAST_UPDATED_DATE on the main base view.
The value can be overridden during invocation.
The field can also refer to a field on a joined view.
For example, when an incremental transfer of claim lines is required based on the creation date of related relations, set incrementalField to RELA.CREATION_DATE.
Every integration invocation start time is logged with a timestamp. The current invocation timestamp acts as the high watermark for the data transfer. The timestamp of the previous matching exchange, if available, acts as the low watermark.
If the previous invocation was not successful, the current invocation can still collect data incrementally. The operator should recover or restart the previous invocation as needed.
The where clause part of the configuration, after all parameters are resolved into actual values, is used as part of the criteria for retrieving the previous exchange.
If the resolved where clause has not been used before for the same integration step, the system cannot determine a low watermark for that selection.
In that case, the system runs the transfer with only the high watermark and stores the resolved where clause for future incremental runs.
The following example periodically transfers data from the Relations table.
The configured where clause is:
RELA.SUBTYPE = '{subtype}'
The examples below show how the system adds watermark conditions at runtime.
At time 1, relation data is transferred for subtype "PERS" using the following where clause:
RELA.SUBTYPE = 'pers' AND RELA.LAST_UPDATED_DATE < TO_TIMESTAMP('Time 1')
As this is the first time the transfer runs for this specific resolved where clause, no low watermark is available.
The system uses only the high watermark and stores the resolved where clause for future runs.
At time 2, relation data is transferred for subtype "ORGA" using the following where clause:
RELA.SUBTYPE = 'orga' AND RELA.LAST_UPDATED_DATE < TO_TIMESTAMP('Time 2')
Again, this is the first time a transfer runs for this specific resolved where clause.
The system treats it separately from the previous "PERS" transfer and uses only the high watermark.
At time 3, relation data is again transferred for subtype "PERS" using the following where clause:
RELA.SUBTYPE = 'pers' AND RELA.LAST_UPDATED_DATE >= TO_TIMESTAMP('Time 1') AND RELA.LAST_UPDATED_DATE < TO_TIMESTAMP('Time 3')
The resolved where clause matches the transfer that ran at time 1.
The system uses the timestamp from the time 1 exchange as the low watermark and the current exchange timestamp as the high watermark.
Only records updated between those two timestamps are included.
Custom Watermarks in Data Transfer
In addition to system-derived watermarks, data transfer supports custom watermarks to provide fine-grained control over the synchronization range. Custom watermarks are implemented through exchange properties and can be set by dynamic logic or passed as Exchange API headers.
For details on passing custom watermark headers while invoking an exchange, see Exchange Integration Point in the Developer Guide.
The exchange property names must use the integration step code as a prefix:
-
<integrationStepCode>-lowWaterMark -
<integrationStepCode>-highWaterMark
For example, a user can define the following exchange properties where TRANSFER_PERS is the integration step code:
TRANSFER_PERS-lowWaterMark = 2025-08-01T00:00:00Z TRANSFER_PERS-highWaterMark = 2025-08-31T23:59:59Z
These values override the default timestamp-based filters and are injected into the transfer logic at runtime. When custom watermarks are provided, the system uses those values instead of the system-derived watermarks. This is useful when the exchange creation timestamp is not the required synchronization range, for example when rerunning a specific time period or recovering missed data.
At time 4, relation data is transferred for subtype PERS using a where clause with custom watermarks.
RELA.SUBTYPE = 'pers' AND RELA.LAST_UPDATED_DATE >= TO_TIMESTAMP('2025-08-01T00:00:00Z') AND RELA.LAST_UPDATED_DATE < TO_TIMESTAMP('2025-08-31T23:59:59Z')
The custom watermark values must use the supported date-time format, for example 2025-08-31T23:59:59Z.
If an invalid custom watermark value is provided, the exchange fails and the exchange step log includes the failure reason.
Processing
The Data Transfer process consists of three steps, each crucial for successfully transferring data between the systems. These steps ensure that data is properly sent, tracked, and collected, with built-in handling for success, failure, and timeout scenarios.
-
Invoke Data Transfer
-
Check Data Transfer Status
-
Data Transfer Collect Data
Step 1: Invoke Data Transfer
This step invokes the data transfer process in the target destination. The response is stored as a log line.
In a failure scenario, an HTTP 4xx/5xx response is received and the response is stored as a log line. For more information, see Response Messages.
Step 2: Check Data Transfer Status
This step involves checking the status of the data transfer request. The polling interval can be configured for real-time adjustments based on the environment and data size, and polling continues until the transfer completes, fails, or times out. Any failure messages are included in the extraInfo attribute in the /api/datatransfer/{id} response, which should be stored in this step.
The <Integration Step Code>-pollingInterval allows the polling frequency to be adjusted (for example, every second instead of every minute).
Scenarios
-
Success
-
Mark the exchange sub-step as DONE.
-
Store the response as a raw payload.
-
-
Failure
-
Mark the exchange sub-step as FAILED.
-
Update the exchange status to FAILED.
-
Store the response as a raw payload.
-
Generate a log line with the message from the
extraInfoattribute.
-
-
Timeout
-
Mark the exchange sub-step as FAILED.
-
Update the exchange status to TIMEDOUT.
-
Store the response as a raw payload.
-
Generate a log line with the following message.
-
Data transfer process timed out after [TIME_IN_SECONDS] seconds. To check the data transfer process status, use the GET request: /api/datatransfer/{id}. The data transfer process cannot be recovered from Oracle Insurance Gateway.
Custom Step
This step enables a user to collect data from the source applications using the Dynamic Logic. The dynamic logic provides an ability to perform external callouts using Callout Mechanism.
This step is useful to gather data when the source application is not an Oracle Health Insurance application and therefore the Query or Extract types cannot be used. This step can also be used to process data (perform DMLs) within Oracle Health Insurance using HTTP API framework, for example, process a request from a customer relationship management system for a specific member and update member data using persons API.
The following table lists the fields valid for this type of step.
| Field | Description | Mandatory? |
|---|---|---|
Code |
Code of the Custom step |
Yes |
Sequence |
The execution sequence for this step. The integration steps get executed in ascending order of their sequence. |
Yes |
Subtype |
CUSTOM |
Yes |
Condition Invocation Check |
A dynamic logic condition to check if this step should be invoked. The step gets skipped if the condition returns false |
No |
Function Post Process |
A dynamic logic function to post process the results of the step. |
No |
Destination |
Reference to REST destination. For more details on destinations refer to the "Destinations" page in the Building Blocks chapter of the Configuration Guide. |
Yes |
Output Name |
Dynamic logic bind variable name. This variable holds the raw payload returned by the dynamic logic function Data Collection Custom and is available as a bind variable to all subsequent dynamic logic/steps. If the output name is not specified, then the integration step code is used as the bind variable. |
No |
Type Config |
The configuration of the custom step includes supplying information about
|
Yes |
Sample Configuration: Getting payment status from an external financial system
Assume that a POST request with query parameters - member code, start date and end date - to a financial system endpoint 'http://{external application host:port}/{external application context}/{URI path}' returns member’s payment status history.
To achieve this the Custom step is configured as follows:
| Field | Configuration | Remarks |
|---|---|---|
code |
getPaymentStatus |
|
sequence |
1 |
Assuming this is the first step in the Integration |
destination |
financials |
Refers to a rest destination with code "financials" |
outputName |
paymentStatus |
The response can be accessed in the dynamic logic of the subsequent step using "paymentStatus" bind variable. |
type config |
customFunctionCode : paymentStatusCallout |
An example of the JSON payload for the creation of a custom step using API is given below. Note that this JSON is part of the larger payload needed to create an integration
{
"code" : "getPaymentStatus",
"sequence" : 1,
"subtype" : "CUSTOM",
"destination" : { "code" : "financials"},
"outputName" : "paymentStatus",
"typeConfig" : {
"customFunctionCode":"paymentStatusCallout"
}
}
Dynamic Logic - paymentStatusCallout is configured as:
String paymentStatus = webTarget("financials")
.queryParam("memberCode", properties.get("memberCode"))
.queryParam("startDate", properties.get("startDate"))
.queryParam("endDate", properties.get("endDate"))
.request()
.header("Accept", "application/json")
.buildGet()
.invoke()
.readEntity(String.class)
return paymentStatus
The rest destination "financials" is configured as :
| Field | Configuration | Remarks |
|---|---|---|
code |
financials |
|
credentialKey |
|
The following must to done to set up credential key "financials_user"
Example: To setup authentication using Generic API - properties (assuming the external financial application uses basic authentication) the following must be done POST http(s)://{host:port}/<oig application context>/generic/properties
with payload
For details on Generic API refer to Developer Guide, section Integration Generic API and Operations. |
addressKey |
address.key.financials.baseurl |
The address key can be setup using generic properties resource or properties page Example: To setup address key using Generic API - properties the following must be done: POST http(s)://{host:port}/{oig application context}/generic/properties
with payload:
|
typeConfig |
path: must be set to the URI Path (relative to addressKey value) httpMethod: POST |