Connector Error Handling and Recovery
This page documents steps to recover a connector from a failed state using connector configurations. A connector runs in the background using the task processing framework to check entities for any change and sends messages to an external system for every change. The functionality of a connector has three stages that are active with the application:
-
Aggregating Change Events: Translates change events into aggregate events periodically.
-
Transforming Aggregate Events: Creates an activity for each unprocessed aggregate event. The activity transforms the event into a message and publishes that message.
-
Re-publishing Messages: Picks up messages that failed delivery and re-publishes them.
This process is continuous and vulnerable to errors.
Check on System Properties
You can check the system properties of a connector:
Stage | System Properties | Default | Additional Information |
---|---|---|---|
Aggregating Change Events |
ohi.connector.event.aggregation.activated |
False |
You need to change it to |
Aggregating Change Events |
ohi.connector.event.aggregation.interval |
300 |
For every 300 seconds, the connector checks new change events to create new aggregate events. |
Re-publishing Messages |
ohi.connector.message.republishing.activated |
False |
You need to change it to |
Re-publishing Messages |
ohi.connector.message.republishing.interval |
300 |
For every 300 seconds, the connector checks for the failed messages to republish. |
Message publishing |
ohi.connector.configuration.<0>.httpmethod |
POST |
If the connector sends messages in a PUT request, you must change the default operation to PUT. These operations are specific to the connector configuration. |
Message publishing |
ohi.connector.configuration.<0>.uri |
You need to set it to the URL endpoint where the messages are published. The URL is specific to the connector configuration. |
Check the Lifespan of a Task
Different tasks run for each stage. You can check how long a task has been running by looking at the creation date of the latest task and the lifespan of the task. A connector creates a new task every five seconds over the lifespan of a task. If a task lasts longer than its lifespan, it can be a reason for a failing connector configuration.
Stage | Number of Tasks | Task Type | Subject ID | Lifespan |
---|---|---|---|---|
Aggregating Change Events |
1 |
IntegrationEventAggregateCreator |
-1 |
The value of the |
Transforming Aggregate Events |
1 per enabled connector |
IntegrationTransformation |
id of Connector Configuration |
The time interval of the connector configuration. |
Re-publishing Messages |
1 |
RepublishingFailedMessages |
-1 |
The value of the |
Check the Latest Task
It is important to check for the latest task to determine whether the task is stuck anywhere. The connector must create new tasks periodically.
The lifespan varies for each task. For tasks in aggregate and republishing stages, the lifespan is available in a system property (see Check on System Properties section). For connector configuration tasks, the lifespan is available in the time-interval column of the connector configuration.
Check the Latest Task with Database Access
-
You can run the following query to fetch the last ten aggregate tasks:
select * from ohi_tasks where taty_id = 8720 -- IntegrationEventAggregateCreator task type order by creation_date desc fetch first 10 rows only ;
-
You can run the following query to fetch the last ten tasks for each connector configuration:
select tasks.* from ( select tasks.*, row_number() over(partition by tasks.subject_id order by tasks.creation_date desc) rn from ohi_tasks tasks where tasks.subject_id in (select id from int_connector_configurations where ind_enabled = 'Y') ) tasks where tasks.rn <= 10 order by tasks.creation_date desc ;
-
You can run the following query to fetch the last ten republishing tasks:
select * from ohi_tasks where taty_id = 8920 -- RepublishingFailedMessages task type order by creation_date desc fetch first 10 rows only ;
Check the Latest Task without Database Access
-
You can run the following query to fetch the latest aggregate tasks:
http://[hostName]:[portNumber]/[api-context-root]/generic/tasks?q=taskType.id.eq(8720)&orderBy=creationDate:desc&limit=10
-
You can run the following query to fetch the last ten tasks for each connector configuration: First, check the ID of the enabled connector configurations and then check their corresponding statuses.
http://[hostName]:[portNumber]/[api-context-root]/generic/connectorconfigurations?q=enabled.eq('true') http://[hostName]:[portNumber]/[api-context-root]/generic/tasks?q=subjectId.eq('[CONNECTOR_CONFIGURATION_ID]')&orderBy=creationDate:desc&limit=10
-
You can run the following query to fetch the last ten republishing tasks:
http://[hostName]:[portNumber]/[api-context-root]/generic/tasks?q=taskType.id.eq(8920)&orderBy=creationDate:desc&limit=10
Check the Task Status
It is important to know the status of the tasks running in the connector.
The tasks are in the PROCESSING
status when active and in the PENDING
status otherwise.
When a task is in the ERRORED
or COMPLETED
status, it means that the task is inactive.
-
You can use the TaskProcessing IP to reactivate an inactive task. See Recover from Application Failures for more information.
-
There are cases when the tasks are in the
PENDING
status as they are stuck in the Oracle AQ. You can run the following query to check the queue and for how long that task has been there.The time zone of the database may be different from your local timezone. It is important to consider the time zone when determining the lifespan of a task. The task’s duration in the queue must not be longer than five seconds over the lifespan of a task. select otqt.* , otqt.user_data.priority , otqt.user_data.task_id , otqt.user_data.delay , otqt.user_data.ref_code , otqt.user_data.task_name , otqt.user_data.subject_code from ohi_task_queue_table otqt where otqt.user_data.task_id = '[TASK_ID]' order by otqt.enq_time desc ;
-
Following are some ways to restart a task when it is not active:
Table 3. Restarting Tasks Status Default Behaviour Possible Causes Steps to Restart PENDING
The task is in the queue and waits to be picked up. The duration of the status must last five seconds over the lifespan of the task.
This shows issues with the Oracle AQ.
If the task is in AQ, check the KM note: State of the Messages in Oracle AQ Are Not Changed from WAITING to READY After the Delay Expires. If the task is not in the AQ, restart the task using the Task Recovery IP.
PROCESSING
The task is active and lasts a few seconds.
There can be issues with the database connectivity.
Restart the latest task using the Task Recovery IP.
ERRORED
unknown
You need to restart the latest task with the Task Recovery IP.
Other Checks
Checks with Database Access
Following is a list of steps you can perform to debug errors in tasks:
-
You can run the following query that uses the ID of the changed entity to check whether the connector creates an aggregate event for a change:
select * from int_aggregate_change_events where aggregate_id = [entity_ID];
If there are no aggregate events, check the task’s lifespan and status, system properties, and the change event rule of the message type.
-
You can run the following query with the ID of an aggregate event to check if the connector is creating an activity:
select acty.* from act_activities acty , int_assigned_aggr_events asae where asae.acet_id = [aggregate_event_ID];
If the connector is not creating the activity, check the duration of the tasks.
-
You can run the following query that uses the ID of an aggregate event to check whether the connector is creating a message for the change:
select * from int_outbound_messages where acet_id = [aggregate_event_ID];
If the connector is not creating any outbound message, check the message transformation logic that is not creating messages for this change.
-
You can run the following query that uses the ID of an activity to check whether the connector is publishing messages:
select * from int_message_publish_results where acty_id = [activity_ID];
Message Publishing Results help with determining successful publishing and thereafter, successful republishing of a message. There are links to an incident file that offer more information.
Checks to Perform without Database Access
-
You can run the following query that uses the ID of a changed entity to check whether the connector is creating an aggregate event:
http://[hostName]:[portNumber]/[api-context-root]/generic/aggregatechangeevents?q=aggregateId.eq('[ENTITY_ID]')
If there are no aggregate events, you can check the task’s lifespan and status, system properties, or the change event rule of type message for this change.
-
You can run the following query that uses the ID of an aggregate event to check if the connector is creating an activity:
http://[hostName]:[portNumber]/[api-context-root]/generic/activities?q=activityType.id.eq('1107420')&orderBy=creationDate:desc
If the connector is not creating an activity, you can check the duration of the task. The API does not filter on aggregate change events. You need to compare time of the change and the time when the connector creates the activity based on the lifespan of the tasks.
-
You can run the following query that uses the ID of an aggregate event to check if the connector is creating messages:
http://[hostName]:[portNumber]/[api-context-root]/generic/outboundmessages?q=changeEvent.id.eq('AGGREGATE_EVENT_ID')
If the connector is not creating outbound messages, you can check the message transformation logic that is not creating a message for a change.
-
You can run the following query that uses the ID of an activity to check if the connector is publishing results:
http://[hostName]:[portNumber]/[api-context-root]/generic/messagepublishingresults?q=activity.id.eq('[ACTIVITY_ID')
You can check the Message Publish Results for information on successful publishing and thereafter, republishing of a message. There are links to the incident file that offer more information.