Troubleshoot Integration Runtime

An Integration Runs Slowly

Several common issues can result in performance issues for an integration. For example, when the target application takes longer than expected to respond, the integration also runs slower than expected.

How to Determine If Your Integration Is Running Slowly

See Determine an Integration's Typical Processing Times, and then take the appropriate action:

If the integration has always run slower than expected, determine whether it contains a common pitfall.

See Common Integration Pattern Pitfalls and Design Best Practices.
If the integration has detectable performance outliers, follow the troubleshooting steps on this page.

Step 1. Investigate the Target Application

The target application for an invoke connection might be taking longer than usual to respond. A slow target application is the most common cause of slow-running integrations.

To determine whether the target application caused the slowness for your integration, check the logs of the application. The logs help you determine whether the application was experiencing an issue when Oracle Integration connected to it.

Step 2. Check the Number of Incoming Messages

When an integration receives a large number of messages to process, Oracle Integration adds the messages to a queue and processes them until it clears the queue. A long queue can impact the overall performance of your system. For example, a retailer might experience higher message volume during a promotion.

To see how many messages Oracle Integration processed in the last hour:

In the navigation pane, click Observability, then Integrations.
Click Filter , and update the following filtering requirements:
- Time window: Last 1 hour.
- Status: All.
Click Apply.
Review the numbers in the Processed column.

Keep in mind that the number of messages is just one factor in performance. Smaller messages that contain very large incoming payloads can also impact performance.

Step 3. Determine Whether the Message Counts Are Too High

Work with an administrator to check whether your organization purchased enough message packs for the messages that you're using. See Edit the Edition, License Type, and Message Packs of an Instance in Provisioning and Administering Oracle Integration 3.
If the integration is synchronous, determine whether the integration is trying to process too many requests at a time. See Calculate Requests Per Second in Provisioning and Administering Oracle Integration 3.

Step 4. Check for a Clustered Schedule (Schedule Integrations Only)

Schedule integrations can run slower than expected when too many integrations start at the same time.

To determine whether your schedule integrations are clustered together, see View the Calendar of Schedule Integration Runs.

Step 5. Look For Inefficiencies

If the integration processes records, you might be able to fine-tune the efficiency of the processing.

Action for processing records	Recommendations
For-each action	You typically can process 10,000 or fewer records using the for-each action without experiencing performance issues. To process 10,000 or more records, use a for-each action within a parallel action. The parallel action allows you to process the records in parallel threads, thereby improving the performance of the integration.
Stage file action	You typically can read and write 10,000 or fewer records within a file using the stage file action without experiencing timeouts. To process a file with 10,000 or more records, use a stage file action within a parallel action. The parallel action allows you to read and write the records in chunks, thereby improving the performance of the integration.

Action for processing records

Recommendations

For-each action

You typically can process 10,000 or fewer records using the for-each action without experiencing performance issues.

To process 10,000 or more records, use a for-each action within a parallel action. The parallel action allows you to process the records in parallel threads, thereby improving the performance of the integration.

Stage file action

You typically can read and write 10,000 or fewer records within a file using the stage file action without experiencing timeouts.

To process a file with 10,000 or more records, use a stage file action within a parallel action. The parallel action allows you to read and write the records in chunks, thereby improving the performance of the integration.

Haven't Solved the Problem?

Get help from Oracle by entering a service request.

Fewer Integration Instances Ran Than Usual

Consider a scenario in which 100 integration instances typically run by a specific time of day, but only 80 integration instances have run by that time today. You need to quickly determine what happened with your missing integration instances.

Note:

The Observability pages show information about the integrations that are running and have already run, not any missing instances. To identify whether any integration instances are missing, you must know the integrations that usually run.

Why Integrations Don't Run

An integration might not run for a number of reasons. For example:

Someone deactivated an integration.
The external application that triggers an integration is down.
Someone changed the schedule for a schedule integration, such as changing it from running hourly to running daily.

To Troubleshoot Missing Integration Instances:

Determine how many integration instances have run so far.
1. In the navigation pane, click Observability, then Instances.
2. Click Filter , and update the filtering requirements for the page.
  
  For instance, update the time window to the last 1 hour, 6 hours, or 1 day.
3. At the top of the page, review the count of the number of integration instances that ran.
If fewer integration instances have run than expected, troubleshoot the missing instances.
For instance:
- Check whether the integrations are active.
- Check whether an external application that triggers an integration is down.
- Check whether someone changed the schedule for a schedule integration.
- If needed, consult the audit trail to determine who last updated an integration.
  
  See Check the Audit History for an Integration or Other Component.

Using Basic Authentication to Call Services When Oracle Identity Cloud Service is Configured With Multi-Factor Authentication Causes a 401-Unauthorized Error

If Oracle Identity Cloud Service is configured with multi-factor authentication (for example, user name / password + multi-factor authentication) and you use basic authentication to invoke Oracle Integration services, you receive a 401-Unauthorized error. None of the adapters or the connectivity agent in Oracle Integration support multi-factor authentication.

As an alternative:

Disable multi-factor authentication from the account with which you are using basic authentication.
Use OAuth instead.

See Managing Multi-Factor Authentication.

Perform Remedial Actions When the Payload Exceeds the Size Limit

When the message payload in an integration exceeds the size limit, the message is rejected and an error message is displayed. The error message describes remedial actions to perform.

Issue	Error Message	Remedial Action to Perform
Payload size limit exceeded	When REST, File, SOAP, FTP, and SAP Adapter connections that support payload sizes of up to 50 MB exceed the limit: `CASDK-0059: Unable to process the response received from the client application. The response received by the OIC integration flow is rejected. The response received has been rejected since the content received of length 5723727000 bytes exceeds the maximum allowed threshold of 52428800 bytes (50MB).` When all other adapter connections that support payload sizes of up to 10 MB exceed the limit: `CASDK-0059: Unable to process the response received from the client application. The response received by the OIC integration flow is rejected. The response received has been rejected since the content received of length 1723727000 bytes exceeds the maximum allowed threshold of 10485760 bytes (10MB).`	The preferred way to send large data sets is by using attachments. Check whether the target application exposes a SOAP API to return data sent as an attachment. If yes, the integration can be designed to configure an invoke connection using the SOAP Adapter to accept an MTOM attachment. The attachment can be an archive containing an XML/comma-separated value (CSV) document.
Attachment size limit exceeded	`CASDK-0059: Unable to process the response received from the client application. The response received by the OIC integration flow is rejected. The response received has been rejected since the content received of length bytes exceeds the maximum allowed threshold of 1073741824 bytes (1GB).`	Check whether the attachment can be broken and sent through multiple APIs. For example, if the attachment is a ZIP archive containing a large CSV file and unstructured documents, have one API that returns the CSV file and another API that retrieves the unstructured documents.

For additional details about 50 MB payload support, limits, and best practices, see Service Limits in Provisioning and Administering Oracle Integration 3.

Remove Unprintable Control Characters from the Incoming Payload

Oracle Integration does not support certain non-printable control characters in the following range.

['\u0000', '\u0001', '\u0002', '\u0003', '\u0004', '\u0005', '\u0006',
'\u0007', '\u0008', '\u000b', '\u000c', '\u000e', '\u000f', '\u0010', '
\u0011', '\u0012', '\u0013', '\u0014', '\u0015', '\u0016', '\u0017', '\
u0018', '\u0019']

The presence of these control characters in the incoming payload results in the following error during processing:

{
  "type" : "10.4.1",
  "title" : "exception occurred while translating the request from native to
xml.",
  "detail" : "oracle.cloud.connector.api.CloudInvocationException:
ORABPEL-15235\n\nTranslation Failure.\nFailed to translate JSON to XML.
org.codehaus.jackson.JsonParseException: Illegal unquoted character
((CTRL-CHAR, code 1)): has to be escaped using backslash to be included in
string value\n at [Source: java.io.BufferedReader@.#; line: 2, column:
22]\nThe incoming data does not conform to the NXSD schema. Please correct
the problem.\n",
  "o:errorCode" : "TRANSLATION-ERROR-00",
  "o:errorDetails" : [ {
    "type" : "NA",
    "instance" : "NA",
    "title" : "NA",
    "o:errorPath" : "NA",
    "o:errorCode" : "NA"
  } ]
}

If you receive this error, contact Oracle Support for information on how to preprocess incoming JSON payloads and filter unprintable control characters.

Notification Action Returns a <from> value is invalid Error

If the notification action in an integration validates successfully, but the following error occurs during integration runtime, the value specified in the Parameter Value field of the notification action may be incorrect.

:
summary=<summary>The <from> value is invalid.
Xpath expression associated with <from> in copy assign activity is invalid.
There is an user error that results in missing element value(s) in the xpath query.
Please review the payload and modeling to ensure that all elements defined in the <from> xpath query have valid non-null values .
</summary>
:
,query=<query>fn:current-dateTime()</query>

The error can be confirmed by viewing errors for the integration instance on the Instances page or the response of a REST testing tool such as Postman.

Verify that the value specified in the Parameter section of the notification action is correct. For example, this value may require conversion to a string:

string(fn:current-dateTime())

Synchronous Integration Timeout Failure

If a synchronous integration fails to complete within 300 seconds, the following error appears in the activity stream that is viewable from the Instances page or Errors page.

CompositeException in: io.reactivex.internal.operators.single.SingleOnErrorReturn$OnErrorReturn.onError2 exceptions occurred

No details are returned when the call is processed (for example, no 502 status error is returned).

Optimize your integration and/or the endpoint you invoke to complete instance processing in under 300 seconds.

See Service Limits in Provisioning and Administering Oracle Integration 3.

Activity Stream Details

You can enable tracing with the payload, view the activity stream logs on the Instances page, and download the activity stream logs with the REST APIs.

Enable Tracing with the Payload and View the Activity Stream on the Instances Page

Documentation is available that describes how to enable tracing with the payload and view the activity stream for integrations on the Instances page:

Download the Activity Stream Logs with the REST APIs

Documentation is available that describes how to download activity stream logs in Oracle Integration or with the REST APIs:

See Retrieve Activity Stream Messages
See Download a Log File (by specifying the icsflowlog parameter)

Note:

If you decide to use absolute values (10 MB, instead of a certain size), downloading the file downloads the latest 10 MB file only.

Message Dequeuing Takes Three to Four Seconds in Asynchronous Integrations

You may observe that Oracle Integration sometimes takes three to four seconds to go from Message Received:Instance created and enqueued for processing to message dequeued and processed by Trigger trigger_name. This increases the end-to-end processing time of a particular instance.

This is the expected behavior with asynchronous requests. Asynchronous requests are processed using a queue. The first two entries in the flow logs show the enqueue and dequeue times. Therefore, a few seconds lag is expected.

403 Error When Accessing an API Protected Using OAuth

OAuth enables Oracle Integration to access a user's resources on their behalf. REST and other REST-based adapters (such as Google Calendar Adapter, Microsoft Office 365 Outlook Adapter, and others) often access APIs that are protected using OAuth. If these calls result in a 403 error (forbidden), it usually indicates a lack of permissions to access the API.

You need to carefully examine your OAuth configuration and work with the target API owner/administrator to resolve this issue. There may be several conditions leading to this, but the exact cause is best described by the authorization server. Some probable causes of 403 errors are as follows:

The access token was procured for a scope that does not cover the API being accessed.
During the phase to provide consent, the user that provided the consent is not the resource owner or owner of the API being accessed.
The access token expired or is not valid to access the resource any longer.
The authorization server revoked access privileges to a particular resource.

Troubleshoot Certificate Import Failures

The certificate that you receive from a Certificate Authority (CA) such as Verisign, Entrust, or others can fail during import into Oracle Integration with the following error in the log file:

java.security.cert.CertificateException: Unable to initialize,
java.io.IOException: extra data given to DerValue constructor

To resolve this issue, either:

Obtain a fixed certificate from the CA.
Convert the certificate to a Distinguished Encoding Rules (DER) binary certificate using tools such as openssl. For example:
```
openssl x509 -outform der -in sfdc-client.crt -out sfdc-client.der 
```