1. Using Integrations in Oracle Integration Generation 2
  2. Troubleshoot Oracle Integration
  3. Troubleshoot Integration Runtime

Troubleshoot Integration Runtime

Learn about troubleshooting integration runtime in Oracle Integration.

Topics:

Using Basic Authentication to Call Services When Oracle Identity Cloud Service is Configured With Multi-Factor Authentication Causes a 401-Unauthorized Error

If Oracle Identity Cloud Service is configured with multi-factor authentication (for example, user name / password + multi-factor authentication) and you use basic authentication to invoke Oracle Integration services, you receive a 401-Unauthorized error. None of the adapters or the connectivity agent in Oracle Integration support multi-factor authentication.

As an alternative:
  • Disable multi-factor authentication from the account with which you are using basic authentication.
  • Use OAuth instead.

See Managing Multi-Factor Authentication.

Perform Remedial Actions When the Payload Exceeds the Size Limit

When the message payload in an integration exceeds the size limit, the message is rejected and an error message is displayed. The error message describes remedial actions to perform.

Issue Error Message Remedial Action to Perform
Payload size limit exceeded
  • When REST, File, SOAP, FTP, and SAP Adapter connections that support payload sizes of up to 50 MB exceed the limit:
    CASDK-0059: Unable to process the response received from the client application.
 The response received by the OIC integration flow is rejected. 
The response received has been rejected since the content received of length 5723727000 
bytes exceeds the maximum allowed threshold of 52428800 bytes (50MB).
  • When all other adapter connections that support payload sizes of up to 10 MB exceed the limit:
    CASDK-0059: Unable to process the response received from the client application.
 The response received by the OIC integration flow is rejected. 
The response received has been rejected since the content received of length 1723727000 bytes exceeds the maximum allowed threshold of 10485760 bytes (10MB).
 The preferred way to send large data sets is by using attachments. Check whether the target application exposes a SOAP API to return data sent as an attachment. If yes, the integration can be designed to configure an invoke connection using the SOAP Adapter to accept an MTOM attachment. The attachment can be an archive containing an XML/comma-separated value (CSV) document.
Attachment size limit exceeded 
CASDK-0059: Unable to process the response received from the client application. 
The response received by the OIC integration flow is rejected.
The response received has been rejected since the content received of length  
bytes exceeds the maximum allowed threshold of 1073741824 bytes (1GB).
 Check whether the attachment can be broken and sent through multiple APIs. For example, if the attachment is a ZIP archive containing a large CSV file and unstructured documents, have one API that returns the CSV file and another API that retrieves the unstructured documents.

For additional details about 50 MB payload support, limits, and best practices, see Service Limits in Provisioning and Administering Oracle Integration Generation 2.

Remove Unprintable Control Characters from the Incoming Payload

Oracle Integration does not support certain non-printable control characters in the following range. 

['\u0000', '\u0001', '\u0002', '\u0003', '\u0004', '\u0005', '\u0006',
'\u0007', '\u0008', '\u000b', '\u000c', '\u000e', '\u000f', '\u0010', '
\u0011', '\u0012', '\u0013', '\u0014', '\u0015', '\u0016', '\u0017', '\
u0018', '\u0019']
The presence of these control characters in the incoming payload results in the following error during processing:
{
  "type" : "10.4.1",
  "title" : "exception occurred while translating the request from native to
xml.",
  "detail" : "oracle.cloud.connector.api.CloudInvocationException:
ORABPEL-15235\n\nTranslation Failure.\nFailed to translate JSON to XML.
org.codehaus.jackson.JsonParseException: Illegal unquoted character
((CTRL-CHAR, code 1)): has to be escaped using backslash to be included in
string value\n at [Source: java.io.BufferedReader@.#; line: 2, column:
22]\nThe incoming data does not conform to the NXSD schema. Please correct
the problem.\n",
  "o:errorCode" : "TRANSLATION-ERROR-00",
  "o:errorDetails" : [ {
    "type" : "NA",
    "instance" : "NA",
    "title" : "NA",
    "o:errorPath" : "NA",
    "o:errorCode" : "NA"
  } ]
}

If you receive this error, contact Oracle Support for information on how to preprocess incoming JSON payloads and filter unprintable control characters.

Notification Action Returns a <from> value is invalid Error

If the notification action in an orchestrated integration validates successfully, but the following error occurs during integration runtime, the value specified in the Parameter Value of the notification action may not be correct. 

:
summary=<summary>The <from> value is invalid.
Xpath expression associated with <from> in copy assign activity is invalid.
There is an user error that results in missing element value(s) in the xpath query.
Please review the payload and modeling to ensure that all elements defined in the <from> xpath query have valid non-null values .
</summary>
:
,query=<query>fn:current-dateTime()</query>

The error can be confirmed by viewing errors for the integration instance on the Track Instances page or the response of a REST testing tool such as Postman.

Verify that the value specified in the Parameter Value of the notification action is correct. For example, this value may require conversion to a string:
string(fn:current-dateTime())

Synchronous Integrations Occasionally Fail with a 504 Gateway Timeout Error

Synchronous integrations can fail with the following error. 

Bad response: 504 Gateway Time-out from url
https://Instance_URL/Integration_Name/Version_Number/

This error occurs because synchronous integrations (integrations that return a response payload) are limited to 300 seconds and return a timeout error if they exceed that limit.

Optimize your integration and/or the endpoint you invoke to complete instance execution in under 300 seconds.

See Service Limits in Provisioning and Administering Oracle Integration Generation 2.

Invalid <From> Value When Using a Fault Handler in an Integration

An integration configured with a fault handler can fail with the following error indicating that the <from> value is invalid. 

ICSWC-0001) :
Error message is - The <from> value is invalid.
Xpath expression associated with <from> in copy assign activity is invalid.
There is a user error that results in missing element value(s) in the xpath
query.
Please review the payload and modeling to ensure that all elements defined in
the <from> xpath query have valid non-null values.

The fault format indicates that the endpoint configuration predates the fault handling infrastructure. Therefore, this fault cannot be handled inside a fault handler associated with a scope or default handler. This occurs when you import a pre-17.4.5 integration from Oracle Integration Cloud Service into Oracle Integration. 


<soap-env:Fault xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">
<faultcode
xmlns:faultsrc="http://xmlns.oracle.com/cloud/adapter/REST/GetLocation_REQUEST">faultsrc:APIInvocationError</faultcode>
<faultstring>ICS runtime execution error</faultstring>
<detail>
<ICSfaultVar/>
<reason><genericRestFault><errorCode>400</errorCode><er
rorPath&gt;&lt;![CDATA[GET
https://myinsance.com/maxrest_b1dk/rest/os/ICSLOCATION?_lid=icsuser&_lpwd=M%24lee019&LOCATION=101104&STATUS=OPERATING returned a
response status of 400 Bad
Request]]></errorPath><instance><![CDATA[Error 400:
BMXAA7901E - You cannot log in at this time. Contact the system
administrator. ]]></instance></genericRestFault&gt;
:Application Error</reason>
<operation>execute</operation>
<request_payload/>
</detail>
</soap-env:Fault>

Solution

Do not modify the integration to include fault handling. If you need to modify, edit the adapter endpoint configuration. After editing the adapter endpoint configuration, the integration is updated to throw a typed fault conforming to the APIInvocationError format mentioned below. This enables the fault to be handled inside fault handlers. 


<soap-env:Fault xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">
<faultcode
xmlns:faultsrc="http://xmlns.oracle.com/cloud/adapter/REST/InvokeMonitoring_REQUEST">faultsrc:APIInvocationError</faultcode>
<faultstring>ICS runtime execution error</faultstring>
<detail>
<ICSfaultVar>
<nstrgdfl:APIInvocationError
xmlns:nstrgdfl="http://xmlns.oracle.com/cloud/generic/rest/fault/REST/InvokeMonitoring">
<nstrgdfl:type/>
<nstrgdfl:title/>
<nstrgdfl:detail/>
<nstrgdfl:errorCode/>
<nstrgdfl:errorDetails>
<nstrgdfl:type>http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1</nstrgdfl:type>
<nstrgdfl:title>Bad Request</nstrgdfl:title>
<nstrgdfl:errorCode>400</nstrgdfl:errorCode>
<nstrgdfl:errorPath>&lt;![CDATA[POST
http://myinstance.com:7003/ic/api/integration/v1/monitoring/errors/(id)/resubmit returned a response status of 400 Bad
Request]]></nstrgdfl:errorPath>
<nstrgdfl:instance>&lt;![CDATA[{"status":"HTTP 400 Bad
Request","title":"Invalid ID format. Please provide only
numbers.","type":"https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1"}.A 400 Bad Request Error indicates that the target service is unable
(or refuses) to process the request sent by the client (Oracle Integration
Cloud), due to an issue that is perceived by the server to be a client
problem. You can trace the cURL representation of the request sent to the
target service from the Oracle Integration Cloud server logs. Try invoking
the target service using cURL. It may also be that one of the intermediaries
(proxy, LBR) could be returning this code. ]]&gt;</nstrgdfl:instance>
</nstrgdfl:errorDetails>
</nstrgdfl:APIInvocationError>
</ICSfaultVar>
<reason>Fault Details : <nstrgdfl:APIInvocationError
xmlns:nstrgdfl="http://xmlns.oracle.com/cloud/generic/rest/fault/REST/InvokeMonitoring"><nstrgdfl:type/><nstrgdfl:title/><nstrgdfl:detail/
><nstrgdfl:errorCode/><nstrgdfl:errorDetails&gt;&lt;nstrgdfl:type&
gt;http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1</nstrgdfl:type><nstrgdfl:title>Bad
Request</nstrgdfl:title><nstrgdfl:errorCode>400</nstrgdfl:error
Code><nstrgdfl:errorPath&gt;&amp;lt;![CDATA[POST
http://myinstance.com:7003/ic/api/integration/v1/monitoring/errors/(id)/resubmit returned a response status of 400 Bad
Request]]></nstrgdfl:errorPath&gt;&lt;nstrgdfl:instance&gt;&amp;lt;!
[CDATA[{"status":"HTTP 400 Bad Request","title":"Invalid ID format. Please
provide only
numbers.","type":"https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1"}.A 400 Bad Request Error indicates that the target service is unable
(or refuses) to process the request sent by the client (Oracle Integration
Cloud), due to an issue that is perceived by the server to be a client
problem. You can trace the cURL representation of the request sent to the
target service from the Oracle Integration Cloud server logs. Try invoking
the target service using cURL. It may also be that one of the intermediaries
(proxy, LBR) could be returning this code.
]]></nstrgdfl:instance></nstrgdfl:errorDetails&gt;&lt;/nstrgdf
l:APIInvocationError> :Application Error</reason>
<operation>execute</operation>
<request_payload/>
</detail>
</soap-env:Fault>

Activity Stream Logs Deleted

The activity stream logs in Oracle Integration can be automatically deleted under certain scenarios. These scenarios are described below, along with links to documentation that describes how to capture the activity stream logs in the Oracle Cloud Infrastructure Console, enable tracing with the payload and view the activity stream logs on the Track Instance page, and download the activity stream logs with the REST APIs.

Automatic Rotation of the Activity Stream Log

Activity stream logs in Oracle Integration can be automatically deleted under the following scenarios:

  • You have too many integrations with tracing and payload logging enabled.
  • You have too many instances of an integration.

The activity stream log size is 100 MB. If either of these scenarios occurs, the file is rotated and you do not see the activity stream. Oracle does not recommend enabling tracing and payload logging in production environments.

Instead, design an integration that can push the payload and other information such as instance ID, user, integration name, code, version, and so on to the storage location of your choice and call this integration from critical integrations through a local invoke connection to persist the log information.

The activity stream payload data is derived from the ics-flow.log files that are created on the integration instance. These files are rotated after they reach a size of 10 MB, and only 10 of these files are retained. There is no option to reconfigure the file size or file rotation settings. Depending on the number of transactions (instance IDs generated) and payload sizes, the flow log files can be rotated frequently. This limits the time window during which the activity stream can show payload and trace data. This is the expected behavior when selecting the Enable tracing and Include payload options during integration activation.

There is no guarantee that the activity stream payload data is persisted or available for a certain amount of time. The availability of activity stream data depends on your integrations. The Enable tracing and Include payload options are intended for design-time and troubleshooting use. Use these options in a production only when absolutely necessary. As a best practice, set business identifiers to track individual fields of the payload for monitoring and debugging purposes. These tracking fields are displayed for each integration instance in the Track Instances page and are retained for the entire instance retention period.

To reduce the size of integration logs (and thereby increase payload data availability in the activity stream), you can do the following:
  • Only activate your integrations using the Enable tracing and Include payload options when absolutely required (for debugging purposes).
  • Do not set the Include payload option at activation for integrations that handle very large payloads or a high number of payloads (for example, through a looping mechanism).

Capture the Activity Stream in the Oracle Cloud Infrastructure Console

Documentation is available that describes how to capture the activity stream in the Oracle Cloud Infrastructure Console:

Enable Tracing with the Payload and View the Activity Stream on the Track Instances Page

Documentation is available that describes how to enable tracing with the payload and view the activity stream for orchestrated integrations on the Track Instances page:

Download the Activity Stream Logs with the REST APIs

Documentation is available that describes how to download activity stream logs in Oracle Integration or with the REST APIs:

Note:

If you decide to use absolute values (10 MB, instead of a certain size), downloading the file downloads the latest 10 MB file only. The rotated files are not available for download.

Message Dequeuing Takes Three to Four Seconds in Asynchronous Integrations

You may observe that Oracle Integration sometimes takes three to four seconds to go from Message Received:Instance created and enqueued for processing to message dequeued and processed by Trigger trigger_name. This increases the end-to-end processing time of a particular instance.

This is the expected behavior with asynchronous requests. Asynchronous requests are processed using a queue. The first two entries in the flow logs show the enqueue and dequeue times. Therefore, a few seconds lag is expected.

403 Error When Accessing an API Protected Using OAuth

OAuth allows Oracle Integration to access a user's resources on their behalf. REST and other REST-based adapters (such as Google Calendar, Microsoft Email, and others) often access APIs that are protected using OAuth. If these calls result in a 403 error (forbidden), it usually implies a lack of permissions to access the API.

You need to carefully examine your OAuth configuration and work with the target API owner/administrator to resolve this issue. There may be several conditions leading to this, but the exact cause is best described by the authorization server. Some probable causes of 403 errors are as follows:

  • The access token was procured for a scope that does not cover the API being accessed.

  • During the phase to provide consent, the user that provided the consent is not the resource owner or owner of the API being accessed.

  • The access token expired or is not valid to access the resource any longer.

  • The authorization server revoked access privileges to a particular resource.

Troubleshoot Certificate Import Failures

The certificate that you receive from a Certificate Authority (CA) such as Verisign, Entrust, or others can fail during import into Oracle Integration with the following error in the log file: 

java.security.cert.CertificateException: Unable to initialize,
java.io.IOException: extra data given to DerValue constructor
To resolve this issue, either:

  • Obtain a fixed certificate from the CA.

  • Convert the certificate to a Distinguished Encoding Rules (DER) binary certificate using tools such as openssl. For example: 

    openssl x509 -outform der -in sfdc-client.crt -out sfdc-client.der