Troubleshoot Connectivity Agent Issues

This section describes how to troubleshoot and resolve connectivity agent issues in Oracle Integration.

Connectivity Agent Installation Error if /etc/hosts File is Not Configured

During connectivity agent installation, if the agent installer cannot find details about the host name/virtual machine (VM) on which the agent is being installed, you may receive the following error:
Caused by: java.net.UnknownHostException: : Unknown
name or service

Resolve this error by adding an entry for the IP address and host name in the /etc/hosts file of the host name/VM.

Restrictions on Using the Stage File Action with the Connectivity Agent

Be aware of several restrictions when using some operations of the stage file action with the connectivity agent. See Restrictions on Using Stage File Action Operations with the File/Attachment Features of the Connectivity Agent.

Connectivity Agent Installation Fails When Run as a Federated User

Connectivity agent installation fails when run as a federated user. If you rely on user federation, you must create a nonfederated user to use specifically for installing the connectivity agent. This user enables the connectivity agent to communicate with Oracle Integration.

Note:

If you ever need to restart the connectivity agent, you must ensure that the username/password credentials for this user are still valid.

Java Memory Errors

The agent Java Virtual Machine can stop running when allocating memory with the following seemingly out-of-memory error:
There is insufficient memory for the Java Runtime Environment to continue.
-----------------------------
Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x00007f6847afd000, 12288, 0) failed; error='Cannot
allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing
reserved memory.
# An error report file with more information is saved as:

However, this issue is unrelated to the out-of-memory error that is sometimes observed when the Java heap is not large enough.

This error occurs when Java requests more memory from the operating system, which does not have any:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map

This may be related to a physical machine or server or virtual machine resources on the physical machine/virtual machine/server on which the agent is executing. As an example, when the agent is running on the same virtual machine as the database, and the database may be consuming most of the resources. Oracle recommends that you set up the agent on a separate compute.

Authorization Error When Restarting the Agent

If you manually stop the agent and attempt to restart it, and receive an authorization error, ensure that the user name and password used to start the agent in the InstallerProfile.cfg file are correct. This error can occur if the password for this user name expired and was changed in the My Services Console by the administrator, but was not updated in the InstallerProfile.cfg file. This task is only required if you manually stop and restart the agent. This task is not required for agent upgrades, which occur automatically and do not use these credentials.

Add or Change the Non-Proxy Host Configuration After Installation in the CpiAgent.properties File

If you need to add, change, or bypass the non-proxy host configuration after agent installation, do not edit the InstallerConfing.cfg file. Proxy host changes made to that file after agent installation do not take effect. Instead, update the host with the proxy_nonProxyHosts parameter in the Agent_Installation_Location/agenthome/agent/config/CpiAgent.properties file for your changes to take effect. After editing this file, restart the agent.

Class Loading Conflict When Sharing the JDK Instance with Another Product

When using the connectivity agent, the following error occurs because your JDK instance is shared with another product for which JAR files have been added in the JDK's endorsed directory. This results in a class loading conflict with the agent:
ClassCastException: com.sun.xml.messaging.saaj.soap.ver1_1.Message1_1Impl
cannot be cast to oracle.j2ee.ws.saaj.soap.MessageImpl
Ensure that your connectivity agent is running with a JDK installation that is not modified because of use with other products.

Connectivity Agent Log File Location

The connectivity agent agent-diagnostic0.log file is available under agenthome/logs.

Error When Using the Connectivity Agent in an Oracle Integration Classic (User-Managed) Environment

If you are using Oracle Integration Classic (user-managed) and agent interactions fail with the following error:

java.sql.SQLException: ORA-03146: Invalid buffer length for TTC field 

You must apply the following patch to the Oracle Database Cloud Service instance used with your Oracle Integration instance.

  1. Go to http://support.oracle.com and obtain patch 26482376.

  2. Apply the patch to the Oracle Database Cloud Service instance.

  3. Run the following command against the database. (Note that running this command helps even without applying the patch.)

    alter system set events '24921 trace name context forever, level=105989

Agent Behavior in a Decommissioned Instance or HTTP 404/401 Error Response Codes

The following code in the logs indicates that agent run time message processing has halted. This occurs if an HTTP 404/401 error code is received by the agent for a continuous period of 24 hours. The decommission of an Oracle Integration instance also triggers this behavior. When the conditions leading to this error have been resolved, the agent must be restarted manually. See Restart the Agent.
[2018-10-13T04:30:13.501Z] [SEVERE] [ThreadID: 18] [Logger:
oracle.cloud.cpi.agent.transport.AQRuntimeConsumer] [SRC Class:
oracle.cloud.cpi.agent.transport.AQRuntimeConsumer; Method: run] Terminate
flag activated. Signalling termination of agent runtime poller thread with Id

CPI_RUNTIME_REQ_RES_CHANNEL_worker_0.

Verify Endpoint Accessibility when Agent is Installed with a Proxy Host

When the agent is installed with a proxy host, carefully check that the endpoint to access through the agent is reachable through the proxy host. If it is not reachable through the proxy host, you must configure the on-premises endpoint host in the proxy_NON_PROXY_HOSTS parameter of the Agent_Installation_Location/agenthome/agent/config /CpiAgent.properties file.

Unlock the Agent Group

When an agent group is in edit mode and the browser crashes, the agent group becomes locked, which prevents it from being edited. This results in the following error:

ICS-10507: The agent group cannot be updated because it is locked.

To unlock the agent group:

  1. Log in again as the same user who was editing the agent group when the browser crashed, then log out. This action unlocks the agent group.

or

  1. Wait 30 minutes for the lock to expire after the timeout starts.

Failure to Send a Response Due to a java.net.SocketException: Connection Reset Error

For connectivity agent installations running on an Oracle Integration Classic VM and connecting to Oracle Integration (running on Oracle Cloud Infrastructure), the design time and runtime operation involving the connectivity agent sometime fails with a java.net.SocketException: Connection reset error.

This can occur because of a Maximum Transmission Unit (MTU) mismatch.

Here is the complete error:

[2019-01-03T16:35:12.670Z] [SEVERE] [ThreadID: 50] [Logger:
oracle.cloud.cpi.agent.transport.CpiAgentAQTransporter] [SRC Class:
oracle.cloud.cpi.agent.transport.CpiAgentAQTransporter; Method:
sendOneWayPacket] Exception while sending response back to
ICSjava.net.SocketException: Connection reset
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException:
Connection reset
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConne
 
ctionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)

To make this connection work, set the MTU of the Oracle Integration Classic VM (where the agent is installed) to 1500 from the current value of 8900.

Perform the following steps:

  1. Run ifconfig -a as the root user (sudo) and note down the network interface.
  2. Run the following command as the root user (sudo) for the network interface (assuming the network interface is eth0):
    ifconfig eth0 mtu 1500

Anytime the agent VM is restarted (note that it is not an agent restart, but the host where the agent is installed), the changes must be done for the network interface before restarting the agent.

Agent Installation on Linux Fails When Using an Installer Copied Using winscp

During connectivity agent installation in Linux environments, the installation sometimes fails with the following error:

On premise agent is throwing the following error: java.lang.RuntimeException:

Agent Startup Failed - java.lang.IllegalArgumentException: URI is not
absolute 

Installation failure occurs for the following reason:

  1. The agent is to be installed in a Linux environment.
  2. The agent installer ZIP file is downloaded in a Windows environment and transferred to a Linux environment for installation using a Windows tool called winscp.
  3. Even when the binary option is enabled in the winscp tool, the installer ZIP used to run and install the agent fails with the above error.

As a workaround, perform the following steps:

  1. Download the agent installer to a Linux environment directly and do not transfer it from a Windows environment.
  2. If the Oracle Integration user interface is accessible from a Linux environment, use the download install option provided on the Agents page.
  3. If the Oracle Integration user interface instance is not accessible, use the following REST command to download the installer to a Linux environment:
    curl -k -v -X GET -u OIC_user:OIC_password 
    -H 'Content-Type:application/json'
    'https://OIC_URL:443/icsapis/v1/agent/binaries/connectivity' 
    -o download_location/oic_connectivity_agent.zip

Agent Performance Tuning

Only modify the agentWorkerThreads property in the Agent_Installation_Location/agenthome/agent/config/CpiAgent.properties file when Oracle Integration has been scaled out to handle additional loads. In that case, the agent can be tuned to handle additional loads by changing the agentWorkerThreads property value. The maximum value that can be assigned to each agent is 10.

Integration Activation Error Due to Change in JDK Location

If integration activation fails with the following error, this is likely the result of the agent installation using a JDK whose location has been changed (for example, removed). This can occur if the agent was installed and running with a version of the JDK whose location was removed and a newer version was installed in a different location. If the JDK installed with the agent is removed, ensure that you restart the agent with the newer version (and location) of the JDK.
Caused by: java.lang.Error: Circular loading of installed providers 
detected at 
java.nio.file.spi.FileSystemProvider.installedProviders(FileSystemProvider.jav 
a:161)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:324)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:276)
at
oracle.cloud.cpi.agent.store.StoreUtils.extractZipBundle(StoreUtils.java:49)
at
oracle.cloud.cpi.agent.store.FileAgentStore.addActivation(FileAgentStore.java:
40)
at
oracle.cloud.cpi.agent.ActivationCpiCommand.doRequest(ActivationCpiCommand.jav 
a:80) 

Polling Flows on the Connectivity Agent Deactivated Due to Oracle Integration Being Quiesced or Unavailable

If Oracle Integration is quiesced or unavailable, any polling flows on the connectivity agent (database, JMS, file, and so on) used as a trigger are deactivated. The flows are then reactivated when Oracle Integration is unquiesced or becomes available again for message processing. However, if Oracle Integration remains unavailable for more than five minutes, the polling flows are deactivated on the agent side. The connectivity agent must be restarted for the trigger endpoints to be reactivated and resume polling of EIS endpoints.

Troubleshoot Network Connectivity Issues

Ensure that network connectivity is working correctly when persistent connectivity failures are encountered with an Oracle Integration instance.

Note:

This does not apply to any intermittent failures because the connectivity agent is resilient to temporary conditions and recovers when the situation is resolved.
  • Run the following command:
    nslookup hostname
  • Run the following command for a period of five minutes to also capture any transient failures:
    while true;
    do
    curl https://Oracle_Integration_hostname/ic/home >> file.txt 2>&1;
    sleep 1;
    echo "trying again";
    done;

HTTP 401 Unauthorized Error Occurs During Connectivity Agent Installation

If connectivity agent installation or restart fails with an HTTP 401 Unauthorized error, it implies the username/password used to bootstrap the connectivity agent are incorrect. Ensure that the username/password credentials being used to install/restart the connectivity agent are valid. You can use postman or curl to access the following API and ensure that it succeeds with an HTTP 200 response for the username/password specified. That eliminates any bad username/password combination you may currently be using.
  • Option 1 - Use postman or curl:

    curl -k -X GET -u user:password https://Oracle_Integration_Host/icsapis/v2/environment 

    If the above API succeeds with an HTTP 200 response, use the same username and password to also bootstrap the connectivity agent.

  • Option 1 - Use a Chrome browser or any other browser to run the following:
    https://Oracle_Integration_Host/icsapis/v2/environment