Troubleshoot the Connectivity Agent
This section describes how to troubleshoot and resolve connectivity agent issues in Oracle Integration.
Topics:
- Troubleshoot Agent Unavailable Issues
- Unable to Start Connectivity Agent Because Agent Status Cannot Be Obtained
- Connectivity Agent Startup Fails With a "java.lang.RuntimeException: Agent Startup Failed - Unable to get agent status" Error
- Connectivity Agent Installation Error if /etc/hosts File is Not Configured
- Restrictions on Using the Stage File Action with the Connectivity Agent
- Connectivity Agent Installation Fails When Run as a Federated User
-
Add or Change the Non-Proxy Host Configuration After Installation in the CpiAgent.properties File
-
Class Loading Conflict When Sharing the JDK Instance with Another Product
-
Error When Using the Connectivity Agent in an Oracle Integration Classic (User-Managed) Environment
-
Agent Behavior in a Decommissioned Instance or HTTP 404/401 Error Response Codes
-
Verify Endpoint Accessibility when Agent is Installed with a Proxy Host
-
Failure to Send a Response Due to java.net.SocketException: Connection Reset Error
-
Agent Installation on Linux Fails When Using an Installer Copied Using winscp
- Integration Activation Error Due to Change in JDK Location
- Polling Flows on the Connectivity Agent Deactivated Due to Oracle Integration Being Quiesced or Unavailable
- Troubleshoot Network Connectivity Issues
- HTTP 401 Unauthorized Error Occurs During Connectivity Agent Installation
Troubleshoot Agent Unavailable Issues
Connectivity agent status can be determined by selecting Observability > Agents to access the Monitor agents page. If the agent status is displayed as unavailable, follow this troubleshooting workflow.
Number | Step | Details |
---|---|---|
1 |
Check the agent process is running |
Check whether the agent is running. Run one of these commands from the host where the agent is installed. If
you don't see Linux:
Windows: If the agent
process is not running, start it:Linux: Start the connectivity agent as a background process:
Windows:
Note: Starting the agent as a Windows service is not supported.If you cannot start the agent, continue to check network connectivity. If the agent process is running, restart it: Follow these steps to restart the agent. |
2 | Check network connectivity |
Check that the agent host can connect to the Oracle Integration instance: On the host where the agent is installed, run this command, replacing with your own path:
If the response is 302 Found, the agent is able to resolve the Oracle Integration instance host. You have network connectivity. Continue to check user credentials. If the response is not 302 Found, there may be network issues. Check the
|
3 | Check user credentials | Check that the user account used to run the agent has a valid user name and
password:
If the response to checking user credentials is 200, the user name and password used by the agent are valid. There may be something else wrong. Check the diagnostic log for errors:
If the response to checking user credentials
is not 200, the credentials the agent is using aren't
valid. The user may be locked, the password may have expired, or
the Reset user credentials and start the agent:
|
4 |
Log a Service Request |
If you've gone through the troubleshooting steps, can't find anything in the diagnostic log, and the agent status is still unavailable, create a service request with Oracle Support. Log in, select the Service Requests tab, and click Create Technical SR. Include the diagnostic log in the service request. |
Unable to Start Connectivity Agent Because Agent Status Cannot Be Obtained
java -jar connectivityagent.jar &
[Logger: oracle.cloud.cpi.agent.util.CpiUtils] [SRC Class: oracle.cloud.cpi.agent.util.CpiUtils;
Method: checkIfAlreadyRunning] Agent Instance status check failed. Response received -
[Logger: oracle.cloud.cpi.agent.ProxyCpiAgentLoader] [SRC Class: oracle.cloud.cpi.agent.ProxyCpiAgentLoader;
Method: initialize] Agent Startup Failed java.lang.RuntimeException: Unable to get agent status :
at oracle.cloud.cpi.agent.util.CpiUtils.checkIfAlreadyRunning(CpiUtils.java:1058)
at oracle.cloud.cpi.agent.ProxyCpiAgentLoader.initialize(ProxyCpiAgentLoader.java:114)
at oracle.cloud.cpi.agent.ProxyCpiAgentLoader.runAgent(ProxyCpiAgentLoader.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.cloud.cpi.agent.CpiAgentLoader.startAgent(CpiAgentLoader.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.cloud.cpi.agent.CpiBootstrapLoader.startAgent(CpiBootstrapLoader.java:183)
at oracle.cloud.cpi.agent.AgentStartCommandTask.run(CpiBootstrapLoader.java:427)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[Logger: oracle.cloud.cpi.agent.AgentStartCommandTask] [SRC Class: oracle.cloud.cpi.agent.AgentStartCommandTask; Method: run]
Error during agent install/startup or while re-starting agent after upgrade: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.cloud.cpi.agent.CpiBootstrapLoader.startAgent(CpiBootstrapLoader.java:183)
at oracle.cloud.cpi.agent.AgentStartCommandTask.run(CpiBootstrapLoader.java:427)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.cloud.cpi.agent.CpiAgentLoader.startAgent(CpiAgentLoader.java:27)
... 13 more
Caused by: java.lang.RuntimeException: Agent Startup Failed - Unable to get agent status
- Run the following command using the credentials specified in
the
InstallerProfile.cfg
file during connectivity agent installation:curl -v -k -X GET -u uid:pwd https://Oracle_Integration_URL:443/ic/api/integration/v1/environment
If the
curl
command returns a401 Unauthorized
error, that means the server is reachable and has returned a response indicating that invalid credentials were sent to the server. - Update the credentials in the
InstallerProfile.cfg
file:- Back up
InstallerProfile.cfg
. - Change the user name and password fields to values used with your earlier REST test. Ensure that you can log in to Oracle Integration using these credentials.
oic_USER=Oracle_Integration_username oic_PASSWORD=Oracle_Integration_password
- Back up
- Save the file.
- Restart the connectivity agent.
Upon a successful restart, the connectivity agent updates the entries with encrypted values.
Connectivity Agent Startup Fails With a "java.lang.RuntimeException: Agent Startup Failed - Unable to get agent status" Error
[2021-06-19T10:06:06.567Z] [SEVERE] [ThreadID: 1] [Logger: oracle.cloud.cpi.agent.CpiBootstrapLoader]
[SRC Class: oracle.cloud.cpi.agent.CpiBootstrapLoader; Method: main] Error encountered !!! java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException
at oracle.cloud.cpi.agent.AgentStartCommandTask.run(CpiBootstrapLoader.java:452)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.cloud.cpi.agent.CpiBootstrapLoader.startAgent(CpiBootstrapLoader.java:192)
at oracle.cloud.cpi.agent.AgentStartCommandTask.run(CpiBootstrapLoader.java:436)
...
...
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.cloud.cpi.agent.CpiAgentLoader.startAgent(CpiAgentLoader.java:27)
...
...
Caused by: java.lang.RuntimeException: Agent Startup Failed - Unable to get agent status :
at oracle.cloud.cpi.agent.ProxyCpiAgentLoader.initialize(ProxyCpiAgentLoader.java:151)
at oracle.cloud.cpi.agent.ProxyCpiAgentLoader.runAgent(ProxyCpiAgentLoader.java:53)
...
...
The issue occurs because the user account configured for the connectivity agent is locked or the password has expired.
- Enter the following
curl
command and see whether you receive a status200
response. Ensure that you use the exact same user name and password as used for the connectivity agent configuration.curl -v -k -X GET -u uid:pwd https://Oracle_Integration_URL:443/ic/api/integration/v1/environment
If the above
curl
command response is not200
, there is an issue with the user configured in theInstallerProfile.cfg
file. - Check the user account in the Oracle Cloud Infrastructure Console and whether the user is locked or the password has expired. Try logging in to Oracle Integration with the exact user name and password credentials and ensure that the user can get into Oracle Integration.
- Once the user can get into Oracle Integration, update the
oic_USER
andoic_PASSWORD
parameters in theInstallerProfile.cfg
file with the correct values and restart the agent. This addresses the issue.
Connectivity Agent Installation Error if /etc/hosts File is Not Configured
Caused by: java.net.UnknownHostException: : Unknown
name or service
Resolve this error by adding an entry for the IP address and host name in
the /etc/hosts
file of the host name/VM.
Restrictions on Using the Stage File Action with the Connectivity Agent
Be aware of several restrictions when using some operations of the stage file action with the connectivity agent. See Restrictions on Using Stage File Action Operations with the File/Attachment Features of the Connectivity Agent.
Connectivity Agent Installation Fails When Run as a Federated User
Note:
If you ever need to restart the connectivity agent, you must ensure that the user name/password credentials for this user are still valid.Java Memory Errors
There is insufficient memory for the Java Runtime Environment to continue.
-----------------------------
Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x00007f6847afd000, 12288, 0) failed; error='Cannot
allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing
reserved memory.
# An error report file with more information is saved as:
However, this issue is unrelated to the out-of-memory error that is sometimes observed when the Java heap is not large enough.
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map
This may be related to resources on the physical machine/virtual machine/server on which the agent is running. As an example, when the agent is running on the same virtual machine as the database, and the database may be consuming most of the resources. Oracle recommends that you set up the agent on a separate compute.
Authorization Error When Restarting the Agent
If you manually stop the agent and attempt to restart it, and receive an
authorization error, ensure that the user name and password used to start the agent
in the InstallerProfile.cfg
file are correct. This error can occur
if the password for this user name expired and was changed in the Oracle Cloud
Infrastructure Console by the administrator, but was not updated in the
InstallerProfile.cfg
file. This task is only required if
you manually stop and restart the agent. This task is not required for agent
upgrades, which occur automatically and do not use these credentials.
Add or Change the Non-Proxy Host Configuration After Installation in the CpiAgent.properties File
If you need to add, change, or bypass the non-proxy host configuration after agent installation, do not edit the InstallerConfing.cfg
file. Proxy host changes made to that file after agent installation do not take effect. Instead, update the host with the proxy_nonProxyHosts
parameter in the Agent_Installation_Location/agenthome/agent/config/CpiAgent.properties
file for your changes to take effect. After editing this file, restart the agent.
Class Loading Conflict When Sharing the JDK Instance with Another Product
ClassCastException: com.sun.xml.messaging.saaj.soap.ver1_1.Message1_1Impl
cannot be cast to oracle.j2ee.ws.saaj.soap.MessageImpl
Ensure that your connectivity agent is running with a JDK installation that is not modified because of use with other products.Connectivity Agent Log File Location
The connectivity agent agent-diagnostic0.log
file is available under agenthome/logs
.
Error When Using the Connectivity Agent in an Oracle Integration Classic (User-Managed) Environment
If you are using Oracle Integration Classic (user-managed) and agent interactions fail with the following error:
java.sql.SQLException: ORA-03146: Invalid buffer length for TTC field
You must apply the following patch to the Oracle Database Cloud Service instance used with your Oracle Integration instance.
-
Go to http://support.oracle.com and obtain patch 26482376.
-
Apply the patch to the Oracle Database Cloud Service instance.
-
Run the following command against the database. (Note that running this command helps even without applying the patch.)
alter system set events '24921 trace name context forever, level=105989
Agent Behavior in a Decommissioned Instance or HTTP 404/401 Error Response Codes
404
/401
error code
is continuously received by the agent for 24 hours. The decommission of an Oracle Integration instance also triggers this behavior. When the conditions
leading to this error have been resolved, the agent must be restarted manually. See
Restart the Connectivity Agent.[2018-10-13T04:30:13.501Z] [SEVERE] [ThreadID: 18] [Logger:
oracle.cloud.cpi.agent.transport.AQRuntimeConsumer] [SRC Class:
oracle.cloud.cpi.agent.transport.AQRuntimeConsumer; Method: run] Terminate
flag activated. Signalling termination of agent runtime poller thread with Id
CPI_RUNTIME_REQ_RES_CHANNEL_worker_0.
Verify Endpoint Accessibility when Agent is Installed with a Proxy Host
When the agent is installed with a proxy host, carefully check that the endpoint to access through the agent is reachable through the proxy host. If it is not reachable through the proxy host, you must configure the on-premises endpoint host in the proxy_NON_PROXY_HOSTS
parameter of the Agent_Installation_Location/agenthome/agent/config /CpiAgent.properties
file.
Unlock the Agent Group
When an agent group is in edit mode and the browser crashes, the agent group becomes locked, which prevents it from being edited. This results in the following error:
ICS-10507: The agent group cannot be updated because it is locked.
To unlock the agent group:
-
Log in again as the same user who was editing the agent group when the browser crashed, then log out. This action unlocks the agent group.
or
- Wait 30 minutes for the lock to expire after the timeout starts.
Failure to Send a Response Due to a java.net.SocketException: Connection Reset Error
For connectivity agent installations running on an Oracle Integration Classic VM and connecting to Oracle Integration (running on
Oracle Cloud Infrastructure), the design-time and runtime operation involving the
connectivity agent sometimes fails with a java.net.SocketException:
Connection reset
error.
This can occur because of a Maximum Transmission Unit (MTU) mismatch.
Here is the complete error:
[2019-01-03T16:35:12.670Z] [SEVERE] [ThreadID: 50] [Logger:
oracle.cloud.cpi.agent.transport.CpiAgentAQTransporter] [SRC Class:
oracle.cloud.cpi.agent.transport.CpiAgentAQTransporter; Method:
sendOneWayPacket] Exception while sending response back to
ICSjava.net.SocketException: Connection reset
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException:
Connection reset
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConne
ctionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
To make this connection work, set the MTU of the Oracle Integration Classic VM (where the agent is installed) to 1500
from the current value of 8900
.
Perform the following steps:
- Run
ifconfig -a
as the root user (sudo) and note down the network interface. - Run the following command as the root user (sudo) for the network interface (assuming the network interface is
eth0
):ifconfig eth0 mtu 1500
Anytime the agent VM is restarted (note that it is not an agent restart, but the host where the agent is installed), the changes must be done for the network interface before restarting the agent.
Agent Installation on Linux Fails When Using an Installer Copied Using winscp
During connectivity agent installation in Linux environments, the installation sometimes fails with the following error:
On premise agent is throwing the following error: java.lang.RuntimeException:
Agent Startup Failed - java.lang.IllegalArgumentException: URI is not
absolute
Installation failure occurs for the following reason:
- The agent is to be installed in a Linux environment.
- The agent installer ZIP file is downloaded in a Windows environment and transferred to a Linux environment for installation using a Windows tool called
winscp
. - Even when the binary option is enabled in the
winscp
tool, the installer ZIP used to run and install the agent fails with the above error.
As a workaround, perform the following steps:
- Download the agent installer to a Linux environment directly and do not transfer it from a Windows environment.
- If the Oracle Integration user interface is accessible from a Linux environment, use the download install option provided on the Agents page.
- If the Oracle Integration user interface instance is not accessible, use the
following REST command to download the installer to a Linux
environment:
curl -k -v -X GET -u OIC_user:OIC_password -H 'Content-Type:application/json' 'https://OIC_URL:443/icsapis/v1/agent/binaries/connectivity' -o download_location/oic_connectivity_agent.zip
Integration Activation Error Due to Change in JDK Location
Caused by: java.lang.Error: Circular loading of installed providers
detected at
java.nio.file.spi.FileSystemProvider.installedProviders(FileSystemProvider.jav
a:161)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:324)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:276)
at
oracle.cloud.cpi.agent.store.StoreUtils.extractZipBundle(StoreUtils.java:49)
at
oracle.cloud.cpi.agent.store.FileAgentStore.addActivation(FileAgentStore.java:
40)
at
oracle.cloud.cpi.agent.ActivationCpiCommand.doRequest(ActivationCpiCommand.jav
a:80)
Polling Flows on the Connectivity Agent Deactivated Due to Oracle Integration Being Quiesced or Unavailable
If Oracle Integration is quiesced or unavailable, any polling flows on the connectivity agent (database, JMS, file, and so on) used as a trigger are deactivated. The flows are then reactivated when Oracle Integration is unquiesced or becomes available again for message processing. However, if Oracle Integration remains unavailable for more than five minutes, the polling flows are deactivated on the agent side. Restart the connectivity agent to reactivate the trigger endpoints and resume polling of EIS endpoints.
Troubleshoot Network Connectivity Issues
Note:
This does not apply to any intermittent failures because the connectivity agent is resilient to temporary conditions and recovers when the situation is resolved.- Run the following command:
nslookup hostname
- Run the following command for a period of five minutes to also capture any transient failures:
while true; do curl https://Oracle_Integration_hostname/ic/home >> file.txt 2>&1; sleep 1; echo "trying again"; done;
HTTP 401 Unauthorized Error Occurs During Connectivity Agent Installation
HTTP 401
Unauthorized
error, it implies the user name/password used to bootstrap
the connectivity agent are incorrect. Ensure that the user name/password credentials
used to install/restart the connectivity agent are valid. You can use
postman
or curl
to access the following API
and ensure that it succeeds with an HTTP 200
response for the user
name/password specified. That eliminates any bad user name/password combination you
may currently be using.
-
Option 1 - Use
postman
orcurl
:curl -k -X GET -u user:password https://Oracle_Integration_Host/ic/api/integration/v1/environment
If the above API succeeds with an
HTTP 200
response, use the same user name and password to also bootstrap the connectivity agent. - Option 2 - Use a Chrome browser or any other browser to run the
following:
https://Oracle_Integration_Host/ic/api/integration/v1/environment