Troubleshoot APM Java Agent Deployment
Installation Issues
Connection Exception
During installation, the APM Java Agent issues several network requests.
Error while accessing the server: java .net.ConnectException: Connection timed out
ERROR: Agent configuration download failed
This error message indicates that the installer tried to issue a request to the OMC server or gateway, but did not receive any response. Usually the reason for that is that network traffic to the server should go through a proxy. The APM Java Agent Installer takes proxy settings from the following parameters: -ph, -pp, -pt
. Note that the APM Java Agent Installer does not take into account environment variables like HTTP_PROXY_HOST,
etc.
Troubleshooting — Agent Startup
APM Java Agent reports Remote certificate is not trusted
When installing the APM Java Agent, if the agent reports that the remote certificate is not trusted, create and add a remote certificate.
The remote certificate is the certificate presented to the agent, usually by Oracle Management Cloud, during the agent's attempt to establish an SSL connection. However, if the agent traffic goes through an intermediary, then, it is the certificate of the intermediary (for example, proxy) that might run into this error.
The certificate is included in the agent log information, and can be used to create a .cer
file.
If the Java Agent's logs contain the SEVERE message Remote certificate is not trusted, AND if the Java agent's traffic goes through a proxy which presents a certificate not signed by a well-known certification authority, then, add the proxy's certificate(s) to the agent's trust list:
-
Edit your
AgentStartup.properties
file which it's located under<APM agent install dir>/config
folder. -
In the
pathToCertificates
property, add the full path to your proxy certificate.cer
file. -
Save the file.
-
Restart your application server.
If you don't have the proxy certificate handy, you could also copy it from the agent's log:
-
From the agent’s log file, copy the block of lines starting with the line
and ending with the line-----BEGIN REMOTE CERTIFICATE-----
-----END REMOTE CERTIFICATE-----
-
Delete the word
REMOTE
(along with the trailing space) from both theBEGIN
line and theEND
line. -
Delete lines 2-6 (the information that describes the certificate).
-
Save the file as a
.cer
file.The contents of the
.cer
file will look like this:-----BEGIN CERTIFICATE----- certificate base64 content -----END CERTIFICATE-----
Check for Agent trace in your Container log
If you have correctly provisioned the APM Agent on your container, the following traces should be visible in your container's log at startup.
Oracle WebLogic:
-
Check the container console log for the following lines:
APM agent - preprocessing initialized APM agent - log directory location is /Users/JC/Oracle/wls12130/user_projects/domains/agentDomain/apmagent/logs/AdminServer
-
Check if the agent log files (For example,
AgentStartup.log
) are being created.
If the above tasks are not being performed, verify your agent installation. Check if the APM Agent is added to the server startup script.
JAVA_OPTIONS="${JAVA_OPTIONS} -javaagent:$DOMAIN_HOME/apmagent/lib/system/ApmAgentInstrumentation.jar"
If the above lines are missing, add them to the container startup script and restart.Apache TomCat:
-
Check if the file
catalina.out
contains lines similar to the following:APM agent - preprocessing initialized APM agent - log directory location is /scratch/tomcat/apache-tomee-plus-1.7.2/apmagent/logs/tomcat_instance
-
Ensure that the
-javaagent
parameter is passed at startup. To do that, check if the filecatalina.sh
contains the following lines:CATALINA_OPTS="${CATALINA_OPTS} -javaagent:${CATALINA_HOME}/apmagent/lib/system/ApmAgentInstrumentation.jar"
Container does not start
AgentStartup.log
file.0xb<2015-06-25T14:08:10.169+0200> INFO <STARTUP> The container is still initializing and is therefore not ready for observation processing
0xb<2015-06-25T14:08:10.169+0200> INFO <STARTUP> Another message will get logged once thecontainer is ready and agent initialization can start
0xb<2015-06-25T14:08:10.169+0200> INFO <STARTUP> If you do not see such a message, pleasecheck the container startup logs.
These traces will be followed by a message explaining the container status. If there is any container startup issue, the container output logs (and not the agent logs) will contain information about the problem. If the container waits too long to start, the agent fails its initialization sequence.
By default, the APM Agent waits a maximum of 10 minutes for the container to be ready. You can increase this time by changing the system property oracle.apmaas.agent.container.startupWaitTime
in the AgentStartup.properties file. This property is specified in seconds.
SSL connection fails
0xb<2015-06-25T15:17:04.968+0200> INFO <common.agentToEngine.transport> Read custom certificate from /var/opt/ORCLemaas/sec/cert.cer
0xb<2015-06-25T15:17:04.968+0200> WARNING <common.agentToEngine.transport> Remote certificate is not trusted
[SubjectDN=CN=*.example.com]
[IssuerDN = CN=*.example.com]
[NotBefore = Thu May 21 22:43:40 CEST 2015]
[NotAfter = Sun May 18 22:43:40 CEST 2025]
[SerialNumber = b5d3145ced001866f475ecdde44cbd58] <Ref: GBIZRBMPLYN3AVXZWNJDUTSJBINCBRQI>
0xb<2015-06-25T15:17:04.969+0200> INFO <common.agentToEngine.transport>
-----BEGIN CERTIFICATE-----
MIIBszCCARwCEQC10xRc7QAYZvR17N3kTL1YMA0GCSqGSIb3DQEBCwUAMBoxGDAWBgNVBAMMDyoudXMub3JhY2xlLmNvbTAeFw0xNTA1MjEyMDQzNDBaFw0yNTA1
MTgyMDQzNDBaMBoxGDAWBgNVBAMMDyoudXMub3JhY2xlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAi/Y/58x4NGOeToiHn7b+T/QjpG7ZutA1by3x
f71Y8qvvFDO7AD1VsrG464YkauinR/DQOCovhvXwyYG/HnRE2SCVS9nOTotve37QaD92Bs6Mt0Gku1/2X3HYa6JxjQ+l4VwmhItYFEMPMfe0ZHtQpz+44psQxOS1
rT402EIA0DsCAwEAATANBgkqhkiG9w0BAQsFAAOBgQAqBSxip3+yjX3j5gk4OButP8b9S3Qbl1pR1KwWx22NCuSW7a8KL3C+BPQPtR0YpxxgMC4F/VOGkEkOBrjY
mG7fULYU8f7ab8ck6oHHdl0CPztp/mxRDpWSizBNKlUCSThxKqvSVEtEZrsh5zhn0VofiRlbZwZBWu4C5ObbjvZ8iw==
-----END CERTIFICATE-----
This enables you to compare the remote certificate with the trusted certificate(s) used by the APM Agent. If you can trust the remote certificate, and want to bypass the trust check, define the property oracle.apmaas.common.trustRemoteSSLHost
, and set it to true
in the AgentStartup.properties
file.
Unsupported Cryptographic Protocol
OMC uses TLS 1.2 for SSL cryptographic protocol. An INFO message is logged with the cryptographic protocols support by the current JVM.
0xb<2018-02-21T19:49:53.007+0000> INFO <common.agentToEngine.transport> Supported crypto protocols: [TLSv1]
0xb<2018-02-21T19:49:54.852+0000> WARNING <common.agentToEngine.transport> Error connecting to https://<Oracle_cloud_host_name>/static/regmanager/agents
Unable to establish SSL connection to destination server. The current Java version may not support TLS 1.2 cryptographic protocol.
(set -Djavax.net.debug=ssl to confirm this, since this exception could happen for other reasons too)
If confirmed, the solution is to either update Java to a version that supports TLS 1.2 (Java 1.6.0_121 or later) OR
send the agent traffic through an Oracle Gateway Agent
<javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure> <Ref: LWT7DYU6HP2DVWY4EQWJX7ZZIA5SML5A>
-
Upgrade Java to at least Java 1.6.0_b121
-
Route Agent traffic through an Oracle Gateway Agent.
Communication fails
If there is any communication failure, it gets logged as a WARNING, for example:
0xb<2015-06-25T14:08:26.407+0200> WARNING <agentToEngine.emaas> Response NOT OK ServiceInfo: RegistryService - agent: null ,tenant: apm_testtenantx1 serviceName: DataReceiver.storage ,version: null Transport info: HTTP method: GET ,URL:https://abc.com:4443/registry/instances?status=UP&serviceName=DataReceiver.storage ,response status: 503 ,response headers: null=HTTP/1.1 503 Service Unavailable , X-92eeb115-fa68-449e-9df7-c2d3ec508ca0-reroute= , Content-Language=en , Access-Control-Allow-Headers=Origin, X-Requested-With, Content-Type, Accept,X-USER-IDENTITY-DOMAIN-NAME,Authorization,x-sso-client , X-ORACLE-DMS-ECID=0056LPv4ofAEWNI_IpWByf0002M^0000_d , Access-Control-Allow-Origin=* , Date=Thu, 25 Jun 2015 12:08:26 GMT , Content-Length=0 , Set-Cookie=_WL_AUTHCOOKIE_EMCS_JSESSIONID=lNpT3Z9FK9OqhZ2lSqQS; path=/registry; secure; HttpOnly EMCS_JSESSIONID=WUYqnnu9LFdZXmiWulLknPsGK0pIP_T1QaFgvNvPol0Jwre8OJtF!-1121177946; expires=Thu, 25-Jun-2015 12:08:36 GMT; path=/registry; secure; HttpOnly , Access-Control-Allow-Methods=GET, OPTIONS, HEAD , Connection=close , Content-Type=application/json; qs=1 <Ref: IQWJEECVYYIRYOFG5HDJUJW43RYOTU4W>
Also, additional messages that are more user-friendly, might get logged:
0xb<2015-06-25T14:08:26.408+0200> WARNING <PROCESSING> The registry service replied to the agent with an error code. Please check the log and service status for more information. <Ref: 74RXY6IPUAFSBWSROR2VXMEGJRI7HEIU>
0xb<2015-06-25T14:08:26.408+0200> WARNING <PROCESSING> The agent could not get a data receiver address from the registry service location 'https://abc.com:4443/registry/'. <Ref: Z5GXIKFE6C52BKKWHGXOQYQMCGCEYQB3>
If the agent cannot connect to the remote services, the agent cannot complete its initialization and will keep retrying until communication channels are working.
0x37<2015-06-25T15:17:04.983+0200> WARNING <STARTUP> No managed entity Id could be retrieved from the target model and security service. Since the agent needs a meId to be operational, the agent will now keep trying to lookup a value. A message will get logged once it succeeds. No traffic will get monitored until then. <Ref: MX5B6MBD4UJIVP2JDFJUNU65AHHA4FLE>
0xb<2015-06-25T15:17:05.984+0200> INFO <STARTUP> Agent startup is waiting for the full initialization of its communications with remote services, and successful delivery of its core observations. Another message will get logged when the agent is operational and its initialization complete.
0x37<2015-06-25T15:22:30.751+0200> INFO <bootstrap> Agent still trying to lookup an managed entity id value from the target model and security service. Agent still not operational.
0x37<2015-06-25T15:28:05.325+0200> INFO <bootstrap> Agent still trying to lookup an managed entity id value from the target model and security service. Agent still not operational.
0x37<2015-06-25T15:33:40.481+0200> INFO <bootstrap> Agent still trying to lookup an managed entity id value from the target model and security service. Agent still not operational.
0x37<2015-06-25T15:39:17.542+0200> INFO <bootstrap> Successfully looked up a managed entity Id: 63EB5524C11743EEA47C09C3CBB94CB6
0x37<2015-06-25T15:39:17.542+0200> INFO <bootstrap> Agent core discovery observations successfully sent
0x37<2015-06-25T15:39:17.542+0200> INFO <bootstrap> Agent startup successfully completed - the agent is now operational and monitoring traffic
RegistrationKey not correct
The agent might start by trying to get its existing managed entity ID, assuming that it already registered during a prior startup (bold INFO log below). If it fails or if the agent just notices that it needs to perform an initial registration, it tries to register. A failure to register (read INFO below), is the sign of an invalid registration key.
0xb<2015-06-26T15:20:09.134+0200> WARNING <agentToEngine.emaas> Response NOT OK ServiceInfo: SecurityServiceRegister - agent: null ,tenant: apm_testtenantx1 ,registrationKey: pTaz5UiPcLbnKvxyVayD4V ,entityName: null ,clientId: null Transport info: HTTP method: POST ,URL: https://abc.com:4443/microservice/96122404-13cf-46cd-a9fb-afdeb4a1df21/agents ,response status: 404 ,response headers: null=HTTP/1.1 404 Not Found , Content-Language=en , X-ORACLE-DMS-ECID=0056MiPMKrWDGfQ_I_T4if0006dN00003u , Transfer-Encoding=chunked , Date=Fri, 26 Jun 2015 13:20:08 GMT , Keep-Alive=timeout=5, max=100 , Connection=Keep-Alive , Content-Type=application/json Transport content: Received status 404 from dependent service http://abc.com:7001/clientservices-persistence/registration = 404, Check service logs for string =CS-1435324808534 for more details <Ref: 37DZJ3LK64ACGZOWPFFBNFFPFYJ5UXSO>
0xb<2015-06-26T15:20:09.135+0200> INFO <agentToEngine.emaas> The agent could not get a managed entity ID value from the Security Service (returned status: 404) - the agent might not be registered. Trying to register now.
0x1c<2015-06-26T15:20:09.325+0200> INFO <ACTION.JAXWS> JAXWS probe adding server side handler
0xb<2015-06-26T15:20:09.565+0200> WARNING <agentToEngine.emaas> Response NOT OK ServiceInfo: SecurityServiceRegister - agent: null ,tenant: apm_testtenantx1 ,registrationKey: pTaz5UiPcLbnKvxyVayD4V ,entityName: null ,clientId: null Transport info: HTTP method: POST ,URL: https://abc.com:4443/microservice/96122404-13cf-46cd-a9fb-afdeb4a1df21/agents ,response status: 500 ,response headers: null=HTTP/1.1 500 Internal Server Error , Content-Language=en , X-ORACLE-DMS-ECID=0056MiPPlRxDGfQ_I_T4if0006dN00003v , Transfer-Encoding=chunked , Date=Fri, 26 Jun 2015 13:20:09 GMT , Connection=close , Content-Type=application/json Transport content: Received status 400 from dependent service http://abc.com:7001/targetmodel/api/v1/data/mes, Check service logs for string =CS-1435324809465 for more details <Ref: NEXSTWGNGQTZNVFAUTP4DPQNIYILIXUB>
0xb<2015-06-26T15:20:09.565+0200> INFO <agentToEngine.emaas> Registration attempt to the Security Service did not return a managed entity ID. Will keep retrying.
0xb<2015-06-26T15:20:09.565+0200> WARNING <PROCESSING> The agent failed getting a managed identity Id - please check the logs for additional information. <Ref: P223WCKDF6KETCRSVCFVTWGS6ZO2MW2Q>
The registration key is specified in the AgentStartup.properties
file, and you can change its value if the registration key is not correct.
Invalid credentials
If credentials to authenticate OMC are not correct, a transport message WARNING gets logged, and HTTP 401 status is returned. Depending on your setup, the credentials will be either located within a wallet or encrypted within the AgentHttpBasic.properties
file.
0xb<2015-06-27T06:38:49.697+0200> WARNING <agentToEngine.emaas> Http credentials were not authorized to access the service. Will attempt to read credentials again ServiceInfo: RegistryService - agent: null ,tenant: apm_testtenantx1 serviceName: DataReceiver.storage ,version: null Transport info: HTTP method: GET ,URL:https://abc.com:4443/registry/instances?status=UP&serviceName=DataReceiver.storage ,response status: 401 ,response headers: null=HTTP/1.1 401 Unauthorized , Content-Language=en , WWW-Authenticate=Basic realm="weblogic" , Date=Sat, 27 Jun 2015 04:38:49 GMT , Content-Length=1468 , Keep-Alive=timeout=5, max=100 , Connection=Keep-Alive , Content-Type=text/html; charset=UTF-8 <Ref: DPKGO5GY2GMNIMOOV7FLSKTMQCLU2FGU>
0xb<2015-06-27T06:38:49.699+0200> WARNING <PROCESSING> The agent could not authenticate to the registry service. Make sure that the credentials specified are correct. There is no need to restart the container if you update the agent credentials as the agent will keep trying to connect until it succeeds, using the more recent set of available credentials. <Ref: XD2BEOVFLOGTE5XK6PH3HFBNUPWYX6EF>
Note that the remote service might have a lockout period. Fixing credentials to the correct values might not be sufficient to reconnect immediately. Wait for the lockout period to expire before the agent can reconnect.
OSGi (Open Services Gateway initiative) property setting
If the application you would like to monitor has a dependency on OSGi, make these manual settings to ensure proper framework boot delegation so that the application that is being monitored does not break.
-
WebLogic Server: On your WebLogic Server, ensure the monitored OSGi framework instances have the Java system property,
oracle.apmaas.*
added to theFramework Boot delegation
property as follows:-Dorg.osgi.framework.bootdelegation=oracle.apmaas.*
Refer to the WebLogic documentation on ways to change the WebLogic OSGi settings.
-
Atlassian JIRA Felix OSGi container (Tomcat): Add the following option to the JIRA container's startup options:
-Datlassian.org.osgi.framework.bootdelegation=oracle.apmaas.*,sun.*,org.apache.xerces,org.apache.xerces.*,org.apache.naming,org.apache.naming.*,org.apache.catalina,org.apache.catalina.*
Logs not created
If APM Agent logs are not created even when the application is running, check if the -javaagent
option for ApmAgentInstrumentation.jar
was added correctly to the server startup command.
Security Access errors while starting APM Java Agent
If you run the APM Java Agent with a Java security manager and see an error message with the following content:
java.security.AccessControlException: access denied()
or
access denied()
grant codeBase "file:<path_to>/apmagent/-" { permission java.security.AllPermission; };
Unable to open TomEE service during installation
While configuring APM Agent as a Windows Service on TomEE, and you run TomEE.exe
and see the following error:
The specified service does not exist as an installed service. Unable to open the service 'TomEE'.
This means that the Windows service name is not the default value, that is, TomEE.
Workaround: Specify the exact service name you have provided for TomEE in the command prompt:
TomEE.exe//ES//<service_name>
Unable to get OAuth Token from IDCS Server
If the agent startup log shows that it cannot get the initial OAuth authentication token, preceded by a warning showing a failure to reach the IDCS server, check to ensure that there is no firewall blocking access to the IDCS server. If there is a firewall, you will need to allow access to the IDCS server.
Trust Manager or Trust Anchor related errors
javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error:
java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be
non-empty
then, the trust store may be invalid.Workaround: Check if the property javax.net.ssl.trustStore
is being passed. If yes, check that the full path to the trust store is specified. If yes, check the trust store's validity using the JDK Keytool utility.
Could not Generate DH Keypair
If APM Java agent provisioning fails with an SSL exception with the error Could not generate DH keypair, this issue could be due to a JDK bug that has been fixed. Check the version of your JDK, and update your JDK to a patch level that resolves this issue (for example, this problem happens with JDK 1.7.0_65, and updating to 1.7.0_201 fixes the issue).
Websphere Application Server doesn't start after uninstalling APM Agent
-javaagent
is specified correctly:
- In an editor, open the file
$WAS_HOME/config/cells/<celll-name>/nodes/<node-name>/servers/<server-name>/server.xml
. - Search for
genericJvmArguments
and look for the-javaagent
option. - Remove the
-javaagent
option, and save the file. - Replace the current
server.policy
startup script of your WebSphere server with the original one you had before installing the APM agent. - Restart the Websphere server.
Oracle Forms monitoring is not working after deploying APM Java Agent
If Oracle Forms monitoring is not working after deploying APM Java Agent, you can check the log file: AgentErrors.log
and look for the following errors:
- Connect timed out message:
Unable to POST to collector due to IOException: connect timed out Exception in thread "main" java.net.SocketTimeoutException: connect timed out
This error message indicates that you might need a proxy server. To fix it, add the following parameters to the fileAgentStartup.properties
which it's located under<APM agent install dir>/config
folder.oracle.apmaas.common.proxyHost = my-proxy.example.com oracle.apmaas.common.proxyPort = 80
- Handshake failure message:
Unable to POST to collector due to IOException: Received fatal alert: handshake_failure Exception in thread "main" javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
This error message indicates that you might be running an older JDK version. Collector needs you to use TLSv1.1 or TLSv1.2 to connect it. Please ensure that your JDK support any of these TLS versions and then set it up doing the following:oracle.apmaas.agent.forms.tlsProtocol = TLSv1.2
- No valid certificate message:
Unable to POST to collector due to IOException: java.security.cert.CertificateException: No valid server certificate found Exception in thread "main" javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No valid server certificate found
This error message indicates that you do not have the correct collector certificate in the
certificates
folder. Try downloading the certificate from the collector and place it in the folder:apmagent/config/certificates
.You can download the certificate via a browser by navigating to the collector URL and saving it as a DER encoded binary file.
- If Forms Name is not configured then the
AgentErrors.log
file reports the following error:0x4b<2019-11-27T05:59:50.304-0800> WARNING <HANDLER.FORMS> <005a5Hcz7E2Dg^0_rx9DiY00057l000kDc> FormWindow Message detected without INDEX_FORM_MODULE property. Please enable the Oracle Forms to send forms name by setting its property FORMS_RUEI_SEND_FORM_NAME=TRUE in your Oracle Forms environment. This can be set in file 'default.env' by your Forms Administrator. Please refer to APM agent documentation for more info on this configuration <Ref: ZHZDUUVYBRU7XVFZG6R5LF64Q3JMJCAF>
Spring Boot 2.2 with Tomcat is not being detected by APM Agent
<2020-05-19T21:11:56.478+0000> SEVERE <STARTUP> Failed to get container information after waiting for 600 seconds <Ref: WZ7PXNQDOUY5PK4YALTWGAZ4UHVLWUKG>
<2020-05-19T21:11:56.487+0000> SEVERE <STARTUP> Agent failed to start <Ref: ZQNZ2FKNIJJCZTD4DUWDCAXKKNE4W3ZS>
-
If you are using Spring Boot 2.2 with Tomcat, enter the following two properties in the application.properties file:
spring.jmx.enabled=true server.tomcat.mbeanregistry.enabled=true
By default, the application.properties file is located inside the spring-boot app executable jar, under the
BOOT-INF/classes
directory. Spring boot allows you to have many locations for this file and multiple formats. For more details, see Spring Boot Application Property Files.For information about the Spring Boot 2.2, see Spring Boot 2.2 Release Notes.
-
If you are using other Application Server, you can force the APM Agent to use a specific application server name by setting the custom value provided using the Custom AppServer feature.
To activate the Custom AppServer feature, do the following:
-
Set up the following Java system property in the Java startup argument:
oracle.apmaas.agent.custom.appserver.name
When
oracle.apmaas.agent.custom.appserver.name
property is specified, Java APM Agent will look for the custom-appserver.properties file in the server config directory such asapmagent/config/<dir_name>/custom-appserver.properties
. -
Create the
custom-appserver.properties
file if it doesn't already exist.If
custom-appserver.properties
file exists, Java APM Agent uses it to populate the app server container details, and discovers the app server based on the provided details.If
custom-appserver.properties
file does not exist, Java APM Agent assumes this is a J2SE application with default properties and a J2SE app server will be discovered.The
custom-appserver.properties
file should be created manually before Java APM Agent is run.
custom-appserver.properties file
The
custom-appserver.properties
file has the following properties:Property Name Defaults for J2SE Defaults fro custom-appserver.properties file type "Java SE" No default name System.getProperty("oracle.apmaas.agent.custom.appserver.name") + "(" + System.getProperty("user.dir") + ")" No default version RuntimeMXBean.getSpecVersion() vendor RuntimeMXBean.getVmVendor() path System.getProperty("user.dir") System.getProperty("user.dir") ports sslPorts Sample ofcustom-appserver.properties
filetype=Jetty name=My Jetty Sandbox version=9.2.5 vendor=Eclipse ports=8080 sslPorts=8443
-