11 Troubleshooting JVM Diagnostics

This chapter describes the errors you may encounter while deploying and using JVM Diagnostics and the workaround steps you can follow to resolve each of them. It contains the following sections:

Section 11.1, "Cross Tier Functionality Errors"
Section 11.2, "Trace Errors"
Section 11.3, "Deployment Script Execution Errors"
Section 11.4, "LoadHeap Errors"
Section 11.5, "Errors on JVM Diagnostics UI Pages"
Section 11.6, "Frequently Asked Questions"

11.1 Cross Tier Functionality Errors

This section lists the errors that show the status of the JVM Diagnostics Manager.

Table 11-1 Cross Tier Functionality Errors

Error Message	Workaround Steps
DBWait link not displayed on JVM Threads Real Time Analysis page. No data displayed in Top DBStates / SQLs tables.	Cross tier functionality errors may occur due the following: Incorrect database credentials Database Agent errors If the database credentials are incorrect: Navigate to Middleware > JVM Diagnostics > Setup > Databases page and verify the credentials for the registered database to be monitored. Enter the id of user who has installed the database in the OS User field. Specify the database application user credentials in the DB User field. Specify the database system user credentials in the DB User (Explain Plan) field. If database agent errors occur, ensure that the database agent is running on a machine on which the database is installed with the correct IP address and port number. Navigate to the Threads > Real Time Analysis page for the JVM. If you see a thread that is in the DB Wait state, it should be a hyperlink. Click on the hyperlink to drill down to the database details and view the database session information including the SQL statements executed. If the DB Wait state is not a hyperlink, navigate to the Middleware > JVM Diagnostics > Setup page and set the Cross Tier Log Level to 6 and send JVMD Manager logs report issue.

Error Message

Workaround Steps

DBWait link not displayed on JVM Threads Real
Time Analysis page.

No data displayed in Top DBStates / SQLs tables.

Cross tier functionality errors may occur due the following:

Incorrect database credentials
Database Agent errors

If the database credentials are incorrect:

Navigate to Middleware > JVM Diagnostics > Setup > Databases page and verify the credentials for the registered database to be monitored.
Enter the id of user who has installed the database in the OS User field.
Specify the database application user credentials in the DB User field.
Specify the database system user credentials in the DB User (Explain Plan) field.

If database agent errors occur, ensure that the database agent is running on a machine on which the database is installed with the correct IP address and port number.

Navigate to the Threads > Real Time Analysis page for the JVM. If you see a thread that is in the DB Wait state, it should be a hyperlink. Click on the hyperlink to drill down to the database details and view the database session information including the SQL statements executed.
If the DB Wait state is not a hyperlink, navigate to the Middleware > JVM Diagnostics > Setup page and set the Cross Tier Log Level to 6 and send JVMD Manager logs report issue.

11.2 Trace Errors

This section lists errors that occur during tracing.

Table 11-2 Trace Errors

Error Message	Workaround Steps
weblogic.transaction.internal.TimedOutException: Transaction timed out after 30 seconds	This error occurs if the Poll Duration has a large value and results in a timeout. This error does not affect the Trace functionality and can be ignored.

Error Message

Workaround Steps

weblogic.transaction.internal.TimedOutException:
Transaction timed out after 30 seconds

This error occurs if the Poll Duration has a large value and results in a timeout.

This error does not affect the Trace functionality and can be ignored.

11.3 Deployment Script Execution Errors

This section lists the errors that occur when you run the deployment script.

Table 11-3 Script Execution Errors

Error Message	Workaround Steps
ScriptException: Error occured while performing deploy: The action you performed timed out after 600,000 milliseconds	This error occurs when you are deploying the JVM Diagnostics Agent. To resolve this issue, check if the lock for the target WebLogic domain Administration Console has already been acquired. If it has been acquired, release it and run the script again. Login to the WebLogic Administration Console: http://<machine address>:<webogic port>/console. If you see the Activate Changes and Undo All Changes buttons in the left pane, click on these buttons to clear them. If the buttons are not cleared, click Undo All Changes and run the script again.

Error Message

Workaround Steps

ScriptException: Error occured while performing
deploy: The action you performed timed out after
600,000 milliseconds

This error occurs when you are deploying the JVM Diagnostics Agent. To resolve this issue, check if the lock for the target WebLogic domain Administration Console has already been acquired. If it has been acquired, release it and run the script again.

Login to the WebLogic Administration Console: http://<machine address>:<webogic port>/console.
If you see the Activate Changes and Undo All Changes buttons in the left pane, click on these buttons to clear them. If the buttons are not cleared, click Undo All Changes and run the script again.

11.4 LoadHeap Errors

This section lists loadheap errors.

Table 11-4 LoadHeap Errors

Error Message	Workaround Steps
glibc detected * free(): invalid next size (fast): 0x0965d090" ./loadheap.sh: line 237: 32357 Aborted ./bin/${bindir}/processlog in=$infile hdr=${sumdata} obj=${objdata} rel=${reldata} root=${rootdata} osum=${objsumdata} rrel=${rootrel} heap=${heap _id} skip=$skipgarbage db=$dbtype $* Error processing file /tmp/heapdump6.txt	Check if the heapdump operation has been successfully completed. Open the `heapdump6.txt` file and check if there is a `heapdump finished` string at the end of the file. If you see this string, load the finished dump file.
Heapdump already in progress, cannot take another heapdump	Check if the `heapdump` operation has been successfully completed. Open the `heapdump6.txt` file and check if there is a `heapdump finished` string at the end of the file.
loadheap.sh created unusable unique indexes.	Run the `loadheap/sql/cleanup.sql` shipped with `loadheap.zip` to fix the unique indexes.

11.5 Errors on JVM Diagnostics UI Pages

This section lists the user interface errors.

Table 11-5 JVM Diagnostics UI Page Errors

Error Message Workaround Steps

Error Message	Workaround Steps
JAM Console: Socket timed out after recv -- client adc2100083.us.oracle.com:7001 is not Active [0] secs JAM Console jamlooptimeout = [3]JAM CONSOLE: JVM 1 is not active JAM Cons Err Processing Request: 128 JVM 1 is not active jamDAL: jamreq returned 128 return status < 0 from jamDalInst.processRequest	To resolve this error, increase the Agent Request Timeout (secs) and Agent Loop Request Timeout (secs)
Agent is up and running but is not displayed in the real time pages.	If the log file shows `JAMMANAGER: OLD AGENT or NULL POOl or wrong jamoptimization level`, this indicates that the old jamagent/Dbagent is being used. To resolve this issue, download the latest `jamagent` from Setup > Download page.
You do not have the necessary privileges to view this page	Ensure that you have the required JVM Diagnostics Administrator or User privileges to view the JVM Diagnostics data.

JAM Console: Socket timed out after recv --
client adc2100083.us.oracle.com:7001 is not
Active [0] secs JAM Console jamlooptimeout =
[3]JAM CONSOLE: JVM 1 is not active JAM Cons Err
Processing Request: 128 JVM 1 is not active
jamDAL: jamreq returned 128 return status < 0
from jamDalInst.processRequest

To resolve this error, increase the Agent Request Timeout (secs) and Agent Loop Request Timeout (secs)

Agent is up and running but is not displayed in
the real time pages.

If the log file shows JAMMANAGER: OLD AGENT or NULL POOl or wrong jamoptimization level, this indicates that the old jamagent/Dbagent is being used.

To resolve this issue, download the latest jamagent from Setup > Download page.

You do not have the necessary privileges to view
this page

Ensure that you have the required JVM Diagnostics Administrator or User privileges to view the JVM Diagnostics data.

11.6 Frequently Asked Questions

This section lists some of the questions you may have while using JVM Diagnostics. It includes the following:

Location of the JVM Diagnostics Logs
JVM Diagnostics Manager Status
JVM Diagnostics Agent Status
Monitoring Status
Creating Less Privileged Users
Usage of Try Changing Threads Parameter
Significance of Optimization Levels
Manually Deploying the JVM Diagnostics Agent
Log Manager Level
Repository Space Requirements

11.6.1 Location of the JVM Diagnostics Logs

You can find the JVM Diagnostics logs in the following locations:

The JVM Diagnostics Manager Log file is located at $EMGC_JVMDMANAGER1/logs/EMGC_JVMDMANAGER1.out
UI related errors are logged in:
- $T_WORK/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
- $T_WORK/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.log
Communication errors between the JVM Diagnostics Manager and the Console are logged in $T_WORK/gc_inst/em/EMGC_OMS1/sysman/log/emoms.log

11.6.2 JVM Diagnostics Manager Status

To check the status of the JVM Diagnostics Manager:

Click Middleware > JVM Diagnostics > Setup. In the Console Setup page, check if the status of the JVM Diagnostics Manager is Up or Down.
Check the status of all Managers at Middleware > JVM Diagnostics > Setup > JVMs & Managers page.
Check the JVM Diagnostics Agent log file to verify the connection between Agent and the Manager. If you see an error - JAM Agent ERROR: Cannot connect to Console:Connection refused, this indicates that the JVM Diagnostics Manager is not running.
Check if the message JAM Console: Agent connection from:[Hostname] is present in the JVM Diagnostics Manager log file. If this message appears, it indicates that the JVM Diagnostics Manager is running and is connected to the Agent.

11.6.3 JVM Diagnostics Agent Status

To check the status of the JVM Diagnostics Agent:

Click Middleware > JVM Diagnostics > Threads > Real Time Analysis to view the pool to which the JVM belongs. Check the JVM Status in the Connected JVMs table.
- If the status is Not Active, this indicates that the Agent is not connected to the Manager. Check the agent logs to verify if it is running and the IP address and port number of the Manager is correct.
- If the status is No AD4J Agent Deployed, the JVM Diagnostics Agent must be deployed on that JVM.
Navigate to the Threads > Real Analysis page for the JVM. If the JVM Diagnostics Agent is running, the active threads data must be visible. If the JVM Diagnostics Agent is not running, you will see a message - JVM is inactive, Please try again after some time.

11.6.4 Monitoring Status

To verify if the JVM Diagnostics Manager is monitoring the data:

Click Middleware >JVM Diagnostics > Setup. In the Console Setup page, verify that the Enable Monitoring checkbox is checked.
Navigate to the Monitoring page under Setup and check if monitoring status is On for the Pool to which the JVM being monitored belongs.
Navigate to the JVM Pools page under Setup and verify if the Poll Enabled checkbox has been checked for the Pool to which the JVM being monitored belongs. Monitoring should now be enabled.

11.6.5 Running the create_jvm_diagnostic_db_user.sh Script

You can run the create_jvm_diagnostic_db_user.sh script if you want to create less privileged users who can only load heaps using the loadHeap script.

11.6.6 Usage of the Try Changing Threads Parameter

This parameter should be used only when the JVM is highly active.

11.6.7 Significance of Optimization Levels

The JVM Diagnostics Agent supports three optimization levels:

Level 0 indicates that the JVM Diagnostics Agent is using a JVMTI based engine. This level is supported for JDK 6 series on almost all supported platforms.
Level 1 is a hybrid between level 0 and level 2. It is supported only for very few JDKs on selected platforms.
Level 2 uses Runtime Object Analysis technique for monitoring as it is efficient at run time.

11.6.8 Custom Provisioning Agent Deployment

You can customize the JVMD Agent deployment in the production environment by running custom provisioning scripts.

After the OMS has been installed, the jvmd.zip file can be found in the plugins/oracle.sysman.emas.oms.plugin_12.1.0.0.0 directory in the Middleware installation directory. The zip file contains a set of scripts in the customprov directory. Details on using these scripts are described in the README.TXT present in the same directory. To use the custom provisioning scripts, follow these steps:

Login to Enterprise Manager and download the jamagent.war from the Setup > Middleware Diagnostics > Setup JVM Diagnostics > Downloads.
Make a copy of the deployment profile that includes the location of the downloaded jamagent.war, domains, and server details.
Run the perl script on the deployment profile which will deploy the JVMD Agent to all the specified servers.

11.6.9 Log Manager Level

The default log manager level is 3. You can temporarily increase this to a higher level if you encounter some issues. Log levels 1 to 5 are supported where:

1 - Error
2 - Warning
3 - Info
4 - Debug
5 - Trace

11.6.10 Repository Space Requirements

For monitoring data, Oracle recommends 50 MB per JVM per day with the default setting of a 24 hour purge interval. This amount can vary based upon runtime factors (e.g depth of call stacks, etc.) within your environment. Hence, you must check the tablespace growth periodically and if required, you may need to change the space requirements. This will ensure that database growth due to standard monitoring will occur smoothly without sudden spikes. Tablespace sizing can be affected by the following:

Heap Dumps: Analyzing heaps requires a large amount of tablespace. As a standard practice, we recommend that you must have 5 times the size of heap dump file being loaded in your tablespace. Since you know the size of your dump file, make sure that there is adequate space to accommodate the dump file before it is loaded into the database.
Thread Traces: While these are smaller than heaps. they are loaded into the database automatically when a user initiates a trace at the console. The size of these threads can vary dramatically depending on the number of active threads during the trace, the duration of the trace, and the sample interval of the trace. This should usually be under 100MB but if several thread traces have been initiated, it could fill up the database quickly. Before initiating the traces, you must ensure that there is adequate space in the database.