| Oracle® Enterprise Manager Cloud Control Getting Started with Oracle Fusion Middleware Management 12c Release 2 (12.1.0.2) Part Number E24215-04 |
|
|
PDF · Mobi · ePub |
| Oracle® Enterprise Manager Cloud Control Getting Started with Oracle Fusion Middleware Management 12c Release 2 (12.1.0.2) Part Number E24215-04 |
|
|
PDF · Mobi · ePub |
This chapter describes the errors you may encounter while deploying and using JVM Diagnostics and the workaround steps you can follow to resolve each of them. It contains the following sections:
This section lists the errors that show the status of the JVM Diagnostics Engine. Cross tier functionality errors may occur due the following:
Incorrect database credentials
Database Agent errors
Mismatched database connection information
Error: DBWait link not displayed on JVM Threads Real Time Analysis page. No data displayed in Top DBStates / SQLs tables.
Solution:
If the database credentials are incorrect:
From the Setup menu, select Middleware Diagnostics, then select Setup JVM Diagnostics in the Middleware Diagnostics page. Click on the Register Databases tab and verify the credentials for the registered database to be monitored.
Enter the id of user who has installed the database in the OS User field.
Specify the database application user credentials in the DB User field.
Specify the database system user credentials in the DB User (Explain Plan) field.
If database agent errors occur, ensure that the database agent is running on a machine on which the database is installed with the correct IP address and port number.
Select the Live Thread Analysis option from the Java Virtual Machine menu. If you see a thread that is in the DB Wait state, it should be a hyperlink. Click on the hyperlink to drill down to the database details and view the database session information including the SQL statements executed.
If the DB Wait state is not a hyperlink, from the Setup menu, select Application Performance Management, select the JVM Diagnostics Engine row and click Configure. Set the Cross Tier Log Level to 6 temporarily when the application is making DB calls and send JVMD Manager logs report issue
If the error occurs due to mismatched database connection information, check if the database has been registered. If it has been registered, click the DBWait link to examine the JDBC connection string. The details need to match the registration information. For example, if it shows SID in JDBC connection string, the database registered needs to have SID; and the same goes for service_name. Another example of such information that requires matching is the hostname of the database.
This section lists errors that occur during tracing. The following error occurs if the Poll Duration has a large value and causes a timeout.
Error: weblogic.transaction.internal.TimedOutException: Transaction timed out after 30 seconds.
Solution: This error does not affect the Trace functionality and can be ignored.
This section lists the errors that occur when you run the deployment script.
Error: Script Exception: Error occurred while performing deploy: The action you performed timed out after 600,000 milliseconds.
Solution: To resolve this issue, check if the lock for the target WebLogic domain Administration Console has already been acquired. If it has been acquired, release it and run the script again by following these steps:
Login to the WebLogic Administration Console: http://<machine address>:<webogic port>/console.
If you see the Activate Changes and Undo All Changes buttons in the left pane, click on these buttons to clear them. If the buttons are not cleared, click Undo All Changes and run the script again.
Error: If the user name and password for the WebLogic Administration Server are incorrect, you may see the following error:
Caused by: java.lang.SecurityException: User: <username>, failed to be authenticated.
This message is typically embedded in a long error message trail.
You may also see the following exception:
javax.naming.AuthenticationException [Root exception is java.lang.SecurityException: User: weblogic, failed to be authenticated.] at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslat or.java:42) at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:788) at weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialContextFact oryDelegate.java:682) atweblogic.jndi.WLInitialContextFactoryDelegate.newContext(WLInitialContextFactoryDelegate.java:469) at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialConte xtFactoryDelegate.java:376) at weblogic.jndi.Environment.getContext(Environment.java:315) at weblogic.jndi.Environment.getContext(Environment.java:285) at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactor y.java:117) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:235) at javax.naming.InitialContext.initializeDefaultInitCtx(InitialContext.java:318) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:348) at javax.naming.InitialContext.internalInit(InitialContext.java:286) at javax.naming.InitialContext.<init>(InitialContext.java:211)
Solution: Enter the correct user name and password for the WebLogic Administration Server and run the script again.
Error: If the path to the weblogic.jar (on the Management Agent) as specified in the agent auto deployment page is incorrect, or if the user does not have read permissions on the weblogic.jar (typically seen on hosted machines with the name aime), the following exception may occur:
Exception in thread "main" java.lang.NoClassDefFoundError: javax/enterprise/deploy/spi/exceptions/TargetException Caused by: java.lang.ClassNotFoundException: javax.enterprise.deploy.spi.exceptions.TargetException at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Error: If the WebLogic Administration Console is locked, the agent deployment job may not work as expected. You will see a message that the agent.log files cannot be deployment since the WebLogic Domain is locked.
Solution: JVM Diagnostics Agents are deployed by using t3/t3s protocols. Make sure thet3/t3s ports are open.
Error: If you are deploying to an SSL enabled WebLogic Domain using the demo certificate, you may see an error if the WebLogic Server demo certificate has not been imported to the keystore.
Solution: You must import the WebLogic Server demo certificate to the keystore of the Management Agent.
Error: While copying the deployer.zip or javadiagnosticagent.ear files, errors like broken pipe appear.
Solution: The Oracle Management Service and the Management Agent must be installed by the same user or users belonging to the same group.
Error: Agent deployment fails on WebLogic 10.0.
Solution: To deploy the JVM Diagnostics Agent, weblogic.jar and MANIFEST.MF (of the weblogic.jar), are required. If, in some WebLogic 10.0 installations, some of the jar files are missing, agent deployment will fail.
Error: JVMD AGENT DEPLOYMENT FAILED FOR WEBLOGIC 9.2 TARGET.
The following exception occurs:
EM Agent home : /scratch/aime/agsh_0819/core/12.1.0.2.0 MIDDLEWARE_HOME : /scratch/aime/mw923 IS_WEBLOGIC9 : true em agent state dir : /scratch/aime/agsh_0819/agent_inst acsera home : /tmp/ad4j_1345730608009/4910760210525348050 wls admin url : t3://adc6160336.us.oracle.com:7001 wls username : weblogic target : AdminServer? weblogic jar path : /scratch/aime/mw923/weblogic92/server/lib/weblogic.jar&&ls /scratch/aime/mw923/weblogic92/server/lib/wljmxclient.jar&&ls /scratch/aime/mw923/weblogic92/server/lib/wlcipher.jar application name : HttpDeployer? agent keystore location : /scratch/aime/agsh_0819/agent_inst/sysman/config/montrust/AgentTrust.jks Command used for deployment: /scratch/aime/agsh_0819/core/12.1.0.2.0/jdk/bin/java -cp /tmp/ad4j_1345730608009/4910760210525348050/ADPAgent/lib/mips.jar:/scratch/aim e/mw923/weblogic92/server/lib/weblogic.jar&&ls /scratch/aime/mw923/weblogic92/server/lib/wljmxclient.jar&&ls /scratch/aime/mw923/weblogic92/server/lib/wlcipher.jar -Dweblogic.security.SSL.ignoreHostnameVerify=true -Djava.security.egd=file:/dev/./urandom -Dweblogic.security.SSL.trustedCAKeyStore=/scratch/aime/agsh_0819/agent_inst/ sysman/config/montrust/AgentTrust.jks-Dsun.lang.ClassLoader.allowArraySyntax= true -Dbea.home=/scratch/aime/mw923 com.acsera.ejb.Deployer.RemoteHttpDeployerShell -deploy -adminurl t3://adc6160336.us.oracle.com:7001 -upload -source /tmp/ad4j_1345730608009/4910760210525348050/ADPAgent/deploy/HttpDeployer.ear -targets AdminServer? -username weblogic -name HttpDeployer? -usenonexclusivelock
The application will be first undeployed on the targeted server
Usage: java [-options] class [args...]
(to execute a class)
or java [-options] -jar jarfile
(to execute a jar file)
where options include:
d32 use a 32-bit data model if available
-d64 use a 64-bit data model if available -client to select the "client" VM -server to select the "server" VM -hotspot is a synonym for the "client" VM [deprecated] The default VM is server, because you are running on a server-class machine. -cp <class search path of directories and zip/jar files> -classpath <class search path of directories and zip/jar files> A : separated list of directories, JAR archives, and ZIP archives to search for class files. -D<name>=<value> set a system property -verbose[:class|gc|jni] enable verbose output -version print product version and exit -version:<value> require the specified version to run -showversion print product version and continue -jre-restrict-search | -jre-no-restrict-search include/exclude user private JREs in the version search -? -help print this help message -X print help on non-standard options -ea[:<packagename>...|:<classname>] -enableassertions[:<packagename>...|:<classname>] enable assertions -da[:<packagename>...|:<classname>] -disableassertions[:<packagename>...|:<classname>] disable assertions -esa | -enablesystemassertions enable system assertions -dsa | -disablesystemassertions disable system assertions -agentlib:<libname>[=<options>] load native agent library <libname>, e.g. -agentlib:hprof see also, -agentlib:jdwp=help and -agentlib:hprof=help -agentpath:<pathname>[=<options>] load native agent library by full pathname -javaagent:<jarpath>[=<options>] load Java programming language agent, see java.lang.instrument -splash:<imagepath> show splash screen with specified image /scratch/aime/mw923/weblogic92/server/lib/wljmxclient.jar ls: invalid line width: eblogic.security.SSL.ignoreHostnameVerify=true Status returned from the java process is 512
This section lists loadheap errors.
Error: The following error occurs during the heapdump operation.
glibc detected * free(): invalid next size (fast): 0x0965d090" ./loadheap.sh:
line 237: 32357 Aborted ./bin/${bindir}/processlog in=$infile hdr=${sumdata}
obj=${objdata} rel=${reldata} root=${rootdata} osum=${objsumdata}
rrel=${rootrel} heap=${heap_id} skip=$skipgarbage db=$dbtype $* Error
processing file /tmp/heapdump6.txt
Solution: Check if the heapdump operation has been successfully completed. Open the heapdump6.txt file and check if there is a heapdump finished string at the end of the file. If you see this string, load the finished dump file.
Error: Heapdump already in progress, cannot take another heapdump.
Solution: Check if the heapdump operation has been successfully completed. Open the heapdump6.txt file and check if there is a heapdump finished string at the end of the file.
Error: loadheap.sh created unusable unique indexes.
Solution: Run the loadheap/sql/cleanup.sql shipped with loadheap.zip to fix the unique indexes.
The following error occurs when you try to deploy the JVM Diagnostics Agent on IBM JDK 1.6:
Error: The following can occur when the JVM Diagnostics Agent is deployed on JDK 1.6.
Jam Agent : can_tag_objects capability is not set. Copy /tmp/libjamcapability.so to another directory and restart Java with argument -agentpath: <Absolute path of libjamcapability.so>
Solution: Deploy the latest jamagent.war and add -agentpath:<Absolute path of libjamcapability.so after copying to another directory> to the java arguments.
This message appears only after the JVM Diagnostics Agent has connected to JVM Diagnostics Engine. Secondly, this argument should be a JVM argument (and not a program argument)
If the server is started using the WebLogic Administration Console (through nodemanager). these arguments can be specified in the Administration Console under Server Start. If the server is started from the command line (startWeblogic.sh or startManagedServer.sh), these arguments have to be specified in the startWeblogic.sh. If there are multiple servers, make sure a check for the server name is present in the startWeblogic.sh to ensure that the path for the libjamcapability.so is separate for each server.
A sample entry to be made in startWeblogic.sh is below:
if [ "${SERVER_NAME}" = "AdminServer" ] ; then
echo "********************************************* MODIFIED ADMIN SERVER"
JAVA_OPTIONS="${JAVA_OPTIONS} -agentpath:<Absolute path of
libjamcapability.so.X after copying to another directory>
export JAVA_OPTIONS
fi
The message "Capabilities Added by libjamcapability.so" during server startup (before the jamagent logs appear) confirms that libjamcapability.so was loaded fine.
This section lists the user interface errors.
Error: This is an Agent timeout error:
JAM Console:Socket timed out after recv -- client adc2100083.us.oracle.com:7001 is not Active [0] secs JAM Console jamlooptimeout=[3] JAM CONSOLE: JVM 1 is not active JAM Cons ErrProcessing Request:128 JVM 1 is not active jamDAL: jamreq returned 128 return status < 0 from jamDalInst.processRequest
Solution: To resolve this error, increase the Agent Request Timeout (secs) and Agent Loop Request Timeout (secs).
Error: The JVM Diagnostics Agent is up and running but is not displayed in the real time pages.
Solution: If the log file shows JAMMANAGER: OLD AGENT or NULL POOL or wrong optimization level, this indicates that the old JVM Diagnostics Agent or Dbagent is being used. To resolve this issue, select Application Performance Management from the Setup menu. The list of Application Performance Management Engines is displayed. Select the JVM Diagnostics Engine row, click Configure then click the Register Databases tab. Click the Downloads button in the Registered DB Agents region, and select JVMD Agent from the JVMD Component list. Specify the JVM Diagnostics Agent web.xml Parameters, click Download, then click OK to download the jamagent.war.
Error: You do not have the necessary privileges to view this page.
Solution: Ensure that you have the required JVM Diagnostics Administrator or User privileges to view the JVM Diagnostics data.
This section lists some of the questions you may have while using JVM Diagnostics. It includes the following:
You can find the JVM Diagnostics logs in the following locations:
The JVM Diagnostics Engine Log file is located at $EMGC_JVMDMANAGER1/logs/EMGC_JVMDMANAGER1.out
UI related errors are logged in:
$T_WORK/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
$T_WORK/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.log
Communication errors between the JVM Diagnostics Engine and the Console are logged in $T_WORK/gc_inst/em/EMGC_OMS1/sysman/log/emoms.log
To check the status of the JVM Diagnostics Engine, follow these steps:
From the Setup menu, select Middleware Diagnostics, then click Setup JVM Diagnostics.
Check the JVM Diagnostics Agent log file to verify the connection between Agent and the Manager. If you see an error - JAM Agent ERROR: Cannot connect to Console:Connection refused, this indicates that the JVM Diagnostics Engine is not running.
Check if the message JAM Console: Agent connection from:[Hostname] is present in the JVM Diagnostics Engine log file. If this message appears, it indicates that the JVM Diagnostics Engine is running and is connected to the Agent.
To check the status of the JVM Diagnostics Agent:
From the Targets menu, select Middleware, then click on a Java Virtual Machine target. Select the Live Thread Analysis option from the Java Virtual Machine menu. Check the JVM Status in the Connected JVMs table.
If the status is Not Active, this indicates that the Agent is not connected to the Manager. Check the agent logs to verify if it is running and the IP address and port number of the Manager is correct.
If the status is No JVMD Agent Deployed, the JVM Diagnostics Agent must be deployed on that JVM.
If the JVM Diagnostics Agent is running, the active threads data must be visible. If the JVM Diagnostics Agent is not running, you will see a message - JVM is inactive, Please try again after some time.
To verify if the JVM Diagnostics Engine is monitoring the data:
From the Setup menu, select Middleware Diagnostics, then click Setup JVM Diagnostics in the Middleware Diagnostics page. In the JVMD Configuration page, verify that the Enable Monitoring check box is checked.
Navigate to the Monitoring page under Setup and check if monitoring status is On for the Pool to which the JVM being monitored belongs.
Navigate to the JVM Pools page under Setup and verify if the Poll Enabled check box has been checked for the Pool to which the JVM being monitored belongs. Monitoring should now be enabled.
You can run the create_jvm_diagnostic_db_user.sh script if you want to create less privileged users who can only load heaps using the loadHeap script.
This parameter should be used only when the JVM is highly active.
The JVM Diagnostics Agent supports three optimization levels:
Level 0 indicates that the JVM Diagnostics Agent is using a JVMTI based engine. This level is supported for JDK 6 series on almost all supported platforms.
Level 1 is a hybrid between level 0 and level 2. It is supported only for very few JDKs on selected platforms.
Level 2 uses Runtime Object Analysis technique for monitoring as it is efficient at run time.
You can customize the JVMD Agent deployment in the production environment by running custom provisioning scripts.
After the OMS has been installed, the jvmd.zip file can be found in the plugins/oracle.sysman.emas.oms.plugin_12.1.0.0.0 directory in the Middleware installation directory. The zip file contains a set of scripts in the customprov directory. Details on using these scripts are described in the README.TXT present in the same directory. To use the custom provisioning scripts, follow these steps:
From the Setup menu, select Middleware Diagnostics, then click the Setup JVM Diagnostics in the Middleware Diagnostics page. Click the Register Databases tab and download the jamagent.war file
Make a copy of the deployment profile that includes the location of the downloaded jamagent.war, domains, and server details.
Run the Perl script on the deployment profile which will deploy the JVMD Agent to all the specified servers.
The default log manager level is 3. You can temporarily increase this to a higher level if you encounter some issues. Log levels 1 to 5 are supported where:
1 - Error
2 - Warning
3 - Info
4 - Debug
5 - Trace
For monitoring data, Oracle recommends 50 MB per JVM per day with the default setting of a 24 hour purge interval. This amount can vary based upon runtime factors (e.g depth of call stacks, etc.) within your environment. Hence, you must check the tablespace growth periodically and if required, you may need to change the space requirements. This will ensure that database growth due to standard monitoring will occur smoothly without sudden spikes. Tablespace sizing can be affected by the following:
Heap Dumps: Analyzing heaps requires a large amount of tablespace. As a standard practice, we recommend that you must have 5 times the size of heap dump file being loaded in your tablespace. Since you know the size of your dump file, make sure that there is adequate space to accommodate the dump file before it is loaded into the database.
Thread Traces: While these are smaller than heaps. they are loaded into the database automatically when a user initiates a trace at the console. The size of these threads can vary dramatically depending on the number of active threads during the trace, the duration of the trace, and the sample interval of the trace. This should usually be under 100MB but if several thread traces have been initiated, it could fill up the database quickly. Before initiating the traces, you must ensure that there is adequate space in the database.
|
 Copyright © 2011, 2013, Oracle and/or its affiliates. All rights reserved. Legal Notices |
|