C Troubleshooting Tricks

This topic contains a few known issues in the Advanced Management Console and how to troubleshoot them.

This topic contains the following sections:

Agent

Tips and Tricks to Ensure that the Advanced Management Console Agent Runs Efficiently:

  1. Verify that the Advanced Management Console agent is running.

    1. From the Windows Control Panel, open Services, locate AMC Agent and verify whether it’s running. If not, right click the AMC Agent and select Start.

  2. Verify that the Advanced Management Console agent is communicating. Do the following to verify:

    1. Open C:\ProgramData\Oracle\Java_AMC\agent.log.0

    2. Check the log file to validate that the agent does not have any communication errors to the server. For example:
      2016-06-14 14:54:19 Communication: sending com.oracle.amc.agent.client.GetCommandMessage@1c00735
      2016-06-14 14:54:19 Communication: exception: java.net.UnknownHostException: <amc-server-host> 
      16-06-14 14:54:19 com.oracle.amc.agent.task.GetCommandTask: got communication error; rescheduling try #1
      
    3. Check the log file to validate that the Agent is posting Java Runtime Environment (JRE) installs to the server and that the server is delivering updates to the Agent (for example, (DRS)). To initiate the agent to communicate with the server, right click the AMC Agent in the Services dialog box of the Windows Control Panel, and select Restart.

      Note:

      Starting Advanced Management Console 2.7, the updated Native Launcher has changed the number of times the agent restarts as well as the delay between each retries. For example, if you retry 3 times, reset on the 4th time, and wait for an hour, then it goes back to the 3 retries again. The interval between the 3 retries is 30 seconds. However, agents that have been updated from an older release version of Advanced Management Console don't reflect this behavior.
  3. Verify that the desktop hosting the agent shows up in the Desktops tab of the Advanced Management Console.

  4. Verify that Advanced Management Console Server has pushed a Deployment Rule Set (DRS). Do the following to let the Advanced Management Console update DeploymentRuleSet.jar.

    1. Check the status of DRS pushing command to the desktop in the Status tab of the Advanced Management Console.

    2. Check the agent log in C:\ProgramData\Oracle\Java_AMC\agent.log.0 for messages relating to DRS upload.

    3. From the Java Control Panel, open the Security Tab, and click View the Active Deployment Rule Set to ensure that the most current version of the rules are shown.

    4. If the rules are not updated, then check C:\Windows\Sun\Java\Deployment to locate a DeploymentRuleSet.tmp file. If you see DeploymentRuleSet.tmp in this location, then ensure that all applets and Java Web Start applications are terminated. Restart the AMC Agent service from the Services dialog box of the Windows Control Panel.

      Once the agent is restarted, ensure that the DeploymentRuleSet.tmp file is no longer available and the DeploymentRuleSet.jar is updated to the latest version.
  5. Verify DRS processing:

    1. Set the deployment logging properties in the <USER_DIR>\AppData\LocalLow\Sun\Deployment\deployment.properties as follows:

      deployment.trace=true
      deployment.trace.level=ALL
      
    2. Check the log files in <USER_DIR>\AppData\LocalLow\Sun\Deployment\log.  (Verify both plugin* and javaws* trace files.) Usually, there are two paired trace files produced for an application. One is the log message for the Main plugin Java Version and the other is the trace file for the Java Version defined from the Deployment Rule.

    3. Locate the word ruleset in the log file to debug the ruleset processing and find all the ruleset logging.

      1. Verify that the Deployment Rule can be read by the plugin. If there is a certificate error, then ensure that the deployment.properties file is available in the following location: C:\Windows\Sun\Java\Deployment. Also, ensure that the deployment.properties contains the following property: deployment.user.security.trusted.cacerts=C:\\Windows\\Sun\\Java\Deployment\\drscacerts. The Advanced Management Console agent should automatically update this file.

      2. If only one trace file is produced and you expected more, there is an issue with the browser bits compared to the Java versions installed. For example, the browser is a 32 bit, but only a 64 bit Java is installed. Look for the word Exception in the log file.

  6. Verify Internet Explorer (IE) and Java Bits: IE 11 32 bit and 64 bit is the same application in Windows. To swap between 32 bit and 64 bit, in the Advanced Options tab of the Internet Options dialog box, select the Enable Enhanced Protection Mode* for  64 bit, and ensure that the check box is not selected for 32 bit.

    In Internet Options of IE 11, you can set it as follows:

    1. Go to the Control Panel and open the Action Center. Click Change User Account Control Settings and verify that the slider is not set to Never notify.  If User Account Control is set to Never notify, then the Enhanced Protected Mode can't be enabled.

    2. In the Security tab of Internet Options, verify that the Enable Protected Mode check box is selected.

    3. In the Advanced tab of Internet Options, verify that Enable Enhanced Protection Mode* is enabled.

  7. Reinstall the agent: When you want to re-install Advanced Management Console agent, either keep its \conf folder intact or run AMCAgent -remove. This also saves you from hitting the agent registration limit from the same IP address.

Server

Tips and Tricks to Ensure that the Advanced Management Console Server Runs Efficiently:

  1. You may encounter error messages, such as 404, HTTP_NOT_FOUND while initializing the Advanced Management Console:
    • If you get error messages, such as, Connection refused or Cannot connect to the server while trying to load /amcwebui in the browsers, then the WebLogic server where Advanced Management Console is deployed to is either not started, or is unreachable.

    • If you get a warning message indicating that the page or the server is not trusted by the host, then it means that an unknown SSL certificate is used by the WebLogic Server.

    • If an HTTP Page Not Found (404) error is displayed, then it means that one of the following can be the issue:

      • WebLogic Server is started, but Advanced Management Console is not installed.

      • WebLogic Server is started and Advanced Management Console is installed (deployed), but the Advanced Management Console is not targeted to the correct managed server.

    • If the Advanced Management Console server has been already initialized, but loading /amcwebui still shows the initialization wizard, then there may be a broken database connection. Verify that you can connect to the database from the Advanced Management Console server and that the data source is configured correctly (check whether or not the user credentials are valid), and then restart the Advanced Management Console or the Managed Server, where the Advanced Management Console is deployed to.

    • If the WebLogic Server is started and the Advanced Management Console is installed, but the dependencies, such as the Data Source or JAX-RS 2.0 library are not deployed to the managed servers, then /amcwebui fails to be loaded and an HTTP Page Not Found (404) error is displayed.

    • If you deploy the Advanced Management Console on an unsupported version of the Oracle WebLogic server, then the page gets redirected to the initialization page when loading /amcwebui and a Datasource Cannot be Found error message gets displayed when you continue to initialize the Advanced Management Console manually.

  2. Ensure the following: agent desktops should get registered with the Advanced Management Console and also the Java Usage Tracker records should get registered with the Advanced Management Console. If not, check the WebLogic Server log in the following location: <WebLogicHome>\user_projects\domains\base_domain\servers\AMCServer\logs\AMCServer.log.

  3. If you have an Out of Memory (OOM) exception, then you need to increase the Java Heap Space. By default the WebLogic Server uses 512 m. Set this to at least 4 GB. Set USER_MEM_ARGS=-Xmx4g before you invoke setDomainEnv.cmd.

  4. Certificate issues can occur if system proxy is not defined. To define proxy servers:

    1. If connection to Internet requires proxy server, then set the following properties: http.proxyHosthttp.proxyPorthttp.nonProxyHostshttps.proxyHosthttps.proxyPort. See Setting Up Java Heap Size and Proxy Servers.

  5. At times, there is a very slow network connection to the server and some Advanced Management Console threads can be marked as Stuck by the Oracle WebLogic Server. For example, if you download an agent bundle from the Advanced Management Console UI, it can take more than 10 minutes. It can take a long time, if agents download JREs to install as well.

    • As a workaround to this issue, you need to configure the WebLogic Server to use a greater timeout than (default) 600s, in case you see a lot of threads marked as Stuck. To configure the timeout, in the WebLogic Administration Console, go to the Server, expand Configuration, and then select Overload, and then update Max Stuck Thread Time to a greater value.

Important Directory Locations (In a Windows Environment)

The following is a list of important directory locations:

  • C:\Windows\Sun\Java\Deployment\: If you want to use self-signed certificates, then ensure that the following files are available in the C:\Windows\Sun\Java\Deployment\ location:

    • deployment.properties:

      The content of this file is as follows:
      #Updated by AMC   #Fri Aug 19 08:15:55 EDT 2016   
      deployment.user.security.trusted.cacerts=C\:\\windows\\Sun\\Java\\Deployment\\drscacerts
      
    • drscacerts: The Keystore pointed to in the deployment.properties file containing the self-signed certificate.

    • If you have a signing certificate from a certificate authority, then it’s sufficient if you have the DeploymentRuleSet.jar file in your Windows directory, for example, C:\Windows\Sun\Java\Deployment\DeploymentRuleSet.jar.
  • C:\Users\user\AppData\LocalLow\Sun\Java\Deployment\: The client log files and the deployment.properties file are located in this directory. You can capture additional information by adding the following properties to the deployment.properties file:
    deployment.trace=true
    deployment.trace.level=all
    
  • $JRE_HOME%\lib\management\usagetracker.properties: To troubleshoot the Java Usage Tracker, add the following in the usagetracker.properties file:
    com.oracle.usagetracker.logToFile = ${user.home}/.java_usagetracker
    com.oracle.usagetracker.verbose = true
    
    If you still find issues, then add the properties to C:\Program Files\Java\conf.
  • %ProgramData%\Oracle\Java_AMC\: The Advanced Management Console log conversations with the Advanced Management Console server are available here. The content of the log file is as follows:
    %AMC_HOME%\Middleware\Oracle_Home\user_projects\domains\base_domain\servers\AdminServer\logs\{base_domain.log and AdminServer.log}
    %AMC_HOME%\Middleware\Oracle_Home\user_projects\domains\base_domain\servers\AMC-Server\logs\AMC-Server.log
    

Important Directory Locations (In an OS X Environment)

The following is a list of important directory locations:

  • /Library/Application Support/Oracle/Java: The Desktop-wide Java settings, such as the Java Usage Tracker configuration file (usagetracker.properties) are stored here. The Deployment rule set is in Deployment/<sub-folder>.

  • /Library/Application Support/Oracle/Java_AMC: This is the Advanced Management Console agent folder that also contains log files apart from other folders.

  • ~/Library/Application Support/Oracle/Java/Deployment: This is the user-specific-deployment home directory. Settings from the Java Control Panel are stored for each user in this folder.

  • /Library/Internet Plug-Ins/JavaAppletPlugin.plugin: This folder contains the system-wide JREs that are used to run Java applets in browsers. In an OS X environment, typically, there maybe just one JRE installed.